Re: [Qemu-devel] [PULL 5/6] console: remove do_safe_dpy_refresh

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Laurent Vivier (lvivier@redhat.com) wrote:
> On 18/07/2017 15:07, Laurent Vivier wrote:
> > On 21/06/2017 15:23, Gerd Hoffmann wrote:
> >> Drop the temporary workaround for the broken display updates.
> >> All display adapters are updated, so this should be safe without
> >> causing regressions.
> > 
> > It seems it breaks QMP command 'migrate "exec:cat>mig"'.
> > 
> > The command hangs and doesn't create the file.
> > 
> > It happens with qemu-system-ppc64 on x86 (so TCG mode).
> > 
> > my command:
> > 
> >    ./ppc64-softmmu/qemu-system-ppc64 -serial mon:stdio
> > 
> > I wait SLOF fails to find an OS, and:
> > 
> >     Ctrl-a c
> >     (qemu) migrate -d "exec:cat>mig"
> > 
> > The file is not created and the command hangs:
> > 
> > #0  in __lll_lock_wait
> > #1  in pthread_mutex_lock
> > #2  in qemu_mutex_lock
> > #3  in rcu_init_lock
> > #4  in fork
> > #5  in qemu_fork
> > #6  in qio_channel_command_new_spawn
> > #7  in exec_start_outgoing_migration
> > #8  in qmp_migrate
> > ...
> > 
> > It looks like a deadlock.
> 
> I think this patch is not the cause of the problem, the one it removes
> just unlocks the deadlock by playing with locks.
> 
> We have a rcu_init_lock() on fork() because of:
> 
> utils/rcu.c:
> 
> static void __attribute__((__constructor__)) rcu_init(void)
> {
> #ifdef CONFIG_POSIX
>     pthread_atfork(rcu_init_lock, rcu_init_unlock, rcu_init_unlock);
> #endif
>     rcu_init_complete();
> }
> 
> The QMP thread hangs on:
> 
> (gdb) p rcu_sync_lock
> $1 = {lock = {__data = {__lock = 2, __count = 0, __owner = 23865,
>       __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {
>         __prev = 0x0, __next = 0x0}},
>     __size = "\002\000\000\000\000\000\000\000\071]\000\000\001", '\000'
> <repeats 26 times>, __align = 2}, initialized = true}
> 
> 
> The lock is already taken by thread 2:
> 
> (gdb) info thread
>   Id   Target Id         Frame
>   1    Thread 0x7f1cf02fdf00 (LWP 23864) "qemu-system-ppc"
> 0x00007f1cd914b37d in __lll_lock_wait () from /lib64/libpthread.so.0
> * 2    Thread 0x7f1cc9762700 (LWP 23865) "qemu-system-ppc"
> 0x00007f1cd410daa9 in syscall () from /lib64/libc.so.6
>   3    Thread 0x7f1cbf8d5700 (LWP 23866) "qemu-system-ppc"
> 0x00007f1cd914b37d in __lll_lock_wait () from /lib64/libpthread.so.0
> 
> (gdb) bt
> #0  0x00007f1cd410daa9 in syscall () at /lib64/libc.so.6
> #1  0x000055ab028ddda2 in qemu_futex_wait
> #2  0x000055ab028ddda2 in qemu_event_wait
> #3  0x000055ab028eda2b in wait_for_readers
> #4  0x000055ab028eda2b in synchronize_rcu
> #5  0x000055ab028edc5b in call_rcu_thread
> #6  0x00007f1cd914273a in start_thread ()
> #7  0x00007f1cd4113e0f in clone ()
> 
> So it seems we cannot fork() from QMP?
> [cc: Paolo]
> 
> Any comments?

I remembered hitting this in the past - but I can only
trigger it rarely for me on x86;  the following script triggers it
after ~100 iterations on my laptop:

#!/bin/bash

while true
do
  OURPIPE=/tmp/delaystop.$$
  mknod $OURPIPE p

  ./try/x86_64-softmmu/qemu-system-x86_64 -nographic -M pc,accel=kvm -smp 8 < $OURPIPE &
  QEMUPID=$!
  exec 10> "$OURPIPE"

  # Flip the mon to hmp
  echo -e '\001c' >&10
  # just a test
  echo 'migrate  -d "exec: cat > /dev/null"' >&10
  sleep $(printf ".%05d" $RANDOM)
  echo "info status" >&10
  echo "q" >&10

  rm $OURPIPE
  wait $QEMUPID || break
done

(From my notes I mentioned that to Paolo about 18months ago after
he nailed a different case)

Dave

> Laurent
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK