[Qemu-devel] vhost-user slave deadlock question

[Qemu-devel] vhost-user slave deadlock question

[Dr. David Alan Gilbert]

Hi,
  I've added a few commands to vhost-user for virtio-fs and
am hitting a deadlock and am trying to figure out what the correct fix
is; suggestions welcome.

My setup is:
  Messages sent over the virtio queues can cause the daemon to need
  to send a request back to qemu along the slave, and qemu must respond with an
  OK/error.  Lets call this command 'setupmapping'.
  In my case I'm reading vhost-user commands in one thread and
  processing the queues in another.

  That normally works OK

My problem:
  If qemu crashes or quits it stops the queues synchronously at a point
when the main loop in qemu wont respond to anything else.  However
if we're unlucky the daemon has already sent a message to qemu and
is waiting for the response; but that response can't arrive because
qemu is shutting down.  So the queue shutdown request never completes.
Then if I kill the daemon forcibly, qemu's handler for the slavefd
wakes up and tries to read data - but it's device has gone and it
crashes.

The trace is:
(Where vuf_* is my device and the structure is pretty much the same
as the others).

vm_state_notify->virtio_set_status->vuf_set_status->vuf_stop->vhost_dev_stop->vhost_virtqueue_stop->vhost_user_get_vring_base->vhost_user_read

So it feels like we need to shut down the slave FD when we shut
down the device;  but it's not clear to me at what level.
In some ways it feels like we need a way to get out if this
hole even if we shut down one queue synchronously.

Is anyone fighting similar cases?

Dave



--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK