Re: vfio hang when unbinding after using qemu as user + vhost_net

[Alex Williamson <alex.williamson@redhat.com>]

On Fri, 5 Feb 2016 16:48:07 +0100
Dominique Martinet <asmadeus@codewreck.org> wrote:

> Alex Williamson wrote on Fri, Feb 05, 2016:
> > I just debugged your case earlier in the week and the bug is with
> > the test case.  
> 
> Thank you for the extra information and sorry for double work.
> Getting technical informations through support is hard...
> 
> > When vhost is used it takes a reference to the process mm (qemu).
> > That reference includes the mmap regions on the vfio device  file.
> > vhost releases those references when the vhostfd file descriptor
> > is released.
> > So in the scenario you have here, killing qemu doesn't release the
> > vhostfd file descriptor because it's still opened in the script.
> > The vfio device is not released because there's still a reference
> > to the mmap.  You've essentially put yourself into a deadlock
> > The solution is to close the vhostfd file descriptor in your test
> > script after launching qemu (echo 10<&-).  Then qemu will hold the
> > last reference to vhostfd and killing qemu will release that file
> > descriptor and everything is released as intended.  
> 
> doh, I was sure I also had the hang when giving /dev/vhost-net to qemu
> and letting it open the fd so I wasn't looking at it at all, but
> you've got it.
> (I think I didn't have the hang as root because I didn't bother with
> vhostfd then either, the devil is in the details...)
> 
> Closing the fd even after qemu has stopped will free the resource and
> let me unbind, so I will just make sure to order vhost-net-related to
> be closed before vfio stuff for now.
> 
> 
> Really not an obvious lock at first glance though, not sure how this
> could be 'fixed' now you've explained it so I'll just let you guys
> decide how to handle it. This has been very helpful.
> 
> 
> > I don't believe standard management tools like libvirt have this
> > problem.  
> 
> I'm pretty sure this would have been pointed at ages ago if libvirt
> had the problem :)
> 
> Thank you for your time,

Yes, it's unfortunate that management tools need to be aware of these
sorts of semantics but it's somewhat fundamental in the mechanism of
providing access through file descriptors.  When QEMU is killed,
there's no opportunity for cleanup, so all of the kernel interfaces
need to do this automatically, but the trigger for that is when the
release callbacks for the open files get called.  Therefore the owner
of that file, the shell in the example case, needs to not only give
QEMU access, but let it hold the only outstanding references.

Another unique feature of your test case is that the unbind is called
from the same shell that owns that same open vhost file descriptor.  The
unbind of course blocks until the device is unused because there is no
opportunity for a -EBUSY return through that path in the kernel.  If
the unbind is executed from a separate shell, then we don't have that
interaction, qemu can be killed, which doesn't immediately release the
vhostfd, but the shell is not blocked and can exit and everything
releases as expected again.

Anyway, I appreciate your concise test case, I wasn't aware of that
interaction either, but it was an interesting problem to investigate.
Thanks,

Alex