Re: vfio hang when unbinding after using qemu as user + vhost_net

[Dominique Martinet <asmadeus@codewreck.org>]

Alex Williamson wrote on Fri, Feb 05, 2016:
> I just debugged your case earlier in the week and the bug is with the
> test case.

Thank you for the extra information and sorry for double work.
Getting technical informations through support is hard...

> When vhost is used it takes a reference to the process mm (qemu).
> That reference includes the mmap regions on the vfio device  file.
> vhost releases those references when the vhostfd file descriptor
> is released.
> So in the scenario you have here, killing qemu doesn't release the
> vhostfd file descriptor because it's still opened in the script.  The
> vfio device is not released because there's still a reference to the
> mmap.  You've essentially put yourself into a deadlock
> The solution is to close the vhostfd file descriptor in your test
> script after launching qemu (echo 10<&-).  Then qemu will hold the
> last reference to vhostfd and killing qemu will release that file
> descriptor and everything is released as intended.

doh, I was sure I also had the hang when giving /dev/vhost-net to qemu
and letting it open the fd so I wasn't looking at it at all, but you've
got it.
(I think I didn't have the hang as root because I didn't bother with
vhostfd then either, the devil is in the details...)

Closing the fd even after qemu has stopped will free the resource and
let me unbind, so I will just make sure to order vhost-net-related to be
closed before vfio stuff for now.


Really not an obvious lock at first glance though, not sure how this
could be 'fixed' now you've explained it so I'll just let you guys
decide how to handle it. This has been very helpful.


> I don't believe standard management tools like libvirt have this
> problem.

I'm pretty sure this would have been pointed at ages ago if libvirt had
the problem :)

Thank you for your time,
-- 
Dominique Martinet