Re: vfio hang when unbinding after using qemu as user + vhost_net

[Alex Williamson <alex.williamson@redhat.com>]

On Fri, 5 Feb 2016 13:36:05 +0100
Dominique Martinet <asmadeus@codewreck.org> wrote:

> Hi,
> 
> I've got a weird hang here when unbinding a device from vfio-pci.
> (confirmed still hanging on 4.4.1, but originally stuck with at least
> redhat's 3.10.0-327.4.4 which could contain anything so if you'd like
> me to test a specific older version for regression please just give
> me a tag)
> 
> 
> Here's a reproducer for my hardware. I'm binding a mlx4 IB card
> (vendor:device id are 15b3:1003, pci address is 0000:90:00.0) to a VM,
> starting with vhost net on a freshly created tuntap:
> 
> ------->8------  
> modprove vhost_net
> modprobe vfio-pci
> 
> # register the model, unbind the card from current driver, bind to
> # vfio-pci and give device to qemu
> echo "15b3 1003" > /sys/bus/pci/drivers/vfio-pci/new_id
> echo 0000:90:00.0 > /sys/bus/pci/devices/0000:90:00.0/driver/unbind
> echo 0000:90:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
> chown qemu: /dev/vfio/32
> 
> # create tuntap, bring it up and open vhost-net fd
> ip tuntap add dev tap-hang-0 mode tap user qemu
> ip link set tap-hang-0 mtu 9000 up
> exec 10<>/dev/vhost-net
> 
> # run qemu. This inits the first devices and exits because of
> # nonexistant device, so no guest OS is ever involved
> su qemu -s/bin/sh -c '/usr/libexec/qemu-kvm --enable-kvm \
>   -m 16G -smp 24 -device vfio-pci,id=ib0,host=90:00.0   \
>   -netdev
> type=tap,id=guest0,ifname=tap-hang-0,script=no,downscript=no,vhost=on,vhostfd=10
> \ -device virtio-net-pci,netdev=guest0,mac=52:54:00:ff:17:12 -device
> nonexistant'
> 
> echo 0000:90:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind
> ------->8------  
> 
> The last command here hangs, I get this in dmesg:
> [  163.458817] vfio-pci 0000:90:00.0: Relaying device request to user
> (#0) [  255.802784] vfio-pci 0000:90:00.0: Relaying device request to
> user (#10) [  355.805452] vfio-pci 0000:90:00.0: Relaying device
> request to user (#20) [  455.808150] vfio-pci 0000:90:00.0: Relaying
> device request to user (#30) [  555.810916] vfio-pci 0000:90:00.0:
> Relaying device request to user (#40) [  655.813904] vfio-pci
> 0000:90:00.0: Relaying device request to user (#50) [  755.816818]
> vfio-pci 0000:90:00.0: Relaying device request to user (#60)
> 
> On pressing ^C I get:
> [  205.793450] vfio-pci 0000:90:00.0: Device is currently in use,
> task "bash" (9719) blocked until device is released
> 
> 
> Two extra observations:
>  - this does not happen if I do not add vhost-net (so adding a network
> interface without vhost=on will not hang)
>  - the exact same script running qemu as root will not hang either
> 
> I ran qemu with strace to compare the output, and after uniformizing
> pointers I do not notice any real difference (no EACCES or similar
> error at least), so the difference is probably somewhere in kernel
> land.
> 
> 
> From what I gather we're basically stuck in vfio_del_group_dev()
> because of a ref leak or something...
> 
> Does that ring a bell for anyone?
> Is there anywhere I could tune verbosity to help debug this?

I just debugged your case earlier in the week and the bug is with the
test case.  When vhost is used it takes a reference to the process mm
(qemu).  That reference includes the mmap regions on the vfio device
file.  vhost releases those references when the vhostfd file descriptor
is released.  So in the scenario you have here, killing qemu doesn't
release the vhostfd file descriptor because it's still opened in the
script.  The vfio device is not released because there's still a
reference to the mmap.  You've essentially put yourself into a
deadlock.  The solution is to close the vhostfd file descriptor in your
test script after launching qemu (echo 10<&-).  Then qemu will hold the
last reference to vhostfd and killing qemu will release that file
descriptor and everything is released as intended.  I don't believe
standard management tools like libvirt have this problem.  Thanks,

Alex