[Qemu-devel] hw/vfio/pci: duplicated invocation of vfio_msix_vector_release() in vfio_msix_disable()

[Qemu-devel] hw/vfio/pci: duplicated invocation of vfio_msix_vector_release() in vfio_msix_disable()

[Guoheyi]

Hi folks,

Recently we found there are 2 invocations of vfio_msix_vector_release() 
in vfio_msix_disable(). The 1st one is in msix_unset_vector_notifiers(), 
for we set device's msix_vector_release_notifier to 
vfio_msix_vector_release() in vfio_msix_enable(), while the 2nd is the 
explicit one in vfio_msix_disable(). Both invocations switch VFIO to use 
the non-bypass eventfd.

Is there any special reason for doing this? Or can we remove one of them?

Thanks,

Heyi

Re: [Qemu-devel] hw/vfio/pci: duplicated invocation of vfio_msix_vector_release() in vfio_msix_disable()

[Alex Williamson]

On Tue, 10 Sep 2019 19:01:22 +0800
Guoheyi <guoheyi@huawei.com> wrote:

> Hi folks,
> 
> Recently we found there are 2 invocations of vfio_msix_vector_release() 
> in vfio_msix_disable(). The 1st one is in msix_unset_vector_notifiers(), 
> for we set device's msix_vector_release_notifier to 
> vfio_msix_vector_release() in vfio_msix_enable(), while the 2nd is the 
> explicit one in vfio_msix_disable(). Both invocations switch VFIO to use 
> the non-bypass eventfd.
> 
> Is there any special reason for doing this? Or can we remove one of them?

The comment certainly suggests there is, and git blame trivially finds:

commit 3e40ba0faf0822fa78336fe6cd9d677ea9b14f1b
Author: Alex Williamson <alex.williamson@redhat.com>
Date:   Fri Dec 6 11:16:40 2013 -0700

    vfio-pci: Release all MSI-X vectors when disabled
    
    We were relying on msix_unset_vector_notifiers() to release all the
    vectors when we disable MSI-X, but this only happens when MSI-X is
    still enabled on the device.  Perform further cleanup by releasing
    any remaining vectors listed as in-use after this call.  This caused
    a leak of IRQ routes on hotplug depending on how the guest OS prepared
    the device for removal.
    
    Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
    Cc: qemu-stable@nongnu.org

diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
index f367537737d2..9aecaa82bc34 100644
--- a/hw/misc/vfio.c
+++ b/hw/misc/vfio.c
@@ -905,8 +905,20 @@ static void vfio_disable_msi_common(VFIODevice *vdev)
 
 static void vfio_disable_msix(VFIODevice *vdev)
 {
+    int i;
+
     msix_unset_vector_notifiers(&vdev->pdev);
 
+    /*
+     * MSI-X will only release vectors if MSI-X is still enabled on the
+     * device, check through the rest and release it ourselves if necessary.
+     */
+    for (i = 0; i < vdev->nr_vectors; i++) {
+        if (vdev->msi_vectors[i].use) {
+            vfio_msix_vector_release(&vdev->pdev, i);
+        }
+    }
+
     if (vdev->nr_vectors) {
         vfio_disable_irqindex(vdev, VFIO_PCI_MSIX_IRQ_INDEX);
     }