From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:43563) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVEm7-0008In-7V for qemu-devel@nongnu.org; Mon, 28 Nov 2011 22:54:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RVEm6-0003tM-3m for qemu-devel@nongnu.org; Mon, 28 Nov 2011 22:54:31 -0500 Received: from mx1.redhat.com ([209.132.183.28]:20650) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVEm5-0003tF-Sr for qemu-devel@nongnu.org; Mon, 28 Nov 2011 22:54:30 -0500 Message-ID: <1322538856.19120.126.camel@bling.home> From: Alex Williamson Date: Mon, 28 Nov 2011 20:54:16 -0700 In-Reply-To: <4ED43CFE.8040009@au1.ibm.com> References: <20111103195452.21259.93021.stgit@bling.home> <4ED43AD9.5090509@au1.ibm.com> <4ED43CFE.8040009@au1.ibm.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Mime-Version: 1.0 Subject: Re: [Qemu-devel] [RFC PATCH] vfio: VFIO Driver core framework List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexey Kardashevskiy Cc: aafabbri@cisco.com, kvm@vger.kernel.org, pmac@au1.ibm.com, qemu-devel@nongnu.org, joerg.roedel@amd.com, konrad.wilk@oracle.com, agraf@suse.de, dwg@au1.ibm.com, chrisw@sous-sol.org, B08248@freescale.com, iommu@lists.linux-foundation.org, avi@redhat.com, linux-pci@vger.kernel.org, B07421@freescale.com, benve@cisco.com On Tue, 2011-11-29 at 13:01 +1100, Alexey Kardashevskiy wrote: > Hi all, > > Another problem I hit on POWER - MSI interrupts allocation. The existing VFIO does not expect a PBH > to support less interrupts that a device might request. In my case, PHB's limit is 8 interrupts > while my test card (10Gb ethernet CXGB3) wants 9. Below are the patches to demonstrate the idea. Seems reasonable. I assume we'd need similar for vfio_pci_setup_msi, though I haven't seen anything use more than a single MSI interrupt. Thanks, Alex > KERNEL patch: > > diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c > index 7d45c6b..d44b9bf 100644 > --- a/drivers/vfio/pci/vfio_pci_intrs.c > +++ b/drivers/vfio/pci/vfio_pci_intrs.c > @@ -458,17 +458,32 @@ int vfio_pci_setup_msix(struct vfio_pci_device *vdev, int nvec, int __user *inta > vdev->msix[i].entry = i; > vdev->ev_msix[i] = ctx; > } > - if (!ret) > + if (!ret) { > ret = pci_enable_msix(pdev, vdev->msix, nvec); > + /* > + The kernel is unable to allocate requested number of IRQs > + and returned the available number. > + */ > + if (0 < ret) { > + ret = pci_enable_msix(pdev, vdev->msix, ret); > + } > + } > vdev->msix_nvec = 0; > - for (i = 0; i < nvec && !ret; i++) { > - ret = request_irq(vdev->msix[i].vector, msihandler, 0, > - "vfio", vdev->ev_msix[i]); > - if (ret) > - break; > - vdev->msix_nvec = i+1; > + if (0 == ret) { > + vdev->msix_nvec = 0; > + ret = 0; > + for (i = 0; i < nvec && !ret; i++) { > + ret = request_irq(vdev->msix[i].vector, msihandler, 0, > + "vfio", vdev->ev_msix[i]); > + if (ret) > + break; > + vdev->msix_nvec = i+1; > + } > + if ((0 == vdev->msix_nvec) && (0 != ret)) > + vfio_pci_drop_msix(vdev); > + else > + ret = vdev->msix_nvec; > } > - if (ret) > - vfio_pci_drop_msix(vdev); > + > return ret; > } > > === end === > > > QEMU patch: > > diff --git a/hw/vfio_pci.c b/hw/vfio_pci.c > index 020961a..980eec7 100644 > --- a/hw/vfio_pci.c > +++ b/hw/vfio_pci.c > @@ -341,7 +341,8 @@ static void vfio_enable_msi(VFIODevice *vdev, bool msix) > } > } > > - if (ioctl(vdev->fd, VFIO_DEVICE_SET_IRQ_EVENTFDS, fds)) { > + ret = ioctl(vdev->fd, VFIO_DEVICE_SET_IRQ_EVENTFDS, fds); > + if (0 > ret) { > fprintf(stderr, "vfio: Error: Failed to setup MSI/X fds %s\n", > strerror(errno)); > for (i = 0; i < vdev->nr_vectors; i++) { > @@ -355,6 +356,8 @@ static void vfio_enable_msi(VFIODevice *vdev, bool msix) > qemu_free(vdev->msi_vectors); > vdev->nr_vectors = 0; > return; > + } else if (0 < ret) { > + vdev->nr_vectors = ret; > } > > vdev->interrupt = msix ? INT_MSIX : INT_MSI; > > > === end ===