From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48610) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bmE3m-0001XQ-Qs for qemu-devel@nongnu.org; Tue, 20 Sep 2016 01:57:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bmE3h-0003x0-Ol for qemu-devel@nongnu.org; Tue, 20 Sep 2016 01:57:37 -0400 Date: Tue, 20 Sep 2016 15:57:19 +1000 From: David Gibson Message-ID: <20160920055719.GU20488@umbus> References: <1473919908-20653-1-git-send-email-david@gibson.dropbear.id.au> <20160918053327.GB11536@pxdev.xzpeter.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="/asd063ZXkk5ILJU" Content-Disposition: inline In-Reply-To: <20160918053327.GB11536@pxdev.xzpeter.org> Subject: Re: [Qemu-devel] [PATCH] vfio: Fix regression in MSI routing configuration List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Xu Cc: alex.williamson@redhat.com, qemu-devel@nongnu.org, qemu-ppc@nongnu.org, gwshan@au1.ibm.com --/asd063ZXkk5ILJU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Sep 18, 2016 at 01:33:27PM +0800, Peter Xu wrote: > Good to know that we have a solution. :) >=20 > I think this is a good fix, however I still do not understand why this > is happening... Please see below comment. >=20 > On Thu, Sep 15, 2016 at 04:11:48PM +1000, David Gibson wrote: > > d1f6af6 "kvm-irqchip: simplify kvm_irqchip_add_msi_route" was a cleanup > > of kvmchip routing configuration, that was mostly intended for x86. > > However, it also contains a subtle change in behaviour which breaks EEH= [1] > > error recovery on certain VFIO passthrough devices on spapr guests. So= far > > it's only been seen on a BCM5719 NIC on a POWER8 server, but there may = be > > other hardware with the same problem. It's also possible there could be > > circumstances where it causes a bug on x86 as well, though I don't know= of > > any obvious candidates. > >=20 > > Prior to d1f6af6, both vfio_msix_vector_do_use() and > > vfio_add_kvm_msi_virq() used msg =3D=3D NULL as a special flag to mark = this > > as the "dummy" vector used to make the host hardware state sync with the > > guest expected hardware state in terms of MSI configuration. > >=20 > > Specifically that flag caused vfio_add_kvm_msi_virq() to become a no-op, > > meaning the dummy irq would always be delivered via qemu. >=20 > AFAICT, QEMU is not delivering that *dummy* IRQ as well. IIUC here the > dummy IRQ refers to the one in vfio_msix_enable(): >=20 > vfio_msix_vector_do_use(&vdev->pdev, 0, NULL, NULL); > vfio_msix_vector_release(&vdev->pdev, 0); >=20 > Here we registered this dummy one to sync up the guest and host > hardware state for some reason. However, the handler is NULL here, > means that even QEMU won't handle it if it happens. And the only > difference is that during this extremely small period, kvm will handle > the interrupt with an uninitialized MSI message, but assuming that > would never happen since no one should start to use MSI yet. After > release(), interrupts will be dropped for all cases. >=20 > So looks like it is not related to "whether QEMU or KVM will handle > the interrupt", but something else. >=20 > Generally speaking, the process of registering the IRQ should be > something like (using QEMU with d1f6af6 change): >=20 > 1. vfio_msix_enable(): when MSIX is enabled. this will trigger > vfio_add_kvm_msi_virq(), but quickly it is unregistered (virq will > be there, but VFIO will assign the VFIO handler to > vector->interrupt, rather than vector->kvm_interrupt, so that virq > won't take effect). >=20 > 2. vfio_msix_vector_use(): when an MSIX entry is finally used and > setup. this will trigger vfio_update_kvm_msi_virq(), to update the > interrupt information to the "real one". >=20 > IMHO we should always receive interrupts after step (2). And as we > have traced (with David), step (2) was updating the correct interrupt > information even with commit d1f6af6. But I think I might be wrong (or > say, the above assumption is not correct), because commit d1f6af6 > indeed caused problem with EHH and this specific hardware. I am just > do not know why it is triggering the issue. And that's why I want to > know whether there is anything special with that NIC's driver. It's quite possible there's something unusual about the driver - this didn't happen for the other NIC I tried, or for an XHCI. Unfortunately, I don't know what it would be, and I really have no idea where one would start looking. > Again, this is totally a discussion only, and I am totally agree with > current change to fix the issue. Just in case someone knows the reason > behind this. >=20 > Thanks, >=20 > -- peterx >=20 --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --/asd063ZXkk5ILJU Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJX4M+/AAoJEGw4ysog2bOSixoQANqPX4OV1sa0B3oNp39B8eDS XAd/EKBfV3ameW4knD+XgHXgFCxvVoHv9Q86W7P92eek17XYsq+NBdjiGEm5du5E 4YaEROwIVU7EcK96nQi0dvpkURYEmLSuq993AYJglHZA61rHSyGySRvOvhzrrDyY 5wFkfMfXrAPzIi6vBAo0GokEkB+ERbPhduXPAMwps1V51khgKoz6QW7sk53wJGub yWoCmcKOdezsDFWUaD9BXxMTVMUwQSIcIPmfM156UOfxyCXO5SPQ3coVr3vBih6r T8MbqhPkBKh+RZL5UsrudzIRQmdQIBi+2uFjnRsEVCQ80rA6k5mxKRzB9QmyNxa4 xNSlI/Ons4GZpxmYQ+BcjpPi2X3qmK6AEQzaLZPPZZ/uOX6wpUhZ0qiWZzwx1XU+ PxPp8TSJnUHk2KRyIV4uZ2Q6zRtLz9rqEVT+LbOs2U0z363C8WfmY3D75ZsslcoF lI0uHb8/YfDS8bgcb9L5VQiLnelGy/6Pj+mGeKmXHesO8wROE3R8KiDzQgUA0lPC 3Ifc40pDUP6r1O8qk2klPGfUlqrPru7gG3VYTVbOupNMMrl8Q4i0X1h5EI8p9kPH mYLzJU158bdrcl+B9j9gKSmURoCeBxy/Yubj+rOqcAzs4nTJFDcVpSLSszAFQndA E0t8w5+HiZTUfakao+BY =izuJ -----END PGP SIGNATURE----- --/asd063ZXkk5ILJU--