From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from aserp1040.oracle.com ([141.146.126.69]:17972 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751229AbdAMWcb (ORCPT ); Fri, 13 Jan 2017 17:32:31 -0500 Date: Fri, 13 Jan 2017 17:30:35 -0500 From: Konrad Rzeszutek Wilk To: Dan Streetman Cc: Stefano Stabellini , jgross@suse.com, Boris Ostrovsky , Dan Streetman , Bjorn Helgaas , xen-devel@lists.xenproject.org, linux-kernel , linux-pci@vger.kernel.org Subject: Re: [PATCH] xen: do not re-use pirq number cached in pci device msi msg data Message-ID: <20170113223035.GE23066@char.us.oracle.com> References: <04f2a09f-59be-a720-bc98-4afb53171790@oracle.com> <20170113200751.20125-1-ddstreet@ieee.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <20170113200751.20125-1-ddstreet@ieee.org> Sender: linux-pci-owner@vger.kernel.org List-ID: On Fri, Jan 13, 2017 at 03:07:51PM -0500, Dan Streetman wrote: > Revert the main part of commit: > af42b8d12f8a ("xen: fix MSI setup and teardown for PV on HVM guests") > > That commit introduced reading the pci device's msi message data to see > if a pirq was previously configured for the device's msi/msix, and re-use > that pirq. At the time, that was the correct behavior. However, a > later change to Qemu caused it to call into the Xen hypervisor to unmap > all pirqs for a pci device, when the pci device disables its MSI/MSIX > vectors; specifically the Qemu commit: > c976437c7dba9c7444fb41df45468968aaa326ad > ("qemu-xen: free all the pirqs for msi/msix when driver unload") > > Once Qemu added this pirq unmapping, it was no longer correct for the > kernel to re-use the pirq number cached in the pci device msi message > data. All Qemu releases since 2.1.0 contain the patch that unmaps the > pirqs when the pci device disables its MSI/MSIX vectors. > > This bug is causing failures to initialize multiple NVMe controllers > under Xen, because the NVMe driver sets up a single MSIX vector for > each controller (concurrently), and then after using that to talk to > the controller for some configuration data, it disables the single MSIX > vector and re-configures all the MSIX vectors it needs. So the MSIX > setup code tries to re-use the cached pirq from the first vector > for each controller, but the hypervisor has already given away that > pirq to another controller, and its initialization fails. > > This is discussed in more detail at: > https://lists.xen.org/archives/html/xen-devel/2017-01/msg00447.html > > Fixes: af42b8d12f8a ("xen: fix MSI setup and teardown for PV on HVM guests") > Signed-off-by: Dan Streetman Acked-by: Konrad Rzeszutek Wilk From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: [PATCH] xen: do not re-use pirq number cached in pci device msi msg data Date: Fri, 13 Jan 2017 17:30:35 -0500 Message-ID: <20170113223035.GE23066@char.us.oracle.com> References: <04f2a09f-59be-a720-bc98-4afb53171790@oracle.com> <20170113200751.20125-1-ddstreet@ieee.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cSAN2-0001I1-CP for xen-devel@lists.xenproject.org; Fri, 13 Jan 2017 22:30:52 +0000 Content-Disposition: inline In-Reply-To: <20170113200751.20125-1-ddstreet@ieee.org> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Dan Streetman Cc: jgross@suse.com, Stefano Stabellini , linux-pci@vger.kernel.org, linux-kernel , Bjorn Helgaas , xen-devel@lists.xenproject.org, Boris Ostrovsky , Dan Streetman List-Id: xen-devel@lists.xenproject.org T24gRnJpLCBKYW4gMTMsIDIwMTcgYXQgMDM6MDc6NTFQTSAtMDUwMCwgRGFuIFN0cmVldG1hbiB3 cm90ZToKPiBSZXZlcnQgdGhlIG1haW4gcGFydCBvZiBjb21taXQ6Cj4gYWY0MmI4ZDEyZjhhICgi eGVuOiBmaXggTVNJIHNldHVwIGFuZCB0ZWFyZG93biBmb3IgUFYgb24gSFZNIGd1ZXN0cyIpCj4g Cj4gVGhhdCBjb21taXQgaW50cm9kdWNlZCByZWFkaW5nIHRoZSBwY2kgZGV2aWNlJ3MgbXNpIG1l c3NhZ2UgZGF0YSB0byBzZWUKPiBpZiBhIHBpcnEgd2FzIHByZXZpb3VzbHkgY29uZmlndXJlZCBm b3IgdGhlIGRldmljZSdzIG1zaS9tc2l4LCBhbmQgcmUtdXNlCj4gdGhhdCBwaXJxLiAgQXQgdGhl IHRpbWUsIHRoYXQgd2FzIHRoZSBjb3JyZWN0IGJlaGF2aW9yLiAgSG93ZXZlciwgYQo+IGxhdGVy IGNoYW5nZSB0byBRZW11IGNhdXNlZCBpdCB0byBjYWxsIGludG8gdGhlIFhlbiBoeXBlcnZpc29y IHRvIHVubWFwCj4gYWxsIHBpcnFzIGZvciBhIHBjaSBkZXZpY2UsIHdoZW4gdGhlIHBjaSBkZXZp Y2UgZGlzYWJsZXMgaXRzIE1TSS9NU0lYCj4gdmVjdG9yczsgc3BlY2lmaWNhbGx5IHRoZSBRZW11 IGNvbW1pdDoKPiBjOTc2NDM3YzdkYmE5Yzc0NDRmYjQxZGY0NTQ2ODk2OGFhYTMyNmFkCj4gKCJx ZW11LXhlbjogZnJlZSBhbGwgdGhlIHBpcnFzIGZvciBtc2kvbXNpeCB3aGVuIGRyaXZlciB1bmxv YWQiKQo+IAo+IE9uY2UgUWVtdSBhZGRlZCB0aGlzIHBpcnEgdW5tYXBwaW5nLCBpdCB3YXMgbm8g bG9uZ2VyIGNvcnJlY3QgZm9yIHRoZQo+IGtlcm5lbCB0byByZS11c2UgdGhlIHBpcnEgbnVtYmVy IGNhY2hlZCBpbiB0aGUgcGNpIGRldmljZSBtc2kgbWVzc2FnZQo+IGRhdGEuICBBbGwgUWVtdSBy ZWxlYXNlcyBzaW5jZSAyLjEuMCBjb250YWluIHRoZSBwYXRjaCB0aGF0IHVubWFwcyB0aGUKPiBw aXJxcyB3aGVuIHRoZSBwY2kgZGV2aWNlIGRpc2FibGVzIGl0cyBNU0kvTVNJWCB2ZWN0b3JzLgo+ IAo+IFRoaXMgYnVnIGlzIGNhdXNpbmcgZmFpbHVyZXMgdG8gaW5pdGlhbGl6ZSBtdWx0aXBsZSBO Vk1lIGNvbnRyb2xsZXJzCj4gdW5kZXIgWGVuLCBiZWNhdXNlIHRoZSBOVk1lIGRyaXZlciBzZXRz IHVwIGEgc2luZ2xlIE1TSVggdmVjdG9yIGZvcgo+IGVhY2ggY29udHJvbGxlciAoY29uY3VycmVu dGx5KSwgYW5kIHRoZW4gYWZ0ZXIgdXNpbmcgdGhhdCB0byB0YWxrIHRvCj4gdGhlIGNvbnRyb2xs ZXIgZm9yIHNvbWUgY29uZmlndXJhdGlvbiBkYXRhLCBpdCBkaXNhYmxlcyB0aGUgc2luZ2xlIE1T SVgKPiB2ZWN0b3IgYW5kIHJlLWNvbmZpZ3VyZXMgYWxsIHRoZSBNU0lYIHZlY3RvcnMgaXQgbmVl ZHMuICBTbyB0aGUgTVNJWAo+IHNldHVwIGNvZGUgdHJpZXMgdG8gcmUtdXNlIHRoZSBjYWNoZWQg cGlycSBmcm9tIHRoZSBmaXJzdCB2ZWN0b3IKPiBmb3IgZWFjaCBjb250cm9sbGVyLCBidXQgdGhl IGh5cGVydmlzb3IgaGFzIGFscmVhZHkgZ2l2ZW4gYXdheSB0aGF0Cj4gcGlycSB0byBhbm90aGVy IGNvbnRyb2xsZXIsIGFuZCBpdHMgaW5pdGlhbGl6YXRpb24gZmFpbHMuCj4gCj4gVGhpcyBpcyBk aXNjdXNzZWQgaW4gbW9yZSBkZXRhaWwgYXQ6Cj4gaHR0cHM6Ly9saXN0cy54ZW4ub3JnL2FyY2hp dmVzL2h0bWwveGVuLWRldmVsLzIwMTctMDEvbXNnMDA0NDcuaHRtbAo+IAo+IEZpeGVzOiBhZjQy YjhkMTJmOGEgKCJ4ZW46IGZpeCBNU0kgc2V0dXAgYW5kIHRlYXJkb3duIGZvciBQViBvbiBIVk0g Z3Vlc3RzIikKPiBTaWduZWQtb2ZmLWJ5OiBEYW4gU3RyZWV0bWFuIDxkYW4uc3RyZWV0bWFuQGNh bm9uaWNhbC5jb20+CgpBY2tlZC1ieTogS29ucmFkIFJ6ZXN6dXRlayBXaWxrIDxrb25yYWQud2ls a0BvcmFjbGUuY29tPgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KWGVuLWRldmVsIG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRw czovL2xpc3RzLnhlbi5vcmcveGVuLWRldmVsCg==