From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13634C33C9E for ; Sat, 18 Jan 2020 01:08:50 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D97612464B for ; Sat, 18 Jan 2020 01:08:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D97612464B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iscb0-0000E2-Vf; Sat, 18 Jan 2020 01:08:14 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iscaz-0000Dx-8J for xen-devel@lists.xenproject.org; Sat, 18 Jan 2020 01:08:13 +0000 X-Inumbo-ID: f951aa4c-398e-11ea-9fd7-bc764e2007e4 Received: from mga06.intel.com (unknown [134.134.136.31]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id f951aa4c-398e-11ea-9fd7-bc764e2007e4; Sat, 18 Jan 2020 01:08:02 +0000 (UTC) X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jan 2020 17:07:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,332,1574150400"; d="scan'208";a="264378530" Received: from gao-cwp.sh.intel.com (HELO gao-cwp) ([10.239.159.154]) by fmsmga001.fm.intel.com with ESMTP; 17 Jan 2020 17:07:33 -0800 Date: Sat, 18 Jan 2020 09:13:39 +0800 From: Chao Gao To: Rich Persaud Message-ID: <20200118011338.GA29391@gao-cwp> References: <20190926101347.GD28704@reaktio.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) Subject: Re: [Xen-devel] [PATCH] xen: xen-pciback: Reset MSI-X state when exposing a device X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: "jgross@suse.com" , "sstabellini@kernel.org" , Jason Andryuk , "Spassov, Stanislav" , "linux-kernel@vger.kernel.org" , Marek =?iso-8859-1?Q?Marczykowski-G=F3recki?= , "baijiaju1990@gmail.com" , "jbeulich@suse.com" , "xen-devel@lists.xenproject.org" , "boris.ostrovsky@oracle.com" , "Woodhouse, David" , "roger.pau@citrix.com" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" T24gRnJpLCBKYW4gMTcsIDIwMjAgYXQgMDE6NTc6NDNQTSAtMDUwMCwgUmljaCBQZXJzYXVkIHdy b3RlOgo+T24gU2VwIDI2LCAyMDE5LCBhdCAwNjoxNywgUGFzaSBLw6Rya2vDpGluZW4gPHBhc2lr QGlraS5maT4gd3JvdGU6Cj4+IAo+PiDvu79IZWxsbyBTdGFuaXNsYXYsCj4+IAo+Pj4gT24gRnJp LCBTZXAgMTMsIDIwMTkgYXQgMTE6Mjg6MjBQTSArMDgwMCwgQ2hhbyBHYW8gd3JvdGU6Cj4+Pj4g T24gRnJpLCBTZXAgMTMsIDIwMTkgYXQgMTA6MDI6MjRBTSArMDAwMCwgU3Bhc3NvdiwgU3Rhbmlz bGF2IHdyb3RlOgo+Pj4+IE9uIFRodSwgRGVjIDEzLCAyMDE4IGF0IDA3OjU0LCBDaGFvIEdhbyB3 cm90ZToKPj4+Pj4gT24gVGh1LCBEZWMgMTMsIDIwMTggYXQgMTI6NTQ6NTJBTSAtMDcwMCwgSmFu IEJldWxpY2ggd3JvdGU6Cj4+Pj4+Pj4+PiBPbiAxMy4xMi4xOCBhdCAwNDo0NiwgPGNoYW8uZ2Fv QGludGVsLmNvbT4gd3JvdGU6Cj4+Pj4+Pj4gT24gV2VkLCBEZWMgMTIsIDIwMTggYXQgMDg6MjE6 MzlBTSAtMDcwMCwgSmFuIEJldWxpY2ggd3JvdGU6Cj4+Pj4+Pj4+Pj4+IE9uIDEyLjEyLjE4IGF0 IDE2OjE4LCA8Y2hhby5nYW9AaW50ZWwuY29tPiB3cm90ZToKPj4+Pj4+Pj4+IE9uIFdlZCwgRGVj IDEyLCAyMDE4IGF0IDAxOjUxOjAxQU0gLTA3MDAsIEphbiBCZXVsaWNoIHdyb3RlOgo+Pj4+Pj4+ Pj4+Pj4+IE9uIDEyLjEyLjE4IGF0IDA4OjA2LCA8Y2hhby5nYW9AaW50ZWwuY29tPiB3cm90ZToK Pj4+Pj4+Pj4+Pj4gT24gV2VkLCBEZWMgMDUsIDIwMTggYXQgMDk6MDE6MzNBTSAtMDUwMCwgQm9y aXMgT3N0cm92c2t5IHdyb3RlOgo+Pj4+Pj4+Pj4+Pj4gT24gMTIvNS8xOCA0OjMyIEFNLCBSb2dl ciBQYXUgTW9ubsOpIHdyb3RlOgo+Pj4+Pj4+Pj4+Pj4+IE9uIFdlZCwgRGVjIDA1LCAyMDE4IGF0 IDEwOjE5OjE3QU0gKzA4MDAsIENoYW8gR2FvIHdyb3RlOgo+Pj4+Pj4+Pj4+Pj4+PiBJIGZpbmQg c29tZSBwYXNzLXRocnUgZGV2aWNlcyBkb24ndCB3b3JrIGFueSBtb3JlIGFjcm9zcyBndWVzdCBy ZWJvb3QuCj4+Pj4+Pj4+Pj4+Pj4+IEFzc2lnbmluZyBpdCB0byBhbm90aGVyIGd1ZXN0IGFsc28g bWVldHMgdGhlIHNhbWUgaXNzdWUuIEFuZCB0aGUgb25seQo+Pj4+Pj4+Pj4+Pj4+PiB3YXkgdG8g bWFrZSBpdCB3b3JrIGFnYWluIGlzIHVuLWJpbmRpbmcgYW5kIGJpbmRpbmcgaXQgdG8gcGNpYmFj ay4KPj4+Pj4+Pj4+Pj4+Pj4gU29tZW9uZSByZXBvcnRlZCB0aGlzIGlzc3VlIG9uZSB5ZWFyIGFn byBbMV0uIE1vcmUgZGV0YWlsIGFsc28gY2FuIGJlCj4+Pj4+Pj4+Pj4+Pj4+IGZvdW5kIGluIFsy XS4KPj4+Pj4+Pj4+Pj4+Pj4gCj4+Pj4+Pj4+Pj4+Pj4+IFRoZSByb290LWNhdXNlIGlzIFhlbidz IGludGVybmFsIE1TSS1YIHN0YXRlIGlzbid0IHJlc2V0IHByb3Blcmx5Cj4+Pj4+Pj4+Pj4+Pj4+ IGR1cmluZyByZWJvb3Qgb3IgcmUtYXNzaWdubWVudC4gSW4gdGhlIGFib3ZlIGNhc2UsIFhlbiBz ZXQgbWFza2FsbCBiaXQKPj4+Pj4+Pj4+Pj4+Pj4gdG8gbWFzayBhbGwgTVNJIGludGVycnVwdHMg YWZ0ZXIgaXQgZGV0ZWN0ZWQgYSBwb3RlbnRpYWwgc2VjdXJpdHkKPj4+Pj4+Pj4+Pj4+Pj4gaXNz dWUuIEV2ZW4gYWZ0ZXIgZGV2aWNlIHJlc2V0LCBYZW4gZGlkbid0IHJlc2V0IGl0cyBpbnRlcm5h bCBtYXNrYWxsCj4+Pj4+Pj4+Pj4+Pj4+IGJpdC4gQXMgYSByZXN1bHQsIG1hc2thbGwgYml0IHdv dWxkIGJlIHNldCBhZ2FpbiBpbiBuZXh0IHdyaXRlIHRvCj4+Pj4+Pj4+Pj4+Pj4+IE1TSS1YIG1l c3NhZ2UgY29udHJvbCByZWdpc3Rlci4KPj4+Pj4+Pj4+Pj4+Pj4gCj4+Pj4+Pj4+Pj4+Pj4+IEdp dmVuIHRoYXQgUEhZU0RFVk9QU19wcmVwYXJlX21zaXgoKSBhbHNvIHRyaWdnZXJzIFhlbiByZXNl dHRpbmcgTVNJLVgKPj4+Pj4+Pj4+Pj4+Pj4gaW50ZXJuYWwgc3RhdGUgb2YgYSBkZXZpY2UsIHdl IGVtcGxveSBpdCB0byBmaXggdGhpcyBpc3N1ZSByYXRoZXIgdGhhbgo+Pj4+Pj4+Pj4+Pj4+PiBp bnRyb2R1Y2luZyBhbm90aGVyIGRlZGljYXRlZCBzdWItaHlwZXJjYWxsLgo+Pj4+Pj4+Pj4+Pj4+ PiAKPj4+Pj4+Pj4+Pj4+Pj4gTm90ZSB0aGF0IFBIWVNERVZPUFNfcmVsZWFzZV9tc2l4KCkgd2ls bCBmYWlsIGlmIHRoZSBtYXBwaW5nIGJldHdlZW4KPj4+Pj4+Pj4+Pj4+Pj4gdGhlIGRldmljZSdz IG1zaXggYW5kIHBpcnEgaGFzIGJlZW4gY3JlYXRlZC4gVGhpcyBsaW1pdGF0aW9uIHByZXZlbnRz Cj4+Pj4+Pj4+Pj4+Pj4+IHVzIGNhbGxpbmcgdGhpcyBmdW5jdGlvbiB3aGVuIGRldGFjaGluZyBh IGRldmljZSBmcm9tIGEgZ3Vlc3QgZHVyaW5nCj4+Pj4+Pj4+Pj4+Pj4+IGd1ZXN0IHNodXRkb3du LiBUaHVzIGl0IGlzIGNhbGxlZCByaWdodCBiZWZvcmUgY2FsbGluZwo+Pj4+Pj4+Pj4+Pj4+PiBQ SFlTREVWT1BTX3ByZXBhcmVfbXNpeCgpLgo+Pj4+Pj4+Pj4+Pj4+IHMvUEhZU0RFVk9QUy9QSFlT REVWT1AvIChubyBmaW5hbCBTKS4gQW5kIHRoZW4gSSB3b3VsZCBhbHNvIGRyb3AgdGhlCj4+Pj4+ Pj4+Pj4+Pj4gKCkgYXQgdGhlIGVuZCBvZiB0aGUgaHlwZXJjYWxsIG5hbWUgc2luY2UgaXQncyBu b3QgYSBmdW5jdGlvbi4KPj4+Pj4+Pj4+Pj4+PiAKPj4+Pj4+Pj4+Pj4+PiBJJ20gYWxzbyB3b25k ZXJpbmcgd2h5IHRoZSByZWxlYXNlIGNhbid0IGJlIGRvbmUgd2hlbiB0aGUgZGV2aWNlIGlzCj4+ Pj4+Pj4+Pj4+Pj4gZGV0YWNoZWQgZnJvbSB0aGUgZ3Vlc3QgKG9yIHRoZSBndWVzdCBoYXMgYmVl biBzaHV0IGRvd24pLiBUaGlzIG1ha2VzCj4+Pj4+Pj4+Pj4+Pj4gbWUgd29ycnkgYWJvdXQgdGhl IHJhY2luZXNzIG9mIHRoZSBhdHRhY2gvZGV0YWNoIHByb2NlZHVyZTogaWYgdGhlcmUncwo+Pj4+ Pj4+Pj4+Pj4+IGEgc3RhdGUgd2hlcmUgcGNpYmFjayBhc3N1bWVzIHRoZSBkZXZpY2UgaGFzIGJl ZW4gZGV0YWNoZWQgZnJvbSB0aGUKPj4+Pj4+Pj4+Pj4+PiBndWVzdCwgYnV0IHRoZXJlIGFyZSBz dGlsbCBwaXJxcyBib3VuZCwgYW4gYXR0ZW1wdCB0byBhdHRhY2ggdG8KPj4+Pj4+Pj4+Pj4+PiBh bm90aGVyIGd1ZXN0IGluIHN1Y2ggc3RhdGUgd2lsbCBmYWlsLgo+Pj4+Pj4+Pj4+Pj4gCj4+Pj4+ Pj4+Pj4+PiBJIHdvbmRlciB3aGV0aGVyIHRoaXMgYWRkaXRpb25hbCByZXNldCBmdW5jdGlvbmFs aXR5IGNvdWxkIGJlIGRvbmUgb3V0Cj4+Pj4+Pj4+Pj4+PiBvZiB4ZW5fcGNpYmtfeGVuYnVzX3Jl bW92ZSgpLiBXZSBmaXJzdCBkbyBhIChiZXN0IGVmZm9ydCkgZGV2aWNlIHJlc2V0Cj4+Pj4+Pj4+ Pj4+PiBhbmQgdGhlbiBkbyB0aGUgZXh0cmEgdGhpbmdzIHRoYXQgYXJlIG5vdCBwcm9wZXJseSBk b25lIHRoZXJlLgo+Pj4+Pj4+Pj4+PiAKPj4+Pj4+Pj4+Pj4gTm8uIEl0IGNhbm5vdCBiZSBkb25l IGluIHhlbl9wY2lia194ZW5idXNfcmVtb3ZlKCkgd2l0aG91dCBtb2RpZnlpbmcKPj4+Pj4+Pj4+ Pj4gdGhlIGhhbmRsZXIgb2YgUEhZU0RFVk9QX3JlbGVhc2VfbXNpeC4gVG8gZG8gYSBzdWNjZXNz ZnVsIFhlbiBpbnRlcm5hbAo+Pj4+Pj4+Pj4+PiBNU0ktWCBzdGF0ZSByZXNldCwgUEhZU0RFVk9Q X3tyZWxlYXNlLCBwcmVwYXJlfV9tc2l4IHNob3VsZCBiZSBmaW5pc2hlZAo+Pj4+Pj4+Pj4+PiB3 aXRob3V0IGVycm9yLiBCdXQgQVRNLCB4ZW4gZXhwZWN0cyB0aGF0IG5vIG1zaSBpcyBib3VuZCB0 byBwaXJxIHdoZW4KPj4+Pj4+Pj4+Pj4gZG9pbmcgUEhZU0RFVk9QX3JlbGVhc2VfbXNpeC4gT3Ro ZXJ3aXNlIGl0IGZhaWxzIHdpdGggZXJyb3IgY29kZSAtRUJVU1kuCj4+Pj4+Pj4+Pj4+IEhvd2V2 ZXIsIHRoZSBleHBlY3RhdGlvbiBpc24ndCBndWFyYW50ZWVkIGluIHhlbl9wY2lia194ZW5idXNf cmVtb3ZlKCkuCj4+Pj4+Pj4+Pj4+IEluIHNvbWUgY2FzZXMsIGlmIHFlbXUgZmFpbHMgdG8gdW5t YXAgTVNJcywgTVNJcyBhcmUgdW5tYXBwZWQgYnkgWGVuCj4+Pj4+Pj4+Pj4+IGF0IGxhc3QgbWlu dXRlLCB3aGljaCBoYXBwZW5zIGFmdGVyIGRldmljZSByZXNldCBpbiAKPj4+Pj4+Pj4+Pj4geGVu X3BjaWJrX3hlbmJ1c19yZW1vdmUoKS4KPj4+Pj4+Pj4+PiAKPj4+Pj4+Pj4+PiBCdXQgdGhhdCBt YXkgbmVlZCB0YWtpbmcgY2FyZSBvZjogSSBkb24ndCB0aGluayBpdCBpcyBhIGdvb2QgaWRlYSB0 byBoYXZlCj4+Pj4+Pj4+Pj4gYW55dGhpbmcgbGVmdCBmcm9tIHRoZSBwcmlvciBvd25pbmcgZG9t YWluIHdoZW4gdGhlIGRldmljZSBnZXRzIHJlc2V0Lgo+Pj4+Pj4+Pj4+IEkuZS4gbGVmdCBvdmVy IElSUSBiaW5kaW5ncyBzaG91bGQgcGVyaGFwcyBiZSBmb3JjaWJseSBjbGVhcmVkIGJlZm9yZQo+ Pj4+Pj4+Pj4+IGludm9raW5nIHRoZSByZXNldDsKPj4+Pj4+Pj4+IAo+Pj4+Pj4+Pj4gQWdyZWUu IEhvdyBhYm91dCBwY2liYWNrIHRvIHRyYWNrIHRoZSBlc3RhYmxpc2hlZCBJUlEgYmluZGluZ3M/ IFRoZW4KPj4+Pj4+Pj4+IHBjaWJhY2sgY2FuIGNsZWFyIGlycSBiaW5kaW5nIGJlZm9yZSBpbnZv a2luZyB0aGUgcmVzZXQuCj4+Pj4+Pj4+IAo+Pj4+Pj4+PiBIb3cgd291bGQgcGNpYmFjayBldmVu IGtub3cgb2YgdGhvc2UgbWFwcGluZ3MsIHdoZW4gaXQncyBxZW11Cj4+Pj4+Pj4+IHdobyBlc3Rh Ymxpc2hlcyAoYW5kIG1hbmFnZXMpIHRoZW0/Cj4+Pj4+Pj4gCj4+Pj4+Pj4gSSBtZWFudCB0byBl eHBvc2Ugc29tZSBpbnRlcmZhY2VzIGZyb20gcGNpYmFjay4gQW5kIHBjaWJhY2sgc2VydmVzCj4+ Pj4+Pj4gYXMgdGhlIHByb3h5IG9mIElSUSAodW4pYmluZGluZyBBUElzLgo+Pj4+Pj4gCj4+Pj4+ PiBJZiBhdCBhbGwgcG9zc2libGUgd2Ugc2hvdWxkIGF2b2lkIGhhdmluZyB0byBjaGFuZ2UgbW9y ZSBwYXJ0aWVzIChxZW11LAo+Pj4+Pj4gbGlieGMsIGtlcm5lbCwgaHlwZXJ2aXNvcikgdGhhbiBy ZWFsbHkgbmVjZXNzYXJ5LiBSZW1lbWJlciB0aGF0IHN1Y2gKPj4+Pj4+IGEgYnVnIGZpeCBtYXkg d2FudCBiYWNrcG9ydGluZywgYW5kIG1ha2luZyBzdXJlIGFmZmVjdGVkIHBlb3BsZSBoYXZlCj4+ Pj4+PiBhbGwgcmVsZXZhbnQgY29tcG9uZW50cyB1cGRhdGVkIGlzIGluY3JlYXNpbmdseSBkaWZm aWN1bHQgd2l0aCB0aGVpcgo+Pj4+Pj4gbnVtYmVyIGdyb3dpbmcuCj4+Pj4+PiAKPj4+Pj4+Pj4+ PiBpbiBmYWN0IEknZCBleHBlY3QgdGhpcyB0byBoYXBwZW4gaW4gdGhlIGNvdXJzZSBvZgo+Pj4+ Pj4+Pj4+IGRvbWFpbiBkZXN0cnVjdGlvbiwgYW5kIEknZCBleHBlY3QgdGhlIGRldmljZSByZXNl dCB0byBjb21lIGFmdGVyIHRoZQo+Pj4+Pj4+Pj4+IGRvbWFpbiB3YXMgY2xlYW5lZCB1cC4gUGVy aGFwcyBzaW1wbHkgYW4gb3JkZXJpbmcgaXNzdWUgaW4gdGhlIHRvb2wKPj4+Pj4+Pj4+PiBzdGFj az8KPj4+Pj4+Pj4+IAo+Pj4+Pj4+Pj4gSSBkb24ndCB0aGluayByZXZlcnNpbmcgdGhlIHNlcXVl bmNlcyBvZiBkZXZpY2UgcmVzZXQgYW5kIGRvbWFpbgo+Pj4+Pj4+Pj4gZGVzdHJ1Y3Rpb24gd291 bGQgYmUgc2ltcGxlLiBGdXJ0aGVybW9yZSwgZHVyaW5nIGRldmljZSBob3QtdW5wbHVnLAo+Pj4+ Pj4+Pj4gZGV2aWNlIHJlc2V0IGlzIGRvbmUgd2hlbiB0aGUgb3duZXIgaXMgYWxpdmUuIFNvIGlm IHdlIHVzZSBkb21haW4KPj4+Pj4+Pj4+IGRlc3RydWN0aW9uIHRvIGVuZm9yY2UgYWxsIGlycSBi aW5kaW5nIGNsZWFyZWQsIGluIHRoZW9yeSwgaXQgd29uJ3QgYmUKPj4+Pj4+Pj4+IGFwcGxpY2Fi bGUgdG8gaG90LXVucGx1ZyBjYXNlIChpZiBxZW11J3MgaG90LXVucGx1ZyBsb2dpYyBpcwo+Pj4+ Pj4+Pj4gY29tcHJvbWlzZWQpLgo+Pj4+Pj4+PiAKPj4+Pj4+Pj4gRXZlbiBpbiB0aGUgaG90LXVu cGx1ZyBjYXNlIHRoZSB0b29sIHN0YWNrIGNvdWxkIGlzc3VlIHVuYmluZAo+Pj4+Pj4+PiByZXF1 ZXN0cywgYmVoaW5kIHRoZSBiYWNrIG9mIHRoZSBwb3NzaWJseSBjb21wcm9taXNlZCBxZW11LAo+ Pj4+Pj4+PiBvbmNlIG5laXRoZXIgdGhlIGd1ZXN0IG5vciBxZW11IGhhdmUgYWNjZXNzIHRvIHRo ZSBkZXZpY2UKPj4+Pj4+Pj4gYW55bW9yZS4KPj4+Pj4+PiAKPj4+Pj4+PiBCdXQgY3VycmVudGx5 LCB0b29sIHN0YWNrIGRvZXNuJ3Qga25vdyB0aGUgcmVtYWluaW5nIElSUSBiaW5kaW5ncy4KPj4+ Pj4+PiBJZiB0b29sIHN0YWNrIGNhbiBtYWludGFpbmUgSVJRIGJpbmRpbmcgaW5mb3JtYXRpb24g b2YgYSBwYXNzLXRocnUKPj4+Pj4+PiBkZXZpY2UgKHN0b3JlZCBpbiBYZW5zdG9yZT8pLCB3ZSBj YW4gY29tZSB1cCB3aXRoIGEgY2xlYW4gc29sdXRpb24KPj4+Pj4+PiB3aXRob3V0IG1vZGlmeWlu ZyBsaW51eCBrZXJuZWwgYW5kIFhlbi4KPj4+Pj4+IAo+Pj4+Pj4gSWYgdGhlcmUncyBubyB3YXkg Zm9yIHRoZSB0b29sIHN0YWNrIHRvIGVpdGhlciBmaW5kIG91dCB0aGUgYmluZGluZ3MKPj4+Pj4+ IG9yICJibGluZGx5IiBpc3N1ZSB1bmJpbmQgcmVxdWVzdHMgKGFjY2VwdGluZyB0aGVtIHRvIGZh aWwpLCB0aGVuIGEKPj4+Pj4+ICJ3aWxkY2FyZCIgdW5iaW5kIG9wZXJhdGlvbiBtYXkgd2FudCBh ZGRpbmcuIE9yLCBwZXJoYXBzIGV2ZW4KPj4+Pj4+IGJldHRlciwgWEVOX0RPTUNUTF9kZWFzc2ln bl9kZXZpY2UgY291bGQgdW5iaW5kIGFueXRoaW5nIGxlZnQKPj4+Pj4+IGluIHBsYWNlIGZvciB0 aGUgc3BlY2lmaWVkIGRldmljZS4KPj4+Pj4gCj4+Pj4+IEdvb2QgaWRlYS4gSSB3aWxsIHRha2Ug dGhpcyBhZHZpY2UuCj4+Pj4+IAo+Pj4+PiBUaGFua3MKPj4+Pj4gQ2hhbwo+Pj4+IAo+Pj4+IEkg YW0gaGF2aW5nIHRoZSBzYW1lIGlzc3VlLCBhbmQgY2Fubm90IGZpbmQgYSBmaXggaW4gZWl0aGVy IHhlbi1wY2liYWNrIG9yIHRoZSBYZW4gY29kZWJhc2UuCj4+Pj4gV2FzIGEgc29sdXRpb24gZXZl ciBwdXNoZWQgYXMgYSByZXN1bHQgb2YgdGhpcyB0aHJlYWQ/Cj4+Pj4gCj4+PiAKPj4+IEkgc3Vi bWl0dGVkIHBhdGNoZXMgWzFdIHRvIFhlbiBjb21tdW5pdHkuIEJ1dCBJIGRpZG4ndCBnZXQgaXQg bWVyZ2VkLgo+Pj4gV2UgbWFkZSBhIGNoYW5nZSBpbiBkZXZpY2UgZHJpdmVyIHRvIGRpc2FibGUg TVNJLVggZHVyaW5nIGd1ZXN0IE9TCj4+PiBzaHV0ZG93biB0byBtaXRpZ2F0ZSB0aGUgaXNzdWUu IEJ1dCB3aGVuIGd1ZXN0IG9yIHFlbXUgd2FzIGNyYXNoZWQsIHdlCj4+PiBlbmNvdW50ZXJlZCB0 aGlzIGlzc3VlIGFnYWluLiBJIGhhdmUgbm8gcGxhbiB0byBnZXQgYmFjayB0byB0aGVzZQo+Pj4g cGF0Y2hlcy4gQnV0IGlmIHlvdSB3YW50IHRvIGZpeCB0aGUgaXNzdWUgY29tcGxldGVseSBhbG9u ZyB3aGF0IHRoZQo+Pj4gcGF0Y2hlcyBiZWxvdyBkaWQsIHBsZWFzZSBnbyBhaGVhZC4KPj4+IAo+ Pj4gWzFdOiBodHRwczovL2xpc3RzLnhlbnByb2plY3Qub3JnL2FyY2hpdmVzL2h0bWwveGVuLWRl dmVsLzIwMTktMDEvbXNnMDEyMjcuaHRtbAo+Pj4gCj4+PiBUaGFua3MKPj4+IENoYW8KPj4+IAo+ PiAKPj4gU3RhbmlzbGF2OiBBcmUgeW91IGFibGUgdG8gY29udGludWUgdGhlIHdvcmsgd2l0aCB0 aGVzZSBwYXRjaGVzLCB0byBnZXQgdGhlbSBtZXJnZWQ/IAo+Cj5XaGF0IGZ1cnRoZXIgd29yayBp cyBuZWVkZWQgZm9yIHRoZXNlIHBhdGNoZXM/ICBBcmUgdGhleSBvbmx5IG5lZWRlZCBmb3IgSW50 ZWwgaTIxMCBOSUMgUENJIHBhc3N0aHJvdWdoLCBvciBhcmUgb3RoZXIgZGV2aWNlcyBhZmZlY3Rl ZD8KCkFsbCBNU0ktWCBjYXBhYmxlIGRldmljZXMgd2VyZSBhZmZlY3RlZC4gVGhpcyBpc3N1ZSBp cyBmaXhlZCBpbiBYZW4gYnkgUm9nZXIncyBwYXRjaAooaHR0cHM6Ly94ZW5iaXRzLnhlbi5vcmcv Z2l0d2ViLz9wPXhlbi5naXQ7YT1jb21taXQ7aD01NzVlMThkNTRkMTllZGE3ODdmNjQ3N2E0YWNk M2M1MGY3Mjc1MWE5KS4KClRoYW5rcwpDaGFvCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZlbEBsaXN0 cy54ZW5wcm9qZWN0Lm9yZwpodHRwczovL2xpc3RzLnhlbnByb2plY3Qub3JnL21haWxtYW4vbGlz dGluZm8veGVuLWRldmVs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08596C33C9E for ; Sat, 18 Jan 2020 01:07:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C96E62464B for ; Sat, 18 Jan 2020 01:07:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730422AbgARBHh (ORCPT ); Fri, 17 Jan 2020 20:07:37 -0500 Received: from mga17.intel.com ([192.55.52.151]:52872 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728778AbgARBHg (ORCPT ); Fri, 17 Jan 2020 20:07:36 -0500 X-Amp-Result: UNSCANNABLE X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jan 2020 17:07:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,332,1574150400"; d="scan'208";a="264378530" Received: from gao-cwp.sh.intel.com (HELO gao-cwp) ([10.239.159.154]) by fmsmga001.fm.intel.com with ESMTP; 17 Jan 2020 17:07:33 -0800 Date: Sat, 18 Jan 2020 09:13:39 +0800 From: Chao Gao To: Rich Persaud Cc: Pasi =?iso-8859-1?Q?K=E4rkk=E4inen?= , "Spassov, Stanislav" , "jgross@suse.com" , "sstabellini@kernel.org" , "linux-kernel@vger.kernel.org" , "baijiaju1990@gmail.com" , "jbeulich@suse.com" , "xen-devel@lists.xenproject.org" , "boris.ostrovsky@oracle.com" , "roger.pau@citrix.com" , "Woodhouse, David" , Marek =?iso-8859-1?Q?Marczykowski-G=F3recki?= , Jason Andryuk Subject: Re: [Xen-devel] [PATCH] xen: xen-pciback: Reset MSI-X state when exposing a device Message-ID: <20200118011338.GA29391@gao-cwp> References: <20190926101347.GD28704@reaktio.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jan 17, 2020 at 01:57:43PM -0500, Rich Persaud wrote: >On Sep 26, 2019, at 06:17, Pasi Kärkkäinen wrote: >> >> Hello Stanislav, >> >>> On Fri, Sep 13, 2019 at 11:28:20PM +0800, Chao Gao wrote: >>>> On Fri, Sep 13, 2019 at 10:02:24AM +0000, Spassov, Stanislav wrote: >>>> On Thu, Dec 13, 2018 at 07:54, Chao Gao wrote: >>>>> On Thu, Dec 13, 2018 at 12:54:52AM -0700, Jan Beulich wrote: >>>>>>>>> On 13.12.18 at 04:46, wrote: >>>>>>> On Wed, Dec 12, 2018 at 08:21:39AM -0700, Jan Beulich wrote: >>>>>>>>>>> On 12.12.18 at 16:18, wrote: >>>>>>>>> On Wed, Dec 12, 2018 at 01:51:01AM -0700, Jan Beulich wrote: >>>>>>>>>>>>> On 12.12.18 at 08:06, wrote: >>>>>>>>>>> On Wed, Dec 05, 2018 at 09:01:33AM -0500, Boris Ostrovsky wrote: >>>>>>>>>>>> On 12/5/18 4:32 AM, Roger Pau Monné wrote: >>>>>>>>>>>>> On Wed, Dec 05, 2018 at 10:19:17AM +0800, Chao Gao wrote: >>>>>>>>>>>>>> I find some pass-thru devices don't work any more across guest reboot. >>>>>>>>>>>>>> Assigning it to another guest also meets the same issue. And the only >>>>>>>>>>>>>> way to make it work again is un-binding and binding it to pciback. >>>>>>>>>>>>>> Someone reported this issue one year ago [1]. More detail also can be >>>>>>>>>>>>>> found in [2]. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The root-cause is Xen's internal MSI-X state isn't reset properly >>>>>>>>>>>>>> during reboot or re-assignment. In the above case, Xen set maskall bit >>>>>>>>>>>>>> to mask all MSI interrupts after it detected a potential security >>>>>>>>>>>>>> issue. Even after device reset, Xen didn't reset its internal maskall >>>>>>>>>>>>>> bit. As a result, maskall bit would be set again in next write to >>>>>>>>>>>>>> MSI-X message control register. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Given that PHYSDEVOPS_prepare_msix() also triggers Xen resetting MSI-X >>>>>>>>>>>>>> internal state of a device, we employ it to fix this issue rather than >>>>>>>>>>>>>> introducing another dedicated sub-hypercall. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Note that PHYSDEVOPS_release_msix() will fail if the mapping between >>>>>>>>>>>>>> the device's msix and pirq has been created. This limitation prevents >>>>>>>>>>>>>> us calling this function when detaching a device from a guest during >>>>>>>>>>>>>> guest shutdown. Thus it is called right before calling >>>>>>>>>>>>>> PHYSDEVOPS_prepare_msix(). >>>>>>>>>>>>> s/PHYSDEVOPS/PHYSDEVOP/ (no final S). And then I would also drop the >>>>>>>>>>>>> () at the end of the hypercall name since it's not a function. >>>>>>>>>>>>> >>>>>>>>>>>>> I'm also wondering why the release can't be done when the device is >>>>>>>>>>>>> detached from the guest (or the guest has been shut down). This makes >>>>>>>>>>>>> me worry about the raciness of the attach/detach procedure: if there's >>>>>>>>>>>>> a state where pciback assumes the device has been detached from the >>>>>>>>>>>>> guest, but there are still pirqs bound, an attempt to attach to >>>>>>>>>>>>> another guest in such state will fail. >>>>>>>>>>>> >>>>>>>>>>>> I wonder whether this additional reset functionality could be done out >>>>>>>>>>>> of xen_pcibk_xenbus_remove(). We first do a (best effort) device reset >>>>>>>>>>>> and then do the extra things that are not properly done there. >>>>>>>>>>> >>>>>>>>>>> No. It cannot be done in xen_pcibk_xenbus_remove() without modifying >>>>>>>>>>> the handler of PHYSDEVOP_release_msix. To do a successful Xen internal >>>>>>>>>>> MSI-X state reset, PHYSDEVOP_{release, prepare}_msix should be finished >>>>>>>>>>> without error. But ATM, xen expects that no msi is bound to pirq when >>>>>>>>>>> doing PHYSDEVOP_release_msix. Otherwise it fails with error code -EBUSY. >>>>>>>>>>> However, the expectation isn't guaranteed in xen_pcibk_xenbus_remove(). >>>>>>>>>>> In some cases, if qemu fails to unmap MSIs, MSIs are unmapped by Xen >>>>>>>>>>> at last minute, which happens after device reset in >>>>>>>>>>> xen_pcibk_xenbus_remove(). >>>>>>>>>> >>>>>>>>>> But that may need taking care of: I don't think it is a good idea to have >>>>>>>>>> anything left from the prior owning domain when the device gets reset. >>>>>>>>>> I.e. left over IRQ bindings should perhaps be forcibly cleared before >>>>>>>>>> invoking the reset; >>>>>>>>> >>>>>>>>> Agree. How about pciback to track the established IRQ bindings? Then >>>>>>>>> pciback can clear irq binding before invoking the reset. >>>>>>>> >>>>>>>> How would pciback even know of those mappings, when it's qemu >>>>>>>> who establishes (and manages) them? >>>>>>> >>>>>>> I meant to expose some interfaces from pciback. And pciback serves >>>>>>> as the proxy of IRQ (un)binding APIs. >>>>>> >>>>>> If at all possible we should avoid having to change more parties (qemu, >>>>>> libxc, kernel, hypervisor) than really necessary. Remember that such >>>>>> a bug fix may want backporting, and making sure affected people have >>>>>> all relevant components updated is increasingly difficult with their >>>>>> number growing. >>>>>> >>>>>>>>>> in fact I'd expect this to happen in the course of >>>>>>>>>> domain destruction, and I'd expect the device reset to come after the >>>>>>>>>> domain was cleaned up. Perhaps simply an ordering issue in the tool >>>>>>>>>> stack? >>>>>>>>> >>>>>>>>> I don't think reversing the sequences of device reset and domain >>>>>>>>> destruction would be simple. Furthermore, during device hot-unplug, >>>>>>>>> device reset is done when the owner is alive. So if we use domain >>>>>>>>> destruction to enforce all irq binding cleared, in theory, it won't be >>>>>>>>> applicable to hot-unplug case (if qemu's hot-unplug logic is >>>>>>>>> compromised). >>>>>>>> >>>>>>>> Even in the hot-unplug case the tool stack could issue unbind >>>>>>>> requests, behind the back of the possibly compromised qemu, >>>>>>>> once neither the guest nor qemu have access to the device >>>>>>>> anymore. >>>>>>> >>>>>>> But currently, tool stack doesn't know the remaining IRQ bindings. >>>>>>> If tool stack can maintaine IRQ binding information of a pass-thru >>>>>>> device (stored in Xenstore?), we can come up with a clean solution >>>>>>> without modifying linux kernel and Xen. >>>>>> >>>>>> If there's no way for the tool stack to either find out the bindings >>>>>> or "blindly" issue unbind requests (accepting them to fail), then a >>>>>> "wildcard" unbind operation may want adding. Or, perhaps even >>>>>> better, XEN_DOMCTL_deassign_device could unbind anything left >>>>>> in place for the specified device. >>>>> >>>>> Good idea. I will take this advice. >>>>> >>>>> Thanks >>>>> Chao >>>> >>>> I am having the same issue, and cannot find a fix in either xen-pciback or the Xen codebase. >>>> Was a solution ever pushed as a result of this thread? >>>> >>> >>> I submitted patches [1] to Xen community. But I didn't get it merged. >>> We made a change in device driver to disable MSI-X during guest OS >>> shutdown to mitigate the issue. But when guest or qemu was crashed, we >>> encountered this issue again. I have no plan to get back to these >>> patches. But if you want to fix the issue completely along what the >>> patches below did, please go ahead. >>> >>> [1]: https://lists.xenproject.org/archives/html/xen-devel/2019-01/msg01227.html >>> >>> Thanks >>> Chao >>> >> >> Stanislav: Are you able to continue the work with these patches, to get them merged? > >What further work is needed for these patches? Are they only needed for Intel i210 NIC PCI passthrough, or are other devices affected? All MSI-X capable devices were affected. This issue is fixed in Xen by Roger's patch (https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=575e18d54d19eda787f6477a4acd3c50f72751a9). Thanks Chao