From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37B0FC49ED7 for ; Fri, 13 Sep 2019 15:24:36 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 13A0920693 for ; Fri, 13 Sep 2019 15:24:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 13A0920693 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1i8nQu-0000tk-SG; Fri, 13 Sep 2019 15:24:24 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1i8nQt-0000tc-Cz for xen-devel@lists.xenproject.org; Fri, 13 Sep 2019 15:24:23 +0000 X-Inumbo-ID: 8e84e5b0-d63a-11e9-b299-bc764e2007e4 Received: from mga03.intel.com (unknown [134.134.136.65]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 8e84e5b0-d63a-11e9-b299-bc764e2007e4; Fri, 13 Sep 2019 15:24:20 +0000 (UTC) X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Sep 2019 08:24:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,492,1559545200"; d="scan'208";a="179706190" Received: from gao-cwp.sh.intel.com (HELO gao-cwp) ([10.239.159.26]) by orsmga008.jf.intel.com with ESMTP; 13 Sep 2019 08:24:16 -0700 Date: Fri, 13 Sep 2019 23:28:20 +0800 From: Chao Gao To: "Spassov, Stanislav" Message-ID: <20190913152818.GA688@gao-cwp> References: <1543976357-1053-1-git-send-email-chao.gao@intel.com> <2c0ad3bf96551ea6e96e812229507221b76876c6.camel@amazon.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <2c0ad3bf96551ea6e96e812229507221b76876c6.camel@amazon.de> User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Re: [Xen-devel] [PATCH] xen: xen-pciback: Reset MSI-X state when exposing a device X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: "jgross@suse.com" , "sstabellini@kernel.org" , "linux-kernel@vger.kernel.org" , "baijiaju1990@gmail.com" , "jbeulich@suse.com" , "xen-devel@lists.xenproject.org" , "boris.ostrovsky@oracle.com" , "Woodhouse, David" , "roger.pau@citrix.com" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" T24gRnJpLCBTZXAgMTMsIDIwMTkgYXQgMTA6MDI6MjRBTSArMDAwMCwgU3Bhc3NvdiwgU3Rhbmlz bGF2IHdyb3RlOgo+T24gVGh1LCBEZWMgMTMsIDIwMTggYXQgMDc6NTQsIENoYW8gR2FvIHdyb3Rl Ogo+Pk9uIFRodSwgRGVjIDEzLCAyMDE4IGF0IDEyOjU0OjUyQU0gLTA3MDAsIEphbiBCZXVsaWNo IHdyb3RlOgo+Pj4+Pj4gT24gMTMuMTIuMTggYXQgMDQ6NDYsIDxjaGFvLmdhb0BpbnRlbC5jb20+ IHdyb3RlOgo+Pj4+IE9uIFdlZCwgRGVjIDEyLCAyMDE4IGF0IDA4OjIxOjM5QU0gLTA3MDAsIEph biBCZXVsaWNoIHdyb3RlOgo+Pj4+Pj4+PiBPbiAxMi4xMi4xOCBhdCAxNjoxOCwgPGNoYW8uZ2Fv QGludGVsLmNvbT4gd3JvdGU6Cj4+Pj4+PiBPbiBXZWQsIERlYyAxMiwgMjAxOCBhdCAwMTo1MTow MUFNIC0wNzAwLCBKYW4gQmV1bGljaCB3cm90ZToKPj4+Pj4+Pj4+PiBPbiAxMi4xMi4xOCBhdCAw ODowNiwgPGNoYW8uZ2FvQGludGVsLmNvbT4gd3JvdGU6Cj4+Pj4+Pj4+IE9uIFdlZCwgRGVjIDA1 LCAyMDE4IGF0IDA5OjAxOjMzQU0gLTA1MDAsIEJvcmlzIE9zdHJvdnNreSB3cm90ZToKPj4+Pj4+ Pj4+T24gMTIvNS8xOCA0OjMyIEFNLCBSb2dlciBQYXUgTW9ubsOpIHdyb3RlOgo+Pj4+Pj4+Pj4+ IE9uIFdlZCwgRGVjIDA1LCAyMDE4IGF0IDEwOjE5OjE3QU0gKzA4MDAsIENoYW8gR2FvIHdyb3Rl Ogo+Pj4+Pj4+Pj4+PiBJIGZpbmQgc29tZSBwYXNzLXRocnUgZGV2aWNlcyBkb24ndCB3b3JrIGFu eSBtb3JlIGFjcm9zcyBndWVzdCByZWJvb3QuCj4+Pj4+Pj4+Pj4+IEFzc2lnbmluZyBpdCB0byBh bm90aGVyIGd1ZXN0IGFsc28gbWVldHMgdGhlIHNhbWUgaXNzdWUuIEFuZCB0aGUgb25seQo+Pj4+ Pj4+Pj4+PiB3YXkgdG8gbWFrZSBpdCB3b3JrIGFnYWluIGlzIHVuLWJpbmRpbmcgYW5kIGJpbmRp bmcgaXQgdG8gcGNpYmFjay4KPj4+Pj4+Pj4+Pj4gU29tZW9uZSByZXBvcnRlZCB0aGlzIGlzc3Vl IG9uZSB5ZWFyIGFnbyBbMV0uIE1vcmUgZGV0YWlsIGFsc28gY2FuIGJlCj4+Pj4+Pj4+Pj4+IGZv dW5kIGluIFsyXS4KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gVGhlIHJvb3QtY2F1c2UgaXMgWGVu J3MgaW50ZXJuYWwgTVNJLVggc3RhdGUgaXNuJ3QgcmVzZXQgcHJvcGVybHkKPj4+Pj4+Pj4+Pj4g ZHVyaW5nIHJlYm9vdCBvciByZS1hc3NpZ25tZW50LiBJbiB0aGUgYWJvdmUgY2FzZSwgWGVuIHNl dCBtYXNrYWxsIGJpdAo+Pj4+Pj4+Pj4+PiB0byBtYXNrIGFsbCBNU0kgaW50ZXJydXB0cyBhZnRl ciBpdCBkZXRlY3RlZCBhIHBvdGVudGlhbCBzZWN1cml0eQo+Pj4+Pj4+Pj4+PiBpc3N1ZS4gRXZl biBhZnRlciBkZXZpY2UgcmVzZXQsIFhlbiBkaWRuJ3QgcmVzZXQgaXRzIGludGVybmFsIG1hc2th bGwKPj4+Pj4+Pj4+Pj4gYml0LiBBcyBhIHJlc3VsdCwgbWFza2FsbCBiaXQgd291bGQgYmUgc2V0 IGFnYWluIGluIG5leHQgd3JpdGUgdG8KPj4+Pj4+Pj4+Pj4gTVNJLVggbWVzc2FnZSBjb250cm9s IHJlZ2lzdGVyLgo+Pj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4+PiBHaXZlbiB0aGF0IFBIWVNERVZPUFNf cHJlcGFyZV9tc2l4KCkgYWxzbyB0cmlnZ2VycyBYZW4gcmVzZXR0aW5nIE1TSS1YCj4+Pj4+Pj4+ Pj4+IGludGVybmFsIHN0YXRlIG9mIGEgZGV2aWNlLCB3ZSBlbXBsb3kgaXQgdG8gZml4IHRoaXMg aXNzdWUgcmF0aGVyIHRoYW4KPj4+Pj4+Pj4+Pj4gaW50cm9kdWNpbmcgYW5vdGhlciBkZWRpY2F0 ZWQgc3ViLWh5cGVyY2FsbC4KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gTm90ZSB0aGF0IFBIWVNE RVZPUFNfcmVsZWFzZV9tc2l4KCkgd2lsbCBmYWlsIGlmIHRoZSBtYXBwaW5nIGJldHdlZW4KPj4+ Pj4+Pj4+Pj4gdGhlIGRldmljZSdzIG1zaXggYW5kIHBpcnEgaGFzIGJlZW4gY3JlYXRlZC4gVGhp cyBsaW1pdGF0aW9uIHByZXZlbnRzCj4+Pj4+Pj4+Pj4+IHVzIGNhbGxpbmcgdGhpcyBmdW5jdGlv biB3aGVuIGRldGFjaGluZyBhIGRldmljZSBmcm9tIGEgZ3Vlc3QgZHVyaW5nCj4+Pj4+Pj4+Pj4+ IGd1ZXN0IHNodXRkb3duLiBUaHVzIGl0IGlzIGNhbGxlZCByaWdodCBiZWZvcmUgY2FsbGluZwo+ Pj4+Pj4+Pj4+PiBQSFlTREVWT1BTX3ByZXBhcmVfbXNpeCgpLgo+Pj4+Pj4+Pj4+IHMvUEhZU0RF Vk9QUy9QSFlTREVWT1AvIChubyBmaW5hbCBTKS4gQW5kIHRoZW4gSSB3b3VsZCBhbHNvIGRyb3Ag dGhlCj4+Pj4+Pj4+Pj4gKCkgYXQgdGhlIGVuZCBvZiB0aGUgaHlwZXJjYWxsIG5hbWUgc2luY2Ug aXQncyBub3QgYSBmdW5jdGlvbi4KPj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4+IEknbSBhbHNvIHdvbmRl cmluZyB3aHkgdGhlIHJlbGVhc2UgY2FuJ3QgYmUgZG9uZSB3aGVuIHRoZSBkZXZpY2UgaXMKPj4+ Pj4+Pj4+PiBkZXRhY2hlZCBmcm9tIHRoZSBndWVzdCAob3IgdGhlIGd1ZXN0IGhhcyBiZWVuIHNo dXQgZG93bikuIFRoaXMgbWFrZXMKPj4+Pj4+Pj4+PiBtZSB3b3JyeSBhYm91dCB0aGUgcmFjaW5l c3Mgb2YgdGhlIGF0dGFjaC9kZXRhY2ggcHJvY2VkdXJlOiBpZiB0aGVyZSdzCj4+Pj4+Pj4+Pj4g YSBzdGF0ZSB3aGVyZSBwY2liYWNrIGFzc3VtZXMgdGhlIGRldmljZSBoYXMgYmVlbiBkZXRhY2hl ZCBmcm9tIHRoZQo+Pj4+Pj4+Pj4+IGd1ZXN0LCBidXQgdGhlcmUgYXJlIHN0aWxsIHBpcnFzIGJv dW5kLCBhbiBhdHRlbXB0IHRvIGF0dGFjaCB0bwo+Pj4+Pj4+Pj4+IGFub3RoZXIgZ3Vlc3QgaW4g c3VjaCBzdGF0ZSB3aWxsIGZhaWwuCj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj5JIHdvbmRlciB3aGV0aGVy IHRoaXMgYWRkaXRpb25hbCByZXNldCBmdW5jdGlvbmFsaXR5IGNvdWxkIGJlIGRvbmUgb3V0Cj4+ Pj4+Pj4+Pm9mIHhlbl9wY2lia194ZW5idXNfcmVtb3ZlKCkuIFdlIGZpcnN0IGRvIGEgKGJlc3Qg ZWZmb3J0KSBkZXZpY2UgcmVzZXQKPj4+Pj4+Pj4+YW5kIHRoZW4gZG8gdGhlIGV4dHJhIHRoaW5n cyB0aGF0IGFyZSBub3QgcHJvcGVybHkgZG9uZSB0aGVyZS4KPj4+Pj4+Pj4gCj4+Pj4+Pj4+IE5v LiBJdCBjYW5ub3QgYmUgZG9uZSBpbiB4ZW5fcGNpYmtfeGVuYnVzX3JlbW92ZSgpIHdpdGhvdXQg bW9kaWZ5aW5nCj4+Pj4+Pj4+IHRoZSBoYW5kbGVyIG9mIFBIWVNERVZPUF9yZWxlYXNlX21zaXgu IFRvIGRvIGEgc3VjY2Vzc2Z1bCBYZW4gaW50ZXJuYWwKPj4+Pj4+Pj4gTVNJLVggc3RhdGUgcmVz ZXQsIFBIWVNERVZPUF97cmVsZWFzZSwgcHJlcGFyZX1fbXNpeCBzaG91bGQgYmUgZmluaXNoZWQK Pj4+Pj4+Pj4gd2l0aG91dCBlcnJvci4gQnV0IEFUTSwgeGVuIGV4cGVjdHMgdGhhdCBubyBtc2kg aXMgYm91bmQgdG8gcGlycSB3aGVuCj4+Pj4+Pj4+IGRvaW5nIFBIWVNERVZPUF9yZWxlYXNlX21z aXguIE90aGVyd2lzZSBpdCBmYWlscyB3aXRoIGVycm9yIGNvZGUgLUVCVVNZLgo+Pj4+Pj4+PiBI b3dldmVyLCB0aGUgZXhwZWN0YXRpb24gaXNuJ3QgZ3VhcmFudGVlZCBpbiB4ZW5fcGNpYmtfeGVu YnVzX3JlbW92ZSgpLgo+Pj4+Pj4+PiBJbiBzb21lIGNhc2VzLCBpZiBxZW11IGZhaWxzIHRvIHVu bWFwIE1TSXMsIE1TSXMgYXJlIHVubWFwcGVkIGJ5IFhlbgo+Pj4+Pj4+PiBhdCBsYXN0IG1pbnV0 ZSwgd2hpY2ggaGFwcGVucyBhZnRlciBkZXZpY2UgcmVzZXQgaW4gCj4+Pj4+Pj4+IHhlbl9wY2li a194ZW5idXNfcmVtb3ZlKCkuCj4+Pj4+Pj4KPj4+Pj4+PkJ1dCB0aGF0IG1heSBuZWVkIHRha2lu ZyBjYXJlIG9mOiBJIGRvbid0IHRoaW5rIGl0IGlzIGEgZ29vZCBpZGVhIHRvIGhhdmUKPj4+Pj4+ PmFueXRoaW5nIGxlZnQgZnJvbSB0aGUgcHJpb3Igb3duaW5nIGRvbWFpbiB3aGVuIHRoZSBkZXZp Y2UgZ2V0cyByZXNldC4KPj4+Pj4+PkkuZS4gbGVmdCBvdmVyIElSUSBiaW5kaW5ncyBzaG91bGQg cGVyaGFwcyBiZSBmb3JjaWJseSBjbGVhcmVkIGJlZm9yZQo+Pj4+Pj4+aW52b2tpbmcgdGhlIHJl c2V0Owo+Pj4+Pj4gCj4+Pj4+PiBBZ3JlZS4gSG93IGFib3V0IHBjaWJhY2sgdG8gdHJhY2sgdGhl IGVzdGFibGlzaGVkIElSUSBiaW5kaW5ncz8gVGhlbgo+Pj4+Pj4gcGNpYmFjayBjYW4gY2xlYXIg aXJxIGJpbmRpbmcgYmVmb3JlIGludm9raW5nIHRoZSByZXNldC4KPj4+Pj4KPj4+Pj5Ib3cgd291 bGQgcGNpYmFjayBldmVuIGtub3cgb2YgdGhvc2UgbWFwcGluZ3MsIHdoZW4gaXQncyBxZW11Cj4+ Pj4+d2hvIGVzdGFibGlzaGVzIChhbmQgbWFuYWdlcykgdGhlbT8KPj4+PiAKPj4+PiBJIG1lYW50 IHRvIGV4cG9zZSBzb21lIGludGVyZmFjZXMgZnJvbSBwY2liYWNrLiBBbmQgcGNpYmFjayBzZXJ2 ZXMKPj4+PiBhcyB0aGUgcHJveHkgb2YgSVJRICh1biliaW5kaW5nIEFQSXMuCj4+Pgo+Pj5JZiBh dCBhbGwgcG9zc2libGUgd2Ugc2hvdWxkIGF2b2lkIGhhdmluZyB0byBjaGFuZ2UgbW9yZSBwYXJ0 aWVzIChxZW11LAo+Pj5saWJ4Yywga2VybmVsLCBoeXBlcnZpc29yKSB0aGFuIHJlYWxseSBuZWNl c3NhcnkuIFJlbWVtYmVyIHRoYXQgc3VjaAo+Pj5hIGJ1ZyBmaXggbWF5IHdhbnQgYmFja3BvcnRp bmcsIGFuZCBtYWtpbmcgc3VyZSBhZmZlY3RlZCBwZW9wbGUgaGF2ZQo+Pj5hbGwgcmVsZXZhbnQg Y29tcG9uZW50cyB1cGRhdGVkIGlzIGluY3JlYXNpbmdseSBkaWZmaWN1bHQgd2l0aCB0aGVpcgo+ Pj5udW1iZXIgZ3Jvd2luZy4KPj4+Cj4+Pj4+Pj5pbiBmYWN0IEknZCBleHBlY3QgdGhpcyB0byBo YXBwZW4gaW4gdGhlIGNvdXJzZSBvZgo+Pj4+Pj4+ZG9tYWluIGRlc3RydWN0aW9uLCBhbmQgSSdk IGV4cGVjdCB0aGUgZGV2aWNlIHJlc2V0IHRvIGNvbWUgYWZ0ZXIgdGhlCj4+Pj4+Pj5kb21haW4g d2FzIGNsZWFuZWQgdXAuIFBlcmhhcHMgc2ltcGx5IGFuIG9yZGVyaW5nIGlzc3VlIGluIHRoZSB0 b29sCj4+Pj4+Pj5zdGFjaz8KPj4+Pj4+IAo+Pj4+Pj4gSSBkb24ndCB0aGluayByZXZlcnNpbmcg dGhlIHNlcXVlbmNlcyBvZiBkZXZpY2UgcmVzZXQgYW5kIGRvbWFpbgo+Pj4+Pj4gZGVzdHJ1Y3Rp b24gd291bGQgYmUgc2ltcGxlLiBGdXJ0aGVybW9yZSwgZHVyaW5nIGRldmljZSBob3QtdW5wbHVn LAo+Pj4+Pj4gZGV2aWNlIHJlc2V0IGlzIGRvbmUgd2hlbiB0aGUgb3duZXIgaXMgYWxpdmUuIFNv IGlmIHdlIHVzZSBkb21haW4KPj4+Pj4+IGRlc3RydWN0aW9uIHRvIGVuZm9yY2UgYWxsIGlycSBi aW5kaW5nIGNsZWFyZWQsIGluIHRoZW9yeSwgaXQgd29uJ3QgYmUKPj4+Pj4+IGFwcGxpY2FibGUg dG8gaG90LXVucGx1ZyBjYXNlIChpZiBxZW11J3MgaG90LXVucGx1ZyBsb2dpYyBpcwo+Pj4+Pj4g Y29tcHJvbWlzZWQpLgo+Pj4+Pgo+Pj4+PkV2ZW4gaW4gdGhlIGhvdC11bnBsdWcgY2FzZSB0aGUg dG9vbCBzdGFjayBjb3VsZCBpc3N1ZSB1bmJpbmQKPj4+Pj5yZXF1ZXN0cywgYmVoaW5kIHRoZSBi YWNrIG9mIHRoZSBwb3NzaWJseSBjb21wcm9taXNlZCBxZW11LAo+Pj4+Pm9uY2UgbmVpdGhlciB0 aGUgZ3Vlc3Qgbm9yIHFlbXUgaGF2ZSBhY2Nlc3MgdG8gdGhlIGRldmljZQo+Pj4+PmFueW1vcmUu Cj4+Pj4gCj4+Pj4gQnV0IGN1cnJlbnRseSwgdG9vbCBzdGFjayBkb2Vzbid0IGtub3cgdGhlIHJl bWFpbmluZyBJUlEgYmluZGluZ3MuCj4+Pj4gSWYgdG9vbCBzdGFjayBjYW4gbWFpbnRhaW5lIElS USBiaW5kaW5nIGluZm9ybWF0aW9uIG9mIGEgcGFzcy10aHJ1Cj4+Pj4gZGV2aWNlIChzdG9yZWQg aW4gWGVuc3RvcmU/KSwgd2UgY2FuIGNvbWUgdXAgd2l0aCBhIGNsZWFuIHNvbHV0aW9uCj4+Pj4g d2l0aG91dCBtb2RpZnlpbmcgbGludXgga2VybmVsIGFuZCBYZW4uCj4+Pgo+Pj5JZiB0aGVyZSdz IG5vIHdheSBmb3IgdGhlIHRvb2wgc3RhY2sgdG8gZWl0aGVyIGZpbmQgb3V0IHRoZSBiaW5kaW5n cwo+Pj5vciAiYmxpbmRseSIgaXNzdWUgdW5iaW5kIHJlcXVlc3RzIChhY2NlcHRpbmcgdGhlbSB0 byBmYWlsKSwgdGhlbiBhCj4+PiJ3aWxkY2FyZCIgdW5iaW5kIG9wZXJhdGlvbiBtYXkgd2FudCBh ZGRpbmcuIE9yLCBwZXJoYXBzIGV2ZW4KPj4+YmV0dGVyLCBYRU5fRE9NQ1RMX2RlYXNzaWduX2Rl dmljZSBjb3VsZCB1bmJpbmQgYW55dGhpbmcgbGVmdAo+Pj5pbiBwbGFjZSBmb3IgdGhlIHNwZWNp ZmllZCBkZXZpY2UuCj4+Cj4+R29vZCBpZGVhLiBJIHdpbGwgdGFrZSB0aGlzIGFkdmljZS4KPj4K Pj5UaGFua3MKPj5DaGFvCj4KPkkgYW0gaGF2aW5nIHRoZSBzYW1lIGlzc3VlLCBhbmQgY2Fubm90 IGZpbmQgYSBmaXggaW4gZWl0aGVyIHhlbi1wY2liYWNrIG9yIHRoZSBYZW4gY29kZWJhc2UuCj5X YXMgYSBzb2x1dGlvbiBldmVyIHB1c2hlZCBhcyBhIHJlc3VsdCBvZiB0aGlzIHRocmVhZD8KPgoK SSBzdWJtaXR0ZWQgcGF0Y2hlcyBbMV0gdG8gWGVuIGNvbW11bml0eS4gQnV0IEkgZGlkbid0IGdl dCBpdCBtZXJnZWQuCldlIG1hZGUgYSBjaGFuZ2UgaW4gZGV2aWNlIGRyaXZlciB0byBkaXNhYmxl IE1TSS1YIGR1cmluZyBndWVzdCBPUwpzaHV0ZG93biB0byBtaXRpZ2F0ZSB0aGUgaXNzdWUuIEJ1 dCB3aGVuIGd1ZXN0IG9yIHFlbXUgd2FzIGNyYXNoZWQsIHdlCmVuY291bnRlcmVkIHRoaXMgaXNz dWUgYWdhaW4uIEkgaGF2ZSBubyBwbGFuIHRvIGdldCBiYWNrIHRvIHRoZXNlCnBhdGNoZXMuIEJ1 dCBpZiB5b3Ugd2FudCB0byBmaXggdGhlIGlzc3VlIGNvbXBsZXRlbHkgYWxvbmcgd2hhdCB0aGUK cGF0Y2hlcyBiZWxvdyBkaWQsIHBsZWFzZSBnbyBhaGVhZC4KClsxXTogaHR0cHM6Ly9saXN0cy54 ZW5wcm9qZWN0Lm9yZy9hcmNoaXZlcy9odG1sL3hlbi1kZXZlbC8yMDE5LTAxL21zZzAxMjI3Lmh0 bWwKClRoYW5rcwpDaGFvCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fXwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZlbEBsaXN0cy54ZW5wcm9qZWN0 Lm9yZwpodHRwczovL2xpc3RzLnhlbnByb2plY3Qub3JnL21haWxtYW4vbGlzdGluZm8veGVuLWRl dmVs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36284C49ED7 for ; Fri, 13 Sep 2019 15:24:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0828B20693 for ; Fri, 13 Sep 2019 15:24:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390687AbfIMPYU (ORCPT ); Fri, 13 Sep 2019 11:24:20 -0400 Received: from mga09.intel.com ([134.134.136.24]:46516 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388354AbfIMPYT (ORCPT ); Fri, 13 Sep 2019 11:24:19 -0400 X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Sep 2019 08:24:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,492,1559545200"; d="scan'208";a="179706190" Received: from gao-cwp.sh.intel.com (HELO gao-cwp) ([10.239.159.26]) by orsmga008.jf.intel.com with ESMTP; 13 Sep 2019 08:24:16 -0700 Date: Fri, 13 Sep 2019 23:28:20 +0800 From: Chao Gao To: "Spassov, Stanislav" Cc: "linux-kernel@vger.kernel.org" , "boris.ostrovsky@oracle.com" , "baijiaju1990@gmail.com" , "Woodhouse, David" , "sstabellini@kernel.org" , "jgross@suse.com" , "roger.pau@citrix.com" , "jbeulich@suse.com" , "xen-devel@lists.xenproject.org" Subject: Re: [PATCH] xen: xen-pciback: Reset MSI-X state when exposing a device Message-ID: <20190913152818.GA688@gao-cwp> References: <1543976357-1053-1-git-send-email-chao.gao@intel.com> <2c0ad3bf96551ea6e96e812229507221b76876c6.camel@amazon.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <2c0ad3bf96551ea6e96e812229507221b76876c6.camel@amazon.de> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Sep 13, 2019 at 10:02:24AM +0000, Spassov, Stanislav wrote: >On Thu, Dec 13, 2018 at 07:54, Chao Gao wrote: >>On Thu, Dec 13, 2018 at 12:54:52AM -0700, Jan Beulich wrote: >>>>>> On 13.12.18 at 04:46, wrote: >>>> On Wed, Dec 12, 2018 at 08:21:39AM -0700, Jan Beulich wrote: >>>>>>>> On 12.12.18 at 16:18, wrote: >>>>>> On Wed, Dec 12, 2018 at 01:51:01AM -0700, Jan Beulich wrote: >>>>>>>>>> On 12.12.18 at 08:06, wrote: >>>>>>>> On Wed, Dec 05, 2018 at 09:01:33AM -0500, Boris Ostrovsky wrote: >>>>>>>>>On 12/5/18 4:32 AM, Roger Pau Monné wrote: >>>>>>>>>> On Wed, Dec 05, 2018 at 10:19:17AM +0800, Chao Gao wrote: >>>>>>>>>>> I find some pass-thru devices don't work any more across guest reboot. >>>>>>>>>>> Assigning it to another guest also meets the same issue. And the only >>>>>>>>>>> way to make it work again is un-binding and binding it to pciback. >>>>>>>>>>> Someone reported this issue one year ago [1]. More detail also can be >>>>>>>>>>> found in [2]. >>>>>>>>>>> >>>>>>>>>>> The root-cause is Xen's internal MSI-X state isn't reset properly >>>>>>>>>>> during reboot or re-assignment. In the above case, Xen set maskall bit >>>>>>>>>>> to mask all MSI interrupts after it detected a potential security >>>>>>>>>>> issue. Even after device reset, Xen didn't reset its internal maskall >>>>>>>>>>> bit. As a result, maskall bit would be set again in next write to >>>>>>>>>>> MSI-X message control register. >>>>>>>>>>> >>>>>>>>>>> Given that PHYSDEVOPS_prepare_msix() also triggers Xen resetting MSI-X >>>>>>>>>>> internal state of a device, we employ it to fix this issue rather than >>>>>>>>>>> introducing another dedicated sub-hypercall. >>>>>>>>>>> >>>>>>>>>>> Note that PHYSDEVOPS_release_msix() will fail if the mapping between >>>>>>>>>>> the device's msix and pirq has been created. This limitation prevents >>>>>>>>>>> us calling this function when detaching a device from a guest during >>>>>>>>>>> guest shutdown. Thus it is called right before calling >>>>>>>>>>> PHYSDEVOPS_prepare_msix(). >>>>>>>>>> s/PHYSDEVOPS/PHYSDEVOP/ (no final S). And then I would also drop the >>>>>>>>>> () at the end of the hypercall name since it's not a function. >>>>>>>>>> >>>>>>>>>> I'm also wondering why the release can't be done when the device is >>>>>>>>>> detached from the guest (or the guest has been shut down). This makes >>>>>>>>>> me worry about the raciness of the attach/detach procedure: if there's >>>>>>>>>> a state where pciback assumes the device has been detached from the >>>>>>>>>> guest, but there are still pirqs bound, an attempt to attach to >>>>>>>>>> another guest in such state will fail. >>>>>>>>> >>>>>>>>>I wonder whether this additional reset functionality could be done out >>>>>>>>>of xen_pcibk_xenbus_remove(). We first do a (best effort) device reset >>>>>>>>>and then do the extra things that are not properly done there. >>>>>>>> >>>>>>>> No. It cannot be done in xen_pcibk_xenbus_remove() without modifying >>>>>>>> the handler of PHYSDEVOP_release_msix. To do a successful Xen internal >>>>>>>> MSI-X state reset, PHYSDEVOP_{release, prepare}_msix should be finished >>>>>>>> without error. But ATM, xen expects that no msi is bound to pirq when >>>>>>>> doing PHYSDEVOP_release_msix. Otherwise it fails with error code -EBUSY. >>>>>>>> However, the expectation isn't guaranteed in xen_pcibk_xenbus_remove(). >>>>>>>> In some cases, if qemu fails to unmap MSIs, MSIs are unmapped by Xen >>>>>>>> at last minute, which happens after device reset in >>>>>>>> xen_pcibk_xenbus_remove(). >>>>>>> >>>>>>>But that may need taking care of: I don't think it is a good idea to have >>>>>>>anything left from the prior owning domain when the device gets reset. >>>>>>>I.e. left over IRQ bindings should perhaps be forcibly cleared before >>>>>>>invoking the reset; >>>>>> >>>>>> Agree. How about pciback to track the established IRQ bindings? Then >>>>>> pciback can clear irq binding before invoking the reset. >>>>> >>>>>How would pciback even know of those mappings, when it's qemu >>>>>who establishes (and manages) them? >>>> >>>> I meant to expose some interfaces from pciback. And pciback serves >>>> as the proxy of IRQ (un)binding APIs. >>> >>>If at all possible we should avoid having to change more parties (qemu, >>>libxc, kernel, hypervisor) than really necessary. Remember that such >>>a bug fix may want backporting, and making sure affected people have >>>all relevant components updated is increasingly difficult with their >>>number growing. >>> >>>>>>>in fact I'd expect this to happen in the course of >>>>>>>domain destruction, and I'd expect the device reset to come after the >>>>>>>domain was cleaned up. Perhaps simply an ordering issue in the tool >>>>>>>stack? >>>>>> >>>>>> I don't think reversing the sequences of device reset and domain >>>>>> destruction would be simple. Furthermore, during device hot-unplug, >>>>>> device reset is done when the owner is alive. So if we use domain >>>>>> destruction to enforce all irq binding cleared, in theory, it won't be >>>>>> applicable to hot-unplug case (if qemu's hot-unplug logic is >>>>>> compromised). >>>>> >>>>>Even in the hot-unplug case the tool stack could issue unbind >>>>>requests, behind the back of the possibly compromised qemu, >>>>>once neither the guest nor qemu have access to the device >>>>>anymore. >>>> >>>> But currently, tool stack doesn't know the remaining IRQ bindings. >>>> If tool stack can maintaine IRQ binding information of a pass-thru >>>> device (stored in Xenstore?), we can come up with a clean solution >>>> without modifying linux kernel and Xen. >>> >>>If there's no way for the tool stack to either find out the bindings >>>or "blindly" issue unbind requests (accepting them to fail), then a >>>"wildcard" unbind operation may want adding. Or, perhaps even >>>better, XEN_DOMCTL_deassign_device could unbind anything left >>>in place for the specified device. >> >>Good idea. I will take this advice. >> >>Thanks >>Chao > >I am having the same issue, and cannot find a fix in either xen-pciback or the Xen codebase. >Was a solution ever pushed as a result of this thread? > I submitted patches [1] to Xen community. But I didn't get it merged. We made a change in device driver to disable MSI-X during guest OS shutdown to mitigate the issue. But when guest or qemu was crashed, we encountered this issue again. I have no plan to get back to these patches. But if you want to fix the issue completely along what the patches below did, please go ahead. [1]: https://lists.xenproject.org/archives/html/xen-devel/2019-01/msg01227.html Thanks Chao