From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexey Gerasimenko Subject: [RFC PATCH 00/30] Xen Q35 Bringup patches + support for PCIe Extended Capabilities for passed through devices Date: Tue, 13 Mar 2018 04:33:45 +1000 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1evSHF-0006zc-Gs for xen-devel@lists.xenproject.org; Mon, 12 Mar 2018 18:34:29 +0000 Received: by mail-pl0-x22a.google.com with SMTP id c11-v6so9830184plo.0 for ; Mon, 12 Mar 2018 11:34:28 -0700 (PDT) List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" To: xen-devel@lists.xenproject.org Cc: Alexey Gerasimenko , qemu-devel@nongnu.org List-Id: xen-devel@lists.xenproject.org VGhpcyBwYXRjaCBzZXJpZXMgaW50cm9kdWNlcyBzdXBwb3J0IG9mIFEzNSBlbXVsYXRpb24gZm9y IFhlbiBIVk0gZ3Vlc3RzCih2aWEgUUVNVSkuIFRoaXMgZmVhdHVyZSBpcyBwcmVzZW50IGluIG90 aGVyIHZpcnR1YWxpemF0aW9uIHByb2R1Y3RzIGFuZApYZW4gY2FuIGdyZWF0bHkgYmVuZWZpdCBm cm9tIHRoaXMgZmVhdHVyZSBhcyB3ZWxsLgoKVGhlIG1haW4gZ29hbCBmb3IgaW1wbGVtZW50aW5n IFEzNSBlbXVsYXRpb24gZm9yIFhlbiB3YXMgZXh0ZW5kaW5nIFBDSS9HUFUKcGFzc3Rocm91Z2gg Y2FwYWJpbGl0aWVzLiBJdCdzIHRoZSBtYWluIGFkdmFudGFnZSBvZiBRMzUgZW11bGF0aW9uCi0g YXZhaWxhYmlsaXR5IG9mIGV4dHJhIGZlYXR1cmVzIGZvciBQQ0llIGRldmljZSBwYXNzdGhyb3Vn aC4gVGhlIG1vc3QKaW1wb3J0YW50IFBDSWUtc3BlY2lmaWMgcGFzc3Rocm91Z2ggZmVhdHVyZSBR MzUgcHJvdmlkZXMgaXMgYSBzdXBwb3J0IGZvcgpQQ0llIGNvbmZpZyBzcGFjZSBFQ0FNIChha2Eg TU1DT05GSUcpIHRvIGFsbG93IGFjY2Vzc2VzIHRvIGV4dGVuZGVkIFBDSWUKY29uZmlnIHNwYWNl ICg+MjU2KSwgd2hpY2ggaXMgTU1JTy1iYXNlZC4gIExvdHMgb2YgUENJZSBkZXZpY2VzIGFuZCB0 aGVpcgpkcml2ZXJzIG1ha2UgdXNlIG9mIFBDSWUgRXh0ZW5kZWQgQ2FwYWJpbGl0aWVzLCB3aG9z ZSBjYW4gYmUgYWNjZXNzZWQgb25seQp1c2luZyBFQ0FNIGFuZCBvZmZzZXRzIGFib3ZlIDB4MTAw IGluIFBDSSBjb25maWcgc3BhY2UuIFN1cHBvcnRpbmcgRUNBTQppcyBhIG1hbmRhdG9yeSBmZWF0 dXJlIGZvciBQQ0llIHBhc3N0aHJvdWdoLiBOb3Qgb25seSB0aGlzIGFsbG93cwpwYXNzdGhyb3Vn aCBQQ0llIGRldmljZXMgdG8gZnVuY3Rpb24gcHJvcGVybHksIGJ1dCBvcGVucyBhIHJvYWQgdG8g ZXh0ZW5kClhlbiBQQ0llIHBhc3N0aHJvdWdoIGZlYXR1cmVzIGZ1cnRoZXIgLS0gZWcuIHByb3Zp ZGluZyBzdXBwb3J0IGZvciBBRVIuIE9uZQpvZiBwb3NzaWJsZSBkaXJlY3Rpb25zIGlzIHByb3Zp ZGluZyBzdXBwb3J0IGZvciBQQ0llIFJlc2l6YWJsZSBCQVJzIC0tCmEgZmVhdHVyZSB3aGljaCBs aWtlbHkgdG8gYmVjb21lIGNvbW1vbiBmb3IgbW9kZXJuIEdQVXMgYXMgdmlkZW8gbWVtb3J5CnNp emVzIGluY3JlYXNlLgoKUTM1IGVtdWxhdGlvbiBtYXkgYWxzbyBiZSB1c2VmdWwgZm9yIG90aGVy IHB1cnBvc2VzLiBJbiBmYWN0LCB0aGUgZW11bGF0aW9uCm9mIGEgbW9yZSByZWNlbnQgY2hpcHNl dCBwYXJ0aWFsbHkgY2xvc2VzIGEgaHVnZSBnYXAgYmV0d2VlbiBhIHNldCBvZgpyZXF1aXJlZCBw bGF0Zm9ybSBmZWF0dXJlcyBhbmQgdGhlIGFjdHVhbCBlbXVsYXRlZCBwbGF0Zm9ybSBjYXBhYmls aXRpZXMKLSBsb3Qgb2YgcmVxdWlyZWQgZnVuY3Rpb25hbGl0eSBpcyBhY3R1YWxseSBtaXNzaW5n IGluIGEgcmVhbCBpNDQwIGNoaXBzZXQuCk9uZSBjYW4gbG9vayBhdCBJR0QgcGFzc3RocnUgc3Vw cG9ydCBwYXRjaGVzIGZyb20gSW50ZWwgZm9yIGV4YW1wbGU6CmFjY29yZGluZyB0byBjb2RlIGNv bW1lbnRzLCB0aGV5IGhhZCB0byBjcmVhdGUgYSBkdW1teSBQQ0ktSVNBIGJyaWRnZQphdCBCREYg MDoxRi4wIGluIG9yZGVyIHRvIG1ha2UgdGhlIG9sZCBpNDQwIHN5c3RlbSBsb29rIG1vcmUgbW9k ZXJuLCBqdXN0CnRvIG1ha2UgaXQgY29tcGF0aWJsZSB3aXRoIElHRCBkcml2ZXIuIFVzaW5nIFEz NSBlbXVsYXRpb24gd2l0aCBpdHMgb3duCmVtdWxhdGVkIExQQyBicmlkZ2UgYWxsb3dzIHRvIGF2 b2lkIHdvcmthcm91bmRzIGxpa2UgdGhpcy4gaTQ0MCBvbiBpdHMgb3duCmlzIGEgZmFpcmx5IG91 dGRhdGVkIHN5c3RlbSBhbmQgZG9lc24ndCByZWFsbHkgc3VwcG9ydCBsb3Qgb2YgdGhpbmdzLCBs aWtlCk1NSU8gaG9sZSBhYm92ZSA0R2IgKGFsdGhvdWdoIGl0IGlzIGFjdHVhbGx5IGVtdWxhdGVk KS4gQWxzbywgZHVlIHRvIHRoZQppNDQwIGNoaXBzZXQncyBhZ2UgdGhlIG9ubHkgZmFjdCBvZiBp dHMgdXNhZ2UgbWF5IGJlIHVzZWQgYXMgYSByZWxpYWJsZQptZXRob2QgdG8gZGV0ZWN0IGEgdmly dHVhbGl6ZWQgZW52aXJvbm1lbnQgYnkgc29tZSBtYWxpY2lvdXMgc29mdHdhcmUKZXNwZWNpYWxs eSBjb25zaWRlcmluZyB0aGUgZmFjdCB0aGF0IGk0NDAgZW11bGF0aW9uIGlzIHNoYXJlZCBhbW9u ZwptdWx0aXBsZSB2aXJ0dWFsaXphdGlvbiBwcm9kdWN0cy4KCk9uIHRvcCBvZiB0aGlzIHNlcmll cyBJJ3ZlIGFsc28gaW1wbGVtZW50ZWQgYSBzb2x1dGlvbiB3aGljaCBzb2x2ZXMKZXhpc3Rpbmcg WGVuIHB1enpsZSB3aXRoIEhWTSBtZW1vcnkgbGF5b3V0IC0tIGhhbmRsaW5nIG9mIFZSQU0sIFJN UlJzIGFuZApNTUlPIGhvbGUgaW4gZ2VuZXJhbC4gVGhpcyAicHV6emxlIiAobWVtb3J5IGxheW91 dCBpbmNvbnNpc3RlbmN5IGJldHdlZW4KbGlieGwvbGlieGMsIGh2bWxvYWRlciBhbmQgUUVNVSkg aXMgYSBzb3J0IG9mIGZ1bmRhbWVudGFsIHByb2JsZW0gd2hpY2gKcGxhZ3VlcyBYZW4gZm9yIHll YXJzIGFuZCBhbW9uZyBmZXcgb3RoZXIgaXNzdWVzIHByZXZlbnRzIFhlbiB0byBiZWNvbWUgYQpk ZWNlbnQgR1BVL1BDSWUgcGFzc3Rocm91Z2ggcGxhdGZvcm0gKHdoaWNoIGl0IHNob3VsZCBiZSku IFRoaXMgc29sdXRpb24KYWxzbyBhbGxvd3MgdG8gbGF0ZXIgcmVzb2x2ZSBjdXJyZW50IFBDSSBw YXNzdGhyb3VnaCBpbmNvbXBhdGliaWxpdHkKaXNzdWVzLCBlZy4gd2l0aCBQb3B1bGF0ZS1vbi1E ZW1hbmQuIEluIGZhY3QsIGk0NDAgc3VwcG9ydCBoYXMgYmVlbiBhZGRlZAphcyB3ZWxsLCBidXQg aXQncyBhIGJpdCBoYWNreSBhcyBpdCB1c2VzIE5CIHJlZ2lzdGVycyB3aGljaCBhcmUgbm90IHBy ZXNlbnQKaW4gYSByZWFsIGk0NDAgKHdlbGwsIG9uZSBtb3JlIG5vbi1leGlzdGluZyBpNDQwIGZl YXR1cmUgd29uJ3QgaGFybSBhbnl3YXkKYXMgdGhlcmUgYXJlIHBsZW50eSBvZiB0aGVtIGFscmVh ZHkpLiBJJ20gcGxhbm5pbmcgdG8gc2VuZCBSRkMgcGF0Y2hlcyBvZgp0aGlzIHNvbHV0aW9uIHJp Z2h0IGFmdGVyIGN1cnJlbnQgcGF0Y2hlcyB3aWxsIGJlIHJldmlld2VkIGFuZCByZWxhdGVkIGNv ZGUKc2V0dGxlLCB0byByZWJhc2UgcGF0Y2hlcyBvbiB0b3Agb2YgaXQuIEFsc28sIGEgZ29vZCBk ZXNjcmlwdGlvbiBpcwpyZXF1aXJlZCBhcyB0aGUgY2hhbmdlIGlzIHJhdGhlciByYWRpY2FsLgoK VGhlIGdvb2QgdGhpbmcgaXMgdGhhdCBwcm92aWRpbmcgUTM1IHN1cHBvcnQgZm9yIFhlbiBhdCB0 aGlzIHN0YWdlIG5laXRoZXIKYnJlYWsgYW55IGV4aXN0aW5nIGZ1bmN0aW9uYWxpdHkgbm9yIGFm ZmVjdCB0aGUgbGVnYWN5IGk0NDAgZW11bGF0aW9uCmluIGFueSB3YXkgLSBRMzUgZW11bGF0aW9u IGNhbiBiZSBlbmFibGVkIG9uIGRlbWFuZCBvbmx5LCB1c2luZyBhIG5ldwpkb21haW4gY29uZmln IG9wdGlvbi4gQWxzbywgb25seSBleGlzdGluZyBpbnRlcmZhY2VzIGFyZSB1c2VkLCBubyBuZXcK aHlwZWNhbGxzIHdlcmUgaW50cm9kdWNlZCwgbm8gQVBJIGNoYW5nZXMsIGV0Yy4gQWx0aG91Z2gg aW4gdGhlIGZ1dHVyZQp3ZSdsbCBoYXZlIHRvIGNoYW5nZSBzb21lIGh5cGVyY2FsbC9RTVAvZXRj IGludGVyZmFjZXMgdG8gcmVtb3ZlCmxpbWl0YXRpb25zIGFuZCBleHRlbmQgdGhlIFEzNS9QQ0ll IHBhc3N0aHJ1IHN1cHBvcnQgZnVydGhlci4KCkN1cnJlbnQgZmVhdHVyZXMgYW5kIGxpbWl0YXRp b25zOgotIEFsbCBiYXNpYyBmdW5jdGlvbmFsaXR5IHdvcmtzIG5vcm1hbGx5IC0gTVAsIG5ldHdv cmtpbmcsIHN0b3JhZ2UgKEFIQ0kpLAogIHBvd2VyaW5nIGRvd24gVk1zIHZpYSBBQ1BJIHNvZnQg b2ZmLCBldGMKLSBYZW4gUGxhdGZvcm0gRGV2aWNlIGFuZCBQViBkZXZpY2VzIGFyZSBzdXBwb3J0 ZWQgLS0gUFYgZHJpdmVycyBmb3IgdmJkLAogIHZpZiwgZXRjIG1heSBiZSBpbnN0YWxsZWQgYW5k IHVzZWQKLSBQQ0llIEVDQU0gZnVsbHkgc3VwcG9ydGVkLCB3aXRoIGFsbG9jYXRpbmcgc3BhY2Ug Zm9yIFBDSUVYQkFSIGluIE1NSU8KICBob2xlLCBBQ1BJIE1DRkcgZ2VuZXJhdGlvbiwgZXRjLgot IFhlbiBpcyBsaW1pdGVkIHRvIG1heCA0IFBJUlFzIGluIG11bHRpcGxlIHBsYWNlcywgd2hpbGUg UTM1IGhhdmUgc3VwcG9ydAogIG9mIDggUElSUXMgLyBQQ0kgcm91dGVyIGxpbmtzLiBUaGlzIHdh cyB3b3JrYXJvdW5kZWQgYnkgZGVzY3JpYmluZyBvbmx5CiAgNCB1c2FibGUgSVJRIGxpbmsgZW50 cmllcyBpbiBBQ1BJIHRhYmxlcyBhbmQgZGlzYWJsaW5nIFBJUlFFLi5QSVJRSCAtLSBsaWtlCiAg d2UncmUgb24gYSByZWFsIHN5c3RlbSB3aGljaCBoYXMgb25seSBzb21lIG9mIDggYXZhaWxhYmxl IFBJUlFzIHBoeXNpY2FsbHkKICBjb25uZWN0ZWQgb24gdGhlIGNoaXBzZXQuIEV4dGVuZGluZyB0 aGUgbnVtYmVyIG9mIFBDSSBsaW5rcyBzdXBwb3J0ZWQKICBpcyB0cml2aWFsLCBidXQgdGhpcyBz dGVwIHdpbGwgY2hhbmdlIHRoZSBzYXZlL21pZ3JhdGlvbiBzdHJlYW0gZm9ybWF0CiAgYSBiaXQu Li4gYWx0aG91Z2ggYXMgaXQgc2VlbXMgdGhlcmUgd2FzIGFjdHVhbGx5IHNvbWUgcGxhY2UgZm9y IHRoaXMKICBleHRlbnNpb24gYmVpbmcgbGVmdCAtLSBlZy4gZmllbGQgdWludDhfdCByb3V0ZVs0 XSBmb2xsb3dlZCBieSB1aW50OF90CiAgcGFkMFs0XSBpbiBodm1faHdfcGNpX2xpbmsgc3RydWN0 dXJlLiBBbnl3YXksIHRoZXJlIGlzIG5vIHByb2JsZW0gYWN0dWFsbHkKICBhcyB3ZSBub3JtYWxs eSBkZWFsIHdpdGggQVBJQyBtb2RlIChvciBNU0lzKSBmb3IgSVJRIGRlbGl2ZXJ5LCB3aGlsZSBQ SUMKICBtb2RlIHdpdGggUENJIHJvdXRpbmcgbmVlZGVkIG9ubHkgZm9yIGxlZ2FjeSBjb21wYXRp YmlsaXR5Ci0gUENJIGhvdHBsdWcgY3VycmVudGx5IGltcGxlbWVudGVkIHZpYSBBQ1BJIGhvdHBs dWcsIGluIGEgd2F5IHNpbWlsYXIKICB0byBpNDQwLiBJbiBmdXR1cmUsIHRoaXMgbWlnaHQgYmUg Y2hhbmdlZCB0byBuYXRpdmUgUENJZSBob3RwbHVnIGZhY2lsaXRpZXMKICAoaWYgdGhlcmUgd2ls bCBiZSBhIGJlbmVmaXQpLgotIEZvciBQQ0llIHBhc3N0aHJvdWdoIHRvIHdvcmsgb24gV2luZG93 cyA3IGFuZCBhYm92ZSwgYSBzcGVjaWZpYwogIHdvcmthcm91bmQgd2FzIGltcGxlbWVudGVkLCB3 aGljaCBhbGxvd3MgdG8gdXNlIFBDSWUgZGV2aWNlIHBhc3N0aHJvdWdoCiAgb24gdGhvc2UgZ3Vl c3QgT1NlcyBub3JtYWxseS4gSW4gZnV0dXJlLCB0aGlzIHNob3VsZCBiZSBjaGFuZ2VkIHRvIGEg bmV3CiAgZW11bGF0ZWQgUENJIGFyY2hpdGVjdHVyZSBmb3IgWGVuIC0tIHByb3ZpZGluZyBzdXBw b3J0IGZvciBzaW1wbGUgUENJCiAgaGllcmFyY2hpZXMsIG5lc3RlZCBNTUlPIHNwYWNlcywgZXRj LiBCYXNpY2FsbHksIHdlIG5lZWQgYXQgbGVhc3QKICB0byBwcm92aWRlIHN1cHBvcnQgZm9yIFBD SS1QQ0kgYnJpZGdlcyAoUENJZSBSb290IFBvcnRzIGluIG91ciBjYXNlKS4KICBDdXJyZW50bHkg WGVuIGxpbWl0ZWQgdG8gYnVzIDAgaW4gbWFueSBwbGFjZXMsIGV2ZW4gaW4gaHlwZXJjYWxsCiAg cGFyYW1ldGVycy4gQSBkZXRhaWxlZCBkZXNjcmlwdGlvbiBvZiB0aGUgaXNzdWUgY2FuIGJlIGZv dW5kIGluIHRoZSBwYXRjaAogIG5hbWVkICJ4ZW4vcHQ6IFhlbiBQQ0llIHBhc3N0aHJvdWdoIHN1 cHBvcnQgZm9yIFEzNTogYnlwYXNzIFBDSWUgdG9wb2xvZ3kKICBjaGVjayIuCi0gVk0gbWlncmF0 aW9uIHdhcyBub3QgdGVzdGVkIGFzIHRoZSBmZWF0dXJlIHByaW1hcmlseSB0YXJnZXRzIHRoZSBQ Q0llCiAgcGFzc3Rocm91Z2ggd2hpY2ggZG9lc24ndCBjb21wYXRpYmxlIHdpdGggbWlncmF0aW9u IGFueXdheS4KCkhvdyB0byB1c2UgdGhlIFEzNSBmZWF0dXJlOgoKQSBuZXcgZG9tYWluIGNvbmZp ZyBvcHRpb24gd2FzIGltcGxlbWVudGVkOiBkZXZpY2VfbW9kZWxfbWFjaGluZS4gSXQncwphIHN0 cmluZyB3aGljaCBoYXMgZm9sbG93aW5nIHBvc3NpYmxlIHZhbHVlczoKLSAiaTQ0MCIgLS0gaTQ0 MCBlbXVsYXRpb24gKGRlZmF1bHQpCi0gInEzNSIgIC0tIGVtdWxhdGUgYSBRMzUgbWFjaGluZS4g QnkgZGVmYXVsdCwgdGhlIHN0b3JhZ2UgaW50ZXJmYWNlIGlzCiAgQUhDSS4KCk5vdGUgdGhhdCBv bWl0dGluZyBkZXZpY2VfbW9kZWxfbWFjaGluZSBwYXJhbWV0ZXIgbWVhbnMgaTQ0MCBzeXN0ZW0K YnkgZGVmYXVsdCwgc28gdGhlIGRlZmF1bHQgYmVoYXZpb3IgZG9lc24ndCBjaGFuZ2UgZm9yIG9s ZCBkb21haW4gY29uZmlnCmZpbGVzLgoKU28sIGluIG9yZGVyIHRvIGVuYWJsZSBRMzUgZW11bGF0 aW9uIG9uZSBuZWVkIHRvIHNwZWNpZnkgdGhlIGZvbGxvd2luZwpvcHRpb24gaW4gdGhlIGRvbWFp biBjb25maWcgZmlsZToKZGV2aWNlX21vZGVsX21hY2hpbmU9InEzNSIKCkl0IGlzIHJlY29tbWVu ZGVkIHRvIGluc3RhbGwgdGhlIGd1ZXN0IE9TIGZyb20gc2NyYXRjaCB0byBhdm9pZCBpc3N1ZXMg ZHVlCnRvIHRoZSBlbXVsYXRlZCBwbGF0Zm9ybSBjaGFuZ2UuCgpPbmUgZXh0cmEgbm90ZSAtIGlm IHlvdSdyZSBnb2luZyB0byBiYWNrcG9ydCB0aGlzIHNlcmllcyB0byBzb21lIG9sZGVyIFFFTVUK dmVyc2lvbiwgbWFrZSBzdXJlIHlvdSBoYXZlIHRoaXMgcGF0Y2ggZm9yIEFIQ0kgRE1BIGJ1ZyBh cHBsaWVkOiBbMV0uCk90aGVyd2lzZSB5b3Ugd2lsbCBlbmNvdW50ZXIgIHJhbmRvbSBRMzUgZ3Vl c3QgaGFuZ3VwcyB3aXRoICJCYWQgUkFNCm9mZnNldCIgbWVzc2FnZSBsb2dnZWQgaW4gL3Zhci9s b2cveGVuLiBSZWNlbnQgUUVNVSB2ZXJzaW9ucyBoYXZlIHRoaXMKcGF0Y2ggY29tbWl0ZWQgYWxy ZWFkeS4KCkFsc28sIGEgY29tbWl0IFsyXSBpcyByZXF1aXJlZCB0byBiZSBhcHBsaWVkIChmb3Ig eGVuLXB0LmMpIC0tIGl0IGlzCmF2YWlsYWJsZSBpbiB0aGUgdXBzdHJlYW0gUUVNVSBjdXJyZW50 bHksIGJ1dCBub3QgcHJlc2VudCBpbiBxZW11LXhlbi4KClRoaXMgaXMgbXkgZmlyc3QgKHNvbWV3 aGF0KSBsYXJnZSBjb250cmlidXRpb24gdG8gWGVuLCBzbyBzb21lIG1pc3Rha2VzCmFyZSB0byBi ZSBleHBlY3RlZC4gTW9zdCB0ZXN0aW5nIHdhcyBkb25lIHVzaW5nIHByZXZpb3VzIHZlcnNpb24g b2YgcGF0Y2hlcwphbmQgWGVuIDQuOC54LgoKSSBwbGFuIHRvIHN1cHBvcnQgYW5kIGV4dGVuZCB0 aGlzIHNlcmllcyBmdXJ0aGVyLCBmb3Igbm93IEkgZXhwZWN0IHNvbWUKY29tbWVudHMvc3VnZ2Vz dGlvbnMvdGVzdGluZyByZXN1bHRzL2J1Z3JlcG9ydHMuCgpbMV06IGh0dHBzOi8vbGlzdHMueGVu Lm9yZy9hcmNoaXZlcy9odG1sL3hlbi1kZXZlbC8yMDE3LTA3L21zZzAxMDc3Lmh0bWwKWzJdOiBo dHRwczovL2xpc3RzLmdudS5vcmcvYXJjaGl2ZS9odG1sL3FlbXUtZGV2ZWwvMjAxNy0xMi9tc2cw MzU3Mi5odG1sCgpYZW4gY2hhbmdlczoKQWxleGV5IEdlcmFzaW1lbmtvICgxMik6CiAgbGliYWNw aTogbmV3IERTRFQgQUNQSSB0YWJsZSBmb3IgUTM1CiAgTWFrZWZpbGU6IGJ1aWxkIGFuZCB1c2Ug bmV3IERTRFQgdGFibGUgZm9yIFEzNQogIGh2bWxvYWRlcjogYWRkIGZ1bmN0aW9uIHRvIHF1ZXJ5 IGFuIGVtdWxhdGVkIG1hY2hpbmUgdHlwZSAoaTQ0MC9RMzUpCiAgaHZtbG9hZGVyOiBhZGQgQUNQ SSBlbmFibGluZyBmb3IgUTM1CiAgaHZtbG9hZGVyOiBhZGQgUTM1IERTRFQgdGFibGUgbG9hZGlu ZwogIGh2bWxvYWRlcjogYWRkIGJhc2ljIFEzNSBzdXBwb3J0CiAgaHZtbG9hZGVyOiBhbGxvY2F0 ZSBNTUNPTkZJRyBhcmVhIGluIHRoZSBNTUlPIGhvbGUgKyBtaW5vciBjb2RlCiAgICByZWZhY3Rv cmluZwogIGxpYnhsOiBRMzUgc3VwcG9ydCAobmV3IG9wdGlvbiBkZXZpY2VfbW9kZWxfbWFjaGlu ZSkKICBsaWJ4bDogWGVuIFBsYXRmb3JtIGRldmljZSBzdXBwb3J0IGZvciBRMzUKICBsaWJhY3Bp OiBidWlsZCBBQ1BJIE1DRkcgdGFibGUgaWYgcmVxdWVzdGVkCiAgaHZtbG9hZGVyOiB1c2UgbGli YWNwaSB0byBidWlsZCBNQ0ZHIHRhYmxlCiAgZG9jczogcHJvdmlkZSBkZXNjcmlwdGlvbiBmb3Ig ZGV2aWNlX21vZGVsX21hY2hpbmUgb3B0aW9uCgogZG9jcy9tYW4veGwuY2ZnLnBvZC41LmluICAg ICAgICAgICAgIHwgIDI3ICsrCiB0b29scy9maXJtd2FyZS9odm1sb2FkZXIvTWFrZWZpbGUgICAg fCAgIDIgKy0KIHRvb2xzL2Zpcm13YXJlL2h2bWxvYWRlci9jb25maWcuaCAgICB8ICAgNSArCiB0 b29scy9maXJtd2FyZS9odm1sb2FkZXIvaHZtbG9hZGVyLmMgfCAgMTEgKy0KIHRvb2xzL2Zpcm13 YXJlL2h2bWxvYWRlci9wY2kuYyAgICAgICB8IDI4OSArKysrKysrKysrKystLS0tLS0KIHRvb2xz L2Zpcm13YXJlL2h2bWxvYWRlci9wY2lfcmVncy5oICB8ICAgNyArCiB0b29scy9maXJtd2FyZS9o dm1sb2FkZXIvdXRpbC5jICAgICAgfCAxMzAgKysrKysrKystCiB0b29scy9maXJtd2FyZS9odm1s b2FkZXIvdXRpbC5oICAgICAgfCAgMTAgKwogdG9vbHMvbGliYWNwaS9NYWtlZmlsZSAgICAgICAg ICAgICAgIHwgICA5ICstCiB0b29scy9saWJhY3BpL2FjcGkyXzAuaCAgICAgICAgICAgICAgfCAg MjEgKysKIHRvb2xzL2xpYmFjcGkvYnVpbGQuYyAgICAgICAgICAgICAgICB8ICA0MiArKysKIHRv b2xzL2xpYmFjcGkvZHNkdF9xMzUuYXNsICAgICAgICAgICB8IDU1MSArKysrKysrKysrKysrKysr KysrKysrKysrKysrKysrKysrKwogdG9vbHMvbGliYWNwaS9saWJhY3BpLmggICAgICAgICAgICAg IHwgICA0ICsKIHRvb2xzL2xpYnhsL2xpYnhsX2RtLmMgICAgICAgICAgICAgICB8ICAyMCArLQog dG9vbHMvbGlieGwvbGlieGxfdHlwZXMuaWRsICAgICAgICAgIHwgICA3ICsKIHRvb2xzL3hsL3hs X3BhcnNlLmMgICAgICAgICAgICAgICAgICB8ICAxNCArCiAxNiBmaWxlcyBjaGFuZ2VkLCAxMDUx IGluc2VydGlvbnMoKyksIDk4IGRlbGV0aW9ucygtKQogY3JlYXRlIG1vZGUgMTAwNjQ0IHRvb2xz L2xpYmFjcGkvZHNkdF9xMzUuYXNsCgpRRU1VIGNoYW5nZXM6CkFsZXhleSBHZXJhc2ltZW5rbyAo MTgpOgogIHBjL3hlbjogWGVuIFEzNSBzdXBwb3J0OiBwcm92aWRlIElSUSBoYW5kbGluZyBmb3Ig UENJIGRldmljZXMKICBwYy9xMzU6IEFwcGx5IFBDSSBidXMgQlNFTCBwcm9wZXJ0eSBmb3IgWGVu IFBDSSBkZXZpY2UgaG90cGx1ZwogIHEzNS9hY3BpL3hlbjogUHJvdmlkZSBBQ1BJIFBDSSBob3Rw bHVnIGludGVyZmFjZSBmb3IgWGVuIG9uIFEzNQogIHEzNS94ZW46IEFkZCBYZW4gcGxhdGZvcm0g ZGV2aWNlIHN1cHBvcnQgZm9yIFEzNQogIHEzNTogRml4IGluY29ycmVjdCB2YWx1ZXMgZm9yIFBD SUVYQkFSIG1hc2tzCiAgeGVuL3B0OiBYZW5Ib3N0UENJRGV2aWNlOiBwcm92aWRlIGZ1bmN0aW9u cyBmb3IgUENJIENhcGFiaWxpdGllcyBhbmQKICAgIFBDSWUgRXh0ZW5kZWQgQ2FwYWJpbGl0aWVz IGVudW1lcmF0aW9uCiAgeGVuL3B0OiBhdm9pZCByZWFkaW5nIFBDSWUgZGV2aWNlIHR5cGUgYW5k IGNhcCB2ZXJzaW9uIG11bHRpcGxlIHRpbWVzCiAgeGVuL3B0OiBkZXRlcm1pbmUgdGhlIGxlZ2Fj eS9QQ0llIG1vZGUgZm9yIGEgcGFzc2VkIHRocm91Z2ggZGV2aWNlCiAgeGVuL3B0OiBYZW4gUENJ ZSBwYXNzdGhyb3VnaCBzdXBwb3J0IGZvciBRMzU6IGJ5cGFzcyBQQ0llIHRvcG9sb2d5CiAgICBj aGVjawogIHhlbi9wdDogYWRkIHN1cHBvcnQgZm9yIFBDSWUgRXh0ZW5kZWQgQ2FwYWJpbGl0aWVz IGFuZCBsYXJnZXIgY29uZmlnCiAgICBzcGFjZQogIHhlbi9wdDogaGFuZGxlIFBDSWUgRXh0ZW5k ZWQgQ2FwYWJpbGl0aWVzIE5leHQgcmVnaXN0ZXIKICB4ZW4vcHQ6IGFsbG93IHRvIGhpZGUgUENJ ZSBFeHRlbmRlZCBDYXBhYmlsaXRpZXMKICB4ZW4vcHQ6IGFkZCBWZW5kb3Itc3BlY2lmaWMgUENJ ZSBFeHRlbmRlZCBDYXBhYmlsaXR5IGRlc2NyaXB0b3IgYW5kCiAgICBzaXppbmcKICB4ZW4vcHQ6 IGFkZCBmaXhlZC1zaXplIFBDSWUgRXh0ZW5kZWQgQ2FwYWJpbGl0aWVzIGRlc2NyaXB0b3JzCiAg eGVuL3B0OiBhZGQgQUVSIFBDSWUgRXh0ZW5kZWQgQ2FwYWJpbGl0eSBkZXNjcmlwdG9yIGFuZCBz aXppbmcKICB4ZW4vcHQ6IGFkZCBkZXNjcmlwdG9ycyBhbmQgc2l6ZSBjYWxjdWxhdGlvbiBmb3IK ICAgIFJDTEQvQUNTL1BNVVgvRFBBL01DQVNUL1RQSC9EUEMgUENJZSBFeHRlbmRlZCBDYXBhYmls aXRpZXMKICB4ZW4vcHQ6IGFkZCBSZXNpemFibGUgQkFSIFBDSWUgRXh0ZW5kZWQgQ2FwYWJpbGl0 eSBkZXNjcmlwdG9yIGFuZAogICAgc2l6aW5nCiAgeGVuL3B0OiBhZGQgVkMvVkM5L01GVkMgUENJ ZSBFeHRlbmRlZCBDYXBhYmlsaXRpZXMgZGVzY3JpcHRvcnMgYW5kCiAgICBzaXppbmcKCiBody9h Y3BpL2ljaDkuYyAgICAgICAgICAgICAgIHwgICAyNCArCiBody9hY3BpL3BjaWhwLmMgICAgICAg ICAgICAgIHwgICAgOCArLQogaHcvY29yZS9tYWNoaW5lLmMgICAgICAgICAgICB8ICAgMjEgKwog aHcvaTM4Ni9wY19xMzUuYyAgICAgICAgICAgICB8ICAgMjcgKy0KIGh3L2kzODYveGVuL3hlbi1o dm0uYyAgICAgICAgfCAgIDMyICstCiBody9pc2EvbHBjX2ljaDkuYyAgICAgICAgICAgIHwgICAg NCArCiBody9wY2ktaG9zdC9waWl4LmMgICAgICAgICAgIHwgICAgMiArLQogaHcvcGNpLWhvc3Qv cTM1LmMgICAgICAgICAgICB8ICAgMTQgKy0KIGh3L3hlbi94ZW4taG9zdC1wY2ktZGV2aWNlLmMg fCAgMTEwICsrKystCiBody94ZW4veGVuLWhvc3QtcGNpLWRldmljZS5oIHwgICAgNiArLQogaHcv eGVuL3hlbl9wdC5jICAgICAgICAgICAgICB8ICAgNTMgKy0KIGh3L3hlbi94ZW5fcHQuaCAgICAg ICAgICAgICAgfCAgIDE5ICstCiBody94ZW4veGVuX3B0X2NvbmZpZ19pbml0LmMgIHwgMTEwOSAr KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKystLS0KIGluY2x1ZGUvaHcvYWNw aS9pY2g5LmggICAgICAgfCAgICAyICsKIGluY2x1ZGUvaHcvYWNwaS9wY2locC5oICAgICAgfCAg ICAyICsKIGluY2x1ZGUvaHcvYm9hcmRzLmggICAgICAgICAgfCAgICAxICsKIGluY2x1ZGUvaHcv aTM4Ni9pY2g5LmggICAgICAgfCAgICAxICsKIGluY2x1ZGUvaHcvaTM4Ni9wYy5oICAgICAgICAg fCAgICAzICsKIGluY2x1ZGUvaHcvcGNpLWhvc3QvcTM1LmggICAgfCAgICA0ICstCiBpbmNsdWRl L2h3L3hlbi94ZW4uaCAgICAgICAgIHwgICAgNSArLQogcWVtdS1vcHRpb25zLmh4ICAgICAgICAg ICAgICB8ICAgIDEgKwogc3R1YnMveGVuLWh2bS5jICAgICAgICAgICAgICB8ICAgIDggKy0KIDIy IGZpbGVzIGNoYW5nZWQsIDEzMzMgaW5zZXJ0aW9ucygrKSwgMTIzIGRlbGV0aW9ucygtKQoKLS0g CjIuMTEuMAoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f Clhlbi1kZXZlbCBtYWlsaW5nIGxpc3QKWGVuLWRldmVsQGxpc3RzLnhlbnByb2plY3Qub3JnCmh0 dHBzOi8vbGlzdHMueGVucHJvamVjdC5vcmcvbWFpbG1hbi9saXN0aW5mby94ZW4tZGV2ZWw= From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44355) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evSHI-00077k-0r for qemu-devel@nongnu.org; Mon, 12 Mar 2018 14:34:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evSHE-0004pv-RU for qemu-devel@nongnu.org; Mon, 12 Mar 2018 14:34:32 -0400 Received: from mail-pl0-x233.google.com ([2607:f8b0:400e:c01::233]:37955) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1evSHE-0004pf-IH for qemu-devel@nongnu.org; Mon, 12 Mar 2018 14:34:28 -0400 Received: by mail-pl0-x233.google.com with SMTP id m22-v6so9833576pls.5 for ; Mon, 12 Mar 2018 11:34:28 -0700 (PDT) From: Alexey Gerasimenko Date: Tue, 13 Mar 2018 04:33:45 +1000 Message-Id: Subject: [Qemu-devel] [RFC PATCH 00/30] Xen Q35 Bringup patches + support for PCIe Extended Capabilities for passed through devices List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: xen-devel@lists.xenproject.org Cc: Alexey Gerasimenko , qemu-devel@nongnu.org This patch series introduces support of Q35 emulation for Xen HVM guests (via QEMU). This feature is present in other virtualization products and Xen can greatly benefit from this feature as well. The main goal for implementing Q35 emulation for Xen was extending PCI/GPU passthrough capabilities. It's the main advantage of Q35 emulation - availability of extra features for PCIe device passthrough. The most important PCIe-specific passthrough feature Q35 provides is a support for PCIe config space ECAM (aka MMCONFIG) to allow accesses to extended PCIe config space (>256), which is MMIO-based. Lots of PCIe devices and their drivers make use of PCIe Extended Capabilities, whose can be accessed only using ECAM and offsets above 0x100 in PCI config space. Supporting ECAM is a mandatory feature for PCIe passthrough. Not only this allows passthrough PCIe devices to function properly, but opens a road to extend Xen PCIe passthrough features further -- eg. providing support for AER. One of possible directions is providing support for PCIe Resizable BARs -- a feature which likely to become common for modern GPUs as video memory sizes increase. Q35 emulation may also be useful for other purposes. In fact, the emulation of a more recent chipset partially closes a huge gap between a set of required platform features and the actual emulated platform capabilities - lot of required functionality is actually missing in a real i440 chipset. One can look at IGD passthru support patches from Intel for example: according to code comments, they had to create a dummy PCI-ISA bridge at BDF 0:1F.0 in order to make the old i440 system look more modern, just to make it compatible with IGD driver. Using Q35 emulation with its own emulated LPC bridge allows to avoid workarounds like this. i440 on its own is a fairly outdated system and doesn't really support lot of things, like MMIO hole above 4Gb (although it is actually emulated). Also, due to the i440 chipset's age the only fact of its usage may be used as a reliable method to detect a virtualized environment by some malicious software especially considering the fact that i440 emulation is shared among multiple virtualization products. On top of this series I've also implemented a solution which solves existing Xen puzzle with HVM memory layout -- handling of VRAM, RMRRs and MMIO hole in general. This "puzzle" (memory layout inconsistency between libxl/libxc, hvmloader and QEMU) is a sort of fundamental problem which plagues Xen for years and among few other issues prevents Xen to become a decent GPU/PCIe passthrough platform (which it should be). This solution also allows to later resolve current PCI passthrough incompatibility issues, eg. with Populate-on-Demand. In fact, i440 support has been added as well, but it's a bit hacky as it uses NB registers which are not present in a real i440 (well, one more non-existing i440 feature won't harm anyway as there are plenty of them already). I'm planning to send RFC patches of this solution right after current patches will be reviewed and related code settle, to rebase patches on top of it. Also, a good description is required as the change is rather radical. The good thing is that providing Q35 support for Xen at this stage neither break any existing functionality nor affect the legacy i440 emulation in any way - Q35 emulation can be enabled on demand only, using a new domain config option. Also, only existing interfaces are used, no new hypecalls were introduced, no API changes, etc. Although in the future we'll have to change some hypercall/QMP/etc interfaces to remove limitations and extend the Q35/PCIe passthru support further. Current features and limitations: - All basic functionality works normally - MP, networking, storage (AHCI), powering down VMs via ACPI soft off, etc - Xen Platform Device and PV devices are supported -- PV drivers for vbd, vif, etc may be installed and used - PCIe ECAM fully supported, with allocating space for PCIEXBAR in MMIO hole, ACPI MCFG generation, etc. - Xen is limited to max 4 PIRQs in multiple places, while Q35 have support of 8 PIRQs / PCI router links. This was workarounded by describing only 4 usable IRQ link entries in ACPI tables and disabling PIRQE..PIRQH -- like we're on a real system which has only some of 8 available PIRQs physically connected on the chipset. Extending the number of PCI links supported is trivial, but this step will change the save/migration stream format a bit... although as it seems there was actually some place for this extension being left -- eg. field uint8_t route[4] followed by uint8_t pad0[4] in hvm_hw_pci_link structure. Anyway, there is no problem actually as we normally deal with APIC mode (or MSIs) for IRQ delivery, while PIC mode with PCI routing needed only for legacy compatibility - PCI hotplug currently implemented via ACPI hotplug, in a way similar to i440. In future, this might be changed to native PCIe hotplug facilities (if there will be a benefit). - For PCIe passthrough to work on Windows 7 and above, a specific workaround was implemented, which allows to use PCIe device passthrough on those guest OSes normally. In future, this should be changed to a new emulated PCI architecture for Xen -- providing support for simple PCI hierarchies, nested MMIO spaces, etc. Basically, we need at least to provide support for PCI-PCI bridges (PCIe Root Ports in our case). Currently Xen limited to bus 0 in many places, even in hypercall parameters. A detailed description of the issue can be found in the patch named "xen/pt: Xen PCIe passthrough support for Q35: bypass PCIe topology check". - VM migration was not tested as the feature primarily targets the PCIe passthrough which doesn't compatible with migration anyway. How to use the Q35 feature: A new domain config option was implemented: device_model_machine. It's a string which has following possible values: - "i440" -- i440 emulation (default) - "q35" -- emulate a Q35 machine. By default, the storage interface is AHCI. Note that omitting device_model_machine parameter means i440 system by default, so the default behavior doesn't change for old domain config files. So, in order to enable Q35 emulation one need to specify the following option in the domain config file: device_model_machine="q35" It is recommended to install the guest OS from scratch to avoid issues due to the emulated platform change. One extra note - if you're going to backport this series to some older QEMU version, make sure you have this patch for AHCI DMA bug applied: [1]. Otherwise you will encounter random Q35 guest hangups with "Bad RAM offset" message logged in /var/log/xen. Recent QEMU versions have this patch commited already. Also, a commit [2] is required to be applied (for xen-pt.c) -- it is available in the upstream QEMU currently, but not present in qemu-xen. This is my first (somewhat) large contribution to Xen, so some mistakes are to be expected. Most testing was done using previous version of patches and Xen 4.8.x. I plan to support and extend this series further, for now I expect some comments/suggestions/testing results/bugreports. [1]: https://lists.xen.org/archives/html/xen-devel/2017-07/msg01077.html [2]: https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03572.html Xen changes: Alexey Gerasimenko (12): libacpi: new DSDT ACPI table for Q35 Makefile: build and use new DSDT table for Q35 hvmloader: add function to query an emulated machine type (i440/Q35) hvmloader: add ACPI enabling for Q35 hvmloader: add Q35 DSDT table loading hvmloader: add basic Q35 support hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring libxl: Q35 support (new option device_model_machine) libxl: Xen Platform device support for Q35 libacpi: build ACPI MCFG table if requested hvmloader: use libacpi to build MCFG table docs: provide description for device_model_machine option docs/man/xl.cfg.pod.5.in | 27 ++ tools/firmware/hvmloader/Makefile | 2 +- tools/firmware/hvmloader/config.h | 5 + tools/firmware/hvmloader/hvmloader.c | 11 +- tools/firmware/hvmloader/pci.c | 289 ++++++++++++------ tools/firmware/hvmloader/pci_regs.h | 7 + tools/firmware/hvmloader/util.c | 130 ++++++++- tools/firmware/hvmloader/util.h | 10 + tools/libacpi/Makefile | 9 +- tools/libacpi/acpi2_0.h | 21 ++ tools/libacpi/build.c | 42 +++ tools/libacpi/dsdt_q35.asl | 551 +++++++++++++++++++++++++++++++++++ tools/libacpi/libacpi.h | 4 + tools/libxl/libxl_dm.c | 20 +- tools/libxl/libxl_types.idl | 7 + tools/xl/xl_parse.c | 14 + 16 files changed, 1051 insertions(+), 98 deletions(-) create mode 100644 tools/libacpi/dsdt_q35.asl QEMU changes: Alexey Gerasimenko (18): pc/xen: Xen Q35 support: provide IRQ handling for PCI devices pc/q35: Apply PCI bus BSEL property for Xen PCI device hotplug q35/acpi/xen: Provide ACPI PCI hotplug interface for Xen on Q35 q35/xen: Add Xen platform device support for Q35 q35: Fix incorrect values for PCIEXBAR masks xen/pt: XenHostPCIDevice: provide functions for PCI Capabilities and PCIe Extended Capabilities enumeration xen/pt: avoid reading PCIe device type and cap version multiple times xen/pt: determine the legacy/PCIe mode for a passed through device xen/pt: Xen PCIe passthrough support for Q35: bypass PCIe topology check xen/pt: add support for PCIe Extended Capabilities and larger config space xen/pt: handle PCIe Extended Capabilities Next register xen/pt: allow to hide PCIe Extended Capabilities xen/pt: add Vendor-specific PCIe Extended Capability descriptor and sizing xen/pt: add fixed-size PCIe Extended Capabilities descriptors xen/pt: add AER PCIe Extended Capability descriptor and sizing xen/pt: add descriptors and size calculation for RCLD/ACS/PMUX/DPA/MCAST/TPH/DPC PCIe Extended Capabilities xen/pt: add Resizable BAR PCIe Extended Capability descriptor and sizing xen/pt: add VC/VC9/MFVC PCIe Extended Capabilities descriptors and sizing hw/acpi/ich9.c | 24 + hw/acpi/pcihp.c | 8 +- hw/core/machine.c | 21 + hw/i386/pc_q35.c | 27 +- hw/i386/xen/xen-hvm.c | 32 +- hw/isa/lpc_ich9.c | 4 + hw/pci-host/piix.c | 2 +- hw/pci-host/q35.c | 14 +- hw/xen/xen-host-pci-device.c | 110 ++++- hw/xen/xen-host-pci-device.h | 6 +- hw/xen/xen_pt.c | 53 +- hw/xen/xen_pt.h | 19 +- hw/xen/xen_pt_config_init.c | 1109 +++++++++++++++++++++++++++++++++++++++--- include/hw/acpi/ich9.h | 2 + include/hw/acpi/pcihp.h | 2 + include/hw/boards.h | 1 + include/hw/i386/ich9.h | 1 + include/hw/i386/pc.h | 3 + include/hw/pci-host/q35.h | 4 +- include/hw/xen/xen.h | 5 +- qemu-options.hx | 1 + stubs/xen-hvm.c | 8 +- 22 files changed, 1333 insertions(+), 123 deletions(-) -- 2.11.0