From mboxrd@z Thu Jan 1 00:00:00 1970 From: Auger Eric Subject: Re: [PATCH v6 09/22] vfio: VFIO_IOMMU_BIND/UNBIND_MSI Date: Wed, 10 Apr 2019 15:02:23 +0200 Message-ID: <2cdd4142-98e5-14de-2f34-264244f24d01@redhat.com> References: <20190317172232.1068-1-eric.auger@redhat.com> <20190317172232.1068-10-eric.auger@redhat.com> <20190321170159.38358f38@x1.home> <16931d58-9c88-8cfb-a392-408ea7afdf16@redhat.com> <20190322160947.3f8dacdb@x1.home> <20190403113800.51503693@x1.home> <4b27b707-4917-2c3e-a13c-3b978e850d09@redhat.com> <20190410123531.GA19023@debian> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Return-path: In-Reply-To: <20190410123531.GA19023@debian> Content-Language: en-US Sender: linux-kernel-owner@vger.kernel.org To: =?UTF-8?Q?Vincent_Stehl=c3=a9?= Cc: Alex Williamson , eric.auger.pro@gmail.com, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, joro@8bytes.org, jacob.jun.pan@linux.intel.com, yi.l.liu@linux.intel.com, jean-philippe.brucker@arm.com, will.deacon@arm.com, robin.murphy@arm.com, kevin.tian@intel.com, ashok.raj@intel.com, marc.zyngier@arm.com, christoffer.dall@arm.com, peter.maydell@linaro.org List-Id: iommu@lists.linux-foundation.org Hi Vincent, On 4/10/19 2:35 PM, Vincent Stehlé wrote: > On Thu, Apr 04, 2019 at 08:55:25AM +0200, Auger Eric wrote: >> Hi Marc, Robin, Alex, > (..) >> Do you think this is a reasonable assumption to consider devices within >> the same host iommu group share the same MSI doorbell? > > Hi Eric, > > I am not sure this assumption always hold. > > Marc, Robin and Alex can correct me, but for example I think the following > topology is valid for Arm systems: > > +------------+ +------------+ > | Endpoint A | | Endpoint B | > +------------+ +------------+ > v v > /---------\ > | Non-ACS | > | Switch | > \---------/ > v > +---------------+ > | PCIe | > | Root Complex | > +---------------+ > v > +-----------+ > | SMMU | > +-----------+ > v > +--------------------------+ > | System interconnect | > +--------------------------+ > v v > +-----------+ +-----------+ > | ITS A | | ITS B | > +-----------+ +-----------+ > > All PCIe Endpoints and ITS could be in the same ITS Group 0, meaning > devices could send their MSI at any ITS in hardware. > > For Linux the two PCIe Endpoints would be in the same iommu group, because > the switch in this example does not support ACS. > > I think the devicetree msi-map property could be used to "map" the RID of > Endpoint A to ITS A and the RID of Endpoint B to ITS B, which would violate > the assumption. > > See the monolithic example in [1], the example system in [2], appendices > D, E and F in [3] and the msi-map property in [4]. Thank you for the review & links. I understand the above topology is perfectly valid. Now the question is: is it sufficiently common to care about it? At the moment VFIO/vIOMMU assignment of devices belonging to the same group isn't upstream yet. Work is ongoing by Alex to support it. It uses a PCIe-to-PCI bridge on guest side and it looks this topology is not supported by the SMMUv3 driver. Then comes the trouble of using several ITS in nested mode. If this topology is sufficiently rare I propose we to do not support it in this VFIO/vIOMMU use case. in v7 I introduced a check that aims to verify devices attached to the same nested iommu_domain share the same msi_domain. Thanks Eric > > Best regards, > Vincent. > > [1] https://static.docs.arm.com/100336/0102/corelink_gic600_generic_interrupt_controller_technical_reference_manual_100336_0102_00_en.pdf > [2] http://infocenter.arm.com/help/topic/com.arm.doc.den0049d/DEN0049D_IO_Remapping_Table.pdf > [3] https://static.docs.arm.com/den0029/50/Q1-DEN0029B_SBSA_5.0.pdf > [4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/pci-msi.txt > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60A05C10F14 for ; Wed, 10 Apr 2019 13:02:53 +0000 (UTC) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3814B2083E for ; Wed, 10 Apr 2019 13:02:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3814B2083E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id E2C4A1807; Wed, 10 Apr 2019 13:02:52 +0000 (UTC) Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 6B8D914F2 for ; Wed, 10 Apr 2019 13:02:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 0139B17E for ; Wed, 10 Apr 2019 13:02:38 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3AF973D3A; Wed, 10 Apr 2019 13:02:38 +0000 (UTC) Received: from [10.36.117.161] (ovpn-117-161.ams2.redhat.com [10.36.117.161]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D05FF619A2; Wed, 10 Apr 2019 13:02:26 +0000 (UTC) Subject: Re: [PATCH v6 09/22] vfio: VFIO_IOMMU_BIND/UNBIND_MSI To: =?UTF-8?Q?Vincent_Stehl=c3=a9?= References: <20190317172232.1068-1-eric.auger@redhat.com> <20190317172232.1068-10-eric.auger@redhat.com> <20190321170159.38358f38@x1.home> <16931d58-9c88-8cfb-a392-408ea7afdf16@redhat.com> <20190322160947.3f8dacdb@x1.home> <20190403113800.51503693@x1.home> <4b27b707-4917-2c3e-a13c-3b978e850d09@redhat.com> <20190410123531.GA19023@debian> From: Auger Eric Message-ID: <2cdd4142-98e5-14de-2f34-264244f24d01@redhat.com> Date: Wed, 10 Apr 2019 15:02:23 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <20190410123531.GA19023@debian> Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Wed, 10 Apr 2019 13:02:38 +0000 (UTC) Cc: yi.l.liu@linux.intel.com, kevin.tian@intel.com, ashok.raj@intel.com, kvm@vger.kernel.org, peter.maydell@linaro.org, jean-philippe.brucker@arm.com, will.deacon@arm.com, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, christoffer.dall@arm.com, marc.zyngier@arm.com, Alex Williamson , robin.murphy@arm.com, kvmarm@lists.cs.columbia.edu, eric.auger.pro@gmail.com X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 Sender: iommu-bounces@lists.linux-foundation.org Errors-To: iommu-bounces@lists.linux-foundation.org Message-ID: <20190410130223.V50TP0QjkghaDmUHMHtTHfr7aWfYCFCMjWaVBis9XS0@z> SGkgVmluY2VudCwKCk9uIDQvMTAvMTkgMjozNSBQTSwgVmluY2VudCBTdGVobMOpIHdyb3RlOgo+ IE9uIFRodSwgQXByIDA0LCAyMDE5IGF0IDA4OjU1OjI1QU0gKzAyMDAsIEF1Z2VyIEVyaWMgd3Jv dGU6Cj4+IEhpIE1hcmMsIFJvYmluLCBBbGV4LAo+ICguLikKPj4gRG8geW91IHRoaW5rIHRoaXMg aXMgYSByZWFzb25hYmxlIGFzc3VtcHRpb24gdG8gY29uc2lkZXIgZGV2aWNlcyB3aXRoaW4KPj4g dGhlIHNhbWUgaG9zdCBpb21tdSBncm91cCBzaGFyZSB0aGUgc2FtZSBNU0kgZG9vcmJlbGw/Cj4g Cj4gSGkgRXJpYywKPiAKPiBJIGFtIG5vdCBzdXJlIHRoaXMgYXNzdW1wdGlvbiBhbHdheXMgaG9s ZC4KPiAKPiBNYXJjLCBSb2JpbiBhbmQgQWxleCBjYW4gY29ycmVjdCBtZSwgYnV0IGZvciBleGFt cGxlIEkgdGhpbmsgdGhlIGZvbGxvd2luZwo+IHRvcG9sb2d5IGlzIHZhbGlkIGZvciBBcm0gc3lz dGVtczoKPiAKPiAgKy0tLS0tLS0tLS0tLSsgICstLS0tLS0tLS0tLS0rCj4gIHwgRW5kcG9pbnQg QSB8ICB8IEVuZHBvaW50IEIgfAo+ICArLS0tLS0tLS0tLS0tKyAgKy0tLS0tLS0tLS0tLSsKPiAg ICAgICAgICAgICB2ICAgICB2Cj4gICAgICAgICAgIC8tLS0tLS0tLS1cCj4gICAgICAgICAgfCAg Tm9uLUFDUyAgfAo+ICAgICAgICAgIHwgIFN3aXRjaCAgIHwKPiAgICAgICAgICAgXC0tLS0tLS0t LS8KPiAgICAgICAgICAgICAgICB2Cj4gICAgICAgICstLS0tLS0tLS0tLS0tLS0rCj4gICAgICAg IHwgICAgIFBDSWUgICAgICB8Cj4gICAgICAgIHwgUm9vdCBDb21wbGV4ICB8Cj4gICAgICAgICst LS0tLS0tLS0tLS0tLS0rCj4gICAgICAgICAgICAgICAgdgo+ICAgICAgICAgICstLS0tLS0tLS0t LSsKPiAgICAgICAgICB8ICAgU01NVSAgICB8Cj4gICAgICAgICAgKy0tLS0tLS0tLS0tKwo+ICAg ICAgICAgICAgICAgIHYKPiAgICstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsKPiAgIHwgICBT eXN0ZW0gaW50ZXJjb25uZWN0ICAgIHwKPiAgICstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsK PiAgICAgICAgIHYgICAgICAgICAgICAgIHYKPiAgICstLS0tLS0tLS0tLSsgICstLS0tLS0tLS0t LSsKPiAgIHwgICBJVFMgQSAgIHwgIHwgICBJVFMgQiAgIHwKPiAgICstLS0tLS0tLS0tLSsgICst LS0tLS0tLS0tLSsKPiAKPiBBbGwgUENJZSBFbmRwb2ludHMgYW5kIElUUyBjb3VsZCBiZSBpbiB0 aGUgc2FtZSBJVFMgR3JvdXAgMCwgbWVhbmluZwo+IGRldmljZXMgY291bGQgc2VuZCB0aGVpciBN U0kgYXQgYW55IElUUyBpbiBoYXJkd2FyZS4KPiAKPiBGb3IgTGludXggdGhlIHR3byBQQ0llIEVu ZHBvaW50cyB3b3VsZCBiZSBpbiB0aGUgc2FtZSBpb21tdSBncm91cCwgYmVjYXVzZQo+IHRoZSBz d2l0Y2ggaW4gdGhpcyBleGFtcGxlIGRvZXMgbm90IHN1cHBvcnQgQUNTLgo+IAo+IEkgdGhpbmsg dGhlIGRldmljZXRyZWUgbXNpLW1hcCBwcm9wZXJ0eSBjb3VsZCBiZSB1c2VkIHRvICJtYXAiIHRo ZSBSSUQgb2YKPiBFbmRwb2ludCBBIHRvIElUUyBBIGFuZCB0aGUgUklEIG9mIEVuZHBvaW50IEIg dG8gSVRTIEIsIHdoaWNoIHdvdWxkIHZpb2xhdGUKPiB0aGUgYXNzdW1wdGlvbi4KPiAKPiBTZWUg dGhlIG1vbm9saXRoaWMgZXhhbXBsZSBpbiBbMV0sIHRoZSBleGFtcGxlIHN5c3RlbSBpbiBbMl0s IGFwcGVuZGljZXMKPiBELCBFIGFuZCBGIGluIFszXSBhbmQgdGhlIG1zaS1tYXAgcHJvcGVydHkg aW4gWzRdLgoKVGhhbmsgeW91IGZvciB0aGUgcmV2aWV3ICYgbGlua3MuCgpJIHVuZGVyc3RhbmQg dGhlIGFib3ZlIHRvcG9sb2d5IGlzIHBlcmZlY3RseSB2YWxpZC4gTm93IHRoZSBxdWVzdGlvbiBp czoKaXMgaXQgc3VmZmljaWVudGx5IGNvbW1vbiB0byBjYXJlIGFib3V0IGl0PwoKQXQgdGhlIG1v bWVudCBWRklPL3ZJT01NVSBhc3NpZ25tZW50IG9mIGRldmljZXMgYmVsb25naW5nIHRvIHRoZSBz YW1lCmdyb3VwIGlzbid0IHVwc3RyZWFtIHlldC4gV29yayBpcyBvbmdvaW5nIGJ5IEFsZXggdG8g c3VwcG9ydCBpdC4gSXQgdXNlcwphIFBDSWUtdG8tUENJIGJyaWRnZSBvbiBndWVzdCBzaWRlIGFu ZCBpdCBsb29rcyB0aGlzIHRvcG9sb2d5IGlzIG5vdApzdXBwb3J0ZWQgYnkgdGhlIFNNTVV2MyBk cml2ZXIuIFRoZW4gY29tZXMgdGhlIHRyb3VibGUgb2YgdXNpbmcgc2V2ZXJhbApJVFMgaW4gbmVz dGVkIG1vZGUuCgpJZiB0aGlzIHRvcG9sb2d5IGlzIHN1ZmZpY2llbnRseSByYXJlIEkgcHJvcG9z ZSB3ZSB0byBkbyBub3Qgc3VwcG9ydCBpdAppbiB0aGlzIFZGSU8vdklPTU1VIHVzZSBjYXNlLiBp biB2NyBJIGludHJvZHVjZWQgYSBjaGVjayB0aGF0IGFpbXMgdG8KdmVyaWZ5IGRldmljZXMgYXR0 YWNoZWQgdG8gdGhlIHNhbWUgbmVzdGVkIGlvbW11X2RvbWFpbiBzaGFyZSB0aGUgc2FtZQptc2lf ZG9tYWluLgoKVGhhbmtzCgpFcmljCj4gCj4gQmVzdCByZWdhcmRzLAo+IFZpbmNlbnQuCj4gCj4g WzFdIGh0dHBzOi8vc3RhdGljLmRvY3MuYXJtLmNvbS8xMDAzMzYvMDEwMi9jb3JlbGlua19naWM2 MDBfZ2VuZXJpY19pbnRlcnJ1cHRfY29udHJvbGxlcl90ZWNobmljYWxfcmVmZXJlbmNlX21hbnVh bF8xMDAzMzZfMDEwMl8wMF9lbi5wZGYKPiBbMl0gaHR0cDovL2luZm9jZW50ZXIuYXJtLmNvbS9o ZWxwL3RvcGljL2NvbS5hcm0uZG9jLmRlbjAwNDlkL0RFTjAwNDlEX0lPX1JlbWFwcGluZ19UYWJs ZS5wZGYKPiBbM10gaHR0cHM6Ly9zdGF0aWMuZG9jcy5hcm0uY29tL2RlbjAwMjkvNTAvUTEtREVO MDAyOUJfU0JTQV81LjAucGRmCj4gWzRdIGh0dHBzOi8vZ2l0Lmtlcm5lbC5vcmcvcHViL3NjbS9s aW51eC9rZXJuZWwvZ2l0L3RvcnZhbGRzL2xpbnV4LmdpdC90cmVlL0RvY3VtZW50YXRpb24vZGV2 aWNldHJlZS9iaW5kaW5ncy9wY2kvcGNpLW1zaS50eHQKPiAKX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KaW9tbXUgbWFpbGluZyBsaXN0CmlvbW11QGxpc3Rz LmxpbnV4LWZvdW5kYXRpb24ub3JnCmh0dHBzOi8vbGlzdHMubGludXhmb3VuZGF0aW9uLm9yZy9t YWlsbWFuL2xpc3RpbmZvL2lvbW11