From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from e23smtp02.au.ibm.com ([202.81.31.144]:40916 "EHLO e23smtp02.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751199AbcCRLwM (ORCPT ); Fri, 18 Mar 2016 07:52:12 -0400 Received: from localhost by e23smtp02.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 18 Mar 2016 21:52:09 +1000 Subject: Re: [RFC PATCH v4 7/7] powerpc/powernv/pci-ioda: Add IOMMU_CAP_INTR_REMAP for IODA host bridge To: Alex Williamson References: <1457336918-3893-1-git-send-email-xyjxie@linux.vnet.ibm.com> <1457336918-3893-8-git-send-email-xyjxie@linux.vnet.ibm.com> <20160316103207.6fffb9b5@t450s.home> <56EA9735.2010401@linux.vnet.ibm.com> <20160317064842.764d8f22@ul30vt.home> Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, bhelgaas@google.com, corbet@lwn.net, aik@ozlabs.ru, benh@kernel.crashing.org, paulus@samba.org, mpe@ellerman.id.au, warrier@linux.vnet.ibm.com, zhong@linux.vnet.ibm.com, nikunj@linux.vnet.ibm.com From: Yongji Xie Message-ID: <56EBEBB6.5030800@linux.vnet.ibm.com> Date: Fri, 18 Mar 2016 19:51:18 +0800 MIME-Version: 1.0 In-Reply-To: <20160317064842.764d8f22@ul30vt.home> Content-Type: text/plain; charset=utf-8; format=flowed Sender: linux-pci-owner@vger.kernel.org List-ID: On 2016/3/17 20:48, Alex Williamson wrote: > On Thu, 17 Mar 2016 19:38:29 +0800 > Yongji Xie wrote: > >> On 2016/3/17 0:32, Alex Williamson wrote: >>> On Mon, 7 Mar 2016 15:48:38 +0800 >>> Yongji Xie wrote: >>> >>>> This patch adds IOMMU_CAP_INTR_REMAP for IODA host bridge so that >>>> we can mmap MSI-X table in vfio driver. >>>> >>>> Signed-off-by: Yongji Xie >>>> --- >>>> arch/powerpc/platforms/powernv/pci-ioda.c | 17 +++++++++++++++++ >>>> 1 file changed, 17 insertions(+) >>>> >>>> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c >>>> index f90dc04..f01b9ab 100644 >>>> --- a/arch/powerpc/platforms/powernv/pci-ioda.c >>>> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c >>>> @@ -1955,6 +1955,20 @@ static struct iommu_table_ops pnv_ioda2_iommu_ops = { >>>> .free = pnv_ioda2_table_free, >>>> }; >>>> >>>> +static bool pnv_ioda_iommu_capable(enum iommu_cap cap) >>>> +{ >>>> + switch (cap) { >>>> + case IOMMU_CAP_INTR_REMAP: >>>> + return true; >>>> + default: >>>> + return false; >>>> + } >>>> +} >>>> + >>>> +static struct iommu_ops pnv_ioda_iommu_ops = { >>>> + .capable = pnv_ioda_iommu_capable, >>>> +}; >>>> + >>>> static void pnv_pci_ioda_setup_dma_pe(struct pnv_phb *phb, >>>> struct pnv_ioda_pe *pe, unsigned int base, >>>> unsigned int segs) >>>> @@ -3078,6 +3092,9 @@ static void pnv_pci_ioda_fixup(void) >>>> >>>> /* Link NPU IODA tables to their PCI devices. */ >>>> pnv_npu_ioda_fixup(); >>>> + >>>> + /* Add IOMMU_CAP_INTR_REMAP */ >>>> + bus_set_iommu(&pci_bus_type, &pnv_ioda_iommu_ops); >>>> } >>>> >>>> /* >>> Doesn't this set you up for a world of hurt? bus_set_iommu() calls >>> iommu_bus_init() which sets up notifiers, which maybe you don't care >>> about, but it also means that iommu_domain_alloc(&pci_bus_type) will >>> segfault because you're not providing a domain_alloc callback here. >> It seems to be hard to add IOMMU_CAP_INTR_REMAP on >> PPC64 platform. >> >> And can we add a new ioctl in vfio_iommu_driver to check >> if interrupt remapping is supported so that we can use our >> own way to determine that on PPC64 platform? > I'd prefer not. At the vfio user API level, the question is whether > the user can mmap over the msix table, testing a property/ioctl on the > iommu driver seems like an odd way to discover that. We should be > determining whether that's safe in the kernel and exporting that info > on the vfio device itself, where it seems like we have various ways we > could do this within the existing ioctls. Thanks, > > Alex > Yes, you are right. It's not a good idea to add a new ioctl in vfio_iommu_driver. Now I'd like to talk about the way to determining whether it's safe to mmap over the msix table. We currently use IOMMU_CAP_INTR_REMAP to determine that. But there are some problems on PPC64 which never set iommu_ops and ARM SMMU which set this capability but not provide interrupt isolation. Can we add a variable/property which can be set in vfio_iommu_driver->ops->attach_group() and used in vfio_pci_driver to determine whether we can allow mmapping msix table? If so, we can still use IOMMU_CAP_INTR_REMAP, or some arch-independent ways when IOMMU_CAP_INTR_REMAP doesn't work. Thanks, Yongji Xie From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yongji Xie Subject: Re: [RFC PATCH v4 7/7] powerpc/powernv/pci-ioda: Add IOMMU_CAP_INTR_REMAP for IODA host bridge Date: Fri, 18 Mar 2016 19:51:18 +0800 Message-ID: <56EBEBB6.5030800@linux.vnet.ibm.com> References: <1457336918-3893-1-git-send-email-xyjxie@linux.vnet.ibm.com> <1457336918-3893-8-git-send-email-xyjxie@linux.vnet.ibm.com> <20160316103207.6fffb9b5@t450s.home> <56EA9735.2010401@linux.vnet.ibm.com> <20160317064842.764d8f22@ul30vt.home> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Cc: nikunj@linux.vnet.ibm.com, zhong@linux.vnet.ibm.com, kvm@vger.kernel.org, linux-doc@vger.kernel.org, aik@ozlabs.ru, linux-pci@vger.kernel.org, corbet@lwn.net, linux-kernel@vger.kernel.org, warrier@linux.vnet.ibm.com, paulus@samba.org, bhelgaas@google.com, linuxppc-dev@lists.ozlabs.org To: Alex Williamson Return-path: In-Reply-To: <20160317064842.764d8f22@ul30vt.home> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+glppe-linuxppc-embedded-2=m.gmane.org@lists.ozlabs.org Sender: "Linuxppc-dev" List-Id: kvm.vger.kernel.org T24gMjAxNi8zLzE3IDIwOjQ4LCBBbGV4IFdpbGxpYW1zb24gd3JvdGU6Cj4gT24gVGh1LCAxNyBN YXIgMjAxNiAxOTozODoyOSArMDgwMAo+IFlvbmdqaSBYaWUgPHh5anhpZUBsaW51eC52bmV0Lmli bS5jb20+IHdyb3RlOgo+Cj4+IE9uIDIwMTYvMy8xNyAwOjMyLCBBbGV4IFdpbGxpYW1zb24gd3Jv dGU6Cj4+PiBPbiBNb24sICA3IE1hciAyMDE2IDE1OjQ4OjM4ICswODAwCj4+PiBZb25namkgWGll IDx4eWp4aWVAbGludXgudm5ldC5pYm0uY29tPiB3cm90ZToKPj4+ICAgCj4+Pj4gVGhpcyBwYXRj aCBhZGRzIElPTU1VX0NBUF9JTlRSX1JFTUFQIGZvciBJT0RBIGhvc3QgYnJpZGdlIHNvIHRoYXQK Pj4+PiB3ZSBjYW4gbW1hcCBNU0ktWCB0YWJsZSBpbiB2ZmlvIGRyaXZlci4KPj4+Pgo+Pj4+IFNp Z25lZC1vZmYtYnk6IFlvbmdqaSBYaWUgPHh5anhpZUBsaW51eC52bmV0LmlibS5jb20+Cj4+Pj4g LS0tCj4+Pj4gICAgYXJjaC9wb3dlcnBjL3BsYXRmb3Jtcy9wb3dlcm52L3BjaS1pb2RhLmMgfCAg IDE3ICsrKysrKysrKysrKysrKysrCj4+Pj4gICAgMSBmaWxlIGNoYW5nZWQsIDE3IGluc2VydGlv bnMoKykKPj4+Pgo+Pj4+IGRpZmYgLS1naXQgYS9hcmNoL3Bvd2VycGMvcGxhdGZvcm1zL3Bvd2Vy bnYvcGNpLWlvZGEuYyBiL2FyY2gvcG93ZXJwYy9wbGF0Zm9ybXMvcG93ZXJudi9wY2ktaW9kYS5j Cj4+Pj4gaW5kZXggZjkwZGMwNC4uZjAxYjlhYiAxMDA2NDQKPj4+PiAtLS0gYS9hcmNoL3Bvd2Vy cGMvcGxhdGZvcm1zL3Bvd2VybnYvcGNpLWlvZGEuYwo+Pj4+ICsrKyBiL2FyY2gvcG93ZXJwYy9w bGF0Zm9ybXMvcG93ZXJudi9wY2ktaW9kYS5jCj4+Pj4gQEAgLTE5NTUsNiArMTk1NSwyMCBAQCBz dGF0aWMgc3RydWN0IGlvbW11X3RhYmxlX29wcyBwbnZfaW9kYTJfaW9tbXVfb3BzID0gewo+Pj4+ ICAgIAkuZnJlZSA9IHBudl9pb2RhMl90YWJsZV9mcmVlLAo+Pj4+ICAgIH07Cj4+Pj4gICAgCj4+ Pj4gK3N0YXRpYyBib29sIHBudl9pb2RhX2lvbW11X2NhcGFibGUoZW51bSBpb21tdV9jYXAgY2Fw KQo+Pj4+ICt7Cj4+Pj4gKwlzd2l0Y2ggKGNhcCkgewo+Pj4+ICsJY2FzZSBJT01NVV9DQVBfSU5U Ul9SRU1BUDoKPj4+PiArCQlyZXR1cm4gdHJ1ZTsKPj4+PiArCWRlZmF1bHQ6Cj4+Pj4gKwkJcmV0 dXJuIGZhbHNlOwo+Pj4+ICsJfQo+Pj4+ICt9Cj4+Pj4gKwo+Pj4+ICtzdGF0aWMgc3RydWN0IGlv bW11X29wcyBwbnZfaW9kYV9pb21tdV9vcHMgPSB7Cj4+Pj4gKwkuY2FwYWJsZSA9IHBudl9pb2Rh X2lvbW11X2NhcGFibGUsCj4+Pj4gK307Cj4+Pj4gKwo+Pj4+ICAgIHN0YXRpYyB2b2lkIHBudl9w Y2lfaW9kYV9zZXR1cF9kbWFfcGUoc3RydWN0IHBudl9waGIgKnBoYiwKPj4+PiAgICAJCQkJICAg ICAgc3RydWN0IHBudl9pb2RhX3BlICpwZSwgdW5zaWduZWQgaW50IGJhc2UsCj4+Pj4gICAgCQkJ CSAgICAgIHVuc2lnbmVkIGludCBzZWdzKQo+Pj4+IEBAIC0zMDc4LDYgKzMwOTIsOSBAQCBzdGF0 aWMgdm9pZCBwbnZfcGNpX2lvZGFfZml4dXAodm9pZCkKPj4+PiAgICAKPj4+PiAgICAJLyogTGlu ayBOUFUgSU9EQSB0YWJsZXMgdG8gdGhlaXIgUENJIGRldmljZXMuICovCj4+Pj4gICAgCXBudl9u cHVfaW9kYV9maXh1cCgpOwo+Pj4+ICsKPj4+PiArCS8qIEFkZCBJT01NVV9DQVBfSU5UUl9SRU1B UCAqLwo+Pj4+ICsJYnVzX3NldF9pb21tdSgmcGNpX2J1c190eXBlLCAmcG52X2lvZGFfaW9tbXVf b3BzKTsKPj4+PiAgICB9Cj4+Pj4gICAgCj4+Pj4gICAgLyoKPj4+IERvZXNuJ3QgdGhpcyBzZXQg eW91IHVwIGZvciBhIHdvcmxkIG9mIGh1cnQ/ICBidXNfc2V0X2lvbW11KCkgY2FsbHMKPj4+IGlv bW11X2J1c19pbml0KCkgd2hpY2ggc2V0cyB1cCBub3RpZmllcnMsIHdoaWNoIG1heWJlIHlvdSBk b24ndCBjYXJlCj4+PiBhYm91dCwgYnV0IGl0IGFsc28gbWVhbnMgdGhhdCBpb21tdV9kb21haW5f YWxsb2MoJnBjaV9idXNfdHlwZSkgd2lsbAo+Pj4gc2VnZmF1bHQgYmVjYXVzZSB5b3UncmUgbm90 IHByb3ZpZGluZyBhIGRvbWFpbl9hbGxvYyBjYWxsYmFjayBoZXJlLgo+PiBJdCBzZWVtcyB0byBi ZSBoYXJkIHRvIGFkZCBJT01NVV9DQVBfSU5UUl9SRU1BUCBvbgo+PiBQUEM2NCBwbGF0Zm9ybS4K Pj4KPj4gQW5kIGNhbiB3ZSBhZGQgYSBuZXcgaW9jdGwgaW4gdmZpb19pb21tdV9kcml2ZXIgdG8g Y2hlY2sKPj4gaWYgaW50ZXJydXB0IHJlbWFwcGluZyBpcyBzdXBwb3J0ZWQgc28gdGhhdCB3ZSBj YW4gdXNlIG91cgo+PiBvd24gd2F5IHRvIGRldGVybWluZSB0aGF0IG9uIFBQQzY0IHBsYXRmb3Jt Pwo+IEknZCBwcmVmZXIgbm90LiAgQXQgdGhlIHZmaW8gdXNlciBBUEkgbGV2ZWwsIHRoZSBxdWVz dGlvbiBpcyB3aGV0aGVyCj4gdGhlIHVzZXIgY2FuIG1tYXAgb3ZlciB0aGUgbXNpeCB0YWJsZSwg dGVzdGluZyBhIHByb3BlcnR5L2lvY3RsIG9uIHRoZQo+IGlvbW11IGRyaXZlciBzZWVtcyBsaWtl IGFuIG9kZCB3YXkgdG8gZGlzY292ZXIgdGhhdC4gIFdlIHNob3VsZCBiZQo+IGRldGVybWluaW5n IHdoZXRoZXIgdGhhdCdzIHNhZmUgaW4gdGhlIGtlcm5lbCBhbmQgZXhwb3J0aW5nIHRoYXQgaW5m bwo+IG9uIHRoZSB2ZmlvIGRldmljZSBpdHNlbGYsIHdoZXJlIGl0IHNlZW1zIGxpa2Ugd2UgaGF2 ZSB2YXJpb3VzIHdheXMgd2UKPiBjb3VsZCBkbyB0aGlzIHdpdGhpbiB0aGUgZXhpc3RpbmcgaW9j dGxzLiAgVGhhbmtzLAo+Cj4gQWxleAo+CgpZZXMsIHlvdSBhcmUgcmlnaHQuIEl0J3Mgbm90IGEg Z29vZCBpZGVhIHRvIGFkZCBhIG5ldyBpb2N0bCBpbgp2ZmlvX2lvbW11X2RyaXZlci4gTm93IEkn ZCBsaWtlIHRvIHRhbGsgYWJvdXQgdGhlIHdheSB0bwpkZXRlcm1pbmluZyB3aGV0aGVyIGl0J3Mg c2FmZSB0byBtbWFwIG92ZXIgdGhlIG1zaXggdGFibGUuCgpXZSBjdXJyZW50bHkgdXNlIElPTU1V X0NBUF9JTlRSX1JFTUFQIHRvIGRldGVybWluZSB0aGF0LgpCdXQgdGhlcmUgYXJlIHNvbWUgcHJv YmxlbXMgb24gUFBDNjQgd2hpY2ggbmV2ZXIgc2V0CmlvbW11X29wcyBhbmQgQVJNIFNNTVUgd2hp Y2ggc2V0IHRoaXMgY2FwYWJpbGl0eSBidXQgbm90CnByb3ZpZGUgaW50ZXJydXB0IGlzb2xhdGlv bi4gQ2FuIHdlIGFkZCBhIHZhcmlhYmxlL3Byb3BlcnR5CndoaWNoIGNhbiBiZSBzZXQgaW4gdmZp b19pb21tdV9kcml2ZXItPm9wcy0+YXR0YWNoX2dyb3VwKCkKYW5kIHVzZWQgaW4gdmZpb19wY2lf ZHJpdmVyIHRvIGRldGVybWluZSB3aGV0aGVyIHdlIGNhbiBhbGxvdwptbWFwcGluZyBtc2l4IHRh YmxlPyAgSWYgc28sIHdlIGNhbiBzdGlsbCB1c2UKSU9NTVVfQ0FQX0lOVFJfUkVNQVAsIG9yIHNv bWUgYXJjaC1pbmRlcGVuZGVudCB3YXlzCndoZW4gSU9NTVVfQ0FQX0lOVFJfUkVNQVAgZG9lc24n dCB3b3JrLgoKVGhhbmtzLApZb25namkgWGllCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXwpMaW51eHBwYy1kZXYgbWFpbGluZyBsaXN0CkxpbnV4cHBjLWRl dkBsaXN0cy5vemxhYnMub3JnCmh0dHBzOi8vbGlzdHMub3psYWJzLm9yZy9saXN0aW5mby9saW51 eHBwYy1kZXY=