From mboxrd@z Thu Jan 1 00:00:00 1970 From: Will Deacon Subject: Re: [RESEND PATCH v17 2/5] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device Date: Mon, 26 Nov 2018 19:31:52 +0000 Message-ID: <20181126193151.GC534@arm.com> References: <20181116112430.31248-1-vivek.gautam@codeaurora.org> <20181116112430.31248-3-vivek.gautam@codeaurora.org> <20181121173757.GA9801@arm.com> <20181123183555.GE21183@arm.com> <9064c01e-cef0-9306-078a-8d303cd6614b@codeaurora.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: freedreno-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org Sender: "Freedreno" To: Vivek Gautam Cc: Mark Rutland , "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS" , Linux PM , sboyd-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, linux-arm-msm , Joerg Roedel , "Rafael J. Wysocki" , open list , "list-Y9sIeH5OGRo@public.gmane.org:IOMMU DRIVERS" , alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, robh+dt , freedreno , Robin Murphy List-Id: linux-arm-msm@vger.kernel.org T24gTW9uLCBOb3YgMjYsIDIwMTggYXQgMDQ6NTY6NDJQTSArMDUzMCwgVml2ZWsgR2F1dGFtIHdy b3RlOgo+IE9uIDExLzI2LzIwMTggMTE6MzMgQU0sIFZpdmVrIEdhdXRhbSB3cm90ZToKPiA+T24g MTEvMjQvMjAxOCAxMjowNiBBTSwgV2lsbCBEZWFjb24gd3JvdGU6Cj4gPj5PbiBUaHUsIE5vdiAy MiwgMjAxOCBhdCAwNTozMjoyNFBNICswNTMwLCBWaXZlayBHYXV0YW0gd3JvdGU6Cj4gPj4+T24g V2VkLCBOb3YgMjEsIDIwMTggYXQgMTE6MDkgUE0gV2lsbCBEZWFjb24gPHdpbGwuZGVhY29uQGFy bS5jb20+Cj4gPj4+d3JvdGU6Cj4gPj4+Pk9uIEZyaSwgTm92IDE2LCAyMDE4IGF0IDA0OjU0OjI3 UE0gKzA1MzAsIFZpdmVrIEdhdXRhbSB3cm90ZToKPiA+Pj4+PkZyb206IFNyaWNoYXJhbiBSIDxz cmljaGFyYW5AY29kZWF1cm9yYS5vcmc+Cj4gPj4+Pj4KPiA+Pj4+PlRoZSBzbW11IGRldmljZSBw cm9iZS9yZW1vdmUgYW5kIGFkZC9yZW1vdmUgbWFzdGVyIGRldmljZSBjYWxsYmFja3MKPiA+Pj4+ PmdldHMgY2FsbGVkIHdoZW4gdGhlIHNtbXUgaXMgbm90IGxpbmtlZCB0byBpdHMgbWFzdGVyLCB0 aGF0IGlzCj4gPj4+Pj53aXRob3V0Cj4gPj4+Pj50aGUgY29udGV4dCBvZiB0aGUgbWFzdGVyIGRl dmljZS4gU28gY2FsbGluZyBydW50aW1lIGFwaXMgaW4gdGhvc2UKPiA+Pj4+PnBsYWNlcwo+ID4+ Pj4+c2VwYXJhdGVseS4KPiA+Pj4+Pkdsb2JhbCBsb2NrcyBhcmUgYWxzbyBpbml0aWFsaXplZCBi ZWZvcmUgZW5hYmxpbmcgcnVudGltZSBwbSBhcyB0aGUKPiA+Pj4+PnJ1bnRpbWVfcmVzdW1lKCkg Y2FsbHMgZGV2aWNlX3Jlc2V0KCkgd2hpY2ggZG9lcyB0bGJfc3luY19nbG9iYWwoKQo+ID4+Pj4+ dGhhdCB1bHRpbWF0ZWx5IHJlcXVpcmVzIGxvY2tzIHRvIGJlIGluaXRpYWxpemVkLgo+ID4+Pj4+ Cj4gPj4+Pj5TaWduZWQtb2ZmLWJ5OiBTcmljaGFyYW4gUiA8c3JpY2hhcmFuQGNvZGVhdXJvcmEu b3JnPgo+ID4+Pj4+W3ZpdmVrOiBDbGVhbnVwIHBtIHJ1bnRpbWUgY2FsbHNdCj4gPj4+Pj5TaWdu ZWQtb2ZmLWJ5OiBWaXZlayBHYXV0YW0gPHZpdmVrLmdhdXRhbUBjb2RlYXVyb3JhLm9yZz4KPiA+ Pj4+PlJldmlld2VkLWJ5OiBUb21hc3ogRmlnYSA8dGZpZ2FAY2hyb21pdW0ub3JnPgo+ID4+Pj4+ VGVzdGVkLWJ5OiBTcmluaXZhcyBLYW5kYWdhdGxhIDxzcmluaXZhcy5rYW5kYWdhdGxhQGxpbmFy by5vcmc+Cj4gPj4+Pj5SZXZpZXdlZC1ieTogUm9iaW4gTXVycGh5IDxyb2Jpbi5tdXJwaHlAYXJt LmNvbT4KPiA+Pj4+Pi0tLQo+ID4+Pj4+wqAgZHJpdmVycy9pb21tdS9hcm0tc21tdS5jIHwgMTAx Cj4gPj4+Pj4rKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKystLS0tLQo+ ID4+Pj4+wqAgMSBmaWxlIGNoYW5nZWQsIDkxIGluc2VydGlvbnMoKyksIDEwIGRlbGV0aW9ucygt KQo+ID4+Pj5HaXZlbiB0aGF0IHlvdSdyZSBkb2luZyB0aGUgZ2V0L3B1dCBpbiB0aGUgVExCSSBv cHMgdW5jb25kaXRpb25hbGx5Ogo+ID4+Pj4KPiA+Pj4+PsKgIHN0YXRpYyB2b2lkIGFybV9zbW11 X2ZsdXNoX2lvdGxiX2FsbChzdHJ1Y3QgaW9tbXVfZG9tYWluICpkb21haW4pCj4gPj4+Pj7CoCB7 Cj4gPj4+Pj7CoMKgwqDCoMKgwqAgc3RydWN0IGFybV9zbW11X2RvbWFpbiAqc21tdV9kb21haW4g PSB0b19zbW11X2RvbWFpbihkb21haW4pOwo+ID4+Pj4+K8KgwqDCoMKgIHN0cnVjdCBhcm1fc21t dV9kZXZpY2UgKnNtbXUgPSBzbW11X2RvbWFpbi0+c21tdTsKPiA+Pj4+Pgo+ID4+Pj4+LcKgwqDC oMKgIGlmIChzbW11X2RvbWFpbi0+dGxiX29wcykKPiA+Pj4+PivCoMKgwqDCoCBpZiAoc21tdV9k b21haW4tPnRsYl9vcHMpIHsKPiA+Pj4+PivCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgYXJtX3Nt bXVfcnBtX2dldChzbW11KTsKPiA+Pj4+PnNtbXVfZG9tYWluLT50bGJfb3BzLT50bGJfZmx1c2hf YWxsKHNtbXVfZG9tYWluKTsKPiA+Pj4+PivCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgYXJtX3Nt bXVfcnBtX3B1dChzbW11KTsKPiA+Pj4+PivCoMKgwqDCoCB9Cj4gPj4+Pj7CoCB9Cj4gPj4+Pj4K PiA+Pj4+PsKgIHN0YXRpYyB2b2lkIGFybV9zbW11X2lvdGxiX3N5bmMoc3RydWN0IGlvbW11X2Rv bWFpbiAqZG9tYWluKQo+ID4+Pj4+wqAgewo+ID4+Pj4+wqDCoMKgwqDCoMKgIHN0cnVjdCBhcm1f c21tdV9kb21haW4gKnNtbXVfZG9tYWluID0gdG9fc21tdV9kb21haW4oZG9tYWluKTsKPiA+Pj4+ PivCoMKgwqDCoCBzdHJ1Y3QgYXJtX3NtbXVfZGV2aWNlICpzbW11ID0gc21tdV9kb21haW4tPnNt bXU7Cj4gPj4+Pj4KPiA+Pj4+Pi3CoMKgwqDCoCBpZiAoc21tdV9kb21haW4tPnRsYl9vcHMpCj4g Pj4+Pj4rwqDCoMKgwqAgaWYgKHNtbXVfZG9tYWluLT50bGJfb3BzKSB7Cj4gPj4+Pj4rwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgIGFybV9zbW11X3JwbV9nZXQoc21tdSk7Cj4gPj4+Pj5zbW11X2Rv bWFpbi0+dGxiX29wcy0+dGxiX3N5bmMoc21tdV9kb21haW4pOwo+ID4+Pj4+K8KgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoCBhcm1fc21tdV9ycG1fcHV0KHNtbXUpOwo+ID4+Pj4+K8KgwqDCoMKgIH0K PiA+Pj4+V2h5IGRvIHlvdSBuZWVkIHRoZW0gYXJvdW5kIHRoZSBtYXAvdW5tYXAgY2FsbHMgYXMg d2VsbD8KPiA+Pj5XZSBzdGlsbCBoYXZlIC50bGJfYWRkX2ZsdXNoIHBhdGg/Cj4gPj5Paywgc28g d2UgY291bGQgYWRkIHRoZSBvcHMgYXJvdW5kIHRoYXQgYXMgd2VsbC4gUmlnaHQgbm93LCB3ZSd2 ZSBnb3QKPiA+PnRoZSBydW50aW1lIHBtIGhvb2tzIGNyb3NzaW5nIHR3byBwYXJ0cyBvZiB0aGUg QVBJLgo+ID4KPiA+U3VyZSwgd2lsbCBkbyB0aGF0IHRoZW4sIGFuZCByZW1vdmUgdGhlIHJ1bnRp bWUgcG0gaG9va3MgZnJvbSBtYXAvdW5tYXAuCj4gCj4gSSBtaXNzZWQgdGhpcyBlYXJsaWVyIC0K PiBXZSBhcmUgYWRkaW5nIHJ1bnRpbWUgcG0gaG9va3MgaW4gdGhlICdpb21tdV9vcHMnIGNhbGxi YWNrcyBhbmQgbm90IHJlYWxseQo+IHRvCj4gJ3RsYl9vcHMnLiBTbyBob3cgdGhlIHJ1bnRpbWUg cG0gaG9va3MgY3Jvc3NpbmcgdGhlIHBhdGhzPwo+ICcubWFwLy51bm1hcCcgaW9tbXVfb3BzIGRv bid0IGNhbGwgJy5mbHVzaF9pb3RsYl9hbGwnIG9yICcuaW90bGJfc3luYycKPiBpb21tdV9vcHMK PiBhbnl3aGVyZS4KPiAKPiBFLmcuLCBvbmx5IGNhbGxlcnMgdG8gZG9tYWluLT5vcHMtPmZsdXNo X2lvdGxiX2FsbCgpIGFyZToKPiBpb21tdV9kbWFfZmx1c2hfaW90bGJfYWxsKCksIG9yIGlvbW11 X2ZsdXNoX3RsYl9hbGwoKSB3aGljaCBhcmUgbm90IGluCj4gbWFwL3VubWFwIHBhdGhzLgoKWWVz LCBzb3JyeSwgSSBnb3QgY29uZnVzZWQgaGVyZSBhbmQgY29tcGxldGVseSBtaXNsZWQgeW91LiBJ biB3aGljaCBjYXNlLAp5b3VyIG9yaWdpbmFsIHBhdGNoIGlzIG9rIGJlY2F1c2UgaXQgaW50ZXJj ZXB0cyB0aGUgY29yZSBJT01NVSBBUEkgdmlhCmlvbW11X29wcy4gQXBvbG9naWVzLgoKQXQgdGhh dCBsZXZlbCwgc2hvdWxkIHdlIGFsc28gYW5ub3RhdGUgYXJtX3NtbXVfaW92YV90b19waHlzX2hh cmQoKQpmb3IgdGhlIGlvdmFfdG9fcGh5cygpIGltcGxlbWVudGF0aW9uPwoKV2l0aCB0aGF0IGRl dGFpbCBhbmQgY2xvY2sgYml0cyBzb3J0ZWQgb3V0LCB3ZSBzaG91bGQgYmUgYWJsZSB0byBnZXQg dGhpcwpxdWV1ZWQgYXQgbGFzdC4KCldpbGwKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18KRnJlZWRyZW5vIG1haWxpbmcgbGlzdApGcmVlZHJlbm9AbGlzdHMu ZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlz dGluZm8vZnJlZWRyZW5vCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3667DC43441 for ; Mon, 26 Nov 2018 19:31:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0400E20855 for ; Mon, 26 Nov 2018 19:31:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0400E20855 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727049AbeK0G0n (ORCPT ); Tue, 27 Nov 2018 01:26:43 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:45968 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726343AbeK0G0m (ORCPT ); Tue, 27 Nov 2018 01:26:42 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1DCC82381; Mon, 26 Nov 2018 11:31:35 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E1D333F5AF; Mon, 26 Nov 2018 11:31:34 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 6FDF31AE0839; Mon, 26 Nov 2018 19:31:52 +0000 (GMT) Date: Mon, 26 Nov 2018 19:31:52 +0000 From: Will Deacon To: Vivek Gautam Cc: Mark Rutland , "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS" , alex.williamson@redhat.com, Linux PM , sboyd@kernel.org, freedreno , "Rafael J. Wysocki" , open list , "list@263.net:IOMMU DRIVERS" , Joerg Roedel , robh+dt , linux-arm-msm , Robin Murphy Subject: Re: [RESEND PATCH v17 2/5] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device Message-ID: <20181126193151.GC534@arm.com> References: <20181116112430.31248-1-vivek.gautam@codeaurora.org> <20181116112430.31248-3-vivek.gautam@codeaurora.org> <20181121173757.GA9801@arm.com> <20181123183555.GE21183@arm.com> <9064c01e-cef0-9306-078a-8d303cd6614b@codeaurora.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Nov 26, 2018 at 04:56:42PM +0530, Vivek Gautam wrote: > On 11/26/2018 11:33 AM, Vivek Gautam wrote: > >On 11/24/2018 12:06 AM, Will Deacon wrote: > >>On Thu, Nov 22, 2018 at 05:32:24PM +0530, Vivek Gautam wrote: > >>>On Wed, Nov 21, 2018 at 11:09 PM Will Deacon > >>>wrote: > >>>>On Fri, Nov 16, 2018 at 04:54:27PM +0530, Vivek Gautam wrote: > >>>>>From: Sricharan R > >>>>> > >>>>>The smmu device probe/remove and add/remove master device callbacks > >>>>>gets called when the smmu is not linked to its master, that is > >>>>>without > >>>>>the context of the master device. So calling runtime apis in those > >>>>>places > >>>>>separately. > >>>>>Global locks are also initialized before enabling runtime pm as the > >>>>>runtime_resume() calls device_reset() which does tlb_sync_global() > >>>>>that ultimately requires locks to be initialized. > >>>>> > >>>>>Signed-off-by: Sricharan R > >>>>>[vivek: Cleanup pm runtime calls] > >>>>>Signed-off-by: Vivek Gautam > >>>>>Reviewed-by: Tomasz Figa > >>>>>Tested-by: Srinivas Kandagatla > >>>>>Reviewed-by: Robin Murphy > >>>>>--- > >>>>>  drivers/iommu/arm-smmu.c | 101 > >>>>>++++++++++++++++++++++++++++++++++++++++++----- > >>>>>  1 file changed, 91 insertions(+), 10 deletions(-) > >>>>Given that you're doing the get/put in the TLBI ops unconditionally: > >>>> > >>>>>  static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain) > >>>>>  { > >>>>>       struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); > >>>>>+     struct arm_smmu_device *smmu = smmu_domain->smmu; > >>>>> > >>>>>-     if (smmu_domain->tlb_ops) > >>>>>+     if (smmu_domain->tlb_ops) { > >>>>>+             arm_smmu_rpm_get(smmu); > >>>>>smmu_domain->tlb_ops->tlb_flush_all(smmu_domain); > >>>>>+             arm_smmu_rpm_put(smmu); > >>>>>+     } > >>>>>  } > >>>>> > >>>>>  static void arm_smmu_iotlb_sync(struct iommu_domain *domain) > >>>>>  { > >>>>>       struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); > >>>>>+     struct arm_smmu_device *smmu = smmu_domain->smmu; > >>>>> > >>>>>-     if (smmu_domain->tlb_ops) > >>>>>+     if (smmu_domain->tlb_ops) { > >>>>>+             arm_smmu_rpm_get(smmu); > >>>>>smmu_domain->tlb_ops->tlb_sync(smmu_domain); > >>>>>+             arm_smmu_rpm_put(smmu); > >>>>>+     } > >>>>Why do you need them around the map/unmap calls as well? > >>>We still have .tlb_add_flush path? > >>Ok, so we could add the ops around that as well. Right now, we've got > >>the runtime pm hooks crossing two parts of the API. > > > >Sure, will do that then, and remove the runtime pm hooks from map/unmap. > > I missed this earlier - > We are adding runtime pm hooks in the 'iommu_ops' callbacks and not really > to > 'tlb_ops'. So how the runtime pm hooks crossing the paths? > '.map/.unmap' iommu_ops don't call '.flush_iotlb_all' or '.iotlb_sync' > iommu_ops > anywhere. > > E.g., only callers to domain->ops->flush_iotlb_all() are: > iommu_dma_flush_iotlb_all(), or iommu_flush_tlb_all() which are not in > map/unmap paths. Yes, sorry, I got confused here and completely misled you. In which case, your original patch is ok because it intercepts the core IOMMU API via iommu_ops. Apologies. At that level, should we also annotate arm_smmu_iova_to_phys_hard() for the iova_to_phys() implementation? With that detail and clock bits sorted out, we should be able to get this queued at last. Will