From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5203AC4360F for ; Fri, 5 Apr 2019 09:51:52 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1EB0621850 for ; Fri, 5 Apr 2019 09:51:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="kAPVt78D" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EB0621850 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mjlS8hsHFc3ewLuQZviUyJPotw8pd2oSa0wPO8HPjuc=; b=kAPVt78DleT2kgZ/qEDyZKcdG +fHlJOycJehnV2rCJ/np0soDfFMJ1u0gBWgNX+FRgYWE+dKIrzheagQv1VEJepRC4hDQtvuvHp5b2 IwEMDUdouDtsJxxrMHCDAJsAVWd7z8SEN0VvXn06m+uRCl2dOZozgfibW1S19mEMRKwFnv9/TLZDX apw0Brw3kDX2WKEZ6a4stt8266colEc2DtS68VGQwpD96st6iTKDWFeWsh7MKaprjm4sBNjZkLz3w Uzsg2FyWoMeTmc9JwoCqqo9agcQ5mGtM1mHyHl9kD1uV14gqGO8pROwAa9xtFKj2UNFzWphSm/I93 kcnjFq+2A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hCLVj-0002lN-8W; Fri, 05 Apr 2019 09:51:47 +0000 Received: from foss.arm.com ([217.140.101.70]) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hCLVg-0002jg-DK for linux-arm-kernel@lists.infradead.org; Fri, 05 Apr 2019 09:51:45 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4FEAF168F; Fri, 5 Apr 2019 02:51:42 -0700 (PDT) Received: from [10.1.196.75] (e110467-lin.cambridge.arm.com [10.1.196.75]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 571B83F721; Fri, 5 Apr 2019 02:51:40 -0700 (PDT) Subject: Re: [PATCH v3 1/3] iommu: io-pgtable: Add ARM Mali midgard MMU page table format To: Steven Price , Rob Herring , dri-devel@lists.freedesktop.org References: <20190401074730.12241-1-robh@kernel.org> <20190401074730.12241-2-robh@kernel.org> <4ab58561-17e8-364c-a315-e55955ffdd49@arm.com> From: Robin Murphy Message-ID: Date: Fri, 5 Apr 2019 10:51:38 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <4ab58561-17e8-364c-a315-e55955ffdd49@arm.com> Content-Language: en-GB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190405_025144_456315_1D1E4BA1 X-CRM114-Status: GOOD ( 36.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Neil Armstrong , Maxime Ripard , Will Deacon , linux-kernel@vger.kernel.org, David Airlie , iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, Sean Paul , Alyssa Rosenzweig Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Steve, On 05/04/2019 10:42, Steven Price wrote: > First let me say congratulations to everyone working on Panfrost - it's > an impressive achievement! > > Full disclosure: I used to work on the Mali kbase driver. And have been > playing around with running the Mali user-space blob with the Panfrost > kernel driver. > > On 01/04/2019 08:47, Rob Herring wrote: >> ARM Mali midgard GPU is similar to standard 64-bit stage 1 page tables, but >> have a few differences. Add a new format type to represent the format. The >> input address size is 48-bits and the output address size is 40-bits (and >> possibly less?). Note that the later bifrost GPUs follow the standard >> 64-bit stage 1 format. >> >> The differences in the format compared to 64-bit stage 1 format are: >> >> The 3rd level page entry bits are 0x1 instead of 0x3 for page entries. >> >> The access flags are not read-only and unprivileged, but read and write. >> This is similar to stage 2 entries, but the memory attributes field matches >> stage 1 being an index. >> >> The nG bit is not set by the vendor driver. This one didn't seem to matter, >> but we'll keep it aligned to the vendor driver. > > The nG bit should be ignored by the hardware. > > The MMU in Midgard/Bifrost has a quirk similar to > IO_PGTABLE_QUIRK_TLBI_ON_MAP - you must perform a cache flush for the > GPU to (reliably) pick up new page table mappings. > > You may not have seen this because of the use of the JS_CONFIG_START_MMU > bit - this effectively performs a cache flush and TLB invalidate before > starting a job, however when using a GPU like T760 (e.g. on the Firefly > RK3288) this bit isn't being set. In my testing on the Firefly board I > saw GPU page faults because of this. > > There's two options for fixing this - a patch like below adds the quirk > mode to the MMU. Or alternatively always set JS_CONFIG_START_MMU on > jobs. In my testing both options solve the page faults. > > To be honest I don't know the reasoning behind kbase making the > JS_CONFIG_START_MMU bit conditional - I'm not aware of any reason why it > can't always be set. My guess is performance, but I haven't benchmarked > the difference between this and JS_CONFIG_START_MMU. > > -----8<---------- > From e3f75c7f04e43238dfc579029b8c11fb6b4a0c18 Mon Sep 17 00:00:00 2001 > From: Steven Price > Date: Thu, 4 Apr 2019 15:53:17 +0100 > Subject: [PATCH] iommu: io-pgtable: IO_PGTABLE_QUIRK_TLBI_ON_MAP for LPAE > > Midgard/Bifrost GPUs require a TLB invalidation when mapping pages, > implement IO_PGTABLE_QUIRK_TLBI_ON_MAP for LPAE iommu page table > formats and add the quirk bit to Panfrost. > > Signed-off-by: Steven Price > --- > drivers/gpu/drm/panfrost/panfrost_mmu.c | 1 + > drivers/iommu/io-pgtable-arm.c | 11 +++++++++-- > 2 files changed, 10 insertions(+), 2 deletions(-) > > diff --git a/drivers/gpu/drm/panfrost/panfrost_mmu.c > b/drivers/gpu/drm/panfrost/panfrost_mmu.c > index f3aad8591cf4..094312074d66 100644 > --- a/drivers/gpu/drm/panfrost/panfrost_mmu.c > +++ b/drivers/gpu/drm/panfrost/panfrost_mmu.c > @@ -343,6 +343,7 @@ int panfrost_mmu_init(struct panfrost_device *pfdev) > mmu_write(pfdev, MMU_INT_MASK, ~0); > > pfdev->mmu->pgtbl_cfg = (struct io_pgtable_cfg) { > + .quirks = IO_PGTABLE_QUIRK_TLBI_ON_MAP, > .pgsize_bitmap = SZ_4K, // | SZ_2M | SZ_1G), > .ias = 48, > .oas = 40, /* Should come from dma mask? */ > diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c > index 84beea1f47a7..45fd7bbdf9aa 100644 > --- a/drivers/iommu/io-pgtable-arm.c > +++ b/drivers/iommu/io-pgtable-arm.c > @@ -505,7 +505,13 @@ static int arm_lpae_map(struct io_pgtable_ops *ops, > unsigned long iova, > * Synchronise all PTE updates for the new mapping before there's > * a chance for anything to kick off a table walk for the new iova. > */ > - wmb(); > + if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_TLBI_ON_MAP) { > + io_pgtable_tlb_add_flush(&data->iop, iova, size, > + ARM_LPAE_BLOCK_SIZE(2, data), false); For correctness (in case this ever ends up used for something with VMSA-like invalidation behaviour), the granule would need to be "size" here, rather than effectively hard-coded. However, since Mali's invalidations appear to operate on arbitrary ranges, it would probably be a lot more efficient for the driver to handle this case directly, by just issuing a single big invalidation after the for_each_sg() loop in panfrost_mmu_map(). Robin. > + io_pgtable_tlb_sync(&data->iop); > + } else { > + wmb(); > + } > > return ret; > } > @@ -800,7 +806,8 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg > *cfg, void *cookie) > struct arm_lpae_io_pgtable *data; > > if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS | IO_PGTABLE_QUIRK_NO_DMA | > - IO_PGTABLE_QUIRK_NON_STRICT)) > + IO_PGTABLE_QUIRK_NON_STRICT | > + IO_PGTABLE_QUIRK_TLBI_ON_MAP)) > return NULL; > > data = arm_lpae_alloc_pgtable(cfg); > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel