From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,UNPARSEABLE_RELAY,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F61BC47404 for ; Wed, 9 Oct 2019 13:53:55 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 30AA22067B for ; Wed, 9 Oct 2019 13:53:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Uvh4xiOs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 30AA22067B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=mediatek.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Date:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ljMRbWYLFeBTtg2h0X1BklWCExdoQwOSR/6HlD41ZBM=; b=Uvh4xiOssO1g+K beIJGzjtDJRq8d/mn1bTI4H+sicSaXL/PJ9UwjfAefEQ3F/0LdrG8OxdXYj/p6jqspiTKuLmCdx42 LxIq7ED51vJhSVr59IxL9jrWtQPDrKLrXqJHsq/onRo0m3/jIeb7o0DymZ8uqp6g4g5/aqsfLG6Ur LL5SJDofxjMX2R2RDVzdqzb9P6+7lhnGNg2QLPHftpAQ76Zhn7c01dG6HsbO8I6wvqh7JEJWvdPef Nhxsx1RDf/NpLA1xyEvnJN5QkxFKenHdVPHMW7yMMkDzlH4zzjb7eeOJYlB22YKr6TdbWPKXCuxp5 Z7xlndOGjsubVKqBP7Cw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iICPW-0006PA-2w; Wed, 09 Oct 2019 13:53:50 +0000 Received: from mailgw02.mediatek.com ([216.200.240.185]) by bombadil.infradead.org with esmtps (Exim 4.92.2 #3 (Red Hat Linux)) id 1iICPS-0006O4-6f; Wed, 09 Oct 2019 13:53:48 +0000 X-UUID: 9641ad5ebd0646cd8cad0fb38942ac0b-20191009 X-UUID: 9641ad5ebd0646cd8cad0fb38942ac0b-20191009 Received: from mtkcas68.mediatek.inc [(172.29.94.19)] by mailgw02.mediatek.com (envelope-from ) (musrelay.mediatek.com ESMTP with TLS) with ESMTP id 371666275; Wed, 09 Oct 2019 05:53:43 -0800 Received: from MTKMBS32N2.mediatek.inc (172.27.4.72) by MTKMBS62N1.mediatek.inc (172.29.193.41) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Wed, 9 Oct 2019 06:53:32 -0700 Received: from MTKCAS32.mediatek.inc (172.27.4.184) by MTKMBS32N2.mediatek.inc (172.27.4.72) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Wed, 9 Oct 2019 21:38:24 +0800 Received: from [10.17.3.153] (172.27.4.253) by MTKCAS32.mediatek.inc (172.27.4.170) with Microsoft SMTP Server id 15.0.1395.4 via Frontend Transport; Wed, 9 Oct 2019 21:38:24 +0800 Message-ID: <1570628307.19130.53.camel@mhfsdcap03> Subject: Re: [PATCH] iommu/mediatek: Move the tlb_sync into tlb_flush From: Yong Wu To: Tomasz Figa Date: Wed, 9 Oct 2019 21:38:27 +0800 In-Reply-To: References: <1569822142-14303-1-git-send-email-yong.wu@mediatek.com> <1570522162.19130.38.camel@mhfsdcap03> X-Mailer: Evolution 3.10.4-0ubuntu2 MIME-Version: 1.0 X-TM-SNTS-SMTP: 9DC8DFD7C36EE50E1052EBAAB997ADF22F64BBEC2A1D9E8D110DA80EEC3D152C2000:8 X-MTK: N X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191009_065346_248462_F47C0EEB X-CRM114-Status: GOOD ( 40.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: youlin.pei@mediatek.com, anan.sun@mediatek.com, Nicolas Boichat , cui.zhang@mediatek.com, srv_heupstream , Joerg Roedel , Will Deacon , Linux Kernel Mailing List , Evan Green , chao.hao@mediatek.com, "list@263.net:IOMMU DRIVERS , Joerg Roedel , " , "moderated list:ARM/Mediatek SoC support" , Matthias Brugger , Robin Murphy , "list@263.net:IOMMU DRIVERS , Joerg Roedel , " Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, 2019-10-09 at 16:56 +0900, Tomasz Figa wrote: > On Tue, Oct 8, 2019 at 5:09 PM Yong Wu wrote: > > > > Hi Tomasz, > > > > Sorry for reply late. > > > > On Wed, 2019-10-02 at 14:18 +0900, Tomasz Figa wrote: > > > Hi Yong, > > > > > > On Mon, Sep 30, 2019 at 2:42 PM Yong Wu wrote: > > > > > > > > The commit 4d689b619445 ("iommu/io-pgtable-arm-v7s: Convert to IOMMU API > > > > TLB sync") help move the tlb_sync of unmap from v7s into the iommu > > > > framework. It helps add a new function "mtk_iommu_iotlb_sync", But it > > > > lacked the dom->pgtlock, then it will cause the variable > > > > "tlb_flush_active" may be changed unexpectedly, we could see this warning > > > > log randomly: > > > > > > > > > > Thanks for the patch! Please see my comments inline. > > > > > > > mtk-iommu 10205000.iommu: Partial TLB flush timed out, falling back to > > > > full flush > > > > > > > > To fix this issue, we can add dom->pgtlock in the "mtk_iommu_iotlb_sync". > > > > And when checking this issue, we find that __arm_v7s_unmap call > > > > io_pgtable_tlb_add_flush consecutively when it is supersection/largepage, > > > > this also is potential unsafe for us. There is no tlb flush queue in the > > > > MediaTek M4U HW. The HW always expect the tlb_flush/tlb_sync one by one. > > > > If v7s don't always gurarantee the sequence, Thus, In this patch I move > > > > the tlb_sync into tlb_flush(also rename the function deleting "_nosync"). > > > > and we don't care if it is leaf, rearrange the callback functions. Also, > > > > the tlb flush/sync was already finished in v7s, then iotlb_sync and > > > > iotlb_sync_all is unnecessary. > > > > > > Performance-wise, we could do much better. Instead of synchronously > > > syncing at the end of mtk_iommu_tlb_add_flush(), we could sync at the > > > beginning, if there was any previous flush still pending. We would > > > also have to keep the .iotlb_sync() callback, to take care of waiting > > > for the last flush. That would allow better pipelining with CPU in > > > cases like this: > > > > > > for (all pages in range) { > > > change page table(); > > > flush(); > > > } > > > > > > "change page table()" could execute while the IOMMU is flushing the > > > previous change. > > > > Do you mean adding a new tlb_sync before tlb_flush_no_sync, like below: > > > > mtk_iommu_tlb_add_flush_nosync { > > + mtk_iommu_tlb_sync(); > > tlb_flush_no_sync(); > > data->tlb_flush_active = true; > > } > > > > mtk_iommu_tlb_sync { > > if (!data->tlb_flush_active) > > return; > > tlb_sync(); > > data->tlb_flush_active = false; > > } > > > > This way look improve the flow, But adjusting the flow is not the root > > cause of this issue. the problem is "data->tlb_flush_active" may be > > changed from mtk_iommu_iotlb_sync which don't have a dom->pglock. > > That was not the only problem with existing code. Existing code also > assumed that add_flush and sync always go in pairs, but that's not > true. Yes. Thus I put the tlb_flush always followed by tlb_sync to make sure they always go in pairs. > > My suggestion is to fix the locking in the driver and keep the sync > deferred as much as possible, so that performance is not degraded. I I really didn't get this timeout warning log in previous kernel(Many tlb_flush followed by one tlb_sync), But deferring the sync is not suggested by our DE, thus I still would like to fix the sequence in this patch with putting them together. > changed my mind, though. I think we would need to make more changes to > the driver to make it implement the flushing efficiently, so let's go > with the current simple approach for now and improve incrementally. > > > > > Currently the synchronisation of the tlb_flush/tlb_sync flow are > > controlled by the variable "data->tlb_flush_active". > > > > In this patch putting the tlb_flush/tlb_sync together looks make > > the flow simpler: > > a) Don't need the sensitive variable "tlb_flush_active". > > b) Remove mtk_iommu_iotlb_sync, Don't need add lock in it. > > c) Simplify the tlb_flush_walk/tlb_flush_leaf. > > is it ok? > > > > Okay, let's do so as a first step to fix the issue. Then we can > optimize in follow up patches. Thanks the confirm, I have sent a quick v2. > > > > > > > > > > > > Besides, there are two minor changes: > > > > a) Use writel for the register F_MMU_INV_RANGE which is for triggering the > > > > HW work. We expect all the setting(iova_start/iova_end...) have already > > > > been finished before F_MMU_INV_RANGE. > > > > b) Reduce the tlb timeout value from 100000us to 1000us. the original value > > > > is so long that affect the multimedia performance. > > > > > > By definition, timeout is something that should not normally happen. > > > Too long timeout affecting multimedia performance would suggest that > > > the timeout was actually happening, which is the core problem, not the > > > length of the timeout. Could you provide more details on this? > > > > As description above, this issue is because there is no dom->pgtlock in > > the mtk_iommu_iotlb_sync. I have tried that the issue will disappear > > after adding lock in it. > > > > Although the issue is fixed after this patch, I still would like to > > reduce the timeout value for somehow error happen in the future. 100ms > > is unnecessary for us. It looks a minor improvement rather than fixing > > the issue. I will use a new patch for it. > > > > Okay, makes sense. > > Best regards, > Tomasz _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel