From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 384AFEE498E for ; Fri, 18 Aug 2023 19:11:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379564AbjHRTK4 (ORCPT ); Fri, 18 Aug 2023 15:10:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379586AbjHRTK3 (ORCPT ); Fri, 18 Aug 2023 15:10:29 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id C70C0421B; Fri, 18 Aug 2023 12:10:27 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0405DD75; Fri, 18 Aug 2023 12:11:08 -0700 (PDT) Received: from [10.57.91.158] (unknown [10.57.91.158]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AFD243F762; Fri, 18 Aug 2023 12:10:19 -0700 (PDT) Message-ID: Date: Fri, 18 Aug 2023 20:10:15 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: [PATCH v11 4/6] iommu/s390: Force ISM devices to use IOMMU_DOMAIN_DMA Content-Language: en-GB To: Niklas Schnelle , Joerg Roedel , Matthew Rosato , Will Deacon , Wenjia Zhang , Jason Gunthorpe Cc: Gerd Bayer , Julian Ruess , Pierre Morel , Alexandra Winter , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Suravee Suthikulpanit , Hector Martin , Sven Peter , Alyssa Rosenzweig , David Woodhouse , Lu Baolu , Andy Gross , Bjorn Andersson , Konrad Dybcio , Yong Wu , Matthias Brugger , AngeloGioacchino Del Regno , Gerald Schaefer , Orson Zhai , Baolin Wang , Chunyan Zhang , Chen-Yu Tsai , Jernej Skrabec , Samuel Holland , Thierry Reding , Krishna Reddy , Jonathan Hunter , Jonathan Corbet , linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, asahi@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, linux-sunxi@lists.linux.dev, linux-tegra@vger.kernel.org, linux-doc@vger.kernel.org References: <20230717-dma_iommu-v11-0-a7a0b83c355c@linux.ibm.com> <20230717-dma_iommu-v11-4-a7a0b83c355c@linux.ibm.com> From: Robin Murphy In-Reply-To: <20230717-dma_iommu-v11-4-a7a0b83c355c@linux.ibm.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-s390@vger.kernel.org On 2023-07-17 12:00, Niklas Schnelle wrote: > ISM devices are virtual PCI devices used for cross-LPAR communication. > Unlike real PCI devices ISM devices do not use the hardware IOMMU but > inspects IOMMU translation tables directly on IOTLB flush (s390 RPCIT > instruction). > > While ISM devices keep their DMA allocations static and only very rarely > DMA unmap at all, For each IOTLB flush that occurs after unmap the ISM > devices will inspect the area of the IOVA space indicated by the flush. > This means that for the global IOTLB flushes used by the flush queue > mechanism the entire IOVA space would be inspected. In principle this > would be fine, albeit potentially unnecessarily slow, it turns out > however that ISM devices are sensitive to seeing IOVA addresses that are > currently in use in the IOVA range being flushed. Seeing such in-use > IOVA addresses will cause the ISM device to enter an error state and > become unusable. > > Fix this by forcing IOMMU_DOMAIN_DMA to be used for ISM devices. This > makes sure IOTLB flushes only cover IOVAs that have been unmapped and > also restricts the range of the IOTLB flush potentially reducing latency > spikes. Would it not be simpler to return false for IOMMU_CAP_DEFERRED_FLUSH for these devices? Cheers, Robin. > Signed-off-by: Niklas Schnelle > --- > drivers/iommu/s390-iommu.c | 10 ++++++++++ > 1 file changed, 10 insertions(+) > > diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c > index f6d6c60e5634..020cc538e4c4 100644 > --- a/drivers/iommu/s390-iommu.c > +++ b/drivers/iommu/s390-iommu.c > @@ -710,6 +710,15 @@ struct zpci_iommu_ctrs *zpci_get_iommu_ctrs(struct zpci_dev *zdev) > return &zdev->s390_domain->ctrs; > } > > +static int s390_iommu_def_domain_type(struct device *dev) > +{ > + struct zpci_dev *zdev = to_zpci_dev(dev); > + > + if (zdev->pft == PCI_FUNC_TYPE_ISM) > + return IOMMU_DOMAIN_DMA; > + return 0; > +} > + > int zpci_init_iommu(struct zpci_dev *zdev) > { > u64 aperture_size; > @@ -789,6 +798,7 @@ static const struct iommu_ops s390_iommu_ops = { > .probe_device = s390_iommu_probe_device, > .probe_finalize = s390_iommu_probe_finalize, > .release_device = s390_iommu_release_device, > + .def_domain_type = s390_iommu_def_domain_type, > .device_group = generic_device_group, > .pgsize_bitmap = SZ_4K, > .get_resv_regions = s390_iommu_get_resv_regions, >