From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0000830FB6 for ; Wed, 27 Sep 2023 16:16:36 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 6D67392; Wed, 27 Sep 2023 09:16:35 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F1F6B1FB; Wed, 27 Sep 2023 09:17:11 -0700 (PDT) Received: from [10.1.196.40] (e121345-lin.cambridge.arm.com [10.1.196.40]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4C3893F59C; Wed, 27 Sep 2023 09:16:28 -0700 (PDT) Message-ID: <96c0892f-20e6-db1f-f310-9b09d419aa6a@arm.com> Date: Wed, 27 Sep 2023 17:16:23 +0100 Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: [PATCH v12 0/6] iommu/dma: s390 DMA API conversion and optimized IOTLB flushing Content-Language: en-GB To: Jason Gunthorpe , Niklas Schnelle Cc: Joerg Roedel , Matthew Rosato , Will Deacon , Wenjia Zhang , Gerd Bayer , Julian Ruess , Pierre Morel , Alexandra Winter , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Suravee Suthikulpanit , Hector Martin , Sven Peter , Alyssa Rosenzweig , David Woodhouse , Lu Baolu , Andy Gross , Bjorn Andersson , Konrad Dybcio , Yong Wu , Matthias Brugger , AngeloGioacchino Del Regno , Gerald Schaefer , Orson Zhai , Baolin Wang , Chunyan Zhang , Chen-Yu Tsai , Jernej Skrabec , Samuel Holland , Thierry Reding , Krishna Reddy , Jonathan Hunter , Jonathan Corbet , linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, asahi@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, linux-sunxi@lists.linux.dev, linux-tegra@vger.kernel.org, linux-doc@vger.kernel.org References: <20230825-dma_iommu-v12-0-4134455994a7@linux.ibm.com> <20230926160832.GM13795@ziepe.ca> <6dab29f58ac1ccd58caaee031f98f4d0d382cbcd.camel@linux.ibm.com> <20230927154009.GN13795@ziepe.ca> From: Robin Murphy In-Reply-To: <20230927154009.GN13795@ziepe.ca> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-3.4 required=5.0 tests=BAYES_00,NICE_REPLY_A, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net On 27/09/2023 4:40 pm, Jason Gunthorpe wrote: > On Wed, Sep 27, 2023 at 05:24:20PM +0200, Niklas Schnelle wrote: > >> Ok, another update. On trying it out again this problem actually also >> occurs when applying this v12 on top of v6.6-rc3 too. Also I guess >> unlike my prior thinking it probably doesn't occur with >> iommu.forcedac=1 since that still allows IOVAs below 4 GiB and we might >> be the only ones who don't support those. From my point of view this >> sounds like a mlx5_core issue they really should call >> dma_set_mask_and_coherent() before their first call to >> dma_alloc_coherent() not after. So I guess I'll send a v13 of this >> series rebased on iommu/core and with an additional mlx5 patch and then >> let's hope we can get that merged in a way that doesn't leave us with >> broken ConnectX VFs for too long. > > Yes, OK. It definitely sounds wrong that mlx5 is doing dma allocations before > setting it's dma_set_mask_and_coherent(). Please link to this thread > and we can get Leon or Saeed to ack it for Joerg. > > (though wondering why s390 is the only case that ever hit this?) Probably because most systems happen to be able to satisfy the allocation within the default 32-bit mask - the whole bottom 4GB of IOVA space being reserved is pretty atypical. TBH it makes me wonder the opposite - how this ever worked on s390 before? And I think the answer to that is "by pure chance", since upon inspection the existing s390_pci_dma_ops implementation appears to pay absolutely no attention to the device's DMA masks whatsoever :( Robin.