From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F10AE1C2BA for ; Wed, 27 Sep 2023 09:26:28 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 24A64EB; Wed, 27 Sep 2023 02:26:27 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 01C3F1FB; Wed, 27 Sep 2023 02:27:05 -0700 (PDT) Received: from [10.57.0.175] (unknown [10.57.0.175]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DE5E73F59C; Wed, 27 Sep 2023 02:26:19 -0700 (PDT) Message-ID: <068aeebd-df24-5097-96e3-ebaaa7a763fd@arm.com> Date: Wed, 27 Sep 2023 10:26:14 +0100 Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: [PATCH v12 0/6] iommu/dma: s390 DMA API conversion and optimized IOTLB flushing Content-Language: en-GB To: Niklas Schnelle , Jason Gunthorpe , Joerg Roedel Cc: Matthew Rosato , Will Deacon , Wenjia Zhang , Gerd Bayer , Julian Ruess , Pierre Morel , Alexandra Winter , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Suravee Suthikulpanit , Hector Martin , Sven Peter , Alyssa Rosenzweig , David Woodhouse , Lu Baolu , Andy Gross , Bjorn Andersson , Konrad Dybcio , Yong Wu , Matthias Brugger , AngeloGioacchino Del Regno , Gerald Schaefer , Orson Zhai , Baolin Wang , Chunyan Zhang , Chen-Yu Tsai , Jernej Skrabec , Samuel Holland , Thierry Reding , Krishna Reddy , Jonathan Hunter , Jonathan Corbet , linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, asahi@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-arm-msm@vger.kernel.org, linux-mediatek@lists.infradead.org, linux-sunxi@lists.linux.dev, linux-tegra@vger.kernel.org, linux-doc@vger.kernel.org References: <20230825-dma_iommu-v12-0-4134455994a7@linux.ibm.com> <20230926160832.GM13795@ziepe.ca> From: Robin Murphy In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-3.4 required=5.0 tests=BAYES_00,NICE_REPLY_A, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net On 2023-09-27 09:55, Niklas Schnelle wrote: > On Tue, 2023-09-26 at 13:08 -0300, Jason Gunthorpe wrote: >> On Tue, Sep 26, 2023 at 05:04:28PM +0200, Joerg Roedel wrote: >>> Hi Niklas, >>> >>> On Fri, Aug 25, 2023 at 12:11:15PM +0200, Niklas Schnelle wrote: >>>> Niklas Schnelle (6): >>>> iommu: Allow .iotlb_sync_map to fail and handle s390's -ENOMEM return >>>> s390/pci: prepare is_passed_through() for dma-iommu >>>> s390/pci: Use dma-iommu layer >>>> iommu/s390: Disable deferred flush for ISM devices >>>> iommu/dma: Allow a single FQ in addition to per-CPU FQs >>>> iommu/dma: Use a large flush queue and timeout for shadow_on_flush >>> >>> Turned out this series has non-trivial conflicts with Jasons >>> default-domain work so I had to remove it from the IOMMU tree for now. >>> Can you please rebase it to the latest iommu/core branch and re-send? I >>> will take it into the tree again then. >> >> Niklas, I think you just 'take yours' to resolve this. All the >> IOMMU_DOMAIN_PLATFORM related and .default_domain = parts should be >> removed. Let me know if you need anything >> >> Thanks, >> Jason > > Hi Joerg, Hi Jason, > > I've run into an unfortunate problem, not with the rebase itself but > with the iommu/core branch. > > Jason is right, I basically need to just remove the platform ops and > .default_domain ops. This seems to work fine for an NVMe both in the > host and also when using the IOMMU with vfio-pci + KVM. I've already > pushed the result of that to my git.kernel.org: > https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/log/?h=b4/dma_iommu > > The problem is that something seems to be broken in the iommu/core > branch. Regardless of whether I have my DMA API conversion on top or > with the base iommu/core branch I can not use ConnectX-4 VFs. > > # lspci > 111a:00:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] > # dmesg | grep mlx > [ 3.189749] mlx5_core 111a:00:00.0: mlx5_mdev_init:1802:(pid 464): Failed initializing cmdif SW structs, aborting > [ 3.189783] mlx5_core: probe of 111a:00:00.0 failed with error -12 > > This same card works on v6.6-rc3 both with and without my DMA API > conversion patch series applied. Looking at mlx5_mdev_init() -> > mlx5_cmd_init(). The -ENOMEM seems to come from the following > dma_pool_create(): > > cmd->pool = dma_pool_create("mlx5_cmd", mlx5_core_dma_dev(dev), size, align, 0); > > I'll try to debug this further but wanted to let you know already in > case you have some ideas. I could imagine that potentially something in the initial default domain conversion somehow interferes with the DMA ops in a way that ends up causing alloc_cmd_page() to fail (maybe calling zpci_dma_init_device() at the wrong point, or too many times?). FWIW I see nothing that would obviously affect dma_pool_create() itself. Robin. > Either way as it doesn't seem to be related > to the DMA API conversion I can sent that out again regardless if you > want, really don't want to miss another cycle. > > Thanks, > Niklas