Re: [PATCH v12 0/6] iommu/dma: s390 DMA API conversion and optimized IOTLB flushing

From: Niklas Schnelle <schnelle@linux.ibm.com>
To: Joerg Roedel <joro@8bytes.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>,
	Matthew Rosato <mjrosato@linux.ibm.com>,
	Will Deacon <will@kernel.org>,
	Wenjia Zhang <wenjia@linux.ibm.com>,
	Robin Murphy <robin.murphy@arm.com>,
	Gerd Bayer <gbayer@linux.ibm.com>,
	Julian Ruess <julianr@linux.ibm.com>,
	Pierre Morel <pmorel@linux.ibm.com>,
	Alexandra Winter <wintera@linux.ibm.com>,
	Heiko Carstens <hca@linux.ibm.com>,
	Vasily Gorbik <gor@linux.ibm.com>,
	Alexander Gordeev <agordeev@linux.ibm.com>,
	Christian Borntraeger <borntraeger@linux.ibm.com>,
	Sven Schnelle <svens@linux.ibm.com>,
	Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
	Hector Martin <marcan@marcan.st>, Sven Peter <sven@svenpeter.dev>,
	Alyssa Rosenzweig <alyssa@rosenzweig.io>,
	David Woodhouse <dwmw2@infradead.org>,
	Lu Baolu <baolu.lu@linux.intel.com>,
	Andy Gross <agross@kernel.org>,
	Bjorn Andersson <andersson@kernel.org>,
	Konrad Dybcio <konrad.dybcio@linaro.org>,
	Yong Wu <yong.wu@mediatek.com>,
	Matthias Brugger <matthias.bgg@gmail.com>,
	AngeloGioacchino Del Regno
	<angelogioacchino.delregno@collabora.com>,
	Gerald Schaefer <gerald.schaefer@linux.ibm.com>,
	Orson Zhai <orsonzhai@gmail.com>,
	Baolin Wang <baolin.wang@linux.alibaba.com>,
	Chunyan Zhang <zhang.lyra@gmail.com>,
	Chen-Yu Tsai <wens@csie.org>,
	Jernej Skrabec <jernej.skrabec@gmail.com>,
	Samuel Holland <samuel@sholland.org>,
	Thierry Reding <thierry.reding@gmail.com>,
	Krishna Reddy <vdumpa@nvidia.com>,
	Jonathan Hunter <jonathanh@nvidia.com>,
	Jonathan Corbet <corbet@lwn.net>,
	linux-s390@vger.kernel.org, netdev@vger.kernel.org,
	linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
	asahi@lists.linux.dev, linux-arm-kernel@lists.infradead.org,
	linux-arm-msm@vger.kernel.org,
	linux-mediatek@lists.infradead.org, linux-sunxi@lists.linux.dev,
	linux-tegra@vger.kernel.org, linux-doc@vger.kernel.org
Subject: Re: [PATCH v12 0/6] iommu/dma: s390 DMA API conversion and optimized IOTLB flushing
Date: Wed, 27 Sep 2023 15:20:07 +0200	[thread overview]
Message-ID: <e1efbbd827e34800bd7fb0ea687645cc6c65e1ab.camel@linux.ibm.com> (raw)
In-Reply-To: <b06a14de270a63050b0d027c24b333dba25001a4.camel@linux.ibm.com>

On Wed, 2023-09-27 at 13:24 +0200, Niklas Schnelle wrote:
> On Wed, 2023-09-27 at 11:55 +0200, Joerg Roedel wrote:
> > Hi Niklas,
> > 
> > On Wed, Sep 27, 2023 at 10:55:23AM +0200, Niklas Schnelle wrote:
> > > The problem is that something seems to  be broken in the iommu/core
> > > branch. Regardless of whether I have my DMA API conversion on top or
> > > with the base iommu/core branch I can not use ConnectX-4 VFs.
> > 
> > Have you already tried to bisect the issue in the iommu/core branch?
> > The result might sched some light on the issue.
> > 
> > Regards,
> > 
> > 	Joerg
> 
> Hi Joerg,
> 
> Working on it, somehow I must have messed up earlier. It now looks like
> it might in fact be caused by my DMA API conversion rebase and the
> "s390/pci: Use dma-iommu layer" commit. Maybe there is some interaction
> with Jason's patches that I haven't thought about. So sorry for any
> wrong blame.
> 
> Thanks,
> Niklas

Hi,

I tracked the problem down from mlx5_core's alloc_cmd_page() via
dma_alloc_coherent(), ops->alloc, iommu_dma_alloc_remap(), and
__iommu_dma_alloc_noncontiguous() to a failed iommu_dma_alloc_iova().
The allocation here is for 4K so nothing crazy.

On second look I also noticed:

nvme 2007:00:00.0: Using 42-bit DMA addresses

for the NVMe that is working. The problem here seems to be that we set
iommu_dma_forcedac = true in s390_iommu_probe_finalize() because we
have currently have a reserved region over the first 4 GiB anyway so
will always use IOVAs larger than that. That however is too late since
iommu_dma_set_pci_32bit_workaround() is already checked in
__iommu_probe_device() which is called just before ops-
>probe_finalize(). So I moved setting iommu_dma_forcedac = true to
zpci_init_iommu() and that gets rid of the notice for the NVMe but I
still get a failure of iommu_dma_alloc_iova() in
__iommu_dma_alloc_noncontiguous(). So I'll keep digging.

Thanks,
Niklas

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel