From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E11EC43334 for ; Thu, 30 Jun 2022 21:22:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E64738E0002; Thu, 30 Jun 2022 17:22:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E14588E0001; Thu, 30 Jun 2022 17:22:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDBAB8E0002; Thu, 30 Jun 2022 17:22:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BEE3B8E0001 for ; Thu, 30 Jun 2022 17:22:06 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8CBFE34100 for ; Thu, 30 Jun 2022 21:22:06 +0000 (UTC) X-FDA: 79636175052.07.76C45EF Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf18.hostedemail.com (Postfix) with ESMTP id DCFF71C0043 for ; Thu, 30 Jun 2022 21:22:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:In-Reply-To:From:References:Cc:To: MIME-Version:Date:Message-ID:content-disposition; bh=lcE0RG7zHjzAvuVO13dlZQZgYUdkoKNtHGwpbVB8WSY=; b=VJXKRmwbrfTeru4GStEgHnOFcn 0clAFOb+uSboDe5lSH6bYXIDyM3A7FV1m1Gv40/1WK+WmKuyU4GcovIZln8u/dOZNroWCwnZTCILD UngbBjeNR9R/2lSaYm9F4nCbknUpbtSWveSI8iaLewadz2cRFT6Ts07ys3AhLJ5hULu5Qjjr9GbyR sj5HECu0OR4z2h2JvEHhTUp5YwFApvTKp5tE7hTFr3k6HO4YBl53QWu7GFrHxx2lVEGROAalhImzW soOQzHUJ3ws78RKLCrt/8AG1InmS7nqUSlYNP+aV7cYdnXQFj+ilO5CQMJjbF5y0acg1lAZ0SkjVv HipxMV4Q==; Received: from s0106a84e3fe8c3f3.cg.shawcable.net ([24.64.144.200] helo=[192.168.0.10]) by ale.deltatee.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1o71bm-003Olq-2k; Thu, 30 Jun 2022 15:21:55 -0600 Message-ID: Date: Thu, 30 Jun 2022 15:21:46 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Content-Language: en-CA To: Robin Murphy , linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Minturn Dave B , Martin Oliveira , Ralph Campbell , Jason Gunthorpe , John Hubbard , Dave Hansen , Matthew Wilcox , =?UTF-8?Q?Christian_K=c3=b6nig?= , Jason Gunthorpe , Chaitanya Kulkarni , Jason Ekstrand , Daniel Vetter , Bjorn Helgaas , Dan Williams , Stephen Bates , Ira Weiny , Christoph Hellwig , Xiong Jianxin References: <20220615161233.17527-1-logang@deltatee.com> <20220615161233.17527-9-logang@deltatee.com> <7f0673e1-433b-65fb-1d2b-c3e4adeebf87@arm.com> <626de61d-e85e-bc9f-9e3d-836a408c859f@deltatee.com> From: Logan Gunthorpe In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SA-Exim-Connect-IP: 24.64.144.200 X-SA-Exim-Rcpt-To: robin.murphy@arm.com, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, dave.b.minturn@intel.com, martin.oliveira@eideticom.com, rcampbell@nvidia.com, jgg@nvidia.com, jhubbard@nvidia.com, dave.hansen@linux.intel.com, willy@infradead.org, christian.koenig@amd.com, jgg@ziepe.ca, ckulkarnilinux@gmail.com, jason@jlekstrand.net, daniel.vetter@ffwll.ch, helgaas@kernel.org, dan.j.williams@intel.com, sbates@raithlin.com, ira.weiny@intel.com, hch@lst.de, jianxin.xiong@intel.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v7 08/21] iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656624126; a=rsa-sha256; cv=none; b=A+iCqaFqRZRGAKTKIwsbQG2uYW7QMPpmQ064887qxZxoJNzA0xBaYfXM7fl8Py4JYm/YTo CKs1PKD02p0uBx0rtvPTzgNqCYpolqQvxYroT/EM1QFbzkN6iRgCWcLwm7M433WFp3gJ6z khH1dSwTIARi09jog+FQvMHeuMJAgz0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=VJXKRmwb; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf18.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656624126; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lcE0RG7zHjzAvuVO13dlZQZgYUdkoKNtHGwpbVB8WSY=; b=PgKbWwGFZvKpFGlUHYTWvmOAxZfzqLOOpQfLz3owUBCamlnzUFl23cl4votLU9kxnvlNHX BbO83j4elojhcJSKCeuWGmHyyVahLuaQl0RSJ9Ue3vK3lCqPDHlEk/ejdKA8zMFIxrfA3W 1BfQRprjw5XHAsAHwY0zKsoBrk6I4YY= X-Rspamd-Queue-Id: DCFF71C0043 X-Rspam-User: Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=VJXKRmwb; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf18.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com X-Rspamd-Server: rspam02 X-Stat-Signature: e5s9a7xsq3mge7et4hqtfrxrg6b5bqz4 X-HE-Tag: 1656624125-150701 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2022-06-30 08:56, Robin Murphy wrote: > On 2022-06-29 23:41, Logan Gunthorpe wrote: >> >> >> On 2022-06-29 13:15, Robin Murphy wrote: >>> On 2022-06-29 16:57, Logan Gunthorpe wrote: >>>> >>>> >>>> >>>> On 2022-06-29 06:07, Robin Murphy wrote: >>>>> On 2022-06-15 17:12, Logan Gunthorpe wrote: >>>>>> When a PCI P2PDMA page is seen, set the IOVA length of the segment >>>>>> to zero so that it is not mapped into the IOVA. Then, in >>>>>> finalise_sg(), >>>>>> apply the appropriate bus address to the segment. The IOVA is not >>>>>> created if the scatterlist only consists of P2PDMA pages. >>>>>> >>>>>> A P2PDMA page may have three possible outcomes when being mapped: >>>>>>      1) If the data path between the two devices doesn't go through >>>>>>         the root port, then it should be mapped with a PCI bus >>>>>> address >>>>>>      2) If the data path goes through the host bridge, it should be >>>>>> mapped >>>>>>         normally with an IOMMU IOVA. >>>>>>      3) It is not possible for the two devices to communicate and >>>>>> thus >>>>>>         the mapping operation should fail (and it will return >>>>>> -EREMOTEIO). >>>>>> >>>>>> Similar to dma-direct, the sg_dma_mark_pci_p2pdma() flag is used to >>>>>> indicate bus address segments. On unmap, P2PDMA segments are skipped >>>>>> over when determining the start and end IOVA addresses. >>>>>> >>>>>> With this change, the flags variable in the dma_map_ops is set to >>>>>> DMA_F_PCI_P2PDMA_SUPPORTED to indicate support for P2PDMA pages. >>>>>> >>>>>> Signed-off-by: Logan Gunthorpe >>>>>> Reviewed-by: Jason Gunthorpe >>>>>> --- >>>>>>     drivers/iommu/dma-iommu.c | 68 >>>>>> +++++++++++++++++++++++++++++++++++---- >>>>>>     1 file changed, 61 insertions(+), 7 deletions(-) >>>>>> >>>>>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c >>>>>> index f90251572a5d..b01ca0c6a7ab 100644 >>>>>> --- a/drivers/iommu/dma-iommu.c >>>>>> +++ b/drivers/iommu/dma-iommu.c >>>>>> @@ -21,6 +21,7 @@ >>>>>>     #include >>>>>>     #include >>>>>>     #include >>>>>> +#include >>>>>>     #include >>>>>>     #include >>>>>>     #include >>>>>> @@ -1062,6 +1063,16 @@ static int __finalise_sg(struct device *dev, >>>>>> struct scatterlist *sg, int nents, >>>>>>             sg_dma_address(s) = DMA_MAPPING_ERROR; >>>>>>             sg_dma_len(s) = 0; >>>>>>     +        if (is_pci_p2pdma_page(sg_page(s)) && !s_iova_len) { >>>>> >>>>> Logically, should we not be able to use sg_is_dma_bus_address() >>>>> here? I >>>>> think it should be feasible, and simpler, to prepare the p2p segments >>>>> up-front, such that at this point all we need to do is restore the >>>>> original length (if even that, see below). >>>> >>>> Per my previous email, no, because sg_is_dma_bus_address() is not set >>>> yet and not meant to tell you something about the page. That flag will >>>> be set below by pci_p2pdma_map_bus_segment() and then checkd in >>>> iommu_dma_unmap_sg() to determine if the dma_address in the segment >>>> needs to be unmapped. >>> >>> I know it's not set yet as-is; I'm suggesting things should be >>> restructured so that it *would be*. In the logical design of this code, >>> the DMA addresses are effectively determined in iommu_dma_map_sg(), and >>> __finalise_sg() merely converts them from a relative to an absolute form >>> (along with undoing the other trickery). Thus the call to >>> pci_p2pdma_map_bus_segment() absolutely belongs in the main >>> iommu_map_sg() loop. >> >> I don't see how that can work: __finalise_sg() does more than convert >> them from relative to absolute, it also figures out which SG entry will >> contain which dma_address segment. Which segment a P2PDMA address needs >> to be programmed into depends on the how 'cur' is calculated which in >> turn depends on things like seg_mask and max_len. This calculation is >> not done in iommu_dma_map_sg() so I don't see how there's any hope of >> assigning the bus address for the P2P segments in that function. >> >> If there's a way to restructure things so that's possible that I'm not >> seeing, I'm open to it but it's certainly not immediately obvious. > > Huh? It's still virtually the same thing; iommu_dma_map_sg() calls > pci_p2pdma_map_bus_segment(s) and sets s->length to 0 if > PCI_P2PDMA_MAP_BUS_ADDR, then __finalise_sg() can use > sg_is_dma_bus_address(s) in place of is_pci_p2pdma_page(sg_page(s)), and > just propagate the DMA address and original length from s to cur. > > Here you've written a patch which looks to correctly interrupt any > ongoing concatenation state and convey some data from the given input > segment to the appropriate output segment, so I'm baffled by why you'd > think you couldn't do what you've already done. Ah, I understand now, thanks for the patience. It took me a couple of read throughs before I got it, but I figured it out and now have a working implementation that looks really nice. It's a big improvement not needing the two different P2PDMA helpers. I'll send a v8 of just the first 13 patches next week after a bit more testing. I've put a draft git branch here if you want to look at it before that: https://github.com/sbates130272/linux-p2pmem p2pdma_map_v8 Thanks! Logan