From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F135C54E58 for ; Thu, 21 Mar 2024 22:39:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80CAB6B008C; Thu, 21 Mar 2024 18:39:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BBF76B0092; Thu, 21 Mar 2024 18:39:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6AAB66B0095; Thu, 21 Mar 2024 18:39:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5C5386B008C for ; Thu, 21 Mar 2024 18:39:22 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2255B1C1901 for ; Thu, 21 Mar 2024 22:39:22 +0000 (UTC) X-FDA: 81922513764.03.D986FCA Received: from verein.lst.de (verein.lst.de [213.95.11.211]) by imf25.hostedemail.com (Postfix) with ESMTP id BFAFFA001D for ; Thu, 21 Mar 2024 22:39:19 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of hch@lst.de designates 213.95.11.211 as permitted sender) smtp.mailfrom=hch@lst.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711060760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JGqzUGhdWzxNRZfpPnkopvOYOGGkyEkyp/xSmTZjR+E=; b=YUIeZKMVt5ku/+1mxPAhXq5pJ8rMtxDB8LuiWcXOUsFT6U+mVu6vAEndx5AoIbiEjbPWdE 7OZvLY7u8oAIpr716XyO91h6kaKdSLINEy0D+VePATAK82zDL17BHdgyPKjuHO0VsmroFB V/TOvlqLoeiPbcxJExZj4ran0REJFek= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of hch@lst.de designates 213.95.11.211 as permitted sender) smtp.mailfrom=hch@lst.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711060760; a=rsa-sha256; cv=none; b=AgivFJqLSBlNF7uMPCAAqziLicWLFlPapwOmQSPl57p0w9iqTW3kwePSFWvH5UcnpyjSKm nOYfs5cTpUHCd34RXvklJiRxuyjr09akZMiSL24TVl/4HkWjMhBCdSUeFUfMJVjZJ6m0k8 1cGSJKyL1xCYK7o4UF7tuQOiyUB6J30= Received: by verein.lst.de (Postfix, from userid 2407) id 879B968BEB; Thu, 21 Mar 2024 23:39:11 +0100 (CET) Date: Thu, 21 Mar 2024 23:39:10 +0100 From: Christoph Hellwig To: Jason Gunthorpe Cc: Christoph Hellwig , Leon Romanovsky , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Chaitanya Kulkarni , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?iso-8859-1?B?Suly9G1l?= Glisse , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, linux-mm@kvack.org, Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , Dan Williams , "jack@suse.com" , Zhu Yanjun Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps Message-ID: <20240321223910.GA22663@lst.de> References: <20240306162022.GB28427@lst.de> <20240306174456.GO9225@ziepe.ca> <20240306221400.GA8663@lst.de> <20240307000036.GP9225@ziepe.ca> <20240307150505.GA28978@lst.de> <20240307210116.GQ9225@ziepe.ca> <20240308164920.GA17991@lst.de> <20240308202342.GZ9225@ziepe.ca> <20240309161418.GA27113@lst.de> <20240319153620.GB66976@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240319153620.GB66976@ziepe.ca> User-Agent: Mutt/1.5.17 (2007-11-01) X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: BFAFFA001D X-Stat-Signature: ntkxfmyr1fqfccix3eoyhg3h81chk5fn X-HE-Tag: 1711060759-860248 X-HE-Meta: U2FsdGVkX19HjtOZ5myesbD0Co+Y2VHHmMkK8GsBylW9yh0JWZM4sVxTXrxzHGNgeMkalbtV2X042ZUUG2bQpUmPO+F4ti8zzvDLo51BIgKXQiAgeIAOYEIe8HxK9wk07F2zV2rVcy8vDLpYCXjaRhlluYIfsAK2dDgCbAZE404CFQjuBkHg7KPDESnR4VCA7+PH8JXfqSqADX2TI3MXX4g8QPFAhRY3GMxkuOlDsPYSa/VjrD5GfEyF031Olts267+YWLcK2tarul5DOcFFj5UkPkU65YEKz/UFcIhPsDSXTKOsJ7Sez/6+ZXGVEYQ5VwA6jMtAS2pdzNYD4lJm6v/HTDnjOSPnICx7yHlGHzfLQCOmAyuvEYdlzN+wmnwsL3+QEArWfmwGB8ZyEHyeym86AEpHLhFPhpmKIse++/5IrHod65j+S51uFJZtMhnVNHD6s2A1U1c2amRqV5ryCBjbJAIy/P+h93pYixPvKjopqqA3UcPQEv7tpkTeURONI+rutZphPY4cuza8gpqE7ghFLvCFL1rWGX+Rg+YLg8fSVBgbRIlhVLNT5kjedRqm/xIMKPlnE3aOUE1jGZjZyJKb0CRWR38ns0hfuvtAOy7TbwPJiACOTjtmoXzYV3unvq78Vr8OYRt53LUKbZO4gRAtLMw3QyBdY48GVzG/eI5ek9ly0VUsnfHR7qlKCdl5JWWy9vvTkEbXz9sG2NzTIeLB0dV2+AQSexNNv08CwqUVrtXqhmfWQgDtVB6mRGXYrCbptkeClClOSvtAUsyn4jApwbeKN9bEQDASWVbJFIYeeiuIjSq8TruWMqffWLXLsP5MY0l35QpboFZ8s6EswVC6tVylzBBE0fFIi5DVv29KnRrdf3XM2hoxm8vjf8lTn9N9NyWxPR0lofzz12N9XQaUhQ2NjUO1IGca7lFxbmwm5xhSTCnuKoiEsP0N03RkRVKXRxyELwYujUAS5pA TqST98rK /4EhtDMpxyeu+DdkMMR/oQVwqBWDkTNU6oJgu9knVv8dzNEcjSI3it6Kgr+fTACIMFpJFIYRhGHOIAfc4my96iFymxg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Mar 19, 2024 at 12:36:20PM -0300, Jason Gunthorpe wrote: > I kind of understand your thinking on the DMA side, but I don't see > how this is good for users of the API beyond BIO. > > How will this make RDMA better? We have one MR, the MR has pages, the > HW doesn't care about the SW distinction of p2p, swiotlb, direct, > encrypted, iommu, etc. It needs to create one HW page list for > whatever user VA range was given. Well, the hardware (as in the PCIe card) never cares. But the setup path for the IOMMU does, and something in the OS needs to know about it. So unless we want to stash away a 'is this P2P' flag in every page / SG entry / bvec, or a do a lookup to find that out for each of them we need to manage chunks at these boundaries. And that's what I'm proposing. > Or worse, whatever thing is inside a DMABUF from a DRM > driver. DMABUF's can have a (dynamic!) mixture of P2P and regular > AFAIK based on the GPU's migration behavior. And that's fine. We just need to track it efficiently. > > Or triple worse, ODP can dynamically change on a page by page basis > the type depending on what hmm_range_fault() returns. Same. If this changes all the time you need to track it. And we should find a way to shared the code if we have multiple users for it. But most DMA API consumers will never see P2P, and when they see it it will be static. So don't build the DMA API to automically do the (not exactly super cheap) checks and add complexity for it. > So I take it as a requirement that RDMA MUST make single MR's out of a > hodgepodge of page types. RDMA MRs cannot be split. Multiple MR's are > not a functional replacement for a single MR. But MRs consolidate multiple dma addresses anyway. > Go back to the start of what are we trying to do here: > 1) Make a DMA API that can support hmm_range_fault() users in a > sensible and performant way > 2) Make a DMA API that can support RDMA MR's backed by DMABUF's, and > user VA's without restriction > 3) Allow to remove scatterlist from BIO paths > 4) Provide a DMABUF API that is not scatterlist that can feed into > the new DMA API - again supporting DMABUF's hodgepodge of types. > > I'd like to do all of these things. I know 3 is your highest priority, > but it is my lowest :) Well, 3 an 4. And 3 is not just limited to bio, but all the other pointless scatterlist uses.