From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0755CC5475B for ; Fri, 8 Mar 2024 16:49:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73DDD6B03BC; Fri, 8 Mar 2024 11:49:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EE8D6B03BD; Fri, 8 Mar 2024 11:49:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58F676B03BE; Fri, 8 Mar 2024 11:49:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 42AAF6B03BC for ; Fri, 8 Mar 2024 11:49:30 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DF621120466 for ; Fri, 8 Mar 2024 16:49:29 +0000 (UTC) X-FDA: 81874457658.02.4D52667 Received: from verein.lst.de (verein.lst.de [213.95.11.211]) by imf28.hostedemail.com (Postfix) with ESMTP id E2C8BC0008 for ; Fri, 8 Mar 2024 16:49:27 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of hch@lst.de designates 213.95.11.211 as permitted sender) smtp.mailfrom=hch@lst.de; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709916568; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FJdRNrfaUk+clLIWSPTwanrknh7+Ussqz1t90RvWT2o=; b=LbcIl4CxV/6BUCcllmVhdtkcf77DfmvzpH2O8eHSfNq4sj52n4YAKNukeMaG5dBoTM7Adm e5WAb9yloUJNuZv/dBg16KmLdCrCS4efBjb/1VpwjnD0OVsuojurP8C6OQE9/qSd9CyeHm ODVYvQnyD4I1aPAdW020FS9aYJVNdRQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of hch@lst.de designates 213.95.11.211 as permitted sender) smtp.mailfrom=hch@lst.de; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709916568; a=rsa-sha256; cv=none; b=QsDTSn8BYGh0X3Gm1HgRMyAcagAygQd/K7pPMOzMmI3MWQJGZVfx4qbbQxWyr0UsKCyzAX /gFezW0USolo/2wAW3cUtV24THCcCiqNfDTPVMfcfuu9W9On16adn4xC0KnH0HL7PbBNHt RYj7KZA/GE+pk3zpASc2zKO4X0aNSqQ= Received: by verein.lst.de (Postfix, from userid 2407) id E414168BEB; Fri, 8 Mar 2024 17:49:20 +0100 (CET) Date: Fri, 8 Mar 2024 17:49:20 +0100 From: Christoph Hellwig To: Jason Gunthorpe Cc: Christoph Hellwig , Leon Romanovsky , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Chaitanya Kulkarni , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?iso-8859-1?B?Suly9G1l?= Glisse , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, linux-mm@kvack.org, Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , Dan Williams , "jack@suse.com" , Zhu Yanjun Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps Message-ID: <20240308164920.GA17991@lst.de> References: <47afacda-3023-4eb7-b227-5f725c3187c2@arm.com> <20240305122935.GB36868@unreal> <20240306144416.GB19711@lst.de> <20240306154328.GM9225@ziepe.ca> <20240306162022.GB28427@lst.de> <20240306174456.GO9225@ziepe.ca> <20240306221400.GA8663@lst.de> <20240307000036.GP9225@ziepe.ca> <20240307150505.GA28978@lst.de> <20240307210116.GQ9225@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240307210116.GQ9225@ziepe.ca> User-Agent: Mutt/1.5.17 (2007-11-01) X-Rspamd-Queue-Id: E2C8BC0008 X-Rspam-User: X-Stat-Signature: sp3dw8itcc1ied8ffw1kusdx9hbu3tmx X-Rspamd-Server: rspam01 X-HE-Tag: 1709916567-480238 X-HE-Meta: U2FsdGVkX1/xRP1ZFFfJnbRgb/mduemk6vWHrZ9wsmvJdHJNE7o6273OROYYibmTk/EvAO9mT8Gs4JQP01X6zrAbk8yzUh73OP9Nlsxc2B5kWErGMbCeSySBQgIzaGNjiXr0fQJcOeOejdhzgNbSeEAzFTYap2P4JMLUxAwochdU1vmTNJoqr0DytfY9So+2n+0JDHB+BGNIaPj/mvokNJ7PcaxfmGHMtAXDzcuvZeNV6V0qWIg2ox/jfH/XWXc4M7hASu7pRLi9tr8/sc1zj/baGTNHxRHy8RpJUtYiZwEM9r0++wVAUbUCTF265JH5leFXI0LJHQ/4SK8yzWpVs8cKJNtRXFNnHeoUyN/kWutBBon/OOG7zQfb2FhC5w5IqdJnHR4776uf1+AZCTLNWwj1YYA1XHc1y/PfwhNXklQv2l8fShL3xpBXnj0ygob12n2GdSsoRbbfGSRtrRFhsn+YAQfnHFHTxrUDbWoKoZORZloqc4qmHod0XDVScDlMTRn9wAocKIiB5K6sLOWbNVsKSzG+BBonqrztPGXSjeapA/FtJYKk+/Za3HLv/wMAWgfm8aXWpGLvlw7uYL0oJAfNQjn6Xth/eClZ3VWwqDQD1PBGt8Jno6ViMDjP32IA0Mvy61CVDIwS45y5N/2O4W8vdmbkYomnJ4LoAEeOjWo01jtShRhp4SqTy//CvfKQDrXE/Bqp/U0dnIL35VDrWcZvai2jBsB2ACtTyFZIK7H5zyDF2g72c0jYFJPfv18A61g4i/T2bj3gE0F0/o5GNdQKzd1LurRB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Mar 07, 2024 at 05:01:16PM -0400, Jason Gunthorpe wrote: > > > > It's just kinda hard to do. For aligned IOMMU mapping you'd only > > have one dma_addr_t mappings (or maybe a few if P2P regions are > > involved), so this probably doesn't matter. For direct mappings > > you'd have a few, but maybe the better answer is to use THP > > more aggressively and reduce the number of segments. > > Right, those things have all been done. 100GB of huge pages is still > using a fair amount of memory for storing dma_addr_t's. > > It is hard to do perfectly, but I think it is not so bad if we focus > on the direct only case and simple systems that can exclude swiotlb > early on. Even with direct mappings only we still need to take care of cache synchronization. > > If all flows includes multiple non-coalesced regions that just makes > > things very complicated, and that's exactly what I'd want to avoid. > > I don't see how to avoid it unless we say RDMA shouldn't use this API, > which is kind of the whole point from my perspective.. The DMA API callers really need to know what is P2P or not for various reasons. And they should generally have that information available, either from pin_user_pages that needs to special case it or from the in-kernel I/O submitter that build it from P2P and normal memory. > Sure, 3 SGL entries is fine, that isn't what I'm pointing at > > I'm saying that today if you give such a scatterlist to dma_map_sg() > it scans it and computes the IOVA space need, allocates one IOVA > space, then subdivides that single space up into the 3 HW SGLs you > show. > > If you don't preserve that then we are calling, 4k at a time, a > dma_map_page() which is not anywhere close to the same outcome as what > dma_map_sg did. I may not get contiguous IOVA, I may not get 3 SGLs, > and we call into the IOVA allocator a huge number of times. Again, your callers must know what is a P2P region and what is not. I don't think it is a hard burdern to do mappings at that granularity, and we can encapsulate this in nice helpes for say the block layer and pin_user_pages callers to start. > > It needs to work following the same basic structure of dma_map_sg, > unfolding that logic into helpers so that the driver can provide > the data structure: > > - Scan the io ranges and figure out how much IOVA needed > (dma_io_summarize_range) That is in general a function of the upper layer and not the DMA code. > - Allocate the IOVA (dma_init_io) And this step is only needed for the iommu case. > > That's why I really just want 2 cases. If the caller guarantees the > > range is coalescable and there is an IOMMU use the iommu-API like > > API, else just iter over map_single/page. > > But how does the caller even know if it is coalescable? Other than the > trivial case of a single CPU range, that is a complicated detail based > on what pages are inside the range combined with the capability of the > device doing DMA. I don't see a simple way for the caller to figure > this out. You need to sweep every page and collect some information on > it. The above is to abstract that detail. dma_get_merge_boundary already provides this information in terms of the device capabilities. And given that the callers knows what is P2P and what is not we have all the information that is needed.