From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED2DAC27C55 for ; Mon, 10 Jun 2024 16:18:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AD1C6B0092; Mon, 10 Jun 2024 12:18:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 35CC76B0096; Mon, 10 Jun 2024 12:18:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2260F6B0098; Mon, 10 Jun 2024 12:18:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 05C926B0096 for ; Mon, 10 Jun 2024 12:18:39 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EFA0116100A for ; Mon, 10 Jun 2024 16:18:37 +0000 (UTC) X-FDA: 82215487074.21.AAA93D2 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf05.hostedemail.com (Postfix) with ESMTP id 6576210001D for ; Mon, 10 Jun 2024 16:18:34 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=txBNSTf1; spf=pass (imf05.hostedemail.com: domain of leon@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718036316; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MZNAwitC0c5tHXIBqT6sv+PXfqUESdg/lRJ7EWEoqJo=; b=nH5MjTZWlXf1x5KwZIkYTSqp8G0QMg05Pnc8Eerf+54qD1xBWm8f8JIqLpJnjey78XjcJY J9LRT4yfXd3oUzgt+dAMIWofga9otHfq3YK6TJKHcIVO4FBPHQ7Fs0VXYVqtOp2DhiEtwd YBvnrHHK9Fxm2FAXaZDFFgQlJMXvAJ0= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=txBNSTf1; spf=pass (imf05.hostedemail.com: domain of leon@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718036316; a=rsa-sha256; cv=none; b=PKvLiB6b5pghYs0+0vrwXF48m4pUqFlei0b8vt1BTjTe0m4WMwxHfodYQu8RyQqwI1zpeP 3HLiNM5coSc32WTWUGkQ9xPMbshiVuNuqFArweU05hJpBdylN6YhIxtScDSbgw19JNgEvr 1Ed26+iqAT/7gLa2xT/9gA2CDOZT8K4= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id A2277CE16B8; Mon, 10 Jun 2024 16:18:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 270DCC2BBFC; Mon, 10 Jun 2024 16:18:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718036311; bh=uZtlK/h+IV89qKq/D7HEEGiIqU3n+5uR+aMUW6bSTYI=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=txBNSTf1+1t4Xc9gPPh5vQRc5XfFN8nPv3Ro8SB/Yq/j5yEFp8jrN0I+5Vvp5KhOC 31eIcQtv4gSFNYkJWu5tiUYcRsGWbQIFh2b2tiBQYfLYxsJieRJiOTdFhWwTvQJF/M fUzjRO+rL2bVJQVsXagyKG+uCYHiF1TBWgw463NfDzmu1rm2XEkNCph2ZdlBvKMPM0 Gr/XbPplhRTbNTmuXHWyp/dAQvRpoJf5jrDLG/1hP72FEcChZfoVi0qm5BQFx6ewcw vpY5DbRGJh+Zpb2hIoVs4KxbjfSUVKLSS1z59lmFu0jrgvvdpMja9j94fZRFY7GnNT znfhgKqB0z4zQ== Date: Mon, 10 Jun 2024 19:18:26 +0300 From: Leon Romanovsky To: "Zeng, Oak" Cc: Jason Gunthorpe , Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Chaitanya Kulkarni , "Brost, Matthew" , "Hellstrom, Thomas" , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , "Tian, Kevin" , Alex Williamson , =?iso-8859-1?B?Suly9G1l?= Glisse , Andrew Morton , "linux-doc@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "iommu@lists.linux.dev" , "linux-nvme@lists.infradead.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , "Williams, Dan J" , "jack@suse.com" , Zhu Yanjun , "Bommu, Krishnaiah" , "Ghimiray, Himal Prasad" Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps Message-ID: <20240610161826.GA4966@unreal> References: <20240503164239.GB901876@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6576210001D X-Stat-Signature: gnuxybk9qeseyja4f5fbzkz4izpczj6a X-HE-Tag: 1718036314-112218 X-HE-Meta: U2FsdGVkX1/6wZCs+6VePga8wXATO0rNo8HwSxzO39NZsx9yPPI1ydlqsxmJMPYU6nxFvE2oOVvwIIi4HGgcXmnPgRyrJyhHaM33E4Bk+WA7cOwCnTTSegb657NWRqKPUKf2TZL42/UB+DWkAE7YWmK7VTl3scz7nX4EQoAkGBeCtVf9UA9n5A9HtUIoAdEJwus7ghVohngcVpqBNwBoCj/67JngGsMCuqUl9Pq2/HnEnxPAqogszc3GO4ORnrNyzhkrnNkeFwr/mQQ4CHUH6CD4LmRdHNMtAXo9lnOcrG8FJi07hve2xeOpBaRkcxAFHPrk6gnz4nY7bWWV1c/GBs0vHb+gGqCZAXojX0lw0cPrbVkRooe0gp4uSODiKXELl+86LaS+IkzDhwTjqlK1fSwzrhARVyR6TbFaMnNwpDulmrLhjCQ72c5sYjnS8/2CtDuPEh7gdcnhB+M311iZ19IaWZXVfGU7TWRJTSMpSo+1xHptUWWtHyYdXkTTSeGpxMzEqcepenTME6prx/ehLniMyrN9Ny98zr1iL1w4EStmzVjY18ZZVTiWr00b2/VtMRdWncrMwHLI7BZMfq57aLafPMm2UMvdTMJB0Ixsmv/o7nCeHJQWvgZ+LIdrjH5nbSbSD/vpq/HkOuGxZaNXGSKNfIQiizlklNtsarEwe8WBHVlbuX8v9603w6QEiGBwZmxug4gO3mIo0sYHOqbUo/yjVUGiQghdGauoe/ZG+hQPYxceY85QIYnzsF28db994RopvEsuVHLJkhh5bWslyW94Lcrfq6boi4tBgs5H0wIoarGYLZOsT2/9sZ4E34mWRlwxbL1Kt0h92d9uc59WwuER/oTtpcn4+nE1+gujnYcuxPfPfWdT5TTXLWcRiO7n2CUIuOBqQaBFCy25KAX7UiUlIbBIzjf06ILScS/33DMZzS+L4zbCXzvTOnHCetWfTfAfv8PQ6mQJQZDnt5z WIfzAHJM 0XvOmpDuX5Rjqrg6EQlKMyfm753WB288RjBGb5weRKA4Ux+BLmfPMMtnEd14Nx6lDjjsZnBBEnNgAUf6zaejhFHqwrcUfFXY7qW0JnwOgqgq+d+IGP1bTTpVIqMO3BE0TyR4ZVsvpZovaxYYD9i6TiFNH3g5jiIkysGVNGZkwINcMFFMV7AaNBtV6JpxsJnM1AQyGhnCkXi03QwvN5iQsS2UhwC1rcUFWcYkXnpLp3EyWnl2sU2ZW5wcpM2E15DQ6WvY2h/p1kirQFMBAn0dfy4mzQsa4Eq2Kdnfa8JwO3FAQwUzY/LAQ9bVGhLOFjyvdkHbe27U6MISqLk2Jgx2Dvyx8d1Rlm3Uy9/BQnky61kS4WRsm//VCQd7e0kyvtxL03SI3O34+iVC5wiqL9kXCidmFmv6xSFNB2IL7zZFMlROy6OfEKRlqujiIXER5hsagbC6x+/X3dm9fljeq0lLFvW3QvTcpkpG/l0zcaTKD2oZEM0o+62WdWvGfIpjSXPXSz/CbToMVvGHS0MgR++bblKuh/MMrsVfzMjdwAYco1OjxAkysUI+5FtvF5q3q5GG3iokPyNRYtoZ4ecTaV99ziCy2jwNxdbkDZrKm0Txqtp5R0VSVLJP3a202taz8WMb3NceFpXVvle1T/2aymPsX8rCrz56W8HeYiP3ZiaMsK9d55n1t2MxpGGMJW32OuA8ayrw14rKrmJcW6H8lKEn1U3hasPOkLPm6EPsyqwUbjmLLn00nIpDhsn4+kXn5bpUL50U59u6i9OYwf9aJyGI/VnwF0jkFJQRE8/bKEeDU0W2vzRYmHPRwLMXPdUstj4VYgfxyik6VliLen4/VQyjotjXxqlEX9CYQ3nLgpnYARH9p0mVoSxVUNs8TCd9vljog2Pv305/gKZGJ51XCQr9aXSRSISHAymC3llu9QjR8xx7StAI0pThUuuWaAtnjU/KHi79NQo6srRKfWX5YDt9O0XRuZTJ1 BzKmKlcs 06qXUnpmLZam5F6WvasmPwIoS2jlVc43Su/mVjcmG8taTnHzf+fBNDWfrLEHBXSComgjcaTW4yiDu3DaVmdO/YvSfUjpYhLWXHN7m+D6yE0NMKFcTKxN1daDOCPT1uC44V0qh5/Z0ETkhSCN5zxsW5O+Nl5ciqyb7VfVXMJ7mXCG6rkn6dGuKw9TLy0yiSBrpJlVfVkzLHCBkPL1ujaoXA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jun 10, 2024 at 03:12:25PM +0000, Zeng, Oak wrote: > Hi Jason, Leon, > > I come back to this thread to ask a question. Per the discussion in another thread, I have integrated the new dma-mapping API (the first 6 patches of this series) to DRM subsystem. The new API seems fit pretty good to our purpose, better than scatter-gather dma-mapping. So we want to continue work with you to adopt this new API. Sounds great, thanks for the feedback. > > Did you test the new API in RDMA subsystem? This version was tested in our regression tests, but there is a chance that you are hitting flows that were not relevant for RDMA case. > Or this RFC series was just some untested codes sending out to get people's design feedback? RFC was fully tested in VFIO and RDMA paths, but not NVMe patch. > Do you have refined version for us to try? I ask because we are seeing some issues but not sure whether it is caused by the new API. We are debugging but it would be good to also ask at the same time. Yes, as an outcome of the feedback in this thread, I implemented a new version. Unfortunately, there are some personal matters that are preventing from me to send it right away. https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/log/?h=dma-split-v1 There are some differences in the API, but the main idea is the same. This version is not fully tested yet. Thanks > > Cc Himal/Krishna who are also working/testing the new API. > > Thanks, > Oak > > > -----Original Message----- > > From: Jason Gunthorpe > > Sent: Friday, May 3, 2024 12:43 PM > > To: Zeng, Oak > > Cc: leon@kernel.org; Christoph Hellwig ; Robin Murphy > > ; Marek Szyprowski > > ; Joerg Roedel ; Will > > Deacon ; Chaitanya Kulkarni ; > > Brost, Matthew ; Hellstrom, Thomas > > ; Jonathan Corbet ; Jens > > Axboe ; Keith Busch ; Sagi > > Grimberg ; Yishai Hadas ; > > Shameer Kolothum ; Tian, Kevin > > ; Alex Williamson ; > > Jérôme Glisse ; Andrew Morton > foundation.org>; linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org; > > linux-block@vger.kernel.org; linux-rdma@vger.kernel.org; > > iommu@lists.linux.dev; linux-nvme@lists.infradead.org; > > kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche > > ; Damien Le Moal > > ; Amir Goldstein > > ; josef@toxicpanda.com; Martin K. Petersen > > ; daniel@iogearbox.net; Williams, Dan J > > ; jack@suse.com; Leon Romanovsky > > ; Zhu Yanjun > > Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to > > two steps > > > > On Thu, May 02, 2024 at 11:32:55PM +0000, Zeng, Oak wrote: > > > > > > Instead of teaching DMA to know these specific datatypes, let's separate > > > > existing DMA mapping routine to two steps and give an option to > > advanced > > > > callers (subsystems) perform all calculations internally in advance and > > > > map pages later when it is needed. > > > > > > I looked into how this scheme can be applied to DRM subsystem and GPU > > drivers. > > > > > > I figured RDMA can apply this scheme because RDMA can calculate the > > > iova size. Per my limited knowledge of rdma, user can register a > > > memory region (the reg_user_mr vfunc) and memory region's sized is > > > used to pre-allocate iova space. And in the RDMA use case, it seems > > > the user registered region can be very big, e.g., 512MiB or even GiB > > > > In RDMA the iova would be linked to the SVA granual we discussed > > previously. > > > > > In GPU driver, we have a few use cases where we need dma-mapping. Just > > name two: > > > > > > 1) userptr: it is user malloc'ed/mmap'ed memory and registers to gpu > > > (in Intel's driver it is through a vm_bind api, similar to mmap). A > > > userptr can be of any random size, depending on user malloc > > > size. Today we use dma-map-sg for this use case. The down side of > > > our approach is, during userptr invalidation, even if user only > > > munmap partially of an userptr, we invalidate the whole userptr from > > > gpu page table, because there is no way for us to partially > > > dma-unmap the whole sg list. I think we can try your new API in this > > > case. The main benefit of the new approach is the partial munmap > > > case. > > > > Yes, this is one of the main things it will improve. > > > > > We will have to pre-allocate iova for each userptr, and we have many > > > userptrs of random size... So we might be not as efficient as RDMA > > > case where I assume user register a few big memory regions. > > > > You are already doing this. dma_map_sg() does exactly the same IOVA > > allocation under the covers. > > > > > 2) system allocator: it is malloc'ed/mmap'ed memory be used for GPU > > > program directly, without any other extra driver API call. We call > > > this use case system allocator. > > > > > For system allocator, driver have no knowledge of which virtual > > > address range is valid in advance. So when GPU access a > > > malloc'ed/mmap'ed address, we have a page fault. We then look up a > > > CPU vma which contains the fault address. I guess we can use the CPU > > > vma size to allocate the iova space of the same size? > > > > No. You'd follow what we discussed in the other thread. > > > > If you do a full SVA then you'd split your MM space into granuals and > > when a fault hits a granual you'd allocate the IOVA for the whole > > granual. RDMA ODP is using a 512M granual currently. > > > > If you are doing sub ranges then you'd probably allocate the IOVA for > > the well defined sub range (assuming the typical use case isn't huge) > > > > > But there will be a true difficulty to apply your scheme to this use > > > case. It is related to the STICKY flag. As I understand it, the > > > sticky flag is designed for driver to mark "this page/pfn has been > > > populated, no need to re-populate again", roughly...Unlike userptr > > > and RDMA use cases where the backing store of a buffer is always in > > > system memory, in the system allocator use case, the backing store > > > can be changing b/t system memory and GPU's device private > > > memory. Even worse, we have to assume the data migration b/t system > > > and GPU is dynamic. When data is migrated to GPU, we don't need > > > dma-map. And when migration happens to a pfn with STICKY flag, we > > > still need to repopulate this pfn. So you can see, it is not easy to > > > apply this scheme to this use case. At least I can't see an obvious > > > way. > > > > You are already doing this today, you are keeping the sg list around > > until you unmap it. > > > > Instead of keeping the sg list you'd keep a much smaller datastructure > > per-granual. The sticky bit is simply a convient way for ODP to manage > > the smaller data structure, you don't have to use it. > > > > But you do need to keep track of what pages in the granual have been > > DMA mapped - sg list was doing this before. This could be a simple > > bitmap array matching the granual size. > > > > Looking (far) forward we may be able to have a "replace" API that > > allows installing a new page unconditionally regardless of what is > > already there. > > > > Jason