From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3487FC27C77 for ; Tue, 11 Jun 2024 15:45:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BCA056B00AD; Tue, 11 Jun 2024 11:45:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B7A9F6B00AF; Tue, 11 Jun 2024 11:45:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1A256B00B5; Tue, 11 Jun 2024 11:45:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 852A66B00AD for ; Tue, 11 Jun 2024 11:45:23 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id ED84512064D for ; Tue, 11 Jun 2024 15:45:22 +0000 (UTC) X-FDA: 82219032084.05.8460984 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf12.hostedemail.com (Postfix) with ESMTP id 1C1FF40022 for ; Tue, 11 Jun 2024 15:45:20 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Em6r9Li6; spf=pass (imf12.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718120721; a=rsa-sha256; cv=none; b=3gxWMUJigmUDRhsH8EeB4hRdIgmvimUoGWAW+NxSHReBR07h4mlu3hZ8CfVBMHd1jVlvuT 1Jm7/Fi4i9SIpg4oAl6yZNPOo3FA3mii60rlKNWHyNqUrIhrH2K6jMqUGWOLTm4bxeqPP/ gTEAAtROtHy8DFYOmxgDiA8zKWjsWlg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Em6r9Li6; spf=pass (imf12.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718120721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oc2eqbMXkAVLIaVnQZ8lbsbNYnsCeobEtpDB3bTLDUU=; b=7ad1lWbJF/gO4/Jrnf53OCoDM4kHaV+iWuccSGHjTDWp9yXkr14THLiG5UC9v+nJvTz588 Nwjb8gvZjbPBT4Sw2cbNvKRaZ6dLwMvuvJS8oTscMXROf4+d0aGBFMPi5kEk+vIpySzHjX +hkiti3EVtukgRFCVq/DrNksPTcEMu4= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1FC1860F00; Tue, 11 Jun 2024 15:45:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 06EE2C2BD10; Tue, 11 Jun 2024 15:45:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718120719; bh=alWpFRyGJPPxpA/qvEhIVLYBDfG5gKLw2O2vjFdIXBU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Em6r9Li64MS4dfA1Fm6Omrjb4w5oyfw2XsNlofj9qeCLh5G5cxfGwuZ4yQPTUgZzW 6kxO2WwG1YbfeA57p9rgmFg6DOXF7B/o2lriNTV968OIdIJj6h+vJ28mFcLCklf+8k loMrtXfftGiWNK48qosW/nixEPYc2CM6BDV0pD+OBtk1iB7sMjJs6yl/UgKehavFC0 ucx5PHa+Nq4Qc5SPa5Zz1cwEjWeiDWr0yVMVZnqrufT33mec/2ulkMu2y2JlMDYEFe LOKNY2IV8maq2WwEhsDrM6P7s8EXMvED5vtpoqf+uru1kkwcRDd4cMFZ+KsLRjANr5 5MbGSb9lFaR2g== Date: Tue, 11 Jun 2024 18:45:15 +0300 From: Leon Romanovsky To: "Zeng, Oak" Cc: Jason Gunthorpe , Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Chaitanya Kulkarni , "Brost, Matthew" , "Hellstrom, Thomas" , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , "Tian, Kevin" , Alex Williamson , =?iso-8859-1?B?Suly9G1l?= Glisse , Andrew Morton , "linux-doc@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "iommu@lists.linux.dev" , "linux-nvme@lists.infradead.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , "Williams, Dan J" , "jack@suse.com" , Zhu Yanjun , "Bommu, Krishnaiah" , "Ghimiray, Himal Prasad" Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps Message-ID: <20240611154515.GC4966@unreal> References: <20240503164239.GB901876@ziepe.ca> <20240610161826.GA4966@unreal> <20240610172501.GJ791043@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Stat-Signature: 7o7w6pwoxo788mnrrdxcs4c8i5qzp7ud X-Rspamd-Queue-Id: 1C1FF40022 X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1718120720-451947 X-HE-Meta: U2FsdGVkX18RTOZuvE9nViwEE7o/jHlBXUs06AJl9tJ6jSOondfKsY3Gt+ihP+5WtG4kCkTooup0+2LUAxiXa+f9IVou++PwG2nHCz4nHc84OFPBD+b9cpBqM5tlFznidpOca44Vve9WiI/hmVneZAMUjMHoRXlWQoRSpDAu149/c+H/a8nNYVwElfnAgaXZppDiztIEuLknM/niPHh2wqMgskCZyE9fRevtMAJub7Wj611UMey1fboB72Ov/H1bXB2dD7vhjEhw0uLjY76Wamj500vmmG6SH7og+johWnWbDqPG7ttxl5ekKw7KL2+i9qKmKd1rAcEiKEWCODKQGzeu79SO0cyGClIKYtqjnIL7nqsiOgu5sofgDStVHzqJs7Z75rUpLKQrkV1rtC45kBwu9loiTeg5ty9MdznaPYsb+i+VOsfsyw8jrzSoGIaD1y32+WvsdqAc9z+obmPasGT3crRaKlu5Tcsj2XZqbIq2EYXdu4RUA2HLG3TKUdUR2fPbBxtuGl4YTy4lPtB8YbcEu1wgbq9LEKW7Kd0T78kjmyJkt1pooTrL+kkz/7PfR1kbJDspqCnx0PQTF/A+6AsyhGx0zZJFG7Dok4vuLpSyZ3TrbaMyFNP1rwpkonB0OMWYWGaP9KRS05DfEIVL0YEZ5Xq6pmvdFqQ0quxhK7nO2O13jvIEr/J4KVjr+iCHgXHS1bO2k2jKbo/E3rxpnjOpzWgcamE3WRHRELAiAspQD/eHtMr1cx2DlIfuW7QEk6PaBoclPU09Y9Zlkhp7HeOQmAvYvgxeDLtY/PH5gaaH9SfVE/1jk/60d6UwKm16sbsbNsQeJwuyjNFQ0ecwTNUnXxfDx7D0nDgBSjwKFtnncOaJdI2+F51hyKS0qaXsTmQEWymvh8HE48Wa2S5BP0WxtxvAJgvNOiad+ezar2uivh1dcZVCKi7Gz0EHO89uOIqmh1chjVr4ckoi1H7 sI4Zv/U1 41PCQSBFFa4pC5NTEXg7Fa80zKhjejjAdUTH4n/DjipFTBSiXPzL8Z1QIFNDOhswfMrd6yUeiopHXY2xl4scwVb2JKmh8SOUbvBFoXo91NRUDl+WnESKzfWdr9+jMttkUBUIqC90K5IU6IsS3gBc7siWe4t44UKnqOzWbCneLOST+dsngzY/8OSOsVUIcnmislOYWnB9k4YhGq4pZNIXaenl5KbBPINo+Df3IrafAp8qSE9FS1D4KVr5gOokdHtFROlKMnFox3UYRV9bymJxamW4YA4wQ52J/oIc445qow9muwoXgP0pb5WXCp5WASqOt7v81cRzOIx2SOWxb4VEQYzcAZAV2cfM/14WU5mC0itarA4h1c+7puLmVHVuQZyP9jzDAHpF5WpVngP2KKE7zFO3kyZeenTLyB/lkPIZa+QVDrbwBv8jwwrHvVCzaeDqMnvgxWgjoS3Db9bOdTMLTg9H4jrD909CjUzJC1Mou8RtsETQ+4fxoUfmv49uEJWc97rD3uzP5x8Knd1rLUZ9fV0sA+O1z+wrfB2Y2c7118t6Uy3N/2KqQOzYlVTv4dZJKSuQJJxmepm67w546TKhw77ij30LFaQfaEA13SPu8SSKPgSkLXQBsbNcbzrrVphRCuMmo05ap1c6ttQlws9cpEHQxTVw3+mdEVhCRHyUb5Hwk79YKyKcpsYT03hWMjcSlh7LLXw+VtsZl1zoEYKmbGw0h+H8PuKJoPtlcTzohf1Gzyv84Q7TVpyDxplLkFRp3P7fed7BC9N9QtwjrYpaY7P5w11e9WHJY9TblNgEL0LucGktJjtUGD4lh276aDzfr0P2r++5CRy1ROyC8ZgGBrAzq20ssTXg++EoEM32p0MlWxhfHViqy9TdJGsvXIYI+thmEZNUrp3+XKcmUst66MVv/EAHZGPU0nruONiwECViGz+WlDBDrL//B+TZlzC1Cps5yLDJJUV1OZWfQyN/y4TG+bUrB wHcM2Gqf dbcR3Q0+dVj1q708vS4/q6TsAer4sR8lV9zR6luQ4HEVii2GlTdfv3J7VWOrFXUTrqlIUWMKCCfGnd3A16MOHh2B0IxvSYgBNQMVFL64R2AzU9r+vXpr8ASP7/PO2hyrWGJQMl4tuR1y5FISfaSnAed/ZZVuWAsiIM6NBQ4rdHMxVLCcB0TG0XCvZ3B1D84hRqxIsVSWD0GhvrQ0DjBeIA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jun 10, 2024 at 09:28:04PM +0000, Zeng, Oak wrote: > Hi Jason, Leon, > > I was able to fix the issue from my side. Things work fine now. I got two questions though: > > 1) The value returned from dma_link_range function is not contiguous, see below print. The "linked pa" is the function return. > I think dma_map_sgtable API would return some contiguous dma address. Is the dma-map_sgtable api is more efficient regarding the iommu page table? i.e., try to use bigger page size, such as use 2M page size when it is possible. With your new API, does it also have such consideration? I vaguely remembered Jason mentioned such thing, but my print below doesn't look like so. Maybe I need to test bigger range (only 16 pages range in the test of below printing). Comment? My API gives you the flexibility to use any page size you want. You can use 2M pages instead of 4K pages. The API doesn't enforce any page size. > > [17584.665126] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, linked pa = 18ef3f000 > [17584.665146] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, linked pa = 190d00000 > [17584.665150] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, linked pa = 190024000 > [17584.665153] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, linked pa = 178e89000 > > 2) in the comment of dma_link_range function, it is said: " @dma_offset needs to be advanced by the caller with the size of previous page that was linked + DMA address returned for the previous page". > Is this description correct? I don't understand the part "+ DMA address returned for the previous page ". > In my codes, let's say I call this function to link 10 pages, the first dma_offset is 0, second is 4k, third 8k. This worked for me. I didn't add the previously returned dma address. > Maybe I need more test. But any comment? You did it perfectly right. This is the correct way to advance dma_offset. Thanks > > Thanks, > Oak > > > -----Original Message----- > > From: Jason Gunthorpe > > Sent: Monday, June 10, 2024 1:25 PM > > To: Zeng, Oak > > Cc: Leon Romanovsky ; Christoph Hellwig ; > > Robin Murphy ; Marek Szyprowski > > ; Joerg Roedel ; Will > > Deacon ; Chaitanya Kulkarni ; > > Brost, Matthew ; Hellstrom, Thomas > > ; Jonathan Corbet ; Jens > > Axboe ; Keith Busch ; Sagi > > Grimberg ; Yishai Hadas ; > > Shameer Kolothum ; Tian, Kevin > > ; Alex Williamson ; > > Jérôme Glisse ; Andrew Morton > foundation.org>; linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org; > > linux-block@vger.kernel.org; linux-rdma@vger.kernel.org; > > iommu@lists.linux.dev; linux-nvme@lists.infradead.org; > > kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche > > ; Damien Le Moal > > ; Amir Goldstein > > ; josef@toxicpanda.com; Martin K. Petersen > > ; daniel@iogearbox.net; Williams, Dan J > > ; jack@suse.com; Zhu Yanjun > > ; Bommu, Krishnaiah > > ; Ghimiray, Himal Prasad > > > > Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to > > two steps > > > > On Mon, Jun 10, 2024 at 04:40:19PM +0000, Zeng, Oak wrote: > > > Thanks Leon and Yanjun for the reply! > > > > > > Based on the reply, we will continue use the current version for > > > test (as it is tested for vfio and rdma). We will switch to v1 once > > > it is fully tested/reviewed. > > > > I'm glad you are finding it useful, one of my interests with this work > > is to improve all the HMM users. > > > > Jason