From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A09BC27C75 for ; Tue, 11 Jun 2024 19:11:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A6C286B00A7; Tue, 11 Jun 2024 15:11:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F43F6B00A9; Tue, 11 Jun 2024 15:11:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86DE26B00AA; Tue, 11 Jun 2024 15:11:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6585C6B00A7 for ; Tue, 11 Jun 2024 15:11:36 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 01892140565 for ; Tue, 11 Jun 2024 19:11:35 +0000 (UTC) X-FDA: 82219551792.30.CBB4764 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf15.hostedemail.com (Postfix) with ESMTP id 1AE28A000D for ; Tue, 11 Jun 2024 19:11:31 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qGeF1z59; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf15.hostedemail.com: domain of leon@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718133092; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Zh0CHXTE9xV7aaJDa4YGe0BUqZW5H+N0XIc3nGbXrw0=; b=D2jDmZiBKYt5CxFmY87wVzuUqBhuQDSjGfIDjsn/DPo//dYTUI01WZgOweVdnyFCAwCR+h ymgVhbYYBbfw89BF6b5E6B0ak3q3QFsqyQWLKNkLM32jGaIjPReTV0wYBakE8mZDT8o2lK mwYzoMrqNIVFyZgNi6opADZ+0FrlO4w= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qGeF1z59; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf15.hostedemail.com: domain of leon@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718133092; a=rsa-sha256; cv=none; b=BUlVgYMACr5O4pF48wMwsawDbkxNx5M7HsYT4qdiRd/UJSB61uY6wIl/jaUJfre1qhCNV5 zVudoIKXdlxsxEcCxXTrISwHazvEd2W1lIb1Nw8bFPWYrho5+o1FgDtEYvAdC0WH0YsQ9Q K2Z+94WDA7pgkK1mJoqWPlBeDoeEhmw= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 61999CE1D9F; Tue, 11 Jun 2024 19:11:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E2943C4AF48; Tue, 11 Jun 2024 19:11:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718133086; bh=VL91sYxrS8ClC4b8/FdOk7/0y+PMB1c26MCrYyZDHlk=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=qGeF1z59jrpZ1RnFDkbFTHlzMrh2tlbdyzzMeTbp/apFSEEZTKtHkH+3loP9CksbW En1EFU2PvaPwg8kwPv58hUH/00Z1iJaK+TYLFlY7jq60EmSHn58BFrPjyBuLSg6kUy szjRV8n4ki19tOLkxEzrvNUz/1FvmLxUyHFx8FXvupsoPReOnwziyyb2DWhwA8gLT0 rcx8zCFMkZDVxz7DB+vMIFm9V2odi09cuXNc6Wz54by/flNmIEo+NsJWB8ixRZyOOy ZaePydnV8sQeml50d7Nyw2E27Fj9eEs+p6aDVCIPkBDnMmJmhcJStvrDcHXQWZUDvZ +Q+PdkPi3MXew== Date: Tue, 11 Jun 2024 22:11:21 +0300 From: Leon Romanovsky To: "Zeng, Oak" Cc: Jason Gunthorpe , Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Chaitanya Kulkarni , "Brost, Matthew" , "Hellstrom, Thomas" , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , "Tian, Kevin" , Alex Williamson , =?iso-8859-1?B?Suly9G1l?= Glisse , Andrew Morton , "linux-doc@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "iommu@lists.linux.dev" , "linux-nvme@lists.infradead.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , "Williams, Dan J" , "jack@suse.com" , Zhu Yanjun , "Bommu, Krishnaiah" , "Ghimiray, Himal Prasad" Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps Message-ID: <20240611191121.GD4966@unreal> References: <20240503164239.GB901876@ziepe.ca> <20240610161826.GA4966@unreal> <20240610172501.GJ791043@ziepe.ca> <20240611154515.GC4966@unreal> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 1AE28A000D X-Stat-Signature: ts6mmfbnhenp4z33edzo7ofoaktf676j X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1718133091-763804 X-HE-Meta: U2FsdGVkX1/zndD/ZvLesilzQCe9Iaqp/a72LzvKg3iHI/Msl2awM5shSFsPURQFfYzq+a9hJZE0F8gnV0Tl1ez2buEtZC+oTqpyp+uy52fsedPyCvagF5BVGoS3YUp07nOBcWzb756SJw0H1BjXTOdAu59iTVunTbRwciDdCLGaC9Dy+55iWecKzhew9mv2lESzmfqpBuzptcWsASbZ3urPkDWS4mATjELXyp6eYDiE9jgj/fN8HIH7w1tQ8l6Yt2qDiU09gxydSse/YzfjZ10tO/kQLIEu/YbrMi0FmPYY3+tloMeqwxI9vEtFbplwW+sv9NdueQZ0LWf82N/jucWomWGoTyQTY1mMqPGpiKrtuPbBLdXXi86ProSw4xVAysDk4c/4bL4p5gtZqSqG9gSFMhLbXI6MRQn9keAdjb4puQmXbdWlyPGfG6w1OMW8ppn1357eLXkbslXThfm9L2SKcuHTMgnN49xGtq+TdCld8NpcWSbsVCKz03KsF8wShzYxMmprqXvH1WtcblwODnxum0kfvyGl7yDzIgTVxo6OcQrOom4qr0FzDVX7/KPu1p6oaPC6jkHJxhH5hNFyigPriQKAqPOyHVBpFwIDh7o0L1cco5j85g0uhBIac2PY8NBPWeSapj6xcd6E/v6sY40QfC4jcMKTrI/UmV1eWRYmkFtn1FrVSGzzN7aytQgsfMG3dXiZpeY+OzLRJMAynK7SwpxeMMVjMnlYLxor5M3XzIBdaxmKrp2bdYwkEjIwVc9Xjm1Ljqs+YzRUamiWFWELjYQjSwuKP4GEyNK/834kgomXQxb7H/Ghh+E+c1WPRbcYyzfU8tDLU78Y/3lkuqVKNpWoggKFPk/saoCRF7BYFdm9UAbQwiR7ZVw4IPa1K35r3SQpgdoqloqykckcZWOaA02+GSFnKGoPfeulXg+19IR7W48O8/XEThHamB99BL/ejnXz2fUevXXCX+q I5ZtDIon MQuazjEF5+e+1+6TdWmF19I5vVIJoc3yk58A+2ydE14PGAfwQgPQVFSqwKhhXX0zXiJR++RwXZRE9EsqFtR9RHsWNGNsZrMj72Es4KozWs8DZ8PUMQaseURAwo5Uxg9duWDvCGuETNiAhBim01klpaIX97H4+y4J55FVU4klGwgQbRAQbwbFztH6ogU/o6VlMinOiUj5LMzDJNeuJL81Z4BtZc5pjhcfk75xmnwBkfDh8cwAYJt2bcOVnqjdny6g9vYgL4rBOdUdzj9l6DSqIkbabSdFB19QV7cXsLqDYcUXrSFC/n0jpbPjZydV8LUPGjuOXvjt+wAwsmpHvEizgIsVG55POZbupgC9bOKtFLOzptTzQD4wAIEnssbKzA7BlLbgZDI3m8zC2Vbc6KATXgYsRGYpUvpUj3KCNaZ5OB4XNzMRq2cjTk0GNwcEecWc2cB+M6BpbDUXmPSDsglbAaWbCttF6ATkDi/iGU1z3Jp3WXF5/+n9o34yiTcPYSe1jM2LT6uOWf4jbNb63gPXamohqzQeIQPfrXL9nnSBdN8KlqyDle19nlMB1VjUQ7/z30k23UowcSd74QZzBserHHHAft5l0PLFVG+/xMfAOAsVg5SYkN3mGFVXGBl0CUbNkTKvZ3ZsHYE8u6FsWCrVjFUEM50yxhosupAlSO8MQymgnDMA8+iKS7/m7PjtbWc6aG7OszF8b9uodduu+ZwCP3JMT+/JGRQQgU2QHEf6vm8jfZj25qlrQ6v21DIsk2tj8rHV/DFYDDHUEspSH3r1R48yVKumyDAxPh4iEU04PLLoUIeijDF3gZoUyllA8UnwAeyi/sKrEeuYjUaTPK2+r90oJ1wBL49hH6SC4gFB1FTwF9qzAzcuBbU0IjBfQTZpmqMxh2Tm+j/hZsgpMYevAbLm1culuqXUDdSneWnAFTVMcdEYxrX64zYt+jnMFK8okD4SHBzEsS+FdoET2YkZH3pjwQmNO wII4VTVk v9qD7RUBT+aHqK9aenkQvjYLlzkEGwX71jxxmv4LbV1e5Iz8zkNZvyzRQMAHH83Ij1VwRYRw/rzi6SmTMukILPZDwVdCW+s1agtZ3ih6vHiLPFnpyEXrCHBKOsnA/6FlaofxVcXXpMQhd2FOiAiS75mlwEoDxPyD06NvxcDLqYklHRkxI5Lf6Zk1Mc2Z5h6KaBxMD4lSFSlqfiWYhS4Ciw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jun 11, 2024 at 06:26:23PM +0000, Zeng, Oak wrote: > Thank you Leon. That is helpful. > > I also have another very naïve question. I don't understand what is the iova address. I previously thought the iova address space is the same as the dma_address space when iommu is involved. I thought the dma_alloc_iova would allocate some contiguous iova address range and later dma_link_range function would link a physical page to the iova address and return the iova address. In other words, I thought the dma_address is iova address, and the iommu page table translate a dma_address or iova address to the physical address. This is right understanding. > > But from my print below, my above understanding is obviously wrong: the iova.dma_addr is 0 and the dma_address returned from dma_link_range is none zero... Can you help me what is iova address? Is iova address iommu related? Since dma_link_range returns a non-iova address, does this function allocate the dma-address itself? Is dma-address correlated with iova address? This is a combination of two things: 1. Need to support HMM specific logic 2. Outcome of v0 version where I implemented dma_link_range() to perform fallback to DMA direct mode. See patch 2 and 3. https://lore.kernel.org/all/54a3554639bfb963c9919c5d7c1f449021bebdb3.1709635535.git.leon@kernel.org/ https://lore.kernel.org/all/f1049f0fc280288ae2f0c1e02388cde91b0f7876.1709635535.git.leon@kernel.org/ So dma-iova == 0 means that you are working in direct mode and not with IOMMU, e.g. you can translate from physical address to DMA address by simple call to phys_to_dma(). One of the comments was that it is not desired behaviour and I need to create separate functions that will be in use only when IOMMU is used. See the difference between v0 and v1 for dma_link_range() function. v0: https://lore.kernel.org/all/f1049f0fc280288ae2f0c1e02388cde91b0f7876.1709635535.git.leon@kernel.org/ v1: https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/commit/?h=dma-split-v1&id=5aa29f2620ef86ac58c17a0297929a0b9e8d7790 And HMM variant of dma_link_range() function, which saves from you the need to copy/paste same HMM logic from RDMA to DRM. https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/commit/?h=dma-split-v1&id=4d8d8d4fbe7891b1412f03ecaff88bc492e2e4eb Thanks > > Oak > > > -----Original Message----- > > From: Leon Romanovsky > > Sent: Tuesday, June 11, 2024 11:45 AM > > To: Zeng, Oak > > Cc: Jason Gunthorpe ; Christoph Hellwig ; Robin > > Murphy ; Marek Szyprowski > > ; Joerg Roedel ; Will > > Deacon ; Chaitanya Kulkarni ; > > Brost, Matthew ; Hellstrom, Thomas > > ; Jonathan Corbet ; Jens > > Axboe ; Keith Busch ; Sagi > > Grimberg ; Yishai Hadas ; > > Shameer Kolothum ; Tian, Kevin > > ; Alex Williamson ; > > Jérôme Glisse ; Andrew Morton > foundation.org>; linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org; > > linux-block@vger.kernel.org; linux-rdma@vger.kernel.org; > > iommu@lists.linux.dev; linux-nvme@lists.infradead.org; > > kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche > > ; Damien Le Moal > > ; Amir Goldstein > > ; josef@toxicpanda.com; Martin K. Petersen > > ; daniel@iogearbox.net; Williams, Dan J > > ; jack@suse.com; Zhu Yanjun > > ; Bommu, Krishnaiah > > ; Ghimiray, Himal Prasad > > > > Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to > > two steps > > > > On Mon, Jun 10, 2024 at 09:28:04PM +0000, Zeng, Oak wrote: > > > Hi Jason, Leon, > > > > > > I was able to fix the issue from my side. Things work fine now. I got two > > questions though: > > > > > > 1) The value returned from dma_link_range function is not contiguous, see > > below print. The "linked pa" is the function return. > > > I think dma_map_sgtable API would return some contiguous dma address. > > Is the dma-map_sgtable api is more efficient regarding the iommu page table? > > i.e., try to use bigger page size, such as use 2M page size when it is possible. > > With your new API, does it also have such consideration? I vaguely > > remembered Jason mentioned such thing, but my print below doesn't look > > like so. Maybe I need to test bigger range (only 16 pages range in the test of > > below printing). Comment? > > > > My API gives you the flexibility to use any page size you want. You can > > use 2M pages instead of 4K pages. The API doesn't enforce any page size. > > > > > > > > [17584.665126] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, > > linked pa = 18ef3f000 > > > [17584.665146] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, > > linked pa = 190d00000 > > > [17584.665150] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, > > linked pa = 190024000 > > > [17584.665153] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, > > linked pa = 178e89000 > > > > > > 2) in the comment of dma_link_range function, it is said: " @dma_offset > > needs to be advanced by the caller with the size of previous page that was > > linked + DMA address returned for the previous page". > > > Is this description correct? I don't understand the part "+ DMA address > > returned for the previous page ". > > > In my codes, let's say I call this function to link 10 pages, the first > > dma_offset is 0, second is 4k, third 8k. This worked for me. I didn't add the > > previously returned dma address. > > > Maybe I need more test. But any comment? > > > > You did it perfectly right. This is the correct way to advance dma_offset. > > > > Thanks > > > > > > > > Thanks, > > > Oak > > > > > > > -----Original Message----- > > > > From: Jason Gunthorpe > > > > Sent: Monday, June 10, 2024 1:25 PM > > > > To: Zeng, Oak > > > > Cc: Leon Romanovsky ; Christoph Hellwig > > ; > > > > Robin Murphy ; Marek Szyprowski > > > > ; Joerg Roedel ; Will > > > > Deacon ; Chaitanya Kulkarni ; > > > > Brost, Matthew ; Hellstrom, Thomas > > > > ; Jonathan Corbet ; > > Jens > > > > Axboe ; Keith Busch ; Sagi > > > > Grimberg ; Yishai Hadas ; > > > > Shameer Kolothum ; Tian, > > Kevin > > > > ; Alex Williamson ; > > > > Jérôme Glisse ; Andrew Morton > > > foundation.org>; linux-doc@vger.kernel.org; linux- > > kernel@vger.kernel.org; > > > > linux-block@vger.kernel.org; linux-rdma@vger.kernel.org; > > > > iommu@lists.linux.dev; linux-nvme@lists.infradead.org; > > > > kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche > > > > ; Damien Le Moal > > > > ; Amir Goldstein > > > > ; josef@toxicpanda.com; Martin K. Petersen > > > > ; daniel@iogearbox.net; Williams, Dan J > > > > ; jack@suse.com; Zhu Yanjun > > > > ; Bommu, Krishnaiah > > > > ; Ghimiray, Himal Prasad > > > > > > > > Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to > > > > two steps > > > > > > > > On Mon, Jun 10, 2024 at 04:40:19PM +0000, Zeng, Oak wrote: > > > > > Thanks Leon and Yanjun for the reply! > > > > > > > > > > Based on the reply, we will continue use the current version for > > > > > test (as it is tested for vfio and rdma). We will switch to v1 once > > > > > it is fully tested/reviewed. > > > > > > > > I'm glad you are finding it useful, one of my interests with this work > > > > is to improve all the HMM users. > > > > > > > > Jason >