From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09B4BC87FC8 for ; Mon, 21 Jul 2025 06:49:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4784C6B0093; Mon, 21 Jul 2025 02:49:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44FC76B0095; Mon, 21 Jul 2025 02:49:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38C7E6B0096; Mon, 21 Jul 2025 02:49:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2801B6B0093 for ; Mon, 21 Jul 2025 02:49:14 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9A49BC061C for ; Mon, 21 Jul 2025 06:49:13 +0000 (UTC) X-FDA: 83687344986.17.3FE0718 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf14.hostedemail.com (Postfix) with ESMTP id F22F210000B for ; Mon, 21 Jul 2025 06:49:11 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=hXfMo9FY; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf14.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1753080552; a=rsa-sha256; cv=none; b=00xK/ym8uBVBvgjKlLvZb2OckP2eTlJ5VLDxZZSN21MVmm5aBdCLaKjKl/zdo99nYdYZQn 6k64WdBs3ooR4GMuUPOTCTF8WWmq2i9w2sGiMjWX0c31NY8VSZtt2wlq2s3UbxEGEZtlw6 2C/SmTsc9iyyNz0MPnhPQ1YXyZM/vzo= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=hXfMo9FY; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf14.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1753080552; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4A6TidLWG0bJAEFEsQexAjA4qhZNO02M3kKFPiPgk24=; b=JiBLttYIIAPNfbSWWUoGbn2Ltes+JosifbeS4RBizQF5N4evubjrGVJaHpGwTYe8oagOxi LUcZsi40u3V/5Tc0S7+Tlok5Zh3OGZUFi6ERv7Dpfr79FFojpwmfb/C+4uC32rUSlLQLxL C3SeRPkADmwoGoxnuOCbHHgD7Zw/QAM= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 273A7A527F6; Mon, 21 Jul 2025 06:49:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 19A71C4CEF4; Mon, 21 Jul 2025 06:49:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753080550; bh=R4gstohCCiEcB83QHp4vePC6Vl/FRbG01v8oYCtIJb4=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=hXfMo9FYah+I/klWyq3E4MM8uAEW+sVZ6ODTWsq5JUGj0H3aocK/YvbN4TFt88Hvo Rn06q7poo3PqWnBummfdHCv3BPx6GuFba9cIihgDNPzUzVyMGM+iS0hPLeAsvnHP3l U16vKWM2EvhSN9ENkKqsKLL6P+hah8+TgBzcVXJobxUE7ZVqOKbmm4hm0TXfKYNChx SSFWMkt4DOwbPfkKpjrcSjqtEzEmHFxNyFpppZlLc8Cu6Ru5oSdrYYLOQST7PAChe3 CoqaxGFV0y1PwQZKEVuQqLnMxUFjEdPhV9/PxlWmOLfWgwSQKGbvScbg47BtI5mlQ2 IcE2ZN/gKY35Q== Date: Mon, 21 Jul 2025 09:49:04 +0300 From: Leon Romanovsky To: Yonatan Maman Cc: =?iso-8859-1?B?Suly9G1l?= Glisse , Andrew Morton , Jason Gunthorpe , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Alistair Popple , Ben Skeggs , Michael Guralnik , Or Har-Toov , Daisuke Matsuda , Shay Drory , linux-mm@kvack.org, linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 0/5] *** GPU Direct RDMA (P2P DMA) for Device Private Pages *** Message-ID: <20250721064904.GK402218@unreal> References: <20250718115112.3881129-1-ymaman@nvidia.com> <20250720103003.GH402218@unreal> <35ff6080-9cb8-43cf-b77a-9ef3afd2ae59@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <35ff6080-9cb8-43cf-b77a-9ef3afd2ae59@nvidia.com> X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: F22F210000B X-Stat-Signature: jktku7fumczthy3gp774isbub3fphe5r X-HE-Tag: 1753080551-952104 X-HE-Meta: U2FsdGVkX18/7jIeQluoIKcH5E+MjixV8ePlAJ0bOyM9Dr0otgVqj42SIvoR4blGZlHLT8P+xAyghsOasOomONBpa/irTi5C6AMZ2+/BTf7hHVSzvlHt65TjbQRibdNGl4XaP8rPv6ZhlBJO+9d9Ksif7TfXVhlNI+TUPMUVsFf1fme7HBwTvCp/hT9vvMJNZ9XGB2DqqR9ajnvCcmT/CNdYXyn5PddAmPwqoaUXaSHqTTqK3F//rYix3uEUKO8nusz8Y3FXALafZq4ag8/MUgU+ePERJdWvnTIsrL8ChM7K3afUyCeCEU55T4/qQzpMfFozqZVjxtL2sR+qu8qnAoMU1G7Hb7RSIDvQRzLqx3yH/xgpNKjQ4/sHi8A9I7X0hXLLfliv8B2B3++1A0/fy3ZGpJTUterlZUJS65jZ4J6tshJF2tKPICcPSxPS3EAzEBhO6AniyrGIZD1KGGmd3xr6kQNIswJAofXKW/U4DrzT5LTj5gD5vjw/fWuGcl4JpX+mYu9756t4i5pGCCZV4LFmtPXN3LKNkppp856FXbjC7y1zrryVIlpiq3NzxOudZuDHPQeRcK28b42hwCsorcAPAdymquKmg0gGwPZAXKJYLvQzd8KddAlspdw/Yg64XZ9GHX4cV0Pr+O+3xvJhXOTnlff8Hb7qpyZ3D8rwopVyS/NweUpizbskC32liirNEBWPlu6UgPSUnEXHfIOCDGaVMZLXdJ96AEgWmfBj+GWUv7fFrbOzIpNELiVuYwW8lXF1imRV+DctQxsy13sxgOINTZ8VSUw6+gv2BVj45CYU4T8txpeqEJfBNcISPD3jWcatAWkEWAR7B0nNdTC18rLCIAE/xOT8NzmGUdys+4q3uENYVqiF9idWC5FDNkg6CF3tsMmum/gxAssvZWz8kQPtNLu2V/8lTtbCsoINko/GbAl1T5uqF/T5vaR7iBwB+HvHbGndcI82q4I79X6 UJfYEaBn Xg+Ig+H83GNcK4Vsdvze9mLErivZWjJx700PBimnaOkPDfD0PaacO7KYjFL8nCR6lhd6NArmawH+DzkhB7AJG01+y+ILvEiVBMW+BA4xw0WTuI212+pFL7saPxxICDnbsSalkbHEPva9lAt30SKwm2aoxGcx4B1YO9gZtYxX9mqCZ41Qf/L9itST+a02CohOON4z9TV/HhwNcg71O9JF2c4ClZ3zwlrNPBsvgFrPdWfoXg+QgqEBgtKgDLZviYRsrYAqXrnSned2py+MkSKhG9kFOdcdtttrusw4p9FncEGkvS/Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jul 21, 2025 at 12:03:51AM +0300, Yonatan Maman wrote: > > > On 20/07/2025 13:30, Leon Romanovsky wrote: > > External email: Use caution opening links or attachments > > > > > > On Fri, Jul 18, 2025 at 02:51:07PM +0300, Yonatan Maman wrote: > > > From: Yonatan Maman > > > > > > This patch series aims to enable Peer-to-Peer (P2P) DMA access in > > > GPU-centric applications that utilize RDMA and private device pages. This > > > enhancement reduces data transfer overhead by allowing the GPU to directly > > > expose device private page data to devices such as NICs, eliminating the > > > need to traverse system RAM, which is the native method for exposing > > > device private page data. > > > > > > To fully support Peer-to-Peer for device private pages, the following > > > changes are proposed: > > > > > > `Memory Management (MM)` > > > * Leverage struct pagemap_ops to support P2P page operations: This > > > modification ensures that the GPU can directly map device private pages > > > for P2P DMA. > > > * Utilize hmm_range_fault to support P2P connections for device private > > > pages (instead of Page fault) > > > > > > `IB Drivers` > > > Add TRY_P2P_REQ flag for the hmm_range_fault call: This flag indicates the > > > need for P2P mapping, enabling IB drivers to efficiently handle P2P DMA > > > requests. > > > > > > `Nouveau driver` > > > Add support for the Nouveau p2p_page callback function: This update > > > integrates P2P DMA support into the Nouveau driver, allowing it to handle > > > P2P page operations seamlessly. > > > > > > `MLX5 Driver` > > > Utilize NIC Address Translation Service (ATS) for ODP memory, to optimize > > > DMA P2P for private device pages. Also, when P2P DMA mapping fails due to > > > inaccessible bridges, the system falls back to standard DMA, which uses host > > > memory, for the affected PFNs > > > > I'm probably missing something very important, but why can't you always > > perform p2p if two devices support it? It is strange that IB and not HMM > > has a fallback mode. > > > > Thanks > > > > P2P mapping can fail even when both devices support it, due to PCIe bridge > limitations or IOMMU restrictions that block direct P2P access. Yes, it is how p2p works. The decision "if p2p is supported or not" is calculated by pci_p2pdma_map_type(). That function needs to get which two devices will be connected. In proposed HMM_PFN_ALLOW_P2P flag, you don't provide device information and for the system with more than 2 p2p devices, you will get completely random result. > The fallback is in IB rather than HMM because HMM only manages memory pages - it doesn't > do DMA mapping. The IB driver does the actual DMA operations, so it knows > when P2P mapping fails and can fall back to copying through system memory. The thing is that in proposed patch, IB doesn't check that p2p is established with right device. https://lore.kernel.org/all/20250718115112.3881129-5-ymaman@nvidia.com/ > In fact, hmm_range_fault doesn't have information about the destination > device that will perform the DMA mapping. So probably you need to teach HMM to perform page_faults on specific device. Thansk > > > > > > Previous version: > > > https://lore.kernel.org/linux-mm/20241201103659.420677-1-ymaman@nvidia.com/ > > > https://lore.kernel.org/linux-mm/20241015152348.3055360-1-ymaman@nvidia.com/ > > > > > > Yonatan Maman (5): > > > mm/hmm: HMM API to enable P2P DMA for device private pages > > > nouveau/dmem: HMM P2P DMA for private dev pages > > > IB/core: P2P DMA for device private pages > > > RDMA/mlx5: Enable P2P DMA with fallback mechanism > > > RDMA/mlx5: Enabling ATS for ODP memory > > > > > > drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++ > > > drivers/infiniband/core/umem_odp.c | 4 + > > > drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 +- > > > drivers/infiniband/hw/mlx5/odp.c | 24 +++++- > > > include/linux/hmm.h | 3 +- > > > include/linux/memremap.h | 8 ++ > > > mm/hmm.c | 57 ++++++++++--- > > > 7 files changed, 195 insertions(+), 17 deletions(-) > > > > > > -- > > > 2.34.1 > > > > > >