From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 57438FEFB4D for ; Fri, 27 Feb 2026 13:48:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=n0vB6jKlYb80N4+dkhk+Q893ehToFfyA8SjCjvVbTBk=; b=eFOel4kjm4Ao7K vZoW7N6YVFA7UuCY2OgSI3cMy2QBlAhHSonIR123Cb8aMn8ClLADpTWHHKJfE7ahiS981H18SYXnI onFY0ESkKGCYDMBIkqWonFksRXNILj2syRpSShkNTo98hCOrhYnrEti+ZFBrBggdC7vyTi9Om3tB2 /tnGEsJzgUL1OpcorA+to84qHQLbjp4M7R6t6WIuqsUALUeQZx/LHnPDJvUvbBD2cBJrP9NHCoW3p jWdivDUjS1ezPO3lGC/nmJyXPav4N9XkZ5HInG/ufN+eikIE+OCXe+GvtjsP90ADa0sLj7VZtY9oS FHeW5oBoz5KDd1Ekxyhw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1vvyCY-00000008Ow4-2ncq; Fri, 27 Feb 2026 13:48:18 +0000 Received: from mail-southcentralusazon11013038.outbound.protection.outlook.com ([40.93.196.38] helo=SA9PR02CU001.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1vvyCW-00000008OvZ-0MDS for linux-riscv@lists.infradead.org; Fri, 27 Feb 2026 13:48:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=f9AfKfGNMQ10RTen/SvrcLcSv+ZbbRxxbKR5/D3n4SflnjaTNYg8KYmYRJNjtlL1VEzRjyU0ESwnphg72RM/j6Nm4aumH23xe4sDNJsXoGoSQXtRzZ+QthCdHKdqyhPh2KPT67JJogpfBaqJMHT+iWQ1NWxZyx3onXDkp0Wnm2L58Q3qTm1oyq7WJSITiQCy6xFAzfNW2KHkPY+QAYv6IyUuxBlmZMKXIxLDgD0cDEN36WprAriJemxGPH8qPYVv/uKWwp7gwUfO1PC8R30w/9ef/iLllt639wR03VZAeWH97SjFF3R3s9zdoothK0r18/bxUXtZ0qjzwINHB6v16A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Fk+OXJQCHLO4K9H5hRfiRa7MqHe46cXgZlinnCfZMdc=; b=aupzz1koVdOcfkrNFPbbfpeIOeCxo7nujSkl7Krew0hjD4mZ/zZf/zp3d9G+q+gI0/g6MZgmea+zFaxT9VS6zrw/dkujmw33PnmER6mwiwkEe9Yas3BRBPie+7Ioka4eonhobEO3H7ZbF0EEfhqBBhM5IbOc0JsgZ6TXHnw1+wyVJrxrSBIZNTXWBElbDHiUYABgNRUihLsCL0RW8DpsnZ3gRSD57tHdxfvrn+JfICir+hpK/xIGKmlFX2nuecIjX2HyEOtFSe/0TKjkfdRQT9qkP5GTiQNdE9tRcCMHjZNE3OeE/IgcdEesIeSCHoBzBBqAsq/7357O5q3MRiUe5w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fk+OXJQCHLO4K9H5hRfiRa7MqHe46cXgZlinnCfZMdc=; b=JOMlcEzWCKlJDfRoQ3PseBO3EkWIrRWCu3RN/l8BRMFKemuAgD+RrwtOx58IVs7u64uTieyIFAV1CPzQcysdxyg5SnbNC3ciE74R1w+bEcVTw85ce9PP6WQwT3EXY4oTrnYE1zKkspsww39WS0zBOiyMWjcpsLYwmxNwZ3/nC/5XRKNnRptafaWRD4tC3PQek3+xThoulaHM9kPeAaFRj5Vy5jiCwAcyBd7N64FoPnVYXvm1LiXs/ps7R5v0Kpxg7FdJNwPFwyNU+xMDLXOOZZrvb3Cl45QfehuY79Et/1YLMN0BGV3s+6iNFkoIGyajPbXSGckwqjucFdsqoVVqtw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by CY8PR12MB7610.namprd12.prod.outlook.com (2603:10b6:930:9a::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9654.16; Fri, 27 Feb 2026 13:48:06 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9654.014; Fri, 27 Feb 2026 13:48:06 +0000 Date: Fri, 27 Feb 2026 09:48:05 -0400 From: Jason Gunthorpe To: Alexey Kardashevskiy Cc: Alexandre Ghiti , Anup Patel , Albert Ou , Jonathan Corbet , iommu@lists.linux.dev, Joerg Roedel , Justin Stitt , linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-riscv@lists.infradead.org, llvm@lists.linux.dev, Bill Wendling , Nathan Chancellor , Nick Desaulniers , Miguel Ojeda , Palmer Dabbelt , Paul Walmsley , Robin Murphy , Shuah Khan , Suravee Suthikulpanit , Will Deacon , Alejandro Jimenez , James Gowans , Kevin Tian , Michael Roth , Pasha Tatashin , patches@lists.linux.dev, Samiullah Khawaja , Vasant Hegde Subject: Re: [PATCH v8 07/15] iommupt: Add map_pages op Message-ID: <20260227134805.GJ5933@nvidia.com> References: <7-v8-d50aeee4481d+55efb-iommu_pt_jgg@nvidia.com> <20260117154347.GF1134360@nvidia.com> <20260226150440.GB5933@nvidia.com> Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: MN0P223CA0005.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:52b::22) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|CY8PR12MB7610:EE_ X-MS-Office365-Filtering-Correlation-Id: 5bf65e5a-b1e3-449a-edd8-08de7606d606 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: VkMuTb/sL8yVWA6g7iByX8T02BsA844qtaoDkazWxYxGqQFjDSNVe+gJTJpnhnKHGe3+BEdYDOpNcr1bX6imVQfEQZrgsqRHlvU12sDxd2lFI4OWGGRwSiEbIHBbd/afOit7NrB3M8JffA0V7+uOLdlpU27MISYm3K/we9vGpslPjp6fF7WoaX9wd+RDc8lmYGZyQ3cn5KrgNzqckheOUQR5rT+8jy10RLBEAMC69RaKOWDX6PpSwCy8RWioVdvEGoFXkmRVamRJjB1GoYURMvWk3HE3pBQLahoyOk46kqSPDY1WzWU7LnR4n+9mdE0g/NpG9mXFU2eE3DcDOkiVtnbDnEn3afRJ3tBXP8TvVpY6v9mY0sZIUGLs9W4HmkctqYhnwF6ymJR42duS4GhewcqJaIzAtAyTVAfVKXfQd/KRD+L21qhQfsw9ZxQGFjUlMFyUU5VtDQjMLkamawJrsXGp9/m2ef33bcX0SLPsFP5AFP5idozLGNPSz5OCgdAKsQhOG8zFZYwE37/CWZNn+vEFJXHUjPU5UqcFdZwXbxDSwDotUdF5o14LY6+3osdeRWpqn8rY/UNvDqJSku1ElFdPY1yxEC1RSrxmcdfQdd2HY85UHkCHJ31wKdMi20oVkB6wmsYFSFvYjHg7rTu0z//0VdrSmA8ySpjbQmbm+zdouE8AQ4bNK4QxXo0fmsXtV3cnb7uigJ4EHaM3jKgyh9i3Jz5w4ag60lEXdX50y7Q= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?3t3jB5oxX8vQuyC4DwRSPqCUJiiXpayAmjsTbMkWwX5R40RHsBvDg3J+3zMS?= =?us-ascii?Q?hErH9hBH+RxKx2fq/Cop2mHMfB4To+M3kmG0Jc5KO+PCpjXYK/qFzlhPPtoN?= =?us-ascii?Q?C1WoShhvvuDXlG8A2y8jeT7ZDJh/IfGY7D/jVnm7kMG7Br+InLGHNdTWuZUH?= =?us-ascii?Q?+b5XLTO8e85RpG5CPGHpGnQWTDxa1lJMDKdM+yNm0eabo58gySlJRa1TbqIh?= =?us-ascii?Q?LWS9T814y+M25DOIckZmNKdYOY9zX8lHPsiR//gDPTWbgOjX+hX2KK5sPQPk?= =?us-ascii?Q?0piG53xWv3/WpFjyQqyMNe/sKavEftVnFJkBUSyFVby1TZOZyJ4Cp6yzguEN?= =?us-ascii?Q?thY+dsIKFrAqSR6xVnMntQUeVuU0l2fu6Du+tWtOi1ti3OPbXmOKvQgjguJj?= =?us-ascii?Q?Xvt6uplY3xNBfEnx7piEfmcCMcHSw9AateIWCysZunTrRqGHnqNvT8AKyO4L?= =?us-ascii?Q?GACvURJytyYlL9vgsNvVu5TAThXKHXpfQicTjteZia9nYmULRFXxrbEDQDAm?= =?us-ascii?Q?ooRULYGgY+Mu7JjlUq/AMhtkpLGL1dOgqWdJUO5qpsgTlgNgl8WHhTN8sFvR?= =?us-ascii?Q?W0pNsO0Q3yeMY3X4zmSxMyGQrR64Pk4t0+3BDarGBJDz9QI/VfRwKlwCk1sI?= =?us-ascii?Q?xYbKiSrEJm9WGxtrHw3z77YliarOzqOQ721LapwkyB1L3Z95CQplrGdb5/7L?= =?us-ascii?Q?2KX0AdR/gRZoVmQM9jQ81/eBvH2Moubq4ns8LcL2p+jg0jAgi6heI3WbqSXp?= =?us-ascii?Q?I3NJJVeM9h4jmp7UkXql+z561BWyVZhozrd79NDMa4+m/4Gtcuj/66kAJLAi?= =?us-ascii?Q?KYEMzRujSuR1oQR6dwCh/l0MhdCzMHU85ilMnZWOsAjW0jx/p6dJvpiIevcs?= =?us-ascii?Q?m/ZUJ93yXVb/jCe8hEbGM9kec5pWS3owHw+5AoBCA9FNzEYnSjeEUU2ajpHv?= =?us-ascii?Q?MC7Cp9Df6OHj0OToT3HSuv/nwbh77+epqEahIEw22qx3JuM4AS40silufrN9?= =?us-ascii?Q?5Dbvsam7qVueTzvaxNLbqcxyQ29byi7hKQ6nH4fndYSuvH7cEwbqqlxKo9x9?= =?us-ascii?Q?Eqo4XefHIsh01oD0i1NuqO/Vdl1sAstxK7hoVOxVMHgPbdrlbKKJrOpon6z0?= =?us-ascii?Q?5VknczDvilYwZTxZNw4ik2V7RR+yYNANmxMeAO3XGU939BoScnP8SdH4i17w?= =?us-ascii?Q?F7sxeJfZ/4i1AeyhDNL0EuwWnvQbRhLkStzehVq93DQnXFduTY2snloMb0Ga?= =?us-ascii?Q?RgsjXUu3nfvPSRGI8e6mcgEBzvX0pViVRMpz888AccFxKR+c+5OQAc6mCH6T?= =?us-ascii?Q?WwPwDY25FofWwChkBpFFweEUBXWRKi0NoqKmwIkDKZT4AwsFnDHFAp4PE92H?= =?us-ascii?Q?nqZifouWIueQEfAZbXv0bEoAEoT9hNDxP1xd1dHOoAZeq+sjoqvEL2N0xfy4?= =?us-ascii?Q?xRBJnknqcOMZKhIiQmuLTKREk3HonN2Py7W6q3jTH0agCGyMfK4SvzOt6zwg?= =?us-ascii?Q?ZfGH183MgvQPW9v+Q/g9BRqDrcLx8QRG2azMRsqcs+jX+ifq2hhKx/nDm7iX?= =?us-ascii?Q?0kfX44Nxlea5yy4Q0qzJ55+S6m1JnSPQCQ1Zpq07Rg76hblMYoAEmCZfWUXa?= =?us-ascii?Q?rBSz92b4NR4ushX80oz5dQ+rUZsNdFv0+vGiwaEMwkxwymPQ1+KTNgrKMcs/?= =?us-ascii?Q?s4wipwHaHLkqZl8dx3i1Xaki2zoX8b8fFYlcfYRqtue6rp5cIWKrbjE7GWeQ?= =?us-ascii?Q?YPnS8XYrgg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5bf65e5a-b1e3-449a-edd8-08de7606d606 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2026 13:48:06.8714 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qAYMNnxFUh9uuY4SuTzbpIl548Oc9osOceFD7YpM2MJzk+tfDumHxV2O5xIsLHLs X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7610 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260227_054816_132670_B5BD7EF4 X-CRM114-Status: GOOD ( 23.88 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Fri, Feb 27, 2026 at 12:39:28PM +1100, Alexey Kardashevskiy wrote: > > > On 27/2/26 02:04, Jason Gunthorpe wrote: > > On Thu, Feb 26, 2026 at 10:11:56AM +1100, Alexey Kardashevskiy wrote: > > > > The flow would be some thing like.. > > > > 1) Create an IOAS > > > > 2) Create a HWPT. If there is some known upper bound on RMP/etc page > > > > size then limit the HWPT page size to the upper bound > > > > 3) Map stuff into the ioas > > > > 4) Build the RMP/etc and map ranges of page granularity > > > > 5) Call iommufd to adjust the page size within ranges > > > > > > I am about to try this approach now. 5) means splitting bigger pages > > > to smaller and I remember you working on that hitless IO PDEs > > > smashing, do you have something to play with? I could not spot > > > anything on github but do not want to reinvent. Thanks, > > > > I thought this thread had concluded you needed to use the HW engines > > The HW engine has to be used for smashing while DMAing to 2M page > being smashed. It is not needed when the insecure->trusted switch > happens and IOMMU now needs to match already configured RMP. Oh? I'm surprised shared->private is different that private->shared.. Regardless, I think if you go this path you have to stick to 4k IOPTEs and avoid the HW engine. Maybe that is good enough to start. > > for this and if so then KVM should maintain the IOMMU S2 where it can > > synchronize things and access the HW engines? > > I want to explore the idea of using the gmemfd->iommufd notification > mechanism for smashing too (as these smashes are always the result > of page state changes and this requires a notification on its own as > we figured out) and plumb that HW engine to the IOMMU side, > somewhere in the AMD IOMMU driver. Hard to imagine KVM learning > about IOMMU. Equally hard to imagine IOMMU changing the RMP.. Since you explained the HW engine changes both I don't know what you will do. Maybe guestmemfd needs to own the RMP updates and it can somehow invoke the HW engine and co-ordinate all the parts. This sounds very hard as well, so IDK. Jason _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv