From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BN8PR05CU002.outbound.protection.outlook.com (mail-eastus2azon11011009.outbound.protection.outlook.com [52.101.57.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2EEF3D16FD; Tue, 21 Apr 2026 13:47:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.57.9 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776779281; cv=fail; b=F5kzROKecA21AowSCrPAsqyPgLcokyq9BvrIlRZYG3NQoTWZNf0EPcybvy41jdBiQZ2sfeWbAtCvJjgtLgOuW3l7Xan6TUVC9JeidBZX6A9F7m8vGUYUZhSG8o+BU3aUWNX8lgZWa7J+dxEbN5DX9QxTQ9473Laf6g96v0CpIyM= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776779281; c=relaxed/simple; bh=yN+755d/A3qxMxUZ9TmbLYxrlw4OEtD1ZK3Hc5e4fME=; h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To: Content-Type:MIME-Version; b=gElypdcnpp98R3yHQkPSKReJu4HQbE/LBwb2Bet0Tho460wkbEweeeZimJKcykZ7rASG46MQ3bcebwRsePN+ivy6gCvT3nrjXiC5hRPxUXxrphzQhHSDxGPAKubkfj1gAswhhPmYij5yEQrD7wiO1p8D6od3VYYVpODXWrb9E9Q= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=oLFdQpg4; arc=fail smtp.client-ip=52.101.57.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="oLFdQpg4" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Bi4dATKhuQ/olWiPETEdDY0hkn9fQ897SXlmzC93XnvDgpDvF3z3I0/lcW6dTwdwZQbk126+qeZvXNucwBtmR4zKaeGIZ0JB1fXkPGoHRvE6eUbHUCl2ft5GgptGigBZFAHW9DkKk+TO16WwdHQn36GSAsBUFKiZa2JH2BYQtD9WYQt38HDy7+4s2TepG67tv0N4TZTra5N82VOl9OtWAXGw7iihcbyVSTo0Sm6837bZyKhltrecqwau+mlpQ8alB6+krE1xQzYfQXxrkzYIwsiYVXfXZxc4UeiN5t7TB3bOAMMsces8dTGy6I0RNmjaZFN8UAiaOJcuA2gTIZe7tA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BANkajuJjAIf2QXm30c6lkUIxkLy27aSZ2YGGINKVHA=; b=xlhbc+fO2iTOg6DyKqg17E8s5JDbqZdWLIwtC5tR+/6+pTPOYro6NFTRQJK8hx8Sxxfc28XemmPUAYk3cDyD7wdVUlcOQBI3Zd2VlOIIg7WnmRZ5rPKj/tBcHir+rRf3R/wutaqjFQoXUIXYZHkc5oqQb7Ds5XBh7HcToctrlM6glLbSS1UsF6RijnKnOpUUyVOoqRyWEJrF67+IpfY0l3c3SIZ7QjBy75b8ibOfwOBYDCvZUpo2HJ29a14z0tDMu8/08UL7/jV1n90zJk/l7PZun90NNxv8LyJgB7f2WhOEWucjf3V2FQcoOIhZS/98+nvTqRspE2A0XjSUvb/uiw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BANkajuJjAIf2QXm30c6lkUIxkLy27aSZ2YGGINKVHA=; b=oLFdQpg4weYaMROR9v9RW0czWLvaLNnCDS94h4HIlGbrGmIgjApV5U6YIFvODlKHURcg6gMRHCMJtfaeKZMrweYJsJ42aZW44GtmQa3+cn8iC6Zl5D7uShEYdONcAQxhsboI5vZXTGe4TMDAnH4PyT5kcIK/PhtDYyNjo1To5Q9CxRPSWuQdffYWOV3FEIXQQ8cObBEu7Bjjh5l0IJazjaHyB0vT7MwwH/Dal9Bp2Mw4lUIfBXS3GGhTkc/AKjwFGDJ/TW36lXsdijlJOQYIhAKQYVYuzhrcvBPwF7xNzqY+9ElfYCuRv0Phf3iBkNDRTF7hWaXNlbgp5ItsQ1UOIQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS0PR12MB6486.namprd12.prod.outlook.com (2603:10b6:8:c5::21) by MW6PR12MB8836.namprd12.prod.outlook.com (2603:10b6:303:241::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9846.15; Tue, 21 Apr 2026 13:47:46 +0000 Received: from DS0PR12MB6486.namprd12.prod.outlook.com ([fe80::88a9:f314:c95f:8b33]) by DS0PR12MB6486.namprd12.prod.outlook.com ([fe80::88a9:f314:c95f:8b33%4]) with mapi id 15.20.9846.016; Tue, 21 Apr 2026 13:47:46 +0000 Message-ID: <9f30b572-04be-4adc-b5f0-a286ea601996@nvidia.com> Date: Tue, 21 Apr 2026 09:47:39 -0400 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v11 07/20] gpu: nova-core: mm: Add TLB flush support To: Danilo Krummrich Cc: linux-kernel@vger.kernel.org, Miguel Ojeda , Boqun Feng , Gary Guo , Bjorn Roy Baron , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Dave Airlie , Daniel Almeida , Koen Koning , dri-devel@lists.freedesktop.org, rust-for-linux@vger.kernel.org, Nikola Djukic , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Jonathan Corbet , Alex Deucher , Christian Koenig , Jani Nikula , Joonas Lahtinen , Rodrigo Vivi , Tvrtko Ursulin , Huang Rui , Matthew Auld , Lucas De Marchi , Thomas Hellstrom , Helge Deller , Alex Gaynor , Boqun Feng , John Hubbard , Alistair Popple , Timur Tabi , Edwin Peer , Alexandre Courbot , Andrea Righi , Andy Ritger , Zhi Wang , Balbir Singh , Philipp Stanner , Elle Rhumsaa , alexeyi@nvidia.com, Eliot Courtney , joel@joelfernandes.org, linux-doc@vger.kernel.org, amd-gfx@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-fbdev@vger.kernel.org References: <20260415210548.3776595-1-joelagnelf@nvidia.com> <20260415210548.3776595-7-joelagnelf@nvidia.com> <20260416212312.GA667928@joelbox2> <1e3c423e-1cf5-48b1-b012-c4af0eb6b95f@nvidia.com> Content-Language: en-US From: Joel Fernandes In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: MN2PR01CA0063.prod.exchangelabs.com (2603:10b6:208:23f::32) To DS0PR12MB6486.namprd12.prod.outlook.com (2603:10b6:8:c5::21) Precedence: bulk X-Mailing-List: linux-fbdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR12MB6486:EE_|MW6PR12MB8836:EE_ X-MS-Office365-Filtering-Correlation-Id: 88a20582-ec18-4f85-cbfc-08de9fac9177 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024|18002099003|56012099003|22082099003; X-Microsoft-Antispam-Message-Info: s7bAiRe0k05jLoatk32p9vvwxat9RdbYPaFNc6ahrjsyOdHLOpY2yiJf25T5ZVKZRPTnUDy9td2tsRieO8PJto2txfenHu4ARLkYSOZV0nVsZqDyLv8dGHMi+tMK0btoXpob/ugC5fwhvzy6/2V57HmjfkUhEH+YGNVjpmOij1Fq6G+d6pFk9PPXTRZ3KpfCnTGYOFfjPeXxBQBBBopskBmV8dWD8FYrxw0cl7y4O6Km5VFbEUwQUWimr7iT7LvyesaUpHXDaGPYYvm2GJJ37wZZ/jNw20QORmIsJNR0pVfNZEiemJrtgxP0MMDfMoYEhRxkdy5SxKIQvrjD5Bh4cXImDhNGdODxHrBPgwJgMjwu7JkaiWGo0neTyZJJ1Q+KT8PBc0Uff6hTAfIJtIhdqUiRP6z3J+dZiU9IwjtheZ9L7I8SM0ZFXoKTpgEZP2Dh/f/BbYBA3fJ+LmWnpZ2Ax4MLNuOh6dmO23daUGYx53p87D55Zh26vOBbNu2WDp0w7Adx+1kntry1dl3MeKTev0hAe8W8YF+4D8Tu8OvuiKbDYVwseFE2pkjDbrfGJApVTl04fOD/L4xglDz7l1sznK3JtmSZINGPQlkBTPJ6sBjSkLFfcSyIkLo/GKvVRnxn0jsaOTQy3rogBk54cbyeyJMWGLVyOQURBDl75Aj9vCVfugYTLG4qoP8scqd5fxqdTeate+0lGgP6Ogkojy9fFCwmbaFgVD9rTuF9hPAckTQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR12MB6486.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024)(18002099003)(56012099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OHhSZ1ZRaGZWTU1PYWdMTjZDTGlEOFRzRWxxMHFsYVlBM1diT0IvamZkY2F1?= =?utf-8?B?SWJJUTJYNkxQMGZPRmVGUEIyYUhpbTcxazRaRXJEckRoSVpRWVk5d3RCdkQ3?= =?utf-8?B?T2lseGg4OFpBaGJDd1FZSExWeStQdEFpaUpPZTczc2NDWTNuWW9US2xZSDN4?= =?utf-8?B?eWlQaUFMMTNSY0ZOSVRxZ204a0xpQjlMblNhcjlQVTc2ZnNGYWVnUG54b1Fv?= =?utf-8?B?TkpCTnl3WVIrZmwrbGtUTGJRUC9jOXdKQjd2TEhTQVNNUm9kQ01aZU1QT3FR?= =?utf-8?B?MEYxOFdldTRIZld0SG41blJDb2ZGV0VIM2ZXMVl6ZnlZVUlyQ1poYkdwdGRR?= =?utf-8?B?eE9QR0s2OG9HUUhSQ1RDOXhwbFJQWGhtNStiWU1XUWlXQ2Z3QUJSRnVDZFBq?= =?utf-8?B?QmFONHRhUFR1Y25FYVNjNGxrOTQ2RnF3aC9oNlJxK0F1Y3JEUGlRUnpmS2dI?= =?utf-8?B?Vm5mSEcxRnNxT3F0Kyt2blM4WTlNVlZWb0xsRlUzbkoxYzRZR0hWbGtzQmEx?= =?utf-8?B?eEFJN1FLbWFUaEtxdXpEem5JL3lwNWJraXgvOTJISVhHMEl1ekYyN1JiMDVV?= =?utf-8?B?bTRReUtlTGlDNDUvRlZneTdqb2pGZFlHOGovTzZrZUNxLzhvRkdTaGVKNEtu?= =?utf-8?B?cTE2NkZiNjltNVBKUkFmWmd4OFd1am80eHU4TzRSUm95V0xwckN4U2NFVEs3?= =?utf-8?B?MEJxTGQ0VzBKT3owWnViSzhiZ0xEaHpIM2FGQXJKbEE0YW5oVWRjVzNrazZY?= =?utf-8?B?S2JKZVpVcXdoZWtEU3ptUzdoVm9HaHdpMTMrRlhBYVZZcVprUUNNcjBkbnZ0?= =?utf-8?B?bDZFNkh6bE50eDBiZzVxMC9vTHVRYjU1T05DdWI5MXUvcVhvc0ZHMjJ4UUZK?= =?utf-8?B?cTlLMDVxQkl3dUFaaUNyZjVuazk2clR4Y1gwR3IrR25kMDJEUEpYTS96MTBm?= =?utf-8?B?R2ZFUDM2VTFGS2k5cGg4aHFNR3dQd0ZGQ3d1cjJLcUpCaW91RXUva3dzaHpk?= =?utf-8?B?aVJMUSsva2I2cWVYSWd6ZGY5VW1FNDNtUlR2THFlUmt3WUJmek9TRVhXN2pj?= =?utf-8?B?N3dKTW9lbnBmUDEvMndSQkVwYStMcS9yUG9ZcFJQSjFpK0tFdTJqamxZQzdh?= =?utf-8?B?M0YvUWpFZWVXMXI0STdMZERGSzJoTzZ5TmcwMDY4R0dOZk9OMHYzTmtqVkM5?= =?utf-8?B?ZVFINUVVMmdsR2pjbkdVMFRKM2FDTU40MkRwdGJ2V0NIUXB4VmluekVveXpI?= =?utf-8?B?VEtaaWxqVW9lbVNrNDF1UzJFN3g2cXRTN0ZRaXlhdFlrNzZoanZ0Unc2RHYr?= =?utf-8?B?dlAzZDErUHNTSUFnKy9rQ2l2S1k3UnVGMkpQZzRITk1qSGRGSkhVZ3czcFRV?= =?utf-8?B?TkhTWVJleFFtNkFFWVdEcHBGVW5MUlF1N0drRnB2dUkrOTIrMTcwRjdmMGIz?= =?utf-8?B?OWxxQlR3RHFTK0JKS0xxaDFIOFE1VHF4RVFYS3loQW5PSi9KN3ExTyttMzB3?= =?utf-8?B?bUR4cjE0MkVnRjd1VE1ZU3BqS2RiMjRmK1NDN0hUeTVaN1M4ZXU1WTh6YXJw?= =?utf-8?B?cGZmdU1na1k4eDZvcUpyemVhc3JGOTNZVi8rVjl3Tk1nSWhQL1Fnd1hXWTh2?= =?utf-8?B?RHZVWTlBTVpIVFh0ZkxuOG4rbXlVdEdud2UweW01cENOcitlMWJJcWRSdXQ1?= =?utf-8?B?WFNzajlIY1JCT1FrbVNMbUlqOWRxbDN0RDQvRTRobktVdnRPVFhWb2pyMjlM?= =?utf-8?B?WFFBN21MaHdzMDNTNi9kbHliODMvZWQ4VldvRkZKWjNmMTh0SkFhbUE4WElN?= =?utf-8?B?TGx2aTdZTGhPUlFJdlkxNHgrdXlPVHdDekNnMmh2eDVsRzdnLytnSlFjMEZa?= =?utf-8?B?TVVQcURlZVA4VVc0UG9qdHM3ZlpPWU91QW9qZG1lVFlJd3hzelZNTUorOGJN?= =?utf-8?B?dEpxZm1xY2hGcGJzMGhzTTFZeTh2YXZpNC8xQjBpWjV0M2ZuSllHMit0TUs1?= =?utf-8?B?NG5FRjVBdGtOSEpHRmlKWDlMazdaQzJ0OERzbk5yTTlQajFMN2EvOGJJRFR1?= =?utf-8?B?OWhpOG5PNW5kUmppTG13NkdQOGkrUURVZVlTYm13dHZ6R05xUTJGMTltWWs5?= =?utf-8?B?RG1Gbkg0WnFtNUZUdWRvMUNKTUpqZ0RPL2RGWmI5cmYrLzNWOEZlbVdPaWpt?= =?utf-8?B?MUFrRytJcVJtdHJwckpHekpFU21ZK3VzMngrTXRuZ00zMEN4WWZJZEo0L3I5?= =?utf-8?B?TEJ3andkZUZabmRvZFJQMUEvVjFwSjV3anVlNzRGRVcrQnpMRHNyVWR4MnBm?= =?utf-8?B?aWphLzRuRlNXQTMrb21zckFSMDI1ZHl4NzJMWXNVTCtpSllaVlhiQT09?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 88a20582-ec18-4f85-cbfc-08de9fac9177 X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB6486.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Apr 2026 13:47:46.0041 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: BlgT4LXriJeAxNHVSSly1Mgo9Lo0EVnFFLby0fYum9Ln3Zb+6wlyxT//ISJ5ly4vluxGBgADgQR47i6fp6dThg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8836 On 4/16/2026 6:53 PM, Danilo Krummrich wrote: > On Fri Apr 17, 2026 at 12:18 AM CEST, Joel Fernandes wrote: >> On 4/16/2026 5:45 PM, Danilo Krummrich wrote: >>> Why do we need the try_access() dance in the first place? I assume this ends up >>> being called from the BarAccess destructor? >> >> BarAccess is different. The try_access() calls here are in tlb.rs and >> pramin.rs for Bar0. > > Yes, and we shouldn't need them in the first place; we should have a > &Device in all call paths this is called from. > >>> If so, I think this is solvable. Gary and me are currently working on >>> higher-ranked types and a chained Devres type. >> >> Hmm, the issue here is we cannot hold revocable guard while sleeping, but >> we have read the bar as a condition in the body of the poll. > > No, you should just require a &Device; or maybe we can utilize the > mentioned higher-ranked types and DevresChain once we have it. But in any case > you shouldn't need try_access() here. > >>> With that, such use-cases should be cleanly solvable without the need for >>> try_access(). >>> >>> Besides that, I can't find where BarAccess is ever constructed. >> >> BarUser::map() constructs it. > > I'm well aware, but absolutely nothing calls BarUser::map(). :) > >>> It already has a lifetime 'a for &'a Bar1, so I don't see why you can't do the >>> same for Bar0.> >>> But again, I don't see this being constructed and I'm not sure the whole >>> construct works in the first place. >> >> BarAccess uses &'a Bar1 because it's a short-lived scoped object. In long >> lived objects I am trying to avoid this. > > Don't get me wrong, if a lifetime is sufficient -- that's great! But I'm > suspicious whether it actually is, since BarAccess is never actually constructed > and hence I can't see how it would be used. Actually, it is constructed in the self-test. Maybe you missed the self-test patch? #[cfg(CONFIG_NOVA_MM_SELFTESTS)] fn run_mm_selftests(self: Pin<&mut Self>, pdev: &pci::Device) -> Result { // PRAMIN aperture self-tests. crate::mm::pramin::run_self_test(pdev.as_ref(), self.mm.pramin())?; // BAR1 self-tests. let bar1 = Arc::pin_init(pdev.iomap_region(1, c"nova-core/bar1"), GFP_KERNEL)?; let bar1_access = bar1.access(pdev.as_ref())?; crate::mm::bar_user::run_self_test( pdev.as_ref(), &self.mm, bar1_access, self.gsp_static_info.bar1_pde_base, self.spec.chipset, )?; Ok(()) } However, you do raise a good point -- may we cannot assume that BAR1 reference is short lived. One thing though is typically BAR1 size is limited, so we should not allow anyone to use it for long-lived usecases to prevent filling up the small BAR1 aperture into VRAM. The user of the bar1 code is vGPU at the moment, I am sure in the future we will have other uses of the CPU directly accessing VRAM. Zhi, can you share any thoughts here? Is BAR1 for vGPU expected to be held long-lived? I agree with your points about Device and I will explore that as well.