From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11011045.outbound.protection.outlook.com [52.101.62.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 789713D47C3; Mon, 31 Aug 2026 08:52:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.45 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788166366; cv=fail; b=lg2UIcAdCM/mXDdprH6yHMQOoKYt4uzDKxm8SjRDWP9qYWqVqyoeOPaTjZWkNVOj2xWw3zv5LJPpXm5ZRfv/ds447Zo1d1AvUg87QymZ6htoirYiwalrIh9Z12lPhsh9OOS/soFveI27TmNCGc1kb6d133QJe6mSBIhWUirrYEk= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788166366; c=relaxed/simple; bh=i+Y1HBAyaZ/xrtZTK0i8uAEG9sQ/7RTKpAxWv3XM/1E=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=ovZf/kvSNfSuAUL+xAtLR4UhhBCawA9gk2psdLtCIAXX1g09RnE+IeAYwLSEwhPiR0eoECERoxyVwcuvYz8CbD0LL++B9cGjQV53Rt0dWKxBAVPC8FYktM81C+bDWKuvDN5UcwmimenfLrACfHRmOMdwAmuqH6xFc8IY4AlNY2s= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Q6XCxOOq; arc=fail smtp.client-ip=52.101.62.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Q6XCxOOq" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=H4BeOGrtjX6Qp1IC5L/TOSWeyZzJdgpX48lFmSeoltNrlJVUmutswmnm0DjbCoSTp2mh0PbSaOtVwENblqRDcNZixepGvfEjFvClDSn1z8lSvK3yKpn08K7gtq0IPbUEylyPCQZ9HmKEzWA+vOhNFarA6uX03qu1EEdIvhYG/0gHFJBrpWOWrRwX0l5apfHoo8s+L7yIbZZiotTW+N2ralQc+eFmEK4DM8KBN9hqr91X3TDzoxbw4AU0/W1dDlcBdo8Dki+tRRVImnAt0XEDVX0GjL167aAYqzSg7bibREvUgPggMYi+r+IL4S8ObEnGVCjCsB79o1qTcDpzIVlS0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zPhYQ7x9APpu7aSRNjARiXg57eCh8M208nw4SrUpdak=; b=ZcLiPg4efVFJuxua1QTA1rAhieac4gu7+kdYJCA78araJmPKOKgwvWlz3DYtDxxJx2jf9QxKAISIiiYlJV+3VirZmdZv0ekksd93aM4yXEpOo9Ap9q9sNN49TCKYgkysNMCJW3S4TdoA58ZcxjjYuTMBqQ3v5EXIrLPZkNmNZfks5HzylWFItky/Z+/jOaj9+eAr3UJTaW6UmGSHvSDwLegPG6phKUrLdrkkTnUe43UhOx0J/xfNXvnHVKzQ9MzSL4BN3AkOpr/AAEuUI6uLxQr5I+DtkDfgnDU+EhtmU2OvgpXPMsXHgC69FR+NKftnz5FN79NroS8lGOIxeyBW3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zPhYQ7x9APpu7aSRNjARiXg57eCh8M208nw4SrUpdak=; b=Q6XCxOOqQ3OC1OnCP4vgRAm+/Yv1g/+YLGWikImTjjGD4pIJNEyADd3NePNXmuKV1nZjV5+txYgymtA5qtnAuUxJkBB4h5ybDTDiEQdEIFu/IrGL5hhYD/GryXLVCU6si+qhM9DMrh31ThHsIVEyo/ryNGyjaf5q3gBOusZ6jH/TZQrtIbuoH6pvXEgjVx9ZZvYFFR2MTX9IQ5QmtUNhBZy75QT/Iqz+1nSKrfAf8UJwvIsNOjIxiWOQNTZClL++Igd4NI3fif6km1XO1e3j54eHtmEOlJnjjjMUQAhnWS2gkwCOPfsUVyUoAk2AZC5zMfS66qjASmWSLWRBtpYlvQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BL0PR12MB2370.namprd12.prod.outlook.com (2603:10b6:207:47::27) by BN7PPF683A477A9.namprd12.prod.outlook.com (2603:10b6:40f:fc02::6d3) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Mon, 31 Aug 2026 08:52:37 +0000 Received: from BL0PR12MB2370.namprd12.prod.outlook.com ([fe80::86cf:c3ec:2cf5:74c8]) by BL0PR12MB2370.namprd12.prod.outlook.com ([fe80::86cf:c3ec:2cf5:74c8%7]) with mapi id 15.21.0360.008; Mon, 31 Aug 2026 08:52:37 +0000 Date: Mon, 31 Aug 2026 16:52:29 +0800 From: Richard Cheng To: "Lucero Palau, Alejandro" Cc: dave@stgolabs.net, jic23@kernel.org, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, djbw@kernel.org, iweiny@kernel.org, ming.li@zohomail.com, gourry@gourry.net, rrichter@amd.com, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, newtonl@nvidia.com, kristinc@nvidia.com, kaihengf@nvidia.com, kobak@nvidia.com Subject: Re: [RFC PATCH 0/3] cxl: Auto-create a region for Type-2 memdev attach Message-ID: References: <20260805074042.30173-1-icheng@nvidia.com> <53689d31-3707-4c15-9eb9-eb6c77ab3ca0@amd.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <53689d31-3707-4c15-9eb9-eb6c77ab3ca0@amd.com> X-ClientProxiedBy: SI2PR04CA0008.apcprd04.prod.outlook.com (2603:1096:4:197::20) To BL0PR12MB2370.namprd12.prod.outlook.com (2603:10b6:207:47::27) Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL0PR12MB2370:EE_|BN7PPF683A477A9:EE_ X-MS-Office365-Filtering-Correlation-Id: 64180029-ea0d-44da-0c2a-08df073d34cd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|366016|23010399003|56012099006|3023799007|11063799006|10067099003|4143699003|6133799003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: jprvyQLe3Z1hbnyMj6ahEPfo4nLAWs8CIyR19Ta2UuUYeFkR8CUG4vYo9v7aoi9khnUynvofXxBDVNzZKzYLX4ZD+Yc3C7yMohJz1mR5oVmQxNbzInQOffC/JeCCitjSTINk+ez6CAXYsbphyuUyXfVlLVSPfzCuuaNu/UW5d1U8cnAemN6n0WJCMM0N/IJ6xNhSWEktmvt5POO3PQQRlQIJimQEv8kyZouHiVCBY6uf6u6+IKiAl8DOzMxrFeVIfnB752ZZXY1NziR11ZbgVjBiUJMf5QrS8jf0TSJO+opJiqn2jkZK6eCqlh37XI+BgpzuqEv+hvp8FGCM1gJObW+pXH1deBiASQu/z1Q9Khu+lwZMBgyvXmaJbVI+Vv5qGJfDviWLl3eUtl4uX1q7VgbsjTJhQmbAgtrSczFpGNA1caSimGWXsTFqJBJgiWHy7aigVcegBl9PXJAEG9r+0iM3a6dai7kar1CZO2heT+zZjI/S2+WydrwEfWkEZmFVSWIDEjKvokS3/9uNN5AcKtWiHrj88TP5mxg0/vlzS8/DOvz0OFVTdKN8oXaoLNc/ckDJhqSJLdpM5ocGREphQSO+VSzhrCRXxoKM3+hEwBniRhIya1c+fGS4zMAjlQAum8DaKTPMpa3gOsSoimGEXwdY3247fCMFIeW0iW5TSSo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR12MB2370.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(366016)(23010399003)(56012099006)(3023799007)(11063799006)(10067099003)(4143699003)(6133799003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?qQCKS6/+VHPKXuvXMGGs9ObRAqMdykbanFFDe9hvYmkzS2kCHJl0MM4hB+ar?= =?us-ascii?Q?QbvICE6wh2yRhPJNjOEw9SZ9Qph9trnIb16oDsMjuFbqsWUUe3Bl05EodVOp?= =?us-ascii?Q?pZXnscaafe+mGFTcPkwNiaQIh8UIsYb2sC9J+5bI24Xz8Ikh7Dgo+Fr/9YPP?= =?us-ascii?Q?rqrd5SOrIYcGQdxJKLpW+opKtvWDCNUWtDMQTghtiUIADoUrYwQAydCh+xRj?= =?us-ascii?Q?60LsC1a/mxShMfvUomGgmiH+T5TAswEGN8zoaFv4THNy5lBJTT89thdNmFuy?= =?us-ascii?Q?BiP7yafL/91kJt4LdZ/hIJlW7VaS+3y5lvyWZsz59R9Lit3jCdL7MpUluh2h?= =?us-ascii?Q?2IY4GKmMWF4Xura+NBophVhskdbX2tyb2KGAZRvdl3Isx/nvYlOcModjuB5M?= =?us-ascii?Q?i/NX1ceK1Xao1X71bZ1UcC/to2jwYVzphpa4MPX6d8vzMDZJfYJfGws5JauF?= =?us-ascii?Q?g+vEj+hxp0p40EyxU7oza8yaS4iUkNPSGQJPCYz2KiU4DFoehABvJD6rhk+i?= =?us-ascii?Q?ypLDMLsN0ocQRSxrzh3CYzpR0xBd9hDDPc3Ql6pmOpt6k0WvS+R7DVPYKZzn?= =?us-ascii?Q?Obw4aGKPBCfpQewnEBbwjznzvMj4Te5yCL1UTsaWuFlfyQ1WMRsgE0RPm2sA?= =?us-ascii?Q?amQNf/y7dB4u3wadD6ZvWSCVt1AZZoFrz2z1PRwgCubErRjoKlOu6CyQe/tk?= =?us-ascii?Q?Truaas4XP3MED2m4bIQ0LYN0DhijrgVm0PEcKRiiYlqMdq1Ws2E9I+2OrA3z?= =?us-ascii?Q?vNUPn//Of3XR2ay9akBCShmMQ49PNw0rWnK7kMYzPrsCF3JXOi2CooKUq/EF?= =?us-ascii?Q?EDy6veiKby8xDDsofOwkAusjRlUSlMreyw3wecG7T/IGfm0u0aUPjVz5zk3h?= =?us-ascii?Q?vekOBOH4iKb965SeEwaiGqTsA6zIoL8IA8L3d0K4YPmFdErmhsEfcnkpvPEc?= =?us-ascii?Q?/IS7NoA9bdK6nKOMQ3vdA0XSbprRE90kBlFIcdOPmnBQBlTMIWh4f7Smgv/Q?= =?us-ascii?Q?mp7AK7MNJKcepiUc6krqxMWHoKbw7pjuWe7UcMCz5vQNDy1+DMXm/fB3KlTb?= =?us-ascii?Q?zOSg9MoDhCTXtYj1asgqxljBmXtKjPnrfuPSePahDuYgqAxhOFbntOoQnjPw?= =?us-ascii?Q?5a+6mm0XvZgZ3SmIeXyRpgiTDmz8jJ4ydBEZIsapEaGX9jLajXtEjwec1fyj?= =?us-ascii?Q?It+6uMgBCi/WQRktZDHyZABfu6r+xWuCXISxWOj+cMamSf96Km/Ti5wA4Jxt?= =?us-ascii?Q?oVIlh1aNfoaU6b56bhvqsbcRvWJDDIX4/IXBFa0Hx/Rbj7ksBVnhk8x8TqOa?= =?us-ascii?Q?J1DNoNSIqAHNw5K2YScLDdwFifCVq1n8XJ3eWQUy+rZCA1w6XxWAUaBUTsE4?= =?us-ascii?Q?EaCAvP8mP2LLA7ZgQHbtcj+RYUnK8vu1N9EIkLH2CJLsVoAC/dFwYUq/PLPZ?= =?us-ascii?Q?I/qorQaYjeZn8k24+6CV+qSVrNe0Jgo5qRpB5Qbnv2x0QAZuqgdjRUvx7Ufs?= =?us-ascii?Q?F7oe2evHPeYYp8CKlTqBwXCxhaETmaquwHkqpILFyYSlfu66zRIksHydzc3l?= =?us-ascii?Q?JFxU+RCJJgFxzZplGfxjgdSueOi7mrewFh22CoAKzBsjl+15aTtbJk4NWiZS?= =?us-ascii?Q?g6k5au7Z+gEjYA/yprmUy8i9U9w62Toc0Wr2i49Ybb0ItW1GqSA1SQ4bcnx9?= =?us-ascii?Q?F+l86PoxD/3xSBJ/Hh1mq/ruUhFSInL6XZASB06DzWsGJHc/?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 64180029-ea0d-44da-0c2a-08df073d34cd X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB2370.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Aug 2026 08:52:37.4402 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: IFc0aSU15NMOZpZ4sPXk5oAqSFvDwINbEn7td1LGVGwz1m5n5+LmI5KxUalXedgkS3M1l3SUfwoCYobFNLMz7Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PPF683A477A9 On Tue, Aug 25, 2026 at 10:50:06AM +0800, Lucero Palau, Alejandro wrote: > Hi Richard, > > On 20/08/2026 10:41, Richard Cheng wrote: > > On Wed, Aug 12, 2026 at 10:58:02AM +0800, Alejandro Lucero Palau wrote: > > > Hi Richard, > > > > > > > > > Some comments below. > > > > > > > > > Thanks! > > > > > > > > > On 8/5/26 08:40, Richard Cheng wrote: > > > > > > Hi Alejandro, > > > > Thanks for the review and explanation. I've read them all. > > > > I think you are right that this RFC doesn't currently have a production platform > > where the system FW publishes a Type-2 CFMWS but leaves the EP decoder > > uncommitted. > > > > However, the config appears to be permitted by the CXL model. A CFMWS describes > > a FW-established root HPA window and the restrictions governing its use, > > including Type-2 v.s. Type-3 and volatile v.s. PMEM. The CFMWS def also > > describes OSPM assigning HPA ranges from those windows to discovered CXL.mem > > devices [1]. > > > Right. I'm not saying this should not be supported, just pointing out the > use case does not make sense with current BIOS functionality. I think BIOS > will/could support a config option for just leaving a Type2 HDM uncommitted, > but then why the kernel should do the same a default BIOS config would do? > Agreed. The kernel shouldn't recreate the config that BIOS would normally provide. And that's why I think we should move region createion out of devm_cxl_probe_mem(). That helper should discover and attach to an already committed region. If FW leaves the decoders unconfigured intentionally , a driver may explicitly request a region and provide the size it needs. In my mind the new model should be - FW-committed config is only discovered and attached - an uncommitted config remains untouched unless a driver explicitly requests it - CXL core supplieds the allocation, validation, programming, accounting and teardown mechnism - the requesting driver owns the policy and the use of the region So far I think PMEM reconstruction form label data maybe be a potential use case, though it's not provided in kernel right now, we can work on it in the future, or work on that one first and we'll continue the auto-create region part. What do you think ? > > > > > The Linux CXL doc similarly states that only root decoders are required to be > > programmed during probe. Switch and EP decoder may remain available for runtime > > programming when the platform supports it [2]. > > > Tangential to this discussion, but I have problems with this assertion. Any > switch or EP HDM programming will need a root port HDM programming as well. > Not sure which root decoders will need to be programmed at boot time: a > CFMWS is "programmed" by the BIOS and root decoders will need to be > programmed as well for any Type2/switch found with an enabled link. > > Here root decoder I mean the logical object created from a CFMWS. I didn't mean switch or EP decoder can be programmed independently. My point was that FW establishes the CFMWS window, while the platform may leave downstream decoder path for later programming. > > > > I raise the RFC intended for the question of how Linux should support that > > architecturally permitted config. > > > > The cxl_test config added in patch 3/3 constructs this scenario synthetically. > > This demonstrates the proposed kernel behavior, but I agree I don't know > > whether there exists a deployed FW scenario. > > > > > > And I agree that devm_cxl_probe_mem() shouldn't silently change from > > "attach to a FW-established region" into "allocate resources and program a new > > region". Those operations should have different semantics and ownership > > expectations. > > > Glad with the consensus :-) > > > > > > I am planning to rebase onto cxl/nexxt and rework the proposal as the following, > > please take a look and see if that matches your imagination or not. > > * Keep devm_cxl_probe_mem() behavior unchanged for FW-committed regions > > * Make region creation an explicit request from the accelerator provider, > > rather than an automatic fallback during memdev attach > > * Have the provider specify the required size. CXL core shouldn't assume > > that it maybe consume the entire volatile DPA partition as you mentioned. > > * Separate the reusable region-provisioning mechanism from the initial Type-2 > > policy. > > * The common mechanism should handle HPA/DPA allocation, decoder-path > > construction, commit , rollback and managed teardown. > > * The initial type-2 caller would constrain that to volatile DEVMEM, IW=1 > > and a provider-requested size. > > > > Oh and I'll replace "x1" with "IW=1" and explain the initial decoder, > > root-selection and granularity restriction more clearly. > > > > How does that sound to you ? > > > It sounds perfect! > > > FWIW, you likely saw Gregory's comment (discord) on this work requiring the > support for PMEM or at least the awareness PMEM support will need to use > same interface. His opinion and mine came from Dan's vision on this, and > your work will be the base for such PMEM support. I do not have an impending > reason for working on this PMEM support, but I am really interested in how > Type3 PMEMs can leverage CXL.mem for improving storage needs, and currently > reading/thinking about all this ... > > > Thanks! > Hmmm for this part I have no idea for now, I'll study more and discuss with you guys. Best regards, Richard Cheng. > > > [1]: https://computeexpresslink.org/wp-content/uploads/2024/02/CEDT_ECN_1.0A_Eval.pdf > > [2]: https://docs.kernel.org/driver-api/cxl/linux/cxl-driver.html#runtime-programming > > > > Best regards, > > Richard Cheng. > > > > Testing result is in the following. > > > > - Built clean with clang/LLVM on arm64 > > > > - cxl_test, type2_test=1. accel0 takes the unchanged attach path. accel1 > > > > drives auto_create -> a committed 512 MB RAM region. The test asserts > > > > the 512 MB HPA range. committed state and 256 byte granularity confirmed > > > > via sysfs. > > > > - Unbind tears the region down with no orphaned decoder, rebind re-creates > > > > a fresh committed region. > > > > - Mock test only. Real accelerators whose FW commits a decoder take the > > > > attach path, and vfio-cxl binds only FW-committed devices, so auto-create > > > > has no real-HW caller yet. > > > > > > > > Best regards, > > > > Richard Cheng. > > > > > > > > Richard Cheng (3): > > > > cxl/region: Reset software-created regions on memdev detach > > > > cxl/region: Auto-create a region for memdev attach > > > > cxl/test: Exercise Type-2 automatic region creation > > > > > > > > drivers/cxl/core/region.c | 422 +++++++++++++++++++++++++++++---- > > > > tools/testing/cxl/test/accel.c | 7 + > > > > tools/testing/cxl/test/cxl.c | 61 ++++- > > > > 3 files changed, 439 insertions(+), 51 deletions(-) > > > > > > > > > > > > base-commit: 1c6b4ceafc3b994871c29340e0c1ddb0af5800e7