[PATCH v19 00/22] Type2 device basic support

[PATCH v19 00/22] Type2 device basic support

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

The patchset should be applied on the described base commit then applying
Terry's v11 about CXL error handling. The first 3 patches come from Dan's
for-6.18/cxl-probe-order branch.

v19 changes:

  Removal of cxl_acquire_endpoint and driver callback for unexpected cxl
  module removal. Dan's patches made them unnecessary.

  patch 4: remove code already moved by Terry's patches (Ben Cheatham)

  patch 6: removed unrelated change (Ben Cheatham)

  patch 7: fix error report inconsistencies (Jonathan, Dave)

  patch 9: remove unnecessary comment (Ben Cheatham)

  patch 11: fix __free usage (Jonathan Cameron, Ben Cheatham)

  patch 13: style fixes (Jonathan Cameron, Dave Jiag)

  patch 14: move code to previous patch (Jonathan Cameron)

  patch 18: group code in one locking (Dave Jian)
	    use __free helper (Ben Cheatham)


v18 changes:

  patch 1: minor changes and fixing docs generation (Jonathan, Dan)
 
  patch4: merged with v17 patch5

  patch 5: merging v17 patches 6 and 7

  patch 6: adding helpers for clarity

  patch 9:
	- minor changes (Dave)
	- simplifying flags check (Dan)

  patch 10: minor changes (Jonathan)

  patch 11:
	- minor changes (Dave)
	- fix mess (Jonathan, Dave)

  patch 18: minor changes (Jonathan, Dan)
  
v17 changes: (Dan Williams review)
 - use devm for cxl_dev_state allocation
 - using current cxl struct for checking capability registers found by
   the driver.
 - simplify dpa initialization without a mailbox not supporting pmem
 - add cxl_acquire_endpoint for protection during initialization
 - add callback/action to cxl_create_region for a driver notified about cxl
   core kernel modules removal.
 - add sfc function to disable CXL-based PIO buffers if such a callback
   is invoked.
 - Always manage a Type2 created region as private not allowing DAX.

v16 changes:
 - rebase against rc4 (Dave Jiang)
 - remove duplicate line (Ben Cheatham)

v15 changes:
 - remove reference to unused header file (Jonathan Cameron)
 - add proper kernel docs to exported functions (Alison Schofield)
 - using an array to map the enums to strings (Alison Schofield)
 - clarify comment when using bitmap_subset (Jonathan Cameron)
 - specify link to type2 support in all patches (Alison Schofield)

  Patches changed (minor): 4, 11

v14 changes:
 - static null initialization of bitmaps (Jonathan Cameron)
 - Fixing cxl tests (Alison Schofield)
 - Fixing robot compilation problems

  Patches changed (minor): 1, 4, 6, 13

v13 changes:
 - using names for headers checking more consistent (Jonathan Cameron)
 - using helper for caps bit setting (Jonathan Cameron)
 - provide generic function for reporting missing capabilities (Jonathan Cameron)
 - rename cxl_pci_setup_memdev_regs to cxl_pci_accel_setup_memdev_regs (Jonathan Cameron)
 - cxl_dpa_info size to be set by the Type2 driver (Jonathan Cameron)
 - avoiding rc variable when possible (Jonathan Cameron)
 - fix spelling (Simon Horman)
 - use scoped_guard (Dave Jiang)
 - use enum instead of bool (Dave Jiang)
 - dropping patch with hardware symbols

v12 changes:
 - use new macro cxl_dev_state_create in pci driver (Ben Cheatham)
 - add public/private sections in now exported cxl_dev_state struct (Ben
   Cheatham)
 - fix cxl/pci.h regarding file name for checking if defined
 - Clarify capabilities found vs expected in error message. (Ben
   Cheatham)
 - Clarify new CXL_DECODER_F flag (Ben Cheatham)
 - Fix changes about cxl memdev creation support moving code to the
   proper patch. (Ben Cheatham)
 - Avoid debug and function duplications (Ben Cheatham)

v11 changes:
 - Dropping the use of cxl_memdev_state and going back to using
   cxl_dev_state.
 - Using a helper for an accel driver to allocate its own cxl-related
   struct embedding cxl_dev_state.
 - Exporting the required structs in include/cxl/cxl.h for an accel
   driver being able to know the cxl_dev_state size required in the
   previously mentioned helper for allocation.
 - Avoid using any struct for dpa initialization by the accel driver
   adding a specific function for creating dpa partitions by accel
   drivers without a mailbox.

v10 changes:
 - Using cxl_memdev_state instead of cxl_dev_state for type2 which has a
   memory after all and facilitates the setup.
 - Adapt core for using cxl_memdev_state allowing accel drivers to work
   with them without further awareness of internal cxl structs.
 - Using last DPA changes for creating DPA partitions with accel driver
   hardcoding mds values when no mailbox.
 - capabilities not a new field but built up when current register maps
   is performed and returned to the caller for checking.
 - HPA free space supporting interleaving.
 - DPA free space droping max-min for a simple alloc size.

v9 changes:
 - adding forward definitions (Jonathan Cameron)
 - using set_bit instead of bitmap_set (Jonathan Cameron)
 - fix rebase problem (Jonathan Cameron)
 - Improve error path (Jonathan Cameron)
 - fix build problems with cxl region dependency (robot)
 - fix error path (Simon Horman)

v8 changes:
 - Change error path labeling inside sfc cxl code (Edward Cree)
 - Properly handling checks and error in sfc cxl code (Simon Horman)
 - Fix bug when checking resource_size (Simon Horman)
 - Avoid bisect problems reordering patches (Edward Cree)
 - Fix buffer allocation size in sfc (Simon Horman)

v7 changes:

 - fixing kernel test robot complains
 - fix type with Type3 mandatory capabilities (Zhi Wang)
 - optimize code in cxl_request_resource (Kalesh Anakkur Purayil)
 - add sanity check when dealing with resources arithmetics (Fan Ni)
 - fix typos and blank lines (Fan Ni)
 - keep previous log errors/warnings in sfc driver (Martin Habets)
 - add WARN_ON_ONCE if region given is NULL

v6 changes:

 - update sfc mcdi_pcol.h with full hardware changes most not related to
   this patchset. This is an automatic file created from hardware design
   changes and not touched by software. It is updated from time to time
   and it required update for the sfc driver CXL support.
 - remove CXL capabilities definitions not used by the patchset or
   previous kernel code. (Dave Jiang, Jonathan Cameron)
 - Use bitmap_subset instead of reinventing the wheel ... (Ben Cheatham)
 - Use cxl_accel_memdev for new device_type created (Ben Cheatham)
 - Fix construct_region use of rwsem (Zhi Wang)
 - Obtain region range instead of region params (Allison Schofield, Dave
   Jiang)

v5 changes:

 - Fix SFC configuration based on kernel CXL configuration
 - Add subset check for capabilities.
 - fix region creation when HDM decoders programmed by firmware/BIOS (Ben
   Cheatham)
 - Add option for creating dax region based on driver decission (Ben
   Cheatham)
 - Using sfc probe_data struct for keeping sfc cxl data

v4 changes:

 - Use bitmap for capabilities new field (Jonathan Cameron)
 - Use cxl_mem attributes for sysfs based on device type (Dave Jian)
 - Add conditional cxl sfc compilation relying on kernel CXL config (kernel test robot)
 - Add sfc changes in different patches for facilitating backport (Jonathan Cameron)
 - Remove patch for dealing with cxl modules dependencies and using sfc kconfig plus
   MODULE_SOFTDEP instead.

v3 changes:

 - cxl_dev_state not defined as opaque but only manipulated by accel drivers
   through accessors.
 - accessors names not identified as only for accel drivers.
 - move pci code from pci driver (drivers/cxl/pci.c) to generic pci code
   (drivers/cxl/core/pci.c).
 - capabilities field from u8 to u32 and initialised by CXL regs discovering
   code.
 - add capabilities check and removing current check by CXL regs discovering
   code.
 - Not fail if CXL Device Registers not found. Not mandatory for Type2.
 - add timeout in acquire_endpoint for solving a race with the endpoint port
   creation.
 - handle EPROBE_DEFER by sfc driver.
 - Limiting interleave ways to 1 for accel driver HPA/DPA requests.
 - factoring out interleave ways and granularity helpers from type2 region
   creation patch.
 - restricting region_creation for type2 to one endpoint decoder.

v2 changes:

I have removed the introduction about the concerns with BIOS/UEFI after the
discussion leading to confirm the need of the functionality implemented, at
least is some scenarios.

There are two main changes from the RFC:

1) Following concerns about drivers using CXL core without restrictions, the CXL
struct to work with is opaque to those drivers, therefore functions are
implemented for modifying or reading those structs indirectly.

2) The driver for using the added functionality is not a test driver but a real
one: the SFC ethernet network driver. It uses the CXL region mapped for PIO
buffers instead of regions inside PCIe BARs.

RFC:

Current CXL kernel code is focused on supporting Type3 CXL devices, aka memory
expanders. Type2 CXL devices, aka device accelerators, share some functionalities
but require some special handling.

First of all, Type2 are by definition specific to drivers doing something and not just
a memory expander, so it is expected to work with the CXL specifics. This implies the CXL
setup needs to be done by such a driver instead of by a generic CXL PCI driver
as for memory expanders. Most of such setup needs to use current CXL core code
and therefore needs to be accessible to those vendor drivers. This is accomplished
exporting opaque CXL structs and adding and exporting functions for working with
those structs indirectly.

Some of the patches are based on a patchset sent by Dan Williams [1] which was just
partially integrated, most related to making things ready for Type2 but none
related to specific Type2 support. Those patches based on Dan´s work have Dan´s
signing as co-developer, and a link to the original patch.

A final note about CXL.cache is needed. This patchset does not cover it at all,
although the emulated Type2 device advertises it. From the kernel point of view
supporting CXL.cache will imply to be sure the CXL path supports what the Type2
device needs. A device accelerator will likely be connected to a Root Switch,
but other configurations can not be discarded. Therefore the kernel will need to
check not just HPA, DPA, interleave and granularity, but also the available
CXL.cache support and resources in each switch in the CXL path to the Type2
device. I expect to contribute to this support in the following months, and
it would be good to discuss about it when possible.

[1] https://lore.kernel.org/linux-cxl/98b1f61a-e6c2-71d4-c368-50d958501b0c@intel.com/T/

Alejandro Lucero (21):
  cxl/mem: Arrange for always-synchronous memdev attach
  cxl/port: Arrange for always synchronous endpoint attach
  cxl: Add type2 device basic support
  sfc: add cxl support
  cxl: Move pci generic code
  cxl: allow Type2 drivers to map cxl component regs
  cxl: Support dpa initialization without a mailbox
  cxl: Prepare memdev creation for type2
  sfc: create type2 cxl memdev
  cxl: Define a driver interface for HPA free space enumeration
  sfc: get root decoder
  cxl: Define a driver interface for DPA allocation
  sfc: get endpoint decoder
  cxl: Make region type based on endpoint type
  cxl/region: Factor out interleave ways setup
  cxl/region: Factor out interleave granularity setup
  cxl: Allow region creation by type2 drivers
  cxl: Avoid dax creation for accelerators
  sfc: create cxl region
  cxl: Add function for obtaining region range
  sfc: support pio mapping based on cxl

Dan Williams (1):
  cxl/mem: Introduce a memdev creation ->probe() operation

 drivers/cxl/Kconfig                   |   2 +-
 drivers/cxl/core/core.h               |   9 +-
 drivers/cxl/core/hdm.c                |  85 ++++++
 drivers/cxl/core/mbox.c               |  63 +---
 drivers/cxl/core/memdev.c             | 209 +++++++++----
 drivers/cxl/core/pci.c                |  63 ++++
 drivers/cxl/core/port.c               |   1 +
 drivers/cxl/core/region.c             | 418 +++++++++++++++++++++++---
 drivers/cxl/core/regs.c               |   2 +-
 drivers/cxl/cxl.h                     | 125 +-------
 drivers/cxl/cxlmem.h                  |  90 +-----
 drivers/cxl/cxlpci.h                  |  21 +-
 drivers/cxl/mem.c                     | 146 +++++----
 drivers/cxl/pci.c                     |  88 +-----
 drivers/cxl/port.c                    |  46 ++-
 drivers/cxl/private.h                 |  17 ++
 drivers/net/ethernet/sfc/Kconfig      |  10 +
 drivers/net/ethernet/sfc/Makefile     |   1 +
 drivers/net/ethernet/sfc/ef10.c       |  50 ++-
 drivers/net/ethernet/sfc/efx.c        |  15 +-
 drivers/net/ethernet/sfc/efx.h        |   1 -
 drivers/net/ethernet/sfc/efx_cxl.c    | 165 ++++++++++
 drivers/net/ethernet/sfc/efx_cxl.h    |  40 +++
 drivers/net/ethernet/sfc/net_driver.h |  12 +
 drivers/net/ethernet/sfc/nic.h        |   3 +
 include/cxl/cxl.h                     | 291 ++++++++++++++++++
 include/cxl/pci.h                     |  21 ++
 tools/testing/cxl/Kbuild              |   1 -
 tools/testing/cxl/test/mem.c          |   5 +-
 tools/testing/cxl/test/mock.c         |  17 --
 30 files changed, 1476 insertions(+), 541 deletions(-)
 create mode 100644 drivers/cxl/private.h
 create mode 100644 drivers/net/ethernet/sfc/efx_cxl.c
 create mode 100644 drivers/net/ethernet/sfc/efx_cxl.h
 create mode 100644 include/cxl/cxl.h
 create mode 100644 include/cxl/pci.h


base-commit: f11a5f89910a7ae970fbce4fdc02d86a8ba8570f
prerequisite-patch-id: 44c914dd079e40d716f3f2d91653247eca731594
prerequisite-patch-id: b13ca5c11c44a736563477d67b1dceadfe3ea19e
prerequisite-patch-id: d0d82965bbea8a2b5ea2f763f19de4dfaa8479c3
prerequisite-patch-id: dd0f24b3bdb938f2f123bc26b31cd5fe659e05eb
prerequisite-patch-id: 2ea41ec399f2360a84e86e97a8f940a62561931a
prerequisite-patch-id: 367b61b5a313db6324f9cf917d46df580f3bbd3b
prerequisite-patch-id: 1805332a9f191bc3547927d96de5926356dac03c
prerequisite-patch-id: 40657fd517f8e835a091c07e93d6abc08f85d395
prerequisite-patch-id: 901eb0d91816499446964b2a9089db59656da08d
prerequisite-patch-id: 79856c0199d6872fd2f76a5829dba7fa46f225d6
prerequisite-patch-id: 6f3503e59a3d745e5ecff4aaed668e2d32da7e4b
prerequisite-patch-id: e9dc88f1b91dce5dc3d46ff2b5bf184aba06439d
prerequisite-patch-id: 196fe106100aad619d5be7266959bbeef29b7c8b
prerequisite-patch-id: 7e719ed404f664ee8d9b98d56f58326f55ea2175
prerequisite-patch-id: 560f95992e13a08279034d5f77aacc9e971332dd
prerequisite-patch-id: 8656445ee654056695ff2894e28c8f1014df919e
prerequisite-patch-id: 001d831149eb8f9ae17b394e4bcd06d844dd39d9
prerequisite-patch-id: 421368aa5eac2af63ef2dc427af2ec11ad45c925
prerequisite-patch-id: 18fd00d4743711d835ad546cfbb558d9f97dcdfc
prerequisite-patch-id: d89bf9e6d3ea5d332ec2c8e441f1fe6d84e726d3
prerequisite-patch-id: 3a6953d11b803abeb437558f3893a3b6a08acdbb
prerequisite-patch-id: 0dd42a82e73765950bd069d421d555ded8bfeb25
prerequisite-patch-id: da6e0df31ad0d5a945e0a0d29204ba75f0c97344
-- 
2.34.1

[PATCH v19 01/22] cxl/mem: Arrange for always-synchronous memdev attach

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

In preparation for CXL accelerator drivers that have a hard dependency on
CXL capability initialization, arrange for the endpoint probe result to be
conveyed to the caller of devm_cxl_add_memdev().

As it stands cxl_pci does not care about the attach state of the cxl_memdev
because all generic memory expansion functionality can be handled by the
cxl_core. For accelerators, that driver needs to know perform driver
specific initialization if CXL is available, or exectute a fallback to PCIe
only operation.

By moving devm_cxl_add_memdev() to cxl_mem.ko it removes async module
loading as one reason that a memdev may not be attached upon return from
devm_cxl_add_memdev().

The diff is busy as this moves cxl_memdev_alloc() down below the definition
of cxl_memdev_fops and introduces devm_cxl_memdev_add_or_reset() to
preclude needing to export more symbols from the cxl_core.

Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/cxl/Kconfig       |  2 +-
 drivers/cxl/core/memdev.c | 97 ++++++++++++++++-----------------------
 drivers/cxl/mem.c         | 30 ++++++++++++
 drivers/cxl/private.h     | 11 +++++
 4 files changed, 82 insertions(+), 58 deletions(-)
 create mode 100644 drivers/cxl/private.h

diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 028201e24523..111e05615f09 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -22,6 +22,7 @@ if CXL_BUS
 config CXL_PCI
 	tristate "PCI manageability"
 	default CXL_BUS
+	select CXL_MEM
 	help
 	  The CXL specification defines a "CXL memory device" sub-class in the
 	  PCI "memory controller" base class of devices. Device's identified by
@@ -89,7 +90,6 @@ config CXL_PMEM
 
 config CXL_MEM
 	tristate "CXL: Memory Expansion"
-	depends on CXL_PCI
 	default CXL_BUS
 	help
 	  The CXL.mem protocol allows a device to act as a provider of "System
diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
index c569e00a511f..2bef231008df 100644
--- a/drivers/cxl/core/memdev.c
+++ b/drivers/cxl/core/memdev.c
@@ -8,6 +8,7 @@
 #include <linux/idr.h>
 #include <linux/pci.h>
 #include <cxlmem.h>
+#include "private.h"
 #include "trace.h"
 #include "core.h"
 
@@ -620,42 +621,30 @@ static void detach_memdev(struct work_struct *work)
 
 static struct lock_class_key cxl_memdev_key;
 
-static struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
-					   const struct file_operations *fops)
+struct cxl_memdev *devm_cxl_memdev_add_or_reset(struct device *host,
+						struct cxl_memdev *cxlmd)
 {
-	struct cxl_memdev *cxlmd;
-	struct device *dev;
-	struct cdev *cdev;
+	struct device *dev = &cxlmd->dev;
+	struct cdev *cdev = &cxlmd->cdev;
 	int rc;
 
-	cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
-	if (!cxlmd)
-		return ERR_PTR(-ENOMEM);
-
-	rc = ida_alloc_max(&cxl_memdev_ida, CXL_MEM_MAX_DEVS - 1, GFP_KERNEL);
-	if (rc < 0)
-		goto err;
-	cxlmd->id = rc;
-	cxlmd->depth = -1;
-
-	dev = &cxlmd->dev;
-	device_initialize(dev);
-	lockdep_set_class(&dev->mutex, &cxl_memdev_key);
-	dev->parent = cxlds->dev;
-	dev->bus = &cxl_bus_type;
-	dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
-	dev->type = &cxl_memdev_type;
-	device_set_pm_not_required(dev);
-	INIT_WORK(&cxlmd->detach_work, detach_memdev);
+	rc = cdev_device_add(cdev, dev);
+	if (rc) {
+		/*
+		 * The cdev was briefly live, shutdown any ioctl operations that
+		 * saw that state.
+		 */
+		cxl_memdev_shutdown(dev);
+		put_device(dev);
+		return ERR_PTR(rc);
+	}
 
-	cdev = &cxlmd->cdev;
-	cdev_init(cdev, fops);
+	rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
+	if (rc)
+		return ERR_PTR(rc);
 	return cxlmd;
-
-err:
-	kfree(cxlmd);
-	return ERR_PTR(rc);
 }
+EXPORT_SYMBOL_NS_GPL(devm_cxl_memdev_add_or_reset, "CXL");
 
 static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
 			       unsigned long arg)
@@ -1023,50 +1012,44 @@ static const struct file_operations cxl_memdev_fops = {
 	.llseek = noop_llseek,
 };
 
-struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
-				       struct cxl_dev_state *cxlds)
+struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds)
 {
 	struct cxl_memdev *cxlmd;
 	struct device *dev;
 	struct cdev *cdev;
 	int rc;
 
-	cxlmd = cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
-	if (IS_ERR(cxlmd))
-		return cxlmd;
+	cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
+	if (!cxlmd)
+		return ERR_PTR(-ENOMEM);
 
-	dev = &cxlmd->dev;
-	rc = dev_set_name(dev, "mem%d", cxlmd->id);
-	if (rc)
+	rc = ida_alloc_max(&cxl_memdev_ida, CXL_MEM_MAX_DEVS - 1, GFP_KERNEL);
+	if (rc < 0)
 		goto err;
-
-	/*
-	 * Activate ioctl operations, no cxl_memdev_rwsem manipulation
-	 * needed as this is ordered with cdev_add() publishing the device.
-	 */
+	cxlmd->id = rc;
+	cxlmd->depth = -1;
 	cxlmd->cxlds = cxlds;
 	cxlds->cxlmd = cxlmd;
 
-	cdev = &cxlmd->cdev;
-	rc = cdev_device_add(cdev, dev);
-	if (rc)
-		goto err;
+	dev = &cxlmd->dev;
+	device_initialize(dev);
+	lockdep_set_class(&dev->mutex, &cxl_memdev_key);
+	dev->parent = cxlds->dev;
+	dev->bus = &cxl_bus_type;
+	dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
+	dev->type = &cxl_memdev_type;
+	device_set_pm_not_required(dev);
+	INIT_WORK(&cxlmd->detach_work, detach_memdev);
 
-	rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
-	if (rc)
-		return ERR_PTR(rc);
+	cdev = &cxlmd->cdev;
+	cdev_init(cdev, &cxl_memdev_fops);
 	return cxlmd;
 
 err:
-	/*
-	 * The cdev was briefly live, shutdown any ioctl operations that
-	 * saw that state.
-	 */
-	cxl_memdev_shutdown(dev);
-	put_device(dev);
+	kfree(cxlmd);
 	return ERR_PTR(rc);
 }
-EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
+EXPORT_SYMBOL_NS_GPL(cxl_memdev_alloc, "CXL");
 
 static void sanitize_teardown_notifier(void *data)
 {
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index f7dc0ba8905d..144749b9c818 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -7,6 +7,7 @@
 
 #include "cxlmem.h"
 #include "cxlpci.h"
+#include "private.h"
 #include "core/core.h"
 
 /**
@@ -203,6 +204,34 @@ static int cxl_mem_probe(struct device *dev)
 	return devm_add_action_or_reset(dev, enable_suspend, NULL);
 }
 
+/**
+ * devm_cxl_add_memdev - Add a CXL memory device
+ * @host: devres alloc/release context and parent for the memdev
+ * @cxlds: CXL device state to associate with the memdev
+ *
+ * Upon return the device will have had a chance to attach to the
+ * cxl_mem driver, but may fail if the CXL topology is not ready
+ * (hardware CXL link down, or software platform CXL root not attached)
+ */
+struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
+				       struct cxl_dev_state *cxlds)
+{
+	struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds);
+	int rc;
+
+	if (IS_ERR(cxlmd))
+		return cxlmd;
+
+	rc = dev_set_name(&cxlmd->dev, "mem%d", cxlmd->id);
+	if (rc) {
+		put_device(&cxlmd->dev);
+		return ERR_PTR(rc);
+	}
+
+	return devm_cxl_memdev_add_or_reset(host, cxlmd);
+}
+EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
+
 static ssize_t trigger_poison_list_store(struct device *dev,
 					 struct device_attribute *attr,
 					 const char *buf, size_t len)
@@ -250,6 +279,7 @@ static struct cxl_driver cxl_mem_driver = {
 	.probe = cxl_mem_probe,
 	.id = CXL_DEVICE_MEMORY_EXPANDER,
 	.drv = {
+		.probe_type = PROBE_FORCE_SYNCHRONOUS,
 		.dev_groups = cxl_mem_groups,
 	},
 };
diff --git a/drivers/cxl/private.h b/drivers/cxl/private.h
new file mode 100644
index 000000000000..bdeb66e4a04f
--- /dev/null
+++ b/drivers/cxl/private.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2025 Intel Corporation. */
+
+/* Private interfaces betwen common drivers ("cxl_mem") and the cxl_core */
+
+#ifndef __CXL_PRIVATE_H__
+#define __CXL_PRIVATE_H__
+struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds);
+struct cxl_memdev *devm_cxl_memdev_add_or_reset(struct device *host,
+						struct cxl_memdev *cxlmd);
+#endif /* __CXL_PRIVATE_H__ */
-- 
2.34.1

[PATCH v19 02/22] cxl/port: Arrange for always synchronous endpoint attach

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Make it so that upon return from devm_cxl_add_endpoint() that cxl_mem_probe() can
assume that the endpoint has had a chance to complete cxl_port_probe().
I.e. cxl_port module loading has completed prior to device registration.

MODULE_SOFTDEP() is not sufficient for this purpose, but a hard link-time
dependency is reliable.

Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/cxl/mem.c     | 43 -------------------------------------------
 drivers/cxl/port.c    | 41 +++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/private.h |  7 ++++++-
 3 files changed, 47 insertions(+), 44 deletions(-)

diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 144749b9c818..56a1a4e14455 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -47,44 +47,6 @@ static int cxl_mem_dpa_show(struct seq_file *file, void *data)
 	return 0;
 }
 
-static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
-				 struct cxl_dport *parent_dport)
-{
-	struct cxl_port *parent_port = parent_dport->port;
-	struct cxl_port *endpoint, *iter, *down;
-	int rc;
-
-	/*
-	 * Now that the path to the root is established record all the
-	 * intervening ports in the chain.
-	 */
-	for (iter = parent_port, down = NULL; !is_cxl_root(iter);
-	     down = iter, iter = to_cxl_port(iter->dev.parent)) {
-		struct cxl_ep *ep;
-
-		ep = cxl_ep_load(iter, cxlmd);
-		ep->next = down;
-	}
-
-	/* Note: endpoint port component registers are derived from @cxlds */
-	endpoint = devm_cxl_add_port(host, &cxlmd->dev, CXL_RESOURCE_NONE,
-				     parent_dport);
-	if (IS_ERR(endpoint))
-		return PTR_ERR(endpoint);
-
-	rc = cxl_endpoint_autoremove(cxlmd, endpoint);
-	if (rc)
-		return rc;
-
-	if (!endpoint->dev.driver) {
-		dev_err(&cxlmd->dev, "%s failed probe\n",
-			dev_name(&endpoint->dev));
-		return -ENXIO;
-	}
-
-	return 0;
-}
-
 static int cxl_debugfs_poison_inject(void *data, u64 dpa)
 {
 	struct cxl_memdev *cxlmd = data;
@@ -290,8 +252,3 @@ MODULE_DESCRIPTION("CXL: Memory Expansion");
 MODULE_LICENSE("GPL v2");
 MODULE_IMPORT_NS("CXL");
 MODULE_ALIAS_CXL(CXL_DEVICE_MEMORY_EXPANDER);
-/*
- * create_endpoint() wants to validate port driver attach immediately after
- * endpoint registration.
- */
-MODULE_SOFTDEP("pre: cxl_port");
diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c
index e66c7f2e1955..83f5a09839ab 100644
--- a/drivers/cxl/port.c
+++ b/drivers/cxl/port.c
@@ -6,6 +6,7 @@
 
 #include "cxlmem.h"
 #include "cxlpci.h"
+#include "private.h"
 #include "core/core.h"
 
 /**
@@ -200,10 +201,50 @@ static struct cxl_driver cxl_port_driver = {
 	.probe = cxl_port_probe,
 	.id = CXL_DEVICE_PORT,
 	.drv = {
+		.probe_type = PROBE_FORCE_SYNCHRONOUS,
 		.dev_groups = cxl_port_attribute_groups,
 	},
 };
 
+int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
+			  struct cxl_dport *parent_dport)
+{
+	struct cxl_port *parent_port = parent_dport->port;
+	struct cxl_port *endpoint, *iter, *down;
+	int rc;
+
+	/*
+	 * Now that the path to the root is established record all the
+	 * intervening ports in the chain.
+	 */
+	for (iter = parent_port, down = NULL; !is_cxl_root(iter);
+	     down = iter, iter = to_cxl_port(iter->dev.parent)) {
+		struct cxl_ep *ep;
+
+		ep = cxl_ep_load(iter, cxlmd);
+		ep->next = down;
+	}
+
+	/* Note: endpoint port component registers are derived from @cxlds */
+	endpoint = devm_cxl_add_port(host, &cxlmd->dev, CXL_RESOURCE_NONE,
+				     parent_dport);
+	if (IS_ERR(endpoint))
+		return PTR_ERR(endpoint);
+
+	rc = cxl_endpoint_autoremove(cxlmd, endpoint);
+	if (rc)
+		return rc;
+
+	if (!endpoint->dev.driver) {
+		dev_err(&cxlmd->dev, "%s failed probe\n",
+			dev_name(&endpoint->dev));
+		return -ENXIO;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(devm_cxl_add_endpoint, "CXL");
+
 static int __init cxl_port_init(void)
 {
 	return cxl_driver_register(&cxl_port_driver);
diff --git a/drivers/cxl/private.h b/drivers/cxl/private.h
index bdeb66e4a04f..e15ff7f4b119 100644
--- a/drivers/cxl/private.h
+++ b/drivers/cxl/private.h
@@ -1,11 +1,16 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 /* Copyright(c) 2025 Intel Corporation. */
 
-/* Private interfaces betwen common drivers ("cxl_mem") and the cxl_core */
+/*
+ * Private interfaces betwen common drivers ("cxl_mem", "cxl_port") and
+ * the cxl_core.
+ */
 
 #ifndef __CXL_PRIVATE_H__
 #define __CXL_PRIVATE_H__
 struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds);
 struct cxl_memdev *devm_cxl_memdev_add_or_reset(struct device *host,
 						struct cxl_memdev *cxlmd);
+int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
+			  struct cxl_dport *parent_dport);
 #endif /* __CXL_PRIVATE_H__ */
-- 
2.34.1

[PATCH v19 03/22] cxl/mem: Introduce a memdev creation ->probe() operation

[alejandro.lucero-palau]

From: Dan Williams <dan.j.williams@intel.com>

Allow for a driver to pass a routine to be called in cxl_mem_probe()
context. This ability mirrors the semantics of faux_device_create() and
allows for the caller to run CXL-topology-attach dependent logic on behalf
of the caller.

This capability is needed for CXL accelerator device drivers that need to
make decisions about enabling CXL dependent functionality in the device, or
falling back to PCIe-only operation.

The probe callback runs after the port topology is successfully attached
for the given memdev.

Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/cxl/core/memdev.c    |  5 ++++-
 drivers/cxl/cxlmem.h         | 10 +++++++++-
 drivers/cxl/mem.c            | 34 +++++++++++++++++++++++++++++++---
 drivers/cxl/pci.c            |  2 +-
 drivers/cxl/private.h        |  3 ++-
 tools/testing/cxl/test/mem.c |  2 +-
 6 files changed, 48 insertions(+), 8 deletions(-)

diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
index 2bef231008df..628f91c60c90 100644
--- a/drivers/cxl/core/memdev.c
+++ b/drivers/cxl/core/memdev.c
@@ -1012,7 +1012,8 @@ static const struct file_operations cxl_memdev_fops = {
 	.llseek = noop_llseek,
 };
 
-struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds)
+struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
+				    const struct cxl_memdev_ops *ops)
 {
 	struct cxl_memdev *cxlmd;
 	struct device *dev;
@@ -1028,6 +1029,8 @@ struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds)
 		goto err;
 	cxlmd->id = rc;
 	cxlmd->depth = -1;
+	cxlmd->ops = ops;
+	cxlmd->endpoint = ERR_PTR(-ENXIO);
 	cxlmd->cxlds = cxlds;
 	cxlds->cxlmd = cxlmd;
 
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 751478dfc410..82e8188c76a0 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -34,6 +34,10 @@
 	(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) !=                       \
 	 CXLMDEV_RESET_NEEDED_NOT)
 
+struct cxl_memdev_ops {
+	int (*probe)(struct cxl_memdev *cxlmd);
+};
+
 /**
  * struct cxl_memdev - CXL bus object representing a Type-3 Memory Device
  * @dev: driver core device object
@@ -43,6 +47,7 @@
  * @cxl_nvb: coordinate removal of @cxl_nvd if present
  * @cxl_nvd: optional bridge to an nvdimm if the device supports pmem
  * @endpoint: connection to the CXL port topology for this memory device
+ * @ops: incremental caller specific probe routine
  * @id: id number of this memdev instance.
  * @depth: endpoint port depth
  * @scrub_cycle: current scrub cycle set for this device
@@ -59,6 +64,7 @@ struct cxl_memdev {
 	struct cxl_nvdimm_bridge *cxl_nvb;
 	struct cxl_nvdimm *cxl_nvd;
 	struct cxl_port *endpoint;
+	const struct cxl_memdev_ops *ops;
 	int id;
 	int depth;
 	u8 scrub_cycle;
@@ -96,7 +102,9 @@ static inline bool is_cxl_endpoint(struct cxl_port *port)
 }
 
 struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
-				       struct cxl_dev_state *cxlds);
+				       struct cxl_dev_state *cxlds,
+				       const struct cxl_memdev_ops *ops);
+
 int devm_cxl_sanitize_setup_notifier(struct device *host,
 				     struct cxl_memdev *cxlmd);
 struct cxl_memdev_state;
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 56a1a4e14455..aeb2e3e8282a 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -145,6 +145,12 @@ static int cxl_mem_probe(struct device *dev)
 			return rc;
 	}
 
+	if (cxlmd->ops) {
+		rc = cxlmd->ops->probe(cxlmd);
+		if (rc)
+			return rc;
+	}
+
 	rc = devm_cxl_memdev_edac_register(cxlmd);
 	if (rc)
 		dev_dbg(dev, "CXL memdev EDAC registration failed rc=%d\n", rc);
@@ -170,15 +176,18 @@ static int cxl_mem_probe(struct device *dev)
  * devm_cxl_add_memdev - Add a CXL memory device
  * @host: devres alloc/release context and parent for the memdev
  * @cxlds: CXL device state to associate with the memdev
+ * @ops: optional operations to run in cxl_mem::{probe,remove}() context
  *
  * Upon return the device will have had a chance to attach to the
  * cxl_mem driver, but may fail if the CXL topology is not ready
  * (hardware CXL link down, or software platform CXL root not attached)
+ *
  */
 struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
-				       struct cxl_dev_state *cxlds)
+				       struct cxl_dev_state *cxlds,
+				       const struct cxl_memdev_ops *ops)
 {
-	struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds);
+	struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds, ops);
 	int rc;
 
 	if (IS_ERR(cxlmd))
@@ -190,7 +199,26 @@ struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
 		return ERR_PTR(rc);
 	}
 
-	return devm_cxl_memdev_add_or_reset(host, cxlmd);
+	cxlmd = devm_cxl_memdev_add_or_reset(host, cxlmd);
+	if (IS_ERR(cxlmd))
+		return cxlmd;
+
+	/*
+	 * If ops is provided fail if the driver is not attached upon
+	 * return. The ->endpoint ERR_PTR may have a more precise error
+	 * code to convey. Note that failure here could be the result of
+	 * a race to teardown the CXL port topology. I.e.
+	 * cxl_mem_probe() could have succeeded and then cxl_mem unbound
+	 * before the lock is acquired.
+	 */
+	guard(device)(&cxlmd->dev);
+	if (ops && !cxlmd->dev.driver) {
+		if (IS_ERR(cxlmd->endpoint))
+			return ERR_CAST(cxlmd->endpoint);
+		return ERR_PTR(-ENXIO);
+	}
+
+	return cxlmd;
 }
 EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
 
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 6803c2fb906b..0a3108d552c8 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -1007,7 +1007,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		dev_dbg(&pdev->dev, "No CXL Features discovered\n");
 
-	cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlds);
+	cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlds, NULL);
 	if (IS_ERR(cxlmd))
 		return PTR_ERR(cxlmd);
 
diff --git a/drivers/cxl/private.h b/drivers/cxl/private.h
index e15ff7f4b119..f8a5658e7090 100644
--- a/drivers/cxl/private.h
+++ b/drivers/cxl/private.h
@@ -8,7 +8,8 @@
 
 #ifndef __CXL_PRIVATE_H__
 #define __CXL_PRIVATE_H__
-struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds);
+struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
+				    const struct cxl_memdev_ops *ops);
 struct cxl_memdev *devm_cxl_memdev_add_or_reset(struct device *host,
 						struct cxl_memdev *cxlmd);
 int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index 0f1d91f57ba3..3d553661ca75 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -1768,7 +1768,7 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 
 	cxl_mock_add_event_logs(&mdata->mes);
 
-	cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlds);
+	cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlds, NULL);
 	if (IS_ERR(cxlmd))
 		return PTR_ERR(cxlmd);
 
-- 
2.34.1

[PATCH v19 04/22] cxl: Add type2 device basic support

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Differentiate CXL memory expanders (type 3) from CXL device accelerators
(type 2) with a new function for initializing cxl_dev_state and a macro
for helping accel drivers to embed cxl_dev_state inside a private
struct.

Move structs to include/cxl as the size of the accel driver private
struct embedding cxl_dev_state needs to know the size of this struct.

Use same new initialization with the type3 pci driver.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
---
 drivers/cxl/core/mbox.c      |  12 +-
 drivers/cxl/core/memdev.c    |  32 +++++
 drivers/cxl/cxl.h            |  97 +--------------
 drivers/cxl/cxlmem.h         |  86 +------------
 drivers/cxl/pci.c            |  15 +--
 include/cxl/cxl.h            | 226 +++++++++++++++++++++++++++++++++++
 tools/testing/cxl/test/mem.c |   3 +-
 7 files changed, 276 insertions(+), 195 deletions(-)
 create mode 100644 include/cxl/cxl.h

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index fa6dd0c94656..bee84d0101d1 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -1514,23 +1514,21 @@ int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_mailbox_init, "CXL");
 
-struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev)
+struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev, u64 serial,
+						 u16 dvsec)
 {
 	struct cxl_memdev_state *mds;
 	int rc;
 
-	mds = devm_kzalloc(dev, sizeof(*mds), GFP_KERNEL);
+	mds = devm_cxl_dev_state_create(dev, CXL_DEVTYPE_CLASSMEM, serial,
+					dvsec, struct cxl_memdev_state, cxlds,
+					true);
 	if (!mds) {
 		dev_err(dev, "No memory available\n");
 		return ERR_PTR(-ENOMEM);
 	}
 
 	mutex_init(&mds->event.log_lock);
-	mds->cxlds.dev = dev;
-	mds->cxlds.reg_map.host = dev;
-	mds->cxlds.cxl_mbox.host = dev;
-	mds->cxlds.reg_map.resource = CXL_RESOURCE_NONE;
-	mds->cxlds.type = CXL_DEVTYPE_CLASSMEM;
 
 	rc = devm_cxl_register_mce_notifier(dev, &mds->mce_notifier);
 	if (rc == -EOPNOTSUPP)
diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
index 628f91c60c90..66d3940fb60a 100644
--- a/drivers/cxl/core/memdev.c
+++ b/drivers/cxl/core/memdev.c
@@ -621,6 +621,38 @@ static void detach_memdev(struct work_struct *work)
 
 static struct lock_class_key cxl_memdev_key;
 
+static void cxl_dev_state_init(struct cxl_dev_state *cxlds, struct device *dev,
+			       enum cxl_devtype type, u64 serial, u16 dvsec,
+			       bool has_mbox)
+{
+	*cxlds = (struct cxl_dev_state) {
+		.dev = dev,
+		.type = type,
+		.serial = serial,
+		.cxl_dvsec = dvsec,
+		.reg_map.host = dev,
+		.reg_map.resource = CXL_RESOURCE_NONE,
+	};
+
+	if (has_mbox)
+		cxlds->cxl_mbox.host = dev;
+}
+
+struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev,
+						 enum cxl_devtype type,
+						 u64 serial, u16 dvsec,
+						 size_t size, bool has_mbox)
+{
+	struct cxl_dev_state *cxlds = devm_kzalloc(dev, size, GFP_KERNEL);
+
+	if (!cxlds)
+		return NULL;
+
+	cxl_dev_state_init(cxlds, dev, type, serial, dvsec, has_mbox);
+	return cxlds;
+}
+EXPORT_SYMBOL_NS_GPL(_devm_cxl_dev_state_create, "CXL");
+
 struct cxl_memdev *devm_cxl_memdev_add_or_reset(struct device *host,
 						struct cxl_memdev *cxlmd)
 {
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 32fccad9a7f6..db8e74c55309 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -11,6 +11,7 @@
 #include <linux/log2.h>
 #include <linux/node.h>
 #include <linux/io.h>
+#include <cxl/cxl.h>
 
 extern const struct nvdimm_security_ops *cxl_security_ops;
 
@@ -200,97 +201,6 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
 #define   CXLDEV_MBOX_BG_CMD_COMMAND_VENDOR_MASK GENMASK_ULL(63, 48)
 #define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20
 
-/*
- * Using struct_group() allows for per register-block-type helper routines,
- * without requiring block-type agnostic code to include the prefix.
- */
-struct cxl_regs {
-	/*
-	 * Common set of CXL Component register block base pointers
-	 * @hdm_decoder: CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure
-	 * @ras: CXL 2.0 8.2.5.9 CXL RAS Capability Structure
-	 */
-	struct_group_tagged(cxl_component_regs, component,
-		void __iomem *hdm_decoder;
-		void __iomem *ras;
-	);
-	/*
-	 * Common set of CXL Device register block base pointers
-	 * @status: CXL 2.0 8.2.8.3 Device Status Registers
-	 * @mbox: CXL 2.0 8.2.8.4 Mailbox Registers
-	 * @memdev: CXL 2.0 8.2.8.5 Memory Device Registers
-	 */
-	struct_group_tagged(cxl_device_regs, device_regs,
-		void __iomem *status, *mbox, *memdev;
-	);
-
-	struct_group_tagged(cxl_pmu_regs, pmu_regs,
-		void __iomem *pmu;
-	);
-
-	/*
-	 * RCH downstream port specific RAS register
-	 * @aer: CXL 3.0 8.2.1.1 RCH Downstream Port RCRB
-	 */
-	struct_group_tagged(cxl_rch_regs, rch_regs,
-		void __iomem *dport_aer;
-	);
-
-	/*
-	 * RCD upstream port specific PCIe cap register
-	 * @pcie_cap: CXL 3.0 8.2.1.2 RCD Upstream Port RCRB
-	 */
-	struct_group_tagged(cxl_rcd_regs, rcd_regs,
-		void __iomem *rcd_pcie_cap;
-	);
-};
-
-struct cxl_reg_map {
-	bool valid;
-	int id;
-	unsigned long offset;
-	unsigned long size;
-};
-
-struct cxl_component_reg_map {
-	struct cxl_reg_map hdm_decoder;
-	struct cxl_reg_map ras;
-};
-
-struct cxl_device_reg_map {
-	struct cxl_reg_map status;
-	struct cxl_reg_map mbox;
-	struct cxl_reg_map memdev;
-};
-
-struct cxl_pmu_reg_map {
-	struct cxl_reg_map pmu;
-};
-
-/**
- * struct cxl_register_map - DVSEC harvested register block mapping parameters
- * @host: device for devm operations and logging
- * @base: virtual base of the register-block-BAR + @block_offset
- * @resource: physical resource base of the register block
- * @max_size: maximum mapping size to perform register search
- * @reg_type: see enum cxl_regloc_type
- * @component_map: cxl_reg_map for component registers
- * @device_map: cxl_reg_maps for device registers
- * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units
- */
-struct cxl_register_map {
-	struct device *host;
-	void __iomem *base;
-	resource_size_t resource;
-	resource_size_t max_size;
-	u8 reg_type;
-	union {
-		struct cxl_component_reg_map component_map;
-		struct cxl_device_reg_map device_map;
-		struct cxl_pmu_reg_map pmu_map;
-	};
-};
-
 void cxl_probe_component_regs(struct device *dev, void __iomem *base,
 			      struct cxl_component_reg_map *map);
 void cxl_probe_device_regs(struct device *dev, void __iomem *base,
@@ -484,11 +394,6 @@ struct cxl_region_params {
 	resource_size_t cache_size;
 };
 
-enum cxl_partition_mode {
-	CXL_PARTMODE_RAM,
-	CXL_PARTMODE_PMEM,
-};
-
 /*
  * Indicate whether this region has been assembled by autodetection or
  * userspace assembly. Prevent endpoint decoders outside of automatic
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 82e8188c76a0..86aa4899d511 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -6,7 +6,7 @@
 #include <linux/pci.h>
 #include <linux/cdev.h>
 #include <linux/uuid.h>
-#include <linux/node.h>
+#include <cxl/cxl.h>
 #include <cxl/event.h>
 #include <cxl/mailbox.h>
 #include "cxl.h"
@@ -373,87 +373,6 @@ struct cxl_security_state {
 	struct kernfs_node *sanitize_node;
 };
 
-/*
- * enum cxl_devtype - delineate type-2 from a generic type-3 device
- * @CXL_DEVTYPE_DEVMEM - Vendor specific CXL Type-2 device implementing HDM-D or
- *			 HDM-DB, no requirement that this device implements a
- *			 mailbox, or other memory-device-standard manageability
- *			 flows.
- * @CXL_DEVTYPE_CLASSMEM - Common class definition of a CXL Type-3 device with
- *			   HDM-H and class-mandatory memory device registers
- */
-enum cxl_devtype {
-	CXL_DEVTYPE_DEVMEM,
-	CXL_DEVTYPE_CLASSMEM,
-};
-
-/**
- * struct cxl_dpa_perf - DPA performance property entry
- * @dpa_range: range for DPA address
- * @coord: QoS performance data (i.e. latency, bandwidth)
- * @cdat_coord: raw QoS performance data from CDAT
- * @qos_class: QoS Class cookies
- */
-struct cxl_dpa_perf {
-	struct range dpa_range;
-	struct access_coordinate coord[ACCESS_COORDINATE_MAX];
-	struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX];
-	int qos_class;
-};
-
-/**
- * struct cxl_dpa_partition - DPA partition descriptor
- * @res: shortcut to the partition in the DPA resource tree (cxlds->dpa_res)
- * @perf: performance attributes of the partition from CDAT
- * @mode: operation mode for the DPA capacity, e.g. ram, pmem, dynamic...
- */
-struct cxl_dpa_partition {
-	struct resource res;
-	struct cxl_dpa_perf perf;
-	enum cxl_partition_mode mode;
-};
-
-/**
- * struct cxl_dev_state - The driver device state
- *
- * cxl_dev_state represents the CXL driver/device state.  It provides an
- * interface to mailbox commands as well as some cached data about the device.
- * Currently only memory devices are represented.
- *
- * @dev: The device associated with this CXL state
- * @cxlmd: The device representing the CXL.mem capabilities of @dev
- * @reg_map: component and ras register mapping parameters
- * @regs: Parsed register blocks
- * @cxl_dvsec: Offset to the PCIe device DVSEC
- * @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH)
- * @media_ready: Indicate whether the device media is usable
- * @dpa_res: Overall DPA resource tree for the device
- * @part: DPA partition array
- * @nr_partitions: Number of DPA partitions
- * @serial: PCIe Device Serial Number
- * @type: Generic Memory Class device or Vendor Specific Memory device
- * @cxl_mbox: CXL mailbox context
- * @cxlfs: CXL features context
- */
-struct cxl_dev_state {
-	struct device *dev;
-	struct cxl_memdev *cxlmd;
-	struct cxl_register_map reg_map;
-	struct cxl_regs regs;
-	int cxl_dvsec;
-	bool rcd;
-	bool media_ready;
-	struct resource dpa_res;
-	struct cxl_dpa_partition part[CXL_NR_PARTITIONS_MAX];
-	unsigned int nr_partitions;
-	u64 serial;
-	enum cxl_devtype type;
-	struct cxl_mailbox cxl_mbox;
-#ifdef CONFIG_CXL_FEATURES
-	struct cxl_features_state *cxlfs;
-#endif
-};
-
 static inline resource_size_t cxl_pmem_size(struct cxl_dev_state *cxlds)
 {
 	/*
@@ -858,7 +777,8 @@ int cxl_dev_state_identify(struct cxl_memdev_state *mds);
 int cxl_await_media_ready(struct cxl_dev_state *cxlds);
 int cxl_enumerate_cmds(struct cxl_memdev_state *mds);
 int cxl_mem_dpa_fetch(struct cxl_memdev_state *mds, struct cxl_dpa_info *info);
-struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev);
+struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev, u64 serial,
+						 u16 dvsec);
 void set_exclusive_cxl_commands(struct cxl_memdev_state *mds,
 				unsigned long *cmds);
 void clear_exclusive_cxl_commands(struct cxl_memdev_state *mds,
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 0a3108d552c8..58532b1216dd 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -11,6 +11,7 @@
 #include <linux/pci.h>
 #include <linux/aer.h>
 #include <linux/io.h>
+#include <cxl/cxl.h>
 #include <cxl/mailbox.h>
 #include "cxlmem.h"
 #include "cxlpci.h"
@@ -912,6 +913,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	int rc, pmu_count;
 	unsigned int i;
 	bool irq_avail;
+	u16 dvsec;
 
 	/*
 	 * Double check the anonymous union trickery in struct cxl_regs
@@ -925,19 +927,18 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		return rc;
 	pci_set_master(pdev);
 
-	mds = cxl_memdev_state_create(&pdev->dev);
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		pci_warn(pdev, "Device DVSEC not present, skip CXL.mem init\n");
+
+	mds = cxl_memdev_state_create(&pdev->dev, pci_get_dsn(pdev), dvsec);
 	if (IS_ERR(mds))
 		return PTR_ERR(mds);
 	cxlds = &mds->cxlds;
 	pci_set_drvdata(pdev, cxlds);
 
 	cxlds->rcd = is_cxl_restricted(pdev);
-	cxlds->serial = pci_get_dsn(pdev);
-	cxlds->cxl_dvsec = pci_find_dvsec_capability(
-		pdev, PCI_VENDOR_ID_CXL, PCI_DVSEC_CXL_DEVICE);
-	if (!cxlds->cxl_dvsec)
-		dev_warn(&pdev->dev,
-			 "Device DVSEC not present, skip CXL.mem init\n");
 
 	rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map);
 	if (rc)
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
new file mode 100644
index 000000000000..13d448686189
--- /dev/null
+++ b/include/cxl/cxl.h
@@ -0,0 +1,226 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2020 Intel Corporation. */
+/* Copyright(c) 2025 Advanced Micro Devices, Inc. */
+
+#ifndef __CXL_CXL_H__
+#define __CXL_CXL_H__
+
+#include <linux/node.h>
+#include <linux/ioport.h>
+#include <cxl/mailbox.h>
+
+/**
+ * enum cxl_devtype - delineate type-2 from a generic type-3 device
+ * @CXL_DEVTYPE_DEVMEM: Vendor specific CXL Type-2 device implementing HDM-D or
+ *			 HDM-DB, no requirement that this device implements a
+ *			 mailbox, or other memory-device-standard manageability
+ *			 flows.
+ * @CXL_DEVTYPE_CLASSMEM: Common class definition of a CXL Type-3 device with
+ *			   HDM-H and class-mandatory memory device registers
+ */
+enum cxl_devtype {
+	CXL_DEVTYPE_DEVMEM,
+	CXL_DEVTYPE_CLASSMEM,
+};
+
+struct device;
+
+/*
+ * Using struct_group() allows for per register-block-type helper routines,
+ * without requiring block-type agnostic code to include the prefix.
+ */
+struct cxl_regs {
+	/*
+	 * Common set of CXL Component register block base pointers
+	 * @hdm_decoder: CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure
+	 * @ras: CXL 2.0 8.2.5.9 CXL RAS Capability Structure
+	 */
+	struct_group_tagged(cxl_component_regs, component,
+		void __iomem *hdm_decoder;
+		void __iomem *ras;
+	);
+	/*
+	 * Common set of CXL Device register block base pointers
+	 * @status: CXL 2.0 8.2.8.3 Device Status Registers
+	 * @mbox: CXL 2.0 8.2.8.4 Mailbox Registers
+	 * @memdev: CXL 2.0 8.2.8.5 Memory Device Registers
+	 */
+	struct_group_tagged(cxl_device_regs, device_regs,
+		void __iomem *status, *mbox, *memdev;
+	);
+
+	struct_group_tagged(cxl_pmu_regs, pmu_regs,
+		void __iomem *pmu;
+	);
+
+	/*
+	 * RCH downstream port specific RAS register
+	 * @aer: CXL 3.0 8.2.1.1 RCH Downstream Port RCRB
+	 */
+	struct_group_tagged(cxl_rch_regs, rch_regs,
+		void __iomem *dport_aer;
+	);
+
+	/*
+	 * RCD upstream port specific PCIe cap register
+	 * @pcie_cap: CXL 3.0 8.2.1.2 RCD Upstream Port RCRB
+	 */
+	struct_group_tagged(cxl_rcd_regs, rcd_regs,
+		void __iomem *rcd_pcie_cap;
+	);
+};
+
+struct cxl_reg_map {
+	bool valid;
+	int id;
+	unsigned long offset;
+	unsigned long size;
+};
+
+struct cxl_component_reg_map {
+	struct cxl_reg_map hdm_decoder;
+	struct cxl_reg_map ras;
+};
+
+struct cxl_device_reg_map {
+	struct cxl_reg_map status;
+	struct cxl_reg_map mbox;
+	struct cxl_reg_map memdev;
+};
+
+struct cxl_pmu_reg_map {
+	struct cxl_reg_map pmu;
+};
+
+/**
+ * struct cxl_register_map - DVSEC harvested register block mapping parameters
+ * @host: device for devm operations and logging
+ * @base: virtual base of the register-block-BAR + @block_offset
+ * @resource: physical resource base of the register block
+ * @max_size: maximum mapping size to perform register search
+ * @reg_type: see enum cxl_regloc_type
+ * @component_map: cxl_reg_map for component registers
+ * @device_map: cxl_reg_maps for device registers
+ * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units
+ */
+struct cxl_register_map {
+	struct device *host;
+	void __iomem *base;
+	resource_size_t resource;
+	resource_size_t max_size;
+	u8 reg_type;
+	union {
+		struct cxl_component_reg_map component_map;
+		struct cxl_device_reg_map device_map;
+		struct cxl_pmu_reg_map pmu_map;
+	};
+};
+
+/**
+ * struct cxl_dpa_perf - DPA performance property entry
+ * @dpa_range: range for DPA address
+ * @coord: QoS performance data (i.e. latency, bandwidth)
+ * @cdat_coord: raw QoS performance data from CDAT
+ * @qos_class: QoS Class cookies
+ */
+struct cxl_dpa_perf {
+	struct range dpa_range;
+	struct access_coordinate coord[ACCESS_COORDINATE_MAX];
+	struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX];
+	int qos_class;
+};
+
+enum cxl_partition_mode {
+	CXL_PARTMODE_RAM,
+	CXL_PARTMODE_PMEM,
+};
+
+/**
+ * struct cxl_dpa_partition - DPA partition descriptor
+ * @res: shortcut to the partition in the DPA resource tree (cxlds->dpa_res)
+ * @perf: performance attributes of the partition from CDAT
+ * @mode: operation mode for the DPA capacity, e.g. ram, pmem, dynamic...
+ */
+struct cxl_dpa_partition {
+	struct resource res;
+	struct cxl_dpa_perf perf;
+	enum cxl_partition_mode mode;
+};
+
+#define CXL_NR_PARTITIONS_MAX 2
+
+/**
+ * struct cxl_dev_state - The driver device state
+ *
+ * cxl_dev_state represents the CXL driver/device state.  It provides an
+ * interface to mailbox commands as well as some cached data about the device.
+ * Currently only memory devices are represented.
+ *
+ * @dev: The device associated with this CXL state
+ * @cxlmd: The device representing the CXL.mem capabilities of @dev
+ * @reg_map: component and ras register mapping parameters
+ * @regs: Parsed register blocks
+ * @cxl_dvsec: Offset to the PCIe device DVSEC
+ * @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH)
+ * @media_ready: Indicate whether the device media is usable
+ * @dpa_res: Overall DPA resource tree for the device
+ * @part: DPA partition array
+ * @nr_partitions: Number of DPA partitions
+ * @serial: PCIe Device Serial Number
+ * @type: Generic Memory Class device or Vendor Specific Memory device
+ * @cxl_mbox: CXL mailbox context
+ * @cxlfs: CXL features context
+ */
+struct cxl_dev_state {
+	/* public for Type2 drivers */
+	struct device *dev;
+	struct cxl_memdev *cxlmd;
+
+	/* private for Type2 drivers */
+	struct cxl_register_map reg_map;
+	struct cxl_regs regs;
+	int cxl_dvsec;
+	bool rcd;
+	bool media_ready;
+	struct resource dpa_res;
+	struct cxl_dpa_partition part[CXL_NR_PARTITIONS_MAX];
+	unsigned int nr_partitions;
+	u64 serial;
+	enum cxl_devtype type;
+	struct cxl_mailbox cxl_mbox;
+#ifdef CONFIG_CXL_FEATURES
+	struct cxl_features_state *cxlfs;
+#endif
+};
+
+struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev,
+						 enum cxl_devtype type,
+						 u64 serial, u16 dvsec,
+						 size_t size, bool has_mbox);
+
+/**
+ * cxl_dev_state_create - safely create and cast a cxl dev state embedded in a
+ * driver specific struct.
+ *
+ * @parent: device behind the request
+ * @type: CXL device type
+ * @serial: device identification
+ * @dvsec: dvsec capability offset
+ * @drv_struct: driver struct embedding a cxl_dev_state struct
+ * @member: drv_struct member as cxl_dev_state
+ * @mbox: true if mailbox supported
+ *
+ * Returns a pointer to the drv_struct allocated and embedding a cxl_dev_state
+ * struct initialized.
+ *
+ * Introduced for Type2 driver support.
+ */
+#define devm_cxl_dev_state_create(parent, type, serial, dvsec, drv_struct, member, mbox)	\
+	({										\
+		static_assert(__same_type(struct cxl_dev_state,				\
+			      ((drv_struct *)NULL)->member));				\
+		static_assert(offsetof(drv_struct, member) == 0);			\
+		(drv_struct *)_devm_cxl_dev_state_create(parent, type, serial, dvsec,	\
+						      sizeof(drv_struct), mbox);	\
+	})
+#endif /* __CXL_CXL_H__ */
diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index 3d553661ca75..369907048362 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -1717,7 +1717,7 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 	if (rc)
 		return rc;
 
-	mds = cxl_memdev_state_create(dev);
+	mds = cxl_memdev_state_create(dev, pdev->id + 1, 0);
 	if (IS_ERR(mds))
 		return PTR_ERR(mds);
 
@@ -1733,7 +1733,6 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 	mds->event.buf = (struct cxl_get_event_payload *) mdata->event_buf;
 	INIT_DELAYED_WORK(&mds->security.poll_dwork, cxl_mockmem_sanitize_work);
 
-	cxlds->serial = pdev->id + 1;
 	if (is_rcd(pdev))
 		cxlds->rcd = true;
 
-- 
2.34.1

[PATCH v19 05/22] sfc: add cxl support

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Add CXL initialization based on new CXL API for accel drivers and make
it dependent on kernel CXL configuration.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Acked-by: Edward Cree <ecree.xilinx@gmail.com>
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/net/ethernet/sfc/Kconfig      |  9 +++++
 drivers/net/ethernet/sfc/Makefile     |  1 +
 drivers/net/ethernet/sfc/efx.c        | 15 ++++++-
 drivers/net/ethernet/sfc/efx_cxl.c    | 56 +++++++++++++++++++++++++++
 drivers/net/ethernet/sfc/efx_cxl.h    | 40 +++++++++++++++++++
 drivers/net/ethernet/sfc/net_driver.h | 10 +++++
 6 files changed, 130 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/sfc/efx_cxl.c
 create mode 100644 drivers/net/ethernet/sfc/efx_cxl.h

diff --git a/drivers/net/ethernet/sfc/Kconfig b/drivers/net/ethernet/sfc/Kconfig
index c4c43434f314..979f2801e2a8 100644
--- a/drivers/net/ethernet/sfc/Kconfig
+++ b/drivers/net/ethernet/sfc/Kconfig
@@ -66,6 +66,15 @@ config SFC_MCDI_LOGGING
 	  Driver-Interface) commands and responses, allowing debugging of
 	  driver/firmware interaction.  The tracing is actually enabled by
 	  a sysfs file 'mcdi_logging' under the PCI device.
+config SFC_CXL
+	bool "Solarflare SFC9100-family CXL support"
+	depends on SFC && CXL_BUS >= SFC
+	default SFC
+	help
+	  This enables SFC CXL support if the kernel is configuring CXL for
+	  using CTPIO with CXL.mem. The SFC device with CXL support and
+	  with a CXL-aware firmware can be used for minimizing latencies
+	  when sending through CTPIO.
 
 source "drivers/net/ethernet/sfc/falcon/Kconfig"
 source "drivers/net/ethernet/sfc/siena/Kconfig"
diff --git a/drivers/net/ethernet/sfc/Makefile b/drivers/net/ethernet/sfc/Makefile
index d99039ec468d..bb0f1891cde6 100644
--- a/drivers/net/ethernet/sfc/Makefile
+++ b/drivers/net/ethernet/sfc/Makefile
@@ -13,6 +13,7 @@ sfc-$(CONFIG_SFC_SRIOV)	+= sriov.o ef10_sriov.o ef100_sriov.o ef100_rep.o \
                            mae.o tc.o tc_bindings.o tc_counters.o \
                            tc_encap_actions.o tc_conntrack.o
 
+sfc-$(CONFIG_SFC_CXL)	+= efx_cxl.o
 obj-$(CONFIG_SFC)	+= sfc.o
 
 obj-$(CONFIG_SFC_FALCON) += falcon/
diff --git a/drivers/net/ethernet/sfc/efx.c b/drivers/net/ethernet/sfc/efx.c
index 112e55b98ed3..537668278375 100644
--- a/drivers/net/ethernet/sfc/efx.c
+++ b/drivers/net/ethernet/sfc/efx.c
@@ -34,6 +34,7 @@
 #include "selftest.h"
 #include "sriov.h"
 #include "efx_devlink.h"
+#include "efx_cxl.h"
 
 #include "mcdi_port_common.h"
 #include "mcdi_pcol.h"
@@ -981,12 +982,15 @@ static void efx_pci_remove(struct pci_dev *pci_dev)
 	efx_pci_remove_main(efx);
 
 	efx_fini_io(efx);
+
+	probe_data = container_of(efx, struct efx_probe_data, efx);
+	efx_cxl_exit(probe_data);
+
 	pci_dbg(efx->pci_dev, "shutdown successful\n");
 
 	efx_fini_devlink_and_unlock(efx);
 	efx_fini_struct(efx);
 	free_netdev(efx->net_dev);
-	probe_data = container_of(efx, struct efx_probe_data, efx);
 	kfree(probe_data);
 };
 
@@ -1190,6 +1194,15 @@ static int efx_pci_probe(struct pci_dev *pci_dev,
 	if (rc)
 		goto fail2;
 
+	/* A successful cxl initialization implies a CXL region created to be
+	 * used for PIO buffers. If there is no CXL support, or initialization
+	 * fails, efx_cxl_pio_initialised will be false and legacy PIO buffers
+	 * defined at specific PCI BAR regions will be used.
+	 */
+	rc = efx_cxl_init(probe_data);
+	if (rc)
+		pci_err(pci_dev, "CXL initialization failed with error %d\n", rc);
+
 	rc = efx_pci_probe_post_io(efx);
 	if (rc) {
 		/* On failure, retry once immediately.
diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
new file mode 100644
index 000000000000..8e0481d8dced
--- /dev/null
+++ b/drivers/net/ethernet/sfc/efx_cxl.c
@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/****************************************************************************
+ *
+ * Driver for AMD network controllers and boards
+ * Copyright (C) 2025, Advanced Micro Devices, Inc.
+ */
+
+#include <linux/pci.h>
+
+#include "net_driver.h"
+#include "efx_cxl.h"
+
+#define EFX_CTPIO_BUFFER_SIZE	SZ_256M
+
+int efx_cxl_init(struct efx_probe_data *probe_data)
+{
+	struct efx_nic *efx = &probe_data->efx;
+	struct pci_dev *pci_dev = efx->pci_dev;
+	struct efx_cxl *cxl;
+	u16 dvsec;
+
+	probe_data->cxl_pio_initialised = false;
+
+	/* Is the device configured with and using CXL? */
+	if (!pcie_is_cxl(pci_dev))
+		return 0;
+
+	dvsec = pci_find_dvsec_capability(pci_dev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec) {
+		pci_err(pci_dev, "CXL_DVSEC_PCIE_DEVICE capability not found\n");
+		return 0;
+	}
+
+	pci_dbg(pci_dev, "CXL_DVSEC_PCIE_DEVICE capability found\n");
+
+	/* Create a cxl_dev_state embedded in the cxl struct using cxl core api
+	 * specifying no mbox available.
+	 */
+	cxl = devm_cxl_dev_state_create(&pci_dev->dev, CXL_DEVTYPE_DEVMEM,
+					pci_dev->dev.id, dvsec, struct efx_cxl,
+					cxlds, false);
+
+	if (!cxl)
+		return -ENOMEM;
+
+	probe_data->cxl = cxl;
+
+	return 0;
+}
+
+void efx_cxl_exit(struct efx_probe_data *probe_data)
+{
+}
+
+MODULE_IMPORT_NS("CXL");
diff --git a/drivers/net/ethernet/sfc/efx_cxl.h b/drivers/net/ethernet/sfc/efx_cxl.h
new file mode 100644
index 000000000000..961639cef692
--- /dev/null
+++ b/drivers/net/ethernet/sfc/efx_cxl.h
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/****************************************************************************
+ * Driver for AMD network controllers and boards
+ * Copyright (C) 2025, Advanced Micro Devices, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation, incorporated herein by reference.
+ */
+
+#ifndef EFX_CXL_H
+#define EFX_CXL_H
+
+#ifdef CONFIG_SFC_CXL
+
+#include <cxl/cxl.h>
+
+struct cxl_root_decoder;
+struct cxl_port;
+struct cxl_endpoint_decoder;
+struct cxl_region;
+struct efx_probe_data;
+
+struct efx_cxl {
+	struct cxl_dev_state cxlds;
+	struct cxl_memdev *cxlmd;
+	struct cxl_root_decoder *cxlrd;
+	struct cxl_port *endpoint;
+	struct cxl_endpoint_decoder *cxled;
+	struct cxl_region *efx_region;
+	void __iomem *ctpio_cxl;
+};
+
+int efx_cxl_init(struct efx_probe_data *probe_data);
+void efx_cxl_exit(struct efx_probe_data *probe_data);
+#else
+static inline int efx_cxl_init(struct efx_probe_data *probe_data) { return 0; }
+static inline void efx_cxl_exit(struct efx_probe_data *probe_data) {}
+#endif
+#endif
diff --git a/drivers/net/ethernet/sfc/net_driver.h b/drivers/net/ethernet/sfc/net_driver.h
index 5c0f306fb019..0e685b8a9980 100644
--- a/drivers/net/ethernet/sfc/net_driver.h
+++ b/drivers/net/ethernet/sfc/net_driver.h
@@ -1199,14 +1199,24 @@ struct efx_nic {
 	atomic_t n_rx_noskb_drops;
 };
 
+#ifdef CONFIG_SFC_CXL
+struct efx_cxl;
+#endif
+
 /**
  * struct efx_probe_data - State after hardware probe
  * @pci_dev: The PCI device
  * @efx: Efx NIC details
+ * @cxl: details of related cxl objects
+ * @cxl_pio_initialised: cxl initialization outcome.
  */
 struct efx_probe_data {
 	struct pci_dev *pci_dev;
 	struct efx_nic efx;
+#ifdef CONFIG_SFC_CXL
+	struct efx_cxl *cxl;
+	bool cxl_pio_initialised;
+#endif
 };
 
 static inline struct efx_nic *efx_netdev_priv(struct net_device *dev)
-- 
2.34.1

[PATCH v19 06/22] cxl: Move pci generic code

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Inside cxl/core/pci.c there are helpers for CXL PCIe initialization
meanwhile cxl/pci.c implements the functionality for a Type3 device
initialization.

Move helper functions from cxl/pci.c to cxl/core/pci.c in order to be
exported and shared with CXL Type2 device initialization.

Fix cxl mock tests affected by the code move, deleting a function which
indeed was not being used since commit 733b57f262b0("cxl/pci: Early
setup RCH dport component registers from RCRB").

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
Reviewed-by: Fan Ni <fan.ni@samsung.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/cxl/core/core.h       |  2 +
 drivers/cxl/core/pci.c        | 62 +++++++++++++++++++++++++++++++
 drivers/cxl/core/regs.c       |  1 -
 drivers/cxl/cxl.h             |  2 -
 drivers/cxl/cxlpci.h          | 13 +++++++
 drivers/cxl/pci.c             | 70 -----------------------------------
 tools/testing/cxl/Kbuild      |  1 -
 tools/testing/cxl/test/mock.c | 17 ---------
 8 files changed, 77 insertions(+), 91 deletions(-)

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index 385bfd38b778..d96213c02fd6 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -207,4 +207,6 @@ int cxl_set_feature(struct cxl_mailbox *cxl_mbox, const uuid_t *feat_uuid,
 		    u16 *return_code);
 #endif
 
+resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
+					   struct cxl_dport *dport);
 #endif /* __CXL_CORE_H__ */
diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
index d677691f8a05..2059017ba8b7 100644
--- a/drivers/cxl/core/pci.c
+++ b/drivers/cxl/core/pci.c
@@ -727,6 +727,68 @@ bool cxl_endpoint_decoder_reset_detected(struct cxl_port *port)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_endpoint_decoder_reset_detected, "CXL");
 
+static int cxl_rcrb_get_comp_regs(struct pci_dev *pdev,
+				  struct cxl_register_map *map,
+				  struct cxl_dport *dport)
+{
+	resource_size_t component_reg_phys;
+
+	*map = (struct cxl_register_map) {
+		.host = &pdev->dev,
+		.resource = CXL_RESOURCE_NONE,
+	};
+
+	struct cxl_port *port __free(put_cxl_port) =
+		cxl_pci_find_port(pdev, &dport);
+	if (!port)
+		return -EPROBE_DEFER;
+
+	component_reg_phys = cxl_rcd_component_reg_phys(&pdev->dev, dport);
+	if (component_reg_phys == CXL_RESOURCE_NONE)
+		return -ENXIO;
+
+	map->resource = component_reg_phys;
+	map->reg_type = CXL_REGLOC_RBI_COMPONENT;
+	map->max_size = CXL_COMPONENT_REG_BLOCK_SIZE;
+
+	return 0;
+}
+
+int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
+		       struct cxl_register_map *map)
+{
+	int rc;
+
+	rc = cxl_find_regblock(pdev, type, map);
+
+	/*
+	 * If the Register Locator DVSEC does not exist, check if it
+	 * is an RCH and try to extract the Component Registers from
+	 * an RCRB.
+	 */
+	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev)) {
+		struct cxl_dport *dport;
+		struct cxl_port *port __free(put_cxl_port) =
+			cxl_pci_find_port(pdev, &dport);
+		if (!port)
+			return -EPROBE_DEFER;
+
+		rc = cxl_rcrb_get_comp_regs(pdev, map, dport);
+		if (rc)
+			return rc;
+
+		rc = cxl_dport_map_rcd_linkcap(pdev, dport);
+		if (rc)
+			return rc;
+
+	} else if (rc) {
+		return rc;
+	}
+
+	return cxl_setup_regs(map);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_pci_setup_regs, "CXL");
+
 int cxl_pci_get_bandwidth(struct pci_dev *pdev, struct access_coordinate *c)
 {
 	int speed, bw;
diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index fb70ffbba72d..fc7fbd4f39d2 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -641,4 +641,3 @@ resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
 		return CXL_RESOURCE_NONE;
 	return __rcrb_to_component(dev, &dport->rcrb, CXL_RCRB_UPSTREAM);
 }
-EXPORT_SYMBOL_NS_GPL(cxl_rcd_component_reg_phys, "CXL");
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index db8e74c55309..e197c36c7525 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -221,8 +221,6 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
 		      struct cxl_register_map *map);
 int cxl_setup_regs(struct cxl_register_map *map);
 struct cxl_dport;
-resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
-					   struct cxl_dport *dport);
 int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev, struct cxl_dport *dport);
 
 #define CXL_RESOURCE_NONE ((resource_size_t) -1)
diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h
index ccf0ca36bc00..4b11757a46ab 100644
--- a/drivers/cxl/cxlpci.h
+++ b/drivers/cxl/cxlpci.h
@@ -74,9 +74,22 @@ static inline bool cxl_pci_flit_256(struct pci_dev *pdev)
 	return lnksta2 & PCI_EXP_LNKSTA2_FLIT;
 }
 
+/*
+ * Assume that the caller has already validated that @pdev has CXL
+ * capabilities, any RCIEp with CXL capabilities is treated as a
+ * Restricted CXL Device (RCD) and finds upstream port and endpoint
+ * registers in a Root Complex Register Block (RCRB).
+ */
+static inline bool is_cxl_restricted(struct pci_dev *pdev)
+{
+	return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END;
+}
+
 int devm_cxl_port_enumerate_dports(struct cxl_port *port);
 struct cxl_dev_state;
 int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
 			struct cxl_endpoint_dvsec_info *info);
 void read_cdat_data(struct cxl_port *port);
+int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
+		       struct cxl_register_map *map);
 #endif /* __CXL_PCI_H__ */
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 58532b1216dd..6544a3ca41b0 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -467,76 +467,6 @@ static int cxl_pci_setup_mailbox(struct cxl_memdev_state *mds, bool irq_avail)
 	return 0;
 }
 
-/*
- * Assume that any RCIEP that emits the CXL memory expander class code
- * is an RCD
- */
-static bool is_cxl_restricted(struct pci_dev *pdev)
-{
-	return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END;
-}
-
-static int cxl_rcrb_get_comp_regs(struct pci_dev *pdev,
-				  struct cxl_register_map *map,
-				  struct cxl_dport *dport)
-{
-	resource_size_t component_reg_phys;
-
-	*map = (struct cxl_register_map) {
-		.host = &pdev->dev,
-		.resource = CXL_RESOURCE_NONE,
-	};
-
-	struct cxl_port *port __free(put_cxl_port) =
-		cxl_pci_find_port(pdev, &dport);
-	if (!port)
-		return -EPROBE_DEFER;
-
-	component_reg_phys = cxl_rcd_component_reg_phys(&pdev->dev, dport);
-	if (component_reg_phys == CXL_RESOURCE_NONE)
-		return -ENXIO;
-
-	map->resource = component_reg_phys;
-	map->reg_type = CXL_REGLOC_RBI_COMPONENT;
-	map->max_size = CXL_COMPONENT_REG_BLOCK_SIZE;
-
-	return 0;
-}
-
-static int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
-			      struct cxl_register_map *map)
-{
-	int rc;
-
-	rc = cxl_find_regblock(pdev, type, map);
-
-	/*
-	 * If the Register Locator DVSEC does not exist, check if it
-	 * is an RCH and try to extract the Component Registers from
-	 * an RCRB.
-	 */
-	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev)) {
-		struct cxl_dport *dport;
-		struct cxl_port *port __free(put_cxl_port) =
-			cxl_pci_find_port(pdev, &dport);
-		if (!port)
-			return -EPROBE_DEFER;
-
-		rc = cxl_rcrb_get_comp_regs(pdev, map, dport);
-		if (rc)
-			return rc;
-
-		rc = cxl_dport_map_rcd_linkcap(pdev, dport);
-		if (rc)
-			return rc;
-
-	} else if (rc) {
-		return rc;
-	}
-
-	return cxl_setup_regs(map);
-}
-
 static int cxl_pci_ras_unmask(struct pci_dev *pdev)
 {
 	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
diff --git a/tools/testing/cxl/Kbuild b/tools/testing/cxl/Kbuild
index 385301aeaeb3..629880c5b9ed 100644
--- a/tools/testing/cxl/Kbuild
+++ b/tools/testing/cxl/Kbuild
@@ -12,7 +12,6 @@ ldflags-y += --wrap=cxl_await_media_ready
 ldflags-y += --wrap=cxl_hdm_decode_init
 ldflags-y += --wrap=cxl_dvsec_rr_decode
 ldflags-y += --wrap=devm_cxl_add_rch_dport
-ldflags-y += --wrap=cxl_rcd_component_reg_phys
 ldflags-y += --wrap=cxl_endpoint_parse_cdat
 ldflags-y += --wrap=cxl_dport_init_ras_reporting
 
diff --git a/tools/testing/cxl/test/mock.c b/tools/testing/cxl/test/mock.c
index 1989ae020df3..c471400116a1 100644
--- a/tools/testing/cxl/test/mock.c
+++ b/tools/testing/cxl/test/mock.c
@@ -268,23 +268,6 @@ struct cxl_dport *__wrap_devm_cxl_add_rch_dport(struct cxl_port *port,
 }
 EXPORT_SYMBOL_NS_GPL(__wrap_devm_cxl_add_rch_dport, "CXL");
 
-resource_size_t __wrap_cxl_rcd_component_reg_phys(struct device *dev,
-						  struct cxl_dport *dport)
-{
-	int index;
-	resource_size_t component_reg_phys;
-	struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
-
-	if (ops && ops->is_mock_port(dev))
-		component_reg_phys = CXL_RESOURCE_NONE;
-	else
-		component_reg_phys = cxl_rcd_component_reg_phys(dev, dport);
-	put_cxl_mock_ops(index);
-
-	return component_reg_phys;
-}
-EXPORT_SYMBOL_NS_GPL(__wrap_cxl_rcd_component_reg_phys, "CXL");
-
 void __wrap_cxl_endpoint_parse_cdat(struct cxl_port *port)
 {
 	int index;
-- 
2.34.1

[PATCH v19 07/22] cxl: allow Type2 drivers to map cxl component regs

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Export cxl core functions for a Type2 driver being able to discover and
map the device component registers.

Use it in sfc driver cxl initialization.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
---
 drivers/cxl/core/pci.c             |  1 +
 drivers/cxl/core/port.c            |  1 +
 drivers/cxl/core/regs.c            |  1 +
 drivers/cxl/cxl.h                  |  7 ------
 drivers/cxl/cxlpci.h               | 12 ----------
 drivers/cxl/pci.c                  |  1 +
 drivers/net/ethernet/sfc/efx_cxl.c | 35 ++++++++++++++++++++++++++++++
 include/cxl/cxl.h                  | 19 ++++++++++++++++
 include/cxl/pci.h                  | 21 ++++++++++++++++++
 9 files changed, 79 insertions(+), 19 deletions(-)
 create mode 100644 include/cxl/pci.h

diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
index 2059017ba8b7..33dbec3d18c5 100644
--- a/drivers/cxl/core/pci.c
+++ b/drivers/cxl/core/pci.c
@@ -5,6 +5,7 @@
 #include <linux/device.h>
 #include <linux/delay.h>
 #include <linux/pci.h>
+#include <cxl/pci.h>
 #include <linux/pci-doe.h>
 #include <cxlpci.h>
 #include <cxlmem.h>
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index bb326dc95d5f..240c3c5bcdc8 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -11,6 +11,7 @@
 #include <linux/idr.h>
 #include <linux/node.h>
 #include <cxl/einj.h>
+#include <cxl/pci.h>
 #include <cxlmem.h>
 #include <cxlpci.h>
 #include <cxl.h>
diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index fc7fbd4f39d2..dcf444f1fe48 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -4,6 +4,7 @@
 #include <linux/device.h>
 #include <linux/slab.h>
 #include <linux/pci.h>
+#include <cxl/pci.h>
 #include <cxlmem.h>
 #include <cxlpci.h>
 #include <pmu.h>
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index e197c36c7525..793d4dfe51a2 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -38,10 +38,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
 #define   CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24)
 #define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)
 
-#define   CXL_CM_CAP_CAP_ID_RAS 0x2
-#define   CXL_CM_CAP_CAP_ID_HDM 0x5
-#define   CXL_CM_CAP_CAP_HDM_VERSION 1
-
 /* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
 #define CXL_HDM_DECODER_CAP_OFFSET 0x0
 #define   CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
@@ -205,9 +201,6 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
 			      struct cxl_component_reg_map *map);
 void cxl_probe_device_regs(struct device *dev, void __iomem *base,
 			   struct cxl_device_reg_map *map);
-int cxl_map_component_regs(const struct cxl_register_map *map,
-			   struct cxl_component_regs *regs,
-			   unsigned long map_mask);
 int cxl_map_device_regs(const struct cxl_register_map *map,
 			struct cxl_device_regs *regs);
 int cxl_map_pmu_regs(struct cxl_register_map *map, struct cxl_pmu_regs *regs);
diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h
index 4b11757a46ab..2247823acf6f 100644
--- a/drivers/cxl/cxlpci.h
+++ b/drivers/cxl/cxlpci.h
@@ -13,16 +13,6 @@
  */
 #define CXL_PCI_DEFAULT_MAX_VECTORS 16
 
-/* Register Block Identifier (RBI) */
-enum cxl_regloc_type {
-	CXL_REGLOC_RBI_EMPTY = 0,
-	CXL_REGLOC_RBI_COMPONENT,
-	CXL_REGLOC_RBI_VIRT,
-	CXL_REGLOC_RBI_MEMDEV,
-	CXL_REGLOC_RBI_PMU,
-	CXL_REGLOC_RBI_TYPES
-};
-
 /*
  * Table Access DOE, CDAT Read Entry Response
  *
@@ -90,6 +80,4 @@ struct cxl_dev_state;
 int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
 			struct cxl_endpoint_dvsec_info *info);
 void read_cdat_data(struct cxl_port *port);
-int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
-		       struct cxl_register_map *map);
 #endif /* __CXL_PCI_H__ */
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 6544a3ca41b0..2e9f71d3a214 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -12,6 +12,7 @@
 #include <linux/aer.h>
 #include <linux/io.h>
 #include <cxl/cxl.h>
+#include <cxl/pci.h>
 #include <cxl/mailbox.h>
 #include "cxlmem.h"
 #include "cxlpci.h"
diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
index 8e0481d8dced..34126bc4826c 100644
--- a/drivers/net/ethernet/sfc/efx_cxl.c
+++ b/drivers/net/ethernet/sfc/efx_cxl.c
@@ -7,6 +7,8 @@
 
 #include <linux/pci.h>
 
+#include <cxl/cxl.h>
+#include <cxl/pci.h>
 #include "net_driver.h"
 #include "efx_cxl.h"
 
@@ -18,6 +20,7 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 	struct pci_dev *pci_dev = efx->pci_dev;
 	struct efx_cxl *cxl;
 	u16 dvsec;
+	int rc;
 
 	probe_data->cxl_pio_initialised = false;
 
@@ -44,6 +47,38 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 	if (!cxl)
 		return -ENOMEM;
 
+	rc = cxl_pci_setup_regs(pci_dev, CXL_REGLOC_RBI_COMPONENT,
+				&cxl->cxlds.reg_map);
+	if (rc) {
+		pci_err(pci_dev, "No component registers\n");
+		return rc;
+	}
+
+	if (!cxl->cxlds.reg_map.component_map.hdm_decoder.valid) {
+		pci_err(pci_dev, "Expected HDM component register not found\n");
+		return -ENODEV;
+	}
+
+	if (!cxl->cxlds.reg_map.component_map.ras.valid) {
+		pci_err(pci_dev, "Expected RAS component register not found\n");
+		return -ENODEV;
+	}
+
+	rc = cxl_map_component_regs(&cxl->cxlds.reg_map,
+				    &cxl->cxlds.regs.component,
+				    BIT(CXL_CM_CAP_CAP_ID_RAS));
+	if (rc) {
+		pci_err(pci_dev, "Failed to map RAS capability.\n");
+		return rc;
+	}
+
+	/*
+	 * Set media ready explicitly as there are neither mailbox for checking
+	 * this state nor the CXL register involved, both not mandatory for
+	 * type2.
+	 */
+	cxl->cxlds.media_ready = true;
+
 	probe_data->cxl = cxl;
 
 	return 0;
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index 13d448686189..7f2e23bce1f7 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -70,6 +70,10 @@ struct cxl_regs {
 	);
 };
 
+#define   CXL_CM_CAP_CAP_ID_RAS 0x2
+#define   CXL_CM_CAP_CAP_ID_HDM 0x5
+#define   CXL_CM_CAP_CAP_HDM_VERSION 1
+
 struct cxl_reg_map {
 	bool valid;
 	int id;
@@ -223,4 +227,19 @@ struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev,
 		(drv_struct *)_devm_cxl_dev_state_create(parent, type, serial, dvsec,	\
 						      sizeof(drv_struct), mbox);	\
 	})
+
+/**
+ * cxl_map_component_regs - map cxl component registers
+ *
+ * @map: cxl register map to update with the mappings
+ * @regs: cxl component registers to work with
+ * @map_mask: cxl component regs to map
+ *
+ * Returns integer: success (0) or error (-ENOMEM)
+ *
+ * Made public for Type2 driver support.
+ */
+int cxl_map_component_regs(const struct cxl_register_map *map,
+			   struct cxl_component_regs *regs,
+			   unsigned long map_mask);
 #endif /* __CXL_CXL_H__ */
diff --git a/include/cxl/pci.h b/include/cxl/pci.h
new file mode 100644
index 000000000000..a172439f08c6
--- /dev/null
+++ b/include/cxl/pci.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+
+#ifndef __CXL_CXL_PCI_H__
+#define __CXL_CXL_PCI_H__
+
+/* Register Block Identifier (RBI) */
+enum cxl_regloc_type {
+	CXL_REGLOC_RBI_EMPTY = 0,
+	CXL_REGLOC_RBI_COMPONENT,
+	CXL_REGLOC_RBI_VIRT,
+	CXL_REGLOC_RBI_MEMDEV,
+	CXL_REGLOC_RBI_PMU,
+	CXL_REGLOC_RBI_TYPES
+};
+
+struct cxl_register_map;
+
+int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
+		       struct cxl_register_map *map);
+#endif
-- 
2.34.1

[PATCH v19 08/22] cxl: Support dpa initialization without a mailbox

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Type3 relies on mailbox CXL_MBOX_OP_IDENTIFY command for initializing
memdev state params which end up being used for DPA initialization.

Allow a Type2 driver to initialize DPA simply by giving the size of its
volatile hardware partition.

Move related functions to memdev.

Add sfc driver as the client.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
---
 drivers/cxl/core/core.h            |  2 +
 drivers/cxl/core/mbox.c            | 51 +----------------------
 drivers/cxl/core/memdev.c          | 66 ++++++++++++++++++++++++++++++
 drivers/net/ethernet/sfc/efx_cxl.c |  5 +++
 include/cxl/cxl.h                  |  1 +
 5 files changed, 75 insertions(+), 50 deletions(-)

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index d96213c02fd6..c4dddbec5d6e 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -90,6 +90,8 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
 struct dentry *cxl_debugfs_create_dir(const char *dir);
 int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
 		     enum cxl_partition_mode mode);
+struct cxl_memdev_state;
+int cxl_mem_get_partition_info(struct cxl_memdev_state *mds);
 int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u64 size);
 int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
 resource_size_t cxl_dpa_size(struct cxl_endpoint_decoder *cxled);
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index bee84d0101d1..d57a0c2d39fb 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -1144,7 +1144,7 @@ EXPORT_SYMBOL_NS_GPL(cxl_mem_get_event_records, "CXL");
  *
  * See CXL @8.2.9.5.2.1 Get Partition Info
  */
-static int cxl_mem_get_partition_info(struct cxl_memdev_state *mds)
+int cxl_mem_get_partition_info(struct cxl_memdev_state *mds)
 {
 	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
 	struct cxl_mbox_get_partition_info pi;
@@ -1300,55 +1300,6 @@ int cxl_mem_sanitize(struct cxl_memdev *cxlmd, u16 cmd)
 	return -EBUSY;
 }
 
-static void add_part(struct cxl_dpa_info *info, u64 start, u64 size, enum cxl_partition_mode mode)
-{
-	int i = info->nr_partitions;
-
-	if (size == 0)
-		return;
-
-	info->part[i].range = (struct range) {
-		.start = start,
-		.end = start + size - 1,
-	};
-	info->part[i].mode = mode;
-	info->nr_partitions++;
-}
-
-int cxl_mem_dpa_fetch(struct cxl_memdev_state *mds, struct cxl_dpa_info *info)
-{
-	struct cxl_dev_state *cxlds = &mds->cxlds;
-	struct device *dev = cxlds->dev;
-	int rc;
-
-	if (!cxlds->media_ready) {
-		info->size = 0;
-		return 0;
-	}
-
-	info->size = mds->total_bytes;
-
-	if (mds->partition_align_bytes == 0) {
-		add_part(info, 0, mds->volatile_only_bytes, CXL_PARTMODE_RAM);
-		add_part(info, mds->volatile_only_bytes,
-			 mds->persistent_only_bytes, CXL_PARTMODE_PMEM);
-		return 0;
-	}
-
-	rc = cxl_mem_get_partition_info(mds);
-	if (rc) {
-		dev_err(dev, "Failed to query partition information\n");
-		return rc;
-	}
-
-	add_part(info, 0, mds->active_volatile_bytes, CXL_PARTMODE_RAM);
-	add_part(info, mds->active_volatile_bytes, mds->active_persistent_bytes,
-		 CXL_PARTMODE_PMEM);
-
-	return 0;
-}
-EXPORT_SYMBOL_NS_GPL(cxl_mem_dpa_fetch, "CXL");
-
 int cxl_get_dirty_count(struct cxl_memdev_state *mds, u32 *count)
 {
 	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
index 66d3940fb60a..1f1890a93276 100644
--- a/drivers/cxl/core/memdev.c
+++ b/drivers/cxl/core/memdev.c
@@ -556,6 +556,72 @@ bool is_cxl_memdev(const struct device *dev)
 }
 EXPORT_SYMBOL_NS_GPL(is_cxl_memdev, "CXL");
 
+static void add_part(struct cxl_dpa_info *info, u64 start, u64 size, enum cxl_partition_mode mode)
+{
+	int i = info->nr_partitions;
+
+	if (size == 0)
+		return;
+
+	info->part[i].range = (struct range) {
+		.start = start,
+		.end = start + size - 1,
+	};
+	info->part[i].mode = mode;
+	info->nr_partitions++;
+}
+
+int cxl_mem_dpa_fetch(struct cxl_memdev_state *mds, struct cxl_dpa_info *info)
+{
+	struct cxl_dev_state *cxlds = &mds->cxlds;
+	struct device *dev = cxlds->dev;
+	int rc;
+
+	if (!cxlds->media_ready) {
+		info->size = 0;
+		return 0;
+	}
+
+	info->size = mds->total_bytes;
+
+	if (mds->partition_align_bytes == 0) {
+		add_part(info, 0, mds->volatile_only_bytes, CXL_PARTMODE_RAM);
+		add_part(info, mds->volatile_only_bytes,
+			 mds->persistent_only_bytes, CXL_PARTMODE_PMEM);
+		return 0;
+	}
+
+	rc = cxl_mem_get_partition_info(mds);
+	if (rc) {
+		dev_err(dev, "Failed to query partition information\n");
+		return rc;
+	}
+
+	add_part(info, 0, mds->active_volatile_bytes, CXL_PARTMODE_RAM);
+	add_part(info, mds->active_volatile_bytes, mds->active_persistent_bytes,
+		 CXL_PARTMODE_PMEM);
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_mem_dpa_fetch, "CXL");
+
+/**
+ * cxl_set_capacity: initialize dpa by a driver without a mailbox.
+ *
+ * @cxlds: pointer to cxl_dev_state
+ * @capacity: device volatile memory size
+ */
+int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity)
+{
+	struct cxl_dpa_info range_info = {
+		.size = capacity,
+	};
+
+	add_part(&range_info, 0, capacity, CXL_PARTMODE_RAM);
+	return cxl_dpa_setup(cxlds, &range_info);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_set_capacity, "CXL");
+
 /**
  * set_exclusive_cxl_commands() - atomically disable user cxl commands
  * @mds: The device state to operate on
diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
index 34126bc4826c..0b10a2e6aceb 100644
--- a/drivers/net/ethernet/sfc/efx_cxl.c
+++ b/drivers/net/ethernet/sfc/efx_cxl.c
@@ -79,6 +79,11 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 	 */
 	cxl->cxlds.media_ready = true;
 
+	if (cxl_set_capacity(&cxl->cxlds, EFX_CTPIO_BUFFER_SIZE)) {
+		pci_err(pci_dev, "dpa capacity setup failed\n");
+		return -ENODEV;
+	}
+
 	probe_data->cxl = cxl;
 
 	return 0;
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index 7f2e23bce1f7..fb2f8f2395d5 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -242,4 +242,5 @@ struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev,
 int cxl_map_component_regs(const struct cxl_register_map *map,
 			   struct cxl_component_regs *regs,
 			   unsigned long map_mask);
+int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity);
 #endif /* __CXL_CXL_H__ */
-- 
2.34.1

[PATCH v19 09/22] cxl: Prepare memdev creation for type2

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Current cxl core is relying on a CXL_DEVTYPE_CLASSMEM type device when
creating a memdev leading to problems when obtaining cxl_memdev_state
references from a CXL_DEVTYPE_DEVMEM type.

Modify check for obtaining cxl_memdev_state adding CXL_DEVTYPE_DEVMEM
support.

Make devm_cxl_add_memdev accessible from a accel driver.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/cxl/core/memdev.c | 15 +++++++++++--
 drivers/cxl/cxlmem.h      |  8 -------
 drivers/cxl/mem.c         | 45 +++++++++++++++++++++++++++++----------
 include/cxl/cxl.h         |  7 ++++++
 4 files changed, 54 insertions(+), 21 deletions(-)

diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
index 1f1890a93276..fd3058638c35 100644
--- a/drivers/cxl/core/memdev.c
+++ b/drivers/cxl/core/memdev.c
@@ -7,6 +7,7 @@
 #include <linux/slab.h>
 #include <linux/idr.h>
 #include <linux/pci.h>
+#include <cxl/cxl.h>
 #include <cxlmem.h>
 #include "private.h"
 #include "trace.h"
@@ -550,9 +551,16 @@ static const struct device_type cxl_memdev_type = {
 	.groups = cxl_memdev_attribute_groups,
 };
 
+static const struct device_type cxl_accel_memdev_type = {
+	.name = "cxl_accel_memdev",
+	.release = cxl_memdev_release,
+	.devnode = cxl_memdev_devnode,
+};
+
 bool is_cxl_memdev(const struct device *dev)
 {
-	return dev->type == &cxl_memdev_type;
+	return (dev->type == &cxl_memdev_type ||
+		dev->type == &cxl_accel_memdev_type);
 }
 EXPORT_SYMBOL_NS_GPL(is_cxl_memdev, "CXL");
 
@@ -1138,7 +1146,10 @@ struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
 	dev->parent = cxlds->dev;
 	dev->bus = &cxl_bus_type;
 	dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
-	dev->type = &cxl_memdev_type;
+	if (cxlds->type == CXL_DEVTYPE_DEVMEM)
+		dev->type = &cxl_accel_memdev_type;
+	else
+		dev->type = &cxl_memdev_type;
 	device_set_pm_not_required(dev);
 	INIT_WORK(&cxlmd->detach_work, detach_memdev);
 
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 86aa4899d511..0e02cd9f0bad 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -34,10 +34,6 @@
 	(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) !=                       \
 	 CXLMDEV_RESET_NEEDED_NOT)
 
-struct cxl_memdev_ops {
-	int (*probe)(struct cxl_memdev *cxlmd);
-};
-
 /**
  * struct cxl_memdev - CXL bus object representing a Type-3 Memory Device
  * @dev: driver core device object
@@ -101,10 +97,6 @@ static inline bool is_cxl_endpoint(struct cxl_port *port)
 	return is_cxl_memdev(port->uport_dev);
 }
 
-struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
-				       struct cxl_dev_state *cxlds,
-				       const struct cxl_memdev_ops *ops);
-
 int devm_cxl_sanitize_setup_notifier(struct device *host,
 				     struct cxl_memdev *cxlmd);
 struct cxl_memdev_state;
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index aeb2e3e8282a..f732e35507a3 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -67,6 +67,26 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
 DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_clear_fops, NULL,
 			 cxl_debugfs_poison_clear, "%llx\n");
 
+static void cxl_memdev_poison_enable(struct cxl_memdev_state *mds,
+				     struct cxl_memdev *cxlmd,
+				     struct dentry *dentry)
+{
+	/*
+	 * Avoid poison debugfs for DEVMEM aka accelerators as they rely on
+	 * cxl_memdev_state.
+	 */
+	if (!mds)
+		return;
+
+	if (test_bit(CXL_POISON_ENABLED_INJECT, mds->poison.enabled_cmds))
+		debugfs_create_file("inject_poison", 0200, dentry, cxlmd,
+				    &cxl_poison_inject_fops);
+
+	if (test_bit(CXL_POISON_ENABLED_CLEAR, mds->poison.enabled_cmds))
+		debugfs_create_file("clear_poison", 0200, dentry, cxlmd,
+				    &cxl_poison_clear_fops);
+}
+
 static int cxl_mem_probe(struct device *dev)
 {
 	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
@@ -94,12 +114,7 @@ static int cxl_mem_probe(struct device *dev)
 	dentry = cxl_debugfs_create_dir(dev_name(dev));
 	debugfs_create_devm_seqfile(dev, "dpamem", dentry, cxl_mem_dpa_show);
 
-	if (test_bit(CXL_POISON_ENABLED_INJECT, mds->poison.enabled_cmds))
-		debugfs_create_file("inject_poison", 0200, dentry, cxlmd,
-				    &cxl_poison_inject_fops);
-	if (test_bit(CXL_POISON_ENABLED_CLEAR, mds->poison.enabled_cmds))
-		debugfs_create_file("clear_poison", 0200, dentry, cxlmd,
-				    &cxl_poison_clear_fops);
+	cxl_memdev_poison_enable(mds, cxlmd, dentry);
 
 	rc = devm_add_action_or_reset(dev, remove_debugfs, dentry);
 	if (rc)
@@ -238,16 +253,24 @@ static ssize_t trigger_poison_list_store(struct device *dev,
 }
 static DEVICE_ATTR_WO(trigger_poison_list);
 
-static umode_t cxl_mem_visible(struct kobject *kobj, struct attribute *a, int n)
+static bool cxl_poison_attr_visible(struct kobject *kobj, struct attribute *a)
 {
 	struct device *dev = kobj_to_dev(kobj);
 	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
 	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
 
-	if (a == &dev_attr_trigger_poison_list.attr)
-		if (!test_bit(CXL_POISON_ENABLED_LIST,
-			      mds->poison.enabled_cmds))
-			return 0;
+	if (!mds ||
+	    !test_bit(CXL_POISON_ENABLED_LIST, mds->poison.enabled_cmds))
+		return false;
+
+	return true;
+}
+
+static umode_t cxl_mem_visible(struct kobject *kobj, struct attribute *a, int n)
+{
+	if (a == &dev_attr_trigger_poison_list.attr &&
+	    !cxl_poison_attr_visible(kobj, a))
+		return 0;
 
 	return a->mode;
 }
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index fb2f8f2395d5..043fc31c764e 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -153,6 +153,10 @@ struct cxl_dpa_partition {
 
 #define CXL_NR_PARTITIONS_MAX 2
 
+struct cxl_memdev_ops {
+	int (*probe)(struct cxl_memdev *cxlmd);
+};
+
 /**
  * struct cxl_dev_state - The driver device state
  *
@@ -243,4 +247,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
 			   struct cxl_component_regs *regs,
 			   unsigned long map_mask);
 int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity);
+struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
+				       struct cxl_dev_state *cxlds,
+				       const struct cxl_memdev_ops *ops);
 #endif /* __CXL_CXL_H__ */
-- 
2.34.1

[PATCH v19 10/22] sfc: create type2 cxl memdev

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Use cxl API for creating a cxl memory device using the type2
cxl_dev_state struct.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Martin Habets <habetsm.xilinx@gmail.com>
Reviewed-by: Fan Ni <fan.ni@samsung.com>
Acked-by: Edward Cree <ecree.xilinx@gmail.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/net/ethernet/sfc/efx_cxl.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
index 0b10a2e6aceb..f6eda93e67e2 100644
--- a/drivers/net/ethernet/sfc/efx_cxl.c
+++ b/drivers/net/ethernet/sfc/efx_cxl.c
@@ -84,6 +84,12 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 		return -ENODEV;
 	}
 
+	cxl->cxlmd = devm_cxl_add_memdev(&pci_dev->dev, &cxl->cxlds, NULL);
+	if (IS_ERR(cxl->cxlmd)) {
+		pci_err(pci_dev, "CXL accel memdev creation failed");
+		return PTR_ERR(cxl->cxlmd);
+	}
+
 	probe_data->cxl = cxl;
 
 	return 0;
-- 
2.34.1

[PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

CXL region creation involves allocating capacity from Device Physical Address
(DPA) and assigning it to decode a given Host Physical Address (HPA). Before
determining how much DPA to allocate the amount of available HPA must be
determined. Also, not all HPA is created equal, some HPA targets RAM, some
targets PMEM, some is prepared for device-memory flows like HDM-D and HDM-DB,
and some is HDM-H (host-only).

In order to support Type2 CXL devices, wrap all of those concerns into
an API that retrieves a root decoder (platform CXL window) that fits the
specified constraints and the capacity available for a new region.

Add a complementary function for releasing the reference to such root
decoder.

Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 drivers/cxl/core/region.c | 162 ++++++++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h         |   3 +
 include/cxl/cxl.h         |   6 ++
 3 files changed, 171 insertions(+)

diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index e9bf42d91689..c5b66204ecde 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -703,6 +703,168 @@ static int free_hpa(struct cxl_region *cxlr)
 	return 0;
 }
 
+struct cxlrd_max_context {
+	struct device * const *host_bridges;
+	int interleave_ways;
+	unsigned long flags;
+	resource_size_t max_hpa;
+	struct cxl_root_decoder *cxlrd;
+};
+
+static int find_max_hpa(struct device *dev, void *data)
+{
+	struct cxlrd_max_context *ctx = data;
+	struct cxl_switch_decoder *cxlsd;
+	struct cxl_root_decoder *cxlrd;
+	struct resource *res, *prev;
+	struct cxl_decoder *cxld;
+	resource_size_t max;
+	int found = 0;
+
+	if (!is_root_decoder(dev))
+		return 0;
+
+	cxlrd = to_cxl_root_decoder(dev);
+	cxlsd = &cxlrd->cxlsd;
+	cxld = &cxlsd->cxld;
+
+	if ((cxld->flags & ctx->flags) != ctx->flags) {
+		dev_dbg(dev, "flags not matching: %08lx vs %08lx\n",
+			cxld->flags, ctx->flags);
+		return 0;
+	}
+
+	for (int i = 0; i < ctx->interleave_ways; i++) {
+		for (int j = 0; j < ctx->interleave_ways; j++) {
+			if (ctx->host_bridges[i] == cxlsd->target[j]->dport_dev) {
+				found++;
+				break;
+			}
+		}
+	}
+
+	if (found != ctx->interleave_ways) {
+		dev_dbg(dev,
+			"Not enough host bridges. Found %d for %d interleave ways requested\n",
+			found, ctx->interleave_ways);
+		return 0;
+	}
+
+	/*
+	 * Walk the root decoder resource range relying on cxl_rwsem.region to
+	 * preclude sibling arrival/departure and find the largest free space
+	 * gap.
+	 */
+	lockdep_assert_held_read(&cxl_rwsem.region);
+	res = cxlrd->res->child;
+
+	/* With no resource child the whole parent resource is available */
+	if (!res)
+		max = resource_size(cxlrd->res);
+	else
+		max = 0;
+
+	for (prev = NULL; res; prev = res, res = res->sibling) {
+		struct resource *next = res->sibling;
+		resource_size_t free = 0;
+
+		/*
+		 * Sanity check for preventing arithmetic problems below as a
+		 * resource with size 0 could imply using the end field below
+		 * when set to unsigned zero - 1 or all f in hex.
+		 */
+		if (prev && !resource_size(prev))
+			continue;
+
+		if (!prev && res->start > cxlrd->res->start) {
+			free = res->start - cxlrd->res->start;
+			max = max(free, max);
+		}
+		if (prev && res->start > prev->end + 1) {
+			free = res->start - prev->end + 1;
+			max = max(free, max);
+		}
+		if (next && res->end + 1 < next->start) {
+			free = next->start - res->end + 1;
+			max = max(free, max);
+		}
+		if (!next && res->end + 1 < cxlrd->res->end + 1) {
+			free = cxlrd->res->end + 1 - res->end + 1;
+			max = max(free, max);
+		}
+	}
+
+	dev_dbg(cxlrd_dev(cxlrd), "found %pa bytes of free space\n", &max);
+	if (max > ctx->max_hpa) {
+		if (ctx->cxlrd)
+			put_device(cxlrd_dev(ctx->cxlrd));
+		get_device(cxlrd_dev(cxlrd));
+		ctx->cxlrd = cxlrd;
+		ctx->max_hpa = max;
+	}
+	return 0;
+}
+
+/**
+ * cxl_get_hpa_freespace - find a root decoder with free capacity per constraints
+ * @cxlmd: the mem device requiring the HPA
+ * @interleave_ways: number of entries in @host_bridges
+ * @flags: CXL_DECODER_F flags for selecting RAM vs PMEM, and Type2 device
+ * @max_avail_contig: output parameter of max contiguous bytes available in the
+ *		      returned decoder
+ *
+ * Returns a pointer to a struct cxl_root_decoder
+ *
+ * The return tuple of a 'struct cxl_root_decoder' and 'bytes available given
+ * in (@max_avail_contig))' is a point in time snapshot. If by the time the
+ * caller goes to use this decoder and its capacity is reduced then caller needs
+ * to loop and retry.
+ *
+ * The returned root decoder has an elevated reference count that needs to be
+ * put with cxl_put_root_decoder(cxlrd).
+ */
+struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
+					       int interleave_ways,
+					       unsigned long flags,
+					       resource_size_t *max_avail_contig)
+{
+	struct cxl_port *endpoint = cxlmd->endpoint;
+	struct cxlrd_max_context ctx = {
+		.flags = flags,
+	};
+	struct cxl_port *root_port;
+
+	if (!endpoint) {
+		dev_dbg(&cxlmd->dev, "endpoint not linked to memdev\n");
+		return ERR_PTR(-ENXIO);
+	}
+
+	ctx.host_bridges = &endpoint->host_bridge;
+
+	struct cxl_root *root __free(put_cxl_root) = find_cxl_root(endpoint);
+	if (!root) {
+		dev_dbg(&endpoint->dev, "endpoint is not related to a root port\n");
+		return ERR_PTR(-ENXIO);
+	}
+
+	root_port = &root->port;
+	scoped_guard(rwsem_read, &cxl_rwsem.region)
+		device_for_each_child(&root_port->dev, &ctx, find_max_hpa);
+
+	if (!ctx.cxlrd)
+		return ERR_PTR(-ENOMEM);
+
+	*max_avail_contig = ctx.max_hpa;
+	return ctx.cxlrd;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_hpa_freespace, "CXL");
+
+void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd)
+{
+	put_device(cxlrd_dev(cxlrd));
+}
+EXPORT_SYMBOL_NS_GPL(cxl_put_root_decoder, "CXL");
+
 static ssize_t size_store(struct device *dev, struct device_attribute *attr,
 			  const char *buf, size_t len)
 {
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 793d4dfe51a2..076640e91ee0 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -664,6 +664,9 @@ struct cxl_root_decoder *to_cxl_root_decoder(struct device *dev);
 struct cxl_switch_decoder *to_cxl_switch_decoder(struct device *dev);
 struct cxl_endpoint_decoder *to_cxl_endpoint_decoder(struct device *dev);
 bool is_root_decoder(struct device *dev);
+
+#define cxlrd_dev(cxlrd) (&(cxlrd)->cxlsd.cxld.dev)
+
 bool is_switch_decoder(struct device *dev);
 bool is_endpoint_decoder(struct device *dev);
 struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port,
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index 043fc31c764e..2ec514c77021 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -250,4 +250,10 @@ int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity);
 struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
 				       struct cxl_dev_state *cxlds,
 				       const struct cxl_memdev_ops *ops);
+struct cxl_port;
+struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
+					       int interleave_ways,
+					       unsigned long flags,
+					       resource_size_t *max);
+void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd);
 #endif /* __CXL_CXL_H__ */
-- 
2.34.1

[PATCH v19 12/22] sfc: get root decoder

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Use cxl api for getting HPA (Host Physical Address) to use from a
CXL root decoder.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Martin Habets <habetsm.xilinx@gmail.com>
Acked-by: Edward Cree <ecree.xilinx@gmail.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
---
 drivers/cxl/cxl.h                  | 15 ---------------
 drivers/net/ethernet/sfc/Kconfig   |  1 +
 drivers/net/ethernet/sfc/efx_cxl.c | 20 ++++++++++++++++++++
 include/cxl/cxl.h                  | 14 ++++++++++++++
 4 files changed, 35 insertions(+), 15 deletions(-)

diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 076640e91ee0..ab490b5a9457 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -219,21 +219,6 @@ int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev, struct cxl_dport *dport);
 #define CXL_RESOURCE_NONE ((resource_size_t) -1)
 #define CXL_TARGET_STRLEN 20
 
-/*
- * cxl_decoder flags that define the type of memory / devices this
- * decoder supports as well as configuration lock status See "CXL 2.0
- * 8.2.5.12.7 CXL HDM Decoder 0 Control Register" for details.
- * Additionally indicate whether decoder settings were autodetected,
- * user customized.
- */
-#define CXL_DECODER_F_RAM   BIT(0)
-#define CXL_DECODER_F_PMEM  BIT(1)
-#define CXL_DECODER_F_TYPE2 BIT(2)
-#define CXL_DECODER_F_TYPE3 BIT(3)
-#define CXL_DECODER_F_LOCK  BIT(4)
-#define CXL_DECODER_F_ENABLE    BIT(5)
-#define CXL_DECODER_F_MASK  GENMASK(5, 0)
-
 enum cxl_decoder_type {
 	CXL_DECODER_DEVMEM = 2,
 	CXL_DECODER_HOSTONLYMEM = 3,
diff --git a/drivers/net/ethernet/sfc/Kconfig b/drivers/net/ethernet/sfc/Kconfig
index 979f2801e2a8..e959d9b4f4ce 100644
--- a/drivers/net/ethernet/sfc/Kconfig
+++ b/drivers/net/ethernet/sfc/Kconfig
@@ -69,6 +69,7 @@ config SFC_MCDI_LOGGING
 config SFC_CXL
 	bool "Solarflare SFC9100-family CXL support"
 	depends on SFC && CXL_BUS >= SFC
+	depends on CXL_REGION
 	default SFC
 	help
 	  This enables SFC CXL support if the kernel is configuring CXL for
diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
index f6eda93e67e2..d7c34c978434 100644
--- a/drivers/net/ethernet/sfc/efx_cxl.c
+++ b/drivers/net/ethernet/sfc/efx_cxl.c
@@ -18,6 +18,7 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 {
 	struct efx_nic *efx = &probe_data->efx;
 	struct pci_dev *pci_dev = efx->pci_dev;
+	resource_size_t max_size;
 	struct efx_cxl *cxl;
 	u16 dvsec;
 	int rc;
@@ -90,6 +91,23 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 		return PTR_ERR(cxl->cxlmd);
 	}
 
+	cxl->cxlrd = cxl_get_hpa_freespace(cxl->cxlmd, 1,
+					   CXL_DECODER_F_RAM | CXL_DECODER_F_TYPE2,
+					   &max_size);
+
+	if (IS_ERR(cxl->cxlrd)) {
+		dev_err(&pci_dev->dev, "cxl_get_hpa_freespace failed\n");
+		return PTR_ERR(cxl->cxlrd);
+	}
+
+	if (max_size < EFX_CTPIO_BUFFER_SIZE) {
+		dev_err(&pci_dev->dev,
+			"%s: not enough free HPA space %pap < %u\n",
+			__func__, &max_size, EFX_CTPIO_BUFFER_SIZE);
+		cxl_put_root_decoder(cxl->cxlrd);
+		return -ENOSPC;
+	}
+
 	probe_data->cxl = cxl;
 
 	return 0;
@@ -97,6 +115,8 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 
 void efx_cxl_exit(struct efx_probe_data *probe_data)
 {
+	if (probe_data->cxl)
+		cxl_put_root_decoder(probe_data->cxl->cxlrd);
 }
 
 MODULE_IMPORT_NS("CXL");
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index 2ec514c77021..2966b95e80a6 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -153,6 +153,20 @@ struct cxl_dpa_partition {
 
 #define CXL_NR_PARTITIONS_MAX 2
 
+/*
+ * cxl_decoder flags that define the type of memory / devices this
+ * decoder supports as well as configuration lock status See "CXL 2.0
+ * 8.2.5.12.7 CXL HDM Decoder 0 Control Register" for details.
+ * Additionally indicate whether decoder settings were autodetected,
+ * user customized.
+ */
+#define CXL_DECODER_F_RAM   BIT(0)
+#define CXL_DECODER_F_PMEM  BIT(1)
+#define CXL_DECODER_F_TYPE2 BIT(2)
+#define CXL_DECODER_F_TYPE3 BIT(3)
+#define CXL_DECODER_F_LOCK  BIT(4)
+#define CXL_DECODER_F_ENABLE    BIT(5)
+
 struct cxl_memdev_ops {
 	int (*probe)(struct cxl_memdev *cxlmd);
 };
-- 
2.34.1

[PATCH v19 13/22] cxl: Define a driver interface for DPA allocation

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Region creation involves finding available DPA (device-physical-address)
capacity to map into HPA (host-physical-address) space.

In order to support CXL Type2 devices, define an API, cxl_request_dpa(),
that tries to allocate the DPA memory the driver requires to operate.The
memory requested should not be bigger than the max available HPA obtained
previously with cxl_get_hpa_freespace().

Based on https://lore.kernel.org/linux-cxl/168592158743.1948938.7622563891193802610.stgit@dwillia2-xfh.jf.intel.com/

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
---
 drivers/cxl/core/hdm.c | 85 ++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h      |  1 +
 include/cxl/cxl.h      |  5 +++
 3 files changed, 91 insertions(+)

diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
index e9e1d555cec6..70a15694c35d 100644
--- a/drivers/cxl/core/hdm.c
+++ b/drivers/cxl/core/hdm.c
@@ -3,6 +3,7 @@
 #include <linux/seq_file.h>
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <cxl/cxl.h>
 
 #include "cxlmem.h"
 #include "core.h"
@@ -556,6 +557,13 @@ bool cxl_resource_contains_addr(const struct resource *res, const resource_size_
 	return resource_contains(res, &_addr);
 }
 
+/**
+ * cxl_dpa_free - release DPA (Device Physical Address)
+ *
+ * @cxled: endpoint decoder linked to the DPA
+ *
+ * Returns 0 or error.
+ */
 int cxl_dpa_free(struct cxl_endpoint_decoder *cxled)
 {
 	struct cxl_port *port = cxled_to_port(cxled);
@@ -582,6 +590,7 @@ int cxl_dpa_free(struct cxl_endpoint_decoder *cxled)
 	devm_cxl_dpa_release(cxled);
 	return 0;
 }
+EXPORT_SYMBOL_NS_GPL(cxl_dpa_free, "CXL");
 
 int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
 		     enum cxl_partition_mode mode)
@@ -613,6 +622,82 @@ int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
 	return 0;
 }
 
+static int find_free_decoder(struct device *dev, const void *data)
+{
+	struct cxl_endpoint_decoder *cxled;
+	struct cxl_port *port;
+
+	if (!is_endpoint_decoder(dev))
+		return 0;
+
+	cxled = to_cxl_endpoint_decoder(dev);
+	port = cxled_to_port(cxled);
+
+	return cxled->cxld.id == (port->hdm_end + 1);
+}
+
+static struct cxl_endpoint_decoder *
+cxl_find_free_decoder(struct cxl_memdev *cxlmd)
+{
+	struct cxl_port *endpoint = cxlmd->endpoint;
+	struct device *dev;
+
+	guard(rwsem_read)(&cxl_rwsem.dpa);
+	dev = device_find_child(&endpoint->dev, NULL,
+				find_free_decoder);
+	if (dev)
+		return to_cxl_endpoint_decoder(dev);
+
+	return NULL;
+}
+
+/**
+ * cxl_request_dpa - search and reserve DPA given input constraints
+ * @cxlmd: memdev with an endpoint port with available decoders
+ * @mode: CXL partition mode (ram vs pmem)
+ * @alloc: dpa size required
+ *
+ * Returns a pointer to a 'struct cxl_endpoint_decoder' on success or
+ * an errno encoded pointer on failure.
+ *
+ * Given that a region needs to allocate from limited HPA capacity it
+ * may be the case that a device has more mappable DPA capacity than
+ * available HPA. The expectation is that @alloc is a driver known
+ * value based on the device capacity but which could not be fully
+ * available due to HPA constraints.
+ *
+ * Returns a pinned cxl_decoder with at least @alloc bytes of capacity
+ * reserved, or an error pointer. The caller is also expected to own the
+ * lifetime of the memdev registration associated with the endpoint to
+ * pin the decoder registered as well.
+ */
+struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
+					     enum cxl_partition_mode mode,
+					     resource_size_t alloc)
+{
+	int rc;
+
+	if (!IS_ALIGNED(alloc, SZ_256M))
+		return ERR_PTR(-EINVAL);
+
+	struct cxl_endpoint_decoder *cxled __free(put_cxled) =
+		cxl_find_free_decoder(cxlmd);
+
+	if (!cxled)
+		return ERR_PTR(-ENODEV);
+
+	rc = cxl_dpa_set_part(cxled, mode);
+	if (rc)
+		return ERR_PTR(rc);
+
+	rc = cxl_dpa_alloc(cxled, alloc);
+	if (rc)
+		return ERR_PTR(rc);
+
+	return no_free_ptr(cxled);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_request_dpa, "CXL");
+
 static int __cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u64 size)
 {
 	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index ab490b5a9457..6ca0827cfaa5 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -625,6 +625,7 @@ struct cxl_root *find_cxl_root(struct cxl_port *port);
 
 DEFINE_FREE(put_cxl_root, struct cxl_root *, if (_T) put_device(&_T->port.dev))
 DEFINE_FREE(put_cxl_port, struct cxl_port *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
+DEFINE_FREE(put_cxled, struct cxl_endpoint_decoder *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->cxld.dev))
 DEFINE_FREE(put_cxl_root_decoder, struct cxl_root_decoder *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->cxlsd.cxld.dev))
 DEFINE_FREE(put_cxl_region, struct cxl_region *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
 
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index 2966b95e80a6..1cbe53ad0416 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -7,6 +7,7 @@
 
 #include <linux/node.h>
 #include <linux/ioport.h>
+#include <linux/range.h>
 #include <cxl/mailbox.h>
 
 /**
@@ -270,4 +271,8 @@ struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
 					       unsigned long flags,
 					       resource_size_t *max);
 void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd);
+struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
+					     enum cxl_partition_mode mode,
+					     resource_size_t alloc);
+int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
 #endif /* __CXL_CXL_H__ */
-- 
2.34.1

[PATCH v19 14/22] sfc: get endpoint decoder

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Use cxl api for getting DPA (Device Physical Address) to use through an
endpoint decoder.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Martin Habets <habetsm.xilinx@gmail.com>
Acked-by: Edward Cree <ecree.xilinx@gmail.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
---
 drivers/net/ethernet/sfc/efx_cxl.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
index d7c34c978434..1a50bb2c0913 100644
--- a/drivers/net/ethernet/sfc/efx_cxl.c
+++ b/drivers/net/ethernet/sfc/efx_cxl.c
@@ -108,6 +108,14 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 		return -ENOSPC;
 	}
 
+	cxl->cxled = cxl_request_dpa(cxl->cxlmd, CXL_PARTMODE_RAM,
+				     EFX_CTPIO_BUFFER_SIZE);
+	if (IS_ERR(cxl->cxled)) {
+		pci_err(pci_dev, "CXL accel request DPA failed");
+		cxl_put_root_decoder(cxl->cxlrd);
+		return PTR_ERR(cxl->cxled);
+	}
+
 	probe_data->cxl = cxl;
 
 	return 0;
@@ -115,8 +123,10 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 
 void efx_cxl_exit(struct efx_probe_data *probe_data)
 {
-	if (probe_data->cxl)
+	if (probe_data->cxl) {
+		cxl_dpa_free(probe_data->cxl->cxled);
 		cxl_put_root_decoder(probe_data->cxl->cxlrd);
+	}
 }
 
 MODULE_IMPORT_NS("CXL");
-- 
2.34.1

[PATCH v19 15/22] cxl: Make region type based on endpoint type

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Current code is expecting Type3 or CXL_DECODER_HOSTONLYMEM devices only.
Support for Type2 implies region type needs to be based on the endpoint
type HDM-D[B] instead.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Zhi Wang <zhiw@nvidia.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
Reviewed-by: Davidlohr Bueso <daves@stgolabs.net>
---
 drivers/cxl/core/region.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index c5b66204ecde..732bc5c6c2bd 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -2749,7 +2749,8 @@ static ssize_t create_ram_region_show(struct device *dev,
 }
 
 static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
-					  enum cxl_partition_mode mode, int id)
+					  enum cxl_partition_mode mode, int id,
+					  enum cxl_decoder_type target_type)
 {
 	int rc;
 
@@ -2771,7 +2772,7 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 		return ERR_PTR(-EBUSY);
 	}
 
-	return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
+	return devm_cxl_add_region(cxlrd, id, mode, target_type);
 }
 
 static ssize_t create_region_store(struct device *dev, const char *buf,
@@ -2785,7 +2786,7 @@ static ssize_t create_region_store(struct device *dev, const char *buf,
 	if (rc != 1)
 		return -EINVAL;
 
-	cxlr = __create_region(cxlrd, mode, id);
+	cxlr = __create_region(cxlrd, mode, id, CXL_DECODER_HOSTONLYMEM);
 	if (IS_ERR(cxlr))
 		return PTR_ERR(cxlr);
 
@@ -3577,7 +3578,8 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 
 	do {
 		cxlr = __create_region(cxlrd, cxlds->part[part].mode,
-				       atomic_read(&cxlrd->region_id));
+				       atomic_read(&cxlrd->region_id),
+				       cxled->cxld.target_type);
 	} while (IS_ERR(cxlr) && PTR_ERR(cxlr) == -EBUSY);
 
 	if (IS_ERR(cxlr)) {
-- 
2.34.1

[PATCH v19 16/22] cxl/region: Factor out interleave ways setup

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Region creation based on Type3 devices is triggered from user space
allowing memory combination through interleaving.

In preparation for kernel driven region creation, that is Type2 drivers
triggering region creation backed with its advertised CXL memory, factor
out a common helper from the user-sysfs region setup for interleave ways.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Zhi Wang <zhiw@nvidia.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
---
 drivers/cxl/core/region.c | 43 ++++++++++++++++++++++++---------------
 1 file changed, 27 insertions(+), 16 deletions(-)

diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index 732bc5c6c2bd..005a9c975020 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -483,22 +483,14 @@ static ssize_t interleave_ways_show(struct device *dev,
 
 static const struct attribute_group *get_cxl_region_target_group(void);
 
-static ssize_t interleave_ways_store(struct device *dev,
-				     struct device_attribute *attr,
-				     const char *buf, size_t len)
+static int set_interleave_ways(struct cxl_region *cxlr, int val)
 {
-	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev->parent);
+	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
 	struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
-	struct cxl_region *cxlr = to_cxl_region(dev);
 	struct cxl_region_params *p = &cxlr->params;
-	unsigned int val, save;
-	int rc;
+	int save, rc;
 	u8 iw;
 
-	rc = kstrtouint(buf, 0, &val);
-	if (rc)
-		return rc;
-
 	rc = ways_to_eiw(val, &iw);
 	if (rc)
 		return rc;
@@ -513,9 +505,7 @@ static ssize_t interleave_ways_store(struct device *dev,
 		return -EINVAL;
 	}
 
-	ACQUIRE(rwsem_write_kill, rwsem)(&cxl_rwsem.region);
-	if ((rc = ACQUIRE_ERR(rwsem_write_kill, &rwsem)))
-		return rc;
+	lockdep_assert_held_write(&cxl_rwsem.region);
 
 	if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE)
 		return -EBUSY;
@@ -523,10 +513,31 @@ static ssize_t interleave_ways_store(struct device *dev,
 	save = p->interleave_ways;
 	p->interleave_ways = val;
 	rc = sysfs_update_group(&cxlr->dev.kobj, get_cxl_region_target_group());
-	if (rc) {
+	if (rc)
 		p->interleave_ways = save;
+
+	return rc;
+}
+
+static ssize_t interleave_ways_store(struct device *dev,
+				     struct device_attribute *attr,
+				     const char *buf, size_t len)
+{
+	struct cxl_region *cxlr = to_cxl_region(dev);
+	unsigned int val;
+	int rc;
+
+	rc = kstrtouint(buf, 0, &val);
+	if (rc)
+		return rc;
+
+	ACQUIRE(rwsem_write_kill, rwsem)(&cxl_rwsem.region);
+	if ((rc = ACQUIRE_ERR(rwsem_write_kill, &rwsem)))
+		return rc;
+
+	rc = set_interleave_ways(cxlr, val);
+	if (rc)
 		return rc;
-	}
 
 	return len;
 }
-- 
2.34.1

[PATCH v19 17/22] cxl/region: Factor out interleave granularity setup

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Region creation based on Type3 devices is triggered from user space
allowing memory combination through interleaving.

In preparation for kernel driven region creation, that is Type2 drivers
triggering region creation backed with its advertised CXL memory, factor
out a common helper from the user-sysfs region setup forinterleave
granularity.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Zhi Wang <zhiw@nvidia.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
---
 drivers/cxl/core/region.c | 39 +++++++++++++++++++++++++--------------
 1 file changed, 25 insertions(+), 14 deletions(-)

diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index 005a9c975020..26dfc15e57cd 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -557,21 +557,14 @@ static ssize_t interleave_granularity_show(struct device *dev,
 	return sysfs_emit(buf, "%d\n", p->interleave_granularity);
 }
 
-static ssize_t interleave_granularity_store(struct device *dev,
-					    struct device_attribute *attr,
-					    const char *buf, size_t len)
+static int set_interleave_granularity(struct cxl_region *cxlr, int val)
 {
-	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev->parent);
+	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
 	struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
-	struct cxl_region *cxlr = to_cxl_region(dev);
 	struct cxl_region_params *p = &cxlr->params;
-	int rc, val;
+	int rc;
 	u16 ig;
 
-	rc = kstrtoint(buf, 0, &val);
-	if (rc)
-		return rc;
-
 	rc = granularity_to_eig(val, &ig);
 	if (rc)
 		return rc;
@@ -587,14 +580,32 @@ static ssize_t interleave_granularity_store(struct device *dev,
 	if (cxld->interleave_ways > 1 && val != cxld->interleave_granularity)
 		return -EINVAL;
 
-	ACQUIRE(rwsem_write_kill, rwsem)(&cxl_rwsem.region);
-	if ((rc = ACQUIRE_ERR(rwsem_write_kill, &rwsem)))
-		return rc;
-
+	lockdep_assert_held_write(&cxl_rwsem.region);
 	if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE)
 		return -EBUSY;
 
 	p->interleave_granularity = val;
+	return 0;
+}
+
+static ssize_t interleave_granularity_store(struct device *dev,
+					    struct device_attribute *attr,
+					    const char *buf, size_t len)
+{
+	struct cxl_region *cxlr = to_cxl_region(dev);
+	int rc, val;
+
+	rc = kstrtoint(buf, 0, &val);
+	if (rc)
+		return rc;
+
+	ACQUIRE(rwsem_write_kill, rwsem)(&cxl_rwsem.region);
+	if ((rc = ACQUIRE_ERR(rwsem_write_kill, &rwsem)))
+		return rc;
+
+	rc = set_interleave_granularity(cxlr, val);
+	if (rc)
+		return rc;
 
 	return len;
 }
-- 
2.34.1

[PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Creating a CXL region requires userspace intervention through the cxl
sysfs files. Type2 support should allow accelerator drivers to create
such cxl region from kernel code.

Adding that functionality and integrating it with current support for
memory expanders.

Support an action by the type2 driver to be linked to the created region
for unwinding the resources allocated properly.

Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 drivers/cxl/core/core.h   |   5 --
 drivers/cxl/core/region.c | 134 +++++++++++++++++++++++++++++++++++---
 drivers/cxl/port.c        |   5 +-
 include/cxl/cxl.h         |  11 ++++
 4 files changed, 141 insertions(+), 14 deletions(-)

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index c4dddbec5d6e..83abaca9f418 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -14,11 +14,6 @@ extern const struct device_type cxl_pmu_type;
 
 extern struct attribute_group cxl_base_attribute_group;
 
-enum cxl_detach_mode {
-	DETACH_ONLY,
-	DETACH_INVALIDATE,
-};
-
 #ifdef CONFIG_CXL_REGION
 extern struct device_attribute dev_attr_create_pmem_region;
 extern struct device_attribute dev_attr_create_ram_region;
diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index 26dfc15e57cd..e3b6d85cd43e 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -2375,6 +2375,7 @@ int cxl_decoder_detach(struct cxl_region *cxlr,
 	}
 	return 0;
 }
+EXPORT_SYMBOL_NS_GPL(cxl_decoder_detach, "CXL");
 
 static int __attach_target(struct cxl_region *cxlr,
 			   struct cxl_endpoint_decoder *cxled, int pos,
@@ -2860,6 +2861,14 @@ cxl_find_region_by_name(struct cxl_root_decoder *cxlrd, const char *name)
 	return to_cxl_region(region_dev);
 }
 
+static void drop_region(struct cxl_region *cxlr)
+{
+	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
+	struct cxl_port *port = cxlrd_to_port(cxlrd);
+
+	devm_release_action(port->uport_dev, unregister_region, cxlr);
+}
+
 static ssize_t delete_region_store(struct device *dev,
 				   struct device_attribute *attr,
 				   const char *buf, size_t len)
@@ -3588,14 +3597,12 @@ static int __construct_region(struct cxl_region *cxlr,
 	return 0;
 }
 
-/* Establish an empty region covering the given HPA range */
-static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
-					   struct cxl_endpoint_decoder *cxled)
+static struct cxl_region *construct_region_begin(struct cxl_root_decoder *cxlrd,
+						 struct cxl_endpoint_decoder *cxled)
 {
 	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
-	struct cxl_port *port = cxlrd_to_port(cxlrd);
 	struct cxl_dev_state *cxlds = cxlmd->cxlds;
-	int rc, part = READ_ONCE(cxled->part);
+	int part = READ_ONCE(cxled->part);
 	struct cxl_region *cxlr;
 
 	do {
@@ -3604,13 +3611,24 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 				       cxled->cxld.target_type);
 	} while (IS_ERR(cxlr) && PTR_ERR(cxlr) == -EBUSY);
 
-	if (IS_ERR(cxlr)) {
+	if (IS_ERR(cxlr))
 		dev_err(cxlmd->dev.parent,
 			"%s:%s: %s failed assign region: %ld\n",
 			dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev),
 			__func__, PTR_ERR(cxlr));
-		return cxlr;
-	}
+
+	return cxlr;
+}
+
+/* Establish an empty region covering the given HPA range */
+static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
+					   struct cxl_endpoint_decoder *cxled)
+{
+	struct cxl_port *port = cxlrd_to_port(cxlrd);
+	struct cxl_region *cxlr;
+	int rc;
+
+	cxlr = construct_region_begin(cxlrd, cxled);
 
 	rc = __construct_region(cxlr, cxlrd, cxled);
 	if (rc) {
@@ -3621,6 +3639,106 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 	return cxlr;
 }
 
+DEFINE_FREE(cxl_region_drop, struct cxl_region *, if (_T) drop_region(_T))
+
+static struct cxl_region *
+__construct_new_region(struct cxl_root_decoder *cxlrd,
+		       struct cxl_endpoint_decoder **cxled, int ways)
+{
+	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
+	struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
+	struct cxl_region_params *p;
+	resource_size_t size = 0;
+	int rc, i;
+
+	struct cxl_region *cxlr __free(cxl_region_drop) =
+		construct_region_begin(cxlrd, cxled[0]);
+	if (IS_ERR(cxlr))
+		return cxlr;
+
+	guard(rwsem_write)(&cxl_rwsem.region);
+
+	/*
+	 * Sanity check. This should not happen with an accel driver handling
+	 * the region creation.
+	 */
+	p = &cxlr->params;
+	if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
+		dev_err(cxlmd->dev.parent,
+			"%s:%s: %s  unexpected region state\n",
+			dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
+			__func__);
+		return ERR_PTR(-EBUSY);
+	}
+
+	rc = set_interleave_ways(cxlr, ways);
+	if (rc)
+		return ERR_PTR(rc);
+
+	rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
+	if (rc)
+		return ERR_PTR(rc);
+
+	scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
+		for (i = 0; i < ways; i++) {
+			if (!cxled[i]->dpa_res)
+				break;
+			size += resource_size(cxled[i]->dpa_res);
+		}
+		if (i < ways)
+			return ERR_PTR(-EINVAL);
+
+		rc = alloc_hpa(cxlr, size);
+		if (rc)
+			return ERR_PTR(rc);
+
+		for (i = 0; i < ways; i++) {
+			rc = cxl_region_attach(cxlr, cxled[i], 0);
+			if (rc)
+				return ERR_PTR(rc);
+		}
+	}
+
+	rc = cxl_region_decode_commit(cxlr);
+	if (rc)
+		return ERR_PTR(rc);
+
+	p->state = CXL_CONFIG_COMMIT;
+
+	return no_free_ptr(cxlr);
+}
+
+/**
+ * cxl_create_region - Establish a region given an endpoint decoder
+ * @cxlrd: root decoder to allocate HPA
+ * @cxled: endpoint decoders with reserved DPA capacity
+ * @ways: interleave ways required
+ *
+ * Returns a fully formed region in the commit state and attached to the
+ * cxl_region driver.
+ */
+struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
+				     struct cxl_endpoint_decoder **cxled,
+				     int ways)
+{
+	struct cxl_region *cxlr;
+
+	mutex_lock(&cxlrd->range_lock);
+	cxlr = __construct_new_region(cxlrd, cxled, ways);
+	mutex_unlock(&cxlrd->range_lock);
+	if (IS_ERR(cxlr))
+		return cxlr;
+
+	if (device_attach(&cxlr->dev) <= 0) {
+		dev_err(&cxlr->dev, "failed to create region\n");
+		drop_region(cxlr);
+		return ERR_PTR(-ENODEV);
+	}
+
+	return cxlr;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_create_region, "CXL");
+
 static struct cxl_region *
 cxl_find_region_by_range(struct cxl_root_decoder *cxlrd, struct range *hpa)
 {
diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c
index 83f5a09839ab..e6c0bd0fc9f9 100644
--- a/drivers/cxl/port.c
+++ b/drivers/cxl/port.c
@@ -35,6 +35,7 @@ static void schedule_detach(void *cxlmd)
 static int discover_region(struct device *dev, void *unused)
 {
 	struct cxl_endpoint_decoder *cxled;
+	struct cxl_memdev *cxlmd;
 	int rc;
 
 	if (!is_endpoint_decoder(dev))
@@ -44,7 +45,9 @@ static int discover_region(struct device *dev, void *unused)
 	if ((cxled->cxld.flags & CXL_DECODER_F_ENABLE) == 0)
 		return 0;
 
-	if (cxled->state != CXL_DECODER_STATE_AUTO)
+	cxlmd = cxled_to_memdev(cxled);
+	if (cxled->state != CXL_DECODER_STATE_AUTO ||
+	    cxlmd->cxlds->type == CXL_DEVTYPE_DEVMEM)
 		return 0;
 
 	/*
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index 1cbe53ad0416..c6fd8fbd36c4 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -275,4 +275,15 @@ struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
 					     enum cxl_partition_mode mode,
 					     resource_size_t alloc);
 int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
+struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
+				     struct cxl_endpoint_decoder **cxled,
+				     int ways);
+enum cxl_detach_mode {
+	DETACH_ONLY,
+	DETACH_INVALIDATE,
+};
+
+int cxl_decoder_detach(struct cxl_region *cxlr,
+		       struct cxl_endpoint_decoder *cxled, int pos,
+		       enum cxl_detach_mode mode);
 #endif /* __CXL_CXL_H__ */
-- 
2.34.1

[PATCH v19 19/22] cxl: Avoid dax creation for accelerators

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

By definition a type2 cxl device will use the host managed memory for
specific functionality, therefore it should not be available to other
uses.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Davidlohr Bueso <daves@stgolabs.net>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
---
 drivers/cxl/core/region.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index e3b6d85cd43e..05f0b77aa229 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -3898,6 +3898,13 @@ static int cxl_region_probe(struct device *dev)
 	if (rc)
 		return rc;
 
+	/*
+	 * HDM-D[B] (device-memory) regions have accelerator specific usage.
+	 * Skip device-dax registration.
+	 */
+	if (cxlr->type == CXL_DECODER_DEVMEM)
+		return 0;
+
 	switch (cxlr->mode) {
 	case CXL_PARTMODE_PMEM:
 		rc = devm_cxl_region_edac_register(cxlr);
-- 
2.34.1

[PATCH v19 20/22] sfc: create cxl region

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

Use cxl api for creating a region using the endpoint decoder related to
a DPA range.

Add a callback for unwinding sfc cxl initialization when the endpoint port
is destroyed by potential cxl_acpi or cxl_mem modules removal.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
---
 drivers/net/ethernet/sfc/efx_cxl.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
index 1a50bb2c0913..79fe99d83f9f 100644
--- a/drivers/net/ethernet/sfc/efx_cxl.c
+++ b/drivers/net/ethernet/sfc/efx_cxl.c
@@ -116,6 +116,14 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 		return PTR_ERR(cxl->cxled);
 	}
 
+	cxl->efx_region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1);
+	if (IS_ERR(cxl->efx_region)) {
+		pci_err(pci_dev, "CXL accel create region failed");
+		cxl_put_root_decoder(cxl->cxlrd);
+		cxl_dpa_free(cxl->cxled);
+		return PTR_ERR(cxl->efx_region);
+	}
+
 	probe_data->cxl = cxl;
 
 	return 0;
@@ -124,6 +132,8 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 void efx_cxl_exit(struct efx_probe_data *probe_data)
 {
 	if (probe_data->cxl) {
+		cxl_decoder_detach(NULL, probe_data->cxl->cxled, 0,
+				   DETACH_INVALIDATE);
 		cxl_dpa_free(probe_data->cxl->cxled);
 		cxl_put_root_decoder(probe_data->cxl->cxlrd);
 	}
-- 
2.34.1

[PATCH v19 21/22] cxl: Add function for obtaining region range

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

A CXL region struct contains the physical address to work with.

Type2 drivers can create a CXL region but have not access to the
related struct as it is defined as private by the kernel CXL core.
Add a function for getting the cxl region range to be used for mapping
such memory range by a Type2 driver.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
Reviewed-by: Zhi Wang <zhiw@nvidia.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/region.c | 23 +++++++++++++++++++++++
 include/cxl/cxl.h         |  2 ++
 2 files changed, 25 insertions(+)

diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index 05f0b77aa229..88c07ecd9e68 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -2754,6 +2754,29 @@ static struct cxl_region *devm_cxl_add_region(struct cxl_root_decoder *cxlrd,
 	return ERR_PTR(rc);
 }
 
+/**
+ * cxl_get_region_range - obtain range linked to a CXL region
+ *
+ * @region: a pointer to struct cxl_region
+ * @range: a pointer to a struct range to be set
+ *
+ * Returns 0 or error.
+ */
+int cxl_get_region_range(struct cxl_region *region, struct range *range)
+{
+	if (WARN_ON_ONCE(!region))
+		return -ENODEV;
+
+	if (!region->params.res)
+		return -ENOSPC;
+
+	range->start = region->params.res->start;
+	range->end = region->params.res->end;
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_region_range, "CXL");
+
 static ssize_t __create_region_show(struct cxl_root_decoder *cxlrd, char *buf)
 {
 	return sysfs_emit(buf, "region%u\n", atomic_read(&cxlrd->region_id));
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index c6fd8fbd36c4..e5d1e5a20e06 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -286,4 +286,6 @@ enum cxl_detach_mode {
 int cxl_decoder_detach(struct cxl_region *cxlr,
 		       struct cxl_endpoint_decoder *cxled, int pos,
 		       enum cxl_detach_mode mode);
+struct range;
+int cxl_get_region_range(struct cxl_region *region, struct range *range);
 #endif /* __CXL_CXL_H__ */
-- 
2.34.1

[PATCH v19 22/22] sfc: support pio mapping based on cxl

[alejandro.lucero-palau]

From: Alejandro Lucero <alucerop@amd.com>

A PIO buffer is a region of device memory to which the driver can write a
packet for TX, with the device handling the transmit doorbell without
requiring a DMA for getting the packet data, which helps reducing latency
in certain exchanges. With CXL mem protocol this latency can be lowered
further.

With a device supporting CXL and successfully initialised, use the cxl
region to map the memory range and use this mapping for PIO buffers.

Add the disabling of those CXL-based PIO buffers if the callback for
potential cxl endpoint removal by the CXL code happens.

Signed-off-by: Alejandro Lucero <alucerop@amd.com>
---
 drivers/net/ethernet/sfc/ef10.c       | 50 +++++++++++++++++++++++----
 drivers/net/ethernet/sfc/efx.h        |  1 -
 drivers/net/ethernet/sfc/efx_cxl.c    | 31 ++++++++++++++---
 drivers/net/ethernet/sfc/net_driver.h |  2 ++
 drivers/net/ethernet/sfc/nic.h        |  3 ++
 5 files changed, 75 insertions(+), 12 deletions(-)

diff --git a/drivers/net/ethernet/sfc/ef10.c b/drivers/net/ethernet/sfc/ef10.c
index 47349c148c0c..1a13fdbbc1b3 100644
--- a/drivers/net/ethernet/sfc/ef10.c
+++ b/drivers/net/ethernet/sfc/ef10.c
@@ -24,6 +24,7 @@
 #include <linux/wait.h>
 #include <linux/workqueue.h>
 #include <net/udp_tunnel.h>
+#include "efx_cxl.h"
 
 /* Hardware control for EF10 architecture including 'Huntington'. */
 
@@ -106,7 +107,7 @@ static int efx_ef10_get_vf_index(struct efx_nic *efx)
 
 static int efx_ef10_init_datapath_caps(struct efx_nic *efx)
 {
-	MCDI_DECLARE_BUF(outbuf, MC_CMD_GET_CAPABILITIES_V4_OUT_LEN);
+	MCDI_DECLARE_BUF(outbuf, MC_CMD_GET_CAPABILITIES_V7_OUT_LEN);
 	struct efx_ef10_nic_data *nic_data = efx->nic_data;
 	size_t outlen;
 	int rc;
@@ -177,6 +178,12 @@ static int efx_ef10_init_datapath_caps(struct efx_nic *efx)
 			  efx->num_mac_stats);
 	}
 
+	if (outlen < MC_CMD_GET_CAPABILITIES_V7_OUT_LEN)
+		nic_data->datapath_caps3 = 0;
+	else
+		nic_data->datapath_caps3 = MCDI_DWORD(outbuf,
+						      GET_CAPABILITIES_V7_OUT_FLAGS3);
+
 	return 0;
 }
 
@@ -919,6 +926,9 @@ static void efx_ef10_forget_old_piobufs(struct efx_nic *efx)
 static void efx_ef10_remove(struct efx_nic *efx)
 {
 	struct efx_ef10_nic_data *nic_data = efx->nic_data;
+#ifdef CONFIG_SFC_CXL
+	struct efx_probe_data *probe_data;
+#endif
 	int rc;
 
 #ifdef CONFIG_SFC_SRIOV
@@ -949,7 +959,12 @@ static void efx_ef10_remove(struct efx_nic *efx)
 
 	efx_mcdi_rx_free_indir_table(efx);
 
+#ifdef CONFIG_SFC_CXL
+	probe_data = container_of(efx, struct efx_probe_data, efx);
+	if (nic_data->wc_membase && !probe_data->cxl_pio_in_use)
+#else
 	if (nic_data->wc_membase)
+#endif
 		iounmap(nic_data->wc_membase);
 
 	rc = efx_mcdi_free_vis(efx);
@@ -1140,6 +1155,9 @@ static int efx_ef10_dimension_resources(struct efx_nic *efx)
 	unsigned int channel_vis, pio_write_vi_base, max_vis;
 	struct efx_ef10_nic_data *nic_data = efx->nic_data;
 	unsigned int uc_mem_map_size, wc_mem_map_size;
+#ifdef CONFIG_SFC_CXL
+	struct efx_probe_data *probe_data;
+#endif
 	void __iomem *membase;
 	int rc;
 
@@ -1263,8 +1281,25 @@ static int efx_ef10_dimension_resources(struct efx_nic *efx)
 	iounmap(efx->membase);
 	efx->membase = membase;
 
-	/* Set up the WC mapping if needed */
-	if (wc_mem_map_size) {
+	if (!wc_mem_map_size)
+		goto skip_pio;
+
+	/* Set up the WC mapping */
+
+#ifdef CONFIG_SFC_CXL
+	probe_data = container_of(efx, struct efx_probe_data, efx);
+	if ((nic_data->datapath_caps3 &
+	    (1 << MC_CMD_GET_CAPABILITIES_V7_OUT_CXL_CONFIG_ENABLE_LBN)) &&
+	    probe_data->cxl_pio_initialised) {
+		/* Using PIO through CXL mapping? */
+		nic_data->pio_write_base = probe_data->cxl->ctpio_cxl +
+					   (pio_write_vi_base * efx->vi_stride +
+					    ER_DZ_TX_PIOBUF - uc_mem_map_size);
+		probe_data->cxl_pio_in_use = true;
+	} else
+#endif
+	{
+		/* Using legacy PIO BAR mapping */
 		nic_data->wc_membase = ioremap_wc(efx->membase_phys +
 						  uc_mem_map_size,
 						  wc_mem_map_size);
@@ -1279,12 +1314,13 @@ static int efx_ef10_dimension_resources(struct efx_nic *efx)
 			nic_data->wc_membase +
 			(pio_write_vi_base * efx->vi_stride + ER_DZ_TX_PIOBUF -
 			 uc_mem_map_size);
-
-		rc = efx_ef10_link_piobufs(efx);
-		if (rc)
-			efx_ef10_free_piobufs(efx);
 	}
 
+	rc = efx_ef10_link_piobufs(efx);
+	if (rc)
+		efx_ef10_free_piobufs(efx);
+
+skip_pio:
 	netif_dbg(efx, probe, efx->net_dev,
 		  "memory BAR at %pa (virtual %p+%x UC, %p+%x WC)\n",
 		  &efx->membase_phys, efx->membase, uc_mem_map_size,
diff --git a/drivers/net/ethernet/sfc/efx.h b/drivers/net/ethernet/sfc/efx.h
index 45e191686625..057d30090894 100644
--- a/drivers/net/ethernet/sfc/efx.h
+++ b/drivers/net/ethernet/sfc/efx.h
@@ -236,5 +236,4 @@ static inline bool efx_rwsem_assert_write_locked(struct rw_semaphore *sem)
 
 int efx_xdp_tx_buffers(struct efx_nic *efx, int n, struct xdp_frame **xdpfs,
 		       bool flush);
-
 #endif /* EFX_EFX_H */
diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
index 79fe99d83f9f..a84ce45398c1 100644
--- a/drivers/net/ethernet/sfc/efx_cxl.c
+++ b/drivers/net/ethernet/sfc/efx_cxl.c
@@ -11,6 +11,7 @@
 #include <cxl/pci.h>
 #include "net_driver.h"
 #include "efx_cxl.h"
+#include "efx.h"
 
 #define EFX_CTPIO_BUFFER_SIZE	SZ_256M
 
@@ -20,6 +21,7 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 	struct pci_dev *pci_dev = efx->pci_dev;
 	resource_size_t max_size;
 	struct efx_cxl *cxl;
+	struct range range;
 	u16 dvsec;
 	int rc;
 
@@ -119,19 +121,40 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
 	cxl->efx_region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1);
 	if (IS_ERR(cxl->efx_region)) {
 		pci_err(pci_dev, "CXL accel create region failed");
-		cxl_put_root_decoder(cxl->cxlrd);
-		cxl_dpa_free(cxl->cxled);
-		return PTR_ERR(cxl->efx_region);
+		rc = PTR_ERR(cxl->efx_region);
+		goto err_dpa;
+	}
+
+	rc = cxl_get_region_range(cxl->efx_region, &range);
+	if (rc) {
+		pci_err(pci_dev, "CXL getting regions params failed");
+		goto err_detach;
+	}
+
+	cxl->ctpio_cxl = ioremap(range.start, range.end - range.start + 1);
+	if (!cxl->ctpio_cxl) {
+		pci_err(pci_dev, "CXL ioremap region (%pra) failed", &range);
+		rc = -ENOMEM;
+		goto err_detach;
 	}
 
 	probe_data->cxl = cxl;
+	probe_data->cxl_pio_initialised = true;
 
 	return 0;
+
+err_detach:
+	cxl_decoder_detach(NULL, cxl->cxled, 0, DETACH_INVALIDATE);
+err_dpa:
+	cxl_put_root_decoder(cxl->cxlrd);
+	cxl_dpa_free(cxl->cxled);
+	return rc;
 }
 
 void efx_cxl_exit(struct efx_probe_data *probe_data)
 {
-	if (probe_data->cxl) {
+	if (probe_data->cxl_pio_initialised) {
+		iounmap(probe_data->cxl->ctpio_cxl);
 		cxl_decoder_detach(NULL, probe_data->cxl->cxled, 0,
 				   DETACH_INVALIDATE);
 		cxl_dpa_free(probe_data->cxl->cxled);
diff --git a/drivers/net/ethernet/sfc/net_driver.h b/drivers/net/ethernet/sfc/net_driver.h
index 0e685b8a9980..894b62d6ada9 100644
--- a/drivers/net/ethernet/sfc/net_driver.h
+++ b/drivers/net/ethernet/sfc/net_driver.h
@@ -1209,6 +1209,7 @@ struct efx_cxl;
  * @efx: Efx NIC details
  * @cxl: details of related cxl objects
  * @cxl_pio_initialised: cxl initialization outcome.
+ * @cxl_pio_in_use: PIO using CXL mapping
  */
 struct efx_probe_data {
 	struct pci_dev *pci_dev;
@@ -1216,6 +1217,7 @@ struct efx_probe_data {
 #ifdef CONFIG_SFC_CXL
 	struct efx_cxl *cxl;
 	bool cxl_pio_initialised;
+	bool cxl_pio_in_use;
 #endif
 };
 
diff --git a/drivers/net/ethernet/sfc/nic.h b/drivers/net/ethernet/sfc/nic.h
index 9fa5c4c713ab..c87cc9214690 100644
--- a/drivers/net/ethernet/sfc/nic.h
+++ b/drivers/net/ethernet/sfc/nic.h
@@ -152,6 +152,8 @@ enum {
  *	%MC_CMD_GET_CAPABILITIES response)
  * @datapath_caps2: Further Capabilities of datapath firmware (FLAGS2 field of
  * %MC_CMD_GET_CAPABILITIES response)
+ * @datapath_caps3: Further Capabilities of datapath firmware (FLAGS3 field of
+ * %MC_CMD_GET_CAPABILITIES response)
  * @rx_dpcpu_fw_id: Firmware ID of the RxDPCPU
  * @tx_dpcpu_fw_id: Firmware ID of the TxDPCPU
  * @must_probe_vswitching: Flag: vswitching has yet to be setup after MC reboot
@@ -186,6 +188,7 @@ struct efx_ef10_nic_data {
 	bool must_check_datapath_caps;
 	u32 datapath_caps;
 	u32 datapath_caps2;
+	u32 datapath_caps3;
 	unsigned int rx_dpcpu_fw_id;
 	unsigned int tx_dpcpu_fw_id;
 	bool must_probe_vswitching;
-- 
2.34.1

Re: [PATCH v19 01/22] cxl/mem: Arrange for always-synchronous memdev attach

[Jonathan Cameron]

On Mon, 6 Oct 2025 11:01:09 +0100
<alejandro.lucero-palau@amd.com> wrote:

> From: Alejandro Lucero <alucerop@amd.com>
> 
> In preparation for CXL accelerator drivers that have a hard dependency on
> CXL capability initialization, arrange for the endpoint probe result to be
> conveyed to the caller of devm_cxl_add_memdev().
> 
> As it stands cxl_pci does not care about the attach state of the cxl_memdev
> because all generic memory expansion functionality can be handled by the
> cxl_core. For accelerators, that driver needs to know perform driver
> specific initialization if CXL is available, or exectute a fallback to PCIe
> only operation.
> 
> By moving devm_cxl_add_memdev() to cxl_mem.ko it removes async module
> loading as one reason that a memdev may not be attached upon return from
> devm_cxl_add_memdev().
> 
> The diff is busy as this moves cxl_memdev_alloc() down below the definition
> of cxl_memdev_fops and introduces devm_cxl_memdev_add_or_reset() to
> preclude needing to export more symbols from the cxl_core.
> 
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Alejandro, SoB chain broken here which makes this currently unmergeable.

Should definitely have your SoB as you sent the patch to the list and need
to make a statement that you believe it to be fine to do so (see the Certificate
of origin stuff in the docs).  Also, From should always be one of the authors.
If Dan wrote this as the SoB suggests then From should be set to him..

git commit --amend --author="Dan Williams <dan.j.williams@intel.com>"

Will fix that up.  Then either you add your SoB on basis you just 'handled'
the patch but didn't make substantial changes, or your SoB and a Codeveloped-by
if you did make major changes.  If it is minor stuff you can an
a sign off with # what changed 
comment next to it.

A few minor comments inline.

Thanks,

Jonathan


> ---
>  drivers/cxl/Kconfig       |  2 +-
>  drivers/cxl/core/memdev.c | 97 ++++++++++++++++-----------------------
>  drivers/cxl/mem.c         | 30 ++++++++++++
>  drivers/cxl/private.h     | 11 +++++
>  4 files changed, 82 insertions(+), 58 deletions(-)
>  create mode 100644 drivers/cxl/private.h
> 
> diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
> index 028201e24523..111e05615f09 100644
> --- a/drivers/cxl/Kconfig
> +++ b/drivers/cxl/Kconfig
> @@ -22,6 +22,7 @@ if CXL_BUS
>  config CXL_PCI
>  	tristate "PCI manageability"
>  	default CXL_BUS
> +	select CXL_MEM
>  	help
>  	  The CXL specification defines a "CXL memory device" sub-class in the
>  	  PCI "memory controller" base class of devices. Device's identified by
> @@ -89,7 +90,6 @@ config CXL_PMEM
>  
>  config CXL_MEM
>  	tristate "CXL: Memory Expansion"
> -	depends on CXL_PCI
>  	default CXL_BUS
>  	help
>  	  The CXL.mem protocol allows a device to act as a provider of "System
> diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
> index c569e00a511f..2bef231008df 100644
> --- a/drivers/cxl/core/memdev.c
> +++ b/drivers/cxl/core/memdev.c

> -
> -err:
> -	kfree(cxlmd);
> -	return ERR_PTR(rc);
>  }
> +EXPORT_SYMBOL_NS_GPL(devm_cxl_memdev_add_or_reset, "CXL");
>  
>  static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
>  			       unsigned long arg)
> @@ -1023,50 +1012,44 @@ static const struct file_operations cxl_memdev_fops = {
>  	.llseek = noop_llseek,
>  };
>  
> -struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
> -				       struct cxl_dev_state *cxlds)
> +struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds)
>  {
>  	struct cxl_memdev *cxlmd;
>  	struct device *dev;
>  	struct cdev *cdev;
>  	int rc;
>  
> -	cxlmd = cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
> -	if (IS_ERR(cxlmd))
> -		return cxlmd;
> +	cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);

It's a little bit non obvious due to the device initialize mid way
through this, but given there are no error paths after that you can
currently just do.
	struct cxl_memdev *cxlmd __free(kfree) =
		cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
and
	return_ptr(cxlmd);

in the good path.  That lets you then just return rather than having
the goto err: handling for the error case that currently frees this
manually.

Unlike the change below, this one I think is definitely worth making.


> +	if (!cxlmd)
> +		return ERR_PTR(-ENOMEM);
>  
> -	dev = &cxlmd->dev;
> -	rc = dev_set_name(dev, "mem%d", cxlmd->id);
> -	if (rc)
> +	rc = ida_alloc_max(&cxl_memdev_ida, CXL_MEM_MAX_DEVS - 1, GFP_KERNEL);
> +	if (rc < 0)
>  		goto err;
> -
> -	/*
> -	 * Activate ioctl operations, no cxl_memdev_rwsem manipulation
> -	 * needed as this is ordered with cdev_add() publishing the device.
> -	 */
> +	cxlmd->id = rc;
> +	cxlmd->depth = -1;
>  	cxlmd->cxlds = cxlds;
>  	cxlds->cxlmd = cxlmd;
>  
> -	cdev = &cxlmd->cdev;
> -	rc = cdev_device_add(cdev, dev);
> -	if (rc)
> -		goto err;
> +	dev = &cxlmd->dev;
> +	device_initialize(dev);
> +	lockdep_set_class(&dev->mutex, &cxl_memdev_key);
> +	dev->parent = cxlds->dev;
> +	dev->bus = &cxl_bus_type;
> +	dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
> +	dev->type = &cxl_memdev_type;
> +	device_set_pm_not_required(dev);
> +	INIT_WORK(&cxlmd->detach_work, detach_memdev);
>  
> -	rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
> -	if (rc)
> -		return ERR_PTR(rc);
> +	cdev = &cxlmd->cdev;
> +	cdev_init(cdev, &cxl_memdev_fops);
>  	return cxlmd;
>  
>  err:
> -	/*
> -	 * The cdev was briefly live, shutdown any ioctl operations that
> -	 * saw that state.
> -	 */
> -	cxl_memdev_shutdown(dev);
> -	put_device(dev);
> +	kfree(cxlmd);
>  	return ERR_PTR(rc);
>  }
> -EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
> +EXPORT_SYMBOL_NS_GPL(cxl_memdev_alloc, "CXL");
>  
>  static void sanitize_teardown_notifier(void *data)
>  {
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index f7dc0ba8905d..144749b9c818 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -7,6 +7,7 @@
>  
>  #include "cxlmem.h"
>  #include "cxlpci.h"
> +#include "private.h"
>  #include "core/core.h"
>  
>  /**
> @@ -203,6 +204,34 @@ static int cxl_mem_probe(struct device *dev)
>  	return devm_add_action_or_reset(dev, enable_suspend, NULL);
>  }
>  
> +/**
> + * devm_cxl_add_memdev - Add a CXL memory device
> + * @host: devres alloc/release context and parent for the memdev
> + * @cxlds: CXL device state to associate with the memdev
> + *
> + * Upon return the device will have had a chance to attach to the
> + * cxl_mem driver, but may fail if the CXL topology is not ready
> + * (hardware CXL link down, or software platform CXL root not attached)
> + */
> +struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
> +				       struct cxl_dev_state *cxlds)
> +{
> +	struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds);

Bit marginal but you could do a DEFINE_FREE() for cxlmd 
similar to the one that exists for put_cxl_port

You would then need to steal the pointer for the devm_ call at the
end of this function.


> +	int rc;
> +
> +	if (IS_ERR(cxlmd))
> +		return cxlmd;
> +
> +	rc = dev_set_name(&cxlmd->dev, "mem%d", cxlmd->id);
> +	if (rc) {
> +		put_device(&cxlmd->dev);
> +		return ERR_PTR(rc);
> +	}
> +
> +	return devm_cxl_memdev_add_or_reset(host, cxlmd);
> +}
> +EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");

Re: [PATCH v19 01/22] cxl/mem: Arrange for always-synchronous memdev attach

[Jonathan Cameron]

On Tue, 7 Oct 2025 13:40:53 +0100
Jonathan Cameron <jonathan.cameron@huawei.com> wrote:

> On Mon, 6 Oct 2025 11:01:09 +0100
> <alejandro.lucero-palau@amd.com> wrote:
> 
> > From: Alejandro Lucero <alucerop@amd.com>
> > 
> > In preparation for CXL accelerator drivers that have a hard dependency on
> > CXL capability initialization, arrange for the endpoint probe result to be
> > conveyed to the caller of devm_cxl_add_memdev().
> > 
> > As it stands cxl_pci does not care about the attach state of the cxl_memdev
> > because all generic memory expansion functionality can be handled by the
> > cxl_core. For accelerators, that driver needs to know perform driver
> > specific initialization if CXL is available, or exectute a fallback to PCIe
> > only operation.
> > 
> > By moving devm_cxl_add_memdev() to cxl_mem.ko it removes async module
> > loading as one reason that a memdev may not be attached upon return from
> > devm_cxl_add_memdev().
> > 
> > The diff is busy as this moves cxl_memdev_alloc() down below the definition
> > of cxl_memdev_fops and introduces devm_cxl_memdev_add_or_reset() to
> > preclude needing to export more symbols from the cxl_core.
> > 
> > Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> 
> Alejandro, SoB chain broken here which makes this currently unmergeable.
> 
> Should definitely have your SoB as you sent the patch to the list and need
> to make a statement that you believe it to be fine to do so (see the Certificate
> of origin stuff in the docs).  Also, From should always be one of the authors.
> If Dan wrote this as the SoB suggests then From should be set to him..
> 
> git commit --amend --author="Dan Williams <dan.j.williams@intel.com>"
> 
> Will fix that up.  Then either you add your SoB on basis you just 'handled'
> the patch but didn't make substantial changes, or your SoB and a Codeveloped-by
> if you did make major changes.  If it is minor stuff you can an
> a sign off with # what changed 
> comment next to it.
> 
> A few minor comments inline.

oops. I see this was part of Dan's series that has merged by now...

Never mind unless you want to take any of the suggested tweaks forwards.

J
> 
> Thanks,
> 
> Jonathan
> 
> 
> > ---
> >  drivers/cxl/Kconfig       |  2 +-
> >  drivers/cxl/core/memdev.c | 97 ++++++++++++++++-----------------------
> >  drivers/cxl/mem.c         | 30 ++++++++++++
> >  drivers/cxl/private.h     | 11 +++++
> >  4 files changed, 82 insertions(+), 58 deletions(-)
> >  create mode 100644 drivers/cxl/private.h
> > 
> > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
> > index 028201e24523..111e05615f09 100644
> > --- a/drivers/cxl/Kconfig
> > +++ b/drivers/cxl/Kconfig
> > @@ -22,6 +22,7 @@ if CXL_BUS
> >  config CXL_PCI
> >  	tristate "PCI manageability"
> >  	default CXL_BUS
> > +	select CXL_MEM
> >  	help
> >  	  The CXL specification defines a "CXL memory device" sub-class in the
> >  	  PCI "memory controller" base class of devices. Device's identified by
> > @@ -89,7 +90,6 @@ config CXL_PMEM
> >  
> >  config CXL_MEM
> >  	tristate "CXL: Memory Expansion"
> > -	depends on CXL_PCI
> >  	default CXL_BUS
> >  	help
> >  	  The CXL.mem protocol allows a device to act as a provider of "System
> > diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
> > index c569e00a511f..2bef231008df 100644
> > --- a/drivers/cxl/core/memdev.c
> > +++ b/drivers/cxl/core/memdev.c
> 
> > -
> > -err:
> > -	kfree(cxlmd);
> > -	return ERR_PTR(rc);
> >  }
> > +EXPORT_SYMBOL_NS_GPL(devm_cxl_memdev_add_or_reset, "CXL");
> >  
> >  static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
> >  			       unsigned long arg)
> > @@ -1023,50 +1012,44 @@ static const struct file_operations cxl_memdev_fops = {
> >  	.llseek = noop_llseek,
> >  };
> >  
> > -struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
> > -				       struct cxl_dev_state *cxlds)
> > +struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds)
> >  {
> >  	struct cxl_memdev *cxlmd;
> >  	struct device *dev;
> >  	struct cdev *cdev;
> >  	int rc;
> >  
> > -	cxlmd = cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
> > -	if (IS_ERR(cxlmd))
> > -		return cxlmd;
> > +	cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
> 
> It's a little bit non obvious due to the device initialize mid way
> through this, but given there are no error paths after that you can
> currently just do.
> 	struct cxl_memdev *cxlmd __free(kfree) =
> 		cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
> and
> 	return_ptr(cxlmd);
> 
> in the good path.  That lets you then just return rather than having
> the goto err: handling for the error case that currently frees this
> manually.
> 
> Unlike the change below, this one I think is definitely worth making.
> 
> 
> > +	if (!cxlmd)
> > +		return ERR_PTR(-ENOMEM);
> >  
> > -	dev = &cxlmd->dev;
> > -	rc = dev_set_name(dev, "mem%d", cxlmd->id);
> > -	if (rc)
> > +	rc = ida_alloc_max(&cxl_memdev_ida, CXL_MEM_MAX_DEVS - 1, GFP_KERNEL);
> > +	if (rc < 0)
> >  		goto err;
> > -
> > -	/*
> > -	 * Activate ioctl operations, no cxl_memdev_rwsem manipulation
> > -	 * needed as this is ordered with cdev_add() publishing the device.
> > -	 */
> > +	cxlmd->id = rc;
> > +	cxlmd->depth = -1;
> >  	cxlmd->cxlds = cxlds;
> >  	cxlds->cxlmd = cxlmd;
> >  
> > -	cdev = &cxlmd->cdev;
> > -	rc = cdev_device_add(cdev, dev);
> > -	if (rc)
> > -		goto err;
> > +	dev = &cxlmd->dev;
> > +	device_initialize(dev);
> > +	lockdep_set_class(&dev->mutex, &cxl_memdev_key);
> > +	dev->parent = cxlds->dev;
> > +	dev->bus = &cxl_bus_type;
> > +	dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
> > +	dev->type = &cxl_memdev_type;
> > +	device_set_pm_not_required(dev);
> > +	INIT_WORK(&cxlmd->detach_work, detach_memdev);
> >  
> > -	rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
> > -	if (rc)
> > -		return ERR_PTR(rc);
> > +	cdev = &cxlmd->cdev;
> > +	cdev_init(cdev, &cxl_memdev_fops);
> >  	return cxlmd;
> >  
> >  err:
> > -	/*
> > -	 * The cdev was briefly live, shutdown any ioctl operations that
> > -	 * saw that state.
> > -	 */
> > -	cxl_memdev_shutdown(dev);
> > -	put_device(dev);
> > +	kfree(cxlmd);
> >  	return ERR_PTR(rc);
> >  }
> > -EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
> > +EXPORT_SYMBOL_NS_GPL(cxl_memdev_alloc, "CXL");
> >  
> >  static void sanitize_teardown_notifier(void *data)
> >  {
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index f7dc0ba8905d..144749b9c818 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -7,6 +7,7 @@
> >  
> >  #include "cxlmem.h"
> >  #include "cxlpci.h"
> > +#include "private.h"
> >  #include "core/core.h"
> >  
> >  /**
> > @@ -203,6 +204,34 @@ static int cxl_mem_probe(struct device *dev)
> >  	return devm_add_action_or_reset(dev, enable_suspend, NULL);
> >  }
> >  
> > +/**
> > + * devm_cxl_add_memdev - Add a CXL memory device
> > + * @host: devres alloc/release context and parent for the memdev
> > + * @cxlds: CXL device state to associate with the memdev
> > + *
> > + * Upon return the device will have had a chance to attach to the
> > + * cxl_mem driver, but may fail if the CXL topology is not ready
> > + * (hardware CXL link down, or software platform CXL root not attached)
> > + */
> > +struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
> > +				       struct cxl_dev_state *cxlds)
> > +{
> > +	struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds);
> 
> Bit marginal but you could do a DEFINE_FREE() for cxlmd 
> similar to the one that exists for put_cxl_port
> 
> You would then need to steal the pointer for the devm_ call at the
> end of this function.
> 
> 
> > +	int rc;
> > +
> > +	if (IS_ERR(cxlmd))
> > +		return cxlmd;
> > +
> > +	rc = dev_set_name(&cxlmd->dev, "mem%d", cxlmd->id);
> > +	if (rc) {
> > +		put_device(&cxlmd->dev);
> > +		return ERR_PTR(rc);
> > +	}
> > +
> > +	return devm_cxl_memdev_add_or_reset(host, cxlmd);
> > +}
> > +EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");

Re: [PATCH v19 06/22] cxl: Move pci generic code

[Jonathan Cameron]

On Mon, 6 Oct 2025 11:01:14 +0100
alejandro.lucero-palau@amd.com wrote:

> From: Alejandro Lucero <alucerop@amd.com>
> 
> Inside cxl/core/pci.c there are helpers for CXL PCIe initialization
> meanwhile cxl/pci.c implements the functionality for a Type3 device
> initialization.
> 
> Move helper functions from cxl/pci.c to cxl/core/pci.c in order to be
> exported and shared with CXL Type2 device initialization.
> 
> Fix cxl mock tests affected by the code move, deleting a function which
> indeed was not being used since commit 733b57f262b0("cxl/pci: Early
> setup RCH dport component registers from RCRB").
> 

Trivial but can we pull out that code removal as a separate patch?
It's something Dave would probably pick up immediately. 

> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
> Reviewed-by: Fan Ni <fan.ni@samsung.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Reviewed-by: Alison Schofield <alison.schofield@intel.com>
> Reviewed-by: Dan Williams <dan.j.williams@intel.com>


> diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h
> index ccf0ca36bc00..4b11757a46ab 100644
> --- a/drivers/cxl/cxlpci.h
> +++ b/drivers/cxl/cxlpci.h
> @@ -74,9 +74,22 @@ static inline bool cxl_pci_flit_256(struct pci_dev *pdev)
>  	return lnksta2 & PCI_EXP_LNKSTA2_FLIT;
>  }
>  
> +/*
> + * Assume that the caller has already validated that @pdev has CXL
> + * capabilities, any RCIEp with CXL capabilities is treated as a

In PCI spec they are RCiEP so we should match that rather than lowercase
for the P.

> + * Restricted CXL Device (RCD) and finds upstream port and endpoint
> + * registers in a Root Complex Register Block (RCRB).
> + */
> +static inline bool is_cxl_restricted(struct pci_dev *pdev)
> +{
> +	return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END;
> +}

Re: [PATCH v19 07/22] cxl: allow Type2 drivers to map cxl component regs

[Jonathan Cameron]

On Mon, 6 Oct 2025 11:01:15 +0100
alejandro.lucero-palau@amd.com wrote:

> From: Alejandro Lucero <alucerop@amd.com>
> 
I'd amend the patch title to 
	cxl/sfc: Map CXL component regs.

And talk about exports in the description.

Other options are fine but the patch title should indicate
this is being used by the sfc driver.

> Export cxl core functions for a Type2 driver being able to discover and
> map the device component registers.
> 
> Use it in sfc driver cxl initialization.
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Dan Williams <dan.j.williams@intel.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
> ---
>  drivers/cxl/core/pci.c             |  1 +
>  drivers/cxl/core/port.c            |  1 +
>  drivers/cxl/core/regs.c            |  1 +
>  drivers/cxl/cxl.h                  |  7 ------
>  drivers/cxl/cxlpci.h               | 12 ----------
>  drivers/cxl/pci.c                  |  1 +
>  drivers/net/ethernet/sfc/efx_cxl.c | 35 ++++++++++++++++++++++++++++++
>  include/cxl/cxl.h                  | 19 ++++++++++++++++
>  include/cxl/pci.h                  | 21 ++++++++++++++++++
>  9 files changed, 79 insertions(+), 19 deletions(-)
>  create mode 100644 include/cxl/pci.h

Re: [PATCH v19 08/22] cxl: Support dpa initialization without a mailbox

[Jonathan Cameron]

On Mon, 6 Oct 2025 11:01:16 +0100
<alejandro.lucero-palau@amd.com> wrote:

> From: Alejandro Lucero <alucerop@amd.com>

I think this also needs to mention sfc in the patch title as we'll
need an appropriate RB/Ack for that part.

Lazy option would be
cxl/sfc: Support dpa initialization without a mailbox

but there are probably better options.

> 
> Type3 relies on mailbox CXL_MBOX_OP_IDENTIFY command for initializing
> memdev state params which end up being used for DPA initialization.
> 
> Allow a Type2 driver to initialize DPA simply by giving the size of its
> volatile hardware partition.
> 
> Move related functions to memdev.
> 
> Add sfc driver as the client.
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Dan Williams <dan.j.williams@intel.com>
> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>

Re: [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration

[Jonathan Cameron]

On Mon, 6 Oct 2025 11:01:19 +0100
alejandro.lucero-palau@amd.com wrote:

> From: Alejandro Lucero <alucerop@amd.com>
> 
> CXL region creation involves allocating capacity from Device Physical Address
> (DPA) and assigning it to decode a given Host Physical Address (HPA). Before
> determining how much DPA to allocate the amount of available HPA must be
> determined. Also, not all HPA is created equal, some HPA targets RAM, some
> targets PMEM, some is prepared for device-memory flows like HDM-D and HDM-DB,
> and some is HDM-H (host-only).
> 
> In order to support Type2 CXL devices, wrap all of those concerns into
> an API that retrieves a root decoder (platform CXL window) that fits the
> specified constraints and the capacity available for a new region.
> 
> Add a complementary function for releasing the reference to such root
> decoder.
> 
> Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
One thing I noticed on a fresh read through...

> ---
>  drivers/cxl/core/region.c | 162 ++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxl.h         |   3 +
>  include/cxl/cxl.h         |   6 ++
>  3 files changed, 171 insertions(+)
> 
> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> index e9bf42d91689..c5b66204ecde 100644
> --- a/drivers/cxl/core/region.c
> +++ b/drivers/cxl/core/region.c
> @@ -703,6 +703,168 @@ static int free_hpa(struct cxl_region *cxlr)
>  	return 0;
>  }
>  
> +struct cxlrd_max_context {
> +	struct device * const *host_bridges;
> +	int interleave_ways;
> +	unsigned long flags;
> +	resource_size_t max_hpa;
> +	struct cxl_root_decoder *cxlrd;
> +};
> +
> +static int find_max_hpa(struct device *dev, void *data)
> +{
> +	struct cxlrd_max_context *ctx = data;
> +	struct cxl_switch_decoder *cxlsd;
> +	struct cxl_root_decoder *cxlrd;
> +	struct resource *res, *prev;
> +	struct cxl_decoder *cxld;
> +	resource_size_t max;
> +	int found = 0;
> +
> +	if (!is_root_decoder(dev))
> +		return 0;
> +
> +	cxlrd = to_cxl_root_decoder(dev);
> +	cxlsd = &cxlrd->cxlsd;
> +	cxld = &cxlsd->cxld;
> +
> +	if ((cxld->flags & ctx->flags) != ctx->flags) {
> +		dev_dbg(dev, "flags not matching: %08lx vs %08lx\n",
> +			cxld->flags, ctx->flags);
> +		return 0;
> +	}
> +
> +	for (int i = 0; i < ctx->interleave_ways; i++) {
I think ctx->interleave_ways == 0 as it's never set, so found is never
set, but then the check below succeeds as found == 0 and ctx->interleave_ways == 0

Definitely doesn't feel intentional!

> +		for (int j = 0; j < ctx->interleave_ways; j++) {
> +			if (ctx->host_bridges[i] == cxlsd->target[j]->dport_dev) {
> +				found++;
> +				break;
> +			}
> +		}
> +	}
> +
> +	if (found != ctx->interleave_ways) {
> +		dev_dbg(dev,
> +			"Not enough host bridges. Found %d for %d interleave ways requested\n",
> +			found, ctx->interleave_ways);
> +		return 0;
> +	}

> +}
> +
> +/**
> + * cxl_get_hpa_freespace - find a root decoder with free capacity per constraints
> + * @cxlmd: the mem device requiring the HPA
> + * @interleave_ways: number of entries in @host_bridges
> + * @flags: CXL_DECODER_F flags for selecting RAM vs PMEM, and Type2 device
> + * @max_avail_contig: output parameter of max contiguous bytes available in the
> + *		      returned decoder
> + *
> + * Returns a pointer to a struct cxl_root_decoder
> + *
> + * The return tuple of a 'struct cxl_root_decoder' and 'bytes available given
> + * in (@max_avail_contig))' is a point in time snapshot. If by the time the
> + * caller goes to use this decoder and its capacity is reduced then caller needs
> + * to loop and retry.
> + *
> + * The returned root decoder has an elevated reference count that needs to be
> + * put with cxl_put_root_decoder(cxlrd).
> + */
> +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
> +					       int interleave_ways,

Currently unused.  I think you mean to set the field in ctx


> +					       unsigned long flags,
> +					       resource_size_t *max_avail_contig)
> +{
> +	struct cxl_port *endpoint = cxlmd->endpoint;
> +	struct cxlrd_max_context ctx = {
> +		.flags = flags,
> +	};
> +	struct cxl_port *root_port;
> +
> +	if (!endpoint) {
> +		dev_dbg(&cxlmd->dev, "endpoint not linked to memdev\n");
> +		return ERR_PTR(-ENXIO);
> +	}
> +
> +	ctx.host_bridges = &endpoint->host_bridge;
> +
> +	struct cxl_root *root __free(put_cxl_root) = find_cxl_root(endpoint);
> +	if (!root) {
> +		dev_dbg(&endpoint->dev, "endpoint is not related to a root port\n");
> +		return ERR_PTR(-ENXIO);
> +	}
> +
> +	root_port = &root->port;
> +	scoped_guard(rwsem_read, &cxl_rwsem.region)
> +		device_for_each_child(&root_port->dev, &ctx, find_max_hpa);
> +
> +	if (!ctx.cxlrd)
> +		return ERR_PTR(-ENOMEM);
> +
> +	*max_avail_contig = ctx.max_hpa;
> +	return ctx.cxlrd;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_get_hpa_freespace, "CXL");

Re: [PATCH v19 13/22] cxl: Define a driver interface for DPA allocation

[Jonathan Cameron]

On Mon, 6 Oct 2025 11:01:21 +0100
<alejandro.lucero-palau@amd.com> wrote:

> From: Alejandro Lucero <alucerop@amd.com>
> 
> Region creation involves finding available DPA (device-physical-address)
> capacity to map into HPA (host-physical-address) space.
> 
> In order to support CXL Type2 devices, define an API, cxl_request_dpa(),
> that tries to allocate the DPA memory the driver requires to operate.The
> memory requested should not be bigger than the max available HPA obtained
> previously with cxl_get_hpa_freespace().
> 
> Based on https://lore.kernel.org/linux-cxl/168592158743.1948938.7622563891193802610.stgit@dwillia2-xfh.jf.intel.com/
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>

A few minor things inline.  Depending on how much this changed
from the 'Based on' it might be appropriate to keep a SoB / author
set to Dan, but I'll let him request that if he feels appropriate
(or you can make that decision if Dan is busy).

A few things inline.  All trivial

Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>

>  int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>  		     enum cxl_partition_mode mode)
> @@ -613,6 +622,82 @@ int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>  	return 0;
>  }
>  
> +static int find_free_decoder(struct device *dev, const void *data)
> +{
> +	struct cxl_endpoint_decoder *cxled;
> +	struct cxl_port *port;
> +
> +	if (!is_endpoint_decoder(dev))
> +		return 0;
> +
> +	cxled = to_cxl_endpoint_decoder(dev);
> +	port = cxled_to_port(cxled);
> +
> +	return cxled->cxld.id == (port->hdm_end + 1);
> +}
> +
> +static struct cxl_endpoint_decoder *
> +cxl_find_free_decoder(struct cxl_memdev *cxlmd)
> +{
> +	struct cxl_port *endpoint = cxlmd->endpoint;
> +	struct device *dev;
> +
> +	guard(rwsem_read)(&cxl_rwsem.dpa);
> +	dev = device_find_child(&endpoint->dev, NULL,
> +				find_free_decoder);
> +	if (dev)
> +		return to_cxl_endpoint_decoder(dev);
> +
> +	return NULL;
Trivial but I'd prefer to see the 'error' like thing out of line

	if (!dev)
		return NULL;

	return to_cxl_endpoint_decoder(dev);

> +}
> +
> +/**
> + * cxl_request_dpa - search and reserve DPA given input constraints
> + * @cxlmd: memdev with an endpoint port with available decoders
> + * @mode: CXL partition mode (ram vs pmem)
> + * @alloc: dpa size required
> + *
> + * Returns a pointer to a 'struct cxl_endpoint_decoder' on success or
> + * an errno encoded pointer on failure.
> + *
> + * Given that a region needs to allocate from limited HPA capacity it
> + * may be the case that a device has more mappable DPA capacity than
> + * available HPA. The expectation is that @alloc is a driver known
> + * value based on the device capacity but which could not be fully
> + * available due to HPA constraints.
> + *
> + * Returns a pinned cxl_decoder with at least @alloc bytes of capacity
> + * reserved, or an error pointer. The caller is also expected to own the
> + * lifetime of the memdev registration associated with the endpoint to
> + * pin the decoder registered as well.
> + */
> +struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
> +					     enum cxl_partition_mode mode,
> +					     resource_size_t alloc)
> +{
> +	int rc;
> +
> +	if (!IS_ALIGNED(alloc, SZ_256M))
> +		return ERR_PTR(-EINVAL);
> +
> +	struct cxl_endpoint_decoder *cxled __free(put_cxled) =
> +		cxl_find_free_decoder(cxlmd);
> +
> +	if (!cxled)
> +		return ERR_PTR(-ENODEV);
> +
> +	rc = cxl_dpa_set_part(cxled, mode);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	rc = cxl_dpa_alloc(cxled, alloc);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	return no_free_ptr(cxled);

return_ptr() (it's exactly the same implementation).

> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_request_dpa, "CXL");

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Jonathan Cameron]

On Mon, 6 Oct 2025 11:01:26 +0100
<alejandro.lucero-palau@amd.com> wrote:

> From: Alejandro Lucero <alucerop@amd.com>
> 
> Creating a CXL region requires userspace intervention through the cxl
> sysfs files. Type2 support should allow accelerator drivers to create
> such cxl region from kernel code.
> 
> Adding that functionality and integrating it with current support for
> memory expanders.
> 
> Support an action by the type2 driver to be linked to the created region
> for unwinding the resources allocated properly.
> 
> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

One new question below (and a trivial thing beyond that).

If you adopt one of the suggested ways of tidying that up, then keep the RB
if not I'll want to take another look so drop it.

>  #ifdef CONFIG_CXL_REGION
>  extern struct device_attribute dev_attr_create_pmem_region;
>  extern struct device_attribute dev_attr_create_ram_region;
> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> index 26dfc15e57cd..e3b6d85cd43e 100644
> --- a/drivers/cxl/core/region.c
> +++ b/drivers/cxl/core/region.c

> +
> +/* Establish an empty region covering the given HPA range */
> +static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
> +					   struct cxl_endpoint_decoder *cxled)
> +{
> +	struct cxl_port *port = cxlrd_to_port(cxlrd);
> +	struct cxl_region *cxlr;
> +	int rc;
> +
> +	cxlr = construct_region_begin(cxlrd, cxled);
>  
>  	rc = __construct_region(cxlr, cxlrd, cxled);
>  	if (rc) {
> @@ -3621,6 +3639,106 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
>  	return cxlr;
>  }
>  
> +DEFINE_FREE(cxl_region_drop, struct cxl_region *, if (_T) drop_region(_T))
> +
> +static struct cxl_region *
> +__construct_new_region(struct cxl_root_decoder *cxlrd,
> +		       struct cxl_endpoint_decoder **cxled, int ways)

Why pass in an array of struct cxl_endpoint_decoder * if this is only
ever going to use the first element?

I think we need to indicate that somehow.  Could just pass in the
relevant decoder (I assume there is only one?)  Or pass the array
and an index (here 0).


> +{
> +	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
> +	struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
> +	struct cxl_region_params *p;
> +	resource_size_t size = 0;
> +	int rc, i;
> +
> +	struct cxl_region *cxlr __free(cxl_region_drop) =
> +		construct_region_begin(cxlrd, cxled[0]);
> +	if (IS_ERR(cxlr))
> +		return cxlr;
> +
> +	guard(rwsem_write)(&cxl_rwsem.region);
> +
> +	/*
> +	 * Sanity check. This should not happen with an accel driver handling
> +	 * the region creation.
> +	 */
> +	p = &cxlr->params;
> +	if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
> +		dev_err(cxlmd->dev.parent,
> +			"%s:%s: %s  unexpected region state\n",
> +			dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
> +			__func__);
> +		return ERR_PTR(-EBUSY);
> +	}
> +
> +	rc = set_interleave_ways(cxlr, ways);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
> +		for (i = 0; i < ways; i++) {
> +			if (!cxled[i]->dpa_res)
> +				break;
> +			size += resource_size(cxled[i]->dpa_res);
> +		}
> +		if (i < ways)
> +			return ERR_PTR(-EINVAL);
> +
> +		rc = alloc_hpa(cxlr, size);
> +		if (rc)
> +			return ERR_PTR(rc);
> +
> +		for (i = 0; i < ways; i++) {
> +			rc = cxl_region_attach(cxlr, cxled[i], 0);
> +			if (rc)
> +				return ERR_PTR(rc);
> +		}
> +	}
> +
> +	rc = cxl_region_decode_commit(cxlr);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	p->state = CXL_CONFIG_COMMIT;
> +
> +	return no_free_ptr(cxlr);

return_ptr() 
> +}

Re: [PATCH v19 20/22] sfc: create cxl region

[Jonathan Cameron]

On Mon, 6 Oct 2025 11:01:28 +0100
<alejandro.lucero-palau@amd.com> wrote:

> From: Alejandro Lucero <alucerop@amd.com>
> 
> Use cxl api for creating a region using the endpoint decoder related to
> a DPA range.
> 
> Add a callback for unwinding sfc cxl initialization when the endpoint port
> is destroyed by potential cxl_acpi or cxl_mem modules removal.
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> ---
>  drivers/net/ethernet/sfc/efx_cxl.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
> index 1a50bb2c0913..79fe99d83f9f 100644
> --- a/drivers/net/ethernet/sfc/efx_cxl.c
> +++ b/drivers/net/ethernet/sfc/efx_cxl.c
> @@ -116,6 +116,14 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>  		return PTR_ERR(cxl->cxled);
>  	}
>  
> +	cxl->efx_region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1);

This follows from earlier comment on patch 18.  I think cxl->cxled is the right
thing to pass in here rather than pretending it's an array by passing
in a pointer to that.

> +	if (IS_ERR(cxl->efx_region)) {
> +		pci_err(pci_dev, "CXL accel create region failed");
> +		cxl_put_root_decoder(cxl->cxlrd);
> +		cxl_dpa_free(cxl->cxled);
> +		return PTR_ERR(cxl->efx_region);
> +	}
> +
>  	probe_data->cxl = cxl;
>  
>  	return 0;
> @@ -124,6 +132,8 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>  void efx_cxl_exit(struct efx_probe_data *probe_data)
>  {
>  	if (probe_data->cxl) {
> +		cxl_decoder_detach(NULL, probe_data->cxl->cxled, 0,
> +				   DETACH_INVALIDATE);
>  		cxl_dpa_free(probe_data->cxl->cxled);
>  		cxl_put_root_decoder(probe_data->cxl->cxlrd);
>  	}

Re: [PATCH v19 22/22] sfc: support pio mapping based on cxl

[Jonathan Cameron]

On Mon, 6 Oct 2025 11:01:30 +0100
<alejandro.lucero-palau@amd.com> wrote:

> From: Alejandro Lucero <alucerop@amd.com>
> 
> A PIO buffer is a region of device memory to which the driver can write a
> packet for TX, with the device handling the transmit doorbell without
> requiring a DMA for getting the packet data, which helps reducing latency
> in certain exchanges. With CXL mem protocol this latency can be lowered
> further.
> 
> With a device supporting CXL and successfully initialised, use the cxl
> region to map the memory range and use this mapping for PIO buffers.
> 
> Add the disabling of those CXL-based PIO buffers if the callback for
> potential cxl endpoint removal by the CXL code happens.
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>

A few minor things inline.  The ifdef complexity in here is moderately
nasty though.  Might be worth seeing if any of that can be moved
to stubs or IS_ENABLED() checks.


> diff --git a/drivers/net/ethernet/sfc/efx.h b/drivers/net/ethernet/sfc/efx.h
> index 45e191686625..057d30090894 100644
> --- a/drivers/net/ethernet/sfc/efx.h
> +++ b/drivers/net/ethernet/sfc/efx.h
> @@ -236,5 +236,4 @@ static inline bool efx_rwsem_assert_write_locked(struct rw_semaphore *sem)
>  
>  int efx_xdp_tx_buffers(struct efx_nic *efx, int n, struct xdp_frame **xdpfs,
>  		       bool flush);
> -

stray change that you should clean out.

>  #endif /* EFX_EFX_H */
> diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
> index 79fe99d83f9f..a84ce45398c1 100644
> --- a/drivers/net/ethernet/sfc/efx_cxl.c
> +++ b/drivers/net/ethernet/sfc/efx_cxl.c
> @@ -11,6 +11,7 @@
>  #include <cxl/pci.h>
>  #include "net_driver.h"
>  #include "efx_cxl.h"
> +#include "efx.h"
>  
>  #define EFX_CTPIO_BUFFER_SIZE	SZ_256M
>  
> @@ -20,6 +21,7 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>  	struct pci_dev *pci_dev = efx->pci_dev;
>  	resource_size_t max_size;
>  	struct efx_cxl *cxl;
> +	struct range range;
>  	u16 dvsec;
>  	int rc;
>  
> @@ -119,19 +121,40 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>  	cxl->efx_region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1);
>  	if (IS_ERR(cxl->efx_region)) {
>  		pci_err(pci_dev, "CXL accel create region failed");
> -		cxl_put_root_decoder(cxl->cxlrd);
> -		cxl_dpa_free(cxl->cxled);
> -		return PTR_ERR(cxl->efx_region);
> +		rc = PTR_ERR(cxl->efx_region);
> +		goto err_dpa;

It's a somewhat trivial thing but you could reduce churn by
moving the err_dpa block introduction back to where this lot
was first added. 

> +	}
> +
> +	rc = cxl_get_region_range(cxl->efx_region, &range);
> +	if (rc) {
> +		pci_err(pci_dev, "CXL getting regions params failed");
> +		goto err_detach;
> +	}
> +
> +	cxl->ctpio_cxl = ioremap(range.start, range.end - range.start + 1);
> +	if (!cxl->ctpio_cxl) {
> +		pci_err(pci_dev, "CXL ioremap region (%pra) failed", &range);
> +		rc = -ENOMEM;
> +		goto err_detach;
>  	}
>  
>  	probe_data->cxl = cxl;
> +	probe_data->cxl_pio_initialised = true;
>  
>  	return 0;
> +
> +err_detach:
> +	cxl_decoder_detach(NULL, cxl->cxled, 0, DETACH_INVALIDATE);
> +err_dpa:
> +	cxl_put_root_decoder(cxl->cxlrd);
> +	cxl_dpa_free(cxl->cxled);
> +	return rc;
>  }
>  
>  void efx_cxl_exit(struct efx_probe_data *probe_data)
>  {
> -	if (probe_data->cxl) {
> +	if (probe_data->cxl_pio_initialised) {
> +		iounmap(probe_data->cxl->ctpio_cxl);
>  		cxl_decoder_detach(NULL, probe_data->cxl->cxled, 0,
>  				   DETACH_INVALIDATE);
>  		cxl_dpa_free(probe_data->cxl->cxled);

Re: [PATCH v19 00/22] Type2 device basic support

[Dave Jiang]

On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
> From: Alejandro Lucero <alucerop@amd.com>
> 
> The patchset should be applied on the described base commit then applying
> Terry's v11 about CXL error handling. The first 3 patches come from Dan's
> for-6.18/cxl-probe-order branch.

I Alejandro, I can't seem to apply with this instruction:

✔ ~/git/kernel-review [review L|…9]
16:35 $ git reset --hard f11a5f89910a7ae970fbce4fdc02d86a8ba8570f
HEAD is now at f11a5f89910a Documentation/ABI/testing/debugfs-cxl: Add 'cxl' to clear_poison path
✔ ~/git/kernel-review [review L|…9]
16:35 $ b4 shazam https://lore.kernel.org/linux-cxl/20251006100130.2623388-1-alejandro.lucero-palau@amd.com/T/#m712c7d01ffc7350d9ef638b932b9693a96fe47a9
Grabbing thread from lore.kernel.org/all/20251006100130.2623388-1-alejandro.lucero-palau@amd.com/t.mbox.gz
Checking for newer revisions
Grabbing search results from lore.kernel.org
Analyzing 33 messages in the thread
Analyzing 620 code-review messages
Checking attestation on all messages, may take a moment...
---
  ✓ [PATCH v19 1/22] cxl/mem: Arrange for always-synchronous memdev attach
  ✓ [PATCH v19 2/22] cxl/port: Arrange for always synchronous endpoint attach
  ✓ [PATCH v19 3/22] cxl/mem: Introduce a memdev creation ->probe() operation
  ✓ [PATCH v19 4/22] cxl: Add type2 device basic support
  ✓ [PATCH v19 5/22] sfc: add cxl support
  ✓ [PATCH v19 6/22] cxl: Move pci generic code
  ✓ [PATCH v19 7/22] cxl: allow Type2 drivers to map cxl component regs
  ✓ [PATCH v19 8/22] cxl: Support dpa initialization without a mailbox
  ✓ [PATCH v19 9/22] cxl: Prepare memdev creation for type2
  ✓ [PATCH v19 10/22] sfc: create type2 cxl memdev
  ✓ [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration
  ✓ [PATCH v19 12/22] sfc: get root decoder
  ✓ [PATCH v19 13/22] cxl: Define a driver interface for DPA allocation
    + Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
  ✓ [PATCH v19 14/22] sfc: get endpoint decoder
  ✓ [PATCH v19 15/22] cxl: Make region type based on endpoint type
    + Reviewed-by: Davidlohr Bueso <dave@stgolabs.net> (✓ DKIM/stgolabs.net)
  ✓ [PATCH v19 16/22] cxl/region: Factor out interleave ways setup
  ✓ [PATCH v19 17/22] cxl/region: Factor out interleave granularity setup
  ✓ [PATCH v19 18/22] cxl: Allow region creation by type2 drivers
  ✓ [PATCH v19 19/22] cxl: Avoid dax creation for accelerators
  ✓ [PATCH v19 20/22] sfc: create cxl region
  ✓ [PATCH v19 21/22] cxl: Add function for obtaining region range
  ✓ [PATCH v19 22/22] sfc: support pio mapping based on cxl
  ---
  ✓ Signed: DKIM/amd.com
  ---
  NOTE: install patatt for end-to-end signature verification
---
Total patches: 22
---
 Deps: looking for dependencies matching 23 patch-ids
 Deps: Applying prerequisite patch: [PATCH v11 01/23] cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
 Deps: Applying prerequisite patch: [PATCH v11 02/23] CXL/AER: Remove CONFIG_PCIEAER_CXL and replace with CONFIG_CXL_RAS
 Deps: Applying prerequisite patch: [PATCH v11 03/23] cxl/pci: Remove unnecessary CXL Endpoint handling helper functions
 Deps: Applying prerequisite patch: [PATCH v11 04/23] cxl/pci: Remove unnecessary CXL RCH handling helper functions
 Deps: Applying prerequisite patch: [PATCH v11 05/23] cxl: Move CXL driver RCH error handling into CONFIG_CXL_RCH_RAS conditional block
 Deps: Applying prerequisite patch: [PATCH v11 06/23] CXL/AER: Introduce rch_aer.c into AER driver for handling CXL RCH errors
 Deps: Applying prerequisite patch: [PATCH v11 08/23] PCI/CXL: Introduce pcie_is_cxl()
 Deps: Applying prerequisite patch: [PATCH v11 09/23] PCI/AER: Report CXL or PCIe bus error type in trace logging
 Deps: Applying prerequisite patch: [PATCH v11 10/23] CXL/AER: Update PCI class code check to use FIELD_GET()
 Deps: Applying prerequisite patch: [PATCH v11 11/23] cxl/pci: Update RAS handler interfaces to also support CXL Ports
 Deps: Applying prerequisite patch: [PATCH v12 12/25] cxl/pci: Log message if RAS registers are unmapped
 Deps: Applying prerequisite patch: [PATCH v11 13/23] cxl/pci: Unify CXL trace logging for CXL Endpoints and CXL Ports
 Deps: Applying prerequisite patch: [PATCH v12 14/25] cxl/pci: Update cxl_handle_cor_ras() to return early if no RAS errors
 Deps: Applying prerequisite patch: [PATCH v11 15/23] cxl/pci: Map CXL Endpoint Port and CXL Switch Port RAS registers
 Deps: Applying prerequisite patch: [PATCH v11 17/23] CXL/AER: Introduce cxl_aer.c into AER driver for forwarding CXL errors
 Deps: Applying prerequisite patch: [PATCH v11 18/23] PCI/AER: Dequeue forwarded CXL error
 Deps: Applying prerequisite patch: [PATCH v11 19/23] CXL/PCI: Introduce CXL Port protocol error handlers
 Deps: Applying prerequisite patch: [PATCH v11 20/23] CXL/PCI: Export and rename merge_result() to pci_ers_merge_result()
 Deps: Applying prerequisite patch: [PATCH v11 21/23] CXL/PCI: Introduce CXL uncorrectable protocol error recovery
 Deps: Applying prerequisite patch: [PATCH v11 22/23] CXL/PCI: Enable CXL protocol errors during CXL Port probe
 Deps: Applying prerequisite patch: [PATCH v11 23/23] CXL/PCI: Disable CXL protocol error interrupts during CXL Port cleanup
Applying: cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
Applying: CXL/AER: Remove CONFIG_PCIEAER_CXL and replace with CONFIG_CXL_RAS
Applying: cxl/pci: Remove unnecessary CXL Endpoint handling helper functions
Applying: cxl/pci: Remove unnecessary CXL RCH handling helper functions
Applying: cxl: Move CXL driver RCH error handling into CONFIG_CXL_RCH_RAS conditional block
Applying: CXL/AER: Introduce rch_aer.c into AER driver for handling CXL RCH errors
Applying: PCI/CXL: Introduce pcie_is_cxl()
Patch failed at 0007 PCI/CXL: Introduce pcie_is_cxl()
error: patch failed: include/uapi/linux/pci_regs.h:1274
error: include/uapi/linux/pci_regs.h: patch does not apply
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

I also tried applying Terry's v11 first (which applied) and then this series failed as well.

DJ

> 
> v19 changes:
> 
>   Removal of cxl_acquire_endpoint and driver callback for unexpected cxl
>   module removal. Dan's patches made them unnecessary.
> 
>   patch 4: remove code already moved by Terry's patches (Ben Cheatham)
> 
>   patch 6: removed unrelated change (Ben Cheatham)
> 
>   patch 7: fix error report inconsistencies (Jonathan, Dave)
> 
>   patch 9: remove unnecessary comment (Ben Cheatham)
> 
>   patch 11: fix __free usage (Jonathan Cameron, Ben Cheatham)
> 
>   patch 13: style fixes (Jonathan Cameron, Dave Jiag)
> 
>   patch 14: move code to previous patch (Jonathan Cameron)
> 
>   patch 18: group code in one locking (Dave Jian)
> 	    use __free helper (Ben Cheatham)
> 
> 
> v18 changes:
> 
>   patch 1: minor changes and fixing docs generation (Jonathan, Dan)
>  
>   patch4: merged with v17 patch5
> 
>   patch 5: merging v17 patches 6 and 7
> 
>   patch 6: adding helpers for clarity
> 
>   patch 9:
> 	- minor changes (Dave)
> 	- simplifying flags check (Dan)
> 
>   patch 10: minor changes (Jonathan)
> 
>   patch 11:
> 	- minor changes (Dave)
> 	- fix mess (Jonathan, Dave)
> 
>   patch 18: minor changes (Jonathan, Dan)
>   
> v17 changes: (Dan Williams review)
>  - use devm for cxl_dev_state allocation
>  - using current cxl struct for checking capability registers found by
>    the driver.
>  - simplify dpa initialization without a mailbox not supporting pmem
>  - add cxl_acquire_endpoint for protection during initialization
>  - add callback/action to cxl_create_region for a driver notified about cxl
>    core kernel modules removal.
>  - add sfc function to disable CXL-based PIO buffers if such a callback
>    is invoked.
>  - Always manage a Type2 created region as private not allowing DAX.
> 
> v16 changes:
>  - rebase against rc4 (Dave Jiang)
>  - remove duplicate line (Ben Cheatham)
> 
> v15 changes:
>  - remove reference to unused header file (Jonathan Cameron)
>  - add proper kernel docs to exported functions (Alison Schofield)
>  - using an array to map the enums to strings (Alison Schofield)
>  - clarify comment when using bitmap_subset (Jonathan Cameron)
>  - specify link to type2 support in all patches (Alison Schofield)
> 
>   Patches changed (minor): 4, 11
> 
> v14 changes:
>  - static null initialization of bitmaps (Jonathan Cameron)
>  - Fixing cxl tests (Alison Schofield)
>  - Fixing robot compilation problems
> 
>   Patches changed (minor): 1, 4, 6, 13
> 
> v13 changes:
>  - using names for headers checking more consistent (Jonathan Cameron)
>  - using helper for caps bit setting (Jonathan Cameron)
>  - provide generic function for reporting missing capabilities (Jonathan Cameron)
>  - rename cxl_pci_setup_memdev_regs to cxl_pci_accel_setup_memdev_regs (Jonathan Cameron)
>  - cxl_dpa_info size to be set by the Type2 driver (Jonathan Cameron)
>  - avoiding rc variable when possible (Jonathan Cameron)
>  - fix spelling (Simon Horman)
>  - use scoped_guard (Dave Jiang)
>  - use enum instead of bool (Dave Jiang)
>  - dropping patch with hardware symbols
> 
> v12 changes:
>  - use new macro cxl_dev_state_create in pci driver (Ben Cheatham)
>  - add public/private sections in now exported cxl_dev_state struct (Ben
>    Cheatham)
>  - fix cxl/pci.h regarding file name for checking if defined
>  - Clarify capabilities found vs expected in error message. (Ben
>    Cheatham)
>  - Clarify new CXL_DECODER_F flag (Ben Cheatham)
>  - Fix changes about cxl memdev creation support moving code to the
>    proper patch. (Ben Cheatham)
>  - Avoid debug and function duplications (Ben Cheatham)
> 
> v11 changes:
>  - Dropping the use of cxl_memdev_state and going back to using
>    cxl_dev_state.
>  - Using a helper for an accel driver to allocate its own cxl-related
>    struct embedding cxl_dev_state.
>  - Exporting the required structs in include/cxl/cxl.h for an accel
>    driver being able to know the cxl_dev_state size required in the
>    previously mentioned helper for allocation.
>  - Avoid using any struct for dpa initialization by the accel driver
>    adding a specific function for creating dpa partitions by accel
>    drivers without a mailbox.
> 
> v10 changes:
>  - Using cxl_memdev_state instead of cxl_dev_state for type2 which has a
>    memory after all and facilitates the setup.
>  - Adapt core for using cxl_memdev_state allowing accel drivers to work
>    with them without further awareness of internal cxl structs.
>  - Using last DPA changes for creating DPA partitions with accel driver
>    hardcoding mds values when no mailbox.
>  - capabilities not a new field but built up when current register maps
>    is performed and returned to the caller for checking.
>  - HPA free space supporting interleaving.
>  - DPA free space droping max-min for a simple alloc size.
> 
> v9 changes:
>  - adding forward definitions (Jonathan Cameron)
>  - using set_bit instead of bitmap_set (Jonathan Cameron)
>  - fix rebase problem (Jonathan Cameron)
>  - Improve error path (Jonathan Cameron)
>  - fix build problems with cxl region dependency (robot)
>  - fix error path (Simon Horman)
> 
> v8 changes:
>  - Change error path labeling inside sfc cxl code (Edward Cree)
>  - Properly handling checks and error in sfc cxl code (Simon Horman)
>  - Fix bug when checking resource_size (Simon Horman)
>  - Avoid bisect problems reordering patches (Edward Cree)
>  - Fix buffer allocation size in sfc (Simon Horman)
> 
> v7 changes:
> 
>  - fixing kernel test robot complains
>  - fix type with Type3 mandatory capabilities (Zhi Wang)
>  - optimize code in cxl_request_resource (Kalesh Anakkur Purayil)
>  - add sanity check when dealing with resources arithmetics (Fan Ni)
>  - fix typos and blank lines (Fan Ni)
>  - keep previous log errors/warnings in sfc driver (Martin Habets)
>  - add WARN_ON_ONCE if region given is NULL
> 
> v6 changes:
> 
>  - update sfc mcdi_pcol.h with full hardware changes most not related to
>    this patchset. This is an automatic file created from hardware design
>    changes and not touched by software. It is updated from time to time
>    and it required update for the sfc driver CXL support.
>  - remove CXL capabilities definitions not used by the patchset or
>    previous kernel code. (Dave Jiang, Jonathan Cameron)
>  - Use bitmap_subset instead of reinventing the wheel ... (Ben Cheatham)
>  - Use cxl_accel_memdev for new device_type created (Ben Cheatham)
>  - Fix construct_region use of rwsem (Zhi Wang)
>  - Obtain region range instead of region params (Allison Schofield, Dave
>    Jiang)
> 
> v5 changes:
> 
>  - Fix SFC configuration based on kernel CXL configuration
>  - Add subset check for capabilities.
>  - fix region creation when HDM decoders programmed by firmware/BIOS (Ben
>    Cheatham)
>  - Add option for creating dax region based on driver decission (Ben
>    Cheatham)
>  - Using sfc probe_data struct for keeping sfc cxl data
> 
> v4 changes:
> 
>  - Use bitmap for capabilities new field (Jonathan Cameron)
>  - Use cxl_mem attributes for sysfs based on device type (Dave Jian)
>  - Add conditional cxl sfc compilation relying on kernel CXL config (kernel test robot)
>  - Add sfc changes in different patches for facilitating backport (Jonathan Cameron)
>  - Remove patch for dealing with cxl modules dependencies and using sfc kconfig plus
>    MODULE_SOFTDEP instead.
> 
> v3 changes:
> 
>  - cxl_dev_state not defined as opaque but only manipulated by accel drivers
>    through accessors.
>  - accessors names not identified as only for accel drivers.
>  - move pci code from pci driver (drivers/cxl/pci.c) to generic pci code
>    (drivers/cxl/core/pci.c).
>  - capabilities field from u8 to u32 and initialised by CXL regs discovering
>    code.
>  - add capabilities check and removing current check by CXL regs discovering
>    code.
>  - Not fail if CXL Device Registers not found. Not mandatory for Type2.
>  - add timeout in acquire_endpoint for solving a race with the endpoint port
>    creation.
>  - handle EPROBE_DEFER by sfc driver.
>  - Limiting interleave ways to 1 for accel driver HPA/DPA requests.
>  - factoring out interleave ways and granularity helpers from type2 region
>    creation patch.
>  - restricting region_creation for type2 to one endpoint decoder.
> 
> v2 changes:
> 
> I have removed the introduction about the concerns with BIOS/UEFI after the
> discussion leading to confirm the need of the functionality implemented, at
> least is some scenarios.
> 
> There are two main changes from the RFC:
> 
> 1) Following concerns about drivers using CXL core without restrictions, the CXL
> struct to work with is opaque to those drivers, therefore functions are
> implemented for modifying or reading those structs indirectly.
> 
> 2) The driver for using the added functionality is not a test driver but a real
> one: the SFC ethernet network driver. It uses the CXL region mapped for PIO
> buffers instead of regions inside PCIe BARs.
> 
> RFC:
> 
> Current CXL kernel code is focused on supporting Type3 CXL devices, aka memory
> expanders. Type2 CXL devices, aka device accelerators, share some functionalities
> but require some special handling.
> 
> First of all, Type2 are by definition specific to drivers doing something and not just
> a memory expander, so it is expected to work with the CXL specifics. This implies the CXL
> setup needs to be done by such a driver instead of by a generic CXL PCI driver
> as for memory expanders. Most of such setup needs to use current CXL core code
> and therefore needs to be accessible to those vendor drivers. This is accomplished
> exporting opaque CXL structs and adding and exporting functions for working with
> those structs indirectly.
> 
> Some of the patches are based on a patchset sent by Dan Williams [1] which was just
> partially integrated, most related to making things ready for Type2 but none
> related to specific Type2 support. Those patches based on Dan´s work have Dan´s
> signing as co-developer, and a link to the original patch.
> 
> A final note about CXL.cache is needed. This patchset does not cover it at all,
> although the emulated Type2 device advertises it. From the kernel point of view
> supporting CXL.cache will imply to be sure the CXL path supports what the Type2
> device needs. A device accelerator will likely be connected to a Root Switch,
> but other configurations can not be discarded. Therefore the kernel will need to
> check not just HPA, DPA, interleave and granularity, but also the available
> CXL.cache support and resources in each switch in the CXL path to the Type2
> device. I expect to contribute to this support in the following months, and
> it would be good to discuss about it when possible.
> 
> [1] https://lore.kernel.org/linux-cxl/98b1f61a-e6c2-71d4-c368-50d958501b0c@intel.com/T/
> 
> Alejandro Lucero (21):
>   cxl/mem: Arrange for always-synchronous memdev attach
>   cxl/port: Arrange for always synchronous endpoint attach
>   cxl: Add type2 device basic support
>   sfc: add cxl support
>   cxl: Move pci generic code
>   cxl: allow Type2 drivers to map cxl component regs
>   cxl: Support dpa initialization without a mailbox
>   cxl: Prepare memdev creation for type2
>   sfc: create type2 cxl memdev
>   cxl: Define a driver interface for HPA free space enumeration
>   sfc: get root decoder
>   cxl: Define a driver interface for DPA allocation
>   sfc: get endpoint decoder
>   cxl: Make region type based on endpoint type
>   cxl/region: Factor out interleave ways setup
>   cxl/region: Factor out interleave granularity setup
>   cxl: Allow region creation by type2 drivers
>   cxl: Avoid dax creation for accelerators
>   sfc: create cxl region
>   cxl: Add function for obtaining region range
>   sfc: support pio mapping based on cxl
> 
> Dan Williams (1):
>   cxl/mem: Introduce a memdev creation ->probe() operation
> 
>  drivers/cxl/Kconfig                   |   2 +-
>  drivers/cxl/core/core.h               |   9 +-
>  drivers/cxl/core/hdm.c                |  85 ++++++
>  drivers/cxl/core/mbox.c               |  63 +---
>  drivers/cxl/core/memdev.c             | 209 +++++++++----
>  drivers/cxl/core/pci.c                |  63 ++++
>  drivers/cxl/core/port.c               |   1 +
>  drivers/cxl/core/region.c             | 418 +++++++++++++++++++++++---
>  drivers/cxl/core/regs.c               |   2 +-
>  drivers/cxl/cxl.h                     | 125 +-------
>  drivers/cxl/cxlmem.h                  |  90 +-----
>  drivers/cxl/cxlpci.h                  |  21 +-
>  drivers/cxl/mem.c                     | 146 +++++----
>  drivers/cxl/pci.c                     |  88 +-----
>  drivers/cxl/port.c                    |  46 ++-
>  drivers/cxl/private.h                 |  17 ++
>  drivers/net/ethernet/sfc/Kconfig      |  10 +
>  drivers/net/ethernet/sfc/Makefile     |   1 +
>  drivers/net/ethernet/sfc/ef10.c       |  50 ++-
>  drivers/net/ethernet/sfc/efx.c        |  15 +-
>  drivers/net/ethernet/sfc/efx.h        |   1 -
>  drivers/net/ethernet/sfc/efx_cxl.c    | 165 ++++++++++
>  drivers/net/ethernet/sfc/efx_cxl.h    |  40 +++
>  drivers/net/ethernet/sfc/net_driver.h |  12 +
>  drivers/net/ethernet/sfc/nic.h        |   3 +
>  include/cxl/cxl.h                     | 291 ++++++++++++++++++
>  include/cxl/pci.h                     |  21 ++
>  tools/testing/cxl/Kbuild              |   1 -
>  tools/testing/cxl/test/mem.c          |   5 +-
>  tools/testing/cxl/test/mock.c         |  17 --
>  30 files changed, 1476 insertions(+), 541 deletions(-)
>  create mode 100644 drivers/cxl/private.h
>  create mode 100644 drivers/net/ethernet/sfc/efx_cxl.c
>  create mode 100644 drivers/net/ethernet/sfc/efx_cxl.h
>  create mode 100644 include/cxl/cxl.h
>  create mode 100644 include/cxl/pci.h
> 
> 
> base-commit: f11a5f89910a7ae970fbce4fdc02d86a8ba8570f
> prerequisite-patch-id: 44c914dd079e40d716f3f2d91653247eca731594
> prerequisite-patch-id: b13ca5c11c44a736563477d67b1dceadfe3ea19e
> prerequisite-patch-id: d0d82965bbea8a2b5ea2f763f19de4dfaa8479c3
> prerequisite-patch-id: dd0f24b3bdb938f2f123bc26b31cd5fe659e05eb
> prerequisite-patch-id: 2ea41ec399f2360a84e86e97a8f940a62561931a
> prerequisite-patch-id: 367b61b5a313db6324f9cf917d46df580f3bbd3b
> prerequisite-patch-id: 1805332a9f191bc3547927d96de5926356dac03c
> prerequisite-patch-id: 40657fd517f8e835a091c07e93d6abc08f85d395
> prerequisite-patch-id: 901eb0d91816499446964b2a9089db59656da08d
> prerequisite-patch-id: 79856c0199d6872fd2f76a5829dba7fa46f225d6
> prerequisite-patch-id: 6f3503e59a3d745e5ecff4aaed668e2d32da7e4b
> prerequisite-patch-id: e9dc88f1b91dce5dc3d46ff2b5bf184aba06439d
> prerequisite-patch-id: 196fe106100aad619d5be7266959bbeef29b7c8b
> prerequisite-patch-id: 7e719ed404f664ee8d9b98d56f58326f55ea2175
> prerequisite-patch-id: 560f95992e13a08279034d5f77aacc9e971332dd
> prerequisite-patch-id: 8656445ee654056695ff2894e28c8f1014df919e
> prerequisite-patch-id: 001d831149eb8f9ae17b394e4bcd06d844dd39d9
> prerequisite-patch-id: 421368aa5eac2af63ef2dc427af2ec11ad45c925
> prerequisite-patch-id: 18fd00d4743711d835ad546cfbb558d9f97dcdfc
> prerequisite-patch-id: d89bf9e6d3ea5d332ec2c8e441f1fe6d84e726d3
> prerequisite-patch-id: 3a6953d11b803abeb437558f3893a3b6a08acdbb
> prerequisite-patch-id: 0dd42a82e73765950bd069d421d555ded8bfeb25
> prerequisite-patch-id: da6e0df31ad0d5a945e0a0d29204ba75f0c97344

Re: [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration

[Cheatham, Benjamin]

On 10/6/2025 5:01 AM, alejandro.lucero-palau@amd.com wrote:
> From: Alejandro Lucero <alucerop@amd.com>
> 
> CXL region creation involves allocating capacity from Device Physical Address
> (DPA) and assigning it to decode a given Host Physical Address (HPA). Before
> determining how much DPA to allocate the amount of available HPA must be
> determined. Also, not all HPA is created equal, some HPA targets RAM, some
> targets PMEM, some is prepared for device-memory flows like HDM-D and HDM-DB,
> and some is HDM-H (host-only).
> 
> In order to support Type2 CXL devices, wrap all of those concerns into
> an API that retrieves a root decoder (platform CXL window) that fits the
> specified constraints and the capacity available for a new region.
> 
> Add a complementary function for releasing the reference to such root
> decoder.
> 
> Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---

Hey Alejandro,

I've been testing this on my setup and noticed a few issues when BIOS sets up the HDM decoders. It came down to 2 issues:
	1) Enabling "Specific Purpose Memory" added a Soft Reserve resource below the CXL window resource and broke
	this patch (more below)
	2) The endpoint decoder was already set up which broke DPA allocation and then CXL region creation (see my response
	to patch 18/22 for fix and explanation)

The fix I did for 1 is a bit hacky but it's essentially checking none of the resources below the CXL window are onlined as
system memory. It's roughly equivalent to what's being done in the CXL_PARTMODE_RAM case of cxl_region_probe(), but
I'm restricting the resources to "Soft Reserved" to be safe.

The diff for 2 is pretty big. If you don't want to take it at this point I can send it as a follow up. In that case I'd definitely
add that auto regions won't work in at least the cover letter (and in the description of 18/22 as well?).

---

diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index acaca64764bf..2d60131edff3 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -784,6 +784,19 @@ static int find_max_hpa(struct device *dev, void *data)
        lockdep_assert_held_read(&cxl_rwsem.region);
        res = cxlrd->res->child;

+       /*
+        * BIOS may have marked the CXL window as soft reserved. Make sure it's
+        * free to use.
+        */
+       while (res && resource_size(res) == resource_size(cxlrd->res)) {
+               if ((res->flags & IORESOURCE_BUSY) ||
+                   (res->flags & IORESOURCE_SYSRAM) ||
+                   strcmp(res->name, "Soft Reserved") != 0)
+                       return 0;
+
+               res = res->child;
+       }
+
        /* With no resource child the whole parent resource is available */
        if (!res)
                max = resource_size(cxlrd->res);

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Cheatham, Benjamin]

On 10/6/2025 5:01 AM, alejandro.lucero-palau@amd.com wrote:
> From: Alejandro Lucero <alucerop@amd.com>
> 
> Creating a CXL region requires userspace intervention through the cxl
> sysfs files. Type2 support should allow accelerator drivers to create
> such cxl region from kernel code.
> 
> Adding that functionality and integrating it with current support for
> memory expanders.
> 
> Support an action by the type2 driver to be linked to the created region
> for unwinding the resources allocated properly.
> 
> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---

Fix for this one should be split between 13/22 and this patch, but the majority of it is in this one. The idea is
if we don't find a free decoder we check for pre-programmed decoders and use that instead. Unfortunately, this
invalidates some of the assumptions made by __construct_new_region().

__construct_new_region() assumes that 1) the underlying HPA is unallocated and 2) the HDM decoders aren't programmed. Neither
of those are true for a decoder that's programmed by BIOS. The HPA is allocated as part of endpoint_port_probe()
(see devm_cxl_enumerate_decoders() in cxl/core/hdm.c specifically) and the HDM decoders are enabled and committed by BIOS before
we ever see them. So the idea here is to split the second half of __construct_new_region() into the 2 cases: un-programmed decoders
(__setup_new_region()) and pre-programmed decoders (__setup_new_auto_region()). The main differences between the two is we don't
allocate the HPA region or commit the HDM decoders and just insert the region resource below the CXL window instead in the auto case.

I'm not sure if I've done everything correctly, but I don't see any errors and get the following iomem tree:
	1050000000-144fffffff : CXL Window 0
  	  1050000000-144fffffff : region0
    	    1050000000-144fffffff : Soft Reserved

---

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index 4af5de5e0a44..a5fa8dd0e63f 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -137,6 +137,8 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
                        struct cxl_endpoint_dvsec_info *info);
 int cxl_port_get_possible_dports(struct cxl_port *port);

+bool is_auto_decoder(struct cxl_endpoint_decoder *cxled);
+
 #ifdef CONFIG_CXL_FEATURES
 struct cxl_feat_entry *
 cxl_feature_info(struct cxl_features_state *cxlfs, const uuid_t *uuid);
diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
index 1f7aa79c1541..8f6236a88c0b 100644
--- a/drivers/cxl/core/hdm.c
+++ b/drivers/cxl/core/hdm.c
@@ -712,16 +712,33 @@ static int find_free_decoder(struct device *dev, const void *data)
        return 1;
 }

+bool is_auto_decoder(struct cxl_endpoint_decoder *cxled)
+{
+       return cxled->state == CXL_DECODER_STATE_AUTO && cxled->pos < 0 &&
+              (cxled->cxld.flags & CXL_DECODER_F_ENABLE);
+}
+
+static int find_auto_decoder(struct device *dev, const void *data)
+{
+       if (!is_endpoint_decoder(dev))
+               return 0;
+
+       return is_auto_decoder(to_cxl_endpoint_decoder(dev));
+}
+
 static struct cxl_endpoint_decoder *
 cxl_find_free_decoder(struct cxl_memdev *cxlmd)
 {
        struct cxl_port *endpoint = cxlmd->endpoint;
        struct device *dev;

-       scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
-               dev = device_find_child(&endpoint->dev, NULL,
-                                       find_free_decoder);
-       }
+       guard(rwsem_read)(&cxl_rwsem.dpa);
+       dev = device_find_child(&endpoint->dev, NULL,
+                               find_free_decoder);
+       if (dev)
+               return to_cxl_endpoint_decoder(dev);
+
+       dev = device_find_child(&endpoint->dev, NULL, find_auto_decoder);
        if (dev)
                return to_cxl_endpoint_decoder(dev);

@@ -761,6 +778,9 @@ struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
        if (!cxled)
                return ERR_PTR(-ENODEV);

+       if (is_auto_decoder(cxled))
+               return_ptr(cxled);
+
        rc = cxl_dpa_set_part(cxled, mode);
        if (rc)
                return ERR_PTR(rc);
diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index 2d60131edff3..004e01ad0e5f 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -3699,48 +3699,74 @@ cxl_find_region_by_range(struct cxl_root_decoder *cxlrd, struct range *hpa)
 }

 static struct cxl_region *
-__construct_new_region(struct cxl_root_decoder *cxlrd,
-                      struct cxl_endpoint_decoder **cxled, int ways)
+__setup_new_auto_region(struct cxl_region *cxlr, struct cxl_root_decoder *cxlrd,
+                       struct cxl_endpoint_decoder **cxled, int ways)
 {
-       struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
-       struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
-       struct cxl_region_params *p;
+       struct range *hpa = &cxled[0]->cxld.hpa_range;
+       struct cxl_region_params *p = &cxlr->params;
        resource_size_t size = 0;
-       struct cxl_region *cxlr;
-       int rc, i;
+       struct resource *res;
+       int rc = -EINVAL, i = 0;

-       cxlr = construct_region_begin(cxlrd, cxled[0]);
-       if (IS_ERR(cxlr))
-               return cxlr;
+       scoped_guard(rwsem_read, &cxl_rwsem.dpa)
+       {
+               for (i = 0; i < ways; i++) {
+                       if (!cxled[i]->dpa_res)
+                               goto err;

-       guard(rwsem_write)(&cxl_rwsem.region);
+                       if (!is_auto_decoder(cxled[i]))
+                               goto err;

-       /*
-        * Sanity check. This should not happen with an accel driver handling
-        * the region creation.
-        */
-       p = &cxlr->params;
-       if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
-               dev_err(cxlmd->dev.parent,
-                       "%s:%s: %s  unexpected region state\n",
-                       dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
-                       __func__);
-               rc = -EBUSY;
-               goto err;
+                       size += resource_size(cxled[i]->dpa_res);
+               }
        }

-       rc = set_interleave_ways(cxlr, ways);
-       if (rc)
+       set_bit(CXL_REGION_F_AUTO, &cxlr->flags);
+
+       p->res = kmalloc(sizeof(*res), GFP_KERNEL);
+       if (!p->res) {
+               rc = -ENOMEM;
                goto err;
+       }

-       rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
+       *p->res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa),
+                                      dev_name(&cxlr->dev));
+
+       rc = insert_resource(cxlrd->res, p->res);
        if (rc)
-               goto err;
+               dev_warn(&cxlr->dev, "Could not insert resource\n");
+
+       p->state = CXL_CONFIG_INTERLEAVE_ACTIVE;
+       scoped_guard(rwsem_read, &cxl_rwsem.dpa)
+       {
+               for (i = 0; i < ways; i++) {
+                       rc = cxl_region_attach(cxlr, cxled[i], -1);
+                       if (rc)
+                               goto err;
+               }
+       }
+
+       return cxlr;
+
+err:
+       drop_region(cxlr);
+       return ERR_PTR(rc);
+}
+
+static struct cxl_region *
+__setup_new_region(struct cxl_region *cxlr, struct cxl_root_decoder *cxlrd,
+                  struct cxl_endpoint_decoder **cxled, int ways)
+{
+       struct cxl_region_params *p = &cxlr->params;
+       resource_size_t size = 0;
+       int rc = -EINVAL, i = 0;

-       scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
+       scoped_guard(rwsem_read, &cxl_rwsem.dpa)
+       {
                for (i = 0; i < ways; i++) {
                        if (!cxled[i]->dpa_res)
                                break;
+
                        size += resource_size(cxled[i]->dpa_res);
                }
        }
@@ -3752,7 +3778,8 @@ __construct_new_region(struct cxl_root_decoder *cxlrd,
        if (rc)
                goto err;

-       scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
+       scoped_guard(rwsem_read, &cxl_rwsem.dpa)
+       {
                for (i = 0; i < ways; i++) {
                        rc = cxl_region_attach(cxlr, cxled[i], 0);
                        if (rc)
@@ -3760,16 +3787,61 @@ __construct_new_region(struct cxl_root_decoder *cxlrd,
                }
        }

+       rc = cxl_region_decode_commit(cxlr);
        if (rc)
                goto err;

-       rc = cxl_region_decode_commit(cxlr);
+       p->state = CXL_CONFIG_COMMIT;
+       return cxlr;
+
+err:
+       drop_region(cxlr);
+       return ERR_PTR(rc);
+}
+
+static struct cxl_region *
+__construct_new_region(struct cxl_root_decoder *cxlrd,
+                      struct cxl_endpoint_decoder **cxled, int ways)
+{
+       struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
+       struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
+       struct cxl_region_params *p;
+       struct cxl_region *cxlr;
+       int rc;
+
+       cxlr = construct_region_begin(cxlrd, cxled[0]);
+       if (IS_ERR(cxlr))
+               return cxlr;
+
+       guard(rwsem_write)(&cxl_rwsem.region);
+
+       /*
+        * Sanity check. This should not happen with an accel driver handling
+        * the region creation.
+        */
+       p = &cxlr->params;
+       if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
+               dev_err(cxlmd->dev.parent,
+                       "%s:%s: %s  unexpected region state\n",
+                       dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
+                       __func__);
+               rc = -EBUSY;
+               goto err;
+       }
+
+       rc = set_interleave_ways(cxlr, ways);
        if (rc)
                goto err;

-       p->state = CXL_CONFIG_COMMIT;
+       rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
+       if (rc)
+               goto err;
+
+       if (is_auto_decoder(cxled[0]))
+               return __setup_new_auto_region(cxlr, cxlrd, cxled, ways);
+       else
+               return __setup_new_region(cxlr, cxlrd, cxled, ways);

-       return cxlr;
 err:
        drop_region(cxlr);
        return ERR_PTR(rc);

Re: [PATCH v19 00/22] Type2 device basic support

[Alejandro Lucero Palau]

Hi Dave,

On 10/8/25 00:41, Dave Jiang wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> The patchset should be applied on the described base commit then applying
>> Terry's v11 about CXL error handling. The first 3 patches come from Dan's
>> for-6.18/cxl-probe-order branch.
> I Alejandro, I can't seem to apply with this instruction:
>
> ✔ ~/git/kernel-review [review L|…9]
> 16:35 $ git reset --hard f11a5f89910a7ae970fbce4fdc02d86a8ba8570f
> HEAD is now at f11a5f89910a Documentation/ABI/testing/debugfs-cxl: Add 'cxl' to clear_poison path
> ✔ ~/git/kernel-review [review L|…9]
> 16:35 $ b4 shazam https://lore.kernel.org/linux-cxl/20251006100130.2623388-1-alejandro.lucero-palau@amd.com/T/#m712c7d01ffc7350d9ef638b932b9693a96fe47a9
> Grabbing thread from lore.kernel.org/all/20251006100130.2623388-1-alejandro.lucero-palau@amd.com/t.mbox.gz
> Checking for newer revisions
> Grabbing search results from lore.kernel.org
> Analyzing 33 messages in the thread
> Analyzing 620 code-review messages
> Checking attestation on all messages, may take a moment...
> ---
>    ✓ [PATCH v19 1/22] cxl/mem: Arrange for always-synchronous memdev attach
>    ✓ [PATCH v19 2/22] cxl/port: Arrange for always synchronous endpoint attach
>    ✓ [PATCH v19 3/22] cxl/mem: Introduce a memdev creation ->probe() operation
>    ✓ [PATCH v19 4/22] cxl: Add type2 device basic support
>    ✓ [PATCH v19 5/22] sfc: add cxl support
>    ✓ [PATCH v19 6/22] cxl: Move pci generic code
>    ✓ [PATCH v19 7/22] cxl: allow Type2 drivers to map cxl component regs
>    ✓ [PATCH v19 8/22] cxl: Support dpa initialization without a mailbox
>    ✓ [PATCH v19 9/22] cxl: Prepare memdev creation for type2
>    ✓ [PATCH v19 10/22] sfc: create type2 cxl memdev
>    ✓ [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration
>    ✓ [PATCH v19 12/22] sfc: get root decoder
>    ✓ [PATCH v19 13/22] cxl: Define a driver interface for DPA allocation
>      + Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
>    ✓ [PATCH v19 14/22] sfc: get endpoint decoder
>    ✓ [PATCH v19 15/22] cxl: Make region type based on endpoint type
>      + Reviewed-by: Davidlohr Bueso <dave@stgolabs.net> (✓ DKIM/stgolabs.net)
>    ✓ [PATCH v19 16/22] cxl/region: Factor out interleave ways setup
>    ✓ [PATCH v19 17/22] cxl/region: Factor out interleave granularity setup
>    ✓ [PATCH v19 18/22] cxl: Allow region creation by type2 drivers
>    ✓ [PATCH v19 19/22] cxl: Avoid dax creation for accelerators
>    ✓ [PATCH v19 20/22] sfc: create cxl region
>    ✓ [PATCH v19 21/22] cxl: Add function for obtaining region range
>    ✓ [PATCH v19 22/22] sfc: support pio mapping based on cxl
>    ---
>    ✓ Signed: DKIM/amd.com
>    ---
>    NOTE: install patatt for end-to-end signature verification
> ---
> Total patches: 22
> ---
>   Deps: looking for dependencies matching 23 patch-ids
>   Deps: Applying prerequisite patch: [PATCH v11 01/23] cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
>   Deps: Applying prerequisite patch: [PATCH v11 02/23] CXL/AER: Remove CONFIG_PCIEAER_CXL and replace with CONFIG_CXL_RAS
>   Deps: Applying prerequisite patch: [PATCH v11 03/23] cxl/pci: Remove unnecessary CXL Endpoint handling helper functions
>   Deps: Applying prerequisite patch: [PATCH v11 04/23] cxl/pci: Remove unnecessary CXL RCH handling helper functions
>   Deps: Applying prerequisite patch: [PATCH v11 05/23] cxl: Move CXL driver RCH error handling into CONFIG_CXL_RCH_RAS conditional block
>   Deps: Applying prerequisite patch: [PATCH v11 06/23] CXL/AER: Introduce rch_aer.c into AER driver for handling CXL RCH errors
>   Deps: Applying prerequisite patch: [PATCH v11 08/23] PCI/CXL: Introduce pcie_is_cxl()
>   Deps: Applying prerequisite patch: [PATCH v11 09/23] PCI/AER: Report CXL or PCIe bus error type in trace logging
>   Deps: Applying prerequisite patch: [PATCH v11 10/23] CXL/AER: Update PCI class code check to use FIELD_GET()
>   Deps: Applying prerequisite patch: [PATCH v11 11/23] cxl/pci: Update RAS handler interfaces to also support CXL Ports
>   Deps: Applying prerequisite patch: [PATCH v12 12/25] cxl/pci: Log message if RAS registers are unmapped
>   Deps: Applying prerequisite patch: [PATCH v11 13/23] cxl/pci: Unify CXL trace logging for CXL Endpoints and CXL Ports
>   Deps: Applying prerequisite patch: [PATCH v12 14/25] cxl/pci: Update cxl_handle_cor_ras() to return early if no RAS errors
>   Deps: Applying prerequisite patch: [PATCH v11 15/23] cxl/pci: Map CXL Endpoint Port and CXL Switch Port RAS registers
>   Deps: Applying prerequisite patch: [PATCH v11 17/23] CXL/AER: Introduce cxl_aer.c into AER driver for forwarding CXL errors
>   Deps: Applying prerequisite patch: [PATCH v11 18/23] PCI/AER: Dequeue forwarded CXL error
>   Deps: Applying prerequisite patch: [PATCH v11 19/23] CXL/PCI: Introduce CXL Port protocol error handlers
>   Deps: Applying prerequisite patch: [PATCH v11 20/23] CXL/PCI: Export and rename merge_result() to pci_ers_merge_result()
>   Deps: Applying prerequisite patch: [PATCH v11 21/23] CXL/PCI: Introduce CXL uncorrectable protocol error recovery
>   Deps: Applying prerequisite patch: [PATCH v11 22/23] CXL/PCI: Enable CXL protocol errors during CXL Port probe
>   Deps: Applying prerequisite patch: [PATCH v11 23/23] CXL/PCI: Disable CXL protocol error interrupts during CXL Port cleanup
> Applying: cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
> Applying: CXL/AER: Remove CONFIG_PCIEAER_CXL and replace with CONFIG_CXL_RAS
> Applying: cxl/pci: Remove unnecessary CXL Endpoint handling helper functions
> Applying: cxl/pci: Remove unnecessary CXL RCH handling helper functions
> Applying: cxl: Move CXL driver RCH error handling into CONFIG_CXL_RCH_RAS conditional block
> Applying: CXL/AER: Introduce rch_aer.c into AER driver for handling CXL RCH errors
> Applying: PCI/CXL: Introduce pcie_is_cxl()
> Patch failed at 0007 PCI/CXL: Introduce pcie_is_cxl()
> error: patch failed: include/uapi/linux/pci_regs.h:1274
> error: include/uapi/linux/pci_regs.h: patch does not apply
> hint: Use 'git am --show-current-patch=diff' to see the failed patch
> hint: When you have resolved this problem, run "git am --continue".
> hint: If you prefer to skip this patch, run "git am --skip" instead.
> hint: To restore the original branch and stop patching, run "git am --abort".
> hint: Disable this message with "git config set advice.mergeConflict false"
>
> I also tried applying Terry's v11 first (which applied) and then this series failed as well.


You need to apply Terry's v11 patches for sure on the commit base. If 
not the v19 series can not be applied cleanly.


I have tried this again a couple of times and it works for me:


1)  git reset --hard f11a5f89910a

2) git am Enable-CXL-PCIe-Port-Protocol-Error-handling-and-logging.patch

3) git am Type2-device-basic-support.patch


Hopefully you can reproduce this as well. Maybe Ben Cheatham can comment 
on this, if he had problems applying them, as it seems he was able to 
work with it.


Thank you


> DJ
>
>> v19 changes:
>>
>>    Removal of cxl_acquire_endpoint and driver callback for unexpected cxl
>>    module removal. Dan's patches made them unnecessary.
>>
>>    patch 4: remove code already moved by Terry's patches (Ben Cheatham)
>>
>>    patch 6: removed unrelated change (Ben Cheatham)
>>
>>    patch 7: fix error report inconsistencies (Jonathan, Dave)
>>
>>    patch 9: remove unnecessary comment (Ben Cheatham)
>>
>>    patch 11: fix __free usage (Jonathan Cameron, Ben Cheatham)
>>
>>    patch 13: style fixes (Jonathan Cameron, Dave Jiag)
>>
>>    patch 14: move code to previous patch (Jonathan Cameron)
>>
>>    patch 18: group code in one locking (Dave Jian)
>>            use __free helper (Ben Cheatham)
>>
>>
>> v18 changes:
>>
>>    patch 1: minor changes and fixing docs generation (Jonathan, Dan)
>>
>>    patch4: merged with v17 patch5
>>
>>    patch 5: merging v17 patches 6 and 7
>>
>>    patch 6: adding helpers for clarity
>>
>>    patch 9:
>>        - minor changes (Dave)
>>        - simplifying flags check (Dan)
>>
>>    patch 10: minor changes (Jonathan)
>>
>>    patch 11:
>>        - minor changes (Dave)
>>        - fix mess (Jonathan, Dave)
>>
>>    patch 18: minor changes (Jonathan, Dan)
>>
>> v17 changes: (Dan Williams review)
>>   - use devm for cxl_dev_state allocation
>>   - using current cxl struct for checking capability registers found by
>>     the driver.
>>   - simplify dpa initialization without a mailbox not supporting pmem
>>   - add cxl_acquire_endpoint for protection during initialization
>>   - add callback/action to cxl_create_region for a driver notified about cxl
>>     core kernel modules removal.
>>   - add sfc function to disable CXL-based PIO buffers if such a callback
>>     is invoked.
>>   - Always manage a Type2 created region as private not allowing DAX.
>>
>> v16 changes:
>>   - rebase against rc4 (Dave Jiang)
>>   - remove duplicate line (Ben Cheatham)
>>
>> v15 changes:
>>   - remove reference to unused header file (Jonathan Cameron)
>>   - add proper kernel docs to exported functions (Alison Schofield)
>>   - using an array to map the enums to strings (Alison Schofield)
>>   - clarify comment when using bitmap_subset (Jonathan Cameron)
>>   - specify link to type2 support in all patches (Alison Schofield)
>>
>>    Patches changed (minor): 4, 11
>>
>> v14 changes:
>>   - static null initialization of bitmaps (Jonathan Cameron)
>>   - Fixing cxl tests (Alison Schofield)
>>   - Fixing robot compilation problems
>>
>>    Patches changed (minor): 1, 4, 6, 13
>>
>> v13 changes:
>>   - using names for headers checking more consistent (Jonathan Cameron)
>>   - using helper for caps bit setting (Jonathan Cameron)
>>   - provide generic function for reporting missing capabilities (Jonathan Cameron)
>>   - rename cxl_pci_setup_memdev_regs to cxl_pci_accel_setup_memdev_regs (Jonathan Cameron)
>>   - cxl_dpa_info size to be set by the Type2 driver (Jonathan Cameron)
>>   - avoiding rc variable when possible (Jonathan Cameron)
>>   - fix spelling (Simon Horman)
>>   - use scoped_guard (Dave Jiang)
>>   - use enum instead of bool (Dave Jiang)
>>   - dropping patch with hardware symbols
>>
>> v12 changes:
>>   - use new macro cxl_dev_state_create in pci driver (Ben Cheatham)
>>   - add public/private sections in now exported cxl_dev_state struct (Ben
>>     Cheatham)
>>   - fix cxl/pci.h regarding file name for checking if defined
>>   - Clarify capabilities found vs expected in error message. (Ben
>>     Cheatham)
>>   - Clarify new CXL_DECODER_F flag (Ben Cheatham)
>>   - Fix changes about cxl memdev creation support moving code to the
>>     proper patch. (Ben Cheatham)
>>   - Avoid debug and function duplications (Ben Cheatham)
>>
>> v11 changes:
>>   - Dropping the use of cxl_memdev_state and going back to using
>>     cxl_dev_state.
>>   - Using a helper for an accel driver to allocate its own cxl-related
>>     struct embedding cxl_dev_state.
>>   - Exporting the required structs in include/cxl/cxl.h for an accel
>>     driver being able to know the cxl_dev_state size required in the
>>     previously mentioned helper for allocation.
>>   - Avoid using any struct for dpa initialization by the accel driver
>>     adding a specific function for creating dpa partitions by accel
>>     drivers without a mailbox.
>>
>> v10 changes:
>>   - Using cxl_memdev_state instead of cxl_dev_state for type2 which has a
>>     memory after all and facilitates the setup.
>>   - Adapt core for using cxl_memdev_state allowing accel drivers to work
>>     with them without further awareness of internal cxl structs.
>>   - Using last DPA changes for creating DPA partitions with accel driver
>>     hardcoding mds values when no mailbox.
>>   - capabilities not a new field but built up when current register maps
>>     is performed and returned to the caller for checking.
>>   - HPA free space supporting interleaving.
>>   - DPA free space droping max-min for a simple alloc size.
>>
>> v9 changes:
>>   - adding forward definitions (Jonathan Cameron)
>>   - using set_bit instead of bitmap_set (Jonathan Cameron)
>>   - fix rebase problem (Jonathan Cameron)
>>   - Improve error path (Jonathan Cameron)
>>   - fix build problems with cxl region dependency (robot)
>>   - fix error path (Simon Horman)
>>
>> v8 changes:
>>   - Change error path labeling inside sfc cxl code (Edward Cree)
>>   - Properly handling checks and error in sfc cxl code (Simon Horman)
>>   - Fix bug when checking resource_size (Simon Horman)
>>   - Avoid bisect problems reordering patches (Edward Cree)
>>   - Fix buffer allocation size in sfc (Simon Horman)
>>
>> v7 changes:
>>
>>   - fixing kernel test robot complains
>>   - fix type with Type3 mandatory capabilities (Zhi Wang)
>>   - optimize code in cxl_request_resource (Kalesh Anakkur Purayil)
>>   - add sanity check when dealing with resources arithmetics (Fan Ni)
>>   - fix typos and blank lines (Fan Ni)
>>   - keep previous log errors/warnings in sfc driver (Martin Habets)
>>   - add WARN_ON_ONCE if region given is NULL
>>
>> v6 changes:
>>
>>   - update sfc mcdi_pcol.h with full hardware changes most not related to
>>     this patchset. This is an automatic file created from hardware design
>>     changes and not touched by software. It is updated from time to time
>>     and it required update for the sfc driver CXL support.
>>   - remove CXL capabilities definitions not used by the patchset or
>>     previous kernel code. (Dave Jiang, Jonathan Cameron)
>>   - Use bitmap_subset instead of reinventing the wheel ... (Ben Cheatham)
>>   - Use cxl_accel_memdev for new device_type created (Ben Cheatham)
>>   - Fix construct_region use of rwsem (Zhi Wang)
>>   - Obtain region range instead of region params (Allison Schofield, Dave
>>     Jiang)
>>
>> v5 changes:
>>
>>   - Fix SFC configuration based on kernel CXL configuration
>>   - Add subset check for capabilities.
>>   - fix region creation when HDM decoders programmed by firmware/BIOS (Ben
>>     Cheatham)
>>   - Add option for creating dax region based on driver decission (Ben
>>     Cheatham)
>>   - Using sfc probe_data struct for keeping sfc cxl data
>>
>> v4 changes:
>>
>>   - Use bitmap for capabilities new field (Jonathan Cameron)
>>   - Use cxl_mem attributes for sysfs based on device type (Dave Jian)
>>   - Add conditional cxl sfc compilation relying on kernel CXL config (kernel test robot)
>>   - Add sfc changes in different patches for facilitating backport (Jonathan Cameron)
>>   - Remove patch for dealing with cxl modules dependencies and using sfc kconfig plus
>>     MODULE_SOFTDEP instead.
>>
>> v3 changes:
>>
>>   - cxl_dev_state not defined as opaque but only manipulated by accel drivers
>>     through accessors.
>>   - accessors names not identified as only for accel drivers.
>>   - move pci code from pci driver (drivers/cxl/pci.c) to generic pci code
>>     (drivers/cxl/core/pci.c).
>>   - capabilities field from u8 to u32 and initialised by CXL regs discovering
>>     code.
>>   - add capabilities check and removing current check by CXL regs discovering
>>     code.
>>   - Not fail if CXL Device Registers not found. Not mandatory for Type2.
>>   - add timeout in acquire_endpoint for solving a race with the endpoint port
>>     creation.
>>   - handle EPROBE_DEFER by sfc driver.
>>   - Limiting interleave ways to 1 for accel driver HPA/DPA requests.
>>   - factoring out interleave ways and granularity helpers from type2 region
>>     creation patch.
>>   - restricting region_creation for type2 to one endpoint decoder.
>>
>> v2 changes:
>>
>> I have removed the introduction about the concerns with BIOS/UEFI after the
>> discussion leading to confirm the need of the functionality implemented, at
>> least is some scenarios.
>>
>> There are two main changes from the RFC:
>>
>> 1) Following concerns about drivers using CXL core without restrictions, the CXL
>> struct to work with is opaque to those drivers, therefore functions are
>> implemented for modifying or reading those structs indirectly.
>>
>> 2) The driver for using the added functionality is not a test driver but a real
>> one: the SFC ethernet network driver. It uses the CXL region mapped for PIO
>> buffers instead of regions inside PCIe BARs.
>>
>> RFC:
>>
>> Current CXL kernel code is focused on supporting Type3 CXL devices, aka memory
>> expanders. Type2 CXL devices, aka device accelerators, share some functionalities
>> but require some special handling.
>>
>> First of all, Type2 are by definition specific to drivers doing something and not just
>> a memory expander, so it is expected to work with the CXL specifics. This implies the CXL
>> setup needs to be done by such a driver instead of by a generic CXL PCI driver
>> as for memory expanders. Most of such setup needs to use current CXL core code
>> and therefore needs to be accessible to those vendor drivers. This is accomplished
>> exporting opaque CXL structs and adding and exporting functions for working with
>> those structs indirectly.
>>
>> Some of the patches are based on a patchset sent by Dan Williams [1] which was just
>> partially integrated, most related to making things ready for Type2 but none
>> related to specific Type2 support. Those patches based on Dan´s work have Dan´s
>> signing as co-developer, and a link to the original patch.
>>
>> A final note about CXL.cache is needed. This patchset does not cover it at all,
>> although the emulated Type2 device advertises it. From the kernel point of view
>> supporting CXL.cache will imply to be sure the CXL path supports what the Type2
>> device needs. A device accelerator will likely be connected to a Root Switch,
>> but other configurations can not be discarded. Therefore the kernel will need to
>> check not just HPA, DPA, interleave and granularity, but also the available
>> CXL.cache support and resources in each switch in the CXL path to the Type2
>> device. I expect to contribute to this support in the following months, and
>> it would be good to discuss about it when possible.
>>
>> [1] https://lore.kernel.org/linux-cxl/98b1f61a-e6c2-71d4-c368-50d958501b0c@intel.com/T/
>>
>> Alejandro Lucero (21):
>>    cxl/mem: Arrange for always-synchronous memdev attach
>>    cxl/port: Arrange for always synchronous endpoint attach
>>    cxl: Add type2 device basic support
>>    sfc: add cxl support
>>    cxl: Move pci generic code
>>    cxl: allow Type2 drivers to map cxl component regs
>>    cxl: Support dpa initialization without a mailbox
>>    cxl: Prepare memdev creation for type2
>>    sfc: create type2 cxl memdev
>>    cxl: Define a driver interface for HPA free space enumeration
>>    sfc: get root decoder
>>    cxl: Define a driver interface for DPA allocation
>>    sfc: get endpoint decoder
>>    cxl: Make region type based on endpoint type
>>    cxl/region: Factor out interleave ways setup
>>    cxl/region: Factor out interleave granularity setup
>>    cxl: Allow region creation by type2 drivers
>>    cxl: Avoid dax creation for accelerators
>>    sfc: create cxl region
>>    cxl: Add function for obtaining region range
>>    sfc: support pio mapping based on cxl
>>
>> Dan Williams (1):
>>    cxl/mem: Introduce a memdev creation ->probe() operation
>>
>>   drivers/cxl/Kconfig                   |   2 +-
>>   drivers/cxl/core/core.h               |   9 +-
>>   drivers/cxl/core/hdm.c                |  85 ++++++
>>   drivers/cxl/core/mbox.c               |  63 +---
>>   drivers/cxl/core/memdev.c             | 209 +++++++++----
>>   drivers/cxl/core/pci.c                |  63 ++++
>>   drivers/cxl/core/port.c               |   1 +
>>   drivers/cxl/core/region.c             | 418 +++++++++++++++++++++++---
>>   drivers/cxl/core/regs.c               |   2 +-
>>   drivers/cxl/cxl.h                     | 125 +-------
>>   drivers/cxl/cxlmem.h                  |  90 +-----
>>   drivers/cxl/cxlpci.h                  |  21 +-
>>   drivers/cxl/mem.c                     | 146 +++++----
>>   drivers/cxl/pci.c                     |  88 +-----
>>   drivers/cxl/port.c                    |  46 ++-
>>   drivers/cxl/private.h                 |  17 ++
>>   drivers/net/ethernet/sfc/Kconfig      |  10 +
>>   drivers/net/ethernet/sfc/Makefile     |   1 +
>>   drivers/net/ethernet/sfc/ef10.c       |  50 ++-
>>   drivers/net/ethernet/sfc/efx.c        |  15 +-
>>   drivers/net/ethernet/sfc/efx.h        |   1 -
>>   drivers/net/ethernet/sfc/efx_cxl.c    | 165 ++++++++++
>>   drivers/net/ethernet/sfc/efx_cxl.h    |  40 +++
>>   drivers/net/ethernet/sfc/net_driver.h |  12 +
>>   drivers/net/ethernet/sfc/nic.h        |   3 +
>>   include/cxl/cxl.h                     | 291 ++++++++++++++++++
>>   include/cxl/pci.h                     |  21 ++
>>   tools/testing/cxl/Kbuild              |   1 -
>>   tools/testing/cxl/test/mem.c          |   5 +-
>>   tools/testing/cxl/test/mock.c         |  17 --
>>   30 files changed, 1476 insertions(+), 541 deletions(-)
>>   create mode 100644 drivers/cxl/private.h
>>   create mode 100644 drivers/net/ethernet/sfc/efx_cxl.c
>>   create mode 100644 drivers/net/ethernet/sfc/efx_cxl.h
>>   create mode 100644 include/cxl/cxl.h
>>   create mode 100644 include/cxl/pci.h
>>
>>
>> base-commit: f11a5f89910a7ae970fbce4fdc02d86a8ba8570f
>> prerequisite-patch-id: 44c914dd079e40d716f3f2d91653247eca731594
>> prerequisite-patch-id: b13ca5c11c44a736563477d67b1dceadfe3ea19e
>> prerequisite-patch-id: d0d82965bbea8a2b5ea2f763f19de4dfaa8479c3
>> prerequisite-patch-id: dd0f24b3bdb938f2f123bc26b31cd5fe659e05eb
>> prerequisite-patch-id: 2ea41ec399f2360a84e86e97a8f940a62561931a
>> prerequisite-patch-id: 367b61b5a313db6324f9cf917d46df580f3bbd3b
>> prerequisite-patch-id: 1805332a9f191bc3547927d96de5926356dac03c
>> prerequisite-patch-id: 40657fd517f8e835a091c07e93d6abc08f85d395
>> prerequisite-patch-id: 901eb0d91816499446964b2a9089db59656da08d
>> prerequisite-patch-id: 79856c0199d6872fd2f76a5829dba7fa46f225d6
>> prerequisite-patch-id: 6f3503e59a3d745e5ecff4aaed668e2d32da7e4b
>> prerequisite-patch-id: e9dc88f1b91dce5dc3d46ff2b5bf184aba06439d
>> prerequisite-patch-id: 196fe106100aad619d5be7266959bbeef29b7c8b
>> prerequisite-patch-id: 7e719ed404f664ee8d9b98d56f58326f55ea2175
>> prerequisite-patch-id: 560f95992e13a08279034d5f77aacc9e971332dd
>> prerequisite-patch-id: 8656445ee654056695ff2894e28c8f1014df919e
>> prerequisite-patch-id: 001d831149eb8f9ae17b394e4bcd06d844dd39d9
>> prerequisite-patch-id: 421368aa5eac2af63ef2dc427af2ec11ad45c925
>> prerequisite-patch-id: 18fd00d4743711d835ad546cfbb558d9f97dcdfc
>> prerequisite-patch-id: d89bf9e6d3ea5d332ec2c8e441f1fe6d84e726d3
>> prerequisite-patch-id: 3a6953d11b803abeb437558f3893a3b6a08acdbb
>> prerequisite-patch-id: 0dd42a82e73765950bd069d421d555ded8bfeb25
>> prerequisite-patch-id: da6e0df31ad0d5a945e0a0d29204ba75f0c97344
>

Re: [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration

[Alejandro Lucero Palau]

On 10/9/25 21:55, Cheatham, Benjamin wrote:
> On 10/6/2025 5:01 AM, alejandro.lucero-palau@amd.com wrote:
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> CXL region creation involves allocating capacity from Device Physical Address
>> (DPA) and assigning it to decode a given Host Physical Address (HPA). Before
>> determining how much DPA to allocate the amount of available HPA must be
>> determined. Also, not all HPA is created equal, some HPA targets RAM, some
>> targets PMEM, some is prepared for device-memory flows like HDM-D and HDM-DB,
>> and some is HDM-H (host-only).
>>
>> In order to support Type2 CXL devices, wrap all of those concerns into
>> an API that retrieves a root decoder (platform CXL window) that fits the
>> specified constraints and the capacity available for a new region.
>>
>> Add a complementary function for releasing the reference to such root
>> decoder.
>>
>> Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
> Hey Alejandro,


Hi Ben,


> I've been testing this on my setup and noticed a few issues when BIOS sets up the HDM decoders. It came down to 2 issues:
> 	1) Enabling "Specific Purpose Memory" added a Soft Reserve resource below the CXL window resource and broke
> 	this patch (more below)


Maybe we should talk (first) about this internally as it is about AMD 
BIOS (I guess). I have been talking with the BIOS team about this 
EFI_MEMORY_SP vs EFI_RESERVED_MEMORY, and I'm afraid the discussion is 
not over yet :-).


> 	2) The endpoint decoder was already set up which broke DPA allocation and then CXL region creation (see my response
> 	to patch 18/22 for fix and explanation)


Yes, if the BIOS configures the device HDM decoder, the current patchset 
does not do the right thing. As I said in the cover letter, my 
expectation at the time, and hopefully in the future as well, although 
I'm not sure about it, was the BIOS not doing so. Most of the 
implementation is based on QEMU, so I found this problem when dealing 
with a real system with a Type2 aware BIOS ... . I was tempted to 
include support for this case, but I did not do so for several reasons:


1) I want to think the patchset is close to being accepted and changes 
at this state could delay it further. After more than a year, and 
because this patchset is about "initial Type2 support", I think it is 
better to do so in a follow-up work, and when there is a client 
requiring it.

2) Because that conversation with BIOS guys, I prefer to be sure what to 
do, as there are other things we need to clarify and in my opinion, far 
more important than current Type2 support.

3) CXL is a fast moving part of the kernel and I bet we will find 
another case which the current patchset is not dealing with properly. In 
fact there is another report of devices with the BAR with CXL 
information being also used by the driver for other purposes and 
existing a problem when mapping the CXL registers after the driver did 
map the whole BAR.


So, I think the current patchset where most of the API for Type2 drivers 
is implemented should go as soon as possible, which will facilitate 
those follow-up works for the case you describe and the other one about 
BAR mappings. If not, even if retirement is still far away for me, I'll 
be concerned about the impending future of this work ... but of course, 
this is my suggestion, so let's see other opinions about it.


Thank you.


>
> The fix I did for 1 is a bit hacky but it's essentially checking none of the resources below the CXL window are onlined as
> system memory. It's roughly equivalent to what's being done in the CXL_PARTMODE_RAM case of cxl_region_probe(), but
> I'm restricting the resources to "Soft Reserved" to be safe.
>
> The diff for 2 is pretty big. If you don't want to take it at this point I can send it as a follow up. In that case I'd definitely
> add that auto regions won't work in at least the cover letter (and in the description of 18/22 as well?).
>
> ---
>
> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> index acaca64764bf..2d60131edff3 100644
> --- a/drivers/cxl/core/region.c
> +++ b/drivers/cxl/core/region.c
> @@ -784,6 +784,19 @@ static int find_max_hpa(struct device *dev, void *data)
>          lockdep_assert_held_read(&cxl_rwsem.region);
>          res = cxlrd->res->child;
>
> +       /*
> +        * BIOS may have marked the CXL window as soft reserved. Make sure it's
> +        * free to use.
> +        */
> +       while (res && resource_size(res) == resource_size(cxlrd->res)) {
> +               if ((res->flags & IORESOURCE_BUSY) ||
> +                   (res->flags & IORESOURCE_SYSRAM) ||
> +                   strcmp(res->name, "Soft Reserved") != 0)
> +                       return 0;
> +
> +               res = res->child;
> +       }
> +
>          /* With no resource child the whole parent resource is available */
>          if (!res)
>                  max = resource_size(cxlrd->res);

Re: [PATCH v19 00/22] Type2 device basic support

[Dave Jiang]

On 10/10/25 3:39 AM, Alejandro Lucero Palau wrote:
> Hi Dave,
> 
> On 10/8/25 00:41, Dave Jiang wrote:
>> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>>
>>
>> On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
>>> From: Alejandro Lucero <alucerop@amd.com>
>>>
>>> The patchset should be applied on the described base commit then applying
>>> Terry's v11 about CXL error handling. The first 3 patches come from Dan's
>>> for-6.18/cxl-probe-order branch.
>> I Alejandro, I can't seem to apply with this instruction:
>>
>> ✔ ~/git/kernel-review [review L|…9]
>> 16:35 $ git reset --hard f11a5f89910a7ae970fbce4fdc02d86a8ba8570f
>> HEAD is now at f11a5f89910a Documentation/ABI/testing/debugfs-cxl: Add 'cxl' to clear_poison path
>> ✔ ~/git/kernel-review [review L|…9]
>> 16:35 $ b4 shazam https://lore.kernel.org/linux-cxl/20251006100130.2623388-1-alejandro.lucero-palau@amd.com/T/#m712c7d01ffc7350d9ef638b932b9693a96fe47a9
>> Grabbing thread from lore.kernel.org/all/20251006100130.2623388-1-alejandro.lucero-palau@amd.com/t.mbox.gz
>> Checking for newer revisions
>> Grabbing search results from lore.kernel.org
>> Analyzing 33 messages in the thread
>> Analyzing 620 code-review messages
>> Checking attestation on all messages, may take a moment...
>> ---
>>    ✓ [PATCH v19 1/22] cxl/mem: Arrange for always-synchronous memdev attach
>>    ✓ [PATCH v19 2/22] cxl/port: Arrange for always synchronous endpoint attach
>>    ✓ [PATCH v19 3/22] cxl/mem: Introduce a memdev creation ->probe() operation
>>    ✓ [PATCH v19 4/22] cxl: Add type2 device basic support
>>    ✓ [PATCH v19 5/22] sfc: add cxl support
>>    ✓ [PATCH v19 6/22] cxl: Move pci generic code
>>    ✓ [PATCH v19 7/22] cxl: allow Type2 drivers to map cxl component regs
>>    ✓ [PATCH v19 8/22] cxl: Support dpa initialization without a mailbox
>>    ✓ [PATCH v19 9/22] cxl: Prepare memdev creation for type2
>>    ✓ [PATCH v19 10/22] sfc: create type2 cxl memdev
>>    ✓ [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration
>>    ✓ [PATCH v19 12/22] sfc: get root decoder
>>    ✓ [PATCH v19 13/22] cxl: Define a driver interface for DPA allocation
>>      + Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
>>    ✓ [PATCH v19 14/22] sfc: get endpoint decoder
>>    ✓ [PATCH v19 15/22] cxl: Make region type based on endpoint type
>>      + Reviewed-by: Davidlohr Bueso <dave@stgolabs.net> (✓ DKIM/stgolabs.net)
>>    ✓ [PATCH v19 16/22] cxl/region: Factor out interleave ways setup
>>    ✓ [PATCH v19 17/22] cxl/region: Factor out interleave granularity setup
>>    ✓ [PATCH v19 18/22] cxl: Allow region creation by type2 drivers
>>    ✓ [PATCH v19 19/22] cxl: Avoid dax creation for accelerators
>>    ✓ [PATCH v19 20/22] sfc: create cxl region
>>    ✓ [PATCH v19 21/22] cxl: Add function for obtaining region range
>>    ✓ [PATCH v19 22/22] sfc: support pio mapping based on cxl
>>    ---
>>    ✓ Signed: DKIM/amd.com
>>    ---
>>    NOTE: install patatt for end-to-end signature verification
>> ---
>> Total patches: 22
>> ---
>>   Deps: looking for dependencies matching 23 patch-ids
>>   Deps: Applying prerequisite patch: [PATCH v11 01/23] cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
>>   Deps: Applying prerequisite patch: [PATCH v11 02/23] CXL/AER: Remove CONFIG_PCIEAER_CXL and replace with CONFIG_CXL_RAS
>>   Deps: Applying prerequisite patch: [PATCH v11 03/23] cxl/pci: Remove unnecessary CXL Endpoint handling helper functions
>>   Deps: Applying prerequisite patch: [PATCH v11 04/23] cxl/pci: Remove unnecessary CXL RCH handling helper functions
>>   Deps: Applying prerequisite patch: [PATCH v11 05/23] cxl: Move CXL driver RCH error handling into CONFIG_CXL_RCH_RAS conditional block
>>   Deps: Applying prerequisite patch: [PATCH v11 06/23] CXL/AER: Introduce rch_aer.c into AER driver for handling CXL RCH errors
>>   Deps: Applying prerequisite patch: [PATCH v11 08/23] PCI/CXL: Introduce pcie_is_cxl()
>>   Deps: Applying prerequisite patch: [PATCH v11 09/23] PCI/AER: Report CXL or PCIe bus error type in trace logging
>>   Deps: Applying prerequisite patch: [PATCH v11 10/23] CXL/AER: Update PCI class code check to use FIELD_GET()
>>   Deps: Applying prerequisite patch: [PATCH v11 11/23] cxl/pci: Update RAS handler interfaces to also support CXL Ports
>>   Deps: Applying prerequisite patch: [PATCH v12 12/25] cxl/pci: Log message if RAS registers are unmapped
>>   Deps: Applying prerequisite patch: [PATCH v11 13/23] cxl/pci: Unify CXL trace logging for CXL Endpoints and CXL Ports
>>   Deps: Applying prerequisite patch: [PATCH v12 14/25] cxl/pci: Update cxl_handle_cor_ras() to return early if no RAS errors
>>   Deps: Applying prerequisite patch: [PATCH v11 15/23] cxl/pci: Map CXL Endpoint Port and CXL Switch Port RAS registers
>>   Deps: Applying prerequisite patch: [PATCH v11 17/23] CXL/AER: Introduce cxl_aer.c into AER driver for forwarding CXL errors
>>   Deps: Applying prerequisite patch: [PATCH v11 18/23] PCI/AER: Dequeue forwarded CXL error
>>   Deps: Applying prerequisite patch: [PATCH v11 19/23] CXL/PCI: Introduce CXL Port protocol error handlers
>>   Deps: Applying prerequisite patch: [PATCH v11 20/23] CXL/PCI: Export and rename merge_result() to pci_ers_merge_result()
>>   Deps: Applying prerequisite patch: [PATCH v11 21/23] CXL/PCI: Introduce CXL uncorrectable protocol error recovery
>>   Deps: Applying prerequisite patch: [PATCH v11 22/23] CXL/PCI: Enable CXL protocol errors during CXL Port probe
>>   Deps: Applying prerequisite patch: [PATCH v11 23/23] CXL/PCI: Disable CXL protocol error interrupts during CXL Port cleanup
>> Applying: cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
>> Applying: CXL/AER: Remove CONFIG_PCIEAER_CXL and replace with CONFIG_CXL_RAS
>> Applying: cxl/pci: Remove unnecessary CXL Endpoint handling helper functions
>> Applying: cxl/pci: Remove unnecessary CXL RCH handling helper functions
>> Applying: cxl: Move CXL driver RCH error handling into CONFIG_CXL_RCH_RAS conditional block
>> Applying: CXL/AER: Introduce rch_aer.c into AER driver for handling CXL RCH errors
>> Applying: PCI/CXL: Introduce pcie_is_cxl()
>> Patch failed at 0007 PCI/CXL: Introduce pcie_is_cxl()
>> error: patch failed: include/uapi/linux/pci_regs.h:1274
>> error: include/uapi/linux/pci_regs.h: patch does not apply
>> hint: Use 'git am --show-current-patch=diff' to see the failed patch
>> hint: When you have resolved this problem, run "git am --continue".
>> hint: If you prefer to skip this patch, run "git am --skip" instead.
>> hint: To restore the original branch and stop patching, run "git am --abort".
>> hint: Disable this message with "git config set advice.mergeConflict false"
>>
>> I also tried applying Terry's v11 first (which applied) and then this series failed as well.
> 
> 
> You need to apply Terry's v11 patches for sure on the commit base. If not the v19 series can not be applied cleanly.
> 
> 
> I have tried this again a couple of times and it works for me:
> 
> 
> 1)  git reset --hard f11a5f89910a
> 
> 2) git am Enable-CXL-PCIe-Port-Protocol-Error-handling-and-logging.patch
> 
> 3) git am Type2-device-basic-support.patch

1) $ git reset --hard f11a5f89910a
2) $ b4 shazam -v11 https://lore.kernel.org/linux-cxl/20250827013539.903682-1-terry.bowman@amd.com/T/#m9c6963513137b67d281414e88b57fe4f346bedab
(success)
3) $ b4 shazam https://lore.kernel.org/linux-cxl/0b41e061-53e1-49ef-9f24-01e01143b709@amd.com/T/#m712c7d01ffc7350d9ef638b932b9693a96fe47a9
...
Applying: cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
Patch failed at 0001 cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
error: patch failed: drivers/cxl/Kconfig:233
error: drivers/cxl/Kconfig: patch does not apply
error: patch failed: drivers/cxl/core/Makefile:14
error: drivers/cxl/core/Makefile: patch does not apply
error: patch failed: drivers/cxl/core/core.h:143
error: drivers/cxl/core/core.h: patch does not apply
error: patch failed: drivers/cxl/core/pci.c:6
error: drivers/cxl/core/pci.c: patch does not apply
error: patch failed: drivers/cxl/core/ras.c:5
error: drivers/cxl/core/ras.c: patch does not apply
error: patch failed: drivers/cxl/cxl.h:761
error: drivers/cxl/cxl.h: patch does not apply
error: patch failed: drivers/cxl/cxlpci.h:132
error: drivers/cxl/cxlpci.h: patch does not apply
error: patch failed: tools/testing/cxl/Kbuild:61
error: tools/testing/cxl/Kbuild: patch does not apply
...

DJ

> 
> 
> Hopefully you can reproduce this as well. Maybe Ben Cheatham can comment on this, if he had problems applying them, as it seems he was able to work with it.
> 
> 
> Thank you
> 
> 
>> DJ
>>
>>> v19 changes:
>>>
>>>    Removal of cxl_acquire_endpoint and driver callback for unexpected cxl
>>>    module removal. Dan's patches made them unnecessary.
>>>
>>>    patch 4: remove code already moved by Terry's patches (Ben Cheatham)
>>>
>>>    patch 6: removed unrelated change (Ben Cheatham)
>>>
>>>    patch 7: fix error report inconsistencies (Jonathan, Dave)
>>>
>>>    patch 9: remove unnecessary comment (Ben Cheatham)
>>>
>>>    patch 11: fix __free usage (Jonathan Cameron, Ben Cheatham)
>>>
>>>    patch 13: style fixes (Jonathan Cameron, Dave Jiag)
>>>
>>>    patch 14: move code to previous patch (Jonathan Cameron)
>>>
>>>    patch 18: group code in one locking (Dave Jian)
>>>            use __free helper (Ben Cheatham)
>>>
>>>
>>> v18 changes:
>>>
>>>    patch 1: minor changes and fixing docs generation (Jonathan, Dan)
>>>
>>>    patch4: merged with v17 patch5
>>>
>>>    patch 5: merging v17 patches 6 and 7
>>>
>>>    patch 6: adding helpers for clarity
>>>
>>>    patch 9:
>>>        - minor changes (Dave)
>>>        - simplifying flags check (Dan)
>>>
>>>    patch 10: minor changes (Jonathan)
>>>
>>>    patch 11:
>>>        - minor changes (Dave)
>>>        - fix mess (Jonathan, Dave)
>>>
>>>    patch 18: minor changes (Jonathan, Dan)
>>>
>>> v17 changes: (Dan Williams review)
>>>   - use devm for cxl_dev_state allocation
>>>   - using current cxl struct for checking capability registers found by
>>>     the driver.
>>>   - simplify dpa initialization without a mailbox not supporting pmem
>>>   - add cxl_acquire_endpoint for protection during initialization
>>>   - add callback/action to cxl_create_region for a driver notified about cxl
>>>     core kernel modules removal.
>>>   - add sfc function to disable CXL-based PIO buffers if such a callback
>>>     is invoked.
>>>   - Always manage a Type2 created region as private not allowing DAX.
>>>
>>> v16 changes:
>>>   - rebase against rc4 (Dave Jiang)
>>>   - remove duplicate line (Ben Cheatham)
>>>
>>> v15 changes:
>>>   - remove reference to unused header file (Jonathan Cameron)
>>>   - add proper kernel docs to exported functions (Alison Schofield)
>>>   - using an array to map the enums to strings (Alison Schofield)
>>>   - clarify comment when using bitmap_subset (Jonathan Cameron)
>>>   - specify link to type2 support in all patches (Alison Schofield)
>>>
>>>    Patches changed (minor): 4, 11
>>>
>>> v14 changes:
>>>   - static null initialization of bitmaps (Jonathan Cameron)
>>>   - Fixing cxl tests (Alison Schofield)
>>>   - Fixing robot compilation problems
>>>
>>>    Patches changed (minor): 1, 4, 6, 13
>>>
>>> v13 changes:
>>>   - using names for headers checking more consistent (Jonathan Cameron)
>>>   - using helper for caps bit setting (Jonathan Cameron)
>>>   - provide generic function for reporting missing capabilities (Jonathan Cameron)
>>>   - rename cxl_pci_setup_memdev_regs to cxl_pci_accel_setup_memdev_regs (Jonathan Cameron)
>>>   - cxl_dpa_info size to be set by the Type2 driver (Jonathan Cameron)
>>>   - avoiding rc variable when possible (Jonathan Cameron)
>>>   - fix spelling (Simon Horman)
>>>   - use scoped_guard (Dave Jiang)
>>>   - use enum instead of bool (Dave Jiang)
>>>   - dropping patch with hardware symbols
>>>
>>> v12 changes:
>>>   - use new macro cxl_dev_state_create in pci driver (Ben Cheatham)
>>>   - add public/private sections in now exported cxl_dev_state struct (Ben
>>>     Cheatham)
>>>   - fix cxl/pci.h regarding file name for checking if defined
>>>   - Clarify capabilities found vs expected in error message. (Ben
>>>     Cheatham)
>>>   - Clarify new CXL_DECODER_F flag (Ben Cheatham)
>>>   - Fix changes about cxl memdev creation support moving code to the
>>>     proper patch. (Ben Cheatham)
>>>   - Avoid debug and function duplications (Ben Cheatham)
>>>
>>> v11 changes:
>>>   - Dropping the use of cxl_memdev_state and going back to using
>>>     cxl_dev_state.
>>>   - Using a helper for an accel driver to allocate its own cxl-related
>>>     struct embedding cxl_dev_state.
>>>   - Exporting the required structs in include/cxl/cxl.h for an accel
>>>     driver being able to know the cxl_dev_state size required in the
>>>     previously mentioned helper for allocation.
>>>   - Avoid using any struct for dpa initialization by the accel driver
>>>     adding a specific function for creating dpa partitions by accel
>>>     drivers without a mailbox.
>>>
>>> v10 changes:
>>>   - Using cxl_memdev_state instead of cxl_dev_state for type2 which has a
>>>     memory after all and facilitates the setup.
>>>   - Adapt core for using cxl_memdev_state allowing accel drivers to work
>>>     with them without further awareness of internal cxl structs.
>>>   - Using last DPA changes for creating DPA partitions with accel driver
>>>     hardcoding mds values when no mailbox.
>>>   - capabilities not a new field but built up when current register maps
>>>     is performed and returned to the caller for checking.
>>>   - HPA free space supporting interleaving.
>>>   - DPA free space droping max-min for a simple alloc size.
>>>
>>> v9 changes:
>>>   - adding forward definitions (Jonathan Cameron)
>>>   - using set_bit instead of bitmap_set (Jonathan Cameron)
>>>   - fix rebase problem (Jonathan Cameron)
>>>   - Improve error path (Jonathan Cameron)
>>>   - fix build problems with cxl region dependency (robot)
>>>   - fix error path (Simon Horman)
>>>
>>> v8 changes:
>>>   - Change error path labeling inside sfc cxl code (Edward Cree)
>>>   - Properly handling checks and error in sfc cxl code (Simon Horman)
>>>   - Fix bug when checking resource_size (Simon Horman)
>>>   - Avoid bisect problems reordering patches (Edward Cree)
>>>   - Fix buffer allocation size in sfc (Simon Horman)
>>>
>>> v7 changes:
>>>
>>>   - fixing kernel test robot complains
>>>   - fix type with Type3 mandatory capabilities (Zhi Wang)
>>>   - optimize code in cxl_request_resource (Kalesh Anakkur Purayil)
>>>   - add sanity check when dealing with resources arithmetics (Fan Ni)
>>>   - fix typos and blank lines (Fan Ni)
>>>   - keep previous log errors/warnings in sfc driver (Martin Habets)
>>>   - add WARN_ON_ONCE if region given is NULL
>>>
>>> v6 changes:
>>>
>>>   - update sfc mcdi_pcol.h with full hardware changes most not related to
>>>     this patchset. This is an automatic file created from hardware design
>>>     changes and not touched by software. It is updated from time to time
>>>     and it required update for the sfc driver CXL support.
>>>   - remove CXL capabilities definitions not used by the patchset or
>>>     previous kernel code. (Dave Jiang, Jonathan Cameron)
>>>   - Use bitmap_subset instead of reinventing the wheel ... (Ben Cheatham)
>>>   - Use cxl_accel_memdev for new device_type created (Ben Cheatham)
>>>   - Fix construct_region use of rwsem (Zhi Wang)
>>>   - Obtain region range instead of region params (Allison Schofield, Dave
>>>     Jiang)
>>>
>>> v5 changes:
>>>
>>>   - Fix SFC configuration based on kernel CXL configuration
>>>   - Add subset check for capabilities.
>>>   - fix region creation when HDM decoders programmed by firmware/BIOS (Ben
>>>     Cheatham)
>>>   - Add option for creating dax region based on driver decission (Ben
>>>     Cheatham)
>>>   - Using sfc probe_data struct for keeping sfc cxl data
>>>
>>> v4 changes:
>>>
>>>   - Use bitmap for capabilities new field (Jonathan Cameron)
>>>   - Use cxl_mem attributes for sysfs based on device type (Dave Jian)
>>>   - Add conditional cxl sfc compilation relying on kernel CXL config (kernel test robot)
>>>   - Add sfc changes in different patches for facilitating backport (Jonathan Cameron)
>>>   - Remove patch for dealing with cxl modules dependencies and using sfc kconfig plus
>>>     MODULE_SOFTDEP instead.
>>>
>>> v3 changes:
>>>
>>>   - cxl_dev_state not defined as opaque but only manipulated by accel drivers
>>>     through accessors.
>>>   - accessors names not identified as only for accel drivers.
>>>   - move pci code from pci driver (drivers/cxl/pci.c) to generic pci code
>>>     (drivers/cxl/core/pci.c).
>>>   - capabilities field from u8 to u32 and initialised by CXL regs discovering
>>>     code.
>>>   - add capabilities check and removing current check by CXL regs discovering
>>>     code.
>>>   - Not fail if CXL Device Registers not found. Not mandatory for Type2.
>>>   - add timeout in acquire_endpoint for solving a race with the endpoint port
>>>     creation.
>>>   - handle EPROBE_DEFER by sfc driver.
>>>   - Limiting interleave ways to 1 for accel driver HPA/DPA requests.
>>>   - factoring out interleave ways and granularity helpers from type2 region
>>>     creation patch.
>>>   - restricting region_creation for type2 to one endpoint decoder.
>>>
>>> v2 changes:
>>>
>>> I have removed the introduction about the concerns with BIOS/UEFI after the
>>> discussion leading to confirm the need of the functionality implemented, at
>>> least is some scenarios.
>>>
>>> There are two main changes from the RFC:
>>>
>>> 1) Following concerns about drivers using CXL core without restrictions, the CXL
>>> struct to work with is opaque to those drivers, therefore functions are
>>> implemented for modifying or reading those structs indirectly.
>>>
>>> 2) The driver for using the added functionality is not a test driver but a real
>>> one: the SFC ethernet network driver. It uses the CXL region mapped for PIO
>>> buffers instead of regions inside PCIe BARs.
>>>
>>> RFC:
>>>
>>> Current CXL kernel code is focused on supporting Type3 CXL devices, aka memory
>>> expanders. Type2 CXL devices, aka device accelerators, share some functionalities
>>> but require some special handling.
>>>
>>> First of all, Type2 are by definition specific to drivers doing something and not just
>>> a memory expander, so it is expected to work with the CXL specifics. This implies the CXL
>>> setup needs to be done by such a driver instead of by a generic CXL PCI driver
>>> as for memory expanders. Most of such setup needs to use current CXL core code
>>> and therefore needs to be accessible to those vendor drivers. This is accomplished
>>> exporting opaque CXL structs and adding and exporting functions for working with
>>> those structs indirectly.
>>>
>>> Some of the patches are based on a patchset sent by Dan Williams [1] which was just
>>> partially integrated, most related to making things ready for Type2 but none
>>> related to specific Type2 support. Those patches based on Dan´s work have Dan´s
>>> signing as co-developer, and a link to the original patch.
>>>
>>> A final note about CXL.cache is needed. This patchset does not cover it at all,
>>> although the emulated Type2 device advertises it. From the kernel point of view
>>> supporting CXL.cache will imply to be sure the CXL path supports what the Type2
>>> device needs. A device accelerator will likely be connected to a Root Switch,
>>> but other configurations can not be discarded. Therefore the kernel will need to
>>> check not just HPA, DPA, interleave and granularity, but also the available
>>> CXL.cache support and resources in each switch in the CXL path to the Type2
>>> device. I expect to contribute to this support in the following months, and
>>> it would be good to discuss about it when possible.
>>>
>>> [1] https://lore.kernel.org/linux-cxl/98b1f61a-e6c2-71d4-c368-50d958501b0c@intel.com/T/
>>>
>>> Alejandro Lucero (21):
>>>    cxl/mem: Arrange for always-synchronous memdev attach
>>>    cxl/port: Arrange for always synchronous endpoint attach
>>>    cxl: Add type2 device basic support
>>>    sfc: add cxl support
>>>    cxl: Move pci generic code
>>>    cxl: allow Type2 drivers to map cxl component regs
>>>    cxl: Support dpa initialization without a mailbox
>>>    cxl: Prepare memdev creation for type2
>>>    sfc: create type2 cxl memdev
>>>    cxl: Define a driver interface for HPA free space enumeration
>>>    sfc: get root decoder
>>>    cxl: Define a driver interface for DPA allocation
>>>    sfc: get endpoint decoder
>>>    cxl: Make region type based on endpoint type
>>>    cxl/region: Factor out interleave ways setup
>>>    cxl/region: Factor out interleave granularity setup
>>>    cxl: Allow region creation by type2 drivers
>>>    cxl: Avoid dax creation for accelerators
>>>    sfc: create cxl region
>>>    cxl: Add function for obtaining region range
>>>    sfc: support pio mapping based on cxl
>>>
>>> Dan Williams (1):
>>>    cxl/mem: Introduce a memdev creation ->probe() operation
>>>
>>>   drivers/cxl/Kconfig                   |   2 +-
>>>   drivers/cxl/core/core.h               |   9 +-
>>>   drivers/cxl/core/hdm.c                |  85 ++++++
>>>   drivers/cxl/core/mbox.c               |  63 +---
>>>   drivers/cxl/core/memdev.c             | 209 +++++++++----
>>>   drivers/cxl/core/pci.c                |  63 ++++
>>>   drivers/cxl/core/port.c               |   1 +
>>>   drivers/cxl/core/region.c             | 418 +++++++++++++++++++++++---
>>>   drivers/cxl/core/regs.c               |   2 +-
>>>   drivers/cxl/cxl.h                     | 125 +-------
>>>   drivers/cxl/cxlmem.h                  |  90 +-----
>>>   drivers/cxl/cxlpci.h                  |  21 +-
>>>   drivers/cxl/mem.c                     | 146 +++++----
>>>   drivers/cxl/pci.c                     |  88 +-----
>>>   drivers/cxl/port.c                    |  46 ++-
>>>   drivers/cxl/private.h                 |  17 ++
>>>   drivers/net/ethernet/sfc/Kconfig      |  10 +
>>>   drivers/net/ethernet/sfc/Makefile     |   1 +
>>>   drivers/net/ethernet/sfc/ef10.c       |  50 ++-
>>>   drivers/net/ethernet/sfc/efx.c        |  15 +-
>>>   drivers/net/ethernet/sfc/efx.h        |   1 -
>>>   drivers/net/ethernet/sfc/efx_cxl.c    | 165 ++++++++++
>>>   drivers/net/ethernet/sfc/efx_cxl.h    |  40 +++
>>>   drivers/net/ethernet/sfc/net_driver.h |  12 +
>>>   drivers/net/ethernet/sfc/nic.h        |   3 +
>>>   include/cxl/cxl.h                     | 291 ++++++++++++++++++
>>>   include/cxl/pci.h                     |  21 ++
>>>   tools/testing/cxl/Kbuild              |   1 -
>>>   tools/testing/cxl/test/mem.c          |   5 +-
>>>   tools/testing/cxl/test/mock.c         |  17 --
>>>   30 files changed, 1476 insertions(+), 541 deletions(-)
>>>   create mode 100644 drivers/cxl/private.h
>>>   create mode 100644 drivers/net/ethernet/sfc/efx_cxl.c
>>>   create mode 100644 drivers/net/ethernet/sfc/efx_cxl.h
>>>   create mode 100644 include/cxl/cxl.h
>>>   create mode 100644 include/cxl/pci.h
>>>
>>>
>>> base-commit: f11a5f89910a7ae970fbce4fdc02d86a8ba8570f
>>> prerequisite-patch-id: 44c914dd079e40d716f3f2d91653247eca731594
>>> prerequisite-patch-id: b13ca5c11c44a736563477d67b1dceadfe3ea19e
>>> prerequisite-patch-id: d0d82965bbea8a2b5ea2f763f19de4dfaa8479c3
>>> prerequisite-patch-id: dd0f24b3bdb938f2f123bc26b31cd5fe659e05eb
>>> prerequisite-patch-id: 2ea41ec399f2360a84e86e97a8f940a62561931a
>>> prerequisite-patch-id: 367b61b5a313db6324f9cf917d46df580f3bbd3b
>>> prerequisite-patch-id: 1805332a9f191bc3547927d96de5926356dac03c
>>> prerequisite-patch-id: 40657fd517f8e835a091c07e93d6abc08f85d395
>>> prerequisite-patch-id: 901eb0d91816499446964b2a9089db59656da08d
>>> prerequisite-patch-id: 79856c0199d6872fd2f76a5829dba7fa46f225d6
>>> prerequisite-patch-id: 6f3503e59a3d745e5ecff4aaed668e2d32da7e4b
>>> prerequisite-patch-id: e9dc88f1b91dce5dc3d46ff2b5bf184aba06439d
>>> prerequisite-patch-id: 196fe106100aad619d5be7266959bbeef29b7c8b
>>> prerequisite-patch-id: 7e719ed404f664ee8d9b98d56f58326f55ea2175
>>> prerequisite-patch-id: 560f95992e13a08279034d5f77aacc9e971332dd
>>> prerequisite-patch-id: 8656445ee654056695ff2894e28c8f1014df919e
>>> prerequisite-patch-id: 001d831149eb8f9ae17b394e4bcd06d844dd39d9
>>> prerequisite-patch-id: 421368aa5eac2af63ef2dc427af2ec11ad45c925
>>> prerequisite-patch-id: 18fd00d4743711d835ad546cfbb558d9f97dcdfc
>>> prerequisite-patch-id: d89bf9e6d3ea5d332ec2c8e441f1fe6d84e726d3
>>> prerequisite-patch-id: 3a6953d11b803abeb437558f3893a3b6a08acdbb
>>> prerequisite-patch-id: 0dd42a82e73765950bd069d421d555ded8bfeb25
>>> prerequisite-patch-id: da6e0df31ad0d5a945e0a0d29204ba75f0c97344
>>

Re: [PATCH v19 00/22] Type2 device basic support

[Dave Jiang]

On 10/10/25 8:57 AM, Dave Jiang wrote:
> 
> 
> On 10/10/25 3:39 AM, Alejandro Lucero Palau wrote:
>> Hi Dave,
>>
>> On 10/8/25 00:41, Dave Jiang wrote:
>>> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>>>
>>>
>>> On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
>>>> From: Alejandro Lucero <alucerop@amd.com>
>>>>
>>>> The patchset should be applied on the described base commit then applying
>>>> Terry's v11 about CXL error handling. The first 3 patches come from Dan's
>>>> for-6.18/cxl-probe-order branch.
>>> I Alejandro, I can't seem to apply with this instruction:
>>>
>>> ✔ ~/git/kernel-review [review L|…9]
>>> 16:35 $ git reset --hard f11a5f89910a7ae970fbce4fdc02d86a8ba8570f
>>> HEAD is now at f11a5f89910a Documentation/ABI/testing/debugfs-cxl: Add 'cxl' to clear_poison path
>>> ✔ ~/git/kernel-review [review L|…9]
>>> 16:35 $ b4 shazam https://lore.kernel.org/linux-cxl/20251006100130.2623388-1-alejandro.lucero-palau@amd.com/T/#m712c7d01ffc7350d9ef638b932b9693a96fe47a9
>>> Grabbing thread from lore.kernel.org/all/20251006100130.2623388-1-alejandro.lucero-palau@amd.com/t.mbox.gz
>>> Checking for newer revisions
>>> Grabbing search results from lore.kernel.org
>>> Analyzing 33 messages in the thread
>>> Analyzing 620 code-review messages
>>> Checking attestation on all messages, may take a moment...
>>> ---
>>>    ✓ [PATCH v19 1/22] cxl/mem: Arrange for always-synchronous memdev attach
>>>    ✓ [PATCH v19 2/22] cxl/port: Arrange for always synchronous endpoint attach
>>>    ✓ [PATCH v19 3/22] cxl/mem: Introduce a memdev creation ->probe() operation
>>>    ✓ [PATCH v19 4/22] cxl: Add type2 device basic support
>>>    ✓ [PATCH v19 5/22] sfc: add cxl support
>>>    ✓ [PATCH v19 6/22] cxl: Move pci generic code
>>>    ✓ [PATCH v19 7/22] cxl: allow Type2 drivers to map cxl component regs
>>>    ✓ [PATCH v19 8/22] cxl: Support dpa initialization without a mailbox
>>>    ✓ [PATCH v19 9/22] cxl: Prepare memdev creation for type2
>>>    ✓ [PATCH v19 10/22] sfc: create type2 cxl memdev
>>>    ✓ [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration
>>>    ✓ [PATCH v19 12/22] sfc: get root decoder
>>>    ✓ [PATCH v19 13/22] cxl: Define a driver interface for DPA allocation
>>>      + Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
>>>    ✓ [PATCH v19 14/22] sfc: get endpoint decoder
>>>    ✓ [PATCH v19 15/22] cxl: Make region type based on endpoint type
>>>      + Reviewed-by: Davidlohr Bueso <dave@stgolabs.net> (✓ DKIM/stgolabs.net)
>>>    ✓ [PATCH v19 16/22] cxl/region: Factor out interleave ways setup
>>>    ✓ [PATCH v19 17/22] cxl/region: Factor out interleave granularity setup
>>>    ✓ [PATCH v19 18/22] cxl: Allow region creation by type2 drivers
>>>    ✓ [PATCH v19 19/22] cxl: Avoid dax creation for accelerators
>>>    ✓ [PATCH v19 20/22] sfc: create cxl region
>>>    ✓ [PATCH v19 21/22] cxl: Add function for obtaining region range
>>>    ✓ [PATCH v19 22/22] sfc: support pio mapping based on cxl
>>>    ---
>>>    ✓ Signed: DKIM/amd.com
>>>    ---
>>>    NOTE: install patatt for end-to-end signature verification
>>> ---
>>> Total patches: 22
>>> ---
>>>   Deps: looking for dependencies matching 23 patch-ids
>>>   Deps: Applying prerequisite patch: [PATCH v11 01/23] cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
>>>   Deps: Applying prerequisite patch: [PATCH v11 02/23] CXL/AER: Remove CONFIG_PCIEAER_CXL and replace with CONFIG_CXL_RAS
>>>   Deps: Applying prerequisite patch: [PATCH v11 03/23] cxl/pci: Remove unnecessary CXL Endpoint handling helper functions
>>>   Deps: Applying prerequisite patch: [PATCH v11 04/23] cxl/pci: Remove unnecessary CXL RCH handling helper functions
>>>   Deps: Applying prerequisite patch: [PATCH v11 05/23] cxl: Move CXL driver RCH error handling into CONFIG_CXL_RCH_RAS conditional block
>>>   Deps: Applying prerequisite patch: [PATCH v11 06/23] CXL/AER: Introduce rch_aer.c into AER driver for handling CXL RCH errors
>>>   Deps: Applying prerequisite patch: [PATCH v11 08/23] PCI/CXL: Introduce pcie_is_cxl()
>>>   Deps: Applying prerequisite patch: [PATCH v11 09/23] PCI/AER: Report CXL or PCIe bus error type in trace logging
>>>   Deps: Applying prerequisite patch: [PATCH v11 10/23] CXL/AER: Update PCI class code check to use FIELD_GET()
>>>   Deps: Applying prerequisite patch: [PATCH v11 11/23] cxl/pci: Update RAS handler interfaces to also support CXL Ports
>>>   Deps: Applying prerequisite patch: [PATCH v12 12/25] cxl/pci: Log message if RAS registers are unmapped
>>>   Deps: Applying prerequisite patch: [PATCH v11 13/23] cxl/pci: Unify CXL trace logging for CXL Endpoints and CXL Ports
>>>   Deps: Applying prerequisite patch: [PATCH v12 14/25] cxl/pci: Update cxl_handle_cor_ras() to return early if no RAS errors
>>>   Deps: Applying prerequisite patch: [PATCH v11 15/23] cxl/pci: Map CXL Endpoint Port and CXL Switch Port RAS registers
>>>   Deps: Applying prerequisite patch: [PATCH v11 17/23] CXL/AER: Introduce cxl_aer.c into AER driver for forwarding CXL errors
>>>   Deps: Applying prerequisite patch: [PATCH v11 18/23] PCI/AER: Dequeue forwarded CXL error
>>>   Deps: Applying prerequisite patch: [PATCH v11 19/23] CXL/PCI: Introduce CXL Port protocol error handlers
>>>   Deps: Applying prerequisite patch: [PATCH v11 20/23] CXL/PCI: Export and rename merge_result() to pci_ers_merge_result()
>>>   Deps: Applying prerequisite patch: [PATCH v11 21/23] CXL/PCI: Introduce CXL uncorrectable protocol error recovery
>>>   Deps: Applying prerequisite patch: [PATCH v11 22/23] CXL/PCI: Enable CXL protocol errors during CXL Port probe
>>>   Deps: Applying prerequisite patch: [PATCH v11 23/23] CXL/PCI: Disable CXL protocol error interrupts during CXL Port cleanup
>>> Applying: cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
>>> Applying: CXL/AER: Remove CONFIG_PCIEAER_CXL and replace with CONFIG_CXL_RAS
>>> Applying: cxl/pci: Remove unnecessary CXL Endpoint handling helper functions
>>> Applying: cxl/pci: Remove unnecessary CXL RCH handling helper functions
>>> Applying: cxl: Move CXL driver RCH error handling into CONFIG_CXL_RCH_RAS conditional block
>>> Applying: CXL/AER: Introduce rch_aer.c into AER driver for handling CXL RCH errors
>>> Applying: PCI/CXL: Introduce pcie_is_cxl()
>>> Patch failed at 0007 PCI/CXL: Introduce pcie_is_cxl()
>>> error: patch failed: include/uapi/linux/pci_regs.h:1274
>>> error: include/uapi/linux/pci_regs.h: patch does not apply
>>> hint: Use 'git am --show-current-patch=diff' to see the failed patch
>>> hint: When you have resolved this problem, run "git am --continue".
>>> hint: If you prefer to skip this patch, run "git am --skip" instead.
>>> hint: To restore the original branch and stop patching, run "git am --abort".
>>> hint: Disable this message with "git config set advice.mergeConflict false"
>>>
>>> I also tried applying Terry's v11 first (which applied) and then this series failed as well.
>>
>>
>> You need to apply Terry's v11 patches for sure on the commit base. If not the v19 series can not be applied cleanly.
>>
>>
>> I have tried this again a couple of times and it works for me:
>>
>>
>> 1)  git reset --hard f11a5f89910a
>>
>> 2) git am Enable-CXL-PCIe-Port-Protocol-Error-handling-and-logging.patch
>>
>> 3) git am Type2-device-basic-support.patch
> 
> 1) $ git reset --hard f11a5f89910a
> 2) $ b4 shazam -v11 https://lore.kernel.org/linux-cxl/20250827013539.903682-1-terry.bowman@amd.com/T/#m9c6963513137b67d281414e88b57fe4f346bedab
> (success)
> 3) $ b4 shazam https://lore.kernel.org/linux-cxl/0b41e061-53e1-49ef-9f24-01e01143b709@amd.com/T/#m712c7d01ffc7350d9ef638b932b9693a96fe47a9
> ...
> Applying: cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
> Patch failed at 0001 cxl: Remove ifdef blocks of CONFIG_PCIEAER_CXL from core/pci.c
> error: patch failed: drivers/cxl/Kconfig:233
> error: drivers/cxl/Kconfig: patch does not apply
> error: patch failed: drivers/cxl/core/Makefile:14
> error: drivers/cxl/core/Makefile: patch does not apply
> error: patch failed: drivers/cxl/core/core.h:143
> error: drivers/cxl/core/core.h: patch does not apply
> error: patch failed: drivers/cxl/core/pci.c:6
> error: drivers/cxl/core/pci.c: patch does not apply
> error: patch failed: drivers/cxl/core/ras.c:5
> error: drivers/cxl/core/ras.c: patch does not apply
> error: patch failed: drivers/cxl/cxl.h:761
> error: drivers/cxl/cxl.h: patch does not apply
> error: patch failed: drivers/cxl/cxlpci.h:132
> error: drivers/cxl/cxlpci.h: patch does not apply
> error: patch failed: tools/testing/cxl/Kbuild:61
> error: tools/testing/cxl/Kbuild: patch does not apply
> ...
> 
> DJ
> 

Alison is able to apply with an older version of b4. Maybe it's something to do with the newer versions. Anyhow, she pushed a branch internally for me so I can use it for code review.

DJ

>>
>>
>> Hopefully you can reproduce this as well. Maybe Ben Cheatham can comment on this, if he had problems applying them, as it seems he was able to work with it.
>>
>>
>> Thank you
>>
>>
>>> DJ
>>>
>>>> v19 changes:
>>>>
>>>>    Removal of cxl_acquire_endpoint and driver callback for unexpected cxl
>>>>    module removal. Dan's patches made them unnecessary.
>>>>
>>>>    patch 4: remove code already moved by Terry's patches (Ben Cheatham)
>>>>
>>>>    patch 6: removed unrelated change (Ben Cheatham)
>>>>
>>>>    patch 7: fix error report inconsistencies (Jonathan, Dave)
>>>>
>>>>    patch 9: remove unnecessary comment (Ben Cheatham)
>>>>
>>>>    patch 11: fix __free usage (Jonathan Cameron, Ben Cheatham)
>>>>
>>>>    patch 13: style fixes (Jonathan Cameron, Dave Jiag)
>>>>
>>>>    patch 14: move code to previous patch (Jonathan Cameron)
>>>>
>>>>    patch 18: group code in one locking (Dave Jian)
>>>>            use __free helper (Ben Cheatham)
>>>>
>>>>
>>>> v18 changes:
>>>>
>>>>    patch 1: minor changes and fixing docs generation (Jonathan, Dan)
>>>>
>>>>    patch4: merged with v17 patch5
>>>>
>>>>    patch 5: merging v17 patches 6 and 7
>>>>
>>>>    patch 6: adding helpers for clarity
>>>>
>>>>    patch 9:
>>>>        - minor changes (Dave)
>>>>        - simplifying flags check (Dan)
>>>>
>>>>    patch 10: minor changes (Jonathan)
>>>>
>>>>    patch 11:
>>>>        - minor changes (Dave)
>>>>        - fix mess (Jonathan, Dave)
>>>>
>>>>    patch 18: minor changes (Jonathan, Dan)
>>>>
>>>> v17 changes: (Dan Williams review)
>>>>   - use devm for cxl_dev_state allocation
>>>>   - using current cxl struct for checking capability registers found by
>>>>     the driver.
>>>>   - simplify dpa initialization without a mailbox not supporting pmem
>>>>   - add cxl_acquire_endpoint for protection during initialization
>>>>   - add callback/action to cxl_create_region for a driver notified about cxl
>>>>     core kernel modules removal.
>>>>   - add sfc function to disable CXL-based PIO buffers if such a callback
>>>>     is invoked.
>>>>   - Always manage a Type2 created region as private not allowing DAX.
>>>>
>>>> v16 changes:
>>>>   - rebase against rc4 (Dave Jiang)
>>>>   - remove duplicate line (Ben Cheatham)
>>>>
>>>> v15 changes:
>>>>   - remove reference to unused header file (Jonathan Cameron)
>>>>   - add proper kernel docs to exported functions (Alison Schofield)
>>>>   - using an array to map the enums to strings (Alison Schofield)
>>>>   - clarify comment when using bitmap_subset (Jonathan Cameron)
>>>>   - specify link to type2 support in all patches (Alison Schofield)
>>>>
>>>>    Patches changed (minor): 4, 11
>>>>
>>>> v14 changes:
>>>>   - static null initialization of bitmaps (Jonathan Cameron)
>>>>   - Fixing cxl tests (Alison Schofield)
>>>>   - Fixing robot compilation problems
>>>>
>>>>    Patches changed (minor): 1, 4, 6, 13
>>>>
>>>> v13 changes:
>>>>   - using names for headers checking more consistent (Jonathan Cameron)
>>>>   - using helper for caps bit setting (Jonathan Cameron)
>>>>   - provide generic function for reporting missing capabilities (Jonathan Cameron)
>>>>   - rename cxl_pci_setup_memdev_regs to cxl_pci_accel_setup_memdev_regs (Jonathan Cameron)
>>>>   - cxl_dpa_info size to be set by the Type2 driver (Jonathan Cameron)
>>>>   - avoiding rc variable when possible (Jonathan Cameron)
>>>>   - fix spelling (Simon Horman)
>>>>   - use scoped_guard (Dave Jiang)
>>>>   - use enum instead of bool (Dave Jiang)
>>>>   - dropping patch with hardware symbols
>>>>
>>>> v12 changes:
>>>>   - use new macro cxl_dev_state_create in pci driver (Ben Cheatham)
>>>>   - add public/private sections in now exported cxl_dev_state struct (Ben
>>>>     Cheatham)
>>>>   - fix cxl/pci.h regarding file name for checking if defined
>>>>   - Clarify capabilities found vs expected in error message. (Ben
>>>>     Cheatham)
>>>>   - Clarify new CXL_DECODER_F flag (Ben Cheatham)
>>>>   - Fix changes about cxl memdev creation support moving code to the
>>>>     proper patch. (Ben Cheatham)
>>>>   - Avoid debug and function duplications (Ben Cheatham)
>>>>
>>>> v11 changes:
>>>>   - Dropping the use of cxl_memdev_state and going back to using
>>>>     cxl_dev_state.
>>>>   - Using a helper for an accel driver to allocate its own cxl-related
>>>>     struct embedding cxl_dev_state.
>>>>   - Exporting the required structs in include/cxl/cxl.h for an accel
>>>>     driver being able to know the cxl_dev_state size required in the
>>>>     previously mentioned helper for allocation.
>>>>   - Avoid using any struct for dpa initialization by the accel driver
>>>>     adding a specific function for creating dpa partitions by accel
>>>>     drivers without a mailbox.
>>>>
>>>> v10 changes:
>>>>   - Using cxl_memdev_state instead of cxl_dev_state for type2 which has a
>>>>     memory after all and facilitates the setup.
>>>>   - Adapt core for using cxl_memdev_state allowing accel drivers to work
>>>>     with them without further awareness of internal cxl structs.
>>>>   - Using last DPA changes for creating DPA partitions with accel driver
>>>>     hardcoding mds values when no mailbox.
>>>>   - capabilities not a new field but built up when current register maps
>>>>     is performed and returned to the caller for checking.
>>>>   - HPA free space supporting interleaving.
>>>>   - DPA free space droping max-min for a simple alloc size.
>>>>
>>>> v9 changes:
>>>>   - adding forward definitions (Jonathan Cameron)
>>>>   - using set_bit instead of bitmap_set (Jonathan Cameron)
>>>>   - fix rebase problem (Jonathan Cameron)
>>>>   - Improve error path (Jonathan Cameron)
>>>>   - fix build problems with cxl region dependency (robot)
>>>>   - fix error path (Simon Horman)
>>>>
>>>> v8 changes:
>>>>   - Change error path labeling inside sfc cxl code (Edward Cree)
>>>>   - Properly handling checks and error in sfc cxl code (Simon Horman)
>>>>   - Fix bug when checking resource_size (Simon Horman)
>>>>   - Avoid bisect problems reordering patches (Edward Cree)
>>>>   - Fix buffer allocation size in sfc (Simon Horman)
>>>>
>>>> v7 changes:
>>>>
>>>>   - fixing kernel test robot complains
>>>>   - fix type with Type3 mandatory capabilities (Zhi Wang)
>>>>   - optimize code in cxl_request_resource (Kalesh Anakkur Purayil)
>>>>   - add sanity check when dealing with resources arithmetics (Fan Ni)
>>>>   - fix typos and blank lines (Fan Ni)
>>>>   - keep previous log errors/warnings in sfc driver (Martin Habets)
>>>>   - add WARN_ON_ONCE if region given is NULL
>>>>
>>>> v6 changes:
>>>>
>>>>   - update sfc mcdi_pcol.h with full hardware changes most not related to
>>>>     this patchset. This is an automatic file created from hardware design
>>>>     changes and not touched by software. It is updated from time to time
>>>>     and it required update for the sfc driver CXL support.
>>>>   - remove CXL capabilities definitions not used by the patchset or
>>>>     previous kernel code. (Dave Jiang, Jonathan Cameron)
>>>>   - Use bitmap_subset instead of reinventing the wheel ... (Ben Cheatham)
>>>>   - Use cxl_accel_memdev for new device_type created (Ben Cheatham)
>>>>   - Fix construct_region use of rwsem (Zhi Wang)
>>>>   - Obtain region range instead of region params (Allison Schofield, Dave
>>>>     Jiang)
>>>>
>>>> v5 changes:
>>>>
>>>>   - Fix SFC configuration based on kernel CXL configuration
>>>>   - Add subset check for capabilities.
>>>>   - fix region creation when HDM decoders programmed by firmware/BIOS (Ben
>>>>     Cheatham)
>>>>   - Add option for creating dax region based on driver decission (Ben
>>>>     Cheatham)
>>>>   - Using sfc probe_data struct for keeping sfc cxl data
>>>>
>>>> v4 changes:
>>>>
>>>>   - Use bitmap for capabilities new field (Jonathan Cameron)
>>>>   - Use cxl_mem attributes for sysfs based on device type (Dave Jian)
>>>>   - Add conditional cxl sfc compilation relying on kernel CXL config (kernel test robot)
>>>>   - Add sfc changes in different patches for facilitating backport (Jonathan Cameron)
>>>>   - Remove patch for dealing with cxl modules dependencies and using sfc kconfig plus
>>>>     MODULE_SOFTDEP instead.
>>>>
>>>> v3 changes:
>>>>
>>>>   - cxl_dev_state not defined as opaque but only manipulated by accel drivers
>>>>     through accessors.
>>>>   - accessors names not identified as only for accel drivers.
>>>>   - move pci code from pci driver (drivers/cxl/pci.c) to generic pci code
>>>>     (drivers/cxl/core/pci.c).
>>>>   - capabilities field from u8 to u32 and initialised by CXL regs discovering
>>>>     code.
>>>>   - add capabilities check and removing current check by CXL regs discovering
>>>>     code.
>>>>   - Not fail if CXL Device Registers not found. Not mandatory for Type2.
>>>>   - add timeout in acquire_endpoint for solving a race with the endpoint port
>>>>     creation.
>>>>   - handle EPROBE_DEFER by sfc driver.
>>>>   - Limiting interleave ways to 1 for accel driver HPA/DPA requests.
>>>>   - factoring out interleave ways and granularity helpers from type2 region
>>>>     creation patch.
>>>>   - restricting region_creation for type2 to one endpoint decoder.
>>>>
>>>> v2 changes:
>>>>
>>>> I have removed the introduction about the concerns with BIOS/UEFI after the
>>>> discussion leading to confirm the need of the functionality implemented, at
>>>> least is some scenarios.
>>>>
>>>> There are two main changes from the RFC:
>>>>
>>>> 1) Following concerns about drivers using CXL core without restrictions, the CXL
>>>> struct to work with is opaque to those drivers, therefore functions are
>>>> implemented for modifying or reading those structs indirectly.
>>>>
>>>> 2) The driver for using the added functionality is not a test driver but a real
>>>> one: the SFC ethernet network driver. It uses the CXL region mapped for PIO
>>>> buffers instead of regions inside PCIe BARs.
>>>>
>>>> RFC:
>>>>
>>>> Current CXL kernel code is focused on supporting Type3 CXL devices, aka memory
>>>> expanders. Type2 CXL devices, aka device accelerators, share some functionalities
>>>> but require some special handling.
>>>>
>>>> First of all, Type2 are by definition specific to drivers doing something and not just
>>>> a memory expander, so it is expected to work with the CXL specifics. This implies the CXL
>>>> setup needs to be done by such a driver instead of by a generic CXL PCI driver
>>>> as for memory expanders. Most of such setup needs to use current CXL core code
>>>> and therefore needs to be accessible to those vendor drivers. This is accomplished
>>>> exporting opaque CXL structs and adding and exporting functions for working with
>>>> those structs indirectly.
>>>>
>>>> Some of the patches are based on a patchset sent by Dan Williams [1] which was just
>>>> partially integrated, most related to making things ready for Type2 but none
>>>> related to specific Type2 support. Those patches based on Dan´s work have Dan´s
>>>> signing as co-developer, and a link to the original patch.
>>>>
>>>> A final note about CXL.cache is needed. This patchset does not cover it at all,
>>>> although the emulated Type2 device advertises it. From the kernel point of view
>>>> supporting CXL.cache will imply to be sure the CXL path supports what the Type2
>>>> device needs. A device accelerator will likely be connected to a Root Switch,
>>>> but other configurations can not be discarded. Therefore the kernel will need to
>>>> check not just HPA, DPA, interleave and granularity, but also the available
>>>> CXL.cache support and resources in each switch in the CXL path to the Type2
>>>> device. I expect to contribute to this support in the following months, and
>>>> it would be good to discuss about it when possible.
>>>>
>>>> [1] https://lore.kernel.org/linux-cxl/98b1f61a-e6c2-71d4-c368-50d958501b0c@intel.com/T/
>>>>
>>>> Alejandro Lucero (21):
>>>>    cxl/mem: Arrange for always-synchronous memdev attach
>>>>    cxl/port: Arrange for always synchronous endpoint attach
>>>>    cxl: Add type2 device basic support
>>>>    sfc: add cxl support
>>>>    cxl: Move pci generic code
>>>>    cxl: allow Type2 drivers to map cxl component regs
>>>>    cxl: Support dpa initialization without a mailbox
>>>>    cxl: Prepare memdev creation for type2
>>>>    sfc: create type2 cxl memdev
>>>>    cxl: Define a driver interface for HPA free space enumeration
>>>>    sfc: get root decoder
>>>>    cxl: Define a driver interface for DPA allocation
>>>>    sfc: get endpoint decoder
>>>>    cxl: Make region type based on endpoint type
>>>>    cxl/region: Factor out interleave ways setup
>>>>    cxl/region: Factor out interleave granularity setup
>>>>    cxl: Allow region creation by type2 drivers
>>>>    cxl: Avoid dax creation for accelerators
>>>>    sfc: create cxl region
>>>>    cxl: Add function for obtaining region range
>>>>    sfc: support pio mapping based on cxl
>>>>
>>>> Dan Williams (1):
>>>>    cxl/mem: Introduce a memdev creation ->probe() operation
>>>>
>>>>   drivers/cxl/Kconfig                   |   2 +-
>>>>   drivers/cxl/core/core.h               |   9 +-
>>>>   drivers/cxl/core/hdm.c                |  85 ++++++
>>>>   drivers/cxl/core/mbox.c               |  63 +---
>>>>   drivers/cxl/core/memdev.c             | 209 +++++++++----
>>>>   drivers/cxl/core/pci.c                |  63 ++++
>>>>   drivers/cxl/core/port.c               |   1 +
>>>>   drivers/cxl/core/region.c             | 418 +++++++++++++++++++++++---
>>>>   drivers/cxl/core/regs.c               |   2 +-
>>>>   drivers/cxl/cxl.h                     | 125 +-------
>>>>   drivers/cxl/cxlmem.h                  |  90 +-----
>>>>   drivers/cxl/cxlpci.h                  |  21 +-
>>>>   drivers/cxl/mem.c                     | 146 +++++----
>>>>   drivers/cxl/pci.c                     |  88 +-----
>>>>   drivers/cxl/port.c                    |  46 ++-
>>>>   drivers/cxl/private.h                 |  17 ++
>>>>   drivers/net/ethernet/sfc/Kconfig      |  10 +
>>>>   drivers/net/ethernet/sfc/Makefile     |   1 +
>>>>   drivers/net/ethernet/sfc/ef10.c       |  50 ++-
>>>>   drivers/net/ethernet/sfc/efx.c        |  15 +-
>>>>   drivers/net/ethernet/sfc/efx.h        |   1 -
>>>>   drivers/net/ethernet/sfc/efx_cxl.c    | 165 ++++++++++
>>>>   drivers/net/ethernet/sfc/efx_cxl.h    |  40 +++
>>>>   drivers/net/ethernet/sfc/net_driver.h |  12 +
>>>>   drivers/net/ethernet/sfc/nic.h        |   3 +
>>>>   include/cxl/cxl.h                     | 291 ++++++++++++++++++
>>>>   include/cxl/pci.h                     |  21 ++
>>>>   tools/testing/cxl/Kbuild              |   1 -
>>>>   tools/testing/cxl/test/mem.c          |   5 +-
>>>>   tools/testing/cxl/test/mock.c         |  17 --
>>>>   30 files changed, 1476 insertions(+), 541 deletions(-)
>>>>   create mode 100644 drivers/cxl/private.h
>>>>   create mode 100644 drivers/net/ethernet/sfc/efx_cxl.c
>>>>   create mode 100644 drivers/net/ethernet/sfc/efx_cxl.h
>>>>   create mode 100644 include/cxl/cxl.h
>>>>   create mode 100644 include/cxl/pci.h
>>>>
>>>>
>>>> base-commit: f11a5f89910a7ae970fbce4fdc02d86a8ba8570f
>>>> prerequisite-patch-id: 44c914dd079e40d716f3f2d91653247eca731594
>>>> prerequisite-patch-id: b13ca5c11c44a736563477d67b1dceadfe3ea19e
>>>> prerequisite-patch-id: d0d82965bbea8a2b5ea2f763f19de4dfaa8479c3
>>>> prerequisite-patch-id: dd0f24b3bdb938f2f123bc26b31cd5fe659e05eb
>>>> prerequisite-patch-id: 2ea41ec399f2360a84e86e97a8f940a62561931a
>>>> prerequisite-patch-id: 367b61b5a313db6324f9cf917d46df580f3bbd3b
>>>> prerequisite-patch-id: 1805332a9f191bc3547927d96de5926356dac03c
>>>> prerequisite-patch-id: 40657fd517f8e835a091c07e93d6abc08f85d395
>>>> prerequisite-patch-id: 901eb0d91816499446964b2a9089db59656da08d
>>>> prerequisite-patch-id: 79856c0199d6872fd2f76a5829dba7fa46f225d6
>>>> prerequisite-patch-id: 6f3503e59a3d745e5ecff4aaed668e2d32da7e4b
>>>> prerequisite-patch-id: e9dc88f1b91dce5dc3d46ff2b5bf184aba06439d
>>>> prerequisite-patch-id: 196fe106100aad619d5be7266959bbeef29b7c8b
>>>> prerequisite-patch-id: 7e719ed404f664ee8d9b98d56f58326f55ea2175
>>>> prerequisite-patch-id: 560f95992e13a08279034d5f77aacc9e971332dd
>>>> prerequisite-patch-id: 8656445ee654056695ff2894e28c8f1014df919e
>>>> prerequisite-patch-id: 001d831149eb8f9ae17b394e4bcd06d844dd39d9
>>>> prerequisite-patch-id: 421368aa5eac2af63ef2dc427af2ec11ad45c925
>>>> prerequisite-patch-id: 18fd00d4743711d835ad546cfbb558d9f97dcdfc
>>>> prerequisite-patch-id: d89bf9e6d3ea5d332ec2c8e441f1fe6d84e726d3
>>>> prerequisite-patch-id: 3a6953d11b803abeb437558f3893a3b6a08acdbb
>>>> prerequisite-patch-id: 0dd42a82e73765950bd069d421d555ded8bfeb25
>>>> prerequisite-patch-id: da6e0df31ad0d5a945e0a0d29204ba75f0c97344
>>>
> 
> 

Re: [PATCH v19 01/22] cxl/mem: Arrange for always-synchronous memdev attach

[Dave Jiang]

On 10/7/25 5:42 AM, Jonathan Cameron wrote:
> On Tue, 7 Oct 2025 13:40:53 +0100
> Jonathan Cameron <jonathan.cameron@huawei.com> wrote:
> 
>> On Mon, 6 Oct 2025 11:01:09 +0100
>> <alejandro.lucero-palau@amd.com> wrote:
>>
>>> From: Alejandro Lucero <alucerop@amd.com>
>>>
>>> In preparation for CXL accelerator drivers that have a hard dependency on
>>> CXL capability initialization, arrange for the endpoint probe result to be
>>> conveyed to the caller of devm_cxl_add_memdev().
>>>
>>> As it stands cxl_pci does not care about the attach state of the cxl_memdev
>>> because all generic memory expansion functionality can be handled by the
>>> cxl_core. For accelerators, that driver needs to know perform driver
>>> specific initialization if CXL is available, or exectute a fallback to PCIe
>>> only operation.
>>>
>>> By moving devm_cxl_add_memdev() to cxl_mem.ko it removes async module
>>> loading as one reason that a memdev may not be attached upon return from
>>> devm_cxl_add_memdev().
>>>
>>> The diff is busy as this moves cxl_memdev_alloc() down below the definition
>>> of cxl_memdev_fops and introduces devm_cxl_memdev_add_or_reset() to
>>> preclude needing to export more symbols from the cxl_core.
>>>
>>> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
>>
>> Alejandro, SoB chain broken here which makes this currently unmergeable.
>>
>> Should definitely have your SoB as you sent the patch to the list and need
>> to make a statement that you believe it to be fine to do so (see the Certificate
>> of origin stuff in the docs).  Also, From should always be one of the authors.
>> If Dan wrote this as the SoB suggests then From should be set to him..
>>
>> git commit --amend --author="Dan Williams <dan.j.williams@intel.com>"
>>
>> Will fix that up.  Then either you add your SoB on basis you just 'handled'
>> the patch but didn't make substantial changes, or your SoB and a Codeveloped-by
>> if you did make major changes.  If it is minor stuff you can an
>> a sign off with # what changed 
>> comment next to it.
>>
>> A few minor comments inline.
> 
> oops. I see this was part of Dan's series that has merged by now...

Dan's changes has not been merged AFAIK.

DJ

> 
> Never mind unless you want to take any of the suggested tweaks forwards.
> 
> J
>>
>> Thanks,
>>
>> Jonathan
>>
>>
>>> ---
>>>  drivers/cxl/Kconfig       |  2 +-
>>>  drivers/cxl/core/memdev.c | 97 ++++++++++++++++-----------------------
>>>  drivers/cxl/mem.c         | 30 ++++++++++++
>>>  drivers/cxl/private.h     | 11 +++++
>>>  4 files changed, 82 insertions(+), 58 deletions(-)
>>>  create mode 100644 drivers/cxl/private.h
>>>
>>> diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
>>> index 028201e24523..111e05615f09 100644
>>> --- a/drivers/cxl/Kconfig
>>> +++ b/drivers/cxl/Kconfig
>>> @@ -22,6 +22,7 @@ if CXL_BUS
>>>  config CXL_PCI
>>>  	tristate "PCI manageability"
>>>  	default CXL_BUS
>>> +	select CXL_MEM
>>>  	help
>>>  	  The CXL specification defines a "CXL memory device" sub-class in the
>>>  	  PCI "memory controller" base class of devices. Device's identified by
>>> @@ -89,7 +90,6 @@ config CXL_PMEM
>>>  
>>>  config CXL_MEM
>>>  	tristate "CXL: Memory Expansion"
>>> -	depends on CXL_PCI
>>>  	default CXL_BUS
>>>  	help
>>>  	  The CXL.mem protocol allows a device to act as a provider of "System
>>> diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
>>> index c569e00a511f..2bef231008df 100644
>>> --- a/drivers/cxl/core/memdev.c
>>> +++ b/drivers/cxl/core/memdev.c
>>
>>> -
>>> -err:
>>> -	kfree(cxlmd);
>>> -	return ERR_PTR(rc);
>>>  }
>>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_memdev_add_or_reset, "CXL");
>>>  
>>>  static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
>>>  			       unsigned long arg)
>>> @@ -1023,50 +1012,44 @@ static const struct file_operations cxl_memdev_fops = {
>>>  	.llseek = noop_llseek,
>>>  };
>>>  
>>> -struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>>> -				       struct cxl_dev_state *cxlds)
>>> +struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds)
>>>  {
>>>  	struct cxl_memdev *cxlmd;
>>>  	struct device *dev;
>>>  	struct cdev *cdev;
>>>  	int rc;
>>>  
>>> -	cxlmd = cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
>>> -	if (IS_ERR(cxlmd))
>>> -		return cxlmd;
>>> +	cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
>>
>> It's a little bit non obvious due to the device initialize mid way
>> through this, but given there are no error paths after that you can
>> currently just do.
>> 	struct cxl_memdev *cxlmd __free(kfree) =
>> 		cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
>> and
>> 	return_ptr(cxlmd);
>>
>> in the good path.  That lets you then just return rather than having
>> the goto err: handling for the error case that currently frees this
>> manually.
>>
>> Unlike the change below, this one I think is definitely worth making.
>>
>>
>>> +	if (!cxlmd)
>>> +		return ERR_PTR(-ENOMEM);
>>>  
>>> -	dev = &cxlmd->dev;
>>> -	rc = dev_set_name(dev, "mem%d", cxlmd->id);
>>> -	if (rc)
>>> +	rc = ida_alloc_max(&cxl_memdev_ida, CXL_MEM_MAX_DEVS - 1, GFP_KERNEL);
>>> +	if (rc < 0)
>>>  		goto err;
>>> -
>>> -	/*
>>> -	 * Activate ioctl operations, no cxl_memdev_rwsem manipulation
>>> -	 * needed as this is ordered with cdev_add() publishing the device.
>>> -	 */
>>> +	cxlmd->id = rc;
>>> +	cxlmd->depth = -1;
>>>  	cxlmd->cxlds = cxlds;
>>>  	cxlds->cxlmd = cxlmd;
>>>  
>>> -	cdev = &cxlmd->cdev;
>>> -	rc = cdev_device_add(cdev, dev);
>>> -	if (rc)
>>> -		goto err;
>>> +	dev = &cxlmd->dev;
>>> +	device_initialize(dev);
>>> +	lockdep_set_class(&dev->mutex, &cxl_memdev_key);
>>> +	dev->parent = cxlds->dev;
>>> +	dev->bus = &cxl_bus_type;
>>> +	dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
>>> +	dev->type = &cxl_memdev_type;
>>> +	device_set_pm_not_required(dev);
>>> +	INIT_WORK(&cxlmd->detach_work, detach_memdev);
>>>  
>>> -	rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
>>> -	if (rc)
>>> -		return ERR_PTR(rc);
>>> +	cdev = &cxlmd->cdev;
>>> +	cdev_init(cdev, &cxl_memdev_fops);
>>>  	return cxlmd;
>>>  
>>>  err:
>>> -	/*
>>> -	 * The cdev was briefly live, shutdown any ioctl operations that
>>> -	 * saw that state.
>>> -	 */
>>> -	cxl_memdev_shutdown(dev);
>>> -	put_device(dev);
>>> +	kfree(cxlmd);
>>>  	return ERR_PTR(rc);
>>>  }
>>> -EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
>>> +EXPORT_SYMBOL_NS_GPL(cxl_memdev_alloc, "CXL");
>>>  
>>>  static void sanitize_teardown_notifier(void *data)
>>>  {
>>> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
>>> index f7dc0ba8905d..144749b9c818 100644
>>> --- a/drivers/cxl/mem.c
>>> +++ b/drivers/cxl/mem.c
>>> @@ -7,6 +7,7 @@
>>>  
>>>  #include "cxlmem.h"
>>>  #include "cxlpci.h"
>>> +#include "private.h"
>>>  #include "core/core.h"
>>>  
>>>  /**
>>> @@ -203,6 +204,34 @@ static int cxl_mem_probe(struct device *dev)
>>>  	return devm_add_action_or_reset(dev, enable_suspend, NULL);
>>>  }
>>>  
>>> +/**
>>> + * devm_cxl_add_memdev - Add a CXL memory device
>>> + * @host: devres alloc/release context and parent for the memdev
>>> + * @cxlds: CXL device state to associate with the memdev
>>> + *
>>> + * Upon return the device will have had a chance to attach to the
>>> + * cxl_mem driver, but may fail if the CXL topology is not ready
>>> + * (hardware CXL link down, or software platform CXL root not attached)
>>> + */
>>> +struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>>> +				       struct cxl_dev_state *cxlds)
>>> +{
>>> +	struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds);
>>
>> Bit marginal but you could do a DEFINE_FREE() for cxlmd 
>> similar to the one that exists for put_cxl_port
>>
>> You would then need to steal the pointer for the devm_ call at the
>> end of this function.
>>
>>
>>> +	int rc;
>>> +
>>> +	if (IS_ERR(cxlmd))
>>> +		return cxlmd;
>>> +
>>> +	rc = dev_set_name(&cxlmd->dev, "mem%d", cxlmd->id);
>>> +	if (rc) {
>>> +		put_device(&cxlmd->dev);
>>> +		return ERR_PTR(rc);
>>> +	}
>>> +
>>> +	return devm_cxl_memdev_add_or_reset(host, cxlmd);
>>> +}
>>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
> 

Re: [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration

[Dave Jiang]

On 10/10/25 4:16 AM, Alejandro Lucero Palau wrote:
> 
> On 10/9/25 21:55, Cheatham, Benjamin wrote:
>> On 10/6/2025 5:01 AM, alejandro.lucero-palau@amd.com wrote:
>>> From: Alejandro Lucero <alucerop@amd.com>
>>>
>>> CXL region creation involves allocating capacity from Device Physical Address
>>> (DPA) and assigning it to decode a given Host Physical Address (HPA). Before
>>> determining how much DPA to allocate the amount of available HPA must be
>>> determined. Also, not all HPA is created equal, some HPA targets RAM, some
>>> targets PMEM, some is prepared for device-memory flows like HDM-D and HDM-DB,
>>> and some is HDM-H (host-only).
>>>
>>> In order to support Type2 CXL devices, wrap all of those concerns into
>>> an API that retrieves a root decoder (platform CXL window) that fits the
>>> specified constraints and the capacity available for a new region.
>>>
>>> Add a complementary function for releasing the reference to such root
>>> decoder.
>>>
>>> Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/
>>>
>>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>> ---
>> Hey Alejandro,
> 
> 
> Hi Ben,
> 
> 
>> I've been testing this on my setup and noticed a few issues when BIOS sets up the HDM decoders. It came down to 2 issues:
>>     1) Enabling "Specific Purpose Memory" added a Soft Reserve resource below the CXL window resource and broke
>>     this patch (more below)
> 
> 
> Maybe we should talk (first) about this internally as it is about AMD BIOS (I guess). I have been talking with the BIOS team about this EFI_MEMORY_SP vs EFI_RESERVED_MEMORY, and I'm afraid the discussion is not over yet :-).
> 
> 
>>     2) The endpoint decoder was already set up which broke DPA allocation and then CXL region creation (see my response
>>     to patch 18/22 for fix and explanation)
> 
> 
> Yes, if the BIOS configures the device HDM decoder, the current patchset does not do the right thing. As I said in the cover letter, my expectation at the time, and hopefully in the future as well, although I'm not sure about it, was the BIOS not doing so. Most of the implementation is based on QEMU, so I found this problem when dealing with a real system with a Type2 aware BIOS ... . I was tempted to include support for this case, but I did not do so for several reasons:
> 
> 
> 1) I want to think the patchset is close to being accepted and changes at this state could delay it further. After more than a year, and because this patchset is about "initial Type2 support", I think it is better to do so in a follow-up work, and when there is a client requiring it.
> 
> 2) Because that conversation with BIOS guys, I prefer to be sure what to do, as there are other things we need to clarify and in my opinion, far more important than current Type2 support.
> 
> 3) CXL is a fast moving part of the kernel and I bet we will find another case which the current patchset is not dealing with properly. In fact there is another report of devices with the BAR with CXL information being also used by the driver for other purposes and existing a problem when mapping the CXL registers after the driver did map the whole BAR.
> 
> 
> So, I think the current patchset where most of the API for Type2 drivers is implemented should go as soon as possible, which will facilitate those follow-up works for the case you describe and the other one about BAR mappings. If not, even if retirement is still far away for me, I'll be concerned about the impending future of this work ... but of course, this is my suggestion, so let's see other opinions about it.
>

My 2 cents is that if the current code set works as intended for the hardware/user you have then lets get that enabled and we can deal with the other cases as next iterations.

Are there any use cases where the BIOS need to pre-commit the type2 decoders instead of allowing the driver/OS to do so?

DJ 
> 
> Thank you.
> 
> 
>>
>> The fix I did for 1 is a bit hacky but it's essentially checking none of the resources below the CXL window are onlined as
>> system memory. It's roughly equivalent to what's being done in the CXL_PARTMODE_RAM case of cxl_region_probe(), but
>> I'm restricting the resources to "Soft Reserved" to be safe.
>>
>> The diff for 2 is pretty big. If you don't want to take it at this point I can send it as a follow up. In that case I'd definitely
>> add that auto regions won't work in at least the cover letter (and in the description of 18/22 as well?).
>>
>> ---
>>
>> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
>> index acaca64764bf..2d60131edff3 100644
>> --- a/drivers/cxl/core/region.c
>> +++ b/drivers/cxl/core/region.c
>> @@ -784,6 +784,19 @@ static int find_max_hpa(struct device *dev, void *data)
>>          lockdep_assert_held_read(&cxl_rwsem.region);
>>          res = cxlrd->res->child;
>>
>> +       /*
>> +        * BIOS may have marked the CXL window as soft reserved. Make sure it's
>> +        * free to use.
>> +        */
>> +       while (res && resource_size(res) == resource_size(cxlrd->res)) {
>> +               if ((res->flags & IORESOURCE_BUSY) ||
>> +                   (res->flags & IORESOURCE_SYSRAM) ||
>> +                   strcmp(res->name, "Soft Reserved") != 0)
>> +                       return 0;
>> +
>> +               res = res->child;
>> +       }
>> +
>>          /* With no resource child the whole parent resource is available */
>>          if (!res)
>>                  max = resource_size(cxlrd->res);

Re: [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration

[Dave Jiang]

On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
> From: Alejandro Lucero <alucerop@amd.com>
> 
> CXL region creation involves allocating capacity from Device Physical Address
> (DPA) and assigning it to decode a given Host Physical Address (HPA). Before
> determining how much DPA to allocate the amount of available HPA must be
> determined. Also, not all HPA is created equal, some HPA targets RAM, some
> targets PMEM, some is prepared for device-memory flows like HDM-D and HDM-DB,
> and some is HDM-H (host-only).
> 
> In order to support Type2 CXL devices, wrap all of those concerns into
> an API that retrieves a root decoder (platform CXL window) that fits the
> specified constraints and the capacity available for a new region.
> 
> Add a complementary function for releasing the reference to such root
> decoder.
> 
> Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
>  drivers/cxl/core/region.c | 162 ++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxl.h         |   3 +
>  include/cxl/cxl.h         |   6 ++
>  3 files changed, 171 insertions(+)
> 
> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> index e9bf42d91689..c5b66204ecde 100644
> --- a/drivers/cxl/core/region.c
> +++ b/drivers/cxl/core/region.c
> @@ -703,6 +703,168 @@ static int free_hpa(struct cxl_region *cxlr)
>  	return 0;
>  }
>  
> +struct cxlrd_max_context {
> +	struct device * const *host_bridges;
> +	int interleave_ways;
> +	unsigned long flags;
> +	resource_size_t max_hpa;
> +	struct cxl_root_decoder *cxlrd;
> +};
> +
> +static int find_max_hpa(struct device *dev, void *data)
> +{
> +	struct cxlrd_max_context *ctx = data;
> +	struct cxl_switch_decoder *cxlsd;
> +	struct cxl_root_decoder *cxlrd;
> +	struct resource *res, *prev;
> +	struct cxl_decoder *cxld;
> +	resource_size_t max;
> +	int found = 0;
> +
> +	if (!is_root_decoder(dev))
> +		return 0;
> +
> +	cxlrd = to_cxl_root_decoder(dev);
> +	cxlsd = &cxlrd->cxlsd;
> +	cxld = &cxlsd->cxld;
> +
> +	if ((cxld->flags & ctx->flags) != ctx->flags) {
> +		dev_dbg(dev, "flags not matching: %08lx vs %08lx\n",
> +			cxld->flags, ctx->flags);
> +		return 0;
> +	}
> +
> +	for (int i = 0; i < ctx->interleave_ways; i++) {
> +		for (int j = 0; j < ctx->interleave_ways; j++) {
> +			if (ctx->host_bridges[i] == cxlsd->target[j]->dport_dev) {
> +				found++;
> +				break;
> +			}
> +		}
> +	}
> +
> +	if (found != ctx->interleave_ways) {
> +		dev_dbg(dev,
> +			"Not enough host bridges. Found %d for %d interleave ways requested\n",
> +			found, ctx->interleave_ways);
> +		return 0;
> +	}
> +
> +	/*
> +	 * Walk the root decoder resource range relying on cxl_rwsem.region to
> +	 * preclude sibling arrival/departure and find the largest free space
> +	 * gap.
> +	 */
> +	lockdep_assert_held_read(&cxl_rwsem.region);
> +	res = cxlrd->res->child;
> +
> +	/* With no resource child the whole parent resource is available */
> +	if (!res)
> +		max = resource_size(cxlrd->res);
> +	else
> +		max = 0;
> +
> +	for (prev = NULL; res; prev = res, res = res->sibling) {
> +		struct resource *next = res->sibling;
> +		resource_size_t free = 0;
> +
> +		/*
> +		 * Sanity check for preventing arithmetic problems below as a
> +		 * resource with size 0 could imply using the end field below
> +		 * when set to unsigned zero - 1 or all f in hex.
> +		 */
> +		if (prev && !resource_size(prev))
> +			continue;
> +
> +		if (!prev && res->start > cxlrd->res->start) {
> +			free = res->start - cxlrd->res->start;
> +			max = max(free, max);
> +		}
> +		if (prev && res->start > prev->end + 1) {
> +			free = res->start - prev->end + 1;
> +			max = max(free, max);
> +		}
> +		if (next && res->end + 1 < next->start) {
> +			free = next->start - res->end + 1;
> +			max = max(free, max);
> +		}
> +		if (!next && res->end + 1 < cxlrd->res->end + 1) {
> +			free = cxlrd->res->end + 1 - res->end + 1;
> +			max = max(free, max);
> +		}
> +	}
> +
> +	dev_dbg(cxlrd_dev(cxlrd), "found %pa bytes of free space\n", &max);
> +	if (max > ctx->max_hpa) {
> +		if (ctx->cxlrd)
> +			put_device(cxlrd_dev(ctx->cxlrd));
> +		get_device(cxlrd_dev(cxlrd));
> +		ctx->cxlrd = cxlrd;
> +		ctx->max_hpa = max;
> +	}
> +	return 0;
> +}
> +
> +/**
> + * cxl_get_hpa_freespace - find a root decoder with free capacity per constraints
> + * @cxlmd: the mem device requiring the HPA
> + * @interleave_ways: number of entries in @host_bridges
> + * @flags: CXL_DECODER_F flags for selecting RAM vs PMEM, and Type2 device
> + * @max_avail_contig: output parameter of max contiguous bytes available in the
> + *		      returned decoder
> + *
> + * Returns a pointer to a struct cxl_root_decoder
> + *
> + * The return tuple of a 'struct cxl_root_decoder' and 'bytes available given
> + * in (@max_avail_contig))' is a point in time snapshot. If by the time the
> + * caller goes to use this decoder and its capacity is reduced then caller needs
> + * to loop and retry.
> + *
> + * The returned root decoder has an elevated reference count that needs to be
> + * put with cxl_put_root_decoder(cxlrd).
> + */
> +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
> +					       int interleave_ways,
> +					       unsigned long flags,
> +					       resource_size_t *max_avail_contig)
> +{
> +	struct cxl_port *endpoint = cxlmd->endpoint;

Do the assignment right before the check. Would help prevent future issues of use before check.

> +	struct cxlrd_max_context ctx = {
> +		.flags = flags,
> +	};
> +	struct cxl_port *root_port;
> +
> +	if (!endpoint) {
> +		dev_dbg(&cxlmd->dev, "endpoint not linked to memdev\n");
> +		return ERR_PTR(-ENXIO);
> +	}
> +
> +	ctx.host_bridges = &endpoint->host_bridge;

Would there ever be a scenario where there would be multiple host bridges that requires this to be an array? I'm not seeing that usage in this patch series.

DJ

> +
> +	struct cxl_root *root __free(put_cxl_root) = find_cxl_root(endpoint);
> +	if (!root) {
> +		dev_dbg(&endpoint->dev, "endpoint is not related to a root port\n");
> +		return ERR_PTR(-ENXIO);
> +	}
> +
> +	root_port = &root->port;
> +	scoped_guard(rwsem_read, &cxl_rwsem.region)
> +		device_for_each_child(&root_port->dev, &ctx, find_max_hpa);
> +
> +	if (!ctx.cxlrd)
> +		return ERR_PTR(-ENOMEM);
> +
> +	*max_avail_contig = ctx.max_hpa;
> +	return ctx.cxlrd;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_get_hpa_freespace, "CXL");
> +
> +void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd)
> +{
> +	put_device(cxlrd_dev(cxlrd));
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_put_root_decoder, "CXL");
> +
>  static ssize_t size_store(struct device *dev, struct device_attribute *attr,
>  			  const char *buf, size_t len)
>  {
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 793d4dfe51a2..076640e91ee0 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -664,6 +664,9 @@ struct cxl_root_decoder *to_cxl_root_decoder(struct device *dev);
>  struct cxl_switch_decoder *to_cxl_switch_decoder(struct device *dev);
>  struct cxl_endpoint_decoder *to_cxl_endpoint_decoder(struct device *dev);
>  bool is_root_decoder(struct device *dev);
> +
> +#define cxlrd_dev(cxlrd) (&(cxlrd)->cxlsd.cxld.dev)
> +
>  bool is_switch_decoder(struct device *dev);
>  bool is_endpoint_decoder(struct device *dev);
>  struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port,
> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
> index 043fc31c764e..2ec514c77021 100644
> --- a/include/cxl/cxl.h
> +++ b/include/cxl/cxl.h
> @@ -250,4 +250,10 @@ int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity);
>  struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>  				       struct cxl_dev_state *cxlds,
>  				       const struct cxl_memdev_ops *ops);
> +struct cxl_port;
> +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
> +					       int interleave_ways,
> +					       unsigned long flags,
> +					       resource_size_t *max);
> +void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd);
>  #endif /* __CXL_CXL_H__ */

Re: [PATCH v19 13/22] cxl: Define a driver interface for DPA allocation

[Dave Jiang]

On 10/7/25 6:52 AM, Jonathan Cameron wrote:
> On Mon, 6 Oct 2025 11:01:21 +0100
> <alejandro.lucero-palau@amd.com> wrote:
> 
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> Region creation involves finding available DPA (device-physical-address)
>> capacity to map into HPA (host-physical-address) space.
>>
>> In order to support CXL Type2 devices, define an API, cxl_request_dpa(),
>> that tries to allocate the DPA memory the driver requires to operate.The
>> memory requested should not be bigger than the max available HPA obtained
>> previously with cxl_get_hpa_freespace().
>>
>> Based on https://lore.kernel.org/linux-cxl/168592158743.1948938.7622563891193802610.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> 
> A few minor things inline.  Depending on how much this changed
> from the 'Based on' it might be appropriate to keep a SoB / author
> set to Dan, but I'll let him request that if he feels appropriate
> (or you can make that decision if Dan is busy).
> 
> A few things inline.  All trivial
> 
> Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
> 
>>  int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>>  		     enum cxl_partition_mode mode)
>> @@ -613,6 +622,82 @@ int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>>  	return 0;
>>  }
>>  
>> +static int find_free_decoder(struct device *dev, const void *data)
>> +{
>> +	struct cxl_endpoint_decoder *cxled;
>> +	struct cxl_port *port;
>> +
>> +	if (!is_endpoint_decoder(dev))
>> +		return 0;
>> +
>> +	cxled = to_cxl_endpoint_decoder(dev);
>> +	port = cxled_to_port(cxled);
>> +
>> +	return cxled->cxld.id == (port->hdm_end + 1);
>> +}
>> +
>> +static struct cxl_endpoint_decoder *
>> +cxl_find_free_decoder(struct cxl_memdev *cxlmd)
>> +{
>> +	struct cxl_port *endpoint = cxlmd->endpoint;
>> +	struct device *dev;
>> +
>> +	guard(rwsem_read)(&cxl_rwsem.dpa);
>> +	dev = device_find_child(&endpoint->dev, NULL,
>> +				find_free_decoder);
>> +	if (dev)
>> +		return to_cxl_endpoint_decoder(dev);
>> +
>> +	return NULL;
> Trivial but I'd prefer to see the 'error' like thing out of line
> 
> 	if (!dev)
> 		return NULL;
> 
> 	return to_cxl_endpoint_decoder(dev);
> 
>> +}
>> +
>> +/**
>> + * cxl_request_dpa - search and reserve DPA given input constraints
>> + * @cxlmd: memdev with an endpoint port with available decoders
>> + * @mode: CXL partition mode (ram vs pmem)
>> + * @alloc: dpa size required
>> + *
>> + * Returns a pointer to a 'struct cxl_endpoint_decoder' on success or
>> + * an errno encoded pointer on failure.
>> + *
>> + * Given that a region needs to allocate from limited HPA capacity it
>> + * may be the case that a device has more mappable DPA capacity than
>> + * available HPA. The expectation is that @alloc is a driver known
>> + * value based on the device capacity but which could not be fully
>> + * available due to HPA constraints.
>> + *
>> + * Returns a pinned cxl_decoder with at least @alloc bytes of capacity
>> + * reserved, or an error pointer. The caller is also expected to own the
>> + * lifetime of the memdev registration associated with the endpoint to
>> + * pin the decoder registered as well.
>> + */
>> +struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
>> +					     enum cxl_partition_mode mode,
>> +					     resource_size_t alloc)
>> +{
>> +	int rc;
>> +
>> +	if (!IS_ALIGNED(alloc, SZ_256M))
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	struct cxl_endpoint_decoder *cxled __free(put_cxled) =
>> +		cxl_find_free_decoder(cxlmd);
>> +
>> +	if (!cxled)
>> +		return ERR_PTR(-ENODEV);
>> +
>> +	rc = cxl_dpa_set_part(cxled, mode);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	rc = cxl_dpa_alloc(cxled, alloc);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	return no_free_ptr(cxled);
> 
> return_ptr() (it's exactly the same implementation).

I don't care either way. But as I recall Dan preferred the 'return no_free_ptr()' version to make it obvious.

DJ

> 
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_request_dpa, "CXL");
> 

Re: [PATCH v19 13/22] cxl: Define a driver interface for DPA allocation

[Dave Jiang]

On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
> From: Alejandro Lucero <alucerop@amd.com>
> 
> Region creation involves finding available DPA (device-physical-address)
> capacity to map into HPA (host-physical-address) space.
> 
> In order to support CXL Type2 devices, define an API, cxl_request_dpa(),
> that tries to allocate the DPA memory the driver requires to operate.The
> memory requested should not be bigger than the max available HPA obtained
> previously with cxl_get_hpa_freespace().
> 
> Based on https://lore.kernel.org/linux-cxl/168592158743.1948938.7622563891193802610.stgit@dwillia2-xfh.jf.intel.com/
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>

just a nit below

Reviewed-by: Dave Jiang <dave.jiang@intel.com>> ---
>  drivers/cxl/core/hdm.c | 85 ++++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxl.h      |  1 +
>  include/cxl/cxl.h      |  5 +++
>  3 files changed, 91 insertions(+)
> 
> diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
> index e9e1d555cec6..70a15694c35d 100644
> --- a/drivers/cxl/core/hdm.c
> +++ b/drivers/cxl/core/hdm.c
> @@ -3,6 +3,7 @@
>  #include <linux/seq_file.h>
>  #include <linux/device.h>
>  #include <linux/delay.h>
> +#include <cxl/cxl.h>
>  
>  #include "cxlmem.h"
>  #include "core.h"
> @@ -556,6 +557,13 @@ bool cxl_resource_contains_addr(const struct resource *res, const resource_size_
>  	return resource_contains(res, &_addr);
>  }
>  
> +/**
> + * cxl_dpa_free - release DPA (Device Physical Address)
> + *

no blank line needed here

DJ> + * @cxled: endpoint decoder linked to the DPA
> + *
> + * Returns 0 or error.
> + */
>  int cxl_dpa_free(struct cxl_endpoint_decoder *cxled)
>  {
>  	struct cxl_port *port = cxled_to_port(cxled);
> @@ -582,6 +590,7 @@ int cxl_dpa_free(struct cxl_endpoint_decoder *cxled)
>  	devm_cxl_dpa_release(cxled);
>  	return 0;
>  }
> +EXPORT_SYMBOL_NS_GPL(cxl_dpa_free, "CXL");
>  
>  int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>  		     enum cxl_partition_mode mode)
> @@ -613,6 +622,82 @@ int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>  	return 0;
>  }
>  
> +static int find_free_decoder(struct device *dev, const void *data)
> +{
> +	struct cxl_endpoint_decoder *cxled;
> +	struct cxl_port *port;
> +
> +	if (!is_endpoint_decoder(dev))
> +		return 0;
> +
> +	cxled = to_cxl_endpoint_decoder(dev);
> +	port = cxled_to_port(cxled);
> +
> +	return cxled->cxld.id == (port->hdm_end + 1);
> +}
> +
> +static struct cxl_endpoint_decoder *
> +cxl_find_free_decoder(struct cxl_memdev *cxlmd)
> +{
> +	struct cxl_port *endpoint = cxlmd->endpoint;
> +	struct device *dev;
> +
> +	guard(rwsem_read)(&cxl_rwsem.dpa);
> +	dev = device_find_child(&endpoint->dev, NULL,
> +				find_free_decoder);
> +	if (dev)
> +		return to_cxl_endpoint_decoder(dev);
> +
> +	return NULL;
> +}
> +
> +/**
> + * cxl_request_dpa - search and reserve DPA given input constraints
> + * @cxlmd: memdev with an endpoint port with available decoders
> + * @mode: CXL partition mode (ram vs pmem)
> + * @alloc: dpa size required
> + *
> + * Returns a pointer to a 'struct cxl_endpoint_decoder' on success or
> + * an errno encoded pointer on failure.
> + *
> + * Given that a region needs to allocate from limited HPA capacity it
> + * may be the case that a device has more mappable DPA capacity than
> + * available HPA. The expectation is that @alloc is a driver known
> + * value based on the device capacity but which could not be fully
> + * available due to HPA constraints.
> + *
> + * Returns a pinned cxl_decoder with at least @alloc bytes of capacity
> + * reserved, or an error pointer. The caller is also expected to own the
> + * lifetime of the memdev registration associated with the endpoint to
> + * pin the decoder registered as well.
> + */
> +struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
> +					     enum cxl_partition_mode mode,
> +					     resource_size_t alloc)
> +{
> +	int rc;
> +
> +	if (!IS_ALIGNED(alloc, SZ_256M))
> +		return ERR_PTR(-EINVAL);
> +
> +	struct cxl_endpoint_decoder *cxled __free(put_cxled) =
> +		cxl_find_free_decoder(cxlmd);
> +
> +	if (!cxled)
> +		return ERR_PTR(-ENODEV);
> +
> +	rc = cxl_dpa_set_part(cxled, mode);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	rc = cxl_dpa_alloc(cxled, alloc);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	return no_free_ptr(cxled);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_request_dpa, "CXL");
> +
>  static int __cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u64 size)
>  {
>  	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index ab490b5a9457..6ca0827cfaa5 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -625,6 +625,7 @@ struct cxl_root *find_cxl_root(struct cxl_port *port);
>  
>  DEFINE_FREE(put_cxl_root, struct cxl_root *, if (_T) put_device(&_T->port.dev))
>  DEFINE_FREE(put_cxl_port, struct cxl_port *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
> +DEFINE_FREE(put_cxled, struct cxl_endpoint_decoder *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->cxld.dev))
>  DEFINE_FREE(put_cxl_root_decoder, struct cxl_root_decoder *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->cxlsd.cxld.dev))
>  DEFINE_FREE(put_cxl_region, struct cxl_region *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
>  
> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
> index 2966b95e80a6..1cbe53ad0416 100644
> --- a/include/cxl/cxl.h
> +++ b/include/cxl/cxl.h
> @@ -7,6 +7,7 @@
>  
>  #include <linux/node.h>
>  #include <linux/ioport.h>
> +#include <linux/range.h>
>  #include <cxl/mailbox.h>
>  
>  /**
> @@ -270,4 +271,8 @@ struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
>  					       unsigned long flags,
>  					       resource_size_t *max);
>  void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd);
> +struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
> +					     enum cxl_partition_mode mode,
> +					     resource_size_t alloc);
> +int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
>  #endif /* __CXL_CXL_H__ */

Re: [PATCH v19 14/22] sfc: get endpoint decoder

[Dave Jiang]

On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
> From: Alejandro Lucero <alucerop@amd.com>
> 
> Use cxl api for getting DPA (Device Physical Address) to use through an
> endpoint decoder.
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Martin Habets <habetsm.xilinx@gmail.com>
> Acked-by: Edward Cree <ecree.xilinx@gmail.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
> ---
>  drivers/net/ethernet/sfc/efx_cxl.c | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
> index d7c34c978434..1a50bb2c0913 100644
> --- a/drivers/net/ethernet/sfc/efx_cxl.c
> +++ b/drivers/net/ethernet/sfc/efx_cxl.c
> @@ -108,6 +108,14 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>  		return -ENOSPC;
>  	}
>  
> +	cxl->cxled = cxl_request_dpa(cxl->cxlmd, CXL_PARTMODE_RAM,
> +				     EFX_CTPIO_BUFFER_SIZE);
> +	if (IS_ERR(cxl->cxled)) {
> +		pci_err(pci_dev, "CXL accel request DPA failed");
> +		cxl_put_root_decoder(cxl->cxlrd);

Might be good to create a __free() macro for freeing the root decoder instead of having to do this for every error path in this function.

DJ

> +		return PTR_ERR(cxl->cxled);
> +	}
> +
>  	probe_data->cxl = cxl;
>  
>  	return 0;
> @@ -115,8 +123,10 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>  
>  void efx_cxl_exit(struct efx_probe_data *probe_data)
>  {
> -	if (probe_data->cxl)
> +	if (probe_data->cxl) {
> +		cxl_dpa_free(probe_data->cxl->cxled);
>  		cxl_put_root_decoder(probe_data->cxl->cxlrd);
> +	}
>  }
>  
>  MODULE_IMPORT_NS("CXL");

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Dave Jiang]

On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
> From: Alejandro Lucero <alucerop@amd.com>
> 
> Creating a CXL region requires userspace intervention through the cxl
> sysfs files. Type2 support should allow accelerator drivers to create
> such cxl region from kernel code.
> 
> Adding that functionality and integrating it with current support for
> memory expanders.
> 
> Support an action by the type2 driver to be linked to the created region
> for unwinding the resources allocated properly.
> 
> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
> 
> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
>  drivers/cxl/core/core.h   |   5 --
>  drivers/cxl/core/region.c | 134 +++++++++++++++++++++++++++++++++++---
>  drivers/cxl/port.c        |   5 +-
>  include/cxl/cxl.h         |  11 ++++
>  4 files changed, 141 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index c4dddbec5d6e..83abaca9f418 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -14,11 +14,6 @@ extern const struct device_type cxl_pmu_type;
>  
>  extern struct attribute_group cxl_base_attribute_group;
>  
> -enum cxl_detach_mode {
> -	DETACH_ONLY,
> -	DETACH_INVALIDATE,
> -};
> -
>  #ifdef CONFIG_CXL_REGION
>  extern struct device_attribute dev_attr_create_pmem_region;
>  extern struct device_attribute dev_attr_create_ram_region;
> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> index 26dfc15e57cd..e3b6d85cd43e 100644
> --- a/drivers/cxl/core/region.c
> +++ b/drivers/cxl/core/region.c
> @@ -2375,6 +2375,7 @@ int cxl_decoder_detach(struct cxl_region *cxlr,
>  	}
>  	return 0;
>  }
> +EXPORT_SYMBOL_NS_GPL(cxl_decoder_detach, "CXL");
>  
>  static int __attach_target(struct cxl_region *cxlr,
>  			   struct cxl_endpoint_decoder *cxled, int pos,
> @@ -2860,6 +2861,14 @@ cxl_find_region_by_name(struct cxl_root_decoder *cxlrd, const char *name)
>  	return to_cxl_region(region_dev);
>  }
>  
> +static void drop_region(struct cxl_region *cxlr)
> +{
> +	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
> +	struct cxl_port *port = cxlrd_to_port(cxlrd);
> +
> +	devm_release_action(port->uport_dev, unregister_region, cxlr);
> +}
> +
>  static ssize_t delete_region_store(struct device *dev,
>  				   struct device_attribute *attr,
>  				   const char *buf, size_t len)
> @@ -3588,14 +3597,12 @@ static int __construct_region(struct cxl_region *cxlr,
>  	return 0;
>  }
>  
> -/* Establish an empty region covering the given HPA range */
> -static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
> -					   struct cxl_endpoint_decoder *cxled)
> +static struct cxl_region *construct_region_begin(struct cxl_root_decoder *cxlrd,
> +						 struct cxl_endpoint_decoder *cxled)
>  {
>  	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
> -	struct cxl_port *port = cxlrd_to_port(cxlrd);
>  	struct cxl_dev_state *cxlds = cxlmd->cxlds;
> -	int rc, part = READ_ONCE(cxled->part);
> +	int part = READ_ONCE(cxled->part);
>  	struct cxl_region *cxlr;
>  
>  	do {
> @@ -3604,13 +3611,24 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
>  				       cxled->cxld.target_type);
>  	} while (IS_ERR(cxlr) && PTR_ERR(cxlr) == -EBUSY);
>  
> -	if (IS_ERR(cxlr)) {
> +	if (IS_ERR(cxlr))
>  		dev_err(cxlmd->dev.parent,
>  			"%s:%s: %s failed assign region: %ld\n",
>  			dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev),
>  			__func__, PTR_ERR(cxlr));
> -		return cxlr;
> -	}
> +
> +	return cxlr;
> +}
> +
> +/* Establish an empty region covering the given HPA range */
> +static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
> +					   struct cxl_endpoint_decoder *cxled)
> +{
> +	struct cxl_port *port = cxlrd_to_port(cxlrd);
> +	struct cxl_region *cxlr;
> +	int rc;
> +
> +	cxlr = construct_region_begin(cxlrd, cxled);

Need to check the returned cxlr.

DJ

>  
>  	rc = __construct_region(cxlr, cxlrd, cxled);
>  	if (rc) {
> @@ -3621,6 +3639,106 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
>  	return cxlr;
>  }
>  
> +DEFINE_FREE(cxl_region_drop, struct cxl_region *, if (_T) drop_region(_T))
> +
> +static struct cxl_region *
> +__construct_new_region(struct cxl_root_decoder *cxlrd,
> +		       struct cxl_endpoint_decoder **cxled, int ways)
> +{
> +	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
> +	struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
> +	struct cxl_region_params *p;
> +	resource_size_t size = 0;
> +	int rc, i;
> +
> +	struct cxl_region *cxlr __free(cxl_region_drop) =
> +		construct_region_begin(cxlrd, cxled[0]);
> +	if (IS_ERR(cxlr))
> +		return cxlr;
> +
> +	guard(rwsem_write)(&cxl_rwsem.region);
> +
> +	/*
> +	 * Sanity check. This should not happen with an accel driver handling
> +	 * the region creation.
> +	 */
> +	p = &cxlr->params;
> +	if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
> +		dev_err(cxlmd->dev.parent,
> +			"%s:%s: %s  unexpected region state\n",
> +			dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
> +			__func__);
> +		return ERR_PTR(-EBUSY);
> +	}
> +
> +	rc = set_interleave_ways(cxlr, ways);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
> +		for (i = 0; i < ways; i++) {
> +			if (!cxled[i]->dpa_res)
> +				break;
> +			size += resource_size(cxled[i]->dpa_res);
> +		}
> +		if (i < ways)
> +			return ERR_PTR(-EINVAL);
> +
> +		rc = alloc_hpa(cxlr, size);
> +		if (rc)
> +			return ERR_PTR(rc);
> +
> +		for (i = 0; i < ways; i++) {
> +			rc = cxl_region_attach(cxlr, cxled[i], 0);
> +			if (rc)
> +				return ERR_PTR(rc);
> +		}
> +	}
> +
> +	rc = cxl_region_decode_commit(cxlr);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	p->state = CXL_CONFIG_COMMIT;
> +
> +	return no_free_ptr(cxlr);
> +}
> +
> +/**
> + * cxl_create_region - Establish a region given an endpoint decoder
> + * @cxlrd: root decoder to allocate HPA
> + * @cxled: endpoint decoders with reserved DPA capacity
> + * @ways: interleave ways required
> + *
> + * Returns a fully formed region in the commit state and attached to the
> + * cxl_region driver.
> + */
> +struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
> +				     struct cxl_endpoint_decoder **cxled,
> +				     int ways)
> +{
> +	struct cxl_region *cxlr;
> +
> +	mutex_lock(&cxlrd->range_lock);
> +	cxlr = __construct_new_region(cxlrd, cxled, ways);
> +	mutex_unlock(&cxlrd->range_lock);
> +	if (IS_ERR(cxlr))
> +		return cxlr;
> +
> +	if (device_attach(&cxlr->dev) <= 0) {
> +		dev_err(&cxlr->dev, "failed to create region\n");
> +		drop_region(cxlr);
> +		return ERR_PTR(-ENODEV);
> +	}
> +
> +	return cxlr;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_create_region, "CXL");
> +
>  static struct cxl_region *
>  cxl_find_region_by_range(struct cxl_root_decoder *cxlrd, struct range *hpa)
>  {
> diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c
> index 83f5a09839ab..e6c0bd0fc9f9 100644
> --- a/drivers/cxl/port.c
> +++ b/drivers/cxl/port.c
> @@ -35,6 +35,7 @@ static void schedule_detach(void *cxlmd)
>  static int discover_region(struct device *dev, void *unused)
>  {
>  	struct cxl_endpoint_decoder *cxled;
> +	struct cxl_memdev *cxlmd;
>  	int rc;
>  
>  	if (!is_endpoint_decoder(dev))
> @@ -44,7 +45,9 @@ static int discover_region(struct device *dev, void *unused)
>  	if ((cxled->cxld.flags & CXL_DECODER_F_ENABLE) == 0)
>  		return 0;
>  
> -	if (cxled->state != CXL_DECODER_STATE_AUTO)
> +	cxlmd = cxled_to_memdev(cxled);
> +	if (cxled->state != CXL_DECODER_STATE_AUTO ||
> +	    cxlmd->cxlds->type == CXL_DEVTYPE_DEVMEM)
>  		return 0;
>  
>  	/*
> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
> index 1cbe53ad0416..c6fd8fbd36c4 100644
> --- a/include/cxl/cxl.h
> +++ b/include/cxl/cxl.h
> @@ -275,4 +275,15 @@ struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
>  					     enum cxl_partition_mode mode,
>  					     resource_size_t alloc);
>  int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
> +struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
> +				     struct cxl_endpoint_decoder **cxled,
> +				     int ways);
> +enum cxl_detach_mode {
> +	DETACH_ONLY,
> +	DETACH_INVALIDATE,
> +};
> +
> +int cxl_decoder_detach(struct cxl_region *cxlr,
> +		       struct cxl_endpoint_decoder *cxled, int pos,
> +		       enum cxl_detach_mode mode);
>  #endif /* __CXL_CXL_H__ */

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Dave Jiang]

On 10/9/25 1:56 PM, Cheatham, Benjamin wrote:
> On 10/6/2025 5:01 AM, alejandro.lucero-palau@amd.com wrote:
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> Creating a CXL region requires userspace intervention through the cxl
>> sysfs files. Type2 support should allow accelerator drivers to create
>> such cxl region from kernel code.
>>
>> Adding that functionality and integrating it with current support for
>> memory expanders.
>>
>> Support an action by the type2 driver to be linked to the created region
>> for unwinding the resources allocated properly.
>>
>> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
> 
> Fix for this one should be split between 13/22 and this patch, but the majority of it is in this one. The idea is
> if we don't find a free decoder we check for pre-programmed decoders and use that instead. Unfortunately, this
> invalidates some of the assumptions made by __construct_new_region().

Wouldn't you look for a pre-programmed decoder first and construct the auto region before you try to manually create one? Also for a type 2 device, would the driver know what it wants and what the region configuration should look like? Would it be a single region either it's auto or manual, or would there be a configuration of multiple regions possible? To me a type 2 region is more intentional where the driver would know exactly what it needs and thus trying to get that from the cxl core. 

DJ


> 
> __construct_new_region() assumes that 1) the underlying HPA is unallocated and 2) the HDM decoders aren't programmed. Neither
> of those are true for a decoder that's programmed by BIOS. The HPA is allocated as part of endpoint_port_probe()
> (see devm_cxl_enumerate_decoders() in cxl/core/hdm.c specifically) and the HDM decoders are enabled and committed by BIOS before
> we ever see them. So the idea here is to split the second half of __construct_new_region() into the 2 cases: un-programmed decoders
> (__setup_new_region()) and pre-programmed decoders (__setup_new_auto_region()). The main differences between the two is we don't
> allocate the HPA region or commit the HDM decoders and just insert the region resource below the CXL window instead in the auto case.
> 
> I'm not sure if I've done everything correctly, but I don't see any errors and get the following iomem tree:
> 	1050000000-144fffffff : CXL Window 0
>   	  1050000000-144fffffff : region0
>     	    1050000000-144fffffff : Soft Reserved
> 
> ---
> 
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index 4af5de5e0a44..a5fa8dd0e63f 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -137,6 +137,8 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
>                         struct cxl_endpoint_dvsec_info *info);
>  int cxl_port_get_possible_dports(struct cxl_port *port);
> 
> +bool is_auto_decoder(struct cxl_endpoint_decoder *cxled);
> +
>  #ifdef CONFIG_CXL_FEATURES
>  struct cxl_feat_entry *
>  cxl_feature_info(struct cxl_features_state *cxlfs, const uuid_t *uuid);
> diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
> index 1f7aa79c1541..8f6236a88c0b 100644
> --- a/drivers/cxl/core/hdm.c
> +++ b/drivers/cxl/core/hdm.c
> @@ -712,16 +712,33 @@ static int find_free_decoder(struct device *dev, const void *data)
>         return 1;
>  }
> 
> +bool is_auto_decoder(struct cxl_endpoint_decoder *cxled)
> +{
> +       return cxled->state == CXL_DECODER_STATE_AUTO && cxled->pos < 0 &&
> +              (cxled->cxld.flags & CXL_DECODER_F_ENABLE);
> +}
> +
> +static int find_auto_decoder(struct device *dev, const void *data)
> +{
> +       if (!is_endpoint_decoder(dev))
> +               return 0;
> +
> +       return is_auto_decoder(to_cxl_endpoint_decoder(dev));
> +}
> +
>  static struct cxl_endpoint_decoder *
>  cxl_find_free_decoder(struct cxl_memdev *cxlmd)
>  {
>         struct cxl_port *endpoint = cxlmd->endpoint;
>         struct device *dev;
> 
> -       scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
> -               dev = device_find_child(&endpoint->dev, NULL,
> -                                       find_free_decoder);
> -       }
> +       guard(rwsem_read)(&cxl_rwsem.dpa);
> +       dev = device_find_child(&endpoint->dev, NULL,
> +                               find_free_decoder);
> +       if (dev)
> +               return to_cxl_endpoint_decoder(dev);
> +
> +       dev = device_find_child(&endpoint->dev, NULL, find_auto_decoder);
>         if (dev)
>                 return to_cxl_endpoint_decoder(dev);
> 
> @@ -761,6 +778,9 @@ struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
>         if (!cxled)
>                 return ERR_PTR(-ENODEV);
> 
> +       if (is_auto_decoder(cxled))
> +               return_ptr(cxled);
> +
>         rc = cxl_dpa_set_part(cxled, mode);
>         if (rc)
>                 return ERR_PTR(rc);
> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> index 2d60131edff3..004e01ad0e5f 100644
> --- a/drivers/cxl/core/region.c
> +++ b/drivers/cxl/core/region.c
> @@ -3699,48 +3699,74 @@ cxl_find_region_by_range(struct cxl_root_decoder *cxlrd, struct range *hpa)
>  }
> 
>  static struct cxl_region *
> -__construct_new_region(struct cxl_root_decoder *cxlrd,
> -                      struct cxl_endpoint_decoder **cxled, int ways)
> +__setup_new_auto_region(struct cxl_region *cxlr, struct cxl_root_decoder *cxlrd,
> +                       struct cxl_endpoint_decoder **cxled, int ways)
>  {
> -       struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
> -       struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
> -       struct cxl_region_params *p;
> +       struct range *hpa = &cxled[0]->cxld.hpa_range;
> +       struct cxl_region_params *p = &cxlr->params;
>         resource_size_t size = 0;
> -       struct cxl_region *cxlr;
> -       int rc, i;
> +       struct resource *res;
> +       int rc = -EINVAL, i = 0;
> 
> -       cxlr = construct_region_begin(cxlrd, cxled[0]);
> -       if (IS_ERR(cxlr))
> -               return cxlr;
> +       scoped_guard(rwsem_read, &cxl_rwsem.dpa)
> +       {
> +               for (i = 0; i < ways; i++) {
> +                       if (!cxled[i]->dpa_res)
> +                               goto err;
> 
> -       guard(rwsem_write)(&cxl_rwsem.region);
> +                       if (!is_auto_decoder(cxled[i]))
> +                               goto err;
> 
> -       /*
> -        * Sanity check. This should not happen with an accel driver handling
> -        * the region creation.
> -        */
> -       p = &cxlr->params;
> -       if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
> -               dev_err(cxlmd->dev.parent,
> -                       "%s:%s: %s  unexpected region state\n",
> -                       dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
> -                       __func__);
> -               rc = -EBUSY;
> -               goto err;
> +                       size += resource_size(cxled[i]->dpa_res);
> +               }
>         }
> 
> -       rc = set_interleave_ways(cxlr, ways);
> -       if (rc)
> +       set_bit(CXL_REGION_F_AUTO, &cxlr->flags);
> +
> +       p->res = kmalloc(sizeof(*res), GFP_KERNEL);
> +       if (!p->res) {
> +               rc = -ENOMEM;
>                 goto err;
> +       }
> 
> -       rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
> +       *p->res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa),
> +                                      dev_name(&cxlr->dev));
> +
> +       rc = insert_resource(cxlrd->res, p->res);
>         if (rc)
> -               goto err;
> +               dev_warn(&cxlr->dev, "Could not insert resource\n");
> +
> +       p->state = CXL_CONFIG_INTERLEAVE_ACTIVE;
> +       scoped_guard(rwsem_read, &cxl_rwsem.dpa)
> +       {
> +               for (i = 0; i < ways; i++) {
> +                       rc = cxl_region_attach(cxlr, cxled[i], -1);
> +                       if (rc)
> +                               goto err;
> +               }
> +       }
> +
> +       return cxlr;
> +
> +err:
> +       drop_region(cxlr);
> +       return ERR_PTR(rc);
> +}
> +
> +static struct cxl_region *
> +__setup_new_region(struct cxl_region *cxlr, struct cxl_root_decoder *cxlrd,
> +                  struct cxl_endpoint_decoder **cxled, int ways)
> +{
> +       struct cxl_region_params *p = &cxlr->params;
> +       resource_size_t size = 0;
> +       int rc = -EINVAL, i = 0;
> 
> -       scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
> +       scoped_guard(rwsem_read, &cxl_rwsem.dpa)
> +       {
>                 for (i = 0; i < ways; i++) {
>                         if (!cxled[i]->dpa_res)
>                                 break;
> +
>                         size += resource_size(cxled[i]->dpa_res);
>                 }
>         }
> @@ -3752,7 +3778,8 @@ __construct_new_region(struct cxl_root_decoder *cxlrd,
>         if (rc)
>                 goto err;
> 
> -       scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
> +       scoped_guard(rwsem_read, &cxl_rwsem.dpa)
> +       {
>                 for (i = 0; i < ways; i++) {
>                         rc = cxl_region_attach(cxlr, cxled[i], 0);
>                         if (rc)
> @@ -3760,16 +3787,61 @@ __construct_new_region(struct cxl_root_decoder *cxlrd,
>                 }
>         }
> 
> +       rc = cxl_region_decode_commit(cxlr);
>         if (rc)
>                 goto err;
> 
> -       rc = cxl_region_decode_commit(cxlr);
> +       p->state = CXL_CONFIG_COMMIT;
> +       return cxlr;
> +
> +err:
> +       drop_region(cxlr);
> +       return ERR_PTR(rc);
> +}
> +
> +static struct cxl_region *
> +__construct_new_region(struct cxl_root_decoder *cxlrd,
> +                      struct cxl_endpoint_decoder **cxled, int ways)
> +{
> +       struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
> +       struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
> +       struct cxl_region_params *p;
> +       struct cxl_region *cxlr;
> +       int rc;
> +
> +       cxlr = construct_region_begin(cxlrd, cxled[0]);
> +       if (IS_ERR(cxlr))
> +               return cxlr;
> +
> +       guard(rwsem_write)(&cxl_rwsem.region);
> +
> +       /*
> +        * Sanity check. This should not happen with an accel driver handling
> +        * the region creation.
> +        */
> +       p = &cxlr->params;
> +       if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
> +               dev_err(cxlmd->dev.parent,
> +                       "%s:%s: %s  unexpected region state\n",
> +                       dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
> +                       __func__);
> +               rc = -EBUSY;
> +               goto err;
> +       }
> +
> +       rc = set_interleave_ways(cxlr, ways);
>         if (rc)
>                 goto err;
> 
> -       p->state = CXL_CONFIG_COMMIT;
> +       rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
> +       if (rc)
> +               goto err;
> +
> +       if (is_auto_decoder(cxled[0]))
> +               return __setup_new_auto_region(cxlr, cxlrd, cxled, ways);
> +       else
> +               return __setup_new_region(cxlr, cxlrd, cxled, ways);
> 
> -       return cxlr;
>  err:
>         drop_region(cxlr);
>         return ERR_PTR(rc);

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Cheatham, Benjamin]

On 10/15/2025 4:42 PM, Dave Jiang wrote:
> 
> 
> On 10/9/25 1:56 PM, Cheatham, Benjamin wrote:
>> On 10/6/2025 5:01 AM, alejandro.lucero-palau@amd.com wrote:
>>> From: Alejandro Lucero <alucerop@amd.com>
>>>
>>> Creating a CXL region requires userspace intervention through the cxl
>>> sysfs files. Type2 support should allow accelerator drivers to create
>>> such cxl region from kernel code.
>>>
>>> Adding that functionality and integrating it with current support for
>>> memory expanders.
>>>
>>> Support an action by the type2 driver to be linked to the created region
>>> for unwinding the resources allocated properly.
>>>
>>> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
>>>
>>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>> ---
>>
>> Fix for this one should be split between 13/22 and this patch, but the majority of it is in this one. The idea is
>> if we don't find a free decoder we check for pre-programmed decoders and use that instead. Unfortunately, this
>> invalidates some of the assumptions made by __construct_new_region().
> 
> Wouldn't you look for a pre-programmed decoder first and construct the auto region before you try to manually create one? Also for a type 2 device, would the driver know what it wants and what the region configuration should look like? Would it be a single region either it's auto or manual, or would there be a configuration of multiple regions possible? To me a type 2 region is more intentional where the driver would know exactly what it needs and thus trying to get that from the cxl core. 
> 

Since this is a fix I didn't want to supersede the current behavior. A better solution would've been to add a flag to allow the type 2 driver
to set up an expected region type.

As for multiple regions, I have no clue. I haven't heard of any reason why a type 2 device would need multiple regions, but it's still very
early days. I don't think there's anything in this set that keeps you from using multiple regions though.

Thanks,
Ben

> DJ
> 

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Alejandro Lucero Palau]

On 10/15/25 22:36, Dave Jiang wrote:
>
> On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> Creating a CXL region requires userspace intervention through the cxl
>> sysfs files. Type2 support should allow accelerator drivers to create
>> such cxl region from kernel code.
>>
>> Adding that functionality and integrating it with current support for
>> memory expanders.
>>
>> Support an action by the type2 driver to be linked to the created region
>> for unwinding the resources allocated properly.
>>
>> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
>>   drivers/cxl/core/core.h   |   5 --
>>   drivers/cxl/core/region.c | 134 +++++++++++++++++++++++++++++++++++---
>>   drivers/cxl/port.c        |   5 +-
>>   include/cxl/cxl.h         |  11 ++++
>>   4 files changed, 141 insertions(+), 14 deletions(-)
>>
>> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
>> index c4dddbec5d6e..83abaca9f418 100644
>> --- a/drivers/cxl/core/core.h
>> +++ b/drivers/cxl/core/core.h
>> @@ -14,11 +14,6 @@ extern const struct device_type cxl_pmu_type;
>>   
>>   extern struct attribute_group cxl_base_attribute_group;
>>   
>> -enum cxl_detach_mode {
>> -	DETACH_ONLY,
>> -	DETACH_INVALIDATE,
>> -};
>> -
>>   #ifdef CONFIG_CXL_REGION
>>   extern struct device_attribute dev_attr_create_pmem_region;
>>   extern struct device_attribute dev_attr_create_ram_region;
>> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
>> index 26dfc15e57cd..e3b6d85cd43e 100644
>> --- a/drivers/cxl/core/region.c
>> +++ b/drivers/cxl/core/region.c
>> @@ -2375,6 +2375,7 @@ int cxl_decoder_detach(struct cxl_region *cxlr,
>>   	}
>>   	return 0;
>>   }
>> +EXPORT_SYMBOL_NS_GPL(cxl_decoder_detach, "CXL");
>>   
>>   static int __attach_target(struct cxl_region *cxlr,
>>   			   struct cxl_endpoint_decoder *cxled, int pos,
>> @@ -2860,6 +2861,14 @@ cxl_find_region_by_name(struct cxl_root_decoder *cxlrd, const char *name)
>>   	return to_cxl_region(region_dev);
>>   }
>>   
>> +static void drop_region(struct cxl_region *cxlr)
>> +{
>> +	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
>> +	struct cxl_port *port = cxlrd_to_port(cxlrd);
>> +
>> +	devm_release_action(port->uport_dev, unregister_region, cxlr);
>> +}
>> +
>>   static ssize_t delete_region_store(struct device *dev,
>>   				   struct device_attribute *attr,
>>   				   const char *buf, size_t len)
>> @@ -3588,14 +3597,12 @@ static int __construct_region(struct cxl_region *cxlr,
>>   	return 0;
>>   }
>>   
>> -/* Establish an empty region covering the given HPA range */
>> -static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
>> -					   struct cxl_endpoint_decoder *cxled)
>> +static struct cxl_region *construct_region_begin(struct cxl_root_decoder *cxlrd,
>> +						 struct cxl_endpoint_decoder *cxled)
>>   {
>>   	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
>> -	struct cxl_port *port = cxlrd_to_port(cxlrd);
>>   	struct cxl_dev_state *cxlds = cxlmd->cxlds;
>> -	int rc, part = READ_ONCE(cxled->part);
>> +	int part = READ_ONCE(cxled->part);
>>   	struct cxl_region *cxlr;
>>   
>>   	do {
>> @@ -3604,13 +3611,24 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
>>   				       cxled->cxld.target_type);
>>   	} while (IS_ERR(cxlr) && PTR_ERR(cxlr) == -EBUSY);
>>   
>> -	if (IS_ERR(cxlr)) {
>> +	if (IS_ERR(cxlr))
>>   		dev_err(cxlmd->dev.parent,
>>   			"%s:%s: %s failed assign region: %ld\n",
>>   			dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev),
>>   			__func__, PTR_ERR(cxlr));
>> -		return cxlr;
>> -	}
>> +
>> +	return cxlr;
>> +}
>> +
>> +/* Establish an empty region covering the given HPA range */
>> +static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
>> +					   struct cxl_endpoint_decoder *cxled)
>> +{
>> +	struct cxl_port *port = cxlrd_to_port(cxlrd);
>> +	struct cxl_region *cxlr;
>> +	int rc;
>> +
>> +	cxlr = construct_region_begin(cxlrd, cxled);
> Need to check the returned cxlr.


Hi Dave,


Yes, with the refactoring this was mistakenly dropped. I does not help 
this is the codepath not user by Type2 ...


I'll add the check.


Thank you


> DJ
>
>>   
>>   	rc = __construct_region(cxlr, cxlrd, cxled);
>>   	if (rc) {
>> @@ -3621,6 +3639,106 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
>>   	return cxlr;
>>   }
>>   
>> +DEFINE_FREE(cxl_region_drop, struct cxl_region *, if (_T) drop_region(_T))
>> +
>> +static struct cxl_region *
>> +__construct_new_region(struct cxl_root_decoder *cxlrd,
>> +		       struct cxl_endpoint_decoder **cxled, int ways)
>> +{
>> +	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
>> +	struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
>> +	struct cxl_region_params *p;
>> +	resource_size_t size = 0;
>> +	int rc, i;
>> +
>> +	struct cxl_region *cxlr __free(cxl_region_drop) =
>> +		construct_region_begin(cxlrd, cxled[0]);
>> +	if (IS_ERR(cxlr))
>> +		return cxlr;
>> +
>> +	guard(rwsem_write)(&cxl_rwsem.region);
>> +
>> +	/*
>> +	 * Sanity check. This should not happen with an accel driver handling
>> +	 * the region creation.
>> +	 */
>> +	p = &cxlr->params;
>> +	if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
>> +		dev_err(cxlmd->dev.parent,
>> +			"%s:%s: %s  unexpected region state\n",
>> +			dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
>> +			__func__);
>> +		return ERR_PTR(-EBUSY);
>> +	}
>> +
>> +	rc = set_interleave_ways(cxlr, ways);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
>> +		for (i = 0; i < ways; i++) {
>> +			if (!cxled[i]->dpa_res)
>> +				break;
>> +			size += resource_size(cxled[i]->dpa_res);
>> +		}
>> +		if (i < ways)
>> +			return ERR_PTR(-EINVAL);
>> +
>> +		rc = alloc_hpa(cxlr, size);
>> +		if (rc)
>> +			return ERR_PTR(rc);
>> +
>> +		for (i = 0; i < ways; i++) {
>> +			rc = cxl_region_attach(cxlr, cxled[i], 0);
>> +			if (rc)
>> +				return ERR_PTR(rc);
>> +		}
>> +	}
>> +
>> +	rc = cxl_region_decode_commit(cxlr);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	p->state = CXL_CONFIG_COMMIT;
>> +
>> +	return no_free_ptr(cxlr);
>> +}
>> +
>> +/**
>> + * cxl_create_region - Establish a region given an endpoint decoder
>> + * @cxlrd: root decoder to allocate HPA
>> + * @cxled: endpoint decoders with reserved DPA capacity
>> + * @ways: interleave ways required
>> + *
>> + * Returns a fully formed region in the commit state and attached to the
>> + * cxl_region driver.
>> + */
>> +struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
>> +				     struct cxl_endpoint_decoder **cxled,
>> +				     int ways)
>> +{
>> +	struct cxl_region *cxlr;
>> +
>> +	mutex_lock(&cxlrd->range_lock);
>> +	cxlr = __construct_new_region(cxlrd, cxled, ways);
>> +	mutex_unlock(&cxlrd->range_lock);
>> +	if (IS_ERR(cxlr))
>> +		return cxlr;
>> +
>> +	if (device_attach(&cxlr->dev) <= 0) {
>> +		dev_err(&cxlr->dev, "failed to create region\n");
>> +		drop_region(cxlr);
>> +		return ERR_PTR(-ENODEV);
>> +	}
>> +
>> +	return cxlr;
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_create_region, "CXL");
>> +
>>   static struct cxl_region *
>>   cxl_find_region_by_range(struct cxl_root_decoder *cxlrd, struct range *hpa)
>>   {
>> diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c
>> index 83f5a09839ab..e6c0bd0fc9f9 100644
>> --- a/drivers/cxl/port.c
>> +++ b/drivers/cxl/port.c
>> @@ -35,6 +35,7 @@ static void schedule_detach(void *cxlmd)
>>   static int discover_region(struct device *dev, void *unused)
>>   {
>>   	struct cxl_endpoint_decoder *cxled;
>> +	struct cxl_memdev *cxlmd;
>>   	int rc;
>>   
>>   	if (!is_endpoint_decoder(dev))
>> @@ -44,7 +45,9 @@ static int discover_region(struct device *dev, void *unused)
>>   	if ((cxled->cxld.flags & CXL_DECODER_F_ENABLE) == 0)
>>   		return 0;
>>   
>> -	if (cxled->state != CXL_DECODER_STATE_AUTO)
>> +	cxlmd = cxled_to_memdev(cxled);
>> +	if (cxled->state != CXL_DECODER_STATE_AUTO ||
>> +	    cxlmd->cxlds->type == CXL_DEVTYPE_DEVMEM)
>>   		return 0;
>>   
>>   	/*
>> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
>> index 1cbe53ad0416..c6fd8fbd36c4 100644
>> --- a/include/cxl/cxl.h
>> +++ b/include/cxl/cxl.h
>> @@ -275,4 +275,15 @@ struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
>>   					     enum cxl_partition_mode mode,
>>   					     resource_size_t alloc);
>>   int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
>> +struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd,
>> +				     struct cxl_endpoint_decoder **cxled,
>> +				     int ways);
>> +enum cxl_detach_mode {
>> +	DETACH_ONLY,
>> +	DETACH_INVALIDATE,
>> +};
>> +
>> +int cxl_decoder_detach(struct cxl_region *cxlr,
>> +		       struct cxl_endpoint_decoder *cxled, int pos,
>> +		       enum cxl_detach_mode mode);
>>   #endif /* __CXL_CXL_H__ */
>

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Alejandro Lucero Palau]

On 10/16/25 14:23, Cheatham, Benjamin wrote:
> On 10/15/2025 4:42 PM, Dave Jiang wrote:
>>
>> On 10/9/25 1:56 PM, Cheatham, Benjamin wrote:
>>> On 10/6/2025 5:01 AM, alejandro.lucero-palau@amd.com wrote:
>>>> From: Alejandro Lucero <alucerop@amd.com>
>>>>
>>>> Creating a CXL region requires userspace intervention through the cxl
>>>> sysfs files. Type2 support should allow accelerator drivers to create
>>>> such cxl region from kernel code.
>>>>
>>>> Adding that functionality and integrating it with current support for
>>>> memory expanders.
>>>>
>>>> Support an action by the type2 driver to be linked to the created region
>>>> for unwinding the resources allocated properly.
>>>>
>>>> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
>>>>
>>>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>>>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>>> ---
>>> Fix for this one should be split between 13/22 and this patch, but the majority of it is in this one. The idea is
>>> if we don't find a free decoder we check for pre-programmed decoders and use that instead. Unfortunately, this
>>> invalidates some of the assumptions made by __construct_new_region().
>> Wouldn't you look for a pre-programmed decoder first and construct the auto region before you try to manually create one? Also for a type 2 device, would the driver know what it wants and what the region configuration should look like? Would it be a single region either it's auto or manual, or would there be a configuration of multiple regions possible? To me a type 2 region is more intentional where the driver would know exactly what it needs and thus trying to get that from the cxl core.
>>
> Since this is a fix I didn't want to supersede the current behavior. A better solution would've been to add a flag to allow the type 2 driver
> to set up an expected region type.
>
> As for multiple regions, I have no clue. I haven't heard of any reason why a type 2 device would need multiple regions, but it's still very
> early days. I don't think there's anything in this set that keeps you from using multiple regions though.


What Dave says is correct, so Type2 should  support these two 
possibilities, an HDM decoder already programmed by the BIOS and the 
BIOS doing nothing, at least with the Type2 HDM decoders. This patchset 
supports the latter, but the former is more than possible, even if the 
logic and what we have discussed since the RFC points to type2 driver 
having the full control.


However, I would prefer to do that other support as a follow-up as the 
functionality added is enough for the initial client, the sfc driver, 
requiring this new Type2 support. The reason is clear: I do not want to 
delay this "basic Type2 support" more than necessary, and as I stated in 
another comment, I bet we will see other things to support soon, so 
better to increasingly add those after a first work set the base. Of 
course, the base needs to be right.


Thanks,

Alejandro


> Thanks,
> Ben
>
>> DJ
>>

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Dave Jiang]

On 10/20/25 6:24 AM, Alejandro Lucero Palau wrote:
> 
> On 10/16/25 14:23, Cheatham, Benjamin wrote:
>> On 10/15/2025 4:42 PM, Dave Jiang wrote:
>>>
>>> On 10/9/25 1:56 PM, Cheatham, Benjamin wrote:
>>>> On 10/6/2025 5:01 AM, alejandro.lucero-palau@amd.com wrote:
>>>>> From: Alejandro Lucero <alucerop@amd.com>
>>>>>
>>>>> Creating a CXL region requires userspace intervention through the cxl
>>>>> sysfs files. Type2 support should allow accelerator drivers to create
>>>>> such cxl region from kernel code.
>>>>>
>>>>> Adding that functionality and integrating it with current support for
>>>>> memory expanders.
>>>>>
>>>>> Support an action by the type2 driver to be linked to the created region
>>>>> for unwinding the resources allocated properly.
>>>>>
>>>>> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
>>>>>
>>>>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>>>>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>>>> ---
>>>> Fix for this one should be split between 13/22 and this patch, but the majority of it is in this one. The idea is
>>>> if we don't find a free decoder we check for pre-programmed decoders and use that instead. Unfortunately, this
>>>> invalidates some of the assumptions made by __construct_new_region().
>>> Wouldn't you look for a pre-programmed decoder first and construct the auto region before you try to manually create one? Also for a type 2 device, would the driver know what it wants and what the region configuration should look like? Would it be a single region either it's auto or manual, or would there be a configuration of multiple regions possible? To me a type 2 region is more intentional where the driver would know exactly what it needs and thus trying to get that from the cxl core.
>>>
>> Since this is a fix I didn't want to supersede the current behavior. A better solution would've been to add a flag to allow the type 2 driver
>> to set up an expected region type.
>>
>> As for multiple regions, I have no clue. I haven't heard of any reason why a type 2 device would need multiple regions, but it's still very
>> early days. I don't think there's anything in this set that keeps you from using multiple regions though.
> 
> 
> What Dave says is correct, so Type2 should  support these two possibilities, an HDM decoder already programmed by the BIOS and the BIOS doing nothing, at least with the Type2 HDM decoders. This patchset supports the latter, but the former is more than possible, even if the logic and what we have discussed since the RFC points to type2 driver having the full control.
> 
> 
> However, I would prefer to do that other support as a follow-up as the functionality added is enough for the initial client, the sfc driver, requiring this new Type2 support. The reason is clear: I do not want to delay this "basic Type2 support" more than necessary, and as I stated in another comment, I bet we will see other things to support soon, so better to increasingly add those after a first work set the base. Of course, the base needs to be right.

I'm fine with the follow on approach. We probably should make a note somewhere in a commit log somewhere that only manual assemble mode is currently supported. And need to reject the BIOS created region and exit type2 CXL setup if present?

DJ

> 
> 
> Thanks,
> 
> Alejandro
> 
> 
>> Thanks,
>> Ben
>>
>>> DJ
>>>

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Alejandro Lucero Palau]

On 10/20/25 14:59, Dave Jiang wrote:
>
> On 10/20/25 6:24 AM, Alejandro Lucero Palau wrote:
>> On 10/16/25 14:23, Cheatham, Benjamin wrote:
>>> On 10/15/2025 4:42 PM, Dave Jiang wrote:
>>>> On 10/9/25 1:56 PM, Cheatham, Benjamin wrote:
>>>>> On 10/6/2025 5:01 AM, alejandro.lucero-palau@amd.com wrote:
>>>>>> From: Alejandro Lucero <alucerop@amd.com>
>>>>>>
>>>>>> Creating a CXL region requires userspace intervention through the cxl
>>>>>> sysfs files. Type2 support should allow accelerator drivers to create
>>>>>> such cxl region from kernel code.
>>>>>>
>>>>>> Adding that functionality and integrating it with current support for
>>>>>> memory expanders.
>>>>>>
>>>>>> Support an action by the type2 driver to be linked to the created region
>>>>>> for unwinding the resources allocated properly.
>>>>>>
>>>>>> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
>>>>>>
>>>>>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>>>>>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>>>>> ---
>>>>> Fix for this one should be split between 13/22 and this patch, but the majority of it is in this one. The idea is
>>>>> if we don't find a free decoder we check for pre-programmed decoders and use that instead. Unfortunately, this
>>>>> invalidates some of the assumptions made by __construct_new_region().
>>>> Wouldn't you look for a pre-programmed decoder first and construct the auto region before you try to manually create one? Also for a type 2 device, would the driver know what it wants and what the region configuration should look like? Would it be a single region either it's auto or manual, or would there be a configuration of multiple regions possible? To me a type 2 region is more intentional where the driver would know exactly what it needs and thus trying to get that from the cxl core.
>>>>
>>> Since this is a fix I didn't want to supersede the current behavior. A better solution would've been to add a flag to allow the type 2 driver
>>> to set up an expected region type.
>>>
>>> As for multiple regions, I have no clue. I haven't heard of any reason why a type 2 device would need multiple regions, but it's still very
>>> early days. I don't think there's anything in this set that keeps you from using multiple regions though.
>>
>> What Dave says is correct, so Type2 should  support these two possibilities, an HDM decoder already programmed by the BIOS and the BIOS doing nothing, at least with the Type2 HDM decoders. This patchset supports the latter, but the former is more than possible, even if the logic and what we have discussed since the RFC points to type2 driver having the full control.
>>
>>
>> However, I would prefer to do that other support as a follow-up as the functionality added is enough for the initial client, the sfc driver, requiring this new Type2 support. The reason is clear: I do not want to delay this "basic Type2 support" more than necessary, and as I stated in another comment, I bet we will see other things to support soon, so better to increasingly add those after a first work set the base. Of course, the base needs to be right.
> I'm fine with the follow on approach. We probably should make a note somewhere in a commit log somewhere that only manual assemble mode is currently supported. And need to reject the BIOS created region and exit type2 CXL setup if present?


Yes, it makes sense, and it is a minor change.


I'll do so.


Thanks


>
> DJ
>
>>
>> Thanks,
>>
>> Alejandro
>>
>>
>>> Thanks,
>>> Ben
>>>
>>>> DJ
>>>>

Re: [PATCH v19 01/22] cxl/mem: Arrange for always-synchronous memdev attach

[Alejandro Lucero Palau]

On 10/7/25 13:40, Jonathan Cameron wrote:
> On Mon, 6 Oct 2025 11:01:09 +0100
> <alejandro.lucero-palau@amd.com> wrote:
>
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> In preparation for CXL accelerator drivers that have a hard dependency on
>> CXL capability initialization, arrange for the endpoint probe result to be
>> conveyed to the caller of devm_cxl_add_memdev().
>>
>> As it stands cxl_pci does not care about the attach state of the cxl_memdev
>> because all generic memory expansion functionality can be handled by the
>> cxl_core. For accelerators, that driver needs to know perform driver
>> specific initialization if CXL is available, or exectute a fallback to PCIe
>> only operation.
>>
>> By moving devm_cxl_add_memdev() to cxl_mem.ko it removes async module
>> loading as one reason that a memdev may not be attached upon return from
>> devm_cxl_add_memdev().
>>
>> The diff is busy as this moves cxl_memdev_alloc() down below the definition
>> of cxl_memdev_fops and introduces devm_cxl_memdev_add_or_reset() to
>> preclude needing to export more symbols from the cxl_core.
>>
>> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> Alejandro, SoB chain broken here which makes this currently unmergeable.
>
> Should definitely have your SoB as you sent the patch to the list and need
> to make a statement that you believe it to be fine to do so (see the Certificate
> of origin stuff in the docs).  Also, From should always be one of the authors.
> If Dan wrote this as the SoB suggests then From should be set to him..
>
> git commit --amend --author="Dan Williams <dan.j.williams@intel.com>"
>
> Will fix that up.  Then either you add your SoB on basis you just 'handled'
> the patch but didn't make substantial changes, or your SoB and a Codeveloped-by
> if you did make major changes.  If it is minor stuff you can an
> a sign off with # what changed
> comment next to it.


Understood. I'll ask Dan what he prefers.


>
> A few minor comments inline.
>
> Thanks,
>
> Jonathan
>
>
>> ---
>>   drivers/cxl/Kconfig       |  2 +-
>>   drivers/cxl/core/memdev.c | 97 ++++++++++++++++-----------------------
>>   drivers/cxl/mem.c         | 30 ++++++++++++
>>   drivers/cxl/private.h     | 11 +++++
>>   4 files changed, 82 insertions(+), 58 deletions(-)
>>   create mode 100644 drivers/cxl/private.h
>>
>> diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
>> index 028201e24523..111e05615f09 100644
>> --- a/drivers/cxl/Kconfig
>> +++ b/drivers/cxl/Kconfig
>> @@ -22,6 +22,7 @@ if CXL_BUS
>>   config CXL_PCI
>>   	tristate "PCI manageability"
>>   	default CXL_BUS
>> +	select CXL_MEM
>>   	help
>>   	  The CXL specification defines a "CXL memory device" sub-class in the
>>   	  PCI "memory controller" base class of devices. Device's identified by
>> @@ -89,7 +90,6 @@ config CXL_PMEM
>>   
>>   config CXL_MEM
>>   	tristate "CXL: Memory Expansion"
>> -	depends on CXL_PCI
>>   	default CXL_BUS
>>   	help
>>   	  The CXL.mem protocol allows a device to act as a provider of "System
>> diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
>> index c569e00a511f..2bef231008df 100644
>> --- a/drivers/cxl/core/memdev.c
>> +++ b/drivers/cxl/core/memdev.c
>> -
>> -err:
>> -	kfree(cxlmd);
>> -	return ERR_PTR(rc);
>>   }
>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_memdev_add_or_reset, "CXL");
>>   
>>   static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
>>   			       unsigned long arg)
>> @@ -1023,50 +1012,44 @@ static const struct file_operations cxl_memdev_fops = {
>>   	.llseek = noop_llseek,
>>   };
>>   
>> -struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>> -				       struct cxl_dev_state *cxlds)
>> +struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds)
>>   {
>>   	struct cxl_memdev *cxlmd;
>>   	struct device *dev;
>>   	struct cdev *cdev;
>>   	int rc;
>>   
>> -	cxlmd = cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
>> -	if (IS_ERR(cxlmd))
>> -		return cxlmd;
>> +	cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
> It's a little bit non obvious due to the device initialize mid way
> through this, but given there are no error paths after that you can
> currently just do.
> 	struct cxl_memdev *cxlmd __free(kfree) =
> 		cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
> and
> 	return_ptr(cxlmd);
>
> in the good path.  That lets you then just return rather than having
> the goto err: handling for the error case that currently frees this
> manually.
>
> Unlike the change below, this one I think is definitely worth making.


I agree so I'll do it. The below suggestion is also needed ...


>
>> +	if (!cxlmd)
>> +		return ERR_PTR(-ENOMEM);
>>   
>> -	dev = &cxlmd->dev;
>> -	rc = dev_set_name(dev, "mem%d", cxlmd->id);
>> -	if (rc)
>> +	rc = ida_alloc_max(&cxl_memdev_ida, CXL_MEM_MAX_DEVS - 1, GFP_KERNEL);
>> +	if (rc < 0)
>>   		goto err;
>> -
>> -	/*
>> -	 * Activate ioctl operations, no cxl_memdev_rwsem manipulation
>> -	 * needed as this is ordered with cdev_add() publishing the device.
>> -	 */
>> +	cxlmd->id = rc;
>> +	cxlmd->depth = -1;
>>   	cxlmd->cxlds = cxlds;
>>   	cxlds->cxlmd = cxlmd;
>>   
>> -	cdev = &cxlmd->cdev;
>> -	rc = cdev_device_add(cdev, dev);
>> -	if (rc)
>> -		goto err;
>> +	dev = &cxlmd->dev;
>> +	device_initialize(dev);
>> +	lockdep_set_class(&dev->mutex, &cxl_memdev_key);
>> +	dev->parent = cxlds->dev;
>> +	dev->bus = &cxl_bus_type;
>> +	dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
>> +	dev->type = &cxl_memdev_type;
>> +	device_set_pm_not_required(dev);
>> +	INIT_WORK(&cxlmd->detach_work, detach_memdev);
>>   
>> -	rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
>> -	if (rc)
>> -		return ERR_PTR(rc);
>> +	cdev = &cxlmd->cdev;
>> +	cdev_init(cdev, &cxl_memdev_fops);
>>   	return cxlmd;
>>   
>>   err:
>> -	/*
>> -	 * The cdev was briefly live, shutdown any ioctl operations that
>> -	 * saw that state.
>> -	 */
>> -	cxl_memdev_shutdown(dev);
>> -	put_device(dev);
>> +	kfree(cxlmd);
>>   	return ERR_PTR(rc);
>>   }
>> -EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
>> +EXPORT_SYMBOL_NS_GPL(cxl_memdev_alloc, "CXL");
>>   
>>   static void sanitize_teardown_notifier(void *data)
>>   {
>> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
>> index f7dc0ba8905d..144749b9c818 100644
>> --- a/drivers/cxl/mem.c
>> +++ b/drivers/cxl/mem.c
>> @@ -7,6 +7,7 @@
>>   
>>   #include "cxlmem.h"
>>   #include "cxlpci.h"
>> +#include "private.h"
>>   #include "core/core.h"
>>   
>>   /**
>> @@ -203,6 +204,34 @@ static int cxl_mem_probe(struct device *dev)
>>   	return devm_add_action_or_reset(dev, enable_suspend, NULL);
>>   }
>>   
>> +/**
>> + * devm_cxl_add_memdev - Add a CXL memory device
>> + * @host: devres alloc/release context and parent for the memdev
>> + * @cxlds: CXL device state to associate with the memdev
>> + *
>> + * Upon return the device will have had a chance to attach to the
>> + * cxl_mem driver, but may fail if the CXL topology is not ready
>> + * (hardware CXL link down, or software platform CXL root not attached)
>> + */
>> +struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>> +				       struct cxl_dev_state *cxlds)
>> +{
>> +	struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds);
> Bit marginal but you could do a DEFINE_FREE() for cxlmd
> similar to the one that exists for put_cxl_port
>
> You would then need to steal the pointer for the devm_ call at the
> end of this function.


We are not freeing cxlmd in case of errors after we got the allocation, 
so I think it makes sense.


Thank you.


>
>> +	int rc;
>> +
>> +	if (IS_ERR(cxlmd))
>> +		return cxlmd;
>> +
>> +	rc = dev_set_name(&cxlmd->dev, "mem%d", cxlmd->id);
>> +	if (rc) {
>> +		put_device(&cxlmd->dev);
>> +		return ERR_PTR(rc);
>> +	}
>> +
>> +	return devm_cxl_memdev_add_or_reset(host, cxlmd);
>> +}
>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");

Re: [PATCH v19 01/22] cxl/mem: Arrange for always-synchronous memdev attach

[Koralahalli Channabasappa, Smita]

Hi Alejandro,

I need patches 1–3 from this series as prerequisites for the 
Soft-Reserved coordination work, so I wanted to check in on your plans 
for the next revision.

Link to the discussion: 
https://lore.kernel.org/all/aPbOfFPIhtu5npaG@aschofie-mobl2.lan/

Are patches 1–3 already being updated as part of your v20 work?
If so, I can wait and pick them up from v20 directly.

If v20 is still in progress and may take some time, I can probably carry 
patches 1–3 at the start of my series, and if that helps, I can fold in 
the review comments by Jonathan while keeping authorship as is. I would 
only adjust wording in the commit descriptions to reflect the 
Soft-Reserved coordination context.

Alternatively, if you prefer to continue carrying them in the Type-2 
series, I can simply reference them as prerequisites instead.

I’m fine with either approach just trying to avoid duplicated effort and 
keep review in one place.

Thanks
Smita

On 10/29/2025 4:20 AM, Alejandro Lucero Palau wrote:
> 
> On 10/7/25 13:40, Jonathan Cameron wrote:
>> On Mon, 6 Oct 2025 11:01:09 +0100
>> <alejandro.lucero-palau@amd.com> wrote:
>>
>>> From: Alejandro Lucero <alucerop@amd.com>
>>>
>>> In preparation for CXL accelerator drivers that have a hard 
>>> dependency on
>>> CXL capability initialization, arrange for the endpoint probe result 
>>> to be
>>> conveyed to the caller of devm_cxl_add_memdev().
>>>
>>> As it stands cxl_pci does not care about the attach state of the 
>>> cxl_memdev
>>> because all generic memory expansion functionality can be handled by the
>>> cxl_core. For accelerators, that driver needs to know perform driver
>>> specific initialization if CXL is available, or exectute a fallback 
>>> to PCIe
>>> only operation.
>>>
>>> By moving devm_cxl_add_memdev() to cxl_mem.ko it removes async module
>>> loading as one reason that a memdev may not be attached upon return from
>>> devm_cxl_add_memdev().
>>>
>>> The diff is busy as this moves cxl_memdev_alloc() down below the 
>>> definition
>>> of cxl_memdev_fops and introduces devm_cxl_memdev_add_or_reset() to
>>> preclude needing to export more symbols from the cxl_core.
>>>
>>> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
>> Alejandro, SoB chain broken here which makes this currently unmergeable.
>>
>> Should definitely have your SoB as you sent the patch to the list and 
>> need
>> to make a statement that you believe it to be fine to do so (see the 
>> Certificate
>> of origin stuff in the docs).  Also, From should always be one of the 
>> authors.
>> If Dan wrote this as the SoB suggests then From should be set to him..
>>
>> git commit --amend --author="Dan Williams <dan.j.williams@intel.com>"
>>
>> Will fix that up.  Then either you add your SoB on basis you just 
>> 'handled'
>> the patch but didn't make substantial changes, or your SoB and a 
>> Codeveloped-by
>> if you did make major changes.  If it is minor stuff you can an
>> a sign off with # what changed
>> comment next to it.
> 
> 
> Understood. I'll ask Dan what he prefers.
> 
> 
>>
>> A few minor comments inline.
>>
>> Thanks,
>>
>> Jonathan
>>
>>
>>> ---
>>>   drivers/cxl/Kconfig       |  2 +-
>>>   drivers/cxl/core/memdev.c | 97 ++++++++++++++++-----------------------
>>>   drivers/cxl/mem.c         | 30 ++++++++++++
>>>   drivers/cxl/private.h     | 11 +++++
>>>   4 files changed, 82 insertions(+), 58 deletions(-)
>>>   create mode 100644 drivers/cxl/private.h
>>>
>>> diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
>>> index 028201e24523..111e05615f09 100644
>>> --- a/drivers/cxl/Kconfig
>>> +++ b/drivers/cxl/Kconfig
>>> @@ -22,6 +22,7 @@ if CXL_BUS
>>>   config CXL_PCI
>>>       tristate "PCI manageability"
>>>       default CXL_BUS
>>> +    select CXL_MEM
>>>       help
>>>         The CXL specification defines a "CXL memory device" sub-class 
>>> in the
>>>         PCI "memory controller" base class of devices. Device's 
>>> identified by
>>> @@ -89,7 +90,6 @@ config CXL_PMEM
>>>   config CXL_MEM
>>>       tristate "CXL: Memory Expansion"
>>> -    depends on CXL_PCI
>>>       default CXL_BUS
>>>       help
>>>         The CXL.mem protocol allows a device to act as a provider of 
>>> "System
>>> diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
>>> index c569e00a511f..2bef231008df 100644
>>> --- a/drivers/cxl/core/memdev.c
>>> +++ b/drivers/cxl/core/memdev.c
>>> -
>>> -err:
>>> -    kfree(cxlmd);
>>> -    return ERR_PTR(rc);
>>>   }
>>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_memdev_add_or_reset, "CXL");
>>>   static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned 
>>> int cmd,
>>>                      unsigned long arg)
>>> @@ -1023,50 +1012,44 @@ static const struct file_operations 
>>> cxl_memdev_fops = {
>>>       .llseek = noop_llseek,
>>>   };
>>> -struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>>> -                       struct cxl_dev_state *cxlds)
>>> +struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds)
>>>   {
>>>       struct cxl_memdev *cxlmd;
>>>       struct device *dev;
>>>       struct cdev *cdev;
>>>       int rc;
>>> -    cxlmd = cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
>>> -    if (IS_ERR(cxlmd))
>>> -        return cxlmd;
>>> +    cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
>> It's a little bit non obvious due to the device initialize mid way
>> through this, but given there are no error paths after that you can
>> currently just do.
>>     struct cxl_memdev *cxlmd __free(kfree) =
>>         cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
>> and
>>     return_ptr(cxlmd);
>>
>> in the good path.  That lets you then just return rather than having
>> the goto err: handling for the error case that currently frees this
>> manually.
>>
>> Unlike the change below, this one I think is definitely worth making.
> 
> 
> I agree so I'll do it. The below suggestion is also needed ...
> 
> 
>>
>>> +    if (!cxlmd)
>>> +        return ERR_PTR(-ENOMEM);
>>> -    dev = &cxlmd->dev;
>>> -    rc = dev_set_name(dev, "mem%d", cxlmd->id);
>>> -    if (rc)
>>> +    rc = ida_alloc_max(&cxl_memdev_ida, CXL_MEM_MAX_DEVS - 1, 
>>> GFP_KERNEL);
>>> +    if (rc < 0)
>>>           goto err;
>>> -
>>> -    /*
>>> -     * Activate ioctl operations, no cxl_memdev_rwsem manipulation
>>> -     * needed as this is ordered with cdev_add() publishing the device.
>>> -     */
>>> +    cxlmd->id = rc;
>>> +    cxlmd->depth = -1;
>>>       cxlmd->cxlds = cxlds;
>>>       cxlds->cxlmd = cxlmd;
>>> -    cdev = &cxlmd->cdev;
>>> -    rc = cdev_device_add(cdev, dev);
>>> -    if (rc)
>>> -        goto err;
>>> +    dev = &cxlmd->dev;
>>> +    device_initialize(dev);
>>> +    lockdep_set_class(&dev->mutex, &cxl_memdev_key);
>>> +    dev->parent = cxlds->dev;
>>> +    dev->bus = &cxl_bus_type;
>>> +    dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
>>> +    dev->type = &cxl_memdev_type;
>>> +    device_set_pm_not_required(dev);
>>> +    INIT_WORK(&cxlmd->detach_work, detach_memdev);
>>> -    rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
>>> -    if (rc)
>>> -        return ERR_PTR(rc);
>>> +    cdev = &cxlmd->cdev;
>>> +    cdev_init(cdev, &cxl_memdev_fops);
>>>       return cxlmd;
>>>   err:
>>> -    /*
>>> -     * The cdev was briefly live, shutdown any ioctl operations that
>>> -     * saw that state.
>>> -     */
>>> -    cxl_memdev_shutdown(dev);
>>> -    put_device(dev);
>>> +    kfree(cxlmd);
>>>       return ERR_PTR(rc);
>>>   }
>>> -EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
>>> +EXPORT_SYMBOL_NS_GPL(cxl_memdev_alloc, "CXL");
>>>   static void sanitize_teardown_notifier(void *data)
>>>   {
>>> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
>>> index f7dc0ba8905d..144749b9c818 100644
>>> --- a/drivers/cxl/mem.c
>>> +++ b/drivers/cxl/mem.c
>>> @@ -7,6 +7,7 @@
>>>   #include "cxlmem.h"
>>>   #include "cxlpci.h"
>>> +#include "private.h"
>>>   #include "core/core.h"
>>>   /**
>>> @@ -203,6 +204,34 @@ static int cxl_mem_probe(struct device *dev)
>>>       return devm_add_action_or_reset(dev, enable_suspend, NULL);
>>>   }
>>> +/**
>>> + * devm_cxl_add_memdev - Add a CXL memory device
>>> + * @host: devres alloc/release context and parent for the memdev
>>> + * @cxlds: CXL device state to associate with the memdev
>>> + *
>>> + * Upon return the device will have had a chance to attach to the
>>> + * cxl_mem driver, but may fail if the CXL topology is not ready
>>> + * (hardware CXL link down, or software platform CXL root not attached)
>>> + */
>>> +struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>>> +                       struct cxl_dev_state *cxlds)
>>> +{
>>> +    struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds);
>> Bit marginal but you could do a DEFINE_FREE() for cxlmd
>> similar to the one that exists for put_cxl_port
>>
>> You would then need to steal the pointer for the devm_ call at the
>> end of this function.
> 
> 
> We are not freeing cxlmd in case of errors after we got the allocation, 
> so I think it makes sense.
> 
> 
> Thank you.
> 
> 
>>
>>> +    int rc;
>>> +
>>> +    if (IS_ERR(cxlmd))
>>> +        return cxlmd;
>>> +
>>> +    rc = dev_set_name(&cxlmd->dev, "mem%d", cxlmd->id);
>>> +    if (rc) {
>>> +        put_device(&cxlmd->dev);
>>> +        return ERR_PTR(rc);
>>> +    }
>>> +
>>> +    return devm_cxl_memdev_add_or_reset(host, cxlmd);
>>> +}
>>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
> 

Re: [PATCH v19 01/22] cxl/mem: Arrange for always-synchronous memdev attach

[Alejandro Lucero Palau]

Hi Smita,


On 10/30/25 19:57, Koralahalli Channabasappa, Smita wrote:
> Hi Alejandro,
>
> I need patches 1–3 from this series as prerequisites for the 
> Soft-Reserved coordination work, so I wanted to check in on your plans 
> for the next revision.
>
> Link to the discussion: 
> https://lore.kernel.org/all/aPbOfFPIhtu5npaG@aschofie-mobl2.lan/
>
> Are patches 1–3 already being updated as part of your v20 work?
> If so, I can wait and pick them up from v20 directly.


Yes, I'm sending v20 later today. This patch has some changes for fixing 
the reported problems.


>
> If v20 is still in progress and may take some time, I can probably 
> carry patches 1–3 at the start of my series, and if that helps, I can 
> fold in the review comments by Jonathan while keeping authorship as 
> is. I would only adjust wording in the commit descriptions to reflect 
> the Soft-Reserved coordination context.
>
> Alternatively, if you prefer to continue carrying them in the Type-2 
> series, I can simply reference them as prerequisites instead.
>
> I’m fine with either approach just trying to avoid duplicated effort 
> and keep review in one place.


Let's see how v20 is received regarding the potential merge. If not 
impending, you could take those patches.


Thanks


>
> Thanks
> Smita
>
> On 10/29/2025 4:20 AM, Alejandro Lucero Palau wrote:
>>
>> On 10/7/25 13:40, Jonathan Cameron wrote:
>>> On Mon, 6 Oct 2025 11:01:09 +0100
>>> <alejandro.lucero-palau@amd.com> wrote:
>>>
>>>> From: Alejandro Lucero <alucerop@amd.com>
>>>>
>>>> In preparation for CXL accelerator drivers that have a hard 
>>>> dependency on
>>>> CXL capability initialization, arrange for the endpoint probe 
>>>> result to be
>>>> conveyed to the caller of devm_cxl_add_memdev().
>>>>
>>>> As it stands cxl_pci does not care about the attach state of the 
>>>> cxl_memdev
>>>> because all generic memory expansion functionality can be handled 
>>>> by the
>>>> cxl_core. For accelerators, that driver needs to know perform driver
>>>> specific initialization if CXL is available, or exectute a fallback 
>>>> to PCIe
>>>> only operation.
>>>>
>>>> By moving devm_cxl_add_memdev() to cxl_mem.ko it removes async module
>>>> loading as one reason that a memdev may not be attached upon return 
>>>> from
>>>> devm_cxl_add_memdev().
>>>>
>>>> The diff is busy as this moves cxl_memdev_alloc() down below the 
>>>> definition
>>>> of cxl_memdev_fops and introduces devm_cxl_memdev_add_or_reset() to
>>>> preclude needing to export more symbols from the cxl_core.
>>>>
>>>> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
>>> Alejandro, SoB chain broken here which makes this currently 
>>> unmergeable.
>>>
>>> Should definitely have your SoB as you sent the patch to the list 
>>> and need
>>> to make a statement that you believe it to be fine to do so (see the 
>>> Certificate
>>> of origin stuff in the docs).  Also, From should always be one of 
>>> the authors.
>>> If Dan wrote this as the SoB suggests then From should be set to him..
>>>
>>> git commit --amend --author="Dan Williams <dan.j.williams@intel.com>"
>>>
>>> Will fix that up.  Then either you add your SoB on basis you just 
>>> 'handled'
>>> the patch but didn't make substantial changes, or your SoB and a 
>>> Codeveloped-by
>>> if you did make major changes.  If it is minor stuff you can an
>>> a sign off with # what changed
>>> comment next to it.
>>
>>
>> Understood. I'll ask Dan what he prefers.
>>
>>
>>>
>>> A few minor comments inline.
>>>
>>> Thanks,
>>>
>>> Jonathan
>>>
>>>
>>>> ---
>>>>   drivers/cxl/Kconfig       |  2 +-
>>>>   drivers/cxl/core/memdev.c | 97 
>>>> ++++++++++++++++-----------------------
>>>>   drivers/cxl/mem.c         | 30 ++++++++++++
>>>>   drivers/cxl/private.h     | 11 +++++
>>>>   4 files changed, 82 insertions(+), 58 deletions(-)
>>>>   create mode 100644 drivers/cxl/private.h
>>>>
>>>> diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
>>>> index 028201e24523..111e05615f09 100644
>>>> --- a/drivers/cxl/Kconfig
>>>> +++ b/drivers/cxl/Kconfig
>>>> @@ -22,6 +22,7 @@ if CXL_BUS
>>>>   config CXL_PCI
>>>>       tristate "PCI manageability"
>>>>       default CXL_BUS
>>>> +    select CXL_MEM
>>>>       help
>>>>         The CXL specification defines a "CXL memory device" 
>>>> sub-class in the
>>>>         PCI "memory controller" base class of devices. Device's 
>>>> identified by
>>>> @@ -89,7 +90,6 @@ config CXL_PMEM
>>>>   config CXL_MEM
>>>>       tristate "CXL: Memory Expansion"
>>>> -    depends on CXL_PCI
>>>>       default CXL_BUS
>>>>       help
>>>>         The CXL.mem protocol allows a device to act as a provider 
>>>> of "System
>>>> diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
>>>> index c569e00a511f..2bef231008df 100644
>>>> --- a/drivers/cxl/core/memdev.c
>>>> +++ b/drivers/cxl/core/memdev.c
>>>> -
>>>> -err:
>>>> -    kfree(cxlmd);
>>>> -    return ERR_PTR(rc);
>>>>   }
>>>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_memdev_add_or_reset, "CXL");
>>>>   static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned 
>>>> int cmd,
>>>>                      unsigned long arg)
>>>> @@ -1023,50 +1012,44 @@ static const struct file_operations 
>>>> cxl_memdev_fops = {
>>>>       .llseek = noop_llseek,
>>>>   };
>>>> -struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>>>> -                       struct cxl_dev_state *cxlds)
>>>> +struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds)
>>>>   {
>>>>       struct cxl_memdev *cxlmd;
>>>>       struct device *dev;
>>>>       struct cdev *cdev;
>>>>       int rc;
>>>> -    cxlmd = cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
>>>> -    if (IS_ERR(cxlmd))
>>>> -        return cxlmd;
>>>> +    cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
>>> It's a little bit non obvious due to the device initialize mid way
>>> through this, but given there are no error paths after that you can
>>> currently just do.
>>>     struct cxl_memdev *cxlmd __free(kfree) =
>>>         cxl_memdev_alloc(cxlds, &cxl_memdev_fops);
>>> and
>>>     return_ptr(cxlmd);
>>>
>>> in the good path.  That lets you then just return rather than having
>>> the goto err: handling for the error case that currently frees this
>>> manually.
>>>
>>> Unlike the change below, this one I think is definitely worth making.
>>
>>
>> I agree so I'll do it. The below suggestion is also needed ...
>>
>>
>>>
>>>> +    if (!cxlmd)
>>>> +        return ERR_PTR(-ENOMEM);
>>>> -    dev = &cxlmd->dev;
>>>> -    rc = dev_set_name(dev, "mem%d", cxlmd->id);
>>>> -    if (rc)
>>>> +    rc = ida_alloc_max(&cxl_memdev_ida, CXL_MEM_MAX_DEVS - 1, 
>>>> GFP_KERNEL);
>>>> +    if (rc < 0)
>>>>           goto err;
>>>> -
>>>> -    /*
>>>> -     * Activate ioctl operations, no cxl_memdev_rwsem manipulation
>>>> -     * needed as this is ordered with cdev_add() publishing the 
>>>> device.
>>>> -     */
>>>> +    cxlmd->id = rc;
>>>> +    cxlmd->depth = -1;
>>>>       cxlmd->cxlds = cxlds;
>>>>       cxlds->cxlmd = cxlmd;
>>>> -    cdev = &cxlmd->cdev;
>>>> -    rc = cdev_device_add(cdev, dev);
>>>> -    if (rc)
>>>> -        goto err;
>>>> +    dev = &cxlmd->dev;
>>>> +    device_initialize(dev);
>>>> +    lockdep_set_class(&dev->mutex, &cxl_memdev_key);
>>>> +    dev->parent = cxlds->dev;
>>>> +    dev->bus = &cxl_bus_type;
>>>> +    dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
>>>> +    dev->type = &cxl_memdev_type;
>>>> +    device_set_pm_not_required(dev);
>>>> +    INIT_WORK(&cxlmd->detach_work, detach_memdev);
>>>> -    rc = devm_add_action_or_reset(host, cxl_memdev_unregister, 
>>>> cxlmd);
>>>> -    if (rc)
>>>> -        return ERR_PTR(rc);
>>>> +    cdev = &cxlmd->cdev;
>>>> +    cdev_init(cdev, &cxl_memdev_fops);
>>>>       return cxlmd;
>>>>   err:
>>>> -    /*
>>>> -     * The cdev was briefly live, shutdown any ioctl operations that
>>>> -     * saw that state.
>>>> -     */
>>>> -    cxl_memdev_shutdown(dev);
>>>> -    put_device(dev);
>>>> +    kfree(cxlmd);
>>>>       return ERR_PTR(rc);
>>>>   }
>>>> -EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
>>>> +EXPORT_SYMBOL_NS_GPL(cxl_memdev_alloc, "CXL");
>>>>   static void sanitize_teardown_notifier(void *data)
>>>>   {
>>>> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
>>>> index f7dc0ba8905d..144749b9c818 100644
>>>> --- a/drivers/cxl/mem.c
>>>> +++ b/drivers/cxl/mem.c
>>>> @@ -7,6 +7,7 @@
>>>>   #include "cxlmem.h"
>>>>   #include "cxlpci.h"
>>>> +#include "private.h"
>>>>   #include "core/core.h"
>>>>   /**
>>>> @@ -203,6 +204,34 @@ static int cxl_mem_probe(struct device *dev)
>>>>       return devm_add_action_or_reset(dev, enable_suspend, NULL);
>>>>   }
>>>> +/**
>>>> + * devm_cxl_add_memdev - Add a CXL memory device
>>>> + * @host: devres alloc/release context and parent for the memdev
>>>> + * @cxlds: CXL device state to associate with the memdev
>>>> + *
>>>> + * Upon return the device will have had a chance to attach to the
>>>> + * cxl_mem driver, but may fail if the CXL topology is not ready
>>>> + * (hardware CXL link down, or software platform CXL root not 
>>>> attached)
>>>> + */
>>>> +struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>>>> +                       struct cxl_dev_state *cxlds)
>>>> +{
>>>> +    struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds);
>>> Bit marginal but you could do a DEFINE_FREE() for cxlmd
>>> similar to the one that exists for put_cxl_port
>>>
>>> You would then need to steal the pointer for the devm_ call at the
>>> end of this function.
>>
>>
>> We are not freeing cxlmd in case of errors after we got the 
>> allocation, so I think it makes sense.
>>
>>
>> Thank you.
>>
>>
>>>
>>>> +    int rc;
>>>> +
>>>> +    if (IS_ERR(cxlmd))
>>>> +        return cxlmd;
>>>> +
>>>> +    rc = dev_set_name(&cxlmd->dev, "mem%d", cxlmd->id);
>>>> +    if (rc) {
>>>> +        put_device(&cxlmd->dev);
>>>> +        return ERR_PTR(rc);
>>>> +    }
>>>> +
>>>> +    return devm_cxl_memdev_add_or_reset(host, cxlmd);
>>>> +}
>>>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");
>>
>

Re: [PATCH v19 06/22] cxl: Move pci generic code

[Alejandro Lucero Palau]

On 10/7/25 14:01, Jonathan Cameron wrote:
> On Mon, 6 Oct 2025 11:01:14 +0100
> alejandro.lucero-palau@amd.com wrote:
>
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> Inside cxl/core/pci.c there are helpers for CXL PCIe initialization
>> meanwhile cxl/pci.c implements the functionality for a Type3 device
>> initialization.
>>
>> Move helper functions from cxl/pci.c to cxl/core/pci.c in order to be
>> exported and shared with CXL Type2 device initialization.
>>
>> Fix cxl mock tests affected by the code move, deleting a function which
>> indeed was not being used since commit 733b57f262b0("cxl/pci: Early
>> setup RCH dport component registers from RCRB").
>>
> Trivial but can we pull out that code removal as a separate patch?
> It's something Dave would probably pick up immediately.


The justification for the removal comes from the changes introduced in 
this patch, so I think it should be fine to keep it as it is now, but if 
Dave prefers, I will do so. Not going it for v20 though.


>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
>> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
>> Reviewed-by: Fan Ni <fan.ni@samsung.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> Reviewed-by: Alison Schofield <alison.schofield@intel.com>
>> Reviewed-by: Dan Williams <dan.j.williams@intel.com>
>
>> diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h
>> index ccf0ca36bc00..4b11757a46ab 100644
>> --- a/drivers/cxl/cxlpci.h
>> +++ b/drivers/cxl/cxlpci.h
>> @@ -74,9 +74,22 @@ static inline bool cxl_pci_flit_256(struct pci_dev *pdev)
>>   	return lnksta2 & PCI_EXP_LNKSTA2_FLIT;
>>   }
>>   
>> +/*
>> + * Assume that the caller has already validated that @pdev has CXL
>> + * capabilities, any RCIEp with CXL capabilities is treated as a
> In PCI spec they are RCiEP so we should match that rather than lowercase
> for the P.


I'll fix it.

Thanks!


>> + * Restricted CXL Device (RCD) and finds upstream port and endpoint
>> + * registers in a Root Complex Register Block (RCRB).
>> + */
>> +static inline bool is_cxl_restricted(struct pci_dev *pdev)
>> +{
>> +	return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END;
>> +}

Re: [PATCH v19 07/22] cxl: allow Type2 drivers to map cxl component regs

[Alejandro Lucero Palau]

On 10/7/25 14:18, Jonathan Cameron wrote:
> On Mon, 6 Oct 2025 11:01:15 +0100
> alejandro.lucero-palau@amd.com wrote:
>
>> From: Alejandro Lucero <alucerop@amd.com>
>>
> I'd amend the patch title to
> 	cxl/sfc: Map CXL component regs.
>
> And talk about exports in the description.
>
> Other options are fine but the patch title should indicate
> this is being used by the sfc driver.


Yes. I'll do so.

Thanks


>> Export cxl core functions for a Type2 driver being able to discover and
>> map the device component registers.
>>
>> Use it in sfc driver cxl initialization.
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Dan Williams <dan.j.williams@intel.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
>> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
>> ---
>>   drivers/cxl/core/pci.c             |  1 +
>>   drivers/cxl/core/port.c            |  1 +
>>   drivers/cxl/core/regs.c            |  1 +
>>   drivers/cxl/cxl.h                  |  7 ------
>>   drivers/cxl/cxlpci.h               | 12 ----------
>>   drivers/cxl/pci.c                  |  1 +
>>   drivers/net/ethernet/sfc/efx_cxl.c | 35 ++++++++++++++++++++++++++++++
>>   include/cxl/cxl.h                  | 19 ++++++++++++++++
>>   include/cxl/pci.h                  | 21 ++++++++++++++++++
>>   9 files changed, 79 insertions(+), 19 deletions(-)
>>   create mode 100644 include/cxl/pci.h

Re: [PATCH v19 08/22] cxl: Support dpa initialization without a mailbox

[Alejandro Lucero Palau]

On 10/7/25 14:22, Jonathan Cameron wrote:
> On Mon, 6 Oct 2025 11:01:16 +0100
> <alejandro.lucero-palau@amd.com> wrote:
>
>> From: Alejandro Lucero <alucerop@amd.com>
> I think this also needs to mention sfc in the patch title as we'll
> need an appropriate RB/Ack for that part.


Right. I'll do so.

Thanks


> Lazy option would be
> cxl/sfc: Support dpa initialization without a mailbox
>
> but there are probably better options.
>
>> Type3 relies on mailbox CXL_MBOX_OP_IDENTIFY command for initializing
>> memdev state params which end up being used for DPA initialization.
>>
>> Allow a Type2 driver to initialize DPA simply by giving the size of its
>> volatile hardware partition.
>>
>> Move related functions to memdev.
>>
>> Add sfc driver as the client.
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Dan Williams <dan.j.williams@intel.com>
>> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
>> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>

Re: [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration

[Alejandro Lucero Palau]

On 10/7/25 14:43, Jonathan Cameron wrote:
> On Mon, 6 Oct 2025 11:01:19 +0100
> alejandro.lucero-palau@amd.com wrote:
>
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> CXL region creation involves allocating capacity from Device Physical Address
>> (DPA) and assigning it to decode a given Host Physical Address (HPA). Before
>> determining how much DPA to allocate the amount of available HPA must be
>> determined. Also, not all HPA is created equal, some HPA targets RAM, some
>> targets PMEM, some is prepared for device-memory flows like HDM-D and HDM-DB,
>> and some is HDM-H (host-only).
>>
>> In order to support Type2 CXL devices, wrap all of those concerns into
>> an API that retrieves a root decoder (platform CXL window) that fits the
>> specified constraints and the capacity available for a new region.
>>
>> Add a complementary function for releasing the reference to such root
>> decoder.
>>
>> Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> One thing I noticed on a fresh read through...
>
>> ---
>>   drivers/cxl/core/region.c | 162 ++++++++++++++++++++++++++++++++++++++
>>   drivers/cxl/cxl.h         |   3 +
>>   include/cxl/cxl.h         |   6 ++
>>   3 files changed, 171 insertions(+)
>>
>> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
>> index e9bf42d91689..c5b66204ecde 100644
>> --- a/drivers/cxl/core/region.c
>> +++ b/drivers/cxl/core/region.c
>> @@ -703,6 +703,168 @@ static int free_hpa(struct cxl_region *cxlr)
>>   	return 0;
>>   }
>>   
>> +struct cxlrd_max_context {
>> +	struct device * const *host_bridges;
>> +	int interleave_ways;
>> +	unsigned long flags;
>> +	resource_size_t max_hpa;
>> +	struct cxl_root_decoder *cxlrd;
>> +};
>> +
>> +static int find_max_hpa(struct device *dev, void *data)
>> +{
>> +	struct cxlrd_max_context *ctx = data;
>> +	struct cxl_switch_decoder *cxlsd;
>> +	struct cxl_root_decoder *cxlrd;
>> +	struct resource *res, *prev;
>> +	struct cxl_decoder *cxld;
>> +	resource_size_t max;
>> +	int found = 0;
>> +
>> +	if (!is_root_decoder(dev))
>> +		return 0;
>> +
>> +	cxlrd = to_cxl_root_decoder(dev);
>> +	cxlsd = &cxlrd->cxlsd;
>> +	cxld = &cxlsd->cxld;
>> +
>> +	if ((cxld->flags & ctx->flags) != ctx->flags) {
>> +		dev_dbg(dev, "flags not matching: %08lx vs %08lx\n",
>> +			cxld->flags, ctx->flags);
>> +		return 0;
>> +	}
>> +
>> +	for (int i = 0; i < ctx->interleave_ways; i++) {
> I think ctx->interleave_ways == 0 as it's never set, so found is never
> set, but then the check below succeeds as found == 0 and ctx->interleave_ways == 0
>
> Definitely doesn't feel intentional!


Yes, that is the consequence of the other thing you spotted later.


>
>> +		for (int j = 0; j < ctx->interleave_ways; j++) {
>> +			if (ctx->host_bridges[i] == cxlsd->target[j]->dport_dev) {
>> +				found++;
>> +				break;
>> +			}
>> +		}
>> +	}
>> +
>> +	if (found != ctx->interleave_ways) {
>> +		dev_dbg(dev,
>> +			"Not enough host bridges. Found %d for %d interleave ways requested\n",
>> +			found, ctx->interleave_ways);
>> +		return 0;
>> +	}
>> +}
>> +
>> +/**
>> + * cxl_get_hpa_freespace - find a root decoder with free capacity per constraints
>> + * @cxlmd: the mem device requiring the HPA
>> + * @interleave_ways: number of entries in @host_bridges
>> + * @flags: CXL_DECODER_F flags for selecting RAM vs PMEM, and Type2 device
>> + * @max_avail_contig: output parameter of max contiguous bytes available in the
>> + *		      returned decoder
>> + *
>> + * Returns a pointer to a struct cxl_root_decoder
>> + *
>> + * The return tuple of a 'struct cxl_root_decoder' and 'bytes available given
>> + * in (@max_avail_contig))' is a point in time snapshot. If by the time the
>> + * caller goes to use this decoder and its capacity is reduced then caller needs
>> + * to loop and retry.
>> + *
>> + * The returned root decoder has an elevated reference count that needs to be
>> + * put with cxl_put_root_decoder(cxlrd).
>> + */
>> +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
>> +					       int interleave_ways,
> Currently unused.  I think you mean to set the field in ctx


Right. Refactoring went wrong.

I'll fix it.

Thanks!


>
>> +					       unsigned long flags,
>> +					       resource_size_t *max_avail_contig)
>> +{
>> +	struct cxl_port *endpoint = cxlmd->endpoint;
>> +	struct cxlrd_max_context ctx = {
>> +		.flags = flags,
>> +	};
>> +	struct cxl_port *root_port;
>> +
>> +	if (!endpoint) {
>> +		dev_dbg(&cxlmd->dev, "endpoint not linked to memdev\n");
>> +		return ERR_PTR(-ENXIO);
>> +	}
>> +
>> +	ctx.host_bridges = &endpoint->host_bridge;
>> +
>> +	struct cxl_root *root __free(put_cxl_root) = find_cxl_root(endpoint);
>> +	if (!root) {
>> +		dev_dbg(&endpoint->dev, "endpoint is not related to a root port\n");
>> +		return ERR_PTR(-ENXIO);
>> +	}
>> +
>> +	root_port = &root->port;
>> +	scoped_guard(rwsem_read, &cxl_rwsem.region)
>> +		device_for_each_child(&root_port->dev, &ctx, find_max_hpa);
>> +
>> +	if (!ctx.cxlrd)
>> +		return ERR_PTR(-ENOMEM);
>> +
>> +	*max_avail_contig = ctx.max_hpa;
>> +	return ctx.cxlrd;
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_get_hpa_freespace, "CXL");

Re: [PATCH v19 11/22] cxl: Define a driver interface for HPA free space enumeration

[Alejandro Lucero Palau]

On 10/15/25 19:17, Dave Jiang wrote:
>
> On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> CXL region creation involves allocating capacity from Device Physical Address
>> (DPA) and assigning it to decode a given Host Physical Address (HPA). Before
>> determining how much DPA to allocate the amount of available HPA must be
>> determined. Also, not all HPA is created equal, some HPA targets RAM, some
>> targets PMEM, some is prepared for device-memory flows like HDM-D and HDM-DB,
>> and some is HDM-H (host-only).
>>
>> In order to support Type2 CXL devices, wrap all of those concerns into
>> an API that retrieves a root decoder (platform CXL window) that fits the
>> specified constraints and the capacity available for a new region.
>>
>> Add a complementary function for releasing the reference to such root
>> decoder.
>>
>> Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
>>   drivers/cxl/core/region.c | 162 ++++++++++++++++++++++++++++++++++++++
>>   drivers/cxl/cxl.h         |   3 +
>>   include/cxl/cxl.h         |   6 ++
>>   3 files changed, 171 insertions(+)
>>
>> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
>> index e9bf42d91689..c5b66204ecde 100644
>> --- a/drivers/cxl/core/region.c
>> +++ b/drivers/cxl/core/region.c
>> @@ -703,6 +703,168 @@ static int free_hpa(struct cxl_region *cxlr)
>>   	return 0;
>>   }
>>   
>> +struct cxlrd_max_context {
>> +	struct device * const *host_bridges;
>> +	int interleave_ways;
>> +	unsigned long flags;
>> +	resource_size_t max_hpa;
>> +	struct cxl_root_decoder *cxlrd;
>> +};
>> +
>> +static int find_max_hpa(struct device *dev, void *data)
>> +{
>> +	struct cxlrd_max_context *ctx = data;
>> +	struct cxl_switch_decoder *cxlsd;
>> +	struct cxl_root_decoder *cxlrd;
>> +	struct resource *res, *prev;
>> +	struct cxl_decoder *cxld;
>> +	resource_size_t max;
>> +	int found = 0;
>> +
>> +	if (!is_root_decoder(dev))
>> +		return 0;
>> +
>> +	cxlrd = to_cxl_root_decoder(dev);
>> +	cxlsd = &cxlrd->cxlsd;
>> +	cxld = &cxlsd->cxld;
>> +
>> +	if ((cxld->flags & ctx->flags) != ctx->flags) {
>> +		dev_dbg(dev, "flags not matching: %08lx vs %08lx\n",
>> +			cxld->flags, ctx->flags);
>> +		return 0;
>> +	}
>> +
>> +	for (int i = 0; i < ctx->interleave_ways; i++) {
>> +		for (int j = 0; j < ctx->interleave_ways; j++) {
>> +			if (ctx->host_bridges[i] == cxlsd->target[j]->dport_dev) {
>> +				found++;
>> +				break;
>> +			}
>> +		}
>> +	}
>> +
>> +	if (found != ctx->interleave_ways) {
>> +		dev_dbg(dev,
>> +			"Not enough host bridges. Found %d for %d interleave ways requested\n",
>> +			found, ctx->interleave_ways);
>> +		return 0;
>> +	}
>> +
>> +	/*
>> +	 * Walk the root decoder resource range relying on cxl_rwsem.region to
>> +	 * preclude sibling arrival/departure and find the largest free space
>> +	 * gap.
>> +	 */
>> +	lockdep_assert_held_read(&cxl_rwsem.region);
>> +	res = cxlrd->res->child;
>> +
>> +	/* With no resource child the whole parent resource is available */
>> +	if (!res)
>> +		max = resource_size(cxlrd->res);
>> +	else
>> +		max = 0;
>> +
>> +	for (prev = NULL; res; prev = res, res = res->sibling) {
>> +		struct resource *next = res->sibling;
>> +		resource_size_t free = 0;
>> +
>> +		/*
>> +		 * Sanity check for preventing arithmetic problems below as a
>> +		 * resource with size 0 could imply using the end field below
>> +		 * when set to unsigned zero - 1 or all f in hex.
>> +		 */
>> +		if (prev && !resource_size(prev))
>> +			continue;
>> +
>> +		if (!prev && res->start > cxlrd->res->start) {
>> +			free = res->start - cxlrd->res->start;
>> +			max = max(free, max);
>> +		}
>> +		if (prev && res->start > prev->end + 1) {
>> +			free = res->start - prev->end + 1;
>> +			max = max(free, max);
>> +		}
>> +		if (next && res->end + 1 < next->start) {
>> +			free = next->start - res->end + 1;
>> +			max = max(free, max);
>> +		}
>> +		if (!next && res->end + 1 < cxlrd->res->end + 1) {
>> +			free = cxlrd->res->end + 1 - res->end + 1;
>> +			max = max(free, max);
>> +		}
>> +	}
>> +
>> +	dev_dbg(cxlrd_dev(cxlrd), "found %pa bytes of free space\n", &max);
>> +	if (max > ctx->max_hpa) {
>> +		if (ctx->cxlrd)
>> +			put_device(cxlrd_dev(ctx->cxlrd));
>> +		get_device(cxlrd_dev(cxlrd));
>> +		ctx->cxlrd = cxlrd;
>> +		ctx->max_hpa = max;
>> +	}
>> +	return 0;
>> +}
>> +
>> +/**
>> + * cxl_get_hpa_freespace - find a root decoder with free capacity per constraints
>> + * @cxlmd: the mem device requiring the HPA
>> + * @interleave_ways: number of entries in @host_bridges
>> + * @flags: CXL_DECODER_F flags for selecting RAM vs PMEM, and Type2 device
>> + * @max_avail_contig: output parameter of max contiguous bytes available in the
>> + *		      returned decoder
>> + *
>> + * Returns a pointer to a struct cxl_root_decoder
>> + *
>> + * The return tuple of a 'struct cxl_root_decoder' and 'bytes available given
>> + * in (@max_avail_contig))' is a point in time snapshot. If by the time the
>> + * caller goes to use this decoder and its capacity is reduced then caller needs
>> + * to loop and retry.
>> + *
>> + * The returned root decoder has an elevated reference count that needs to be
>> + * put with cxl_put_root_decoder(cxlrd).
>> + */
>> +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
>> +					       int interleave_ways,
>> +					       unsigned long flags,
>> +					       resource_size_t *max_avail_contig)
>> +{
>> +	struct cxl_port *endpoint = cxlmd->endpoint;
> Do the assignment right before the check. Would help prevent future issues of use before check.
>

I'll do, but the pointer declaration will be placed the last one with 
the new line length, just before the assignment. I bet some reviewers in 
v20 will tell to do the assignment along with the declaration ...


>> +	struct cxlrd_max_context ctx = {
>> +		.flags = flags,
>> +	};
>> +	struct cxl_port *root_port;
>> +
>> +	if (!endpoint) {
>> +		dev_dbg(&cxlmd->dev, "endpoint not linked to memdev\n");
>> +		return ERR_PTR(-ENXIO);
>> +	}
>> +
>> +	ctx.host_bridges = &endpoint->host_bridge;
> Would there ever be a scenario where there would be multiple host bridges that requires this to be an array? I'm not seeing that usage in this patch series.


It has been suggested by Dan the cxl core modified for Type2 needs to 
cover other impending requirements, and this one comes from Dan's 
original patchset.

I could avoid the array as it is not being used for sfc, but let's ask 
Dan specifically.



>
> DJ
>
>> +
>> +	struct cxl_root *root __free(put_cxl_root) = find_cxl_root(endpoint);
>> +	if (!root) {
>> +		dev_dbg(&endpoint->dev, "endpoint is not related to a root port\n");
>> +		return ERR_PTR(-ENXIO);
>> +	}
>> +
>> +	root_port = &root->port;
>> +	scoped_guard(rwsem_read, &cxl_rwsem.region)
>> +		device_for_each_child(&root_port->dev, &ctx, find_max_hpa);
>> +
>> +	if (!ctx.cxlrd)
>> +		return ERR_PTR(-ENOMEM);
>> +
>> +	*max_avail_contig = ctx.max_hpa;
>> +	return ctx.cxlrd;
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_get_hpa_freespace, "CXL");
>> +
>> +void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd)
>> +{
>> +	put_device(cxlrd_dev(cxlrd));
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_put_root_decoder, "CXL");
>> +
>>   static ssize_t size_store(struct device *dev, struct device_attribute *attr,
>>   			  const char *buf, size_t len)
>>   {
>> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
>> index 793d4dfe51a2..076640e91ee0 100644
>> --- a/drivers/cxl/cxl.h
>> +++ b/drivers/cxl/cxl.h
>> @@ -664,6 +664,9 @@ struct cxl_root_decoder *to_cxl_root_decoder(struct device *dev);
>>   struct cxl_switch_decoder *to_cxl_switch_decoder(struct device *dev);
>>   struct cxl_endpoint_decoder *to_cxl_endpoint_decoder(struct device *dev);
>>   bool is_root_decoder(struct device *dev);
>> +
>> +#define cxlrd_dev(cxlrd) (&(cxlrd)->cxlsd.cxld.dev)
>> +
>>   bool is_switch_decoder(struct device *dev);
>>   bool is_endpoint_decoder(struct device *dev);
>>   struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port,
>> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
>> index 043fc31c764e..2ec514c77021 100644
>> --- a/include/cxl/cxl.h
>> +++ b/include/cxl/cxl.h
>> @@ -250,4 +250,10 @@ int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity);
>>   struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
>>   				       struct cxl_dev_state *cxlds,
>>   				       const struct cxl_memdev_ops *ops);
>> +struct cxl_port;
>> +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
>> +					       int interleave_ways,
>> +					       unsigned long flags,
>> +					       resource_size_t *max);
>> +void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd);
>>   #endif /* __CXL_CXL_H__ */
>

Re: [PATCH v19 13/22] cxl: Define a driver interface for DPA allocation

[Alejandro Lucero Palau]

On 10/7/25 14:52, Jonathan Cameron wrote:
> On Mon, 6 Oct 2025 11:01:21 +0100
> <alejandro.lucero-palau@amd.com> wrote:
>
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> Region creation involves finding available DPA (device-physical-address)
>> capacity to map into HPA (host-physical-address) space.
>>
>> In order to support CXL Type2 devices, define an API, cxl_request_dpa(),
>> that tries to allocate the DPA memory the driver requires to operate.The
>> memory requested should not be bigger than the max available HPA obtained
>> previously with cxl_get_hpa_freespace().
>>
>> Based on https://lore.kernel.org/linux-cxl/168592158743.1948938.7622563891193802610.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> A few minor things inline.  Depending on how much this changed
> from the 'Based on' it might be appropriate to keep a SoB / author
> set to Dan, but I'll let him request that if he feels appropriate
> (or you can make that decision if Dan is busy).
>
> A few things inline.  All trivial
>
> Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
>
>>   int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>>   		     enum cxl_partition_mode mode)
>> @@ -613,6 +622,82 @@ int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>>   	return 0;
>>   }
>>   
>> +static int find_free_decoder(struct device *dev, const void *data)
>> +{
>> +	struct cxl_endpoint_decoder *cxled;
>> +	struct cxl_port *port;
>> +
>> +	if (!is_endpoint_decoder(dev))
>> +		return 0;
>> +
>> +	cxled = to_cxl_endpoint_decoder(dev);
>> +	port = cxled_to_port(cxled);
>> +
>> +	return cxled->cxld.id == (port->hdm_end + 1);
>> +}
>> +
>> +static struct cxl_endpoint_decoder *
>> +cxl_find_free_decoder(struct cxl_memdev *cxlmd)
>> +{
>> +	struct cxl_port *endpoint = cxlmd->endpoint;
>> +	struct device *dev;
>> +
>> +	guard(rwsem_read)(&cxl_rwsem.dpa);
>> +	dev = device_find_child(&endpoint->dev, NULL,
>> +				find_free_decoder);
>> +	if (dev)
>> +		return to_cxl_endpoint_decoder(dev);
>> +
>> +	return NULL;
> Trivial but I'd prefer to see the 'error' like thing out of line
>
> 	if (!dev)
> 		return NULL;
>
> 	return to_cxl_endpoint_decoder(dev);


It makes sense. I'll do so.

Thanks!


>> +}
>> +
>> +/**
>> + * cxl_request_dpa - search and reserve DPA given input constraints
>> + * @cxlmd: memdev with an endpoint port with available decoders
>> + * @mode: CXL partition mode (ram vs pmem)
>> + * @alloc: dpa size required
>> + *
>> + * Returns a pointer to a 'struct cxl_endpoint_decoder' on success or
>> + * an errno encoded pointer on failure.
>> + *
>> + * Given that a region needs to allocate from limited HPA capacity it
>> + * may be the case that a device has more mappable DPA capacity than
>> + * available HPA. The expectation is that @alloc is a driver known
>> + * value based on the device capacity but which could not be fully
>> + * available due to HPA constraints.
>> + *
>> + * Returns a pinned cxl_decoder with at least @alloc bytes of capacity
>> + * reserved, or an error pointer. The caller is also expected to own the
>> + * lifetime of the memdev registration associated with the endpoint to
>> + * pin the decoder registered as well.
>> + */
>> +struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
>> +					     enum cxl_partition_mode mode,
>> +					     resource_size_t alloc)
>> +{
>> +	int rc;
>> +
>> +	if (!IS_ALIGNED(alloc, SZ_256M))
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	struct cxl_endpoint_decoder *cxled __free(put_cxled) =
>> +		cxl_find_free_decoder(cxlmd);
>> +
>> +	if (!cxled)
>> +		return ERR_PTR(-ENODEV);
>> +
>> +	rc = cxl_dpa_set_part(cxled, mode);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	rc = cxl_dpa_alloc(cxled, alloc);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	return no_free_ptr(cxled);
> return_ptr() (it's exactly the same implementation).
>
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_request_dpa, "CXL");

Re: [PATCH v19 13/22] cxl: Define a driver interface for DPA allocation

[Alejandro Lucero Palau]

On 10/15/25 21:08, Dave Jiang wrote:
>
> On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> Region creation involves finding available DPA (device-physical-address)
>> capacity to map into HPA (host-physical-address) space.
>>
>> In order to support CXL Type2 devices, define an API, cxl_request_dpa(),
>> that tries to allocate the DPA memory the driver requires to operate.The
>> memory requested should not be bigger than the max available HPA obtained
>> previously with cxl_get_hpa_freespace().
>>
>> Based on https://lore.kernel.org/linux-cxl/168592158743.1948938.7622563891193802610.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> just a nit below
>
> Reviewed-by: Dave Jiang <dave.jiang@intel.com>> ---
>>   drivers/cxl/core/hdm.c | 85 ++++++++++++++++++++++++++++++++++++++++++
>>   drivers/cxl/cxl.h      |  1 +
>>   include/cxl/cxl.h      |  5 +++
>>   3 files changed, 91 insertions(+)
>>
>> diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
>> index e9e1d555cec6..70a15694c35d 100644
>> --- a/drivers/cxl/core/hdm.c
>> +++ b/drivers/cxl/core/hdm.c
>> @@ -3,6 +3,7 @@
>>   #include <linux/seq_file.h>
>>   #include <linux/device.h>
>>   #include <linux/delay.h>
>> +#include <cxl/cxl.h>
>>   
>>   #include "cxlmem.h"
>>   #include "core.h"
>> @@ -556,6 +557,13 @@ bool cxl_resource_contains_addr(const struct resource *res, const resource_size_
>>   	return resource_contains(res, &_addr);
>>   }
>>   
>> +/**
>> + * cxl_dpa_free - release DPA (Device Physical Address)
>> + *
> no blank line needed here
>
> DJ> + * @cxled: endpoint decoder linked to the DPA


I'll fix it.

Thanks!


>> + *
>> + * Returns 0 or error.
>> + */
>>   int cxl_dpa_free(struct cxl_endpoint_decoder *cxled)
>>   {
>>   	struct cxl_port *port = cxled_to_port(cxled);
>> @@ -582,6 +590,7 @@ int cxl_dpa_free(struct cxl_endpoint_decoder *cxled)
>>   	devm_cxl_dpa_release(cxled);
>>   	return 0;
>>   }
>> +EXPORT_SYMBOL_NS_GPL(cxl_dpa_free, "CXL");
>>   
>>   int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>>   		     enum cxl_partition_mode mode)
>> @@ -613,6 +622,82 @@ int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled,
>>   	return 0;
>>   }
>>   
>> +static int find_free_decoder(struct device *dev, const void *data)
>> +{
>> +	struct cxl_endpoint_decoder *cxled;
>> +	struct cxl_port *port;
>> +
>> +	if (!is_endpoint_decoder(dev))
>> +		return 0;
>> +
>> +	cxled = to_cxl_endpoint_decoder(dev);
>> +	port = cxled_to_port(cxled);
>> +
>> +	return cxled->cxld.id == (port->hdm_end + 1);
>> +}
>> +
>> +static struct cxl_endpoint_decoder *
>> +cxl_find_free_decoder(struct cxl_memdev *cxlmd)
>> +{
>> +	struct cxl_port *endpoint = cxlmd->endpoint;
>> +	struct device *dev;
>> +
>> +	guard(rwsem_read)(&cxl_rwsem.dpa);
>> +	dev = device_find_child(&endpoint->dev, NULL,
>> +				find_free_decoder);
>> +	if (dev)
>> +		return to_cxl_endpoint_decoder(dev);
>> +
>> +	return NULL;
>> +}
>> +
>> +/**
>> + * cxl_request_dpa - search and reserve DPA given input constraints
>> + * @cxlmd: memdev with an endpoint port with available decoders
>> + * @mode: CXL partition mode (ram vs pmem)
>> + * @alloc: dpa size required
>> + *
>> + * Returns a pointer to a 'struct cxl_endpoint_decoder' on success or
>> + * an errno encoded pointer on failure.
>> + *
>> + * Given that a region needs to allocate from limited HPA capacity it
>> + * may be the case that a device has more mappable DPA capacity than
>> + * available HPA. The expectation is that @alloc is a driver known
>> + * value based on the device capacity but which could not be fully
>> + * available due to HPA constraints.
>> + *
>> + * Returns a pinned cxl_decoder with at least @alloc bytes of capacity
>> + * reserved, or an error pointer. The caller is also expected to own the
>> + * lifetime of the memdev registration associated with the endpoint to
>> + * pin the decoder registered as well.
>> + */
>> +struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
>> +					     enum cxl_partition_mode mode,
>> +					     resource_size_t alloc)
>> +{
>> +	int rc;
>> +
>> +	if (!IS_ALIGNED(alloc, SZ_256M))
>> +		return ERR_PTR(-EINVAL);
>> +
>> +	struct cxl_endpoint_decoder *cxled __free(put_cxled) =
>> +		cxl_find_free_decoder(cxlmd);
>> +
>> +	if (!cxled)
>> +		return ERR_PTR(-ENODEV);
>> +
>> +	rc = cxl_dpa_set_part(cxled, mode);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	rc = cxl_dpa_alloc(cxled, alloc);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	return no_free_ptr(cxled);
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_request_dpa, "CXL");
>> +
>>   static int __cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u64 size)
>>   {
>>   	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
>> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
>> index ab490b5a9457..6ca0827cfaa5 100644
>> --- a/drivers/cxl/cxl.h
>> +++ b/drivers/cxl/cxl.h
>> @@ -625,6 +625,7 @@ struct cxl_root *find_cxl_root(struct cxl_port *port);
>>   
>>   DEFINE_FREE(put_cxl_root, struct cxl_root *, if (_T) put_device(&_T->port.dev))
>>   DEFINE_FREE(put_cxl_port, struct cxl_port *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
>> +DEFINE_FREE(put_cxled, struct cxl_endpoint_decoder *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->cxld.dev))
>>   DEFINE_FREE(put_cxl_root_decoder, struct cxl_root_decoder *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->cxlsd.cxld.dev))
>>   DEFINE_FREE(put_cxl_region, struct cxl_region *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
>>   
>> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
>> index 2966b95e80a6..1cbe53ad0416 100644
>> --- a/include/cxl/cxl.h
>> +++ b/include/cxl/cxl.h
>> @@ -7,6 +7,7 @@
>>   
>>   #include <linux/node.h>
>>   #include <linux/ioport.h>
>> +#include <linux/range.h>
>>   #include <cxl/mailbox.h>
>>   
>>   /**
>> @@ -270,4 +271,8 @@ struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
>>   					       unsigned long flags,
>>   					       resource_size_t *max);
>>   void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd);
>> +struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd,
>> +					     enum cxl_partition_mode mode,
>> +					     resource_size_t alloc);
>> +int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
>>   #endif /* __CXL_CXL_H__ */
>

Re: [PATCH v19 14/22] sfc: get endpoint decoder

[Alejandro Lucero Palau]

On 10/15/25 21:15, Dave Jiang wrote:
>
> On 10/6/25 3:01 AM, alejandro.lucero-palau@amd.com wrote:
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> Use cxl api for getting DPA (Device Physical Address) to use through an
>> endpoint decoder.
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Martin Habets <habetsm.xilinx@gmail.com>
>> Acked-by: Edward Cree <ecree.xilinx@gmail.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
>> ---
>>   drivers/net/ethernet/sfc/efx_cxl.c | 12 +++++++++++-
>>   1 file changed, 11 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
>> index d7c34c978434..1a50bb2c0913 100644
>> --- a/drivers/net/ethernet/sfc/efx_cxl.c
>> +++ b/drivers/net/ethernet/sfc/efx_cxl.c
>> @@ -108,6 +108,14 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>>   		return -ENOSPC;
>>   	}
>>   
>> +	cxl->cxled = cxl_request_dpa(cxl->cxlmd, CXL_PARTMODE_RAM,
>> +				     EFX_CTPIO_BUFFER_SIZE);
>> +	if (IS_ERR(cxl->cxled)) {
>> +		pci_err(pci_dev, "CXL accel request DPA failed");
>> +		cxl_put_root_decoder(cxl->cxlrd);
> Might be good to create a __free() macro for freeing the root decoder instead of having to do this for every error path in this function.


We are in netdev territory here and this is not recommended, so I have 
tried to avoid other uses of this kernel releasing functionality here 
since the RFC.

Thanks


> DJ
>
>> +		return PTR_ERR(cxl->cxled);
>> +	}
>> +
>>   	probe_data->cxl = cxl;
>>   
>>   	return 0;
>> @@ -115,8 +123,10 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>>   
>>   void efx_cxl_exit(struct efx_probe_data *probe_data)
>>   {
>> -	if (probe_data->cxl)
>> +	if (probe_data->cxl) {
>> +		cxl_dpa_free(probe_data->cxl->cxled);
>>   		cxl_put_root_decoder(probe_data->cxl->cxlrd);
>> +	}
>>   }
>>   
>>   MODULE_IMPORT_NS("CXL");

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Alejandro Lucero Palau]

On 10/7/25 15:11, Jonathan Cameron wrote:
> On Mon, 6 Oct 2025 11:01:26 +0100
> <alejandro.lucero-palau@amd.com> wrote:
>
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> Creating a CXL region requires userspace intervention through the cxl
>> sysfs files. Type2 support should allow accelerator drivers to create
>> such cxl region from kernel code.
>>
>> Adding that functionality and integrating it with current support for
>> memory expanders.
>>
>> Support an action by the type2 driver to be linked to the created region
>> for unwinding the resources allocated properly.
>>
>> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> One new question below (and a trivial thing beyond that).
>
> If you adopt one of the suggested ways of tidying that up, then keep the RB
> if not I'll want to take another look so drop it.
>

I will drop your RB since it is not clear to me if your suggestion below 
makes sense.


>>   #ifdef CONFIG_CXL_REGION
>>   extern struct device_attribute dev_attr_create_pmem_region;
>>   extern struct device_attribute dev_attr_create_ram_region;
>> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
>> index 26dfc15e57cd..e3b6d85cd43e 100644
>> --- a/drivers/cxl/core/region.c
>> +++ b/drivers/cxl/core/region.c
>> +
>> +/* Establish an empty region covering the given HPA range */
>> +static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
>> +					   struct cxl_endpoint_decoder *cxled)
>> +{
>> +	struct cxl_port *port = cxlrd_to_port(cxlrd);
>> +	struct cxl_region *cxlr;
>> +	int rc;
>> +
>> +	cxlr = construct_region_begin(cxlrd, cxled);
>>   
>>   	rc = __construct_region(cxlr, cxlrd, cxled);
>>   	if (rc) {
>> @@ -3621,6 +3639,106 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
>>   	return cxlr;
>>   }
>>   
>> +DEFINE_FREE(cxl_region_drop, struct cxl_region *, if (_T) drop_region(_T))
>> +
>> +static struct cxl_region *
>> +__construct_new_region(struct cxl_root_decoder *cxlrd,
>> +		       struct cxl_endpoint_decoder **cxled, int ways)
> Why pass in an array of struct cxl_endpoint_decoder * if this is only
> ever going to use the first element?
>
> I think we need to indicate that somehow.  Could just pass in the
> relevant decoder (I assume there is only one?)  Or pass the array
> and an index (here 0).


Just the first one is use for creating the region, what means the 
struct/object which will be initialised with the attaching phase later on.

The region will be created with the target type and mode of the first 
decoder used, but the attaching implies to check the other decoders 
align with this. And all the decoders are used for that and for 
calculating the hpa size to request.

So I do not think there is a problem here, at least regarding your concern.


>
>> +{
>> +	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
>> +	struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
>> +	struct cxl_region_params *p;
>> +	resource_size_t size = 0;
>> +	int rc, i;
>> +
>> +	struct cxl_region *cxlr __free(cxl_region_drop) =
>> +		construct_region_begin(cxlrd, cxled[0]);
>> +	if (IS_ERR(cxlr))
>> +		return cxlr;
>> +
>> +	guard(rwsem_write)(&cxl_rwsem.region);
>> +
>> +	/*
>> +	 * Sanity check. This should not happen with an accel driver handling
>> +	 * the region creation.
>> +	 */
>> +	p = &cxlr->params;
>> +	if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
>> +		dev_err(cxlmd->dev.parent,
>> +			"%s:%s: %s  unexpected region state\n",
>> +			dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
>> +			__func__);
>> +		return ERR_PTR(-EBUSY);
>> +	}
>> +
>> +	rc = set_interleave_ways(cxlr, ways);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
>> +		for (i = 0; i < ways; i++) {
>> +			if (!cxled[i]->dpa_res)
>> +				break;
>> +			size += resource_size(cxled[i]->dpa_res);
>> +		}
>> +		if (i < ways)
>> +			return ERR_PTR(-EINVAL);
>> +
>> +		rc = alloc_hpa(cxlr, size);
>> +		if (rc)
>> +			return ERR_PTR(rc);
>> +
>> +		for (i = 0; i < ways; i++) {
>> +			rc = cxl_region_attach(cxlr, cxled[i], 0);
>> +			if (rc)
>> +				return ERR_PTR(rc);
>> +		}
>> +	}
>> +
>> +	rc = cxl_region_decode_commit(cxlr);
>> +	if (rc)
>> +		return ERR_PTR(rc);
>> +
>> +	p->state = CXL_CONFIG_COMMIT;
>> +
>> +	return no_free_ptr(cxlr);
> return_ptr()
>> +}

Re: [PATCH v19 22/22] sfc: support pio mapping based on cxl

[Alejandro Lucero Palau]

On 10/7/25 15:48, Jonathan Cameron wrote:
> On Mon, 6 Oct 2025 11:01:30 +0100
> <alejandro.lucero-palau@amd.com> wrote:
>
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> A PIO buffer is a region of device memory to which the driver can write a
>> packet for TX, with the device handling the transmit doorbell without
>> requiring a DMA for getting the packet data, which helps reducing latency
>> in certain exchanges. With CXL mem protocol this latency can be lowered
>> further.
>>
>> With a device supporting CXL and successfully initialised, use the cxl
>> region to map the memory range and use this mapping for PIO buffers.
>>
>> Add the disabling of those CXL-based PIO buffers if the callback for
>> potential cxl endpoint removal by the CXL code happens.
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> A few minor things inline.  The ifdef complexity in here is moderately
> nasty though.  Might be worth seeing if any of that can be moved
> to stubs or IS_ENABLED() checks.
>

I'm afraid that would imply main refactoring in main efx/sfc code and I 
prefer to avoid so. The sfc maintainer has the last word though.


>> diff --git a/drivers/net/ethernet/sfc/efx.h b/drivers/net/ethernet/sfc/efx.h
>> index 45e191686625..057d30090894 100644
>> --- a/drivers/net/ethernet/sfc/efx.h
>> +++ b/drivers/net/ethernet/sfc/efx.h
>> @@ -236,5 +236,4 @@ static inline bool efx_rwsem_assert_write_locked(struct rw_semaphore *sem)
>>   
>>   int efx_xdp_tx_buffers(struct efx_nic *efx, int n, struct xdp_frame **xdpfs,
>>   		       bool flush);
>> -
> stray change that you should clean out.


Oh, yes. I'll remove it.


>>   #endif /* EFX_EFX_H */
>> diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c
>> index 79fe99d83f9f..a84ce45398c1 100644
>> --- a/drivers/net/ethernet/sfc/efx_cxl.c
>> +++ b/drivers/net/ethernet/sfc/efx_cxl.c
>> @@ -11,6 +11,7 @@
>>   #include <cxl/pci.h>
>>   #include "net_driver.h"
>>   #include "efx_cxl.h"
>> +#include "efx.h"
>>   
>>   #define EFX_CTPIO_BUFFER_SIZE	SZ_256M
>>   
>> @@ -20,6 +21,7 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>>   	struct pci_dev *pci_dev = efx->pci_dev;
>>   	resource_size_t max_size;
>>   	struct efx_cxl *cxl;
>> +	struct range range;
>>   	u16 dvsec;
>>   	int rc;
>>   
>> @@ -119,19 +121,40 @@ int efx_cxl_init(struct efx_probe_data *probe_data)
>>   	cxl->efx_region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1);
>>   	if (IS_ERR(cxl->efx_region)) {
>>   		pci_err(pci_dev, "CXL accel create region failed");
>> -		cxl_put_root_decoder(cxl->cxlrd);
>> -		cxl_dpa_free(cxl->cxled);
>> -		return PTR_ERR(cxl->efx_region);
>> +		rc = PTR_ERR(cxl->efx_region);
>> +		goto err_dpa;
> It's a somewhat trivial thing but you could reduce churn by
> moving the err_dpa block introduction back to where this lot
> was first added.
>

The error path now has the minimum lines possible using the gotos when 
that saves at least one code line.

Thanks!

Re: [PATCH v19 06/22] cxl: Move pci generic code

[Jonathan Cameron]

On Mon, 10 Nov 2025 11:23:53 +0000
Alejandro Lucero Palau <alucerop@amd.com> wrote:

> On 10/7/25 14:01, Jonathan Cameron wrote:
> > On Mon, 6 Oct 2025 11:01:14 +0100
> > alejandro.lucero-palau@amd.com wrote:
> >  
> >> From: Alejandro Lucero <alucerop@amd.com>
> >>
> >> Inside cxl/core/pci.c there are helpers for CXL PCIe initialization
> >> meanwhile cxl/pci.c implements the functionality for a Type3 device
> >> initialization.
> >>
> >> Move helper functions from cxl/pci.c to cxl/core/pci.c in order to be
> >> exported and shared with CXL Type2 device initialization.
> >>
> >> Fix cxl mock tests affected by the code move, deleting a function which
> >> indeed was not being used since commit 733b57f262b0("cxl/pci: Early
> >> setup RCH dport component registers from RCRB").
> >>  
> > Trivial but can we pull out that code removal as a separate patch?
> > It's something Dave would probably pick up immediately.  
> 
> 
> The justification for the removal comes from the changes introduced in 
> this patch, so I think it should be fine to keep it as it is now, but if 
> Dave prefers, I will do so. Not going it for v20 though.
Ok. My confusion is this commit message says the function was not being
used since commit ...
Which isn't this patch.  Either that is true and the removal is unrelated
or that description needs a tweak.

Jonathan

Re: [PATCH v19 18/22] cxl: Allow region creation by type2 drivers

[Jonathan Cameron]

On Mon, 10 Nov 2025 13:47:40 +0000
Alejandro Lucero Palau <alucerop@amd.com> wrote:

> On 10/7/25 15:11, Jonathan Cameron wrote:
> > On Mon, 6 Oct 2025 11:01:26 +0100
> > <alejandro.lucero-palau@amd.com> wrote:
> >  
> >> From: Alejandro Lucero <alucerop@amd.com>
> >>
> >> Creating a CXL region requires userspace intervention through the cxl
> >> sysfs files. Type2 support should allow accelerator drivers to create
> >> such cxl region from kernel code.
> >>
> >> Adding that functionality and integrating it with current support for
> >> memory expanders.
> >>
> >> Support an action by the type2 driver to be linked to the created region
> >> for unwinding the resources allocated properly.
> >>
> >> Based on https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/
> >>
> >> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> >> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>  
> > One new question below (and a trivial thing beyond that).
> >
> > If you adopt one of the suggested ways of tidying that up, then keep the RB
> > if not I'll want to take another look so drop it.
> >  
> 
> I will drop your RB since it is not clear to me if your suggestion below 
> makes sense.
> 
> 
> >>   #ifdef CONFIG_CXL_REGION
> >>   extern struct device_attribute dev_attr_create_pmem_region;
> >>   extern struct device_attribute dev_attr_create_ram_region;
> >> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> >> index 26dfc15e57cd..e3b6d85cd43e 100644
> >> --- a/drivers/cxl/core/region.c
> >> +++ b/drivers/cxl/core/region.c
> >> +
> >> +/* Establish an empty region covering the given HPA range */
> >> +static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
> >> +					   struct cxl_endpoint_decoder *cxled)
> >> +{
> >> +	struct cxl_port *port = cxlrd_to_port(cxlrd);
> >> +	struct cxl_region *cxlr;
> >> +	int rc;
> >> +
> >> +	cxlr = construct_region_begin(cxlrd, cxled);
> >>   
> >>   	rc = __construct_region(cxlr, cxlrd, cxled);
> >>   	if (rc) {
> >> @@ -3621,6 +3639,106 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
> >>   	return cxlr;
> >>   }
> >>   
> >> +DEFINE_FREE(cxl_region_drop, struct cxl_region *, if (_T) drop_region(_T))
> >> +
> >> +static struct cxl_region *
> >> +__construct_new_region(struct cxl_root_decoder *cxlrd,
> >> +		       struct cxl_endpoint_decoder **cxled, int ways)  
> > Why pass in an array of struct cxl_endpoint_decoder * if this is only
> > ever going to use the first element?
> >
> > I think we need to indicate that somehow.  Could just pass in the
> > relevant decoder (I assume there is only one?)  Or pass the array
> > and an index (here 0).  
> 
> 
> Just the first one is use for creating the region, what means the 
> struct/object which will be initialised with the attaching phase later on.

OK. I think I now see what this is doing.

> 
> The region will be created with the target type and mode of the first 
> decoder used, but the attaching implies to check the other decoders 
> align with this. And all the decoders are used for that and for 
> calculating the hpa size to request.
I'd missed the indexing over decoders later in the function.

One minor thing inline noticed whilst walking through your explanation.
> 
> So I do not think there is a problem here, at least regarding your concern.
> 
> 
> >  
> >> +{
> >> +	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]);
> >> +	struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
> >> +	struct cxl_region_params *p;
> >> +	resource_size_t size = 0;
> >> +	int rc, i;
> >> +
> >> +	struct cxl_region *cxlr __free(cxl_region_drop) =
> >> +		construct_region_begin(cxlrd, cxled[0]);
> >> +	if (IS_ERR(cxlr))
> >> +		return cxlr;
> >> +
> >> +	guard(rwsem_write)(&cxl_rwsem.region);
> >> +
> >> +	/*
> >> +	 * Sanity check. This should not happen with an accel driver handling
> >> +	 * the region creation.
> >> +	 */
> >> +	p = &cxlr->params;
> >> +	if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) {
> >> +		dev_err(cxlmd->dev.parent,
> >> +			"%s:%s: %s  unexpected region state\n",
> >> +			dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev),
> >> +			__func__);
> >> +		return ERR_PTR(-EBUSY);
> >> +	}
> >> +
> >> +	rc = set_interleave_ways(cxlr, ways);
> >> +	if (rc)
> >> +		return ERR_PTR(rc);
> >> +
> >> +	rc = set_interleave_granularity(cxlr, cxld->interleave_granularity);
> >> +	if (rc)
> >> +		return ERR_PTR(rc);
> >> +
> >> +	scoped_guard(rwsem_read, &cxl_rwsem.dpa) {
> >> +		for (i = 0; i < ways; i++) {
> >> +			if (!cxled[i]->dpa_res)
> >> +				break;
> >> +			size += resource_size(cxled[i]->dpa_res);
> >> +		}
> >> +		if (i < ways)
> >> +			return ERR_PTR(-EINVAL);
I'll try and remember to point this out in the v20 review, but
whilst I was looking at your reply here, I wondered why this isn't

		for (i = 0; i < ways; i++) {
			if (!cxled[i]->dpa_res)
				return ERR_PTR(-EINVAL);
			size += resource_size(cxled[i]->dpa_res);
		}

Given that's the only way we can get if (i < ways) true. The loop
has to have exited before the final post increment.

> >> +
> >> +		rc = alloc_hpa(cxlr, size);
> >> +		if (rc)
> >> +			return ERR_PTR(rc);
> >> +
> >> +		for (i = 0; i < ways; i++) {
> >> +			rc = cxl_region_attach(cxlr, cxled[i], 0);
> >> +			if (rc)
> >> +				return ERR_PTR(rc);
> >> +		}
> >> +	}
> >> +
> >> +	rc = cxl_region_decode_commit(cxlr);
> >> +	if (rc)
> >> +		return ERR_PTR(rc);
> >> +
> >> +	p->state = CXL_CONFIG_COMMIT;
> >> +
> >> +	return no_free_ptr(cxlr);  
> > return_ptr()  
> >> +}  
>