* [RFC PATCH 00/18] Initial CXL.cache device support
@ 2025-08-12 21:29 Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 01/18] cxl/mem: Change cxl_memdev_ops to cxl_dev_ops Ben Cheatham
` (18 more replies)
0 siblings, 19 replies; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
This patch series adds initial CXL.cache support. What I have here only
allows for adding a cache device to the system, programming/validating
the system configuration with respect to cache devices, and some basic
cache reporting/management.
The general philosophy is to have an endpoint/vendor-specific driver
that runs through the same steps of adding a cxl_memdev, but for the
cache portion of the device (both type 1 & 2). Getting cache support for
a CXL device should be as simple as: get cache information, set up the
memory region (see below), and then calling devm_cxl_add_cachedev().
There's a couple of things missing from this set:
1) Missing an endpoint driver
I plan on submitting a reference driver (type 1 or 2) with v1, I
figured I'd send out what I have before going much further.
2) Mapping/Reserving host memory used for CXL cache(s)
I'm thinking this will be handled by the endpoint driver, but I'm
not sure what mechanism would be used and whether to integrate it
with the cxl_cache/core. Any thoughts/ideas here are appreciated!
3) RAS Support
Same situation as 1) above.
Some quick notes: The actual cache parts of this set are untested due
to problems with my set up, but I did make sure nothing here breaks type
3 support. I'll have this fixed before sending out a v1. This series is
based on the for-6.18/cxl-probe-order (commit 5e29cbd1077b) branch in
the CXL repo, with v7 of Dave's deferred dport probe set on top [1]. I
added Dave's set to help with getting around constraints with HDM
decoders in CXL.cache device only configurations (see 08/18 for more).
Patch Breakdown:
- 1 & 2: Preliminary changes for struct cxl_cachedev
- 3: Add struct cxl_cachedev
- 4-8: Preliminary changes for adding cache devices to port
hierarchy
- 9: Function for getting CXL cache info
- 10: cxl_cache driver (mirrors cxl_mem)
- 11-16: Checking CXL.cache capabilities for system configuration
validity
- 17-18: Cache device attributes
[1]:
Link: https://lore.kernel.org/linux-cxl/20250714223527.461147-1-dave.jiang@intel.com/
Ben Cheatham (18):
cxl/mem: Change cxl_memdev_ops to cxl_dev_ops
cxl: Move struct cxl_dev_state definition
cxl/core: Add CXL.cache device struct
cxl: Replace cxl_mem_find_port() with cxl_dev_find_port()
cxl: Change cxl_ep_load() to use struct device * parameter
cxl/port, mem: Make adding an endpoint device type agnostic
cxl/port: Split endpoint port probe on device type
cxl/port: Update switch_port_probe() for CXL cache devices
cxl/core: Add function for getting CXL cache info
cxl/cache: Add cxl_cache driver
cxl/core: Add CXL snoop filter setup and checking
cxl/cache: Add CXL Cache ID Route Table mapping
cxl/cache: Implement Cache ID Route Table programming
cxl/cache: Add Cache ID Decoder capability mapping
cxl/cache: Implement Cache ID Decoder programming
cxl/cache: Add cache device counting for CXL ports
cxl/core: Add cache device attributes
cxl/core: Add cache device cache management attributes
drivers/cxl/Kconfig | 14 +
drivers/cxl/Makefile | 2 +
drivers/cxl/cache.c | 276 +++++++++++++++++++
drivers/cxl/core/Makefile | 1 +
drivers/cxl/core/cachedev.c | 292 ++++++++++++++++++++
drivers/cxl/core/hdm.c | 31 +++
drivers/cxl/core/memdev.c | 2 +-
drivers/cxl/core/pci.c | 134 ++++++++++
drivers/cxl/core/port.c | 518 +++++++++++++++++++++++++++++++++---
drivers/cxl/core/region.c | 25 +-
drivers/cxl/core/regs.c | 28 ++
drivers/cxl/cxl.h | 193 +++++++++++++-
drivers/cxl/cxlcache.h | 42 +++
drivers/cxl/cxlmem.h | 121 +--------
drivers/cxl/cxlpci.h | 10 +
drivers/cxl/mem.c | 14 +-
drivers/cxl/pci.c | 2 +-
drivers/cxl/port.c | 54 ++--
drivers/cxl/private.h | 8 +-
19 files changed, 1569 insertions(+), 198 deletions(-)
create mode 100644 drivers/cxl/cache.c
create mode 100644 drivers/cxl/core/cachedev.c
create mode 100644 drivers/cxl/cxlcache.h
--
2.34.1
^ permalink raw reply [flat|nested] 55+ messages in thread
* [RFC PATCH 01/18] cxl/mem: Change cxl_memdev_ops to cxl_dev_ops
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 02/18] cxl: Move struct cxl_dev_state definition Ben Cheatham
` (17 subsequent siblings)
18 siblings, 0 replies; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Change struct cxl_memdev_ops to take a struct device * instead of struct
cxl_memdev *. This is done in preparation of a new cache device type
that will also use struct cxl_dev_state and need the functionality
provided by cxl_memdev_ops.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/core/memdev.c | 2 +-
drivers/cxl/cxlmem.h | 8 ++++----
drivers/cxl/mem.c | 4 ++--
drivers/cxl/private.h | 2 +-
4 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
index b891a5cba4f4..8c3c58c92bcf 100644
--- a/drivers/cxl/core/memdev.c
+++ b/drivers/cxl/core/memdev.c
@@ -1013,7 +1013,7 @@ static const struct file_operations cxl_memdev_fops = {
};
struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
- const struct cxl_memdev_ops *ops)
+ const struct cxl_dev_ops *ops)
{
struct cxl_memdev *cxlmd;
struct device *dev;
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 075642d9dff2..5e0e551c9d6b 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -34,8 +34,8 @@
(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
CXLMDEV_RESET_NEEDED_NOT)
-struct cxl_memdev_ops {
- int (*probe)(struct cxl_memdev *cxlmd);
+struct cxl_dev_ops {
+ int (*probe)(struct device *cxldev);
};
/**
@@ -64,7 +64,7 @@ struct cxl_memdev {
struct cxl_nvdimm_bridge *cxl_nvb;
struct cxl_nvdimm *cxl_nvd;
struct cxl_port *endpoint;
- const struct cxl_memdev_ops *ops;
+ const struct cxl_dev_ops *ops;
int id;
int depth;
u8 scrub_cycle;
@@ -103,7 +103,7 @@ static inline bool is_cxl_endpoint(struct cxl_port *port)
struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
struct cxl_dev_state *cxlds,
- const struct cxl_memdev_ops *ops);
+ const struct cxl_dev_ops *ops);
int devm_cxl_sanitize_setup_notifier(struct device *host,
struct cxl_memdev *cxlmd);
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 13d9e089ecaf..c311536def38 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -144,7 +144,7 @@ static int cxl_mem_probe(struct device *dev)
}
if (cxlmd->ops) {
- rc = cxlmd->ops->probe(cxlmd);
+ rc = cxlmd->ops->probe(&cxlmd->dev);
if (rc)
return rc;
}
@@ -183,7 +183,7 @@ static int cxl_mem_probe(struct device *dev)
*/
struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
struct cxl_dev_state *cxlds,
- const struct cxl_memdev_ops *ops)
+ const struct cxl_dev_ops *ops)
{
struct cxl_memdev *cxlmd = cxl_memdev_alloc(cxlds, ops);
int rc;
diff --git a/drivers/cxl/private.h b/drivers/cxl/private.h
index f41fc6fd036f..8413db891343 100644
--- a/drivers/cxl/private.h
+++ b/drivers/cxl/private.h
@@ -9,7 +9,7 @@
#ifndef __CXL_PRIVATE_H__
#define __CXL_PRIVATE_H__
struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
- const struct cxl_memdev_ops *ops);
+ const struct cxl_dev_ops *ops);
struct cxl_memdev *devm_cxl_memdev_add_or_reset(struct device *host,
struct cxl_memdev *cxlmd);
int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 02/18] cxl: Move struct cxl_dev_state definition
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 01/18] cxl/mem: Change cxl_memdev_ops to cxl_dev_ops Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 11:33 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 03/18] cxl/core: Add CXL.cache device struct Ben Cheatham
` (16 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Move struct cxl_dev_state (and associated structs) from cxlmem.h to
cxl.h. This is in preparation of adding a new CXL cache device type
that will also use cxl_dev_state.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/cxl.h | 114 +++++++++++++++++++++++++++++++++++++++++++
drivers/cxl/cxlmem.h | 113 ------------------------------------------
2 files changed, 114 insertions(+), 113 deletions(-)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 1de4fec5c8f4..751e5860423a 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -11,6 +11,7 @@
#include <linux/log2.h>
#include <linux/node.h>
#include <linux/io.h>
+#include <cxl/mailbox.h>
extern const struct nvdimm_security_ops *cxl_security_ops;
@@ -696,6 +697,119 @@ struct cxl_ep {
struct cxl_port *next;
};
+
+/*
+ * enum cxl_devtype - delineate type-2 from a generic type-3 device
+ * @CXL_DEVTYPE_DEVMEM - Vendor specific CXL Type-2 device implementing HDM-D or
+ * HDM-DB, no requirement that this device implements a
+ * mailbox, or other memory-device-standard manageability
+ * flows.
+ * @CXL_DEVTYPE_CLASSMEM - Common class definition of a CXL Type-3 device with
+ * HDM-H and class-mandatory memory device registers
+ */
+enum cxl_devtype {
+ CXL_DEVTYPE_DEVMEM,
+ CXL_DEVTYPE_CLASSMEM,
+};
+
+/**
+ * struct cxl_dpa_perf - DPA performance property entry
+ * @dpa_range: range for DPA address
+ * @coord: QoS performance data (i.e. latency, bandwidth)
+ * @cdat_coord: raw QoS performance data from CDAT
+ * @qos_class: QoS Class cookies
+ */
+struct cxl_dpa_perf {
+ struct range dpa_range;
+ struct access_coordinate coord[ACCESS_COORDINATE_MAX];
+ struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX];
+ int qos_class;
+};
+
+/**
+ * struct cxl_dpa_partition - DPA partition descriptor
+ * @res: shortcut to the partition in the DPA resource tree (cxlds->dpa_res)
+ * @perf: performance attributes of the partition from CDAT
+ * @mode: operation mode for the DPA capacity, e.g. ram, pmem, dynamic...
+ */
+struct cxl_dpa_partition {
+ struct resource res;
+ struct cxl_dpa_perf perf;
+ enum cxl_partition_mode mode;
+};
+
+#define CXL_NR_PARTITIONS_MAX 2
+
+struct cxl_dpa_info {
+ u64 size;
+ struct cxl_dpa_part_info {
+ struct range range;
+ enum cxl_partition_mode mode;
+ } part[CXL_NR_PARTITIONS_MAX];
+ int nr_partitions;
+};
+
+struct cxl_dev_ops {
+ int (*probe)(struct device *cxldev);
+};
+
+/**
+ * struct cxl_dev_state - The driver device state
+ *
+ * cxl_dev_state represents the CXL driver/device state. It provides an
+ * interface to mailbox commands as well as some cached data about the device.
+ * Currently only memory devices are represented.
+ *
+ * @dev: The device associated with this CXL state
+ * @cxlmd: The device representing the CXL.mem capabilities of @dev
+ * @reg_map: component and ras register mapping parameters
+ * @regs: Parsed register blocks
+ * @cxl_dvsec: Offset to the PCIe device DVSEC
+ * @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH)
+ * @media_ready: Indicate whether the device media is usable
+ * @dpa_res: Overall DPA resource tree for the device
+ * @part: DPA partition array
+ * @nr_partitions: Number of DPA partitions
+ * @serial: PCIe Device Serial Number
+ * @type: Generic Memory Class device or Vendor Specific Memory device
+ * @cxl_mbox: CXL mailbox context
+ * @cxlfs: CXL features context
+ */
+struct cxl_dev_state {
+ struct device *dev;
+ struct cxl_memdev *cxlmd;
+ struct cxl_register_map reg_map;
+ struct cxl_regs regs;
+ int cxl_dvsec;
+ bool rcd;
+ bool media_ready;
+ struct resource dpa_res;
+ struct cxl_dpa_partition part[CXL_NR_PARTITIONS_MAX];
+ unsigned int nr_partitions;
+ u64 serial;
+ enum cxl_devtype type;
+ struct cxl_mailbox cxl_mbox;
+#ifdef CONFIG_CXL_FEATURES
+ struct cxl_features_state *cxlfs;
+#endif
+};
+
+static inline resource_size_t cxl_pmem_size(struct cxl_dev_state *cxlds)
+{
+ /*
+ * Static PMEM may be at partition index 0 when there is no static RAM
+ * capacity.
+ */
+ for (int i = 0; i < cxlds->nr_partitions; i++)
+ if (cxlds->part[i].mode == CXL_PARTMODE_PMEM)
+ return resource_size(&cxlds->part[i].res);
+ return 0;
+}
+
+static inline struct cxl_dev_state *mbox_to_cxlds(struct cxl_mailbox *cxl_mbox)
+{
+ return dev_get_drvdata(cxl_mbox->host);
+}
/**
* struct cxl_region_ref - track a region's interest in a port
* @port: point in topology to install this reference
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 5e0e551c9d6b..9c9d8a878785 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -34,10 +34,6 @@
(FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
CXLMDEV_RESET_NEEDED_NOT)
-struct cxl_dev_ops {
- int (*probe)(struct device *cxldev);
-};
-
/**
* struct cxl_memdev - CXL bus object representing a Type-3 Memory Device
* @dev: driver core device object
@@ -113,17 +109,6 @@ int devm_cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled,
resource_size_t base, resource_size_t len,
resource_size_t skipped);
-#define CXL_NR_PARTITIONS_MAX 2
-
-struct cxl_dpa_info {
- u64 size;
- struct cxl_dpa_part_info {
- struct range range;
- enum cxl_partition_mode mode;
- } part[CXL_NR_PARTITIONS_MAX];
- int nr_partitions;
-};
-
int cxl_dpa_setup(struct cxl_dev_state *cxlds, const struct cxl_dpa_info *info);
static inline struct cxl_ep *cxl_ep_load(struct cxl_port *port,
@@ -373,104 +358,6 @@ struct cxl_security_state {
struct kernfs_node *sanitize_node;
};
-/*
- * enum cxl_devtype - delineate type-2 from a generic type-3 device
- * @CXL_DEVTYPE_DEVMEM - Vendor specific CXL Type-2 device implementing HDM-D or
- * HDM-DB, no requirement that this device implements a
- * mailbox, or other memory-device-standard manageability
- * flows.
- * @CXL_DEVTYPE_CLASSMEM - Common class definition of a CXL Type-3 device with
- * HDM-H and class-mandatory memory device registers
- */
-enum cxl_devtype {
- CXL_DEVTYPE_DEVMEM,
- CXL_DEVTYPE_CLASSMEM,
-};
-
-/**
- * struct cxl_dpa_perf - DPA performance property entry
- * @dpa_range: range for DPA address
- * @coord: QoS performance data (i.e. latency, bandwidth)
- * @cdat_coord: raw QoS performance data from CDAT
- * @qos_class: QoS Class cookies
- */
-struct cxl_dpa_perf {
- struct range dpa_range;
- struct access_coordinate coord[ACCESS_COORDINATE_MAX];
- struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX];
- int qos_class;
-};
-
-/**
- * struct cxl_dpa_partition - DPA partition descriptor
- * @res: shortcut to the partition in the DPA resource tree (cxlds->dpa_res)
- * @perf: performance attributes of the partition from CDAT
- * @mode: operation mode for the DPA capacity, e.g. ram, pmem, dynamic...
- */
-struct cxl_dpa_partition {
- struct resource res;
- struct cxl_dpa_perf perf;
- enum cxl_partition_mode mode;
-};
-
-/**
- * struct cxl_dev_state - The driver device state
- *
- * cxl_dev_state represents the CXL driver/device state. It provides an
- * interface to mailbox commands as well as some cached data about the device.
- * Currently only memory devices are represented.
- *
- * @dev: The device associated with this CXL state
- * @cxlmd: The device representing the CXL.mem capabilities of @dev
- * @reg_map: component and ras register mapping parameters
- * @regs: Parsed register blocks
- * @cxl_dvsec: Offset to the PCIe device DVSEC
- * @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH)
- * @media_ready: Indicate whether the device media is usable
- * @dpa_res: Overall DPA resource tree for the device
- * @part: DPA partition array
- * @nr_partitions: Number of DPA partitions
- * @serial: PCIe Device Serial Number
- * @type: Generic Memory Class device or Vendor Specific Memory device
- * @cxl_mbox: CXL mailbox context
- * @cxlfs: CXL features context
- */
-struct cxl_dev_state {
- struct device *dev;
- struct cxl_memdev *cxlmd;
- struct cxl_register_map reg_map;
- struct cxl_regs regs;
- int cxl_dvsec;
- bool rcd;
- bool media_ready;
- struct resource dpa_res;
- struct cxl_dpa_partition part[CXL_NR_PARTITIONS_MAX];
- unsigned int nr_partitions;
- u64 serial;
- enum cxl_devtype type;
- struct cxl_mailbox cxl_mbox;
-#ifdef CONFIG_CXL_FEATURES
- struct cxl_features_state *cxlfs;
-#endif
-};
-
-static inline resource_size_t cxl_pmem_size(struct cxl_dev_state *cxlds)
-{
- /*
- * Static PMEM may be at partition index 0 when there is no static RAM
- * capacity.
- */
- for (int i = 0; i < cxlds->nr_partitions; i++)
- if (cxlds->part[i].mode == CXL_PARTMODE_PMEM)
- return resource_size(&cxlds->part[i].res);
- return 0;
-}
-
-static inline struct cxl_dev_state *mbox_to_cxlds(struct cxl_mailbox *cxl_mbox)
-{
- return dev_get_drvdata(cxl_mbox->host);
-}
-
/**
* struct cxl_memdev_state - Generic Type-3 Memory Device Class driver data
*
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 03/18] cxl/core: Add CXL.cache device struct
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 01/18] cxl/mem: Change cxl_memdev_ops to cxl_dev_ops Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 02/18] cxl: Move struct cxl_dev_state definition Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 11:48 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 04/18] cxl: Replace cxl_mem_find_port() with cxl_dev_find_port() Ben Cheatham
` (15 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Add a new CXL.cache device (struct cxl_cachedev) that is the cache
analogue to struct cxl_memdev. This device will be created by
endpoint vendor-specific drivers to enable and manage the cache
capabilities of the endpoint.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/core/Makefile | 1 +
drivers/cxl/core/cachedev.c | 91 +++++++++++++++++++++++++++++++++++++
drivers/cxl/core/port.c | 3 ++
drivers/cxl/cxl.h | 3 ++
drivers/cxl/cxlcache.h | 23 ++++++++++
drivers/cxl/private.h | 4 ++
6 files changed, 125 insertions(+)
create mode 100644 drivers/cxl/core/cachedev.c
create mode 100644 drivers/cxl/cxlcache.h
diff --git a/drivers/cxl/core/Makefile b/drivers/cxl/core/Makefile
index 79e2ef81fde8..227b903d6a47 100644
--- a/drivers/cxl/core/Makefile
+++ b/drivers/cxl/core/Makefile
@@ -9,6 +9,7 @@ cxl_core-y := port.o
cxl_core-y += pmem.o
cxl_core-y += regs.o
cxl_core-y += memdev.o
+cxl_core-y += cachedev.o
cxl_core-y += mbox.o
cxl_core-y += pci.o
cxl_core-y += hdm.o
diff --git a/drivers/cxl/core/cachedev.c b/drivers/cxl/core/cachedev.c
new file mode 100644
index 000000000000..aaf4a1b94178
--- /dev/null
+++ b/drivers/cxl/core/cachedev.c
@@ -0,0 +1,91 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2025 Advanced Micro Devices, Inc. */
+#include <linux/device.h>
+#include <linux/pci.h>
+
+#include "../cxlcache.h"
+#include "private.h"
+
+static DEFINE_IDA(cxl_cachedev_ida);
+
+static void cxl_cachedev_release(struct device *dev)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+
+ ida_free(&cxl_cachedev_ida, cxlcd->id);
+ kfree(cxlcd);
+}
+
+static void cxl_cachedev_unregister(void *dev)
+{
+ struct cxl_cachedev *cxlcd = dev;
+
+ cxlcd->cxlds = NULL;
+ put_device(&cxlcd->dev);
+}
+
+static char *cxl_cachedev_devnode(const struct device *dev, umode_t *mode,
+ kuid_t *uid, kgid_t *gid)
+{
+ return kasprintf(GFP_KERNEL, "cxl/%s", dev_name(dev));
+}
+
+static const struct device_type cxl_cachedev_type = {
+ .name = "cxl_cachedev",
+ .release = cxl_cachedev_release,
+ .devnode = cxl_cachedev_devnode,
+};
+
+bool is_cxl_cachedev(const struct device *dev)
+{
+ return dev->type == &cxl_cachedev_type;
+}
+EXPORT_SYMBOL_NS_GPL(is_cxl_cachedev, "CXL");
+
+static struct lock_class_key cxl_cachedev_key;
+
+struct cxl_cachedev *cxl_cachedev_alloc(struct cxl_dev_state *cxlds,
+ const struct cxl_dev_ops *ops)
+{
+ struct device *dev;
+ int rc;
+
+ struct cxl_cachedev *cxlcd __free(kfree) =
+ kzalloc(sizeof(*cxlcd), GFP_KERNEL);
+ if (!cxlcd)
+ return ERR_PTR(-ENOMEM);
+
+ rc = ida_alloc(&cxl_cachedev_ida, GFP_KERNEL);
+ if (rc < 0)
+ return ERR_PTR(rc);
+
+ cxlcd->id = rc;
+ cxlcd->depth = -1;
+ cxlcd->cxlds = cxlds;
+ cxlds->cxlcd = cxlcd;
+ cxlcd->endpoint = ERR_PTR(-ENXIO);
+ cxlcd->ops = ops;
+
+ dev = &cxlcd->dev;
+ device_initialize(dev);
+ lockdep_set_class(&dev->mutex, &cxl_cachedev_key);
+ dev->parent = cxlds->dev;
+ dev->bus = &cxl_bus_type;
+ dev->type = &cxl_cachedev_type;
+ device_set_pm_not_required(dev);
+
+ return_ptr(cxlcd);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_cachedev_alloc, "CXL");
+
+struct cxl_cachedev *devm_cxl_cachedev_add_or_reset(struct device *host,
+ struct cxl_cachedev *cxlcd)
+{
+ int rc;
+
+ rc = devm_add_action_or_reset(host, cxl_cachedev_unregister, cxlcd);
+ if (rc)
+ return ERR_PTR(rc);
+ return cxlcd;
+}
+EXPORT_SYMBOL_NS_GPL(devm_cxl_cachedev_add_or_reset, "CXL");
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index cc025bf0376e..0730a60cb722 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -11,6 +11,7 @@
#include <linux/idr.h>
#include <linux/node.h>
#include <cxl/einj.h>
+#include <cxlcache.h>
#include <cxlmem.h>
#include <cxlpci.h>
#include <cxl.h>
@@ -78,6 +79,8 @@ static int cxl_device_id(const struct device *dev)
return CXL_DEVICE_REGION;
if (dev->type == &cxl_pmu_type)
return CXL_DEVICE_PMU;
+ if (is_cxl_cachedev(dev))
+ return CXL_DEVICE_ACCELERATOR;
return 0;
}
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 751e5860423a..9ff67a58ceac 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -762,6 +762,7 @@ struct cxl_dev_ops {
*
* @dev: The device associated with this CXL state
* @cxlmd: The device representing the CXL.mem capabilities of @dev
+ * @cxlcd: The device representing the CXL.cache capabilities of @dev
* @reg_map: component and ras register mapping parameters
* @regs: Parsed register blocks
* @cxl_dvsec: Offset to the PCIe device DVSEC
@@ -778,6 +779,7 @@ struct cxl_dev_ops {
struct cxl_dev_state {
struct device *dev;
struct cxl_memdev *cxlmd;
+ struct cxl_cachedev *cxlcd;
struct cxl_register_map reg_map;
struct cxl_regs regs;
int cxl_dvsec;
@@ -968,6 +970,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);
#define CXL_DEVICE_PMEM_REGION 7
#define CXL_DEVICE_DAX_REGION 8
#define CXL_DEVICE_PMU 9
+#define CXL_DEVICE_ACCELERATOR 10
#define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
#define CXL_MODALIAS_FMT "cxl:t%d"
diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
new file mode 100644
index 000000000000..3ccbeba9fcd6
--- /dev/null
+++ b/drivers/cxl/cxlcache.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __CXL_CACHE_H__
+#define __CXL_CACHE_H__
+#include <linux/pci.h>
+
+#include "cxl.h"
+
+struct cxl_cachedev {
+ struct device dev;
+ struct cxl_dev_state *cxlds;
+ struct cxl_port *endpoint;
+ const struct cxl_dev_ops *ops;
+ int id;
+ int depth;
+};
+
+static inline struct cxl_cachedev *to_cxl_cachedev(struct device *dev)
+{
+ return container_of(dev, struct cxl_cachedev, dev);
+}
+
+bool is_cxl_cachedev(const struct device *dev);
+#endif
diff --git a/drivers/cxl/private.h b/drivers/cxl/private.h
index 8413db891343..49c588b95c19 100644
--- a/drivers/cxl/private.h
+++ b/drivers/cxl/private.h
@@ -12,6 +12,10 @@ struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
const struct cxl_dev_ops *ops);
struct cxl_memdev *devm_cxl_memdev_add_or_reset(struct device *host,
struct cxl_memdev *cxlmd);
+struct cxl_cachedev *cxl_cachedev_alloc(struct cxl_dev_state *cxlds,
+ const struct cxl_dev_ops *ops);
+struct cxl_cachedev *devm_cxl_cachedev_add_or_reset(struct device *host,
+ struct cxl_cachedev *cxlcd);
int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
struct cxl_dport *parent_dport);
#endif /* __CXL_PRIVATE_H__ */
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 04/18] cxl: Replace cxl_mem_find_port() with cxl_dev_find_port()
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (2 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 03/18] cxl/core: Add CXL.cache device struct Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 05/18] cxl: Change cxl_ep_load() to use struct device * parameter Ben Cheatham
` (14 subsequent siblings)
18 siblings, 0 replies; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Change cxl_mem_find_port() to take a struct device * instead of a struct
cxl_memdev * in preparation for adding cache devices to the port
hierarchy.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/core/port.c | 11 +++++++----
drivers/cxl/cxl.h | 2 +-
drivers/cxl/mem.c | 2 +-
drivers/cxl/pci.c | 2 +-
4 files changed, 10 insertions(+), 7 deletions(-)
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 0730a60cb722..0d27c9e8aa64 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -1960,7 +1960,7 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)
*/
if (i >= CXL_ITER_LEVEL_SWITCH) {
struct cxl_port *pport __free(put_cxl_port) =
- cxl_mem_find_port(cxlmd, &dport);
+ cxl_dev_find_port(&cxlmd->dev, &dport);
if (!pport)
goto retry;
}
@@ -2003,12 +2003,15 @@ struct cxl_port *cxl_pci_find_port(struct pci_dev *pdev,
}
EXPORT_SYMBOL_NS_GPL(cxl_pci_find_port, "CXL");
-struct cxl_port *cxl_mem_find_port(struct cxl_memdev *cxlmd,
+struct cxl_port *cxl_dev_find_port(struct device *cxldev,
struct cxl_dport **dport)
{
- return find_cxl_port(grandparent(&cxlmd->dev), dport);
+ if (!is_cxl_memdev(cxldev) && !is_cxl_cachedev(cxldev))
+ return NULL;
+
+ return find_cxl_port(grandparent(cxldev), dport);
}
-EXPORT_SYMBOL_NS_GPL(cxl_mem_find_port, "CXL");
+EXPORT_SYMBOL_NS_GPL(cxl_dev_find_port, "CXL");
static int decoder_populate_targets(struct cxl_switch_decoder *cxlsd,
struct cxl_port *port, int *target_map)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 9ff67a58ceac..3ad25229a48b 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -874,7 +874,7 @@ void cxl_bus_rescan(void);
void cxl_bus_drain(void);
struct cxl_port *cxl_pci_find_port(struct pci_dev *pdev,
struct cxl_dport **dport);
-struct cxl_port *cxl_mem_find_port(struct cxl_memdev *cxlmd,
+struct cxl_port *cxl_dev_find_port(struct device *cxldev,
struct cxl_dport **dport);
bool schedule_cxl_memdev_detach(struct cxl_memdev *cxlmd);
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index c311536def38..cdac7bfa2289 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -109,7 +109,7 @@ static int cxl_mem_probe(struct device *dev)
return rc;
struct cxl_port *parent_port __free(put_cxl_port) =
- cxl_mem_find_port(cxlmd, &dport);
+ cxl_dev_find_port(&cxlmd->dev, &dport);
if (!parent_port) {
dev_err(dev, "CXL port topology not found\n");
return -ENXIO;
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 2ff19359dc0d..a0863b04a8f0 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -827,7 +827,7 @@ static ssize_t rcd_pcie_cap_emit(struct device *dev, u16 offset, char *buf, size
struct device *root_dev;
struct cxl_dport *dport;
struct cxl_port *root __free(put_cxl_port) =
- cxl_mem_find_port(cxlmd, &dport);
+ cxl_dev_find_port(&cxlmd->dev, &dport);
if (!root)
return -ENXIO;
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 05/18] cxl: Change cxl_ep_load() to use struct device * parameter
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (3 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 04/18] cxl: Replace cxl_mem_find_port() with cxl_dev_find_port() Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 06/18] cxl/port, mem: Make adding an endpoint device type agnostic Ben Cheatham
` (13 subsequent siblings)
18 siblings, 0 replies; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Once cache devices are added the devices stored in cxl_ep refs will no
longer be gauranteed to be cxl_memdevs. Change cxl_ep_load() in
preparation of adding cache devices to the port hierarchy.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/core/port.c | 4 ++--
drivers/cxl/core/region.c | 25 +++++++++++++------------
drivers/cxl/cxlmem.h | 4 ++--
drivers/cxl/port.c | 2 +-
4 files changed, 18 insertions(+), 17 deletions(-)
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 0d27c9e8aa64..0eb406d4a131 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -1532,7 +1532,7 @@ static int port_has_memdev(struct device *dev, const void *data)
if (port->depth != ctx->depth)
return 0;
- return !!cxl_ep_load(port, ctx->cxlmd);
+ return !!cxl_ep_load(port, &ctx->cxlmd->dev);
}
static void cxl_detach_ep(void *data)
@@ -1557,7 +1557,7 @@ static void cxl_detach_ep(void *data)
parent_port = to_cxl_port(port->dev.parent);
device_lock(&parent_port->dev);
device_lock(&port->dev);
- ep = cxl_ep_load(port, cxlmd);
+ ep = cxl_ep_load(port, &cxlmd->dev);
dev_dbg(&cxlmd->dev, "disconnect %s from %s\n",
ep ? dev_name(ep->ep) : "", dev_name(&port->dev));
cxl_ep_remove(port, ep);
diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index ad60c93be803..9bd44cf4615e 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -257,8 +257,8 @@ static void cxl_region_decode_reset(struct cxl_region *cxlr, int count)
while (!is_cxl_root(to_cxl_port(iter->dev.parent)))
iter = to_cxl_port(iter->dev.parent);
- for (ep = cxl_ep_load(iter, cxlmd); iter;
- iter = ep->next, ep = cxl_ep_load(iter, cxlmd)) {
+ for (ep = cxl_ep_load(iter, &cxlmd->dev); iter;
+ iter = ep->next, ep = cxl_ep_load(iter, &cxlmd->dev)) {
struct cxl_region_ref *cxl_rr;
struct cxl_decoder *cxld;
@@ -319,8 +319,8 @@ static int cxl_region_decode_commit(struct cxl_region *cxlr)
if (rc) {
/* programming @iter failed, teardown */
- for (ep = cxl_ep_load(iter, cxlmd); ep && iter;
- iter = ep->next, ep = cxl_ep_load(iter, cxlmd)) {
+ for (ep = cxl_ep_load(iter, &cxlmd->dev); ep && iter;
+ iter = ep->next, ep = cxl_ep_load(iter, &cxlmd->dev)) {
cxl_rr = cxl_rr_load(iter, cxlr);
cxld = cxl_rr->decoder;
if (cxld->reset)
@@ -1005,7 +1005,8 @@ static int cxl_rr_ep_add(struct cxl_region_ref *cxl_rr,
struct cxl_port *port = cxl_rr->port;
struct cxl_region *cxlr = cxl_rr->region;
struct cxl_decoder *cxld = cxl_rr->decoder;
- struct cxl_ep *ep = cxl_ep_load(port, cxled_to_memdev(cxled));
+ struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
+ struct cxl_ep *ep = cxl_ep_load(port, &cxlmd->dev);
if (ep) {
rc = xa_insert(&cxl_rr->endpoints, (unsigned long)cxled, ep,
@@ -1080,7 +1081,7 @@ static int cxl_port_attach_region(struct cxl_port *port,
struct cxl_endpoint_decoder *cxled, int pos)
{
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
- struct cxl_ep *ep = cxl_ep_load(port, cxlmd);
+ struct cxl_ep *ep = cxl_ep_load(port, &cxlmd->dev);
struct cxl_region_ref *cxl_rr;
bool nr_targets_inc = false;
struct cxl_decoder *cxld;
@@ -1257,7 +1258,7 @@ static int check_last_peer(struct cxl_endpoint_decoder *cxled,
}
cxled_peer = p->targets[pos - distance];
cxlmd_peer = cxled_to_memdev(cxled_peer);
- ep_peer = cxl_ep_load(port, cxlmd_peer);
+ ep_peer = cxl_ep_load(port, &cxlmd_peer->dev);
if (ep->dport != ep_peer->dport) {
dev_dbg(&cxlr->dev,
"%s:%s: %s:%s pos %d mismatched peer %s:%s\n",
@@ -1324,7 +1325,7 @@ static int cxl_port_setup_targets(struct cxl_port *port,
struct cxl_port *parent_port = to_cxl_port(port->dev.parent);
struct cxl_region_ref *cxl_rr = cxl_rr_load(port, cxlr);
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
- struct cxl_ep *ep = cxl_ep_load(port, cxlmd);
+ struct cxl_ep *ep = cxl_ep_load(port, &cxlmd->dev);
struct cxl_region_params *p = &cxlr->params;
struct cxl_decoder *cxld = cxl_rr->decoder;
struct cxl_switch_decoder *cxlsd;
@@ -1573,8 +1574,8 @@ static void cxl_region_teardown_targets(struct cxl_region *cxlr)
while (!is_cxl_root(to_cxl_port(iter->dev.parent)))
iter = to_cxl_port(iter->dev.parent);
- for (ep = cxl_ep_load(iter, cxlmd); iter;
- iter = ep->next, ep = cxl_ep_load(iter, cxlmd))
+ for (ep = cxl_ep_load(iter, &cxlmd->dev); iter;
+ iter = ep->next, ep = cxl_ep_load(iter, &cxlmd->dev))
cxl_port_reset_targets(iter, cxlr);
}
}
@@ -1610,8 +1611,8 @@ static int cxl_region_setup_targets(struct cxl_region *cxlr)
* Descend the topology tree programming / validating
* targets while looking for conflicts.
*/
- for (ep = cxl_ep_load(iter, cxlmd); iter;
- iter = ep->next, ep = cxl_ep_load(iter, cxlmd)) {
+ for (ep = cxl_ep_load(iter, &cxlmd->dev); iter;
+ iter = ep->next, ep = cxl_ep_load(iter, &cxlmd->dev)) {
rc = cxl_port_setup_targets(iter, cxlr, cxled);
if (rc) {
cxl_region_teardown_targets(cxlr);
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 9c9d8a878785..f6af7276fa2e 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -112,12 +112,12 @@ int devm_cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled,
int cxl_dpa_setup(struct cxl_dev_state *cxlds, const struct cxl_dpa_info *info);
static inline struct cxl_ep *cxl_ep_load(struct cxl_port *port,
- struct cxl_memdev *cxlmd)
+ struct device *cxldev)
{
if (!port)
return NULL;
- return xa_load(&port->endpoints, (unsigned long)&cxlmd->dev);
+ return xa_load(&port->endpoints, (unsigned long)cxldev);
}
/*
diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c
index b991c96622d7..7d88a8bec8e8 100644
--- a/drivers/cxl/port.c
+++ b/drivers/cxl/port.c
@@ -214,7 +214,7 @@ int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
down = iter, iter = to_cxl_port(iter->dev.parent)) {
struct cxl_ep *ep;
- ep = cxl_ep_load(iter, cxlmd);
+ ep = cxl_ep_load(iter, &cxlmd->dev);
ep->next = down;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 06/18] cxl/port, mem: Make adding an endpoint device type agnostic
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (4 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 05/18] cxl: Change cxl_ep_load() to use struct device * parameter Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 11:53 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 07/18] cxl/port: Split endpoint port probe on device type Ben Cheatham
` (12 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Change devm_cxl_enumerate_ports() and devm_cxl_add_endpoint() to take a
generic struct device * instead of struct cxl_memdev in preparation of
adding cache devices to the port hierarchy.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/core/port.c | 112 +++++++++++++++++++++++++++-------------
drivers/cxl/cxl.h | 4 +-
drivers/cxl/mem.c | 8 +--
drivers/cxl/port.c | 11 ++--
drivers/cxl/private.h | 2 +-
5 files changed, 90 insertions(+), 47 deletions(-)
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 0eb406d4a131..5144ddac7145 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -1451,10 +1451,21 @@ static struct device *endpoint_host(struct cxl_port *endpoint)
return &port->dev;
}
+static struct cxl_port *cxl_dev_get_endpoint(struct device *cxldev)
+{
+ if (is_cxl_memdev(cxldev))
+ return to_cxl_memdev(cxldev)->endpoint;
+
+ if (is_cxl_cachedev(cxldev))
+ return to_cxl_cachedev(cxldev)->endpoint;
+
+ return NULL;
+}
+
static void delete_endpoint(void *data)
{
- struct cxl_memdev *cxlmd = data;
- struct cxl_port *endpoint = cxlmd->endpoint;
+ struct device *cxldev = data;
+ struct cxl_port *endpoint = cxl_dev_get_endpoint(cxldev);
struct device *host = endpoint_host(endpoint);
scoped_guard(device, host) {
@@ -1463,21 +1474,34 @@ static void delete_endpoint(void *data)
devm_release_action(host, cxl_unlink_uport, endpoint);
devm_release_action(host, unregister_port, endpoint);
}
- cxlmd->endpoint = NULL;
+
+ if (is_cxl_memdev(cxldev))
+ to_cxl_memdev(cxldev)->endpoint = NULL;
+
+ if (is_cxl_cachedev(cxldev))
+ to_cxl_cachedev(cxldev)->endpoint = NULL;
}
put_device(&endpoint->dev);
put_device(host);
}
-int cxl_endpoint_autoremove(struct cxl_memdev *cxlmd, struct cxl_port *endpoint)
+static void cxl_dev_set_depth(struct device *cxldev, int depth)
+{
+ if (is_cxl_memdev(cxldev))
+ to_cxl_memdev(cxldev)->depth = depth;
+
+ if (is_cxl_cachedev(cxldev))
+ to_cxl_cachedev(cxldev)->depth = depth;
+}
+
+int cxl_endpoint_autoremove(struct device *cxldev, struct cxl_port *endpoint)
{
struct device *host = endpoint_host(endpoint);
- struct device *dev = &cxlmd->dev;
get_device(host);
get_device(&endpoint->dev);
- cxlmd->depth = endpoint->depth;
- return devm_add_action_or_reset(dev, delete_endpoint, cxlmd);
+ cxl_dev_set_depth(cxldev, endpoint->depth);
+ return devm_add_action_or_reset(cxldev, delete_endpoint, cxldev);
}
EXPORT_SYMBOL_NS_GPL(cxl_endpoint_autoremove, "CXL");
@@ -1711,7 +1735,7 @@ static struct cxl_dport *devm_cxl_add_dport_by_uport(struct device *uport_dev,
return devm_cxl_port_add_dport(port, dport_dev);
}
-static int add_port_attach_ep(struct cxl_memdev *cxlmd,
+static int add_port_attach_ep(struct device *cxldev,
struct device *uport_dev,
struct device *dport_dev)
{
@@ -1726,7 +1750,7 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd,
* CXL-root 'cxl_port' on a previous iteration, fail for now to
* be re-probed after platform driver attaches.
*/
- dev_dbg(&cxlmd->dev, "%s is a root dport\n",
+ dev_dbg(cxldev, "%s is a root dport\n",
dev_name(dport_dev));
return -ENXIO;
}
@@ -1747,9 +1771,10 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd,
struct cxl_dport *new_dport;
if (!parent_port->dev.driver) {
- dev_warn(&cxlmd->dev,
- "port %s:%s disabled, failed to enumerate CXL.mem\n",
- dev_name(&parent_port->dev), dev_name(uport_dev));
+ dev_warn(cxldev,
+ "port %s:%s disabled, failed to enumerate CXL.%s\n",
+ dev_name(&parent_port->dev), dev_name(uport_dev),
+ is_cxl_memdev(cxldev) ? "mem" : "cache");
return -ENXIO;
}
@@ -1786,9 +1811,9 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd,
dport = new_dport;
}
- dev_dbg(&cxlmd->dev, "add to new port %s:%s\n",
+ dev_dbg(cxldev, "add to new port %s:%s\n",
dev_name(&port->dev), dev_name(port->uport_dev));
- rc = cxl_add_ep(dport, &cxlmd->dev);
+ rc = cxl_add_ep(dport, cxldev);
if (rc == -EBUSY) {
/*
* "can't" happen, but this error code means
@@ -1802,14 +1827,13 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd,
#define CXL_ITER_LEVEL_SWITCH 1
-static int get_hostbridge_port_devices(struct cxl_memdev *cxlmd,
+static int get_hostbridge_port_devices(struct device *cxldev,
struct device **hb_uport_dev,
struct device **hb_dport_dev)
{
- struct device *dev = &cxlmd->dev;
struct device *iter;
- for (iter = dev; iter; iter = grandparent(iter)) {
+ for (iter = cxldev; iter; iter = grandparent(iter)) {
struct device *dport_dev = grandparent(iter);
struct device *uport_dev = dport_dev->parent;
@@ -1823,13 +1847,13 @@ static int get_hostbridge_port_devices(struct cxl_memdev *cxlmd,
return -ENODEV;
}
-static int cxl_hostbridge_port_setup(struct cxl_memdev *cxlmd)
+static int cxl_hostbridge_port_setup(struct device *cxldev)
{
struct device *hb_uport_dev, *hb_dport_dev;
struct cxl_dport *dport = NULL;
int rc;
- rc = get_hostbridge_port_devices(cxlmd, &hb_uport_dev, &hb_dport_dev);
+ rc = get_hostbridge_port_devices(cxldev, &hb_uport_dev, &hb_dport_dev);
if (rc)
return -ENODEV;
@@ -1852,7 +1876,7 @@ static int cxl_hostbridge_port_setup(struct cxl_memdev *cxlmd)
guard(device)(&port->dev);
dport = devm_cxl_port_add_dport(port, hb_dport_dev);
if (IS_ERR(dport) && PTR_ERR(dport) != -EEXIST) {
- dev_dbg(&cxlmd->dev,
+ dev_dbg(cxldev,
"failed to add dport %s to port %s: %ld\n",
dev_name(hb_dport_dev), dev_name(&port->dev),
PTR_ERR(dport));
@@ -1872,7 +1896,7 @@ static int cxl_hostbridge_port_setup(struct cxl_memdev *cxlmd)
/* Add the dport that goes with the newly created port */
dport = devm_cxl_add_dport_by_uport(hb_uport_dev, hb_dport_dev);
if (IS_ERR(dport) && PTR_ERR(dport) != -EEXIST) {
- dev_dbg(&cxlmd->dev,
+ dev_dbg(cxldev,
"failed to add dport %s to port %s: %ld\n",
dev_name(hb_dport_dev), dev_name(&cxl_root->port.dev),
PTR_ERR(dport));
@@ -1882,27 +1906,45 @@ static int cxl_hostbridge_port_setup(struct cxl_memdev *cxlmd)
return 0;
}
-int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)
+static struct cxl_dev_state *cxl_get_dev_state(struct device *dev)
{
- struct device *dev = &cxlmd->dev;
+ if (is_cxl_memdev(dev))
+ return to_cxl_memdev(dev)->cxlds;
+
+ if (is_cxl_cachedev(dev))
+ return to_cxl_cachedev(dev)->cxlds;
+
+ return ERR_PTR(-EINVAL);
+}
+
+int devm_cxl_enumerate_ports(struct device *cxldev)
+{
+ struct cxl_dev_state *cxlds = cxl_get_dev_state(cxldev);
struct device *dgparent;
struct device *iter;
int rc, i;
+ /* An error here means @cxldev isn't a supported device type */
+ if (IS_ERR(cxlds))
+ return PTR_ERR(cxlds);
+
/*
* Skip intermediate port enumeration in the RCH case, there
* are no ports in between a host bridge and an endpoint.
*/
- if (cxlmd->cxlds->rcd)
+ if (cxlds->rcd)
return 0;
- rc = cxl_hostbridge_port_setup(cxlmd);
+ rc = cxl_hostbridge_port_setup(cxldev);
if (rc)
return rc;
- rc = devm_add_action_or_reset(&cxlmd->dev, cxl_detach_ep, cxlmd);
- if (rc)
- return rc;
+ if (is_cxl_memdev(cxldev)) {
+ rc = devm_add_action_or_reset(cxldev, cxl_detach_ep,
+ to_cxl_memdev(cxldev));
+ if (rc)
+ return rc;
+ }
/*
* Scan for and add all cxl_ports in this device's ancestry.
@@ -1910,7 +1952,7 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)
* attempt fails.
*/
retry:
- for (i = 0, iter = dev; iter; i++, iter = grandparent(iter)) {
+ for (i = 0, iter = cxldev; iter; i++, iter = grandparent(iter)) {
struct device *dport_dev = grandparent(iter);
struct device *uport_dev;
struct cxl_dport *dport;
@@ -1920,22 +1962,22 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)
uport_dev = dport_dev->parent;
if (!uport_dev) {
- dev_warn(dev, "at %s no parent for dport: %s\n",
+ dev_warn(cxldev, "at %s no parent for dport: %s\n",
dev_name(iter), dev_name(dport_dev));
return -ENXIO;
}
- dev_dbg(dev, "scan: iter: %s dport_dev: %s parent: %s\n",
+ dev_dbg(cxldev, "scan: iter: %s dport_dev: %s parent: %s\n",
dev_name(iter), dev_name(dport_dev),
dev_name(uport_dev));
struct cxl_port *port __free(put_cxl_port) =
find_cxl_port(dport_dev, &dport);
if (port) {
- dev_dbg(&cxlmd->dev,
+ dev_dbg(cxldev,
"found already registered port %s:%s\n",
dev_name(&port->dev),
dev_name(port->uport_dev));
- rc = cxl_add_ep(dport, &cxlmd->dev);
+ rc = cxl_add_ep(dport, cxldev);
/*
* If the endpoint already exists in the port's list,
@@ -1960,7 +2002,7 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)
*/
if (i >= CXL_ITER_LEVEL_SWITCH) {
struct cxl_port *pport __free(put_cxl_port) =
- cxl_dev_find_port(&cxlmd->dev, &dport);
+ cxl_dev_find_port(cxldev, &dport);
if (!pport)
goto retry;
}
@@ -1979,7 +2021,7 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)
if (IS_ERR(dport))
return PTR_ERR(dport);
} else {
- rc = add_port_attach_ep(cxlmd, uport_dev, dport_dev);
+ rc = add_port_attach_ep(cxldev, uport_dev, dport_dev);
/* port missing, try to add parent */
if (rc == -EAGAIN)
continue;
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 3ad25229a48b..c857c112772f 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -869,7 +869,7 @@ DEFINE_FREE(put_cxl_port, struct cxl_port *, if (!IS_ERR_OR_NULL(_T)) put_device
DEFINE_FREE(put_cxl_root_decoder, struct cxl_root_decoder *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->cxlsd.cxld.dev))
DEFINE_FREE(put_cxl_region, struct cxl_region *, if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
-int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd);
+int devm_cxl_enumerate_ports(struct device *cxldev);
void cxl_bus_rescan(void);
void cxl_bus_drain(void);
struct cxl_port *cxl_pci_find_port(struct pci_dev *pdev,
@@ -913,7 +913,7 @@ static inline int cxl_root_decoder_autoremove(struct device *host,
{
return cxl_decoder_autoremove(host, &cxlrd->cxlsd.cxld);
}
-int cxl_endpoint_autoremove(struct cxl_memdev *cxlmd, struct cxl_port *endpoint);
+int cxl_endpoint_autoremove(struct device *cxldev, struct cxl_port *endpoint);
/**
* struct cxl_endpoint_dvsec_info - Cached DVSEC info
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index cdac7bfa2289..c615e6c4ee60 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -104,9 +104,11 @@ static int cxl_mem_probe(struct device *dev)
if (rc)
return rc;
- rc = devm_cxl_enumerate_ports(cxlmd);
- if (rc)
+ rc = devm_cxl_enumerate_ports(&cxlmd->dev);
+ if (rc) {
+ cxlmd->endpoint = ERR_PTR(rc);
return rc;
+ }
struct cxl_port *parent_port __free(put_cxl_port) =
cxl_dev_find_port(&cxlmd->dev, &dport);
@@ -138,7 +140,7 @@ static int cxl_mem_probe(struct device *dev)
return -ENXIO;
}
- rc = devm_cxl_add_endpoint(endpoint_parent, cxlmd, dport);
+ rc = devm_cxl_add_endpoint(endpoint_parent, &cxlmd->dev, dport);
if (rc)
return rc;
}
diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c
index 7d88a8bec8e8..aa6ee6cc714e 100644
--- a/drivers/cxl/port.c
+++ b/drivers/cxl/port.c
@@ -199,7 +199,7 @@ static struct cxl_driver cxl_port_driver = {
},
};
-int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
+int devm_cxl_add_endpoint(struct device *host, struct device *cxldev,
struct cxl_dport *parent_dport)
{
struct cxl_port *parent_port = parent_dport->port;
@@ -214,23 +214,22 @@ int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
down = iter, iter = to_cxl_port(iter->dev.parent)) {
struct cxl_ep *ep;
- ep = cxl_ep_load(iter, &cxlmd->dev);
+ ep = cxl_ep_load(iter, cxldev);
ep->next = down;
}
/* Note: endpoint port component registers are derived from @cxlds */
- endpoint = devm_cxl_add_port(host, &cxlmd->dev, CXL_RESOURCE_NONE,
+ endpoint = devm_cxl_add_port(host, cxldev, CXL_RESOURCE_NONE,
parent_dport);
if (IS_ERR(endpoint))
return PTR_ERR(endpoint);
- rc = cxl_endpoint_autoremove(cxlmd, endpoint);
+ rc = cxl_endpoint_autoremove(cxldev, endpoint);
if (rc)
return rc;
if (!endpoint->dev.driver) {
- dev_err(&cxlmd->dev, "%s failed probe\n",
- dev_name(&endpoint->dev));
+ dev_err(cxldev, "%s failed probe\n", dev_name(&endpoint->dev));
return -ENXIO;
}
diff --git a/drivers/cxl/private.h b/drivers/cxl/private.h
index 49c588b95c19..04ec30ab5f31 100644
--- a/drivers/cxl/private.h
+++ b/drivers/cxl/private.h
@@ -16,6 +16,6 @@ struct cxl_cachedev *cxl_cachedev_alloc(struct cxl_dev_state *cxlds,
const struct cxl_dev_ops *ops);
struct cxl_cachedev *devm_cxl_cachedev_add_or_reset(struct device *host,
struct cxl_cachedev *cxlcd);
-int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
+int devm_cxl_add_endpoint(struct device *host, struct device *cxldev,
struct cxl_dport *parent_dport);
#endif /* __CXL_PRIVATE_H__ */
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 07/18] cxl/port: Split endpoint port probe on device type
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (5 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 06/18] cxl/port, mem: Make adding an endpoint device type agnostic Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 11:57 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 08/18] cxl/port: Update switch_port_probe() for CXL cache devices Ben Cheatham
` (11 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
The current endpoint port probe assumes the device has CXL.mem
capabilities. With the addition of cache devices, this is no longer
gauranteed. With this in mind, split endpoint port probe into separate
functions for CXL.cache and CXL.mem endpoints.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/port.c | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c
index aa6ee6cc714e..fbff4900c69d 100644
--- a/drivers/cxl/port.c
+++ b/drivers/cxl/port.c
@@ -4,6 +4,7 @@
#include <linux/module.h>
#include <linux/slab.h>
+#include "cxlcache.h"
#include "cxlmem.h"
#include "cxlpci.h"
#include "private.h"
@@ -88,7 +89,15 @@ static int cxl_switch_port_probe(struct cxl_port *port)
return -ENXIO;
}
-static int cxl_endpoint_port_probe(struct cxl_port *port)
+static int cxl_cache_endpoint_port_probe(struct cxl_port *port)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(port->uport_dev);
+
+ get_device(&cxlcd->dev);
+ return 0;
+}
+
+static int cxl_mem_endpoint_port_probe(struct cxl_port *port)
{
struct cxl_endpoint_dvsec_info info = { .port = port };
struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport_dev);
@@ -133,6 +142,17 @@ static int cxl_endpoint_port_probe(struct cxl_port *port)
return 0;
}
+static int cxl_endpoint_port_probe(struct cxl_port *port)
+{
+ if (is_cxl_memdev(port->uport_dev))
+ return cxl_mem_endpoint_port_probe(port);
+
+ if (is_cxl_cachedev(port->uport_dev))
+ return cxl_cache_endpoint_port_probe(port);
+
+ return -EINVAL;
+}
+
static int cxl_port_probe(struct device *dev)
{
struct cxl_port *port = to_cxl_port(dev);
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 08/18] cxl/port: Update switch_port_probe() for CXL cache devices
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (6 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 07/18] cxl/port: Split endpoint port probe on device type Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 12:03 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 09/18] cxl/core: Add function for getting CXL cache info Ben Cheatham
` (10 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
CXL switch port probe currently assumes there are only CXL.mem-capable
devices below the switch with preconfigured HDM decoders. Now that
CXL.cache devices can be added below a switch port, it's possible that
they either have no HDM decoders (type 1) or the CXL.mem capabilities of
the endpoint haven't been set up yet (type 2).
The HDM decoders being disabled (if present) is no longer a gauranteed
failure condition when only cache devices are present below the port.
Knowing what kind of devices are under the switch port isn't possible
until the endpoints are probed, so delay HDM setup and validation until
endpoint port probe.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/core/hdm.c | 31 +++++++++++++++++++++++++++++++
drivers/cxl/cxl.h | 1 +
drivers/cxl/port.c | 23 +++++++++++++----------
3 files changed, 45 insertions(+), 10 deletions(-)
diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
index 865a71bce251..6e04f1f4c166 100644
--- a/drivers/cxl/core/hdm.c
+++ b/drivers/cxl/core/hdm.c
@@ -205,6 +205,37 @@ struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port,
}
EXPORT_SYMBOL_NS_GPL(devm_cxl_setup_hdm, "CXL");
+int cxl_validate_endpoint_hdm_setup(struct cxl_port *endpoint)
+{
+ struct cxl_port *sp = parent_port_of(endpoint);
+ struct cxl_hdm *cxlhdm;
+ int rc;
+
+ for (; sp && !is_cxl_root(sp); sp = parent_port_of(sp)) {
+ cxlhdm = dev_get_drvdata(&sp->dev);
+ if (cxlhdm && cxlhdm->decoder_count > 0)
+ continue;
+
+ if (!sp->reg_map.component_map.hdm_decoder.valid || !cxlhdm) {
+ dev_err(&sp->dev, "Missing HDM decoder capability\n");
+ return -ENXIO;
+ }
+
+ if (sp->total_dports == 1) {
+ dev_dbg(&sp->dev, "Fallback to passthrough decoder\n");
+ rc = devm_cxl_add_passthrough_decoder(sp);
+ if (rc)
+ return rc;
+ }
+ }
+
+ if (is_cxl_root(sp))
+ return 0;
+
+ return -ENODEV;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_validate_endpoint_hdm_setup, "CXL");
+
static void __cxl_dpa_debug(struct seq_file *file, struct resource *r, int depth)
{
unsigned long long start = r->start, end = r->end;
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index c857c112772f..0dc4e5c96459 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -932,6 +932,7 @@ struct cxl_endpoint_dvsec_info {
struct cxl_hdm;
struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port,
struct cxl_endpoint_dvsec_info *info);
+int cxl_validate_endpoint_hdm_setup(struct cxl_port *endpoint);
int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,
struct cxl_endpoint_dvsec_info *info);
int devm_cxl_add_passthrough_decoder(struct cxl_port *port);
diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c
index fbff4900c69d..56030066a652 100644
--- a/drivers/cxl/port.c
+++ b/drivers/cxl/port.c
@@ -75,18 +75,17 @@ static int cxl_switch_port_probe(struct cxl_port *port)
if (!IS_ERR(cxlhdm))
return devm_cxl_enumerate_decoders(cxlhdm, NULL);
- if (PTR_ERR(cxlhdm) != -ENODEV) {
- dev_err(&port->dev, "Failed to map HDM decoder capability\n");
- return PTR_ERR(cxlhdm);
- }
-
- if (port->total_dports == 1) {
- dev_dbg(&port->dev, "Fallback to passthrough decoder\n");
- return devm_cxl_add_passthrough_decoder(port);
+ /*
+ * This isn't an error if only CXL.cache devices are below the port or
+ * a passthrough is possible. Finish set up in
+ * cxl_validate_endpoint_hdm_setup() during CXL.mem endpoint probe.
+ */
+ if (PTR_ERR(cxlhdm) == -ENODEV) {
+ dev_dbg(&port->dev, "HDM decoder capability not found\n");
+ return 0;
}
- dev_err(&port->dev, "HDM decoder capability not found\n");
- return -ENXIO;
+ return PTR_ERR(cxlhdm);
}
static int cxl_cache_endpoint_port_probe(struct cxl_port *port)
@@ -116,6 +115,10 @@ static int cxl_mem_endpoint_port_probe(struct cxl_port *port)
return PTR_ERR(cxlhdm);
}
+ rc = cxl_validate_endpoint_hdm_setup(port);
+ if (rc)
+ return rc;
+
/* Cache the data early to ensure is_visible() works */
read_cdat_data(port);
cxl_endpoint_parse_cdat(port);
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 09/18] cxl/core: Add function for getting CXL cache info
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (7 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 08/18] cxl/port: Update switch_port_probe() for CXL cache devices Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 10/18] cxl/cache: Add cxl_cache driver Ben Cheatham
` (9 subsequent siblings)
18 siblings, 0 replies; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Add a function for getting common CXL.cache information. This
information will be stored in the struct cxl_cache_state member
(cinfo) of struct cxl_dev_state for easy access by endpoint drivers.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/core/pci.c | 45 ++++++++++++++++++++++++++++++++++++++++++
drivers/cxl/cxl.h | 7 +++++++
drivers/cxl/cxlcache.h | 2 ++
drivers/cxl/cxlpci.h | 4 ++++
4 files changed, 58 insertions(+)
diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
index 717e25131db7..2279c2690c59 100644
--- a/drivers/cxl/core/pci.c
+++ b/drivers/cxl/core/pci.c
@@ -7,6 +7,7 @@
#include <linux/pci.h>
#include <linux/pci-doe.h>
#include <linux/aer.h>
+#include <cxlcache.h>
#include <cxlpci.h>
#include <cxlmem.h>
#include <cxl.h>
@@ -1187,3 +1188,47 @@ int cxl_port_update_total_dports(struct cxl_port *port)
return 0;
}
EXPORT_SYMBOL_NS_GPL(cxl_port_update_total_dports, "CXL");
+
+int cxl_accel_get_cache_info(struct cxl_dev_state *cxlds)
+{
+ struct cxl_cache_state *cstate = &cxlds->cstate;
+ struct pci_dev *pdev;
+ int dvsec, rc;
+ u16 cap, cap2;
+
+ if (!dev_is_pci(cxlds->dev))
+ return -EINVAL;
+ pdev = to_pci_dev(cxlds->dev);
+
+ dvsec = cxlds->cxl_dvsec;
+
+ rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAP_OFFSET, &cap);
+ if (rc)
+ return rc;
+
+ if (!(cap & CXL_DVSEC_CACHE_CAPABLE))
+ return -ENXIO;
+
+ rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAP2_OFFSET, &cap2);
+ if (rc)
+ return rc;
+
+ /* CXL 3.2 8.1.3.7 DVSEC CXL Capability2 for encoding */
+ switch (FIELD_GET(CXL_DVSEC_CACHE_UNIT_MASK, cap2)) {
+ case 1:
+ cstate->unit = 64;
+ break;
+ case 2:
+ cstate->unit = 1024;
+ break;
+ default:
+ return -ENXIO;
+ }
+
+ cstate->size = FIELD_GET(CXL_DVSEC_CACHE_SIZE_MASK, cap2) * cstate->unit;
+ if (!cstate->size)
+ return -ENXIO;
+
+ return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_accel_get_cache_info, "CXL");
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 0dc4e5c96459..1771c42a1e3b 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -753,6 +753,11 @@ struct cxl_dev_ops {
int (*probe)(struct device *cxldev);
};
+struct cxl_cache_state {
+ u64 size;
+ u32 unit;
+};
+
/**
* struct cxl_dev_state - The driver device state
*
@@ -774,6 +779,7 @@ struct cxl_dev_ops {
* @serial: PCIe Device Serial Number
* @type: Generic Memory Class device or Vendor Specific Memory device
* @cxl_mbox: CXL mailbox context
+ * @cstate: CXL cache state and capabilities
* @cxlfs: CXL features context
*/
struct cxl_dev_state {
@@ -791,6 +797,7 @@ struct cxl_dev_state {
u64 serial;
enum cxl_devtype type;
struct cxl_mailbox cxl_mbox;
+ struct cxl_cache_state cstate;
#ifdef CONFIG_CXL_FEATURES
struct cxl_features_state *cxlfs;
#endif
diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
index 3ccbeba9fcd6..3880312cd49d 100644
--- a/drivers/cxl/cxlcache.h
+++ b/drivers/cxl/cxlcache.h
@@ -20,4 +20,6 @@ static inline struct cxl_cachedev *to_cxl_cachedev(struct device *dev)
}
bool is_cxl_cachedev(const struct device *dev);
+
+int cxl_accel_get_cache_info(struct cxl_dev_state *cxlds);
#endif
diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h
index 7827a183128d..e5dd132be7e7 100644
--- a/drivers/cxl/cxlpci.h
+++ b/drivers/cxl/cxlpci.h
@@ -17,10 +17,14 @@
/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */
#define CXL_DVSEC_PCIE_DEVICE 0
#define CXL_DVSEC_CAP_OFFSET 0xA
+#define CXL_DVSEC_CACHE_CAPABLE BIT(0)
#define CXL_DVSEC_MEM_CAPABLE BIT(2)
#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4)
#define CXL_DVSEC_CTRL_OFFSET 0xC
#define CXL_DVSEC_MEM_ENABLE BIT(2)
+#define CXL_DVSEC_CAP2_OFFSET 0x16
+#define CXL_DVSEC_CACHE_UNIT_MASK GENMASK(3, 0)
+#define CXL_DVSEC_CACHE_SIZE_MASK GENMASK(15, 8)
#define CXL_DVSEC_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10))
#define CXL_DVSEC_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10))
#define CXL_DVSEC_MEM_INFO_VALID BIT(0)
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 10/18] cxl/cache: Add cxl_cache driver
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (8 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 09/18] cxl/core: Add function for getting CXL cache info Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 12:11 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 11/18] cxl/core: Add CXL snoop filter setup and checking Ben Cheatham
` (8 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Add the cxl_cache driver which will manage struct cxl_cachdev devices.
This driver will provide common management functions for some of a cache
capable endpoint. This driver will also be responsible for validating
the system's CXL.cache configuration.
The driver expects the device's cache capabilities to be prefetched by
the endpoint-specific driver. The required capabilities can be gotten by
calling cxl_accel_get_cache_info().
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/Kconfig | 14 +++++
drivers/cxl/Makefile | 2 +
drivers/cxl/cache.c | 116 +++++++++++++++++++++++++++++++++++++++++
drivers/cxl/cxlcache.h | 3 ++
4 files changed, 135 insertions(+)
create mode 100644 drivers/cxl/cache.c
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 48b7314afdb8..6b07212b554a 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -233,4 +233,18 @@ config CXL_MCE
def_bool y
depends on X86_MCE && MEMORY_FAILURE
+config CXL_CACHE
+ tristate "CXL: Cache Management Support"
+ depends on CXL_BUS
+ help
+ Enables a driver that manages the CXL.cache capabilities of a CXL.cache
+ capable CXL device. This driver validates and provides support for
+ programming the CXL cache device topology. This driver is required for
+ using multiple CXL.cache devices (Type 1 or Type 2) below a given
+ CXL 3.0+ capable PCIe Root Port. This driver only provides CXL cache
+ management and reporting capabilities, a vendor-specific device driver
+ is expected to enable the full capabilities of the device.
+
+ If unsure, say 'm'.
+
endif
diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
index 2caa90fa4bf2..1017206d6780 100644
--- a/drivers/cxl/Makefile
+++ b/drivers/cxl/Makefile
@@ -13,9 +13,11 @@ obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
obj-$(CONFIG_CXL_MEM) += cxl_mem.o
obj-$(CONFIG_CXL_PCI) += cxl_pci.o
+obj-$(CONFIG_CXL_CACHE) += cxl_cache.o
cxl_port-y := port.o
cxl_acpi-y := acpi.o
cxl_pmem-y := pmem.o security.o
cxl_mem-y := mem.o
cxl_pci-y := pci.o
+cxl_cache-y := cache.o
diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
new file mode 100644
index 000000000000..6e5161ae3107
--- /dev/null
+++ b/drivers/cxl/cache.c
@@ -0,0 +1,116 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2025 Advanced Micro Devices, Inc. */
+#include "cxlcache.h"
+#include "private.h"
+
+/**
+ * DOC: cxl cache
+ *
+ * The cxl_cache driver is responsible for validating the CXL.cache system
+ * configuration and managing portions of the CXL cache of CXL.cache enabled
+ * devices in the system. This driver does not discover devices, a
+ * device-specific driver is required for discovery and portions of set up.
+ */
+
+/**
+ * devm_cxl_add_cachedev - Add a CXL cache device
+ * @host: devres alloc/release context and parent for the cachedev
+ * @cxlds: CXL device state to associate with the cachedev
+ * @ops: optional operations to run in cxl_cache::{probe,remove}() context
+ *
+ * Upon return the device will have had a chance to attach to the
+ * cxl_cache driver. This may fail if the CXL topology is not ready
+ * (hardware CXL link down, or software platform CXL root not attached)
+ * or the CXL.cache system configuration is invalid.
+ */
+struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
+ struct cxl_dev_state *cxlds,
+ const struct cxl_dev_ops *ops)
+{
+ struct cxl_cachedev *cxlcd;
+ int rc;
+
+ cxlcd = cxl_cachedev_alloc(cxlds, ops);
+ if (IS_ERR(cxlcd))
+ return cxlcd;
+
+ rc = dev_set_name(&cxlcd->dev, "cache%d", cxlcd->id);
+ if (rc) {
+ put_device(&cxlcd->dev);
+ return ERR_PTR(rc);
+ }
+
+ cxlcd = devm_cxl_cachedev_add_or_reset(host, cxlcd);
+ if (IS_ERR(cxlcd))
+ return cxlcd;
+
+ guard(device)(&cxlcd->dev);
+ if (ops && !cxlcd->dev.driver) {
+ if (IS_ERR(cxlcd->endpoint))
+ return ERR_CAST(cxlcd->endpoint);
+ return ERR_PTR(-ENXIO);
+ }
+
+ return cxlcd;
+}
+EXPORT_SYMBOL_NS_GPL(devm_cxl_add_cachedev, "CXL");
+
+static int cxl_cache_probe(struct device *dev)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct device *endpoint_parent;
+ struct cxl_dport *dport;
+ int rc;
+
+ rc = devm_cxl_enumerate_ports(&cxlcd->dev);
+ if (rc)
+ return rc;
+
+ struct cxl_port *parent_port __free(put_cxl_port) =
+ cxl_dev_find_port(&cxlcd->dev, &dport);
+ if (!parent_port) {
+ dev_err(dev, "CXL port topology not found\n");
+ return -ENXIO;
+ }
+
+ if (dport->rch)
+ endpoint_parent = parent_port->uport_dev;
+ else
+ endpoint_parent = &parent_port->dev;
+
+ scoped_guard(device, endpoint_parent) {
+ if (!endpoint_parent->driver) {
+ dev_err(dev, "CXL port topology %s not enabled\n",
+ dev_name(endpoint_parent));
+ return -ENXIO;
+ }
+
+ rc = devm_cxl_add_endpoint(endpoint_parent, &cxlcd->dev, dport);
+ if (rc)
+ return rc;
+ }
+
+ if (cxlcd->ops) {
+ rc = cxlcd->ops->probe(&cxlcd->dev);
+ if (rc)
+ return rc;
+ }
+
+ return 0;
+}
+
+static struct cxl_driver cxl_cache_driver = {
+ .name = "cxl_cache",
+ .probe = cxl_cache_probe,
+ .drv = {
+ .probe_type = PROBE_FORCE_SYNCHRONOUS,
+ },
+ .id = CXL_DEVICE_ACCELERATOR,
+};
+
+module_cxl_driver(cxl_cache_driver);
+
+MODULE_DESCRIPTION("CXL: Cache Management");
+MODULE_LICENSE("GPL");
+MODULE_IMPORT_NS("CXL");
+MODULE_ALIAS_CXL(CXL_DEVICE_ACCELERATOR);
diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
index 3880312cd49d..a707e402b000 100644
--- a/drivers/cxl/cxlcache.h
+++ b/drivers/cxl/cxlcache.h
@@ -22,4 +22,7 @@ static inline struct cxl_cachedev *to_cxl_cachedev(struct device *dev)
bool is_cxl_cachedev(const struct device *dev);
int cxl_accel_get_cache_info(struct cxl_dev_state *cxlds);
+struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
+ struct cxl_dev_state *cxlds,
+ const struct cxl_dev_ops *ops);
#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 11/18] cxl/core: Add CXL snoop filter setup and checking
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (9 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 10/18] cxl/cache: Add cxl_cache driver Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 14:18 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 12/18] cxl/cache: Add CXL Cache ID Route Table mapping Ben Cheatham
` (7 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Add functions to set up and validate the snoop filter configuration of a
CXL-capable PCIe Root Port (struct cxl_dport). Allocate space in the
snoop filter as part of cxl_cachedev probe.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/cache.c | 26 +++++++
drivers/cxl/core/port.c | 145 ++++++++++++++++++++++++++++++++++++++++
drivers/cxl/core/regs.c | 7 ++
drivers/cxl/cxl.h | 14 ++++
drivers/cxl/cxlcache.h | 3 +
5 files changed, 195 insertions(+)
diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
index 6e5161ae3107..7ef36e4bbce8 100644
--- a/drivers/cxl/cache.c
+++ b/drivers/cxl/cache.c
@@ -55,11 +55,29 @@ struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
}
EXPORT_SYMBOL_NS_GPL(devm_cxl_add_cachedev, "CXL");
+static int find_snoop_gid(struct cxl_cachedev *cxlcd, u32 *gid)
+{
+ struct cxl_dport *iter = cxlcd->endpoint->parent_dport;
+
+ while (iter && !is_cxl_root(iter->port)) {
+ if (iter->snoop_id != CXL_SNOOP_ID_NO_ID) {
+ *gid = iter->snoop_id;
+ return 0;
+ }
+
+ iter = iter->port->parent_dport;
+ }
+
+ *gid = CXL_SNOOP_ID_NO_ID;
+ return -ENXIO;
+}
+
static int cxl_cache_probe(struct device *dev)
{
struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
struct device *endpoint_parent;
struct cxl_dport *dport;
+ u32 gid;
int rc;
rc = devm_cxl_enumerate_ports(&cxlcd->dev);
@@ -96,6 +114,14 @@ static int cxl_cache_probe(struct device *dev)
return rc;
}
+ rc = find_snoop_gid(cxlcd, &gid);
+ if (rc)
+ return rc;
+
+ rc = devm_cxl_snoop_filter_alloc(gid, cxlcd->cxlds);
+ if (rc)
+ return rc;
+
return 0;
}
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 5144ddac7145..26ea22a2638c 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -35,6 +35,9 @@ static DEFINE_IDA(cxl_port_ida);
static DEFINE_XARRAY(cxl_root_buses);
static DEFINE_XARRAY(cxl_root_ports);
+DECLARE_RWSEM(cxl_snoop_rwsem);
+static DEFINE_XARRAY(cxl_snoop_filters);
+
/*
* The terminal device in PCI is NULL and @platform_bus
* for platform devices (for cxl_test)
@@ -809,6 +812,139 @@ static int cxl_dport_setup_regs(struct device *host, struct cxl_dport *dport,
return rc;
}
+struct cxl_snoop_filter {
+ u64 size;
+ struct xarray dports;
+ struct xarray devs;
+ struct mutex lock; /* Used for filter allocations */
+};
+
+static struct cxl_snoop_filter *cxl_create_snoop_filter(struct cxl_dport *dport,
+ u32 gid, u32 size)
+{
+ struct cxl_snoop_filter *sf;
+ int rc;
+
+ sf = xa_load(&cxl_snoop_filters, gid);
+ if (sf)
+ return sf;
+
+ guard(rwsem_write)(&cxl_snoop_rwsem);
+ sf = kzalloc(sizeof(*sf), GFP_KERNEL);
+ if (!sf)
+ return ERR_PTR(-ENOMEM);
+
+ sf->size = size;
+ xa_init(&sf->dports);
+ xa_init(&sf->devs);
+ mutex_init(&sf->lock);
+
+ rc = xa_insert(&sf->dports, (unsigned long)dport->dport_dev, dport,
+ GFP_KERNEL);
+ if (rc)
+ goto err;
+
+ rc = xa_insert(&cxl_snoop_filters, gid, sf, GFP_KERNEL);
+ if (rc)
+ goto err;
+
+ return 0;
+
+err:
+ xa_destroy(&sf->dports);
+ xa_destroy(&sf->devs);
+ mutex_destroy(&sf->lock);
+ kfree(sf);
+ return ERR_PTR(rc);
+}
+
+static void cxl_destroy_snoop_filters(void)
+{
+ struct cxl_snoop_filter *sf;
+ unsigned long gid;
+
+ guard(rwsem_write)(&cxl_snoop_rwsem);
+ xa_for_each(&cxl_snoop_filters, gid, sf) {
+ xa_destroy(&sf->dports);
+ xa_destroy(&sf->devs);
+ mutex_destroy(&sf->lock);
+ }
+
+ xa_destroy(&cxl_snoop_filters);
+}
+
+static void cxl_snoop_filter_free(void *data)
+{
+ struct cxl_cache_state *cstate = data;
+ struct cxl_snoop_filter *sf;
+
+ sf = xa_load(&cxl_snoop_filters, cstate->snoop_id);
+ if (!sf)
+ return;
+
+ guard(mutex)(&sf->lock);
+ sf->size += cstate->size;
+}
+
+int devm_cxl_snoop_filter_alloc(u32 gid, struct cxl_dev_state *cxlds)
+{
+ struct cxl_cache_state *cstate = &cxlds->cstate;
+ struct cxl_snoop_filter *sf;
+
+ sf = xa_load(&cxl_snoop_filters, gid);
+ if (!sf)
+ return -ENODEV;
+
+ guard(mutex)(&sf->lock);
+ if (cstate->size > sf->size)
+ return -EBUSY;
+
+ sf->size -= cstate->size;
+ cstate->snoop_id = gid;
+ return devm_add_action_or_reset(cxlds->dev, cxl_snoop_filter_free,
+ cstate);
+}
+EXPORT_SYMBOL_NS_GPL(devm_cxl_snoop_filter_alloc, "CXL");
+
+int cxl_dport_setup_snoop_filter(struct cxl_dport *dport)
+{
+ struct cxl_port *port = dport->port, *parent_port;
+ struct cxl_snoop_filter *sf;
+ u32 snoop, gid, size;
+ int rc;
+
+ /* Only supported by CXL-enabled PCIe root ports */
+ parent_port = parent_port_of(port);
+ if (!parent_port || !is_cxl_root(parent_port))
+ return 0;
+
+ if (!dport->reg_map.component_map.snoop.valid) {
+ dev_dbg(dport->dport_dev, "missing snoop filter capability\n");
+ return -ENXIO;
+ }
+
+ rc = cxl_map_component_regs(&dport->reg_map, &dport->regs.component,
+ BIT(CXL_CM_CAP_CAP_ID_SNOOP));
+ if (rc)
+ return rc;
+
+ snoop = readl(dport->regs.snoop + CXL_SNOOP_GROUP_ID_OFFSET);
+ gid = FIELD_GET(CXL_SNOOP_GROUP_ID_MASK, snoop);
+
+ snoop = readl(dport->regs.snoop + CXL_SNOOP_FILTER_SIZE_OFFSET);
+ size = FIELD_GET(CXL_SNOOP_FILTER_SIZE_MASK, snoop);
+ if (!size)
+ return -ENXIO;
+
+ sf = cxl_create_snoop_filter(dport, gid, size);
+ if (IS_ERR(sf))
+ return PTR_ERR(sf);
+
+ dport->snoop_id = gid;
+ return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_dport_setup_snoop_filter, "CXL");
+
DEFINE_SHOW_ATTRIBUTE(einj_cxl_available_error_type);
static int cxl_einj_inject(void *data, u64 type)
@@ -1202,6 +1338,7 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
dport->dport_dev = dport_dev;
dport->port_id = port_id;
dport->port = port;
+ dport->snoop_id = CXL_SNOOP_ID_NO_ID;
if (rcrb == CXL_RESOURCE_NONE) {
rc = cxl_dport_setup_regs(&port->dev, dport,
@@ -1686,10 +1823,17 @@ static int update_decoders_with_dport(struct cxl_port *port, struct cxl_dport *d
static int cxl_port_setup_with_dport(struct cxl_port *port,
struct cxl_dport *dport)
{
+ int rc;
+
device_lock_assert(&port->dev);
cxl_switch_parse_cdat(dport);
+ rc = cxl_dport_setup_snoop_filter(dport);
+ if (rc)
+ dev_dbg(dport->dport_dev,
+ "failed to set up snoop filter capability: %d\n", rc);
+
return update_decoders_with_dport(port, dport);
}
@@ -2711,6 +2855,7 @@ static void cxl_core_exit(void)
bus_unregister(&cxl_bus_type);
destroy_workqueue(cxl_bus_wq);
cxl_memdev_exit();
+ cxl_destroy_snoop_filters();
debugfs_remove_recursive(cxl_debugfs);
}
diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index 5ca7b0eed568..0c8962fcad9b 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -92,6 +92,12 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
length = CXL_RAS_CAPABILITY_LENGTH;
rmap = &map->ras;
break;
+ case CXL_CM_CAP_CAP_ID_SNOOP:
+ dev_dbg(dev, "found Snoop filter capability (0x%x)\n",
+ offset);
+ length = CXL_SNOOP_FILTER_CAPABILITY_LENGTH;
+ rmap = &map->snoop;
+ break;
default:
dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id,
offset);
@@ -211,6 +217,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
} mapinfo[] = {
{ &map->component_map.hdm_decoder, ®s->hdm_decoder },
{ &map->component_map.ras, ®s->ras },
+ { &map->component_map.snoop, ®s->snoop },
};
int i;
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 1771c42a1e3b..2881e0826829 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -40,6 +40,7 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
#define CXL_CM_CAP_CAP_ID_RAS 0x2
#define CXL_CM_CAP_CAP_ID_HDM 0x5
+#define CXL_CM_CAP_CAP_ID_SNOOP 0x9
#define CXL_CM_CAP_CAP_HDM_VERSION 1
/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
@@ -151,6 +152,13 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
#define CXL_HEADERLOG_SIZE SZ_512
#define CXL_HEADERLOG_SIZE_U32 SZ_512 / sizeof(u32)
+/* CXL 3.2 8.2.4.23 CXL Snoop Filter Capability Structure */
+#define CXL_SNOOP_GROUP_ID_OFFSET 0x0
+#define CXL_SNOOP_GROUP_ID_MASK GENMASK(15, 0)
+#define CXL_SNOOP_FILTER_SIZE_OFFSET 0x4
+#define CXL_SNOOP_FILTER_SIZE_MASK GENMASK(31, 0)
+#define CXL_SNOOP_FILTER_CAPABILITY_LENGTH 0x8
+
/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
#define CXLDEV_CAP_ARRAY_OFFSET 0x0
#define CXLDEV_CAP_ARRAY_CAP_ID 0
@@ -214,6 +222,7 @@ struct cxl_regs {
struct_group_tagged(cxl_component_regs, component,
void __iomem *hdm_decoder;
void __iomem *ras;
+ void __iomem *snoop;
);
/*
* Common set of CXL Device register block base pointers
@@ -256,6 +265,7 @@ struct cxl_reg_map {
struct cxl_component_reg_map {
struct cxl_reg_map hdm_decoder;
struct cxl_reg_map ras;
+ struct cxl_reg_map snoop;
};
struct cxl_device_reg_map {
@@ -658,6 +668,8 @@ struct cxl_rcrb_info {
u16 aer_cap;
};
+#define CXL_SNOOP_ID_NO_ID (-1)
+
/**
* struct cxl_dport - CXL downstream port
* @dport_dev: PCI bridge or firmware device representing the downstream link
@@ -682,6 +694,7 @@ struct cxl_dport {
struct access_coordinate coord[ACCESS_COORDINATE_MAX];
long link_latency;
int gpf_dvsec;
+ int snoop_id;
};
/**
@@ -756,6 +769,7 @@ struct cxl_dev_ops {
struct cxl_cache_state {
u64 size;
u32 unit;
+ u16 snoop_id;
};
/**
diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
index a707e402b000..c4862f7e6edc 100644
--- a/drivers/cxl/cxlcache.h
+++ b/drivers/cxl/cxlcache.h
@@ -25,4 +25,7 @@ int cxl_accel_get_cache_info(struct cxl_dev_state *cxlds);
struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
struct cxl_dev_state *cxlds,
const struct cxl_dev_ops *ops);
+
+int cxl_dport_setup_snoop_filter(struct cxl_dport *dport);
+int devm_cxl_snoop_filter_alloc(u32 gid, struct cxl_dev_state *cxlds);
#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 12/18] cxl/cache: Add CXL Cache ID Route Table mapping
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (10 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 11/18] cxl/core: Add CXL snoop filter setup and checking Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 15:09 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 13/18] cxl/cache: Implement Cache ID Route Table programming Ben Cheatham
` (6 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
The CXL Cache ID Route Table is an optional capability implemented by
CXL host bridges and CXL upstream switch ports. This capability is
required for having multiple CXL.cache devices below a port (CXL 3.2
8.2.4.28).
Add mapping and allocation of cache ids to be used by the cache id
route table and decoder capabilities. It's possible the id has already
been programmed by BIOS, in that case use the preprogrammed value.
Programming/validation of the capabilities by the OS will come in later
commits.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/cache.c | 87 ++++++++++++++++++++++++++++++++++++++++-
drivers/cxl/core/regs.c | 13 ++++++
drivers/cxl/cxl.h | 16 ++++++++
3 files changed, 115 insertions(+), 1 deletion(-)
diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
index 7ef36e4bbce8..fd2ae1fa44bc 100644
--- a/drivers/cxl/cache.c
+++ b/drivers/cxl/cache.c
@@ -55,6 +55,90 @@ struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
}
EXPORT_SYMBOL_NS_GPL(devm_cxl_add_cachedev, "CXL");
+static DEFINE_IDA(cache_id_ida);
+
+static int map_cache_idrt_cap(struct cxl_port *port)
+{
+ if (port->regs.cidrt)
+ return 0;
+
+ if (!port->reg_map.component_map.cidrt.valid)
+ return -ENXIO;
+
+ return cxl_map_component_regs(&port->reg_map, &port->regs.component,
+ BIT(CXL_CM_CAP_CAP_ID_CIDRT));
+}
+
+static void free_cache_id(void *data)
+{
+ struct cxl_cache_state *cstate = data;
+
+ ida_free(&cache_id_ida, cstate->cache_id);
+}
+
+static int find_cache_id(struct cxl_cachedev *cxlcd)
+{
+ struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
+ u32 cap, count, entry, portn;
+ struct cxl_dport *dport;
+ struct cxl_port *port;
+ int id;
+
+ if (cstate->cache_id != CXL_CACHE_ID_NO_ID)
+ return cstate->cache_id;
+
+ dport = cxlcd->endpoint->parent_dport;
+ if (!dport)
+ return -ENODEV;
+ port = dport->port;
+
+ cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
+ count = FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK, cap);
+
+ for (id = 0; id < count; id++) {
+ entry = readl(port->regs.cidrt +
+ CXL_CACHE_IDRT_TARGETN_OFFSET(id));
+ portn = FIELD_GET(CXL_CACHE_IDRT_TARGETN_PORT_MASK, entry);
+ if (portn != dport->port_id &&
+ !(entry & CXL_CACHE_IDRT_TARGETN_VALID))
+ continue;
+
+ cstate->cache_id = id;
+ return 0;
+ }
+
+ id = ida_alloc(&cache_id_ida, GFP_KERNEL);
+ if (id < 0)
+ return id;
+
+ cstate->cache_id = id;
+ return devm_add_action_or_reset(&cxlcd->dev, free_cache_id, cstate);
+}
+
+static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
+{
+ struct cxl_port *endpoint = cxlcd->endpoint, *port;
+ struct cxl_dport *dport;
+ int rc;
+
+ for (dport = endpoint->parent_dport, port = parent_port_of(endpoint);
+ dport && port && !is_cxl_root(port);
+ dport = port->parent_dport, port = parent_port_of(port)) {
+ guard(device)(&port->dev);
+ rc = map_cache_idrt_cap(port);
+ if (rc)
+ return rc;
+
+ if (port == parent_port_of(endpoint)) {
+ rc = find_cache_id(cxlcd);
+ if (rc)
+ return rc;
+ }
+ }
+
+ return 0;
+}
+
static int find_snoop_gid(struct cxl_cachedev *cxlcd, u32 *gid)
{
struct cxl_dport *iter = cxlcd->endpoint->parent_dport;
@@ -122,7 +206,8 @@ static int cxl_cache_probe(struct device *dev)
if (rc)
return rc;
- return 0;
+
+ return devm_cxl_cachedev_allocate_cache_id(cxlcd);
}
static struct cxl_driver cxl_cache_driver = {
diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index 0c8962fcad9b..0924127bd8fd 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -98,6 +98,18 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
length = CXL_SNOOP_FILTER_CAPABILITY_LENGTH;
rmap = &map->snoop;
break;
+ case CXL_CM_CAP_CAP_ID_CIDRT: {
+ int entry_cnt;
+
+ dev_dbg(dev,
+ "found Cache ID route table capability (0x%x)\n",
+ offset);
+ entry_cnt = FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK,
+ hdr);
+ length = 2 * entry_cnt + 0x010;
+ rmap = &map->cidrt;
+ break;
+ }
default:
dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id,
offset);
@@ -218,6 +230,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
{ &map->component_map.hdm_decoder, ®s->hdm_decoder },
{ &map->component_map.ras, ®s->ras },
{ &map->component_map.snoop, ®s->snoop },
+ { &map->component_map.cidrt, ®s->cidrt },
};
int i;
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 2881e0826829..c147846855e2 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -41,6 +41,7 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
#define CXL_CM_CAP_CAP_ID_RAS 0x2
#define CXL_CM_CAP_CAP_ID_HDM 0x5
#define CXL_CM_CAP_CAP_ID_SNOOP 0x9
+#define CXL_CM_CAP_CAP_ID_CIDRT 0xD
#define CXL_CM_CAP_CAP_HDM_VERSION 1
/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
@@ -159,6 +160,15 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
#define CXL_SNOOP_FILTER_SIZE_MASK GENMASK(31, 0)
#define CXL_SNOOP_FILTER_CAPABILITY_LENGTH 0x8
+/* CXL 3.2 8.2.4.28 CXL Cache ID Route Table Capability Structure */
+#define CXL_CACHE_IDRT_CAP_OFFSET 0x0
+#define CXL_CACHE_IDRT_CAP_CNT_MASK GENMASK(4, 0)
+#define CXL_CACHE_IDRT_STAT_OFFSET 0x8
+#define CXL_CACHE_IDRT_STAT_COMMITTED BIT(0)
+#define CXL_CACHE_IDRT_TARGETN_OFFSET(n) (0x10 + (2 * (n)))
+#define CXL_CACHE_IDRT_TARGETN_VALID BIT(0)
+#define CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
+
/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
#define CXLDEV_CAP_ARRAY_OFFSET 0x0
#define CXLDEV_CAP_ARRAY_CAP_ID 0
@@ -223,6 +233,7 @@ struct cxl_regs {
void __iomem *hdm_decoder;
void __iomem *ras;
void __iomem *snoop;
+ void __iomem *cidrt;
);
/*
* Common set of CXL Device register block base pointers
@@ -266,6 +277,7 @@ struct cxl_component_reg_map {
struct cxl_reg_map hdm_decoder;
struct cxl_reg_map ras;
struct cxl_reg_map snoop;
+ struct cxl_reg_map cidrt;
};
struct cxl_device_reg_map {
@@ -618,6 +630,7 @@ struct cxl_port {
struct cxl_dport *parent_dport;
struct ida decoder_ida;
struct cxl_register_map reg_map;
+ struct cxl_regs regs;
int total_dports;
int nr_dports;
int hdm_end;
@@ -766,10 +779,13 @@ struct cxl_dev_ops {
int (*probe)(struct device *cxldev);
};
+#define CXL_CACHE_ID_NO_ID (-1)
+
struct cxl_cache_state {
u64 size;
u32 unit;
u16 snoop_id;
+ int cache_id;
};
/**
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 13/18] cxl/cache: Implement Cache ID Route Table programming
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (11 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 12/18] cxl/cache: Add CXL Cache ID Route Table mapping Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 15:07 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 14/18] cxl/cache: Add Cache ID Decoder capability mapping Ben Cheatham
` (5 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
The CXL Cache ID Route Table is an optional capability for CXL-enabled
upstream switch ports and CXL host bridges. This capability is required
for having multiple CXL.cache enabled devices under a single port
(CXL 3.2 8.2.4.28).
Implement route table programming. In the case BIOS has already
programmed the route table(s), check the configuration for validity.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/cache.c | 4 ++
drivers/cxl/core/port.c | 135 ++++++++++++++++++++++++++++++++++++++++
drivers/cxl/cxl.h | 10 +++
drivers/cxl/cxlcache.h | 4 ++
4 files changed, 153 insertions(+)
diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
index fd2ae1fa44bc..24559b9ba8e8 100644
--- a/drivers/cxl/cache.c
+++ b/drivers/cxl/cache.c
@@ -134,6 +134,10 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
if (rc)
return rc;
}
+
+ rc = devm_cxl_port_program_cache_idrt(port, dport, cxlcd);
+ if (rc)
+ return rc;
}
return 0;
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 26ea22a2638c..f491796f6e60 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -945,6 +945,141 @@ int cxl_dport_setup_snoop_filter(struct cxl_dport *dport)
}
EXPORT_SYMBOL_NS_GPL(cxl_dport_setup_snoop_filter, "CXL");
+static int cxl_port_commit_idrt(struct cxl_port *port)
+{
+ unsigned long timeout, start, end;
+ u32 cap, ctrl, status;
+ int i;
+
+ cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
+ if (!(cap & CXL_CACHE_IDRT_CAP_COMMIT))
+ return 0;
+
+ ctrl = readl(port->regs.cidrt + CXL_CACHE_IDRT_CTRL_OFFSET);
+ if (ctrl & CXL_CACHE_IDRT_CTRL_COMMIT) {
+ ctrl &= ~CXL_CACHE_IDRT_CTRL_COMMIT;
+ writel(ctrl, port->regs.cidrt + CXL_CACHE_IDRT_CTRL_OFFSET);
+ }
+
+ ctrl |= CXL_CACHE_IDRT_CTRL_COMMIT;
+ writel(ctrl, port->regs.cidrt + CXL_CACHE_IDRT_CTRL_OFFSET);
+
+ status = readl(port->regs.cidrt + CXL_CACHE_IDRT_STAT_OFFSET);
+
+ i = FIELD_GET(CXL_CACHE_IDRT_STAT_TIMEOUT_SCALE_MASK, status);
+ timeout = 1 * HZ;
+ while (i-- > 3)
+ timeout *= 10;
+
+ timeout *= FIELD_GET(CXL_CACHE_IDRT_STAT_TIMEOUT_BASE_MASK, status);
+ start = jiffies;
+ do {
+ status = readl(port->regs.cidrt + CXL_CACHE_IDRT_STAT_OFFSET);
+ if (status & CXL_CACHE_IDRT_STAT_ERR_COMMIT)
+ return -EBUSY;
+
+ if (status & CXL_CACHE_IDRT_STAT_COMMITTED)
+ return 0;
+
+ end = jiffies;
+ } while (time_before(end, start + timeout));
+
+ return -ETIMEDOUT;
+}
+
+struct cxl_cache_idrt_entry {
+ struct cxl_port *port;
+ int port_num;
+ int id;
+};
+
+static int cxl_port_enable_idrt_entry(struct cxl_cache_idrt_entry *entry)
+{
+ struct cxl_port *port = entry->port;
+ int id = entry->id, port_num;
+ u32 entry_n;
+
+ entry_n = readl(port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
+ if (entry_n & CXL_CACHE_IDRT_TARGETN_VALID) {
+ port_num = FIELD_GET(CXL_CACHE_IDRT_TARGETN_PORT_MASK, entry_n);
+ if (port_num != entry->port_num) {
+ dev_err(&port->dev,
+ "Cache ID Route Table entry%d: Invalid port id %d, expected %d\n",
+ id, port_num, entry->port_num);
+ return -EINVAL;
+ }
+
+ return 0;
+ }
+
+ entry_n &= ~CXL_CACHE_IDRT_TARGETN_PORT_MASK;
+ entry_n |= FIELD_PREP(CXL_CACHE_IDRT_TARGETN_PORT_MASK, entry->port_num);
+ entry_n |= CXL_CACHE_IDRT_TARGETN_VALID;
+ writel(entry_n, port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
+
+ return cxl_port_commit_idrt(port);
+}
+
+static void free_cache_idrt_entry(void *data)
+{
+ struct cxl_cache_idrt_entry *entry = data;
+ struct cxl_port *port = entry->port;
+ int id = entry->id;
+ u32 cap, entry_n;
+
+ cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
+ if (FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK, cap) < id)
+ return;
+
+ entry_n = readl(port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
+ entry_n &= ~CXL_CACHE_IDRT_TARGETN_VALID;
+ writel(entry_n, port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
+
+ cxl_port_commit_idrt(port);
+
+ kfree(entry);
+}
+
+int devm_cxl_port_program_cache_idrt(struct cxl_port *port,
+ struct cxl_dport *dport,
+ struct cxl_cachedev *cxlcd)
+{
+ struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
+ int id = cstate->cache_id, rc;
+ u32 cap, max_hdmd;
+
+ cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
+ if (FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK, cap) < id)
+ return -EINVAL;
+
+ if (is_cxl_root(parent_port_of(port))) {
+ max_hdmd = FIELD_GET(CXL_CACHE_IDRT_CAP_T2_MAX_MASK, cap);
+ if (cxlcd->cxlds->type == CXL_DEVTYPE_DEVMEM && cstate->hdmd)
+ port->nr_hdmd++;
+
+ if (port->nr_hdmd > max_hdmd) {
+ port->nr_hdmd--;
+ return -EINVAL;
+ }
+ }
+
+ struct cxl_cache_idrt_entry *entry __free(kfree) =
+ kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ return -ENXIO;
+
+ entry->port_num = dport->port_id;
+ entry->port = port;
+ entry->id = id;
+
+ rc = cxl_port_enable_idrt_entry(entry);
+ if (rc)
+ return rc;
+
+ return devm_add_action(&cxlcd->dev, free_cache_idrt_entry, entry);
+}
+EXPORT_SYMBOL_NS_GPL(devm_cxl_port_program_cache_idrt, "CXL");
+
DEFINE_SHOW_ATTRIBUTE(einj_cxl_available_error_type);
static int cxl_einj_inject(void *data, u64 type)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index c147846855e2..1a2918aaee62 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -163,8 +163,15 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
/* CXL 3.2 8.2.4.28 CXL Cache ID Route Table Capability Structure */
#define CXL_CACHE_IDRT_CAP_OFFSET 0x0
#define CXL_CACHE_IDRT_CAP_CNT_MASK GENMASK(4, 0)
+#define CXL_CACHE_IDRT_CAP_T2_MAX_MASK GENMASK(11, 8)
+#define CXL_CACHE_IDRT_CAP_COMMIT BIT(16)
+#define CXL_CACHE_IDRT_CTRL_OFFSET 0x4
+#define CXL_CACHE_IDRT_CTRL_COMMIT BIT(0)
#define CXL_CACHE_IDRT_STAT_OFFSET 0x8
#define CXL_CACHE_IDRT_STAT_COMMITTED BIT(0)
+#define CXL_CACHE_IDRT_STAT_ERR_COMMIT BIT(0)
+#define CXL_CACHE_IDRT_STAT_TIMEOUT_SCALE_MASK GENMASK(11, 8)
+#define CXL_CACHE_IDRT_STAT_TIMEOUT_BASE_MASK GENMASK(15, 12)
#define CXL_CACHE_IDRT_TARGETN_OFFSET(n) (0x10 + (2 * (n)))
#define CXL_CACHE_IDRT_TARGETN_VALID BIT(0)
#define CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
@@ -618,6 +625,7 @@ struct cxl_dax_region {
* @cdat: Cached CDAT data
* @cdat_available: Should a CDAT attribute be available in sysfs
* @pci_latency: Upstream latency in picoseconds
+ * @nr_hdmd: Number of HDM-D devices below port
*/
struct cxl_port {
struct device dev;
@@ -643,6 +651,7 @@ struct cxl_port {
} cdat;
bool cdat_available;
long pci_latency;
+ int nr_hdmd;
};
/**
@@ -786,6 +795,7 @@ struct cxl_cache_state {
u32 unit;
u16 snoop_id;
int cache_id;
+ bool hdmd;
};
/**
diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
index c4862f7e6edc..d28222123102 100644
--- a/drivers/cxl/cxlcache.h
+++ b/drivers/cxl/cxlcache.h
@@ -28,4 +28,8 @@ struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
int cxl_dport_setup_snoop_filter(struct cxl_dport *dport);
int devm_cxl_snoop_filter_alloc(u32 gid, struct cxl_dev_state *cxlds);
+
+int devm_cxl_port_program_cache_idrt(struct cxl_port *port,
+ struct cxl_dport *dport,
+ struct cxl_cachedev *cxlcd);
#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 14/18] cxl/cache: Add Cache ID Decoder capability mapping
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (12 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 13/18] cxl/cache: Implement Cache ID Route Table programming Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 14:12 ` Alireza Sanaee
2025-08-12 21:29 ` [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming Ben Cheatham
` (4 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
The CXL Cache ID Decoder capability is an optional capability present on
CXL downstream switch and CXL-enabled PCIe Root ports. This capability
is required for having multiple CXL.cache enabled devices under the same
port (CXL 3.2 8.2.4.29).
Add mapping of the capability as part of allocating a CXL cache id for
an endpoint. Implement mapping/validation of the capability in a later
commit.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/cache.c | 25 +++++++++++++++++++++++++
drivers/cxl/core/regs.c | 8 ++++++++
drivers/cxl/cxl.h | 6 ++++++
3 files changed, 39 insertions(+)
diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
index 24559b9ba8e8..f123d596187d 100644
--- a/drivers/cxl/cache.c
+++ b/drivers/cxl/cache.c
@@ -69,6 +69,27 @@ static int map_cache_idrt_cap(struct cxl_port *port)
BIT(CXL_CM_CAP_CAP_ID_CIDRT));
}
+static int map_cache_idd_cap(struct cxl_dport *dport)
+{
+ int agents;
+
+ if (dport->regs.cidd)
+ return 0;
+
+ /*
+ * A missing Cache ID Decoder capability is only an issue
+ * if there are multiple cache agents in the VCS
+ */
+ if (!dport->reg_map.component_map.cidd.valid) {
+ agents = atomic_read(&dport->port->cache_agents);
+ return agents > 1 ? -ENXIO : 0;
+ }
+
+ return cxl_map_component_regs(&dport->reg_map,
+ &dport->regs.component,
+ BIT(CXL_CM_CAP_CAP_ID_CIDD));
+}
+
static void free_cache_id(void *data)
{
struct cxl_cache_state *cstate = data;
@@ -138,6 +159,10 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
rc = devm_cxl_port_program_cache_idrt(port, dport, cxlcd);
if (rc)
return rc;
+
+ rc = map_cache_idd_cap(dport);
+ if (rc)
+ return rc;
}
return 0;
diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index 0924127bd8fd..1362f4156ee6 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -110,6 +110,13 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
rmap = &map->cidrt;
break;
}
+ case CXL_CM_CAP_CAP_ID_CIDD:
+ dev_dbg(dev,
+ "found Cache ID decoder capability (0x%x)\n",
+ offset);
+ length = CXL_CACHE_IDD_CAPABILITY_LENGTH;
+ rmap = &map->cidd;
+ break;
default:
dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id,
offset);
@@ -231,6 +238,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
{ &map->component_map.ras, ®s->ras },
{ &map->component_map.snoop, ®s->snoop },
{ &map->component_map.cidrt, ®s->cidrt },
+ { &map->component_map.cidd, ®s->cidd },
};
int i;
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 1a2918aaee62..04fde1a994d0 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -42,6 +42,7 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
#define CXL_CM_CAP_CAP_ID_HDM 0x5
#define CXL_CM_CAP_CAP_ID_SNOOP 0x9
#define CXL_CM_CAP_CAP_ID_CIDRT 0xD
+#define CXL_CM_CAP_CAP_ID_CIDD 0xE
#define CXL_CM_CAP_CAP_HDM_VERSION 1
/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
@@ -176,6 +177,9 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
#define CXL_CACHE_IDRT_TARGETN_VALID BIT(0)
#define CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
+/* CXL 3.2 8.2.4.29 CXL Cache ID Decoder Capability Structure */
+#define CXL_CACHE_IDD_CAPABILITY_LENGTH 0xC
+
/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
#define CXLDEV_CAP_ARRAY_OFFSET 0x0
#define CXLDEV_CAP_ARRAY_CAP_ID 0
@@ -241,6 +245,7 @@ struct cxl_regs {
void __iomem *ras;
void __iomem *snoop;
void __iomem *cidrt;
+ void __iomem *cidd;
);
/*
* Common set of CXL Device register block base pointers
@@ -285,6 +290,7 @@ struct cxl_component_reg_map {
struct cxl_reg_map ras;
struct cxl_reg_map snoop;
struct cxl_reg_map cidrt;
+ struct cxl_reg_map cidd;
};
struct cxl_device_reg_map {
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (13 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 14/18] cxl/cache: Add Cache ID Decoder capability mapping Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 13:44 ` Alireza Sanaee
2025-08-19 15:26 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 16/18] cxl/cache: Add cache device counting for CXL ports Ben Cheatham
` (3 subsequent siblings)
18 siblings, 2 replies; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
The CXL Cache ID Decoder capability is an optional capability present on
CXL downstream switch and CXL-enabled PCIe Root ports. This capability
is required for having multiple CXL.cache enabled devices under the same
port (CXL 3.2 8.2.4.29).
Implement cache id decoder programming. Validate the configuration in the
case BIOS already programmed the id(s).
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/cache.c | 4 ++
drivers/cxl/core/port.c | 109 ++++++++++++++++++++++++++++++++++++++++
drivers/cxl/cxl.h | 14 ++++++
drivers/cxl/cxlcache.h | 3 ++
4 files changed, 130 insertions(+)
diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
index f123d596187d..fa3223165a18 100644
--- a/drivers/cxl/cache.c
+++ b/drivers/cxl/cache.c
@@ -163,6 +163,10 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
rc = map_cache_idd_cap(dport);
if (rc)
return rc;
+
+ rc = cxl_dport_program_cache_idd(dport, cxlcd);
+ if (rc)
+ return rc;
}
return 0;
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index f491796f6e60..3606ad557bfb 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -1080,6 +1080,115 @@ int devm_cxl_port_program_cache_idrt(struct cxl_port *port,
}
EXPORT_SYMBOL_NS_GPL(devm_cxl_port_program_cache_idrt, "CXL");
+static int cxl_dport_commit_idd(struct cxl_dport *dport)
+{
+ unsigned long timeout, start, end;
+ u32 cap, ctrl, status;
+ int i;
+
+ cap = readl(dport->regs.cidd + CXL_CACHE_IDD_CAP_OFFSET);
+ if (!(cap & CXL_CACHE_IDD_CAP_COMMIT))
+ return 0;
+
+ ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
+ if (ctrl & CXL_CACHE_IDD_CTRL_COMMIT) {
+ ctrl &= ~CXL_CACHE_IDD_CTRL_COMMIT;
+ writel(ctrl, dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
+ }
+
+ ctrl |= CXL_CACHE_IDD_CTRL_COMMIT;
+ writel(ctrl, dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
+
+ status = readl(dport->regs.cidd + CXL_CACHE_IDD_STAT_OFFSET);
+
+ i = FIELD_GET(CXL_CACHE_IDD_STAT_TIMEOUT_SCALE_MASK, status);
+ timeout = 1 * HZ;
+ while (i-- > 3)
+ timeout *= 10;
+
+ timeout *= FIELD_GET(CXL_CACHE_IDD_STAT_TIMEOUT_BASE_MASK, status);
+ start = jiffies;
+ do {
+ status = readl(dport->regs.cidd + CXL_CACHE_IDD_STAT_OFFSET);
+ if (status & CXL_CACHE_IDD_STAT_ERR_COMMIT)
+ return -EBUSY;
+
+ if (status & CXL_CACHE_IDD_STAT_COMMITTED)
+ return 0;
+
+ end = jiffies;
+ } while (time_before(end, start + timeout));
+
+ return -ETIMEDOUT;
+}
+
+static int cxl_cache_idd_alread_programmed(struct cxl_dport *dport,
+ struct cxl_cachedev *cxlcd)
+{
+ struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
+ u32 ctrl, status, cache_id;
+
+ status = readl(dport->regs.cidd + CXL_CACHE_IDD_STAT_OFFSET);
+ if (!(status & CXL_CACHE_IDD_STAT_COMMITTED))
+ return 0;
+
+ /*
+ * The decoder is probably already committed by BIOS, validate
+ * the configuration
+ */
+ ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
+ /* TODO: Check for 68B flit mode */
+ if (dport == cxlcd->endpoint->parent_dport &&
+ !(ctrl & CXL_CACHE_IDD_CTRL_ASGN_ID))
+ return -EINVAL;
+
+ if (dport != cxlcd->endpoint->parent_dport &&
+ !(ctrl & CXL_CACHE_IDD_CTRL_FWD_ID))
+ return -EINVAL;
+
+ cache_id = FIELD_GET(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK, ctrl);
+ if (cache_id != cstate->cache_id)
+ return -EINVAL;
+
+ if (!!(ctrl & CXL_CACHE_IDD_CTRL_TYPE2) != cstate->hdmd)
+ return -EINVAL;
+
+ return 1;
+}
+
+int cxl_dport_program_cache_idd(struct cxl_dport *dport,
+ struct cxl_cachedev *cxlcd)
+{
+ u32 ctrl;
+ int rc;
+
+ rc = cxl_cache_idd_alread_programmed(dport, cxlcd);
+ if (rc)
+ return rc < 0 ? rc : 0;
+
+ ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
+
+ if (cxlcd->cxlds->cstate.hdmd) {
+ ctrl |= CXL_CACHE_IDD_CTRL_TYPE2;
+ ctrl &= ~CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK;
+ ctrl |= FIELD_PREP(CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK, cxlcd->id);
+ }
+
+ /* TODO: Check for 68B flit mode */
+ if (dport == cxlcd->endpoint->parent_dport)
+ ctrl |= CXL_CACHE_IDD_CTRL_ASGN_ID;
+ else
+ ctrl |= CXL_CACHE_IDD_CTRL_FWD_ID;
+
+ ctrl &= ~CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK;
+ ctrl |= FIELD_PREP(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK,
+ cxlcd->cxlds->cstate.cache_id);
+
+ writel(ctrl, dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
+ return cxl_dport_commit_idd(dport);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_dport_program_cache_idd, "CXL");
+
DEFINE_SHOW_ATTRIBUTE(einj_cxl_available_error_type);
static int cxl_einj_inject(void *data, u64 type)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 04fde1a994d0..9c475d8c1573 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -178,6 +178,20 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
#define CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
/* CXL 3.2 8.2.4.29 CXL Cache ID Decoder Capability Structure */
+#define CXL_CACHE_IDD_CAP_OFFSET 0x0
+#define CXL_CACHE_IDD_CAP_COMMIT BIT(0)
+#define CXL_CACHE_IDD_CTRL_OFFSET 0x4
+#define CXL_CACHE_IDD_CTRL_FWD_ID BIT(0)
+#define CXL_CACHE_IDD_CTRL_ASGN_ID BIT(1)
+#define CXL_CACHE_IDD_CTRL_TYPE2 BIT(2)
+#define CXL_CACHE_IDD_CTRL_COMMIT BIT(3)
+#define CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK GENMASK(11, 8)
+#define CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK GENMASK(19, 16)
+#define CXL_CACHE_IDD_STAT_OFFSET 0x8
+#define CXL_CACHE_IDD_STAT_COMMITTED BIT(0)
+#define CXL_CACHE_IDD_STAT_ERR_COMMIT BIT(1)
+#define CXL_CACHE_IDD_STAT_TIMEOUT_SCALE_MASK GENMASK(11, 8)
+#define CXL_CACHE_IDD_STAT_TIMEOUT_BASE_MASK GENMASK(15, 12)
#define CXL_CACHE_IDD_CAPABILITY_LENGTH 0xC
/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
index d28222123102..60563b391844 100644
--- a/drivers/cxl/cxlcache.h
+++ b/drivers/cxl/cxlcache.h
@@ -32,4 +32,7 @@ int devm_cxl_snoop_filter_alloc(u32 gid, struct cxl_dev_state *cxlds);
int devm_cxl_port_program_cache_idrt(struct cxl_port *port,
struct cxl_dport *dport,
struct cxl_cachedev *cxlcd);
+int cxl_dport_program_cache_idd(struct cxl_dport *dport,
+ struct cxl_cachedev *cxlcd);
+
#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 16/18] cxl/cache: Add cache device counting for CXL ports
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (14 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 15:30 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 17/18] cxl/core: Add cache device attributes Ben Cheatham
` (2 subsequent siblings)
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Having more than one CXL.cache enabled device under a port requires CXL
cache id capabilities (CXL 3.2 8.2.4.28/29).
Add tracking of how many cache devices are under a port. If the required
capabilities are absent and more than one device is enabled under the
port, fail allocation of a cache id.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/cache.c | 42 ++++++++++++++++++++++++++++-------------
drivers/cxl/core/port.c | 1 +
drivers/cxl/cxl.h | 2 ++
3 files changed, 32 insertions(+), 13 deletions(-)
diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
index fa3223165a18..886f9794472c 100644
--- a/drivers/cxl/cache.c
+++ b/drivers/cxl/cache.c
@@ -62,8 +62,12 @@ static int map_cache_idrt_cap(struct cxl_port *port)
if (port->regs.cidrt)
return 0;
+ /*
+ * A missing Cache ID Route Table capability is only an issue
+ * if there are multiple cache devices under the port
+ */
if (!port->reg_map.component_map.cidrt.valid)
- return -ENXIO;
+ return port->cache_devs ? -ENXIO : 0;
return cxl_map_component_regs(&port->reg_map, &port->regs.component,
BIT(CXL_CM_CAP_CAP_ID_CIDRT));
@@ -71,19 +75,11 @@ static int map_cache_idrt_cap(struct cxl_port *port)
static int map_cache_idd_cap(struct cxl_dport *dport)
{
- int agents;
-
if (dport->regs.cidd)
return 0;
- /*
- * A missing Cache ID Decoder capability is only an issue
- * if there are multiple cache agents in the VCS
- */
- if (!dport->reg_map.component_map.cidd.valid) {
- agents = atomic_read(&dport->port->cache_agents);
- return agents > 1 ? -ENXIO : 0;
- }
+ if (!dport->reg_map.component_map.cidd.valid)
+ return dport->port->cache_devs ? -ENXIO : 0;
return cxl_map_component_regs(&dport->reg_map,
&dport->regs.component,
@@ -136,6 +132,19 @@ static int find_cache_id(struct cxl_cachedev *cxlcd)
return devm_add_action_or_reset(&cxlcd->dev, free_cache_id, cstate);
}
+static void cxl_deactivate_cache(void *data)
+{
+ struct cxl_cachedev *cxlcd = data;
+ struct cxl_port *port = cxlcd->endpoint;
+
+ for (port = cxlcd->endpoint; port && !is_cxl_root(port);
+ port = parent_port_of(port)) {
+ scoped_guard(device, &port->dev) {
+ port->cache_devs--;
+ }
+ }
+}
+
static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
{
struct cxl_port *endpoint = cxlcd->endpoint, *port;
@@ -167,9 +176,17 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
rc = cxl_dport_program_cache_idd(dport, cxlcd);
if (rc)
return rc;
+
+ port->cache_devs++;
}
- return 0;
+ if (!is_cxl_root(port)) {
+ cxl_deactivate_cache(cxlcd);
+ return -ENODEV;
+ }
+
+ return devm_add_action_or_reset(&cxlcd->dev, cxl_deactivate_cache,
+ cxlcd);
}
static int find_snoop_gid(struct cxl_cachedev *cxlcd, u32 *gid)
@@ -239,7 +256,6 @@ static int cxl_cache_probe(struct device *dev)
if (rc)
return rc;
-
return devm_cxl_cachedev_allocate_cache_id(cxlcd);
}
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 3606ad557bfb..f9f1f0ff08c9 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -618,6 +618,7 @@ struct cxl_port *parent_port_of(struct cxl_port *port)
return NULL;
return port->parent_dport->port;
}
+EXPORT_SYMBOL_NS_GPL(parent_port_of, "CXL");
static void unregister_port(void *_port)
{
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 9c475d8c1573..c67da9a378df 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -646,6 +646,7 @@ struct cxl_dax_region {
* @cdat_available: Should a CDAT attribute be available in sysfs
* @pci_latency: Upstream latency in picoseconds
* @nr_hdmd: Number of HDM-D devices below port
+ * @cache_devs: Number of CXL.cache devices below this port
*/
struct cxl_port {
struct device dev;
@@ -672,6 +673,7 @@ struct cxl_port {
bool cdat_available;
long pci_latency;
int nr_hdmd;
+ int cache_devs;
};
/**
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 17/18] cxl/core: Add cache device attributes
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (15 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 16/18] cxl/cache: Add cache device counting for CXL ports Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 15:38 ` Jonathan Cameron
2025-08-12 21:29 ` [RFC PATCH 18/18] cxl/core: Add cache device cache management attributes Ben Cheatham
2025-08-13 11:25 ` [RFC PATCH 00/18] Initial CXL.cache device support Alejandro Lucero Palau
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Add sysfs attributes for getting the serial number, numa node, CXL cache
unit, and CXL cache size to struct cxl_cachedev.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/core/cachedev.c | 102 ++++++++++++++++++++++++++++++++++++
1 file changed, 102 insertions(+)
diff --git a/drivers/cxl/core/cachedev.c b/drivers/cxl/core/cachedev.c
index aaf4a1b94178..1cdd94520ceb 100644
--- a/drivers/cxl/core/cachedev.c
+++ b/drivers/cxl/core/cachedev.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (C) 2025 Advanced Micro Devices, Inc. */
+#include <linux/string_helpers.h>
#include <linux/device.h>
#include <linux/pci.h>
@@ -30,10 +31,111 @@ static char *cxl_cachedev_devnode(const struct device *dev, umode_t *mode,
return kasprintf(GFP_KERNEL, "cxl/%s", dev_name(dev));
}
+static ssize_t serial_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct cxl_dev_state *cxlds = cxlcd->cxlds;
+
+ return sysfs_emit(buf, "%#llx\n", cxlds->serial);
+}
+static DEVICE_ATTR_RO(serial);
+
+static ssize_t numa_node_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ return sysfs_emit(buf, "%d\n", dev_to_node(dev));
+}
+static DEVICE_ATTR_RO(numa_node);
+
+static struct attribute *cxl_cachedev_attributes[] = {
+ &dev_attr_serial.attr,
+ &dev_attr_numa_node.attr,
+};
+
+static umode_t cxl_cachedev_visible(struct kobject *kobj, struct attribute *a,
+ int n)
+{
+ if (!IS_ENABLED(CONFIG_NUMA) && a == &dev_attr_numa_node.attr)
+ return 0;
+ return a->mode;
+}
+
+static struct attribute_group cxl_cachedev_attribute_group = {
+ .attrs = cxl_cachedev_attributes,
+ .is_visible = cxl_cachedev_visible,
+};
+
+static ssize_t cache_size_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct cxl_dev_state *cxlds = cxlcd->cxlds;
+ struct cxl_cache_state cstate = cxlds->cstate;
+ char size_buf[32];
+ int rc;
+
+ rc = string_get_size(cstate.size, SZ_1K, STRING_UNITS_2, size_buf,
+ sizeof(size_buf) - 1);
+ if (rc <= 0)
+ return -ENXIO;
+
+ return sysfs_emit(buf, "%s\n", size_buf);
+}
+static DEVICE_ATTR_RO(cache_size);
+
+static ssize_t cache_unit_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct cxl_dev_state *cxlds = cxlcd->cxlds;
+ struct cxl_cache_state cstate = cxlds->cstate;
+ char unit_buf[32];
+ int rc;
+
+ rc = string_get_size(cstate.size, SZ_1K, STRING_UNITS_2, unit_buf,
+ sizeof(unit_buf) - 1);
+ if (rc <= 0)
+ return -ENXIO;
+
+ return sysfs_emit(buf, "%s\n", unit_buf);
+}
+static DEVICE_ATTR_RO(cache_unit);
+
+static struct attribute *cxl_cachedev_cache_attributes[] = {
+ &dev_attr_cache_size.attr,
+ &dev_attr_cache_unit.attr,
+ NULL,
+};
+
+static umode_t cxl_cachedev_cache_visible(struct kobject *kobj,
+ struct attribute *a, int n)
+{
+ struct device *dev = kobj_to_dev(kobj);
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct cxl_dev_state *cxlds = cxlcd->cxlds;
+
+ if (!cxlds || cxlds->cstate.size == 0)
+ return 0;
+ return a->mode;
+}
+
+static struct attribute_group cxl_cachedev_cache_attribute_group = {
+ .attrs = cxl_cachedev_cache_attributes,
+ .is_visible = cxl_cachedev_cache_visible,
+};
+
+static const struct attribute_group *cxl_cachedev_attribute_groups[] = {
+ &cxl_cachedev_attribute_group,
+ &cxl_cachedev_cache_attribute_group,
+ NULL,
+};
+
static const struct device_type cxl_cachedev_type = {
.name = "cxl_cachedev",
.release = cxl_cachedev_release,
.devnode = cxl_cachedev_devnode,
+ .groups = cxl_cachedev_attribute_groups,
};
bool is_cxl_cachedev(const struct device *dev)
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* [RFC PATCH 18/18] cxl/core: Add cache device cache management attributes
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (16 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 17/18] cxl/core: Add cache device attributes Ben Cheatham
@ 2025-08-12 21:29 ` Ben Cheatham
2025-08-19 15:53 ` Jonathan Cameron
2025-08-13 11:25 ` [RFC PATCH 00/18] Initial CXL.cache device support Alejandro Lucero Palau
18 siblings, 1 reply; 55+ messages in thread
From: Ben Cheatham @ 2025-08-12 21:29 UTC (permalink / raw)
To: linux-cxl; +Cc: Ben Cheatham
Add functions to cxl/core/pci.c to manage the cache of a CXL.cache
enabled endpoint. Add these new functions to sysfs attributes for
userspace accessibility.
Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
---
drivers/cxl/core/cachedev.c | 99 +++++++++++++++++++++++++++++++++++++
drivers/cxl/core/pci.c | 89 +++++++++++++++++++++++++++++++++
drivers/cxl/cxlcache.h | 4 ++
drivers/cxl/cxlpci.h | 6 +++
4 files changed, 198 insertions(+)
diff --git a/drivers/cxl/core/cachedev.c b/drivers/cxl/core/cachedev.c
index 1cdd94520ceb..85989cdd1eb8 100644
--- a/drivers/cxl/core/cachedev.c
+++ b/drivers/cxl/core/cachedev.c
@@ -5,6 +5,7 @@
#include <linux/pci.h>
#include "../cxlcache.h"
+#include "../cxlpci.h"
#include "private.h"
static DEFINE_IDA(cxl_cachedev_ida);
@@ -125,9 +126,107 @@ static struct attribute_group cxl_cachedev_cache_attribute_group = {
.is_visible = cxl_cachedev_cache_visible,
};
+static ssize_t cache_disable_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t n)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct cxl_dev_state *cxlds = cxlcd->cxlds;
+ bool disable;
+ int rc;
+
+ rc = kstrtobool(buf, &disable);
+ if (rc)
+ return rc;
+
+ rc = cxl_accel_set_cache_disable(cxlds, disable);
+ if (rc)
+ return rc;
+
+ return n;
+}
+
+static ssize_t cache_disable_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct cxl_dev_state *cxlds = cxlcd->cxlds;
+ int rc;
+
+ rc = cxl_accel_caching_disabled(cxlds);
+ if (rc < 0)
+ return rc;
+
+ return sysfs_emit(buf, "%d\n", rc);
+}
+static DEVICE_ATTR_RW(cache_disable);
+
+static ssize_t cache_invalid_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct cxl_dev_state *cxlds = cxlcd->cxlds;
+ int rc;
+
+ rc = cxl_accel_cache_invalid(cxlds);
+ if (rc < 0)
+ return rc;
+
+ return sysfs_emit(buf, "%d\n", rc);
+}
+static DEVICE_ATTR_RO(cache_invalid);
+
+static ssize_t init_wbinvd_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t n)
+{
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct cxl_dev_state *cxlds = cxlcd->cxlds;
+ int rc;
+
+ rc = cxl_accel_initiate_wbinvd(cxlds);
+ return rc ? rc : n;
+}
+static DEVICE_ATTR_WO(init_wbinvd);
+
+static struct attribute *cxl_cachedev_mgmt_attributes[] = {
+ &dev_attr_cache_disable.attr,
+ &dev_attr_cache_invalid.attr,
+ &dev_attr_init_wbinvd.attr,
+ NULL,
+};
+
+static umode_t cxl_cachedev_mgmt_visible(struct kobject *kobj,
+ struct attribute *a, int n)
+{
+ struct device *dev = kobj_to_dev(kobj);
+ struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
+ struct cxl_dev_state *cxlds = cxlcd->cxlds;
+ struct pci_dev *pdev = to_pci_dev(cxlds->dev);
+ u16 cap;
+ int rc;
+
+ rc = pci_read_config_word(pdev, cxlds->cxl_dvsec + CXL_DVSEC_CAP_OFFSET,
+ &cap);
+ if (rc)
+ return 0;
+
+ if (!(cap & CXL_DVSEC_WBINVD_CAPABLE) &&
+ a == &dev_attr_init_wbinvd.attr)
+ return 0;
+
+ return a->mode;
+}
+
+static struct attribute_group cxl_cachedev_mgmt_attribute_group = {
+ .attrs = cxl_cachedev_mgmt_attributes,
+ .is_visible = cxl_cachedev_mgmt_visible,
+};
+
static const struct attribute_group *cxl_cachedev_attribute_groups[] = {
&cxl_cachedev_attribute_group,
&cxl_cachedev_cache_attribute_group,
+ &cxl_cachedev_mgmt_attribute_group,
NULL,
};
diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
index 2279c2690c59..667f75043f9e 100644
--- a/drivers/cxl/core/pci.c
+++ b/drivers/cxl/core/pci.c
@@ -1232,3 +1232,92 @@ int cxl_accel_get_cache_info(struct cxl_dev_state *cxlds)
return 0;
}
EXPORT_SYMBOL_NS_GPL(cxl_accel_get_cache_info, "CXL");
+
+int cxl_accel_caching_disabled(struct cxl_dev_state *cxlds)
+{
+ struct pci_dev *pdev = to_pci_dev(cxlds->dev);
+ int dvsec, rc;
+ u16 ctrl2;
+
+ device_lock_assert(cxlds->dev);
+
+ if (!dev_is_pci(cxlds->dev))
+ return -EINVAL;
+ pdev = to_pci_dev(cxlds->dev);
+
+ dvsec = cxlds->cxl_dvsec;
+ rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &ctrl2);
+ if (rc)
+ return rc;
+
+ return !!(ctrl2 & CXL_DVSEC_DISABLE_CACHING);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_accel_caching_disabled, "CXL");
+
+int cxl_accel_set_cache_disable(struct cxl_dev_state *cxlds, bool disable)
+{
+ struct pci_dev *pdev = to_pci_dev(cxlds->dev);
+ int dvsec, rc;
+ u16 ctrl2;
+
+ if (!dev_is_pci(cxlds->dev))
+ return -EINVAL;
+
+ guard(device)(cxlds->dev);
+ dvsec = cxlds->cxl_dvsec;
+ rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &ctrl2);
+ if (rc)
+ return rc;
+
+ if (!!(ctrl2 & CXL_DVSEC_DISABLE_CACHING) == disable)
+ return 1;
+
+ ctrl2 &= ~CXL_DVSEC_DISABLE_CACHING;
+ ctrl2 |= FIELD_PREP(CXL_DVSEC_DISABLE_CACHING, disable);
+ return pci_write_config_word(pdev, dvsec + CXL_DVSEC_DISABLE_CACHING,
+ ctrl2);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_accel_set_cache_disable, "CXL");
+
+int cxl_accel_initiate_wbinvd(struct cxl_dev_state *cxlds)
+{
+ struct pci_dev *pdev = to_pci_dev(cxlds->dev);
+ int dvsec, rc;
+ u16 ctrl2;
+
+ if (!dev_is_pci(cxlds->dev))
+ return -EINVAL;
+
+ if (cxl_accel_caching_disabled(cxlds) != 1)
+ return -EBUSY;
+
+ guard(device)(cxlds->dev);
+ dvsec = cxlds->cxl_dvsec;
+ rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &ctrl2);
+ if (rc)
+ return rc;
+
+ ctrl2 |= CXL_DVSEC_INIT_WBINVD;
+ return pci_write_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET,
+ ctrl2);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_accel_initiate_wbinvd, "CXL");
+
+int cxl_accel_cache_invalid(struct cxl_dev_state *cxlds)
+{
+ struct pci_dev *pdev = to_pci_dev(cxlds->dev);
+ int dvsec, rc;
+ u16 stat2;
+
+ if (!dev_is_pci(cxlds->dev))
+ return -EINVAL;
+
+ guard(device)(cxlds->dev);
+ dvsec = cxlds->cxl_dvsec;
+ rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_STAT2_OFFSET, &stat2);
+ if (rc)
+ return rc;
+
+ return !!(stat2 & CXL_DVSEC_CACHE_INVALID);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_accel_cache_invalid, "CXL");
diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
index 60563b391844..14e6f543ade9 100644
--- a/drivers/cxl/cxlcache.h
+++ b/drivers/cxl/cxlcache.h
@@ -35,4 +35,8 @@ int devm_cxl_port_program_cache_idrt(struct cxl_port *port,
int cxl_dport_program_cache_idd(struct cxl_dport *dport,
struct cxl_cachedev *cxlcd);
+int cxl_accel_set_cache_disable(struct cxl_dev_state *cxlds, bool disable);
+int cxl_accel_caching_disabled(struct cxl_dev_state *cxlds);
+int cxl_accel_cache_invalid(struct cxl_dev_state *cxlds);
+int cxl_accel_initiate_wbinvd(struct cxl_dev_state *cxlds);
#endif
diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h
index e5dd132be7e7..dc0800d79cf6 100644
--- a/drivers/cxl/cxlpci.h
+++ b/drivers/cxl/cxlpci.h
@@ -20,8 +20,14 @@
#define CXL_DVSEC_CACHE_CAPABLE BIT(0)
#define CXL_DVSEC_MEM_CAPABLE BIT(2)
#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4)
+#define CXL_DVSEC_WBINVD_CAPABLE BIT(6)
#define CXL_DVSEC_CTRL_OFFSET 0xC
#define CXL_DVSEC_MEM_ENABLE BIT(2)
+#define CXL_DVSEC_CTRL2_OFFSET 0x10
+#define CXL_DVSEC_DISABLE_CACHING BIT(0)
+#define CXL_DVSEC_INIT_WBINVD BIT(1)
+#define CXL_DVSEC_STAT2_OFFSET 0x12
+#define CXL_DVSEC_CACHE_INVALID BIT(0)
#define CXL_DVSEC_CAP2_OFFSET 0x16
#define CXL_DVSEC_CACHE_UNIT_MASK GENMASK(3, 0)
#define CXL_DVSEC_CACHE_SIZE_MASK GENMASK(15, 8)
--
2.34.1
^ permalink raw reply related [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 00/18] Initial CXL.cache device support
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
` (17 preceding siblings ...)
2025-08-12 21:29 ` [RFC PATCH 18/18] cxl/core: Add cache device cache management attributes Ben Cheatham
@ 2025-08-13 11:25 ` Alejandro Lucero Palau
2025-08-19 15:57 ` Jonathan Cameron
2025-08-22 18:02 ` Cheatham, Benjamin
18 siblings, 2 replies; 55+ messages in thread
From: Alejandro Lucero Palau @ 2025-08-13 11:25 UTC (permalink / raw)
To: Ben Cheatham, linux-cxl
On 8/12/25 22:29, Ben Cheatham wrote:
> This patch series adds initial CXL.cache support. What I have here only
> allows for adding a cache device to the system, programming/validating
> the system configuration with respect to cache devices, and some basic
> cache reporting/management.
>
> The general philosophy is to have an endpoint/vendor-specific driver
> that runs through the same steps of adding a cxl_memdev, but for the
> cache portion of the device (both type 1 & 2). Getting cache support for
> a CXL device should be as simple as: get cache information, set up the
> memory region (see below), and then calling devm_cxl_add_cachedev().
>
> There's a couple of things missing from this set:
>
> 1) Missing an endpoint driver
>
> I plan on submitting a reference driver (type 1 or 2) with v1, I
> figured I'd send out what I have before going much further.
>
> 2) Mapping/Reserving host memory used for CXL cache(s)
>
> I'm thinking this will be handled by the endpoint driver, but I'm
> not sure what mechanism would be used and whether to integrate it
> with the cxl_cache/core. Any thoughts/ideas here are appreciated!
Hi Ben,
Good to see you working on this. I'll go through the series these next days.
FWIW, I did send a RFC last November about my concerns for
properly/safely supporting CXL.cache.
I had no time to keep thinking about it. Also, I sent a copy to the
kernel iommu mailing list, but it did not get traction then. Anyways, as
a summary, I think mapping for CXL.cache requires not just the endpoint
driver doing it freely but through the cxl core for ensuring it is done
properly and safely. But I have to admit this could be a
misunderstanding about how the hardware is going to allow CXL.cache.
Thanks
> 3) RAS Support
>
> Same situation as 1) above.
>
> Some quick notes: The actual cache parts of this set are untested due
> to problems with my set up, but I did make sure nothing here breaks type
> 3 support. I'll have this fixed before sending out a v1. This series is
> based on the for-6.18/cxl-probe-order (commit 5e29cbd1077b) branch in
> the CXL repo, with v7 of Dave's deferred dport probe set on top [1]. I
> added Dave's set to help with getting around constraints with HDM
> decoders in CXL.cache device only configurations (see 08/18 for more).
>
> Patch Breakdown:
> - 1 & 2: Preliminary changes for struct cxl_cachedev
> - 3: Add struct cxl_cachedev
> - 4-8: Preliminary changes for adding cache devices to port
> hierarchy
> - 9: Function for getting CXL cache info
> - 10: cxl_cache driver (mirrors cxl_mem)
> - 11-16: Checking CXL.cache capabilities for system configuration
> validity
> - 17-18: Cache device attributes
>
> [1]:
> Link: https://lore.kernel.org/linux-cxl/20250714223527.461147-1-dave.jiang@intel.com/
>
> Ben Cheatham (18):
> cxl/mem: Change cxl_memdev_ops to cxl_dev_ops
> cxl: Move struct cxl_dev_state definition
> cxl/core: Add CXL.cache device struct
> cxl: Replace cxl_mem_find_port() with cxl_dev_find_port()
> cxl: Change cxl_ep_load() to use struct device * parameter
> cxl/port, mem: Make adding an endpoint device type agnostic
> cxl/port: Split endpoint port probe on device type
> cxl/port: Update switch_port_probe() for CXL cache devices
> cxl/core: Add function for getting CXL cache info
> cxl/cache: Add cxl_cache driver
> cxl/core: Add CXL snoop filter setup and checking
> cxl/cache: Add CXL Cache ID Route Table mapping
> cxl/cache: Implement Cache ID Route Table programming
> cxl/cache: Add Cache ID Decoder capability mapping
> cxl/cache: Implement Cache ID Decoder programming
> cxl/cache: Add cache device counting for CXL ports
> cxl/core: Add cache device attributes
> cxl/core: Add cache device cache management attributes
>
> drivers/cxl/Kconfig | 14 +
> drivers/cxl/Makefile | 2 +
> drivers/cxl/cache.c | 276 +++++++++++++++++++
> drivers/cxl/core/Makefile | 1 +
> drivers/cxl/core/cachedev.c | 292 ++++++++++++++++++++
> drivers/cxl/core/hdm.c | 31 +++
> drivers/cxl/core/memdev.c | 2 +-
> drivers/cxl/core/pci.c | 134 ++++++++++
> drivers/cxl/core/port.c | 518 +++++++++++++++++++++++++++++++++---
> drivers/cxl/core/region.c | 25 +-
> drivers/cxl/core/regs.c | 28 ++
> drivers/cxl/cxl.h | 193 +++++++++++++-
> drivers/cxl/cxlcache.h | 42 +++
> drivers/cxl/cxlmem.h | 121 +--------
> drivers/cxl/cxlpci.h | 10 +
> drivers/cxl/mem.c | 14 +-
> drivers/cxl/pci.c | 2 +-
> drivers/cxl/port.c | 54 ++--
> drivers/cxl/private.h | 8 +-
> 19 files changed, 1569 insertions(+), 198 deletions(-)
> create mode 100644 drivers/cxl/cache.c
> create mode 100644 drivers/cxl/core/cachedev.c
> create mode 100644 drivers/cxl/cxlcache.h
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 02/18] cxl: Move struct cxl_dev_state definition
2025-08-12 21:29 ` [RFC PATCH 02/18] cxl: Move struct cxl_dev_state definition Ben Cheatham
@ 2025-08-19 11:33 ` Jonathan Cameron
2025-08-22 18:00 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 11:33 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:05 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> Move struct cxl_dev_state (and associated structs) from cxlmem.h to
> cxl.h. This is in preparation of adding a new CXL cache device type
> that will also use cxl_dev_state.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
Hi Ben,
I'm just going to take a first (likely superficial look at this) so
comments may well be lightweight. I need to get my head around it
before offering more substantial stuff.
Jonathan
> ---
> drivers/cxl/cxl.h | 114 +++++++++++++++++++++++++++++++++++++++++++
> drivers/cxl/cxlmem.h | 113 ------------------------------------------
> 2 files changed, 114 insertions(+), 113 deletions(-)
>
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 1de4fec5c8f4..751e5860423a 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -11,6 +11,7 @@
> #include <linux/log2.h>
> #include <linux/node.h>
> #include <linux/io.h>
> +#include <cxl/mailbox.h>
>
> extern const struct nvdimm_security_ops *cxl_security_ops;
>
> @@ -696,6 +697,119 @@ struct cxl_ep {
> struct cxl_port *next;
> };
>
> +
One blank line probably enough.
> +/*
> + * enum cxl_devtype - delineate type-2 from a generic type-3 device
> + * @CXL_DEVTYPE_DEVMEM - Vendor specific CXL Type-2 device implementing HDM-D or
> + * HDM-DB, no requirement that this device implements a
> + * mailbox, or other memory-device-standard manageability
> + * flows.
> + * @CXL_DEVTYPE_CLASSMEM - Common class definition of a CXL Type-3 device with
> + * HDM-H and class-mandatory memory device registers
> + */
> +enum cxl_devtype {
> + CXL_DEVTYPE_DEVMEM,
> + CXL_DEVTYPE_CLASSMEM,
> +};
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 03/18] cxl/core: Add CXL.cache device struct
2025-08-12 21:29 ` [RFC PATCH 03/18] cxl/core: Add CXL.cache device struct Ben Cheatham
@ 2025-08-19 11:48 ` Jonathan Cameron
2025-08-22 18:00 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 11:48 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:06 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> Add a new CXL.cache device (struct cxl_cachedev) that is the cache
> analogue to struct cxl_memdev. This device will be created by
> endpoint vendor-specific drivers to enable and manage the cache
> capabilities of the endpoint.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
Whilst this creates the device it never adds which seems like an odd omission.
If that's intentional add some more text on why.
Jonathan
> diff --git a/drivers/cxl/core/cachedev.c b/drivers/cxl/core/cachedev.c
> new file mode 100644
> index 000000000000..aaf4a1b94178
> --- /dev/null
> +++ b/drivers/cxl/core/cachedev.c
> @@ -0,0 +1,91 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright (C) 2025 Advanced Micro Devices, Inc. */
> +#include <linux/device.h>
> +#include <linux/pci.h>
> +
> +#include "../cxlcache.h"
> +#include "private.h"
> +
> +static DEFINE_IDA(cxl_cachedev_ida);
> +
> +static void cxl_cachedev_release(struct device *dev)
> +{
> + struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
> +
> + ida_free(&cxl_cachedev_ida, cxlcd->id);
> + kfree(cxlcd);
> +}
> +
> +static void cxl_cachedev_unregister(void *dev)
> +{
> + struct cxl_cachedev *cxlcd = dev;
> +
It's not doing what we'd normally expect in an unregister
as no device_del() Which is fair because it was never added below.
> + cxlcd->cxlds = NULL;
> + put_device(&cxlcd->dev);
> +}
> +
> +struct cxl_cachedev *devm_cxl_cachedev_add_or_reset(struct device *host,
> + struct cxl_cachedev *cxlcd)
> +{
> + int rc;
Odd to not have a device_add() of any type in here.
Maybe introduced in later patches, but I think you need to drag that into
this one for it to make any real sense.
> +
> + rc = devm_add_action_or_reset(host, cxl_cachedev_unregister, cxlcd);
> + if (rc)
> + return ERR_PTR(rc);
I'd throw a blank line in here.
> + return cxlcd;
> +}
> +EXPORT_SYMBOL_NS_GPL(devm_cxl_cachedev_add_or_reset, "CXL");
> diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
> new file mode 100644
> index 000000000000..3ccbeba9fcd6
> --- /dev/null
> +++ b/drivers/cxl/cxlcache.h
> @@ -0,0 +1,23 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +#ifndef __CXL_CACHE_H__
> +#define __CXL_CACHE_H__
> +#include <linux/pci.h>
> +
> +#include "cxl.h"
> +
I'd add docs from the start.
> +struct cxl_cachedev {
> + struct device dev;
> + struct cxl_dev_state *cxlds;
> + struct cxl_port *endpoint;
> + const struct cxl_dev_ops *ops;
> + int id;
> + int depth;
> +};
> +
> +static inline struct cxl_cachedev *to_cxl_cachedev(struct device *dev)
> +{
> + return container_of(dev, struct cxl_cachedev, dev);
> +}
> +
> +bool is_cxl_cachedev(const struct device *dev);
> +#endif
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 06/18] cxl/port, mem: Make adding an endpoint device type agnostic
2025-08-12 21:29 ` [RFC PATCH 06/18] cxl/port, mem: Make adding an endpoint device type agnostic Ben Cheatham
@ 2025-08-19 11:53 ` Jonathan Cameron
2025-08-22 18:00 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 11:53 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:09 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> Change devm_cxl_enumerate_ports() and devm_cxl_add_endpoint() to take a
> generic struct device * instead of struct cxl_memdev in preparation of
> adding cache devices to the port hierarchy.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index cdac7bfa2289..c615e6c4ee60 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -104,9 +104,11 @@ static int cxl_mem_probe(struct device *dev)
> if (rc)
> return rc;
>
> - rc = devm_cxl_enumerate_ports(cxlmd);
> - if (rc)
> + rc = devm_cxl_enumerate_ports(&cxlmd->dev);
> + if (rc) {
Not immediately obvious to me why this needs to be set. Perhaps a comment?
> + cxlmd->endpoint = ERR_PTR(rc);
> return rc;
> + }
>
> struct cxl_port *parent_port __free(put_cxl_port) =
> cxl_dev_find_port(&cxlmd->dev, &dport);
> @@ -138,7 +140,7 @@ static int cxl_mem_probe(struct device *dev)
> return -ENXIO;
> }
>
> - rc = devm_cxl_add_endpoint(endpoint_parent, cxlmd, dport);
> + rc = devm_cxl_add_endpoint(endpoint_parent, &cxlmd->dev, dport);
> if (rc)
> return rc;
> }
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 07/18] cxl/port: Split endpoint port probe on device type
2025-08-12 21:29 ` [RFC PATCH 07/18] cxl/port: Split endpoint port probe on device type Ben Cheatham
@ 2025-08-19 11:57 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 11:57 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:10 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> The current endpoint port probe assumes the device has CXL.mem
> capabilities. With the addition of cache devices, this is no longer
> gauranteed. With this in mind, split endpoint port probe into separate
guaranteed
> functions for CXL.cache and CXL.mem endpoints.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 08/18] cxl/port: Update switch_port_probe() for CXL cache devices
2025-08-12 21:29 ` [RFC PATCH 08/18] cxl/port: Update switch_port_probe() for CXL cache devices Ben Cheatham
@ 2025-08-19 12:03 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 12:03 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:11 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> CXL switch port probe currently assumes there are only CXL.mem-capable
> devices below the switch with preconfigured HDM decoders. Now that
> CXL.cache devices can be added below a switch port, it's possible that
> they either have no HDM decoders (type 1) or the CXL.mem capabilities of
> the endpoint haven't been set up yet (type 2).
>
> The HDM decoders being disabled (if present) is no longer a gauranteed
> failure condition when only cache devices are present below the port.
> Knowing what kind of devices are under the switch port isn't possible
> until the endpoints are probed, so delay HDM setup and validation until
> endpoint port probe.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> ---
> drivers/cxl/core/hdm.c | 31 +++++++++++++++++++++++++++++++
> drivers/cxl/cxl.h | 1 +
> drivers/cxl/port.c | 23 +++++++++++++----------
> 3 files changed, 45 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
> index 865a71bce251..6e04f1f4c166 100644
> --- a/drivers/cxl/core/hdm.c
> +++ b/drivers/cxl/core/hdm.c
> @@ -205,6 +205,37 @@ struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port,
> }
> EXPORT_SYMBOL_NS_GPL(devm_cxl_setup_hdm, "CXL");
>
> +int cxl_validate_endpoint_hdm_setup(struct cxl_port *endpoint)
> +{
> + struct cxl_port *sp = parent_port_of(endpoint);
> + struct cxl_hdm *cxlhdm;
> + int rc;
> +
> + for (; sp && !is_cxl_root(sp); sp = parent_port_of(sp)) {
Might as well set sp as loop init rather than earlier.
> + cxlhdm = dev_get_drvdata(&sp->dev);
> + if (cxlhdm && cxlhdm->decoder_count > 0)
> + continue;
> +
> + if (!sp->reg_map.component_map.hdm_decoder.valid || !cxlhdm) {
> + dev_err(&sp->dev, "Missing HDM decoder capability\n");
> + return -ENXIO;
> + }
> +
> + if (sp->total_dports == 1) {
> + dev_dbg(&sp->dev, "Fallback to passthrough decoder\n");
> + rc = devm_cxl_add_passthrough_decoder(sp);
> + if (rc)
> + return rc;
> + }
> + }
> +
> + if (is_cxl_root(sp))
> + return 0;
Could move this test into the loop, alowing the loop condition to be simplified
to just sp;
> +
> + return -ENODEV;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_validate_endpoint_hdm_setup, "CXL");
> +
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 10/18] cxl/cache: Add cxl_cache driver
2025-08-12 21:29 ` [RFC PATCH 10/18] cxl/cache: Add cxl_cache driver Ben Cheatham
@ 2025-08-19 12:11 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 12:11 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:13 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> Add the cxl_cache driver which will manage struct cxl_cachdev devices.
> This driver will provide common management functions for some of a cache
> capable endpoint. This driver will also be responsible for validating
> the system's CXL.cache configuration.
>
> The driver expects the device's cache capabilities to be prefetched by
> the endpoint-specific driver. The required capabilities can be gotten by
> calling cxl_accel_get_cache_info().
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> +static struct cxl_driver cxl_cache_driver = {
> + .name = "cxl_cache",
> + .probe = cxl_cache_probe,
> + .drv = {
> + .probe_type = PROBE_FORCE_SYNCHRONOUS,
Why? I'd add a comment.
> + },
> + .id = CXL_DEVICE_ACCELERATOR,
> +};
> +
> +module_cxl_driver(cxl_cache_driver);
> +
> +MODULE_DESCRIPTION("CXL: Cache Management");
> +MODULE_LICENSE("GPL");
> +MODULE_IMPORT_NS("CXL");
> +MODULE_ALIAS_CXL(CXL_DEVICE_ACCELERATOR);
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming
2025-08-12 21:29 ` [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming Ben Cheatham
@ 2025-08-19 13:44 ` Alireza Sanaee
2025-08-20 8:55 ` Alireza Sanaee
2025-08-19 15:26 ` Jonathan Cameron
1 sibling, 1 reply; 55+ messages in thread
From: Alireza Sanaee @ 2025-08-19 13:44 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:18 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> The CXL Cache ID Decoder capability is an optional capability present
> on CXL downstream switch and CXL-enabled PCIe Root ports. This
> capability is required for having multiple CXL.cache enabled devices
> under the same port (CXL 3.2 8.2.4.29).
>
> Implement cache id decoder programming. Validate the configuration in
> the case BIOS already programmed the id(s).
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> ---
> drivers/cxl/cache.c | 4 ++
> drivers/cxl/core/port.c | 109
> ++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h |
> 14 ++++++ drivers/cxl/cxlcache.h | 3 ++
> 4 files changed, 130 insertions(+)
>
> diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
> index f123d596187d..fa3223165a18 100644
> --- a/drivers/cxl/cache.c
> +++ b/drivers/cxl/cache.c
> @@ -163,6 +163,10 @@ static int
> devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd) rc =
> map_cache_idd_cap(dport); if (rc)
> return rc;
> +
> + rc = cxl_dport_program_cache_idd(dport, cxlcd);
> + if (rc)
> + return rc;
> }
>
> return 0;
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index f491796f6e60..3606ad557bfb 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -1080,6 +1080,115 @@ int devm_cxl_port_program_cache_idrt(struct
> cxl_port *port, }
> EXPORT_SYMBOL_NS_GPL(devm_cxl_port_program_cache_idrt, "CXL");
>
> +static int cxl_dport_commit_idd(struct cxl_dport *dport)
> +{
> + unsigned long timeout, start, end;
> + u32 cap, ctrl, status;
> + int i;
> +
> + cap = readl(dport->regs.cidd + CXL_CACHE_IDD_CAP_OFFSET);
> + if (!(cap & CXL_CACHE_IDD_CAP_COMMIT))
> + return 0;
> +
> + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> + if (ctrl & CXL_CACHE_IDD_CTRL_COMMIT) {
> + ctrl &= ~CXL_CACHE_IDD_CTRL_COMMIT;
> + writel(ctrl, dport->regs.cidd +
> CXL_CACHE_IDD_CTRL_OFFSET);
> + }
> +
> + ctrl |= CXL_CACHE_IDD_CTRL_COMMIT;
> + writel(ctrl, dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> +
> + status = readl(dport->regs.cidd + CXL_CACHE_IDD_STAT_OFFSET);
> +
> + i = FIELD_GET(CXL_CACHE_IDD_STAT_TIMEOUT_SCALE_MASK, status);
> + timeout = 1 * HZ;
> + while (i-- > 3)
> + timeout *= 10;
> +
> + timeout *= FIELD_GET(CXL_CACHE_IDD_STAT_TIMEOUT_BASE_MASK,
> status);
> + start = jiffies;
> + do {
> + status = readl(dport->regs.cidd +
> CXL_CACHE_IDD_STAT_OFFSET);
> + if (status & CXL_CACHE_IDD_STAT_ERR_COMMIT)
> + return -EBUSY;
> +
> + if (status & CXL_CACHE_IDD_STAT_COMMITTED)
> + return 0;
> +
> + end = jiffies;
> + } while (time_before(end, start + timeout));
> +
> + return -ETIMEDOUT;
> +}
> +
> +static int cxl_cache_idd_alread_programmed(struct cxl_dport *dport,
> + struct cxl_cachedev
> *cxlcd) +{
> + struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
> + u32 ctrl, status, cache_id;
> +
> + status = readl(dport->regs.cidd + CXL_CACHE_IDD_STAT_OFFSET);
> + if (!(status & CXL_CACHE_IDD_STAT_COMMITTED))
> + return 0;
> +
> + /*
> + * The decoder is probably already committed by BIOS,
> validate
> + * the configuration
> + */
> + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> + /* TODO: Check for 68B flit mode */
> + if (dport == cxlcd->endpoint->parent_dport &&
> + !(ctrl & CXL_CACHE_IDD_CTRL_ASGN_ID))
> + return -EINVAL;
> +
> + if (dport != cxlcd->endpoint->parent_dport &&
> + !(ctrl & CXL_CACHE_IDD_CTRL_FWD_ID))
> + return -EINVAL;
> +
> + cache_id = FIELD_GET(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK, ctrl);
> + if (cache_id != cstate->cache_id)
> + return -EINVAL;
> +
> + if (!!(ctrl & CXL_CACHE_IDD_CTRL_TYPE2) != cstate->hdmd)
> + return -EINVAL;
> +
> + return 1;
> +}
> +
> +int cxl_dport_program_cache_idd(struct cxl_dport *dport,
> + struct cxl_cachedev *cxlcd)
> +{
> + u32 ctrl;
> + int rc;
> +
> + rc = cxl_cache_idd_alread_programmed(dport, cxlcd);
> + if (rc)
> + return rc < 0 ? rc : 0;
> +
> + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> +
> + if (cxlcd->cxlds->cstate.hdmd) {
> + ctrl |= CXL_CACHE_IDD_CTRL_TYPE2;
> + ctrl &= ~CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK;
> + ctrl |= FIELD_PREP(CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK,
> cxlcd->id);
> + }
> +
> + /* TODO: Check for 68B flit mode */
> + if (dport == cxlcd->endpoint->parent_dport)
> + ctrl |= CXL_CACHE_IDD_CTRL_ASGN_ID;
> + else
> + ctrl |= CXL_CACHE_IDD_CTRL_FWD_ID;
> +
> + ctrl &= ~CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK;
> + ctrl |= FIELD_PREP(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK,
> + cxlcd->cxlds->cstate.cache_id);
> +
> + writel(ctrl, dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> + return cxl_dport_commit_idd(dport);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_dport_program_cache_idd, "CXL");
> +
> DEFINE_SHOW_ATTRIBUTE(einj_cxl_available_error_type);
>
> static int cxl_einj_inject(void *data, u64 type)
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 04fde1a994d0..9c475d8c1573 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -178,6 +178,20 @@ static inline int ways_to_eiw(unsigned int ways,
> u8 *eiw) #define CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
>
> /* CXL 3.2 8.2.4.29 CXL Cache ID Decoder Capability Structure */
> +#define CXL_CACHE_IDD_CAP_OFFSET 0x0
> +#define CXL_CACHE_IDD_CAP_COMMIT BIT(0)
> +#define CXL_CACHE_IDD_CTRL_OFFSET 0x4
> +#define CXL_CACHE_IDD_CTRL_FWD_ID BIT(0)
> +#define CXL_CACHE_IDD_CTRL_ASGN_ID BIT(1)
> +#define CXL_CACHE_IDD_CTRL_TYPE2 BIT(2)
> +#define CXL_CACHE_IDD_CTRL_COMMIT BIT(3)
> +#define CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK GENMASK(11, 8)
> +#define CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK GENMASK(19, 16)
> +#define CXL_CACHE_IDD_STAT_OFFSET 0x8
> +#define CXL_CACHE_IDD_STAT_COMMITTED BIT(0)
> +#define CXL_CACHE_IDD_STAT_ERR_COMMIT BIT(1)
> +#define CXL_CACHE_IDD_STAT_TIMEOUT_SCALE_MASK GENMASK(11, 8)
> +#define CXL_CACHE_IDD_STAT_TIMEOUT_BASE_MASK GENMASK(15, 12)
> #define CXL_CACHE_IDD_CAPABILITY_LENGTH 0xC
Hi Ben,
This looks like some styling issues.
>
> /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
> diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
> index d28222123102..60563b391844 100644
> --- a/drivers/cxl/cxlcache.h
> +++ b/drivers/cxl/cxlcache.h
> @@ -32,4 +32,7 @@ int devm_cxl_snoop_filter_alloc(u32 gid, struct
> cxl_dev_state *cxlds); int devm_cxl_port_program_cache_idrt(struct
> cxl_port *port, struct cxl_dport *dport,
> struct cxl_cachedev *cxlcd);
> +int cxl_dport_program_cache_idd(struct cxl_dport *dport,
> + struct cxl_cachedev *cxlcd);
> +
> #endif
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 14/18] cxl/cache: Add Cache ID Decoder capability mapping
2025-08-12 21:29 ` [RFC PATCH 14/18] cxl/cache: Add Cache ID Decoder capability mapping Ben Cheatham
@ 2025-08-19 14:12 ` Alireza Sanaee
2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Alireza Sanaee @ 2025-08-19 14:12 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:17 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> The CXL Cache ID Decoder capability is an optional capability present
> on CXL downstream switch and CXL-enabled PCIe Root ports. This
> capability is required for having multiple CXL.cache enabled devices
> under the same port (CXL 3.2 8.2.4.29).
>
> Add mapping of the capability as part of allocating a CXL cache id for
> an endpoint. Implement mapping/validation of the capability in a later
> commit.
Maybe you keep ID or Cache or Decoder consistent everywhere in the
commit message.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> ---
> drivers/cxl/cache.c | 25 +++++++++++++++++++++++++
> drivers/cxl/core/regs.c | 8 ++++++++
> drivers/cxl/cxl.h | 6 ++++++
> 3 files changed, 39 insertions(+)
>
> diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
> index 24559b9ba8e8..f123d596187d 100644
> --- a/drivers/cxl/cache.c
> +++ b/drivers/cxl/cache.c
> @@ -69,6 +69,27 @@ static int map_cache_idrt_cap(struct cxl_port
> *port) BIT(CXL_CM_CAP_CAP_ID_CIDRT));
> }
>
> +static int map_cache_idd_cap(struct cxl_dport *dport)
> +{
> + int agents;
> +
> + if (dport->regs.cidd)
> + return 0;
> +
> + /*
> + * A missing Cache ID Decoder capability is only an issue
> + * if there are multiple cache agents in the VCS
> + */
> + if (!dport->reg_map.component_map.cidd.valid) {
> + agents = atomic_read(&dport->port->cache_agents);
> + return agents > 1 ? -ENXIO : 0;
> + }
> +
> + return cxl_map_component_regs(&dport->reg_map,
> + &dport->regs.component,
> + BIT(CXL_CM_CAP_CAP_ID_CIDD));
Less relevant to this patch. It looks like cxl_map_component_reg
receives redundant parameters, both regs.component and reg_map comes
from the same structure and other occurrences in the kernel also grab
components and reg_map from port, so not sure why two separate
parameters are there.
> +}
> +
> static void free_cache_id(void *data)
> {
> struct cxl_cache_state *cstate = data;
> @@ -138,6 +159,10 @@ static int
> devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd) rc =
> devm_cxl_port_program_cache_idrt(port, dport, cxlcd); if (rc)
> return rc;
> +
> + rc = map_cache_idd_cap(dport);
> + if (rc)
> + return rc;
> }
>
> return 0;
> diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
> index 0924127bd8fd..1362f4156ee6 100644
> --- a/drivers/cxl/core/regs.c
> +++ b/drivers/cxl/core/regs.c
> @@ -110,6 +110,13 @@ void cxl_probe_component_regs(struct device
> *dev, void __iomem *base, rmap = &map->cidrt;
> break;
> }
> + case CXL_CM_CAP_CAP_ID_CIDD:
> + dev_dbg(dev,
> + "found Cache ID decoder capability
> (0x%x)\n",
> + offset);
> + length = CXL_CACHE_IDD_CAPABILITY_LENGTH;
> + rmap = &map->cidd;
> + break;
> default:
> dev_dbg(dev, "Unknown CM cap ID: %d
> (0x%x)\n", cap_id, offset);
> @@ -231,6 +238,7 @@ int cxl_map_component_regs(const struct
> cxl_register_map *map, { &map->component_map.ras, ®s->ras },
> { &map->component_map.snoop, ®s->snoop },
> { &map->component_map.cidrt, ®s->cidrt },
> + { &map->component_map.cidd, ®s->cidd },
> };
> int i;
>
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 1a2918aaee62..04fde1a994d0 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -42,6 +42,7 @@ extern const struct nvdimm_security_ops
> *cxl_security_ops; #define CXL_CM_CAP_CAP_ID_HDM 0x5
> #define CXL_CM_CAP_CAP_ID_SNOOP 0x9
> #define CXL_CM_CAP_CAP_ID_CIDRT 0xD
> +#define CXL_CM_CAP_CAP_ID_CIDD 0xE
> #define CXL_CM_CAP_CAP_HDM_VERSION 1
>
> /* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability
> Structure */ @@ -176,6 +177,9 @@ static inline int
> ways_to_eiw(unsigned int ways, u8 *eiw) #define
> CXL_CACHE_IDRT_TARGETN_VALID BIT(0) #define
> CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
> +/* CXL 3.2 8.2.4.29 CXL Cache ID Decoder Capability Structure */
> +#define CXL_CACHE_IDD_CAPABILITY_LENGTH 0xC
> +
> /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
> #define CXLDEV_CAP_ARRAY_OFFSET 0x0
> #define CXLDEV_CAP_ARRAY_CAP_ID 0
> @@ -241,6 +245,7 @@ struct cxl_regs {
> void __iomem *ras;
> void __iomem *snoop;
> void __iomem *cidrt;
> + void __iomem *cidd;
> );
> /*
> * Common set of CXL Device register block base pointers
> @@ -285,6 +290,7 @@ struct cxl_component_reg_map {
> struct cxl_reg_map ras;
> struct cxl_reg_map snoop;
> struct cxl_reg_map cidrt;
> + struct cxl_reg_map cidd;
> };
>
> struct cxl_device_reg_map {
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 11/18] cxl/core: Add CXL snoop filter setup and checking
2025-08-12 21:29 ` [RFC PATCH 11/18] cxl/core: Add CXL snoop filter setup and checking Ben Cheatham
@ 2025-08-19 14:18 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 14:18 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:14 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> Add functions to set up and validate the snoop filter configuration of a
> CXL-capable PCIe Root Port (struct cxl_dport). Allocate space in the
> snoop filter as part of cxl_cachedev probe.
>
Any thoughts on how to handle allocation policy, when combined with Cache_SF_Coverage
hints or indeed just overcommiting the snoop filter and relying on snoops
to clear space.
The spec is rather ambiguous on when these can be used as opposed to
just rejecting devices if we are out of space.
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
A few comments inline.
Thanks,
Jonathan
>
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index 5144ddac7145..26ea22a2638c 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -35,6 +35,9 @@ static DEFINE_IDA(cxl_port_ida);
> static DEFINE_XARRAY(cxl_root_buses);
> static DEFINE_XARRAY(cxl_root_ports);
>
> +DECLARE_RWSEM(cxl_snoop_rwsem);
static?
> +static DEFINE_XARRAY(cxl_snoop_filters);
> +
> /*
> * The terminal device in PCI is NULL and @platform_bus
> * for platform devices (for cxl_test)
> @@ -809,6 +812,139 @@ static int cxl_dport_setup_regs(struct device *host, struct cxl_dport *dport,
> return rc;
> }
>
> +struct cxl_snoop_filter {
> + u64 size;
> + struct xarray dports;
> + struct xarray devs;
> + struct mutex lock; /* Used for filter allocations */
> +};
> +
> +static struct cxl_snoop_filter *cxl_create_snoop_filter(struct cxl_dport *dport,
> + u32 gid, u32 size)
> +{
> + struct cxl_snoop_filter *sf;
> + int rc;
> +
> + sf = xa_load(&cxl_snoop_filters, gid);
> + if (sf)
> + return sf;
> +
> + guard(rwsem_write)(&cxl_snoop_rwsem);
> + sf = kzalloc(sizeof(*sf), GFP_KERNEL);
> + if (!sf)
> + return ERR_PTR(-ENOMEM);
> +
> + sf->size = size;
> + xa_init(&sf->dports);
> + xa_init(&sf->devs);
> + mutex_init(&sf->lock);
> +
> + rc = xa_insert(&sf->dports, (unsigned long)dport->dport_dev, dport,
So the aim here is include all dports with this GID in sf->dports?
If so why isn't one added if the xa_load() above finds sf?
> + GFP_KERNEL);
> + if (rc)
> + goto err;
> +
> + rc = xa_insert(&cxl_snoop_filters, gid, sf, GFP_KERNEL);
> + if (rc)
> + goto err;
> +
> + return 0;
> +
> +err:
> + xa_destroy(&sf->dports);
> + xa_destroy(&sf->devs);
> + mutex_destroy(&sf->lock);
> + kfree(sf);
> + return ERR_PTR(rc);
> +}
> +
> +static void cxl_destroy_snoop_filters(void)
> +{
> + struct cxl_snoop_filter *sf;
> + unsigned long gid;
> +
> + guard(rwsem_write)(&cxl_snoop_rwsem);
> + xa_for_each(&cxl_snoop_filters, gid, sf) {
> + xa_destroy(&sf->dports);
> + xa_destroy(&sf->devs);
> + mutex_destroy(&sf->lock);
Not freeing sf?
> + }
> +
> + xa_destroy(&cxl_snoop_filters);
> +}
> +
> +static void cxl_snoop_filter_free(void *data)
> +{
> + struct cxl_cache_state *cstate = data;
> + struct cxl_snoop_filter *sf;
> +
> + sf = xa_load(&cxl_snoop_filters, cstate->snoop_id);
> + if (!sf)
I'd put a debug print here at least and possibly scream as fairly
sure this means we are trying to free something we never had.
> + return;
> +
> + guard(mutex)(&sf->lock);
> + sf->size += cstate->size;
> +}
> +
> +int devm_cxl_snoop_filter_alloc(u32 gid, struct cxl_dev_state *cxlds)
I'd rename this as it is allocating part of the capacity rather
than allocating the snoop filter representation.
_alloc_for_device maybe?
> +{
> + struct cxl_cache_state *cstate = &cxlds->cstate;
> + struct cxl_snoop_filter *sf;
> +
> + sf = xa_load(&cxl_snoop_filters, gid);
> + if (!sf)
> + return -ENODEV;
> +
> + guard(mutex)(&sf->lock);
> + if (cstate->size > sf->size)
> + return -EBUSY;
> +
> + sf->size -= cstate->size;
> + cstate->snoop_id = gid;
> + return devm_add_action_or_reset(cxlds->dev, cxl_snoop_filter_free,
> + cstate);
> +}
> +EXPORT_SYMBOL_NS_GPL(devm_cxl_snoop_filter_alloc, "CXL");
> +
> +int cxl_dport_setup_snoop_filter(struct cxl_dport *dport)
> +{
> + struct cxl_port *port = dport->port, *parent_port;
> + struct cxl_snoop_filter *sf;
> + u32 snoop, gid, size;
> + int rc;
> +
> + /* Only supported by CXL-enabled PCIe root ports */
I'd go with the CXL spec term which is simply "CXL root port".
> + parent_port = parent_port_of(port);
> + if (!parent_port || !is_cxl_root(parent_port))
> + return 0;
> +
> + if (!dport->reg_map.component_map.snoop.valid) {
> + dev_dbg(dport->dport_dev, "missing snoop filter capability\n");
> + return -ENXIO;
> + }
> +
> + rc = cxl_map_component_regs(&dport->reg_map, &dport->regs.component,
> + BIT(CXL_CM_CAP_CAP_ID_SNOOP));
> + if (rc)
> + return rc;
> +
> + snoop = readl(dport->regs.snoop + CXL_SNOOP_GROUP_ID_OFFSET);
> + gid = FIELD_GET(CXL_SNOOP_GROUP_ID_MASK, snoop);
> +
> + snoop = readl(dport->regs.snoop + CXL_SNOOP_FILTER_SIZE_OFFSET);
> + size = FIELD_GET(CXL_SNOOP_FILTER_SIZE_MASK, snoop);
I'm not sure I'd bother with masking when it is the whole register with
no reserved bits.
> + if (!size)
> + return -ENXIO;
> +
> + sf = cxl_create_snoop_filter(dport, gid, size);
> + if (IS_ERR(sf))
> + return PTR_ERR(sf);
> +
> + dport->snoop_id = gid;
> + return 0;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_dport_setup_snoop_filter, "CXL");
Export seems unnecessary given it's used later in this file.
Maybe that changes in later patches.
> +
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 1771c42a1e3b..2881e0826829 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -40,6 +40,7 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
>
> #define CXL_CM_CAP_CAP_ID_RAS 0x2
> #define CXL_CM_CAP_CAP_ID_HDM 0x5
> +#define CXL_CM_CAP_CAP_ID_SNOOP 0x9
Snoop filter capability is 0x8 I think.
0x9 is the timeout and isolation capability.
> #define CXL_CM_CAP_CAP_HDM_VERSION 1
>
> struct cxl_device_reg_map {
> @@ -658,6 +668,8 @@ struct cxl_rcrb_info {
> u16 aer_cap;
> };
>
> +#define CXL_SNOOP_ID_NO_ID (-1)
> +
> /**
> * struct cxl_dport - CXL downstream port
> * @dport_dev: PCI bridge or firmware device representing the downstream link
> @@ -682,6 +694,7 @@ struct cxl_dport {
> struct access_coordinate coord[ACCESS_COORDINATE_MAX];
> long link_latency;
> int gpf_dvsec;
> + int snoop_id;
> };
>
> /**
> @@ -756,6 +769,7 @@ struct cxl_dev_ops {
> struct cxl_cache_state {
> u64 size;
> u32 unit;
> + u16 snoop_id;
> };
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 13/18] cxl/cache: Implement Cache ID Route Table programming
2025-08-12 21:29 ` [RFC PATCH 13/18] cxl/cache: Implement Cache ID Route Table programming Ben Cheatham
@ 2025-08-19 15:07 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 15:07 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:16 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> The CXL Cache ID Route Table is an optional capability for CXL-enabled
> upstream switch ports and CXL host bridges. This capability is required
> for having multiple CXL.cache enabled devices under a single port
> (CXL 3.2 8.2.4.28).
>
> Implement route table programming. In the case BIOS has already
> programmed the route table(s), check the configuration for validity.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
Whilst it's a bit big, I'd smash this patch with the previous. They don't
really separate as we end up with things half done but not returning errors.
A lot of my initial comments on that patch were along the lines of
'you didn't check this' when turns out you did in this patch.
> ---
> drivers/cxl/cache.c | 4 ++
> drivers/cxl/core/port.c | 135 ++++++++++++++++++++++++++++++++++++++++
> drivers/cxl/cxl.h | 10 +++
> drivers/cxl/cxlcache.h | 4 ++
> 4 files changed, 153 insertions(+)
>
> diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
> index fd2ae1fa44bc..24559b9ba8e8 100644
> --- a/drivers/cxl/cache.c
> +++ b/drivers/cxl/cache.c
> @@ -134,6 +134,10 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
> if (rc)
> return rc;
> }
> +
> + rc = devm_cxl_port_program_cache_idrt(port, dport, cxlcd);
> + if (rc)
> + return rc;
> }
>
> return 0;
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index 26ea22a2638c..f491796f6e60 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -945,6 +945,141 @@ int cxl_dport_setup_snoop_filter(struct cxl_dport *dport)
> }
> EXPORT_SYMBOL_NS_GPL(cxl_dport_setup_snoop_filter, "CXL");
>
> +static int cxl_port_commit_idrt(struct cxl_port *port)
> +{
> + unsigned long timeout, start, end;
> + u32 cap, ctrl, status;
> + int i;
> +
> + cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
> + if (!(cap & CXL_CACHE_IDRT_CAP_COMMIT))
> + return 0;
> +
> + ctrl = readl(port->regs.cidrt + CXL_CACHE_IDRT_CTRL_OFFSET);
> + if (ctrl & CXL_CACHE_IDRT_CTRL_COMMIT) {
> + ctrl &= ~CXL_CACHE_IDRT_CTRL_COMMIT;
> + writel(ctrl, port->regs.cidrt + CXL_CACHE_IDRT_CTRL_OFFSET);
> + }
> +
> + ctrl |= CXL_CACHE_IDRT_CTRL_COMMIT;
> + writel(ctrl, port->regs.cidrt + CXL_CACHE_IDRT_CTRL_OFFSET);
> +
> + status = readl(port->regs.cidrt + CXL_CACHE_IDRT_STAT_OFFSET);
> +
> + i = FIELD_GET(CXL_CACHE_IDRT_STAT_TIMEOUT_SCALE_MASK, status);
> + timeout = 1 * HZ;
> + while (i-- > 3)
> + timeout *= 10;
I've no idea how that relates to the spec. Perhaps some comments.
> +
> + timeout *= FIELD_GET(CXL_CACHE_IDRT_STAT_TIMEOUT_BASE_MASK, status);
> + start = jiffies;
> + do {
> + status = readl(port->regs.cidrt + CXL_CACHE_IDRT_STAT_OFFSET);
> + if (status & CXL_CACHE_IDRT_STAT_ERR_COMMIT)
> + return -EBUSY;
> +
> + if (status & CXL_CACHE_IDRT_STAT_COMMITTED)
> + return 0;
> +
> + end = jiffies;
> + } while (time_before(end, start + timeout));
> +
> + return -ETIMEDOUT;
> +}
> +
> +struct cxl_cache_idrt_entry {
> + struct cxl_port *port;
> + int port_num;
> + int id;
> +};
> +
> +static int cxl_port_enable_idrt_entry(struct cxl_cache_idrt_entry *entry)
> +{
> + struct cxl_port *port = entry->port;
> + int id = entry->id, port_num;
> + u32 entry_n;
> +
> + entry_n = readl(port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
It's 16bit so readw?
> + if (entry_n & CXL_CACHE_IDRT_TARGETN_VALID) {
> + port_num = FIELD_GET(CXL_CACHE_IDRT_TARGETN_PORT_MASK, entry_n);
> + if (port_num != entry->port_num) {
> + dev_err(&port->dev,
> + "Cache ID Route Table entry%d: Invalid port id %d, expected %d\n",
> + id, port_num, entry->port_num);
> + return -EINVAL;
> + }
> +
> + return 0;
> + }
> +
> + entry_n &= ~CXL_CACHE_IDRT_TARGETN_PORT_MASK;
Maybe just start from 0. There aren't any other fields as long as we stick
to not reading and writing neighboring entry.
> + entry_n |= FIELD_PREP(CXL_CACHE_IDRT_TARGETN_PORT_MASK, entry->port_num);
> + entry_n |= CXL_CACHE_IDRT_TARGETN_VALID;
> + writel(entry_n, port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
writew()
> +
> + return cxl_port_commit_idrt(port);
> +}
> +
> +static void free_cache_idrt_entry(void *data)
> +{
> + struct cxl_cache_idrt_entry *entry = data;
> + struct cxl_port *port = entry->port;
> + int id = entry->id;
> + u32 cap, entry_n;
> +
> + cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
> + if (FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK, cap) < id)
How could this happen?
> + return;
> +
> + entry_n = readl(port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
> + entry_n &= ~CXL_CACHE_IDRT_TARGETN_VALID;
> + writel(entry_n, port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
> +
> + cxl_port_commit_idrt(port);
> +
> + kfree(entry);
> +}
> +
> +int devm_cxl_port_program_cache_idrt(struct cxl_port *port,
> + struct cxl_dport *dport,
> + struct cxl_cachedev *cxlcd)
> +{
> + struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
> + int id = cstate->cache_id, rc;
I'm always moaning about these but to me combining assignment and
non assignment declarations just ends up hiding stuff.
I'd always burn a line to keep them separate.
> + u32 cap, max_hdmd;
> +
> + cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
> + if (FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK, cap) < id)
Check if in 68B flit mode as if we are this number may need capping
to a much lower level (field description talks about this.
> + return -EINVAL;
> +
> + if (is_cxl_root(parent_port_of(port))) {
> + max_hdmd = FIELD_GET(CXL_CACHE_IDRT_CAP_T2_MAX_MASK, cap);
> + if (cxlcd->cxlds->type == CXL_DEVTYPE_DEVMEM && cstate->hdmd)
> + port->nr_hdmd++;
> +
> + if (port->nr_hdmd > max_hdmd) {
> + port->nr_hdmd--;
> + return -EINVAL;
> + }
> + }
> +
> + struct cxl_cache_idrt_entry *entry __free(kfree) =
> + kzalloc(sizeof(*entry), GFP_KERNEL);
> + if (!entry)
> + return -ENXIO;
> +
> + entry->port_num = dport->port_id;
> + entry->port = port;
> + entry->id = id;
> +
> + rc = cxl_port_enable_idrt_entry(entry);
> + if (rc)
> + return rc;
> +
> + return devm_add_action(&cxlcd->dev, free_cache_idrt_entry, entry);
At this point entry is freed. So by the time that action gets called
it's a use after free. Maybe also need the _or_reset() variant
in case this call itself fails.
> +}
> +EXPORT_SYMBOL_NS_GPL(devm_cxl_port_program_cache_idrt, "CXL");
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index c147846855e2..1a2918aaee62 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -163,8 +163,15 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
> /* CXL 3.2 8.2.4.28 CXL Cache ID Route Table Capability Structure */
> #define CXL_CACHE_IDRT_CAP_OFFSET 0x0
> #define CXL_CACHE_IDRT_CAP_CNT_MASK GENMASK(4, 0)
> +#define CXL_CACHE_IDRT_CAP_T2_MAX_MASK GENMASK(11, 8)
> +#define CXL_CACHE_IDRT_CAP_COMMIT BIT(16)
I'd get explicit in that name. It is an odd bit of design to have
the option for device to not require it so making it clear that
the absense of this bit doesn't mean it never makes the entrees
active / committed. Maybe REQUIRES_COMMIT
> +#define CXL_CACHE_IDRT_CTRL_OFFSET 0x4
> +#define CXL_CACHE_IDRT_CTRL_COMMIT BIT(0)
> #define CXL_CACHE_IDRT_STAT_OFFSET 0x8
> #define CXL_CACHE_IDRT_STAT_COMMITTED BIT(0)
> +#define CXL_CACHE_IDRT_STAT_ERR_COMMIT BIT(0)
Nope. They aren't both BIT(0)
> +#define CXL_CACHE_IDRT_STAT_TIMEOUT_SCALE_MASK GENMASK(11, 8)
> +#define CXL_CACHE_IDRT_STAT_TIMEOUT_BASE_MASK GENMASK(15, 12)
> #define CXL_CACHE_IDRT_TARGETN_OFFSET(n) (0x10 + (2 * (n)))
> #define CXL_CACHE_IDRT_TARGETN_VALID BIT(0)
> #define CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
> @@ -618,6 +625,7 @@ struct cxl_dax_region {
> * @cdat: Cached CDAT data
> * @cdat_available: Should a CDAT attribute be available in sysfs
> * @pci_latency: Upstream latency in picoseconds
> + * @nr_hdmd: Number of HDM-D devices below port
> */
> struct cxl_port {
> struct device dev;
> @@ -643,6 +651,7 @@ struct cxl_port {
> } cdat;
> bool cdat_available;
> long pci_latency;
> + int nr_hdmd;
> };
>
> /**
> @@ -786,6 +795,7 @@ struct cxl_cache_state {
> u32 unit;
> u16 snoop_id;
> int cache_id;
> + bool hdmd;
> };
>
> /**
> diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
> index c4862f7e6edc..d28222123102 100644
> --- a/drivers/cxl/cxlcache.h
> +++ b/drivers/cxl/cxlcache.h
> @@ -28,4 +28,8 @@ struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
>
> int cxl_dport_setup_snoop_filter(struct cxl_dport *dport);
> int devm_cxl_snoop_filter_alloc(u32 gid, struct cxl_dev_state *cxlds);
> +
> +int devm_cxl_port_program_cache_idrt(struct cxl_port *port,
> + struct cxl_dport *dport,
> + struct cxl_cachedev *cxlcd);
> #endif
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 12/18] cxl/cache: Add CXL Cache ID Route Table mapping
2025-08-12 21:29 ` [RFC PATCH 12/18] cxl/cache: Add CXL Cache ID Route Table mapping Ben Cheatham
@ 2025-08-19 15:09 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 15:09 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:15 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> The CXL Cache ID Route Table is an optional capability implemented by
> CXL host bridges and CXL upstream switch ports. This capability is
> required for having multiple CXL.cache devices below a port (CXL 3.2
> 8.2.4.28).
>
> Add mapping and allocation of cache ids to be used by the cache id
> route table and decoder capabilities. It's possible the id has already
> been programmed by BIOS, in that case use the preprogrammed value.
> Programming/validation of the capabilities by the OS will come in later
> commits.
Fine to leave this for later, but I'd not set the cached value to
something we haven't written. Just leave it unset or error out until the
support is there.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> ---
> drivers/cxl/cache.c | 87 ++++++++++++++++++++++++++++++++++++++++-
> drivers/cxl/core/regs.c | 13 ++++++
> drivers/cxl/cxl.h | 16 ++++++++
> 3 files changed, 115 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
> index 7ef36e4bbce8..fd2ae1fa44bc 100644
> --- a/drivers/cxl/cache.c
> +++ b/drivers/cxl/cache.c
> @@ -55,6 +55,90 @@ struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
> }
> EXPORT_SYMBOL_NS_GPL(devm_cxl_add_cachedev, "CXL");
>
> +static int find_cache_id(struct cxl_cachedev *cxlcd)
> +{
> + struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
> + u32 cap, count, entry, portn;
> + struct cxl_dport *dport;
> + struct cxl_port *port;
> + int id;
> +
> + if (cstate->cache_id != CXL_CACHE_ID_NO_ID)
> + return cstate->cache_id;
> +
> + dport = cxlcd->endpoint->parent_dport;
> + if (!dport)
> + return -ENODEV;
> + port = dport->port;
> +
> + cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
> + count = FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK, cap);
Need to do the clamping dance for when operating in 68B flit mode?
Described in the register field definition in 82.2.4.28.1
> +
> + for (id = 0; id < count; id++) {
I'd pull portn, entry and anything else you can into this scope
> + entry = readl(port->regs.cidrt +
> + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
> + portn = FIELD_GET(CXL_CACHE_IDRT_TARGETN_PORT_MASK, entry);
> + if (portn != dport->port_id &&
> + !(entry & CXL_CACHE_IDRT_TARGETN_VALID))
> + continue;
> +
> + cstate->cache_id = id;
> + return 0;
> + }
> +
> + id = ida_alloc(&cache_id_ida, GFP_KERNEL);
> + if (id < 0)
> + return id;
ida_alloc_max(ida, count - 1, GFP_KERNEL)
probably to avoid allocating an ID off the top.
If some IDs were programmed by the BIOS, but not one for this
device (maybe it turned up late) do you reserve the relevant
values in the ida?
> +
> + cstate->cache_id = id;
Having this half state at the end of this patch makes things complex
to reason about. If you really want to split it then return an error in
paths you haven't handled yet.
> + return devm_add_action_or_reset(&cxlcd->dev, free_cache_id, cstate);
> +}
> +
> +static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
> +{
> + struct cxl_port *endpoint = cxlcd->endpoint, *port;
> + struct cxl_dport *dport;
> + int rc;
> +
> + for (dport = endpoint->parent_dport, port = parent_port_of(endpoint);
> + dport && port && !is_cxl_root(port);
> + dport = port->parent_dport, port = parent_port_of(port)) {
> + guard(device)(&port->dev);
> + rc = map_cache_idrt_cap(port);
> + if (rc)
> + return rc;
> +
> + if (port == parent_port_of(endpoint)) {
> + rc = find_cache_id(cxlcd);
> + if (rc)
> + return rc;
> + }
> + }
> +
> + return 0;
> +}
>
> static struct cxl_driver cxl_cache_driver = {
> diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
> index 0c8962fcad9b..0924127bd8fd 100644
> --- a/drivers/cxl/core/regs.c
> +++ b/drivers/cxl/core/regs.c
> @@ -98,6 +98,18 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
> length = CXL_SNOOP_FILTER_CAPABILITY_LENGTH;
> rmap = &map->snoop;
> break;
> + case CXL_CM_CAP_CAP_ID_CIDRT: {
> + int entry_cnt;
> +
> + dev_dbg(dev,
> + "found Cache ID route table capability (0x%x)\n",
> + offset);
> + entry_cnt = FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK,
> + hdr);
> + length = 2 * entry_cnt + 0x010;
I'd drop the leading 0 so 0x10 that will bring it in line with the macro below.
> + rmap = &map->cidrt;
> + break;
> + }
> default:
> dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id,
> offset);
> @@ -218,6 +230,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
> { &map->component_map.hdm_decoder, ®s->hdm_decoder },
> { &map->component_map.ras, ®s->ras },
> { &map->component_map.snoop, ®s->snoop },
> + { &map->component_map.cidrt, ®s->cidrt },
> };
> int i;
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming
2025-08-12 21:29 ` [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming Ben Cheatham
2025-08-19 13:44 ` Alireza Sanaee
@ 2025-08-19 15:26 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
1 sibling, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 15:26 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:18 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> The CXL Cache ID Decoder capability is an optional capability present on
> CXL downstream switch and CXL-enabled PCIe Root ports. This capability
> is required for having multiple CXL.cache enabled devices under the same
> port (CXL 3.2 8.2.4.29).
>
> Implement cache id decoder programming. Validate the configuration in the
> case BIOS already programmed the id(s).
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
Comments from earlier on how the timeout is calculate apply here as well.
A few other things inline.
Thanks,
Jonathan
> ---
> drivers/cxl/cache.c | 4 ++
> drivers/cxl/core/port.c | 109 ++++++++++++++++++++++++++++++++++++++++
> drivers/cxl/cxl.h | 14 ++++++
> drivers/cxl/cxlcache.h | 3 ++
> 4 files changed, 130 insertions(+)
>
> diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
> index f123d596187d..fa3223165a18 100644
> --- a/drivers/cxl/cache.c
> +++ b/drivers/cxl/cache.c
> @@ -163,6 +163,10 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
> rc = map_cache_idd_cap(dport);
> if (rc)
> return rc;
> +
> + rc = cxl_dport_program_cache_idd(dport, cxlcd);
> + if (rc)
> + return rc;
> }
>
> return 0;
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index f491796f6e60..3606ad557bfb 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> +static int cxl_cache_idd_alread_programmed(struct cxl_dport *dport,
already?
> + struct cxl_cachedev *cxlcd)
> +{
> + struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
> + u32 ctrl, status, cache_id;
> +
> + status = readl(dport->regs.cidd + CXL_CACHE_IDD_STAT_OFFSET);
> + if (!(status & CXL_CACHE_IDD_STAT_COMMITTED))
> + return 0;
> +
> + /*
> + * The decoder is probably already committed by BIOS, validate
> + * the configuration
> + */
> + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> + /* TODO: Check for 68B flit mode */
> + if (dport == cxlcd->endpoint->parent_dport &&
> + !(ctrl & CXL_CACHE_IDD_CTRL_ASGN_ID))
> + return -EINVAL;
> +
> + if (dport != cxlcd->endpoint->parent_dport &&
> + !(ctrl & CXL_CACHE_IDD_CTRL_FWD_ID))
> + return -EINVAL;
> +
> + cache_id = FIELD_GET(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK, ctrl);
> + if (cache_id != cstate->cache_id)
> + return -EINVAL;
> +
> + if (!!(ctrl & CXL_CACHE_IDD_CTRL_TYPE2) != cstate->hdmd)
> + return -EINVAL;
> +
> + return 1;
> +}
> +
> +int cxl_dport_program_cache_idd(struct cxl_dport *dport,
> + struct cxl_cachedev *cxlcd)
> +{
> + u32 ctrl;
> + int rc;
> +
> + rc = cxl_cache_idd_alread_programmed(dport, cxlcd);
> + if (rc)
> + return rc < 0 ? rc : 0;
> +
> + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> +
> + if (cxlcd->cxlds->cstate.hdmd) {
> + ctrl |= CXL_CACHE_IDD_CTRL_TYPE2;
> + ctrl &= ~CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK;
> + ctrl |= FIELD_PREP(CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK, cxlcd->id);
> + }
> +
> + /* TODO: Check for 68B flit mode */
> + if (dport == cxlcd->endpoint->parent_dport)
> + ctrl |= CXL_CACHE_IDD_CTRL_ASGN_ID;
> + else
> + ctrl |= CXL_CACHE_IDD_CTRL_FWD_ID;
> +
> + ctrl &= ~CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK;
> + ctrl |= FIELD_PREP(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK,
> + cxlcd->cxlds->cstate.cache_id);
> +
> + writel(ctrl, dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> + return cxl_dport_commit_idd(dport);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_dport_program_cache_idd, "CXL");
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 04fde1a994d0..9c475d8c1573 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -178,6 +178,20 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
> #define CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
>
> /* CXL 3.2 8.2.4.29 CXL Cache ID Decoder Capability Structure */
> +#define CXL_CACHE_IDD_CAP_OFFSET 0x0
> +#define CXL_CACHE_IDD_CAP_COMMIT BIT(0)
Similar to before I'd name this something to indicate it's about
it being explicit / required by this device for things
to work.
> +#define CXL_CACHE_IDD_CTRL_OFFSET 0x4
> +#define CXL_CACHE_IDD_CTRL_FWD_ID BIT(0)
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 16/18] cxl/cache: Add cache device counting for CXL ports
2025-08-12 21:29 ` [RFC PATCH 16/18] cxl/cache: Add cache device counting for CXL ports Ben Cheatham
@ 2025-08-19 15:30 ` Jonathan Cameron
2025-08-22 18:02 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 15:30 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:19 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> Having more than one CXL.cache enabled device under a port requires CXL
> cache id capabilities (CXL 3.2 8.2.4.28/29).
>
> Add tracking of how many cache devices are under a port. If the required
> capabilities are absent and more than one device is enabled under the
> port, fail allocation of a cache id.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
...
>
> +static void cxl_deactivate_cache(void *data)
> +{
> + struct cxl_cachedev *cxlcd = data;
> + struct cxl_port *port = cxlcd->endpoint;
> +
> + for (port = cxlcd->endpoint; port && !is_cxl_root(port);
> + port = parent_port_of(port)) {
> + scoped_guard(device, &port->dev) {
I think you need to check this succeeded if the aim is to ensure
that port didn't go away before this decrement.
It might have gone away just before we tried to take the guard.
> + port->cache_devs--;
> + }
> + }
> +}
> +
> static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
> {
> struct cxl_port *endpoint = cxlcd->endpoint, *port;
> @@ -167,9 +176,17 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
> rc = cxl_dport_program_cache_idd(dport, cxlcd);
> if (rc)
> return rc;
> +
> + port->cache_devs++;
> }
>
> - return 0;
> + if (!is_cxl_root(port)) {
> + cxl_deactivate_cache(cxlcd);
> + return -ENODEV;
> + }
> +
> + return devm_add_action_or_reset(&cxlcd->dev, cxl_deactivate_cache,
> + cxlcd);
> }
>
> static int find_snoop_gid(struct cxl_cachedev *cxlcd, u32 *gid)
> @@ -239,7 +256,6 @@ static int cxl_cache_probe(struct device *dev)
> if (rc)
> return rc;
>
> -
No comment. :)
> return devm_cxl_cachedev_allocate_cache_id(cxlcd);
> }
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 17/18] cxl/core: Add cache device attributes
2025-08-12 21:29 ` [RFC PATCH 17/18] cxl/core: Add cache device attributes Ben Cheatham
@ 2025-08-19 15:38 ` Jonathan Cameron
2025-08-22 18:02 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 15:38 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:20 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> Add sysfs attributes for getting the serial number, numa node, CXL cache
> unit, and CXL cache size to struct cxl_cachedev.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
A few minor comments inline.
> ---
> drivers/cxl/core/cachedev.c | 102 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 102 insertions(+)
>
> diff --git a/drivers/cxl/core/cachedev.c b/drivers/cxl/core/cachedev.c
> index aaf4a1b94178..1cdd94520ceb 100644
> --- a/drivers/cxl/core/cachedev.c
> +++ b/drivers/cxl/core/cachedev.c
> @@ -1,5 +1,6 @@
> // SPDX-License-Identifier: GPL-2.0-only
> /* Copyright (C) 2025 Advanced Micro Devices, Inc. */
> +#include <linux/string_helpers.h>
> #include <linux/device.h>
> #include <linux/pci.h>
>
> @@ -30,10 +31,111 @@ static char *cxl_cachedev_devnode(const struct device *dev, umode_t *mode,
> return kasprintf(GFP_KERNEL, "cxl/%s", dev_name(dev));
> }
>
> +static ssize_t serial_show(struct device *dev, struct device_attribute *attr,
> + char *buf)
> +{
> + struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
> + struct cxl_dev_state *cxlds = cxlcd->cxlds;
> +
> + return sysfs_emit(buf, "%#llx\n", cxlds->serial);
Just as an FYI there is a discussion about putting similar in place for the pci device.
https://lore.kernel.org/linux-pci/20250811173931.8068-2-thepacketgeek@gmail.com/
> +}
> +static DEVICE_ATTR_RO(serial);
> +
> +static ssize_t numa_node_show(struct device *dev, struct device_attribute *attr,
> + char *buf)
> +{
> + return sysfs_emit(buf, "%d\n", dev_to_node(dev));
I grumped about this when it was added for the memdev (duplication) but lost
then so I'll give in this time without a fight :)
> +}
> +static DEVICE_ATTR_RO(numa_node);
> +
> +static struct attribute *cxl_cachedev_attributes[] = {
> + &dev_attr_serial.attr,
> + &dev_attr_numa_node.attr,
Missing NULL?
> +};
> +
> +static umode_t cxl_cachedev_visible(struct kobject *kobj, struct attribute *a,
> + int n)
> +{
> + if (!IS_ENABLED(CONFIG_NUMA) && a == &dev_attr_numa_node.attr)
> + return 0;
> + return a->mode;
> +}
> +static DEVICE_ATTR_RO(cache_unit);
> +
> +static struct attribute *cxl_cachedev_cache_attributes[] = {
> + &dev_attr_cache_size.attr,
> + &dev_attr_cache_unit.attr,
> + NULL,
No comma.
> +};
> +
> +static const struct attribute_group *cxl_cachedev_attribute_groups[] = {
> + &cxl_cachedev_attribute_group,
> + &cxl_cachedev_cache_attribute_group,
> + NULL,
No trailing comma needed given it's a terminator.
> +};
> +
> static const struct device_type cxl_cachedev_type = {
> .name = "cxl_cachedev",
> .release = cxl_cachedev_release,
> .devnode = cxl_cachedev_devnode,
> + .groups = cxl_cachedev_attribute_groups,
> };
>
> bool is_cxl_cachedev(const struct device *dev)
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 18/18] cxl/core: Add cache device cache management attributes
2025-08-12 21:29 ` [RFC PATCH 18/18] cxl/core: Add cache device cache management attributes Ben Cheatham
@ 2025-08-19 15:53 ` Jonathan Cameron
2025-08-22 18:02 ` Cheatham, Benjamin
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 15:53 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 12 Aug 2025 16:29:21 -0500
Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
> Add functions to cxl/core/pci.c to manage the cache of a CXL.cache
> enabled endpoint. Add these new functions to sysfs attributes for
> userspace accessibility.
>
> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
Documentation/ABI needs something on this.
> ---
> drivers/cxl/core/cachedev.c | 99 +++++++++++++++++++++++++++++++++++++
> drivers/cxl/core/pci.c | 89 +++++++++++++++++++++++++++++++++
> drivers/cxl/cxlcache.h | 4 ++
> drivers/cxl/cxlpci.h | 6 +++
> 4 files changed, 198 insertions(+)
>
> diff --git a/drivers/cxl/core/cachedev.c b/drivers/cxl/core/cachedev.c
> index 1cdd94520ceb..85989cdd1eb8 100644
> --- a/drivers/cxl/core/cachedev.c
> +++ b/drivers/cxl/core/cachedev.c
> @@ -5,6 +5,7 @@
> #include <linux/pci.h>
>
> #include "../cxlcache.h"
> +#include "../cxlpci.h"
> #include "private.h"
>
> static DEFINE_IDA(cxl_cachedev_ida);
> @@ -125,9 +126,107 @@ static struct attribute_group cxl_cachedev_cache_attribute_group = {
> .is_visible = cxl_cachedev_cache_visible,
> };
>
> +static ssize_t cache_disable_store(struct device *dev,
> + struct device_attribute *attr,
> + const char *buf, size_t n)
> +{
> + struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
> + struct cxl_dev_state *cxlds = cxlcd->cxlds;
> + bool disable;
> + int rc;
> +
> + rc = kstrtobool(buf, &disable);
> + if (rc)
> + return rc;
> +
> + rc = cxl_accel_set_cache_disable(cxlds, disable);
> + if (rc)
> + return rc;
Can return 1 which is unlikely to be what want.
> +
> + return n;
> +}
> +
> +static struct attribute *cxl_cachedev_mgmt_attributes[] = {
> + &dev_attr_cache_disable.attr,
> + &dev_attr_cache_invalid.attr,
> + &dev_attr_init_wbinvd.attr,
> + NULL,
No comma.
> +};
> +static struct attribute_group cxl_cachedev_mgmt_attribute_group = {
> + .attrs = cxl_cachedev_mgmt_attributes,
> + .is_visible = cxl_cachedev_mgmt_visible,
> +};
> +
> static const struct attribute_group *cxl_cachedev_attribute_groups[] = {
> &cxl_cachedev_attribute_group,
> &cxl_cachedev_cache_attribute_group,
> + &cxl_cachedev_mgmt_attribute_group,
> NULL,
No comma
> };
>
> diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> index 2279c2690c59..667f75043f9e 100644
> --- a/drivers/cxl/core/pci.c
> +++ b/drivers/cxl/core/pci.c
> @@ -1232,3 +1232,92 @@ int cxl_accel_get_cache_info(struct cxl_dev_state *cxlds)
> return 0;
> }
> EXPORT_SYMBOL_NS_GPL(cxl_accel_get_cache_info, "CXL");
> +
> +int cxl_accel_caching_disabled(struct cxl_dev_state *cxlds)
> +{
> + struct pci_dev *pdev = to_pci_dev(cxlds->dev);
> + int dvsec, rc;
> + u16 ctrl2;
> +
> + device_lock_assert(cxlds->dev);
What took this and why do we need to check?
> +
> + if (!dev_is_pci(cxlds->dev))
> + return -EINVAL;
> + pdev = to_pci_dev(cxlds->dev);
> +
> + dvsec = cxlds->cxl_dvsec;
> + rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &ctrl2);
> + if (rc)
> + return rc;
> +
> + return !!(ctrl2 & CXL_DVSEC_DISABLE_CACHING);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_accel_caching_disabled, "CXL");
> +
> +int cxl_accel_set_cache_disable(struct cxl_dev_state *cxlds, bool disable)
> +{
> + struct pci_dev *pdev = to_pci_dev(cxlds->dev);
> + int dvsec, rc;
> + u16 ctrl2;
> +
> + if (!dev_is_pci(cxlds->dev))
> + return -EINVAL;
> +
> + guard(device)(cxlds->dev);
> + dvsec = cxlds->cxl_dvsec;
> + rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &ctrl2);
> + if (rc)
> + return rc;
> +
> + if (!!(ctrl2 & CXL_DVSEC_DISABLE_CACHING) == disable)
Maybe FIELD_GET() is cleaner than the !!
> + return 1;
> +
> + ctrl2 &= ~CXL_DVSEC_DISABLE_CACHING;
> + ctrl2 |= FIELD_PREP(CXL_DVSEC_DISABLE_CACHING, disable);
> + return pci_write_config_word(pdev, dvsec + CXL_DVSEC_DISABLE_CACHING,
> + ctrl2);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_accel_set_cache_disable, "CXL");
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 00/18] Initial CXL.cache device support
2025-08-13 11:25 ` [RFC PATCH 00/18] Initial CXL.cache device support Alejandro Lucero Palau
@ 2025-08-19 15:57 ` Jonathan Cameron
2025-08-19 16:05 ` Jonathan Cameron
2025-08-22 18:02 ` Cheatham, Benjamin
1 sibling, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 15:57 UTC (permalink / raw)
To: Alejandro Lucero Palau; +Cc: Ben Cheatham, linux-cxl
On Wed, 13 Aug 2025 12:25:12 +0100
Alejandro Lucero Palau <alucerop@amd.com> wrote:
> On 8/12/25 22:29, Ben Cheatham wrote:
> > This patch series adds initial CXL.cache support. What I have here only
> > allows for adding a cache device to the system, programming/validating
> > the system configuration with respect to cache devices, and some basic
> > cache reporting/management.
> >
> > The general philosophy is to have an endpoint/vendor-specific driver
> > that runs through the same steps of adding a cxl_memdev, but for the
> > cache portion of the device (both type 1 & 2). Getting cache support for
> > a CXL device should be as simple as: get cache information, set up the
> > memory region (see below), and then calling devm_cxl_add_cachedev().
> >
> > There's a couple of things missing from this set:
> >
> > 1) Missing an endpoint driver
> >
> > I plan on submitting a reference driver (type 1 or 2) with v1, I
> > figured I'd send out what I have before going much further.
> >
> > 2) Mapping/Reserving host memory used for CXL cache(s)
> >
> > I'm thinking this will be handled by the endpoint driver, but I'm
> > not sure what mechanism would be used and whether to integrate it
> > with the cxl_cache/core. Any thoughts/ideas here are appreciated!
>
>
> Hi Ben,
>
>
> Good to see you working on this. I'll go through the series these next days.
>
>
> FWIW, I did send a RFC last November about my concerns for
> properly/safely supporting CXL.cache.
>
>
> I had no time to keep thinking about it. Also, I sent a copy to the
> kernel iommu mailing list, but it did not get traction then. Anyways, as
> a summary, I think mapping for CXL.cache requires not just the endpoint
> driver doing it freely but through the cxl core for ensuring it is done
> properly and safely. But I have to admit this could be a
> misunderstanding about how the hardware is going to allow CXL.cache.
Perhaps it's time to restart that discussion.
https://lore.kernel.org/linux-cxl/cc2525a6-0f6a-c1c8-83e1-6396661efc8a@amd.com/
>
>
> Thanks
>
>
> > 3) RAS Support
> >
> > Same situation as 1) above.
> >
> > Some quick notes: The actual cache parts of this set are untested due
> > to problems with my set up, but I did make sure nothing here breaks type
> > 3 support. I'll have this fixed before sending out a v1. This series is
> > based on the for-6.18/cxl-probe-order (commit 5e29cbd1077b) branch in
> > the CXL repo, with v7 of Dave's deferred dport probe set on top [1]. I
> > added Dave's set to help with getting around constraints with HDM
> > decoders in CXL.cache device only configurations (see 08/18 for more).
> >
> > Patch Breakdown:
> > - 1 & 2: Preliminary changes for struct cxl_cachedev
> > - 3: Add struct cxl_cachedev
> > - 4-8: Preliminary changes for adding cache devices to port
> > hierarchy
> > - 9: Function for getting CXL cache info
> > - 10: cxl_cache driver (mirrors cxl_mem)
> > - 11-16: Checking CXL.cache capabilities for system configuration
> > validity
> > - 17-18: Cache device attributes
> >
> > [1]:
> > Link: https://lore.kernel.org/linux-cxl/20250714223527.461147-1-dave.jiang@intel.com/
> >
> > Ben Cheatham (18):
> > cxl/mem: Change cxl_memdev_ops to cxl_dev_ops
> > cxl: Move struct cxl_dev_state definition
> > cxl/core: Add CXL.cache device struct
> > cxl: Replace cxl_mem_find_port() with cxl_dev_find_port()
> > cxl: Change cxl_ep_load() to use struct device * parameter
> > cxl/port, mem: Make adding an endpoint device type agnostic
> > cxl/port: Split endpoint port probe on device type
> > cxl/port: Update switch_port_probe() for CXL cache devices
> > cxl/core: Add function for getting CXL cache info
> > cxl/cache: Add cxl_cache driver
> > cxl/core: Add CXL snoop filter setup and checking
> > cxl/cache: Add CXL Cache ID Route Table mapping
> > cxl/cache: Implement Cache ID Route Table programming
> > cxl/cache: Add Cache ID Decoder capability mapping
> > cxl/cache: Implement Cache ID Decoder programming
> > cxl/cache: Add cache device counting for CXL ports
> > cxl/core: Add cache device attributes
> > cxl/core: Add cache device cache management attributes
> >
> > drivers/cxl/Kconfig | 14 +
> > drivers/cxl/Makefile | 2 +
> > drivers/cxl/cache.c | 276 +++++++++++++++++++
> > drivers/cxl/core/Makefile | 1 +
> > drivers/cxl/core/cachedev.c | 292 ++++++++++++++++++++
> > drivers/cxl/core/hdm.c | 31 +++
> > drivers/cxl/core/memdev.c | 2 +-
> > drivers/cxl/core/pci.c | 134 ++++++++++
> > drivers/cxl/core/port.c | 518 +++++++++++++++++++++++++++++++++---
> > drivers/cxl/core/region.c | 25 +-
> > drivers/cxl/core/regs.c | 28 ++
> > drivers/cxl/cxl.h | 193 +++++++++++++-
> > drivers/cxl/cxlcache.h | 42 +++
> > drivers/cxl/cxlmem.h | 121 +--------
> > drivers/cxl/cxlpci.h | 10 +
> > drivers/cxl/mem.c | 14 +-
> > drivers/cxl/pci.c | 2 +-
> > drivers/cxl/port.c | 54 ++--
> > drivers/cxl/private.h | 8 +-
> > 19 files changed, 1569 insertions(+), 198 deletions(-)
> > create mode 100644 drivers/cxl/cache.c
> > create mode 100644 drivers/cxl/core/cachedev.c
> > create mode 100644 drivers/cxl/cxlcache.h
> >
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 00/18] Initial CXL.cache device support
2025-08-19 15:57 ` Jonathan Cameron
@ 2025-08-19 16:05 ` Jonathan Cameron
2025-08-26 10:42 ` Alejandro Lucero Palau
0 siblings, 1 reply; 55+ messages in thread
From: Jonathan Cameron @ 2025-08-19 16:05 UTC (permalink / raw)
To: Alejandro Lucero Palau; +Cc: Ben Cheatham, linux-cxl
On Tue, 19 Aug 2025 16:57:54 +0100
Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> On Wed, 13 Aug 2025 12:25:12 +0100
> Alejandro Lucero Palau <alucerop@amd.com> wrote:
>
> > On 8/12/25 22:29, Ben Cheatham wrote:
> > > This patch series adds initial CXL.cache support. What I have here only
> > > allows for adding a cache device to the system, programming/validating
> > > the system configuration with respect to cache devices, and some basic
> > > cache reporting/management.
> > >
> > > The general philosophy is to have an endpoint/vendor-specific driver
> > > that runs through the same steps of adding a cxl_memdev, but for the
> > > cache portion of the device (both type 1 & 2). Getting cache support for
> > > a CXL device should be as simple as: get cache information, set up the
> > > memory region (see below), and then calling devm_cxl_add_cachedev().
> > >
> > > There's a couple of things missing from this set:
> > >
> > > 1) Missing an endpoint driver
> > >
> > > I plan on submitting a reference driver (type 1 or 2) with v1, I
> > > figured I'd send out what I have before going much further.
> > >
> > > 2) Mapping/Reserving host memory used for CXL cache(s)
> > >
> > > I'm thinking this will be handled by the endpoint driver, but I'm
> > > not sure what mechanism would be used and whether to integrate it
> > > with the cxl_cache/core. Any thoughts/ideas here are appreciated!
> >
> >
> > Hi Ben,
> >
> >
> > Good to see you working on this. I'll go through the series these next days.
> >
> >
> > FWIW, I did send a RFC last November about my concerns for
> > properly/safely supporting CXL.cache.
> >
> >
> > I had no time to keep thinking about it. Also, I sent a copy to the
> > kernel iommu mailing list, but it did not get traction then. Anyways, as
> > a summary, I think mapping for CXL.cache requires not just the endpoint
> > driver doing it freely but through the cxl core for ensuring it is done
> > properly and safely. But I have to admit this could be a
> > misunderstanding about how the hardware is going to allow CXL.cache.
>
> Perhaps it's time to restart that discussion.
>
> https://lore.kernel.org/linux-cxl/cc2525a6-0f6a-c1c8-83e1-6396661efc8a@amd.com/
>
I hit send too fast. For that thread I'd suggest we break it down into
the various issues you raise. Make it a bit more bite sized!
Some such as trust should be relatively easy to tick off with the work
Lukas is doing on attestation (or the version that uses confidential
compute infrastructure)
Jonathan
> >
> >
> > Thanks
> >
> >
> > > 3) RAS Support
> > >
> > > Same situation as 1) above.
> > >
> > > Some quick notes: The actual cache parts of this set are untested due
> > > to problems with my set up, but I did make sure nothing here breaks type
> > > 3 support. I'll have this fixed before sending out a v1. This series is
> > > based on the for-6.18/cxl-probe-order (commit 5e29cbd1077b) branch in
> > > the CXL repo, with v7 of Dave's deferred dport probe set on top [1]. I
> > > added Dave's set to help with getting around constraints with HDM
> > > decoders in CXL.cache device only configurations (see 08/18 for more).
> > >
> > > Patch Breakdown:
> > > - 1 & 2: Preliminary changes for struct cxl_cachedev
> > > - 3: Add struct cxl_cachedev
> > > - 4-8: Preliminary changes for adding cache devices to port
> > > hierarchy
> > > - 9: Function for getting CXL cache info
> > > - 10: cxl_cache driver (mirrors cxl_mem)
> > > - 11-16: Checking CXL.cache capabilities for system configuration
> > > validity
> > > - 17-18: Cache device attributes
> > >
> > > [1]:
> > > Link: https://lore.kernel.org/linux-cxl/20250714223527.461147-1-dave.jiang@intel.com/
> > >
> > > Ben Cheatham (18):
> > > cxl/mem: Change cxl_memdev_ops to cxl_dev_ops
> > > cxl: Move struct cxl_dev_state definition
> > > cxl/core: Add CXL.cache device struct
> > > cxl: Replace cxl_mem_find_port() with cxl_dev_find_port()
> > > cxl: Change cxl_ep_load() to use struct device * parameter
> > > cxl/port, mem: Make adding an endpoint device type agnostic
> > > cxl/port: Split endpoint port probe on device type
> > > cxl/port: Update switch_port_probe() for CXL cache devices
> > > cxl/core: Add function for getting CXL cache info
> > > cxl/cache: Add cxl_cache driver
> > > cxl/core: Add CXL snoop filter setup and checking
> > > cxl/cache: Add CXL Cache ID Route Table mapping
> > > cxl/cache: Implement Cache ID Route Table programming
> > > cxl/cache: Add Cache ID Decoder capability mapping
> > > cxl/cache: Implement Cache ID Decoder programming
> > > cxl/cache: Add cache device counting for CXL ports
> > > cxl/core: Add cache device attributes
> > > cxl/core: Add cache device cache management attributes
> > >
> > > drivers/cxl/Kconfig | 14 +
> > > drivers/cxl/Makefile | 2 +
> > > drivers/cxl/cache.c | 276 +++++++++++++++++++
> > > drivers/cxl/core/Makefile | 1 +
> > > drivers/cxl/core/cachedev.c | 292 ++++++++++++++++++++
> > > drivers/cxl/core/hdm.c | 31 +++
> > > drivers/cxl/core/memdev.c | 2 +-
> > > drivers/cxl/core/pci.c | 134 ++++++++++
> > > drivers/cxl/core/port.c | 518 +++++++++++++++++++++++++++++++++---
> > > drivers/cxl/core/region.c | 25 +-
> > > drivers/cxl/core/regs.c | 28 ++
> > > drivers/cxl/cxl.h | 193 +++++++++++++-
> > > drivers/cxl/cxlcache.h | 42 +++
> > > drivers/cxl/cxlmem.h | 121 +--------
> > > drivers/cxl/cxlpci.h | 10 +
> > > drivers/cxl/mem.c | 14 +-
> > > drivers/cxl/pci.c | 2 +-
> > > drivers/cxl/port.c | 54 ++--
> > > drivers/cxl/private.h | 8 +-
> > > 19 files changed, 1569 insertions(+), 198 deletions(-)
> > > create mode 100644 drivers/cxl/cache.c
> > > create mode 100644 drivers/cxl/core/cachedev.c
> > > create mode 100644 drivers/cxl/cxlcache.h
> > >
> >
>
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming
2025-08-19 13:44 ` Alireza Sanaee
@ 2025-08-20 8:55 ` Alireza Sanaee
0 siblings, 0 replies; 55+ messages in thread
From: Alireza Sanaee @ 2025-08-20 8:55 UTC (permalink / raw)
To: Ben Cheatham; +Cc: linux-cxl
On Tue, 19 Aug 2025 14:44:45 +0100
Alireza Sanaee <alireza.sanaee@huawei.com> wrote:
> On Tue, 12 Aug 2025 16:29:18 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
> > The CXL Cache ID Decoder capability is an optional capability
> > present on CXL downstream switch and CXL-enabled PCIe Root ports.
> > This capability is required for having multiple CXL.cache enabled
> > devices under the same port (CXL 3.2 8.2.4.29).
> >
> > Implement cache id decoder programming. Validate the configuration
> > in the case BIOS already programmed the id(s).
> >
> > Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> > ---
> > drivers/cxl/cache.c | 4 ++
> > drivers/cxl/core/port.c | 109
> > ++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h |
> > 14 ++++++ drivers/cxl/cxlcache.h | 3 ++
> > 4 files changed, 130 insertions(+)
> >
> > diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
> > index f123d596187d..fa3223165a18 100644
> > --- a/drivers/cxl/cache.c
> > +++ b/drivers/cxl/cache.c
> > @@ -163,6 +163,10 @@ static int
> > devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd) rc =
> > map_cache_idd_cap(dport); if (rc)
> > return rc;
> > +
> > + rc = cxl_dport_program_cache_idd(dport, cxlcd);
> > + if (rc)
> > + return rc;
> > }
> >
> > return 0;
> > diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> > index f491796f6e60..3606ad557bfb 100644
> > --- a/drivers/cxl/core/port.c
> > +++ b/drivers/cxl/core/port.c
> > @@ -1080,6 +1080,115 @@ int devm_cxl_port_program_cache_idrt(struct
> > cxl_port *port, }
> > EXPORT_SYMBOL_NS_GPL(devm_cxl_port_program_cache_idrt, "CXL");
> >
> > +static int cxl_dport_commit_idd(struct cxl_dport *dport)
> > +{
> > + unsigned long timeout, start, end;
> > + u32 cap, ctrl, status;
> > + int i;
> > +
> > + cap = readl(dport->regs.cidd + CXL_CACHE_IDD_CAP_OFFSET);
> > + if (!(cap & CXL_CACHE_IDD_CAP_COMMIT))
> > + return 0;
> > +
> > + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> > + if (ctrl & CXL_CACHE_IDD_CTRL_COMMIT) {
> > + ctrl &= ~CXL_CACHE_IDD_CTRL_COMMIT;
> > + writel(ctrl, dport->regs.cidd +
> > CXL_CACHE_IDD_CTRL_OFFSET);
> > + }
> > +
> > + ctrl |= CXL_CACHE_IDD_CTRL_COMMIT;
> > + writel(ctrl, dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> > +
> > + status = readl(dport->regs.cidd +
> > CXL_CACHE_IDD_STAT_OFFSET); +
> > + i = FIELD_GET(CXL_CACHE_IDD_STAT_TIMEOUT_SCALE_MASK,
> > status);
> > + timeout = 1 * HZ;
> > + while (i-- > 3)
> > + timeout *= 10;
> > +
> > + timeout *= FIELD_GET(CXL_CACHE_IDD_STAT_TIMEOUT_BASE_MASK,
> > status);
> > + start = jiffies;
> > + do {
> > + status = readl(dport->regs.cidd +
> > CXL_CACHE_IDD_STAT_OFFSET);
> > + if (status & CXL_CACHE_IDD_STAT_ERR_COMMIT)
> > + return -EBUSY;
> > +
> > + if (status & CXL_CACHE_IDD_STAT_COMMITTED)
> > + return 0;
> > +
> > + end = jiffies;
> > + } while (time_before(end, start + timeout));
> > +
> > + return -ETIMEDOUT;
> > +}
> > +
> > +static int cxl_cache_idd_alread_programmed(struct cxl_dport *dport,
> > + struct cxl_cachedev
> > *cxlcd) +{
> > + struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
> > + u32 ctrl, status, cache_id;
> > +
> > + status = readl(dport->regs.cidd +
> > CXL_CACHE_IDD_STAT_OFFSET);
> > + if (!(status & CXL_CACHE_IDD_STAT_COMMITTED))
> > + return 0;
> > +
> > + /*
> > + * The decoder is probably already committed by BIOS,
> > validate
> > + * the configuration
> > + */
> > + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> > + /* TODO: Check for 68B flit mode */
> > + if (dport == cxlcd->endpoint->parent_dport &&
> > + !(ctrl & CXL_CACHE_IDD_CTRL_ASGN_ID))
> > + return -EINVAL;
> > +
> > + if (dport != cxlcd->endpoint->parent_dport &&
> > + !(ctrl & CXL_CACHE_IDD_CTRL_FWD_ID))
> > + return -EINVAL;
> > +
> > + cache_id = FIELD_GET(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK,
> > ctrl);
> > + if (cache_id != cstate->cache_id)
> > + return -EINVAL;
> > +
> > + if (!!(ctrl & CXL_CACHE_IDD_CTRL_TYPE2) != cstate->hdmd)
> > + return -EINVAL;
> > +
> > + return 1;
> > +}
> > +
> > +int cxl_dport_program_cache_idd(struct cxl_dport *dport,
> > + struct cxl_cachedev *cxlcd)
> > +{
> > + u32 ctrl;
> > + int rc;
> > +
> > + rc = cxl_cache_idd_alread_programmed(dport, cxlcd);
> > + if (rc)
> > + return rc < 0 ? rc : 0;
> > +
> > + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> > +
> > + if (cxlcd->cxlds->cstate.hdmd) {
> > + ctrl |= CXL_CACHE_IDD_CTRL_TYPE2;
> > + ctrl &= ~CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK;
> > + ctrl |=
> > FIELD_PREP(CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK, cxlcd->id);
> > + }
> > +
> > + /* TODO: Check for 68B flit mode */
> > + if (dport == cxlcd->endpoint->parent_dport)
> > + ctrl |= CXL_CACHE_IDD_CTRL_ASGN_ID;
> > + else
> > + ctrl |= CXL_CACHE_IDD_CTRL_FWD_ID;
> > +
> > + ctrl &= ~CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK;
> > + ctrl |= FIELD_PREP(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK,
> > + cxlcd->cxlds->cstate.cache_id);
> > +
> > + writel(ctrl, dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
> > + return cxl_dport_commit_idd(dport);
> > +}
> > +EXPORT_SYMBOL_NS_GPL(cxl_dport_program_cache_idd, "CXL");
> > +
> > DEFINE_SHOW_ATTRIBUTE(einj_cxl_available_error_type);
> >
> > static int cxl_einj_inject(void *data, u64 type)
> > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> > index 04fde1a994d0..9c475d8c1573 100644
> > --- a/drivers/cxl/cxl.h
> > +++ b/drivers/cxl/cxl.h
> > @@ -178,6 +178,20 @@ static inline int ways_to_eiw(unsigned int
> > ways, u8 *eiw) #define CXL_CACHE_IDRT_TARGETN_PORT_MASK
> > GENMASK(15, 8)
> > /* CXL 3.2 8.2.4.29 CXL Cache ID Decoder Capability Structure */
> > +#define CXL_CACHE_IDD_CAP_OFFSET 0x0
> > +#define CXL_CACHE_IDD_CAP_COMMIT BIT(0)
> > +#define CXL_CACHE_IDD_CTRL_OFFSET 0x4
> > +#define CXL_CACHE_IDD_CTRL_FWD_ID BIT(0)
> > +#define CXL_CACHE_IDD_CTRL_ASGN_ID BIT(1)
> > +#define CXL_CACHE_IDD_CTRL_TYPE2 BIT(2)
> > +#define CXL_CACHE_IDD_CTRL_COMMIT BIT(3)
> > +#define CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK GENMASK(11, 8)
> > +#define CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK GENMASK(19, 16)
> > +#define CXL_CACHE_IDD_STAT_OFFSET 0x8
> > +#define CXL_CACHE_IDD_STAT_COMMITTED BIT(0)
> > +#define CXL_CACHE_IDD_STAT_ERR_COMMIT BIT(1)
> > +#define CXL_CACHE_IDD_STAT_TIMEOUT_SCALE_MASK GENMASK(11, 8)
> > +#define CXL_CACHE_IDD_STAT_TIMEOUT_BASE_MASK GENMASK(15, 12)
> > #define CXL_CACHE_IDD_CAPABILITY_LENGTH 0xC
>
> Hi Ben,
>
> This looks like some styling issues.
Hi Ben,
Sorry I misread this.
> >
> > /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
> > diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
> > index d28222123102..60563b391844 100644
> > --- a/drivers/cxl/cxlcache.h
> > +++ b/drivers/cxl/cxlcache.h
> > @@ -32,4 +32,7 @@ int devm_cxl_snoop_filter_alloc(u32 gid, struct
> > cxl_dev_state *cxlds); int devm_cxl_port_program_cache_idrt(struct
> > cxl_port *port, struct cxl_dport *dport,
> > struct cxl_cachedev *cxlcd);
> > +int cxl_dport_program_cache_idd(struct cxl_dport *dport,
> > + struct cxl_cachedev *cxlcd);
> > +
> > #endif
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 02/18] cxl: Move struct cxl_dev_state definition
2025-08-19 11:33 ` Jonathan Cameron
@ 2025-08-22 18:00 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:00 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 6:33 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:05 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> Move struct cxl_dev_state (and associated structs) from cxlmem.h to
>> cxl.h. This is in preparation of adding a new CXL cache device type
>> that will also use cxl_dev_state.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> Hi Ben,
>
> I'm just going to take a first (likely superficial look at this) so
> comments may well be lightweight. I need to get my head around it
> before offering more substantial stuff.
>
> Jonathan
No problem, thanks for taking a look!
>> ---
>> drivers/cxl/cxl.h | 114 +++++++++++++++++++++++++++++++++++++++++++
>> drivers/cxl/cxlmem.h | 113 ------------------------------------------
>> 2 files changed, 114 insertions(+), 113 deletions(-)
>>
>> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
>> index 1de4fec5c8f4..751e5860423a 100644
>> --- a/drivers/cxl/cxl.h
>> +++ b/drivers/cxl/cxl.h
>> @@ -11,6 +11,7 @@
>> #include <linux/log2.h>
>> #include <linux/node.h>
>> #include <linux/io.h>
>> +#include <cxl/mailbox.h>
>>
>> extern const struct nvdimm_security_ops *cxl_security_ops;
>>
>> @@ -696,6 +697,119 @@ struct cxl_ep {
>> struct cxl_port *next;
>> };
>>
>> +
> One blank line probably enough.
>
Yep, messed up the copy/paste. Will remove.
>> +/*
>> + * enum cxl_devtype - delineate type-2 from a generic type-3 device
>> + * @CXL_DEVTYPE_DEVMEM - Vendor specific CXL Type-2 device implementing HDM-D or
>> + * HDM-DB, no requirement that this device implements a
>> + * mailbox, or other memory-device-standard manageability
>> + * flows.
>> + * @CXL_DEVTYPE_CLASSMEM - Common class definition of a CXL Type-3 device with
>> + * HDM-H and class-mandatory memory device registers
>> + */
>> +enum cxl_devtype {
>> + CXL_DEVTYPE_DEVMEM,
>> + CXL_DEVTYPE_CLASSMEM,
>> +};
>
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 03/18] cxl/core: Add CXL.cache device struct
2025-08-19 11:48 ` Jonathan Cameron
@ 2025-08-22 18:00 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:00 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 6:48 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:06 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> Add a new CXL.cache device (struct cxl_cachedev) that is the cache
>> analogue to struct cxl_memdev. This device will be created by
>> endpoint vendor-specific drivers to enable and manage the cache
>> capabilities of the endpoint.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>
> Whilst this creates the device it never adds which seems like an odd omission.
> If that's intentional add some more text on why.
>
> Jonathan
Yeah, I just wanted to keep the size of the patch down. It would probably make more sense
if I moved this patch next to the patch where these devices get added (10/18 IIRC).
I'll try and squash the two together (or go more fine-grained if it's too big).
>
>> diff --git a/drivers/cxl/core/cachedev.c b/drivers/cxl/core/cachedev.c
>> new file mode 100644
>> index 000000000000..aaf4a1b94178
>> --- /dev/null
>> +++ b/drivers/cxl/core/cachedev.c
>> @@ -0,0 +1,91 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/* Copyright (C) 2025 Advanced Micro Devices, Inc. */
>> +#include <linux/device.h>
>> +#include <linux/pci.h>
>> +
>> +#include "../cxlcache.h"
>> +#include "private.h"
>> +
>> +static DEFINE_IDA(cxl_cachedev_ida);
>> +
>> +static void cxl_cachedev_release(struct device *dev)
>> +{
>> + struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
>> +
>> + ida_free(&cxl_cachedev_ida, cxlcd->id);
>> + kfree(cxlcd);
>> +}
>> +
>> +static void cxl_cachedev_unregister(void *dev)
>> +{
>> + struct cxl_cachedev *cxlcd = dev;
>> +
>
> It's not doing what we'd normally expect in an unregister
> as no device_del() Which is fair because it was never added below.
>
Yeah, that's an omission on my part. I don't think I ever actually added the device_del(),
will do.
>> + cxlcd->cxlds = NULL;
>> + put_device(&cxlcd->dev);
>> +}
>> +
>
>> +struct cxl_cachedev *devm_cxl_cachedev_add_or_reset(struct device *host,
>> + struct cxl_cachedev *cxlcd)
>> +{
>> + int rc;
>
> Odd to not have a device_add() of any type in here.
> Maybe introduced in later patches, but I think you need to drag that into
> this one for it to make any real sense.
>
>> +
>> + rc = devm_add_action_or_reset(host, cxl_cachedev_unregister, cxlcd);
>> + if (rc)
>> + return ERR_PTR(rc);
> I'd throw a blank line in here.
>
Sure.
>> + return cxlcd;
>> +}
>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_cachedev_add_or_reset, "CXL");
>
>
>> diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
>> new file mode 100644
>> index 000000000000..3ccbeba9fcd6
>> --- /dev/null
>> +++ b/drivers/cxl/cxlcache.h
>> @@ -0,0 +1,23 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> +#ifndef __CXL_CACHE_H__
>> +#define __CXL_CACHE_H__
>> +#include <linux/pci.h>
>> +
>> +#include "cxl.h"
>> +
>
> I'd add docs from the start.
>
Will do!
>> +struct cxl_cachedev {
>> + struct device dev;
>> + struct cxl_dev_state *cxlds;
>> + struct cxl_port *endpoint;
>> + const struct cxl_dev_ops *ops;
>> + int id;
>> + int depth;
>> +};
>> +
>> +static inline struct cxl_cachedev *to_cxl_cachedev(struct device *dev)
>> +{
>> + return container_of(dev, struct cxl_cachedev, dev);
>> +}
>> +
>> +bool is_cxl_cachedev(const struct device *dev);
>> +#endif
>
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 06/18] cxl/port, mem: Make adding an endpoint device type agnostic
2025-08-19 11:53 ` Jonathan Cameron
@ 2025-08-22 18:00 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:00 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 6:53 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:09 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> Change devm_cxl_enumerate_ports() and devm_cxl_add_endpoint() to take a
>> generic struct device * instead of struct cxl_memdev in preparation of
>> adding cache devices to the port hierarchy.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>
>> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
>> index cdac7bfa2289..c615e6c4ee60 100644
>> --- a/drivers/cxl/mem.c
>> +++ b/drivers/cxl/mem.c
>> @@ -104,9 +104,11 @@ static int cxl_mem_probe(struct device *dev)
>> if (rc)
>> return rc;
>>
>> - rc = devm_cxl_enumerate_ports(cxlmd);
>> - if (rc)
>> + rc = devm_cxl_enumerate_ports(&cxlmd->dev);
>> + if (rc) {
> Not immediately obvious to me why this needs to be set. Perhaps a comment?
It's based on the changes Dan made on the 6.18 probe order branch. It's a change made for type 2
drivers to get more fine-grained error reporting when probe fails IIRC.
I don't mind adding a comment though.
>> + cxlmd->endpoint = ERR_PTR(rc);
>> return rc;
>> + }
>>
>> struct cxl_port *parent_port __free(put_cxl_port) =
>> cxl_dev_find_port(&cxlmd->dev, &dport);
>> @@ -138,7 +140,7 @@ static int cxl_mem_probe(struct device *dev)
>> return -ENXIO;
>> }
>>
>> - rc = devm_cxl_add_endpoint(endpoint_parent, cxlmd, dport);
>> + rc = devm_cxl_add_endpoint(endpoint_parent, &cxlmd->dev, dport);
>> if (rc)
>> return rc;
>> }
>
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 07/18] cxl/port: Split endpoint port probe on device type
2025-08-19 11:57 ` Jonathan Cameron
@ 2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:01 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 6:57 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:10 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> The current endpoint port probe assumes the device has CXL.mem
>> capabilities. With the addition of cache devices, this is no longer
>> gauranteed. With this in mind, split endpoint port probe into separate
>
> guaranteed
I can never spell this word right, will fix it.
>
>> functions for CXL.cache and CXL.mem endpoints.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 08/18] cxl/port: Update switch_port_probe() for CXL cache devices
2025-08-19 12:03 ` Jonathan Cameron
@ 2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:01 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 7:03 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:11 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> CXL switch port probe currently assumes there are only CXL.mem-capable
>> devices below the switch with preconfigured HDM decoders. Now that
>> CXL.cache devices can be added below a switch port, it's possible that
>> they either have no HDM decoders (type 1) or the CXL.mem capabilities of
>> the endpoint haven't been set up yet (type 2).
>>
>> The HDM decoders being disabled (if present) is no longer a gauranteed
>> failure condition when only cache devices are present below the port.
>> Knowing what kind of devices are under the switch port isn't possible
>> until the endpoints are probed, so delay HDM setup and validation until
>> endpoint port probe.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>> ---
>> drivers/cxl/core/hdm.c | 31 +++++++++++++++++++++++++++++++
>> drivers/cxl/cxl.h | 1 +
>> drivers/cxl/port.c | 23 +++++++++++++----------
>> 3 files changed, 45 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
>> index 865a71bce251..6e04f1f4c166 100644
>> --- a/drivers/cxl/core/hdm.c
>> +++ b/drivers/cxl/core/hdm.c
>> @@ -205,6 +205,37 @@ struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port,
>> }
>> EXPORT_SYMBOL_NS_GPL(devm_cxl_setup_hdm, "CXL");
>>
>> +int cxl_validate_endpoint_hdm_setup(struct cxl_port *endpoint)
>> +{
>> + struct cxl_port *sp = parent_port_of(endpoint);
>> + struct cxl_hdm *cxlhdm;
>> + int rc;
>> +
>> + for (; sp && !is_cxl_root(sp); sp = parent_port_of(sp)) {
>
> Might as well set sp as loop init rather than earlier.
>
Yep, will do.
>> + cxlhdm = dev_get_drvdata(&sp->dev);
>> + if (cxlhdm && cxlhdm->decoder_count > 0)
>> + continue;
>> +
>> + if (!sp->reg_map.component_map.hdm_decoder.valid || !cxlhdm) {
>> + dev_err(&sp->dev, "Missing HDM decoder capability\n");
>> + return -ENXIO;
>> + }
>> +
>> + if (sp->total_dports == 1) {
>> + dev_dbg(&sp->dev, "Fallback to passthrough decoder\n");
>> + rc = devm_cxl_add_passthrough_decoder(sp);
>> + if (rc)
>> + return rc;
>> + }
>> + }
>> +
>> + if (is_cxl_root(sp))
>> + return 0;
>
> Could move this test into the loop, alowing the loop condition to be simplified
> to just sp;
>
Sure!
>> +
>> + return -ENODEV;
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_validate_endpoint_hdm_setup, "CXL");
>> +
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 11/18] cxl/core: Add CXL snoop filter setup and checking
2025-08-19 14:18 ` Jonathan Cameron
@ 2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:01 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 9:18 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:14 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> Add functions to set up and validate the snoop filter configuration of a
>> CXL-capable PCIe Root Port (struct cxl_dport). Allocate space in the
>> snoop filter as part of cxl_cachedev probe.
>>
>
> Any thoughts on how to handle allocation policy, when combined with Cache_SF_Coverage
> hints or indeed just overcommiting the snoop filter and relying on snoops
> to clear space.
>
I'm not sure. The coverage fields are described as performance hints which makes me think
that devices may not pay attention to the fields to begin with. If it's common for devices
to ignore those field, it seems rejecting devices based on snoop capacity is the way to go.
If it's expected for devices to pay attention then I could try to come up with an allocation
scheme, but I would rather be conservative to start.
> The spec is rather ambiguous on when these can be used as opposed to
> just rejecting devices if we are out of space.
The spec is ambiguous on most things relating to CXL.cache in my experience :/.
>
>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>
> A few comments inline.
>
> Thanks,
>
> Jonathan
>
>>
>> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
>> index 5144ddac7145..26ea22a2638c 100644
>> --- a/drivers/cxl/core/port.c
>> +++ b/drivers/cxl/core/port.c
>> @@ -35,6 +35,9 @@ static DEFINE_IDA(cxl_port_ida);
>> static DEFINE_XARRAY(cxl_root_buses);
>> static DEFINE_XARRAY(cxl_root_ports);
>>
>> +DECLARE_RWSEM(cxl_snoop_rwsem);
>
> static?
>
Yep.
>> +static DEFINE_XARRAY(cxl_snoop_filters);
>> +
>> /*
>> * The terminal device in PCI is NULL and @platform_bus
>> * for platform devices (for cxl_test)
>> @@ -809,6 +812,139 @@ static int cxl_dport_setup_regs(struct device *host, struct cxl_dport *dport,
>> return rc;
>> }
>>
>> +struct cxl_snoop_filter {
>> + u64 size;
>> + struct xarray dports;
>> + struct xarray devs;
>> + struct mutex lock; /* Used for filter allocations */
>> +};
>> +
>> +static struct cxl_snoop_filter *cxl_create_snoop_filter(struct cxl_dport *dport,
>> + u32 gid, u32 size)
>> +{
>> + struct cxl_snoop_filter *sf;
>> + int rc;
>> +
>> + sf = xa_load(&cxl_snoop_filters, gid);
>> + if (sf)
>> + return sf;
>> +
>> + guard(rwsem_write)(&cxl_snoop_rwsem);
>> + sf = kzalloc(sizeof(*sf), GFP_KERNEL);
>> + if (!sf)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + sf->size = size;
>> + xa_init(&sf->dports);
>> + xa_init(&sf->devs);
>> + mutex_init(&sf->lock);
>> +
>> + rc = xa_insert(&sf->dports, (unsigned long)dport->dport_dev, dport,
>
> So the aim here is include all dports with this GID in sf->dports?
> If so why isn't one added if the xa_load() above finds sf?
>
It should be, I just missed it :/.
>> + GFP_KERNEL);
>> + if (rc)
>> + goto err;
>> +
>> + rc = xa_insert(&cxl_snoop_filters, gid, sf, GFP_KERNEL);
>> + if (rc)
>> + goto err;
>> +
>> + return 0;
>> +
>> +err:
>> + xa_destroy(&sf->dports);
>> + xa_destroy(&sf->devs);
>> + mutex_destroy(&sf->lock);
>> + kfree(sf);
>> + return ERR_PTR(rc);
>> +}
>> +
>> +static void cxl_destroy_snoop_filters(void)
>> +{
>> + struct cxl_snoop_filter *sf;
>> + unsigned long gid;
>> +
>> + guard(rwsem_write)(&cxl_snoop_rwsem);
>> + xa_for_each(&cxl_snoop_filters, gid, sf) {
>> + xa_destroy(&sf->dports);
>> + xa_destroy(&sf->devs);
>> + mutex_destroy(&sf->lock);
>
> Not freeing sf?
>
Missed it, will add.
>> + }
>> +
>> + xa_destroy(&cxl_snoop_filters);
>> +}
>> +
>> +static void cxl_snoop_filter_free(void *data)
>> +{
>> + struct cxl_cache_state *cstate = data;
>> + struct cxl_snoop_filter *sf;
>> +
>> + sf = xa_load(&cxl_snoop_filters, cstate->snoop_id);
>> + if (!sf)
>
> I'd put a debug print here at least and possibly scream as fairly
> sure this means we are trying to free something we never had.
>
Sure.
>> + return;
>> +
>> + guard(mutex)(&sf->lock);
>> + sf->size += cstate->size;
>> +}
>> +
>> +int devm_cxl_snoop_filter_alloc(u32 gid, struct cxl_dev_state *cxlds)
>
> I'd rename this as it is allocating part of the capacity rather
> than allocating the snoop filter representation.
> _alloc_for_device maybe?
>
Yeah, that sounds good to me.
>
>> +{
>> + struct cxl_cache_state *cstate = &cxlds->cstate;
>> + struct cxl_snoop_filter *sf;
>> +
>> + sf = xa_load(&cxl_snoop_filters, gid);
>> + if (!sf)
>> + return -ENODEV;
>> +
>> + guard(mutex)(&sf->lock);
>> + if (cstate->size > sf->size)
>> + return -EBUSY;
>> +
>> + sf->size -= cstate->size;
>> + cstate->snoop_id = gid;
>> + return devm_add_action_or_reset(cxlds->dev, cxl_snoop_filter_free,
>> + cstate);
>> +}
>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_snoop_filter_alloc, "CXL");
>> +
>> +int cxl_dport_setup_snoop_filter(struct cxl_dport *dport)
>> +{
>> + struct cxl_port *port = dport->port, *parent_port;
>> + struct cxl_snoop_filter *sf;
>> + u32 snoop, gid, size;
>> + int rc;
>> +
>> + /* Only supported by CXL-enabled PCIe root ports */
>
> I'd go with the CXL spec term which is simply "CXL root port".
>
Yeah I can do that. I wanted to be clear here since "root port" is an overloaded
term in the driver.
>> + parent_port = parent_port_of(port);
>> + if (!parent_port || !is_cxl_root(parent_port))
>> + return 0;
>> +
>> + if (!dport->reg_map.component_map.snoop.valid) {
>> + dev_dbg(dport->dport_dev, "missing snoop filter capability\n");
>> + return -ENXIO;
>> + }
>> +
>> + rc = cxl_map_component_regs(&dport->reg_map, &dport->regs.component,
>> + BIT(CXL_CM_CAP_CAP_ID_SNOOP));
>> + if (rc)
>> + return rc;
>> +
>> + snoop = readl(dport->regs.snoop + CXL_SNOOP_GROUP_ID_OFFSET);
>> + gid = FIELD_GET(CXL_SNOOP_GROUP_ID_MASK, snoop);
>> +
>> + snoop = readl(dport->regs.snoop + CXL_SNOOP_FILTER_SIZE_OFFSET);
>> + size = FIELD_GET(CXL_SNOOP_FILTER_SIZE_MASK, snoop);
>
> I'm not sure I'd bother with masking when it is the whole register with
> no reserved bits.
>
Ok, I'll remove it.
>> + if (!size)
>> + return -ENXIO;
>> +
>> + sf = cxl_create_snoop_filter(dport, gid, size);
>> + if (IS_ERR(sf))
>> + return PTR_ERR(sf);
>> +
>> + dport->snoop_id = gid;
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_dport_setup_snoop_filter, "CXL");
>
> Export seems unnecessary given it's used later in this file.
> Maybe that changes in later patches.
>
>> +
>
>
>> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
>> index 1771c42a1e3b..2881e0826829 100644
>> --- a/drivers/cxl/cxl.h
>> +++ b/drivers/cxl/cxl.h
>> @@ -40,6 +40,7 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
>>
>> #define CXL_CM_CAP_CAP_ID_RAS 0x2
>> #define CXL_CM_CAP_CAP_ID_HDM 0x5
>> +#define CXL_CM_CAP_CAP_ID_SNOOP 0x9
>
> Snoop filter capability is 0x8 I think.
> 0x9 is the timeout and isolation capability.
>
That's what I get for working on both of them at the same time. I'll
fix it.
>
>> #define CXL_CM_CAP_CAP_HDM_VERSION 1
>
>
>>
>> struct cxl_device_reg_map {
>> @@ -658,6 +668,8 @@ struct cxl_rcrb_info {
>> u16 aer_cap;
>> };
>>
>> +#define CXL_SNOOP_ID_NO_ID (-1)
>> +
>> /**
>> * struct cxl_dport - CXL downstream port
>> * @dport_dev: PCI bridge or firmware device representing the downstream link
>> @@ -682,6 +694,7 @@ struct cxl_dport {
>> struct access_coordinate coord[ACCESS_COORDINATE_MAX];
>> long link_latency;
>> int gpf_dvsec;
>> + int snoop_id;
>> };
>>
>> /**
>> @@ -756,6 +769,7 @@ struct cxl_dev_ops {
>> struct cxl_cache_state {
>> u64 size;
>> u32 unit;
>> + u16 snoop_id;
>> };
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 10/18] cxl/cache: Add cxl_cache driver
2025-08-19 12:11 ` Jonathan Cameron
@ 2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:01 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 7:11 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:13 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> Add the cxl_cache driver which will manage struct cxl_cachdev devices.
>> This driver will provide common management functions for some of a cache
>> capable endpoint. This driver will also be responsible for validating
>> the system's CXL.cache configuration.
>>
>> The driver expects the device's cache capabilities to be prefetched by
>> the endpoint-specific driver. The required capabilities can be gotten by
>> calling cxl_accel_get_cache_info().
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>
>
>> +static struct cxl_driver cxl_cache_driver = {
>> + .name = "cxl_cache",
>> + .probe = cxl_cache_probe,
>> + .drv = {
>> + .probe_type = PROBE_FORCE_SYNCHRONOUS,
>
> Why? I'd add a comment.
>
To match cxl_mem probe. This series is based on the probe changes Dan did
on the for-6.18/cxl-probe-order branch. I could add a comment, but it looks
like I should probably just do a better job highlighting the base for this
series in the cover letter.
>> + },
>> + .id = CXL_DEVICE_ACCELERATOR,
>> +};
>> +
>> +module_cxl_driver(cxl_cache_driver);
>> +
>> +MODULE_DESCRIPTION("CXL: Cache Management");
>> +MODULE_LICENSE("GPL");
>> +MODULE_IMPORT_NS("CXL");
>> +MODULE_ALIAS_CXL(CXL_DEVICE_ACCELERATOR);
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 12/18] cxl/cache: Add CXL Cache ID Route Table mapping
2025-08-19 15:09 ` Jonathan Cameron
@ 2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:01 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 10:09 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:15 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> The CXL Cache ID Route Table is an optional capability implemented by
>> CXL host bridges and CXL upstream switch ports. This capability is
>> required for having multiple CXL.cache devices below a port (CXL 3.2
>> 8.2.4.28).
>>
>> Add mapping and allocation of cache ids to be used by the cache id
>> route table and decoder capabilities. It's possible the id has already
>> been programmed by BIOS, in that case use the preprogrammed value.
>> Programming/validation of the capabilities by the OS will come in later
>> commits.
>
> Fine to leave this for later, but I'd not set the cached value to
> something we haven't written. Just leave it unset or error out until the
> support is there.
>
Makes sense, I'll change it.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>> ---
>> drivers/cxl/cache.c | 87 ++++++++++++++++++++++++++++++++++++++++-
>> drivers/cxl/core/regs.c | 13 ++++++
>> drivers/cxl/cxl.h | 16 ++++++++
>> 3 files changed, 115 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
>> index 7ef36e4bbce8..fd2ae1fa44bc 100644
>> --- a/drivers/cxl/cache.c
>> +++ b/drivers/cxl/cache.c
>> @@ -55,6 +55,90 @@ struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
>> }
>> EXPORT_SYMBOL_NS_GPL(devm_cxl_add_cachedev, "CXL");
>>
>> +static int find_cache_id(struct cxl_cachedev *cxlcd)
>> +{
>> + struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
>> + u32 cap, count, entry, portn;
>> + struct cxl_dport *dport;
>> + struct cxl_port *port;
>> + int id;
>> +
>> + if (cstate->cache_id != CXL_CACHE_ID_NO_ID)
>> + return cstate->cache_id;
>> +
>> + dport = cxlcd->endpoint->parent_dport;
>> + if (!dport)
>> + return -ENODEV;
>> + port = dport->port;
>> +
>> + cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
>> + count = FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK, cap);
>
> Need to do the clamping dance for when operating in 68B flit mode?
> Described in the register field definition in 82.2.4.28.1
>
Yeah, it couldn't hurt. I think the "extra" entries will just be marked invalid,
so no point checking them in that case.
>> +
>> + for (id = 0; id < count; id++) {
>
> I'd pull portn, entry and anything else you can into this scope
>
Sure
>> + entry = readl(port->regs.cidrt +
>> + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
>> + portn = FIELD_GET(CXL_CACHE_IDRT_TARGETN_PORT_MASK, entry);
>> + if (portn != dport->port_id &&
>> + !(entry & CXL_CACHE_IDRT_TARGETN_VALID))
>> + continue;
>> +
>> + cstate->cache_id = id;
>> + return 0;
>> + }
>> +
>> + id = ida_alloc(&cache_id_ida, GFP_KERNEL);
>> + if (id < 0)
>> + return id;
>
> ida_alloc_max(ida, count - 1, GFP_KERNEL)
> probably to avoid allocating an ID off the top.
For sure
>
> If some IDs were programmed by the BIOS, but not one for this
> device (maybe it turned up late) do you reserve the relevant
> values in the ida?
>
I don't think I did, oversight on my part. I'll add it.
>> +
>> + cstate->cache_id = id;
>
> Having this half state at the end of this patch makes things complex
> to reason about. If you really want to split it then return an error in
> paths you haven't handled yet.
>
Ok, I'll look at either squashing the patches or adding errors ends up simpler.
>
>> + return devm_add_action_or_reset(&cxlcd->dev, free_cache_id, cstate);
>> +}
>> +
>> +static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
>> +{
>> + struct cxl_port *endpoint = cxlcd->endpoint, *port;
>> + struct cxl_dport *dport;
>> + int rc;
>> +
>> + for (dport = endpoint->parent_dport, port = parent_port_of(endpoint);
>> + dport && port && !is_cxl_root(port);
>> + dport = port->parent_dport, port = parent_port_of(port)) {
>> + guard(device)(&port->dev);
>> + rc = map_cache_idrt_cap(port);
>> + if (rc)
>> + return rc;
>> +
>> + if (port == parent_port_of(endpoint)) {
>> + rc = find_cache_id(cxlcd);
>> + if (rc)
>> + return rc;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>
>>
>> static struct cxl_driver cxl_cache_driver = {
>> diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
>> index 0c8962fcad9b..0924127bd8fd 100644
>> --- a/drivers/cxl/core/regs.c
>> +++ b/drivers/cxl/core/regs.c
>> @@ -98,6 +98,18 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
>> length = CXL_SNOOP_FILTER_CAPABILITY_LENGTH;
>> rmap = &map->snoop;
>> break;
>> + case CXL_CM_CAP_CAP_ID_CIDRT: {
>> + int entry_cnt;
>> +
>> + dev_dbg(dev,
>> + "found Cache ID route table capability (0x%x)\n",
>> + offset);
>> + entry_cnt = FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK,
>> + hdr);
>> + length = 2 * entry_cnt + 0x010;
>
> I'd drop the leading 0 so 0x10 that will bring it in line with the macro below.
>
Yeah, it's not supposed to be there. I'll remove it.
>> + rmap = &map->cidrt;
>> + break;
>> + }
>> default:
>> dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id,
>> offset);
>> @@ -218,6 +230,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
>> { &map->component_map.hdm_decoder, ®s->hdm_decoder },
>> { &map->component_map.ras, ®s->ras },
>> { &map->component_map.snoop, ®s->snoop },
>> + { &map->component_map.cidrt, ®s->cidrt },
>> };
>> int i;
>
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 13/18] cxl/cache: Implement Cache ID Route Table programming
2025-08-19 15:07 ` Jonathan Cameron
@ 2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:01 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 10:07 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:16 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> The CXL Cache ID Route Table is an optional capability for CXL-enabled
>> upstream switch ports and CXL host bridges. This capability is required
>> for having multiple CXL.cache enabled devices under a single port
>> (CXL 3.2 8.2.4.28).
>>
>> Implement route table programming. In the case BIOS has already
>> programmed the route table(s), check the configuration for validity.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>
> Whilst it's a bit big, I'd smash this patch with the previous. They don't
> really separate as we end up with things half done but not returning errors.
>
> A lot of my initial comments on that patch were along the lines of
> 'you didn't check this' when turns out you did in this patch.
>
Yeah, I'll probably squash it (same with next two patches). I'll go back and take a look at
re-organizing this series to flow a bit better in general.
>> ---
>> drivers/cxl/cache.c | 4 ++
>> drivers/cxl/core/port.c | 135 ++++++++++++++++++++++++++++++++++++++++
>> drivers/cxl/cxl.h | 10 +++
>> drivers/cxl/cxlcache.h | 4 ++
>> 4 files changed, 153 insertions(+)
>>
>> diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
>> index fd2ae1fa44bc..24559b9ba8e8 100644
>> --- a/drivers/cxl/cache.c
>> +++ b/drivers/cxl/cache.c
>> @@ -134,6 +134,10 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
>> if (rc)
>> return rc;
>> }
>> +
>> + rc = devm_cxl_port_program_cache_idrt(port, dport, cxlcd);
>> + if (rc)
>> + return rc;
>> }
>>
>> return 0;
>> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
>> index 26ea22a2638c..f491796f6e60 100644
>> --- a/drivers/cxl/core/port.c
>> +++ b/drivers/cxl/core/port.c
>> @@ -945,6 +945,141 @@ int cxl_dport_setup_snoop_filter(struct cxl_dport *dport)
>> }
>> EXPORT_SYMBOL_NS_GPL(cxl_dport_setup_snoop_filter, "CXL");
>>
>> +static int cxl_port_commit_idrt(struct cxl_port *port)
>> +{
>> + unsigned long timeout, start, end;
>> + u32 cap, ctrl, status;
>> + int i;
>> +
>> + cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
>> + if (!(cap & CXL_CACHE_IDRT_CAP_COMMIT))
>> + return 0;
>> +
>> + ctrl = readl(port->regs.cidrt + CXL_CACHE_IDRT_CTRL_OFFSET);
>> + if (ctrl & CXL_CACHE_IDRT_CTRL_COMMIT) {
>> + ctrl &= ~CXL_CACHE_IDRT_CTRL_COMMIT;
>> + writel(ctrl, port->regs.cidrt + CXL_CACHE_IDRT_CTRL_OFFSET);
>> + }
>> +
>> + ctrl |= CXL_CACHE_IDRT_CTRL_COMMIT;
>> + writel(ctrl, port->regs.cidrt + CXL_CACHE_IDRT_CTRL_OFFSET);
>> +
>> + status = readl(port->regs.cidrt + CXL_CACHE_IDRT_STAT_OFFSET);
>> +
>> + i = FIELD_GET(CXL_CACHE_IDRT_STAT_TIMEOUT_SCALE_MASK, status);
>> + timeout = 1 * HZ;
>> + while (i-- > 3)
>> + timeout *= 10;
>
> I've no idea how that relates to the spec. Perhaps some comments.
It doesn't really, just an arbitrary timeout. I'll add a comment.
>
>> +
>> + timeout *= FIELD_GET(CXL_CACHE_IDRT_STAT_TIMEOUT_BASE_MASK, status);
>> + start = jiffies;
>> + do {
>> + status = readl(port->regs.cidrt + CXL_CACHE_IDRT_STAT_OFFSET);
>> + if (status & CXL_CACHE_IDRT_STAT_ERR_COMMIT)
>> + return -EBUSY;
>> +
>> + if (status & CXL_CACHE_IDRT_STAT_COMMITTED)
>> + return 0;
>> +
>> + end = jiffies;
>> + } while (time_before(end, start + timeout));
>> +
>> + return -ETIMEDOUT;
>> +}
>> +
>> +struct cxl_cache_idrt_entry {
>> + struct cxl_port *port;
>> + int port_num;
>> + int id;
>> +};
>> +
>> +static int cxl_port_enable_idrt_entry(struct cxl_cache_idrt_entry *entry)
>> +{
>> + struct cxl_port *port = entry->port;
>> + int id = entry->id, port_num;
>> + u32 entry_n;
>> +
>> + entry_n = readl(port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
>
> It's 16bit so readw?
>
Yep, will fix.
>> + if (entry_n & CXL_CACHE_IDRT_TARGETN_VALID) {
>> + port_num = FIELD_GET(CXL_CACHE_IDRT_TARGETN_PORT_MASK, entry_n);
>> + if (port_num != entry->port_num) {
>> + dev_err(&port->dev,
>> + "Cache ID Route Table entry%d: Invalid port id %d, expected %d\n",
>> + id, port_num, entry->port_num);
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> + }
>> +
>> + entry_n &= ~CXL_CACHE_IDRT_TARGETN_PORT_MASK;
>
> Maybe just start from 0. There aren't any other fields as long as we stick
> to not reading and writing neighboring entry.
>
Makes sense, I'll change it.
>> + entry_n |= FIELD_PREP(CXL_CACHE_IDRT_TARGETN_PORT_MASK, entry->port_num);
>> + entry_n |= CXL_CACHE_IDRT_TARGETN_VALID;
>> + writel(entry_n, port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
>
> writew()
>
>> +
>> + return cxl_port_commit_idrt(port);
>> +}
>> +
>> +static void free_cache_idrt_entry(void *data)
>> +{
>> + struct cxl_cache_idrt_entry *entry = data;
>> + struct cxl_port *port = entry->port;
>> + int id = entry->id;
>> + u32 cap, entry_n;
>> +
>> + cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
>> + if (FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK, cap) < id)
>
> How could this happen?
>
I'll double check, but it probably can't. I'll remove if so.
>> + return;
>> +
>> + entry_n = readl(port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
>> + entry_n &= ~CXL_CACHE_IDRT_TARGETN_VALID;
>> + writel(entry_n, port->regs.cidrt + CXL_CACHE_IDRT_TARGETN_OFFSET(id));
>> +
>> + cxl_port_commit_idrt(port);
>> +
>> + kfree(entry);
>> +}
>> +
>> +int devm_cxl_port_program_cache_idrt(struct cxl_port *port,
>> + struct cxl_dport *dport,
>> + struct cxl_cachedev *cxlcd)
>> +{
>> + struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
>> + int id = cstate->cache_id, rc;
>
> I'm always moaning about these but to me combining assignment and
> non assignment declarations just ends up hiding stuff.
> I'd always burn a line to keep them separate.
>
Sure.
>> + u32 cap, max_hdmd;
>> +
>> + cap = readl(port->regs.cidrt + CXL_CACHE_IDRT_CAP_OFFSET);
>> + if (FIELD_GET(CXL_CACHE_IDRT_CAP_CNT_MASK, cap) < id)
>
> Check if in 68B flit mode as if we are this number may need capping
> to a much lower level (field description talks about this.
>
Yep, will do.
>> + return -EINVAL;
>> +
>> + if (is_cxl_root(parent_port_of(port))) {
>> + max_hdmd = FIELD_GET(CXL_CACHE_IDRT_CAP_T2_MAX_MASK, cap);
>> + if (cxlcd->cxlds->type == CXL_DEVTYPE_DEVMEM && cstate->hdmd)
>> + port->nr_hdmd++;
>> +
>> + if (port->nr_hdmd > max_hdmd) {
>> + port->nr_hdmd--;
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + struct cxl_cache_idrt_entry *entry __free(kfree) =
>> + kzalloc(sizeof(*entry), GFP_KERNEL);
>> + if (!entry)
>> + return -ENXIO;
>> +
>> + entry->port_num = dport->port_id;
>> + entry->port = port;
>> + entry->id = id;
>> +
>> + rc = cxl_port_enable_idrt_entry(entry);
>> + if (rc)
>> + return rc;
>> +
>> + return devm_add_action(&cxlcd->dev, free_cache_idrt_entry, entry);
>
> At this point entry is freed. So by the time that action gets called
> it's a use after free. Maybe also need the _or_reset() variant
> in case this call itself fails.
>
I'll fix it and change to _or_reset() as well.
>> +}
>> +EXPORT_SYMBOL_NS_GPL(devm_cxl_port_program_cache_idrt, "CXL");
>
>> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
>> index c147846855e2..1a2918aaee62 100644
>> --- a/drivers/cxl/cxl.h
>> +++ b/drivers/cxl/cxl.h
>> @@ -163,8 +163,15 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
>> /* CXL 3.2 8.2.4.28 CXL Cache ID Route Table Capability Structure */
>> #define CXL_CACHE_IDRT_CAP_OFFSET 0x0
>> #define CXL_CACHE_IDRT_CAP_CNT_MASK GENMASK(4, 0)
>> +#define CXL_CACHE_IDRT_CAP_T2_MAX_MASK GENMASK(11, 8)
>> +#define CXL_CACHE_IDRT_CAP_COMMIT BIT(16)
>
> I'd get explicit in that name. It is an odd bit of design to have
> the option for device to not require it so making it clear that
> the absense of this bit doesn't mean it never makes the entrees
> active / committed. Maybe REQUIRES_COMMIT
>
I agree, I'll change it. Some of these names get a bit long, so I was probably
trying to cut down on the length...
>> +#define CXL_CACHE_IDRT_CTRL_OFFSET 0x4
>> +#define CXL_CACHE_IDRT_CTRL_COMMIT BIT(0)
>> #define CXL_CACHE_IDRT_STAT_OFFSET 0x8
>> #define CXL_CACHE_IDRT_STAT_COMMITTED BIT(0)
>> +#define CXL_CACHE_IDRT_STAT_ERR_COMMIT BIT(0)
>
> Nope. They aren't both BIT(0)
Woops, thanks for pointing that out.
>
>> +#define CXL_CACHE_IDRT_STAT_TIMEOUT_SCALE_MASK GENMASK(11, 8)
>> +#define CXL_CACHE_IDRT_STAT_TIMEOUT_BASE_MASK GENMASK(15, 12)
>> #define CXL_CACHE_IDRT_TARGETN_OFFSET(n) (0x10 + (2 * (n)))
>> #define CXL_CACHE_IDRT_TARGETN_VALID BIT(0)
>> #define CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
>> @@ -618,6 +625,7 @@ struct cxl_dax_region {
>> * @cdat: Cached CDAT data
>> * @cdat_available: Should a CDAT attribute be available in sysfs
>> * @pci_latency: Upstream latency in picoseconds
>> + * @nr_hdmd: Number of HDM-D devices below port
>> */
>> struct cxl_port {
>> struct device dev;
>> @@ -643,6 +651,7 @@ struct cxl_port {
>> } cdat;
>> bool cdat_available;
>> long pci_latency;
>> + int nr_hdmd;
>> };
>>
>> /**
>> @@ -786,6 +795,7 @@ struct cxl_cache_state {
>> u32 unit;
>> u16 snoop_id;
>> int cache_id;
>> + bool hdmd;
>> };
>>
>> /**
>> diff --git a/drivers/cxl/cxlcache.h b/drivers/cxl/cxlcache.h
>> index c4862f7e6edc..d28222123102 100644
>> --- a/drivers/cxl/cxlcache.h
>> +++ b/drivers/cxl/cxlcache.h
>> @@ -28,4 +28,8 @@ struct cxl_cachedev *devm_cxl_add_cachedev(struct device *host,
>>
>> int cxl_dport_setup_snoop_filter(struct cxl_dport *dport);
>> int devm_cxl_snoop_filter_alloc(u32 gid, struct cxl_dev_state *cxlds);
>> +
>> +int devm_cxl_port_program_cache_idrt(struct cxl_port *port,
>> + struct cxl_dport *dport,
>> + struct cxl_cachedev *cxlcd);
>> #endif
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 14/18] cxl/cache: Add Cache ID Decoder capability mapping
2025-08-19 14:12 ` Alireza Sanaee
@ 2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:01 UTC (permalink / raw)
To: Alireza Sanaee; +Cc: linux-cxl
On 8/19/2025 9:12 AM, Alireza Sanaee wrote:
> On Tue, 12 Aug 2025 16:29:17 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> The CXL Cache ID Decoder capability is an optional capability present
>> on CXL downstream switch and CXL-enabled PCIe Root ports. This
>> capability is required for having multiple CXL.cache enabled devices
>> under the same port (CXL 3.2 8.2.4.29).
>>
>> Add mapping of the capability as part of allocating a CXL cache id for
>> an endpoint. Implement mapping/validation of the capability in a later
>> commit.
>
> Maybe you keep ID or Cache or Decoder consistent everywhere in the
> commit message.
The initial mention was to match the spec, but I don't mind unifying it all.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>> ---
>> drivers/cxl/cache.c | 25 +++++++++++++++++++++++++
>> drivers/cxl/core/regs.c | 8 ++++++++
>> drivers/cxl/cxl.h | 6 ++++++
>> 3 files changed, 39 insertions(+)
>>
>> diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
>> index 24559b9ba8e8..f123d596187d 100644
>> --- a/drivers/cxl/cache.c
>> +++ b/drivers/cxl/cache.c
>> @@ -69,6 +69,27 @@ static int map_cache_idrt_cap(struct cxl_port
>> *port) BIT(CXL_CM_CAP_CAP_ID_CIDRT));
>> }
>>
>> +static int map_cache_idd_cap(struct cxl_dport *dport)
>> +{
>> + int agents;
>> +
>> + if (dport->regs.cidd)
>> + return 0;
>> +
>> + /*
>> + * A missing Cache ID Decoder capability is only an issue
>> + * if there are multiple cache agents in the VCS
>> + */
>> + if (!dport->reg_map.component_map.cidd.valid) {
>> + agents = atomic_read(&dport->port->cache_agents);
>> + return agents > 1 ? -ENXIO : 0;
>> + }
>> +
>> + return cxl_map_component_regs(&dport->reg_map,
>> + &dport->regs.component,
>> + BIT(CXL_CM_CAP_CAP_ID_CIDD));
> Less relevant to this patch. It looks like cxl_map_component_reg
> receives redundant parameters, both regs.component and reg_map comes
> from the same structure and other occurrences in the kernel also grab
> components and reg_map from port, so not sure why two separate
> parameters are there.
I *think* they used to be separate and it's a relatively recent change to
store the parameters together under the same port/dport. Good point though,
someone could probably take a look at the usage and simplify things accordingly.
>> +}
>> +
>> static void free_cache_id(void *data)
>> {
>> struct cxl_cache_state *cstate = data;
>> @@ -138,6 +159,10 @@ static int
>> devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd) rc =
>> devm_cxl_port_program_cache_idrt(port, dport, cxlcd); if (rc)
>> return rc;
>> +
>> + rc = map_cache_idd_cap(dport);
>> + if (rc)
>> + return rc;
>> }
>>
>> return 0;
>> diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
>> index 0924127bd8fd..1362f4156ee6 100644
>> --- a/drivers/cxl/core/regs.c
>> +++ b/drivers/cxl/core/regs.c
>> @@ -110,6 +110,13 @@ void cxl_probe_component_regs(struct device
>> *dev, void __iomem *base, rmap = &map->cidrt;
>> break;
>> }
>> + case CXL_CM_CAP_CAP_ID_CIDD:
>> + dev_dbg(dev,
>> + "found Cache ID decoder capability
>> (0x%x)\n",
>> + offset);
>> + length = CXL_CACHE_IDD_CAPABILITY_LENGTH;
>> + rmap = &map->cidd;
>> + break;
>> default:
>> dev_dbg(dev, "Unknown CM cap ID: %d
>> (0x%x)\n", cap_id, offset);
>> @@ -231,6 +238,7 @@ int cxl_map_component_regs(const struct
>> cxl_register_map *map, { &map->component_map.ras, ®s->ras },
>> { &map->component_map.snoop, ®s->snoop },
>> { &map->component_map.cidrt, ®s->cidrt },
>> + { &map->component_map.cidd, ®s->cidd },
>> };
>> int i;
>>
>> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
>> index 1a2918aaee62..04fde1a994d0 100644
>> --- a/drivers/cxl/cxl.h
>> +++ b/drivers/cxl/cxl.h
>> @@ -42,6 +42,7 @@ extern const struct nvdimm_security_ops
>> *cxl_security_ops; #define CXL_CM_CAP_CAP_ID_HDM 0x5
>> #define CXL_CM_CAP_CAP_ID_SNOOP 0x9
>> #define CXL_CM_CAP_CAP_ID_CIDRT 0xD
>> +#define CXL_CM_CAP_CAP_ID_CIDD 0xE
>> #define CXL_CM_CAP_CAP_HDM_VERSION 1
>>
>> /* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability
>> Structure */ @@ -176,6 +177,9 @@ static inline int
>> ways_to_eiw(unsigned int ways, u8 *eiw) #define
>> CXL_CACHE_IDRT_TARGETN_VALID BIT(0) #define
>> CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
>> +/* CXL 3.2 8.2.4.29 CXL Cache ID Decoder Capability Structure */
>> +#define CXL_CACHE_IDD_CAPABILITY_LENGTH 0xC
>> +
>> /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
>> #define CXLDEV_CAP_ARRAY_OFFSET 0x0
>> #define CXLDEV_CAP_ARRAY_CAP_ID 0
>> @@ -241,6 +245,7 @@ struct cxl_regs {
>> void __iomem *ras;
>> void __iomem *snoop;
>> void __iomem *cidrt;
>> + void __iomem *cidd;
>> );
>> /*
>> * Common set of CXL Device register block base pointers
>> @@ -285,6 +290,7 @@ struct cxl_component_reg_map {
>> struct cxl_reg_map ras;
>> struct cxl_reg_map snoop;
>> struct cxl_reg_map cidrt;
>> + struct cxl_reg_map cidd;
>> };
>>
>> struct cxl_device_reg_map {
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming
2025-08-19 15:26 ` Jonathan Cameron
@ 2025-08-22 18:01 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:01 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 10:26 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:18 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> The CXL Cache ID Decoder capability is an optional capability present on
>> CXL downstream switch and CXL-enabled PCIe Root ports. This capability
>> is required for having multiple CXL.cache enabled devices under the same
>> port (CXL 3.2 8.2.4.29).
>>
>> Implement cache id decoder programming. Validate the configuration in the
>> case BIOS already programmed the id(s).
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> Comments from earlier on how the timeout is calculate apply here as well.
> A few other things inline.
>
> Thanks,
>
> Jonathan
>
>> ---
>> drivers/cxl/cache.c | 4 ++
>> drivers/cxl/core/port.c | 109 ++++++++++++++++++++++++++++++++++++++++
>> drivers/cxl/cxl.h | 14 ++++++
>> drivers/cxl/cxlcache.h | 3 ++
>> 4 files changed, 130 insertions(+)
>>
>> diff --git a/drivers/cxl/cache.c b/drivers/cxl/cache.c
>> index f123d596187d..fa3223165a18 100644
>> --- a/drivers/cxl/cache.c
>> +++ b/drivers/cxl/cache.c
>> @@ -163,6 +163,10 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
>> rc = map_cache_idd_cap(dport);
>> if (rc)
>> return rc;
>> +
>> + rc = cxl_dport_program_cache_idd(dport, cxlcd);
>> + if (rc)
>> + return rc;
>> }
>>
>> return 0;
>> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
>> index f491796f6e60..3606ad557bfb 100644
>> --- a/drivers/cxl/core/port.c
>> +++ b/drivers/cxl/core/port.c
>
>
>> +static int cxl_cache_idd_alread_programmed(struct cxl_dport *dport,
> already?
Yep. Could probably do with a rename though, it's a bit weird...
>> + struct cxl_cachedev *cxlcd)
>> +{
>> + struct cxl_cache_state *cstate = &cxlcd->cxlds->cstate;
>> + u32 ctrl, status, cache_id;
>> +
>> + status = readl(dport->regs.cidd + CXL_CACHE_IDD_STAT_OFFSET);
>> + if (!(status & CXL_CACHE_IDD_STAT_COMMITTED))
>> + return 0;
>> +
>> + /*
>> + * The decoder is probably already committed by BIOS, validate
>> + * the configuration
>> + */
>> + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
>> + /* TODO: Check for 68B flit mode */
>> + if (dport == cxlcd->endpoint->parent_dport &&
>> + !(ctrl & CXL_CACHE_IDD_CTRL_ASGN_ID))
>> + return -EINVAL;
>> +
>> + if (dport != cxlcd->endpoint->parent_dport &&
>> + !(ctrl & CXL_CACHE_IDD_CTRL_FWD_ID))
>> + return -EINVAL;
>> +
>> + cache_id = FIELD_GET(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK, ctrl);
>> + if (cache_id != cstate->cache_id)
>> + return -EINVAL;
>> +
>> + if (!!(ctrl & CXL_CACHE_IDD_CTRL_TYPE2) != cstate->hdmd)
>> + return -EINVAL;
>> +
>> + return 1;
>> +}
>> +
>> +int cxl_dport_program_cache_idd(struct cxl_dport *dport,
>> + struct cxl_cachedev *cxlcd)
>> +{
>> + u32 ctrl;
>> + int rc;
>> +
>> + rc = cxl_cache_idd_alread_programmed(dport, cxlcd);
>> + if (rc)
>> + return rc < 0 ? rc : 0;
>> +
>> + ctrl = readl(dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
>> +
>> + if (cxlcd->cxlds->cstate.hdmd) {
>> + ctrl |= CXL_CACHE_IDD_CTRL_TYPE2;
>> + ctrl &= ~CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK;
>> + ctrl |= FIELD_PREP(CXL_CACHE_IDD_CTRL_TYPE2_ID_MASK, cxlcd->id);
>> + }
>> +
>> + /* TODO: Check for 68B flit mode */
>> + if (dport == cxlcd->endpoint->parent_dport)
>> + ctrl |= CXL_CACHE_IDD_CTRL_ASGN_ID;
>> + else
>> + ctrl |= CXL_CACHE_IDD_CTRL_FWD_ID;
>> +
>> + ctrl &= ~CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK;
>> + ctrl |= FIELD_PREP(CXL_CACHE_IDD_CTRL_LOCAL_ID_MASK,
>> + cxlcd->cxlds->cstate.cache_id);
>> +
>> + writel(ctrl, dport->regs.cidd + CXL_CACHE_IDD_CTRL_OFFSET);
>> + return cxl_dport_commit_idd(dport);
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_dport_program_cache_idd, "CXL");
>
>> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
>> index 04fde1a994d0..9c475d8c1573 100644
>> --- a/drivers/cxl/cxl.h
>> +++ b/drivers/cxl/cxl.h
>> @@ -178,6 +178,20 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
>> #define CXL_CACHE_IDRT_TARGETN_PORT_MASK GENMASK(15, 8)
>>
>> /* CXL 3.2 8.2.4.29 CXL Cache ID Decoder Capability Structure */
>> +#define CXL_CACHE_IDD_CAP_OFFSET 0x0
>> +#define CXL_CACHE_IDD_CAP_COMMIT BIT(0)
>
> Similar to before I'd name this something to indicate it's about
> it being explicit / required by this device for things
> to work.
>
Yeah, will do.
>> +#define CXL_CACHE_IDD_CTRL_OFFSET 0x4
>> +#define CXL_CACHE_IDD_CTRL_FWD_ID BIT(0)
>
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 16/18] cxl/cache: Add cache device counting for CXL ports
2025-08-19 15:30 ` Jonathan Cameron
@ 2025-08-22 18:02 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:02 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 10:30 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:19 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> Having more than one CXL.cache enabled device under a port requires CXL
>> cache id capabilities (CXL 3.2 8.2.4.28/29).
>>
>> Add tracking of how many cache devices are under a port. If the required
>> capabilities are absent and more than one device is enabled under the
>> port, fail allocation of a cache id.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>
> ...
>
>>
>> +static void cxl_deactivate_cache(void *data)
>> +{
>> + struct cxl_cachedev *cxlcd = data;
>> + struct cxl_port *port = cxlcd->endpoint;
>> +
>> + for (port = cxlcd->endpoint; port && !is_cxl_root(port);
>> + port = parent_port_of(port)) {
>> + scoped_guard(device, &port->dev) {
>
> I think you need to check this succeeded if the aim is to ensure
> that port didn't go away before this decrement.
> It might have gone away just before we tried to take the guard.
>
I see your point here, I'll change it. I don't think I was even considering
if the port still existed, this was just for concurrent access if multiple devices were
leaving at once.
>> + port->cache_devs--;
>> + }
>> + }
>> +}
>> +
>> static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
>> {
>> struct cxl_port *endpoint = cxlcd->endpoint, *port;
>> @@ -167,9 +176,17 @@ static int devm_cxl_cachedev_allocate_cache_id(struct cxl_cachedev *cxlcd)
>> rc = cxl_dport_program_cache_idd(dport, cxlcd);
>> if (rc)
>> return rc;
>> +
>> + port->cache_devs++;
>> }
>>
>> - return 0;
>> + if (!is_cxl_root(port)) {
>> + cxl_deactivate_cache(cxlcd);
>> + return -ENODEV;
>> + }
>> +
>> + return devm_add_action_or_reset(&cxlcd->dev, cxl_deactivate_cache,
>> + cxlcd);
>> }
>>
>> static int find_snoop_gid(struct cxl_cachedev *cxlcd, u32 *gid)
>> @@ -239,7 +256,6 @@ static int cxl_cache_probe(struct device *dev)
>> if (rc)
>> return rc;
>>
>> -
>
> No comment. :)
>
>> return devm_cxl_cachedev_allocate_cache_id(cxlcd);
>> }
>>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 17/18] cxl/core: Add cache device attributes
2025-08-19 15:38 ` Jonathan Cameron
@ 2025-08-22 18:02 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:02 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 10:38 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:20 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> Add sysfs attributes for getting the serial number, numa node, CXL cache
>> unit, and CXL cache size to struct cxl_cachedev.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
> A few minor comments inline.
>
>> ---
>> drivers/cxl/core/cachedev.c | 102 ++++++++++++++++++++++++++++++++++++
>> 1 file changed, 102 insertions(+)
>>
>> diff --git a/drivers/cxl/core/cachedev.c b/drivers/cxl/core/cachedev.c
>> index aaf4a1b94178..1cdd94520ceb 100644
>> --- a/drivers/cxl/core/cachedev.c
>> +++ b/drivers/cxl/core/cachedev.c
>> @@ -1,5 +1,6 @@
>> // SPDX-License-Identifier: GPL-2.0-only
>> /* Copyright (C) 2025 Advanced Micro Devices, Inc. */
>> +#include <linux/string_helpers.h>
>> #include <linux/device.h>
>> #include <linux/pci.h>
>>
>> @@ -30,10 +31,111 @@ static char *cxl_cachedev_devnode(const struct device *dev, umode_t *mode,
>> return kasprintf(GFP_KERNEL, "cxl/%s", dev_name(dev));
>> }
>>
>> +static ssize_t serial_show(struct device *dev, struct device_attribute *attr,
>> + char *buf)
>> +{
>> + struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
>> + struct cxl_dev_state *cxlds = cxlcd->cxlds;
>> +
>> + return sysfs_emit(buf, "%#llx\n", cxlds->serial);
> Just as an FYI there is a discussion about putting similar in place for the pci device.
>
> https://lore.kernel.org/linux-pci/20250811173931.8068-2-thepacketgeek@gmail.com/
Wasn't aware of that, I'll probably remove this one then.
>
>> +}
>> +static DEVICE_ATTR_RO(serial);
>> +
>> +static ssize_t numa_node_show(struct device *dev, struct device_attribute *attr,
>> + char *buf)
>> +{
>> + return sysfs_emit(buf, "%d\n", dev_to_node(dev));
>
> I grumped about this when it was added for the memdev (duplication) but lost
> then so I'll give in this time without a fight :)
>
I don't mind removing it honestly, it's just there for parity with cxl_memdev. It really
confuses people who are debugging CXL memory in the terminal but don't know much about the
driver in my experience.
>> +}
>> +static DEVICE_ATTR_RO(numa_node);
>> +
>> +static struct attribute *cxl_cachedev_attributes[] = {
>> + &dev_attr_serial.attr,
>> + &dev_attr_numa_node.attr,
>
> Missing NULL?
>
Yeah, will add it.
>> +};
>> +
>> +static umode_t cxl_cachedev_visible(struct kobject *kobj, struct attribute *a,
>> + int n)
>> +{
>> + if (!IS_ENABLED(CONFIG_NUMA) && a == &dev_attr_numa_node.attr)
>> + return 0;
>> + return a->mode;
>> +}
>
>> +static DEVICE_ATTR_RO(cache_unit);
>> +
>> +static struct attribute *cxl_cachedev_cache_attributes[] = {
>> + &dev_attr_cache_size.attr,
>> + &dev_attr_cache_unit.attr,
>> + NULL,
>
> No comma.
>
>> +};
>
>> +
>> +static const struct attribute_group *cxl_cachedev_attribute_groups[] = {
>> + &cxl_cachedev_attribute_group,
>> + &cxl_cachedev_cache_attribute_group,
>> + NULL,
>
> No trailing comma needed given it's a terminator.
>
Will fix (with above as well).
>> +};
>> +
>> static const struct device_type cxl_cachedev_type = {
>> .name = "cxl_cachedev",
>> .release = cxl_cachedev_release,
>> .devnode = cxl_cachedev_devnode,
>> + .groups = cxl_cachedev_attribute_groups,
>> };
>>
>> bool is_cxl_cachedev(const struct device *dev)
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 18/18] cxl/core: Add cache device cache management attributes
2025-08-19 15:53 ` Jonathan Cameron
@ 2025-08-22 18:02 ` Cheatham, Benjamin
0 siblings, 0 replies; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:02 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: linux-cxl
On 8/19/2025 10:53 AM, Jonathan Cameron wrote:
> On Tue, 12 Aug 2025 16:29:21 -0500
> Ben Cheatham <Benjamin.Cheatham@amd.com> wrote:
>
>> Add functions to cxl/core/pci.c to manage the cache of a CXL.cache
>> enabled endpoint. Add these new functions to sysfs attributes for
>> userspace accessibility.
>>
>> Signed-off-by: Ben Cheatham <Benjamin.Cheatham@amd.com>
>
> Documentation/ABI needs something on this.
Yeah I'll add it. Skimped on the docs since it's just RFC at the moment.
>
>> ---
>> drivers/cxl/core/cachedev.c | 99 +++++++++++++++++++++++++++++++++++++
>> drivers/cxl/core/pci.c | 89 +++++++++++++++++++++++++++++++++
>> drivers/cxl/cxlcache.h | 4 ++
>> drivers/cxl/cxlpci.h | 6 +++
>> 4 files changed, 198 insertions(+)
>>
>> diff --git a/drivers/cxl/core/cachedev.c b/drivers/cxl/core/cachedev.c
>> index 1cdd94520ceb..85989cdd1eb8 100644
>> --- a/drivers/cxl/core/cachedev.c
>> +++ b/drivers/cxl/core/cachedev.c
>> @@ -5,6 +5,7 @@
>> #include <linux/pci.h>
>>
>> #include "../cxlcache.h"
>> +#include "../cxlpci.h"
>> #include "private.h"
>>
>> static DEFINE_IDA(cxl_cachedev_ida);
>> @@ -125,9 +126,107 @@ static struct attribute_group cxl_cachedev_cache_attribute_group = {
>> .is_visible = cxl_cachedev_cache_visible,
>> };
>>
>> +static ssize_t cache_disable_store(struct device *dev,
>> + struct device_attribute *attr,
>> + const char *buf, size_t n)
>> +{
>> + struct cxl_cachedev *cxlcd = to_cxl_cachedev(dev);
>> + struct cxl_dev_state *cxlds = cxlcd->cxlds;
>> + bool disable;
>> + int rc;
>> +
>> + rc = kstrtobool(buf, &disable);
>> + if (rc)
>> + return rc;
>> +
>> + rc = cxl_accel_set_cache_disable(cxlds, disable);
>> + if (rc)
>> + return rc;
>
> Can return 1 which is unlikely to be what want.
>
You are right, will fix it.
>> +
>> + return n;
>> +}
>
>> +
>> +static struct attribute *cxl_cachedev_mgmt_attributes[] = {
>> + &dev_attr_cache_disable.attr,
>> + &dev_attr_cache_invalid.attr,
>> + &dev_attr_init_wbinvd.attr,
>> + NULL,
>
> No comma.
>
>> +};
>
>
>> +static struct attribute_group cxl_cachedev_mgmt_attribute_group = {
>> + .attrs = cxl_cachedev_mgmt_attributes,
>> + .is_visible = cxl_cachedev_mgmt_visible,
>> +};
>> +
>> static const struct attribute_group *cxl_cachedev_attribute_groups[] = {
>> &cxl_cachedev_attribute_group,
>> &cxl_cachedev_cache_attribute_group,
>> + &cxl_cachedev_mgmt_attribute_group,
>> NULL,
> No comma
>
I'll remove it (same as above)
>> };
>>
>> diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
>> index 2279c2690c59..667f75043f9e 100644
>> --- a/drivers/cxl/core/pci.c
>> +++ b/drivers/cxl/core/pci.c
>> @@ -1232,3 +1232,92 @@ int cxl_accel_get_cache_info(struct cxl_dev_state *cxlds)
>> return 0;
>> }
>> EXPORT_SYMBOL_NS_GPL(cxl_accel_get_cache_info, "CXL");
>> +
>> +int cxl_accel_caching_disabled(struct cxl_dev_state *cxlds)
>> +{
>> + struct pci_dev *pdev = to_pci_dev(cxlds->dev);
>> + int dvsec, rc;
>> + u16 ctrl2;
>> +
>> + device_lock_assert(cxlds->dev);
> What took this and why do we need to check?
It's taken by the function below and the cache invalidate function. Though looking at the
patch, I put the guard in cxl_accel_initiate_wbinvd() in the wrong spot, it should be right
before calling this function.
I'm checking this because it's exported and I expect it to be useful for type 1/2 vendor
drivers.
>> +
>> + if (!dev_is_pci(cxlds->dev))
>> + return -EINVAL;
>> + pdev = to_pci_dev(cxlds->dev);
>> +
>> + dvsec = cxlds->cxl_dvsec;
>> + rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &ctrl2);
>> + if (rc)
>> + return rc;
>> +
>> + return !!(ctrl2 & CXL_DVSEC_DISABLE_CACHING);
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_accel_caching_disabled, "CXL");
>> +
>> +int cxl_accel_set_cache_disable(struct cxl_dev_state *cxlds, bool disable)
>> +{
>> + struct pci_dev *pdev = to_pci_dev(cxlds->dev);
>> + int dvsec, rc;
>> + u16 ctrl2;
>> +
>> + if (!dev_is_pci(cxlds->dev))
>> + return -EINVAL;
>> +
>> + guard(device)(cxlds->dev);
>> + dvsec = cxlds->cxl_dvsec;
>> + rc = pci_read_config_word(pdev, dvsec + CXL_DVSEC_CTRL2_OFFSET, &ctrl2);
>> + if (rc)
>> + return rc;
>> +
>> + if (!!(ctrl2 & CXL_DVSEC_DISABLE_CACHING) == disable)
>
> Maybe FIELD_GET() is cleaner than the !!
Sure, I'll change it.
Thanks,
Ben
>
>> + return 1;
>> +
>> + ctrl2 &= ~CXL_DVSEC_DISABLE_CACHING;
>> + ctrl2 |= FIELD_PREP(CXL_DVSEC_DISABLE_CACHING, disable);
>> + return pci_write_config_word(pdev, dvsec + CXL_DVSEC_DISABLE_CACHING,
>> + ctrl2);
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_accel_set_cache_disable, "CXL");
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 00/18] Initial CXL.cache device support
2025-08-13 11:25 ` [RFC PATCH 00/18] Initial CXL.cache device support Alejandro Lucero Palau
2025-08-19 15:57 ` Jonathan Cameron
@ 2025-08-22 18:02 ` Cheatham, Benjamin
2025-08-26 10:44 ` Alejandro Lucero Palau
1 sibling, 1 reply; 55+ messages in thread
From: Cheatham, Benjamin @ 2025-08-22 18:02 UTC (permalink / raw)
To: Alejandro Lucero Palau, linux-cxl
On 8/13/2025 6:25 AM, Alejandro Lucero Palau wrote:
> On 8/12/25 22:29, Ben Cheatham wrote:
>> This patch series adds initial CXL.cache support. What I have here only
>> allows for adding a cache device to the system, programming/validating
>> the system configuration with respect to cache devices, and some basic
>> cache reporting/management.
>>
>> The general philosophy is to have an endpoint/vendor-specific driver
>> that runs through the same steps of adding a cxl_memdev, but for the
>> cache portion of the device (both type 1 & 2). Getting cache support for
>> a CXL device should be as simple as: get cache information, set up the
>> memory region (see below), and then calling devm_cxl_add_cachedev().
>>
>> There's a couple of things missing from this set:
>>
>> 1) Missing an endpoint driver
>>
>> I plan on submitting a reference driver (type 1 or 2) with v1, I
>> figured I'd send out what I have before going much further.
>>
>> 2) Mapping/Reserving host memory used for CXL cache(s)
>>
>> I'm thinking this will be handled by the endpoint driver, but I'm
>> not sure what mechanism would be used and whether to integrate it
>> with the cxl_cache/core. Any thoughts/ideas here are appreciated!
>
>
> Hi Ben,
>
>
> Good to see you working on this. I'll go through the series these next days.
>
>
> FWIW, I did send a RFC last November about my concerns for properly/safely supporting CXL.cache.
Yeah, I saw Jonathan linked it in his reply as well. I've read it but didn't really have anything to say about safety/iommu schemes.
This set is really just for the first part of the summary at the end of that email ("CXL core handling per device CXL.cache enabling...").
>
>
> I had no time to keep thinking about it. Also, I sent a copy to the kernel iommu mailing list, but it did not get traction then. Anyways, as a summary, I think mapping for CXL.cache requires not just the endpoint driver doing it freely but through the cxl core for ensuring it is done properly and safely. But I have to admit this could be a misunderstanding about how the hardware is going to allow CXL.cache.
I agree! I just wasn't sure of the approach to take here so I opted to leave it out until we came to a consensus.
Thanks,
Ben
>
>
> Thanks
>
>
>> 3) RAS Support
>>
>> Same situation as 1) above.
>>
>> Some quick notes: The actual cache parts of this set are untested due
>> to problems with my set up, but I did make sure nothing here breaks type
>> 3 support. I'll have this fixed before sending out a v1. This series is
>> based on the for-6.18/cxl-probe-order (commit 5e29cbd1077b) branch in
>> the CXL repo, with v7 of Dave's deferred dport probe set on top [1]. I
>> added Dave's set to help with getting around constraints with HDM
>> decoders in CXL.cache device only configurations (see 08/18 for more).
>>
>> Patch Breakdown:
>> - 1 & 2: Preliminary changes for struct cxl_cachedev
>> - 3: Add struct cxl_cachedev
>> - 4-8: Preliminary changes for adding cache devices to port
>> hierarchy
>> - 9: Function for getting CXL cache info
>> - 10: cxl_cache driver (mirrors cxl_mem)
>> - 11-16: Checking CXL.cache capabilities for system configuration
>> validity
>> - 17-18: Cache device attributes
>>
>> [1]:
>> Link: https://lore.kernel.org/linux-cxl/20250714223527.461147-1-dave.jiang@intel.com/
>>
>> Ben Cheatham (18):
>> cxl/mem: Change cxl_memdev_ops to cxl_dev_ops
>> cxl: Move struct cxl_dev_state definition
>> cxl/core: Add CXL.cache device struct
>> cxl: Replace cxl_mem_find_port() with cxl_dev_find_port()
>> cxl: Change cxl_ep_load() to use struct device * parameter
>> cxl/port, mem: Make adding an endpoint device type agnostic
>> cxl/port: Split endpoint port probe on device type
>> cxl/port: Update switch_port_probe() for CXL cache devices
>> cxl/core: Add function for getting CXL cache info
>> cxl/cache: Add cxl_cache driver
>> cxl/core: Add CXL snoop filter setup and checking
>> cxl/cache: Add CXL Cache ID Route Table mapping
>> cxl/cache: Implement Cache ID Route Table programming
>> cxl/cache: Add Cache ID Decoder capability mapping
>> cxl/cache: Implement Cache ID Decoder programming
>> cxl/cache: Add cache device counting for CXL ports
>> cxl/core: Add cache device attributes
>> cxl/core: Add cache device cache management attributes
>>
>> drivers/cxl/Kconfig | 14 +
>> drivers/cxl/Makefile | 2 +
>> drivers/cxl/cache.c | 276 +++++++++++++++++++
>> drivers/cxl/core/Makefile | 1 +
>> drivers/cxl/core/cachedev.c | 292 ++++++++++++++++++++
>> drivers/cxl/core/hdm.c | 31 +++
>> drivers/cxl/core/memdev.c | 2 +-
>> drivers/cxl/core/pci.c | 134 ++++++++++
>> drivers/cxl/core/port.c | 518 +++++++++++++++++++++++++++++++++---
>> drivers/cxl/core/region.c | 25 +-
>> drivers/cxl/core/regs.c | 28 ++
>> drivers/cxl/cxl.h | 193 +++++++++++++-
>> drivers/cxl/cxlcache.h | 42 +++
>> drivers/cxl/cxlmem.h | 121 +--------
>> drivers/cxl/cxlpci.h | 10 +
>> drivers/cxl/mem.c | 14 +-
>> drivers/cxl/pci.c | 2 +-
>> drivers/cxl/port.c | 54 ++--
>> drivers/cxl/private.h | 8 +-
>> 19 files changed, 1569 insertions(+), 198 deletions(-)
>> create mode 100644 drivers/cxl/cache.c
>> create mode 100644 drivers/cxl/core/cachedev.c
>> create mode 100644 drivers/cxl/cxlcache.h
>>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 00/18] Initial CXL.cache device support
2025-08-19 16:05 ` Jonathan Cameron
@ 2025-08-26 10:42 ` Alejandro Lucero Palau
0 siblings, 0 replies; 55+ messages in thread
From: Alejandro Lucero Palau @ 2025-08-26 10:42 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: Ben Cheatham, linux-cxl
On 8/19/25 17:05, Jonathan Cameron wrote:
> On Tue, 19 Aug 2025 16:57:54 +0100
> Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>
>> On Wed, 13 Aug 2025 12:25:12 +0100
>> Alejandro Lucero Palau <alucerop@amd.com> wrote:
>>
>>> On 8/12/25 22:29, Ben Cheatham wrote:
>>>> This patch series adds initial CXL.cache support. What I have here only
>>>> allows for adding a cache device to the system, programming/validating
>>>> the system configuration with respect to cache devices, and some basic
>>>> cache reporting/management.
>>>>
>>>> The general philosophy is to have an endpoint/vendor-specific driver
>>>> that runs through the same steps of adding a cxl_memdev, but for the
>>>> cache portion of the device (both type 1 & 2). Getting cache support for
>>>> a CXL device should be as simple as: get cache information, set up the
>>>> memory region (see below), and then calling devm_cxl_add_cachedev().
>>>>
>>>> There's a couple of things missing from this set:
>>>>
>>>> 1) Missing an endpoint driver
>>>>
>>>> I plan on submitting a reference driver (type 1 or 2) with v1, I
>>>> figured I'd send out what I have before going much further.
>>>>
>>>> 2) Mapping/Reserving host memory used for CXL cache(s)
>>>>
>>>> I'm thinking this will be handled by the endpoint driver, but I'm
>>>> not sure what mechanism would be used and whether to integrate it
>>>> with the cxl_cache/core. Any thoughts/ideas here are appreciated!
>>>
>>> Hi Ben,
>>>
>>>
>>> Good to see you working on this. I'll go through the series these next days.
>>>
>>>
>>> FWIW, I did send a RFC last November about my concerns for
>>> properly/safely supporting CXL.cache.
>>>
>>>
>>> I had no time to keep thinking about it. Also, I sent a copy to the
>>> kernel iommu mailing list, but it did not get traction then. Anyways, as
>>> a summary, I think mapping for CXL.cache requires not just the endpoint
>>> driver doing it freely but through the cxl core for ensuring it is done
>>> properly and safely. But I have to admit this could be a
>>> misunderstanding about how the hardware is going to allow CXL.cache.
>> Perhaps it's time to restart that discussion.
>>
>> https://lore.kernel.org/linux-cxl/cc2525a6-0f6a-c1c8-83e1-6396661efc8a@amd.com/
>>
> I hit send too fast. For that thread I'd suggest we break it down into
> the various issues you raise. Make it a bit more bite sized!
>
> Some such as trust should be relatively easy to tick off with the work
> Lukas is doing on attestation (or the version that uses confidential
> compute infrastructure)
I think that is a good approach. I will try to do so asap for maybe
presenting the discussion in Plumbers.
Thanks
>
> Jonathan
>
>
^ permalink raw reply [flat|nested] 55+ messages in thread
* Re: [RFC PATCH 00/18] Initial CXL.cache device support
2025-08-22 18:02 ` Cheatham, Benjamin
@ 2025-08-26 10:44 ` Alejandro Lucero Palau
0 siblings, 0 replies; 55+ messages in thread
From: Alejandro Lucero Palau @ 2025-08-26 10:44 UTC (permalink / raw)
To: Cheatham, Benjamin, linux-cxl
On 8/22/25 19:02, Cheatham, Benjamin wrote:
>
> On 8/13/2025 6:25 AM, Alejandro Lucero Palau wrote:
>> On 8/12/25 22:29, Ben Cheatham wrote:
>>> This patch series adds initial CXL.cache support. What I have here only
>>> allows for adding a cache device to the system, programming/validating
>>> the system configuration with respect to cache devices, and some basic
>>> cache reporting/management.
>>>
>>> The general philosophy is to have an endpoint/vendor-specific driver
>>> that runs through the same steps of adding a cxl_memdev, but for the
>>> cache portion of the device (both type 1 & 2). Getting cache support for
>>> a CXL device should be as simple as: get cache information, set up the
>>> memory region (see below), and then calling devm_cxl_add_cachedev().
>>>
>>> There's a couple of things missing from this set:
>>>
>>> 1) Missing an endpoint driver
>>>
>>> I plan on submitting a reference driver (type 1 or 2) with v1, I
>>> figured I'd send out what I have before going much further.
>>>
>>> 2) Mapping/Reserving host memory used for CXL cache(s)
>>>
>>> I'm thinking this will be handled by the endpoint driver, but I'm
>>> not sure what mechanism would be used and whether to integrate it
>>> with the cxl_cache/core. Any thoughts/ideas here are appreciated!
>>
>> Hi Ben,
>>
>>
>> Good to see you working on this. I'll go through the series these next days.
>>
>>
>> FWIW, I did send a RFC last November about my concerns for properly/safely supporting CXL.cache.
> Yeah, I saw Jonathan linked it in his reply as well. I've read it but didn't really have anything to say about safety/iommu schemes.
>
> This set is really just for the first part of the summary at the end of that email ("CXL core handling per device CXL.cache enabling...").
Yes, and it is good to have it.
>>
>> I had no time to keep thinking about it. Also, I sent a copy to the kernel iommu mailing list, but it did not get traction then. Anyways, as a summary, I think mapping for CXL.cache requires not just the endpoint driver doing it freely but through the cxl core for ensuring it is done properly and safely. But I have to admit this could be a misunderstanding about how the hardware is going to allow CXL.cache.
> I agree! I just wasn't sure of the approach to take here so I opted to leave it out until we came to a consensus.
I will try to start that discussion again ...
Thank you
> Thanks,
> Ben
>
>>
>> Thanks
>>
>>
>>> 3) RAS Support
>>>
>>> Same situation as 1) above.
>>>
>>> Some quick notes: The actual cache parts of this set are untested due
>>> to problems with my set up, but I did make sure nothing here breaks type
>>> 3 support. I'll have this fixed before sending out a v1. This series is
>>> based on the for-6.18/cxl-probe-order (commit 5e29cbd1077b) branch in
>>> the CXL repo, with v7 of Dave's deferred dport probe set on top [1]. I
>>> added Dave's set to help with getting around constraints with HDM
>>> decoders in CXL.cache device only configurations (see 08/18 for more).
>>>
>>> Patch Breakdown:
>>> - 1 & 2: Preliminary changes for struct cxl_cachedev
>>> - 3: Add struct cxl_cachedev
>>> - 4-8: Preliminary changes for adding cache devices to port
>>> hierarchy
>>> - 9: Function for getting CXL cache info
>>> - 10: cxl_cache driver (mirrors cxl_mem)
>>> - 11-16: Checking CXL.cache capabilities for system configuration
>>> validity
>>> - 17-18: Cache device attributes
>>>
>>> [1]:
>>> Link: https://lore.kernel.org/linux-cxl/20250714223527.461147-1-dave.jiang@intel.com/
>>>
>>> Ben Cheatham (18):
>>> cxl/mem: Change cxl_memdev_ops to cxl_dev_ops
>>> cxl: Move struct cxl_dev_state definition
>>> cxl/core: Add CXL.cache device struct
>>> cxl: Replace cxl_mem_find_port() with cxl_dev_find_port()
>>> cxl: Change cxl_ep_load() to use struct device * parameter
>>> cxl/port, mem: Make adding an endpoint device type agnostic
>>> cxl/port: Split endpoint port probe on device type
>>> cxl/port: Update switch_port_probe() for CXL cache devices
>>> cxl/core: Add function for getting CXL cache info
>>> cxl/cache: Add cxl_cache driver
>>> cxl/core: Add CXL snoop filter setup and checking
>>> cxl/cache: Add CXL Cache ID Route Table mapping
>>> cxl/cache: Implement Cache ID Route Table programming
>>> cxl/cache: Add Cache ID Decoder capability mapping
>>> cxl/cache: Implement Cache ID Decoder programming
>>> cxl/cache: Add cache device counting for CXL ports
>>> cxl/core: Add cache device attributes
>>> cxl/core: Add cache device cache management attributes
>>>
>>> drivers/cxl/Kconfig | 14 +
>>> drivers/cxl/Makefile | 2 +
>>> drivers/cxl/cache.c | 276 +++++++++++++++++++
>>> drivers/cxl/core/Makefile | 1 +
>>> drivers/cxl/core/cachedev.c | 292 ++++++++++++++++++++
>>> drivers/cxl/core/hdm.c | 31 +++
>>> drivers/cxl/core/memdev.c | 2 +-
>>> drivers/cxl/core/pci.c | 134 ++++++++++
>>> drivers/cxl/core/port.c | 518 +++++++++++++++++++++++++++++++++---
>>> drivers/cxl/core/region.c | 25 +-
>>> drivers/cxl/core/regs.c | 28 ++
>>> drivers/cxl/cxl.h | 193 +++++++++++++-
>>> drivers/cxl/cxlcache.h | 42 +++
>>> drivers/cxl/cxlmem.h | 121 +--------
>>> drivers/cxl/cxlpci.h | 10 +
>>> drivers/cxl/mem.c | 14 +-
>>> drivers/cxl/pci.c | 2 +-
>>> drivers/cxl/port.c | 54 ++--
>>> drivers/cxl/private.h | 8 +-
>>> 19 files changed, 1569 insertions(+), 198 deletions(-)
>>> create mode 100644 drivers/cxl/cache.c
>>> create mode 100644 drivers/cxl/core/cachedev.c
>>> create mode 100644 drivers/cxl/cxlcache.h
>>>
^ permalink raw reply [flat|nested] 55+ messages in thread
end of thread, other threads:[~2025-08-26 10:44 UTC | newest]
Thread overview: 55+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-08-12 21:29 [RFC PATCH 00/18] Initial CXL.cache device support Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 01/18] cxl/mem: Change cxl_memdev_ops to cxl_dev_ops Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 02/18] cxl: Move struct cxl_dev_state definition Ben Cheatham
2025-08-19 11:33 ` Jonathan Cameron
2025-08-22 18:00 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 03/18] cxl/core: Add CXL.cache device struct Ben Cheatham
2025-08-19 11:48 ` Jonathan Cameron
2025-08-22 18:00 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 04/18] cxl: Replace cxl_mem_find_port() with cxl_dev_find_port() Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 05/18] cxl: Change cxl_ep_load() to use struct device * parameter Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 06/18] cxl/port, mem: Make adding an endpoint device type agnostic Ben Cheatham
2025-08-19 11:53 ` Jonathan Cameron
2025-08-22 18:00 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 07/18] cxl/port: Split endpoint port probe on device type Ben Cheatham
2025-08-19 11:57 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 08/18] cxl/port: Update switch_port_probe() for CXL cache devices Ben Cheatham
2025-08-19 12:03 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 09/18] cxl/core: Add function for getting CXL cache info Ben Cheatham
2025-08-12 21:29 ` [RFC PATCH 10/18] cxl/cache: Add cxl_cache driver Ben Cheatham
2025-08-19 12:11 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 11/18] cxl/core: Add CXL snoop filter setup and checking Ben Cheatham
2025-08-19 14:18 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 12/18] cxl/cache: Add CXL Cache ID Route Table mapping Ben Cheatham
2025-08-19 15:09 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 13/18] cxl/cache: Implement Cache ID Route Table programming Ben Cheatham
2025-08-19 15:07 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 14/18] cxl/cache: Add Cache ID Decoder capability mapping Ben Cheatham
2025-08-19 14:12 ` Alireza Sanaee
2025-08-22 18:01 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 15/18] cxl/cache: Implement Cache ID Decoder programming Ben Cheatham
2025-08-19 13:44 ` Alireza Sanaee
2025-08-20 8:55 ` Alireza Sanaee
2025-08-19 15:26 ` Jonathan Cameron
2025-08-22 18:01 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 16/18] cxl/cache: Add cache device counting for CXL ports Ben Cheatham
2025-08-19 15:30 ` Jonathan Cameron
2025-08-22 18:02 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 17/18] cxl/core: Add cache device attributes Ben Cheatham
2025-08-19 15:38 ` Jonathan Cameron
2025-08-22 18:02 ` Cheatham, Benjamin
2025-08-12 21:29 ` [RFC PATCH 18/18] cxl/core: Add cache device cache management attributes Ben Cheatham
2025-08-19 15:53 ` Jonathan Cameron
2025-08-22 18:02 ` Cheatham, Benjamin
2025-08-13 11:25 ` [RFC PATCH 00/18] Initial CXL.cache device support Alejandro Lucero Palau
2025-08-19 15:57 ` Jonathan Cameron
2025-08-19 16:05 ` Jonathan Cameron
2025-08-26 10:42 ` Alejandro Lucero Palau
2025-08-22 18:02 ` Cheatham, Benjamin
2025-08-26 10:44 ` Alejandro Lucero Palau
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).