[PATCH v1 0/19] cxl: Add CXL feature commands support via fwctl

[PATCH v1 0/19] cxl: Add CXL feature commands support via fwctl

[Dave Jiang]

v1:
- Create a CXL features driver to handle all feature command related bits
  including FWCTL support. (Dan)
- Tested against CXL CLI unit tests for FWCTL.
- See individual patches for detailed changes from RFC v2.

RFC v2:
- Dropped 1/13 and 2/13 from previous version. Merged upstream already.
- Combined changes from Shiju for "get supported features"
- Addressed comments from Jonathan and Jason
- See specific changes in individual patch revision history
- Added hardware command info to FWCTL
- Added filtering to set feature command
- Added documentation

This series add support for CXL feature commands using the FWCTL framework [1].
The code has been tested with CXL CLI unit tests for FWCTL.

Patches 1-7 are shared with EDAC series [1] and there will be an immuable branch
once accepted post review.

A CXL features driver is created to handle all CXL mailbox feature commands related
functionalities as suggested by Dan. The 3 mailbox commands "Get Supported Features",
"Get Feature", and "Set Feature" are implemented. The "get" commands under the
FWCTL_RPC_CONFIGURATION policy, the "set" command checks the policy depending
on the effect of the feature. All mailbox commands for CXL provides an effects
table that describes the effects of a command when performed on the device.
For CXL features, there is also an effects field that describes the effects
a feature write operation has on the device per feature. The security policy
is checked against this feature specific effects field. Looking for discussion
on matching the CXL spec defined effects with the FWCTL security policy.

The code is based off of v3 of FWCTL series [2] posted by Jason and rebased on top of
v6.13-rc5.

[1]: https://lore.kernel.org/linux-cxl/20250106121017.1620-1-shiju.jose@huawei.com/T/#t
[2]: https://lore.kernel.org/linux-cxl/0-v3-960f17f90f17+516-fwctl_jgg@nvidia.com/#r
---

Dave Jiang (17):
      cxl: Refactor user ioctl command path from mds to mailbox
      cxl: Add skeletal features driver
      cxl: Enumerate feature commands
      cxl: Add Get Supported Features command for kernel usage
      cxl: Add features driver attribute to emit number of features supported
      cxl/test: Add Get Supported Features mailbox command support
      cxl: Setup exclusive CXL features that are reserved for the kernel
      cxl: Add FWCTL support to the CXL features driver
      cxl: Add support for get driver information
      cxl: Move cxl_mem.h under uapi to cxl exclusive directory
      cxl: Move cxl feature command structs to user header
      cxl: Add support for fwctl RPC command to enable CXL feature commands
      cxl: Add support to handle user feature commands for get feature
      cxl: Add support to handle user feature commands for set feature
      cxl/test: Add Get Feature support to cxl_test
      cxl/test: Add Set Feature support to cxl_test
      fwctl/cxl: Add documentation to FWCTL CXL

Shiju Jose (2):
      cxl/mbox: Add GET_FEATURE mailbox command
      cxl/mbox: Add SET_FEATURE mailbox command

 Documentation/userspace-api/fwctl/fwctl-cxl.rst |  81 +++++++
 Documentation/userspace-api/fwctl/index.rst     |   1 +
 drivers/cxl/Kconfig                             |   9 +
 drivers/cxl/Makefile                            |   3 +
 drivers/cxl/core/Makefile                       |   1 +
 drivers/cxl/core/core.h                         |   7 +-
 drivers/cxl/core/features.c                     | 287 +++++++++++++++++++++++
 drivers/cxl/core/mbox.c                         | 143 ++++++++----
 drivers/cxl/core/memdev.c                       |  22 +-
 drivers/cxl/core/port.c                         |   3 +
 drivers/cxl/cxl.h                               |   3 +
 drivers/cxl/cxlmem.h                            |  45 +---
 drivers/cxl/features.c                          | 558 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/pci.c                               |  19 ++
 include/cxl/features.h                          |  75 ++++++
 include/cxl/mailbox.h                           |  46 +++-
 include/uapi/cxl/features.h                     | 135 +++++++++++
 include/uapi/{linux/cxl_mem.h => cxl/mem.h}     |   0
 include/uapi/fwctl/cxl.h                        |  60 +++++
 include/uapi/fwctl/fwctl.h                      |   1 +
 tools/testing/cxl/Kbuild                        |   1 +
 tools/testing/cxl/test/mem.c                    | 194 +++++++++++++++
 22 files changed, 1597 insertions(+), 97 deletions(-)

[PATCH v1 01/19] cxl: Refactor user ioctl command path from mds to mailbox

[Dave Jiang]

With 'struct cxl_mailbox' context introduced, the helper functions
cxl_query_cmd() and cxl_send_cmd() can take a cxl_mailbox directly
rather than a cxl_memdev parameter. Refactor to use cxl_mailbox
directly.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/core.h   |  6 ++-
 drivers/cxl/core/mbox.c   | 89 +++++++++++++++++++--------------------
 drivers/cxl/core/memdev.c | 22 +++++++---
 drivers/cxl/cxlmem.h      | 40 ------------------
 include/cxl/mailbox.h     | 41 +++++++++++++++++-
 5 files changed, 102 insertions(+), 96 deletions(-)

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index 800466f96a68..23761340e65c 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -4,6 +4,8 @@
 #ifndef __CXL_CORE_H__
 #define __CXL_CORE_H__
 
+#include <cxl/mailbox.h>
+
 extern const struct device_type cxl_nvdimm_bridge_type;
 extern const struct device_type cxl_nvdimm_type;
 extern const struct device_type cxl_pmu_type;
@@ -65,9 +67,9 @@ static inline void cxl_region_exit(void)
 
 struct cxl_send_command;
 struct cxl_mem_query_commands;
-int cxl_query_cmd(struct cxl_memdev *cxlmd,
+int cxl_query_cmd(struct cxl_mailbox *cxl_mbox,
 		  struct cxl_mem_query_commands __user *q);
-int cxl_send_cmd(struct cxl_memdev *cxlmd, struct cxl_send_command __user *s);
+int cxl_send_cmd(struct cxl_mailbox *cxl_mailbox, struct cxl_send_command __user *s);
 void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
 				   resource_size_t length);
 
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 548564c770c0..bdb8f060f2c1 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -349,40 +349,40 @@ static bool cxl_payload_from_user_allowed(u16 opcode, void *payload_in)
 	return true;
 }
 
-static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox,
-			     struct cxl_memdev_state *mds, u16 opcode,
+static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox_cmd,
+			     struct cxl_mailbox *cxl_mbox, u16 opcode,
 			     size_t in_size, size_t out_size, u64 in_payload)
 {
-	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
-	*mbox = (struct cxl_mbox_cmd) {
+	*mbox_cmd = (struct cxl_mbox_cmd) {
 		.opcode = opcode,
 		.size_in = in_size,
 	};
 
 	if (in_size) {
-		mbox->payload_in = vmemdup_user(u64_to_user_ptr(in_payload),
-						in_size);
-		if (IS_ERR(mbox->payload_in))
-			return PTR_ERR(mbox->payload_in);
+		mbox_cmd->payload_in = vmemdup_user(u64_to_user_ptr(in_payload),
+						    in_size);
+		if (IS_ERR(mbox_cmd->payload_in))
+			return PTR_ERR(mbox_cmd->payload_in);
 
-		if (!cxl_payload_from_user_allowed(opcode, mbox->payload_in)) {
-			dev_dbg(mds->cxlds.dev, "%s: input payload not allowed\n",
+		if (!cxl_payload_from_user_allowed(opcode,
+						   mbox_cmd->payload_in)) {
+			dev_dbg(cxl_mbox->host, "%s: input payload not allowed\n",
 				cxl_mem_opcode_to_name(opcode));
-			kvfree(mbox->payload_in);
+			kvfree(mbox_cmd->payload_in);
 			return -EBUSY;
 		}
 	}
 
 	/* Prepare to handle a full payload for variable sized output */
 	if (out_size == CXL_VARIABLE_PAYLOAD)
-		mbox->size_out = cxl_mbox->payload_size;
+		mbox_cmd->size_out = cxl_mbox->payload_size;
 	else
-		mbox->size_out = out_size;
+		mbox_cmd->size_out = out_size;
 
-	if (mbox->size_out) {
-		mbox->payload_out = kvzalloc(mbox->size_out, GFP_KERNEL);
-		if (!mbox->payload_out) {
-			kvfree(mbox->payload_in);
+	if (mbox_cmd->size_out) {
+		mbox_cmd->payload_out = kvzalloc(mbox_cmd->size_out, GFP_KERNEL);
+		if (!mbox_cmd->payload_out) {
+			kvfree(mbox_cmd->payload_in);
 			return -ENOMEM;
 		}
 	}
@@ -397,10 +397,8 @@ static void cxl_mbox_cmd_dtor(struct cxl_mbox_cmd *mbox)
 
 static int cxl_to_mem_cmd_raw(struct cxl_mem_command *mem_cmd,
 			      const struct cxl_send_command *send_cmd,
-			      struct cxl_memdev_state *mds)
+			      struct cxl_mailbox *cxl_mbox)
 {
-	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
-
 	if (send_cmd->raw.rsvd)
 		return -EINVAL;
 
@@ -415,7 +413,7 @@ static int cxl_to_mem_cmd_raw(struct cxl_mem_command *mem_cmd,
 	if (!cxl_mem_raw_command_allowed(send_cmd->raw.opcode))
 		return -EPERM;
 
-	dev_WARN_ONCE(mds->cxlds.dev, true, "raw command path used\n");
+	dev_WARN_ONCE(cxl_mbox->host, true, "raw command path used\n");
 
 	*mem_cmd = (struct cxl_mem_command) {
 		.info = {
@@ -431,7 +429,7 @@ static int cxl_to_mem_cmd_raw(struct cxl_mem_command *mem_cmd,
 
 static int cxl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
 			  const struct cxl_send_command *send_cmd,
-			  struct cxl_memdev_state *mds)
+			  struct cxl_mailbox *cxl_mbox)
 {
 	struct cxl_mem_command *c = &cxl_mem_commands[send_cmd->id];
 	const struct cxl_command_info *info = &c->info;
@@ -446,11 +444,11 @@ static int cxl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
 		return -EINVAL;
 
 	/* Check that the command is enabled for hardware */
-	if (!test_bit(info->id, mds->enabled_cmds))
+	if (!test_bit(info->id, cxl_mbox->enabled_cmds))
 		return -ENOTTY;
 
 	/* Check that the command is not claimed for exclusive kernel use */
-	if (test_bit(info->id, mds->exclusive_cmds))
+	if (test_bit(info->id, cxl_mbox->exclusive_cmds))
 		return -EBUSY;
 
 	/* Check the input buffer is the expected size */
@@ -479,7 +477,7 @@ static int cxl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
 /**
  * cxl_validate_cmd_from_user() - Check fields for CXL_MEM_SEND_COMMAND.
  * @mbox_cmd: Sanitized and populated &struct cxl_mbox_cmd.
- * @mds: The driver data for the operation
+ * @cxl_mbox: CXL mailbox context
  * @send_cmd: &struct cxl_send_command copied in from userspace.
  *
  * Return:
@@ -494,10 +492,9 @@ static int cxl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
  * safe to send to the hardware.
  */
 static int cxl_validate_cmd_from_user(struct cxl_mbox_cmd *mbox_cmd,
-				      struct cxl_memdev_state *mds,
+				      struct cxl_mailbox *cxl_mbox,
 				      const struct cxl_send_command *send_cmd)
 {
-	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
 	struct cxl_mem_command mem_cmd;
 	int rc;
 
@@ -514,24 +511,23 @@ static int cxl_validate_cmd_from_user(struct cxl_mbox_cmd *mbox_cmd,
 
 	/* Sanitize and construct a cxl_mem_command */
 	if (send_cmd->id == CXL_MEM_COMMAND_ID_RAW)
-		rc = cxl_to_mem_cmd_raw(&mem_cmd, send_cmd, mds);
+		rc = cxl_to_mem_cmd_raw(&mem_cmd, send_cmd, cxl_mbox);
 	else
-		rc = cxl_to_mem_cmd(&mem_cmd, send_cmd, mds);
+		rc = cxl_to_mem_cmd(&mem_cmd, send_cmd, cxl_mbox);
 
 	if (rc)
 		return rc;
 
 	/* Sanitize and construct a cxl_mbox_cmd */
-	return cxl_mbox_cmd_ctor(mbox_cmd, mds, mem_cmd.opcode,
+	return cxl_mbox_cmd_ctor(mbox_cmd, cxl_mbox, mem_cmd.opcode,
 				 mem_cmd.info.size_in, mem_cmd.info.size_out,
 				 send_cmd->in.payload);
 }
 
-int cxl_query_cmd(struct cxl_memdev *cxlmd,
+int cxl_query_cmd(struct cxl_mailbox *cxl_mbox,
 		  struct cxl_mem_query_commands __user *q)
 {
-	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
-	struct device *dev = &cxlmd->dev;
+	struct device *dev = cxl_mbox->host;
 	struct cxl_mem_command *cmd;
 	u32 n_commands;
 	int j = 0;
@@ -552,9 +548,9 @@ int cxl_query_cmd(struct cxl_memdev *cxlmd,
 	cxl_for_each_cmd(cmd) {
 		struct cxl_command_info info = cmd->info;
 
-		if (test_bit(info.id, mds->enabled_cmds))
+		if (test_bit(info.id, cxl_mbox->enabled_cmds))
 			info.flags |= CXL_MEM_COMMAND_FLAG_ENABLED;
-		if (test_bit(info.id, mds->exclusive_cmds))
+		if (test_bit(info.id, cxl_mbox->exclusive_cmds))
 			info.flags |= CXL_MEM_COMMAND_FLAG_EXCLUSIVE;
 
 		if (copy_to_user(&q->commands[j++], &info, sizeof(info)))
@@ -569,7 +565,7 @@ int cxl_query_cmd(struct cxl_memdev *cxlmd,
 
 /**
  * handle_mailbox_cmd_from_user() - Dispatch a mailbox command for userspace.
- * @mds: The driver data for the operation
+ * @cxl_mbox: The mailbox context for the operation.
  * @mbox_cmd: The validated mailbox command.
  * @out_payload: Pointer to userspace's output payload.
  * @size_out: (Input) Max payload size to copy out.
@@ -590,13 +586,12 @@ int cxl_query_cmd(struct cxl_memdev *cxlmd,
  *
  * See cxl_send_cmd().
  */
-static int handle_mailbox_cmd_from_user(struct cxl_memdev_state *mds,
+static int handle_mailbox_cmd_from_user(struct cxl_mailbox *cxl_mbox,
 					struct cxl_mbox_cmd *mbox_cmd,
 					u64 out_payload, s32 *size_out,
 					u32 *retval)
 {
-	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
-	struct device *dev = mds->cxlds.dev;
+	struct device *dev = cxl_mbox->host;
 	int rc;
 
 	dev_dbg(dev,
@@ -633,10 +628,9 @@ static int handle_mailbox_cmd_from_user(struct cxl_memdev_state *mds,
 	return rc;
 }
 
-int cxl_send_cmd(struct cxl_memdev *cxlmd, struct cxl_send_command __user *s)
+int cxl_send_cmd(struct cxl_mailbox *cxl_mbox, struct cxl_send_command __user *s)
 {
-	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
-	struct device *dev = &cxlmd->dev;
+	struct device *dev = cxl_mbox->host;
 	struct cxl_send_command send;
 	struct cxl_mbox_cmd mbox_cmd;
 	int rc;
@@ -646,11 +640,11 @@ int cxl_send_cmd(struct cxl_memdev *cxlmd, struct cxl_send_command __user *s)
 	if (copy_from_user(&send, s, sizeof(send)))
 		return -EFAULT;
 
-	rc = cxl_validate_cmd_from_user(&mbox_cmd, mds, &send);
+	rc = cxl_validate_cmd_from_user(&mbox_cmd, cxl_mbox, &send);
 	if (rc)
 		return rc;
 
-	rc = handle_mailbox_cmd_from_user(mds, &mbox_cmd, send.out.payload,
+	rc = handle_mailbox_cmd_from_user(cxl_mbox, &mbox_cmd, send.out.payload,
 					  &send.out.size, &send.retval);
 	if (rc)
 		return rc;
@@ -724,6 +718,7 @@ static int cxl_xfer_log(struct cxl_memdev_state *mds, uuid_t *uuid,
  */
 static void cxl_walk_cel(struct cxl_memdev_state *mds, size_t size, u8 *cel)
 {
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
 	struct cxl_cel_entry *cel_entry;
 	const int cel_entries = size / sizeof(*cel_entry);
 	struct device *dev = mds->cxlds.dev;
@@ -737,7 +732,7 @@ static void cxl_walk_cel(struct cxl_memdev_state *mds, size_t size, u8 *cel)
 		int enabled = 0;
 
 		if (cmd) {
-			set_bit(cmd->info.id, mds->enabled_cmds);
+			set_bit(cmd->info.id, cxl_mbox->enabled_cmds);
 			enabled++;
 		}
 
@@ -807,6 +802,7 @@ static const uuid_t log_uuid[] = {
  */
 int cxl_enumerate_cmds(struct cxl_memdev_state *mds)
 {
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
 	struct cxl_mbox_get_supported_logs *gsl;
 	struct device *dev = mds->cxlds.dev;
 	struct cxl_mem_command *cmd;
@@ -845,7 +841,7 @@ int cxl_enumerate_cmds(struct cxl_memdev_state *mds)
 		/* In case CEL was bogus, enable some default commands. */
 		cxl_for_each_cmd(cmd)
 			if (cmd->flags & CXL_CMD_FLAG_FORCE_ENABLE)
-				set_bit(cmd->info.id, mds->enabled_cmds);
+				set_bit(cmd->info.id, cxl_mbox->enabled_cmds);
 
 		/* Found the required CEL */
 		rc = 0;
@@ -1448,6 +1444,7 @@ struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev)
 	mutex_init(&mds->event.log_lock);
 	mds->cxlds.dev = dev;
 	mds->cxlds.reg_map.host = dev;
+	mds->cxlds.cxl_mbox.host = dev;
 	mds->cxlds.reg_map.resource = CXL_RESOURCE_NONE;
 	mds->cxlds.type = CXL_DEVTYPE_CLASSMEM;
 	mds->ram_perf.qos_class = CXL_QOS_CLASS_INVALID;
diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
index ae3dfcbe8938..2e2e035abdaa 100644
--- a/drivers/cxl/core/memdev.c
+++ b/drivers/cxl/core/memdev.c
@@ -564,9 +564,11 @@ EXPORT_SYMBOL_NS_GPL(is_cxl_memdev, "CXL");
 void set_exclusive_cxl_commands(struct cxl_memdev_state *mds,
 				unsigned long *cmds)
 {
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+
 	down_write(&cxl_memdev_rwsem);
-	bitmap_or(mds->exclusive_cmds, mds->exclusive_cmds, cmds,
-		  CXL_MEM_COMMAND_ID_MAX);
+	bitmap_or(cxl_mbox->exclusive_cmds, cxl_mbox->exclusive_cmds,
+		  cmds, CXL_MEM_COMMAND_ID_MAX);
 	up_write(&cxl_memdev_rwsem);
 }
 EXPORT_SYMBOL_NS_GPL(set_exclusive_cxl_commands, "CXL");
@@ -579,9 +581,11 @@ EXPORT_SYMBOL_NS_GPL(set_exclusive_cxl_commands, "CXL");
 void clear_exclusive_cxl_commands(struct cxl_memdev_state *mds,
 				  unsigned long *cmds)
 {
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+
 	down_write(&cxl_memdev_rwsem);
-	bitmap_andnot(mds->exclusive_cmds, mds->exclusive_cmds, cmds,
-		      CXL_MEM_COMMAND_ID_MAX);
+	bitmap_andnot(cxl_mbox->exclusive_cmds, cxl_mbox->exclusive_cmds,
+		      cmds, CXL_MEM_COMMAND_ID_MAX);
 	up_write(&cxl_memdev_rwsem);
 }
 EXPORT_SYMBOL_NS_GPL(clear_exclusive_cxl_commands, "CXL");
@@ -656,11 +660,14 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
 static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
 			       unsigned long arg)
 {
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+
 	switch (cmd) {
 	case CXL_MEM_QUERY_COMMANDS:
-		return cxl_query_cmd(cxlmd, (void __user *)arg);
+		return cxl_query_cmd(cxl_mbox, (void __user *)arg);
 	case CXL_MEM_SEND_COMMAND:
-		return cxl_send_cmd(cxlmd, (void __user *)arg);
+		return cxl_send_cmd(cxl_mbox, (void __user *)arg);
 	default:
 		return -ENOTTY;
 	}
@@ -994,10 +1001,11 @@ static void cxl_remove_fw_upload(void *fwl)
 int devm_cxl_setup_fw_upload(struct device *host, struct cxl_memdev_state *mds)
 {
 	struct cxl_dev_state *cxlds = &mds->cxlds;
+	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
 	struct device *dev = &cxlds->cxlmd->dev;
 	struct fw_upload *fwl;
 
-	if (!test_bit(CXL_MEM_COMMAND_ID_GET_FW_INFO, mds->enabled_cmds))
+	if (!test_bit(CXL_MEM_COMMAND_ID_GET_FW_INFO, cxl_mbox->enabled_cmds))
 		return 0;
 
 	fwl = firmware_upload_register(THIS_MODULE, dev, dev_name(dev),
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 2a25d1957ddb..a0a49809cd76 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -106,42 +106,6 @@ static inline struct cxl_ep *cxl_ep_load(struct cxl_port *port,
 	return xa_load(&port->endpoints, (unsigned long)&cxlmd->dev);
 }
 
-/**
- * struct cxl_mbox_cmd - A command to be submitted to hardware.
- * @opcode: (input) The command set and command submitted to hardware.
- * @payload_in: (input) Pointer to the input payload.
- * @payload_out: (output) Pointer to the output payload. Must be allocated by
- *		 the caller.
- * @size_in: (input) Number of bytes to load from @payload_in.
- * @size_out: (input) Max number of bytes loaded into @payload_out.
- *            (output) Number of bytes generated by the device. For fixed size
- *            outputs commands this is always expected to be deterministic. For
- *            variable sized output commands, it tells the exact number of bytes
- *            written.
- * @min_out: (input) internal command output payload size validation
- * @poll_count: (input) Number of timeouts to attempt.
- * @poll_interval_ms: (input) Time between mailbox background command polling
- *                    interval timeouts.
- * @return_code: (output) Error code returned from hardware.
- *
- * This is the primary mechanism used to send commands to the hardware.
- * All the fields except @payload_* correspond exactly to the fields described in
- * Command Register section of the CXL 2.0 8.2.8.4.5. @payload_in and
- * @payload_out are written to, and read from the Command Payload Registers
- * defined in CXL 2.0 8.2.8.4.8.
- */
-struct cxl_mbox_cmd {
-	u16 opcode;
-	void *payload_in;
-	void *payload_out;
-	size_t size_in;
-	size_t size_out;
-	size_t min_out;
-	int poll_count;
-	int poll_interval_ms;
-	u16 return_code;
-};
-
 /*
  * Per CXL 3.0 Section 8.2.8.4.5.1
  */
@@ -461,8 +425,6 @@ static inline struct cxl_dev_state *mbox_to_cxlds(struct cxl_mailbox *cxl_mbox)
  * @lsa_size: Size of Label Storage Area
  *                (CXL 2.0 8.2.9.5.1.1 Identify Memory Device)
  * @firmware_version: Firmware version for the memory device.
- * @enabled_cmds: Hardware commands found enabled in CEL.
- * @exclusive_cmds: Commands that are kernel-internal only
  * @total_bytes: sum of all possible capacities
  * @volatile_only_bytes: hard volatile capacity
  * @persistent_only_bytes: hard persistent capacity
@@ -485,8 +447,6 @@ struct cxl_memdev_state {
 	struct cxl_dev_state cxlds;
 	size_t lsa_size;
 	char firmware_version[0x10];
-	DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
-	DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
 	u64 total_bytes;
 	u64 volatile_only_bytes;
 	u64 persistent_only_bytes;
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index bacd111e75f1..cc894f07a435 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -3,12 +3,49 @@
 #ifndef __CXL_MBOX_H__
 #define __CXL_MBOX_H__
 #include <linux/rcuwait.h>
+#include <uapi/linux/cxl_mem.h>
 
-struct cxl_mbox_cmd;
+/**
+ * struct cxl_mbox_cmd - A command to be submitted to hardware.
+ * @opcode: (input) The command set and command submitted to hardware.
+ * @payload_in: (input) Pointer to the input payload.
+ * @payload_out: (output) Pointer to the output payload. Must be allocated by
+ *		 the caller.
+ * @size_in: (input) Number of bytes to load from @payload_in.
+ * @size_out: (input) Max number of bytes loaded into @payload_out.
+ *            (output) Number of bytes generated by the device. For fixed size
+ *            outputs commands this is always expected to be deterministic. For
+ *            variable sized output commands, it tells the exact number of bytes
+ *            written.
+ * @min_out: (input) internal command output payload size validation
+ * @poll_count: (input) Number of timeouts to attempt.
+ * @poll_interval_ms: (input) Time between mailbox background command polling
+ *                    interval timeouts.
+ * @return_code: (output) Error code returned from hardware.
+ *
+ * This is the primary mechanism used to send commands to the hardware.
+ * All the fields except @payload_* correspond exactly to the fields described in
+ * Command Register section of the CXL 2.0 8.2.8.4.5. @payload_in and
+ * @payload_out are written to, and read from the Command Payload Registers
+ * defined in CXL 2.0 8.2.8.4.8.
+ */
+struct cxl_mbox_cmd {
+	u16 opcode;
+	void *payload_in;
+	void *payload_out;
+	size_t size_in;
+	size_t size_out;
+	size_t min_out;
+	int poll_count;
+	int poll_interval_ms;
+	u16 return_code;
+};
 
 /**
  * struct cxl_mailbox - context for CXL mailbox operations
  * @host: device that hosts the mailbox
+ * @enabled_cmds: mailbox commands that are enabled by the driver
+ * @exclusive_cmds: mailbox commands that are exclusive to the kernel
  * @payload_size: Size of space for payload
  *                (CXL 3.1 8.2.8.4.3 Mailbox Capabilities Register)
  * @mbox_mutex: mutex protects device mailbox and firmware
@@ -17,6 +54,8 @@ struct cxl_mbox_cmd;
  */
 struct cxl_mailbox {
 	struct device *host;
+	DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
+	DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
 	size_t payload_size;
 	struct mutex mbox_mutex; /* lock to protect mailbox context */
 	struct rcuwait mbox_wait;
-- 
2.47.1

[PATCH v1 02/19] cxl: Add skeletal features driver

[Dave Jiang]

Add the basic bits of a features driver to handle all CXL feature related
services. The driver is expected to handle all CXL mailbox feature command
related operations.

Suggested-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/Kconfig                   | 11 +++++
 drivers/cxl/Makefile                  |  3 ++
 drivers/cxl/core/Makefile             |  1 +
 drivers/cxl/core/core.h               |  1 +
 drivers/cxl/core/features.c           | 71 +++++++++++++++++++++++++++
 drivers/cxl/core/port.c               |  3 ++
 drivers/cxl/cxl.h                     |  1 +
 drivers/cxl/features.c                | 44 +++++++++++++++++
 drivers/cxl/pci.c                     | 19 +++++++
 include/cxl/features.h                | 23 +++++++++
 include/cxl/mailbox.h                 |  3 ++
 tools/testing/cxl/Kbuild              |  7 +++
 tools/testing/cxl/cxl_features_test.c |  6 +++
 13 files changed, 193 insertions(+)
 create mode 100644 drivers/cxl/core/features.c
 create mode 100644 drivers/cxl/features.c
 create mode 100644 include/cxl/features.h
 create mode 100644 tools/testing/cxl/cxl_features_test.c

diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 876469e23f7a..9a6ffd81ac0e 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -146,4 +146,15 @@ config CXL_REGION_INVALIDATION_TEST
 	  If unsure, or if this kernel is meant for production environments,
 	  say N.
 
+config CXL_FEATURES
+	tristate "CXL: Features support"
+	default CXL_BUS
+	help
+	  Enable CXL features support that are tied to a CXL mailbox.
+	  The support for features including the feature mailbox
+	  commands and also FWCTL support of the commands via user
+	  space.
+
+	  If unsure say 'y'.
+
 endif
diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
index 2caa90fa4bf2..4696fc218df4 100644
--- a/drivers/cxl/Makefile
+++ b/drivers/cxl/Makefile
@@ -7,15 +7,18 @@
 # - 'mem' and 'pmem' before endpoint drivers so that memdevs are
 #   immediately enabled
 # - 'pci' last, also mirrors the hardware enumeration hierarchy
+# - 'features' comes after pci device is enumerated
 obj-y += core/
 obj-$(CONFIG_CXL_PORT) += cxl_port.o
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
 obj-$(CONFIG_CXL_MEM) += cxl_mem.o
 obj-$(CONFIG_CXL_PCI) += cxl_pci.o
+obj-$(CONFIG_CXL_FEATURES) += cxl_features.o
 
 cxl_port-y := port.o
 cxl_acpi-y := acpi.o
 cxl_pmem-y := pmem.o security.o
 cxl_mem-y := mem.o
 cxl_pci-y := pci.o
+cxl_features-y := features.o
diff --git a/drivers/cxl/core/Makefile b/drivers/cxl/core/Makefile
index 9259bcc6773c..73b6348afd67 100644
--- a/drivers/cxl/core/Makefile
+++ b/drivers/cxl/core/Makefile
@@ -14,5 +14,6 @@ cxl_core-y += pci.o
 cxl_core-y += hdm.o
 cxl_core-y += pmu.o
 cxl_core-y += cdat.o
+cxl_core-y += features.o
 cxl_core-$(CONFIG_TRACING) += trace.o
 cxl_core-$(CONFIG_CXL_REGION) += region.o
diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index 23761340e65c..e8a3df226643 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -9,6 +9,7 @@
 extern const struct device_type cxl_nvdimm_bridge_type;
 extern const struct device_type cxl_nvdimm_type;
 extern const struct device_type cxl_pmu_type;
+extern const struct device_type cxl_features_type;
 
 extern struct attribute_group cxl_base_attribute_group;
 
diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
new file mode 100644
index 000000000000..eb6eb191a32e
--- /dev/null
+++ b/drivers/cxl/core/features.c
@@ -0,0 +1,71 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2024-2025 Intel Corporation. All rights reserved. */
+#include <linux/device.h>
+#include "cxl.h"
+#include "core.h"
+
+#define CXL_FEATURE_MAX_DEVS 65536
+static DEFINE_IDA(cxl_features_ida);
+
+static void cxl_features_release(struct device *dev)
+{
+	struct cxl_features *features = to_cxl_features(dev);
+
+	ida_free(&cxl_features_ida, features->id);
+	kfree(features);
+}
+
+static void remove_features_dev(void *dev)
+{
+	device_unregister(dev);
+}
+
+const struct device_type cxl_features_type = {
+	.name = "features",
+	.release = cxl_features_release,
+};
+EXPORT_SYMBOL_NS_GPL(cxl_features_type, "CXL");
+
+struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
+					struct device *parent)
+{
+	struct device *dev;
+	int rc;
+
+	struct cxl_features *features __free(kfree) =
+		kzalloc(sizeof(*features), GFP_KERNEL);
+	if (!features)
+		return ERR_PTR(-ENOMEM);
+
+	rc = ida_alloc_max(&cxl_features_ida, CXL_FEATURE_MAX_DEVS - 1,
+			   GFP_KERNEL);
+	if (rc < 0)
+		return ERR_PTR(rc);
+
+	features->id = rc;
+	features->cxl_mbox = cxl_mbox;
+	dev = &features->dev;
+	device_initialize(dev);
+	device_set_pm_not_required(dev);
+	dev->parent = parent;
+	dev->bus = &cxl_bus_type;
+	dev->type = &cxl_features_type;
+	rc = dev_set_name(dev, "features%d", features->id);
+	if (rc)
+		goto err;
+
+	rc = device_add(dev);
+	if (rc)
+		goto err;
+
+	rc = devm_add_action_or_reset(parent, remove_features_dev, dev);
+	if (rc)
+		goto err;
+
+	return no_free_ptr(features);
+
+err:
+	put_device(dev);
+	return ERR_PTR(rc);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_features_alloc, "CXL");
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 78a5c2c25982..cc53a597cae6 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -74,6 +74,9 @@ static int cxl_device_id(const struct device *dev)
 		return CXL_DEVICE_REGION;
 	if (dev->type == &cxl_pmu_type)
 		return CXL_DEVICE_PMU;
+	if (dev->type == &cxl_features_type)
+		return CXL_DEVICE_FEATURES;
+
 	return 0;
 }
 
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index f6015f24ad38..ee29d1a1c8df 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -855,6 +855,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);
 #define CXL_DEVICE_PMEM_REGION		7
 #define CXL_DEVICE_DAX_REGION		8
 #define CXL_DEVICE_PMU			9
+#define CXL_DEVICE_FEATURES		10
 
 #define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
 #define CXL_MODALIAS_FMT "cxl:t%d"
diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
new file mode 100644
index 000000000000..644add26975f
--- /dev/null
+++ b/drivers/cxl/features.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2024,2025 Intel Corporation. All rights reserved. */
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <cxl/features.h>
+
+#include "cxl.h"
+
+static int cxl_features_probe(struct device *dev)
+{
+	struct cxl_features *features = to_cxl_features(dev);
+	struct cxl_features_state *cfs __free(kfree) =
+		kzalloc(sizeof(*cfs), GFP_KERNEL);
+
+	if (!cfs)
+		return -ENOMEM;
+
+	cfs->features = features;
+	dev_set_drvdata(dev, no_free_ptr(cfs));
+
+	return 0;
+}
+
+static void cxl_features_remove(struct device *dev)
+{
+	struct cxl_features_state *cfs = dev_get_drvdata(dev);
+
+	kfree(cfs);
+}
+
+static struct cxl_driver cxl_features_driver = {
+	.name = "cxl_features",
+	.probe = cxl_features_probe,
+	.remove = cxl_features_remove,
+	.id = CXL_DEVICE_FEATURES,
+};
+
+module_cxl_driver(cxl_features_driver);
+
+MODULE_DESCRIPTION("CXL: Features");
+MODULE_LICENSE("GPL");
+MODULE_IMPORT_NS("CXL");
+MODULE_ALIAS_CXL(CXL_DEVICE_FEATURES);
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 6d94ff4a4f1a..eb68dd3f8b21 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -386,6 +386,21 @@ static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox,
 	return rc;
 }
 
+static int cxl_pci_setup_features(struct cxl_memdev_state *mds)
+{
+	struct cxl_dev_state *cxlds = &mds->cxlds;
+	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
+	struct cxl_features *features;
+
+	features = cxl_features_alloc(cxl_mbox, cxlds->dev);
+	if (IS_ERR(features))
+		return PTR_ERR(features);
+
+	cxl_mbox->features = features;
+
+	return 0;
+}
+
 static int cxl_pci_setup_mailbox(struct cxl_memdev_state *mds, bool irq_avail)
 {
 	struct cxl_dev_state *cxlds = &mds->cxlds;
@@ -980,6 +995,10 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
+	rc = cxl_pci_setup_features(mds);
+	if (rc)
+		return rc;
+
 	rc = cxl_set_timestamp(mds);
 	if (rc)
 		return rc;
diff --git a/include/cxl/features.h b/include/cxl/features.h
new file mode 100644
index 000000000000..b92da1e92780
--- /dev/null
+++ b/include/cxl/features.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2024-2025 Intel Corporation. */
+#ifndef __CXL_FEATURES_H__
+#define __CXL_FEATURES_H__
+
+struct cxl_mailbox;
+
+struct cxl_features {
+	int id;
+	struct device dev;
+	struct cxl_mailbox *cxl_mbox;
+};
+#define to_cxl_features(dev) container_of(dev, struct cxl_features, dev)
+
+struct cxl_features_state {
+	struct cxl_features *features;
+	int num_features;
+};
+
+struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
+					struct device *parent);
+
+#endif
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index cc894f07a435..6caab0d406ba 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -3,6 +3,7 @@
 #ifndef __CXL_MBOX_H__
 #define __CXL_MBOX_H__
 #include <linux/rcuwait.h>
+#include <cxl/features.h>
 #include <uapi/linux/cxl_mem.h>
 
 /**
@@ -50,6 +51,7 @@ struct cxl_mbox_cmd {
  *                (CXL 3.1 8.2.8.4.3 Mailbox Capabilities Register)
  * @mbox_mutex: mutex protects device mailbox and firmware
  * @mbox_wait: rcuwait for mailbox
+ * @features: pointer to cxl_features device
  * @mbox_send: @dev specific transport for transmitting mailbox commands
  */
 struct cxl_mailbox {
@@ -59,6 +61,7 @@ struct cxl_mailbox {
 	size_t payload_size;
 	struct mutex mbox_mutex; /* lock to protect mailbox context */
 	struct rcuwait mbox_wait;
+	struct cxl_features *features;
 	int (*mbox_send)(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd *cmd);
 };
 
diff --git a/tools/testing/cxl/Kbuild b/tools/testing/cxl/Kbuild
index b1256fee3567..8b62341bb5df 100644
--- a/tools/testing/cxl/Kbuild
+++ b/tools/testing/cxl/Kbuild
@@ -50,6 +50,12 @@ cxl_mem-y := $(CXL_SRC)/mem.o
 cxl_mem-y += config_check.o
 cxl_mem-y += cxl_mem_test.o
 
+obj-m += cxl_features.o
+
+cxl_features-y := $(CXL_SRC)/features.o
+cxl_features-y += config_check.o
+cxl_features-y += cxl_features_test.o
+
 obj-m += cxl_core.o
 
 cxl_core-y := $(CXL_CORE_SRC)/port.o
@@ -61,6 +67,7 @@ cxl_core-y += $(CXL_CORE_SRC)/pci.o
 cxl_core-y += $(CXL_CORE_SRC)/hdm.o
 cxl_core-y += $(CXL_CORE_SRC)/pmu.o
 cxl_core-y += $(CXL_CORE_SRC)/cdat.o
+cxl_core-y += $(CXL_CORE_SRC)/features.o
 cxl_core-$(CONFIG_TRACING) += $(CXL_CORE_SRC)/trace.o
 cxl_core-$(CONFIG_CXL_REGION) += $(CXL_CORE_SRC)/region.o
 cxl_core-y += config_check.o
diff --git a/tools/testing/cxl/cxl_features_test.c b/tools/testing/cxl/cxl_features_test.c
new file mode 100644
index 000000000000..b117977b5ab2
--- /dev/null
+++ b/tools/testing/cxl/cxl_features_test.c
@@ -0,0 +1,6 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2025 Intel Corporation. All rights reserved. */
+
+#include "watermark.h"
+
+cxl_test_watermark(cxl_features);
-- 
2.47.1

[PATCH v1 03/19] cxl: Enumerate feature commands

[Dave Jiang]

Add feature commands enumeration code in order to detect and enumerate
the 3 feature related commands "get supported features", "get feature",
and "set feature". The enumeration will help determine whether the driver
can issue any of the 3 commands to the device.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/mbox.c | 41 +++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/cxlmem.h    |  3 +++
 include/cxl/features.h  |  7 +++++++
 include/cxl/mailbox.h   |  1 +
 4 files changed, 52 insertions(+)

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index bdb8f060f2c1..5e21ff99d70f 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -38,6 +38,21 @@ static bool cxl_raw_allow_all;
 	.flags = _flags,                                                       \
 	}
 
+#define cxl_for_each_feature_cmd(cmd)                                          \
+	for ((cmd) = &cxl_feature_commands[0];                                 \
+	     ((cmd) - cxl_feature_commands) < ARRAY_SIZE(cxl_feature_commands); (cmd)++)
+
+#define CXL_FEATURE_CMD(_id, sin, sout, _flags)                                \
+	[CXL_FEATURE_ID_##_id] = {                                             \
+	.info =	{                                                              \
+			.id = CXL_FEATURE_ID_##_id,                            \
+			.size_in = sin,                                        \
+			.size_out = sout,                                      \
+		},                                                             \
+	.opcode = CXL_MBOX_OP_##_id,                                           \
+	.flags = _flags,                                                       \
+	}
+
 #define CXL_VARIABLE_PAYLOAD	~0U
 /*
  * This table defines the supported mailbox commands for the driver. This table
@@ -69,6 +84,13 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
 	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
 };
 
+#define CXL_FEATURE_COMMAND_ID_MAX 3
+static struct cxl_mem_command cxl_feature_commands[CXL_FEATURE_COMMAND_ID_MAX] = {
+	CXL_FEATURE_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
+	CXL_FEATURE_CMD(GET_FEATURE, 0xf, CXL_VARIABLE_PAYLOAD, 0),
+	CXL_FEATURE_CMD(SET_FEATURE, CXL_VARIABLE_PAYLOAD, 0, 0),
+};
+
 /*
  * Commands that RAW doesn't permit. The rationale for each:
  *
@@ -212,6 +234,17 @@ static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)
 	return NULL;
 }
 
+static struct cxl_mem_command *cxl_find_feature_command(u16 opcode)
+{
+	struct cxl_mem_command *c;
+
+	cxl_for_each_feature_cmd(c)
+		if (c->opcode == opcode)
+			return c;
+
+	return NULL;
+}
+
 static const char *cxl_mem_opcode_to_name(u16 opcode)
 {
 	struct cxl_mem_command *c;
@@ -734,6 +767,14 @@ static void cxl_walk_cel(struct cxl_memdev_state *mds, size_t size, u8 *cel)
 		if (cmd) {
 			set_bit(cmd->info.id, cxl_mbox->enabled_cmds);
 			enabled++;
+		} else {
+			struct cxl_mem_command *fcmd =
+				cxl_find_feature_command(opcode);
+
+			if (fcmd) {
+				set_bit(fcmd->info.id, cxl_mbox->feature_cmds);
+				enabled++;
+			}
 		}
 
 		if (cxl_is_poison_command(opcode)) {
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index a0a49809cd76..55c55685cb39 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -490,6 +490,9 @@ enum cxl_opcode {
 	CXL_MBOX_OP_GET_LOG_CAPS	= 0x0402,
 	CXL_MBOX_OP_CLEAR_LOG           = 0x0403,
 	CXL_MBOX_OP_GET_SUP_LOG_SUBLIST = 0x0405,
+	CXL_MBOX_OP_GET_SUPPORTED_FEATURES	= 0x0500,
+	CXL_MBOX_OP_GET_FEATURE		= 0x0501,
+	CXL_MBOX_OP_SET_FEATURE		= 0x0502,
 	CXL_MBOX_OP_IDENTIFY		= 0x4000,
 	CXL_MBOX_OP_GET_PARTITION_INFO	= 0x4100,
 	CXL_MBOX_OP_SET_PARTITION_INFO	= 0x4101,
diff --git a/include/cxl/features.h b/include/cxl/features.h
index b92da1e92780..7a8be3c621a1 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -5,6 +5,13 @@
 
 struct cxl_mailbox;
 
+enum feature_cmds {
+	CXL_FEATURE_ID_GET_SUPPORTED_FEATURES = 0,
+	CXL_FEATURE_ID_GET_FEATURE,
+	CXL_FEATURE_ID_SET_FEATURE,
+	CXL_FEATURE_ID_MAX,
+};
+
 struct cxl_features {
 	int id;
 	struct device dev;
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index 6caab0d406ba..263fc346aeb1 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -58,6 +58,7 @@ struct cxl_mailbox {
 	struct device *host;
 	DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
 	DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
+	DECLARE_BITMAP(feature_cmds, CXL_FEATURE_ID_MAX);
 	size_t payload_size;
 	struct mutex mbox_mutex; /* lock to protect mailbox context */
 	struct rcuwait mbox_wait;
-- 
2.47.1

[PATCH v1 04/19] cxl: Add Get Supported Features command for kernel usage

[Dave Jiang]

CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
The command retrieve the list of supported device-specific features
(identified by UUID) and general information about each Feature.

The driver will retrieve the feature entries in order to make checks and
provide information for the Get Feature and Set Feature command. One of
the main piece of information retrieved are the effects a Set Feature
command would have for a particular feature.

Co-developed-by: Shiju Jose <shiju.jose@huawei.com>
Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v1:
- Change input param from cxlds to cxl_mbox
- Move mbox_out declaration inline. (Jonathan)
- Fix __counted_by() input. (Jonathan)
- Return count for cxl_get_supported_features_count(). (Dan)
- Remove goto from cxl_get_supported_features(). (Dan)
- Return cxl_get_supported_feature_entry() directly. (Dan)
- Drop user path enumeration. (Dan)
- Move to the feature driver model. (Dan)
- Add support for 0 feature data requested.
- Add missing increment of feature data ptr during copy.
---
 drivers/cxl/core/features.c |  28 +++++++
 drivers/cxl/core/mbox.c     |   3 +-
 drivers/cxl/cxl.h           |   2 +
 drivers/cxl/features.c      | 146 +++++++++++++++++++++++++++++++++++-
 include/cxl/features.h      |  32 ++++++++
 5 files changed, 209 insertions(+), 2 deletions(-)

diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
index eb6eb191a32e..66a4b82910e6 100644
--- a/drivers/cxl/core/features.c
+++ b/drivers/cxl/core/features.c
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /* Copyright(c) 2024-2025 Intel Corporation. All rights reserved. */
 #include <linux/device.h>
+#include <cxl/mailbox.h>
 #include "cxl.h"
 #include "core.h"
 
@@ -69,3 +70,30 @@ struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
 	return ERR_PTR(rc);
 }
 EXPORT_SYMBOL_NS_GPL(cxl_features_alloc, "CXL");
+
+struct cxl_feat_entry *
+cxl_get_supported_feature_entry(struct cxl_features *features,
+				const uuid_t *feat_uuid)
+{
+	struct cxl_feat_entry *feat_entry;
+	struct cxl_features_state *cfs;
+	int count;
+
+	cfs = dev_get_drvdata(&features->dev);
+	if (!cfs)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	if (!cfs->num_features)
+		return ERR_PTR(-ENOENT);
+
+	/* Check CXL dev supports the feature */
+	feat_entry = cfs->entries;
+	for (count = 0; count < cfs->num_features;
+	     count++, feat_entry++) {
+		if (uuid_equal(&feat_entry->uuid, feat_uuid))
+			return feat_entry;
+	}
+
+	return ERR_PTR(-ENOENT);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_supported_feature_entry, "CXL");
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 5e21ff99d70f..0b4946205910 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -234,7 +234,7 @@ static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)
 	return NULL;
 }
 
-static struct cxl_mem_command *cxl_find_feature_command(u16 opcode)
+struct cxl_mem_command *cxl_find_feature_command(u16 opcode)
 {
 	struct cxl_mem_command *c;
 
@@ -244,6 +244,7 @@ static struct cxl_mem_command *cxl_find_feature_command(u16 opcode)
 
 	return NULL;
 }
+EXPORT_SYMBOL_NS_GPL(cxl_find_feature_command, "CXL");
 
 static const char *cxl_mem_opcode_to_name(u16 opcode)
 {
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index ee29d1a1c8df..1284614d71d0 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -912,6 +912,8 @@ void cxl_coordinates_combine(struct access_coordinate *out,
 
 bool cxl_endpoint_decoder_reset_detected(struct cxl_port *port);
 
+struct cxl_mem_command *cxl_find_feature_command(u16 opcode);
+
 /*
  * Unit test builds overrides this to __weak, find the 'strong' version
  * of these symbols in tools/testing/cxl/.
diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
index 644add26975f..a5949312a4ab 100644
--- a/drivers/cxl/features.c
+++ b/drivers/cxl/features.c
@@ -3,20 +3,164 @@
 #include <linux/device.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <cxl/mailbox.h>
 #include <cxl/features.h>
 
 #include "cxl.h"
+#include "cxlmem.h"
+
+static void cxl_free_feature_entries(void *entries)
+{
+	kvfree(entries);
+}
+
+static int cxl_get_supported_features_count(struct cxl_mailbox *cxl_mbox)
+{
+	struct cxl_mbox_get_sup_feats_out mbox_out;
+	struct cxl_mbox_get_sup_feats_in mbox_in;
+	struct cxl_mbox_cmd mbox_cmd;
+	int rc;
+
+	memset(&mbox_in, 0, sizeof(mbox_in));
+	mbox_in.count = cpu_to_le32(sizeof(mbox_out));
+	memset(&mbox_out, 0, sizeof(mbox_out));
+	mbox_cmd = (struct cxl_mbox_cmd) {
+		.opcode = CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
+		.size_in = sizeof(mbox_in),
+		.payload_in = &mbox_in,
+		.size_out = sizeof(mbox_out),
+		.payload_out = &mbox_out,
+		.min_out = sizeof(mbox_out),
+	};
+	rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+	if (rc < 0)
+		return rc;
+
+	return le16_to_cpu(mbox_out.supported_feats);
+}
+
+static int cxl_get_supported_features(struct cxl_features_state *cfs)
+{
+	int remain_feats, max_size, max_feats, start, rc, hdr_size;
+	struct cxl_mailbox *cxl_mbox = cfs->features->cxl_mbox;
+	int feat_size = sizeof(struct cxl_feat_entry);
+	struct cxl_mbox_get_sup_feats_in mbox_in;
+	struct cxl_feat_entry *entry;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_mem_command *cmd;
+	int count;
+
+	/* Get supported features is optional, need to check */
+	cmd = cxl_find_feature_command(CXL_MBOX_OP_GET_SUPPORTED_FEATURES);
+	if (!cmd)
+		return -EOPNOTSUPP;
+	if (!test_bit(cmd->info.id, cxl_mbox->feature_cmds))
+		return -EOPNOTSUPP;
+
+	count = cxl_get_supported_features_count(cxl_mbox);
+	if (count == 0)
+		return 0;
+	if (count < 0)
+		return -ENXIO;
+
+	struct cxl_feat_entry *entries __free(kvfree) =
+		kvmalloc(count * sizeof(*entries), GFP_KERNEL);
+	if (!entries)
+		return -ENOMEM;
+
+	struct cxl_mbox_get_sup_feats_out *mbox_out __free(kvfree) =
+		kvmalloc(cxl_mbox->payload_size, GFP_KERNEL);
+	if (!mbox_out)
+		return -ENOMEM;
+
+	hdr_size = sizeof(*mbox_out);
+	max_size = cxl_mbox->payload_size - hdr_size;
+	/* max feat entries that can fit in mailbox max payload size */
+	max_feats = max_size / feat_size;
+	entry = entries;
+
+	start = 0;
+	remain_feats = count;
+	do {
+		int retrieved, alloc_size, copy_feats;
+		int num_entries;
+
+		if (remain_feats > max_feats) {
+			alloc_size = sizeof(*mbox_out) + max_feats * feat_size;
+			remain_feats = remain_feats - max_feats;
+			copy_feats = max_feats;
+		} else {
+			alloc_size = sizeof(*mbox_out) + remain_feats * feat_size;
+			copy_feats = remain_feats;
+			remain_feats = 0;
+		}
+
+		memset(&mbox_in, 0, sizeof(mbox_in));
+		mbox_in.count = cpu_to_le32(alloc_size);
+		mbox_in.start_idx = cpu_to_le16(start);
+		memset(mbox_out, 0, alloc_size);
+		mbox_cmd = (struct cxl_mbox_cmd) {
+			.opcode = CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
+			.size_in = sizeof(mbox_in),
+			.payload_in = &mbox_in,
+			.size_out = alloc_size,
+			.payload_out = mbox_out,
+			.min_out = hdr_size,
+		};
+		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+		if (rc < 0)
+			return rc;
+
+		if (mbox_cmd.size_out <= hdr_size)
+			return -ENXIO;
+
+		/*
+		 * Make sure retrieved out buffer is multiple of feature
+		 * entries.
+		 */
+		retrieved = mbox_cmd.size_out - hdr_size;
+		if (retrieved % feat_size)
+			return -ENXIO;
+
+		num_entries = le16_to_cpu(mbox_out->num_entries);
+		/*
+		 * If the reported output entries * defined entry size !=
+		 * retrieved output bytes, then the output package is incorrect.
+		 */
+		if (num_entries * feat_size != retrieved)
+			return -ENXIO;
+
+		memcpy(entry, mbox_out->ents, retrieved);
+		entry++;
+		/*
+		 * If the number of output entries is less than expected, add the
+		 * remaining entries to the next batch.
+		 */
+		remain_feats += copy_feats - num_entries;
+		start += num_entries;
+	} while (remain_feats);
+
+	cfs->num_features = count;
+	cfs->entries = no_free_ptr(entries);
+	return devm_add_action_or_reset(&cfs->features->dev,
+					cxl_free_feature_entries, cfs->entries);
+}
 
 static int cxl_features_probe(struct device *dev)
 {
 	struct cxl_features *features = to_cxl_features(dev);
+	int rc;
+
 	struct cxl_features_state *cfs __free(kfree) =
 		kzalloc(sizeof(*cfs), GFP_KERNEL);
-
 	if (!cfs)
 		return -ENOMEM;
 
 	cfs->features = features;
+	rc = cxl_get_supported_features(cfs);
+	if (rc)
+		return rc;
+
 	dev_set_drvdata(dev, no_free_ptr(cfs));
 
 	return 0;
diff --git a/include/cxl/features.h b/include/cxl/features.h
index 7a8be3c621a1..429b9782667c 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -3,6 +3,8 @@
 #ifndef __CXL_FEATURES_H__
 #define __CXL_FEATURES_H__
 
+#include <linux/uuid.h>
+
 struct cxl_mailbox;
 
 enum feature_cmds {
@@ -19,12 +21,42 @@ struct cxl_features {
 };
 #define to_cxl_features(dev) container_of(dev, struct cxl_features, dev)
 
+/* Get Supported Features (0x500h) CXL r3.1 8.2.9.6.1 */
+struct cxl_mbox_get_sup_feats_in {
+	__le32 count;
+	__le16 start_idx;
+	u8 reserved[2];
+} __packed;
+
+struct cxl_feat_entry {
+	uuid_t uuid;
+	__le16 id;
+	__le16 get_feat_size;
+	__le16 set_feat_size;
+	__le32 flags;
+	u8 get_feat_ver;
+	u8 set_feat_ver;
+	__le16 effects;
+	u8 reserved[18];
+} __packed;
+
+struct cxl_mbox_get_sup_feats_out {
+	__le16 num_entries;
+	__le16 supported_feats;
+	u8 reserved[4];
+	struct cxl_feat_entry ents[] __counted_by_le(num_entries);
+} __packed;
+
 struct cxl_features_state {
 	struct cxl_features *features;
 	int num_features;
+	struct cxl_feat_entry *entries;
 };
 
 struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
 					struct device *parent);
+struct cxl_feat_entry *
+cxl_get_supported_feature_entry(struct cxl_features *features,
+				const uuid_t *feat_uuid);
 
 #endif
-- 
2.47.1

[PATCH v1 05/19] cxl: Add features driver attribute to emit number of features supported

[Dave Jiang]

Enable sysfs attribute emission of the number of features supported by the
driver/device. This is useful for userspace to determine the number of features
to query for.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/features.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
index a5949312a4ab..0d1fc3da9e35 100644
--- a/drivers/cxl/features.c
+++ b/drivers/cxl/features.c
@@ -173,11 +173,38 @@ static void cxl_features_remove(struct device *dev)
 	kfree(cfs);
 }
 
+static ssize_t features_show(struct device *dev, struct device_attribute *attr,
+			     char *buf)
+{
+	struct cxl_features_state *cfs = dev_get_drvdata(dev);
+
+	if (!cfs)
+		return -ENOENT;
+
+	return sysfs_emit(buf, "%d\n", cfs->num_features);
+}
+
+static DEVICE_ATTR_RO(features);
+
+static struct attribute *cxl_features_attrs[] = {
+	&dev_attr_features.attr,
+	NULL
+};
+
+static struct attribute_group cxl_features_group = {
+	.attrs = cxl_features_attrs,
+};
+
+__ATTRIBUTE_GROUPS(cxl_features);
+
 static struct cxl_driver cxl_features_driver = {
 	.name = "cxl_features",
 	.probe = cxl_features_probe,
 	.remove = cxl_features_remove,
 	.id = CXL_DEVICE_FEATURES,
+	.drv = {
+		.dev_groups = cxl_features_groups,
+	},
 };
 
 module_cxl_driver(cxl_features_driver);
-- 
2.47.1

[PATCH v1 06/19] cxl/test: Add Get Supported Features mailbox command support

[Dave Jiang]

Add cxl-test emulation of Get Supported Features mailbox command.
Currently only adding a test feature with feature identifier of
all f's for testing.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Acked-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v1:
- Fix setting of num_entries endieness issue. (Jonathan)
---
 include/cxl/features.h       | 23 ++++++++++
 tools/testing/cxl/test/mem.c | 87 ++++++++++++++++++++++++++++++++++++
 2 files changed, 110 insertions(+)

diff --git a/include/cxl/features.h b/include/cxl/features.h
index 429b9782667c..8ff7d90932d6 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -28,6 +28,29 @@ struct cxl_mbox_get_sup_feats_in {
 	u8 reserved[2];
 } __packed;
 
+/* CXL spec r3.2 Table 8-87 command effects */
+#define CXL_CMD_CONFIG_CHANGE_COLD_RESET	BIT(0)
+#define CXL_CMD_CONFIG_CHANGE_IMMEDIATE		BIT(1)
+#define CXL_CMD_DATA_CHANGE_IMMEDIATE		BIT(2)
+#define CXL_CMD_POLICY_CHANGE_IMMEDIATE		BIT(3)
+#define CXL_CMD_LOG_CHANGE_IMMEDIATE		BIT(4)
+#define CXL_CMD_SECURITY_STATE_CHANGE		BIT(5)
+#define CXL_CMD_BACKGROUND			BIT(6)
+#define CXL_CMD_BGCMD_ABORT_SUPPORTED		BIT(7)
+#define CXL_CMD_EFFECTS_EXTEND			BIT(9)
+#define CXL_CMD_CONFIG_CHANGE_CONV_RESET	BIT(10)
+#define CXL_CMD_CONFIG_CHANGE_CXL_RESET		BIT(11)
+
+/*
+ * CXL spec r3.2 Table 8-109
+ * Get Supported Features Supported Feature Entry
+ * Atribute Flags
+ */
+#define CXL_FEATURE_F_CHANGEABLE		BIT(0)
+#define CXL_FEATURE_F_PERSIST_FW_UPDATE		BIT(4)
+#define CXL_FEATURE_F_DEFAULT_SEL		BIT(5)
+#define CXL_FEATURE_F_SAVED_SEL			BIT(6)
+
 struct cxl_feat_entry {
 	uuid_t uuid;
 	__le16 id;
diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index 347c1e7b37bd..208337432bfe 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -44,6 +44,10 @@ static struct cxl_cel_entry mock_cel[] = {
 		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_SUPPORTED_LOGS),
 		.effect = CXL_CMD_EFFECT_NONE,
 	},
+	{
+		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_SUPPORTED_FEATURES),
+		.effect = CXL_CMD_EFFECT_NONE,
+	},
 	{
 		.opcode = cpu_to_le16(CXL_MBOX_OP_IDENTIFY),
 		.effect = CXL_CMD_EFFECT_NONE,
@@ -1337,6 +1341,67 @@ static int mock_activate_fw(struct cxl_mockmem_data *mdata,
 	return -EINVAL;
 }
 
+#define CXL_VENDOR_FEATURE_TEST							\
+	UUID_INIT(0xffffffff, 0xffff, 0xffff, 0xff, 0xff, 0xff, 0xff, 0xff,	\
+		  0xff, 0xff, 0xff)
+
+static void fill_feature_vendor_test(struct cxl_feat_entry *feat)
+{
+	feat->uuid = CXL_VENDOR_FEATURE_TEST;
+	feat->id = 0;
+	feat->get_feat_size = cpu_to_le16(0x4);
+	feat->set_feat_size = cpu_to_le16(0x4);
+	feat->flags = cpu_to_le32(CXL_FEATURE_F_CHANGEABLE |
+				  CXL_FEATURE_F_DEFAULT_SEL |
+				  CXL_FEATURE_F_SAVED_SEL);
+	feat->get_feat_ver = 1;
+	feat->set_feat_ver = 1;
+	feat->effects = cpu_to_le16(CXL_CMD_CONFIG_CHANGE_COLD_RESET |
+				    CXL_CMD_EFFECTS_EXTEND);
+}
+
+static int mock_get_supported_features(struct cxl_mockmem_data *mdata,
+				       struct cxl_mbox_cmd *cmd)
+{
+	struct cxl_mbox_get_sup_feats_in *in = cmd->payload_in;
+	struct cxl_mbox_get_sup_feats_out *out = cmd->payload_out;
+	struct cxl_feat_entry *feat;
+	u16 start_idx, count;
+
+	if (cmd->size_out < sizeof(*out)) {
+		cmd->return_code = CXL_MBOX_CMD_RC_PAYLOADLEN;
+		return -EINVAL;
+	}
+
+	/*
+	 * Current emulation only supports 1 feature
+	 */
+	start_idx = le16_to_cpu(in->start_idx);
+	if (start_idx != 0) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	count = le16_to_cpu(in->count);
+	if (count < sizeof(*out)) {
+		cmd->return_code = CXL_MBOX_CMD_RC_PAYLOADLEN;
+		return -EINVAL;
+	}
+
+	out->supported_feats = cpu_to_le16(1);
+	cmd->return_code = 0;
+	if (count < sizeof(*out) + sizeof(*feat)) {
+		out->num_entries = 0;
+		return 0;
+	}
+
+	out->num_entries = cpu_to_le16(1);
+	feat = out->ents;
+	fill_feature_vendor_test(feat);
+
+	return 0;
+}
+
 static int cxl_mock_mbox_send(struct cxl_mailbox *cxl_mbox,
 			      struct cxl_mbox_cmd *cmd)
 {
@@ -1422,6 +1487,9 @@ static int cxl_mock_mbox_send(struct cxl_mailbox *cxl_mbox,
 	case CXL_MBOX_OP_ACTIVATE_FW:
 		rc = mock_activate_fw(mdata, cmd);
 		break;
+	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
+		rc = mock_get_supported_features(mdata, cmd);
+		break;
 	default:
 		break;
 	}
@@ -1469,6 +1537,21 @@ static int cxl_mock_mailbox_create(struct cxl_dev_state *cxlds)
 	return 0;
 }
 
+static int cxl_mock_setup_features(struct cxl_memdev_state *mds)
+{
+	struct cxl_dev_state *cxlds = &mds->cxlds;
+	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
+	struct cxl_features *features;
+
+	features = cxl_features_alloc(cxl_mbox, cxlds->dev);
+	if (IS_ERR(features))
+		return PTR_ERR(features);
+
+	cxl_mbox->features = features;
+
+	return 0;
+}
+
 static int cxl_mock_mem_probe(struct platform_device *pdev)
 {
 	struct device *dev = &pdev->dev;
@@ -1524,6 +1607,10 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 	if (rc)
 		return rc;
 
+	rc = cxl_mock_setup_features(mds);
+	if (rc)
+		dev_dbg(dev, "cxl_test: No features setup\n");
+
 	rc = cxl_poison_state_init(mds);
 	if (rc)
 		return rc;
-- 
2.47.1

[PATCH v1 07/19] cxl/mbox: Add GET_FEATURE mailbox command

[Dave Jiang]

From: Shiju Jose <shiju.jose@huawei.com>

Add support for GET_FEATURE mailbox command.

CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
The settings of a feature can be retrieved using Get Feature command.
CXL spec 3.1 section 8.2.9.6.2 describes Get Feature command.

Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v1:
- pass in cxl_mbox instead of cxlds (Dan)
- Move to the feature driver model. (Dan)
---
 drivers/cxl/core/features.c | 74 +++++++++++++++++++++++++++++++++++++
 drivers/cxl/features.c      |  6 +--
 include/cxl/features.h      | 27 ++++++++++++++
 3 files changed, 102 insertions(+), 5 deletions(-)

diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
index 66a4b82910e6..ab9386b53a95 100644
--- a/drivers/cxl/core/features.c
+++ b/drivers/cxl/core/features.c
@@ -4,6 +4,7 @@
 #include <cxl/mailbox.h>
 #include "cxl.h"
 #include "core.h"
+#include "cxlmem.h"
 
 #define CXL_FEATURE_MAX_DEVS 65536
 static DEFINE_IDA(cxl_features_ida);
@@ -97,3 +98,76 @@ cxl_get_supported_feature_entry(struct cxl_features *features,
 	return ERR_PTR(-ENOENT);
 }
 EXPORT_SYMBOL_NS_GPL(cxl_get_supported_feature_entry, "CXL");
+
+bool cxl_feature_enabled(struct cxl_features_state *cfs, u16 opcode)
+{
+	struct cxl_mailbox *cxl_mbox = cfs->features->cxl_mbox;
+	struct cxl_mem_command *cmd;
+
+	cmd = cxl_find_feature_command(opcode);
+	if (!cmd)
+		return false;
+
+	return test_bit(cmd->info.id, cxl_mbox->feature_cmds);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_feature_enabled, "CXL");
+
+size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
+		       enum cxl_get_feat_selection selection,
+		       void *feat_out, size_t feat_out_size, u16 offset,
+		       u16 *return_code)
+{
+	size_t data_to_rd_size, size_out;
+	struct cxl_features_state *cfs;
+	struct cxl_mbox_get_feat_in pi;
+	struct cxl_mailbox *cxl_mbox;
+	struct cxl_mbox_cmd mbox_cmd;
+	size_t data_rcvd_size = 0;
+	int rc;
+
+	if (return_code)
+		*return_code = CXL_MBOX_CMD_RC_INPUT;
+
+	cfs = dev_get_drvdata(&features->dev);
+	if (!cfs)
+		return 0;
+
+	if (!cxl_feature_enabled(cfs, CXL_MBOX_OP_GET_FEATURE))
+		return 0;
+
+	if (!feat_out || !feat_out_size)
+		return 0;
+
+	cxl_mbox = features->cxl_mbox;
+	size_out = min(feat_out_size, cxl_mbox->payload_size);
+	pi.uuid = feat_uuid;
+	pi.selection = selection;
+	do {
+		data_to_rd_size = min(feat_out_size - data_rcvd_size,
+				      cxl_mbox->payload_size);
+		pi.offset = cpu_to_le16(offset + data_rcvd_size);
+		pi.count = cpu_to_le16(data_to_rd_size);
+
+		mbox_cmd = (struct cxl_mbox_cmd) {
+			.opcode = CXL_MBOX_OP_GET_FEATURE,
+			.size_in = sizeof(pi),
+			.payload_in = &pi,
+			.size_out = size_out,
+			.payload_out = feat_out + data_rcvd_size,
+			.min_out = data_to_rd_size,
+		};
+		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+		if (rc < 0 || !mbox_cmd.size_out) {
+			if (return_code)
+				*return_code = mbox_cmd.return_code;
+			return 0;
+		}
+		data_rcvd_size += mbox_cmd.size_out;
+	} while (data_rcvd_size < feat_out_size);
+
+	if (return_code)
+		*return_code = CXL_MBOX_CMD_RC_SUCCESS;
+
+	return data_rcvd_size;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_feature, "CXL");
diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
index 0d1fc3da9e35..6b061bef3a6a 100644
--- a/drivers/cxl/features.c
+++ b/drivers/cxl/features.c
@@ -47,14 +47,10 @@ static int cxl_get_supported_features(struct cxl_features_state *cfs)
 	struct cxl_mbox_get_sup_feats_in mbox_in;
 	struct cxl_feat_entry *entry;
 	struct cxl_mbox_cmd mbox_cmd;
-	struct cxl_mem_command *cmd;
 	int count;
 
 	/* Get supported features is optional, need to check */
-	cmd = cxl_find_feature_command(CXL_MBOX_OP_GET_SUPPORTED_FEATURES);
-	if (!cmd)
-		return -EOPNOTSUPP;
-	if (!test_bit(cmd->info.id, cxl_mbox->feature_cmds))
+	if (!cxl_feature_enabled(cfs, CXL_MBOX_OP_GET_SUPPORTED_FEATURES))
 		return -EOPNOTSUPP;
 
 	count = cxl_get_supported_features_count(cxl_mbox);
diff --git a/include/cxl/features.h b/include/cxl/features.h
index 8ff7d90932d6..872edd0a4417 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -76,10 +76,37 @@ struct cxl_features_state {
 	struct cxl_feat_entry *entries;
 };
 
+/*
+ * Get Feature CXL 3.1 Spec 8.2.9.6.2
+ */
+
+/*
+ * Get Feature input payload
+ * CXL rev 3.1 section 8.2.9.6.2 Table 8-99
+ */
+enum cxl_get_feat_selection {
+	CXL_GET_FEAT_SEL_CURRENT_VALUE,
+	CXL_GET_FEAT_SEL_DEFAULT_VALUE,
+	CXL_GET_FEAT_SEL_SAVED_VALUE,
+	CXL_GET_FEAT_SEL_MAX
+};
+
+struct cxl_mbox_get_feat_in {
+	uuid_t uuid;
+	__le16 offset;
+	__le16 count;
+	u8 selection;
+}  __packed;
+
+bool cxl_feature_enabled(struct cxl_features_state *cfs, u16 opcode);
 struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
 					struct device *parent);
 struct cxl_feat_entry *
 cxl_get_supported_feature_entry(struct cxl_features *features,
 				const uuid_t *feat_uuid);
+size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
+		       enum cxl_get_feat_selection selection,
+		       void *feat_out, size_t feat_out_size, u16 offset,
+		       u16 *return_code);
 
 #endif
-- 
2.47.1

[PATCH v1 08/19] cxl/mbox: Add SET_FEATURE mailbox command

[Dave Jiang]

From: Shiju Jose <shiju.jose@huawei.com>

Add support for SET_FEATURE mailbox command.

CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
CXL devices supports features with changeable attributes.
The settings of a feature can be optionally modified using Set Feature
command.
CXL spec 3.1 section 8.2.9.6.3 describes Set Feature command.

Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v1:
- CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER enum must be set to 0.
---
 drivers/cxl/core/features.c  | 92 ++++++++++++++++++++++++++++++++++++
 include/cxl/features.h       | 32 +++++++++++++
 tools/testing/cxl/test/mem.c |  1 +
 3 files changed, 125 insertions(+)

diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
index ab9386b53a95..932e82b52f90 100644
--- a/drivers/cxl/core/features.c
+++ b/drivers/cxl/core/features.c
@@ -171,3 +171,95 @@ size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
 	return data_rcvd_size;
 }
 EXPORT_SYMBOL_NS_GPL(cxl_get_feature, "CXL");
+
+/*
+ * FEAT_DATA_MIN_PAYLOAD_SIZE - min extra number of bytes should be
+ * available in the mailbox for storing the actual feature data so that
+ * the feature data transfer would work as expected.
+ */
+#define FEAT_DATA_MIN_PAYLOAD_SIZE 10
+int cxl_set_feature(struct cxl_features *features,
+		    const uuid_t feat_uuid, u8 feat_version,
+		    void *feat_data, size_t feat_data_size,
+		    u32 feat_flag, u16 offset, u16 *return_code)
+{
+	struct cxl_memdev_set_feat_pi {
+		struct cxl_mbox_set_feat_hdr hdr;
+		u8 feat_data[];
+	}  __packed;
+	size_t data_in_size, data_sent_size = 0;
+	struct cxl_features_state *cfs;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_mailbox *cxl_mbox;
+	size_t hdr_size;
+	int rc = 0;
+
+	if (return_code)
+		*return_code = CXL_MBOX_CMD_RC_INPUT;
+
+	cfs = dev_get_drvdata(&features->dev);
+	if (!cfs)
+		return -EOPNOTSUPP;
+
+	if (!cxl_feature_enabled(cfs, CXL_MBOX_OP_SET_FEATURE))
+		return -EOPNOTSUPP;
+
+	cxl_mbox = features->cxl_mbox;
+
+	struct cxl_memdev_set_feat_pi *pi __free(kfree) =
+					kmalloc(cxl_mbox->payload_size, GFP_KERNEL);
+	pi->hdr.uuid = feat_uuid;
+	pi->hdr.version = feat_version;
+	feat_flag &= ~CXL_SET_FEAT_FLAG_DATA_TRANSFER_MASK;
+	feat_flag |= CXL_SET_FEAT_FLAG_DATA_SAVED_ACROSS_RESET;
+	hdr_size = sizeof(pi->hdr);
+	/*
+	 * Check minimum mbox payload size is available for
+	 * the feature data transfer.
+	 */
+	if (hdr_size + FEAT_DATA_MIN_PAYLOAD_SIZE > cxl_mbox->payload_size)
+		return -ENOMEM;
+
+	if ((hdr_size + feat_data_size) <= cxl_mbox->payload_size) {
+		pi->hdr.flags = cpu_to_le32(feat_flag |
+				       CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER);
+		data_in_size = feat_data_size;
+	} else {
+		pi->hdr.flags = cpu_to_le32(feat_flag |
+				       CXL_SET_FEAT_FLAG_INITIATE_DATA_TRANSFER);
+		data_in_size = cxl_mbox->payload_size - hdr_size;
+	}
+
+	do {
+		pi->hdr.offset = cpu_to_le16(offset + data_sent_size);
+		memcpy(pi->feat_data, feat_data + data_sent_size, data_in_size);
+		mbox_cmd = (struct cxl_mbox_cmd) {
+			.opcode = CXL_MBOX_OP_SET_FEATURE,
+			.size_in = hdr_size + data_in_size,
+			.payload_in = pi,
+		};
+		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+		if (rc < 0) {
+			if (return_code)
+				*return_code = mbox_cmd.return_code;
+			return rc;
+		}
+
+		data_sent_size += data_in_size;
+		if (data_sent_size >= feat_data_size) {
+			if (return_code)
+				*return_code = CXL_MBOX_CMD_RC_SUCCESS;
+			return 0;
+		}
+
+		if ((feat_data_size - data_sent_size) <= (cxl_mbox->payload_size - hdr_size)) {
+			data_in_size = feat_data_size - data_sent_size;
+			pi->hdr.flags = cpu_to_le32(feat_flag |
+					       CXL_SET_FEAT_FLAG_FINISH_DATA_TRANSFER);
+		} else {
+			pi->hdr.flags = cpu_to_le32(feat_flag |
+					       CXL_SET_FEAT_FLAG_CONTINUE_DATA_TRANSFER);
+		}
+	} while (true);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_set_feature, "CXL");
diff --git a/include/cxl/features.h b/include/cxl/features.h
index 872edd0a4417..370c8d57c728 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -98,6 +98,35 @@ struct cxl_mbox_get_feat_in {
 	u8 selection;
 }  __packed;
 
+/*
+ * Set Feature CXL 3.1 Spec 8.2.9.6.3
+ */
+
+/*
+ * Set Feature input payload
+ * CXL rev 3.1 section 8.2.9.6.3 Table 8-101
+ */
+/* Set Feature : Payload in flags */
+#define CXL_SET_FEAT_FLAG_DATA_TRANSFER_MASK	GENMASK(2, 0)
+enum cxl_set_feat_flag_data_transfer {
+	CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER = 0,
+	CXL_SET_FEAT_FLAG_INITIATE_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_CONTINUE_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_FINISH_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_ABORT_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_DATA_TRANSFER_MAX
+};
+
+#define CXL_SET_FEAT_FLAG_DATA_SAVED_ACROSS_RESET	BIT(3)
+
+struct cxl_mbox_set_feat_hdr {
+	uuid_t uuid;
+	__le32 flags;
+	__le16 offset;
+	u8 version;
+	u8 rsvd[9];
+}  __packed;
+
 bool cxl_feature_enabled(struct cxl_features_state *cfs, u16 opcode);
 struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
 					struct device *parent);
@@ -108,5 +137,8 @@ size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
 		       enum cxl_get_feat_selection selection,
 		       void *feat_out, size_t feat_out_size, u16 offset,
 		       u16 *return_code);
+int cxl_set_feature(struct cxl_features *features, const uuid_t feat_uuid,
+		    u8 feat_version, void *feat_data, size_t feat_data_size,
+		    u32 feat_flag, u16 offset, u16 *return_code);
 
 #endif
diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index 208337432bfe..2b111142b2c0 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -9,6 +9,7 @@
 #include <linux/sizes.h>
 #include <linux/bits.h>
 #include <cxl/mailbox.h>
+#include <cxl/features.h>
 #include <linux/unaligned.h>
 #include <crypto/sha2.h>
 #include <cxlmem.h>
-- 
2.47.1

[PATCH v1 09/19] cxl: Setup exclusive CXL features that are reserved for the kernel

[Dave Jiang]

Certain features will be exclusively used by components such as in
kernel RAS driver. Setup an exclusion list that can be later filtered
out before exposing to user space.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/features.c | 22 ++++++++++++++++++++++
 drivers/cxl/features.c      |  6 +++++-
 include/cxl/features.h      | 34 ++++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 1 deletion(-)

diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
index 932e82b52f90..c836b3a64561 100644
--- a/drivers/cxl/core/features.c
+++ b/drivers/cxl/core/features.c
@@ -6,6 +6,17 @@
 #include "core.h"
 #include "cxlmem.h"
 
+static const uuid_t cxl_exclusive_feats[] = {
+	CXL_FEAT_PATROL_SCRUB_UUID,
+	CXL_FEAT_ECS_UUID,
+	CXL_FEAT_SPPR_UUID,
+	CXL_FEAT_HPPR_UUID,
+	CXL_FEAT_CACHELINE_SPARING_UUID,
+	CXL_FEAT_ROW_SPARING_UUID,
+	CXL_FEAT_BANK_SPARING_UUID,
+	CXL_FEAT_RANK_SPARING_UUID,
+};
+
 #define CXL_FEATURE_MAX_DEVS 65536
 static DEFINE_IDA(cxl_features_ida);
 
@@ -263,3 +274,14 @@ int cxl_set_feature(struct cxl_features *features,
 	} while (true);
 }
 EXPORT_SYMBOL_NS_GPL(cxl_set_feature, "CXL");
+
+bool is_cxl_feature_exclusive(struct cxl_feat_entry *entry)
+{
+	for (int i = 0; i < ARRAY_SIZE(cxl_exclusive_feats); i++) {
+		if (uuid_equal(&entry->uuid, &cxl_exclusive_feats[i]))
+			return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_NS_GPL(is_cxl_feature_exclusive, "CXL");
diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
index 6b061bef3a6a..1c4bcd7465cb 100644
--- a/drivers/cxl/features.c
+++ b/drivers/cxl/features.c
@@ -47,6 +47,7 @@ static int cxl_get_supported_features(struct cxl_features_state *cfs)
 	struct cxl_mbox_get_sup_feats_in mbox_in;
 	struct cxl_feat_entry *entry;
 	struct cxl_mbox_cmd mbox_cmd;
+	int user_feats = 0;
 	int count;
 
 	/* Get supported features is optional, need to check */
@@ -127,6 +128,8 @@ static int cxl_get_supported_features(struct cxl_features_state *cfs)
 			return -ENXIO;
 
 		memcpy(entry, mbox_out->ents, retrieved);
+		if (!is_cxl_feature_exclusive(entry))
+			user_feats++;
 		entry++;
 		/*
 		 * If the number of output entries is less than expected, add the
@@ -138,6 +141,7 @@ static int cxl_get_supported_features(struct cxl_features_state *cfs)
 
 	cfs->num_features = count;
 	cfs->entries = no_free_ptr(entries);
+	cfs->num_user_features = user_feats;
 	return devm_add_action_or_reset(&cfs->features->dev,
 					cxl_free_feature_entries, cfs->entries);
 }
@@ -177,7 +181,7 @@ static ssize_t features_show(struct device *dev, struct device_attribute *attr,
 	if (!cfs)
 		return -ENOENT;
 
-	return sysfs_emit(buf, "%d\n", cfs->num_features);
+	return sysfs_emit(buf, "%d\n", cfs->num_user_features);
 }
 
 static DEVICE_ATTR_RO(features);
diff --git a/include/cxl/features.h b/include/cxl/features.h
index 370c8d57c728..f746358ba5f9 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -5,6 +5,38 @@
 
 #include <linux/uuid.h>
 
+#define CXL_FEAT_PATROL_SCRUB_UUID						\
+	UUID_INIT(0x96dad7d6, 0xfde8, 0x482b, 0xa7, 0x33, 0x75, 0x77, 0x4e,	\
+		  0x06, 0xdb, 0x8a)
+
+#define CXL_FEAT_ECS_UUID							\
+	UUID_INIT(0xe5b13f22, 0x2328, 0x4a14, 0xb8, 0xba, 0xb9, 0x69, 0x1e,	\
+		  0x89, 0x33, 0x86)
+
+#define CXL_FEAT_SPPR_UUID							\
+	UUID_INIT(0x892ba475, 0xfad8, 0x474e, 0x9d, 0x3e, 0x69, 0x2c, 0x91,	\
+		  0x75, 0x68, 0xbb)
+
+#define CXL_FEAT_HPPR_UUID							\
+	UUID_INIT(0x80ea4521, 0x786f, 0x4127, 0xaf, 0xb1, 0xec, 0x74, 0x59,	\
+		  0xfb, 0x0e, 0x24)
+
+#define CXL_FEAT_CACHELINE_SPARING_UUID						\
+	UUID_INIT(0x96C33386, 0x91dd, 0x44c7, 0x9e, 0xcb, 0xfd, 0xaf, 0x65,	\
+		  0x03, 0xba, 0xc4)
+
+#define CXL_FEAT_ROW_SPARING_UUID						\
+	UUID_INIT(0x450ebf67, 0xb135, 0x4f97, 0xa4, 0x98, 0xc2, 0xd5, 0x7f,	\
+		  0x27, 0x9b, 0xed)
+
+#define CXL_FEAT_BANK_SPARING_UUID						\
+	UUID_INIT(0x78b79636, 0x90ac, 0x4b64, 0xa4, 0xef, 0xfa, 0xac, 0x5d,	\
+		  0x18, 0xa8, 0x63)
+
+#define CXL_FEAT_RANK_SPARING_UUID						\
+	UUID_INIT(0x34dbaff5, 0x0552, 0x4281, 0x8f, 0x76, 0xda, 0x0b, 0x5e,	\
+		  0x7a, 0x76, 0xa7)
+
 struct cxl_mailbox;
 
 enum feature_cmds {
@@ -73,6 +105,7 @@ struct cxl_mbox_get_sup_feats_out {
 struct cxl_features_state {
 	struct cxl_features *features;
 	int num_features;
+	int num_user_features;
 	struct cxl_feat_entry *entries;
 };
 
@@ -140,5 +173,6 @@ size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
 int cxl_set_feature(struct cxl_features *features, const uuid_t feat_uuid,
 		    u8 feat_version, void *feat_data, size_t feat_data_size,
 		    u32 feat_flag, u16 offset, u16 *return_code);
+bool is_cxl_feature_exclusive(struct cxl_feat_entry *entry);
 
 #endif
-- 
2.47.1

[PATCH v1 10/19] cxl: Add FWCTL support to the CXL features driver

[Dave Jiang]

Add fwctl support code to allow sending of CXL feature commands from
userspace through as ioctls via FWCTL. Provide initial setup bits.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v1:
- Add missing dev_set_name() (Jason)
---
 drivers/cxl/Kconfig        |  1 +
 drivers/cxl/features.c     | 44 ++++++++++++++++++++++++++++++++++++--
 include/cxl/features.h     |  2 ++
 include/uapi/fwctl/fwctl.h |  1 +
 4 files changed, 46 insertions(+), 2 deletions(-)

diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 9a6ffd81ac0e..5de06c3806ce 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -148,6 +148,7 @@ config CXL_REGION_INVALIDATION_TEST
 
 config CXL_FEATURES
 	tristate "CXL: Features support"
+	depends on FWCTL
 	default CXL_BUS
 	help
 	  Enable CXL features support that are tied to a CXL mailbox.
diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
index 1c4bcd7465cb..5a2b771586d3 100644
--- a/drivers/cxl/features.c
+++ b/drivers/cxl/features.c
@@ -9,6 +9,37 @@
 #include "cxl.h"
 #include "cxlmem.h"
 
+static int cxlctl_open_uctx(struct fwctl_uctx *uctx)
+{
+	return 0;
+}
+
+static void cxlctl_close_uctx(struct fwctl_uctx *uctx)
+{
+}
+
+static void *cxlctl_info(struct fwctl_uctx *uctx, size_t *length)
+{
+	/* Place holder */
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
+static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
+			   void *rpc_in, size_t in_len, size_t *out_len)
+{
+	/* Place holder */
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
+static const struct fwctl_ops cxlctl_ops = {
+	.device_type = FWCTL_DEVICE_TYPE_CXL,
+	.uctx_size = sizeof(struct cxl_features),
+	.open_uctx = cxlctl_open_uctx,
+	.close_uctx = cxlctl_close_uctx,
+	.info = cxlctl_info,
+	.fw_rpc = cxlctl_fw_rpc,
+};
+
 static void cxl_free_feature_entries(void *entries)
 {
 	kvfree(entries);
@@ -146,13 +177,16 @@ static int cxl_get_supported_features(struct cxl_features_state *cfs)
 					cxl_free_feature_entries, cfs->entries);
 }
 
+DEFINE_FREE(cfs, struct cxl_features_state *, if (_T) fwctl_put(&_T->fwctl))
+
 static int cxl_features_probe(struct device *dev)
 {
 	struct cxl_features *features = to_cxl_features(dev);
 	int rc;
 
-	struct cxl_features_state *cfs __free(kfree) =
-		kzalloc(sizeof(*cfs), GFP_KERNEL);
+	struct cxl_features_state *cfs __free(cfs) =
+		fwctl_alloc_device(dev, &cxlctl_ops,
+				   struct cxl_features_state, fwctl);
 	if (!cfs)
 		return -ENOMEM;
 
@@ -161,6 +195,10 @@ static int cxl_features_probe(struct device *dev)
 	if (rc)
 		return rc;
 
+	rc = fwctl_register(&cfs->fwctl);
+	if (rc)
+		return rc;
+
 	dev_set_drvdata(dev, no_free_ptr(cfs));
 
 	return 0;
@@ -170,6 +208,7 @@ static void cxl_features_remove(struct device *dev)
 {
 	struct cxl_features_state *cfs = dev_get_drvdata(dev);
 
+	fwctl_unregister(&cfs->fwctl);
 	kfree(cfs);
 }
 
@@ -212,4 +251,5 @@ module_cxl_driver(cxl_features_driver);
 MODULE_DESCRIPTION("CXL: Features");
 MODULE_LICENSE("GPL");
 MODULE_IMPORT_NS("CXL");
+MODULE_IMPORT_NS("FWCTL");
 MODULE_ALIAS_CXL(CXL_DEVICE_FEATURES);
diff --git a/include/cxl/features.h b/include/cxl/features.h
index f746358ba5f9..69697c069e63 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -4,6 +4,7 @@
 #define __CXL_FEATURES_H__
 
 #include <linux/uuid.h>
+#include <linux/fwctl.h>
 
 #define CXL_FEAT_PATROL_SCRUB_UUID						\
 	UUID_INIT(0x96dad7d6, 0xfde8, 0x482b, 0xa7, 0x33, 0x75, 0x77, 0x4e,	\
@@ -103,6 +104,7 @@ struct cxl_mbox_get_sup_feats_out {
 } __packed;
 
 struct cxl_features_state {
+	struct fwctl_device fwctl;
 	struct cxl_features *features;
 	int num_features;
 	int num_user_features;
diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
index f9b27fb5c161..2c3817592308 100644
--- a/include/uapi/fwctl/fwctl.h
+++ b/include/uapi/fwctl/fwctl.h
@@ -43,6 +43,7 @@ enum {
 enum fwctl_device_type {
 	FWCTL_DEVICE_TYPE_ERROR = 0,
 	FWCTL_DEVICE_TYPE_MLX5 = 1,
+	FWCTL_DEVICE_TYPE_CXL,
 };
 
 /**
-- 
2.47.1

[PATCH v1 11/19] cxl: Add support for get driver information

[Dave Jiang]

Add definition for fwctl_ops->info() to return driver information. The
function will return a mask of the feature mailbox commands supported
by the fwctl char device.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v1:
- Add missed setting of *length for ->info().
- Use BIT() instead of enum directly.
---
 drivers/cxl/features.c   | 30 ++++++++++++++++++++++++++++--
 include/cxl/features.h   |  7 -------
 include/cxl/mailbox.h    |  1 +
 include/uapi/fwctl/cxl.h | 31 +++++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+), 9 deletions(-)
 create mode 100644 include/uapi/fwctl/cxl.h

diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
index 5a2b771586d3..cc72e73ae8d6 100644
--- a/drivers/cxl/features.c
+++ b/drivers/cxl/features.c
@@ -9,6 +9,8 @@
 #include "cxl.h"
 #include "cxlmem.h"
 
+#define to_cxl_features_state(fwctl) container_of(fwctl, struct cxl_features_state, fwctl)
+
 static int cxlctl_open_uctx(struct fwctl_uctx *uctx)
 {
 	return 0;
@@ -18,10 +20,34 @@ static void cxlctl_close_uctx(struct fwctl_uctx *uctx)
 {
 }
 
+static void set_command_mask(struct cxl_features_state *cfs, u32 *mask)
+{
+	struct cxl_mailbox *cxl_mbox = cfs->features->cxl_mbox;
+
+	*mask = 0;
+	if (test_bit(CXL_FEATURE_ID_GET_SUPPORTED_FEATURES,
+		     cxl_mbox->feature_cmds))
+		*mask |= BIT(CXL_FEATURE_ID_GET_SUPPORTED_FEATURES);
+	if (test_bit(CXL_FEATURE_ID_GET_FEATURE, cxl_mbox->feature_cmds))
+		*mask |= BIT(CXL_FEATURE_ID_GET_FEATURE);
+	if (test_bit(CXL_FEATURE_ID_SET_FEATURE, cxl_mbox->feature_cmds))
+		*mask |= BIT(CXL_FEATURE_ID_SET_FEATURE);
+}
+
 static void *cxlctl_info(struct fwctl_uctx *uctx, size_t *length)
 {
-	/* Place holder */
-	return ERR_PTR(-EOPNOTSUPP);
+	struct fwctl_device *fwctl = uctx->fwctl;
+	struct cxl_features_state *cfs = to_cxl_features_state(fwctl);
+	struct fwctl_info_cxl *info;
+
+	info = kzalloc(sizeof(*info), GFP_KERNEL);
+	if (!info)
+		return ERR_PTR(-ENOMEM);
+
+	set_command_mask(cfs, &info->cmd_mask);
+	*length = sizeof(*info);
+
+	return info;
 }
 
 static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
diff --git a/include/cxl/features.h b/include/cxl/features.h
index 69697c069e63..e38ee777328d 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -40,13 +40,6 @@
 
 struct cxl_mailbox;
 
-enum feature_cmds {
-	CXL_FEATURE_ID_GET_SUPPORTED_FEATURES = 0,
-	CXL_FEATURE_ID_GET_FEATURE,
-	CXL_FEATURE_ID_SET_FEATURE,
-	CXL_FEATURE_ID_MAX,
-};
-
 struct cxl_features {
 	int id;
 	struct device dev;
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index 263fc346aeb1..1157b19175a5 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -4,6 +4,7 @@
 #define __CXL_MBOX_H__
 #include <linux/rcuwait.h>
 #include <cxl/features.h>
+#include <uapi/fwctl/cxl.h>
 #include <uapi/linux/cxl_mem.h>
 
 /**
diff --git a/include/uapi/fwctl/cxl.h b/include/uapi/fwctl/cxl.h
new file mode 100644
index 000000000000..79b822dbfafd
--- /dev/null
+++ b/include/uapi/fwctl/cxl.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (c) 2024, Intel Corporation
+ *
+ * These are definitions for the mailbox command interface of CXL subsystem.
+ */
+#ifndef _UAPI_FWCTL_CXL_H_
+#define _UAPI_FWCTL_CXL_H_
+
+#include <linux/types.h>
+
+enum feature_cmds {
+	CXL_FEATURE_ID_GET_SUPPORTED_FEATURES = 0,
+	CXL_FEATURE_ID_GET_FEATURE,
+	CXL_FEATURE_ID_SET_FEATURE,
+	CXL_FEATURE_ID_MAX,
+};
+
+/**
+ * struct fwctl_info_cxl - ioctl(FWCTL_INFO) out_device_data
+ * @cmd_mask: Mask indicate which commands are supported based on 'enum feature_cmds'
+ *
+ * Return basic information about the FW interface available.
+ *
+ * nr_commands is number of hardware commands the driver supports. Use
+ * FWCTL_CMD_HW_INFO ioctl to request additional information.
+ */
+struct fwctl_info_cxl {
+	__u32 cmd_mask;
+};
+#endif
-- 
2.47.1

[PATCH v1 12/19] cxl: Move cxl_mem.h under uapi to cxl exclusive directory

[Dave Jiang]

In preparation of more cxl headers, create a cxl directory under
uapi.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 MAINTAINERS                                 | 1 -
 drivers/cxl/cxlmem.h                        | 2 +-
 include/cxl/mailbox.h                       | 2 +-
 include/uapi/{linux/cxl_mem.h => cxl/mem.h} | 0
 4 files changed, 2 insertions(+), 3 deletions(-)
 rename include/uapi/{linux/cxl_mem.h => cxl/mem.h} (100%)

diff --git a/MAINTAINERS b/MAINTAINERS
index 877eb301b1e5..d5a38bdbf2c1 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -5775,7 +5775,6 @@ S:	Maintained
 F:	Documentation/driver-api/cxl
 F:	drivers/cxl/
 F:	include/cxl/
-F:	include/uapi/linux/cxl_mem.h
 F:	tools/testing/cxl/
 
 COMPUTE EXPRESS LINK PMU (CPMU)
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 55c55685cb39..e82c84b2edb5 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -2,7 +2,7 @@
 /* Copyright(c) 2020-2021 Intel Corporation. */
 #ifndef __CXL_MEM_H__
 #define __CXL_MEM_H__
-#include <uapi/linux/cxl_mem.h>
+#include <uapi/cxl/mem.h>
 #include <linux/pci.h>
 #include <linux/cdev.h>
 #include <linux/uuid.h>
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index 1157b19175a5..9b7bd59f5fa4 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -4,8 +4,8 @@
 #define __CXL_MBOX_H__
 #include <linux/rcuwait.h>
 #include <cxl/features.h>
+#include <uapi/cxl/mem.h>
 #include <uapi/fwctl/cxl.h>
-#include <uapi/linux/cxl_mem.h>
 
 /**
  * struct cxl_mbox_cmd - A command to be submitted to hardware.
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/cxl/mem.h
similarity index 100%
rename from include/uapi/linux/cxl_mem.h
rename to include/uapi/cxl/mem.h
-- 
2.47.1

[PATCH v1 13/19] cxl: Move cxl feature command structs to user header

[Dave Jiang]

In perparation for cxl fwctl enabling, move data structures related to
cxl feature commands to a user header file.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v1:
- Fixup user headers from 0day report
---
 include/cxl/features.h      | 101 +------------------------------
 include/uapi/cxl/features.h | 117 ++++++++++++++++++++++++++++++++++++
 2 files changed, 118 insertions(+), 100 deletions(-)
 create mode 100644 include/uapi/cxl/features.h

diff --git a/include/cxl/features.h b/include/cxl/features.h
index e38ee777328d..9f234659ca2d 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -5,6 +5,7 @@
 
 #include <linux/uuid.h>
 #include <linux/fwctl.h>
+#include <uapi/cxl/features.h>
 
 #define CXL_FEAT_PATROL_SCRUB_UUID						\
 	UUID_INIT(0x96dad7d6, 0xfde8, 0x482b, 0xa7, 0x33, 0x75, 0x77, 0x4e,	\
@@ -47,55 +48,6 @@ struct cxl_features {
 };
 #define to_cxl_features(dev) container_of(dev, struct cxl_features, dev)
 
-/* Get Supported Features (0x500h) CXL r3.1 8.2.9.6.1 */
-struct cxl_mbox_get_sup_feats_in {
-	__le32 count;
-	__le16 start_idx;
-	u8 reserved[2];
-} __packed;
-
-/* CXL spec r3.2 Table 8-87 command effects */
-#define CXL_CMD_CONFIG_CHANGE_COLD_RESET	BIT(0)
-#define CXL_CMD_CONFIG_CHANGE_IMMEDIATE		BIT(1)
-#define CXL_CMD_DATA_CHANGE_IMMEDIATE		BIT(2)
-#define CXL_CMD_POLICY_CHANGE_IMMEDIATE		BIT(3)
-#define CXL_CMD_LOG_CHANGE_IMMEDIATE		BIT(4)
-#define CXL_CMD_SECURITY_STATE_CHANGE		BIT(5)
-#define CXL_CMD_BACKGROUND			BIT(6)
-#define CXL_CMD_BGCMD_ABORT_SUPPORTED		BIT(7)
-#define CXL_CMD_EFFECTS_EXTEND			BIT(9)
-#define CXL_CMD_CONFIG_CHANGE_CONV_RESET	BIT(10)
-#define CXL_CMD_CONFIG_CHANGE_CXL_RESET		BIT(11)
-
-/*
- * CXL spec r3.2 Table 8-109
- * Get Supported Features Supported Feature Entry
- * Atribute Flags
- */
-#define CXL_FEATURE_F_CHANGEABLE		BIT(0)
-#define CXL_FEATURE_F_PERSIST_FW_UPDATE		BIT(4)
-#define CXL_FEATURE_F_DEFAULT_SEL		BIT(5)
-#define CXL_FEATURE_F_SAVED_SEL			BIT(6)
-
-struct cxl_feat_entry {
-	uuid_t uuid;
-	__le16 id;
-	__le16 get_feat_size;
-	__le16 set_feat_size;
-	__le32 flags;
-	u8 get_feat_ver;
-	u8 set_feat_ver;
-	__le16 effects;
-	u8 reserved[18];
-} __packed;
-
-struct cxl_mbox_get_sup_feats_out {
-	__le16 num_entries;
-	__le16 supported_feats;
-	u8 reserved[4];
-	struct cxl_feat_entry ents[] __counted_by_le(num_entries);
-} __packed;
-
 struct cxl_features_state {
 	struct fwctl_device fwctl;
 	struct cxl_features *features;
@@ -104,57 +56,6 @@ struct cxl_features_state {
 	struct cxl_feat_entry *entries;
 };
 
-/*
- * Get Feature CXL 3.1 Spec 8.2.9.6.2
- */
-
-/*
- * Get Feature input payload
- * CXL rev 3.1 section 8.2.9.6.2 Table 8-99
- */
-enum cxl_get_feat_selection {
-	CXL_GET_FEAT_SEL_CURRENT_VALUE,
-	CXL_GET_FEAT_SEL_DEFAULT_VALUE,
-	CXL_GET_FEAT_SEL_SAVED_VALUE,
-	CXL_GET_FEAT_SEL_MAX
-};
-
-struct cxl_mbox_get_feat_in {
-	uuid_t uuid;
-	__le16 offset;
-	__le16 count;
-	u8 selection;
-}  __packed;
-
-/*
- * Set Feature CXL 3.1 Spec 8.2.9.6.3
- */
-
-/*
- * Set Feature input payload
- * CXL rev 3.1 section 8.2.9.6.3 Table 8-101
- */
-/* Set Feature : Payload in flags */
-#define CXL_SET_FEAT_FLAG_DATA_TRANSFER_MASK	GENMASK(2, 0)
-enum cxl_set_feat_flag_data_transfer {
-	CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER = 0,
-	CXL_SET_FEAT_FLAG_INITIATE_DATA_TRANSFER,
-	CXL_SET_FEAT_FLAG_CONTINUE_DATA_TRANSFER,
-	CXL_SET_FEAT_FLAG_FINISH_DATA_TRANSFER,
-	CXL_SET_FEAT_FLAG_ABORT_DATA_TRANSFER,
-	CXL_SET_FEAT_FLAG_DATA_TRANSFER_MAX
-};
-
-#define CXL_SET_FEAT_FLAG_DATA_SAVED_ACROSS_RESET	BIT(3)
-
-struct cxl_mbox_set_feat_hdr {
-	uuid_t uuid;
-	__le32 flags;
-	__le16 offset;
-	u8 version;
-	u8 rsvd[9];
-}  __packed;
-
 bool cxl_feature_enabled(struct cxl_features_state *cfs, u16 opcode);
 struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
 					struct device *parent);
diff --git a/include/uapi/cxl/features.h b/include/uapi/cxl/features.h
new file mode 100644
index 000000000000..18b74252058a
--- /dev/null
+++ b/include/uapi/cxl/features.h
@@ -0,0 +1,117 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (c) 2024,2025, Intel Corporation
+ *
+ * These are definitions for the mailbox command interface of CXL subsystem.
+ */
+#ifndef _UAPI_CXL_FEATURES_H_
+#define _UAPI_CXL_FEATURES_H_
+
+#include <linux/types.h>
+#ifndef __KERNEL__
+#include <uuid/uuid.h>
+#else
+#include <linux/uuid.h>
+#endif
+
+/* Get Supported Features (0x500h) CXL r3.1 8.2.9.6.1 */
+struct cxl_mbox_get_sup_feats_in {
+	__le32 count;
+	__le16 start_idx;
+	__u8 reserved[2];
+} __attribute__ ((__packed__));
+
+/* CXL spec r3.2 Table 8-87 command effects */
+#define CXL_CMD_CONFIG_CHANGE_COLD_RESET	BIT(0)
+#define CXL_CMD_CONFIG_CHANGE_IMMEDIATE		BIT(1)
+#define CXL_CMD_DATA_CHANGE_IMMEDIATE		BIT(2)
+#define CXL_CMD_POLICY_CHANGE_IMMEDIATE		BIT(3)
+#define CXL_CMD_LOG_CHANGE_IMMEDIATE		BIT(4)
+#define CXL_CMD_SECURITY_STATE_CHANGE		BIT(5)
+#define CXL_CMD_BACKGROUND			BIT(6)
+#define CXL_CMD_BGCMD_ABORT_SUPPORTED		BIT(7)
+#define CXL_CMD_EFFECTS_EXTEND			BIT(9)
+#define CXL_CMD_CONFIG_CHANGE_CONV_RESET	BIT(10)
+#define CXL_CMD_CONFIG_CHANGE_CXL_RESET		BIT(11)
+
+/*
+ * CXL spec r3.2 Table 8-109
+ * Get Supported Features Supported Feature Entry
+ * Attribute Flags
+ */
+#define CXL_FEATURE_F_CHANGEABLE		BIT(0)
+#define CXL_FEATURE_F_PERSIST_FW_UPDATE		BIT(4)
+#define CXL_FEATURE_F_DEFAULT_SEL		BIT(5)
+#define CXL_FEATURE_F_SAVED_SEL			BIT(6)
+
+struct cxl_feat_entry {
+	uuid_t uuid;
+	__le16 id;
+	__le16 get_feat_size;
+	__le16 set_feat_size;
+	__le32 flags;
+	__u8 get_feat_ver;
+	__u8 set_feat_ver;
+	__le16 effects;
+	__u8 reserved[18];
+} __attribute__ ((__packed__));
+
+struct cxl_mbox_get_sup_feats_out {
+	__le16 num_entries;
+	__le16 supported_feats;
+	__u8 reserved[4];
+	struct cxl_feat_entry ents[] __counted_by_le(num_entries);
+} __attribute__ ((__packed__));
+
+/*
+ * Get Feature CXL 3.1 Spec 8.2.9.6.2
+ */
+
+/*
+ * Get Feature input payload
+ * CXL rev 3.1 section 8.2.9.6.2 Table 8-99
+ */
+enum cxl_get_feat_selection {
+	CXL_GET_FEAT_SEL_CURRENT_VALUE,
+	CXL_GET_FEAT_SEL_DEFAULT_VALUE,
+	CXL_GET_FEAT_SEL_SAVED_VALUE,
+	CXL_GET_FEAT_SEL_MAX
+};
+
+struct cxl_mbox_get_feat_in {
+	uuid_t uuid;
+	__le16 offset;
+	__le16 count;
+	__u8 selection;
+} __attribute__ ((__packed__));
+
+/*
+ * Set Feature CXL 3.1 Spec 8.2.9.6.3
+ */
+
+/*
+ * Set Feature input payload
+ * CXL rev 3.1 section 8.2.9.6.3 Table 8-101
+ */
+/* Set Feature : Payload in flags */
+#define CXL_SET_FEAT_FLAG_DATA_TRANSFER_MASK	GENMASK(2, 0)
+enum cxl_set_feat_flag_data_transfer {
+	CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER = 0,
+	CXL_SET_FEAT_FLAG_INITIATE_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_CONTINUE_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_FINISH_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_ABORT_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_DATA_TRANSFER_MAX
+};
+
+#define CXL_SET_FEAT_FLAG_DATA_SAVED_ACROSS_RESET	BIT(3)
+
+struct cxl_mbox_set_feat_hdr {
+	uuid_t uuid;
+	__le32 flags;
+	__le16 offset;
+	__u8 version;
+	__u8 rsvd[9];
+} __attribute__ ((__packed__));
+
+#endif
-- 
2.47.1

[PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Dave Jiang]

fwctl provides a fwctl_ops->fw_rpc() callback in order to issue ioctls
to a device. The cxl fwctl driver will start by supporting the CXL
feature commands: Get Supported Features, Get Feature, and Set Feature.

The fw_rpc() callback provides 'enum fwctl_rpc_scope' parameter where
it indicates the security scope of the call. The Get Supported Features
and Get Feature calls can be executed with the scope of
FWCTL_RPC_CONFIGRATION. The Set Feature call is gated by the effects
of the feature reported by Get Supported Features call for the specific
feature.

Only "get supported features" is supported in this patch. Additional
commands will be added in follow on patches. "Get supported features"
will filter the features that are exclusive to the kernel and only
report out features that are not kernel only.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/mbox.c     |  12 +++
 drivers/cxl/features.c      | 198 +++++++++++++++++++++++++++++++++++-
 include/cxl/features.h      |   1 +
 include/uapi/cxl/features.h |  13 +++
 include/uapi/fwctl/cxl.h    |  29 ++++++
 5 files changed, 250 insertions(+), 3 deletions(-)

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 0b4946205910..f818e452dbca 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -246,6 +246,18 @@ struct cxl_mem_command *cxl_find_feature_command(u16 opcode)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_find_feature_command, "CXL");
 
+struct cxl_mem_command *cxl_find_feature_command_by_id(u32 id)
+{
+	struct cxl_mem_command *c;
+
+	cxl_for_each_feature_cmd(c)
+		if (c->info.id == id)
+			return c;
+
+	return NULL;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_find_feature_command_by_id, "CXL");
+
 static const char *cxl_mem_opcode_to_name(u16 opcode)
 {
 	struct cxl_mem_command *c;
diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
index cc72e73ae8d6..e65fc8479d21 100644
--- a/drivers/cxl/features.c
+++ b/drivers/cxl/features.c
@@ -50,11 +50,203 @@ static void *cxlctl_info(struct fwctl_uctx *uctx, size_t *length)
 	return info;
 }
 
+static struct cxl_feat_entry *
+get_support_feature_info(struct cxl_features_state *cfs,
+			 const struct fwctl_rpc_cxl *rpc_in)
+{
+	struct cxl_feat_entry *feat;
+	uuid_t uuid;
+
+	if (rpc_in->op_size < sizeof(uuid))
+		return ERR_PTR(-EINVAL);
+
+	if (copy_from_user(&uuid, u64_to_user_ptr(rpc_in->in_payload),
+			   sizeof(uuid)))
+		return ERR_PTR(-EFAULT);
+
+	for (int i = 0; i < cfs->num_features; i++) {
+		feat = &cfs->entries[i];
+		if (uuid_equal(&uuid, &feat->uuid))
+			return feat;
+	}
+
+	return ERR_PTR(-ENOENT);
+}
+
+static void *cxlctl_get_supported_features(struct cxl_features_state *cfs,
+					   const struct fwctl_rpc_cxl *rpc_in,
+					   size_t *out_len)
+{
+	struct cxl_mbox_get_sup_feats_out *feat_out;
+	struct cxl_mbox_get_sup_feats_in feat_in;
+	struct cxl_feat_entry *saved, *pos;
+	int requested, copied;
+	size_t out_size;
+	u32 count;
+	u16 start;
+
+	if (rpc_in->op_size != sizeof(feat_in))
+		return ERR_PTR(-EINVAL);
+
+	if (copy_from_user(&feat_in, u64_to_user_ptr(rpc_in->in_payload),
+			   rpc_in->op_size))
+		return ERR_PTR(-EFAULT);
+
+	count = le32_to_cpu(feat_in.count);
+	start = le16_to_cpu(feat_in.start_idx);
+	requested = count / sizeof(*pos);
+
+	/*
+	 * Make sure that the total requested number of entries is not greater
+	 * than the total number of supported features allowed for userspace.
+	 */
+	if (start >= cfs->num_user_features)
+		return ERR_PTR(-EINVAL);
+
+	requested = min_t(int, requested, cfs->num_user_features - start);
+
+	out_size = sizeof(struct fwctl_rpc_cxl_out) + sizeof(*feat_out) +
+		   requested * sizeof(*pos);
+
+	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
+		kvzalloc(out_size, GFP_KERNEL);
+	if (!rpc_out)
+		return ERR_PTR(-ENOMEM);
+
+	rpc_out->size = sizeof(*feat_out) + requested * sizeof(*pos);
+	feat_out = (struct cxl_mbox_get_sup_feats_out *)rpc_out->payload;
+	if (requested == 0) {
+		feat_out->num_entries = cpu_to_le16(requested);
+		feat_out->supported_feats = cpu_to_le16(cfs->num_user_features);
+		rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
+		*out_len = out_size;
+		return no_free_ptr(rpc_out);
+	}
+
+	pos = &feat_out->ents[0];
+	saved = &cfs->entries[0];
+
+	copied = 0;
+	for (int i = 0; i < cfs->num_features; i++, saved++) {
+		if (is_cxl_feature_exclusive(saved))
+			continue;
+
+		memcpy(pos, saved, sizeof(*saved));
+		copied++;
+		if (copied == requested)
+			break;
+		pos++;
+	}
+
+	feat_out->num_entries = cpu_to_le16(requested);
+	feat_out->supported_feats = cpu_to_le16(cfs->num_user_features);
+	rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
+	*out_len = out_size;
+
+	return no_free_ptr(rpc_out);
+}
+
+static bool cxlctl_validate_set_features(struct cxl_features_state *cfs,
+					 const struct fwctl_rpc_cxl *rpc_in,
+					 enum fwctl_rpc_scope scope)
+{
+	struct cxl_feat_entry *feat;
+	u16 effects, mask;
+	u32 flags;
+
+	feat = get_support_feature_info(cfs, rpc_in);
+	if (IS_ERR(feat))
+		return false;
+
+	/* Ensure that the attribute is changeable */
+	flags = le32_to_cpu(feat->flags);
+	if (!(flags & CXL_FEATURE_F_CHANGEABLE))
+		return false;
+
+	effects = le16_to_cpu(feat->effects);
+	/* Currently no user background command support */
+	if (effects & CXL_CMD_BACKGROUND)
+		return false;
+
+	mask = CXL_CMD_CONFIG_CHANGE_IMMEDIATE |
+	       CXL_CMD_DATA_CHANGE_IMMEDIATE |
+	       CXL_CMD_POLICY_CHANGE_IMMEDIATE |
+	       CXL_CMD_LOG_CHANGE_IMMEDIATE;
+	if (effects & mask && scope >= FWCTL_RPC_DEBUG_WRITE_FULL)
+		return true;
+
+	/* These effects supported for all scope */
+	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
+	     (effects & CXL_CMD_EFFECTS_EXTEND &&
+	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
+	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
+	    scope >= FWCTL_RPC_DEBUG_WRITE)
+		return true;
+
+	return false;
+}
+
+static struct cxl_mem_command *
+cxlctl_get_valid_hw_command(struct cxl_features_state *cfs,
+			    const struct fwctl_rpc_cxl *rpc_in,
+			    enum fwctl_rpc_scope scope)
+{
+	struct cxl_mem_command *cmd;
+
+	if (!cfs->num_features)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	cmd = cxl_find_feature_command_by_id(rpc_in->command_id);
+	if (!cmd)
+		return ERR_PTR(-ENOENT);
+
+	if (!cxl_feature_enabled(cfs, cmd->opcode))
+		return ERR_PTR(-EPERM);
+
+	switch (cmd->opcode) {
+	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
+	case CXL_MBOX_OP_GET_FEATURE:
+		if (scope >= FWCTL_RPC_CONFIGURATION)
+			return cmd;
+		break;
+	case CXL_MBOX_OP_SET_FEATURE:
+		if (cxlctl_validate_set_features(cfs, rpc_in, scope))
+			return cmd;
+		break;
+	default:
+		return ERR_PTR(-EINVAL);
+	}
+
+	return ERR_PTR(-EINVAL);
+}
+
+static void *cxlctl_handle_commands(struct cxl_features_state *cfs,
+				    const struct fwctl_rpc_cxl *rpc_in,
+				    size_t *out_len, struct cxl_mem_command *cmd)
+{
+	switch (cmd->opcode) {
+	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
+		return cxlctl_get_supported_features(cfs, rpc_in, out_len);
+	case CXL_MBOX_OP_GET_FEATURE:
+	case CXL_MBOX_OP_SET_FEATURE:
+	default:
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+}
+
 static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
-			   void *rpc_in, size_t in_len, size_t *out_len)
+			   void *in, size_t in_len, size_t *out_len)
 {
-	/* Place holder */
-	return ERR_PTR(-EOPNOTSUPP);
+	struct fwctl_device *fwctl = uctx->fwctl;
+	struct cxl_features_state *cfs = to_cxl_features_state(fwctl);
+	const struct fwctl_rpc_cxl *rpc_in = in;
+	struct cxl_mem_command *cmd;
+
+	cmd = cxlctl_get_valid_hw_command(cfs, rpc_in, scope);
+	if (IS_ERR(cmd))
+		return (void *)cmd;
+
+	return cxlctl_handle_commands(cfs, rpc_in, out_len, cmd);
 }
 
 static const struct fwctl_ops cxlctl_ops = {
diff --git a/include/cxl/features.h b/include/cxl/features.h
index 9f234659ca2d..b5dacc1665c7 100644
--- a/include/cxl/features.h
+++ b/include/cxl/features.h
@@ -69,6 +69,7 @@ size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
 int cxl_set_feature(struct cxl_features *features, const uuid_t feat_uuid,
 		    u8 feat_version, void *feat_data, size_t feat_data_size,
 		    u32 feat_flag, u16 offset, u16 *return_code);
+struct cxl_mem_command *cxl_find_feature_command_by_id(u32 id);
 bool is_cxl_feature_exclusive(struct cxl_feat_entry *entry);
 
 #endif
diff --git a/include/uapi/cxl/features.h b/include/uapi/cxl/features.h
index 18b74252058a..55fa5b2a5d17 100644
--- a/include/uapi/cxl/features.h
+++ b/include/uapi/cxl/features.h
@@ -14,6 +14,19 @@
 #include <linux/uuid.h>
 #endif
 
+/* CXL spec r3.2 Table 8-87 command effects */
+#define CXL_CMD_CONFIG_CHANGE_COLD_RESET	BIT(0)
+#define CXL_CMD_CONFIG_CHANGE_IMMEDIATE		BIT(1)
+#define CXL_CMD_DATA_CHANGE_IMMEDIATE		BIT(2)
+#define CXL_CMD_POLICY_CHANGE_IMMEDIATE		BIT(3)
+#define CXL_CMD_LOG_CHANGE_IMMEDIATE		BIT(4)
+#define CXL_CMD_SECURITY_STATE_CHANGE		BIT(5)
+#define CXL_CMD_BACKGROUND			BIT(6)
+#define CXL_CMD_BGCMD_ABORT_SUPPORTED		BIT(7)
+#define CXL_CMD_EFFECTS_EXTEND			BIT(9)
+#define CXL_CMD_CONFIG_CHANGE_CONV_RESET	BIT(10)
+#define CXL_CMD_CONFIG_CHANGE_CXL_RESET		BIT(11)
+
 /* Get Supported Features (0x500h) CXL r3.1 8.2.9.6.1 */
 struct cxl_mbox_get_sup_feats_in {
 	__le32 count;
diff --git a/include/uapi/fwctl/cxl.h b/include/uapi/fwctl/cxl.h
index 79b822dbfafd..ad23c65ce724 100644
--- a/include/uapi/fwctl/cxl.h
+++ b/include/uapi/fwctl/cxl.h
@@ -28,4 +28,33 @@ enum feature_cmds {
 struct fwctl_info_cxl {
 	__u32 cmd_mask;
 };
+
+/**
+ * struct fwctl_rpc_cxl - ioctl(FWCTL_RPC) input for CXL
+ * @command_id: the defined command id by 'enum feature_cmds'
+ * @flags: Flags for the command (input).
+ * @op_size: Size of input payload.
+ * @reserved1: Reserved. Must be 0s.
+ * @in_payload: User address of the hardware op input structure
+ */
+struct fwctl_rpc_cxl {
+	__u32 command_id;
+	__u32 flags;
+	__u32 op_size;
+	__u32 reserved1;
+	__aligned_u64 in_payload;
+};
+
+/**
+ * struct fwctl_rpc_cxl_out - ioctl(FWCTL_RPC) output for CXL
+ * @size: Size of the output payload
+ * @retval: Return value from device
+ * @payload: Return data from device
+ */
+struct fwctl_rpc_cxl_out {
+	__u32 size;
+	__u32 retval;
+	__u8 payload[] __counted_by(size);
+};
+
 #endif
-- 
2.47.1

[PATCH v1 15/19] cxl: Add support to handle user feature commands for get feature

[Dave Jiang]

Add helper function to parse the user data from fwctl RPC ioctl and
send the parsed input parameters to cxl_get_feature() call.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/features.c | 46 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
index e65fc8479d21..7c726f1512d0 100644
--- a/drivers/cxl/features.c
+++ b/drivers/cxl/features.c
@@ -146,6 +146,51 @@ static void *cxlctl_get_supported_features(struct cxl_features_state *cfs,
 	return no_free_ptr(rpc_out);
 }
 
+static void *cxlctl_get_feature(struct cxl_features_state *cfs,
+				const struct fwctl_rpc_cxl *rpc_in,
+				size_t *out_len)
+{
+	struct cxl_mbox_get_feat_in feat_in;
+	u16 offset, count, return_code;
+	size_t out_size = *out_len;
+	void *feat_out;
+
+	if (rpc_in->op_size != sizeof(feat_in))
+		return ERR_PTR(-EINVAL);
+
+	if (copy_from_user(&feat_in, u64_to_user_ptr(rpc_in->in_payload),
+			   rpc_in->op_size))
+		return ERR_PTR(-EFAULT);
+
+	offset = le16_to_cpu(feat_in.offset);
+	count = le16_to_cpu(feat_in.count);
+
+	if (!count)
+		return ERR_PTR(-EINVAL);
+
+	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
+		kvzalloc(out_size, GFP_KERNEL);
+	if (!rpc_out)
+		return ERR_PTR(-ENOMEM);
+
+	feat_out = (void *)rpc_out->payload;
+	out_size = cxl_get_feature(cfs->features, feat_in.uuid,
+				   feat_in.selection, feat_out,
+				   count, offset, &return_code);
+	*out_len = sizeof(struct fwctl_rpc_cxl_out);
+	if (!out_size) {
+		rpc_out->size = 0;
+		rpc_out->retval = return_code;
+		return no_free_ptr(rpc_out);
+	}
+
+	rpc_out->size = out_size;
+	rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
+	*out_len += out_size;
+
+	return no_free_ptr(rpc_out);
+}
+
 static bool cxlctl_validate_set_features(struct cxl_features_state *cfs,
 					 const struct fwctl_rpc_cxl *rpc_in,
 					 enum fwctl_rpc_scope scope)
@@ -228,6 +273,7 @@ static void *cxlctl_handle_commands(struct cxl_features_state *cfs,
 	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
 		return cxlctl_get_supported_features(cfs, rpc_in, out_len);
 	case CXL_MBOX_OP_GET_FEATURE:
+		return cxlctl_get_feature(cfs, rpc_in, out_len);
 	case CXL_MBOX_OP_SET_FEATURE:
 	default:
 		return ERR_PTR(-EOPNOTSUPP);
-- 
2.47.1

[PATCH v1 16/19] cxl: Add support to handle user feature commands for set feature

[Dave Jiang]

Add helper function to parse the user data from fwctl RPC ioctl and
send the parsed input parameters to cxl_set_feature() call.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/features.c      | 48 +++++++++++++++++++++++++++++++++++++
 include/uapi/cxl/features.h |  5 ++++
 2 files changed, 53 insertions(+)

diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
index 7c726f1512d0..4c5c93278e58 100644
--- a/drivers/cxl/features.c
+++ b/drivers/cxl/features.c
@@ -191,6 +191,53 @@ static void *cxlctl_get_feature(struct cxl_features_state *cfs,
 	return no_free_ptr(rpc_out);
 }
 
+static void *cxlctl_set_feature(struct cxl_features_state *cfs,
+				const struct fwctl_rpc_cxl *rpc_in,
+				size_t *out_len)
+{
+	size_t out_size, data_size;
+	u16 offset, return_code;
+	u32 flags;
+	int rc;
+
+	if (rpc_in->op_size <= sizeof(struct cxl_mbox_set_feat_hdr))
+		return ERR_PTR(-EINVAL);
+
+	struct cxl_mbox_set_feat_in *feat_in __free(kvfree) =
+		kvzalloc(rpc_in->op_size, GFP_KERNEL);
+	if (!feat_in)
+		return ERR_PTR(-ENOMEM);
+
+	if (copy_from_user(feat_in, u64_to_user_ptr(rpc_in->in_payload),
+			   rpc_in->op_size))
+		return ERR_PTR(-EFAULT);
+
+	offset = le16_to_cpu(feat_in->hdr.offset);
+	flags = le32_to_cpu(feat_in->hdr.flags);
+	out_size = *out_len;
+
+	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
+		kvzalloc(out_size, GFP_KERNEL);
+	if (!rpc_out)
+		return ERR_PTR(-ENOMEM);
+
+	rpc_out->size = 0;
+
+	data_size = rpc_in->op_size - sizeof(feat_in->hdr);
+	rc = cxl_set_feature(cfs->features, feat_in->hdr.uuid,
+			     feat_in->hdr.version, feat_in->data,
+			     data_size, flags, offset, &return_code);
+	if (rc) {
+		rpc_out->retval = return_code;
+		return no_free_ptr(rpc_out);
+	}
+
+	rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
+	*out_len = sizeof(*rpc_out);
+
+	return no_free_ptr(rpc_out);
+}
+
 static bool cxlctl_validate_set_features(struct cxl_features_state *cfs,
 					 const struct fwctl_rpc_cxl *rpc_in,
 					 enum fwctl_rpc_scope scope)
@@ -275,6 +322,7 @@ static void *cxlctl_handle_commands(struct cxl_features_state *cfs,
 	case CXL_MBOX_OP_GET_FEATURE:
 		return cxlctl_get_feature(cfs, rpc_in, out_len);
 	case CXL_MBOX_OP_SET_FEATURE:
+		return cxlctl_set_feature(cfs, rpc_in, out_len);
 	default:
 		return ERR_PTR(-EOPNOTSUPP);
 	}
diff --git a/include/uapi/cxl/features.h b/include/uapi/cxl/features.h
index 55fa5b2a5d17..67ed8b08f7e9 100644
--- a/include/uapi/cxl/features.h
+++ b/include/uapi/cxl/features.h
@@ -127,4 +127,9 @@ struct cxl_mbox_set_feat_hdr {
 	__u8 rsvd[9];
 } __attribute__ ((__packed__));
 
+struct cxl_mbox_set_feat_in {
+	struct cxl_mbox_set_feat_hdr hdr;
+	__u8 data[];
+} __attribute__ ((__packed__));
+
 #endif
-- 
2.47.1

[PATCH v1 17/19] cxl/test: Add Get Feature support to cxl_test

[Dave Jiang]

Add emulation of Get Feature command to cxl_test. The feature for
device patrol scrub is returned by the emulation code. This is
the only feature currently supported by cxl_test. It returns
the information for the device patrol scrub feature.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 tools/testing/cxl/test/mem.c | 56 ++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index 2b111142b2c0..5b7b0420e3a0 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -49,6 +49,10 @@ static struct cxl_cel_entry mock_cel[] = {
 		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_SUPPORTED_FEATURES),
 		.effect = CXL_CMD_EFFECT_NONE,
 	},
+	{
+		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_FEATURE),
+		.effect = CXL_CMD_EFFECT_NONE,
+	},
 	{
 		.opcode = cpu_to_le16(CXL_MBOX_OP_IDENTIFY),
 		.effect = CXL_CMD_EFFECT_NONE,
@@ -150,6 +154,10 @@ struct mock_event_store {
 	u32 ev_status;
 };
 
+struct vendor_test_feat {
+	__le32 data;
+} __packed;
+
 struct cxl_mockmem_data {
 	void *lsa;
 	void *fw;
@@ -166,6 +174,7 @@ struct cxl_mockmem_data {
 	u8 event_buf[SZ_4K];
 	u64 timestamp;
 	unsigned long sanitize_timeout;
+	struct vendor_test_feat test_feat;
 };
 
 static struct mock_event_log *event_find_log(struct device *dev, int log_type)
@@ -1361,6 +1370,44 @@ static void fill_feature_vendor_test(struct cxl_feat_entry *feat)
 				    CXL_CMD_EFFECTS_EXTEND);
 }
 
+static int mock_get_test_feature(struct cxl_mockmem_data *mdata,
+				 struct cxl_mbox_cmd *cmd)
+{
+	struct vendor_test_feat *output = cmd->payload_out;
+	struct cxl_mbox_get_feat_in *input = cmd->payload_in;
+	u16 offset = le16_to_cpu(input->offset);
+	u16 count = le16_to_cpu(input->count);
+	u8 *ptr;
+
+	if (offset > sizeof(*output)) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	if (offset + count > sizeof(*output)) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	ptr = (u8 *)&mdata->test_feat + offset;
+	memcpy((u8 *)output + offset, ptr, count);
+
+	return 0;
+}
+
+static int mock_get_feature(struct cxl_mockmem_data *mdata,
+			    struct cxl_mbox_cmd *cmd)
+{
+	struct cxl_mbox_get_feat_in *input = cmd->payload_in;
+
+	if (uuid_equal(&input->uuid, &CXL_VENDOR_FEATURE_TEST))
+		return mock_get_test_feature(mdata, cmd);
+
+	cmd->return_code = CXL_MBOX_CMD_RC_UNSUPPORTED;
+
+	return -EOPNOTSUPP;
+}
+
 static int mock_get_supported_features(struct cxl_mockmem_data *mdata,
 				       struct cxl_mbox_cmd *cmd)
 {
@@ -1491,6 +1538,9 @@ static int cxl_mock_mbox_send(struct cxl_mailbox *cxl_mbox,
 	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
 		rc = mock_get_supported_features(mdata, cmd);
 		break;
+	case CXL_MBOX_OP_GET_FEATURE:
+		rc = mock_get_feature(mdata, cmd);
+		break;
 	default:
 		break;
 	}
@@ -1553,6 +1603,11 @@ static int cxl_mock_setup_features(struct cxl_memdev_state *mds)
 	return 0;
 }
 
+static void cxl_mock_test_feat_init(struct cxl_mockmem_data *mdata)
+{
+	mdata->test_feat.data = cpu_to_le32(0xdeadbeef);
+}
+
 static int cxl_mock_mem_probe(struct platform_device *pdev)
 {
 	struct device *dev = &pdev->dev;
@@ -1644,6 +1699,7 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 		return rc;
 
 	cxl_mem_get_event_records(mds, CXLDEV_EVENT_STATUS_ALL);
+	cxl_mock_test_feat_init(mdata);
 
 	return 0;
 }
-- 
2.47.1

[PATCH v1 18/19] cxl/test: Add Set Feature support to cxl_test

[Dave Jiang]

Add emulation to support Set Feature mailbox command to cxl_test.
The only feature supported is the device patrol scrub feature. The
set feature allows activation of patrol scrub for the cxl_test
emulated device. The command does not support partial data transfer
even though the spec allows it. This restriction is to reduce complexity
of the emulation given the patrol scrub feature is very minimal.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 tools/testing/cxl/test/mem.c | 50 ++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index 5b7b0420e3a0..f1d775968607 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -53,6 +53,10 @@ static struct cxl_cel_entry mock_cel[] = {
 		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_FEATURE),
 		.effect = CXL_CMD_EFFECT_NONE,
 	},
+	{
+		.opcode = cpu_to_le16(CXL_MBOX_OP_SET_FEATURE),
+		.effect = cpu_to_le16(EFFECT(CONF_CHANGE_IMMEDIATE)),
+	},
 	{
 		.opcode = cpu_to_le16(CXL_MBOX_OP_IDENTIFY),
 		.effect = CXL_CMD_EFFECT_NONE,
@@ -1408,6 +1412,49 @@ static int mock_get_feature(struct cxl_mockmem_data *mdata,
 	return -EOPNOTSUPP;
 }
 
+static int mock_set_test_feature(struct cxl_mockmem_data *mdata,
+				 struct cxl_mbox_cmd *cmd)
+{
+	struct cxl_mbox_set_feat_in *input = cmd->payload_in;
+	struct vendor_test_feat *test = (struct vendor_test_feat *)input->data;
+	u32 action;
+
+	action = FIELD_GET(CXL_SET_FEAT_FLAG_DATA_TRANSFER_MASK,
+			   le32_to_cpu(input->hdr.flags));
+	/*
+	 * While it is spec compliant to support other set actions, it is not
+	 * necessary to add the complication in the emulation currently. Reject
+	 * anything besides full xfer.
+	 */
+	if (action != CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	/* Offset should be reserved when doing full transfer */
+	if (input->hdr.offset) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	memcpy(&mdata->test_feat.data, &test->data, sizeof(u32));
+
+	return 0;
+}
+
+static int mock_set_feature(struct cxl_mockmem_data *mdata,
+			    struct cxl_mbox_cmd *cmd)
+{
+	struct cxl_mbox_set_feat_in *input = cmd->payload_in;
+
+	if (uuid_equal(&input->hdr.uuid, &CXL_VENDOR_FEATURE_TEST))
+		return mock_set_test_feature(mdata, cmd);
+
+	cmd->return_code = CXL_MBOX_CMD_RC_UNSUPPORTED;
+
+	return -EOPNOTSUPP;
+}
+
 static int mock_get_supported_features(struct cxl_mockmem_data *mdata,
 				       struct cxl_mbox_cmd *cmd)
 {
@@ -1541,6 +1588,9 @@ static int cxl_mock_mbox_send(struct cxl_mailbox *cxl_mbox,
 	case CXL_MBOX_OP_GET_FEATURE:
 		rc = mock_get_feature(mdata, cmd);
 		break;
+	case CXL_MBOX_OP_SET_FEATURE:
+		rc = mock_set_feature(mdata, cmd);
+		break;
 	default:
 		break;
 	}
-- 
2.47.1

[PATCH v1 19/19] fwctl/cxl: Add documentation to FWCTL CXL

[Dave Jiang]

Add policy and operational documentation for FWCTL CXL.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 .../userspace-api/fwctl/fwctl-cxl.rst         | 81 +++++++++++++++++++
 Documentation/userspace-api/fwctl/index.rst   |  1 +
 2 files changed, 82 insertions(+)
 create mode 100644 Documentation/userspace-api/fwctl/fwctl-cxl.rst

diff --git a/Documentation/userspace-api/fwctl/fwctl-cxl.rst b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
new file mode 100644
index 000000000000..f774bf2ecb5f
--- /dev/null
+++ b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
@@ -0,0 +1,81 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+================
+fwctl cxl driver
+================
+
+:Author: Dave Jiang
+
+Overview
+========
+
+The CXL spec defines a set of commands that can be issued to the mailbox of a
+CXL device or switch. It also left room for vendor specific commands to be
+issued to the mailbox as well. fwctl provides a path to issue a set of allowed
+mailbox commands from user space to the device moderated by the kernel driver.
+
+While there are a large set of mailbox commands, only the feature related
+commands will be allowed to be issued through fwctl initially. No background
+commands will be supported at this time.
+
+CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
+CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
+CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)
+
+The "Get Supported Features" return data may be filtered by the kernel driver to
+drop any features that are forbidden by the kernel or being exclusively used by
+the kernel. The "Get Supported Features" command and the "Get Features" falls
+under the fwctl policy of FWCTL_RPC_CONFIGURATION.
+
+For "Set Feature" command, the access policy currently is broken down into two
+categories depending on the set feature effects reported by the device. If the
+set feature will cause immediate change to the device, the fwctl access policy
+must be FWCTL_RPC_DEBUG_WRITE_FULL. The effects for this level are
+"immediate config change", "immediate data change", "immediate policy change",
+or "immediate log change" for the set effects mask. If the effects are "config
+change with cold reset" or "config change with conventional reset", then the
+fwctl access policy must be FWCTL_RPC_DEBUG_WRITE or higher.
+
+fwctl cxl User API
+==================
+
+.. kernel-doc:: include/uapi/fwctl/cxl.h
+.. kernel-doc:: include/uapi/cxl/features.h
+
+1. Driver info query
+--------------------
+
+First step for the app is to issue the ioctl(FWCTL_CMD_INFO) in order to retrieve
+the information from the driver. A ``struct fwctl_info`` needs to be filled out
+with the ``fwctl_info.out_device_type`` set to ``FWCTL_DEVICE_TYPE_CXL``. The
+return data should be ``struct fwctl_info_cxl`` that contains a command mask
+that indicates which of the commands are supported by the driver.
+
+2. Send hardware command
+------------------------
+
+Next step is to send the 'Get Supported Features' command to the driver from
+user space via ioctl(FWCTL_RPC). A ``struct fwctl_rpc_cxl`` should be pointed to
+by ``fwctl_rpc.in``. ``struct fwctl_rpc_cxl.in_payload`` should point to
+the hardware input structure that is defined by the CXL spec. ``fwctl_rpc.out``
+points to the buffer that contains a ``struct fwctl_rpc_cxl_out`` that includes
+the hardware output data inlined as ``fwctl_rpc_cxl_out.payload``. This command
+can be called twice. First time to retrieve the number of features supported.
+A second time to retrieve the specific feature details.
+
+After getting the specific feature details, a get/set feature command can be
+appropriately programmed and sent. For a "set feature" command, the retrieved
+feature info contains an effects field that details the resulting
+"set feature" command will trigger. That should inform the user whether
+the system is configured to allowed the "set feature" command or not.
+
+It is recommended to take a look at CXL CLI test directory for a detailed
+user code example on how to exercise this path.
+
+fwctl cxl Kernel API
+====================
+
+.. kernel-doc:: drivers/cxl/core/features.c
+.. kernel-doc:: drivers/cxl/features.c
+   :export:
+.. kernel-doc:: include/cxl/features.h
diff --git a/Documentation/userspace-api/fwctl/index.rst b/Documentation/userspace-api/fwctl/index.rst
index 06959fbf1547..d9d40a468a31 100644
--- a/Documentation/userspace-api/fwctl/index.rst
+++ b/Documentation/userspace-api/fwctl/index.rst
@@ -10,3 +10,4 @@ to securely construct and execute RPCs inside device firmware.
    :maxdepth: 1
 
    fwctl
+   fwctl-cxl
-- 
2.47.1

Re: [PATCH v1 02/19] cxl: Add skeletal features driver

[Dan Williams]

Dave Jiang wrote:
> Add the basic bits of a features driver to handle all CXL feature related
> services. The driver is expected to handle all CXL mailbox feature command
> related operations.
> 
> Suggested-by: Dan Williams <dan.j.williams@intel.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/Kconfig                   | 11 +++++
>  drivers/cxl/Makefile                  |  3 ++
>  drivers/cxl/core/Makefile             |  1 +
>  drivers/cxl/core/core.h               |  1 +
>  drivers/cxl/core/features.c           | 71 +++++++++++++++++++++++++++
>  drivers/cxl/core/port.c               |  3 ++
>  drivers/cxl/cxl.h                     |  1 +
>  drivers/cxl/features.c                | 44 +++++++++++++++++
>  drivers/cxl/pci.c                     | 19 +++++++
>  include/cxl/features.h                | 23 +++++++++
>  include/cxl/mailbox.h                 |  3 ++
>  tools/testing/cxl/Kbuild              |  7 +++
>  tools/testing/cxl/cxl_features_test.c |  6 +++
>  13 files changed, 193 insertions(+)
>  create mode 100644 drivers/cxl/core/features.c
>  create mode 100644 drivers/cxl/features.c
>  create mode 100644 include/cxl/features.h
>  create mode 100644 tools/testing/cxl/cxl_features_test.c
> 
> diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
> index 876469e23f7a..9a6ffd81ac0e 100644
> --- a/drivers/cxl/Kconfig
> +++ b/drivers/cxl/Kconfig
> @@ -146,4 +146,15 @@ config CXL_REGION_INVALIDATION_TEST
>  	  If unsure, or if this kernel is meant for production environments,
>  	  say N.
>  
> +config CXL_FEATURES
> +	tristate "CXL: Features support"
> +	default CXL_BUS

Not sure this default makes sense. All of the other "default CXL_BUS" is
because only an expert config would ever turn on CXL.mem support but
leave one of CONFIG_CXL_{PCI,ACPI,MEM,REGION,PORT} disabled.

Features are purely incremental.

> +	help
> +	  Enable CXL features support that are tied to a CXL mailbox.
> +	  The support for features including the feature mailbox
> +	  commands and also FWCTL support of the commands via user
> +	  space.

Perhaps a couple prominent examples of Features an end user might know
by name? Otherwise "Features" is too generic a term for making the case
for turning this on.

> +
> +	  If unsure say 'y'.

...say 'n', someone had better have a feature in mind for this optional
functionality.

> +
>  endif
> diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
> index 2caa90fa4bf2..4696fc218df4 100644
> --- a/drivers/cxl/Makefile
> +++ b/drivers/cxl/Makefile
> @@ -7,15 +7,18 @@
>  # - 'mem' and 'pmem' before endpoint drivers so that memdevs are
>  #   immediately enabled
>  # - 'pci' last, also mirrors the hardware enumeration hierarchy
> +# - 'features' comes after pci device is enumerated
>  obj-y += core/
>  obj-$(CONFIG_CXL_PORT) += cxl_port.o
>  obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
>  obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
>  obj-$(CONFIG_CXL_MEM) += cxl_mem.o
>  obj-$(CONFIG_CXL_PCI) += cxl_pci.o
> +obj-$(CONFIG_CXL_FEATURES) += cxl_features.o
>  
>  cxl_port-y := port.o
>  cxl_acpi-y := acpi.o
>  cxl_pmem-y := pmem.o security.o
>  cxl_mem-y := mem.o
>  cxl_pci-y := pci.o
> +cxl_features-y := features.o
> diff --git a/drivers/cxl/core/Makefile b/drivers/cxl/core/Makefile
> index 9259bcc6773c..73b6348afd67 100644
> --- a/drivers/cxl/core/Makefile
> +++ b/drivers/cxl/core/Makefile
> @@ -14,5 +14,6 @@ cxl_core-y += pci.o
>  cxl_core-y += hdm.o
>  cxl_core-y += pmu.o
>  cxl_core-y += cdat.o
> +cxl_core-y += features.o
>  cxl_core-$(CONFIG_TRACING) += trace.o
>  cxl_core-$(CONFIG_CXL_REGION) += region.o
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index 23761340e65c..e8a3df226643 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -9,6 +9,7 @@
>  extern const struct device_type cxl_nvdimm_bridge_type;
>  extern const struct device_type cxl_nvdimm_type;
>  extern const struct device_type cxl_pmu_type;
> +extern const struct device_type cxl_features_type;
>  
>  extern struct attribute_group cxl_base_attribute_group;
>  
> diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
> new file mode 100644
> index 000000000000..eb6eb191a32e
> --- /dev/null
> +++ b/drivers/cxl/core/features.c
> @@ -0,0 +1,71 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright(c) 2024-2025 Intel Corporation. All rights reserved. */
> +#include <linux/device.h>
> +#include "cxl.h"
> +#include "core.h"
> +
> +#define CXL_FEATURE_MAX_DEVS 65536
> +static DEFINE_IDA(cxl_features_ida);
> +
> +static void cxl_features_release(struct device *dev)
> +{
> +	struct cxl_features *features = to_cxl_features(dev);

Perhaps make this and other local 'struct cxl_feature' pointers 'cxlf'?

Just to keep with the theme of 'cxl<short-suffix>' for local pointers?


> +
> +	ida_free(&cxl_features_ida, features->id);
> +	kfree(features);
> +}
> +
> +static void remove_features_dev(void *dev)
> +{
> +	device_unregister(dev);
> +}
> +
> +const struct device_type cxl_features_type = {
> +	.name = "features",
> +	.release = cxl_features_release,
> +};
> +EXPORT_SYMBOL_NS_GPL(cxl_features_type, "CXL");
> +
> +struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
> +					struct device *parent)

So this function is alloc + add + devm auto-release. I would call it:

    devm_cxl_add_features()

> +{
> +	struct device *dev;
> +	int rc;
> +
> +	struct cxl_features *features __free(kfree) =
> +		kzalloc(sizeof(*features), GFP_KERNEL);
> +	if (!features)
> +		return ERR_PTR(-ENOMEM);
> +
> +	rc = ida_alloc_max(&cxl_features_ida, CXL_FEATURE_MAX_DEVS - 1,
> +			   GFP_KERNEL);

Does this need its own ida? I expect it could just use the same id as
the memdev, and that saves some potential confusion of which
memory-devices support features and which features interface correlates
with which device.

...or how do you imagine that working?

> +	if (rc < 0)
> +		return ERR_PTR(rc);
> +
> +	features->id = rc;
> +	features->cxl_mbox = cxl_mbox;
> +	dev = &features->dev;
> +	device_initialize(dev);

At this point @features stops being freed by kzalloc and starts being
freed by put_device(). Which means that no_free_ptr() on @features needs
to be called before put_device().

So I would put all of that in a function that does the
alloc+device_initialize() then you can do:

DEFINE_FREE(put_cxl_features, struct cxl_features *,
            if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
struct cxl_features *cxlf = __free(put_cxl_features) = alloc_features(...);

> +	device_set_pm_not_required(dev);
> +	dev->parent = parent;
> +	dev->bus = &cxl_bus_type;
> +	dev->type = &cxl_features_type;
> +	rc = dev_set_name(dev, "features%d", features->id);
> +	if (rc)
> +		goto err;
> +
> +	rc = device_add(dev);
> +	if (rc)
> +		goto err;
> +
> +	rc = devm_add_action_or_reset(parent, remove_features_dev, dev);
> +	if (rc)
> +		goto err;

...otherwise all of the gotos above causes a double-free of @features as
far as I can see.

Mixing goto and scoped-base-cleanup... often a bug.

> +
> +	return no_free_ptr(features);
> +
> +err:
> +	put_device(dev);
> +	return ERR_PTR(rc);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_features_alloc, "CXL");
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index 78a5c2c25982..cc53a597cae6 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -74,6 +74,9 @@ static int cxl_device_id(const struct device *dev)
>  		return CXL_DEVICE_REGION;
>  	if (dev->type == &cxl_pmu_type)
>  		return CXL_DEVICE_PMU;
> +	if (dev->type == &cxl_features_type)
> +		return CXL_DEVICE_FEATURES;
> +
>  	return 0;
>  }
>  
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index f6015f24ad38..ee29d1a1c8df 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -855,6 +855,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);
>  #define CXL_DEVICE_PMEM_REGION		7
>  #define CXL_DEVICE_DAX_REGION		8
>  #define CXL_DEVICE_PMU			9
> +#define CXL_DEVICE_FEATURES		10
>  
>  #define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
>  #define CXL_MODALIAS_FMT "cxl:t%d"
> diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
> new file mode 100644
> index 000000000000..644add26975f
> --- /dev/null
> +++ b/drivers/cxl/features.c
> @@ -0,0 +1,44 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright(c) 2024,2025 Intel Corporation. All rights reserved. */
> +#include <linux/device.h>
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <cxl/features.h>
> +
> +#include "cxl.h"
> +
> +static int cxl_features_probe(struct device *dev)
> +{
> +	struct cxl_features *features = to_cxl_features(dev);
> +	struct cxl_features_state *cfs __free(kfree) =
> +		kzalloc(sizeof(*cfs), GFP_KERNEL);

Just makes this:

	struct cxl_features_state *cfs = devm_kzalloc(sizeof(*cfs), GFP_KERNEL);
> +
> +	if (!cfs)
> +		return -ENOMEM;
> +
> +	cfs->features = features;
> +	dev_set_drvdata(dev, no_free_ptr(cfs));

...skip the no_free_ptr()

> +
> +	return 0;
> +}
> +
> +static void cxl_features_remove(struct device *dev)
> +{
> +	struct cxl_features_state *cfs = dev_get_drvdata(dev);
> +
> +	kfree(cfs);
> +}

...and skip the need for a ->remove() routine altogether.

> +
> +static struct cxl_driver cxl_features_driver = {
> +	.name = "cxl_features",
> +	.probe = cxl_features_probe,
> +	.remove = cxl_features_remove,
> +	.id = CXL_DEVICE_FEATURES,
> +};
> +
> +module_cxl_driver(cxl_features_driver);
> +
> +MODULE_DESCRIPTION("CXL: Features");

Maybe a few more words:

    "CXL 'Feature' command support via FWCTL"

> +MODULE_LICENSE("GPL");
> +MODULE_IMPORT_NS("CXL");
> +MODULE_ALIAS_CXL(CXL_DEVICE_FEATURES);
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 6d94ff4a4f1a..eb68dd3f8b21 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -386,6 +386,21 @@ static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox,
>  	return rc;
>  }
>  
> +static int cxl_pci_setup_features(struct cxl_memdev_state *mds)
> +{
> +	struct cxl_dev_state *cxlds = &mds->cxlds;

Looks like this function wants to take @cxlds directly and skip passing
in @mds.

> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
> +	struct cxl_features *features;
> +
> +	features = cxl_features_alloc(cxl_mbox, cxlds->dev);
> +	if (IS_ERR(features))
> +		return PTR_ERR(features);
> +
> +	cxl_mbox->features = features;
> +
> +	return 0;
> +}
> +
>  static int cxl_pci_setup_mailbox(struct cxl_memdev_state *mds, bool irq_avail)
>  {
>  	struct cxl_dev_state *cxlds = &mds->cxlds;
> @@ -980,6 +995,10 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>  	if (rc)
>  		return rc;
>  
> +	rc = cxl_pci_setup_features(mds);
> +	if (rc)
> +		return rc;
> +
>  	rc = cxl_set_timestamp(mds);
>  	if (rc)
>  		return rc;
> diff --git a/include/cxl/features.h b/include/cxl/features.h
> new file mode 100644
> index 000000000000..b92da1e92780
> --- /dev/null
> +++ b/include/cxl/features.h
> @@ -0,0 +1,23 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/* Copyright(c) 2024-2025 Intel Corporation. */
> +#ifndef __CXL_FEATURES_H__
> +#define __CXL_FEATURES_H__
> +
> +struct cxl_mailbox;
> +
> +struct cxl_features {
> +	int id;
> +	struct device dev;
> +	struct cxl_mailbox *cxl_mbox;

A bit uncomfortable with a 'struct cxl_features' instance having a
pointer to a 'struct cxl_mailbox' without holding a refcount.

This all works becase cxl_pci_setup_features() knows that the @parent
it is passing to cxl_features_alloc() is device that will also
unregister the @features device before destroying the 'struct
cxl_mailbox' instance in @cxlds.


> +};
> +#define to_cxl_features(dev) container_of(dev, struct cxl_features, dev)
> +
> +struct cxl_features_state {
> +	struct cxl_features *features;
> +	int num_features;
> +};
> +
> +struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
> +					struct device *parent);

...but this is proposed as a public function outside of cxl_pci. How
does cxl_features_alloc() understand the lifetime relationships between
its @cxl_mbox and @parent arguments?

I would have cxl_features_alloc() (devm_cxl_add_features()) take a
'struct cxl_dev_state *' argument which makes it clear that the features
device must be unregistered before 'struct cxl_dev_state *' can be
destroyed.

...or something to constrain the freedom of a new globally public
export. Does this need to be made available to any other drivers besides
cxl_pci for now?

I.e. should this be include/cxl/features.h or drivers/cxl/features.h?

Re: [PATCH v1 02/19] cxl: Add skeletal features driver

[Dave Jiang]

On 1/22/25 8:59 PM, Dan Williams wrote:
> Dave Jiang wrote:
>> Add the basic bits of a features driver to handle all CXL feature related
>> services. The driver is expected to handle all CXL mailbox feature command
>> related operations.
>>
>> Suggested-by: Dan Williams <dan.j.williams@intel.com>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>>  drivers/cxl/Kconfig                   | 11 +++++
>>  drivers/cxl/Makefile                  |  3 ++
>>  drivers/cxl/core/Makefile             |  1 +
>>  drivers/cxl/core/core.h               |  1 +
>>  drivers/cxl/core/features.c           | 71 +++++++++++++++++++++++++++
>>  drivers/cxl/core/port.c               |  3 ++
>>  drivers/cxl/cxl.h                     |  1 +
>>  drivers/cxl/features.c                | 44 +++++++++++++++++
>>  drivers/cxl/pci.c                     | 19 +++++++
>>  include/cxl/features.h                | 23 +++++++++
>>  include/cxl/mailbox.h                 |  3 ++
>>  tools/testing/cxl/Kbuild              |  7 +++
>>  tools/testing/cxl/cxl_features_test.c |  6 +++
>>  13 files changed, 193 insertions(+)
>>  create mode 100644 drivers/cxl/core/features.c
>>  create mode 100644 drivers/cxl/features.c
>>  create mode 100644 include/cxl/features.h
>>  create mode 100644 tools/testing/cxl/cxl_features_test.c
>>
>> diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
>> index 876469e23f7a..9a6ffd81ac0e 100644
>> --- a/drivers/cxl/Kconfig
>> +++ b/drivers/cxl/Kconfig
>> @@ -146,4 +146,15 @@ config CXL_REGION_INVALIDATION_TEST
>>  	  If unsure, or if this kernel is meant for production environments,
>>  	  say N.
>>  
>> +config CXL_FEATURES
>> +	tristate "CXL: Features support"
>> +	default CXL_BUS
> 
> Not sure this default makes sense. All of the other "default CXL_BUS" is
> because only an expert config would ever turn on CXL.mem support but
> leave one of CONFIG_CXL_{PCI,ACPI,MEM,REGION,PORT} disabled.
> 
> Features are purely incremental.
> 
>> +	help
>> +	  Enable CXL features support that are tied to a CXL mailbox.
>> +	  The support for features including the feature mailbox
>> +	  commands and also FWCTL support of the commands via user
>> +	  space.
> 
> Perhaps a couple prominent examples of Features an end user might know
> by name? Otherwise "Features" is too generic a term for making the case
> for turning this on.
> 
>> +
>> +	  If unsure say 'y'.
> 
> ...say 'n', someone had better have a feature in mind for this optional
> functionality.
> 
>> +
>>  endif
>> diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
>> index 2caa90fa4bf2..4696fc218df4 100644
>> --- a/drivers/cxl/Makefile
>> +++ b/drivers/cxl/Makefile
>> @@ -7,15 +7,18 @@
>>  # - 'mem' and 'pmem' before endpoint drivers so that memdevs are
>>  #   immediately enabled
>>  # - 'pci' last, also mirrors the hardware enumeration hierarchy
>> +# - 'features' comes after pci device is enumerated
>>  obj-y += core/
>>  obj-$(CONFIG_CXL_PORT) += cxl_port.o
>>  obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
>>  obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
>>  obj-$(CONFIG_CXL_MEM) += cxl_mem.o
>>  obj-$(CONFIG_CXL_PCI) += cxl_pci.o
>> +obj-$(CONFIG_CXL_FEATURES) += cxl_features.o
>>  
>>  cxl_port-y := port.o
>>  cxl_acpi-y := acpi.o
>>  cxl_pmem-y := pmem.o security.o
>>  cxl_mem-y := mem.o
>>  cxl_pci-y := pci.o
>> +cxl_features-y := features.o
>> diff --git a/drivers/cxl/core/Makefile b/drivers/cxl/core/Makefile
>> index 9259bcc6773c..73b6348afd67 100644
>> --- a/drivers/cxl/core/Makefile
>> +++ b/drivers/cxl/core/Makefile
>> @@ -14,5 +14,6 @@ cxl_core-y += pci.o
>>  cxl_core-y += hdm.o
>>  cxl_core-y += pmu.o
>>  cxl_core-y += cdat.o
>> +cxl_core-y += features.o
>>  cxl_core-$(CONFIG_TRACING) += trace.o
>>  cxl_core-$(CONFIG_CXL_REGION) += region.o
>> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
>> index 23761340e65c..e8a3df226643 100644
>> --- a/drivers/cxl/core/core.h
>> +++ b/drivers/cxl/core/core.h
>> @@ -9,6 +9,7 @@
>>  extern const struct device_type cxl_nvdimm_bridge_type;
>>  extern const struct device_type cxl_nvdimm_type;
>>  extern const struct device_type cxl_pmu_type;
>> +extern const struct device_type cxl_features_type;
>>  
>>  extern struct attribute_group cxl_base_attribute_group;
>>  
>> diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
>> new file mode 100644
>> index 000000000000..eb6eb191a32e
>> --- /dev/null
>> +++ b/drivers/cxl/core/features.c
>> @@ -0,0 +1,71 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/* Copyright(c) 2024-2025 Intel Corporation. All rights reserved. */
>> +#include <linux/device.h>
>> +#include "cxl.h"
>> +#include "core.h"
>> +
>> +#define CXL_FEATURE_MAX_DEVS 65536
>> +static DEFINE_IDA(cxl_features_ida);
>> +
>> +static void cxl_features_release(struct device *dev)
>> +{
>> +	struct cxl_features *features = to_cxl_features(dev);
> 
> Perhaps make this and other local 'struct cxl_feature' pointers 'cxlf'?
> 
> Just to keep with the theme of 'cxl<short-suffix>' for local pointers?
> 
> 
>> +
>> +	ida_free(&cxl_features_ida, features->id);
>> +	kfree(features);
>> +}
>> +
>> +static void remove_features_dev(void *dev)
>> +{
>> +	device_unregister(dev);
>> +}
>> +
>> +const struct device_type cxl_features_type = {
>> +	.name = "features",
>> +	.release = cxl_features_release,
>> +};
>> +EXPORT_SYMBOL_NS_GPL(cxl_features_type, "CXL");
>> +
>> +struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
>> +					struct device *parent)
> 
> So this function is alloc + add + devm auto-release. I would call it:
> 
>     devm_cxl_add_features()
> 
>> +{
>> +	struct device *dev;
>> +	int rc;
>> +
>> +	struct cxl_features *features __free(kfree) =
>> +		kzalloc(sizeof(*features), GFP_KERNEL);
>> +	if (!features)
>> +		return ERR_PTR(-ENOMEM);
>> +
>> +	rc = ida_alloc_max(&cxl_features_ida, CXL_FEATURE_MAX_DEVS - 1,
>> +			   GFP_KERNEL);
> 
> Does this need its own ida? I expect it could just use the same id as
> the memdev, and that saves some potential confusion of which
> memory-devices support features and which features interface correlates
> with which device.
> 
> ...or how do you imagine that working?

One of the issue was that the feature commands query happens before memdev is created. So that ida doesn't exist yet. In order to share, we'll need to do a split initialization of the feature device. But really I was trying to separate the association of features from the memdev as they are independent of each other. i.e a type2 may not have memdev but have mailbox and features.

DJ 

> 
>> +	if (rc < 0)
>> +		return ERR_PTR(rc);
>> +
>> +	features->id = rc;
>> +	features->cxl_mbox = cxl_mbox;
>> +	dev = &features->dev;
>> +	device_initialize(dev);
> 
> At this point @features stops being freed by kzalloc and starts being
> freed by put_device(). Which means that no_free_ptr() on @features needs
> to be called before put_device().
> 
> So I would put all of that in a function that does the
> alloc+device_initialize() then you can do:
> 
> DEFINE_FREE(put_cxl_features, struct cxl_features *,
>             if (!IS_ERR_OR_NULL(_T)) put_device(&_T->dev))
> struct cxl_features *cxlf = __free(put_cxl_features) = alloc_features(...);
> 
>> +	device_set_pm_not_required(dev);
>> +	dev->parent = parent;
>> +	dev->bus = &cxl_bus_type;
>> +	dev->type = &cxl_features_type;
>> +	rc = dev_set_name(dev, "features%d", features->id);
>> +	if (rc)
>> +		goto err;
>> +
>> +	rc = device_add(dev);
>> +	if (rc)
>> +		goto err;
>> +
>> +	rc = devm_add_action_or_reset(parent, remove_features_dev, dev);
>> +	if (rc)
>> +		goto err;
> 
> ...otherwise all of the gotos above causes a double-free of @features as
> far as I can see.
> 
> Mixing goto and scoped-base-cleanup... often a bug.
> 
>> +
>> +	return no_free_ptr(features);
>> +
>> +err:
>> +	put_device(dev);
>> +	return ERR_PTR(rc);
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_features_alloc, "CXL");
>> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
>> index 78a5c2c25982..cc53a597cae6 100644
>> --- a/drivers/cxl/core/port.c
>> +++ b/drivers/cxl/core/port.c
>> @@ -74,6 +74,9 @@ static int cxl_device_id(const struct device *dev)
>>  		return CXL_DEVICE_REGION;
>>  	if (dev->type == &cxl_pmu_type)
>>  		return CXL_DEVICE_PMU;
>> +	if (dev->type == &cxl_features_type)
>> +		return CXL_DEVICE_FEATURES;
>> +
>>  	return 0;
>>  }
>>  
>> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
>> index f6015f24ad38..ee29d1a1c8df 100644
>> --- a/drivers/cxl/cxl.h
>> +++ b/drivers/cxl/cxl.h
>> @@ -855,6 +855,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);
>>  #define CXL_DEVICE_PMEM_REGION		7
>>  #define CXL_DEVICE_DAX_REGION		8
>>  #define CXL_DEVICE_PMU			9
>> +#define CXL_DEVICE_FEATURES		10
>>  
>>  #define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
>>  #define CXL_MODALIAS_FMT "cxl:t%d"
>> diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
>> new file mode 100644
>> index 000000000000..644add26975f
>> --- /dev/null
>> +++ b/drivers/cxl/features.c
>> @@ -0,0 +1,44 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/* Copyright(c) 2024,2025 Intel Corporation. All rights reserved. */
>> +#include <linux/device.h>
>> +#include <linux/module.h>
>> +#include <linux/pci.h>
>> +#include <cxl/features.h>
>> +
>> +#include "cxl.h"
>> +
>> +static int cxl_features_probe(struct device *dev)
>> +{
>> +	struct cxl_features *features = to_cxl_features(dev);
>> +	struct cxl_features_state *cfs __free(kfree) =
>> +		kzalloc(sizeof(*cfs), GFP_KERNEL);
> 
> Just makes this:
> 
> 	struct cxl_features_state *cfs = devm_kzalloc(sizeof(*cfs), GFP_KERNEL);
>> +
>> +	if (!cfs)
>> +		return -ENOMEM;
>> +
>> +	cfs->features = features;
>> +	dev_set_drvdata(dev, no_free_ptr(cfs));
> 
> ...skip the no_free_ptr()
> 
>> +
>> +	return 0;
>> +}
>> +
>> +static void cxl_features_remove(struct device *dev)
>> +{
>> +	struct cxl_features_state *cfs = dev_get_drvdata(dev);
>> +
>> +	kfree(cfs);
>> +}
> 
> ...and skip the need for a ->remove() routine altogether.
> 
>> +
>> +static struct cxl_driver cxl_features_driver = {
>> +	.name = "cxl_features",
>> +	.probe = cxl_features_probe,
>> +	.remove = cxl_features_remove,
>> +	.id = CXL_DEVICE_FEATURES,
>> +};
>> +
>> +module_cxl_driver(cxl_features_driver);
>> +
>> +MODULE_DESCRIPTION("CXL: Features");
> 
> Maybe a few more words:
> 
>     "CXL 'Feature' command support via FWCTL"
> 
>> +MODULE_LICENSE("GPL");
>> +MODULE_IMPORT_NS("CXL");
>> +MODULE_ALIAS_CXL(CXL_DEVICE_FEATURES);
>> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
>> index 6d94ff4a4f1a..eb68dd3f8b21 100644
>> --- a/drivers/cxl/pci.c
>> +++ b/drivers/cxl/pci.c
>> @@ -386,6 +386,21 @@ static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox,
>>  	return rc;
>>  }
>>  
>> +static int cxl_pci_setup_features(struct cxl_memdev_state *mds)
>> +{
>> +	struct cxl_dev_state *cxlds = &mds->cxlds;
> 
> Looks like this function wants to take @cxlds directly and skip passing
> in @mds.
> 
>> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
>> +	struct cxl_features *features;
>> +
>> +	features = cxl_features_alloc(cxl_mbox, cxlds->dev);
>> +	if (IS_ERR(features))
>> +		return PTR_ERR(features);
>> +
>> +	cxl_mbox->features = features;
>> +
>> +	return 0;
>> +}
>> +
>>  static int cxl_pci_setup_mailbox(struct cxl_memdev_state *mds, bool irq_avail)
>>  {
>>  	struct cxl_dev_state *cxlds = &mds->cxlds;
>> @@ -980,6 +995,10 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>>  	if (rc)
>>  		return rc;
>>  
>> +	rc = cxl_pci_setup_features(mds);
>> +	if (rc)
>> +		return rc;
>> +
>>  	rc = cxl_set_timestamp(mds);
>>  	if (rc)
>>  		return rc;
>> diff --git a/include/cxl/features.h b/include/cxl/features.h
>> new file mode 100644
>> index 000000000000..b92da1e92780
>> --- /dev/null
>> +++ b/include/cxl/features.h
>> @@ -0,0 +1,23 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> +/* Copyright(c) 2024-2025 Intel Corporation. */
>> +#ifndef __CXL_FEATURES_H__
>> +#define __CXL_FEATURES_H__
>> +
>> +struct cxl_mailbox;
>> +
>> +struct cxl_features {
>> +	int id;
>> +	struct device dev;
>> +	struct cxl_mailbox *cxl_mbox;
> 
> A bit uncomfortable with a 'struct cxl_features' instance having a
> pointer to a 'struct cxl_mailbox' without holding a refcount.
> 
> This all works becase cxl_pci_setup_features() knows that the @parent
> it is passing to cxl_features_alloc() is device that will also
> unregister the @features device before destroying the 'struct
> cxl_mailbox' instance in @cxlds.
> 
> 
>> +};
>> +#define to_cxl_features(dev) container_of(dev, struct cxl_features, dev)
>> +
>> +struct cxl_features_state {
>> +	struct cxl_features *features;
>> +	int num_features;
>> +};
>> +
>> +struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
>> +					struct device *parent);
> 
> ...but this is proposed as a public function outside of cxl_pci. How
> does cxl_features_alloc() understand the lifetime relationships between
> its @cxl_mbox and @parent arguments?
> 
> I would have cxl_features_alloc() (devm_cxl_add_features()) take a
> 'struct cxl_dev_state *' argument which makes it clear that the features
> device must be unregistered before 'struct cxl_dev_state *' can be
> destroyed.
> 
> ...or something to constrain the freedom of a new globally public
> export. Does this need to be made available to any other drivers besides
> cxl_pci for now?
> 
> I.e. should this be include/cxl/features.h or drivers/cxl/features.h?

Re: [PATCH v1 0/19] cxl: Add CXL feature commands support via fwctl

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:31 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> v1:
> - Create a CXL features driver to handle all feature command related bits
>   including FWCTL support. (Dan)
> - Tested against CXL CLI unit tests for FWCTL.
> - See individual patches for detailed changes from RFC v2.
> 
> RFC v2:
> - Dropped 1/13 and 2/13 from previous version. Merged upstream already.
> - Combined changes from Shiju for "get supported features"
> - Addressed comments from Jonathan and Jason
> - See specific changes in individual patch revision history
> - Added hardware command info to FWCTL
> - Added filtering to set feature command
> - Added documentation
> 
> This series add support for CXL feature commands using the FWCTL framework [1].
> The code has been tested with CXL CLI unit tests for FWCTL.
> 
> Patches 1-7 are shared with EDAC series [1] and there will be an immuable branch
> once accepted post review.

Looks like immutable should be maybe 3-8?  Perhaps reorder for next version
as odd to have middle of series in an immutable branch.

> 
> A CXL features driver is created to handle all CXL mailbox feature commands related
> functionalities as suggested by Dan. The 3 mailbox commands "Get Supported Features",
> "Get Feature", and "Set Feature" are implemented. The "get" commands under the
> FWCTL_RPC_CONFIGURATION policy, the "set" command checks the policy depending
> on the effect of the feature. All mailbox commands for CXL provides an effects
> table that describes the effects of a command when performed on the device.
> For CXL features, there is also an effects field that describes the effects
> a feature write operation has on the device per feature. The security policy
> is checked against this feature specific effects field. Looking for discussion
> on matching the CXL spec defined effects with the FWCTL security policy.
> 
> The code is based off of v3 of FWCTL series [2] posted by Jason and rebased on top of
> v6.13-rc5.
> 
> [1]: https://lore.kernel.org/linux-cxl/20250106121017.1620-1-shiju.jose@huawei.com/T/#t
> [2]: https://lore.kernel.org/linux-cxl/0-v3-960f17f90f17+516-fwctl_jgg@nvidia.com/#r
> ---
> 
> Dave Jiang (17):
>       cxl: Refactor user ioctl command path from mds to mailbox
>       cxl: Add skeletal features driver
>       cxl: Enumerate feature commands
>       cxl: Add Get Supported Features command for kernel usage
>       cxl: Add features driver attribute to emit number of features supported
>       cxl/test: Add Get Supported Features mailbox command support
>       cxl: Setup exclusive CXL features that are reserved for the kernel
>       cxl: Add FWCTL support to the CXL features driver
>       cxl: Add support for get driver information
>       cxl: Move cxl_mem.h under uapi to cxl exclusive directory
>       cxl: Move cxl feature command structs to user header
>       cxl: Add support for fwctl RPC command to enable CXL feature commands
>       cxl: Add support to handle user feature commands for get feature
>       cxl: Add support to handle user feature commands for set feature
>       cxl/test: Add Get Feature support to cxl_test
>       cxl/test: Add Set Feature support to cxl_test
>       fwctl/cxl: Add documentation to FWCTL CXL
> 
> Shiju Jose (2):
>       cxl/mbox: Add GET_FEATURE mailbox command
>       cxl/mbox: Add SET_FEATURE mailbox command
> 
>  Documentation/userspace-api/fwctl/fwctl-cxl.rst |  81 +++++++
>  Documentation/userspace-api/fwctl/index.rst     |   1 +
>  drivers/cxl/Kconfig                             |   9 +
>  drivers/cxl/Makefile                            |   3 +
>  drivers/cxl/core/Makefile                       |   1 +
>  drivers/cxl/core/core.h                         |   7 +-
>  drivers/cxl/core/features.c                     | 287 +++++++++++++++++++++++
>  drivers/cxl/core/mbox.c                         | 143 ++++++++----
>  drivers/cxl/core/memdev.c                       |  22 +-
>  drivers/cxl/core/port.c                         |   3 +
>  drivers/cxl/cxl.h                               |   3 +
>  drivers/cxl/cxlmem.h                            |  45 +---
>  drivers/cxl/features.c                          | 558 ++++++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/pci.c                               |  19 ++
>  include/cxl/features.h                          |  75 ++++++
>  include/cxl/mailbox.h                           |  46 +++-
>  include/uapi/cxl/features.h                     | 135 +++++++++++
>  include/uapi/{linux/cxl_mem.h => cxl/mem.h}     |   0
>  include/uapi/fwctl/cxl.h                        |  60 +++++
>  include/uapi/fwctl/fwctl.h                      |   1 +
>  tools/testing/cxl/Kbuild                        |   1 +
>  tools/testing/cxl/test/mem.c                    | 194 +++++++++++++++
>  22 files changed, 1597 insertions(+), 97 deletions(-)

Re: [PATCH v1 02/19] cxl: Add skeletal features driver

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:33 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add the basic bits of a features driver to handle all CXL feature related
> services. The driver is expected to handle all CXL mailbox feature command
> related operations.
> 
> Suggested-by: Dan Williams <dan.j.williams@intel.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
I've tried not to duplicate Dan's feedback but might well
have done in places.

I have more or less completely forgotten earlier discussions
so may well repeat comments long addressed.

Jonathan


> diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
> new file mode 100644
> index 000000000000..eb6eb191a32e
> --- /dev/null
> +++ b/drivers/cxl/core/features.c
> @@ -0,0 +1,71 @@

> +struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
> +					struct device *parent)
> +{
> +	struct device *dev;
> +	int rc;
> +
> +	struct cxl_features *features __free(kfree) =
> +		kzalloc(sizeof(*features), GFP_KERNEL);
> +	if (!features)
> +		return ERR_PTR(-ENOMEM);
> +
> +	rc = ida_alloc_max(&cxl_features_ida, CXL_FEATURE_MAX_DEVS - 1,
> +			   GFP_KERNEL);
> +	if (rc < 0)
> +		return ERR_PTR(rc);
> +
> +	features->id = rc;
> +	features->cxl_mbox = cxl_mbox;
> +	dev = &features->dev;
> +	device_initialize(dev);
> +	device_set_pm_not_required(dev);
> +	dev->parent = parent;
> +	dev->bus = &cxl_bus_type;
> +	dev->type = &cxl_features_type;
> +	rc = dev_set_name(dev, "features%d", features->id);
> +	if (rc)
> +		goto err;
> +
> +	rc = device_add(dev);
> +	if (rc)
> +		goto err;
> +
> +	rc = devm_add_action_or_reset(parent, remove_features_dev, dev);
> +	if (rc)
		return rc;

If this fails the or_reset() means it has called device_unregister()
so put_device() should not be needed I think (though I may have missed
a reference counter increase somewhere).

On top of the release freeing features that Dan called out.


> +		goto err;
> +
> +	return no_free_ptr(features);
> +
> +err:
> +	put_device(dev);
> +	return ERR_PTR(rc);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_features_alloc, "CXL");

> diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
> new file mode 100644
> index 000000000000..644add26975f
> --- /dev/null
> +++ b/drivers/cxl/features.c
> @@ -0,0 +1,44 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright(c) 2024,2025 Intel Corporation. All rights reserved. */
> +#include <linux/device.h>
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <cxl/features.h>
> +
> +#include "cxl.h"
> +
> +static int cxl_features_probe(struct device *dev)
> +{
> +	struct cxl_features *features = to_cxl_features(dev);
> +	struct cxl_features_state *cfs __free(kfree) =
> +		kzalloc(sizeof(*cfs), GFP_KERNEL);

Maybe devm_ and no need for the scoped free handling
or for now the remove function?


> +
> +	if (!cfs)
> +		return -ENOMEM;
> +
> +	cfs->features = features;
> +	dev_set_drvdata(dev, no_free_ptr(cfs));
> +
> +	return 0;
> +}
> +
> +static void cxl_features_remove(struct device *dev)
> +{
> +	struct cxl_features_state *cfs = dev_get_drvdata(dev);
> +
> +	kfree(cfs);
> +}
> +
> +static struct cxl_driver cxl_features_driver = {
> +	.name = "cxl_features",
> +	.probe = cxl_features_probe,
> +	.remove = cxl_features_remove,
> +	.id = CXL_DEVICE_FEATURES,
> +};
> +
> +module_cxl_driver(cxl_features_driver);
> +
> +MODULE_DESCRIPTION("CXL: Features");
> +MODULE_LICENSE("GPL");
> +MODULE_IMPORT_NS("CXL");
> +MODULE_ALIAS_CXL(CXL_DEVICE_FEATURES);

> diff --git a/include/cxl/features.h b/include/cxl/features.h
> new file mode 100644
> index 000000000000..b92da1e92780
> --- /dev/null
> +++ b/include/cxl/features.h
> @@ -0,0 +1,23 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/* Copyright(c) 2024-2025 Intel Corporation. */
> +#ifndef __CXL_FEATURES_H__
> +#define __CXL_FEATURES_H__

#include <linux/device.h> for struct device definition.

> +
> +struct cxl_mailbox;
> +
> +struct cxl_features {
> +	int id;
> +	struct device dev;

Trivial thing but I'd put the device first as we tend
to end up with maths like to_cxl_features() that can be simpler
if it is there.


> +	struct cxl_mailbox *cxl_mbox;
> +};
> +#define to_cxl_features(dev) container_of(dev, struct cxl_features, dev)
> +
> +struct cxl_features_state {
> +	struct cxl_features *features;
> +	int num_features;
> +};
> +
> +struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
> +					struct device *parent);
> +
> +#endif

Re: [PATCH v1 03/19] cxl: Enumerate feature commands

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:34 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add feature commands enumeration code in order to detect and enumerate
> the 3 feature related commands "get supported features", "get feature",
> and "set feature". The enumeration will help determine whether the driver
> can issue any of the 3 commands to the device.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/mbox.c | 41 +++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxlmem.h    |  3 +++
>  include/cxl/features.h  |  7 +++++++
>  include/cxl/mailbox.h   |  1 +
>  4 files changed, 52 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index bdb8f060f2c1..5e21ff99d70f 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -38,6 +38,21 @@ static bool cxl_raw_allow_all;
>  	.flags = _flags,                                                       \
>  	}
>  
> +#define cxl_for_each_feature_cmd(cmd)                                          \
> +	for ((cmd) = &cxl_feature_commands[0];                                 \
> +	     ((cmd) - cxl_feature_commands) < ARRAY_SIZE(cxl_feature_commands); (cmd)++)
> +
> +#define CXL_FEATURE_CMD(_id, sin, sout, _flags)                                \
> +	[CXL_FEATURE_ID_##_id] = {                                             \
> +	.info =	{                                                              \
> +			.id = CXL_FEATURE_ID_##_id,                            \
> +			.size_in = sin,                                        \
> +			.size_out = sout,                                      \
> +		},                                                             \
> +	.opcode = CXL_MBOX_OP_##_id,                                           \
> +	.flags = _flags,                                                       \
> +	}
> +
>  #define CXL_VARIABLE_PAYLOAD	~0U
>  /*
>   * This table defines the supported mailbox commands for the driver. This table
> @@ -69,6 +84,13 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
>  };
>  
> +#define CXL_FEATURE_COMMAND_ID_MAX 3
Why do we need this define separately from CXL_FEATURE_ID_MAX as I'm fairly sure they 
must always be the same.  A [] array would work here too.

> +static struct cxl_mem_command cxl_feature_commands[CXL_FEATURE_COMMAND_ID_MAX] = {
> +	CXL_FEATURE_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
> +	CXL_FEATURE_CMD(GET_FEATURE, 0xf, CXL_VARIABLE_PAYLOAD, 0),
> +	CXL_FEATURE_CMD(SET_FEATURE, CXL_VARIABLE_PAYLOAD, 0, 0),
> +};

> diff --git a/include/cxl/features.h b/include/cxl/features.h
> index b92da1e92780..7a8be3c621a1 100644
> --- a/include/cxl/features.h
> +++ b/include/cxl/features.h
> @@ -5,6 +5,13 @@
>  
>  struct cxl_mailbox;
>  
> +enum feature_cmds {
> +	CXL_FEATURE_ID_GET_SUPPORTED_FEATURES = 0,
> +	CXL_FEATURE_ID_GET_FEATURE,
> +	CXL_FEATURE_ID_SET_FEATURE,
> +	CXL_FEATURE_ID_MAX,

No comma on that one as it's meant to always be the terminator.

> +};

Re: [PATCH v1 04/19] cxl: Add Get Supported Features command for kernel usage

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:35 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
> The command retrieve the list of supported device-specific features
> (identified by UUID) and general information about each Feature.
> 
> The driver will retrieve the feature entries in order to make checks and
> provide information for the Get Feature and Set Feature command. One of
> the main piece of information retrieved are the effects a Set Feature
> command would have for a particular feature.
> 
> Co-developed-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
One stray change below that either wants dropping or pushing to
earlier patch.

Not sure why I haven't given a tag on this one before now
in either of the patch sets it features in.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> ---
> v1:
> - Change input param from cxlds to cxl_mbox
> - Move mbox_out declaration inline. (Jonathan)
> - Fix __counted_by() input. (Jonathan)
> - Return count for cxl_get_supported_features_count(). (Dan)
> - Remove goto from cxl_get_supported_features(). (Dan)
> - Return cxl_get_supported_feature_entry() directly. (Dan)
> - Drop user path enumeration. (Dan)
> - Move to the feature driver model. (Dan)
> - Add support for 0 feature data requested.
> - Add missing increment of feature data ptr during cop

> diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
> index 644add26975f..a5949312a4ab 100644
> --- a/drivers/cxl/features.c
> +++ b/drivers/cxl/features.c

>  static int cxl_features_probe(struct device *dev)
>  {
>  	struct cxl_features *features = to_cxl_features(dev);
> +	int rc;
> +
>  	struct cxl_features_state *cfs __free(kfree) =
>  		kzalloc(sizeof(*cfs), GFP_KERNEL);
> -
Stray change that belongs in earlier patch if anywhere.


>  	if (!cfs)
>  		return -ENOMEM;
>  
>  	cfs->features = features;
> +	rc = cxl_get_supported_features(cfs);
> +	if (rc)
> +		return rc;
> +
>  	dev_set_drvdata(dev, no_free_ptr(cfs));
>  
>  	return 0;

Re: [PATCH v1 05/19] cxl: Add features driver attribute to emit number of features supported

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:36 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Enable sysfs attribute emission of the number of features supported by the
> driver/device. This is useful for userspace to determine the number of features
> to query for.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Seems reasonable.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Re: [PATCH v1 06/19] cxl/test: Add Get Supported Features mailbox command support

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:37 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add cxl-test emulation of Get Supported Features mailbox command.
> Currently only adding a test feature with feature identifier of
> all f's for testing.
> 
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Acked-by: Dan Williams <dan.j.williams@intel.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v1:
> - Fix setting of num_entries endieness issue. (Jonathan)
> ---
>  include/cxl/features.h       | 23 ++++++++++
>  tools/testing/cxl/test/mem.c | 87 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 110 insertions(+)
> 
> diff --git a/include/cxl/features.h b/include/cxl/features.h
> index 429b9782667c..8ff7d90932d6 100644
> --- a/include/cxl/features.h
> +++ b/include/cxl/features.h
> @@ -28,6 +28,29 @@ struct cxl_mbox_get_sup_feats_in {
>  	u8 reserved[2];
>  } __packed;
>  
> +/* CXL spec r3.2 Table 8-87 command effects */
> +#define CXL_CMD_CONFIG_CHANGE_COLD_RESET	BIT(0)
> +#define CXL_CMD_CONFIG_CHANGE_IMMEDIATE		BIT(1)
> +#define CXL_CMD_DATA_CHANGE_IMMEDIATE		BIT(2)
> +#define CXL_CMD_POLICY_CHANGE_IMMEDIATE		BIT(3)
> +#define CXL_CMD_LOG_CHANGE_IMMEDIATE		BIT(4)
> +#define CXL_CMD_SECURITY_STATE_CHANGE		BIT(5)
> +#define CXL_CMD_BACKGROUND			BIT(6)
> +#define CXL_CMD_BGCMD_ABORT_SUPPORTED		BIT(7)
> +#define CXL_CMD_EFFECTS_EXTEND			BIT(9)
> +#define CXL_CMD_CONFIG_CHANGE_CONV_RESET	BIT(10)
> +#define CXL_CMD_CONFIG_CHANGE_CXL_RESET		BIT(11)
> +
> +/*
> + * CXL spec r3.2 Table 8-109
> + * Get Supported Features Supported Feature Entry
> + * Atribute Flags
> + */
> +#define CXL_FEATURE_F_CHANGEABLE		BIT(0)
> +#define CXL_FEATURE_F_PERSIST_FW_UPDATE		BIT(4)
> +#define CXL_FEATURE_F_DEFAULT_SEL		BIT(5)
> +#define CXL_FEATURE_F_SAVED_SEL			BIT(6)
> +
>  struct cxl_feat_entry {
>  	uuid_t uuid;
>  	__le16 id;

A little odd to bring these defs in with the CXL test support
but I suppose that's really just a patch ordering thing.

Jonathan

Re: [PATCH v1 07/19] cxl/mbox: Add GET_FEATURE mailbox command

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:38 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> From: Shiju Jose <shiju.jose@huawei.com>
> 
> Add support for GET_FEATURE mailbox command.
> 
> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
> The settings of a feature can be retrieved using Get Feature command.
> CXL spec 3.1 section 8.2.9.6.2 describes Get Feature command.
> 
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Re: [PATCH v1 08/19] cxl/mbox: Add SET_FEATURE mailbox command

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:39 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> From: Shiju Jose <shiju.jose@huawei.com>
> 
> Add support for SET_FEATURE mailbox command.
> 
> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
> CXL devices supports features with changeable attributes.
> The settings of a feature can be optionally modified using Set Feature
> command.
> CXL spec 3.1 section 8.2.9.6.3 describes Set Feature command.
> 
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Trivial comment inline.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> ---
> v1:
> - CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER enum must be set to 0.
> ---
>  drivers/cxl/core/features.c  | 92 ++++++++++++++++++++++++++++++++++++
>  include/cxl/features.h       | 32 +++++++++++++
>  tools/testing/cxl/test/mem.c |  1 +
>  3 files changed, 125 insertions(+)
> 
> diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
> index ab9386b53a95..932e82b52f90 100644
> --- a/drivers/cxl/core/features.c
> +++ b/drivers/cxl/core/features.c
> @@ -171,3 +171,95 @@ size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
>  	return data_rcvd_size;
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_get_feature, "CXL");
> +
> +/*
> + * FEAT_DATA_MIN_PAYLOAD_SIZE - min extra number of bytes should be
> + * available in the mailbox for storing the actual feature data so that
> + * the feature data transfer would work as expected.
> + */
> +#define FEAT_DATA_MIN_PAYLOAD_SIZE 10
> +int cxl_set_feature(struct cxl_features *features,
> +		    const uuid_t feat_uuid, u8 feat_version,
> +		    void *feat_data, size_t feat_data_size,
> +		    u32 feat_flag, u16 offset, u16 *return_code)
> +{
> +	struct cxl_memdev_set_feat_pi {
> +		struct cxl_mbox_set_feat_hdr hdr;
> +		u8 feat_data[];
> +	}  __packed;
> +	size_t data_in_size, data_sent_size = 0;
> +	struct cxl_features_state *cfs;
> +	struct cxl_mbox_cmd mbox_cmd;
> +	struct cxl_mailbox *cxl_mbox;
> +	size_t hdr_size;
> +	int rc = 0;
> +
> +	if (return_code)
> +		*return_code = CXL_MBOX_CMD_RC_INPUT;
> +
> +	cfs = dev_get_drvdata(&features->dev);
> +	if (!cfs)
> +		return -EOPNOTSUPP;
> +
> +	if (!cxl_feature_enabled(cfs, CXL_MBOX_OP_SET_FEATURE))
> +		return -EOPNOTSUPP;
> +
> +	cxl_mbox = features->cxl_mbox;
> +
> +	struct cxl_memdev_set_feat_pi *pi __free(kfree) =
> +					kmalloc(cxl_mbox->payload_size, GFP_KERNEL);

I'd just align one tab after line above. This sort of 'maximal' alignment
is fiddly to deal with.


> +	pi->hdr.uuid = feat_uuid;
> +	pi->hdr.version = feat_version;
> +	feat_flag &= ~CXL_SET_FEAT_FLAG_DATA_TRANSFER_MASK;
> +	feat_flag |= CXL_SET_FEAT_FLAG_DATA_SAVED_ACROSS_RESET;
> +	hdr_size = sizeof(pi->hdr);
> +	/*
> +	 * Check minimum mbox payload size is available for
> +	 * the feature data transfer.
> +	 */
> +	if (hdr_size + FEAT_DATA_MIN_PAYLOAD_SIZE > cxl_mbox->payload_size)
> +		return -ENOMEM;
> +
> +	if ((hdr_size + feat_data_size) <= cxl_mbox->payload_size) {
> +		pi->hdr.flags = cpu_to_le32(feat_flag |
> +				       CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER);
> +		data_in_size = feat_data_size;
> +	} else {
> +		pi->hdr.flags = cpu_to_le32(feat_flag |
> +				       CXL_SET_FEAT_FLAG_INITIATE_DATA_TRANSFER);
> +		data_in_size = cxl_mbox->payload_size - hdr_size;
> +	}
> +
> +	do {
> +		pi->hdr.offset = cpu_to_le16(offset + data_sent_size);
> +		memcpy(pi->feat_data, feat_data + data_sent_size, data_in_size);
> +		mbox_cmd = (struct cxl_mbox_cmd) {
> +			.opcode = CXL_MBOX_OP_SET_FEATURE,
> +			.size_in = hdr_size + data_in_size,
> +			.payload_in = pi,
> +		};
> +		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
> +		if (rc < 0) {
> +			if (return_code)
> +				*return_code = mbox_cmd.return_code;
> +			return rc;
> +		}
> +
> +		data_sent_size += data_in_size;
> +		if (data_sent_size >= feat_data_size) {
> +			if (return_code)
> +				*return_code = CXL_MBOX_CMD_RC_SUCCESS;
> +			return 0;
> +		}
> +
> +		if ((feat_data_size - data_sent_size) <= (cxl_mbox->payload_size - hdr_size)) {
> +			data_in_size = feat_data_size - data_sent_size;
> +			pi->hdr.flags = cpu_to_le32(feat_flag |
> +					       CXL_SET_FEAT_FLAG_FINISH_DATA_TRANSFER);
> +		} else {
> +			pi->hdr.flags = cpu_to_le32(feat_flag |
> +					       CXL_SET_FEAT_FLAG_CONTINUE_DATA_TRANSFER);
> +		}
> +	} while (true);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_set_feature, "CXL");

Re: [PATCH v1 09/19] cxl: Setup exclusive CXL features that are reserved for the kernel

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:40 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Certain features will be exclusively used by components such as in
> kernel RAS driver. Setup an exclusion list that can be later filtered
> out before exposing to user space.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

Nice
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

This also finally led me to the footnote I'd lost in the spec
that says that row sparing is preferred to PPR where possible :)

Jonathan

Re: [PATCH v1 10/19] cxl: Add FWCTL support to the CXL features driver

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:41 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add fwctl support code to allow sending of CXL feature commands from
> userspace through as ioctls via FWCTL. Provide initial setup bits.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
> index f9b27fb5c161..2c3817592308 100644
> --- a/include/uapi/fwctl/fwctl.h
> +++ b/include/uapi/fwctl/fwctl.h
> @@ -43,6 +43,7 @@ enum {
>  enum fwctl_device_type {
>  	FWCTL_DEVICE_TYPE_ERROR = 0,
>  	FWCTL_DEVICE_TYPE_MLX5 = 1,
> +	FWCTL_DEVICE_TYPE_CXL,
Seems we are giving explicit numbers but not for CXL?
I don't greatly care either way but we should be consistent.  Jason?
>  };
>  
>  /**

Re: [PATCH v1 11/19] cxl: Add support for get driver information

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:42 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add definition for fwctl_ops->info() to return driver information. The
> function will return a mask of the feature mailbox commands supported
> by the fwctl char device.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

One follow through style thing otherwise lgtm
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> diff --git a/include/uapi/fwctl/cxl.h b/include/uapi/fwctl/cxl.h
> new file mode 100644
> index 000000000000..79b822dbfafd
> --- /dev/null
> +++ b/include/uapi/fwctl/cxl.h
> @@ -0,0 +1,31 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Copyright (c) 2024, Intel Corporation
> + *
> + * These are definitions for the mailbox command interface of CXL subsystem.
> + */
> +#ifndef _UAPI_FWCTL_CXL_H_
> +#define _UAPI_FWCTL_CXL_H_
> +
> +#include <linux/types.h>
> +
> +enum feature_cmds {
> +	CXL_FEATURE_ID_GET_SUPPORTED_FEATURES = 0,
> +	CXL_FEATURE_ID_GET_FEATURE,
> +	CXL_FEATURE_ID_SET_FEATURE,
> +	CXL_FEATURE_ID_MAX,

No trailing comma on this terminating item.

> +};
> +
> +/**
> + * struct fwctl_info_cxl - ioctl(FWCTL_INFO) out_device_data
> + * @cmd_mask: Mask indicate which commands are supported based on 'enum feature_cmds'
> + *
> + * Return basic information about the FW interface available.
> + *
> + * nr_commands is number of hardware commands the driver supports. Use
> + * FWCTL_CMD_HW_INFO ioctl to request additional information.
> + */
> +struct fwctl_info_cxl {
> +	__u32 cmd_mask;
> +};
> +#endif

Re: [PATCH v1 12/19] cxl: Move cxl_mem.h under uapi to cxl exclusive directory

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:43 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> In preparation of more cxl headers, create a cxl directory under
> uapi.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Re: [PATCH v1 13/19] cxl: Move cxl feature command structs to user header

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:44 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> In perparation for cxl fwctl enabling, move data structures related to
> cxl feature commands to a user header file.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

Reviewed-by; Jonathan Cameron <Jonathan.Cameron@huawei.com>

Re: [PATCH v1 13/19] cxl: Move cxl feature command structs to user header

[Jonathan Cameron]

On Thu, 23 Jan 2025 18:12:54 +0000
Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:

> On Wed, 22 Jan 2025 16:50:44 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
> > In perparation for cxl fwctl enabling, move data structures related to
> > cxl feature commands to a user header file.
> > 
> > Signed-off-by: Dave Jiang <dave.jiang@intel.com>  
> 
> Reviewed-by; Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

I missed the cancel button....

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:45 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> fwctl provides a fwctl_ops->fw_rpc() callback in order to issue ioctls
> to a device. The cxl fwctl driver will start by supporting the CXL
> feature commands: Get Supported Features, Get Feature, and Set Feature.
> 
> The fw_rpc() callback provides 'enum fwctl_rpc_scope' parameter where
> it indicates the security scope of the call. The Get Supported Features
> and Get Feature calls can be executed with the scope of
> FWCTL_RPC_CONFIGRATION. The Set Feature call is gated by the effects
> of the feature reported by Get Supported Features call for the specific
> feature.
> 
> Only "get supported features" is supported in this patch. Additional
> commands will be added in follow on patches. "Get supported features"
> will filter the features that are exclusive to the kernel and only
> report out features that are not kernel only.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
A few comments inline.

Jonathan

>

> diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
> index cc72e73ae8d6..e65fc8479d21 100644
> --- a/drivers/cxl/features.c
> +++ b/drivers/cxl/features.c


> +static struct cxl_mem_command *
> +cxlctl_get_valid_hw_command(struct cxl_features_state *cfs,
> +			    const struct fwctl_rpc_cxl *rpc_in,
> +			    enum fwctl_rpc_scope scope)
> +{
> +	struct cxl_mem_command *cmd;
> +
> +	if (!cfs->num_features)
> +		return ERR_PTR(-EOPNOTSUPP);
> +
> +	cmd = cxl_find_feature_command_by_id(rpc_in->command_id);
> +	if (!cmd)
> +		return ERR_PTR(-ENOENT);
> +
> +	if (!cxl_feature_enabled(cfs, cmd->opcode))
> +		return ERR_PTR(-EPERM);
> +
> +	switch (cmd->opcode) {
> +	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
> +	case CXL_MBOX_OP_GET_FEATURE:
> +		if (scope >= FWCTL_RPC_CONFIGURATION)
> +			return cmd;
> +		break;
> +	case CXL_MBOX_OP_SET_FEATURE:
> +		if (cxlctl_validate_set_features(cfs, rpc_in, scope))
> +			return cmd;
> +		break;

return ERR_PTR(-EINVAL); here perhaps.

> +	default:
> +		return ERR_PTR(-EINVAL);
> +	}
> +
> +	return ERR_PTR(-EINVAL);
Maybe better to return in the error cases above to make it clear locally
to them that they are errors.

> +}

> diff --git a/include/uapi/cxl/features.h b/include/uapi/cxl/features.h
> index 18b74252058a..55fa5b2a5d17 100644
> --- a/include/uapi/cxl/features.h
> +++ b/include/uapi/cxl/features.h
> @@ -14,6 +14,19 @@
>  #include <linux/uuid.h>
>  #endif
>  
> +/* CXL spec r3.2 Table 8-87 command effects */
> +#define CXL_CMD_CONFIG_CHANGE_COLD_RESET	BIT(0)

These are duplicated with those in include/cxl/features.
Can we avoid that?


> +#define CXL_CMD_CONFIG_CHANGE_IMMEDIATE		BIT(1)
> +#define CXL_CMD_DATA_CHANGE_IMMEDIATE		BIT(2)
> +#define CXL_CMD_POLICY_CHANGE_IMMEDIATE		BIT(3)
> +#define CXL_CMD_LOG_CHANGE_IMMEDIATE		BIT(4)
> +#define CXL_CMD_SECURITY_STATE_CHANGE		BIT(5)
> +#define CXL_CMD_BACKGROUND			BIT(6)
> +#define CXL_CMD_BGCMD_ABORT_SUPPORTED		BIT(7)
> +#define CXL_CMD_EFFECTS_EXTEND			BIT(9)
> +#define CXL_CMD_CONFIG_CHANGE_CONV_RESET	BIT(10)
> +#define CXL_CMD_CONFIG_CHANGE_CXL_RESET		BIT(11)
> +
>  /* Get Supported Features (0x500h) CXL r3.1 8.2.9.6.1 */
>  struct cxl_mbox_get_sup_feats_in {
>  	__le32 count;

Re: [PATCH v1 15/19] cxl: Add support to handle user feature commands for get feature

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:46 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add helper function to parse the user data from fwctl RPC ioctl and
> send the parsed input parameters to cxl_get_feature() call.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Trivial comment inline.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> ---
>  drivers/cxl/features.c | 46 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
> 
> diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
> index e65fc8479d21..7c726f1512d0 100644
> --- a/drivers/cxl/features.c
> +++ b/drivers/cxl/features.c
> @@ -146,6 +146,51 @@ static void *cxlctl_get_supported_features(struct cxl_features_state *cfs,
>  	return no_free_ptr(rpc_out);
>  }
>  
> +static void *cxlctl_get_feature(struct cxl_features_state *cfs,
> +				const struct fwctl_rpc_cxl *rpc_in,
> +				size_t *out_len)
> +{
> +	struct cxl_mbox_get_feat_in feat_in;
> +	u16 offset, count, return_code;
> +	size_t out_size = *out_len;
> +	void *feat_out;
> +
> +	if (rpc_in->op_size != sizeof(feat_in))
> +		return ERR_PTR(-EINVAL);
> +
> +	if (copy_from_user(&feat_in, u64_to_user_ptr(rpc_in->in_payload),
> +			   rpc_in->op_size))
> +		return ERR_PTR(-EFAULT);
> +
> +	offset = le16_to_cpu(feat_in.offset);
> +	count = le16_to_cpu(feat_in.count);
> +
> +	if (!count)
> +		return ERR_PTR(-EINVAL);
> +
> +	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
> +		kvzalloc(out_size, GFP_KERNEL);
> +	if (!rpc_out)
> +		return ERR_PTR(-ENOMEM);
> +
> +	feat_out = (void *)rpc_out->payload;

Fairly sure no need for the cast.  Should be fine given it's a void *
that you are assigning it to from another pointer type.
As such, maybe just use it inline in next call?

> +	out_size = cxl_get_feature(cfs->features, feat_in.uuid,
> +				   feat_in.selection, feat_out,
> +				   count, offset, &return_code);
> +	*out_len = sizeof(struct fwctl_rpc_cxl_out);
> +	if (!out_size) {
> +		rpc_out->size = 0;
> +		rpc_out->retval = return_code;
> +		return no_free_ptr(rpc_out);
> +	}
> +
> +	rpc_out->size = out_size;
> +	rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
> +	*out_len += out_size;
> +
> +	return no_free_ptr(rpc_out);
> +}

Re: [PATCH v1 16/19] cxl: Add support to handle user feature commands for set feature

[Jonathan Cameron]

On Wed, 22 Jan 2025 16:50:47 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add helper function to parse the user data from fwctl RPC ioctl and
> send the parsed input parameters to cxl_set_feature() call.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
LGTM
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Re: [PATCH v1 10/19] cxl: Add FWCTL support to the CXL features driver

[Jason Gunthorpe]

On Thu, Jan 23, 2025 at 06:04:30PM +0000, Jonathan Cameron wrote:
> > diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
> > index f9b27fb5c161..2c3817592308 100644
> > --- a/include/uapi/fwctl/fwctl.h
> > +++ b/include/uapi/fwctl/fwctl.h
> > @@ -43,6 +43,7 @@ enum {
> >  enum fwctl_device_type {
> >  	FWCTL_DEVICE_TYPE_ERROR = 0,
> >  	FWCTL_DEVICE_TYPE_MLX5 = 1,
> > +	FWCTL_DEVICE_TYPE_CXL,
> Seems we are giving explicit numbers but not for CXL?
> I don't greatly care either way but we should be consistent.  Jason?

Please write out the numbers in the enum, it aids backporters in the
long run.

Jason

Re: [PATCH v1 02/19] cxl: Add skeletal features driver

[Dan Williams]

Dave Jiang wrote:
> 
> 
> On 1/22/25 8:59 PM, Dan Williams wrote:
> > Dave Jiang wrote:
> >> Add the basic bits of a features driver to handle all CXL feature related
> >> services. The driver is expected to handle all CXL mailbox feature command
> >> related operations.
> >>
> >> Suggested-by: Dan Williams <dan.j.williams@intel.com>
> >> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
[..]
> > 
> > Does this need its own ida? I expect it could just use the same id as
> > the memdev, and that saves some potential confusion of which
> > memory-devices support features and which features interface correlates
> > with which device.
> > 
> > ...or how do you imagine that working?
> 
> One of the issue was that the feature commands query happens before
> memdev is created. So that ida doesn't exist yet. In order to share,
> we'll need to do a split initialization of the feature device. But
> really I was trying to separate the association of features from the
> memdev as they are independent of each other. i.e a type2 may not have
> memdev but have mailbox and features.

We could do what platform_device does and let the id be optionally
auto-selected or specified by the caller?

However, I think that's a can we can kick down the road until a Type-2
user with a mailbox, but without a memdev shows up. No signs of that on
the horizon unless someone wants to chime in to the contrary.

The Features commands of most interest to date are around memory RAS, so
it is likely those will have a memdev. The Type-2 series from Alejandro
registers a memdev for example.

Re: [PATCH v1 03/19] cxl: Enumerate feature commands

[Dan Williams]

Dave Jiang wrote:
> Add feature commands enumeration code in order to detect and enumerate
> the 3 feature related commands "get supported features", "get feature",
> and "set feature". The enumeration will help determine whether the driver
> can issue any of the 3 commands to the device.

> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/mbox.c | 41 +++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxlmem.h    |  3 +++
>  include/cxl/features.h  |  7 +++++++
>  include/cxl/mailbox.h   |  1 +
>  4 files changed, 52 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index bdb8f060f2c1..5e21ff99d70f 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -38,6 +38,21 @@ static bool cxl_raw_allow_all;
>  	.flags = _flags,                                                       \
>  	}
>  
> +#define cxl_for_each_feature_cmd(cmd)                                          \
> +	for ((cmd) = &cxl_feature_commands[0];                                 \
> +	     ((cmd) - cxl_feature_commands) < ARRAY_SIZE(cxl_feature_commands); (cmd)++)
> +
> +#define CXL_FEATURE_CMD(_id, sin, sout, _flags)                                \
> +	[CXL_FEATURE_ID_##_id] = {                                             \
> +	.info =	{                                                              \
> +			.id = CXL_FEATURE_ID_##_id,                            \
> +			.size_in = sin,                                        \
> +			.size_out = sout,                                      \
> +		},                                                             \
> +	.opcode = CXL_MBOX_OP_##_id,                                           \
> +	.flags = _flags,                                                       \
> +	}
> +
>  #define CXL_VARIABLE_PAYLOAD	~0U
>  /*
>   * This table defines the supported mailbox commands for the driver. This table
> @@ -69,6 +84,13 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
>  };
>  
> +#define CXL_FEATURE_COMMAND_ID_MAX 3
> +static struct cxl_mem_command cxl_feature_commands[CXL_FEATURE_COMMAND_ID_MAX] = {
> +	CXL_FEATURE_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
> +	CXL_FEATURE_CMD(GET_FEATURE, 0xf, CXL_VARIABLE_PAYLOAD, 0),
> +	CXL_FEATURE_CMD(SET_FEATURE, CXL_VARIABLE_PAYLOAD, 0, 0),
> +};

It's not immediately clear to me what this is far. The
cxl_mem_commands[] array is to put all the common work of defining new
commands for the CXL ioctl path into a table. Given these are all
fronted by FWCTL, does the driver need to do anything more than the
opcodes are supported.

In other words, I worry this approach is simply copying the organization
of cxl_walk_cel() for consistency sake. I had similar feedback on the
DCD series:

http://lore.kernel.org/67786b76a0c5_f58f294b1@dwillia2-xfh.jf.intel.com.notmuch

Just check for the opcodes you need and set a flag.

Re: [PATCH v1 04/19] cxl: Add Get Supported Features command for kernel usage

[Dan Williams]

Dave Jiang wrote:
> CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
> The command retrieve the list of supported device-specific features
> (identified by UUID) and general information about each Feature.
> 
> The driver will retrieve the feature entries in order to make checks and
> provide information for the Get Feature and Set Feature command. One of
> the main piece of information retrieved are the effects a Set Feature
> command would have for a particular feature.
> 
> Co-developed-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v1:
> - Change input param from cxlds to cxl_mbox
> - Move mbox_out declaration inline. (Jonathan)
> - Fix __counted_by() input. (Jonathan)
> - Return count for cxl_get_supported_features_count(). (Dan)
> - Remove goto from cxl_get_supported_features(). (Dan)
> - Return cxl_get_supported_feature_entry() directly. (Dan)
> - Drop user path enumeration. (Dan)
> - Move to the feature driver model. (Dan)
> - Add support for 0 feature data requested.
> - Add missing increment of feature data ptr during copy.
> ---
>  drivers/cxl/core/features.c |  28 +++++++
>  drivers/cxl/core/mbox.c     |   3 +-
>  drivers/cxl/cxl.h           |   2 +
>  drivers/cxl/features.c      | 146 +++++++++++++++++++++++++++++++++++-
>  include/cxl/features.h      |  32 ++++++++
>  5 files changed, 209 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
> index eb6eb191a32e..66a4b82910e6 100644
> --- a/drivers/cxl/core/features.c
> +++ b/drivers/cxl/core/features.c
> @@ -1,6 +1,7 @@
>  // SPDX-License-Identifier: GPL-2.0-only
>  /* Copyright(c) 2024-2025 Intel Corporation. All rights reserved. */
>  #include <linux/device.h>
> +#include <cxl/mailbox.h>
>  #include "cxl.h"
>  #include "core.h"
>  
> @@ -69,3 +70,30 @@ struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
>  	return ERR_PTR(rc);
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_features_alloc, "CXL");
> +
> +struct cxl_feat_entry *
> +cxl_get_supported_feature_entry(struct cxl_features *features,
> +				const uuid_t *feat_uuid)
> +{
> +	struct cxl_feat_entry *feat_entry;
> +	struct cxl_features_state *cfs;
> +	int count;
> +
> +	cfs = dev_get_drvdata(&features->dev);
> +	if (!cfs)
> +		return ERR_PTR(-EOPNOTSUPP);

How could this function be called outside of the driver being enabled?

If the driver might not be enabled when this is called, what stops it
from being disabled immediately after this check?

Went looking for the caller of this function... none in this patch set.

If this is for the EDAC use case, that enabling had better be triggered
from a known context where these questions have a satisfactory answer.
As it stands I think you can just delete it, right?

This has me thinking that the EDAC integration should just be a feature
of the cxl_features_driver.

...that last sentence made me itch because I don't want to have to
untangle future sentences like: A new feature of the features driver
features a new Feature to complement the existing Features and other
features.

So I might need to ask you to be explicit about at least using capital
'Feature' when talking about a command, 'cxl_features_device' when
talking about the device and 'cxl_features_driver' when talking about
the driver.

> diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
> index 644add26975f..a5949312a4ab 100644
> --- a/drivers/cxl/features.c
> +++ b/drivers/cxl/features.c
[..]
> +static int cxl_get_supported_features(struct cxl_features_state *cfs)
> +{
> +	int remain_feats, max_size, max_feats, start, rc, hdr_size;
> +	struct cxl_mailbox *cxl_mbox = cfs->features->cxl_mbox;
> +	int feat_size = sizeof(struct cxl_feat_entry);
> +	struct cxl_mbox_get_sup_feats_in mbox_in;
> +	struct cxl_feat_entry *entry;
> +	struct cxl_mbox_cmd mbox_cmd;
> +	struct cxl_mem_command *cmd;
> +	int count;
> +
> +	/* Get supported features is optional, need to check */
> +	cmd = cxl_find_feature_command(CXL_MBOX_OP_GET_SUPPORTED_FEATURES);
> +	if (!cmd)
> +		return -EOPNOTSUPP;
> +	if (!test_bit(cmd->info.id, cxl_mbox->feature_cmds))
> +		return -EOPNOTSUPP;

First, who is calling this function without knowing that features are
supported? Like how did we get here if something already did that first
level enumeration.

Otherwise, that looks like a lot of work just to ask the question, "did
the mailbox see CXL_MBOX_OP_GET_SUPPORTED_FEATURES" at init?

That could probably all be boiled down into an enum like:

enum cxl_features_capability {
	CXL_FEATURES_NONE,
	CXL_FEATURES_RO,
	CXL_FEATURES_RW,
};

...for tracking Feature command enumeration and optional write support.

The cxl_mem_commands organization is quirk of the CXL ioctl
implementation cxl_features is free to not copy it.

Longer term we need to think about how to make cxl_mailbox more
independent of the memory expander / cxl_pci use case. Whereby, perhaps
the consumer of the mailbox passes is in a callback that takes action on
each found command. Then we get out of this game of how to cache command
capabilities that are awkward fits depending on command set or driver
use case.

[..]

Re: [PATCH v1 05/19] cxl: Add features driver attribute to emit number of features supported

[Dan Williams]

Dave Jiang wrote:
> Enable sysfs attribute emission of the number of features supported by the
> driver/device. This is useful for userspace to determine the number of features
> to query for.
> 

It would need Documentation in Documentation/ABI/ for userspace to even
know it is there, but I am not convinced it's needed given we do not
have a similar sysfs attribute for CXL commands.

Is not the expectation for userspace to call Get Supported Features
itself?

Re: [PATCH v1 06/19] cxl/test: Add Get Supported Features mailbox command support

[Dan Williams]

Dave Jiang wrote:
> Add cxl-test emulation of Get Supported Features mailbox command.
> Currently only adding a test feature with feature identifier of
> all f's for testing.
> 
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Acked-by: Dan Williams <dan.j.williams@intel.com>

I see the ndctl pieces now, so lets upgrade this to:

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

...modulo trivial comment below:

> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v1:
> - Fix setting of num_entries endieness issue. (Jonathan)
> ---
>  include/cxl/features.h       | 23 ++++++++++
>  tools/testing/cxl/test/mem.c | 87 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 110 insertions(+)
> 
> diff --git a/include/cxl/features.h b/include/cxl/features.h
> index 429b9782667c..8ff7d90932d6 100644
> --- a/include/cxl/features.h
> +++ b/include/cxl/features.h
> @@ -28,6 +28,29 @@ struct cxl_mbox_get_sup_feats_in {
>  	u8 reserved[2];
>  } __packed;
>  
> +/* CXL spec r3.2 Table 8-87 command effects */
> +#define CXL_CMD_CONFIG_CHANGE_COLD_RESET	BIT(0)
> +#define CXL_CMD_CONFIG_CHANGE_IMMEDIATE		BIT(1)
> +#define CXL_CMD_DATA_CHANGE_IMMEDIATE		BIT(2)
> +#define CXL_CMD_POLICY_CHANGE_IMMEDIATE		BIT(3)
> +#define CXL_CMD_LOG_CHANGE_IMMEDIATE		BIT(4)
> +#define CXL_CMD_SECURITY_STATE_CHANGE		BIT(5)
> +#define CXL_CMD_BACKGROUND			BIT(6)
> +#define CXL_CMD_BGCMD_ABORT_SUPPORTED		BIT(7)
> +#define CXL_CMD_EFFECTS_EXTEND			BIT(9)
> +#define CXL_CMD_CONFIG_CHANGE_CONV_RESET	BIT(10)
> +#define CXL_CMD_CONFIG_CHANGE_CXL_RESET		BIT(11)

cxl-test already has CXL_CMD_EFFECT_ definitions. Move those to a shared
location to avoid duplicated definitions with divergent naming.

Re: [PATCH v1 04/19] cxl: Add Get Supported Features command for kernel usage

[Jason Gunthorpe]

On Thu, Jan 23, 2025 at 04:30:17PM -0800, Dan Williams wrote:

> > +struct cxl_feat_entry *
> > +cxl_get_supported_feature_entry(struct cxl_features *features,
> > +				const uuid_t *feat_uuid)
> > +{
> > +	struct cxl_feat_entry *feat_entry;
> > +	struct cxl_features_state *cfs;
> > +	int count;
> > +
> > +	cfs = dev_get_drvdata(&features->dev);
> > +	if (!cfs)
> > +		return ERR_PTR(-EOPNOTSUPP);
> 
> How could this function be called outside of the driver being enabled?

+1

I hate these "defensive" checks. Either your API is always called
under a properly bound driver, or your caller is horribly wrongly
locked and the above does nothing but hide bugs.

defensive "can't happen" assertions should be marked with WARN_ON(),
but I wouldn't use that here - the oops on NULL backtrace is
sufficiently clear. :)

Jason

Re: [PATCH v1 07/19] cxl/mbox: Add GET_FEATURE mailbox command

[Dan Williams]

Dave Jiang wrote:
> From: Shiju Jose <shiju.jose@huawei.com>
> 
> Add support for GET_FEATURE mailbox command.
> 
> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
> The settings of a feature can be retrieved using Get Feature command.
> CXL spec 3.1 section 8.2.9.6.2 describes Get Feature command.
> 
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v1:
> - pass in cxl_mbox instead of cxlds (Dan)
> - Move to the feature driver model. (Dan)
> ---
>  drivers/cxl/core/features.c | 74 +++++++++++++++++++++++++++++++++++++
>  drivers/cxl/features.c      |  6 +--
>  include/cxl/features.h      | 27 ++++++++++++++
>  3 files changed, 102 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
> index 66a4b82910e6..ab9386b53a95 100644
> --- a/drivers/cxl/core/features.c
> +++ b/drivers/cxl/core/features.c
> @@ -4,6 +4,7 @@
>  #include <cxl/mailbox.h>
>  #include "cxl.h"
>  #include "core.h"
> +#include "cxlmem.h"
>  
>  #define CXL_FEATURE_MAX_DEVS 65536
>  static DEFINE_IDA(cxl_features_ida);
> @@ -97,3 +98,76 @@ cxl_get_supported_feature_entry(struct cxl_features *features,
>  	return ERR_PTR(-ENOENT);
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_get_supported_feature_entry, "CXL");
> +
> +bool cxl_feature_enabled(struct cxl_features_state *cfs, u16 opcode)
> +{
> +	struct cxl_mailbox *cxl_mbox = cfs->features->cxl_mbox;
> +	struct cxl_mem_command *cmd;
> +
> +	cmd = cxl_find_feature_command(opcode);
> +	if (!cmd)
> +		return false;
> +
> +	return test_bit(cmd->info.id, cxl_mbox->feature_cmds);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_feature_enabled, "CXL");
> +
> +size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
> +		       enum cxl_get_feat_selection selection,
> +		       void *feat_out, size_t feat_out_size, u16 offset,

Is @feat_out guaranteed to be a kernel pointer, or might it be an __user
pointer?

Shouldn't @feat_uuid be a 'uuid_t *' rather than a 'uuid_t'?

> +		       u16 *return_code)
> +{
> +	size_t data_to_rd_size, size_out;
> +	struct cxl_features_state *cfs;
> +	struct cxl_mbox_get_feat_in pi;
> +	struct cxl_mailbox *cxl_mbox;
> +	struct cxl_mbox_cmd mbox_cmd;
> +	size_t data_rcvd_size = 0;
> +	int rc;
> +
> +	if (return_code)
> +		*return_code = CXL_MBOX_CMD_RC_INPUT;
> +
> +	cfs = dev_get_drvdata(&features->dev);
> +	if (!cfs)
> +		return 0;
> +
> +	if (!cxl_feature_enabled(cfs, CXL_MBOX_OP_GET_FEATURE))
> +		return 0;

Per previous feedback, just make the caller responsible for knowing this
in advance, not checking every call. With that gone this function loses
its dependency on 'struct cxl_features' and 'struct cxl_features_state'
and can take 'struct cxl_mbox' directly.

If for some reason the caller did not know that Get Feature was missing
the device will still fail it anyway, so that boilerplate is not serving
any useful purpose.

Re: [PATCH v1 08/19] cxl/mbox: Add SET_FEATURE mailbox command

[Dan Williams]

Dave Jiang wrote:
> From: Shiju Jose <shiju.jose@huawei.com>
> 
> Add support for SET_FEATURE mailbox command.
> 
> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
> CXL devices supports features with changeable attributes.
> The settings of a feature can be optionally modified using Set Feature
> command.
> CXL spec 3.1 section 8.2.9.6.3 describes Set Feature command.
> 
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v1:
> - CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER enum must be set to 0.
> ---
>  drivers/cxl/core/features.c  | 92 ++++++++++++++++++++++++++++++++++++
>  include/cxl/features.h       | 32 +++++++++++++
>  tools/testing/cxl/test/mem.c |  1 +
>  3 files changed, 125 insertions(+)
> 
> diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
> index ab9386b53a95..932e82b52f90 100644
> --- a/drivers/cxl/core/features.c
> +++ b/drivers/cxl/core/features.c
> @@ -171,3 +171,95 @@ size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
>  	return data_rcvd_size;
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_get_feature, "CXL");
> +
> +/*
> + * FEAT_DATA_MIN_PAYLOAD_SIZE - min extra number of bytes should be
> + * available in the mailbox for storing the actual feature data so that
> + * the feature data transfer would work as expected.
> + */
> +#define FEAT_DATA_MIN_PAYLOAD_SIZE 10
> +int cxl_set_feature(struct cxl_features *features,
> +		    const uuid_t feat_uuid, u8 feat_version,
> +		    void *feat_data, size_t feat_data_size,
> +		    u32 feat_flag, u16 offset, u16 *return_code)
> +{
> +	struct cxl_memdev_set_feat_pi {
> +		struct cxl_mbox_set_feat_hdr hdr;
> +		u8 feat_data[];
> +	}  __packed;
> +	size_t data_in_size, data_sent_size = 0;
> +	struct cxl_features_state *cfs;
> +	struct cxl_mbox_cmd mbox_cmd;
> +	struct cxl_mailbox *cxl_mbox;
> +	size_t hdr_size;
> +	int rc = 0;
> +
> +	if (return_code)
> +		*return_code = CXL_MBOX_CMD_RC_INPUT;
> +
> +	cfs = dev_get_drvdata(&features->dev);
> +	if (!cfs)
> +		return -EOPNOTSUPP;
> +
> +	if (!cxl_feature_enabled(cfs, CXL_MBOX_OP_SET_FEATURE))
> +		return -EOPNOTSUPP;

Similar boilerplate removal and function signature feedback as the last patch.

> +
> +	cxl_mbox = features->cxl_mbox;
> +
> +	struct cxl_memdev_set_feat_pi *pi __free(kfree) =
> +					kmalloc(cxl_mbox->payload_size, GFP_KERNEL);

Unless you know for a fact that @pi will always be fully overwritten, I
would kzalloc() just to be safe.

Other than that this looks ok to me.
[..]

Re: [PATCH v1 09/19] cxl: Setup exclusive CXL features that are reserved for the kernel

[Dan Williams]

Dave Jiang wrote:
> Certain features will be exclusively used by components such as in
> kernel RAS driver. Setup an exclusion list that can be later filtered
> out before exposing to user space.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/features.c | 22 ++++++++++++++++++++++
>  drivers/cxl/features.c      |  6 +++++-
>  include/cxl/features.h      | 34 ++++++++++++++++++++++++++++++++++
>  3 files changed, 61 insertions(+), 1 deletion(-)

LGTM

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

Re: [PATCH v1 10/19] cxl: Add FWCTL support to the CXL features driver

[Dan Williams]

Dave Jiang wrote:
> Add fwctl support code to allow sending of CXL feature commands from
> userspace through as ioctls via FWCTL. Provide initial setup bits.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v1:
> - Add missing dev_set_name() (Jason)
> ---
>  drivers/cxl/Kconfig        |  1 +
>  drivers/cxl/features.c     | 44 ++++++++++++++++++++++++++++++++++++--
>  include/cxl/features.h     |  2 ++
>  include/uapi/fwctl/fwctl.h |  1 +
>  4 files changed, 46 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
> index 9a6ffd81ac0e..5de06c3806ce 100644
> --- a/drivers/cxl/Kconfig
> +++ b/drivers/cxl/Kconfig
> @@ -148,6 +148,7 @@ config CXL_REGION_INVALIDATION_TEST
>  
>  config CXL_FEATURES
>  	tristate "CXL: Features support"
> +	depends on FWCTL
>  	default CXL_BUS
>  	help
>  	  Enable CXL features support that are tied to a CXL mailbox.
> diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
> index 1c4bcd7465cb..5a2b771586d3 100644
> --- a/drivers/cxl/features.c
> +++ b/drivers/cxl/features.c
> @@ -9,6 +9,37 @@
>  #include "cxl.h"
>  #include "cxlmem.h"
>  
> +static int cxlctl_open_uctx(struct fwctl_uctx *uctx)
> +{
> +	return 0;
> +}
> +
> +static void cxlctl_close_uctx(struct fwctl_uctx *uctx)
> +{
> +}
> +
> +static void *cxlctl_info(struct fwctl_uctx *uctx, size_t *length)
> +{
> +	/* Place holder */
> +	return ERR_PTR(-EOPNOTSUPP);
> +}
> +
> +static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
> +			   void *rpc_in, size_t in_len, size_t *out_len)
> +{
> +	/* Place holder */
> +	return ERR_PTR(-EOPNOTSUPP);
> +}
> +
> +static const struct fwctl_ops cxlctl_ops = {
> +	.device_type = FWCTL_DEVICE_TYPE_CXL,
> +	.uctx_size = sizeof(struct cxl_features),
> +	.open_uctx = cxlctl_open_uctx,
> +	.close_uctx = cxlctl_close_uctx,
> +	.info = cxlctl_info,
> +	.fw_rpc = cxlctl_fw_rpc,
> +};
> +
>  static void cxl_free_feature_entries(void *entries)
>  {
>  	kvfree(entries);
> @@ -146,13 +177,16 @@ static int cxl_get_supported_features(struct cxl_features_state *cfs)
>  					cxl_free_feature_entries, cfs->entries);
>  }
>  
> +DEFINE_FREE(cfs, struct cxl_features_state *, if (_T) fwctl_put(&_T->fwctl))
> +
>  static int cxl_features_probe(struct device *dev)
>  {
>  	struct cxl_features *features = to_cxl_features(dev);
>  	int rc;
>  
> -	struct cxl_features_state *cfs __free(kfree) =
> -		kzalloc(sizeof(*cfs), GFP_KERNEL);
> +	struct cxl_features_state *cfs __free(cfs) =
> +		fwctl_alloc_device(dev, &cxlctl_ops,
> +				   struct cxl_features_state, fwctl);
>  	if (!cfs)
>  		return -ENOMEM;
>  
> @@ -161,6 +195,10 @@ static int cxl_features_probe(struct device *dev)
>  	if (rc)
>  		return rc;
>  
> +	rc = fwctl_register(&cfs->fwctl);
> +	if (rc)
> +		return rc;

Repeating our offline discussion...

After looking at what cxl_features_driver does and the fact that the
EDAC scrub support is enabled by registration calls from cxl_mem and
cxl_region. Let's just have cxl_mem or cxl_pci register fwctl relative
to a cxl_memdev. So, similar to how a cxl_memdev hosts an ioctl interface
and a fw_upload interface, add a fwctl interface with no new
device+driver objects.

When / if other components like CXL switches want to support fwctl that
can also be had as just a fwctl registration relative to a cxl_port
device which would have been awkward to handle with a sub-device+driver.

> diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
> index f9b27fb5c161..2c3817592308 100644
> --- a/include/uapi/fwctl/fwctl.h
> +++ b/include/uapi/fwctl/fwctl.h
> @@ -43,6 +43,7 @@ enum {
>  enum fwctl_device_type {
>  	FWCTL_DEVICE_TYPE_ERROR = 0,
>  	FWCTL_DEVICE_TYPE_MLX5 = 1,
> +	FWCTL_DEVICE_TYPE_CXL,

Jonathan and Jason already handled the comment here.

Re: [PATCH v1 11/19] cxl: Add support for get driver information

[Dan Williams]

Dave Jiang wrote:
> Add definition for fwctl_ops->info() to return driver information. The
> function will return a mask of the feature mailbox commands supported
> by the fwctl char device.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v1:
> - Add missed setting of *length for ->info().
> - Use BIT() instead of enum directly.
> ---
>  drivers/cxl/features.c   | 30 ++++++++++++++++++++++++++++--
>  include/cxl/features.h   |  7 -------
>  include/cxl/mailbox.h    |  1 +
>  include/uapi/fwctl/cxl.h | 31 +++++++++++++++++++++++++++++++
>  4 files changed, 60 insertions(+), 9 deletions(-)
>  create mode 100644 include/uapi/fwctl/cxl.h
[..]
> diff --git a/include/uapi/fwctl/cxl.h b/include/uapi/fwctl/cxl.h
> new file mode 100644
> index 000000000000..79b822dbfafd
> --- /dev/null
> +++ b/include/uapi/fwctl/cxl.h
> @@ -0,0 +1,31 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Copyright (c) 2024, Intel Corporation
> + *
> + * These are definitions for the mailbox command interface of CXL subsystem.
> + */
> +#ifndef _UAPI_FWCTL_CXL_H_
> +#define _UAPI_FWCTL_CXL_H_
> +
> +#include <linux/types.h>
> +
> +enum feature_cmds {
> +	CXL_FEATURE_ID_GET_SUPPORTED_FEATURES = 0,
> +	CXL_FEATURE_ID_GET_FEATURE,
> +	CXL_FEATURE_ID_SET_FEATURE,
> +	CXL_FEATURE_ID_MAX,
> +};
> +
> +/**
> + * struct fwctl_info_cxl - ioctl(FWCTL_INFO) out_device_data
> + * @cmd_mask: Mask indicate which commands are supported based on 'enum feature_cmds'
> + *
> + * Return basic information about the FW interface available.
> + *
> + * nr_commands is number of hardware commands the driver supports. Use
> + * FWCTL_CMD_HW_INFO ioctl to request additional information.
> + */
> +struct fwctl_info_cxl {
> +	__u32 cmd_mask;
> +};
> +#endif

I do not understand the value of this. If the fwctl device shows up at
all for CXL you already know the device must support Get Supported
Features, and Get Feature. Set Feature could be optionally be supported
by the device but Get Supported Features will already tell userspace if
a feature is writable via Set Feature Size in its output payload.

So cmd_mask is vestigial information.

I would just do this for now.

struct fwctl_info_cxl {
	__u32 reserved;
};

Make it zero so that we have place holder to provide future
meta-information about the interface, like an empty flags argument for a
new syscall.

Re: [PATCH v1 12/19] cxl: Move cxl_mem.h under uapi to cxl exclusive directory

[Dan Williams]

Dave Jiang wrote:
> In preparation of more cxl headers, create a cxl directory under
> uapi.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

This theoretically build breaks userspace that was looking at the old
directory path. What's wrong with adding to cxl_mem.h?

Re: [PATCH v1 13/19] cxl: Move cxl feature command structs to user header

[Dan Williams]

Dave Jiang wrote:
> In perparation for cxl fwctl enabling, move data structures related to

s/perparation/preparation/

> cxl feature commands to a user header file.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v1:
> - Fixup user headers from 0day report
> ---
>  include/cxl/features.h      | 101 +------------------------------
>  include/uapi/cxl/features.h | 117 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 118 insertions(+), 100 deletions(-)
>  create mode 100644 include/uapi/cxl/features.h

I think it's ok to just extend the cxl_mem ABI. This is interface is
going to hang off a cxl_mem device.

Otherwise looks ok to me:

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Dan Williams]

Dave Jiang wrote:
> fwctl provides a fwctl_ops->fw_rpc() callback in order to issue ioctls
> to a device. The cxl fwctl driver will start by supporting the CXL
> feature commands: Get Supported Features, Get Feature, and Set Feature.
> 
> The fw_rpc() callback provides 'enum fwctl_rpc_scope' parameter where
> it indicates the security scope of the call. The Get Supported Features
> and Get Feature calls can be executed with the scope of
> FWCTL_RPC_CONFIGRATION. The Set Feature call is gated by the effects
> of the feature reported by Get Supported Features call for the specific
> feature.
> 
> Only "get supported features" is supported in this patch. Additional
> commands will be added in follow on patches. "Get supported features"
> will filter the features that are exclusive to the kernel and only
> report out features that are not kernel only.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/mbox.c     |  12 +++
>  drivers/cxl/features.c      | 198 +++++++++++++++++++++++++++++++++++-
>  include/cxl/features.h      |   1 +
>  include/uapi/cxl/features.h |  13 +++
>  include/uapi/fwctl/cxl.h    |  29 ++++++
>  5 files changed, 250 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 0b4946205910..f818e452dbca 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -246,6 +246,18 @@ struct cxl_mem_command *cxl_find_feature_command(u16 opcode)
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_find_feature_command, "CXL");
>  
> +struct cxl_mem_command *cxl_find_feature_command_by_id(u32 id)
> +{
> +	struct cxl_mem_command *c;
> +
> +	cxl_for_each_feature_cmd(c)
> +		if (c->info.id == id)
> +			return c;
> +
> +	return NULL;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_find_feature_command_by_id, "CXL");

I assume this will go with the deletion of a cxl_mem_command array for
Features. Just do a simple hard-coded switch statement.

> +
>  static const char *cxl_mem_opcode_to_name(u16 opcode)
>  {
>  	struct cxl_mem_command *c;
> diff --git a/drivers/cxl/features.c b/drivers/cxl/features.c
> index cc72e73ae8d6..e65fc8479d21 100644
> --- a/drivers/cxl/features.c
> +++ b/drivers/cxl/features.c
> @@ -50,11 +50,203 @@ static void *cxlctl_info(struct fwctl_uctx *uctx, size_t *length)
>  	return info;
>  }
>  
> +static struct cxl_feat_entry *
> +get_support_feature_info(struct cxl_features_state *cfs,
> +			 const struct fwctl_rpc_cxl *rpc_in)
> +{
> +	struct cxl_feat_entry *feat;
> +	uuid_t uuid;
> +
> +	if (rpc_in->op_size < sizeof(uuid))
> +		return ERR_PTR(-EINVAL);
> +
> +	if (copy_from_user(&uuid, u64_to_user_ptr(rpc_in->in_payload),
> +			   sizeof(uuid)))
> +		return ERR_PTR(-EFAULT);
> +
> +	for (int i = 0; i < cfs->num_features; i++) {
> +		feat = &cfs->entries[i];
> +		if (uuid_equal(&uuid, &feat->uuid))
> +			return feat;
> +	}
> +
> +	return ERR_PTR(-ENOENT);

I expect user space to be surprised that it is getting "No such file or
directory" after successfully opening the fwctl fd. Just use EINVAL for
attempts to access Features that were not captured via the init-time Get
Supported Features.

> +}
> +
> +static void *cxlctl_get_supported_features(struct cxl_features_state *cfs,
> +					   const struct fwctl_rpc_cxl *rpc_in,
> +					   size_t *out_len)
> +{
> +	struct cxl_mbox_get_sup_feats_out *feat_out;
> +	struct cxl_mbox_get_sup_feats_in feat_in;
> +	struct cxl_feat_entry *saved, *pos;
> +	int requested, copied;
> +	size_t out_size;
> +	u32 count;
> +	u16 start;
> +
> +	if (rpc_in->op_size != sizeof(feat_in))
> +		return ERR_PTR(-EINVAL);
> +
> +	if (copy_from_user(&feat_in, u64_to_user_ptr(rpc_in->in_payload),
> +			   rpc_in->op_size))
> +		return ERR_PTR(-EFAULT);
> +
> +	count = le32_to_cpu(feat_in.count);
> +	start = le16_to_cpu(feat_in.start_idx);
> +	requested = count / sizeof(*pos);
> +
> +	/*
> +	 * Make sure that the total requested number of entries is not greater
> +	 * than the total number of supported features allowed for userspace.
> +	 */
> +	if (start >= cfs->num_user_features)
> +		return ERR_PTR(-EINVAL);
> +
> +	requested = min_t(int, requested, cfs->num_user_features - start);
> +
> +	out_size = sizeof(struct fwctl_rpc_cxl_out) + sizeof(*feat_out) +
> +		   requested * sizeof(*pos);
> +
> +	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
> +		kvzalloc(out_size, GFP_KERNEL);
> +	if (!rpc_out)
> +		return ERR_PTR(-ENOMEM);
> +
> +	rpc_out->size = sizeof(*feat_out) + requested * sizeof(*pos);
> +	feat_out = (struct cxl_mbox_get_sup_feats_out *)rpc_out->payload;
> +	if (requested == 0) {
> +		feat_out->num_entries = cpu_to_le16(requested);
> +		feat_out->supported_feats = cpu_to_le16(cfs->num_user_features);
> +		rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
> +		*out_len = out_size;
> +		return no_free_ptr(rpc_out);
> +	}
> +
> +	pos = &feat_out->ents[0];
> +	saved = &cfs->entries[0];
> +
> +	copied = 0;
> +	for (int i = 0; i < cfs->num_features; i++, saved++) {
> +		if (is_cxl_feature_exclusive(saved))
> +			continue;

I think it's fine to let userspace see that exclusive features are
present, just need to return EBUSY if userspace actually tries to use
them.

> +
> +		memcpy(pos, saved, sizeof(*saved));
> +		copied++;
> +		if (copied == requested)
> +			break;
> +		pos++;
> +	}
> +
> +	feat_out->num_entries = cpu_to_le16(requested);
> +	feat_out->supported_feats = cpu_to_le16(cfs->num_user_features);
> +	rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
> +	*out_len = out_size;
> +
> +	return no_free_ptr(rpc_out);
> +}
> +
> +static bool cxlctl_validate_set_features(struct cxl_features_state *cfs,
> +					 const struct fwctl_rpc_cxl *rpc_in,
> +					 enum fwctl_rpc_scope scope)
> +{
> +	struct cxl_feat_entry *feat;
> +	u16 effects, mask;
> +	u32 flags;
> +
> +	feat = get_support_feature_info(cfs, rpc_in);
> +	if (IS_ERR(feat))
> +		return false;
> +
> +	/* Ensure that the attribute is changeable */
> +	flags = le32_to_cpu(feat->flags);
> +	if (!(flags & CXL_FEATURE_F_CHANGEABLE))
> +		return false;
> +
> +	effects = le16_to_cpu(feat->effects);
> +	/* Currently no user background command support */
> +	if (effects & CXL_CMD_BACKGROUND)
> +		return false;
> +
> +	mask = CXL_CMD_CONFIG_CHANGE_IMMEDIATE |
> +	       CXL_CMD_DATA_CHANGE_IMMEDIATE |
> +	       CXL_CMD_POLICY_CHANGE_IMMEDIATE |
> +	       CXL_CMD_LOG_CHANGE_IMMEDIATE;
> +	if (effects & mask && scope >= FWCTL_RPC_DEBUG_WRITE_FULL)
> +		return true;
> +
> +	/* These effects supported for all scope */
> +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
> +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
> +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
> +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
> +	    scope >= FWCTL_RPC_DEBUG_WRITE)
> +		return true;

Looks good for the known bits, but this needs to return false for the
currently reserved bits because the driver can not assume a security
model for future effects. If a future spec adds
FWCTL_RPC_DEBUG_WRITE-safe effects, a new kernel is needed to allow
those Feature commands through.

Sidenote: I wonder why the spec wasted one of its bits on an extend bit,
but here we are. The 'extend' concept is typically something like
"bit15: go look at this other field in this payload as this 16-bit field
was exhausted", not "bit9: the bits above this originally defined 16 bit
field now has more bits", oh well.

> +
> +	return false;
> +}
> +
> +static struct cxl_mem_command *
> +cxlctl_get_valid_hw_command(struct cxl_features_state *cfs,
> +			    const struct fwctl_rpc_cxl *rpc_in,
> +			    enum fwctl_rpc_scope scope)
> +{
> +	struct cxl_mem_command *cmd;
> +
> +	if (!cfs->num_features)
> +		return ERR_PTR(-EOPNOTSUPP);
> +
> +	cmd = cxl_find_feature_command_by_id(rpc_in->command_id);
> +	if (!cmd)
> +		return ERR_PTR(-ENOENT);
> +
> +	if (!cxl_feature_enabled(cfs, cmd->opcode))
> +		return ERR_PTR(-EPERM);

Why "Permission denied", because the base command is missing? The
cxl_fwctl interface would not even be availble to userspace if the above
was true.

No need to worry about optional Set Feature becuase that should be
consistent with the Set Feature Size data in Get Supported Features.

> +

Per above, need to add EBUSY for exclusive features in this path.

> +	switch (cmd->opcode) {
> +	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
> +	case CXL_MBOX_OP_GET_FEATURE:

I would still check if a Get Feature has a configuration change
side-effect. You never know if someone purposefully or accidentally
introduces "read causes side effects" commands.

[..]

Re: [PATCH v1 15/19] cxl: Add support to handle user feature commands for get feature

[Dan Williams]

Dave Jiang wrote:
> Add helper function to parse the user data from fwctl RPC ioctl and
> send the parsed input parameters to cxl_get_feature() call.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---

Looks good to me:

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

...modulo reworking @cfs related entanglements.

Re: [PATCH v1 16/19] cxl: Add support to handle user feature commands for set feature

[Dan Williams]

Dave Jiang wrote:
> Add helper function to parse the user data from fwctl RPC ioctl and
> send the parsed input parameters to cxl_set_feature() call.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---

Looks good.

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

Re: [PATCH v1 17/19] cxl/test: Add Get Feature support to cxl_test

[Dan Williams]

Dave Jiang wrote:
> Add emulation of Get Feature command to cxl_test. The feature for
> device patrol scrub is returned by the emulation code. This is
> the only feature currently supported by cxl_test. It returns
> the information for the device patrol scrub feature.
> 
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

Looks good for what it is. I wouldn't mind it being a bit more cheeky
and testing the "Get Feature has a data change effect" and "Get Feature
reports a reserved effect" corner cases. Those negative tests can be
added later.

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

Re: [PATCH v1 18/19] cxl/test: Add Set Feature support to cxl_test

[Dan Williams]

Dave Jiang wrote:
> Add emulation to support Set Feature mailbox command to cxl_test.
> The only feature supported is the device patrol scrub feature. The
> set feature allows activation of patrol scrub for the cxl_test
> emulated device. The command does not support partial data transfer
> even though the spec allows it. This restriction is to reduce complexity
> of the emulation given the patrol scrub feature is very minimal.
> 
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

For cxl-test it does not need to be a real spec feature, it can just be
a scratch pad like Feature with a made up UUID and effects.

Again, future work, for this one:

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

Re: [PATCH v1 19/19] fwctl/cxl: Add documentation to FWCTL CXL

[Dan Williams]

Dave Jiang wrote:
> Add policy and operational documentation for FWCTL CXL.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  .../userspace-api/fwctl/fwctl-cxl.rst         | 81 +++++++++++++++++++
>  Documentation/userspace-api/fwctl/index.rst   |  1 +
>  2 files changed, 82 insertions(+)
>  create mode 100644 Documentation/userspace-api/fwctl/fwctl-cxl.rst
> 
> diff --git a/Documentation/userspace-api/fwctl/fwctl-cxl.rst b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
> new file mode 100644
> index 000000000000..f774bf2ecb5f
> --- /dev/null
> +++ b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
> @@ -0,0 +1,81 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +================
> +fwctl cxl driver
> +================
> +
> +:Author: Dave Jiang
> +
> +Overview
> +========
> +
> +The CXL spec defines a set of commands that can be issued to the mailbox of a
> +CXL device or switch. It also left room for vendor specific commands to be
> +issued to the mailbox as well. fwctl provides a path to issue a set of allowed
> +mailbox commands from user space to the device moderated by the kernel driver.
> +
> +While there are a large set of mailbox commands, only the feature related
> +commands will be allowed to be issued through fwctl initially. No background
> +commands will be supported at this time.

Just describe what the CXL FWTCTL for Features Interface is, not what
it would/could be. Someone reading this doc on docs.kernel.org without
access to git history is going to wonder what "initially", and "at this
time" means. That's better changelog fodder, not interface documentation.

> +
> +CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
> +CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
> +CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)
> +
> +The "Get Supported Features" return data may be filtered by the kernel driver to
> +drop any features that are forbidden by the kernel or being exclusively used by
> +the kernel. The "Get Supported Features" command and the "Get Features" falls
> +under the fwctl policy of FWCTL_RPC_CONFIGURATION.
> +
> +For "Set Feature" command, the access policy currently is broken down into two
> +categories depending on the set feature effects reported by the device. If the

Capitalize Set Feature throughout when it is reference a specification
term.

> +set feature will cause immediate change to the device, the fwctl access policy
> +must be FWCTL_RPC_DEBUG_WRITE_FULL. The effects for this level are
> +"immediate config change", "immediate data change", "immediate policy change",
> +or "immediate log change" for the set effects mask. If the effects are "config
> +change with cold reset" or "config change with conventional reset", then the
> +fwctl access policy must be FWCTL_RPC_DEBUG_WRITE or higher.
> +
> +fwctl cxl User API
> +==================
> +
> +.. kernel-doc:: include/uapi/fwctl/cxl.h
> +.. kernel-doc:: include/uapi/cxl/features.h
> +
> +1. Driver info query
> +--------------------
> +
> +First step for the app is to issue the ioctl(FWCTL_CMD_INFO) in order to retrieve
> +the information from the driver. A ``struct fwctl_info`` needs to be filled out
> +with the ``fwctl_info.out_device_type`` set to ``FWCTL_DEVICE_TYPE_CXL``. The
> +return data should be ``struct fwctl_info_cxl`` that contains a command mask
> +that indicates which of the commands are supported by the driver.

Per the code review, succesful invocation of the info command implies
the Features capability is operational and returns an all zeros
payload to indicate first generation interface support.

> +
> +2. Send hardware command
> +------------------------
> +
> +Next step is to send the 'Get Supported Features' command to the driver from
> +user space via ioctl(FWCTL_RPC). A ``struct fwctl_rpc_cxl`` should be pointed to
> +by ``fwctl_rpc.in``. ``struct fwctl_rpc_cxl.in_payload`` should point to
> +the hardware input structure that is defined by the CXL spec. ``fwctl_rpc.out``
> +points to the buffer that contains a ``struct fwctl_rpc_cxl_out`` that includes
> +the hardware output data inlined as ``fwctl_rpc_cxl_out.payload``. This command
> +can be called twice. First time to retrieve the number of features supported.
> +A second time to retrieve the specific feature details.

Please convert this from passive to active voice. I.e. s/should be
pointed to by/points to/

Some sample code to enumerat and issue a Get/Set Feature for a made up
Feature UUID would be welcome here.

> +
> +After getting the specific feature details, a get/set feature command can be
> +appropriately programmed and sent. For a "set feature" command, the retrieved
> +feature info contains an effects field that details the resulting
> +"set feature" command will trigger. That should inform the user whether
> +the system is configured to allowed the "set feature" command or not.
> +
> +It is recommended to take a look at CXL CLI test directory for a detailed
> +user code example on how to exercise this path.

Should be small enough to copy here directly. Otherwise provide a URL
directly to github or similar for someone reading this on
docs.kernel.org.

> +
> +fwctl cxl Kernel API
> +====================
> +
> +.. kernel-doc:: drivers/cxl/core/features.c
> +.. kernel-doc:: drivers/cxl/features.c
> +   :export:
> +.. kernel-doc:: include/cxl/features.h
> diff --git a/Documentation/userspace-api/fwctl/index.rst b/Documentation/userspace-api/fwctl/index.rst
> index 06959fbf1547..d9d40a468a31 100644
> --- a/Documentation/userspace-api/fwctl/index.rst
> +++ b/Documentation/userspace-api/fwctl/index.rst
> @@ -10,3 +10,4 @@ to securely construct and execute RPCs inside device firmware.
>     :maxdepth: 1
>  
>     fwctl
> +   fwctl-cxl
> -- 
> 2.47.1
> 
> 

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Jonathan Cameron]

> > +}
> > +
> > +static void *cxlctl_get_supported_features(struct cxl_features_state *cfs,
> > +					   const struct fwctl_rpc_cxl *rpc_in,
> > +					   size_t *out_len)
> > +{
> > +	struct cxl_mbox_get_sup_feats_out *feat_out;
> > +	struct cxl_mbox_get_sup_feats_in feat_in;
> > +	struct cxl_feat_entry *saved, *pos;
> > +	int requested, copied;
> > +	size_t out_size;
> > +	u32 count;
> > +	u16 start;
> > +
> > +	if (rpc_in->op_size != sizeof(feat_in))
> > +		return ERR_PTR(-EINVAL);
> > +
> > +	if (copy_from_user(&feat_in, u64_to_user_ptr(rpc_in->in_payload),
> > +			   rpc_in->op_size))
> > +		return ERR_PTR(-EFAULT);
> > +
> > +	count = le32_to_cpu(feat_in.count);
> > +	start = le16_to_cpu(feat_in.start_idx);
> > +	requested = count / sizeof(*pos);
> > +
> > +	/*
> > +	 * Make sure that the total requested number of entries is not greater
> > +	 * than the total number of supported features allowed for userspace.
> > +	 */
> > +	if (start >= cfs->num_user_features)
> > +		return ERR_PTR(-EINVAL);
> > +
> > +	requested = min_t(int, requested, cfs->num_user_features - start);
> > +
> > +	out_size = sizeof(struct fwctl_rpc_cxl_out) + sizeof(*feat_out) +
> > +		   requested * sizeof(*pos);
> > +
> > +	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
> > +		kvzalloc(out_size, GFP_KERNEL);
> > +	if (!rpc_out)
> > +		return ERR_PTR(-ENOMEM);
> > +
> > +	rpc_out->size = sizeof(*feat_out) + requested * sizeof(*pos);
> > +	feat_out = (struct cxl_mbox_get_sup_feats_out *)rpc_out->payload;
> > +	if (requested == 0) {
> > +		feat_out->num_entries = cpu_to_le16(requested);
> > +		feat_out->supported_feats = cpu_to_le16(cfs->num_user_features);
> > +		rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
> > +		*out_len = out_size;
> > +		return no_free_ptr(rpc_out);
> > +	}
> > +
> > +	pos = &feat_out->ents[0];
> > +	saved = &cfs->entries[0];
> > +
> > +	copied = 0;
> > +	for (int i = 0; i < cfs->num_features; i++, saved++) {
> > +		if (is_cxl_feature_exclusive(saved))
> > +			continue;  
> 
> I think it's fine to let userspace see that exclusive features are
> present, just need to return EBUSY if userspace actually tries to use
> them.

To me, a poke it and see interface is really ugly.

In many cases we could let "get" through even if the we are using the interface
via some other kernel path and have it as exclusive.
(I don't know how useful that is, but maybe it makes sense).

If we ever do that, the only way to discover if an interface is available
will be to try the set interface.  Depending on design of feature
that might have side effects - hopefully get never does!

Alternatives:
1. Flag. Maybe add something that makes it discoverable if a feature is
   in exclusive mode or not.

2. Query type interface.  So a way to actually ask if a given feature is
   usable.

3. What we have here.  To me the simplest solution is hide what we can't
   be used.
 


> > +	/* These effects supported for all scope */
> > +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
> > +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
> > +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
> > +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
> > +	    scope >= FWCTL_RPC_DEBUG_WRITE)
> > +		return true;  
> 
> Looks good for the known bits, but this needs to return false for the
> currently reserved bits because the driver can not assume a security
> model for future effects. If a future spec adds
> FWCTL_RPC_DEBUG_WRITE-safe effects, a new kernel is needed to allow
> those Feature commands through.
> 
> Sidenote: I wonder why the spec wasted one of its bits on an extend bit,
> but here we are. The 'extend' concept is typically something like
> "bit15: go look at this other field in this payload as this 16-bit field
> was exhausted", not "bit9: the bits above this originally defined 16 bit
> field now has more bits", oh well.

It's odd but corner case of going from 'unknown' state for the remaining
pair of bits to 0 means this and 1 means this. Naming though doesn't match
the spec that calls it CEL[11:10] valid.  Would be good to name
it closer to that as we may well have something in bits 12 and 15 in future
and it doesn't refer to them.

Jonathan

Re: [PATCH v1 04/19] cxl: Add Get Supported Features command for kernel usage

[Jonathan Cameron]

On Thu, 23 Jan 2025 16:30:17 -0800
Dan Williams <dan.j.williams@intel.com> wrote:

> Dave Jiang wrote:
> > CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
> > The command retrieve the list of supported device-specific features
> > (identified by UUID) and general information about each Feature.
> > 
> > The driver will retrieve the feature entries in order to make checks and
> > provide information for the Get Feature and Set Feature command. One of
> > the main piece of information retrieved are the effects a Set Feature
> > command would have for a particular feature.
> > 
> > Co-developed-by: Shiju Jose <shiju.jose@huawei.com>
> > Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> > Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> > ---
> > v1:
> > - Change input param from cxlds to cxl_mbox
> > - Move mbox_out declaration inline. (Jonathan)
> > - Fix __counted_by() input. (Jonathan)
> > - Return count for cxl_get_supported_features_count(). (Dan)
> > - Remove goto from cxl_get_supported_features(). (Dan)
> > - Return cxl_get_supported_feature_entry() directly. (Dan)
> > - Drop user path enumeration. (Dan)
> > - Move to the feature driver model. (Dan)
> > - Add support for 0 feature data requested.
> > - Add missing increment of feature data ptr during copy.
> > ---
> >  drivers/cxl/core/features.c |  28 +++++++
> >  drivers/cxl/core/mbox.c     |   3 +-
> >  drivers/cxl/cxl.h           |   2 +
> >  drivers/cxl/features.c      | 146 +++++++++++++++++++++++++++++++++++-
> >  include/cxl/features.h      |  32 ++++++++
> >  5 files changed, 209 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
> > index eb6eb191a32e..66a4b82910e6 100644
> > --- a/drivers/cxl/core/features.c
> > +++ b/drivers/cxl/core/features.c
> > @@ -1,6 +1,7 @@
> >  // SPDX-License-Identifier: GPL-2.0-only
> >  /* Copyright(c) 2024-2025 Intel Corporation. All rights reserved. */
> >  #include <linux/device.h>
> > +#include <cxl/mailbox.h>
> >  #include "cxl.h"
> >  #include "core.h"
> >  
> > @@ -69,3 +70,30 @@ struct cxl_features *cxl_features_alloc(struct cxl_mailbox *cxl_mbox,
> >  	return ERR_PTR(rc);
> >  }
> >  EXPORT_SYMBOL_NS_GPL(cxl_features_alloc, "CXL");
> > +
> > +struct cxl_feat_entry *
> > +cxl_get_supported_feature_entry(struct cxl_features *features,
> > +				const uuid_t *feat_uuid)
> > +{
> > +	struct cxl_feat_entry *feat_entry;
> > +	struct cxl_features_state *cfs;
> > +	int count;
> > +
> > +	cfs = dev_get_drvdata(&features->dev);
> > +	if (!cfs)
> > +		return ERR_PTR(-EOPNOTSUPP);  
> 
> How could this function be called outside of the driver being enabled?
> 
> If the driver might not be enabled when this is called, what stops it
> from being disabled immediately after this check?
> 
> Went looking for the caller of this function... none in this patch set.
> 
> If this is for the EDAC use case, that enabling had better be triggered
> from a known context where these questions have a satisfactory answer.
> As it stands I think you can just delete it, right?
> 
> This has me thinking that the EDAC integration should just be a feature
> of the cxl_features_driver.

I'm not 100% sure I see what you mean here, but if you mean pushing
the registration code into cxl_features.ko...

I replied to this in the other thread, but just to add a bit here.
I think this creates a worse spiders web than we have today. Not
all EDAC features we might add (e.g. device self test) have anything to do
with get/set features.  I'm not sure we'll support that one in EDAC soon
though it might make sense if there is generality with non CXL systems
(I've been meaning to look into that.)

To me, the CXL features support is providing a service to some of the EDAC
features, but they use other services from CXL drivers and potentially
don't use CXL features at all.  For now we have the repair drivers
that only use the features support for a tiny bit of what they do
(tweaking device initiated aspects and detailed record generation
 - if we even support those yet).

At some point we may have core CXL functionality depending on
get/set features as well as there are things we definitely don't
want in either EDAC or fwctl like metabits storage controls.
So I'd be careful adding any thing to cxl_features beyond stuff
that is definitely clustered with fwctl bit or helpers.

Soon I'd also expect us to add 'features' to fwctl interface
that aren't using get/set features. I think you suggested that we
could do this by making up some guids for them and pushing though
this same interface.

Jonathan

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Dan Williams]

Jonathan Cameron wrote:
> 
> > > +}
> > > +
> > > +static void *cxlctl_get_supported_features(struct cxl_features_state *cfs,
> > > +					   const struct fwctl_rpc_cxl *rpc_in,
> > > +					   size_t *out_len)
> > > +{
> > > +	struct cxl_mbox_get_sup_feats_out *feat_out;
> > > +	struct cxl_mbox_get_sup_feats_in feat_in;
> > > +	struct cxl_feat_entry *saved, *pos;
> > > +	int requested, copied;
> > > +	size_t out_size;
> > > +	u32 count;
> > > +	u16 start;
> > > +
> > > +	if (rpc_in->op_size != sizeof(feat_in))
> > > +		return ERR_PTR(-EINVAL);
> > > +
> > > +	if (copy_from_user(&feat_in, u64_to_user_ptr(rpc_in->in_payload),
> > > +			   rpc_in->op_size))
> > > +		return ERR_PTR(-EFAULT);
> > > +
> > > +	count = le32_to_cpu(feat_in.count);
> > > +	start = le16_to_cpu(feat_in.start_idx);
> > > +	requested = count / sizeof(*pos);
> > > +
> > > +	/*
> > > +	 * Make sure that the total requested number of entries is not greater
> > > +	 * than the total number of supported features allowed for userspace.
> > > +	 */
> > > +	if (start >= cfs->num_user_features)
> > > +		return ERR_PTR(-EINVAL);
> > > +
> > > +	requested = min_t(int, requested, cfs->num_user_features - start);
> > > +
> > > +	out_size = sizeof(struct fwctl_rpc_cxl_out) + sizeof(*feat_out) +
> > > +		   requested * sizeof(*pos);
> > > +
> > > +	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
> > > +		kvzalloc(out_size, GFP_KERNEL);
> > > +	if (!rpc_out)
> > > +		return ERR_PTR(-ENOMEM);
> > > +
> > > +	rpc_out->size = sizeof(*feat_out) + requested * sizeof(*pos);
> > > +	feat_out = (struct cxl_mbox_get_sup_feats_out *)rpc_out->payload;
> > > +	if (requested == 0) {
> > > +		feat_out->num_entries = cpu_to_le16(requested);
> > > +		feat_out->supported_feats = cpu_to_le16(cfs->num_user_features);
> > > +		rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
> > > +		*out_len = out_size;
> > > +		return no_free_ptr(rpc_out);
> > > +	}
> > > +
> > > +	pos = &feat_out->ents[0];
> > > +	saved = &cfs->entries[0];
> > > +
> > > +	copied = 0;
> > > +	for (int i = 0; i < cfs->num_features; i++, saved++) {
> > > +		if (is_cxl_feature_exclusive(saved))
> > > +			continue;  
> > 
> > I think it's fine to let userspace see that exclusive features are
> > present, just need to return EBUSY if userspace actually tries to use
> > them.
> 
> To me, a poke it and see interface is really ugly.

That smells more like a matter of documentation. "Doctor it hurts when I
try to use the documented kernel-exclusive commands?"

> In many cases we could let "get" through even if the we are using the interface
> via some other kernel path and have it as exclusive.
> (I don't know how useful that is, but maybe it makes sense).
> 
> If we ever do that, the only way to discover if an interface is available
> will be to try the set interface.  Depending on design of feature
> that might have side effects - hopefully get never does!

I would not put it past some future device to make that mistake.

> 
> Alternatives:
> 1. Flag. Maybe add something that makes it discoverable if a feature is
>    in exclusive mode or not.

I notice that all existing defined Features set a non-zero "Get Feature
Size" in their Supported Feature Entry. I would not say "no" to just
zero-ing out Get Feature Size as a hint that "you might get EBUSY due to
kernel exclusivity with this command", but that still feels like
overkill compared to documentation.

> 2. Query type interface.  So a way to actually ask if a given feature is
>    usable.

Not sure we really need a programmatic way to read the documentation.

The CXL_MEM_COMMAND_FLAG_EXCLUSIVE flag is for cases where the
exclusivity is transient. For these features the exclusivity is
permanent, and I hope we never need to cross that
transient-exclusivity-bridge for Features.

> 3. What we have here.  To me the simplest solution is hide what we can't
>    be used.

It is inconsistent that we do not do this for the other kernel exclusive
commands in userspace retreived Command Effects Log. The ABI here is raw
Get Supported Features payload.

> > > +	/* These effects supported for all scope */
> > > +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
> > > +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
> > > +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
> > > +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
> > > +	    scope >= FWCTL_RPC_DEBUG_WRITE)
> > > +		return true;  
> > 
> > Looks good for the known bits, but this needs to return false for the
> > currently reserved bits because the driver can not assume a security
> > model for future effects. If a future spec adds
> > FWCTL_RPC_DEBUG_WRITE-safe effects, a new kernel is needed to allow
> > those Feature commands through.
> > 
> > Sidenote: I wonder why the spec wasted one of its bits on an extend bit,
> > but here we are. The 'extend' concept is typically something like
> > "bit15: go look at this other field in this payload as this 16-bit field
> > was exhausted", not "bit9: the bits above this originally defined 16 bit
> > field now has more bits", oh well.
> 
> It's odd but corner case of going from 'unknown' state for the remaining
> pair of bits to 0 means this and 1 means this.

I don't understand. 0 means no effect to worry about whether it is
defined or not.

> Naming though doesn't match the spec that calls it CEL[11:10] valid.
> Would be good to name it closer to that as we may well have something
> in bits 12 and 15 in future and it doesn't refer to them.

Hopefully we can head off another "valid2" mistake, and I don't think
Linux needs to define anything for this bit. That bit's definition is:

   "Bit[9]: 1 is recommended, 0 is permitted (CEL[11:10] Valid)"

...which translates to "useless". If 11 or 10 are set, I don't care what
value 9 has. If 12:15 are set, I don't care if there is a future valid2
bit gating whether or not to use them. Valid bits are for cases that go
outside of what Reserved 0 compatibility rules can convey, and I think
Reserved 0 compatiblity fully covers us in this case.

So, if a device use case breaks because they set 10, but clear 9 and
expect software to ignore 10 then they get to keep all the pieces
because they have already broken the expectations of Reserved 0
compat-software created before 9 existed.

Re: [PATCH v1 04/19] cxl: Add Get Supported Features command for kernel usage

[Dan Williams]

Jonathan Cameron wrote:
> On Thu, 23 Jan 2025 16:30:17 -0800
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
> > Dave Jiang wrote:
> > > CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
> > > The command retrieve the list of supported device-specific features
> > > (identified by UUID) and general information about each Feature.
> > > 
> > > The driver will retrieve the feature entries in order to make checks and
> > > provide information for the Get Feature and Set Feature command. One of
> > > the main piece of information retrieved are the effects a Set Feature
> > > command would have for a particular feature.
> > > 
> > > Co-developed-by: Shiju Jose <shiju.jose@huawei.com>
> > > Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> > > Signed-off-by: Dave Jiang <dave.jiang@intel.com>
[..]
> > This has me thinking that the EDAC integration should just be a feature
> > of the cxl_features_driver.
> 
> I'm not 100% sure I see what you mean here, but if you mean pushing
> the registration code into cxl_features.ko...
> 
> I replied to this in the other thread, but just to add a bit here.
> I think this creates a worse spiders web than we have today. Not
> all EDAC features we might add (e.g. device self test) have anything to do
> with get/set features.  I'm not sure we'll support that one in EDAC soon
> though it might make sense if there is generality with non CXL systems
> (I've been meaning to look into that.)

I came to the same conclusion and am now skeptical of cxl_features.ko vs
just extended the memdev ABI. We have long since shipped the expectation
that command passthrough is a compile time policy option. Each component
that wants a CXL-Features-FWCTL interface can just register one via CXL
core helpers. In this case it will be similar to what currently happens
with the fw_upload interface registered against the memdev, not a
cxl_fw_upload.ko driver.

> To me, the CXL features support is providing a service to some of the EDAC
> features, but they use other services from CXL drivers and potentially
> don't use CXL features at all.  For now we have the repair drivers
> that only use the features support for a tiny bit of what they do
> (tweaking device initiated aspects and detailed record generation
>  - if we even support those yet).
> 
> At some point we may have core CXL functionality depending on
> get/set features as well as there are things we definitely don't
> want in either EDAC or fwctl like metabits storage controls.
> So I'd be careful adding any thing to cxl_features beyond stuff
> that is definitely clustered with fwctl bit or helpers.
> 
> Soon I'd also expect us to add 'features' to fwctl interface
> that aren't using get/set features. I think you suggested that we
> could do this by making up some guids for them and pushing though
> this same interface.

Yes, although I hope the availability of FWCTL encourages new and
interesting device functionality to just be a Feature, so that we don't
have to play Linux GUID games. That might be what triggers the creation
of a query command because that could lead to future scenarios where the
kernel needs to dynamically mark a Feature as exclusive that was
previously allowed before the kernel knew about it.

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Jonathan Cameron]

On Mon, 27 Jan 2025 16:40:48 -0800
Dan Williams <dan.j.williams@intel.com> wrote:

> Jonathan Cameron wrote:
> >   
> > > > +}
> > > > +
> > > > +static void *cxlctl_get_supported_features(struct cxl_features_state *cfs,
> > > > +					   const struct fwctl_rpc_cxl *rpc_in,
> > > > +					   size_t *out_len)
> > > > +{
> > > > +	struct cxl_mbox_get_sup_feats_out *feat_out;
> > > > +	struct cxl_mbox_get_sup_feats_in feat_in;
> > > > +	struct cxl_feat_entry *saved, *pos;
> > > > +	int requested, copied;
> > > > +	size_t out_size;
> > > > +	u32 count;
> > > > +	u16 start;
> > > > +
> > > > +	if (rpc_in->op_size != sizeof(feat_in))
> > > > +		return ERR_PTR(-EINVAL);
> > > > +
> > > > +	if (copy_from_user(&feat_in, u64_to_user_ptr(rpc_in->in_payload),
> > > > +			   rpc_in->op_size))
> > > > +		return ERR_PTR(-EFAULT);
> > > > +
> > > > +	count = le32_to_cpu(feat_in.count);
> > > > +	start = le16_to_cpu(feat_in.start_idx);
> > > > +	requested = count / sizeof(*pos);
> > > > +
> > > > +	/*
> > > > +	 * Make sure that the total requested number of entries is not greater
> > > > +	 * than the total number of supported features allowed for userspace.
> > > > +	 */
> > > > +	if (start >= cfs->num_user_features)
> > > > +		return ERR_PTR(-EINVAL);
> > > > +
> > > > +	requested = min_t(int, requested, cfs->num_user_features - start);
> > > > +
> > > > +	out_size = sizeof(struct fwctl_rpc_cxl_out) + sizeof(*feat_out) +
> > > > +		   requested * sizeof(*pos);
> > > > +
> > > > +	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
> > > > +		kvzalloc(out_size, GFP_KERNEL);
> > > > +	if (!rpc_out)
> > > > +		return ERR_PTR(-ENOMEM);
> > > > +
> > > > +	rpc_out->size = sizeof(*feat_out) + requested * sizeof(*pos);
> > > > +	feat_out = (struct cxl_mbox_get_sup_feats_out *)rpc_out->payload;
> > > > +	if (requested == 0) {
> > > > +		feat_out->num_entries = cpu_to_le16(requested);
> > > > +		feat_out->supported_feats = cpu_to_le16(cfs->num_user_features);
> > > > +		rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
> > > > +		*out_len = out_size;
> > > > +		return no_free_ptr(rpc_out);
> > > > +	}
> > > > +
> > > > +	pos = &feat_out->ents[0];
> > > > +	saved = &cfs->entries[0];
> > > > +
> > > > +	copied = 0;
> > > > +	for (int i = 0; i < cfs->num_features; i++, saved++) {
> > > > +		if (is_cxl_feature_exclusive(saved))
> > > > +			continue;    
> > > 
> > > I think it's fine to let userspace see that exclusive features are
> > > present, just need to return EBUSY if userspace actually tries to use
> > > them.  
> > 
> > To me, a poke it and see interface is really ugly.  
> 
> That smells more like a matter of documentation. "Doctor it hurts when I
> try to use the documented kernel-exclusive commands?"

To me this is a nasty interface design.
If I'm writing a tool to enumerate what is exposed etc then it will
have to poke every get command just to list if an interface is available.
Hopefully none of them have side effects!

> 
> > In many cases we could let "get" through even if the we are using the interface
> > via some other kernel path and have it as exclusive.
> > (I don't know how useful that is, but maybe it makes sense).
> > 
> > If we ever do that, the only way to discover if an interface is available
> > will be to try the set interface.  Depending on design of feature
> > that might have side effects - hopefully get never does!  
> 
> I would not put it past some future device to make that mistake.

True.  Though I'd be up for a quirk list to block such commands if we
see them.  Can't deal with them until we know though.

> 
> > 
> > Alternatives:
> > 1. Flag. Maybe add something that makes it discoverable if a feature is
> >    in exclusive mode or not.  
> 
> I notice that all existing defined Features set a non-zero "Get Feature
> Size" in their Supported Feature Entry. I would not say "no" to just
> zero-ing out Get Feature Size as a hint that "you might get EBUSY due to
> kernel exclusivity with this command", but that still feels like
> overkill compared to documentation.

'might' is no use. Would have to be definitely as otherwise userspace
can't know the size that field conveyed.

> 
> > 2. Query type interface.  So a way to actually ask if a given feature is
> >    usable.  
> 
> Not sure we really need a programmatic way to read the documentation.
> 
> The CXL_MEM_COMMAND_FLAG_EXCLUSIVE flag is for cases where the
> exclusivity is transient. For these features the exclusivity is
> permanent, and I hope we never need to cross that
> transient-exclusivity-bridge for Features.

It's permanent today, but I can definitely see that not always being
the case - we may well have future kernel does things in X fashion but
for legacy support disable that CONFIG option. Not nice but definitely
plausible.

> 
> > 3. What we have here.  To me the simplest solution is hide what we can't
> >    be used.  
> 
> It is inconsistent that we do not do this for the other kernel exclusive
> commands in userspace retreived Command Effects Log. The ABI here is raw
> Get Supported Features payload.

If they were exposed via similar paths I'd agree consistency matters
but I 'hope' no one is going to have a tool that mixes fwctl and the
legacy path.  In my head we add all the useful commands to fwctl
and that legacy path ends up effectively deprecated.

Anyhow, I don't feel that strongly about this, it's just a case
of doesn't smell of roses to me.

> 
> > > > +	/* These effects supported for all scope */
> > > > +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
> > > > +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
> > > > +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
> > > > +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
> > > > +	    scope >= FWCTL_RPC_DEBUG_WRITE)
> > > > +		return true;    
> > > 
> > > Looks good for the known bits, but this needs to return false for the
> > > currently reserved bits because the driver can not assume a security
> > > model for future effects. If a future spec adds
> > > FWCTL_RPC_DEBUG_WRITE-safe effects, a new kernel is needed to allow
> > > those Feature commands through.
> > > 
> > > Sidenote: I wonder why the spec wasted one of its bits on an extend bit,
> > > but here we are. The 'extend' concept is typically something like
> > > "bit15: go look at this other field in this payload as this 16-bit field
> > > was exhausted", not "bit9: the bits above this originally defined 16 bit
> > > field now has more bits", oh well.  
> > 
> > It's odd but corner case of going from 'unknown' state for the remaining
> > pair of bits to 0 means this and 1 means this.  
> 
> I don't understand. 0 means no effect to worry about whether it is
> defined or not.
> 
> > Naming though doesn't match the spec that calls it CEL[11:10] valid.
> > Would be good to name it closer to that as we may well have something
> > in bits 12 and 15 in future and it doesn't refer to them.  
> 
> Hopefully we can head off another "valid2" mistake, and I don't think
> Linux needs to define anything for this bit. That bit's definition is:
> 
>    "Bit[9]: 1 is recommended, 0 is permitted (CEL[11:10] Valid)"
> 
> ...which translates to "useless". If 11 or 10 are set, I don't care what
> value 9 has.
>
> If 12:15 are set, I don't care if there is a future valid2
> bit gating whether or not to use them. Valid bits are for cases that go
> outside of what Reserved 0 compatibility rules can convey, and I think
> Reserved 0 compatiblity fully covers us in this case.

Seems the spec authors disagreed. (obviously I can't comment on that
discussion).

Using just what anyone can see (if they have the spec)
It was a clear spec hole and there wasn't an obvious default for 0 to
mean so it was a 'read your device docs and act appropriately' case
before this stuff was added.

There may be corner cases where the right answer if we know the
feature is not persisted over a reset but instead panic or take
some heavy weight action. Same can be true the other way around
in that we may have to do something heavy to manually reset something
we don't want to persist over reset. Hopefully not but we'll see.

> 
> So, if a device use case breaks because they set 10, but clear 9 and
> expect software to ignore 10 then they get to keep all the pieces
> because they have already broken the expectations of Reserved 0
> compat-software created before 9 existed.

True, but a compliance test should have dealt with that (seems they
are yet to catch up with this though).

Jonathan

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Dave Jiang]

On 1/28/25 5:01 AM, Jonathan Cameron wrote:
> On Mon, 27 Jan 2025 16:40:48 -0800
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
>> Jonathan Cameron wrote:
>>>   
>>>>> +}
>>>>> +
>>>>> +static void *cxlctl_get_supported_features(struct cxl_features_state *cfs,
>>>>> +					   const struct fwctl_rpc_cxl *rpc_in,
>>>>> +					   size_t *out_len)
>>>>> +{
>>>>> +	struct cxl_mbox_get_sup_feats_out *feat_out;
>>>>> +	struct cxl_mbox_get_sup_feats_in feat_in;
>>>>> +	struct cxl_feat_entry *saved, *pos;
>>>>> +	int requested, copied;
>>>>> +	size_t out_size;
>>>>> +	u32 count;
>>>>> +	u16 start;
>>>>> +
>>>>> +	if (rpc_in->op_size != sizeof(feat_in))
>>>>> +		return ERR_PTR(-EINVAL);
>>>>> +
>>>>> +	if (copy_from_user(&feat_in, u64_to_user_ptr(rpc_in->in_payload),
>>>>> +			   rpc_in->op_size))
>>>>> +		return ERR_PTR(-EFAULT);
>>>>> +
>>>>> +	count = le32_to_cpu(feat_in.count);
>>>>> +	start = le16_to_cpu(feat_in.start_idx);
>>>>> +	requested = count / sizeof(*pos);
>>>>> +
>>>>> +	/*
>>>>> +	 * Make sure that the total requested number of entries is not greater
>>>>> +	 * than the total number of supported features allowed for userspace.
>>>>> +	 */
>>>>> +	if (start >= cfs->num_user_features)
>>>>> +		return ERR_PTR(-EINVAL);
>>>>> +
>>>>> +	requested = min_t(int, requested, cfs->num_user_features - start);
>>>>> +
>>>>> +	out_size = sizeof(struct fwctl_rpc_cxl_out) + sizeof(*feat_out) +
>>>>> +		   requested * sizeof(*pos);
>>>>> +
>>>>> +	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
>>>>> +		kvzalloc(out_size, GFP_KERNEL);
>>>>> +	if (!rpc_out)
>>>>> +		return ERR_PTR(-ENOMEM);
>>>>> +
>>>>> +	rpc_out->size = sizeof(*feat_out) + requested * sizeof(*pos);
>>>>> +	feat_out = (struct cxl_mbox_get_sup_feats_out *)rpc_out->payload;
>>>>> +	if (requested == 0) {
>>>>> +		feat_out->num_entries = cpu_to_le16(requested);
>>>>> +		feat_out->supported_feats = cpu_to_le16(cfs->num_user_features);
>>>>> +		rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
>>>>> +		*out_len = out_size;
>>>>> +		return no_free_ptr(rpc_out);
>>>>> +	}
>>>>> +
>>>>> +	pos = &feat_out->ents[0];
>>>>> +	saved = &cfs->entries[0];
>>>>> +
>>>>> +	copied = 0;
>>>>> +	for (int i = 0; i < cfs->num_features; i++, saved++) {
>>>>> +		if (is_cxl_feature_exclusive(saved))
>>>>> +			continue;    
>>>>
>>>> I think it's fine to let userspace see that exclusive features are
>>>> present, just need to return EBUSY if userspace actually tries to use
>>>> them.  
>>>
>>> To me, a poke it and see interface is really ugly.  
>>
>> That smells more like a matter of documentation. "Doctor it hurts when I
>> try to use the documented kernel-exclusive commands?"
> 
> To me this is a nasty interface design.
> If I'm writing a tool to enumerate what is exposed etc then it will
> have to poke every get command just to list if an interface is available.
> Hopefully none of them have side effects!
> 
>>
>>> In many cases we could let "get" through even if the we are using the interface
>>> via some other kernel path and have it as exclusive.
>>> (I don't know how useful that is, but maybe it makes sense).
>>>
>>> If we ever do that, the only way to discover if an interface is available
>>> will be to try the set interface.  Depending on design of feature
>>> that might have side effects - hopefully get never does!  
>>
>> I would not put it past some future device to make that mistake.
> 
> True.  Though I'd be up for a quirk list to block such commands if we
> see them.  Can't deal with them until we know though.
> 
>>
>>>
>>> Alternatives:
>>> 1. Flag. Maybe add something that makes it discoverable if a feature is
>>>    in exclusive mode or not.  
>>
>> I notice that all existing defined Features set a non-zero "Get Feature
>> Size" in their Supported Feature Entry. I would not say "no" to just
>> zero-ing out Get Feature Size as a hint that "you might get EBUSY due to
>> kernel exclusivity with this command", but that still feels like
>> overkill compared to documentation.
> 
> 'might' is no use. Would have to be definitely as otherwise userspace
> can't know the size that field conveyed.

We could zero the "Set Feature Size" in the "Get Supported Features Supported Feature Entry". To indicate that the Feature cannot be changed. That would be spec compliant. Or is there concern that reading the feature may do something for some devices decide to be weird?

DJ


> 
>>
>>> 2. Query type interface.  So a way to actually ask if a given feature is
>>>    usable.  
>>
>> Not sure we really need a programmatic way to read the documentation.
>>
>> The CXL_MEM_COMMAND_FLAG_EXCLUSIVE flag is for cases where the
>> exclusivity is transient. For these features the exclusivity is
>> permanent, and I hope we never need to cross that
>> transient-exclusivity-bridge for Features.
> 
> It's permanent today, but I can definitely see that not always being
> the case - we may well have future kernel does things in X fashion but
> for legacy support disable that CONFIG option. Not nice but definitely
> plausible.
> 
>>
>>> 3. What we have here.  To me the simplest solution is hide what we can't
>>>    be used.  
>>
>> It is inconsistent that we do not do this for the other kernel exclusive
>> commands in userspace retreived Command Effects Log. The ABI here is raw
>> Get Supported Features payload.
> 
> If they were exposed via similar paths I'd agree consistency matters
> but I 'hope' no one is going to have a tool that mixes fwctl and the
> legacy path.  In my head we add all the useful commands to fwctl
> and that legacy path ends up effectively deprecated.
> 
> Anyhow, I don't feel that strongly about this, it's just a case
> of doesn't smell of roses to me.
> 
>>
>>>>> +	/* These effects supported for all scope */
>>>>> +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
>>>>> +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
>>>>> +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
>>>>> +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
>>>>> +	    scope >= FWCTL_RPC_DEBUG_WRITE)
>>>>> +		return true;    
>>>>
>>>> Looks good for the known bits, but this needs to return false for the
>>>> currently reserved bits because the driver can not assume a security
>>>> model for future effects. If a future spec adds
>>>> FWCTL_RPC_DEBUG_WRITE-safe effects, a new kernel is needed to allow
>>>> those Feature commands through.
>>>>
>>>> Sidenote: I wonder why the spec wasted one of its bits on an extend bit,
>>>> but here we are. The 'extend' concept is typically something like
>>>> "bit15: go look at this other field in this payload as this 16-bit field
>>>> was exhausted", not "bit9: the bits above this originally defined 16 bit
>>>> field now has more bits", oh well.  
>>>
>>> It's odd but corner case of going from 'unknown' state for the remaining
>>> pair of bits to 0 means this and 1 means this.  
>>
>> I don't understand. 0 means no effect to worry about whether it is
>> defined or not.
>>
>>> Naming though doesn't match the spec that calls it CEL[11:10] valid.
>>> Would be good to name it closer to that as we may well have something
>>> in bits 12 and 15 in future and it doesn't refer to them.  
>>
>> Hopefully we can head off another "valid2" mistake, and I don't think
>> Linux needs to define anything for this bit. That bit's definition is:
>>
>>    "Bit[9]: 1 is recommended, 0 is permitted (CEL[11:10] Valid)"
>>
>> ...which translates to "useless". If 11 or 10 are set, I don't care what
>> value 9 has.
>>
>> If 12:15 are set, I don't care if there is a future valid2
>> bit gating whether or not to use them. Valid bits are for cases that go
>> outside of what Reserved 0 compatibility rules can convey, and I think
>> Reserved 0 compatiblity fully covers us in this case.
> 
> Seems the spec authors disagreed. (obviously I can't comment on that
> discussion).
> 
> Using just what anyone can see (if they have the spec)
> It was a clear spec hole and there wasn't an obvious default for 0 to
> mean so it was a 'read your device docs and act appropriately' case
> before this stuff was added.
> 
> There may be corner cases where the right answer if we know the
> feature is not persisted over a reset but instead panic or take
> some heavy weight action. Same can be true the other way around
> in that we may have to do something heavy to manually reset something
> we don't want to persist over reset. Hopefully not but we'll see.
> 
>>
>> So, if a device use case breaks because they set 10, but clear 9 and
>> expect software to ignore 10 then they get to keep all the pieces
>> because they have already broken the expectations of Reserved 0
>> compat-software created before 9 existed.
> 
> True, but a compliance test should have dealt with that (seems they
> are yet to catch up with this though).
> 
> Jonathan
> 

Re: [PATCH v1 07/19] cxl/mbox: Add GET_FEATURE mailbox command

[Dave Jiang]

On 1/24/25 3:58 PM, Dan Williams wrote:
> Dave Jiang wrote:
>> From: Shiju Jose <shiju.jose@huawei.com>
>>
>> Add support for GET_FEATURE mailbox command.
>>
>> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
>> The settings of a feature can be retrieved using Get Feature command.
>> CXL spec 3.1 section 8.2.9.6.2 describes Get Feature command.
>>
>> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>> v1:
>> - pass in cxl_mbox instead of cxlds (Dan)
>> - Move to the feature driver model. (Dan)
>> ---
>>  drivers/cxl/core/features.c | 74 +++++++++++++++++++++++++++++++++++++
>>  drivers/cxl/features.c      |  6 +--
>>  include/cxl/features.h      | 27 ++++++++++++++
>>  3 files changed, 102 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
>> index 66a4b82910e6..ab9386b53a95 100644
>> --- a/drivers/cxl/core/features.c
>> +++ b/drivers/cxl/core/features.c
>> @@ -4,6 +4,7 @@
>>  #include <cxl/mailbox.h>
>>  #include "cxl.h"
>>  #include "core.h"
>> +#include "cxlmem.h"
>>  
>>  #define CXL_FEATURE_MAX_DEVS 65536
>>  static DEFINE_IDA(cxl_features_ida);
>> @@ -97,3 +98,76 @@ cxl_get_supported_feature_entry(struct cxl_features *features,
>>  	return ERR_PTR(-ENOENT);
>>  }
>>  EXPORT_SYMBOL_NS_GPL(cxl_get_supported_feature_entry, "CXL");
>> +
>> +bool cxl_feature_enabled(struct cxl_features_state *cfs, u16 opcode)
>> +{
>> +	struct cxl_mailbox *cxl_mbox = cfs->features->cxl_mbox;
>> +	struct cxl_mem_command *cmd;
>> +
>> +	cmd = cxl_find_feature_command(opcode);
>> +	if (!cmd)
>> +		return false;
>> +
>> +	return test_bit(cmd->info.id, cxl_mbox->feature_cmds);
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_feature_enabled, "CXL");
>> +
>> +size_t cxl_get_feature(struct cxl_features *features, const uuid_t feat_uuid,
>> +		       enum cxl_get_feat_selection selection,
>> +		       void *feat_out, size_t feat_out_size, u16 offset,
> 
> Is @feat_out guaranteed to be a kernel pointer, or might it be an __user
> pointer?

@feat_out will be a kernel pointer. The way FWCTL is setup, the callback returns a kernel buffer that FWCTL core will copy out to user space and then free.

> 
> Shouldn't @feat_uuid be a 'uuid_t *' rather than a 'uuid_t'?

right. also code needs to use uuid_copy().

> 
>> +		       u16 *return_code)
>> +{
>> +	size_t data_to_rd_size, size_out;
>> +	struct cxl_features_state *cfs;
>> +	struct cxl_mbox_get_feat_in pi;
>> +	struct cxl_mailbox *cxl_mbox;
>> +	struct cxl_mbox_cmd mbox_cmd;
>> +	size_t data_rcvd_size = 0;
>> +	int rc;
>> +
>> +	if (return_code)
>> +		*return_code = CXL_MBOX_CMD_RC_INPUT;
>> +
>> +	cfs = dev_get_drvdata(&features->dev);
>> +	if (!cfs)
>> +		return 0;
>> +
>> +	if (!cxl_feature_enabled(cfs, CXL_MBOX_OP_GET_FEATURE))
>> +		return 0;
> 
> Per previous feedback, just make the caller responsible for knowing this
> in advance, not checking every call. With that gone this function loses
> its dependency on 'struct cxl_features' and 'struct cxl_features_state'
> and can take 'struct cxl_mbox' directly.
> 
> If for some reason the caller did not know that Get Feature was missing
> the device will still fail it anyway, so that boilerplate is not serving
> any useful purpose.

ok

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Jonathan Cameron]

On Tue, 28 Jan 2025 08:55:57 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> On 1/28/25 5:01 AM, Jonathan Cameron wrote:
> > On Mon, 27 Jan 2025 16:40:48 -0800
> > Dan Williams <dan.j.williams@intel.com> wrote:
> >   
> >> Jonathan Cameron wrote:  
> >>>     
> >>>>> +}
> >>>>> +
> >>>>> +static void *cxlctl_get_supported_features(struct cxl_features_state *cfs,
> >>>>> +					   const struct fwctl_rpc_cxl *rpc_in,
> >>>>> +					   size_t *out_len)
> >>>>> +{
> >>>>> +	struct cxl_mbox_get_sup_feats_out *feat_out;
> >>>>> +	struct cxl_mbox_get_sup_feats_in feat_in;
> >>>>> +	struct cxl_feat_entry *saved, *pos;
> >>>>> +	int requested, copied;
> >>>>> +	size_t out_size;
> >>>>> +	u32 count;
> >>>>> +	u16 start;
> >>>>> +
> >>>>> +	if (rpc_in->op_size != sizeof(feat_in))
> >>>>> +		return ERR_PTR(-EINVAL);
> >>>>> +
> >>>>> +	if (copy_from_user(&feat_in, u64_to_user_ptr(rpc_in->in_payload),
> >>>>> +			   rpc_in->op_size))
> >>>>> +		return ERR_PTR(-EFAULT);
> >>>>> +
> >>>>> +	count = le32_to_cpu(feat_in.count);
> >>>>> +	start = le16_to_cpu(feat_in.start_idx);
> >>>>> +	requested = count / sizeof(*pos);
> >>>>> +
> >>>>> +	/*
> >>>>> +	 * Make sure that the total requested number of entries is not greater
> >>>>> +	 * than the total number of supported features allowed for userspace.
> >>>>> +	 */
> >>>>> +	if (start >= cfs->num_user_features)
> >>>>> +		return ERR_PTR(-EINVAL);
> >>>>> +
> >>>>> +	requested = min_t(int, requested, cfs->num_user_features - start);
> >>>>> +
> >>>>> +	out_size = sizeof(struct fwctl_rpc_cxl_out) + sizeof(*feat_out) +
> >>>>> +		   requested * sizeof(*pos);
> >>>>> +
> >>>>> +	struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) =
> >>>>> +		kvzalloc(out_size, GFP_KERNEL);
> >>>>> +	if (!rpc_out)
> >>>>> +		return ERR_PTR(-ENOMEM);
> >>>>> +
> >>>>> +	rpc_out->size = sizeof(*feat_out) + requested * sizeof(*pos);
> >>>>> +	feat_out = (struct cxl_mbox_get_sup_feats_out *)rpc_out->payload;
> >>>>> +	if (requested == 0) {
> >>>>> +		feat_out->num_entries = cpu_to_le16(requested);
> >>>>> +		feat_out->supported_feats = cpu_to_le16(cfs->num_user_features);
> >>>>> +		rpc_out->retval = CXL_MBOX_CMD_RC_SUCCESS;
> >>>>> +		*out_len = out_size;
> >>>>> +		return no_free_ptr(rpc_out);
> >>>>> +	}
> >>>>> +
> >>>>> +	pos = &feat_out->ents[0];
> >>>>> +	saved = &cfs->entries[0];
> >>>>> +
> >>>>> +	copied = 0;
> >>>>> +	for (int i = 0; i < cfs->num_features; i++, saved++) {
> >>>>> +		if (is_cxl_feature_exclusive(saved))
> >>>>> +			continue;      
> >>>>
> >>>> I think it's fine to let userspace see that exclusive features are
> >>>> present, just need to return EBUSY if userspace actually tries to use
> >>>> them.    
> >>>
> >>> To me, a poke it and see interface is really ugly.    
> >>
> >> That smells more like a matter of documentation. "Doctor it hurts when I
> >> try to use the documented kernel-exclusive commands?"  
> > 
> > To me this is a nasty interface design.
> > If I'm writing a tool to enumerate what is exposed etc then it will
> > have to poke every get command just to list if an interface is available.
> > Hopefully none of them have side effects!
> >   
> >>  
> >>> In many cases we could let "get" through even if the we are using the interface
> >>> via some other kernel path and have it as exclusive.
> >>> (I don't know how useful that is, but maybe it makes sense).
> >>>
> >>> If we ever do that, the only way to discover if an interface is available
> >>> will be to try the set interface.  Depending on design of feature
> >>> that might have side effects - hopefully get never does!    
> >>
> >> I would not put it past some future device to make that mistake.  
> > 
> > True.  Though I'd be up for a quirk list to block such commands if we
> > see them.  Can't deal with them until we know though.
> >   
> >>  
> >>>
> >>> Alternatives:
> >>> 1. Flag. Maybe add something that makes it discoverable if a feature is
> >>>    in exclusive mode or not.    
> >>
> >> I notice that all existing defined Features set a non-zero "Get Feature
> >> Size" in their Supported Feature Entry. I would not say "no" to just
> >> zero-ing out Get Feature Size as a hint that "you might get EBUSY due to
> >> kernel exclusivity with this command", but that still feels like
> >> overkill compared to documentation.  
> > 
> > 'might' is no use. Would have to be definitely as otherwise userspace
> > can't know the size that field conveyed.  
> 
> We could zero the "Set Feature Size" in the "Get Supported Features Supported Feature Entry". To indicate that the Feature cannot be changed. That would be spec compliant. Or is there concern that reading the feature may do something for some devices decide to be weird?

Let's hope not.  I'm fine with this suggestion. Seems like a reasonable
compromise.

Jonathan

> 
> DJ
> 
> 
> >   
> >>  
> >>> 2. Query type interface.  So a way to actually ask if a given feature is
> >>>    usable.    
> >>
> >> Not sure we really need a programmatic way to read the documentation.
> >>
> >> The CXL_MEM_COMMAND_FLAG_EXCLUSIVE flag is for cases where the
> >> exclusivity is transient. For these features the exclusivity is
> >> permanent, and I hope we never need to cross that
> >> transient-exclusivity-bridge for Features.  
> > 
> > It's permanent today, but I can definitely see that not always being
> > the case - we may well have future kernel does things in X fashion but
> > for legacy support disable that CONFIG option. Not nice but definitely
> > plausible.
> >   
> >>  
> >>> 3. What we have here.  To me the simplest solution is hide what we can't
> >>>    be used.    
> >>
> >> It is inconsistent that we do not do this for the other kernel exclusive
> >> commands in userspace retreived Command Effects Log. The ABI here is raw
> >> Get Supported Features payload.  
> > 
> > If they were exposed via similar paths I'd agree consistency matters
> > but I 'hope' no one is going to have a tool that mixes fwctl and the
> > legacy path.  In my head we add all the useful commands to fwctl
> > and that legacy path ends up effectively deprecated.
> > 
> > Anyhow, I don't feel that strongly about this, it's just a case
> > of doesn't smell of roses to me.
> >   
> >>  
> >>>>> +	/* These effects supported for all scope */
> >>>>> +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
> >>>>> +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
> >>>>> +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
> >>>>> +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
> >>>>> +	    scope >= FWCTL_RPC_DEBUG_WRITE)
> >>>>> +		return true;      
> >>>>
> >>>> Looks good for the known bits, but this needs to return false for the
> >>>> currently reserved bits because the driver can not assume a security
> >>>> model for future effects. If a future spec adds
> >>>> FWCTL_RPC_DEBUG_WRITE-safe effects, a new kernel is needed to allow
> >>>> those Feature commands through.
> >>>>
> >>>> Sidenote: I wonder why the spec wasted one of its bits on an extend bit,
> >>>> but here we are. The 'extend' concept is typically something like
> >>>> "bit15: go look at this other field in this payload as this 16-bit field
> >>>> was exhausted", not "bit9: the bits above this originally defined 16 bit
> >>>> field now has more bits", oh well.    
> >>>
> >>> It's odd but corner case of going from 'unknown' state for the remaining
> >>> pair of bits to 0 means this and 1 means this.    
> >>
> >> I don't understand. 0 means no effect to worry about whether it is
> >> defined or not.
> >>  
> >>> Naming though doesn't match the spec that calls it CEL[11:10] valid.
> >>> Would be good to name it closer to that as we may well have something
> >>> in bits 12 and 15 in future and it doesn't refer to them.    
> >>
> >> Hopefully we can head off another "valid2" mistake, and I don't think
> >> Linux needs to define anything for this bit. That bit's definition is:
> >>
> >>    "Bit[9]: 1 is recommended, 0 is permitted (CEL[11:10] Valid)"
> >>
> >> ...which translates to "useless". If 11 or 10 are set, I don't care what
> >> value 9 has.
> >>
> >> If 12:15 are set, I don't care if there is a future valid2
> >> bit gating whether or not to use them. Valid bits are for cases that go
> >> outside of what Reserved 0 compatibility rules can convey, and I think
> >> Reserved 0 compatiblity fully covers us in this case.  
> > 
> > Seems the spec authors disagreed. (obviously I can't comment on that
> > discussion).
> > 
> > Using just what anyone can see (if they have the spec)
> > It was a clear spec hole and there wasn't an obvious default for 0 to
> > mean so it was a 'read your device docs and act appropriately' case
> > before this stuff was added.
> > 
> > There may be corner cases where the right answer if we know the
> > feature is not persisted over a reset but instead panic or take
> > some heavy weight action. Same can be true the other way around
> > in that we may have to do something heavy to manually reset something
> > we don't want to persist over reset. Hopefully not but we'll see.
> >   
> >>
> >> So, if a device use case breaks because they set 10, but clear 9 and
> >> expect software to ignore 10 then they get to keep all the pieces
> >> because they have already broken the expectations of Reserved 0
> >> compat-software created before 9 existed.  
> > 
> > True, but a compliance test should have dealt with that (seems they
> > are yet to catch up with this though).
> > 
> > Jonathan
> >   
> 
> 

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Dan Williams]

Jonathan Cameron wrote:
[..]
> > > > I think it's fine to let userspace see that exclusive features are
> > > > present, just need to return EBUSY if userspace actually tries to use
> > > > them.  
> > > 
> > > To me, a poke it and see interface is really ugly.  
> > 
> > That smells more like a matter of documentation. "Doctor it hurts when I
> > try to use the documented kernel-exclusive commands?"
> 
> To me this is a nasty interface design.
> If I'm writing a tool to enumerate what is exposed etc then it will
> have to poke every get command just to list if an interface is available.
> Hopefully none of them have side effects!

The kernel exclusive list is documented. How did this tool get written
in such a way to understand how to get data out of the interface but
without reading the documentation on how to consume that data?

> > > In many cases we could let "get" through even if the we are using the interface
> > > via some other kernel path and have it as exclusive.
> > > (I don't know how useful that is, but maybe it makes sense).
> > > 
> > > If we ever do that, the only way to discover if an interface is available
> > > will be to try the set interface.  Depending on design of feature
> > > that might have side effects - hopefully get never does!  
> > 
> > I would not put it past some future device to make that mistake.
> 
> True.  Though I'd be up for a quirk list to block such commands if we
> see them.  Can't deal with them until we know though.

True, I overlooked that Get Feature is documented as having no effects
and the only way to communicate an Effect is for the Set Feature
command.

> > > Alternatives:
> > > 1. Flag. Maybe add something that makes it discoverable if a feature is
> > >    in exclusive mode or not.  
> > 
> > I notice that all existing defined Features set a non-zero "Get Feature
> > Size" in their Supported Feature Entry. I would not say "no" to just
> > zero-ing out Get Feature Size as a hint that "you might get EBUSY due to
> > kernel exclusivity with this command", but that still feels like
> > overkill compared to documentation.
> 
> 'might' is no use. Would have to be definitely as otherwise userspace
> can't know the size that field conveyed.

Agree, and this falls back to "documentation by itself is sufficient".

> > > 2. Query type interface.  So a way to actually ask if a given feature is
> > >    usable.  
> > 
> > Not sure we really need a programmatic way to read the documentation.
> > 
> > The CXL_MEM_COMMAND_FLAG_EXCLUSIVE flag is for cases where the
> > exclusivity is transient. For these features the exclusivity is
> > permanent, and I hope we never need to cross that
> > transient-exclusivity-bridge for Features.
> 
> It's permanent today, but I can definitely see that not always being
> the case - we may well have future kernel does things in X fashion but
> for legacy support disable that CONFIG option. Not nice but definitely
> plausible.

Then we cross that bridge and build some new ABI to communicate
transient exclusivity, and that new state of the world will be
documented as to how to discover that new capability.

Something like a Linux specific feature that returns a list of transient
and permanently exclusive Features.

In the meantime no need to hide useful information from userspace.

> > > 3. What we have here.  To me the simplest solution is hide what we can't
> > >    be used.  
> > 
> > It is inconsistent that we do not do this for the other kernel exclusive
> > commands in userspace retreived Command Effects Log. The ABI here is raw
> > Get Supported Features payload.
> 
> If they were exposed via similar paths I'd agree consistency matters
> but I 'hope' no one is going to have a tool that mixes fwctl and the
> legacy path.  In my head we add all the useful commands to fwctl
> and that legacy path ends up effectively deprecated.
> 
> Anyhow, I don't feel that strongly about this, it's just a case
> of doesn't smell of roses to me.
> 
> > 
> > > > > +	/* These effects supported for all scope */
> > > > > +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
> > > > > +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
> > > > > +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
> > > > > +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
> > > > > +	    scope >= FWCTL_RPC_DEBUG_WRITE)
> > > > > +		return true;    
> > > > 
> > > > Looks good for the known bits, but this needs to return false for the
> > > > currently reserved bits because the driver can not assume a security
> > > > model for future effects. If a future spec adds
> > > > FWCTL_RPC_DEBUG_WRITE-safe effects, a new kernel is needed to allow
> > > > those Feature commands through.
> > > > 
> > > > Sidenote: I wonder why the spec wasted one of its bits on an extend bit,
> > > > but here we are. The 'extend' concept is typically something like
> > > > "bit15: go look at this other field in this payload as this 16-bit field
> > > > was exhausted", not "bit9: the bits above this originally defined 16 bit
> > > > field now has more bits", oh well.  
> > > 
> > > It's odd but corner case of going from 'unknown' state for the remaining
> > > pair of bits to 0 means this and 1 means this.  
> > 
> > I don't understand. 0 means no effect to worry about whether it is
> > defined or not.
> > 
> > > Naming though doesn't match the spec that calls it CEL[11:10] valid.
> > > Would be good to name it closer to that as we may well have something
> > > in bits 12 and 15 in future and it doesn't refer to them.  
> > 
> > Hopefully we can head off another "valid2" mistake, and I don't think
> > Linux needs to define anything for this bit. That bit's definition is:
> > 
> >    "Bit[9]: 1 is recommended, 0 is permitted (CEL[11:10] Valid)"
> > 
> > ...which translates to "useless". If 11 or 10 are set, I don't care what
> > value 9 has.
> >
> > If 12:15 are set, I don't care if there is a future valid2
> > bit gating whether or not to use them. Valid bits are for cases that go
> > outside of what Reserved 0 compatibility rules can convey, and I think
> > Reserved 0 compatiblity fully covers us in this case.
> 
> Seems the spec authors disagreed. (obviously I can't comment on that
> discussion).
> 
> Using just what anyone can see (if they have the spec)
> It was a clear spec hole and there wasn't an obvious default for 0 to
> mean so it was a 'read your device docs and act appropriately' case
> before this stuff was added.

I see v2 is still trying to pretend this bit matters.

Are you saying that because the unused bits were marked 0 instead of
"Reserved" that software needs to play this game of checking an extend
bit?

What breaks if software treats those bits as Reserved0? What breaks if
software ignores bit9?

> There may be corner cases where the right answer if we know the
> feature is not persisted over a reset but instead panic or take
> some heavy weight action. Same can be true the other way around
> in that we may have to do something heavy to manually reset something
> we don't want to persist over reset. Hopefully not but we'll see.

...but how is that relevant to the FWCTL use case which just wants to
know if the operation is going to have an immediate effect that changes
a parameter behind the kernel's back?

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Jonathan Cameron]

On Mon, 3 Feb 2025 17:43:38 -0800
Dan Williams <dan.j.williams@intel.com> wrote:

> Jonathan Cameron wrote:
> [..]
> > > > > I think it's fine to let userspace see that exclusive features are
> > > > > present, just need to return EBUSY if userspace actually tries to use
> > > > > them.    
> > > > 
> > > > To me, a poke it and see interface is really ugly.    
> > > 
> > > That smells more like a matter of documentation. "Doctor it hurts when I
> > > try to use the documented kernel-exclusive commands?"  
> > 
> > To me this is a nasty interface design.
> > If I'm writing a tool to enumerate what is exposed etc then it will
> > have to poke every get command just to list if an interface is available.
> > Hopefully none of them have side effects!  
> 
> The kernel exclusive list is documented. How did this tool get written
> in such a way to understand how to get data out of the interface but
> without reading the documentation on how to consume that data?

Today's kernel exclusive list is documented.  Kind of tricky to know what
is on that list in a few years time.

> 
> > > > 2. Query type interface.  So a way to actually ask if a given feature is
> > > >    usable.    
> > > 
> > > Not sure we really need a programmatic way to read the documentation.
> > > 
> > > The CXL_MEM_COMMAND_FLAG_EXCLUSIVE flag is for cases where the
> > > exclusivity is transient. For these features the exclusivity is
> > > permanent, and I hope we never need to cross that
> > > transient-exclusivity-bridge for Features.  
> > 
> > It's permanent today, but I can definitely see that not always being
> > the case - we may well have future kernel does things in X fashion but
> > for legacy support disable that CONFIG option. Not nice but definitely
> > plausible.  
> 
> Then we cross that bridge and build some new ABI to communicate
> transient exclusivity, and that new state of the world will be
> documented as to how to discover that new capability.
> 
> Something like a Linux specific feature that returns a list of transient
> and permanently exclusive Features.
> 
> In the meantime no need to hide useful information from userspace.

Ok.  Seems like a more complex long term solution, but meh I don't care
that much.

> 
> > > > 3. What we have here.  To me the simplest solution is hide what we can't
> > > >    be used.    
> > > 
> > > It is inconsistent that we do not do this for the other kernel exclusive
> > > commands in userspace retreived Command Effects Log. The ABI here is raw
> > > Get Supported Features payload.  
> > 
> > If they were exposed via similar paths I'd agree consistency matters
> > but I 'hope' no one is going to have a tool that mixes fwctl and the
> > legacy path.  In my head we add all the useful commands to fwctl
> > and that legacy path ends up effectively deprecated.
> > 
> > Anyhow, I don't feel that strongly about this, it's just a case
> > of doesn't smell of roses to me.
> >   
> > >   
> > > > > > +	/* These effects supported for all scope */
> > > > > > +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
> > > > > > +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
> > > > > > +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
> > > > > > +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
> > > > > > +	    scope >= FWCTL_RPC_DEBUG_WRITE)
> > > > > > +		return true;      
> > > > > 
> > > > > Looks good for the known bits, but this needs to return false for the
> > > > > currently reserved bits because the driver can not assume a security
> > > > > model for future effects. If a future spec adds
> > > > > FWCTL_RPC_DEBUG_WRITE-safe effects, a new kernel is needed to allow
> > > > > those Feature commands through.
> > > > > 
> > > > > Sidenote: I wonder why the spec wasted one of its bits on an extend bit,
> > > > > but here we are. The 'extend' concept is typically something like
> > > > > "bit15: go look at this other field in this payload as this 16-bit field
> > > > > was exhausted", not "bit9: the bits above this originally defined 16 bit
> > > > > field now has more bits", oh well.    
> > > > 
> > > > It's odd but corner case of going from 'unknown' state for the remaining
> > > > pair of bits to 0 means this and 1 means this.    
> > > 
> > > I don't understand. 0 means no effect to worry about whether it is
> > > defined or not.
> > >   
> > > > Naming though doesn't match the spec that calls it CEL[11:10] valid.
> > > > Would be good to name it closer to that as we may well have something
> > > > in bits 12 and 15 in future and it doesn't refer to them.    
> > > 
> > > Hopefully we can head off another "valid2" mistake, and I don't think
> > > Linux needs to define anything for this bit. That bit's definition is:
> > > 
> > >    "Bit[9]: 1 is recommended, 0 is permitted (CEL[11:10] Valid)"
> > > 
> > > ...which translates to "useless". If 11 or 10 are set, I don't care what
> > > value 9 has.
> > >
> > > If 12:15 are set, I don't care if there is a future valid2
> > > bit gating whether or not to use them. Valid bits are for cases that go
> > > outside of what Reserved 0 compatibility rules can convey, and I think
> > > Reserved 0 compatiblity fully covers us in this case.  
> > 
> > Seems the spec authors disagreed. (obviously I can't comment on that
> > discussion).
> > 
> > Using just what anyone can see (if they have the spec)
> > It was a clear spec hole and there wasn't an obvious default for 0 to
> > mean so it was a 'read your device docs and act appropriately' case
> > before this stuff was added.  
> 
> I see v2 is still trying to pretend this bit matters.
> 
> Are you saying that because the unused bits were marked 0 instead of
> "Reserved" that software needs to play this game of checking an extend
> bit?
> 
> What breaks if software treats those bits as Reserved0? What breaks if
> software ignores bit9?

More generally we can't assume it doesn't happen if those bits are 0.
In this particular case perhaps it doesn't matter.

Hmm. The aim is to decide if the permissions DEBUG_WRITE is enough.

We have already verified it doesn't happen immediately. So now
the question is does it result in a change conventional reset.

Without bit 9 we have no idea either way.  So question is do
we assume it does, or assume it doesn't?

+	/* These effects supported for all scope */
+	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
+	     (effects & CXL_CMD_EFFECTS_EXTEND &&
+	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
+	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
+	    scope >= FWCTL_RPC_DEBUG_WRITE)
+		return true;
+
+	return false;

Right now we return false if it was conventional reset but
we don't know that because the hardware doesn't say either way.

I'm stuck on what the right thing to do is.  My assumption
is that cold reset will always apply if conv or cxl reset
do. So maybe who cares and we shouldn't check these two
at all?

Fun corner: there must be something that causes an effect
to happen (or in what sense is this a 'set') and if we have
excluded everything else maybe we should assume that
it must be one of these two resets?


> 
> > There may be corner cases where the right answer if we know the
> > feature is not persisted over a reset but instead panic or take
> > some heavy weight action. Same can be true the other way around
> > in that we may have to do something heavy to manually reset something
> > we don't want to persist over reset. Hopefully not but we'll see.  
> 
> ...but how is that relevant to the FWCTL use case which just wants to
> know if the operation is going to have an immediate effect that changes
> a parameter behind the kernel's back?

In that case, should we just let everything through and not check for
cold reset either?

Jonathan

> 

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Dan Williams]

Jonathan Cameron wrote:
> On Mon, 3 Feb 2025 17:43:38 -0800
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
> > Jonathan Cameron wrote:
> > [..]
> > > > > > I think it's fine to let userspace see that exclusive features are
> > > > > > present, just need to return EBUSY if userspace actually tries to use
> > > > > > them.    
> > > > > 
> > > > > To me, a poke it and see interface is really ugly.    
> > > > 
> > > > That smells more like a matter of documentation. "Doctor it hurts when I
> > > > try to use the documented kernel-exclusive commands?"  
> > > 
> > > To me this is a nasty interface design.
> > > If I'm writing a tool to enumerate what is exposed etc then it will
> > > have to poke every get command just to list if an interface is available.
> > > Hopefully none of them have side effects!  
> > 
> > The kernel exclusive list is documented. How did this tool get written
> > in such a way to understand how to get data out of the interface but
> > without reading the documentation on how to consume that data?
> 
> Today's kernel exclusive list is documented.  Kind of tricky to know what
> is on that list in a few years time.

I expect new software for new capabilities considers new documentation.

[..]
> > What breaks if software treats those bits as Reserved0? What breaks if
> > software ignores bit9?
> 
> More generally we can't assume it doesn't happen if those bits are 0.
> In this particular case perhaps it doesn't matter.
> 
> Hmm. The aim is to decide if the permissions DEBUG_WRITE is enough.
> 
> We have already verified it doesn't happen immediately. So now
> the question is does it result in a change conventional reset.
> 
> Without bit 9 we have no idea either way.  So question is do
> we assume it does, or assume it doesn't?

If bit10 is set, assume it does, otherwise not, but as you say below,
does any of this matter once immediate config change effects are
accounted?

I think no.

> 
> +	/* These effects supported for all scope */
> +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
> +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
> +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
> +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
> +	    scope >= FWCTL_RPC_DEBUG_WRITE)
> +		return true;
> +
> +	return false;
> 
> Right now we return false if it was conventional reset but
> we don't know that because the hardware doesn't say either way.
>
> I'm stuck on what the right thing to do is.  My assumption
> is that cold reset will always apply if conv or cxl reset
> do. So maybe who cares and we shouldn't check these two
> at all?

That's true, if the security policy is that reset based configuration
changes are always ok, then bits [0, 9:11] do not matter.

> Fun corner: there must be something that causes an effect
> to happen (or in what sense is this a 'set') and if we have
> excluded everything else maybe we should assume that
> it must be one of these two resets?

Yes, fun. "What exactly did this effectless 'Set' do?". I think I would
be ok with blocking "effectless" Set just to make the device commit to
a security model for all of its 'Set'-able Features.

> > > There may be corner cases where the right answer if we know the
> > > feature is not persisted over a reset but instead panic or take
> > > some heavy weight action. Same can be true the other way around
> > > in that we may have to do something heavy to manually reset something
> > > we don't want to persist over reset. Hopefully not but we'll see.  
> > 
> > ...but how is that relevant to the FWCTL use case which just wants to
> > know if the operation is going to have an immediate effect that changes
> > a parameter behind the kernel's back?
> 
> In that case, should we just let everything through and not check for
> cold reset either?

I think so, yes. Filter the known problematic effects for live changes,
and require the device to claim at least one of [0:5, 10:11] in its
effects, and filter Features that claim [12:15] effects for now.

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Jonathan Cameron]

On Tue, 4 Feb 2025 14:26:02 -0800
Dan Williams <dan.j.williams@intel.com> wrote:

> Jonathan Cameron wrote:
> > On Mon, 3 Feb 2025 17:43:38 -0800
> > Dan Williams <dan.j.williams@intel.com> wrote:
> >   
> > > Jonathan Cameron wrote:
> > > [..]  
> > > > > > > I think it's fine to let userspace see that exclusive features are
> > > > > > > present, just need to return EBUSY if userspace actually tries to use
> > > > > > > them.      
> > > > > > 
> > > > > > To me, a poke it and see interface is really ugly.      
> > > > > 
> > > > > That smells more like a matter of documentation. "Doctor it hurts when I
> > > > > try to use the documented kernel-exclusive commands?"    
> > > > 
> > > > To me this is a nasty interface design.
> > > > If I'm writing a tool to enumerate what is exposed etc then it will
> > > > have to poke every get command just to list if an interface is available.
> > > > Hopefully none of them have side effects!    
> > > 
> > > The kernel exclusive list is documented. How did this tool get written
> > > in such a way to understand how to get data out of the interface but
> > > without reading the documentation on how to consume that data?  
> > 
> > Today's kernel exclusive list is documented.  Kind of tricky to know what
> > is on that list in a few years time.  
> 
> I expect new software for new capabilities considers new documentation.

Time travel issue.  New capabilities are supported as features the day
they turn up. Tools are written. 

Sometime later we might decide kernel support is needed and make them
exclusive.

Ah well, we'll cope.
> 
> [..]
> > > What breaks if software treats those bits as Reserved0? What breaks if
> > > software ignores bit9?  
> > 
> > More generally we can't assume it doesn't happen if those bits are 0.
> > In this particular case perhaps it doesn't matter.
> > 
> > Hmm. The aim is to decide if the permissions DEBUG_WRITE is enough.
> > 
> > We have already verified it doesn't happen immediately. So now
> > the question is does it result in a change conventional reset.
> > 
> > Without bit 9 we have no idea either way.  So question is do
> > we assume it does, or assume it doesn't?  
> 
> If bit10 is set, assume it does, otherwise not, but as you say below,
> does any of this matter once immediate config change effects are
> accounted?

If it's set and 9 isn't - hardware bug! Not really our problem.


> 
> I think no.
> 
> > 
> > +	/* These effects supported for all scope */
> > +	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
> > +	     (effects & CXL_CMD_EFFECTS_EXTEND &&
> > +	      (effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET ||
> > +	       effects & CXL_CMD_CONFIG_CHANGE_CXL_RESET))) &&
> > +	    scope >= FWCTL_RPC_DEBUG_WRITE)
> > +		return true;
> > +
> > +	return false;
> > 
> > Right now we return false if it was conventional reset but
> > we don't know that because the hardware doesn't say either way.
> >
> > I'm stuck on what the right thing to do is.  My assumption
> > is that cold reset will always apply if conv or cxl reset
> > do. So maybe who cares and we shouldn't check these two
> > at all?  
> 
> That's true, if the security policy is that reset based configuration
> changes are always ok, then bits [0, 9:11] do not matter.
> 
> > Fun corner: there must be something that causes an effect
> > to happen (or in what sense is this a 'set') and if we have
> > excluded everything else maybe we should assume that
> > it must be one of these two resets?  
> 
> Yes, fun. "What exactly did this effectless 'Set' do?". I think I would
> be ok with blocking "effectless" Set just to make the device commit to
> a security model for all of its 'Set'-able Features.
> 
> > > > There may be corner cases where the right answer if we know the
> > > > feature is not persisted over a reset but instead panic or take
> > > > some heavy weight action. Same can be true the other way around
> > > > in that we may have to do something heavy to manually reset something
> > > > we don't want to persist over reset. Hopefully not but we'll see.    
> > > 
> > > ...but how is that relevant to the FWCTL use case which just wants to
> > > know if the operation is going to have an immediate effect that changes
> > > a parameter behind the kernel's back?  
> > 
> > In that case, should we just let everything through and not check for
> > cold reset either?  
> 
> I think so, yes. Filter the known problematic effects for live changes,
> and require the device to claim at least one of [0:5, 10:11] in its
> effects, and filter Features that claim [12:15] effects for now.
Ok.  With a big comment to say any device setting 10:11 that
doesn't set 9 is buggy but we don't care.

Otherwise someone will 'fix' it.

Jonathan

Re: [PATCH v1 14/19] cxl: Add support for fwctl RPC command to enable CXL feature commands

[Dan Williams]

Jonathan Cameron wrote:
> On Tue, 4 Feb 2025 14:26:02 -0800
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
> > Jonathan Cameron wrote:
> > > On Mon, 3 Feb 2025 17:43:38 -0800
> > > Dan Williams <dan.j.williams@intel.com> wrote:
> > >   
> > > > Jonathan Cameron wrote:
> > > > [..]  
> > > > > > > > I think it's fine to let userspace see that exclusive features are
> > > > > > > > present, just need to return EBUSY if userspace actually tries to use
> > > > > > > > them.      
> > > > > > > 
> > > > > > > To me, a poke it and see interface is really ugly.      
> > > > > > 
> > > > > > That smells more like a matter of documentation. "Doctor it hurts when I
> > > > > > try to use the documented kernel-exclusive commands?"    
> > > > > 
> > > > > To me this is a nasty interface design.
> > > > > If I'm writing a tool to enumerate what is exposed etc then it will
> > > > > have to poke every get command just to list if an interface is available.
> > > > > Hopefully none of them have side effects!    
> > > > 
> > > > The kernel exclusive list is documented. How did this tool get written
> > > > in such a way to understand how to get data out of the interface but
> > > > without reading the documentation on how to consume that data?  
> > > 
> > > Today's kernel exclusive list is documented.  Kind of tricky to know what
> > > is on that list in a few years time.  
> > 
> > I expect new software for new capabilities considers new documentation.
> 
> Time travel issue.  New capabilities are supported as features the day
> they turn up. Tools are written. 
> 
> Sometime later we might decide kernel support is needed and make them
> exclusive.
> 
> Ah well, we'll cope.

Yes, that is always going to be a risk especially with the vendor
experimentation that this is going to allow. However, vendors will also
need to heed the FWCTL documentation that this interface is for
non-primary functionality of the device.

I am comforted by the fact that all immediate-effect commands will be
relegated to a debug / permissive security posture. A high value Feature
likely graduates out of vendor-specific passthrough to a kernel mediated
interface when folks inevitably demand that it also work with
restrictive (lockdown) security stance. I have seen this dynamic play
out in the past when folks realize, for example, that debugfs is
disabled on lockdown kernels.

In the meantime the kernel is not going out of its way to preemptively
hide the presence of Features.

> > [..]
> > > > What breaks if software treats those bits as Reserved0? What breaks if
> > > > software ignores bit9?  
> > > 
> > > More generally we can't assume it doesn't happen if those bits are 0.
> > > In this particular case perhaps it doesn't matter.
> > > 
> > > Hmm. The aim is to decide if the permissions DEBUG_WRITE is enough.
> > > 
> > > We have already verified it doesn't happen immediately. So now
> > > the question is does it result in a change conventional reset.
> > > 
> > > Without bit 9 we have no idea either way.  So question is do
> > > we assume it does, or assume it doesn't?  
> > 
> > If bit10 is set, assume it does, otherwise not, but as you say below,
> > does any of this matter once immediate config change effects are
> > accounted?
> 
> If it's set and 9 isn't - hardware bug! Not really our problem.

...right, a bug we can safely ignore.

[..]
> > I think so, yes. Filter the known problematic effects for live changes,
> > and require the device to claim at least one of [0:5, 10:11] in its
> > effects, and filter Features that claim [12:15] effects for now.
> Ok.  With a big comment to say any device setting 10:11 that
> doesn't set 9 is buggy but we don't care.
> 
> Otherwise someone will 'fix' it.

Works for me.