[RFC PATCH v2 0/20] fwctl/cxl: Add CXL feature commands support via fwctl

[RFC PATCH v2 0/20] fwctl/cxl: Add CXL feature commands support via fwctl

[Dave Jiang]

v2:
- Dropped 1/13 and 2/13 from previous version. Merged upstream already.
- Combined changes from Shiju for "get supported features"
- Addressed comments from Jonathan and Jason
- See specific changes in individual patch revision history
- Added hardware command info to FWCTL
- Added filtering to set feature command
- Added documentation

This series add support for CXL feature commands using the FWCTL framework [1].
The code is untested and I'm looking for architectural and implementation feedback.
While CXL currently has a chardev for user ioctls to send some mailbox
commands to a memory device, the fwctl framework provides more security policies
that can be a potential vehicle to move CXL ioctl path to that.

For this RFC, the mailbox commands "Get Supported Features", "Get Feature", and
"Set Feature" commands are implemented. The "get" commands under the
FWCTL_RPC_DEBUG_READ_ONLY policy, the "set" command checks the policy depending
on the effect of the feature. All mailbox commands for CXL provides an effects
table that describes the effects of a command when performed on the device.
For CXL features, there is also an effects field that describes the effects
a feature write operation has on the device per feature. The security policy
is checked against this feature specific effects field. Looking for discussion
on matching the CXL spec defined effects with the FWCTL security policy.

The code is based off of v3 of FWCTL series [1] posted by Jason and rebased on top of
v6.12-rc6.

Jason,
Please see 11/20 and 13/20 in the series WRT the new hw info ioctl I introduced.
Let me know if that looks reasonable or if I should do that in a different way to
achieve what I want.

Jonathan and Shiju,
16/20 adds filtering of set feature command. Please check and see if that looks
reasonable for how we want to apply restrictions to the set feature command.

Shiju,
Please pick up 2/20 for your EDAC series. I made some corrections, including a pointer
math issue in cxl_get_supported_features() while looping through the feature
entries after saving them.

Jonathan,
I added documentation in 20/20. Maybe we can flesh out the usage policy there.

[1]: https://lore.kernel.org/linux-cxl/0-v3-960f17f90f17+516-fwctl_jgg@nvidia.com/#r

---

Dave Jiang (18):
      cxl: Refactor user ioctl command path from mds to mailbox
      cxl: Add Get Supported Features command for kernel usage
      cxl/test: Add Get Supported Features mailbox command support
      cxl: Add Get Feature command support for user submission
      cxl: Add Set Feature command support for user submission
      cxl: Move cxl_driver related bits to be usable by external drivers
      fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands
      fwctl/cxl: Add support for get driver information
      fwctl: FWCTL_HW_INFO to return hardware information
      cxl: Save Command Effects Log (CEL) effects for enabled commands
      fwctl/cxl: Add hw_info callback
      cxl: Move defines and error codes from cxlmem.h to cxl/mailbox.h
      fwctl/cxl: Add support for fwctl RPC command to enable CXL feature commands
      fwctl/cxl: Add support to filter exclusive features
      cxl/test: Add Get Feature support to cxl_test
      cxl/test: Add Set Feature support to cxl_test
      fwctl: Move fwctl documentation to its own directory
      fwctl/cxl: Add documentation to FWCTL CXL

Shiju Jose (2):
      cxl/mbox: Add GET_FEATURE mailbox command
      cxl/mbox: Add SET_FEATURE mailbox command

 Documentation/userspace-api/fwctl/fwctl-cxl.rst   |  91 +++++
 Documentation/userspace-api/{ => fwctl}/fwctl.rst |   0
 Documentation/userspace-api/fwctl/index.rst       |  13 +
 Documentation/userspace-api/index.rst             |   2 +-
 MAINTAINERS                                       |  10 +-
 drivers/cxl/core/core.h                           |   9 +-
 drivers/cxl/core/mbox.c                           | 859 ++++++++++++++++++++++++++++++++++++++----
 drivers/cxl/core/memdev.c                         |  24 +-
 drivers/cxl/cxl.h                                 |  32 +-
 drivers/cxl/cxlmem.h                              | 184 ++++-----
 drivers/cxl/pci.c                                 |   4 +
 drivers/fwctl/Kconfig                             |   9 +
 drivers/fwctl/Makefile                            |   1 +
 drivers/fwctl/cxl/Makefile                        |   4 +
 drivers/fwctl/cxl/cxl.c                           | 237 ++++++++++++
 drivers/fwctl/main.c                              |  33 ++
 include/cxl/cxl.h                                 |  40 ++
 include/cxl/features.h                            |  52 +++
 include/cxl/mailbox.h                             | 183 ++++++++-
 include/linux/fwctl.h                             |   6 +
 include/uapi/fwctl/cxl.h                          |  51 +++
 include/uapi/fwctl/fwctl.h                        |  59 ++-
 include/uapi/linux/cxl_mem.h                      |  20 +-
 tools/testing/cxl/test/mem.c                      | 201 ++++++++++
 24 files changed, 1903 insertions(+), 221 deletions(-)      

[RFC PATCH v2 01/20] cxl: Refactor user ioctl command path from mds to mailbox

[Dave Jiang]

With 'struct cxl_mailbox' context introduced, the helper functions
cxl_query_cmd() and cxl_send_cmd() can take a cxl_mailbox directly
rather than a cxl_memdev parameter. Refactor to use cxl_mailbox
directly.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/core.h   |  6 ++-
 drivers/cxl/core/mbox.c   | 89 +++++++++++++++++++--------------------
 drivers/cxl/core/memdev.c | 22 +++++++---
 drivers/cxl/cxlmem.h      | 40 ------------------
 include/cxl/mailbox.h     | 41 +++++++++++++++++-
 5 files changed, 102 insertions(+), 96 deletions(-)

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index 0c62b4069ba0..ff30d1c99ca7 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -4,6 +4,8 @@
 #ifndef __CXL_CORE_H__
 #define __CXL_CORE_H__
 
+#include <cxl/mailbox.h>
+
 extern const struct device_type cxl_nvdimm_bridge_type;
 extern const struct device_type cxl_nvdimm_type;
 extern const struct device_type cxl_pmu_type;
@@ -65,9 +67,9 @@ static inline void cxl_region_exit(void)
 
 struct cxl_send_command;
 struct cxl_mem_query_commands;
-int cxl_query_cmd(struct cxl_memdev *cxlmd,
+int cxl_query_cmd(struct cxl_mailbox *cxl_mbox,
 		  struct cxl_mem_query_commands __user *q);
-int cxl_send_cmd(struct cxl_memdev *cxlmd, struct cxl_send_command __user *s);
+int cxl_send_cmd(struct cxl_mailbox *cxl_mailbox, struct cxl_send_command __user *s);
 void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
 				   resource_size_t length);
 
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 5175138c4fb7..880ac1dba3cc 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -349,40 +349,40 @@ static bool cxl_payload_from_user_allowed(u16 opcode, void *payload_in)
 	return true;
 }
 
-static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox,
-			     struct cxl_memdev_state *mds, u16 opcode,
+static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox_cmd,
+			     struct cxl_mailbox *cxl_mbox, u16 opcode,
 			     size_t in_size, size_t out_size, u64 in_payload)
 {
-	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
-	*mbox = (struct cxl_mbox_cmd) {
+	*mbox_cmd = (struct cxl_mbox_cmd) {
 		.opcode = opcode,
 		.size_in = in_size,
 	};
 
 	if (in_size) {
-		mbox->payload_in = vmemdup_user(u64_to_user_ptr(in_payload),
-						in_size);
-		if (IS_ERR(mbox->payload_in))
-			return PTR_ERR(mbox->payload_in);
+		mbox_cmd->payload_in = vmemdup_user(u64_to_user_ptr(in_payload),
+						    in_size);
+		if (IS_ERR(mbox_cmd->payload_in))
+			return PTR_ERR(mbox_cmd->payload_in);
 
-		if (!cxl_payload_from_user_allowed(opcode, mbox->payload_in)) {
-			dev_dbg(mds->cxlds.dev, "%s: input payload not allowed\n",
+		if (!cxl_payload_from_user_allowed(opcode,
+						   mbox_cmd->payload_in)) {
+			dev_dbg(cxl_mbox->host, "%s: input payload not allowed\n",
 				cxl_mem_opcode_to_name(opcode));
-			kvfree(mbox->payload_in);
+			kvfree(mbox_cmd->payload_in);
 			return -EBUSY;
 		}
 	}
 
 	/* Prepare to handle a full payload for variable sized output */
 	if (out_size == CXL_VARIABLE_PAYLOAD)
-		mbox->size_out = cxl_mbox->payload_size;
+		mbox_cmd->size_out = cxl_mbox->payload_size;
 	else
-		mbox->size_out = out_size;
+		mbox_cmd->size_out = out_size;
 
-	if (mbox->size_out) {
-		mbox->payload_out = kvzalloc(mbox->size_out, GFP_KERNEL);
-		if (!mbox->payload_out) {
-			kvfree(mbox->payload_in);
+	if (mbox_cmd->size_out) {
+		mbox_cmd->payload_out = kvzalloc(mbox_cmd->size_out, GFP_KERNEL);
+		if (!mbox_cmd->payload_out) {
+			kvfree(mbox_cmd->payload_in);
 			return -ENOMEM;
 		}
 	}
@@ -397,10 +397,8 @@ static void cxl_mbox_cmd_dtor(struct cxl_mbox_cmd *mbox)
 
 static int cxl_to_mem_cmd_raw(struct cxl_mem_command *mem_cmd,
 			      const struct cxl_send_command *send_cmd,
-			      struct cxl_memdev_state *mds)
+			      struct cxl_mailbox *cxl_mbox)
 {
-	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
-
 	if (send_cmd->raw.rsvd)
 		return -EINVAL;
 
@@ -415,7 +413,7 @@ static int cxl_to_mem_cmd_raw(struct cxl_mem_command *mem_cmd,
 	if (!cxl_mem_raw_command_allowed(send_cmd->raw.opcode))
 		return -EPERM;
 
-	dev_WARN_ONCE(mds->cxlds.dev, true, "raw command path used\n");
+	dev_WARN_ONCE(cxl_mbox->host, true, "raw command path used\n");
 
 	*mem_cmd = (struct cxl_mem_command) {
 		.info = {
@@ -431,7 +429,7 @@ static int cxl_to_mem_cmd_raw(struct cxl_mem_command *mem_cmd,
 
 static int cxl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
 			  const struct cxl_send_command *send_cmd,
-			  struct cxl_memdev_state *mds)
+			  struct cxl_mailbox *cxl_mbox)
 {
 	struct cxl_mem_command *c = &cxl_mem_commands[send_cmd->id];
 	const struct cxl_command_info *info = &c->info;
@@ -446,11 +444,11 @@ static int cxl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
 		return -EINVAL;
 
 	/* Check that the command is enabled for hardware */
-	if (!test_bit(info->id, mds->enabled_cmds))
+	if (!test_bit(info->id, cxl_mbox->enabled_cmds))
 		return -ENOTTY;
 
 	/* Check that the command is not claimed for exclusive kernel use */
-	if (test_bit(info->id, mds->exclusive_cmds))
+	if (test_bit(info->id, cxl_mbox->exclusive_cmds))
 		return -EBUSY;
 
 	/* Check the input buffer is the expected size */
@@ -479,7 +477,7 @@ static int cxl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
 /**
  * cxl_validate_cmd_from_user() - Check fields for CXL_MEM_SEND_COMMAND.
  * @mbox_cmd: Sanitized and populated &struct cxl_mbox_cmd.
- * @mds: The driver data for the operation
+ * @cxl_mbox: CXL mailbox context
  * @send_cmd: &struct cxl_send_command copied in from userspace.
  *
  * Return:
@@ -494,10 +492,9 @@ static int cxl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
  * safe to send to the hardware.
  */
 static int cxl_validate_cmd_from_user(struct cxl_mbox_cmd *mbox_cmd,
-				      struct cxl_memdev_state *mds,
+				      struct cxl_mailbox *cxl_mbox,
 				      const struct cxl_send_command *send_cmd)
 {
-	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
 	struct cxl_mem_command mem_cmd;
 	int rc;
 
@@ -514,24 +511,23 @@ static int cxl_validate_cmd_from_user(struct cxl_mbox_cmd *mbox_cmd,
 
 	/* Sanitize and construct a cxl_mem_command */
 	if (send_cmd->id == CXL_MEM_COMMAND_ID_RAW)
-		rc = cxl_to_mem_cmd_raw(&mem_cmd, send_cmd, mds);
+		rc = cxl_to_mem_cmd_raw(&mem_cmd, send_cmd, cxl_mbox);
 	else
-		rc = cxl_to_mem_cmd(&mem_cmd, send_cmd, mds);
+		rc = cxl_to_mem_cmd(&mem_cmd, send_cmd, cxl_mbox);
 
 	if (rc)
 		return rc;
 
 	/* Sanitize and construct a cxl_mbox_cmd */
-	return cxl_mbox_cmd_ctor(mbox_cmd, mds, mem_cmd.opcode,
+	return cxl_mbox_cmd_ctor(mbox_cmd, cxl_mbox, mem_cmd.opcode,
 				 mem_cmd.info.size_in, mem_cmd.info.size_out,
 				 send_cmd->in.payload);
 }
 
-int cxl_query_cmd(struct cxl_memdev *cxlmd,
+int cxl_query_cmd(struct cxl_mailbox *cxl_mbox,
 		  struct cxl_mem_query_commands __user *q)
 {
-	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
-	struct device *dev = &cxlmd->dev;
+	struct device *dev = cxl_mbox->host;
 	struct cxl_mem_command *cmd;
 	u32 n_commands;
 	int j = 0;
@@ -552,9 +548,9 @@ int cxl_query_cmd(struct cxl_memdev *cxlmd,
 	cxl_for_each_cmd(cmd) {
 		struct cxl_command_info info = cmd->info;
 
-		if (test_bit(info.id, mds->enabled_cmds))
+		if (test_bit(info.id, cxl_mbox->enabled_cmds))
 			info.flags |= CXL_MEM_COMMAND_FLAG_ENABLED;
-		if (test_bit(info.id, mds->exclusive_cmds))
+		if (test_bit(info.id, cxl_mbox->exclusive_cmds))
 			info.flags |= CXL_MEM_COMMAND_FLAG_EXCLUSIVE;
 
 		if (copy_to_user(&q->commands[j++], &info, sizeof(info)))
@@ -569,7 +565,7 @@ int cxl_query_cmd(struct cxl_memdev *cxlmd,
 
 /**
  * handle_mailbox_cmd_from_user() - Dispatch a mailbox command for userspace.
- * @mds: The driver data for the operation
+ * @cxl_mbox: The mailbox context for the operation.
  * @mbox_cmd: The validated mailbox command.
  * @out_payload: Pointer to userspace's output payload.
  * @size_out: (Input) Max payload size to copy out.
@@ -590,13 +586,12 @@ int cxl_query_cmd(struct cxl_memdev *cxlmd,
  *
  * See cxl_send_cmd().
  */
-static int handle_mailbox_cmd_from_user(struct cxl_memdev_state *mds,
+static int handle_mailbox_cmd_from_user(struct cxl_mailbox *cxl_mbox,
 					struct cxl_mbox_cmd *mbox_cmd,
 					u64 out_payload, s32 *size_out,
 					u32 *retval)
 {
-	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
-	struct device *dev = mds->cxlds.dev;
+	struct device *dev = cxl_mbox->host;
 	int rc;
 
 	dev_dbg(dev,
@@ -633,10 +628,9 @@ static int handle_mailbox_cmd_from_user(struct cxl_memdev_state *mds,
 	return rc;
 }
 
-int cxl_send_cmd(struct cxl_memdev *cxlmd, struct cxl_send_command __user *s)
+int cxl_send_cmd(struct cxl_mailbox *cxl_mbox, struct cxl_send_command __user *s)
 {
-	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
-	struct device *dev = &cxlmd->dev;
+	struct device *dev = cxl_mbox->host;
 	struct cxl_send_command send;
 	struct cxl_mbox_cmd mbox_cmd;
 	int rc;
@@ -646,11 +640,11 @@ int cxl_send_cmd(struct cxl_memdev *cxlmd, struct cxl_send_command __user *s)
 	if (copy_from_user(&send, s, sizeof(send)))
 		return -EFAULT;
 
-	rc = cxl_validate_cmd_from_user(&mbox_cmd, mds, &send);
+	rc = cxl_validate_cmd_from_user(&mbox_cmd, cxl_mbox, &send);
 	if (rc)
 		return rc;
 
-	rc = handle_mailbox_cmd_from_user(mds, &mbox_cmd, send.out.payload,
+	rc = handle_mailbox_cmd_from_user(cxl_mbox, &mbox_cmd, send.out.payload,
 					  &send.out.size, &send.retval);
 	if (rc)
 		return rc;
@@ -724,6 +718,7 @@ static int cxl_xfer_log(struct cxl_memdev_state *mds, uuid_t *uuid,
  */
 static void cxl_walk_cel(struct cxl_memdev_state *mds, size_t size, u8 *cel)
 {
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
 	struct cxl_cel_entry *cel_entry;
 	const int cel_entries = size / sizeof(*cel_entry);
 	struct device *dev = mds->cxlds.dev;
@@ -737,7 +732,7 @@ static void cxl_walk_cel(struct cxl_memdev_state *mds, size_t size, u8 *cel)
 		int enabled = 0;
 
 		if (cmd) {
-			set_bit(cmd->info.id, mds->enabled_cmds);
+			set_bit(cmd->info.id, cxl_mbox->enabled_cmds);
 			enabled++;
 		}
 
@@ -807,6 +802,7 @@ static const uuid_t log_uuid[] = {
  */
 int cxl_enumerate_cmds(struct cxl_memdev_state *mds)
 {
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
 	struct cxl_mbox_get_supported_logs *gsl;
 	struct device *dev = mds->cxlds.dev;
 	struct cxl_mem_command *cmd;
@@ -845,7 +841,7 @@ int cxl_enumerate_cmds(struct cxl_memdev_state *mds)
 		/* In case CEL was bogus, enable some default commands. */
 		cxl_for_each_cmd(cmd)
 			if (cmd->flags & CXL_CMD_FLAG_FORCE_ENABLE)
-				set_bit(cmd->info.id, mds->enabled_cmds);
+				set_bit(cmd->info.id, cxl_mbox->enabled_cmds);
 
 		/* Found the required CEL */
 		rc = 0;
@@ -1448,6 +1444,7 @@ struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev)
 	mutex_init(&mds->event.log_lock);
 	mds->cxlds.dev = dev;
 	mds->cxlds.reg_map.host = dev;
+	mds->cxlds.cxl_mbox.host = dev;
 	mds->cxlds.reg_map.resource = CXL_RESOURCE_NONE;
 	mds->cxlds.type = CXL_DEVTYPE_CLASSMEM;
 	mds->ram_perf.qos_class = CXL_QOS_CLASS_INVALID;
diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
index 84fefb76dafa..4d544a55ac3e 100644
--- a/drivers/cxl/core/memdev.c
+++ b/drivers/cxl/core/memdev.c
@@ -564,9 +564,11 @@ EXPORT_SYMBOL_NS_GPL(is_cxl_memdev, CXL);
 void set_exclusive_cxl_commands(struct cxl_memdev_state *mds,
 				unsigned long *cmds)
 {
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+
 	down_write(&cxl_memdev_rwsem);
-	bitmap_or(mds->exclusive_cmds, mds->exclusive_cmds, cmds,
-		  CXL_MEM_COMMAND_ID_MAX);
+	bitmap_or(cxl_mbox->exclusive_cmds, cxl_mbox->exclusive_cmds,
+		  cmds, CXL_MEM_COMMAND_ID_MAX);
 	up_write(&cxl_memdev_rwsem);
 }
 EXPORT_SYMBOL_NS_GPL(set_exclusive_cxl_commands, CXL);
@@ -579,9 +581,11 @@ EXPORT_SYMBOL_NS_GPL(set_exclusive_cxl_commands, CXL);
 void clear_exclusive_cxl_commands(struct cxl_memdev_state *mds,
 				  unsigned long *cmds)
 {
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+
 	down_write(&cxl_memdev_rwsem);
-	bitmap_andnot(mds->exclusive_cmds, mds->exclusive_cmds, cmds,
-		      CXL_MEM_COMMAND_ID_MAX);
+	bitmap_andnot(cxl_mbox->exclusive_cmds, cxl_mbox->exclusive_cmds,
+		      cmds, CXL_MEM_COMMAND_ID_MAX);
 	up_write(&cxl_memdev_rwsem);
 }
 EXPORT_SYMBOL_NS_GPL(clear_exclusive_cxl_commands, CXL);
@@ -656,11 +660,14 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds,
 static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
 			       unsigned long arg)
 {
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+
 	switch (cmd) {
 	case CXL_MEM_QUERY_COMMANDS:
-		return cxl_query_cmd(cxlmd, (void __user *)arg);
+		return cxl_query_cmd(cxl_mbox, (void __user *)arg);
 	case CXL_MEM_SEND_COMMAND:
-		return cxl_send_cmd(cxlmd, (void __user *)arg);
+		return cxl_send_cmd(cxl_mbox, (void __user *)arg);
 	default:
 		return -ENOTTY;
 	}
@@ -994,10 +1001,11 @@ static void cxl_remove_fw_upload(void *fwl)
 int devm_cxl_setup_fw_upload(struct device *host, struct cxl_memdev_state *mds)
 {
 	struct cxl_dev_state *cxlds = &mds->cxlds;
+	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
 	struct device *dev = &cxlds->cxlmd->dev;
 	struct fw_upload *fwl;
 
-	if (!test_bit(CXL_MEM_COMMAND_ID_GET_FW_INFO, mds->enabled_cmds))
+	if (!test_bit(CXL_MEM_COMMAND_ID_GET_FW_INFO, cxl_mbox->enabled_cmds))
 		return 0;
 
 	fwl = firmware_upload_register(THIS_MODULE, dev, dev_name(dev),
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 2a25d1957ddb..a0a49809cd76 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -106,42 +106,6 @@ static inline struct cxl_ep *cxl_ep_load(struct cxl_port *port,
 	return xa_load(&port->endpoints, (unsigned long)&cxlmd->dev);
 }
 
-/**
- * struct cxl_mbox_cmd - A command to be submitted to hardware.
- * @opcode: (input) The command set and command submitted to hardware.
- * @payload_in: (input) Pointer to the input payload.
- * @payload_out: (output) Pointer to the output payload. Must be allocated by
- *		 the caller.
- * @size_in: (input) Number of bytes to load from @payload_in.
- * @size_out: (input) Max number of bytes loaded into @payload_out.
- *            (output) Number of bytes generated by the device. For fixed size
- *            outputs commands this is always expected to be deterministic. For
- *            variable sized output commands, it tells the exact number of bytes
- *            written.
- * @min_out: (input) internal command output payload size validation
- * @poll_count: (input) Number of timeouts to attempt.
- * @poll_interval_ms: (input) Time between mailbox background command polling
- *                    interval timeouts.
- * @return_code: (output) Error code returned from hardware.
- *
- * This is the primary mechanism used to send commands to the hardware.
- * All the fields except @payload_* correspond exactly to the fields described in
- * Command Register section of the CXL 2.0 8.2.8.4.5. @payload_in and
- * @payload_out are written to, and read from the Command Payload Registers
- * defined in CXL 2.0 8.2.8.4.8.
- */
-struct cxl_mbox_cmd {
-	u16 opcode;
-	void *payload_in;
-	void *payload_out;
-	size_t size_in;
-	size_t size_out;
-	size_t min_out;
-	int poll_count;
-	int poll_interval_ms;
-	u16 return_code;
-};
-
 /*
  * Per CXL 3.0 Section 8.2.8.4.5.1
  */
@@ -461,8 +425,6 @@ static inline struct cxl_dev_state *mbox_to_cxlds(struct cxl_mailbox *cxl_mbox)
  * @lsa_size: Size of Label Storage Area
  *                (CXL 2.0 8.2.9.5.1.1 Identify Memory Device)
  * @firmware_version: Firmware version for the memory device.
- * @enabled_cmds: Hardware commands found enabled in CEL.
- * @exclusive_cmds: Commands that are kernel-internal only
  * @total_bytes: sum of all possible capacities
  * @volatile_only_bytes: hard volatile capacity
  * @persistent_only_bytes: hard persistent capacity
@@ -485,8 +447,6 @@ struct cxl_memdev_state {
 	struct cxl_dev_state cxlds;
 	size_t lsa_size;
 	char firmware_version[0x10];
-	DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
-	DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
 	u64 total_bytes;
 	u64 volatile_only_bytes;
 	u64 persistent_only_bytes;
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index bacd111e75f1..cc894f07a435 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -3,12 +3,49 @@
 #ifndef __CXL_MBOX_H__
 #define __CXL_MBOX_H__
 #include <linux/rcuwait.h>
+#include <uapi/linux/cxl_mem.h>
 
-struct cxl_mbox_cmd;
+/**
+ * struct cxl_mbox_cmd - A command to be submitted to hardware.
+ * @opcode: (input) The command set and command submitted to hardware.
+ * @payload_in: (input) Pointer to the input payload.
+ * @payload_out: (output) Pointer to the output payload. Must be allocated by
+ *		 the caller.
+ * @size_in: (input) Number of bytes to load from @payload_in.
+ * @size_out: (input) Max number of bytes loaded into @payload_out.
+ *            (output) Number of bytes generated by the device. For fixed size
+ *            outputs commands this is always expected to be deterministic. For
+ *            variable sized output commands, it tells the exact number of bytes
+ *            written.
+ * @min_out: (input) internal command output payload size validation
+ * @poll_count: (input) Number of timeouts to attempt.
+ * @poll_interval_ms: (input) Time between mailbox background command polling
+ *                    interval timeouts.
+ * @return_code: (output) Error code returned from hardware.
+ *
+ * This is the primary mechanism used to send commands to the hardware.
+ * All the fields except @payload_* correspond exactly to the fields described in
+ * Command Register section of the CXL 2.0 8.2.8.4.5. @payload_in and
+ * @payload_out are written to, and read from the Command Payload Registers
+ * defined in CXL 2.0 8.2.8.4.8.
+ */
+struct cxl_mbox_cmd {
+	u16 opcode;
+	void *payload_in;
+	void *payload_out;
+	size_t size_in;
+	size_t size_out;
+	size_t min_out;
+	int poll_count;
+	int poll_interval_ms;
+	u16 return_code;
+};
 
 /**
  * struct cxl_mailbox - context for CXL mailbox operations
  * @host: device that hosts the mailbox
+ * @enabled_cmds: mailbox commands that are enabled by the driver
+ * @exclusive_cmds: mailbox commands that are exclusive to the kernel
  * @payload_size: Size of space for payload
  *                (CXL 3.1 8.2.8.4.3 Mailbox Capabilities Register)
  * @mbox_mutex: mutex protects device mailbox and firmware
@@ -17,6 +54,8 @@ struct cxl_mbox_cmd;
  */
 struct cxl_mailbox {
 	struct device *host;
+	DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
+	DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
 	size_t payload_size;
 	struct mutex mbox_mutex; /* lock to protect mailbox context */
 	struct rcuwait mbox_wait;
-- 
2.47.0

[RFC PATCH v2 02/20] cxl: Add Get Supported Features command for kernel usage

[Dave Jiang]

CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
The command retrieve the list of supported device-specific features
(identified by UUID) and general information about each Feature.

The driver will retrieve the feature entries in order to make checks and
provide information for the Get Feature and Set Feature command. One of
the main piece of information retrieved are the effects a Set Feature
command would have for a particular feature.

Co-developed-by: Shiju Jose <shiju.jose@huawei.com>
Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---

v2:
- fix feature entry pointer math.
- free mbox_out in cxl_get_supported_features(). (Shiju, Jonathan)
- replace sizeof(struct) with feat_size. (Shiju)
- replace reserved in feature structs with u8 type. (Shiju)
- change cxl_feat_entry->effects to set_effects. (Shiju)
- rearrange assigment of cxl_mbox->entries. (Jonathan)
- Separate no features from actual error. (Jonathan)
- Fix missing kdoc for entries. (Jonathan)
---
 drivers/cxl/core/mbox.c      | 178 +++++++++++++++++++++++++++++++++++
 drivers/cxl/cxlmem.h         |  31 ++++++
 drivers/cxl/pci.c            |   4 +
 include/cxl/mailbox.h        |   4 +
 include/uapi/linux/cxl_mem.h |   1 +
 5 files changed, 218 insertions(+)

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 880ac1dba3cc..4ba56f3d5a65 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -67,6 +67,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
 	CXL_CMD(SET_SHUTDOWN_STATE, 0x1, 0, 0),
 	CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
 	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
+	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
 };
 
 /*
@@ -790,6 +791,183 @@ static const uuid_t log_uuid[] = {
 	[VENDOR_DEBUG_UUID] = DEFINE_CXL_VENDOR_DEBUG_UUID,
 };
 
+static void cxl_free_features(void *features)
+{
+	kvfree(features);
+}
+
+static int cxl_get_supported_features_count(struct cxl_dev_state *cxlds)
+{
+	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
+	struct cxl_mbox_get_sup_feats_out mbox_out;
+	struct cxl_mbox_get_sup_feats_in mbox_in;
+	struct cxl_mbox_cmd mbox_cmd;
+	int rc;
+
+	memset(&mbox_in, 0, sizeof(mbox_in));
+	mbox_in.count = cpu_to_le32(sizeof(mbox_out));
+	memset(&mbox_out, 0, sizeof(mbox_out));
+	mbox_cmd = (struct cxl_mbox_cmd) {
+		.opcode = CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
+		.size_in = sizeof(mbox_in),
+		.payload_in = &mbox_in,
+		.size_out = sizeof(mbox_out),
+		.payload_out = &mbox_out,
+		.min_out = sizeof(mbox_out),
+	};
+	rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+	if (rc < 0)
+		return rc;
+
+	cxl_mbox->num_features = le16_to_cpu(mbox_out.supported_feats);
+
+	return 0;
+}
+
+int cxl_get_supported_features(struct cxl_dev_state *cxlds)
+{
+	struct cxl_mbox_get_sup_feats_out *mbox_out __free(kvfree) = NULL;
+	int remain_feats, max_size, max_feats, start, rc;
+	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
+	int feat_size = sizeof(struct cxl_feat_entry);
+	struct cxl_mbox_get_sup_feats_in mbox_in;
+	int hdr_size = sizeof(*mbox_out);
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_mem_command *cmd;
+	struct cxl_feat_entry *entry;
+
+	/* Get supported features is optional, need to check */
+	cmd = cxl_mem_find_command(CXL_MBOX_OP_GET_SUPPORTED_FEATURES);
+	if (!cmd)
+		return -EOPNOTSUPP;
+	if (!test_bit(cmd->info.id, cxl_mbox->enabled_cmds))
+		return -EOPNOTSUPP;
+
+	rc = cxl_get_supported_features_count(cxlds);
+	if (rc)
+		return rc;
+
+	if (!cxl_mbox->num_features) {
+		dev_dbg(cxl_mbox->host, "No CXL features enumerated.\n");
+		return 0;
+	}
+
+	struct cxl_feat_entry *entries __free(kvfree) =
+		kvmalloc(cxl_mbox->num_features * feat_size, GFP_KERNEL);
+
+	if (!entries)
+		return -ENOMEM;
+
+	max_size = cxl_mbox->payload_size - hdr_size;
+	/* max feat entries that can fit in mailbox max payload size */
+	max_feats = max_size / feat_size;
+	entry = &entries[0];
+
+	mbox_out = kvmalloc(cxl_mbox->payload_size, GFP_KERNEL);
+	if (!mbox_out)
+		return -ENOMEM;
+
+	start = 0;
+	remain_feats = cxl_mbox->num_features;
+	do {
+		int retrieved, alloc_size, copy_feats;
+		int num_entries;
+
+		if (remain_feats > max_feats) {
+			alloc_size = sizeof(*mbox_out) + max_feats * feat_size;
+			remain_feats = remain_feats - max_feats;
+			copy_feats = max_feats;
+		} else {
+			alloc_size = sizeof(*mbox_out) + remain_feats * feat_size;
+			copy_feats = remain_feats;
+			remain_feats = 0;
+		}
+
+		memset(&mbox_in, 0, sizeof(mbox_in));
+		mbox_in.count = cpu_to_le32(alloc_size);
+		mbox_in.start_idx = cpu_to_le16(start);
+		memset(mbox_out, 0, alloc_size);
+		mbox_cmd = (struct cxl_mbox_cmd) {
+			.opcode = CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
+			.size_in = sizeof(mbox_in),
+			.payload_in = &mbox_in,
+			.size_out = alloc_size,
+			.payload_out = mbox_out,
+			.min_out = hdr_size,
+		};
+		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+		if (rc < 0)
+			return rc;
+
+		if (mbox_cmd.size_out <= hdr_size) {
+			rc = -ENXIO;
+			goto err;
+		}
+
+		/*
+		 * Make sure retrieved out buffer is multiple of feature
+		 * entries.
+		 */
+		retrieved = mbox_cmd.size_out - hdr_size;
+		if (retrieved % feat_size) {
+			rc = -ENXIO;
+			goto err;
+		}
+
+		num_entries = le16_to_cpu(mbox_out->num_entries);
+		/*
+		 * If the reported output entries * defined entry size !=
+		 * retrieved output bytes, then the output package is incorrect.
+		 */
+		if (num_entries * feat_size != retrieved) {
+			rc = -ENXIO;
+			goto err;
+		}
+
+		memcpy(entry, mbox_out->ents, retrieved);
+		entry++;
+		/*
+		 * If the number of output entries is less than expected, add the
+		 * remaining entries to the next batch.
+		 */
+		remain_feats += copy_feats - num_entries;
+		start += num_entries;
+	} while (remain_feats);
+
+	cxl_mbox->entries = no_free_ptr(entries);
+	rc = devm_add_action_or_reset(cxl_mbox->host, cxl_free_features,
+				      cxl_mbox->entries);
+	if (rc)
+		return rc;
+
+	return 0;
+
+err:
+	cxl_mbox->num_features = 0;
+	return rc;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_supported_features, CXL);
+
+int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t *feat_uuid,
+				    struct cxl_feat_entry *feat_entry_out)
+{
+	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
+	struct cxl_feat_entry *feat_entry;
+	int count;
+
+	/* Check CXL dev supports the feature */
+	feat_entry = &cxl_mbox->entries[0];
+	for (count = 0; count < cxl_mbox->num_features; count++, feat_entry++) {
+		if (uuid_equal(&feat_entry->uuid, feat_uuid)) {
+			memcpy(feat_entry_out, feat_entry, sizeof(*feat_entry_out));
+			return 0;
+		}
+	}
+
+	return -EOPNOTSUPP;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_supported_feature_entry, CXL);
+
 /**
  * cxl_enumerate_cmds() - Enumerate commands for a device.
  * @mds: The driver data for the operation
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index a0a49809cd76..6685dd76985a 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -490,6 +490,7 @@ enum cxl_opcode {
 	CXL_MBOX_OP_GET_LOG_CAPS	= 0x0402,
 	CXL_MBOX_OP_CLEAR_LOG           = 0x0403,
 	CXL_MBOX_OP_GET_SUP_LOG_SUBLIST = 0x0405,
+	CXL_MBOX_OP_GET_SUPPORTED_FEATURES	= 0x0500,
 	CXL_MBOX_OP_IDENTIFY		= 0x4000,
 	CXL_MBOX_OP_GET_PARTITION_INFO	= 0x4100,
 	CXL_MBOX_OP_SET_PARTITION_INFO	= 0x4101,
@@ -773,6 +774,32 @@ enum {
 	CXL_PMEM_SEC_PASS_USER,
 };
 
+/* Get Supported Features (0x500h) CXL r3.1 8.2.9.6.1 */
+struct cxl_mbox_get_sup_feats_in {
+	__le32 count;
+	__le16 start_idx;
+	u8 reserved[2];
+} __packed;
+
+struct cxl_feat_entry {
+	uuid_t uuid;
+	__le16 id;
+	__le16 get_feat_size;
+	__le16 set_feat_size;
+	__le32 flags;
+	u8 get_feat_ver;
+	u8 set_feat_ver;
+	__le16 set_effects;
+	u8 reserved[18];
+} __packed;
+
+struct cxl_mbox_get_sup_feats_out {
+	__le16 num_entries;
+	__le16 supported_feats;
+	u8 reserved[4];
+	struct cxl_feat_entry ents[] __counted_by_le(supported_feats);
+} __packed;
+
 int cxl_internal_send_cmd(struct cxl_mailbox *cxl_mbox,
 			  struct cxl_mbox_cmd *cmd);
 int cxl_dev_state_identify(struct cxl_memdev_state *mds);
@@ -832,4 +859,8 @@ struct cxl_hdm {
 struct seq_file;
 struct dentry *cxl_debugfs_create_dir(const char *dir);
 void cxl_dpa_debug(struct seq_file *file, struct cxl_dev_state *cxlds);
+
+int cxl_get_supported_features(struct cxl_dev_state *cxlds);
+int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t *feat_uuid,
+				    struct cxl_feat_entry *feat_entry_out);
 #endif /* __CXL_MEM_H__ */
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 188412d45e0d..cbb86ecf0e2f 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -887,6 +887,10 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
+	rc = cxl_get_supported_features(cxlds);
+	if (rc)
+		dev_dbg(&pdev->dev, "No features enumerated.\n");
+
 	rc = cxl_set_timestamp(mds);
 	if (rc)
 		return rc;
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index cc894f07a435..03c4b8ad84c1 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -50,6 +50,8 @@ struct cxl_mbox_cmd {
  *                (CXL 3.1 8.2.8.4.3 Mailbox Capabilities Register)
  * @mbox_mutex: mutex protects device mailbox and firmware
  * @mbox_wait: rcuwait for mailbox
+ * @num_features: number of supported features entries
+ * @features: list of supported feature entries
  * @mbox_send: @dev specific transport for transmitting mailbox commands
  */
 struct cxl_mailbox {
@@ -59,6 +61,8 @@ struct cxl_mailbox {
 	size_t payload_size;
 	struct mutex mbox_mutex; /* lock to protect mailbox context */
 	struct rcuwait mbox_wait;
+	int num_features;
+	struct cxl_feat_entry *entries;
 	int (*mbox_send)(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd *cmd);
 };
 
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index c6c0fe27495d..bd2535962f70 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -50,6 +50,7 @@
 	___C(GET_LOG_CAPS, "Get Log Capabilities"),			  \
 	___C(CLEAR_LOG, "Clear Log"),					  \
 	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
+	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),		  \
 	___C(MAX, "invalid / last command")
 
 #define ___C(a, b) CXL_MEM_COMMAND_ID_##a
-- 
2.47.0

[RFC PATCH v2 03/20] cxl/test: Add Get Supported Features mailbox command support

[Dave Jiang]

Add cxl-test emulation of Get Supported Features mailbox command.
Currently only adding a test feature with feature identifier of
all f's for testing.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 tools/testing/cxl/test/mem.c | 69 ++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index 71916e0e1546..e0c7e49795ed 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -44,6 +44,10 @@ static struct cxl_cel_entry mock_cel[] = {
 		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_SUPPORTED_LOGS),
 		.effect = CXL_CMD_EFFECT_NONE,
 	},
+	{
+		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_SUPPORTED_FEATURES),
+		.effect = CXL_CMD_EFFECT_NONE,
+	},
 	{
 		.opcode = cpu_to_le16(CXL_MBOX_OP_IDENTIFY),
 		.effect = CXL_CMD_EFFECT_NONE,
@@ -1337,6 +1341,64 @@ static int mock_activate_fw(struct cxl_mockmem_data *mdata,
 	return -EINVAL;
 }
 
+#define CXL_VENDOR_FEATURE_TEST							\
+	UUID_INIT(0xffffffff, 0xffff, 0xffff, 0xff, 0xff, 0xff, 0xff, 0xff,	\
+		  0xff, 0xff, 0xff)
+
+static void fill_feature_vendor_test(struct cxl_feat_entry *feat)
+{
+	feat->uuid = CXL_VENDOR_FEATURE_TEST;
+	feat->id = 0;
+	feat->get_feat_size = cpu_to_le16(0x4);
+	feat->set_feat_size = cpu_to_le16(0x4);
+	feat->flags = cpu_to_le32(BIT(5));
+	feat->get_feat_ver = 1;
+	feat->set_feat_ver = 1;
+	feat->set_effects = cpu_to_le16(BIT(0) | BIT(9));
+}
+
+static int mock_get_supported_features(struct cxl_mockmem_data *mdata,
+				       struct cxl_mbox_cmd *cmd)
+{
+	struct cxl_mbox_get_sup_feats_in *in = cmd->payload_in;
+	struct cxl_mbox_get_sup_feats_out *out = cmd->payload_out;
+	struct cxl_feat_entry *feat;
+	u16 start_idx, count;
+
+	if (cmd->size_out < sizeof(*out)) {
+		cmd->return_code = CXL_MBOX_CMD_RC_PAYLOADLEN;
+		return -EINVAL;
+	}
+
+	/*
+	 * Current emulation only supports 1 feature
+	 */
+	start_idx = le16_to_cpu(in->start_idx);
+	if (start_idx != 0) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	count = le16_to_cpu(in->count);
+	if (count < sizeof(*out)) {
+		cmd->return_code = CXL_MBOX_CMD_RC_PAYLOADLEN;
+		return -EINVAL;
+	}
+
+	out->supported_feats = cpu_to_le16(1);
+	cmd->return_code = 0;
+	if (count < sizeof(*out) + sizeof(*feat)) {
+		out->num_entries = 0;
+		return 0;
+	}
+
+	out->num_entries = 1;
+	feat = out->ents;
+	fill_feature_vendor_test(feat);
+
+	return 0;
+}
+
 static int cxl_mock_mbox_send(struct cxl_mailbox *cxl_mbox,
 			      struct cxl_mbox_cmd *cmd)
 {
@@ -1422,6 +1484,9 @@ static int cxl_mock_mbox_send(struct cxl_mailbox *cxl_mbox,
 	case CXL_MBOX_OP_ACTIVATE_FW:
 		rc = mock_activate_fw(mdata, cmd);
 		break;
+	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
+		rc = mock_get_supported_features(mdata, cmd);
+		break;
 	default:
 		break;
 	}
@@ -1524,6 +1589,10 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 	if (rc)
 		return rc;
 
+	rc = cxl_get_supported_features(cxlds);
+	if (rc)
+		dev_dbg(dev, "No features enumerated\n");
+
 	rc = cxl_poison_state_init(mds);
 	if (rc)
 		return rc;
-- 
2.47.0

[RFC PATCH v2 04/20] cxl/mbox: Add GET_FEATURE mailbox command

[Dave Jiang]

From: Shiju Jose <shiju.jose@huawei.com>

Add support for GET_FEATURE mailbox command.

CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
The settings of a feature can be retrieved using Get Feature command.
CXL spec 3.1 section 8.2.9.6.2 describes Get Feature command.

Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
This patch is not needed by this series except the opcode enumeration.
---
 drivers/cxl/core/mbox.c | 41 +++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/cxlmem.h    | 26 ++++++++++++++++++++++++++
 2 files changed, 67 insertions(+)

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 4ba56f3d5a65..f70bd7d28ff8 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -968,6 +968,47 @@ int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t *f
 }
 EXPORT_SYMBOL_NS_GPL(cxl_get_supported_feature_entry, CXL);
 
+size_t cxl_get_feature(struct cxl_dev_state *cxlds, const uuid_t feat_uuid,
+		       enum cxl_get_feat_selection selection,
+		       void *feat_out, size_t feat_out_size)
+{
+	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
+	size_t data_to_rd_size, size_out;
+	struct cxl_mbox_get_feat_in pi;
+	struct cxl_mbox_cmd mbox_cmd;
+	size_t data_rcvd_size = 0;
+	int rc;
+
+	if (!feat_out || !feat_out_size)
+		return 0;
+
+	size_out = min(feat_out_size, cxl_mbox->payload_size);
+	pi.uuid = feat_uuid;
+	pi.selection = selection;
+	do {
+		data_to_rd_size = min(feat_out_size - data_rcvd_size,
+				      cxl_mbox->payload_size);
+		pi.offset = cpu_to_le16(data_rcvd_size);
+		pi.count = cpu_to_le16(data_to_rd_size);
+
+		mbox_cmd = (struct cxl_mbox_cmd) {
+			.opcode = CXL_MBOX_OP_GET_FEATURE,
+			.size_in = sizeof(pi),
+			.payload_in = &pi,
+			.size_out = size_out,
+			.payload_out = feat_out + data_rcvd_size,
+			.min_out = data_to_rd_size,
+		};
+		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+		if (rc < 0 || !mbox_cmd.size_out)
+			return 0;
+		data_rcvd_size += mbox_cmd.size_out;
+	} while (data_rcvd_size < feat_out_size);
+
+	return data_rcvd_size;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_feature, CXL);
+
 /**
  * cxl_enumerate_cmds() - Enumerate commands for a device.
  * @mds: The driver data for the operation
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 6685dd76985a..8fa3a817a0dd 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -491,6 +491,7 @@ enum cxl_opcode {
 	CXL_MBOX_OP_CLEAR_LOG           = 0x0403,
 	CXL_MBOX_OP_GET_SUP_LOG_SUBLIST = 0x0405,
 	CXL_MBOX_OP_GET_SUPPORTED_FEATURES	= 0x0500,
+	CXL_MBOX_OP_GET_FEATURE		= 0x0501,
 	CXL_MBOX_OP_IDENTIFY		= 0x4000,
 	CXL_MBOX_OP_GET_PARTITION_INFO	= 0x4100,
 	CXL_MBOX_OP_SET_PARTITION_INFO	= 0x4101,
@@ -800,6 +801,28 @@ struct cxl_mbox_get_sup_feats_out {
 	struct cxl_feat_entry ents[] __counted_by_le(supported_feats);
 } __packed;
 
+/*
+ * Get Feature CXL 3.1 Spec 8.2.9.6.2
+ */
+
+/*
+ * Get Feature input payload
+ * CXL rev 3.1 section 8.2.9.6.2 Table 8-99
+ */
+enum cxl_get_feat_selection {
+	CXL_GET_FEAT_SEL_CURRENT_VALUE,
+	CXL_GET_FEAT_SEL_DEFAULT_VALUE,
+	CXL_GET_FEAT_SEL_SAVED_VALUE,
+	CXL_GET_FEAT_SEL_MAX
+};
+
+struct cxl_mbox_get_feat_in {
+	uuid_t uuid;
+	__le16 offset;
+	__le16 count;
+	u8 selection;
+}  __packed;
+
 int cxl_internal_send_cmd(struct cxl_mailbox *cxl_mbox,
 			  struct cxl_mbox_cmd *cmd);
 int cxl_dev_state_identify(struct cxl_memdev_state *mds);
@@ -863,4 +886,7 @@ void cxl_dpa_debug(struct seq_file *file, struct cxl_dev_state *cxlds);
 int cxl_get_supported_features(struct cxl_dev_state *cxlds);
 int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t *feat_uuid,
 				    struct cxl_feat_entry *feat_entry_out);
+size_t cxl_get_feature(struct cxl_dev_state *cxlds, const uuid_t feat_uuid,
+		       enum cxl_get_feat_selection selection,
+		       void *feat_out, size_t feat_out_size);
 #endif /* __CXL_MEM_H__ */
-- 
2.47.0

[RFC PATCH v2 05/20] cxl: Add Get Feature command support for user submission

[Dave Jiang]

Add enumeration of Get Feature mailbox command for the kernel to recognize
the command being passed in from user space.

CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)

The feature requested is identified by specific UUID.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/mbox.c      | 1 +
 include/uapi/linux/cxl_mem.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index f70bd7d28ff8..f49665d3b4d6 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -68,6 +68,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
 	CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
 	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
 	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
+	CXL_CMD(GET_FEATURE, 0x15, CXL_VARIABLE_PAYLOAD, 0),
 };
 
 /*
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index bd2535962f70..90dcb9723997 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -51,6 +51,7 @@
 	___C(CLEAR_LOG, "Clear Log"),					  \
 	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
 	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),		  \
+	___C(GET_FEATURE, "Get Feature"),				  \
 	___C(MAX, "invalid / last command")
 
 #define ___C(a, b) CXL_MEM_COMMAND_ID_##a
-- 
2.47.0

[RFC PATCH v2 06/20] cxl/mbox: Add SET_FEATURE mailbox command

[Dave Jiang]

From: Shiju Jose <shiju.jose@huawei.com>

Add support for SET_FEATURE mailbox command.

CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
CXL devices supports features with changeable attributes.
The settings of a feature can be optionally modified using Set Feature
command.
CXL spec 3.1 section 8.2.9.6.3 describes Set Feature command.

Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
This patch is not needed except for the opcode enumeration.
---
 drivers/cxl/core/mbox.c | 73 +++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/cxlmem.h    | 34 +++++++++++++++++++
 2 files changed, 107 insertions(+)

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index f49665d3b4d6..4b9abf9a5b2b 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -1010,6 +1010,79 @@ size_t cxl_get_feature(struct cxl_dev_state *cxlds, const uuid_t feat_uuid,
 }
 EXPORT_SYMBOL_NS_GPL(cxl_get_feature, CXL);
 
+/*
+ * FEAT_DATA_MIN_PAYLOAD_SIZE - min extra number of bytes should be
+ * available in the mailbox for storing the actual feature data so that
+ * the feature data transfer would work as expected.
+ */
+#define FEAT_DATA_MIN_PAYLOAD_SIZE 10
+int cxl_set_feature(struct cxl_dev_state *cxlds,
+		    const uuid_t feat_uuid, u8 feat_version,
+		    void *feat_data, size_t feat_data_size,
+		    u8 feat_flag)
+{
+	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
+	struct cxl_memdev_set_feat_pi {
+		struct cxl_mbox_set_feat_hdr hdr;
+		u8 feat_data[];
+	}  __packed;
+	size_t data_in_size, data_sent_size = 0;
+	struct cxl_mbox_cmd mbox_cmd;
+	size_t hdr_size;
+	int rc = 0;
+
+	struct cxl_memdev_set_feat_pi *pi __free(kfree) =
+					kmalloc(cxl_mbox->payload_size, GFP_KERNEL);
+	pi->hdr.uuid = feat_uuid;
+	pi->hdr.version = feat_version;
+	feat_flag &= ~CXL_SET_FEAT_FLAG_DATA_TRANSFER_MASK;
+	feat_flag |= CXL_SET_FEAT_FLAG_DATA_SAVED_ACROSS_RESET;
+	hdr_size = sizeof(pi->hdr);
+	/*
+	 * Check minimum mbox payload size is available for
+	 * the feature data transfer.
+	 */
+	if (hdr_size + FEAT_DATA_MIN_PAYLOAD_SIZE > cxl_mbox->payload_size)
+		return -ENOMEM;
+
+	if ((hdr_size + feat_data_size) <= cxl_mbox->payload_size) {
+		pi->hdr.flags = cpu_to_le32(feat_flag |
+				       CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER);
+		data_in_size = feat_data_size;
+	} else {
+		pi->hdr.flags = cpu_to_le32(feat_flag |
+				       CXL_SET_FEAT_FLAG_INITIATE_DATA_TRANSFER);
+		data_in_size = cxl_mbox->payload_size - hdr_size;
+	}
+
+	do {
+		pi->hdr.offset = cpu_to_le16(data_sent_size);
+		memcpy(pi->feat_data, feat_data + data_sent_size, data_in_size);
+		mbox_cmd = (struct cxl_mbox_cmd) {
+			.opcode = CXL_MBOX_OP_SET_FEATURE,
+			.size_in = hdr_size + data_in_size,
+			.payload_in = pi,
+		};
+		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+		if (rc < 0)
+			return rc;
+
+		data_sent_size += data_in_size;
+		if (data_sent_size >= feat_data_size)
+			return 0;
+
+		if ((feat_data_size - data_sent_size) <= (cxl_mbox->payload_size - hdr_size)) {
+			data_in_size = feat_data_size - data_sent_size;
+			pi->hdr.flags = cpu_to_le32(feat_flag |
+					       CXL_SET_FEAT_FLAG_FINISH_DATA_TRANSFER);
+		} else {
+			pi->hdr.flags = cpu_to_le32(feat_flag |
+					       CXL_SET_FEAT_FLAG_CONTINUE_DATA_TRANSFER);
+		}
+	} while (true);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_set_feature, CXL);
+
 /**
  * cxl_enumerate_cmds() - Enumerate commands for a device.
  * @mds: The driver data for the operation
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 8fa3a817a0dd..2c15817c80a5 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -492,6 +492,7 @@ enum cxl_opcode {
 	CXL_MBOX_OP_GET_SUP_LOG_SUBLIST = 0x0405,
 	CXL_MBOX_OP_GET_SUPPORTED_FEATURES	= 0x0500,
 	CXL_MBOX_OP_GET_FEATURE		= 0x0501,
+	CXL_MBOX_OP_SET_FEATURE		= 0x0502,
 	CXL_MBOX_OP_IDENTIFY		= 0x4000,
 	CXL_MBOX_OP_GET_PARTITION_INFO	= 0x4100,
 	CXL_MBOX_OP_SET_PARTITION_INFO	= 0x4101,
@@ -823,6 +824,35 @@ struct cxl_mbox_get_feat_in {
 	u8 selection;
 }  __packed;
 
+/*
+ * Set Feature CXL 3.1 Spec 8.2.9.6.3
+ */
+
+/*
+ * Set Feature input payload
+ * CXL rev 3.1 section 8.2.9.6.3 Table 8-101
+ */
+/* Set Feature : Payload in flags */
+#define CXL_SET_FEAT_FLAG_DATA_TRANSFER_MASK	GENMASK(2, 0)
+enum cxl_set_feat_flag_data_transfer {
+	CXL_SET_FEAT_FLAG_FULL_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_INITIATE_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_CONTINUE_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_FINISH_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_ABORT_DATA_TRANSFER,
+	CXL_SET_FEAT_FLAG_DATA_TRANSFER_MAX
+};
+
+#define CXL_SET_FEAT_FLAG_DATA_SAVED_ACROSS_RESET	BIT(3)
+
+struct cxl_mbox_set_feat_hdr {
+	uuid_t uuid;
+	__le32 flags;
+	__le16 offset;
+	u8 version;
+	u8 rsvd[9];
+}  __packed;
+
 int cxl_internal_send_cmd(struct cxl_mailbox *cxl_mbox,
 			  struct cxl_mbox_cmd *cmd);
 int cxl_dev_state_identify(struct cxl_memdev_state *mds);
@@ -889,4 +919,8 @@ int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t *f
 size_t cxl_get_feature(struct cxl_dev_state *cxlds, const uuid_t feat_uuid,
 		       enum cxl_get_feat_selection selection,
 		       void *feat_out, size_t feat_out_size);
+int cxl_set_feature(struct cxl_dev_state *cxlds,
+		    const uuid_t feat_uuid, u8 feat_version,
+		    void *feat_data, size_t feat_data_size,
+		    u8 feat_flag);
 #endif /* __CXL_MEM_H__ */
-- 
2.47.0

[RFC PATCH v2 07/20] cxl: Add Set Feature command support for user submission

[Dave Jiang]

Add enumeration of Set Feature mailbox command for the kernel to recognize
the command being passed in from user space.

CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)

The feature requested is identified by specific UUID.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/mbox.c      | 1 +
 include/uapi/linux/cxl_mem.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 4b9abf9a5b2b..739444d34130 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -69,6 +69,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
 	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
 	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
 	CXL_CMD(GET_FEATURE, 0x15, CXL_VARIABLE_PAYLOAD, 0),
+	CXL_CMD(SET_FEATURE, CXL_VARIABLE_PAYLOAD, 0, 0),
 };
 
 /*
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 90dcb9723997..469dd481795f 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -52,6 +52,7 @@
 	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
 	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),		  \
 	___C(GET_FEATURE, "Get Feature"),				  \
+	___C(SET_FEATURE, "Set Feature"),				  \
 	___C(MAX, "invalid / last command")
 
 #define ___C(a, b) CXL_MEM_COMMAND_ID_##a
-- 
2.47.0

[RFC PATCH v2 08/20] cxl: Move cxl_driver related bits to be usable by external drivers

[Dave Jiang]

Move the code related to cxl_driver to allow external drivers such as
cxl_fwctl to utilize the cxl_bus.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/cxl.h | 32 +-------------------------------
 include/cxl/cxl.h | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+), 31 deletions(-)
 create mode 100644 include/cxl/cxl.h

diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 5406e3ab3d4a..18e78b2c3612 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -11,6 +11,7 @@
 #include <linux/log2.h>
 #include <linux/node.h>
 #include <linux/io.h>
+#include <cxl/cxl.h>
 
 extern const struct nvdimm_security_ops *cxl_security_ops;
 
@@ -819,37 +820,6 @@ bool is_cxl_region(struct device *dev);
 
 extern struct bus_type cxl_bus_type;
 
-struct cxl_driver {
-	const char *name;
-	int (*probe)(struct device *dev);
-	void (*remove)(struct device *dev);
-	struct device_driver drv;
-	int id;
-};
-
-#define to_cxl_drv(__drv)	container_of_const(__drv, struct cxl_driver, drv)
-
-int __cxl_driver_register(struct cxl_driver *cxl_drv, struct module *owner,
-			  const char *modname);
-#define cxl_driver_register(x) __cxl_driver_register(x, THIS_MODULE, KBUILD_MODNAME)
-void cxl_driver_unregister(struct cxl_driver *cxl_drv);
-
-#define module_cxl_driver(__cxl_driver) \
-	module_driver(__cxl_driver, cxl_driver_register, cxl_driver_unregister)
-
-#define CXL_DEVICE_NVDIMM_BRIDGE	1
-#define CXL_DEVICE_NVDIMM		2
-#define CXL_DEVICE_PORT			3
-#define CXL_DEVICE_ROOT			4
-#define CXL_DEVICE_MEMORY_EXPANDER	5
-#define CXL_DEVICE_REGION		6
-#define CXL_DEVICE_PMEM_REGION		7
-#define CXL_DEVICE_DAX_REGION		8
-#define CXL_DEVICE_PMU			9
-
-#define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
-#define CXL_MODALIAS_FMT "cxl:t%d"
-
 struct cxl_nvdimm_bridge *to_cxl_nvdimm_bridge(struct device *dev);
 struct cxl_nvdimm_bridge *devm_cxl_add_nvdimm_bridge(struct device *host,
 						     struct cxl_port *port);
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
new file mode 100644
index 000000000000..07a2983275a0
--- /dev/null
+++ b/include/cxl/cxl.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2024 Intel Corporation. */
+#ifndef __CXL_GLOBAL_H__
+#define __CXL_GLOBAL_H__
+
+struct cxl_driver {
+	const char *name;
+	int (*probe)(struct device *dev);
+	void (*remove)(struct device *dev);
+	struct device_driver drv;
+	int id;
+};
+
+#define to_cxl_drv(__drv)      container_of_const(__drv, struct cxl_driver, drv)
+
+int __cxl_driver_register(struct cxl_driver *cxl_drv, struct module *owner,
+			  const char *modname);
+#define cxl_driver_register(x) __cxl_driver_register(x, THIS_MODULE, KBUILD_MODNAME)
+void cxl_driver_unregister(struct cxl_driver *cxl_drv);
+
+#define module_cxl_driver(__cxl_driver) \
+	module_driver(__cxl_driver, cxl_driver_register, cxl_driver_unregister)
+
+#define CXL_DEVICE_NVDIMM_BRIDGE	1
+#define CXL_DEVICE_NVDIMM		2
+#define CXL_DEVICE_PORT			3
+#define CXL_DEVICE_ROOT			4
+#define CXL_DEVICE_MEMORY_EXPANDER	5
+#define CXL_DEVICE_REGION		6
+#define CXL_DEVICE_PMEM_REGION		7
+#define CXL_DEVICE_DAX_REGION		8
+#define CXL_DEVICE_PMU			9
+
+#define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
+#define CXL_MODALIAS_FMT "cxl:t%d"
+
+#endif
-- 
2.47.0

[RFC PATCH v2 09/20] fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands

[Dave Jiang]

Add a cxl fwctl driver to allow sending of CXL feature
commands from userspace through as ioctls. Create a driver skeleton for
initial setup.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v2:
- Move to cxl_bus from auxiliary_bus (Jonathan)
---
 MAINTAINERS                |  8 ++++
 drivers/cxl/core/mbox.c    | 41 +++++++++++++++-
 drivers/fwctl/Kconfig      |  9 ++++
 drivers/fwctl/Makefile     |  1 +
 drivers/fwctl/cxl/Makefile |  4 ++
 drivers/fwctl/cxl/cxl.c    | 97 ++++++++++++++++++++++++++++++++++++++
 include/cxl/cxl.h          |  1 +
 include/cxl/mailbox.h      | 17 +++++++
 include/uapi/fwctl/fwctl.h |  1 +
 9 files changed, 178 insertions(+), 1 deletion(-)
 create mode 100644 drivers/fwctl/cxl/Makefile
 create mode 100644 drivers/fwctl/cxl/cxl.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 93f74bc58d99..6b52c4168b94 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9389,6 +9389,14 @@ L:	linux-kernel@vger.kernel.org
 S:	Maintained
 F:	drivers/fwctl/mlx5/
 
+FWCTL CXL DRIVER
+M:	Dave Jiang <dave.jiang@intel.com>
+R:	Dan Williams <dan.j.williams@intel.com>
+R:	Jonathan Cameron <jonathan.cameron@huawei.com>
+L:	linux-cxl@vger.kernel.org
+S:	Maintained
+F:	drivers/fwctl/cxl/
+
 GALAXYCORE GC0308 CAMERA SENSOR DRIVER
 M:	Sebastian Reichel <sre@kernel.org>
 L:	linux-media@vger.kernel.org
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 739444d34130..12758e763650 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -1712,8 +1712,29 @@ int cxl_poison_state_init(struct cxl_memdev_state *mds)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_poison_state_init, CXL);
 
+static void cxl_fwctl_release(struct device *dev)
+{
+	struct cxl_fwctl *fwctl = to_cxl_fwctl(dev);
+
+	kfree(fwctl);
+}
+
+static void remove_fwctl_dev(void *dev)
+{
+	device_unregister(dev);
+}
+
+static const struct device_type cxl_fwctl_type = {
+	.name = "cxl_fwctl",
+	.release = cxl_fwctl_release,
+};
+
 int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host)
 {
+	struct cxl_fwctl *fwctl __free(kfree) = kzalloc(sizeof(*fwctl), GFP_KERNEL);
+	struct device *dev;
+	int rc;
+
 	if (!cxl_mbox || !host)
 		return -EINVAL;
 
@@ -1721,7 +1742,25 @@ int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host)
 	mutex_init(&cxl_mbox->mbox_mutex);
 	rcuwait_init(&cxl_mbox->mbox_wait);
 
-	return 0;
+	fwctl->cxl_mbox = cxl_mbox;
+	dev = &fwctl->dev;
+	device_initialize(dev);
+	device_set_pm_not_required(dev);
+	dev->parent = host;
+	dev->bus = &cxl_bus_type;
+	dev->type = &cxl_fwctl_type;
+
+	rc = device_add(dev);
+	if (rc)
+		goto err;
+
+	cxl_mbox->fwctl = no_free_ptr(fwctl);
+
+	return devm_add_action_or_reset(host, remove_fwctl_dev, dev);
+
+err:
+	put_device(&fwctl->dev);
+	return rc;
 }
 EXPORT_SYMBOL_NS_GPL(cxl_mailbox_init, CXL);
 
diff --git a/drivers/fwctl/Kconfig b/drivers/fwctl/Kconfig
index e5ee2d46d431..e49903a9d0d3 100644
--- a/drivers/fwctl/Kconfig
+++ b/drivers/fwctl/Kconfig
@@ -19,5 +19,14 @@ config FWCTL_MLX5
 	  This will allow configuration and debug tools to work out of the box on
 	  mainstream kernel.
 
+	  If you don't know what to do here, say N.
+
+config FWCTL_CXL
+	tristate "CXL fwctl driver"
+	depends on CXL_BUS
+	help
+	  CXLCTL provides interface for the user process to access user allowed
+	  mailbox commands for CXL device.
+
 	  If you don't know what to do here, say N.
 endif
diff --git a/drivers/fwctl/Makefile b/drivers/fwctl/Makefile
index 1c535f694d7f..bd356e6f2e5a 100644
--- a/drivers/fwctl/Makefile
+++ b/drivers/fwctl/Makefile
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_FWCTL) += fwctl.o
 obj-$(CONFIG_FWCTL_MLX5) += mlx5/
+obj-$(CONFIG_FWCTL_CXL) += cxl/
 
 fwctl-y += main.o
diff --git a/drivers/fwctl/cxl/Makefile b/drivers/fwctl/cxl/Makefile
new file mode 100644
index 000000000000..623194521572
--- /dev/null
+++ b/drivers/fwctl/cxl/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_FWCTL_CXL) += cxl_fwctl.o
+
+cxl_fwctl-y += cxl.o
diff --git a/drivers/fwctl/cxl/cxl.c b/drivers/fwctl/cxl/cxl.c
new file mode 100644
index 000000000000..c6a11cbbd937
--- /dev/null
+++ b/drivers/fwctl/cxl/cxl.c
@@ -0,0 +1,97 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2024, Intel Corporation
+ */
+#include <linux/fwctl.h>
+#include <linux/device.h>
+#include <cxl/cxl.h>
+#include <cxl/mailbox.h>
+
+struct cxlctl_uctx {
+	struct fwctl_uctx uctx;
+	u32 nr_commands;
+};
+
+struct cxlctl_dev {
+	struct fwctl_device fwctl;
+	struct cxl_mailbox *mbox;
+};
+
+DEFINE_FREE(cxlctl, struct cxlctl_dev *, if (_T) fwctl_put(&_T->fwctl))
+
+static int cxlctl_open_uctx(struct fwctl_uctx *uctx)
+{
+	return 0;
+}
+
+static void cxlctl_close_uctx(struct fwctl_uctx *uctx)
+{
+}
+
+static void *cxlctl_info(struct fwctl_uctx *uctx, size_t *length)
+{
+	/* Place holder */
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
+static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
+			   void *rpc_in, size_t in_len, size_t *out_len)
+{
+	/* Place holder */
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
+static const struct fwctl_ops cxlctl_ops = {
+	.device_type = FWCTL_DEVICE_TYPE_CXL,
+	.uctx_size = sizeof(struct cxlctl_uctx),
+	.open_uctx = cxlctl_open_uctx,
+	.close_uctx = cxlctl_close_uctx,
+	.info = cxlctl_info,
+	.fw_rpc = cxlctl_fw_rpc,
+};
+
+static int cxlctl_probe(struct device *dev)
+{
+	struct cxl_fwctl *cxl_fwctl = to_cxl_fwctl(dev);
+	struct cxl_mailbox *cxl_mbox = cxl_fwctl->cxl_mbox;
+	struct cxlctl_dev *cxlctl __free(cxlctl) =
+		fwctl_alloc_device(cxl_mbox->host, &cxlctl_ops,
+				   struct cxlctl_dev, fwctl);
+	int rc;
+
+	if (!cxlctl)
+		return -ENOMEM;
+
+	cxlctl->mbox = cxl_mbox;
+
+	rc = fwctl_register(&cxlctl->fwctl);
+	if (rc)
+		return rc;
+
+	dev_set_drvdata(dev, no_free_ptr(cxlctl));
+
+	return 0;
+}
+
+static void cxlctl_remove(struct device *dev)
+{
+	struct cxlctl_dev *ctldev  = dev_get_drvdata(dev);
+
+	fwctl_unregister(&ctldev->fwctl);
+}
+
+static struct cxl_driver cxl_fwctl_driver = {
+	.name = "cxl_fwctl",
+	.probe = cxlctl_probe,
+	.remove = cxlctl_remove,
+	.id = CXL_DEVICE_FWCTL,
+};
+
+module_cxl_driver(cxl_fwctl_driver);
+
+MODULE_IMPORT_NS(CXL);
+MODULE_IMPORT_NS(FWCTL);
+MODULE_DESCRIPTION("CXL fwctl driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_CXL(CXL_DEVICE_FWCTL);
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index 07a2983275a0..6bfd7942a3f7 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -30,6 +30,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);
 #define CXL_DEVICE_PMEM_REGION		7
 #define CXL_DEVICE_DAX_REGION		8
 #define CXL_DEVICE_PMU			9
+#define CXL_DEVICE_FWCTL		10
 
 #define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
 #define CXL_MODALIAS_FMT "cxl:t%d"
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index 03c4b8ad84c1..16d21f63464c 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -3,6 +3,7 @@
 #ifndef __CXL_MBOX_H__
 #define __CXL_MBOX_H__
 #include <linux/rcuwait.h>
+#include <linux/auxiliary_bus.h>
 #include <uapi/linux/cxl_mem.h>
 
 /**
@@ -41,8 +42,23 @@ struct cxl_mbox_cmd {
 	u16 return_code;
 };
 
+struct cxl_mailbox;
+
+/**
+ * struct cxl_fwctl - context for FWCTL
+ * @dev: device for fwctl
+ * @cxl_mbox: pointer to cxl mailbox context
+ */
+struct cxl_fwctl {
+	struct device dev;
+	struct cxl_mailbox *cxl_mbox;
+};
+
+#define to_cxl_fwctl(dev) container_of(dev, struct cxl_fwctl, dev)
+
 /**
  * struct cxl_mailbox - context for CXL mailbox operations
+ * @fwctl: points to fwctl context
  * @host: device that hosts the mailbox
  * @enabled_cmds: mailbox commands that are enabled by the driver
  * @exclusive_cmds: mailbox commands that are exclusive to the kernel
@@ -55,6 +71,7 @@ struct cxl_mbox_cmd {
  * @mbox_send: @dev specific transport for transmitting mailbox commands
  */
 struct cxl_mailbox {
+	struct cxl_fwctl *fwctl;
 	struct device *host;
 	DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
 	DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
index f9b27fb5c161..4e4d30104667 100644
--- a/include/uapi/fwctl/fwctl.h
+++ b/include/uapi/fwctl/fwctl.h
@@ -43,6 +43,7 @@ enum {
 enum fwctl_device_type {
 	FWCTL_DEVICE_TYPE_ERROR = 0,
 	FWCTL_DEVICE_TYPE_MLX5 = 1,
+	FWCTL_DEVICE_TYPE_CXL = 2,
 };
 
 /**
-- 
2.47.0

[RFC PATCH v2 10/20] fwctl/cxl: Add support for get driver information

[Dave Jiang]

Add definition for fwctl_ops->info() to return driver information. The
function will return the number of device mailbox commands supported by the
fwctl char device.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v2:
- Change driver info to report number of commands supported by the driver and device.
---
 drivers/cxl/core/mbox.c  | 46 ++++++++++++++++++++++++++++++++++++++++
 drivers/fwctl/cxl/cxl.c  | 23 ++++++++++++++++++--
 include/cxl/cxl.h        |  2 ++
 include/cxl/mailbox.h    |  1 +
 include/uapi/fwctl/cxl.h | 22 +++++++++++++++++++
 5 files changed, 92 insertions(+), 2 deletions(-)
 create mode 100644 include/uapi/fwctl/cxl.h

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 12758e763650..f464eb42f08a 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -118,6 +118,14 @@ static u8 security_command_sets[] = {
 	0x46, /* Security Passthrough */
 };
 
+#define FWCTL_CXL_MAX_COMMANDS 3
+/* Command set that is allowed with FWCTL-CXL */
+static u16 fwctl_command_sets[] = {
+	CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
+	CXL_MBOX_OP_GET_FEATURE,
+	CXL_MBOX_OP_SET_FEATURE,
+};
+
 static bool cxl_is_security_command(u16 opcode)
 {
 	int i;
@@ -527,6 +535,44 @@ static int cxl_validate_cmd_from_user(struct cxl_mbox_cmd *mbox_cmd,
 				 send_cmd->in.payload);
 }
 
+static struct cxl_mem_command *
+fwctl_cxl_find_command(struct cxl_mailbox *cxl_mbox, u16 opcode)
+{
+	struct cxl_command_info *info;
+	struct cxl_mem_command *cmd;
+
+	cmd = cxl_mem_find_command(opcode);
+	if (!cmd)
+		return NULL;
+
+	info = &cmd->info;
+	if (test_bit(info->id, cxl_mbox->enabled_cmds) &&
+	    !test_bit(info->id, cxl_mbox->exclusive_cmds))
+		return cmd;
+
+	return NULL;
+}
+
+/**
+ * cxl_mailbox_user_commands_supported() - Return number of user mailbox
+ *                                         commands supported.
+ * @cxl_mbox: cxl mailbox context
+ *
+ * Return: number of commands supported
+ */
+int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox)
+{
+	int nr_cmds = 0;
+
+	for (int i = 0; i < FWCTL_CXL_MAX_COMMANDS; i++) {
+		nr_cmds += !!fwctl_cxl_find_command(cxl_mbox,
+						    fwctl_command_sets[i]);
+	}
+
+	return nr_cmds;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_mailbox_user_commands_supported, CXL);
+
 int cxl_query_cmd(struct cxl_mailbox *cxl_mbox,
 		  struct cxl_mem_query_commands __user *q)
 {
diff --git a/drivers/fwctl/cxl/cxl.c b/drivers/fwctl/cxl/cxl.c
index c6a11cbbd937..5eb5eabf2bff 100644
--- a/drivers/fwctl/cxl/cxl.c
+++ b/drivers/fwctl/cxl/cxl.c
@@ -6,6 +6,7 @@
 #include <linux/device.h>
 #include <cxl/cxl.h>
 #include <cxl/mailbox.h>
+#include <uapi/fwctl/cxl.h>
 
 struct cxlctl_uctx {
 	struct fwctl_uctx uctx;
@@ -21,6 +22,15 @@ DEFINE_FREE(cxlctl, struct cxlctl_dev *, if (_T) fwctl_put(&_T->fwctl))
 
 static int cxlctl_open_uctx(struct fwctl_uctx *uctx)
 {
+	struct cxlctl_uctx *cxlctl_uctx =
+		container_of(uctx, struct cxlctl_uctx, uctx);
+	struct fwctl_device *fwctl = uctx->fwctl;
+	struct cxlctl_dev *cxlctl =
+		container_of(fwctl, struct cxlctl_dev, fwctl);
+
+	cxlctl_uctx->nr_commands =
+		cxl_mailbox_user_commands_supported(cxlctl->mbox);
+
 	return 0;
 }
 
@@ -30,8 +40,17 @@ static void cxlctl_close_uctx(struct fwctl_uctx *uctx)
 
 static void *cxlctl_info(struct fwctl_uctx *uctx, size_t *length)
 {
-	/* Place holder */
-	return ERR_PTR(-EOPNOTSUPP);
+	struct cxlctl_uctx *cxlctl_uctx =
+		container_of(uctx, struct cxlctl_uctx, uctx);
+	struct fwctl_info_cxl *info;
+
+	info = kzalloc(sizeof(*info), GFP_KERNEL);
+	if (!info)
+		return ERR_PTR(-ENOMEM);
+
+	info->nr_commands = cxlctl_uctx->nr_commands;
+
+	return info;
 }
 
 static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index 6bfd7942a3f7..d0939e3bcbc0 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -3,6 +3,8 @@
 #ifndef __CXL_GLOBAL_H__
 #define __CXL_GLOBAL_H__
 
+#include <linux/device.h>
+
 struct cxl_driver {
 	const char *name;
 	int (*probe)(struct device *dev);
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index 16d21f63464c..af01fb78ae10 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -84,5 +84,6 @@ struct cxl_mailbox {
 };
 
 int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host);
+int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox);
 
 #endif
diff --git a/include/uapi/fwctl/cxl.h b/include/uapi/fwctl/cxl.h
new file mode 100644
index 000000000000..a32c4c752db6
--- /dev/null
+++ b/include/uapi/fwctl/cxl.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (c) 2024, Intel Corporation
+ *
+ * These are definitions for the mailbox command interface of CXL subsystem.
+ */
+#ifndef _UAPI_FWCTL_CXL_H_
+#define _UAPI_FWCTL_CXL_H_
+
+#include <linux/types.h>
+
+/**
+ * struct fwctl_info_cxl - ioctl(FWCTL_INFO) out_device_data
+ * @uctx_caps: The number of commands the driver and device supports
+ *
+ * Return basic information about the FW interface available.
+ */
+struct fwctl_info_cxl {
+	__u32 nr_commands;
+};
+
+#endif
-- 
2.47.0

[RFC PATCH v2 11/20] fwctl: FWCTL_HW_INFO to return hardware information

[Dave Jiang]

Add an optional ioctl FWCTL_HW_INFO to pass command specific information
to user space. An array of 'struct fwctl_command_info' will be returned
from the ioctl. These commands are send to the driver via FWCTL_RPC call.
The command info struct contains the command id, the related hardware
opcode, input and output size for the command, and the effects the command
has if it's a write command.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/fwctl/main.c       | 33 ++++++++++++++++++++++
 include/linux/fwctl.h      |  6 ++++
 include/uapi/fwctl/fwctl.h | 58 +++++++++++++++++++++++++++++++++++++-
 3 files changed, 96 insertions(+), 1 deletion(-)

diff --git a/drivers/fwctl/main.c b/drivers/fwctl/main.c
index 54a7356e586c..073e3d20558b 100644
--- a/drivers/fwctl/main.c
+++ b/drivers/fwctl/main.c
@@ -77,6 +77,38 @@ static int fwctl_cmd_info(struct fwctl_ucmd *ucmd)
 	return ucmd_respond(ucmd, sizeof(*cmd));
 }
 
+static int fwctl_hw_cmd_info(struct fwctl_ucmd *ucmd)
+{
+	struct fwctl_device *fwctl = ucmd->uctx->fwctl;
+	struct fwctl_hw_info *cmd = ucmd->cmd;
+	size_t out_len;
+	int nr_cmds;
+
+	if (cmd->flags)
+		return -EOPNOTSUPP;
+
+	nr_cmds = cmd->nr_cmds;
+	if (!nr_cmds)
+		return -EINVAL;
+
+	out_len = sizeof(struct fwctl_hw_info_out);
+	out_len += sizeof(struct fwctl_command_info) * nr_cmds;
+
+	if (cmd->device_data_len < out_len)
+		return -EINVAL;
+
+	void *outbuf __free(kvfree) = fwctl->ops->hw_info(ucmd->uctx, nr_cmds,
+							  &out_len);
+	if (IS_ERR(outbuf))
+		return PTR_ERR(outbuf);
+
+	if (copy_to_user(u64_to_user_ptr(cmd->out_device_data), outbuf, out_len))
+		return -EFAULT;
+
+	cmd->device_data_len = out_len;
+	return ucmd_respond(ucmd, sizeof(*cmd));
+}
+
 static int fwctl_cmd_rpc(struct fwctl_ucmd *ucmd)
 {
 	struct fwctl_device *fwctl = ucmd->uctx->fwctl;
@@ -156,6 +188,7 @@ struct fwctl_ioctl_op {
 	}
 static const struct fwctl_ioctl_op fwctl_ioctl_ops[] = {
 	IOCTL_OP(FWCTL_INFO, fwctl_cmd_info, struct fwctl_info, out_device_data),
+	IOCTL_OP(FWCTL_HW_INFO, fwctl_hw_cmd_info, struct fwctl_info, out_device_data),
 	IOCTL_OP(FWCTL_RPC, fwctl_cmd_rpc, struct fwctl_rpc, out),
 };
 
diff --git a/include/linux/fwctl.h b/include/linux/fwctl.h
index 6eac9497ff1a..ccef64da710c 100644
--- a/include/linux/fwctl.h
+++ b/include/linux/fwctl.h
@@ -47,6 +47,12 @@ struct fwctl_ops {
 	 * ignore length on input, the core code will handle everything.
 	 */
 	void *(*info)(struct fwctl_uctx *uctx, size_t *length);
+	/**
+	 * @hw_info: Implement FWCTL_HW_INFO. Return a kmalloc() memory that is
+	 * copied to out_device_dataa. On input commands indicates the number
+	 * of command info structs requested.
+	 */
+	void *(*hw_info)(struct fwctl_uctx *uctx, int commands, size_t *out_len);
 	/**
 	 * @fw_rpc: Implement FWCTL_RPC. Deliver rpc_in/in_len to the FW and
 	 * return the response and set out_len. rpc_in can be returned as the
diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
index 4e4d30104667..7334907e27c1 100644
--- a/include/uapi/fwctl/fwctl.h
+++ b/include/uapi/fwctl/fwctl.h
@@ -37,7 +37,8 @@
 enum {
 	FWCTL_CMD_BASE = 0,
 	FWCTL_CMD_INFO = 0,
-	FWCTL_CMD_RPC = 1,
+	FWCTL_CMD_HW_INFO = 1,
+	FWCTL_CMD_RPC = 2,
 };
 
 enum fwctl_device_type {
@@ -69,6 +70,61 @@ struct fwctl_info {
 };
 #define FWCTL_INFO _IO(FWCTL_TYPE, FWCTL_CMD_INFO)
 
+/**
+ * struct fwctl_command_info - Hardware command information returned from a query.
+ * @id: Driver ID number for the command
+ * @opcode: Hardware command opcode
+ * @set_effects: Effects to the OS and hardware when command is executed.
+ *               Provided by the device.
+ * @size_in: Expected input size, or ~0U if variable length.
+ * @size_out: Expected output size, or ~0U if variable length.
+ *
+ * Represents a single command that is supported by both the driver and the
+ * hardware. This is returned as part of an array from the FWCTL_HW_INFO ioctl.
+ */
+struct fwctl_command_info {
+	__u32 id;
+	__u16 opcode;
+	__u16 effects;
+	__u32 size_in;
+	__u32 size_out;
+};
+
+/**
+ * struct fwctl_hw_info_out - output struct for FWCTL_HW_INFO
+ * @nr_cmds: Number of commands for output
+ * @reserved: Reserved u32 for alignment.
+ * @commands: Array of 'struct fwctl_command_info'
+ */
+struct fwctl_hw_info_out {
+	__u32 nr_cmds;
+	__u32 reserved;
+
+	struct fwctl_command_info commands[] __counted_by(nr_cmds);
+};
+
+/**
+ * struct fwctl_hw_info - ioctl(FWCTL_HW_INFO)
+ * @size: sizeof(struct fwctl_hw_info)
+ * @flags: Must be 0
+ * @device_data_len: On input the length of the out_device_data memory. On
+ *	output the size of the kernel's device_data which may be larger or
+ *	smaller than the input. Maybe 0 on input.
+ * @nr_cmds: Number of commands requested. 0 returns only nr_cmds in output.
+ * @out_device_data: Pointer to a memory of device_data_len bytes. Kernel will
+ *	fill the entire memory, zeroing as required.
+ *
+ * Returns hardware commands information about this fwctl instance.
+ */
+struct fwctl_hw_info {
+	__u32 size;
+	__u32 flags;
+	__u32 device_data_len;
+	__u32 nr_cmds;
+	__aligned_u64 out_device_data;
+};
+#define FWCTL_HW_INFO _IO(FWCTL_TYPE, FWCTL_CMD_HW_INFO)
+
 /**
  * enum fwctl_rpc_scope - Scope of access for the RPC
  *
-- 
2.47.0

[RFC PATCH v2 12/20] cxl: Save Command Effects Log (CEL) effects for enabled commands

[Dave Jiang]

Store the command effects via what device provided via the CEL in the
'struct cxl_command_info' structure. Steal the upper 16 bits to store
the effects value in order to keep user API compatibility.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/mbox.c      | 2 ++
 include/uapi/fwctl/fwctl.h   | 4 ++--
 include/uapi/linux/cxl_mem.h | 5 +++--
 3 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index f464eb42f08a..fba6bdd30a82 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -795,6 +795,8 @@ static void cxl_walk_cel(struct cxl_memdev_state *mds, size_t size, u8 *cel)
 			enabled++;
 		}
 
+		cmd->info.effects = le16_to_cpu(cel_entry[i].effect);
+
 		dev_dbg(dev, "Opcode 0x%04x %s\n", opcode,
 			enabled ? "enabled" : "unsupported by driver");
 	}
diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
index 7334907e27c1..04da549cd4ae 100644
--- a/include/uapi/fwctl/fwctl.h
+++ b/include/uapi/fwctl/fwctl.h
@@ -74,8 +74,8 @@ struct fwctl_info {
  * struct fwctl_command_info - Hardware command information returned from a query.
  * @id: Driver ID number for the command
  * @opcode: Hardware command opcode
- * @set_effects: Effects to the OS and hardware when command is executed.
- *               Provided by the device.
+ * @effects: Effects to the OS and hardware when command is executed.
+ *           Provided by the device.
  * @size_in: Expected input size, or ~0U if variable length.
  * @size_out: Expected output size, or ~0U if variable length.
  *
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 469dd481795f..9dd37849c450 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -118,6 +118,7 @@ static const __u8 cxl_deprecated_commands[]
  *         the label-storage-area can not be written while the kernel is
  *         actively managing that space.
  *
+ * @effects: Command effects reported by the device.
  * @size_in: Expected input size, or ~0 if variable length.
  * @size_out: Expected output size, or ~0 if variable length.
  *
@@ -136,11 +137,11 @@ static const __u8 cxl_deprecated_commands[]
 struct cxl_command_info {
 	__u32 id;
 
-	__u32 flags;
+	__u16 flags;
 #define CXL_MEM_COMMAND_FLAG_MASK		GENMASK(1, 0)
 #define CXL_MEM_COMMAND_FLAG_ENABLED		BIT(0)
 #define CXL_MEM_COMMAND_FLAG_EXCLUSIVE		BIT(1)
-
+	__u16 effects;
 	__u32 size_in;
 	__u32 size_out;
 };
-- 
2.47.0

[RFC PATCH v2 13/20] fwctl/cxl: Add hw_info callback

[Dave Jiang]

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/mbox.c | 42 +++++++++++++++++++++++++++++++++++++++++
 drivers/fwctl/cxl/cxl.c | 25 ++++++++++++++++++++++++
 include/cxl/mailbox.h   |  2 ++
 3 files changed, 69 insertions(+)

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index fba6bdd30a82..65fceabb9fe7 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -9,6 +9,8 @@
 #include <cxlmem.h>
 #include <cxl.h>
 
+#include <uapi/fwctl/fwctl.h>
+
 #include "core.h"
 #include "trace.h"
 
@@ -573,6 +575,46 @@ int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_mailbox_user_commands_supported, CXL);
 
+/**
+ * cxl_mailbox_user_commands_info_get() - Retrieve array of command info
+ * @cxl_mbox: cxl mailbox context
+ * @nr_cmds: number of commands to retrieve
+ * @outbuf: Output buffer to store array 'struct fwctl_command_info'
+ * @out_len: size of final output buffer
+ *
+ * Return: 0 for success, or -errno for failure.
+ */
+int cxl_mailbox_user_commands_info_get(struct cxl_mailbox *cxl_mbox, int nr_cmds,
+				       void *outbuf, size_t *out_len)
+{
+	struct fwctl_command_info *entry;
+	struct cxl_command_info *info;
+	struct cxl_mem_command *cmd;
+
+	if (nr_cmds > FWCTL_CXL_MAX_COMMANDS)
+		return -EINVAL;
+
+	entry = outbuf;
+	for (int i = 0; i < nr_cmds; i++) {
+		cmd = fwctl_cxl_find_command(cxl_mbox, fwctl_command_sets[i]);
+		if (!cmd)
+			continue;
+		info = &cmd->info;
+		memset(entry, 0, sizeof(*entry));
+		entry->id = info->id;
+		entry->opcode = fwctl_command_sets[i];
+		entry->effects = info->effects;
+		entry->size_in = info->size_in;
+		entry->size_out = info->size_out;
+		entry++;
+	}
+
+	*out_len = sizeof(*entry) * nr_cmds;
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_mailbox_user_commands_info_get, CXL);
+
 int cxl_query_cmd(struct cxl_mailbox *cxl_mbox,
 		  struct cxl_mem_query_commands __user *q)
 {
diff --git a/drivers/fwctl/cxl/cxl.c b/drivers/fwctl/cxl/cxl.c
index 5eb5eabf2bff..ce8960a9beaa 100644
--- a/drivers/fwctl/cxl/cxl.c
+++ b/drivers/fwctl/cxl/cxl.c
@@ -53,6 +53,30 @@ static void *cxlctl_info(struct fwctl_uctx *uctx, size_t *length)
 	return info;
 }
 
+static void *cxlctl_hw_info(struct fwctl_uctx *uctx, int commands, size_t *out_len)
+{
+	struct cxlctl_uctx *cxlctl_uctx =
+		container_of(uctx, struct cxlctl_uctx, uctx);
+	struct fwctl_device *fwctl = uctx->fwctl;
+	struct cxlctl_dev *cxlctl =
+		container_of(fwctl, struct cxlctl_dev, fwctl);
+	int rc;
+
+	if (commands > cxlctl_uctx->nr_commands)
+		return ERR_PTR(-EINVAL);
+
+	void *out __free(kvfree) = kvzalloc(*out_len, GFP_KERNEL);
+	if (!out)
+		return ERR_PTR(-ENOMEM);
+
+	rc = cxl_mailbox_user_commands_info_get(cxlctl->mbox,
+						commands, out, out_len);
+	if (rc)
+		return ERR_PTR(rc);
+
+	return_ptr(out);
+}
+
 static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
 			   void *rpc_in, size_t in_len, size_t *out_len)
 {
@@ -66,6 +90,7 @@ static const struct fwctl_ops cxlctl_ops = {
 	.open_uctx = cxlctl_open_uctx,
 	.close_uctx = cxlctl_close_uctx,
 	.info = cxlctl_info,
+	.hw_info = cxlctl_hw_info,
 	.fw_rpc = cxlctl_fw_rpc,
 };
 
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index af01fb78ae10..f48eebb855f7 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -85,5 +85,7 @@ struct cxl_mailbox {
 
 int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host);
 int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox);
+int cxl_mailbox_user_commands_info_get(struct cxl_mailbox *cxl_mbox, int nr_cmds,
+				       void *outbuf, size_t *out_len);
 
 #endif
-- 
2.47.0

[RFC PATCH v2 14/20] cxl: Move defines and error codes from cxlmem.h to cxl/mailbox.h

[Dave Jiang]

Moving some internal definitions to cxl/mailbox.h in order to be accessed
by FWCTL CXL driver.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/cxlmem.h         |  83 ---------------------------
 include/cxl/mailbox.h        | 106 +++++++++++++++++++++++++++++++++++
 tools/testing/cxl/test/mem.c |   2 +-
 3 files changed, 107 insertions(+), 84 deletions(-)

diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 2c15817c80a5..890ed3199235 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -473,53 +473,6 @@ to_cxl_memdev_state(struct cxl_dev_state *cxlds)
 	return container_of(cxlds, struct cxl_memdev_state, cxlds);
 }
 
-enum cxl_opcode {
-	CXL_MBOX_OP_INVALID		= 0x0000,
-	CXL_MBOX_OP_RAW			= CXL_MBOX_OP_INVALID,
-	CXL_MBOX_OP_GET_EVENT_RECORD	= 0x0100,
-	CXL_MBOX_OP_CLEAR_EVENT_RECORD	= 0x0101,
-	CXL_MBOX_OP_GET_EVT_INT_POLICY	= 0x0102,
-	CXL_MBOX_OP_SET_EVT_INT_POLICY	= 0x0103,
-	CXL_MBOX_OP_GET_FW_INFO		= 0x0200,
-	CXL_MBOX_OP_TRANSFER_FW		= 0x0201,
-	CXL_MBOX_OP_ACTIVATE_FW		= 0x0202,
-	CXL_MBOX_OP_GET_TIMESTAMP	= 0x0300,
-	CXL_MBOX_OP_SET_TIMESTAMP	= 0x0301,
-	CXL_MBOX_OP_GET_SUPPORTED_LOGS	= 0x0400,
-	CXL_MBOX_OP_GET_LOG		= 0x0401,
-	CXL_MBOX_OP_GET_LOG_CAPS	= 0x0402,
-	CXL_MBOX_OP_CLEAR_LOG           = 0x0403,
-	CXL_MBOX_OP_GET_SUP_LOG_SUBLIST = 0x0405,
-	CXL_MBOX_OP_GET_SUPPORTED_FEATURES	= 0x0500,
-	CXL_MBOX_OP_GET_FEATURE		= 0x0501,
-	CXL_MBOX_OP_SET_FEATURE		= 0x0502,
-	CXL_MBOX_OP_IDENTIFY		= 0x4000,
-	CXL_MBOX_OP_GET_PARTITION_INFO	= 0x4100,
-	CXL_MBOX_OP_SET_PARTITION_INFO	= 0x4101,
-	CXL_MBOX_OP_GET_LSA		= 0x4102,
-	CXL_MBOX_OP_SET_LSA		= 0x4103,
-	CXL_MBOX_OP_GET_HEALTH_INFO	= 0x4200,
-	CXL_MBOX_OP_GET_ALERT_CONFIG	= 0x4201,
-	CXL_MBOX_OP_SET_ALERT_CONFIG	= 0x4202,
-	CXL_MBOX_OP_GET_SHUTDOWN_STATE	= 0x4203,
-	CXL_MBOX_OP_SET_SHUTDOWN_STATE	= 0x4204,
-	CXL_MBOX_OP_GET_POISON		= 0x4300,
-	CXL_MBOX_OP_INJECT_POISON	= 0x4301,
-	CXL_MBOX_OP_CLEAR_POISON	= 0x4302,
-	CXL_MBOX_OP_GET_SCAN_MEDIA_CAPS	= 0x4303,
-	CXL_MBOX_OP_SCAN_MEDIA		= 0x4304,
-	CXL_MBOX_OP_GET_SCAN_MEDIA	= 0x4305,
-	CXL_MBOX_OP_SANITIZE		= 0x4400,
-	CXL_MBOX_OP_SECURE_ERASE	= 0x4401,
-	CXL_MBOX_OP_GET_SECURITY_STATE	= 0x4500,
-	CXL_MBOX_OP_SET_PASSPHRASE	= 0x4501,
-	CXL_MBOX_OP_DISABLE_PASSPHRASE	= 0x4502,
-	CXL_MBOX_OP_UNLOCK		= 0x4503,
-	CXL_MBOX_OP_FREEZE_SECURITY	= 0x4504,
-	CXL_MBOX_OP_PASSPHRASE_SECURE_ERASE	= 0x4505,
-	CXL_MBOX_OP_MAX			= 0x10000
-};
-
 #define DEFINE_CXL_CEL_UUID                                                    \
 	UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96, 0xb1, 0x62,     \
 		  0x3b, 0x3f, 0x17)
@@ -717,30 +670,6 @@ struct cxl_mbox_clear_poison {
 	u8 write_data[CXL_POISON_LEN_MULT];
 } __packed;
 
-/**
- * struct cxl_mem_command - Driver representation of a memory device command
- * @info: Command information as it exists for the UAPI
- * @opcode: The actual bits used for the mailbox protocol
- * @flags: Set of flags effecting driver behavior.
- *
- *  * %CXL_CMD_FLAG_FORCE_ENABLE: In cases of error, commands with this flag
- *    will be enabled by the driver regardless of what hardware may have
- *    advertised.
- *
- * The cxl_mem_command is the driver's internal representation of commands that
- * are supported by the driver. Some of these commands may not be supported by
- * the hardware. The driver will use @info to validate the fields passed in by
- * the user then submit the @opcode to the hardware.
- *
- * See struct cxl_command_info.
- */
-struct cxl_mem_command {
-	struct cxl_command_info info;
-	enum cxl_opcode opcode;
-	u32 flags;
-#define CXL_CMD_FLAG_FORCE_ENABLE BIT(0)
-};
-
 #define CXL_PMEM_SEC_STATE_USER_PASS_SET	0x01
 #define CXL_PMEM_SEC_STATE_MASTER_PASS_SET	0x02
 #define CXL_PMEM_SEC_STATE_LOCKED		0x04
@@ -783,18 +712,6 @@ struct cxl_mbox_get_sup_feats_in {
 	u8 reserved[2];
 } __packed;
 
-struct cxl_feat_entry {
-	uuid_t uuid;
-	__le16 id;
-	__le16 get_feat_size;
-	__le16 set_feat_size;
-	__le32 flags;
-	u8 get_feat_ver;
-	u8 set_feat_ver;
-	__le16 set_effects;
-	u8 reserved[18];
-} __packed;
-
 struct cxl_mbox_get_sup_feats_out {
 	__le16 num_entries;
 	__le16 supported_feats;
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index f48eebb855f7..e753d5d1d708 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -4,6 +4,7 @@
 #define __CXL_MBOX_H__
 #include <linux/rcuwait.h>
 #include <linux/auxiliary_bus.h>
+#include <uapi/fwctl/cxl.h>
 #include <uapi/linux/cxl_mem.h>
 
 /**
@@ -83,6 +84,111 @@ struct cxl_mailbox {
 	int (*mbox_send)(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd *cmd);
 };
 
+enum cxl_opcode {
+	CXL_MBOX_OP_INVALID		= 0x0000,
+	CXL_MBOX_OP_RAW			= CXL_MBOX_OP_INVALID,
+	CXL_MBOX_OP_GET_EVENT_RECORD	= 0x0100,
+	CXL_MBOX_OP_CLEAR_EVENT_RECORD	= 0x0101,
+	CXL_MBOX_OP_GET_EVT_INT_POLICY	= 0x0102,
+	CXL_MBOX_OP_SET_EVT_INT_POLICY	= 0x0103,
+	CXL_MBOX_OP_GET_FW_INFO		= 0x0200,
+	CXL_MBOX_OP_TRANSFER_FW		= 0x0201,
+	CXL_MBOX_OP_ACTIVATE_FW		= 0x0202,
+	CXL_MBOX_OP_GET_TIMESTAMP	= 0x0300,
+	CXL_MBOX_OP_SET_TIMESTAMP	= 0x0301,
+	CXL_MBOX_OP_GET_SUPPORTED_LOGS	= 0x0400,
+	CXL_MBOX_OP_GET_LOG		= 0x0401,
+	CXL_MBOX_OP_GET_LOG_CAPS	= 0x0402,
+	CXL_MBOX_OP_CLEAR_LOG           = 0x0403,
+	CXL_MBOX_OP_GET_SUP_LOG_SUBLIST = 0x0405,
+	CXL_MBOX_OP_GET_SUPPORTED_FEATURES	= 0x0500,
+	CXL_MBOX_OP_GET_FEATURE		= 0x0501,
+	CXL_MBOX_OP_SET_FEATURE		= 0x0502,
+	CXL_MBOX_OP_IDENTIFY		= 0x4000,
+	CXL_MBOX_OP_GET_PARTITION_INFO	= 0x4100,
+	CXL_MBOX_OP_SET_PARTITION_INFO	= 0x4101,
+	CXL_MBOX_OP_GET_LSA		= 0x4102,
+	CXL_MBOX_OP_SET_LSA		= 0x4103,
+	CXL_MBOX_OP_GET_HEALTH_INFO	= 0x4200,
+	CXL_MBOX_OP_GET_ALERT_CONFIG	= 0x4201,
+	CXL_MBOX_OP_SET_ALERT_CONFIG	= 0x4202,
+	CXL_MBOX_OP_GET_SHUTDOWN_STATE	= 0x4203,
+	CXL_MBOX_OP_SET_SHUTDOWN_STATE	= 0x4204,
+	CXL_MBOX_OP_GET_POISON		= 0x4300,
+	CXL_MBOX_OP_INJECT_POISON	= 0x4301,
+	CXL_MBOX_OP_CLEAR_POISON	= 0x4302,
+	CXL_MBOX_OP_GET_SCAN_MEDIA_CAPS	= 0x4303,
+	CXL_MBOX_OP_SCAN_MEDIA		= 0x4304,
+	CXL_MBOX_OP_GET_SCAN_MEDIA	= 0x4305,
+	CXL_MBOX_OP_SANITIZE		= 0x4400,
+	CXL_MBOX_OP_SECURE_ERASE	= 0x4401,
+	CXL_MBOX_OP_GET_SECURITY_STATE	= 0x4500,
+	CXL_MBOX_OP_SET_PASSPHRASE	= 0x4501,
+	CXL_MBOX_OP_DISABLE_PASSPHRASE	= 0x4502,
+	CXL_MBOX_OP_UNLOCK		= 0x4503,
+	CXL_MBOX_OP_FREEZE_SECURITY	= 0x4504,
+	CXL_MBOX_OP_PASSPHRASE_SECURE_ERASE	= 0x4505,
+	CXL_MBOX_OP_MAX			= 0x10000
+};
+
+#define CXL_CMD_CONFIG_CHANGE_COLD_RESET	BIT(0)
+#define CXL_CMD_CONFIG_CHANGE_IMMEDIATE		BIT(1)
+#define CXL_CMD_DATA_CHANGE_IMMEDIATE		BIT(2)
+#define CXL_CMD_POLICY_CHANGE_IMMEDIATE		BIT(3)
+#define CXL_CMD_LOG_CHANGE_IMMEDIATE		BIT(4)
+#define CXL_CMD_SECURITY_STATE_CHANGE		BIT(5)
+#define CXL_CMD_BACKGROUND			BIT(6)
+#define CXL_CMD_BGCMD_ABORT_SUPPORTED		BIT(7)
+#define CXL_CMD_CONFIG_CHANGE_CONV_RESET	(BIT(9) | BIT(10))
+#define CXL_CMD_CONFIG_CHANGE_CXL_RESET		(BIT(9) | BIT(11))
+
+/**
+ * struct cxl_feat_entry - CXL Spec r3.1 Table 8-97
+ * @uuid: Feature identifier
+ * @id: Feature Index
+ * @get_feat_size: Get Feature Size
+ * @set_feat_size: Set Feature Size
+ * @flags: Attribute Flags
+ * @get_feat_ver: Get Feature Version
+ * @set_feat_ver: Set Feature Version
+ * @reserved: reserved, must be 0
+ */
+struct cxl_feat_entry {
+	uuid_t uuid;
+	__le16 id;
+	__le16 get_feat_size;
+	__le16 set_feat_size;
+	__le32 flags;
+	u8 get_feat_ver;
+	u8 set_feat_ver;
+	__le16 effects;
+	u8 reserved[18];
+} __packed;
+
+/**
+ * struct cxl_mem_command - Driver representation of a memory device command
+ * @info: Command information as it exists for the UAPI
+ * @opcode: The actual bits used for the mailbox protocol
+ * @flags: Set of flags effecting driver behavior.
+ *
+ *  * %CXL_CMD_FLAG_FORCE_ENABLE: In cases of error, commands with this flag
+ *    will be enabled by the driver regardless of what hardware may have
+ *    advertised.
+ *
+ * The cxl_mem_command is the driver's internal representation of commands that
+ * are supported by the driver. Some of these commands may not be supported by
+ * the hardware. The driver will use @info to validate the fields passed in by
+ * the user then submit the @opcode to the hardware.
+ *
+ * See struct cxl_command_info.
+ */
+struct cxl_mem_command {
+	struct cxl_command_info info;
+	enum cxl_opcode opcode;
+	u32 flags;
+#define CXL_CMD_FLAG_FORCE_ENABLE BIT(0)
+};
+
 int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host);
 int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox);
 int cxl_mailbox_user_commands_info_get(struct cxl_mailbox *cxl_mbox, int nr_cmds,
diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index e0c7e49795ed..9103bac054b9 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -1354,7 +1354,7 @@ static void fill_feature_vendor_test(struct cxl_feat_entry *feat)
 	feat->flags = cpu_to_le32(BIT(5));
 	feat->get_feat_ver = 1;
 	feat->set_feat_ver = 1;
-	feat->set_effects = cpu_to_le16(BIT(0) | BIT(9));
+	feat->effects = cpu_to_le16(BIT(0) | BIT(9));
 }
 
 static int mock_get_supported_features(struct cxl_mockmem_data *mdata,
-- 
2.47.0

[RFC PATCH v2 15/20] fwctl/cxl: Add support for fwctl RPC command to enable CXL feature commands

[Dave Jiang]

fwctl provides a fwctl_ops->fw_rpc() callback in order to issue ioctls
to a device. The cxl fwctl driver will start by supporting the CXL
feature commands: Get Supported Features, Get Feature, and Set Feature.

The fw_rpc() callback provides 'enum fwctl_rpc_scope' parameter where
it indicates the security scope of the call. The Get Supported Features
and Get Feature calls can be executed with the scope of
FWCTL_RPC_DEBUG_READ_ONLY. The Set Feature call is gated by the effects
of the feature reported by Get Supported Features call for the specific
feature.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/core.h      |   5 +-
 drivers/cxl/core/mbox.c      | 260 +++++++++++++++++++++++++++++++----
 drivers/cxl/core/memdev.c    |   4 +-
 drivers/fwctl/cxl/cxl.c      | 101 +++++++++++++-
 include/cxl/mailbox.h        |   6 +
 include/uapi/fwctl/cxl.h     |  29 ++++
 include/uapi/linux/cxl_mem.h |  12 ++
 7 files changed, 387 insertions(+), 30 deletions(-)

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index ff30d1c99ca7..2b28fd8845d6 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -6,6 +6,8 @@
 
 #include <cxl/mailbox.h>
 
+extern struct rw_semaphore cxl_memdev_rwsem;
+
 extern const struct device_type cxl_nvdimm_bridge_type;
 extern const struct device_type cxl_nvdimm_type;
 extern const struct device_type cxl_pmu_type;
@@ -69,7 +71,8 @@ struct cxl_send_command;
 struct cxl_mem_query_commands;
 int cxl_query_cmd(struct cxl_mailbox *cxl_mbox,
 		  struct cxl_mem_query_commands __user *q);
-int cxl_send_cmd(struct cxl_mailbox *cxl_mailbox, struct cxl_send_command __user *s);
+int cxl_send_cmd_from_user(struct cxl_mailbox *cxl_mbox,
+			   struct cxl_send_command __user *s);
 void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
 				   resource_size_t length);
 
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 65fceabb9fe7..e7c5c709ac79 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -5,6 +5,7 @@
 #include <linux/ktime.h>
 #include <linux/mutex.h>
 #include <linux/unaligned.h>
+#include <cxl/mailbox.h>
 #include <cxlpci.h>
 #include <cxlmem.h>
 #include <cxl.h>
@@ -225,6 +226,38 @@ static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)
 	return NULL;
 }
 
+static struct cxl_mem_command *cxl_mem_find_command_by_id(int id)
+{
+	struct cxl_mem_command *c;
+
+	cxl_for_each_cmd(c)
+		if (c->info.id == id)
+			return c;
+
+	return NULL;
+
+}
+
+struct cxl_mem_command *
+cxl_get_mem_command_for_fwctl(struct cxl_mailbox *cxl_mbox, u32 id)
+{
+	struct cxl_mem_command *cmd;
+
+	if (id > CXL_MEM_COMMAND_ID_MAX)
+		return NULL;
+
+	cmd = &cxl_mem_commands[id];
+
+	if (!test_bit(cmd->info.id, cxl_mbox->enabled_cmds))
+		return NULL;
+
+	if (test_bit(cmd->info.id, cxl_mbox->exclusive_cmds))
+		return NULL;
+
+	return cmd;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_get_mem_command_for_fwctl, CXL);
+
 static const char *cxl_mem_opcode_to_name(u16 opcode)
 {
 	struct cxl_mem_command *c;
@@ -387,10 +420,13 @@ static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox_cmd,
 	}
 
 	/* Prepare to handle a full payload for variable sized output */
-	if (out_size == CXL_VARIABLE_PAYLOAD)
-		mbox_cmd->size_out = cxl_mbox->payload_size;
-	else
+	if (out_size == CXL_VARIABLE_PAYLOAD) {
+		/* Adding extra 8 bytes for FWCTL, should not impact operation */
+		mbox_cmd->size_out = cxl_mbox->payload_size +
+			sizeof(struct fwctl_rpc_cxl_out);
+	} else {
 		mbox_cmd->size_out = out_size;
+	}
 
 	if (mbox_cmd->size_out) {
 		mbox_cmd->payload_out = kvzalloc(mbox_cmd->size_out, GFP_KERNEL);
@@ -487,6 +523,73 @@ static int cxl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
 	return 0;
 }
 
+static int cxl_fwctl_to_mem_cmd(struct cxl_mem_command *mem_cmd,
+				const struct cxl_send_command *send_cmd,
+				struct cxl_mailbox *cxl_mbox)
+{
+	struct cxl_mem_command *c = &cxl_mem_commands[send_cmd->id];
+	const struct cxl_command_info *info = &c->info;
+
+	if (send_cmd->flags & ~CXL_MEM_COMMAND_FLAG_MASK)
+		return -EINVAL;
+
+	if (send_cmd->rsvd)
+		return -EINVAL;
+
+	if (send_cmd->in.rsvd || send_cmd->out.rsvd)
+		return -EINVAL;
+
+	/* Check the input buffer is the expected size */
+	if (info->size_in != CXL_VARIABLE_PAYLOAD &&
+	    info->size_in != send_cmd->in.size)
+		return -ENOMEM;
+
+	/* Check the output buffer is at least large enough */
+	if (info->size_out != CXL_VARIABLE_PAYLOAD &&
+	    send_cmd->out.size < info->size_out)
+		return -ENOMEM;
+
+	*mem_cmd = (struct cxl_mem_command) {
+		.info = {
+			.id = info->id,
+			.flags = info->flags,
+			.size_in = send_cmd->in.size,
+			.size_out = send_cmd->out.size,
+		},
+		.opcode = c->opcode
+	};
+
+	return 0;
+}
+
+static int verify_send_command(const struct cxl_send_command *send_cmd,
+			       struct cxl_mailbox *cxl_mbox)
+{
+	if (send_cmd->id == 0 || send_cmd->id >= CXL_MEM_COMMAND_ID_MAX)
+		return -ENOTTY;
+
+	/*
+	 * The user can never specify an input payload larger than what hardware
+	 * supports, but output can be arbitrarily large (simply write out as
+	 * much data as the hardware provides).
+	 */
+	if (send_cmd->in.size > cxl_mbox->payload_size)
+		return -EINVAL;
+
+	return 0;
+}
+
+/* Sanitize and construct a cxl_mbox_cmd */
+static int construct_mbox_cmd(struct cxl_mbox_cmd *mbox_cmd,
+			      struct cxl_mem_command *mem_cmd,
+			      struct cxl_mailbox *cxl_mbox,
+			      const struct cxl_send_command *send_cmd)
+{
+	return cxl_mbox_cmd_ctor(mbox_cmd, cxl_mbox, mem_cmd->opcode,
+				 mem_cmd->info.size_in, mem_cmd->info.size_out,
+				 send_cmd->in.payload);
+}
+
 /**
  * cxl_validate_cmd_from_user() - Check fields for CXL_MEM_SEND_COMMAND.
  * @mbox_cmd: Sanitized and populated &struct cxl_mbox_cmd.
@@ -511,16 +614,9 @@ static int cxl_validate_cmd_from_user(struct cxl_mbox_cmd *mbox_cmd,
 	struct cxl_mem_command mem_cmd;
 	int rc;
 
-	if (send_cmd->id == 0 || send_cmd->id >= CXL_MEM_COMMAND_ID_MAX)
-		return -ENOTTY;
-
-	/*
-	 * The user can never specify an input payload larger than what hardware
-	 * supports, but output can be arbitrarily large (simply write out as
-	 * much data as the hardware provides).
-	 */
-	if (send_cmd->in.size > cxl_mbox->payload_size)
-		return -EINVAL;
+	rc = verify_send_command(send_cmd, cxl_mbox);
+	if (rc)
+		return rc;
 
 	/* Sanitize and construct a cxl_mem_command */
 	if (send_cmd->id == CXL_MEM_COMMAND_ID_RAW)
@@ -531,10 +627,26 @@ static int cxl_validate_cmd_from_user(struct cxl_mbox_cmd *mbox_cmd,
 	if (rc)
 		return rc;
 
-	/* Sanitize and construct a cxl_mbox_cmd */
-	return cxl_mbox_cmd_ctor(mbox_cmd, cxl_mbox, mem_cmd.opcode,
-				 mem_cmd.info.size_in, mem_cmd.info.size_out,
-				 send_cmd->in.payload);
+	return construct_mbox_cmd(mbox_cmd, &mem_cmd, cxl_mbox, send_cmd);
+}
+
+static int cxl_validate_cmd_from_fwctl(struct cxl_mbox_cmd *mbox_cmd,
+				       struct cxl_mailbox *cxl_mbox,
+				       const struct cxl_send_command *send_cmd)
+{
+	struct cxl_mem_command mem_cmd;
+	int rc;
+
+	rc = verify_send_command(send_cmd, cxl_mbox);
+	if (rc)
+		return rc;
+
+	/* Sanitize and construct a cxl_mem_command */
+	rc = cxl_fwctl_to_mem_cmd(&mem_cmd, send_cmd, cxl_mbox);
+	if (rc)
+		return rc;
+
+	return construct_mbox_cmd(mbox_cmd, &mem_cmd, cxl_mbox, send_cmd);
 }
 
 static struct cxl_mem_command *
@@ -719,7 +831,112 @@ static int handle_mailbox_cmd_from_user(struct cxl_mailbox *cxl_mbox,
 	return rc;
 }
 
-int cxl_send_cmd(struct cxl_mailbox *cxl_mbox, struct cxl_send_command __user *s)
+static int cxl_send_cmd(struct cxl_mailbox *cxl_mbox, struct cxl_send_command *send,
+			struct cxl_mbox_cmd *mbox_cmd)
+{
+	int rc;
+
+	rc = cxl_validate_cmd_from_user(mbox_cmd, cxl_mbox, send);
+	if (rc)
+		return rc;
+
+	rc = handle_mailbox_cmd_from_user(cxl_mbox, mbox_cmd, send->out.payload,
+					  &send->out.size, &send->retval);
+	if (rc)
+		return rc;
+
+	return 0;
+}
+
+/**
+ * handle_mailbox_cmd_from_fwctl() - Dispatch a mailbox command for userspace.
+ * @cxl_mbox: The mailbox context for the operation.
+ * @mbox_cmd: The validated mailbox command.
+ *
+ * Return:
+ *  * %0	- Mailbox transaction succeeded. This implies the mailbox
+ *		  protocol completed successfully not that the operation itself
+ *		  was successful.
+ *  * %-ENOMEM  - Couldn't allocate a bounce buffer.
+ *  * %-EINTR	- Mailbox acquisition interrupted.
+ *  * %-EXXX	- Transaction level failures.
+ *
+ * Dispatches a mailbox command on behalf of a userspace request.
+ * The output payload is copied to userspace by fwctl.
+ *
+ * See cxl_send_cmd().
+ */
+static int handle_mailbox_cmd_from_fwctl(struct cxl_mailbox *cxl_mbox,
+					 struct cxl_mbox_cmd *mbox_cmd)
+{
+	struct device *dev = cxl_mbox->host;
+	struct fwctl_rpc_cxl_out *orig_out;
+	int rc;
+
+	/*
+	 * Save the payload_out pointer and move it to where hardware output
+	 * can be copied to.
+	 */
+	orig_out = mbox_cmd->payload_out;
+	mbox_cmd->payload_out = (void *)orig_out + sizeof(*orig_out);
+
+	dev_dbg(dev,
+		"Submitting %s command for user\n"
+		"\topcode: %x\n"
+		"\tsize: %zx\n",
+		cxl_mem_opcode_to_name(mbox_cmd->opcode),
+		mbox_cmd->opcode, mbox_cmd->size_in);
+
+	rc = cxl_mbox->mbox_send(cxl_mbox, mbox_cmd);
+	if (rc)
+		return rc;
+
+	orig_out->retval = mbox_cmd->return_code;
+	mbox_cmd->payload_out = (void *)orig_out;
+
+	return 0;
+}
+
+int cxl_mbox_send_cmd(struct cxl_mailbox *cxl_mbox,
+		      struct fwctl_rpc_cxl *rpc_in,
+		      struct cxl_mbox_cmd *mbox_cmd, size_t *out_len)
+{
+	struct cxl_send_command send_cmd = {
+		.id = rpc_in->id,
+		.flags = rpc_in->flags,
+		.in.size = rpc_in->op_size,
+		.in.payload = rpc_in->in_payload,
+		.out.size = *out_len,
+	};
+	struct cxl_mem_command *cmd;
+	int rc;
+
+	cmd = cxl_mem_find_command_by_id(rpc_in->id);
+	if (!cmd)
+		return -EINVAL;
+	send_cmd.raw.opcode = cmd->opcode;
+
+	rc = cxl_validate_cmd_from_fwctl(mbox_cmd, cxl_mbox, &send_cmd);
+	if (rc)
+		return rc;
+
+	rc = handle_mailbox_cmd_from_fwctl(cxl_mbox, mbox_cmd);
+	if (rc)
+		return rc;
+
+	guard(rwsem_read)(&cxl_memdev_rwsem);
+	rc = cxl_mbox->mbox_send(cxl_mbox, mbox_cmd);
+	if (rc)
+		return rc;
+
+	*out_len = mbox_cmd->size_out;
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_mbox_send_cmd, CXL);
+
+int cxl_send_cmd_from_user(struct cxl_mailbox *cxl_mbox,
+			   struct cxl_send_command __user *s)
 {
 	struct device *dev = cxl_mbox->host;
 	struct cxl_send_command send;
@@ -731,12 +948,7 @@ int cxl_send_cmd(struct cxl_mailbox *cxl_mbox, struct cxl_send_command __user *s
 	if (copy_from_user(&send, s, sizeof(send)))
 		return -EFAULT;
 
-	rc = cxl_validate_cmd_from_user(&mbox_cmd, cxl_mbox, &send);
-	if (rc)
-		return rc;
-
-	rc = handle_mailbox_cmd_from_user(cxl_mbox, &mbox_cmd, send.out.payload,
-					  &send.out.size, &send.retval);
+	rc = cxl_send_cmd(cxl_mbox, &send, &mbox_cmd);
 	if (rc)
 		return rc;
 
diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
index 4d544a55ac3e..7fa16930cd85 100644
--- a/drivers/cxl/core/memdev.c
+++ b/drivers/cxl/core/memdev.c
@@ -11,7 +11,7 @@
 #include "trace.h"
 #include "core.h"
 
-static DECLARE_RWSEM(cxl_memdev_rwsem);
+DECLARE_RWSEM(cxl_memdev_rwsem);
 
 /*
  * An entire PCI topology full of devices should be enough for any
@@ -667,7 +667,7 @@ static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
 	case CXL_MEM_QUERY_COMMANDS:
 		return cxl_query_cmd(cxl_mbox, (void __user *)arg);
 	case CXL_MEM_SEND_COMMAND:
-		return cxl_send_cmd(cxl_mbox, (void __user *)arg);
+		return cxl_send_cmd_from_user(cxl_mbox, (void __user *)arg);
 	default:
 		return -ENOTTY;
 	}
diff --git a/drivers/fwctl/cxl/cxl.c b/drivers/fwctl/cxl/cxl.c
index ce8960a9beaa..164f6774a2c1 100644
--- a/drivers/fwctl/cxl/cxl.c
+++ b/drivers/fwctl/cxl/cxl.c
@@ -77,11 +77,106 @@ static void *cxlctl_hw_info(struct fwctl_uctx *uctx, int commands, size_t *out_l
 	return_ptr(out);
 }
 
+static bool cxlctl_validate_set_features(struct cxl_mailbox *cxl_mbox,
+					 const struct fwctl_rpc_cxl *rpc_in,
+					 enum fwctl_rpc_scope scope)
+{
+	struct cxl_feat_entry *feat;
+	bool found = false;
+	uuid_t uuid;
+	u16 effects, mask;
+
+	if (rpc_in->op_size < sizeof(struct cxl_set_feature_input))
+		return false;
+
+	if (copy_from_user(&uuid, u64_to_user_ptr(rpc_in->in_payload),
+			   sizeof(uuid)))
+		return false;
+
+	for (int i = 0; i < cxl_mbox->num_features; i++) {
+		feat = &cxl_mbox->entries[i];
+		if (uuid_equal(&uuid, &feat->uuid)) {
+			found = true;
+			break;
+		}
+	}
+
+	if (!found)
+		return false;
+
+	effects = le16_to_cpu(feat->effects);
+	/* Currently no user background command support */
+	if (effects & CXL_CMD_BACKGROUND)
+		return false;
+
+	mask = CXL_CMD_CONFIG_CHANGE_IMMEDIATE |
+	       CXL_CMD_DATA_CHANGE_IMMEDIATE |
+	       CXL_CMD_POLICY_CHANGE_IMMEDIATE |
+	       CXL_CMD_LOG_CHANGE_IMMEDIATE;
+	if (effects & mask && scope >= FWCTL_RPC_DEBUG_WRITE_FULL)
+		return true;
+
+	/* These effects supported for all scope */
+	if ((effects & CXL_CMD_CONFIG_CHANGE_COLD_RESET ||
+	     effects & CXL_CMD_CONFIG_CHANGE_CONV_RESET) &&
+	    scope >= FWCTL_RPC_DEBUG_WRITE)
+		return true;
+
+	return false;
+}
+
+static bool cxlctl_validate_hw_cmds(struct cxl_mailbox *cxl_mbox,
+				    const struct fwctl_rpc_cxl *rpc_in,
+				    enum fwctl_rpc_scope scope)
+{
+	struct cxl_mem_command *cmd;
+
+	/*
+	 * Only supporting feature commands for now.
+	 */
+	if (!cxl_mbox->num_features)
+		return false;
+
+	cmd = cxl_get_mem_command_for_fwctl(cxl_mbox, rpc_in->id);
+	if (!cmd)
+		return false;
+
+	switch (cmd->opcode) {
+	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
+		if (scope >= FWCTL_RPC_CONFIGURATION)
+			return true;
+		return false;
+	case CXL_MBOX_OP_GET_FEATURE:
+		if (scope >= FWCTL_RPC_DEBUG_READ_ONLY)
+			return true;
+		return false;
+	case CXL_MBOX_OP_SET_FEATURE:
+		return cxlctl_validate_set_features(cxl_mbox, rpc_in, scope);
+	default:
+		return false;
+	}
+}
+
 static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
-			   void *rpc_in, size_t in_len, size_t *out_len)
+			   void *in, size_t in_len, size_t *out_len)
 {
-	/* Place holder */
-	return ERR_PTR(-EOPNOTSUPP);
+	struct cxlctl_dev *cxlctl =
+		container_of(uctx->fwctl, struct cxlctl_dev, fwctl);
+	struct cxl_mailbox *cxl_mbox = cxlctl->mbox;
+	struct fwctl_rpc_cxl *rpc_in = in;
+	struct cxl_mbox_cmd mbox_cmd;
+	int rc;
+
+	if (!cxlctl_validate_hw_cmds(cxlctl->mbox, rpc_in, scope))
+		return ERR_PTR(-EPERM);
+
+	rc = cxl_mbox_send_cmd(cxl_mbox, rpc_in, &mbox_cmd, out_len);
+	if (rc)
+		return ERR_PTR(rc);
+
+	*out_len = mbox_cmd.size_out;
+
+	return mbox_cmd.payload_out;
 }
 
 static const struct fwctl_ops cxlctl_ops = {
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index e753d5d1d708..3e5e9c9362f5 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -193,5 +193,11 @@ int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host);
 int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox);
 int cxl_mailbox_user_commands_info_get(struct cxl_mailbox *cxl_mbox, int nr_cmds,
 				       void *outbuf, size_t *out_len);
+struct cxl_mem_command *cxl_get_mem_command(u32 id);
+struct cxl_mem_command *
+cxl_get_mem_command_for_fwctl(struct cxl_mailbox *cxl_mbox, u32 id);
+int cxl_mbox_send_cmd(struct cxl_mailbox *cxl_mbox,
+		      struct fwctl_rpc_cxl *rpc_in,
+		      struct cxl_mbox_cmd *mbox_cmd, size_t *out_len);
 
 #endif
diff --git a/include/uapi/fwctl/cxl.h b/include/uapi/fwctl/cxl.h
index a32c4c752db6..99804fc72f28 100644
--- a/include/uapi/fwctl/cxl.h
+++ b/include/uapi/fwctl/cxl.h
@@ -19,4 +19,33 @@ struct fwctl_info_cxl {
 	__u32 nr_commands;
 };
 
+/**
+ * struct fwctl_rpc_cxl - ioctl(FWCTL_RPC) input for CXL
+ * @id: The command id to send to the memory device. This must be one of the
+ *	commands returned by the query command.
+ * @flags: Flags for the command (input).
+ * @op_size: Size of hw operation
+ * @reserved: Reserved. Must be 0s.
+ * @in_payload: User address of the hardware op input
+ */
+struct fwctl_rpc_cxl {
+	__u32 id;
+	__u32 flags;
+	__u32 op_size;
+	__u32 reserved;
+	__aligned_u64 in_payload;
+};
+
+/**
+ * struct fwctl_rpc_cxl_out - ioctl9FWCTL_RPC) output for CXL
+ * @size: Size of the output payload
+ * @retval: Return value from device
+ * @payload: Return data from device
+ */
+struct fwctl_rpc_cxl_out {
+	__u32 size;
+	__u32 retval;
+	__u8 payload[];
+};
+
 #endif
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 9dd37849c450..4e7c8c03cfe8 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -234,4 +234,16 @@ struct cxl_send_command {
 	} out;
 };
 
+/*
+ * CXL spec r3.1 Table 8-101 Set Feature Input Payload
+ */
+struct cxl_set_feature_input {
+	__u8 uuid[16];
+	__u32 flags;
+	__u16 offset;
+	__u8 version;
+	__u8 reserved[9];
+	__u8 data[];
+} __packed;
+
 #endif
-- 
2.47.0

[RFC PATCH v2 16/20] fwctl/cxl: Add support to filter exclusive features

[Dave Jiang]

Add support to allow filtering of CXL features that are exclusive to the
kernel and make them not visible to user space. The "get supported
features" mailbox command returned to the userspace is emulated and
will skip the features that is marked exclusive for the kernel. The
exclusion allows certain the feature setting of certain commands such
as claimed by RAS to be exclusive to the kernel.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/cxl/core/mbox.c | 115 ++++++++++++++++++++++++++++++++++++----
 drivers/fwctl/cxl/cxl.c |   7 +--
 include/cxl/features.h  |  52 ++++++++++++++++++
 include/cxl/mailbox.h   |   8 ++-
 4 files changed, 168 insertions(+), 14 deletions(-)
 create mode 100644 include/cxl/features.h

diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index e7c5c709ac79..12ace2951f7c 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -5,6 +5,7 @@
 #include <linux/ktime.h>
 #include <linux/mutex.h>
 #include <linux/unaligned.h>
+#include <cxl/features.h>
 #include <cxl/mailbox.h>
 #include <cxlpci.h>
 #include <cxlmem.h>
@@ -897,6 +898,78 @@ static int handle_mailbox_cmd_from_fwctl(struct cxl_mailbox *cxl_mbox,
 	return 0;
 }
 
+/*
+ * Emulate the 'get supported features' mailbox command and only copy out the
+ * features that are not marked as exclusive for kernel.
+ */
+static void cxl_mbox_get_supported_features_filtered(struct cxl_mailbox *cxl_mbox,
+						     struct cxl_mbox_cmd *mbox_cmd)
+{
+	struct cxl_mbox_get_sup_feats_in *feat_in = mbox_cmd->payload_in;
+	struct cxl_mbox_get_sup_feats_out *feat_out = mbox_cmd->payload_out;
+	const int feat_out_size = sizeof(*feat_out);
+	struct cxl_feat_entry *pos;
+	const int feat_ent_size = sizeof(*pos);
+	int out_count, ents, u;
+	u32 count;
+	u16 start;
+
+	count = le32_to_cpu(feat_in->count);
+	start = le16_to_cpu(feat_in->start_idx);
+	ents = count / sizeof(struct cxl_feat_entry);
+	ents -= start;
+	if (ents < 0) {
+		mbox_cmd->size_out = 0;
+		mbox_cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return;
+	}
+
+	if (mbox_cmd->size_out < feat_out_size) {
+		mbox_cmd->size_out = 0;
+		mbox_cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return;
+	}
+
+	feat_out->supported_feats =
+		cpu_to_le16(cxl_mbox->num_user_feats);
+	if (ents == 0) {
+		feat_out->num_entries = cpu_to_le16(0);
+		mbox_cmd->size_out = feat_out_size;
+		mbox_cmd->return_code = CXL_MBOX_CMD_RC_SUCCESS;
+		return;
+	}
+
+	pos = &feat_out->ents[0];
+	out_count = feat_out_size;
+	for (int i = 0, u = 0; i < cxl_mbox->num_features; i++) {
+		struct cxl_feature *feat = &cxl_mbox->entries[i];
+
+		if (feat->exclusive)
+			continue;
+
+		if (u < start) {
+			u++;
+			continue;
+		}
+
+		memcpy(pos, &feat->entry, feat_ent_size);
+		out_count += feat_ent_size;
+		pos++;
+		u++;
+
+		if (u == count)
+			break;
+
+		/* Make sure it does not go over total output buffer size */
+		if (out_count + feat_ent_size >= mbox_cmd->size_out)
+			break;
+	}
+
+	feat_out->num_entries = cpu_to_le16(u);
+	mbox_cmd->size_out = out_count;
+	mbox_cmd->return_code = CXL_MBOX_CMD_RC_SUCCESS;
+}
+
 int cxl_mbox_send_cmd(struct cxl_mailbox *cxl_mbox,
 		      struct fwctl_rpc_cxl *rpc_in,
 		      struct cxl_mbox_cmd *mbox_cmd, size_t *out_len)
@@ -925,9 +998,13 @@ int cxl_mbox_send_cmd(struct cxl_mailbox *cxl_mbox,
 		return rc;
 
 	guard(rwsem_read)(&cxl_memdev_rwsem);
-	rc = cxl_mbox->mbox_send(cxl_mbox, mbox_cmd);
-	if (rc)
-		return rc;
+	if (send_cmd.raw.opcode != CXL_MBOX_OP_GET_SUPPORTED_FEATURES) {
+		rc = cxl_mbox->mbox_send(cxl_mbox, mbox_cmd);
+		if (rc)
+			return rc;
+	} else {
+		cxl_mbox_get_supported_features_filtered(cxl_mbox, mbox_cmd);
+	}
 
 	*out_len = mbox_cmd->size_out;
 
@@ -1128,6 +1205,18 @@ static int cxl_get_supported_features_count(struct cxl_dev_state *cxlds)
 	return 0;
 }
 
+static bool is_feature_exclusive(struct cxl_feature *feat)
+{
+	int i;
+
+	for (i = 0; i < CXL_FEAT_UUID_MAX; i++) {
+		if (uuid_equal(&feat->entry.uuid, &cxl_exclusive_feats[i]))
+			return true;
+	}
+
+	return false;
+}
+
 int cxl_get_supported_features(struct cxl_dev_state *cxlds)
 {
 	struct cxl_mbox_get_sup_feats_out *mbox_out __free(kvfree) = NULL;
@@ -1138,7 +1227,7 @@ int cxl_get_supported_features(struct cxl_dev_state *cxlds)
 	int hdr_size = sizeof(*mbox_out);
 	struct cxl_mbox_cmd mbox_cmd;
 	struct cxl_mem_command *cmd;
-	struct cxl_feat_entry *entry;
+	struct cxl_feature *feat;
 
 	/* Get supported features is optional, need to check */
 	cmd = cxl_mem_find_command(CXL_MBOX_OP_GET_SUPPORTED_FEATURES);
@@ -1156,7 +1245,7 @@ int cxl_get_supported_features(struct cxl_dev_state *cxlds)
 		return 0;
 	}
 
-	struct cxl_feat_entry *entries __free(kvfree) =
+	struct cxl_feature *entries __free(kvfree) =
 		kvmalloc(cxl_mbox->num_features * feat_size, GFP_KERNEL);
 
 	if (!entries)
@@ -1165,7 +1254,7 @@ int cxl_get_supported_features(struct cxl_dev_state *cxlds)
 	max_size = cxl_mbox->payload_size - hdr_size;
 	/* max feat entries that can fit in mailbox max payload size */
 	max_feats = max_size / feat_size;
-	entry = &entries[0];
+	feat = &entries[0];
 
 	mbox_out = kvmalloc(cxl_mbox->payload_size, GFP_KERNEL);
 	if (!mbox_out)
@@ -1228,8 +1317,13 @@ int cxl_get_supported_features(struct cxl_dev_state *cxlds)
 			goto err;
 		}
 
-		memcpy(entry, mbox_out->ents, retrieved);
-		entry++;
+		memcpy(&feat->entry, mbox_out->ents, retrieved);
+		if (is_feature_exclusive(feat))
+			feat->exclusive = true;
+		else
+			cxl_mbox->num_user_feats++;
+		feat++;
+
 		/*
 		 * If the number of output entries is less than expected, add the
 		 * remaining entries to the next batch.
@@ -1256,13 +1350,14 @@ int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t *f
 				    struct cxl_feat_entry *feat_entry_out)
 {
 	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
-	struct cxl_feat_entry *feat_entry;
+	struct cxl_feature *feat_entry;
 	int count;
 
 	/* Check CXL dev supports the feature */
 	feat_entry = &cxl_mbox->entries[0];
 	for (count = 0; count < cxl_mbox->num_features; count++, feat_entry++) {
-		if (uuid_equal(&feat_entry->uuid, feat_uuid)) {
+		if (uuid_equal(&feat_entry->entry.uuid, feat_uuid) &&
+		    !is_feature_exclusive(feat_entry)) {
 			memcpy(feat_entry_out, feat_entry, sizeof(*feat_entry_out));
 			return 0;
 		}
diff --git a/drivers/fwctl/cxl/cxl.c b/drivers/fwctl/cxl/cxl.c
index 164f6774a2c1..3915a59c7408 100644
--- a/drivers/fwctl/cxl/cxl.c
+++ b/drivers/fwctl/cxl/cxl.c
@@ -81,7 +81,7 @@ static bool cxlctl_validate_set_features(struct cxl_mailbox *cxl_mbox,
 					 const struct fwctl_rpc_cxl *rpc_in,
 					 enum fwctl_rpc_scope scope)
 {
-	struct cxl_feat_entry *feat;
+	struct cxl_feature *feat;
 	bool found = false;
 	uuid_t uuid;
 	u16 effects, mask;
@@ -95,7 +95,8 @@ static bool cxlctl_validate_set_features(struct cxl_mailbox *cxl_mbox,
 
 	for (int i = 0; i < cxl_mbox->num_features; i++) {
 		feat = &cxl_mbox->entries[i];
-		if (uuid_equal(&uuid, &feat->uuid)) {
+		if (uuid_equal(&uuid, &feat->entry.uuid) &&
+		    !feat->exclusive) {
 			found = true;
 			break;
 		}
@@ -104,7 +105,7 @@ static bool cxlctl_validate_set_features(struct cxl_mailbox *cxl_mbox,
 	if (!found)
 		return false;
 
-	effects = le16_to_cpu(feat->effects);
+	effects = le16_to_cpu(feat->entry.effects);
 	/* Currently no user background command support */
 	if (effects & CXL_CMD_BACKGROUND)
 		return false;
diff --git a/include/cxl/features.h b/include/cxl/features.h
new file mode 100644
index 000000000000..6074a3f79d70
--- /dev/null
+++ b/include/cxl/features.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2024 Intel Corporation. */
+#ifndef __CXL_FEATS_H_
+#define __CXL_FEATS_H_
+
+#include <linux/uuid.h>
+
+#define CXL_FEAT_PATROL_SCRUB_UUID						\
+	UUID_INIT(0x96dad7d6, 0xfde8, 0x482b, 0xa7, 0x33, 0x75, 0x77, 0x4e,	\
+		  0x06, 0xdb, 0x8a)
+
+#define CXL_FEAT_ECS_UUID							\
+	UUID_INIT(0xe5b13f22, 0x2328, 0x4a14, 0xb8, 0xba, 0xb9, 0x69, 0x1e,	\
+		  0x89, 0x33, 0x86)
+
+#define CXL_FEAT_SPPR_UUID							\
+	UUID_INIT(0x892ba475, 0xfad8, 0x474e, 0x9d, 0x3e, 0x69, 0x2c, 0x91,	\
+		  0x75, 0x68, 0xbb)
+
+#define CXL_FEAT_HPPR_UUID							\
+	UUID_INIT(0x80ea4521, 0x786f, 0x4127, 0xaf, 0xb1, 0xec, 0x74, 0x59,	\
+		  0xfb, 0x0e, 0x24)
+
+#define CXL_FEAT_CACHELINE_SPARING_UUID						\
+	UUID_INIT(0x96C33386, 0x91dd, 0x44c7, 0x9e, 0xcb, 0xfd, 0xaf, 0x65,	\
+		  0x03, 0xba, 0xc4)
+
+#define CXL_FEAT_ROW_SPARING_UUID						\
+	UUID_INIT(0x450ebf67, 0xb135, 0x4f97, 0xa4, 0x98, 0xc2, 0xd5, 0x7f,	\
+		  0x27, 0x9b, 0xed)
+
+#define CXL_FEAT_BANK_SPARING_UUID						\
+	UUID_INIT(0x78b79636, 0x90ac, 0x4b64, 0xa4, 0xef, 0xfa, 0xac, 0x5d,	\
+		  0x18, 0xa8, 0x63)
+
+#define CXL_FEAT_RANK_SPARING_UUID						\
+	UUID_INIT(0x34dbaff5, 0x0552, 0x4281, 0x8f, 0x76, 0xda, 0x0b, 0x5e,	\
+		  0x7a, 0x76, 0xa7)
+
+#define CXL_FEAT_UUID_MAX	8
+static uuid_t cxl_exclusive_feats[CXL_FEAT_UUID_MAX] = {
+	CXL_FEAT_PATROL_SCRUB_UUID,
+	CXL_FEAT_ECS_UUID,
+	CXL_FEAT_SPPR_UUID,
+	CXL_FEAT_HPPR_UUID,
+	CXL_FEAT_CACHELINE_SPARING_UUID,
+	CXL_FEAT_ROW_SPARING_UUID,
+	CXL_FEAT_BANK_SPARING_UUID,
+	CXL_FEAT_RANK_SPARING_UUID,
+};
+
+#endif
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index 3e5e9c9362f5..f44ac93ebdf4 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -80,7 +80,8 @@ struct cxl_mailbox {
 	struct mutex mbox_mutex; /* lock to protect mailbox context */
 	struct rcuwait mbox_wait;
 	int num_features;
-	struct cxl_feat_entry *entries;
+	int num_user_feats;
+	struct cxl_feature *entries;
 	int (*mbox_send)(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd *cmd);
 };
 
@@ -165,6 +166,11 @@ struct cxl_feat_entry {
 	u8 reserved[18];
 } __packed;
 
+struct cxl_feature {
+	bool exclusive;
+	struct cxl_feat_entry entry;
+};
+
 /**
  * struct cxl_mem_command - Driver representation of a memory device command
  * @info: Command information as it exists for the UAPI
-- 
2.47.0

[RFC PATCH v2 17/20] cxl/test: Add Get Feature support to cxl_test

[Dave Jiang]

Add emulation of Get Feature command to cxl_test. The feature for
device patrol scrub is returned by the emulation code. This is
the only feature currently supported by cxl_test. It returns
the information for the device patrol scrub feature.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 tools/testing/cxl/test/mem.c | 63 ++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index 9103bac054b9..547d7289d8df 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -48,6 +48,10 @@ static struct cxl_cel_entry mock_cel[] = {
 		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_SUPPORTED_FEATURES),
 		.effect = CXL_CMD_EFFECT_NONE,
 	},
+	{
+		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_FEATURE),
+		.effect = CXL_CMD_EFFECT_NONE,
+	},
 	{
 		.opcode = cpu_to_le16(CXL_MBOX_OP_IDENTIFY),
 		.effect = CXL_CMD_EFFECT_NONE,
@@ -149,6 +153,10 @@ struct mock_event_store {
 	u32 ev_status;
 };
 
+struct vendor_test_feat {
+	__le32 data;
+} __packed;
+
 struct cxl_mockmem_data {
 	void *lsa;
 	void *fw;
@@ -165,6 +173,7 @@ struct cxl_mockmem_data {
 	u8 event_buf[SZ_4K];
 	u64 timestamp;
 	unsigned long sanitize_timeout;
+	struct vendor_test_feat test_feat;
 };
 
 static struct mock_event_log *event_find_log(struct device *dev, int log_type)
@@ -1357,6 +1366,51 @@ static void fill_feature_vendor_test(struct cxl_feat_entry *feat)
 	feat->effects = cpu_to_le16(BIT(0) | BIT(9));
 }
 
+struct get_feat_input {
+	uuid_t uuid;
+	__le16 offset;
+	__le16 count;
+	u8 selection;
+} __packed;
+
+static int mock_get_test_feature(struct cxl_mockmem_data *mdata,
+				 struct cxl_mbox_cmd *cmd)
+{
+	struct vendor_test_feat *output = cmd->payload_out;
+	struct get_feat_input *input = cmd->payload_in;
+	u16 offset = le16_to_cpu(input->offset);
+	u16 count = le16_to_cpu(input->count);
+	u8 *ptr;
+
+	if (offset > sizeof(*output)) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	if (offset + count > sizeof(*output)) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	ptr = (u8 *)&mdata->test_feat + offset;
+	memcpy((u8 *)output + offset, ptr, count);
+
+	return 0;
+}
+
+static int mock_get_feature(struct cxl_mockmem_data *mdata,
+			    struct cxl_mbox_cmd *cmd)
+{
+	struct get_feat_input *input = cmd->payload_in;
+
+	if (uuid_equal(&input->uuid, &CXL_VENDOR_FEATURE_TEST))
+		return mock_get_test_feature(mdata, cmd);
+
+	cmd->return_code = CXL_MBOX_CMD_RC_UNSUPPORTED;
+
+	return -EOPNOTSUPP;
+}
+
 static int mock_get_supported_features(struct cxl_mockmem_data *mdata,
 				       struct cxl_mbox_cmd *cmd)
 {
@@ -1487,6 +1541,9 @@ static int cxl_mock_mbox_send(struct cxl_mailbox *cxl_mbox,
 	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
 		rc = mock_get_supported_features(mdata, cmd);
 		break;
+	case CXL_MBOX_OP_GET_FEATURE:
+		rc = mock_get_feature(mdata, cmd);
+		break;
 	default:
 		break;
 	}
@@ -1534,6 +1591,11 @@ static int cxl_mock_mailbox_create(struct cxl_dev_state *cxlds)
 	return 0;
 }
 
+static void cxl_mock_test_feat_init(struct cxl_mockmem_data *mdata)
+{
+	mdata->test_feat.data = cpu_to_le32(0xdeadbeef);
+}
+
 static int cxl_mock_mem_probe(struct platform_device *pdev)
 {
 	struct device *dev = &pdev->dev;
@@ -1625,6 +1687,7 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 		return rc;
 
 	cxl_mem_get_event_records(mds, CXLDEV_EVENT_STATUS_ALL);
+	cxl_mock_test_feat_init(mdata);
 
 	return 0;
 }
-- 
2.47.0

[RFC PATCH v2 18/20] cxl/test: Add Set Feature support to cxl_test

[Dave Jiang]

Add emulation to support Set Feature mailbox command to cxl_test.
The only feature supported is the device patrol scrub feature. The
set feature allows activation of patrol scrub for the cxl_test
emulated device. The command does not support partial data transfer
even though the spec allows it. This restriction is to reduce complexity
of the emulation given the patrol scrub feature is very minimal.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 tools/testing/cxl/test/mem.c | 69 ++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
index 547d7289d8df..25f07843cb84 100644
--- a/tools/testing/cxl/test/mem.c
+++ b/tools/testing/cxl/test/mem.c
@@ -52,6 +52,10 @@ static struct cxl_cel_entry mock_cel[] = {
 		.opcode = cpu_to_le16(CXL_MBOX_OP_GET_FEATURE),
 		.effect = CXL_CMD_EFFECT_NONE,
 	},
+	{
+		.opcode = cpu_to_le16(CXL_MBOX_OP_SET_FEATURE),
+		.effect = cpu_to_le16(EFFECT(CONF_CHANGE_IMMEDIATE)),
+	},
 	{
 		.opcode = cpu_to_le16(CXL_MBOX_OP_IDENTIFY),
 		.effect = CXL_CMD_EFFECT_NONE,
@@ -1411,6 +1415,68 @@ static int mock_get_feature(struct cxl_mockmem_data *mdata,
 	return -EOPNOTSUPP;
 }
 
+struct set_feat_input {
+	uuid_t uuid;
+	__le32 flags;
+	__le16 offset;
+	u8 version;
+	u8 rsvd[9];
+	u8 data[];
+} __packed;
+
+enum set_feature_flags {
+	SET_FEATURE_F_FULL_XFER = 0,
+	SET_FEATURE_F_INIT_XFER,
+	SET_FEATURE_F_CONT_XFER,
+	SET_FEATURE_F_FIN_XFER,
+	SET_FEATURE_F_ABORT_XFER,
+};
+
+#define SET_FEATURE_F_ACTION_MASK	GENMASK(2, 0)
+
+static int mock_set_test_feature(struct cxl_mockmem_data *mdata,
+				 struct cxl_mbox_cmd *cmd)
+{
+	struct set_feat_input *input = cmd->payload_in;
+	struct vendor_test_feat *test = (struct vendor_test_feat *)input->data;
+	u32 action;
+
+	action = FIELD_GET(SET_FEATURE_F_ACTION_MASK,
+			   le32_to_cpu(input->flags));
+	/*
+	 * While it is spec compliant to support other set actions, it is not
+	 * necessary to add the complication in the emulation currently. Reject
+	 * anything besides full xfer.
+	 */
+	if (action != SET_FEATURE_F_FULL_XFER) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	/* Offset should be reserved when doing full transfer */
+	if (input->offset) {
+		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
+		return -EINVAL;
+	}
+
+	memcpy(&mdata->test_feat.data, &test->data, sizeof(u32));
+
+	return 0;
+}
+
+static int mock_set_feature(struct cxl_mockmem_data *mdata,
+			    struct cxl_mbox_cmd *cmd)
+{
+	struct set_feat_input *input = cmd->payload_in;
+
+	if (uuid_equal(&input->uuid, &CXL_VENDOR_FEATURE_TEST))
+		return mock_set_test_feature(mdata, cmd);
+
+	cmd->return_code = CXL_MBOX_CMD_RC_UNSUPPORTED;
+
+	return -EOPNOTSUPP;
+}
+
 static int mock_get_supported_features(struct cxl_mockmem_data *mdata,
 				       struct cxl_mbox_cmd *cmd)
 {
@@ -1544,6 +1610,9 @@ static int cxl_mock_mbox_send(struct cxl_mailbox *cxl_mbox,
 	case CXL_MBOX_OP_GET_FEATURE:
 		rc = mock_get_feature(mdata, cmd);
 		break;
+	case CXL_MBOX_OP_SET_FEATURE:
+		rc = mock_set_feature(mdata, cmd);
+		break;
 	default:
 		break;
 	}
-- 
2.47.0

[RFC PATCH v2 19/20] fwctl: Move fwctl documentation to its own directory

[Dave Jiang]

Move fwctl.rst to its own directory in prepraration for adding more
documentation.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 Documentation/userspace-api/{ => fwctl}/fwctl.rst |  0
 Documentation/userspace-api/fwctl/index.rst       | 12 ++++++++++++
 Documentation/userspace-api/index.rst             |  2 +-
 MAINTAINERS                                       |  2 +-
 4 files changed, 14 insertions(+), 2 deletions(-)
 rename Documentation/userspace-api/{ => fwctl}/fwctl.rst (100%)
 create mode 100644 Documentation/userspace-api/fwctl/index.rst

diff --git a/Documentation/userspace-api/fwctl.rst b/Documentation/userspace-api/fwctl/fwctl.rst
similarity index 100%
rename from Documentation/userspace-api/fwctl.rst
rename to Documentation/userspace-api/fwctl/fwctl.rst
diff --git a/Documentation/userspace-api/fwctl/index.rst b/Documentation/userspace-api/fwctl/index.rst
new file mode 100644
index 000000000000..06959fbf1547
--- /dev/null
+++ b/Documentation/userspace-api/fwctl/index.rst
@@ -0,0 +1,12 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Firmware Control (FWCTL) Userspace API
+======================================
+
+A framework that define a common set of limited rules that allows user space
+to securely construct and execute RPCs inside device firmware.
+
+.. toctree::
+   :maxdepth: 1
+
+   fwctl
diff --git a/Documentation/userspace-api/index.rst b/Documentation/userspace-api/index.rst
index 2bc43a658074..c2f067b3de01 100644
--- a/Documentation/userspace-api/index.rst
+++ b/Documentation/userspace-api/index.rst
@@ -44,7 +44,7 @@ Devices and I/O
 
    accelerators/ocxl
    dma-buf-alloc-exchange
-   fwctl
+   fwctl/index
    gpio/index
    iommufd
    media/index
diff --git a/MAINTAINERS b/MAINTAINERS
index 6b52c4168b94..1699316f86ae 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9377,7 +9377,7 @@ FWCTL SUBSYSTEM
 M:	Jason Gunthorpe <jgg@nvidia.com>
 M:	Saeed Mahameed <saeedm@nvidia.com>
 S:	Maintained
-F:	Documentation/userspace-api/fwctl.rst
+F:	Documentation/userspace-api/fwctl/
 F:	drivers/fwctl/
 F:	include/linux/fwctl.h
 F:	include/uapi/fwctl/
-- 
2.47.0

[RFC PATCH v2 20/20] fwctl/cxl: Add documentation to FWCTL CXL

[Dave Jiang]

Add policy and operational documentation for FWCTL CXL.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 .../userspace-api/fwctl/fwctl-cxl.rst         | 91 +++++++++++++++++++
 Documentation/userspace-api/fwctl/index.rst   |  1 +
 2 files changed, 92 insertions(+)
 create mode 100644 Documentation/userspace-api/fwctl/fwctl-cxl.rst

diff --git a/Documentation/userspace-api/fwctl/fwctl-cxl.rst b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
new file mode 100644
index 000000000000..158e9aa45609
--- /dev/null
+++ b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
@@ -0,0 +1,91 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+================
+fwctl cxl driver
+================
+
+:Author: Dave Jiang
+
+Overview
+========
+
+The CXL spec defines a set of commands that can be issued to the mailbox of a
+CXL device or switch. It also left room for vendor specific commands to be
+issued to the mailbox as well. fwctl provides a path to issue a set of allowed
+mailbox commands from user space to the device moderated by the kernel driver.
+
+While there are a large set of mailbox commands, only the feature related
+commands will be allowed to be issued through fwctl initially. No background
+commands will be supported at this time.
+
+CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
+CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
+CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)
+
+The "Get Supported Features" return data may be filtered by the kernel driver to
+drop any features that are forbidden by the kernel or being exclusively used by
+the kernel. The "Get Supported Featuers" command falls under the fwctl policy
+of FWCTL_RPC_CONFIGURATION.
+
+The "Get Features" commands falls under the policy FWCTL_RPC_DEBUG_READ_ONLY.
+
+For "Set Feature" command, the access policy currently is broken down into two
+categories depending on the set feature effects reported by the device. If the
+set feature will cause immediate change to the device, the fwctl access policy
+must be FWCTL_RPC_DEBUG_WRITE_FULL. The effects for this level are
+"immediate config change", "immediate data change", "immediate policy change",
+or "immediate log change" for the set effects mask. If the effects are "config
+change with cold reset" or "config change with conventional reset", then the
+fwctl access policy must be FWCTL_RPC_DEBUG_WRITE or higher.
+
+fwctl cxl User API
+==================
+
+.. kernel-doc:: include/uapi/fwctl/cxl.h
+
+1. Driver info query
+--------------------
+
+First step for the app is to issue the ioctl(FWCTL_CMD_INFO) in order to retrieve
+the information from the driver. A ``struct fwctl_info`` needs to be filled out
+with the ``fwctl_info.out_device_type`` set to ``FWCTL_DEVICE_TYPE_CXL``. The
+return data should be ``struct fwctl_info_cxl`` that contains the number of
+commands supported by the driver.
+
+2. Hardware command info query
+------------------------------
+
+Second step is to retrieve the command information from the driver. A ``struct
+fwctl_hw_info`` is sent with the ioctl(FWCTL_CMD_HW_INFO).
+``fwctl_hw_info.nr_cmds`` should be set to the number retrieved from the driver
+info query. A user buffer of ``struct fwctl_hw_info_out`` including an inlined
+array of ``struct fwctl_hw_info_out`` numbered of nr_cmds should be allocated and
+set as ``fwctl_hw_info.out_device_data``. For each command supported, its
+relevant information is returned such as the driver command id, the opcode it
+supports, the effect reported by the device if it's a write command, and its
+expected input and output sizes.
+
+3. Send hardware command
+------------------------
+
+Last step is to send hardware command to the driver from user space via
+ioctl(FWCTL_RPC). A ``struct fwctl_rpc_cxl`` should be pointed to by
+``fwctl_rpc.in``. ``struct fwctl_rpc_cxl.in_payload`` should point to
+the hardware input structure that is defined by the CXL spec. ``fwctl_rpc.out``
+points to the buffer that contains a ``struct fwctl_rpc_cxl_out`` that includes
+the hardware output data inlined as ``fwctl_rpc_cxl_out.payload``.
+
+For a get/set feature command, it is recommended to run the "get supported
+features" command first to retrieve the desired feature information first
+before sending a get or set feature command. For a "set feature" command,
+the retrieved info contains an effects field that details the resulting
+"set feature" command will trigger. That should inform the user whether
+the system is configured to allowed the "set feature" command or not.
+
+
+fwctl cxl Kernel API
+====================
+
+.. kernel-doc:: drivers/fwctl/cxl/cxl.c
+   :export:
+.. kernel-doc:: include/cxl/mailbox.h
diff --git a/Documentation/userspace-api/fwctl/index.rst b/Documentation/userspace-api/fwctl/index.rst
index 06959fbf1547..d9d40a468a31 100644
--- a/Documentation/userspace-api/fwctl/index.rst
+++ b/Documentation/userspace-api/fwctl/index.rst
@@ -10,3 +10,4 @@ to securely construct and execute RPCs inside device firmware.
    :maxdepth: 1
 
    fwctl
+   fwctl-cxl
-- 
2.47.0

Re: [RFC PATCH v2 19/20] fwctl: Move fwctl documentation to its own directory

[Jason Gunthorpe]

On Fri, Nov 15, 2024 at 02:25:52PM -0700, Dave Jiang wrote:
> Move fwctl.rst to its own directory in prepraration for adding more
> documentation.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

I folded this into my tree

Thanks,
Jason

Re: [RFC PATCH v2 09/20] fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands

[Jason Gunthorpe]

On Fri, Nov 15, 2024 at 02:25:42PM -0700, Dave Jiang wrote:
> @@ -1721,7 +1742,25 @@ int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host)
>  	mutex_init(&cxl_mbox->mbox_mutex);
>  	rcuwait_init(&cxl_mbox->mbox_wait);
>  
> -	return 0;
> +	fwctl->cxl_mbox = cxl_mbox;
> +	dev = &fwctl->dev;
> +	device_initialize(dev);
> +	device_set_pm_not_required(dev);
> +	dev->parent = host;
> +	dev->bus = &cxl_bus_type;
> +	dev->type = &cxl_fwctl_type;
> +
> +	rc = device_add(dev);
> +	if (rc)
> +		goto err;

If you don't call dev_set_name(), don't you have to set dev->id to
something unique?

> diff --git a/drivers/fwctl/Kconfig b/drivers/fwctl/Kconfig
> index e5ee2d46d431..e49903a9d0d3 100644
> --- a/drivers/fwctl/Kconfig
> +++ b/drivers/fwctl/Kconfig
> @@ -19,5 +19,14 @@ config FWCTL_MLX5
>  	  This will allow configuration and debug tools to work out of the box on
>  	  mainstream kernel.
>  
> +	  If you don't know what to do here, say N.
> +
> +config FWCTL_CXL
> +	tristate "CXL fwctl driver"
> +	depends on CXL_BUS
> +	help
> +	  CXLCTL provides interface for the user process to access user allowed
> +	  mailbox commands for CXL device.
> +
>  	  If you don't know what to do here, say N.
>  endif

Keep sorted

> diff --git a/drivers/fwctl/Makefile b/drivers/fwctl/Makefile
> index 1c535f694d7f..bd356e6f2e5a 100644
> --- a/drivers/fwctl/Makefile
> +++ b/drivers/fwctl/Makefile
> @@ -1,5 +1,6 @@
>  # SPDX-License-Identifier: GPL-2.0
>  obj-$(CONFIG_FWCTL) += fwctl.o
>  obj-$(CONFIG_FWCTL_MLX5) += mlx5/
> +obj-$(CONFIG_FWCTL_CXL) += cxl/

Keep sorted

Jason

Re: [RFC PATCH v2 10/20] fwctl/cxl: Add support for get driver information

[Jason Gunthorpe]

On Fri, Nov 15, 2024 at 02:25:43PM -0700, Dave Jiang wrote:

> +/**
> + * cxl_mailbox_user_commands_supported() - Return number of user mailbox
> + *                                         commands supported.
> + * @cxl_mbox: cxl mailbox context
> + *
> + * Return: number of commands supported
> + */
> +int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox)
> +{
> +	int nr_cmds = 0;
> +
> +	for (int i = 0; i < FWCTL_CXL_MAX_COMMANDS; i++) {
> +		nr_cmds += !!fwctl_cxl_find_command(cxl_mbox,
> +						    fwctl_command_sets[i]);

I'd just use an if here, adding a boolean is weird

> +/**
> + * struct fwctl_info_cxl - ioctl(FWCTL_INFO) out_device_data
> + * @uctx_caps: The number of commands the driver and device supports

Comment needs an update to nr_commands

Maybe write a few words about what userspace is supposed to do with
nr_commands?

Jason

Re: [RFC PATCH v2 09/20] fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands

[Dave Jiang]

On 11/20/24 11:01 AM, Jason Gunthorpe wrote:
> On Fri, Nov 15, 2024 at 02:25:42PM -0700, Dave Jiang wrote:
>> @@ -1721,7 +1742,25 @@ int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host)
>>  	mutex_init(&cxl_mbox->mbox_mutex);
>>  	rcuwait_init(&cxl_mbox->mbox_wait);
>>  
>> -	return 0;
>> +	fwctl->cxl_mbox = cxl_mbox;
>> +	dev = &fwctl->dev;
>> +	device_initialize(dev);
>> +	device_set_pm_not_required(dev);
>> +	dev->parent = host;
>> +	dev->bus = &cxl_bus_type;
>> +	dev->type = &cxl_fwctl_type;
>> +
>> +	rc = device_add(dev);
>> +	if (rc)
>> +		goto err;
> 
> If you don't call dev_set_name(), don't you have to set dev->id to
> something unique?

Yep. I missed that and found it yesterday while testing.

> 
>> diff --git a/drivers/fwctl/Kconfig b/drivers/fwctl/Kconfig
>> index e5ee2d46d431..e49903a9d0d3 100644
>> --- a/drivers/fwctl/Kconfig
>> +++ b/drivers/fwctl/Kconfig
>> @@ -19,5 +19,14 @@ config FWCTL_MLX5
>>  	  This will allow configuration and debug tools to work out of the box on
>>  	  mainstream kernel.
>>  
>> +	  If you don't know what to do here, say N.
>> +
>> +config FWCTL_CXL
>> +	tristate "CXL fwctl driver"
>> +	depends on CXL_BUS
>> +	help
>> +	  CXLCTL provides interface for the user process to access user allowed
>> +	  mailbox commands for CXL device.
>> +
>>  	  If you don't know what to do here, say N.
>>  endif
> 
> Keep sorted
> 

ok

>> diff --git a/drivers/fwctl/Makefile b/drivers/fwctl/Makefile
>> index 1c535f694d7f..bd356e6f2e5a 100644
>> --- a/drivers/fwctl/Makefile
>> +++ b/drivers/fwctl/Makefile
>> @@ -1,5 +1,6 @@
>>  # SPDX-License-Identifier: GPL-2.0
>>  obj-$(CONFIG_FWCTL) += fwctl.o
>>  obj-$(CONFIG_FWCTL_MLX5) += mlx5/
>> +obj-$(CONFIG_FWCTL_CXL) += cxl/
> 
> Keep sorted

ok

> 
> Jason

Re: [RFC PATCH v2 15/20] fwctl/cxl: Add support for fwctl RPC command to enable CXL feature commands

[Jason Gunthorpe]

On Fri, Nov 15, 2024 at 02:25:48PM -0700, Dave Jiang wrote:
> +static bool cxlctl_validate_set_features(struct cxl_mailbox *cxl_mbox,
> +					 const struct fwctl_rpc_cxl *rpc_in,
> +					 enum fwctl_rpc_scope scope)
> +{
> +	struct cxl_feat_entry *feat;
> +	bool found = false;
> +	uuid_t uuid;
> +	u16 effects, mask;
> +
> +	if (rpc_in->op_size < sizeof(struct cxl_set_feature_input))
> +		return false;
> +
> +	if (copy_from_user(&uuid, u64_to_user_ptr(rpc_in->in_payload),
> +			   sizeof(uuid)))
> +		return false;
> +
> +	for (int i = 0; i < cxl_mbox->num_features; i++) {
> +		feat = &cxl_mbox->entries[i];
> +		if (uuid_equal(&uuid, &feat->uuid)) {
> +			found = true;
> +			break;
> +		}
> +	}
> +
> +	if (!found)
> +		return false;
> +
> +	effects = le16_to_cpu(feat->effects);
> +	/* Currently no user background command support */
> +	if (effects & CXL_CMD_BACKGROUND)
> +		return false;
> +
> +	mask = CXL_CMD_CONFIG_CHANGE_IMMEDIATE |
> +	       CXL_CMD_DATA_CHANGE_IMMEDIATE |
> +	       CXL_CMD_POLICY_CHANGE_IMMEDIATE |
> +	       CXL_CMD_LOG_CHANGE_IMMEDIATE;
> +	if (effects & mask && scope >= FWCTL_RPC_DEBUG_WRITE_FULL)
> +		return true;

I'd put brackets (effects & mask) &&

Jason

Re: [RFC PATCH v2 11/20] fwctl: FWCTL_HW_INFO to return hardware information

[Jason Gunthorpe]

On Fri, Nov 15, 2024 at 02:25:44PM -0700, Dave Jiang wrote:
> --- a/include/linux/fwctl.h
> +++ b/include/linux/fwctl.h
> @@ -47,6 +47,12 @@ struct fwctl_ops {
>  	 * ignore length on input, the core code will handle everything.
>  	 */
>  	void *(*info)(struct fwctl_uctx *uctx, size_t *length);
> +	/**
> +	 * @hw_info: Implement FWCTL_HW_INFO. Return a kmalloc() memory that is
> +	 * copied to out_device_dataa. On input commands indicates the number

Spelling 'data'

> diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
> index 4e4d30104667..7334907e27c1 100644
> --- a/include/uapi/fwctl/fwctl.h
> +++ b/include/uapi/fwctl/fwctl.h
> @@ -37,7 +37,8 @@
>  enum {
>  	FWCTL_CMD_BASE = 0,
>  	FWCTL_CMD_INFO = 0,
> -	FWCTL_CMD_RPC = 1,
> +	FWCTL_CMD_HW_INFO = 1,
> +	FWCTL_CMD_RPC = 2,
>  };

Let's follow uapi practicies from the start, give HW_INFO 2 instead of
renumbering rpc (or we need to order this patch before RPC)

> +/**
> + * struct fwctl_command_info - Hardware command information returned from a query.
> + * @id: Driver ID number for the command
> + * @opcode: Hardware command opcode
> + * @set_effects: Effects to the OS and hardware when command is executed.
> + *               Provided by the device.

This seems like it needs a common set of bit definitions if it is
going to be in a common struct

> + * @size_in: Expected input size, or ~0U if variable length.
> + * @size_out: Expected output size, or ~0U if variable length.
> + *
> + * Represents a single command that is supported by both the driver and the
> + * hardware. This is returned as part of an array from the FWCTL_HW_INFO ioctl.
> + */
> +struct fwctl_command_info {
> +	__u32 id;
> +	__u16 opcode;
> +	__u16 effects;
> +	__u32 size_in;
> +	__u32 size_out;
> +};

What is the difference btween id and opcode? That should be explained
in the comment

I'd suggest to make the raw HW id to be u64 'just in case'

> +/**
> + * struct fwctl_hw_info - ioctl(FWCTL_HW_INFO)
> + * @size: sizeof(struct fwctl_hw_info)
> + * @flags: Must be 0
> + * @device_data_len: On input the length of the out_device_data memory. On
> + *	output the size of the kernel's device_data which may be larger or
> + *	smaller than the input. Maybe 0 on input.
> + * @nr_cmds: Number of commands requested. 0 returns only nr_cmds in output.
> + * @out_device_data: Pointer to a memory of device_data_len bytes. Kernel will
> + *	fill the entire memory, zeroing as required.
> + *
> + * Returns hardware commands information about this fwctl instance.
> + */
> +struct fwctl_hw_info {
> +	__u32 size;
> +	__u32 flags;
> +	__u32 device_data_len;
> +	__u32 nr_cmds;
> +	__aligned_u64 out_device_data;
> +};

It isn't really device data, so this should be called out_hw_info

Jason

Re: [RFC PATCH v2 0/20] fwctl/cxl: Add CXL feature commands support via fwctl

[Jason Gunthorpe]

On Fri, Nov 15, 2024 at 02:25:33PM -0700, Dave Jiang wrote:
> v2:
> - Dropped 1/13 and 2/13 from previous version. Merged upstream already.
> - Combined changes from Shiju for "get supported features"
> - Addressed comments from Jonathan and Jason
> - See specific changes in individual patch revision history
> - Added hardware command info to FWCTL
> - Added filtering to set feature command
> - Added documentation

I looked through it, and I think it is good enough that the next
posting should not be RFC from my perspective..

The amount of CXL core patches has reduced quite a lot, it would be good to
order the patches so CXL was grouped together at the start, that would
make it easier to create a shared branch down the road

For instance very roughly something like:

>       cxl: Refactor user ioctl command path from mds to mailbox
>       cxl: Add Get Supported Features command for kernel usage
>       cxl/mbox: Add GET_FEATURE mailbox command
>       cxl: Add Get Feature command support for user submission
>       cxl/mbox: Add SET_FEATURE mailbox command
>       cxl: Add Set Feature command support for user submission
>       cxl: Move cxl_driver related bits to be usable by external drivers
>       cxl: Save Command Effects Log (CEL) effects for enabled commands
>       cxl: Move defines and error codes from cxlmem.h to cxl/mailbox.h
>       cxl/test: Add Get Supported Features mailbox command support
>       cxl/test: Add Get Feature support to cxl_test
>       cxl/test: Add Set Feature support to cxl_test

>       fwctl: FWCTL_HW_INFO to return hardware information
>       fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands
>       fwctl/cxl: Add support for get driver information
>       fwctl/cxl: Add hw_info callback
>       fwctl/cxl: Add support for fwctl RPC command to enable CXL feature commands
>       fwctl/cxl: Add support to filter exclusive features
>       fwctl/cxl: Add documentation to FWCTL CXL

Thanks,
Jason

Re: [RFC PATCH v2 01/20] cxl: Refactor user ioctl command path from mds to mailbox

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:34 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> With 'struct cxl_mailbox' context introduced, the helper functions
> cxl_query_cmd() and cxl_send_cmd() can take a cxl_mailbox directly
> rather than a cxl_memdev parameter. Refactor to use cxl_mailbox
> directly.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

To me this stands on it's own as a good change.

Re: [RFC PATCH v2 02/20] cxl: Add Get Supported Features command for kernel usage

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:35 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
> The command retrieve the list of supported device-specific features
> (identified by UUID) and general information about each Feature.
> 
> The driver will retrieve the feature entries in order to make checks and
> provide information for the Get Feature and Set Feature command. One of
> the main piece of information retrieved are the effects a Set Feature
> command would have for a particular feature.
> 
> Co-developed-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

A few things inline.

> ---
> 
> v2:
> - fix feature entry pointer math.
> - free mbox_out in cxl_get_supported_features(). (Shiju, Jonathan)
> - replace sizeof(struct) with feat_size. (Shiju)
> - replace reserved in feature structs with u8 type. (Shiju)
> - change cxl_feat_entry->effects to set_effects. (Shiju)
> - rearrange assigment of cxl_mbox->entries. (Jonathan)
> - Separate no features from actual error. (Jonathan)
> - Fix missing kdoc for entries. (Jonathan)
> ---
>  drivers/cxl/core/mbox.c      | 178 +++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxlmem.h         |  31 ++++++
>  drivers/cxl/pci.c            |   4 +
>  include/cxl/mailbox.h        |   4 +
>  include/uapi/linux/cxl_mem.h |   1 +
>  5 files changed, 218 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 880ac1dba3cc..4ba56f3d5a65 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c

> +int cxl_get_supported_features(struct cxl_dev_state *cxlds)
> +{
> +	struct cxl_mbox_get_sup_feats_out *mbox_out __free(kvfree) = NULL;

If there is a reason this isn't inline, definitely needs a comment.

Otherwise move it inline as per the comments in cleanup.h


> +	int remain_feats, max_size, max_feats, start, rc;
> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
> +	int feat_size = sizeof(struct cxl_feat_entry);
> +	struct cxl_mbox_get_sup_feats_in mbox_in;
> +	int hdr_size = sizeof(*mbox_out);
> +	struct cxl_mbox_cmd mbox_cmd;
> +	struct cxl_mem_command *cmd;
> +	struct cxl_feat_entry *entry;
> +
> +	/* Get supported features is optional, need to check */
> +	cmd = cxl_mem_find_command(CXL_MBOX_OP_GET_SUPPORTED_FEATURES);
> +	if (!cmd)
> +		return -EOPNOTSUPP;
> +	if (!test_bit(cmd->info.id, cxl_mbox->enabled_cmds))
> +		return -EOPNOTSUPP;
> +
> +	rc = cxl_get_supported_features_count(cxlds);
> +	if (rc)
> +		return rc;
> +
> +	if (!cxl_mbox->num_features) {
> +		dev_dbg(cxl_mbox->host, "No CXL features enumerated.\n");
> +		return 0;
> +	}
> +
> +	struct cxl_feat_entry *entries __free(kvfree) =
> +		kvmalloc(cxl_mbox->num_features * feat_size, GFP_KERNEL);
> +
> +	if (!entries)
> +		return -ENOMEM;
> +
> +	max_size = cxl_mbox->payload_size - hdr_size;
> +	/* max feat entries that can fit in mailbox max payload size */
> +	max_feats = max_size / feat_size;
> +	entry = &entries[0];
> +
> +	mbox_out = kvmalloc(cxl_mbox->payload_size, GFP_KERNEL);
> +	if (!mbox_out)
> +		return -ENOMEM;
> +
> +	start = 0;
> +	remain_feats = cxl_mbox->num_features;
> +	do {
> +		int retrieved, alloc_size, copy_feats;
> +		int num_entries;
> +
> +		if (remain_feats > max_feats) {
> +			alloc_size = sizeof(*mbox_out) + max_feats * feat_size;
> +			remain_feats = remain_feats - max_feats;
> +			copy_feats = max_feats;
> +		} else {
> +			alloc_size = sizeof(*mbox_out) + remain_feats * feat_size;
> +			copy_feats = remain_feats;
> +			remain_feats = 0;
> +		}
> +
> +		memset(&mbox_in, 0, sizeof(mbox_in));
> +		mbox_in.count = cpu_to_le32(alloc_size);
> +		mbox_in.start_idx = cpu_to_le16(start);
> +		memset(mbox_out, 0, alloc_size);
> +		mbox_cmd = (struct cxl_mbox_cmd) {
> +			.opcode = CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
> +			.size_in = sizeof(mbox_in),
> +			.payload_in = &mbox_in,
> +			.size_out = alloc_size,
> +			.payload_out = mbox_out,
> +			.min_out = hdr_size,
> +		};
> +		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
> +		if (rc < 0)
> +			return rc;
> +
> +		if (mbox_cmd.size_out <= hdr_size) {
> +			rc = -ENXIO;
> +			goto err;
> +		}
> +
> +		/*
> +		 * Make sure retrieved out buffer is multiple of feature
> +		 * entries.
> +		 */
> +		retrieved = mbox_cmd.size_out - hdr_size;
> +		if (retrieved % feat_size) {
> +			rc = -ENXIO;
> +			goto err;
> +		}
> +
> +		num_entries = le16_to_cpu(mbox_out->num_entries);
> +		/*
> +		 * If the reported output entries * defined entry size !=
> +		 * retrieved output bytes, then the output package is incorrect.
> +		 */
> +		if (num_entries * feat_size != retrieved) {
> +			rc = -ENXIO;
> +			goto err;
> +		}
> +
> +		memcpy(entry, mbox_out->ents, retrieved);
> +		entry++;
> +		/*
> +		 * If the number of output entries is less than expected, add the
> +		 * remaining entries to the next batch.
> +		 */
> +		remain_feats += copy_feats - num_entries;
> +		start += num_entries;
> +	} while (remain_feats);
> +
> +	cxl_mbox->entries = no_free_ptr(entries);
> +	rc = devm_add_action_or_reset(cxl_mbox->host, cxl_free_features,
> +				      cxl_mbox->entries);
> +	if (rc)
> +		return rc;
> +
> +	return 0;
> +
> +err:
> +	cxl_mbox->num_features = 0;
> +	return rc;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_get_supported_features, CXL);

> diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> index a0a49809cd76..6685dd76985a 100644
> --- a/drivers/cxl/cxlmem.h
> +++ b/drivers/cxl/cxlmem.h

...

> +struct cxl_mbox_get_sup_feats_out {
> +	__le16 num_entries;
> +	__le16 supported_feats;
> +	u8 reserved[4];
> +	struct cxl_feat_entry ents[] __counted_by_le(supported_feats);
Wrong counted_by. That one is the total number of supported_feats.
num_entries is the one for the number in the output payload.

Might well have been me getting this wrong in earlier review ;(

Jonathan


> +} __packed;

Re: [RFC PATCH v2 03/20] cxl/test: Add Get Supported Features mailbox command support

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:36 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add cxl-test emulation of Get Supported Features mailbox command.
> Currently only adding a test feature with feature identifier of
> all f's for testing.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Hi Dave,

One endian issue inline. With that tidied up.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> +
> +static int mock_get_supported_features(struct cxl_mockmem_data *mdata,
> +				       struct cxl_mbox_cmd *cmd)
> +{
> +	struct cxl_mbox_get_sup_feats_in *in = cmd->payload_in;
> +	struct cxl_mbox_get_sup_feats_out *out = cmd->payload_out;
> +	struct cxl_feat_entry *feat;
> +	u16 start_idx, count;
> +
> +	if (cmd->size_out < sizeof(*out)) {
> +		cmd->return_code = CXL_MBOX_CMD_RC_PAYLOADLEN;
> +		return -EINVAL;
> +	}
> +
> +	/*
> +	 * Current emulation only supports 1 feature
> +	 */
> +	start_idx = le16_to_cpu(in->start_idx);
> +	if (start_idx != 0) {
> +		cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
> +		return -EINVAL;
> +	}
> +
> +	count = le16_to_cpu(in->count);
> +	if (count < sizeof(*out)) {
> +		cmd->return_code = CXL_MBOX_CMD_RC_PAYLOADLEN;
> +		return -EINVAL;
> +	}
> +
> +	out->supported_feats = cpu_to_le16(1);
> +	cmd->return_code = 0;
> +	if (count < sizeof(*out) + sizeof(*feat)) {
> +		out->num_entries = 0;
> +		return 0;
> +	}
> +
> +	out->num_entries = 1;

cpu_to_le16(1) I think.

> +	feat = out->ents;
> +	fill_feature_vendor_test(feat);
> +
> +	return 0;
> +}

Re: [RFC PATCH v2 05/20] cxl: Add Get Feature command support for user submission

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:38 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add enumeration of Get Feature mailbox command for the kernel to recognize
> the command being passed in from user space.
> 
> CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
> 
> The feature requested is identified by specific UUID.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Hmm. Should we push this into Shiju's series if thought is that will land
first?  Seems harmless but I find it hard to care that much ;)

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
>  drivers/cxl/core/mbox.c      | 1 +
>  include/uapi/linux/cxl_mem.h | 1 +
>  2 files changed, 2 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index f70bd7d28ff8..f49665d3b4d6 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -68,6 +68,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
>  	CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
>  	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
> +	CXL_CMD(GET_FEATURE, 0x15, CXL_VARIABLE_PAYLOAD, 0),
>  };
>  
>  /*
> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> index bd2535962f70..90dcb9723997 100644
> --- a/include/uapi/linux/cxl_mem.h
> +++ b/include/uapi/linux/cxl_mem.h
> @@ -51,6 +51,7 @@
>  	___C(CLEAR_LOG, "Clear Log"),					  \
>  	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
>  	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),		  \
> +	___C(GET_FEATURE, "Get Feature"),				  \
>  	___C(MAX, "invalid / last command")
>  
>  #define ___C(a, b) CXL_MEM_COMMAND_ID_##a

Re: [RFC PATCH v2 07/20] cxl: Add Set Feature command support for user submission

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:40 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add enumeration of Set Feature mailbox command for the kernel to recognize
> the command being passed in from user space.

This looks unwise to make available on the ioctl. 

> 
> CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)
> 
> The feature requested is identified by specific UUID.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/mbox.c      | 1 +
>  include/uapi/linux/cxl_mem.h | 1 +
>  2 files changed, 2 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 4b9abf9a5b2b..739444d34130 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -69,6 +69,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
>  	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
>  	CXL_CMD(GET_FEATURE, 0x15, CXL_VARIABLE_PAYLOAD, 0),
> +	CXL_CMD(SET_FEATURE, CXL_VARIABLE_PAYLOAD, 0, 0),
>  };
>  
>  /*
> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> index 90dcb9723997..469dd481795f 100644
> --- a/include/uapi/linux/cxl_mem.h
> +++ b/include/uapi/linux/cxl_mem.h
> @@ -52,6 +52,7 @@
>  	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
>  	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),		  \
>  	___C(GET_FEATURE, "Get Feature"),				  \
> +	___C(SET_FEATURE, "Set Feature"),				  \
>  	___C(MAX, "invalid / last command")
>  
>  #define ___C(a, b) CXL_MEM_COMMAND_ID_##a

Re: [RFC PATCH v2 08/20] cxl: Move cxl_driver related bits to be usable by external drivers

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:41 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Move the code related to cxl_driver to allow external drivers such as
> cxl_fwctl to utilize the cxl_bus.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Hmm. Will have to see if this can allow dropping of at least some
of the includes in the cxl_pmu driver 

#include "../cxl/cxlpci.h"
#include "../cxl/cxl.h"
currently which isn't pretty.  I can't remember exactly what we
need though and that can definitely follow this series as an
additional cleanup.

In general sensible thing to do.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> ---
>  drivers/cxl/cxl.h | 32 +-------------------------------
>  include/cxl/cxl.h | 37 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 38 insertions(+), 31 deletions(-)
>  create mode 100644 include/cxl/cxl.h
> 
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 5406e3ab3d4a..18e78b2c3612 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -11,6 +11,7 @@
>  #include <linux/log2.h>
>  #include <linux/node.h>
>  #include <linux/io.h>
> +#include <cxl/cxl.h>
>  
>  extern const struct nvdimm_security_ops *cxl_security_ops;
>  
> @@ -819,37 +820,6 @@ bool is_cxl_region(struct device *dev);
>  
>  extern struct bus_type cxl_bus_type;
>  
> -struct cxl_driver {
> -	const char *name;
> -	int (*probe)(struct device *dev);
> -	void (*remove)(struct device *dev);
> -	struct device_driver drv;
> -	int id;
> -};
> -
> -#define to_cxl_drv(__drv)	container_of_const(__drv, struct cxl_driver, drv)
> -
> -int __cxl_driver_register(struct cxl_driver *cxl_drv, struct module *owner,
> -			  const char *modname);
> -#define cxl_driver_register(x) __cxl_driver_register(x, THIS_MODULE, KBUILD_MODNAME)
> -void cxl_driver_unregister(struct cxl_driver *cxl_drv);
> -
> -#define module_cxl_driver(__cxl_driver) \
> -	module_driver(__cxl_driver, cxl_driver_register, cxl_driver_unregister)
> -
> -#define CXL_DEVICE_NVDIMM_BRIDGE	1
> -#define CXL_DEVICE_NVDIMM		2
> -#define CXL_DEVICE_PORT			3
> -#define CXL_DEVICE_ROOT			4
> -#define CXL_DEVICE_MEMORY_EXPANDER	5
> -#define CXL_DEVICE_REGION		6
> -#define CXL_DEVICE_PMEM_REGION		7
> -#define CXL_DEVICE_DAX_REGION		8
> -#define CXL_DEVICE_PMU			9
> -
> -#define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
> -#define CXL_MODALIAS_FMT "cxl:t%d"
> -
>  struct cxl_nvdimm_bridge *to_cxl_nvdimm_bridge(struct device *dev);
>  struct cxl_nvdimm_bridge *devm_cxl_add_nvdimm_bridge(struct device *host,
>  						     struct cxl_port *port);
> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
> new file mode 100644
> index 000000000000..07a2983275a0
> --- /dev/null
> +++ b/include/cxl/cxl.h
> @@ -0,0 +1,37 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/* Copyright(c) 2024 Intel Corporation. */
> +#ifndef __CXL_GLOBAL_H__
> +#define __CXL_GLOBAL_H__
> +
> +struct cxl_driver {
> +	const char *name;
> +	int (*probe)(struct device *dev);
> +	void (*remove)(struct device *dev);
> +	struct device_driver drv;
> +	int id;
> +};
> +
> +#define to_cxl_drv(__drv)      container_of_const(__drv, struct cxl_driver, drv)
> +
> +int __cxl_driver_register(struct cxl_driver *cxl_drv, struct module *owner,
> +			  const char *modname);
> +#define cxl_driver_register(x) __cxl_driver_register(x, THIS_MODULE, KBUILD_MODNAME)
> +void cxl_driver_unregister(struct cxl_driver *cxl_drv);
> +
> +#define module_cxl_driver(__cxl_driver) \
> +	module_driver(__cxl_driver, cxl_driver_register, cxl_driver_unregister)
> +
> +#define CXL_DEVICE_NVDIMM_BRIDGE	1
> +#define CXL_DEVICE_NVDIMM		2
> +#define CXL_DEVICE_PORT			3
> +#define CXL_DEVICE_ROOT			4
> +#define CXL_DEVICE_MEMORY_EXPANDER	5
> +#define CXL_DEVICE_REGION		6
> +#define CXL_DEVICE_PMEM_REGION		7
> +#define CXL_DEVICE_DAX_REGION		8
> +#define CXL_DEVICE_PMU			9
> +
> +#define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
> +#define CXL_MODALIAS_FMT "cxl:t%d"
> +
> +#endif

Re: [RFC PATCH v2 09/20] fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:42 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add a cxl fwctl driver to allow sending of CXL feature
> commands from userspace through as ioctls. Create a driver skeleton for
> initial setup.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Other than what Jason already raised (most of which I missed ;)
LGTM.  With that all resolved.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>


> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
> index 07a2983275a0..6bfd7942a3f7 100644
> --- a/include/cxl/cxl.h
> +++ b/include/cxl/cxl.h
> @@ -30,6 +30,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);
>  #define CXL_DEVICE_PMEM_REGION		7
>  #define CXL_DEVICE_DAX_REGION		8
>  #define CXL_DEVICE_PMU			9
> +#define CXL_DEVICE_FWCTL		10
Clash with HMU, but this will hopefully land well before that does and
not exactly a challenge to resolve!

J

Re: [RFC PATCH v2 07/20] cxl: Add Set Feature command support for user submission

[Dave Jiang]

On 11/21/24 10:49 AM, Jonathan Cameron wrote:
> On Fri, 15 Nov 2024 14:25:40 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
>> Add enumeration of Set Feature mailbox command for the kernel to recognize
>> the command being passed in from user space.
> 
> This looks unwise to make available on the ioctl. 

Do you mean the current memdev char dev ioctl? Maybe we can introduce a restricted_commands mask to block that on the legacy ioctl? Trying to make the two ioctl paths co-exist is a bit of a pain right now.

DJ
 
> 
>>
>> CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)
>>
>> The feature requested is identified by specific UUID.
>>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>>  drivers/cxl/core/mbox.c      | 1 +
>>  include/uapi/linux/cxl_mem.h | 1 +
>>  2 files changed, 2 insertions(+)
>>
>> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
>> index 4b9abf9a5b2b..739444d34130 100644
>> --- a/drivers/cxl/core/mbox.c
>> +++ b/drivers/cxl/core/mbox.c
>> @@ -69,6 +69,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
>>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
>>  	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
>>  	CXL_CMD(GET_FEATURE, 0x15, CXL_VARIABLE_PAYLOAD, 0),
>> +	CXL_CMD(SET_FEATURE, CXL_VARIABLE_PAYLOAD, 0, 0),
>>  };
>>  
>>  /*
>> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
>> index 90dcb9723997..469dd481795f 100644
>> --- a/include/uapi/linux/cxl_mem.h
>> +++ b/include/uapi/linux/cxl_mem.h
>> @@ -52,6 +52,7 @@
>>  	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
>>  	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),		  \
>>  	___C(GET_FEATURE, "Get Feature"),				  \
>> +	___C(SET_FEATURE, "Set Feature"),				  \
>>  	___C(MAX, "invalid / last command")
>>  
>>  #define ___C(a, b) CXL_MEM_COMMAND_ID_##a
> 

Re: [RFC PATCH v2 10/20] fwctl/cxl: Add support for get driver information

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:43 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add definition for fwctl_ops->info() to return driver information. The
> function will return the number of device mailbox commands supported by the
> fwctl char device.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Few minor things inline

Jonathan
> ---
> v2:
> - Change driver info to report number of commands supported by the driver and device.
> ---
>  drivers/cxl/core/mbox.c  | 46 ++++++++++++++++++++++++++++++++++++++++
>  drivers/fwctl/cxl/cxl.c  | 23 ++++++++++++++++++--
>  include/cxl/cxl.h        |  2 ++
>  include/cxl/mailbox.h    |  1 +
>  include/uapi/fwctl/cxl.h | 22 +++++++++++++++++++
>  5 files changed, 92 insertions(+), 2 deletions(-)
>  create mode 100644 include/uapi/fwctl/cxl.h
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 12758e763650..f464eb42f08a 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -118,6 +118,14 @@ static u8 security_command_sets[] = {
>  	0x46, /* Security Passthrough */
>  };
>  
> +#define FWCTL_CXL_MAX_COMMANDS 3

Use this to size the array? Will let us know if it gets
out of sync.

> +/* Command set that is allowed with FWCTL-CXL */
> +static u16 fwctl_command_sets[] = {
> +	CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
> +	CXL_MBOX_OP_GET_FEATURE,
> +	CXL_MBOX_OP_SET_FEATURE,
> +};
> +

>  static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
> index 6bfd7942a3f7..d0939e3bcbc0 100644
> --- a/include/cxl/cxl.h
> +++ b/include/cxl/cxl.h
> @@ -3,6 +3,8 @@
>  #ifndef __CXL_GLOBAL_H__
>  #define __CXL_GLOBAL_H__
>  
> +#include <linux/device.h>

Why in this patch?

> +
>  struct cxl_driver {
>  	const char *name;
>  	int (*probe)(struct device *dev);

Re: [RFC PATCH v2 11/20] fwctl: FWCTL_HW_INFO to return hardware information

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:44 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add an optional ioctl FWCTL_HW_INFO to pass command specific information
> to user space. An array of 'struct fwctl_command_info' will be returned
> from the ioctl. These commands are send to the driver via FWCTL_RPC call.
> The command info struct contains the command id, the related hardware
> opcode, input and output size for the command, and the effects the command
> has if it's a write command.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Minor stuff inline.

Jonathan


> diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
> index 4e4d30104667..7334907e27c1 100644
> --- a/include/uapi/fwctl/fwctl.h
> +++ b/include/uapi/fwctl/fwctl.h

>  enum fwctl_device_type {
> @@ -69,6 +70,61 @@ struct fwctl_info {
>  };
>  #define FWCTL_INFO _IO(FWCTL_TYPE, FWCTL_CMD_INFO)
>  
> +/**
> + * struct fwctl_command_info - Hardware command information returned from a query.
> + * @id: Driver ID number for the command
> + * @opcode: Hardware command opcode
> + * @set_effects: Effects to the OS and hardware when command is executed.
> + *               Provided by the device.

Run kernel doc over the files.  (wrong name)

> + * @size_in: Expected input size, or ~0U if variable length.
> + * @size_out: Expected output size, or ~0U if variable length.
> + *
> + * Represents a single command that is supported by both the driver and the
> + * hardware. This is returned as part of an array from the FWCTL_HW_INFO ioctl.
> + */
> +struct fwctl_command_info {
> +	__u32 id;
> +	__u16 opcode;
> +	__u16 effects;
> +	__u32 size_in;
> +	__u32 size_out;
> +};
> +
> +/**
> + * struct fwctl_hw_info_out - output struct for FWCTL_HW_INFO
> + * @nr_cmds: Number of commands for output
> + * @reserved: Reserved u32 for alignment.

As currently defined, why do we need to pad? I'm not against
padding just not understanding the comment.
It's more fun if there is a u64 in there (we all love 32 bit
x86 data alignment).

> + * @commands: Array of 'struct fwctl_command_info'
> + */
> +struct fwctl_hw_info_out {
> +	__u32 nr_cmds;
> +	__u32 reserved;
> +
> +	struct fwctl_command_info commands[] __counted_by(nr_cmds);
> +};

Re: [RFC PATCH v2 12/20] cxl: Save Command Effects Log (CEL) effects for enabled commands

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:45 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Store the command effects via what device provided via the CEL in the
> 'struct cxl_command_info' structure. Steal the upper 16 bits to store
> the effects value in order to keep user API compatibility.

Trivial comment inline.

> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/mbox.c      | 2 ++
>  include/uapi/fwctl/fwctl.h   | 4 ++--
>  include/uapi/linux/cxl_mem.h | 5 +++--
>  3 files changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index f464eb42f08a..fba6bdd30a82 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -795,6 +795,8 @@ static void cxl_walk_cel(struct cxl_memdev_state *mds, size_t size, u8 *cel)
>  			enabled++;
>  		}
>  
> +		cmd->info.effects = le16_to_cpu(cel_entry[i].effect);
> +
>  		dev_dbg(dev, "Opcode 0x%04x %s\n", opcode,
>  			enabled ? "enabled" : "unsupported by driver");
>  	}
> diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
> index 7334907e27c1..04da549cd4ae 100644
> --- a/include/uapi/fwctl/fwctl.h
> +++ b/include/uapi/fwctl/fwctl.h
> @@ -74,8 +74,8 @@ struct fwctl_info {
>   * struct fwctl_command_info - Hardware command information returned from a query.
>   * @id: Driver ID number for the command
>   * @opcode: Hardware command opcode
> - * @set_effects: Effects to the OS and hardware when command is executed.
> - *               Provided by the device.
> + * @effects: Effects to the OS and hardware when command is executed.
> + *           Provided by the device.

Wrong patch...

>   * @size_in: Expected input size, or ~0U if variable length.
>   * @size_out: Expected output size, or ~0U if variable length.
>   *
> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> index 469dd481795f..9dd37849c450 100644
> --- a/include/uapi/linux/cxl_mem.h
> +++ b/include/uapi/linux/cxl_mem.h
> @@ -118,6 +118,7 @@ static const __u8 cxl_deprecated_commands[]
>   *         the label-storage-area can not be written while the kernel is
>   *         actively managing that space.
>   *
> + * @effects: Command effects reported by the device.
>   * @size_in: Expected input size, or ~0 if variable length.
>   * @size_out: Expected output size, or ~0 if variable length.
>   *
> @@ -136,11 +137,11 @@ static const __u8 cxl_deprecated_commands[]
>  struct cxl_command_info {
>  	__u32 id;
>  
> -	__u32 flags;
> +	__u16 flags;
>  #define CXL_MEM_COMMAND_FLAG_MASK		GENMASK(1, 0)
>  #define CXL_MEM_COMMAND_FLAG_ENABLED		BIT(0)
>  #define CXL_MEM_COMMAND_FLAG_EXCLUSIVE		BIT(1)
> -
> +	__u16 effects;
>  	__u32 size_in;
>  	__u32 size_out;
>  };

Re: [RFC PATCH v2 13/20] fwctl/cxl: Add hw_info callback

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:46 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

Needs a bit more info.


> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

Otherwise trivial comment inline.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> diff --git a/drivers/fwctl/cxl/cxl.c b/drivers/fwctl/cxl/cxl.c
> index 5eb5eabf2bff..ce8960a9beaa 100644
> --- a/drivers/fwctl/cxl/cxl.c
> +++ b/drivers/fwctl/cxl/cxl.c
> @@ -53,6 +53,30 @@ static void *cxlctl_info(struct fwctl_uctx *uctx, size_t *length)
>  	return info;
>  }
>  
> +static void *cxlctl_hw_info(struct fwctl_uctx *uctx, int commands, size_t *out_len)
> +{
> +	struct cxlctl_uctx *cxlctl_uctx =
> +		container_of(uctx, struct cxlctl_uctx, uctx);
Some of this container stuff is starting to look very familiar.
Maybe worth some macros?

> +	struct fwctl_device *fwctl = uctx->fwctl;
> +	struct cxlctl_dev *cxlctl =
> +		container_of(fwctl, struct cxlctl_dev, fwctl);
> +	int rc;
> +
> +	if (commands > cxlctl_uctx->nr_commands)
> +		return ERR_PTR(-EINVAL);
> +
> +	void *out __free(kvfree) = kvzalloc(*out_len, GFP_KERNEL);
> +	if (!out)
> +		return ERR_PTR(-ENOMEM);
> +
> +	rc = cxl_mailbox_user_commands_info_get(cxlctl->mbox,
> +						commands, out, out_len);
> +	if (rc)
> +		return ERR_PTR(rc);
> +
> +	return_ptr(out);
> +}
> +

Re: [RFC PATCH v2 14/20] cxl: Move defines and error codes from cxlmem.h to cxl/mailbox.h

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:47 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Moving some internal definitions to cxl/mailbox.h in order to be accessed
> by FWCTL CXL driver.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
I'm not keen on the rename being in this patch which is otherwise
just a move.  If that can't get pushed back to earlier patch
then do it as a precursor.

Jonathan

> diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
> index f48eebb855f7..e753d5d1d708 100644
> --- a/include/cxl/mailbox.h
> +++ b/include/cxl/mailbox.h

> +/**
> + * struct cxl_feat_entry - CXL Spec r3.1 Table 8-97
> + * @uuid: Feature identifier
> + * @id: Feature Index
> + * @get_feat_size: Get Feature Size
> + * @set_feat_size: Set Feature Size
> + * @flags: Attribute Flags
> + * @get_feat_ver: Get Feature Version
> + * @set_feat_ver: Set Feature Version
> + * @reserved: reserved, must be 0
> + */
> +struct cxl_feat_entry {
> +	uuid_t uuid;
> +	__le16 id;
> +	__le16 get_feat_size;
> +	__le16 set_feat_size;
> +	__le32 flags;
> +	u8 get_feat_ver;
> +	u8 set_feat_ver;
> +	__le16 effects;
Not documented.

> +	u8 reserved[18];
> +} __packed;

> diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c
> index e0c7e49795ed..9103bac054b9 100644
> --- a/tools/testing/cxl/test/mem.c
> +++ b/tools/testing/cxl/test/mem.c
> @@ -1354,7 +1354,7 @@ static void fill_feature_vendor_test(struct cxl_feat_entry *feat)
>  	feat->flags = cpu_to_le32(BIT(5));
>  	feat->get_feat_ver = 1;
>  	feat->set_feat_ver = 1;
> -	feat->set_effects = cpu_to_le16(BIT(0) | BIT(9));
> +	feat->effects = cpu_to_le16(BIT(0) | BIT(9));

Feels like the rename should be in the earlier patch.

>  }
>  
>  static int mock_get_supported_features(struct cxl_mockmem_data *mdata,

Re: [RFC PATCH v2 0/20] fwctl/cxl: Add CXL feature commands support via fwctl

[Jonathan Cameron]

On Wed, 20 Nov 2024 14:57:20 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Fri, Nov 15, 2024 at 02:25:33PM -0700, Dave Jiang wrote:
> > v2:
> > - Dropped 1/13 and 2/13 from previous version. Merged upstream already.
> > - Combined changes from Shiju for "get supported features"
> > - Addressed comments from Jonathan and Jason
> > - See specific changes in individual patch revision history
> > - Added hardware command info to FWCTL
> > - Added filtering to set feature command
> > - Added documentation  
> 
> I looked through it, and I think it is good enough that the next
> posting should not be RFC from my perspective..
> 
> The amount of CXL core patches has reduced quite a lot, it would be good to
> order the patches so CXL was grouped together at the start, that would
> make it easier to create a shared branch down the road
> 
> For instance very roughly something like:
> 
> >       cxl: Refactor user ioctl command path from mds to mailbox
> >       cxl: Add Get Supported Features command for kernel usage
> >       cxl/mbox: Add GET_FEATURE mailbox command
> >       cxl: Add Get Feature command support for user submission
> >       cxl/mbox: Add SET_FEATURE mailbox command

Some of these are common to the EDAC RAS features set which
I'm really hoping finally lands next cycle.

So we will need to be a little careful in how we merge things.

Going to need to be careful how we handle this.  Maybe Dave should
pick up the feature stuff as an immutable branch after rc1 and
both series get rebuilt on top of that.  Need to discuss if
Borislav is happy with that for potential merge of the RAS
features series (EDAC).

Anyhow, time up for me today. I'll look at the rest of the series
tomorrow.

Jonathan

> >       cxl: Add Set Feature command support for user submission
> >       cxl: Move cxl_driver related bits to be usable by external drivers
> >       cxl: Save Command Effects Log (CEL) effects for enabled commands
> >       cxl: Move defines and error codes from cxlmem.h to cxl/mailbox.h
> >       cxl/test: Add Get Supported Features mailbox command support
> >       cxl/test: Add Get Feature support to cxl_test
> >       cxl/test: Add Set Feature support to cxl_test  
> 
> >       fwctl: FWCTL_HW_INFO to return hardware information
> >       fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands
> >       fwctl/cxl: Add support for get driver information
> >       fwctl/cxl: Add hw_info callback
> >       fwctl/cxl: Add support for fwctl RPC command to enable CXL feature commands
> >       fwctl/cxl: Add support to filter exclusive features
> >       fwctl/cxl: Add documentation to FWCTL CXL  
> 
> Thanks,
> Jason

Re: [RFC PATCH v2 07/20] cxl: Add Set Feature command support for user submission

[Jonathan Cameron]

On Thu, 21 Nov 2024 11:08:44 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> On 11/21/24 10:49 AM, Jonathan Cameron wrote:
> > On Fri, 15 Nov 2024 14:25:40 -0700
> > Dave Jiang <dave.jiang@intel.com> wrote:
> >   
> >> Add enumeration of Set Feature mailbox command for the kernel to recognize
> >> the command being passed in from user space.  
> > 
> > This looks unwise to make available on the ioctl.   
> 
> Do you mean the current memdev char dev ioctl? Maybe we can introduce a restricted_commands mask to block that on the legacy ioctl? Trying to make the two ioctl paths co-exist is a bit of a pain right now.
> 
Yes - I'd rather we didn't introduce new commands on the legacy ioctl
at all, but if we do keeping it to the definitely safe ones (GET_xxx etc)
would be better.

Another mask is a way to solve this, though feels like opt in is
going to be easier to maintain long term than opt out.
Mind you we can modify how exactly it is done later as not
critical yet.

Jonathan

> DJ
>  
> >   
> >>
> >> CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)
> >>
> >> The feature requested is identified by specific UUID.
> >>
> >> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> >> ---
> >>  drivers/cxl/core/mbox.c      | 1 +
> >>  include/uapi/linux/cxl_mem.h | 1 +
> >>  2 files changed, 2 insertions(+)
> >>
> >> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> >> index 4b9abf9a5b2b..739444d34130 100644
> >> --- a/drivers/cxl/core/mbox.c
> >> +++ b/drivers/cxl/core/mbox.c
> >> @@ -69,6 +69,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
> >>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
> >>  	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
> >>  	CXL_CMD(GET_FEATURE, 0x15, CXL_VARIABLE_PAYLOAD, 0),
> >> +	CXL_CMD(SET_FEATURE, CXL_VARIABLE_PAYLOAD, 0, 0),
> >>  };
> >>  
> >>  /*
> >> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> >> index 90dcb9723997..469dd481795f 100644
> >> --- a/include/uapi/linux/cxl_mem.h
> >> +++ b/include/uapi/linux/cxl_mem.h
> >> @@ -52,6 +52,7 @@
> >>  	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
> >>  	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),		  \
> >>  	___C(GET_FEATURE, "Get Feature"),				  \
> >> +	___C(SET_FEATURE, "Set Feature"),				  \
> >>  	___C(MAX, "invalid / last command")
> >>  
> >>  #define ___C(a, b) CXL_MEM_COMMAND_ID_##a  
> >   
> 
> 

Re: [RFC PATCH v2 15/20] fwctl/cxl: Add support for fwctl RPC command to enable CXL feature commands

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:48 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> fwctl provides a fwctl_ops->fw_rpc() callback in order to issue ioctls
> to a device. The cxl fwctl driver will start by supporting the CXL
> feature commands: Get Supported Features, Get Feature, and Set Feature.
> 
> The fw_rpc() callback provides 'enum fwctl_rpc_scope' parameter where
> it indicates the security scope of the call. The Get Supported Features
> and Get Feature calls can be executed with the scope of
> FWCTL_RPC_DEBUG_READ_ONLY. The Set Feature call is gated by the effects
> of the feature reported by Get Supported Features call for the specific
> feature.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

A few comments inline

Jonathan


>  
> -int cxl_send_cmd(struct cxl_mailbox *cxl_mbox, struct cxl_send_command __user *s)
> +static int cxl_send_cmd(struct cxl_mailbox *cxl_mbox, struct cxl_send_command *send,
> +			struct cxl_mbox_cmd *mbox_cmd)
> +{
> +	int rc;
> +
> +	rc = cxl_validate_cmd_from_user(mbox_cmd, cxl_mbox, send);
> +	if (rc)
> +		return rc;
> +
> +	rc = handle_mailbox_cmd_from_user(cxl_mbox, mbox_cmd, send->out.payload,
> +					  &send->out.size, &send->retval);
> +	if (rc)
> +		return rc;

return handle_mailbox_cmd_from_user()

> +
> +	return 0;
> +}
> +
> +/**
> + * handle_mailbox_cmd_from_fwctl() - Dispatch a mailbox command for userspace.
> + * @cxl_mbox: The mailbox context for the operation.
> + * @mbox_cmd: The validated mailbox command.
> + *
> + * Return:
> + *  * %0	- Mailbox transaction succeeded. This implies the mailbox
> + *		  protocol completed successfully not that the operation itself
> + *		  was successful.
> + *  * %-ENOMEM  - Couldn't allocate a bounce buffer.
> + *  * %-EINTR	- Mailbox acquisition interrupted.
> + *  * %-EXXX	- Transaction level failures.
> + *
> + * Dispatches a mailbox command on behalf of a userspace request.
> + * The output payload is copied to userspace by fwctl.
> + *
> + * See cxl_send_cmd().
> + */
> +static int handle_mailbox_cmd_from_fwctl(struct cxl_mailbox *cxl_mbox,
> +					 struct cxl_mbox_cmd *mbox_cmd)
> +{
> +	struct device *dev = cxl_mbox->host;
Not sure I'd bother for one use.

> +	struct fwctl_rpc_cxl_out *orig_out;
> +	int rc;
> +
> +	/*
> +	 * Save the payload_out pointer and move it to where hardware output
> +	 * can be copied to.
> +	 */
> +	orig_out = mbox_cmd->payload_out;
> +	mbox_cmd->payload_out = (void *)orig_out + sizeof(*orig_out);
Messing around with void * seems awkward.
Isn't this just
	mbox_cmd->payload_out = orig_out + 1;

or (I prefer the above)
	mbox_cmd->payload_out += sizeof(*orig_out);

> +
> +	dev_dbg(dev,
> +		"Submitting %s command for user\n"
> +		"\topcode: %x\n"
> +		"\tsize: %zx\n",
> +		cxl_mem_opcode_to_name(mbox_cmd->opcode),
> +		mbox_cmd->opcode, mbox_cmd->size_in);
> +
> +	rc = cxl_mbox->mbox_send(cxl_mbox, mbox_cmd);
> +	if (rc)
> +		return rc;
> +
> +	orig_out->retval = mbox_cmd->return_code;
> +	mbox_cmd->payload_out = (void *)orig_out;

No need to cast a pointer to void *

> +
> +	return 0;
> +}
> +
> +int cxl_mbox_send_cmd(struct cxl_mailbox *cxl_mbox,
> +		      struct fwctl_rpc_cxl *rpc_in,
> +		      struct cxl_mbox_cmd *mbox_cmd, size_t *out_len)
> +{
> +	struct cxl_send_command send_cmd = {
> +		.id = rpc_in->id,
> +		.flags = rpc_in->flags,
> +		.in.size = rpc_in->op_size,
> +		.in.payload = rpc_in->in_payload,
> +		.out.size = *out_len,
> +	};
> +	struct cxl_mem_command *cmd;
> +	int rc;
> +
> +	cmd = cxl_mem_find_command_by_id(rpc_in->id);
> +	if (!cmd)
> +		return -EINVAL;
> +	send_cmd.raw.opcode = cmd->opcode;
> +
> +	rc = cxl_validate_cmd_from_fwctl(mbox_cmd, cxl_mbox, &send_cmd);

I'd not really expect to see fwctl specific code in a function called
simply cxl_mbox_send_cmd().  Perhaps a rename is appropriate?

> +	if (rc)
> +		return rc;
> +
> +	rc = handle_mailbox_cmd_from_fwctl(cxl_mbox, mbox_cmd);
> +	if (rc)
> +		return rc;
> +
> +	guard(rwsem_read)(&cxl_memdev_rwsem);
> +	rc = cxl_mbox->mbox_send(cxl_mbox, mbox_cmd);
> +	if (rc)
> +		return rc;
> +
> +	*out_len = mbox_cmd->size_out;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_mbox_send_cmd, CXL);

> diff --git a/drivers/fwctl/cxl/cxl.c b/drivers/fwctl/cxl/cxl.c
> index ce8960a9beaa..164f6774a2c1 100644
> --- a/drivers/fwctl/cxl/cxl.c
> +++ b/drivers/fwctl/cxl/cxl.c
> @@ -77,11 +77,106 @@ static void *cxlctl_hw_info(struct fwctl_uctx *uctx, int commands, size_t *out_l
>  	return_ptr(out);
>  }
>  
> +static bool cxlctl_validate_set_features(struct cxl_mailbox *cxl_mbox,
> +					 const struct fwctl_rpc_cxl *rpc_in,
> +					 enum fwctl_rpc_scope scope)
> +{
> +	struct cxl_feat_entry *feat;
> +	bool found = false;
> +	uuid_t uuid;
> +	u16 effects, mask;

Pick an order. Reverse xmas tree maybe.


> +
> +static bool cxlctl_validate_hw_cmds(struct cxl_mailbox *cxl_mbox,
> +				    const struct fwctl_rpc_cxl *rpc_in,
> +				    enum fwctl_rpc_scope scope)
> +{
> +	struct cxl_mem_command *cmd;
> +
> +	/*
> +	 * Only supporting feature commands for now.
> +	 */
> +	if (!cxl_mbox->num_features)
> +		return false;
> +
> +	cmd = cxl_get_mem_command_for_fwctl(cxl_mbox, rpc_in->id);
> +	if (!cmd)
> +		return false;
> +
> +	switch (cmd->opcode) {
> +	case CXL_MBOX_OP_GET_SUPPORTED_FEATURES:
> +		if (scope >= FWCTL_RPC_CONFIGURATION)
> +			return true;
> +		return false;

		return scope >= FWCTL_RFC_CONFIGURATION;

> +	case CXL_MBOX_OP_GET_FEATURE:
> +		if (scope >= FWCTL_RPC_DEBUG_READ_ONLY)
> +			return true;
> +		return false;
		return scope >= FWCTL_RPC_DEBUG_READ_ONLY;

> +	case CXL_MBOX_OP_SET_FEATURE:
> +		return cxlctl_validate_set_features(cxl_mbox, rpc_in, scope);
> +	default:
> +		return false;
> +	}
> +}
> +

>  static const struct fwctl_ops cxlctl_ops = {
> diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
> index e753d5d1d708..3e5e9c9362f5 100644
> --- a/include/cxl/mailbox.h
> +++ b/include/cxl/mailbox.h
> @@ -193,5 +193,11 @@ int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host);
>  int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox);
>  int cxl_mailbox_user_commands_info_get(struct cxl_mailbox *cxl_mbox, int nr_cmds,
>  				       void *outbuf, size_t *out_len);
> +struct cxl_mem_command *cxl_get_mem_command(u32 id);
> +struct cxl_mem_command *
> +cxl_get_mem_command_for_fwctl(struct cxl_mailbox *cxl_mbox, u32 id);
> +int cxl_mbox_send_cmd(struct cxl_mailbox *cxl_mbox,
> +		      struct fwctl_rpc_cxl *rpc_in,
> +		      struct cxl_mbox_cmd *mbox_cmd, size_t *out_len);
>  
>  #endif
> diff --git a/include/uapi/fwctl/cxl.h b/include/uapi/fwctl/cxl.h
> index a32c4c752db6..99804fc72f28 100644
> --- a/include/uapi/fwctl/cxl.h
> +++ b/include/uapi/fwctl/cxl.h


> +/**
> + * struct fwctl_rpc_cxl_out - ioctl9FWCTL_RPC) output for CXL
(FWCTL_RPC)  9/(

> + * @size: Size of the output payload
> + * @retval: Return value from device
> + * @payload: Return data from device
> + */
> +struct fwctl_rpc_cxl_out {
> +	__u32 size;
> +	__u32 retval;
> +	__u8 payload[];
Can we add counted_by to uapi headers?

Seems there are some existing examples so I guess that is fine.

> +};
> +
>  #endif
> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> index 9dd37849c450..4e7c8c03cfe8 100644
> --- a/include/uapi/linux/cxl_mem.h
> +++ b/include/uapi/linux/cxl_mem.h
> @@ -234,4 +234,16 @@ struct cxl_send_command {
>  	} out;
>  };
>  
> +/*
> + * CXL spec r3.1 Table 8-101 Set Feature Input Payload
> + */
> +struct cxl_set_feature_input {
> +	__u8 uuid[16];
> +	__u32 flags;
> +	__u16 offset;
> +	__u8 version;
> +	__u8 reserved[9];

This first bit matches with cxl_mbox_set_feat_hdr.
Good avoid the duplication.


> +	__u8 data[];
> +} __packed;
> +
>  #endif

Re: [RFC PATCH v2 16/20] fwctl/cxl: Add support to filter exclusive features

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:49 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add support to allow filtering of CXL features that are exclusive to the
> kernel and make them not visible to user space. The "get supported
> features" mailbox command returned to the userspace is emulated and
> will skip the features that is marked exclusive for the kernel. The
> exclusion allows certain the feature setting of certain commands such
> as claimed by RAS to be exclusive to the kernel.

Well behaved software is only going to issue features that were returned
by get_supported_features.  However we need to block badly behaved software
too.  So a check is needed in the set_feature() / get_feature() paths.

A few queries inline but basically LGTM other than that.

Jonathan

> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/mbox.c | 115 ++++++++++++++++++++++++++++++++++++----
>  drivers/fwctl/cxl/cxl.c |   7 +--
>  include/cxl/features.h  |  52 ++++++++++++++++++
>  include/cxl/mailbox.h   |   8 ++-
>  4 files changed, 168 insertions(+), 14 deletions(-)
>  create mode 100644 include/cxl/features.h
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index e7c5c709ac79..12ace2951f7c 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -5,6 +5,7 @@
>  #include <linux/ktime.h>
>  #include <linux/mutex.h>
>  #include <linux/unaligned.h>
> +#include <cxl/features.h>
>  #include <cxl/mailbox.h>
>  #include <cxlpci.h>
>  #include <cxlmem.h>
> @@ -897,6 +898,78 @@ static int handle_mailbox_cmd_from_fwctl(struct cxl_mailbox *cxl_mbox,
>  	return 0;
>  }
>  
> +/*
> + * Emulate the 'get supported features' mailbox command and only copy out the
> + * features that are not marked as exclusive for kernel.
> + */
> +static void cxl_mbox_get_supported_features_filtered(struct cxl_mailbox *cxl_mbox,
> +						     struct cxl_mbox_cmd *mbox_cmd)
> +{
> +	struct cxl_mbox_get_sup_feats_in *feat_in = mbox_cmd->payload_in;
> +	struct cxl_mbox_get_sup_feats_out *feat_out = mbox_cmd->payload_out;
> +	const int feat_out_size = sizeof(*feat_out);
Worth while?  I'd just use sizeof(*feat_out) inline.
Compiler will sort out out and it is self documenting.

> +	struct cxl_feat_entry *pos;
> +	const int feat_ent_size = sizeof(*pos);
Same for this.

> +	int out_count, ents, u;
> +	u32 count;
> +	u16 start;
> +
> +	count = le32_to_cpu(feat_in->count);
> +	start = le16_to_cpu(feat_in->start_idx);
> +	ents = count / sizeof(struct cxl_feat_entry);
> +	ents -= start;

You've lost me on this maths. ents is what?


> +	if (ents < 0) {
> +		mbox_cmd->size_out = 0;
> +		mbox_cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
> +		return;
> +	}
> +
> +	if (mbox_cmd->size_out < feat_out_size) {
> +		mbox_cmd->size_out = 0;
> +		mbox_cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
> +		return;
> +	}
> +
> +	feat_out->supported_feats =
> +		cpu_to_le16(cxl_mbox->num_user_feats);
> +	if (ents == 0) {
> +		feat_out->num_entries = cpu_to_le16(0);
> +		mbox_cmd->size_out = feat_out_size;
> +		mbox_cmd->return_code = CXL_MBOX_CMD_RC_SUCCESS;
> +		return;
> +	}
> +
> +	pos = &feat_out->ents[0];
> +	out_count = feat_out_size;
> +	for (int i = 0, u = 0; i < cxl_mbox->num_features; i++) {
> +		struct cxl_feature *feat = &cxl_mbox->entries[i];
> +
> +		if (feat->exclusive)
> +			continue;
> +
> +		if (u < start) {
> +			u++;
> +			continue;
> +		}
> +
> +		memcpy(pos, &feat->entry, feat_ent_size);
> +		out_count += feat_ent_size;
> +		pos++;
> +		u++;
> +
> +		if (u == count)
> +			break;
> +
> +		/* Make sure it does not go over total output buffer size */
> +		if (out_count + feat_ent_size >= mbox_cmd->size_out)
> +			break;
> +	}
> +
> +	feat_out->num_entries = cpu_to_le16(u);
> +	mbox_cmd->size_out = out_count;
> +	mbox_cmd->return_code = CXL_MBOX_CMD_RC_SUCCESS;
> +}


> diff --git a/include/cxl/features.h b/include/cxl/features.h
> new file mode 100644
> index 000000000000..6074a3f79d70
> --- /dev/null
> +++ b/include/cxl/features.h
> @@ -0,0 +1,52 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/* Copyright(c) 2024 Intel Corporation. */
> +#ifndef __CXL_FEATS_H_
> +#define __CXL_FEATS_H_
> +
> +#include <linux/uuid.h>
> +
> +#define CXL_FEAT_PATROL_SCRUB_UUID						\
> +	UUID_INIT(0x96dad7d6, 0xfde8, 0x482b, 0xa7, 0x33, 0x75, 0x77, 0x4e,	\
> +		  0x06, 0xdb, 0x8a)

We should figure out ordering for this one to avoid introducing the UUIDs in
Shiju's set then pulling them here.   Shiju, maybe just use the UUID
part of this file as a patch in your series?

The exclusive array belongs only with this patch.

I guess it doesn't really matter if we end up with repeated UUID definitions
and cleanup after though.

> +
> +#define CXL_FEAT_ECS_UUID							\
> +	UUID_INIT(0xe5b13f22, 0x2328, 0x4a14, 0xb8, 0xba, 0xb9, 0x69, 0x1e,	\
> +		  0x89, 0x33, 0x86)
> +
> +#define CXL_FEAT_SPPR_UUID							\
> +	UUID_INIT(0x892ba475, 0xfad8, 0x474e, 0x9d, 0x3e, 0x69, 0x2c, 0x91,	\
> +		  0x75, 0x68, 0xbb)
> +
> +#define CXL_FEAT_HPPR_UUID							\
> +	UUID_INIT(0x80ea4521, 0x786f, 0x4127, 0xaf, 0xb1, 0xec, 0x74, 0x59,	\
> +		  0xfb, 0x0e, 0x24)
> +
> +#define CXL_FEAT_CACHELINE_SPARING_UUID						\
> +	UUID_INIT(0x96C33386, 0x91dd, 0x44c7, 0x9e, 0xcb, 0xfd, 0xaf, 0x65,	\
> +		  0x03, 0xba, 0xc4)
> +
> +#define CXL_FEAT_ROW_SPARING_UUID						\
> +	UUID_INIT(0x450ebf67, 0xb135, 0x4f97, 0xa4, 0x98, 0xc2, 0xd5, 0x7f,	\
> +		  0x27, 0x9b, 0xed)
> +
> +#define CXL_FEAT_BANK_SPARING_UUID						\
> +	UUID_INIT(0x78b79636, 0x90ac, 0x4b64, 0xa4, 0xef, 0xfa, 0xac, 0x5d,	\
> +		  0x18, 0xa8, 0x63)
> +
> +#define CXL_FEAT_RANK_SPARING_UUID						\
> +	UUID_INIT(0x34dbaff5, 0x0552, 0x4281, 0x8f, 0x76, 0xda, 0x0b, 0x5e,	\
> +		  0x7a, 0x76, 0xa7)
> +
> +#define CXL_FEAT_UUID_MAX	8

Why not size automatically and use ARRAY_SIZE() to figure out size?

> +static uuid_t cxl_exclusive_feats[CXL_FEAT_UUID_MAX] = {
> +	CXL_FEAT_PATROL_SCRUB_UUID,
> +	CXL_FEAT_ECS_UUID,
> +	CXL_FEAT_SPPR_UUID,
> +	CXL_FEAT_HPPR_UUID,
> +	CXL_FEAT_CACHELINE_SPARING_UUID,
> +	CXL_FEAT_ROW_SPARING_UUID,
> +	CXL_FEAT_BANK_SPARING_UUID,
> +	CXL_FEAT_RANK_SPARING_UUID,
> +};
> +
> +#endif

Re: [RFC PATCH v2 17/20] cxl/test: Add Get Feature support to cxl_test

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:50 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add emulation of Get Feature command to cxl_test. The feature for
> device patrol scrub is returned by the emulation code. This is
> the only feature currently supported by cxl_test. It returns
> the information for the device patrol scrub feature.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
LGTM
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Re: [RFC PATCH v2 18/20] cxl/test: Add Set Feature support to cxl_test

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:51 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add emulation to support Set Feature mailbox command to cxl_test.
> The only feature supported is the device patrol scrub feature. The
> set feature allows activation of patrol scrub for the cxl_test
> emulated device. The command does not support partial data transfer
> even though the spec allows it. This restriction is to reduce complexity
> of the emulation given the patrol scrub feature is very minimal.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Re: [RFC PATCH v2 20/20] fwctl/cxl: Add documentation to FWCTL CXL

[Jonathan Cameron]

On Fri, 15 Nov 2024 14:25:53 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> Add policy and operational documentation for FWCTL CXL.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  .../userspace-api/fwctl/fwctl-cxl.rst         | 91 +++++++++++++++++++
>  Documentation/userspace-api/fwctl/index.rst   |  1 +
>  2 files changed, 92 insertions(+)
>  create mode 100644 Documentation/userspace-api/fwctl/fwctl-cxl.rst
> 
> diff --git a/Documentation/userspace-api/fwctl/fwctl-cxl.rst b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
> new file mode 100644
> index 000000000000..158e9aa45609
> --- /dev/null
> +++ b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
> @@ -0,0 +1,91 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +================
> +fwctl cxl driver
> +================
> +
> +:Author: Dave Jiang
> +
> +Overview
> +========
> +
> +The CXL spec defines a set of commands that can be issued to the mailbox of a
> +CXL device or switch. It also left room for vendor specific commands to be
> +issued to the mailbox as well. fwctl provides a path to issue a set of allowed
> +mailbox commands from user space to the device moderated by the kernel driver.
> +
> +While there are a large set of mailbox commands, only the feature related
> +commands will be allowed to be issued through fwctl initially. No background
> +commands will be supported at this time.
> +
> +CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
> +CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
> +CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)
> +
> +The "Get Supported Features" return data may be filtered by the kernel driver to
> +drop any features that are forbidden by the kernel or being exclusively used by
> +the kernel. The "Get Supported Featuers" command falls under the fwctl policy
spell check. Features

> +of FWCTL_RPC_CONFIGURATION.
As per earlier feedback I think we need to reject get/set even if they are not
listed in get supported features so that bad userspace doesn't get away
with sending them anyway.  This is particularly necessary as I suspect we'll
be removing features in the long run from what we let be accessed this way.

I'm not sure how we document the potential for removal of a feature that
was reported by an earlier kernel.

Jonathan

Re: [RFC PATCH v2 05/20] cxl: Add Get Feature command support for user submission

[Dave Jiang]

On 11/21/24 10:47 AM, Jonathan Cameron wrote:
> On Fri, 15 Nov 2024 14:25:38 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
>> Add enumeration of Get Feature mailbox command for the kernel to recognize
>> the command being passed in from user space.
>>
>> CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
>>
>> The feature requested is identified by specific UUID.
>>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> Hmm. Should we push this into Shiju's series if thought is that will land
> first?  Seems harmless but I find it hard to care that much ;)

Yes if Shiju can fold this into his get feature patch that'd be great.

At some point we should stabilize the common patches between the two series and get that committed to cxl/next.

> 
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
>>  drivers/cxl/core/mbox.c      | 1 +
>>  include/uapi/linux/cxl_mem.h | 1 +
>>  2 files changed, 2 insertions(+)
>>
>> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
>> index f70bd7d28ff8..f49665d3b4d6 100644
>> --- a/drivers/cxl/core/mbox.c
>> +++ b/drivers/cxl/core/mbox.c
>> @@ -68,6 +68,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
>>  	CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
>>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
>>  	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),
>> +	CXL_CMD(GET_FEATURE, 0x15, CXL_VARIABLE_PAYLOAD, 0),
>>  };
>>  
>>  /*
>> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
>> index bd2535962f70..90dcb9723997 100644
>> --- a/include/uapi/linux/cxl_mem.h
>> +++ b/include/uapi/linux/cxl_mem.h
>> @@ -51,6 +51,7 @@
>>  	___C(CLEAR_LOG, "Clear Log"),					  \
>>  	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
>>  	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),		  \
>> +	___C(GET_FEATURE, "Get Feature"),				  \
>>  	___C(MAX, "invalid / last command")
>>  
>>  #define ___C(a, b) CXL_MEM_COMMAND_ID_##a
> 

Re: [RFC PATCH v2 11/20] fwctl: FWCTL_HW_INFO to return hardware information

[Dave Jiang]

On 11/21/24 11:20 AM, Jonathan Cameron wrote:
> On Fri, 15 Nov 2024 14:25:44 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
>> Add an optional ioctl FWCTL_HW_INFO to pass command specific information
>> to user space. An array of 'struct fwctl_command_info' will be returned
>> from the ioctl. These commands are send to the driver via FWCTL_RPC call.
>> The command info struct contains the command id, the related hardware
>> opcode, input and output size for the command, and the effects the command
>> has if it's a write command.
>>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> Minor stuff inline.
> 
> Jonathan
> 
> 
>> diff --git a/include/uapi/fwctl/fwctl.h b/include/uapi/fwctl/fwctl.h
>> index 4e4d30104667..7334907e27c1 100644
>> --- a/include/uapi/fwctl/fwctl.h
>> +++ b/include/uapi/fwctl/fwctl.h
> 
>>  enum fwctl_device_type {
>> @@ -69,6 +70,61 @@ struct fwctl_info {
>>  };
>>  #define FWCTL_INFO _IO(FWCTL_TYPE, FWCTL_CMD_INFO)
>>  
>> +/**
>> + * struct fwctl_command_info - Hardware command information returned from a query.
>> + * @id: Driver ID number for the command
>> + * @opcode: Hardware command opcode
>> + * @set_effects: Effects to the OS and hardware when command is executed.
>> + *               Provided by the device.
> 
> Run kernel doc over the files.  (wrong name)
> 
>> + * @size_in: Expected input size, or ~0U if variable length.
>> + * @size_out: Expected output size, or ~0U if variable length.
>> + *
>> + * Represents a single command that is supported by both the driver and the
>> + * hardware. This is returned as part of an array from the FWCTL_HW_INFO ioctl.
>> + */
>> +struct fwctl_command_info {
>> +	__u32 id;
>> +	__u16 opcode;
>> +	__u16 effects;
>> +	__u32 size_in;
>> +	__u32 size_out;
>> +};
>> +
>> +/**
>> + * struct fwctl_hw_info_out - output struct for FWCTL_HW_INFO
>> + * @nr_cmds: Number of commands for output
>> + * @reserved: Reserved u32 for alignment.
> 
> As currently defined, why do we need to pad? I'm not against
> padding just not understanding the comment.
> It's more fun if there is a u64 in there (we all love 32 bit
> x86 data alignment).

If 32bit alignment is fine then I'll drop it.

> 
>> + * @commands: Array of 'struct fwctl_command_info'
>> + */
>> +struct fwctl_hw_info_out {
>> +	__u32 nr_cmds;
>> +	__u32 reserved;
>> +
>> +	struct fwctl_command_info commands[] __counted_by(nr_cmds);
>> +};
> 
> 

Re: [RFC PATCH v2 10/20] fwctl/cxl: Add support for get driver information

[Dave Jiang]

On 11/21/24 11:11 AM, Jonathan Cameron wrote:
> On Fri, 15 Nov 2024 14:25:43 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
>> Add definition for fwctl_ops->info() to return driver information. The
>> function will return the number of device mailbox commands supported by the
>> fwctl char device.
>>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> Few minor things inline
> 
> Jonathan
>> ---
>> v2:
>> - Change driver info to report number of commands supported by the driver and device.
>> ---
>>  drivers/cxl/core/mbox.c  | 46 ++++++++++++++++++++++++++++++++++++++++
>>  drivers/fwctl/cxl/cxl.c  | 23 ++++++++++++++++++--
>>  include/cxl/cxl.h        |  2 ++
>>  include/cxl/mailbox.h    |  1 +
>>  include/uapi/fwctl/cxl.h | 22 +++++++++++++++++++
>>  5 files changed, 92 insertions(+), 2 deletions(-)
>>  create mode 100644 include/uapi/fwctl/cxl.h
>>
>> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
>> index 12758e763650..f464eb42f08a 100644
>> --- a/drivers/cxl/core/mbox.c
>> +++ b/drivers/cxl/core/mbox.c
>> @@ -118,6 +118,14 @@ static u8 security_command_sets[] = {
>>  	0x46, /* Security Passthrough */
>>  };
>>  
>> +#define FWCTL_CXL_MAX_COMMANDS 3
> 
> Use this to size the array? Will let us know if it gets
> out of sync.
> 
>> +/* Command set that is allowed with FWCTL-CXL */
>> +static u16 fwctl_command_sets[] = {
>> +	CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
>> +	CXL_MBOX_OP_GET_FEATURE,
>> +	CXL_MBOX_OP_SET_FEATURE,
>> +};
>> +
> 
>>  static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
>> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
>> index 6bfd7942a3f7..d0939e3bcbc0 100644
>> --- a/include/cxl/cxl.h
>> +++ b/include/cxl/cxl.h
>> @@ -3,6 +3,8 @@
>>  #ifndef __CXL_GLOBAL_H__
>>  #define __CXL_GLOBAL_H__
>>  
>> +#include <linux/device.h>
> 
> Why in this patch?

I got a 0-day report with this patch complaining about missing 'struct device' definition on some weird arch. May be a current issue that needs a standalone fix.
 
> 
>> +
>>  struct cxl_driver {
>>  	const char *name;
>>  	int (*probe)(struct device *dev);
> 

Re: [RFC PATCH v2 10/20] fwctl/cxl: Add support for get driver information

[Jonathan Cameron]

On Fri, 22 Nov 2024 16:22:25 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> On 11/21/24 11:11 AM, Jonathan Cameron wrote:
> > On Fri, 15 Nov 2024 14:25:43 -0700
> > Dave Jiang <dave.jiang@intel.com> wrote:
> >   
> >> Add definition for fwctl_ops->info() to return driver information. The
> >> function will return the number of device mailbox commands supported by the
> >> fwctl char device.
> >>
> >> Signed-off-by: Dave Jiang <dave.jiang@intel.com>  
> > Few minor things inline
> > 
> > Jonathan  
> >> ---
> >> v2:
> >> - Change driver info to report number of commands supported by the driver and device.
> >> ---
> >>  drivers/cxl/core/mbox.c  | 46 ++++++++++++++++++++++++++++++++++++++++
> >>  drivers/fwctl/cxl/cxl.c  | 23 ++++++++++++++++++--
> >>  include/cxl/cxl.h        |  2 ++
> >>  include/cxl/mailbox.h    |  1 +
> >>  include/uapi/fwctl/cxl.h | 22 +++++++++++++++++++
> >>  5 files changed, 92 insertions(+), 2 deletions(-)
> >>  create mode 100644 include/uapi/fwctl/cxl.h
> >>
> >> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> >> index 12758e763650..f464eb42f08a 100644
> >> --- a/drivers/cxl/core/mbox.c
> >> +++ b/drivers/cxl/core/mbox.c
> >> @@ -118,6 +118,14 @@ static u8 security_command_sets[] = {
> >>  	0x46, /* Security Passthrough */
> >>  };
> >>  
> >> +#define FWCTL_CXL_MAX_COMMANDS 3  
> > 
> > Use this to size the array? Will let us know if it gets
> > out of sync.
> >   
> >> +/* Command set that is allowed with FWCTL-CXL */
> >> +static u16 fwctl_command_sets[] = {
> >> +	CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
> >> +	CXL_MBOX_OP_GET_FEATURE,
> >> +	CXL_MBOX_OP_SET_FEATURE,
> >> +};
> >> +  
> >   
> >>  static void *cxlctl_fw_rpc(struct fwctl_uctx *uctx, enum fwctl_rpc_scope scope,
> >> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
> >> index 6bfd7942a3f7..d0939e3bcbc0 100644
> >> --- a/include/cxl/cxl.h
> >> +++ b/include/cxl/cxl.h
> >> @@ -3,6 +3,8 @@
> >>  #ifndef __CXL_GLOBAL_H__
> >>  #define __CXL_GLOBAL_H__
> >>  
> >> +#include <linux/device.h>  
> > 
> > Why in this patch?  
> 
> I got a 0-day report with this patch complaining about missing 'struct device' definition on some weird arch. May be a current issue that needs a standalone fix.


Hmm. struct device only needs a forward definition but not so much for
struct device_driver.

Pull this back to patch 8 I think.

>  
> >   
> >> +
> >>  struct cxl_driver {
> >>  	const char *name;
> >>  	int (*probe)(struct device *dev);  
> >   
> 
> 

RE: [RFC PATCH v2 05/20] cxl: Add Get Feature command support for user submission

[Shiju Jose]

>-----Original Message-----
>From: Dave Jiang <dave.jiang@intel.com>
>Sent: 22 November 2024 20:15
>To: Jonathan Cameron <jonathan.cameron@huawei.com>
>Cc: linux-cxl@vger.kernel.org; dan.j.williams@intel.com; ira.weiny@intel.com;
>vishal.l.verma@intel.com; alison.schofield@intel.com; dave@stgolabs.net;
>jgg@nvidia.com; Shiju Jose <shiju.jose@huawei.com>; Shiju Jose
><shiju.jose@huawei.com>
>Subject: Re: [RFC PATCH v2 05/20] cxl: Add Get Feature command support for
>user submission
>
>
>
>On 11/21/24 10:47 AM, Jonathan Cameron wrote:
>> On Fri, 15 Nov 2024 14:25:38 -0700
>> Dave Jiang <dave.jiang@intel.com> wrote:
>>
>>> Add enumeration of Get Feature mailbox command for the kernel to
>>> recognize the command being passed in from user space.
>>>
>>> CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
>>>
>>> The feature requested is identified by specific UUID.
>>>
>>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> Hmm. Should we push this into Shiju's series if thought is that will
>> land first?  Seems harmless but I find it hard to care that much ;)
>
>Yes if Shiju can fold this into his get feature patch that'd be great.
>
>At some point we should stabilize the common patches between the two series
>and get that committed to cxl/next.

Hi Dave,
I merged this patch to the get feature patch.
The rearranged v17 patches for CXL feature commands( including the merged patch)
and are available here, 
(next version may be send after receiving feedbacks from Borislav on EDAC patches).

https://github.com/shijujose4/linux.git
Branch: edac-enhancement-ras-features_v17_updated

Thanks,
Shiju
>
>>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>> ---
>>>  drivers/cxl/core/mbox.c      | 1 +
>>>  include/uapi/linux/cxl_mem.h | 1 +
>>>  2 files changed, 2 insertions(+)
>>>
>>> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index
>>> f70bd7d28ff8..f49665d3b4d6 100644
>>> --- a/drivers/cxl/core/mbox.c
>>> +++ b/drivers/cxl/core/mbox.c
>>> @@ -68,6 +68,7 @@ static struct cxl_mem_command
>cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
>>>  	CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
>>>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
>>>  	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD,
>0),
>>> +	CXL_CMD(GET_FEATURE, 0x15, CXL_VARIABLE_PAYLOAD, 0),
>>>  };
>>>
>>>  /*
>>> diff --git a/include/uapi/linux/cxl_mem.h
>>> b/include/uapi/linux/cxl_mem.h index bd2535962f70..90dcb9723997
>>> 100644
>>> --- a/include/uapi/linux/cxl_mem.h
>>> +++ b/include/uapi/linux/cxl_mem.h
>>> @@ -51,6 +51,7 @@
>>>  	___C(CLEAR_LOG, "Clear Log"),					  \
>>>  	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
>>>  	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),
>	  \
>>> +	___C(GET_FEATURE, "Get Feature"),				  \
>>>  	___C(MAX, "invalid / last command")
>>>
>>>  #define ___C(a, b) CXL_MEM_COMMAND_ID_##a
>>
>

Re: [RFC PATCH v2 16/20] fwctl/cxl: Add support to filter exclusive features

[Dave Jiang]

On 11/22/24 8:05 AM, Jonathan Cameron wrote:
> On Fri, 15 Nov 2024 14:25:49 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
>> Add support to allow filtering of CXL features that are exclusive to the
>> kernel and make them not visible to user space. The "get supported
>> features" mailbox command returned to the userspace is emulated and
>> will skip the features that is marked exclusive for the kernel. The
>> exclusion allows certain the feature setting of certain commands such
>> as claimed by RAS to be exclusive to the kernel.
> 
> Well behaved software is only going to issue features that were returned
> by get_supported_features.  However we need to block badly behaved software
> too.  So a check is needed in the set_feature() / get_feature() paths.
> 
> A few queries inline but basically LGTM other than that.
> 
> Jonathan
> 
>>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>>  drivers/cxl/core/mbox.c | 115 ++++++++++++++++++++++++++++++++++++----
>>  drivers/fwctl/cxl/cxl.c |   7 +--
>>  include/cxl/features.h  |  52 ++++++++++++++++++
>>  include/cxl/mailbox.h   |   8 ++-
>>  4 files changed, 168 insertions(+), 14 deletions(-)
>>  create mode 100644 include/cxl/features.h
>>
>> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
>> index e7c5c709ac79..12ace2951f7c 100644
>> --- a/drivers/cxl/core/mbox.c
>> +++ b/drivers/cxl/core/mbox.c
>> @@ -5,6 +5,7 @@
>>  #include <linux/ktime.h>
>>  #include <linux/mutex.h>
>>  #include <linux/unaligned.h>
>> +#include <cxl/features.h>
>>  #include <cxl/mailbox.h>
>>  #include <cxlpci.h>
>>  #include <cxlmem.h>
>> @@ -897,6 +898,78 @@ static int handle_mailbox_cmd_from_fwctl(struct cxl_mailbox *cxl_mbox,
>>  	return 0;
>>  }
>>  
>> +/*
>> + * Emulate the 'get supported features' mailbox command and only copy out the
>> + * features that are not marked as exclusive for kernel.
>> + */
>> +static void cxl_mbox_get_supported_features_filtered(struct cxl_mailbox *cxl_mbox,
>> +						     struct cxl_mbox_cmd *mbox_cmd)
>> +{
>> +	struct cxl_mbox_get_sup_feats_in *feat_in = mbox_cmd->payload_in;
>> +	struct cxl_mbox_get_sup_feats_out *feat_out = mbox_cmd->payload_out;
>> +	const int feat_out_size = sizeof(*feat_out);
> Worth while?  I'd just use sizeof(*feat_out) inline.
> Compiler will sort out out and it is self documenting.

ok

> 
>> +	struct cxl_feat_entry *pos;
>> +	const int feat_ent_size = sizeof(*pos);
> Same for this.

ok

> 
>> +	int out_count, ents, u;
>> +	u32 count;
>> +	u16 start;
>> +
>> +	count = le32_to_cpu(feat_in->count);
>> +	start = le16_to_cpu(feat_in->start_idx);
>> +	ents = count / sizeof(struct cxl_feat_entry);
>> +	ents -= start;
> 
> You've lost me on this maths. ents is what?

So I'm trying to validate the number of entries I need to copy out when 'start' index gets thrown in the fray. So it calculates the number of entries the output buffer is capable off, and then minus the start index. Although now I think about it, I should compare that to the (total driver supported features - start). Having to deal with 'start' with this "emulation" is a pain.  

> 
> 
>> +	if (ents < 0) {
>> +		mbox_cmd->size_out = 0;
>> +		mbox_cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
>> +		return;
>> +	}
>> +
>> +	if (mbox_cmd->size_out < feat_out_size) {
>> +		mbox_cmd->size_out = 0;
>> +		mbox_cmd->return_code = CXL_MBOX_CMD_RC_INPUT;
>> +		return;
>> +	}
>> +
>> +	feat_out->supported_feats =
>> +		cpu_to_le16(cxl_mbox->num_user_feats);
>> +	if (ents == 0) {
>> +		feat_out->num_entries = cpu_to_le16(0);
>> +		mbox_cmd->size_out = feat_out_size;
>> +		mbox_cmd->return_code = CXL_MBOX_CMD_RC_SUCCESS;
>> +		return;
>> +	}
>> +
>> +	pos = &feat_out->ents[0];
>> +	out_count = feat_out_size;
>> +	for (int i = 0, u = 0; i < cxl_mbox->num_features; i++) {
>> +		struct cxl_feature *feat = &cxl_mbox->entries[i];
>> +
>> +		if (feat->exclusive)
>> +			continue;
>> +
>> +		if (u < start) {
>> +			u++;
>> +			continue;
>> +		}
>> +
>> +		memcpy(pos, &feat->entry, feat_ent_size);
>> +		out_count += feat_ent_size;
>> +		pos++;
>> +		u++;
>> +
>> +		if (u == count)
>> +			break;
>> +
>> +		/* Make sure it does not go over total output buffer size */
>> +		if (out_count + feat_ent_size >= mbox_cmd->size_out)
>> +			break;
>> +	}
>> +
>> +	feat_out->num_entries = cpu_to_le16(u);
>> +	mbox_cmd->size_out = out_count;
>> +	mbox_cmd->return_code = CXL_MBOX_CMD_RC_SUCCESS;
>> +}
> 
> 
>> diff --git a/include/cxl/features.h b/include/cxl/features.h
>> new file mode 100644
>> index 000000000000..6074a3f79d70
>> --- /dev/null
>> +++ b/include/cxl/features.h
>> @@ -0,0 +1,52 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> +/* Copyright(c) 2024 Intel Corporation. */
>> +#ifndef __CXL_FEATS_H_
>> +#define __CXL_FEATS_H_
>> +
>> +#include <linux/uuid.h>
>> +
>> +#define CXL_FEAT_PATROL_SCRUB_UUID						\
>> +	UUID_INIT(0x96dad7d6, 0xfde8, 0x482b, 0xa7, 0x33, 0x75, 0x77, 0x4e,	\
>> +		  0x06, 0xdb, 0x8a)
> 
> We should figure out ordering for this one to avoid introducing the UUIDs in
> Shiju's set then pulling them here.   Shiju, maybe just use the UUID
> part of this file as a patch in your series?
> 
> The exclusive array belongs only with this patch.
> 
> I guess it doesn't really matter if we end up with repeated UUID definitions
> and cleanup after though.
> 
>> +
>> +#define CXL_FEAT_ECS_UUID							\
>> +	UUID_INIT(0xe5b13f22, 0x2328, 0x4a14, 0xb8, 0xba, 0xb9, 0x69, 0x1e,	\
>> +		  0x89, 0x33, 0x86)
>> +
>> +#define CXL_FEAT_SPPR_UUID							\
>> +	UUID_INIT(0x892ba475, 0xfad8, 0x474e, 0x9d, 0x3e, 0x69, 0x2c, 0x91,	\
>> +		  0x75, 0x68, 0xbb)
>> +
>> +#define CXL_FEAT_HPPR_UUID							\
>> +	UUID_INIT(0x80ea4521, 0x786f, 0x4127, 0xaf, 0xb1, 0xec, 0x74, 0x59,	\
>> +		  0xfb, 0x0e, 0x24)
>> +
>> +#define CXL_FEAT_CACHELINE_SPARING_UUID						\
>> +	UUID_INIT(0x96C33386, 0x91dd, 0x44c7, 0x9e, 0xcb, 0xfd, 0xaf, 0x65,	\
>> +		  0x03, 0xba, 0xc4)
>> +
>> +#define CXL_FEAT_ROW_SPARING_UUID						\
>> +	UUID_INIT(0x450ebf67, 0xb135, 0x4f97, 0xa4, 0x98, 0xc2, 0xd5, 0x7f,	\
>> +		  0x27, 0x9b, 0xed)
>> +
>> +#define CXL_FEAT_BANK_SPARING_UUID						\
>> +	UUID_INIT(0x78b79636, 0x90ac, 0x4b64, 0xa4, 0xef, 0xfa, 0xac, 0x5d,	\
>> +		  0x18, 0xa8, 0x63)
>> +
>> +#define CXL_FEAT_RANK_SPARING_UUID						\
>> +	UUID_INIT(0x34dbaff5, 0x0552, 0x4281, 0x8f, 0x76, 0xda, 0x0b, 0x5e,	\
>> +		  0x7a, 0x76, 0xa7)
>> +
>> +#define CXL_FEAT_UUID_MAX	8
> 
> Why not size automatically and use ARRAY_SIZE() to figure out size?

will do

> 
>> +static uuid_t cxl_exclusive_feats[CXL_FEAT_UUID_MAX] = {
>> +	CXL_FEAT_PATROL_SCRUB_UUID,
>> +	CXL_FEAT_ECS_UUID,
>> +	CXL_FEAT_SPPR_UUID,
>> +	CXL_FEAT_HPPR_UUID,
>> +	CXL_FEAT_CACHELINE_SPARING_UUID,
>> +	CXL_FEAT_ROW_SPARING_UUID,
>> +	CXL_FEAT_BANK_SPARING_UUID,
>> +	CXL_FEAT_RANK_SPARING_UUID,
>> +};
>> +
>> +#endif
> 

Re: [RFC PATCH v2 20/20] fwctl/cxl: Add documentation to FWCTL CXL

[Dave Jiang]

On 11/22/24 8:26 AM, Jonathan Cameron wrote:
> On Fri, 15 Nov 2024 14:25:53 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
>> Add policy and operational documentation for FWCTL CXL.
>>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>>  .../userspace-api/fwctl/fwctl-cxl.rst         | 91 +++++++++++++++++++
>>  Documentation/userspace-api/fwctl/index.rst   |  1 +
>>  2 files changed, 92 insertions(+)
>>  create mode 100644 Documentation/userspace-api/fwctl/fwctl-cxl.rst
>>
>> diff --git a/Documentation/userspace-api/fwctl/fwctl-cxl.rst b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
>> new file mode 100644
>> index 000000000000..158e9aa45609
>> --- /dev/null
>> +++ b/Documentation/userspace-api/fwctl/fwctl-cxl.rst
>> @@ -0,0 +1,91 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +================
>> +fwctl cxl driver
>> +================
>> +
>> +:Author: Dave Jiang
>> +
>> +Overview
>> +========
>> +
>> +The CXL spec defines a set of commands that can be issued to the mailbox of a
>> +CXL device or switch. It also left room for vendor specific commands to be
>> +issued to the mailbox as well. fwctl provides a path to issue a set of allowed
>> +mailbox commands from user space to the device moderated by the kernel driver.
>> +
>> +While there are a large set of mailbox commands, only the feature related
>> +commands will be allowed to be issued through fwctl initially. No background
>> +commands will be supported at this time.
>> +
>> +CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
>> +CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
>> +CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)
>> +
>> +The "Get Supported Features" return data may be filtered by the kernel driver to
>> +drop any features that are forbidden by the kernel or being exclusively used by
>> +the kernel. The "Get Supported Featuers" command falls under the fwctl policy
> spell check. Features
> 
>> +of FWCTL_RPC_CONFIGURATION.
> As per earlier feedback I think we need to reject get/set even if they are not
> listed in get supported features so that bad userspace doesn't get away
> with sending them anyway.  This is particularly necessary as I suspect we'll
> be removing features in the long run from what we let be accessed this way.
> 
> I'm not sure how we document the potential for removal of a feature that
> was reported by an earlier kernel.

Userspace should query the driver for whatever features that are "available". Given one feature may exist for a device while another does not? So maybe the expectation from the app should be feature does not exist unless reported by the driver that it exists. 

> 
> Jonathan
> 
> 
> 
> 

Re: [RFC PATCH v2 01/20] cxl: Refactor user ioctl command path from mds to mailbox

[Dan Williams]

Dave Jiang wrote:
> With 'struct cxl_mailbox' context introduced, the helper functions
> cxl_query_cmd() and cxl_send_cmd() can take a cxl_mailbox directly
> rather than a cxl_memdev parameter. Refactor to use cxl_mailbox
> directly.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>

LGTM

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

Re: [RFC PATCH v2 02/20] cxl: Add Get Supported Features command for kernel usage

[Dan Williams]

Dave Jiang wrote:
> CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h)
> The command retrieve the list of supported device-specific features
> (identified by UUID) and general information about each Feature.
> 
> The driver will retrieve the feature entries in order to make checks and
> provide information for the Get Feature and Set Feature command. One of
> the main piece of information retrieved are the effects a Set Feature
> command would have for a particular feature.
> 
> Co-developed-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> 
> v2:
> - fix feature entry pointer math.
> - free mbox_out in cxl_get_supported_features(). (Shiju, Jonathan)
> - replace sizeof(struct) with feat_size. (Shiju)
> - replace reserved in feature structs with u8 type. (Shiju)
> - change cxl_feat_entry->effects to set_effects. (Shiju)
> - rearrange assigment of cxl_mbox->entries. (Jonathan)
> - Separate no features from actual error. (Jonathan)
> - Fix missing kdoc for entries. (Jonathan)
> ---
>  drivers/cxl/core/mbox.c      | 178 +++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxlmem.h         |  31 ++++++
>  drivers/cxl/pci.c            |   4 +
>  include/cxl/mailbox.h        |   4 +
>  include/uapi/linux/cxl_mem.h |   1 +
>  5 files changed, 218 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 880ac1dba3cc..4ba56f3d5a65 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -67,6 +67,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
>  	CXL_CMD(SET_SHUTDOWN_STATE, 0x1, 0, 0),
>  	CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
> +	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD, 0),

Since this patch is only about internal usage then no need for a
cxl_mem_commands entry which is the translation from CXL mailbox
op-codes to user CXL commands ids.

I think ideally CXL features ABI is only ever a fwctl concern. Lets try
to see if we can get away with never defining a CXL user command code
for them.

>  };
>  
>  /*
> @@ -790,6 +791,183 @@ static const uuid_t log_uuid[] = {
>  	[VENDOR_DEBUG_UUID] = DEFINE_CXL_VENDOR_DEBUG_UUID,
>  };
>  
> +static void cxl_free_features(void *features)
> +{
> +	kvfree(features);
> +}
> +
> +static int cxl_get_supported_features_count(struct cxl_dev_state *cxlds)
> +{
> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
> +	struct cxl_mbox_get_sup_feats_out mbox_out;
> +	struct cxl_mbox_get_sup_feats_in mbox_in;
> +	struct cxl_mbox_cmd mbox_cmd;
> +	int rc;
> +
> +	memset(&mbox_in, 0, sizeof(mbox_in));
> +	mbox_in.count = cpu_to_le32(sizeof(mbox_out));
> +	memset(&mbox_out, 0, sizeof(mbox_out));
> +	mbox_cmd = (struct cxl_mbox_cmd) {
> +		.opcode = CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
> +		.size_in = sizeof(mbox_in),
> +		.payload_in = &mbox_in,
> +		.size_out = sizeof(mbox_out),
> +		.payload_out = &mbox_out,
> +		.min_out = sizeof(mbox_out),
> +	};
> +	rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
> +	if (rc < 0)
> +		return rc;
> +
> +	cxl_mbox->num_features = le16_to_cpu(mbox_out.supported_feats);

I think this function should return a positive number of features or a
negative error code, more below...

> +
> +	return 0;
> +}
> +
> +int cxl_get_supported_features(struct cxl_dev_state *cxlds)
> +{
> +	struct cxl_mbox_get_sup_feats_out *mbox_out __free(kvfree) = NULL;
> +	int remain_feats, max_size, max_feats, start, rc;
> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
> +	int feat_size = sizeof(struct cxl_feat_entry);
> +	struct cxl_mbox_get_sup_feats_in mbox_in;
> +	int hdr_size = sizeof(*mbox_out);
> +	struct cxl_mbox_cmd mbox_cmd;
> +	struct cxl_mem_command *cmd;
> +	struct cxl_feat_entry *entry;
> +
> +	/* Get supported features is optional, need to check */
> +	cmd = cxl_mem_find_command(CXL_MBOX_OP_GET_SUPPORTED_FEATURES);
> +	if (!cmd)
> +		return -EOPNOTSUPP;
> +	if (!test_bit(cmd->info.id, cxl_mbox->enabled_cmds))
> +		return -EOPNOTSUPP;
> +
> +	rc = cxl_get_supported_features_count(cxlds);
> +	if (rc)
> +		return rc;
> +
> +	if (!cxl_mbox->num_features) {
> +		dev_dbg(cxl_mbox->host, "No CXL features enumerated.\n");
> +		return 0;
> +	}
> +
> +	struct cxl_feat_entry *entries __free(kvfree) =
> +		kvmalloc(cxl_mbox->num_features * feat_size, GFP_KERNEL);
> +
> +	if (!entries)
> +		return -ENOMEM;
> +
> +	max_size = cxl_mbox->payload_size - hdr_size;
> +	/* max feat entries that can fit in mailbox max payload size */
> +	max_feats = max_size / feat_size;
> +	entry = &entries[0];
> +
> +	mbox_out = kvmalloc(cxl_mbox->payload_size, GFP_KERNEL);
> +	if (!mbox_out)
> +		return -ENOMEM;
> +
> +	start = 0;
> +	remain_feats = cxl_mbox->num_features;
> +	do {
[..]
> +		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
> +		if (rc < 0)
> +			return rc;

Hmm, why "return rc" here, but "goto err" below?

> +		if (mbox_cmd.size_out <= hdr_size) {
> +			rc = -ENXIO;
> +			goto err;

So one of my new code smells for "needs more care" is cases where
scoped-based-cleanup functions are still mixed with usage of "goto".

In this case there are two dependent things to cleanup on error, "reset
cxl_mbox->num_features" and "free @entries". So, to eliminate the goto,
how about eliminate one of the things to cleanup with something like
this:

	struct cxl_features {
		int num_features;
		struct cxl_feat_entry entries[] __counted_by(num_features);
	};
	
	count = cxl_get_supported_features_count(cxl_mbox);
	if (count <= 0)
		return ...
	
	struct cxl_features *features = __free(kvfree) =
	        kvmalloc(struct_size(features, entries, count), GFP_KERNEL);
	
	features->num_features = count;

...and then use "cxl_mbox->features == NULL" as your "no features" case.


> +		}
> +
> +		/*
> +		 * Make sure retrieved out buffer is multiple of feature
> +		 * entries.
> +		 */
> +		retrieved = mbox_cmd.size_out - hdr_size;
> +		if (retrieved % feat_size) {
> +			rc = -ENXIO;
> +			goto err;
> +		}
> +
> +		num_entries = le16_to_cpu(mbox_out->num_entries);
> +		/*
> +		 * If the reported output entries * defined entry size !=
> +		 * retrieved output bytes, then the output package is incorrect.
> +		 */
> +		if (num_entries * feat_size != retrieved) {
> +			rc = -ENXIO;
> +			goto err;
> +		}
> +
> +		memcpy(entry, mbox_out->ents, retrieved);
> +		entry++;
> +		/*
> +		 * If the number of output entries is less than expected, add the
> +		 * remaining entries to the next batch.
> +		 */
> +		remain_feats += copy_feats - num_entries;
> +		start += num_entries;
> +	} while (remain_feats);
> +
> +	cxl_mbox->entries = no_free_ptr(entries);
> +	rc = devm_add_action_or_reset(cxl_mbox->host, cxl_free_features,
> +				      cxl_mbox->entries);
> +	if (rc)
> +		return rc;
> +
> +	return 0;
> +
> +err:
> +	cxl_mbox->num_features = 0;
> +	return rc;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_get_supported_features, CXL);
> +
> +int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t *feat_uuid,
> +				    struct cxl_feat_entry *feat_entry_out)
> +{
> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
> +	struct cxl_feat_entry *feat_entry;
> +	int count;
> +
> +	/* Check CXL dev supports the feature */
> +	feat_entry = &cxl_mbox->entries[0];
> +	for (count = 0; count < cxl_mbox->num_features; count++, feat_entry++) {
> +		if (uuid_equal(&feat_entry->uuid, feat_uuid)) {
> +			memcpy(feat_entry_out, feat_entry, sizeof(*feat_entry_out));

Why copy-out? Would it not be sufficient to just return the index in the
features->entries[] array?

[..]
> diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
> index cc894f07a435..03c4b8ad84c1 100644
> --- a/include/cxl/mailbox.h
> +++ b/include/cxl/mailbox.h
> @@ -50,6 +50,8 @@ struct cxl_mbox_cmd {
>   *                (CXL 3.1 8.2.8.4.3 Mailbox Capabilities Register)
>   * @mbox_mutex: mutex protects device mailbox and firmware
>   * @mbox_wait: rcuwait for mailbox
> + * @num_features: number of supported features entries
> + * @features: list of supported feature entries
>   * @mbox_send: @dev specific transport for transmitting mailbox commands
>   */
>  struct cxl_mailbox {
> @@ -59,6 +61,8 @@ struct cxl_mailbox {
>  	size_t payload_size;
>  	struct mutex mbox_mutex; /* lock to protect mailbox context */
>  	struct rcuwait mbox_wait;
> +	int num_features;
> +	struct cxl_feat_entry *entries;
>  	int (*mbox_send)(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd *cmd);
>  };
>  
> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> index c6c0fe27495d..bd2535962f70 100644
> --- a/include/uapi/linux/cxl_mem.h
> +++ b/include/uapi/linux/cxl_mem.h
> @@ -50,6 +50,7 @@
>  	___C(GET_LOG_CAPS, "Get Log Capabilities"),			  \
>  	___C(CLEAR_LOG, "Clear Log"),					  \
>  	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
> +	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),		  \

Again, for a patch that claims to be kernel internal only enabling, lets
keep it internal-only.

Re: [RFC PATCH v2 03/20] cxl/test: Add Get Supported Features mailbox command support

[Dan Williams]

Dave Jiang wrote:
> Add cxl-test emulation of Get Supported Features mailbox command.
> Currently only adding a test feature with feature identifier of
> all f's for testing.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  tools/testing/cxl/test/mem.c | 69 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 69 insertions(+)

Looks good to me:

Acked-by: Dan Williams <dan.j.williams@intel.com>

I explicitly only ack this because the real review is when the
corresponding test in cxl-cli is available, but this can go upstream in
the meantime.

Re: [RFC PATCH v2 04/20] cxl/mbox: Add GET_FEATURE mailbox command

[Dan Williams]

Dave Jiang wrote:
> From: Shiju Jose <shiju.jose@huawei.com>
> 
> Add support for GET_FEATURE mailbox command.
> 
> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
> The settings of a feature can be retrieved using Get Feature command.
> CXL spec 3.1 section 8.2.9.6.2 describes Get Feature command.
> 
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> This patch is not needed by this series except the opcode enumeration.
> ---

Wait, then why is it included?

I would say fold the small bit that you need into the previous patch,
especially if we think that fwctl can cover all the "get feature" use
cases.

>  drivers/cxl/core/mbox.c | 41 +++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxlmem.h    | 26 ++++++++++++++++++++++++++
>  2 files changed, 67 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 4ba56f3d5a65..f70bd7d28ff8 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -968,6 +968,47 @@ int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t *f
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_get_supported_feature_entry, CXL);
>  
> +size_t cxl_get_feature(struct cxl_dev_state *cxlds, const uuid_t feat_uuid,
> +		       enum cxl_get_feat_selection selection,
> +		       void *feat_out, size_t feat_out_size)
> +{
> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;

cxlds is otherwise unused in this function, just pass in @cxl_mbox
directly.

Re: [RFC PATCH v2 05/20] cxl: Add Get Feature command support for user submission

[Dan Williams]

Dave Jiang wrote:
> Add enumeration of Get Feature mailbox command for the kernel to recognize
> the command being passed in from user space.
> 
> CXL spec r3.1 8.2.9.6.2 Get Feature (Opcode 0501h)
> 
> The feature requested is identified by specific UUID.

Lets identify the consumer for this ABI that can't use fwctl.

Re: [RFC PATCH v2 06/20] cxl/mbox: Add SET_FEATURE mailbox command

[Dan Williams]

Dave Jiang wrote:
> From: Shiju Jose <shiju.jose@huawei.com>
> 
> Add support for SET_FEATURE mailbox command.
> 
> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
> CXL devices supports features with changeable attributes.
> The settings of a feature can be optionally modified using Set Feature
> command.
> CXL spec 3.1 section 8.2.9.6.3 describes Set Feature command.
> 
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> This patch is not needed except for the opcode enumeration.
> ---

Same reaction as a the last patch.

We can always circle back to add this on top if really necessary, but
lets keep this set on topic with the cover letter of enabling
fwctl for features.

Re: [RFC PATCH v2 07/20] cxl: Add Set Feature command support for user submission

[Dan Williams]

Dave Jiang wrote:
> Add enumeration of Set Feature mailbox command for the kernel to recognize
> the command being passed in from user space.
> 
> CXL spec r3.1 8.2.9.6.3 Set Feature (Opcode 0502h)
> 
> The feature requested is identified by specific UUID.

The changelog needs to be more than "thing exists, enable thing". When
its enabling a "forever" ABI there needs to be a story around why Linux
cares about the new ABI forever.

Now this story applies equally to fwctl support for set features, but
that at least has built-in support for a security model which saves you
from needing to tell that side of the story.

I assume we can drop this for the purpose of fwctl enabling?

Re: [RFC PATCH v2 09/20] fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands

[Dan Williams]

Dave Jiang wrote:
> Add a cxl fwctl driver to allow sending of CXL feature
> commands from userspace through as ioctls. Create a driver skeleton for
> initial setup.

I don't understand why fwctl support of the *existing* mailbox needs
yet another sub-device and driver?

In the mlx5 case, as I understand it, multiple families of device can
register the common mlx5_fwctl auxiliary device to share a common driver
for that piece. For CXL that shared object is the 'struct cxl_mailbox'.

The changelog does not help with a rationale for this split, so I am
tempted to just say fwctl is always resident if your cxl device has
registered a cxl_mailbox and the CONFIG_FWCTL_CXL boolean is set to
true.

> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v2:
> - Move to cxl_bus from auxiliary_bus (Jonathan)
> ---
>  MAINTAINERS                |  8 ++++
>  drivers/cxl/core/mbox.c    | 41 +++++++++++++++-
>  drivers/fwctl/Kconfig      |  9 ++++
>  drivers/fwctl/Makefile     |  1 +
>  drivers/fwctl/cxl/Makefile |  4 ++
>  drivers/fwctl/cxl/cxl.c    | 97 ++++++++++++++++++++++++++++++++++++++

Is there some private functionality in drivers/fwctl/ that
drivers/fwctl/cxl/ would need access?

That was the rationale for drivers/dax/cxl.c because it uses the private
drivers/dax/bus.h to get its work done.

Now, I am not opposed to drivers/fwctl/cxl.c as a library that the CXL
core can use to implement CXL fwctl support, if it helps to keep all
fwctl users co-located, but I don't see a need to make it a new CXL
sub-device.

>  include/cxl/cxl.h          |  1 +
>  include/cxl/mailbox.h      | 17 +++++++
>  include/uapi/fwctl/fwctl.h |  1 +
>  9 files changed, 178 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/fwctl/cxl/Makefile
>  create mode 100644 drivers/fwctl/cxl/cxl.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 93f74bc58d99..6b52c4168b94 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -9389,6 +9389,14 @@ L:	linux-kernel@vger.kernel.org
>  S:	Maintained
>  F:	drivers/fwctl/mlx5/
>  
> +FWCTL CXL DRIVER
> +M:	Dave Jiang <dave.jiang@intel.com>
> +R:	Dan Williams <dan.j.williams@intel.com>
> +R:	Jonathan Cameron <jonathan.cameron@huawei.com>
> +L:	linux-cxl@vger.kernel.org
> +S:	Maintained
> +F:	drivers/fwctl/cxl/

I don't think we need a new MAINTAINERS entry for this. Just extend the
CXL entry:

@@ -5772,6 +5772,7 @@ L:        linux-cxl@vger.kernel.org
 S:     Maintained
 F:     Documentation/driver-api/cxl
 F:     drivers/cxl/
+F:     drivers/fwctl/cxl*
 F:     include/cxl/
 F:     include/uapi/linux/cxl_mem.h

...and call it a day.

Re: [RFC PATCH v2 10/20] fwctl/cxl: Add support for get driver information

[Dan Williams]

Dave Jiang wrote:
> Add definition for fwctl_ops->info() to return driver information. The
> function will return the number of device mailbox commands supported by the
> fwctl char device.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v2:
> - Change driver info to report number of commands supported by the driver and device.
> ---
>  drivers/cxl/core/mbox.c  | 46 ++++++++++++++++++++++++++++++++++++++++
>  drivers/fwctl/cxl/cxl.c  | 23 ++++++++++++++++++--
>  include/cxl/cxl.h        |  2 ++
>  include/cxl/mailbox.h    |  1 +
>  include/uapi/fwctl/cxl.h | 22 +++++++++++++++++++
>  5 files changed, 92 insertions(+), 2 deletions(-)
>  create mode 100644 include/uapi/fwctl/cxl.h
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 12758e763650..f464eb42f08a 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -118,6 +118,14 @@ static u8 security_command_sets[] = {
>  	0x46, /* Security Passthrough */
>  };
>  
> +#define FWCTL_CXL_MAX_COMMANDS 3
> +/* Command set that is allowed with FWCTL-CXL */
> +static u16 fwctl_command_sets[] = {
> +	CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
> +	CXL_MBOX_OP_GET_FEATURE,
> +	CXL_MBOX_OP_SET_FEATURE,
> +};
> +
>  static bool cxl_is_security_command(u16 opcode)
>  {
>  	int i;
> @@ -527,6 +535,44 @@ static int cxl_validate_cmd_from_user(struct cxl_mbox_cmd *mbox_cmd,
>  				 send_cmd->in.payload);
>  }
>  
> +static struct cxl_mem_command *
> +fwctl_cxl_find_command(struct cxl_mailbox *cxl_mbox, u16 opcode)
> +{
> +	struct cxl_command_info *info;
> +	struct cxl_mem_command *cmd;
> +
> +	cmd = cxl_mem_find_command(opcode);
> +	if (!cmd)
> +		return NULL;
> +
> +	info = &cmd->info;
> +	if (test_bit(info->id, cxl_mbox->enabled_cmds) &&
> +	    !test_bit(info->id, cxl_mbox->exclusive_cmds))

If fwctl_cxl is an internal command submission path, it will always be
exclusive to the kernel. If {set,get}_features is exclusively CXL then
it gets even simpler to reason about.

> +		return cmd;
> +
> +	return NULL;
> +}
> +
> +/**
> + * cxl_mailbox_user_commands_supported() - Return number of user mailbox
> + *                                         commands supported.
> + * @cxl_mbox: cxl mailbox context
> + *
> + * Return: number of commands supported
> + */
> +int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox)
> +{
> +	int nr_cmds = 0;
> +
> +	for (int i = 0; i < FWCTL_CXL_MAX_COMMANDS; i++) {
> +		nr_cmds += !!fwctl_cxl_find_command(cxl_mbox,
> +						    fwctl_command_sets[i]);

This seems overly complicated for a submission that only needs to worry
about 3 commands max. You might even just statically claim to support
all three and let things naturally fail when no features are enabled, or
the device only supports get and not set.

I.e. what happens if this says SET_FEATURE is supported and someone
passes a read-only feature, or the SET FEATURE command is missing? That
would seem to be the job of input validation, not this registration
path, right?

> +	}
> +
> +	return nr_cmds;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_mailbox_user_commands_supported, CXL);
> +
>  int cxl_query_cmd(struct cxl_mailbox *cxl_mbox,
>  		  struct cxl_mem_query_commands __user *q)
>  {
> diff --git a/drivers/fwctl/cxl/cxl.c b/drivers/fwctl/cxl/cxl.c
> index c6a11cbbd937..5eb5eabf2bff 100644
> --- a/drivers/fwctl/cxl/cxl.c
> +++ b/drivers/fwctl/cxl/cxl.c
> @@ -6,6 +6,7 @@
>  #include <linux/device.h>
>  #include <cxl/cxl.h>
>  #include <cxl/mailbox.h>
> +#include <uapi/fwctl/cxl.h>
>  
>  struct cxlctl_uctx {
>  	struct fwctl_uctx uctx;
> @@ -21,6 +22,15 @@ DEFINE_FREE(cxlctl, struct cxlctl_dev *, if (_T) fwctl_put(&_T->fwctl))
>  
>  static int cxlctl_open_uctx(struct fwctl_uctx *uctx)
>  {
> +	struct cxlctl_uctx *cxlctl_uctx =
> +		container_of(uctx, struct cxlctl_uctx, uctx);

This is copy-pasta'd enough times that I think a:

struct cxlctl_uctx *to_uctx(struct fwctl_uctx *uctx)

...helper would not hurt.

Re: [RFC PATCH v2 11/20] fwctl: FWCTL_HW_INFO to return hardware information

[Dan Williams]

Dave Jiang wrote:
> Add an optional ioctl FWCTL_HW_INFO to pass command specific information
> to user space. An array of 'struct fwctl_command_info' will be returned
> from the ioctl. These commands are send to the driver via FWCTL_RPC call.
> The command info struct contains the command id, the related hardware
> opcode, input and output size for the command, and the effects the command
> has if it's a write command.

Maybe this made more immediate sense to Jason, but I do not see why CXL
needs a FWCTL_CMD_HWINFO when FWCTL_CMD_RPC can just convey the GET
SUPPORTED FEATURES payload?

Now, if we ever want to support more commands outside FEATURES, I expect
that will need to look like synthetic "feature" that the driver
publishes to fit it into the existing paradigm.

In other words CXL tools are already going to need to have knowledge of
feature payload formats, why does fwctl need to package it up in another
form?

If, for example, I was looking to support GET TIMESTAMP over CXL fwctl,
which I am not, I would say that would be a synthetic Linux Kernel CXL
feature UUID that gets documented and synthesized and fwctl core is none
the wiser.

Re: [RFC PATCH v2 13/20] fwctl/cxl: Add hw_info callback

[Dan Williams]

Dave Jiang wrote:
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/mbox.c | 42 +++++++++++++++++++++++++++++++++++++++++
>  drivers/fwctl/cxl/cxl.c | 25 ++++++++++++++++++++++++
>  include/cxl/mailbox.h   |  2 ++
>  3 files changed, 69 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index fba6bdd30a82..65fceabb9fe7 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -9,6 +9,8 @@
>  #include <cxlmem.h>
>  #include <cxl.h>
>  
> +#include <uapi/fwctl/fwctl.h>
> +
>  #include "core.h"
>  #include "trace.h"
>  
> @@ -573,6 +575,46 @@ int cxl_mailbox_user_commands_supported(struct cxl_mailbox *cxl_mbox)
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_mailbox_user_commands_supported, CXL);
>  
> +/**
> + * cxl_mailbox_user_commands_info_get() - Retrieve array of command info
> + * @cxl_mbox: cxl mailbox context
> + * @nr_cmds: number of commands to retrieve
> + * @outbuf: Output buffer to store array 'struct fwctl_command_info'
> + * @out_len: size of final output buffer
> + *
> + * Return: 0 for success, or -errno for failure.
> + */
> +int cxl_mailbox_user_commands_info_get(struct cxl_mailbox *cxl_mbox, int nr_cmds,
> +				       void *outbuf, size_t *out_len)
> +{
> +	struct fwctl_command_info *entry;
> +	struct cxl_command_info *info;
> +	struct cxl_mem_command *cmd;
> +
> +	if (nr_cmds > FWCTL_CXL_MAX_COMMANDS)
> +		return -EINVAL;
> +
> +	entry = outbuf;
> +	for (int i = 0; i < nr_cmds; i++) {
> +		cmd = fwctl_cxl_find_command(cxl_mbox, fwctl_command_sets[i]);
> +		if (!cmd)
> +			continue;
> +		info = &cmd->info;
> +		memset(entry, 0, sizeof(*entry));
> +		entry->id = info->id;
> +		entry->opcode = fwctl_command_sets[i];
> +		entry->effects = info->effects;
> +		entry->size_in = info->size_in;
> +		entry->size_out = info->size_out;
> +		entry++;
> +	}

Yeah, I am going to double down and just say that if userspace wants to
know the available features, it just issues GET SUPPORTED FEATURES
directly over FWCTL_CMD_RPC, and the kernel can return a cached and / or
augmented instance of that payload.

Everything this dynamic command could return could be automatically
generated by a script that converts CXL command effects to FWCTL
security scope, no need to ask for that information programmatically.

Otherwise I am missing something subtle about why this infrastructure is
needed?

Re: [RFC PATCH v2 14/20] cxl: Move defines and error codes from cxlmem.h to cxl/mailbox.h

[Dan Williams]

Dave Jiang wrote:
> Moving some internal definitions to cxl/mailbox.h in order to be accessed
> by FWCTL CXL driver.

I would first like to see some rationale for: drivers/fwctl/cxl.c over
drivers/cxl/core/fwctl.c. If the consensus is drivers/fwctl/cxl.c then I
think only the opcodes and 'struct cxl_features' needs to be exported.
cxl_mem_command() deals with CXL command ids which fwctl should never
care about.

Re: [RFC PATCH v2 15/20] fwctl/cxl: Add support for fwctl RPC command to enable CXL feature commands

[Dan Williams]

Dave Jiang wrote:
> fwctl provides a fwctl_ops->fw_rpc() callback in order to issue ioctls
> to a device. The cxl fwctl driver will start by supporting the CXL
> feature commands: Get Supported Features, Get Feature, and Set Feature.
> 
> The fw_rpc() callback provides 'enum fwctl_rpc_scope' parameter where
> it indicates the security scope of the call. The Get Supported Features

I would not classify GET SUPPORTED FEATURES as debug, I think it is safe to
be at the lowest FWCTL_RPC_CONFIGURATION, because supported features is
just the provisioned state of features in the device.

It is even more safe because the payload can be cached and augmented by
the kernel.

> and Get Feature calls can be executed with the scope of
> FWCTL_RPC_DEBUG_READ_ONLY.

I think this is a good default, although for well known spec defined
features we should audit if any are suitable to just mark as
FWCTL_RPC_CONFIGURATION.

> The Set Feature call is gated by the effects
> of the feature reported by Get Supported Features call for the specific
> feature.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/core.h      |   5 +-
>  drivers/cxl/core/mbox.c      | 260 +++++++++++++++++++++++++++++++----
>  drivers/cxl/core/memdev.c    |   4 +-
>  drivers/fwctl/cxl/cxl.c      | 101 +++++++++++++-
>  include/cxl/mailbox.h        |   6 +
>  include/uapi/fwctl/cxl.h     |  29 ++++
>  include/uapi/linux/cxl_mem.h |  12 ++
>  7 files changed, 387 insertions(+), 30 deletions(-)

Ok, now I see why the initial patches were touching the cxl_mem_command
array, this patch is converting fwctl requests and plugging them into
the "user ioctl" path.

I would have expected to skip the need to define CXL command ids for
fwctl and it by a raw passthrough of opcode to cxl_internal_send_cmd().
Maybe plugging into the user ioctl path saved from copy_{to,from}_user()
overhead for this? I would much rather fwctl leave the CXL user ioctl
path alone and keep this all self contained on top of
cxl_internal_send_cmd() if possible.

...but do holler if that unexpectedly blows up the complexity.

Re: [RFC PATCH v2 11/20] fwctl: FWCTL_HW_INFO to return hardware information

[Dave Jiang]

On 12/5/24 10:32 PM, Dan Williams wrote:
> Dave Jiang wrote:
>> Add an optional ioctl FWCTL_HW_INFO to pass command specific information
>> to user space. An array of 'struct fwctl_command_info' will be returned
>> from the ioctl. These commands are send to the driver via FWCTL_RPC call.
>> The command info struct contains the command id, the related hardware
>> opcode, input and output size for the command, and the effects the command
>> has if it's a write command.
> 
> Maybe this made more immediate sense to Jason, but I do not see why CXL
> needs a FWCTL_CMD_HWINFO when FWCTL_CMD_RPC can just convey the GET
> SUPPORTED FEATURES payload?

That is because I wrote the code with intention of supporting the existing chardev commands in addition to features. If we are only supporting feature commands fwctl, then we can drop this and make things simpler.

So heads up for Jason, we can drop this common ioctl.

DJ

> 
> Now, if we ever want to support more commands outside FEATURES, I expect
> that will need to look like synthetic "feature" that the driver
> publishes to fit it into the existing paradigm.
> 
> In other words CXL tools are already going to need to have knowledge of
> feature payload formats, why does fwctl need to package it up in another
> form?
> 
> If, for example, I was looking to support GET TIMESTAMP over CXL fwctl,
> which I am not, I would say that would be a synthetic Linux Kernel CXL
> feature UUID that gets documented and synthesized and fwctl core is none
> the wiser.

Re: [RFC PATCH v2 20/20] fwctl/cxl: Add documentation to FWCTL CXL

[Dan Williams]

Dave Jiang wrote:
[..]
> > As per earlier feedback I think we need to reject get/set even if they are not
> > listed in get supported features so that bad userspace doesn't get away
> > with sending them anyway.  This is particularly necessary as I suspect we'll
> > be removing features in the long run from what we let be accessed this way.
> > 
> > I'm not sure how we document the potential for removal of a feature that
> > was reported by an earlier kernel.
> 
> Userspace should query the driver for whatever features that are
> "available". Given one feature may exist for a device while another
> does not? So maybe the expectation from the app should be feature does
> not exist unless reported by the driver that it exists. 

Right, the whole point of this fwctl experiment is to make the CXL
device and firmware more responsible for honoring security and
compability contracts. If a device drops a feature in a firmware update
causing an application to regress, take it up with the feature vendor.
If the device violates the security expectations of the kernel that's
the device's mess to clean up.

Even with the legacy kernel mediated CXL command path both of those
risks still existed. Nothing in mainline stops a device from removing or
regressing a command it reported as available with a previous firmware
version, and nothing in mainline stops a device from cynically violating
the kernel's security model.

The fwctl interface lets the CXL driver document a security model on top
of CXL specification defined Command Effects, and then its up to market and
regulatory forces to keep implementations within that model. That frees
up the subsystem to focus on the primary functions of the device and
ABIs that read beyond "ioctl passthroughs that the kernel need never
care about".

RE: [RFC PATCH v2 02/20] cxl: Add Get Supported Features command for kernel usage

[Shiju Jose]

Hi Dan,

>-----Original Message-----
>From: Dan Williams <dan.j.williams@intel.com>
>Sent: 06 December 2024 00:33
>To: Dave Jiang <dave.jiang@intel.com>; linux-cxl@vger.kernel.org
>Cc: dan.j.williams@intel.com; ira.weiny@intel.com; vishal.l.verma@intel.com;
>alison.schofield@intel.com; Jonathan Cameron
><jonathan.cameron@huawei.com>; dave@stgolabs.net; jgg@nvidia.com; Shiju
>Jose <shiju.jose@huawei.com>
>Subject: Re: [RFC PATCH v2 02/20] cxl: Add Get Supported Features command
>for kernel usage
>
>Dave Jiang wrote:
>> CXL spec r3.1 8.2.9.6.1 Get Supported Features (Opcode 0500h) The
>> command retrieve the list of supported device-specific features
>> (identified by UUID) and general information about each Feature.
>>
>> The driver will retrieve the feature entries in order to make checks
>> and provide information for the Get Feature and Set Feature command.
>> One of the main piece of information retrieved are the effects a Set
>> Feature command would have for a particular feature.
>>
>> Co-developed-by: Shiju Jose <shiju.jose@huawei.com>
>> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>>
>> v2:
>> - fix feature entry pointer math.
>> - free mbox_out in cxl_get_supported_features(). (Shiju, Jonathan)
>> - replace sizeof(struct) with feat_size. (Shiju)
>> - replace reserved in feature structs with u8 type. (Shiju)
>> - change cxl_feat_entry->effects to set_effects. (Shiju)
>> - rearrange assigment of cxl_mbox->entries. (Jonathan)
>> - Separate no features from actual error. (Jonathan)
>> - Fix missing kdoc for entries. (Jonathan)
>> ---
>>  drivers/cxl/core/mbox.c      | 178 +++++++++++++++++++++++++++++++++++
>>  drivers/cxl/cxlmem.h         |  31 ++++++
>>  drivers/cxl/pci.c            |   4 +
>>  include/cxl/mailbox.h        |   4 +
>>  include/uapi/linux/cxl_mem.h |   1 +
>>  5 files changed, 218 insertions(+)
>>
>> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index
>> 880ac1dba3cc..4ba56f3d5a65 100644
>> --- a/drivers/cxl/core/mbox.c
>> +++ b/drivers/cxl/core/mbox.c
>> @@ -67,6 +67,7 @@ static struct cxl_mem_command
>cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
>>  	CXL_CMD(SET_SHUTDOWN_STATE, 0x1, 0, 0),
>>  	CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
>>  	CXL_CMD(GET_TIMESTAMP, 0, 0x8, 0),
>> +	CXL_CMD(GET_SUPPORTED_FEATURES, 0x8, CXL_VARIABLE_PAYLOAD,
>0),
>
>Since this patch is only about internal usage then no need for a
>cxl_mem_commands entry which is the translation from CXL mailbox op-codes
>to user CXL commands ids.
>
>I think ideally CXL features ABI is only ever a fwctl concern. Lets try to see if we
>can get away with never defining a CXL user command code for them.
>
>>  };
>>
>>  /*
>> @@ -790,6 +791,183 @@ static const uuid_t log_uuid[] = {
>>  	[VENDOR_DEBUG_UUID] = DEFINE_CXL_VENDOR_DEBUG_UUID,  };
>>
>> +static void cxl_free_features(void *features) {
>> +	kvfree(features);
>> +}
>> +
>> +static int cxl_get_supported_features_count(struct cxl_dev_state
>> +*cxlds) {
>> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
>> +	struct cxl_mbox_get_sup_feats_out mbox_out;
>> +	struct cxl_mbox_get_sup_feats_in mbox_in;
>> +	struct cxl_mbox_cmd mbox_cmd;
>> +	int rc;
>> +
>> +	memset(&mbox_in, 0, sizeof(mbox_in));
>> +	mbox_in.count = cpu_to_le32(sizeof(mbox_out));
>> +	memset(&mbox_out, 0, sizeof(mbox_out));
>> +	mbox_cmd = (struct cxl_mbox_cmd) {
>> +		.opcode = CXL_MBOX_OP_GET_SUPPORTED_FEATURES,
>> +		.size_in = sizeof(mbox_in),
>> +		.payload_in = &mbox_in,
>> +		.size_out = sizeof(mbox_out),
>> +		.payload_out = &mbox_out,
>> +		.min_out = sizeof(mbox_out),
>> +	};
>> +	rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
>> +	if (rc < 0)
>> +		return rc;
>> +
>> +	cxl_mbox->num_features = le16_to_cpu(mbox_out.supported_feats);
>
>I think this function should return a positive number of features or a negative
>error code, more below...

I  added this change in the EDAC CXL features setup and tested fine.

>
>> +
>> +	return 0;
>> +}
>> +
>> +int cxl_get_supported_features(struct cxl_dev_state *cxlds) {
>> +	struct cxl_mbox_get_sup_feats_out *mbox_out __free(kvfree) = NULL;
>> +	int remain_feats, max_size, max_feats, start, rc;
>> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
>> +	int feat_size = sizeof(struct cxl_feat_entry);
>> +	struct cxl_mbox_get_sup_feats_in mbox_in;
>> +	int hdr_size = sizeof(*mbox_out);
>> +	struct cxl_mbox_cmd mbox_cmd;
>> +	struct cxl_mem_command *cmd;
>> +	struct cxl_feat_entry *entry;
>> +
>> +	/* Get supported features is optional, need to check */
>> +	cmd =
>cxl_mem_find_command(CXL_MBOX_OP_GET_SUPPORTED_FEATURES);
>> +	if (!cmd)
>> +		return -EOPNOTSUPP;
>> +	if (!test_bit(cmd->info.id, cxl_mbox->enabled_cmds))
>> +		return -EOPNOTSUPP;
>> +
>> +	rc = cxl_get_supported_features_count(cxlds);
>> +	if (rc)
>> +		return rc;
>> +
>> +	if (!cxl_mbox->num_features) {
>> +		dev_dbg(cxl_mbox->host, "No CXL features enumerated.\n");
>> +		return 0;
>> +	}
>> +
>> +	struct cxl_feat_entry *entries __free(kvfree) =
>> +		kvmalloc(cxl_mbox->num_features * feat_size, GFP_KERNEL);
>> +
>> +	if (!entries)
>> +		return -ENOMEM;
>> +
>> +	max_size = cxl_mbox->payload_size - hdr_size;
>> +	/* max feat entries that can fit in mailbox max payload size */
>> +	max_feats = max_size / feat_size;
>> +	entry = &entries[0];
>> +
>> +	mbox_out = kvmalloc(cxl_mbox->payload_size, GFP_KERNEL);
>> +	if (!mbox_out)
>> +		return -ENOMEM;
>> +
>> +	start = 0;
>> +	remain_feats = cxl_mbox->num_features;
>> +	do {
>[..]
>> +		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
>> +		if (rc < 0)
>> +			return rc;
>
>Hmm, why "return rc" here, but "goto err" below?
>
>> +		if (mbox_cmd.size_out <= hdr_size) {
>> +			rc = -ENXIO;
>> +			goto err;
>
>So one of my new code smells for "needs more care" is cases where scoped-
>based-cleanup functions are still mixed with usage of "goto".
>
>In this case there are two dependent things to cleanup on error, "reset
>cxl_mbox->num_features" and "free @entries". So, to eliminate the goto, how
>about eliminate one of the things to cleanup with something like
>this:
>
>	struct cxl_features {
>		int num_features;
>		struct cxl_feat_entry entries[] __counted_by(num_features);
>	};
>
>	count = cxl_get_supported_features_count(cxl_mbox);
>	if (count <= 0)
>		return ...
>
>	struct cxl_features *features = __free(kvfree) =
>	        kvmalloc(struct_size(features, entries, count), GFP_KERNEL);
>
>	features->num_features = count;
>
>...and then use "cxl_mbox->features == NULL" as your "no features" case.

I  tried these changes in the EDAC CXL features setup and tested fine.

>
>
>> +		}
>> +
>> +		/*
>> +		 * Make sure retrieved out buffer is multiple of feature
>> +		 * entries.
>> +		 */
>> +		retrieved = mbox_cmd.size_out - hdr_size;
>> +		if (retrieved % feat_size) {
>> +			rc = -ENXIO;
>> +			goto err;
>> +		}
>> +
>> +		num_entries = le16_to_cpu(mbox_out->num_entries);
>> +		/*
>> +		 * If the reported output entries * defined entry size !=
>> +		 * retrieved output bytes, then the output package is incorrect.
>> +		 */
>> +		if (num_entries * feat_size != retrieved) {
>> +			rc = -ENXIO;
>> +			goto err;
>> +		}
>> +
>> +		memcpy(entry, mbox_out->ents, retrieved);
>> +		entry++;
>> +		/*
>> +		 * If the number of output entries is less than expected, add the
>> +		 * remaining entries to the next batch.
>> +		 */
>> +		remain_feats += copy_feats - num_entries;
>> +		start += num_entries;
>> +	} while (remain_feats);
>> +
>> +	cxl_mbox->entries = no_free_ptr(entries);
>> +	rc = devm_add_action_or_reset(cxl_mbox->host, cxl_free_features,
>> +				      cxl_mbox->entries);
>> +	if (rc)
>> +		return rc;
>> +
>> +	return 0;
>> +
>> +err:
>> +	cxl_mbox->num_features = 0;
>> +	return rc;
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_get_supported_features, CXL);
>> +
>> +int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t
>*feat_uuid,
>> +				    struct cxl_feat_entry *feat_entry_out) {
>> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
>> +	struct cxl_feat_entry *feat_entry;
>> +	int count;
>> +
>> +	/* Check CXL dev supports the feature */
>> +	feat_entry = &cxl_mbox->entries[0];
>> +	for (count = 0; count < cxl_mbox->num_features; count++, feat_entry++)
>{
>> +		if (uuid_equal(&feat_entry->uuid, feat_uuid)) {
>> +			memcpy(feat_entry_out, feat_entry,
>sizeof(*feat_entry_out));
>
>Why copy-out? Would it not be sufficient to just return the index in the
>features->entries[] array?

I  added this change in the EDAC CXL features setup and tested fine.

>
>[..]
>> diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h index
>> cc894f07a435..03c4b8ad84c1 100644
>> --- a/include/cxl/mailbox.h
>> +++ b/include/cxl/mailbox.h
>> @@ -50,6 +50,8 @@ struct cxl_mbox_cmd {
>>   *                (CXL 3.1 8.2.8.4.3 Mailbox Capabilities Register)
>>   * @mbox_mutex: mutex protects device mailbox and firmware
>>   * @mbox_wait: rcuwait for mailbox
>> + * @num_features: number of supported features entries
>> + * @features: list of supported feature entries
>>   * @mbox_send: @dev specific transport for transmitting mailbox commands
>>   */
>>  struct cxl_mailbox {
>> @@ -59,6 +61,8 @@ struct cxl_mailbox {
>>  	size_t payload_size;
>>  	struct mutex mbox_mutex; /* lock to protect mailbox context */
>>  	struct rcuwait mbox_wait;
>> +	int num_features;
>> +	struct cxl_feat_entry *entries;
>>  	int (*mbox_send)(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd
>> *cmd);  };
>>
>> diff --git a/include/uapi/linux/cxl_mem.h
>> b/include/uapi/linux/cxl_mem.h index c6c0fe27495d..bd2535962f70 100644
>> --- a/include/uapi/linux/cxl_mem.h
>> +++ b/include/uapi/linux/cxl_mem.h
>> @@ -50,6 +50,7 @@
>>  	___C(GET_LOG_CAPS, "Get Log Capabilities"),			  \
>>  	___C(CLEAR_LOG, "Clear Log"),					  \
>>  	___C(GET_SUP_LOG_SUBLIST, "Get Supported Logs Sub-List"),	  \
>> +	___C(GET_SUPPORTED_FEATURES, "Get Supported Features"),
>	  \
>
>Again, for a patch that claims to be kernel internal only enabling, lets keep it
>internal-only.

Thanks,
Shiju

RE: [RFC PATCH v2 04/20] cxl/mbox: Add GET_FEATURE mailbox command

[Shiju Jose]

>-----Original Message-----
>From: Dan Williams <dan.j.williams@intel.com>
>Sent: 06 December 2024 00:44
>To: Dave Jiang <dave.jiang@intel.com>; linux-cxl@vger.kernel.org
>Cc: dan.j.williams@intel.com; ira.weiny@intel.com; vishal.l.verma@intel.com;
>alison.schofield@intel.com; Jonathan Cameron
><jonathan.cameron@huawei.com>; dave@stgolabs.net; jgg@nvidia.com; Shiju
>Jose <shiju.jose@huawei.com>
>Subject: Re: [RFC PATCH v2 04/20] cxl/mbox: Add GET_FEATURE mailbox
>command
>
>Dave Jiang wrote:
>> From: Shiju Jose <shiju.jose@huawei.com>
>>
>> Add support for GET_FEATURE mailbox command.
>>
>> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
>> The settings of a feature can be retrieved using Get Feature command.
>> CXL spec 3.1 section 8.2.9.6.2 describes Get Feature command.
>>
>> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>> This patch is not needed by this series except the opcode enumeration.
>> ---
>
>Wait, then why is it included?
>
>I would say fold the small bit that you need into the previous patch, especially if
>we think that fwctl can cover all the "get feature" use cases.
>
>>  drivers/cxl/core/mbox.c | 41
>+++++++++++++++++++++++++++++++++++++++++
>>  drivers/cxl/cxlmem.h    | 26 ++++++++++++++++++++++++++
>>  2 files changed, 67 insertions(+)
>>
>> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index
>> 4ba56f3d5a65..f70bd7d28ff8 100644
>> --- a/drivers/cxl/core/mbox.c
>> +++ b/drivers/cxl/core/mbox.c
>> @@ -968,6 +968,47 @@ int cxl_get_supported_feature_entry(struct
>> cxl_dev_state *cxlds, const uuid_t *f  }
>> EXPORT_SYMBOL_NS_GPL(cxl_get_supported_feature_entry, CXL);
>>
>> +size_t cxl_get_feature(struct cxl_dev_state *cxlds, const uuid_t feat_uuid,
>> +		       enum cxl_get_feat_selection selection,
>> +		       void *feat_out, size_t feat_out_size) {
>> +	struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;
>
>cxlds is otherwise unused in this function, just pass in @cxl_mbox directly.

Hi Dan,

I  modified to cxl_get_feature(struct cxl_mailbox *cxl_mbox, ..) in the EDAC CXL features setup and tested fine.

Thanks,
Shiju

RE: [RFC PATCH v2 06/20] cxl/mbox: Add SET_FEATURE mailbox command

[Shiju Jose]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

>-----Original Message-----
>From: Dan Williams <dan.j.williams@intel.com>
>Sent: 06 December 2024 00:48
>To: Dave Jiang <dave.jiang@intel.com>; linux-cxl@vger.kernel.org
>Cc: dan.j.williams@intel.com; ira.weiny@intel.com; vishal.l.verma@intel.com;
>alison.schofield@intel.com; Jonathan Cameron
><jonathan.cameron@huawei.com>; dave@stgolabs.net; jgg@nvidia.com; Shiju
>Jose <shiju.jose@huawei.com>
>Subject: Re: [RFC PATCH v2 06/20] cxl/mbox: Add SET_FEATURE mailbox
>command
>
>Dave Jiang wrote:
>> From: Shiju Jose <shiju.jose@huawei.com>
>>
>> Add support for SET_FEATURE mailbox command.
>>
>> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
>> CXL devices supports features with changeable attributes.
>> The settings of a feature can be optionally modified using Set Feature
>> command.
>> CXL spec 3.1 section 8.2.9.6.3 describes Set Feature command.
>>
>> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>> This patch is not needed except for the opcode enumeration.
>> ---
>
>Same reaction as a the last patch.
>
>We can always circle back to add this on top if really necessary, but lets keep
>this set on topic with the cover letter of enabling fwctl for features.

Hi Dan, 

I  modified to cxl_set_feature(struct cxl_mailbox *cxl_mbox, ..) in the EDAC CXL features setup and tested fine.

Thanks,
Shiju

[-- Attachment #2: Type: message/rfc822, Size: 6627 bytes --]

From: Dan Williams <dan.j.williams@intel.com>
To: Dave Jiang <dave.jiang@intel.com>, "linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>
Cc: "dan.j.williams@intel.com" <dan.j.williams@intel.com>, "ira.weiny@intel.com" <ira.weiny@intel.com>, "vishal.l.verma@intel.com" <vishal.l.verma@intel.com>, "alison.schofield@intel.com" <alison.schofield@intel.com>, Jonathan Cameron <jonathan.cameron@huawei.com>, "dave@stgolabs.net" <dave@stgolabs.net>, "jgg@nvidia.com" <jgg@nvidia.com>, Shiju Jose <shiju.jose@huawei.com>
Subject: Re: [RFC PATCH v2 04/20] cxl/mbox: Add GET_FEATURE mailbox command
Date: Fri, 6 Dec 2024 00:44:20 +0000
Message-ID: <675248e44c037_25073294f4@dwillia2-xfh.jf.intel.com.notmuch>

Dave Jiang wrote:
> From: Shiju Jose <shiju.jose@huawei.com>
>
> Add support for GET_FEATURE mailbox command.
>
> CXL spec 3.1 section 8.2.9.6 describes optional device specific features.
> The settings of a feature can be retrieved using Get Feature command.
> CXL spec 3.1 section 8.2.9.6.2 describes Get Feature command.
>
> Signed-off-by: Shiju Jose <shiju.jose@huawei.com>
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> This patch is not needed by this series except the opcode enumeration.
> ---

Wait, then why is it included?

I would say fold the small bit that you need into the previous patch,
especially if we think that fwctl can cover all the "get feature" use
cases.

>  drivers/cxl/core/mbox.c | 41 +++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxlmem.h    | 26 ++++++++++++++++++++++++++
>  2 files changed, 67 insertions(+)
>
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 4ba56f3d5a65..f70bd7d28ff8 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -968,6 +968,47 @@ int cxl_get_supported_feature_entry(struct cxl_dev_state *cxlds, const uuid_t *f
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_get_supported_feature_entry, CXL);
>
> +size_t cxl_get_feature(struct cxl_dev_state *cxlds, const uuid_t feat_uuid,
> +                    enum cxl_get_feat_selection selection,
> +                    void *feat_out, size_t feat_out_size)
> +{
> +     struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox;

cxlds is otherwise unused in this function, just pass in @cxl_mbox
directly.

Re: [RFC PATCH v2 09/20] fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands

[Jason Gunthorpe]

On Thu, Dec 05, 2024 at 05:21:13PM -0800, Dan Williams wrote:
> Dave Jiang wrote:
> > Add a cxl fwctl driver to allow sending of CXL feature
> > commands from userspace through as ioctls. Create a driver skeleton for
> > initial setup.
> 
> I don't understand why fwctl support of the *existing* mailbox needs
> yet another sub-device and driver?

It is how to modprobe the fwctl driver independently from the main cxl
codebase.

You want the fwctl parts to be in their own loadable module even if
cxl core is built in, and the loadable module should auto load when
cxl is present in the system.

Spawning a new struct device is the best way to do this.

> In the mlx5 case, as I understand it, multiple families of device can
> register the common mlx5_fwctl auxiliary device to share a common driver
> for that piece. For CXL that shared object is the 'struct cxl_mailbox'.

There is only one mlx5_core driver. The use of auxiliary bus in mlx5
is purely to split the driver up into many modules in many
subsystems. Each subsystem probes its own driver that autoloads if the
underlying HW supports that subsystem.

Basically, it creates modularity within the kernel for complex
multi-subsystem devices.

> Now, I am not opposed to drivers/fwctl/cxl.c as a library that the CXL
> core can use to implement CXL fwctl support, if it helps to keep all
> fwctl users co-located, but I don't see a need to make it a new CXL
> sub-device.

I was expecting that subsystem drivers would reside in the subsystem
directory, so it shouldn't be a library, it should be the fwctl
particular parts here.

> @@ -5772,6 +5772,7 @@ L:        linux-cxl@vger.kernel.org
>  S:     Maintained
>  F:     Documentation/driver-api/cxl
>  F:     drivers/cxl/
> +F:     drivers/fwctl/cxl*
>  F:     include/cxl/
>  F:     include/uapi/linux/cxl_mem.h
> 
> ...and call it a day.

Makes sense

Jason

Re: [RFC PATCH v2 11/20] fwctl: FWCTL_HW_INFO to return hardware information

[Jason Gunthorpe]

On Fri, Dec 06, 2024 at 11:39:03AM -0700, Dave Jiang wrote:
> 
> 
> On 12/5/24 10:32 PM, Dan Williams wrote:
> > Dave Jiang wrote:
> >> Add an optional ioctl FWCTL_HW_INFO to pass command specific information
> >> to user space. An array of 'struct fwctl_command_info' will be returned
> >> from the ioctl. These commands are send to the driver via FWCTL_RPC call.
> >> The command info struct contains the command id, the related hardware
> >> opcode, input and output size for the command, and the effects the command
> >> has if it's a write command.
> > 
> > Maybe this made more immediate sense to Jason, but I do not see why CXL
> > needs a FWCTL_CMD_HWINFO when FWCTL_CMD_RPC can just convey the GET
> > SUPPORTED FEATURES payload?
> 
> That is because I wrote the code with intention of supporting the
> existing chardev commands in addition to features. If we are only
> supporting feature commands fwctl, then we can drop this and make
> things simpler.

If that is what this series is doing then lets drop it, if you need it
later it is easy to add

Jason

Re: [RFC PATCH v2 09/20] fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands

[Dan Williams]

Jason Gunthorpe wrote:
> On Thu, Dec 05, 2024 at 05:21:13PM -0800, Dan Williams wrote:
> > Dave Jiang wrote:
> > > Add a cxl fwctl driver to allow sending of CXL feature
> > > commands from userspace through as ioctls. Create a driver skeleton for
> > > initial setup.
> > 
> > I don't understand why fwctl support of the *existing* mailbox needs
> > yet another sub-device and driver?
> 
> It is how to modprobe the fwctl driver independently from the main cxl
> codebase.
> 
> You want the fwctl parts to be in their own loadable module even if
> cxl core is built in, and the loadable module should auto load when
> cxl is present in the system.
> 
> Spawning a new struct device is the best way to do this.

I agree with that, and CXL is already a multi-device subsystem for the
same purpose. However, a device just for the few subcommands that go
through fwctl feels too fine grained. I would say either move cxl_mbox
to be its own device with a driver that does CXL native commands and CXL
'feature' commands via fwctl, or create a 'CXL features' device that
support any functionality implemented by the 'features' command set of
which fwctl is just the direct command submission ABI for that driver
which may have other ABIs like EDAC associated with it.

The latter is similar to the proposed cxl_fwctl device, but focuses more
on the 'CXL features' aspect and less on the 'this is a fwctl module'.

Re: [RFC PATCH v2 09/20] fwctl/cxl: Add driver for CXL mailbox for handling CXL features commands

[Jason Gunthorpe]

On Mon, Dec 09, 2024 at 12:35:14PM -0800, Dan Williams wrote:
> Jason Gunthorpe wrote:
> > On Thu, Dec 05, 2024 at 05:21:13PM -0800, Dan Williams wrote:
> > > Dave Jiang wrote:
> > > > Add a cxl fwctl driver to allow sending of CXL feature
> > > > commands from userspace through as ioctls. Create a driver skeleton for
> > > > initial setup.
> > > 
> > > I don't understand why fwctl support of the *existing* mailbox needs
> > > yet another sub-device and driver?
> > 
> > It is how to modprobe the fwctl driver independently from the main cxl
> > codebase.
> > 
> > You want the fwctl parts to be in their own loadable module even if
> > cxl core is built in, and the loadable module should auto load when
> > cxl is present in the system.
> > 
> > Spawning a new struct device is the best way to do this.
> 
> I agree with that, and CXL is already a multi-device subsystem for the
> same purpose. However, a device just for the few subcommands that go
> through fwctl feels too fine grained. I would say either move cxl_mbox
> to be its own device with a driver that does CXL native commands and CXL
> 'feature' commands via fwctl, or create a 'CXL features' device that
> support any functionality implemented by the 'features' command set of
> which fwctl is just the direct command submission ABI for that driver
> which may have other ABIs like EDAC associated with it.
> 
> The latter is similar to the proposed cxl_fwctl device, but focuses more
> on the 'CXL features' aspect and less on the 'this is a fwctl module'.

Well, the advantage of slicing it according to subsystem is you can
slice the code according to subsystem too. If you have a complex cxl
device then it can't just have a simple driver in fwctl, with its own
loadlable module.

I like the idea that the fwctl interfaces all have their own loadlable
modules, it gives the admin an option to blacklist the module if they
don't want fwctl for some reason without impacting anything else.

The original version of these patches used the auxbus, which is
intended for *exactly* this purpose. I think that got a bit lost with
the feedback to use the cxl bus instead?

Jason

Re: [RFC PATCH v2 0/20] fwctl/cxl: Add CXL feature commands support via fwctl

[Jason Gunthorpe]

On Fri, Nov 15, 2024 at 02:25:33PM -0700, Dave Jiang wrote:

> This series add support for CXL feature commands using the FWCTL framework [1].
> The code is untested and I'm looking for architectural and implementation feedback.
> While CXL currently has a chardev for user ioctls to send some mailbox
> commands to a memory device, the fwctl framework provides more security policies
> that can be a potential vehicle to move CXL ioctl path to that.

How are things going on the CXL side here? Do we want to make a push
to get this done in 6.14? (ie starting in two weeks?) This is approx
the timeline I had in mind for people to get their drivers ready.

Thanks,
Jason

Re: [RFC PATCH v2 0/20] fwctl/cxl: Add CXL feature commands support via fwctl

[Dave Jiang]

On 1/21/25 1:30 PM, Jason Gunthorpe wrote:
> On Fri, Nov 15, 2024 at 02:25:33PM -0700, Dave Jiang wrote:
> 
>> This series add support for CXL feature commands using the FWCTL framework [1].
>> The code is untested and I'm looking for architectural and implementation feedback.
>> While CXL currently has a chardev for user ioctls to send some mailbox
>> commands to a memory device, the fwctl framework provides more security policies
>> that can be a potential vehicle to move CXL ioctl path to that.
> 
> How are things going on the CXL side here? Do we want to make a push
> to get this done in 6.14? (ie starting in two weeks?) This is approx
> the timeline I had in mind for people to get their drivers ready.

Really close. I'm trying to finish testing and wrap up the unit tests on the user side. I'm hoping I can post the v1 post RFC this or next week.

> 
> Thanks,
> Jason
>