Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[Xu Yilun]

On Sun, Dec 22, 2024 at 05:53:30PM -0800, Yidong Zhang wrote:
> 
> 
> On 11/18/24 23:07, Xu Yilun wrote:
> > 
> > > > > +obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
> > > > IMHO the naming vmgmt is hard to understand, any better idea?
> > > The "v" stand for Versal. We would change to amd-vpci for Versal based pcie
> > "v" + "pci" is quite a misleading term, maybe just versal-pci?
> 
> Hi Yilun,
> 
> I sent the V2 patch and refactored the driver as versal-pci now.
> One more thing that I kept in V2 was the firmware_upload. I forgot to
> mention that I'd love to switch to the newly proposed interface once
> it is ready. I saw the proposal was now as config_fs and it was not merged

Good to know that.

I didn't start reviewing the v2 yet. But one thing is that now the
versal-pci FPGA manager has no user because of the ongoing uAPI, so
cannot be merged, and I won't pay much effort on this series for now.

Thanks,
Yilun

> yet.
> 
> Happy Holidays.
> 
> Thanks,
> David

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[Xu Yilun]

On Wed, Dec 25, 2024 at 10:10:06PM -0800, Yidong Zhang wrote:
> 
> 
> On 3/12/23 11:03, Xu Yilun wrote:
> > Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> > 
> > 
> > On Sun, Dec 22, 2024 at 05:53:30PM -0800, Yidong Zhang wrote:
> > > 
> > > 
> > > On 11/18/24 23:07, Xu Yilun wrote:
> > > > 
> > > > > > > +obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
> > > > > > IMHO the naming vmgmt is hard to understand, any better idea?
> > > > > The "v" stand for Versal. We would change to amd-vpci for Versal based pcie
> > > > "v" + "pci" is quite a misleading term, maybe just versal-pci?
> > > 
> > > Hi Yilun,
> > > 
> > > I sent the V2 patch and refactored the driver as versal-pci now.
> > > One more thing that I kept in V2 was the firmware_upload. I forgot to
> > > mention that I'd love to switch to the newly proposed interface once
> > > it is ready. I saw the proposal was now as config_fs and it was not merged
> > 
> > Good to know that.
> > 
> > I didn't start reviewing the v2 yet. But one thing is that now the
> > versal-pci FPGA manager has no user because of the ongoing uAPI, so
> > cannot be merged, and I won't pay much effort on this series for now.
> 
> Hi Yilun,
> 
> Can we add this as TODO in the future when the uAPI solution is ready to
> switch? We spent some time to refactor the driver and address most of your

Sorry, generally kernel won't merge code that has no user, which means
the code are not tested.

> comments in the V2. Hopefully, can you please start reviewing the fpga_mgr
> and other driver code?

Yes, I can review the code when possible. But will not merge it.

> 
> We'd think that we use the firmware_upload for 1st approach and start
> letting user use this driver.
> 
> We definitely will switch to the new uAPI as soon as it is ready in the
> linux fpga driver. But we'd not like this uAPI holds up everything we
> already spent times.

If you want speed up, then collaborate to develop/review the dependent
patchset, that's how the community works. I actually don't want to be
the only reviewer in this mail list, and others just send patches.

Thanks,
Yilun

[PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[David Zhang]

From: Yidong Zhang <yidong.zhang@amd.com>

AMD Versal based PCIe card, including V70, is designed for AI inference
efficiency and is tuned for video analytics and natural language processing
applications.

Add the driver to support AMD Versal card management physical function.
Only very basic functionalities are added.
  - module and PCI device initialization
  - fpga framework ops callbacks
  - communication with user physical function

Co-developed-by: DMG Karthik <Karthik.DMG@amd.com>
Signed-off-by: DMG Karthik <Karthik.DMG@amd.com>
Co-developed-by: Nishad Saraf <nishads@amd.com>
Signed-off-by: Nishad Saraf <nishads@amd.com>
Co-developed-by: Prapul Krishnamurthy <prapulk@amd.com>
Signed-off-by: Prapul Krishnamurthy <prapulk@amd.com>
Signed-off-by: Yidong Zhang <yidong.zhang@amd.com>
---
 MAINTAINERS                    |   7 +
 drivers/fpga/Kconfig           |   3 +
 drivers/fpga/Makefile          |   3 +
 drivers/fpga/amd/Kconfig       |  17 ++
 drivers/fpga/amd/Makefile      |   6 +
 drivers/fpga/amd/vmgmt-comms.c | 344 ++++++++++++++++++++++++++++
 drivers/fpga/amd/vmgmt-comms.h |  14 ++
 drivers/fpga/amd/vmgmt.c       | 395 +++++++++++++++++++++++++++++++++
 drivers/fpga/amd/vmgmt.h       | 100 +++++++++
 include/uapi/linux/vmgmt.h     |  25 +++
 10 files changed, 914 insertions(+)
 create mode 100644 drivers/fpga/amd/Kconfig
 create mode 100644 drivers/fpga/amd/Makefile
 create mode 100644 drivers/fpga/amd/vmgmt-comms.c
 create mode 100644 drivers/fpga/amd/vmgmt-comms.h
 create mode 100644 drivers/fpga/amd/vmgmt.c
 create mode 100644 drivers/fpga/amd/vmgmt.h
 create mode 100644 include/uapi/linux/vmgmt.h

diff --git a/MAINTAINERS b/MAINTAINERS
index a097afd76ded..645f00ccb342 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1185,6 +1185,13 @@ M:	Sanjay R Mehta <sanju.mehta@amd.com>
 S:	Maintained
 F:	drivers/spi/spi-amd.c
 
+AMD VERSAL PCI DRIVER
+M:	Yidong Zhang <yidong.zhang@amd.com>
+L:	linux-fpga@vger.kernel.org
+S:	Supported
+F:	drivers/fpga/amd/
+F:	include/uapi/linux/vmgmt.h
+
 AMD XGBE DRIVER
 M:	"Shyam Sundar S K" <Shyam-sundar.S-k@amd.com>
 L:	netdev@vger.kernel.org
diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 37b35f58f0df..dce060a7bd8f 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -290,4 +290,7 @@ config FPGA_MGR_LATTICE_SYSCONFIG_SPI
 
 source "drivers/fpga/tests/Kconfig"
 
+# Driver files
+source "drivers/fpga/amd/Kconfig"
+
 endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index aeb89bb13517..5e8a3869f9a0 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -58,5 +58,8 @@ obj-$(CONFIG_FPGA_DFL_NIOS_INTEL_PAC_N3000)	+= dfl-n3000-nios.o
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o
 
+# AMD PCIe Versal Management Driver
+obj-y					+= amd/
+
 # KUnit tests
 obj-$(CONFIG_FPGA_KUNIT_TESTS)		+= tests/
diff --git a/drivers/fpga/amd/Kconfig b/drivers/fpga/amd/Kconfig
new file mode 100644
index 000000000000..126bc579a333
--- /dev/null
+++ b/drivers/fpga/amd/Kconfig
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config AMD_VERSAL_MGMT
+	tristate "AMD PCIe Versal Management Driver"
+	select FW_LOADER
+	select FW_UPLOAD
+	select REGMAP_MMIO
+	depends on FPGA_BRIDGE
+	depends on FPGA_REGION
+	depends on HAS_IOMEM
+	depends on PCI
+	help
+	  AMD PCIe Versal Management Driver provides management services to
+	  download firmware, program bitstream, collect sensor data, control
+	  resets, and communicate with the User function.
+
+	  If "M" is selected, the driver module will be amd-vmgmt.
diff --git a/drivers/fpga/amd/Makefile b/drivers/fpga/amd/Makefile
new file mode 100644
index 000000000000..3e4c6dd3b787
--- /dev/null
+++ b/drivers/fpga/amd/Makefile
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_AMD_VERSAL_MGMT)			+= amd-vmgmt.o
+
+amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)		:= vmgmt.o	\
+						   vmgmt-comms.o
diff --git a/drivers/fpga/amd/vmgmt-comms.c b/drivers/fpga/amd/vmgmt-comms.c
new file mode 100644
index 000000000000..bed0d369a744
--- /dev/null
+++ b/drivers/fpga/amd/vmgmt-comms.c
@@ -0,0 +1,344 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for Versal PCIe device
+ *
+ * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/jiffies.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/timer.h>
+#include <linux/uuid.h>
+#include <linux/workqueue.h>
+
+#include "vmgmt.h"
+#include "vmgmt-comms.h"
+
+#define COMMS_PROTOCOL_VERSION			1
+#define COMMS_PCI_BAR_OFF			0x2000000
+#define COMMS_TIMER				(HZ / 10)
+#define COMMS_DATA_LEN				16
+#define COMMS_DATA_TYPE_MASK			GENMASK(7, 0)
+#define COMMS_DATA_EOM_MASK			BIT(31)
+#define COMMS_MSG_END				BIT(31)
+
+#define COMMS_REG_WRDATA_OFF			0x0
+#define COMMS_REG_RDDATA_OFF			0x8
+#define COMMS_REG_STATUS_OFF			0x10
+#define COMMS_REG_ERROR_OFF			0x14
+#define COMMS_REG_RIT_OFF			0x1C
+#define COMMS_REG_IS_OFF			0x20
+#define COMMS_REG_IE_OFF			0x24
+#define COMMS_REG_CTRL_OFF			0x2C
+#define COMMS_REGS_SIZE				0x1000
+
+#define COMMS_IRQ_DISABLE_ALL			0
+#define COMMS_IRQ_RECEIVE_ENABLE		BIT(1)
+#define COMMS_IRQ_CLEAR_ALL			GENMASK(2, 0)
+#define COMMS_CLEAR_FIFO			GENMASK(1, 0)
+#define COMMS_RECEIVE_THRESHOLD			15
+
+enum comms_req_ops {
+	COMMS_REQ_OPS_UNKNOWN			= 0,
+	COMMS_REQ_OPS_HOT_RESET			= 5,
+	COMMS_REQ_OPS_GET_PROTOCOL_VERSION	= 19,
+	COMMS_REQ_OPS_GET_XCLBIN_UUID		= 20,
+	COMMS_REQ_OPS_MAX,
+};
+
+enum comms_msg_type {
+	COMMS_MSG_INVALID			= 0,
+	COMMS_MSG_START				= 2,
+	COMMS_MSG_BODY				= 3,
+};
+
+enum comms_msg_service_type {
+	COMMS_MSG_SRV_RESPONSE			= BIT(0),
+	COMMS_MSG_SRV_REQUEST			= BIT(1),
+};
+
+struct comms_hw_msg {
+	struct {
+		u32		type;
+		u32		payload_size;
+	} header;
+	struct {
+		u64		id;
+		u32		flags;
+		u32		size;
+		u32		payload[COMMS_DATA_LEN - 6];
+	} body;
+} __packed;
+
+struct comms_srv_req {
+	u64			flags;
+	u32			opcode;
+	u32			data[];
+};
+
+struct comms_srv_ver_resp {
+	u32			version;
+};
+
+struct comms_srv_uuid_resp {
+	uuid_t			uuid;
+};
+
+struct comms_msg {
+	u64			id;
+	u32			flags;
+	u32			len;
+	u32			bytes_read;
+	u32			data[10];
+};
+
+struct comms_device {
+	struct vmgmt_device	*vdev;
+	struct regmap		*regmap;
+	struct timer_list	timer;
+	struct work_struct	work;
+};
+
+static bool comms_regmap_rd_regs(struct device *dev, unsigned int reg)
+{
+	switch (reg) {
+	case COMMS_REG_RDDATA_OFF:
+	case COMMS_REG_IS_OFF:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static bool comms_regmap_wr_regs(struct device *dev, unsigned int reg)
+{
+	switch (reg) {
+	case COMMS_REG_WRDATA_OFF:
+	case COMMS_REG_IS_OFF:
+	case COMMS_REG_IE_OFF:
+	case COMMS_REG_CTRL_OFF:
+	case COMMS_REG_RIT_OFF:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static bool comms_regmap_nir_regs(struct device *dev, unsigned int reg)
+{
+	switch (reg) {
+	case COMMS_REG_RDDATA_OFF:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static const struct regmap_config comms_regmap_config = {
+	.name = "comms_config",
+	.reg_bits = 32,
+	.reg_stride = 4,
+	.val_bits = 32,
+	.readable_reg = comms_regmap_rd_regs,
+	.writeable_reg = comms_regmap_wr_regs,
+	.readable_noinc_reg = comms_regmap_nir_regs,
+};
+
+static inline struct comms_device *to_ccdev_work(struct work_struct *w)
+{
+	return container_of(w, struct comms_device, work);
+}
+
+static inline struct comms_device *to_ccdev_timer(struct timer_list *t)
+{
+	return container_of(t, struct comms_device, timer);
+}
+
+static u32 comms_set_uuid_resp(struct vmgmt_device *vdev, void *payload)
+{
+	struct comms_srv_uuid_resp *resp;
+	u32 resp_len = sizeof(*resp);
+
+	resp = (struct comms_srv_uuid_resp *)payload;
+	uuid_copy(&resp->uuid, &vdev->xclbin_uuid);
+	vmgmt_dbg(vdev, "xclbin UUID: %pUb", &resp->uuid);
+
+	return resp_len;
+}
+
+static u32 comms_set_protocol_resp(void *payload)
+{
+	struct comms_srv_ver_resp *resp = (struct comms_srv_ver_resp *)payload;
+	u32 resp_len = sizeof(*resp);
+
+	resp->version = COMMS_PROTOCOL_VERSION;
+
+	return sizeof(resp_len);
+}
+
+static void comms_send_response(struct comms_device *ccdev,
+				struct comms_msg *msg)
+{
+	struct comms_srv_req *req = (struct comms_srv_req *)msg->data;
+	struct vmgmt_device *vdev = ccdev->vdev;
+	struct comms_hw_msg response = {0};
+	u32 size;
+	int ret;
+	u8 i;
+
+	switch (req->opcode) {
+	case COMMS_REQ_OPS_GET_PROTOCOL_VERSION:
+		size = comms_set_protocol_resp(response.body.payload);
+		break;
+	case COMMS_REQ_OPS_GET_XCLBIN_UUID:
+		size = comms_set_uuid_resp(vdev, response.body.payload);
+		break;
+	default:
+		vmgmt_err(vdev, "Unsupported request opcode: %d", req->opcode);
+		*response.body.payload = -1;
+		size = sizeof(int);
+	}
+
+	vmgmt_dbg(vdev, "Response opcode: %d", req->opcode);
+
+	response.header.type = COMMS_MSG_START | COMMS_MSG_END;
+	response.header.payload_size = size;
+
+	response.body.flags = COMMS_MSG_SRV_RESPONSE;
+	response.body.size = size;
+	response.body.id = msg->id;
+
+	for (i = 0; i < COMMS_DATA_LEN; i++) {
+		ret = regmap_write(ccdev->regmap, COMMS_REG_WRDATA_OFF, ((u32 *)&response)[i]);
+		if (ret < 0) {
+			vmgmt_err(vdev, "regmap write failed: %d", ret);
+			return;
+		}
+	}
+}
+
+#define STATUS_IS_READY(status) ((status) & BIT(1))
+#define STATUS_IS_ERROR(status) ((status) & BIT(2))
+
+static void comms_check_request(struct work_struct *w)
+{
+	struct comms_device *ccdev = to_ccdev_work(w);
+	u32 status = 0, request[COMMS_DATA_LEN] = {0};
+	struct comms_hw_msg *hw_msg;
+	struct comms_msg msg;
+	u8 type, eom;
+	int ret;
+	int i;
+
+	ret = regmap_read(ccdev->regmap, COMMS_REG_IS_OFF, &status);
+	if (ret) {
+		vmgmt_err(ccdev->vdev, "regmap read failed: %d", ret);
+		return;
+	}
+	if (!STATUS_IS_READY(status))
+		return;
+	if (STATUS_IS_ERROR(status)) {
+		vmgmt_err(ccdev->vdev, "An error has occurred with comms");
+		return;
+	}
+
+	/* ACK status */
+	regmap_write(ccdev->regmap, COMMS_REG_IS_OFF, status);
+
+	for (i = 0; i < COMMS_DATA_LEN; i++) {
+		if (regmap_read(ccdev->regmap, COMMS_REG_RDDATA_OFF, &request[i]) < 0) {
+			vmgmt_err(ccdev->vdev, "regmap read failed");
+			return;
+		}
+	}
+
+	hw_msg = (struct comms_hw_msg *)request;
+	type = FIELD_GET(COMMS_DATA_TYPE_MASK, hw_msg->header.type);
+	eom = FIELD_GET(COMMS_DATA_EOM_MASK, hw_msg->header.type);
+
+	/* Only support fixed size 64B messages */
+	if (!eom || type != COMMS_MSG_START) {
+		vmgmt_err(ccdev->vdev, "Unsupported message format or length");
+		return;
+	}
+
+	msg.flags = hw_msg->body.flags;
+	msg.len = hw_msg->body.size;
+	msg.id = hw_msg->body.id;
+
+	if (msg.flags != COMMS_MSG_SRV_REQUEST) {
+		vmgmt_err(ccdev->vdev, "Unsupported service request");
+		return;
+	}
+
+	if (hw_msg->body.size > sizeof(msg.data) * sizeof(msg.data[0])) {
+		vmgmt_err(ccdev->vdev, "msg is too big: %d", hw_msg->body.size);
+		return;
+	}
+	memcpy(msg.data, hw_msg->body.payload, hw_msg->body.size);
+
+	/* Now decode and respond appropriately */
+	comms_send_response(ccdev, &msg);
+}
+
+static void comms_sched_work(struct timer_list *t)
+{
+	struct comms_device *ccdev = to_ccdev_timer(t);
+
+	/* Schedule a work in the general workqueue */
+	schedule_work(&ccdev->work);
+	/* Periodic timer */
+	mod_timer(&ccdev->timer, jiffies + COMMS_TIMER);
+}
+
+static void comms_config(struct comms_device *ccdev)
+{
+	/* Disable interrupts */
+	regmap_write(ccdev->regmap, COMMS_REG_IE_OFF, COMMS_IRQ_DISABLE_ALL);
+	/* Clear request and response FIFOs */
+	regmap_write(ccdev->regmap, COMMS_REG_CTRL_OFF, COMMS_CLEAR_FIFO);
+	/* Clear interrupts */
+	regmap_write(ccdev->regmap, COMMS_REG_IS_OFF, COMMS_IRQ_CLEAR_ALL);
+	/* Setup RIT reg */
+	regmap_write(ccdev->regmap, COMMS_REG_RIT_OFF, COMMS_RECEIVE_THRESHOLD);
+	/* Enable RIT interrupt */
+	regmap_write(ccdev->regmap, COMMS_REG_IE_OFF, COMMS_IRQ_RECEIVE_ENABLE);
+
+	/* Create and schedule timer to do recurring work */
+	INIT_WORK(&ccdev->work, &comms_check_request);
+	timer_setup(&ccdev->timer, &comms_sched_work, 0);
+	mod_timer(&ccdev->timer, jiffies + COMMS_TIMER);
+}
+
+void vmgmtm_comms_fini(struct comms_device *ccdev)
+{
+	/* First stop scheduling new work then cancel work */
+	del_timer_sync(&ccdev->timer);
+	cancel_work_sync(&ccdev->work);
+}
+
+struct comms_device *vmgmtm_comms_init(struct vmgmt_device *vdev)
+{
+	struct comms_device *ccdev;
+
+	ccdev = devm_kzalloc(&vdev->pdev->dev, sizeof(*ccdev), GFP_KERNEL);
+	if (!ccdev)
+		return ERR_PTR(-ENOMEM);
+
+	ccdev->vdev = vdev;
+
+	ccdev->regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
+					      vdev->tbl + COMMS_PCI_BAR_OFF,
+					      &comms_regmap_config);
+	if (IS_ERR(ccdev->regmap)) {
+		vmgmt_err(vdev, "Comms regmap init failed");
+		return ERR_CAST(ccdev->regmap);
+	}
+
+	comms_config(ccdev);
+	return ccdev;
+}
diff --git a/drivers/fpga/amd/vmgmt-comms.h b/drivers/fpga/amd/vmgmt-comms.h
new file mode 100644
index 000000000000..0afb14c8bd32
--- /dev/null
+++ b/drivers/fpga/amd/vmgmt-comms.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Driver for Versal PCIe device
+ *
+ * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#ifndef __VMGMT_COMMS_H
+#define __VMGMT_COMMS_H
+
+struct comms_device *vmgmtm_comms_init(struct vmgmt_device *vdev);
+void vmgmtm_comms_fini(struct comms_device *ccdev);
+
+#endif	/* __VMGMT_COMMS_H */
diff --git a/drivers/fpga/amd/vmgmt.c b/drivers/fpga/amd/vmgmt.c
new file mode 100644
index 000000000000..b72eff9e8bc0
--- /dev/null
+++ b/drivers/fpga/amd/vmgmt.c
@@ -0,0 +1,395 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for Versal PCIe device
+ *
+ * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#include <linux/cdev.h>
+#include <linux/device/class.h>
+#include <linux/err.h>
+#include <linux/firmware.h>
+#include <linux/fs.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/idr.h>
+#include <linux/kdev_t.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/types.h>
+#include <linux/uuid.h>
+#include <linux/vmgmt.h>
+
+#include "vmgmt.h"
+#include "vmgmt-comms.h"
+
+#define DRV_NAME			"amd-vmgmt"
+#define CLASS_NAME			DRV_NAME
+
+#define PCI_DEVICE_ID_V70PQ2		0x50B0
+#define VERSAL_XCLBIN_MAGIC_ID		"xclbin2"
+
+static DEFINE_IDA(vmgmt_dev_minor_ida);
+static dev_t vmgmt_devnode;
+struct class *vmgmt_class;
+static struct fpga_bridge_ops vmgmt_br_ops;
+
+struct vmgmt_fpga_region {
+	struct fpga_device *fdev;
+	uuid_t *uuid;
+};
+
+static inline struct vmgmt_device *vmgmt_inode_to_vdev(struct inode *inode)
+{
+	return (struct vmgmt_device *)container_of(inode->i_cdev, struct vmgmt_device, cdev);
+}
+
+static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)
+{
+	struct fpga_device *fdev = mgr->priv;
+
+	return fdev->state;
+}
+
+static const struct fpga_manager_ops vmgmt_fpga_ops = {
+	.state = vmgmt_fpga_state,
+};
+
+static int vmgmt_get_bridges(struct fpga_region *region)
+{
+	struct fpga_device *fdev = region->priv;
+
+	return fpga_bridge_get_to_list(&fdev->vdev->pdev->dev, region->info,
+				       &region->bridge_list);
+}
+
+static void vmgmt_fpga_fini(struct fpga_device *fdev)
+{
+	fpga_region_unregister(fdev->region);
+	fpga_bridge_unregister(fdev->bridge);
+	fpga_mgr_unregister(fdev->mgr);
+}
+
+static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
+{
+	struct device *dev = &vdev->pdev->dev;
+	struct fpga_region_info region = { 0 };
+	struct fpga_manager_info info = { 0 };
+	struct fpga_device *fdev;
+	int ret;
+
+	fdev = devm_kzalloc(dev, sizeof(*fdev), GFP_KERNEL);
+	if (!fdev)
+		return ERR_PTR(-ENOMEM);
+
+	fdev->vdev = vdev;
+
+	info = (struct fpga_manager_info) {
+		.name = "AMD Versal FPGA Manager",
+		.mops = &vmgmt_fpga_ops,
+		.priv = fdev,
+	};
+
+	fdev->mgr = fpga_mgr_register_full(dev, &info);
+	if (IS_ERR(fdev->mgr)) {
+		ret = PTR_ERR(fdev->mgr);
+		vmgmt_err(vdev, "Failed to register FPGA manager, err %d", ret);
+		return ERR_PTR(ret);
+	}
+
+	/* create fgpa bridge, region for the base shell */
+	fdev->bridge = fpga_bridge_register(dev, "AMD Versal FPGA Bridge",
+					    &vmgmt_br_ops, fdev);
+	if (IS_ERR(fdev->bridge)) {
+		vmgmt_err(vdev, "Failed to register FPGA bridge, err %ld",
+			  PTR_ERR(fdev->bridge));
+		ret = PTR_ERR(fdev->bridge);
+		goto unregister_fpga_mgr;
+	}
+
+	region = (struct fpga_region_info) {
+		.compat_id = (struct fpga_compat_id *)&vdev->intf_uuid,
+		.get_bridges = vmgmt_get_bridges,
+		.mgr = fdev->mgr,
+		.priv = fdev,
+	};
+
+	fdev->region = fpga_region_register_full(dev, &region);
+	if (IS_ERR(fdev->region)) {
+		vmgmt_err(vdev, "Failed to register FPGA region, err %ld",
+			  PTR_ERR(fdev->region));
+		ret = PTR_ERR(fdev->region);
+		goto unregister_fpga_bridge;
+	}
+
+	return fdev;
+
+unregister_fpga_bridge:
+	fpga_bridge_unregister(fdev->bridge);
+
+unregister_fpga_mgr:
+	fpga_mgr_unregister(fdev->mgr);
+
+	return ERR_PTR(ret);
+}
+
+static int vmgmt_open(struct inode *inode, struct file *filep)
+{
+	struct vmgmt_device *vdev = vmgmt_inode_to_vdev(inode);
+
+	if (WARN_ON(!vdev))
+		return -ENODEV;
+
+	filep->private_data = vdev;
+
+	return 0;
+}
+
+static int vmgmt_release(struct inode *inode, struct file *filep)
+{
+	filep->private_data = NULL;
+
+	return 0;
+}
+
+static const struct file_operations vmgmt_fops = {
+	.owner = THIS_MODULE,
+	.open = vmgmt_open,
+	.release = vmgmt_release,
+};
+
+static void vmgmt_chrdev_destroy(struct vmgmt_device *vdev)
+{
+	device_destroy(vmgmt_class, vdev->cdev.dev);
+	cdev_del(&vdev->cdev);
+	ida_free(&vmgmt_dev_minor_ida, vdev->minor);
+}
+
+static int vmgmt_chrdev_create(struct vmgmt_device *vdev)
+{
+	u32 devid;
+	int ret;
+
+	vdev->minor = ida_alloc(&vmgmt_dev_minor_ida, GFP_KERNEL);
+	if (vdev->minor < 0) {
+		vmgmt_err(vdev, "Failed to allocate chrdev ID");
+		return -ENODEV;
+	}
+
+	cdev_init(&vdev->cdev, &vmgmt_fops);
+
+	vdev->cdev.owner = THIS_MODULE;
+	vdev->cdev.dev = MKDEV(MAJOR(vmgmt_devnode), vdev->minor);
+	ret = cdev_add(&vdev->cdev, vdev->cdev.dev, 1);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to add char device: %d\n", ret);
+		ida_free(&vmgmt_dev_minor_ida, vdev->minor);
+		return -ENODEV;
+	}
+
+	devid = PCI_DEVID(vdev->pdev->bus->number, vdev->pdev->devfn);
+	vdev->device = device_create(vmgmt_class, &vdev->pdev->dev,
+				     vdev->cdev.dev, NULL, "%s%x", DRV_NAME,
+				     devid);
+	if (IS_ERR(vdev->device)) {
+		vmgmt_err(vdev, "Failed to create device: %ld\n",
+			  PTR_ERR(vdev->device));
+		cdev_del(&vdev->cdev);
+		ida_free(&vmgmt_dev_minor_ida, vdev->minor);
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
+{
+	struct firmware_device *fwdev = fw_upload->dd_handle;
+
+	vmgmt_warn(fwdev->vdev, "canceled");
+}
+
+static const struct fw_upload_ops vmgmt_fw_ops = {
+	.cancel = vmgmt_fw_cancel,
+};
+
+static void vmgmt_fw_upload_fini(struct firmware_device *fwdev)
+{
+	firmware_upload_unregister(fwdev->fw);
+	kfree(fwdev->name);
+}
+
+static struct firmware_device *vmgmt_fw_upload_init(struct vmgmt_device *vdev)
+{
+	struct device *dev = &vdev->pdev->dev;
+	struct firmware_device *fwdev;
+	u32 devid;
+
+	fwdev = devm_kzalloc(dev, sizeof(*fwdev), GFP_KERNEL);
+	if (!fwdev)
+		return ERR_PTR(-ENOMEM);
+
+	devid = PCI_DEVID(vdev->pdev->bus->number, vdev->pdev->devfn);
+	fwdev->name = kasprintf(GFP_KERNEL, "%s%x", DRV_NAME, devid);
+	if (!fwdev->name)
+		return ERR_PTR(-ENOMEM);
+
+	fwdev->fw = firmware_upload_register(THIS_MODULE, dev, fwdev->name,
+					     &vmgmt_fw_ops, fwdev);
+	if (IS_ERR(fwdev->fw)) {
+		kfree(fwdev->name);
+		return ERR_CAST(fwdev->fw);
+	}
+
+	fwdev->vdev = vdev;
+
+	return fwdev;
+}
+
+static void vmgmt_device_teardown(struct vmgmt_device *vdev)
+{
+	vmgmt_fpga_fini(vdev->fdev);
+	vmgmt_fw_upload_fini(vdev->fwdev);
+	vmgmtm_comms_fini(vdev->ccdev);
+}
+
+static int vmgmt_device_setup(struct vmgmt_device *vdev)
+{
+	int ret;
+
+	vdev->fwdev = vmgmt_fw_upload_init(vdev);
+	if (IS_ERR(vdev->fwdev)) {
+		ret = PTR_ERR(vdev->fwdev);
+		vmgmt_err(vdev, "Failed to init FW uploader, err %d", ret);
+		goto done;
+	}
+
+	vdev->ccdev = vmgmtm_comms_init(vdev);
+	if (IS_ERR(vdev->ccdev)) {
+		ret = PTR_ERR(vdev->ccdev);
+		vmgmt_err(vdev, "Failed to init comms channel, err %d", ret);
+		goto upload_fini;
+	}
+
+	vdev->fdev = vmgmt_fpga_init(vdev);
+	if (IS_ERR(vdev->fdev)) {
+		ret = PTR_ERR(vdev->fdev);
+		vmgmt_err(vdev, "Failed to init FPGA maanger, err %d", ret);
+		goto comms_fini;
+	}
+
+	return 0;
+comms_fini:
+	vmgmtm_comms_fini(vdev->ccdev);
+upload_fini:
+	vmgmt_fw_upload_fini(vdev->fwdev);
+done:
+	return ret;
+}
+
+static void vmgmt_remove(struct pci_dev *pdev)
+{
+	struct vmgmt_device *vdev = pci_get_drvdata(pdev);
+
+	vmgmt_chrdev_destroy(vdev);
+	vmgmt_device_teardown(vdev);
+}
+
+static int vmgmt_probe(struct pci_dev *pdev,
+		       const struct pci_device_id *pdev_id)
+{
+	struct vmgmt_device *vdev;
+	int ret;
+
+	vdev = devm_kzalloc(&pdev->dev, sizeof(*vdev), GFP_KERNEL);
+	if (!vdev)
+		return -ENOMEM;
+
+	pci_set_drvdata(pdev, vdev);
+	vdev->pdev = pdev;
+
+	ret = pcim_enable_device(pdev);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to enable device %d", ret);
+		return ret;
+	}
+
+	ret = pcim_iomap_regions(vdev->pdev, AMD_VMGMT_BAR_MASK, "amd-vmgmt");
+	if (ret) {
+		vmgmt_err(vdev, "Failed iomap regions %d", ret);
+		return -ENOMEM;
+	}
+
+	vdev->tbl = pcim_iomap_table(vdev->pdev)[AMD_VMGMT_BAR];
+	if (IS_ERR(vdev->tbl)) {
+		vmgmt_err(vdev, "Failed to map RM shared memory BAR%d", AMD_VMGMT_BAR);
+		return -ENOMEM;
+	}
+
+	ret = vmgmt_device_setup(vdev);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to setup Versal device %d", ret);
+		return ret;
+	}
+
+	ret = vmgmt_chrdev_create(vdev);
+	if (ret) {
+		vmgmt_device_teardown(vdev);
+		return ret;
+	}
+
+	vmgmt_dbg(vdev, "Successfully probed %s driver!", DRV_NAME);
+	return 0;
+}
+
+static const struct pci_device_id vmgmt_pci_ids[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_XILINX, PCI_DEVICE_ID_V70PQ2), },
+	{ 0 }
+};
+
+MODULE_DEVICE_TABLE(pci, vmgmt_pci_ids);
+
+static struct pci_driver amd_vmgmt_driver = {
+	.name = DRV_NAME,
+	.id_table = vmgmt_pci_ids,
+	.probe = vmgmt_probe,
+	.remove = vmgmt_remove,
+};
+
+static int amd_vmgmt_init(void)
+{
+	int ret;
+
+	vmgmt_class = class_create(CLASS_NAME);
+	if (IS_ERR(vmgmt_class))
+		return PTR_ERR(vmgmt_class);
+
+	ret = alloc_chrdev_region(&vmgmt_devnode, 0, MINORMASK, DRV_NAME);
+	if (ret)
+		goto chr_err;
+
+	ret = pci_register_driver(&amd_vmgmt_driver);
+	if (ret)
+		goto pci_err;
+
+	return 0;
+
+pci_err:
+	unregister_chrdev_region(vmgmt_devnode, MINORMASK);
+chr_err:
+	class_destroy(vmgmt_class);
+	return ret;
+}
+
+static void amd_vmgmt_exit(void)
+{
+	pci_unregister_driver(&amd_vmgmt_driver);
+	unregister_chrdev_region(vmgmt_devnode, MINORMASK);
+	class_destroy(vmgmt_class);
+}
+
+module_init(amd_vmgmt_init);
+module_exit(amd_vmgmt_exit);
+
+MODULE_DESCRIPTION("AMD PCIe Versal Management Driver");
+MODULE_AUTHOR("XRT Team <runtimeca39d@amd.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/fpga/amd/vmgmt.h b/drivers/fpga/amd/vmgmt.h
new file mode 100644
index 000000000000..4dc8a43f825e
--- /dev/null
+++ b/drivers/fpga/amd/vmgmt.h
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Driver for Versal PCIe device
+ *
+ * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#ifndef __VMGMT_H
+#define __VMGMT_H
+
+#include <linux/cdev.h>
+#include <linux/dev_printk.h>
+#include <linux/jiffies.h>
+#include <linux/list.h>
+#include <linux/firmware.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/pci.h>
+#include <linux/uuid.h>
+#include <linux/regmap.h>
+
+#define AMD_VMGMT_BAR			0
+#define AMD_VMGMT_BAR_MASK		BIT(0)
+
+#define vmgmt_info(vdev, fmt, args...)					\
+	dev_info(&(vdev)->pdev->dev, "%s: "fmt, __func__, ##args)
+
+#define vmgmt_warn(vdev, fmt, args...)					\
+	dev_warn(&(vdev)->pdev->dev, "%s: "fmt, __func__, ##args)
+
+#define vmgmt_err(vdev, fmt, args...)					\
+	dev_err(&(vdev)->pdev->dev, "%s: "fmt, __func__, ##args)
+
+#define vmgmt_dbg(vdev, fmt, args...)					\
+	dev_dbg(&(vdev)->pdev->dev, fmt, ##args)
+
+struct vmgmt_device;
+struct comms_device;
+struct rm_cmd;
+
+struct axlf_header {
+	u64				length;
+	unsigned char			reserved1[24];
+	uuid_t				rom_uuid;
+	unsigned char			reserved2[64];
+	uuid_t				uuid;
+	unsigned char			reserved3[24];
+} __packed;
+
+struct axlf {
+	char				magic[8];
+	unsigned char			reserved[296];
+	struct axlf_header		header;
+} __packed;
+
+struct fw_tnx {
+	struct rm_cmd		*cmd;
+	int			opcode;
+	int			id;
+};
+
+struct fpga_device {
+	enum fpga_mgr_states	state;
+	struct fpga_manager	*mgr;
+	struct fpga_bridge	*bridge;
+	struct fpga_region	*region;
+	struct vmgmt_device	*vdev;
+	struct fw_tnx		fw;
+};
+
+struct firmware_device {
+	struct vmgmt_device	*vdev;
+	struct fw_upload	*fw;
+	char			*name;
+	u32			fw_name_id;
+	struct rm_cmd		*cmd;
+	int			id;
+	uuid_t			uuid;
+};
+
+struct vmgmt_device {
+	struct pci_dev		*pdev;
+
+	struct rm_device	*rdev;
+	struct comms_device	*ccdev;
+	struct fpga_device	*fdev;
+	struct firmware_device	*fwdev;
+	struct cdev		cdev;
+	struct device		*device;
+
+	int                     minor;
+	void __iomem		*tbl;
+	uuid_t			xclbin_uuid;
+	uuid_t			intf_uuid;
+
+	void                    *debugfs_root;
+};
+
+#endif	/* __VMGMT_H */
diff --git a/include/uapi/linux/vmgmt.h b/include/uapi/linux/vmgmt.h
new file mode 100644
index 000000000000..2269ceb5c131
--- /dev/null
+++ b/include/uapi/linux/vmgmt.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Header file for Versal PCIe device user API
+ *
+ * Copyright (C) 2024 AMD Corporation, Inc.
+ */
+
+#ifndef _UAPI_LINUX_VMGMT_H
+#define _UAPI_LINUX_VMGMT_H
+
+#include <linux/ioctl.h>
+
+#define VERSAL_MGMT_MAGIC	0xB7
+#define VERSAL_MGMT_BASE	0
+
+/**
+ * VERSAL_MGMT_LOAD_XCLBIN_IOCTL - Download XCLBIN to the device
+ *
+ * This IOCTL is used to download XCLBIN down to the device.
+ * Return: 0 on success, -errno on failure.
+ */
+#define VERSAL_MGMT_LOAD_XCLBIN_IOCTL	_IOW(VERSAL_MGMT_MAGIC,		\
+					     VERSAL_MGMT_BASE + 0, void *)
+
+#endif /* _UAPI_LINUX_VMGMT_H */
-- 
2.34.1

[PATCH V1 2/3] drivers/fpga/amd: Add communication with firmware

[David Zhang]

From: Yidong Zhang <yidong.zhang@amd.com>

Add queue based communication between host driver and firmware on the
card. The remote queue (rm) can send/receive messages and providing
firmware downloading services.

Co-developed-by: Nishad Saraf <nishads@amd.com>
Signed-off-by: Nishad Saraf <nishads@amd.com>
Co-developed-by: Prapul Krishnamurthy <prapulk@amd.com>
Signed-off-by: Prapul Krishnamurthy <prapulk@amd.com>
Signed-off-by: Yidong Zhang <yidong.zhang@amd.com>
---
 drivers/fpga/amd/Makefile         |   6 +-
 drivers/fpga/amd/vmgmt-rm-queue.c |  38 +++
 drivers/fpga/amd/vmgmt-rm-queue.h |  15 +
 drivers/fpga/amd/vmgmt-rm.c       | 543 ++++++++++++++++++++++++++++++
 drivers/fpga/amd/vmgmt-rm.h       | 222 ++++++++++++
 drivers/fpga/amd/vmgmt.c          | 305 ++++++++++++++++-
 drivers/fpga/amd/vmgmt.h          |   7 +-
 7 files changed, 1130 insertions(+), 6 deletions(-)
 create mode 100644 drivers/fpga/amd/vmgmt-rm-queue.c
 create mode 100644 drivers/fpga/amd/vmgmt-rm-queue.h
 create mode 100644 drivers/fpga/amd/vmgmt-rm.c
 create mode 100644 drivers/fpga/amd/vmgmt-rm.h

diff --git a/drivers/fpga/amd/Makefile b/drivers/fpga/amd/Makefile
index 3e4c6dd3b787..97cfff6be204 100644
--- a/drivers/fpga/amd/Makefile
+++ b/drivers/fpga/amd/Makefile
@@ -2,5 +2,7 @@
 
 obj-$(CONFIG_AMD_VERSAL_MGMT)			+= amd-vmgmt.o
 
-amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)		:= vmgmt.o	\
-						   vmgmt-comms.o
+amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)		:= vmgmt.o		\
+						   vmgmt-comms.o	\
+						   vmgmt-rm.o		\
+						   vmgmt-rm-queue.o
diff --git a/drivers/fpga/amd/vmgmt-rm-queue.c b/drivers/fpga/amd/vmgmt-rm-queue.c
new file mode 100644
index 000000000000..fe805373ea32
--- /dev/null
+++ b/drivers/fpga/amd/vmgmt-rm-queue.c
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for Versal PCIe device
+ *
+ * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/completion.h>
+#include <linux/err.h>
+#include <linux/firmware.h>
+#include <linux/idr.h>
+#include <linux/jiffies.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/semaphore.h>
+#include <linux/timer.h>
+#include <linux/uuid.h>
+#include <linux/workqueue.h>
+
+#include "vmgmt.h"
+#include "vmgmt-rm.h"
+#include "vmgmt-rm-queue.h"
+
+int rm_queue_send_cmd(struct rm_cmd *cmd, unsigned long timeout)
+{
+	return 0;
+}
+
+void rm_queue_fini(struct rm_device *rdev)
+{
+}
+
+int rm_queue_init(struct rm_device *rdev)
+{
+	return 0;
+}
diff --git a/drivers/fpga/amd/vmgmt-rm-queue.h b/drivers/fpga/amd/vmgmt-rm-queue.h
new file mode 100644
index 000000000000..6fd0e0026a13
--- /dev/null
+++ b/drivers/fpga/amd/vmgmt-rm-queue.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Driver for Versal PCIe device
+ *
+ * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#ifndef __VMGMT_RM_QUEUE_H
+#define __VMGMT_RM_QUEUE_H
+
+int rm_queue_init(struct rm_device *rdev);
+void rm_queue_fini(struct rm_device *rdev);
+int rm_queue_send_cmd(struct rm_cmd *cmd, unsigned long timeout);
+
+#endif	/* __VMGMT_RM_QUEUE_H */
diff --git a/drivers/fpga/amd/vmgmt-rm.c b/drivers/fpga/amd/vmgmt-rm.c
new file mode 100644
index 000000000000..856d5af52c8d
--- /dev/null
+++ b/drivers/fpga/amd/vmgmt-rm.c
@@ -0,0 +1,543 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for Versal PCIe device
+ *
+ * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/completion.h>
+#include <linux/err.h>
+#include <linux/firmware.h>
+#include <linux/idr.h>
+#include <linux/jiffies.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/semaphore.h>
+#include <linux/timer.h>
+#include <linux/uuid.h>
+#include <linux/workqueue.h>
+
+#include "vmgmt.h"
+#include "vmgmt-rm.h"
+#include "vmgmt-rm-queue.h"
+
+static DEFINE_IDA(rm_cmd_ids);
+
+static const struct regmap_config rm_shmem_regmap_config = {
+	.name = "rm_shmem_config",
+	.reg_bits = 32,
+	.reg_stride = 4,
+	.val_bits = 32,
+};
+
+static const struct regmap_config rm_io_regmap_config = {
+	.name = "rm_io_config",
+	.reg_bits = 32,
+	.reg_stride = 4,
+	.val_bits = 32,
+};
+
+static void rm_uninstall_health_monitor(struct rm_device *rdev);
+
+static inline struct rm_device *to_rdev_health_monitor(struct work_struct *w)
+{
+	return container_of(w, struct rm_device, health_monitor);
+}
+
+static inline struct rm_device *to_rdev_health_timer(struct timer_list *t)
+{
+	return container_of(t, struct rm_device, health_timer);
+}
+
+static inline int rm_shmem_read(struct rm_device *rdev, u32 offset, u32 *value)
+{
+	return regmap_read(rdev->shmem_regmap, offset, value);
+}
+
+static inline int rm_shmem_bulk_read(struct rm_device *rdev, u32 offset,
+				     u32 *value, u32 size)
+{
+	return regmap_bulk_read(rdev->shmem_regmap, offset, value,
+				DIV_ROUND_UP(size, 4));
+}
+
+static inline int rm_shmem_bulk_write(struct rm_device *rdev, u32 offset,
+				      u32 *value, u32 size)
+{
+	return regmap_bulk_write(rdev->shmem_regmap, offset, value,
+				DIV_ROUND_UP(size, 4));
+}
+
+void rm_queue_destory_cmd(struct rm_cmd *cmd)
+{
+	ida_free(&rm_cmd_ids, cmd->sq_msg.hdr.id);
+	kfree(cmd);
+}
+
+int rm_queue_copy_response(struct rm_cmd *cmd, void *buffer, ssize_t len)
+{
+	struct rm_cmd_cq_log_page *result = &cmd->cq_msg.data.page;
+	u64 off = cmd->sq_msg.data.page.address;
+
+	if (!result->len || len < result->len) {
+		vmgmt_err(cmd->rdev->vdev, "Invalid response or buffer size");
+		return -EINVAL;
+	}
+
+	return rm_shmem_bulk_read(cmd->rdev, off, (u32 *)buffer, result->len);
+}
+
+static void rm_queue_payload_fini(struct rm_cmd *cmd)
+{
+	up(&cmd->rdev->cq.data_lock);
+}
+
+static int rm_queue_payload_init(struct rm_cmd *cmd,
+				 enum rm_cmd_log_page_type type)
+{
+	struct rm_device *rdev = cmd->rdev;
+	int ret;
+
+	ret = down_interruptible(&rdev->cq.data_lock);
+	if (ret)
+		return ret;
+
+	cmd->sq_msg.data.page.address = rdev->cq.data_offset;
+	cmd->sq_msg.data.page.size = rdev->cq.data_size;
+	cmd->sq_msg.data.page.reserved1 = 0;
+	cmd->sq_msg.data.page.type = FIELD_PREP(RM_CMD_LOG_PAGE_TYPE_MASK,
+						type);
+	return 0;
+}
+
+void rm_queue_data_fini(struct rm_cmd *cmd)
+{
+	up(&cmd->rdev->sq.data_lock);
+}
+
+int rm_queue_data_init(struct rm_cmd *cmd, const char *buffer, ssize_t size)
+{
+	struct rm_device *rdev = cmd->rdev;
+	int ret;
+
+	if (!size || size > rdev->sq.data_size) {
+		vmgmt_err(rdev->vdev, "Unsupported file size");
+		return -ENOMEM;
+	}
+
+	ret = down_interruptible(&rdev->sq.data_lock);
+	if (ret)
+		return ret;
+
+	ret = rm_shmem_bulk_write(cmd->rdev, rdev->sq.data_offset,
+				  (u32 *)buffer, size);
+	if (ret) {
+		vmgmt_err(rdev->vdev, "Failed to copy binary to SQ buffer");
+		up(&cmd->rdev->sq.data_lock);
+		return ret;
+	}
+
+	cmd->sq_msg.data.bin.address = rdev->sq.data_offset;
+	cmd->sq_msg.data.bin.size = size;
+	return 0;
+}
+
+int rm_queue_create_cmd(struct rm_device *rdev, enum rm_queue_opcode opcode,
+			struct rm_cmd **cmd_ptr)
+{
+	struct rm_cmd *cmd = NULL;
+	int ret, id;
+	u16 size;
+
+	if (rdev->firewall_tripped)
+		return -ENODEV;
+
+	cmd = kzalloc(sizeof(*cmd), GFP_KERNEL);
+	if (!cmd)
+		return -ENOMEM;
+	cmd->rdev = rdev;
+
+	switch (opcode) {
+	case RM_QUEUE_OP_LOAD_XCLBIN:
+		fallthrough;
+	case RM_QUEUE_OP_LOAD_FW:
+		fallthrough;
+	case RM_QUEUE_OP_LOAD_APU_FW:
+		size = sizeof(struct rm_cmd_sq_bin);
+		break;
+	case RM_QUEUE_OP_GET_LOG_PAGE:
+		size = sizeof(struct rm_cmd_sq_log_page);
+		break;
+	case RM_QUEUE_OP_IDENTIFY:
+		size = 0;
+		break;
+	case RM_QUEUE_OP_VMR_CONTROL:
+		size = sizeof(struct rm_cmd_sq_ctrl);
+		break;
+	default:
+		vmgmt_err(rdev->vdev, "Invalid cmd opcode %d", opcode);
+		ret = -EINVAL;
+		goto error;
+	};
+
+	cmd->opcode = opcode;
+	cmd->sq_msg.hdr.opcode = FIELD_PREP(RM_CMD_SQ_HDR_OPS_MSK, opcode);
+	cmd->sq_msg.hdr.msg_size = FIELD_PREP(RM_CMD_SQ_HDR_SIZE_MSK, size);
+
+	id = ida_alloc_range(&rm_cmd_ids, RM_CMD_ID_MIN, RM_CMD_ID_MAX, GFP_KERNEL);
+	if (id < 0) {
+		vmgmt_err(rdev->vdev, "Failed to alloc cmd ID: %d", id);
+		ret = id;
+		goto error;
+	}
+	cmd->sq_msg.hdr.id = id;
+
+	init_completion(&cmd->executed);
+
+	*cmd_ptr = cmd;
+	return 0;
+error:
+	kfree(cmd);
+	return ret;
+}
+
+static int rm_queue_verify(struct rm_device *rdev)
+{
+	struct vmgmt_device *vdev = rdev->vdev;
+	struct rm_cmd_cq_identify *result;
+	struct rm_cmd *cmd;
+	u32 major, minor;
+	int ret;
+
+	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_IDENTIFY, &cmd);
+	if (ret)
+		return ret;
+
+	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
+	if (ret)
+		goto error;
+
+	result = &cmd->cq_msg.data.identify;
+	major = result->major;
+	minor = result->minor;
+	vmgmt_dbg(vdev, "VMR version %d.%d", major, minor);
+	if (!major) {
+		vmgmt_err(vdev, "VMR version is unsupported");
+		ret = -EOPNOTSUPP;
+	}
+
+error:
+	rm_queue_destory_cmd(cmd);
+	return ret;
+}
+
+static int rm_check_apu_status(struct rm_device *rdev, bool *status)
+{
+	struct rm_cmd_cq_control *result;
+	struct rm_cmd *cmd;
+	int ret;
+
+	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_VMR_CONTROL, &cmd);
+	if (ret)
+		return ret;
+
+	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
+	if (ret)
+		goto error;
+
+	result = &cmd->cq_msg.data.ctrl;
+	*status = FIELD_GET(RM_CMD_VMR_CONTROL_PS_MASK, result->status);
+
+	rm_queue_destory_cmd(cmd);
+	return 0;
+
+error:
+	rm_queue_destory_cmd(cmd);
+	return ret;
+}
+
+static int rm_download_apu_fw(struct rm_device *rdev, char *data, ssize_t size)
+{
+	struct rm_cmd *cmd;
+	int ret;
+
+	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_LOAD_APU_FW, &cmd);
+	if (ret)
+		return ret;
+
+	ret = rm_queue_data_init(cmd, data, size);
+	if (ret)
+		goto done;
+
+	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
+
+done:
+	rm_queue_destory_cmd(cmd);
+	return ret;
+}
+
+int rm_boot_apu(struct rm_device *rdev)
+{
+	char *bin = "xilinx/xrt-versal-apu.xsabin";
+	const struct firmware *fw = NULL;
+	bool status;
+	int ret;
+
+	ret = rm_check_apu_status(rdev, &status);
+	if (ret) {
+		vmgmt_err(rdev->vdev, "Failed to get APU status");
+		return ret;
+	}
+
+	if (status) {
+		vmgmt_dbg(rdev->vdev, "APU online. Skipping APU FW download");
+		return 0;
+	}
+
+	ret = request_firmware(&fw, bin, &rdev->vdev->pdev->dev);
+	if (ret) {
+		vmgmt_warn(rdev->vdev, "Request APU FW %s failed %d", bin, ret);
+		return ret;
+	}
+
+	vmgmt_dbg(rdev->vdev, "Starting... APU FW download");
+	ret = rm_download_apu_fw(rdev, (char *)fw->data, fw->size);
+	vmgmt_dbg(rdev->vdev, "Finished... APU FW download %d", ret);
+
+	if (ret)
+		vmgmt_err(rdev->vdev, "Failed to download APU FW, ret:%d", ret);
+
+	release_firmware(fw);
+
+	return ret;
+}
+
+static void rm_check_health(struct work_struct *w)
+{
+	struct rm_device *rdev = to_rdev_health_monitor(w);
+	ssize_t len = PAGE_SIZE;
+	char *buffer = NULL;
+	struct rm_cmd *cmd;
+	int ret;
+
+	buffer = vzalloc(len);
+	if (!buffer)
+		return;
+
+	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_GET_LOG_PAGE, &cmd);
+	if (ret)
+		return;
+
+	ret = rm_queue_payload_init(cmd, RM_CMD_LOG_PAGE_AXI_TRIP_STATUS);
+	if (ret)
+		goto error;
+
+	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
+	if (ret == -ETIME || ret == -EINVAL)
+		goto payload_fini;
+
+	if (cmd->cq_msg.data.page.len) {
+		ret = rm_queue_copy_response(cmd, buffer, len);
+		if (ret)
+			goto payload_fini;
+
+		vmgmt_err(rdev->vdev, "%s", buffer);
+		rdev->firewall_tripped = 1;
+	}
+
+	vfree(buffer);
+
+	rm_queue_payload_fini(cmd);
+	rm_queue_destory_cmd(cmd);
+
+	return;
+
+payload_fini:
+	rm_queue_payload_fini(cmd);
+error:
+	rm_queue_destory_cmd(cmd);
+	vfree(buffer);
+}
+
+static void rm_sched_health_check(struct timer_list *t)
+{
+	struct rm_device *rdev = to_rdev_health_timer(t);
+
+	if (rdev->firewall_tripped) {
+		vmgmt_err(rdev->vdev, "Firewall tripped, health check paused. Please reset card");
+		return;
+	}
+	/* Schedule a work in the general workqueue */
+	schedule_work(&rdev->health_monitor);
+	/* Periodic timer */
+	mod_timer(&rdev->health_timer, jiffies + RM_HEALTH_CHECK_TIMER);
+}
+
+static void rm_uninstall_health_monitor(struct rm_device *rdev)
+{
+	del_timer_sync(&rdev->health_timer);
+	cancel_work_sync(&rdev->health_monitor);
+}
+
+static void rm_install_health_monitor(struct rm_device *rdev)
+{
+	INIT_WORK(&rdev->health_monitor, &rm_check_health);
+	timer_setup(&rdev->health_timer, &rm_sched_health_check, 0);
+	mod_timer(&rdev->health_timer, jiffies + RM_HEALTH_CHECK_TIMER);
+}
+
+void vmgmt_rm_fini(struct rm_device *rdev)
+{
+	rm_uninstall_health_monitor(rdev);
+	rm_queue_fini(rdev);
+}
+
+struct rm_device *vmgmt_rm_init(struct vmgmt_device *vdev)
+{
+	struct rm_header *header;
+	struct rm_device *rdev;
+	u32 status;
+	int ret;
+
+	rdev = devm_kzalloc(&vdev->pdev->dev, sizeof(*rdev), GFP_KERNEL);
+	if (!rdev)
+		return ERR_PTR(-ENOMEM);
+
+	rdev->vdev = vdev;
+	header = &rdev->rm_metadata;
+
+	rdev->shmem_regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
+						   vdev->tbl + RM_PCI_SHMEM_BAR_OFF,
+						   &rm_shmem_regmap_config);
+	if (IS_ERR(rdev->shmem_regmap)) {
+		vmgmt_err(vdev, "Failed to init RM shared memory regmap");
+		return ERR_CAST(rdev->shmem_regmap);
+	}
+
+	ret = rm_shmem_bulk_read(rdev, RM_HDR_OFF, (u32 *)header,
+				 sizeof(*header));
+	if (ret) {
+		vmgmt_err(vdev, "Failed to read RM shared mem, ret %d", ret);
+		ret = -ENODEV;
+		goto err;
+	}
+
+	if (header->magic != RM_HDR_MAGIC_NUM) {
+		vmgmt_err(vdev, "Invalid RM header 0x%x", header->magic);
+		ret = -ENODEV;
+		goto err;
+	}
+
+	ret = rm_shmem_read(rdev, header->status_off, &status);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to read RM shared mem, ret %d", ret);
+		ret = -ENODEV;
+		goto err;
+	}
+
+	if (!status) {
+		vmgmt_err(vdev, "RM status %d is not ready", status);
+		ret = -ENODEV;
+		goto err;
+	}
+
+	rdev->queue_buffer_size = header->data_end - header->data_start + 1;
+	rdev->queue_buffer_start = header->data_start;
+	rdev->queue_base = header->queue_base;
+
+	rdev->io_regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
+						vdev->tbl + RM_PCI_IO_BAR_OFF,
+						&rm_io_regmap_config);
+	if (IS_ERR(rdev->io_regmap)) {
+		vmgmt_err(vdev, "Failed to init RM IO regmap");
+		ret = PTR_ERR(rdev->io_regmap);
+		goto err;
+	}
+
+	ret = rm_queue_init(rdev);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to init cmd queue, ret %d", ret);
+		ret = -ENODEV;
+		goto err;
+	}
+
+	ret = rm_queue_verify(rdev);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to verify cmd queue, ret %d", ret);
+		ret = -ENODEV;
+		goto queue_fini;
+	}
+
+	ret = rm_boot_apu(rdev);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to bringup APU, ret %d", ret);
+		ret = -ENODEV;
+		goto queue_fini;
+	}
+
+	rm_install_health_monitor(rdev);
+
+	return rdev;
+queue_fini:
+	rm_queue_fini(rdev);
+err:
+	return ERR_PTR(ret);
+}
+
+int vmgmt_rm_get_fw_id(struct rm_device *rdev, uuid_t *uuid)
+{
+	char str[UUID_STRING_LEN];
+	ssize_t len = PAGE_SIZE;
+	char *buffer = NULL;
+	struct rm_cmd *cmd;
+	u8 i, j;
+	int ret;
+
+	buffer = vmalloc(len);
+	if (!buffer)
+		return -ENOMEM;
+
+	memset(buffer, 0, len);
+
+	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_GET_LOG_PAGE, &cmd);
+	if (ret)
+		return ret;
+
+	ret = rm_queue_payload_init(cmd, RM_CMD_LOG_PAGE_FW_ID);
+	if (ret)
+		goto error;
+
+	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
+	if (ret)
+		goto payload;
+
+	ret = rm_queue_copy_response(cmd, buffer, len);
+	if (ret)
+		goto payload;
+
+	/* parse uuid into a valid uuid string format */
+	for (i  = 0, j = 0; i < strlen(buffer); i++) {
+		str[j++] = buffer[i];
+		if (j == 8 || j == 13 || j == 18 || j == 23)
+			str[j++] = '-';
+	}
+
+	uuid_parse(str, uuid);
+	vmgmt_dbg(rdev->vdev, "Interface uuid %pU", uuid);
+
+	vfree(buffer);
+
+	rm_queue_payload_fini(cmd);
+	rm_queue_destory_cmd(cmd);
+
+	return 0;
+
+payload:
+	rm_queue_payload_fini(cmd);
+error:
+	rm_queue_destory_cmd(cmd);
+	vfree(buffer);
+	return ret;
+}
diff --git a/drivers/fpga/amd/vmgmt-rm.h b/drivers/fpga/amd/vmgmt-rm.h
new file mode 100644
index 000000000000..a74f93cefbe8
--- /dev/null
+++ b/drivers/fpga/amd/vmgmt-rm.h
@@ -0,0 +1,222 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Driver for Versal PCIe device
+ *
+ * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#ifndef __VMGMT_RM_H
+#define __VMGMT_RM_H
+
+#define RM_HDR_OFF			0x0
+#define RM_HDR_MAGIC_NUM		0x564D5230
+#define RM_QUEUE_HDR_MAGIC_NUM		0x5847513F
+#define RM_PCI_IO_BAR_OFF		0x2010000
+#define RM_PCI_IO_SIZE			0x1000
+#define RM_PCI_SHMEM_BAR_OFF		0x8000000
+#define RM_PCI_SHMEM_SIZE		0x8000000 /* 128 MB */
+#define RM_PCI_SHMEM_HDR_SIZE		0x28
+
+#define RM_QUEUE_HDR_MAGIC_NUM_OFF	0x0
+#define RM_IO_SQ_PIDX_OFF		0x0
+#define RM_IO_CQ_PIDX_OFF		0x100
+
+#define RM_CMD_ID_MIN			1
+#define RM_CMD_ID_MAX			(BIT(17) - 1)
+#define RM_CMD_SQ_HDR_OPS_MSK		GENMASK(15, 0)
+#define RM_CMD_SQ_HDR_SIZE_MSK		GENMASK(14, 0)
+#define RM_CMD_SQ_SLOT_SIZE		512
+#define RM_CMD_CQ_SLOT_SIZE		16
+#define RM_CMD_CQ_BUFFER_SIZE		(1024 * 1024)
+#define RM_CMD_CQ_BUFFER_OFFSET		0x0
+#define RM_CMD_LOG_PAGE_TYPE_MASK	GENMASK(15, 0)
+#define RM_CMD_VMR_CONTROL_MSK		GENMASK(10, 8)
+#define RM_CMD_VMR_CONTROL_PS_MASK	BIT(9)
+
+#define RM_CMD_WAIT_CONFIG_TIMEOUT	msecs_to_jiffies(10 * 1000)
+#define RM_CMD_WAIT_DOWNLOAD_TIMEOUT	msecs_to_jiffies(300 * 1000)
+
+#define RM_COMPLETION_TIMER		(HZ / 10)
+#define RM_HEALTH_CHECK_TIMER		(HZ)
+
+#define RM_INVALID_SLOT			0
+
+enum rm_queue_opcode {
+	RM_QUEUE_OP_LOAD_XCLBIN		= 0x0,
+	RM_QUEUE_OP_GET_LOG_PAGE	= 0x8,
+	RM_QUEUE_OP_LOAD_FW		= 0xA,
+	RM_QUEUE_OP_LOAD_APU_FW		= 0xD,
+	RM_QUEUE_OP_VMR_CONTROL		= 0xE,
+	RM_QUEUE_OP_IDENTIFY		= 0x202,
+};
+
+struct rm_cmd_sq_hdr {
+	u16 opcode;
+	u16 msg_size;
+	u16 id;
+	u16 reserved;
+} __packed;
+
+struct rm_cmd_cq_hdr {
+	u16 id;
+	u16 reserved;
+} __packed;
+
+struct rm_cmd_sq_bin {
+	u64			address;
+	u32			size;
+	u32			reserved1;
+	u32			reserved2;
+	u32			reserved3;
+	u64			reserved4;
+} __packed;
+
+struct rm_cmd_sq_log_page {
+	u64			address;
+	u32			size;
+	u32			reserved1;
+	u32			type;
+	u32			reserved2;
+} __packed;
+
+struct rm_cmd_sq_ctrl {
+	u32			status;
+} __packed;
+
+struct rm_cmd_sq_data {
+	union {
+		struct rm_cmd_sq_log_page	page;
+		struct rm_cmd_sq_bin		bin;
+		struct rm_cmd_sq_ctrl		ctrl;
+	};
+} __packed;
+
+struct rm_cmd_cq_identify {
+	u16			major;
+	u16			minor;
+	u32			reserved;
+} __packed;
+
+struct rm_cmd_cq_log_page {
+	u32			len;
+	u32			reserved;
+} __packed;
+
+struct rm_cmd_cq_control {
+	u16			status;
+	u16			reserved1;
+	u32			reserved2;
+} __packed;
+
+struct rm_cmd_cq_data {
+	union {
+		struct rm_cmd_cq_identify	identify;
+		struct rm_cmd_cq_log_page	page;
+		struct rm_cmd_cq_control	ctrl;
+		u32				reserved[2];
+	};
+	u32			rcode;
+} __packed;
+
+struct rm_cmd_sq_msg {
+	struct rm_cmd_sq_hdr	hdr;
+	struct rm_cmd_sq_data	data;
+} __packed;
+
+struct rm_cmd_cq_msg {
+	struct rm_cmd_cq_hdr	hdr;
+	struct rm_cmd_cq_data	data;
+} __packed;
+
+struct rm_cmd {
+	struct rm_device	*rdev;
+	struct list_head	list;
+	struct completion	executed;
+	struct rm_cmd_sq_msg	sq_msg;
+	struct rm_cmd_cq_msg	cq_msg;
+	enum rm_queue_opcode	opcode;
+	void			*buffer;
+	ssize_t			size;
+};
+
+enum rm_queue_type {
+	RM_QUEUE_SQ,
+	RM_QUEUE_CQ
+};
+
+enum rm_cmd_log_page_type {
+	RM_CMD_LOG_PAGE_AXI_TRIP_STATUS	= 0x0,
+	RM_CMD_LOG_PAGE_FW_ID		= 0xA,
+};
+
+struct rm_queue {
+	enum rm_queue_type	type;
+	u32			pidx;
+	u32			cidx;
+	u32			offset;
+	u32			data_offset;
+	u32			data_size;
+	struct semaphore	data_lock;
+};
+
+struct rm_queue_header {
+	u32			magic;
+	u32			version;
+	u32			size;
+	u32			sq_off;
+	u32			sq_slot_size;
+	u32			cq_off;
+	u32			sq_cidx;
+	u32			cq_cidx;
+};
+
+struct rm_header {
+	u32			magic;
+	u32			queue_base;
+	u32			queue_size;
+	u32			status_off;
+	u32			status_len;
+	u32			log_index;
+	u32			log_off;
+	u32			log_size;
+	u32			data_start;
+	u32			data_end;
+};
+
+struct rm_device {
+	struct vmgmt_device	*vdev;
+	struct regmap		*shmem_regmap;
+	struct regmap		*io_regmap;
+
+	struct rm_header	rm_metadata;
+	u32			queue_buffer_start;
+	u32			queue_buffer_size;
+	u32			queue_base;
+
+	/* Lock to queue access */
+	struct mutex		queue;
+	struct rm_queue		sq;
+	struct rm_queue		cq;
+	u32			queue_size;
+
+	struct timer_list	msg_timer;
+	struct work_struct	msg_monitor;
+	struct timer_list	health_timer;
+	struct work_struct	health_monitor;
+	struct list_head	submitted_cmds;
+
+	int			firewall_tripped;
+};
+
+int rm_queue_create_cmd(struct rm_device *rdev, enum rm_queue_opcode opcode,
+			struct rm_cmd **cmd_ptr);
+void rm_queue_destory_cmd(struct rm_cmd *cmd);
+
+int rm_queue_data_init(struct rm_cmd *cmd, const char *buffer, ssize_t size);
+void rm_queue_data_fini(struct rm_cmd *cmd);
+
+int rm_queue_copy_response(struct rm_cmd *cmd, void *buffer, ssize_t len);
+
+int rm_boot_apu(struct rm_device *rdev);
+
+#endif	/* __VMGMT_RM_H */
diff --git a/drivers/fpga/amd/vmgmt.c b/drivers/fpga/amd/vmgmt.c
index b72eff9e8bc0..198213a13c7d 100644
--- a/drivers/fpga/amd/vmgmt.c
+++ b/drivers/fpga/amd/vmgmt.c
@@ -21,6 +21,8 @@
 
 #include "vmgmt.h"
 #include "vmgmt-comms.h"
+#include "vmgmt-rm.h"
+#include "vmgmt-rm-queue.h"
 
 #define DRV_NAME			"amd-vmgmt"
 #define CLASS_NAME			DRV_NAME
@@ -43,6 +45,61 @@ static inline struct vmgmt_device *vmgmt_inode_to_vdev(struct inode *inode)
 	return (struct vmgmt_device *)container_of(inode->i_cdev, struct vmgmt_device, cdev);
 }
 
+static int vmgmt_fpga_write_init(struct fpga_manager *mgr,
+				 struct fpga_image_info *info, const char *buf,
+				 size_t count)
+{
+	struct fpga_device *fdev = mgr->priv;
+	struct fw_tnx *tnx = &fdev->fw;
+	int ret;
+
+	ret = rm_queue_create_cmd(fdev->vdev->rdev, tnx->opcode, &tnx->cmd);
+	if (ret) {
+		fdev->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
+		return ret;
+	}
+
+	fdev->state = FPGA_MGR_STATE_WRITE_INIT;
+	return ret;
+}
+
+static int vmgmt_fpga_write(struct fpga_manager *mgr, const char *buf,
+			    size_t count)
+{
+	struct fpga_device *fdev = mgr->priv;
+	int ret;
+
+	ret = rm_queue_data_init(fdev->fw.cmd, buf, count);
+	if (ret) {
+		fdev->state = FPGA_MGR_STATE_WRITE_ERR;
+		rm_queue_destory_cmd(fdev->fw.cmd);
+		return ret;
+	}
+
+	fdev->state = FPGA_MGR_STATE_WRITE;
+	return ret;
+}
+
+static int vmgmt_fpga_write_complete(struct fpga_manager *mgr,
+				     struct fpga_image_info *info)
+{
+	struct fpga_device *fdev = mgr->priv;
+	int ret;
+
+	ret = rm_queue_send_cmd(fdev->fw.cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
+	if (ret) {
+		fdev->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
+		vmgmt_err(fdev->vdev, "Send cmd failed:%d, cid:%d", ret, fdev->fw.id);
+	} else {
+		fdev->state = FPGA_MGR_STATE_WRITE_COMPLETE;
+	}
+
+	rm_queue_data_fini(fdev->fw.cmd);
+	rm_queue_destory_cmd(fdev->fw.cmd);
+	memset(&fdev->fw, 0, sizeof(fdev->fw));
+	return ret;
+}
+
 static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)
 {
 	struct fpga_device *fdev = mgr->priv;
@@ -51,6 +108,9 @@ static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)
 }
 
 static const struct fpga_manager_ops vmgmt_fpga_ops = {
+	.write_init = vmgmt_fpga_write_init,
+	.write = vmgmt_fpga_write,
+	.write_complete = vmgmt_fpga_write_complete,
 	.state = vmgmt_fpga_state,
 };
 
@@ -96,6 +156,13 @@ static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
 		return ERR_PTR(ret);
 	}
 
+	ret = vmgmt_rm_get_fw_id(vdev->rdev, &vdev->intf_uuid);
+	if (ret) {
+		vmgmt_warn(vdev, "Failed to get interface uuid");
+		ret = -EINVAL;
+		goto unregister_fpga_mgr;
+	}
+
 	/* create fgpa bridge, region for the base shell */
 	fdev->bridge = fpga_bridge_register(dev, "AMD Versal FPGA Bridge",
 					    &vmgmt_br_ops, fdev);
@@ -132,6 +199,149 @@ static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
 	return ERR_PTR(ret);
 }
 
+static int vmgmt_region_program(struct fpga_region *region, const void *data)
+{
+	struct fpga_device *fdev = region->priv;
+	struct vmgmt_device *vdev = fdev->vdev;
+	const struct axlf *xclbin = data;
+	struct fpga_image_info *info;
+	int ret;
+
+	info = fpga_image_info_alloc(&vdev->pdev->dev);
+	if (!info)
+		return -ENOMEM;
+
+	region->info = info;
+
+	info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+	info->count = xclbin->header.length;
+	info->buf = (char *)xclbin;
+
+	ret = fpga_region_program_fpga(region);
+	if (ret) {
+		vmgmt_err(vdev, "Programming xclbin failed: %d", ret);
+		goto exit;
+	}
+
+	/* free bridges to allow reprogram */
+	if (region->get_bridges)
+		fpga_bridges_put(&region->bridge_list);
+
+exit:
+	fpga_image_info_free(info);
+	return ret;
+}
+
+static int vmgmt_fpga_region_match(struct device *dev, const void *data)
+{
+	const struct vmgmt_fpga_region *arg = data;
+	const struct fpga_region *match_region;
+	struct fpga_device *fdev = arg->fdev;
+	uuid_t compat_uuid;
+
+	if (dev->parent != &fdev->vdev->pdev->dev)
+		return false;
+
+	match_region = to_fpga_region(dev);
+
+	import_uuid(&compat_uuid, (const char *)match_region->compat_id);
+	if (uuid_equal(&compat_uuid, arg->uuid)) {
+		vmgmt_dbg(fdev->vdev, "Region match found");
+		return true;
+	}
+
+	vmgmt_err(fdev->vdev, "download uuid %pUb is not the same as device uuid %pUb",
+		  arg->uuid, &compat_uuid);
+	return false;
+}
+
+static long vmgmt_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
+{
+	struct vmgmt_device *vdev = (struct vmgmt_device *)filep->private_data;
+	struct vmgmt_fpga_region reg = { 0 };
+	struct fpga_region *region = NULL;
+	struct axlf *axlf = NULL;
+	void *data = NULL;
+	size_t size = 0;
+	int ret = 0;
+
+	axlf = vmalloc(sizeof(*axlf));
+	if (!axlf)
+		return -ENOMEM;
+
+	ret = copy_from_user((void *)axlf, (void *)arg, sizeof(*axlf));
+	if (ret) {
+		vmgmt_err(vdev, "Failed to copy axlf: %d", ret);
+		ret = -EFAULT;
+		goto exit;
+	}
+
+	ret = memcmp(axlf->magic, VERSAL_XCLBIN_MAGIC_ID,
+		     sizeof(VERSAL_XCLBIN_MAGIC_ID));
+	if (ret) {
+		vmgmt_err(vdev, "unknown axlf magic %s", axlf->magic);
+		ret = -EINVAL;
+		goto exit;
+	}
+
+	/* axlf should never be over 1G and less than size of struct axlf */
+	size = axlf->header.length;
+	if (size < sizeof(struct axlf) || size > 1024 * 1024 * 1024) {
+		vmgmt_err(vdev, "axlf length %zu is invalid", size);
+		ret = -EINVAL;
+		goto exit;
+	}
+
+	data = vmalloc(size);
+	if (!data) {
+		ret = -ENOMEM;
+		goto exit;
+	}
+
+	ret = copy_from_user((void *)data, (void *)arg, size);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to copy data: %d", ret);
+		ret = -EFAULT;
+		goto exit;
+	}
+
+	switch (cmd) {
+	case VERSAL_MGMT_LOAD_XCLBIN_IOCTL:
+		vdev->fdev->fw.opcode = RM_QUEUE_OP_LOAD_XCLBIN;
+		break;
+	default:
+		vmgmt_err(vdev, "Invalid IOCTL command: %d", cmd);
+		ret = -EINVAL;
+		goto exit;
+	}
+
+	reg.uuid = &axlf->header.rom_uuid;
+	reg.fdev = vdev->fdev;
+
+	region = fpga_region_class_find(NULL, &reg, vmgmt_fpga_region_match);
+	if (!region) {
+		vmgmt_err(vdev, "Failed to find compatible region");
+		ret = -ENOENT;
+		goto exit;
+	}
+
+	ret = vmgmt_region_program(region, data);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to program region");
+		goto exit;
+	}
+
+	vmgmt_info(vdev, "Downloaded axlf %pUb of size %zu Bytes",
+		   &axlf->header.uuid, size);
+	uuid_copy(&vdev->xclbin_uuid, &axlf->header.uuid);
+
+exit:
+	vfree(data);
+	vfree(axlf);
+
+	return ret;
+}
+
 static int vmgmt_open(struct inode *inode, struct file *filep)
 {
 	struct vmgmt_device *vdev = vmgmt_inode_to_vdev(inode);
@@ -155,6 +365,7 @@ static const struct file_operations vmgmt_fops = {
 	.owner = THIS_MODULE,
 	.open = vmgmt_open,
 	.release = vmgmt_release,
+	.unlocked_ioctl = vmgmt_ioctl,
 };
 
 static void vmgmt_chrdev_destroy(struct vmgmt_device *vdev)
@@ -201,6 +412,69 @@ static int vmgmt_chrdev_create(struct vmgmt_device *vdev)
 	return 0;
 }
 
+static enum fw_upload_err vmgmt_fw_prepare(struct fw_upload *fw_upload,
+					   const u8 *data, u32 size)
+{
+	struct firmware_device *fwdev = fw_upload->dd_handle;
+	struct axlf *xsabin = (struct axlf *)data;
+	int ret;
+
+	ret = memcmp(xsabin->magic, VERSAL_XCLBIN_MAGIC_ID,
+		     sizeof(VERSAL_XCLBIN_MAGIC_ID));
+	if (ret) {
+		vmgmt_err(fwdev->vdev, "Invalid device firmware");
+		return FW_UPLOAD_ERR_INVALID_SIZE;
+	}
+
+	/* Firmware size should never be over 1G and less than size of struct axlf */
+	if (!size || size != xsabin->header.length || size < sizeof(*xsabin) ||
+	    size > 1024 * 1024 * 1024) {
+		vmgmt_err(fwdev->vdev, "Invalid device firmware size");
+		return FW_UPLOAD_ERR_INVALID_SIZE;
+	}
+
+	ret = rm_queue_create_cmd(fwdev->vdev->rdev, RM_QUEUE_OP_LOAD_FW,
+				  &fwdev->cmd);
+	if (ret)
+		return FW_UPLOAD_ERR_RW_ERROR;
+
+	uuid_copy(&fwdev->uuid, &xsabin->header.uuid);
+	return FW_UPLOAD_ERR_NONE;
+}
+
+static enum fw_upload_err vmgmt_fw_write(struct fw_upload *fw_upload,
+					 const u8 *data, u32 offset, u32 size,
+					 u32 *written)
+{
+	struct firmware_device *fwdev = fw_upload->dd_handle;
+	int ret;
+
+	ret = rm_queue_data_init(fwdev->cmd, data, size);
+	if (ret)
+		return FW_UPLOAD_ERR_RW_ERROR;
+
+	*written = size;
+	return FW_UPLOAD_ERR_NONE;
+}
+
+static enum fw_upload_err vmgmt_fw_poll_complete(struct fw_upload *fw_upload)
+{
+	struct firmware_device *fwdev = fw_upload->dd_handle;
+	int ret;
+
+	vmgmt_info(fwdev->vdev, "Programming device firmware: %pUb", &fwdev->uuid);
+
+	ret = rm_queue_send_cmd(fwdev->cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
+	if (ret) {
+		vmgmt_err(fwdev->vdev, "Send cmd failedi:%d, cid %d", ret, fwdev->id);
+		return FW_UPLOAD_ERR_HW_ERROR;
+	}
+
+	vmgmt_info(fwdev->vdev, "Successfully programmed device firmware: %pUb",
+		   &fwdev->uuid);
+	return FW_UPLOAD_ERR_NONE;
+}
+
 static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
 {
 	struct firmware_device *fwdev = fw_upload->dd_handle;
@@ -208,8 +482,26 @@ static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
 	vmgmt_warn(fwdev->vdev, "canceled");
 }
 
+static void vmgmt_fw_cleanup(struct fw_upload *fw_upload)
+{
+	struct firmware_device *fwdev = fw_upload->dd_handle;
+
+	if (!fwdev->cmd)
+		return;
+
+	rm_queue_data_fini(fwdev->cmd);
+	rm_queue_destory_cmd(fwdev->cmd);
+
+	fwdev->cmd = NULL;
+	fwdev->id = 0;
+}
+
 static const struct fw_upload_ops vmgmt_fw_ops = {
+	.prepare = vmgmt_fw_prepare,
+	.write = vmgmt_fw_write,
+	.poll_complete = vmgmt_fw_poll_complete,
 	.cancel = vmgmt_fw_cancel,
+	.cleanup = vmgmt_fw_cleanup,
 };
 
 static void vmgmt_fw_upload_fini(struct firmware_device *fwdev)
@@ -250,17 +542,25 @@ static void vmgmt_device_teardown(struct vmgmt_device *vdev)
 	vmgmt_fpga_fini(vdev->fdev);
 	vmgmt_fw_upload_fini(vdev->fwdev);
 	vmgmtm_comms_fini(vdev->ccdev);
+	vmgmt_rm_fini(vdev->rdev);
 }
 
 static int vmgmt_device_setup(struct vmgmt_device *vdev)
 {
 	int ret;
 
+	vdev->rdev = vmgmt_rm_init(vdev);
+	if (IS_ERR(vdev->rdev)) {
+		ret = PTR_ERR(vdev->rdev);
+		vmgmt_err(vdev, "Failed to init runtime manager, err %d", ret);
+		return ret;
+	}
+
 	vdev->fwdev = vmgmt_fw_upload_init(vdev);
 	if (IS_ERR(vdev->fwdev)) {
 		ret = PTR_ERR(vdev->fwdev);
 		vmgmt_err(vdev, "Failed to init FW uploader, err %d", ret);
-		goto done;
+		goto rm_fini;
 	}
 
 	vdev->ccdev = vmgmtm_comms_init(vdev);
@@ -282,7 +582,8 @@ static int vmgmt_device_setup(struct vmgmt_device *vdev)
 	vmgmtm_comms_fini(vdev->ccdev);
 upload_fini:
 	vmgmt_fw_upload_fini(vdev->fwdev);
-done:
+rm_fini:
+	vmgmt_rm_fini(vdev->rdev);
 	return ret;
 }
 
diff --git a/drivers/fpga/amd/vmgmt.h b/drivers/fpga/amd/vmgmt.h
index 4dc8a43f825e..c767d1372881 100644
--- a/drivers/fpga/amd/vmgmt.h
+++ b/drivers/fpga/amd/vmgmt.h
@@ -19,6 +19,7 @@
 #include <linux/pci.h>
 #include <linux/uuid.h>
 #include <linux/regmap.h>
+#include <linux/vmalloc.h>
 
 #define AMD_VMGMT_BAR			0
 #define AMD_VMGMT_BAR_MASK		BIT(0)
@@ -93,8 +94,10 @@ struct vmgmt_device {
 	void __iomem		*tbl;
 	uuid_t			xclbin_uuid;
 	uuid_t			intf_uuid;
-
-	void                    *debugfs_root;
 };
 
+struct rm_device *vmgmt_rm_init(struct vmgmt_device *vdev);
+void vmgmt_rm_fini(struct rm_device *rdev);
+int vmgmt_rm_get_fw_id(struct rm_device *rdev, uuid_t *uuid);
+
 #endif	/* __VMGMT_H */
-- 
2.34.1

[PATCH V1 3/3] drivers/fpga/amd: Add remote queue service APIs

[David Zhang]

From: Yidong Zhang <yidong.zhang@amd.com>

Adds remote queue services inlcuding init, fini, and send command.

Co-developed-by: Nishad Saraf <nishads@amd.com>
Signed-off-by: Nishad Saraf <nishads@amd.com>
Co-developed-by: Prapul Krishnamurthy <prapulk@amd.com>
Signed-off-by: Prapul Krishnamurthy <prapulk@amd.com>
Signed-off-by: Yidong Zhang <yidong.zhang@amd.com>
---
 drivers/fpga/amd/vmgmt-rm-queue.c | 342 +++++++++++++++++++++++++++++-
 1 file changed, 341 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/amd/vmgmt-rm-queue.c b/drivers/fpga/amd/vmgmt-rm-queue.c
index fe805373ea32..f68439833d51 100644
--- a/drivers/fpga/amd/vmgmt-rm-queue.c
+++ b/drivers/fpga/amd/vmgmt-rm-queue.c
@@ -23,16 +23,356 @@
 #include "vmgmt-rm.h"
 #include "vmgmt-rm-queue.h"
 
+static inline struct rm_device *to_rdev_msg_monitor(struct work_struct *w)
+{
+	return container_of(w, struct rm_device, msg_monitor);
+}
+
+static inline struct rm_device *to_rdev_msg_timer(struct timer_list *t)
+{
+	return container_of(t, struct rm_device, msg_timer);
+}
+
+static inline int rm_queue_write(struct rm_device *rdev, u32 offset, u32 value)
+{
+	return regmap_write(rdev->shmem_regmap, rdev->queue_base + offset, value);
+}
+
+static inline int rm_queue_read(struct rm_device *rdev, u32 offset, u32 *value)
+{
+	return regmap_read(rdev->shmem_regmap, rdev->queue_base + offset, value);
+}
+
+static inline int rm_queue_bulk_read(struct rm_device *rdev, u32 offset,
+				     u32 *value, u32 size)
+{
+	if (size & 0x3) {
+		vmgmt_err(rdev->vdev, "size %d is not 4 Bytes aligned", size);
+		return -EINVAL;
+	}
+
+	return regmap_bulk_read(rdev->shmem_regmap, rdev->queue_base + offset,
+				value, DIV_ROUND_UP(size, 4));
+}
+
+static inline int rm_queue_bulk_write(struct rm_device *rdev, u32 offset,
+				      u32 *value, u32 size)
+{
+	if (size & 0x3) {
+		vmgmt_err(rdev->vdev, "size %d is not 4 Bytes aligned", size);
+		return -EINVAL;
+	}
+
+	return regmap_bulk_write(rdev->shmem_regmap, rdev->queue_base + offset,
+				 value, DIV_ROUND_UP(size, 4));
+}
+
+static inline int rm_queue_get_cidx(struct rm_device *rdev,
+				    enum rm_queue_type type, u32 *value)
+{
+	u32 off;
+
+	if (type == RM_QUEUE_SQ)
+		off = offsetof(struct rm_queue_header, sq_cidx);
+	else
+		off = offsetof(struct rm_queue_header, cq_cidx);
+
+	return rm_queue_read(rdev, off, value);
+}
+
+static inline int rm_queue_set_cidx(struct rm_device *rdev,
+				    enum rm_queue_type type, u32 value)
+{
+	u32 off;
+
+	if (type == RM_QUEUE_SQ)
+		off = offsetof(struct rm_queue_header, sq_cidx);
+	else
+		off = offsetof(struct rm_queue_header, cq_cidx);
+
+	return rm_queue_write(rdev, off, value);
+}
+
+static inline int rm_queue_get_pidx(struct rm_device *rdev,
+				    enum rm_queue_type type, u32 *value)
+{
+	if (type == RM_QUEUE_SQ)
+		return regmap_read(rdev->io_regmap, RM_IO_SQ_PIDX_OFF, value);
+	else
+		return regmap_read(rdev->io_regmap, RM_IO_CQ_PIDX_OFF, value);
+}
+
+static inline int rm_queue_set_pidx(struct rm_device *rdev,
+				    enum rm_queue_type type, u32 value)
+{
+	if (type == RM_QUEUE_SQ)
+		return regmap_write(rdev->io_regmap, RM_IO_SQ_PIDX_OFF, value);
+	else
+		return regmap_write(rdev->io_regmap, RM_IO_CQ_PIDX_OFF, value);
+}
+
+static inline u32 rm_queue_get_sq_slot_offset(struct rm_device *rdev)
+{
+	u32 index;
+
+	if ((rdev->sq.pidx - rdev->sq.cidx) >= rdev->queue_size)
+		return RM_INVALID_SLOT;
+
+	index = rdev->sq.pidx & (rdev->queue_size - 1);
+	return rdev->sq.offset + RM_CMD_SQ_SLOT_SIZE * index;
+}
+
+static inline u32 rm_queue_get_cq_slot_offset(struct rm_device *rdev)
+{
+	u32 index;
+
+	index = rdev->cq.cidx & (rdev->queue_size - 1);
+	return rdev->cq.offset + RM_CMD_CQ_SLOT_SIZE * index;
+}
+
+static int rm_queue_submit_cmd(struct rm_cmd *cmd)
+{
+	struct vmgmt_device *vdev = cmd->rdev->vdev;
+	struct rm_device *rdev = cmd->rdev;
+	u32 offset;
+	int ret;
+
+	mutex_lock(&rdev->queue);
+
+	offset = rm_queue_get_sq_slot_offset(rdev);
+	if (!offset) {
+		vmgmt_err(vdev, "No SQ slot available");
+		ret = -ENOSPC;
+		goto exit;
+	}
+
+	ret = rm_queue_bulk_write(rdev, offset, (u32 *)&cmd->sq_msg,
+				  sizeof(cmd->sq_msg));
+	if (ret) {
+		vmgmt_err(vdev, "Failed to write msg to ring, ret %d", ret);
+		goto exit;
+	}
+
+	ret = rm_queue_set_pidx(rdev, RM_QUEUE_SQ, ++rdev->sq.pidx);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to update PIDX, ret %d", ret);
+		goto exit;
+	}
+
+	list_add_tail(&cmd->list, &rdev->submitted_cmds);
+exit:
+	mutex_unlock(&rdev->queue);
+	return ret;
+}
+
+static void rm_queue_withdraw_cmd(struct rm_cmd *cmd)
+{
+	mutex_lock(&cmd->rdev->queue);
+	list_del(&cmd->list);
+	mutex_unlock(&cmd->rdev->queue);
+}
+
+static int rm_queue_wait_cmd_timeout(struct rm_cmd *cmd, unsigned long timeout)
+{
+	struct vmgmt_device *vdev = cmd->rdev->vdev;
+	int ret;
+
+	if (wait_for_completion_timeout(&cmd->executed, timeout)) {
+		ret = cmd->cq_msg.data.rcode;
+		if (!ret)
+			return 0;
+
+		vmgmt_err(vdev, "CMD returned with a failure: %d", ret);
+		return ret;
+	}
+
+	/*
+	 * each cmds will be cleaned up by complete before it times out.
+	 * if we reach here, the cmd should be cleared and hot reset should
+	 * be issued.
+	 */
+	vmgmt_err(vdev, "cmd is timedout after, please reset the card");
+	rm_queue_withdraw_cmd(cmd);
+	return -ETIME;
+}
+
 int rm_queue_send_cmd(struct rm_cmd *cmd, unsigned long timeout)
 {
-	return 0;
+	int ret;
+
+	ret = rm_queue_submit_cmd(cmd);
+	if (ret)
+		return ret;
+
+	return rm_queue_wait_cmd_timeout(cmd, timeout);
+}
+
+static int rm_process_msg(struct rm_device *rdev)
+{
+	struct rm_cmd *cmd, *next;
+	struct vmgmt_device *vdev = rdev->vdev;
+	struct rm_cmd_cq_hdr header;
+	u32 offset;
+	int ret;
+
+	offset = rm_queue_get_cq_slot_offset(rdev);
+	if (!offset) {
+		vmgmt_err(vdev, "Invalid CQ offset");
+		return -EINVAL;
+	}
+
+	ret = rm_queue_bulk_read(rdev, offset, (u32 *)&header, sizeof(header));
+	if (ret) {
+		vmgmt_err(vdev, "Failed to read queue msg, %d", ret);
+		return ret;
+	}
+
+	list_for_each_entry_safe(cmd, next, &rdev->submitted_cmds, list) {
+		u32 value = 0;
+
+		if (cmd->sq_msg.hdr.id != header.id)
+			continue;
+
+		ret = rm_queue_bulk_read(rdev, offset + sizeof(cmd->cq_msg.hdr),
+					 (u32 *)&cmd->cq_msg.data,
+					 sizeof(cmd->cq_msg.data));
+		if (ret)
+			vmgmt_warn(vdev, "Failed to read queue msg, %d", ret);
+
+		ret = rm_queue_write(rdev, offset, value);
+		if (ret)
+			vmgmt_warn(vdev, "Failed to write queue msg, %d", ret);
+
+		list_del(&cmd->list);
+		complete(&cmd->executed);
+		return 0;
+	}
+
+	vmgmt_err(vdev, "Unknown cmd ID %d found in CQ", header.id);
+	return -EFAULT;
+}
+
+static void rm_check_msg(struct work_struct *w)
+{
+	struct rm_device *rdev = to_rdev_msg_monitor(w);
+	int ret;
+
+	mutex_lock(&rdev->queue);
+
+	ret = rm_queue_get_cidx(rdev, RM_QUEUE_SQ, &rdev->sq.cidx);
+	if (ret)
+		goto error;
+
+	ret = rm_queue_get_pidx(rdev, RM_QUEUE_CQ, &rdev->cq.pidx);
+	if (ret)
+		goto error;
+
+	while (rdev->cq.cidx < rdev->cq.pidx) {
+		ret = rm_process_msg(rdev);
+		if (ret)
+			break;
+
+		rdev->cq.cidx++;
+
+		ret = rm_queue_set_cidx(rdev, RM_QUEUE_CQ, rdev->cq.cidx);
+		if (ret)
+			break;
+	};
+
+error:
+	mutex_unlock(&rdev->queue);
+}
+
+static void rm_sched_work(struct timer_list *t)
+{
+	struct rm_device *rdev = to_rdev_msg_timer(t);
+
+	/* Schedule a work in the general workqueue */
+	schedule_work(&rdev->msg_monitor);
+	/* Periodic timer */
+	mod_timer(&rdev->msg_timer, jiffies + RM_COMPLETION_TIMER);
 }
 
 void rm_queue_fini(struct rm_device *rdev)
 {
+	del_timer_sync(&rdev->msg_timer);
+	cancel_work_sync(&rdev->msg_monitor);
+	mutex_destroy(&rdev->queue);
 }
 
 int rm_queue_init(struct rm_device *rdev)
 {
+	struct vmgmt_device *vdev = rdev->vdev;
+	struct rm_queue_header header = {0};
+	int ret;
+
+	INIT_LIST_HEAD(&rdev->submitted_cmds);
+	mutex_init(&rdev->queue);
+
+	ret = rm_queue_bulk_read(rdev, RM_HDR_OFF, (u32 *)&header,
+				 sizeof(header));
+	if (ret) {
+		vmgmt_err(vdev, "Failed to read RM shared mem, ret %d", ret);
+		goto error;
+	}
+
+	if (header.magic != RM_QUEUE_HDR_MAGIC_NUM) {
+		vmgmt_err(vdev, "Invalid RM queue header");
+		ret = -ENODEV;
+		goto error;
+	}
+
+	if (!header.version) {
+		vmgmt_err(vdev, "Invalid RM queue header");
+		ret = -ENODEV;
+		goto error;
+	}
+
+	sema_init(&rdev->sq.data_lock, 1);
+	sema_init(&rdev->cq.data_lock, 1);
+	rdev->queue_size = header.size;
+	rdev->sq.offset = header.sq_off;
+	rdev->cq.offset = header.cq_off;
+	rdev->sq.type = RM_QUEUE_SQ;
+	rdev->cq.type = RM_QUEUE_CQ;
+	rdev->sq.data_size = rdev->queue_buffer_size - RM_CMD_CQ_BUFFER_SIZE;
+	rdev->cq.data_size = RM_CMD_CQ_BUFFER_SIZE;
+	rdev->sq.data_offset = rdev->queue_buffer_start +
+			       RM_CMD_CQ_BUFFER_OFFSET + RM_CMD_CQ_BUFFER_SIZE;
+	rdev->cq.data_offset = rdev->queue_buffer_start +
+			       RM_CMD_CQ_BUFFER_OFFSET;
+	rdev->sq.cidx = header.sq_cidx;
+	rdev->cq.cidx = header.cq_cidx;
+
+	ret = rm_queue_get_pidx(rdev, RM_QUEUE_SQ, &rdev->sq.pidx);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to read sq.pidx, ret %d", ret);
+		goto error;
+	}
+
+	ret = rm_queue_get_pidx(rdev, RM_QUEUE_CQ, &rdev->cq.pidx);
+	if (ret) {
+		vmgmt_err(vdev, "Failed to read cq.pidx, ret %d", ret);
+		goto error;
+	}
+
+	if (rdev->cq.cidx != rdev->cq.pidx) {
+		vmgmt_warn(vdev, "Clearing stale completions");
+		rdev->cq.cidx = rdev->cq.pidx;
+		ret = rm_queue_set_cidx(rdev, RM_QUEUE_CQ, rdev->cq.cidx);
+		if (ret) {
+			vmgmt_err(vdev, "Failed to cleanup CQ, ret %d", ret);
+			goto error;
+		}
+	}
+
+	/* Create and schedule timer to do recurring work */
+	INIT_WORK(&rdev->msg_monitor, &rm_check_msg);
+	timer_setup(&rdev->msg_timer, &rm_sched_work, 0);
+	mod_timer(&rdev->msg_timer, jiffies + RM_COMPLETION_TIMER);
+
 	return 0;
+error:
+	mutex_destroy(&rdev->queue);
+	return ret;
 }
-- 
2.34.1

Re: [PATCH V1 2/3] drivers/fpga/amd: Add communication with firmware

[kernel test robot]

Hi David,

kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.12-rc2 next-20241008]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/David-Zhang/drivers-fpga-amd-Add-communication-with-firmware/20241008-060253
base:   linus/master
patch link:    https://lore.kernel.org/r/20241007220128.3023169-2-yidong.zhang%40amd.com
patch subject: [PATCH V1 2/3] drivers/fpga/amd: Add communication with firmware
config: i386-randconfig-054-20241009 (https://download.01.org/0day-ci/archive/20241009/202410091338.c38eM1Hd-lkp@intel.com/config)
compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202410091338.c38eM1Hd-lkp@intel.com/

cocci warnings: (new ones prefixed by >>)
>> drivers/fpga/amd/vmgmt-rm.c:183:2-3: Unneeded semicolon
--
>> drivers/fpga/amd/vmgmt-rm.c:498:10-17: WARNING: vzalloc should be used for buffer, instead of vmalloc/memset

vim +183 drivers/fpga/amd/vmgmt-rm.c

   146	
   147	int rm_queue_create_cmd(struct rm_device *rdev, enum rm_queue_opcode opcode,
   148				struct rm_cmd **cmd_ptr)
   149	{
   150		struct rm_cmd *cmd = NULL;
   151		int ret, id;
   152		u16 size;
   153	
   154		if (rdev->firewall_tripped)
   155			return -ENODEV;
   156	
   157		cmd = kzalloc(sizeof(*cmd), GFP_KERNEL);
   158		if (!cmd)
   159			return -ENOMEM;
   160		cmd->rdev = rdev;
   161	
   162		switch (opcode) {
   163		case RM_QUEUE_OP_LOAD_XCLBIN:
   164			fallthrough;
   165		case RM_QUEUE_OP_LOAD_FW:
   166			fallthrough;
   167		case RM_QUEUE_OP_LOAD_APU_FW:
   168			size = sizeof(struct rm_cmd_sq_bin);
   169			break;
   170		case RM_QUEUE_OP_GET_LOG_PAGE:
   171			size = sizeof(struct rm_cmd_sq_log_page);
   172			break;
   173		case RM_QUEUE_OP_IDENTIFY:
   174			size = 0;
   175			break;
   176		case RM_QUEUE_OP_VMR_CONTROL:
   177			size = sizeof(struct rm_cmd_sq_ctrl);
   178			break;
   179		default:
   180			vmgmt_err(rdev->vdev, "Invalid cmd opcode %d", opcode);
   181			ret = -EINVAL;
   182			goto error;
 > 183		};
   184	
   185		cmd->opcode = opcode;
   186		cmd->sq_msg.hdr.opcode = FIELD_PREP(RM_CMD_SQ_HDR_OPS_MSK, opcode);
   187		cmd->sq_msg.hdr.msg_size = FIELD_PREP(RM_CMD_SQ_HDR_SIZE_MSK, size);
   188	
   189		id = ida_alloc_range(&rm_cmd_ids, RM_CMD_ID_MIN, RM_CMD_ID_MAX, GFP_KERNEL);
   190		if (id < 0) {
   191			vmgmt_err(rdev->vdev, "Failed to alloc cmd ID: %d", id);
   192			ret = id;
   193			goto error;
   194		}
   195		cmd->sq_msg.hdr.id = id;
   196	
   197		init_completion(&cmd->executed);
   198	
   199		*cmd_ptr = cmd;
   200		return 0;
   201	error:
   202		kfree(cmd);
   203		return ret;
   204	}
   205	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH V1 3/3] drivers/fpga/amd: Add remote queue service APIs

[kernel test robot]

Hi David,

kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.12-rc2 next-20241008]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/David-Zhang/drivers-fpga-amd-Add-communication-with-firmware/20241008-060253
base:   linus/master
patch link:    https://lore.kernel.org/r/20241007220128.3023169-3-yidong.zhang%40amd.com
patch subject: [PATCH V1 3/3] drivers/fpga/amd: Add remote queue service APIs
config: i386-randconfig-054-20241009 (https://download.01.org/0day-ci/archive/20241009/202410091512.rcCqJO6z-lkp@intel.com/config)
compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202410091512.rcCqJO6z-lkp@intel.com/

cocci warnings: (new ones prefixed by >>)
>> drivers/fpga/amd/vmgmt-rm-queue.c:280:2-3: Unneeded semicolon

vim +280 drivers/fpga/amd/vmgmt-rm-queue.c

   254	
   255	static void rm_check_msg(struct work_struct *w)
   256	{
   257		struct rm_device *rdev = to_rdev_msg_monitor(w);
   258		int ret;
   259	
   260		mutex_lock(&rdev->queue);
   261	
   262		ret = rm_queue_get_cidx(rdev, RM_QUEUE_SQ, &rdev->sq.cidx);
   263		if (ret)
   264			goto error;
   265	
   266		ret = rm_queue_get_pidx(rdev, RM_QUEUE_CQ, &rdev->cq.pidx);
   267		if (ret)
   268			goto error;
   269	
   270		while (rdev->cq.cidx < rdev->cq.pidx) {
   271			ret = rm_process_msg(rdev);
   272			if (ret)
   273				break;
   274	
   275			rdev->cq.cidx++;
   276	
   277			ret = rm_queue_set_cidx(rdev, RM_QUEUE_CQ, rdev->cq.cidx);
   278			if (ret)
   279				break;
 > 280		};
   281	
   282	error:
   283		mutex_unlock(&rdev->queue);
   284	}
   285	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[kernel test robot]

Hi David,

kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.12-rc2 next-20241008]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/David-Zhang/drivers-fpga-amd-Add-communication-with-firmware/20241008-060253
base:   linus/master
patch link:    https://lore.kernel.org/r/20241007220128.3023169-1-yidong.zhang%40amd.com
patch subject: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card
config: x86_64-randconfig-121-20241009 (https://download.01.org/0day-ci/archive/20241009/202410091652.aQHmRoj1-lkp@intel.com/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241009/202410091652.aQHmRoj1-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202410091652.aQHmRoj1-lkp@intel.com/

sparse warnings: (new ones prefixed by >>)
>> drivers/fpga/amd/vmgmt.c:33:14: sparse: sparse: symbol 'vmgmt_class' was not declared. Should it be static?

vim +/vmgmt_class +33 drivers/fpga/amd/vmgmt.c

    30	
    31	static DEFINE_IDA(vmgmt_dev_minor_ida);
    32	static dev_t vmgmt_devnode;
  > 33	struct class *vmgmt_class;
    34	static struct fpga_bridge_ops vmgmt_br_ops;
    35	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH V1 2/3] drivers/fpga/amd: Add communication with firmware

[kernel test robot]

Hi David,

kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.12-rc2 next-20241008]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/David-Zhang/drivers-fpga-amd-Add-communication-with-firmware/20241008-060253
base:   linus/master
patch link:    https://lore.kernel.org/r/20241007220128.3023169-2-yidong.zhang%40amd.com
patch subject: [PATCH V1 2/3] drivers/fpga/amd: Add communication with firmware
config: x86_64-randconfig-121-20241009 (https://download.01.org/0day-ci/archive/20241009/202410091855.yLTZGOfr-lkp@intel.com/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241009/202410091855.yLTZGOfr-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202410091855.yLTZGOfr-lkp@intel.com/

sparse warnings: (new ones prefixed by >>)
   drivers/fpga/amd/vmgmt.c:35:14: sparse: sparse: symbol 'vmgmt_class' was not declared. Should it be static?
>> drivers/fpga/amd/vmgmt.c:272:45: sparse: sparse: incorrect type in argument 2 (different address spaces) @@     expected void const [noderef] __user *from @@     got void * @@
   drivers/fpga/amd/vmgmt.c:272:45: sparse:     expected void const [noderef] __user *from
   drivers/fpga/amd/vmgmt.c:272:45: sparse:     got void *
   drivers/fpga/amd/vmgmt.c:301:45: sparse: sparse: incorrect type in argument 2 (different address spaces) @@     expected void const [noderef] __user *from @@     got void * @@
   drivers/fpga/amd/vmgmt.c:301:45: sparse:     expected void const [noderef] __user *from
   drivers/fpga/amd/vmgmt.c:301:45: sparse:     got void *

vim +272 drivers/fpga/amd/vmgmt.c

   257	
   258	static long vmgmt_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
   259	{
   260		struct vmgmt_device *vdev = (struct vmgmt_device *)filep->private_data;
   261		struct vmgmt_fpga_region reg = { 0 };
   262		struct fpga_region *region = NULL;
   263		struct axlf *axlf = NULL;
   264		void *data = NULL;
   265		size_t size = 0;
   266		int ret = 0;
   267	
   268		axlf = vmalloc(sizeof(*axlf));
   269		if (!axlf)
   270			return -ENOMEM;
   271	
 > 272		ret = copy_from_user((void *)axlf, (void *)arg, sizeof(*axlf));
   273		if (ret) {
   274			vmgmt_err(vdev, "Failed to copy axlf: %d", ret);
   275			ret = -EFAULT;
   276			goto exit;
   277		}
   278	
   279		ret = memcmp(axlf->magic, VERSAL_XCLBIN_MAGIC_ID,
   280			     sizeof(VERSAL_XCLBIN_MAGIC_ID));
   281		if (ret) {
   282			vmgmt_err(vdev, "unknown axlf magic %s", axlf->magic);
   283			ret = -EINVAL;
   284			goto exit;
   285		}
   286	
   287		/* axlf should never be over 1G and less than size of struct axlf */
   288		size = axlf->header.length;
   289		if (size < sizeof(struct axlf) || size > 1024 * 1024 * 1024) {
   290			vmgmt_err(vdev, "axlf length %zu is invalid", size);
   291			ret = -EINVAL;
   292			goto exit;
   293		}
   294	
   295		data = vmalloc(size);
   296		if (!data) {
   297			ret = -ENOMEM;
   298			goto exit;
   299		}
   300	
   301		ret = copy_from_user((void *)data, (void *)arg, size);
   302		if (ret) {
   303			vmgmt_err(vdev, "Failed to copy data: %d", ret);
   304			ret = -EFAULT;
   305			goto exit;
   306		}
   307	
   308		switch (cmd) {
   309		case VERSAL_MGMT_LOAD_XCLBIN_IOCTL:
   310			vdev->fdev->fw.opcode = RM_QUEUE_OP_LOAD_XCLBIN;
   311			break;
   312		default:
   313			vmgmt_err(vdev, "Invalid IOCTL command: %d", cmd);
   314			ret = -EINVAL;
   315			goto exit;
   316		}
   317	
   318		reg.uuid = &axlf->header.rom_uuid;
   319		reg.fdev = vdev->fdev;
   320	
   321		region = fpga_region_class_find(NULL, &reg, vmgmt_fpga_region_match);
   322		if (!region) {
   323			vmgmt_err(vdev, "Failed to find compatible region");
   324			ret = -ENOENT;
   325			goto exit;
   326		}
   327	
   328		ret = vmgmt_region_program(region, data);
   329		if (ret) {
   330			vmgmt_err(vdev, "Failed to program region");
   331			goto exit;
   332		}
   333	
   334		vmgmt_info(vdev, "Downloaded axlf %pUb of size %zu Bytes",
   335			   &axlf->header.uuid, size);
   336		uuid_copy(&vdev->xclbin_uuid, &axlf->header.uuid);
   337	
   338	exit:
   339		vfree(data);
   340		vfree(axlf);
   341	
   342		return ret;
   343	}
   344	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[Xu Yilun]

On Mon, Oct 07, 2024 at 03:01:26PM -0700, David Zhang wrote:
> From: Yidong Zhang <yidong.zhang@amd.com>
> 
> AMD Versal based PCIe card, including V70, is designed for AI inference
> efficiency and is tuned for video analytics and natural language processing
> applications.
> 
> Add the driver to support AMD Versal card management physical function.
> Only very basic functionalities are added.

I think this is not "basic" enough.  If possible please add your following
functionalities one by one.

>   - module and PCI device initialization
>   - fpga framework ops callbacks
>   - communication with user physical function

So IIUC this is a multifunction PCI device? Management PF & User PF?
Next time please add some description about the architecture overview
of this card, as well as how the SW stack is supposed to make the card
work.

> 
> Co-developed-by: DMG Karthik <Karthik.DMG@amd.com>
> Signed-off-by: DMG Karthik <Karthik.DMG@amd.com>
> Co-developed-by: Nishad Saraf <nishads@amd.com>
> Signed-off-by: Nishad Saraf <nishads@amd.com>
> Co-developed-by: Prapul Krishnamurthy <prapulk@amd.com>
> Signed-off-by: Prapul Krishnamurthy <prapulk@amd.com>
> Signed-off-by: Yidong Zhang <yidong.zhang@amd.com>
> ---
>  MAINTAINERS                    |   7 +
>  drivers/fpga/Kconfig           |   3 +
>  drivers/fpga/Makefile          |   3 +
>  drivers/fpga/amd/Kconfig       |  17 ++
>  drivers/fpga/amd/Makefile      |   6 +
>  drivers/fpga/amd/vmgmt-comms.c | 344 ++++++++++++++++++++++++++++
>  drivers/fpga/amd/vmgmt-comms.h |  14 ++
>  drivers/fpga/amd/vmgmt.c       | 395 +++++++++++++++++++++++++++++++++
>  drivers/fpga/amd/vmgmt.h       | 100 +++++++++
>  include/uapi/linux/vmgmt.h     |  25 +++
>  10 files changed, 914 insertions(+)
>  create mode 100644 drivers/fpga/amd/Kconfig
>  create mode 100644 drivers/fpga/amd/Makefile
>  create mode 100644 drivers/fpga/amd/vmgmt-comms.c
>  create mode 100644 drivers/fpga/amd/vmgmt-comms.h
>  create mode 100644 drivers/fpga/amd/vmgmt.c
>  create mode 100644 drivers/fpga/amd/vmgmt.h
>  create mode 100644 include/uapi/linux/vmgmt.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index a097afd76ded..645f00ccb342 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1185,6 +1185,13 @@ M:	Sanjay R Mehta <sanju.mehta@amd.com>
>  S:	Maintained
>  F:	drivers/spi/spi-amd.c
>  
> +AMD VERSAL PCI DRIVER
> +M:	Yidong Zhang <yidong.zhang@amd.com>
> +L:	linux-fpga@vger.kernel.org
> +S:	Supported
> +F:	drivers/fpga/amd/
> +F:	include/uapi/linux/vmgmt.h
> +
>  AMD XGBE DRIVER
>  M:	"Shyam Sundar S K" <Shyam-sundar.S-k@amd.com>
>  L:	netdev@vger.kernel.org
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index 37b35f58f0df..dce060a7bd8f 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -290,4 +290,7 @@ config FPGA_MGR_LATTICE_SYSCONFIG_SPI
>  
>  source "drivers/fpga/tests/Kconfig"
>  
> +# Driver files
> +source "drivers/fpga/amd/Kconfig"
> +
>  endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index aeb89bb13517..5e8a3869f9a0 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -58,5 +58,8 @@ obj-$(CONFIG_FPGA_DFL_NIOS_INTEL_PAC_N3000)	+= dfl-n3000-nios.o
>  # Drivers for FPGAs which implement DFL
>  obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o
>  
> +# AMD PCIe Versal Management Driver
> +obj-y					+= amd/
> +
>  # KUnit tests
>  obj-$(CONFIG_FPGA_KUNIT_TESTS)		+= tests/
> diff --git a/drivers/fpga/amd/Kconfig b/drivers/fpga/amd/Kconfig
> new file mode 100644
> index 000000000000..126bc579a333
> --- /dev/null
> +++ b/drivers/fpga/amd/Kconfig
> @@ -0,0 +1,17 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +
> +config AMD_VERSAL_MGMT
> +	tristate "AMD PCIe Versal Management Driver"
> +	select FW_LOADER
> +	select FW_UPLOAD
> +	select REGMAP_MMIO
> +	depends on FPGA_BRIDGE
> +	depends on FPGA_REGION
> +	depends on HAS_IOMEM
> +	depends on PCI
> +	help
> +	  AMD PCIe Versal Management Driver provides management services to
> +	  download firmware, program bitstream, collect sensor data, control
> +	  resets, and communicate with the User function.
> +
> +	  If "M" is selected, the driver module will be amd-vmgmt.
> diff --git a/drivers/fpga/amd/Makefile b/drivers/fpga/amd/Makefile
> new file mode 100644
> index 000000000000..3e4c6dd3b787
> --- /dev/null
> +++ b/drivers/fpga/amd/Makefile
> @@ -0,0 +1,6 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +obj-$(CONFIG_AMD_VERSAL_MGMT)			+= amd-vmgmt.o

IMHO the naming vmgmt is hard to understand, any better idea?

> +
> +amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)		:= vmgmt.o	\
> +						   vmgmt-comms.o
> diff --git a/drivers/fpga/amd/vmgmt-comms.c b/drivers/fpga/amd/vmgmt-comms.c
> new file mode 100644
> index 000000000000..bed0d369a744
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-comms.c
> @@ -0,0 +1,344 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/device.h>
> +#include <linux/err.h>
> +#include <linux/jiffies.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/timer.h>
> +#include <linux/uuid.h>
> +#include <linux/workqueue.h>
> +
> +#include "vmgmt.h"
> +#include "vmgmt-comms.h"
> +
> +#define COMMS_PROTOCOL_VERSION			1
> +#define COMMS_PCI_BAR_OFF			0x2000000
> +#define COMMS_TIMER				(HZ / 10)
> +#define COMMS_DATA_LEN				16
> +#define COMMS_DATA_TYPE_MASK			GENMASK(7, 0)
> +#define COMMS_DATA_EOM_MASK			BIT(31)
> +#define COMMS_MSG_END				BIT(31)
> +
> +#define COMMS_REG_WRDATA_OFF			0x0
> +#define COMMS_REG_RDDATA_OFF			0x8
> +#define COMMS_REG_STATUS_OFF			0x10
> +#define COMMS_REG_ERROR_OFF			0x14
> +#define COMMS_REG_RIT_OFF			0x1C
> +#define COMMS_REG_IS_OFF			0x20
> +#define COMMS_REG_IE_OFF			0x24
> +#define COMMS_REG_CTRL_OFF			0x2C
> +#define COMMS_REGS_SIZE				0x1000
> +
> +#define COMMS_IRQ_DISABLE_ALL			0
> +#define COMMS_IRQ_RECEIVE_ENABLE		BIT(1)
> +#define COMMS_IRQ_CLEAR_ALL			GENMASK(2, 0)
> +#define COMMS_CLEAR_FIFO			GENMASK(1, 0)
> +#define COMMS_RECEIVE_THRESHOLD			15
> +
> +enum comms_req_ops {
> +	COMMS_REQ_OPS_UNKNOWN			= 0,
> +	COMMS_REQ_OPS_HOT_RESET			= 5,
> +	COMMS_REQ_OPS_GET_PROTOCOL_VERSION	= 19,
> +	COMMS_REQ_OPS_GET_XCLBIN_UUID		= 20,
> +	COMMS_REQ_OPS_MAX,
> +};
> +
> +enum comms_msg_type {
> +	COMMS_MSG_INVALID			= 0,
> +	COMMS_MSG_START				= 2,
> +	COMMS_MSG_BODY				= 3,
> +};
> +
> +enum comms_msg_service_type {
> +	COMMS_MSG_SRV_RESPONSE			= BIT(0),
> +	COMMS_MSG_SRV_REQUEST			= BIT(1),
> +};
> +
> +struct comms_hw_msg {
> +	struct {
> +		u32		type;
> +		u32		payload_size;
> +	} header;
> +	struct {
> +		u64		id;
> +		u32		flags;
> +		u32		size;
> +		u32		payload[COMMS_DATA_LEN - 6];
> +	} body;
> +} __packed;
> +
> +struct comms_srv_req {
> +	u64			flags;
> +	u32			opcode;
> +	u32			data[];
> +};
> +
> +struct comms_srv_ver_resp {
> +	u32			version;
> +};
> +
> +struct comms_srv_uuid_resp {
> +	uuid_t			uuid;
> +};
> +
> +struct comms_msg {
> +	u64			id;
> +	u32			flags;
> +	u32			len;
> +	u32			bytes_read;
> +	u32			data[10];
> +};
> +
> +struct comms_device {
> +	struct vmgmt_device	*vdev;
> +	struct regmap		*regmap;
> +	struct timer_list	timer;
> +	struct work_struct	work;
> +};
> +
> +static bool comms_regmap_rd_regs(struct device *dev, unsigned int reg)
> +{
> +	switch (reg) {
> +	case COMMS_REG_RDDATA_OFF:
> +	case COMMS_REG_IS_OFF:
> +		return true;
> +	default:
> +		return false;
> +	}
> +}
> +
> +static bool comms_regmap_wr_regs(struct device *dev, unsigned int reg)
> +{
> +	switch (reg) {
> +	case COMMS_REG_WRDATA_OFF:
> +	case COMMS_REG_IS_OFF:
> +	case COMMS_REG_IE_OFF:
> +	case COMMS_REG_CTRL_OFF:
> +	case COMMS_REG_RIT_OFF:
> +		return true;
> +	default:
> +		return false;
> +	}
> +}
> +
> +static bool comms_regmap_nir_regs(struct device *dev, unsigned int reg)
> +{
> +	switch (reg) {
> +	case COMMS_REG_RDDATA_OFF:
> +		return true;
> +	default:
> +		return false;
> +	}
> +}
> +
> +static const struct regmap_config comms_regmap_config = {
> +	.name = "comms_config",
> +	.reg_bits = 32,
> +	.reg_stride = 4,
> +	.val_bits = 32,
> +	.readable_reg = comms_regmap_rd_regs,
> +	.writeable_reg = comms_regmap_wr_regs,
> +	.readable_noinc_reg = comms_regmap_nir_regs,
> +};
> +
> +static inline struct comms_device *to_ccdev_work(struct work_struct *w)
> +{
> +	return container_of(w, struct comms_device, work);
> +}
> +
> +static inline struct comms_device *to_ccdev_timer(struct timer_list *t)
> +{
> +	return container_of(t, struct comms_device, timer);
> +}
> +
> +static u32 comms_set_uuid_resp(struct vmgmt_device *vdev, void *payload)
> +{
> +	struct comms_srv_uuid_resp *resp;
> +	u32 resp_len = sizeof(*resp);
> +
> +	resp = (struct comms_srv_uuid_resp *)payload;
> +	uuid_copy(&resp->uuid, &vdev->xclbin_uuid);
> +	vmgmt_dbg(vdev, "xclbin UUID: %pUb", &resp->uuid);
> +
> +	return resp_len;
> +}
> +
> +static u32 comms_set_protocol_resp(void *payload)
> +{
> +	struct comms_srv_ver_resp *resp = (struct comms_srv_ver_resp *)payload;
> +	u32 resp_len = sizeof(*resp);
> +
> +	resp->version = COMMS_PROTOCOL_VERSION;
> +
> +	return sizeof(resp_len);
> +}
> +
> +static void comms_send_response(struct comms_device *ccdev,
> +				struct comms_msg *msg)
> +{
> +	struct comms_srv_req *req = (struct comms_srv_req *)msg->data;
> +	struct vmgmt_device *vdev = ccdev->vdev;
> +	struct comms_hw_msg response = {0};
> +	u32 size;
> +	int ret;
> +	u8 i;
> +
> +	switch (req->opcode) {
> +	case COMMS_REQ_OPS_GET_PROTOCOL_VERSION:
> +		size = comms_set_protocol_resp(response.body.payload);
> +		break;
> +	case COMMS_REQ_OPS_GET_XCLBIN_UUID:
> +		size = comms_set_uuid_resp(vdev, response.body.payload);
> +		break;
> +	default:
> +		vmgmt_err(vdev, "Unsupported request opcode: %d", req->opcode);
> +		*response.body.payload = -1;
> +		size = sizeof(int);
> +	}
> +
> +	vmgmt_dbg(vdev, "Response opcode: %d", req->opcode);
> +
> +	response.header.type = COMMS_MSG_START | COMMS_MSG_END;
> +	response.header.payload_size = size;
> +
> +	response.body.flags = COMMS_MSG_SRV_RESPONSE;
> +	response.body.size = size;
> +	response.body.id = msg->id;
> +
> +	for (i = 0; i < COMMS_DATA_LEN; i++) {
> +		ret = regmap_write(ccdev->regmap, COMMS_REG_WRDATA_OFF, ((u32 *)&response)[i]);
> +		if (ret < 0) {
> +			vmgmt_err(vdev, "regmap write failed: %d", ret);
> +			return;
> +		}
> +	}
> +}
> +
> +#define STATUS_IS_READY(status) ((status) & BIT(1))
> +#define STATUS_IS_ERROR(status) ((status) & BIT(2))
> +
> +static void comms_check_request(struct work_struct *w)
> +{
> +	struct comms_device *ccdev = to_ccdev_work(w);
> +	u32 status = 0, request[COMMS_DATA_LEN] = {0};
> +	struct comms_hw_msg *hw_msg;
> +	struct comms_msg msg;
> +	u8 type, eom;
> +	int ret;
> +	int i;
> +
> +	ret = regmap_read(ccdev->regmap, COMMS_REG_IS_OFF, &status);
> +	if (ret) {
> +		vmgmt_err(ccdev->vdev, "regmap read failed: %d", ret);
> +		return;
> +	}
> +	if (!STATUS_IS_READY(status))
> +		return;
> +	if (STATUS_IS_ERROR(status)) {
> +		vmgmt_err(ccdev->vdev, "An error has occurred with comms");
> +		return;
> +	}
> +
> +	/* ACK status */
> +	regmap_write(ccdev->regmap, COMMS_REG_IS_OFF, status);
> +
> +	for (i = 0; i < COMMS_DATA_LEN; i++) {
> +		if (regmap_read(ccdev->regmap, COMMS_REG_RDDATA_OFF, &request[i]) < 0) {
> +			vmgmt_err(ccdev->vdev, "regmap read failed");
> +			return;
> +		}
> +	}
> +
> +	hw_msg = (struct comms_hw_msg *)request;
> +	type = FIELD_GET(COMMS_DATA_TYPE_MASK, hw_msg->header.type);
> +	eom = FIELD_GET(COMMS_DATA_EOM_MASK, hw_msg->header.type);
> +
> +	/* Only support fixed size 64B messages */
> +	if (!eom || type != COMMS_MSG_START) {
> +		vmgmt_err(ccdev->vdev, "Unsupported message format or length");
> +		return;
> +	}
> +
> +	msg.flags = hw_msg->body.flags;
> +	msg.len = hw_msg->body.size;
> +	msg.id = hw_msg->body.id;
> +
> +	if (msg.flags != COMMS_MSG_SRV_REQUEST) {
> +		vmgmt_err(ccdev->vdev, "Unsupported service request");
> +		return;
> +	}
> +
> +	if (hw_msg->body.size > sizeof(msg.data) * sizeof(msg.data[0])) {
> +		vmgmt_err(ccdev->vdev, "msg is too big: %d", hw_msg->body.size);
> +		return;
> +	}
> +	memcpy(msg.data, hw_msg->body.payload, hw_msg->body.size);

Why is the memcpy() necessary? I just see the data move from stack to
stack, finally they will be released all.

> +
> +	/* Now decode and respond appropriately */
> +	comms_send_response(ccdev, &msg);
> +}
> +
> +static void comms_sched_work(struct timer_list *t)
> +{
> +	struct comms_device *ccdev = to_ccdev_timer(t);
> +
> +	/* Schedule a work in the general workqueue */
> +	schedule_work(&ccdev->work);
> +	/* Periodic timer */
> +	mod_timer(&ccdev->timer, jiffies + COMMS_TIMER);
> +}
> +
> +static void comms_config(struct comms_device *ccdev)
> +{
> +	/* Disable interrupts */
> +	regmap_write(ccdev->regmap, COMMS_REG_IE_OFF, COMMS_IRQ_DISABLE_ALL);
> +	/* Clear request and response FIFOs */
> +	regmap_write(ccdev->regmap, COMMS_REG_CTRL_OFF, COMMS_CLEAR_FIFO);
> +	/* Clear interrupts */
> +	regmap_write(ccdev->regmap, COMMS_REG_IS_OFF, COMMS_IRQ_CLEAR_ALL);
> +	/* Setup RIT reg */
> +	regmap_write(ccdev->regmap, COMMS_REG_RIT_OFF, COMMS_RECEIVE_THRESHOLD);
> +	/* Enable RIT interrupt */
> +	regmap_write(ccdev->regmap, COMMS_REG_IE_OFF, COMMS_IRQ_RECEIVE_ENABLE);
> +
> +	/* Create and schedule timer to do recurring work */
> +	INIT_WORK(&ccdev->work, &comms_check_request);
> +	timer_setup(&ccdev->timer, &comms_sched_work, 0);
> +	mod_timer(&ccdev->timer, jiffies + COMMS_TIMER);

Do we have to use raw timer+workqueue for a normal periodic task? Could
delayed_work work for you?

And do we have to do endless periodic query? Couldn't the user PF driver
trigger the service? Where is the user PF driver?

> +}
> +
> +void vmgmtm_comms_fini(struct comms_device *ccdev)
> +{
> +	/* First stop scheduling new work then cancel work */
> +	del_timer_sync(&ccdev->timer);
> +	cancel_work_sync(&ccdev->work);
> +}
> +
> +struct comms_device *vmgmtm_comms_init(struct vmgmt_device *vdev)

So 'comms' means 'communication with user PF ', is it? I thought it was
some common services at first, especially it is introduced in the first
basic patch.

Any better name?

> +{
> +	struct comms_device *ccdev;
> +
> +	ccdev = devm_kzalloc(&vdev->pdev->dev, sizeof(*ccdev), GFP_KERNEL);
> +	if (!ccdev)
> +		return ERR_PTR(-ENOMEM);
> +
> +	ccdev->vdev = vdev;
> +
> +	ccdev->regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
> +					      vdev->tbl + COMMS_PCI_BAR_OFF,
> +					      &comms_regmap_config);

I'm not sure why a regmap is needed. All register accessing is within
the same module/file, and I assume a base+offset is enough to position
the register addr.

> +	if (IS_ERR(ccdev->regmap)) {
> +		vmgmt_err(vdev, "Comms regmap init failed");
> +		return ERR_CAST(ccdev->regmap);
> +	}
> +
> +	comms_config(ccdev);
> +	return ccdev;
> +}
> diff --git a/drivers/fpga/amd/vmgmt-comms.h b/drivers/fpga/amd/vmgmt-comms.h
> new file mode 100644
> index 000000000000..0afb14c8bd32
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-comms.h
> @@ -0,0 +1,14 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#ifndef __VMGMT_COMMS_H
> +#define __VMGMT_COMMS_H
> +
> +struct comms_device *vmgmtm_comms_init(struct vmgmt_device *vdev);
> +void vmgmtm_comms_fini(struct comms_device *ccdev);
> +
> +#endif	/* __VMGMT_COMMS_H */
> diff --git a/drivers/fpga/amd/vmgmt.c b/drivers/fpga/amd/vmgmt.c
> new file mode 100644
> index 000000000000..b72eff9e8bc0
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt.c
> @@ -0,0 +1,395 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#include <linux/cdev.h>
> +#include <linux/device/class.h>
> +#include <linux/err.h>
> +#include <linux/firmware.h>
> +#include <linux/fs.h>
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/idr.h>
> +#include <linux/kdev_t.h>
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/types.h>
> +#include <linux/uuid.h>
> +#include <linux/vmgmt.h>
> +
> +#include "vmgmt.h"
> +#include "vmgmt-comms.h"
> +
> +#define DRV_NAME			"amd-vmgmt"
> +#define CLASS_NAME			DRV_NAME
> +
> +#define PCI_DEVICE_ID_V70PQ2		0x50B0
> +#define VERSAL_XCLBIN_MAGIC_ID		"xclbin2"
> +
> +static DEFINE_IDA(vmgmt_dev_minor_ida);
> +static dev_t vmgmt_devnode;
> +struct class *vmgmt_class;
> +static struct fpga_bridge_ops vmgmt_br_ops;
> +
> +struct vmgmt_fpga_region {
> +	struct fpga_device *fdev;
> +	uuid_t *uuid;
> +};
> +
> +static inline struct vmgmt_device *vmgmt_inode_to_vdev(struct inode *inode)
> +{
> +	return (struct vmgmt_device *)container_of(inode->i_cdev, struct vmgmt_device, cdev);
> +}
> +
> +static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)
> +{
> +	struct fpga_device *fdev = mgr->priv;
> +
> +	return fdev->state;
> +}
> +
> +static const struct fpga_manager_ops vmgmt_fpga_ops = {
> +	.state = vmgmt_fpga_state,

If you want to add a skeleton, then add all skeleton ops with no
implementation. This makes me think the fpga_manager need .state() only.

> +};
> +
> +static int vmgmt_get_bridges(struct fpga_region *region)
> +{
> +	struct fpga_device *fdev = region->priv;
> +
> +	return fpga_bridge_get_to_list(&fdev->vdev->pdev->dev, region->info,
> +				       &region->bridge_list);
> +}
> +
> +static void vmgmt_fpga_fini(struct fpga_device *fdev)
> +{
> +	fpga_region_unregister(fdev->region);
> +	fpga_bridge_unregister(fdev->bridge);
> +	fpga_mgr_unregister(fdev->mgr);
> +}
> +
> +static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
> +{
> +	struct device *dev = &vdev->pdev->dev;
> +	struct fpga_region_info region = { 0 };
> +	struct fpga_manager_info info = { 0 };
> +	struct fpga_device *fdev;
> +	int ret;
> +
> +	fdev = devm_kzalloc(dev, sizeof(*fdev), GFP_KERNEL);
> +	if (!fdev)
> +		return ERR_PTR(-ENOMEM);
> +
> +	fdev->vdev = vdev;
> +
> +	info = (struct fpga_manager_info) {
> +		.name = "AMD Versal FPGA Manager",
> +		.mops = &vmgmt_fpga_ops,
> +		.priv = fdev,
> +	};
> +
> +	fdev->mgr = fpga_mgr_register_full(dev, &info);
> +	if (IS_ERR(fdev->mgr)) {
> +		ret = PTR_ERR(fdev->mgr);
> +		vmgmt_err(vdev, "Failed to register FPGA manager, err %d", ret);
> +		return ERR_PTR(ret);
> +	}
> +
> +	/* create fgpa bridge, region for the base shell */
> +	fdev->bridge = fpga_bridge_register(dev, "AMD Versal FPGA Bridge",
> +					    &vmgmt_br_ops, fdev);

I didn't find the br_ops anywhere in this patchset. So how to gate the
FPGA region when it is being reprogrammed? What is the physical link
between the FPGA region and outside visitors?

> +	if (IS_ERR(fdev->bridge)) {
> +		vmgmt_err(vdev, "Failed to register FPGA bridge, err %ld",
> +			  PTR_ERR(fdev->bridge));
> +		ret = PTR_ERR(fdev->bridge);
> +		goto unregister_fpga_mgr;
> +	}
> +
> +	region = (struct fpga_region_info) {
> +		.compat_id = (struct fpga_compat_id *)&vdev->intf_uuid,
> +		.get_bridges = vmgmt_get_bridges,
> +		.mgr = fdev->mgr,
> +		.priv = fdev,
> +	};
> +
> +	fdev->region = fpga_region_register_full(dev, &region);

I assume the fpga region represents the user PF, is it? If you
reprogram the FPGA region, how does the user PF driver aware the HW is
changing?

> +	if (IS_ERR(fdev->region)) {
> +		vmgmt_err(vdev, "Failed to register FPGA region, err %ld",
> +			  PTR_ERR(fdev->region));
> +		ret = PTR_ERR(fdev->region);
> +		goto unregister_fpga_bridge;
> +	}
> +
> +	return fdev;
> +
> +unregister_fpga_bridge:
> +	fpga_bridge_unregister(fdev->bridge);
> +
> +unregister_fpga_mgr:
> +	fpga_mgr_unregister(fdev->mgr);
> +
> +	return ERR_PTR(ret);
> +}
> +
> +static int vmgmt_open(struct inode *inode, struct file *filep)
> +{
> +	struct vmgmt_device *vdev = vmgmt_inode_to_vdev(inode);
> +
> +	if (WARN_ON(!vdev))
> +		return -ENODEV;
> +
> +	filep->private_data = vdev;
> +
> +	return 0;
> +}
> +
> +static int vmgmt_release(struct inode *inode, struct file *filep)
> +{
> +	filep->private_data = NULL;
> +
> +	return 0;
> +}
> +
> +static const struct file_operations vmgmt_fops = {
> +	.owner = THIS_MODULE,
> +	.open = vmgmt_open,
> +	.release = vmgmt_release,
> +};
> +
> +static void vmgmt_chrdev_destroy(struct vmgmt_device *vdev)
> +{
> +	device_destroy(vmgmt_class, vdev->cdev.dev);
> +	cdev_del(&vdev->cdev);
> +	ida_free(&vmgmt_dev_minor_ida, vdev->minor);
> +}
> +
> +static int vmgmt_chrdev_create(struct vmgmt_device *vdev)
> +{
> +	u32 devid;
> +	int ret;
> +
> +	vdev->minor = ida_alloc(&vmgmt_dev_minor_ida, GFP_KERNEL);
> +	if (vdev->minor < 0) {
> +		vmgmt_err(vdev, "Failed to allocate chrdev ID");
> +		return -ENODEV;
> +	}
> +
> +	cdev_init(&vdev->cdev, &vmgmt_fops);
> +
> +	vdev->cdev.owner = THIS_MODULE;
> +	vdev->cdev.dev = MKDEV(MAJOR(vmgmt_devnode), vdev->minor);
> +	ret = cdev_add(&vdev->cdev, vdev->cdev.dev, 1);
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to add char device: %d\n", ret);
> +		ida_free(&vmgmt_dev_minor_ida, vdev->minor);
> +		return -ENODEV;
> +	}
> +
> +	devid = PCI_DEVID(vdev->pdev->bus->number, vdev->pdev->devfn);
> +	vdev->device = device_create(vmgmt_class, &vdev->pdev->dev,
> +				     vdev->cdev.dev, NULL, "%s%x", DRV_NAME,
> +				     devid);
> +	if (IS_ERR(vdev->device)) {
> +		vmgmt_err(vdev, "Failed to create device: %ld\n",
> +			  PTR_ERR(vdev->device));
> +		cdev_del(&vdev->cdev);
> +		ida_free(&vmgmt_dev_minor_ida, vdev->minor);
> +		return -ENODEV;
> +	}
> +
> +	return 0;
> +}
> +
> +static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
> +{
> +	struct firmware_device *fwdev = fw_upload->dd_handle;
> +
> +	vmgmt_warn(fwdev->vdev, "canceled");
> +}
> +
> +static const struct fw_upload_ops vmgmt_fw_ops = {
> +	.cancel = vmgmt_fw_cancel,

Same concern.

> +};
> +
> +static void vmgmt_fw_upload_fini(struct firmware_device *fwdev)
> +{
> +	firmware_upload_unregister(fwdev->fw);
> +	kfree(fwdev->name);
> +}
> +
> +static struct firmware_device *vmgmt_fw_upload_init(struct vmgmt_device *vdev)
> +{
> +	struct device *dev = &vdev->pdev->dev;
> +	struct firmware_device *fwdev;
> +	u32 devid;
> +
> +	fwdev = devm_kzalloc(dev, sizeof(*fwdev), GFP_KERNEL);
> +	if (!fwdev)
> +		return ERR_PTR(-ENOMEM);
> +
> +	devid = PCI_DEVID(vdev->pdev->bus->number, vdev->pdev->devfn);
> +	fwdev->name = kasprintf(GFP_KERNEL, "%s%x", DRV_NAME, devid);
> +	if (!fwdev->name)
> +		return ERR_PTR(-ENOMEM);
> +
> +	fwdev->fw = firmware_upload_register(THIS_MODULE, dev, fwdev->name,
> +					     &vmgmt_fw_ops, fwdev);
> +	if (IS_ERR(fwdev->fw)) {
> +		kfree(fwdev->name);
> +		return ERR_CAST(fwdev->fw);
> +	}
> +
> +	fwdev->vdev = vdev;
> +
> +	return fwdev;
> +}
> +
> +static void vmgmt_device_teardown(struct vmgmt_device *vdev)
> +{
> +	vmgmt_fpga_fini(vdev->fdev);
> +	vmgmt_fw_upload_fini(vdev->fwdev);
> +	vmgmtm_comms_fini(vdev->ccdev);
> +}
> +
> +static int vmgmt_device_setup(struct vmgmt_device *vdev)
> +{
> +	int ret;
> +
> +	vdev->fwdev = vmgmt_fw_upload_init(vdev);
> +	if (IS_ERR(vdev->fwdev)) {
> +		ret = PTR_ERR(vdev->fwdev);
> +		vmgmt_err(vdev, "Failed to init FW uploader, err %d", ret);
> +		goto done;
> +	}
> +
> +	vdev->ccdev = vmgmtm_comms_init(vdev);
> +	if (IS_ERR(vdev->ccdev)) {
> +		ret = PTR_ERR(vdev->ccdev);
> +		vmgmt_err(vdev, "Failed to init comms channel, err %d", ret);
> +		goto upload_fini;
> +	}
> +
> +	vdev->fdev = vmgmt_fpga_init(vdev);
> +	if (IS_ERR(vdev->fdev)) {
> +		ret = PTR_ERR(vdev->fdev);
> +		vmgmt_err(vdev, "Failed to init FPGA maanger, err %d", ret);
> +		goto comms_fini;
> +	}
> +
> +	return 0;
> +comms_fini:
> +	vmgmtm_comms_fini(vdev->ccdev);
> +upload_fini:
> +	vmgmt_fw_upload_fini(vdev->fwdev);
> +done:
> +	return ret;
> +}
> +
> +static void vmgmt_remove(struct pci_dev *pdev)
> +{
> +	struct vmgmt_device *vdev = pci_get_drvdata(pdev);
> +
> +	vmgmt_chrdev_destroy(vdev);
> +	vmgmt_device_teardown(vdev);
> +}
> +
> +static int vmgmt_probe(struct pci_dev *pdev,
> +		       const struct pci_device_id *pdev_id)
> +{
> +	struct vmgmt_device *vdev;
> +	int ret;
> +
> +	vdev = devm_kzalloc(&pdev->dev, sizeof(*vdev), GFP_KERNEL);
> +	if (!vdev)
> +		return -ENOMEM;
> +
> +	pci_set_drvdata(pdev, vdev);
> +	vdev->pdev = pdev;
> +
> +	ret = pcim_enable_device(pdev);
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to enable device %d", ret);
> +		return ret;
> +	}
> +
> +	ret = pcim_iomap_regions(vdev->pdev, AMD_VMGMT_BAR_MASK, "amd-vmgmt");
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed iomap regions %d", ret);
> +		return -ENOMEM;
> +	}
> +
> +	vdev->tbl = pcim_iomap_table(vdev->pdev)[AMD_VMGMT_BAR];
> +	if (IS_ERR(vdev->tbl)) {
> +		vmgmt_err(vdev, "Failed to map RM shared memory BAR%d", AMD_VMGMT_BAR);
> +		return -ENOMEM;
> +	}

Deprecating pcim_iomap_regions & pcim_iomap_table is WIP. FYI.

  https://lore.kernel.org/all/20241016094911.24818-5-pstanner@redhat.com/

> +
> +	ret = vmgmt_device_setup(vdev);
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to setup Versal device %d", ret);
> +		return ret;
> +	}
> +
> +	ret = vmgmt_chrdev_create(vdev);
> +	if (ret) {
> +		vmgmt_device_teardown(vdev);
> +		return ret;
> +	}
> +
> +	vmgmt_dbg(vdev, "Successfully probed %s driver!", DRV_NAME);
> +	return 0;
> +}
> +
> +static const struct pci_device_id vmgmt_pci_ids[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_XILINX, PCI_DEVICE_ID_V70PQ2), },
> +	{ 0 }
> +};
> +
> +MODULE_DEVICE_TABLE(pci, vmgmt_pci_ids);
> +
> +static struct pci_driver amd_vmgmt_driver = {
> +	.name = DRV_NAME,
> +	.id_table = vmgmt_pci_ids,
> +	.probe = vmgmt_probe,
> +	.remove = vmgmt_remove,
> +};
> +
> +static int amd_vmgmt_init(void)
> +{
> +	int ret;
> +
> +	vmgmt_class = class_create(CLASS_NAME);
> +	if (IS_ERR(vmgmt_class))
> +		return PTR_ERR(vmgmt_class);
> +
> +	ret = alloc_chrdev_region(&vmgmt_devnode, 0, MINORMASK, DRV_NAME);
> +	if (ret)
> +		goto chr_err;
> +
> +	ret = pci_register_driver(&amd_vmgmt_driver);
> +	if (ret)
> +		goto pci_err;
> +
> +	return 0;
> +
> +pci_err:
> +	unregister_chrdev_region(vmgmt_devnode, MINORMASK);
> +chr_err:
> +	class_destroy(vmgmt_class);
> +	return ret;
> +}
> +
> +static void amd_vmgmt_exit(void)
> +{
> +	pci_unregister_driver(&amd_vmgmt_driver);
> +	unregister_chrdev_region(vmgmt_devnode, MINORMASK);
> +	class_destroy(vmgmt_class);
> +}
> +
> +module_init(amd_vmgmt_init);
> +module_exit(amd_vmgmt_exit);
> +
> +MODULE_DESCRIPTION("AMD PCIe Versal Management Driver");
> +MODULE_AUTHOR("XRT Team <runtimeca39d@amd.com>");
> +MODULE_LICENSE("GPL");
> diff --git a/drivers/fpga/amd/vmgmt.h b/drivers/fpga/amd/vmgmt.h
> new file mode 100644
> index 000000000000..4dc8a43f825e
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt.h
> @@ -0,0 +1,100 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#ifndef __VMGMT_H
> +#define __VMGMT_H
> +
> +#include <linux/cdev.h>
> +#include <linux/dev_printk.h>
> +#include <linux/jiffies.h>
> +#include <linux/list.h>
> +#include <linux/firmware.h>
> +#include <linux/fpga/fpga-bridge.h>
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/fpga/fpga-region.h>
> +#include <linux/pci.h>
> +#include <linux/uuid.h>
> +#include <linux/regmap.h>
> +
> +#define AMD_VMGMT_BAR			0
> +#define AMD_VMGMT_BAR_MASK		BIT(0)
> +
> +#define vmgmt_info(vdev, fmt, args...)					\
> +	dev_info(&(vdev)->pdev->dev, "%s: "fmt, __func__, ##args)
> +
> +#define vmgmt_warn(vdev, fmt, args...)					\
> +	dev_warn(&(vdev)->pdev->dev, "%s: "fmt, __func__, ##args)
> +
> +#define vmgmt_err(vdev, fmt, args...)					\
> +	dev_err(&(vdev)->pdev->dev, "%s: "fmt, __func__, ##args)
> +
> +#define vmgmt_dbg(vdev, fmt, args...)					\
> +	dev_dbg(&(vdev)->pdev->dev, fmt, ##args)
> +
> +struct vmgmt_device;
> +struct comms_device;
> +struct rm_cmd;
> +
> +struct axlf_header {
> +	u64				length;
> +	unsigned char			reserved1[24];
> +	uuid_t				rom_uuid;
> +	unsigned char			reserved2[64];
> +	uuid_t				uuid;
> +	unsigned char			reserved3[24];
> +} __packed;
> +
> +struct axlf {
> +	char				magic[8];
> +	unsigned char			reserved[296];
> +	struct axlf_header		header;
> +} __packed;
> +
> +struct fw_tnx {
> +	struct rm_cmd		*cmd;
> +	int			opcode;
> +	int			id;
> +};
> +
> +struct fpga_device {
> +	enum fpga_mgr_states	state;
> +	struct fpga_manager	*mgr;
> +	struct fpga_bridge	*bridge;
> +	struct fpga_region	*region;
> +	struct vmgmt_device	*vdev;
> +	struct fw_tnx		fw;
> +};
> +
> +struct firmware_device {
> +	struct vmgmt_device	*vdev;
> +	struct fw_upload	*fw;
> +	char			*name;
> +	u32			fw_name_id;
> +	struct rm_cmd		*cmd;
> +	int			id;
> +	uuid_t			uuid;
> +};
> +
> +struct vmgmt_device {
> +	struct pci_dev		*pdev;
> +
> +	struct rm_device	*rdev;
> +	struct comms_device	*ccdev;
> +	struct fpga_device	*fdev;
> +	struct firmware_device	*fwdev;
> +	struct cdev		cdev;
> +	struct device		*device;
> +
> +	int                     minor;
> +	void __iomem		*tbl;
> +	uuid_t			xclbin_uuid;
> +	uuid_t			intf_uuid;
> +
> +	void                    *debugfs_root;
> +};
> +
> +#endif	/* __VMGMT_H */
> diff --git a/include/uapi/linux/vmgmt.h b/include/uapi/linux/vmgmt.h
> new file mode 100644
> index 000000000000..2269ceb5c131
> --- /dev/null
> +++ b/include/uapi/linux/vmgmt.h
> @@ -0,0 +1,25 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Header file for Versal PCIe device user API
> + *
> + * Copyright (C) 2024 AMD Corporation, Inc.
> + */
> +
> +#ifndef _UAPI_LINUX_VMGMT_H
> +#define _UAPI_LINUX_VMGMT_H
> +
> +#include <linux/ioctl.h>
> +
> +#define VERSAL_MGMT_MAGIC	0xB7
> +#define VERSAL_MGMT_BASE	0
> +
> +/**
> + * VERSAL_MGMT_LOAD_XCLBIN_IOCTL - Download XCLBIN to the device
> + *
> + * This IOCTL is used to download XCLBIN down to the device.
> + * Return: 0 on success, -errno on failure.
> + */
> +#define VERSAL_MGMT_LOAD_XCLBIN_IOCTL	_IOW(VERSAL_MGMT_MAGIC,		\
> +					     VERSAL_MGMT_BASE + 0, void *)

Many definitions are added in a batch but some are not used in this
patch. Please reorganize the patches for easer review, even for first
version.

Thanks,
Yilun

> +
> +#endif /* _UAPI_LINUX_VMGMT_H */
> -- 
> 2.34.1
> 
> 

Re: [PATCH V1 2/3] drivers/fpga/amd: Add communication with firmware

[Xu Yilun]

On Mon, Oct 07, 2024 at 03:01:27PM -0700, David Zhang wrote:
> From: Yidong Zhang <yidong.zhang@amd.com>
> 
> Add queue based communication between host driver and firmware on the
> card. The remote queue (rm) can send/receive messages and providing

Abbrevate 'remote queue' to 'rm', or 'remote' to 'rm'. I see you use
rm_queue in the code.

> firmware downloading services.
> 
> Co-developed-by: Nishad Saraf <nishads@amd.com>
> Signed-off-by: Nishad Saraf <nishads@amd.com>
> Co-developed-by: Prapul Krishnamurthy <prapulk@amd.com>
> Signed-off-by: Prapul Krishnamurthy <prapulk@amd.com>
> Signed-off-by: Yidong Zhang <yidong.zhang@amd.com>
> ---
>  drivers/fpga/amd/Makefile         |   6 +-
>  drivers/fpga/amd/vmgmt-rm-queue.c |  38 +++
>  drivers/fpga/amd/vmgmt-rm-queue.h |  15 +
>  drivers/fpga/amd/vmgmt-rm.c       | 543 ++++++++++++++++++++++++++++++
>  drivers/fpga/amd/vmgmt-rm.h       | 222 ++++++++++++
>  drivers/fpga/amd/vmgmt.c          | 305 ++++++++++++++++-
>  drivers/fpga/amd/vmgmt.h          |   7 +-
>  7 files changed, 1130 insertions(+), 6 deletions(-)
>  create mode 100644 drivers/fpga/amd/vmgmt-rm-queue.c
>  create mode 100644 drivers/fpga/amd/vmgmt-rm-queue.h
>  create mode 100644 drivers/fpga/amd/vmgmt-rm.c
>  create mode 100644 drivers/fpga/amd/vmgmt-rm.h
> 
> diff --git a/drivers/fpga/amd/Makefile b/drivers/fpga/amd/Makefile
> index 3e4c6dd3b787..97cfff6be204 100644
> --- a/drivers/fpga/amd/Makefile
> +++ b/drivers/fpga/amd/Makefile
> @@ -2,5 +2,7 @@
>  
>  obj-$(CONFIG_AMD_VERSAL_MGMT)			+= amd-vmgmt.o
>  
> -amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)		:= vmgmt.o	\
> -						   vmgmt-comms.o
> +amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)		:= vmgmt.o		\
> +						   vmgmt-comms.o	\
> +						   vmgmt-rm.o		\
> +						   vmgmt-rm-queue.o
> diff --git a/drivers/fpga/amd/vmgmt-rm-queue.c b/drivers/fpga/amd/vmgmt-rm-queue.c
> new file mode 100644
> index 000000000000..fe805373ea32
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-rm-queue.c
> @@ -0,0 +1,38 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/completion.h>
> +#include <linux/err.h>
> +#include <linux/firmware.h>
> +#include <linux/idr.h>
> +#include <linux/jiffies.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/semaphore.h>
> +#include <linux/timer.h>
> +#include <linux/uuid.h>
> +#include <linux/workqueue.h>
> +
> +#include "vmgmt.h"
> +#include "vmgmt-rm.h"
> +#include "vmgmt-rm-queue.h"
> +
> +int rm_queue_send_cmd(struct rm_cmd *cmd, unsigned long timeout)
> +{
> +	return 0;
> +}
> +
> +void rm_queue_fini(struct rm_device *rdev)
> +{
> +}
> +
> +int rm_queue_init(struct rm_device *rdev)
> +{
> +	return 0;
> +}
> diff --git a/drivers/fpga/amd/vmgmt-rm-queue.h b/drivers/fpga/amd/vmgmt-rm-queue.h
> new file mode 100644
> index 000000000000..6fd0e0026a13
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-rm-queue.h
> @@ -0,0 +1,15 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#ifndef __VMGMT_RM_QUEUE_H
> +#define __VMGMT_RM_QUEUE_H
> +
> +int rm_queue_init(struct rm_device *rdev);
> +void rm_queue_fini(struct rm_device *rdev);
> +int rm_queue_send_cmd(struct rm_cmd *cmd, unsigned long timeout);
> +
> +#endif	/* __VMGMT_RM_QUEUE_H */
> diff --git a/drivers/fpga/amd/vmgmt-rm.c b/drivers/fpga/amd/vmgmt-rm.c
> new file mode 100644
> index 000000000000..856d5af52c8d
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-rm.c
> @@ -0,0 +1,543 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/completion.h>
> +#include <linux/err.h>
> +#include <linux/firmware.h>
> +#include <linux/idr.h>
> +#include <linux/jiffies.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/semaphore.h>
> +#include <linux/timer.h>
> +#include <linux/uuid.h>
> +#include <linux/workqueue.h>
> +
> +#include "vmgmt.h"
> +#include "vmgmt-rm.h"
> +#include "vmgmt-rm-queue.h"
> +
> +static DEFINE_IDA(rm_cmd_ids);
> +
> +static const struct regmap_config rm_shmem_regmap_config = {
> +	.name = "rm_shmem_config",
> +	.reg_bits = 32,
> +	.reg_stride = 4,
> +	.val_bits = 32,
> +};
> +
> +static const struct regmap_config rm_io_regmap_config = {
> +	.name = "rm_io_config",
> +	.reg_bits = 32,
> +	.reg_stride = 4,
> +	.val_bits = 32,
> +};
> +
> +static void rm_uninstall_health_monitor(struct rm_device *rdev);
> +
> +static inline struct rm_device *to_rdev_health_monitor(struct work_struct *w)
> +{
> +	return container_of(w, struct rm_device, health_monitor);
> +}
> +
> +static inline struct rm_device *to_rdev_health_timer(struct timer_list *t)
> +{
> +	return container_of(t, struct rm_device, health_timer);
> +}
> +
> +static inline int rm_shmem_read(struct rm_device *rdev, u32 offset, u32 *value)
> +{
> +	return regmap_read(rdev->shmem_regmap, offset, value);
> +}
> +
> +static inline int rm_shmem_bulk_read(struct rm_device *rdev, u32 offset,
> +				     u32 *value, u32 size)
> +{
> +	return regmap_bulk_read(rdev->shmem_regmap, offset, value,
> +				DIV_ROUND_UP(size, 4));
> +}
> +
> +static inline int rm_shmem_bulk_write(struct rm_device *rdev, u32 offset,
> +				      u32 *value, u32 size)
> +{
> +	return regmap_bulk_write(rdev->shmem_regmap, offset, value,
> +				DIV_ROUND_UP(size, 4));
> +}
> +
> +void rm_queue_destory_cmd(struct rm_cmd *cmd)
> +{
> +	ida_free(&rm_cmd_ids, cmd->sq_msg.hdr.id);
> +	kfree(cmd);
> +}
> +
> +int rm_queue_copy_response(struct rm_cmd *cmd, void *buffer, ssize_t len)
> +{
> +	struct rm_cmd_cq_log_page *result = &cmd->cq_msg.data.page;
> +	u64 off = cmd->sq_msg.data.page.address;
> +
> +	if (!result->len || len < result->len) {
> +		vmgmt_err(cmd->rdev->vdev, "Invalid response or buffer size");
> +		return -EINVAL;
> +	}
> +
> +	return rm_shmem_bulk_read(cmd->rdev, off, (u32 *)buffer, result->len);
> +}
> +
> +static void rm_queue_payload_fini(struct rm_cmd *cmd)
> +{
> +	up(&cmd->rdev->cq.data_lock);
> +}
> +
> +static int rm_queue_payload_init(struct rm_cmd *cmd,
> +				 enum rm_cmd_log_page_type type)
> +{
> +	struct rm_device *rdev = cmd->rdev;
> +	int ret;
> +
> +	ret = down_interruptible(&rdev->cq.data_lock);
> +	if (ret)
> +		return ret;
> +
> +	cmd->sq_msg.data.page.address = rdev->cq.data_offset;
> +	cmd->sq_msg.data.page.size = rdev->cq.data_size;
> +	cmd->sq_msg.data.page.reserved1 = 0;
> +	cmd->sq_msg.data.page.type = FIELD_PREP(RM_CMD_LOG_PAGE_TYPE_MASK,
> +						type);
> +	return 0;
> +}
> +
> +void rm_queue_data_fini(struct rm_cmd *cmd)
> +{
> +	up(&cmd->rdev->sq.data_lock);
> +}
> +
> +int rm_queue_data_init(struct rm_cmd *cmd, const char *buffer, ssize_t size)
> +{
> +	struct rm_device *rdev = cmd->rdev;
> +	int ret;
> +
> +	if (!size || size > rdev->sq.data_size) {
> +		vmgmt_err(rdev->vdev, "Unsupported file size");
> +		return -ENOMEM;
> +	}
> +
> +	ret = down_interruptible(&rdev->sq.data_lock);
> +	if (ret)
> +		return ret;
> +
> +	ret = rm_shmem_bulk_write(cmd->rdev, rdev->sq.data_offset,
> +				  (u32 *)buffer, size);
> +	if (ret) {
> +		vmgmt_err(rdev->vdev, "Failed to copy binary to SQ buffer");
> +		up(&cmd->rdev->sq.data_lock);
> +		return ret;
> +	}
> +
> +	cmd->sq_msg.data.bin.address = rdev->sq.data_offset;
> +	cmd->sq_msg.data.bin.size = size;
> +	return 0;
> +}
> +
> +int rm_queue_create_cmd(struct rm_device *rdev, enum rm_queue_opcode opcode,
> +			struct rm_cmd **cmd_ptr)
> +{
> +	struct rm_cmd *cmd = NULL;
> +	int ret, id;
> +	u16 size;
> +
> +	if (rdev->firewall_tripped)
> +		return -ENODEV;
> +
> +	cmd = kzalloc(sizeof(*cmd), GFP_KERNEL);
> +	if (!cmd)
> +		return -ENOMEM;
> +	cmd->rdev = rdev;
> +
> +	switch (opcode) {
> +	case RM_QUEUE_OP_LOAD_XCLBIN:
> +		fallthrough;
> +	case RM_QUEUE_OP_LOAD_FW:
> +		fallthrough;
> +	case RM_QUEUE_OP_LOAD_APU_FW:
> +		size = sizeof(struct rm_cmd_sq_bin);
> +		break;
> +	case RM_QUEUE_OP_GET_LOG_PAGE:
> +		size = sizeof(struct rm_cmd_sq_log_page);
> +		break;
> +	case RM_QUEUE_OP_IDENTIFY:
> +		size = 0;
> +		break;
> +	case RM_QUEUE_OP_VMR_CONTROL:
> +		size = sizeof(struct rm_cmd_sq_ctrl);
> +		break;
> +	default:
> +		vmgmt_err(rdev->vdev, "Invalid cmd opcode %d", opcode);
> +		ret = -EINVAL;
> +		goto error;
> +	};
> +
> +	cmd->opcode = opcode;
> +	cmd->sq_msg.hdr.opcode = FIELD_PREP(RM_CMD_SQ_HDR_OPS_MSK, opcode);
> +	cmd->sq_msg.hdr.msg_size = FIELD_PREP(RM_CMD_SQ_HDR_SIZE_MSK, size);
> +
> +	id = ida_alloc_range(&rm_cmd_ids, RM_CMD_ID_MIN, RM_CMD_ID_MAX, GFP_KERNEL);
> +	if (id < 0) {
> +		vmgmt_err(rdev->vdev, "Failed to alloc cmd ID: %d", id);
> +		ret = id;
> +		goto error;
> +	}
> +	cmd->sq_msg.hdr.id = id;
> +
> +	init_completion(&cmd->executed);
> +
> +	*cmd_ptr = cmd;
> +	return 0;
> +error:
> +	kfree(cmd);
> +	return ret;
> +}
> +
> +static int rm_queue_verify(struct rm_device *rdev)
> +{
> +	struct vmgmt_device *vdev = rdev->vdev;
> +	struct rm_cmd_cq_identify *result;
> +	struct rm_cmd *cmd;
> +	u32 major, minor;
> +	int ret;
> +
> +	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_IDENTIFY, &cmd);
> +	if (ret)
> +		return ret;
> +
> +	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
> +	if (ret)
> +		goto error;
> +
> +	result = &cmd->cq_msg.data.identify;
> +	major = result->major;
> +	minor = result->minor;
> +	vmgmt_dbg(vdev, "VMR version %d.%d", major, minor);
> +	if (!major) {
> +		vmgmt_err(vdev, "VMR version is unsupported");
> +		ret = -EOPNOTSUPP;
> +	}
> +
> +error:
> +	rm_queue_destory_cmd(cmd);
> +	return ret;
> +}
> +
> +static int rm_check_apu_status(struct rm_device *rdev, bool *status)
> +{
> +	struct rm_cmd_cq_control *result;
> +	struct rm_cmd *cmd;
> +	int ret;
> +
> +	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_VMR_CONTROL, &cmd);
> +	if (ret)
> +		return ret;
> +
> +	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
> +	if (ret)
> +		goto error;
> +
> +	result = &cmd->cq_msg.data.ctrl;
> +	*status = FIELD_GET(RM_CMD_VMR_CONTROL_PS_MASK, result->status);
> +
> +	rm_queue_destory_cmd(cmd);
> +	return 0;
> +
> +error:
> +	rm_queue_destory_cmd(cmd);
> +	return ret;
> +}
> +
> +static int rm_download_apu_fw(struct rm_device *rdev, char *data, ssize_t size)
> +{
> +	struct rm_cmd *cmd;
> +	int ret;
> +
> +	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_LOAD_APU_FW, &cmd);
> +	if (ret)
> +		return ret;
> +
> +	ret = rm_queue_data_init(cmd, data, size);
> +	if (ret)
> +		goto done;
> +
> +	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
> +
> +done:
> +	rm_queue_destory_cmd(cmd);
> +	return ret;
> +}
> +
> +int rm_boot_apu(struct rm_device *rdev)
> +{
> +	char *bin = "xilinx/xrt-versal-apu.xsabin";

So apu fw never changes, is it?

> +	const struct firmware *fw = NULL;
> +	bool status;
> +	int ret;
> +
> +	ret = rm_check_apu_status(rdev, &status);
> +	if (ret) {
> +		vmgmt_err(rdev->vdev, "Failed to get APU status");
> +		return ret;
> +	}
> +
> +	if (status) {
> +		vmgmt_dbg(rdev->vdev, "APU online. Skipping APU FW download");
> +		return 0;
> +	}
> +
> +	ret = request_firmware(&fw, bin, &rdev->vdev->pdev->dev);
> +	if (ret) {
> +		vmgmt_warn(rdev->vdev, "Request APU FW %s failed %d", bin, ret);
> +		return ret;
> +	}
> +
> +	vmgmt_dbg(rdev->vdev, "Starting... APU FW download");
> +	ret = rm_download_apu_fw(rdev, (char *)fw->data, fw->size);
> +	vmgmt_dbg(rdev->vdev, "Finished... APU FW download %d", ret);
> +
> +	if (ret)
> +		vmgmt_err(rdev->vdev, "Failed to download APU FW, ret:%d", ret);
> +
> +	release_firmware(fw);
> +
> +	return ret;
> +}
> +
> +static void rm_check_health(struct work_struct *w)
> +{
> +	struct rm_device *rdev = to_rdev_health_monitor(w);
> +	ssize_t len = PAGE_SIZE;
> +	char *buffer = NULL;
> +	struct rm_cmd *cmd;
> +	int ret;
> +
> +	buffer = vzalloc(len);
> +	if (!buffer)
> +		return;
> +
> +	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_GET_LOG_PAGE, &cmd);
> +	if (ret)
> +		return;

Memory Leak!

> +
> +	ret = rm_queue_payload_init(cmd, RM_CMD_LOG_PAGE_AXI_TRIP_STATUS);
> +	if (ret)
> +		goto error;
> +
> +	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
> +	if (ret == -ETIME || ret == -EINVAL)
> +		goto payload_fini;
> +
> +	if (cmd->cq_msg.data.page.len) {

If no data, the heath is good?

> +		ret = rm_queue_copy_response(cmd, buffer, len);
> +		if (ret)
> +			goto payload_fini;
> +
> +		vmgmt_err(rdev->vdev, "%s", buffer);

Any concern reading out of bound.

If the buffer is only used in this block, put its allocation/free here.

> +		rdev->firewall_tripped = 1;
> +	}
> +
> +	vfree(buffer);
> +
> +	rm_queue_payload_fini(cmd);
> +	rm_queue_destory_cmd(cmd);
> +
> +	return;
> +
> +payload_fini:
> +	rm_queue_payload_fini(cmd);
> +error:
> +	rm_queue_destory_cmd(cmd);
> +	vfree(buffer);
> +}
> +
> +static void rm_sched_health_check(struct timer_list *t)
> +{
> +	struct rm_device *rdev = to_rdev_health_timer(t);
> +
> +	if (rdev->firewall_tripped) {
> +		vmgmt_err(rdev->vdev, "Firewall tripped, health check paused. Please reset card");
> +		return;
> +	}
> +	/* Schedule a work in the general workqueue */
> +	schedule_work(&rdev->health_monitor);
> +	/* Periodic timer */
> +	mod_timer(&rdev->health_timer, jiffies + RM_HEALTH_CHECK_TIMER);

Again, is it necessary we endless poll. Coundn't we check on cmd create?

> +}
> +
> +static void rm_uninstall_health_monitor(struct rm_device *rdev)
> +{
> +	del_timer_sync(&rdev->health_timer);
> +	cancel_work_sync(&rdev->health_monitor);
> +}
> +
> +static void rm_install_health_monitor(struct rm_device *rdev)
> +{
> +	INIT_WORK(&rdev->health_monitor, &rm_check_health);
> +	timer_setup(&rdev->health_timer, &rm_sched_health_check, 0);
> +	mod_timer(&rdev->health_timer, jiffies + RM_HEALTH_CHECK_TIMER);
> +}
> +
> +void vmgmt_rm_fini(struct rm_device *rdev)
> +{
> +	rm_uninstall_health_monitor(rdev);
> +	rm_queue_fini(rdev);
> +}
> +
> +struct rm_device *vmgmt_rm_init(struct vmgmt_device *vdev)
> +{
> +	struct rm_header *header;
> +	struct rm_device *rdev;
> +	u32 status;
> +	int ret;
> +
> +	rdev = devm_kzalloc(&vdev->pdev->dev, sizeof(*rdev), GFP_KERNEL);
> +	if (!rdev)
> +		return ERR_PTR(-ENOMEM);
> +
> +	rdev->vdev = vdev;
> +	header = &rdev->rm_metadata;
> +
> +	rdev->shmem_regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
> +						   vdev->tbl + RM_PCI_SHMEM_BAR_OFF,
> +						   &rm_shmem_regmap_config);
> +	if (IS_ERR(rdev->shmem_regmap)) {
> +		vmgmt_err(vdev, "Failed to init RM shared memory regmap");
> +		return ERR_CAST(rdev->shmem_regmap);
> +	}
> +
> +	ret = rm_shmem_bulk_read(rdev, RM_HDR_OFF, (u32 *)header,
> +				 sizeof(*header));
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to read RM shared mem, ret %d", ret);
> +		ret = -ENODEV;
> +		goto err;
> +	}
> +
> +	if (header->magic != RM_HDR_MAGIC_NUM) {
> +		vmgmt_err(vdev, "Invalid RM header 0x%x", header->magic);
> +		ret = -ENODEV;
> +		goto err;
> +	}
> +
> +	ret = rm_shmem_read(rdev, header->status_off, &status);
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to read RM shared mem, ret %d", ret);
> +		ret = -ENODEV;
> +		goto err;
> +	}
> +
> +	if (!status) {
> +		vmgmt_err(vdev, "RM status %d is not ready", status);
> +		ret = -ENODEV;
> +		goto err;
> +	}
> +
> +	rdev->queue_buffer_size = header->data_end - header->data_start + 1;
> +	rdev->queue_buffer_start = header->data_start;
> +	rdev->queue_base = header->queue_base;
> +
> +	rdev->io_regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
> +						vdev->tbl + RM_PCI_IO_BAR_OFF,
> +						&rm_io_regmap_config);
> +	if (IS_ERR(rdev->io_regmap)) {
> +		vmgmt_err(vdev, "Failed to init RM IO regmap");
> +		ret = PTR_ERR(rdev->io_regmap);
> +		goto err;
> +	}
> +
> +	ret = rm_queue_init(rdev);
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to init cmd queue, ret %d", ret);
> +		ret = -ENODEV;
> +		goto err;
> +	}
> +
> +	ret = rm_queue_verify(rdev);
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to verify cmd queue, ret %d", ret);
> +		ret = -ENODEV;
> +		goto queue_fini;
> +	}
> +
> +	ret = rm_boot_apu(rdev);
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to bringup APU, ret %d", ret);
> +		ret = -ENODEV;
> +		goto queue_fini;
> +	}
> +
> +	rm_install_health_monitor(rdev);
> +
> +	return rdev;
> +queue_fini:
> +	rm_queue_fini(rdev);
> +err:
> +	return ERR_PTR(ret);
> +}
> +
> +int vmgmt_rm_get_fw_id(struct rm_device *rdev, uuid_t *uuid)
> +{
> +	char str[UUID_STRING_LEN];
> +	ssize_t len = PAGE_SIZE;
> +	char *buffer = NULL;
> +	struct rm_cmd *cmd;
> +	u8 i, j;
> +	int ret;
> +
> +	buffer = vmalloc(len);
> +	if (!buffer)
> +		return -ENOMEM;
> +
> +	memset(buffer, 0, len);

vzalloc()?

> +
> +	ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_GET_LOG_PAGE, &cmd);
> +	if (ret)
> +		return ret;
> +
> +	ret = rm_queue_payload_init(cmd, RM_CMD_LOG_PAGE_FW_ID);
> +	if (ret)
> +		goto error;
> +
> +	ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
> +	if (ret)
> +		goto payload;
> +
> +	ret = rm_queue_copy_response(cmd, buffer, len);
> +	if (ret)
> +		goto payload;
> +
> +	/* parse uuid into a valid uuid string format */
> +	for (i  = 0, j = 0; i < strlen(buffer); i++) {
> +		str[j++] = buffer[i];
> +		if (j == 8 || j == 13 || j == 18 || j == 23)
> +			str[j++] = '-';
> +	}
> +
> +	uuid_parse(str, uuid);
> +	vmgmt_dbg(rdev->vdev, "Interface uuid %pU", uuid);
> +
> +	vfree(buffer);
> +
> +	rm_queue_payload_fini(cmd);
> +	rm_queue_destory_cmd(cmd);
> +
> +	return 0;
> +
> +payload:
> +	rm_queue_payload_fini(cmd);
> +error:
> +	rm_queue_destory_cmd(cmd);
> +	vfree(buffer);
> +	return ret;
> +}
> diff --git a/drivers/fpga/amd/vmgmt-rm.h b/drivers/fpga/amd/vmgmt-rm.h
> new file mode 100644
> index 000000000000..a74f93cefbe8
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-rm.h
> @@ -0,0 +1,222 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#ifndef __VMGMT_RM_H
> +#define __VMGMT_RM_H
> +
> +#define RM_HDR_OFF			0x0
> +#define RM_HDR_MAGIC_NUM		0x564D5230
> +#define RM_QUEUE_HDR_MAGIC_NUM		0x5847513F
> +#define RM_PCI_IO_BAR_OFF		0x2010000
> +#define RM_PCI_IO_SIZE			0x1000
> +#define RM_PCI_SHMEM_BAR_OFF		0x8000000
> +#define RM_PCI_SHMEM_SIZE		0x8000000 /* 128 MB */
> +#define RM_PCI_SHMEM_HDR_SIZE		0x28
> +
> +#define RM_QUEUE_HDR_MAGIC_NUM_OFF	0x0
> +#define RM_IO_SQ_PIDX_OFF		0x0
> +#define RM_IO_CQ_PIDX_OFF		0x100
> +
> +#define RM_CMD_ID_MIN			1
> +#define RM_CMD_ID_MAX			(BIT(17) - 1)
> +#define RM_CMD_SQ_HDR_OPS_MSK		GENMASK(15, 0)
> +#define RM_CMD_SQ_HDR_SIZE_MSK		GENMASK(14, 0)
> +#define RM_CMD_SQ_SLOT_SIZE		512
> +#define RM_CMD_CQ_SLOT_SIZE		16
> +#define RM_CMD_CQ_BUFFER_SIZE		(1024 * 1024)
> +#define RM_CMD_CQ_BUFFER_OFFSET		0x0
> +#define RM_CMD_LOG_PAGE_TYPE_MASK	GENMASK(15, 0)
> +#define RM_CMD_VMR_CONTROL_MSK		GENMASK(10, 8)
> +#define RM_CMD_VMR_CONTROL_PS_MASK	BIT(9)
> +
> +#define RM_CMD_WAIT_CONFIG_TIMEOUT	msecs_to_jiffies(10 * 1000)
> +#define RM_CMD_WAIT_DOWNLOAD_TIMEOUT	msecs_to_jiffies(300 * 1000)
> +
> +#define RM_COMPLETION_TIMER		(HZ / 10)
> +#define RM_HEALTH_CHECK_TIMER		(HZ)
> +
> +#define RM_INVALID_SLOT			0
> +
> +enum rm_queue_opcode {
> +	RM_QUEUE_OP_LOAD_XCLBIN		= 0x0,
> +	RM_QUEUE_OP_GET_LOG_PAGE	= 0x8,
> +	RM_QUEUE_OP_LOAD_FW		= 0xA,
> +	RM_QUEUE_OP_LOAD_APU_FW		= 0xD,
> +	RM_QUEUE_OP_VMR_CONTROL		= 0xE,
> +	RM_QUEUE_OP_IDENTIFY		= 0x202,
> +};
> +
> +struct rm_cmd_sq_hdr {
> +	u16 opcode;
> +	u16 msg_size;
> +	u16 id;
> +	u16 reserved;
> +} __packed;
> +
> +struct rm_cmd_cq_hdr {
> +	u16 id;
> +	u16 reserved;
> +} __packed;
> +
> +struct rm_cmd_sq_bin {
> +	u64			address;
> +	u32			size;
> +	u32			reserved1;
> +	u32			reserved2;
> +	u32			reserved3;
> +	u64			reserved4;
> +} __packed;
> +
> +struct rm_cmd_sq_log_page {
> +	u64			address;
> +	u32			size;
> +	u32			reserved1;
> +	u32			type;
> +	u32			reserved2;
> +} __packed;
> +
> +struct rm_cmd_sq_ctrl {
> +	u32			status;
> +} __packed;
> +
> +struct rm_cmd_sq_data {
> +	union {
> +		struct rm_cmd_sq_log_page	page;
> +		struct rm_cmd_sq_bin		bin;
> +		struct rm_cmd_sq_ctrl		ctrl;
> +	};
> +} __packed;
> +
> +struct rm_cmd_cq_identify {
> +	u16			major;
> +	u16			minor;
> +	u32			reserved;
> +} __packed;
> +
> +struct rm_cmd_cq_log_page {
> +	u32			len;
> +	u32			reserved;
> +} __packed;
> +
> +struct rm_cmd_cq_control {
> +	u16			status;
> +	u16			reserved1;
> +	u32			reserved2;
> +} __packed;
> +
> +struct rm_cmd_cq_data {
> +	union {
> +		struct rm_cmd_cq_identify	identify;
> +		struct rm_cmd_cq_log_page	page;
> +		struct rm_cmd_cq_control	ctrl;
> +		u32				reserved[2];
> +	};
> +	u32			rcode;
> +} __packed;
> +
> +struct rm_cmd_sq_msg {
> +	struct rm_cmd_sq_hdr	hdr;
> +	struct rm_cmd_sq_data	data;
> +} __packed;
> +
> +struct rm_cmd_cq_msg {
> +	struct rm_cmd_cq_hdr	hdr;
> +	struct rm_cmd_cq_data	data;
> +} __packed;
> +
> +struct rm_cmd {
> +	struct rm_device	*rdev;
> +	struct list_head	list;
> +	struct completion	executed;
> +	struct rm_cmd_sq_msg	sq_msg;
> +	struct rm_cmd_cq_msg	cq_msg;
> +	enum rm_queue_opcode	opcode;
> +	void			*buffer;
> +	ssize_t			size;
> +};
> +
> +enum rm_queue_type {
> +	RM_QUEUE_SQ,
> +	RM_QUEUE_CQ
> +};
> +
> +enum rm_cmd_log_page_type {
> +	RM_CMD_LOG_PAGE_AXI_TRIP_STATUS	= 0x0,
> +	RM_CMD_LOG_PAGE_FW_ID		= 0xA,
> +};
> +
> +struct rm_queue {
> +	enum rm_queue_type	type;
> +	u32			pidx;
> +	u32			cidx;
> +	u32			offset;
> +	u32			data_offset;
> +	u32			data_size;
> +	struct semaphore	data_lock;
> +};
> +
> +struct rm_queue_header {
> +	u32			magic;
> +	u32			version;
> +	u32			size;
> +	u32			sq_off;
> +	u32			sq_slot_size;
> +	u32			cq_off;
> +	u32			sq_cidx;
> +	u32			cq_cidx;
> +};
> +
> +struct rm_header {
> +	u32			magic;
> +	u32			queue_base;
> +	u32			queue_size;
> +	u32			status_off;
> +	u32			status_len;
> +	u32			log_index;
> +	u32			log_off;
> +	u32			log_size;
> +	u32			data_start;
> +	u32			data_end;
> +};
> +
> +struct rm_device {
> +	struct vmgmt_device	*vdev;
> +	struct regmap		*shmem_regmap;
> +	struct regmap		*io_regmap;
> +
> +	struct rm_header	rm_metadata;
> +	u32			queue_buffer_start;
> +	u32			queue_buffer_size;
> +	u32			queue_base;
> +
> +	/* Lock to queue access */
> +	struct mutex		queue;
> +	struct rm_queue		sq;
> +	struct rm_queue		cq;
> +	u32			queue_size;
> +
> +	struct timer_list	msg_timer;
> +	struct work_struct	msg_monitor;
> +	struct timer_list	health_timer;
> +	struct work_struct	health_monitor;
> +	struct list_head	submitted_cmds;
> +
> +	int			firewall_tripped;
> +};
> +
> +int rm_queue_create_cmd(struct rm_device *rdev, enum rm_queue_opcode opcode,
> +			struct rm_cmd **cmd_ptr);
> +void rm_queue_destory_cmd(struct rm_cmd *cmd);
> +
> +int rm_queue_data_init(struct rm_cmd *cmd, const char *buffer, ssize_t size);
> +void rm_queue_data_fini(struct rm_cmd *cmd);
> +
> +int rm_queue_copy_response(struct rm_cmd *cmd, void *buffer, ssize_t len);
> +
> +int rm_boot_apu(struct rm_device *rdev);
> +
> +#endif	/* __VMGMT_RM_H */
> diff --git a/drivers/fpga/amd/vmgmt.c b/drivers/fpga/amd/vmgmt.c
> index b72eff9e8bc0..198213a13c7d 100644
> --- a/drivers/fpga/amd/vmgmt.c
> +++ b/drivers/fpga/amd/vmgmt.c
> @@ -21,6 +21,8 @@
>  
>  #include "vmgmt.h"
>  #include "vmgmt-comms.h"
> +#include "vmgmt-rm.h"
> +#include "vmgmt-rm-queue.h"
>  
>  #define DRV_NAME			"amd-vmgmt"
>  #define CLASS_NAME			DRV_NAME
> @@ -43,6 +45,61 @@ static inline struct vmgmt_device *vmgmt_inode_to_vdev(struct inode *inode)
>  	return (struct vmgmt_device *)container_of(inode->i_cdev, struct vmgmt_device, cdev);
>  }
>  
> +static int vmgmt_fpga_write_init(struct fpga_manager *mgr,
> +				 struct fpga_image_info *info, const char *buf,
> +				 size_t count)
> +{
> +	struct fpga_device *fdev = mgr->priv;
> +	struct fw_tnx *tnx = &fdev->fw;
> +	int ret;
> +
> +	ret = rm_queue_create_cmd(fdev->vdev->rdev, tnx->opcode, &tnx->cmd);
> +	if (ret) {
> +		fdev->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> +		return ret;
> +	}
> +
> +	fdev->state = FPGA_MGR_STATE_WRITE_INIT;
> +	return ret;
> +}
> +
> +static int vmgmt_fpga_write(struct fpga_manager *mgr, const char *buf,
> +			    size_t count)
> +{
> +	struct fpga_device *fdev = mgr->priv;
> +	int ret;
> +
> +	ret = rm_queue_data_init(fdev->fw.cmd, buf, count);
> +	if (ret) {
> +		fdev->state = FPGA_MGR_STATE_WRITE_ERR;
> +		rm_queue_destory_cmd(fdev->fw.cmd);
> +		return ret;
> +	}
> +
> +	fdev->state = FPGA_MGR_STATE_WRITE;
> +	return ret;
> +}
> +
> +static int vmgmt_fpga_write_complete(struct fpga_manager *mgr,
> +				     struct fpga_image_info *info)
> +{
> +	struct fpga_device *fdev = mgr->priv;
> +	int ret;
> +
> +	ret = rm_queue_send_cmd(fdev->fw.cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
> +	if (ret) {
> +		fdev->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
> +		vmgmt_err(fdev->vdev, "Send cmd failed:%d, cid:%d", ret, fdev->fw.id);
> +	} else {
> +		fdev->state = FPGA_MGR_STATE_WRITE_COMPLETE;
> +	}
> +
> +	rm_queue_data_fini(fdev->fw.cmd);
> +	rm_queue_destory_cmd(fdev->fw.cmd);
> +	memset(&fdev->fw, 0, sizeof(fdev->fw));
> +	return ret;
> +}
> +
>  static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)
>  {
>  	struct fpga_device *fdev = mgr->priv;
> @@ -51,6 +108,9 @@ static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)
>  }
>  
>  static const struct fpga_manager_ops vmgmt_fpga_ops = {
> +	.write_init = vmgmt_fpga_write_init,
> +	.write = vmgmt_fpga_write,
> +	.write_complete = vmgmt_fpga_write_complete,
>  	.state = vmgmt_fpga_state,
>  };
>  
> @@ -96,6 +156,13 @@ static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
>  		return ERR_PTR(ret);
>  	}
>  
> +	ret = vmgmt_rm_get_fw_id(vdev->rdev, &vdev->intf_uuid);
> +	if (ret) {
> +		vmgmt_warn(vdev, "Failed to get interface uuid");
> +		ret = -EINVAL;
> +		goto unregister_fpga_mgr;
> +	}
> +
>  	/* create fgpa bridge, region for the base shell */
>  	fdev->bridge = fpga_bridge_register(dev, "AMD Versal FPGA Bridge",
>  					    &vmgmt_br_ops, fdev);
> @@ -132,6 +199,149 @@ static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
>  	return ERR_PTR(ret);
>  }
>  
> +static int vmgmt_region_program(struct fpga_region *region, const void *data)
> +{
> +	struct fpga_device *fdev = region->priv;
> +	struct vmgmt_device *vdev = fdev->vdev;
> +	const struct axlf *xclbin = data;
> +	struct fpga_image_info *info;
> +	int ret;
> +
> +	info = fpga_image_info_alloc(&vdev->pdev->dev);
> +	if (!info)
> +		return -ENOMEM;
> +
> +	region->info = info;
> +
> +	info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
> +	info->count = xclbin->header.length;
> +	info->buf = (char *)xclbin;
> +
> +	ret = fpga_region_program_fpga(region);
> +	if (ret) {
> +		vmgmt_err(vdev, "Programming xclbin failed: %d", ret);
> +		goto exit;
> +	}
> +
> +	/* free bridges to allow reprogram */
> +	if (region->get_bridges)
> +		fpga_bridges_put(&region->bridge_list);
> +
> +exit:
> +	fpga_image_info_free(info);
> +	return ret;
> +}
> +
> +static int vmgmt_fpga_region_match(struct device *dev, const void *data)
> +{
> +	const struct vmgmt_fpga_region *arg = data;
> +	const struct fpga_region *match_region;
> +	struct fpga_device *fdev = arg->fdev;
> +	uuid_t compat_uuid;
> +
> +	if (dev->parent != &fdev->vdev->pdev->dev)
> +		return false;
> +
> +	match_region = to_fpga_region(dev);
> +
> +	import_uuid(&compat_uuid, (const char *)match_region->compat_id);
> +	if (uuid_equal(&compat_uuid, arg->uuid)) {
> +		vmgmt_dbg(fdev->vdev, "Region match found");
> +		return true;
> +	}
> +
> +	vmgmt_err(fdev->vdev, "download uuid %pUb is not the same as device uuid %pUb",
> +		  arg->uuid, &compat_uuid);
> +	return false;
> +}
> +
> +static long vmgmt_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
> +{
> +	struct vmgmt_device *vdev = (struct vmgmt_device *)filep->private_data;
> +	struct vmgmt_fpga_region reg = { 0 };
> +	struct fpga_region *region = NULL;
> +	struct axlf *axlf = NULL;
> +	void *data = NULL;
> +	size_t size = 0;
> +	int ret = 0;
> +
> +	axlf = vmalloc(sizeof(*axlf));
> +	if (!axlf)
> +		return -ENOMEM;
> +
> +	ret = copy_from_user((void *)axlf, (void *)arg, sizeof(*axlf));
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to copy axlf: %d", ret);
> +		ret = -EFAULT;
> +		goto exit;
> +	}
> +
> +	ret = memcmp(axlf->magic, VERSAL_XCLBIN_MAGIC_ID,
> +		     sizeof(VERSAL_XCLBIN_MAGIC_ID));
> +	if (ret) {
> +		vmgmt_err(vdev, "unknown axlf magic %s", axlf->magic);
> +		ret = -EINVAL;
> +		goto exit;
> +	}
> +
> +	/* axlf should never be over 1G and less than size of struct axlf */
> +	size = axlf->header.length;
> +	if (size < sizeof(struct axlf) || size > 1024 * 1024 * 1024) {
> +		vmgmt_err(vdev, "axlf length %zu is invalid", size);
> +		ret = -EINVAL;
> +		goto exit;
> +	}
> +
> +	data = vmalloc(size);
> +	if (!data) {
> +		ret = -ENOMEM;
> +		goto exit;
> +	}
> +
> +	ret = copy_from_user((void *)data, (void *)arg, size);
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to copy data: %d", ret);
> +		ret = -EFAULT;
> +		goto exit;
> +	}
> +
> +	switch (cmd) {
> +	case VERSAL_MGMT_LOAD_XCLBIN_IOCTL:
> +		vdev->fdev->fw.opcode = RM_QUEUE_OP_LOAD_XCLBIN;
> +		break;
> +	default:
> +		vmgmt_err(vdev, "Invalid IOCTL command: %d", cmd);
> +		ret = -EINVAL;
> +		goto exit;
> +	}
> +
> +	reg.uuid = &axlf->header.rom_uuid;
> +	reg.fdev = vdev->fdev;
> +
> +	region = fpga_region_class_find(NULL, &reg, vmgmt_fpga_region_match);
> +	if (!region) {
> +		vmgmt_err(vdev, "Failed to find compatible region");
> +		ret = -ENOENT;
> +		goto exit;
> +	}
> +
> +	ret = vmgmt_region_program(region, data);
> +	if (ret) {
> +		vmgmt_err(vdev, "Failed to program region");
> +		goto exit;
> +	}
> +
> +	vmgmt_info(vdev, "Downloaded axlf %pUb of size %zu Bytes",
> +		   &axlf->header.uuid, size);
> +	uuid_copy(&vdev->xclbin_uuid, &axlf->header.uuid);
> +
> +exit:
> +	vfree(data);
> +	vfree(axlf);
> +
> +	return ret;
> +}
> +
>  static int vmgmt_open(struct inode *inode, struct file *filep)
>  {
>  	struct vmgmt_device *vdev = vmgmt_inode_to_vdev(inode);
> @@ -155,6 +365,7 @@ static const struct file_operations vmgmt_fops = {
>  	.owner = THIS_MODULE,
>  	.open = vmgmt_open,
>  	.release = vmgmt_release,
> +	.unlocked_ioctl = vmgmt_ioctl,
>  };
>  
>  static void vmgmt_chrdev_destroy(struct vmgmt_device *vdev)
> @@ -201,6 +412,69 @@ static int vmgmt_chrdev_create(struct vmgmt_device *vdev)
>  	return 0;
>  }
>  
> +static enum fw_upload_err vmgmt_fw_prepare(struct fw_upload *fw_upload,
> +					   const u8 *data, u32 size)
> +{
> +	struct firmware_device *fwdev = fw_upload->dd_handle;
> +	struct axlf *xsabin = (struct axlf *)data;
> +	int ret;
> +
> +	ret = memcmp(xsabin->magic, VERSAL_XCLBIN_MAGIC_ID,
> +		     sizeof(VERSAL_XCLBIN_MAGIC_ID));
> +	if (ret) {
> +		vmgmt_err(fwdev->vdev, "Invalid device firmware");
> +		return FW_UPLOAD_ERR_INVALID_SIZE;
> +	}
> +
> +	/* Firmware size should never be over 1G and less than size of struct axlf */
> +	if (!size || size != xsabin->header.length || size < sizeof(*xsabin) ||
> +	    size > 1024 * 1024 * 1024) {
> +		vmgmt_err(fwdev->vdev, "Invalid device firmware size");
> +		return FW_UPLOAD_ERR_INVALID_SIZE;
> +	}
> +
> +	ret = rm_queue_create_cmd(fwdev->vdev->rdev, RM_QUEUE_OP_LOAD_FW,
> +				  &fwdev->cmd);
> +	if (ret)
> +		return FW_UPLOAD_ERR_RW_ERROR;
> +
> +	uuid_copy(&fwdev->uuid, &xsabin->header.uuid);
> +	return FW_UPLOAD_ERR_NONE;
> +}
> +
> +static enum fw_upload_err vmgmt_fw_write(struct fw_upload *fw_upload,
> +					 const u8 *data, u32 offset, u32 size,
> +					 u32 *written)
> +{
> +	struct firmware_device *fwdev = fw_upload->dd_handle;
> +	int ret;
> +
> +	ret = rm_queue_data_init(fwdev->cmd, data, size);
> +	if (ret)
> +		return FW_UPLOAD_ERR_RW_ERROR;
> +
> +	*written = size;
> +	return FW_UPLOAD_ERR_NONE;
> +}
> +
> +static enum fw_upload_err vmgmt_fw_poll_complete(struct fw_upload *fw_upload)
> +{
> +	struct firmware_device *fwdev = fw_upload->dd_handle;
> +	int ret;
> +
> +	vmgmt_info(fwdev->vdev, "Programming device firmware: %pUb", &fwdev->uuid);
> +
> +	ret = rm_queue_send_cmd(fwdev->cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
> +	if (ret) {
> +		vmgmt_err(fwdev->vdev, "Send cmd failedi:%d, cid %d", ret, fwdev->id);
> +		return FW_UPLOAD_ERR_HW_ERROR;
> +	}
> +

Basically I didn't see any difference on HW operations between FPGA
reprogram & firmware loading. So why use different SW interfaces?

My idea is FPGA reprogramming is not just for image loading, but also
device re-enumeration. As I mentioned before, programing the fpga region
without notifying the impacted driver makes kernel crash.

I see there is an effort to introduce a unified FPGA reprograming
interface that should address the concerns. I think we should stop
adding vendor interfaces and make effort on the unified one.

https://lore.kernel.org/linux-fpga/20240726063819.2274324-1-nava.kishore.manne@amd.com/

> +	vmgmt_info(fwdev->vdev, "Successfully programmed device firmware: %pUb",
> +		   &fwdev->uuid);
> +	return FW_UPLOAD_ERR_NONE;
> +}
> +
>  static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
>  {
>  	struct firmware_device *fwdev = fw_upload->dd_handle;
> @@ -208,8 +482,26 @@ static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
>  	vmgmt_warn(fwdev->vdev, "canceled");

Nothing to do?


I'll stop here, please better organize the patches next time submitting
to make reviewers easier, even if you know it is an RFC.

Thanks,
Yilun

RE: [PATCH V1 2/3] drivers/fpga/amd: Add communication with firmware

[Zhang, Yidong (David)]

[AMD Official Use Only - AMD Internal Distribution Only]

Hi Yilun,

Thanks for reviewing the patches for us. I will fix those with detailed info with next V2 patches.
My cover latter was sent with the other patches, but the second patch failed due to "D M K, Karthik" is not a valid email address, I just changed his name to "DMK" 😊.

I will address your comments with next patches set and send a clean patch.
Thank you so much for taking your time helping us on this.

Thanks,
David

-----Original Message-----
From: Xu Yilun <yilun.xu@linux.intel.com>
Sent: Friday, October 18, 2024 1:11 AM
To: Zhang, Yidong (David) <yidong.zhang@amd.com>
Cc: linux-kernel@vger.kernel.org; linux-fpga@vger.kernel.org; mdf@kernel.org; hao.wu@intel.com; yilun.xu@intel.com; Hou, Lizhi <lizhi.hou@amd.com>; Saraf, Nishad <nishad.saraf@amd.com>; Krishnamurthy, Prapul <Prapul.Krishnamurthy@amd.com>
Subject: Re: [PATCH V1 2/3] drivers/fpga/amd: Add communication with firmware

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


On Mon, Oct 07, 2024 at 03:01:27PM -0700, David Zhang wrote:
> From: Yidong Zhang <yidong.zhang@amd.com>
>
> Add queue based communication between host driver and firmware on the
> card. The remote queue (rm) can send/receive messages and providing

Abbrevate 'remote queue' to 'rm', or 'remote' to 'rm'. I see you use rm_queue in the code.

> firmware downloading services.
>
> Co-developed-by: Nishad Saraf <nishads@amd.com>
> Signed-off-by: Nishad Saraf <nishads@amd.com>
> Co-developed-by: Prapul Krishnamurthy <prapulk@amd.com>
> Signed-off-by: Prapul Krishnamurthy <prapulk@amd.com>
> Signed-off-by: Yidong Zhang <yidong.zhang@amd.com>
> ---
>  drivers/fpga/amd/Makefile         |   6 +-
>  drivers/fpga/amd/vmgmt-rm-queue.c |  38 +++
> drivers/fpga/amd/vmgmt-rm-queue.h |  15 +
>  drivers/fpga/amd/vmgmt-rm.c       | 543 ++++++++++++++++++++++++++++++
>  drivers/fpga/amd/vmgmt-rm.h       | 222 ++++++++++++
>  drivers/fpga/amd/vmgmt.c          | 305 ++++++++++++++++-
>  drivers/fpga/amd/vmgmt.h          |   7 +-
>  7 files changed, 1130 insertions(+), 6 deletions(-)  create mode
> 100644 drivers/fpga/amd/vmgmt-rm-queue.c  create mode 100644
> drivers/fpga/amd/vmgmt-rm-queue.h  create mode 100644
> drivers/fpga/amd/vmgmt-rm.c  create mode 100644
> drivers/fpga/amd/vmgmt-rm.h
>
> diff --git a/drivers/fpga/amd/Makefile b/drivers/fpga/amd/Makefile
> index 3e4c6dd3b787..97cfff6be204 100644
> --- a/drivers/fpga/amd/Makefile
> +++ b/drivers/fpga/amd/Makefile
> @@ -2,5 +2,7 @@
>
>  obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
>
> -amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)          := vmgmt.o      \
> -                                                vmgmt-comms.o
> +amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)          := vmgmt.o              \
> +                                                vmgmt-comms.o        \
> +                                                vmgmt-rm.o           \
> +                                                vmgmt-rm-queue.o
> diff --git a/drivers/fpga/amd/vmgmt-rm-queue.c
> b/drivers/fpga/amd/vmgmt-rm-queue.c
> new file mode 100644
> index 000000000000..fe805373ea32
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-rm-queue.c
> @@ -0,0 +1,38 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/completion.h>
> +#include <linux/err.h>
> +#include <linux/firmware.h>
> +#include <linux/idr.h>
> +#include <linux/jiffies.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/semaphore.h>
> +#include <linux/timer.h>
> +#include <linux/uuid.h>
> +#include <linux/workqueue.h>
> +
> +#include "vmgmt.h"
> +#include "vmgmt-rm.h"
> +#include "vmgmt-rm-queue.h"
> +
> +int rm_queue_send_cmd(struct rm_cmd *cmd, unsigned long timeout) {
> +     return 0;
> +}
> +
> +void rm_queue_fini(struct rm_device *rdev) { }
> +
> +int rm_queue_init(struct rm_device *rdev) {
> +     return 0;
> +}
> diff --git a/drivers/fpga/amd/vmgmt-rm-queue.h
> b/drivers/fpga/amd/vmgmt-rm-queue.h
> new file mode 100644
> index 000000000000..6fd0e0026a13
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-rm-queue.h
> @@ -0,0 +1,15 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#ifndef __VMGMT_RM_QUEUE_H
> +#define __VMGMT_RM_QUEUE_H
> +
> +int rm_queue_init(struct rm_device *rdev); void rm_queue_fini(struct
> +rm_device *rdev); int rm_queue_send_cmd(struct rm_cmd *cmd, unsigned
> +long timeout);
> +
> +#endif       /* __VMGMT_RM_QUEUE_H */
> diff --git a/drivers/fpga/amd/vmgmt-rm.c b/drivers/fpga/amd/vmgmt-rm.c
> new file mode 100644 index 000000000000..856d5af52c8d
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-rm.c
> @@ -0,0 +1,543 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/completion.h>
> +#include <linux/err.h>
> +#include <linux/firmware.h>
> +#include <linux/idr.h>
> +#include <linux/jiffies.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/semaphore.h>
> +#include <linux/timer.h>
> +#include <linux/uuid.h>
> +#include <linux/workqueue.h>
> +
> +#include "vmgmt.h"
> +#include "vmgmt-rm.h"
> +#include "vmgmt-rm-queue.h"
> +
> +static DEFINE_IDA(rm_cmd_ids);
> +
> +static const struct regmap_config rm_shmem_regmap_config = {
> +     .name = "rm_shmem_config",
> +     .reg_bits = 32,
> +     .reg_stride = 4,
> +     .val_bits = 32,
> +};
> +
> +static const struct regmap_config rm_io_regmap_config = {
> +     .name = "rm_io_config",
> +     .reg_bits = 32,
> +     .reg_stride = 4,
> +     .val_bits = 32,
> +};
> +
> +static void rm_uninstall_health_monitor(struct rm_device *rdev);
> +
> +static inline struct rm_device *to_rdev_health_monitor(struct
> +work_struct *w) {
> +     return container_of(w, struct rm_device, health_monitor); }
> +
> +static inline struct rm_device *to_rdev_health_timer(struct
> +timer_list *t) {
> +     return container_of(t, struct rm_device, health_timer); }
> +
> +static inline int rm_shmem_read(struct rm_device *rdev, u32 offset,
> +u32 *value) {
> +     return regmap_read(rdev->shmem_regmap, offset, value); }
> +
> +static inline int rm_shmem_bulk_read(struct rm_device *rdev, u32 offset,
> +                                  u32 *value, u32 size) {
> +     return regmap_bulk_read(rdev->shmem_regmap, offset, value,
> +                             DIV_ROUND_UP(size, 4)); }
> +
> +static inline int rm_shmem_bulk_write(struct rm_device *rdev, u32 offset,
> +                                   u32 *value, u32 size) {
> +     return regmap_bulk_write(rdev->shmem_regmap, offset, value,
> +                             DIV_ROUND_UP(size, 4)); }
> +
> +void rm_queue_destory_cmd(struct rm_cmd *cmd) {
> +     ida_free(&rm_cmd_ids, cmd->sq_msg.hdr.id);
> +     kfree(cmd);
> +}
> +
> +int rm_queue_copy_response(struct rm_cmd *cmd, void *buffer, ssize_t
> +len) {
> +     struct rm_cmd_cq_log_page *result = &cmd->cq_msg.data.page;
> +     u64 off = cmd->sq_msg.data.page.address;
> +
> +     if (!result->len || len < result->len) {
> +             vmgmt_err(cmd->rdev->vdev, "Invalid response or buffer size");
> +             return -EINVAL;
> +     }
> +
> +     return rm_shmem_bulk_read(cmd->rdev, off, (u32 *)buffer,
> +result->len); }
> +
> +static void rm_queue_payload_fini(struct rm_cmd *cmd) {
> +     up(&cmd->rdev->cq.data_lock);
> +}
> +
> +static int rm_queue_payload_init(struct rm_cmd *cmd,
> +                              enum rm_cmd_log_page_type type) {
> +     struct rm_device *rdev = cmd->rdev;
> +     int ret;
> +
> +     ret = down_interruptible(&rdev->cq.data_lock);
> +     if (ret)
> +             return ret;
> +
> +     cmd->sq_msg.data.page.address = rdev->cq.data_offset;
> +     cmd->sq_msg.data.page.size = rdev->cq.data_size;
> +     cmd->sq_msg.data.page.reserved1 = 0;
> +     cmd->sq_msg.data.page.type = FIELD_PREP(RM_CMD_LOG_PAGE_TYPE_MASK,
> +                                             type);
> +     return 0;
> +}
> +
> +void rm_queue_data_fini(struct rm_cmd *cmd) {
> +     up(&cmd->rdev->sq.data_lock);
> +}
> +
> +int rm_queue_data_init(struct rm_cmd *cmd, const char *buffer,
> +ssize_t size) {
> +     struct rm_device *rdev = cmd->rdev;
> +     int ret;
> +
> +     if (!size || size > rdev->sq.data_size) {
> +             vmgmt_err(rdev->vdev, "Unsupported file size");
> +             return -ENOMEM;
> +     }
> +
> +     ret = down_interruptible(&rdev->sq.data_lock);
> +     if (ret)
> +             return ret;
> +
> +     ret = rm_shmem_bulk_write(cmd->rdev, rdev->sq.data_offset,
> +                               (u32 *)buffer, size);
> +     if (ret) {
> +             vmgmt_err(rdev->vdev, "Failed to copy binary to SQ buffer");
> +             up(&cmd->rdev->sq.data_lock);
> +             return ret;
> +     }
> +
> +     cmd->sq_msg.data.bin.address = rdev->sq.data_offset;
> +     cmd->sq_msg.data.bin.size = size;
> +     return 0;
> +}
> +
> +int rm_queue_create_cmd(struct rm_device *rdev, enum rm_queue_opcode opcode,
> +                     struct rm_cmd **cmd_ptr) {
> +     struct rm_cmd *cmd = NULL;
> +     int ret, id;
> +     u16 size;
> +
> +     if (rdev->firewall_tripped)
> +             return -ENODEV;
> +
> +     cmd = kzalloc(sizeof(*cmd), GFP_KERNEL);
> +     if (!cmd)
> +             return -ENOMEM;
> +     cmd->rdev = rdev;
> +
> +     switch (opcode) {
> +     case RM_QUEUE_OP_LOAD_XCLBIN:
> +             fallthrough;
> +     case RM_QUEUE_OP_LOAD_FW:
> +             fallthrough;
> +     case RM_QUEUE_OP_LOAD_APU_FW:
> +             size = sizeof(struct rm_cmd_sq_bin);
> +             break;
> +     case RM_QUEUE_OP_GET_LOG_PAGE:
> +             size = sizeof(struct rm_cmd_sq_log_page);
> +             break;
> +     case RM_QUEUE_OP_IDENTIFY:
> +             size = 0;
> +             break;
> +     case RM_QUEUE_OP_VMR_CONTROL:
> +             size = sizeof(struct rm_cmd_sq_ctrl);
> +             break;
> +     default:
> +             vmgmt_err(rdev->vdev, "Invalid cmd opcode %d", opcode);
> +             ret = -EINVAL;
> +             goto error;
> +     };
> +
> +     cmd->opcode = opcode;
> +     cmd->sq_msg.hdr.opcode = FIELD_PREP(RM_CMD_SQ_HDR_OPS_MSK, opcode);
> +     cmd->sq_msg.hdr.msg_size = FIELD_PREP(RM_CMD_SQ_HDR_SIZE_MSK,
> + size);
> +
> +     id = ida_alloc_range(&rm_cmd_ids, RM_CMD_ID_MIN, RM_CMD_ID_MAX, GFP_KERNEL);
> +     if (id < 0) {
> +             vmgmt_err(rdev->vdev, "Failed to alloc cmd ID: %d", id);
> +             ret = id;
> +             goto error;
> +     }
> +     cmd->sq_msg.hdr.id = id;
> +
> +     init_completion(&cmd->executed);
> +
> +     *cmd_ptr = cmd;
> +     return 0;
> +error:
> +     kfree(cmd);
> +     return ret;
> +}
> +
> +static int rm_queue_verify(struct rm_device *rdev) {
> +     struct vmgmt_device *vdev = rdev->vdev;
> +     struct rm_cmd_cq_identify *result;
> +     struct rm_cmd *cmd;
> +     u32 major, minor;
> +     int ret;
> +
> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_IDENTIFY, &cmd);
> +     if (ret)
> +             return ret;
> +
> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
> +     if (ret)
> +             goto error;
> +
> +     result = &cmd->cq_msg.data.identify;
> +     major = result->major;
> +     minor = result->minor;
> +     vmgmt_dbg(vdev, "VMR version %d.%d", major, minor);
> +     if (!major) {
> +             vmgmt_err(vdev, "VMR version is unsupported");
> +             ret = -EOPNOTSUPP;
> +     }
> +
> +error:
> +     rm_queue_destory_cmd(cmd);
> +     return ret;
> +}
> +
> +static int rm_check_apu_status(struct rm_device *rdev, bool *status)
> +{
> +     struct rm_cmd_cq_control *result;
> +     struct rm_cmd *cmd;
> +     int ret;
> +
> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_VMR_CONTROL, &cmd);
> +     if (ret)
> +             return ret;
> +
> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
> +     if (ret)
> +             goto error;
> +
> +     result = &cmd->cq_msg.data.ctrl;
> +     *status = FIELD_GET(RM_CMD_VMR_CONTROL_PS_MASK, result->status);
> +
> +     rm_queue_destory_cmd(cmd);
> +     return 0;
> +
> +error:
> +     rm_queue_destory_cmd(cmd);
> +     return ret;
> +}
> +
> +static int rm_download_apu_fw(struct rm_device *rdev, char *data,
> +ssize_t size) {
> +     struct rm_cmd *cmd;
> +     int ret;
> +
> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_LOAD_APU_FW, &cmd);
> +     if (ret)
> +             return ret;
> +
> +     ret = rm_queue_data_init(cmd, data, size);
> +     if (ret)
> +             goto done;
> +
> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
> +
> +done:
> +     rm_queue_destory_cmd(cmd);
> +     return ret;
> +}
> +
> +int rm_boot_apu(struct rm_device *rdev) {
> +     char *bin = "xilinx/xrt-versal-apu.xsabin";

So apu fw never changes, is it?

> +     const struct firmware *fw = NULL;
> +     bool status;
> +     int ret;
> +
> +     ret = rm_check_apu_status(rdev, &status);
> +     if (ret) {
> +             vmgmt_err(rdev->vdev, "Failed to get APU status");
> +             return ret;
> +     }
> +
> +     if (status) {
> +             vmgmt_dbg(rdev->vdev, "APU online. Skipping APU FW download");
> +             return 0;
> +     }
> +
> +     ret = request_firmware(&fw, bin, &rdev->vdev->pdev->dev);
> +     if (ret) {
> +             vmgmt_warn(rdev->vdev, "Request APU FW %s failed %d", bin, ret);
> +             return ret;
> +     }
> +
> +     vmgmt_dbg(rdev->vdev, "Starting... APU FW download");
> +     ret = rm_download_apu_fw(rdev, (char *)fw->data, fw->size);
> +     vmgmt_dbg(rdev->vdev, "Finished... APU FW download %d", ret);
> +
> +     if (ret)
> +             vmgmt_err(rdev->vdev, "Failed to download APU FW,
> + ret:%d", ret);
> +
> +     release_firmware(fw);
> +
> +     return ret;
> +}
> +
> +static void rm_check_health(struct work_struct *w) {
> +     struct rm_device *rdev = to_rdev_health_monitor(w);
> +     ssize_t len = PAGE_SIZE;
> +     char *buffer = NULL;
> +     struct rm_cmd *cmd;
> +     int ret;
> +
> +     buffer = vzalloc(len);
> +     if (!buffer)
> +             return;
> +
> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_GET_LOG_PAGE, &cmd);
> +     if (ret)
> +             return;

Memory Leak!

> +
> +     ret = rm_queue_payload_init(cmd, RM_CMD_LOG_PAGE_AXI_TRIP_STATUS);
> +     if (ret)
> +             goto error;
> +
> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
> +     if (ret == -ETIME || ret == -EINVAL)
> +             goto payload_fini;
> +
> +     if (cmd->cq_msg.data.page.len) {

If no data, the heath is good?

> +             ret = rm_queue_copy_response(cmd, buffer, len);
> +             if (ret)
> +                     goto payload_fini;
> +
> +             vmgmt_err(rdev->vdev, "%s", buffer);

Any concern reading out of bound.

If the buffer is only used in this block, put its allocation/free here.

> +             rdev->firewall_tripped = 1;
> +     }
> +
> +     vfree(buffer);
> +
> +     rm_queue_payload_fini(cmd);
> +     rm_queue_destory_cmd(cmd);
> +
> +     return;
> +
> +payload_fini:
> +     rm_queue_payload_fini(cmd);
> +error:
> +     rm_queue_destory_cmd(cmd);
> +     vfree(buffer);
> +}
> +
> +static void rm_sched_health_check(struct timer_list *t) {
> +     struct rm_device *rdev = to_rdev_health_timer(t);
> +
> +     if (rdev->firewall_tripped) {
> +             vmgmt_err(rdev->vdev, "Firewall tripped, health check paused. Please reset card");
> +             return;
> +     }
> +     /* Schedule a work in the general workqueue */
> +     schedule_work(&rdev->health_monitor);
> +     /* Periodic timer */
> +     mod_timer(&rdev->health_timer, jiffies + RM_HEALTH_CHECK_TIMER);

Again, is it necessary we endless poll. Coundn't we check on cmd create?

> +}
> +
> +static void rm_uninstall_health_monitor(struct rm_device *rdev) {
> +     del_timer_sync(&rdev->health_timer);
> +     cancel_work_sync(&rdev->health_monitor);
> +}
> +
> +static void rm_install_health_monitor(struct rm_device *rdev) {
> +     INIT_WORK(&rdev->health_monitor, &rm_check_health);
> +     timer_setup(&rdev->health_timer, &rm_sched_health_check, 0);
> +     mod_timer(&rdev->health_timer, jiffies + RM_HEALTH_CHECK_TIMER);
> +}
> +
> +void vmgmt_rm_fini(struct rm_device *rdev) {
> +     rm_uninstall_health_monitor(rdev);
> +     rm_queue_fini(rdev);
> +}
> +
> +struct rm_device *vmgmt_rm_init(struct vmgmt_device *vdev) {
> +     struct rm_header *header;
> +     struct rm_device *rdev;
> +     u32 status;
> +     int ret;
> +
> +     rdev = devm_kzalloc(&vdev->pdev->dev, sizeof(*rdev), GFP_KERNEL);
> +     if (!rdev)
> +             return ERR_PTR(-ENOMEM);
> +
> +     rdev->vdev = vdev;
> +     header = &rdev->rm_metadata;
> +
> +     rdev->shmem_regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
> +                                                vdev->tbl + RM_PCI_SHMEM_BAR_OFF,
> +                                                &rm_shmem_regmap_config);
> +     if (IS_ERR(rdev->shmem_regmap)) {
> +             vmgmt_err(vdev, "Failed to init RM shared memory regmap");
> +             return ERR_CAST(rdev->shmem_regmap);
> +     }
> +
> +     ret = rm_shmem_bulk_read(rdev, RM_HDR_OFF, (u32 *)header,
> +                              sizeof(*header));
> +     if (ret) {
> +             vmgmt_err(vdev, "Failed to read RM shared mem, ret %d", ret);
> +             ret = -ENODEV;
> +             goto err;
> +     }
> +
> +     if (header->magic != RM_HDR_MAGIC_NUM) {
> +             vmgmt_err(vdev, "Invalid RM header 0x%x", header->magic);
> +             ret = -ENODEV;
> +             goto err;
> +     }
> +
> +     ret = rm_shmem_read(rdev, header->status_off, &status);
> +     if (ret) {
> +             vmgmt_err(vdev, "Failed to read RM shared mem, ret %d", ret);
> +             ret = -ENODEV;
> +             goto err;
> +     }
> +
> +     if (!status) {
> +             vmgmt_err(vdev, "RM status %d is not ready", status);
> +             ret = -ENODEV;
> +             goto err;
> +     }
> +
> +     rdev->queue_buffer_size = header->data_end - header->data_start + 1;
> +     rdev->queue_buffer_start = header->data_start;
> +     rdev->queue_base = header->queue_base;
> +
> +     rdev->io_regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
> +                                             vdev->tbl + RM_PCI_IO_BAR_OFF,
> +                                             &rm_io_regmap_config);
> +     if (IS_ERR(rdev->io_regmap)) {
> +             vmgmt_err(vdev, "Failed to init RM IO regmap");
> +             ret = PTR_ERR(rdev->io_regmap);
> +             goto err;
> +     }
> +
> +     ret = rm_queue_init(rdev);
> +     if (ret) {
> +             vmgmt_err(vdev, "Failed to init cmd queue, ret %d", ret);
> +             ret = -ENODEV;
> +             goto err;
> +     }
> +
> +     ret = rm_queue_verify(rdev);
> +     if (ret) {
> +             vmgmt_err(vdev, "Failed to verify cmd queue, ret %d", ret);
> +             ret = -ENODEV;
> +             goto queue_fini;
> +     }
> +
> +     ret = rm_boot_apu(rdev);
> +     if (ret) {
> +             vmgmt_err(vdev, "Failed to bringup APU, ret %d", ret);
> +             ret = -ENODEV;
> +             goto queue_fini;
> +     }
> +
> +     rm_install_health_monitor(rdev);
> +
> +     return rdev;
> +queue_fini:
> +     rm_queue_fini(rdev);
> +err:
> +     return ERR_PTR(ret);
> +}
> +
> +int vmgmt_rm_get_fw_id(struct rm_device *rdev, uuid_t *uuid) {
> +     char str[UUID_STRING_LEN];
> +     ssize_t len = PAGE_SIZE;
> +     char *buffer = NULL;
> +     struct rm_cmd *cmd;
> +     u8 i, j;
> +     int ret;
> +
> +     buffer = vmalloc(len);
> +     if (!buffer)
> +             return -ENOMEM;
> +
> +     memset(buffer, 0, len);

vzalloc()?

> +
> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_GET_LOG_PAGE, &cmd);
> +     if (ret)
> +             return ret;
> +
> +     ret = rm_queue_payload_init(cmd, RM_CMD_LOG_PAGE_FW_ID);
> +     if (ret)
> +             goto error;
> +
> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
> +     if (ret)
> +             goto payload;
> +
> +     ret = rm_queue_copy_response(cmd, buffer, len);
> +     if (ret)
> +             goto payload;
> +
> +     /* parse uuid into a valid uuid string format */
> +     for (i  = 0, j = 0; i < strlen(buffer); i++) {
> +             str[j++] = buffer[i];
> +             if (j == 8 || j == 13 || j == 18 || j == 23)
> +                     str[j++] = '-';
> +     }
> +
> +     uuid_parse(str, uuid);
> +     vmgmt_dbg(rdev->vdev, "Interface uuid %pU", uuid);
> +
> +     vfree(buffer);
> +
> +     rm_queue_payload_fini(cmd);
> +     rm_queue_destory_cmd(cmd);
> +
> +     return 0;
> +
> +payload:
> +     rm_queue_payload_fini(cmd);
> +error:
> +     rm_queue_destory_cmd(cmd);
> +     vfree(buffer);
> +     return ret;
> +}
> diff --git a/drivers/fpga/amd/vmgmt-rm.h b/drivers/fpga/amd/vmgmt-rm.h
> new file mode 100644 index 000000000000..a74f93cefbe8
> --- /dev/null
> +++ b/drivers/fpga/amd/vmgmt-rm.h
> @@ -0,0 +1,222 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for Versal PCIe device
> + *
> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
> + */
> +
> +#ifndef __VMGMT_RM_H
> +#define __VMGMT_RM_H
> +
> +#define RM_HDR_OFF                   0x0
> +#define RM_HDR_MAGIC_NUM             0x564D5230
> +#define RM_QUEUE_HDR_MAGIC_NUM               0x5847513F
> +#define RM_PCI_IO_BAR_OFF            0x2010000
> +#define RM_PCI_IO_SIZE                       0x1000
> +#define RM_PCI_SHMEM_BAR_OFF         0x8000000
> +#define RM_PCI_SHMEM_SIZE            0x8000000 /* 128 MB */
> +#define RM_PCI_SHMEM_HDR_SIZE                0x28
> +
> +#define RM_QUEUE_HDR_MAGIC_NUM_OFF   0x0
> +#define RM_IO_SQ_PIDX_OFF            0x0
> +#define RM_IO_CQ_PIDX_OFF            0x100
> +
> +#define RM_CMD_ID_MIN                        1
> +#define RM_CMD_ID_MAX                        (BIT(17) - 1)
> +#define RM_CMD_SQ_HDR_OPS_MSK                GENMASK(15, 0)
> +#define RM_CMD_SQ_HDR_SIZE_MSK               GENMASK(14, 0)
> +#define RM_CMD_SQ_SLOT_SIZE          512
> +#define RM_CMD_CQ_SLOT_SIZE          16
> +#define RM_CMD_CQ_BUFFER_SIZE                (1024 * 1024)
> +#define RM_CMD_CQ_BUFFER_OFFSET              0x0
> +#define RM_CMD_LOG_PAGE_TYPE_MASK    GENMASK(15, 0)
> +#define RM_CMD_VMR_CONTROL_MSK               GENMASK(10, 8)
> +#define RM_CMD_VMR_CONTROL_PS_MASK   BIT(9)
> +
> +#define RM_CMD_WAIT_CONFIG_TIMEOUT   msecs_to_jiffies(10 * 1000)
> +#define RM_CMD_WAIT_DOWNLOAD_TIMEOUT msecs_to_jiffies(300 * 1000)
> +
> +#define RM_COMPLETION_TIMER          (HZ / 10)
> +#define RM_HEALTH_CHECK_TIMER                (HZ)
> +
> +#define RM_INVALID_SLOT                      0
> +
> +enum rm_queue_opcode {
> +     RM_QUEUE_OP_LOAD_XCLBIN         = 0x0,
> +     RM_QUEUE_OP_GET_LOG_PAGE        = 0x8,
> +     RM_QUEUE_OP_LOAD_FW             = 0xA,
> +     RM_QUEUE_OP_LOAD_APU_FW         = 0xD,
> +     RM_QUEUE_OP_VMR_CONTROL         = 0xE,
> +     RM_QUEUE_OP_IDENTIFY            = 0x202,
> +};
> +
> +struct rm_cmd_sq_hdr {
> +     u16 opcode;
> +     u16 msg_size;
> +     u16 id;
> +     u16 reserved;
> +} __packed;
> +
> +struct rm_cmd_cq_hdr {
> +     u16 id;
> +     u16 reserved;
> +} __packed;
> +
> +struct rm_cmd_sq_bin {
> +     u64                     address;
> +     u32                     size;
> +     u32                     reserved1;
> +     u32                     reserved2;
> +     u32                     reserved3;
> +     u64                     reserved4;
> +} __packed;
> +
> +struct rm_cmd_sq_log_page {
> +     u64                     address;
> +     u32                     size;
> +     u32                     reserved1;
> +     u32                     type;
> +     u32                     reserved2;
> +} __packed;
> +
> +struct rm_cmd_sq_ctrl {
> +     u32                     status;
> +} __packed;
> +
> +struct rm_cmd_sq_data {
> +     union {
> +             struct rm_cmd_sq_log_page       page;
> +             struct rm_cmd_sq_bin            bin;
> +             struct rm_cmd_sq_ctrl           ctrl;
> +     };
> +} __packed;
> +
> +struct rm_cmd_cq_identify {
> +     u16                     major;
> +     u16                     minor;
> +     u32                     reserved;
> +} __packed;
> +
> +struct rm_cmd_cq_log_page {
> +     u32                     len;
> +     u32                     reserved;
> +} __packed;
> +
> +struct rm_cmd_cq_control {
> +     u16                     status;
> +     u16                     reserved1;
> +     u32                     reserved2;
> +} __packed;
> +
> +struct rm_cmd_cq_data {
> +     union {
> +             struct rm_cmd_cq_identify       identify;
> +             struct rm_cmd_cq_log_page       page;
> +             struct rm_cmd_cq_control        ctrl;
> +             u32                             reserved[2];
> +     };
> +     u32                     rcode;
> +} __packed;
> +
> +struct rm_cmd_sq_msg {
> +     struct rm_cmd_sq_hdr    hdr;
> +     struct rm_cmd_sq_data   data;
> +} __packed;
> +
> +struct rm_cmd_cq_msg {
> +     struct rm_cmd_cq_hdr    hdr;
> +     struct rm_cmd_cq_data   data;
> +} __packed;
> +
> +struct rm_cmd {
> +     struct rm_device        *rdev;
> +     struct list_head        list;
> +     struct completion       executed;
> +     struct rm_cmd_sq_msg    sq_msg;
> +     struct rm_cmd_cq_msg    cq_msg;
> +     enum rm_queue_opcode    opcode;
> +     void                    *buffer;
> +     ssize_t                 size;
> +};
> +
> +enum rm_queue_type {
> +     RM_QUEUE_SQ,
> +     RM_QUEUE_CQ
> +};
> +
> +enum rm_cmd_log_page_type {
> +     RM_CMD_LOG_PAGE_AXI_TRIP_STATUS = 0x0,
> +     RM_CMD_LOG_PAGE_FW_ID           = 0xA,
> +};
> +
> +struct rm_queue {
> +     enum rm_queue_type      type;
> +     u32                     pidx;
> +     u32                     cidx;
> +     u32                     offset;
> +     u32                     data_offset;
> +     u32                     data_size;
> +     struct semaphore        data_lock;
> +};
> +
> +struct rm_queue_header {
> +     u32                     magic;
> +     u32                     version;
> +     u32                     size;
> +     u32                     sq_off;
> +     u32                     sq_slot_size;
> +     u32                     cq_off;
> +     u32                     sq_cidx;
> +     u32                     cq_cidx;
> +};
> +
> +struct rm_header {
> +     u32                     magic;
> +     u32                     queue_base;
> +     u32                     queue_size;
> +     u32                     status_off;
> +     u32                     status_len;
> +     u32                     log_index;
> +     u32                     log_off;
> +     u32                     log_size;
> +     u32                     data_start;
> +     u32                     data_end;
> +};
> +
> +struct rm_device {
> +     struct vmgmt_device     *vdev;
> +     struct regmap           *shmem_regmap;
> +     struct regmap           *io_regmap;
> +
> +     struct rm_header        rm_metadata;
> +     u32                     queue_buffer_start;
> +     u32                     queue_buffer_size;
> +     u32                     queue_base;
> +
> +     /* Lock to queue access */
> +     struct mutex            queue;
> +     struct rm_queue         sq;
> +     struct rm_queue         cq;
> +     u32                     queue_size;
> +
> +     struct timer_list       msg_timer;
> +     struct work_struct      msg_monitor;
> +     struct timer_list       health_timer;
> +     struct work_struct      health_monitor;
> +     struct list_head        submitted_cmds;
> +
> +     int                     firewall_tripped;
> +};
> +
> +int rm_queue_create_cmd(struct rm_device *rdev, enum rm_queue_opcode opcode,
> +                     struct rm_cmd **cmd_ptr); void
> +rm_queue_destory_cmd(struct rm_cmd *cmd);
> +
> +int rm_queue_data_init(struct rm_cmd *cmd, const char *buffer,
> +ssize_t size); void rm_queue_data_fini(struct rm_cmd *cmd);
> +
> +int rm_queue_copy_response(struct rm_cmd *cmd, void *buffer, ssize_t
> +len);
> +
> +int rm_boot_apu(struct rm_device *rdev);
> +
> +#endif       /* __VMGMT_RM_H */
> diff --git a/drivers/fpga/amd/vmgmt.c b/drivers/fpga/amd/vmgmt.c index
> b72eff9e8bc0..198213a13c7d 100644
> --- a/drivers/fpga/amd/vmgmt.c
> +++ b/drivers/fpga/amd/vmgmt.c
> @@ -21,6 +21,8 @@
>
>  #include "vmgmt.h"
>  #include "vmgmt-comms.h"
> +#include "vmgmt-rm.h"
> +#include "vmgmt-rm-queue.h"
>
>  #define DRV_NAME                     "amd-vmgmt"
>  #define CLASS_NAME                   DRV_NAME
> @@ -43,6 +45,61 @@ static inline struct vmgmt_device *vmgmt_inode_to_vdev(struct inode *inode)
>       return (struct vmgmt_device *)container_of(inode->i_cdev, struct
> vmgmt_device, cdev);  }
>
> +static int vmgmt_fpga_write_init(struct fpga_manager *mgr,
> +                              struct fpga_image_info *info, const char *buf,
> +                              size_t count) {
> +     struct fpga_device *fdev = mgr->priv;
> +     struct fw_tnx *tnx = &fdev->fw;
> +     int ret;
> +
> +     ret = rm_queue_create_cmd(fdev->vdev->rdev, tnx->opcode, &tnx->cmd);
> +     if (ret) {
> +             fdev->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> +             return ret;
> +     }
> +
> +     fdev->state = FPGA_MGR_STATE_WRITE_INIT;
> +     return ret;
> +}
> +
> +static int vmgmt_fpga_write(struct fpga_manager *mgr, const char *buf,
> +                         size_t count) {
> +     struct fpga_device *fdev = mgr->priv;
> +     int ret;
> +
> +     ret = rm_queue_data_init(fdev->fw.cmd, buf, count);
> +     if (ret) {
> +             fdev->state = FPGA_MGR_STATE_WRITE_ERR;
> +             rm_queue_destory_cmd(fdev->fw.cmd);
> +             return ret;
> +     }
> +
> +     fdev->state = FPGA_MGR_STATE_WRITE;
> +     return ret;
> +}
> +
> +static int vmgmt_fpga_write_complete(struct fpga_manager *mgr,
> +                                  struct fpga_image_info *info) {
> +     struct fpga_device *fdev = mgr->priv;
> +     int ret;
> +
> +     ret = rm_queue_send_cmd(fdev->fw.cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
> +     if (ret) {
> +             fdev->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
> +             vmgmt_err(fdev->vdev, "Send cmd failed:%d, cid:%d", ret, fdev->fw.id);
> +     } else {
> +             fdev->state = FPGA_MGR_STATE_WRITE_COMPLETE;
> +     }
> +
> +     rm_queue_data_fini(fdev->fw.cmd);
> +     rm_queue_destory_cmd(fdev->fw.cmd);
> +     memset(&fdev->fw, 0, sizeof(fdev->fw));
> +     return ret;
> +}
> +
>  static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager
> *mgr)  {
>       struct fpga_device *fdev = mgr->priv; @@ -51,6 +108,9 @@ static
> enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)  }
>
>  static const struct fpga_manager_ops vmgmt_fpga_ops = {
> +     .write_init = vmgmt_fpga_write_init,
> +     .write = vmgmt_fpga_write,
> +     .write_complete = vmgmt_fpga_write_complete,
>       .state = vmgmt_fpga_state,
>  };
>
> @@ -96,6 +156,13 @@ static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
>               return ERR_PTR(ret);
>       }
>
> +     ret = vmgmt_rm_get_fw_id(vdev->rdev, &vdev->intf_uuid);
> +     if (ret) {
> +             vmgmt_warn(vdev, "Failed to get interface uuid");
> +             ret = -EINVAL;
> +             goto unregister_fpga_mgr;
> +     }
> +
>       /* create fgpa bridge, region for the base shell */
>       fdev->bridge = fpga_bridge_register(dev, "AMD Versal FPGA Bridge",
>                                           &vmgmt_br_ops, fdev); @@
> -132,6 +199,149 @@ static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
>       return ERR_PTR(ret);
>  }
>
> +static int vmgmt_region_program(struct fpga_region *region, const
> +void *data) {
> +     struct fpga_device *fdev = region->priv;
> +     struct vmgmt_device *vdev = fdev->vdev;
> +     const struct axlf *xclbin = data;
> +     struct fpga_image_info *info;
> +     int ret;
> +
> +     info = fpga_image_info_alloc(&vdev->pdev->dev);
> +     if (!info)
> +             return -ENOMEM;
> +
> +     region->info = info;
> +
> +     info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
> +     info->count = xclbin->header.length;
> +     info->buf = (char *)xclbin;
> +
> +     ret = fpga_region_program_fpga(region);
> +     if (ret) {
> +             vmgmt_err(vdev, "Programming xclbin failed: %d", ret);
> +             goto exit;
> +     }
> +
> +     /* free bridges to allow reprogram */
> +     if (region->get_bridges)
> +             fpga_bridges_put(&region->bridge_list);
> +
> +exit:
> +     fpga_image_info_free(info);
> +     return ret;
> +}
> +
> +static int vmgmt_fpga_region_match(struct device *dev, const void
> +*data) {
> +     const struct vmgmt_fpga_region *arg = data;
> +     const struct fpga_region *match_region;
> +     struct fpga_device *fdev = arg->fdev;
> +     uuid_t compat_uuid;
> +
> +     if (dev->parent != &fdev->vdev->pdev->dev)
> +             return false;
> +
> +     match_region = to_fpga_region(dev);
> +
> +     import_uuid(&compat_uuid, (const char *)match_region->compat_id);
> +     if (uuid_equal(&compat_uuid, arg->uuid)) {
> +             vmgmt_dbg(fdev->vdev, "Region match found");
> +             return true;
> +     }
> +
> +     vmgmt_err(fdev->vdev, "download uuid %pUb is not the same as device uuid %pUb",
> +               arg->uuid, &compat_uuid);
> +     return false;
> +}
> +
> +static long vmgmt_ioctl(struct file *filep, unsigned int cmd,
> +unsigned long arg) {
> +     struct vmgmt_device *vdev = (struct vmgmt_device *)filep->private_data;
> +     struct vmgmt_fpga_region reg = { 0 };
> +     struct fpga_region *region = NULL;
> +     struct axlf *axlf = NULL;
> +     void *data = NULL;
> +     size_t size = 0;
> +     int ret = 0;
> +
> +     axlf = vmalloc(sizeof(*axlf));
> +     if (!axlf)
> +             return -ENOMEM;
> +
> +     ret = copy_from_user((void *)axlf, (void *)arg, sizeof(*axlf));
> +     if (ret) {
> +             vmgmt_err(vdev, "Failed to copy axlf: %d", ret);
> +             ret = -EFAULT;
> +             goto exit;
> +     }
> +
> +     ret = memcmp(axlf->magic, VERSAL_XCLBIN_MAGIC_ID,
> +                  sizeof(VERSAL_XCLBIN_MAGIC_ID));
> +     if (ret) {
> +             vmgmt_err(vdev, "unknown axlf magic %s", axlf->magic);
> +             ret = -EINVAL;
> +             goto exit;
> +     }
> +
> +     /* axlf should never be over 1G and less than size of struct axlf */
> +     size = axlf->header.length;
> +     if (size < sizeof(struct axlf) || size > 1024 * 1024 * 1024) {
> +             vmgmt_err(vdev, "axlf length %zu is invalid", size);
> +             ret = -EINVAL;
> +             goto exit;
> +     }
> +
> +     data = vmalloc(size);
> +     if (!data) {
> +             ret = -ENOMEM;
> +             goto exit;
> +     }
> +
> +     ret = copy_from_user((void *)data, (void *)arg, size);
> +     if (ret) {
> +             vmgmt_err(vdev, "Failed to copy data: %d", ret);
> +             ret = -EFAULT;
> +             goto exit;
> +     }
> +
> +     switch (cmd) {
> +     case VERSAL_MGMT_LOAD_XCLBIN_IOCTL:
> +             vdev->fdev->fw.opcode = RM_QUEUE_OP_LOAD_XCLBIN;
> +             break;
> +     default:
> +             vmgmt_err(vdev, "Invalid IOCTL command: %d", cmd);
> +             ret = -EINVAL;
> +             goto exit;
> +     }
> +
> +     reg.uuid = &axlf->header.rom_uuid;
> +     reg.fdev = vdev->fdev;
> +
> +     region = fpga_region_class_find(NULL, &reg, vmgmt_fpga_region_match);
> +     if (!region) {
> +             vmgmt_err(vdev, "Failed to find compatible region");
> +             ret = -ENOENT;
> +             goto exit;
> +     }
> +
> +     ret = vmgmt_region_program(region, data);
> +     if (ret) {
> +             vmgmt_err(vdev, "Failed to program region");
> +             goto exit;
> +     }
> +
> +     vmgmt_info(vdev, "Downloaded axlf %pUb of size %zu Bytes",
> +                &axlf->header.uuid, size);
> +     uuid_copy(&vdev->xclbin_uuid, &axlf->header.uuid);
> +
> +exit:
> +     vfree(data);
> +     vfree(axlf);
> +
> +     return ret;
> +}
> +
>  static int vmgmt_open(struct inode *inode, struct file *filep)  {
>       struct vmgmt_device *vdev = vmgmt_inode_to_vdev(inode); @@
> -155,6 +365,7 @@ static const struct file_operations vmgmt_fops = {
>       .owner = THIS_MODULE,
>       .open = vmgmt_open,
>       .release = vmgmt_release,
> +     .unlocked_ioctl = vmgmt_ioctl,
>  };
>
>  static void vmgmt_chrdev_destroy(struct vmgmt_device *vdev) @@ -201,6
> +412,69 @@ static int vmgmt_chrdev_create(struct vmgmt_device *vdev)
>       return 0;
>  }
>
> +static enum fw_upload_err vmgmt_fw_prepare(struct fw_upload *fw_upload,
> +                                        const u8 *data, u32 size) {
> +     struct firmware_device *fwdev = fw_upload->dd_handle;
> +     struct axlf *xsabin = (struct axlf *)data;
> +     int ret;
> +
> +     ret = memcmp(xsabin->magic, VERSAL_XCLBIN_MAGIC_ID,
> +                  sizeof(VERSAL_XCLBIN_MAGIC_ID));
> +     if (ret) {
> +             vmgmt_err(fwdev->vdev, "Invalid device firmware");
> +             return FW_UPLOAD_ERR_INVALID_SIZE;
> +     }
> +
> +     /* Firmware size should never be over 1G and less than size of struct axlf */
> +     if (!size || size != xsabin->header.length || size < sizeof(*xsabin) ||
> +         size > 1024 * 1024 * 1024) {
> +             vmgmt_err(fwdev->vdev, "Invalid device firmware size");
> +             return FW_UPLOAD_ERR_INVALID_SIZE;
> +     }
> +
> +     ret = rm_queue_create_cmd(fwdev->vdev->rdev, RM_QUEUE_OP_LOAD_FW,
> +                               &fwdev->cmd);
> +     if (ret)
> +             return FW_UPLOAD_ERR_RW_ERROR;
> +
> +     uuid_copy(&fwdev->uuid, &xsabin->header.uuid);
> +     return FW_UPLOAD_ERR_NONE;
> +}
> +
> +static enum fw_upload_err vmgmt_fw_write(struct fw_upload *fw_upload,
> +                                      const u8 *data, u32 offset, u32 size,
> +                                      u32 *written) {
> +     struct firmware_device *fwdev = fw_upload->dd_handle;
> +     int ret;
> +
> +     ret = rm_queue_data_init(fwdev->cmd, data, size);
> +     if (ret)
> +             return FW_UPLOAD_ERR_RW_ERROR;
> +
> +     *written = size;
> +     return FW_UPLOAD_ERR_NONE;
> +}
> +
> +static enum fw_upload_err vmgmt_fw_poll_complete(struct fw_upload
> +*fw_upload) {
> +     struct firmware_device *fwdev = fw_upload->dd_handle;
> +     int ret;
> +
> +     vmgmt_info(fwdev->vdev, "Programming device firmware: %pUb",
> + &fwdev->uuid);
> +
> +     ret = rm_queue_send_cmd(fwdev->cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
> +     if (ret) {
> +             vmgmt_err(fwdev->vdev, "Send cmd failedi:%d, cid %d", ret, fwdev->id);
> +             return FW_UPLOAD_ERR_HW_ERROR;
> +     }
> +

Basically I didn't see any difference on HW operations between FPGA reprogram & firmware loading. So why use different SW interfaces?

My idea is FPGA reprogramming is not just for image loading, but also device re-enumeration. As I mentioned before, programing the fpga region without notifying the impacted driver makes kernel crash.

I see there is an effort to introduce a unified FPGA reprograming interface that should address the concerns. I think we should stop adding vendor interfaces and make effort on the unified one.

https://lore.kernel.org/linux-fpga/20240726063819.2274324-1-nava.kishore.manne@amd.com/

> +     vmgmt_info(fwdev->vdev, "Successfully programmed device firmware: %pUb",
> +                &fwdev->uuid);
> +     return FW_UPLOAD_ERR_NONE;
> +}
> +
>  static void vmgmt_fw_cancel(struct fw_upload *fw_upload)  {
>       struct firmware_device *fwdev = fw_upload->dd_handle; @@ -208,8
> +482,26 @@ static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
>       vmgmt_warn(fwdev->vdev, "canceled");

Nothing to do?


I'll stop here, please better organize the patches next time submitting to make reviewers easier, even if you know it is an RFC.

Thanks,
Yilun

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[Yidong Zhang]

On 10/17/24 23:17, Xu Yilun wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> 
> 
> On Mon, Oct 07, 2024 at 03:01:26PM -0700, David Zhang wrote:
>> From: Yidong Zhang <yidong.zhang@amd.com>
>>
>> AMD Versal based PCIe card, including V70, is designed for AI inference
>> efficiency and is tuned for video analytics and natural language processing
>> applications.
>>
>> Add the driver to support AMD Versal card management physical function.
>> Only very basic functionalities are added.
> 
> I think this is not "basic" enough.  If possible please add your following
> functionalities one by one.

Thank you so much for the review. I added more info for each functions 
below.

> 
>>    - module and PCI device initialization
>>    - fpga framework ops callbacks
>>    - communication with user physical function
> 
> So IIUC this is a multifunction PCI device? Management PF & User PF?
> Next time please add some description about the architecture overview
> of this card, as well as how the SW stack is supposed to make the card
> work.
>

I would like to add the following info by next V2 patch:

The driver architecture picture:

   +---------+  Communication +---------+  Remote  +-----+------+
   |         |  Channel       |         |  Queue   |     |      |
   | User PF | <============> | Mgmt PF | <=======>| FW  | FPGA |
   +---------+                +---------+          +-----+------+
     PL Data                    base FW
                                APU FW
                                PL Data (copy)
  - PL (FPGA Program Logic)
  - FW (Firmware)

There are 2 separate drivers from the original XRT[1] design.
  - UserPF driver
  - MgmtPF driver

The new amd-vpci driver will replace the MgmtPF driver for Versal PCIe card.

The XRT[1] is already open-sourced. It includes solution of runtime for 
many different type of PCIe Based cards. It also provides utilities for
managing and programming the devices.

The amd-vpci stands for AMD Versal brand PCIe device management driver.
This driver provides the following functionalities:

    - module and PCI device initialization
      this driver will attach to specific device id of V70 card;
      the driver will initialize itself based on bar resources for
      - communication channel:
        a hardware message service between mgmt PF and user PF
      - remote queue:
        a hardware queue based ring buffer service between mgmt PF and
        PCIe hardware firmware for programing FPGA Program Logic.

    - programing FWs
      - The base FW is downloaded onto flash of the card.
      - The APU FW is downloaded once after a POR (power on reset).
      - Reloading the MgmtPF driver will not change any existing hardware.

    - programing FPGA hardware binaries - PL Data
     - using fpga framework ops to support re-programing FPGA
     - the re-programing request will be initiated from the existing
       UserPF driver only, and the MgmtPF driver load the same PL Data 
after receiving request from the communication channel.

[1] https://github.com/Xilinx/XRT/blob/master/README.rst

>>
>> Co-developed-by: DMG Karthik <Karthik.DMG@amd.com>
>> Signed-off-by: DMG Karthik <Karthik.DMG@amd.com>
>> Co-developed-by: Nishad Saraf <nishads@amd.com>
>> Signed-off-by: Nishad Saraf <nishads@amd.com>
>> Co-developed-by: Prapul Krishnamurthy <prapulk@amd.com>
>> Signed-off-by: Prapul Krishnamurthy <prapulk@amd.com>
>> Signed-off-by: Yidong Zhang <yidong.zhang@amd.com>
>> ---
>>   MAINTAINERS                    |   7 +
>>   drivers/fpga/Kconfig           |   3 +
>>   drivers/fpga/Makefile          |   3 +
>>   drivers/fpga/amd/Kconfig       |  17 ++
>>   drivers/fpga/amd/Makefile      |   6 +
>>   drivers/fpga/amd/vmgmt-comms.c | 344 ++++++++++++++++++++++++++++
>>   drivers/fpga/amd/vmgmt-comms.h |  14 ++
>>   drivers/fpga/amd/vmgmt.c       | 395 +++++++++++++++++++++++++++++++++
>>   drivers/fpga/amd/vmgmt.h       | 100 +++++++++
>>   include/uapi/linux/vmgmt.h     |  25 +++
>>   10 files changed, 914 insertions(+)
>>   create mode 100644 drivers/fpga/amd/Kconfig
>>   create mode 100644 drivers/fpga/amd/Makefile
>>   create mode 100644 drivers/fpga/amd/vmgmt-comms.c
>>   create mode 100644 drivers/fpga/amd/vmgmt-comms.h
>>   create mode 100644 drivers/fpga/amd/vmgmt.c
>>   create mode 100644 drivers/fpga/amd/vmgmt.h
>>   create mode 100644 include/uapi/linux/vmgmt.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index a097afd76ded..645f00ccb342 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1185,6 +1185,13 @@ M:     Sanjay R Mehta <sanju.mehta@amd.com>
>>   S:   Maintained
>>   F:   drivers/spi/spi-amd.c
>>
>> +AMD VERSAL PCI DRIVER
>> +M:   Yidong Zhang <yidong.zhang@amd.com>
>> +L:   linux-fpga@vger.kernel.org
>> +S:   Supported
>> +F:   drivers/fpga/amd/
>> +F:   include/uapi/linux/vmgmt.h
>> +
>>   AMD XGBE DRIVER
>>   M:   "Shyam Sundar S K" <Shyam-sundar.S-k@amd.com>
>>   L:   netdev@vger.kernel.org
>> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
>> index 37b35f58f0df..dce060a7bd8f 100644
>> --- a/drivers/fpga/Kconfig
>> +++ b/drivers/fpga/Kconfig
>> @@ -290,4 +290,7 @@ config FPGA_MGR_LATTICE_SYSCONFIG_SPI
>>
>>   source "drivers/fpga/tests/Kconfig"
>>
>> +# Driver files
>> +source "drivers/fpga/amd/Kconfig"
>> +
>>   endif # FPGA
>> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
>> index aeb89bb13517..5e8a3869f9a0 100644
>> --- a/drivers/fpga/Makefile
>> +++ b/drivers/fpga/Makefile
>> @@ -58,5 +58,8 @@ obj-$(CONFIG_FPGA_DFL_NIOS_INTEL_PAC_N3000) += dfl-n3000-nios.o
>>   # Drivers for FPGAs which implement DFL
>>   obj-$(CONFIG_FPGA_DFL_PCI)           += dfl-pci.o
>>
>> +# AMD PCIe Versal Management Driver
>> +obj-y                                        += amd/
>> +
>>   # KUnit tests
>>   obj-$(CONFIG_FPGA_KUNIT_TESTS)               += tests/
>> diff --git a/drivers/fpga/amd/Kconfig b/drivers/fpga/amd/Kconfig
>> new file mode 100644
>> index 000000000000..126bc579a333
>> --- /dev/null
>> +++ b/drivers/fpga/amd/Kconfig
>> @@ -0,0 +1,17 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +
>> +config AMD_VERSAL_MGMT
>> +     tristate "AMD PCIe Versal Management Driver"
>> +     select FW_LOADER
>> +     select FW_UPLOAD
>> +     select REGMAP_MMIO
>> +     depends on FPGA_BRIDGE
>> +     depends on FPGA_REGION
>> +     depends on HAS_IOMEM
>> +     depends on PCI
>> +     help
>> +       AMD PCIe Versal Management Driver provides management services to
>> +       download firmware, program bitstream, collect sensor data, control
>> +       resets, and communicate with the User function.
>> +
>> +       If "M" is selected, the driver module will be amd-vmgmt.
>> diff --git a/drivers/fpga/amd/Makefile b/drivers/fpga/amd/Makefile
>> new file mode 100644
>> index 000000000000..3e4c6dd3b787
>> --- /dev/null
>> +++ b/drivers/fpga/amd/Makefile
>> @@ -0,0 +1,6 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +
>> +obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
> 
> IMHO the naming vmgmt is hard to understand, any better idea?

The "v" stand for Versal. We would change to amd-vpci for Versal based 
pcie devices.

> 
>> +
>> +amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)          := vmgmt.o      \
>> +                                                vmgmt-comms.o
>> diff --git a/drivers/fpga/amd/vmgmt-comms.c b/drivers/fpga/amd/vmgmt-comms.c
>> new file mode 100644
>> index 000000000000..bed0d369a744
>> --- /dev/null
>> +++ b/drivers/fpga/amd/vmgmt-comms.c
>> @@ -0,0 +1,344 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for Versal PCIe device
>> + *
>> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
>> + */
>> +
>> +#include <linux/bitfield.h>
>> +#include <linux/device.h>
>> +#include <linux/err.h>
>> +#include <linux/jiffies.h>
>> +#include <linux/mutex.h>
>> +#include <linux/pci.h>
>> +#include <linux/timer.h>
>> +#include <linux/uuid.h>
>> +#include <linux/workqueue.h>
>> +
>> +#include "vmgmt.h"
>> +#include "vmgmt-comms.h"
>> +
>> +#define COMMS_PROTOCOL_VERSION                       1
>> +#define COMMS_PCI_BAR_OFF                    0x2000000
>> +#define COMMS_TIMER                          (HZ / 10)
>> +#define COMMS_DATA_LEN                               16
>> +#define COMMS_DATA_TYPE_MASK                 GENMASK(7, 0)
>> +#define COMMS_DATA_EOM_MASK                  BIT(31)
>> +#define COMMS_MSG_END                                BIT(31)
>> +
>> +#define COMMS_REG_WRDATA_OFF                 0x0
>> +#define COMMS_REG_RDDATA_OFF                 0x8
>> +#define COMMS_REG_STATUS_OFF                 0x10
>> +#define COMMS_REG_ERROR_OFF                  0x14
>> +#define COMMS_REG_RIT_OFF                    0x1C
>> +#define COMMS_REG_IS_OFF                     0x20
>> +#define COMMS_REG_IE_OFF                     0x24
>> +#define COMMS_REG_CTRL_OFF                   0x2C
>> +#define COMMS_REGS_SIZE                              0x1000
>> +
>> +#define COMMS_IRQ_DISABLE_ALL                        0
>> +#define COMMS_IRQ_RECEIVE_ENABLE             BIT(1)
>> +#define COMMS_IRQ_CLEAR_ALL                  GENMASK(2, 0)
>> +#define COMMS_CLEAR_FIFO                     GENMASK(1, 0)
>> +#define COMMS_RECEIVE_THRESHOLD                      15
>> +
>> +enum comms_req_ops {
>> +     COMMS_REQ_OPS_UNKNOWN                   = 0,
>> +     COMMS_REQ_OPS_HOT_RESET                 = 5,
>> +     COMMS_REQ_OPS_GET_PROTOCOL_VERSION      = 19,
>> +     COMMS_REQ_OPS_GET_XCLBIN_UUID           = 20,
>> +     COMMS_REQ_OPS_MAX,
>> +};
>> +
>> +enum comms_msg_type {
>> +     COMMS_MSG_INVALID                       = 0,
>> +     COMMS_MSG_START                         = 2,
>> +     COMMS_MSG_BODY                          = 3,
>> +};
>> +
>> +enum comms_msg_service_type {
>> +     COMMS_MSG_SRV_RESPONSE                  = BIT(0),
>> +     COMMS_MSG_SRV_REQUEST                   = BIT(1),
>> +};
>> +
>> +struct comms_hw_msg {
>> +     struct {
>> +             u32             type;
>> +             u32             payload_size;
>> +     } header;
>> +     struct {
>> +             u64             id;
>> +             u32             flags;
>> +             u32             size;
>> +             u32             payload[COMMS_DATA_LEN - 6];
>> +     } body;
>> +} __packed;
>> +
>> +struct comms_srv_req {
>> +     u64                     flags;
>> +     u32                     opcode;
>> +     u32                     data[];
>> +};
>> +
>> +struct comms_srv_ver_resp {
>> +     u32                     version;
>> +};
>> +
>> +struct comms_srv_uuid_resp {
>> +     uuid_t                  uuid;
>> +};
>> +
>> +struct comms_msg {
>> +     u64                     id;
>> +     u32                     flags;
>> +     u32                     len;
>> +     u32                     bytes_read;
>> +     u32                     data[10];
>> +};
>> +
>> +struct comms_device {
>> +     struct vmgmt_device     *vdev;
>> +     struct regmap           *regmap;
>> +     struct timer_list       timer;
>> +     struct work_struct      work;
>> +};
>> +
>> +static bool comms_regmap_rd_regs(struct device *dev, unsigned int reg)
>> +{
>> +     switch (reg) {
>> +     case COMMS_REG_RDDATA_OFF:
>> +     case COMMS_REG_IS_OFF:
>> +             return true;
>> +     default:
>> +             return false;
>> +     }
>> +}
>> +
>> +static bool comms_regmap_wr_regs(struct device *dev, unsigned int reg)
>> +{
>> +     switch (reg) {
>> +     case COMMS_REG_WRDATA_OFF:
>> +     case COMMS_REG_IS_OFF:
>> +     case COMMS_REG_IE_OFF:
>> +     case COMMS_REG_CTRL_OFF:
>> +     case COMMS_REG_RIT_OFF:
>> +             return true;
>> +     default:
>> +             return false;
>> +     }
>> +}
>> +
>> +static bool comms_regmap_nir_regs(struct device *dev, unsigned int reg)
>> +{
>> +     switch (reg) {
>> +     case COMMS_REG_RDDATA_OFF:
>> +             return true;
>> +     default:
>> +             return false;
>> +     }
>> +}
>> +
>> +static const struct regmap_config comms_regmap_config = {
>> +     .name = "comms_config",
>> +     .reg_bits = 32,
>> +     .reg_stride = 4,
>> +     .val_bits = 32,
>> +     .readable_reg = comms_regmap_rd_regs,
>> +     .writeable_reg = comms_regmap_wr_regs,
>> +     .readable_noinc_reg = comms_regmap_nir_regs,
>> +};
>> +
>> +static inline struct comms_device *to_ccdev_work(struct work_struct *w)
>> +{
>> +     return container_of(w, struct comms_device, work);
>> +}
>> +
>> +static inline struct comms_device *to_ccdev_timer(struct timer_list *t)
>> +{
>> +     return container_of(t, struct comms_device, timer);
>> +}
>> +
>> +static u32 comms_set_uuid_resp(struct vmgmt_device *vdev, void *payload)
>> +{
>> +     struct comms_srv_uuid_resp *resp;
>> +     u32 resp_len = sizeof(*resp);
>> +
>> +     resp = (struct comms_srv_uuid_resp *)payload;
>> +     uuid_copy(&resp->uuid, &vdev->xclbin_uuid);
>> +     vmgmt_dbg(vdev, "xclbin UUID: %pUb", &resp->uuid);
>> +
>> +     return resp_len;
>> +}
>> +
>> +static u32 comms_set_protocol_resp(void *payload)
>> +{
>> +     struct comms_srv_ver_resp *resp = (struct comms_srv_ver_resp *)payload;
>> +     u32 resp_len = sizeof(*resp);
>> +
>> +     resp->version = COMMS_PROTOCOL_VERSION;
>> +
>> +     return sizeof(resp_len);
>> +}
>> +
>> +static void comms_send_response(struct comms_device *ccdev,
>> +                             struct comms_msg *msg)
>> +{
>> +     struct comms_srv_req *req = (struct comms_srv_req *)msg->data;
>> +     struct vmgmt_device *vdev = ccdev->vdev;
>> +     struct comms_hw_msg response = {0};
>> +     u32 size;
>> +     int ret;
>> +     u8 i;
>> +
>> +     switch (req->opcode) {
>> +     case COMMS_REQ_OPS_GET_PROTOCOL_VERSION:
>> +             size = comms_set_protocol_resp(response.body.payload);
>> +             break;
>> +     case COMMS_REQ_OPS_GET_XCLBIN_UUID:
>> +             size = comms_set_uuid_resp(vdev, response.body.payload);
>> +             break;
>> +     default:
>> +             vmgmt_err(vdev, "Unsupported request opcode: %d", req->opcode);
>> +             *response.body.payload = -1;
>> +             size = sizeof(int);
>> +     }
>> +
>> +     vmgmt_dbg(vdev, "Response opcode: %d", req->opcode);
>> +
>> +     response.header.type = COMMS_MSG_START | COMMS_MSG_END;
>> +     response.header.payload_size = size;
>> +
>> +     response.body.flags = COMMS_MSG_SRV_RESPONSE;
>> +     response.body.size = size;
>> +     response.body.id = msg->id;
>> +
>> +     for (i = 0; i < COMMS_DATA_LEN; i++) {
>> +             ret = regmap_write(ccdev->regmap, COMMS_REG_WRDATA_OFF, ((u32 *)&response)[i]);
>> +             if (ret < 0) {
>> +                     vmgmt_err(vdev, "regmap write failed: %d", ret);
>> +                     return;
>> +             }
>> +     }
>> +}
>> +
>> +#define STATUS_IS_READY(status) ((status) & BIT(1))
>> +#define STATUS_IS_ERROR(status) ((status) & BIT(2))
>> +
>> +static void comms_check_request(struct work_struct *w)
>> +{
>> +     struct comms_device *ccdev = to_ccdev_work(w);
>> +     u32 status = 0, request[COMMS_DATA_LEN] = {0};
>> +     struct comms_hw_msg *hw_msg;
>> +     struct comms_msg msg;
>> +     u8 type, eom;
>> +     int ret;
>> +     int i;
>> +
>> +     ret = regmap_read(ccdev->regmap, COMMS_REG_IS_OFF, &status);
>> +     if (ret) {
>> +             vmgmt_err(ccdev->vdev, "regmap read failed: %d", ret);
>> +             return;
>> +     }
>> +     if (!STATUS_IS_READY(status))
>> +             return;
>> +     if (STATUS_IS_ERROR(status)) {
>> +             vmgmt_err(ccdev->vdev, "An error has occurred with comms");
>> +             return;
>> +     }
>> +
>> +     /* ACK status */
>> +     regmap_write(ccdev->regmap, COMMS_REG_IS_OFF, status);
>> +
>> +     for (i = 0; i < COMMS_DATA_LEN; i++) {
>> +             if (regmap_read(ccdev->regmap, COMMS_REG_RDDATA_OFF, &request[i]) < 0) {
>> +                     vmgmt_err(ccdev->vdev, "regmap read failed");
>> +                     return;
>> +             }
>> +     }
>> +
>> +     hw_msg = (struct comms_hw_msg *)request;
>> +     type = FIELD_GET(COMMS_DATA_TYPE_MASK, hw_msg->header.type);
>> +     eom = FIELD_GET(COMMS_DATA_EOM_MASK, hw_msg->header.type);
>> +
>> +     /* Only support fixed size 64B messages */
>> +     if (!eom || type != COMMS_MSG_START) {
>> +             vmgmt_err(ccdev->vdev, "Unsupported message format or length");
>> +             return;
>> +     }
>> +
>> +     msg.flags = hw_msg->body.flags;
>> +     msg.len = hw_msg->body.size;
>> +     msg.id = hw_msg->body.id;
>> +
>> +     if (msg.flags != COMMS_MSG_SRV_REQUEST) {
>> +             vmgmt_err(ccdev->vdev, "Unsupported service request");
>> +             return;
>> +     }
>> +
>> +     if (hw_msg->body.size > sizeof(msg.data) * sizeof(msg.data[0])) {
>> +             vmgmt_err(ccdev->vdev, "msg is too big: %d", hw_msg->body.size);
>> +             return;
>> +     }
>> +     memcpy(msg.data, hw_msg->body.payload, hw_msg->body.size);
> 
> Why is the memcpy() necessary? I just see the data move from stack to
> stack, finally they will be released all.

Sure, I will fix this.

> 
>> +
>> +     /* Now decode and respond appropriately */
>> +     comms_send_response(ccdev, &msg);
>> +}
>> +
>> +static void comms_sched_work(struct timer_list *t)
>> +{
>> +     struct comms_device *ccdev = to_ccdev_timer(t);
>> +
>> +     /* Schedule a work in the general workqueue */
>> +     schedule_work(&ccdev->work);
>> +     /* Periodic timer */
>> +     mod_timer(&ccdev->timer, jiffies + COMMS_TIMER);
>> +}
>> +
>> +static void comms_config(struct comms_device *ccdev)
>> +{
>> +     /* Disable interrupts */
>> +     regmap_write(ccdev->regmap, COMMS_REG_IE_OFF, COMMS_IRQ_DISABLE_ALL);
>> +     /* Clear request and response FIFOs */
>> +     regmap_write(ccdev->regmap, COMMS_REG_CTRL_OFF, COMMS_CLEAR_FIFO);
>> +     /* Clear interrupts */
>> +     regmap_write(ccdev->regmap, COMMS_REG_IS_OFF, COMMS_IRQ_CLEAR_ALL);
>> +     /* Setup RIT reg */
>> +     regmap_write(ccdev->regmap, COMMS_REG_RIT_OFF, COMMS_RECEIVE_THRESHOLD);
>> +     /* Enable RIT interrupt */
>> +     regmap_write(ccdev->regmap, COMMS_REG_IE_OFF, COMMS_IRQ_RECEIVE_ENABLE);
>> +
>> +     /* Create and schedule timer to do recurring work */
>> +     INIT_WORK(&ccdev->work, &comms_check_request);
>> +     timer_setup(&ccdev->timer, &comms_sched_work, 0);
>> +     mod_timer(&ccdev->timer, jiffies + COMMS_TIMER);
> 
> Do we have to use raw timer+workqueue for a normal periodic task? Could
> delayed_work work for you?
> 
> And do we have to do endless periodic query? Couldn't the user PF driver
> trigger the service? Where is the user PF driver?

Current Versal based communication channel hardware does not support 
interrupt.

> 
>> +}
>> +
>> +void vmgmtm_comms_fini(struct comms_device *ccdev)
>> +{
>> +     /* First stop scheduling new work then cancel work */
>> +     del_timer_sync(&ccdev->timer);
>> +     cancel_work_sync(&ccdev->work);
>> +}
>> +
>> +struct comms_device *vmgmtm_comms_init(struct vmgmt_device *vdev)
> 
> So 'comms' means 'communication with user PF ', is it? I thought it was
> some common services at first, especially it is introduced in the first
> basic patch.
> 
> Any better name?

We can use comm_chan_* instead.

> 
>> +{
>> +     struct comms_device *ccdev;
>> +
>> +     ccdev = devm_kzalloc(&vdev->pdev->dev, sizeof(*ccdev), GFP_KERNEL);
>> +     if (!ccdev)
>> +             return ERR_PTR(-ENOMEM);
>> +
>> +     ccdev->vdev = vdev;
>> +
>> +     ccdev->regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
>> +                                           vdev->tbl + COMMS_PCI_BAR_OFF,
>> +                                           &comms_regmap_config);
> 
> I'm not sure why a regmap is needed. All register accessing is within
> the same module/file, and I assume a base+offset is enough to position
> the register addr.

I thought the regmap is preferred. We can use some common APIs like 
regmap_bulk_*. The base+offset works too, then we will implement our own 
bulk_* functions. Please let me know.

> 
>> +     if (IS_ERR(ccdev->regmap)) {
>> +             vmgmt_err(vdev, "Comms regmap init failed");
>> +             return ERR_CAST(ccdev->regmap);
>> +     }
>> +
>> +     comms_config(ccdev);
>> +     return ccdev;
>> +}
>> diff --git a/drivers/fpga/amd/vmgmt-comms.h b/drivers/fpga/amd/vmgmt-comms.h
>> new file mode 100644
>> index 000000000000..0afb14c8bd32
>> --- /dev/null
>> +++ b/drivers/fpga/amd/vmgmt-comms.h
>> @@ -0,0 +1,14 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Driver for Versal PCIe device
>> + *
>> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
>> + */
>> +
>> +#ifndef __VMGMT_COMMS_H
>> +#define __VMGMT_COMMS_H
>> +
>> +struct comms_device *vmgmtm_comms_init(struct vmgmt_device *vdev);
>> +void vmgmtm_comms_fini(struct comms_device *ccdev);
>> +
>> +#endif       /* __VMGMT_COMMS_H */
>> diff --git a/drivers/fpga/amd/vmgmt.c b/drivers/fpga/amd/vmgmt.c
>> new file mode 100644
>> index 000000000000..b72eff9e8bc0
>> --- /dev/null
>> +++ b/drivers/fpga/amd/vmgmt.c
>> @@ -0,0 +1,395 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for Versal PCIe device
>> + *
>> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
>> + */
>> +
>> +#include <linux/cdev.h>
>> +#include <linux/device/class.h>
>> +#include <linux/err.h>
>> +#include <linux/firmware.h>
>> +#include <linux/fs.h>
>> +#include <linux/fpga/fpga-mgr.h>
>> +#include <linux/idr.h>
>> +#include <linux/kdev_t.h>
>> +#include <linux/module.h>
>> +#include <linux/pci.h>
>> +#include <linux/types.h>
>> +#include <linux/uuid.h>
>> +#include <linux/vmgmt.h>
>> +
>> +#include "vmgmt.h"
>> +#include "vmgmt-comms.h"
>> +
>> +#define DRV_NAME                     "amd-vmgmt"
>> +#define CLASS_NAME                   DRV_NAME
>> +
>> +#define PCI_DEVICE_ID_V70PQ2         0x50B0
>> +#define VERSAL_XCLBIN_MAGIC_ID               "xclbin2"
>> +
>> +static DEFINE_IDA(vmgmt_dev_minor_ida);
>> +static dev_t vmgmt_devnode;
>> +struct class *vmgmt_class;
>> +static struct fpga_bridge_ops vmgmt_br_ops;
>> +
>> +struct vmgmt_fpga_region {
>> +     struct fpga_device *fdev;
>> +     uuid_t *uuid;
>> +};
>> +
>> +static inline struct vmgmt_device *vmgmt_inode_to_vdev(struct inode *inode)
>> +{
>> +     return (struct vmgmt_device *)container_of(inode->i_cdev, struct vmgmt_device, cdev);
>> +}
>> +
>> +static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)
>> +{
>> +     struct fpga_device *fdev = mgr->priv;
>> +
>> +     return fdev->state;
>> +}
>> +
>> +static const struct fpga_manager_ops vmgmt_fpga_ops = {
>> +     .state = vmgmt_fpga_state,
> 
> If you want to add a skeleton, then add all skeleton ops with no
> implementation. This makes me think the fpga_manager need .state() only.

Sure, I will fix this.

> 
>> +};
>> +
>> +static int vmgmt_get_bridges(struct fpga_region *region)
>> +{
>> +     struct fpga_device *fdev = region->priv;
>> +
>> +     return fpga_bridge_get_to_list(&fdev->vdev->pdev->dev, region->info,
>> +                                    &region->bridge_list);
>> +}
>> +
>> +static void vmgmt_fpga_fini(struct fpga_device *fdev)
>> +{
>> +     fpga_region_unregister(fdev->region);
>> +     fpga_bridge_unregister(fdev->bridge);
>> +     fpga_mgr_unregister(fdev->mgr);
>> +}
>> +
>> +static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
>> +{
>> +     struct device *dev = &vdev->pdev->dev;
>> +     struct fpga_region_info region = { 0 };
>> +     struct fpga_manager_info info = { 0 };
>> +     struct fpga_device *fdev;
>> +     int ret;
>> +
>> +     fdev = devm_kzalloc(dev, sizeof(*fdev), GFP_KERNEL);
>> +     if (!fdev)
>> +             return ERR_PTR(-ENOMEM);
>> +
>> +     fdev->vdev = vdev;
>> +
>> +     info = (struct fpga_manager_info) {
>> +             .name = "AMD Versal FPGA Manager",
>> +             .mops = &vmgmt_fpga_ops,
>> +             .priv = fdev,
>> +     };
>> +
>> +     fdev->mgr = fpga_mgr_register_full(dev, &info);
>> +     if (IS_ERR(fdev->mgr)) {
>> +             ret = PTR_ERR(fdev->mgr);
>> +             vmgmt_err(vdev, "Failed to register FPGA manager, err %d", ret);
>> +             return ERR_PTR(ret);
>> +     }
>> +
>> +     /* create fgpa bridge, region for the base shell */
>> +     fdev->bridge = fpga_bridge_register(dev, "AMD Versal FPGA Bridge",
>> +                                         &vmgmt_br_ops, fdev);
> 
> I didn't find the br_ops anywhere in this patchset. So how to gate the
> FPGA region when it is being reprogrammed? What is the physical link
> between the FPGA region and outside visitors?

The FPGA region gate operation is done in the FW running in this PCIe 
card. The FW will "freeze" the gate before programing the PL. After 
downloading the new hardware. The FW will then "free" the gate.

No physical link between FPGA region and outside visitors, the FW 
handles all requests.

> 
>> +     if (IS_ERR(fdev->bridge)) {
>> +             vmgmt_err(vdev, "Failed to register FPGA bridge, err %ld",
>> +                       PTR_ERR(fdev->bridge));
>> +             ret = PTR_ERR(fdev->bridge);
>> +             goto unregister_fpga_mgr;
>> +     }
>> +
>> +     region = (struct fpga_region_info) {
>> +             .compat_id = (struct fpga_compat_id *)&vdev->intf_uuid,
>> +             .get_bridges = vmgmt_get_bridges,
>> +             .mgr = fdev->mgr,
>> +             .priv = fdev,
>> +     };
>> +
>> +     fdev->region = fpga_region_register_full(dev, &region);
> 
> I assume the fpga region represents the user PF, is it? If you
> reprogram the FPGA region, how does the user PF driver aware the HW is
> changing?

The HW changing request is always requested from the user PF driver. The 
user PF driver will make sure it is safe to change hardware. Then, the 
user PF driver notify the mgmt PF driver by a unique identify of the HW 
bitstream (PL Data).

The mgmt PF driver, the amd-vpci driver, will check the unique identify 
and then find the same PL Data from its local storage which is 
previously installed, and start downloading it.

> 
>> +     if (IS_ERR(fdev->region)) {
>> +             vmgmt_err(vdev, "Failed to register FPGA region, err %ld",
>> +                       PTR_ERR(fdev->region));
>> +             ret = PTR_ERR(fdev->region);
>> +             goto unregister_fpga_bridge;
>> +     }
>> +
>> +     return fdev;
>> +
>> +unregister_fpga_bridge:
>> +     fpga_bridge_unregister(fdev->bridge);
>> +
>> +unregister_fpga_mgr:
>> +     fpga_mgr_unregister(fdev->mgr);
>> +
>> +     return ERR_PTR(ret);
>> +}
>> +
>> +static int vmgmt_open(struct inode *inode, struct file *filep)
>> +{
>> +     struct vmgmt_device *vdev = vmgmt_inode_to_vdev(inode);
>> +
>> +     if (WARN_ON(!vdev))
>> +             return -ENODEV;
>> +
>> +     filep->private_data = vdev;
>> +
>> +     return 0;
>> +}
>> +
>> +static int vmgmt_release(struct inode *inode, struct file *filep)
>> +{
>> +     filep->private_data = NULL;
>> +
>> +     return 0;
>> +}
>> +
>> +static const struct file_operations vmgmt_fops = {
>> +     .owner = THIS_MODULE,
>> +     .open = vmgmt_open,
>> +     .release = vmgmt_release,
>> +};
>> +
>> +static void vmgmt_chrdev_destroy(struct vmgmt_device *vdev)
>> +{
>> +     device_destroy(vmgmt_class, vdev->cdev.dev);
>> +     cdev_del(&vdev->cdev);
>> +     ida_free(&vmgmt_dev_minor_ida, vdev->minor);
>> +}
>> +
>> +static int vmgmt_chrdev_create(struct vmgmt_device *vdev)
>> +{
>> +     u32 devid;
>> +     int ret;
>> +
>> +     vdev->minor = ida_alloc(&vmgmt_dev_minor_ida, GFP_KERNEL);
>> +     if (vdev->minor < 0) {
>> +             vmgmt_err(vdev, "Failed to allocate chrdev ID");
>> +             return -ENODEV;
>> +     }
>> +
>> +     cdev_init(&vdev->cdev, &vmgmt_fops);
>> +
>> +     vdev->cdev.owner = THIS_MODULE;
>> +     vdev->cdev.dev = MKDEV(MAJOR(vmgmt_devnode), vdev->minor);
>> +     ret = cdev_add(&vdev->cdev, vdev->cdev.dev, 1);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to add char device: %d\n", ret);
>> +             ida_free(&vmgmt_dev_minor_ida, vdev->minor);
>> +             return -ENODEV;
>> +     }
>> +
>> +     devid = PCI_DEVID(vdev->pdev->bus->number, vdev->pdev->devfn);
>> +     vdev->device = device_create(vmgmt_class, &vdev->pdev->dev,
>> +                                  vdev->cdev.dev, NULL, "%s%x", DRV_NAME,
>> +                                  devid);
>> +     if (IS_ERR(vdev->device)) {
>> +             vmgmt_err(vdev, "Failed to create device: %ld\n",
>> +                       PTR_ERR(vdev->device));
>> +             cdev_del(&vdev->cdev);
>> +             ida_free(&vmgmt_dev_minor_ida, vdev->minor);
>> +             return -ENODEV;
>> +     }
>> +
>> +     return 0;
>> +}
>> +
>> +static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
>> +{
>> +     struct firmware_device *fwdev = fw_upload->dd_handle;
>> +
>> +     vmgmt_warn(fwdev->vdev, "canceled");
>> +}
>> +
>> +static const struct fw_upload_ops vmgmt_fw_ops = {
>> +     .cancel = vmgmt_fw_cancel,
> 
> Same concern.
> 

I will add all ops.

>> +};
>> +
>> +static void vmgmt_fw_upload_fini(struct firmware_device *fwdev)
>> +{
>> +     firmware_upload_unregister(fwdev->fw);
>> +     kfree(fwdev->name);
>> +}
>> +
>> +static struct firmware_device *vmgmt_fw_upload_init(struct vmgmt_device *vdev)
>> +{
>> +     struct device *dev = &vdev->pdev->dev;
>> +     struct firmware_device *fwdev;
>> +     u32 devid;
>> +
>> +     fwdev = devm_kzalloc(dev, sizeof(*fwdev), GFP_KERNEL);
>> +     if (!fwdev)
>> +             return ERR_PTR(-ENOMEM);
>> +
>> +     devid = PCI_DEVID(vdev->pdev->bus->number, vdev->pdev->devfn);
>> +     fwdev->name = kasprintf(GFP_KERNEL, "%s%x", DRV_NAME, devid);
>> +     if (!fwdev->name)
>> +             return ERR_PTR(-ENOMEM);
>> +
>> +     fwdev->fw = firmware_upload_register(THIS_MODULE, dev, fwdev->name,
>> +                                          &vmgmt_fw_ops, fwdev);
>> +     if (IS_ERR(fwdev->fw)) {
>> +             kfree(fwdev->name);
>> +             return ERR_CAST(fwdev->fw);
>> +     }
>> +
>> +     fwdev->vdev = vdev;
>> +
>> +     return fwdev;
>> +}
>> +
>> +static void vmgmt_device_teardown(struct vmgmt_device *vdev)
>> +{
>> +     vmgmt_fpga_fini(vdev->fdev);
>> +     vmgmt_fw_upload_fini(vdev->fwdev);
>> +     vmgmtm_comms_fini(vdev->ccdev);
>> +}
>> +
>> +static int vmgmt_device_setup(struct vmgmt_device *vdev)
>> +{
>> +     int ret;
>> +
>> +     vdev->fwdev = vmgmt_fw_upload_init(vdev);
>> +     if (IS_ERR(vdev->fwdev)) {
>> +             ret = PTR_ERR(vdev->fwdev);
>> +             vmgmt_err(vdev, "Failed to init FW uploader, err %d", ret);
>> +             goto done;
>> +     }
>> +
>> +     vdev->ccdev = vmgmtm_comms_init(vdev);
>> +     if (IS_ERR(vdev->ccdev)) {
>> +             ret = PTR_ERR(vdev->ccdev);
>> +             vmgmt_err(vdev, "Failed to init comms channel, err %d", ret);
>> +             goto upload_fini;
>> +     }
>> +
>> +     vdev->fdev = vmgmt_fpga_init(vdev);
>> +     if (IS_ERR(vdev->fdev)) {
>> +             ret = PTR_ERR(vdev->fdev);
>> +             vmgmt_err(vdev, "Failed to init FPGA maanger, err %d", ret);
>> +             goto comms_fini;
>> +     }
>> +
>> +     return 0;
>> +comms_fini:
>> +     vmgmtm_comms_fini(vdev->ccdev);
>> +upload_fini:
>> +     vmgmt_fw_upload_fini(vdev->fwdev);
>> +done:
>> +     return ret;
>> +}
>> +
>> +static void vmgmt_remove(struct pci_dev *pdev)
>> +{
>> +     struct vmgmt_device *vdev = pci_get_drvdata(pdev);
>> +
>> +     vmgmt_chrdev_destroy(vdev);
>> +     vmgmt_device_teardown(vdev);
>> +}
>> +
>> +static int vmgmt_probe(struct pci_dev *pdev,
>> +                    const struct pci_device_id *pdev_id)
>> +{
>> +     struct vmgmt_device *vdev;
>> +     int ret;
>> +
>> +     vdev = devm_kzalloc(&pdev->dev, sizeof(*vdev), GFP_KERNEL);
>> +     if (!vdev)
>> +             return -ENOMEM;
>> +
>> +     pci_set_drvdata(pdev, vdev);
>> +     vdev->pdev = pdev;
>> +
>> +     ret = pcim_enable_device(pdev);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to enable device %d", ret);
>> +             return ret;
>> +     }
>> +
>> +     ret = pcim_iomap_regions(vdev->pdev, AMD_VMGMT_BAR_MASK, "amd-vmgmt");
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed iomap regions %d", ret);
>> +             return -ENOMEM;
>> +     }
>> +
>> +     vdev->tbl = pcim_iomap_table(vdev->pdev)[AMD_VMGMT_BAR];
>> +     if (IS_ERR(vdev->tbl)) {
>> +             vmgmt_err(vdev, "Failed to map RM shared memory BAR%d", AMD_VMGMT_BAR);
>> +             return -ENOMEM;
>> +     }
> 
> Deprecating pcim_iomap_regions & pcim_iomap_table is WIP. FYI.
> 
>    https://lore.kernel.org/all/20241016094911.24818-5-pstanner@redhat.com/

Thanks! I will fix this.

> 
>> +
>> +     ret = vmgmt_device_setup(vdev);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to setup Versal device %d", ret);
>> +             return ret;
>> +     }
>> +
>> +     ret = vmgmt_chrdev_create(vdev);
>> +     if (ret) {
>> +             vmgmt_device_teardown(vdev);
>> +             return ret;
>> +     }
>> +
>> +     vmgmt_dbg(vdev, "Successfully probed %s driver!", DRV_NAME);
>> +     return 0;
>> +}
>> +
>> +static const struct pci_device_id vmgmt_pci_ids[] = {
>> +     { PCI_DEVICE(PCI_VENDOR_ID_XILINX, PCI_DEVICE_ID_V70PQ2), },
>> +     { 0 }
>> +};
>> +
>> +MODULE_DEVICE_TABLE(pci, vmgmt_pci_ids);
>> +
>> +static struct pci_driver amd_vmgmt_driver = {
>> +     .name = DRV_NAME,
>> +     .id_table = vmgmt_pci_ids,
>> +     .probe = vmgmt_probe,
>> +     .remove = vmgmt_remove,
>> +};
>> +
>> +static int amd_vmgmt_init(void)
>> +{
>> +     int ret;
>> +
>> +     vmgmt_class = class_create(CLASS_NAME);
>> +     if (IS_ERR(vmgmt_class))
>> +             return PTR_ERR(vmgmt_class);
>> +
>> +     ret = alloc_chrdev_region(&vmgmt_devnode, 0, MINORMASK, DRV_NAME);
>> +     if (ret)
>> +             goto chr_err;
>> +
>> +     ret = pci_register_driver(&amd_vmgmt_driver);
>> +     if (ret)
>> +             goto pci_err;
>> +
>> +     return 0;
>> +
>> +pci_err:
>> +     unregister_chrdev_region(vmgmt_devnode, MINORMASK);
>> +chr_err:
>> +     class_destroy(vmgmt_class);
>> +     return ret;
>> +}
>> +
>> +static void amd_vmgmt_exit(void)
>> +{
>> +     pci_unregister_driver(&amd_vmgmt_driver);
>> +     unregister_chrdev_region(vmgmt_devnode, MINORMASK);
>> +     class_destroy(vmgmt_class);
>> +}
>> +
>> +module_init(amd_vmgmt_init);
>> +module_exit(amd_vmgmt_exit);
>> +
>> +MODULE_DESCRIPTION("AMD PCIe Versal Management Driver");
>> +MODULE_AUTHOR("XRT Team <runtimeca39d@amd.com>");
>> +MODULE_LICENSE("GPL");
>> diff --git a/drivers/fpga/amd/vmgmt.h b/drivers/fpga/amd/vmgmt.h
>> new file mode 100644
>> index 000000000000..4dc8a43f825e
>> --- /dev/null
>> +++ b/drivers/fpga/amd/vmgmt.h
>> @@ -0,0 +1,100 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Driver for Versal PCIe device
>> + *
>> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
>> + */
>> +
>> +#ifndef __VMGMT_H
>> +#define __VMGMT_H
>> +
>> +#include <linux/cdev.h>
>> +#include <linux/dev_printk.h>
>> +#include <linux/jiffies.h>
>> +#include <linux/list.h>
>> +#include <linux/firmware.h>
>> +#include <linux/fpga/fpga-bridge.h>
>> +#include <linux/fpga/fpga-mgr.h>
>> +#include <linux/fpga/fpga-region.h>
>> +#include <linux/pci.h>
>> +#include <linux/uuid.h>
>> +#include <linux/regmap.h>
>> +
>> +#define AMD_VMGMT_BAR                        0
>> +#define AMD_VMGMT_BAR_MASK           BIT(0)
>> +
>> +#define vmgmt_info(vdev, fmt, args...)                                       \
>> +     dev_info(&(vdev)->pdev->dev, "%s: "fmt, __func__, ##args)
>> +
>> +#define vmgmt_warn(vdev, fmt, args...)                                       \
>> +     dev_warn(&(vdev)->pdev->dev, "%s: "fmt, __func__, ##args)
>> +
>> +#define vmgmt_err(vdev, fmt, args...)                                        \
>> +     dev_err(&(vdev)->pdev->dev, "%s: "fmt, __func__, ##args)
>> +
>> +#define vmgmt_dbg(vdev, fmt, args...)                                        \
>> +     dev_dbg(&(vdev)->pdev->dev, fmt, ##args)
>> +
>> +struct vmgmt_device;
>> +struct comms_device;
>> +struct rm_cmd;
>> +
>> +struct axlf_header {
>> +     u64                             length;
>> +     unsigned char                   reserved1[24];
>> +     uuid_t                          rom_uuid;
>> +     unsigned char                   reserved2[64];
>> +     uuid_t                          uuid;
>> +     unsigned char                   reserved3[24];
>> +} __packed;
>> +
>> +struct axlf {
>> +     char                            magic[8];
>> +     unsigned char                   reserved[296];
>> +     struct axlf_header              header;
>> +} __packed;
>> +
>> +struct fw_tnx {
>> +     struct rm_cmd           *cmd;
>> +     int                     opcode;
>> +     int                     id;
>> +};
>> +
>> +struct fpga_device {
>> +     enum fpga_mgr_states    state;
>> +     struct fpga_manager     *mgr;
>> +     struct fpga_bridge      *bridge;
>> +     struct fpga_region      *region;
>> +     struct vmgmt_device     *vdev;
>> +     struct fw_tnx           fw;
>> +};
>> +
>> +struct firmware_device {
>> +     struct vmgmt_device     *vdev;
>> +     struct fw_upload        *fw;
>> +     char                    *name;
>> +     u32                     fw_name_id;
>> +     struct rm_cmd           *cmd;
>> +     int                     id;
>> +     uuid_t                  uuid;
>> +};
>> +
>> +struct vmgmt_device {
>> +     struct pci_dev          *pdev;
>> +
>> +     struct rm_device        *rdev;
>> +     struct comms_device     *ccdev;
>> +     struct fpga_device      *fdev;
>> +     struct firmware_device  *fwdev;
>> +     struct cdev             cdev;
>> +     struct device           *device;
>> +
>> +     int                     minor;
>> +     void __iomem            *tbl;
>> +     uuid_t                  xclbin_uuid;
>> +     uuid_t                  intf_uuid;
>> +
>> +     void                    *debugfs_root;
>> +};
>> +
>> +#endif       /* __VMGMT_H */
>> diff --git a/include/uapi/linux/vmgmt.h b/include/uapi/linux/vmgmt.h
>> new file mode 100644
>> index 000000000000..2269ceb5c131
>> --- /dev/null
>> +++ b/include/uapi/linux/vmgmt.h
>> @@ -0,0 +1,25 @@
>> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>> +/*
>> + * Header file for Versal PCIe device user API
>> + *
>> + * Copyright (C) 2024 AMD Corporation, Inc.
>> + */
>> +
>> +#ifndef _UAPI_LINUX_VMGMT_H
>> +#define _UAPI_LINUX_VMGMT_H
>> +
>> +#include <linux/ioctl.h>
>> +
>> +#define VERSAL_MGMT_MAGIC    0xB7
>> +#define VERSAL_MGMT_BASE     0
>> +
>> +/**
>> + * VERSAL_MGMT_LOAD_XCLBIN_IOCTL - Download XCLBIN to the device
>> + *
>> + * This IOCTL is used to download XCLBIN down to the device.
>> + * Return: 0 on success, -errno on failure.
>> + */
>> +#define VERSAL_MGMT_LOAD_XCLBIN_IOCTL        _IOW(VERSAL_MGMT_MAGIC,         \
>> +                                          VERSAL_MGMT_BASE + 0, void *)
> 
> Many definitions are added in a batch but some are not used in this
> patch. Please reorganize the patches for easer review, even for first
> version.
> 
> Thanks,
> Yilun

Hi Yilun,

Thanks for taking your time, and yes for sure I will make each patch 
more self-contained.

Here is my thoughts on upcoming patches structure:
1st patch, adding driver probe and FPGA framework; the actual ops
of handling communication channel message and remote queue message
will present as no-op with comments.

2nd patch, adding the communication channel services
3rd patch, adding the remote queue services
4th patch, adding the callers of using the remote queue services

Thanks,
David

> 
>> +
>> +#endif /* _UAPI_LINUX_VMGMT_H */
>> --
>> 2.34.1
>>
>>

Re: [PATCH V1 2/3] drivers/fpga/amd: Add communication with firmware

[Yidong Zhang]

On 10/18/24 01:11, Xu Yilun wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> 
> 
> On Mon, Oct 07, 2024 at 03:01:27PM -0700, David Zhang wrote:
>> From: Yidong Zhang <yidong.zhang@amd.com>
>>
>> Add queue based communication between host driver and firmware on the
>> card. The remote queue (rm) can send/receive messages and providing
> 
> Abbrevate 'remote queue' to 'rm', or 'remote' to 'rm'. I see you use
> rm_queue in the code.

I will fix it.

> 
>> firmware downloading services.
>>
>> Co-developed-by: Nishad Saraf <nishads@amd.com>
>> Signed-off-by: Nishad Saraf <nishads@amd.com>
>> Co-developed-by: Prapul Krishnamurthy <prapulk@amd.com>
>> Signed-off-by: Prapul Krishnamurthy <prapulk@amd.com>
>> Signed-off-by: Yidong Zhang <yidong.zhang@amd.com>
>> ---
>>   drivers/fpga/amd/Makefile         |   6 +-
>>   drivers/fpga/amd/vmgmt-rm-queue.c |  38 +++
>>   drivers/fpga/amd/vmgmt-rm-queue.h |  15 +
>>   drivers/fpga/amd/vmgmt-rm.c       | 543 ++++++++++++++++++++++++++++++
>>   drivers/fpga/amd/vmgmt-rm.h       | 222 ++++++++++++
>>   drivers/fpga/amd/vmgmt.c          | 305 ++++++++++++++++-
>>   drivers/fpga/amd/vmgmt.h          |   7 +-
>>   7 files changed, 1130 insertions(+), 6 deletions(-)
>>   create mode 100644 drivers/fpga/amd/vmgmt-rm-queue.c
>>   create mode 100644 drivers/fpga/amd/vmgmt-rm-queue.h
>>   create mode 100644 drivers/fpga/amd/vmgmt-rm.c
>>   create mode 100644 drivers/fpga/amd/vmgmt-rm.h
>>
>> diff --git a/drivers/fpga/amd/Makefile b/drivers/fpga/amd/Makefile
>> index 3e4c6dd3b787..97cfff6be204 100644
>> --- a/drivers/fpga/amd/Makefile
>> +++ b/drivers/fpga/amd/Makefile
>> @@ -2,5 +2,7 @@
>>
>>   obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
>>
>> -amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)          := vmgmt.o      \
>> -                                                vmgmt-comms.o
>> +amd-vmgmt-$(CONFIG_AMD_VERSAL_MGMT)          := vmgmt.o              \
>> +                                                vmgmt-comms.o        \
>> +                                                vmgmt-rm.o           \
>> +                                                vmgmt-rm-queue.o
>> diff --git a/drivers/fpga/amd/vmgmt-rm-queue.c b/drivers/fpga/amd/vmgmt-rm-queue.c
>> new file mode 100644
>> index 000000000000..fe805373ea32
>> --- /dev/null
>> +++ b/drivers/fpga/amd/vmgmt-rm-queue.c
>> @@ -0,0 +1,38 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for Versal PCIe device
>> + *
>> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
>> + */
>> +
>> +#include <linux/bitfield.h>
>> +#include <linux/completion.h>
>> +#include <linux/err.h>
>> +#include <linux/firmware.h>
>> +#include <linux/idr.h>
>> +#include <linux/jiffies.h>
>> +#include <linux/list.h>
>> +#include <linux/mutex.h>
>> +#include <linux/pci.h>
>> +#include <linux/semaphore.h>
>> +#include <linux/timer.h>
>> +#include <linux/uuid.h>
>> +#include <linux/workqueue.h>
>> +
>> +#include "vmgmt.h"
>> +#include "vmgmt-rm.h"
>> +#include "vmgmt-rm-queue.h"
>> +
>> +int rm_queue_send_cmd(struct rm_cmd *cmd, unsigned long timeout)
>> +{
>> +     return 0;
>> +}
>> +
>> +void rm_queue_fini(struct rm_device *rdev)
>> +{
>> +}
>> +
>> +int rm_queue_init(struct rm_device *rdev)
>> +{
>> +     return 0;
>> +}
>> diff --git a/drivers/fpga/amd/vmgmt-rm-queue.h b/drivers/fpga/amd/vmgmt-rm-queue.h
>> new file mode 100644
>> index 000000000000..6fd0e0026a13
>> --- /dev/null
>> +++ b/drivers/fpga/amd/vmgmt-rm-queue.h
>> @@ -0,0 +1,15 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Driver for Versal PCIe device
>> + *
>> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
>> + */
>> +
>> +#ifndef __VMGMT_RM_QUEUE_H
>> +#define __VMGMT_RM_QUEUE_H
>> +
>> +int rm_queue_init(struct rm_device *rdev);
>> +void rm_queue_fini(struct rm_device *rdev);
>> +int rm_queue_send_cmd(struct rm_cmd *cmd, unsigned long timeout);
>> +
>> +#endif       /* __VMGMT_RM_QUEUE_H */
>> diff --git a/drivers/fpga/amd/vmgmt-rm.c b/drivers/fpga/amd/vmgmt-rm.c
>> new file mode 100644
>> index 000000000000..856d5af52c8d
>> --- /dev/null
>> +++ b/drivers/fpga/amd/vmgmt-rm.c
>> @@ -0,0 +1,543 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for Versal PCIe device
>> + *
>> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
>> + */
>> +
>> +#include <linux/bitfield.h>
>> +#include <linux/completion.h>
>> +#include <linux/err.h>
>> +#include <linux/firmware.h>
>> +#include <linux/idr.h>
>> +#include <linux/jiffies.h>
>> +#include <linux/list.h>
>> +#include <linux/mutex.h>
>> +#include <linux/pci.h>
>> +#include <linux/semaphore.h>
>> +#include <linux/timer.h>
>> +#include <linux/uuid.h>
>> +#include <linux/workqueue.h>
>> +
>> +#include "vmgmt.h"
>> +#include "vmgmt-rm.h"
>> +#include "vmgmt-rm-queue.h"
>> +
>> +static DEFINE_IDA(rm_cmd_ids);
>> +
>> +static const struct regmap_config rm_shmem_regmap_config = {
>> +     .name = "rm_shmem_config",
>> +     .reg_bits = 32,
>> +     .reg_stride = 4,
>> +     .val_bits = 32,
>> +};
>> +
>> +static const struct regmap_config rm_io_regmap_config = {
>> +     .name = "rm_io_config",
>> +     .reg_bits = 32,
>> +     .reg_stride = 4,
>> +     .val_bits = 32,
>> +};
>> +
>> +static void rm_uninstall_health_monitor(struct rm_device *rdev);
>> +
>> +static inline struct rm_device *to_rdev_health_monitor(struct work_struct *w)
>> +{
>> +     return container_of(w, struct rm_device, health_monitor);
>> +}
>> +
>> +static inline struct rm_device *to_rdev_health_timer(struct timer_list *t)
>> +{
>> +     return container_of(t, struct rm_device, health_timer);
>> +}
>> +
>> +static inline int rm_shmem_read(struct rm_device *rdev, u32 offset, u32 *value)
>> +{
>> +     return regmap_read(rdev->shmem_regmap, offset, value);
>> +}
>> +
>> +static inline int rm_shmem_bulk_read(struct rm_device *rdev, u32 offset,
>> +                                  u32 *value, u32 size)
>> +{
>> +     return regmap_bulk_read(rdev->shmem_regmap, offset, value,
>> +                             DIV_ROUND_UP(size, 4));
>> +}
>> +
>> +static inline int rm_shmem_bulk_write(struct rm_device *rdev, u32 offset,
>> +                                   u32 *value, u32 size)
>> +{
>> +     return regmap_bulk_write(rdev->shmem_regmap, offset, value,
>> +                             DIV_ROUND_UP(size, 4));
>> +}
>> +
>> +void rm_queue_destory_cmd(struct rm_cmd *cmd)
>> +{
>> +     ida_free(&rm_cmd_ids, cmd->sq_msg.hdr.id);
>> +     kfree(cmd);
>> +}
>> +
>> +int rm_queue_copy_response(struct rm_cmd *cmd, void *buffer, ssize_t len)
>> +{
>> +     struct rm_cmd_cq_log_page *result = &cmd->cq_msg.data.page;
>> +     u64 off = cmd->sq_msg.data.page.address;
>> +
>> +     if (!result->len || len < result->len) {
>> +             vmgmt_err(cmd->rdev->vdev, "Invalid response or buffer size");
>> +             return -EINVAL;
>> +     }
>> +
>> +     return rm_shmem_bulk_read(cmd->rdev, off, (u32 *)buffer, result->len);
>> +}
>> +
>> +static void rm_queue_payload_fini(struct rm_cmd *cmd)
>> +{
>> +     up(&cmd->rdev->cq.data_lock);
>> +}
>> +
>> +static int rm_queue_payload_init(struct rm_cmd *cmd,
>> +                              enum rm_cmd_log_page_type type)
>> +{
>> +     struct rm_device *rdev = cmd->rdev;
>> +     int ret;
>> +
>> +     ret = down_interruptible(&rdev->cq.data_lock);
>> +     if (ret)
>> +             return ret;
>> +
>> +     cmd->sq_msg.data.page.address = rdev->cq.data_offset;
>> +     cmd->sq_msg.data.page.size = rdev->cq.data_size;
>> +     cmd->sq_msg.data.page.reserved1 = 0;
>> +     cmd->sq_msg.data.page.type = FIELD_PREP(RM_CMD_LOG_PAGE_TYPE_MASK,
>> +                                             type);
>> +     return 0;
>> +}
>> +
>> +void rm_queue_data_fini(struct rm_cmd *cmd)
>> +{
>> +     up(&cmd->rdev->sq.data_lock);
>> +}
>> +
>> +int rm_queue_data_init(struct rm_cmd *cmd, const char *buffer, ssize_t size)
>> +{
>> +     struct rm_device *rdev = cmd->rdev;
>> +     int ret;
>> +
>> +     if (!size || size > rdev->sq.data_size) {
>> +             vmgmt_err(rdev->vdev, "Unsupported file size");
>> +             return -ENOMEM;
>> +     }
>> +
>> +     ret = down_interruptible(&rdev->sq.data_lock);
>> +     if (ret)
>> +             return ret;
>> +
>> +     ret = rm_shmem_bulk_write(cmd->rdev, rdev->sq.data_offset,
>> +                               (u32 *)buffer, size);
>> +     if (ret) {
>> +             vmgmt_err(rdev->vdev, "Failed to copy binary to SQ buffer");
>> +             up(&cmd->rdev->sq.data_lock);
>> +             return ret;
>> +     }
>> +
>> +     cmd->sq_msg.data.bin.address = rdev->sq.data_offset;
>> +     cmd->sq_msg.data.bin.size = size;
>> +     return 0;
>> +}
>> +
>> +int rm_queue_create_cmd(struct rm_device *rdev, enum rm_queue_opcode opcode,
>> +                     struct rm_cmd **cmd_ptr)
>> +{
>> +     struct rm_cmd *cmd = NULL;
>> +     int ret, id;
>> +     u16 size;
>> +
>> +     if (rdev->firewall_tripped)
>> +             return -ENODEV;
>> +
>> +     cmd = kzalloc(sizeof(*cmd), GFP_KERNEL);
>> +     if (!cmd)
>> +             return -ENOMEM;
>> +     cmd->rdev = rdev;
>> +
>> +     switch (opcode) {
>> +     case RM_QUEUE_OP_LOAD_XCLBIN:
>> +             fallthrough;
>> +     case RM_QUEUE_OP_LOAD_FW:
>> +             fallthrough;
>> +     case RM_QUEUE_OP_LOAD_APU_FW:
>> +             size = sizeof(struct rm_cmd_sq_bin);
>> +             break;
>> +     case RM_QUEUE_OP_GET_LOG_PAGE:
>> +             size = sizeof(struct rm_cmd_sq_log_page);
>> +             break;
>> +     case RM_QUEUE_OP_IDENTIFY:
>> +             size = 0;
>> +             break;
>> +     case RM_QUEUE_OP_VMR_CONTROL:
>> +             size = sizeof(struct rm_cmd_sq_ctrl);
>> +             break;
>> +     default:
>> +             vmgmt_err(rdev->vdev, "Invalid cmd opcode %d", opcode);
>> +             ret = -EINVAL;
>> +             goto error;
>> +     };
>> +
>> +     cmd->opcode = opcode;
>> +     cmd->sq_msg.hdr.opcode = FIELD_PREP(RM_CMD_SQ_HDR_OPS_MSK, opcode);
>> +     cmd->sq_msg.hdr.msg_size = FIELD_PREP(RM_CMD_SQ_HDR_SIZE_MSK, size);
>> +
>> +     id = ida_alloc_range(&rm_cmd_ids, RM_CMD_ID_MIN, RM_CMD_ID_MAX, GFP_KERNEL);
>> +     if (id < 0) {
>> +             vmgmt_err(rdev->vdev, "Failed to alloc cmd ID: %d", id);
>> +             ret = id;
>> +             goto error;
>> +     }
>> +     cmd->sq_msg.hdr.id = id;
>> +
>> +     init_completion(&cmd->executed);
>> +
>> +     *cmd_ptr = cmd;
>> +     return 0;
>> +error:
>> +     kfree(cmd);
>> +     return ret;
>> +}
>> +
>> +static int rm_queue_verify(struct rm_device *rdev)
>> +{
>> +     struct vmgmt_device *vdev = rdev->vdev;
>> +     struct rm_cmd_cq_identify *result;
>> +     struct rm_cmd *cmd;
>> +     u32 major, minor;
>> +     int ret;
>> +
>> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_IDENTIFY, &cmd);
>> +     if (ret)
>> +             return ret;
>> +
>> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
>> +     if (ret)
>> +             goto error;
>> +
>> +     result = &cmd->cq_msg.data.identify;
>> +     major = result->major;
>> +     minor = result->minor;
>> +     vmgmt_dbg(vdev, "VMR version %d.%d", major, minor);
>> +     if (!major) {
>> +             vmgmt_err(vdev, "VMR version is unsupported");
>> +             ret = -EOPNOTSUPP;
>> +     }
>> +
>> +error:
>> +     rm_queue_destory_cmd(cmd);
>> +     return ret;
>> +}
>> +
>> +static int rm_check_apu_status(struct rm_device *rdev, bool *status)
>> +{
>> +     struct rm_cmd_cq_control *result;
>> +     struct rm_cmd *cmd;
>> +     int ret;
>> +
>> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_VMR_CONTROL, &cmd);
>> +     if (ret)
>> +             return ret;
>> +
>> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
>> +     if (ret)
>> +             goto error;
>> +
>> +     result = &cmd->cq_msg.data.ctrl;
>> +     *status = FIELD_GET(RM_CMD_VMR_CONTROL_PS_MASK, result->status);
>> +
>> +     rm_queue_destory_cmd(cmd);
>> +     return 0;
>> +
>> +error:
>> +     rm_queue_destory_cmd(cmd);
>> +     return ret;
>> +}
>> +
>> +static int rm_download_apu_fw(struct rm_device *rdev, char *data, ssize_t size)
>> +{
>> +     struct rm_cmd *cmd;
>> +     int ret;
>> +
>> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_LOAD_APU_FW, &cmd);
>> +     if (ret)
>> +             return ret;
>> +
>> +     ret = rm_queue_data_init(cmd, data, size);
>> +     if (ret)
>> +             goto done;
>> +
>> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
>> +
>> +done:
>> +     rm_queue_destory_cmd(cmd);
>> +     return ret;
>> +}
>> +
>> +int rm_boot_apu(struct rm_device *rdev)
>> +{
>> +     char *bin = "xilinx/xrt-versal-apu.xsabin";
> 
> So apu fw never changes, is it?

The APU FW does not change when the card is in service.
It works with the base shell FW together.

The location of the "*bin" does not change too.
The binary xsabin can be changed by installing a deb pkg

> 
>> +     const struct firmware *fw = NULL;
>> +     bool status;
>> +     int ret;
>> +
>> +     ret = rm_check_apu_status(rdev, &status);
>> +     if (ret) {
>> +             vmgmt_err(rdev->vdev, "Failed to get APU status");
>> +             return ret;
>> +     }
>> +
>> +     if (status) {
>> +             vmgmt_dbg(rdev->vdev, "APU online. Skipping APU FW download");
>> +             return 0;
>> +     }
>> +
>> +     ret = request_firmware(&fw, bin, &rdev->vdev->pdev->dev);
>> +     if (ret) {
>> +             vmgmt_warn(rdev->vdev, "Request APU FW %s failed %d", bin, ret);
>> +             return ret;
>> +     }
>> +
>> +     vmgmt_dbg(rdev->vdev, "Starting... APU FW download");
>> +     ret = rm_download_apu_fw(rdev, (char *)fw->data, fw->size);
>> +     vmgmt_dbg(rdev->vdev, "Finished... APU FW download %d", ret);
>> +
>> +     if (ret)
>> +             vmgmt_err(rdev->vdev, "Failed to download APU FW, ret:%d", ret);
>> +
>> +     release_firmware(fw);
>> +
>> +     return ret;
>> +}
>> +
>> +static void rm_check_health(struct work_struct *w)
>> +{
>> +     struct rm_device *rdev = to_rdev_health_monitor(w);
>> +     ssize_t len = PAGE_SIZE;
>> +     char *buffer = NULL;
>> +     struct rm_cmd *cmd;
>> +     int ret;
>> +
>> +     buffer = vzalloc(len);
>> +     if (!buffer)
>> +             return;
>> +
>> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_GET_LOG_PAGE, &cmd);
>> +     if (ret)
>> +             return;
> 
> Memory Leak!

I will fix it.

> 
>> +
>> +     ret = rm_queue_payload_init(cmd, RM_CMD_LOG_PAGE_AXI_TRIP_STATUS);
>> +     if (ret)
>> +             goto error;
>> +
>> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
>> +     if (ret == -ETIME || ret == -EINVAL)
>> +             goto payload_fini;
>> +
>> +     if (cmd->cq_msg.data.page.len) {
> 
> If no data, the heath is good?

If not data and ret value is not ZERO, the health is not good.
The current FW guarrents that there will be always some data in this 
situation. I can add code like:
if (ret) {
   if (has data)
     printk(data);
   else
     printk(readable info based on ret value);
}

> 
>> +             ret = rm_queue_copy_response(cmd, buffer, len);
>> +             if (ret)
>> +                     goto payload_fini;
>> +
>> +             vmgmt_err(rdev->vdev, "%s", buffer);
> 
> Any concern reading out of bound.
> 
> If the buffer is only used in this block, put its allocation/free here.

Yes, I will move the buffer here.

> 
>> +             rdev->firewall_tripped = 1;
>> +     }
>> +
>> +     vfree(buffer);
>> +
>> +     rm_queue_payload_fini(cmd);
>> +     rm_queue_destory_cmd(cmd);
>> +
>> +     return;
>> +
>> +payload_fini:
>> +     rm_queue_payload_fini(cmd);
>> +error:
>> +     rm_queue_destory_cmd(cmd);
>> +     vfree(buffer);
>> +}
>> +
>> +static void rm_sched_health_check(struct timer_list *t)
>> +{
>> +     struct rm_device *rdev = to_rdev_health_timer(t);
>> +
>> +     if (rdev->firewall_tripped) {
>> +             vmgmt_err(rdev->vdev, "Firewall tripped, health check paused. Please reset card");
>> +             return;
>> +     }
>> +     /* Schedule a work in the general workqueue */
>> +     schedule_work(&rdev->health_monitor);
>> +     /* Periodic timer */
>> +     mod_timer(&rdev->health_timer, jiffies + RM_HEALTH_CHECK_TIMER);
> 
> Again, is it necessary we endless poll. Coundn't we check on cmd create?

The remote queue hardware only support polling mode.

> 
>> +}
>> +
>> +static void rm_uninstall_health_monitor(struct rm_device *rdev)
>> +{
>> +     del_timer_sync(&rdev->health_timer);
>> +     cancel_work_sync(&rdev->health_monitor);
>> +}
>> +
>> +static void rm_install_health_monitor(struct rm_device *rdev)
>> +{
>> +     INIT_WORK(&rdev->health_monitor, &rm_check_health);
>> +     timer_setup(&rdev->health_timer, &rm_sched_health_check, 0);
>> +     mod_timer(&rdev->health_timer, jiffies + RM_HEALTH_CHECK_TIMER);
>> +}
>> +
>> +void vmgmt_rm_fini(struct rm_device *rdev)
>> +{
>> +     rm_uninstall_health_monitor(rdev);
>> +     rm_queue_fini(rdev);
>> +}
>> +
>> +struct rm_device *vmgmt_rm_init(struct vmgmt_device *vdev)
>> +{
>> +     struct rm_header *header;
>> +     struct rm_device *rdev;
>> +     u32 status;
>> +     int ret;
>> +
>> +     rdev = devm_kzalloc(&vdev->pdev->dev, sizeof(*rdev), GFP_KERNEL);
>> +     if (!rdev)
>> +             return ERR_PTR(-ENOMEM);
>> +
>> +     rdev->vdev = vdev;
>> +     header = &rdev->rm_metadata;
>> +
>> +     rdev->shmem_regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
>> +                                                vdev->tbl + RM_PCI_SHMEM_BAR_OFF,
>> +                                                &rm_shmem_regmap_config);
>> +     if (IS_ERR(rdev->shmem_regmap)) {
>> +             vmgmt_err(vdev, "Failed to init RM shared memory regmap");
>> +             return ERR_CAST(rdev->shmem_regmap);
>> +     }
>> +
>> +     ret = rm_shmem_bulk_read(rdev, RM_HDR_OFF, (u32 *)header,
>> +                              sizeof(*header));
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to read RM shared mem, ret %d", ret);
>> +             ret = -ENODEV;
>> +             goto err;
>> +     }
>> +
>> +     if (header->magic != RM_HDR_MAGIC_NUM) {
>> +             vmgmt_err(vdev, "Invalid RM header 0x%x", header->magic);
>> +             ret = -ENODEV;
>> +             goto err;
>> +     }
>> +
>> +     ret = rm_shmem_read(rdev, header->status_off, &status);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to read RM shared mem, ret %d", ret);
>> +             ret = -ENODEV;
>> +             goto err;
>> +     }
>> +
>> +     if (!status) {
>> +             vmgmt_err(vdev, "RM status %d is not ready", status);
>> +             ret = -ENODEV;
>> +             goto err;
>> +     }
>> +
>> +     rdev->queue_buffer_size = header->data_end - header->data_start + 1;
>> +     rdev->queue_buffer_start = header->data_start;
>> +     rdev->queue_base = header->queue_base;
>> +
>> +     rdev->io_regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
>> +                                             vdev->tbl + RM_PCI_IO_BAR_OFF,
>> +                                             &rm_io_regmap_config);
>> +     if (IS_ERR(rdev->io_regmap)) {
>> +             vmgmt_err(vdev, "Failed to init RM IO regmap");
>> +             ret = PTR_ERR(rdev->io_regmap);
>> +             goto err;
>> +     }
>> +
>> +     ret = rm_queue_init(rdev);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to init cmd queue, ret %d", ret);
>> +             ret = -ENODEV;
>> +             goto err;
>> +     }
>> +
>> +     ret = rm_queue_verify(rdev);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to verify cmd queue, ret %d", ret);
>> +             ret = -ENODEV;
>> +             goto queue_fini;
>> +     }
>> +
>> +     ret = rm_boot_apu(rdev);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to bringup APU, ret %d", ret);
>> +             ret = -ENODEV;
>> +             goto queue_fini;
>> +     }
>> +
>> +     rm_install_health_monitor(rdev);
>> +
>> +     return rdev;
>> +queue_fini:
>> +     rm_queue_fini(rdev);
>> +err:
>> +     return ERR_PTR(ret);
>> +}
>> +
>> +int vmgmt_rm_get_fw_id(struct rm_device *rdev, uuid_t *uuid)
>> +{
>> +     char str[UUID_STRING_LEN];
>> +     ssize_t len = PAGE_SIZE;
>> +     char *buffer = NULL;
>> +     struct rm_cmd *cmd;
>> +     u8 i, j;
>> +     int ret;
>> +
>> +     buffer = vmalloc(len);
>> +     if (!buffer)
>> +             return -ENOMEM;
>> +
>> +     memset(buffer, 0, len);
> 
> vzalloc()?

Yes, I will fix it.

> 
>> +
>> +     ret = rm_queue_create_cmd(rdev, RM_QUEUE_OP_GET_LOG_PAGE, &cmd);
>> +     if (ret)
>> +             return ret;
>> +
>> +     ret = rm_queue_payload_init(cmd, RM_CMD_LOG_PAGE_FW_ID);
>> +     if (ret)
>> +             goto error;
>> +
>> +     ret = rm_queue_send_cmd(cmd, RM_CMD_WAIT_CONFIG_TIMEOUT);
>> +     if (ret)
>> +             goto payload;
>> +
>> +     ret = rm_queue_copy_response(cmd, buffer, len);
>> +     if (ret)
>> +             goto payload;
>> +
>> +     /* parse uuid into a valid uuid string format */
>> +     for (i  = 0, j = 0; i < strlen(buffer); i++) {
>> +             str[j++] = buffer[i];
>> +             if (j == 8 || j == 13 || j == 18 || j == 23)
>> +                     str[j++] = '-';
>> +     }
>> +
>> +     uuid_parse(str, uuid);
>> +     vmgmt_dbg(rdev->vdev, "Interface uuid %pU", uuid);
>> +
>> +     vfree(buffer);
>> +
>> +     rm_queue_payload_fini(cmd);
>> +     rm_queue_destory_cmd(cmd);
>> +
>> +     return 0;
>> +
>> +payload:
>> +     rm_queue_payload_fini(cmd);
>> +error:
>> +     rm_queue_destory_cmd(cmd);
>> +     vfree(buffer);
>> +     return ret;
>> +}
>> diff --git a/drivers/fpga/amd/vmgmt-rm.h b/drivers/fpga/amd/vmgmt-rm.h
>> new file mode 100644
>> index 000000000000..a74f93cefbe8
>> --- /dev/null
>> +++ b/drivers/fpga/amd/vmgmt-rm.h
>> @@ -0,0 +1,222 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Driver for Versal PCIe device
>> + *
>> + * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
>> + */
>> +
>> +#ifndef __VMGMT_RM_H
>> +#define __VMGMT_RM_H
>> +
>> +#define RM_HDR_OFF                   0x0
>> +#define RM_HDR_MAGIC_NUM             0x564D5230
>> +#define RM_QUEUE_HDR_MAGIC_NUM               0x5847513F
>> +#define RM_PCI_IO_BAR_OFF            0x2010000
>> +#define RM_PCI_IO_SIZE                       0x1000
>> +#define RM_PCI_SHMEM_BAR_OFF         0x8000000
>> +#define RM_PCI_SHMEM_SIZE            0x8000000 /* 128 MB */
>> +#define RM_PCI_SHMEM_HDR_SIZE                0x28
>> +
>> +#define RM_QUEUE_HDR_MAGIC_NUM_OFF   0x0
>> +#define RM_IO_SQ_PIDX_OFF            0x0
>> +#define RM_IO_CQ_PIDX_OFF            0x100
>> +
>> +#define RM_CMD_ID_MIN                        1
>> +#define RM_CMD_ID_MAX                        (BIT(17) - 1)
>> +#define RM_CMD_SQ_HDR_OPS_MSK                GENMASK(15, 0)
>> +#define RM_CMD_SQ_HDR_SIZE_MSK               GENMASK(14, 0)
>> +#define RM_CMD_SQ_SLOT_SIZE          512
>> +#define RM_CMD_CQ_SLOT_SIZE          16
>> +#define RM_CMD_CQ_BUFFER_SIZE                (1024 * 1024)
>> +#define RM_CMD_CQ_BUFFER_OFFSET              0x0
>> +#define RM_CMD_LOG_PAGE_TYPE_MASK    GENMASK(15, 0)
>> +#define RM_CMD_VMR_CONTROL_MSK               GENMASK(10, 8)
>> +#define RM_CMD_VMR_CONTROL_PS_MASK   BIT(9)
>> +
>> +#define RM_CMD_WAIT_CONFIG_TIMEOUT   msecs_to_jiffies(10 * 1000)
>> +#define RM_CMD_WAIT_DOWNLOAD_TIMEOUT msecs_to_jiffies(300 * 1000)
>> +
>> +#define RM_COMPLETION_TIMER          (HZ / 10)
>> +#define RM_HEALTH_CHECK_TIMER                (HZ)
>> +
>> +#define RM_INVALID_SLOT                      0
>> +
>> +enum rm_queue_opcode {
>> +     RM_QUEUE_OP_LOAD_XCLBIN         = 0x0,
>> +     RM_QUEUE_OP_GET_LOG_PAGE        = 0x8,
>> +     RM_QUEUE_OP_LOAD_FW             = 0xA,
>> +     RM_QUEUE_OP_LOAD_APU_FW         = 0xD,
>> +     RM_QUEUE_OP_VMR_CONTROL         = 0xE,
>> +     RM_QUEUE_OP_IDENTIFY            = 0x202,
>> +};
>> +
>> +struct rm_cmd_sq_hdr {
>> +     u16 opcode;
>> +     u16 msg_size;
>> +     u16 id;
>> +     u16 reserved;
>> +} __packed;
>> +
>> +struct rm_cmd_cq_hdr {
>> +     u16 id;
>> +     u16 reserved;
>> +} __packed;
>> +
>> +struct rm_cmd_sq_bin {
>> +     u64                     address;
>> +     u32                     size;
>> +     u32                     reserved1;
>> +     u32                     reserved2;
>> +     u32                     reserved3;
>> +     u64                     reserved4;
>> +} __packed;
>> +
>> +struct rm_cmd_sq_log_page {
>> +     u64                     address;
>> +     u32                     size;
>> +     u32                     reserved1;
>> +     u32                     type;
>> +     u32                     reserved2;
>> +} __packed;
>> +
>> +struct rm_cmd_sq_ctrl {
>> +     u32                     status;
>> +} __packed;
>> +
>> +struct rm_cmd_sq_data {
>> +     union {
>> +             struct rm_cmd_sq_log_page       page;
>> +             struct rm_cmd_sq_bin            bin;
>> +             struct rm_cmd_sq_ctrl           ctrl;
>> +     };
>> +} __packed;
>> +
>> +struct rm_cmd_cq_identify {
>> +     u16                     major;
>> +     u16                     minor;
>> +     u32                     reserved;
>> +} __packed;
>> +
>> +struct rm_cmd_cq_log_page {
>> +     u32                     len;
>> +     u32                     reserved;
>> +} __packed;
>> +
>> +struct rm_cmd_cq_control {
>> +     u16                     status;
>> +     u16                     reserved1;
>> +     u32                     reserved2;
>> +} __packed;
>> +
>> +struct rm_cmd_cq_data {
>> +     union {
>> +             struct rm_cmd_cq_identify       identify;
>> +             struct rm_cmd_cq_log_page       page;
>> +             struct rm_cmd_cq_control        ctrl;
>> +             u32                             reserved[2];
>> +     };
>> +     u32                     rcode;
>> +} __packed;
>> +
>> +struct rm_cmd_sq_msg {
>> +     struct rm_cmd_sq_hdr    hdr;
>> +     struct rm_cmd_sq_data   data;
>> +} __packed;
>> +
>> +struct rm_cmd_cq_msg {
>> +     struct rm_cmd_cq_hdr    hdr;
>> +     struct rm_cmd_cq_data   data;
>> +} __packed;
>> +
>> +struct rm_cmd {
>> +     struct rm_device        *rdev;
>> +     struct list_head        list;
>> +     struct completion       executed;
>> +     struct rm_cmd_sq_msg    sq_msg;
>> +     struct rm_cmd_cq_msg    cq_msg;
>> +     enum rm_queue_opcode    opcode;
>> +     void                    *buffer;
>> +     ssize_t                 size;
>> +};
>> +
>> +enum rm_queue_type {
>> +     RM_QUEUE_SQ,
>> +     RM_QUEUE_CQ
>> +};
>> +
>> +enum rm_cmd_log_page_type {
>> +     RM_CMD_LOG_PAGE_AXI_TRIP_STATUS = 0x0,
>> +     RM_CMD_LOG_PAGE_FW_ID           = 0xA,
>> +};
>> +
>> +struct rm_queue {
>> +     enum rm_queue_type      type;
>> +     u32                     pidx;
>> +     u32                     cidx;
>> +     u32                     offset;
>> +     u32                     data_offset;
>> +     u32                     data_size;
>> +     struct semaphore        data_lock;
>> +};
>> +
>> +struct rm_queue_header {
>> +     u32                     magic;
>> +     u32                     version;
>> +     u32                     size;
>> +     u32                     sq_off;
>> +     u32                     sq_slot_size;
>> +     u32                     cq_off;
>> +     u32                     sq_cidx;
>> +     u32                     cq_cidx;
>> +};
>> +
>> +struct rm_header {
>> +     u32                     magic;
>> +     u32                     queue_base;
>> +     u32                     queue_size;
>> +     u32                     status_off;
>> +     u32                     status_len;
>> +     u32                     log_index;
>> +     u32                     log_off;
>> +     u32                     log_size;
>> +     u32                     data_start;
>> +     u32                     data_end;
>> +};
>> +
>> +struct rm_device {
>> +     struct vmgmt_device     *vdev;
>> +     struct regmap           *shmem_regmap;
>> +     struct regmap           *io_regmap;
>> +
>> +     struct rm_header        rm_metadata;
>> +     u32                     queue_buffer_start;
>> +     u32                     queue_buffer_size;
>> +     u32                     queue_base;
>> +
>> +     /* Lock to queue access */
>> +     struct mutex            queue;
>> +     struct rm_queue         sq;
>> +     struct rm_queue         cq;
>> +     u32                     queue_size;
>> +
>> +     struct timer_list       msg_timer;
>> +     struct work_struct      msg_monitor;
>> +     struct timer_list       health_timer;
>> +     struct work_struct      health_monitor;
>> +     struct list_head        submitted_cmds;
>> +
>> +     int                     firewall_tripped;
>> +};
>> +
>> +int rm_queue_create_cmd(struct rm_device *rdev, enum rm_queue_opcode opcode,
>> +                     struct rm_cmd **cmd_ptr);
>> +void rm_queue_destory_cmd(struct rm_cmd *cmd);
>> +
>> +int rm_queue_data_init(struct rm_cmd *cmd, const char *buffer, ssize_t size);
>> +void rm_queue_data_fini(struct rm_cmd *cmd);
>> +
>> +int rm_queue_copy_response(struct rm_cmd *cmd, void *buffer, ssize_t len);
>> +
>> +int rm_boot_apu(struct rm_device *rdev);
>> +
>> +#endif       /* __VMGMT_RM_H */
>> diff --git a/drivers/fpga/amd/vmgmt.c b/drivers/fpga/amd/vmgmt.c
>> index b72eff9e8bc0..198213a13c7d 100644
>> --- a/drivers/fpga/amd/vmgmt.c
>> +++ b/drivers/fpga/amd/vmgmt.c
>> @@ -21,6 +21,8 @@
>>
>>   #include "vmgmt.h"
>>   #include "vmgmt-comms.h"
>> +#include "vmgmt-rm.h"
>> +#include "vmgmt-rm-queue.h"
>>
>>   #define DRV_NAME                     "amd-vmgmt"
>>   #define CLASS_NAME                   DRV_NAME
>> @@ -43,6 +45,61 @@ static inline struct vmgmt_device *vmgmt_inode_to_vdev(struct inode *inode)
>>        return (struct vmgmt_device *)container_of(inode->i_cdev, struct vmgmt_device, cdev);
>>   }
>>
>> +static int vmgmt_fpga_write_init(struct fpga_manager *mgr,
>> +                              struct fpga_image_info *info, const char *buf,
>> +                              size_t count)
>> +{
>> +     struct fpga_device *fdev = mgr->priv;
>> +     struct fw_tnx *tnx = &fdev->fw;
>> +     int ret;
>> +
>> +     ret = rm_queue_create_cmd(fdev->vdev->rdev, tnx->opcode, &tnx->cmd);
>> +     if (ret) {
>> +             fdev->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
>> +             return ret;
>> +     }
>> +
>> +     fdev->state = FPGA_MGR_STATE_WRITE_INIT;
>> +     return ret;
>> +}
>> +
>> +static int vmgmt_fpga_write(struct fpga_manager *mgr, const char *buf,
>> +                         size_t count)
>> +{
>> +     struct fpga_device *fdev = mgr->priv;
>> +     int ret;
>> +
>> +     ret = rm_queue_data_init(fdev->fw.cmd, buf, count);
>> +     if (ret) {
>> +             fdev->state = FPGA_MGR_STATE_WRITE_ERR;
>> +             rm_queue_destory_cmd(fdev->fw.cmd);
>> +             return ret;
>> +     }
>> +
>> +     fdev->state = FPGA_MGR_STATE_WRITE;
>> +     return ret;
>> +}
>> +
>> +static int vmgmt_fpga_write_complete(struct fpga_manager *mgr,
>> +                                  struct fpga_image_info *info)
>> +{
>> +     struct fpga_device *fdev = mgr->priv;
>> +     int ret;
>> +
>> +     ret = rm_queue_send_cmd(fdev->fw.cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
>> +     if (ret) {
>> +             fdev->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
>> +             vmgmt_err(fdev->vdev, "Send cmd failed:%d, cid:%d", ret, fdev->fw.id);
>> +     } else {
>> +             fdev->state = FPGA_MGR_STATE_WRITE_COMPLETE;
>> +     }
>> +
>> +     rm_queue_data_fini(fdev->fw.cmd);
>> +     rm_queue_destory_cmd(fdev->fw.cmd);
>> +     memset(&fdev->fw, 0, sizeof(fdev->fw));
>> +     return ret;
>> +}
>> +
>>   static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)
>>   {
>>        struct fpga_device *fdev = mgr->priv;
>> @@ -51,6 +108,9 @@ static enum fpga_mgr_states vmgmt_fpga_state(struct fpga_manager *mgr)
>>   }
>>
>>   static const struct fpga_manager_ops vmgmt_fpga_ops = {
>> +     .write_init = vmgmt_fpga_write_init,
>> +     .write = vmgmt_fpga_write,
>> +     .write_complete = vmgmt_fpga_write_complete,
>>        .state = vmgmt_fpga_state,
>>   };
>>
>> @@ -96,6 +156,13 @@ static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
>>                return ERR_PTR(ret);
>>        }
>>
>> +     ret = vmgmt_rm_get_fw_id(vdev->rdev, &vdev->intf_uuid);
>> +     if (ret) {
>> +             vmgmt_warn(vdev, "Failed to get interface uuid");
>> +             ret = -EINVAL;
>> +             goto unregister_fpga_mgr;
>> +     }
>> +
>>        /* create fgpa bridge, region for the base shell */
>>        fdev->bridge = fpga_bridge_register(dev, "AMD Versal FPGA Bridge",
>>                                            &vmgmt_br_ops, fdev);
>> @@ -132,6 +199,149 @@ static struct fpga_device *vmgmt_fpga_init(struct vmgmt_device *vdev)
>>        return ERR_PTR(ret);
>>   }
>>
>> +static int vmgmt_region_program(struct fpga_region *region, const void *data)
>> +{
>> +     struct fpga_device *fdev = region->priv;
>> +     struct vmgmt_device *vdev = fdev->vdev;
>> +     const struct axlf *xclbin = data;
>> +     struct fpga_image_info *info;
>> +     int ret;
>> +
>> +     info = fpga_image_info_alloc(&vdev->pdev->dev);
>> +     if (!info)
>> +             return -ENOMEM;
>> +
>> +     region->info = info;
>> +
>> +     info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
>> +     info->count = xclbin->header.length;
>> +     info->buf = (char *)xclbin;
>> +
>> +     ret = fpga_region_program_fpga(region);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Programming xclbin failed: %d", ret);
>> +             goto exit;
>> +     }
>> +
>> +     /* free bridges to allow reprogram */
>> +     if (region->get_bridges)
>> +             fpga_bridges_put(&region->bridge_list);
>> +
>> +exit:
>> +     fpga_image_info_free(info);
>> +     return ret;
>> +}
>> +
>> +static int vmgmt_fpga_region_match(struct device *dev, const void *data)
>> +{
>> +     const struct vmgmt_fpga_region *arg = data;
>> +     const struct fpga_region *match_region;
>> +     struct fpga_device *fdev = arg->fdev;
>> +     uuid_t compat_uuid;
>> +
>> +     if (dev->parent != &fdev->vdev->pdev->dev)
>> +             return false;
>> +
>> +     match_region = to_fpga_region(dev);
>> +
>> +     import_uuid(&compat_uuid, (const char *)match_region->compat_id);
>> +     if (uuid_equal(&compat_uuid, arg->uuid)) {
>> +             vmgmt_dbg(fdev->vdev, "Region match found");
>> +             return true;
>> +     }
>> +
>> +     vmgmt_err(fdev->vdev, "download uuid %pUb is not the same as device uuid %pUb",
>> +               arg->uuid, &compat_uuid);
>> +     return false;
>> +}
>> +
>> +static long vmgmt_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
>> +{
>> +     struct vmgmt_device *vdev = (struct vmgmt_device *)filep->private_data;
>> +     struct vmgmt_fpga_region reg = { 0 };
>> +     struct fpga_region *region = NULL;
>> +     struct axlf *axlf = NULL;
>> +     void *data = NULL;
>> +     size_t size = 0;
>> +     int ret = 0;
>> +
>> +     axlf = vmalloc(sizeof(*axlf));
>> +     if (!axlf)
>> +             return -ENOMEM;
>> +
>> +     ret = copy_from_user((void *)axlf, (void *)arg, sizeof(*axlf));
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to copy axlf: %d", ret);
>> +             ret = -EFAULT;
>> +             goto exit;
>> +     }
>> +
>> +     ret = memcmp(axlf->magic, VERSAL_XCLBIN_MAGIC_ID,
>> +                  sizeof(VERSAL_XCLBIN_MAGIC_ID));
>> +     if (ret) {
>> +             vmgmt_err(vdev, "unknown axlf magic %s", axlf->magic);
>> +             ret = -EINVAL;
>> +             goto exit;
>> +     }
>> +
>> +     /* axlf should never be over 1G and less than size of struct axlf */
>> +     size = axlf->header.length;
>> +     if (size < sizeof(struct axlf) || size > 1024 * 1024 * 1024) {
>> +             vmgmt_err(vdev, "axlf length %zu is invalid", size);
>> +             ret = -EINVAL;
>> +             goto exit;
>> +     }
>> +
>> +     data = vmalloc(size);
>> +     if (!data) {
>> +             ret = -ENOMEM;
>> +             goto exit;
>> +     }
>> +
>> +     ret = copy_from_user((void *)data, (void *)arg, size);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to copy data: %d", ret);
>> +             ret = -EFAULT;
>> +             goto exit;
>> +     }
>> +
>> +     switch (cmd) {
>> +     case VERSAL_MGMT_LOAD_XCLBIN_IOCTL:
>> +             vdev->fdev->fw.opcode = RM_QUEUE_OP_LOAD_XCLBIN;
>> +             break;
>> +     default:
>> +             vmgmt_err(vdev, "Invalid IOCTL command: %d", cmd);
>> +             ret = -EINVAL;
>> +             goto exit;
>> +     }
>> +
>> +     reg.uuid = &axlf->header.rom_uuid;
>> +     reg.fdev = vdev->fdev;
>> +
>> +     region = fpga_region_class_find(NULL, &reg, vmgmt_fpga_region_match);
>> +     if (!region) {
>> +             vmgmt_err(vdev, "Failed to find compatible region");
>> +             ret = -ENOENT;
>> +             goto exit;
>> +     }
>> +
>> +     ret = vmgmt_region_program(region, data);
>> +     if (ret) {
>> +             vmgmt_err(vdev, "Failed to program region");
>> +             goto exit;
>> +     }
>> +
>> +     vmgmt_info(vdev, "Downloaded axlf %pUb of size %zu Bytes",
>> +                &axlf->header.uuid, size);
>> +     uuid_copy(&vdev->xclbin_uuid, &axlf->header.uuid);
>> +
>> +exit:
>> +     vfree(data);
>> +     vfree(axlf);
>> +
>> +     return ret;
>> +}
>> +
>>   static int vmgmt_open(struct inode *inode, struct file *filep)
>>   {
>>        struct vmgmt_device *vdev = vmgmt_inode_to_vdev(inode);
>> @@ -155,6 +365,7 @@ static const struct file_operations vmgmt_fops = {
>>        .owner = THIS_MODULE,
>>        .open = vmgmt_open,
>>        .release = vmgmt_release,
>> +     .unlocked_ioctl = vmgmt_ioctl,
>>   };
>>
>>   static void vmgmt_chrdev_destroy(struct vmgmt_device *vdev)
>> @@ -201,6 +412,69 @@ static int vmgmt_chrdev_create(struct vmgmt_device *vdev)
>>        return 0;
>>   }
>>
>> +static enum fw_upload_err vmgmt_fw_prepare(struct fw_upload *fw_upload,
>> +                                        const u8 *data, u32 size)
>> +{
>> +     struct firmware_device *fwdev = fw_upload->dd_handle;
>> +     struct axlf *xsabin = (struct axlf *)data;
>> +     int ret;
>> +
>> +     ret = memcmp(xsabin->magic, VERSAL_XCLBIN_MAGIC_ID,
>> +                  sizeof(VERSAL_XCLBIN_MAGIC_ID));
>> +     if (ret) {
>> +             vmgmt_err(fwdev->vdev, "Invalid device firmware");
>> +             return FW_UPLOAD_ERR_INVALID_SIZE;
>> +     }
>> +
>> +     /* Firmware size should never be over 1G and less than size of struct axlf */
>> +     if (!size || size != xsabin->header.length || size < sizeof(*xsabin) ||
>> +         size > 1024 * 1024 * 1024) {
>> +             vmgmt_err(fwdev->vdev, "Invalid device firmware size");
>> +             return FW_UPLOAD_ERR_INVALID_SIZE;
>> +     }
>> +
>> +     ret = rm_queue_create_cmd(fwdev->vdev->rdev, RM_QUEUE_OP_LOAD_FW,
>> +                               &fwdev->cmd);
>> +     if (ret)
>> +             return FW_UPLOAD_ERR_RW_ERROR;
>> +
>> +     uuid_copy(&fwdev->uuid, &xsabin->header.uuid);
>> +     return FW_UPLOAD_ERR_NONE;
>> +}
>> +
>> +static enum fw_upload_err vmgmt_fw_write(struct fw_upload *fw_upload,
>> +                                      const u8 *data, u32 offset, u32 size,
>> +                                      u32 *written)
>> +{
>> +     struct firmware_device *fwdev = fw_upload->dd_handle;
>> +     int ret;
>> +
>> +     ret = rm_queue_data_init(fwdev->cmd, data, size);
>> +     if (ret)
>> +             return FW_UPLOAD_ERR_RW_ERROR;
>> +
>> +     *written = size;
>> +     return FW_UPLOAD_ERR_NONE;
>> +}
>> +
>> +static enum fw_upload_err vmgmt_fw_poll_complete(struct fw_upload *fw_upload)
>> +{
>> +     struct firmware_device *fwdev = fw_upload->dd_handle;
>> +     int ret;
>> +
>> +     vmgmt_info(fwdev->vdev, "Programming device firmware: %pUb", &fwdev->uuid);
>> +
>> +     ret = rm_queue_send_cmd(fwdev->cmd, RM_CMD_WAIT_DOWNLOAD_TIMEOUT);
>> +     if (ret) {
>> +             vmgmt_err(fwdev->vdev, "Send cmd failedi:%d, cid %d", ret, fwdev->id);
>> +             return FW_UPLOAD_ERR_HW_ERROR;
>> +     }
>> +
> 
> Basically I didn't see any difference on HW operations between FPGA
> reprogram & firmware loading. So why use different SW interfaces?
> 
> My idea is FPGA reprogramming is not just for image loading, but also
> device re-enumeration. As I mentioned before, programing the fpga region
> without notifying the impacted driver makes kernel crash.
> 
> I see there is an effort to introduce a unified FPGA reprograming
> interface that should address the concerns. I think we should stop
> adding vendor interfaces and make effort on the unified one.
> 
> https://lore.kernel.org/linux-fpga/20240726063819.2274324-1-nava.kishore.manne@amd.com/

This Versal card has 3 data images that we need to download.
1) The base FW - that enables the hardware, only take effect with a cold 
reboot.
2) The APU FW - the additional FW that is compatible with the base FW.
3) The PL Data - the (FPGA Bitstream Binary) can change the hardware PL.

Our original design was using ioctl to download No.1, and using the 
request_firmware to download the No.2. Both No.1 and No.2 will not be 
dynamically re-programable. We took the existing 
"intel-m10-bmc-sec-update.c" in the fpga as an example. This was why we 
converted the No.1 ioctl to fw_upload.

The No.3 will be initiated by the user PF driver only. Then, the mgmt PF 
driver will use request_firmware to load local saved data and then 
download it onto the FPGA. Thus, in this case, we don't need sysfs or 
ioctl. I will add those code in next patch.

Thus, we don't have to use new IOCTLs to load images for No.1, No.2 and 
No.3. The existing request_firmware API works for all.

I also briefly chatted with "nava.kishore.manne@amd.com". He is going to 
add ioctl and leveraging fpga_region_config_info for OF driver for now.

> 
>> +     vmgmt_info(fwdev->vdev, "Successfully programmed device firmware: %pUb",
>> +                &fwdev->uuid);
>> +     return FW_UPLOAD_ERR_NONE;
>> +}
>> +
>>   static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
>>   {
>>        struct firmware_device *fwdev = fw_upload->dd_handle;
>> @@ -208,8 +482,26 @@ static void vmgmt_fw_cancel(struct fw_upload *fw_upload)
>>        vmgmt_warn(fwdev->vdev, "canceled");
> 
> Nothing to do?
>

The rm_queue_withdraw_cmd should be called. I will fix this.

> 
> I'll stop here, please better organize the patches next time submitting
> to make reviewers easier, even if you know it is an RFC.

Thanks for taking your time reviewing all those patches. I will 
reorganize the patches.

> 
> Thanks,
> Yilun

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[Xu Yilun]

> > > +obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
> > 
> > IMHO the naming vmgmt is hard to understand, any better idea?
> 
> The "v" stand for Versal. We would change to amd-vpci for Versal based pcie

"v" + "pci" is quite a misleading term, maybe just versal-pci?

> devices.
> 
>

[...]
 
> > 
> > > +{
> > > +     struct comms_device *ccdev;
> > > +
> > > +     ccdev = devm_kzalloc(&vdev->pdev->dev, sizeof(*ccdev), GFP_KERNEL);
> > > +     if (!ccdev)
> > > +             return ERR_PTR(-ENOMEM);
> > > +
> > > +     ccdev->vdev = vdev;
> > > +
> > > +     ccdev->regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
> > > +                                           vdev->tbl + COMMS_PCI_BAR_OFF,
> > > +                                           &comms_regmap_config);
> > 
> > I'm not sure why a regmap is needed. All register accessing is within
> > the same module/file, and I assume a base+offset is enough to position
> > the register addr.
> 
> I thought the regmap is preferred. We can use some common APIs like
> regmap_bulk_*. The base+offset works too, then we will implement our own
> bulk_* functions. Please let me know.

I didn't see any regmap_bulk_*. AFAICS regmap is not needed here.

[...]

> > > +     /* create fgpa bridge, region for the base shell */
> > > +     fdev->bridge = fpga_bridge_register(dev, "AMD Versal FPGA Bridge",
> > > +                                         &vmgmt_br_ops, fdev);
> > 
> > I didn't find the br_ops anywhere in this patchset. So how to gate the
> > FPGA region when it is being reprogrammed? What is the physical link
> > between the FPGA region and outside visitors?
> 
> The FPGA region gate operation is done in the FW running in this PCIe card.
> The FW will "freeze" the gate before programing the PL. After downloading
> the new hardware. The FW will then "free" the gate.

So no OS operation is needed, then seems no need the fpga_bridge object.

> 
> No physical link between FPGA region and outside visitors, the FW handles
> all requests.
> 
> > 
> > > +     if (IS_ERR(fdev->bridge)) {
> > > +             vmgmt_err(vdev, "Failed to register FPGA bridge, err %ld",
> > > +                       PTR_ERR(fdev->bridge));
> > > +             ret = PTR_ERR(fdev->bridge);
> > > +             goto unregister_fpga_mgr;
> > > +     }
> > > +
> > > +     region = (struct fpga_region_info) {
> > > +             .compat_id = (struct fpga_compat_id *)&vdev->intf_uuid,
> > > +             .get_bridges = vmgmt_get_bridges,
> > > +             .mgr = fdev->mgr,
> > > +             .priv = fdev,
> > > +     };
> > > +
> > > +     fdev->region = fpga_region_register_full(dev, &region);
> > 
> > I assume the fpga region represents the user PF, is it? If you
> > reprogram the FPGA region, how does the user PF driver aware the HW is
> > changing?
> 
> The HW changing request is always requested from the user PF driver. The

I don't understand. In your patch the FPGA reprograming is triggered by
an IOCTL, usually a userspace application calls it. But here says it is 
triggered by the user PF *driver*, which IIUC is a kernel driver.
Anything I missed?

> user PF driver will make sure it is safe to change hardware. Then, the user
> PF driver notify the mgmt PF driver by a unique identify of the HW bitstream
> (PL Data).
> 
> The mgmt PF driver, the amd-vpci driver, will check the unique identify and
> then find the same PL Data from its local storage which is previously
> installed, and start downloading it.

Is the flow included in this patchset? Please elaborate more.

[...]

> > > +/**
> > > + * VERSAL_MGMT_LOAD_XCLBIN_IOCTL - Download XCLBIN to the device
> > > + *
> > > + * This IOCTL is used to download XCLBIN down to the device.
> > > + * Return: 0 on success, -errno on failure.
> > > + */
> > > +#define VERSAL_MGMT_LOAD_XCLBIN_IOCTL        _IOW(VERSAL_MGMT_MAGIC,         \
> > > +                                          VERSAL_MGMT_BASE + 0, void *)
> > 
> > Many definitions are added in a batch but some are not used in this
> > patch. Please reorganize the patches for easer review, even for first
> > version.
> > 
> > Thanks,
> > Yilun
> 
> Hi Yilun,
> 
> Thanks for taking your time, and yes for sure I will make each patch more
> self-contained.
> 
> Here is my thoughts on upcoming patches structure:
> 1st patch, adding driver probe and FPGA framework; the actual ops

Just adding driver probe for 1st patch please.

Thanks,
Yilun

> of handling communication channel message and remote queue message
> will present as no-op with comments.
> 
> 2nd patch, adding the communication channel services
> 3rd patch, adding the remote queue services
> 4th patch, adding the callers of using the remote queue services
> 
> Thanks,
> David
> 
> > 
> > > +
> > > +#endif /* _UAPI_LINUX_VMGMT_H */
> > > --
> > > 2.34.1
> > > 
> > > 

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[Yidong Zhang]

On 11/18/24 23:07, Xu Yilun wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> 
> 
>>>> +obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
>>>
>>> IMHO the naming vmgmt is hard to understand, any better idea?
>>
>> The "v" stand for Versal. We would change to amd-vpci for Versal based pcie
> 
> "v" + "pci" is quite a misleading term, maybe just versal-pci?

Sound good, I will rename the driver to versal-pci.

> 
>> devices.
>>
>>
> 
> [...]
> 
>>>
>>>> +{
>>>> +     struct comms_device *ccdev;
>>>> +
>>>> +     ccdev = devm_kzalloc(&vdev->pdev->dev, sizeof(*ccdev), GFP_KERNEL);
>>>> +     if (!ccdev)
>>>> +             return ERR_PTR(-ENOMEM);
>>>> +
>>>> +     ccdev->vdev = vdev;
>>>> +
>>>> +     ccdev->regmap = devm_regmap_init_mmio(&vdev->pdev->dev,
>>>> +                                           vdev->tbl + COMMS_PCI_BAR_OFF,
>>>> +                                           &comms_regmap_config);
>>>
>>> I'm not sure why a regmap is needed. All register accessing is within
>>> the same module/file, and I assume a base+offset is enough to position
>>> the register addr.
>>
>> I thought the regmap is preferred. We can use some common APIs like
>> regmap_bulk_*. The base+offset works too, then we will implement our own
>> bulk_* functions. Please let me know.
> 
> I didn't see any regmap_bulk_*. AFAICS regmap is not needed here.

The bulk_* is in the patch 0003. I can switch to base+offset in next 
version of the patches.

> 
> [...]
> 
>>>> +     /* create fgpa bridge, region for the base shell */
>>>> +     fdev->bridge = fpga_bridge_register(dev, "AMD Versal FPGA Bridge",
>>>> +                                         &vmgmt_br_ops, fdev);
>>>
>>> I didn't find the br_ops anywhere in this patchset. So how to gate the
>>> FPGA region when it is being reprogrammed? What is the physical link
>>> between the FPGA region and outside visitors?
>>
>> The FPGA region gate operation is done in the FW running in this PCIe card.
>> The FW will "freeze" the gate before programing the PL. After downloading
>> the new hardware. The FW will then "free" the gate.
> 
> So no OS operation is needed, then seems no need the fpga_bridge object.

Thanks, I will simplify this code.

> 
>>
>> No physical link between FPGA region and outside visitors, the FW handles
>> all requests.
>>
>>>
>>>> +     if (IS_ERR(fdev->bridge)) {
>>>> +             vmgmt_err(vdev, "Failed to register FPGA bridge, err %ld",
>>>> +                       PTR_ERR(fdev->bridge));
>>>> +             ret = PTR_ERR(fdev->bridge);
>>>> +             goto unregister_fpga_mgr;
>>>> +     }
>>>> +
>>>> +     region = (struct fpga_region_info) {
>>>> +             .compat_id = (struct fpga_compat_id *)&vdev->intf_uuid,
>>>> +             .get_bridges = vmgmt_get_bridges,
>>>> +             .mgr = fdev->mgr,
>>>> +             .priv = fdev,
>>>> +     };
>>>> +
>>>> +     fdev->region = fpga_region_register_full(dev, &region);
>>>
>>> I assume the fpga region represents the user PF, is it? If you
>>> reprogram the FPGA region, how does the user PF driver aware the HW is
>>> changing?
>>
>> The HW changing request is always requested from the user PF driver. The
> 
> I don't understand. In your patch the FPGA reprograming is triggered by
> an IOCTL, usually a userspace application calls it. But here says it is
> triggered by the user PF *driver*, which IIUC is a kernel driver.
> Anything I missed?
>

I will remove the IOCTL in the patch because this IOCTL is internal test 
only. The official request should always come from the user PF to avoid 
hardware crash due to hardware changes.

The userPF flow is described below.

>> user PF driver will make sure it is safe to change hardware. Then, the user
>> PF driver notify the mgmt PF driver by a unique identify of the HW bitstream
>> (PL Data).
>>
>> The mgmt PF driver, the amd-vpci driver, will check the unique identify and
>> then find the same PL Data from its local storage which is previously
>> installed, and start downloading it.
> 
> Is the flow included in this patchset? Please elaborate more.
> 

The userFP driver will send request to the mgmt PF driver (versal-pci).
The versal-pci driver will then find the same PL Data from its local 
storage (e.g. /lib/xilinx/fw_uuid/xclbin_uuid.xclbin).
The versal-pci driver will then read the firmware and download it.

I will port more code in next patch. The 1st patch did not include 
everything. I can use a single/separate patch for this specific flow so 
that it would be easy for you to review.

> [...]
> 
>>>> +/**
>>>> + * VERSAL_MGMT_LOAD_XCLBIN_IOCTL - Download XCLBIN to the device
>>>> + *
>>>> + * This IOCTL is used to download XCLBIN down to the device.
>>>> + * Return: 0 on success, -errno on failure.
>>>> + */
>>>> +#define VERSAL_MGMT_LOAD_XCLBIN_IOCTL        _IOW(VERSAL_MGMT_MAGIC,         \
>>>> +                                          VERSAL_MGMT_BASE + 0, void *)
>>>
>>> Many definitions are added in a batch but some are not used in this
>>> patch. Please reorganize the patches for easer review, even for first
>>> version.
>>>
>>> Thanks,
>>> Yilun
>>
>> Hi Yilun,
>>
>> Thanks for taking your time, and yes for sure I will make each patch more
>> self-contained.
>>
>> Here is my thoughts on upcoming patches structure:
>> 1st patch, adding driver probe and FPGA framework; the actual ops
> 
> Just adding driver probe for 1st patch please.
> 

Sure.

Thank,
David

> Thanks,
> Yilun
> 
>> of handling communication channel message and remote queue message
>> will present as no-op with comments.
>>
>> 2nd patch, adding the communication channel services
>> 3rd patch, adding the remote queue services
>> 4th patch, adding the callers of using the remote queue services
>>
>> Thanks,
>> David
>>
>>>
>>>> +
>>>> +#endif /* _UAPI_LINUX_VMGMT_H */
>>>> --
>>>> 2.34.1
>>>>
>>>>

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[Yidong Zhang]

On 11/18/24 23:07, Xu Yilun wrote:
> 
>>>> +obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
>>> IMHO the naming vmgmt is hard to understand, any better idea?
>> The "v" stand for Versal. We would change to amd-vpci for Versal based pcie
> "v" + "pci" is quite a misleading term, maybe just versal-pci?

Hi Yilun,

I sent the V2 patch and refactored the driver as versal-pci now.
One more thing that I kept in V2 was the firmware_upload. I forgot to
mention that I'd love to switch to the newly proposed interface once
it is ready. I saw the proposal was now as config_fs and it was not 
merged yet.

Happy Holidays.

Thanks,
David

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[Yidong Zhang]

On 3/12/23 11:03, Xu Yilun wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> 
> 
> On Sun, Dec 22, 2024 at 05:53:30PM -0800, Yidong Zhang wrote:
>>
>>
>> On 11/18/24 23:07, Xu Yilun wrote:
>>>
>>>>>> +obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
>>>>> IMHO the naming vmgmt is hard to understand, any better idea?
>>>> The "v" stand for Versal. We would change to amd-vpci for Versal based pcie
>>> "v" + "pci" is quite a misleading term, maybe just versal-pci?
>>
>> Hi Yilun,
>>
>> I sent the V2 patch and refactored the driver as versal-pci now.
>> One more thing that I kept in V2 was the firmware_upload. I forgot to
>> mention that I'd love to switch to the newly proposed interface once
>> it is ready. I saw the proposal was now as config_fs and it was not merged
> 
> Good to know that.
> 
> I didn't start reviewing the v2 yet. But one thing is that now the
> versal-pci FPGA manager has no user because of the ongoing uAPI, so
> cannot be merged, and I won't pay much effort on this series for now.

Hi Yilun,

Can we add this as TODO in the future when the uAPI solution is ready to 
switch? We spent some time to refactor the driver and address most of 
your comments in the V2. Hopefully, can you please start reviewing the 
fpga_mgr and other driver code?

We'd think that we use the firmware_upload for 1st approach and start 
letting user use this driver.

We definitely will switch to the new uAPI as soon as it is ready in the 
linux fpga driver. But we'd not like this uAPI holds up everything we 
already spent times.

Thanks,
David
> 
> Thanks,
> Yilun
> 
>> yet.
>>
>> Happy Holidays.
>>
>> Thanks,
>> David

Re: [PATCH V1 1/3] drivers/fpga/amd: Add new driver for AMD Versal PCIe card

[Yidong Zhang]

On 3/12/23 14:30, Xu Yilun wrote:
> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> 
> 
> On Wed, Dec 25, 2024 at 10:10:06PM -0800, Yidong Zhang wrote:
>>
>>
>> On 3/12/23 11:03, Xu Yilun wrote:
>>> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
>>>
>>>
>>> On Sun, Dec 22, 2024 at 05:53:30PM -0800, Yidong Zhang wrote:
>>>>
>>>>
>>>> On 11/18/24 23:07, Xu Yilun wrote:
>>>>>
>>>>>>>> +obj-$(CONFIG_AMD_VERSAL_MGMT)                        += amd-vmgmt.o
>>>>>>> IMHO the naming vmgmt is hard to understand, any better idea?
>>>>>> The "v" stand for Versal. We would change to amd-vpci for Versal based pcie
>>>>> "v" + "pci" is quite a misleading term, maybe just versal-pci?
>>>>
>>>> Hi Yilun,
>>>>
>>>> I sent the V2 patch and refactored the driver as versal-pci now.
>>>> One more thing that I kept in V2 was the firmware_upload. I forgot to
>>>> mention that I'd love to switch to the newly proposed interface once
>>>> it is ready. I saw the proposal was now as config_fs and it was not merged
>>>
>>> Good to know that.
>>>
>>> I didn't start reviewing the v2 yet. But one thing is that now the
>>> versal-pci FPGA manager has no user because of the ongoing uAPI, so
>>> cannot be merged, and I won't pay much effort on this series for now.
>>
>> Hi Yilun,
>>
>> Can we add this as TODO in the future when the uAPI solution is ready to
>> switch? We spent some time to refactor the driver and address most of your
>

Hi Yilun,

Happy New Year!

> Sorry, generally kernel won't merge code that has no user, which means
> the code are not tested.

Just confirm "the code are not tested".

We use XRT opensource userPF driver (called xocl.ko) to test the 
versal-pci driver. You can find the source code here:
https://github.com/xdavidz/XRT1/commits/dtb

We are planing to let user use this mgmtPF driver + our open-sourced 
userPF driver + our library as full stack. I will provide full test
result if needed.

We are also working on some linux drm driver. We use the similar way
to prove that the drive has been tested.

Or, you are referring the "user" as user PF driver?
We do have user PF driver as a separate driver.

Please confirm if you are thinking that only userPF + mgmtPF as one 
driver? We architect our driver to 2 drivers due to cloud vendor 
specific requirement.

> 
>> comments in the V2. Hopefully, can you please start reviewing the fpga_mgr
>> and other driver code?
> 
> Yes, I can review the code when possible. But will not merge it.

Thanks. Please see questions above.

> 
>>
>> We'd think that we use the firmware_upload for 1st approach and start
>> letting user use this driver.
>>
>> We definitely will switch to the new uAPI as soon as it is ready in the
>> linux fpga driver. But we'd not like this uAPI holds up everything we
>> already spent times.
> 
> If you want speed up, then collaborate to develop/review the dependent
> patchset, that's how the community works. I actually don't want to be
> the only reviewer in this mail list, and others just send patches.

I'd like to help for the patchset. Let me ping the developer and see how
he would need help on this.

Thanks,
David

> 
> Thanks,
> Yilun