[PATCH] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[PATCH] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

This patch introduces a PCI hotplug controller driver for the OCTEON
PCIe device, a multi-function PCIe device where the first function acts
as a hotplug controller. It is equipped with MSI-x interrupts to notify
the host of hotplug events from the OCTEON firmware.

The driver facilitates the hotplugging of non-controller functions
within the same device. During probe, non-controller functions are
removed and registered as PCI hotplug slots. The slots are added back
only upon request from the device firmware. The driver also allows the
enabling and disabling of the slots via sysfs slot entries, provided by
the PCI hotplug framework.

Signed-off-by: Shijith Thotton <sthotton@marvell.com>
Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
---

This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
controller. The OCTEON PCIe device is a multi-function device where the first
function acts as a PCI hotplug controller.

              +--------------------------------+
              |           Root Port            |
              +--------------------------------+
                              |
                             PCIe
                              |
+---------------------------------------------------------------+
|              OCTEON PCIe Multifunction Device                 |
+---------------------------------------------------------------+
            |                    |              |            |
            |                    |              |            |
+---------------------+  +----------------+  +-----+  +----------------+
|      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
| (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
+---------------------+  +----------------+  +-----+  +----------------+
            |
            |
+-------------------------+
|   Controller Firmware   |
+-------------------------+

The hotplug controller driver facilitates the hotplugging of non-controller
functions in the same device. During the probe of the driver, the non-controller
function are removed and registered as PCI hotplug slots. They are added back
only upon request from the device firmware. The driver also allows the user to
enable/disable the functions using sysfs slot entries provided by PCI hotplug
framework.

This solution adopts a hardware + software approach for several reasons:

1. To reduce hardware implementation cost. Supporting complete hotplug
   capability within the card would require a PCI switch implemented within.

2. In the multi-function device, non-controller functions can act as emulated
   devices. The firmware can dynamically enable or disable them at runtime.

3. Not all root ports support PCI hotplug. This approach provides greater
   flexibility and compatibility across different hardware configurations.

The hotplug controller function is lightweight and is equipped with MSI-x
interrupts to notify the host about hotplug events. Upon receiving an
interrupt, the hotplug register is read, and the required function is enabled
or disabled.

This driver will be beneficial for managing PCI hotplug events on the OCTEON
PCIe device without requiring complex hardware solutions.

 MAINTAINERS                    |   6 +
 drivers/pci/hotplug/Kconfig    |  10 +
 drivers/pci/hotplug/Makefile   |   1 +
 drivers/pci/hotplug/octep_hp.c | 351 +++++++++++++++++++++++++++++++++
 4 files changed, 368 insertions(+)
 create mode 100644 drivers/pci/hotplug/octep_hp.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 42decde38320..7b5a618eed1c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
 R:	vattunuru@marvell.com
 F:	drivers/vdpa/octeon_ep/
 
+MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
+R:	Shijith Thotton <sthotton@marvell.com>
+R:	Vamsi Attunuru <vattunuru@marvell.com>
+S:	Supported
+F:	drivers/pci/hotplug/octep_hp.c
+
 MATROX FRAMEBUFFER DRIVER
 L:	linux-fbdev@vger.kernel.org
 S:	Orphan
diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
index 1472aef0fb81..2e38fd25f7ef 100644
--- a/drivers/pci/hotplug/Kconfig
+++ b/drivers/pci/hotplug/Kconfig
@@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
 
 	  When in doubt, say Y.
 
+config HOTPLUG_PCI_OCTEONEP
+	bool "OCTEON PCI device Hotplug controller driver"
+	depends on HOTPLUG_PCI
+	help
+	  Say Y here if you have an OCTEON PCIe device with a hotplug
+	  controller. This driver enables the non-controller functions of the
+	  device to be registered as hotplug slots.
+
+	  When in doubt, say N.
+
 endif # HOTPLUG_PCI
diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
index 240c99517d5e..40aaf31fe338 100644
--- a/drivers/pci/hotplug/Makefile
+++ b/drivers/pci/hotplug/Makefile
@@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+= rpaphp.o
 obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
 obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
 obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
+obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
 
 # acpiphp_ibm extends acpiphp, so should be linked afterwards.
 
diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
new file mode 100644
index 000000000000..efeb542d4993
--- /dev/null
+++ b/drivers/pci/hotplug/octep_hp.c
@@ -0,0 +1,351 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2024 Marvell. */
+
+#include <linux/delay.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/pci_hotplug.h>
+#include <linux/slab.h>
+
+#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
+#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
+#define OCTEP_HP_DRV_NAME       "octep_hp"
+
+/* Interrupt vectors for hotplug enable and disable events. */
+enum octep_hp_vec_type {
+	OCTEP_HP_VEC_ENA,
+	OCTEP_HP_VEC_DIS,
+};
+
+struct octep_hp_cmd {
+	struct list_head list;
+	enum octep_hp_vec_type vec_type;
+	u64 slot_mask;
+};
+
+struct octep_hp_slot {
+	struct list_head list;
+	struct hotplug_slot slot;
+	u16 slot_number;
+	struct pci_dev *hp_pdev;
+	unsigned int hp_devfn;
+	struct octep_hp_controller *ctrl;
+};
+
+struct octep_hp_controller {
+	void __iomem *base;
+	struct pci_dev *pdev;
+	struct work_struct work;
+	struct list_head slot_list;
+	struct mutex slot_lock; /* Protects slot_list */
+	struct list_head hp_cmd_list;
+	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
+};
+
+static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl, struct octep_hp_slot *hp_slot)
+{
+	mutex_lock(&hp_ctrl->slot_lock);
+	if (hp_slot->hp_pdev) {
+		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already enabled\n", hp_slot->slot_number);
+		mutex_unlock(&hp_ctrl->slot_lock);
+		return;
+	}
+
+	/* Scan the device and add it to the bus */
+	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus, hp_slot->hp_devfn);
+	pci_bus_assign_resources(hp_ctrl->pdev->bus);
+	pci_bus_add_device(hp_slot->hp_pdev);
+
+	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n", hp_slot->slot_number);
+	mutex_unlock(&hp_ctrl->slot_lock);
+}
+
+static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
+				  struct octep_hp_slot *hp_slot)
+{
+	mutex_lock(&hp_ctrl->slot_lock);
+	if (!hp_slot->hp_pdev) {
+		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n", hp_slot->slot_number);
+		mutex_unlock(&hp_ctrl->slot_lock);
+		return;
+	}
+
+	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n", hp_slot->slot_number);
+
+	/* Remove the device from the bus */
+	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
+	hp_slot->hp_pdev = NULL;
+	mutex_unlock(&hp_ctrl->slot_lock);
+}
+
+static int octep_hp_enable_slot(struct hotplug_slot *slot)
+{
+	struct octep_hp_slot *hp_slot = container_of(slot, struct octep_hp_slot, slot);
+
+	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
+	return 0;
+}
+
+static int octep_hp_disable_slot(struct hotplug_slot *slot)
+{
+	struct octep_hp_slot *hp_slot = container_of(slot, struct octep_hp_slot, slot);
+
+	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
+	return 0;
+}
+
+static struct hotplug_slot_ops octep_hp_slot_ops = {
+	.enable_slot = octep_hp_enable_slot,
+	.disable_slot = octep_hp_disable_slot,
+};
+
+#define SLOT_NAME_SIZE 16
+static int octep_hp_register_slot(struct octep_hp_controller *hp_ctrl, struct pci_dev *pdev,
+				  u16 slot_number)
+{
+	char slot_name[SLOT_NAME_SIZE];
+	struct octep_hp_slot *hp_slot;
+	int ret;
+
+	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
+	if (!hp_slot)
+		return -ENOMEM;
+
+	hp_slot->ctrl = hp_ctrl;
+	hp_slot->hp_pdev = pdev;
+	hp_slot->hp_devfn = pdev->devfn;
+	hp_slot->slot_number = slot_number;
+	hp_slot->slot.ops = &octep_hp_slot_ops;
+
+	snprintf(slot_name, SLOT_NAME_SIZE, "octep_hp_%u", slot_number);
+	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus, PCI_SLOT(pdev->devfn), slot_name);
+	if (ret) {
+		dev_err(&pdev->dev, "Failed to register hotplug slot %u\n", slot_number);
+		kfree(hp_slot);
+		return ret;
+	}
+
+	octep_hp_disable_pdev(hp_ctrl, hp_slot);
+	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
+
+	return 0;
+}
+
+static bool octep_hp_slot(struct octep_hp_controller *hp_ctrl, struct pci_dev *pdev)
+{
+	/* Check if the PCI device can be hotplugged */
+	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
+		PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
+}
+
+static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl, struct octep_hp_cmd *hp_cmd)
+{
+	struct octep_hp_slot *hp_slot;
+
+	/* Enable or disable the slots based on the slot mask */
+	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
+		if (hp_cmd->slot_mask & BIT(hp_slot->slot_number)) {
+			if (hp_cmd->vec_type == OCTEP_HP_VEC_ENA)
+				octep_hp_enable_pdev(hp_ctrl, hp_slot);
+			else
+				octep_hp_disable_pdev(hp_ctrl, hp_slot);
+		}
+	}
+}
+
+static void octep_hp_work_handler(struct work_struct *work)
+{
+	struct octep_hp_controller *hp_ctrl = container_of(work, struct octep_hp_controller, work);
+	struct octep_hp_cmd *hp_cmd;
+	unsigned long flags;
+
+	/* Process all the hotplug commands */
+	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
+	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
+		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list, struct octep_hp_cmd, list);
+		list_del(&hp_cmd->list);
+		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
+
+		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
+		kfree(hp_cmd);
+
+		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
+	}
+	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
+}
+
+static irqreturn_t octep_hp_intr_handler(int irq, void *data)
+{
+	struct octep_hp_controller *hp_ctrl = data;
+	struct pci_dev *pdev = hp_ctrl->pdev;
+	enum octep_hp_vec_type vec_type;
+	struct octep_hp_cmd *hp_cmd;
+	u64 slot_mask;
+
+	vec_type = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_ENA)) == irq ?
+		OCTEP_HP_VEC_ENA : OCTEP_HP_VEC_DIS;
+
+	slot_mask = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(vec_type));
+	if (!slot_mask) {
+		dev_err(&pdev->dev, "Invalid slot mask %llx\n", slot_mask);
+		return IRQ_HANDLED;
+	}
+
+	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
+	if (!hp_cmd)
+		return IRQ_HANDLED;
+
+	hp_cmd->slot_mask = slot_mask;
+	hp_cmd->vec_type = vec_type;
+
+	/* Add the command to the list and schedule the work */
+	spin_lock(&hp_ctrl->hp_cmd_lock);
+	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
+	spin_unlock(&hp_ctrl->hp_cmd_lock);
+	schedule_work(&hp_ctrl->work);
+
+	/* Clear the interrupt */
+	writeq(slot_mask, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(vec_type));
+
+	return IRQ_HANDLED;
+}
+
+static void octep_hp_deregister_slots(struct octep_hp_controller *hp_ctrl)
+{
+	struct octep_hp_slot *hp_slot, *tmp;
+
+	/* Deregister all the hotplug slots */
+	list_for_each_entry_safe(hp_slot, tmp, &hp_ctrl->slot_list, list) {
+		pci_hp_deregister(&hp_slot->slot);
+		octep_hp_enable_pdev(hp_ctrl, hp_slot);
+		list_del(&hp_slot->list);
+		kfree(hp_slot);
+	}
+}
+
+static void octep_hp_controller_cleanup(void *data)
+{
+	struct octep_hp_controller *hp_ctrl = data;
+
+	pci_free_irq_vectors(hp_ctrl->pdev);
+	flush_work(&hp_ctrl->work);
+	octep_hp_deregister_slots(hp_ctrl);
+}
+
+static int octep_hp_controller_setup(struct pci_dev *pdev, struct octep_hp_controller *hp_ctrl)
+{
+	struct device *dev = &pdev->dev;
+	int ret;
+
+	ret = pcim_enable_device(pdev);
+	if (ret) {
+		dev_err(dev, "Failed to enable PCI device\n");
+		return ret;
+	}
+
+	ret = pcim_iomap_regions(pdev, BIT(0), OCTEP_HP_DRV_NAME);
+	if (ret) {
+		dev_err(dev, "Failed to request MMIO region\n");
+		return ret;
+	}
+
+	pci_set_master(pdev);
+	pci_set_drvdata(pdev, hp_ctrl);
+
+	INIT_LIST_HEAD(&hp_ctrl->slot_list);
+	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
+	mutex_init(&hp_ctrl->slot_lock);
+	spin_lock_init(&hp_ctrl->hp_cmd_lock);
+	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
+
+	hp_ctrl->pdev = pdev;
+	hp_ctrl->base = pcim_iomap_table(pdev)[0];
+	if (!hp_ctrl->base) {
+		dev_err(dev, "Failed to get device resource map\n");
+		return -ENODEV;
+	}
+
+	ret = pci_alloc_irq_vectors(pdev, 1, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS) + 1,
+				    PCI_IRQ_MSIX);
+	if (ret < 0) {
+		dev_err(dev, "Failed to alloc MSI-X vectors");
+		return ret;
+	}
+
+	ret = devm_add_action(dev, octep_hp_controller_cleanup, hp_ctrl);
+	if (ret) {
+		dev_err(dev, "Failed to add action for controller cleanup\n");
+		return ret;
+	}
+
+	ret = devm_request_irq(dev, pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_ENA)),
+			       octep_hp_intr_handler, 0, "octep_hp_ena", hp_ctrl);
+	if (ret) {
+		dev_err(dev, "Failed to register slot enable interrupt handle\n");
+		return ret;
+	}
+
+	ret = devm_request_irq(dev, pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS)),
+			       octep_hp_intr_handler, 0, "octep_hp_dis", hp_ctrl);
+	if (ret) {
+		dev_err(dev, "Failed to register slot disable interrupt handle\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static int octep_hp_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	struct octep_hp_controller *hp_ctrl;
+	struct pci_dev *tmp_pdev = NULL;
+	u16 slot_number = 0;
+	int ret;
+
+	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
+	if (!hp_ctrl)
+		return -ENOMEM;
+
+	ret = octep_hp_controller_setup(pdev, hp_ctrl);
+	if (ret) {
+		dev_err(&pdev->dev, "Failed to setup octep controller\n");
+		return ret;
+	}
+
+	/*
+	 * Register all hotplug slots. Hotplug controller is the first function of the PCI device.
+	 * The hotplug slots are the remaining functions of the PCI device. They are removed from
+	 * the bus and are added back when the hotplug event is triggered.
+	 */
+	for_each_pci_dev(tmp_pdev) {
+		if (octep_hp_slot(hp_ctrl, tmp_pdev)) {
+			ret = octep_hp_register_slot(hp_ctrl, tmp_pdev, slot_number++);
+			if (ret) {
+				dev_err(&pdev->dev, "Failed to register hotplug slots.\n");
+				return ret;
+			}
+		}
+	}
+
+	return 0;
+}
+
+#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
+static struct pci_device_id octep_hp_pci_map[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, OCTEP_DEVID_HP_CONTROLLER) },
+	{ 0 },
+};
+
+static struct pci_driver octep_hp = {
+	.name     = OCTEP_HP_DRV_NAME,
+	.id_table = octep_hp_pci_map,
+	.probe    = octep_hp_pci_probe,
+};
+
+module_pci_driver(octep_hp);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Marvell");
+MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
-- 
2.25.1

Re: [PATCH] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Ilpo Järvinen]

On Tue, 20 Aug 2024, Shijith Thotton wrote:

> This patch introduces a PCI hotplug controller driver for the OCTEON
> PCIe device, a multi-function PCIe device where the first function acts
> as a hotplug controller. It is equipped with MSI-x interrupts to notify
> the host of hotplug events from the OCTEON firmware.
> 
> The driver facilitates the hotplugging of non-controller functions
> within the same device. During probe, non-controller functions are
> removed and registered as PCI hotplug slots. The slots are added back
> only upon request from the device firmware. The driver also allows the
> enabling and disabling of the slots via sysfs slot entries, provided by
> the PCI hotplug framework.
> 
> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> ---
> 
> This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
> controller. The OCTEON PCIe device is a multi-function device where the first
> function acts as a PCI hotplug controller.
> 
>               +--------------------------------+
>               |           Root Port            |
>               +--------------------------------+
>                               |
>                              PCIe
>                               |
> +---------------------------------------------------------------+
> |              OCTEON PCIe Multifunction Device                 |
> +---------------------------------------------------------------+
>             |                    |              |            |
>             |                    |              |            |
> +---------------------+  +----------------+  +-----+  +----------------+
> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
> +---------------------+  +----------------+  +-----+  +----------------+
>             |
>             |
> +-------------------------+
> |   Controller Firmware   |
> +-------------------------+
> 
> The hotplug controller driver facilitates the hotplugging of non-controller
> functions in the same device. During the probe of the driver, the non-controller
> function are removed and registered as PCI hotplug slots. They are added back
> only upon request from the device firmware. The driver also allows the user to
> enable/disable the functions using sysfs slot entries provided by PCI hotplug
> framework.
> 
> This solution adopts a hardware + software approach for several reasons:
> 
> 1. To reduce hardware implementation cost. Supporting complete hotplug
>    capability within the card would require a PCI switch implemented within.
> 
> 2. In the multi-function device, non-controller functions can act as emulated
>    devices. The firmware can dynamically enable or disable them at runtime.
> 
> 3. Not all root ports support PCI hotplug. This approach provides greater
>    flexibility and compatibility across different hardware configurations.
> 
> The hotplug controller function is lightweight and is equipped with MSI-x
> interrupts to notify the host about hotplug events. Upon receiving an
> interrupt, the hotplug register is read, and the required function is enabled
> or disabled.
> 
> This driver will be beneficial for managing PCI hotplug events on the OCTEON
> PCIe device without requiring complex hardware solutions.
> 
>  MAINTAINERS                    |   6 +
>  drivers/pci/hotplug/Kconfig    |  10 +
>  drivers/pci/hotplug/Makefile   |   1 +
>  drivers/pci/hotplug/octep_hp.c | 351 +++++++++++++++++++++++++++++++++
>  4 files changed, 368 insertions(+)
>  create mode 100644 drivers/pci/hotplug/octep_hp.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 42decde38320..7b5a618eed1c 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
>  R:	vattunuru@marvell.com
>  F:	drivers/vdpa/octeon_ep/
>  
> +MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
> +R:	Shijith Thotton <sthotton@marvell.com>
> +R:	Vamsi Attunuru <vattunuru@marvell.com>
> +S:	Supported
> +F:	drivers/pci/hotplug/octep_hp.c
> +
>  MATROX FRAMEBUFFER DRIVER
>  L:	linux-fbdev@vger.kernel.org
>  S:	Orphan
> diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
> index 1472aef0fb81..2e38fd25f7ef 100644
> --- a/drivers/pci/hotplug/Kconfig
> +++ b/drivers/pci/hotplug/Kconfig
> @@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
>  
>  	  When in doubt, say Y.
>  
> +config HOTPLUG_PCI_OCTEONEP
> +	bool "OCTEON PCI device Hotplug controller driver"
> +	depends on HOTPLUG_PCI
> +	help
> +	  Say Y here if you have an OCTEON PCIe device with a hotplug
> +	  controller. This driver enables the non-controller functions of the
> +	  device to be registered as hotplug slots.
> +
> +	  When in doubt, say N.
> +
>  endif # HOTPLUG_PCI
> diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
> index 240c99517d5e..40aaf31fe338 100644
> --- a/drivers/pci/hotplug/Makefile
> +++ b/drivers/pci/hotplug/Makefile
> @@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+= rpaphp.o
>  obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
>  obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
>  obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
> +obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
>  
>  # acpiphp_ibm extends acpiphp, so should be linked afterwards.
>  
> diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
> new file mode 100644
> index 000000000000..efeb542d4993
> --- /dev/null
> +++ b/drivers/pci/hotplug/octep_hp.c
> @@ -0,0 +1,351 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright (C) 2024 Marvell. */
> +
> +#include <linux/delay.h>
> +#include <linux/init.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/pci_hotplug.h>
> +#include <linux/slab.h>
> +
> +#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
> +#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
> +#define OCTEP_HP_DRV_NAME       "octep_hp"
> +
> +/* Interrupt vectors for hotplug enable and disable events. */
> +enum octep_hp_vec_type {
> +	OCTEP_HP_VEC_ENA,
> +	OCTEP_HP_VEC_DIS,

If I understand your code right, these cannot be arbitrary numbers but has 
to be specific numbers to match the vector number so you should explicitly 
set them to those values.

> +};
> +
> +struct octep_hp_cmd {
> +	struct list_head list;

Missing include for list_head.

> +	enum octep_hp_vec_type vec_type;
> +	u64 slot_mask;
> +};
> +
> +struct octep_hp_slot {
> +	struct list_head list;
> +	struct hotplug_slot slot;
> +	u16 slot_number;
> +	struct pci_dev *hp_pdev;
> +	unsigned int hp_devfn;
> +	struct octep_hp_controller *ctrl;
> +};
> +
> +struct octep_hp_controller {
> +	void __iomem *base;
> +	struct pci_dev *pdev;
> +	struct work_struct work;

Missing include.

> +	struct list_head slot_list;
> +	struct mutex slot_lock; /* Protects slot_list */

Missing include.

> +	struct list_head hp_cmd_list;
> +	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
> +};
> +
> +static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl, struct octep_hp_slot *hp_slot)
> +{
> +	mutex_lock(&hp_ctrl->slot_lock);

Use guard().

> +	if (hp_slot->hp_pdev) {
> +		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already enabled\n", hp_slot->slot_number);
> +		mutex_unlock(&hp_ctrl->slot_lock);
> +		return;
> +	}
> +
> +	/* Scan the device and add it to the bus */
> +	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus, hp_slot->hp_devfn);
> +	pci_bus_assign_resources(hp_ctrl->pdev->bus);
> +	pci_bus_add_device(hp_slot->hp_pdev);
> +
> +	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n", hp_slot->slot_number);

Missing include for dev_dbg().

> +	mutex_unlock(&hp_ctrl->slot_lock);
> +}
> +
> +static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
> +				  struct octep_hp_slot *hp_slot)
> +{
> +	mutex_lock(&hp_ctrl->slot_lock);

Use guard().

> +	if (!hp_slot->hp_pdev) {
> +		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n", hp_slot->slot_number);
> +		mutex_unlock(&hp_ctrl->slot_lock);
> +		return;
> +	}
> +
> +	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n", hp_slot->slot_number);
> +
> +	/* Remove the device from the bus */
> +	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
> +	hp_slot->hp_pdev = NULL;
> +	mutex_unlock(&hp_ctrl->slot_lock);
> +}
> +
> +static int octep_hp_enable_slot(struct hotplug_slot *slot)
> +{
> +	struct octep_hp_slot *hp_slot = container_of(slot, struct octep_hp_slot, slot);

Missing include for container_of().

> +
> +	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
> +	return 0;
> +}
> +
> +static int octep_hp_disable_slot(struct hotplug_slot *slot)
> +{
> +	struct octep_hp_slot *hp_slot = container_of(slot, struct octep_hp_slot, slot);
> +
> +	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
> +	return 0;
> +}
> +
> +static struct hotplug_slot_ops octep_hp_slot_ops = {
> +	.enable_slot = octep_hp_enable_slot,
> +	.disable_slot = octep_hp_disable_slot,
> +};
> +
> +#define SLOT_NAME_SIZE 16
> +static int octep_hp_register_slot(struct octep_hp_controller *hp_ctrl, struct pci_dev *pdev,
> +				  u16 slot_number)
> +{
> +	char slot_name[SLOT_NAME_SIZE];
> +	struct octep_hp_slot *hp_slot;
> +	int ret;
> +
> +	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
> +	if (!hp_slot)
> +		return -ENOMEM;
> +
> +	hp_slot->ctrl = hp_ctrl;
> +	hp_slot->hp_pdev = pdev;
> +	hp_slot->hp_devfn = pdev->devfn;
> +	hp_slot->slot_number = slot_number;
> +	hp_slot->slot.ops = &octep_hp_slot_ops;
> +
> +	snprintf(slot_name, SLOT_NAME_SIZE, "octep_hp_%u", slot_number);

SLOT_NAME_SIZE -> sizeof(slot_name)

> +	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus, PCI_SLOT(pdev->devfn), slot_name);
> +	if (ret) {
> +		dev_err(&pdev->dev, "Failed to register hotplug slot %u\n", slot_number);
> +		kfree(hp_slot);
> +		return ret;
> +	}
> +
> +	octep_hp_disable_pdev(hp_ctrl, hp_slot);
> +	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
> +
> +	return 0;
> +}
> +
> +static bool octep_hp_slot(struct octep_hp_controller *hp_ctrl, struct pci_dev *pdev)
> +{
> +	/* Check if the PCI device can be hotplugged */
> +	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
> +		PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
> +}
> +
> +static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl, struct octep_hp_cmd *hp_cmd)
> +{
> +	struct octep_hp_slot *hp_slot;
> +
> +	/* Enable or disable the slots based on the slot mask */
> +	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
> +		if (hp_cmd->slot_mask & BIT(hp_slot->slot_number)) {

Reverse logic & use continue ?

> +			if (hp_cmd->vec_type == OCTEP_HP_VEC_ENA)
> +				octep_hp_enable_pdev(hp_ctrl, hp_slot);
> +			else
> +				octep_hp_disable_pdev(hp_ctrl, hp_slot);
> +		}
> +	}
> +}
> +
> +static void octep_hp_work_handler(struct work_struct *work)
> +{
> +	struct octep_hp_controller *hp_ctrl = container_of(work, struct octep_hp_controller, work);
> +	struct octep_hp_cmd *hp_cmd;
> +	unsigned long flags;
> +
> +	/* Process all the hotplug commands */
> +	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
> +	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
> +		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list, struct octep_hp_cmd, list);
> +		list_del(&hp_cmd->list);
> +		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
> +
> +		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
> +		kfree(hp_cmd);
> +
> +		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
> +	}
> +	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
> +}
> +
> +static irqreturn_t octep_hp_intr_handler(int irq, void *data)
> +{
> +	struct octep_hp_controller *hp_ctrl = data;
> +	struct pci_dev *pdev = hp_ctrl->pdev;
> +	enum octep_hp_vec_type vec_type;
> +	struct octep_hp_cmd *hp_cmd;
> +	u64 slot_mask;
> +
> +	vec_type = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_ENA)) == irq ?
> +		OCTEP_HP_VEC_ENA : OCTEP_HP_VEC_DIS;

This is a bit unusual and complicated to do all this in one statement. But 
can't you just store that ena irq number into struct octep_hp_controller 
so you don't need to make the call here every time?

> +	slot_mask = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(vec_type));
> +	if (!slot_mask) {
> +		dev_err(&pdev->dev, "Invalid slot mask %llx\n", slot_mask);
> +		return IRQ_HANDLED;
> +	}
> +
> +	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
> +	if (!hp_cmd)
> +		return IRQ_HANDLED;
> +
> +	hp_cmd->slot_mask = slot_mask;
> +	hp_cmd->vec_type = vec_type;
> +
> +	/* Add the command to the list and schedule the work */
> +	spin_lock(&hp_ctrl->hp_cmd_lock);
> +	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
> +	spin_unlock(&hp_ctrl->hp_cmd_lock);
> +	schedule_work(&hp_ctrl->work);
> +
> +	/* Clear the interrupt */
> +	writeq(slot_mask, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(vec_type));
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static void octep_hp_deregister_slots(struct octep_hp_controller *hp_ctrl)
> +{
> +	struct octep_hp_slot *hp_slot, *tmp;
> +
> +	/* Deregister all the hotplug slots */
> +	list_for_each_entry_safe(hp_slot, tmp, &hp_ctrl->slot_list, list) {
> +		pci_hp_deregister(&hp_slot->slot);
> +		octep_hp_enable_pdev(hp_ctrl, hp_slot);
> +		list_del(&hp_slot->list);
> +		kfree(hp_slot);
> +	}
> +}
> +
> +static void octep_hp_controller_cleanup(void *data)
> +{
> +	struct octep_hp_controller *hp_ctrl = data;
> +
> +	pci_free_irq_vectors(hp_ctrl->pdev);
> +	flush_work(&hp_ctrl->work);
> +	octep_hp_deregister_slots(hp_ctrl);
> +}
> +
> +static int octep_hp_controller_setup(struct pci_dev *pdev, struct octep_hp_controller *hp_ctrl)
> +{
> +	struct device *dev = &pdev->dev;
> +	int ret;
> +
> +	ret = pcim_enable_device(pdev);
> +	if (ret) {
> +		dev_err(dev, "Failed to enable PCI device\n");
> +		return ret;
> +	}
> +
> +	ret = pcim_iomap_regions(pdev, BIT(0), OCTEP_HP_DRV_NAME);
> +	if (ret) {
> +		dev_err(dev, "Failed to request MMIO region\n");
> +		return ret;
> +	}
> +
> +	pci_set_master(pdev);
> +	pci_set_drvdata(pdev, hp_ctrl);
> +
> +	INIT_LIST_HEAD(&hp_ctrl->slot_list);
> +	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
> +	mutex_init(&hp_ctrl->slot_lock);
> +	spin_lock_init(&hp_ctrl->hp_cmd_lock);
> +	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
> +
> +	hp_ctrl->pdev = pdev;
> +	hp_ctrl->base = pcim_iomap_table(pdev)[0];
> +	if (!hp_ctrl->base) {
> +		dev_err(dev, "Failed to get device resource map\n");
> +		return -ENODEV;
> +	}
> +
> +	ret = pci_alloc_irq_vectors(pdev, 1, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS) + 1,
> +				    PCI_IRQ_MSIX);
> +	if (ret < 0) {
> +		dev_err(dev, "Failed to alloc MSI-X vectors");
> +		return ret;
> +	}
> +
> +	ret = devm_add_action(dev, octep_hp_controller_cleanup, hp_ctrl);
> +	if (ret) {
> +		dev_err(dev, "Failed to add action for controller cleanup\n");
> +		return ret;
> +	}
> +
> +	ret = devm_request_irq(dev, pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_ENA)),
> +			       octep_hp_intr_handler, 0, "octep_hp_ena", hp_ctrl);
> +	if (ret) {
> +		dev_err(dev, "Failed to register slot enable interrupt handle\n");
> +		return ret;
> +	}
> +
> +	ret = devm_request_irq(dev, pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS)),
> +			       octep_hp_intr_handler, 0, "octep_hp_dis", hp_ctrl);
> +	if (ret) {
> +		dev_err(dev, "Failed to register slot disable interrupt handle\n");
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int octep_hp_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> +{
> +	struct octep_hp_controller *hp_ctrl;
> +	struct pci_dev *tmp_pdev = NULL;
> +	u16 slot_number = 0;
> +	int ret;
> +
> +	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
> +	if (!hp_ctrl)
> +		return -ENOMEM;
> +
> +	ret = octep_hp_controller_setup(pdev, hp_ctrl);
> +	if (ret) {
> +		dev_err(&pdev->dev, "Failed to setup octep controller\n");

So both octep_hp_controller_setup() and this function print an error? 
Wouldn't the one from octep_hp_controller_setup() suffice?

-- 
 i.

> +		return ret;
> +	}
> +
> +	/*
> +	 * Register all hotplug slots. Hotplug controller is the first function of the PCI device.
> +	 * The hotplug slots are the remaining functions of the PCI device. They are removed from
> +	 * the bus and are added back when the hotplug event is triggered.

Reflow the comment to 80 chars.

> +	 */
> +	for_each_pci_dev(tmp_pdev) {
> +		if (octep_hp_slot(hp_ctrl, tmp_pdev)) {

Reverse logic & use continue.

> +			ret = octep_hp_register_slot(hp_ctrl, tmp_pdev, slot_number++);
> +			if (ret) {
> +				dev_err(&pdev->dev, "Failed to register hotplug slots.\n");
> +				return ret;
> +			}
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
> +static struct pci_device_id octep_hp_pci_map[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, OCTEP_DEVID_HP_CONTROLLER) },
> +	{ 0 },
> +};
> +
> +static struct pci_driver octep_hp = {
> +	.name     = OCTEP_HP_DRV_NAME,
> +	.id_table = octep_hp_pci_map,
> +	.probe    = octep_hp_pci_probe,
> +};
> +
> +module_pci_driver(octep_hp);
> +
> +MODULE_LICENSE("GPL");
> +MODULE_AUTHOR("Marvell");
> +MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
> 

Re: [PATCH] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

>On Tue, 20 Aug 2024, Shijith Thotton wrote:
>
>> This patch introduces a PCI hotplug controller driver for the OCTEON
>> PCIe device, a multi-function PCIe device where the first function acts
>> as a hotplug controller. It is equipped with MSI-x interrupts to notify
>> the host of hotplug events from the OCTEON firmware.
>>
>> The driver facilitates the hotplugging of non-controller functions
>> within the same device. During probe, non-controller functions are
>> removed and registered as PCI hotplug slots. The slots are added back
>> only upon request from the device firmware. The driver also allows the
>> enabling and disabling of the slots via sysfs slot entries, provided by
>> the PCI hotplug framework.
>>
>> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>> ---
>>
>> This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
>> controller. The OCTEON PCIe device is a multi-function device where the first
>> function acts as a PCI hotplug controller.
>>
>>               +--------------------------------+
>>               |           Root Port            |
>>               +--------------------------------+
>>                               |
>>                              PCIe
>>                               |
>> +---------------------------------------------------------------+
>> |              OCTEON PCIe Multifunction Device                 |
>> +---------------------------------------------------------------+
>>             |                    |              |            |
>>             |                    |              |            |
>> +---------------------+  +----------------+  +-----+  +----------------+
>> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>> +---------------------+  +----------------+  +-----+  +----------------+
>>             |
>>             |
>> +-------------------------+
>> |   Controller Firmware   |
>> +-------------------------+
>>
>> The hotplug controller driver facilitates the hotplugging of non-controller
>> functions in the same device. During the probe of the driver, the non-
>controller
>> function are removed and registered as PCI hotplug slots. They are added
>back
>> only upon request from the device firmware. The driver also allows the user
>to
>> enable/disable the functions using sysfs slot entries provided by PCI hotplug
>> framework.
>>
>> This solution adopts a hardware + software approach for several reasons:
>>
>> 1. To reduce hardware implementation cost. Supporting complete hotplug
>>    capability within the card would require a PCI switch implemented within.
>>
>> 2. In the multi-function device, non-controller functions can act as emulated
>>    devices. The firmware can dynamically enable or disable them at runtime.
>>
>> 3. Not all root ports support PCI hotplug. This approach provides greater
>>    flexibility and compatibility across different hardware configurations.
>>
>> The hotplug controller function is lightweight and is equipped with MSI-x
>> interrupts to notify the host about hotplug events. Upon receiving an
>> interrupt, the hotplug register is read, and the required function is enabled
>> or disabled.
>>
>> This driver will be beneficial for managing PCI hotplug events on the OCTEON
>> PCIe device without requiring complex hardware solutions.
>>
>>  MAINTAINERS                    |   6 +
>>  drivers/pci/hotplug/Kconfig    |  10 +
>>  drivers/pci/hotplug/Makefile   |   1 +
>>  drivers/pci/hotplug/octep_hp.c | 351
>+++++++++++++++++++++++++++++++++
>>  4 files changed, 368 insertions(+)
>>  create mode 100644 drivers/pci/hotplug/octep_hp.c
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 42decde38320..7b5a618eed1c 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
>>  R:	vattunuru@marvell.com
>>  F:	drivers/vdpa/octeon_ep/
>>
>> +MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
>> +R:	Shijith Thotton <sthotton@marvell.com>
>> +R:	Vamsi Attunuru <vattunuru@marvell.com>
>> +S:	Supported
>> +F:	drivers/pci/hotplug/octep_hp.c
>> +
>>  MATROX FRAMEBUFFER DRIVER
>>  L:	linux-fbdev@vger.kernel.org
>>  S:	Orphan
>> diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
>> index 1472aef0fb81..2e38fd25f7ef 100644
>> --- a/drivers/pci/hotplug/Kconfig
>> +++ b/drivers/pci/hotplug/Kconfig
>> @@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
>>
>>  	  When in doubt, say Y.
>>
>> +config HOTPLUG_PCI_OCTEONEP
>> +	bool "OCTEON PCI device Hotplug controller driver"
>> +	depends on HOTPLUG_PCI
>> +	help
>> +	  Say Y here if you have an OCTEON PCIe device with a hotplug
>> +	  controller. This driver enables the non-controller functions of the
>> +	  device to be registered as hotplug slots.
>> +
>> +	  When in doubt, say N.
>> +
>>  endif # HOTPLUG_PCI
>> diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
>> index 240c99517d5e..40aaf31fe338 100644
>> --- a/drivers/pci/hotplug/Makefile
>> +++ b/drivers/pci/hotplug/Makefile
>> @@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+=
>rpaphp.o
>>  obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
>>  obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
>>  obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
>> +obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
>>
>>  # acpiphp_ibm extends acpiphp, so should be linked afterwards.
>>
>> diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
>> new file mode 100644
>> index 000000000000..efeb542d4993
>> --- /dev/null
>> +++ b/drivers/pci/hotplug/octep_hp.c
>> @@ -0,0 +1,351 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/* Copyright (C) 2024 Marvell. */
>> +
>> +#include <linux/delay.h>
>> +#include <linux/init.h>
>> +#include <linux/kernel.h>
>> +#include <linux/module.h>
>> +#include <linux/pci.h>
>> +#include <linux/pci_hotplug.h>
>> +#include <linux/slab.h>
>> +
>> +#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
>> +#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
>> +#define OCTEP_HP_DRV_NAME       "octep_hp"
>> +
>> +/* Interrupt vectors for hotplug enable and disable events. */
>> +enum octep_hp_vec_type {
>> +	OCTEP_HP_VEC_ENA,
>> +	OCTEP_HP_VEC_DIS,
>
>If I understand your code right, these cannot be arbitrary numbers but has
>to be specific numbers to match the vector number so you should explicitly
>set them to those values.

octep_hp_vec_type is used to specify the type of hotplug interrupt. There are
two types of hotplug interrupts, enable and disable. The vectors for these
interrupts are formed using the macros OCTEP_HP_INTR_VECTOR(x) where x is the
type of interrupt. The vectors for enable and disable interrupts are 16 and 17
respectively.

I have not set them in enum since the vectors are sequential. I will change it as
below to make it clearer.

enum octep_hp_intr_type {
        OCTEP_HP_INTR_INVALID = -1,
        OCTEP_HP_INTR_ENA = 0,
        OCTEP_HP_INTR_DIS = 1,
};

>> +};
>> +
>> +struct octep_hp_cmd {
>> +	struct list_head list;
>
>Missing include for list_head.

Will add.

>
>> +	enum octep_hp_vec_type vec_type;
>> +	u64 slot_mask;
>> +};
>> +
>> +struct octep_hp_slot {
>> +	struct list_head list;
>> +	struct hotplug_slot slot;
>> +	u16 slot_number;
>> +	struct pci_dev *hp_pdev;
>> +	unsigned int hp_devfn;
>> +	struct octep_hp_controller *ctrl;
>> +};
>> +
>> +struct octep_hp_controller {
>> +	void __iomem *base;
>> +	struct pci_dev *pdev;
>> +	struct work_struct work;
>
>Missing include.

Will add.

>
>> +	struct list_head slot_list;
>> +	struct mutex slot_lock; /* Protects slot_list */
>
>Missing include.

Will add.

>
>> +	struct list_head hp_cmd_list;
>> +	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
>> +};
>> +
>> +static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl,
>struct octep_hp_slot *hp_slot)
>> +{
>> +	mutex_lock(&hp_ctrl->slot_lock);
>
>Use guard().

Will change to use guard().

>
>> +	if (hp_slot->hp_pdev) {
>> +		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already
>enabled\n", hp_slot->slot_number);
>> +		mutex_unlock(&hp_ctrl->slot_lock);
>> +		return;
>> +	}
>> +
>> +	/* Scan the device and add it to the bus */
>> +	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus,
>hp_slot->hp_devfn);
>> +	pci_bus_assign_resources(hp_ctrl->pdev->bus);
>> +	pci_bus_add_device(hp_slot->hp_pdev);
>> +
>> +	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n", hp_slot-
>>slot_number);
>
>Missing include for dev_dbg().
>

Will add.

>> +	mutex_unlock(&hp_ctrl->slot_lock);
>> +}
>> +
>> +static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
>> +				  struct octep_hp_slot *hp_slot)
>> +{
>> +	mutex_lock(&hp_ctrl->slot_lock);
>
>Use guard().

Will change.

>
>> +	if (!hp_slot->hp_pdev) {
>> +		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n",
>hp_slot->slot_number);
>> +		mutex_unlock(&hp_ctrl->slot_lock);
>> +		return;
>> +	}
>> +
>> +	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n", hp_slot-
>>slot_number);
>> +
>> +	/* Remove the device from the bus */
>> +	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
>> +	hp_slot->hp_pdev = NULL;
>> +	mutex_unlock(&hp_ctrl->slot_lock);
>> +}
>> +
>> +static int octep_hp_enable_slot(struct hotplug_slot *slot)
>> +{
>> +	struct octep_hp_slot *hp_slot = container_of(slot, struct
>octep_hp_slot, slot);
>
>Missing include for container_of().

Will add.

>
>> +
>> +	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
>> +	return 0;
>> +}
>> +
>> +static int octep_hp_disable_slot(struct hotplug_slot *slot)
>> +{
>> +	struct octep_hp_slot *hp_slot = container_of(slot, struct
>octep_hp_slot, slot);
>> +
>> +	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
>> +	return 0;
>> +}
>> +
>> +static struct hotplug_slot_ops octep_hp_slot_ops = {
>> +	.enable_slot = octep_hp_enable_slot,
>> +	.disable_slot = octep_hp_disable_slot,
>> +};
>> +
>> +#define SLOT_NAME_SIZE 16
>> +static int octep_hp_register_slot(struct octep_hp_controller *hp_ctrl, struct
>pci_dev *pdev,
>> +				  u16 slot_number)
>> +{
>> +	char slot_name[SLOT_NAME_SIZE];
>> +	struct octep_hp_slot *hp_slot;
>> +	int ret;
>> +
>> +	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
>> +	if (!hp_slot)
>> +		return -ENOMEM;
>> +
>> +	hp_slot->ctrl = hp_ctrl;
>> +	hp_slot->hp_pdev = pdev;
>> +	hp_slot->hp_devfn = pdev->devfn;
>> +	hp_slot->slot_number = slot_number;
>> +	hp_slot->slot.ops = &octep_hp_slot_ops;
>> +
>> +	snprintf(slot_name, SLOT_NAME_SIZE, "octep_hp_%u", slot_number);
>
>SLOT_NAME_SIZE -> sizeof(slot_name)

Will change.

>
>> +	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
>PCI_SLOT(pdev->devfn), slot_name);
>> +	if (ret) {
>> +		dev_err(&pdev->dev, "Failed to register hotplug slot %u\n",
>slot_number);
>> +		kfree(hp_slot);
>> +		return ret;
>> +	}
>> +
>> +	octep_hp_disable_pdev(hp_ctrl, hp_slot);
>> +	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
>> +
>> +	return 0;
>> +}
>> +
>> +static bool octep_hp_slot(struct octep_hp_controller *hp_ctrl, struct pci_dev
>*pdev)
>> +{
>> +	/* Check if the PCI device can be hotplugged */
>> +	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
>> +		PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
>> +}
>> +
>> +static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
>struct octep_hp_cmd *hp_cmd)
>> +{
>> +	struct octep_hp_slot *hp_slot;
>> +
>> +	/* Enable or disable the slots based on the slot mask */
>> +	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
>> +		if (hp_cmd->slot_mask & BIT(hp_slot->slot_number)) {
>
>Reverse logic & use continue ?

Sure. Will change.

>
>> +			if (hp_cmd->vec_type == OCTEP_HP_VEC_ENA)
>> +				octep_hp_enable_pdev(hp_ctrl, hp_slot);
>> +			else
>> +				octep_hp_disable_pdev(hp_ctrl, hp_slot);
>> +		}
>> +	}
>> +}
>> +
>> +static void octep_hp_work_handler(struct work_struct *work)
>> +{
>> +	struct octep_hp_controller *hp_ctrl = container_of(work, struct
>octep_hp_controller, work);
>> +	struct octep_hp_cmd *hp_cmd;
>> +	unsigned long flags;
>> +
>> +	/* Process all the hotplug commands */
>> +	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>> +	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
>> +		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list, struct
>octep_hp_cmd, list);
>> +		list_del(&hp_cmd->list);
>> +		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>> +
>> +		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
>> +		kfree(hp_cmd);
>> +
>> +		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>> +	}
>> +	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>> +}
>> +
>> +static irqreturn_t octep_hp_intr_handler(int irq, void *data)
>> +{
>> +	struct octep_hp_controller *hp_ctrl = data;
>> +	struct pci_dev *pdev = hp_ctrl->pdev;
>> +	enum octep_hp_vec_type vec_type;
>> +	struct octep_hp_cmd *hp_cmd;
>> +	u64 slot_mask;
>> +
>> +	vec_type = pci_irq_vector(pdev,
>OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_ENA)) == irq ?
>> +		OCTEP_HP_VEC_ENA : OCTEP_HP_VEC_DIS;
>
>This is a bit unusual and complicated to do all this in one statement. But
>can't you just store that ena irq number into struct octep_hp_controller
>so you don't need to make the call here every time?
>

Will move this logic to a new function to get the vector type based on the irq
number. Avoided storing the irq number in the controller structure as we don't
expect a lot of hotplug interrupts.

>> +	slot_mask = readq(hp_ctrl->base +
>OCTEP_HP_INTR_OFFSET(vec_type));
>> +	if (!slot_mask) {
>> +		dev_err(&pdev->dev, "Invalid slot mask %llx\n", slot_mask);
>> +		return IRQ_HANDLED;
>> +	}
>> +
>> +	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
>> +	if (!hp_cmd)
>> +		return IRQ_HANDLED;
>> +
>> +	hp_cmd->slot_mask = slot_mask;
>> +	hp_cmd->vec_type = vec_type;
>> +
>> +	/* Add the command to the list and schedule the work */
>> +	spin_lock(&hp_ctrl->hp_cmd_lock);
>> +	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
>> +	spin_unlock(&hp_ctrl->hp_cmd_lock);
>> +	schedule_work(&hp_ctrl->work);
>> +
>> +	/* Clear the interrupt */
>> +	writeq(slot_mask, hp_ctrl->base +
>OCTEP_HP_INTR_OFFSET(vec_type));
>> +
>> +	return IRQ_HANDLED;
>> +}
>> +
>> +static void octep_hp_deregister_slots(struct octep_hp_controller *hp_ctrl)
>> +{
>> +	struct octep_hp_slot *hp_slot, *tmp;
>> +
>> +	/* Deregister all the hotplug slots */
>> +	list_for_each_entry_safe(hp_slot, tmp, &hp_ctrl->slot_list, list) {
>> +		pci_hp_deregister(&hp_slot->slot);
>> +		octep_hp_enable_pdev(hp_ctrl, hp_slot);
>> +		list_del(&hp_slot->list);
>> +		kfree(hp_slot);
>> +	}
>> +}
>> +
>> +static void octep_hp_controller_cleanup(void *data)
>> +{
>> +	struct octep_hp_controller *hp_ctrl = data;
>> +
>> +	pci_free_irq_vectors(hp_ctrl->pdev);
>> +	flush_work(&hp_ctrl->work);
>> +	octep_hp_deregister_slots(hp_ctrl);
>> +}
>> +
>> +static int octep_hp_controller_setup(struct pci_dev *pdev, struct
>octep_hp_controller *hp_ctrl)
>> +{
>> +	struct device *dev = &pdev->dev;
>> +	int ret;
>> +
>> +	ret = pcim_enable_device(pdev);
>> +	if (ret) {
>> +		dev_err(dev, "Failed to enable PCI device\n");
>> +		return ret;
>> +	}
>> +
>> +	ret = pcim_iomap_regions(pdev, BIT(0), OCTEP_HP_DRV_NAME);
>> +	if (ret) {
>> +		dev_err(dev, "Failed to request MMIO region\n");
>> +		return ret;
>> +	}
>> +
>> +	pci_set_master(pdev);
>> +	pci_set_drvdata(pdev, hp_ctrl);
>> +
>> +	INIT_LIST_HEAD(&hp_ctrl->slot_list);
>> +	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
>> +	mutex_init(&hp_ctrl->slot_lock);
>> +	spin_lock_init(&hp_ctrl->hp_cmd_lock);
>> +	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
>> +
>> +	hp_ctrl->pdev = pdev;
>> +	hp_ctrl->base = pcim_iomap_table(pdev)[0];
>> +	if (!hp_ctrl->base) {
>> +		dev_err(dev, "Failed to get device resource map\n");
>> +		return -ENODEV;
>> +	}
>> +
>> +	ret = pci_alloc_irq_vectors(pdev, 1,
>OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS) + 1,
>> +				    PCI_IRQ_MSIX);
>> +	if (ret < 0) {
>> +		dev_err(dev, "Failed to alloc MSI-X vectors");
>> +		return ret;
>> +	}
>> +
>> +	ret = devm_add_action(dev, octep_hp_controller_cleanup, hp_ctrl);
>> +	if (ret) {
>> +		dev_err(dev, "Failed to add action for controller cleanup\n");
>> +		return ret;
>> +	}
>> +
>> +	ret = devm_request_irq(dev, pci_irq_vector(pdev,
>OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_ENA)),
>> +			       octep_hp_intr_handler, 0, "octep_hp_ena",
>hp_ctrl);
>> +	if (ret) {
>> +		dev_err(dev, "Failed to register slot enable interrupt
>handle\n");
>> +		return ret;
>> +	}
>> +
>> +	ret = devm_request_irq(dev, pci_irq_vector(pdev,
>OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS)),
>> +			       octep_hp_intr_handler, 0, "octep_hp_dis", hp_ctrl);
>> +	if (ret) {
>> +		dev_err(dev, "Failed to register slot disable interrupt
>handle\n");
>> +		return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int octep_hp_pci_probe(struct pci_dev *pdev, const struct
>pci_device_id *id)
>> +{
>> +	struct octep_hp_controller *hp_ctrl;
>> +	struct pci_dev *tmp_pdev = NULL;
>> +	u16 slot_number = 0;
>> +	int ret;
>> +
>> +	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
>> +	if (!hp_ctrl)
>> +		return -ENOMEM;
>> +
>> +	ret = octep_hp_controller_setup(pdev, hp_ctrl);
>> +	if (ret) {
>> +		dev_err(&pdev->dev, "Failed to setup octep controller\n");
>
>So both octep_hp_controller_setup() and this function print an error?
>Wouldn't the one from octep_hp_controller_setup() suffice?

Will remove the extra print.

>--
> i.
>
>> +		return ret;
>> +	}
>> +
>> +	/*
>> +	 * Register all hotplug slots. Hotplug controller is the first function of
>the PCI device.
>> +	 * The hotplug slots are the remaining functions of the PCI device. They
>are removed from
>> +	 * the bus and are added back when the hotplug event is triggered.
>
>Reflow the comment to 80 chars.

Will change.

>
>> +	 */
>> +	for_each_pci_dev(tmp_pdev) {
>> +		if (octep_hp_slot(hp_ctrl, tmp_pdev)) {
>
>Reverse logic & use continue.
>

Will change.

>> +			ret = octep_hp_register_slot(hp_ctrl, tmp_pdev,
>slot_number++);
>> +			if (ret) {
>> +				dev_err(&pdev->dev, "Failed to register
>hotplug slots.\n");
>> +				return ret;
>> +			}
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
>> +static struct pci_device_id octep_hp_pci_map[] = {
>> +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
>OCTEP_DEVID_HP_CONTROLLER) },
>> +	{ 0 },
>> +};
>> +
>> +static struct pci_driver octep_hp = {
>> +	.name     = OCTEP_HP_DRV_NAME,
>> +	.id_table = octep_hp_pci_map,
>> +	.probe    = octep_hp_pci_probe,
>> +};
>> +
>> +module_pci_driver(octep_hp);
>> +
>> +MODULE_LICENSE("GPL");
>> +MODULE_AUTHOR("Marvell");
>> +MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
>>

Thanks for the review.
Shijith

Re: [PATCH] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Jonathan Cameron]

On Tue, 20 Aug 2024 20:57:20 +0530
Shijith Thotton <sthotton@marvell.com> wrote:

> This patch introduces a PCI hotplug controller driver for the OCTEON
> PCIe device, a multi-function PCIe device where the first function acts
> as a hotplug controller. It is equipped with MSI-x interrupts to notify
> the host of hotplug events from the OCTEON firmware.
> 
> The driver facilitates the hotplugging of non-controller functions
> within the same device. During probe, non-controller functions are
> removed and registered as PCI hotplug slots. The slots are added back
> only upon request from the device firmware. The driver also allows the
> enabling and disabling of the slots via sysfs slot entries, provided by
> the PCI hotplug framework.
> 
> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
I was curious, so drive by review.

What was Vamsi's involvement?  Given Shijith sent this
I'd guess co developer?  In which Co-developed-by tag
should be used.

Various random comments on things inline.

> ---
> 
> This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
> controller. The OCTEON PCIe device is a multi-function device where the first
> function acts as a PCI hotplug controller.
> 
>               +--------------------------------+
>               |           Root Port            |
>               +--------------------------------+
>                               |
>                              PCIe
>                               |
> +---------------------------------------------------------------+
> |              OCTEON PCIe Multifunction Device                 |
> +---------------------------------------------------------------+
>             |                    |              |            |
>             |                    |              |            |
> +---------------------+  +----------------+  +-----+  +----------------+
> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
> +---------------------+  +----------------+  +-----+  +----------------+
>             |
>             |
> +-------------------------+
> |   Controller Firmware   |
> +-------------------------+
> 
> The hotplug controller driver facilitates the hotplugging of non-controller
> functions in the same device. During the probe of the driver, the non-controller
> function are removed and registered as PCI hotplug slots. They are added back
> only upon request from the device firmware. The driver also allows the user to
> enable/disable the functions using sysfs slot entries provided by PCI hotplug
> framework.
> 
> This solution adopts a hardware + software approach for several reasons:
> 
> 1. To reduce hardware implementation cost. Supporting complete hotplug
>    capability within the card would require a PCI switch implemented within.
> 
> 2. In the multi-function device, non-controller functions can act as emulated
>    devices. The firmware can dynamically enable or disable them at runtime.
> 
> 3. Not all root ports support PCI hotplug. This approach provides greater
>    flexibility and compatibility across different hardware configurations.
> 
> The hotplug controller function is lightweight and is equipped with MSI-x
> interrupts to notify the host about hotplug events. Upon receiving an
> interrupt, the hotplug register is read, and the required function is enabled
> or disabled.
> 
> This driver will be beneficial for managing PCI hotplug events on the OCTEON
> PCIe device without requiring complex hardware solutions.
> 
>  MAINTAINERS                    |   6 +
>  drivers/pci/hotplug/Kconfig    |  10 +
>  drivers/pci/hotplug/Makefile   |   1 +
>  drivers/pci/hotplug/octep_hp.c | 351 +++++++++++++++++++++++++++++++++
>  4 files changed, 368 insertions(+)
>  create mode 100644 drivers/pci/hotplug/octep_hp.c
> 


> diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
> new file mode 100644
> index 000000000000..efeb542d4993
> --- /dev/null
> +++ b/drivers/pci/hotplug/octep_hp.c
> @@ -0,0 +1,351 @@

> +struct octep_hp_controller {
> +	void __iomem *base;
> +	struct pci_dev *pdev;
> +	struct work_struct work;
> +	struct list_head slot_list;
> +	struct mutex slot_lock; /* Protects slot_list */
> +	struct list_head hp_cmd_list;
> +	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
> +};

> +static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
> +				  struct octep_hp_slot *hp_slot)
> +{
> +	mutex_lock(&hp_ctrl->slot_lock);

guard(mutex)(&hp_ctrl->slot_lock);

Same for other simple cases where we can avoid explicit unlock in error paths
+ main path.


> +	if (!hp_slot->hp_pdev) {
> +		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n", hp_slot->slot_number);
> +		mutex_unlock(&hp_ctrl->slot_lock);
> +		return;
> +	}
> +
> +	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n", hp_slot->slot_number);
> +
> +	/* Remove the device from the bus */
> +	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
> +	hp_slot->hp_pdev = NULL;
> +	mutex_unlock(&hp_ctrl->slot_lock);
> +}

> +#define SLOT_NAME_SIZE 16
> +static int octep_hp_register_slot(struct octep_hp_controller *hp_ctrl, struct pci_dev *pdev,
> +				  u16 slot_number)
> +{
> +	char slot_name[SLOT_NAME_SIZE];
> +	struct octep_hp_slot *hp_slot;
> +	int ret;
> +
> +	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
> +	if (!hp_slot)
> +		return -ENOMEM;
> +
> +	hp_slot->ctrl = hp_ctrl;
> +	hp_slot->hp_pdev = pdev;
> +	hp_slot->hp_devfn = pdev->devfn;
> +	hp_slot->slot_number = slot_number;
> +	hp_slot->slot.ops = &octep_hp_slot_ops;
> +
> +	snprintf(slot_name, SLOT_NAME_SIZE, "octep_hp_%u", slot_number);
> +	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus, PCI_SLOT(pdev->devfn), slot_name);
> +	if (ret) {
> +		dev_err(&pdev->dev, "Failed to register hotplug slot %u\n", slot_number);
> +		kfree(hp_slot);
> +		return ret;
		return dev_err_probe() in here as well.

> +	}
> +
> +	octep_hp_disable_pdev(hp_ctrl, hp_slot);
> +	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
> +
> +	return 0;
> +}
> +
> +static bool octep_hp_slot(struct octep_hp_controller *hp_ctrl, struct pci_dev *pdev)
> +{
> +	/* Check if the PCI device can be hotplugged */
> +	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
> +		PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
> +}
> +
> +static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl, struct octep_hp_cmd *hp_cmd)

Line break as that's getting long for little reason.

> +{
> +	struct octep_hp_slot *hp_slot;
> +
> +	/* Enable or disable the slots based on the slot mask */
> +	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
> +		if (hp_cmd->slot_mask & BIT(hp_slot->slot_number)) {
> +			if (hp_cmd->vec_type == OCTEP_HP_VEC_ENA)
> +				octep_hp_enable_pdev(hp_ctrl, hp_slot);
> +			else
> +				octep_hp_disable_pdev(hp_ctrl, hp_slot);
> +		}
> +	}
> +}
> +
> +static void octep_hp_work_handler(struct work_struct *work)
> +{
> +	struct octep_hp_controller *hp_ctrl = container_of(work, struct octep_hp_controller, work);

Add a line break.

> +	struct octep_hp_cmd *hp_cmd;
> +	unsigned long flags;

...

> +
> +static void octep_hp_deregister_slots(struct octep_hp_controller *hp_ctrl)
> +{
> +	struct octep_hp_slot *hp_slot, *tmp;
> +
> +	/* Deregister all the hotplug slots */
> +	list_for_each_entry_safe(hp_slot, tmp, &hp_ctrl->slot_list, list) {
> +		pci_hp_deregister(&hp_slot->slot);
> +		octep_hp_enable_pdev(hp_ctrl, hp_slot);
> +		list_del(&hp_slot->list);
> +		kfree(hp_slot);

As below, maybe just register cleanup for one slot at a time.
That will need a bit of magic to get form the slot to the hp_ctrl though.

Plus point is simpler logic to follow.


> +	}
> +}
> +
> +static void octep_hp_controller_cleanup(void *data)
> +{
> +	struct octep_hp_controller *hp_ctrl = data;
> +
> +	pci_free_irq_vectors(hp_ctrl->pdev);
> +	flush_work(&hp_ctrl->work);
I'd prefer to see this broken up so we have
simple steps.
1. Add action 1 in probe.
2. Add cleanup for action 1 in probe
3. Add action 2 in probe etc.

Otherwise it can be a bit tricky to review.

> +	octep_hp_deregister_slots(hp_ctrl);

This is particular is no where near the place where the
slots are registered.
May be better to register one cleanup function call
per slot as then it will also be nice and simple to follow.


> +}
> +
> +static int octep_hp_controller_setup(struct pci_dev *pdev, struct octep_hp_controller *hp_ctrl)
> +{
> +	struct device *dev = &pdev->dev;
> +	int ret;
> +
> +	ret = pcim_enable_device(pdev);
> +	if (ret) {
> +		dev_err(dev, "Failed to enable PCI device\n");
Only called from probe, so
		return dev_err_probe()

> +		return ret;
> +	}
> +
> +	ret = pcim_iomap_regions(pdev, BIT(0), OCTEP_HP_DRV_NAME);
Given there is a patch on list deprecating this, reconsider.
https://lore.kernel.org/all/20240807083018.8734-2-pstanner@redhat.com/
+CC Philipp


> +	if (ret) {
> +		dev_err(dev, "Failed to request MMIO region\n");
> +		return ret;

Same thing on return dev_err_probe here and later.

> +	}
> +
> +	pci_set_master(pdev);
> +	pci_set_drvdata(pdev, hp_ctrl);
> +
> +	INIT_LIST_HEAD(&hp_ctrl->slot_list);
> +	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
> +	mutex_init(&hp_ctrl->slot_lock);
> +	spin_lock_init(&hp_ctrl->hp_cmd_lock);
> +	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
> +
> +	hp_ctrl->pdev = pdev;
> +	hp_ctrl->base = pcim_iomap_table(pdev)[0];
> +	if (!hp_ctrl->base) {
> +		dev_err(dev, "Failed to get device resource map\n");
> +		return -ENODEV;
> +	}
> +
> +	ret = pci_alloc_irq_vectors(pdev, 1, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS) + 1,
> +				    PCI_IRQ_MSIX);
> +	if (ret < 0) {
> +		dev_err(dev, "Failed to alloc MSI-X vectors");
> +		return ret;
> +	}
> +
> +	ret = devm_add_action(dev, octep_hp_controller_cleanup, hp_ctrl);
> +	if (ret) {
> +		dev_err(dev, "Failed to add action for controller cleanup\n");
> +		return ret;
> +	}
> +
> +	ret = devm_request_irq(dev, pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_ENA)),
> +			       octep_hp_intr_handler, 0, "octep_hp_ena", hp_ctrl);
> +	if (ret) {
> +		dev_err(dev, "Failed to register slot enable interrupt handle\n");
> +		return ret;
> +	}
> +
> +	ret = devm_request_irq(dev, pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS)),
> +			       octep_hp_intr_handler, 0, "octep_hp_dis", hp_ctrl);
> +	if (ret) {
> +		dev_err(dev, "Failed to register slot disable interrupt handle\n");
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int octep_hp_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> +{
> +	struct octep_hp_controller *hp_ctrl;
> +	struct pci_dev *tmp_pdev = NULL;
> +	u16 slot_number = 0;
> +	int ret;
> +
> +	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
> +	if (!hp_ctrl)
> +		return -ENOMEM;
> +
> +	ret = octep_hp_controller_setup(pdev, hp_ctrl);
> +	if (ret) {
> +		dev_err(&pdev->dev, "Failed to setup octep controller\n");
> +		return ret;
return dev_err_probe()
actually no. Drop it. There are more informative error prints for all the
conditions. We probably don't care that the end result is this specific
function fails (more than an irq request failed etc).


> +	}
> +
> +	/*
> +	 * Register all hotplug slots. Hotplug controller is the first function of the PCI device.
> +	 * The hotplug slots are the remaining functions of the PCI device. They are removed from
> +	 * the bus and are added back when the hotplug event is triggered.
> +	 */
> +	for_each_pci_dev(tmp_pdev) {
> +		if (octep_hp_slot(hp_ctrl, tmp_pdev)) {
What IIpo said on this.  No one likes deep indent ;)
> +			ret = octep_hp_register_slot(hp_ctrl, tmp_pdev, slot_number++);
> +			if (ret) {
> +				dev_err(&pdev->dev, "Failed to register hotplug slots.\n");
> +				return ret;

return dev_err_probe();

Cleaner in general even if no chance of deferral.



> +			}
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
> +static struct pci_device_id octep_hp_pci_map[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, OCTEP_DEVID_HP_CONTROLLER) },
> +	{ 0 },

{ }
Is sufficient. I can't remember if that's common style in PCI or not though.

> +};

Re: [PATCH] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[kernel test robot]

Hi Shijith,

kernel test robot noticed the following build errors:

[auto build test ERROR on pci/next]
[also build test ERROR on pci/for-linus linus/master v6.11-rc4 next-20240821]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Shijith-Thotton/PCI-hotplug-Add-OCTEON-PCI-hotplug-controller-driver/20240820-233005
base:   https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next
patch link:    https://lore.kernel.org/r/20240820152734.642533-1-sthotton%40marvell.com
patch subject: [PATCH] PCI: hotplug: Add OCTEON PCI hotplug controller driver
config: mips-allyesconfig (https://download.01.org/0day-ci/archive/20240821/202408212127.ed49pQkJ-lkp@intel.com/config)
compiler: mips-linux-gcc (GCC) 14.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240821/202408212127.ed49pQkJ-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202408212127.ed49pQkJ-lkp@intel.com/

All errors (new ones prefixed by >>):

   drivers/pci/hotplug/octep_hp.c: In function 'octep_hp_intr_handler':
>> drivers/pci/hotplug/octep_hp.c:190:21: error: implicit declaration of function 'readq'; did you mean 'readl'? [-Wimplicit-function-declaration]
     190 |         slot_mask = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(vec_type));
         |                     ^~~~~
         |                     readl
>> drivers/pci/hotplug/octep_hp.c:210:9: error: implicit declaration of function 'writeq'; did you mean 'writel'? [-Wimplicit-function-declaration]
     210 |         writeq(slot_mask, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(vec_type));
         |         ^~~~~~
         |         writel


vim +190 drivers/pci/hotplug/octep_hp.c

   178	
   179	static irqreturn_t octep_hp_intr_handler(int irq, void *data)
   180	{
   181		struct octep_hp_controller *hp_ctrl = data;
   182		struct pci_dev *pdev = hp_ctrl->pdev;
   183		enum octep_hp_vec_type vec_type;
   184		struct octep_hp_cmd *hp_cmd;
   185		u64 slot_mask;
   186	
   187		vec_type = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_ENA)) == irq ?
   188			OCTEP_HP_VEC_ENA : OCTEP_HP_VEC_DIS;
   189	
 > 190		slot_mask = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(vec_type));
   191		if (!slot_mask) {
   192			dev_err(&pdev->dev, "Invalid slot mask %llx\n", slot_mask);
   193			return IRQ_HANDLED;
   194		}
   195	
   196		hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
   197		if (!hp_cmd)
   198			return IRQ_HANDLED;
   199	
   200		hp_cmd->slot_mask = slot_mask;
   201		hp_cmd->vec_type = vec_type;
   202	
   203		/* Add the command to the list and schedule the work */
   204		spin_lock(&hp_ctrl->hp_cmd_lock);
   205		list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
   206		spin_unlock(&hp_ctrl->hp_cmd_lock);
   207		schedule_work(&hp_ctrl->work);
   208	
   209		/* Clear the interrupt */
 > 210		writeq(slot_mask, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(vec_type));
   211	
   212		return IRQ_HANDLED;
   213	}
   214	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

>> This patch introduces a PCI hotplug controller driver for the OCTEON
>> PCIe device, a multi-function PCIe device where the first function acts
>> as a hotplug controller. It is equipped with MSI-x interrupts to notify
>> the host of hotplug events from the OCTEON firmware.
>>
>> The driver facilitates the hotplugging of non-controller functions
>> within the same device. During probe, non-controller functions are
>> removed and registered as PCI hotplug slots. The slots are added back
>> only upon request from the device firmware. The driver also allows the
>> enabling and disabling of the slots via sysfs slot entries, provided by
>> the PCI hotplug framework.
>>
>> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>I was curious, so drive by review.
>
>What was Vamsi's involvement?  Given Shijith sent this
>I'd guess co developer?  In which Co-developed-by tag
>should be used.
>

Will add the co-developer tag.

>Various random comments on things inline.
>
>> ---
>>
>> This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
>> controller. The OCTEON PCIe device is a multi-function device where the first
>> function acts as a PCI hotplug controller.
>>
>>               +--------------------------------+
>>               |           Root Port            |
>>               +--------------------------------+
>>                               |
>>                              PCIe
>>                               |
>> +---------------------------------------------------------------+
>> |              OCTEON PCIe Multifunction Device                 |
>> +---------------------------------------------------------------+
>>             |                    |              |            |
>>             |                    |              |            |
>> +---------------------+  +----------------+  +-----+  +----------------+
>> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>> +---------------------+  +----------------+  +-----+  +----------------+
>>             |
>>             |
>> +-------------------------+
>> |   Controller Firmware   |
>> +-------------------------+
>>
>> The hotplug controller driver facilitates the hotplugging of non-controller
>> functions in the same device. During the probe of the driver, the non-
>controller
>> function are removed and registered as PCI hotplug slots. They are added
>back
>> only upon request from the device firmware. The driver also allows the user
>to
>> enable/disable the functions using sysfs slot entries provided by PCI hotplug
>> framework.
>>
>> This solution adopts a hardware + software approach for several reasons:
>>
>> 1. To reduce hardware implementation cost. Supporting complete hotplug
>>    capability within the card would require a PCI switch implemented within.
>>
>> 2. In the multi-function device, non-controller functions can act as emulated
>>    devices. The firmware can dynamically enable or disable them at runtime.
>>
>> 3. Not all root ports support PCI hotplug. This approach provides greater
>>    flexibility and compatibility across different hardware configurations.
>>
>> The hotplug controller function is lightweight and is equipped with MSI-x
>> interrupts to notify the host about hotplug events. Upon receiving an
>> interrupt, the hotplug register is read, and the required function is enabled
>> or disabled.
>>
>> This driver will be beneficial for managing PCI hotplug events on the OCTEON
>> PCIe device without requiring complex hardware solutions.
>>
>>  MAINTAINERS                    |   6 +
>>  drivers/pci/hotplug/Kconfig    |  10 +
>>  drivers/pci/hotplug/Makefile   |   1 +
>>  drivers/pci/hotplug/octep_hp.c | 351
>+++++++++++++++++++++++++++++++++
>>  4 files changed, 368 insertions(+)
>>  create mode 100644 drivers/pci/hotplug/octep_hp.c
>>
>
>
>> diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
>> new file mode 100644
>> index 000000000000..efeb542d4993
>> --- /dev/null
>> +++ b/drivers/pci/hotplug/octep_hp.c
>> @@ -0,0 +1,351 @@
>
>> +struct octep_hp_controller {
>> +	void __iomem *base;
>> +	struct pci_dev *pdev;
>> +	struct work_struct work;
>> +	struct list_head slot_list;
>> +	struct mutex slot_lock; /* Protects slot_list */
>> +	struct list_head hp_cmd_list;
>> +	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
>> +};
>
>> +static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
>> +				  struct octep_hp_slot *hp_slot)
>> +{
>> +	mutex_lock(&hp_ctrl->slot_lock);
>
>guard(mutex)(&hp_ctrl->slot_lock);
>
>Same for other simple cases where we can avoid explicit unlock in error paths
>+ main path.
>
>

Will change to use guard.

>> +	if (!hp_slot->hp_pdev) {
>> +		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n",
>hp_slot->slot_number);
>> +		mutex_unlock(&hp_ctrl->slot_lock);
>> +		return;
>> +	}
>> +
>> +	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n", hp_slot-
>>slot_number);
>> +
>> +	/* Remove the device from the bus */
>> +	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
>> +	hp_slot->hp_pdev = NULL;
>> +	mutex_unlock(&hp_ctrl->slot_lock);
>> +}
>
>> +#define SLOT_NAME_SIZE 16
>> +static int octep_hp_register_slot(struct octep_hp_controller *hp_ctrl, struct
>pci_dev *pdev,
>> +				  u16 slot_number)
>> +{
>> +	char slot_name[SLOT_NAME_SIZE];
>> +	struct octep_hp_slot *hp_slot;
>> +	int ret;
>> +
>> +	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
>> +	if (!hp_slot)
>> +		return -ENOMEM;
>> +
>> +	hp_slot->ctrl = hp_ctrl;
>> +	hp_slot->hp_pdev = pdev;
>> +	hp_slot->hp_devfn = pdev->devfn;
>> +	hp_slot->slot_number = slot_number;
>> +	hp_slot->slot.ops = &octep_hp_slot_ops;
>> +
>> +	snprintf(slot_name, SLOT_NAME_SIZE, "octep_hp_%u", slot_number);
>> +	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
>PCI_SLOT(pdev->devfn), slot_name);
>> +	if (ret) {
>> +		dev_err(&pdev->dev, "Failed to register hotplug slot %u\n",
>slot_number);
>> +		kfree(hp_slot);
>> +		return ret;
>		return dev_err_probe() in here as well.
>

Will change.

>> +	}
>> +
>> +	octep_hp_disable_pdev(hp_ctrl, hp_slot);
>> +	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
>> +
>> +	return 0;
>> +}
>> +
>> +static bool octep_hp_slot(struct octep_hp_controller *hp_ctrl, struct pci_dev
>*pdev)
>> +{
>> +	/* Check if the PCI device can be hotplugged */
>> +	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
>> +		PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
>> +}
>> +
>> +static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
>struct octep_hp_cmd *hp_cmd)
>
>Line break as that's getting long for little reason.
>

Okay. Will also change other places that cross  the 80 mark.

>> +{
>> +	struct octep_hp_slot *hp_slot;
>> +
>> +	/* Enable or disable the slots based on the slot mask */
>> +	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
>> +		if (hp_cmd->slot_mask & BIT(hp_slot->slot_number)) {
>> +			if (hp_cmd->vec_type == OCTEP_HP_VEC_ENA)
>> +				octep_hp_enable_pdev(hp_ctrl, hp_slot);
>> +			else
>> +				octep_hp_disable_pdev(hp_ctrl, hp_slot);
>> +		}
>> +	}
>> +}
>> +
>> +static void octep_hp_work_handler(struct work_struct *work)
>> +{
>> +	struct octep_hp_controller *hp_ctrl = container_of(work, struct
>octep_hp_controller, work);
>
>Add a line break.
>

Okay.

>> +	struct octep_hp_cmd *hp_cmd;
>> +	unsigned long flags;
>
>...
>
>> +
>> +static void octep_hp_deregister_slots(struct octep_hp_controller *hp_ctrl)
>> +{
>> +	struct octep_hp_slot *hp_slot, *tmp;
>> +
>> +	/* Deregister all the hotplug slots */
>> +	list_for_each_entry_safe(hp_slot, tmp, &hp_ctrl->slot_list, list) {
>> +		pci_hp_deregister(&hp_slot->slot);
>> +		octep_hp_enable_pdev(hp_ctrl, hp_slot);
>> +		list_del(&hp_slot->list);
>> +		kfree(hp_slot);
>
>As below, maybe just register cleanup for one slot at a time.
>That will need a bit of magic to get form the slot to the hp_ctrl though.
>
>Plus point is simpler logic to follow.
>
>

Will change.

>> +	}
>> +}
>> +
>> +static void octep_hp_controller_cleanup(void *data)
>> +{
>> +	struct octep_hp_controller *hp_ctrl = data;
>> +
>> +	pci_free_irq_vectors(hp_ctrl->pdev);
>> +	flush_work(&hp_ctrl->work);
>I'd prefer to see this broken up so we have
>simple steps.
>1. Add action 1 in probe.
>2. Add cleanup for action 1 in probe
>3. Add action 2 in probe etc.
>
>Otherwise it can be a bit tricky to review.
>

Will break the cleanup to IRQ and slot specific.

>> +	octep_hp_deregister_slots(hp_ctrl);
>
>This is particular is no where near the place where the
>slots are registered.
>May be better to register one cleanup function call
>per slot as then it will also be nice and simple to follow.
>
>

Will change to cleanup per slot.

>> +}
>> +
>> +static int octep_hp_controller_setup(struct pci_dev *pdev, struct
>octep_hp_controller *hp_ctrl)
>> +{
>> +	struct device *dev = &pdev->dev;
>> +	int ret;
>> +
>> +	ret = pcim_enable_device(pdev);
>> +	if (ret) {
>> +		dev_err(dev, "Failed to enable PCI device\n");
>Only called from probe, so
>		return dev_err_probe()
>
>> +		return ret;
>> +	}
>> +
>> +	ret = pcim_iomap_regions(pdev, BIT(0), OCTEP_HP_DRV_NAME);
>Given there is a patch on list deprecating this, reconsider.
> https://lore.kernel.org/all/20240807083018.8734-2-pstanner@redhat.com/ 
>+CC Philipp
>

Okay. The API is available in origin/devres branch.  Will rebase on top of it.

>> +	if (ret) {
>> +		dev_err(dev, "Failed to request MMIO region\n");
>> +		return ret;
>
>Same thing on return dev_err_probe here and later.
>

Will change.

>> +	}
>> +
>> +	pci_set_master(pdev);
>> +	pci_set_drvdata(pdev, hp_ctrl);
>> +
>> +	INIT_LIST_HEAD(&hp_ctrl->slot_list);
>> +	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
>> +	mutex_init(&hp_ctrl->slot_lock);
>> +	spin_lock_init(&hp_ctrl->hp_cmd_lock);
>> +	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
>> +
>> +	hp_ctrl->pdev = pdev;
>> +	hp_ctrl->base = pcim_iomap_table(pdev)[0];
>> +	if (!hp_ctrl->base) {
>> +		dev_err(dev, "Failed to get device resource map\n");
>> +		return -ENODEV;
>> +	}
>> +
>> +	ret = pci_alloc_irq_vectors(pdev, 1,
>OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS) + 1,
>> +				    PCI_IRQ_MSIX);
>> +	if (ret < 0) {
>> +		dev_err(dev, "Failed to alloc MSI-X vectors");
>> +		return ret;
>> +	}
>> +
>> +	ret = devm_add_action(dev, octep_hp_controller_cleanup, hp_ctrl);
>> +	if (ret) {
>> +		dev_err(dev, "Failed to add action for controller cleanup\n");
>> +		return ret;
>> +	}
>> +
>> +	ret = devm_request_irq(dev, pci_irq_vector(pdev,
>OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_ENA)),
>> +			       octep_hp_intr_handler, 0, "octep_hp_ena",
>hp_ctrl);
>> +	if (ret) {
>> +		dev_err(dev, "Failed to register slot enable interrupt
>handle\n");
>> +		return ret;
>> +	}
>> +
>> +	ret = devm_request_irq(dev, pci_irq_vector(pdev,
>OCTEP_HP_INTR_VECTOR(OCTEP_HP_VEC_DIS)),
>> +			       octep_hp_intr_handler, 0, "octep_hp_dis", hp_ctrl);
>> +	if (ret) {
>> +		dev_err(dev, "Failed to register slot disable interrupt
>handle\n");
>> +		return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int octep_hp_pci_probe(struct pci_dev *pdev, const struct
>pci_device_id *id)
>> +{
>> +	struct octep_hp_controller *hp_ctrl;
>> +	struct pci_dev *tmp_pdev = NULL;
>> +	u16 slot_number = 0;
>> +	int ret;
>> +
>> +	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
>> +	if (!hp_ctrl)
>> +		return -ENOMEM;
>> +
>> +	ret = octep_hp_controller_setup(pdev, hp_ctrl);
>> +	if (ret) {
>> +		dev_err(&pdev->dev, "Failed to setup octep controller\n");
>> +		return ret;
>return dev_err_probe()
>actually no. Drop it. There are more informative error prints for all the
>conditions. We probably don't care that the end result is this specific
>function fails (more than an irq request failed etc).
>

Okay.

>
>> +	}
>> +
>> +	/*
>> +	 * Register all hotplug slots. Hotplug controller is the first function of
>the PCI device.
>> +	 * The hotplug slots are the remaining functions of the PCI device. They
>are removed from
>> +	 * the bus and are added back when the hotplug event is triggered.
>> +	 */
>> +	for_each_pci_dev(tmp_pdev) {
>> +		if (octep_hp_slot(hp_ctrl, tmp_pdev)) {
>What IIpo said on this.  No one likes deep indent ;)
>> +			ret = octep_hp_register_slot(hp_ctrl, tmp_pdev,
>slot_number++);
>> +			if (ret) {
>> +				dev_err(&pdev->dev, "Failed to register
>hotplug slots.\n");
>> +				return ret;
>
>return dev_err_probe();
>
>Cleaner in general even if no chance of deferral.
>
>
>
>> +			}
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
>> +static struct pci_device_id octep_hp_pci_map[] = {
>> +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
>OCTEP_DEVID_HP_CONTROLLER) },
>> +	{ 0 },
>
>{ }
>Is sufficient. I can't remember if that's common style in PCI or not though.
>

Will change.

>> +};

Thanks for the review.
Shijith

[PATCH v2] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

This patch introduces a PCI hotplug controller driver for the OCTEON
PCIe device, a multi-function PCIe device where the first function acts
as a hotplug controller. It is equipped with MSI-x interrupts to notify
the host of hotplug events from the OCTEON firmware.

The driver facilitates the hotplugging of non-controller functions
within the same device. During probe, non-controller functions are
removed and registered as PCI hotplug slots. The slots are added back
only upon request from the device firmware. The driver also allows the
enabling and disabling of the slots via sysfs slot entries, provided by
the PCI hotplug framework.

Signed-off-by: Shijith Thotton <sthotton@marvell.com>
Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
---

This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
controller. The OCTEON PCIe device is a multi-function device where the first
function acts as a PCI hotplug controller.

              +--------------------------------+
              |           Root Port            |
              +--------------------------------+
                              |
                             PCIe
                              |
+---------------------------------------------------------------+
|              OCTEON PCIe Multifunction Device                 |
+---------------------------------------------------------------+
            |                    |              |            |
            |                    |              |            |
+---------------------+  +----------------+  +-----+  +----------------+
|      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
| (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
+---------------------+  +----------------+  +-----+  +----------------+
            |
            |
+-------------------------+
|   Controller Firmware   |
+-------------------------+

The hotplug controller driver facilitates the hotplugging of non-controller
functions in the same device. During the probe of the driver, the non-controller
function are removed and registered as PCI hotplug slots. They are added back
only upon request from the device firmware. The driver also allows the user to
enable/disable the functions using sysfs slot entries provided by PCI hotplug
framework.

This solution adopts a hardware + software approach for several reasons:

1. To reduce hardware implementation cost. Supporting complete hotplug
   capability within the card would require a PCI switch implemented within.

2. In the multi-function device, non-controller functions can act as emulated
   devices. The firmware can dynamically enable or disable them at runtime.

3. Not all root ports support PCI hotplug. This approach provides greater
   flexibility and compatibility across different hardware configurations.

The hotplug controller function is lightweight and is equipped with MSI-x
interrupts to notify the host about hotplug events. Upon receiving an
interrupt, the hotplug register is read, and the required function is enabled
or disabled.

This driver will be beneficial for managing PCI hotplug events on the OCTEON
PCIe device without requiring complex hardware solutions.

Changes in v2:
- Added missing include files.
- Used dev_err_probe() for error handling.
- Used guard() for mutex locking.
- Splited cleanup actions and added per-slot cleanup action.
- Fixed coding style issues.
- Added co-developed-by tag.

 MAINTAINERS                    |   6 +
 drivers/pci/hotplug/Kconfig    |  10 +
 drivers/pci/hotplug/Makefile   |   1 +
 drivers/pci/hotplug/octep_hp.c | 412 +++++++++++++++++++++++++++++++++
 4 files changed, 429 insertions(+)
 create mode 100644 drivers/pci/hotplug/octep_hp.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 42decde38320..7b5a618eed1c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
 R:	vattunuru@marvell.com
 F:	drivers/vdpa/octeon_ep/
 
+MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
+R:	Shijith Thotton <sthotton@marvell.com>
+R:	Vamsi Attunuru <vattunuru@marvell.com>
+S:	Supported
+F:	drivers/pci/hotplug/octep_hp.c
+
 MATROX FRAMEBUFFER DRIVER
 L:	linux-fbdev@vger.kernel.org
 S:	Orphan
diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
index 1472aef0fb81..2e38fd25f7ef 100644
--- a/drivers/pci/hotplug/Kconfig
+++ b/drivers/pci/hotplug/Kconfig
@@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
 
 	  When in doubt, say Y.
 
+config HOTPLUG_PCI_OCTEONEP
+	bool "OCTEON PCI device Hotplug controller driver"
+	depends on HOTPLUG_PCI
+	help
+	  Say Y here if you have an OCTEON PCIe device with a hotplug
+	  controller. This driver enables the non-controller functions of the
+	  device to be registered as hotplug slots.
+
+	  When in doubt, say N.
+
 endif # HOTPLUG_PCI
diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
index 240c99517d5e..40aaf31fe338 100644
--- a/drivers/pci/hotplug/Makefile
+++ b/drivers/pci/hotplug/Makefile
@@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+= rpaphp.o
 obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
 obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
 obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
+obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
 
 # acpiphp_ibm extends acpiphp, so should be linked afterwards.
 
diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
new file mode 100644
index 000000000000..3ac90ffff564
--- /dev/null
+++ b/drivers/pci/hotplug/octep_hp.c
@@ -0,0 +1,412 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2024 Marvell. */
+
+#include <linux/cleanup.h>
+#include <linux/container_of.h>
+#include <linux/delay.h>
+#include <linux/dev_printk.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/pci_hotplug.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/workqueue.h>
+
+#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
+#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
+#define OCTEP_HP_DRV_NAME "octep_hp"
+
+/*
+ * Type of MSI-X interrupts.
+ * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are used to
+ * generate the vector and offset for an interrupt type.
+ */
+enum octep_hp_intr_type {
+	OCTEP_HP_INTR_INVALID = -1,
+	OCTEP_HP_INTR_ENA,
+	OCTEP_HP_INTR_DIS,
+	OCTEP_HP_INTR_MAX,
+};
+
+struct octep_hp_cmd {
+	struct list_head list;
+	enum octep_hp_intr_type intr_type;
+	u64 intr_val;
+};
+
+struct octep_hp_slot {
+	struct list_head list;
+	struct hotplug_slot slot;
+	u16 slot_number;
+	struct pci_dev *hp_pdev;
+	unsigned int hp_devfn;
+	struct octep_hp_controller *ctrl;
+};
+
+struct octep_hp_intr_info {
+	enum octep_hp_intr_type type;
+	int number;
+	char name[16];
+};
+
+struct octep_hp_controller {
+	void __iomem *base;
+	struct pci_dev *pdev;
+	struct octep_hp_intr_info intr[OCTEP_HP_INTR_MAX];
+	struct work_struct work;
+	struct list_head slot_list;
+	struct mutex slot_lock; /* Protects slot_list */
+	struct list_head hp_cmd_list;
+	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
+};
+
+static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl,
+				 struct octep_hp_slot *hp_slot)
+{
+	guard(mutex)(&hp_ctrl->slot_lock);
+	if (hp_slot->hp_pdev) {
+		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already enabled\n",
+			hp_slot->slot_number);
+		return;
+	}
+
+	/* Scan the device and add it to the bus */
+	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus,
+						  hp_slot->hp_devfn);
+	pci_bus_assign_resources(hp_ctrl->pdev->bus);
+	pci_bus_add_device(hp_slot->hp_pdev);
+
+	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n",
+		hp_slot->slot_number);
+}
+
+static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
+				  struct octep_hp_slot *hp_slot)
+{
+	guard(mutex)(&hp_ctrl->slot_lock);
+	if (!hp_slot->hp_pdev) {
+		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n",
+			hp_slot->slot_number);
+		return;
+	}
+
+	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n",
+		hp_slot->slot_number);
+
+	/* Remove the device from the bus */
+	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
+	hp_slot->hp_pdev = NULL;
+}
+
+static int octep_hp_enable_slot(struct hotplug_slot *slot)
+{
+	struct octep_hp_slot *hp_slot =
+		container_of(slot, struct octep_hp_slot, slot);
+
+	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
+	return 0;
+}
+
+static int octep_hp_disable_slot(struct hotplug_slot *slot)
+{
+	struct octep_hp_slot *hp_slot =
+		container_of(slot, struct octep_hp_slot, slot);
+
+	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
+	return 0;
+}
+
+static struct hotplug_slot_ops octep_hp_slot_ops = {
+	.enable_slot = octep_hp_enable_slot,
+	.disable_slot = octep_hp_disable_slot,
+};
+
+#define SLOT_NAME_SIZE 16
+static struct octep_hp_slot *
+octep_hp_register_slot(struct octep_hp_controller *hp_ctrl,
+		       struct pci_dev *pdev, u16 slot_number)
+{
+	char slot_name[SLOT_NAME_SIZE];
+	struct octep_hp_slot *hp_slot;
+	int ret;
+
+	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
+	if (!hp_slot)
+		return ERR_PTR(-ENOMEM);
+
+	hp_slot->ctrl = hp_ctrl;
+	hp_slot->hp_pdev = pdev;
+	hp_slot->hp_devfn = pdev->devfn;
+	hp_slot->slot_number = slot_number;
+	hp_slot->slot.ops = &octep_hp_slot_ops;
+
+	snprintf(slot_name, sizeof(slot_name), "octep_hp_%u", slot_number);
+	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
+			      PCI_SLOT(pdev->devfn), slot_name);
+	if (ret) {
+		kfree(hp_slot);
+		return ERR_PTR(ret);
+	}
+
+	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
+	octep_hp_disable_pdev(hp_ctrl, hp_slot);
+
+	return hp_slot;
+}
+
+static void octep_hp_deregister_slot(void *data)
+{
+	struct octep_hp_slot *hp_slot = data;
+	struct octep_hp_controller *hp_ctrl = hp_slot->ctrl;
+
+	pci_hp_deregister(&hp_slot->slot);
+	octep_hp_enable_pdev(hp_ctrl, hp_slot);
+	list_del(&hp_slot->list);
+	kfree(hp_slot);
+}
+
+static bool octep_hp_is_slot(struct octep_hp_controller *hp_ctrl,
+			     struct pci_dev *pdev)
+{
+	/* Check if the PCI device can be hotplugged */
+	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
+	       PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
+}
+
+static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
+				 struct octep_hp_cmd *hp_cmd)
+{
+	struct octep_hp_slot *hp_slot;
+
+	/*
+	 * Enable or disable the slots based on the slot mask.
+	 * intr_val is a bit mask where each bit represents a slot.
+	 */
+	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
+		if (!(hp_cmd->intr_val & BIT(hp_slot->slot_number)))
+			continue;
+
+		if (hp_cmd->intr_type == OCTEP_HP_INTR_ENA)
+			octep_hp_enable_pdev(hp_ctrl, hp_slot);
+		else
+			octep_hp_disable_pdev(hp_ctrl, hp_slot);
+	}
+}
+
+static void octep_hp_work_handler(struct work_struct *work)
+{
+	struct octep_hp_controller *hp_ctrl;
+	struct octep_hp_cmd *hp_cmd;
+	unsigned long flags;
+
+	hp_ctrl = container_of(work, struct octep_hp_controller, work);
+
+	/* Process all the hotplug commands */
+	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
+	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
+		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list,
+					  struct octep_hp_cmd, list);
+		list_del(&hp_cmd->list);
+		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
+
+		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
+		kfree(hp_cmd);
+
+		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
+	}
+	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
+}
+
+static enum octep_hp_intr_type octep_hp_intr_type(struct octep_hp_intr_info *intr,
+						  int irq)
+{
+	enum octep_hp_intr_type type;
+
+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX; type++) {
+		if (intr[type].number == irq)
+			return type;
+	}
+
+	return OCTEP_HP_INTR_INVALID;
+}
+
+static irqreturn_t octep_hp_intr_handler(int irq, void *data)
+{
+	struct octep_hp_controller *hp_ctrl = data;
+	struct pci_dev *pdev = hp_ctrl->pdev;
+	enum octep_hp_intr_type type;
+	struct octep_hp_cmd *hp_cmd;
+	u64 intr_val;
+
+	type = octep_hp_intr_type(hp_ctrl->intr, irq);
+	if (type == OCTEP_HP_INTR_INVALID) {
+		dev_err(&pdev->dev, "Invalid interrupt %d\n", irq);
+		return IRQ_HANDLED;
+	}
+
+	/* Read and clear the interrupt */
+	intr_val = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
+	writeq(intr_val, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
+
+	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
+	if (!hp_cmd)
+		return IRQ_HANDLED;
+
+	hp_cmd->intr_val = intr_val;
+	hp_cmd->intr_type = type;
+
+	/* Add the command to the list and schedule the work */
+	spin_lock(&hp_ctrl->hp_cmd_lock);
+	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
+	spin_unlock(&hp_ctrl->hp_cmd_lock);
+	schedule_work(&hp_ctrl->work);
+
+	return IRQ_HANDLED;
+}
+
+static void octep_hp_irq_cleanup(void *data)
+{
+	struct octep_hp_controller *hp_ctrl = data;
+
+	pci_free_irq_vectors(hp_ctrl->pdev);
+	flush_work(&hp_ctrl->work);
+}
+
+static int octep_hp_request_irq(struct octep_hp_controller *hp_ctrl,
+				enum octep_hp_intr_type type)
+{
+	struct pci_dev *pdev = hp_ctrl->pdev;
+	struct octep_hp_intr_info *intr;
+	int irq;
+
+	irq = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(type));
+	if (irq < 0)
+		return irq;
+
+	intr = &hp_ctrl->intr[type];
+	intr->number = irq;
+	intr->type = type;
+	snprintf(intr->name, sizeof(intr->name), "octep_hp_%d", type);
+
+	return devm_request_irq(&pdev->dev, irq, octep_hp_intr_handler,
+				IRQF_SHARED, intr->name, hp_ctrl);
+}
+
+static int octep_hp_controller_setup(struct pci_dev *pdev,
+				     struct octep_hp_controller *hp_ctrl)
+{
+	struct device *dev = &pdev->dev;
+	enum octep_hp_intr_type type;
+	int ret;
+
+	ret = pcim_enable_device(pdev);
+	if (ret)
+		return dev_err_probe(dev, ret, "Failed to enable PCI device\n");
+
+	ret = pcim_iomap_regions(pdev, BIT(0), OCTEP_HP_DRV_NAME);
+	if (ret)
+		return dev_err_probe(dev, ret, "Failed to map PCI device region\n");
+
+	pci_set_master(pdev);
+	pci_set_drvdata(pdev, hp_ctrl);
+
+	INIT_LIST_HEAD(&hp_ctrl->slot_list);
+	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
+	mutex_init(&hp_ctrl->slot_lock);
+	spin_lock_init(&hp_ctrl->hp_cmd_lock);
+	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
+
+	hp_ctrl->pdev = pdev;
+	hp_ctrl->base = pcim_iomap_table(pdev)[0];
+	if (!hp_ctrl->base)
+		return dev_err_probe(dev, -ENODEV, "Failed to get device resource map\n");
+
+	ret = pci_alloc_irq_vectors(pdev, 1,
+				    OCTEP_HP_INTR_VECTOR(OCTEP_HP_INTR_MAX),
+				    PCI_IRQ_MSIX);
+	if (ret < 0)
+		return dev_err_probe(dev, ret, "Failed to alloc MSI-X vectors\n");
+
+	ret = devm_add_action(&pdev->dev, octep_hp_irq_cleanup, hp_ctrl);
+	if (ret)
+		return dev_err_probe(&pdev->dev, ret, "Failed to add IRQ cleanup action\n");
+
+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX; type++) {
+		ret = octep_hp_request_irq(hp_ctrl, type);
+		if (ret)
+			return dev_err_probe(dev, ret,
+					     "Failed to request IRQ for vector %d\n",
+					     OCTEP_HP_INTR_VECTOR(type));
+	}
+
+	return 0;
+}
+
+static int octep_hp_pci_probe(struct pci_dev *pdev,
+			      const struct pci_device_id *id)
+{
+	struct octep_hp_controller *hp_ctrl;
+	struct pci_dev *tmp_pdev = NULL;
+	struct octep_hp_slot *hp_slot;
+	u16 slot_number = 0;
+	int ret;
+
+	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
+	if (!hp_ctrl)
+		return -ENOMEM;
+
+	ret = octep_hp_controller_setup(pdev, hp_ctrl);
+	if (ret)
+		return ret;
+
+	/*
+	 * Register all hotplug slots. Hotplug controller is the first function
+	 * of the PCI device. The hotplug slots are the remaining functions of
+	 * the PCI device. They are removed from the bus and are added back when
+	 * the hotplug event is triggered.
+	 */
+	for_each_pci_dev(tmp_pdev) {
+		if (!octep_hp_is_slot(hp_ctrl, tmp_pdev))
+			continue;
+
+		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev, slot_number);
+		if (IS_ERR(hp_slot))
+			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
+					     "Failed to register hotplug slot %u\n",
+					     slot_number);
+
+		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
+				      hp_slot);
+		if (ret)
+			return dev_err_probe(&pdev->dev, ret,
+					     "Failed to add action for deregistering slot %u\n",
+					     slot_number);
+		slot_number++;
+	}
+
+	return 0;
+}
+
+#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
+static struct pci_device_id octep_hp_pci_map[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, OCTEP_DEVID_HP_CONTROLLER) },
+	{ },
+};
+
+static struct pci_driver octep_hp = {
+	.name = OCTEP_HP_DRV_NAME,
+	.id_table = octep_hp_pci_map,
+	.probe = octep_hp_pci_probe,
+};
+
+module_pci_driver(octep_hp);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Marvell");
+MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
-- 
2.25.1

RE: [PATCH] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

>>> This patch introduces a PCI hotplug controller driver for the OCTEON
>>> PCIe device, a multi-function PCIe device where the first function acts
>>> as a hotplug controller. It is equipped with MSI-x interrupts to notify
>>> the host of hotplug events from the OCTEON firmware.
>>>
>>> The driver facilitates the hotplugging of non-controller functions
>>> within the same device. During probe, non-controller functions are
>>> removed and registered as PCI hotplug slots. The slots are added back
>>> only upon request from the device firmware. The driver also allows the
>>> enabling and disabling of the slots via sysfs slot entries, provided by
>>> the PCI hotplug framework.
>>>
>>> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>>> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>>I was curious, so drive by review.
>>
>>What was Vamsi's involvement?  Given Shijith sent this
>>I'd guess co developer?  In which Co-developed-by tag
>>should be used.
>>
...
>>> +}
>>> +
>>> +static int octep_hp_controller_setup(struct pci_dev *pdev, struct
>>octep_hp_controller *hp_ctrl)
>>> +{
>>> +	struct device *dev = &pdev->dev;
>>> +	int ret;
>>> +
>>> +	ret = pcim_enable_device(pdev);
>>> +	if (ret) {
>>> +		dev_err(dev, "Failed to enable PCI device\n");
>>Only called from probe, so
>>		return dev_err_probe()
>>
>>> +		return ret;
>>> +	}
>>> +
>>> +	ret = pcim_iomap_regions(pdev, BIT(0), OCTEP_HP_DRV_NAME);
>>Given there is a patch on list deprecating this, reconsider.
>> https://lore.kernel.org/all/20240807083018.8734-2-
>pstanner@redhat.com/
>>+CC Philipp
>>
>
>Okay. The API is available in origin/devres branch.  Will rebase on top of it.
>

I forgot about the kernel test robot.
I'll hold off on making this change until the patch is merged into pci/next.

Re: [PATCH v2] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Ilpo Järvinen]

On Fri, 23 Aug 2024, Shijith Thotton wrote:

> This patch introduces a PCI hotplug controller driver for the OCTEON
> PCIe device, a multi-function PCIe device where the first function acts
> as a hotplug controller. It is equipped with MSI-x interrupts to notify
> the host of hotplug events from the OCTEON firmware.
> 
> The driver facilitates the hotplugging of non-controller functions
> within the same device. During probe, non-controller functions are
> removed and registered as PCI hotplug slots. The slots are added back
> only upon request from the device firmware. The driver also allows the
> enabling and disabling of the slots via sysfs slot entries, provided by
> the PCI hotplug framework.
> 
> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
> Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> ---
> 
> This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
> controller. The OCTEON PCIe device is a multi-function device where the first
> function acts as a PCI hotplug controller.
> 
>               +--------------------------------+
>               |           Root Port            |
>               +--------------------------------+
>                               |
>                              PCIe
>                               |
> +---------------------------------------------------------------+
> |              OCTEON PCIe Multifunction Device                 |
> +---------------------------------------------------------------+
>             |                    |              |            |
>             |                    |              |            |
> +---------------------+  +----------------+  +-----+  +----------------+
> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
> +---------------------+  +----------------+  +-----+  +----------------+
>             |
>             |
> +-------------------------+
> |   Controller Firmware   |
> +-------------------------+
> 
> The hotplug controller driver facilitates the hotplugging of non-controller
> functions in the same device. During the probe of the driver, the non-controller
> function are removed and registered as PCI hotplug slots. They are added back
> only upon request from the device firmware. The driver also allows the user to
> enable/disable the functions using sysfs slot entries provided by PCI hotplug
> framework.
> 
> This solution adopts a hardware + software approach for several reasons:
> 
> 1. To reduce hardware implementation cost. Supporting complete hotplug
>    capability within the card would require a PCI switch implemented within.
> 
> 2. In the multi-function device, non-controller functions can act as emulated
>    devices. The firmware can dynamically enable or disable them at runtime.
> 
> 3. Not all root ports support PCI hotplug. This approach provides greater
>    flexibility and compatibility across different hardware configurations.
> 
> The hotplug controller function is lightweight and is equipped with MSI-x
> interrupts to notify the host about hotplug events. Upon receiving an
> interrupt, the hotplug register is read, and the required function is enabled
> or disabled.
> 
> This driver will be beneficial for managing PCI hotplug events on the OCTEON
> PCIe device without requiring complex hardware solutions.
> 
> Changes in v2:
> - Added missing include files.
> - Used dev_err_probe() for error handling.
> - Used guard() for mutex locking.
> - Splited cleanup actions and added per-slot cleanup action.
> - Fixed coding style issues.
> - Added co-developed-by tag.
> 
>  MAINTAINERS                    |   6 +
>  drivers/pci/hotplug/Kconfig    |  10 +
>  drivers/pci/hotplug/Makefile   |   1 +
>  drivers/pci/hotplug/octep_hp.c | 412 +++++++++++++++++++++++++++++++++
>  4 files changed, 429 insertions(+)
>  create mode 100644 drivers/pci/hotplug/octep_hp.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 42decde38320..7b5a618eed1c 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
>  R:	vattunuru@marvell.com
>  F:	drivers/vdpa/octeon_ep/
>  
> +MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
> +R:	Shijith Thotton <sthotton@marvell.com>
> +R:	Vamsi Attunuru <vattunuru@marvell.com>
> +S:	Supported
> +F:	drivers/pci/hotplug/octep_hp.c
> +
>  MATROX FRAMEBUFFER DRIVER
>  L:	linux-fbdev@vger.kernel.org
>  S:	Orphan
> diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
> index 1472aef0fb81..2e38fd25f7ef 100644
> --- a/drivers/pci/hotplug/Kconfig
> +++ b/drivers/pci/hotplug/Kconfig
> @@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
>  
>  	  When in doubt, say Y.
>  
> +config HOTPLUG_PCI_OCTEONEP
> +	bool "OCTEON PCI device Hotplug controller driver"
> +	depends on HOTPLUG_PCI
> +	help
> +	  Say Y here if you have an OCTEON PCIe device with a hotplug
> +	  controller. This driver enables the non-controller functions of the
> +	  device to be registered as hotplug slots.
> +
> +	  When in doubt, say N.
> +
>  endif # HOTPLUG_PCI
> diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
> index 240c99517d5e..40aaf31fe338 100644
> --- a/drivers/pci/hotplug/Makefile
> +++ b/drivers/pci/hotplug/Makefile
> @@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+= rpaphp.o
>  obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
>  obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
>  obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
> +obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
>  
>  # acpiphp_ibm extends acpiphp, so should be linked afterwards.
>  
> diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
> new file mode 100644
> index 000000000000..3ac90ffff564
> --- /dev/null
> +++ b/drivers/pci/hotplug/octep_hp.c
> @@ -0,0 +1,412 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright (C) 2024 Marvell. */
> +
> +#include <linux/cleanup.h>
> +#include <linux/container_of.h>
> +#include <linux/delay.h>
> +#include <linux/dev_printk.h>
> +#include <linux/init.h>
> +#include <linux/interrupt.h>
> +#include <linux/io-64-nonatomic-lo-hi.h>
> +#include <linux/kernel.h>
> +#include <linux/list.h>
> +#include <linux/module.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/pci_hotplug.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/workqueue.h>
> +
> +#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
> +#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
> +#define OCTEP_HP_DRV_NAME "octep_hp"
> +
> +/*
> + * Type of MSI-X interrupts.
> + * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are used to
> + * generate the vector and offset for an interrupt type.
> + */
> +enum octep_hp_intr_type {
> +	OCTEP_HP_INTR_INVALID = -1,
> +	OCTEP_HP_INTR_ENA,
> +	OCTEP_HP_INTR_DIS,

Making these numbers explicit (since they cannot be just any numbers) fell 
through cracks.


-- 
 i.

Re: [PATCH v2] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

>> This patch introduces a PCI hotplug controller driver for the OCTEON
>> PCIe device, a multi-function PCIe device where the first function acts
>> as a hotplug controller. It is equipped with MSI-x interrupts to notify
>> the host of hotplug events from the OCTEON firmware.
>>
>> The driver facilitates the hotplugging of non-controller functions
>> within the same device. During probe, non-controller functions are
>> removed and registered as PCI hotplug slots. The slots are added back
>> only upon request from the device firmware. The driver also allows the
>> enabling and disabling of the slots via sysfs slot entries, provided by
>> the PCI hotplug framework.
>>
>> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>> Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
>> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>> ---
>>
>> This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
>> controller. The OCTEON PCIe device is a multi-function device where the first
>> function acts as a PCI hotplug controller.
>>
>>               +--------------------------------+
>>               |           Root Port            |
>>               +--------------------------------+
>>                               |
>>                              PCIe
>>                               |
>> +---------------------------------------------------------------+
>> |              OCTEON PCIe Multifunction Device                 |
>> +---------------------------------------------------------------+
>>             |                    |              |            |
>>             |                    |              |            |
>> +---------------------+  +----------------+  +-----+  +----------------+
>> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>> +---------------------+  +----------------+  +-----+  +----------------+
>>             |
>>             |
>> +-------------------------+
>> |   Controller Firmware   |
>> +-------------------------+
>>
>> The hotplug controller driver facilitates the hotplugging of non-controller
>> functions in the same device. During the probe of the driver, the non-
>controller
>> function are removed and registered as PCI hotplug slots. They are added
>back
>> only upon request from the device firmware. The driver also allows the user
>to
>> enable/disable the functions using sysfs slot entries provided by PCI hotplug
>> framework.
>>
>> This solution adopts a hardware + software approach for several reasons:
>>
>> 1. To reduce hardware implementation cost. Supporting complete hotplug
>>    capability within the card would require a PCI switch implemented within.
>>
>> 2. In the multi-function device, non-controller functions can act as emulated
>>    devices. The firmware can dynamically enable or disable them at runtime.
>>
>> 3. Not all root ports support PCI hotplug. This approach provides greater
>>    flexibility and compatibility across different hardware configurations.
>>
>> The hotplug controller function is lightweight and is equipped with MSI-x
>> interrupts to notify the host about hotplug events. Upon receiving an
>> interrupt, the hotplug register is read, and the required function is enabled
>> or disabled.
>>
>> This driver will be beneficial for managing PCI hotplug events on the OCTEON
>> PCIe device without requiring complex hardware solutions.
>>
>> Changes in v2:
>> - Added missing include files.
>> - Used dev_err_probe() for error handling.
>> - Used guard() for mutex locking.
>> - Splited cleanup actions and added per-slot cleanup action.
>> - Fixed coding style issues.
>> - Added co-developed-by tag.
>>
>>  MAINTAINERS                    |   6 +
>>  drivers/pci/hotplug/Kconfig    |  10 +
>>  drivers/pci/hotplug/Makefile   |   1 +
>>  drivers/pci/hotplug/octep_hp.c | 412
>+++++++++++++++++++++++++++++++++
>>  4 files changed, 429 insertions(+)
>>  create mode 100644 drivers/pci/hotplug/octep_hp.c
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 42decde38320..7b5a618eed1c 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
>>  R:	vattunuru@marvell.com
>>  F:	drivers/vdpa/octeon_ep/
>>
>> +MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
>> +R:	Shijith Thotton <sthotton@marvell.com>
>> +R:	Vamsi Attunuru <vattunuru@marvell.com>
>> +S:	Supported
>> +F:	drivers/pci/hotplug/octep_hp.c
>> +
>>  MATROX FRAMEBUFFER DRIVER
>>  L:	linux-fbdev@vger.kernel.org
>>  S:	Orphan
>> diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
>> index 1472aef0fb81..2e38fd25f7ef 100644
>> --- a/drivers/pci/hotplug/Kconfig
>> +++ b/drivers/pci/hotplug/Kconfig
>> @@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
>>
>>  	  When in doubt, say Y.
>>
>> +config HOTPLUG_PCI_OCTEONEP
>> +	bool "OCTEON PCI device Hotplug controller driver"
>> +	depends on HOTPLUG_PCI
>> +	help
>> +	  Say Y here if you have an OCTEON PCIe device with a hotplug
>> +	  controller. This driver enables the non-controller functions of the
>> +	  device to be registered as hotplug slots.
>> +
>> +	  When in doubt, say N.
>> +
>>  endif # HOTPLUG_PCI
>> diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
>> index 240c99517d5e..40aaf31fe338 100644
>> --- a/drivers/pci/hotplug/Makefile
>> +++ b/drivers/pci/hotplug/Makefile
>> @@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+=
>rpaphp.o
>>  obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
>>  obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
>>  obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
>> +obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
>>
>>  # acpiphp_ibm extends acpiphp, so should be linked afterwards.
>>
>> diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
>> new file mode 100644
>> index 000000000000..3ac90ffff564
>> --- /dev/null
>> +++ b/drivers/pci/hotplug/octep_hp.c
>> @@ -0,0 +1,412 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/* Copyright (C) 2024 Marvell. */
>> +
>> +#include <linux/cleanup.h>
>> +#include <linux/container_of.h>
>> +#include <linux/delay.h>
>> +#include <linux/dev_printk.h>
>> +#include <linux/init.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/io-64-nonatomic-lo-hi.h>
>> +#include <linux/kernel.h>
>> +#include <linux/list.h>
>> +#include <linux/module.h>
>> +#include <linux/mutex.h>
>> +#include <linux/pci.h>
>> +#include <linux/pci_hotplug.h>
>> +#include <linux/slab.h>
>> +#include <linux/spinlock.h>
>> +#include <linux/workqueue.h>
>> +
>> +#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
>> +#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
>> +#define OCTEP_HP_DRV_NAME "octep_hp"
>> +
>> +/*
>> + * Type of MSI-X interrupts.
>> + * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are
>used to
>> + * generate the vector and offset for an interrupt type.
>> + */
>> +enum octep_hp_intr_type {
>> +	OCTEP_HP_INTR_INVALID = -1,
>> +	OCTEP_HP_INTR_ENA,
>> +	OCTEP_HP_INTR_DIS,
>
>Making these numbers explicit (since they cannot be just any numbers) fell
>through cracks.
>

The functionality of the code is good since the enum values are automatically
sequential. The values we need for the interrupt types are OCTEP_HP_INTR_ENA as
0 and OCTEP_HP_INTR_DIS as 1. I will include the explicit assignments in v3.

Thanks,
Shijith

[PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

This patch introduces a PCI hotplug controller driver for the OCTEON
PCIe device, a multi-function PCIe device where the first function acts
as a hotplug controller. It is equipped with MSI-x interrupts to notify
the host of hotplug events from the OCTEON firmware.

The driver facilitates the hotplugging of non-controller functions
within the same device. During probe, non-controller functions are
removed and registered as PCI hotplug slots. The slots are added back
only upon request from the device firmware. The driver also allows the
enabling and disabling of the slots via sysfs slot entries, provided by
the PCI hotplug framework.

Signed-off-by: Shijith Thotton <sthotton@marvell.com>
Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
---

This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
controller. The OCTEON PCIe device is a multi-function device where the first
function acts as a PCI hotplug controller.

              +--------------------------------+
              |           Root Port            |
              +--------------------------------+
                              |
                             PCIe
                              |
+---------------------------------------------------------------+
|              OCTEON PCIe Multifunction Device                 |
+---------------------------------------------------------------+
            |                    |              |            |
            |                    |              |            |
+---------------------+  +----------------+  +-----+  +----------------+
|      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
| (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
+---------------------+  +----------------+  +-----+  +----------------+
            |
            |
+-------------------------+
|   Controller Firmware   |
+-------------------------+

The hotplug controller driver facilitates the hotplugging of non-controller
functions in the same device. During the probe of the driver, the non-controller
function are removed and registered as PCI hotplug slots. They are added back
only upon request from the device firmware. The driver also allows the user to
enable/disable the functions using sysfs slot entries provided by PCI hotplug
framework.

This solution adopts a hardware + software approach for several reasons:

1. To reduce hardware implementation cost. Supporting complete hotplug
   capability within the card would require a PCI switch implemented within.

2. In the multi-function device, non-controller functions can act as emulated
   devices. The firmware can dynamically enable or disable them at runtime.

3. Not all root ports support PCI hotplug. This approach provides greater
   flexibility and compatibility across different hardware configurations.

The hotplug controller function is lightweight and is equipped with MSI-x
interrupts to notify the host about hotplug events. Upon receiving an
interrupt, the hotplug register is read, and the required function is enabled
or disabled.

This driver will be beneficial for managing PCI hotplug events on the OCTEON
PCIe device without requiring complex hardware solutions.

Changes in v2:
- Added missing include files.
- Used dev_err_probe() for error handling.
- Used guard() for mutex locking.
- Splited cleanup actions and added per-slot cleanup action.
- Fixed coding style issues.
- Added co-developed-by tag.

Changes in v3:
- Explicit assignment of enum values.
- Use pcim_iomap_region() instead of pcim_iomap_regions().

 MAINTAINERS                    |   6 +
 drivers/pci/hotplug/Kconfig    |  10 +
 drivers/pci/hotplug/Makefile   |   1 +
 drivers/pci/hotplug/octep_hp.c | 409 +++++++++++++++++++++++++++++++++
 4 files changed, 426 insertions(+)
 create mode 100644 drivers/pci/hotplug/octep_hp.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 42decde38320..7b5a618eed1c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
 R:	vattunuru@marvell.com
 F:	drivers/vdpa/octeon_ep/
 
+MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
+R:	Shijith Thotton <sthotton@marvell.com>
+R:	Vamsi Attunuru <vattunuru@marvell.com>
+S:	Supported
+F:	drivers/pci/hotplug/octep_hp.c
+
 MATROX FRAMEBUFFER DRIVER
 L:	linux-fbdev@vger.kernel.org
 S:	Orphan
diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
index 1472aef0fb81..2e38fd25f7ef 100644
--- a/drivers/pci/hotplug/Kconfig
+++ b/drivers/pci/hotplug/Kconfig
@@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
 
 	  When in doubt, say Y.
 
+config HOTPLUG_PCI_OCTEONEP
+	bool "OCTEON PCI device Hotplug controller driver"
+	depends on HOTPLUG_PCI
+	help
+	  Say Y here if you have an OCTEON PCIe device with a hotplug
+	  controller. This driver enables the non-controller functions of the
+	  device to be registered as hotplug slots.
+
+	  When in doubt, say N.
+
 endif # HOTPLUG_PCI
diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
index 240c99517d5e..40aaf31fe338 100644
--- a/drivers/pci/hotplug/Makefile
+++ b/drivers/pci/hotplug/Makefile
@@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+= rpaphp.o
 obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
 obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
 obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
+obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
 
 # acpiphp_ibm extends acpiphp, so should be linked afterwards.
 
diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
new file mode 100644
index 000000000000..77dc2740f43e
--- /dev/null
+++ b/drivers/pci/hotplug/octep_hp.c
@@ -0,0 +1,409 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2024 Marvell. */
+
+#include <linux/cleanup.h>
+#include <linux/container_of.h>
+#include <linux/delay.h>
+#include <linux/dev_printk.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/pci_hotplug.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/workqueue.h>
+
+#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
+#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
+#define OCTEP_HP_DRV_NAME "octep_hp"
+
+/*
+ * Type of MSI-X interrupts.
+ * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are used to
+ * generate the vector and offset for an interrupt type.
+ */
+enum octep_hp_intr_type {
+	OCTEP_HP_INTR_INVALID = -1,
+	OCTEP_HP_INTR_ENA = 0,
+	OCTEP_HP_INTR_DIS = 1,
+	OCTEP_HP_INTR_MAX = 2,
+};
+
+struct octep_hp_cmd {
+	struct list_head list;
+	enum octep_hp_intr_type intr_type;
+	u64 intr_val;
+};
+
+struct octep_hp_slot {
+	struct list_head list;
+	struct hotplug_slot slot;
+	u16 slot_number;
+	struct pci_dev *hp_pdev;
+	unsigned int hp_devfn;
+	struct octep_hp_controller *ctrl;
+};
+
+struct octep_hp_intr_info {
+	enum octep_hp_intr_type type;
+	int number;
+	char name[16];
+};
+
+struct octep_hp_controller {
+	void __iomem *base;
+	struct pci_dev *pdev;
+	struct octep_hp_intr_info intr[OCTEP_HP_INTR_MAX];
+	struct work_struct work;
+	struct list_head slot_list;
+	struct mutex slot_lock; /* Protects slot_list */
+	struct list_head hp_cmd_list;
+	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
+};
+
+static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl,
+				 struct octep_hp_slot *hp_slot)
+{
+	guard(mutex)(&hp_ctrl->slot_lock);
+	if (hp_slot->hp_pdev) {
+		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already enabled\n",
+			hp_slot->slot_number);
+		return;
+	}
+
+	/* Scan the device and add it to the bus */
+	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus,
+						  hp_slot->hp_devfn);
+	pci_bus_assign_resources(hp_ctrl->pdev->bus);
+	pci_bus_add_device(hp_slot->hp_pdev);
+
+	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n",
+		hp_slot->slot_number);
+}
+
+static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
+				  struct octep_hp_slot *hp_slot)
+{
+	guard(mutex)(&hp_ctrl->slot_lock);
+	if (!hp_slot->hp_pdev) {
+		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n",
+			hp_slot->slot_number);
+		return;
+	}
+
+	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n",
+		hp_slot->slot_number);
+
+	/* Remove the device from the bus */
+	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
+	hp_slot->hp_pdev = NULL;
+}
+
+static int octep_hp_enable_slot(struct hotplug_slot *slot)
+{
+	struct octep_hp_slot *hp_slot =
+		container_of(slot, struct octep_hp_slot, slot);
+
+	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
+	return 0;
+}
+
+static int octep_hp_disable_slot(struct hotplug_slot *slot)
+{
+	struct octep_hp_slot *hp_slot =
+		container_of(slot, struct octep_hp_slot, slot);
+
+	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
+	return 0;
+}
+
+static struct hotplug_slot_ops octep_hp_slot_ops = {
+	.enable_slot = octep_hp_enable_slot,
+	.disable_slot = octep_hp_disable_slot,
+};
+
+#define SLOT_NAME_SIZE 16
+static struct octep_hp_slot *
+octep_hp_register_slot(struct octep_hp_controller *hp_ctrl,
+		       struct pci_dev *pdev, u16 slot_number)
+{
+	char slot_name[SLOT_NAME_SIZE];
+	struct octep_hp_slot *hp_slot;
+	int ret;
+
+	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
+	if (!hp_slot)
+		return ERR_PTR(-ENOMEM);
+
+	hp_slot->ctrl = hp_ctrl;
+	hp_slot->hp_pdev = pdev;
+	hp_slot->hp_devfn = pdev->devfn;
+	hp_slot->slot_number = slot_number;
+	hp_slot->slot.ops = &octep_hp_slot_ops;
+
+	snprintf(slot_name, sizeof(slot_name), "octep_hp_%u", slot_number);
+	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
+			      PCI_SLOT(pdev->devfn), slot_name);
+	if (ret) {
+		kfree(hp_slot);
+		return ERR_PTR(ret);
+	}
+
+	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
+	octep_hp_disable_pdev(hp_ctrl, hp_slot);
+
+	return hp_slot;
+}
+
+static void octep_hp_deregister_slot(void *data)
+{
+	struct octep_hp_slot *hp_slot = data;
+	struct octep_hp_controller *hp_ctrl = hp_slot->ctrl;
+
+	pci_hp_deregister(&hp_slot->slot);
+	octep_hp_enable_pdev(hp_ctrl, hp_slot);
+	list_del(&hp_slot->list);
+	kfree(hp_slot);
+}
+
+static bool octep_hp_is_slot(struct octep_hp_controller *hp_ctrl,
+			     struct pci_dev *pdev)
+{
+	/* Check if the PCI device can be hotplugged */
+	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
+	       PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
+}
+
+static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
+				 struct octep_hp_cmd *hp_cmd)
+{
+	struct octep_hp_slot *hp_slot;
+
+	/*
+	 * Enable or disable the slots based on the slot mask.
+	 * intr_val is a bit mask where each bit represents a slot.
+	 */
+	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
+		if (!(hp_cmd->intr_val & BIT(hp_slot->slot_number)))
+			continue;
+
+		if (hp_cmd->intr_type == OCTEP_HP_INTR_ENA)
+			octep_hp_enable_pdev(hp_ctrl, hp_slot);
+		else
+			octep_hp_disable_pdev(hp_ctrl, hp_slot);
+	}
+}
+
+static void octep_hp_work_handler(struct work_struct *work)
+{
+	struct octep_hp_controller *hp_ctrl;
+	struct octep_hp_cmd *hp_cmd;
+	unsigned long flags;
+
+	hp_ctrl = container_of(work, struct octep_hp_controller, work);
+
+	/* Process all the hotplug commands */
+	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
+	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
+		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list,
+					  struct octep_hp_cmd, list);
+		list_del(&hp_cmd->list);
+		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
+
+		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
+		kfree(hp_cmd);
+
+		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
+	}
+	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
+}
+
+static enum octep_hp_intr_type octep_hp_intr_type(struct octep_hp_intr_info *intr,
+						  int irq)
+{
+	enum octep_hp_intr_type type;
+
+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX; type++) {
+		if (intr[type].number == irq)
+			return type;
+	}
+
+	return OCTEP_HP_INTR_INVALID;
+}
+
+static irqreturn_t octep_hp_intr_handler(int irq, void *data)
+{
+	struct octep_hp_controller *hp_ctrl = data;
+	struct pci_dev *pdev = hp_ctrl->pdev;
+	enum octep_hp_intr_type type;
+	struct octep_hp_cmd *hp_cmd;
+	u64 intr_val;
+
+	type = octep_hp_intr_type(hp_ctrl->intr, irq);
+	if (type == OCTEP_HP_INTR_INVALID) {
+		dev_err(&pdev->dev, "Invalid interrupt %d\n", irq);
+		return IRQ_HANDLED;
+	}
+
+	/* Read and clear the interrupt */
+	intr_val = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
+	writeq(intr_val, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
+
+	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
+	if (!hp_cmd)
+		return IRQ_HANDLED;
+
+	hp_cmd->intr_val = intr_val;
+	hp_cmd->intr_type = type;
+
+	/* Add the command to the list and schedule the work */
+	spin_lock(&hp_ctrl->hp_cmd_lock);
+	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
+	spin_unlock(&hp_ctrl->hp_cmd_lock);
+	schedule_work(&hp_ctrl->work);
+
+	return IRQ_HANDLED;
+}
+
+static void octep_hp_irq_cleanup(void *data)
+{
+	struct octep_hp_controller *hp_ctrl = data;
+
+	pci_free_irq_vectors(hp_ctrl->pdev);
+	flush_work(&hp_ctrl->work);
+}
+
+static int octep_hp_request_irq(struct octep_hp_controller *hp_ctrl,
+				enum octep_hp_intr_type type)
+{
+	struct pci_dev *pdev = hp_ctrl->pdev;
+	struct octep_hp_intr_info *intr;
+	int irq;
+
+	irq = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(type));
+	if (irq < 0)
+		return irq;
+
+	intr = &hp_ctrl->intr[type];
+	intr->number = irq;
+	intr->type = type;
+	snprintf(intr->name, sizeof(intr->name), "octep_hp_%d", type);
+
+	return devm_request_irq(&pdev->dev, irq, octep_hp_intr_handler,
+				IRQF_SHARED, intr->name, hp_ctrl);
+}
+
+static int octep_hp_controller_setup(struct pci_dev *pdev,
+				     struct octep_hp_controller *hp_ctrl)
+{
+	struct device *dev = &pdev->dev;
+	enum octep_hp_intr_type type;
+	int ret;
+
+	ret = pcim_enable_device(pdev);
+	if (ret)
+		return dev_err_probe(dev, ret, "Failed to enable PCI device\n");
+
+	hp_ctrl->base = pcim_iomap_region(pdev, 0, OCTEP_HP_DRV_NAME);
+	if (IS_ERR(hp_ctrl->base))
+		return dev_err_probe(dev, PTR_ERR(hp_ctrl->base),
+				     "Failed to map PCI device region\n");
+
+	pci_set_master(pdev);
+	pci_set_drvdata(pdev, hp_ctrl);
+
+	INIT_LIST_HEAD(&hp_ctrl->slot_list);
+	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
+	mutex_init(&hp_ctrl->slot_lock);
+	spin_lock_init(&hp_ctrl->hp_cmd_lock);
+	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
+	hp_ctrl->pdev = pdev;
+
+	ret = pci_alloc_irq_vectors(pdev, 1,
+				    OCTEP_HP_INTR_VECTOR(OCTEP_HP_INTR_MAX),
+				    PCI_IRQ_MSIX);
+	if (ret < 0)
+		return dev_err_probe(dev, ret, "Failed to alloc MSI-X vectors\n");
+
+	ret = devm_add_action(&pdev->dev, octep_hp_irq_cleanup, hp_ctrl);
+	if (ret)
+		return dev_err_probe(&pdev->dev, ret, "Failed to add IRQ cleanup action\n");
+
+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX; type++) {
+		ret = octep_hp_request_irq(hp_ctrl, type);
+		if (ret)
+			return dev_err_probe(dev, ret,
+					     "Failed to request IRQ for vector %d\n",
+					     OCTEP_HP_INTR_VECTOR(type));
+	}
+
+	return 0;
+}
+
+static int octep_hp_pci_probe(struct pci_dev *pdev,
+			      const struct pci_device_id *id)
+{
+	struct octep_hp_controller *hp_ctrl;
+	struct pci_dev *tmp_pdev = NULL;
+	struct octep_hp_slot *hp_slot;
+	u16 slot_number = 0;
+	int ret;
+
+	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
+	if (!hp_ctrl)
+		return -ENOMEM;
+
+	ret = octep_hp_controller_setup(pdev, hp_ctrl);
+	if (ret)
+		return ret;
+
+	/*
+	 * Register all hotplug slots. Hotplug controller is the first function
+	 * of the PCI device. The hotplug slots are the remaining functions of
+	 * the PCI device. They are removed from the bus and are added back when
+	 * the hotplug event is triggered.
+	 */
+	for_each_pci_dev(tmp_pdev) {
+		if (!octep_hp_is_slot(hp_ctrl, tmp_pdev))
+			continue;
+
+		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev, slot_number);
+		if (IS_ERR(hp_slot))
+			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
+					     "Failed to register hotplug slot %u\n",
+					     slot_number);
+
+		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
+				      hp_slot);
+		if (ret)
+			return dev_err_probe(&pdev->dev, ret,
+					     "Failed to add action for deregistering slot %u\n",
+					     slot_number);
+		slot_number++;
+	}
+
+	return 0;
+}
+
+#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
+static struct pci_device_id octep_hp_pci_map[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, OCTEP_DEVID_HP_CONTROLLER) },
+	{ },
+};
+
+static struct pci_driver octep_hp = {
+	.name = OCTEP_HP_DRV_NAME,
+	.id_table = octep_hp_pci_map,
+	.probe = octep_hp_pci_probe,
+};
+
+module_pci_driver(octep_hp);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Marvell");
+MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
-- 
2.25.1

RE: [PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

Hi Helgaas,

Please check this patch.

>This patch introduces a PCI hotplug controller driver for the OCTEON
>PCIe device, a multi-function PCIe device where the first function acts
>as a hotplug controller. It is equipped with MSI-x interrupts to notify
>the host of hotplug events from the OCTEON firmware.
>
>The driver facilitates the hotplugging of non-controller functions
>within the same device. During probe, non-controller functions are
>removed and registered as PCI hotplug slots. The slots are added back
>only upon request from the device firmware. The driver also allows the
>enabling and disabling of the slots via sysfs slot entries, provided by
>the PCI hotplug framework.
>
>Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
>Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>---
>
>This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
>controller. The OCTEON PCIe device is a multi-function device where the first
>function acts as a PCI hotplug controller.
>
>              +--------------------------------+
>              |           Root Port            |
>              +--------------------------------+
>                              |
>                             PCIe
>                              |
>+---------------------------------------------------------------+
>|              OCTEON PCIe Multifunction Device                 |
>+---------------------------------------------------------------+
>            |                    |              |            |
>            |                    |              |            |
>+---------------------+  +----------------+  +-----+  +----------------+
>|      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>| (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>+---------------------+  +----------------+  +-----+  +----------------+
>            |
>            |
>+-------------------------+
>|   Controller Firmware   |
>+-------------------------+
>
>The hotplug controller driver facilitates the hotplugging of non-controller
>functions in the same device. During the probe of the driver, the non-controller
>function are removed and registered as PCI hotplug slots. They are added back
>only upon request from the device firmware. The driver also allows the user to
>enable/disable the functions using sysfs slot entries provided by PCI hotplug
>framework.
>
>This solution adopts a hardware + software approach for several reasons:
>
>1. To reduce hardware implementation cost. Supporting complete hotplug
>   capability within the card would require a PCI switch implemented within.
>
>2. In the multi-function device, non-controller functions can act as emulated
>   devices. The firmware can dynamically enable or disable them at runtime.
>
>3. Not all root ports support PCI hotplug. This approach provides greater
>   flexibility and compatibility across different hardware configurations.
>
>The hotplug controller function is lightweight and is equipped with MSI-x
>interrupts to notify the host about hotplug events. Upon receiving an
>interrupt, the hotplug register is read, and the required function is enabled
>or disabled.
>
>This driver will be beneficial for managing PCI hotplug events on the OCTEON
>PCIe device without requiring complex hardware solutions.
>
>Changes in v2:
>- Added missing include files.
>- Used dev_err_probe() for error handling.
>- Used guard() for mutex locking.
>- Splited cleanup actions and added per-slot cleanup action.
>- Fixed coding style issues.
>- Added co-developed-by tag.
>
>Changes in v3:
>- Explicit assignment of enum values.
>- Use pcim_iomap_region() instead of pcim_iomap_regions().
>
> MAINTAINERS                    |   6 +
> drivers/pci/hotplug/Kconfig    |  10 +
> drivers/pci/hotplug/Makefile   |   1 +
> drivers/pci/hotplug/octep_hp.c | 409
>+++++++++++++++++++++++++++++++++
> 4 files changed, 426 insertions(+)
> create mode 100644 drivers/pci/hotplug/octep_hp.c
>
>diff --git a/MAINTAINERS b/MAINTAINERS
>index 42decde38320..7b5a618eed1c 100644
>--- a/MAINTAINERS
>+++ b/MAINTAINERS
>@@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
> R:	vattunuru@marvell.com
> F:	drivers/vdpa/octeon_ep/
>
>+MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
>+R:	Shijith Thotton <sthotton@marvell.com>
>+R:	Vamsi Attunuru <vattunuru@marvell.com>
>+S:	Supported
>+F:	drivers/pci/hotplug/octep_hp.c
>+
> MATROX FRAMEBUFFER DRIVER
> L:	linux-fbdev@vger.kernel.org
> S:	Orphan
>diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
>index 1472aef0fb81..2e38fd25f7ef 100644
>--- a/drivers/pci/hotplug/Kconfig
>+++ b/drivers/pci/hotplug/Kconfig
>@@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
>
> 	  When in doubt, say Y.
>
>+config HOTPLUG_PCI_OCTEONEP
>+	bool "OCTEON PCI device Hotplug controller driver"
>+	depends on HOTPLUG_PCI
>+	help
>+	  Say Y here if you have an OCTEON PCIe device with a hotplug
>+	  controller. This driver enables the non-controller functions of the
>+	  device to be registered as hotplug slots.
>+
>+	  When in doubt, say N.
>+
> endif # HOTPLUG_PCI
>diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
>index 240c99517d5e..40aaf31fe338 100644
>--- a/drivers/pci/hotplug/Makefile
>+++ b/drivers/pci/hotplug/Makefile
>@@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+=
>rpaphp.o
> obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
> obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
> obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
>+obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
>
> # acpiphp_ibm extends acpiphp, so should be linked afterwards.
>
>diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
>new file mode 100644
>index 000000000000..77dc2740f43e
>--- /dev/null
>+++ b/drivers/pci/hotplug/octep_hp.c
>@@ -0,0 +1,409 @@
>+// SPDX-License-Identifier: GPL-2.0-only
>+/* Copyright (C) 2024 Marvell. */
>+
>+#include <linux/cleanup.h>
>+#include <linux/container_of.h>
>+#include <linux/delay.h>
>+#include <linux/dev_printk.h>
>+#include <linux/init.h>
>+#include <linux/interrupt.h>
>+#include <linux/io-64-nonatomic-lo-hi.h>
>+#include <linux/kernel.h>
>+#include <linux/list.h>
>+#include <linux/module.h>
>+#include <linux/mutex.h>
>+#include <linux/pci.h>
>+#include <linux/pci_hotplug.h>
>+#include <linux/slab.h>
>+#include <linux/spinlock.h>
>+#include <linux/workqueue.h>
>+
>+#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
>+#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
>+#define OCTEP_HP_DRV_NAME "octep_hp"
>+
>+/*
>+ * Type of MSI-X interrupts.
>+ * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are
>used to
>+ * generate the vector and offset for an interrupt type.
>+ */
>+enum octep_hp_intr_type {
>+	OCTEP_HP_INTR_INVALID = -1,
>+	OCTEP_HP_INTR_ENA = 0,
>+	OCTEP_HP_INTR_DIS = 1,
>+	OCTEP_HP_INTR_MAX = 2,
>+};
>+
>+struct octep_hp_cmd {
>+	struct list_head list;
>+	enum octep_hp_intr_type intr_type;
>+	u64 intr_val;
>+};
>+
>+struct octep_hp_slot {
>+	struct list_head list;
>+	struct hotplug_slot slot;
>+	u16 slot_number;
>+	struct pci_dev *hp_pdev;
>+	unsigned int hp_devfn;
>+	struct octep_hp_controller *ctrl;
>+};
>+
>+struct octep_hp_intr_info {
>+	enum octep_hp_intr_type type;
>+	int number;
>+	char name[16];
>+};
>+
>+struct octep_hp_controller {
>+	void __iomem *base;
>+	struct pci_dev *pdev;
>+	struct octep_hp_intr_info intr[OCTEP_HP_INTR_MAX];
>+	struct work_struct work;
>+	struct list_head slot_list;
>+	struct mutex slot_lock; /* Protects slot_list */
>+	struct list_head hp_cmd_list;
>+	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
>+};
>+
>+static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl,
>+				 struct octep_hp_slot *hp_slot)
>+{
>+	guard(mutex)(&hp_ctrl->slot_lock);
>+	if (hp_slot->hp_pdev) {
>+		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already
>enabled\n",
>+			hp_slot->slot_number);
>+		return;
>+	}
>+
>+	/* Scan the device and add it to the bus */
>+	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus,
>+						  hp_slot->hp_devfn);
>+	pci_bus_assign_resources(hp_ctrl->pdev->bus);
>+	pci_bus_add_device(hp_slot->hp_pdev);
>+
>+	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n",
>+		hp_slot->slot_number);
>+}
>+
>+static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
>+				  struct octep_hp_slot *hp_slot)
>+{
>+	guard(mutex)(&hp_ctrl->slot_lock);
>+	if (!hp_slot->hp_pdev) {
>+		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n",
>+			hp_slot->slot_number);
>+		return;
>+	}
>+
>+	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n",
>+		hp_slot->slot_number);
>+
>+	/* Remove the device from the bus */
>+	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
>+	hp_slot->hp_pdev = NULL;
>+}
>+
>+static int octep_hp_enable_slot(struct hotplug_slot *slot)
>+{
>+	struct octep_hp_slot *hp_slot =
>+		container_of(slot, struct octep_hp_slot, slot);
>+
>+	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
>+	return 0;
>+}
>+
>+static int octep_hp_disable_slot(struct hotplug_slot *slot)
>+{
>+	struct octep_hp_slot *hp_slot =
>+		container_of(slot, struct octep_hp_slot, slot);
>+
>+	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
>+	return 0;
>+}
>+
>+static struct hotplug_slot_ops octep_hp_slot_ops = {
>+	.enable_slot = octep_hp_enable_slot,
>+	.disable_slot = octep_hp_disable_slot,
>+};
>+
>+#define SLOT_NAME_SIZE 16
>+static struct octep_hp_slot *
>+octep_hp_register_slot(struct octep_hp_controller *hp_ctrl,
>+		       struct pci_dev *pdev, u16 slot_number)
>+{
>+	char slot_name[SLOT_NAME_SIZE];
>+	struct octep_hp_slot *hp_slot;
>+	int ret;
>+
>+	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
>+	if (!hp_slot)
>+		return ERR_PTR(-ENOMEM);
>+
>+	hp_slot->ctrl = hp_ctrl;
>+	hp_slot->hp_pdev = pdev;
>+	hp_slot->hp_devfn = pdev->devfn;
>+	hp_slot->slot_number = slot_number;
>+	hp_slot->slot.ops = &octep_hp_slot_ops;
>+
>+	snprintf(slot_name, sizeof(slot_name), "octep_hp_%u", slot_number);
>+	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
>+			      PCI_SLOT(pdev->devfn), slot_name);
>+	if (ret) {
>+		kfree(hp_slot);
>+		return ERR_PTR(ret);
>+	}
>+
>+	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
>+	octep_hp_disable_pdev(hp_ctrl, hp_slot);
>+
>+	return hp_slot;
>+}
>+
>+static void octep_hp_deregister_slot(void *data)
>+{
>+	struct octep_hp_slot *hp_slot = data;
>+	struct octep_hp_controller *hp_ctrl = hp_slot->ctrl;
>+
>+	pci_hp_deregister(&hp_slot->slot);
>+	octep_hp_enable_pdev(hp_ctrl, hp_slot);
>+	list_del(&hp_slot->list);
>+	kfree(hp_slot);
>+}
>+
>+static bool octep_hp_is_slot(struct octep_hp_controller *hp_ctrl,
>+			     struct pci_dev *pdev)
>+{
>+	/* Check if the PCI device can be hotplugged */
>+	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
>+	       PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
>+}
>+
>+static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
>+				 struct octep_hp_cmd *hp_cmd)
>+{
>+	struct octep_hp_slot *hp_slot;
>+
>+	/*
>+	 * Enable or disable the slots based on the slot mask.
>+	 * intr_val is a bit mask where each bit represents a slot.
>+	 */
>+	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
>+		if (!(hp_cmd->intr_val & BIT(hp_slot->slot_number)))
>+			continue;
>+
>+		if (hp_cmd->intr_type == OCTEP_HP_INTR_ENA)
>+			octep_hp_enable_pdev(hp_ctrl, hp_slot);
>+		else
>+			octep_hp_disable_pdev(hp_ctrl, hp_slot);
>+	}
>+}
>+
>+static void octep_hp_work_handler(struct work_struct *work)
>+{
>+	struct octep_hp_controller *hp_ctrl;
>+	struct octep_hp_cmd *hp_cmd;
>+	unsigned long flags;
>+
>+	hp_ctrl = container_of(work, struct octep_hp_controller, work);
>+
>+	/* Process all the hotplug commands */
>+	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>+	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
>+		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list,
>+					  struct octep_hp_cmd, list);
>+		list_del(&hp_cmd->list);
>+		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>+
>+		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
>+		kfree(hp_cmd);
>+
>+		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>+	}
>+	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>+}
>+
>+static enum octep_hp_intr_type octep_hp_intr_type(struct
>octep_hp_intr_info *intr,
>+						  int irq)
>+{
>+	enum octep_hp_intr_type type;
>+
>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>type++) {
>+		if (intr[type].number == irq)
>+			return type;
>+	}
>+
>+	return OCTEP_HP_INTR_INVALID;
>+}
>+
>+static irqreturn_t octep_hp_intr_handler(int irq, void *data)
>+{
>+	struct octep_hp_controller *hp_ctrl = data;
>+	struct pci_dev *pdev = hp_ctrl->pdev;
>+	enum octep_hp_intr_type type;
>+	struct octep_hp_cmd *hp_cmd;
>+	u64 intr_val;
>+
>+	type = octep_hp_intr_type(hp_ctrl->intr, irq);
>+	if (type == OCTEP_HP_INTR_INVALID) {
>+		dev_err(&pdev->dev, "Invalid interrupt %d\n", irq);
>+		return IRQ_HANDLED;
>+	}
>+
>+	/* Read and clear the interrupt */
>+	intr_val = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>+	writeq(intr_val, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>+
>+	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
>+	if (!hp_cmd)
>+		return IRQ_HANDLED;
>+
>+	hp_cmd->intr_val = intr_val;
>+	hp_cmd->intr_type = type;
>+
>+	/* Add the command to the list and schedule the work */
>+	spin_lock(&hp_ctrl->hp_cmd_lock);
>+	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
>+	spin_unlock(&hp_ctrl->hp_cmd_lock);
>+	schedule_work(&hp_ctrl->work);
>+
>+	return IRQ_HANDLED;
>+}
>+
>+static void octep_hp_irq_cleanup(void *data)
>+{
>+	struct octep_hp_controller *hp_ctrl = data;
>+
>+	pci_free_irq_vectors(hp_ctrl->pdev);
>+	flush_work(&hp_ctrl->work);
>+}
>+
>+static int octep_hp_request_irq(struct octep_hp_controller *hp_ctrl,
>+				enum octep_hp_intr_type type)
>+{
>+	struct pci_dev *pdev = hp_ctrl->pdev;
>+	struct octep_hp_intr_info *intr;
>+	int irq;
>+
>+	irq = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(type));
>+	if (irq < 0)
>+		return irq;
>+
>+	intr = &hp_ctrl->intr[type];
>+	intr->number = irq;
>+	intr->type = type;
>+	snprintf(intr->name, sizeof(intr->name), "octep_hp_%d", type);
>+
>+	return devm_request_irq(&pdev->dev, irq, octep_hp_intr_handler,
>+				IRQF_SHARED, intr->name, hp_ctrl);
>+}
>+
>+static int octep_hp_controller_setup(struct pci_dev *pdev,
>+				     struct octep_hp_controller *hp_ctrl)
>+{
>+	struct device *dev = &pdev->dev;
>+	enum octep_hp_intr_type type;
>+	int ret;
>+
>+	ret = pcim_enable_device(pdev);
>+	if (ret)
>+		return dev_err_probe(dev, ret, "Failed to enable PCI device\n");
>+
>+	hp_ctrl->base = pcim_iomap_region(pdev, 0, OCTEP_HP_DRV_NAME);
>+	if (IS_ERR(hp_ctrl->base))
>+		return dev_err_probe(dev, PTR_ERR(hp_ctrl->base),
>+				     "Failed to map PCI device region\n");
>+
>+	pci_set_master(pdev);
>+	pci_set_drvdata(pdev, hp_ctrl);
>+
>+	INIT_LIST_HEAD(&hp_ctrl->slot_list);
>+	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
>+	mutex_init(&hp_ctrl->slot_lock);
>+	spin_lock_init(&hp_ctrl->hp_cmd_lock);
>+	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
>+	hp_ctrl->pdev = pdev;
>+
>+	ret = pci_alloc_irq_vectors(pdev, 1,
>+
>OCTEP_HP_INTR_VECTOR(OCTEP_HP_INTR_MAX),
>+				    PCI_IRQ_MSIX);
>+	if (ret < 0)
>+		return dev_err_probe(dev, ret, "Failed to alloc MSI-X
>vectors\n");
>+
>+	ret = devm_add_action(&pdev->dev, octep_hp_irq_cleanup, hp_ctrl);
>+	if (ret)
>+		return dev_err_probe(&pdev->dev, ret, "Failed to add IRQ
>cleanup action\n");
>+
>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>type++) {
>+		ret = octep_hp_request_irq(hp_ctrl, type);
>+		if (ret)
>+			return dev_err_probe(dev, ret,
>+					     "Failed to request IRQ for vector
>%d\n",
>+					     OCTEP_HP_INTR_VECTOR(type));
>+	}
>+
>+	return 0;
>+}
>+
>+static int octep_hp_pci_probe(struct pci_dev *pdev,
>+			      const struct pci_device_id *id)
>+{
>+	struct octep_hp_controller *hp_ctrl;
>+	struct pci_dev *tmp_pdev = NULL;
>+	struct octep_hp_slot *hp_slot;
>+	u16 slot_number = 0;
>+	int ret;
>+
>+	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
>+	if (!hp_ctrl)
>+		return -ENOMEM;
>+
>+	ret = octep_hp_controller_setup(pdev, hp_ctrl);
>+	if (ret)
>+		return ret;
>+
>+	/*
>+	 * Register all hotplug slots. Hotplug controller is the first function
>+	 * of the PCI device. The hotplug slots are the remaining functions of
>+	 * the PCI device. They are removed from the bus and are added back
>when
>+	 * the hotplug event is triggered.
>+	 */
>+	for_each_pci_dev(tmp_pdev) {
>+		if (!octep_hp_is_slot(hp_ctrl, tmp_pdev))
>+			continue;
>+
>+		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev,
>slot_number);
>+		if (IS_ERR(hp_slot))
>+			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
>+					     "Failed to register hotplug slot
>%u\n",
>+					     slot_number);
>+
>+		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
>+				      hp_slot);
>+		if (ret)
>+			return dev_err_probe(&pdev->dev, ret,
>+					     "Failed to add action for
>deregistering slot %u\n",
>+					     slot_number);
>+		slot_number++;
>+	}
>+
>+	return 0;
>+}
>+
>+#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
>+static struct pci_device_id octep_hp_pci_map[] = {
>+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
>OCTEP_DEVID_HP_CONTROLLER) },
>+	{ },
>+};
>+
>+static struct pci_driver octep_hp = {
>+	.name = OCTEP_HP_DRV_NAME,
>+	.id_table = octep_hp_pci_map,
>+	.probe = octep_hp_pci_probe,
>+};
>+
>+module_pci_driver(octep_hp);
>+
>+MODULE_LICENSE("GPL");
>+MODULE_AUTHOR("Marvell");
>+MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
>--
>2.25.1

Thanks,
Shijith

RE: [PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

Just a gentle reminder.
Could you please comments if there are improvements needed?

>Hi Helgaas,
>
>Please check this patch.
>
>>This patch introduces a PCI hotplug controller driver for the OCTEON
>>PCIe device, a multi-function PCIe device where the first function acts
>>as a hotplug controller. It is equipped with MSI-x interrupts to notify
>>the host of hotplug events from the OCTEON firmware.
>>
>>The driver facilitates the hotplugging of non-controller functions
>>within the same device. During probe, non-controller functions are
>>removed and registered as PCI hotplug slots. The slots are added back
>>only upon request from the device firmware. The driver also allows the
>>enabling and disabling of the slots via sysfs slot entries, provided by
>>the PCI hotplug framework.
>>
>>Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>>Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
>>Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>>---
>>
>>This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
>>controller. The OCTEON PCIe device is a multi-function device where the first
>>function acts as a PCI hotplug controller.
>>
>>              +--------------------------------+
>>              |           Root Port            |
>>              +--------------------------------+
>>                              |
>>                             PCIe
>>                              |
>>+---------------------------------------------------------------+
>>|              OCTEON PCIe Multifunction Device                 |
>>+---------------------------------------------------------------+
>>            |                    |              |            |
>>            |                    |              |            |
>>+---------------------+  +----------------+  +-----+  +----------------+
>>|      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>>| (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>>+---------------------+  +----------------+  +-----+  +----------------+
>>            |
>>            |
>>+-------------------------+
>>|   Controller Firmware   |
>>+-------------------------+
>>
>>The hotplug controller driver facilitates the hotplugging of non-controller
>>functions in the same device. During the probe of the driver, the non-
>controller
>>function are removed and registered as PCI hotplug slots. They are added back
>>only upon request from the device firmware. The driver also allows the user to
>>enable/disable the functions using sysfs slot entries provided by PCI hotplug
>>framework.
>>
>>This solution adopts a hardware + software approach for several reasons:
>>
>>1. To reduce hardware implementation cost. Supporting complete hotplug
>>   capability within the card would require a PCI switch implemented within.
>>
>>2. In the multi-function device, non-controller functions can act as emulated
>>   devices. The firmware can dynamically enable or disable them at runtime.
>>
>>3. Not all root ports support PCI hotplug. This approach provides greater
>>   flexibility and compatibility across different hardware configurations.
>>
>>The hotplug controller function is lightweight and is equipped with MSI-x
>>interrupts to notify the host about hotplug events. Upon receiving an
>>interrupt, the hotplug register is read, and the required function is enabled
>>or disabled.
>>
>>This driver will be beneficial for managing PCI hotplug events on the OCTEON
>>PCIe device without requiring complex hardware solutions.
>>
>>Changes in v2:
>>- Added missing include files.
>>- Used dev_err_probe() for error handling.
>>- Used guard() for mutex locking.
>>- Splited cleanup actions and added per-slot cleanup action.
>>- Fixed coding style issues.
>>- Added co-developed-by tag.
>>
>>Changes in v3:
>>- Explicit assignment of enum values.
>>- Use pcim_iomap_region() instead of pcim_iomap_regions().
>>
>> MAINTAINERS                    |   6 +
>> drivers/pci/hotplug/Kconfig    |  10 +
>> drivers/pci/hotplug/Makefile   |   1 +
>> drivers/pci/hotplug/octep_hp.c | 409
>>+++++++++++++++++++++++++++++++++
>> 4 files changed, 426 insertions(+)
>> create mode 100644 drivers/pci/hotplug/octep_hp.c
>>
>>diff --git a/MAINTAINERS b/MAINTAINERS
>>index 42decde38320..7b5a618eed1c 100644
>>--- a/MAINTAINERS
>>+++ b/MAINTAINERS
>>@@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
>> R:	vattunuru@marvell.com
>> F:	drivers/vdpa/octeon_ep/
>>
>>+MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
>>+R:	Shijith Thotton <sthotton@marvell.com>
>>+R:	Vamsi Attunuru <vattunuru@marvell.com>
>>+S:	Supported
>>+F:	drivers/pci/hotplug/octep_hp.c
>>+
>> MATROX FRAMEBUFFER DRIVER
>> L:	linux-fbdev@vger.kernel.org
>> S:	Orphan
>>diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
>>index 1472aef0fb81..2e38fd25f7ef 100644
>>--- a/drivers/pci/hotplug/Kconfig
>>+++ b/drivers/pci/hotplug/Kconfig
>>@@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
>>
>> 	  When in doubt, say Y.
>>
>>+config HOTPLUG_PCI_OCTEONEP
>>+	bool "OCTEON PCI device Hotplug controller driver"
>>+	depends on HOTPLUG_PCI
>>+	help
>>+	  Say Y here if you have an OCTEON PCIe device with a hotplug
>>+	  controller. This driver enables the non-controller functions of the
>>+	  device to be registered as hotplug slots.
>>+
>>+	  When in doubt, say N.
>>+
>> endif # HOTPLUG_PCI
>>diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
>>index 240c99517d5e..40aaf31fe338 100644
>>--- a/drivers/pci/hotplug/Makefile
>>+++ b/drivers/pci/hotplug/Makefile
>>@@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+=
>>rpaphp.o
>> obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
>> obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
>> obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
>>+obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
>>
>> # acpiphp_ibm extends acpiphp, so should be linked afterwards.
>>
>>diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
>>new file mode 100644
>>index 000000000000..77dc2740f43e
>>--- /dev/null
>>+++ b/drivers/pci/hotplug/octep_hp.c
>>@@ -0,0 +1,409 @@
>>+// SPDX-License-Identifier: GPL-2.0-only
>>+/* Copyright (C) 2024 Marvell. */
>>+
>>+#include <linux/cleanup.h>
>>+#include <linux/container_of.h>
>>+#include <linux/delay.h>
>>+#include <linux/dev_printk.h>
>>+#include <linux/init.h>
>>+#include <linux/interrupt.h>
>>+#include <linux/io-64-nonatomic-lo-hi.h>
>>+#include <linux/kernel.h>
>>+#include <linux/list.h>
>>+#include <linux/module.h>
>>+#include <linux/mutex.h>
>>+#include <linux/pci.h>
>>+#include <linux/pci_hotplug.h>
>>+#include <linux/slab.h>
>>+#include <linux/spinlock.h>
>>+#include <linux/workqueue.h>
>>+
>>+#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
>>+#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
>>+#define OCTEP_HP_DRV_NAME "octep_hp"
>>+
>>+/*
>>+ * Type of MSI-X interrupts.
>>+ * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are
>>used to
>>+ * generate the vector and offset for an interrupt type.
>>+ */
>>+enum octep_hp_intr_type {
>>+	OCTEP_HP_INTR_INVALID = -1,
>>+	OCTEP_HP_INTR_ENA = 0,
>>+	OCTEP_HP_INTR_DIS = 1,
>>+	OCTEP_HP_INTR_MAX = 2,
>>+};
>>+
>>+struct octep_hp_cmd {
>>+	struct list_head list;
>>+	enum octep_hp_intr_type intr_type;
>>+	u64 intr_val;
>>+};
>>+
>>+struct octep_hp_slot {
>>+	struct list_head list;
>>+	struct hotplug_slot slot;
>>+	u16 slot_number;
>>+	struct pci_dev *hp_pdev;
>>+	unsigned int hp_devfn;
>>+	struct octep_hp_controller *ctrl;
>>+};
>>+
>>+struct octep_hp_intr_info {
>>+	enum octep_hp_intr_type type;
>>+	int number;
>>+	char name[16];
>>+};
>>+
>>+struct octep_hp_controller {
>>+	void __iomem *base;
>>+	struct pci_dev *pdev;
>>+	struct octep_hp_intr_info intr[OCTEP_HP_INTR_MAX];
>>+	struct work_struct work;
>>+	struct list_head slot_list;
>>+	struct mutex slot_lock; /* Protects slot_list */
>>+	struct list_head hp_cmd_list;
>>+	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
>>+};
>>+
>>+static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl,
>>+				 struct octep_hp_slot *hp_slot)
>>+{
>>+	guard(mutex)(&hp_ctrl->slot_lock);
>>+	if (hp_slot->hp_pdev) {
>>+		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already
>>enabled\n",
>>+			hp_slot->slot_number);
>>+		return;
>>+	}
>>+
>>+	/* Scan the device and add it to the bus */
>>+	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus,
>>+						  hp_slot->hp_devfn);
>>+	pci_bus_assign_resources(hp_ctrl->pdev->bus);
>>+	pci_bus_add_device(hp_slot->hp_pdev);
>>+
>>+	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n",
>>+		hp_slot->slot_number);
>>+}
>>+
>>+static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
>>+				  struct octep_hp_slot *hp_slot)
>>+{
>>+	guard(mutex)(&hp_ctrl->slot_lock);
>>+	if (!hp_slot->hp_pdev) {
>>+		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n",
>>+			hp_slot->slot_number);
>>+		return;
>>+	}
>>+
>>+	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n",
>>+		hp_slot->slot_number);
>>+
>>+	/* Remove the device from the bus */
>>+	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
>>+	hp_slot->hp_pdev = NULL;
>>+}
>>+
>>+static int octep_hp_enable_slot(struct hotplug_slot *slot)
>>+{
>>+	struct octep_hp_slot *hp_slot =
>>+		container_of(slot, struct octep_hp_slot, slot);
>>+
>>+	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
>>+	return 0;
>>+}
>>+
>>+static int octep_hp_disable_slot(struct hotplug_slot *slot)
>>+{
>>+	struct octep_hp_slot *hp_slot =
>>+		container_of(slot, struct octep_hp_slot, slot);
>>+
>>+	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
>>+	return 0;
>>+}
>>+
>>+static struct hotplug_slot_ops octep_hp_slot_ops = {
>>+	.enable_slot = octep_hp_enable_slot,
>>+	.disable_slot = octep_hp_disable_slot,
>>+};
>>+
>>+#define SLOT_NAME_SIZE 16
>>+static struct octep_hp_slot *
>>+octep_hp_register_slot(struct octep_hp_controller *hp_ctrl,
>>+		       struct pci_dev *pdev, u16 slot_number)
>>+{
>>+	char slot_name[SLOT_NAME_SIZE];
>>+	struct octep_hp_slot *hp_slot;
>>+	int ret;
>>+
>>+	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
>>+	if (!hp_slot)
>>+		return ERR_PTR(-ENOMEM);
>>+
>>+	hp_slot->ctrl = hp_ctrl;
>>+	hp_slot->hp_pdev = pdev;
>>+	hp_slot->hp_devfn = pdev->devfn;
>>+	hp_slot->slot_number = slot_number;
>>+	hp_slot->slot.ops = &octep_hp_slot_ops;
>>+
>>+	snprintf(slot_name, sizeof(slot_name), "octep_hp_%u", slot_number);
>>+	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
>>+			      PCI_SLOT(pdev->devfn), slot_name);
>>+	if (ret) {
>>+		kfree(hp_slot);
>>+		return ERR_PTR(ret);
>>+	}
>>+
>>+	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
>>+	octep_hp_disable_pdev(hp_ctrl, hp_slot);
>>+
>>+	return hp_slot;
>>+}
>>+
>>+static void octep_hp_deregister_slot(void *data)
>>+{
>>+	struct octep_hp_slot *hp_slot = data;
>>+	struct octep_hp_controller *hp_ctrl = hp_slot->ctrl;
>>+
>>+	pci_hp_deregister(&hp_slot->slot);
>>+	octep_hp_enable_pdev(hp_ctrl, hp_slot);
>>+	list_del(&hp_slot->list);
>>+	kfree(hp_slot);
>>+}
>>+
>>+static bool octep_hp_is_slot(struct octep_hp_controller *hp_ctrl,
>>+			     struct pci_dev *pdev)
>>+{
>>+	/* Check if the PCI device can be hotplugged */
>>+	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
>>+	       PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
>>+}
>>+
>>+static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
>>+				 struct octep_hp_cmd *hp_cmd)
>>+{
>>+	struct octep_hp_slot *hp_slot;
>>+
>>+	/*
>>+	 * Enable or disable the slots based on the slot mask.
>>+	 * intr_val is a bit mask where each bit represents a slot.
>>+	 */
>>+	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
>>+		if (!(hp_cmd->intr_val & BIT(hp_slot->slot_number)))
>>+			continue;
>>+
>>+		if (hp_cmd->intr_type == OCTEP_HP_INTR_ENA)
>>+			octep_hp_enable_pdev(hp_ctrl, hp_slot);
>>+		else
>>+			octep_hp_disable_pdev(hp_ctrl, hp_slot);
>>+	}
>>+}
>>+
>>+static void octep_hp_work_handler(struct work_struct *work)
>>+{
>>+	struct octep_hp_controller *hp_ctrl;
>>+	struct octep_hp_cmd *hp_cmd;
>>+	unsigned long flags;
>>+
>>+	hp_ctrl = container_of(work, struct octep_hp_controller, work);
>>+
>>+	/* Process all the hotplug commands */
>>+	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>>+	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
>>+		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list,
>>+					  struct octep_hp_cmd, list);
>>+		list_del(&hp_cmd->list);
>>+		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>>+
>>+		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
>>+		kfree(hp_cmd);
>>+
>>+		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>>+	}
>>+	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>>+}
>>+
>>+static enum octep_hp_intr_type octep_hp_intr_type(struct
>>octep_hp_intr_info *intr,
>>+						  int irq)
>>+{
>>+	enum octep_hp_intr_type type;
>>+
>>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>>type++) {
>>+		if (intr[type].number == irq)
>>+			return type;
>>+	}
>>+
>>+	return OCTEP_HP_INTR_INVALID;
>>+}
>>+
>>+static irqreturn_t octep_hp_intr_handler(int irq, void *data)
>>+{
>>+	struct octep_hp_controller *hp_ctrl = data;
>>+	struct pci_dev *pdev = hp_ctrl->pdev;
>>+	enum octep_hp_intr_type type;
>>+	struct octep_hp_cmd *hp_cmd;
>>+	u64 intr_val;
>>+
>>+	type = octep_hp_intr_type(hp_ctrl->intr, irq);
>>+	if (type == OCTEP_HP_INTR_INVALID) {
>>+		dev_err(&pdev->dev, "Invalid interrupt %d\n", irq);
>>+		return IRQ_HANDLED;
>>+	}
>>+
>>+	/* Read and clear the interrupt */
>>+	intr_val = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>>+	writeq(intr_val, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>>+
>>+	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
>>+	if (!hp_cmd)
>>+		return IRQ_HANDLED;
>>+
>>+	hp_cmd->intr_val = intr_val;
>>+	hp_cmd->intr_type = type;
>>+
>>+	/* Add the command to the list and schedule the work */
>>+	spin_lock(&hp_ctrl->hp_cmd_lock);
>>+	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
>>+	spin_unlock(&hp_ctrl->hp_cmd_lock);
>>+	schedule_work(&hp_ctrl->work);
>>+
>>+	return IRQ_HANDLED;
>>+}
>>+
>>+static void octep_hp_irq_cleanup(void *data)
>>+{
>>+	struct octep_hp_controller *hp_ctrl = data;
>>+
>>+	pci_free_irq_vectors(hp_ctrl->pdev);
>>+	flush_work(&hp_ctrl->work);
>>+}
>>+
>>+static int octep_hp_request_irq(struct octep_hp_controller *hp_ctrl,
>>+				enum octep_hp_intr_type type)
>>+{
>>+	struct pci_dev *pdev = hp_ctrl->pdev;
>>+	struct octep_hp_intr_info *intr;
>>+	int irq;
>>+
>>+	irq = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(type));
>>+	if (irq < 0)
>>+		return irq;
>>+
>>+	intr = &hp_ctrl->intr[type];
>>+	intr->number = irq;
>>+	intr->type = type;
>>+	snprintf(intr->name, sizeof(intr->name), "octep_hp_%d", type);
>>+
>>+	return devm_request_irq(&pdev->dev, irq, octep_hp_intr_handler,
>>+				IRQF_SHARED, intr->name, hp_ctrl);
>>+}
>>+
>>+static int octep_hp_controller_setup(struct pci_dev *pdev,
>>+				     struct octep_hp_controller *hp_ctrl)
>>+{
>>+	struct device *dev = &pdev->dev;
>>+	enum octep_hp_intr_type type;
>>+	int ret;
>>+
>>+	ret = pcim_enable_device(pdev);
>>+	if (ret)
>>+		return dev_err_probe(dev, ret, "Failed to enable PCI device\n");
>>+
>>+	hp_ctrl->base = pcim_iomap_region(pdev, 0, OCTEP_HP_DRV_NAME);
>>+	if (IS_ERR(hp_ctrl->base))
>>+		return dev_err_probe(dev, PTR_ERR(hp_ctrl->base),
>>+				     "Failed to map PCI device region\n");
>>+
>>+	pci_set_master(pdev);
>>+	pci_set_drvdata(pdev, hp_ctrl);
>>+
>>+	INIT_LIST_HEAD(&hp_ctrl->slot_list);
>>+	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
>>+	mutex_init(&hp_ctrl->slot_lock);
>>+	spin_lock_init(&hp_ctrl->hp_cmd_lock);
>>+	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
>>+	hp_ctrl->pdev = pdev;
>>+
>>+	ret = pci_alloc_irq_vectors(pdev, 1,
>>+
>>OCTEP_HP_INTR_VECTOR(OCTEP_HP_INTR_MAX),
>>+				    PCI_IRQ_MSIX);
>>+	if (ret < 0)
>>+		return dev_err_probe(dev, ret, "Failed to alloc MSI-X
>>vectors\n");
>>+
>>+	ret = devm_add_action(&pdev->dev, octep_hp_irq_cleanup, hp_ctrl);
>>+	if (ret)
>>+		return dev_err_probe(&pdev->dev, ret, "Failed to add IRQ
>>cleanup action\n");
>>+
>>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>>type++) {
>>+		ret = octep_hp_request_irq(hp_ctrl, type);
>>+		if (ret)
>>+			return dev_err_probe(dev, ret,
>>+					     "Failed to request IRQ for vector
>>%d\n",
>>+					     OCTEP_HP_INTR_VECTOR(type));
>>+	}
>>+
>>+	return 0;
>>+}
>>+
>>+static int octep_hp_pci_probe(struct pci_dev *pdev,
>>+			      const struct pci_device_id *id)
>>+{
>>+	struct octep_hp_controller *hp_ctrl;
>>+	struct pci_dev *tmp_pdev = NULL;
>>+	struct octep_hp_slot *hp_slot;
>>+	u16 slot_number = 0;
>>+	int ret;
>>+
>>+	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
>>+	if (!hp_ctrl)
>>+		return -ENOMEM;
>>+
>>+	ret = octep_hp_controller_setup(pdev, hp_ctrl);
>>+	if (ret)
>>+		return ret;
>>+
>>+	/*
>>+	 * Register all hotplug slots. Hotplug controller is the first function
>>+	 * of the PCI device. The hotplug slots are the remaining functions of
>>+	 * the PCI device. They are removed from the bus and are added back
>>when
>>+	 * the hotplug event is triggered.
>>+	 */
>>+	for_each_pci_dev(tmp_pdev) {
>>+		if (!octep_hp_is_slot(hp_ctrl, tmp_pdev))
>>+			continue;
>>+
>>+		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev,
>>slot_number);
>>+		if (IS_ERR(hp_slot))
>>+			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
>>+					     "Failed to register hotplug slot
>>%u\n",
>>+					     slot_number);
>>+
>>+		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
>>+				      hp_slot);
>>+		if (ret)
>>+			return dev_err_probe(&pdev->dev, ret,
>>+					     "Failed to add action for
>>deregistering slot %u\n",
>>+					     slot_number);
>>+		slot_number++;
>>+	}
>>+
>>+	return 0;
>>+}
>>+
>>+#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
>>+static struct pci_device_id octep_hp_pci_map[] = {
>>+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
>>OCTEP_DEVID_HP_CONTROLLER) },
>>+	{ },
>>+};
>>+
>>+static struct pci_driver octep_hp = {
>>+	.name = OCTEP_HP_DRV_NAME,
>>+	.id_table = octep_hp_pci_map,
>>+	.probe = octep_hp_pci_probe,
>>+};
>>+
>>+module_pci_driver(octep_hp);
>>+
>>+MODULE_LICENSE("GPL");
>>+MODULE_AUTHOR("Marvell");
>>+MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
>>--
>>2.25.1
>

Thanks,
Shijith

RE: [PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

Just a gentle reminder to review the patch when you get a chance.
let me know if any adjustments are needed. Thanks!

>Just a gentle reminder.
>Could you please comments if there are improvements needed?
>
>>Hi Helgaas,
>>
>>Please check this patch.
>>
>>>This patch introduces a PCI hotplug controller driver for the OCTEON
>>>PCIe device, a multi-function PCIe device where the first function acts
>>>as a hotplug controller. It is equipped with MSI-x interrupts to notify
>>>the host of hotplug events from the OCTEON firmware.
>>>
>>>The driver facilitates the hotplugging of non-controller functions
>>>within the same device. During probe, non-controller functions are
>>>removed and registered as PCI hotplug slots. The slots are added back
>>>only upon request from the device firmware. The driver also allows the
>>>enabling and disabling of the slots via sysfs slot entries, provided by
>>>the PCI hotplug framework.
>>>
>>>Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>>>Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
>>>Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>>>---
>>>
>>>This patch introduces a PCI hotplug controller driver for OCTEON PCIe
>hotplug
>>>controller. The OCTEON PCIe device is a multi-function device where the first
>>>function acts as a PCI hotplug controller.
>>>
>>>              +--------------------------------+
>>>              |           Root Port            |
>>>              +--------------------------------+
>>>                              |
>>>                             PCIe
>>>                              |
>>>+---------------------------------------------------------------+
>>>|              OCTEON PCIe Multifunction Device                 |
>>>+---------------------------------------------------------------+
>>>            |                    |              |            |
>>>            |                    |              |            |
>>>+---------------------+  +----------------+  +-----+  +----------------+
>>>|      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>>>| (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>>>+---------------------+  +----------------+  +-----+  +----------------+
>>>            |
>>>            |
>>>+-------------------------+
>>>|   Controller Firmware   |
>>>+-------------------------+
>>>
>>>The hotplug controller driver facilitates the hotplugging of non-controller
>>>functions in the same device. During the probe of the driver, the non-
>>controller
>>>function are removed and registered as PCI hotplug slots. They are added
>back
>>>only upon request from the device firmware. The driver also allows the user
>to
>>>enable/disable the functions using sysfs slot entries provided by PCI hotplug
>>>framework.
>>>
>>>This solution adopts a hardware + software approach for several reasons:
>>>
>>>1. To reduce hardware implementation cost. Supporting complete hotplug
>>>   capability within the card would require a PCI switch implemented within.
>>>
>>>2. In the multi-function device, non-controller functions can act as emulated
>>>   devices. The firmware can dynamically enable or disable them at runtime.
>>>
>>>3. Not all root ports support PCI hotplug. This approach provides greater
>>>   flexibility and compatibility across different hardware configurations.
>>>
>>>The hotplug controller function is lightweight and is equipped with MSI-x
>>>interrupts to notify the host about hotplug events. Upon receiving an
>>>interrupt, the hotplug register is read, and the required function is enabled
>>>or disabled.
>>>
>>>This driver will be beneficial for managing PCI hotplug events on the OCTEON
>>>PCIe device without requiring complex hardware solutions.
>>>
>>>Changes in v2:
>>>- Added missing include files.
>>>- Used dev_err_probe() for error handling.
>>>- Used guard() for mutex locking.
>>>- Splited cleanup actions and added per-slot cleanup action.
>>>- Fixed coding style issues.
>>>- Added co-developed-by tag.
>>>
>>>Changes in v3:
>>>- Explicit assignment of enum values.
>>>- Use pcim_iomap_region() instead of pcim_iomap_regions().
>>>
>>> MAINTAINERS                    |   6 +
>>> drivers/pci/hotplug/Kconfig    |  10 +
>>> drivers/pci/hotplug/Makefile   |   1 +
>>> drivers/pci/hotplug/octep_hp.c | 409
>>>+++++++++++++++++++++++++++++++++
>>> 4 files changed, 426 insertions(+)
>>> create mode 100644 drivers/pci/hotplug/octep_hp.c
>>>
>>>diff --git a/MAINTAINERS b/MAINTAINERS
>>>index 42decde38320..7b5a618eed1c 100644
>>>--- a/MAINTAINERS
>>>+++ b/MAINTAINERS
>>>@@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
>>> R:	vattunuru@marvell.com
>>> F:	drivers/vdpa/octeon_ep/
>>>
>>>+MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
>>>+R:	Shijith Thotton <sthotton@marvell.com>
>>>+R:	Vamsi Attunuru <vattunuru@marvell.com>
>>>+S:	Supported
>>>+F:	drivers/pci/hotplug/octep_hp.c
>>>+
>>> MATROX FRAMEBUFFER DRIVER
>>> L:	linux-fbdev@vger.kernel.org
>>> S:	Orphan
>>>diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
>>>index 1472aef0fb81..2e38fd25f7ef 100644
>>>--- a/drivers/pci/hotplug/Kconfig
>>>+++ b/drivers/pci/hotplug/Kconfig
>>>@@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
>>>
>>> 	  When in doubt, say Y.
>>>
>>>+config HOTPLUG_PCI_OCTEONEP
>>>+	bool "OCTEON PCI device Hotplug controller driver"
>>>+	depends on HOTPLUG_PCI
>>>+	help
>>>+	  Say Y here if you have an OCTEON PCIe device with a hotplug
>>>+	  controller. This driver enables the non-controller functions of the
>>>+	  device to be registered as hotplug slots.
>>>+
>>>+	  When in doubt, say N.
>>>+
>>> endif # HOTPLUG_PCI
>>>diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
>>>index 240c99517d5e..40aaf31fe338 100644
>>>--- a/drivers/pci/hotplug/Makefile
>>>+++ b/drivers/pci/hotplug/Makefile
>>>@@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+=
>>>rpaphp.o
>>> obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
>>> obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
>>> obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
>>>+obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
>>>
>>> # acpiphp_ibm extends acpiphp, so should be linked afterwards.
>>>
>>>diff --git a/drivers/pci/hotplug/octep_hp.c
>b/drivers/pci/hotplug/octep_hp.c
>>>new file mode 100644
>>>index 000000000000..77dc2740f43e
>>>--- /dev/null
>>>+++ b/drivers/pci/hotplug/octep_hp.c
>>>@@ -0,0 +1,409 @@
>>>+// SPDX-License-Identifier: GPL-2.0-only
>>>+/* Copyright (C) 2024 Marvell. */
>>>+
>>>+#include <linux/cleanup.h>
>>>+#include <linux/container_of.h>
>>>+#include <linux/delay.h>
>>>+#include <linux/dev_printk.h>
>>>+#include <linux/init.h>
>>>+#include <linux/interrupt.h>
>>>+#include <linux/io-64-nonatomic-lo-hi.h>
>>>+#include <linux/kernel.h>
>>>+#include <linux/list.h>
>>>+#include <linux/module.h>
>>>+#include <linux/mutex.h>
>>>+#include <linux/pci.h>
>>>+#include <linux/pci_hotplug.h>
>>>+#include <linux/slab.h>
>>>+#include <linux/spinlock.h>
>>>+#include <linux/workqueue.h>
>>>+
>>>+#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
>>>+#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
>>>+#define OCTEP_HP_DRV_NAME "octep_hp"
>>>+
>>>+/*
>>>+ * Type of MSI-X interrupts.
>>>+ * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are
>>>used to
>>>+ * generate the vector and offset for an interrupt type.
>>>+ */
>>>+enum octep_hp_intr_type {
>>>+	OCTEP_HP_INTR_INVALID = -1,
>>>+	OCTEP_HP_INTR_ENA = 0,
>>>+	OCTEP_HP_INTR_DIS = 1,
>>>+	OCTEP_HP_INTR_MAX = 2,
>>>+};
>>>+
>>>+struct octep_hp_cmd {
>>>+	struct list_head list;
>>>+	enum octep_hp_intr_type intr_type;
>>>+	u64 intr_val;
>>>+};
>>>+
>>>+struct octep_hp_slot {
>>>+	struct list_head list;
>>>+	struct hotplug_slot slot;
>>>+	u16 slot_number;
>>>+	struct pci_dev *hp_pdev;
>>>+	unsigned int hp_devfn;
>>>+	struct octep_hp_controller *ctrl;
>>>+};
>>>+
>>>+struct octep_hp_intr_info {
>>>+	enum octep_hp_intr_type type;
>>>+	int number;
>>>+	char name[16];
>>>+};
>>>+
>>>+struct octep_hp_controller {
>>>+	void __iomem *base;
>>>+	struct pci_dev *pdev;
>>>+	struct octep_hp_intr_info intr[OCTEP_HP_INTR_MAX];
>>>+	struct work_struct work;
>>>+	struct list_head slot_list;
>>>+	struct mutex slot_lock; /* Protects slot_list */
>>>+	struct list_head hp_cmd_list;
>>>+	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
>>>+};
>>>+
>>>+static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl,
>>>+				 struct octep_hp_slot *hp_slot)
>>>+{
>>>+	guard(mutex)(&hp_ctrl->slot_lock);
>>>+	if (hp_slot->hp_pdev) {
>>>+		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already
>>>enabled\n",
>>>+			hp_slot->slot_number);
>>>+		return;
>>>+	}
>>>+
>>>+	/* Scan the device and add it to the bus */
>>>+	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus,
>>>+						  hp_slot->hp_devfn);
>>>+	pci_bus_assign_resources(hp_ctrl->pdev->bus);
>>>+	pci_bus_add_device(hp_slot->hp_pdev);
>>>+
>>>+	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n",
>>>+		hp_slot->slot_number);
>>>+}
>>>+
>>>+static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
>>>+				  struct octep_hp_slot *hp_slot)
>>>+{
>>>+	guard(mutex)(&hp_ctrl->slot_lock);
>>>+	if (!hp_slot->hp_pdev) {
>>>+		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n",
>>>+			hp_slot->slot_number);
>>>+		return;
>>>+	}
>>>+
>>>+	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n",
>>>+		hp_slot->slot_number);
>>>+
>>>+	/* Remove the device from the bus */
>>>+	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
>>>+	hp_slot->hp_pdev = NULL;
>>>+}
>>>+
>>>+static int octep_hp_enable_slot(struct hotplug_slot *slot)
>>>+{
>>>+	struct octep_hp_slot *hp_slot =
>>>+		container_of(slot, struct octep_hp_slot, slot);
>>>+
>>>+	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
>>>+	return 0;
>>>+}
>>>+
>>>+static int octep_hp_disable_slot(struct hotplug_slot *slot)
>>>+{
>>>+	struct octep_hp_slot *hp_slot =
>>>+		container_of(slot, struct octep_hp_slot, slot);
>>>+
>>>+	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
>>>+	return 0;
>>>+}
>>>+
>>>+static struct hotplug_slot_ops octep_hp_slot_ops = {
>>>+	.enable_slot = octep_hp_enable_slot,
>>>+	.disable_slot = octep_hp_disable_slot,
>>>+};
>>>+
>>>+#define SLOT_NAME_SIZE 16
>>>+static struct octep_hp_slot *
>>>+octep_hp_register_slot(struct octep_hp_controller *hp_ctrl,
>>>+		       struct pci_dev *pdev, u16 slot_number)
>>>+{
>>>+	char slot_name[SLOT_NAME_SIZE];
>>>+	struct octep_hp_slot *hp_slot;
>>>+	int ret;
>>>+
>>>+	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
>>>+	if (!hp_slot)
>>>+		return ERR_PTR(-ENOMEM);
>>>+
>>>+	hp_slot->ctrl = hp_ctrl;
>>>+	hp_slot->hp_pdev = pdev;
>>>+	hp_slot->hp_devfn = pdev->devfn;
>>>+	hp_slot->slot_number = slot_number;
>>>+	hp_slot->slot.ops = &octep_hp_slot_ops;
>>>+
>>>+	snprintf(slot_name, sizeof(slot_name), "octep_hp_%u", slot_number);
>>>+	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
>>>+			      PCI_SLOT(pdev->devfn), slot_name);
>>>+	if (ret) {
>>>+		kfree(hp_slot);
>>>+		return ERR_PTR(ret);
>>>+	}
>>>+
>>>+	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
>>>+	octep_hp_disable_pdev(hp_ctrl, hp_slot);
>>>+
>>>+	return hp_slot;
>>>+}
>>>+
>>>+static void octep_hp_deregister_slot(void *data)
>>>+{
>>>+	struct octep_hp_slot *hp_slot = data;
>>>+	struct octep_hp_controller *hp_ctrl = hp_slot->ctrl;
>>>+
>>>+	pci_hp_deregister(&hp_slot->slot);
>>>+	octep_hp_enable_pdev(hp_ctrl, hp_slot);
>>>+	list_del(&hp_slot->list);
>>>+	kfree(hp_slot);
>>>+}
>>>+
>>>+static bool octep_hp_is_slot(struct octep_hp_controller *hp_ctrl,
>>>+			     struct pci_dev *pdev)
>>>+{
>>>+	/* Check if the PCI device can be hotplugged */
>>>+	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
>>>+	       PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
>>>+}
>>>+
>>>+static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
>>>+				 struct octep_hp_cmd *hp_cmd)
>>>+{
>>>+	struct octep_hp_slot *hp_slot;
>>>+
>>>+	/*
>>>+	 * Enable or disable the slots based on the slot mask.
>>>+	 * intr_val is a bit mask where each bit represents a slot.
>>>+	 */
>>>+	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
>>>+		if (!(hp_cmd->intr_val & BIT(hp_slot->slot_number)))
>>>+			continue;
>>>+
>>>+		if (hp_cmd->intr_type == OCTEP_HP_INTR_ENA)
>>>+			octep_hp_enable_pdev(hp_ctrl, hp_slot);
>>>+		else
>>>+			octep_hp_disable_pdev(hp_ctrl, hp_slot);
>>>+	}
>>>+}
>>>+
>>>+static void octep_hp_work_handler(struct work_struct *work)
>>>+{
>>>+	struct octep_hp_controller *hp_ctrl;
>>>+	struct octep_hp_cmd *hp_cmd;
>>>+	unsigned long flags;
>>>+
>>>+	hp_ctrl = container_of(work, struct octep_hp_controller, work);
>>>+
>>>+	/* Process all the hotplug commands */
>>>+	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>>>+	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
>>>+		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list,
>>>+					  struct octep_hp_cmd, list);
>>>+		list_del(&hp_cmd->list);
>>>+		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>>>+
>>>+		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
>>>+		kfree(hp_cmd);
>>>+
>>>+		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>>>+	}
>>>+	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>>>+}
>>>+
>>>+static enum octep_hp_intr_type octep_hp_intr_type(struct
>>>octep_hp_intr_info *intr,
>>>+						  int irq)
>>>+{
>>>+	enum octep_hp_intr_type type;
>>>+
>>>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>>>type++) {
>>>+		if (intr[type].number == irq)
>>>+			return type;
>>>+	}
>>>+
>>>+	return OCTEP_HP_INTR_INVALID;
>>>+}
>>>+
>>>+static irqreturn_t octep_hp_intr_handler(int irq, void *data)
>>>+{
>>>+	struct octep_hp_controller *hp_ctrl = data;
>>>+	struct pci_dev *pdev = hp_ctrl->pdev;
>>>+	enum octep_hp_intr_type type;
>>>+	struct octep_hp_cmd *hp_cmd;
>>>+	u64 intr_val;
>>>+
>>>+	type = octep_hp_intr_type(hp_ctrl->intr, irq);
>>>+	if (type == OCTEP_HP_INTR_INVALID) {
>>>+		dev_err(&pdev->dev, "Invalid interrupt %d\n", irq);
>>>+		return IRQ_HANDLED;
>>>+	}
>>>+
>>>+	/* Read and clear the interrupt */
>>>+	intr_val = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>>>+	writeq(intr_val, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>>>+
>>>+	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
>>>+	if (!hp_cmd)
>>>+		return IRQ_HANDLED;
>>>+
>>>+	hp_cmd->intr_val = intr_val;
>>>+	hp_cmd->intr_type = type;
>>>+
>>>+	/* Add the command to the list and schedule the work */
>>>+	spin_lock(&hp_ctrl->hp_cmd_lock);
>>>+	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
>>>+	spin_unlock(&hp_ctrl->hp_cmd_lock);
>>>+	schedule_work(&hp_ctrl->work);
>>>+
>>>+	return IRQ_HANDLED;
>>>+}
>>>+
>>>+static void octep_hp_irq_cleanup(void *data)
>>>+{
>>>+	struct octep_hp_controller *hp_ctrl = data;
>>>+
>>>+	pci_free_irq_vectors(hp_ctrl->pdev);
>>>+	flush_work(&hp_ctrl->work);
>>>+}
>>>+
>>>+static int octep_hp_request_irq(struct octep_hp_controller *hp_ctrl,
>>>+				enum octep_hp_intr_type type)
>>>+{
>>>+	struct pci_dev *pdev = hp_ctrl->pdev;
>>>+	struct octep_hp_intr_info *intr;
>>>+	int irq;
>>>+
>>>+	irq = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(type));
>>>+	if (irq < 0)
>>>+		return irq;
>>>+
>>>+	intr = &hp_ctrl->intr[type];
>>>+	intr->number = irq;
>>>+	intr->type = type;
>>>+	snprintf(intr->name, sizeof(intr->name), "octep_hp_%d", type);
>>>+
>>>+	return devm_request_irq(&pdev->dev, irq, octep_hp_intr_handler,
>>>+				IRQF_SHARED, intr->name, hp_ctrl);
>>>+}
>>>+
>>>+static int octep_hp_controller_setup(struct pci_dev *pdev,
>>>+				     struct octep_hp_controller *hp_ctrl)
>>>+{
>>>+	struct device *dev = &pdev->dev;
>>>+	enum octep_hp_intr_type type;
>>>+	int ret;
>>>+
>>>+	ret = pcim_enable_device(pdev);
>>>+	if (ret)
>>>+		return dev_err_probe(dev, ret, "Failed to enable PCI device\n");
>>>+
>>>+	hp_ctrl->base = pcim_iomap_region(pdev, 0, OCTEP_HP_DRV_NAME);
>>>+	if (IS_ERR(hp_ctrl->base))
>>>+		return dev_err_probe(dev, PTR_ERR(hp_ctrl->base),
>>>+				     "Failed to map PCI device region\n");
>>>+
>>>+	pci_set_master(pdev);
>>>+	pci_set_drvdata(pdev, hp_ctrl);
>>>+
>>>+	INIT_LIST_HEAD(&hp_ctrl->slot_list);
>>>+	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
>>>+	mutex_init(&hp_ctrl->slot_lock);
>>>+	spin_lock_init(&hp_ctrl->hp_cmd_lock);
>>>+	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
>>>+	hp_ctrl->pdev = pdev;
>>>+
>>>+	ret = pci_alloc_irq_vectors(pdev, 1,
>>>+
>>>OCTEP_HP_INTR_VECTOR(OCTEP_HP_INTR_MAX),
>>>+				    PCI_IRQ_MSIX);
>>>+	if (ret < 0)
>>>+		return dev_err_probe(dev, ret, "Failed to alloc MSI-X
>>>vectors\n");
>>>+
>>>+	ret = devm_add_action(&pdev->dev, octep_hp_irq_cleanup, hp_ctrl);
>>>+	if (ret)
>>>+		return dev_err_probe(&pdev->dev, ret, "Failed to add IRQ
>>>cleanup action\n");
>>>+
>>>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>>>type++) {
>>>+		ret = octep_hp_request_irq(hp_ctrl, type);
>>>+		if (ret)
>>>+			return dev_err_probe(dev, ret,
>>>+					     "Failed to request IRQ for vector
>>>%d\n",
>>>+					     OCTEP_HP_INTR_VECTOR(type));
>>>+	}
>>>+
>>>+	return 0;
>>>+}
>>>+
>>>+static int octep_hp_pci_probe(struct pci_dev *pdev,
>>>+			      const struct pci_device_id *id)
>>>+{
>>>+	struct octep_hp_controller *hp_ctrl;
>>>+	struct pci_dev *tmp_pdev = NULL;
>>>+	struct octep_hp_slot *hp_slot;
>>>+	u16 slot_number = 0;
>>>+	int ret;
>>>+
>>>+	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
>>>+	if (!hp_ctrl)
>>>+		return -ENOMEM;
>>>+
>>>+	ret = octep_hp_controller_setup(pdev, hp_ctrl);
>>>+	if (ret)
>>>+		return ret;
>>>+
>>>+	/*
>>>+	 * Register all hotplug slots. Hotplug controller is the first function
>>>+	 * of the PCI device. The hotplug slots are the remaining functions of
>>>+	 * the PCI device. They are removed from the bus and are added back
>>>when
>>>+	 * the hotplug event is triggered.
>>>+	 */
>>>+	for_each_pci_dev(tmp_pdev) {
>>>+		if (!octep_hp_is_slot(hp_ctrl, tmp_pdev))
>>>+			continue;
>>>+
>>>+		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev,
>>>slot_number);
>>>+		if (IS_ERR(hp_slot))
>>>+			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
>>>+					     "Failed to register hotplug slot
>>>%u\n",
>>>+					     slot_number);
>>>+
>>>+		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
>>>+				      hp_slot);
>>>+		if (ret)
>>>+			return dev_err_probe(&pdev->dev, ret,
>>>+					     "Failed to add action for
>>>deregistering slot %u\n",
>>>+					     slot_number);
>>>+		slot_number++;
>>>+	}
>>>+
>>>+	return 0;
>>>+}
>>>+
>>>+#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
>>>+static struct pci_device_id octep_hp_pci_map[] = {
>>>+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
>>>OCTEP_DEVID_HP_CONTROLLER) },
>>>+	{ },
>>>+};
>>>+
>>>+static struct pci_driver octep_hp = {
>>>+	.name = OCTEP_HP_DRV_NAME,
>>>+	.id_table = octep_hp_pci_map,
>>>+	.probe = octep_hp_pci_probe,
>>>+};
>>>+
>>>+module_pci_driver(octep_hp);
>>>+
>>>+MODULE_LICENSE("GPL");
>>>+MODULE_AUTHOR("Marvell");
>>>+MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
>>>--
>>>2.25.1
>>

RE: [PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

Ping. Please check this patch.

>>>>This patch introduces a PCI hotplug controller driver for the OCTEON
>>>>PCIe device, a multi-function PCIe device where the first function acts
>>>>as a hotplug controller. It is equipped with MSI-x interrupts to notify
>>>>the host of hotplug events from the OCTEON firmware.
>>>>
>>>>The driver facilitates the hotplugging of non-controller functions
>>>>within the same device. During probe, non-controller functions are
>>>>removed and registered as PCI hotplug slots. The slots are added back
>>>>only upon request from the device firmware. The driver also allows the
>>>>enabling and disabling of the slots via sysfs slot entries, provided by
>>>>the PCI hotplug framework.
>>>>
>>>>Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>>>>Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
>>>>Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>>>>---
>>>>
>>>>This patch introduces a PCI hotplug controller driver for OCTEON PCIe
>>hotplug
>>>>controller. The OCTEON PCIe device is a multi-function device where the
>first
>>>>function acts as a PCI hotplug controller.
>>>>
>>>>              +--------------------------------+
>>>>              |           Root Port            |
>>>>              +--------------------------------+
>>>>                              |
>>>>                             PCIe
>>>>                              |
>>>>+---------------------------------------------------------------+
>>>>|              OCTEON PCIe Multifunction Device                 |
>>>>+---------------------------------------------------------------+
>>>>            |                    |              |            |
>>>>            |                    |              |            |
>>>>+---------------------+  +----------------+  +-----+  +----------------+
>>>>|      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>>>>| (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>>>>+---------------------+  +----------------+  +-----+  +----------------+
>>>>            |
>>>>            |
>>>>+-------------------------+
>>>>|   Controller Firmware   |
>>>>+-------------------------+
>>>>
>>>>The hotplug controller driver facilitates the hotplugging of non-controller
>>>>functions in the same device. During the probe of the driver, the non-
>>>controller
>>>>function are removed and registered as PCI hotplug slots. They are added
>>back
>>>>only upon request from the device firmware. The driver also allows the user
>>to
>>>>enable/disable the functions using sysfs slot entries provided by PCI hotplug
>>>>framework.
>>>>
>>>>This solution adopts a hardware + software approach for several reasons:
>>>>
>>>>1. To reduce hardware implementation cost. Supporting complete hotplug
>>>>   capability within the card would require a PCI switch implemented within.
>>>>
>>>>2. In the multi-function device, non-controller functions can act as
>emulated
>>>>   devices. The firmware can dynamically enable or disable them at runtime.
>>>>
>>>>3. Not all root ports support PCI hotplug. This approach provides greater
>>>>   flexibility and compatibility across different hardware configurations.
>>>>
>>>>The hotplug controller function is lightweight and is equipped with MSI-x
>>>>interrupts to notify the host about hotplug events. Upon receiving an
>>>>interrupt, the hotplug register is read, and the required function is enabled
>>>>or disabled.
>>>>
>>>>This driver will be beneficial for managing PCI hotplug events on the
>OCTEON
>>>>PCIe device without requiring complex hardware solutions.
>>>>
>>>>Changes in v2:
>>>>- Added missing include files.
>>>>- Used dev_err_probe() for error handling.
>>>>- Used guard() for mutex locking.
>>>>- Splited cleanup actions and added per-slot cleanup action.
>>>>- Fixed coding style issues.
>>>>- Added co-developed-by tag.
>>>>
>>>>Changes in v3:
>>>>- Explicit assignment of enum values.
>>>>- Use pcim_iomap_region() instead of pcim_iomap_regions().
>>>>
>>>> MAINTAINERS                    |   6 +
>>>> drivers/pci/hotplug/Kconfig    |  10 +
>>>> drivers/pci/hotplug/Makefile   |   1 +
>>>> drivers/pci/hotplug/octep_hp.c | 409
>>>>+++++++++++++++++++++++++++++++++
>>>> 4 files changed, 426 insertions(+)
>>>> create mode 100644 drivers/pci/hotplug/octep_hp.c
>>>>
>>>>diff --git a/MAINTAINERS b/MAINTAINERS
>>>>index 42decde38320..7b5a618eed1c 100644
>>>>--- a/MAINTAINERS
>>>>+++ b/MAINTAINERS
>>>>@@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
>>>> R:	vattunuru@marvell.com
>>>> F:	drivers/vdpa/octeon_ep/
>>>>
>>>>+MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
>>>>+R:	Shijith Thotton <sthotton@marvell.com>
>>>>+R:	Vamsi Attunuru <vattunuru@marvell.com>
>>>>+S:	Supported
>>>>+F:	drivers/pci/hotplug/octep_hp.c
>>>>+
>>>> MATROX FRAMEBUFFER DRIVER
>>>> L:	linux-fbdev@vger.kernel.org
>>>> S:	Orphan
>>>>diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
>>>>index 1472aef0fb81..2e38fd25f7ef 100644
>>>>--- a/drivers/pci/hotplug/Kconfig
>>>>+++ b/drivers/pci/hotplug/Kconfig
>>>>@@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
>>>>
>>>> 	  When in doubt, say Y.
>>>>
>>>>+config HOTPLUG_PCI_OCTEONEP
>>>>+	bool "OCTEON PCI device Hotplug controller driver"
>>>>+	depends on HOTPLUG_PCI
>>>>+	help
>>>>+	  Say Y here if you have an OCTEON PCIe device with a hotplug
>>>>+	  controller. This driver enables the non-controller functions of the
>>>>+	  device to be registered as hotplug slots.
>>>>+
>>>>+	  When in doubt, say N.
>>>>+
>>>> endif # HOTPLUG_PCI
>>>>diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
>>>>index 240c99517d5e..40aaf31fe338 100644
>>>>--- a/drivers/pci/hotplug/Makefile
>>>>+++ b/drivers/pci/hotplug/Makefile
>>>>@@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+=
>>>>rpaphp.o
>>>> obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
>>>> obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
>>>> obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
>>>>+obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
>>>>
>>>> # acpiphp_ibm extends acpiphp, so should be linked afterwards.
>>>>
>>>>diff --git a/drivers/pci/hotplug/octep_hp.c
>>b/drivers/pci/hotplug/octep_hp.c
>>>>new file mode 100644
>>>>index 000000000000..77dc2740f43e
>>>>--- /dev/null
>>>>+++ b/drivers/pci/hotplug/octep_hp.c
>>>>@@ -0,0 +1,409 @@
>>>>+// SPDX-License-Identifier: GPL-2.0-only
>>>>+/* Copyright (C) 2024 Marvell. */
>>>>+
>>>>+#include <linux/cleanup.h>
>>>>+#include <linux/container_of.h>
>>>>+#include <linux/delay.h>
>>>>+#include <linux/dev_printk.h>
>>>>+#include <linux/init.h>
>>>>+#include <linux/interrupt.h>
>>>>+#include <linux/io-64-nonatomic-lo-hi.h>
>>>>+#include <linux/kernel.h>
>>>>+#include <linux/list.h>
>>>>+#include <linux/module.h>
>>>>+#include <linux/mutex.h>
>>>>+#include <linux/pci.h>
>>>>+#include <linux/pci_hotplug.h>
>>>>+#include <linux/slab.h>
>>>>+#include <linux/spinlock.h>
>>>>+#include <linux/workqueue.h>
>>>>+
>>>>+#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
>>>>+#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
>>>>+#define OCTEP_HP_DRV_NAME "octep_hp"
>>>>+
>>>>+/*
>>>>+ * Type of MSI-X interrupts.
>>>>+ * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are
>>>>used to
>>>>+ * generate the vector and offset for an interrupt type.
>>>>+ */
>>>>+enum octep_hp_intr_type {
>>>>+	OCTEP_HP_INTR_INVALID = -1,
>>>>+	OCTEP_HP_INTR_ENA = 0,
>>>>+	OCTEP_HP_INTR_DIS = 1,
>>>>+	OCTEP_HP_INTR_MAX = 2,
>>>>+};
>>>>+
>>>>+struct octep_hp_cmd {
>>>>+	struct list_head list;
>>>>+	enum octep_hp_intr_type intr_type;
>>>>+	u64 intr_val;
>>>>+};
>>>>+
>>>>+struct octep_hp_slot {
>>>>+	struct list_head list;
>>>>+	struct hotplug_slot slot;
>>>>+	u16 slot_number;
>>>>+	struct pci_dev *hp_pdev;
>>>>+	unsigned int hp_devfn;
>>>>+	struct octep_hp_controller *ctrl;
>>>>+};
>>>>+
>>>>+struct octep_hp_intr_info {
>>>>+	enum octep_hp_intr_type type;
>>>>+	int number;
>>>>+	char name[16];
>>>>+};
>>>>+
>>>>+struct octep_hp_controller {
>>>>+	void __iomem *base;
>>>>+	struct pci_dev *pdev;
>>>>+	struct octep_hp_intr_info intr[OCTEP_HP_INTR_MAX];
>>>>+	struct work_struct work;
>>>>+	struct list_head slot_list;
>>>>+	struct mutex slot_lock; /* Protects slot_list */
>>>>+	struct list_head hp_cmd_list;
>>>>+	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
>>>>+};
>>>>+
>>>>+static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl,
>>>>+				 struct octep_hp_slot *hp_slot)
>>>>+{
>>>>+	guard(mutex)(&hp_ctrl->slot_lock);
>>>>+	if (hp_slot->hp_pdev) {
>>>>+		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already
>>>>enabled\n",
>>>>+			hp_slot->slot_number);
>>>>+		return;
>>>>+	}
>>>>+
>>>>+	/* Scan the device and add it to the bus */
>>>>+	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus,
>>>>+						  hp_slot->hp_devfn);
>>>>+	pci_bus_assign_resources(hp_ctrl->pdev->bus);
>>>>+	pci_bus_add_device(hp_slot->hp_pdev);
>>>>+
>>>>+	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n",
>>>>+		hp_slot->slot_number);
>>>>+}
>>>>+
>>>>+static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
>>>>+				  struct octep_hp_slot *hp_slot)
>>>>+{
>>>>+	guard(mutex)(&hp_ctrl->slot_lock);
>>>>+	if (!hp_slot->hp_pdev) {
>>>>+		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n",
>>>>+			hp_slot->slot_number);
>>>>+		return;
>>>>+	}
>>>>+
>>>>+	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n",
>>>>+		hp_slot->slot_number);
>>>>+
>>>>+	/* Remove the device from the bus */
>>>>+	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
>>>>+	hp_slot->hp_pdev = NULL;
>>>>+}
>>>>+
>>>>+static int octep_hp_enable_slot(struct hotplug_slot *slot)
>>>>+{
>>>>+	struct octep_hp_slot *hp_slot =
>>>>+		container_of(slot, struct octep_hp_slot, slot);
>>>>+
>>>>+	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
>>>>+	return 0;
>>>>+}
>>>>+
>>>>+static int octep_hp_disable_slot(struct hotplug_slot *slot)
>>>>+{
>>>>+	struct octep_hp_slot *hp_slot =
>>>>+		container_of(slot, struct octep_hp_slot, slot);
>>>>+
>>>>+	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
>>>>+	return 0;
>>>>+}
>>>>+
>>>>+static struct hotplug_slot_ops octep_hp_slot_ops = {
>>>>+	.enable_slot = octep_hp_enable_slot,
>>>>+	.disable_slot = octep_hp_disable_slot,
>>>>+};
>>>>+
>>>>+#define SLOT_NAME_SIZE 16
>>>>+static struct octep_hp_slot *
>>>>+octep_hp_register_slot(struct octep_hp_controller *hp_ctrl,
>>>>+		       struct pci_dev *pdev, u16 slot_number)
>>>>+{
>>>>+	char slot_name[SLOT_NAME_SIZE];
>>>>+	struct octep_hp_slot *hp_slot;
>>>>+	int ret;
>>>>+
>>>>+	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
>>>>+	if (!hp_slot)
>>>>+		return ERR_PTR(-ENOMEM);
>>>>+
>>>>+	hp_slot->ctrl = hp_ctrl;
>>>>+	hp_slot->hp_pdev = pdev;
>>>>+	hp_slot->hp_devfn = pdev->devfn;
>>>>+	hp_slot->slot_number = slot_number;
>>>>+	hp_slot->slot.ops = &octep_hp_slot_ops;
>>>>+
>>>>+	snprintf(slot_name, sizeof(slot_name), "octep_hp_%u", slot_number);
>>>>+	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
>>>>+			      PCI_SLOT(pdev->devfn), slot_name);
>>>>+	if (ret) {
>>>>+		kfree(hp_slot);
>>>>+		return ERR_PTR(ret);
>>>>+	}
>>>>+
>>>>+	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
>>>>+	octep_hp_disable_pdev(hp_ctrl, hp_slot);
>>>>+
>>>>+	return hp_slot;
>>>>+}
>>>>+
>>>>+static void octep_hp_deregister_slot(void *data)
>>>>+{
>>>>+	struct octep_hp_slot *hp_slot = data;
>>>>+	struct octep_hp_controller *hp_ctrl = hp_slot->ctrl;
>>>>+
>>>>+	pci_hp_deregister(&hp_slot->slot);
>>>>+	octep_hp_enable_pdev(hp_ctrl, hp_slot);
>>>>+	list_del(&hp_slot->list);
>>>>+	kfree(hp_slot);
>>>>+}
>>>>+
>>>>+static bool octep_hp_is_slot(struct octep_hp_controller *hp_ctrl,
>>>>+			     struct pci_dev *pdev)
>>>>+{
>>>>+	/* Check if the PCI device can be hotplugged */
>>>>+	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
>>>>+	       PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
>>>>+}
>>>>+
>>>>+static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
>>>>+				 struct octep_hp_cmd *hp_cmd)
>>>>+{
>>>>+	struct octep_hp_slot *hp_slot;
>>>>+
>>>>+	/*
>>>>+	 * Enable or disable the slots based on the slot mask.
>>>>+	 * intr_val is a bit mask where each bit represents a slot.
>>>>+	 */
>>>>+	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
>>>>+		if (!(hp_cmd->intr_val & BIT(hp_slot->slot_number)))
>>>>+			continue;
>>>>+
>>>>+		if (hp_cmd->intr_type == OCTEP_HP_INTR_ENA)
>>>>+			octep_hp_enable_pdev(hp_ctrl, hp_slot);
>>>>+		else
>>>>+			octep_hp_disable_pdev(hp_ctrl, hp_slot);
>>>>+	}
>>>>+}
>>>>+
>>>>+static void octep_hp_work_handler(struct work_struct *work)
>>>>+{
>>>>+	struct octep_hp_controller *hp_ctrl;
>>>>+	struct octep_hp_cmd *hp_cmd;
>>>>+	unsigned long flags;
>>>>+
>>>>+	hp_ctrl = container_of(work, struct octep_hp_controller, work);
>>>>+
>>>>+	/* Process all the hotplug commands */
>>>>+	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>>>>+	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
>>>>+		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list,
>>>>+					  struct octep_hp_cmd, list);
>>>>+		list_del(&hp_cmd->list);
>>>>+		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>>>>+
>>>>+		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
>>>>+		kfree(hp_cmd);
>>>>+
>>>>+		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>>>>+	}
>>>>+	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>>>>+}
>>>>+
>>>>+static enum octep_hp_intr_type octep_hp_intr_type(struct
>>>>octep_hp_intr_info *intr,
>>>>+						  int irq)
>>>>+{
>>>>+	enum octep_hp_intr_type type;
>>>>+
>>>>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>>>>type++) {
>>>>+		if (intr[type].number == irq)
>>>>+			return type;
>>>>+	}
>>>>+
>>>>+	return OCTEP_HP_INTR_INVALID;
>>>>+}
>>>>+
>>>>+static irqreturn_t octep_hp_intr_handler(int irq, void *data)
>>>>+{
>>>>+	struct octep_hp_controller *hp_ctrl = data;
>>>>+	struct pci_dev *pdev = hp_ctrl->pdev;
>>>>+	enum octep_hp_intr_type type;
>>>>+	struct octep_hp_cmd *hp_cmd;
>>>>+	u64 intr_val;
>>>>+
>>>>+	type = octep_hp_intr_type(hp_ctrl->intr, irq);
>>>>+	if (type == OCTEP_HP_INTR_INVALID) {
>>>>+		dev_err(&pdev->dev, "Invalid interrupt %d\n", irq);
>>>>+		return IRQ_HANDLED;
>>>>+	}
>>>>+
>>>>+	/* Read and clear the interrupt */
>>>>+	intr_val = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>>>>+	writeq(intr_val, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>>>>+
>>>>+	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
>>>>+	if (!hp_cmd)
>>>>+		return IRQ_HANDLED;
>>>>+
>>>>+	hp_cmd->intr_val = intr_val;
>>>>+	hp_cmd->intr_type = type;
>>>>+
>>>>+	/* Add the command to the list and schedule the work */
>>>>+	spin_lock(&hp_ctrl->hp_cmd_lock);
>>>>+	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
>>>>+	spin_unlock(&hp_ctrl->hp_cmd_lock);
>>>>+	schedule_work(&hp_ctrl->work);
>>>>+
>>>>+	return IRQ_HANDLED;
>>>>+}
>>>>+
>>>>+static void octep_hp_irq_cleanup(void *data)
>>>>+{
>>>>+	struct octep_hp_controller *hp_ctrl = data;
>>>>+
>>>>+	pci_free_irq_vectors(hp_ctrl->pdev);
>>>>+	flush_work(&hp_ctrl->work);
>>>>+}
>>>>+
>>>>+static int octep_hp_request_irq(struct octep_hp_controller *hp_ctrl,
>>>>+				enum octep_hp_intr_type type)
>>>>+{
>>>>+	struct pci_dev *pdev = hp_ctrl->pdev;
>>>>+	struct octep_hp_intr_info *intr;
>>>>+	int irq;
>>>>+
>>>>+	irq = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(type));
>>>>+	if (irq < 0)
>>>>+		return irq;
>>>>+
>>>>+	intr = &hp_ctrl->intr[type];
>>>>+	intr->number = irq;
>>>>+	intr->type = type;
>>>>+	snprintf(intr->name, sizeof(intr->name), "octep_hp_%d", type);
>>>>+
>>>>+	return devm_request_irq(&pdev->dev, irq, octep_hp_intr_handler,
>>>>+				IRQF_SHARED, intr->name, hp_ctrl);
>>>>+}
>>>>+
>>>>+static int octep_hp_controller_setup(struct pci_dev *pdev,
>>>>+				     struct octep_hp_controller *hp_ctrl)
>>>>+{
>>>>+	struct device *dev = &pdev->dev;
>>>>+	enum octep_hp_intr_type type;
>>>>+	int ret;
>>>>+
>>>>+	ret = pcim_enable_device(pdev);
>>>>+	if (ret)
>>>>+		return dev_err_probe(dev, ret, "Failed to enable PCI device\n");
>>>>+
>>>>+	hp_ctrl->base = pcim_iomap_region(pdev, 0, OCTEP_HP_DRV_NAME);
>>>>+	if (IS_ERR(hp_ctrl->base))
>>>>+		return dev_err_probe(dev, PTR_ERR(hp_ctrl->base),
>>>>+				     "Failed to map PCI device region\n");
>>>>+
>>>>+	pci_set_master(pdev);
>>>>+	pci_set_drvdata(pdev, hp_ctrl);
>>>>+
>>>>+	INIT_LIST_HEAD(&hp_ctrl->slot_list);
>>>>+	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
>>>>+	mutex_init(&hp_ctrl->slot_lock);
>>>>+	spin_lock_init(&hp_ctrl->hp_cmd_lock);
>>>>+	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
>>>>+	hp_ctrl->pdev = pdev;
>>>>+
>>>>+	ret = pci_alloc_irq_vectors(pdev, 1,
>>>>+
>>>>OCTEP_HP_INTR_VECTOR(OCTEP_HP_INTR_MAX),
>>>>+				    PCI_IRQ_MSIX);
>>>>+	if (ret < 0)
>>>>+		return dev_err_probe(dev, ret, "Failed to alloc MSI-X
>>>>vectors\n");
>>>>+
>>>>+	ret = devm_add_action(&pdev->dev, octep_hp_irq_cleanup, hp_ctrl);
>>>>+	if (ret)
>>>>+		return dev_err_probe(&pdev->dev, ret, "Failed to add IRQ
>>>>cleanup action\n");
>>>>+
>>>>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>>>>type++) {
>>>>+		ret = octep_hp_request_irq(hp_ctrl, type);
>>>>+		if (ret)
>>>>+			return dev_err_probe(dev, ret,
>>>>+					     "Failed to request IRQ for vector
>>>>%d\n",
>>>>+					     OCTEP_HP_INTR_VECTOR(type));
>>>>+	}
>>>>+
>>>>+	return 0;
>>>>+}
>>>>+
>>>>+static int octep_hp_pci_probe(struct pci_dev *pdev,
>>>>+			      const struct pci_device_id *id)
>>>>+{
>>>>+	struct octep_hp_controller *hp_ctrl;
>>>>+	struct pci_dev *tmp_pdev = NULL;
>>>>+	struct octep_hp_slot *hp_slot;
>>>>+	u16 slot_number = 0;
>>>>+	int ret;
>>>>+
>>>>+	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
>>>>+	if (!hp_ctrl)
>>>>+		return -ENOMEM;
>>>>+
>>>>+	ret = octep_hp_controller_setup(pdev, hp_ctrl);
>>>>+	if (ret)
>>>>+		return ret;
>>>>+
>>>>+	/*
>>>>+	 * Register all hotplug slots. Hotplug controller is the first function
>>>>+	 * of the PCI device. The hotplug slots are the remaining functions of
>>>>+	 * the PCI device. They are removed from the bus and are added back
>>>>when
>>>>+	 * the hotplug event is triggered.
>>>>+	 */
>>>>+	for_each_pci_dev(tmp_pdev) {
>>>>+		if (!octep_hp_is_slot(hp_ctrl, tmp_pdev))
>>>>+			continue;
>>>>+
>>>>+		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev,
>>>>slot_number);
>>>>+		if (IS_ERR(hp_slot))
>>>>+			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
>>>>+					     "Failed to register hotplug slot
>>>>%u\n",
>>>>+					     slot_number);
>>>>+
>>>>+		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
>>>>+				      hp_slot);
>>>>+		if (ret)
>>>>+			return dev_err_probe(&pdev->dev, ret,
>>>>+					     "Failed to add action for
>>>>deregistering slot %u\n",
>>>>+					     slot_number);
>>>>+		slot_number++;
>>>>+	}
>>>>+
>>>>+	return 0;
>>>>+}
>>>>+
>>>>+#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
>>>>+static struct pci_device_id octep_hp_pci_map[] = {
>>>>+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
>>>>OCTEP_DEVID_HP_CONTROLLER) },
>>>>+	{ },
>>>>+};
>>>>+
>>>>+static struct pci_driver octep_hp = {
>>>>+	.name = OCTEP_HP_DRV_NAME,
>>>>+	.id_table = octep_hp_pci_map,
>>>>+	.probe = octep_hp_pci_probe,
>>>>+};
>>>>+
>>>>+module_pci_driver(octep_hp);
>>>>+
>>>>+MODULE_LICENSE("GPL");
>>>>+MODULE_AUTHOR("Marvell");
>>>>+MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
>>>>--
>>>>2.25.1
>>>

RE: [PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

Just a gentle reminder to review the patch. Let me know if any adjustments are
needed. Thanks!

>
>This patch introduces a PCI hotplug controller driver for the OCTEON
>PCIe device, a multi-function PCIe device where the first function acts
>as a hotplug controller. It is equipped with MSI-x interrupts to notify
>the host of hotplug events from the OCTEON firmware.
>
>The driver facilitates the hotplugging of non-controller functions
>within the same device. During probe, non-controller functions are
>removed and registered as PCI hotplug slots. The slots are added back
>only upon request from the device firmware. The driver also allows the
>enabling and disabling of the slots via sysfs slot entries, provided by
>the PCI hotplug framework.
>
>Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
>Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>---
>
>This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
>controller. The OCTEON PCIe device is a multi-function device where the first
>function acts as a PCI hotplug controller.
>
>              +--------------------------------+
>              |           Root Port            |
>              +--------------------------------+
>                              |
>                             PCIe
>                              |
>+---------------------------------------------------------------+
>|              OCTEON PCIe Multifunction Device                 |
>+---------------------------------------------------------------+
>            |                    |              |            |
>            |                    |              |            |
>+---------------------+  +----------------+  +-----+  +----------------+
>|      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>| (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>+---------------------+  +----------------+  +-----+  +----------------+
>            |
>            |
>+-------------------------+
>|   Controller Firmware   |
>+-------------------------+
>
>The hotplug controller driver facilitates the hotplugging of non-controller
>functions in the same device. During the probe of the driver, the non-controller
>function are removed and registered as PCI hotplug slots. They are added back
>only upon request from the device firmware. The driver also allows the user to
>enable/disable the functions using sysfs slot entries provided by PCI hotplug
>framework.
>
>This solution adopts a hardware + software approach for several reasons:
>
>1. To reduce hardware implementation cost. Supporting complete hotplug
>   capability within the card would require a PCI switch implemented within.
>
>2. In the multi-function device, non-controller functions can act as emulated
>   devices. The firmware can dynamically enable or disable them at runtime.
>
>3. Not all root ports support PCI hotplug. This approach provides greater
>   flexibility and compatibility across different hardware configurations.
>
>The hotplug controller function is lightweight and is equipped with MSI-x
>interrupts to notify the host about hotplug events. Upon receiving an
>interrupt, the hotplug register is read, and the required function is enabled
>or disabled.
>
>This driver will be beneficial for managing PCI hotplug events on the OCTEON
>PCIe device without requiring complex hardware solutions.
>
>Changes in v2:
>- Added missing include files.
>- Used dev_err_probe() for error handling.
>- Used guard() for mutex locking.
>- Splited cleanup actions and added per-slot cleanup action.
>- Fixed coding style issues.
>- Added co-developed-by tag.
>
>Changes in v3:
>- Explicit assignment of enum values.
>- Use pcim_iomap_region() instead of pcim_iomap_regions().
>
> MAINTAINERS                    |   6 +
> drivers/pci/hotplug/Kconfig    |  10 +
> drivers/pci/hotplug/Makefile   |   1 +
> drivers/pci/hotplug/octep_hp.c | 409
>+++++++++++++++++++++++++++++++++
> 4 files changed, 426 insertions(+)
> create mode 100644 drivers/pci/hotplug/octep_hp.c
>
>diff --git a/MAINTAINERS b/MAINTAINERS
>index 42decde38320..7b5a618eed1c 100644
>--- a/MAINTAINERS
>+++ b/MAINTAINERS
>@@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
> R:	vattunuru@marvell.com
> F:	drivers/vdpa/octeon_ep/
>
>+MARVELL OCTEON HOTPLUG CONTROLLER DRIVER
>+R:	Shijith Thotton <sthotton@marvell.com>
>+R:	Vamsi Attunuru <vattunuru@marvell.com>
>+S:	Supported
>+F:	drivers/pci/hotplug/octep_hp.c
>+
> MATROX FRAMEBUFFER DRIVER
> L:	linux-fbdev@vger.kernel.org
> S:	Orphan
>diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
>index 1472aef0fb81..2e38fd25f7ef 100644
>--- a/drivers/pci/hotplug/Kconfig
>+++ b/drivers/pci/hotplug/Kconfig
>@@ -173,4 +173,14 @@ config HOTPLUG_PCI_S390
>
> 	  When in doubt, say Y.
>
>+config HOTPLUG_PCI_OCTEONEP
>+	bool "OCTEON PCI device Hotplug controller driver"
>+	depends on HOTPLUG_PCI
>+	help
>+	  Say Y here if you have an OCTEON PCIe device with a hotplug
>+	  controller. This driver enables the non-controller functions of the
>+	  device to be registered as hotplug slots.
>+
>+	  When in doubt, say N.
>+
> endif # HOTPLUG_PCI
>diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
>index 240c99517d5e..40aaf31fe338 100644
>--- a/drivers/pci/hotplug/Makefile
>+++ b/drivers/pci/hotplug/Makefile
>@@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+=
>rpaphp.o
> obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
> obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
> obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
>+obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
>
> # acpiphp_ibm extends acpiphp, so should be linked afterwards.
>
>diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
>new file mode 100644
>index 000000000000..77dc2740f43e
>--- /dev/null
>+++ b/drivers/pci/hotplug/octep_hp.c
>@@ -0,0 +1,409 @@
>+// SPDX-License-Identifier: GPL-2.0-only
>+/* Copyright (C) 2024 Marvell. */
>+
>+#include <linux/cleanup.h>
>+#include <linux/container_of.h>
>+#include <linux/delay.h>
>+#include <linux/dev_printk.h>
>+#include <linux/init.h>
>+#include <linux/interrupt.h>
>+#include <linux/io-64-nonatomic-lo-hi.h>
>+#include <linux/kernel.h>
>+#include <linux/list.h>
>+#include <linux/module.h>
>+#include <linux/mutex.h>
>+#include <linux/pci.h>
>+#include <linux/pci_hotplug.h>
>+#include <linux/slab.h>
>+#include <linux/spinlock.h>
>+#include <linux/workqueue.h>
>+
>+#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
>+#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
>+#define OCTEP_HP_DRV_NAME "octep_hp"
>+
>+/*
>+ * Type of MSI-X interrupts.
>+ * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are
>used to
>+ * generate the vector and offset for an interrupt type.
>+ */
>+enum octep_hp_intr_type {
>+	OCTEP_HP_INTR_INVALID = -1,
>+	OCTEP_HP_INTR_ENA = 0,
>+	OCTEP_HP_INTR_DIS = 1,
>+	OCTEP_HP_INTR_MAX = 2,
>+};
>+
>+struct octep_hp_cmd {
>+	struct list_head list;
>+	enum octep_hp_intr_type intr_type;
>+	u64 intr_val;
>+};
>+
>+struct octep_hp_slot {
>+	struct list_head list;
>+	struct hotplug_slot slot;
>+	u16 slot_number;
>+	struct pci_dev *hp_pdev;
>+	unsigned int hp_devfn;
>+	struct octep_hp_controller *ctrl;
>+};
>+
>+struct octep_hp_intr_info {
>+	enum octep_hp_intr_type type;
>+	int number;
>+	char name[16];
>+};
>+
>+struct octep_hp_controller {
>+	void __iomem *base;
>+	struct pci_dev *pdev;
>+	struct octep_hp_intr_info intr[OCTEP_HP_INTR_MAX];
>+	struct work_struct work;
>+	struct list_head slot_list;
>+	struct mutex slot_lock; /* Protects slot_list */
>+	struct list_head hp_cmd_list;
>+	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
>+};
>+
>+static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl,
>+				 struct octep_hp_slot *hp_slot)
>+{
>+	guard(mutex)(&hp_ctrl->slot_lock);
>+	if (hp_slot->hp_pdev) {
>+		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %u already
>enabled\n",
>+			hp_slot->slot_number);
>+		return;
>+	}
>+
>+	/* Scan the device and add it to the bus */
>+	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus,
>+						  hp_slot->hp_devfn);
>+	pci_bus_assign_resources(hp_ctrl->pdev->bus);
>+	pci_bus_add_device(hp_slot->hp_pdev);
>+
>+	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %u\n",
>+		hp_slot->slot_number);
>+}
>+
>+static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
>+				  struct octep_hp_slot *hp_slot)
>+{
>+	guard(mutex)(&hp_ctrl->slot_lock);
>+	if (!hp_slot->hp_pdev) {
>+		dev_dbg(&hp_ctrl->pdev->dev, "Slot %u already disabled\n",
>+			hp_slot->slot_number);
>+		return;
>+	}
>+
>+	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %u\n",
>+		hp_slot->slot_number);
>+
>+	/* Remove the device from the bus */
>+	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
>+	hp_slot->hp_pdev = NULL;
>+}
>+
>+static int octep_hp_enable_slot(struct hotplug_slot *slot)
>+{
>+	struct octep_hp_slot *hp_slot =
>+		container_of(slot, struct octep_hp_slot, slot);
>+
>+	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
>+	return 0;
>+}
>+
>+static int octep_hp_disable_slot(struct hotplug_slot *slot)
>+{
>+	struct octep_hp_slot *hp_slot =
>+		container_of(slot, struct octep_hp_slot, slot);
>+
>+	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
>+	return 0;
>+}
>+
>+static struct hotplug_slot_ops octep_hp_slot_ops = {
>+	.enable_slot = octep_hp_enable_slot,
>+	.disable_slot = octep_hp_disable_slot,
>+};
>+
>+#define SLOT_NAME_SIZE 16
>+static struct octep_hp_slot *
>+octep_hp_register_slot(struct octep_hp_controller *hp_ctrl,
>+		       struct pci_dev *pdev, u16 slot_number)
>+{
>+	char slot_name[SLOT_NAME_SIZE];
>+	struct octep_hp_slot *hp_slot;
>+	int ret;
>+
>+	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
>+	if (!hp_slot)
>+		return ERR_PTR(-ENOMEM);
>+
>+	hp_slot->ctrl = hp_ctrl;
>+	hp_slot->hp_pdev = pdev;
>+	hp_slot->hp_devfn = pdev->devfn;
>+	hp_slot->slot_number = slot_number;
>+	hp_slot->slot.ops = &octep_hp_slot_ops;
>+
>+	snprintf(slot_name, sizeof(slot_name), "octep_hp_%u", slot_number);
>+	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
>+			      PCI_SLOT(pdev->devfn), slot_name);
>+	if (ret) {
>+		kfree(hp_slot);
>+		return ERR_PTR(ret);
>+	}
>+
>+	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
>+	octep_hp_disable_pdev(hp_ctrl, hp_slot);
>+
>+	return hp_slot;
>+}
>+
>+static void octep_hp_deregister_slot(void *data)
>+{
>+	struct octep_hp_slot *hp_slot = data;
>+	struct octep_hp_controller *hp_ctrl = hp_slot->ctrl;
>+
>+	pci_hp_deregister(&hp_slot->slot);
>+	octep_hp_enable_pdev(hp_ctrl, hp_slot);
>+	list_del(&hp_slot->list);
>+	kfree(hp_slot);
>+}
>+
>+static bool octep_hp_is_slot(struct octep_hp_controller *hp_ctrl,
>+			     struct pci_dev *pdev)
>+{
>+	/* Check if the PCI device can be hotplugged */
>+	return pdev != hp_ctrl->pdev && pdev->bus == hp_ctrl->pdev->bus &&
>+	       PCI_SLOT(pdev->devfn) == PCI_SLOT(hp_ctrl->pdev->devfn);
>+}
>+
>+static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
>+				 struct octep_hp_cmd *hp_cmd)
>+{
>+	struct octep_hp_slot *hp_slot;
>+
>+	/*
>+	 * Enable or disable the slots based on the slot mask.
>+	 * intr_val is a bit mask where each bit represents a slot.
>+	 */
>+	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
>+		if (!(hp_cmd->intr_val & BIT(hp_slot->slot_number)))
>+			continue;
>+
>+		if (hp_cmd->intr_type == OCTEP_HP_INTR_ENA)
>+			octep_hp_enable_pdev(hp_ctrl, hp_slot);
>+		else
>+			octep_hp_disable_pdev(hp_ctrl, hp_slot);
>+	}
>+}
>+
>+static void octep_hp_work_handler(struct work_struct *work)
>+{
>+	struct octep_hp_controller *hp_ctrl;
>+	struct octep_hp_cmd *hp_cmd;
>+	unsigned long flags;
>+
>+	hp_ctrl = container_of(work, struct octep_hp_controller, work);
>+
>+	/* Process all the hotplug commands */
>+	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>+	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
>+		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list,
>+					  struct octep_hp_cmd, list);
>+		list_del(&hp_cmd->list);
>+		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>+
>+		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
>+		kfree(hp_cmd);
>+
>+		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
>+	}
>+	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
>+}
>+
>+static enum octep_hp_intr_type octep_hp_intr_type(struct
>octep_hp_intr_info *intr,
>+						  int irq)
>+{
>+	enum octep_hp_intr_type type;
>+
>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>type++) {
>+		if (intr[type].number == irq)
>+			return type;
>+	}
>+
>+	return OCTEP_HP_INTR_INVALID;
>+}
>+
>+static irqreturn_t octep_hp_intr_handler(int irq, void *data)
>+{
>+	struct octep_hp_controller *hp_ctrl = data;
>+	struct pci_dev *pdev = hp_ctrl->pdev;
>+	enum octep_hp_intr_type type;
>+	struct octep_hp_cmd *hp_cmd;
>+	u64 intr_val;
>+
>+	type = octep_hp_intr_type(hp_ctrl->intr, irq);
>+	if (type == OCTEP_HP_INTR_INVALID) {
>+		dev_err(&pdev->dev, "Invalid interrupt %d\n", irq);
>+		return IRQ_HANDLED;
>+	}
>+
>+	/* Read and clear the interrupt */
>+	intr_val = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>+	writeq(intr_val, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
>+
>+	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
>+	if (!hp_cmd)
>+		return IRQ_HANDLED;
>+
>+	hp_cmd->intr_val = intr_val;
>+	hp_cmd->intr_type = type;
>+
>+	/* Add the command to the list and schedule the work */
>+	spin_lock(&hp_ctrl->hp_cmd_lock);
>+	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
>+	spin_unlock(&hp_ctrl->hp_cmd_lock);
>+	schedule_work(&hp_ctrl->work);
>+
>+	return IRQ_HANDLED;
>+}
>+
>+static void octep_hp_irq_cleanup(void *data)
>+{
>+	struct octep_hp_controller *hp_ctrl = data;
>+
>+	pci_free_irq_vectors(hp_ctrl->pdev);
>+	flush_work(&hp_ctrl->work);
>+}
>+
>+static int octep_hp_request_irq(struct octep_hp_controller *hp_ctrl,
>+				enum octep_hp_intr_type type)
>+{
>+	struct pci_dev *pdev = hp_ctrl->pdev;
>+	struct octep_hp_intr_info *intr;
>+	int irq;
>+
>+	irq = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(type));
>+	if (irq < 0)
>+		return irq;
>+
>+	intr = &hp_ctrl->intr[type];
>+	intr->number = irq;
>+	intr->type = type;
>+	snprintf(intr->name, sizeof(intr->name), "octep_hp_%d", type);
>+
>+	return devm_request_irq(&pdev->dev, irq, octep_hp_intr_handler,
>+				IRQF_SHARED, intr->name, hp_ctrl);
>+}
>+
>+static int octep_hp_controller_setup(struct pci_dev *pdev,
>+				     struct octep_hp_controller *hp_ctrl)
>+{
>+	struct device *dev = &pdev->dev;
>+	enum octep_hp_intr_type type;
>+	int ret;
>+
>+	ret = pcim_enable_device(pdev);
>+	if (ret)
>+		return dev_err_probe(dev, ret, "Failed to enable PCI device\n");
>+
>+	hp_ctrl->base = pcim_iomap_region(pdev, 0, OCTEP_HP_DRV_NAME);
>+	if (IS_ERR(hp_ctrl->base))
>+		return dev_err_probe(dev, PTR_ERR(hp_ctrl->base),
>+				     "Failed to map PCI device region\n");
>+
>+	pci_set_master(pdev);
>+	pci_set_drvdata(pdev, hp_ctrl);
>+
>+	INIT_LIST_HEAD(&hp_ctrl->slot_list);
>+	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
>+	mutex_init(&hp_ctrl->slot_lock);
>+	spin_lock_init(&hp_ctrl->hp_cmd_lock);
>+	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
>+	hp_ctrl->pdev = pdev;
>+
>+	ret = pci_alloc_irq_vectors(pdev, 1,
>+
>OCTEP_HP_INTR_VECTOR(OCTEP_HP_INTR_MAX),
>+				    PCI_IRQ_MSIX);
>+	if (ret < 0)
>+		return dev_err_probe(dev, ret, "Failed to alloc MSI-X
>vectors\n");
>+
>+	ret = devm_add_action(&pdev->dev, octep_hp_irq_cleanup, hp_ctrl);
>+	if (ret)
>+		return dev_err_probe(&pdev->dev, ret, "Failed to add IRQ
>cleanup action\n");
>+
>+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX;
>type++) {
>+		ret = octep_hp_request_irq(hp_ctrl, type);
>+		if (ret)
>+			return dev_err_probe(dev, ret,
>+					     "Failed to request IRQ for vector
>%d\n",
>+					     OCTEP_HP_INTR_VECTOR(type));
>+	}
>+
>+	return 0;
>+}
>+
>+static int octep_hp_pci_probe(struct pci_dev *pdev,
>+			      const struct pci_device_id *id)
>+{
>+	struct octep_hp_controller *hp_ctrl;
>+	struct pci_dev *tmp_pdev = NULL;
>+	struct octep_hp_slot *hp_slot;
>+	u16 slot_number = 0;
>+	int ret;
>+
>+	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
>+	if (!hp_ctrl)
>+		return -ENOMEM;
>+
>+	ret = octep_hp_controller_setup(pdev, hp_ctrl);
>+	if (ret)
>+		return ret;
>+
>+	/*
>+	 * Register all hotplug slots. Hotplug controller is the first function
>+	 * of the PCI device. The hotplug slots are the remaining functions of
>+	 * the PCI device. They are removed from the bus and are added back
>when
>+	 * the hotplug event is triggered.
>+	 */
>+	for_each_pci_dev(tmp_pdev) {
>+		if (!octep_hp_is_slot(hp_ctrl, tmp_pdev))
>+			continue;
>+
>+		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev,
>slot_number);
>+		if (IS_ERR(hp_slot))
>+			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
>+					     "Failed to register hotplug slot
>%u\n",
>+					     slot_number);
>+
>+		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
>+				      hp_slot);
>+		if (ret)
>+			return dev_err_probe(&pdev->dev, ret,
>+					     "Failed to add action for
>deregistering slot %u\n",
>+					     slot_number);
>+		slot_number++;
>+	}
>+
>+	return 0;
>+}
>+
>+#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
>+static struct pci_device_id octep_hp_pci_map[] = {
>+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
>OCTEP_DEVID_HP_CONTROLLER) },
>+	{ },
>+};
>+
>+static struct pci_driver octep_hp = {
>+	.name = OCTEP_HP_DRV_NAME,
>+	.id_table = octep_hp_pci_map,
>+	.probe = octep_hp_pci_probe,
>+};
>+
>+module_pci_driver(octep_hp);
>+
>+MODULE_LICENSE("GPL");
>+MODULE_AUTHOR("Marvell");
>+MODULE_DESCRIPTION("OCTEON PCIe device hotplug controller driver");
>--
>2.25.1

Re: [PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Bjorn Helgaas]

On Mon, Aug 26, 2024 at 04:15:03PM +0530, Shijith Thotton wrote:
> This patch introduces a PCI hotplug controller driver for the OCTEON
> PCIe device, a multi-function PCIe device where the first function acts
> as a hotplug controller. It is equipped with MSI-x interrupts to notify
> the host of hotplug events from the OCTEON firmware.

s/MSI-x/MSI-X/ to match spec and other uses

> The driver facilitates the hotplugging of non-controller functions
> within the same device. During probe, non-controller functions are
> removed and registered as PCI hotplug slots. The slots are added back
> only upon request from the device firmware. The driver also allows the
> enabling and disabling of the slots via sysfs slot entries, provided by
> the PCI hotplug framework.
> 
> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
> Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>

This order is incorrect because the handler (Shijith in this case)
should be last in the chain; see
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?id=v6.11#n396

> This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
> controller. The OCTEON PCIe device is a multi-function device where the first
> function acts as a PCI hotplug controller.
> 
>               +--------------------------------+
>               |           Root Port            |
>               +--------------------------------+
>                               |
>                              PCIe
>                               |
> +---------------------------------------------------------------+
> |              OCTEON PCIe Multifunction Device                 |
> +---------------------------------------------------------------+
>             |                    |              |            |
>             |                    |              |            |
> +---------------------+  +----------------+  +-----+  +----------------+
> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
> +---------------------+  +----------------+  +-----+  +----------------+
>             |
>             |
> +-------------------------+
> |   Controller Firmware   |
> +-------------------------+
> 
> The hotplug controller driver facilitates the hotplugging of non-controller
> functions in the same device. During the probe of the driver, the non-controller
> function are removed and registered as PCI hotplug slots. They are added back
> only upon request from the device firmware. The driver also allows the user to
> enable/disable the functions using sysfs slot entries provided by PCI hotplug
> framework.

I think the diagram and this text would be useful in the commit log.

I would rephrase "functions are removed ..." as "the driver removes
functions" to make it clear that this is purely a software thing and
there's no PCIe-level change.  Similar for adding them back.

> This solution adopts a hardware + software approach for several reasons:
> 
> 1. To reduce hardware implementation cost. Supporting complete hotplug
>    capability within the card would require a PCI switch implemented within.
> 
> 2. In the multi-function device, non-controller functions can act as emulated
>    devices. The firmware can dynamically enable or disable them at runtime.
> 
> 3. Not all root ports support PCI hotplug. This approach provides greater
>    flexibility and compatibility across different hardware configurations.

The downside of all this is the need for special-purpose software,
which slows things down (you need to develop it, others need to review
it) and burdens everybody with ongoing maintenance, e.g., changes to
PCI device removal, resource assignment, slot registration, etc. now
have to consider another case.

> The hotplug controller function is lightweight and is equipped with MSI-x
> interrupts to notify the host about hotplug events. Upon receiving an
> interrupt, the hotplug register is read, and the required function is enabled
> or disabled.
> 
> This driver will be beneficial for managing PCI hotplug events on the OCTEON
> PCIe device without requiring complex hardware solutions.

> +config HOTPLUG_PCI_OCTEONEP
> +	bool "OCTEON PCI device Hotplug controller driver"

s/Marvell PCI device/Cavium OCTEON PCI/ to match other entries.

This Kconfig file isn't completely sorted, but if you move this above
SHPC, it will at least be closer.

> +static int octep_hp_pci_probe(struct pci_dev *pdev,
> +			      const struct pci_device_id *id)
> +{
> +	struct octep_hp_controller *hp_ctrl;
> +	struct pci_dev *tmp_pdev = NULL;
> +	struct octep_hp_slot *hp_slot;
> +	u16 slot_number = 0;
> +	int ret;
> +
> +	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
> +	if (!hp_ctrl)
> +		return -ENOMEM;
> +
> +	ret = octep_hp_controller_setup(pdev, hp_ctrl);
> +	if (ret)
> +		return ret;
> +
> +	/*
> +	 * Register all hotplug slots. Hotplug controller is the first function
> +	 * of the PCI device. The hotplug slots are the remaining functions of
> +	 * the PCI device. They are removed from the bus and are added back when
> +	 * the hotplug event is triggered.

"Logically removed," I guess?  Obviously the PCIe Link remains up
since function 0 is still alive, and I assume these other functions
would still respond to config accesses.

> +	 */
> +	for_each_pci_dev(tmp_pdev) {

Ugh.  We should avoid for_each_pci_dev() when possible.

IIUC you only care about other functions of *this* device, and those
functions should all be on the bus->devices list.  There are many
instances of this, which I think would be sufficient:

  list_for_each_entry(dev, &bus->devices, bus_list)

> +		if (!octep_hp_is_slot(hp_ctrl, tmp_pdev))
> +			continue;

Which would make this check mostly unnecessary.

> +		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev, slot_number);
> +		if (IS_ERR(hp_slot))
> +			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
> +					     "Failed to register hotplug slot %u\n",
> +					     slot_number);
> +
> +		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
> +				      hp_slot);
> +		if (ret)
> +			return dev_err_probe(&pdev->dev, ret,
> +					     "Failed to add action for deregistering slot %u\n",
> +					     slot_number);
> +		slot_number++;
> +	}

AFAICS this driver logs nothing at all in dmesg (except for errors and
a few dev_dbg() uses).  Seems like it might be useful to have some
hints there about the hotplug controller existence, possibly the
connection between the slot name used in sysfs and the PCI function,
and maybe even hot-add and hot-remove events.

> +	return 0;
> +}
> +
> +#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3

Even though this is a private #define, I think something like
PCI_DEVICE_ID_CAVIUM_OCTEP_HP_CTLR would be nice because that's the
typical pattern of include/linux/pci_ids.h.

> +static struct pci_device_id octep_hp_pci_map[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, OCTEP_DEVID_HP_CONTROLLER) },
> +	{ },
> +};

Re: [PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

>> This patch introduces a PCI hotplug controller driver for the OCTEON
>> PCIe device, a multi-function PCIe device where the first function acts
>> as a hotplug controller. It is equipped with MSI-x interrupts to notify
>> the host of hotplug events from the OCTEON firmware.
>
>s/MSI-x/MSI-X/ to match spec and other uses

I will change in v4

>
>> The driver facilitates the hotplugging of non-controller functions
>> within the same device. During probe, non-controller functions are
>> removed and registered as PCI hotplug slots. The slots are added back
>> only upon request from the device firmware. The driver also allows the
>> enabling and disabling of the slots via sysfs slot entries, provided by
>> the PCI hotplug framework.
>>
>> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>> Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
>> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>
>This order is incorrect because the handler (Shijith in this case)
>should be last in the chain; see
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?id=v6.11#n396
>

I will change in v4.

>> This patch introduces a PCI hotplug controller driver for OCTEON PCIe hotplug
>> controller. The OCTEON PCIe device is a multi-function device where the first
>> function acts as a PCI hotplug controller.
>>
>>               +--------------------------------+
>>               |           Root Port            |
>>               +--------------------------------+
>>                               |
>>                              PCIe
>>                               |
>> +---------------------------------------------------------------+
>> |              OCTEON PCIe Multifunction Device                 |
>> +---------------------------------------------------------------+
>>             |                    |              |            |
>>             |                    |              |            |
>> +---------------------+  +----------------+  +-----+  +----------------+
>> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>> +---------------------+  +----------------+  +-----+  +----------------+
>>             |
>>             |
>> +-------------------------+
>> |   Controller Firmware   |
>> +-------------------------+
>>
>> The hotplug controller driver facilitates the hotplugging of non-controller
>> functions in the same device. During the probe of the driver, the non-
>controller
>> function are removed and registered as PCI hotplug slots. They are added
>back
>> only upon request from the device firmware. The driver also allows the user
>to
>> enable/disable the functions using sysfs slot entries provided by PCI hotplug
>> framework.
>
>I think the diagram and this text would be useful in the commit log.
>
>I would rephrase "functions are removed ..." as "the driver removes
>functions" to make it clear that this is purely a software thing and
>there's no PCIe-level change.  Similar for adding them back.
>

I will update the commit log with the diagram and more wordings from above.

>> This solution adopts a hardware + software approach for several reasons:
>>
>> 1. To reduce hardware implementation cost. Supporting complete hotplug
>>    capability within the card would require a PCI switch implemented within.
>>
>> 2. In the multi-function device, non-controller functions can act as emulated
>>    devices. The firmware can dynamically enable or disable them at runtime.
>>
>> 3. Not all root ports support PCI hotplug. This approach provides greater
>>    flexibility and compatibility across different hardware configurations.
>
>The downside of all this is the need for special-purpose software,
>which slows things down (you need to develop it, others need to review
>it) and burdens everybody with ongoing maintenance, e.g., changes to
>PCI device removal, resource assignment, slot registration, etc. now
>have to consider another case.
>
>> The hotplug controller function is lightweight and is equipped with MSI-x
>> interrupts to notify the host about hotplug events. Upon receiving an
>> interrupt, the hotplug register is read, and the required function is enabled
>> or disabled.
>>
>> This driver will be beneficial for managing PCI hotplug events on the OCTEON
>> PCIe device without requiring complex hardware solutions.
>
>> +config HOTPLUG_PCI_OCTEONEP
>> +	bool "OCTEON PCI device Hotplug controller driver"
>
>s/Marvell PCI device/Cavium OCTEON PCI/ to match other entries.
>

Cavium was acquired by Marvell, but we are still using the PCI vendor ID  
`PCI_VENDOR_ID_CAVIUM`. I hope using Marvell is acceptable; please let me  
know if this poses any issues.

>This Kconfig file isn't completely sorted, but if you move this above
>SHPC, it will at least be closer.
>

I will move above SHPC.

>> +static int octep_hp_pci_probe(struct pci_dev *pdev,
>> +			      const struct pci_device_id *id)
>> +{
>> +	struct octep_hp_controller *hp_ctrl;
>> +	struct pci_dev *tmp_pdev = NULL;
>> +	struct octep_hp_slot *hp_slot;
>> +	u16 slot_number = 0;
>> +	int ret;
>> +
>> +	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
>> +	if (!hp_ctrl)
>> +		return -ENOMEM;
>> +
>> +	ret = octep_hp_controller_setup(pdev, hp_ctrl);
>> +	if (ret)
>> +		return ret;
>> +
>> +	/*
>> +	 * Register all hotplug slots. Hotplug controller is the first function
>> +	 * of the PCI device. The hotplug slots are the remaining functions of
>> +	 * the PCI device. They are removed from the bus and are added back
>when
>> +	 * the hotplug event is triggered.
>
>"Logically removed," I guess?  Obviously the PCIe Link remains up
>since function 0 is still alive, and I assume these other functions
>would still respond to config accesses.
>

Yes, it's just a software removal. I will update the text.

>> +	 */
>> +	for_each_pci_dev(tmp_pdev) {
>
>Ugh.  We should avoid for_each_pci_dev() when possible.
>
>IIUC you only care about other functions of *this* device, and those
>functions should all be on the bus->devices list.  There are many
>instances of this, which I think would be sufficient:
>
>  list_for_each_entry(dev, &bus->devices, bus_list)
>

Okay. I will replace for_each_pci_dev() with list_for_each_entry_safe().

>> +		if (!octep_hp_is_slot(hp_ctrl, tmp_pdev))
>> +			continue;
>
>Which would make this check mostly unnecessary.

Yes, I will replace it with the check below to skip the controller function.
  if (tmp_pdev == pdev)
      continue;

>
>> +		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev,
>slot_number);
>> +		if (IS_ERR(hp_slot))
>> +			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
>> +					     "Failed to register hotplug slot
>%u\n",
>> +					     slot_number);
>> +
>> +		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
>> +				      hp_slot);
>> +		if (ret)
>> +			return dev_err_probe(&pdev->dev, ret,
>> +					     "Failed to add action for
>deregistering slot %u\n",
>> +					     slot_number);
>> +		slot_number++;
>> +	}
>
>AFAICS this driver logs nothing at all in dmesg (except for errors and
>a few dev_dbg() uses).  Seems like it might be useful to have some
>hints there about the hotplug controller existence, possibly the
>connection between the slot name used in sysfs and the PCI function,
>and maybe even hot-add and hot-remove events.
>

Okay. I’ll add more informative prints as mentioned above.

>> +	return 0;
>> +}
>> +
>> +#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
>
>Even though this is a private #define, I think something like
>PCI_DEVICE_ID_CAVIUM_OCTEP_HP_CTLR would be nice because that's the
>typical pattern of include/linux/pci_ids.h.
>

The same reason mentioned above for not using the name CAVIUM.
Is it  okay to use `PCI_DEVICE_ID_MARVELL_OCTEP_HP_CTLR`?

>> +static struct pci_device_id octep_hp_pci_map[] = {
>> +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
>OCTEP_DEVID_HP_CONTROLLER) },
>> +	{ },
>> +};

Thanks,
Shijith

Re: [PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Bjorn Helgaas]

On Fri, Nov 08, 2024 at 12:17:22PM +0000, Shijith Thotton wrote:
> >> This patch introduces a PCI hotplug controller driver for the OCTEON
> >> PCIe device, a multi-function PCIe device where the first function acts
> >> as a hotplug controller. It is equipped with MSI-x interrupts to notify
> >> the host of hotplug events from the OCTEON firmware.

> >> +config HOTPLUG_PCI_OCTEONEP
> >> +	bool "OCTEON PCI device Hotplug controller driver"
> >
> >s/Marvell PCI device/Cavium OCTEON PCI/ to match other entries.
> 
> Cavium was acquired by Marvell, but we are still using the PCI
> vendor ID  `PCI_VENDOR_ID_CAVIUM`. I hope using Marvell is
> acceptable; please let me  know if this poses any issues.

Oops, sorry, this was my mistake.  I meant to suggest "Marvell OCTEON
PCI Hotplug driver", but I messed up the editing.

> >> +#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
> >
> >Even though this is a private #define, I think something like
> >PCI_DEVICE_ID_CAVIUM_OCTEP_HP_CTLR would be nice because that's the
> >typical pattern of include/linux/pci_ids.h.
> 
> The same reason mentioned above for not using the name CAVIUM.
> Is it okay to use `PCI_DEVICE_ID_MARVELL_OCTEP_HP_CTLR`?

In this case, I think you should use PCI_DEVICE_ID_CAVIUM... because
the PCI_VENDOR_ID_CAVIUM identifies the Device ID namespace, so the
#defines for the Vendor ID and the Device IDs in that namespace should
match.

> >> +static struct pci_device_id octep_hp_pci_map[] = {
> >> +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
> >OCTEP_DEVID_HP_CONTROLLER) },
> >> +	{ },
> >> +};

Re: [PATCH v3] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

>> >> This patch introduces a PCI hotplug controller driver for the OCTEON
>> >> PCIe device, a multi-function PCIe device where the first function acts
>> >> as a hotplug controller. It is equipped with MSI-x interrupts to notify
>> >> the host of hotplug events from the OCTEON firmware.
>
>> >> +config HOTPLUG_PCI_OCTEONEP
>> >> +	bool "OCTEON PCI device Hotplug controller driver"
>> >
>> >s/Marvell PCI device/Cavium OCTEON PCI/ to match other entries.
>>
>> Cavium was acquired by Marvell, but we are still using the PCI
>> vendor ID  `PCI_VENDOR_ID_CAVIUM`. I hope using Marvell is
>> acceptable; please let me  know if this poses any issues.
>
>Oops, sorry, this was my mistake.  I meant to suggest "Marvell OCTEON
>PCI Hotplug driver", but I messed up the editing.
>
>> >> +#define OCTEP_DEVID_HP_CONTROLLER 0xa0e3
>> >
>> >Even though this is a private #define, I think something like
>> >PCI_DEVICE_ID_CAVIUM_OCTEP_HP_CTLR would be nice because that's the
>> >typical pattern of include/linux/pci_ids.h.
>>
>> The same reason mentioned above for not using the name CAVIUM.
>> Is it okay to use `PCI_DEVICE_ID_MARVELL_OCTEP_HP_CTLR`?
>
>In this case, I think you should use PCI_DEVICE_ID_CAVIUM... because
>the PCI_VENDOR_ID_CAVIUM identifies the Device ID namespace, so the
>#defines for the Vendor ID and the Device IDs in that namespace should
>match.
>

Okay. I will use CAVIUM for the define.

>> >> +static struct pci_device_id octep_hp_pci_map[] = {
>> >> +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM,
>> >OCTEP_DEVID_HP_CONTROLLER) },
>> >> +	{ },
>> >> +};

[PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

This patch introduces a PCI hotplug controller driver for the OCTEON
PCIe device. The OCTEON PCIe device is a multi-function device where the
first function serves as the PCI hotplug controller.

               +--------------------------------+
               |           Root Port            |
               +--------------------------------+
                               |
                              PCIe
                               |
+---------------------------------------------------------------+
|              OCTEON PCIe Multifunction Device                 |
+---------------------------------------------------------------+
             |                    |              |            |
             |                    |              |            |
+---------------------+  +----------------+  +-----+  +----------------+
|      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
| (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
+---------------------+  +----------------+  +-----+  +----------------+
             |
             |
+-------------------------+
|   Controller Firmware   |
+-------------------------+

The hotplug controller driver enables hotplugging of non-controller
functions within the same device. During probing, the driver removes
the non-controller functions and registers them as PCI hotplug slots.
These slots are added back by the driver, only upon request from the
device firmware.

The controller uses MSI-X interrupts to notify the host of hotplug
events initiated by the OCTEON firmware. Additionally, the driver
allows users to enable or disable individual functions via sysfs slot
entries, as provided by the PCI hotplug framework.

Co-developed-by: Vamsi Attunuru <vattunuru@marvell.com>
Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
Signed-off-by: Shijith Thotton <sthotton@marvell.com>
---
Changes in v2:
- Added missing include files.
- Used dev_err_probe() for error handling.
- Used guard() for mutex locking.
- Splited cleanup actions and added per-slot cleanup action.
- Fixed coding style issues.
- Added co-developed-by tag.

Changes in v3:
- Explicit assignment of enum values.
- Use pcim_iomap_region() instead of pcim_iomap_regions().

Changes in v4:
- Updated the commit message.
- Improved wordings and added new log messages.
- Modified device ID define to match namespace.
- Avoided use of for_each_pci_dev().

 MAINTAINERS                    |   6 +
 drivers/pci/hotplug/Kconfig    |  10 +
 drivers/pci/hotplug/Makefile   |   1 +
 drivers/pci/hotplug/octep_hp.c | 427 +++++++++++++++++++++++++++++++++
 4 files changed, 444 insertions(+)
 create mode 100644 drivers/pci/hotplug/octep_hp.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 42decde38320..2258f1459440 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13677,6 +13677,12 @@ R:	schalla@marvell.com
 R:	vattunuru@marvell.com
 F:	drivers/vdpa/octeon_ep/
 
+MARVELL OCTEON HOTPLUG DRIVER
+R:	Shijith Thotton <sthotton@marvell.com>
+R:	Vamsi Attunuru <vattunuru@marvell.com>
+S:	Supported
+F:	drivers/pci/hotplug/octep_hp.c
+
 MATROX FRAMEBUFFER DRIVER
 L:	linux-fbdev@vger.kernel.org
 S:	Orphan
diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
index 1472aef0fb81..123c4c7c2ab5 100644
--- a/drivers/pci/hotplug/Kconfig
+++ b/drivers/pci/hotplug/Kconfig
@@ -118,6 +118,16 @@ config HOTPLUG_PCI_CPCI_GENERIC
 
 	  When in doubt, say N.
 
+config HOTPLUG_PCI_OCTEONEP
+	bool "Marvell OCTEON PCI Hotplug driver"
+	depends on HOTPLUG_PCI
+	help
+	  Say Y here if you have an OCTEON PCIe device with a hotplug
+	  controller. This driver enables the non-controller functions of the
+	  device to be registered as hotplug slots.
+
+	  When in doubt, say N.
+
 config HOTPLUG_PCI_SHPC
 	bool "SHPC PCI Hotplug driver"
 	help
diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
index 240c99517d5e..40aaf31fe338 100644
--- a/drivers/pci/hotplug/Makefile
+++ b/drivers/pci/hotplug/Makefile
@@ -20,6 +20,7 @@ obj-$(CONFIG_HOTPLUG_PCI_RPA)		+= rpaphp.o
 obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)	+= rpadlpar_io.o
 obj-$(CONFIG_HOTPLUG_PCI_ACPI)		+= acpiphp.o
 obj-$(CONFIG_HOTPLUG_PCI_S390)		+= s390_pci_hpc.o
+obj-$(CONFIG_HOTPLUG_PCI_OCTEONEP)	+= octep_hp.o
 
 # acpiphp_ibm extends acpiphp, so should be linked afterwards.
 
diff --git a/drivers/pci/hotplug/octep_hp.c b/drivers/pci/hotplug/octep_hp.c
new file mode 100644
index 000000000000..d2475feb44cc
--- /dev/null
+++ b/drivers/pci/hotplug/octep_hp.c
@@ -0,0 +1,427 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2024 Marvell. */
+
+#include <linux/cleanup.h>
+#include <linux/container_of.h>
+#include <linux/delay.h>
+#include <linux/dev_printk.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/pci_hotplug.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/workqueue.h>
+
+#define OCTEP_HP_INTR_OFFSET(x) (0x20400 + ((x) << 4))
+#define OCTEP_HP_INTR_VECTOR(x) (16 + (x))
+#define OCTEP_HP_DRV_NAME "octep_hp"
+
+/*
+ * Type of MSI-X interrupts.
+ * The macros OCTEP_HP_INTR_VECTOR and OCTEP_HP_INTR_OFFSET are used to
+ * generate the vector and offset for an interrupt type.
+ */
+enum octep_hp_intr_type {
+	OCTEP_HP_INTR_INVALID = -1,
+	OCTEP_HP_INTR_ENA = 0,
+	OCTEP_HP_INTR_DIS = 1,
+	OCTEP_HP_INTR_MAX = 2,
+};
+
+struct octep_hp_cmd {
+	struct list_head list;
+	enum octep_hp_intr_type intr_type;
+	u64 intr_val;
+};
+
+struct octep_hp_slot {
+	struct list_head list;
+	struct hotplug_slot slot;
+	u16 slot_number;
+	struct pci_dev *hp_pdev;
+	unsigned int hp_devfn;
+	struct octep_hp_controller *ctrl;
+};
+
+struct octep_hp_intr_info {
+	enum octep_hp_intr_type type;
+	int number;
+	char name[16];
+};
+
+struct octep_hp_controller {
+	void __iomem *base;
+	struct pci_dev *pdev;
+	struct octep_hp_intr_info intr[OCTEP_HP_INTR_MAX];
+	struct work_struct work;
+	struct list_head slot_list;
+	struct mutex slot_lock; /* Protects slot_list */
+	struct list_head hp_cmd_list;
+	spinlock_t hp_cmd_lock; /* Protects hp_cmd_list */
+};
+
+static void octep_hp_enable_pdev(struct octep_hp_controller *hp_ctrl,
+				 struct octep_hp_slot *hp_slot)
+{
+	guard(mutex)(&hp_ctrl->slot_lock);
+	if (hp_slot->hp_pdev) {
+		dev_dbg(&hp_slot->hp_pdev->dev, "Slot %s is already enabled\n",
+			hotplug_slot_name(&hp_slot->slot));
+		return;
+	}
+
+	/* Scan the device and add it to the bus */
+	hp_slot->hp_pdev = pci_scan_single_device(hp_ctrl->pdev->bus,
+						  hp_slot->hp_devfn);
+	pci_bus_assign_resources(hp_ctrl->pdev->bus);
+	pci_bus_add_device(hp_slot->hp_pdev);
+
+	dev_dbg(&hp_slot->hp_pdev->dev, "Enabled slot %s\n",
+		hotplug_slot_name(&hp_slot->slot));
+}
+
+static void octep_hp_disable_pdev(struct octep_hp_controller *hp_ctrl,
+				  struct octep_hp_slot *hp_slot)
+{
+	guard(mutex)(&hp_ctrl->slot_lock);
+	if (!hp_slot->hp_pdev) {
+		dev_dbg(&hp_ctrl->pdev->dev, "Slot %s is already disabled\n",
+			hotplug_slot_name(&hp_slot->slot));
+		return;
+	}
+
+	dev_dbg(&hp_slot->hp_pdev->dev, "Disabling slot %s\n",
+		hotplug_slot_name(&hp_slot->slot));
+
+	/* Remove the device from the bus */
+	pci_stop_and_remove_bus_device_locked(hp_slot->hp_pdev);
+	hp_slot->hp_pdev = NULL;
+}
+
+static int octep_hp_enable_slot(struct hotplug_slot *slot)
+{
+	struct octep_hp_slot *hp_slot =
+		container_of(slot, struct octep_hp_slot, slot);
+
+	octep_hp_enable_pdev(hp_slot->ctrl, hp_slot);
+	return 0;
+}
+
+static int octep_hp_disable_slot(struct hotplug_slot *slot)
+{
+	struct octep_hp_slot *hp_slot =
+		container_of(slot, struct octep_hp_slot, slot);
+
+	octep_hp_disable_pdev(hp_slot->ctrl, hp_slot);
+	return 0;
+}
+
+static struct hotplug_slot_ops octep_hp_slot_ops = {
+	.enable_slot = octep_hp_enable_slot,
+	.disable_slot = octep_hp_disable_slot,
+};
+
+#define SLOT_NAME_SIZE 16
+static struct octep_hp_slot *
+octep_hp_register_slot(struct octep_hp_controller *hp_ctrl,
+		       struct pci_dev *pdev, u16 slot_number)
+{
+	char slot_name[SLOT_NAME_SIZE];
+	struct octep_hp_slot *hp_slot;
+	int ret;
+
+	hp_slot = kzalloc(sizeof(*hp_slot), GFP_KERNEL);
+	if (!hp_slot)
+		return ERR_PTR(-ENOMEM);
+
+	hp_slot->ctrl = hp_ctrl;
+	hp_slot->hp_pdev = pdev;
+	hp_slot->hp_devfn = pdev->devfn;
+	hp_slot->slot_number = slot_number;
+	hp_slot->slot.ops = &octep_hp_slot_ops;
+
+	snprintf(slot_name, sizeof(slot_name), "octep_hp_%u", slot_number);
+	ret = pci_hp_register(&hp_slot->slot, hp_ctrl->pdev->bus,
+			      PCI_SLOT(pdev->devfn), slot_name);
+	if (ret) {
+		kfree(hp_slot);
+		return ERR_PTR(ret);
+	}
+
+	dev_info(&pdev->dev, "Registered slot %s for device %s\n",
+		 slot_name, pci_name(pdev));
+
+	list_add_tail(&hp_slot->list, &hp_ctrl->slot_list);
+	octep_hp_disable_pdev(hp_ctrl, hp_slot);
+
+	return hp_slot;
+}
+
+static void octep_hp_deregister_slot(void *data)
+{
+	struct octep_hp_slot *hp_slot = data;
+	struct octep_hp_controller *hp_ctrl = hp_slot->ctrl;
+
+	pci_hp_deregister(&hp_slot->slot);
+	octep_hp_enable_pdev(hp_ctrl, hp_slot);
+	list_del(&hp_slot->list);
+	kfree(hp_slot);
+}
+
+static const char *octep_hp_cmd_name(enum octep_hp_intr_type type)
+{
+	switch (type) {
+	case OCTEP_HP_INTR_ENA:
+		return "hotplug enable";
+	case OCTEP_HP_INTR_DIS:
+		return "hotplug disable";
+	default:
+		return "invalid";
+	}
+}
+
+static void octep_hp_cmd_handler(struct octep_hp_controller *hp_ctrl,
+				 struct octep_hp_cmd *hp_cmd)
+{
+	struct octep_hp_slot *hp_slot;
+
+	/*
+	 * Enable or disable the slots based on the slot mask.
+	 * intr_val is a bit mask where each bit represents a slot.
+	 */
+	list_for_each_entry(hp_slot, &hp_ctrl->slot_list, list) {
+		if (!(hp_cmd->intr_val & BIT(hp_slot->slot_number)))
+			continue;
+
+		dev_info(&hp_ctrl->pdev->dev, "Received %s command for slot %s\n",
+			 octep_hp_cmd_name(hp_cmd->intr_type),
+			 hotplug_slot_name(&hp_slot->slot));
+
+		switch (hp_cmd->intr_type) {
+		case OCTEP_HP_INTR_ENA:
+			octep_hp_enable_pdev(hp_ctrl, hp_slot);
+			break;
+		case OCTEP_HP_INTR_DIS:
+			octep_hp_disable_pdev(hp_ctrl, hp_slot);
+			break;
+		default:
+			break;
+		}
+	}
+}
+
+static void octep_hp_work_handler(struct work_struct *work)
+{
+	struct octep_hp_controller *hp_ctrl;
+	struct octep_hp_cmd *hp_cmd;
+	unsigned long flags;
+
+	hp_ctrl = container_of(work, struct octep_hp_controller, work);
+
+	/* Process all the hotplug commands */
+	spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
+	while (!list_empty(&hp_ctrl->hp_cmd_list)) {
+		hp_cmd = list_first_entry(&hp_ctrl->hp_cmd_list,
+					  struct octep_hp_cmd, list);
+		list_del(&hp_cmd->list);
+		spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
+
+		octep_hp_cmd_handler(hp_ctrl, hp_cmd);
+		kfree(hp_cmd);
+
+		spin_lock_irqsave(&hp_ctrl->hp_cmd_lock, flags);
+	}
+	spin_unlock_irqrestore(&hp_ctrl->hp_cmd_lock, flags);
+}
+
+static enum octep_hp_intr_type octep_hp_intr_type(struct octep_hp_intr_info *intr,
+						  int irq)
+{
+	enum octep_hp_intr_type type;
+
+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX; type++) {
+		if (intr[type].number == irq)
+			return type;
+	}
+
+	return OCTEP_HP_INTR_INVALID;
+}
+
+static irqreturn_t octep_hp_intr_handler(int irq, void *data)
+{
+	struct octep_hp_controller *hp_ctrl = data;
+	struct pci_dev *pdev = hp_ctrl->pdev;
+	enum octep_hp_intr_type type;
+	struct octep_hp_cmd *hp_cmd;
+	u64 intr_val;
+
+	type = octep_hp_intr_type(hp_ctrl->intr, irq);
+	if (type == OCTEP_HP_INTR_INVALID) {
+		dev_err(&pdev->dev, "Invalid interrupt %d\n", irq);
+		return IRQ_HANDLED;
+	}
+
+	/* Read and clear the interrupt */
+	intr_val = readq(hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
+	writeq(intr_val, hp_ctrl->base + OCTEP_HP_INTR_OFFSET(type));
+
+	hp_cmd = kzalloc(sizeof(*hp_cmd), GFP_ATOMIC);
+	if (!hp_cmd)
+		return IRQ_HANDLED;
+
+	hp_cmd->intr_val = intr_val;
+	hp_cmd->intr_type = type;
+
+	/* Add the command to the list and schedule the work */
+	spin_lock(&hp_ctrl->hp_cmd_lock);
+	list_add_tail(&hp_cmd->list, &hp_ctrl->hp_cmd_list);
+	spin_unlock(&hp_ctrl->hp_cmd_lock);
+	schedule_work(&hp_ctrl->work);
+
+	return IRQ_HANDLED;
+}
+
+static void octep_hp_irq_cleanup(void *data)
+{
+	struct octep_hp_controller *hp_ctrl = data;
+
+	pci_free_irq_vectors(hp_ctrl->pdev);
+	flush_work(&hp_ctrl->work);
+}
+
+static int octep_hp_request_irq(struct octep_hp_controller *hp_ctrl,
+				enum octep_hp_intr_type type)
+{
+	struct pci_dev *pdev = hp_ctrl->pdev;
+	struct octep_hp_intr_info *intr;
+	int irq;
+
+	irq = pci_irq_vector(pdev, OCTEP_HP_INTR_VECTOR(type));
+	if (irq < 0)
+		return irq;
+
+	intr = &hp_ctrl->intr[type];
+	intr->number = irq;
+	intr->type = type;
+	snprintf(intr->name, sizeof(intr->name), "octep_hp_%d", type);
+
+	return devm_request_irq(&pdev->dev, irq, octep_hp_intr_handler,
+				IRQF_SHARED, intr->name, hp_ctrl);
+}
+
+static int octep_hp_controller_setup(struct pci_dev *pdev,
+				     struct octep_hp_controller *hp_ctrl)
+{
+	struct device *dev = &pdev->dev;
+	enum octep_hp_intr_type type;
+	int ret;
+
+	ret = pcim_enable_device(pdev);
+	if (ret)
+		return dev_err_probe(dev, ret, "Failed to enable PCI device\n");
+
+	hp_ctrl->base = pcim_iomap_region(pdev, 0, OCTEP_HP_DRV_NAME);
+	if (IS_ERR(hp_ctrl->base))
+		return dev_err_probe(dev, PTR_ERR(hp_ctrl->base),
+				     "Failed to map PCI device region\n");
+
+	pci_set_master(pdev);
+	pci_set_drvdata(pdev, hp_ctrl);
+
+	INIT_LIST_HEAD(&hp_ctrl->slot_list);
+	INIT_LIST_HEAD(&hp_ctrl->hp_cmd_list);
+	mutex_init(&hp_ctrl->slot_lock);
+	spin_lock_init(&hp_ctrl->hp_cmd_lock);
+	INIT_WORK(&hp_ctrl->work, octep_hp_work_handler);
+	hp_ctrl->pdev = pdev;
+
+	ret = pci_alloc_irq_vectors(pdev, 1,
+				    OCTEP_HP_INTR_VECTOR(OCTEP_HP_INTR_MAX),
+				    PCI_IRQ_MSIX);
+	if (ret < 0)
+		return dev_err_probe(dev, ret, "Failed to alloc MSI-X vectors\n");
+
+	ret = devm_add_action(&pdev->dev, octep_hp_irq_cleanup, hp_ctrl);
+	if (ret)
+		return dev_err_probe(&pdev->dev, ret, "Failed to add IRQ cleanup action\n");
+
+	for (type = OCTEP_HP_INTR_ENA; type < OCTEP_HP_INTR_MAX; type++) {
+		ret = octep_hp_request_irq(hp_ctrl, type);
+		if (ret)
+			return dev_err_probe(dev, ret,
+					     "Failed to request IRQ for vector %d\n",
+					     OCTEP_HP_INTR_VECTOR(type));
+	}
+
+	return 0;
+}
+
+static int octep_hp_pci_probe(struct pci_dev *pdev,
+			      const struct pci_device_id *id)
+{
+	struct octep_hp_controller *hp_ctrl;
+	struct pci_dev *tmp_pdev, *next;
+	struct octep_hp_slot *hp_slot;
+	u16 slot_number = 0;
+	int ret;
+
+	hp_ctrl = devm_kzalloc(&pdev->dev, sizeof(*hp_ctrl), GFP_KERNEL);
+	if (!hp_ctrl)
+		return -ENOMEM;
+
+	ret = octep_hp_controller_setup(pdev, hp_ctrl);
+	if (ret)
+		return ret;
+
+	/*
+	 * Register all hotplug slots. Hotplug controller is the first function
+	 * of the PCI device. The hotplug slots are the remaining functions of
+	 * the PCI device. The hotplug slot functions are logically removed from
+	 * the bus during probing and are re-enabled by the driver when a
+	 * hotplug event is received.
+	 */
+	list_for_each_entry_safe(tmp_pdev, next, &pdev->bus->devices, bus_list) {
+		if (tmp_pdev == pdev)
+			continue;
+
+		hp_slot = octep_hp_register_slot(hp_ctrl, tmp_pdev, slot_number);
+		if (IS_ERR(hp_slot))
+			return dev_err_probe(&pdev->dev, PTR_ERR(hp_slot),
+					     "Failed to register hotplug slot %u\n",
+					     slot_number);
+
+		ret = devm_add_action(&pdev->dev, octep_hp_deregister_slot,
+				      hp_slot);
+		if (ret)
+			return dev_err_probe(&pdev->dev, ret,
+					     "Failed to add action for deregistering slot %u\n",
+					     slot_number);
+		slot_number++;
+	}
+
+	return 0;
+}
+
+#define PCI_DEVICE_ID_CAVIUM_OCTEP_HP_CTLR  0xa0e3
+static struct pci_device_id octep_hp_pci_map[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_CAVIUM_OCTEP_HP_CTLR) },
+	{ },
+};
+
+static struct pci_driver octep_hp = {
+	.name = OCTEP_HP_DRV_NAME,
+	.id_table = octep_hp_pci_map,
+	.probe = octep_hp_pci_probe,
+};
+
+module_pci_driver(octep_hp);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Marvell");
+MODULE_DESCRIPTION("Marvell OCTEON PCI Hotplug driver");
-- 
2.25.1

Re: [PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Bjorn Helgaas]

On Mon, Nov 11, 2024 at 07:15:11PM +0530, Shijith Thotton wrote:
> This patch introduces a PCI hotplug controller driver for the OCTEON
> PCIe device. The OCTEON PCIe device is a multi-function device where the
> first function serves as the PCI hotplug controller.
> 
>                +--------------------------------+
>                |           Root Port            |
>                +--------------------------------+
>                                |
>                               PCIe
>                                |
> +---------------------------------------------------------------+
> |              OCTEON PCIe Multifunction Device                 |
> +---------------------------------------------------------------+
>              |                    |              |            |
>              |                    |              |            |
> +---------------------+  +----------------+  +-----+  +----------------+
> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
> +---------------------+  +----------------+  +-----+  +----------------+
>              |
>              |
> +-------------------------+
> |   Controller Firmware   |
> +-------------------------+
> 
> The hotplug controller driver enables hotplugging of non-controller
> functions within the same device. During probing, the driver removes
> the non-controller functions and registers them as PCI hotplug slots.
> These slots are added back by the driver, only upon request from the
> device firmware.
> 
> The controller uses MSI-X interrupts to notify the host of hotplug
> events initiated by the OCTEON firmware. Additionally, the driver
> allows users to enable or disable individual functions via sysfs slot
> entries, as provided by the PCI hotplug framework.

Can we say something here about what the benefit of this driver is?
For example, does it save power?

What causes the function 0 firmware to request a hot-add or
hot-removal of another function?

Bjorn

Re: [PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

>> This patch introduces a PCI hotplug controller driver for the OCTEON
>> PCIe device. The OCTEON PCIe device is a multi-function device where the
>> first function serves as the PCI hotplug controller.
>>
>>                +--------------------------------+
>>                |           Root Port            |
>>                +--------------------------------+
>>                                |
>>                               PCIe
>>                                |
>> +---------------------------------------------------------------+
>> |              OCTEON PCIe Multifunction Device                 |
>> +---------------------------------------------------------------+
>>              |                    |              |            |
>>              |                    |              |            |
>> +---------------------+  +----------------+  +-----+  +----------------+
>> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>> +---------------------+  +----------------+  +-----+  +----------------+
>>              |
>>              |
>> +-------------------------+
>> |   Controller Firmware   |
>> +-------------------------+
>>
>> The hotplug controller driver enables hotplugging of non-controller
>> functions within the same device. During probing, the driver removes
>> the non-controller functions and registers them as PCI hotplug slots.
>> These slots are added back by the driver, only upon request from the
>> device firmware.
>>
>> The controller uses MSI-X interrupts to notify the host of hotplug
>> events initiated by the OCTEON firmware. Additionally, the driver
>> allows users to enable or disable individual functions via sysfs slot
>> entries, as provided by the PCI hotplug framework.
>
>Can we say something here about what the benefit of this driver is?
>For example, does it save power?

The driver enables hotplugging of non-controller functions within the device
without requiring a fully implemented switch, reducing both power consumption
and product cost.

>What causes the function 0 firmware to request a hot-add or
>hot-removal of another function?

The firmware will enable the required number of non-controller functions based
on runtime demand, allowing control over these functions. For example, in a
vDPA scenario, each function could act as a different type of device (such as
net, crypto, or storage) depending on the firmware configuration.

Hot removal is useful in cases of live firmware updates.

I will update the commit log with above details.

Thanks,
Shijith

Re: [PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Bjorn Helgaas]

On Tue, Nov 12, 2024 at 09:25:46AM +0000, Shijith Thotton wrote:
> >> This patch introduces a PCI hotplug controller driver for the OCTEON
> >> PCIe device. The OCTEON PCIe device is a multi-function device where the
> >> first function serves as the PCI hotplug controller.
> >>
> >>                +--------------------------------+
> >>                |           Root Port            |
> >>                +--------------------------------+
> >>                                |
> >>                               PCIe
> >>                                |
> >> +---------------------------------------------------------------+
> >> |              OCTEON PCIe Multifunction Device                 |
> >> +---------------------------------------------------------------+
> >>              |                    |              |            |
> >>              |                    |              |            |
> >> +---------------------+  +----------------+  +-----+  +----------------+
> >> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
> >> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
> >> +---------------------+  +----------------+  +-----+  +----------------+
> >>              |
> >>              |
> >> +-------------------------+
> >> |   Controller Firmware   |
> >> +-------------------------+
> >>
> >> The hotplug controller driver enables hotplugging of non-controller
> >> functions within the same device. During probing, the driver removes
> >> the non-controller functions and registers them as PCI hotplug slots.
> >> These slots are added back by the driver, only upon request from the
> >> device firmware.
> >>
> >> The controller uses MSI-X interrupts to notify the host of hotplug
> >> events initiated by the OCTEON firmware. Additionally, the driver
> >> allows users to enable or disable individual functions via sysfs slot
> >> entries, as provided by the PCI hotplug framework.
> >
> >Can we say something here about what the benefit of this driver is?
> >For example, does it save power?
> 
> The driver enables hotplugging of non-controller functions within the device
> without requiring a fully implemented switch, reducing both power consumption
> and product cost.

Reduced product cost is motivation for the hardware design, not for
this hotplug driver.

You didn't explicitly say that when function 0 hot-removes another
function, it reduces overall power consumption.  But I assume that's
the case?

> >What causes the function 0 firmware to request a hot-add or
> >hot-removal of another function?
> 
> The firmware will enable the required number of non-controller
> functions based on runtime demand, allowing control over these
> functions. For example, in a vDPA scenario, each function could act
> as a different type of device (such as net, crypto, or storage)
> depending on the firmware configuration.

What is the path for this runtime demand?  I assume function 0
provides some interface to request a specific kind of functionality
(net, crypo, storage, etc)?

I don't know anything about vDPA, so if that's important here, it
needs a little more context.

> Hot removal is useful in cases of live firmware updates.

So the idea is that function X is hot-removed, which forces the driver
to let go of it, the firmware is updated, and X is hot-added again,
and the driver binds to it again?

And somewhere in there is a reset of function X, and after the reset
X is running the new firmware?

Who/what initiates this whole path?  Some request to function 0,
saying "please remove function X"?

But I guess maybe it doesn't go through function 0, since octeon_hp
claims function 0, and it doesn't provide that functionality.  Maybe
the individual drivers for *other* functions know how to initiate
these things, and those functions internally communicate with function
0 to ask it to start a hot-remove/hot-add sequence?

That wouldn't explain the power reduction plan, though.  A driver for
function X could conceivably tell its device "I'm no longer needed"
and function X could tell function 0 to remove it.  That might enable
some power savings.  But that doesn't have a path to *re-enable*
function X, since function X has been removed and there's no driver to
ask for it to be hot-added again.

Maybe there's some out-of-band management path that can tell function
0 to do things, independent of PCIe?

So confused,
  Bjorn

Re: [PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

>> >> This patch introduces a PCI hotplug controller driver for the OCTEON
>> >> PCIe device. The OCTEON PCIe device is a multi-function device where the
>> >> first function serves as the PCI hotplug controller.
>> >>
>> >>                +--------------------------------+
>> >>                |           Root Port            |
>> >>                +--------------------------------+
>> >>                                |
>> >>                               PCIe
>> >>                                |
>> >> +---------------------------------------------------------------+
>> >> |              OCTEON PCIe Multifunction Device                 |
>> >> +---------------------------------------------------------------+
>> >>              |                    |              |            |
>> >>              |                    |              |            |
>> >> +---------------------+  +----------------+  +-----+  +----------------+
>> >> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>> >> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>> >> +---------------------+  +----------------+  +-----+  +----------------+
>> >>              |
>> >>              |
>> >> +-------------------------+
>> >> |   Controller Firmware   |
>> >> +-------------------------+
>> >>
>> >> The hotplug controller driver enables hotplugging of non-controller
>> >> functions within the same device. During probing, the driver removes
>> >> the non-controller functions and registers them as PCI hotplug slots.
>> >> These slots are added back by the driver, only upon request from the
>> >> device firmware.
>> >>
>> >> The controller uses MSI-X interrupts to notify the host of hotplug
>> >> events initiated by the OCTEON firmware. Additionally, the driver
>> >> allows users to enable or disable individual functions via sysfs slot
>> >> entries, as provided by the PCI hotplug framework.
>> >
>> >Can we say something here about what the benefit of this driver is?
>> >For example, does it save power?
>>
>> The driver enables hotplugging of non-controller functions within the device
>> without requiring a fully implemented switch, reducing both power
>consumption
>> and product cost.
>
>Reduced product cost is motivation for the hardware design, not for
>this hotplug driver.
>
>You didn't explicitly say that when function 0 hot-removes another
>function, it reduces overall power consumption.  But I assume that's
>the case?
>

Yes, I will explain it in detail below

>> >What causes the function 0 firmware to request a hot-add or
>> >hot-removal of another function?
>>
>> The firmware will enable the required number of non-controller
>> functions based on runtime demand, allowing control over these
>> functions. For example, in a vDPA scenario, each function could act
>> as a different type of device (such as net, crypto, or storage)
>> depending on the firmware configuration.
>
>What is the path for this runtime demand?  I assume function 0
>provides some interface to request a specific kind of functionality
>(net, crypo, storage, etc)?
>

Right now, it done via firmware management console.

>I don't know anything about vDPA, so if that's important here, it
>needs a little more context.
>
>> Hot removal is useful in cases of live firmware updates.
>
>So the idea is that function X is hot-removed, which forces the driver
>to let go of it, the firmware is updated, and X is hot-added again,
>and the driver binds to it again?
>

I will explain the process in detail, which should also address the questions
below.

>And somewhere in there is a reset of function X, and after the reset
>X is running the new firmware?
>
>Who/what initiates this whole path?  Some request to function 0,
>saying "please remove function X"?
>
>But I guess maybe it doesn't go through function 0, since octeon_hp
>claims function 0, and it doesn't provide that functionality.  Maybe
>the individual drivers for *other* functions know how to initiate
>these things, and those functions internally communicate with function
>0 to ask it to start a hot-remove/hot-add sequence?
>
>That wouldn't explain the power reduction plan, though.  A driver for
>function X could conceivably tell its device "I'm no longer needed"
>and function X could tell function 0 to remove it.  That might enable
>some power savings.  But that doesn't have a path to *re-enable*
>function X, since function X has been removed and there's no driver to
>ask for it to be hot-added again.
>
>Maybe there's some out-of-band management path that can tell function
>0 to do things, independent of PCIe?
>

Our implementation aims to achieve two main objectives:

1. Enable changing a function's personality at runtime.
2. Reduce power consumption.

The OCTEON PCI device has multiple ARM cores running Linux, with its firmware
composed of multiple components. For example, the firmware includes components
like Virtio-net, NVMe, and Virtio-Crypto, which can be assigned to any function
at runtime. The device firmware is accessible via a management console, allowing
components to be started or stopped. For each component, an associated function
is hot-added on the host to expose its functionality. Initially, after boot, only
Function 0 and the controller firmware are active.

Here's a breakdown:

At Time 0:
- Linux boots on the device, starting the controller firmware.

At Time 1:
- The hotplug driver loads on the host, temporarily removing other functions.

At Time 2:
- A network device firmware component starts on an ARM core (initiated through
  a console command).
- This component sets up the Function 1 configuration space, data, and other
  request handlers for network processing.
- The firmware issues a hot-add request to Function 0 (hotplug driver) on the
  host to enable Function 1.

At Time 3:
- The Function 0 hotplug driver on the host receives the hot-add request and
  enables Function 1 on the host.
- A network driver binds to Function 1 based on device class and ID.

At Time 4:
- The network device firmware component receives a stop signal.
- The firmware issues a hot-remove request for Function 1 on the host.
- The firmware component halts, reducing the device's power consumption.

At Time 5:
- The Function 0 hotplug driver on the host receives the hot-remove request and
  disables Function 1 on the host.

At Time 6:
- A crypto device firmware component starts on an ARM core.
- This component configures the Function 1 configuration space for crypto
  processing and sets up the required firmware handlers.
- The firmware issues a hot-add request to enable Function 1 on the host.

At Time 7:
- The Function 0 hotplug driver on the host receives the hot-add request and                                                                                                                                                                                                      enables Function 1 on the host.
- A crypto driver binds to Function 1 based on device class and ID.

The firmware component for each function only runs and is hot-added when
needed. Only Function 0 and the controller firmware remain active
continuously. This dynamic control reduces power usage by keeping unnecessary
components off. Additionally, a single function can adapt its personality based
on the associated firmware component, enhancing flexibility. 
                                                                                                                                                                                                                   
I hope this clarifies the implementation. Let me know if you have any
questions.

Thanks,
Shijith

Re: [PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Bjorn Helgaas]

On Wed, Nov 13, 2024 at 12:20:20PM +0000, Shijith Thotton wrote:
> >> >> This patch introduces a PCI hotplug controller driver for the OCTEON
> >> >> PCIe device. The OCTEON PCIe device is a multi-function device where the
> >> >> first function serves as the PCI hotplug controller.
> >> >>
> >> >>                +--------------------------------+
> >> >>                |           Root Port            |
> >> >>                +--------------------------------+
> >> >>                                |
> >> >>                               PCIe
> >> >>                                |
> >> >> +---------------------------------------------------------------+
> >> >> |              OCTEON PCIe Multifunction Device                 |
> >> >> +---------------------------------------------------------------+
> >> >>              |                    |              |            |
> >> >>              |                    |              |            |
> >> >> +---------------------+  +----------------+  +-----+  +----------------+
> >> >> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
> >> >> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
> >> >> +---------------------+  +----------------+  +-----+  +----------------+
> >> >>              |
> >> >>              |
> >> >> +-------------------------+
> >> >> |   Controller Firmware   |
> >> >> +-------------------------+
> >> >>
> >> >> The hotplug controller driver enables hotplugging of non-controller
> >> >> functions within the same device. During probing, the driver removes
> >> >> the non-controller functions and registers them as PCI hotplug slots.
> >> >> These slots are added back by the driver, only upon request from the
> >> >> device firmware.
> >> >>
> >> >> The controller uses MSI-X interrupts to notify the host of hotplug
> >> >> events initiated by the OCTEON firmware. Additionally, the driver
> >> >> allows users to enable or disable individual functions via sysfs slot
> >> >> entries, as provided by the PCI hotplug framework.
> >> >
> >> >Can we say something here about what the benefit of this driver is?
> >> >For example, does it save power?
> >>
> >> The driver enables hotplugging of non-controller functions within the device
> >> without requiring a fully implemented switch, reducing both power
> >consumption
> >> and product cost.
> >
> >Reduced product cost is motivation for the hardware design, not for
> >this hotplug driver.
> >
> >You didn't explicitly say that when function 0 hot-removes another
> >function, it reduces overall power consumption.  But I assume that's
> >the case?
> >
> 
> Yes, I will explain it in detail below
> 
> >> >What causes the function 0 firmware to request a hot-add or
> >> >hot-removal of another function?
> >>
> >> The firmware will enable the required number of non-controller
> >> functions based on runtime demand, allowing control over these
> >> functions. For example, in a vDPA scenario, each function could act
> >> as a different type of device (such as net, crypto, or storage)
> >> depending on the firmware configuration.
> >
> >What is the path for this runtime demand?  I assume function 0
> >provides some interface to request a specific kind of functionality
> >(net, crypo, storage, etc)?
> >
> 
> Right now, it done via firmware management console.
> 
> >I don't know anything about vDPA, so if that's important here, it
> >needs a little more context.
> >
> >> Hot removal is useful in cases of live firmware updates.
> >
> >So the idea is that function X is hot-removed, which forces the driver
> >to let go of it, the firmware is updated, and X is hot-added again,
> >and the driver binds to it again?
> >
> 
> I will explain the process in detail, which should also address the questions
> below.
> 
> >And somewhere in there is a reset of function X, and after the reset
> >X is running the new firmware?
> >
> >Who/what initiates this whole path?  Some request to function 0,
> >saying "please remove function X"?
> >
> >But I guess maybe it doesn't go through function 0, since octeon_hp
> >claims function 0, and it doesn't provide that functionality.  Maybe
> >the individual drivers for *other* functions know how to initiate
> >these things, and those functions internally communicate with function
> >0 to ask it to start a hot-remove/hot-add sequence?
> >
> >That wouldn't explain the power reduction plan, though.  A driver for
> >function X could conceivably tell its device "I'm no longer needed"
> >and function X could tell function 0 to remove it.  That might enable
> >some power savings.  But that doesn't have a path to *re-enable*
> >function X, since function X has been removed and there's no driver to
> >ask for it to be hot-added again.
> >
> >Maybe there's some out-of-band management path that can tell function
> >0 to do things, independent of PCIe?
> >
> 
> Our implementation aims to achieve two main objectives:
> 
> 1. Enable changing a function's personality at runtime.
> 2. Reduce power consumption.
> 
> The OCTEON PCI device has multiple ARM cores running Linux, with its firmware
> composed of multiple components. For example, the firmware includes components
> like Virtio-net, NVMe, and Virtio-Crypto, which can be assigned to any function
> at runtime. The device firmware is accessible via a management console, allowing
> components to be started or stopped. For each component, an associated function
> is hot-added on the host to expose its functionality. Initially, after boot, only
> Function 0 and the controller firmware are active.
> 
> Here's a breakdown:
> 
> At Time 0:
> - Linux boots on the device, starting the controller firmware.
> 
> At Time 1:
> - The hotplug driver loads on the host, temporarily removing other functions.
> 
> At Time 2:
> - A network device firmware component starts on an ARM core (initiated through
>   a console command).
> - This component sets up the Function 1 configuration space, data, and other
>   request handlers for network processing.
> - The firmware issues a hot-add request to Function 0 (hotplug driver) on the
>   host to enable Function 1.
> 
> At Time 3:
> - The Function 0 hotplug driver on the host receives the hot-add request and
>   enables Function 1 on the host.
> - A network driver binds to Function 1 based on device class and ID.
> 
> At Time 4:
> - The network device firmware component receives a stop signal.
> - The firmware issues a hot-remove request for Function 1 on the host.
> - The firmware component halts, reducing the device's power consumption.
> 
> At Time 5:
> - The Function 0 hotplug driver on the host receives the hot-remove request and
>   disables Function 1 on the host.
> 
> At Time 6:
> - A crypto device firmware component starts on an ARM core.
> - This component configures the Function 1 configuration space for crypto
>   processing and sets up the required firmware handlers.
> - The firmware issues a hot-add request to enable Function 1 on the host.
> 
> At Time 7:
> - The Function 0 hotplug driver on the host receives the hot-add request and                                                                                                                                                                                                      enables Function 1 on the host.
> - A crypto driver binds to Function 1 based on device class and ID.
> 
> The firmware component for each function only runs and is hot-added when
> needed. Only Function 0 and the controller firmware remain active
> continuously. This dynamic control reduces power usage by keeping unnecessary
> components off. Additionally, a single function can adapt its personality based
> on the associated firmware component, enhancing flexibility. 
>                                                                                                                                                                                                                    
> I hope this clarifies the implementation. Let me know if you have any
> questions.

Thanks very much!  I propose adding text like this to the commit log:

  There is an out-of-band management console interface to firmware
  running on function 0 whereby an administrator can disable functions
  to save power or enable them with one of several personalities
  (virtio-net, virtio-crypto, NVMe, etc) for the other functions.
  Function 0 initiates hotplug events handled by this driver when the
  other functions are enabled or disabled.

I provisionally applied this to pci/hotplug-octeon, but will be happy
to update the text if necessary.

Bjorn

Re: [PATCH v4] PCI: hotplug: Add OCTEON PCI hotplug controller driver

[Shijith Thotton]

>> >> >> This patch introduces a PCI hotplug controller driver for the OCTEON
>> >> >> PCIe device. The OCTEON PCIe device is a multi-function device where
>the
>> >> >> first function serves as the PCI hotplug controller.
>> >> >>
>> >> >>                +--------------------------------+
>> >> >>                |           Root Port            |
>> >> >>                +--------------------------------+
>> >> >>                                |
>> >> >>                               PCIe
>> >> >>                                |
>> >> >> +---------------------------------------------------------------+
>> >> >> |              OCTEON PCIe Multifunction Device                 |
>> >> >> +---------------------------------------------------------------+
>> >> >>              |                    |              |            |
>> >> >>              |                    |              |            |
>> >> >> +---------------------+  +----------------+  +-----+  +----------------+
>> >> >> |      Function 0     |  |   Function 1   |  | ... |  |   Function 7   |
>> >> >> | (Hotplug controller)|  | (Hotplug slot) |  |     |  | (Hotplug slot) |
>> >> >> +---------------------+  +----------------+  +-----+  +----------------+
>> >> >>              |
>> >> >>              |
>> >> >> +-------------------------+
>> >> >> |   Controller Firmware   |
>> >> >> +-------------------------+
>> >> >>
>> >> >> The hotplug controller driver enables hotplugging of non-controller
>> >> >> functions within the same device. During probing, the driver removes
>> >> >> the non-controller functions and registers them as PCI hotplug slots.
>> >> >> These slots are added back by the driver, only upon request from the
>> >> >> device firmware.
>> >> >>
>> >> >> The controller uses MSI-X interrupts to notify the host of hotplug
>> >> >> events initiated by the OCTEON firmware. Additionally, the driver
>> >> >> allows users to enable or disable individual functions via sysfs slot
>> >> >> entries, as provided by the PCI hotplug framework.
>> >> >
>> >> >Can we say something here about what the benefit of this driver is?
>> >> >For example, does it save power?
>> >>
>> >> The driver enables hotplugging of non-controller functions within the
>device
>> >> without requiring a fully implemented switch, reducing both power
>> >consumption
>> >> and product cost.
>> >
>> >Reduced product cost is motivation for the hardware design, not for
>> >this hotplug driver.
>> >
>> >You didn't explicitly say that when function 0 hot-removes another
>> >function, it reduces overall power consumption.  But I assume that's
>> >the case?
>> >
>>
>> Yes, I will explain it in detail below
>>
>> >> >What causes the function 0 firmware to request a hot-add or
>> >> >hot-removal of another function?
>> >>
>> >> The firmware will enable the required number of non-controller
>> >> functions based on runtime demand, allowing control over these
>> >> functions. For example, in a vDPA scenario, each function could act
>> >> as a different type of device (such as net, crypto, or storage)
>> >> depending on the firmware configuration.
>> >
>> >What is the path for this runtime demand?  I assume function 0
>> >provides some interface to request a specific kind of functionality
>> >(net, crypo, storage, etc)?
>> >
>>
>> Right now, it done via firmware management console.
>>
>> >I don't know anything about vDPA, so if that's important here, it
>> >needs a little more context.
>> >
>> >> Hot removal is useful in cases of live firmware updates.
>> >
>> >So the idea is that function X is hot-removed, which forces the driver
>> >to let go of it, the firmware is updated, and X is hot-added again,
>> >and the driver binds to it again?
>> >
>>
>> I will explain the process in detail, which should also address the questions
>> below.
>>
>> >And somewhere in there is a reset of function X, and after the reset
>> >X is running the new firmware?
>> >
>> >Who/what initiates this whole path?  Some request to function 0,
>> >saying "please remove function X"?
>> >
>> >But I guess maybe it doesn't go through function 0, since octeon_hp
>> >claims function 0, and it doesn't provide that functionality.  Maybe
>> >the individual drivers for *other* functions know how to initiate
>> >these things, and those functions internally communicate with function
>> >0 to ask it to start a hot-remove/hot-add sequence?
>> >
>> >That wouldn't explain the power reduction plan, though.  A driver for
>> >function X could conceivably tell its device "I'm no longer needed"
>> >and function X could tell function 0 to remove it.  That might enable
>> >some power savings.  But that doesn't have a path to *re-enable*
>> >function X, since function X has been removed and there's no driver to
>> >ask for it to be hot-added again.
>> >
>> >Maybe there's some out-of-band management path that can tell function
>> >0 to do things, independent of PCIe?
>> >
>>
>> Our implementation aims to achieve two main objectives:
>>
>> 1. Enable changing a function's personality at runtime.
>> 2. Reduce power consumption.
>>
>> The OCTEON PCI device has multiple ARM cores running Linux, with its
>firmware
>> composed of multiple components. For example, the firmware includes
>components
>> like Virtio-net, NVMe, and Virtio-Crypto, which can be assigned to any
>function
>> at runtime. The device firmware is accessible via a management console,
>allowing
>> components to be started or stopped. For each component, an associated
>function
>> is hot-added on the host to expose its functionality. Initially, after boot, only
>> Function 0 and the controller firmware are active.
>>
>> Here's a breakdown:
>>
>> At Time 0:
>> - Linux boots on the device, starting the controller firmware.
>>
>> At Time 1:
>> - The hotplug driver loads on the host, temporarily removing other functions.
>>
>> At Time 2:
>> - A network device firmware component starts on an ARM core (initiated
>through
>>   a console command).
>> - This component sets up the Function 1 configuration space, data, and other
>>   request handlers for network processing.
>> - The firmware issues a hot-add request to Function 0 (hotplug driver) on the
>>   host to enable Function 1.
>>
>> At Time 3:
>> - The Function 0 hotplug driver on the host receives the hot-add request and
>>   enables Function 1 on the host.
>> - A network driver binds to Function 1 based on device class and ID.
>>
>> At Time 4:
>> - The network device firmware component receives a stop signal.
>> - The firmware issues a hot-remove request for Function 1 on the host.
>> - The firmware component halts, reducing the device's power consumption.
>>
>> At Time 5:
>> - The Function 0 hotplug driver on the host receives the hot-remove request
>and
>>   disables Function 1 on the host.
>>
>> At Time 6:
>> - A crypto device firmware component starts on an ARM core.
>> - This component configures the Function 1 configuration space for crypto
>>   processing and sets up the required firmware handlers.
>> - The firmware issues a hot-add request to enable Function 1 on the host.
>>
>> At Time 7:
>> - The Function 0 hotplug driver on the host receives the hot-add request and
>enables Function 1 on the host.
>> - A crypto driver binds to Function 1 based on device class and ID.
>>
>> The firmware component for each function only runs and is hot-added when
>> needed. Only Function 0 and the controller firmware remain active
>> continuously. This dynamic control reduces power usage by keeping
>unnecessary
>> components off. Additionally, a single function can adapt its personality based
>> on the associated firmware component, enhancing flexibility.
>>
>> I hope this clarifies the implementation. Let me know if you have any
>> questions.
>
>Thanks very much!  I propose adding text like this to the commit log:
>
>  There is an out-of-band management console interface to firmware
>  running on function 0 whereby an administrator can disable functions
>  to save power or enable them with one of several personalities
>  (virtio-net, virtio-crypto, NVMe, etc) for the other functions.
>  Function 0 initiates hotplug events handled by this driver when the
>  other functions are enabled or disabled.
>
>I provisionally applied this to pci/hotplug-octeon, but will be happy
>to update the text if necessary.
>

Thank you. The commit log looks good. I hope no further action is required from
my side.

Shijith