[PATCH v3 0/2] PCI: mvebu: Dispose INTx IRQs before to removing INTx domain

[PATCH v3 0/2] PCI: mvebu: Dispose INTx IRQs before to removing INTx domain

[Marek Behún]

Hello Bjorn, Thomas, Ilpo et al.

I've split the patch into two: the first introduces the new helper
function, pci_remove_irq_domain(), and the second uses this new function
instead of irq_domain_remove() in the pci-mvebu controller driver.

Marek

Marek Behún (1):
  PCI: Add pci_remove_irq_domain() helper

Pali Rohár (1):
  PCI: mvebu: Dispose INTx IRQs before to removing INTx domain

 drivers/pci/controller/pci-mvebu.c |  2 +-
 drivers/pci/irq.c                  | 21 +++++++++++++++++++++
 drivers/pci/pci.h                  |  7 +++++++
 3 files changed, 29 insertions(+), 1 deletion(-)

-- 
2.44.2

[PATCH v3 1/2] PCI: Add pci_remove_irq_domain() helper

[Marek Behún]

Add a helper function pci_remove_irq_domain() for disposing all
interrupt mappings of an IRQ domain and then removing said IRQ domain.

As explained in the attached link, the PCI INTX interrupt may be shared,
and so the PCI device drivers do not dispose mapped interrupts when they
are unbound from a device, since other devices may be still using those
mapped interrupts. Thus the interrupts must be disposed by the PCI
controller driver when the IRQ domain is being removed.

This function may be used by PCI controller drivers that wish to be
removable / modular.

Link: https://lore.kernel.org/linux-pci/878qy5rrq7.ffs@tglx/
Signed-off-by: Marek Behún <kabel@kernel.org>
---
 drivers/pci/irq.c | 21 +++++++++++++++++++++
 drivers/pci/pci.h |  7 +++++++
 2 files changed, 28 insertions(+)

diff --git a/drivers/pci/irq.c b/drivers/pci/irq.c
index 4555630be9ec..30c8d930016a 100644
--- a/drivers/pci/irq.c
+++ b/drivers/pci/irq.c
@@ -11,6 +11,7 @@
 #include <linux/errno.h>
 #include <linux/export.h>
 #include <linux/interrupt.h>
+#include <linux/irqdomain.h>
 #include <linux/pci.h>
 
 #include "pci.h"
@@ -259,6 +260,26 @@ bool pci_check_and_unmask_intx(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(pci_check_and_unmask_intx);
 
+#ifdef CONFIG_IRQ_DOMAIN
+/**
+ * pci_remove_irq_domain - dispose all IRQ mappings and remove IRQ domain
+ * @domain: the IRQ domain to be removed
+ *
+ * Disposes all IRQ mappings of a given IRQ domain before removing the domain.
+ */
+void pci_remove_irq_domain(struct irq_domain *domain)
+{
+	for (irq_hw_number_t i = 0; i < domain->hwirq_max; i++) {
+		unsigned int virq = irq_find_mapping(domain, i);
+
+		if (virq)
+			irq_dispose_mapping(virq);
+	}
+
+	irq_domain_remove(domain);
+}
+#endif
+
 /**
  * pcibios_penalize_isa_irq - penalize an ISA IRQ
  * @irq: ISA IRQ to penalize
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index fd44565c4756..1ba6a6f418ac 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -170,6 +170,13 @@ void pci_no_msi(void);
 static inline void pci_no_msi(void) { }
 #endif
 
+struct irq_domain;
+#ifdef CONFIG_IRQ_DOMAIN
+void pci_remove_irq_domain(struct irq_domain *domain);
+#else
+static inline void pci_remove_irq_domain(struct irq_domain *domain) { }
+#endif
+
 void pci_realloc_get_opt(char *);
 
 static inline int pci_no_d1d2(struct pci_dev *dev)
-- 
2.44.2

Re: [PATCH v3 1/2] PCI: Add pci_remove_irq_domain() helper

[Manivannan Sadhasivam]

On Mon, Jul 15, 2024 at 01:48:53PM +0200, Marek Behún wrote:
> Add a helper function pci_remove_irq_domain() for disposing all
> interrupt mappings of an IRQ domain and then removing said IRQ domain.
> 
> As explained in the attached link, the PCI INTX interrupt may be shared,
> and so the PCI device drivers do not dispose mapped interrupts when they
> are unbound from a device, since other devices may be still using those
> mapped interrupts. Thus the interrupts must be disposed by the PCI
> controller driver when the IRQ domain is being removed.
> 
> This function may be used by PCI controller drivers that wish to be
> removable / modular.
> 
> Link: https://lore.kernel.org/linux-pci/878qy5rrq7.ffs@tglx/
> Signed-off-by: Marek Behún <kabel@kernel.org>

Thomas shared the diff leading to this patch. Shouldn't you give some credit to
him?

For the patch though,

Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>

- Mani

> ---
>  drivers/pci/irq.c | 21 +++++++++++++++++++++
>  drivers/pci/pci.h |  7 +++++++
>  2 files changed, 28 insertions(+)
> 
> diff --git a/drivers/pci/irq.c b/drivers/pci/irq.c
> index 4555630be9ec..30c8d930016a 100644
> --- a/drivers/pci/irq.c
> +++ b/drivers/pci/irq.c
> @@ -11,6 +11,7 @@
>  #include <linux/errno.h>
>  #include <linux/export.h>
>  #include <linux/interrupt.h>
> +#include <linux/irqdomain.h>
>  #include <linux/pci.h>
>  
>  #include "pci.h"
> @@ -259,6 +260,26 @@ bool pci_check_and_unmask_intx(struct pci_dev *dev)
>  }
>  EXPORT_SYMBOL_GPL(pci_check_and_unmask_intx);
>  
> +#ifdef CONFIG_IRQ_DOMAIN
> +/**
> + * pci_remove_irq_domain - dispose all IRQ mappings and remove IRQ domain
> + * @domain: the IRQ domain to be removed
> + *
> + * Disposes all IRQ mappings of a given IRQ domain before removing the domain.
> + */
> +void pci_remove_irq_domain(struct irq_domain *domain)
> +{
> +	for (irq_hw_number_t i = 0; i < domain->hwirq_max; i++) {
> +		unsigned int virq = irq_find_mapping(domain, i);
> +
> +		if (virq)
> +			irq_dispose_mapping(virq);
> +	}
> +
> +	irq_domain_remove(domain);
> +}
> +#endif
> +
>  /**
>   * pcibios_penalize_isa_irq - penalize an ISA IRQ
>   * @irq: ISA IRQ to penalize
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index fd44565c4756..1ba6a6f418ac 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -170,6 +170,13 @@ void pci_no_msi(void);
>  static inline void pci_no_msi(void) { }
>  #endif
>  
> +struct irq_domain;
> +#ifdef CONFIG_IRQ_DOMAIN
> +void pci_remove_irq_domain(struct irq_domain *domain);
> +#else
> +static inline void pci_remove_irq_domain(struct irq_domain *domain) { }
> +#endif
> +
>  void pci_realloc_get_opt(char *);
>  
>  static inline int pci_no_d1d2(struct pci_dev *dev)
> -- 
> 2.44.2
> 

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v3 1/2] PCI: Add pci_remove_irq_domain() helper

[Thomas Gleixner]

On Mon, Jul 15 2024 at 13:48, Marek Behún wrote:
> +#ifdef CONFIG_IRQ_DOMAIN
> +/**
> + * pci_remove_irq_domain - dispose all IRQ mappings and remove IRQ domain
> + * @domain: the IRQ domain to be removed
> + *
> + * Disposes all IRQ mappings of a given IRQ domain before removing
> the domain.

Sure, but this lacks information what this is about and where this
should be used.

Thanks,

        tglx

[PATCH v3 2/2] PCI: mvebu: Dispose INTx IRQs before to removing INTx domain

[Marek Behún]

From: Pali Rohár <pali@kernel.org>

The documentation for the irq_domain_remove() function says that all
mappings within the IRQ domain must be disposed before the domain is
removed.

Use the new helper function pci_remove_irq_domain() that also disposes
all IRQ mappings of a IRQ domain before removing sait domain.

Currently, the INTx IRQs are not disposed in pci-mvebu driver .remove()
method, which causes the kernel to crash when unloading the driver and
then reading /sys/kernel/debug/irq/irqs/<num> or /proc/interrupts.

Unmapping of the IRQs at this point of the .remove() method is safe,
since the PCIe bus is already unregistered, and all its devices are
unbound from their drivers and removed. If there was indeed any
remaining use of PCIe resources, then it would mean that PCIe hotplug
code is broken, and we have bigger problems.

Fixes: ec075262648f ("PCI: mvebu: Implement support for legacy INTx interrupts")
Reported-by: Hajo Noerenberg <hajo-linux-bugzilla@noerenberg.de>
Signed-off-by: Pali Rohár <pali@kernel.org>
Reviewed-by: Marek Behún <kabel@kernel.org>
[ Marek: refactored a little, added more explanation to commit message ]
Signed-off-by: Marek Behún <kabel@kernel.org>
Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
 drivers/pci/controller/pci-mvebu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/pci/controller/pci-mvebu.c b/drivers/pci/controller/pci-mvebu.c
index 29fe09c99e7d..590f121bd91a 100644
--- a/drivers/pci/controller/pci-mvebu.c
+++ b/drivers/pci/controller/pci-mvebu.c
@@ -1684,7 +1684,7 @@ static void mvebu_pcie_remove(struct platform_device *pdev)
 
 		/* Remove IRQ domains. */
 		if (port->intx_irq_domain)
-			irq_domain_remove(port->intx_irq_domain);
+			pci_remove_irq_domain(port->intx_irq_domain);
 
 		/* Free config space for emulated root bridge. */
 		pci_bridge_emul_cleanup(&port->bridge);
-- 
2.44.2