Re:Re: [PATCH v11 12/14] irqchip/riscv-aplic: Add support for MSI-mode

From: Ben  <figure1802@126.com>
To: "Anup Patel" <apatel@ventanamicro.com>
Cc: "Palmer Dabbelt" <palmer@dabbelt.com>,
	"Paul Walmsley" <paul.walmsley@sifive.com>,
	"Thomas Gleixner" <tglx@linutronix.de>,
	"Rob Herring" <robh+dt@kernel.org>,
	"Krzysztof Kozlowski" <krzysztof.kozlowski+dt@linaro.org>,
	"Frank Rowand" <frowand.list@gmail.com>,
	"Conor Dooley" <conor+dt@kernel.org>,
	devicetree@vger.kernel.org,
	"Saravana Kannan" <saravanak@google.com>,
	"Marc Zyngier" <maz@kernel.org>,
	"Anup Patel" <anup@brainfault.org>,
	linux-kernel@vger.kernel.org, "Björn Töpel" <bjorn@kernel.org>,
	"Atish Patra" <atishp@atishpatra.org>,
	linux-riscv@lists.infradead.org,
	"Andrew Jones" <ajones@ventanamicro.com>
Subject: Re:Re: [PATCH v11 12/14] irqchip/riscv-aplic: Add support for MSI-mode
Date: Fri, 3 Nov 2023 17:39:56 +0800 (CST)	[thread overview]
Message-ID: <15813ba.5290.18b948db497.Coremail.figure1802@126.com> (raw)
In-Reply-To: <CAK9=C2UsExEDz76dr=gF2nxyF_3p5OtWxC7L8vZuK5s1nbiSoQ@mail.gmail.com>

在 2023-11-02 20:37:42，"Anup Patel" <apatel@ventanamicro.com> 写道：
>On Thu, Nov 2, 2023 at 11:55 AM Ben <figure1802@126.com> wrote:
>>
>>
>> At 2023-10-24 01:27:58, "Anup Patel" <apatel@ventanamicro.com> wrote:
>> >The RISC-V advanced platform-level interrupt controller (APLIC) has
>> >two modes of operation: 1) Direct mode and 2) MSI mode.
>> >(For more details, refer https://github.com/riscv/riscv-aia)
>> >
>> >In APLIC MSI-mode, wired interrupts are forwared as message signaled
>> >interrupts (MSIs) to CPUs via IMSIC.
>> >
>> >We extend the existing APLIC irqchip driver to support MSI-mode for
>> >RISC-V platforms having both wired interrupts and MSIs.
>> >
>> >Signed-off-by: Anup Patel <apatel@ventanamicro.com>
>> >---
>> > drivers/irqchip/Kconfig                |   6 +
>> > drivers/irqchip/Makefile               |   1 +
>> > drivers/irqchip/irq-riscv-aplic-main.c |   2 +-
>> > drivers/irqchip/irq-riscv-aplic-main.h |   8 +
>> > drivers/irqchip/irq-riscv-aplic-msi.c  | 285 +++++++++++++++++++++++++
>> > 5 files changed, 301 insertions(+), 1 deletion(-)
>> > create mode 100644 drivers/irqchip/irq-riscv-aplic-msi.c
>> >
>> >diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
>> >index 1996cc6f666a..7adc4dbe07ff 100644
>> >--- a/drivers/irqchip/Kconfig
>> >+++ b/drivers/irqchip/Kconfig
>> >@@ -551,6 +551,12 @@ config RISCV_APLIC
>> > depends on RISCV
>> > select IRQ_DOMAIN_HIERARCHY
>> >
>> >+config RISCV_APLIC_MSI
>> >+ bool
>> >+ depends on RISCV_APLIC
>> >+ select GENERIC_MSI_IRQ
>> >+ default RISCV_APLIC
>> >+
>> > config RISCV_IMSIC
>> > bool
>> > depends on RISCV
>> >diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
>> >index 7f8289790ed8..47995fdb2c60 100644
>> >--- a/drivers/irqchip/Makefile
>> >+++ b/drivers/irqchip/Makefile
>> >@@ -96,6 +96,7 @@ obj-$(CONFIG_CSKY_MPINTC) += irq-csky-mpintc.o
>> > obj-$(CONFIG_CSKY_APB_INTC) += irq-csky-apb-intc.o
>> > obj-$(CONFIG_RISCV_INTC) += irq-riscv-intc.o
>> > obj-$(CONFIG_RISCV_APLIC) += irq-riscv-aplic-main.o irq-riscv-aplic-direct.o
>> >+obj-$(CONFIG_RISCV_APLIC_MSI) += irq-riscv-aplic-msi.o
>> > obj-$(CONFIG_RISCV_IMSIC) += irq-riscv-imsic-state.o irq-riscv-imsic-early.o irq-riscv-imsic-platform.o
>> > obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
>> > obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
>> >diff --git a/drivers/irqchip/irq-riscv-aplic-main.c b/drivers/irqchip/irq-riscv-aplic-main.c
>> >index 87450708a733..d1b342b66551 100644
>> >--- a/drivers/irqchip/irq-riscv-aplic-main.c
>> >+++ b/drivers/irqchip/irq-riscv-aplic-main.c
>> >@@ -205,7 +205,7 @@ static int aplic_probe(struct platform_device *pdev)
>> > msi_mode = of_property_present(to_of_node(dev->fwnode),
>> > "msi-parent");
>> > if (msi_mode)
>> >- rc = -ENODEV;
>> >+ rc = aplic_msi_setup(dev, regs);
>> > else
>> > rc = aplic_direct_setup(dev, regs);
>> > if (rc) {
>> >diff --git a/drivers/irqchip/irq-riscv-aplic-main.h b/drivers/irqchip/irq-riscv-aplic-main.h
>> >index 474a04229334..78267ec58098 100644
>> >--- a/drivers/irqchip/irq-riscv-aplic-main.h
>> >+++ b/drivers/irqchip/irq-riscv-aplic-main.h
>> >@@ -41,5 +41,13 @@ void aplic_init_hw_global(struct aplic_priv *priv, bool msi_mode);
>> > int aplic_setup_priv(struct aplic_priv *priv, struct device *dev,
>> >     void __iomem *regs);
>> > int aplic_direct_setup(struct device *dev, void __iomem *regs);
>> >+#ifdef CONFIG_RISCV_APLIC_MSI
>> >+int aplic_msi_setup(struct device *dev, void __iomem *regs);
>> >+#else
>> >+static inline int aplic_msi_setup(struct device *dev, void __iomem *regs)
>> >+{
>> >+ return -ENODEV;
>> >+}
>> >+#endif
>> >
>> > #endif
>> >diff --git a/drivers/irqchip/irq-riscv-aplic-msi.c b/drivers/irqchip/irq-riscv-aplic-msi.c
>> >new file mode 100644
>> >index 000000000000..086d00e0429e
>> >--- /dev/null
>> >+++ b/drivers/irqchip/irq-riscv-aplic-msi.c
>> >@@ -0,0 +1,285 @@
>> >+// SPDX-License-Identifier: GPL-2.0
>> >+/*
>> >+ * Copyright (C) 2021 Western Digital Corporation or its affiliates.
>> >+ * Copyright (C) 2022 Ventana Micro Systems Inc.
>> >+ */
>> >+
>> >+#include <linux/bitops.h>
>> >+#include <linux/cpu.h>
>> >+#include <linux/interrupt.h>
>> >+#include <linux/irqchip.h>
>> >+#include <linux/irqchip/riscv-aplic.h>
>> >+#include <linux/irqchip/riscv-imsic.h>
>> >+#include <linux/module.h>
>> >+#include <linux/msi.h>
>> >+#include <linux/of_irq.h>
>> >+#include <linux/platform_device.h>
>> >+#include <linux/printk.h>
>> >+#include <linux/smp.h>
>> >+
>> >+#include "irq-riscv-aplic-main.h"
>> >+
>> >+static void aplic_msi_irq_unmask(struct irq_data *d)
>> >+{
>> >+ aplic_irq_unmask(d);
>> >+ irq_chip_unmask_parent(d);
>> >+}
>> >+
>> >+static void aplic_msi_irq_mask(struct irq_data *d)
>> >+{
>> >+ aplic_irq_mask(d);
>> >+ irq_chip_mask_parent(d);
>> >+}
>> >+
>> >+static void aplic_msi_irq_eoi(struct irq_data *d)
>> >+{
>> >+ struct aplic_priv *priv = irq_data_get_irq_chip_data(d);
>> >+ u32 reg_off, reg_mask;
>> >+
>> >+ /*
>> >+ * EOI handling only required only for level-triggered
>> >+ * interrupts in APLIC MSI mode.
>> >+ */
>> >+
>> >+ reg_off = APLIC_CLRIP_BASE + ((d->hwirq / APLIC_IRQBITS_PER_REG) * 4);
>> >+ reg_mask = BIT(d->hwirq % APLIC_IRQBITS_PER_REG);
>> >+ switch (irqd_get_trigger_type(d)) {
>> >+ case IRQ_TYPE_LEVEL_LOW:
>> >+ if (!(readl(priv->regs + reg_off) & reg_mask))
>> >+ writel(d->hwirq, priv->regs + APLIC_SETIPNUM_LE);
>> >+ break;
>> >+ case IRQ_TYPE_LEVEL_HIGH:
>> >+ if (readl(priv->regs + reg_off) & reg_mask)
>> >+ writel(d->hwirq, priv->regs + APLIC_SETIPNUM_LE);
>> >+ break;
>> >+ }
>> >+}
>> >+
>> >+static struct irq_chip aplic_msi_chip = {
>> >+ .name = "APLIC-MSI",
>> >+ .irq_mask = aplic_msi_irq_mask,
>> >+ .irq_unmask = aplic_msi_irq_unmask,
>> >+ .irq_set_type = aplic_irq_set_type,
>> >+ .irq_eoi = aplic_msi_irq_eoi,
>> >+#ifdef CONFIG_SMP
>> >+ .irq_set_affinity = irq_chip_set_affinity_parent,
>> >+#endif
>> >+ .flags = IRQCHIP_SET_TYPE_MASKED |
>> >+  IRQCHIP_SKIP_SET_WAKE |
>> >+  IRQCHIP_MASK_ON_SUSPEND,
>> >+};
>> >+
>> >+static int aplic_msi_irqdomain_translate(struct irq_domain *d,
>> >+ struct irq_fwspec *fwspec,
>> >+ unsigned long *hwirq,
>> >+ unsigned int *type)
>> >+{
>> >+ struct aplic_priv *priv = platform_msi_get_host_data(d);
>> >+
>> >+ return aplic_irqdomain_translate(fwspec, priv->gsi_base, hwirq, type);
>> >+}
>> >+
>> >+static int aplic_msi_irqdomain_alloc(struct irq_domain *domain,
>> >+     unsigned int virq, unsigned int nr_irqs,
>> >+     void *arg)
>> >+{
>> >+ int i, ret;
>> >+ unsigned int type;
>> >+ irq_hw_number_t hwirq;
>> >+ struct irq_fwspec *fwspec = arg;
>> >+ struct aplic_priv *priv = platform_msi_get_host_data(domain);
>> >+
>> >+ ret = aplic_irqdomain_translate(fwspec, priv->gsi_base, &hwirq, &type);
>> >+ if (ret)
>> >+ return ret;
>> >+
>> >+ ret = platform_msi_device_domain_alloc(domain, virq, nr_irqs);
>> >+ if (ret)
>> >+ return ret;
>> >+
>> >+ for (i = 0; i < nr_irqs; i++) {
>> >+ irq_domain_set_info(domain, virq + i, hwirq + i,
>> >+    &aplic_msi_chip, priv, handle_fasteoi_irq,
>> >+    NULL, NULL);
>> >+ /*
>> >+ * APLIC does not implement irq_disable() so Linux interrupt
>> >+ * subsystem will take a lazy approach for disabling an APLIC
>> >+ * interrupt. This means APLIC interrupts are left unmasked
>> >+ * upon system suspend and interrupts are not processed
>> >+ * immediately upon system wake up. To tackle this, we disable
>> >+ * the lazy approach for all APLIC interrupts.
>> >+ */
>> >+ irq_set_status_flags(virq + i, IRQ_DISABLE_UNLAZY);
>> >+ }
>>
>> For platfrom MSI irq, it will call irq_domain_set_info() and irq_set_status_flags() twice, the first one is here:
>> platform_msi_device_domain_alloc->msi_domain_populate_irqs->irq_domain_alloc_irqs_hierarchy->imsic_irq_domain_alloc->irq_domain_set_info
>>
>> so  i think here this for(...) is not necessary, can be removed.
>
>If we remove then it breaks APLIC MSI-mode because we have
>hierarchical irq domains where the APLIC-MSI domain is a child
>of the IMSIC-PLAT domain.
>
>The irq_domain_set_info() called by IMSIC driver only sets irqchip
>for IMSIC irq whereas irq_domain_set_info() called by APLIC driver
>sets irqchip for APLIC irq. We use a different APLIC irqchip for the
>APLIC domain to mask, unmask, and eoi irqs in an APLIC specific
>way.
>

As your said APLIC-MSI domain is a child of the IMSIC-PLAT domain, so all of platform IRQ or wired IRQ will go to APLIC-MSI domain firstly.
how about the pure MSI interrupt? for example the MSI of PCIe device or device driver call platform_msi_domain_alloc_irqs() to allocate a MSI ?
 in this scenario, it also go into  APLIC-MSI domain firstly?

would you like provide the steps how to test the PCI MSI for your patchset on QEMU? i run a QEMU system, but i cannot found any PCI devices using MSI, especially the virtio devices which using the platform IRQ.

~# cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
 10:      38972      38946      38882      38924  RISC-V INTC   5 Edge      riscv-timer
 11:          0       1149          0          0  APLIC-MSI   8 Level     virtio0
 12:          0          0         21          0  APLIC-MSI   7 Level     virtio1
 13:        149          0          0        218  APLIC-MSI  10 Level     ttyS0
IPI0:        40         53         43         50  Rescheduling interrupts
IPI1:      7518       8899       6679       7959  Function call interrupts
IPI2:         0          0          0          0  CPU stop interrupts
IPI3:         0          0          0          0  CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0  IRQ work interrupts
IPI5:         0          0          0          0  Timer broadcast interrupts

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv