[PATCH 0/6] Altera PCIe host controller driver with MSI support

[PATCH 0/6] Altera PCIe host controller driver with MSI support

[Ley Foon Tan]

This patch adds support for Altera PCIe host controller with MSI feature on
Altera FPGA device families.
Note, the MSI component is a soft IP that external from PCIe port.

It is based on patch series from Marc Zyngier "Per-device MSI domain &
platform MSI" [1] to get rid of struct msi_controller.

[1]: https://lkml.org/lkml/2015/7/23/172

Ley Foon Tan (6):
  arm: add msi.h to Kbuild
  arm: mach-socfpga: enable pci support
  pci:host: Add Altera PCIe host controller driver
  pci: altera: Add Altera PCIe MSI driver
  Documentation: dt-bindings: pci: altera pcie device tree binding
  MAINTAINERS: Add Altera PCIe driver maintainer

 .../devicetree/bindings/pci/altera-pcie-msi.txt    |  27 +
 .../devicetree/bindings/pci/altera-pcie.txt        |  49 ++
 MAINTAINERS                                        |  16 +
 arch/arm/include/asm/Kbuild                        |   1 +
 arch/arm/mach-socfpga/Kconfig                      |   2 +
 drivers/pci/host/Kconfig                           |  16 +
 drivers/pci/host/Makefile                          |   2 +
 drivers/pci/host/pcie-altera-msi.c                 | 318 ++++++++++++
 drivers/pci/host/pcie-altera.c                     | 576 +++++++++++++++++++++
 9 files changed, 1007 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/pci/altera-pcie-msi.txt
 create mode 100644 Documentation/devicetree/bindings/pci/altera-pcie.txt
 create mode 100644 drivers/pci/host/pcie-altera-msi.c
 create mode 100644 drivers/pci/host/pcie-altera.c

-- 
1.8.2.1

[PATCH 1/6] arm: add msi.h to Kbuild

[Ley Foon Tan]

Include asm-generic/msi.h to support CONFIG_GENERIC_MSI_IRQ_DOMAIN.
This to fix compilation error:
"include/linux/msi.h:123:21: fatal error: asm/msi.h:
No such file or directory"

Signed-off-by: Ley Foon Tan <lftan@altera.com>
---
 arch/arm/include/asm/Kbuild | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm/include/asm/Kbuild b/arch/arm/include/asm/Kbuild
index 83c5019..ddc2761 100644
--- a/arch/arm/include/asm/Kbuild
+++ b/arch/arm/include/asm/Kbuild
@@ -14,6 +14,7 @@ generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
 generic-y += msgbuf.h
+generic-y += msi.h
 generic-y += param.h
 generic-y += parport.h
 generic-y += poll.h
-- 
1.8.2.1

[PATCH 2/6] arm: mach-socfpga: enable pci support

[Ley Foon Tan]

Enable CONFIG_ARCH_SUPPORT_MSI and CONFIG_PCI in SOCFPGA platform.

Signed-off-by: Ley Foon Tan <lftan@altera.com>
---
 arch/arm/mach-socfpga/Kconfig | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm/mach-socfpga/Kconfig b/arch/arm/mach-socfpga/Kconfig
index 90efdeb..7ff4576 100644
--- a/arch/arm/mach-socfpga/Kconfig
+++ b/arch/arm/mach-socfpga/Kconfig
@@ -1,6 +1,7 @@
 menuconfig ARCH_SOCFPGA
 	bool "Altera SOCFPGA family" if ARCH_MULTI_V7
 	select ARCH_SUPPORTS_BIG_ENDIAN
+	select ARCH_SUPPORTS_MSI
 	select ARM_AMBA
 	select ARM_GIC
 	select CACHE_L2X0
@@ -9,6 +10,7 @@ menuconfig ARCH_SOCFPGA
 	select HAVE_ARM_SCU
 	select HAVE_ARM_TWD if SMP
 	select MFD_SYSCON
+	select PCI
 
 if ARCH_SOCFPGA
 config SOCFPGA_SUSPEND
-- 
1.8.2.1

Re: [PATCH 2/6] arm: mach-socfpga: enable pci support

[Rob Herring]

On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan@altera.com> wrote:
> Enable CONFIG_ARCH_SUPPORT_MSI and CONFIG_PCI in SOCFPGA platform.
>
> Signed-off-by: Ley Foon Tan <lftan@altera.com>
> ---
>  arch/arm/mach-socfpga/Kconfig | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/arch/arm/mach-socfpga/Kconfig b/arch/arm/mach-socfpga/Kconfig
> index 90efdeb..7ff4576 100644
> --- a/arch/arm/mach-socfpga/Kconfig
> +++ b/arch/arm/mach-socfpga/Kconfig
> @@ -1,6 +1,7 @@
>  menuconfig ARCH_SOCFPGA
>         bool "Altera SOCFPGA family" if ARCH_MULTI_V7
>         select ARCH_SUPPORTS_BIG_ENDIAN
> +       select ARCH_SUPPORTS_MSI

This doesn't exist in mainline having been removed in 3.12.

>         select ARM_AMBA
>         select ARM_GIC
>         select CACHE_L2X0
> @@ -9,6 +10,7 @@ menuconfig ARCH_SOCFPGA
>         select HAVE_ARM_SCU
>         select HAVE_ARM_TWD if SMP
>         select MFD_SYSCON
> +       select PCI

This should not be selected by the platform, but be user selectable
unless perhaps you require PCI to boot.

Rob

Re: [PATCH 2/6] arm: mach-socfpga: enable pci support

[Ley Foon Tan]

On Tue, Jul 28, 2015 at 9:26 PM, Rob Herring <robherring2@gmail.com> wrote:
> On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan@altera.com> wrote:
>> Enable CONFIG_ARCH_SUPPORT_MSI and CONFIG_PCI in SOCFPGA platform.
>>
>> Signed-off-by: Ley Foon Tan <lftan@altera.com>
>> ---
>>  arch/arm/mach-socfpga/Kconfig | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/arch/arm/mach-socfpga/Kconfig b/arch/arm/mach-socfpga/Kconfig
>> index 90efdeb..7ff4576 100644
>> --- a/arch/arm/mach-socfpga/Kconfig
>> +++ b/arch/arm/mach-socfpga/Kconfig
>> @@ -1,6 +1,7 @@
>>  menuconfig ARCH_SOCFPGA
>>         bool "Altera SOCFPGA family" if ARCH_MULTI_V7
>>         select ARCH_SUPPORTS_BIG_ENDIAN
>> +       select ARCH_SUPPORTS_MSI
>
> This doesn't exist in mainline having been removed in 3.12.
Will removed this.
>
>>         select ARM_AMBA
>>         select ARM_GIC
>>         select CACHE_L2X0
>> @@ -9,6 +10,7 @@ menuconfig ARCH_SOCFPGA
>>         select HAVE_ARM_SCU
>>         select HAVE_ARM_TWD if SMP
>>         select MFD_SYSCON
>> +       select PCI
>
> This should not be selected by the platform, but be user selectable
> unless perhaps you require PCI to boot.

Will removed this as well.

Thanks for reviewing.

Regards
Ley Foon

[PATCH 3/6] pci:host: Add Altera PCIe host controller driver

[Ley Foon Tan]

This patch adds the Altera PCIe host controller driver.

Signed-off-by: Ley Foon Tan <lftan@altera.com>
---
 drivers/pci/host/Kconfig       |   9 +
 drivers/pci/host/Makefile      |   1 +
 drivers/pci/host/pcie-altera.c | 576 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 586 insertions(+)
 create mode 100644 drivers/pci/host/pcie-altera.c

diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
index 675c2d1..af19039 100644
--- a/drivers/pci/host/Kconfig
+++ b/drivers/pci/host/Kconfig
@@ -145,4 +145,13 @@ config PCIE_IPROC_BCMA
 	  Say Y here if you want to use the Broadcom iProc PCIe controller
 	  through the BCMA bus interface
 
+config PCIE_ALTERA
+	bool "Altera PCIe controller"
+	depends on ARCH_SOCFPGA
+	depends on OF
+	select PCI_MSI_IRQ_DOMAIN if PCI_MSI
+	help
+	  Say Y here if you want to enable PCIe controller support for Altera
+	  SoCFPGA family of SoCs.
+
 endmenu
diff --git a/drivers/pci/host/Makefile b/drivers/pci/host/Makefile
index 140d66f..6954f76 100644
--- a/drivers/pci/host/Makefile
+++ b/drivers/pci/host/Makefile
@@ -17,3 +17,4 @@ obj-$(CONFIG_PCI_VERSATILE) += pci-versatile.o
 obj-$(CONFIG_PCIE_IPROC) += pcie-iproc.o
 obj-$(CONFIG_PCIE_IPROC_PLATFORM) += pcie-iproc-platform.o
 obj-$(CONFIG_PCIE_IPROC_BCMA) += pcie-iproc-bcma.o
+obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o
diff --git a/drivers/pci/host/pcie-altera.c b/drivers/pci/host/pcie-altera.c
new file mode 100644
index 0000000..afa64e9
--- /dev/null
+++ b/drivers/pci/host/pcie-altera.c
@@ -0,0 +1,576 @@
+/*
+ * Copyright Altera Corporation (C) 2013-2015. All rights reserved
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/of_pci.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+
+#define A2P_ADDR_MAP_LO0		0x1000
+#define A2P_ADDR_MAP_HI0		0x1004
+#define RP_TX_REG0			0x2000
+#define RP_TX_REG1			0x2004
+#define RP_TX_CNTRL			0x2008
+#define RP_TX_EOP			0x2
+#define RP_TX_SOP			0x1
+#define RP_RXCPL_STATUS		0x2010
+#define RP_RXCPL_EOP			0x2
+#define RP_RXCPL_SOP			0x1
+#define RP_RXCPL_REG0			0x2014
+#define RP_RXCPL_REG1			0x2018
+#define P2A_INT_STATUS			0x3060
+#define P2A_INT_STS_ALL			0xF
+#define P2A_INT_ENABLE			0x3070
+#define P2A_INT_ENA_ALL			0xF
+#define RP_LTSSM			0x3C64
+#define LTSSM_L0			0xF
+
+/* TLP configuration type 0 and 1 */
+#define TLP_FMTTYPE_CFGRD0		0x04	/* Configuration Read Type 0 */
+#define TLP_FMTTYPE_CFGWR0		0x44	/* Configuration Write Type 0 */
+#define TLP_FMTTYPE_CFGRD1		0x05	/* Configuration Read Type 1 */
+#define TLP_FMTTYPE_CFGWR1		0x45	/* Configuration Write Type 1 */
+#define TLP_PAYLOAD_SIZE		0x01
+#define TLP_READ_TAG			0x1D
+#define TLP_WRITE_TAG			0x10
+#define TLP_CFG_DW0(fmttype)		(((fmttype) << 24) | TLP_PAYLOAD_SIZE)
+#define TLP_CFG_DW1(reqid, tag)		(((reqid) << 16) | (tag << 8) | 0xF)
+#define TLP_CFG_DW2(bus, devfn, offset)	\
+				(((bus) << 24) | ((devfn) << 16) | (offset))
+#define TLP_REQ_ID(bus, devfn)		(((bus) << 8) | (devfn))
+#define TLP_COMPL_STATUS(hdr)		(((hdr) & 0xE0) >> 13)
+#define TLP_HDR_SIZE			3
+#define TLP_LOOP			10
+
+#define INTX_NUM		4
+
+/* Address translation table entry size */
+#define ATT_ENTRY_SIZE		8
+
+#define DWORD_MASK		3
+
+struct altera_pcie {
+	struct platform_device	*pdev;
+	struct resource		*txs;
+	void __iomem		*cra_base;
+	int			hwirq;
+	u8			root_bus_nr;
+	struct irq_domain		*irq_domain;
+	struct resource		bus_range;
+	struct list_head		resources;
+};
+
+struct tlp_rp_regpair_t {
+	u32 ctrl;
+	u32 reg0;
+	u32 reg1;
+};
+
+static inline struct altera_pcie *sys_to_pcie(struct pci_sys_data *sys)
+{
+	return sys->private_data;
+}
+
+static void altera_pcie_retrain(struct pci_dev *dev)
+{
+	u16 linkcap, linkstat;
+
+	/*
+	 * Set the retrain bit if the PCIe rootport support > 2.5GB/s, but
+	 * current speed is 2.5 GB/s.
+	 */
+	pcie_capability_read_word(dev, PCI_EXP_LNKCAP, &linkcap);
+
+	if ((linkcap & PCI_EXP_LNKCAP_SLS) <= PCI_EXP_LNKCAP_SLS_2_5GB)
+		return;
+
+	pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &linkstat);
+	if ((linkstat & PCI_EXP_LNKSTA_CLS) == PCI_EXP_LNKSTA_CLS_2_5GB)
+		pcie_capability_set_word(dev, PCI_EXP_LNKCTL,
+					PCI_EXP_LNKCTL_RL);
+
+}
+DECLARE_PCI_FIXUP_EARLY(PCI_ANY_ID, PCI_ANY_ID, altera_pcie_retrain);
+
+static inline void cra_writel(struct altera_pcie *pcie, u32 value, u32 reg)
+{
+	writel(value, pcie->cra_base + reg);
+}
+
+static inline u32 cra_readl(struct altera_pcie *pcie, u32 reg)
+{
+	return readl(pcie->cra_base + reg);
+}
+
+static void tlp_read_rx(struct altera_pcie *pcie,
+			struct tlp_rp_regpair_t *tlp_rp_regdata)
+{
+	tlp_rp_regdata->ctrl = cra_readl(pcie, RP_RXCPL_STATUS);
+	tlp_rp_regdata->reg0 = cra_readl(pcie, RP_RXCPL_REG0);
+	tlp_rp_regdata->reg1 = cra_readl(pcie, RP_RXCPL_REG1);
+}
+
+static void tlp_write_tx(struct altera_pcie *pcie,
+			struct tlp_rp_regpair_t *tlp_rp_regdata)
+{
+	cra_writel(pcie, tlp_rp_regdata->reg0, RP_TX_REG0);
+	cra_writel(pcie, tlp_rp_regdata->reg1, RP_TX_REG1);
+	cra_writel(pcie, tlp_rp_regdata->ctrl, RP_TX_CNTRL);
+}
+
+static bool altera_pcie_link_is_up(struct altera_pcie *pcie)
+{
+	return !!(cra_readl(pcie, RP_LTSSM) & LTSSM_L0);
+}
+
+static bool altera_pcie_valid_config(struct altera_pcie *pcie,
+				struct pci_bus *bus, int dev)
+{
+	/* If there is no link, then there is no device */
+	if (bus->number != pcie->root_bus_nr) {
+		if (!altera_pcie_link_is_up(pcie))
+			return false;
+	}
+
+	/* access only one slot on each root port */
+	if (bus->number == pcie->root_bus_nr && dev > 0)
+		return false;
+
+	/*
+	 * Do not read more than one device on the bus directly attached
+	 * to RC.
+	 */
+	if (bus->primary == pcie->root_bus_nr && dev > 0)
+		return false;
+
+	 return true;
+}
+
+static int tlp_read_packet(struct altera_pcie *pcie, u32 *value)
+{
+	u8 loop;
+	struct tlp_rp_regpair_t tlp_rp_regdata;
+
+	for (loop = TLP_LOOP; loop > 0; loop--) {
+
+		tlp_read_rx(pcie, &tlp_rp_regdata);
+
+		if (tlp_rp_regdata.ctrl & RP_RXCPL_EOP) {
+
+			if (value)
+				*value = tlp_rp_regdata.reg0;
+
+			return PCIBIOS_SUCCESSFUL;
+		}
+	}
+
+	return -ENOENT;
+}
+
+static void tlp_write_packet(struct altera_pcie *pcie, u32 *headers, u32 data)
+{
+	struct tlp_rp_regpair_t tlp_rp_regdata;
+
+	tlp_rp_regdata.reg0 = headers[0];
+	tlp_rp_regdata.reg1 = headers[1];
+	tlp_rp_regdata.ctrl = RP_TX_SOP;
+	tlp_write_tx(pcie, &tlp_rp_regdata);
+
+	tlp_rp_regdata.reg0 = headers[2];
+	tlp_rp_regdata.reg1 = data;
+	tlp_rp_regdata.ctrl = RP_TX_EOP;
+	tlp_write_tx(pcie, &tlp_rp_regdata);
+}
+
+static int tlp_cfg_dword_read(struct altera_pcie *pcie, u8 bus, u32 devfn,
+			      int where, u32 *value)
+{
+	int ret;
+	u32 headers[TLP_HDR_SIZE];
+
+	if (bus == pcie->root_bus_nr)
+		headers[0] = TLP_CFG_DW0(TLP_FMTTYPE_CFGRD0);
+	else
+		headers[0] = TLP_CFG_DW0(TLP_FMTTYPE_CFGRD1);
+
+	headers[1] = TLP_CFG_DW1(TLP_REQ_ID(pcie->root_bus_nr, devfn),
+					TLP_READ_TAG);
+	headers[2] = TLP_CFG_DW2(bus, devfn, where);
+
+	tlp_write_packet(pcie, headers, 0);
+
+	ret = tlp_read_packet(pcie, value);
+	if (ret)
+		*value = ~0UL;	/* return 0xFFFFFFFF if error */
+
+	return ret;
+}
+
+static int tlp_cfg_dword_write(struct altera_pcie *pcie, u8 bus, u32 devfn,
+			       int where, u32 value)
+{
+	u32 headers[TLP_HDR_SIZE];
+
+	if (bus == pcie->root_bus_nr)
+		headers[0] = TLP_CFG_DW0(TLP_FMTTYPE_CFGWR0);
+	else
+		headers[0] = TLP_CFG_DW0(TLP_FMTTYPE_CFGWR1);
+
+	headers[1] = TLP_CFG_DW1(TLP_REQ_ID(pcie->root_bus_nr, devfn),
+					TLP_WRITE_TAG);
+	headers[2] = TLP_CFG_DW2(bus, devfn, where);
+
+	tlp_write_packet(pcie, headers, value);
+
+	tlp_read_packet(pcie, NULL);
+
+	/* Keep an eye out for changes to the root bus number */
+	if ((bus == pcie->root_bus_nr) && (where == PCI_PRIMARY_BUS))
+		pcie->root_bus_nr = (u8)(value);
+
+	return PCIBIOS_SUCCESSFUL;
+}
+
+static int altera_pcie_cfg_read(struct pci_bus *bus, unsigned int devfn,
+			       int where, int size, u32 *value)
+{
+	struct altera_pcie *pcie = sys_to_pcie(bus->sysdata);
+	int ret;
+
+	if (!altera_pcie_valid_config(pcie, bus, PCI_SLOT(devfn))) {
+		*value = ~0UL;
+		return PCIBIOS_DEVICE_NOT_FOUND;
+	}
+
+	ret = tlp_cfg_dword_read(pcie, bus->number, devfn,
+		(where & ~DWORD_MASK), value);
+	if (ret != PCIBIOS_SUCCESSFUL)
+		return ret;
+
+	if (size == 1)
+		*value = (*value >> (8 * (where & 0x3))) & 0xff;
+	else if (size == 2)
+		*value = (*value >> (8 * (where & 0x2))) & 0xffff;
+
+	return PCIBIOS_SUCCESSFUL;
+}
+
+static int altera_pcie_cfg_write(struct pci_bus *bus, unsigned int devfn,
+				int where, int size, u32 value)
+{
+	struct altera_pcie *pcie = sys_to_pcie(bus->sysdata);
+	u32 data32 = value;
+	u32 shift = 8 * (where & 3);
+	int ret;
+
+	if (!altera_pcie_valid_config(pcie, bus, PCI_SLOT(devfn)))
+		return PCIBIOS_DEVICE_NOT_FOUND;
+
+	/* write partial */
+	if (size != sizeof(u32)) {
+		ret = tlp_cfg_dword_read(pcie, bus->number, devfn,
+			where & ~DWORD_MASK, &data32);
+		if (ret)
+			return ret;
+
+		if (size == 2)
+			data32 = (data32 & ~(0xffff << shift)) |
+					((value & 0xffff) << shift);
+		 else if (size == 1)
+			data32 = (data32 & ~(0xff << shift)) |
+					((value & 0xff) << shift);
+	}
+
+	return tlp_cfg_dword_write(pcie, bus->number, devfn,
+		(where & ~DWORD_MASK), data32);
+}
+
+static struct pci_ops altera_pcie_ops = {
+	.read = altera_pcie_cfg_read,
+	.write = altera_pcie_cfg_write,
+};
+
+static int altera_pcie_setup(int nr, struct pci_sys_data *sys)
+{
+	struct altera_pcie *pcie = sys->private_data;
+
+	list_splice_init(&pcie->resources, &sys->resources);
+
+	return 1;
+}
+
+static int altera_pcie_map_irq(const struct pci_dev *pdev, u8 slot, u8 pin)
+{
+	struct altera_pcie *pcie = sys_to_pcie(pdev->bus->sysdata);
+	int irq;
+
+	irq = of_irq_parse_and_map_pci(pdev, slot, pin);
+
+	if (!irq)
+		irq = pcie->hwirq;
+
+	return irq;
+}
+
+static struct pci_bus *altera_pcie_scan_bus(int nr, struct pci_sys_data *sys)
+{
+	struct altera_pcie *pcie = sys_to_pcie(sys);
+	struct pci_bus *bus;
+
+	pcie->root_bus_nr = sys->busnr;
+	bus = pci_scan_root_bus(&pcie->pdev->dev, sys->busnr, &altera_pcie_ops,
+				sys, &sys->resources);
+
+	return bus;
+}
+
+static struct hw_pci altera_pcie_hw __initdata = {
+#ifdef CONFIG_PCI_DOMAINS
+	.domain			= 0,
+#endif
+	.nr_controllers		= 1,
+	.ops			= &altera_pcie_ops,
+	.setup			= altera_pcie_setup,
+	.map_irq			= altera_pcie_map_irq,
+	.scan			= altera_pcie_scan_bus,
+};
+
+static int altera_pcie_intx_map(struct irq_domain *domain, unsigned int irq,
+				irq_hw_number_t hwirq)
+{
+	irq_set_chip_and_handler(irq, &dummy_irq_chip, handle_simple_irq);
+	irq_set_chip_data(irq, domain->host_data);
+	set_irq_flags(irq, IRQF_VALID);
+
+	return 0;
+}
+
+static const struct irq_domain_ops intx_domain_ops = {
+	.map = altera_pcie_intx_map,
+};
+
+static irqreturn_t altera_pcie_isr(int irq, void *arg)
+{
+	struct altera_pcie *pcie = arg;
+	u32 status, i;
+
+	status = cra_readl(pcie, P2A_INT_STATUS) & P2A_INT_STS_ALL;
+
+	/* clear interrupts */
+	cra_writel(pcie, status, P2A_INT_STATUS);
+
+	for (i = 0; i < INTX_NUM; i++) {
+		if (status & (1 << i))
+			generic_handle_irq(irq_find_mapping(pcie->irq_domain,
+							i + 1));
+	}
+
+	return IRQ_HANDLED;
+}
+
+static void altera_pcie_release_of_pci_ranges(struct altera_pcie *pcie)
+{
+	pci_free_resource_list(&pcie->resources);
+}
+
+static int altera_pcie_parse_request_of_pci_ranges(struct altera_pcie *pcie)
+{
+	int err, res_valid = 0;
+	struct device *dev = &pcie->pdev->dev;
+	struct device_node *np = dev->of_node;
+	resource_size_t iobase;
+	struct resource_entry *win;
+	int offset = 0;
+
+	err = of_pci_get_host_bridge_resources(np, 0, 0xff, &pcie->resources,
+					       &iobase);
+	if (err)
+		return err;
+
+	resource_list_for_each_entry(win, &pcie->resources) {
+		struct resource *parent, *res = win->res;
+
+		switch (resource_type(res)) {
+		case IORESOURCE_MEM:
+			parent = &iomem_resource;
+			res_valid |= !(res->flags & IORESOURCE_PREFETCH);
+			cra_writel(pcie, res->start,
+				A2P_ADDR_MAP_LO0 + offset);
+			cra_writel(pcie, 0,
+				A2P_ADDR_MAP_HI0 + offset);
+			offset += ATT_ENTRY_SIZE;
+			break;
+		default:
+			continue;
+		}
+
+		err = devm_request_resource(dev, parent, res);
+		if (err)
+			goto out_release_res;
+	}
+
+	if (!res_valid) {
+		dev_err(dev, "non-prefetchable memory resource required\n");
+		err = -EINVAL;
+		goto out_release_res;
+	}
+
+	return 0;
+
+out_release_res:
+	altera_pcie_release_of_pci_ranges(pcie);
+	return err;
+}
+
+static void altera_pcie_free_irq_domain(struct altera_pcie *pcie)
+{
+	int i;
+	u32 irq;
+
+	for (i = 0; i < INTX_NUM; i++) {
+		irq = irq_find_mapping(pcie->irq_domain, i);
+		if (irq > 0)
+			irq_dispose_mapping(irq);
+	}
+
+	irq_domain_remove(pcie->irq_domain);
+}
+
+static int altera_pcie_init_irq_domain(struct altera_pcie *pcie)
+{
+	struct device *dev = &pcie->pdev->dev;
+	struct device_node *node = dev->of_node;
+
+	/* Setup INTx */
+	pcie->irq_domain = irq_domain_add_linear(node, INTX_NUM,
+					&intx_domain_ops, pcie);
+	if (!pcie->irq_domain) {
+		dev_err(dev, "Failed to get a INTx IRQ domain\n");
+		return PTR_ERR(pcie->irq_domain);
+	}
+
+	return 0;
+}
+
+static int altera_pcie_parse_dt(struct altera_pcie *pcie)
+{
+	struct resource *cra;
+	int ret;
+	struct platform_device *pdev = pcie->pdev;
+
+	cra = platform_get_resource_byname(pdev, IORESOURCE_MEM, "Cra");
+	pcie->cra_base = devm_ioremap_resource(&pdev->dev, cra);
+	if (IS_ERR(pcie->cra_base)) {
+		dev_err(&pdev->dev, "get Cra resource failed\n");
+		return PTR_ERR(pcie->cra_base);
+	}
+
+	/* setup IRQ */
+	pcie->hwirq = platform_get_irq(pdev, 0);
+	if (pcie->hwirq <= 0) {
+		dev_err(&pdev->dev, "failed to get IRQ: %d\n", pcie->hwirq);
+		return -EINVAL;
+	}
+	ret = devm_request_irq(&pdev->dev, pcie->hwirq, altera_pcie_isr,
+			IRQF_SHARED, pdev->name, pcie);
+
+	if (ret) {
+		dev_err(&pdev->dev, "failed to request irq %d\n", pcie->hwirq);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int altera_pcie_probe(struct platform_device *pdev)
+{
+	struct altera_pcie *pcie;
+	int ret;
+
+	pcie = devm_kzalloc(&pdev->dev, sizeof(*pcie), GFP_KERNEL);
+	if (!pcie)
+		return -ENOMEM;
+
+	pcie->pdev = pdev;
+
+	ret = altera_pcie_parse_dt(pcie);
+	if (ret) {
+		dev_err(&pdev->dev, "Parsing DT failed\n");
+		return ret;
+	}
+
+	INIT_LIST_HEAD(&pcie->resources);
+
+	ret = altera_pcie_parse_request_of_pci_ranges(pcie);
+	if (ret) {
+		dev_err(&pdev->dev, "Failed add resources\n");
+		return ret;
+	}
+
+	ret = altera_pcie_init_irq_domain(pcie);
+	if (ret) {
+		dev_err(&pdev->dev, "Failed creating IRQ Domain\n");
+		return ret;
+	}
+
+	pcie->root_bus_nr = -1;
+
+	/* clear all interrupts */
+	cra_writel(pcie, P2A_INT_STS_ALL, P2A_INT_STATUS);
+	/* enable all interrupts */
+	cra_writel(pcie, P2A_INT_ENA_ALL, P2A_INT_ENABLE);
+
+	altera_pcie_hw.private_data = (void **)&pcie;
+
+	pci_common_init_dev(&pdev->dev, &altera_pcie_hw);
+
+	platform_set_drvdata(pdev, pcie);
+	return ret;
+}
+
+static int __exit altera_pcie_remove(struct platform_device *pdev)
+{
+	struct altera_pcie *pcie = platform_get_drvdata(pdev);
+
+	altera_pcie_free_irq_domain(pcie);
+	platform_set_drvdata(pdev, NULL);
+	return 0;
+}
+
+static const struct of_device_id altera_pcie_of_match[] = {
+	{ .compatible = "altr,pcie-root-port-1.0", },
+	{},
+};
+MODULE_DEVICE_TABLE(of, altera_pcie_of_match);
+
+static struct platform_driver altera_pcie_driver = {
+	.probe		= altera_pcie_probe,
+	.remove		= altera_pcie_remove,
+	.driver = {
+		.name	= "altera-pcie",
+		.owner	= THIS_MODULE,
+		.of_match_table = altera_pcie_of_match,
+	},
+};
+
+module_platform_driver(altera_pcie_driver);
+
+MODULE_AUTHOR("Ley Foon Tan <lftan@altera.com>");
+MODULE_DESCRIPTION("Altera PCIe host controller driver");
+MODULE_LICENSE("GPL v2");
-- 
1.8.2.1

Re: [PATCH 3/6] pci:host: Add Altera PCIe host controller driver

[Dinh Nguyen]

On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan@altera.com> wrote:
> This patch adds the Altera PCIe host controller driver.
>
> Signed-off-by: Ley Foon Tan <lftan@altera.com>
> ---
>  drivers/pci/host/Kconfig       |   9 +
>  drivers/pci/host/Makefile      |   1 +
>  drivers/pci/host/pcie-altera.c | 576 +++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 586 insertions(+)
>  create mode 100644 drivers/pci/host/pcie-altera.c
>
> diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
> index 675c2d1..af19039 100644
> --- a/drivers/pci/host/Kconfig
> +++ b/drivers/pci/host/Kconfig
> @@ -145,4 +145,13 @@ config PCIE_IPROC_BCMA
>           Say Y here if you want to use the Broadcom iProc PCIe controller
>           through the BCMA bus interface
>
> +config PCIE_ALTERA
> +       bool "Altera PCIe controller"
> +       depends on ARCH_SOCFPGA
> +       depends on OF

I don't think you need this, "depends on ARCH_SOCFPGA" should already
have taken care of this.

> +       select PCI_MSI_IRQ_DOMAIN if PCI_MSI
> +       help
> +         Say Y here if you want to enable PCIe controller support for Altera
> +         SoCFPGA family of SoCs.
> +
>  endmenu

<snip>

> +
> +static int tlp_read_packet(struct altera_pcie *pcie, u32 *value)
> +{
> +       u8 loop;
> +       struct tlp_rp_regpair_t tlp_rp_regdata;
> +
> +       for (loop = TLP_LOOP; loop > 0; loop--) {
> +
> +               tlp_read_rx(pcie, &tlp_rp_regdata);
> +
> +               if (tlp_rp_regdata.ctrl & RP_RXCPL_EOP) {
> +
> +                       if (value)
> +                               *value = tlp_rp_regdata.reg0;
> +
> +                       return PCIBIOS_SUCCESSFUL;
> +               }

Remove all the unnecessary newlines in this function.
> +       }
> +
> +       return -ENOENT;
> +}
> +

<snip>

> +
> +static struct platform_driver altera_pcie_driver = {
> +       .probe          = altera_pcie_probe,
> +       .remove         = altera_pcie_remove,
> +       .driver = {
> +               .name   = "altera-pcie",
> +               .owner  = THIS_MODULE,

Don't need to set owner anymore as this will get populated by the driver core.

Dinh

Re: [PATCH 3/6] pci:host: Add Altera PCIe host controller driver

[Ley Foon Tan]

On Wed, Jul 29, 2015 at 12:45 AM, Dinh Nguyen <dinh.linux@gmail.com> wrote:
> On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan@altera.com> wrote:
>> This patch adds the Altera PCIe host controller driver.
>>
>> Signed-off-by: Ley Foon Tan <lftan@altera.com>
>> ---
>>  drivers/pci/host/Kconfig       |   9 +
>>  drivers/pci/host/Makefile      |   1 +
>>  drivers/pci/host/pcie-altera.c | 576 +++++++++++++++++++++++++++++++++++++++++
>>  3 files changed, 586 insertions(+)
>>  create mode 100644 drivers/pci/host/pcie-altera.c
>>
>> diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
>> index 675c2d1..af19039 100644
>> --- a/drivers/pci/host/Kconfig
>> +++ b/drivers/pci/host/Kconfig
>> @@ -145,4 +145,13 @@ config PCIE_IPROC_BCMA
>>           Say Y here if you want to use the Broadcom iProc PCIe controller
>>           through the BCMA bus interface
>>
>> +config PCIE_ALTERA
>> +       bool "Altera PCIe controller"
>> +       depends on ARCH_SOCFPGA
>> +       depends on OF
>
> I don't think you need this, "depends on ARCH_SOCFPGA" should already
> have taken care of this.
Okay, will remove this.
>
>> +       select PCI_MSI_IRQ_DOMAIN if PCI_MSI
>> +       help
>> +         Say Y here if you want to enable PCIe controller support for Altera
>> +         SoCFPGA family of SoCs.
>> +
>>  endmenu
>
> <snip>
>
>> +
>> +static int tlp_read_packet(struct altera_pcie *pcie, u32 *value)
>> +{
>> +       u8 loop;
>> +       struct tlp_rp_regpair_t tlp_rp_regdata;
>> +
>> +       for (loop = TLP_LOOP; loop > 0; loop--) {
>> +
>> +               tlp_read_rx(pcie, &tlp_rp_regdata);
>> +
>> +               if (tlp_rp_regdata.ctrl & RP_RXCPL_EOP) {
>> +
>> +                       if (value)
>> +                               *value = tlp_rp_regdata.reg0;
>> +
>> +                       return PCIBIOS_SUCCESSFUL;
>> +               }
>
> Remove all the unnecessary newlines in this function.
Noted.

>> +       }
>> +
>> +       return -ENOENT;
>> +}
>> +
>
> <snip>
>
>> +
>> +static struct platform_driver altera_pcie_driver = {
>> +       .probe          = altera_pcie_probe,
>> +       .remove         = altera_pcie_remove,
>> +       .driver = {
>> +               .name   = "altera-pcie",
>> +               .owner  = THIS_MODULE,
>
> Don't need to set owner anymore as this will get populated by the driver core.

Will remove it.

Thanks for reviewing.

Regards
Ley Foon

Re: [PATCH 3/6] pci:host: Add Altera PCIe host controller driver

[Rob Herring]

On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan@altera.com> wrote:
> This patch adds the Altera PCIe host controller driver.
>
> Signed-off-by: Ley Foon Tan <lftan@altera.com>
> ---
>  drivers/pci/host/Kconfig       |   9 +
>  drivers/pci/host/Makefile      |   1 +
>  drivers/pci/host/pcie-altera.c | 576 +++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 586 insertions(+)
>  create mode 100644 drivers/pci/host/pcie-altera.c
>
> diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
> index 675c2d1..af19039 100644
> --- a/drivers/pci/host/Kconfig
> +++ b/drivers/pci/host/Kconfig
> @@ -145,4 +145,13 @@ config PCIE_IPROC_BCMA
>           Say Y here if you want to use the Broadcom iProc PCIe controller
>           through the BCMA bus interface
>
> +config PCIE_ALTERA
> +       bool "Altera PCIe controller"
> +       depends on ARCH_SOCFPGA
> +       depends on OF
> +       select PCI_MSI_IRQ_DOMAIN if PCI_MSI
> +       help
> +         Say Y here if you want to enable PCIe controller support for Altera
> +         SoCFPGA family of SoCs.
> +
>  endmenu
> diff --git a/drivers/pci/host/Makefile b/drivers/pci/host/Makefile
> index 140d66f..6954f76 100644
> --- a/drivers/pci/host/Makefile
> +++ b/drivers/pci/host/Makefile
> @@ -17,3 +17,4 @@ obj-$(CONFIG_PCI_VERSATILE) += pci-versatile.o
>  obj-$(CONFIG_PCIE_IPROC) += pcie-iproc.o
>  obj-$(CONFIG_PCIE_IPROC_PLATFORM) += pcie-iproc-platform.o
>  obj-$(CONFIG_PCIE_IPROC_BCMA) += pcie-iproc-bcma.o
> +obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o
> diff --git a/drivers/pci/host/pcie-altera.c b/drivers/pci/host/pcie-altera.c
> new file mode 100644
> index 0000000..afa64e9
> --- /dev/null
> +++ b/drivers/pci/host/pcie-altera.c
> @@ -0,0 +1,576 @@
> +/*
> + * Copyright Altera Corporation (C) 2013-2015. All rights reserved
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_irq.h>
> +#include <linux/of_pci.h>
> +#include <linux/pci.h>
> +#include <linux/platform_device.h>
> +#include <linux/slab.h>
> +
> +#define A2P_ADDR_MAP_LO0               0x1000
> +#define A2P_ADDR_MAP_HI0               0x1004
> +#define RP_TX_REG0                     0x2000
> +#define RP_TX_REG1                     0x2004
> +#define RP_TX_CNTRL                    0x2008
> +#define RP_TX_EOP                      0x2
> +#define RP_TX_SOP                      0x1
> +#define RP_RXCPL_STATUS                0x2010
> +#define RP_RXCPL_EOP                   0x2
> +#define RP_RXCPL_SOP                   0x1
> +#define RP_RXCPL_REG0                  0x2014
> +#define RP_RXCPL_REG1                  0x2018
> +#define P2A_INT_STATUS                 0x3060
> +#define P2A_INT_STS_ALL                        0xF
> +#define P2A_INT_ENABLE                 0x3070
> +#define P2A_INT_ENA_ALL                        0xF
> +#define RP_LTSSM                       0x3C64
> +#define LTSSM_L0                       0xF
> +
> +/* TLP configuration type 0 and 1 */
> +#define TLP_FMTTYPE_CFGRD0             0x04    /* Configuration Read Type 0 */
> +#define TLP_FMTTYPE_CFGWR0             0x44    /* Configuration Write Type 0 */
> +#define TLP_FMTTYPE_CFGRD1             0x05    /* Configuration Read Type 1 */
> +#define TLP_FMTTYPE_CFGWR1             0x45    /* Configuration Write Type 1 */
> +#define TLP_PAYLOAD_SIZE               0x01
> +#define TLP_READ_TAG                   0x1D
> +#define TLP_WRITE_TAG                  0x10
> +#define TLP_CFG_DW0(fmttype)           (((fmttype) << 24) | TLP_PAYLOAD_SIZE)
> +#define TLP_CFG_DW1(reqid, tag)                (((reqid) << 16) | (tag << 8) | 0xF)
> +#define TLP_CFG_DW2(bus, devfn, offset)        \
> +                               (((bus) << 24) | ((devfn) << 16) | (offset))
> +#define TLP_REQ_ID(bus, devfn)         (((bus) << 8) | (devfn))
> +#define TLP_COMPL_STATUS(hdr)          (((hdr) & 0xE0) >> 13)
> +#define TLP_HDR_SIZE                   3
> +#define TLP_LOOP                       10
> +
> +#define INTX_NUM               4
> +
> +/* Address translation table entry size */
> +#define ATT_ENTRY_SIZE         8
> +
> +#define DWORD_MASK             3
> +
> +struct altera_pcie {
> +       struct platform_device  *pdev;
> +       struct resource         *txs;
> +       void __iomem            *cra_base;
> +       int                     hwirq;
> +       u8                      root_bus_nr;
> +       struct irq_domain               *irq_domain;
> +       struct resource         bus_range;
> +       struct list_head                resources;
> +};
> +
> +struct tlp_rp_regpair_t {
> +       u32 ctrl;
> +       u32 reg0;
> +       u32 reg1;
> +};
> +
> +static inline struct altera_pcie *sys_to_pcie(struct pci_sys_data *sys)
> +{
> +       return sys->private_data;
> +}
> +
> +static void altera_pcie_retrain(struct pci_dev *dev)
> +{
> +       u16 linkcap, linkstat;
> +
> +       /*
> +        * Set the retrain bit if the PCIe rootport support > 2.5GB/s, but
> +        * current speed is 2.5 GB/s.
> +        */
> +       pcie_capability_read_word(dev, PCI_EXP_LNKCAP, &linkcap);
> +
> +       if ((linkcap & PCI_EXP_LNKCAP_SLS) <= PCI_EXP_LNKCAP_SLS_2_5GB)
> +               return;
> +
> +       pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &linkstat);
> +       if ((linkstat & PCI_EXP_LNKSTA_CLS) == PCI_EXP_LNKSTA_CLS_2_5GB)
> +               pcie_capability_set_word(dev, PCI_EXP_LNKCTL,
> +                                       PCI_EXP_LNKCTL_RL);
> +
> +}
> +DECLARE_PCI_FIXUP_EARLY(PCI_ANY_ID, PCI_ANY_ID, altera_pcie_retrain);
> +
> +static inline void cra_writel(struct altera_pcie *pcie, u32 value, u32 reg)
> +{
> +       writel(value, pcie->cra_base + reg);
> +}
> +
> +static inline u32 cra_readl(struct altera_pcie *pcie, u32 reg)
> +{
> +       return readl(pcie->cra_base + reg);
> +}
> +
> +static void tlp_read_rx(struct altera_pcie *pcie,
> +                       struct tlp_rp_regpair_t *tlp_rp_regdata)
> +{
> +       tlp_rp_regdata->ctrl = cra_readl(pcie, RP_RXCPL_STATUS);
> +       tlp_rp_regdata->reg0 = cra_readl(pcie, RP_RXCPL_REG0);
> +       tlp_rp_regdata->reg1 = cra_readl(pcie, RP_RXCPL_REG1);
> +}
> +
> +static void tlp_write_tx(struct altera_pcie *pcie,
> +                       struct tlp_rp_regpair_t *tlp_rp_regdata)
> +{
> +       cra_writel(pcie, tlp_rp_regdata->reg0, RP_TX_REG0);
> +       cra_writel(pcie, tlp_rp_regdata->reg1, RP_TX_REG1);
> +       cra_writel(pcie, tlp_rp_regdata->ctrl, RP_TX_CNTRL);
> +}
> +
> +static bool altera_pcie_link_is_up(struct altera_pcie *pcie)
> +{
> +       return !!(cra_readl(pcie, RP_LTSSM) & LTSSM_L0);
> +}
> +
> +static bool altera_pcie_valid_config(struct altera_pcie *pcie,
> +                               struct pci_bus *bus, int dev)
> +{
> +       /* If there is no link, then there is no device */
> +       if (bus->number != pcie->root_bus_nr) {
> +               if (!altera_pcie_link_is_up(pcie))
> +                       return false;
> +       }
> +
> +       /* access only one slot on each root port */
> +       if (bus->number == pcie->root_bus_nr && dev > 0)
> +               return false;
> +
> +       /*
> +        * Do not read more than one device on the bus directly attached
> +        * to RC.
> +        */
> +       if (bus->primary == pcie->root_bus_nr && dev > 0)
> +               return false;
> +
> +        return true;
> +}
> +
> +static int tlp_read_packet(struct altera_pcie *pcie, u32 *value)
> +{
> +       u8 loop;
> +       struct tlp_rp_regpair_t tlp_rp_regdata;
> +
> +       for (loop = TLP_LOOP; loop > 0; loop--) {
> +
> +               tlp_read_rx(pcie, &tlp_rp_regdata);
> +
> +               if (tlp_rp_regdata.ctrl & RP_RXCPL_EOP) {
> +
> +                       if (value)
> +                               *value = tlp_rp_regdata.reg0;
> +
> +                       return PCIBIOS_SUCCESSFUL;
> +               }
> +       }
> +
> +       return -ENOENT;
> +}
> +
> +static void tlp_write_packet(struct altera_pcie *pcie, u32 *headers, u32 data)
> +{
> +       struct tlp_rp_regpair_t tlp_rp_regdata;
> +
> +       tlp_rp_regdata.reg0 = headers[0];
> +       tlp_rp_regdata.reg1 = headers[1];
> +       tlp_rp_regdata.ctrl = RP_TX_SOP;
> +       tlp_write_tx(pcie, &tlp_rp_regdata);
> +
> +       tlp_rp_regdata.reg0 = headers[2];
> +       tlp_rp_regdata.reg1 = data;
> +       tlp_rp_regdata.ctrl = RP_TX_EOP;
> +       tlp_write_tx(pcie, &tlp_rp_regdata);
> +}
> +
> +static int tlp_cfg_dword_read(struct altera_pcie *pcie, u8 bus, u32 devfn,
> +                             int where, u32 *value)
> +{
> +       int ret;
> +       u32 headers[TLP_HDR_SIZE];
> +
> +       if (bus == pcie->root_bus_nr)
> +               headers[0] = TLP_CFG_DW0(TLP_FMTTYPE_CFGRD0);
> +       else
> +               headers[0] = TLP_CFG_DW0(TLP_FMTTYPE_CFGRD1);
> +
> +       headers[1] = TLP_CFG_DW1(TLP_REQ_ID(pcie->root_bus_nr, devfn),
> +                                       TLP_READ_TAG);
> +       headers[2] = TLP_CFG_DW2(bus, devfn, where);
> +
> +       tlp_write_packet(pcie, headers, 0);
> +
> +       ret = tlp_read_packet(pcie, value);
> +       if (ret)
> +               *value = ~0UL;  /* return 0xFFFFFFFF if error */
> +
> +       return ret;
> +}
> +
> +static int tlp_cfg_dword_write(struct altera_pcie *pcie, u8 bus, u32 devfn,
> +                              int where, u32 value)
> +{
> +       u32 headers[TLP_HDR_SIZE];
> +
> +       if (bus == pcie->root_bus_nr)
> +               headers[0] = TLP_CFG_DW0(TLP_FMTTYPE_CFGWR0);
> +       else
> +               headers[0] = TLP_CFG_DW0(TLP_FMTTYPE_CFGWR1);
> +
> +       headers[1] = TLP_CFG_DW1(TLP_REQ_ID(pcie->root_bus_nr, devfn),
> +                                       TLP_WRITE_TAG);
> +       headers[2] = TLP_CFG_DW2(bus, devfn, where);
> +
> +       tlp_write_packet(pcie, headers, value);
> +
> +       tlp_read_packet(pcie, NULL);
> +
> +       /* Keep an eye out for changes to the root bus number */
> +       if ((bus == pcie->root_bus_nr) && (where == PCI_PRIMARY_BUS))
> +               pcie->root_bus_nr = (u8)(value);
> +
> +       return PCIBIOS_SUCCESSFUL;
> +}
> +
> +static int altera_pcie_cfg_read(struct pci_bus *bus, unsigned int devfn,
> +                              int where, int size, u32 *value)
> +{
> +       struct altera_pcie *pcie = sys_to_pcie(bus->sysdata);
> +       int ret;
> +
> +       if (!altera_pcie_valid_config(pcie, bus, PCI_SLOT(devfn))) {
> +               *value = ~0UL;
> +               return PCIBIOS_DEVICE_NOT_FOUND;
> +       }
> +
> +       ret = tlp_cfg_dword_read(pcie, bus->number, devfn,
> +               (where & ~DWORD_MASK), value);
> +       if (ret != PCIBIOS_SUCCESSFUL)
> +               return ret;
> +
> +       if (size == 1)
> +               *value = (*value >> (8 * (where & 0x3))) & 0xff;
> +       else if (size == 2)
> +               *value = (*value >> (8 * (where & 0x2))) & 0xffff;
> +
> +       return PCIBIOS_SUCCESSFUL;
> +}
> +
> +static int altera_pcie_cfg_write(struct pci_bus *bus, unsigned int devfn,
> +                               int where, int size, u32 value)
> +{
> +       struct altera_pcie *pcie = sys_to_pcie(bus->sysdata);
> +       u32 data32 = value;
> +       u32 shift = 8 * (where & 3);
> +       int ret;
> +
> +       if (!altera_pcie_valid_config(pcie, bus, PCI_SLOT(devfn)))
> +               return PCIBIOS_DEVICE_NOT_FOUND;
> +
> +       /* write partial */
> +       if (size != sizeof(u32)) {
> +               ret = tlp_cfg_dword_read(pcie, bus->number, devfn,
> +                       where & ~DWORD_MASK, &data32);
> +               if (ret)
> +                       return ret;
> +
> +               if (size == 2)
> +                       data32 = (data32 & ~(0xffff << shift)) |
> +                                       ((value & 0xffff) << shift);
> +                else if (size == 1)
> +                       data32 = (data32 & ~(0xff << shift)) |
> +                                       ((value & 0xff) << shift);
> +       }
> +
> +       return tlp_cfg_dword_write(pcie, bus->number, devfn,
> +               (where & ~DWORD_MASK), data32);
> +}
> +
> +static struct pci_ops altera_pcie_ops = {
> +       .read = altera_pcie_cfg_read,
> +       .write = altera_pcie_cfg_write,
> +};
> +
> +static int altera_pcie_setup(int nr, struct pci_sys_data *sys)
> +{
> +       struct altera_pcie *pcie = sys->private_data;
> +
> +       list_splice_init(&pcie->resources, &sys->resources);
> +
> +       return 1;
> +}
> +
> +static int altera_pcie_map_irq(const struct pci_dev *pdev, u8 slot, u8 pin)
> +{
> +       struct altera_pcie *pcie = sys_to_pcie(pdev->bus->sysdata);
> +       int irq;
> +
> +       irq = of_irq_parse_and_map_pci(pdev, slot, pin);
> +
> +       if (!irq)
> +               irq = pcie->hwirq;
> +
> +       return irq;
> +}

This should not be needed as the core code should take care of this.

> +
> +static struct pci_bus *altera_pcie_scan_bus(int nr, struct pci_sys_data *sys)
> +{
> +       struct altera_pcie *pcie = sys_to_pcie(sys);
> +       struct pci_bus *bus;
> +
> +       pcie->root_bus_nr = sys->busnr;
> +       bus = pci_scan_root_bus(&pcie->pdev->dev, sys->busnr, &altera_pcie_ops,
> +                               sys, &sys->resources);
> +
> +       return bus;
> +}
> +
> +static struct hw_pci altera_pcie_hw __initdata = {
> +#ifdef CONFIG_PCI_DOMAINS
> +       .domain                 = 0,
> +#endif
> +       .nr_controllers         = 1,
> +       .ops                    = &altera_pcie_ops,
> +       .setup                  = altera_pcie_setup,
> +       .map_irq                        = altera_pcie_map_irq,
> +       .scan                   = altera_pcie_scan_bus,

You should be able to only have an empty hw_pci struct now.

> +};
> +
> +static int altera_pcie_intx_map(struct irq_domain *domain, unsigned int irq,
> +                               irq_hw_number_t hwirq)
> +{
> +       irq_set_chip_and_handler(irq, &dummy_irq_chip, handle_simple_irq);
> +       irq_set_chip_data(irq, domain->host_data);
> +       set_irq_flags(irq, IRQF_VALID);
> +
> +       return 0;
> +}
> +
> +static const struct irq_domain_ops intx_domain_ops = {
> +       .map = altera_pcie_intx_map,
> +};
> +
> +static irqreturn_t altera_pcie_isr(int irq, void *arg)
> +{
> +       struct altera_pcie *pcie = arg;
> +       u32 status, i;
> +
> +       status = cra_readl(pcie, P2A_INT_STATUS) & P2A_INT_STS_ALL;
> +
> +       /* clear interrupts */
> +       cra_writel(pcie, status, P2A_INT_STATUS);
> +
> +       for (i = 0; i < INTX_NUM; i++) {
> +               if (status & (1 << i))
> +                       generic_handle_irq(irq_find_mapping(pcie->irq_domain,
> +                                                       i + 1));
> +       }
> +
> +       return IRQ_HANDLED;
> +}
> +
> +static void altera_pcie_release_of_pci_ranges(struct altera_pcie *pcie)
> +{
> +       pci_free_resource_list(&pcie->resources);
> +}
> +
> +static int altera_pcie_parse_request_of_pci_ranges(struct altera_pcie *pcie)
> +{
> +       int err, res_valid = 0;
> +       struct device *dev = &pcie->pdev->dev;
> +       struct device_node *np = dev->of_node;
> +       resource_size_t iobase;
> +       struct resource_entry *win;
> +       int offset = 0;
> +
> +       err = of_pci_get_host_bridge_resources(np, 0, 0xff, &pcie->resources,
> +                                              &iobase);
> +       if (err)
> +               return err;
> +
> +       resource_list_for_each_entry(win, &pcie->resources) {
> +               struct resource *parent, *res = win->res;
> +
> +               switch (resource_type(res)) {
> +               case IORESOURCE_MEM:
> +                       parent = &iomem_resource;
> +                       res_valid |= !(res->flags & IORESOURCE_PREFETCH);
> +                       cra_writel(pcie, res->start,
> +                               A2P_ADDR_MAP_LO0 + offset);
> +                       cra_writel(pcie, 0,
> +                               A2P_ADDR_MAP_HI0 + offset);
> +                       offset += ATT_ENTRY_SIZE;
> +                       break;
> +               default:
> +                       continue;
> +               }
> +
> +               err = devm_request_resource(dev, parent, res);
> +               if (err)
> +                       goto out_release_res;
> +       }
> +
> +       if (!res_valid) {
> +               dev_err(dev, "non-prefetchable memory resource required\n");
> +               err = -EINVAL;
> +               goto out_release_res;
> +       }
> +
> +       return 0;
> +
> +out_release_res:
> +       altera_pcie_release_of_pci_ranges(pcie);
> +       return err;
> +}
> +
> +static void altera_pcie_free_irq_domain(struct altera_pcie *pcie)
> +{
> +       int i;
> +       u32 irq;
> +
> +       for (i = 0; i < INTX_NUM; i++) {
> +               irq = irq_find_mapping(pcie->irq_domain, i);
> +               if (irq > 0)
> +                       irq_dispose_mapping(irq);
> +       }
> +
> +       irq_domain_remove(pcie->irq_domain);
> +}
> +
> +static int altera_pcie_init_irq_domain(struct altera_pcie *pcie)
> +{
> +       struct device *dev = &pcie->pdev->dev;
> +       struct device_node *node = dev->of_node;
> +
> +       /* Setup INTx */
> +       pcie->irq_domain = irq_domain_add_linear(node, INTX_NUM,
> +                                       &intx_domain_ops, pcie);
> +       if (!pcie->irq_domain) {
> +               dev_err(dev, "Failed to get a INTx IRQ domain\n");
> +               return PTR_ERR(pcie->irq_domain);
> +       }
> +
> +       return 0;
> +}
> +
> +static int altera_pcie_parse_dt(struct altera_pcie *pcie)
> +{
> +       struct resource *cra;
> +       int ret;
> +       struct platform_device *pdev = pcie->pdev;
> +
> +       cra = platform_get_resource_byname(pdev, IORESOURCE_MEM, "Cra");
> +       pcie->cra_base = devm_ioremap_resource(&pdev->dev, cra);
> +       if (IS_ERR(pcie->cra_base)) {
> +               dev_err(&pdev->dev, "get Cra resource failed\n");
> +               return PTR_ERR(pcie->cra_base);
> +       }
> +
> +       /* setup IRQ */
> +       pcie->hwirq = platform_get_irq(pdev, 0);
> +       if (pcie->hwirq <= 0) {
> +               dev_err(&pdev->dev, "failed to get IRQ: %d\n", pcie->hwirq);
> +               return -EINVAL;
> +       }
> +       ret = devm_request_irq(&pdev->dev, pcie->hwirq, altera_pcie_isr,
> +                       IRQF_SHARED, pdev->name, pcie);
> +
> +       if (ret) {
> +               dev_err(&pdev->dev, "failed to request irq %d\n", pcie->hwirq);
> +               return ret;
> +       }
> +
> +       return 0;
> +}
> +
> +static int altera_pcie_probe(struct platform_device *pdev)
> +{
> +       struct altera_pcie *pcie;
> +       int ret;
> +
> +       pcie = devm_kzalloc(&pdev->dev, sizeof(*pcie), GFP_KERNEL);
> +       if (!pcie)
> +               return -ENOMEM;
> +
> +       pcie->pdev = pdev;
> +
> +       ret = altera_pcie_parse_dt(pcie);
> +       if (ret) {
> +               dev_err(&pdev->dev, "Parsing DT failed\n");
> +               return ret;
> +       }
> +
> +       INIT_LIST_HEAD(&pcie->resources);
> +
> +       ret = altera_pcie_parse_request_of_pci_ranges(pcie);
> +       if (ret) {
> +               dev_err(&pdev->dev, "Failed add resources\n");
> +               return ret;
> +       }
> +
> +       ret = altera_pcie_init_irq_domain(pcie);

I don't think you need an irq domain here.

> +       if (ret) {
> +               dev_err(&pdev->dev, "Failed creating IRQ Domain\n");
> +               return ret;
> +       }
> +
> +       pcie->root_bus_nr = -1;
> +
> +       /* clear all interrupts */
> +       cra_writel(pcie, P2A_INT_STS_ALL, P2A_INT_STATUS);
> +       /* enable all interrupts */
> +       cra_writel(pcie, P2A_INT_ENA_ALL, P2A_INT_ENABLE);
> +
> +       altera_pcie_hw.private_data = (void **)&pcie;
> +
> +       pci_common_init_dev(&pdev->dev, &altera_pcie_hw);

You should not call this, but call pci_scan_root_bus directly. See
pci-host-generic.c or pci-versatile.c for examples.

Rob

Re: [PATCH 3/6] pci:host: Add Altera PCIe host controller driver

[Ley Foon Tan]

On Wed, Jul 29, 2015 at 11:43 AM, Rob Herring <robherring2@gmail.com> wrote:
> On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan@altera.com> wrote:
>> This patch adds the Altera PCIe host controller driver.
>>
>> Signed-off-by: Ley Foon Tan <lftan@altera.com>

>> +
>> +static int altera_pcie_setup(int nr, struct pci_sys_data *sys)
>> +{
>> +       struct altera_pcie *pcie = sys->private_data;
>> +
>> +       list_splice_init(&pcie->resources, &sys->resources);
>> +
>> +       return 1;
>> +}
>> +
>> +static int altera_pcie_map_irq(const struct pci_dev *pdev, u8 slot, u8 pin)
>> +{
>> +       struct altera_pcie *pcie = sys_to_pcie(pdev->bus->sysdata);
>> +       int irq;
>> +
>> +       irq = of_irq_parse_and_map_pci(pdev, slot, pin);
>> +
>> +       if (!irq)
>> +               irq = pcie->hwirq;
>> +
>> +       return irq;
>> +}
>
> This should not be needed as the core code should take care of this.
Okay.

>
>> +
>> +static struct pci_bus *altera_pcie_scan_bus(int nr, struct pci_sys_data *sys)
>> +{
>> +       struct altera_pcie *pcie = sys_to_pcie(sys);
>> +       struct pci_bus *bus;
>> +
>> +       pcie->root_bus_nr = sys->busnr;
>> +       bus = pci_scan_root_bus(&pcie->pdev->dev, sys->busnr, &altera_pcie_ops,
>> +                               sys, &sys->resources);
>> +
>> +       return bus;
>> +}
>> +
>> +static struct hw_pci altera_pcie_hw __initdata = {
>> +#ifdef CONFIG_PCI_DOMAINS
>> +       .domain                 = 0,
>> +#endif
>> +       .nr_controllers         = 1,
>> +       .ops                    = &altera_pcie_ops,
>> +       .setup                  = altera_pcie_setup,
>> +       .map_irq                        = altera_pcie_map_irq,
>> +       .scan                   = altera_pcie_scan_bus,
>
> You should be able to only have an empty hw_pci struct now.
Will remove this.

>> +static void altera_pcie_free_irq_domain(struct altera_pcie *pcie)
>> +{
>> +       int i;
>> +       u32 irq;
>> +
>> +       for (i = 0; i < INTX_NUM; i++) {
>> +               irq = irq_find_mapping(pcie->irq_domain, i);
>> +               if (irq > 0)
>> +                       irq_dispose_mapping(irq);
>> +       }
>> +
>> +       irq_domain_remove(pcie->irq_domain);
>> +}
>> +
>> +static int altera_pcie_init_irq_domain(struct altera_pcie *pcie)
>> +{
>> +       struct device *dev = &pcie->pdev->dev;
>> +       struct device_node *node = dev->of_node;
>> +
>> +       /* Setup INTx */
>> +       pcie->irq_domain = irq_domain_add_linear(node, INTX_NUM,
>> +                                       &intx_domain_ops, pcie);
>> +       if (!pcie->irq_domain) {
>> +               dev_err(dev, "Failed to get a INTx IRQ domain\n");
>> +               return PTR_ERR(pcie->irq_domain);
>> +       }
>> +
>> +       return 0;
>> +}
>> +
>> +static int altera_pcie_parse_dt(struct altera_pcie *pcie)
>> +{
>> +       struct resource *cra;
>> +       int ret;
>> +       struct platform_device *pdev = pcie->pdev;
>> +
>> +       cra = platform_get_resource_byname(pdev, IORESOURCE_MEM, "Cra");
>> +       pcie->cra_base = devm_ioremap_resource(&pdev->dev, cra);
>> +       if (IS_ERR(pcie->cra_base)) {
>> +               dev_err(&pdev->dev, "get Cra resource failed\n");
>> +               return PTR_ERR(pcie->cra_base);
>> +       }
>> +
>> +       /* setup IRQ */
>> +       pcie->hwirq = platform_get_irq(pdev, 0);
>> +       if (pcie->hwirq <= 0) {
>> +               dev_err(&pdev->dev, "failed to get IRQ: %d\n", pcie->hwirq);
>> +               return -EINVAL;
>> +       }
>> +       ret = devm_request_irq(&pdev->dev, pcie->hwirq, altera_pcie_isr,
>> +                       IRQF_SHARED, pdev->name, pcie);
>> +
>> +       if (ret) {
>> +               dev_err(&pdev->dev, "failed to request irq %d\n", pcie->hwirq);
>> +               return ret;
>> +       }
>> +
>> +       return 0;
>> +}
>> +
>> +static int altera_pcie_probe(struct platform_device *pdev)
>> +{
>> +       struct altera_pcie *pcie;
>> +       int ret;
>> +
>> +       pcie = devm_kzalloc(&pdev->dev, sizeof(*pcie), GFP_KERNEL);
>> +       if (!pcie)
>> +               return -ENOMEM;
>> +
>> +       pcie->pdev = pdev;
>> +
>> +       ret = altera_pcie_parse_dt(pcie);
>> +       if (ret) {
>> +               dev_err(&pdev->dev, "Parsing DT failed\n");
>> +               return ret;
>> +       }
>> +
>> +       INIT_LIST_HEAD(&pcie->resources);
>> +
>> +       ret = altera_pcie_parse_request_of_pci_ranges(pcie);
>> +       if (ret) {
>> +               dev_err(&pdev->dev, "Failed add resources\n");
>> +               return ret;
>> +       }
>> +
>> +       ret = altera_pcie_init_irq_domain(pcie);
>
> I don't think you need an irq domain here.
The controller have one single interrupt  for INTx. So, we need IRQ
domain for map and decode
the 4 INTx interrupt and route them to this domain.

>
>> +       if (ret) {
>> +               dev_err(&pdev->dev, "Failed creating IRQ Domain\n");
>> +               return ret;
>> +       }
>> +
>> +       pcie->root_bus_nr = -1;
>> +
>> +       /* clear all interrupts */
>> +       cra_writel(pcie, P2A_INT_STS_ALL, P2A_INT_STATUS);
>> +       /* enable all interrupts */
>> +       cra_writel(pcie, P2A_INT_ENA_ALL, P2A_INT_ENABLE);
>> +
>> +       altera_pcie_hw.private_data = (void **)&pcie;
>> +
>> +       pci_common_init_dev(&pdev->dev, &altera_pcie_hw);
>
> You should not call this, but call pci_scan_root_bus directly. See
> pci-host-generic.c or pci-versatile.c for examples.
Okay, will change to pci_scan_root_bus.

Thanks.

Regards
Ley Foon

Re: [PATCH 3/6] pci:host: Add Altera PCIe host controller driver

[Paul Bolle]

On di, 2015-07-28 at 18:45 +0800, Ley Foon Tan wrote:
> --- a/drivers/pci/host/Kconfig
> +++ b/drivers/pci/host/Kconfig

> +config PCIE_ALTERA
> +	bool "Altera PCIe controller"
> +	depends on ARCH_SOCFPGA
> +	depends on OF
> +	select PCI_MSI_IRQ_DOMAIN if PCI_MSI
> +	help
> +	  Say Y here if you want to enable PCIe controller support for Altera
> +	  SoCFPGA family of SoCs.

> --- a/drivers/pci/host/Makefile
> +++ b/drivers/pci/host/Makefile

> +obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o

> --- /dev/null
> +++ b/drivers/pci/host/pcie-altera.c

> +#include <linux/module.h>

> +static const struct of_device_id altera_pcie_of_match[] = {
> +	{ .compatible = "altr,pcie-root-port-1.0", },
> +	{},
> +};
> +MODULE_DEVICE_TABLE(of, altera_pcie_of_match);


> +static struct platform_driver altera_pcie_driver = {
> +	[...]
> +		.owner	= THIS_MODULE,
> +	]...]
> +};
> +
> +module_platform_driver(altera_pcie_driver);
> +
> +MODULE_AUTHOR("Ley Foon Tan <lftan@altera.com>");
> +MODULE_DESCRIPTION("Altera PCIe host controller driver");
> +MODULE_LICENSE("GPL v2");

PCIE_ALTERA is a bool symbol. So pcie-altera.o is built-in only. Yet
pcie-altera.c uses a number of module-specific constructs. Should
PCIE_ALTERA perhaps be tristate?

Likewise for PCIE_ALTERA_MSI in 4/6.

Thanks,


Paul Bolle

Re: [PATCH 3/6] pci:host: Add Altera PCIe host controller driver

[Ley Foon Tan]

On Wed, Jul 29, 2015 at 4:35 PM, Paul Bolle <pebolle-IWqWACnzNjzz+pZb47iToQ@public.gmane.org> wrote:
> On di, 2015-07-28 at 18:45 +0800, Ley Foon Tan wrote:
>> --- a/drivers/pci/host/Kconfig
>> +++ b/drivers/pci/host/Kconfig
>
>> +config PCIE_ALTERA
>> +     bool "Altera PCIe controller"
>> +     depends on ARCH_SOCFPGA
>> +     depends on OF
>> +     select PCI_MSI_IRQ_DOMAIN if PCI_MSI
>> +     help
>> +       Say Y here if you want to enable PCIe controller support for Altera
>> +       SoCFPGA family of SoCs.
>
>> --- a/drivers/pci/host/Makefile
>> +++ b/drivers/pci/host/Makefile
>
>> +obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o
>
>> --- /dev/null
>> +++ b/drivers/pci/host/pcie-altera.c
>
>> +#include <linux/module.h>
>
>> +static const struct of_device_id altera_pcie_of_match[] = {
>> +     { .compatible = "altr,pcie-root-port-1.0", },
>> +     {},
>> +};
>> +MODULE_DEVICE_TABLE(of, altera_pcie_of_match);
>
>
>> +static struct platform_driver altera_pcie_driver = {
>> +     [...]
>> +             .owner  = THIS_MODULE,
>> +     ]...]
>> +};
>> +
>> +module_platform_driver(altera_pcie_driver);
>> +
>> +MODULE_AUTHOR("Ley Foon Tan <lftan-EIB2kfCEclfQT0dZR+AlfA@public.gmane.org>");
>> +MODULE_DESCRIPTION("Altera PCIe host controller driver");
>> +MODULE_LICENSE("GPL v2");
>
> PCIE_ALTERA is a bool symbol. So pcie-altera.o is built-in only. Yet
> pcie-altera.c uses a number of module-specific constructs. Should
> PCIE_ALTERA perhaps be tristate?
Yes, PCIE_ALTERA can be changed to tristate after we remove call to
pci_common_init_dev to now.
pci_common_init_dev is not a exported symbol.


> Likewise for PCIE_ALTERA_MSI in 4/6.
PCIE_ALTERA_MSI can only be built-in only. It depends on quite a
number of non-exported symbols.
Eg: pci_msi_mask_irq, pci_msi_create_irq_domain, irq_domain_set_info and etc..

Thanks.

Regards
Ley Foon
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/6] pci:host: Add Altera PCIe host controller driver

[Lorenzo Pieralisi]

On Tue, Jul 28, 2015 at 11:45:42AM +0100, Ley Foon Tan wrote:

[...]

> +static int altera_pcie_parse_request_of_pci_ranges(struct altera_pcie *pcie)
> +{
> +       int err, res_valid = 0;
> +       struct device *dev = &pcie->pdev->dev;
> +       struct device_node *np = dev->of_node;
> +       resource_size_t iobase;
> +       struct resource_entry *win;
> +       int offset = 0;
> +
> +       err = of_pci_get_host_bridge_resources(np, 0, 0xff, &pcie->resources,
> +                                              &iobase);
> +       if (err)
> +               return err;

On top of Rob's comments on ARM bios32 dependency removal (ie rewrite
the driver so that it does not use pci_common_init_dev()), if you need IO
access you have to map iobase, see pci_remap_iospace() in pci-host-generic.c

Lorenzo

> +
> +       resource_list_for_each_entry(win, &pcie->resources) {
> +               struct resource *parent, *res = win->res;
> +
> +               switch (resource_type(res)) {
> +               case IORESOURCE_MEM:
> +                       parent = &iomem_resource;
> +                       res_valid |= !(res->flags & IORESOURCE_PREFETCH);
> +                       cra_writel(pcie, res->start,
> +                               A2P_ADDR_MAP_LO0 + offset);
> +                       cra_writel(pcie, 0,
> +                               A2P_ADDR_MAP_HI0 + offset);
> +                       offset += ATT_ENTRY_SIZE;
> +                       break;
> +               default:
> +                       continue;
> +               }
> +
> +               err = devm_request_resource(dev, parent, res);
> +               if (err)
> +                       goto out_release_res;
> +       }
> +
> +       if (!res_valid) {
> +               dev_err(dev, "non-prefetchable memory resource required\n");
> +               err = -EINVAL;
> +               goto out_release_res;
> +       }
> +
> +       return 0;
> +
> +out_release_res:
> +       altera_pcie_release_of_pci_ranges(pcie);
> +       return err;
> +}
> +
> +static void altera_pcie_free_irq_domain(struct altera_pcie *pcie)
> +{
> +       int i;
> +       u32 irq;
> +
> +       for (i = 0; i < INTX_NUM; i++) {
> +               irq = irq_find_mapping(pcie->irq_domain, i);
> +               if (irq > 0)
> +                       irq_dispose_mapping(irq);
> +       }
> +
> +       irq_domain_remove(pcie->irq_domain);
> +}
> +
> +static int altera_pcie_init_irq_domain(struct altera_pcie *pcie)
> +{
> +       struct device *dev = &pcie->pdev->dev;
> +       struct device_node *node = dev->of_node;
> +
> +       /* Setup INTx */
> +       pcie->irq_domain = irq_domain_add_linear(node, INTX_NUM,
> +                                       &intx_domain_ops, pcie);
> +       if (!pcie->irq_domain) {
> +               dev_err(dev, "Failed to get a INTx IRQ domain\n");
> +               return PTR_ERR(pcie->irq_domain);
> +       }
> +
> +       return 0;
> +}
> +
> +static int altera_pcie_parse_dt(struct altera_pcie *pcie)
> +{
> +       struct resource *cra;
> +       int ret;
> +       struct platform_device *pdev = pcie->pdev;
> +
> +       cra = platform_get_resource_byname(pdev, IORESOURCE_MEM, "Cra");
> +       pcie->cra_base = devm_ioremap_resource(&pdev->dev, cra);
> +       if (IS_ERR(pcie->cra_base)) {
> +               dev_err(&pdev->dev, "get Cra resource failed\n");
> +               return PTR_ERR(pcie->cra_base);
> +       }
> +
> +       /* setup IRQ */
> +       pcie->hwirq = platform_get_irq(pdev, 0);
> +       if (pcie->hwirq <= 0) {
> +               dev_err(&pdev->dev, "failed to get IRQ: %d\n", pcie->hwirq);
> +               return -EINVAL;
> +       }
> +       ret = devm_request_irq(&pdev->dev, pcie->hwirq, altera_pcie_isr,
> +                       IRQF_SHARED, pdev->name, pcie);
> +
> +       if (ret) {
> +               dev_err(&pdev->dev, "failed to request irq %d\n", pcie->hwirq);
> +               return ret;
> +       }
> +
> +       return 0;
> +}
> +
> +static int altera_pcie_probe(struct platform_device *pdev)
> +{
> +       struct altera_pcie *pcie;
> +       int ret;
> +
> +       pcie = devm_kzalloc(&pdev->dev, sizeof(*pcie), GFP_KERNEL);
> +       if (!pcie)
> +               return -ENOMEM;
> +
> +       pcie->pdev = pdev;
> +
> +       ret = altera_pcie_parse_dt(pcie);
> +       if (ret) {
> +               dev_err(&pdev->dev, "Parsing DT failed\n");
> +               return ret;
> +       }
> +
> +       INIT_LIST_HEAD(&pcie->resources);
> +
> +       ret = altera_pcie_parse_request_of_pci_ranges(pcie);
> +       if (ret) {
> +               dev_err(&pdev->dev, "Failed add resources\n");
> +               return ret;
> +       }
> +
> +       ret = altera_pcie_init_irq_domain(pcie);
> +       if (ret) {
> +               dev_err(&pdev->dev, "Failed creating IRQ Domain\n");
> +               return ret;
> +       }
> +
> +       pcie->root_bus_nr = -1;
> +
> +       /* clear all interrupts */
> +       cra_writel(pcie, P2A_INT_STS_ALL, P2A_INT_STATUS);
> +       /* enable all interrupts */
> +       cra_writel(pcie, P2A_INT_ENA_ALL, P2A_INT_ENABLE);
> +
> +       altera_pcie_hw.private_data = (void **)&pcie;
> +
> +       pci_common_init_dev(&pdev->dev, &altera_pcie_hw);
> +
> +       platform_set_drvdata(pdev, pcie);
> +       return ret;
> +}
> +
> +static int __exit altera_pcie_remove(struct platform_device *pdev)
> +{
> +       struct altera_pcie *pcie = platform_get_drvdata(pdev);
> +
> +       altera_pcie_free_irq_domain(pcie);
> +       platform_set_drvdata(pdev, NULL);
> +       return 0;
> +}
> +
> +static const struct of_device_id altera_pcie_of_match[] = {
> +       { .compatible = "altr,pcie-root-port-1.0", },
> +       {},
> +};
> +MODULE_DEVICE_TABLE(of, altera_pcie_of_match);
> +
> +static struct platform_driver altera_pcie_driver = {
> +       .probe          = altera_pcie_probe,
> +       .remove         = altera_pcie_remove,
> +       .driver = {
> +               .name   = "altera-pcie",
> +               .owner  = THIS_MODULE,
> +               .of_match_table = altera_pcie_of_match,
> +       },
> +};
> +
> +module_platform_driver(altera_pcie_driver);
> +
> +MODULE_AUTHOR("Ley Foon Tan <lftan@altera.com>");
> +MODULE_DESCRIPTION("Altera PCIe host controller driver");
> +MODULE_LICENSE("GPL v2");
> --
> 1.8.2.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [PATCH 3/6] pci:host: Add Altera PCIe host controller driver

[Ley Foon Tan]

On Wed, Jul 29, 2015 at 9:19 PM, Lorenzo Pieralisi
<lorenzo.pieralisi@arm.com> wrote:
> On Tue, Jul 28, 2015 at 11:45:42AM +0100, Ley Foon Tan wrote:
>
> [...]
>
>> +static int altera_pcie_parse_request_of_pci_ranges(struct altera_pcie *pcie)
>> +{
>> +       int err, res_valid = 0;
>> +       struct device *dev = &pcie->pdev->dev;
>> +       struct device_node *np = dev->of_node;
>> +       resource_size_t iobase;
>> +       struct resource_entry *win;
>> +       int offset = 0;
>> +
>> +       err = of_pci_get_host_bridge_resources(np, 0, 0xff, &pcie->resources,
>> +                                              &iobase);
>> +       if (err)
>> +               return err;
>
> On top of Rob's comments on ARM bios32 dependency removal (ie rewrite
> the driver so that it does not use pci_common_init_dev()), if you need IO
> access you have to map iobase, see pci_remap_iospace() in pci-host-generic.c
Thanks. We doesn't support I/O region.

Regards
Ley Foon

[PATCH 4/6] pci: altera: Add Altera PCIe MSI driver

[Ley Foon Tan]

This patch adds Altera PCIe MSI driver. This soft IP supports configurable
number of vectors, which is a dts parameter.

Signed-off-by: Ley Foon Tan <lftan@altera.com>
---
 drivers/pci/host/Kconfig           |   7 +
 drivers/pci/host/Makefile          |   1 +
 drivers/pci/host/pcie-altera-msi.c | 318 +++++++++++++++++++++++++++++++++++++
 3 files changed, 326 insertions(+)
 create mode 100644 drivers/pci/host/pcie-altera-msi.c

diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
index af19039..a8b87fd 100644
--- a/drivers/pci/host/Kconfig
+++ b/drivers/pci/host/Kconfig
@@ -154,4 +154,11 @@ config PCIE_ALTERA
 	  Say Y here if you want to enable PCIe controller support for Altera
 	  SoCFPGA family of SoCs.
 
+config PCIE_ALTERA_MSI
+	bool "Altera PCIe MSI feature"
+	depends on PCI_MSI && PCIE_ALTERA
+	help
+	  Say Y here if you want PCIe MSI support for the Altera SocFPGA SoC.
+	  This MSI driver supports Altera MSI to GIC controller IP.
+
 endmenu
diff --git a/drivers/pci/host/Makefile b/drivers/pci/host/Makefile
index 6954f76..6c4913d 100644
--- a/drivers/pci/host/Makefile
+++ b/drivers/pci/host/Makefile
@@ -18,3 +18,4 @@ obj-$(CONFIG_PCIE_IPROC) += pcie-iproc.o
 obj-$(CONFIG_PCIE_IPROC_PLATFORM) += pcie-iproc-platform.o
 obj-$(CONFIG_PCIE_IPROC_BCMA) += pcie-iproc-bcma.o
 obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o
+obj-$(CONFIG_PCIE_ALTERA_MSI) += pcie-altera-msi.o
diff --git a/drivers/pci/host/pcie-altera-msi.c b/drivers/pci/host/pcie-altera-msi.c
new file mode 100644
index 0000000..b852b51
--- /dev/null
+++ b/drivers/pci/host/pcie-altera-msi.c
@@ -0,0 +1,318 @@
+/*
+ * Copyright Altera Corporation (C) 2013-2015. All rights reserved
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/msi.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/of_pci.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+
+#define MSI_STATUS		0x0
+#define MSI_ERROR		0x4
+#define MSI_INTMASK		0x8
+
+#define MAX_MSI_VECTORS		32
+struct altera_msi {
+	DECLARE_BITMAP(used, MAX_MSI_VECTORS);
+	struct mutex		lock;	/* proctect used variable */
+	struct platform_device	*pdev;
+	struct irq_domain		*msi_domain;
+	void __iomem		*csr_base;
+	void __iomem		*vector_base;
+	u32			vector_phy;
+	u32			num_of_vectors;
+	int			irq;
+};
+
+static inline void msi_writel(struct altera_msi *msi, u32 value, u32 reg)
+{
+	writel(value, msi->csr_base + reg);
+}
+
+static inline u32 msi_readl(struct altera_msi *msi, u32 reg)
+{
+	return readl(msi->csr_base + reg);
+}
+
+static irqreturn_t altera_msi_isr(int irq, void *data)
+{
+	struct altera_msi *msi = data;
+	unsigned long status;
+	u32 num_of_vectors = msi->num_of_vectors;
+	u32 processed = 0;
+	u32 offset;
+
+	do {
+		status = msi_readl(msi, MSI_STATUS);
+		if (!status)
+			break;
+
+		do {
+			offset = find_first_bit(&status, num_of_vectors);
+			/* Dummy read from vector to clear the interrupt */
+			readl(msi->vector_base + (offset * sizeof(u32)));
+
+			irq = irq_find_mapping(msi->msi_domain->parent, offset);
+			if (irq) {
+				if (test_bit(offset, msi->used))
+					generic_handle_irq(irq);
+				else
+					dev_info(&msi->pdev->dev, "unhandled MSI\n");
+			} else
+				dev_info(&msi->pdev->dev, "unexpected MSI\n");
+
+			/* Clear the bit from status and repeat without reading
+			 * again status register. */
+			clear_bit(offset, &status);
+			processed++;
+		} while (status);
+	} while (1);
+
+	return processed > 0 ? IRQ_HANDLED : IRQ_NONE;
+}
+
+static struct irq_chip altera_msi_irq_chip = {
+	.name = "Altera PCIe MSI",
+	.irq_enable = pci_msi_unmask_irq,
+	.irq_disable = pci_msi_mask_irq,
+	.irq_mask = pci_msi_mask_irq,
+	.irq_unmask = pci_msi_unmask_irq,
+};
+
+static struct msi_domain_info altera_msi_domain_info = {
+	.flags	= (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),
+	.chip	= &altera_msi_irq_chip,
+};
+
+static void altera_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
+{
+	struct altera_msi *msi = irq_data_get_irq_chip_data(data);
+	u32 mask;
+
+	msg->address_lo = msi->vector_phy + (data->hwirq * sizeof(u32));
+	/* 32 bit address only */
+	msg->address_hi = 0;
+	msg->data = data->hwirq;
+
+	mask = msi_readl(msi, MSI_INTMASK);
+	mask |= 1 << data->hwirq;
+	msi_writel(msi, mask, MSI_INTMASK);
+	dev_dbg(&msi->pdev->dev, "msi#%d address_lo 0x%x\n", (int)data->hwirq,
+		msg->address_lo);
+}
+
+static int altera_msi_set_affinity(struct irq_data *irq_data,
+				 const struct cpumask *mask, bool force)
+{
+	 return irq_set_affinity(irq_data->hwirq, mask);
+}
+
+static struct irq_chip altera_msi_bottom_irq_chip = {
+	.name			= "Altera MSI",
+	.irq_compose_msi_msg	= altera_compose_msi_msg,
+	.irq_set_affinity	= altera_msi_set_affinity,
+};
+
+static int altera_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
+				  unsigned int nr_irqs, void *args)
+{
+	struct altera_msi *msi = domain->host_data;
+	int bit;
+
+	mutex_lock(&msi->lock);
+
+	bit = find_first_zero_bit(msi->used, msi->num_of_vectors);
+	if (bit < msi->num_of_vectors)
+		set_bit(bit, msi->used);
+	else
+		bit = -ENOSPC;
+
+	mutex_unlock(&msi->lock);
+
+	if (bit < 0)
+		return bit;
+
+	irq_domain_set_info(domain, virq, bit, &altera_msi_bottom_irq_chip,
+			    domain->host_data, handle_simple_irq,
+			    NULL, NULL);
+	set_irq_flags(virq, IRQF_VALID);
+
+	return 0;
+}
+
+static void altera_irq_domain_free(struct irq_domain *domain,
+				  unsigned int virq, unsigned int nr_irqs)
+{
+	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
+	struct altera_msi *msi = irq_data_get_irq_chip_data(d);
+	u32 mask;
+
+	mutex_lock(&msi->lock);
+
+	if (!test_bit(d->hwirq, msi->used))
+		dev_err(&msi->pdev->dev, "trying to free unused MSI#%lu\n",
+		d->hwirq);
+	else {
+		clear_bit(d->hwirq, msi->used);
+		mask = msi_readl(msi, MSI_INTMASK);
+		mask &= ~(1 << d->hwirq);
+		msi_writel(msi, mask, MSI_INTMASK);
+	}
+
+	mutex_unlock(&msi->lock);
+}
+
+static const struct irq_domain_ops msi_domain_ops = {
+	.alloc	= altera_irq_domain_alloc,
+	.free	= altera_irq_domain_free,
+};
+
+static int altera_allocate_domains(struct altera_msi *msi)
+{
+	struct irq_domain *inner_domain;
+
+	inner_domain = irq_domain_add_linear(NULL, msi->num_of_vectors,
+					     &msi_domain_ops, msi);
+	if (!inner_domain) {
+		dev_err(&msi->pdev->dev, "failed to create IRQ domain\n");
+		return -ENOMEM;
+	}
+
+	msi->msi_domain = pci_msi_create_irq_domain(
+				msi->pdev->dev.of_node,
+				&altera_msi_domain_info, inner_domain);
+	if (!msi->msi_domain) {
+		dev_err(&msi->pdev->dev, "failed to create MSI domain\n");
+		irq_domain_remove(inner_domain);
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
+static void altera_free_domains(struct altera_msi *msi)
+{
+	struct irq_domain *inner_domain = msi->msi_domain->parent;
+
+	irq_domain_remove(msi->msi_domain);
+	irq_domain_remove(inner_domain);
+}
+
+int altera_msi_probe(struct platform_device *pdev)
+{
+	struct altera_msi *msi;
+	struct device_node *np = pdev->dev.of_node;
+	struct resource *res;
+	int ret;
+
+	msi = devm_kzalloc(&pdev->dev, sizeof(struct altera_msi),
+		GFP_KERNEL);
+	if (!msi)
+		return -ENOMEM;
+
+	mutex_init(&msi->lock);
+	msi->pdev = pdev;
+
+	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "csr");
+	msi->csr_base = devm_ioremap_resource(&pdev->dev, res);
+	if (IS_ERR(msi->csr_base)) {
+		dev_err(&pdev->dev, "get csr resource failed\n");
+		return -EADDRNOTAVAIL;
+	}
+
+	res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+			"vector_slave");
+	msi->vector_base = devm_ioremap_resource(&pdev->dev, res);
+	if (IS_ERR(msi->vector_base)) {
+		dev_err(&pdev->dev, "get vector slave resource failed\n");
+		return -EADDRNOTAVAIL;
+	}
+
+	msi->vector_phy = res->start;
+
+	if (of_property_read_u32(np, "num-vectors", &msi->num_of_vectors)) {
+		dev_err(&pdev->dev, "failed to parse the number of vectors\n");
+		return -EINVAL;
+	}
+
+	ret = altera_allocate_domains(msi);
+	if (ret)
+		return ret;
+
+	msi->irq = platform_get_irq(pdev, 0);
+	if (msi->irq <= 0) {
+		dev_err(&pdev->dev, "failed to map IRQ: %d\n", msi->irq);
+		ret = -ENODEV;
+		goto err;
+	}
+
+	ret = devm_request_irq(&pdev->dev, msi->irq, altera_msi_isr, 0,
+				altera_msi_irq_chip.name, msi);
+	if (ret) {
+		dev_err(&pdev->dev, "failed to request IRQ: %d\n", ret);
+		goto err;
+	}
+
+	platform_set_drvdata(pdev, msi);
+
+	return 0;
+
+err:
+	irq_domain_remove(msi->msi_domain);
+	return ret;
+}
+
+static int altera_msi_remove(struct platform_device *pdev)
+{
+	struct altera_msi *msi = platform_get_drvdata(pdev);
+
+	msi_writel(msi, 0, MSI_INTMASK);
+
+	altera_free_domains(msi);
+
+	platform_set_drvdata(pdev, NULL);
+	return 0;
+}
+
+static const struct of_device_id altera_msi_of_match[] = {
+	{ .compatible = "altr,msi-1.0", NULL },
+	{ },
+};
+MODULE_DEVICE_TABLE(of, altera_msi_of_match);
+
+static struct platform_driver altera_msi_driver = {
+	.driver = {
+		.name = "altera-msi",
+		.owner = THIS_MODULE,
+		.of_match_table = altera_msi_of_match,
+	},
+	.probe = altera_msi_probe,
+	.remove = altera_msi_remove,
+};
+
+static int __init altera_msi_init(void)
+{
+	return platform_driver_register(&altera_msi_driver);
+}
+
+subsys_initcall(altera_msi_init);
+
+MODULE_AUTHOR("Ley Foon Tan <lftan@altera.com>");
+MODULE_DESCRIPTION("Altera PCIe MSI support");
+MODULE_LICENSE("GPL v2");
-- 
1.8.2.1

Re: [PATCH 4/6] pci: altera: Add Altera PCIe MSI driver

[Dinh Nguyen]

On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan@altera.com> wrote:
> This patch adds Altera PCIe MSI driver. This soft IP supports configurable
> number of vectors, which is a dts parameter.
>
> Signed-off-by: Ley Foon Tan <lftan@altera.com>
> ---
>  drivers/pci/host/Kconfig           |   7 +
>  drivers/pci/host/Makefile          |   1 +
>  drivers/pci/host/pcie-altera-msi.c | 318 +++++++++++++++++++++++++++++++++++++
>  3 files changed, 326 insertions(+)
>  create mode 100644 drivers/pci/host/pcie-altera-msi.c
>
> diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
> index af19039..a8b87fd 100644
> --- a/drivers/pci/host/Kconfig
> +++ b/drivers/pci/host/Kconfig
> @@ -154,4 +154,11 @@ config PCIE_ALTERA
>           Say Y here if you want to enable PCIe controller support for Altera
>           SoCFPGA family of SoCs.
>
> +config PCIE_ALTERA_MSI
> +       bool "Altera PCIe MSI feature"
> +       depends on PCI_MSI && PCIE_ALTERA
> +       help
> +         Say Y here if you want PCIe MSI support for the Altera SocFPGA SoC.
> +         This MSI driver supports Altera MSI to GIC controller IP.
> +

Couldn't this driver get combined with the pcie-altera driver?

Dinh

Re: [PATCH 4/6] pci: altera: Add Altera PCIe MSI driver

[Ley Foon Tan]

On Wed, Jul 29, 2015 at 1:00 AM, Dinh Nguyen <dinh.linux@gmail.com> wrote:
> On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan@altera.com> wrote:
>> This patch adds Altera PCIe MSI driver. This soft IP supports configurable
>> number of vectors, which is a dts parameter.
>> @@ -154,4 +154,11 @@ config PCIE_ALTERA
>>           Say Y here if you want to enable PCIe controller support for Altera
>>           SoCFPGA family of SoCs.
>>
>> +config PCIE_ALTERA_MSI
>> +       bool "Altera PCIe MSI feature"
>> +       depends on PCI_MSI && PCIE_ALTERA
>> +       help
>> +         Say Y here if you want PCIe MSI support for the Altera SocFPGA SoC.
>> +         This MSI driver supports Altera MSI to GIC controller IP.
>> +
>
> Couldn't this driver get combined with the pcie-altera driver?

This MSI IP is a soft IP and it is optional to PCI system. So, we have
individual driver for it and user can disable it.

Regards
Ley Foon

Re: [PATCH 4/6] pci: altera: Add Altera PCIe MSI driver

[Dinh Nguyen]

On Tue, Jul 28, 2015 at 10:07 PM, Ley Foon Tan <lftan@altera.com> wrote:
> On Wed, Jul 29, 2015 at 1:00 AM, Dinh Nguyen <dinh.linux@gmail.com> wrote:
>> On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan@altera.com> wrote:
>>> This patch adds Altera PCIe MSI driver. This soft IP supports configurable
>>> number of vectors, which is a dts parameter.
>>> @@ -154,4 +154,11 @@ config PCIE_ALTERA
>>>           Say Y here if you want to enable PCIe controller support for Altera
>>>           SoCFPGA family of SoCs.
>>>
>>> +config PCIE_ALTERA_MSI
>>> +       bool "Altera PCIe MSI feature"
>>> +       depends on PCI_MSI && PCIE_ALTERA
>>> +       help
>>> +         Say Y here if you want PCIe MSI support for the Altera SocFPGA SoC.
>>> +         This MSI driver supports Altera MSI to GIC controller IP.
>>> +
>>
>> Couldn't this driver get combined with the pcie-altera driver?
>
> This MSI IP is a soft IP and it is optional to PCI system. So, we have
> individual driver for it and user can disable it.
>

So why can't you use CONFIG_PCI_MSI in the pcie-altera driver? I see
that most, if not
all of the other drivers are combined with MSI.

Dinh

Re: [PATCH 4/6] pci: altera: Add Altera PCIe MSI driver

[Ley Foon Tan]

On Wed, Jul 29, 2015 at 1:00 AM, Dinh Nguyen <dinh.linux-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Tue, Jul 28, 2015 at 5:45 AM, Ley Foon Tan <lftan-EIB2kfCEclfQT0dZR+AlfA@public.gmane.org> wrote:
>> This patch adds Altera PCIe MSI driver. This soft IP supports configurable
>> number of vectors, which is a dts parameter.
>>
>> Signed-off-by: Ley Foon Tan <lftan-EIB2kfCEclfQT0dZR+AlfA@public.gmane.org>
>> ---
>>  drivers/pci/host/Kconfig           |   7 +
>>  drivers/pci/host/Makefile          |   1 +
>>  drivers/pci/host/pcie-altera-msi.c | 318 +++++++++++++++++++++++++++++++++++++
>>  3 files changed, 326 insertions(+)
>>  create mode 100644 drivers/pci/host/pcie-altera-msi.c
>>
>> diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
>> index af19039..a8b87fd 100644
>> --- a/drivers/pci/host/Kconfig
>> +++ b/drivers/pci/host/Kconfig
>> @@ -154,4 +154,11 @@ config PCIE_ALTERA
>>           Say Y here if you want to enable PCIe controller support for Altera
>>           SoCFPGA family of SoCs.
>>
>> +config PCIE_ALTERA_MSI
>> +       bool "Altera PCIe MSI feature"
>> +       depends on PCI_MSI && PCIE_ALTERA
>> +       help
>> +         Say Y here if you want PCIe MSI support for the Altera SocFPGA SoC.
>> +         This MSI driver supports Altera MSI to GIC controller IP.
>> +
>
> Couldn't this driver get combined with the pcie-altera driver?
Prefer to separate them, since they are 2 separate IPs.
Theoretically, this MSI driver can work with other PCIe host as well.

Thanks.

Regard
Ley Foon
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/6] pci: altera: Add Altera PCIe MSI driver

[Marc Zyngier]

Hi Ley,

On 28/07/15 11:45, Ley Foon Tan wrote:
> This patch adds Altera PCIe MSI driver. This soft IP supports configurable
> number of vectors, which is a dts parameter.

Can't you read this configuration from the HW?

> 
> Signed-off-by: Ley Foon Tan <lftan@altera.com>
> ---
>  drivers/pci/host/Kconfig           |   7 +
>  drivers/pci/host/Makefile          |   1 +
>  drivers/pci/host/pcie-altera-msi.c | 318 +++++++++++++++++++++++++++++++++++++
>  3 files changed, 326 insertions(+)
>  create mode 100644 drivers/pci/host/pcie-altera-msi.c
> 
> diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
> index af19039..a8b87fd 100644
> --- a/drivers/pci/host/Kconfig
> +++ b/drivers/pci/host/Kconfig
> @@ -154,4 +154,11 @@ config PCIE_ALTERA
>  	  Say Y here if you want to enable PCIe controller support for Altera
>  	  SoCFPGA family of SoCs.
>  
> +config PCIE_ALTERA_MSI
> +	bool "Altera PCIe MSI feature"
> +	depends on PCI_MSI && PCIE_ALTERA

What is the dependency with PCIE_ALTERA? Isn't that module standalone?

> +	help
> +	  Say Y here if you want PCIe MSI support for the Altera SocFPGA SoC.
> +	  This MSI driver supports Altera MSI to GIC controller IP.
> +
>  endmenu
> diff --git a/drivers/pci/host/Makefile b/drivers/pci/host/Makefile
> index 6954f76..6c4913d 100644
> --- a/drivers/pci/host/Makefile
> +++ b/drivers/pci/host/Makefile
> @@ -18,3 +18,4 @@ obj-$(CONFIG_PCIE_IPROC) += pcie-iproc.o
>  obj-$(CONFIG_PCIE_IPROC_PLATFORM) += pcie-iproc-platform.o
>  obj-$(CONFIG_PCIE_IPROC_BCMA) += pcie-iproc-bcma.o
>  obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o
> +obj-$(CONFIG_PCIE_ALTERA_MSI) += pcie-altera-msi.o
> diff --git a/drivers/pci/host/pcie-altera-msi.c b/drivers/pci/host/pcie-altera-msi.c
> new file mode 100644
> index 0000000..b852b51
> --- /dev/null
> +++ b/drivers/pci/host/pcie-altera-msi.c
> @@ -0,0 +1,318 @@
> +/*
> + * Copyright Altera Corporation (C) 2013-2015. All rights reserved
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/msi.h>
> +#include <linux/of_address.h>
> +#include <linux/of_irq.h>
> +#include <linux/of_pci.h>
> +#include <linux/pci.h>
> +#include <linux/platform_device.h>
> +#include <linux/slab.h>
> +
> +#define MSI_STATUS		0x0
> +#define MSI_ERROR		0x4
> +#define MSI_INTMASK		0x8
> +
> +#define MAX_MSI_VECTORS		32
> +struct altera_msi {
> +	DECLARE_BITMAP(used, MAX_MSI_VECTORS);
> +	struct mutex		lock;	/* proctect used variable */
> +	struct platform_device	*pdev;
> +	struct irq_domain		*msi_domain;
> +	void __iomem		*csr_base;
> +	void __iomem		*vector_base;
> +	u32			vector_phy;

This should be a phys_addr_t. Not everything is 32bit.

> +	u32			num_of_vectors;
> +	int			irq;
> +};
> +
> +static inline void msi_writel(struct altera_msi *msi, u32 value, u32 reg)
> +{
> +	writel(value, msi->csr_base + reg);

You should be able to use the relaxed accessors.

> +}
> +
> +static inline u32 msi_readl(struct altera_msi *msi, u32 reg)
> +{
> +	return readl(msi->csr_base + reg);

Same here.

> +}
> +
> +static irqreturn_t altera_msi_isr(int irq, void *data)
> +{
> +	struct altera_msi *msi = data;
> +	unsigned long status;
> +	u32 num_of_vectors = msi->num_of_vectors;
> +	u32 processed = 0;
> +	u32 offset;
> +
> +	do {
> +		status = msi_readl(msi, MSI_STATUS);
> +		if (!status)
> +			break;
> +
> +		do {
> +			offset = find_first_bit(&status, num_of_vectors);
> +			/* Dummy read from vector to clear the interrupt */
> +			readl(msi->vector_base + (offset * sizeof(u32)));

readl_relaxed

> +
> +			irq = irq_find_mapping(msi->msi_domain->parent, offset);

This would tend to indicate that you don't really need to store the
msi_domain pointer, but the inner_domain pointer instead.

> +			if (irq) {
> +				if (test_bit(offset, msi->used))
> +					generic_handle_irq(irq);
> +				else
> +					dev_info(&msi->pdev->dev, "unhandled MSI\n");
> +			} else
> +				dev_info(&msi->pdev->dev, "unexpected MSI\n");
> +
> +			/* Clear the bit from status and repeat without reading
> +			 * again status register. */
> +			clear_bit(offset, &status);
> +			processed++;
> +		} while (status);
> +	} while (1);
> +
> +	return processed > 0 ? IRQ_HANDLED : IRQ_NONE;

This shouldn't be a simple interrupt interrupt handler, but instead a
chained irqchip. See pci-xgene-msi.c for an example of such a thing.

> +}
> +
> +static struct irq_chip altera_msi_irq_chip = {
> +	.name = "Altera PCIe MSI",
> +	.irq_enable = pci_msi_unmask_irq,
> +	.irq_disable = pci_msi_mask_irq,
> +	.irq_mask = pci_msi_mask_irq,
> +	.irq_unmask = pci_msi_unmask_irq,
> +};
> +
> +static struct msi_domain_info altera_msi_domain_info = {
> +	.flags	= (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),

So you don't support MSIX? That's a bit weird.

> +	.chip	= &altera_msi_irq_chip,
> +};
> +
> +static void altera_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> +{
> +	struct altera_msi *msi = irq_data_get_irq_chip_data(data);
> +	u32 mask;
> +
> +	msg->address_lo = msi->vector_phy + (data->hwirq * sizeof(u32));

Each MSI has a separate doorbell? Interesting... Please use
lower_32_bits on the above expression.

> +	/* 32 bit address only */
> +	msg->address_hi = 0;

So this HW will never be used in a 64bit platform? Oddly enough, I
cannot believe it. Please use upper_32_bits() as the complement of the
above. At least, we'll be future proof.

> +	msg->data = data->hwirq;
> +
> +	mask = msi_readl(msi, MSI_INTMASK);
> +	mask |= 1 << data->hwirq;
> +	msi_writel(msi, mask, MSI_INTMASK);
> +	dev_dbg(&msi->pdev->dev, "msi#%d address_lo 0x%x\n", (int)data->hwirq,
> +		msg->address_lo);
> +}
> +
> +static int altera_msi_set_affinity(struct irq_data *irq_data,
> +				 const struct cpumask *mask, bool force)
> +{
> +	 return irq_set_affinity(irq_data->hwirq, mask);

There is no way this can be right. irq_data->hwirq can *never* be passed
as a Linux IRQ. This really should be the IRQ to the GIC.

Which raises another issue: as you only have a single interrupt to the
GIC, changing the affinity of a single MSI is going to affect all the
other MSIs as well. This doesn't seem like a desirable behaviour.

> +}
> +
> +static struct irq_chip altera_msi_bottom_irq_chip = {
> +	.name			= "Altera MSI",
> +	.irq_compose_msi_msg	= altera_compose_msi_msg,
> +	.irq_set_affinity	= altera_msi_set_affinity,
> +};
> +
> +static int altera_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
> +				  unsigned int nr_irqs, void *args)
> +{
> +	struct altera_msi *msi = domain->host_data;
> +	int bit;
> +
> +	mutex_lock(&msi->lock);
> +
> +	bit = find_first_zero_bit(msi->used, msi->num_of_vectors);
> +	if (bit < msi->num_of_vectors)
> +		set_bit(bit, msi->used);
> +	else
> +		bit = -ENOSPC;

You can loose the "else" clause...

> +
> +	mutex_unlock(&msi->lock);
> +
> +	if (bit < 0)
> +		return bit;

... and test for bit >= msi->num_of_vectors, returning -ENOSPC if out of
vectors.

> +
> +	irq_domain_set_info(domain, virq, bit, &altera_msi_bottom_irq_chip,
> +			    domain->host_data, handle_simple_irq,
> +			    NULL, NULL);
> +	set_irq_flags(virq, IRQF_VALID);
> +
> +	return 0;
> +}
> +
> +static void altera_irq_domain_free(struct irq_domain *domain,
> +				  unsigned int virq, unsigned int nr_irqs)
> +{
> +	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> +	struct altera_msi *msi = irq_data_get_irq_chip_data(d);
> +	u32 mask;
> +
> +	mutex_lock(&msi->lock);
> +
> +	if (!test_bit(d->hwirq, msi->used))
> +		dev_err(&msi->pdev->dev, "trying to free unused MSI#%lu\n",
> +		d->hwirq);
> +	else {
> +		clear_bit(d->hwirq, msi->used);
> +		mask = msi_readl(msi, MSI_INTMASK);
> +		mask &= ~(1 << d->hwirq);
> +		msi_writel(msi, mask, MSI_INTMASK);
> +	}
> +
> +	mutex_unlock(&msi->lock);
> +}
> +
> +static const struct irq_domain_ops msi_domain_ops = {
> +	.alloc	= altera_irq_domain_alloc,
> +	.free	= altera_irq_domain_free,
> +};
> +
> +static int altera_allocate_domains(struct altera_msi *msi)
> +{
> +	struct irq_domain *inner_domain;
> +
> +	inner_domain = irq_domain_add_linear(NULL, msi->num_of_vectors,
> +					     &msi_domain_ops, msi);
> +	if (!inner_domain) {
> +		dev_err(&msi->pdev->dev, "failed to create IRQ domain\n");
> +		return -ENOMEM;
> +	}
> +
> +	msi->msi_domain = pci_msi_create_irq_domain(
> +				msi->pdev->dev.of_node,
> +				&altera_msi_domain_info, inner_domain);
> +	if (!msi->msi_domain) {
> +		dev_err(&msi->pdev->dev, "failed to create MSI domain\n");
> +		irq_domain_remove(inner_domain);
> +		return -ENOMEM;
> +	}
> +
> +	return 0;
> +}
> +
> +static void altera_free_domains(struct altera_msi *msi)
> +{
> +	struct irq_domain *inner_domain = msi->msi_domain->parent;
> +
> +	irq_domain_remove(msi->msi_domain);
> +	irq_domain_remove(inner_domain);
> +}
> +
> +int altera_msi_probe(struct platform_device *pdev)
> +{
> +	struct altera_msi *msi;
> +	struct device_node *np = pdev->dev.of_node;
> +	struct resource *res;
> +	int ret;
> +
> +	msi = devm_kzalloc(&pdev->dev, sizeof(struct altera_msi),
> +		GFP_KERNEL);
> +	if (!msi)
> +		return -ENOMEM;
> +
> +	mutex_init(&msi->lock);
> +	msi->pdev = pdev;
> +
> +	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "csr");
> +	msi->csr_base = devm_ioremap_resource(&pdev->dev, res);
> +	if (IS_ERR(msi->csr_base)) {
> +		dev_err(&pdev->dev, "get csr resource failed\n");
> +		return -EADDRNOTAVAIL;

You're being quite creative when it comes to error codes. I'd expect
this to be used for networking (pci-tegra also uses it, which is even
more disturbing). I'd be more confident with an -ENOMEM.

> +	}
> +
> +	res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> +			"vector_slave");
> +	msi->vector_base = devm_ioremap_resource(&pdev->dev, res);
> +	if (IS_ERR(msi->vector_base)) {
> +		dev_err(&pdev->dev, "get vector slave resource failed\n");
> +		return -EADDRNOTAVAIL;
> +	}
> +
> +	msi->vector_phy = res->start;
> +
> +	if (of_property_read_u32(np, "num-vectors", &msi->num_of_vectors)) {
> +		dev_err(&pdev->dev, "failed to parse the number of vectors\n");
> +		return -EINVAL;
> +	}

Since this is a configurable IP, you should have a register telling you
the number of configured MSI, shouldn't you? Or is the HW really, really
dumb?

> +
> +	ret = altera_allocate_domains(msi);
> +	if (ret)
> +		return ret;
> +
> +	msi->irq = platform_get_irq(pdev, 0);
> +	if (msi->irq <= 0) {
> +		dev_err(&pdev->dev, "failed to map IRQ: %d\n", msi->irq);
> +		ret = -ENODEV;
> +		goto err;
> +	}
> +
> +	ret = devm_request_irq(&pdev->dev, msi->irq, altera_msi_isr, 0,
> +				altera_msi_irq_chip.name, msi);
> +	if (ret) {
> +		dev_err(&pdev->dev, "failed to request IRQ: %d\n", ret);
> +		goto err;
> +	}

Turn this into a call to irq_set_chained_handler.

> +
> +	platform_set_drvdata(pdev, msi);
> +
> +	return 0;
> +
> +err:
> +	irq_domain_remove(msi->msi_domain);

You're leaking the inner domain here.

> +	return ret;
> +}
> +
> +static int altera_msi_remove(struct platform_device *pdev)
> +{
> +	struct altera_msi *msi = platform_get_drvdata(pdev);
> +
> +	msi_writel(msi, 0, MSI_INTMASK);
> +
> +	altera_free_domains(msi);
> +
> +	platform_set_drvdata(pdev, NULL);
> +	return 0;
> +}
> +
> +static const struct of_device_id altera_msi_of_match[] = {
> +	{ .compatible = "altr,msi-1.0", NULL },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(of, altera_msi_of_match);
> +
> +static struct platform_driver altera_msi_driver = {
> +	.driver = {
> +		.name = "altera-msi",
> +		.owner = THIS_MODULE,
> +		.of_match_table = altera_msi_of_match,
> +	},
> +	.probe = altera_msi_probe,
> +	.remove = altera_msi_remove,
> +};
> +
> +static int __init altera_msi_init(void)
> +{
> +	return platform_driver_register(&altera_msi_driver);
> +}
> +
> +subsys_initcall(altera_msi_init);
> +
> +MODULE_AUTHOR("Ley Foon Tan <lftan@altera.com>");
> +MODULE_DESCRIPTION("Altera PCIe MSI support");
> +MODULE_LICENSE("GPL v2");
> 

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 4/6] pci: altera: Add Altera PCIe MSI driver

[Ley Foon Tan]

On Wed, Jul 29, 2015 at 1:58 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> Hi Ley,
>
> On 28/07/15 11:45, Ley Foon Tan wrote:
>> This patch adds Altera PCIe MSI driver. This soft IP supports configurable
>> number of vectors, which is a dts parameter.
>
> Can't you read this configuration from the HW?
No, we can't read from HW.


>>
>> Signed-off-by: Ley Foon Tan <lftan@altera.com>
>> ---
>>  drivers/pci/host/Kconfig           |   7 +
>>  drivers/pci/host/Makefile          |   1 +
>>  drivers/pci/host/pcie-altera-msi.c | 318 +++++++++++++++++++++++++++++++++++++
>>  3 files changed, 326 insertions(+)
>>  create mode 100644 drivers/pci/host/pcie-altera-msi.c
>>
>> diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
>> index af19039..a8b87fd 100644
>> --- a/drivers/pci/host/Kconfig
>> +++ b/drivers/pci/host/Kconfig
>> @@ -154,4 +154,11 @@ config PCIE_ALTERA
>>         Say Y here if you want to enable PCIe controller support for Altera
>>         SoCFPGA family of SoCs.
>>
>> +config PCIE_ALTERA_MSI
>> +     bool "Altera PCIe MSI feature"
>> +     depends on PCI_MSI && PCIE_ALTERA
>
> What is the dependency with PCIE_ALTERA? Isn't that module standalone?
Theoretically it can work with other PCIe hosts. Will remove depends
PCIE_ALTERA.


>> +struct altera_msi {
>> +     DECLARE_BITMAP(used, MAX_MSI_VECTORS);
>> +     struct mutex            lock;   /* proctect used variable */
>> +     struct platform_device  *pdev;
>> +     struct irq_domain               *msi_domain;
>> +     void __iomem            *csr_base;
>> +     void __iomem            *vector_base;
>> +     u32                     vector_phy;
>
> This should be a phys_addr_t. Not everything is 32bit.
Noted.

>
>> +     u32                     num_of_vectors;
>> +     int                     irq;
>> +};
>> +
>> +static inline void msi_writel(struct altera_msi *msi, u32 value, u32 reg)
>> +{
>> +     writel(value, msi->csr_base + reg);
>
> You should be able to use the relaxed accessors.
Noted.

>
>> +}
>> +
>> +static inline u32 msi_readl(struct altera_msi *msi, u32 reg)
>> +{
>> +     return readl(msi->csr_base + reg);
>
> Same here.
Noted.


>> +             status = msi_readl(msi, MSI_STATUS);
>> +             if (!status)
>> +                     break;
>> +
>> +             do {
>> +                     offset = find_first_bit(&status, num_of_vectors);
>> +                     /* Dummy read from vector to clear the interrupt */
>> +                     readl(msi->vector_base + (offset * sizeof(u32)));
>
> readl_relaxed
Noted.

>
>> +
>> +                     irq = irq_find_mapping(msi->msi_domain->parent, offset);
>
> This would tend to indicate that you don't really need to store the
> msi_domain pointer, but the inner_domain pointer instead.
Will store the inner_domain pointer. But, I think we still need
msi_domain for irq_domain_remove.
Or do we have any way to retrieve msi_domain from inner_domain?

>
>> +                     if (irq) {
>> +                             if (test_bit(offset, msi->used))
>> +                                     generic_handle_irq(irq);
>> +                             else
>> +                                     dev_info(&msi->pdev->dev, "unhandled MSI\n");
>> +                     } else
>> +                             dev_info(&msi->pdev->dev, "unexpected MSI\n");
>> +
>> +                     /* Clear the bit from status and repeat without reading
>> +                      * again status register. */
>> +                     clear_bit(offset, &status);
>> +                     processed++;
>> +             } while (status);
>> +     } while (1);
>> +
>> +     return processed > 0 ? IRQ_HANDLED : IRQ_NONE;
>
> This shouldn't be a simple interrupt interrupt handler, but instead a
> chained irqchip. See pci-xgene-msi.c for an example of such a thing.
Noted, will change to use chained irqchip.

>
>> +}
>> +
>> +static struct irq_chip altera_msi_irq_chip = {
>> +     .name = "Altera PCIe MSI",
>> +     .irq_enable = pci_msi_unmask_irq,
>> +     .irq_disable = pci_msi_mask_irq,
>> +     .irq_mask = pci_msi_mask_irq,
>> +     .irq_unmask = pci_msi_unmask_irq,
>> +};
>> +
>> +static struct msi_domain_info altera_msi_domain_info = {
>> +     .flags  = (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),
>
> So you don't support MSIX? That's a bit weird.
Yes, this MSI IP doesn't support it.


>
>> +     .chip   = &altera_msi_irq_chip,
>> +};
>> +
>> +static void altera_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
>> +{
>> +     struct altera_msi *msi = irq_data_get_irq_chip_data(data);
>> +     u32 mask;
>> +
>> +     msg->address_lo = msi->vector_phy + (data->hwirq * sizeof(u32));
>
> Each MSI has a separate doorbell? Interesting... Please use
> lower_32_bits on the above expression.
Correct. Will change to lower_32_bits.

>
>> +     /* 32 bit address only */
>> +     msg->address_hi = 0;
>
> So this HW will never be used in a 64bit platform? Oddly enough, I
> cannot believe it. Please use upper_32_bits() as the complement of the
> above. At least, we'll be future proof.
It can be used in 64 bits platform. Will change to use upper_32_bits() .

>
>> +     msg->data = data->hwirq;
>> +
>> +     mask = msi_readl(msi, MSI_INTMASK);
>> +     mask |= 1 << data->hwirq;
>> +     msi_writel(msi, mask, MSI_INTMASK);
>> +     dev_dbg(&msi->pdev->dev, "msi#%d address_lo 0x%x\n", (int)data->hwirq,
>> +             msg->address_lo);
>> +}
>> +
>> +static int altera_msi_set_affinity(struct irq_data *irq_data,
>> +                              const struct cpumask *mask, bool force)
>> +{
>> +      return irq_set_affinity(irq_data->hwirq, mask);
>
> There is no way this can be right. irq_data->hwirq can *never* be passed
> as a Linux IRQ. This really should be the IRQ to the GIC.
>
> Which raises another issue: as you only have a single interrupt to the
> GIC, changing the affinity of a single MSI is going to affect all the
> other MSIs as well. This doesn't seem like a desirable behaviour.
Do we must provide '.irq_set_affinity" callback to msi domain? I have
tried set it to NULL,
but kernel got the NULL pointer deference error from msi_domain_set_affinity().
Recently, I saw this new patch for pci-tegra.c from [1], it doesn't
set the ".irq_set_affinity".
Just wonder how it can work.

Do you have any recommendation way for this?

[1] https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms.git/commit/drivers/pci/host?h=irq/gsi-irq-domain-v2&id=17c22fc4e60e6ad54c7e3b73868cbc057360fa63

>> +}
>> +
>> +static struct irq_chip altera_msi_bottom_irq_chip = {
>> +     .name                   = "Altera MSI",
>> +     .irq_compose_msi_msg    = altera_compose_msi_msg,
>> +     .irq_set_affinity       = altera_msi_set_affinity,
>> +};
>> +
>> +static int altera_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
>> +                               unsigned int nr_irqs, void *args)
>> +{
>> +     struct altera_msi *msi = domain->host_data;
>> +     int bit;
>> +
>> +     mutex_lock(&msi->lock);
>> +
>> +     bit = find_first_zero_bit(msi->used, msi->num_of_vectors);
>> +     if (bit < msi->num_of_vectors)
>> +             set_bit(bit, msi->used);
>> +     else
>> +             bit = -ENOSPC;
>
> You can loose the "else" clause...
Okay. Will remove it.

>
>> +
>> +     mutex_unlock(&msi->lock);
>> +
>> +     if (bit < 0)
>> +             return bit;
>
> ... and test for bit >= msi->num_of_vectors, returning -ENOSPC if out of
> vectors.
Noted.


>> +int altera_msi_probe(struct platform_device *pdev)
>> +{
>> +     struct altera_msi *msi;
>> +     struct device_node *np = pdev->dev.of_node;
>> +     struct resource *res;
>> +     int ret;
>> +
>> +     msi = devm_kzalloc(&pdev->dev, sizeof(struct altera_msi),
>> +             GFP_KERNEL);
>> +     if (!msi)
>> +             return -ENOMEM;
>> +
>> +     mutex_init(&msi->lock);
>> +     msi->pdev = pdev;
>> +
>> +     res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "csr");
>> +     msi->csr_base = devm_ioremap_resource(&pdev->dev, res);
>> +     if (IS_ERR(msi->csr_base)) {
>> +             dev_err(&pdev->dev, "get csr resource failed\n");
>> +             return -EADDRNOTAVAIL;
>
> You're being quite creative when it comes to error codes. I'd expect
> this to be used for networking (pci-tegra also uses it, which is even
> more disturbing). I'd be more confident with an -ENOMEM.
Okay, will change it to -ENOMEM.

>
>> +     }
>> +
>> +     res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
>> +                     "vector_slave");
>> +     msi->vector_base = devm_ioremap_resource(&pdev->dev, res);
>> +     if (IS_ERR(msi->vector_base)) {
>> +             dev_err(&pdev->dev, "get vector slave resource failed\n");
>> +             return -EADDRNOTAVAIL;
>> +     }
>> +
>> +     msi->vector_phy = res->start;
>> +
>> +     if (of_property_read_u32(np, "num-vectors", &msi->num_of_vectors)) {
>> +             dev_err(&pdev->dev, "failed to parse the number of vectors\n");
>> +             return -EINVAL;
>> +     }
>
> Since this is a configurable IP, you should have a register telling you
> the number of configured MSI, shouldn't you? Or is the HW really, really
> dumb?
Too bad we can't read it from HW.

>
>> +
>> +     ret = altera_allocate_domains(msi);
>> +     if (ret)
>> +             return ret;
>> +
>> +     msi->irq = platform_get_irq(pdev, 0);
>> +     if (msi->irq <= 0) {
>> +             dev_err(&pdev->dev, "failed to map IRQ: %d\n", msi->irq);
>> +             ret = -ENODEV;
>> +             goto err;
>> +     }
>> +
>> +     ret = devm_request_irq(&pdev->dev, msi->irq, altera_msi_isr, 0,
>> +                             altera_msi_irq_chip.name, msi);
>> +     if (ret) {
>> +             dev_err(&pdev->dev, "failed to request IRQ: %d\n", ret);
>> +             goto err;
>> +     }
>
> Turn this into a call to irq_set_chained_handler.
Noted.


>
>> +
>> +     platform_set_drvdata(pdev, msi);
>> +
>> +     return 0;
>> +
>> +err:
>> +     irq_domain_remove(msi->msi_domain);
>
> You're leaking the inner domain here.
Noted. Will change to altera_msi_remove() instead and eventually it
will remove inner_domain and msi_domain.


Thanks for reviewing.

Regards
Ley Foon

Re: [PATCH 4/6] pci: altera: Add Altera PCIe MSI driver

[Marc Zyngier]

On 29/07/15 09:52, Ley Foon Tan wrote:
> On Wed, Jul 29, 2015 at 1:58 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>> Hi Ley,
>>
>> On 28/07/15 11:45, Ley Foon Tan wrote:
>>> This patch adds Altera PCIe MSI driver. This soft IP supports configurable
>>> number of vectors, which is a dts parameter.
>>
>> Can't you read this configuration from the HW?
> No, we can't read from HW.

Ah, that's a shame. Specially on HW that is configurable by design.

[...]

>>> +
>>> +                     irq = irq_find_mapping(msi->msi_domain->parent, offset);
>>
>> This would tend to indicate that you don't really need to store the
>> msi_domain pointer, but the inner_domain pointer instead.
> Will store the inner_domain pointer. But, I think we still need
> msi_domain for irq_domain_remove.
> Or do we have any way to retrieve msi_domain from inner_domain?

Do you have any case where you remove the domains, aside from the
obvious error cases?

[...]

>>> +
>>> +static struct msi_domain_info altera_msi_domain_info = {
>>> +     .flags  = (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),
>>
>> So you don't support MSIX? That's a bit weird.
> Yes, this MSI IP doesn't support it.

This is not really a function of the MSI IP, but of the PCI device. In
your case, this is just a set of doorbells, so I hardly see what would
prevent MSI-X to be supported. Multi-MSI, I can see why.

[...]

>>> +static int altera_msi_set_affinity(struct irq_data *irq_data,
>>> +                              const struct cpumask *mask, bool force)
>>> +{
>>> +      return irq_set_affinity(irq_data->hwirq, mask);
>>
>> There is no way this can be right. irq_data->hwirq can *never* be passed
>> as a Linux IRQ. This really should be the IRQ to the GIC.
>>
>> Which raises another issue: as you only have a single interrupt to the
>> GIC, changing the affinity of a single MSI is going to affect all the
>> other MSIs as well. This doesn't seem like a desirable behaviour.
> Do we must provide '.irq_set_affinity" callback to msi domain? I have
> tried set it to NULL,
> but kernel got the NULL pointer deference error from msi_domain_set_affinity().
> Recently, I saw this new patch for pci-tegra.c from [1], it doesn't
> set the ".irq_set_affinity".
> Just wonder how it can work.
> 
> Do you have any recommendation way for this?
> 
> [1] https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms.git/commit/drivers/pci/host?h=irq/gsi-irq-domain-v2&id=17c22fc4e60e6ad54c7e3b73868cbc057360fa63

Please realize that I *never* tested this patch (I don't think I ever
posted it officially, I just keep here for convenience), and I wouldn't
take it as a reference.

When it comes to msi_domain_set_affinity issue, this look more like an
oversight. I'll cook a patch for that.

Anyway, whichever way you want to do it, you need to fix this. You could
always return -EINVAL in the meantime,

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 4/6] pci: altera: Add Altera PCIe MSI driver

[Ley Foon Tan]

On Wed, Jul 29, 2015 at 5:15 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>
> On 29/07/15 09:52, Ley Foon Tan wrote:
> > On Wed, Jul 29, 2015 at 1:58 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> >> Hi Ley,
> >>
> >> On 28/07/15 11:45, Ley Foon Tan wrote:
> >>> This patch adds Altera PCIe MSI driver. This soft IP supports configurable
> >>> number of vectors, which is a dts parameter.
> >>
> >> Can't you read this configuration from the HW?
> > No, we can't read from HW.
>
> Ah, that's a shame. Specially on HW that is configurable by design.
>
> [...]
>
> >>> +
> >>> +                     irq = irq_find_mapping(msi->msi_domain->parent, offset);
> >>
> >> This would tend to indicate that you don't really need to store the
> >> msi_domain pointer, but the inner_domain pointer instead.
> > Will store the inner_domain pointer. But, I think we still need
> > msi_domain for irq_domain_remove.
> > Or do we have any way to retrieve msi_domain from inner_domain?
>
> Do you have any case where you remove the domains, aside from the
> obvious error cases?
Yes, if there is error in _probe().

> [...]
>
> >>> +
> >>> +static struct msi_domain_info altera_msi_domain_info = {
> >>> +     .flags  = (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),
> >>
> >> So you don't support MSIX? That's a bit weird.
> > Yes, this MSI IP doesn't support it.
>
> This is not really a function of the MSI IP, but of the PCI device. In
> your case, this is just a set of doorbells, so I hardly see what would
> prevent MSI-X to be supported. Multi-MSI, I can see why.
Yes, you are right. Will add MSI_FLAG_PCI_MSIX flag here.

>
> [...]
>
> >>> +static int altera_msi_set_affinity(struct irq_data *irq_data,
> >>> +                              const struct cpumask *mask, bool force)
> >>> +{
> >>> +      return irq_set_affinity(irq_data->hwirq, mask);
> >>
> >> There is no way this can be right. irq_data->hwirq can *never* be passed
> >> as a Linux IRQ. This really should be the IRQ to the GIC.
> >>
> >> Which raises another issue: as you only have a single interrupt to the
> >> GIC, changing the affinity of a single MSI is going to affect all the
> >> other MSIs as well. This doesn't seem like a desirable behaviour.
> > Do we must provide '.irq_set_affinity" callback to msi domain? I have
> > tried set it to NULL,
> > but kernel got the NULL pointer deference error from msi_domain_set_affinity().
> > Recently, I saw this new patch for pci-tegra.c from [1], it doesn't
> > set the ".irq_set_affinity".
> > Just wonder how it can work.
> >
> > Do you have any recommendation way for this?
> >
> > [1] https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms.git/commit/drivers/pci/host?h=irq/gsi-irq-domain-v2&id=17c22fc4e60e6ad54c7e3b73868cbc057360fa63
>
> Please realize that I *never* tested this patch (I don't think I ever
> posted it officially, I just keep here for convenience), and I wouldn't
> take it as a reference.
>
> When it comes to msi_domain_set_affinity issue, this look more like an
> oversight. I'll cook a patch for that.
>
> Anyway, whichever way you want to do it, you need to fix this. You could
> always return -EINVAL in the meantime,
Will add  -EINVAL for now.

Thanks for reviewing.

Regards
Ley Foon

[PATCH 5/6] Documentation: dt-bindings: pci: altera pcie device tree binding

[Ley Foon Tan]

This patch adds the bindings for Altera PCIe host controller driver and
Altera PCIe MSI driver.

Signed-off-by: Ley Foon Tan <lftan@altera.com>
---
 .../devicetree/bindings/pci/altera-pcie-msi.txt    | 27 ++++++++++++
 .../devicetree/bindings/pci/altera-pcie.txt        | 49 ++++++++++++++++++++++
 2 files changed, 76 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/pci/altera-pcie-msi.txt
 create mode 100644 Documentation/devicetree/bindings/pci/altera-pcie.txt

diff --git a/Documentation/devicetree/bindings/pci/altera-pcie-msi.txt b/Documentation/devicetree/bindings/pci/altera-pcie-msi.txt
new file mode 100644
index 0000000..7f330c9
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/altera-pcie-msi.txt
@@ -0,0 +1,27 @@
+* Altera PCIe MSI controller
+
+Required properties:
+- compatible:	should contain "altr,msi-1.0"
+- reg:		specifies the physical base address of the controller and
+		the length of the memory mapped region.
+- reg-names:	Must include the following entries:
+		"csr": CSR registers
+		"vector_slave": vectors region
+-interrupts:	specifies the interrupt source of the parent interrupt
+		controller. The format of the interrupt specifier depends on the
+		parent interrupt controller.
+- num-vectors:	Number of vectors, range 1 to 32.
+- msi-controller:	indicates that this is MSI controller node
+
+
+Example
+msi0: msi@0xFF200000 {
+	compatible = "altr,msi-1.0";
+	reg = <0xFF200000 0x00000010
+		0xFF200010 0x00000080>;
+	reg-names = "csr", "vector_slave";
+	interrupt-parent = <&hps_0_arm_gic_0>;
+	interrupts = <0 42 4>;
+	msi-controller = <1>;
+	num-vectors = <32>;
+};
diff --git a/Documentation/devicetree/bindings/pci/altera-pcie.txt b/Documentation/devicetree/bindings/pci/altera-pcie.txt
new file mode 100644
index 0000000..54c45f5
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/altera-pcie.txt
@@ -0,0 +1,49 @@
+* Altera PCIe controller
+
+Required properties:
+- compatible :	should contain "altr,pcie-root-port-1.0"
+- reg:		A list of physical base address and length for TXS and CRA.
+- reg-names:	Must include the following entries:
+		"Txs": TXS region
+		"Cra": Control register access region
+-interrupts:	specifies the interrupt source of the parent interrupt controller.
+		The format of the interrupt specifier depends on the parent interrupt
+		controller.
+- device_type:	must be "pci"
+- #address-cells:	set to <3>
+- #size-cells:	set to <2>
+- #interrupt-cells:	set to <1>
+- ranges:		Describes the translation of addresses for root ports and standard
+		PCI regions.
+- interrupt-map-mask and interrupt-map: standard PCI properties
+	to define the mapping of the PCIe interface to interrupt
+	numbers.
+
+Optional properties:
+- msi-parent:	Link to the hardware entity that serves as the MSI controller for this PCIe
+		controller.
+- bus-range:	PCI bus numbers covered
+
+Example
+	pcie_0: pcie@0xc00000000 {
+		compatible = "altr,pcie-root-port-1.0";
+		reg = <0xc0000000 0x20000000>,
+			<0xff220000 0x00004000>;
+		reg-names = "Txs", "Cra";
+		interrupt-parent = <&hps_0_arm_gic_0>;
+		interrupts = <0 40 4>;
+		interrupt-controller;
+		#interrupt-cells = <1>;
+		bus-range = <0x0 0xFF>;
+		device_type = "pci";
+		msi-parent = <&msi_to_gic_gen_0>;
+		#address-cells = <3>;
+		#size-cells = <2>;
+		interrupt-map-mask = <0 0 0 7>;
+		interrupt-map = <0 0 0 1 &pcie_0 1>,
+			            <0 0 0 2 &pcie_0 2>,
+			            <0 0 0 3 &pcie_0 3>,
+			            <0 0 0 4 &pcie_0 4>;
+		ranges = <0x82000000 0x00000000 0xc0000000 0xc0000000 0x00000000 0x10000000
+			    0x82000000 0x00000000 0xd0000000 0xd0000000 0x00000000 0x10000000>;
+	};
-- 
1.8.2.1

[PATCH 6/6] MAINTAINERS: Add Altera PCIe driver maintainer

[Ley Foon Tan]

Signed-off-by: Ley Foon Tan <lftan@altera.com>
---
 MAINTAINERS | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index fd60784..32f5287 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7759,6 +7759,14 @@ F:	include/linux/pci*
 F:	arch/x86/pci/
 F:	arch/x86/kernel/quirks.c
 
+PCI DRIVER FOR ALTERA PCIE IP
+M:	Ley Foon Tan <lftan@altera.com>
+L:	rfi@lists.rocketboards.org (moderated for non-subscribers)
+L:	linux-pci@vger.kernel.org
+S:	Supported
+F:	Documentation/devicetree/bindings/pci/altera-pcie.txt
+F:	drivers/pci/host/pcie-altera.c
+
 PCI DRIVER FOR ARM VERSATILE PLATFORM
 M:	Rob Herring <robh@kernel.org>
 L:	linux-pci@vger.kernel.org
@@ -7860,6 +7868,14 @@ L:	linux-pci@vger.kernel.org
 S:	Maintained
 F:	drivers/pci/host/*spear*
 
+PCI MSI DRIVER FOR ALTERA MSI IP
+M:	Ley Foon Tan <lftan@altera.com>
+L:	rfi@lists.rocketboards.org (moderated for non-subscribers)
+L:	linux-pci@vger.kernel.org
+S:	Supported
+F:	Documentation/devicetree/bindings/pci/altera-pcie-msi.txt
+F:	drivers/pci/host/pcie-altera-msi.c
+
 PCI MSI DRIVER FOR APPLIEDMICRO XGENE
 M:	Duc Dang <dhdang@apm.com>
 L:	linux-pci@vger.kernel.org
-- 
1.8.2.1