* [RFC PATCH 0/3] APM X-Gene PCIe driver
@ 2013-12-23 8:02 Tanmay Inamdar
2013-12-23 8:02 ` [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver Tanmay Inamdar
` (3 more replies)
0 siblings, 4 replies; 27+ messages in thread
From: Tanmay Inamdar @ 2013-12-23 8:02 UTC (permalink / raw)
To: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley
Cc: linux-pci, devicetree, linux-arm-kernel, linux-doc, linux-kernel,
patches, jcm, Tanmay Inamdar
This patch adds support for AppliedMicro X-Gene PCIe host controller. The
driver is tested on X-Gene platform with different gen1/2/3 PCIe endpoint
cards.
X-Gene PCIe controller driver has depedency on the pcie arch support for
arm64. The arm64 pcie arch support is not yet part of mainline Linux kernel
and approach for arch support is under discussion with arm64 maintainers.
The reference patch can be found here --> https://lkml.org/lkml/2013/10/23/244
If someone wishes to test PCIe on X-Gene, arch support patch must be applied
before the patches in this patch set.
Tanmay Inamdar (3):
pci: APM X-Gene PCIe controller driver
arm64: dts: APM X-Gene PCIe device tree nodes
dt-bindings: pci: xgene pcie device tree bindings
.../devicetree/bindings/pci/xgene-pcie.txt | 43 +
arch/arm64/boot/dts/apm-mustang.dts | 4 +
arch/arm64/boot/dts/apm-storm.dtsi | 140 +++
drivers/pci/host/Kconfig | 5 +
drivers/pci/host/Makefile | 1 +
drivers/pci/host/pcie-xgene.c | 1017 ++++++++++++++++++++
6 files changed, 1210 insertions(+)
create mode 100644 Documentation/devicetree/bindings/pci/xgene-pcie.txt
create mode 100644 drivers/pci/host/pcie-xgene.c
--
1.7.9.5
^ permalink raw reply [flat|nested] 27+ messages in thread* [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver 2013-12-23 8:02 [RFC PATCH 0/3] APM X-Gene PCIe driver Tanmay Inamdar @ 2013-12-23 8:02 ` Tanmay Inamdar 2014-01-02 21:08 ` Bjorn Helgaas ` (2 more replies) 2013-12-23 8:02 ` [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes Tanmay Inamdar ` (2 subsequent siblings) 3 siblings, 3 replies; 27+ messages in thread From: Tanmay Inamdar @ 2013-12-23 8:02 UTC (permalink / raw) To: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley Cc: linux-pci, devicetree, linux-arm-kernel, linux-doc, linux-kernel, patches, jcm, Tanmay Inamdar This patch adds the AppliedMicro X-gene SOC PCIe controller driver. APM X-Gene PCIe controller supports maximum upto 8 lanes and GEN3 speed. X-Gene has maximum 5 PCIe ports supported. Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> --- drivers/pci/host/Kconfig | 5 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-xgene.c | 1017 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1023 insertions(+) create mode 100644 drivers/pci/host/pcie-xgene.c diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig index 47d46c6..6d8fcbc 100644 --- a/drivers/pci/host/Kconfig +++ b/drivers/pci/host/Kconfig @@ -33,4 +33,9 @@ config PCI_RCAR_GEN2 There are 3 internal PCI controllers available with a single built-in EHCI/OHCI host controller present on each one. +config PCI_XGENE + bool "X-Gene PCIe controller" + depends on ARCH_XGENE + depends on OF + endmenu diff --git a/drivers/pci/host/Makefile b/drivers/pci/host/Makefile index 13fb333..a0bdfa7 100644 --- a/drivers/pci/host/Makefile +++ b/drivers/pci/host/Makefile @@ -4,3 +4,4 @@ obj-$(CONFIG_PCI_IMX6) += pci-imx6.o obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o obj-$(CONFIG_PCI_TEGRA) += pci-tegra.o obj-$(CONFIG_PCI_RCAR_GEN2) += pci-rcar-gen2.o +obj-$(CONFIG_PCI_XGENE) += pcie-xgene.o diff --git a/drivers/pci/host/pcie-xgene.c b/drivers/pci/host/pcie-xgene.c new file mode 100644 index 0000000..c9403c3 --- /dev/null +++ b/drivers/pci/host/pcie-xgene.c @@ -0,0 +1,1017 @@ +/** + * APM X-Gene PCIe Driver + * + * Copyright (c) 2013 Applied Micro Circuits Corporation. + * + * Author: Tanmay Inamdar <tinamdar@apm.com>. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + */ +#include <linux/module.h> +#include <linux/delay.h> +#include <linux/pci.h> +#include <linux/slab.h> +#include <linux/memblock.h> +#include <linux/io.h> +#include <linux/platform_device.h> +#include <linux/of.h> +#include <linux/of_address.h> +#include <linux/of_irq.h> +#include <linux/of_pci.h> +#include <linux/jiffies.h> +#include <linux/clk-private.h> +#ifdef CONFIG_ARM64 +#include <asm/pcibios.h> +#else +#include <asm/mach/pci.h> +#endif + +#define PCIECORE_LTSSM 0x4c +#define PCIECORE_CTLANDSTATUS 0x50 +#define PIPE_PHY_RATE_RD(src) ((0xc000 & (u32)(src)) >> 0xe) +#define INTXSTATUSMASK 0x6c +#define PIM1_1L 0x80 +#define IBAR2 0x98 +#define IR2MSK 0x9c +#define PIM2_1L 0xa0 +#define OMR1BARL 0x100 +#define OMR2BARL 0x118 +#define CFGBARL 0x154 +#define CFGBARH 0x158 +#define CFGCTL 0x15c +#define RTDID 0x160 +#define CFG_CONSTANTS_31_00 0x2000 +#define CFG_CONSTANTS_63_32 0x2004 +#define CFG_CONSTANTS_159_128 0x2010 +#define CFG_CONSTANTS_415_384 0x2030 +#define ENABLE_L1S_POWER_MGMT_SET(dst, src) (((dst) & ~0x02000000) | \ + (((u32)(src) << 0x19) & \ + 0x02000000)) +#define CFG_CONSTANTS_479_448 0x2038 +#define CFG_8G_CONSTANTS_31_0 0x2100 +#define MGMT_US_PORT_TX_PRESET_SET(dst, src) (((dst) & ~0xf00)| \ + (((u32)(src) << 0x8) & 0xf00)) +#define MGMT_DS_PORT_TX_PRESET_SET(dst, src) (((dst) & ~0xf) | \ + (((u32)(src)) & 0xf)) + +#define CFG_8G_CONSTANTS_159_128 0x2110 +#define EQ_UPDN_POST_STEP_SET(dst, src) (((dst) & ~0x30) | \ + (((u32)(src) << 0x4) & \ + 0x30)) +#define CFG_8G_CONSTANTS_287_256 0x2120 +#define CFG_8G_CONSTANTS_319_288 0x2124 +#define CFG_8G_CONSTANTS_351_320 0x2128 +#define CFG_8G_CONSTANTS_383_352 0x212c +#define EQ_PRE_CURSOR_LANE0_SET(dst, src) (((dst) & ~0xff) | \ + (((u32)(src)) & 0xff)) +#define EQ_PRE_CURSOR_LANE1_SET(dst, src) (((dst) & ~0x00ff0000) | \ + (((u32)(src) << 0x10) & \ + 0x00ff0000)) + +#define CFG_CONTROL_63_32 0x2204 +#define CFG_CONTROL_95_64 0x2208 +#define CFG_CONTROL_191_160 0x2214 +#define PCIE_STATUS_31_0 0x2600 +#define MEM_RAM_SHUTDOWN 0xd070 +#define BLOCK_MEM_RDY 0xd074 + +#define PCI_PRIMARY_BUS_MASK 0x00ffffff +#define REVISION_ID_MASK 0x000000ff +#define SLOT_IMPLEMENTED_MASK 0x04000000 +#define DEVICE_PORT_TYPE_MASK 0x03c00000 +#define ADVT_INFINITE_CREDITS 0x00000200 +#define PM_FORCE_RP_MODE_MASK 0x00000400 +#define SWITCH_PORT_MODE_MASK 0x00000800 +#define CLASS_CODE_MASK 0xffffff00 +#define LINK_UP_MASK 0x00000100 +#define AER_OPTIONAL_ERROR_EN 0xffc00000 +#define DWNSTRM_EQ_SKP_PHS_2_3 0x00010000 +#define DIRECT_TO_5GTS_MASK 0x00020000 +#define SUPPORT_5GTS_MASK 0x00010000 +#define DIRECT_TO_8GTS_MASK 0x00008000 +#define SUPPORT_8GTS_MASK 0x00004000 +#define XGENE_PCIE_DEV_CTRL 0x2f0f +#define AXI_EP_CFG_ACCESS 0x10000 +#define ENABLE_ASPM 0x08000000 +#define XGENE_PORT_TYPE_RC 0x05000000 +#define BLOCK_MEM_RDY_VAL 0xFFFFFFFF +#define EN_COHERENCY 0xF0000000 +#define EN_REG 0x00000001 +#define OB_LO_IO 0x00000002 +#define XGENE_PCIE_VENDORID 0x19AA +#define XGENE_PCIE_BRIDGE_DEVICEID 0xE008 +#define XGENE_PCIE_DEVICEID 0xCAFE +#define XGENE_PCIE_TIMEOUT (500*1000) /* us */ +#define XGENE_PCIE_MAX_REGIONS 3 +#define XGENE_LTSSM_DETECT_WAIT 20 +#define XGENE_LTSSM_L0_WAIT 4 +#define XGENE_PCIE_PHY_DRV "pcie-8g" +#define XGENE_PCIE_CLK_DRV "pcieclk" +#define XGENE_PCIE_MAX_PORTS 5 +#define XGENE_PCIE_EP_MEM_SIZE 0x100000 + +enum { + PTYPE_ENDPOINT = 0x0, + PTYPE_LEGACY_ENDPOINT = 0x1, + PTYPE_ROOT_PORT = 0x4, + + LNKW_X1 = 0x1, + LNKW_X4 = 0x4, + LNKW_X8 = 0x8, + + PCIE_GEN1 = 0x0, /* 2.5G */ + PCIE_GEN2 = 0x1, /* 5.0G */ + PCIE_GEN3 = 0x2, /* 8.0G */ +}; + +enum { + XGENE_MEM, + XGENE_MSI, + XGENE_IO, + XGENE_RES /* termination */ +}; + +struct xgene_pcie_ep_info { + void *reg_virt; /* maps to outbound space of RC */ + dma_addr_t reg_phys; /* Physical address of reg space */ +}; + +struct xgene_pcie_port { + struct device_node *node; + struct resource res[XGENE_RES]; + u8 type; + u8 link_up; + u8 link_speed; + u32 first_busno; + void *csr_base; + void *cfg_base; + struct device *dev; + struct xgene_pcie_ep_info ep_info; + struct clk *clk; +}; + +#ifdef CONFIG_64BIT +#define pci_io_offset(s) (s & 0xff00000000) +#else +#define pci_io_offset(s) (s & 0x00000000) +#endif /* CONFIG_64BIT */ + +static inline struct xgene_pcie_port * +xgene_pcie_sys_to_port(struct pci_sys_data *sys) +{ + return (struct xgene_pcie_port *)sys->private_data; +} + +static inline struct xgene_pcie_port * +xgene_pcie_bus_to_port(struct pci_bus *bus) +{ + struct pci_sys_data *sys = bus->sysdata; + return xgene_pcie_sys_to_port(sys); +} + +/* IO ports are memory mapped */ +void __iomem *__pci_ioport_map(struct pci_dev *dev, unsigned long port, + unsigned int nr) +{ + return devm_ioremap_nocache(&dev->dev, port, nr); +} + +/* PCIE Out/In to CSR */ +static inline void xgene_pcie_out32(void *addr, u32 val) +{ + pr_debug("pcie csr wr: 0x%llx 0x%08x\n", (phys_addr_t)addr, val); + writel(val, addr); +} + +static inline void xgene_pcie_in32(void *addr, u32 *val) +{ + *val = readl(addr); + pr_debug("pcie csr rd: 0x%llx 0x%08x\n", (phys_addr_t)addr, *val); +} + +/* PCIE Configuration Out/In */ +static inline void xgene_pcie_cfg_out32(void *addr, u32 val) +{ + writel(val, addr); +} + +static inline void xgene_pcie_cfg_out16(void *addr, u16 val) +{ + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; + u32 val32 = readl((void *)temp_addr); + + switch ((phys_addr_t) addr & 0x3) { + case 2: + val32 &= ~0xFFFF0000; + val32 |= (u32) val << 16; + break; + case 0: + default: + val32 &= ~0xFFFF; + val32 |= val; + break; + } + writel(val32, (void *)temp_addr); +} + +static inline void xgene_pcie_cfg_out8(void *addr, u8 val) +{ + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; + u32 val32 = readl((void *)temp_addr); + + switch ((phys_addr_t) addr & 0x3) { + case 0: + val32 &= ~0xFF; + val32 |= val; + break; + case 1: + val32 &= ~0xFF00; + val32 |= (u32) val << 8; + break; + case 2: + val32 &= ~0xFF0000; + val32 |= (u32) val << 16; + break; + case 3: + default: + val32 &= ~0xFF000000; + val32 |= (u32) val << 24; + break; + } + writel(val32, (void *)temp_addr); +} + +static inline void xgene_pcie_cfg_in32(void *addr, u32 *val) +{ + *val = readl(addr); +} + +static inline void xgene_pcie_cfg_in16(void *addr, u16 *val) +{ + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; + u32 val32; + + val32 = readl((void *)temp_addr); + + switch ((phys_addr_t) addr & 0x3) { + case 2: + *val = val32 >> 16; + break; + case 0: + default: + *val = val32; + break; + } +} + +static inline void xgene_pcie_cfg_in8(void *addr, u8 *val) +{ + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; + u32 val32; + + val32 = readl((void *)temp_addr); + + switch ((phys_addr_t) addr & 0x3) { + case 3: + *val = val32 >> 24; + break; + case 2: + *val = val32 >> 16; + break; + case 1: + *val = val32 >> 8; + break; + case 0: + default: + *val = val32; + break; + } +} + +static void __iomem *xgene_pcie_get_cfg_base(struct pci_bus *bus) +{ + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); + phys_addr_t addr = (phys_addr_t) port->cfg_base; + + if (bus->number >= (port->first_busno + 1)) + addr |= AXI_EP_CFG_ACCESS; + + return (void *)addr; +} + +static void xgene_pcie_set_rtdid_reg(struct pci_bus *bus, uint devfn) +{ + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); + unsigned int b, d, f; + u32 rtdid_val = 0; + + b = bus->number; + d = PCI_SLOT(devfn); + f = PCI_FUNC(devfn); + + if (bus->number == port->first_busno) + rtdid_val = (b << 24) | (d << 19) | (f << 16); + else if (bus->number >= (port->first_busno + 1)) + rtdid_val = (port->first_busno << 24) | + (b << 8) | (d << 3) | f; + + xgene_pcie_out32(port->csr_base + RTDID, rtdid_val); + /* read the register back to ensure flush */ + xgene_pcie_in32(port->csr_base + RTDID, &rtdid_val); +} + +static int xgene_pcie_read_config(struct pci_bus *bus, unsigned int devfn, + int offset, int len, u32 *val) +{ + void __iomem *addr; + u8 val8; + u16 val16; + + if (pci_is_root_bus(bus) && devfn != 0) + return PCIBIOS_DEVICE_NOT_FOUND; + + xgene_pcie_set_rtdid_reg(bus, devfn); + addr = xgene_pcie_get_cfg_base(bus); + switch (len) { + case 1: + xgene_pcie_cfg_in8(addr + offset, &val8); + *val = val8; + break; + case 2: + xgene_pcie_cfg_in16(addr + offset, &val16); + *val = val16; + break; + default: + xgene_pcie_cfg_in32(addr + offset, val); + break; + } + return PCIBIOS_SUCCESSFUL; +} + +static int xgene_pcie_write_config(struct pci_bus *bus, unsigned int devfn, + int offset, int len, u32 val) +{ + void __iomem *addr; + + if (pci_is_root_bus(bus) && devfn != 0) + return PCIBIOS_DEVICE_NOT_FOUND; + + xgene_pcie_set_rtdid_reg(bus, devfn); + addr = xgene_pcie_get_cfg_base(bus); + switch (len) { + case 1: + xgene_pcie_cfg_out8(addr + offset, (u8) val); + break; + case 2: + xgene_pcie_cfg_out16(addr + offset, (u16) val); + break; + default: + xgene_pcie_cfg_out32(addr + offset, val); + break; + } + return PCIBIOS_SUCCESSFUL; +} + +static struct pci_ops xgene_pcie_ops = { + .read = xgene_pcie_read_config, + .write = xgene_pcie_write_config +}; + +static void xgene_pcie_setup_lanes(struct xgene_pcie_port *port) +{ + void *csr_base = port->csr_base; + u32 val; + + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_287_256, &val); + val = EQ_PRE_CURSOR_LANE0_SET(val, 0x7); + val = EQ_PRE_CURSOR_LANE1_SET(val, 0x7); + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_287_256, val); + + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_319_288, &val); + val = EQ_PRE_CURSOR_LANE0_SET(val, 0x7); + val = EQ_PRE_CURSOR_LANE1_SET(val, 0x7); + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_319_288, val); + + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_351_320, &val); + val = EQ_PRE_CURSOR_LANE0_SET(val, 0x7); + val = EQ_PRE_CURSOR_LANE1_SET(val, 0x7); + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_351_320, val); + + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_383_352, &val); + val = EQ_PRE_CURSOR_LANE0_SET(val, 0x7); + val = EQ_PRE_CURSOR_LANE1_SET(val, 0x7); + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_383_352, val); + + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_159_128, &val); + val = EQ_UPDN_POST_STEP_SET(val, 0x1); + val = EQ_UPDN_POST_STEP_SET(val, 0x1); + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_159_128, val); +} + +static void xgene_pcie_setup_link(struct xgene_pcie_port *port) +{ + void *csr_base = port->csr_base; + u32 val; + + xgene_pcie_in32(csr_base + CFG_CONSTANTS_479_448, &val); + switch (port->link_speed) { + case PCIE_GEN1: + val &= ~SUPPORT_5GTS_MASK; + val &= ~SUPPORT_8GTS_MASK; + break; + case PCIE_GEN2: + val |= SUPPORT_5GTS_MASK; + val |= DIRECT_TO_5GTS_MASK; + val &= ~SUPPORT_8GTS_MASK; + val &= ~DIRECT_TO_8GTS_MASK; + break; + case PCIE_GEN3: + val |= DIRECT_TO_8GTS_MASK; + val |= SUPPORT_5GTS_MASK; + val |= SUPPORT_8GTS_MASK; + val |= DIRECT_TO_5GTS_MASK; + break; + } + xgene_pcie_out32(csr_base + CFG_CONSTANTS_479_448, val); + + xgene_pcie_in32(csr_base + CFG_CONSTANTS_479_448, &val); + val &= ~ADVT_INFINITE_CREDITS; + xgene_pcie_out32(csr_base + CFG_CONSTANTS_479_448, val); + + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_31_0, &val); + val |= MGMT_DS_PORT_TX_PRESET_SET(val, 0x7); + val |= MGMT_US_PORT_TX_PRESET_SET(val, 0x7); + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_31_0, val); + + if (port->link_speed == PCIE_GEN3) { + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_31_0, &val); + val |= DWNSTRM_EQ_SKP_PHS_2_3; + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_31_0, val); + } +} + +static void xgene_pcie_program_core(void *csr_base) +{ + u32 val; + + xgene_pcie_in32(csr_base + CFG_CONSTANTS_31_00, &val); + val |= AER_OPTIONAL_ERROR_EN; + xgene_pcie_out32(csr_base + CFG_CONSTANTS_31_00, val); + xgene_pcie_out32(csr_base + INTXSTATUSMASK, 0x0); + xgene_pcie_in32(csr_base + CFG_CONTROL_63_32, &val); + val = (val & ~0xffff) | XGENE_PCIE_DEV_CTRL; + xgene_pcie_out32(csr_base + CFG_CONTROL_63_32, val); +} + +static u64 xgene_pcie_set_ib_mask(void *csr_base, u32 addr, u32 flags, + resource_size_t size) +{ + u64 val64 = 0; + u32 val32 = 0; + u32 val; + + if (size >= SZ_1K) + val64 = (~(size - 1) & PCI_BASE_ADDRESS_MEM_MASK) | flags; + + xgene_pcie_in32(csr_base + addr, &val32); + val = (val32 & 0x0000ffff) | (lower_32_bits(val64) << 16); + xgene_pcie_out32(csr_base + addr, val); + + xgene_pcie_in32(csr_base + addr + 0x04, &val32); + val = (val32 & 0xffff0000) | (lower_32_bits(val64) >> 16); + xgene_pcie_out32(csr_base + addr + 0x04, val); + + xgene_pcie_in32(csr_base + addr + 0x04, &val32); + val = (val32 & 0x0000ffff) | (upper_32_bits(val64) << 16); + xgene_pcie_out32(csr_base + addr + 0x04, val); + + xgene_pcie_in32(csr_base + addr + 0x08, &val32); + val = (val32 & 0xffff0000) | (upper_32_bits(val64) >> 16); + xgene_pcie_out32(csr_base + addr + 0x08, val); + + return val64; +} + +static void xgene_pcie_config_pims(void *csr_base, u32 addr, + u64 pim, resource_size_t size) +{ + u32 val; + + xgene_pcie_out32(csr_base + addr, lower_32_bits(pim)); + val = upper_32_bits(pim) | EN_COHERENCY; + xgene_pcie_out32(csr_base + addr + 0x04, val); + xgene_pcie_out32(csr_base + addr + 0x08, 0x0); + xgene_pcie_out32(csr_base + addr + 0x0c, 0x0); + val = lower_32_bits(size); + xgene_pcie_out32(csr_base + addr + 0x10, val); + val = upper_32_bits(size); + xgene_pcie_out32(csr_base + addr + 0x14, val); +} + +static void xgene_pcie_poll_linkup(struct xgene_pcie_port *port, u32 *lanes) +{ + void *csr_base = port->csr_base; + u32 val32; + u64 start_time, time; + + /* + * A component enters the LTSSM Detect state within + * 20ms of the end of fundamental core reset. + */ + msleep(XGENE_LTSSM_DETECT_WAIT); + port->link_up = 0; + start_time = jiffies; + do { + xgene_pcie_in32(csr_base + PCIECORE_CTLANDSTATUS, &val32); + if (val32 & LINK_UP_MASK) { + port->link_up = 1; + port->link_speed = PIPE_PHY_RATE_RD(val32); + xgene_pcie_in32(csr_base + PCIE_STATUS_31_0, &val32); + *lanes = val32 >> 26; + } + time = jiffies_to_msecs(jiffies - start_time); + } while ((!port->link_up) || (time <= XGENE_LTSSM_L0_WAIT)); +} + +static void xgene_pcie_setup_root_complex(struct xgene_pcie_port *port) +{ + void *csr_base = port->csr_base; + u32 val; + + val = (XGENE_PCIE_BRIDGE_DEVICEID << 16) | XGENE_PCIE_VENDORID; + xgene_pcie_out32(csr_base + CFG_CONSTANTS_31_00, val); + + xgene_pcie_in32(csr_base + CFG_CONSTANTS_63_32, &val); + val &= ~CLASS_CODE_MASK; + val |= PCI_CLASS_BRIDGE_PCI << 16; + xgene_pcie_out32(csr_base + CFG_CONSTANTS_63_32, val); + + xgene_pcie_in32(csr_base + CFG_CONSTANTS_479_448, &val); + val |= SWITCH_PORT_MODE_MASK; + val &= ~PM_FORCE_RP_MODE_MASK; + xgene_pcie_out32(csr_base + CFG_CONSTANTS_479_448, val); + xgene_pcie_setup_link(port); + xgene_pcie_setup_lanes(port); + xgene_pcie_in32(csr_base + CFG_CONTROL_191_160, &val); + val &= ~DEVICE_PORT_TYPE_MASK; + val |= XGENE_PORT_TYPE_RC; + xgene_pcie_out32(csr_base + CFG_CONTROL_191_160, val); + + xgene_pcie_in32(csr_base + CFG_CONTROL_95_64, &val); + val |= ENABLE_ASPM; + xgene_pcie_out32(csr_base + CFG_CONTROL_95_64, val); + + xgene_pcie_in32(csr_base + CFG_CONSTANTS_415_384, &val); + val = ENABLE_L1S_POWER_MGMT_SET(val, 1); + xgene_pcie_out32(csr_base + CFG_CONSTANTS_415_384, val); +} + +static void xgene_pcie_setup_endpoint(struct xgene_pcie_port *port) +{ + void *csr_base = port->csr_base; + u32 val; + + xgene_pcie_in32(csr_base + CFG_CONSTANTS_479_448, &val); + val &= ~SWITCH_PORT_MODE_MASK; + val &= ~PM_FORCE_RP_MODE_MASK; + xgene_pcie_out32(csr_base + CFG_CONSTANTS_479_448, val); + + xgene_pcie_in32(csr_base + CFG_CONTROL_191_160, &val); + val &= ~DEVICE_PORT_TYPE_MASK; + val &= ~SLOT_IMPLEMENTED_MASK; + xgene_pcie_out32(csr_base + CFG_CONTROL_191_160, val); + + xgene_pcie_in32(csr_base + CFG_CONSTANTS_31_00, &val); + val = (XGENE_PCIE_DEVICEID << 16) | XGENE_PCIE_VENDORID; + xgene_pcie_out32(csr_base + CFG_CONSTANTS_31_00, val); + + xgene_pcie_in32(csr_base + CFG_CONSTANTS_63_32, &val); + val &= REVISION_ID_MASK; + val |= PCI_CLASS_BRIDGE_OTHER << 16; + xgene_pcie_out32(csr_base + CFG_CONSTANTS_63_32, val); + + xgene_pcie_setup_link(port); +} + +static void xgene_pcie_setup_port(struct xgene_pcie_port *port) +{ + int type = port->type; + + xgene_pcie_program_core(port->csr_base); + if (type == PTYPE_ROOT_PORT) + xgene_pcie_setup_root_complex(port); + else + xgene_pcie_setup_endpoint(port); +} + +/* Return 0 on success */ +static int xgene_pcie_init_ecc(struct xgene_pcie_port *port) +{ + void *csr_base = port->csr_base; + int timeout = XGENE_PCIE_TIMEOUT; + u32 val; + + xgene_pcie_in32(csr_base + MEM_RAM_SHUTDOWN, &val); + if (val == 0) + return 0; + xgene_pcie_out32(csr_base + MEM_RAM_SHUTDOWN, 0x0); + do { + xgene_pcie_in32(csr_base + BLOCK_MEM_RDY, &val); + udelay(1); + } while ((val != BLOCK_MEM_RDY_VAL) && timeout--); + + return !(timeout > 0); +} + +static int xgene_pcie_init_port(struct xgene_pcie_port *port) +{ + int rc; + + port->clk = clk_get(port->dev, XGENE_PCIE_CLK_DRV); + if (IS_ERR_OR_NULL(port->clk)) { + dev_err(port->dev, "clock not available\n"); + return -ENODEV; + } + + rc = clk_prepare_enable(port->clk); + if (rc) { + dev_err(port->dev, "clock enable failed\n"); + return rc; + } + + rc = xgene_pcie_init_ecc(port); + if (rc) { + dev_err(port->dev, "memory init failed\n"); + return rc; + } + + return 0; +} + +struct device_node *pcibios_get_phb_of_node(struct pci_bus *bus) +{ + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); + + return of_node_get(port->node); +} + +static void xgene_pcie_fixup_bridge(struct pci_dev *dev) +{ + int i; + + for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) { + dev->resource[i].start = dev->resource[i].end = 0; + dev->resource[i].flags = 0; + } +} +DECLARE_PCI_FIXUP_HEADER(XGENE_PCIE_VENDORID, XGENE_PCIE_BRIDGE_DEVICEID, + xgene_pcie_fixup_bridge); + +static void xgene_pcie_setup_primary_bus(struct xgene_pcie_port *port, + u32 first_busno, u32 last_busno) +{ + u32 val; + void *cfg_addr = port->cfg_base; + + xgene_pcie_in32(cfg_addr + PCI_PRIMARY_BUS, &val); + val &= ~PCI_PRIMARY_BUS_MASK; + val |= (last_busno << 16) | ((first_busno + 1) << 8) | (first_busno); + xgene_pcie_out32(cfg_addr + PCI_PRIMARY_BUS, val); +} + +/* + * read configuration values from DTS + */ +static int xgene_pcie_read_dts_config(struct xgene_pcie_port *port) +{ + struct device_node *np = port->node; + struct resource csr_res; + u32 val32; + int ret; + const u8 *val; + + val = of_get_property(np, "device_type", NULL); + if ((val != NULL) && !strcmp(val, "ep")) + port->type = PTYPE_ENDPOINT; + else + port->type = PTYPE_ROOT_PORT; + + ret = of_property_read_u32(np, "link_speed", &val32); + if (ret == 0) + port->link_speed = val32; + else + port->link_speed = PCIE_GEN3; + + /* Get configured CSR space registers address */ + if (of_address_to_resource(np, 0, &csr_res)) + return -EINVAL; + + port->csr_base = devm_ioremap_nocache(port->dev, csr_res.start, + resource_size(&csr_res)); + if (port->csr_base == NULL) + return -ENOMEM; + + return 0; +} + +static int xgene_pcie_alloc_ep_mem(struct xgene_pcie_port *port) +{ + struct xgene_pcie_ep_info *ep = &port->ep_info; + + ep->reg_virt = dma_alloc_coherent(port->dev, XGENE_PCIE_EP_MEM_SIZE, + &ep->reg_phys, GFP_KERNEL); + if (ep->reg_virt == NULL) + return -ENOMEM; + + dev_info(port->dev, "EP: Virt - %p Phys - 0x%llx Size - 0x%x\n", + ep->reg_virt, (u64) ep->reg_phys, XGENE_PCIE_EP_MEM_SIZE); + return 0; +} + +static int xgene_pcie_populate_inbound_regions(struct xgene_pcie_port *port) +{ + struct resource *msi_res = &port->res[XGENE_MSI]; + phys_addr_t ddr_size = memblock_phys_mem_size(); + phys_addr_t ddr_base = memblock_start_of_DRAM(); + void *csr_base = port->csr_base; + void *cfg_addr = port->cfg_base; + u64 val64, size; + u32 val, mask_addr; + u32 flags = PCI_BASE_ADDRESS_MEM_PREFETCH | + PCI_BASE_ADDRESS_MEM_TYPE_64; + + if (port->type == PTYPE_ROOT_PORT) { + mask_addr = CFG_CONSTANTS_159_128; + xgene_pcie_set_ib_mask(csr_base, mask_addr, flags, ddr_size); + val = (lower_32_bits(ddr_base) & PCI_BASE_ADDRESS_MEM_MASK) | + flags; + xgene_pcie_out32(cfg_addr + PCI_BASE_ADDRESS_0, val); + val = upper_32_bits(ddr_base); + xgene_pcie_out32(cfg_addr + PCI_BASE_ADDRESS_1, val); + xgene_pcie_config_pims(csr_base, PIM1_1L, ddr_base, ddr_size); + } else { + struct xgene_pcie_ep_info *ep = &port->ep_info; + if (xgene_pcie_alloc_ep_mem(port)) + return -ENOMEM; + mask_addr = CFG_CONSTANTS_159_128; + size = XGENE_PCIE_EP_MEM_SIZE; + xgene_pcie_set_ib_mask(csr_base, mask_addr, flags, size); + xgene_pcie_config_pims(csr_base, PIM1_1L, ep->reg_phys, size); + } + val64 = 0; + size = resource_size(msi_res); + if (size >= SZ_1M) + val64 = ~(size - 1) | EN_REG; + xgene_pcie_out32(csr_base + IBAR2, msi_res->start); + xgene_pcie_out32(csr_base + IR2MSK, lower_32_bits(val64)); + xgene_pcie_config_pims(csr_base, PIM2_1L, msi_res->start, size); + return 0; +} + +static void xgene_pcie_setup_ob_reg(void *csr_base, u32 addr, u32 index, + struct resource *res) +{ + resource_size_t size = resource_size(res); + u64 val64 = 0; + u32 min_size = 0; + u32 flag = EN_REG; + + switch (index) { + case XGENE_MEM: + min_size = SZ_128M; + break; + case XGENE_IO: + min_size = 128; + flag |= OB_LO_IO; + break; + } + if (size >= min_size) + val64 = ~(size - 1) | flag; + else + pr_warn("resource size 0x%llx less than minimum 0x%x\n", + (u64)size, min_size); + xgene_pcie_out32(csr_base + addr, lower_32_bits(res->start)); + xgene_pcie_out32(csr_base + addr + 0x04, upper_32_bits(res->start)); + xgene_pcie_out32(csr_base + addr + 0x08, lower_32_bits(val64)); + xgene_pcie_out32(csr_base + addr + 0x0c, upper_32_bits(val64)); + xgene_pcie_out32(csr_base + addr + 0x10, 0x0); + xgene_pcie_out32(csr_base + addr + 0x14, 0x0); +} + +static int xgene_pcie_map_cfg(struct xgene_pcie_port *port, + struct of_pci_range *range) +{ + struct device *dev = port->dev; + u64 addr = range->cpu_addr; + resource_size_t size = range->size; + void *csr_base = port->csr_base; + + port->cfg_base = devm_ioremap_nocache(dev, addr, size); + if (port->cfg_base == NULL) { + dev_err(dev, "failed to map cfg region!"); + return -ENOMEM; + } + + xgene_pcie_out32(csr_base + CFGBARL, lower_32_bits(addr)); + xgene_pcie_out32(csr_base + CFGBARH, upper_32_bits(addr)); + xgene_pcie_out32(csr_base + CFGCTL, EN_REG); + + return 0; +} + +static int xgene_pcie_parse_map_ranges(struct xgene_pcie_port *port) +{ + struct device_node *np = port->node; + struct of_pci_range range; + struct of_pci_range_parser parser; + struct device *dev = port->dev; + u32 cfg_map_done = 0; + int ret; + + if (of_pci_range_parser_init(&parser, np)) { + dev_err(dev, "missing ranges property\n"); + return -EINVAL; + } + + /* Get the I/O, memory, config ranges from DT */ + for_each_of_pci_range(&parser, &range) { + struct resource *res = NULL; + u64 restype = range.flags & IORESOURCE_TYPE_BITS; + u64 end = range.cpu_addr + range.size - 1; + dev_dbg(port->dev, "0x%08x 0x%016llx..0x%016llx -> 0x%016llx\n", + range.flags, range.cpu_addr, end, range.pci_addr); + + switch (restype) { + case IORESOURCE_IO: + res = &port->res[XGENE_IO]; + of_pci_range_to_resource(&range, np, res); + xgene_pcie_setup_ob_reg(port->csr_base, OMR1BARL, + XGENE_IO, res); + break; + case IORESOURCE_MEM: + res = &port->res[XGENE_MEM]; + of_pci_range_to_resource(&range, np, res); + xgene_pcie_setup_ob_reg(port->csr_base, OMR2BARL, + XGENE_MEM, res); + break; + case 0: + if (!cfg_map_done) { + /* config region */ + if (port->type == PTYPE_ROOT_PORT) { + ret = xgene_pcie_map_cfg(port, &range); + if (ret) + return ret; + } + cfg_map_done = 1; + } else { + /* msi region */ + res = &port->res[XGENE_MSI]; + of_pci_range_to_resource(&range, np, res); + } + break; + default: + dev_err(dev, "invalid io resource!"); + return -EINVAL; + } + } + + return xgene_pcie_populate_inbound_regions(port); +} + +static int xgene_pcie_setup(int nr, struct pci_sys_data *sys) +{ + struct xgene_pcie_port *pp = xgene_pcie_sys_to_port(sys); + + if (pp == NULL) + return 0; + + if (pp->type == PTYPE_ENDPOINT) + return 0; + + sys->io_offset = pci_io_offset(pp->res[XGENE_IO].start); + sys->mem_offset = pci_io_offset(pp->res[XGENE_MEM].start); + + BUG_ON(request_resource(&iomem_resource, &pp->res[XGENE_IO]) || + request_resource(&iomem_resource, &pp->res[XGENE_MEM])); + + pci_add_resource_offset(&sys->resources, &pp->res[XGENE_MEM], + sys->mem_offset); + pci_add_resource_offset(&sys->resources, &pp->res[XGENE_IO], + sys->io_offset); + return 1; +} + +static int xgene_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) +{ + return of_irq_parse_and_map_pci(dev, slot, pin); +} + +static struct pci_bus __init *xgene_pcie_scan_bus(int nr, + struct pci_sys_data *sys) +{ + struct xgene_pcie_port *pp = xgene_pcie_sys_to_port(sys); + + pp->first_busno = sys->busnr; + xgene_pcie_setup_primary_bus(pp, sys->busnr, 0xff); + return pci_scan_root_bus(NULL, sys->busnr, &xgene_pcie_ops, + sys, &sys->resources); +} + +static struct hw_pci xgene_pcie_hw __initdata = { + .nr_controllers = XGENE_PCIE_MAX_PORTS, + .setup = xgene_pcie_setup, + .scan = xgene_pcie_scan_bus, + .map_irq = xgene_pcie_map_irq, +}; + +static int __init xgene_pcie_probe_bridge(struct platform_device *pdev) +{ + struct device_node *np = of_node_get(pdev->dev.of_node); + struct xgene_pcie_port *port; + u32 lanes = 0; + static int index; + int ret; + + port = devm_kzalloc(&pdev->dev, sizeof(*port), GFP_KERNEL); + if (port == NULL) + return -ENOMEM; + port->node = np; + port->dev = &pdev->dev; + + ret = xgene_pcie_read_dts_config(port); + if (ret) + return ret; + + ret = xgene_pcie_init_port(port); + if (ret) + goto skip; + + xgene_pcie_setup_port(port); + ret = xgene_pcie_parse_map_ranges(port); + if (ret) + goto skip; + + if (port->type == PTYPE_ROOT_PORT) + xgene_pcie_poll_linkup(port, &lanes); +skip: + if (port->type == PTYPE_ROOT_PORT) { + if (!port->link_up) + dev_info(port->dev, "(rc) link down\n"); + else + dev_info(port->dev, "(rc) x%d gen-%d link up\n", + lanes, port->link_speed + 1); + } else + dev_info(port->dev, "(ep)\n"); + + xgene_pcie_hw.private_data[index++] = port; + platform_set_drvdata(pdev, port); + return 0; +} + +static const struct of_device_id xgene_pcie_match_table[] __initconst = { + {.compatible = "apm,xgene-pcie",}, + {}, +}; + +static struct platform_driver xgene_pcie_driver = { + .driver = { + .name = "xgene-pcie", + .owner = THIS_MODULE, + .of_match_table = of_match_ptr(xgene_pcie_match_table), + }, +}; + +static int __init xgene_pcie_init(void) +{ + void *private; + int ret; + + pr_info("X-Gene: PCIe driver\n"); + + /* allocate private data to keep xgene_pcie_port information */ + private = kzalloc((XGENE_PCIE_MAX_PORTS * sizeof(void *)), GFP_KERNEL); + if (private == NULL) + return -ENOMEM; + xgene_pcie_hw.private_data = private; + ret = platform_driver_probe(&xgene_pcie_driver, + xgene_pcie_probe_bridge); + if (ret) + return ret; + pci_common_init(&xgene_pcie_hw); + return 0; +} + +module_init(xgene_pcie_init); + +MODULE_AUTHOR("Tanmay Inamdar <tinamdar@apm.com>"); +MODULE_DESCRIPTION("APM X-Gene PCIe driver"); +MODULE_LICENSE("GPL v2"); -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver 2013-12-23 8:02 ` [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver Tanmay Inamdar @ 2014-01-02 21:08 ` Bjorn Helgaas 2014-01-03 12:07 ` Arnd Bergmann 2014-01-06 1:47 ` Jingoo Han 2 siblings, 0 replies; 27+ messages in thread From: Bjorn Helgaas @ 2014-01-02 21:08 UTC (permalink / raw) To: Tanmay Inamdar Cc: Grant Likely, Catalin Marinas, Rob Landley, linux-pci@vger.kernel.org, devicetree@vger.kernel.org, linux-arm, linux-doc, linux-kernel@vger.kernel.org, patches, jcm On Mon, Dec 23, 2013 at 1:02 AM, Tanmay Inamdar <tinamdar@apm.com> wrote: > This patch adds the AppliedMicro X-gene SOC PCIe controller driver. > APM X-Gene PCIe controller supports maximum upto 8 lanes and > GEN3 speed. X-Gene has maximum 5 PCIe ports supported. > > Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> Since Jason requested "lspci" output, I'm waiting for him to review these before applying them. Also, please include a MAINTAINERS update showing who should approve future changes to this file. Bjorn > --- > drivers/pci/host/Kconfig | 5 + > drivers/pci/host/Makefile | 1 + > drivers/pci/host/pcie-xgene.c | 1017 +++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 1023 insertions(+) > create mode 100644 drivers/pci/host/pcie-xgene.c > > diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig > index 47d46c6..6d8fcbc 100644 > --- a/drivers/pci/host/Kconfig > +++ b/drivers/pci/host/Kconfig > @@ -33,4 +33,9 @@ config PCI_RCAR_GEN2 > There are 3 internal PCI controllers available with a single > built-in EHCI/OHCI host controller present on each one. > > +config PCI_XGENE > + bool "X-Gene PCIe controller" > + depends on ARCH_XGENE > + depends on OF > + > endmenu > diff --git a/drivers/pci/host/Makefile b/drivers/pci/host/Makefile > index 13fb333..a0bdfa7 100644 > --- a/drivers/pci/host/Makefile > +++ b/drivers/pci/host/Makefile > @@ -4,3 +4,4 @@ obj-$(CONFIG_PCI_IMX6) += pci-imx6.o > obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o > obj-$(CONFIG_PCI_TEGRA) += pci-tegra.o > obj-$(CONFIG_PCI_RCAR_GEN2) += pci-rcar-gen2.o > +obj-$(CONFIG_PCI_XGENE) += pcie-xgene.o > diff --git a/drivers/pci/host/pcie-xgene.c b/drivers/pci/host/pcie-xgene.c > new file mode 100644 > index 0000000..c9403c3 > --- /dev/null > +++ b/drivers/pci/host/pcie-xgene.c > @@ -0,0 +1,1017 @@ > +/** > + * APM X-Gene PCIe Driver > + * > + * Copyright (c) 2013 Applied Micro Circuits Corporation. > + * > + * Author: Tanmay Inamdar <tinamdar@apm.com>. > + * > + * This program is free software; you can redistribute it and/or modify it > + * under the terms of the GNU General Public License as published by the > + * Free Software Foundation; either version 2 of the License, or (at your > + * option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + */ > +#include <linux/module.h> > +#include <linux/delay.h> > +#include <linux/pci.h> > +#include <linux/slab.h> > +#include <linux/memblock.h> > +#include <linux/io.h> > +#include <linux/platform_device.h> > +#include <linux/of.h> > +#include <linux/of_address.h> > +#include <linux/of_irq.h> > +#include <linux/of_pci.h> > +#include <linux/jiffies.h> > +#include <linux/clk-private.h> > +#ifdef CONFIG_ARM64 > +#include <asm/pcibios.h> > +#else > +#include <asm/mach/pci.h> > +#endif > + > +#define PCIECORE_LTSSM 0x4c > +#define PCIECORE_CTLANDSTATUS 0x50 > +#define PIPE_PHY_RATE_RD(src) ((0xc000 & (u32)(src)) >> 0xe) > +#define INTXSTATUSMASK 0x6c > +#define PIM1_1L 0x80 > +#define IBAR2 0x98 > +#define IR2MSK 0x9c > +#define PIM2_1L 0xa0 > +#define OMR1BARL 0x100 > +#define OMR2BARL 0x118 > +#define CFGBARL 0x154 > +#define CFGBARH 0x158 > +#define CFGCTL 0x15c > +#define RTDID 0x160 > +#define CFG_CONSTANTS_31_00 0x2000 > +#define CFG_CONSTANTS_63_32 0x2004 > +#define CFG_CONSTANTS_159_128 0x2010 > +#define CFG_CONSTANTS_415_384 0x2030 > +#define ENABLE_L1S_POWER_MGMT_SET(dst, src) (((dst) & ~0x02000000) | \ > + (((u32)(src) << 0x19) & \ > + 0x02000000)) > +#define CFG_CONSTANTS_479_448 0x2038 > +#define CFG_8G_CONSTANTS_31_0 0x2100 > +#define MGMT_US_PORT_TX_PRESET_SET(dst, src) (((dst) & ~0xf00)| \ > + (((u32)(src) << 0x8) & 0xf00)) > +#define MGMT_DS_PORT_TX_PRESET_SET(dst, src) (((dst) & ~0xf) | \ > + (((u32)(src)) & 0xf)) > + > +#define CFG_8G_CONSTANTS_159_128 0x2110 > +#define EQ_UPDN_POST_STEP_SET(dst, src) (((dst) & ~0x30) | \ > + (((u32)(src) << 0x4) & \ > + 0x30)) > +#define CFG_8G_CONSTANTS_287_256 0x2120 > +#define CFG_8G_CONSTANTS_319_288 0x2124 > +#define CFG_8G_CONSTANTS_351_320 0x2128 > +#define CFG_8G_CONSTANTS_383_352 0x212c > +#define EQ_PRE_CURSOR_LANE0_SET(dst, src) (((dst) & ~0xff) | \ > + (((u32)(src)) & 0xff)) > +#define EQ_PRE_CURSOR_LANE1_SET(dst, src) (((dst) & ~0x00ff0000) | \ > + (((u32)(src) << 0x10) & \ > + 0x00ff0000)) > + > +#define CFG_CONTROL_63_32 0x2204 > +#define CFG_CONTROL_95_64 0x2208 > +#define CFG_CONTROL_191_160 0x2214 > +#define PCIE_STATUS_31_0 0x2600 > +#define MEM_RAM_SHUTDOWN 0xd070 > +#define BLOCK_MEM_RDY 0xd074 > + > +#define PCI_PRIMARY_BUS_MASK 0x00ffffff > +#define REVISION_ID_MASK 0x000000ff > +#define SLOT_IMPLEMENTED_MASK 0x04000000 > +#define DEVICE_PORT_TYPE_MASK 0x03c00000 > +#define ADVT_INFINITE_CREDITS 0x00000200 > +#define PM_FORCE_RP_MODE_MASK 0x00000400 > +#define SWITCH_PORT_MODE_MASK 0x00000800 > +#define CLASS_CODE_MASK 0xffffff00 > +#define LINK_UP_MASK 0x00000100 > +#define AER_OPTIONAL_ERROR_EN 0xffc00000 > +#define DWNSTRM_EQ_SKP_PHS_2_3 0x00010000 > +#define DIRECT_TO_5GTS_MASK 0x00020000 > +#define SUPPORT_5GTS_MASK 0x00010000 > +#define DIRECT_TO_8GTS_MASK 0x00008000 > +#define SUPPORT_8GTS_MASK 0x00004000 > +#define XGENE_PCIE_DEV_CTRL 0x2f0f > +#define AXI_EP_CFG_ACCESS 0x10000 > +#define ENABLE_ASPM 0x08000000 > +#define XGENE_PORT_TYPE_RC 0x05000000 > +#define BLOCK_MEM_RDY_VAL 0xFFFFFFFF > +#define EN_COHERENCY 0xF0000000 > +#define EN_REG 0x00000001 > +#define OB_LO_IO 0x00000002 > +#define XGENE_PCIE_VENDORID 0x19AA > +#define XGENE_PCIE_BRIDGE_DEVICEID 0xE008 > +#define XGENE_PCIE_DEVICEID 0xCAFE > +#define XGENE_PCIE_TIMEOUT (500*1000) /* us */ > +#define XGENE_PCIE_MAX_REGIONS 3 > +#define XGENE_LTSSM_DETECT_WAIT 20 > +#define XGENE_LTSSM_L0_WAIT 4 > +#define XGENE_PCIE_PHY_DRV "pcie-8g" > +#define XGENE_PCIE_CLK_DRV "pcieclk" > +#define XGENE_PCIE_MAX_PORTS 5 > +#define XGENE_PCIE_EP_MEM_SIZE 0x100000 > + > +enum { > + PTYPE_ENDPOINT = 0x0, > + PTYPE_LEGACY_ENDPOINT = 0x1, > + PTYPE_ROOT_PORT = 0x4, > + > + LNKW_X1 = 0x1, > + LNKW_X4 = 0x4, > + LNKW_X8 = 0x8, > + > + PCIE_GEN1 = 0x0, /* 2.5G */ > + PCIE_GEN2 = 0x1, /* 5.0G */ > + PCIE_GEN3 = 0x2, /* 8.0G */ > +}; > + > +enum { > + XGENE_MEM, > + XGENE_MSI, > + XGENE_IO, > + XGENE_RES /* termination */ > +}; > + > +struct xgene_pcie_ep_info { > + void *reg_virt; /* maps to outbound space of RC */ > + dma_addr_t reg_phys; /* Physical address of reg space */ > +}; > + > +struct xgene_pcie_port { > + struct device_node *node; > + struct resource res[XGENE_RES]; > + u8 type; > + u8 link_up; > + u8 link_speed; > + u32 first_busno; > + void *csr_base; > + void *cfg_base; > + struct device *dev; > + struct xgene_pcie_ep_info ep_info; > + struct clk *clk; > +}; > + > +#ifdef CONFIG_64BIT > +#define pci_io_offset(s) (s & 0xff00000000) > +#else > +#define pci_io_offset(s) (s & 0x00000000) > +#endif /* CONFIG_64BIT */ > + > +static inline struct xgene_pcie_port * > +xgene_pcie_sys_to_port(struct pci_sys_data *sys) > +{ > + return (struct xgene_pcie_port *)sys->private_data; > +} > + > +static inline struct xgene_pcie_port * > +xgene_pcie_bus_to_port(struct pci_bus *bus) > +{ > + struct pci_sys_data *sys = bus->sysdata; > + return xgene_pcie_sys_to_port(sys); > +} > + > +/* IO ports are memory mapped */ > +void __iomem *__pci_ioport_map(struct pci_dev *dev, unsigned long port, > + unsigned int nr) > +{ > + return devm_ioremap_nocache(&dev->dev, port, nr); > +} > + > +/* PCIE Out/In to CSR */ > +static inline void xgene_pcie_out32(void *addr, u32 val) > +{ > + pr_debug("pcie csr wr: 0x%llx 0x%08x\n", (phys_addr_t)addr, val); > + writel(val, addr); > +} > + > +static inline void xgene_pcie_in32(void *addr, u32 *val) > +{ > + *val = readl(addr); > + pr_debug("pcie csr rd: 0x%llx 0x%08x\n", (phys_addr_t)addr, *val); > +} > + > +/* PCIE Configuration Out/In */ > +static inline void xgene_pcie_cfg_out32(void *addr, u32 val) > +{ > + writel(val, addr); > +} > + > +static inline void xgene_pcie_cfg_out16(void *addr, u16 val) > +{ > + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; > + u32 val32 = readl((void *)temp_addr); > + > + switch ((phys_addr_t) addr & 0x3) { > + case 2: > + val32 &= ~0xFFFF0000; > + val32 |= (u32) val << 16; > + break; > + case 0: > + default: > + val32 &= ~0xFFFF; > + val32 |= val; > + break; > + } > + writel(val32, (void *)temp_addr); > +} > + > +static inline void xgene_pcie_cfg_out8(void *addr, u8 val) > +{ > + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; > + u32 val32 = readl((void *)temp_addr); > + > + switch ((phys_addr_t) addr & 0x3) { > + case 0: > + val32 &= ~0xFF; > + val32 |= val; > + break; > + case 1: > + val32 &= ~0xFF00; > + val32 |= (u32) val << 8; > + break; > + case 2: > + val32 &= ~0xFF0000; > + val32 |= (u32) val << 16; > + break; > + case 3: > + default: > + val32 &= ~0xFF000000; > + val32 |= (u32) val << 24; > + break; > + } > + writel(val32, (void *)temp_addr); > +} > + > +static inline void xgene_pcie_cfg_in32(void *addr, u32 *val) > +{ > + *val = readl(addr); > +} > + > +static inline void xgene_pcie_cfg_in16(void *addr, u16 *val) > +{ > + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; > + u32 val32; > + > + val32 = readl((void *)temp_addr); > + > + switch ((phys_addr_t) addr & 0x3) { > + case 2: > + *val = val32 >> 16; > + break; > + case 0: > + default: > + *val = val32; > + break; > + } > +} > + > +static inline void xgene_pcie_cfg_in8(void *addr, u8 *val) > +{ > + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; > + u32 val32; > + > + val32 = readl((void *)temp_addr); > + > + switch ((phys_addr_t) addr & 0x3) { > + case 3: > + *val = val32 >> 24; > + break; > + case 2: > + *val = val32 >> 16; > + break; > + case 1: > + *val = val32 >> 8; > + break; > + case 0: > + default: > + *val = val32; > + break; > + } > +} > + > +static void __iomem *xgene_pcie_get_cfg_base(struct pci_bus *bus) > +{ > + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); > + phys_addr_t addr = (phys_addr_t) port->cfg_base; > + > + if (bus->number >= (port->first_busno + 1)) > + addr |= AXI_EP_CFG_ACCESS; > + > + return (void *)addr; > +} > + > +static void xgene_pcie_set_rtdid_reg(struct pci_bus *bus, uint devfn) > +{ > + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); > + unsigned int b, d, f; > + u32 rtdid_val = 0; > + > + b = bus->number; > + d = PCI_SLOT(devfn); > + f = PCI_FUNC(devfn); > + > + if (bus->number == port->first_busno) > + rtdid_val = (b << 24) | (d << 19) | (f << 16); > + else if (bus->number >= (port->first_busno + 1)) > + rtdid_val = (port->first_busno << 24) | > + (b << 8) | (d << 3) | f; > + > + xgene_pcie_out32(port->csr_base + RTDID, rtdid_val); > + /* read the register back to ensure flush */ > + xgene_pcie_in32(port->csr_base + RTDID, &rtdid_val); > +} > + > +static int xgene_pcie_read_config(struct pci_bus *bus, unsigned int devfn, > + int offset, int len, u32 *val) > +{ > + void __iomem *addr; > + u8 val8; > + u16 val16; > + > + if (pci_is_root_bus(bus) && devfn != 0) > + return PCIBIOS_DEVICE_NOT_FOUND; > + > + xgene_pcie_set_rtdid_reg(bus, devfn); > + addr = xgene_pcie_get_cfg_base(bus); > + switch (len) { > + case 1: > + xgene_pcie_cfg_in8(addr + offset, &val8); > + *val = val8; > + break; > + case 2: > + xgene_pcie_cfg_in16(addr + offset, &val16); > + *val = val16; > + break; > + default: > + xgene_pcie_cfg_in32(addr + offset, val); > + break; > + } > + return PCIBIOS_SUCCESSFUL; > +} > + > +static int xgene_pcie_write_config(struct pci_bus *bus, unsigned int devfn, > + int offset, int len, u32 val) > +{ > + void __iomem *addr; > + > + if (pci_is_root_bus(bus) && devfn != 0) > + return PCIBIOS_DEVICE_NOT_FOUND; > + > + xgene_pcie_set_rtdid_reg(bus, devfn); > + addr = xgene_pcie_get_cfg_base(bus); > + switch (len) { > + case 1: > + xgene_pcie_cfg_out8(addr + offset, (u8) val); > + break; > + case 2: > + xgene_pcie_cfg_out16(addr + offset, (u16) val); > + break; > + default: > + xgene_pcie_cfg_out32(addr + offset, val); > + break; > + } > + return PCIBIOS_SUCCESSFUL; > +} > + > +static struct pci_ops xgene_pcie_ops = { > + .read = xgene_pcie_read_config, > + .write = xgene_pcie_write_config > +}; > + > +static void xgene_pcie_setup_lanes(struct xgene_pcie_port *port) > +{ > + void *csr_base = port->csr_base; > + u32 val; > + > + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_287_256, &val); > + val = EQ_PRE_CURSOR_LANE0_SET(val, 0x7); > + val = EQ_PRE_CURSOR_LANE1_SET(val, 0x7); > + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_287_256, val); > + > + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_319_288, &val); > + val = EQ_PRE_CURSOR_LANE0_SET(val, 0x7); > + val = EQ_PRE_CURSOR_LANE1_SET(val, 0x7); > + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_319_288, val); > + > + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_351_320, &val); > + val = EQ_PRE_CURSOR_LANE0_SET(val, 0x7); > + val = EQ_PRE_CURSOR_LANE1_SET(val, 0x7); > + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_351_320, val); > + > + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_383_352, &val); > + val = EQ_PRE_CURSOR_LANE0_SET(val, 0x7); > + val = EQ_PRE_CURSOR_LANE1_SET(val, 0x7); > + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_383_352, val); > + > + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_159_128, &val); > + val = EQ_UPDN_POST_STEP_SET(val, 0x1); > + val = EQ_UPDN_POST_STEP_SET(val, 0x1); > + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_159_128, val); > +} > + > +static void xgene_pcie_setup_link(struct xgene_pcie_port *port) > +{ > + void *csr_base = port->csr_base; > + u32 val; > + > + xgene_pcie_in32(csr_base + CFG_CONSTANTS_479_448, &val); > + switch (port->link_speed) { > + case PCIE_GEN1: > + val &= ~SUPPORT_5GTS_MASK; > + val &= ~SUPPORT_8GTS_MASK; > + break; > + case PCIE_GEN2: > + val |= SUPPORT_5GTS_MASK; > + val |= DIRECT_TO_5GTS_MASK; > + val &= ~SUPPORT_8GTS_MASK; > + val &= ~DIRECT_TO_8GTS_MASK; > + break; > + case PCIE_GEN3: > + val |= DIRECT_TO_8GTS_MASK; > + val |= SUPPORT_5GTS_MASK; > + val |= SUPPORT_8GTS_MASK; > + val |= DIRECT_TO_5GTS_MASK; > + break; > + } > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_479_448, val); > + > + xgene_pcie_in32(csr_base + CFG_CONSTANTS_479_448, &val); > + val &= ~ADVT_INFINITE_CREDITS; > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_479_448, val); > + > + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_31_0, &val); > + val |= MGMT_DS_PORT_TX_PRESET_SET(val, 0x7); > + val |= MGMT_US_PORT_TX_PRESET_SET(val, 0x7); > + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_31_0, val); > + > + if (port->link_speed == PCIE_GEN3) { > + xgene_pcie_in32(csr_base + CFG_8G_CONSTANTS_31_0, &val); > + val |= DWNSTRM_EQ_SKP_PHS_2_3; > + xgene_pcie_out32(csr_base + CFG_8G_CONSTANTS_31_0, val); > + } > +} > + > +static void xgene_pcie_program_core(void *csr_base) > +{ > + u32 val; > + > + xgene_pcie_in32(csr_base + CFG_CONSTANTS_31_00, &val); > + val |= AER_OPTIONAL_ERROR_EN; > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_31_00, val); > + xgene_pcie_out32(csr_base + INTXSTATUSMASK, 0x0); > + xgene_pcie_in32(csr_base + CFG_CONTROL_63_32, &val); > + val = (val & ~0xffff) | XGENE_PCIE_DEV_CTRL; > + xgene_pcie_out32(csr_base + CFG_CONTROL_63_32, val); > +} > + > +static u64 xgene_pcie_set_ib_mask(void *csr_base, u32 addr, u32 flags, > + resource_size_t size) > +{ > + u64 val64 = 0; > + u32 val32 = 0; > + u32 val; > + > + if (size >= SZ_1K) > + val64 = (~(size - 1) & PCI_BASE_ADDRESS_MEM_MASK) | flags; > + > + xgene_pcie_in32(csr_base + addr, &val32); > + val = (val32 & 0x0000ffff) | (lower_32_bits(val64) << 16); > + xgene_pcie_out32(csr_base + addr, val); > + > + xgene_pcie_in32(csr_base + addr + 0x04, &val32); > + val = (val32 & 0xffff0000) | (lower_32_bits(val64) >> 16); > + xgene_pcie_out32(csr_base + addr + 0x04, val); > + > + xgene_pcie_in32(csr_base + addr + 0x04, &val32); > + val = (val32 & 0x0000ffff) | (upper_32_bits(val64) << 16); > + xgene_pcie_out32(csr_base + addr + 0x04, val); > + > + xgene_pcie_in32(csr_base + addr + 0x08, &val32); > + val = (val32 & 0xffff0000) | (upper_32_bits(val64) >> 16); > + xgene_pcie_out32(csr_base + addr + 0x08, val); > + > + return val64; > +} > + > +static void xgene_pcie_config_pims(void *csr_base, u32 addr, > + u64 pim, resource_size_t size) > +{ > + u32 val; > + > + xgene_pcie_out32(csr_base + addr, lower_32_bits(pim)); > + val = upper_32_bits(pim) | EN_COHERENCY; > + xgene_pcie_out32(csr_base + addr + 0x04, val); > + xgene_pcie_out32(csr_base + addr + 0x08, 0x0); > + xgene_pcie_out32(csr_base + addr + 0x0c, 0x0); > + val = lower_32_bits(size); > + xgene_pcie_out32(csr_base + addr + 0x10, val); > + val = upper_32_bits(size); > + xgene_pcie_out32(csr_base + addr + 0x14, val); > +} > + > +static void xgene_pcie_poll_linkup(struct xgene_pcie_port *port, u32 *lanes) > +{ > + void *csr_base = port->csr_base; > + u32 val32; > + u64 start_time, time; > + > + /* > + * A component enters the LTSSM Detect state within > + * 20ms of the end of fundamental core reset. > + */ > + msleep(XGENE_LTSSM_DETECT_WAIT); > + port->link_up = 0; > + start_time = jiffies; > + do { > + xgene_pcie_in32(csr_base + PCIECORE_CTLANDSTATUS, &val32); > + if (val32 & LINK_UP_MASK) { > + port->link_up = 1; > + port->link_speed = PIPE_PHY_RATE_RD(val32); > + xgene_pcie_in32(csr_base + PCIE_STATUS_31_0, &val32); > + *lanes = val32 >> 26; > + } > + time = jiffies_to_msecs(jiffies - start_time); > + } while ((!port->link_up) || (time <= XGENE_LTSSM_L0_WAIT)); > +} > + > +static void xgene_pcie_setup_root_complex(struct xgene_pcie_port *port) > +{ > + void *csr_base = port->csr_base; > + u32 val; > + > + val = (XGENE_PCIE_BRIDGE_DEVICEID << 16) | XGENE_PCIE_VENDORID; > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_31_00, val); > + > + xgene_pcie_in32(csr_base + CFG_CONSTANTS_63_32, &val); > + val &= ~CLASS_CODE_MASK; > + val |= PCI_CLASS_BRIDGE_PCI << 16; > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_63_32, val); > + > + xgene_pcie_in32(csr_base + CFG_CONSTANTS_479_448, &val); > + val |= SWITCH_PORT_MODE_MASK; > + val &= ~PM_FORCE_RP_MODE_MASK; > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_479_448, val); > + xgene_pcie_setup_link(port); > + xgene_pcie_setup_lanes(port); > + xgene_pcie_in32(csr_base + CFG_CONTROL_191_160, &val); > + val &= ~DEVICE_PORT_TYPE_MASK; > + val |= XGENE_PORT_TYPE_RC; > + xgene_pcie_out32(csr_base + CFG_CONTROL_191_160, val); > + > + xgene_pcie_in32(csr_base + CFG_CONTROL_95_64, &val); > + val |= ENABLE_ASPM; > + xgene_pcie_out32(csr_base + CFG_CONTROL_95_64, val); > + > + xgene_pcie_in32(csr_base + CFG_CONSTANTS_415_384, &val); > + val = ENABLE_L1S_POWER_MGMT_SET(val, 1); > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_415_384, val); > +} > + > +static void xgene_pcie_setup_endpoint(struct xgene_pcie_port *port) > +{ > + void *csr_base = port->csr_base; > + u32 val; > + > + xgene_pcie_in32(csr_base + CFG_CONSTANTS_479_448, &val); > + val &= ~SWITCH_PORT_MODE_MASK; > + val &= ~PM_FORCE_RP_MODE_MASK; > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_479_448, val); > + > + xgene_pcie_in32(csr_base + CFG_CONTROL_191_160, &val); > + val &= ~DEVICE_PORT_TYPE_MASK; > + val &= ~SLOT_IMPLEMENTED_MASK; > + xgene_pcie_out32(csr_base + CFG_CONTROL_191_160, val); > + > + xgene_pcie_in32(csr_base + CFG_CONSTANTS_31_00, &val); > + val = (XGENE_PCIE_DEVICEID << 16) | XGENE_PCIE_VENDORID; > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_31_00, val); > + > + xgene_pcie_in32(csr_base + CFG_CONSTANTS_63_32, &val); > + val &= REVISION_ID_MASK; > + val |= PCI_CLASS_BRIDGE_OTHER << 16; > + xgene_pcie_out32(csr_base + CFG_CONSTANTS_63_32, val); > + > + xgene_pcie_setup_link(port); > +} > + > +static void xgene_pcie_setup_port(struct xgene_pcie_port *port) > +{ > + int type = port->type; > + > + xgene_pcie_program_core(port->csr_base); > + if (type == PTYPE_ROOT_PORT) > + xgene_pcie_setup_root_complex(port); > + else > + xgene_pcie_setup_endpoint(port); > +} > + > +/* Return 0 on success */ > +static int xgene_pcie_init_ecc(struct xgene_pcie_port *port) > +{ > + void *csr_base = port->csr_base; > + int timeout = XGENE_PCIE_TIMEOUT; > + u32 val; > + > + xgene_pcie_in32(csr_base + MEM_RAM_SHUTDOWN, &val); > + if (val == 0) > + return 0; > + xgene_pcie_out32(csr_base + MEM_RAM_SHUTDOWN, 0x0); > + do { > + xgene_pcie_in32(csr_base + BLOCK_MEM_RDY, &val); > + udelay(1); > + } while ((val != BLOCK_MEM_RDY_VAL) && timeout--); > + > + return !(timeout > 0); > +} > + > +static int xgene_pcie_init_port(struct xgene_pcie_port *port) > +{ > + int rc; > + > + port->clk = clk_get(port->dev, XGENE_PCIE_CLK_DRV); > + if (IS_ERR_OR_NULL(port->clk)) { > + dev_err(port->dev, "clock not available\n"); > + return -ENODEV; > + } > + > + rc = clk_prepare_enable(port->clk); > + if (rc) { > + dev_err(port->dev, "clock enable failed\n"); > + return rc; > + } > + > + rc = xgene_pcie_init_ecc(port); > + if (rc) { > + dev_err(port->dev, "memory init failed\n"); > + return rc; > + } > + > + return 0; > +} > + > +struct device_node *pcibios_get_phb_of_node(struct pci_bus *bus) > +{ > + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); > + > + return of_node_get(port->node); > +} > + > +static void xgene_pcie_fixup_bridge(struct pci_dev *dev) > +{ > + int i; > + > + for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) { > + dev->resource[i].start = dev->resource[i].end = 0; > + dev->resource[i].flags = 0; > + } > +} > +DECLARE_PCI_FIXUP_HEADER(XGENE_PCIE_VENDORID, XGENE_PCIE_BRIDGE_DEVICEID, > + xgene_pcie_fixup_bridge); > + > +static void xgene_pcie_setup_primary_bus(struct xgene_pcie_port *port, > + u32 first_busno, u32 last_busno) > +{ > + u32 val; > + void *cfg_addr = port->cfg_base; > + > + xgene_pcie_in32(cfg_addr + PCI_PRIMARY_BUS, &val); > + val &= ~PCI_PRIMARY_BUS_MASK; > + val |= (last_busno << 16) | ((first_busno + 1) << 8) | (first_busno); > + xgene_pcie_out32(cfg_addr + PCI_PRIMARY_BUS, val); > +} > + > +/* > + * read configuration values from DTS > + */ > +static int xgene_pcie_read_dts_config(struct xgene_pcie_port *port) > +{ > + struct device_node *np = port->node; > + struct resource csr_res; > + u32 val32; > + int ret; > + const u8 *val; > + > + val = of_get_property(np, "device_type", NULL); > + if ((val != NULL) && !strcmp(val, "ep")) > + port->type = PTYPE_ENDPOINT; > + else > + port->type = PTYPE_ROOT_PORT; > + > + ret = of_property_read_u32(np, "link_speed", &val32); > + if (ret == 0) > + port->link_speed = val32; > + else > + port->link_speed = PCIE_GEN3; > + > + /* Get configured CSR space registers address */ > + if (of_address_to_resource(np, 0, &csr_res)) > + return -EINVAL; > + > + port->csr_base = devm_ioremap_nocache(port->dev, csr_res.start, > + resource_size(&csr_res)); > + if (port->csr_base == NULL) > + return -ENOMEM; > + > + return 0; > +} > + > +static int xgene_pcie_alloc_ep_mem(struct xgene_pcie_port *port) > +{ > + struct xgene_pcie_ep_info *ep = &port->ep_info; > + > + ep->reg_virt = dma_alloc_coherent(port->dev, XGENE_PCIE_EP_MEM_SIZE, > + &ep->reg_phys, GFP_KERNEL); > + if (ep->reg_virt == NULL) > + return -ENOMEM; > + > + dev_info(port->dev, "EP: Virt - %p Phys - 0x%llx Size - 0x%x\n", > + ep->reg_virt, (u64) ep->reg_phys, XGENE_PCIE_EP_MEM_SIZE); > + return 0; > +} > + > +static int xgene_pcie_populate_inbound_regions(struct xgene_pcie_port *port) > +{ > + struct resource *msi_res = &port->res[XGENE_MSI]; > + phys_addr_t ddr_size = memblock_phys_mem_size(); > + phys_addr_t ddr_base = memblock_start_of_DRAM(); > + void *csr_base = port->csr_base; > + void *cfg_addr = port->cfg_base; > + u64 val64, size; > + u32 val, mask_addr; > + u32 flags = PCI_BASE_ADDRESS_MEM_PREFETCH | > + PCI_BASE_ADDRESS_MEM_TYPE_64; > + > + if (port->type == PTYPE_ROOT_PORT) { > + mask_addr = CFG_CONSTANTS_159_128; > + xgene_pcie_set_ib_mask(csr_base, mask_addr, flags, ddr_size); > + val = (lower_32_bits(ddr_base) & PCI_BASE_ADDRESS_MEM_MASK) | > + flags; > + xgene_pcie_out32(cfg_addr + PCI_BASE_ADDRESS_0, val); > + val = upper_32_bits(ddr_base); > + xgene_pcie_out32(cfg_addr + PCI_BASE_ADDRESS_1, val); > + xgene_pcie_config_pims(csr_base, PIM1_1L, ddr_base, ddr_size); > + } else { > + struct xgene_pcie_ep_info *ep = &port->ep_info; > + if (xgene_pcie_alloc_ep_mem(port)) > + return -ENOMEM; > + mask_addr = CFG_CONSTANTS_159_128; > + size = XGENE_PCIE_EP_MEM_SIZE; > + xgene_pcie_set_ib_mask(csr_base, mask_addr, flags, size); > + xgene_pcie_config_pims(csr_base, PIM1_1L, ep->reg_phys, size); > + } > + val64 = 0; > + size = resource_size(msi_res); > + if (size >= SZ_1M) > + val64 = ~(size - 1) | EN_REG; > + xgene_pcie_out32(csr_base + IBAR2, msi_res->start); > + xgene_pcie_out32(csr_base + IR2MSK, lower_32_bits(val64)); > + xgene_pcie_config_pims(csr_base, PIM2_1L, msi_res->start, size); > + return 0; > +} > + > +static void xgene_pcie_setup_ob_reg(void *csr_base, u32 addr, u32 index, > + struct resource *res) > +{ > + resource_size_t size = resource_size(res); > + u64 val64 = 0; > + u32 min_size = 0; > + u32 flag = EN_REG; > + > + switch (index) { > + case XGENE_MEM: > + min_size = SZ_128M; > + break; > + case XGENE_IO: > + min_size = 128; > + flag |= OB_LO_IO; > + break; > + } > + if (size >= min_size) > + val64 = ~(size - 1) | flag; > + else > + pr_warn("resource size 0x%llx less than minimum 0x%x\n", > + (u64)size, min_size); > + xgene_pcie_out32(csr_base + addr, lower_32_bits(res->start)); > + xgene_pcie_out32(csr_base + addr + 0x04, upper_32_bits(res->start)); > + xgene_pcie_out32(csr_base + addr + 0x08, lower_32_bits(val64)); > + xgene_pcie_out32(csr_base + addr + 0x0c, upper_32_bits(val64)); > + xgene_pcie_out32(csr_base + addr + 0x10, 0x0); > + xgene_pcie_out32(csr_base + addr + 0x14, 0x0); > +} > + > +static int xgene_pcie_map_cfg(struct xgene_pcie_port *port, > + struct of_pci_range *range) > +{ > + struct device *dev = port->dev; > + u64 addr = range->cpu_addr; > + resource_size_t size = range->size; > + void *csr_base = port->csr_base; > + > + port->cfg_base = devm_ioremap_nocache(dev, addr, size); > + if (port->cfg_base == NULL) { > + dev_err(dev, "failed to map cfg region!"); > + return -ENOMEM; > + } > + > + xgene_pcie_out32(csr_base + CFGBARL, lower_32_bits(addr)); > + xgene_pcie_out32(csr_base + CFGBARH, upper_32_bits(addr)); > + xgene_pcie_out32(csr_base + CFGCTL, EN_REG); > + > + return 0; > +} > + > +static int xgene_pcie_parse_map_ranges(struct xgene_pcie_port *port) > +{ > + struct device_node *np = port->node; > + struct of_pci_range range; > + struct of_pci_range_parser parser; > + struct device *dev = port->dev; > + u32 cfg_map_done = 0; > + int ret; > + > + if (of_pci_range_parser_init(&parser, np)) { > + dev_err(dev, "missing ranges property\n"); > + return -EINVAL; > + } > + > + /* Get the I/O, memory, config ranges from DT */ > + for_each_of_pci_range(&parser, &range) { > + struct resource *res = NULL; > + u64 restype = range.flags & IORESOURCE_TYPE_BITS; > + u64 end = range.cpu_addr + range.size - 1; > + dev_dbg(port->dev, "0x%08x 0x%016llx..0x%016llx -> 0x%016llx\n", > + range.flags, range.cpu_addr, end, range.pci_addr); > + > + switch (restype) { > + case IORESOURCE_IO: > + res = &port->res[XGENE_IO]; > + of_pci_range_to_resource(&range, np, res); > + xgene_pcie_setup_ob_reg(port->csr_base, OMR1BARL, > + XGENE_IO, res); > + break; > + case IORESOURCE_MEM: > + res = &port->res[XGENE_MEM]; > + of_pci_range_to_resource(&range, np, res); > + xgene_pcie_setup_ob_reg(port->csr_base, OMR2BARL, > + XGENE_MEM, res); > + break; > + case 0: > + if (!cfg_map_done) { > + /* config region */ > + if (port->type == PTYPE_ROOT_PORT) { > + ret = xgene_pcie_map_cfg(port, &range); > + if (ret) > + return ret; > + } > + cfg_map_done = 1; > + } else { > + /* msi region */ > + res = &port->res[XGENE_MSI]; > + of_pci_range_to_resource(&range, np, res); > + } > + break; > + default: > + dev_err(dev, "invalid io resource!"); > + return -EINVAL; > + } > + } > + > + return xgene_pcie_populate_inbound_regions(port); > +} > + > +static int xgene_pcie_setup(int nr, struct pci_sys_data *sys) > +{ > + struct xgene_pcie_port *pp = xgene_pcie_sys_to_port(sys); > + > + if (pp == NULL) > + return 0; > + > + if (pp->type == PTYPE_ENDPOINT) > + return 0; > + > + sys->io_offset = pci_io_offset(pp->res[XGENE_IO].start); > + sys->mem_offset = pci_io_offset(pp->res[XGENE_MEM].start); > + > + BUG_ON(request_resource(&iomem_resource, &pp->res[XGENE_IO]) || > + request_resource(&iomem_resource, &pp->res[XGENE_MEM])); > + > + pci_add_resource_offset(&sys->resources, &pp->res[XGENE_MEM], > + sys->mem_offset); > + pci_add_resource_offset(&sys->resources, &pp->res[XGENE_IO], > + sys->io_offset); > + return 1; > +} > + > +static int xgene_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) > +{ > + return of_irq_parse_and_map_pci(dev, slot, pin); > +} > + > +static struct pci_bus __init *xgene_pcie_scan_bus(int nr, > + struct pci_sys_data *sys) > +{ > + struct xgene_pcie_port *pp = xgene_pcie_sys_to_port(sys); > + > + pp->first_busno = sys->busnr; > + xgene_pcie_setup_primary_bus(pp, sys->busnr, 0xff); > + return pci_scan_root_bus(NULL, sys->busnr, &xgene_pcie_ops, > + sys, &sys->resources); > +} > + > +static struct hw_pci xgene_pcie_hw __initdata = { > + .nr_controllers = XGENE_PCIE_MAX_PORTS, > + .setup = xgene_pcie_setup, > + .scan = xgene_pcie_scan_bus, > + .map_irq = xgene_pcie_map_irq, > +}; > + > +static int __init xgene_pcie_probe_bridge(struct platform_device *pdev) > +{ > + struct device_node *np = of_node_get(pdev->dev.of_node); > + struct xgene_pcie_port *port; > + u32 lanes = 0; > + static int index; > + int ret; > + > + port = devm_kzalloc(&pdev->dev, sizeof(*port), GFP_KERNEL); > + if (port == NULL) > + return -ENOMEM; > + port->node = np; > + port->dev = &pdev->dev; > + > + ret = xgene_pcie_read_dts_config(port); > + if (ret) > + return ret; > + > + ret = xgene_pcie_init_port(port); > + if (ret) > + goto skip; > + > + xgene_pcie_setup_port(port); > + ret = xgene_pcie_parse_map_ranges(port); > + if (ret) > + goto skip; > + > + if (port->type == PTYPE_ROOT_PORT) > + xgene_pcie_poll_linkup(port, &lanes); > +skip: > + if (port->type == PTYPE_ROOT_PORT) { > + if (!port->link_up) > + dev_info(port->dev, "(rc) link down\n"); > + else > + dev_info(port->dev, "(rc) x%d gen-%d link up\n", > + lanes, port->link_speed + 1); > + } else > + dev_info(port->dev, "(ep)\n"); > + > + xgene_pcie_hw.private_data[index++] = port; > + platform_set_drvdata(pdev, port); > + return 0; > +} > + > +static const struct of_device_id xgene_pcie_match_table[] __initconst = { > + {.compatible = "apm,xgene-pcie",}, > + {}, > +}; > + > +static struct platform_driver xgene_pcie_driver = { > + .driver = { > + .name = "xgene-pcie", > + .owner = THIS_MODULE, > + .of_match_table = of_match_ptr(xgene_pcie_match_table), > + }, > +}; > + > +static int __init xgene_pcie_init(void) > +{ > + void *private; > + int ret; > + > + pr_info("X-Gene: PCIe driver\n"); > + > + /* allocate private data to keep xgene_pcie_port information */ > + private = kzalloc((XGENE_PCIE_MAX_PORTS * sizeof(void *)), GFP_KERNEL); > + if (private == NULL) > + return -ENOMEM; > + xgene_pcie_hw.private_data = private; > + ret = platform_driver_probe(&xgene_pcie_driver, > + xgene_pcie_probe_bridge); > + if (ret) > + return ret; > + pci_common_init(&xgene_pcie_hw); > + return 0; > +} > + > +module_init(xgene_pcie_init); > + > +MODULE_AUTHOR("Tanmay Inamdar <tinamdar@apm.com>"); > +MODULE_DESCRIPTION("APM X-Gene PCIe driver"); > +MODULE_LICENSE("GPL v2"); > -- > 1.7.9.5 > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver 2013-12-23 8:02 ` [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver Tanmay Inamdar 2014-01-02 21:08 ` Bjorn Helgaas @ 2014-01-03 12:07 ` Arnd Bergmann 2014-01-07 2:41 ` Tanmay Inamdar 2014-01-06 1:47 ` Jingoo Han 2 siblings, 1 reply; 27+ messages in thread From: Arnd Bergmann @ 2014-01-03 12:07 UTC (permalink / raw) To: linux-arm-kernel Cc: Tanmay Inamdar, Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel, jcm On Monday 23 December 2013, Tanmay Inamdar wrote: > This patch adds the AppliedMicro X-gene SOC PCIe controller driver. > APM X-Gene PCIe controller supports maximum upto 8 lanes and > GEN3 speed. X-Gene has maximum 5 PCIe ports supported. > > Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> > --- > drivers/pci/host/Kconfig | 5 + > drivers/pci/host/Makefile | 1 + > drivers/pci/host/pcie-xgene.c | 1017 +++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 1023 insertions(+) > create mode 100644 drivers/pci/host/pcie-xgene.c > > diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig > index 47d46c6..6d8fcbc 100644 > --- a/drivers/pci/host/Kconfig > +++ b/drivers/pci/host/Kconfig > @@ -33,4 +33,9 @@ config PCI_RCAR_GEN2 > There are 3 internal PCI controllers available with a single > built-in EHCI/OHCI host controller present on each one. > > +config PCI_XGENE > + bool "X-Gene PCIe controller" > + depends on ARCH_XGENE > + depends on OF Please add a help text here. > +#ifdef CONFIG_ARM64 > +#include <asm/pcibios.h> > +#else > +#include <asm/mach/pci.h> > +#endif What is !ARM64 case? Is this for PowerPC or ARM? Since you depend on ARCH_XGENE in Kconfig I guess neither case can actually happen, so you can remove the #ifdef. > +#define CFG_CONSTANTS_31_00 0x2000 > +#define CFG_CONSTANTS_63_32 0x2004 > +#define CFG_CONSTANTS_159_128 0x2010 > +#define CFG_CONSTANTS_415_384 0x2030 These macros do not seem helpful. If you don't have meaningful register names, just don't provide any and address the registers by index. > +#define ENABLE_L1S_POWER_MGMT_SET(dst, src) (((dst) & ~0x02000000) | \ > + (((u32)(src) << 0x19) & \ > + 0x02000000)) Makes this an inline function, or open-code it in the caller if there is only one. > +#ifdef CONFIG_64BIT > +#define pci_io_offset(s) (s & 0xff00000000) > +#else > +#define pci_io_offset(s) (s & 0x00000000) > +#endif /* CONFIG_64BIT */ Why is this needed? The I/O space can never be over 0xffffffff, or in practice 0xffff. My best guess is that you have a bug in the function parsing your ranges property, or in the property value. > +static inline struct xgene_pcie_port * > +xgene_pcie_sys_to_port(struct pci_sys_data *sys) > +{ > + return (struct xgene_pcie_port *)sys->private_data; > +} You shouldn't need the cast, or the accessor function, since private_data is already a void pointer. > +/* IO ports are memory mapped */ > +void __iomem *__pci_ioport_map(struct pci_dev *dev, unsigned long port, > + unsigned int nr) > +{ > + return devm_ioremap_nocache(&dev->dev, port, nr); > +} This can't be in the host driver, since you can have only one instance of the function in the system, but you probably want multiple host drivers in a multiplatform kernel on ARM64. Also, the implementation is wrong since the I/O port range already needs to be ioremapped in order for inb/outb to work. There is already a generic implementation of this in include/asm-generic/iomap.h, which correctly calls ioport_map. Make sure that arm64 uses this implementation and provides an ioport_map() function like static inline void __iomem *ioport_map(unsigned long port, unsigned int nr) { return PCI_IOBASE + port; } > +/* PCIE Out/In to CSR */ > +static inline void xgene_pcie_out32(void *addr, u32 val) > +{ > + pr_debug("pcie csr wr: 0x%llx 0x%08x\n", (phys_addr_t)addr, val); > + writel(val, addr); > +} > + > +static inline void xgene_pcie_in32(void *addr, u32 *val) > +{ > + *val = readl(addr); > + pr_debug("pcie csr rd: 0x%llx 0x%08x\n", (phys_addr_t)addr, *val); > +} These add no value, just remove them. If your code is so buggy that you need to print every register access to the debug log, we don't want it ;-) > +static inline void xgene_pcie_cfg_out16(void *addr, u16 val) > +{ > + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; > + u32 val32 = readl((void *)temp_addr); > + > + switch ((phys_addr_t) addr & 0x3) { > + case 2: > + val32 &= ~0xFFFF0000; > + val32 |= (u32) val << 16; > + break; > + case 0: > + default: > + val32 &= ~0xFFFF; > + val32 |= val; > + break; > + } > + writel(val32, (void *)temp_addr); > +} Isn't there a generic version of this? If not, should there be one? Maybe Bjorn can comment. Also, all the typecasts are wrong. Please think about what types you really want and fix them. > +static void xgene_pcie_set_rtdid_reg(struct pci_bus *bus, uint devfn) > +{ > + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); > + unsigned int b, d, f; > + u32 rtdid_val = 0; > + > + b = bus->number; > + d = PCI_SLOT(devfn); > + f = PCI_FUNC(devfn); > + > + if (bus->number == port->first_busno) > + rtdid_val = (b << 24) | (d << 19) | (f << 16); > + else if (bus->number >= (port->first_busno + 1)) > + rtdid_val = (port->first_busno << 24) | > + (b << 8) | (d << 3) | f; > + > + xgene_pcie_out32(port->csr_base + RTDID, rtdid_val); > + /* read the register back to ensure flush */ > + xgene_pcie_in32(port->csr_base + RTDID, &rtdid_val); > +} What is an 'rtdid'? Maybe add some comments > +static void xgene_pcie_setup_lanes(struct xgene_pcie_port *port) > +{ > + void *csr_base = port->csr_base; > + u32 val; > + ... > +static void xgene_pcie_setup_link(struct xgene_pcie_port *port) > +{ > + void *csr_base = port->csr_base; > + u32 val; > + Don't these belong into the PHY driver? Can the setup be done in the firmware instead so we don't have to bother with it in Linux? Presumably you already need PCI support at boot time already if you want to boot from a PCI device. > +static void xgene_pcie_config_pims(void *csr_base, u32 addr, > + u64 pim, resource_size_t size) > +{ > + u32 val; > + > + xgene_pcie_out32(csr_base + addr, lower_32_bits(pim)); > + val = upper_32_bits(pim) | EN_COHERENCY; > + xgene_pcie_out32(csr_base + addr + 0x04, val); > + xgene_pcie_out32(csr_base + addr + 0x08, 0x0); > + xgene_pcie_out32(csr_base + addr + 0x0c, 0x0); > + val = lower_32_bits(size); > + xgene_pcie_out32(csr_base + addr + 0x10, val); > + val = upper_32_bits(size); > + xgene_pcie_out32(csr_base + addr + 0x14, val); > +} I suspect this is for programming the inbound translation window for DMA transactions (maybe add a comment?), and the second 64-bit word is for the bus-side address. Are you sure you want a translation starting at zero, rather than an identity-mapping like this? xgene_pcie_out32(csr_base + addr, lower_32_bits(pim)); val = upper_32_bits(pim) | EN_COHERENCY; xgene_pcie_out32(csr_base + addr + 0x04, val); xgene_pcie_out32(csr_base + addr + 0x08, pim & 0xffffffff); xgene_pcie_out32(csr_base + addr + 0x0c, pim >> 32); > +static void xgene_pcie_setup_port(struct xgene_pcie_port *port) > +{ > + int type = port->type; > + > + xgene_pcie_program_core(port->csr_base); > + if (type == PTYPE_ROOT_PORT) > + xgene_pcie_setup_root_complex(port); > + else > + xgene_pcie_setup_endpoint(port); > +} We don't really have infrastructure for PCIe endpoint devices in Linux, or in the generic DT binding for PCI hosts. We probably really want to add that in the future, but until we have decided on how to do this, please remove all code related to endpoint mode. > +struct device_node *pcibios_get_phb_of_node(struct pci_bus *bus) > +{ > + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); > + > + return of_node_get(port->node); > +} Another pointless wrapper to remove. > +static void xgene_pcie_fixup_bridge(struct pci_dev *dev) > +{ > + int i; > + > + for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) { > + dev->resource[i].start = dev->resource[i].end = 0; > + dev->resource[i].flags = 0; > + } > +} > +DECLARE_PCI_FIXUP_HEADER(XGENE_PCIE_VENDORID, XGENE_PCIE_BRIDGE_DEVICEID, > + xgene_pcie_fixup_bridge); Please add a comment to describe exactly what bug you are working around, and what devices are affected. > +/* > + * read configuration values from DTS > + */ > +static int xgene_pcie_read_dts_config(struct xgene_pcie_port *port) > +{ > + struct device_node *np = port->node; > + struct resource csr_res; > + u32 val32; > + int ret; > + const u8 *val; > + > + val = of_get_property(np, "device_type", NULL); > + if ((val != NULL) && !strcmp(val, "ep")) > + port->type = PTYPE_ENDPOINT; > + else > + port->type = PTYPE_ROOT_PORT; "ep" is not a valid device_type for all I know. > + ret = of_property_read_u32(np, "link_speed", &val32); > + if (ret == 0) > + port->link_speed = val32; > + else > + port->link_speed = PCIE_GEN3; I guess this should be an argument to the phy node. Isn't it the phy that needs to know the link speed rather than the host? > +static int xgene_pcie_alloc_ep_mem(struct xgene_pcie_port *port) > +{ > + struct xgene_pcie_ep_info *ep = &port->ep_info; > + > + ep->reg_virt = dma_alloc_coherent(port->dev, XGENE_PCIE_EP_MEM_SIZE, > + &ep->reg_phys, GFP_KERNEL); > + if (ep->reg_virt == NULL) > + return -ENOMEM; > + > + dev_info(port->dev, "EP: Virt - %p Phys - 0x%llx Size - 0x%x\n", > + ep->reg_virt, (u64) ep->reg_phys, XGENE_PCIE_EP_MEM_SIZE); > + return 0; > +} remove endpoint stuff for now. > +static int xgene_pcie_populate_inbound_regions(struct xgene_pcie_port *port) > +{ > + struct resource *msi_res = &port->res[XGENE_MSI]; > + phys_addr_t ddr_size = memblock_phys_mem_size(); > + phys_addr_t ddr_base = memblock_start_of_DRAM(); This looks fragile. What about discontiguous memory? It's probably better to leave this setup to the firmware, which already has to do it. > +static int xgene_pcie_parse_map_ranges(struct xgene_pcie_port *port) > +{ > + struct device_node *np = port->node; > + struct of_pci_range range; > + struct of_pci_range_parser parser; > + struct device *dev = port->dev; > + u32 cfg_map_done = 0; > + int ret; > + > + if (of_pci_range_parser_init(&parser, np)) { > + dev_err(dev, "missing ranges property\n"); > + return -EINVAL; > + } > + > + /* Get the I/O, memory, config ranges from DT */ > + for_each_of_pci_range(&parser, &range) { > + struct resource *res = NULL; > + u64 restype = range.flags & IORESOURCE_TYPE_BITS; > + u64 end = range.cpu_addr + range.size - 1; > + dev_dbg(port->dev, "0x%08x 0x%016llx..0x%016llx -> 0x%016llx\n", > + range.flags, range.cpu_addr, end, range.pci_addr); > + > + switch (restype) { > + case IORESOURCE_IO: > + res = &port->res[XGENE_IO]; > + of_pci_range_to_resource(&range, np, res); > + xgene_pcie_setup_ob_reg(port->csr_base, OMR1BARL, > + XGENE_IO, res); > + break; > + case IORESOURCE_MEM: > + res = &port->res[XGENE_MEM]; > + of_pci_range_to_resource(&range, np, res); > + xgene_pcie_setup_ob_reg(port->csr_base, OMR2BARL, > + XGENE_MEM, res); > + break; You also need to read out the pci_addr field from the range struct in order to set up the io_offset and mem_offset and the translation windows. Don't assume that they start at zero. > + case 0: > + if (!cfg_map_done) { > + /* config region */ > + if (port->type == PTYPE_ROOT_PORT) { > + ret = xgene_pcie_map_cfg(port, &range); > + if (ret) > + return ret; > + } > + cfg_map_done = 1; > + } else { > + /* msi region */ > + res = &port->res[XGENE_MSI]; > + of_pci_range_to_resource(&range, np, res); > + } > + break; Don't make assumptions about the order of the ranges property. Also, neither the MSI register nor the config space should be in the ranges. > +static int xgene_pcie_setup(int nr, struct pci_sys_data *sys) > +{ > + struct xgene_pcie_port *pp = xgene_pcie_sys_to_port(sys); > + > + if (pp == NULL) > + return 0; > + > + if (pp->type == PTYPE_ENDPOINT) > + return 0; > + > + sys->io_offset = pci_io_offset(pp->res[XGENE_IO].start); Normally we want io_offset to be zero, i.e. have every Bus I/O space window get translated to the same Linux I/O space address, i.e. the number you pass into pci_ioremap_io(). The code here assumes that the Bus I/O address is zero instead and the io_offset adjusts for that, which is a bit confusing. Please change it to read the actual values from the ranges property. > + sys->mem_offset = pci_io_offset(pp->res[XGENE_MEM].start); > + > + BUG_ON(request_resource(&iomem_resource, &pp->res[XGENE_IO]) || > + request_resource(&iomem_resource, &pp->res[XGENE_MEM])); > + > + pci_add_resource_offset(&sys->resources, &pp->res[XGENE_MEM], > + sys->mem_offset); > + pci_add_resource_offset(&sys->resources, &pp->res[XGENE_IO], > + sys->io_offset); &pp->res[XGENE_IO] is in memory space, while the argument you want here is in I/O space. > +static int xgene_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) > +{ > + return of_irq_parse_and_map_pci(dev, slot, pin); > +} Just use the function directly and remove the wrapper. Arnd ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver 2014-01-03 12:07 ` Arnd Bergmann @ 2014-01-07 2:41 ` Tanmay Inamdar 2014-01-07 9:27 ` Arnd Bergmann 0 siblings, 1 reply; 27+ messages in thread From: Tanmay Inamdar @ 2014-01-07 2:41 UTC (permalink / raw) To: Arnd Bergmann Cc: linux-arm-kernel@lists.infradead.org, Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters Thanks for your comments. Please see some inline replies. On Fri, Jan 3, 2014 at 4:07 AM, Arnd Bergmann <arnd@arndb.de> wrote: > On Monday 23 December 2013, Tanmay Inamdar wrote: >> This patch adds the AppliedMicro X-gene SOC PCIe controller driver. >> APM X-Gene PCIe controller supports maximum upto 8 lanes and >> GEN3 speed. X-Gene has maximum 5 PCIe ports supported. >> >> Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> >> --- >> drivers/pci/host/Kconfig | 5 + >> drivers/pci/host/Makefile | 1 + >> drivers/pci/host/pcie-xgene.c | 1017 +++++++++++++++++++++++++++++++++++++++++ >> 3 files changed, 1023 insertions(+) >> create mode 100644 drivers/pci/host/pcie-xgene.c >> >> diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig >> index 47d46c6..6d8fcbc 100644 >> --- a/drivers/pci/host/Kconfig >> +++ b/drivers/pci/host/Kconfig >> @@ -33,4 +33,9 @@ config PCI_RCAR_GEN2 >> There are 3 internal PCI controllers available with a single >> built-in EHCI/OHCI host controller present on each one. >> >> +config PCI_XGENE >> + bool "X-Gene PCIe controller" >> + depends on ARCH_XGENE >> + depends on OF > > Please add a help text here. ok > >> +#ifdef CONFIG_ARM64 >> +#include <asm/pcibios.h> >> +#else >> +#include <asm/mach/pci.h> >> +#endif > > What is !ARM64 case? Is this for PowerPC or ARM? Since you depend on > ARCH_XGENE in Kconfig I guess neither case can actually happen, > so you can remove the #ifdef. ok > >> +#define CFG_CONSTANTS_31_00 0x2000 >> +#define CFG_CONSTANTS_63_32 0x2004 >> +#define CFG_CONSTANTS_159_128 0x2010 >> +#define CFG_CONSTANTS_415_384 0x2030 > > These macros do not seem helpful. If you don't have meaningful register > names, just don't provide any and address the registers by index. ok > >> +#define ENABLE_L1S_POWER_MGMT_SET(dst, src) (((dst) & ~0x02000000) | \ >> + (((u32)(src) << 0x19) & \ >> + 0x02000000)) > > Makes this an inline function, or open-code it in the caller if there > is only one. > ok >> +#ifdef CONFIG_64BIT >> +#define pci_io_offset(s) (s & 0xff00000000) >> +#else >> +#define pci_io_offset(s) (s & 0x00000000) >> +#endif /* CONFIG_64BIT */ > > Why is this needed? The I/O space can never be over 0xffffffff, > or in practice 0xffff. My best guess is that you have a bug in the > function parsing your ranges property, or in the property value. I will recheck the logic. > >> +static inline struct xgene_pcie_port * >> +xgene_pcie_sys_to_port(struct pci_sys_data *sys) >> +{ >> + return (struct xgene_pcie_port *)sys->private_data; >> +} > > You shouldn't need the cast, or the accessor function, since private_data > is already a void pointer. got it. > >> +/* IO ports are memory mapped */ >> +void __iomem *__pci_ioport_map(struct pci_dev *dev, unsigned long port, >> + unsigned int nr) >> +{ >> + return devm_ioremap_nocache(&dev->dev, port, nr); >> +} > > This can't be in the host driver, since you can have only one instance > of the function in the system, but you probably want multiple host > drivers in a multiplatform kernel on ARM64. You are right. It will fail multiplatform kernel. > > Also, the implementation is wrong since the I/O port range already needs > to be ioremapped in order for inb/outb to work. There is already a > generic implementation of this in include/asm-generic/iomap.h, which > correctly calls ioport_map. Make sure that arm64 uses this implementation > and provides an ioport_map() function like > > static inline void __iomem *ioport_map(unsigned long port, unsigned int nr) > { > return PCI_IOBASE + port; > } For X-Gene, IO regions are memory mapped IO regions. So I am not sure if 'ioport_map' would work. > >> +/* PCIE Out/In to CSR */ >> +static inline void xgene_pcie_out32(void *addr, u32 val) >> +{ >> + pr_debug("pcie csr wr: 0x%llx 0x%08x\n", (phys_addr_t)addr, val); >> + writel(val, addr); >> +} >> + >> +static inline void xgene_pcie_in32(void *addr, u32 *val) >> +{ >> + *val = readl(addr); >> + pr_debug("pcie csr rd: 0x%llx 0x%08x\n", (phys_addr_t)addr, *val); >> +} > > These add no value, just remove them. If your code is so buggy that > you need to print every register access to the debug log, we don't > want it ;-) Yep. I will remove it. > >> +static inline void xgene_pcie_cfg_out16(void *addr, u16 val) >> +{ >> + phys_addr_t temp_addr = (phys_addr_t) addr & ~0x3; >> + u32 val32 = readl((void *)temp_addr); >> + >> + switch ((phys_addr_t) addr & 0x3) { >> + case 2: >> + val32 &= ~0xFFFF0000; >> + val32 |= (u32) val << 16; >> + break; >> + case 0: >> + default: >> + val32 &= ~0xFFFF; >> + val32 |= val; >> + break; >> + } >> + writel(val32, (void *)temp_addr); >> +} > > Isn't there a generic version of this? If not, should there be one? > Maybe Bjorn can comment. > > Also, all the typecasts are wrong. Please think about what types > you really want and fix them. ok > >> +static void xgene_pcie_set_rtdid_reg(struct pci_bus *bus, uint devfn) >> +{ >> + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); >> + unsigned int b, d, f; >> + u32 rtdid_val = 0; >> + >> + b = bus->number; >> + d = PCI_SLOT(devfn); >> + f = PCI_FUNC(devfn); >> + >> + if (bus->number == port->first_busno) >> + rtdid_val = (b << 24) | (d << 19) | (f << 16); >> + else if (bus->number >= (port->first_busno + 1)) >> + rtdid_val = (port->first_busno << 24) | >> + (b << 8) | (d << 3) | f; >> + >> + xgene_pcie_out32(port->csr_base + RTDID, rtdid_val); >> + /* read the register back to ensure flush */ >> + xgene_pcie_in32(port->csr_base + RTDID, &rtdid_val); >> +} > > What is an 'rtdid'? Maybe add some comments RTDID should be set with correct bdf to access the EP config space. I will add comments. > >> +static void xgene_pcie_setup_lanes(struct xgene_pcie_port *port) >> +{ >> + void *csr_base = port->csr_base; >> + u32 val; >> + > ... >> +static void xgene_pcie_setup_link(struct xgene_pcie_port *port) >> +{ >> + void *csr_base = port->csr_base; >> + u32 val; >> + > > Don't these belong into the PHY driver? Can the setup be done in the > firmware instead so we don't have to bother with it in Linux? > Presumably you already need PCI support at boot time already if > you want to boot from a PCI device. They do look like phy setup functions but they are part of PCIe core register space. > >> +static void xgene_pcie_config_pims(void *csr_base, u32 addr, >> + u64 pim, resource_size_t size) >> +{ >> + u32 val; >> + >> + xgene_pcie_out32(csr_base + addr, lower_32_bits(pim)); >> + val = upper_32_bits(pim) | EN_COHERENCY; >> + xgene_pcie_out32(csr_base + addr + 0x04, val); >> + xgene_pcie_out32(csr_base + addr + 0x08, 0x0); >> + xgene_pcie_out32(csr_base + addr + 0x0c, 0x0); >> + val = lower_32_bits(size); >> + xgene_pcie_out32(csr_base + addr + 0x10, val); >> + val = upper_32_bits(size); >> + xgene_pcie_out32(csr_base + addr + 0x14, val); >> +} > > I suspect this is for programming the inbound translation window for > DMA transactions (maybe add a comment?), and the second 64-bit word is > for the bus-side address. Are you sure you want a translation starting > at zero, rather than an identity-mapping like this? Actually it is an unused sub-region. I will remove setting to 0. It defaults to 0 anyways. > > xgene_pcie_out32(csr_base + addr, lower_32_bits(pim)); > val = upper_32_bits(pim) | EN_COHERENCY; > xgene_pcie_out32(csr_base + addr + 0x04, val); > xgene_pcie_out32(csr_base + addr + 0x08, pim & 0xffffffff); > xgene_pcie_out32(csr_base + addr + 0x0c, pim >> 32); > >> +static void xgene_pcie_setup_port(struct xgene_pcie_port *port) >> +{ >> + int type = port->type; >> + >> + xgene_pcie_program_core(port->csr_base); >> + if (type == PTYPE_ROOT_PORT) >> + xgene_pcie_setup_root_complex(port); >> + else >> + xgene_pcie_setup_endpoint(port); >> +} > > We don't really have infrastructure for PCIe endpoint devices in Linux, > or in the generic DT binding for PCI hosts. We probably really want to > add that in the future, but until we have decided on how to do this, > please remove all code related to endpoint mode. ok. > >> +struct device_node *pcibios_get_phb_of_node(struct pci_bus *bus) >> +{ >> + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); >> + >> + return of_node_get(port->node); >> +} > > Another pointless wrapper to remove. If I remove this, then we get a failure while parsing irqs "pci 0000:00:00.0: of_irq_parse_pci() failed with rc=-22" > >> +static void xgene_pcie_fixup_bridge(struct pci_dev *dev) >> +{ >> + int i; >> + >> + for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) { >> + dev->resource[i].start = dev->resource[i].end = 0; >> + dev->resource[i].flags = 0; >> + } >> +} >> +DECLARE_PCI_FIXUP_HEADER(XGENE_PCIE_VENDORID, XGENE_PCIE_BRIDGE_DEVICEID, >> + xgene_pcie_fixup_bridge); > > Please add a comment to describe exactly what bug you are working around, > and what devices are affected. ok > >> +/* >> + * read configuration values from DTS >> + */ >> +static int xgene_pcie_read_dts_config(struct xgene_pcie_port *port) >> +{ >> + struct device_node *np = port->node; >> + struct resource csr_res; >> + u32 val32; >> + int ret; >> + const u8 *val; >> + >> + val = of_get_property(np, "device_type", NULL); >> + if ((val != NULL) && !strcmp(val, "ep")) >> + port->type = PTYPE_ENDPOINT; >> + else >> + port->type = PTYPE_ROOT_PORT; > > "ep" is not a valid device_type for all I know. Will remove all EP stuff. > >> + ret = of_property_read_u32(np, "link_speed", &val32); >> + if (ret == 0) >> + port->link_speed = val32; >> + else >> + port->link_speed = PCIE_GEN3; > > I guess this should be an argument to the phy node. Isn't it the phy > that needs to know the link speed rather than the host? yes. some part of it still resides in the core. However I will make it to GEN3 by default. > >> +static int xgene_pcie_alloc_ep_mem(struct xgene_pcie_port *port) >> +{ >> + struct xgene_pcie_ep_info *ep = &port->ep_info; >> + >> + ep->reg_virt = dma_alloc_coherent(port->dev, XGENE_PCIE_EP_MEM_SIZE, >> + &ep->reg_phys, GFP_KERNEL); >> + if (ep->reg_virt == NULL) >> + return -ENOMEM; >> + >> + dev_info(port->dev, "EP: Virt - %p Phys - 0x%llx Size - 0x%x\n", >> + ep->reg_virt, (u64) ep->reg_phys, XGENE_PCIE_EP_MEM_SIZE); >> + return 0; >> +} > > remove endpoint stuff for now. ok > >> +static int xgene_pcie_populate_inbound_regions(struct xgene_pcie_port *port) >> +{ >> + struct resource *msi_res = &port->res[XGENE_MSI]; >> + phys_addr_t ddr_size = memblock_phys_mem_size(); >> + phys_addr_t ddr_base = memblock_start_of_DRAM(); > > This looks fragile. What about discontiguous memory? It's probably better to > leave this setup to the firmware, which already has to do it. Idea is to map whole RAM. The memory controller in X-Gene does not allow holes or discontinuity in RAM. > >> +static int xgene_pcie_parse_map_ranges(struct xgene_pcie_port *port) >> +{ >> + struct device_node *np = port->node; >> + struct of_pci_range range; >> + struct of_pci_range_parser parser; >> + struct device *dev = port->dev; >> + u32 cfg_map_done = 0; >> + int ret; >> + >> + if (of_pci_range_parser_init(&parser, np)) { >> + dev_err(dev, "missing ranges property\n"); >> + return -EINVAL; >> + } >> + >> + /* Get the I/O, memory, config ranges from DT */ >> + for_each_of_pci_range(&parser, &range) { >> + struct resource *res = NULL; >> + u64 restype = range.flags & IORESOURCE_TYPE_BITS; >> + u64 end = range.cpu_addr + range.size - 1; >> + dev_dbg(port->dev, "0x%08x 0x%016llx..0x%016llx -> 0x%016llx\n", >> + range.flags, range.cpu_addr, end, range.pci_addr); >> + >> + switch (restype) { >> + case IORESOURCE_IO: >> + res = &port->res[XGENE_IO]; >> + of_pci_range_to_resource(&range, np, res); >> + xgene_pcie_setup_ob_reg(port->csr_base, OMR1BARL, >> + XGENE_IO, res); >> + break; >> + case IORESOURCE_MEM: >> + res = &port->res[XGENE_MEM]; >> + of_pci_range_to_resource(&range, np, res); >> + xgene_pcie_setup_ob_reg(port->csr_base, OMR2BARL, >> + XGENE_MEM, res); >> + break; > > You also need to read out the pci_addr field from the range struct in order > to set up the io_offset and mem_offset and the translation windows. > Don't assume that they start at zero. ok. > >> + case 0: >> + if (!cfg_map_done) { >> + /* config region */ >> + if (port->type == PTYPE_ROOT_PORT) { >> + ret = xgene_pcie_map_cfg(port, &range); >> + if (ret) >> + return ret; >> + } >> + cfg_map_done = 1; >> + } else { >> + /* msi region */ >> + res = &port->res[XGENE_MSI]; >> + of_pci_range_to_resource(&range, np, res); >> + } >> + break; > > Don't make assumptions about the order of the ranges property. Also, neither > the MSI register nor the config space should be in the ranges. ok. > >> +static int xgene_pcie_setup(int nr, struct pci_sys_data *sys) >> +{ >> + struct xgene_pcie_port *pp = xgene_pcie_sys_to_port(sys); >> + >> + if (pp == NULL) >> + return 0; >> + >> + if (pp->type == PTYPE_ENDPOINT) >> + return 0; >> + >> + sys->io_offset = pci_io_offset(pp->res[XGENE_IO].start); > > Normally we want io_offset to be zero, i.e. have every Bus I/O space > window get translated to the same Linux I/O space address, i.e. > the number you pass into pci_ioremap_io(). The code here assumes > that the Bus I/O address is zero instead and the io_offset adjusts > for that, which is a bit confusing. Please change it to read > the actual values from the ranges property. I will recheck the logic. > >> + sys->mem_offset = pci_io_offset(pp->res[XGENE_MEM].start); >> + >> + BUG_ON(request_resource(&iomem_resource, &pp->res[XGENE_IO]) || >> + request_resource(&iomem_resource, &pp->res[XGENE_MEM])); >> + >> + pci_add_resource_offset(&sys->resources, &pp->res[XGENE_MEM], >> + sys->mem_offset); >> + pci_add_resource_offset(&sys->resources, &pp->res[XGENE_IO], >> + sys->io_offset); > > &pp->res[XGENE_IO] is in memory space, while the argument you want here > is in I/O space. > >> +static int xgene_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) >> +{ >> + return of_irq_parse_and_map_pci(dev, slot, pin); >> +} > > Just use the function directly and remove the wrapper. got it. > > Arnd ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver 2014-01-07 2:41 ` Tanmay Inamdar @ 2014-01-07 9:27 ` Arnd Bergmann 2014-01-10 1:20 ` Tanmay Inamdar 0 siblings, 1 reply; 27+ messages in thread From: Arnd Bergmann @ 2014-01-07 9:27 UTC (permalink / raw) To: Tanmay Inamdar Cc: linux-arm-kernel@lists.infradead.org, Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters On Tuesday 07 January 2014, Tanmay Inamdar wrote: > > Also, the implementation is wrong since the I/O port range already needs > > to be ioremapped in order for inb/outb to work. There is already a > > generic implementation of this in include/asm-generic/iomap.h, which > > correctly calls ioport_map. Make sure that arm64 uses this implementation > > and provides an ioport_map() function like > > > > static inline void __iomem *ioport_map(unsigned long port, unsigned int nr) > > { > > return PCI_IOBASE + port; > > } > > For X-Gene, IO regions are memory mapped IO regions. So I am not sure > if 'ioport_map' > would work. It should. In fact all ARM and ARM64 platforms I have seen (and most powerpc ones) have their IO region memory mapped. The way we handle this in Linux is to map the IO space to a fixed virtual address at the time the host controller is initialized, and all accesses to an IO port translate to a access in this virtual address. See the inb()/outb() implementation on arm and arm64, as well as the arm pci_ioremap_io() function for more details. > >> +static void xgene_pcie_setup_lanes(struct xgene_pcie_port *port) > >> +{ > >> + void *csr_base = port->csr_base; > >> + u32 val; > >> + > > ... > >> +static void xgene_pcie_setup_link(struct xgene_pcie_port *port) > >> +{ > >> + void *csr_base = port->csr_base; > >> + u32 val; > >> + > > > > Don't these belong into the PHY driver? Can the setup be done in the > > firmware instead so we don't have to bother with it in Linux? > > Presumably you already need PCI support at boot time already if > > you want to boot from a PCI device. > > They do look like phy setup functions but they are part of PCIe core > register space. Ok. > >> +static void xgene_pcie_config_pims(void *csr_base, u32 addr, > >> + u64 pim, resource_size_t size) > >> +{ > >> + u32 val; > >> + > >> + xgene_pcie_out32(csr_base + addr, lower_32_bits(pim)); > >> + val = upper_32_bits(pim) | EN_COHERENCY; > >> + xgene_pcie_out32(csr_base + addr + 0x04, val); > >> + xgene_pcie_out32(csr_base + addr + 0x08, 0x0); > >> + xgene_pcie_out32(csr_base + addr + 0x0c, 0x0); > >> + val = lower_32_bits(size); > >> + xgene_pcie_out32(csr_base + addr + 0x10, val); > >> + val = upper_32_bits(size); > >> + xgene_pcie_out32(csr_base + addr + 0x14, val); > >> +} > > > > I suspect this is for programming the inbound translation window for > > DMA transactions (maybe add a comment?), and the second 64-bit word is > > for the bus-side address. Are you sure you want a translation starting > > at zero, rather than an identity-mapping like this? > > Actually it is an unused sub-region. I will remove setting to 0. It > defaults to 0 anyways. Is it always an identity-mapping then? > >> +struct device_node *pcibios_get_phb_of_node(struct pci_bus *bus) > >> +{ > >> + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); > >> + > >> + return of_node_get(port->node); > >> +} > > > > Another pointless wrapper to remove. > > If I remove this, then we get a failure while parsing irqs > "pci 0000:00:00.0: of_irq_parse_pci() failed with rc=-22" I mean it would be just as easy to open-code the function in the callers, and more readable. > >> +static int xgene_pcie_populate_inbound_regions(struct xgene_pcie_port *port) > >> +{ > >> + struct resource *msi_res = &port->res[XGENE_MSI]; > >> + phys_addr_t ddr_size = memblock_phys_mem_size(); > >> + phys_addr_t ddr_base = memblock_start_of_DRAM(); > > > > This looks fragile. What about discontiguous memory? It's probably better to > > leave this setup to the firmware, which already has to do it. > > Idea is to map whole RAM. The memory controller in X-Gene does not > allow holes or discontinuity in RAM. There might be holes in the memory map for other reasons, e.g. some part of memory could be reserved for use by a particular piece of software. There is actually a definition for a "dma-ranges" property that is normally use to communicate this information, i.e. which bus addresses for DMA translate into which parent bus (or memory) addresses. I think it would be more logical to use that property. Arnd ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver 2014-01-07 9:27 ` Arnd Bergmann @ 2014-01-10 1:20 ` Tanmay Inamdar 0 siblings, 0 replies; 27+ messages in thread From: Tanmay Inamdar @ 2014-01-10 1:20 UTC (permalink / raw) To: Arnd Bergmann Cc: linux-arm-kernel@lists.infradead.org, Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters On Tue, Jan 7, 2014 at 1:27 AM, Arnd Bergmann <arnd@arndb.de> wrote: > On Tuesday 07 January 2014, Tanmay Inamdar wrote: >> > Also, the implementation is wrong since the I/O port range already needs >> > to be ioremapped in order for inb/outb to work. There is already a >> > generic implementation of this in include/asm-generic/iomap.h, which >> > correctly calls ioport_map. Make sure that arm64 uses this implementation >> > and provides an ioport_map() function like >> > >> > static inline void __iomem *ioport_map(unsigned long port, unsigned int nr) >> > { >> > return PCI_IOBASE + port; >> > } >> >> For X-Gene, IO regions are memory mapped IO regions. So I am not sure >> if 'ioport_map' >> would work. > > It should. In fact all ARM and ARM64 platforms I have seen (and most powerpc > ones) have their IO region memory mapped. The way we handle this in Linux > is to map the IO space to a fixed virtual address at the time the host > controller is initialized, and all accesses to an IO port translate to > a access in this virtual address. See the inb()/outb() implementation on > arm and arm64, as well as the arm pci_ioremap_io() function for more > details. > Yes. arm64 has to support pci_ioremap_io and pci_iomap in order to support IO regions. Thanks. >> >> +static void xgene_pcie_setup_lanes(struct xgene_pcie_port *port) >> >> +{ >> >> + void *csr_base = port->csr_base; >> >> + u32 val; >> >> + >> > ... >> >> +static void xgene_pcie_setup_link(struct xgene_pcie_port *port) >> >> +{ >> >> + void *csr_base = port->csr_base; >> >> + u32 val; >> >> + >> > >> > Don't these belong into the PHY driver? Can the setup be done in the >> > firmware instead so we don't have to bother with it in Linux? >> > Presumably you already need PCI support at boot time already if >> > you want to boot from a PCI device. >> >> They do look like phy setup functions but they are part of PCIe core >> register space. > > Ok. > >> >> +static void xgene_pcie_config_pims(void *csr_base, u32 addr, >> >> + u64 pim, resource_size_t size) >> >> +{ >> >> + u32 val; >> >> + >> >> + xgene_pcie_out32(csr_base + addr, lower_32_bits(pim)); >> >> + val = upper_32_bits(pim) | EN_COHERENCY; >> >> + xgene_pcie_out32(csr_base + addr + 0x04, val); >> >> + xgene_pcie_out32(csr_base + addr + 0x08, 0x0); >> >> + xgene_pcie_out32(csr_base + addr + 0x0c, 0x0); >> >> + val = lower_32_bits(size); >> >> + xgene_pcie_out32(csr_base + addr + 0x10, val); >> >> + val = upper_32_bits(size); >> >> + xgene_pcie_out32(csr_base + addr + 0x14, val); >> >> +} >> > >> > I suspect this is for programming the inbound translation window for >> > DMA transactions (maybe add a comment?), and the second 64-bit word is >> > for the bus-side address. Are you sure you want a translation starting >> > at zero, rather than an identity-mapping like this? >> >> Actually it is an unused sub-region. I will remove setting to 0. It >> defaults to 0 anyways. > > Is it always an identity-mapping then? > Yes. >> >> +struct device_node *pcibios_get_phb_of_node(struct pci_bus *bus) >> >> +{ >> >> + struct xgene_pcie_port *port = xgene_pcie_bus_to_port(bus); >> >> + >> >> + return of_node_get(port->node); >> >> +} >> > >> > Another pointless wrapper to remove. >> >> If I remove this, then we get a failure while parsing irqs >> "pci 0000:00:00.0: of_irq_parse_pci() failed with rc=-22" > > I mean it would be just as easy to open-code the function in the > callers, and more readable. > >> >> +static int xgene_pcie_populate_inbound_regions(struct xgene_pcie_port *port) >> >> +{ >> >> + struct resource *msi_res = &port->res[XGENE_MSI]; >> >> + phys_addr_t ddr_size = memblock_phys_mem_size(); >> >> + phys_addr_t ddr_base = memblock_start_of_DRAM(); >> > >> > This looks fragile. What about discontiguous memory? It's probably better to >> > leave this setup to the firmware, which already has to do it. >> >> Idea is to map whole RAM. The memory controller in X-Gene does not >> allow holes or discontinuity in RAM. > > There might be holes in the memory map for other reasons, e.g. some part of > memory could be reserved for use by a particular piece of software. > There is actually a definition for a "dma-ranges" property that is normally > use to communicate this information, i.e. which bus addresses for DMA > translate into which parent bus (or memory) addresses. I think it would > be more logical to use that property. Yes. You are right. We will get more flexibility with this. > > Arnd ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver 2013-12-23 8:02 ` [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver Tanmay Inamdar 2014-01-02 21:08 ` Bjorn Helgaas 2014-01-03 12:07 ` Arnd Bergmann @ 2014-01-06 1:47 ` Jingoo Han 2014-01-07 2:45 ` Tanmay Inamdar 2 siblings, 1 reply; 27+ messages in thread From: Jingoo Han @ 2014-01-06 1:47 UTC (permalink / raw) To: 'Tanmay Inamdar' Cc: 'Bjorn Helgaas', 'Grant Likely', 'Catalin Marinas', 'Rob Landley', linux-pci, devicetree, linux-arm-kernel, linux-doc, linux-kernel, patches, 'Jon Masters', 'Jason Gunthorpe', 'Arnd Bergmann', 'Jingoo Han' On Monday, December 23, 2013 5:02 PM, Tanmay Inamdar wrote: > > This patch adds the AppliedMicro X-gene SOC PCIe controller driver. > APM X-Gene PCIe controller supports maximum upto 8 lanes and > GEN3 speed. X-Gene has maximum 5 PCIe ports supported. (+cc Jason Gunthorpe, Arnd Bergmann) Hi Tanmay Inamdar, I added some minor comments. :-) > > Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> > --- > drivers/pci/host/Kconfig | 5 + > drivers/pci/host/Makefile | 1 + > drivers/pci/host/pcie-xgene.c | 1017 +++++++++++++++++++++++++++++++++++++++++ Would you change the file name to 'pci-xgene.c'? Now, all PCI host drivers are using the prefix 'pci-', not 'pcie-'. [.....] > +#include <linux/module.h> > +#include <linux/delay.h> > +#include <linux/pci.h> > +#include <linux/slab.h> > +#include <linux/memblock.h> > +#include <linux/io.h> > +#include <linux/platform_device.h> > +#include <linux/of.h> > +#include <linux/of_address.h> > +#include <linux/of_irq.h> > +#include <linux/of_pci.h> > +#include <linux/jiffies.h> > +#include <linux/clk-private.h> Would you re-order these headers alphabetically? It enhances the readability. [.....] > +static int xgene_pcie_parse_map_ranges(struct xgene_pcie_port *port) > +{ > + struct device_node *np = port->node; > + struct of_pci_range range; > + struct of_pci_range_parser parser; > + struct device *dev = port->dev; > + u32 cfg_map_done = 0; > + int ret; > + > + if (of_pci_range_parser_init(&parser, np)) { > + dev_err(dev, "missing ranges property\n"); > + return -EINVAL; > + } > + > + /* Get the I/O, memory, config ranges from DT */ > + for_each_of_pci_range(&parser, &range) { > + struct resource *res = NULL; > + u64 restype = range.flags & IORESOURCE_TYPE_BITS; > + u64 end = range.cpu_addr + range.size - 1; > + dev_dbg(port->dev, "0x%08x 0x%016llx..0x%016llx -> 0x%016llx\n", > + range.flags, range.cpu_addr, end, range.pci_addr); > + > + switch (restype) { > + case IORESOURCE_IO: > + res = &port->res[XGENE_IO]; > + of_pci_range_to_resource(&range, np, res); > + xgene_pcie_setup_ob_reg(port->csr_base, OMR1BARL, > + XGENE_IO, res); > + break; > + case IORESOURCE_MEM: > + res = &port->res[XGENE_MEM]; > + of_pci_range_to_resource(&range, np, res); > + xgene_pcie_setup_ob_reg(port->csr_base, OMR2BARL, > + XGENE_MEM, res); > + break; > + case 0: > + if (!cfg_map_done) { > + /* config region */ > + if (port->type == PTYPE_ROOT_PORT) { > + ret = xgene_pcie_map_cfg(port, &range); > + if (ret) > + return ret; > + } > + cfg_map_done = 1; > + } else { > + /* msi region */ > + res = &port->res[XGENE_MSI]; > + of_pci_range_to_resource(&range, np, res); As Jason Gunthorpe said, the DT 'range' property should not handle MSI. Please refer to other PCI host drivers such as pci-mvebu.c, pci-tegra.c and pcie-designware.c. Currently, 'struct msi_chip', ' struct irq_domain' are used for implementing MSI feature. Best regards, Jingoo Han ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver 2014-01-06 1:47 ` Jingoo Han @ 2014-01-07 2:45 ` Tanmay Inamdar 2014-01-07 3:31 ` Jingoo Han 0 siblings, 1 reply; 27+ messages in thread From: Tanmay Inamdar @ 2014-01-07 2:45 UTC (permalink / raw) To: Jingoo Han Cc: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, linux-pci, devicetree, linux-arm-kernel@lists.infradead.org, linux-doc, linux-kernel@vger.kernel.org, patches, Jon Masters, Jason Gunthorpe, Arnd Bergmann On Sun, Jan 5, 2014 at 5:47 PM, Jingoo Han <jg1.han@samsung.com> wrote: > On Monday, December 23, 2013 5:02 PM, Tanmay Inamdar wrote: >> >> This patch adds the AppliedMicro X-gene SOC PCIe controller driver. >> APM X-Gene PCIe controller supports maximum upto 8 lanes and >> GEN3 speed. X-Gene has maximum 5 PCIe ports supported. > > (+cc Jason Gunthorpe, Arnd Bergmann) > > Hi Tanmay Inamdar, > > I added some minor comments. :-) > >> >> Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> >> --- >> drivers/pci/host/Kconfig | 5 + >> drivers/pci/host/Makefile | 1 + >> drivers/pci/host/pcie-xgene.c | 1017 +++++++++++++++++++++++++++++++++++++++++ > > Would you change the file name to 'pci-xgene.c'? > Now, all PCI host drivers are using the prefix 'pci-', not 'pcie-'. I guess designware is an exception. There is "drivers/pci/host/pcie-designware.c" > > [.....] > >> +#include <linux/module.h> >> +#include <linux/delay.h> >> +#include <linux/pci.h> >> +#include <linux/slab.h> >> +#include <linux/memblock.h> >> +#include <linux/io.h> >> +#include <linux/platform_device.h> >> +#include <linux/of.h> >> +#include <linux/of_address.h> >> +#include <linux/of_irq.h> >> +#include <linux/of_pci.h> >> +#include <linux/jiffies.h> >> +#include <linux/clk-private.h> > > Would you re-order these headers alphabetically? > It enhances the readability. ok. > > [.....] > >> +static int xgene_pcie_parse_map_ranges(struct xgene_pcie_port *port) >> +{ >> + struct device_node *np = port->node; >> + struct of_pci_range range; >> + struct of_pci_range_parser parser; >> + struct device *dev = port->dev; >> + u32 cfg_map_done = 0; >> + int ret; >> + >> + if (of_pci_range_parser_init(&parser, np)) { >> + dev_err(dev, "missing ranges property\n"); >> + return -EINVAL; >> + } >> + >> + /* Get the I/O, memory, config ranges from DT */ >> + for_each_of_pci_range(&parser, &range) { >> + struct resource *res = NULL; >> + u64 restype = range.flags & IORESOURCE_TYPE_BITS; >> + u64 end = range.cpu_addr + range.size - 1; >> + dev_dbg(port->dev, "0x%08x 0x%016llx..0x%016llx -> 0x%016llx\n", >> + range.flags, range.cpu_addr, end, range.pci_addr); >> + >> + switch (restype) { >> + case IORESOURCE_IO: >> + res = &port->res[XGENE_IO]; >> + of_pci_range_to_resource(&range, np, res); >> + xgene_pcie_setup_ob_reg(port->csr_base, OMR1BARL, >> + XGENE_IO, res); >> + break; >> + case IORESOURCE_MEM: >> + res = &port->res[XGENE_MEM]; >> + of_pci_range_to_resource(&range, np, res); >> + xgene_pcie_setup_ob_reg(port->csr_base, OMR2BARL, >> + XGENE_MEM, res); >> + break; >> + case 0: >> + if (!cfg_map_done) { >> + /* config region */ >> + if (port->type == PTYPE_ROOT_PORT) { >> + ret = xgene_pcie_map_cfg(port, &range); >> + if (ret) >> + return ret; >> + } >> + cfg_map_done = 1; >> + } else { >> + /* msi region */ >> + res = &port->res[XGENE_MSI]; >> + of_pci_range_to_resource(&range, np, res); > > As Jason Gunthorpe said, the DT 'range' property should not > handle MSI. Please refer to other PCI host drivers such as > pci-mvebu.c, pci-tegra.c and pcie-designware.c. Right. I will remove MSI from ranges. > > Currently, 'struct msi_chip', ' struct irq_domain' are used > for implementing MSI feature. > > Best regards, > Jingoo Han > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver 2014-01-07 2:45 ` Tanmay Inamdar @ 2014-01-07 3:31 ` Jingoo Han 0 siblings, 0 replies; 27+ messages in thread From: Jingoo Han @ 2014-01-07 3:31 UTC (permalink / raw) To: 'Tanmay Inamdar' Cc: 'Bjorn Helgaas', 'Grant Likely', 'Catalin Marinas', 'Rob Landley', linux-pci, devicetree, linux-arm-kernel, linux-doc, linux-kernel, 'patches', 'Jon Masters', 'Jason Gunthorpe', 'Arnd Bergmann', 'Jingoo Han', 'Thierry Reding', 'Pratyush Anand', 'Mohit KUMAR' On Tuesday, January 07, 2014 11:45 AM, Tanmay Inamdar wrote: > On Sun, Jan 5, 2014 at 5:47 PM, Jingoo Han <jg1.han@samsung.com> wrote: > > On Monday, December 23, 2013 5:02 PM, Tanmay Inamdar wrote: > >> > >> This patch adds the AppliedMicro X-gene SOC PCIe controller driver. > >> APM X-Gene PCIe controller supports maximum upto 8 lanes and > >> GEN3 speed. X-Gene has maximum 5 PCIe ports supported. > > > > (+cc Jason Gunthorpe, Arnd Bergmann) > > > > Hi Tanmay Inamdar, > > > > I added some minor comments. :-) > > > >> > >> Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> > >> --- > >> drivers/pci/host/Kconfig | 5 + > >> drivers/pci/host/Makefile | 1 + > >> drivers/pci/host/pcie-xgene.c | 1017 +++++++++++++++++++++++++++++++++++++++++ > > > > Would you change the file name to 'pci-xgene.c'? > > Now, all PCI host drivers are using the prefix 'pci-', not 'pcie-'. > > I guess designware is an exception. There is > "drivers/pci/host/pcie-designware.c" (+cc Thierry Reding, Pratyush Anand, Mohit KUMAR) Now, the current naming rule is "PCI-" prefix as below. - Samsung Exynos: "pci"-exynos.c - Freescale i.MX6: "pci"-imx6.c - Marvell: pci-mvebu.c - Nvidia Tegra: pci-tegra.c - Renesas R-Car: pci-rcar-gen2.c According to the Thierry Reding's comment, "I think we should keep these sorted alphabetically. Also Tegra and Marvell are PCIe controllers but they still use the pci- prefix instead of pcie-. Perhaps it'd be good to keep consistency here? I initially chose pci- because from a software point of view it doesn't matter all that much whether it's PCI or PCIe and because the drivers are part of the PCI subsystem. However if Exynos now uses the pcie- prefix it makes it look like Tegra and Marvell are plain old PCI." (https://groups.google.com/forum/#!msg/linux.kernel/qtimJoNSc3w/_1aayHaG54YJ) However, "pcie-designware.c" is common layer driver for other SoC PCI host drivers that use Synopsys Designware PCI IP. Thus, currently it is shared by other SoC PCI host drivers such as pci-exynos.c, and pci-imx6.c. Also, ST PCI driver will use pcie-designware.c as common layer. Originally, "pci"-designware.c was used. However, Pratyush Anand suggested "pci"-designware.c can be renamed to "pcie"-designware.c, because Synopsys PCI IP and Synopsys PCI Express IP are different. So, currently "pcie"-designware.c is used. So, if there is no special reason, "pci-" prefix can be used. Thank you. Best regards, Jingoo Han ^ permalink raw reply [flat|nested] 27+ messages in thread
* [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes 2013-12-23 8:02 [RFC PATCH 0/3] APM X-Gene PCIe driver Tanmay Inamdar 2013-12-23 8:02 ` [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver Tanmay Inamdar @ 2013-12-23 8:02 ` Tanmay Inamdar 2013-12-23 17:46 ` Jason Gunthorpe 2013-12-23 8:02 ` [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings Tanmay Inamdar 2013-12-23 8:56 ` [RFC PATCH 0/3] APM X-Gene PCIe driver Tanmay Inamdar 3 siblings, 1 reply; 27+ messages in thread From: Tanmay Inamdar @ 2013-12-23 8:02 UTC (permalink / raw) To: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley Cc: linux-pci, devicetree, linux-arm-kernel, linux-doc, linux-kernel, patches, jcm, Tanmay Inamdar This patch adds the device tree nodes for APM X-Gene PCIe controller and PCIe clock interface. Since X-Gene SOC supports maximum 5 ports, 5 dts nodes are added. Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> --- arch/arm64/boot/dts/apm-mustang.dts | 4 + arch/arm64/boot/dts/apm-storm.dtsi | 140 +++++++++++++++++++++++++++++++++++ 2 files changed, 144 insertions(+) diff --git a/arch/arm64/boot/dts/apm-mustang.dts b/arch/arm64/boot/dts/apm-mustang.dts index 1247ca1..ab2b95f 100644 --- a/arch/arm64/boot/dts/apm-mustang.dts +++ b/arch/arm64/boot/dts/apm-mustang.dts @@ -24,3 +24,7 @@ reg = < 0x1 0x00000000 0x0 0x80000000 >; /* Updated by bootloader */ }; }; + +&pcie0 { + status = "ok"; +}; diff --git a/arch/arm64/boot/dts/apm-storm.dtsi b/arch/arm64/boot/dts/apm-storm.dtsi index d37d736..b82f430 100644 --- a/arch/arm64/boot/dts/apm-storm.dtsi +++ b/arch/arm64/boot/dts/apm-storm.dtsi @@ -176,6 +176,146 @@ reg-names = "csr-reg"; clock-output-names = "eth8clk"; }; + + pcie0clk: pcie0clk@1f2bc000 { + compatible = "apm,xgene-device-clock"; + #clock-cells = <1>; + clocks = <&socplldiv2 0>; + clock-names = "pcie0clk"; + reg = <0x0 0x1f2bc000 0x0 0x1000>; + reg-names = "csr-reg"; + clock-output-names = "pcie0clk"; + }; + + pcie1clk: pcie1clk@1f2cc000 { + compatible = "apm,xgene-device-clock"; + #clock-cells = <1>; + clocks = <&socplldiv2 0>; + clock-names = "pcie1clk"; + reg = <0x0 0x1f2cc000 0x0 0x1000>; + reg-names = "csr-reg"; + clock-output-names = "pcie1clk"; + }; + + pcie2clk: pcie2clk@1f2dc000 { + compatible = "apm,xgene-device-clock"; + #clock-cells = <1>; + clocks = <&socplldiv2 0>; + clock-names = "pcie2clk"; + reg = <0x0 0x1f2dc000 0x0 0x1000>; + reg-names = "csr-reg"; + clock-output-names = "pcie2clk"; + }; + + pcie3clk: pcie3clk@1f50c000 { + compatible = "apm,xgene-device-clock"; + #clock-cells = <1>; + clocks = <&socplldiv2 0>; + clock-names = "pcie3clk"; + reg = <0x0 0x1f50c000 0x0 0x1000>; + reg-names = "csr-reg"; + clock-output-names = "pcie3clk"; + }; + + pcie4clk: pcie4clk@1f51c000 { + compatible = "apm,xgene-device-clock"; + #clock-cells = <1>; + clocks = <&socplldiv2 0>; + clock-names = "pcie4clk"; + reg = <0x0 0x1f51c000 0x0 0x1000>; + reg-names = "csr-reg"; + clock-output-names = "pcie4clk"; + }; + }; + + pcie0: pcie@1f2b0000 { + status = "disabled"; + device_type = "pci"; + compatible = "apm,xgene-pcie"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + reg = < 0x00 0x1f2b0000 0x0 0x00010000>; + ranges = <0x02000000 0x0 0x00000000 0xe0 0x00000000 0x0 0x10000000 /* mem*/ + 0x01000000 0x0 0x80000000 0xe0 0x80000000 0x0 0x00010000 /* io */ + 0x00000000 0x0 0xd0000000 0xe0 0xd0000000 0x0 0x00200000 /* cfg */ + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xc2 0x1>; + clocks = <&pcie0clk 0>; + clock-names = "pcieclk"; + }; + + pcie1: pcie@1f2c0000 { + status = "disabled"; + device_type = "pci"; + compatible = "apm,xgene-pcie"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + reg = <0x00 0x1f2c0000 0x0 0x00010000>; + ranges = <0x02000000 0x0 0x00000000 0xd0 0x00000000 0x0 0x10000000 /* mem*/ + 0x01000000 0x0 0x80000000 0xd0 0x80000000 0x0 0x00010000 /* io */ + 0x00000000 0x0 0xd0000000 0xd0 0xd0000000 0x0 0x00200000 /* cfg */ + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xc8 0x1>; + clocks = <&pcie1clk 0>; + clock-names = "pcieclk"; + }; + + pcie2: pcie@1f2d0000 { + status = "disabled"; + device_type = "pci"; + compatible = "apm,xgene-pcie"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + reg = <0x00 0x1f2d0000 0x0 0x00010000>; + ranges = <0x02000000 0x0 0x00000000 0x90 0x00000000 0x0 0x10000000 /* mem*/ + 0x01000000 0x0 0x80000000 0x90 0x80000000 0x0 0x00010000 /* io */ + 0x00000000 0x0 0xd0000000 0x90 0xd0000000 0x0 0x00200000 /* cfg */ + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xce 0x1>; + clocks = <&pcie3clk 0>; + clock-names = "pcieclk"; + }; + + pcie3: pcie@1f500000 { + status = "disabled"; + device_type = "pci"; + compatible = "apm,xgene-pcie"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + reg = <0x00 0x1f500000 0x0 0x00010000>; + ranges = <0x02000000 0x0 0x00000000 0xa0 0x00000000 0x0 0x10000000 /* mem*/ + 0x01000000 0x0 0x80000000 0xa0 0x80000000 0x0 0x00010000 /* io */ + 0x00000000 0x0 0xd0000000 0xa0 0xd0000000 0x0 0x00200000 /* cfg */ + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xd4 0x1>; + clocks = <&pcie3clk 0>; + clock-names = "pcieclk"; + }; + + pcie4: pcie@1f510000 { + status = "disabled"; + device_type = "pci"; + compatible = "apm,xgene-pcie"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + reg = <0x00 0x1f510000 0x0 0x00010000>; + ranges = <0x02000000 0x0 0x00000000 0xc0 0x00000000 0x0 0x10000000 /* mem*/ + 0x01000000 0x0 0x80000000 0xc0 0x80000000 0x0 0x00010000 /* io */ + 0x00000000 0x0 0xd0000000 0xc0 0xd0000000 0x0 0x00200000 /* cfg */ + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xda 0x1>; + clocks = <&pcie4clk 0>; + clock-names = "pcieclk"; }; serial0: serial@1c020000 { -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes 2013-12-23 8:02 ` [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes Tanmay Inamdar @ 2013-12-23 17:46 ` Jason Gunthorpe 2014-01-02 21:56 ` Tanmay Inamdar 0 siblings, 1 reply; 27+ messages in thread From: Jason Gunthorpe @ 2013-12-23 17:46 UTC (permalink / raw) To: Tanmay Inamdar Cc: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel, jcm, linux-arm-kernel On Mon, Dec 23, 2013 at 01:32:03PM +0530, Tanmay Inamdar wrote: > This patch adds the device tree nodes for APM X-Gene PCIe controller and > PCIe clock interface. Since X-Gene SOC supports maximum 5 ports, 5 dts nodes > are added. Can you include an lspci dump for PCI DT bindings please? It is impossible to review otherwise.. Regards, Jason ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes 2013-12-23 17:46 ` Jason Gunthorpe @ 2014-01-02 21:56 ` Tanmay Inamdar 2014-01-03 0:52 ` Jason Gunthorpe 0 siblings, 1 reply; 27+ messages in thread From: Tanmay Inamdar @ 2014-01-02 21:56 UTC (permalink / raw) To: Jason Gunthorpe Cc: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters, linux-arm-kernel@lists.infradead.org On Mon, Dec 23, 2013 at 9:46 AM, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote: > On Mon, Dec 23, 2013 at 01:32:03PM +0530, Tanmay Inamdar wrote: >> This patch adds the device tree nodes for APM X-Gene PCIe controller and >> PCIe clock interface. Since X-Gene SOC supports maximum 5 ports, 5 dts nodes >> are added. > > Can you include an lspci dump for PCI DT bindings please? It is > impossible to review otherwise.. > On the X-Gene evaluation platform, there is only one PCIe port enabled. Here is the 'lspci' dump # lspci -vvv 00:00.0 Class 0604: Device 19aa:e008 (rev 04) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Region 0: Memory at <ignored> (64-bit, prefetchable) Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: 00c00000-00cfffff Prefetchable memory behind bridge: 0000000000000000-0000000000bfffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 16384 bytes, PhantFunc 1, Latency L0s <1us, L1 unlimited ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend+ LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Latency L0 unlimited, L1 unlimited ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 10.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState+ RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [180 v1] #19 Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> 01:00.0 Class 0200: Device 15b3:1003 Subsystem: Device 15b3:0049 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 226 Region 0: Memory at e000c00000 (64-bit, non-prefetchable) [size=1M] Region 2: Memory at e000000000 (64-bit, prefetchable) [size=8M] Expansion ROM at e000800000 [size=1M] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] Vital Product Data Product Name: CX312A - ConnectX-3 SFP+ Read-only fields: [PN] Part number: MCX312A-XCBT [EC] Engineering changes: A5 [SN] Serial number: MT1338X00578 [V0] Vendor specific: PCIe Gen3 x8 [RV] Reserved: checksum good, 0 byte(s) reserved Read/write fields: [V1] Vendor specific: N/A [YA] Asset tag: N/A [RW] Read-write area: 105 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 253 byte(s) free [RW] Read-write area: 252 byte(s) free End Capabilities: [9c] MSI-X: Enable- Count=128 Masked- Vector table: BAR=0 offset=0007c000 PBA: BAR=0 offset=0007d000 Capabilities: [60] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [148 v1] Device Serial Number f4-52-14-03-00-0b-dc-90 Capabilities: [154 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [18c v1] #19 Kernel driver in use: mlx4_core > Regards, > Jason ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes 2014-01-02 21:56 ` Tanmay Inamdar @ 2014-01-03 0:52 ` Jason Gunthorpe 2014-01-07 2:56 ` Tanmay Inamdar 0 siblings, 1 reply; 27+ messages in thread From: Jason Gunthorpe @ 2014-01-03 0:52 UTC (permalink / raw) To: Tanmay Inamdar Cc: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters, linux-arm-kernel@lists.infradead.org On Thu, Jan 02, 2014 at 01:56:51PM -0800, Tanmay Inamdar wrote: > On Mon, Dec 23, 2013 at 9:46 AM, Jason Gunthorpe > <jgunthorpe@obsidianresearch.com> wrote: > > On Mon, Dec 23, 2013 at 01:32:03PM +0530, Tanmay Inamdar wrote: > >> This patch adds the device tree nodes for APM X-Gene PCIe controller and > >> PCIe clock interface. Since X-Gene SOC supports maximum 5 ports, 5 dts nodes > >> are added. > > > > Can you include an lspci dump for PCI DT bindings please? It is > > impossible to review otherwise.. > > > > On the X-Gene evaluation platform, there is only one PCIe port > enabled. Here is the 'lspci' dump This is a bit hard to read withouth more context, but: > # lspci -vvv > 00:00.0 Class 0604: Device 19aa:e008 (rev 04) > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- This is an on-chip device? (19aa does not seem to be a VID I can find) Ideally this is the on-chip PCI-PCI bridge which represents the port. The problem I see is that your DT binding has a top level stanza per port. We *really* prefer to see a single stanza for all ports - but this requires the HW to be able to fit into the Linux resource assignment model - a single resource pool for all ports and standard PCI-PCI bridge config access to assign the resource to a port. If your HW can't do this (eg because the port aperture 0xe000000000 is hard wired) then the fall back is to place every port in a distinct domain, with a distinct DT node and have overlapping bus numbers and fixed windows. I don't see PCI domain support in your driver.. There is some kind of an addressing problem because you've done this: +static void xgene_pcie_fixup_bridge(struct pci_dev *dev) +{ + int i; + + for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) { + dev->resource[i].start = dev->resource[i].end = 0; + dev->resource[i].flags = 0; + } +} +DECLARE_PCI_FIXUP_HEADER(XGENE_PCIE_VENDORID, XGENE_PCIE_BRIDGE_DEVICEID, + xgene_pcie_fixup_bridge); Which is usually a sign that something is wonky with how the HW is being fit into the PCI core. > ParErr+ Stepping- SERR+ FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Region 0: Memory at <ignored> (64-bit, prefetchable) > Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 > I/O behind bridge: 0000f000-00000fff > Memory behind bridge: 00c00000-00cfffff [..] > 01:00.0 Class 0200: Device 15b3:1003 > Region 0: Memory at e000c00000 (64-bit, non-prefetchable) [size=1M] > Region 2: Memory at e000000000 (64-bit, prefetchable) > [size=8M] Something funky is going on here too, the 64 bit address e000000000 should be reflected in the 'memory behind bridge' above, not truncated. ranges = <0x02000000 0x0 0x00000000 0x90 0x00000000 0x0 0x10000000 /* mem*/ + 0x01000000 0x0 0x80000000 0x90 0x80000000 0x0 0x00010000 /* io */ + 0x00000000 0x0 0xd0000000 0x90 0xd0000000 0x0 0x00200000 /* cfg */ + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ Ranges has a defined meaning, MSI shouldn't be in ranges, and 'cfg' is only OK if the address encoding exactly matches the funky PCI-E extended configuration address format. You can move these to regs or other properties (MSI is tricky, I'm not aware of DT binding work for MSI :() Also, unrelated, can you please double check that your HW cannot generate 8 and 16 bit configuration write TLPs natively? The xgene_pcie_cfg_out8/16 hack is not desirable if it can be avoided. Regards, Jason ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes 2014-01-03 0:52 ` Jason Gunthorpe @ 2014-01-07 2:56 ` Tanmay Inamdar 2014-01-07 17:27 ` Jason Gunthorpe 0 siblings, 1 reply; 27+ messages in thread From: Tanmay Inamdar @ 2014-01-07 2:56 UTC (permalink / raw) To: Jason Gunthorpe Cc: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters, linux-arm-kernel@lists.infradead.org On Thu, Jan 2, 2014 at 4:52 PM, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote: > On Thu, Jan 02, 2014 at 01:56:51PM -0800, Tanmay Inamdar wrote: >> On Mon, Dec 23, 2013 at 9:46 AM, Jason Gunthorpe >> <jgunthorpe@obsidianresearch.com> wrote: >> > On Mon, Dec 23, 2013 at 01:32:03PM +0530, Tanmay Inamdar wrote: >> >> This patch adds the device tree nodes for APM X-Gene PCIe controller and >> >> PCIe clock interface. Since X-Gene SOC supports maximum 5 ports, 5 dts nodes >> >> are added. >> > >> > Can you include an lspci dump for PCI DT bindings please? It is >> > impossible to review otherwise.. >> > >> >> On the X-Gene evaluation platform, there is only one PCIe port >> enabled. Here is the 'lspci' dump > > This is a bit hard to read withouth more context, but: > >> # lspci -vvv >> 00:00.0 Class 0604: Device 19aa:e008 (rev 04) >> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- > > This is an on-chip device? (19aa does not seem to be a VID I can find) oops.. looks like vendor and device IDs are jumbled. You can look for e008 vendor ID. I will fix it in next version. > > Ideally this is the on-chip PCI-PCI bridge which represents the port. > > The problem I see is that your DT binding has a top level stanza per > port. > > We *really* prefer to see a single stanza for all ports - but this > requires the HW to be able to fit into the Linux resource assignment > model - a single resource pool for all ports and standard PCI-PCI > bridge config access to assign the resource to a port. > > If your HW can't do this (eg because the port aperture 0xe000000000 is > hard wired) then the fall back is to place every port in a distinct > domain, with a distinct DT node and have overlapping bus numbers > and fixed windows. I don't see PCI domain support in your driver.. Thanks for the suggestion. I will think over this. > > There is some kind of an addressing problem because you've done this: > > +static void xgene_pcie_fixup_bridge(struct pci_dev *dev) > +{ > + int i; > + > + for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) { > + dev->resource[i].start = dev->resource[i].end = 0; > + dev->resource[i].flags = 0; > + } > +} > +DECLARE_PCI_FIXUP_HEADER(XGENE_PCIE_VENDORID, XGENE_PCIE_BRIDGE_DEVICEID, > + xgene_pcie_fixup_bridge); > > Which is usually a sign that something is wonky with how the HW is > being fit into the PCI core. We map the whole DDR range (eg 256 GB) into host's BAR. The Linux PCI resource management tries to fit the host's memory into the ranges provided (eg 0xe000000000). Please let me know if there is any use case to do this mapping. > >> ParErr+ Stepping- SERR+ FastB2B- DisINTx- >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- >> <TAbort- <MAbort- >SERR- <PERR- INTx- >> Latency: 0, Cache Line Size: 64 bytes >> Region 0: Memory at <ignored> (64-bit, prefetchable) >> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 >> I/O behind bridge: 0000f000-00000fff >> Memory behind bridge: 00c00000-00cfffff > > [..] > >> 01:00.0 Class 0200: Device 15b3:1003 >> Region 0: Memory at e000c00000 (64-bit, non-prefetchable) [size=1M] >> Region 2: Memory at e000000000 (64-bit, prefetchable) >> [size=8M] > > Something funky is going on here too, the 64 bit address e000000000 > should be reflected in the 'memory behind bridge' above, not > truncated. That's the Mellanox device that is plugged into the system. The device's memory gets mapped at '0xe0xxxxxxxx' > > ranges = <0x02000000 0x0 0x00000000 0x90 0x00000000 0x0 0x10000000 /* mem*/ > + 0x01000000 0x0 0x80000000 0x90 0x80000000 0x0 0x00010000 /* io */ > + 0x00000000 0x0 0xd0000000 0x90 0xd0000000 0x0 0x00200000 /* cfg */ > + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ > > Ranges has a defined meaning, MSI shouldn't be in ranges, and 'cfg' is > only OK if the address encoding exactly matches the funky PCI-E extended > configuration address format. You can move these to regs or other > properties ok > > (MSI is tricky, I'm not aware of DT binding work for MSI :() > It does not. This is the range required to be mapped by the controller to support MSI. I know it is not a standard way of doing. I was just trying to utilize 'of_pci_range_to_resource' api. > Also, unrelated, can you please double check that your HW cannot > generate 8 and 16 bit configuration write TLPs natively? The > xgene_pcie_cfg_out8/16 hack is not desirable if it can be avoided. > Sadly HW cannot generate 8 and 16 bit configuration transactions. > Regards, > Jason ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes 2014-01-07 2:56 ` Tanmay Inamdar @ 2014-01-07 17:27 ` Jason Gunthorpe 2014-01-10 1:30 ` Tanmay Inamdar 0 siblings, 1 reply; 27+ messages in thread From: Jason Gunthorpe @ 2014-01-07 17:27 UTC (permalink / raw) To: Tanmay Inamdar Cc: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters, linux-arm-kernel@lists.infradead.org On Mon, Jan 06, 2014 at 06:56:21PM -0800, Tanmay Inamdar wrote: > > There is some kind of an addressing problem because you've done this: > > > > +static void xgene_pcie_fixup_bridge(struct pci_dev *dev) > > +{ > > + int i; > > + > > + for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) { > > + dev->resource[i].start = dev->resource[i].end = 0; > > + dev->resource[i].flags = 0; > > + } > > +} > > +DECLARE_PCI_FIXUP_HEADER(XGENE_PCIE_VENDORID, XGENE_PCIE_BRIDGE_DEVICEID, > > + xgene_pcie_fixup_bridge); > > > > Which is usually a sign that something is wonky with how the HW is > > being fit into the PCI core. > > We map the whole DDR range (eg 256 GB) into host's BAR. The Linux PCI > resource management tries to fit the host's memory into the ranges > provided (eg 0xe000000000). > Please let me know if there is any use case to do this mapping. If you need to set the bridge's BAR like this, then the bridge is not non-conforming.. Bridge BAR's should be 0 size unless the bridge itself has registers. Do any registers in this config space work properly? Does the secondary status reflect the physical link status properly? If it is *really* broken you might just consider hiding it from the Linux core. > >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > >> <TAbort- <MAbort- >SERR- <PERR- INTx- > >> Latency: 0, Cache Line Size: 64 bytes > >> Region 0: Memory at <ignored> (64-bit, prefetchable) > >> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 > >> I/O behind bridge: 0000f000-00000fff > >> Memory behind bridge: 00c00000-00cfffff > > > > [..] > > > >> 01:00.0 Class 0200: Device 15b3:1003 > >> Region 0: Memory at e000c00000 (64-bit, non-prefetchable) [size=1M] > >> Region 2: Memory at e000000000 (64-bit, prefetchable) > >> [size=8M] > > > > Something funky is going on here too, the 64 bit address e000000000 > > should be reflected in the 'memory behind bridge' above, not > > truncated. > > That's the Mellanox device that is plugged into the system. The > device's memory gets mapped at '0xe0xxxxxxxx' Right, but the bridge setup above has: > >> Memory behind bridge: 00c00000-00cfffff Which is wrong, it doesn't include the range '0xe0xxxxxxxx' Jason ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes 2014-01-07 17:27 ` Jason Gunthorpe @ 2014-01-10 1:30 ` Tanmay Inamdar 0 siblings, 0 replies; 27+ messages in thread From: Tanmay Inamdar @ 2014-01-10 1:30 UTC (permalink / raw) To: Jason Gunthorpe Cc: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters, linux-arm-kernel@lists.infradead.org On Tue, Jan 7, 2014 at 9:27 AM, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote: > On Mon, Jan 06, 2014 at 06:56:21PM -0800, Tanmay Inamdar wrote: > >> > There is some kind of an addressing problem because you've done this: >> > >> > +static void xgene_pcie_fixup_bridge(struct pci_dev *dev) >> > +{ >> > + int i; >> > + >> > + for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) { >> > + dev->resource[i].start = dev->resource[i].end = 0; >> > + dev->resource[i].flags = 0; >> > + } >> > +} >> > +DECLARE_PCI_FIXUP_HEADER(XGENE_PCIE_VENDORID, XGENE_PCIE_BRIDGE_DEVICEID, >> > + xgene_pcie_fixup_bridge); >> > >> > Which is usually a sign that something is wonky with how the HW is >> > being fit into the PCI core. >> >> We map the whole DDR range (eg 256 GB) into host's BAR. The Linux PCI >> resource management tries to fit the host's memory into the ranges >> provided (eg 0xe000000000). >> Please let me know if there is any use case to do this mapping. > > If you need to set the bridge's BAR like this, then the bridge is not > non-conforming.. Bridge BAR's should be 0 size unless the bridge > itself has registers. They are not set to 0 as per our hardware implementation. We have to hide it using the fixup API. I don't know the reason but "arch/powerpc/sysdev/xilinx_pci.c" is doing the same thing. > > Do any registers in this config space work properly? Does the > secondary status reflect the physical link status properly? Link status information is seen correctly. > > If it is *really* broken you might just consider hiding it from the > Linux core. > >> >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- >> >> <TAbort- <MAbort- >SERR- <PERR- INTx- >> >> Latency: 0, Cache Line Size: 64 bytes >> >> Region 0: Memory at <ignored> (64-bit, prefetchable) >> >> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 >> >> I/O behind bridge: 0000f000-00000fff >> >> Memory behind bridge: 00c00000-00cfffff >> > >> > [..] >> > >> >> 01:00.0 Class 0200: Device 15b3:1003 >> >> Region 0: Memory at e000c00000 (64-bit, non-prefetchable) [size=1M] >> >> Region 2: Memory at e000000000 (64-bit, prefetchable) >> >> [size=8M] >> > >> > Something funky is going on here too, the 64 bit address e000000000 >> > should be reflected in the 'memory behind bridge' above, not >> > truncated. >> >> That's the Mellanox device that is plugged into the system. The >> device's memory gets mapped at '0xe0xxxxxxxx' > > Right, but the bridge setup above has: > >> >> Memory behind bridge: 00c00000-00cfffff > > Which is wrong, it doesn't include the range '0xe0xxxxxxxx' > > Jason ^ permalink raw reply [flat|nested] 27+ messages in thread
* [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings 2013-12-23 8:02 [RFC PATCH 0/3] APM X-Gene PCIe driver Tanmay Inamdar 2013-12-23 8:02 ` [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver Tanmay Inamdar 2013-12-23 8:02 ` [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes Tanmay Inamdar @ 2013-12-23 8:02 ` Tanmay Inamdar [not found] ` <1387785725-24262-4-git-send-email-tinamdar-qTEPVZfXA3Y@public.gmane.org> 2013-12-23 8:56 ` [RFC PATCH 0/3] APM X-Gene PCIe driver Tanmay Inamdar 3 siblings, 1 reply; 27+ messages in thread From: Tanmay Inamdar @ 2013-12-23 8:02 UTC (permalink / raw) To: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley Cc: linux-pci, devicetree, linux-arm-kernel, linux-doc, linux-kernel, patches, jcm, Tanmay Inamdar This patch adds the bindings for X-Gene PCIe driver. The driver resides under 'drivers/pci/host/pcie-xgene.c' file. Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> --- .../devicetree/bindings/pci/xgene-pcie.txt | 43 ++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 Documentation/devicetree/bindings/pci/xgene-pcie.txt diff --git a/Documentation/devicetree/bindings/pci/xgene-pcie.txt b/Documentation/devicetree/bindings/pci/xgene-pcie.txt new file mode 100644 index 0000000..d92da4f --- /dev/null +++ b/Documentation/devicetree/bindings/pci/xgene-pcie.txt @@ -0,0 +1,43 @@ +* AppliedMicro X-Gene PCIe interface + +Required properties: +- status: Either "ok" or "disabled". +- device_type: set to "pci" +- compatible: should contain "xgene,pcie" to identify the core. +- reg: base addresses and lengths of the pcie controller configuration + space register. +- #address-cells: set to <3> +- #size-cells: set to <2> +- ranges: ranges for the PCI memory, I/O regions, config and MSI regions +- #interrupt-cells: set to <1> +- interrupt-map-mask and interrupt-map: standard PCI properties + to define the mapping of the PCIe interface to interrupt + numbers. +- clocks: from common clock binding: handle to pci clock. +- clock-names: from common clock binding. Should be "pcieclk". + +Example: + +SoC specific DT Entry: + pcie0: pcie@1f2b0000 { + status = "disabled"; + device_type = "pci"; + compatible = "xgene,pcie"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + reg = < 0x00 0x1f2b0000 0x0 0x00010000>; + ranges = <0x02000000 0x0 0x00000000 0xe0 0x00000000 0x0 0x10000000 /* mem*/ + 0x01000000 0x0 0x80000000 0xe0 0x80000000 0x0 0x00010000 /* io */ + 0x00000000 0x0 0xd0000000 0xe0 0xd0000000 0x0 0x00200000 /* cfg */ + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xc2 0x1>; + clocks = <&pcie0clk 0>; + clock-names = "pcieclk" + }; + +Board specific DT Entry: + &pcie0 { + status = "ok"; + }; -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 27+ messages in thread
[parent not found: <1387785725-24262-4-git-send-email-tinamdar-qTEPVZfXA3Y@public.gmane.org>]
* Re: [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings [not found] ` <1387785725-24262-4-git-send-email-tinamdar-qTEPVZfXA3Y@public.gmane.org> @ 2014-01-03 9:49 ` Arnd Bergmann 2014-01-07 3:04 ` Tanmay Inamdar 0 siblings, 1 reply; 27+ messages in thread From: Arnd Bergmann @ 2014-01-03 9:49 UTC (permalink / raw) To: linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r Cc: Tanmay Inamdar, Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-doc-u79uwXL29TY76Z2rM5mHXA, linux-pci-u79uwXL29TY76Z2rM5mHXA, patches-qTEPVZfXA3Y, linux-kernel-u79uwXL29TY76Z2rM5mHXA, jcm-H+wXaHxf7aLQT0dZR+AlfA On Monday 23 December 2013, Tanmay Inamdar wrote: > This patch adds the bindings for X-Gene PCIe driver. The driver resides > under 'drivers/pci/host/pcie-xgene.c' file. > > Signed-off-by: Tanmay Inamdar <tinamdar-qTEPVZfXA3Y@public.gmane.org> > --- > .../devicetree/bindings/pci/xgene-pcie.txt | 43 ++++++++++++++++++++ > 1 file changed, 43 insertions(+) > create mode 100644 Documentation/devicetree/bindings/pci/xgene-pcie.txt > > diff --git a/Documentation/devicetree/bindings/pci/xgene-pcie.txt b/Documentation/devicetree/bindings/pci/xgene-pcie.txt > new file mode 100644 > index 0000000..d92da4f > --- /dev/null > +++ b/Documentation/devicetree/bindings/pci/xgene-pcie.txt > @@ -0,0 +1,43 @@ > +* AppliedMicro X-Gene PCIe interface > + > +Required properties: > +- status: Either "ok" or "disabled". > +- device_type: set to "pci" > +- compatible: should contain "xgene,pcie" to identify the core. > +- reg: base addresses and lengths of the pcie controller configuration > + space register. > +- #address-cells: set to <3> > +- #size-cells: set to <2> > +- ranges: ranges for the PCI memory, I/O regions, config and MSI regions > +- #interrupt-cells: set to <1> > +- interrupt-map-mask and interrupt-map: standard PCI properties > + to define the mapping of the PCIe interface to interrupt > + numbers. > +- clocks: from common clock binding: handle to pci clock. > +- clock-names: from common clock binding. Should be "pcieclk". > + Better use an anonymous clock? The driver also seems to use a phy that is not defined here. > +Example: > + > +SoC specific DT Entry: > + pcie0: pcie@1f2b0000 { > + status = "disabled"; > + device_type = "pci"; > + compatible = "xgene,pcie"; > + #interrupt-cells = <1>; > + #size-cells = <2>; > + #address-cells = <3>; > + reg = < 0x00 0x1f2b0000 0x0 0x00010000>; > + ranges = <0x02000000 0x0 0x00000000 0xe0 0x00000000 0x0 0x10000000 /* mem*/ This is an awfully small memory space for a 64 bit machine. Is that a hardware bug you are working around? If not, please make it as large as you can to allow for arbitrary extension cards with large BARs, at least 4GB. Also, do you support no prefetchable memory? > + 0x01000000 0x0 0x80000000 0xe0 0x80000000 0x0 0x00010000 /* io */ I/O space at 0x80000000? That won't work with a lot of devices. Can you make it start at bus address 0? > + 0x00000000 0x0 0xd0000000 0xe0 0xd0000000 0x0 0x00200000 /* cfg */ config space is not normally in the ranges property, and I think you will need it in the pcie node itself as a 'reg' property so the code can access it. > + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ Same here. Also I suspect this is a separate irqchip that isn't actually part of the PCIe host. If they are separate devices, I'd strongly recommend modeling them as separate device-nodes in DT to make reuse easier. > + interrupt-map-mask = <0x0 0x0 0x0 0x7>; > + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xc2 0x1>; Only one IRQ for all devices? > + clocks = <&pcie0clk 0>; > + clock-names = "pcieclk" > + }; > + > +Board specific DT Entry: > + &pcie0 { > + status = "ok"; > + }; Arnd -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings 2014-01-03 9:49 ` Arnd Bergmann @ 2014-01-07 3:04 ` Tanmay Inamdar 2014-01-07 15:35 ` Arnd Bergmann 0 siblings, 1 reply; 27+ messages in thread From: Tanmay Inamdar @ 2014-01-07 3:04 UTC (permalink / raw) To: Arnd Bergmann Cc: linux-arm-kernel@lists.infradead.org, Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters On Fri, Jan 3, 2014 at 1:49 AM, Arnd Bergmann <arnd@arndb.de> wrote: > On Monday 23 December 2013, Tanmay Inamdar wrote: >> This patch adds the bindings for X-Gene PCIe driver. The driver resides >> under 'drivers/pci/host/pcie-xgene.c' file. >> >> Signed-off-by: Tanmay Inamdar <tinamdar@apm.com> >> --- >> .../devicetree/bindings/pci/xgene-pcie.txt | 43 ++++++++++++++++++++ >> 1 file changed, 43 insertions(+) >> create mode 100644 Documentation/devicetree/bindings/pci/xgene-pcie.txt >> >> diff --git a/Documentation/devicetree/bindings/pci/xgene-pcie.txt b/Documentation/devicetree/bindings/pci/xgene-pcie.txt >> new file mode 100644 >> index 0000000..d92da4f >> --- /dev/null >> +++ b/Documentation/devicetree/bindings/pci/xgene-pcie.txt >> @@ -0,0 +1,43 @@ >> +* AppliedMicro X-Gene PCIe interface >> + >> +Required properties: >> +- status: Either "ok" or "disabled". >> +- device_type: set to "pci" >> +- compatible: should contain "xgene,pcie" to identify the core. >> +- reg: base addresses and lengths of the pcie controller configuration >> + space register. >> +- #address-cells: set to <3> >> +- #size-cells: set to <2> >> +- ranges: ranges for the PCI memory, I/O regions, config and MSI regions >> +- #interrupt-cells: set to <1> >> +- interrupt-map-mask and interrupt-map: standard PCI properties >> + to define the mapping of the PCIe interface to interrupt >> + numbers. >> +- clocks: from common clock binding: handle to pci clock. >> +- clock-names: from common clock binding. Should be "pcieclk". >> + > > Better use an anonymous clock? Sorry. Can you please elaborate? > > The driver also seems to use a phy that is not defined here. > >> +Example: >> + >> +SoC specific DT Entry: >> + pcie0: pcie@1f2b0000 { >> + status = "disabled"; >> + device_type = "pci"; >> + compatible = "xgene,pcie"; >> + #interrupt-cells = <1>; >> + #size-cells = <2>; >> + #address-cells = <3>; >> + reg = < 0x00 0x1f2b0000 0x0 0x00010000>; >> + ranges = <0x02000000 0x0 0x00000000 0xe0 0x00000000 0x0 0x10000000 /* mem*/ > > This is an awfully small memory space for a 64 bit machine. Is that a hardware bug you > are working around? If not, please make it as large as you can to allow for arbitrary > extension cards with large BARs, at least 4GB. HW does support a lot more than 256MB. I will change it. > > Also, do you support no prefetchable memory? HW has either IO or Memory regions for mapping device's memory space. There is no separate prefetchable memory space. > >> + 0x01000000 0x0 0x80000000 0xe0 0x80000000 0x0 0x00010000 /* io */ > > I/O space at 0x80000000? That won't work with a lot of devices. Can you make it start > at bus address 0? Ok. > >> + 0x00000000 0x0 0xd0000000 0xe0 0xd0000000 0x0 0x00200000 /* cfg */ > > config space is not normally in the ranges property, and I think you will need > it in the pcie node itself as a 'reg' property so the code can access it. pcie-designware.c does it that way. I just followed their implementation. > >> + 0x00000000 0x0 0x79000000 0x00 0x79000000 0x0 0x00800000>; /* msi */ > > Same here. Also I suspect this is a separate irqchip that isn't actually part > of the PCIe host. If they are separate devices, I'd strongly recommend modeling > them as separate device-nodes in DT to make reuse easier. You are right. However MSI irqchip needs PCIe core to map above mentioned space. I will remove it for now and will send it along with MSI patch. > >> + interrupt-map-mask = <0x0 0x0 0x0 0x7>; >> + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xc2 0x1>; > > Only one IRQ for all devices? The node represents a port. I believe that Linux framework uses only one of the legacy IRQs per port. Rest all remain unused. Hence I removed them. Please correct me if I am wrong. > >> + clocks = <&pcie0clk 0>; >> + clock-names = "pcieclk" >> + }; >> + >> +Board specific DT Entry: >> + &pcie0 { >> + status = "ok"; >> + }; > > Arnd ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings 2014-01-07 3:04 ` Tanmay Inamdar @ 2014-01-07 15:35 ` Arnd Bergmann 2014-01-07 15:44 ` Arnd Bergmann ` (2 more replies) 0 siblings, 3 replies; 27+ messages in thread From: Arnd Bergmann @ 2014-01-07 15:35 UTC (permalink / raw) To: Tanmay Inamdar Cc: linux-arm-kernel@lists.infradead.org, Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters On Tuesday 07 January 2014, Tanmay Inamdar wrote: > On Fri, Jan 3, 2014 at 1:49 AM, Arnd Bergmann <arnd@arndb.de> wrote: > >> +Required properties: > >> +- status: Either "ok" or "disabled". > >> +- device_type: set to "pci" > >> +- compatible: should contain "xgene,pcie" to identify the core. > >> +- reg: base addresses and lengths of the pcie controller configuration > >> + space register. > >> +- #address-cells: set to <3> > >> +- #size-cells: set to <2> > >> +- ranges: ranges for the PCI memory, I/O regions, config and MSI regions > >> +- #interrupt-cells: set to <1> > >> +- interrupt-map-mask and interrupt-map: standard PCI properties > >> + to define the mapping of the PCIe interface to interrupt > >> + numbers. > >> +- clocks: from common clock binding: handle to pci clock. > >> +- clock-names: from common clock binding. Should be "pcieclk". > >> + > > > > Better use an anonymous clock? > > Sorry. Can you please elaborate? I mean drop the "clock-names" property. > >> +SoC specific DT Entry: > >> + pcie0: pcie@1f2b0000 { > >> + status = "disabled"; > >> + device_type = "pci"; > >> + compatible = "xgene,pcie"; > >> + #interrupt-cells = <1>; > >> + #size-cells = <2>; > >> + #address-cells = <3>; > >> + reg = < 0x00 0x1f2b0000 0x0 0x00010000>; > >> + ranges = <0x02000000 0x0 0x00000000 0xe0 0x00000000 0x0 0x10000000 /* mem*/ > > > > > > Also, do you support no prefetchable memory? > > HW has either IO or Memory regions for mapping device's memory space. > There is no separate prefetchable memory space. Are you sure the memory is non-prefetchable then? I would have expected 0x42000000 rather than 0x02000000, but I could be misremembering it. > > > >> + 0x00000000 0x0 0xd0000000 0xe0 0xd0000000 0x0 0x00200000 /* cfg */ > > > > config space is not normally in the ranges property, and I think you will need > > it in the pcie node itself as a 'reg' property so the code can access it. > > pcie-designware.c does it that way. I just followed their implementation. I don't remember what led to that, it still seems wrong and I can't find anything in the PCI binding for host bridges telling their config space this way. > >> + interrupt-map-mask = <0x0 0x0 0x0 0x7>; > >> + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xc2 0x1>; > > > > Only one IRQ for all devices? > > The node represents a port. I believe that Linux framework uses only > one of the legacy IRQs per port. Rest all remain unused. Hence I > removed them. Please correct me if I am wrong. Any PCI device can normally have four interrupts (IntA through IntD), which are traditionally separate pins on a PCI bus, but get emulated on PCIe. While it's not common for any normal device to use more than one IRQ, a bridge device will swizzle these (virtual) IRQ lines, so a device behind the bridge actually gets a different host IRQ. Arnd ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings 2014-01-07 15:35 ` Arnd Bergmann @ 2014-01-07 15:44 ` Arnd Bergmann 2014-01-07 18:31 ` Jason Gunthorpe 2014-01-11 0:12 ` Tanmay Inamdar 2 siblings, 0 replies; 27+ messages in thread From: Arnd Bergmann @ 2014-01-07 15:44 UTC (permalink / raw) To: linux-arm-kernel Cc: Tanmay Inamdar, devicetree, Jon Masters, linux-doc, Catalin Marinas, patches, linux-kernel@vger.kernel.org, Grant Likely, Rob Landley, linux-pci, Bjorn Helgaas On Tuesday 07 January 2014 16:35:01 Arnd Bergmann wrote: > > >> +SoC specific DT Entry: > > >> + pcie0: pcie@1f2b0000 { > > >> + status = "disabled"; > > >> + device_type = "pci"; > > >> + compatible = "xgene,pcie"; > > >> + #interrupt-cells = <1>; > > >> + #size-cells = <2>; > > >> + #address-cells = >; > > >> + reg = < 0x00 0x1f2b0000 0x0 0x00010000>; > > >> + ranges = <0x02000000 0x0 0x00000000 0xe0 0x00000000 0x0 0x10000000 /* mem*/ > > > > > > > > > Also, do you support no prefetchable memory? > > > > HW has either IO or Memory regions for mapping device's memory space. > > There is no separate prefetchable memory space. > > Are you sure the memory is non-prefetchable then? I would have expected > 0x42000000 rather than 0x02000000, but I could be misremembering it. Nevermind. I just checked and you are right: if you only have one memory range, it has to be non-prefetchable. Arnd ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings 2014-01-07 15:35 ` Arnd Bergmann 2014-01-07 15:44 ` Arnd Bergmann @ 2014-01-07 18:31 ` Jason Gunthorpe 2014-01-10 1:32 ` Tanmay Inamdar 2014-01-11 0:12 ` Tanmay Inamdar 2 siblings, 1 reply; 27+ messages in thread From: Jason Gunthorpe @ 2014-01-07 18:31 UTC (permalink / raw) To: Arnd Bergmann Cc: Tanmay Inamdar, devicetree, Jon Masters, linux-doc, Catalin Marinas, patches, linux-kernel@vger.kernel.org, Grant Likely, Rob Landley, linux-pci, Bjorn Helgaas, linux-arm-kernel@lists.infradead.org On Tue, Jan 07, 2014 at 04:35:01PM +0100, Arnd Bergmann wrote: > > >> + 0x00000000 0x0 0xd0000000 0xe0 0xd0000000 0x0 0x00200000 /* cfg */ > > > > > > config space is not normally in the ranges property, and I think you will need > > > it in the pcie node itself as a 'reg' property so the code can access it. > > > > pcie-designware.c does it that way. I just followed their implementation. > > I don't remember what led to that, it still seems wrong and I can't > find anything in the PCI binding for host bridges telling their > config space this way. When we discussed the mvebu PCI driver (which is, so far, the most throughly discussed PCI binding) it was concluded that the config space ranges like the above was OK only if it exactly described the standard ECAM layout. Idea being that standard/core code should be able to see that ranges, map the range and issue config accesses via the ECAM rules. > > >> + interrupt-map-mask = <0x0 0x0 0x0 0x7>; > > >> + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xc2 0x1>; > > > > > > Only one IRQ for all devices? > > > > The node represents a port. I believe that Linux framework uses only > > one of the legacy IRQs per port. Rest all remain unused. Hence I > > removed them. Please correct me if I am wrong. > > Any PCI device can normally have four interrupts (IntA through > IntD), which are traditionally separate pins on a PCI bus, but get > emulated on PCIe. While it's not common for any normal device to use > more than one IRQ, a bridge device will swizzle these (virtual) IRQ > lines, so a device behind the bridge actually gets a different host > IRQ. Agree, the binding should handle all four INTA,B,C,D assertions delivered to the port. If HW is able to decode the 4 ints into seperate Linux interrupt numbers then that should be described. If HW routes them all to a single number then interrupt-map-mask should be all 0. Arnd's point about swizzling effects the layout of the interrupt-map. When it is placed at the pcie-controller node level the map will incorporate one swizzle of the on-the-wire INTx messages. If the HW doesn't swizzle the INTx as the TLP passes through the bridge then it probably makes more sense to put the interrupt-map in the DT node of the bridge like mvebu does. Jason ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings 2014-01-07 18:31 ` Jason Gunthorpe @ 2014-01-10 1:32 ` Tanmay Inamdar 0 siblings, 0 replies; 27+ messages in thread From: Tanmay Inamdar @ 2014-01-10 1:32 UTC (permalink / raw) To: Jason Gunthorpe Cc: Arnd Bergmann, devicetree, Jon Masters, linux-doc, Catalin Marinas, patches, linux-kernel@vger.kernel.org, Grant Likely, Rob Landley, linux-pci, Bjorn Helgaas, linux-arm-kernel@lists.infradead.org On Tue, Jan 7, 2014 at 10:31 AM, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote: > On Tue, Jan 07, 2014 at 04:35:01PM +0100, Arnd Bergmann wrote: > >> > >> + 0x00000000 0x0 0xd0000000 0xe0 0xd0000000 0x0 0x00200000 /* cfg */ >> > > >> > > config space is not normally in the ranges property, and I think you will need >> > > it in the pcie node itself as a 'reg' property so the code can access it. >> > >> > pcie-designware.c does it that way. I just followed their implementation. >> >> I don't remember what led to that, it still seems wrong and I can't >> find anything in the PCI binding for host bridges telling their >> config space this way. > > When we discussed the mvebu PCI driver (which is, so far, the most > throughly discussed PCI binding) it was concluded that the config > space ranges like the above was OK only if it exactly described the > standard ECAM layout. > > Idea being that standard/core code should be able to see that ranges, > map the range and issue config accesses via the ECAM rules. Ok. Thanks. > >> > >> + interrupt-map-mask = <0x0 0x0 0x0 0x7>; >> > >> + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xc2 0x1>; >> > > >> > > Only one IRQ for all devices? >> > >> > The node represents a port. I believe that Linux framework uses only >> > one of the legacy IRQs per port. Rest all remain unused. Hence I >> > removed them. Please correct me if I am wrong. >> >> Any PCI device can normally have four interrupts (IntA through >> IntD), which are traditionally separate pins on a PCI bus, but get >> emulated on PCIe. While it's not common for any normal device to use >> more than one IRQ, a bridge device will swizzle these (virtual) IRQ >> lines, so a device behind the bridge actually gets a different host >> IRQ. > > Agree, the binding should handle all four INTA,B,C,D assertions > delivered to the port. > > If HW is able to decode the 4 ints into seperate Linux interrupt > numbers then that should be described. If HW routes them all to a > single number then interrupt-map-mask should be all 0. > > Arnd's point about swizzling effects the layout of the > interrupt-map. When it is placed at the pcie-controller node level the > map will incorporate one swizzle of the on-the-wire INTx messages. If > the HW doesn't swizzle the INTx as the TLP passes through the bridge > then it probably makes more sense to put the interrupt-map in the DT > node of the bridge like mvebu does. > Ok. > Jason ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings 2014-01-07 15:35 ` Arnd Bergmann 2014-01-07 15:44 ` Arnd Bergmann 2014-01-07 18:31 ` Jason Gunthorpe @ 2014-01-11 0:12 ` Tanmay Inamdar 2014-01-11 13:06 ` Arnd Bergmann 2 siblings, 1 reply; 27+ messages in thread From: Tanmay Inamdar @ 2014-01-11 0:12 UTC (permalink / raw) To: Arnd Bergmann Cc: linux-arm-kernel@lists.infradead.org, Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters On Tue, Jan 7, 2014 at 7:35 AM, Arnd Bergmann <arnd@arndb.de> wrote: > On Tuesday 07 January 2014, Tanmay Inamdar wrote: >> On Fri, Jan 3, 2014 at 1:49 AM, Arnd Bergmann <arnd@arndb.de> wrote: >> >> +Required properties: >> >> +- status: Either "ok" or "disabled". >> >> +- device_type: set to "pci" >> >> +- compatible: should contain "xgene,pcie" to identify the core. >> >> +- reg: base addresses and lengths of the pcie controller configuration >> >> + space register. >> >> +- #address-cells: set to <3> >> >> +- #size-cells: set to <2> >> >> +- ranges: ranges for the PCI memory, I/O regions, config and MSI regions >> >> +- #interrupt-cells: set to <1> >> >> +- interrupt-map-mask and interrupt-map: standard PCI properties >> >> + to define the mapping of the PCIe interface to interrupt >> >> + numbers. >> >> +- clocks: from common clock binding: handle to pci clock. >> >> +- clock-names: from common clock binding. Should be "pcieclk". >> >> + >> > >> > Better use an anonymous clock? >> >> Sorry. Can you please elaborate? > > I mean drop the "clock-names" property. > Did you mean clock-names in pcie-clock node or pcie node? I can drop clock-names from pcie clock node. However if I drop clock-names from pcie node, then clk_get call from pcie host driver would result in failure. Right? >> >> +SoC specific DT Entry: >> >> + pcie0: pcie@1f2b0000 { >> >> + status = "disabled"; >> >> + device_type = "pci"; >> >> + compatible = "xgene,pcie"; >> >> + #interrupt-cells = <1>; >> >> + #size-cells = <2>; >> >> + #address-cells = <3>; >> >> + reg = < 0x00 0x1f2b0000 0x0 0x00010000>; >> >> + ranges = <0x02000000 0x0 0x00000000 0xe0 0x00000000 0x0 0x10000000 /* mem*/ >> > >> > >> > Also, do you support no prefetchable memory? >> >> HW has either IO or Memory regions for mapping device's memory space. >> There is no separate prefetchable memory space. > > Are you sure the memory is non-prefetchable then? I would have expected > 0x42000000 rather than 0x02000000, but I could be misremembering it. > >> > >> >> + 0x00000000 0x0 0xd0000000 0xe0 0xd0000000 0x0 0x00200000 /* cfg */ >> > >> > config space is not normally in the ranges property, and I think you will need >> > it in the pcie node itself as a 'reg' property so the code can access it. >> >> pcie-designware.c does it that way. I just followed their implementation. > > I don't remember what led to that, it still seems wrong and I can't find anything > in the PCI binding for host bridges telling their config space this way. > >> >> + interrupt-map-mask = <0x0 0x0 0x0 0x7>; >> >> + interrupt-map = <0x0 0x0 0x0 0x1 &gic 0x0 0xc2 0x1>; >> > >> > Only one IRQ for all devices? >> >> The node represents a port. I believe that Linux framework uses only >> one of the legacy IRQs per port. Rest all remain unused. Hence I >> removed them. Please correct me if I am wrong. > > Any PCI device can normally have four interrupts (IntA through IntD), which are > traditionally separate pins on a PCI bus, but get emulated on PCIe. While it's > not common for any normal device to use more than one IRQ, a bridge device > will swizzle these (virtual) IRQ lines, so a device behind the bridge actually > gets a different host IRQ. > > Arnd ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings 2014-01-11 0:12 ` Tanmay Inamdar @ 2014-01-11 13:06 ` Arnd Bergmann 0 siblings, 0 replies; 27+ messages in thread From: Arnd Bergmann @ 2014-01-11 13:06 UTC (permalink / raw) To: Tanmay Inamdar Cc: linux-arm-kernel@lists.infradead.org, Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley, devicetree, linux-doc, linux-pci, patches, linux-kernel@vger.kernel.org, Jon Masters On Saturday 11 January 2014, Tanmay Inamdar wrote: > On Tue, Jan 7, 2014 at 7:35 AM, Arnd Bergmann <arnd@arndb.de> wrote: > > On Tuesday 07 January 2014, Tanmay Inamdar wrote: > >> On Fri, Jan 3, 2014 at 1:49 AM, Arnd Bergmann <arnd@arndb.de> wrote: > >> > > >> > Better use an anonymous clock? > >> > >> Sorry. Can you please elaborate? > > > > I mean drop the "clock-names" property. > > > Did you mean clock-names in pcie-clock node or pcie node? I can drop > clock-names from pcie clock node. However if I drop clock-names from > pcie node, then clk_get call from pcie host driver would result in > failure. Right? I meant drop it from the pcie node, and change the clk_get call to pass NULL instead of the name, which will get the handle for the only clock provided. You only need clock-names if you have more than one clock in the device node and want to identify them. The pcie-clock node should not have a "clock-names" property at all, unless it has a "clocks" property as well and refers to its clock parent with it. I already noticed in another review that the xgene clocks get this part wrong and that should be fixed for all those clock provides, but it's unrelated to what I was talking about here. Arnd ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [RFC PATCH 0/3] APM X-Gene PCIe driver 2013-12-23 8:02 [RFC PATCH 0/3] APM X-Gene PCIe driver Tanmay Inamdar ` (2 preceding siblings ...) 2013-12-23 8:02 ` [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings Tanmay Inamdar @ 2013-12-23 8:56 ` Tanmay Inamdar 3 siblings, 0 replies; 27+ messages in thread From: Tanmay Inamdar @ 2013-12-23 8:56 UTC (permalink / raw) To: Bjorn Helgaas, Grant Likely, Catalin Marinas, Rob Landley Cc: linux-pci, devicetree, linux-arm-kernel@lists.infradead.org, linux-doc, linux-kernel@vger.kernel.org, patches, Jon Masters, Tanmay Inamdar On Mon, Dec 23, 2013 at 1:32 PM, Tanmay Inamdar <tinamdar@apm.com> wrote: > This patch adds support for AppliedMicro X-Gene PCIe host controller. The > driver is tested on X-Gene platform with different gen1/2/3 PCIe endpoint > cards. > > X-Gene PCIe controller driver has depedency on the pcie arch support for > arm64. The arm64 pcie arch support is not yet part of mainline Linux kernel > and approach for arch support is under discussion with arm64 maintainers. > The reference patch can be found here --> https://lkml.org/lkml/2013/10/23/244 > > If someone wishes to test PCIe on X-Gene, arch support patch must be applied > before the patches in this patch set. > > Tanmay Inamdar (3): > pci: APM X-Gene PCIe controller driver > arm64: dts: APM X-Gene PCIe device tree nodes > dt-bindings: pci: xgene pcie device tree bindings > > .../devicetree/bindings/pci/xgene-pcie.txt | 43 + > arch/arm64/boot/dts/apm-mustang.dts | 4 + > arch/arm64/boot/dts/apm-storm.dtsi | 140 +++ > drivers/pci/host/Kconfig | 5 + > drivers/pci/host/Makefile | 1 + > drivers/pci/host/pcie-xgene.c | 1017 ++++++++++++++++++++ > 6 files changed, 1210 insertions(+) > create mode 100644 Documentation/devicetree/bindings/pci/xgene-pcie.txt > create mode 100644 drivers/pci/host/pcie-xgene.c > > -- > 1.7.9.5 > This patch set might have been delivered to limited audience previously because of some problem in my mail settings. My apologies for spamming. -Tanmay ^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2014-01-11 13:06 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-12-23 8:02 [RFC PATCH 0/3] APM X-Gene PCIe driver Tanmay Inamdar
2013-12-23 8:02 ` [RFC PATCH 1/3] pci: APM X-Gene PCIe controller driver Tanmay Inamdar
2014-01-02 21:08 ` Bjorn Helgaas
2014-01-03 12:07 ` Arnd Bergmann
2014-01-07 2:41 ` Tanmay Inamdar
2014-01-07 9:27 ` Arnd Bergmann
2014-01-10 1:20 ` Tanmay Inamdar
2014-01-06 1:47 ` Jingoo Han
2014-01-07 2:45 ` Tanmay Inamdar
2014-01-07 3:31 ` Jingoo Han
2013-12-23 8:02 ` [RFC PATCH 2/3] arm64: dts: APM X-Gene PCIe device tree nodes Tanmay Inamdar
2013-12-23 17:46 ` Jason Gunthorpe
2014-01-02 21:56 ` Tanmay Inamdar
2014-01-03 0:52 ` Jason Gunthorpe
2014-01-07 2:56 ` Tanmay Inamdar
2014-01-07 17:27 ` Jason Gunthorpe
2014-01-10 1:30 ` Tanmay Inamdar
2013-12-23 8:02 ` [RFC PATCH 3/3] dt-bindings: pci: xgene pcie device tree bindings Tanmay Inamdar
[not found] ` <1387785725-24262-4-git-send-email-tinamdar-qTEPVZfXA3Y@public.gmane.org>
2014-01-03 9:49 ` Arnd Bergmann
2014-01-07 3:04 ` Tanmay Inamdar
2014-01-07 15:35 ` Arnd Bergmann
2014-01-07 15:44 ` Arnd Bergmann
2014-01-07 18:31 ` Jason Gunthorpe
2014-01-10 1:32 ` Tanmay Inamdar
2014-01-11 0:12 ` Tanmay Inamdar
2014-01-11 13:06 ` Arnd Bergmann
2013-12-23 8:56 ` [RFC PATCH 0/3] APM X-Gene PCIe driver Tanmay Inamdar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).