[PATCH v5 00/18] dwc MSI fixes, ARTPEC-6 EP mode support, ARTPEC-7 SoC support

[PATCH v5 00/18] dwc MSI fixes, ARTPEC-6 EP mode support, ARTPEC-7 SoC support

[Niklas Cassel]

This is a series that adds:
- PCI endpoint mode support in the ARTPEC-6 driver.
- ARTPEC-7 SoC support in the ARTPEC-6 driver (the SoCs are very similar).
- Small fixes for MSI in designware-ep and designware-host,
  needed to get endpoint mode support working for ARTPEC-6.
- Cleanups in pci-dra7xx to better prepare for endpoint mode in other
  DWC based PCIe drivers.

New in V5:
-Replaced ep->page_size with epc->mem->page_size in patch
 ("PCI: designware-ep: Pre-allocate memory for MSI in dw_pcie_ep_init") and
 in patch ("PCI: designware-ep: Add generic function for raising MSI irq").
-Included a new patch in this patch series:
 ("PCI: dwc: artpec6: Deassert the core before waiting for PHY").

Niklas Cassel (18):
  PCI: dwc: Use the DMA-API to get the MSI address
  PCI: designware-ep: dw_pcie_ep_set_msi() should only set MMC bits
  PCI: designware-ep: Read-only registers need DBI_RO_WR_EN to be
    writable
  PCI: designware-ep: Pre-allocate memory for MSI in dw_pcie_ep_init
  PCI: designware-ep: Remove static keyword from dw_pcie_ep_reset_bar()
  PCI: designware-ep: Add generic function for raising MSI irq
  PCI: dwc: dra7xx: Refactor Kconfig and Makefile handling for host/ep
    mode
  PCI: dwc: dra7xx: Assign pp->ops in dra7xx_add_pcie_port() rather than
    in probe
  PCI: dwc: dra7xx: Help compiler to remove unused code
  PCI: dwc: artpec6: Remove unused defines
  PCI: dwc: artpec6: Use BIT and GENMASK macros
  PCI: dwc: artpec6: Split artpec6_pcie_establish_link() into smaller
    functions
  bindings: PCI: artpec: Add support for endpoint mode
  PCI: dwc: artpec6: Add support for endpoint mode
  PCI: dwc: Make cpu_addr_fixup take struct dw_pcie as argument
  PCI: dwc: artpec6: Deassert the core before waiting for PHY
  bindings: PCI: artpec: Add support for the ARTPEC-7 SoC
  PCI: dwc: artpec6: Add support for the ARTPEC-7 SoC

 .../devicetree/bindings/pci/axis,artpec6-pcie.txt  |   5 +-
 drivers/pci/dwc/Kconfig                            |  68 +--
 drivers/pci/dwc/Makefile                           |   4 +-
 drivers/pci/dwc/pci-dra7xx.c                       |  27 +-
 drivers/pci/dwc/pcie-artpec6.c                     | 470 ++++++++++++++++++---
 drivers/pci/dwc/pcie-designware-ep.c               |  59 ++-
 drivers/pci/dwc/pcie-designware-host.c             |  15 +-
 drivers/pci/dwc/pcie-designware.c                  |   2 +-
 drivers/pci/dwc/pcie-designware.h                  |  22 +-
 9 files changed, 554 insertions(+), 118 deletions(-)

-- 
2.14.2

[PATCH v5 05/18] PCI: designware-ep: Remove static keyword from dw_pcie_ep_reset_bar()

[Niklas Cassel]

Remove the static keyword from dw_pcie_ep_reset_bar() so that
pci-dra7xx.c does not need its own copy of dw_pcie_ep_reset_bar().

Signed-off-by: Niklas Cassel <niklas.cassel@axis.com>
Acked-by: Kishon Vijay Abraham I <kishon@ti.com>
---
 drivers/pci/dwc/pci-dra7xx.c         | 9 ---------
 drivers/pci/dwc/pcie-designware-ep.c | 2 +-
 drivers/pci/dwc/pcie-designware.h    | 5 +++++
 3 files changed, 6 insertions(+), 10 deletions(-)

diff --git a/drivers/pci/dwc/pci-dra7xx.c b/drivers/pci/dwc/pci-dra7xx.c
index e77a4ceed74c..d330b7d86ce3 100644
--- a/drivers/pci/dwc/pci-dra7xx.c
+++ b/drivers/pci/dwc/pci-dra7xx.c
@@ -337,15 +337,6 @@ static irqreturn_t dra7xx_pcie_irq_handler(int irq, void *arg)
 	return IRQ_HANDLED;
 }
 
-static void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
-{
-	u32 reg;
-
-	reg = PCI_BASE_ADDRESS_0 + (4 * bar);
-	dw_pcie_writel_dbi2(pci, reg, 0x0);
-	dw_pcie_writel_dbi(pci, reg, 0x0);
-}
-
 static void dra7xx_pcie_ep_init(struct dw_pcie_ep *ep)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
diff --git a/drivers/pci/dwc/pcie-designware-ep.c b/drivers/pci/dwc/pcie-designware-ep.c
index 8d8019cdff2a..700ed2f4becf 100644
--- a/drivers/pci/dwc/pcie-designware-ep.c
+++ b/drivers/pci/dwc/pcie-designware-ep.c
@@ -30,7 +30,7 @@ void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
 	pci_epc_linkup(epc);
 }
 
-static void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
+void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
 {
 	u32 reg;
 
diff --git a/drivers/pci/dwc/pcie-designware.h b/drivers/pci/dwc/pcie-designware.h
index 5a1da459eda5..37dfad8d003f 100644
--- a/drivers/pci/dwc/pcie-designware.h
+++ b/drivers/pci/dwc/pcie-designware.h
@@ -338,6 +338,7 @@ static inline int dw_pcie_host_init(struct pcie_port *pp)
 void dw_pcie_ep_linkup(struct dw_pcie_ep *ep);
 int dw_pcie_ep_init(struct dw_pcie_ep *ep);
 void dw_pcie_ep_exit(struct dw_pcie_ep *ep);
+void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar);
 #else
 static inline void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
 {
@@ -351,5 +352,9 @@ static inline int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 static inline void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
 {
 }
+
+static inline void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
+{
+}
 #endif
 #endif /* _PCIE_DESIGNWARE_H */
-- 
2.14.2

[PATCH v5 08/18] PCI: dwc: dra7xx: Assign pp->ops in dra7xx_add_pcie_port() rather than in probe

[Niklas Cassel]

Assign pp->ops in *_add_pcie_port() to match how it is done in other
drivers like exynos, imx7, keystone, armada8k, artpec6, designware-plat,
hisi, kirin and spear13xx.

This is probably a remainder since when dev and ops were assigned as
members to pp. Since we now assign them as members to struct dw_pcie,
the pp->ops assignment should definitely be in dra7xx_add_pcie_port().

This is done so that the compiler (in a later commit) can remove more
code when enabling only one of the two supported modes (host/ep) in
the dra7xx driver.

Signed-off-by: Niklas Cassel <niklas.cassel@axis.com>
Acked-by: Kishon Vijay Abraham I <kishon@ti.com>
---
 drivers/pci/dwc/pci-dra7xx.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/pci/dwc/pci-dra7xx.c b/drivers/pci/dwc/pci-dra7xx.c
index d330b7d86ce3..07c74ae3614e 100644
--- a/drivers/pci/dwc/pci-dra7xx.c
+++ b/drivers/pci/dwc/pci-dra7xx.c
@@ -461,6 +461,8 @@ static int __init dra7xx_add_pcie_port(struct dra7xx_pcie *dra7xx,
 	if (!pci->dbi_base)
 		return -ENOMEM;
 
+	pp->ops = &dra7xx_pcie_host_ops;
+
 	ret = dw_pcie_host_init(pp);
 	if (ret) {
 		dev_err(dev, "failed to initialize host\n");
@@ -590,7 +592,6 @@ static int __init dra7xx_pcie_probe(struct platform_device *pdev)
 	void __iomem *base;
 	struct resource *res;
 	struct dw_pcie *pci;
-	struct pcie_port *pp;
 	struct dra7xx_pcie *dra7xx;
 	struct device *dev = &pdev->dev;
 	struct device_node *np = dev->of_node;
@@ -618,9 +619,6 @@ static int __init dra7xx_pcie_probe(struct platform_device *pdev)
 	pci->dev = dev;
 	pci->ops = &dw_pcie_ops;
 
-	pp = &pci->pp;
-	pp->ops = &dra7xx_pcie_host_ops;
-
 	irq = platform_get_irq(pdev, 0);
 	if (irq < 0) {
 		dev_err(dev, "missing IRQ resource: %d\n", irq);
-- 
2.14.2

[PATCH v5 09/18] PCI: dwc: dra7xx: Help compiler to remove unused code

[Niklas Cassel]

The dra7xx driver supports both host and ep mode.
When enabling support for only one of the modes, help the compiler
to remove code for the mode that we have not enabled in the driver.

By adding if (!IS_ENABLED(CONFIG_PCI_DRA7XX_HOST)) return -ENODEV;
anything after that statement will get silently dropped by the compiler,
including static functions and structures that are referenced indirectly
from there.

Suggested-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Niklas Cassel <niklas.cassel@axis.com>
---
 drivers/pci/dwc/pci-dra7xx.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/pci/dwc/pci-dra7xx.c b/drivers/pci/dwc/pci-dra7xx.c
index 07c74ae3614e..224ff8affdce 100644
--- a/drivers/pci/dwc/pci-dra7xx.c
+++ b/drivers/pci/dwc/pci-dra7xx.c
@@ -694,6 +694,11 @@ static int __init dra7xx_pcie_probe(struct platform_device *pdev)
 
 	switch (mode) {
 	case DW_PCIE_RC_TYPE:
+		if (!IS_ENABLED(CONFIG_PCI_DRA7XX_HOST)) {
+			ret = -ENODEV;
+			goto err_gpio;
+		}
+
 		dra7xx_pcie_writel(dra7xx, PCIECTRL_TI_CONF_DEVICE_TYPE,
 				   DEVICE_TYPE_RC);
 		ret = dra7xx_add_pcie_port(dra7xx, pdev);
@@ -701,6 +706,11 @@ static int __init dra7xx_pcie_probe(struct platform_device *pdev)
 			goto err_gpio;
 		break;
 	case DW_PCIE_EP_TYPE:
+		if (!IS_ENABLED(CONFIG_PCI_DRA7XX_EP)) {
+			ret = -ENODEV;
+			goto err_gpio;
+		}
+
 		dra7xx_pcie_writel(dra7xx, PCIECTRL_TI_CONF_DEVICE_TYPE,
 				   DEVICE_TYPE_EP);
 
-- 
2.14.2

[PATCH v5 15/18] PCI: dwc: Make cpu_addr_fixup take struct dw_pcie as argument

[Niklas Cassel]

There is no need to hard code the cpu to bus address fixup mask.
The PCIe controller has a global address on the AXI bus, however,
from the perspective of the PCIe controller, its base starts at 0x0,
so the local address is 0x0. To get the bus address, simply subtract
the global address from the cpu address. The global address is taken
from device tree.

Also for ARTPEC-7, hard coding the cpu to bus address fixup mask is
not possible, since it uses a High Address Bits Look Up Table, which
means that it can, at runtime, map the PCIe window to an arbitrary
address in the 32-bit address space.

This also fixes a bug for ARTPEC-6, where the cpu to bus address
fixup mask previously was off by one (GENMASK(27, 0), rather than
GENMASK(28, 0)). This is another reason to calculate the mask by
using values from device tree.

Signed-off-by: Niklas Cassel <niklas.cassel@axis.com>
---
 drivers/pci/dwc/pci-dra7xx.c      |  2 +-
 drivers/pci/dwc/pcie-artpec6.c    | 18 ++++++++++++++----
 drivers/pci/dwc/pcie-designware.c |  2 +-
 drivers/pci/dwc/pcie-designware.h |  2 +-
 4 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/drivers/pci/dwc/pci-dra7xx.c b/drivers/pci/dwc/pci-dra7xx.c
index 224ff8affdce..89d87844abb3 100644
--- a/drivers/pci/dwc/pci-dra7xx.c
+++ b/drivers/pci/dwc/pci-dra7xx.c
@@ -110,7 +110,7 @@ static inline void dra7xx_pcie_writel(struct dra7xx_pcie *pcie, u32 offset,
 	writel(value, pcie->base + offset);
 }
 
-static u64 dra7xx_pcie_cpu_addr_fixup(u64 pci_addr)
+static u64 dra7xx_pcie_cpu_addr_fixup(struct dw_pcie *pci, u64 pci_addr)
 {
 	return pci_addr & DRA7XX_CPU_TO_BUS_ADDR;
 }
diff --git a/drivers/pci/dwc/pcie-artpec6.c b/drivers/pci/dwc/pcie-artpec6.c
index e7de4e4649eb..318a2bd0d97e 100644
--- a/drivers/pci/dwc/pcie-artpec6.c
+++ b/drivers/pci/dwc/pcie-artpec6.c
@@ -67,8 +67,6 @@ static const struct of_device_id artpec6_pcie_of_match[];
 #define PHY_STATUS			0x118
 #define  PHY_COSPLLLOCK			BIT(0)
 
-#define ARTPEC6_CPU_TO_BUS_ADDR		GENMASK(27, 0)
-
 static u32 artpec6_pcie_readl(struct artpec6_pcie *artpec6_pcie, u32 offset)
 {
 	u32 val;
@@ -82,9 +80,21 @@ static void artpec6_pcie_writel(struct artpec6_pcie *artpec6_pcie, u32 offset, u
 	regmap_write(artpec6_pcie->regmap, offset, val);
 }
 
-static u64 artpec6_pcie_cpu_addr_fixup(u64 pci_addr)
+static u64 artpec6_pcie_cpu_addr_fixup(struct dw_pcie *pci, u64 pci_addr)
 {
-	return pci_addr & ARTPEC6_CPU_TO_BUS_ADDR;
+	struct artpec6_pcie *artpec6_pcie = to_artpec6_pcie(pci);
+	struct pcie_port *pp = &pci->pp;
+	struct dw_pcie_ep *ep = &pci->ep;
+
+	switch (artpec6_pcie->mode) {
+	case DW_PCIE_RC_TYPE:
+		return pci_addr - pp->cfg0_base;
+	case DW_PCIE_EP_TYPE:
+		return pci_addr - ep->phys_base;
+	default:
+		dev_err(pci->dev, "UNKNOWN device type\n");
+	}
+	return pci_addr;
 }
 
 static int artpec6_pcie_establish_link(struct dw_pcie *pci)
diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie-designware.c
index 88abdddee2ad..800be7a4f087 100644
--- a/drivers/pci/dwc/pcie-designware.c
+++ b/drivers/pci/dwc/pcie-designware.c
@@ -149,7 +149,7 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
 	u32 retries, val;
 
 	if (pci->ops->cpu_addr_fixup)
-		cpu_addr = pci->ops->cpu_addr_fixup(cpu_addr);
+		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
 
 	if (pci->iatu_unroll_enabled) {
 		dw_pcie_prog_outbound_atu_unroll(pci, index, type, cpu_addr,
diff --git a/drivers/pci/dwc/pcie-designware.h b/drivers/pci/dwc/pcie-designware.h
index 24edac035160..cca5a81c1c74 100644
--- a/drivers/pci/dwc/pcie-designware.h
+++ b/drivers/pci/dwc/pcie-designware.h
@@ -205,7 +205,7 @@ struct dw_pcie_ep {
 };
 
 struct dw_pcie_ops {
-	u64	(*cpu_addr_fixup)(u64 cpu_addr);
+	u64	(*cpu_addr_fixup)(struct dw_pcie *pcie, u64 cpu_addr);
 	u32	(*read_dbi)(struct dw_pcie *pcie, void __iomem *base, u32 reg,
 			    size_t size);
 	void	(*write_dbi)(struct dw_pcie *pcie, void __iomem *base, u32 reg,
-- 
2.14.2

Re: [PATCH v5 15/18] PCI: dwc: Make cpu_addr_fixup take struct dw_pcie as argument

[Lorenzo Pieralisi]

On Mon, Nov 20, 2017 at 02:32:18PM +0100, Niklas Cassel wrote:
> There is no need to hard code the cpu to bus address fixup mask.

There is no need or hardcoding it is broken ? There is a difference
between those two.

> The PCIe controller has a global address on the AXI bus, however,
> from the perspective of the PCIe controller, its base starts at 0x0,
> so the local address is 0x0. To get the bus address, simply subtract
> the global address from the cpu address. The global address is taken
> from device tree.

I do not understand this paragraph, I would kindly ask you and Kishon to
explain please this patch and

commit a660083eb06c ("PCI: dwc: designware: Add new *ops* for CPU addr
fixup")

What the cpu_addr_fixup() hook is supposed to do ? And what does the
"addr_space" property represent ?

> Also for ARTPEC-7, hard coding the cpu to bus address fixup mask is
> not possible, since it uses a High Address Bits Look Up Table, which
> means that it can, at runtime, map the PCIe window to an arbitrary
> address in the 32-bit address space.

But you are not changing the ARTPEC-7 mechanism, are you ? I do not
understand this paragraph - I see no change for ARTPEC-7 in this patch.

> This also fixes a bug for ARTPEC-6, where the cpu to bus address
> fixup mask previously was off by one (GENMASK(27, 0), rather than
> GENMASK(28, 0)). This is another reason to calculate the mask by
> using values from device tree.

What this patch does (and how the cpu_addr_fixup() mechanism works)
is not clear to me, please explain, we can rewrite the commit log
with a clear explanation then.

Thanks,
Lorenzo

> Signed-off-by: Niklas Cassel <niklas.cassel@axis.com>
> ---
>  drivers/pci/dwc/pci-dra7xx.c      |  2 +-
>  drivers/pci/dwc/pcie-artpec6.c    | 18 ++++++++++++++----
>  drivers/pci/dwc/pcie-designware.c |  2 +-
>  drivers/pci/dwc/pcie-designware.h |  2 +-
>  4 files changed, 17 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/pci/dwc/pci-dra7xx.c b/drivers/pci/dwc/pci-dra7xx.c
> index 224ff8affdce..89d87844abb3 100644
> --- a/drivers/pci/dwc/pci-dra7xx.c
> +++ b/drivers/pci/dwc/pci-dra7xx.c
> @@ -110,7 +110,7 @@ static inline void dra7xx_pcie_writel(struct dra7xx_pcie *pcie, u32 offset,
>  	writel(value, pcie->base + offset);
>  }
>  
> -static u64 dra7xx_pcie_cpu_addr_fixup(u64 pci_addr)
> +static u64 dra7xx_pcie_cpu_addr_fixup(struct dw_pcie *pci, u64 pci_addr)
>  {
>  	return pci_addr & DRA7XX_CPU_TO_BUS_ADDR;
>  }
> diff --git a/drivers/pci/dwc/pcie-artpec6.c b/drivers/pci/dwc/pcie-artpec6.c
> index e7de4e4649eb..318a2bd0d97e 100644
> --- a/drivers/pci/dwc/pcie-artpec6.c
> +++ b/drivers/pci/dwc/pcie-artpec6.c
> @@ -67,8 +67,6 @@ static const struct of_device_id artpec6_pcie_of_match[];
>  #define PHY_STATUS			0x118
>  #define  PHY_COSPLLLOCK			BIT(0)
>  
> -#define ARTPEC6_CPU_TO_BUS_ADDR		GENMASK(27, 0)
> -
>  static u32 artpec6_pcie_readl(struct artpec6_pcie *artpec6_pcie, u32 offset)
>  {
>  	u32 val;
> @@ -82,9 +80,21 @@ static void artpec6_pcie_writel(struct artpec6_pcie *artpec6_pcie, u32 offset, u
>  	regmap_write(artpec6_pcie->regmap, offset, val);
>  }
>  
> -static u64 artpec6_pcie_cpu_addr_fixup(u64 pci_addr)
> +static u64 artpec6_pcie_cpu_addr_fixup(struct dw_pcie *pci, u64 pci_addr)
>  {
> -	return pci_addr & ARTPEC6_CPU_TO_BUS_ADDR;
> +	struct artpec6_pcie *artpec6_pcie = to_artpec6_pcie(pci);
> +	struct pcie_port *pp = &pci->pp;
> +	struct dw_pcie_ep *ep = &pci->ep;
> +
> +	switch (artpec6_pcie->mode) {
> +	case DW_PCIE_RC_TYPE:
> +		return pci_addr - pp->cfg0_base;
> +	case DW_PCIE_EP_TYPE:
> +		return pci_addr - ep->phys_base;
> +	default:
> +		dev_err(pci->dev, "UNKNOWN device type\n");
> +	}
> +	return pci_addr;
>  }
>  
>  static int artpec6_pcie_establish_link(struct dw_pcie *pci)
> diff --git a/drivers/pci/dwc/pcie-designware.c b/drivers/pci/dwc/pcie-designware.c
> index 88abdddee2ad..800be7a4f087 100644
> --- a/drivers/pci/dwc/pcie-designware.c
> +++ b/drivers/pci/dwc/pcie-designware.c
> @@ -149,7 +149,7 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
>  	u32 retries, val;
>  
>  	if (pci->ops->cpu_addr_fixup)
> -		cpu_addr = pci->ops->cpu_addr_fixup(cpu_addr);
> +		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
>  
>  	if (pci->iatu_unroll_enabled) {
>  		dw_pcie_prog_outbound_atu_unroll(pci, index, type, cpu_addr,
> diff --git a/drivers/pci/dwc/pcie-designware.h b/drivers/pci/dwc/pcie-designware.h
> index 24edac035160..cca5a81c1c74 100644
> --- a/drivers/pci/dwc/pcie-designware.h
> +++ b/drivers/pci/dwc/pcie-designware.h
> @@ -205,7 +205,7 @@ struct dw_pcie_ep {
>  };
>  
>  struct dw_pcie_ops {
> -	u64	(*cpu_addr_fixup)(u64 cpu_addr);
> +	u64	(*cpu_addr_fixup)(struct dw_pcie *pcie, u64 cpu_addr);
>  	u32	(*read_dbi)(struct dw_pcie *pcie, void __iomem *base, u32 reg,
>  			    size_t size);
>  	void	(*write_dbi)(struct dw_pcie *pcie, void __iomem *base, u32 reg,
> -- 
> 2.14.2
> 

Re: [PATCH v5 15/18] PCI: dwc: Make cpu_addr_fixup take struct dw_pcie as argument

[Niklas Cassel]

On Mon, Dec 18, 2017 at 06:10:58PM +0000, Lorenzo Pieralisi wrote:
> On Mon, Nov 20, 2017 at 02:32:18PM +0100, Niklas Cassel wrote:
> > There is no need to hard code the cpu to bus address fixup mask.
> 
> There is no need or hardcoding it is broken ? There is a difference
> between those two.

Well, both :P

The current hardcoding for ARTPEC-6: GENMASK(27, 0), is wrong.
Hardcoding the correct mask GENMASK(28, 0) would be sufficient
for ARTPEC-6.

However, the ARTPEC-7 is a 32-bit CPU (without LPAE), but has
a 64-bit bus. So the ARTPEC-7 can have more devices than fits
in the 32-bit range, therefore a lookup table is placed between
the bus and the CPU, so you can choose what devices to map.
This mapping, which decides what devices to map, is currently
done at boot via devicetree.
So, on ARTPEC-7, it is not possible to hardcode a CPU_TO_BUS
mask, since the PCIe controller's address is only known via DT.

Having a hardcoded value is arguably wrong. It should probably
have been a DT property called something like "cpu-bus-fixup-mask".
However, this property is not needed since we already have
pp->cfg0_base and ep->phys_base, which is derived from already
existing DT properties.

By using pp->cfg0_base and ep->phys_base, we avoid hardcoding a mask
in the driver. This should work for ARTPEC-6, DRA7xx and ARTPEC-7.
I have not changed the code in DRA7xx though, since their existing
code works, but if they want, they could use the same logic as
artpec6_pcie_cpu_addr_fixup, and thus remove their hardcoded mask.

> 
> > The PCIe controller has a global address on the AXI bus, however,
> > from the perspective of the PCIe controller, its base starts at 0x0,
> > so the local address is 0x0. To get the bus address, simply subtract
> > the global address from the cpu address. The global address is taken
> > from device tree.
> 
> I do not understand this paragraph, I would kindly ask you and Kishon to
> explain please this patch and
> 
> commit a660083eb06c ("PCI: dwc: designware: Add new *ops* for CPU addr
> fixup")
> 
> What the cpu_addr_fixup() hook is supposed to do ? And what does the
> "addr_space" property represent ?

The ARTPEC-6 and DRA7xx SoCs both has an interconnect configured is a way
where the local address of the PCIe controller is 0x0.

For ARTPEC-6 the global address of the PCIe controller is 0xf8050000,
so if the CPU writes to address 0xf8050008, from the perspective of the
PCIe controller, it will see a write to address 0x8.

This is normally not a problem, but when reading/writing from an endpoint,
we go via the outbound iATU. The outbound iATU has to be programmed with
a base address (reads/writes in the range [base address + limit] will be
subject to translation).

However, since the address the PCIe controller sees, in reality is
"0xf8050000 + the address the PCIe controller sees", we need to subtract
the global address before programming the base address in the outbound
iATU.

Kishon explains the same thing in commit f4c55c5a3f7f68c0.

For this patch, I tried make the commit message understandable,
without going into too much detail, but there is probably still
room for improvement.

> 
> > Also for ARTPEC-7, hard coding the cpu to bus address fixup mask is
> > not possible, since it uses a High Address Bits Look Up Table, which
> > means that it can, at runtime, map the PCIe window to an arbitrary
> > address in the 32-bit address space.
> 
> But you are not changing the ARTPEC-7 mechanism, are you ? I do not
> understand this paragraph - I see no change for ARTPEC-7 in this patch.

Hopefully this is clarified by the comments above.

> 
> > This also fixes a bug for ARTPEC-6, where the cpu to bus address
> > fixup mask previously was off by one (GENMASK(27, 0), rather than
> > GENMASK(28, 0)). This is another reason to calculate the mask by
> > using values from device tree.
> 
> What this patch does (and how the cpu_addr_fixup() mechanism works)
> is not clear to me, please explain, we can rewrite the commit log
> with a clear explanation then.

Hopefully this is clarified by the comments above.


Regards,
Niklas

Re: [PATCH v5 15/18] PCI: dwc: Make cpu_addr_fixup take struct dw_pcie as argument

[Lorenzo Pieralisi]

On Mon, Dec 18, 2017 at 10:15:43PM +0100, Niklas Cassel wrote:
> On Mon, Dec 18, 2017 at 06:10:58PM +0000, Lorenzo Pieralisi wrote:
> > On Mon, Nov 20, 2017 at 02:32:18PM +0100, Niklas Cassel wrote:
> > > There is no need to hard code the cpu to bus address fixup mask.
> > 
> > There is no need or hardcoding it is broken ? There is a difference
> > between those two.
> 
> Well, both :P
> 
> The current hardcoding for ARTPEC-6: GENMASK(27, 0), is wrong.
> Hardcoding the correct mask GENMASK(28, 0) would be sufficient
> for ARTPEC-6.
> 
> However, the ARTPEC-7 is a 32-bit CPU (without LPAE), but has
> a 64-bit bus. So the ARTPEC-7 can have more devices than fits
> in the 32-bit range, therefore a lookup table is placed between
> the bus and the CPU, so you can choose what devices to map.
> This mapping, which decides what devices to map, is currently
> done at boot via devicetree.
> So, on ARTPEC-7, it is not possible to hardcode a CPU_TO_BUS
> mask, since the PCIe controller's address is only known via DT.
> 
> Having a hardcoded value is arguably wrong. It should probably
> have been a DT property called something like "cpu-bus-fixup-mask".
> However, this property is not needed since we already have
> pp->cfg0_base and ep->phys_base, which is derived from already
> existing DT properties.
> 
> By using pp->cfg0_base and ep->phys_base, we avoid hardcoding a mask
> in the driver. This should work for ARTPEC-6, DRA7xx and ARTPEC-7.
> I have not changed the code in DRA7xx though, since their existing
> code works, but if they want, they could use the same logic as
> artpec6_pcie_cpu_addr_fixup, and thus remove their hardcoded mask.
> 
> > 
> > > The PCIe controller has a global address on the AXI bus, however,
> > > from the perspective of the PCIe controller, its base starts at 0x0,
> > > so the local address is 0x0. To get the bus address, simply subtract
> > > the global address from the cpu address. The global address is taken
> > > from device tree.
> > 
> > I do not understand this paragraph, I would kindly ask you and Kishon to
> > explain please this patch and
> > 
> > commit a660083eb06c ("PCI: dwc: designware: Add new *ops* for CPU addr
> > fixup")
> > 
> > What the cpu_addr_fixup() hook is supposed to do ? And what does the
> > "addr_space" property represent ?
> 
> The ARTPEC-6 and DRA7xx SoCs both has an interconnect configured is a way
> where the local address of the PCIe controller is 0x0.
> 
> For ARTPEC-6 the global address of the PCIe controller is 0xf8050000,
> so if the CPU writes to address 0xf8050008, from the perspective of the
> PCIe controller, it will see a write to address 0x8.
> 
> This is normally not a problem, but when reading/writing from an endpoint,
> we go via the outbound iATU. The outbound iATU has to be programmed with
> a base address (reads/writes in the range [base address + limit] will be
> subject to translation).
> 
> However, since the address the PCIe controller sees, in reality is
> "0xf8050000 + the address the PCIe controller sees", we need to subtract
> the global address before programming the base address in the outbound
> iATU.
> 
> Kishon explains the same thing in commit f4c55c5a3f7f68c0.
> 
> For this patch, I tried make the commit message understandable,
> without going into too much detail, but there is probably still
> room for improvement.
> 
> > 
> > > Also for ARTPEC-7, hard coding the cpu to bus address fixup mask is
> > > not possible, since it uses a High Address Bits Look Up Table, which
> > > means that it can, at runtime, map the PCIe window to an arbitrary
> > > address in the 32-bit address space.
> > 
> > But you are not changing the ARTPEC-7 mechanism, are you ? I do not
> > understand this paragraph - I see no change for ARTPEC-7 in this patch.
> 
> Hopefully this is clarified by the comments above.
> 
> > 
> > > This also fixes a bug for ARTPEC-6, where the cpu to bus address
> > > fixup mask previously was off by one (GENMASK(27, 0), rather than
> > > GENMASK(28, 0)). This is another reason to calculate the mask by
> > > using values from device tree.
> > 
> > What this patch does (and how the cpu_addr_fixup() mechanism works)
> > is not clear to me, please explain, we can rewrite the commit log
> > with a clear explanation then.
> 
> Hopefully this is clarified by the comments above.

It is, thank you - it would be good to convert this thread into commit
log format, you can send it as an update to this patch.

Thanks,
Lorenzo