[PATCH v15,RESEND 00/23] Refactoring Microchip PCIe driver and add StarFive PCIe

[PATCH v15,RESEND 00/23] Refactoring Microchip PCIe driver and add StarFive PCIe

[Minda Chen]

This patchset final purpose is add PCIe driver for StarFive JH7110 SoC.
JH7110 using PLDA XpressRICH PCIe IP. Microchip PolarFire Using the
same IP and have commit their codes, which are mixed with PLDA
controller codes and Microchip platform codes.

For re-use the PLDA controller codes, I request refactoring microchip
codes, move PLDA common codes to PLDA files.
Desigware and Cadence is good example for refactoring codes.

----------------------------------------------------------
The refactoring patches total number is 16,(patch 1-16)
which do NOT contain changing logic of codes.

These patches just contain three type basic operations.
(rename, modify codes to support starfive platform, and moving to common file)
If these patched are all be reviewed. They can be accepted first.

Refactoring patches can be devided to different groups
1. (patch 1- 3 is the prepare work of refactoring)
patch1 is move PLDA XpressRICH PCIe host common properties dt-binding
       docs from microchip,pcie-host.yaml
patch2 is move PolarFire codes to PLDA directory.
patch3 is move PLDA IP register macros to plda-pcie.h

2. (patch4 - 6 is processing and re-use PCIe host instance)
patch4 is add bridge_addr field to PCIe host instance.
patch5 is rename data structure in microchip codes.
patch6 is moving two data structures to head file

3. (patch 7 - 9 are for re-use two PCIe setup function)
patch7 is rename two setup functions in microchip codes, prepare to move
to common file.
patch8 is change the arguments of plda_pcie_setup_iomems()
patch9 is move the two setup functions to common file pcie-plda-host.c

4.(patch 10 - 16 are for re-use interupt processing codes)
patch10 is rename the IRQ related functions, prepare to move to
pcie-plda-host.c
patch 11 - 15 is modify the interrupt event codes, preparing for support starfive
and microchip two platforms.
patch16 is move IRQ related functions to pcie-plda-host.c

------------------------------------------------------------
The remainder patches (patch 17 -22) are not refactoring patch.
They are for adding StarFive codes and dont modify the microchip's
codes.

patch17 is Add event bitmap for mask unused/unimpementes interrupts
patch18 is Add host init/deinit functions.
patch19 is add StarFive JH7110 PCIe dt-binding doc.
patch20 is Add a PCIe delay time macro
patch21 is add StarFive JH7110 Soc PCIe codes.
patch22 is fix some nvme SSD timeout issue. (A workaround to JH7110 PCIe
qurik).
patch23 is Starfive dts config

This patchset is base on v6.8-rc6

previous version:
v9:https://patchwork.kernel.org/project/linux-pci/cover/20231020104341.63157-1-minda.chen@starfivetech.com/
v10:https://patchwork.kernel.org/project/linux-pci/cover/20231031115430.113586-1-minda.chen@starfivetech.com/
v11:https://patchwork.kernel.org/project/linux-pci/cover/20231115114912.71448-1-minda.chen@starfivetech.com/
v12:https://patchwork.kernel.org/project/linux-pci/cover/20231206105839.25805-1-minda.chen@starfivetech.com/
v13:https://patchwork.kernel.org/project/linux-pci/cover/20231214072839.2367-1-minda.chen@starfivetech.com/
v14:https://patchwork.kernel.org/project/linux-pci/cover/20240129005854.3554-1-minda.chen@starfivetech.com/

change:
  v15: 
     resend: rebase on v6.8-rc6
     Add new patch 22.(A workaround to JH7110 PCIe quirk).
  v14:
   resend: correct some commit messages. base on v6.8-rc1
   Add a new patch 17. Add interrupt eventmap to mask non-implemented or unused interrupt.
   patch3: rename the PLDA interrupts macro name.
   patch9: remove the redundant head file.
   patch10,11: modify the commit message suggested by Lorrenzo.
   patch12: modify the commit message.
   patch14, 15: PLDA codes will handle DMA interrupts. Modify the commit message. 
   patch18 remove plda default events. This is implemented by vendor.
   patch21 Add plda_event instance stf_pcie_event.

  v13:
   patch14: 1. Add plda_get_event() function. This can be compare with mc_get_event() easily and 
		track the codes changes in case in the future..
            2. The host event_ops is directly set to plda host port.
	    3. Setting default host event_ops instead of checking event ops.
   patch15:1. Add PLDA event irq_chip instead of event_domain_ops, The
		event_domain_ops can be re-used.
            2. The host event irq_chip is directly set to plda host port.
	    3. Add PLDA event irqchip ops codes.
            4. Remove Conor's review tag due to whole patch have been changed.
   patch16: Also move the new added PLDA event codes.
   patch18: Add plda host init and deinit function only.

  v12:
   patch18: modify the commit message and add starfive review tag.
   Add PCIE_RESET_CONFIG_DEVICE_WAIT_MS to patch 19.
   patch21: Add disable runtime pm function in starfive_pcie_remove()
            Add "depens on ARCH_STARFIVE || COMPILE_TEST" in Starfive PCie Kconfig

  v11:
     check and modify some commit messages again.
     All the codes are the same with v10.   

  v10:
   All the commit message set to fit in 75 columns.
   All the codes fit in less than 80 colunms.
   patch 14: 
	Commit message changes suggested by Conor.
   patch 21:
        Add 100 ms delay macro to pci.h
	generic phy pointer related codes moving to pcie-starfive.c
	This patch Change pcie-starfive only, bus_ops move to patch 16.
	Some Codes changes suggested by Bjorn.

  v9:
   v8 patch 10 squash to v9 patch 12, v8 patch 18 squash to v9 patch 16.
   patch 4 - 16: Add new review tags and add more accurate commit messages.
   patch 18: move the plda_pcie_host_init/deinit from patch 19. Make
             plda driver become to whole driver.


*** BLURB HERE ***

Kevin Xie (2):
  PCI: Add PCIE_RESET_CONFIG_DEVICE_WAIT_MS waiting time value
  PCI: starfive: Offload the NVMe timeout workaround to host drivers.

Minda Chen (21):
  dt-bindings: PCI: Add PLDA XpressRICH PCIe host common properties
  PCI: microchip: Move pcie-microchip-host.c to plda directory
  PCI: microchip: Move PLDA IP register macros to pcie-plda.h
  PCI: microchip: Add bridge_addr field to struct mc_pcie
  PCI: microchip: Rename two PCIe data structures
  PCI: microchip: Move PCIe host data structures to plda-pcie.h
  PCI: microchip: Rename two setup functions
  PCI: microchip: Change the argument of plda_pcie_setup_iomems()
  PCI: microchip: Move setup functions to pcie-plda-host.c
  PCI: microchip: Rename interrupt related functions
  PCI: microchip: Add num_events field to struct plda_pcie_rp
  PCI: microchip: Add request_event_irq() callback function
  PCI: microchip: Add INTx and MSI event num to struct plda_event
  PCI: microchip: Add get_events() callback and add PLDA get_event()
  PCI: microchip: Add event irqchip field to host port and add PLDA
    irqchip
  PCI: microchip: Move IRQ functions to pcie-plda-host.c
  PCI: plda: Add event bitmap field to struct plda_pcie_rp
  PCI: plda: Add host init/deinit and map bus functions
  dt-bindings: PCI: Add StarFive JH7110 PCIe controller
  PCI: starfive: Add JH7110 PCIe controller
  riscv: dts: starfive: add PCIe dts configuration for JH7110

 .../bindings/pci/microchip,pcie-host.yaml     |  55 +-
 .../pci/plda,xpressrich3-axi-common.yaml      |  75 ++
 .../bindings/pci/starfive,jh7110-pcie.yaml    | 120 ++++
 MAINTAINERS                                   |  19 +-
 .../jh7110-starfive-visionfive-2.dtsi         |  64 ++
 arch/riscv/boot/dts/starfive/jh7110.dtsi      |  86 +++
 drivers/pci/controller/Kconfig                |   9 +-
 drivers/pci/controller/Makefile               |   2 +-
 drivers/pci/controller/plda/Kconfig           |  30 +
 drivers/pci/controller/plda/Makefile          |   4 +
 .../{ => plda}/pcie-microchip-host.c          | 615 ++--------------
 drivers/pci/controller/plda/pcie-plda-host.c  | 663 ++++++++++++++++++
 drivers/pci/controller/plda/pcie-plda.h       | 273 ++++++++
 drivers/pci/controller/plda/pcie-starfive.c   | 474 +++++++++++++
 drivers/pci/pci.h                             |  16 +
 15 files changed, 1889 insertions(+), 616 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/pci/plda,xpressrich3-axi-common.yaml
 create mode 100644 Documentation/devicetree/bindings/pci/starfive,jh7110-pcie.yaml
 create mode 100644 drivers/pci/controller/plda/Kconfig
 create mode 100644 drivers/pci/controller/plda/Makefile
 rename drivers/pci/controller/{ => plda}/pcie-microchip-host.c (54%)
 create mode 100644 drivers/pci/controller/plda/pcie-plda-host.c
 create mode 100644 drivers/pci/controller/plda/pcie-plda.h
 create mode 100644 drivers/pci/controller/plda/pcie-starfive.c


base-commit: d206a76d7d2726f3b096037f2079ce0bd3ba329b
-- 
2.17.1

[PATCH v15,RESEND 01/23] dt-bindings: PCI: Add PLDA XpressRICH PCIe host common properties

[Minda Chen]

Add PLDA XpressRICH PCIe host common properties dt-binding doc.
PolarFire PCIe host using PLDA IP. Move common properties from Microchip
PolarFire PCIe host to PLDA files.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Hal Feng <hal.feng@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
Reviewed-by: Rob Herring <robh@kernel.org>
Tested-by: John Clark <inindev@gmail.com>
---
 .../bindings/pci/microchip,pcie-host.yaml     | 55 +-------------
 .../pci/plda,xpressrich3-axi-common.yaml      | 75 +++++++++++++++++++
 2 files changed, 76 insertions(+), 54 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/pci/plda,xpressrich3-axi-common.yaml

diff --git a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
index f7a3c2636355..7c2d51221f65 100644
--- a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
+++ b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
@@ -10,21 +10,13 @@ maintainers:
   - Daire McNamara <daire.mcnamara@microchip.com>
 
 allOf:
-  - $ref: /schemas/pci/pci-bus.yaml#
+  - $ref: plda,xpressrich3-axi-common.yaml#
   - $ref: /schemas/interrupt-controller/msi-controller.yaml#
 
 properties:
   compatible:
     const: microchip,pcie-host-1.0 # PolarFire
 
-  reg:
-    maxItems: 2
-
-  reg-names:
-    items:
-      - const: cfg
-      - const: apb
-
   clocks:
     description:
       Fabric Interface Controllers, FICs, are the interface between the FPGA
@@ -52,18 +44,6 @@ properties:
     items:
       pattern: '^fic[0-3]$'
 
-  interrupts:
-    minItems: 1
-    items:
-      - description: PCIe host controller
-      - description: builtin MSI controller
-
-  interrupt-names:
-    minItems: 1
-    items:
-      - const: pcie
-      - const: msi
-
   ranges:
     maxItems: 1
 
@@ -71,39 +51,6 @@ properties:
     minItems: 1
     maxItems: 6
 
-  msi-controller:
-    description: Identifies the node as an MSI controller.
-
-  msi-parent:
-    description: MSI controller the device is capable of using.
-
-  interrupt-controller:
-    type: object
-    properties:
-      '#address-cells':
-        const: 0
-
-      '#interrupt-cells':
-        const: 1
-
-      interrupt-controller: true
-
-    required:
-      - '#address-cells'
-      - '#interrupt-cells'
-      - interrupt-controller
-
-    additionalProperties: false
-
-required:
-  - reg
-  - reg-names
-  - "#interrupt-cells"
-  - interrupts
-  - interrupt-map-mask
-  - interrupt-map
-  - msi-controller
-
 unevaluatedProperties: false
 
 examples:
diff --git a/Documentation/devicetree/bindings/pci/plda,xpressrich3-axi-common.yaml b/Documentation/devicetree/bindings/pci/plda,xpressrich3-axi-common.yaml
new file mode 100644
index 000000000000..31bb17b11e58
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/plda,xpressrich3-axi-common.yaml
@@ -0,0 +1,75 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/pci/plda,xpressrich3-axi-common.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: PLDA XpressRICH PCIe host common properties
+
+maintainers:
+  - Daire McNamara <daire.mcnamara@microchip.com>
+  - Kevin Xie <kevin.xie@starfivetech.com>
+
+description:
+  Generic PLDA XpressRICH PCIe host common properties.
+
+allOf:
+  - $ref: /schemas/pci/pci-bus.yaml#
+
+properties:
+  reg:
+    maxItems: 2
+
+  reg-names:
+    items:
+      - const: cfg
+      - const: apb
+
+  interrupts:
+    minItems: 1
+    items:
+      - description: PCIe host controller
+      - description: builtin MSI controller
+
+  interrupt-names:
+    minItems: 1
+    items:
+      - const: pcie
+      - const: msi
+
+  msi-controller:
+    description: Identifies the node as an MSI controller.
+
+  msi-parent:
+    description: MSI controller the device is capable of using.
+
+  interrupt-controller:
+    type: object
+    properties:
+      '#address-cells':
+        const: 0
+
+      '#interrupt-cells':
+        const: 1
+
+      interrupt-controller: true
+
+    required:
+      - '#address-cells'
+      - '#interrupt-cells'
+      - interrupt-controller
+
+    additionalProperties: false
+
+required:
+  - reg
+  - reg-names
+  - interrupts
+  - msi-controller
+  - "#interrupt-cells"
+  - interrupt-map-mask
+  - interrupt-map
+
+additionalProperties: true
+
+...
-- 
2.17.1

[PATCH v15,RESEND 02/23] PCI: microchip: Move pcie-microchip-host.c to plda directory

[Minda Chen]

For Microchip Polarfire PCIe host is PLDA XpressRich IP, move to plda
directory. Prepare for refactoring the codes.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 MAINTAINERS                                        |  4 ++--
 drivers/pci/controller/Kconfig                     |  9 +--------
 drivers/pci/controller/Makefile                    |  2 +-
 drivers/pci/controller/plda/Kconfig                | 14 ++++++++++++++
 drivers/pci/controller/plda/Makefile               |  2 ++
 .../controller/{ => plda}/pcie-microchip-host.c    |  2 +-
 6 files changed, 21 insertions(+), 12 deletions(-)
 create mode 100644 drivers/pci/controller/plda/Kconfig
 create mode 100644 drivers/pci/controller/plda/Makefile
 rename drivers/pci/controller/{ => plda}/pcie-microchip-host.c (99%)

diff --git a/MAINTAINERS b/MAINTAINERS
index 2ecaaec6a6bf..9f7e907c8749 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17046,7 +17046,7 @@ M:	Daire McNamara <daire.mcnamara@microchip.com>
 L:	linux-pci@vger.kernel.org
 S:	Supported
 F:	Documentation/devicetree/bindings/pci/microchip*
-F:	drivers/pci/controller/*microchip*
+F:	drivers/pci/controller/plda/*microchip*
 
 PCIE DRIVER FOR QUALCOMM MSM
 M:	Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
@@ -18838,7 +18838,7 @@ F:	drivers/clk/microchip/clk-mpfs*.c
 F:	drivers/firmware/microchip/mpfs-auto-update.c
 F:	drivers/i2c/busses/i2c-microchip-corei2c.c
 F:	drivers/mailbox/mailbox-mpfs.c
-F:	drivers/pci/controller/pcie-microchip-host.c
+F:	drivers/pci/controller/plda/pcie-microchip-host.c
 F:	drivers/pwm/pwm-microchip-core.c
 F:	drivers/reset/reset-mpfs.c
 F:	drivers/rtc/rtc-mpfs.c
diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig
index e534c02ee34f..4d2c188f5835 100644
--- a/drivers/pci/controller/Kconfig
+++ b/drivers/pci/controller/Kconfig
@@ -215,14 +215,6 @@ config PCIE_MT7621
 	help
 	  This selects a driver for the MediaTek MT7621 PCIe Controller.
 
-config PCIE_MICROCHIP_HOST
-	tristate "Microchip AXI PCIe controller"
-	depends on PCI_MSI && OF
-	select PCI_HOST_COMMON
-	help
-	  Say Y here if you want kernel to support the Microchip AXI PCIe
-	  Host Bridge driver.
-
 config PCI_HYPERV_INTERFACE
 	tristate "Microsoft Hyper-V PCI Interface"
 	depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI
@@ -356,4 +348,5 @@ config PCIE_XILINX_CPM
 source "drivers/pci/controller/cadence/Kconfig"
 source "drivers/pci/controller/dwc/Kconfig"
 source "drivers/pci/controller/mobiveil/Kconfig"
+source "drivers/pci/controller/plda/Kconfig"
 endmenu
diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile
index f2b19e6174af..038ccbd9e3ba 100644
--- a/drivers/pci/controller/Makefile
+++ b/drivers/pci/controller/Makefile
@@ -33,7 +33,6 @@ obj-$(CONFIG_PCIE_ROCKCHIP_EP) += pcie-rockchip-ep.o
 obj-$(CONFIG_PCIE_ROCKCHIP_HOST) += pcie-rockchip-host.o
 obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o
 obj-$(CONFIG_PCIE_MEDIATEK_GEN3) += pcie-mediatek-gen3.o
-obj-$(CONFIG_PCIE_MICROCHIP_HOST) += pcie-microchip-host.o
 obj-$(CONFIG_VMD) += vmd.o
 obj-$(CONFIG_PCIE_BRCMSTB) += pcie-brcmstb.o
 obj-$(CONFIG_PCI_LOONGSON) += pci-loongson.o
@@ -44,6 +43,7 @@ obj-$(CONFIG_PCIE_MT7621) += pcie-mt7621.o
 # pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW
 obj-y				+= dwc/
 obj-y				+= mobiveil/
+obj-y				+= plda/
 
 
 # The following drivers are for devices that use the generic ACPI
diff --git a/drivers/pci/controller/plda/Kconfig b/drivers/pci/controller/plda/Kconfig
new file mode 100644
index 000000000000..5cb3be4fc98c
--- /dev/null
+++ b/drivers/pci/controller/plda/Kconfig
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0
+
+menu "PLDA-based PCIe controllers"
+	depends on PCI
+
+config PCIE_MICROCHIP_HOST
+	tristate "Microchip AXI PCIe controller"
+	depends on PCI_MSI && OF
+	select PCI_HOST_COMMON
+	help
+	  Say Y here if you want kernel to support the Microchip AXI PCIe
+	  Host Bridge driver.
+
+endmenu
diff --git a/drivers/pci/controller/plda/Makefile b/drivers/pci/controller/plda/Makefile
new file mode 100644
index 000000000000..e1a265cbf91c
--- /dev/null
+++ b/drivers/pci/controller/plda/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_PCIE_MICROCHIP_HOST) += pcie-microchip-host.o
diff --git a/drivers/pci/controller/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
similarity index 99%
rename from drivers/pci/controller/pcie-microchip-host.c
rename to drivers/pci/controller/plda/pcie-microchip-host.c
index 137fb8570ba2..cb09a8137e25 100644
--- a/drivers/pci/controller/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -18,7 +18,7 @@
 #include <linux/pci-ecam.h>
 #include <linux/platform_device.h>
 
-#include "../pci.h"
+#include "../../pci.h"
 
 /* Number of MSI IRQs */
 #define MC_MAX_NUM_MSI_IRQS			32
-- 
2.17.1

[PATCH v15,RESEND 03/23] PCI: microchip: Move PLDA IP register macros to pcie-plda.h

[Minda Chen]

Move PLDA PCIe host controller IP registers macros to pcie-plda.h,
including bridge registers and PLDA IRQ event number.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 MAINTAINERS                                   |   8 ++
 .../pci/controller/plda/pcie-microchip-host.c | 108 +++---------------
 drivers/pci/controller/plda/pcie-plda.h       | 108 ++++++++++++++++++
 3 files changed, 132 insertions(+), 92 deletions(-)
 create mode 100644 drivers/pci/controller/plda/pcie-plda.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 9f7e907c8749..fc9576e69a71 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -16814,6 +16814,14 @@ S:	Maintained
 F:	Documentation/devicetree/bindings/pci/layerscape-pcie-gen4.txt
 F:	drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
 
+PCI DRIVER FOR PLDA PCIE IP
+M:	Daire McNamara <daire.mcnamara@microchip.com>
+M:	Kevin Xie <kevin.xie@starfivetech.com>
+L:	linux-pci@vger.kernel.org
+S:	Maintained
+F:	Documentation/devicetree/bindings/pci/plda,*
+F:	drivers/pci/controller/plda/*plda*
+
 PCI DRIVER FOR RENESAS R-CAR
 M:	Marek Vasut <marek.vasut+renesas@gmail.com>
 M:	Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index cb09a8137e25..d9030d550482 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -19,6 +19,7 @@
 #include <linux/platform_device.h>
 
 #include "../../pci.h"
+#include "pcie-plda.h"
 
 /* Number of MSI IRQs */
 #define MC_MAX_NUM_MSI_IRQS			32
@@ -30,84 +31,6 @@
 #define MC_PCIE_BRIDGE_ADDR			(MC_PCIE1_BRIDGE_ADDR)
 #define MC_PCIE_CTRL_ADDR			(MC_PCIE1_CTRL_ADDR)
 
-/* PCIe Bridge Phy Regs */
-#define PCIE_PCI_IRQ_DW0			0xa8
-#define  MSIX_CAP_MASK				BIT(31)
-#define  NUM_MSI_MSGS_MASK			GENMASK(6, 4)
-#define  NUM_MSI_MSGS_SHIFT			4
-
-#define IMASK_LOCAL				0x180
-#define  DMA_END_ENGINE_0_MASK			0x00000000u
-#define  DMA_END_ENGINE_0_SHIFT			0
-#define  DMA_END_ENGINE_1_MASK			0x00000000u
-#define  DMA_END_ENGINE_1_SHIFT			1
-#define  DMA_ERROR_ENGINE_0_MASK		0x00000100u
-#define  DMA_ERROR_ENGINE_0_SHIFT		8
-#define  DMA_ERROR_ENGINE_1_MASK		0x00000200u
-#define  DMA_ERROR_ENGINE_1_SHIFT		9
-#define  A_ATR_EVT_POST_ERR_MASK		0x00010000u
-#define  A_ATR_EVT_POST_ERR_SHIFT		16
-#define  A_ATR_EVT_FETCH_ERR_MASK		0x00020000u
-#define  A_ATR_EVT_FETCH_ERR_SHIFT		17
-#define  A_ATR_EVT_DISCARD_ERR_MASK		0x00040000u
-#define  A_ATR_EVT_DISCARD_ERR_SHIFT		18
-#define  A_ATR_EVT_DOORBELL_MASK		0x00000000u
-#define  A_ATR_EVT_DOORBELL_SHIFT		19
-#define  P_ATR_EVT_POST_ERR_MASK		0x00100000u
-#define  P_ATR_EVT_POST_ERR_SHIFT		20
-#define  P_ATR_EVT_FETCH_ERR_MASK		0x00200000u
-#define  P_ATR_EVT_FETCH_ERR_SHIFT		21
-#define  P_ATR_EVT_DISCARD_ERR_MASK		0x00400000u
-#define  P_ATR_EVT_DISCARD_ERR_SHIFT		22
-#define  P_ATR_EVT_DOORBELL_MASK		0x00000000u
-#define  P_ATR_EVT_DOORBELL_SHIFT		23
-#define  PM_MSI_INT_INTA_MASK			0x01000000u
-#define  PM_MSI_INT_INTA_SHIFT			24
-#define  PM_MSI_INT_INTB_MASK			0x02000000u
-#define  PM_MSI_INT_INTB_SHIFT			25
-#define  PM_MSI_INT_INTC_MASK			0x04000000u
-#define  PM_MSI_INT_INTC_SHIFT			26
-#define  PM_MSI_INT_INTD_MASK			0x08000000u
-#define  PM_MSI_INT_INTD_SHIFT			27
-#define  PM_MSI_INT_INTX_MASK			0x0f000000u
-#define  PM_MSI_INT_INTX_SHIFT			24
-#define  PM_MSI_INT_MSI_MASK			0x10000000u
-#define  PM_MSI_INT_MSI_SHIFT			28
-#define  PM_MSI_INT_AER_EVT_MASK		0x20000000u
-#define  PM_MSI_INT_AER_EVT_SHIFT		29
-#define  PM_MSI_INT_EVENTS_MASK			0x40000000u
-#define  PM_MSI_INT_EVENTS_SHIFT		30
-#define  PM_MSI_INT_SYS_ERR_MASK		0x80000000u
-#define  PM_MSI_INT_SYS_ERR_SHIFT		31
-#define  NUM_LOCAL_EVENTS			15
-#define ISTATUS_LOCAL				0x184
-#define IMASK_HOST				0x188
-#define ISTATUS_HOST				0x18c
-#define IMSI_ADDR				0x190
-#define ISTATUS_MSI				0x194
-
-/* PCIe Master table init defines */
-#define ATR0_PCIE_WIN0_SRCADDR_PARAM		0x600u
-#define  ATR0_PCIE_ATR_SIZE			0x25
-#define  ATR0_PCIE_ATR_SIZE_SHIFT		1
-#define ATR0_PCIE_WIN0_SRC_ADDR			0x604u
-#define ATR0_PCIE_WIN0_TRSL_ADDR_LSB		0x608u
-#define ATR0_PCIE_WIN0_TRSL_ADDR_UDW		0x60cu
-#define ATR0_PCIE_WIN0_TRSL_PARAM		0x610u
-
-/* PCIe AXI slave table init defines */
-#define ATR0_AXI4_SLV0_SRCADDR_PARAM		0x800u
-#define  ATR_SIZE_SHIFT				1
-#define  ATR_IMPL_ENABLE			1
-#define ATR0_AXI4_SLV0_SRC_ADDR			0x804u
-#define ATR0_AXI4_SLV0_TRSL_ADDR_LSB		0x808u
-#define ATR0_AXI4_SLV0_TRSL_ADDR_UDW		0x80cu
-#define ATR0_AXI4_SLV0_TRSL_PARAM		0x810u
-#define  PCIE_TX_RX_INTERFACE			0x00000000u
-#define  PCIE_CONFIG_INTERFACE			0x00000001u
-
-#define ATR_ENTRY_SIZE				32
-
 /* PCIe Controller Phy Regs */
 #define SEC_ERROR_EVENT_CNT			0x20
 #define DED_ERROR_EVENT_CNT			0x24
@@ -179,20 +102,21 @@
 #define EVENT_LOCAL_DMA_END_ENGINE_1		12
 #define EVENT_LOCAL_DMA_ERROR_ENGINE_0		13
 #define EVENT_LOCAL_DMA_ERROR_ENGINE_1		14
-#define EVENT_LOCAL_A_ATR_EVT_POST_ERR		15
-#define EVENT_LOCAL_A_ATR_EVT_FETCH_ERR		16
-#define EVENT_LOCAL_A_ATR_EVT_DISCARD_ERR	17
-#define EVENT_LOCAL_A_ATR_EVT_DOORBELL		18
-#define EVENT_LOCAL_P_ATR_EVT_POST_ERR		19
-#define EVENT_LOCAL_P_ATR_EVT_FETCH_ERR		20
-#define EVENT_LOCAL_P_ATR_EVT_DISCARD_ERR	21
-#define EVENT_LOCAL_P_ATR_EVT_DOORBELL		22
-#define EVENT_LOCAL_PM_MSI_INT_INTX		23
-#define EVENT_LOCAL_PM_MSI_INT_MSI		24
-#define EVENT_LOCAL_PM_MSI_INT_AER_EVT		25
-#define EVENT_LOCAL_PM_MSI_INT_EVENTS		26
-#define EVENT_LOCAL_PM_MSI_INT_SYS_ERR		27
-#define NUM_EVENTS				28
+#define NUM_MC_EVENTS				15
+#define EVENT_LOCAL_A_ATR_EVT_POST_ERR		(NUM_MC_EVENTS + PLDA_AXI_POST_ERR)
+#define EVENT_LOCAL_A_ATR_EVT_FETCH_ERR		(NUM_MC_EVENTS + PLDA_AXI_FETCH_ERR)
+#define EVENT_LOCAL_A_ATR_EVT_DISCARD_ERR	(NUM_MC_EVENTS + PLDA_AXI_DISCARD_ERR)
+#define EVENT_LOCAL_A_ATR_EVT_DOORBELL		(NUM_MC_EVENTS + PLDA_AXI_DOORBELL)
+#define EVENT_LOCAL_P_ATR_EVT_POST_ERR		(NUM_MC_EVENTS + PLDA_PCIE_POST_ERR)
+#define EVENT_LOCAL_P_ATR_EVT_FETCH_ERR		(NUM_MC_EVENTS + PLDA_PCIE_FETCH_ERR)
+#define EVENT_LOCAL_P_ATR_EVT_DISCARD_ERR	(NUM_MC_EVENTS + PLDA_PCIE_DISCARD_ERR)
+#define EVENT_LOCAL_P_ATR_EVT_DOORBELL		(NUM_MC_EVENTS + PLDA_PCIE_DOORBELL)
+#define EVENT_LOCAL_PM_MSI_INT_INTX		(NUM_MC_EVENTS + PLDA_INTX)
+#define EVENT_LOCAL_PM_MSI_INT_MSI		(NUM_MC_EVENTS + PLDA_MSI)
+#define EVENT_LOCAL_PM_MSI_INT_AER_EVT		(NUM_MC_EVENTS + PLDA_AER_EVENT)
+#define EVENT_LOCAL_PM_MSI_INT_EVENTS		(NUM_MC_EVENTS + PLDA_MISC_EVENTS)
+#define EVENT_LOCAL_PM_MSI_INT_SYS_ERR		(NUM_MC_EVENTS + PLDA_SYS_ERR)
+#define NUM_EVENTS				(NUM_MC_EVENTS + PLDA_INT_EVENT_NUM)
 
 #define PCIE_EVENT_CAUSE(x, s)	\
 	[EVENT_PCIE_ ## x] = { __stringify(x), s }
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
new file mode 100644
index 000000000000..65e0f3b72184
--- /dev/null
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * PLDA PCIe host controller driver
+ */
+
+#ifndef _PCIE_PLDA_H
+#define _PCIE_PLDA_H
+
+/* PCIe Bridge Phy Regs */
+#define PCIE_PCI_IRQ_DW0			0xa8
+#define  MSIX_CAP_MASK				BIT(31)
+#define  NUM_MSI_MSGS_MASK			GENMASK(6, 4)
+#define  NUM_MSI_MSGS_SHIFT			4
+
+#define IMASK_LOCAL				0x180
+#define  DMA_END_ENGINE_0_MASK			0x00000000u
+#define  DMA_END_ENGINE_0_SHIFT			0
+#define  DMA_END_ENGINE_1_MASK			0x00000000u
+#define  DMA_END_ENGINE_1_SHIFT			1
+#define  DMA_ERROR_ENGINE_0_MASK		0x00000100u
+#define  DMA_ERROR_ENGINE_0_SHIFT		8
+#define  DMA_ERROR_ENGINE_1_MASK		0x00000200u
+#define  DMA_ERROR_ENGINE_1_SHIFT		9
+#define  A_ATR_EVT_POST_ERR_MASK		0x00010000u
+#define  A_ATR_EVT_POST_ERR_SHIFT		16
+#define  A_ATR_EVT_FETCH_ERR_MASK		0x00020000u
+#define  A_ATR_EVT_FETCH_ERR_SHIFT		17
+#define  A_ATR_EVT_DISCARD_ERR_MASK		0x00040000u
+#define  A_ATR_EVT_DISCARD_ERR_SHIFT		18
+#define  A_ATR_EVT_DOORBELL_MASK		0x00000000u
+#define  A_ATR_EVT_DOORBELL_SHIFT		19
+#define  P_ATR_EVT_POST_ERR_MASK		0x00100000u
+#define  P_ATR_EVT_POST_ERR_SHIFT		20
+#define  P_ATR_EVT_FETCH_ERR_MASK		0x00200000u
+#define  P_ATR_EVT_FETCH_ERR_SHIFT		21
+#define  P_ATR_EVT_DISCARD_ERR_MASK		0x00400000u
+#define  P_ATR_EVT_DISCARD_ERR_SHIFT		22
+#define  P_ATR_EVT_DOORBELL_MASK		0x00000000u
+#define  P_ATR_EVT_DOORBELL_SHIFT		23
+#define  PM_MSI_INT_INTA_MASK			0x01000000u
+#define  PM_MSI_INT_INTA_SHIFT			24
+#define  PM_MSI_INT_INTB_MASK			0x02000000u
+#define  PM_MSI_INT_INTB_SHIFT			25
+#define  PM_MSI_INT_INTC_MASK			0x04000000u
+#define  PM_MSI_INT_INTC_SHIFT			26
+#define  PM_MSI_INT_INTD_MASK			0x08000000u
+#define  PM_MSI_INT_INTD_SHIFT			27
+#define  PM_MSI_INT_INTX_MASK			0x0f000000u
+#define  PM_MSI_INT_INTX_SHIFT			24
+#define  PM_MSI_INT_MSI_MASK			0x10000000u
+#define  PM_MSI_INT_MSI_SHIFT			28
+#define  PM_MSI_INT_AER_EVT_MASK		0x20000000u
+#define  PM_MSI_INT_AER_EVT_SHIFT		29
+#define  PM_MSI_INT_EVENTS_MASK			0x40000000u
+#define  PM_MSI_INT_EVENTS_SHIFT		30
+#define  PM_MSI_INT_SYS_ERR_MASK		0x80000000u
+#define  PM_MSI_INT_SYS_ERR_SHIFT		31
+#define  NUM_LOCAL_EVENTS			15
+#define ISTATUS_LOCAL				0x184
+#define IMASK_HOST				0x188
+#define ISTATUS_HOST				0x18c
+#define IMSI_ADDR				0x190
+#define ISTATUS_MSI				0x194
+
+/* PCIe Master table init defines */
+#define ATR0_PCIE_WIN0_SRCADDR_PARAM		0x600u
+#define  ATR0_PCIE_ATR_SIZE			0x25
+#define  ATR0_PCIE_ATR_SIZE_SHIFT		1
+#define ATR0_PCIE_WIN0_SRC_ADDR			0x604u
+#define ATR0_PCIE_WIN0_TRSL_ADDR_LSB		0x608u
+#define ATR0_PCIE_WIN0_TRSL_ADDR_UDW		0x60cu
+#define ATR0_PCIE_WIN0_TRSL_PARAM		0x610u
+
+/* PCIe AXI slave table init defines */
+#define ATR0_AXI4_SLV0_SRCADDR_PARAM		0x800u
+#define  ATR_SIZE_SHIFT				1
+#define  ATR_IMPL_ENABLE			1
+#define ATR0_AXI4_SLV0_SRC_ADDR			0x804u
+#define ATR0_AXI4_SLV0_TRSL_ADDR_LSB		0x808u
+#define ATR0_AXI4_SLV0_TRSL_ADDR_UDW		0x80cu
+#define ATR0_AXI4_SLV0_TRSL_PARAM		0x810u
+#define  PCIE_TX_RX_INTERFACE			0x00000000u
+#define  PCIE_CONFIG_INTERFACE			0x00000001u
+
+#define ATR_ENTRY_SIZE				32
+
+enum plda_int_event {
+	PLDA_AXI_POST_ERR,
+	PLDA_AXI_FETCH_ERR,
+	PLDA_AXI_DISCARD_ERR,
+	PLDA_AXI_DOORBELL,
+	PLDA_PCIE_POST_ERR,
+	PLDA_PCIE_FETCH_ERR,
+	PLDA_PCIE_DISCARD_ERR,
+	PLDA_PCIE_DOORBELL,
+	PLDA_INTX,
+	PLDA_MSI,
+	PLDA_AER_EVENT,
+	PLDA_MISC_EVENTS,
+	PLDA_SYS_ERR,
+	PLDA_INT_EVENT_NUM
+};
+
+#define PLDA_NUM_DMA_EVENTS			16
+
+#define PLDA_MAX_EVENT_NUM			(PLDA_NUM_DMA_EVENTS + PLDA_INT_EVENT_NUM)
+
+#endif
-- 
2.17.1

[PATCH v15,RESEND 04/23] PCI: microchip: Add bridge_addr field to struct mc_pcie

[Minda Chen]

For bridge address base is common PLDA field, Add this to struct mc_pcie
first.

INTx and MSI codes interrupts codes will get the bridge base address from
port->bridge_addr. These codes will be changed to common codes.
axi_base_addr is Microchip its own data.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../pci/controller/plda/pcie-microchip-host.c | 23 ++++++++-----------
 1 file changed, 9 insertions(+), 14 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index d9030d550482..c55ede80a6d0 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -195,6 +195,7 @@ struct mc_pcie {
 	struct irq_domain *event_domain;
 	raw_spinlock_t lock;
 	struct mc_msi msi;
+	void __iomem *bridge_addr;
 };
 
 struct cause {
@@ -339,8 +340,7 @@ static void mc_handle_msi(struct irq_desc *desc)
 	struct irq_chip *chip = irq_desc_get_chip(desc);
 	struct device *dev = port->dev;
 	struct mc_msi *msi = &port->msi;
-	void __iomem *bridge_base_addr =
-		port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
+	void __iomem *bridge_base_addr = port->bridge_addr;
 	unsigned long status;
 	u32 bit;
 	int ret;
@@ -365,8 +365,7 @@ static void mc_handle_msi(struct irq_desc *desc)
 static void mc_msi_bottom_irq_ack(struct irq_data *data)
 {
 	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
-	void __iomem *bridge_base_addr =
-		port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
+	void __iomem *bridge_base_addr = port->bridge_addr;
 	u32 bitpos = data->hwirq;
 
 	writel_relaxed(BIT(bitpos), bridge_base_addr + ISTATUS_MSI);
@@ -488,8 +487,7 @@ static void mc_handle_intx(struct irq_desc *desc)
 	struct mc_pcie *port = irq_desc_get_handler_data(desc);
 	struct irq_chip *chip = irq_desc_get_chip(desc);
 	struct device *dev = port->dev;
-	void __iomem *bridge_base_addr =
-		port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
+	void __iomem *bridge_base_addr = port->bridge_addr;
 	unsigned long status;
 	u32 bit;
 	int ret;
@@ -514,8 +512,7 @@ static void mc_handle_intx(struct irq_desc *desc)
 static void mc_ack_intx_irq(struct irq_data *data)
 {
 	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
-	void __iomem *bridge_base_addr =
-		port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
+	void __iomem *bridge_base_addr = port->bridge_addr;
 	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
 
 	writel_relaxed(mask, bridge_base_addr + ISTATUS_LOCAL);
@@ -524,8 +521,7 @@ static void mc_ack_intx_irq(struct irq_data *data)
 static void mc_mask_intx_irq(struct irq_data *data)
 {
 	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
-	void __iomem *bridge_base_addr =
-		port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
+	void __iomem *bridge_base_addr = port->bridge_addr;
 	unsigned long flags;
 	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
 	u32 val;
@@ -540,8 +536,7 @@ static void mc_mask_intx_irq(struct irq_data *data)
 static void mc_unmask_intx_irq(struct irq_data *data)
 {
 	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
-	void __iomem *bridge_base_addr =
-		port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
+	void __iomem *bridge_base_addr = port->bridge_addr;
 	unsigned long flags;
 	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
 	u32 val;
@@ -896,8 +891,7 @@ static void mc_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
 static int mc_pcie_setup_windows(struct platform_device *pdev,
 				 struct mc_pcie *port)
 {
-	void __iomem *bridge_base_addr =
-		port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
+	void __iomem *bridge_base_addr = port->bridge_addr;
 	struct pci_host_bridge *bridge = platform_get_drvdata(pdev);
 	struct resource_entry *entry;
 	u64 pci_addr;
@@ -1081,6 +1075,7 @@ static int mc_host_probe(struct platform_device *pdev)
 	mc_disable_interrupts(port);
 
 	bridge_base_addr = port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
+	port->bridge_addr = bridge_base_addr;
 
 	/* Allow enabling MSI by disabling MSI-X */
 	val = readl(bridge_base_addr + PCIE_PCI_IRQ_DW0);
-- 
2.17.1

[PATCH v15,RESEND 05/23] PCI: microchip: Rename two PCIe data structures

[Minda Chen]

Add PLDA PCIe related data structures by rename data structure name from
mc_* to plda_*.

axi_base_addr is stayed in struct mc_pcie for it's microchip its own data.

The event interrupt codes is still using struct mc_pcie because the event
interrupt codes can not be re-used.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../pci/controller/plda/pcie-microchip-host.c | 96 ++++++++++---------
 1 file changed, 53 insertions(+), 43 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index c55ede80a6d0..df0736f688ce 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -22,7 +22,7 @@
 #include "pcie-plda.h"
 
 /* Number of MSI IRQs */
-#define MC_MAX_NUM_MSI_IRQS			32
+#define PLDA_MAX_NUM_MSI_IRQS			32
 
 /* PCIe Bridge Phy and Controller Phy offsets */
 #define MC_PCIE1_BRIDGE_ADDR			0x00008000u
@@ -179,25 +179,29 @@ struct event_map {
 	u32 event_bit;
 };
 
-struct mc_msi {
+struct plda_msi {
 	struct mutex lock;		/* Protect used bitmap */
 	struct irq_domain *msi_domain;
 	struct irq_domain *dev_domain;
 	u32 num_vectors;
 	u64 vector_phy;
-	DECLARE_BITMAP(used, MC_MAX_NUM_MSI_IRQS);
+	DECLARE_BITMAP(used, PLDA_MAX_NUM_MSI_IRQS);
 };
 
-struct mc_pcie {
-	void __iomem *axi_base_addr;
+struct plda_pcie_rp {
 	struct device *dev;
 	struct irq_domain *intx_domain;
 	struct irq_domain *event_domain;
 	raw_spinlock_t lock;
-	struct mc_msi msi;
+	struct plda_msi msi;
 	void __iomem *bridge_addr;
 };
 
+struct mc_pcie {
+	struct plda_pcie_rp plda;
+	void __iomem *axi_base_addr;
+};
+
 struct cause {
 	const char *sym;
 	const char *str;
@@ -313,7 +317,7 @@ static struct mc_pcie *port;
 
 static void mc_pcie_enable_msi(struct mc_pcie *port, void __iomem *ecam)
 {
-	struct mc_msi *msi = &port->msi;
+	struct plda_msi *msi = &port->plda.msi;
 	u16 reg;
 	u8 queue_size;
 
@@ -336,10 +340,10 @@ static void mc_pcie_enable_msi(struct mc_pcie *port, void __iomem *ecam)
 
 static void mc_handle_msi(struct irq_desc *desc)
 {
-	struct mc_pcie *port = irq_desc_get_handler_data(desc);
+	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
 	struct irq_chip *chip = irq_desc_get_chip(desc);
 	struct device *dev = port->dev;
-	struct mc_msi *msi = &port->msi;
+	struct plda_msi *msi = &port->msi;
 	void __iomem *bridge_base_addr = port->bridge_addr;
 	unsigned long status;
 	u32 bit;
@@ -364,7 +368,7 @@ static void mc_handle_msi(struct irq_desc *desc)
 
 static void mc_msi_bottom_irq_ack(struct irq_data *data)
 {
-	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	void __iomem *bridge_base_addr = port->bridge_addr;
 	u32 bitpos = data->hwirq;
 
@@ -373,7 +377,7 @@ static void mc_msi_bottom_irq_ack(struct irq_data *data)
 
 static void mc_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 {
-	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	phys_addr_t addr = port->msi.vector_phy;
 
 	msg->address_lo = lower_32_bits(addr);
@@ -400,8 +404,8 @@ static struct irq_chip mc_msi_bottom_irq_chip = {
 static int mc_irq_msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
 				   unsigned int nr_irqs, void *args)
 {
-	struct mc_pcie *port = domain->host_data;
-	struct mc_msi *msi = &port->msi;
+	struct plda_pcie_rp *port = domain->host_data;
+	struct plda_msi *msi = &port->msi;
 	unsigned long bit;
 
 	mutex_lock(&msi->lock);
@@ -425,8 +429,8 @@ static void mc_irq_msi_domain_free(struct irq_domain *domain, unsigned int virq,
 				   unsigned int nr_irqs)
 {
 	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
-	struct mc_pcie *port = irq_data_get_irq_chip_data(d);
-	struct mc_msi *msi = &port->msi;
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(d);
+	struct plda_msi *msi = &port->msi;
 
 	mutex_lock(&msi->lock);
 
@@ -456,11 +460,11 @@ static struct msi_domain_info mc_msi_domain_info = {
 	.chip = &mc_msi_irq_chip,
 };
 
-static int mc_allocate_msi_domains(struct mc_pcie *port)
+static int mc_allocate_msi_domains(struct plda_pcie_rp *port)
 {
 	struct device *dev = port->dev;
 	struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
-	struct mc_msi *msi = &port->msi;
+	struct plda_msi *msi = &port->msi;
 
 	mutex_init(&port->msi.lock);
 
@@ -484,7 +488,7 @@ static int mc_allocate_msi_domains(struct mc_pcie *port)
 
 static void mc_handle_intx(struct irq_desc *desc)
 {
-	struct mc_pcie *port = irq_desc_get_handler_data(desc);
+	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
 	struct irq_chip *chip = irq_desc_get_chip(desc);
 	struct device *dev = port->dev;
 	void __iomem *bridge_base_addr = port->bridge_addr;
@@ -511,7 +515,7 @@ static void mc_handle_intx(struct irq_desc *desc)
 
 static void mc_ack_intx_irq(struct irq_data *data)
 {
-	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	void __iomem *bridge_base_addr = port->bridge_addr;
 	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
 
@@ -520,7 +524,7 @@ static void mc_ack_intx_irq(struct irq_data *data)
 
 static void mc_mask_intx_irq(struct irq_data *data)
 {
-	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	void __iomem *bridge_base_addr = port->bridge_addr;
 	unsigned long flags;
 	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
@@ -535,7 +539,7 @@ static void mc_mask_intx_irq(struct irq_data *data)
 
 static void mc_unmask_intx_irq(struct irq_data *data)
 {
-	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	void __iomem *bridge_base_addr = port->bridge_addr;
 	unsigned long flags;
 	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
@@ -625,21 +629,22 @@ static u32 local_events(struct mc_pcie *port)
 	return val;
 }
 
-static u32 get_events(struct mc_pcie *port)
+static u32 get_events(struct plda_pcie_rp *port)
 {
+	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
 	u32 events = 0;
 
-	events |= pcie_events(port);
-	events |= sec_errors(port);
-	events |= ded_errors(port);
-	events |= local_events(port);
+	events |= pcie_events(mc_port);
+	events |= sec_errors(mc_port);
+	events |= ded_errors(mc_port);
+	events |= local_events(mc_port);
 
 	return events;
 }
 
 static irqreturn_t mc_event_handler(int irq, void *dev_id)
 {
-	struct mc_pcie *port = dev_id;
+	struct plda_pcie_rp *port = dev_id;
 	struct device *dev = port->dev;
 	struct irq_data *data;
 
@@ -655,7 +660,7 @@ static irqreturn_t mc_event_handler(int irq, void *dev_id)
 
 static void mc_handle_event(struct irq_desc *desc)
 {
-	struct mc_pcie *port = irq_desc_get_handler_data(desc);
+	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
 	unsigned long events;
 	u32 bit;
 	struct irq_chip *chip = irq_desc_get_chip(desc);
@@ -672,12 +677,13 @@ static void mc_handle_event(struct irq_desc *desc)
 
 static void mc_ack_event_irq(struct irq_data *data)
 {
-	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
 	u32 event = data->hwirq;
 	void __iomem *addr;
 	u32 mask;
 
-	addr = port->axi_base_addr + event_descs[event].base +
+	addr = mc_port->axi_base_addr + event_descs[event].base +
 		event_descs[event].offset;
 	mask = event_descs[event].mask;
 	mask |= event_descs[event].enb_mask;
@@ -687,13 +693,14 @@ static void mc_ack_event_irq(struct irq_data *data)
 
 static void mc_mask_event_irq(struct irq_data *data)
 {
-	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
 	u32 event = data->hwirq;
 	void __iomem *addr;
 	u32 mask;
 	u32 val;
 
-	addr = port->axi_base_addr + event_descs[event].base +
+	addr = mc_port->axi_base_addr + event_descs[event].base +
 		event_descs[event].mask_offset;
 	mask = event_descs[event].mask;
 	if (event_descs[event].enb_mask) {
@@ -717,13 +724,14 @@ static void mc_mask_event_irq(struct irq_data *data)
 
 static void mc_unmask_event_irq(struct irq_data *data)
 {
-	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
 	u32 event = data->hwirq;
 	void __iomem *addr;
 	u32 mask;
 	u32 val;
 
-	addr = port->axi_base_addr + event_descs[event].base +
+	addr = mc_port->axi_base_addr + event_descs[event].base +
 		event_descs[event].mask_offset;
 	mask = event_descs[event].mask;
 
@@ -811,7 +819,7 @@ static int mc_pcie_init_clks(struct device *dev)
 	return 0;
 }
 
-static int mc_pcie_init_irq_domains(struct mc_pcie *port)
+static int mc_pcie_init_irq_domains(struct plda_pcie_rp *port)
 {
 	struct device *dev = port->dev;
 	struct device_node *node = dev->of_node;
@@ -889,7 +897,7 @@ static void mc_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
 }
 
 static int mc_pcie_setup_windows(struct platform_device *pdev,
-				 struct mc_pcie *port)
+				 struct plda_pcie_rp *port)
 {
 	void __iomem *bridge_base_addr = port->bridge_addr;
 	struct pci_host_bridge *bridge = platform_get_drvdata(pdev);
@@ -970,7 +978,7 @@ static void mc_disable_interrupts(struct mc_pcie *port)
 	writel_relaxed(GENMASK(31, 0), bridge_base_addr + ISTATUS_HOST);
 }
 
-static int mc_init_interrupts(struct platform_device *pdev, struct mc_pcie *port)
+static int mc_init_interrupts(struct platform_device *pdev, struct plda_pcie_rp *port)
 {
 	struct device *dev = &pdev->dev;
 	int irq;
@@ -1043,12 +1051,12 @@ static int mc_platform_init(struct pci_config_window *cfg)
 	mc_pcie_enable_msi(port, cfg->win);
 
 	/* Configure non-config space outbound ranges */
-	ret = mc_pcie_setup_windows(pdev, port);
+	ret = mc_pcie_setup_windows(pdev, &port->plda);
 	if (ret)
 		return ret;
 
 	/* Address translation is up; safe to enable interrupts */
-	ret = mc_init_interrupts(pdev, port);
+	ret = mc_init_interrupts(pdev, &port->plda);
 	if (ret)
 		return ret;
 
@@ -1059,6 +1067,7 @@ static int mc_host_probe(struct platform_device *pdev)
 {
 	struct device *dev = &pdev->dev;
 	void __iomem *bridge_base_addr;
+	struct plda_pcie_rp *plda;
 	int ret;
 	u32 val;
 
@@ -1066,7 +1075,8 @@ static int mc_host_probe(struct platform_device *pdev)
 	if (!port)
 		return -ENOMEM;
 
-	port->dev = dev;
+	plda = &port->plda;
+	plda->dev = dev;
 
 	port->axi_base_addr = devm_platform_ioremap_resource(pdev, 1);
 	if (IS_ERR(port->axi_base_addr))
@@ -1075,7 +1085,7 @@ static int mc_host_probe(struct platform_device *pdev)
 	mc_disable_interrupts(port);
 
 	bridge_base_addr = port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
-	port->bridge_addr = bridge_base_addr;
+	plda->bridge_addr = bridge_base_addr;
 
 	/* Allow enabling MSI by disabling MSI-X */
 	val = readl(bridge_base_addr + PCIE_PCI_IRQ_DW0);
@@ -1087,10 +1097,10 @@ static int mc_host_probe(struct platform_device *pdev)
 	val &= NUM_MSI_MSGS_MASK;
 	val >>= NUM_MSI_MSGS_SHIFT;
 
-	port->msi.num_vectors = 1 << val;
+	plda->msi.num_vectors = 1 << val;
 
 	/* Pick vector address from design */
-	port->msi.vector_phy = readl_relaxed(bridge_base_addr + IMSI_ADDR);
+	plda->msi.vector_phy = readl_relaxed(bridge_base_addr + IMSI_ADDR);
 
 	ret = mc_pcie_init_clks(dev);
 	if (ret) {
-- 
2.17.1

[PATCH v15,RESEND 06/23] PCI: microchip: Move PCIe host data structures to plda-pcie.h

[Minda Chen]

Move the common data structures definition to head file for these two data
structures can be re-used.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../pci/controller/plda/pcie-microchip-host.c | 20 ------------------
 drivers/pci/controller/plda/pcie-plda.h       | 21 +++++++++++++++++++
 2 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index df0736f688ce..a554a56cc0e8 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -21,9 +21,6 @@
 #include "../../pci.h"
 #include "pcie-plda.h"
 
-/* Number of MSI IRQs */
-#define PLDA_MAX_NUM_MSI_IRQS			32
-
 /* PCIe Bridge Phy and Controller Phy offsets */
 #define MC_PCIE1_BRIDGE_ADDR			0x00008000u
 #define MC_PCIE1_CTRL_ADDR			0x0000a000u
@@ -179,23 +176,6 @@ struct event_map {
 	u32 event_bit;
 };
 
-struct plda_msi {
-	struct mutex lock;		/* Protect used bitmap */
-	struct irq_domain *msi_domain;
-	struct irq_domain *dev_domain;
-	u32 num_vectors;
-	u64 vector_phy;
-	DECLARE_BITMAP(used, PLDA_MAX_NUM_MSI_IRQS);
-};
-
-struct plda_pcie_rp {
-	struct device *dev;
-	struct irq_domain *intx_domain;
-	struct irq_domain *event_domain;
-	raw_spinlock_t lock;
-	struct plda_msi msi;
-	void __iomem *bridge_addr;
-};
 
 struct mc_pcie {
 	struct plda_pcie_rp plda;
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index 65e0f3b72184..9ca66916c609 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -6,6 +6,9 @@
 #ifndef _PCIE_PLDA_H
 #define _PCIE_PLDA_H
 
+/* Number of MSI IRQs */
+#define PLDA_MAX_NUM_MSI_IRQS			32
+
 /* PCIe Bridge Phy Regs */
 #define PCIE_PCI_IRQ_DW0			0xa8
 #define  MSIX_CAP_MASK				BIT(31)
@@ -105,4 +108,22 @@ enum plda_int_event {
 
 #define PLDA_MAX_EVENT_NUM			(PLDA_NUM_DMA_EVENTS + PLDA_INT_EVENT_NUM)
 
+struct plda_msi {
+	struct mutex lock;		/* Protect used bitmap */
+	struct irq_domain *msi_domain;
+	struct irq_domain *dev_domain;
+	u32 num_vectors;
+	u64 vector_phy;
+	DECLARE_BITMAP(used, PLDA_MAX_NUM_MSI_IRQS);
+};
+
+struct plda_pcie_rp {
+	struct device *dev;
+	struct irq_domain *intx_domain;
+	struct irq_domain *event_domain;
+	raw_spinlock_t lock;
+	struct plda_msi msi;
+	void __iomem *bridge_addr;
+};
+
 #endif
-- 
2.17.1

[PATCH v15,RESEND 07/23] PCI: microchip: Rename two setup functions

[Minda Chen]

Rename two setup functions to plda prefix. Prepare to re-use these two
setup function.

For two setup functions names are similar, rename mc_pcie_setup_windows()
to plda_pcie_setup_iomems().

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../pci/controller/plda/pcie-microchip-host.c | 24 +++++++++----------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index a554a56cc0e8..9b367927cd32 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -838,9 +838,9 @@ static int mc_pcie_init_irq_domains(struct plda_pcie_rp *port)
 	return mc_allocate_msi_domains(port);
 }
 
-static void mc_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
-				 phys_addr_t axi_addr, phys_addr_t pci_addr,
-				 size_t size)
+static void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
+				   phys_addr_t axi_addr, phys_addr_t pci_addr,
+				   size_t size)
 {
 	u32 atr_sz = ilog2(size) - 1;
 	u32 val;
@@ -876,8 +876,8 @@ static void mc_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
 	writel(0, bridge_base_addr + ATR0_PCIE_WIN0_SRC_ADDR);
 }
 
-static int mc_pcie_setup_windows(struct platform_device *pdev,
-				 struct plda_pcie_rp *port)
+static int plda_pcie_setup_iomems(struct platform_device *pdev,
+				  struct plda_pcie_rp *port)
 {
 	void __iomem *bridge_base_addr = port->bridge_addr;
 	struct pci_host_bridge *bridge = platform_get_drvdata(pdev);
@@ -888,9 +888,9 @@ static int mc_pcie_setup_windows(struct platform_device *pdev,
 	resource_list_for_each_entry(entry, &bridge->windows) {
 		if (resource_type(entry->res) == IORESOURCE_MEM) {
 			pci_addr = entry->res->start - entry->offset;
-			mc_pcie_setup_window(bridge_base_addr, index,
-					     entry->res->start, pci_addr,
-					     resource_size(entry->res));
+			plda_pcie_setup_window(bridge_base_addr, index,
+					       entry->res->start, pci_addr,
+					       resource_size(entry->res));
 			index++;
 		}
 	}
@@ -1023,15 +1023,15 @@ static int mc_platform_init(struct pci_config_window *cfg)
 	int ret;
 
 	/* Configure address translation table 0 for PCIe config space */
-	mc_pcie_setup_window(bridge_base_addr, 0, cfg->res.start,
-			     cfg->res.start,
-			     resource_size(&cfg->res));
+	plda_pcie_setup_window(bridge_base_addr, 0, cfg->res.start,
+			       cfg->res.start,
+			       resource_size(&cfg->res));
 
 	/* Need some fixups in config space */
 	mc_pcie_enable_msi(port, cfg->win);
 
 	/* Configure non-config space outbound ranges */
-	ret = mc_pcie_setup_windows(pdev, &port->plda);
+	ret = plda_pcie_setup_iomems(pdev, &port->plda);
 	if (ret)
 		return ret;
 
-- 
2.17.1

[PATCH v15,RESEND 08/23] PCI: microchip: Change the argument of plda_pcie_setup_iomems()

[Minda Chen]

If other vendor do not select PCI_HOST_COMMON, the driver data is not
struct pci_host_bridge.

Move calling platform_get_drvdata() to mc_platform_init().

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 drivers/pci/controller/plda/pcie-microchip-host.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index 9b367927cd32..805870aed61d 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -876,11 +876,10 @@ static void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
 	writel(0, bridge_base_addr + ATR0_PCIE_WIN0_SRC_ADDR);
 }
 
-static int plda_pcie_setup_iomems(struct platform_device *pdev,
+static int plda_pcie_setup_iomems(struct pci_host_bridge *bridge,
 				  struct plda_pcie_rp *port)
 {
 	void __iomem *bridge_base_addr = port->bridge_addr;
-	struct pci_host_bridge *bridge = platform_get_drvdata(pdev);
 	struct resource_entry *entry;
 	u64 pci_addr;
 	u32 index = 1;
@@ -1018,6 +1017,7 @@ static int mc_platform_init(struct pci_config_window *cfg)
 {
 	struct device *dev = cfg->parent;
 	struct platform_device *pdev = to_platform_device(dev);
+	struct pci_host_bridge *bridge = platform_get_drvdata(pdev);
 	void __iomem *bridge_base_addr =
 		port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
 	int ret;
@@ -1031,7 +1031,7 @@ static int mc_platform_init(struct pci_config_window *cfg)
 	mc_pcie_enable_msi(port, cfg->win);
 
 	/* Configure non-config space outbound ranges */
-	ret = plda_pcie_setup_iomems(pdev, &port->plda);
+	ret = plda_pcie_setup_iomems(bridge, &port->plda);
 	if (ret)
 		return ret;
 
-- 
2.17.1

[PATCH v15,RESEND 09/23] PCI: microchip: Move setup functions to pcie-plda-host.c

[Minda Chen]

Move setup functions to common pcie-plda-host.c. So these two functions
can be re-used.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 drivers/pci/controller/plda/Kconfig           |  4 +
 drivers/pci/controller/plda/Makefile          |  1 +
 .../pci/controller/plda/pcie-microchip-host.c | 59 ---------------
 drivers/pci/controller/plda/pcie-plda-host.c  | 73 +++++++++++++++++++
 drivers/pci/controller/plda/pcie-plda.h       |  5 ++
 5 files changed, 83 insertions(+), 59 deletions(-)
 create mode 100644 drivers/pci/controller/plda/pcie-plda-host.c

diff --git a/drivers/pci/controller/plda/Kconfig b/drivers/pci/controller/plda/Kconfig
index 5cb3be4fc98c..e54a82ee94f5 100644
--- a/drivers/pci/controller/plda/Kconfig
+++ b/drivers/pci/controller/plda/Kconfig
@@ -3,10 +3,14 @@
 menu "PLDA-based PCIe controllers"
 	depends on PCI
 
+config PCIE_PLDA_HOST
+	bool
+
 config PCIE_MICROCHIP_HOST
 	tristate "Microchip AXI PCIe controller"
 	depends on PCI_MSI && OF
 	select PCI_HOST_COMMON
+	select PCIE_PLDA_HOST
 	help
 	  Say Y here if you want kernel to support the Microchip AXI PCIe
 	  Host Bridge driver.
diff --git a/drivers/pci/controller/plda/Makefile b/drivers/pci/controller/plda/Makefile
index e1a265cbf91c..4340ab007f44 100644
--- a/drivers/pci/controller/plda/Makefile
+++ b/drivers/pci/controller/plda/Makefile
@@ -1,2 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_PCIE_PLDA_HOST) += pcie-plda-host.o
 obj-$(CONFIG_PCIE_MICROCHIP_HOST) += pcie-microchip-host.o
diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index 805870aed61d..573ad31c578a 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -838,65 +838,6 @@ static int mc_pcie_init_irq_domains(struct plda_pcie_rp *port)
 	return mc_allocate_msi_domains(port);
 }
 
-static void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
-				   phys_addr_t axi_addr, phys_addr_t pci_addr,
-				   size_t size)
-{
-	u32 atr_sz = ilog2(size) - 1;
-	u32 val;
-
-	if (index == 0)
-		val = PCIE_CONFIG_INTERFACE;
-	else
-		val = PCIE_TX_RX_INTERFACE;
-
-	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
-	       ATR0_AXI4_SLV0_TRSL_PARAM);
-
-	val = lower_32_bits(axi_addr) | (atr_sz << ATR_SIZE_SHIFT) |
-			    ATR_IMPL_ENABLE;
-	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
-	       ATR0_AXI4_SLV0_SRCADDR_PARAM);
-
-	val = upper_32_bits(axi_addr);
-	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
-	       ATR0_AXI4_SLV0_SRC_ADDR);
-
-	val = lower_32_bits(pci_addr);
-	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
-	       ATR0_AXI4_SLV0_TRSL_ADDR_LSB);
-
-	val = upper_32_bits(pci_addr);
-	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
-	       ATR0_AXI4_SLV0_TRSL_ADDR_UDW);
-
-	val = readl(bridge_base_addr + ATR0_PCIE_WIN0_SRCADDR_PARAM);
-	val |= (ATR0_PCIE_ATR_SIZE << ATR0_PCIE_ATR_SIZE_SHIFT);
-	writel(val, bridge_base_addr + ATR0_PCIE_WIN0_SRCADDR_PARAM);
-	writel(0, bridge_base_addr + ATR0_PCIE_WIN0_SRC_ADDR);
-}
-
-static int plda_pcie_setup_iomems(struct pci_host_bridge *bridge,
-				  struct plda_pcie_rp *port)
-{
-	void __iomem *bridge_base_addr = port->bridge_addr;
-	struct resource_entry *entry;
-	u64 pci_addr;
-	u32 index = 1;
-
-	resource_list_for_each_entry(entry, &bridge->windows) {
-		if (resource_type(entry->res) == IORESOURCE_MEM) {
-			pci_addr = entry->res->start - entry->offset;
-			plda_pcie_setup_window(bridge_base_addr, index,
-					       entry->res->start, pci_addr,
-					       resource_size(entry->res));
-			index++;
-		}
-	}
-
-	return 0;
-}
-
 static inline void mc_clear_secs(struct mc_pcie *port)
 {
 	void __iomem *ctrl_base_addr = port->axi_base_addr + MC_PCIE_CTRL_ADDR;
diff --git a/drivers/pci/controller/plda/pcie-plda-host.c b/drivers/pci/controller/plda/pcie-plda-host.c
new file mode 100644
index 000000000000..05ea68baebfb
--- /dev/null
+++ b/drivers/pci/controller/plda/pcie-plda-host.c
@@ -0,0 +1,73 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PLDA PCIe XpressRich host controller driver
+ *
+ * Copyright (C) 2023 Microchip Co. Ltd
+ *
+ * Author: Daire McNamara <daire.mcnamara@microchip.com>
+ */
+
+#include <linux/pci-ecam.h>
+
+#include "pcie-plda.h"
+
+void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
+			    phys_addr_t axi_addr, phys_addr_t pci_addr,
+			    size_t size)
+{
+	u32 atr_sz = ilog2(size) - 1;
+	u32 val;
+
+	if (index == 0)
+		val = PCIE_CONFIG_INTERFACE;
+	else
+		val = PCIE_TX_RX_INTERFACE;
+
+	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
+	       ATR0_AXI4_SLV0_TRSL_PARAM);
+
+	val = lower_32_bits(axi_addr) | (atr_sz << ATR_SIZE_SHIFT) |
+			    ATR_IMPL_ENABLE;
+	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
+	       ATR0_AXI4_SLV0_SRCADDR_PARAM);
+
+	val = upper_32_bits(axi_addr);
+	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
+	       ATR0_AXI4_SLV0_SRC_ADDR);
+
+	val = lower_32_bits(pci_addr);
+	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
+	       ATR0_AXI4_SLV0_TRSL_ADDR_LSB);
+
+	val = upper_32_bits(pci_addr);
+	writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
+	       ATR0_AXI4_SLV0_TRSL_ADDR_UDW);
+
+	val = readl(bridge_base_addr + ATR0_PCIE_WIN0_SRCADDR_PARAM);
+	val |= (ATR0_PCIE_ATR_SIZE << ATR0_PCIE_ATR_SIZE_SHIFT);
+	writel(val, bridge_base_addr + ATR0_PCIE_WIN0_SRCADDR_PARAM);
+	writel(0, bridge_base_addr + ATR0_PCIE_WIN0_SRC_ADDR);
+}
+EXPORT_SYMBOL_GPL(plda_pcie_setup_window);
+
+int plda_pcie_setup_iomems(struct pci_host_bridge *bridge,
+			   struct plda_pcie_rp *port)
+{
+	void __iomem *bridge_base_addr = port->bridge_addr;
+	struct resource_entry *entry;
+	u64 pci_addr;
+	u32 index = 1;
+
+	resource_list_for_each_entry(entry, &bridge->windows) {
+		if (resource_type(entry->res) == IORESOURCE_MEM) {
+			pci_addr = entry->res->start - entry->offset;
+			plda_pcie_setup_window(bridge_base_addr, index,
+					       entry->res->start, pci_addr,
+					       resource_size(entry->res));
+			index++;
+		}
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(plda_pcie_setup_iomems);
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index 9ca66916c609..e277a5452b5d 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -126,4 +126,9 @@ struct plda_pcie_rp {
 	void __iomem *bridge_addr;
 };
 
+void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
+			    phys_addr_t axi_addr, phys_addr_t pci_addr,
+			    size_t size);
+int plda_pcie_setup_iomems(struct pci_host_bridge *bridge,
+			   struct plda_pcie_rp *port);
 #endif
-- 
2.17.1

[PATCH v15,RESEND 10/23] PCI: microchip: Rename interrupt related functions

[Minda Chen]

Rename mc_* to plda_* for IRQ functions and related IRQ domain ops data
instances.

MSI, INTx interrupt code and IRQ init code are all can be re-used.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Acked-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../pci/controller/plda/pcie-microchip-host.c | 109 +++++++++---------
 1 file changed, 57 insertions(+), 52 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index 573ad31c578a..18bc352db389 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -318,7 +318,7 @@ static void mc_pcie_enable_msi(struct mc_pcie *port, void __iomem *ecam)
 		       ecam + MC_MSI_CAP_CTRL_OFFSET + PCI_MSI_ADDRESS_HI);
 }
 
-static void mc_handle_msi(struct irq_desc *desc)
+static void plda_handle_msi(struct irq_desc *desc)
 {
 	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
 	struct irq_chip *chip = irq_desc_get_chip(desc);
@@ -346,7 +346,7 @@ static void mc_handle_msi(struct irq_desc *desc)
 	chained_irq_exit(chip, desc);
 }
 
-static void mc_msi_bottom_irq_ack(struct irq_data *data)
+static void plda_msi_bottom_irq_ack(struct irq_data *data)
 {
 	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	void __iomem *bridge_base_addr = port->bridge_addr;
@@ -355,7 +355,7 @@ static void mc_msi_bottom_irq_ack(struct irq_data *data)
 	writel_relaxed(BIT(bitpos), bridge_base_addr + ISTATUS_MSI);
 }
 
-static void mc_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
+static void plda_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 {
 	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	phys_addr_t addr = port->msi.vector_phy;
@@ -368,21 +368,23 @@ static void mc_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 		(int)data->hwirq, msg->address_hi, msg->address_lo);
 }
 
-static int mc_msi_set_affinity(struct irq_data *irq_data,
-			       const struct cpumask *mask, bool force)
+static int plda_msi_set_affinity(struct irq_data *irq_data,
+				 const struct cpumask *mask, bool force)
 {
 	return -EINVAL;
 }
 
-static struct irq_chip mc_msi_bottom_irq_chip = {
-	.name = "Microchip MSI",
-	.irq_ack = mc_msi_bottom_irq_ack,
-	.irq_compose_msi_msg = mc_compose_msi_msg,
-	.irq_set_affinity = mc_msi_set_affinity,
+static struct irq_chip plda_msi_bottom_irq_chip = {
+	.name = "PLDA MSI",
+	.irq_ack = plda_msi_bottom_irq_ack,
+	.irq_compose_msi_msg = plda_compose_msi_msg,
+	.irq_set_affinity = plda_msi_set_affinity,
 };
 
-static int mc_irq_msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
-				   unsigned int nr_irqs, void *args)
+static int plda_irq_msi_domain_alloc(struct irq_domain *domain,
+				     unsigned int virq,
+				     unsigned int nr_irqs,
+				     void *args)
 {
 	struct plda_pcie_rp *port = domain->host_data;
 	struct plda_msi *msi = &port->msi;
@@ -397,7 +399,7 @@ static int mc_irq_msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
 
 	set_bit(bit, msi->used);
 
-	irq_domain_set_info(domain, virq, bit, &mc_msi_bottom_irq_chip,
+	irq_domain_set_info(domain, virq, bit, &plda_msi_bottom_irq_chip,
 			    domain->host_data, handle_edge_irq, NULL, NULL);
 
 	mutex_unlock(&msi->lock);
@@ -405,8 +407,9 @@ static int mc_irq_msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
 	return 0;
 }
 
-static void mc_irq_msi_domain_free(struct irq_domain *domain, unsigned int virq,
-				   unsigned int nr_irqs)
+static void plda_irq_msi_domain_free(struct irq_domain *domain,
+				     unsigned int virq,
+				     unsigned int nr_irqs)
 {
 	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
 	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(d);
@@ -423,24 +426,24 @@ static void mc_irq_msi_domain_free(struct irq_domain *domain, unsigned int virq,
 }
 
 static const struct irq_domain_ops msi_domain_ops = {
-	.alloc	= mc_irq_msi_domain_alloc,
-	.free	= mc_irq_msi_domain_free,
+	.alloc	= plda_irq_msi_domain_alloc,
+	.free	= plda_irq_msi_domain_free,
 };
 
-static struct irq_chip mc_msi_irq_chip = {
-	.name = "Microchip PCIe MSI",
+static struct irq_chip plda_msi_irq_chip = {
+	.name = "PLDA PCIe MSI",
 	.irq_ack = irq_chip_ack_parent,
 	.irq_mask = pci_msi_mask_irq,
 	.irq_unmask = pci_msi_unmask_irq,
 };
 
-static struct msi_domain_info mc_msi_domain_info = {
+static struct msi_domain_info plda_msi_domain_info = {
 	.flags = (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
 		  MSI_FLAG_PCI_MSIX),
-	.chip = &mc_msi_irq_chip,
+	.chip = &plda_msi_irq_chip,
 };
 
-static int mc_allocate_msi_domains(struct plda_pcie_rp *port)
+static int plda_allocate_msi_domains(struct plda_pcie_rp *port)
 {
 	struct device *dev = port->dev;
 	struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
@@ -455,7 +458,8 @@ static int mc_allocate_msi_domains(struct plda_pcie_rp *port)
 		return -ENOMEM;
 	}
 
-	msi->msi_domain = pci_msi_create_irq_domain(fwnode, &mc_msi_domain_info,
+	msi->msi_domain = pci_msi_create_irq_domain(fwnode,
+						    &plda_msi_domain_info,
 						    msi->dev_domain);
 	if (!msi->msi_domain) {
 		dev_err(dev, "failed to create MSI domain\n");
@@ -466,7 +470,7 @@ static int mc_allocate_msi_domains(struct plda_pcie_rp *port)
 	return 0;
 }
 
-static void mc_handle_intx(struct irq_desc *desc)
+static void plda_handle_intx(struct irq_desc *desc)
 {
 	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
 	struct irq_chip *chip = irq_desc_get_chip(desc);
@@ -493,7 +497,7 @@ static void mc_handle_intx(struct irq_desc *desc)
 	chained_irq_exit(chip, desc);
 }
 
-static void mc_ack_intx_irq(struct irq_data *data)
+static void plda_ack_intx_irq(struct irq_data *data)
 {
 	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	void __iomem *bridge_base_addr = port->bridge_addr;
@@ -502,7 +506,7 @@ static void mc_ack_intx_irq(struct irq_data *data)
 	writel_relaxed(mask, bridge_base_addr + ISTATUS_LOCAL);
 }
 
-static void mc_mask_intx_irq(struct irq_data *data)
+static void plda_mask_intx_irq(struct irq_data *data)
 {
 	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	void __iomem *bridge_base_addr = port->bridge_addr;
@@ -517,7 +521,7 @@ static void mc_mask_intx_irq(struct irq_data *data)
 	raw_spin_unlock_irqrestore(&port->lock, flags);
 }
 
-static void mc_unmask_intx_irq(struct irq_data *data)
+static void plda_unmask_intx_irq(struct irq_data *data)
 {
 	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
 	void __iomem *bridge_base_addr = port->bridge_addr;
@@ -532,24 +536,24 @@ static void mc_unmask_intx_irq(struct irq_data *data)
 	raw_spin_unlock_irqrestore(&port->lock, flags);
 }
 
-static struct irq_chip mc_intx_irq_chip = {
-	.name = "Microchip PCIe INTx",
-	.irq_ack = mc_ack_intx_irq,
-	.irq_mask = mc_mask_intx_irq,
-	.irq_unmask = mc_unmask_intx_irq,
+static struct irq_chip plda_intx_irq_chip = {
+	.name = "PLDA PCIe INTx",
+	.irq_ack = plda_ack_intx_irq,
+	.irq_mask = plda_mask_intx_irq,
+	.irq_unmask = plda_unmask_intx_irq,
 };
 
-static int mc_pcie_intx_map(struct irq_domain *domain, unsigned int irq,
-			    irq_hw_number_t hwirq)
+static int plda_pcie_intx_map(struct irq_domain *domain, unsigned int irq,
+			      irq_hw_number_t hwirq)
 {
-	irq_set_chip_and_handler(irq, &mc_intx_irq_chip, handle_level_irq);
+	irq_set_chip_and_handler(irq, &plda_intx_irq_chip, handle_level_irq);
 	irq_set_chip_data(irq, domain->host_data);
 
 	return 0;
 }
 
 static const struct irq_domain_ops intx_domain_ops = {
-	.map = mc_pcie_intx_map,
+	.map = plda_pcie_intx_map,
 };
 
 static inline u32 reg_to_event(u32 reg, struct event_map field)
@@ -609,7 +613,7 @@ static u32 local_events(struct mc_pcie *port)
 	return val;
 }
 
-static u32 get_events(struct plda_pcie_rp *port)
+static u32 mc_get_events(struct plda_pcie_rp *port)
 {
 	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
 	u32 events = 0;
@@ -638,7 +642,7 @@ static irqreturn_t mc_event_handler(int irq, void *dev_id)
 	return IRQ_HANDLED;
 }
 
-static void mc_handle_event(struct irq_desc *desc)
+static void plda_handle_event(struct irq_desc *desc)
 {
 	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
 	unsigned long events;
@@ -647,7 +651,7 @@ static void mc_handle_event(struct irq_desc *desc)
 
 	chained_irq_enter(chip, desc);
 
-	events = get_events(port);
+	events = mc_get_events(port);
 
 	for_each_set_bit(bit, &events, NUM_EVENTS)
 		generic_handle_domain_irq(port->event_domain, bit);
@@ -741,8 +745,8 @@ static struct irq_chip mc_event_irq_chip = {
 	.irq_unmask = mc_unmask_event_irq,
 };
 
-static int mc_pcie_event_map(struct irq_domain *domain, unsigned int irq,
-			     irq_hw_number_t hwirq)
+static int plda_pcie_event_map(struct irq_domain *domain, unsigned int irq,
+			       irq_hw_number_t hwirq)
 {
 	irq_set_chip_and_handler(irq, &mc_event_irq_chip, handle_level_irq);
 	irq_set_chip_data(irq, domain->host_data);
@@ -750,8 +754,8 @@ static int mc_pcie_event_map(struct irq_domain *domain, unsigned int irq,
 	return 0;
 }
 
-static const struct irq_domain_ops event_domain_ops = {
-	.map = mc_pcie_event_map,
+static const struct irq_domain_ops plda_event_domain_ops = {
+	.map = plda_pcie_event_map,
 };
 
 static inline void mc_pcie_deinit_clk(void *data)
@@ -799,7 +803,7 @@ static int mc_pcie_init_clks(struct device *dev)
 	return 0;
 }
 
-static int mc_pcie_init_irq_domains(struct plda_pcie_rp *port)
+static int plda_pcie_init_irq_domains(struct plda_pcie_rp *port)
 {
 	struct device *dev = port->dev;
 	struct device_node *node = dev->of_node;
@@ -813,7 +817,8 @@ static int mc_pcie_init_irq_domains(struct plda_pcie_rp *port)
 	}
 
 	port->event_domain = irq_domain_add_linear(pcie_intc_node, NUM_EVENTS,
-						   &event_domain_ops, port);
+						   &plda_event_domain_ops,
+						   port);
 	if (!port->event_domain) {
 		dev_err(dev, "failed to get event domain\n");
 		of_node_put(pcie_intc_node);
@@ -835,7 +840,7 @@ static int mc_pcie_init_irq_domains(struct plda_pcie_rp *port)
 	of_node_put(pcie_intc_node);
 	raw_spin_lock_init(&port->lock);
 
-	return mc_allocate_msi_domains(port);
+	return plda_allocate_msi_domains(port);
 }
 
 static inline void mc_clear_secs(struct mc_pcie *port)
@@ -898,14 +903,14 @@ static void mc_disable_interrupts(struct mc_pcie *port)
 	writel_relaxed(GENMASK(31, 0), bridge_base_addr + ISTATUS_HOST);
 }
 
-static int mc_init_interrupts(struct platform_device *pdev, struct plda_pcie_rp *port)
+static int plda_init_interrupts(struct platform_device *pdev, struct plda_pcie_rp *port)
 {
 	struct device *dev = &pdev->dev;
 	int irq;
 	int i, intx_irq, msi_irq, event_irq;
 	int ret;
 
-	ret = mc_pcie_init_irq_domains(port);
+	ret = plda_pcie_init_irq_domains(port);
 	if (ret) {
 		dev_err(dev, "failed creating IRQ domains\n");
 		return ret;
@@ -938,7 +943,7 @@ static int mc_init_interrupts(struct platform_device *pdev, struct plda_pcie_rp
 	}
 
 	/* Plug the INTx chained handler */
-	irq_set_chained_handler_and_data(intx_irq, mc_handle_intx, port);
+	irq_set_chained_handler_and_data(intx_irq, plda_handle_intx, port);
 
 	msi_irq = irq_create_mapping(port->event_domain,
 				     EVENT_LOCAL_PM_MSI_INT_MSI);
@@ -946,10 +951,10 @@ static int mc_init_interrupts(struct platform_device *pdev, struct plda_pcie_rp
 		return -ENXIO;
 
 	/* Plug the MSI chained handler */
-	irq_set_chained_handler_and_data(msi_irq, mc_handle_msi, port);
+	irq_set_chained_handler_and_data(msi_irq, plda_handle_msi, port);
 
 	/* Plug the main event chained handler */
-	irq_set_chained_handler_and_data(irq, mc_handle_event, port);
+	irq_set_chained_handler_and_data(irq, plda_handle_event, port);
 
 	return 0;
 }
@@ -977,7 +982,7 @@ static int mc_platform_init(struct pci_config_window *cfg)
 		return ret;
 
 	/* Address translation is up; safe to enable interrupts */
-	ret = mc_init_interrupts(pdev, &port->plda);
+	ret = plda_init_interrupts(pdev, &port->plda);
 	if (ret)
 		return ret;
 
-- 
2.17.1

[PATCH v15,RESEND 11/23] PCI: microchip: Add num_events field to struct plda_pcie_rp

[Minda Chen]

The number of events is different across platforms. In order to share
interrupt processing code, add a variable that defines the number of
events so that it can be set per-platform instead of hardcoding it.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 drivers/pci/controller/plda/pcie-microchip-host.c | 8 +++++---
 drivers/pci/controller/plda/pcie-plda.h           | 1 +
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index 18bc352db389..0a5cd8b214cd 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -653,7 +653,7 @@ static void plda_handle_event(struct irq_desc *desc)
 
 	events = mc_get_events(port);
 
-	for_each_set_bit(bit, &events, NUM_EVENTS)
+	for_each_set_bit(bit, &events, port->num_events)
 		generic_handle_domain_irq(port->event_domain, bit);
 
 	chained_irq_exit(chip, desc);
@@ -816,7 +816,8 @@ static int plda_pcie_init_irq_domains(struct plda_pcie_rp *port)
 		return -EINVAL;
 	}
 
-	port->event_domain = irq_domain_add_linear(pcie_intc_node, NUM_EVENTS,
+	port->event_domain = irq_domain_add_linear(pcie_intc_node,
+						   port->num_events,
 						   &plda_event_domain_ops,
 						   port);
 	if (!port->event_domain) {
@@ -920,7 +921,7 @@ static int plda_init_interrupts(struct platform_device *pdev, struct plda_pcie_r
 	if (irq < 0)
 		return -ENODEV;
 
-	for (i = 0; i < NUM_EVENTS; i++) {
+	for (i = 0; i < port->num_events; i++) {
 		event_irq = irq_create_mapping(port->event_domain, i);
 		if (!event_irq) {
 			dev_err(dev, "failed to map hwirq %d\n", i);
@@ -1012,6 +1013,7 @@ static int mc_host_probe(struct platform_device *pdev)
 
 	bridge_base_addr = port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
 	plda->bridge_addr = bridge_base_addr;
+	plda->num_events = NUM_EVENTS;
 
 	/* Allow enabling MSI by disabling MSI-X */
 	val = readl(bridge_base_addr + PCIE_PCI_IRQ_DW0);
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index e277a5452b5d..f7e900b395f8 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -124,6 +124,7 @@ struct plda_pcie_rp {
 	raw_spinlock_t lock;
 	struct plda_msi msi;
 	void __iomem *bridge_addr;
+	int num_events;
 };
 
 void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
-- 
2.17.1

[PATCH v15,RESEND 12/23] PCI: microchip: Add request_event_irq() callback function

[Minda Chen]

As PLDA dts binding doc(Documentation/devicetree/bindings/pci/
plda,xpressrich3-axi-common.yaml) showed, PLDA PCIe contains an interrupt
controller. Microchip Polarfire PCIe add some PCIe interrupts base on
PLDA IP interrupt controller.

Microchip Polarfire PCIe additional intrerrupts:
(defined in drivers/pci/controller/plda/pcie-microchip-host.c)
EVENT_PCIE_L2_EXIT
EVENT_PCIE_HOTRST_EXIT
EVENT_PCIE_DLUP_EXIT
EVENT_SEC_TX_RAM_SEC_ERR
EVENT_SEC_RX_RAM_SEC_ERR
....

Both codes of request interrupts and mc_event_handler() contain
additional interrupts symbol names, these can not be re-used. So add a
new plda_event_handler() functions, which implements PLDA interrupt
defalt handler, add request_event_irq() callback function to compat
Microchip Polorfire PCIe additional interrupts.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Acked-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../pci/controller/plda/pcie-microchip-host.c | 31 ++++++++++++++++---
 drivers/pci/controller/plda/pcie-plda.h       |  5 +++
 2 files changed, 32 insertions(+), 4 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index 0a5cd8b214cd..bf5ce33ee275 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -642,6 +642,11 @@ static irqreturn_t mc_event_handler(int irq, void *dev_id)
 	return IRQ_HANDLED;
 }
 
+static irqreturn_t plda_event_handler(int irq, void *dev_id)
+{
+	return IRQ_HANDLED;
+}
+
 static void plda_handle_event(struct irq_desc *desc)
 {
 	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
@@ -803,6 +808,17 @@ static int mc_pcie_init_clks(struct device *dev)
 	return 0;
 }
 
+static int mc_request_event_irq(struct plda_pcie_rp *plda, int event_irq,
+				int event)
+{
+	return devm_request_irq(plda->dev, event_irq, mc_event_handler,
+				0, event_cause[event].sym, plda);
+}
+
+static const struct plda_event mc_event = {
+	.request_event_irq = mc_request_event_irq,
+};
+
 static int plda_pcie_init_irq_domains(struct plda_pcie_rp *port)
 {
 	struct device *dev = port->dev;
@@ -904,7 +920,9 @@ static void mc_disable_interrupts(struct mc_pcie *port)
 	writel_relaxed(GENMASK(31, 0), bridge_base_addr + ISTATUS_HOST);
 }
 
-static int plda_init_interrupts(struct platform_device *pdev, struct plda_pcie_rp *port)
+static int plda_init_interrupts(struct platform_device *pdev,
+				struct plda_pcie_rp *port,
+				const struct plda_event *event)
 {
 	struct device *dev = &pdev->dev;
 	int irq;
@@ -928,8 +946,13 @@ static int plda_init_interrupts(struct platform_device *pdev, struct plda_pcie_r
 			return -ENXIO;
 		}
 
-		ret = devm_request_irq(dev, event_irq, mc_event_handler,
-				       0, event_cause[i].sym, port);
+		if (event->request_event_irq)
+			ret = event->request_event_irq(port, event_irq, i);
+		else
+			ret = devm_request_irq(dev, event_irq,
+					       plda_event_handler,
+					       0, NULL, port);
+
 		if (ret) {
 			dev_err(dev, "failed to request IRQ %d\n", event_irq);
 			return ret;
@@ -983,7 +1006,7 @@ static int mc_platform_init(struct pci_config_window *cfg)
 		return ret;
 
 	/* Address translation is up; safe to enable interrupts */
-	ret = plda_init_interrupts(pdev, &port->plda);
+	ret = plda_init_interrupts(pdev, &port->plda, &mc_event);
 	if (ret)
 		return ret;
 
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index f7e900b395f8..935686bba837 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -127,6 +127,11 @@ struct plda_pcie_rp {
 	int num_events;
 };
 
+struct plda_event {
+	int (*request_event_irq)(struct plda_pcie_rp *pcie,
+				 int event_irq, int event);
+};
+
 void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
 			    phys_addr_t axi_addr, phys_addr_t pci_addr,
 			    size_t size);
-- 
2.17.1

[PATCH v15,RESEND 13/23] PCI: microchip: Add INTx and MSI event num to struct plda_event

[Minda Chen]

The INTx and MSI interrupt event num is different across platforms, so
add two event num fields in struct plda_event.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Acked-by: Conor Dooley <conor.dooley@microchip.com>
---
 drivers/pci/controller/plda/pcie-microchip-host.c | 6 ++++--
 drivers/pci/controller/plda/pcie-plda.h           | 2 ++
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index bf5ce33ee275..8a51d3aa7e88 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -817,6 +817,8 @@ static int mc_request_event_irq(struct plda_pcie_rp *plda, int event_irq,
 
 static const struct plda_event mc_event = {
 	.request_event_irq = mc_request_event_irq,
+	.intx_event        = EVENT_LOCAL_PM_MSI_INT_INTX,
+	.msi_event         = EVENT_LOCAL_PM_MSI_INT_MSI,
 };
 
 static int plda_pcie_init_irq_domains(struct plda_pcie_rp *port)
@@ -960,7 +962,7 @@ static int plda_init_interrupts(struct platform_device *pdev,
 	}
 
 	intx_irq = irq_create_mapping(port->event_domain,
-				      EVENT_LOCAL_PM_MSI_INT_INTX);
+				      event->intx_event);
 	if (!intx_irq) {
 		dev_err(dev, "failed to map INTx interrupt\n");
 		return -ENXIO;
@@ -970,7 +972,7 @@ static int plda_init_interrupts(struct platform_device *pdev,
 	irq_set_chained_handler_and_data(intx_irq, plda_handle_intx, port);
 
 	msi_irq = irq_create_mapping(port->event_domain,
-				     EVENT_LOCAL_PM_MSI_INT_MSI);
+				     event->msi_event);
 	if (!msi_irq)
 		return -ENXIO;
 
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index 935686bba837..89172ce18237 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -130,6 +130,8 @@ struct plda_pcie_rp {
 struct plda_event {
 	int (*request_event_irq)(struct plda_pcie_rp *pcie,
 				 int event_irq, int event);
+	int intx_event;
+	int msi_event;
 };
 
 void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
-- 
2.17.1

[PATCH v15,RESEND 14/23] PCI: microchip: Add get_events() callback and add PLDA get_event()

[Minda Chen]

As PLDA dts binding doc(Documentation/devicetree/bindings/pci/
plda,xpressrich3-axi-common.yaml) showed, PLDA PCIe contains an interrupt
controller.

PolarFire implements its own PCIe interrupts, additional to the regular
PCIe interrupts, due to lack of an MSI controller, so the interrupt to
event number mapping is different to the PLDA regular interrupts,
necessitating a custom get_events() implementation.

Microchip Polarfire PCIe additional intrerrupts:
(defined in drivers/pci/controller/plda/pcie-microchip-host.c)
EVENT_PCIE_L2_EXIT
EVENT_PCIE_HOTRST_EXIT
EVENT_PCIE_DLUP_EXIT
EVENT_SEC_TX_RAM_SEC_ERR
EVENT_SEC_RX_RAM_SEC_ERR
....

plda_get_events() adds interrupt register to PLDA event num mapping codes.
All The PLDA interrupts can be seen in new added graph.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Acked-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../pci/controller/plda/pcie-microchip-host.c | 35 ++++++++++++++++++-
 drivers/pci/controller/plda/pcie-plda.h       | 32 +++++++++++++++++
 2 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index 8a51d3aa7e88..b3df373a2141 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -626,6 +626,26 @@ static u32 mc_get_events(struct plda_pcie_rp *port)
 	return events;
 }
 
+static u32 plda_get_events(struct plda_pcie_rp *port)
+{
+	u32 events, val, origin;
+
+	origin = readl_relaxed(port->bridge_addr + ISTATUS_LOCAL);
+
+	/* MSI event and sys events */
+	val = (origin & SYS_AND_MSI_MASK) >> PM_MSI_INT_MSI_SHIFT;
+	events = val << (PM_MSI_INT_MSI_SHIFT - PCI_NUM_INTX + 1);
+
+	/* INTx events */
+	if (origin & PM_MSI_INT_INTX_MASK)
+		events |= BIT(PM_MSI_INT_INTX_SHIFT);
+
+	/* remains are same with register */
+	events |= origin & GENMASK(P_ATR_EVT_DOORBELL_SHIFT, 0);
+
+	return events;
+}
+
 static irqreturn_t mc_event_handler(int irq, void *dev_id)
 {
 	struct plda_pcie_rp *port = dev_id;
@@ -656,7 +676,7 @@ static void plda_handle_event(struct irq_desc *desc)
 
 	chained_irq_enter(chip, desc);
 
-	events = mc_get_events(port);
+	events = port->event_ops->get_events(port);
 
 	for_each_set_bit(bit, &events, port->num_events)
 		generic_handle_domain_irq(port->event_domain, bit);
@@ -750,6 +770,10 @@ static struct irq_chip mc_event_irq_chip = {
 	.irq_unmask = mc_unmask_event_irq,
 };
 
+static const struct plda_event_ops plda_event_ops = {
+	.get_events = plda_get_events,
+};
+
 static int plda_pcie_event_map(struct irq_domain *domain, unsigned int irq,
 			       irq_hw_number_t hwirq)
 {
@@ -815,6 +839,10 @@ static int mc_request_event_irq(struct plda_pcie_rp *plda, int event_irq,
 				0, event_cause[event].sym, plda);
 }
 
+static const struct plda_event_ops mc_event_ops = {
+	.get_events = mc_get_events,
+};
+
 static const struct plda_event mc_event = {
 	.request_event_irq = mc_request_event_irq,
 	.intx_event        = EVENT_LOCAL_PM_MSI_INT_INTX,
@@ -931,6 +959,9 @@ static int plda_init_interrupts(struct platform_device *pdev,
 	int i, intx_irq, msi_irq, event_irq;
 	int ret;
 
+	if (!port->event_ops)
+		port->event_ops = &plda_event_ops;
+
 	ret = plda_pcie_init_irq_domains(port);
 	if (ret) {
 		dev_err(dev, "failed creating IRQ domains\n");
@@ -1007,6 +1038,8 @@ static int mc_platform_init(struct pci_config_window *cfg)
 	if (ret)
 		return ret;
 
+	port->plda.event_ops = &mc_event_ops;
+
 	/* Address translation is up; safe to enable interrupts */
 	ret = plda_init_interrupts(pdev, &port->plda, &mc_event);
 	if (ret)
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index 89172ce18237..e0e5e7cc8434 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -58,6 +58,7 @@
 #define  PM_MSI_INT_EVENTS_SHIFT		30
 #define  PM_MSI_INT_SYS_ERR_MASK		0x80000000u
 #define  PM_MSI_INT_SYS_ERR_SHIFT		31
+#define  SYS_AND_MSI_MASK			GENMASK(31, 28)
 #define  NUM_LOCAL_EVENTS			15
 #define ISTATUS_LOCAL				0x184
 #define IMASK_HOST				0x188
@@ -108,6 +109,36 @@ enum plda_int_event {
 
 #define PLDA_MAX_EVENT_NUM			(PLDA_NUM_DMA_EVENTS + PLDA_INT_EVENT_NUM)
 
+/*
+ * PLDA interrupt register
+ *
+ * 31         27     23              15           7          0
+ * +--+--+--+-+------+-+-+-+-+-+-+-+-+-----------+-----------+
+ * |12|11|10|9| intx |7|6|5|4|3|2|1|0| DMA error | DMA end   |
+ * +--+--+--+-+------+-+-+-+-+-+-+-+-+-----------+-----------+
+ * bit 0-7  DMA interrupt end : reserved for vendor implement
+ * bit 8-15 DMA error : reserved for vendor implement
+ * 0:  AXI post error (PLDA_AXI_POST_ERR)
+ * 1:  AXI fetch error (PLDA_AXI_FETCH_ERR)
+ * 2:  AXI discard error (PLDA_AXI_DISCARD_ERR)
+ * 3:  AXI doorbell (PLDA_PCIE_DOORBELL)
+ * 4:  PCIe post error (PLDA_PCIE_POST_ERR)
+ * 5:  PCIe fetch error (PLDA_PCIE_FETCH_ERR)
+ * 6:  PCIe discard error (PLDA_PCIE_DISCARD_ERR)
+ * 7:  PCIe doorbell (PLDA_PCIE_DOORBELL)
+ * 8:  4 INTx interruts (PLDA_INTX)
+ * 9:  MSI interrupt (PLDA_MSI)
+ * 10: AER event (PLDA_AER_EVENT)
+ * 11: PM/LTR/Hotplug (PLDA_MISC_EVENTS)
+ * 12: System error (PLDA_SYS_ERR)
+ */
+
+struct plda_pcie_rp;
+
+struct plda_event_ops {
+	u32 (*get_events)(struct plda_pcie_rp *pcie);
+};
+
 struct plda_msi {
 	struct mutex lock;		/* Protect used bitmap */
 	struct irq_domain *msi_domain;
@@ -123,6 +154,7 @@ struct plda_pcie_rp {
 	struct irq_domain *event_domain;
 	raw_spinlock_t lock;
 	struct plda_msi msi;
+	const struct plda_event_ops *event_ops;
 	void __iomem *bridge_addr;
 	int num_events;
 };
-- 
2.17.1

[PATCH v15,RESEND 15/23] PCI: microchip: Add event irqchip field to host port and add PLDA irqchip

[Minda Chen]

As PLDA dts binding doc(Documentation/devicetree/bindings/pci/
plda,xpressrich3-axi-common.yaml) showed, PLDA PCIe contains an interrupt
controller.

Microchip PolarFire PCIE event IRQs includes PLDA interrupts and
Polarfire their own interrupts. The interrupt irqchip ops includes
ack/mask/unmask interrupt ops, which will write correct registers.
Microchip Polarfire PCIe additional interrupts require to write Polarfire
SoC self-defined registers. So Microchip PCIe event irqchip ops can not
be re-used.

To support PLDA its own event IRQ process, implements PLDA irqchip ops and
add event irqchip field to struct pcie_plda_rp.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Acked-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../pci/controller/plda/pcie-microchip-host.c | 66 ++++++++++++++++++-
 drivers/pci/controller/plda/pcie-plda.h       |  3 +
 2 files changed, 68 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index b3df373a2141..beaf5c27da84 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -770,6 +770,64 @@ static struct irq_chip mc_event_irq_chip = {
 	.irq_unmask = mc_unmask_event_irq,
 };
 
+static u32 plda_hwirq_to_mask(int hwirq)
+{
+	u32 mask;
+
+	/* hwirq 23 - 0 are the same with register */
+	if (hwirq < EVENT_PM_MSI_INT_INTX)
+		mask = BIT(hwirq);
+	else if (hwirq == EVENT_PM_MSI_INT_INTX)
+		mask = PM_MSI_INT_INTX_MASK;
+	else
+		mask = BIT(hwirq + PCI_NUM_INTX - 1);
+
+	return mask;
+}
+
+static void plda_ack_event_irq(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+
+	writel_relaxed(plda_hwirq_to_mask(data->hwirq),
+		       port->bridge_addr + ISTATUS_LOCAL);
+}
+
+static void plda_mask_event_irq(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	u32 mask, val;
+
+	mask = plda_hwirq_to_mask(data->hwirq);
+
+	raw_spin_lock(&port->lock);
+	val = readl_relaxed(port->bridge_addr + IMASK_LOCAL);
+	val &= ~mask;
+	writel_relaxed(val, port->bridge_addr + IMASK_LOCAL);
+	raw_spin_unlock(&port->lock);
+}
+
+static void plda_unmask_event_irq(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	u32 mask, val;
+
+	mask = plda_hwirq_to_mask(data->hwirq);
+
+	raw_spin_lock(&port->lock);
+	val = readl_relaxed(port->bridge_addr + IMASK_LOCAL);
+	val |= mask;
+	writel_relaxed(val, port->bridge_addr + IMASK_LOCAL);
+	raw_spin_unlock(&port->lock);
+}
+
+static struct irq_chip plda_event_irq_chip = {
+	.name = "PLDA PCIe EVENT",
+	.irq_ack = plda_ack_event_irq,
+	.irq_mask = plda_mask_event_irq,
+	.irq_unmask = plda_unmask_event_irq,
+};
+
 static const struct plda_event_ops plda_event_ops = {
 	.get_events = plda_get_events,
 };
@@ -777,7 +835,9 @@ static const struct plda_event_ops plda_event_ops = {
 static int plda_pcie_event_map(struct irq_domain *domain, unsigned int irq,
 			       irq_hw_number_t hwirq)
 {
-	irq_set_chip_and_handler(irq, &mc_event_irq_chip, handle_level_irq);
+	struct plda_pcie_rp *port = (void *)domain->host_data;
+
+	irq_set_chip_and_handler(irq, port->event_irq_chip, handle_level_irq);
 	irq_set_chip_data(irq, domain->host_data);
 
 	return 0;
@@ -962,6 +1022,9 @@ static int plda_init_interrupts(struct platform_device *pdev,
 	if (!port->event_ops)
 		port->event_ops = &plda_event_ops;
 
+	if (!port->event_irq_chip)
+		port->event_irq_chip = &plda_event_irq_chip;
+
 	ret = plda_pcie_init_irq_domains(port);
 	if (ret) {
 		dev_err(dev, "failed creating IRQ domains\n");
@@ -1039,6 +1102,7 @@ static int mc_platform_init(struct pci_config_window *cfg)
 		return ret;
 
 	port->plda.event_ops = &mc_event_ops;
+	port->plda.event_irq_chip = &mc_event_irq_chip;
 
 	/* Address translation is up; safe to enable interrupts */
 	ret = plda_init_interrupts(pdev, &port->plda, &mc_event);
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index e0e5e7cc8434..a3ce01735bea 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -107,6 +107,8 @@ enum plda_int_event {
 
 #define PLDA_NUM_DMA_EVENTS			16
 
+#define EVENT_PM_MSI_INT_INTX			(PLDA_NUM_DMA_EVENTS + PLDA_INTX)
+#define EVENT_PM_MSI_INT_MSI			(PLDA_NUM_DMA_EVENTS + PLDA_MSI)
 #define PLDA_MAX_EVENT_NUM			(PLDA_NUM_DMA_EVENTS + PLDA_INT_EVENT_NUM)
 
 /*
@@ -155,6 +157,7 @@ struct plda_pcie_rp {
 	raw_spinlock_t lock;
 	struct plda_msi msi;
 	const struct plda_event_ops *event_ops;
+	const struct irq_chip *event_irq_chip;
 	void __iomem *bridge_addr;
 	int num_events;
 };
-- 
2.17.1

[PATCH v15,RESEND 16/23] PCI: microchip: Move IRQ functions to pcie-plda-host.c

[Minda Chen]

Move IRQ related functions to pcie-plda-host.c for re-use these codes.

The re-use code including MSI, INTx, event interrupts and IRQ init
functions.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Acked-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../pci/controller/plda/pcie-microchip-host.c | 467 -----------------
 drivers/pci/controller/plda/pcie-plda-host.c  | 473 ++++++++++++++++++
 drivers/pci/controller/plda/pcie-plda.h       |   3 +
 3 files changed, 476 insertions(+), 467 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index beaf5c27da84..105964306b71 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -318,244 +318,6 @@ static void mc_pcie_enable_msi(struct mc_pcie *port, void __iomem *ecam)
 		       ecam + MC_MSI_CAP_CTRL_OFFSET + PCI_MSI_ADDRESS_HI);
 }
 
-static void plda_handle_msi(struct irq_desc *desc)
-{
-	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
-	struct irq_chip *chip = irq_desc_get_chip(desc);
-	struct device *dev = port->dev;
-	struct plda_msi *msi = &port->msi;
-	void __iomem *bridge_base_addr = port->bridge_addr;
-	unsigned long status;
-	u32 bit;
-	int ret;
-
-	chained_irq_enter(chip, desc);
-
-	status = readl_relaxed(bridge_base_addr + ISTATUS_LOCAL);
-	if (status & PM_MSI_INT_MSI_MASK) {
-		writel_relaxed(status & PM_MSI_INT_MSI_MASK, bridge_base_addr + ISTATUS_LOCAL);
-		status = readl_relaxed(bridge_base_addr + ISTATUS_MSI);
-		for_each_set_bit(bit, &status, msi->num_vectors) {
-			ret = generic_handle_domain_irq(msi->dev_domain, bit);
-			if (ret)
-				dev_err_ratelimited(dev, "bad MSI IRQ %d\n",
-						    bit);
-		}
-	}
-
-	chained_irq_exit(chip, desc);
-}
-
-static void plda_msi_bottom_irq_ack(struct irq_data *data)
-{
-	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
-	void __iomem *bridge_base_addr = port->bridge_addr;
-	u32 bitpos = data->hwirq;
-
-	writel_relaxed(BIT(bitpos), bridge_base_addr + ISTATUS_MSI);
-}
-
-static void plda_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
-{
-	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
-	phys_addr_t addr = port->msi.vector_phy;
-
-	msg->address_lo = lower_32_bits(addr);
-	msg->address_hi = upper_32_bits(addr);
-	msg->data = data->hwirq;
-
-	dev_dbg(port->dev, "msi#%x address_hi %#x address_lo %#x\n",
-		(int)data->hwirq, msg->address_hi, msg->address_lo);
-}
-
-static int plda_msi_set_affinity(struct irq_data *irq_data,
-				 const struct cpumask *mask, bool force)
-{
-	return -EINVAL;
-}
-
-static struct irq_chip plda_msi_bottom_irq_chip = {
-	.name = "PLDA MSI",
-	.irq_ack = plda_msi_bottom_irq_ack,
-	.irq_compose_msi_msg = plda_compose_msi_msg,
-	.irq_set_affinity = plda_msi_set_affinity,
-};
-
-static int plda_irq_msi_domain_alloc(struct irq_domain *domain,
-				     unsigned int virq,
-				     unsigned int nr_irqs,
-				     void *args)
-{
-	struct plda_pcie_rp *port = domain->host_data;
-	struct plda_msi *msi = &port->msi;
-	unsigned long bit;
-
-	mutex_lock(&msi->lock);
-	bit = find_first_zero_bit(msi->used, msi->num_vectors);
-	if (bit >= msi->num_vectors) {
-		mutex_unlock(&msi->lock);
-		return -ENOSPC;
-	}
-
-	set_bit(bit, msi->used);
-
-	irq_domain_set_info(domain, virq, bit, &plda_msi_bottom_irq_chip,
-			    domain->host_data, handle_edge_irq, NULL, NULL);
-
-	mutex_unlock(&msi->lock);
-
-	return 0;
-}
-
-static void plda_irq_msi_domain_free(struct irq_domain *domain,
-				     unsigned int virq,
-				     unsigned int nr_irqs)
-{
-	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
-	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(d);
-	struct plda_msi *msi = &port->msi;
-
-	mutex_lock(&msi->lock);
-
-	if (test_bit(d->hwirq, msi->used))
-		__clear_bit(d->hwirq, msi->used);
-	else
-		dev_err(port->dev, "trying to free unused MSI%lu\n", d->hwirq);
-
-	mutex_unlock(&msi->lock);
-}
-
-static const struct irq_domain_ops msi_domain_ops = {
-	.alloc	= plda_irq_msi_domain_alloc,
-	.free	= plda_irq_msi_domain_free,
-};
-
-static struct irq_chip plda_msi_irq_chip = {
-	.name = "PLDA PCIe MSI",
-	.irq_ack = irq_chip_ack_parent,
-	.irq_mask = pci_msi_mask_irq,
-	.irq_unmask = pci_msi_unmask_irq,
-};
-
-static struct msi_domain_info plda_msi_domain_info = {
-	.flags = (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
-		  MSI_FLAG_PCI_MSIX),
-	.chip = &plda_msi_irq_chip,
-};
-
-static int plda_allocate_msi_domains(struct plda_pcie_rp *port)
-{
-	struct device *dev = port->dev;
-	struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
-	struct plda_msi *msi = &port->msi;
-
-	mutex_init(&port->msi.lock);
-
-	msi->dev_domain = irq_domain_add_linear(NULL, msi->num_vectors,
-						&msi_domain_ops, port);
-	if (!msi->dev_domain) {
-		dev_err(dev, "failed to create IRQ domain\n");
-		return -ENOMEM;
-	}
-
-	msi->msi_domain = pci_msi_create_irq_domain(fwnode,
-						    &plda_msi_domain_info,
-						    msi->dev_domain);
-	if (!msi->msi_domain) {
-		dev_err(dev, "failed to create MSI domain\n");
-		irq_domain_remove(msi->dev_domain);
-		return -ENOMEM;
-	}
-
-	return 0;
-}
-
-static void plda_handle_intx(struct irq_desc *desc)
-{
-	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
-	struct irq_chip *chip = irq_desc_get_chip(desc);
-	struct device *dev = port->dev;
-	void __iomem *bridge_base_addr = port->bridge_addr;
-	unsigned long status;
-	u32 bit;
-	int ret;
-
-	chained_irq_enter(chip, desc);
-
-	status = readl_relaxed(bridge_base_addr + ISTATUS_LOCAL);
-	if (status & PM_MSI_INT_INTX_MASK) {
-		status &= PM_MSI_INT_INTX_MASK;
-		status >>= PM_MSI_INT_INTX_SHIFT;
-		for_each_set_bit(bit, &status, PCI_NUM_INTX) {
-			ret = generic_handle_domain_irq(port->intx_domain, bit);
-			if (ret)
-				dev_err_ratelimited(dev, "bad INTx IRQ %d\n",
-						    bit);
-		}
-	}
-
-	chained_irq_exit(chip, desc);
-}
-
-static void plda_ack_intx_irq(struct irq_data *data)
-{
-	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
-	void __iomem *bridge_base_addr = port->bridge_addr;
-	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
-
-	writel_relaxed(mask, bridge_base_addr + ISTATUS_LOCAL);
-}
-
-static void plda_mask_intx_irq(struct irq_data *data)
-{
-	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
-	void __iomem *bridge_base_addr = port->bridge_addr;
-	unsigned long flags;
-	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
-	u32 val;
-
-	raw_spin_lock_irqsave(&port->lock, flags);
-	val = readl_relaxed(bridge_base_addr + IMASK_LOCAL);
-	val &= ~mask;
-	writel_relaxed(val, bridge_base_addr + IMASK_LOCAL);
-	raw_spin_unlock_irqrestore(&port->lock, flags);
-}
-
-static void plda_unmask_intx_irq(struct irq_data *data)
-{
-	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
-	void __iomem *bridge_base_addr = port->bridge_addr;
-	unsigned long flags;
-	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
-	u32 val;
-
-	raw_spin_lock_irqsave(&port->lock, flags);
-	val = readl_relaxed(bridge_base_addr + IMASK_LOCAL);
-	val |= mask;
-	writel_relaxed(val, bridge_base_addr + IMASK_LOCAL);
-	raw_spin_unlock_irqrestore(&port->lock, flags);
-}
-
-static struct irq_chip plda_intx_irq_chip = {
-	.name = "PLDA PCIe INTx",
-	.irq_ack = plda_ack_intx_irq,
-	.irq_mask = plda_mask_intx_irq,
-	.irq_unmask = plda_unmask_intx_irq,
-};
-
-static int plda_pcie_intx_map(struct irq_domain *domain, unsigned int irq,
-			      irq_hw_number_t hwirq)
-{
-	irq_set_chip_and_handler(irq, &plda_intx_irq_chip, handle_level_irq);
-	irq_set_chip_data(irq, domain->host_data);
-
-	return 0;
-}
-
-static const struct irq_domain_ops intx_domain_ops = {
-	.map = plda_pcie_intx_map,
-};
-
 static inline u32 reg_to_event(u32 reg, struct event_map field)
 {
 	return (reg & field.reg_mask) ? BIT(field.event_bit) : 0;
@@ -626,26 +388,6 @@ static u32 mc_get_events(struct plda_pcie_rp *port)
 	return events;
 }
 
-static u32 plda_get_events(struct plda_pcie_rp *port)
-{
-	u32 events, val, origin;
-
-	origin = readl_relaxed(port->bridge_addr + ISTATUS_LOCAL);
-
-	/* MSI event and sys events */
-	val = (origin & SYS_AND_MSI_MASK) >> PM_MSI_INT_MSI_SHIFT;
-	events = val << (PM_MSI_INT_MSI_SHIFT - PCI_NUM_INTX + 1);
-
-	/* INTx events */
-	if (origin & PM_MSI_INT_INTX_MASK)
-		events |= BIT(PM_MSI_INT_INTX_SHIFT);
-
-	/* remains are same with register */
-	events |= origin & GENMASK(P_ATR_EVT_DOORBELL_SHIFT, 0);
-
-	return events;
-}
-
 static irqreturn_t mc_event_handler(int irq, void *dev_id)
 {
 	struct plda_pcie_rp *port = dev_id;
@@ -662,28 +404,6 @@ static irqreturn_t mc_event_handler(int irq, void *dev_id)
 	return IRQ_HANDLED;
 }
 
-static irqreturn_t plda_event_handler(int irq, void *dev_id)
-{
-	return IRQ_HANDLED;
-}
-
-static void plda_handle_event(struct irq_desc *desc)
-{
-	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
-	unsigned long events;
-	u32 bit;
-	struct irq_chip *chip = irq_desc_get_chip(desc);
-
-	chained_irq_enter(chip, desc);
-
-	events = port->event_ops->get_events(port);
-
-	for_each_set_bit(bit, &events, port->num_events)
-		generic_handle_domain_irq(port->event_domain, bit);
-
-	chained_irq_exit(chip, desc);
-}
-
 static void mc_ack_event_irq(struct irq_data *data)
 {
 	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
@@ -770,83 +490,6 @@ static struct irq_chip mc_event_irq_chip = {
 	.irq_unmask = mc_unmask_event_irq,
 };
 
-static u32 plda_hwirq_to_mask(int hwirq)
-{
-	u32 mask;
-
-	/* hwirq 23 - 0 are the same with register */
-	if (hwirq < EVENT_PM_MSI_INT_INTX)
-		mask = BIT(hwirq);
-	else if (hwirq == EVENT_PM_MSI_INT_INTX)
-		mask = PM_MSI_INT_INTX_MASK;
-	else
-		mask = BIT(hwirq + PCI_NUM_INTX - 1);
-
-	return mask;
-}
-
-static void plda_ack_event_irq(struct irq_data *data)
-{
-	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
-
-	writel_relaxed(plda_hwirq_to_mask(data->hwirq),
-		       port->bridge_addr + ISTATUS_LOCAL);
-}
-
-static void plda_mask_event_irq(struct irq_data *data)
-{
-	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
-	u32 mask, val;
-
-	mask = plda_hwirq_to_mask(data->hwirq);
-
-	raw_spin_lock(&port->lock);
-	val = readl_relaxed(port->bridge_addr + IMASK_LOCAL);
-	val &= ~mask;
-	writel_relaxed(val, port->bridge_addr + IMASK_LOCAL);
-	raw_spin_unlock(&port->lock);
-}
-
-static void plda_unmask_event_irq(struct irq_data *data)
-{
-	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
-	u32 mask, val;
-
-	mask = plda_hwirq_to_mask(data->hwirq);
-
-	raw_spin_lock(&port->lock);
-	val = readl_relaxed(port->bridge_addr + IMASK_LOCAL);
-	val |= mask;
-	writel_relaxed(val, port->bridge_addr + IMASK_LOCAL);
-	raw_spin_unlock(&port->lock);
-}
-
-static struct irq_chip plda_event_irq_chip = {
-	.name = "PLDA PCIe EVENT",
-	.irq_ack = plda_ack_event_irq,
-	.irq_mask = plda_mask_event_irq,
-	.irq_unmask = plda_unmask_event_irq,
-};
-
-static const struct plda_event_ops plda_event_ops = {
-	.get_events = plda_get_events,
-};
-
-static int plda_pcie_event_map(struct irq_domain *domain, unsigned int irq,
-			       irq_hw_number_t hwirq)
-{
-	struct plda_pcie_rp *port = (void *)domain->host_data;
-
-	irq_set_chip_and_handler(irq, port->event_irq_chip, handle_level_irq);
-	irq_set_chip_data(irq, domain->host_data);
-
-	return 0;
-}
-
-static const struct irq_domain_ops plda_event_domain_ops = {
-	.map = plda_pcie_event_map,
-};
-
 static inline void mc_pcie_deinit_clk(void *data)
 {
 	struct clk *clk = data;
@@ -909,47 +552,6 @@ static const struct plda_event mc_event = {
 	.msi_event         = EVENT_LOCAL_PM_MSI_INT_MSI,
 };
 
-static int plda_pcie_init_irq_domains(struct plda_pcie_rp *port)
-{
-	struct device *dev = port->dev;
-	struct device_node *node = dev->of_node;
-	struct device_node *pcie_intc_node;
-
-	/* Setup INTx */
-	pcie_intc_node = of_get_next_child(node, NULL);
-	if (!pcie_intc_node) {
-		dev_err(dev, "failed to find PCIe Intc node\n");
-		return -EINVAL;
-	}
-
-	port->event_domain = irq_domain_add_linear(pcie_intc_node,
-						   port->num_events,
-						   &plda_event_domain_ops,
-						   port);
-	if (!port->event_domain) {
-		dev_err(dev, "failed to get event domain\n");
-		of_node_put(pcie_intc_node);
-		return -ENOMEM;
-	}
-
-	irq_domain_update_bus_token(port->event_domain, DOMAIN_BUS_NEXUS);
-
-	port->intx_domain = irq_domain_add_linear(pcie_intc_node, PCI_NUM_INTX,
-						  &intx_domain_ops, port);
-	if (!port->intx_domain) {
-		dev_err(dev, "failed to get an INTx IRQ domain\n");
-		of_node_put(pcie_intc_node);
-		return -ENOMEM;
-	}
-
-	irq_domain_update_bus_token(port->intx_domain, DOMAIN_BUS_WIRED);
-
-	of_node_put(pcie_intc_node);
-	raw_spin_lock_init(&port->lock);
-
-	return plda_allocate_msi_domains(port);
-}
-
 static inline void mc_clear_secs(struct mc_pcie *port)
 {
 	void __iomem *ctrl_base_addr = port->axi_base_addr + MC_PCIE_CTRL_ADDR;
@@ -1010,75 +612,6 @@ static void mc_disable_interrupts(struct mc_pcie *port)
 	writel_relaxed(GENMASK(31, 0), bridge_base_addr + ISTATUS_HOST);
 }
 
-static int plda_init_interrupts(struct platform_device *pdev,
-				struct plda_pcie_rp *port,
-				const struct plda_event *event)
-{
-	struct device *dev = &pdev->dev;
-	int irq;
-	int i, intx_irq, msi_irq, event_irq;
-	int ret;
-
-	if (!port->event_ops)
-		port->event_ops = &plda_event_ops;
-
-	if (!port->event_irq_chip)
-		port->event_irq_chip = &plda_event_irq_chip;
-
-	ret = plda_pcie_init_irq_domains(port);
-	if (ret) {
-		dev_err(dev, "failed creating IRQ domains\n");
-		return ret;
-	}
-
-	irq = platform_get_irq(pdev, 0);
-	if (irq < 0)
-		return -ENODEV;
-
-	for (i = 0; i < port->num_events; i++) {
-		event_irq = irq_create_mapping(port->event_domain, i);
-		if (!event_irq) {
-			dev_err(dev, "failed to map hwirq %d\n", i);
-			return -ENXIO;
-		}
-
-		if (event->request_event_irq)
-			ret = event->request_event_irq(port, event_irq, i);
-		else
-			ret = devm_request_irq(dev, event_irq,
-					       plda_event_handler,
-					       0, NULL, port);
-
-		if (ret) {
-			dev_err(dev, "failed to request IRQ %d\n", event_irq);
-			return ret;
-		}
-	}
-
-	intx_irq = irq_create_mapping(port->event_domain,
-				      event->intx_event);
-	if (!intx_irq) {
-		dev_err(dev, "failed to map INTx interrupt\n");
-		return -ENXIO;
-	}
-
-	/* Plug the INTx chained handler */
-	irq_set_chained_handler_and_data(intx_irq, plda_handle_intx, port);
-
-	msi_irq = irq_create_mapping(port->event_domain,
-				     event->msi_event);
-	if (!msi_irq)
-		return -ENXIO;
-
-	/* Plug the MSI chained handler */
-	irq_set_chained_handler_and_data(msi_irq, plda_handle_msi, port);
-
-	/* Plug the main event chained handler */
-	irq_set_chained_handler_and_data(irq, plda_handle_event, port);
-
-	return 0;
-}
-
 static int mc_platform_init(struct pci_config_window *cfg)
 {
 	struct device *dev = cfg->parent;
diff --git a/drivers/pci/controller/plda/pcie-plda-host.c b/drivers/pci/controller/plda/pcie-plda-host.c
index 05ea68baebfb..98c51e594efe 100644
--- a/drivers/pci/controller/plda/pcie-plda-host.c
+++ b/drivers/pci/controller/plda/pcie-plda-host.c
@@ -7,10 +7,483 @@
  * Author: Daire McNamara <daire.mcnamara@microchip.com>
  */
 
+#include <linux/irqchip/chained_irq.h>
+#include <linux/irqdomain.h>
+#include <linux/msi.h>
+#include <linux/pci_regs.h>
 #include <linux/pci-ecam.h>
 
 #include "pcie-plda.h"
 
+static void plda_handle_msi(struct irq_desc *desc)
+{
+	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
+	struct irq_chip *chip = irq_desc_get_chip(desc);
+	struct device *dev = port->dev;
+	struct plda_msi *msi = &port->msi;
+	void __iomem *bridge_base_addr = port->bridge_addr;
+	unsigned long status;
+	u32 bit;
+	int ret;
+
+	chained_irq_enter(chip, desc);
+
+	status = readl_relaxed(bridge_base_addr + ISTATUS_LOCAL);
+	if (status & PM_MSI_INT_MSI_MASK) {
+		writel_relaxed(status & PM_MSI_INT_MSI_MASK,
+			       bridge_base_addr + ISTATUS_LOCAL);
+		status = readl_relaxed(bridge_base_addr + ISTATUS_MSI);
+		for_each_set_bit(bit, &status, msi->num_vectors) {
+			ret = generic_handle_domain_irq(msi->dev_domain, bit);
+			if (ret)
+				dev_err_ratelimited(dev, "bad MSI IRQ %d\n",
+						    bit);
+		}
+	}
+
+	chained_irq_exit(chip, desc);
+}
+
+static void plda_msi_bottom_irq_ack(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	void __iomem *bridge_base_addr = port->bridge_addr;
+	u32 bitpos = data->hwirq;
+
+	writel_relaxed(BIT(bitpos), bridge_base_addr + ISTATUS_MSI);
+}
+
+static void plda_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	phys_addr_t addr = port->msi.vector_phy;
+
+	msg->address_lo = lower_32_bits(addr);
+	msg->address_hi = upper_32_bits(addr);
+	msg->data = data->hwirq;
+
+	dev_dbg(port->dev, "msi#%x address_hi %#x address_lo %#x\n",
+		(int)data->hwirq, msg->address_hi, msg->address_lo);
+}
+
+static int plda_msi_set_affinity(struct irq_data *irq_data,
+				 const struct cpumask *mask, bool force)
+{
+	return -EINVAL;
+}
+
+static struct irq_chip plda_msi_bottom_irq_chip = {
+	.name = "PLDA MSI",
+	.irq_ack = plda_msi_bottom_irq_ack,
+	.irq_compose_msi_msg = plda_compose_msi_msg,
+	.irq_set_affinity = plda_msi_set_affinity,
+};
+
+static int plda_irq_msi_domain_alloc(struct irq_domain *domain,
+				     unsigned int virq,
+				     unsigned int nr_irqs,
+				     void *args)
+{
+	struct plda_pcie_rp *port = domain->host_data;
+	struct plda_msi *msi = &port->msi;
+	unsigned long bit;
+
+	mutex_lock(&msi->lock);
+	bit = find_first_zero_bit(msi->used, msi->num_vectors);
+	if (bit >= msi->num_vectors) {
+		mutex_unlock(&msi->lock);
+		return -ENOSPC;
+	}
+
+	set_bit(bit, msi->used);
+
+	irq_domain_set_info(domain, virq, bit, &plda_msi_bottom_irq_chip,
+			    domain->host_data, handle_edge_irq, NULL, NULL);
+
+	mutex_unlock(&msi->lock);
+
+	return 0;
+}
+
+static void plda_irq_msi_domain_free(struct irq_domain *domain,
+				     unsigned int virq,
+				     unsigned int nr_irqs)
+{
+	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(d);
+	struct plda_msi *msi = &port->msi;
+
+	mutex_lock(&msi->lock);
+
+	if (test_bit(d->hwirq, msi->used))
+		__clear_bit(d->hwirq, msi->used);
+	else
+		dev_err(port->dev, "trying to free unused MSI%lu\n", d->hwirq);
+
+	mutex_unlock(&msi->lock);
+}
+
+static const struct irq_domain_ops msi_domain_ops = {
+	.alloc	= plda_irq_msi_domain_alloc,
+	.free	= plda_irq_msi_domain_free,
+};
+
+static struct irq_chip plda_msi_irq_chip = {
+	.name = "PLDA PCIe MSI",
+	.irq_ack = irq_chip_ack_parent,
+	.irq_mask = pci_msi_mask_irq,
+	.irq_unmask = pci_msi_unmask_irq,
+};
+
+static struct msi_domain_info plda_msi_domain_info = {
+	.flags = (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
+		  MSI_FLAG_PCI_MSIX),
+	.chip = &plda_msi_irq_chip,
+};
+
+static int plda_allocate_msi_domains(struct plda_pcie_rp *port)
+{
+	struct device *dev = port->dev;
+	struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
+	struct plda_msi *msi = &port->msi;
+
+	mutex_init(&port->msi.lock);
+
+	msi->dev_domain = irq_domain_add_linear(NULL, msi->num_vectors,
+						&msi_domain_ops, port);
+	if (!msi->dev_domain) {
+		dev_err(dev, "failed to create IRQ domain\n");
+		return -ENOMEM;
+	}
+
+	msi->msi_domain = pci_msi_create_irq_domain(fwnode,
+						    &plda_msi_domain_info,
+						    msi->dev_domain);
+	if (!msi->msi_domain) {
+		dev_err(dev, "failed to create MSI domain\n");
+		irq_domain_remove(msi->dev_domain);
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
+static void plda_handle_intx(struct irq_desc *desc)
+{
+	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
+	struct irq_chip *chip = irq_desc_get_chip(desc);
+	struct device *dev = port->dev;
+	void __iomem *bridge_base_addr = port->bridge_addr;
+	unsigned long status;
+	u32 bit;
+	int ret;
+
+	chained_irq_enter(chip, desc);
+
+	status = readl_relaxed(bridge_base_addr + ISTATUS_LOCAL);
+	if (status & PM_MSI_INT_INTX_MASK) {
+		status &= PM_MSI_INT_INTX_MASK;
+		status >>= PM_MSI_INT_INTX_SHIFT;
+		for_each_set_bit(bit, &status, PCI_NUM_INTX) {
+			ret = generic_handle_domain_irq(port->intx_domain, bit);
+			if (ret)
+				dev_err_ratelimited(dev, "bad INTx IRQ %d\n",
+						    bit);
+		}
+	}
+
+	chained_irq_exit(chip, desc);
+}
+
+static void plda_ack_intx_irq(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	void __iomem *bridge_base_addr = port->bridge_addr;
+	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
+
+	writel_relaxed(mask, bridge_base_addr + ISTATUS_LOCAL);
+}
+
+static void plda_mask_intx_irq(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	void __iomem *bridge_base_addr = port->bridge_addr;
+	unsigned long flags;
+	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
+	u32 val;
+
+	raw_spin_lock_irqsave(&port->lock, flags);
+	val = readl_relaxed(bridge_base_addr + IMASK_LOCAL);
+	val &= ~mask;
+	writel_relaxed(val, bridge_base_addr + IMASK_LOCAL);
+	raw_spin_unlock_irqrestore(&port->lock, flags);
+}
+
+static void plda_unmask_intx_irq(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	void __iomem *bridge_base_addr = port->bridge_addr;
+	unsigned long flags;
+	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
+	u32 val;
+
+	raw_spin_lock_irqsave(&port->lock, flags);
+	val = readl_relaxed(bridge_base_addr + IMASK_LOCAL);
+	val |= mask;
+	writel_relaxed(val, bridge_base_addr + IMASK_LOCAL);
+	raw_spin_unlock_irqrestore(&port->lock, flags);
+}
+
+static struct irq_chip plda_intx_irq_chip = {
+	.name = "PLDA PCIe INTx",
+	.irq_ack = plda_ack_intx_irq,
+	.irq_mask = plda_mask_intx_irq,
+	.irq_unmask = plda_unmask_intx_irq,
+};
+
+static int plda_pcie_intx_map(struct irq_domain *domain, unsigned int irq,
+			      irq_hw_number_t hwirq)
+{
+	irq_set_chip_and_handler(irq, &plda_intx_irq_chip, handle_level_irq);
+	irq_set_chip_data(irq, domain->host_data);
+
+	return 0;
+}
+
+static const struct irq_domain_ops intx_domain_ops = {
+	.map = plda_pcie_intx_map,
+};
+
+static u32 plda_get_events(struct plda_pcie_rp *port)
+{
+	u32 events, val, origin;
+
+	origin = readl_relaxed(port->bridge_addr + ISTATUS_LOCAL);
+
+	/* MSI event and sys events */
+	val = (origin & SYS_AND_MSI_MASK) >> PM_MSI_INT_MSI_SHIFT;
+	events = val << (PM_MSI_INT_MSI_SHIFT - PCI_NUM_INTX + 1);
+
+	/* INTx events */
+	if (origin & PM_MSI_INT_INTX_MASK)
+		events |= BIT(PM_MSI_INT_INTX_SHIFT);
+
+	/* remains are same with register */
+	events |= origin & GENMASK(P_ATR_EVT_DOORBELL_SHIFT, 0);
+
+	return events;
+}
+
+static irqreturn_t plda_event_handler(int irq, void *dev_id)
+{
+	return IRQ_HANDLED;
+}
+
+static void plda_handle_event(struct irq_desc *desc)
+{
+	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
+	unsigned long events;
+	u32 bit;
+	struct irq_chip *chip = irq_desc_get_chip(desc);
+
+	chained_irq_enter(chip, desc);
+
+	events = port->event_ops->get_events(port);
+
+	for_each_set_bit(bit, &events, port->num_events)
+		generic_handle_domain_irq(port->event_domain, bit);
+
+	chained_irq_exit(chip, desc);
+}
+
+static u32 plda_hwirq_to_mask(int hwirq)
+{
+	u32 mask;
+
+	/* hwirq 23 - 0 are the same with register */
+	if (hwirq < EVENT_PM_MSI_INT_INTX)
+		mask = BIT(hwirq);
+	else if (hwirq == EVENT_PM_MSI_INT_INTX)
+		mask = PM_MSI_INT_INTX_MASK;
+	else
+		mask = BIT(hwirq + PCI_NUM_INTX - 1);
+
+	return mask;
+}
+
+static void plda_ack_event_irq(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+
+	writel_relaxed(plda_hwirq_to_mask(data->hwirq),
+		       port->bridge_addr + ISTATUS_LOCAL);
+}
+
+static void plda_mask_event_irq(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	u32 mask, val;
+
+	mask = plda_hwirq_to_mask(data->hwirq);
+
+	raw_spin_lock(&port->lock);
+	val = readl_relaxed(port->bridge_addr + IMASK_LOCAL);
+	val &= ~mask;
+	writel_relaxed(val, port->bridge_addr + IMASK_LOCAL);
+	raw_spin_unlock(&port->lock);
+}
+
+static void plda_unmask_event_irq(struct irq_data *data)
+{
+	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
+	u32 mask, val;
+
+	mask = plda_hwirq_to_mask(data->hwirq);
+
+	raw_spin_lock(&port->lock);
+	val = readl_relaxed(port->bridge_addr + IMASK_LOCAL);
+	val |= mask;
+	writel_relaxed(val, port->bridge_addr + IMASK_LOCAL);
+	raw_spin_unlock(&port->lock);
+}
+
+static struct irq_chip plda_event_irq_chip = {
+	.name = "PLDA PCIe EVENT",
+	.irq_ack = plda_ack_event_irq,
+	.irq_mask = plda_mask_event_irq,
+	.irq_unmask = plda_unmask_event_irq,
+};
+
+static const struct plda_event_ops plda_event_ops = {
+	.get_events = plda_get_events,
+};
+
+static int plda_pcie_event_map(struct irq_domain *domain, unsigned int irq,
+			       irq_hw_number_t hwirq)
+{
+	struct plda_pcie_rp *port = (void *)domain->host_data;
+
+	irq_set_chip_and_handler(irq, port->event_irq_chip, handle_level_irq);
+	irq_set_chip_data(irq, domain->host_data);
+
+	return 0;
+}
+
+static const struct irq_domain_ops plda_event_domain_ops = {
+	.map = plda_pcie_event_map,
+};
+
+static int plda_pcie_init_irq_domains(struct plda_pcie_rp *port)
+{
+	struct device *dev = port->dev;
+	struct device_node *node = dev->of_node;
+	struct device_node *pcie_intc_node;
+
+	/* Setup INTx */
+	pcie_intc_node = of_get_next_child(node, NULL);
+	if (!pcie_intc_node) {
+		dev_err(dev, "failed to find PCIe Intc node\n");
+		return -EINVAL;
+	}
+
+	port->event_domain = irq_domain_add_linear(pcie_intc_node,
+						   port->num_events,
+						   &plda_event_domain_ops,
+						   port);
+	if (!port->event_domain) {
+		dev_err(dev, "failed to get event domain\n");
+		of_node_put(pcie_intc_node);
+		return -ENOMEM;
+	}
+
+	irq_domain_update_bus_token(port->event_domain, DOMAIN_BUS_NEXUS);
+
+	port->intx_domain = irq_domain_add_linear(pcie_intc_node, PCI_NUM_INTX,
+						  &intx_domain_ops, port);
+	if (!port->intx_domain) {
+		dev_err(dev, "failed to get an INTx IRQ domain\n");
+		of_node_put(pcie_intc_node);
+		return -ENOMEM;
+	}
+
+	irq_domain_update_bus_token(port->intx_domain, DOMAIN_BUS_WIRED);
+
+	of_node_put(pcie_intc_node);
+	raw_spin_lock_init(&port->lock);
+
+	return plda_allocate_msi_domains(port);
+}
+
+int plda_init_interrupts(struct platform_device *pdev,
+			 struct plda_pcie_rp *port,
+			 const struct plda_event *event)
+{
+	struct device *dev = &pdev->dev;
+	int irq;
+	int i, intx_irq, msi_irq, event_irq;
+	int ret;
+
+	if (!port->event_ops)
+		port->event_ops = &plda_event_ops;
+
+	if (!port->event_irq_chip)
+		port->event_irq_chip = &plda_event_irq_chip;
+
+	ret = plda_pcie_init_irq_domains(port);
+	if (ret) {
+		dev_err(dev, "failed creating IRQ domains\n");
+		return ret;
+	}
+
+	irq = platform_get_irq(pdev, 0);
+	if (irq < 0)
+		return -ENODEV;
+
+	for (i = 0; i < port->num_events; i++) {
+		event_irq = irq_create_mapping(port->event_domain, i);
+		if (!event_irq) {
+			dev_err(dev, "failed to map hwirq %d\n", i);
+			return -ENXIO;
+		}
+
+		if (event->request_event_irq)
+			ret = event->request_event_irq(port, event_irq, i);
+		else
+			ret = devm_request_irq(dev, event_irq,
+					       plda_event_handler,
+					       0, NULL, port);
+
+		if (ret) {
+			dev_err(dev, "failed to request IRQ %d\n", event_irq);
+			return ret;
+		}
+	}
+
+	intx_irq = irq_create_mapping(port->event_domain,
+				      event->intx_event);
+	if (!intx_irq) {
+		dev_err(dev, "failed to map INTx interrupt\n");
+		return -ENXIO;
+	}
+
+	/* Plug the INTx chained handler */
+	irq_set_chained_handler_and_data(intx_irq, plda_handle_intx, port);
+
+	msi_irq = irq_create_mapping(port->event_domain,
+				     event->msi_event);
+	if (!msi_irq)
+		return -ENXIO;
+
+	/* Plug the MSI chained handler */
+	irq_set_chained_handler_and_data(msi_irq, plda_handle_msi, port);
+
+	/* Plug the main event chained handler */
+	irq_set_chained_handler_and_data(irq, plda_handle_event, port);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(plda_init_interrupts);
+
 void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
 			    phys_addr_t axi_addr, phys_addr_t pci_addr,
 			    size_t size)
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index a3ce01735bea..6672a231a4bc 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -169,6 +169,9 @@ struct plda_event {
 	int msi_event;
 };
 
+int plda_init_interrupts(struct platform_device *pdev,
+			 struct plda_pcie_rp *port,
+			 const struct plda_event *event);
 void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
 			    phys_addr_t axi_addr, phys_addr_t pci_addr,
 			    size_t size);
-- 
2.17.1

[PATCH v15,RESEND 17/23] PCI: plda: Add event bitmap field to struct plda_pcie_rp

[Minda Chen]

For PLDA DMA interrupts are not all implemented. The non-implemented
interrupts should be masked. So add a bitmap field to mask the non-
implemented interrupts.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
---
 drivers/pci/controller/plda/pcie-microchip-host.c | 1 +
 drivers/pci/controller/plda/pcie-plda-host.c      | 6 ++++--
 drivers/pci/controller/plda/pcie-plda.h           | 1 +
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
index 105964306b71..48f60a04b740 100644
--- a/drivers/pci/controller/plda/pcie-microchip-host.c
+++ b/drivers/pci/controller/plda/pcie-microchip-host.c
@@ -636,6 +636,7 @@ static int mc_platform_init(struct pci_config_window *cfg)
 
 	port->plda.event_ops = &mc_event_ops;
 	port->plda.event_irq_chip = &mc_event_irq_chip;
+	port->plda.events_bitmap = GENMASK(NUM_EVENTS - 1, 0);
 
 	/* Address translation is up; safe to enable interrupts */
 	ret = plda_init_interrupts(pdev, &port->plda, &mc_event);
diff --git a/drivers/pci/controller/plda/pcie-plda-host.c b/drivers/pci/controller/plda/pcie-plda-host.c
index 98c51e594efe..a040e7e5492f 100644
--- a/drivers/pci/controller/plda/pcie-plda-host.c
+++ b/drivers/pci/controller/plda/pcie-plda-host.c
@@ -290,6 +290,7 @@ static void plda_handle_event(struct irq_desc *desc)
 
 	events = port->event_ops->get_events(port);
 
+	events &= port->events_bitmap;
 	for_each_set_bit(bit, &events, port->num_events)
 		generic_handle_domain_irq(port->event_domain, bit);
 
@@ -420,8 +421,9 @@ int plda_init_interrupts(struct platform_device *pdev,
 {
 	struct device *dev = &pdev->dev;
 	int irq;
-	int i, intx_irq, msi_irq, event_irq;
+	int intx_irq, msi_irq, event_irq;
 	int ret;
+	u32 i;
 
 	if (!port->event_ops)
 		port->event_ops = &plda_event_ops;
@@ -439,7 +441,7 @@ int plda_init_interrupts(struct platform_device *pdev,
 	if (irq < 0)
 		return -ENODEV;
 
-	for (i = 0; i < port->num_events; i++) {
+	for_each_set_bit(i, &port->events_bitmap, port->num_events) {
 		event_irq = irq_create_mapping(port->event_domain, i);
 		if (!event_irq) {
 			dev_err(dev, "failed to map hwirq %d\n", i);
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index 6672a231a4bc..443109d04d59 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -159,6 +159,7 @@ struct plda_pcie_rp {
 	const struct plda_event_ops *event_ops;
 	const struct irq_chip *event_irq_chip;
 	void __iomem *bridge_addr;
+	unsigned long events_bitmap;
 	int num_events;
 };
 
-- 
2.17.1

[PATCH v15,RESEND 18/23] PCI: plda: Add host init/deinit and map bus functions

[Minda Chen]

Add PLDA host plda_pcie_host_init()/plda_pcie_host_deinit() and map bus
function. So vendor can use it to init PLDA PCIe host core.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Mason Huo <mason.huo@starfivetech.com>
---
 drivers/pci/controller/plda/pcie-plda-host.c | 131 +++++++++++++++++--
 drivers/pci/controller/plda/pcie-plda.h      |  22 ++++
 2 files changed, 139 insertions(+), 14 deletions(-)

diff --git a/drivers/pci/controller/plda/pcie-plda-host.c b/drivers/pci/controller/plda/pcie-plda-host.c
index a040e7e5492f..a18923d7cea6 100644
--- a/drivers/pci/controller/plda/pcie-plda-host.c
+++ b/drivers/pci/controller/plda/pcie-plda-host.c
@@ -3,6 +3,7 @@
  * PLDA PCIe XpressRich host controller driver
  *
  * Copyright (C) 2023 Microchip Co. Ltd
+ *		      StarFive Co. Ltd
  *
  * Author: Daire McNamara <daire.mcnamara@microchip.com>
  */
@@ -15,6 +16,15 @@
 
 #include "pcie-plda.h"
 
+void __iomem *plda_pcie_map_bus(struct pci_bus *bus, unsigned int devfn,
+				int where)
+{
+	struct plda_pcie_rp *pcie = bus->sysdata;
+
+	return pcie->config_base + PCIE_ECAM_OFFSET(bus->number, devfn, where);
+}
+EXPORT_SYMBOL_GPL(plda_pcie_map_bus);
+
 static void plda_handle_msi(struct irq_desc *desc)
 {
 	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
@@ -420,9 +430,7 @@ int plda_init_interrupts(struct platform_device *pdev,
 			 const struct plda_event *event)
 {
 	struct device *dev = &pdev->dev;
-	int irq;
-	int intx_irq, msi_irq, event_irq;
-	int ret;
+	int event_irq, ret;
 	u32 i;
 
 	if (!port->event_ops)
@@ -437,8 +445,8 @@ int plda_init_interrupts(struct platform_device *pdev,
 		return ret;
 	}
 
-	irq = platform_get_irq(pdev, 0);
-	if (irq < 0)
+	port->irq = platform_get_irq(pdev, 0);
+	if (port->irq < 0)
 		return -ENODEV;
 
 	for_each_set_bit(i, &port->events_bitmap, port->num_events) {
@@ -461,26 +469,26 @@ int plda_init_interrupts(struct platform_device *pdev,
 		}
 	}
 
-	intx_irq = irq_create_mapping(port->event_domain,
-				      event->intx_event);
-	if (!intx_irq) {
+	port->intx_irq = irq_create_mapping(port->event_domain,
+					    event->intx_event);
+	if (!port->intx_irq) {
 		dev_err(dev, "failed to map INTx interrupt\n");
 		return -ENXIO;
 	}
 
 	/* Plug the INTx chained handler */
-	irq_set_chained_handler_and_data(intx_irq, plda_handle_intx, port);
+	irq_set_chained_handler_and_data(port->intx_irq, plda_handle_intx, port);
 
-	msi_irq = irq_create_mapping(port->event_domain,
-				     event->msi_event);
-	if (!msi_irq)
+	port->msi_irq = irq_create_mapping(port->event_domain,
+					   event->msi_event);
+	if (!port->msi_irq)
 		return -ENXIO;
 
 	/* Plug the MSI chained handler */
-	irq_set_chained_handler_and_data(msi_irq, plda_handle_msi, port);
+	irq_set_chained_handler_and_data(port->msi_irq, plda_handle_msi, port);
 
 	/* Plug the main event chained handler */
-	irq_set_chained_handler_and_data(irq, plda_handle_event, port);
+	irq_set_chained_handler_and_data(port->irq, plda_handle_event, port);
 
 	return 0;
 }
@@ -546,3 +554,98 @@ int plda_pcie_setup_iomems(struct pci_host_bridge *bridge,
 	return 0;
 }
 EXPORT_SYMBOL_GPL(plda_pcie_setup_iomems);
+
+static void plda_pcie_irq_domain_deinit(struct plda_pcie_rp *pcie)
+{
+	irq_set_chained_handler_and_data(pcie->irq, NULL, NULL);
+	irq_set_chained_handler_and_data(pcie->msi_irq, NULL, NULL);
+	irq_set_chained_handler_and_data(pcie->intx_irq, NULL, NULL);
+
+	irq_domain_remove(pcie->msi.msi_domain);
+	irq_domain_remove(pcie->msi.dev_domain);
+
+	irq_domain_remove(pcie->intx_domain);
+	irq_domain_remove(pcie->event_domain);
+}
+
+int plda_pcie_host_init(struct plda_pcie_rp *port, struct pci_ops *ops,
+			const struct plda_event *plda_event)
+{
+	struct device *dev = port->dev;
+	struct pci_host_bridge *bridge;
+	struct platform_device *pdev = to_platform_device(dev);
+	struct resource *cfg_res;
+	int ret;
+
+	pdev = to_platform_device(dev);
+
+	port->bridge_addr =
+		devm_platform_ioremap_resource_byname(pdev, "apb");
+
+	if (IS_ERR(port->bridge_addr))
+		return dev_err_probe(dev, PTR_ERR(port->bridge_addr),
+				     "failed to map reg memory\n");
+
+	cfg_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "cfg");
+	if (!cfg_res)
+		return dev_err_probe(dev, -ENODEV,
+				     "failed to get config memory\n");
+
+	port->config_base = devm_ioremap_resource(dev, cfg_res);
+	if (IS_ERR(port->config_base))
+		return dev_err_probe(dev, PTR_ERR(port->config_base),
+				     "failed to map config memory\n");
+
+	bridge = devm_pci_alloc_host_bridge(dev, 0);
+	if (!bridge)
+		return dev_err_probe(dev, -ENOMEM,
+				     "failed to alloc bridge\n");
+
+	if (port->host_ops && port->host_ops->host_init) {
+		ret = port->host_ops->host_init(port);
+		if (ret)
+			return ret;
+	}
+
+	port->bridge = bridge;
+	plda_pcie_setup_window(port->bridge_addr, 0, cfg_res->start, 0,
+			       resource_size(cfg_res));
+	plda_pcie_setup_iomems(bridge, port);
+	plda_set_default_msi(&port->msi);
+	ret = plda_init_interrupts(pdev, port, plda_event);
+	if (ret)
+		goto err_host;
+
+	/* Set default bus ops */
+	bridge->ops = ops;
+	bridge->sysdata = port;
+
+	ret = pci_host_probe(bridge);
+	if (ret < 0) {
+		dev_err_probe(dev, ret, "failed to probe pci host\n");
+		goto err_probe;
+	}
+
+	return ret;
+
+err_probe:
+	plda_pcie_irq_domain_deinit(port);
+err_host:
+	if (port->host_ops && port->host_ops->host_deinit)
+		port->host_ops->host_deinit(port);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(plda_pcie_host_init);
+
+void plda_pcie_host_deinit(struct plda_pcie_rp *port)
+{
+	pci_stop_root_bus(port->bridge->bus);
+	pci_remove_root_bus(port->bridge->bus);
+
+	plda_pcie_irq_domain_deinit(port);
+
+	if (port->host_ops && port->host_ops->host_deinit)
+		port->host_ops->host_deinit(port);
+}
+EXPORT_SYMBOL_GPL(plda_pcie_host_deinit);
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index 443109d04d59..7b69891700a4 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -141,6 +141,11 @@ struct plda_event_ops {
 	u32 (*get_events)(struct plda_pcie_rp *pcie);
 };
 
+struct plda_pcie_host_ops {
+	int (*host_init)(struct plda_pcie_rp *pcie);
+	void (*host_deinit)(struct plda_pcie_rp *pcie);
+};
+
 struct plda_msi {
 	struct mutex lock;		/* Protect used bitmap */
 	struct irq_domain *msi_domain;
@@ -152,14 +157,20 @@ struct plda_msi {
 
 struct plda_pcie_rp {
 	struct device *dev;
+	struct pci_host_bridge *bridge;
 	struct irq_domain *intx_domain;
 	struct irq_domain *event_domain;
 	raw_spinlock_t lock;
 	struct plda_msi msi;
 	const struct plda_event_ops *event_ops;
 	const struct irq_chip *event_irq_chip;
+	const struct plda_pcie_host_ops *host_ops;
 	void __iomem *bridge_addr;
+	void __iomem *config_base;
 	unsigned long events_bitmap;
+	int irq;
+	int msi_irq;
+	int intx_irq;
 	int num_events;
 };
 
@@ -170,6 +181,8 @@ struct plda_event {
 	int msi_event;
 };
 
+void __iomem *plda_pcie_map_bus(struct pci_bus *bus, unsigned int devfn,
+				int where);
 int plda_init_interrupts(struct platform_device *pdev,
 			 struct plda_pcie_rp *port,
 			 const struct plda_event *event);
@@ -178,4 +191,13 @@ void plda_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
 			    size_t size);
 int plda_pcie_setup_iomems(struct pci_host_bridge *bridge,
 			   struct plda_pcie_rp *port);
+int plda_pcie_host_init(struct plda_pcie_rp *port, struct pci_ops *ops,
+			const struct plda_event *plda_event);
+void plda_pcie_host_deinit(struct plda_pcie_rp *pcie);
+
+static inline void plda_set_default_msi(struct plda_msi *msi)
+{
+	msi->vector_phy = IMSI_ADDR;
+	msi->num_vectors = PLDA_MAX_NUM_MSI_IRQS;
+}
 #endif
-- 
2.17.1

[PATCH v15,RESEND 19/23] dt-bindings: PCI: Add StarFive JH7110 PCIe controller

[Minda Chen]

Add StarFive JH7110 SoC PCIe controller dt-bindings. JH7110 using PLDA
XpressRICH PCIe host controller IP.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Hal Feng <hal.feng@starfivetech.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
Reviewed-by: Rob Herring <robh@kernel.org>
---
 .../bindings/pci/starfive,jh7110-pcie.yaml    | 120 ++++++++++++++++++
 1 file changed, 120 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/pci/starfive,jh7110-pcie.yaml

diff --git a/Documentation/devicetree/bindings/pci/starfive,jh7110-pcie.yaml b/Documentation/devicetree/bindings/pci/starfive,jh7110-pcie.yaml
new file mode 100644
index 000000000000..67151aaa3948
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/starfive,jh7110-pcie.yaml
@@ -0,0 +1,120 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/pci/starfive,jh7110-pcie.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: StarFive JH7110 PCIe host controller
+
+maintainers:
+  - Kevin Xie <kevin.xie@starfivetech.com>
+
+allOf:
+  - $ref: plda,xpressrich3-axi-common.yaml#
+
+properties:
+  compatible:
+    const: starfive,jh7110-pcie
+
+  clocks:
+    items:
+      - description: NOC bus clock
+      - description: Transport layer clock
+      - description: AXI MST0 clock
+      - description: APB clock
+
+  clock-names:
+    items:
+      - const: noc
+      - const: tl
+      - const: axi_mst0
+      - const: apb
+
+  resets:
+    items:
+      - description: AXI MST0 reset
+      - description: AXI SLAVE0 reset
+      - description: AXI SLAVE reset
+      - description: PCIE BRIDGE reset
+      - description: PCIE CORE reset
+      - description: PCIE APB reset
+
+  reset-names:
+    items:
+      - const: mst0
+      - const: slv0
+      - const: slv
+      - const: brg
+      - const: core
+      - const: apb
+
+  starfive,stg-syscon:
+    $ref: /schemas/types.yaml#/definitions/phandle-array
+    description:
+      The phandle to System Register Controller syscon node.
+
+  perst-gpios:
+    description: GPIO controlled connection to PERST# signal
+    maxItems: 1
+
+  phys:
+    description:
+      Specified PHY is attached to PCIe controller.
+    maxItems: 1
+
+required:
+  - clocks
+  - resets
+  - starfive,stg-syscon
+
+unevaluatedProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/gpio/gpio.h>
+    soc {
+        #address-cells = <2>;
+        #size-cells = <2>;
+
+        pcie@940000000 {
+            compatible = "starfive,jh7110-pcie";
+            reg = <0x9 0x40000000 0x0 0x10000000>,
+                  <0x0 0x2b000000 0x0 0x1000000>;
+            reg-names = "cfg", "apb";
+            #address-cells = <3>;
+            #size-cells = <2>;
+            #interrupt-cells = <1>;
+            device_type = "pci";
+            ranges = <0x82000000  0x0 0x30000000  0x0 0x30000000 0x0 0x08000000>,
+                     <0xc3000000  0x9 0x00000000  0x9 0x00000000 0x0 0x40000000>;
+            starfive,stg-syscon = <&stg_syscon>;
+            bus-range = <0x0 0xff>;
+            interrupt-parent = <&plic>;
+            interrupts = <56>;
+            interrupt-map-mask = <0x0 0x0 0x0 0x7>;
+            interrupt-map = <0x0 0x0 0x0 0x1 &pcie_intc0 0x1>,
+                            <0x0 0x0 0x0 0x2 &pcie_intc0 0x2>,
+                            <0x0 0x0 0x0 0x3 &pcie_intc0 0x3>,
+                            <0x0 0x0 0x0 0x4 &pcie_intc0 0x4>;
+            msi-controller;
+            clocks = <&syscrg 86>,
+                     <&stgcrg 10>,
+                     <&stgcrg 8>,
+                     <&stgcrg 9>;
+            clock-names = "noc", "tl", "axi_mst0", "apb";
+            resets = <&stgcrg 11>,
+                     <&stgcrg 12>,
+                     <&stgcrg 13>,
+                     <&stgcrg 14>,
+                     <&stgcrg 15>,
+                     <&stgcrg 16>;
+            perst-gpios = <&gpios 26 GPIO_ACTIVE_LOW>;
+            phys = <&pciephy0>;
+
+            pcie_intc0: interrupt-controller {
+                #address-cells = <0>;
+                #interrupt-cells = <1>;
+                interrupt-controller;
+            };
+        };
+    };
-- 
2.17.1

[PATCH v15,RESEND 20/23] PCI: Add PCIE_RESET_CONFIG_DEVICE_WAIT_MS waiting time value

[Minda Chen]

From: Kevin Xie <kevin.xie@starfivetech.com>

Add the PCIE_RESET_CONFIG_DEVICE_WAIT_MS macro to define the minimum
waiting time between exit from a conventional reset and sending the
first configuration request to the device.

As described in PCI base specification r6.0, section 6.6.1 <Conventional
Reset>, there are two different use cases of the value:

   - "With a Downstream Port that does not support Link speeds greater
     than 5.0 GT/s, software must wait a minimum of 100 ms following exit
     from a Conventional Reset before sending a Configuration Request to
     the device immediately below that Port."

   - "With a Downstream Port that supports Link speeds greater than
     5.0 GT/s, software must wait a minimum of 100 ms after Link training
     completes before sending a Configuration Request to the device
     immediately below that Port."

Signed-off-by: Kevin Xie <kevin.xie@starfivetech.com>
Reviewed-by: Mason Huo <mason.huo@starfivetech.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
---
 drivers/pci/pci.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index e9750b1b19ba..06d76439bcac 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -22,6 +22,22 @@
  */
 #define PCIE_PME_TO_L2_TIMEOUT_US	10000
 
+/*
+ * As described in PCI base specification r6.0, section 6.6.1 <Conventional
+ * Reset>, there are two different use cases of the value:
+ *
+ * - "With a Downstream Port that does not support Link speeds greater
+ *    than 5.0 GT/s, software must wait a minimum of 100 ms following exit
+ *    from a Conventional Reset before sending a Configuration Request to
+ *    the device immediately below that Port."
+ *
+ * - "With a Downstream Port that supports Link speeds greater than
+ *    5.0 GT/s, software must wait a minimum of 100 ms after Link training
+ *    completes before sending a Configuration Request to the device
+ *    immediately below that Port."
+ */
+#define PCIE_RESET_CONFIG_DEVICE_WAIT_MS	100
+
 extern const unsigned char pcie_link_speed[];
 extern bool pci_early_dump;
 
-- 
2.17.1

[PATCH v15,RESEND 21/23] PCI: starfive: Add JH7110 PCIe controller

[Minda Chen]

Add StarFive JH7110 SoC PCIe controller platform driver codes, JH7110
with PLDA host PCIe core.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Co-developed-by: Kevin Xie <kevin.xie@starfivetech.com>
Reviewed-by: Mason Huo <mason.huo@starfivetech.com>
---
 MAINTAINERS                                 |   7 +
 drivers/pci/controller/plda/Kconfig         |  12 +
 drivers/pci/controller/plda/Makefile        |   1 +
 drivers/pci/controller/plda/pcie-plda.h     |  71 ++-
 drivers/pci/controller/plda/pcie-starfive.c | 473 ++++++++++++++++++++
 5 files changed, 563 insertions(+), 1 deletion(-)
 create mode 100644 drivers/pci/controller/plda/pcie-starfive.c

diff --git a/MAINTAINERS b/MAINTAINERS
index fc9576e69a71..d3b0a0fb754f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17078,6 +17078,13 @@ S:	Maintained
 F:	Documentation/devicetree/bindings/pci/socionext,uniphier-pcie*
 F:	drivers/pci/controller/dwc/pcie-uniphier*
 
+PCIE DRIVER FOR STARFIVE JH71x0
+M:	Kevin Xie <kevin.xie@starfivetech.com>
+L:	linux-pci@vger.kernel.org
+S:	Maintained
+F:	Documentation/devicetree/bindings/pci/starfive*
+F:	drivers/pci/controller/plda/pcie-starfive.c
+
 PCIE DRIVER FOR ST SPEAR13XX
 M:	Pratyush Anand <pratyush.anand@gmail.com>
 L:	linux-pci@vger.kernel.org
diff --git a/drivers/pci/controller/plda/Kconfig b/drivers/pci/controller/plda/Kconfig
index e54a82ee94f5..c0e14146d7e4 100644
--- a/drivers/pci/controller/plda/Kconfig
+++ b/drivers/pci/controller/plda/Kconfig
@@ -15,4 +15,16 @@ config PCIE_MICROCHIP_HOST
 	  Say Y here if you want kernel to support the Microchip AXI PCIe
 	  Host Bridge driver.
 
+config PCIE_STARFIVE_HOST
+	tristate "StarFive PCIe host controller"
+	depends on PCI_MSI && OF
+	depends on ARCH_STARFIVE || COMPILE_TEST
+	select PCIE_PLDA_HOST
+	help
+	  Say Y here if you want to support the StarFive PCIe controller in
+	  host mode. StarFive PCIe controller uses PLDA PCIe core.
+
+	  If you choose to build this driver as module it will be dynamically
+	  linked and module will be called pcie-starfive.ko.
+
 endmenu
diff --git a/drivers/pci/controller/plda/Makefile b/drivers/pci/controller/plda/Makefile
index 4340ab007f44..0ac6851bed48 100644
--- a/drivers/pci/controller/plda/Makefile
+++ b/drivers/pci/controller/plda/Makefile
@@ -1,3 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_PCIE_PLDA_HOST) += pcie-plda-host.o
 obj-$(CONFIG_PCIE_MICROCHIP_HOST) += pcie-microchip-host.o
+obj-$(CONFIG_PCIE_STARFIVE_HOST) += pcie-starfive.o
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index 7b69891700a4..04e385758a2f 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -10,10 +10,20 @@
 #define PLDA_MAX_NUM_MSI_IRQS			32
 
 /* PCIe Bridge Phy Regs */
+#define GEN_SETTINGS				0x80
+#define  RP_ENABLE				1
+#define PCIE_PCI_IDS_DW1			0x9c
+#define  IDS_CLASS_CODE_SHIFT			16
+#define  REVISION_ID_MASK			GENMASK(7, 0)
+#define  CLASS_CODE_ID_MASK			GENMASK(31, 8)
 #define PCIE_PCI_IRQ_DW0			0xa8
 #define  MSIX_CAP_MASK				BIT(31)
 #define  NUM_MSI_MSGS_MASK			GENMASK(6, 4)
 #define  NUM_MSI_MSGS_SHIFT			4
+#define PCI_MISC				0xb4
+#define  PHY_FUNCTION_DIS			BIT(15)
+#define PCIE_WINROM				0xfc
+#define  PREF_MEM_WIN_64_SUPPORT		BIT(3)
 
 #define IMASK_LOCAL				0x180
 #define  DMA_END_ENGINE_0_MASK			0x00000000u
@@ -65,6 +75,8 @@
 #define ISTATUS_HOST				0x18c
 #define IMSI_ADDR				0x190
 #define ISTATUS_MSI				0x194
+#define PMSG_SUPPORT_RX				0x3f0
+#define  PMSG_LTR_SUPPORT			BIT(2)
 
 /* PCIe Master table init defines */
 #define ATR0_PCIE_WIN0_SRCADDR_PARAM		0x600u
@@ -86,6 +98,8 @@
 #define  PCIE_TX_RX_INTERFACE			0x00000000u
 #define  PCIE_CONFIG_INTERFACE			0x00000001u
 
+#define CONFIG_SPACE_ADDR_OFFSET		0x1000u
+
 #define ATR_ENTRY_SIZE				32
 
 enum plda_int_event {
@@ -200,4 +214,59 @@ static inline void plda_set_default_msi(struct plda_msi *msi)
 	msi->vector_phy = IMSI_ADDR;
 	msi->num_vectors = PLDA_MAX_NUM_MSI_IRQS;
 }
-#endif
+
+static inline void plda_pcie_enable_root_port(struct plda_pcie_rp *plda)
+{
+	u32 value;
+
+	value = readl_relaxed(plda->bridge_addr + GEN_SETTINGS);
+	value |= RP_ENABLE;
+	writel_relaxed(value, plda->bridge_addr + GEN_SETTINGS);
+}
+
+static inline void plda_pcie_set_standard_class(struct plda_pcie_rp *plda)
+{
+	u32 value;
+
+	/* set class code and reserve revision id */
+	value = readl_relaxed(plda->bridge_addr + PCIE_PCI_IDS_DW1);
+	value &= REVISION_ID_MASK;
+	value |= (PCI_CLASS_BRIDGE_PCI << IDS_CLASS_CODE_SHIFT);
+	writel_relaxed(value, plda->bridge_addr + PCIE_PCI_IDS_DW1);
+}
+
+static inline void plda_pcie_set_pref_win_64bit(struct plda_pcie_rp *plda)
+{
+	u32 value;
+
+	value = readl_relaxed(plda->bridge_addr + PCIE_WINROM);
+	value |= PREF_MEM_WIN_64_SUPPORT;
+	writel_relaxed(value, plda->bridge_addr + PCIE_WINROM);
+}
+
+static inline void plda_pcie_disable_ltr(struct plda_pcie_rp *plda)
+{
+	u32 value;
+
+	value = readl_relaxed(plda->bridge_addr + PMSG_SUPPORT_RX);
+	value &= ~PMSG_LTR_SUPPORT;
+	writel_relaxed(value, plda->bridge_addr + PMSG_SUPPORT_RX);
+}
+
+static inline void plda_pcie_disable_func(struct plda_pcie_rp *plda)
+{
+	u32 value;
+
+	value = readl_relaxed(plda->bridge_addr + PCI_MISC);
+	value |= PHY_FUNCTION_DIS;
+	writel_relaxed(value, plda->bridge_addr + PCI_MISC);
+}
+
+static inline void plda_pcie_write_rc_bar(struct plda_pcie_rp *plda, u64 val)
+{
+	void __iomem *addr = plda->bridge_addr + CONFIG_SPACE_ADDR_OFFSET;
+
+	writel_relaxed(lower_32_bits(val), addr + PCI_BASE_ADDRESS_0);
+	writel_relaxed(upper_32_bits(val), addr + PCI_BASE_ADDRESS_1);
+}
+#endif /* _PCIE_PLDA_H */
diff --git a/drivers/pci/controller/plda/pcie-starfive.c b/drivers/pci/controller/plda/pcie-starfive.c
new file mode 100644
index 000000000000..9bb9f0e29565
--- /dev/null
+++ b/drivers/pci/controller/plda/pcie-starfive.c
@@ -0,0 +1,473 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * PCIe host controller driver for StarFive JH7110 Soc.
+ *
+ * Copyright (C) 2023 StarFive Technology Co., Ltd.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/gpio/consumer.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/of_pci.h>
+#include <linux/pci.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/regmap.h>
+#include <linux/reset.h>
+#include "../../pci.h"
+
+#include "pcie-plda.h"
+
+#define PCIE_FUNC_NUM			4
+
+/* system control */
+#define STG_SYSCON_PCIE0_BASE			0x48
+#define STG_SYSCON_PCIE1_BASE			0x1f8
+
+#define STG_SYSCON_AR_OFFSET			0x78
+#define STG_SYSCON_AXI4_SLVL_AR_MASK		GENMASK(22, 8)
+#define STG_SYSCON_AXI4_SLVL_PHY_AR(x)		FIELD_PREP(GENMASK(20, 17), x)
+#define STG_SYSCON_AW_OFFSET			0x7c
+#define STG_SYSCON_AXI4_SLVL_AW_MASK		GENMASK(14, 0)
+#define STG_SYSCON_AXI4_SLVL_PHY_AW(x)		FIELD_PREP(GENMASK(12, 9), x)
+#define STG_SYSCON_CLKREQ			BIT(22)
+#define STG_SYSCON_CKREF_SRC_MASK		GENMASK(19, 18)
+#define STG_SYSCON_RP_NEP_OFFSET		0xe8
+#define STG_SYSCON_K_RP_NEP			BIT(8)
+#define STG_SYSCON_LNKSTA_OFFSET		0x170
+#define DATA_LINK_ACTIVE			BIT(5)
+
+/* Parameters for the waiting for link up routine */
+#define LINK_WAIT_MAX_RETRIES	10
+#define LINK_WAIT_USLEEP_MIN	90000
+#define LINK_WAIT_USLEEP_MAX	100000
+
+struct starfive_jh7110_pcie {
+	struct plda_pcie_rp plda;
+	struct reset_control *resets;
+	struct clk_bulk_data *clks;
+	struct regmap *reg_syscon;
+	struct gpio_desc *power_gpio;
+	struct gpio_desc *reset_gpio;
+	struct phy *phy;
+
+	unsigned int stg_pcie_base;
+	int num_clks;
+};
+
+/*
+ * The BAR0/1 of bridge should be hidden during enumeration to
+ * avoid the sizing and resource allocation by PCIe core.
+ */
+static bool starfive_pcie_hide_rc_bar(struct pci_bus *bus, unsigned int devfn,
+				      int offset)
+{
+	if (pci_is_root_bus(bus) && !devfn &&
+	    (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1))
+		return true;
+
+	return false;
+}
+
+static int starfive_pcie_config_write(struct pci_bus *bus, unsigned int devfn,
+				      int where, int size, u32 value)
+{
+	if (starfive_pcie_hide_rc_bar(bus, devfn, where))
+		return PCIBIOS_SUCCESSFUL;
+
+	return pci_generic_config_write(bus, devfn, where, size, value);
+}
+
+static int starfive_pcie_config_read(struct pci_bus *bus, unsigned int devfn,
+				     int where, int size, u32 *value)
+{
+	if (starfive_pcie_hide_rc_bar(bus, devfn, where)) {
+		*value = 0;
+		return PCIBIOS_SUCCESSFUL;
+	}
+
+	return pci_generic_config_read(bus, devfn, where, size, value);
+}
+
+static int starfive_pcie_parse_dt(struct starfive_jh7110_pcie *pcie,
+				  struct device *dev)
+{
+	int domain_nr;
+
+	pcie->num_clks = devm_clk_bulk_get_all(dev, &pcie->clks);
+	if (pcie->num_clks < 0)
+		return dev_err_probe(dev, pcie->num_clks,
+				     "failed to get pcie clocks\n");
+
+	pcie->resets = devm_reset_control_array_get_exclusive(dev);
+	if (IS_ERR(pcie->resets))
+		return dev_err_probe(dev, PTR_ERR(pcie->resets),
+				     "failed to get pcie resets");
+
+	pcie->reg_syscon =
+		syscon_regmap_lookup_by_phandle(dev->of_node,
+						"starfive,stg-syscon");
+
+	if (IS_ERR(pcie->reg_syscon))
+		return dev_err_probe(dev, PTR_ERR(pcie->reg_syscon),
+				     "failed to parse starfive,stg-syscon\n");
+
+	pcie->phy = devm_phy_optional_get(dev, NULL);
+	if (IS_ERR(pcie->phy))
+		return dev_err_probe(dev, PTR_ERR(pcie->phy),
+				     "failed to get pcie phy\n");
+
+	domain_nr = of_get_pci_domain_nr(dev->of_node);
+
+	if (domain_nr < 0 || domain_nr > 1)
+		return dev_err_probe(dev, -ENODEV,
+				     "failed to get valid pcie domain\n");
+
+	if (domain_nr == 0)
+		pcie->stg_pcie_base = STG_SYSCON_PCIE0_BASE;
+	else
+		pcie->stg_pcie_base = STG_SYSCON_PCIE1_BASE;
+
+	pcie->reset_gpio = devm_gpiod_get_optional(dev, "perst",
+						   GPIOD_OUT_HIGH);
+	if (IS_ERR(pcie->reset_gpio))
+		return dev_err_probe(dev, PTR_ERR(pcie->reset_gpio),
+				     "failed to get perst-gpio\n");
+
+	pcie->power_gpio = devm_gpiod_get_optional(dev, "enable",
+						   GPIOD_OUT_LOW);
+	if (IS_ERR(pcie->power_gpio))
+		return dev_err_probe(dev, PTR_ERR(pcie->power_gpio),
+				     "failed to get power-gpio\n");
+
+	return 0;
+}
+
+static struct pci_ops starfive_pcie_ops = {
+	.map_bus	= plda_pcie_map_bus,
+	.read           = starfive_pcie_config_read,
+	.write          = starfive_pcie_config_write,
+};
+
+static int starfive_pcie_clk_rst_init(struct starfive_jh7110_pcie *pcie)
+{
+	struct device *dev = pcie->plda.dev;
+	int ret;
+
+	ret = clk_bulk_prepare_enable(pcie->num_clks, pcie->clks);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to enable clocks\n");
+
+	ret = reset_control_deassert(pcie->resets);
+	if (ret) {
+		clk_bulk_disable_unprepare(pcie->num_clks, pcie->clks);
+		dev_err_probe(dev, ret, "failed to deassert resets\n");
+	}
+
+	return ret;
+}
+
+static void starfive_pcie_clk_rst_deinit(struct starfive_jh7110_pcie *pcie)
+{
+	reset_control_assert(pcie->resets);
+	clk_bulk_disable_unprepare(pcie->num_clks, pcie->clks);
+}
+
+static bool starfive_pcie_link_up(struct plda_pcie_rp *plda)
+{
+	struct starfive_jh7110_pcie *pcie =
+		container_of(plda, struct starfive_jh7110_pcie, plda);
+	int ret;
+	u32 stg_reg_val;
+
+	ret = regmap_read(pcie->reg_syscon,
+			  pcie->stg_pcie_base + STG_SYSCON_LNKSTA_OFFSET,
+			  &stg_reg_val);
+	if (ret) {
+		dev_err(pcie->plda.dev, "failed to read link status\n");
+		return false;
+	}
+
+	return !!(stg_reg_val & DATA_LINK_ACTIVE);
+}
+
+static int starfive_pcie_host_wait_for_link(struct starfive_jh7110_pcie *pcie)
+{
+	int retries;
+
+	/* Check if the link is up or not */
+	for (retries = 0; retries < LINK_WAIT_MAX_RETRIES; retries++) {
+		if (starfive_pcie_link_up(&pcie->plda)) {
+			dev_info(pcie->plda.dev, "port link up\n");
+			return 0;
+		}
+		usleep_range(LINK_WAIT_USLEEP_MIN, LINK_WAIT_USLEEP_MAX);
+	}
+
+	return -ETIMEDOUT;
+}
+
+static int starfive_pcie_enable_phy(struct device *dev,
+				    struct starfive_jh7110_pcie *pcie)
+{
+	int ret;
+
+	if (!pcie->phy)
+		return 0;
+
+	ret = phy_init(pcie->phy);
+	if (ret)
+		return dev_err_probe(dev, ret,
+				     "failed to initialize pcie phy\n");
+
+	ret = phy_set_mode(pcie->phy, PHY_MODE_PCIE);
+	if (ret) {
+		dev_err_probe(dev, ret, "failed to set pcie mode\n");
+		goto err_phy_on;
+	}
+
+	ret = phy_power_on(pcie->phy);
+	if (ret) {
+		dev_err_probe(dev, ret, "failed to power on pcie phy\n");
+		goto err_phy_on;
+	}
+
+	return 0;
+
+err_phy_on:
+	phy_exit(pcie->phy);
+	return ret;
+}
+
+static void starfive_pcie_disable_phy(struct starfive_jh7110_pcie *pcie)
+{
+	phy_power_off(pcie->phy);
+	phy_exit(pcie->phy);
+}
+
+static void starfive_pcie_host_deinit(struct plda_pcie_rp *plda)
+{
+	struct starfive_jh7110_pcie *pcie =
+		container_of(plda, struct starfive_jh7110_pcie, plda);
+
+	starfive_pcie_clk_rst_deinit(pcie);
+	if (pcie->power_gpio)
+		gpiod_set_value_cansleep(pcie->power_gpio, 0);
+	starfive_pcie_disable_phy(pcie);
+}
+
+static int starfive_pcie_host_init(struct plda_pcie_rp *plda)
+{
+	struct starfive_jh7110_pcie *pcie =
+		container_of(plda, struct starfive_jh7110_pcie, plda);
+	struct device *dev = plda->dev;
+	int ret;
+	int i;
+
+	ret = starfive_pcie_enable_phy(dev, pcie);
+	if (ret)
+		return ret;
+
+	regmap_update_bits(pcie->reg_syscon,
+			   pcie->stg_pcie_base + STG_SYSCON_RP_NEP_OFFSET,
+			   STG_SYSCON_K_RP_NEP, STG_SYSCON_K_RP_NEP);
+
+	regmap_update_bits(pcie->reg_syscon,
+			   pcie->stg_pcie_base + STG_SYSCON_AW_OFFSET,
+			   STG_SYSCON_CKREF_SRC_MASK,
+			   FIELD_PREP(STG_SYSCON_CKREF_SRC_MASK, 2));
+
+	regmap_update_bits(pcie->reg_syscon,
+			   pcie->stg_pcie_base + STG_SYSCON_AW_OFFSET,
+			   STG_SYSCON_CLKREQ, STG_SYSCON_CLKREQ);
+
+	ret = starfive_pcie_clk_rst_init(pcie);
+	if (ret)
+		return ret;
+
+	if (pcie->power_gpio)
+		gpiod_set_value_cansleep(pcie->power_gpio, 1);
+
+	if (pcie->reset_gpio)
+		gpiod_set_value_cansleep(pcie->reset_gpio, 1);
+
+	/* Disable physical functions except #0 */
+	for (i = 1; i < PCIE_FUNC_NUM; i++) {
+		regmap_update_bits(pcie->reg_syscon,
+				   pcie->stg_pcie_base + STG_SYSCON_AR_OFFSET,
+				   STG_SYSCON_AXI4_SLVL_AR_MASK,
+				   STG_SYSCON_AXI4_SLVL_PHY_AR(i));
+
+		regmap_update_bits(pcie->reg_syscon,
+				   pcie->stg_pcie_base + STG_SYSCON_AW_OFFSET,
+				   STG_SYSCON_AXI4_SLVL_AW_MASK,
+				   STG_SYSCON_AXI4_SLVL_PHY_AW(i));
+
+		plda_pcie_disable_func(plda);
+	}
+
+	regmap_update_bits(pcie->reg_syscon,
+			   pcie->stg_pcie_base + STG_SYSCON_AR_OFFSET,
+			   STG_SYSCON_AXI4_SLVL_AR_MASK, 0);
+	regmap_update_bits(pcie->reg_syscon,
+			   pcie->stg_pcie_base + STG_SYSCON_AW_OFFSET,
+			   STG_SYSCON_AXI4_SLVL_AW_MASK, 0);
+
+	plda_pcie_enable_root_port(plda);
+	plda_pcie_write_rc_bar(plda, 0);
+
+	/* PCIe PCI Standard Configuration Identification Settings. */
+	plda_pcie_set_standard_class(plda);
+
+	/*
+	 * The LTR message forwarding of PCIe Message Reception was set by core
+	 * as default, but the forward id & addr are also need to be reset.
+	 * If we do not disable LTR message forwarding here, or set a legal
+	 * forwarding address, the kernel will get stuck after the driver probe.
+	 * To workaround, disable the LTR message forwarding support on
+	 * PCIe Message Reception.
+	 */
+	plda_pcie_disable_ltr(plda);
+
+	/* Prefetchable memory window 64-bit addressing support */
+	plda_pcie_set_pref_win_64bit(plda);
+
+	/*
+	 * Ensure that PERST has been asserted for at least 100 ms,
+	 * the sleep value is T_PVPERL from PCIe CEM spec r2.0 (Table 2-4)
+	 */
+	msleep(100);
+	if (pcie->reset_gpio)
+		gpiod_set_value_cansleep(pcie->reset_gpio, 0);
+
+	/*
+	 * With a Downstream Port (<=5GT/s), software must wait a minimum
+	 * of 100ms following exit from a conventional reset before
+	 * sending a configuration request to the device.
+	 */
+	msleep(PCIE_RESET_CONFIG_DEVICE_WAIT_MS);
+
+	if (starfive_pcie_host_wait_for_link(pcie))
+		dev_info(dev, "port link down\n");
+
+	return 0;
+}
+
+static const struct plda_pcie_host_ops sf_host_ops = {
+	.host_init = starfive_pcie_host_init,
+	.host_deinit = starfive_pcie_host_deinit,
+};
+
+static const struct plda_event stf_pcie_event = {
+	.intx_event = EVENT_PM_MSI_INT_INTX,
+	.msi_event  = EVENT_PM_MSI_INT_MSI
+};
+
+static int starfive_pcie_probe(struct platform_device *pdev)
+{
+	struct starfive_jh7110_pcie *pcie;
+	struct device *dev = &pdev->dev;
+	struct plda_pcie_rp *plda;
+	int ret;
+
+	pcie = devm_kzalloc(dev, sizeof(*pcie), GFP_KERNEL);
+	if (!pcie)
+		return -ENOMEM;
+
+	plda = &pcie->plda;
+	plda->dev = dev;
+
+	ret = starfive_pcie_parse_dt(pcie, dev);
+	if (ret)
+		return ret;
+
+	plda->host_ops = &sf_host_ops;
+	plda->num_events = PLDA_MAX_EVENT_NUM;
+	/* mask doorbell event */
+	plda->events_bitmap = GENMASK(PLDA_INT_EVENT_NUM - 1, 0)
+			     & ~BIT(PLDA_AXI_DOORBELL)
+			     & ~BIT(PLDA_PCIE_DOORBELL);
+	plda->events_bitmap <<= PLDA_NUM_DMA_EVENTS;
+	ret = plda_pcie_host_init(&pcie->plda, &starfive_pcie_ops,
+				  &stf_pcie_event);
+	if (ret)
+		return ret;
+
+	pm_runtime_enable(&pdev->dev);
+	pm_runtime_get_sync(&pdev->dev);
+	platform_set_drvdata(pdev, pcie);
+
+	return 0;
+}
+
+static void starfive_pcie_remove(struct platform_device *pdev)
+{
+	struct starfive_jh7110_pcie *pcie = platform_get_drvdata(pdev);
+
+	pm_runtime_put(&pdev->dev);
+	pm_runtime_disable(&pdev->dev);
+	plda_pcie_host_deinit(&pcie->plda);
+	platform_set_drvdata(pdev, NULL);
+}
+
+static int starfive_pcie_suspend_noirq(struct device *dev)
+{
+	struct starfive_jh7110_pcie *pcie = dev_get_drvdata(dev);
+
+	clk_bulk_disable_unprepare(pcie->num_clks, pcie->clks);
+	starfive_pcie_disable_phy(pcie);
+
+	return 0;
+}
+
+static int starfive_pcie_resume_noirq(struct device *dev)
+{
+	struct starfive_jh7110_pcie *pcie = dev_get_drvdata(dev);
+	int ret;
+
+	ret = starfive_pcie_enable_phy(dev, pcie);
+	if (ret)
+		return ret;
+
+	ret = clk_bulk_prepare_enable(pcie->num_clks, pcie->clks);
+	if (ret) {
+		dev_err(dev, "failed to enable clocks\n");
+		starfive_pcie_disable_phy(pcie);
+		return ret;
+	}
+
+	return 0;
+}
+
+static const struct dev_pm_ops starfive_pcie_pm_ops = {
+	NOIRQ_SYSTEM_SLEEP_PM_OPS(starfive_pcie_suspend_noirq,
+				  starfive_pcie_resume_noirq)
+};
+
+static const struct of_device_id starfive_pcie_of_match[] = {
+	{ .compatible = "starfive,jh7110-pcie", },
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, starfive_pcie_of_match);
+
+static struct platform_driver starfive_pcie_driver = {
+	.driver = {
+		.name = "pcie-starfive",
+		.of_match_table = of_match_ptr(starfive_pcie_of_match),
+		.pm = pm_sleep_ptr(&starfive_pcie_pm_ops),
+	},
+	.probe = starfive_pcie_probe,
+	.remove_new = starfive_pcie_remove,
+};
+module_platform_driver(starfive_pcie_driver);
+
+MODULE_DESCRIPTION("StarFive JH7110 PCIe host driver");
+MODULE_LICENSE("GPL v2");
-- 
2.17.1

[PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers.

[Minda Chen]

From: Kevin Xie <kevin.xie@starfivetech.com>

As the Starfive JH7110 hardware can't keep two inbound post write in
order all the time, such as MSI messages and NVMe completions. If the
NVMe completion update later than the MSI, an NVMe IRQ handle will miss.

As a workaround, we will wait a while before going to the generic
handle here.

Verified with NVMe SSD, USB SSD, R8169 NIC.
The performance are stable and even higher after this patch.

Signed-off-by: Kevin Xie <kevin.xie@starfivetech.com>
Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
---
 drivers/pci/controller/plda/pcie-plda-host.c | 12 ++++++++++++
 drivers/pci/controller/plda/pcie-plda.h      |  1 +
 drivers/pci/controller/plda/pcie-starfive.c  |  1 +
 3 files changed, 14 insertions(+)

diff --git a/drivers/pci/controller/plda/pcie-plda-host.c b/drivers/pci/controller/plda/pcie-plda-host.c
index a18923d7cea6..9e077ddf45c0 100644
--- a/drivers/pci/controller/plda/pcie-plda-host.c
+++ b/drivers/pci/controller/plda/pcie-plda-host.c
@@ -13,6 +13,7 @@
 #include <linux/msi.h>
 #include <linux/pci_regs.h>
 #include <linux/pci-ecam.h>
+#include <linux/delay.h>
 
 #include "pcie-plda.h"
 
@@ -44,6 +45,17 @@ static void plda_handle_msi(struct irq_desc *desc)
 			       bridge_base_addr + ISTATUS_LOCAL);
 		status = readl_relaxed(bridge_base_addr + ISTATUS_MSI);
 		for_each_set_bit(bit, &status, msi->num_vectors) {
+			/*
+			 * As the Starfive JH7110 hardware can't keep two
+			 * inbound post write in order all the time, such as
+			 * MSI messages and NVMe completions.
+			 * If the NVMe completion update later than the MSI,
+			 * an NVMe IRQ handle will miss.
+			 * As a workaround, we will wait a while before
+			 * going to the generic handle here.
+			 */
+			if (port->msi_quirk_delay_us)
+				udelay(port->msi_quirk_delay_us);
 			ret = generic_handle_domain_irq(msi->dev_domain, bit);
 			if (ret)
 				dev_err_ratelimited(dev, "bad MSI IRQ %d\n",
diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
index 04e385758a2f..feccf285dfe8 100644
--- a/drivers/pci/controller/plda/pcie-plda.h
+++ b/drivers/pci/controller/plda/pcie-plda.h
@@ -186,6 +186,7 @@ struct plda_pcie_rp {
 	int msi_irq;
 	int intx_irq;
 	int num_events;
+	u16 msi_quirk_delay_us;
 };
 
 struct plda_event {
diff --git a/drivers/pci/controller/plda/pcie-starfive.c b/drivers/pci/controller/plda/pcie-starfive.c
index 9bb9f0e29565..5cfc30572b7f 100644
--- a/drivers/pci/controller/plda/pcie-starfive.c
+++ b/drivers/pci/controller/plda/pcie-starfive.c
@@ -391,6 +391,7 @@ static int starfive_pcie_probe(struct platform_device *pdev)
 
 	plda->host_ops = &sf_host_ops;
 	plda->num_events = PLDA_MAX_EVENT_NUM;
+	plda->msi_quirk_delay_us = 1;
 	/* mask doorbell event */
 	plda->events_bitmap = GENMASK(PLDA_INT_EVENT_NUM - 1, 0)
 			     & ~BIT(PLDA_AXI_DOORBELL)
-- 
2.17.1

[PATCH v15,RESEND 23/23] riscv: dts: starfive: add PCIe dts configuration for JH7110

[Minda Chen]

Add PCIe dts configuraion for JH7110 SoC platform.

Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
Reviewed-by: Hal Feng <hal.feng@starfivetech.com>
---
 .../jh7110-starfive-visionfive-2.dtsi         | 64 ++++++++++++++
 arch/riscv/boot/dts/starfive/jh7110.dtsi      | 86 +++++++++++++++++++
 2 files changed, 150 insertions(+)

diff --git a/arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive-2.dtsi b/arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive-2.dtsi
index b89e9791efa7..ff55a66d7603 100644
--- a/arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive-2.dtsi
+++ b/arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive-2.dtsi
@@ -281,6 +281,22 @@
 	status = "okay";
 };
 
+&pcie0 {
+	perst-gpios = <&sysgpio 26 GPIO_ACTIVE_LOW>;
+	phys = <&pciephy0>;
+	pinctrl-names = "default";
+	pinctrl-0 = <&pcie0_pins>;
+	status = "okay";
+};
+
+&pcie1 {
+	perst-gpios = <&sysgpio 28 GPIO_ACTIVE_LOW>;
+	phys = <&pciephy1>;
+	pinctrl-names = "default";
+	pinctrl-0 = <&pcie1_pins>;
+	status = "okay";
+};
+
 &pwmdac {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pwmdac_pins>;
@@ -497,6 +513,54 @@
 		};
 	};
 
+	pcie0_pins: pcie0-0 {
+		clkreq-pins {
+			pinmux = <GPIOMUX(27, GPOUT_LOW,
+					      GPOEN_DISABLE,
+					      GPI_NONE)>;
+			bias-pull-down;
+			drive-strength = <2>;
+			input-enable;
+			input-schmitt-disable;
+			slew-rate = <0>;
+		};
+
+		wake-pins {
+			pinmux = <GPIOMUX(32, GPOUT_LOW,
+					      GPOEN_DISABLE,
+					      GPI_NONE)>;
+			bias-pull-up;
+			drive-strength = <2>;
+			input-enable;
+			input-schmitt-disable;
+			slew-rate = <0>;
+		};
+	};
+
+	pcie1_pins: pcie1-0 {
+		clkreq-pins {
+			pinmux = <GPIOMUX(29, GPOUT_LOW,
+					      GPOEN_DISABLE,
+					      GPI_NONE)>;
+			bias-pull-down;
+			drive-strength = <2>;
+			input-enable;
+			input-schmitt-disable;
+			slew-rate = <0>;
+		};
+
+		wake-pins {
+			pinmux = <GPIOMUX(21, GPOUT_LOW,
+					      GPOEN_DISABLE,
+					      GPI_NONE)>;
+			bias-pull-up;
+			drive-strength = <2>;
+			input-enable;
+			input-schmitt-disable;
+			slew-rate = <0>;
+		};
+	};
+
 	pwmdac_pins: pwmdac-0 {
 		pwmdac-pins {
 			pinmux = <GPIOMUX(33, GPOUT_SYS_PWMDAC_LEFT,
diff --git a/arch/riscv/boot/dts/starfive/jh7110.dtsi b/arch/riscv/boot/dts/starfive/jh7110.dtsi
index 74ed3b9264d8..db96214fd443 100644
--- a/arch/riscv/boot/dts/starfive/jh7110.dtsi
+++ b/arch/riscv/boot/dts/starfive/jh7110.dtsi
@@ -1138,5 +1138,91 @@
 			#reset-cells = <1>;
 			power-domains = <&pwrc JH7110_PD_VOUT>;
 		};
+
+		pcie0: pcie@940000000 {
+			compatible = "starfive,jh7110-pcie";
+			reg = <0x9 0x40000000 0x0 0x1000000>,
+			      <0x0 0x2b000000 0x0 0x100000>;
+			reg-names = "cfg", "apb";
+			linux,pci-domain = <0>;
+			#address-cells = <3>;
+			#size-cells = <2>;
+			#interrupt-cells = <1>;
+			ranges = <0x82000000  0x0 0x30000000  0x0 0x30000000 0x0 0x08000000>,
+				 <0xc3000000  0x9 0x00000000  0x9 0x00000000 0x0 0x40000000>;
+			interrupts = <56>;
+			interrupt-map-mask = <0x0 0x0 0x0 0x7>;
+			interrupt-map = <0x0 0x0 0x0 0x1 &pcie_intc0 0x1>,
+					<0x0 0x0 0x0 0x2 &pcie_intc0 0x2>,
+					<0x0 0x0 0x0 0x3 &pcie_intc0 0x3>,
+					<0x0 0x0 0x0 0x4 &pcie_intc0 0x4>;
+			msi-controller;
+			device_type = "pci";
+			starfive,stg-syscon = <&stg_syscon>;
+			bus-range = <0x0 0xff>;
+			clocks = <&syscrg JH7110_SYSCLK_NOC_BUS_STG_AXI>,
+				 <&stgcrg JH7110_STGCLK_PCIE0_TL>,
+				 <&stgcrg JH7110_STGCLK_PCIE0_AXI_MST0>,
+				 <&stgcrg JH7110_STGCLK_PCIE0_APB>;
+			clock-names = "noc", "tl", "axi_mst0", "apb";
+			resets = <&stgcrg JH7110_STGRST_PCIE0_AXI_MST0>,
+				 <&stgcrg JH7110_STGRST_PCIE0_AXI_SLV0>,
+				 <&stgcrg JH7110_STGRST_PCIE0_AXI_SLV>,
+				 <&stgcrg JH7110_STGRST_PCIE0_BRG>,
+				 <&stgcrg JH7110_STGRST_PCIE0_CORE>,
+				 <&stgcrg JH7110_STGRST_PCIE0_APB>;
+			reset-names = "mst0", "slv0", "slv", "brg",
+				      "core", "apb";
+			status = "disabled";
+
+			pcie_intc0: interrupt-controller {
+				#address-cells = <0>;
+				#interrupt-cells = <1>;
+				interrupt-controller;
+			};
+		};
+
+		pcie1: pcie@9c0000000 {
+			compatible = "starfive,jh7110-pcie";
+			reg = <0x9 0xc0000000 0x0 0x1000000>,
+			      <0x0 0x2c000000 0x0 0x100000>;
+			reg-names = "cfg", "apb";
+			linux,pci-domain = <1>;
+			#address-cells = <3>;
+			#size-cells = <2>;
+			#interrupt-cells = <1>;
+			ranges = <0x82000000  0x0 0x38000000  0x0 0x38000000 0x0 0x08000000>,
+				 <0xc3000000  0x9 0x80000000  0x9 0x80000000 0x0 0x40000000>;
+			interrupts = <57>;
+			interrupt-map-mask = <0x0 0x0 0x0 0x7>;
+			interrupt-map = <0x0 0x0 0x0 0x1 &pcie_intc1 0x1>,
+					<0x0 0x0 0x0 0x2 &pcie_intc1 0x2>,
+					<0x0 0x0 0x0 0x3 &pcie_intc1 0x3>,
+					<0x0 0x0 0x0 0x4 &pcie_intc1 0x4>;
+			msi-controller;
+			device_type = "pci";
+			starfive,stg-syscon = <&stg_syscon>;
+			bus-range = <0x0 0xff>;
+			clocks = <&syscrg JH7110_SYSCLK_NOC_BUS_STG_AXI>,
+				 <&stgcrg JH7110_STGCLK_PCIE1_TL>,
+				 <&stgcrg JH7110_STGCLK_PCIE1_AXI_MST0>,
+				 <&stgcrg JH7110_STGCLK_PCIE1_APB>;
+			clock-names = "noc", "tl", "axi_mst0", "apb";
+			resets = <&stgcrg JH7110_STGRST_PCIE1_AXI_MST0>,
+				 <&stgcrg JH7110_STGRST_PCIE1_AXI_SLV0>,
+				 <&stgcrg JH7110_STGRST_PCIE1_AXI_SLV>,
+				 <&stgcrg JH7110_STGRST_PCIE1_BRG>,
+				 <&stgcrg JH7110_STGRST_PCIE1_CORE>,
+				 <&stgcrg JH7110_STGRST_PCIE1_APB>;
+			reset-names = "mst0", "slv0", "slv", "brg",
+				      "core", "apb";
+			status = "disabled";
+
+			pcie_intc1: interrupt-controller {
+				#address-cells = <0>;
+				#interrupt-cells = <1>;
+				interrupt-controller;
+			};
+		};
 	};
 };
-- 
2.17.1

Re: [PATCH v15,RESEND 00/23] Refactoring Microchip PCIe driver and add StarFive PCIe

[Aurelien Jarno]

On 2024-02-27 18:34, Minda Chen wrote:
> This patchset final purpose is add PCIe driver for StarFive JH7110 SoC.
> JH7110 using PLDA XpressRICH PCIe IP. Microchip PolarFire Using the
> same IP and have commit their codes, which are mixed with PLDA
> controller codes and Microchip platform codes.
> 
> For re-use the PLDA controller codes, I request refactoring microchip
> codes, move PLDA common codes to PLDA files.
> Desigware and Cadence is good example for refactoring codes.
> 

Thanks for the resend, I confirm that now the mail threading is correct.

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

Re: [PATCH v15,RESEND 05/23] PCI: microchip: Rename two PCIe data structures

[Lorenzo Pieralisi]

On Tue, Feb 27, 2024 at 06:35:04PM +0800, Minda Chen wrote:
> Add PLDA PCIe related data structures by rename data structure name from
> mc_* to plda_*.
> 
> axi_base_addr is stayed in struct mc_pcie for it's microchip its own data.

Here in "for it's microchip" you mean "since it's microchip" own data.

Please fix this "for" usage in all commit logs, I noticed it is
a recurrent mistake you make I can do it but it is tedious.

Don't resend another version yet, I am trying to see what I can
take from this one.

> The event interrupt codes is still using struct mc_pcie because the event

"code", please not "codes". Please fix it everywhere in your patch
series, same as above.

Lorenzo

> interrupt codes can not be re-used.
> 
> Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
> ---
>  .../pci/controller/plda/pcie-microchip-host.c | 96 ++++++++++---------
>  1 file changed, 53 insertions(+), 43 deletions(-)
> 
> diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
> index c55ede80a6d0..df0736f688ce 100644
> --- a/drivers/pci/controller/plda/pcie-microchip-host.c
> +++ b/drivers/pci/controller/plda/pcie-microchip-host.c
> @@ -22,7 +22,7 @@
>  #include "pcie-plda.h"
>  
>  /* Number of MSI IRQs */
> -#define MC_MAX_NUM_MSI_IRQS			32
> +#define PLDA_MAX_NUM_MSI_IRQS			32
>  
>  /* PCIe Bridge Phy and Controller Phy offsets */
>  #define MC_PCIE1_BRIDGE_ADDR			0x00008000u
> @@ -179,25 +179,29 @@ struct event_map {
>  	u32 event_bit;
>  };
>  
> -struct mc_msi {
> +struct plda_msi {
>  	struct mutex lock;		/* Protect used bitmap */
>  	struct irq_domain *msi_domain;
>  	struct irq_domain *dev_domain;
>  	u32 num_vectors;
>  	u64 vector_phy;
> -	DECLARE_BITMAP(used, MC_MAX_NUM_MSI_IRQS);
> +	DECLARE_BITMAP(used, PLDA_MAX_NUM_MSI_IRQS);
>  };
>  
> -struct mc_pcie {
> -	void __iomem *axi_base_addr;
> +struct plda_pcie_rp {
>  	struct device *dev;
>  	struct irq_domain *intx_domain;
>  	struct irq_domain *event_domain;
>  	raw_spinlock_t lock;
> -	struct mc_msi msi;
> +	struct plda_msi msi;
>  	void __iomem *bridge_addr;
>  };
>  
> +struct mc_pcie {
> +	struct plda_pcie_rp plda;
> +	void __iomem *axi_base_addr;
> +};
> +
>  struct cause {
>  	const char *sym;
>  	const char *str;
> @@ -313,7 +317,7 @@ static struct mc_pcie *port;
>  
>  static void mc_pcie_enable_msi(struct mc_pcie *port, void __iomem *ecam)
>  {
> -	struct mc_msi *msi = &port->msi;
> +	struct plda_msi *msi = &port->plda.msi;
>  	u16 reg;
>  	u8 queue_size;
>  
> @@ -336,10 +340,10 @@ static void mc_pcie_enable_msi(struct mc_pcie *port, void __iomem *ecam)
>  
>  static void mc_handle_msi(struct irq_desc *desc)
>  {
> -	struct mc_pcie *port = irq_desc_get_handler_data(desc);
> +	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
>  	struct irq_chip *chip = irq_desc_get_chip(desc);
>  	struct device *dev = port->dev;
> -	struct mc_msi *msi = &port->msi;
> +	struct plda_msi *msi = &port->msi;
>  	void __iomem *bridge_base_addr = port->bridge_addr;
>  	unsigned long status;
>  	u32 bit;
> @@ -364,7 +368,7 @@ static void mc_handle_msi(struct irq_desc *desc)
>  
>  static void mc_msi_bottom_irq_ack(struct irq_data *data)
>  {
> -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
>  	void __iomem *bridge_base_addr = port->bridge_addr;
>  	u32 bitpos = data->hwirq;
>  
> @@ -373,7 +377,7 @@ static void mc_msi_bottom_irq_ack(struct irq_data *data)
>  
>  static void mc_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
>  {
> -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
>  	phys_addr_t addr = port->msi.vector_phy;
>  
>  	msg->address_lo = lower_32_bits(addr);
> @@ -400,8 +404,8 @@ static struct irq_chip mc_msi_bottom_irq_chip = {
>  static int mc_irq_msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
>  				   unsigned int nr_irqs, void *args)
>  {
> -	struct mc_pcie *port = domain->host_data;
> -	struct mc_msi *msi = &port->msi;
> +	struct plda_pcie_rp *port = domain->host_data;
> +	struct plda_msi *msi = &port->msi;
>  	unsigned long bit;
>  
>  	mutex_lock(&msi->lock);
> @@ -425,8 +429,8 @@ static void mc_irq_msi_domain_free(struct irq_domain *domain, unsigned int virq,
>  				   unsigned int nr_irqs)
>  {
>  	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> -	struct mc_pcie *port = irq_data_get_irq_chip_data(d);
> -	struct mc_msi *msi = &port->msi;
> +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(d);
> +	struct plda_msi *msi = &port->msi;
>  
>  	mutex_lock(&msi->lock);
>  
> @@ -456,11 +460,11 @@ static struct msi_domain_info mc_msi_domain_info = {
>  	.chip = &mc_msi_irq_chip,
>  };
>  
> -static int mc_allocate_msi_domains(struct mc_pcie *port)
> +static int mc_allocate_msi_domains(struct plda_pcie_rp *port)
>  {
>  	struct device *dev = port->dev;
>  	struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
> -	struct mc_msi *msi = &port->msi;
> +	struct plda_msi *msi = &port->msi;
>  
>  	mutex_init(&port->msi.lock);
>  
> @@ -484,7 +488,7 @@ static int mc_allocate_msi_domains(struct mc_pcie *port)
>  
>  static void mc_handle_intx(struct irq_desc *desc)
>  {
> -	struct mc_pcie *port = irq_desc_get_handler_data(desc);
> +	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
>  	struct irq_chip *chip = irq_desc_get_chip(desc);
>  	struct device *dev = port->dev;
>  	void __iomem *bridge_base_addr = port->bridge_addr;
> @@ -511,7 +515,7 @@ static void mc_handle_intx(struct irq_desc *desc)
>  
>  static void mc_ack_intx_irq(struct irq_data *data)
>  {
> -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
>  	void __iomem *bridge_base_addr = port->bridge_addr;
>  	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
>  
> @@ -520,7 +524,7 @@ static void mc_ack_intx_irq(struct irq_data *data)
>  
>  static void mc_mask_intx_irq(struct irq_data *data)
>  {
> -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
>  	void __iomem *bridge_base_addr = port->bridge_addr;
>  	unsigned long flags;
>  	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
> @@ -535,7 +539,7 @@ static void mc_mask_intx_irq(struct irq_data *data)
>  
>  static void mc_unmask_intx_irq(struct irq_data *data)
>  {
> -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
>  	void __iomem *bridge_base_addr = port->bridge_addr;
>  	unsigned long flags;
>  	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
> @@ -625,21 +629,22 @@ static u32 local_events(struct mc_pcie *port)
>  	return val;
>  }
>  
> -static u32 get_events(struct mc_pcie *port)
> +static u32 get_events(struct plda_pcie_rp *port)
>  {
> +	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
>  	u32 events = 0;
>  
> -	events |= pcie_events(port);
> -	events |= sec_errors(port);
> -	events |= ded_errors(port);
> -	events |= local_events(port);
> +	events |= pcie_events(mc_port);
> +	events |= sec_errors(mc_port);
> +	events |= ded_errors(mc_port);
> +	events |= local_events(mc_port);
>  
>  	return events;
>  }
>  
>  static irqreturn_t mc_event_handler(int irq, void *dev_id)
>  {
> -	struct mc_pcie *port = dev_id;
> +	struct plda_pcie_rp *port = dev_id;
>  	struct device *dev = port->dev;
>  	struct irq_data *data;
>  
> @@ -655,7 +660,7 @@ static irqreturn_t mc_event_handler(int irq, void *dev_id)
>  
>  static void mc_handle_event(struct irq_desc *desc)
>  {
> -	struct mc_pcie *port = irq_desc_get_handler_data(desc);
> +	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
>  	unsigned long events;
>  	u32 bit;
>  	struct irq_chip *chip = irq_desc_get_chip(desc);
> @@ -672,12 +677,13 @@ static void mc_handle_event(struct irq_desc *desc)
>  
>  static void mc_ack_event_irq(struct irq_data *data)
>  {
> -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> +	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
>  	u32 event = data->hwirq;
>  	void __iomem *addr;
>  	u32 mask;
>  
> -	addr = port->axi_base_addr + event_descs[event].base +
> +	addr = mc_port->axi_base_addr + event_descs[event].base +
>  		event_descs[event].offset;
>  	mask = event_descs[event].mask;
>  	mask |= event_descs[event].enb_mask;
> @@ -687,13 +693,14 @@ static void mc_ack_event_irq(struct irq_data *data)
>  
>  static void mc_mask_event_irq(struct irq_data *data)
>  {
> -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> +	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
>  	u32 event = data->hwirq;
>  	void __iomem *addr;
>  	u32 mask;
>  	u32 val;
>  
> -	addr = port->axi_base_addr + event_descs[event].base +
> +	addr = mc_port->axi_base_addr + event_descs[event].base +
>  		event_descs[event].mask_offset;
>  	mask = event_descs[event].mask;
>  	if (event_descs[event].enb_mask) {
> @@ -717,13 +724,14 @@ static void mc_mask_event_irq(struct irq_data *data)
>  
>  static void mc_unmask_event_irq(struct irq_data *data)
>  {
> -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> +	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
>  	u32 event = data->hwirq;
>  	void __iomem *addr;
>  	u32 mask;
>  	u32 val;
>  
> -	addr = port->axi_base_addr + event_descs[event].base +
> +	addr = mc_port->axi_base_addr + event_descs[event].base +
>  		event_descs[event].mask_offset;
>  	mask = event_descs[event].mask;
>  
> @@ -811,7 +819,7 @@ static int mc_pcie_init_clks(struct device *dev)
>  	return 0;
>  }
>  
> -static int mc_pcie_init_irq_domains(struct mc_pcie *port)
> +static int mc_pcie_init_irq_domains(struct plda_pcie_rp *port)
>  {
>  	struct device *dev = port->dev;
>  	struct device_node *node = dev->of_node;
> @@ -889,7 +897,7 @@ static void mc_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
>  }
>  
>  static int mc_pcie_setup_windows(struct platform_device *pdev,
> -				 struct mc_pcie *port)
> +				 struct plda_pcie_rp *port)
>  {
>  	void __iomem *bridge_base_addr = port->bridge_addr;
>  	struct pci_host_bridge *bridge = platform_get_drvdata(pdev);
> @@ -970,7 +978,7 @@ static void mc_disable_interrupts(struct mc_pcie *port)
>  	writel_relaxed(GENMASK(31, 0), bridge_base_addr + ISTATUS_HOST);
>  }
>  
> -static int mc_init_interrupts(struct platform_device *pdev, struct mc_pcie *port)
> +static int mc_init_interrupts(struct platform_device *pdev, struct plda_pcie_rp *port)
>  {
>  	struct device *dev = &pdev->dev;
>  	int irq;
> @@ -1043,12 +1051,12 @@ static int mc_platform_init(struct pci_config_window *cfg)
>  	mc_pcie_enable_msi(port, cfg->win);
>  
>  	/* Configure non-config space outbound ranges */
> -	ret = mc_pcie_setup_windows(pdev, port);
> +	ret = mc_pcie_setup_windows(pdev, &port->plda);
>  	if (ret)
>  		return ret;
>  
>  	/* Address translation is up; safe to enable interrupts */
> -	ret = mc_init_interrupts(pdev, port);
> +	ret = mc_init_interrupts(pdev, &port->plda);
>  	if (ret)
>  		return ret;
>  
> @@ -1059,6 +1067,7 @@ static int mc_host_probe(struct platform_device *pdev)
>  {
>  	struct device *dev = &pdev->dev;
>  	void __iomem *bridge_base_addr;
> +	struct plda_pcie_rp *plda;
>  	int ret;
>  	u32 val;
>  
> @@ -1066,7 +1075,8 @@ static int mc_host_probe(struct platform_device *pdev)
>  	if (!port)
>  		return -ENOMEM;
>  
> -	port->dev = dev;
> +	plda = &port->plda;
> +	plda->dev = dev;
>  
>  	port->axi_base_addr = devm_platform_ioremap_resource(pdev, 1);
>  	if (IS_ERR(port->axi_base_addr))
> @@ -1075,7 +1085,7 @@ static int mc_host_probe(struct platform_device *pdev)
>  	mc_disable_interrupts(port);
>  
>  	bridge_base_addr = port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
> -	port->bridge_addr = bridge_base_addr;
> +	plda->bridge_addr = bridge_base_addr;
>  
>  	/* Allow enabling MSI by disabling MSI-X */
>  	val = readl(bridge_base_addr + PCIE_PCI_IRQ_DW0);
> @@ -1087,10 +1097,10 @@ static int mc_host_probe(struct platform_device *pdev)
>  	val &= NUM_MSI_MSGS_MASK;
>  	val >>= NUM_MSI_MSGS_SHIFT;
>  
> -	port->msi.num_vectors = 1 << val;
> +	plda->msi.num_vectors = 1 << val;
>  
>  	/* Pick vector address from design */
> -	port->msi.vector_phy = readl_relaxed(bridge_base_addr + IMSI_ADDR);
> +	plda->msi.vector_phy = readl_relaxed(bridge_base_addr + IMSI_ADDR);
>  
>  	ret = mc_pcie_init_clks(dev);
>  	if (ret) {
> -- 
> 2.17.1
> 
> 
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH v15,RESEND 03/23] PCI: microchip: Move PLDA IP register macros to pcie-plda.h

[Lorenzo Pieralisi]

On Tue, Feb 27, 2024 at 06:35:02PM +0800, Minda Chen wrote:
> Move PLDA PCIe host controller IP registers macros to pcie-plda.h,
> including bridge registers and PLDA IRQ event number.
> 
> Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
> ---
>  MAINTAINERS                                   |   8 ++
>  .../pci/controller/plda/pcie-microchip-host.c | 108 +++---------------
>  drivers/pci/controller/plda/pcie-plda.h       | 108 ++++++++++++++++++
>  3 files changed, 132 insertions(+), 92 deletions(-)
>  create mode 100644 drivers/pci/controller/plda/pcie-plda.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 9f7e907c8749..fc9576e69a71 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -16814,6 +16814,14 @@ S:	Maintained
>  F:	Documentation/devicetree/bindings/pci/layerscape-pcie-gen4.txt
>  F:	drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
>  
> +PCI DRIVER FOR PLDA PCIE IP
> +M:	Daire McNamara <daire.mcnamara@microchip.com>
> +M:	Kevin Xie <kevin.xie@starfivetech.com>
> +L:	linux-pci@vger.kernel.org
> +S:	Maintained
> +F:	Documentation/devicetree/bindings/pci/plda,*

This is an unrelated change. It belongs to the patch adding DT bindings
or I am reading it wrong ?

Lorenzo

> +F:	drivers/pci/controller/plda/*plda*
> +
>  PCI DRIVER FOR RENESAS R-CAR
>  M:	Marek Vasut <marek.vasut+renesas@gmail.com>
>  M:	Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c b/drivers/pci/controller/plda/pcie-microchip-host.c
> index cb09a8137e25..d9030d550482 100644
> --- a/drivers/pci/controller/plda/pcie-microchip-host.c
> +++ b/drivers/pci/controller/plda/pcie-microchip-host.c
> @@ -19,6 +19,7 @@
>  #include <linux/platform_device.h>
>  
>  #include "../../pci.h"
> +#include "pcie-plda.h"
>  
>  /* Number of MSI IRQs */
>  #define MC_MAX_NUM_MSI_IRQS			32
> @@ -30,84 +31,6 @@
>  #define MC_PCIE_BRIDGE_ADDR			(MC_PCIE1_BRIDGE_ADDR)
>  #define MC_PCIE_CTRL_ADDR			(MC_PCIE1_CTRL_ADDR)
>  
> -/* PCIe Bridge Phy Regs */
> -#define PCIE_PCI_IRQ_DW0			0xa8
> -#define  MSIX_CAP_MASK				BIT(31)
> -#define  NUM_MSI_MSGS_MASK			GENMASK(6, 4)
> -#define  NUM_MSI_MSGS_SHIFT			4
> -
> -#define IMASK_LOCAL				0x180
> -#define  DMA_END_ENGINE_0_MASK			0x00000000u
> -#define  DMA_END_ENGINE_0_SHIFT			0
> -#define  DMA_END_ENGINE_1_MASK			0x00000000u
> -#define  DMA_END_ENGINE_1_SHIFT			1
> -#define  DMA_ERROR_ENGINE_0_MASK		0x00000100u
> -#define  DMA_ERROR_ENGINE_0_SHIFT		8
> -#define  DMA_ERROR_ENGINE_1_MASK		0x00000200u
> -#define  DMA_ERROR_ENGINE_1_SHIFT		9
> -#define  A_ATR_EVT_POST_ERR_MASK		0x00010000u
> -#define  A_ATR_EVT_POST_ERR_SHIFT		16
> -#define  A_ATR_EVT_FETCH_ERR_MASK		0x00020000u
> -#define  A_ATR_EVT_FETCH_ERR_SHIFT		17
> -#define  A_ATR_EVT_DISCARD_ERR_MASK		0x00040000u
> -#define  A_ATR_EVT_DISCARD_ERR_SHIFT		18
> -#define  A_ATR_EVT_DOORBELL_MASK		0x00000000u
> -#define  A_ATR_EVT_DOORBELL_SHIFT		19
> -#define  P_ATR_EVT_POST_ERR_MASK		0x00100000u
> -#define  P_ATR_EVT_POST_ERR_SHIFT		20
> -#define  P_ATR_EVT_FETCH_ERR_MASK		0x00200000u
> -#define  P_ATR_EVT_FETCH_ERR_SHIFT		21
> -#define  P_ATR_EVT_DISCARD_ERR_MASK		0x00400000u
> -#define  P_ATR_EVT_DISCARD_ERR_SHIFT		22
> -#define  P_ATR_EVT_DOORBELL_MASK		0x00000000u
> -#define  P_ATR_EVT_DOORBELL_SHIFT		23
> -#define  PM_MSI_INT_INTA_MASK			0x01000000u
> -#define  PM_MSI_INT_INTA_SHIFT			24
> -#define  PM_MSI_INT_INTB_MASK			0x02000000u
> -#define  PM_MSI_INT_INTB_SHIFT			25
> -#define  PM_MSI_INT_INTC_MASK			0x04000000u
> -#define  PM_MSI_INT_INTC_SHIFT			26
> -#define  PM_MSI_INT_INTD_MASK			0x08000000u
> -#define  PM_MSI_INT_INTD_SHIFT			27
> -#define  PM_MSI_INT_INTX_MASK			0x0f000000u
> -#define  PM_MSI_INT_INTX_SHIFT			24
> -#define  PM_MSI_INT_MSI_MASK			0x10000000u
> -#define  PM_MSI_INT_MSI_SHIFT			28
> -#define  PM_MSI_INT_AER_EVT_MASK		0x20000000u
> -#define  PM_MSI_INT_AER_EVT_SHIFT		29
> -#define  PM_MSI_INT_EVENTS_MASK			0x40000000u
> -#define  PM_MSI_INT_EVENTS_SHIFT		30
> -#define  PM_MSI_INT_SYS_ERR_MASK		0x80000000u
> -#define  PM_MSI_INT_SYS_ERR_SHIFT		31
> -#define  NUM_LOCAL_EVENTS			15
> -#define ISTATUS_LOCAL				0x184
> -#define IMASK_HOST				0x188
> -#define ISTATUS_HOST				0x18c
> -#define IMSI_ADDR				0x190
> -#define ISTATUS_MSI				0x194
> -
> -/* PCIe Master table init defines */
> -#define ATR0_PCIE_WIN0_SRCADDR_PARAM		0x600u
> -#define  ATR0_PCIE_ATR_SIZE			0x25
> -#define  ATR0_PCIE_ATR_SIZE_SHIFT		1
> -#define ATR0_PCIE_WIN0_SRC_ADDR			0x604u
> -#define ATR0_PCIE_WIN0_TRSL_ADDR_LSB		0x608u
> -#define ATR0_PCIE_WIN0_TRSL_ADDR_UDW		0x60cu
> -#define ATR0_PCIE_WIN0_TRSL_PARAM		0x610u
> -
> -/* PCIe AXI slave table init defines */
> -#define ATR0_AXI4_SLV0_SRCADDR_PARAM		0x800u
> -#define  ATR_SIZE_SHIFT				1
> -#define  ATR_IMPL_ENABLE			1
> -#define ATR0_AXI4_SLV0_SRC_ADDR			0x804u
> -#define ATR0_AXI4_SLV0_TRSL_ADDR_LSB		0x808u
> -#define ATR0_AXI4_SLV0_TRSL_ADDR_UDW		0x80cu
> -#define ATR0_AXI4_SLV0_TRSL_PARAM		0x810u
> -#define  PCIE_TX_RX_INTERFACE			0x00000000u
> -#define  PCIE_CONFIG_INTERFACE			0x00000001u
> -
> -#define ATR_ENTRY_SIZE				32
> -
>  /* PCIe Controller Phy Regs */
>  #define SEC_ERROR_EVENT_CNT			0x20
>  #define DED_ERROR_EVENT_CNT			0x24
> @@ -179,20 +102,21 @@
>  #define EVENT_LOCAL_DMA_END_ENGINE_1		12
>  #define EVENT_LOCAL_DMA_ERROR_ENGINE_0		13
>  #define EVENT_LOCAL_DMA_ERROR_ENGINE_1		14
> -#define EVENT_LOCAL_A_ATR_EVT_POST_ERR		15
> -#define EVENT_LOCAL_A_ATR_EVT_FETCH_ERR		16
> -#define EVENT_LOCAL_A_ATR_EVT_DISCARD_ERR	17
> -#define EVENT_LOCAL_A_ATR_EVT_DOORBELL		18
> -#define EVENT_LOCAL_P_ATR_EVT_POST_ERR		19
> -#define EVENT_LOCAL_P_ATR_EVT_FETCH_ERR		20
> -#define EVENT_LOCAL_P_ATR_EVT_DISCARD_ERR	21
> -#define EVENT_LOCAL_P_ATR_EVT_DOORBELL		22
> -#define EVENT_LOCAL_PM_MSI_INT_INTX		23
> -#define EVENT_LOCAL_PM_MSI_INT_MSI		24
> -#define EVENT_LOCAL_PM_MSI_INT_AER_EVT		25
> -#define EVENT_LOCAL_PM_MSI_INT_EVENTS		26
> -#define EVENT_LOCAL_PM_MSI_INT_SYS_ERR		27
> -#define NUM_EVENTS				28
> +#define NUM_MC_EVENTS				15
> +#define EVENT_LOCAL_A_ATR_EVT_POST_ERR		(NUM_MC_EVENTS + PLDA_AXI_POST_ERR)
> +#define EVENT_LOCAL_A_ATR_EVT_FETCH_ERR		(NUM_MC_EVENTS + PLDA_AXI_FETCH_ERR)
> +#define EVENT_LOCAL_A_ATR_EVT_DISCARD_ERR	(NUM_MC_EVENTS + PLDA_AXI_DISCARD_ERR)
> +#define EVENT_LOCAL_A_ATR_EVT_DOORBELL		(NUM_MC_EVENTS + PLDA_AXI_DOORBELL)
> +#define EVENT_LOCAL_P_ATR_EVT_POST_ERR		(NUM_MC_EVENTS + PLDA_PCIE_POST_ERR)
> +#define EVENT_LOCAL_P_ATR_EVT_FETCH_ERR		(NUM_MC_EVENTS + PLDA_PCIE_FETCH_ERR)
> +#define EVENT_LOCAL_P_ATR_EVT_DISCARD_ERR	(NUM_MC_EVENTS + PLDA_PCIE_DISCARD_ERR)
> +#define EVENT_LOCAL_P_ATR_EVT_DOORBELL		(NUM_MC_EVENTS + PLDA_PCIE_DOORBELL)
> +#define EVENT_LOCAL_PM_MSI_INT_INTX		(NUM_MC_EVENTS + PLDA_INTX)
> +#define EVENT_LOCAL_PM_MSI_INT_MSI		(NUM_MC_EVENTS + PLDA_MSI)
> +#define EVENT_LOCAL_PM_MSI_INT_AER_EVT		(NUM_MC_EVENTS + PLDA_AER_EVENT)
> +#define EVENT_LOCAL_PM_MSI_INT_EVENTS		(NUM_MC_EVENTS + PLDA_MISC_EVENTS)
> +#define EVENT_LOCAL_PM_MSI_INT_SYS_ERR		(NUM_MC_EVENTS + PLDA_SYS_ERR)
> +#define NUM_EVENTS				(NUM_MC_EVENTS + PLDA_INT_EVENT_NUM)
>  
>  #define PCIE_EVENT_CAUSE(x, s)	\
>  	[EVENT_PCIE_ ## x] = { __stringify(x), s }
> diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
> new file mode 100644
> index 000000000000..65e0f3b72184
> --- /dev/null
> +++ b/drivers/pci/controller/plda/pcie-plda.h
> @@ -0,0 +1,108 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * PLDA PCIe host controller driver
> + */
> +
> +#ifndef _PCIE_PLDA_H
> +#define _PCIE_PLDA_H
> +
> +/* PCIe Bridge Phy Regs */
> +#define PCIE_PCI_IRQ_DW0			0xa8
> +#define  MSIX_CAP_MASK				BIT(31)
> +#define  NUM_MSI_MSGS_MASK			GENMASK(6, 4)
> +#define  NUM_MSI_MSGS_SHIFT			4
> +
> +#define IMASK_LOCAL				0x180
> +#define  DMA_END_ENGINE_0_MASK			0x00000000u
> +#define  DMA_END_ENGINE_0_SHIFT			0
> +#define  DMA_END_ENGINE_1_MASK			0x00000000u
> +#define  DMA_END_ENGINE_1_SHIFT			1
> +#define  DMA_ERROR_ENGINE_0_MASK		0x00000100u
> +#define  DMA_ERROR_ENGINE_0_SHIFT		8
> +#define  DMA_ERROR_ENGINE_1_MASK		0x00000200u
> +#define  DMA_ERROR_ENGINE_1_SHIFT		9
> +#define  A_ATR_EVT_POST_ERR_MASK		0x00010000u
> +#define  A_ATR_EVT_POST_ERR_SHIFT		16
> +#define  A_ATR_EVT_FETCH_ERR_MASK		0x00020000u
> +#define  A_ATR_EVT_FETCH_ERR_SHIFT		17
> +#define  A_ATR_EVT_DISCARD_ERR_MASK		0x00040000u
> +#define  A_ATR_EVT_DISCARD_ERR_SHIFT		18
> +#define  A_ATR_EVT_DOORBELL_MASK		0x00000000u
> +#define  A_ATR_EVT_DOORBELL_SHIFT		19
> +#define  P_ATR_EVT_POST_ERR_MASK		0x00100000u
> +#define  P_ATR_EVT_POST_ERR_SHIFT		20
> +#define  P_ATR_EVT_FETCH_ERR_MASK		0x00200000u
> +#define  P_ATR_EVT_FETCH_ERR_SHIFT		21
> +#define  P_ATR_EVT_DISCARD_ERR_MASK		0x00400000u
> +#define  P_ATR_EVT_DISCARD_ERR_SHIFT		22
> +#define  P_ATR_EVT_DOORBELL_MASK		0x00000000u
> +#define  P_ATR_EVT_DOORBELL_SHIFT		23
> +#define  PM_MSI_INT_INTA_MASK			0x01000000u
> +#define  PM_MSI_INT_INTA_SHIFT			24
> +#define  PM_MSI_INT_INTB_MASK			0x02000000u
> +#define  PM_MSI_INT_INTB_SHIFT			25
> +#define  PM_MSI_INT_INTC_MASK			0x04000000u
> +#define  PM_MSI_INT_INTC_SHIFT			26
> +#define  PM_MSI_INT_INTD_MASK			0x08000000u
> +#define  PM_MSI_INT_INTD_SHIFT			27
> +#define  PM_MSI_INT_INTX_MASK			0x0f000000u
> +#define  PM_MSI_INT_INTX_SHIFT			24
> +#define  PM_MSI_INT_MSI_MASK			0x10000000u
> +#define  PM_MSI_INT_MSI_SHIFT			28
> +#define  PM_MSI_INT_AER_EVT_MASK		0x20000000u
> +#define  PM_MSI_INT_AER_EVT_SHIFT		29
> +#define  PM_MSI_INT_EVENTS_MASK			0x40000000u
> +#define  PM_MSI_INT_EVENTS_SHIFT		30
> +#define  PM_MSI_INT_SYS_ERR_MASK		0x80000000u
> +#define  PM_MSI_INT_SYS_ERR_SHIFT		31
> +#define  NUM_LOCAL_EVENTS			15
> +#define ISTATUS_LOCAL				0x184
> +#define IMASK_HOST				0x188
> +#define ISTATUS_HOST				0x18c
> +#define IMSI_ADDR				0x190
> +#define ISTATUS_MSI				0x194
> +
> +/* PCIe Master table init defines */
> +#define ATR0_PCIE_WIN0_SRCADDR_PARAM		0x600u
> +#define  ATR0_PCIE_ATR_SIZE			0x25
> +#define  ATR0_PCIE_ATR_SIZE_SHIFT		1
> +#define ATR0_PCIE_WIN0_SRC_ADDR			0x604u
> +#define ATR0_PCIE_WIN0_TRSL_ADDR_LSB		0x608u
> +#define ATR0_PCIE_WIN0_TRSL_ADDR_UDW		0x60cu
> +#define ATR0_PCIE_WIN0_TRSL_PARAM		0x610u
> +
> +/* PCIe AXI slave table init defines */
> +#define ATR0_AXI4_SLV0_SRCADDR_PARAM		0x800u
> +#define  ATR_SIZE_SHIFT				1
> +#define  ATR_IMPL_ENABLE			1
> +#define ATR0_AXI4_SLV0_SRC_ADDR			0x804u
> +#define ATR0_AXI4_SLV0_TRSL_ADDR_LSB		0x808u
> +#define ATR0_AXI4_SLV0_TRSL_ADDR_UDW		0x80cu
> +#define ATR0_AXI4_SLV0_TRSL_PARAM		0x810u
> +#define  PCIE_TX_RX_INTERFACE			0x00000000u
> +#define  PCIE_CONFIG_INTERFACE			0x00000001u
> +
> +#define ATR_ENTRY_SIZE				32
> +
> +enum plda_int_event {
> +	PLDA_AXI_POST_ERR,
> +	PLDA_AXI_FETCH_ERR,
> +	PLDA_AXI_DISCARD_ERR,
> +	PLDA_AXI_DOORBELL,
> +	PLDA_PCIE_POST_ERR,
> +	PLDA_PCIE_FETCH_ERR,
> +	PLDA_PCIE_DISCARD_ERR,
> +	PLDA_PCIE_DOORBELL,
> +	PLDA_INTX,
> +	PLDA_MSI,
> +	PLDA_AER_EVENT,
> +	PLDA_MISC_EVENTS,
> +	PLDA_SYS_ERR,
> +	PLDA_INT_EVENT_NUM
> +};
> +
> +#define PLDA_NUM_DMA_EVENTS			16
> +
> +#define PLDA_MAX_EVENT_NUM			(PLDA_NUM_DMA_EVENTS + PLDA_INT_EVENT_NUM)
> +
> +#endif
> -- 
> 2.17.1
> 

Re: [PATCH v15,RESEND 03/23] PCI: microchip: Move PLDA IP register macros to pcie-plda.h

[Minda Chen]

> 
> On Tue, Feb 27, 2024 at 06:35:02PM +0800, Minda Chen wrote:
> > Move PLDA PCIe host controller IP registers macros to pcie-plda.h,
> > including bridge registers and PLDA IRQ event number.
> >
> > Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> > Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
> > ---
> >  MAINTAINERS                                   |   8 ++
> >  .../pci/controller/plda/pcie-microchip-host.c | 108 +++---------------
> >  drivers/pci/controller/plda/pcie-plda.h       | 108 ++++++++++++++++++
> >  3 files changed, 132 insertions(+), 92 deletions(-)  create mode
> > 100644 drivers/pci/controller/plda/pcie-plda.h
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS index
> > 9f7e907c8749..fc9576e69a71 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -16814,6 +16814,14 @@ S:	Maintained
> >  F:	Documentation/devicetree/bindings/pci/layerscape-pcie-gen4.txt
> >  F:	drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> >
> > +PCI DRIVER FOR PLDA PCIE IP
> > +M:	Daire McNamara <daire.mcnamara@microchip.com>
> > +M:	Kevin Xie <kevin.xie@starfivetech.com>
> > +L:	linux-pci@vger.kernel.org
> > +S:	Maintained
> > +F:	Documentation/devicetree/bindings/pci/plda,*
> 
> This is an unrelated change. It belongs to the patch adding DT bindings or I am
> reading it wrong ?
> 
> Lorenzo
> 
You are right. Thanks
Moveing "F: Documentation/devicetree/bindings/pci/plda,*" to 1st patch is better.
This patch just add " F:	drivers/pci/controller/plda/*plda*"

> > +F:	drivers/pci/controller/plda/*plda*
> > +
> >  PCI DRIVER FOR RENESAS R-CAR
> >  M:	Marek Vasut <marek.vasut+renesas@gmail.com>
> >  M:	Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c
> > b/drivers/pci/controller/plda/pcie-microchip-host.c
> > index cb09a8137e25..d9030d550482 100644
> > --- a/drivers/pci/controller/plda/pcie-microchip-host.c
> > +++ b/drivers/pci/controller/plda/pcie-microchip-host.c
> > @@ -19,6 +19,7 @@
> >  #include <linux/platform_device.h>
> >
> >  #include "../../pci.h"
> > +#include "pcie-plda.h"
> >
> >  /* Number of MSI IRQs */
> >  #define MC_MAX_NUM_MSI_IRQS			32
> > @@ -30,84 +31,6 @@
> >  #define MC_PCIE_BRIDGE_ADDR			(MC_PCIE1_BRIDGE_ADDR)
> >  #define MC_PCIE_CTRL_ADDR			(MC_PCIE1_CTRL_ADDR)
> >
> > -/* PCIe Bridge Phy Regs */
> > -#define PCIE_PCI_IRQ_DW0			0xa8
> > -#define  MSIX_CAP_MASK				BIT(31)
> > -#define  NUM_MSI_MSGS_MASK			GENMASK(6, 4)
> > -#define  NUM_MSI_MSGS_SHIFT			4
> > -
> > -#define IMASK_LOCAL				0x180
> > -#define  DMA_END_ENGINE_0_MASK			0x00000000u
> > -#define  DMA_END_ENGINE_0_SHIFT			0
> > -#define  DMA_END_ENGINE_1_MASK			0x00000000u
> > -#define  DMA_END_ENGINE_1_SHIFT			1
> > -#define  DMA_ERROR_ENGINE_0_MASK		0x00000100u
> > -#define  DMA_ERROR_ENGINE_0_SHIFT		8
> > -#define  DMA_ERROR_ENGINE_1_MASK		0x00000200u
> > -#define  DMA_ERROR_ENGINE_1_SHIFT		9
> > -#define  A_ATR_EVT_POST_ERR_MASK		0x00010000u
> > -#define  A_ATR_EVT_POST_ERR_SHIFT		16
> > -#define  A_ATR_EVT_FETCH_ERR_MASK		0x00020000u
> > -#define  A_ATR_EVT_FETCH_ERR_SHIFT		17
> > -#define  A_ATR_EVT_DISCARD_ERR_MASK		0x00040000u
> > -#define  A_ATR_EVT_DISCARD_ERR_SHIFT		18
> > -#define  A_ATR_EVT_DOORBELL_MASK		0x00000000u
> > -#define  A_ATR_EVT_DOORBELL_SHIFT		19
> > -#define  P_ATR_EVT_POST_ERR_MASK		0x00100000u
> > -#define  P_ATR_EVT_POST_ERR_SHIFT		20
> > -#define  P_ATR_EVT_FETCH_ERR_MASK		0x00200000u
> > -#define  P_ATR_EVT_FETCH_ERR_SHIFT		21
> > -#define  P_ATR_EVT_DISCARD_ERR_MASK		0x00400000u
> > -#define  P_ATR_EVT_DISCARD_ERR_SHIFT		22
> > -#define  P_ATR_EVT_DOORBELL_MASK		0x00000000u
> > -#define  P_ATR_EVT_DOORBELL_SHIFT		23
> > -#define  PM_MSI_INT_INTA_MASK			0x01000000u
> > -#define  PM_MSI_INT_INTA_SHIFT			24
> > -#define  PM_MSI_INT_INTB_MASK			0x02000000u
> > -#define  PM_MSI_INT_INTB_SHIFT			25
> > -#define  PM_MSI_INT_INTC_MASK			0x04000000u
> > -#define  PM_MSI_INT_INTC_SHIFT			26
> > -#define  PM_MSI_INT_INTD_MASK			0x08000000u
> > -#define  PM_MSI_INT_INTD_SHIFT			27
> > -#define  PM_MSI_INT_INTX_MASK			0x0f000000u
> > -#define  PM_MSI_INT_INTX_SHIFT			24
> > -#define  PM_MSI_INT_MSI_MASK			0x10000000u
> > -#define  PM_MSI_INT_MSI_SHIFT			28
> > -#define  PM_MSI_INT_AER_EVT_MASK		0x20000000u
> > -#define  PM_MSI_INT_AER_EVT_SHIFT		29
> > -#define  PM_MSI_INT_EVENTS_MASK			0x40000000u
> > -#define  PM_MSI_INT_EVENTS_SHIFT		30
> > -#define  PM_MSI_INT_SYS_ERR_MASK		0x80000000u
> > -#define  PM_MSI_INT_SYS_ERR_SHIFT		31
> > -#define  NUM_LOCAL_EVENTS			15
> > -#define ISTATUS_LOCAL				0x184
> > -#define IMASK_HOST				0x188
> > -#define ISTATUS_HOST				0x18c
> > -#define IMSI_ADDR				0x190
> > -#define ISTATUS_MSI				0x194
> > -
> > -/* PCIe Master table init defines */
> > -#define ATR0_PCIE_WIN0_SRCADDR_PARAM		0x600u
> > -#define  ATR0_PCIE_ATR_SIZE			0x25
> > -#define  ATR0_PCIE_ATR_SIZE_SHIFT		1
> > -#define ATR0_PCIE_WIN0_SRC_ADDR			0x604u
> > -#define ATR0_PCIE_WIN0_TRSL_ADDR_LSB		0x608u
> > -#define ATR0_PCIE_WIN0_TRSL_ADDR_UDW		0x60cu
> > -#define ATR0_PCIE_WIN0_TRSL_PARAM		0x610u
> > -
> > -/* PCIe AXI slave table init defines */
> > -#define ATR0_AXI4_SLV0_SRCADDR_PARAM		0x800u
> > -#define  ATR_SIZE_SHIFT				1
> > -#define  ATR_IMPL_ENABLE			1
> > -#define ATR0_AXI4_SLV0_SRC_ADDR			0x804u
> > -#define ATR0_AXI4_SLV0_TRSL_ADDR_LSB		0x808u
> > -#define ATR0_AXI4_SLV0_TRSL_ADDR_UDW		0x80cu
> > -#define ATR0_AXI4_SLV0_TRSL_PARAM		0x810u
> > -#define  PCIE_TX_RX_INTERFACE			0x00000000u
> > -#define  PCIE_CONFIG_INTERFACE			0x00000001u
> > -
> > -#define ATR_ENTRY_SIZE				32
> > -
> >  /* PCIe Controller Phy Regs */
> >  #define SEC_ERROR_EVENT_CNT			0x20
> >  #define DED_ERROR_EVENT_CNT			0x24
> > @@ -179,20 +102,21 @@
> >  #define EVENT_LOCAL_DMA_END_ENGINE_1		12
> >  #define EVENT_LOCAL_DMA_ERROR_ENGINE_0		13
> >  #define EVENT_LOCAL_DMA_ERROR_ENGINE_1		14
> > -#define EVENT_LOCAL_A_ATR_EVT_POST_ERR		15
> > -#define EVENT_LOCAL_A_ATR_EVT_FETCH_ERR		16
> > -#define EVENT_LOCAL_A_ATR_EVT_DISCARD_ERR	17
> > -#define EVENT_LOCAL_A_ATR_EVT_DOORBELL		18
> > -#define EVENT_LOCAL_P_ATR_EVT_POST_ERR		19
> > -#define EVENT_LOCAL_P_ATR_EVT_FETCH_ERR		20
> > -#define EVENT_LOCAL_P_ATR_EVT_DISCARD_ERR	21
> > -#define EVENT_LOCAL_P_ATR_EVT_DOORBELL		22
> > -#define EVENT_LOCAL_PM_MSI_INT_INTX		23
> > -#define EVENT_LOCAL_PM_MSI_INT_MSI		24
> > -#define EVENT_LOCAL_PM_MSI_INT_AER_EVT		25
> > -#define EVENT_LOCAL_PM_MSI_INT_EVENTS		26
> > -#define EVENT_LOCAL_PM_MSI_INT_SYS_ERR		27
> > -#define NUM_EVENTS				28
> > +#define NUM_MC_EVENTS				15
> > +#define EVENT_LOCAL_A_ATR_EVT_POST_ERR		(NUM_MC_EVENTS
> + PLDA_AXI_POST_ERR)
> > +#define EVENT_LOCAL_A_ATR_EVT_FETCH_ERR		(NUM_MC_EVENTS
> + PLDA_AXI_FETCH_ERR)
> > +#define EVENT_LOCAL_A_ATR_EVT_DISCARD_ERR	(NUM_MC_EVENTS +
> PLDA_AXI_DISCARD_ERR)
> > +#define EVENT_LOCAL_A_ATR_EVT_DOORBELL		(NUM_MC_EVENTS
> + PLDA_AXI_DOORBELL)
> > +#define EVENT_LOCAL_P_ATR_EVT_POST_ERR		(NUM_MC_EVENTS
> + PLDA_PCIE_POST_ERR)
> > +#define EVENT_LOCAL_P_ATR_EVT_FETCH_ERR		(NUM_MC_EVENTS
> + PLDA_PCIE_FETCH_ERR)
> > +#define EVENT_LOCAL_P_ATR_EVT_DISCARD_ERR	(NUM_MC_EVENTS +
> PLDA_PCIE_DISCARD_ERR)
> > +#define EVENT_LOCAL_P_ATR_EVT_DOORBELL		(NUM_MC_EVENTS
> + PLDA_PCIE_DOORBELL)
> > +#define EVENT_LOCAL_PM_MSI_INT_INTX		(NUM_MC_EVENTS +
> PLDA_INTX)
> > +#define EVENT_LOCAL_PM_MSI_INT_MSI		(NUM_MC_EVENTS +
> PLDA_MSI)
> > +#define EVENT_LOCAL_PM_MSI_INT_AER_EVT		(NUM_MC_EVENTS
> + PLDA_AER_EVENT)
> > +#define EVENT_LOCAL_PM_MSI_INT_EVENTS		(NUM_MC_EVENTS +
> PLDA_MISC_EVENTS)
> > +#define EVENT_LOCAL_PM_MSI_INT_SYS_ERR		(NUM_MC_EVENTS
> + PLDA_SYS_ERR)
> > +#define NUM_EVENTS				(NUM_MC_EVENTS +
> PLDA_INT_EVENT_NUM)
> >
> >  #define PCIE_EVENT_CAUSE(x, s)	\
> >  	[EVENT_PCIE_ ## x] = { __stringify(x), s } diff --git
> > a/drivers/pci/controller/plda/pcie-plda.h
> > b/drivers/pci/controller/plda/pcie-plda.h
> > new file mode 100644
> > index 000000000000..65e0f3b72184
> > --- /dev/null
> > +++ b/drivers/pci/controller/plda/pcie-plda.h
> > @@ -0,0 +1,108 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * PLDA PCIe host controller driver
> > + */
> > +
> > +#ifndef _PCIE_PLDA_H
> > +#define _PCIE_PLDA_H
> > +
> > +/* PCIe Bridge Phy Regs */
> > +#define PCIE_PCI_IRQ_DW0			0xa8
> > +#define  MSIX_CAP_MASK				BIT(31)
> > +#define  NUM_MSI_MSGS_MASK			GENMASK(6, 4)
> > +#define  NUM_MSI_MSGS_SHIFT			4
> > +
> > +#define IMASK_LOCAL				0x180
> > +#define  DMA_END_ENGINE_0_MASK			0x00000000u
> > +#define  DMA_END_ENGINE_0_SHIFT			0
> > +#define  DMA_END_ENGINE_1_MASK			0x00000000u
> > +#define  DMA_END_ENGINE_1_SHIFT			1
> > +#define  DMA_ERROR_ENGINE_0_MASK		0x00000100u
> > +#define  DMA_ERROR_ENGINE_0_SHIFT		8
> > +#define  DMA_ERROR_ENGINE_1_MASK		0x00000200u
> > +#define  DMA_ERROR_ENGINE_1_SHIFT		9
> > +#define  A_ATR_EVT_POST_ERR_MASK		0x00010000u
> > +#define  A_ATR_EVT_POST_ERR_SHIFT		16
> > +#define  A_ATR_EVT_FETCH_ERR_MASK		0x00020000u
> > +#define  A_ATR_EVT_FETCH_ERR_SHIFT		17
> > +#define  A_ATR_EVT_DISCARD_ERR_MASK		0x00040000u
> > +#define  A_ATR_EVT_DISCARD_ERR_SHIFT		18
> > +#define  A_ATR_EVT_DOORBELL_MASK		0x00000000u
> > +#define  A_ATR_EVT_DOORBELL_SHIFT		19
> > +#define  P_ATR_EVT_POST_ERR_MASK		0x00100000u
> > +#define  P_ATR_EVT_POST_ERR_SHIFT		20
> > +#define  P_ATR_EVT_FETCH_ERR_MASK		0x00200000u
> > +#define  P_ATR_EVT_FETCH_ERR_SHIFT		21
> > +#define  P_ATR_EVT_DISCARD_ERR_MASK		0x00400000u
> > +#define  P_ATR_EVT_DISCARD_ERR_SHIFT		22
> > +#define  P_ATR_EVT_DOORBELL_MASK		0x00000000u
> > +#define  P_ATR_EVT_DOORBELL_SHIFT		23
> > +#define  PM_MSI_INT_INTA_MASK			0x01000000u
> > +#define  PM_MSI_INT_INTA_SHIFT			24
> > +#define  PM_MSI_INT_INTB_MASK			0x02000000u
> > +#define  PM_MSI_INT_INTB_SHIFT			25
> > +#define  PM_MSI_INT_INTC_MASK			0x04000000u
> > +#define  PM_MSI_INT_INTC_SHIFT			26
> > +#define  PM_MSI_INT_INTD_MASK			0x08000000u
> > +#define  PM_MSI_INT_INTD_SHIFT			27
> > +#define  PM_MSI_INT_INTX_MASK			0x0f000000u
> > +#define  PM_MSI_INT_INTX_SHIFT			24
> > +#define  PM_MSI_INT_MSI_MASK			0x10000000u
> > +#define  PM_MSI_INT_MSI_SHIFT			28
> > +#define  PM_MSI_INT_AER_EVT_MASK		0x20000000u
> > +#define  PM_MSI_INT_AER_EVT_SHIFT		29
> > +#define  PM_MSI_INT_EVENTS_MASK			0x40000000u
> > +#define  PM_MSI_INT_EVENTS_SHIFT		30
> > +#define  PM_MSI_INT_SYS_ERR_MASK		0x80000000u
> > +#define  PM_MSI_INT_SYS_ERR_SHIFT		31
> > +#define  NUM_LOCAL_EVENTS			15
> > +#define ISTATUS_LOCAL				0x184
> > +#define IMASK_HOST				0x188
> > +#define ISTATUS_HOST				0x18c
> > +#define IMSI_ADDR				0x190
> > +#define ISTATUS_MSI				0x194
> > +
> > +/* PCIe Master table init defines */
> > +#define ATR0_PCIE_WIN0_SRCADDR_PARAM		0x600u
> > +#define  ATR0_PCIE_ATR_SIZE			0x25
> > +#define  ATR0_PCIE_ATR_SIZE_SHIFT		1
> > +#define ATR0_PCIE_WIN0_SRC_ADDR			0x604u
> > +#define ATR0_PCIE_WIN0_TRSL_ADDR_LSB		0x608u
> > +#define ATR0_PCIE_WIN0_TRSL_ADDR_UDW		0x60cu
> > +#define ATR0_PCIE_WIN0_TRSL_PARAM		0x610u
> > +
> > +/* PCIe AXI slave table init defines */
> > +#define ATR0_AXI4_SLV0_SRCADDR_PARAM		0x800u
> > +#define  ATR_SIZE_SHIFT				1
> > +#define  ATR_IMPL_ENABLE			1
> > +#define ATR0_AXI4_SLV0_SRC_ADDR			0x804u
> > +#define ATR0_AXI4_SLV0_TRSL_ADDR_LSB		0x808u
> > +#define ATR0_AXI4_SLV0_TRSL_ADDR_UDW		0x80cu
> > +#define ATR0_AXI4_SLV0_TRSL_PARAM		0x810u
> > +#define  PCIE_TX_RX_INTERFACE			0x00000000u
> > +#define  PCIE_CONFIG_INTERFACE			0x00000001u
> > +
> > +#define ATR_ENTRY_SIZE				32
> > +
> > +enum plda_int_event {
> > +	PLDA_AXI_POST_ERR,
> > +	PLDA_AXI_FETCH_ERR,
> > +	PLDA_AXI_DISCARD_ERR,
> > +	PLDA_AXI_DOORBELL,
> > +	PLDA_PCIE_POST_ERR,
> > +	PLDA_PCIE_FETCH_ERR,
> > +	PLDA_PCIE_DISCARD_ERR,
> > +	PLDA_PCIE_DOORBELL,
> > +	PLDA_INTX,
> > +	PLDA_MSI,
> > +	PLDA_AER_EVENT,
> > +	PLDA_MISC_EVENTS,
> > +	PLDA_SYS_ERR,
> > +	PLDA_INT_EVENT_NUM
> > +};
> > +
> > +#define PLDA_NUM_DMA_EVENTS			16
> > +
> > +#define PLDA_MAX_EVENT_NUM			(PLDA_NUM_DMA_EVENTS +
> PLDA_INT_EVENT_NUM)
> > +
> > +#endif
> > --
> > 2.17.1
> >

Re: [PATCH v15,RESEND 21/23] PCI: starfive: Add JH7110 PCIe controller

[Lorenzo Pieralisi]

On Tue, Feb 27, 2024 at 06:35:20PM +0800, Minda Chen wrote:
> Add StarFive JH7110 SoC PCIe controller platform driver codes, JH7110
> with PLDA host PCIe core.
> 
> Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> Co-developed-by: Kevin Xie <kevin.xie@starfivetech.com>
> Reviewed-by: Mason Huo <mason.huo@starfivetech.com>
> ---
>  MAINTAINERS                                 |   7 +
>  drivers/pci/controller/plda/Kconfig         |  12 +
>  drivers/pci/controller/plda/Makefile        |   1 +
>  drivers/pci/controller/plda/pcie-plda.h     |  71 ++-
>  drivers/pci/controller/plda/pcie-starfive.c | 473 ++++++++++++++++++++
>  5 files changed, 563 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/pci/controller/plda/pcie-starfive.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index fc9576e69a71..d3b0a0fb754f 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -17078,6 +17078,13 @@ S:	Maintained
>  F:	Documentation/devicetree/bindings/pci/socionext,uniphier-pcie*
>  F:	drivers/pci/controller/dwc/pcie-uniphier*
>  
> +PCIE DRIVER FOR STARFIVE JH71x0
> +M:	Kevin Xie <kevin.xie@starfivetech.com>
> +L:	linux-pci@vger.kernel.org
> +S:	Maintained
> +F:	Documentation/devicetree/bindings/pci/starfive*
> +F:	drivers/pci/controller/plda/pcie-starfive.c
> +
>  PCIE DRIVER FOR ST SPEAR13XX
>  M:	Pratyush Anand <pratyush.anand@gmail.com>
>  L:	linux-pci@vger.kernel.org
> diff --git a/drivers/pci/controller/plda/Kconfig b/drivers/pci/controller/plda/Kconfig
> index e54a82ee94f5..c0e14146d7e4 100644
> --- a/drivers/pci/controller/plda/Kconfig
> +++ b/drivers/pci/controller/plda/Kconfig
> @@ -15,4 +15,16 @@ config PCIE_MICROCHIP_HOST
>  	  Say Y here if you want kernel to support the Microchip AXI PCIe
>  	  Host Bridge driver.
>  
> +config PCIE_STARFIVE_HOST
> +	tristate "StarFive PCIe host controller"
> +	depends on PCI_MSI && OF
> +	depends on ARCH_STARFIVE || COMPILE_TEST
> +	select PCIE_PLDA_HOST
> +	help
> +	  Say Y here if you want to support the StarFive PCIe controller in
> +	  host mode. StarFive PCIe controller uses PLDA PCIe core.
> +
> +	  If you choose to build this driver as module it will be dynamically
> +	  linked and module will be called pcie-starfive.ko.
> +
>  endmenu
> diff --git a/drivers/pci/controller/plda/Makefile b/drivers/pci/controller/plda/Makefile
> index 4340ab007f44..0ac6851bed48 100644
> --- a/drivers/pci/controller/plda/Makefile
> +++ b/drivers/pci/controller/plda/Makefile
> @@ -1,3 +1,4 @@
>  # SPDX-License-Identifier: GPL-2.0
>  obj-$(CONFIG_PCIE_PLDA_HOST) += pcie-plda-host.o
>  obj-$(CONFIG_PCIE_MICROCHIP_HOST) += pcie-microchip-host.o
> +obj-$(CONFIG_PCIE_STARFIVE_HOST) += pcie-starfive.o
> diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
> index 7b69891700a4..04e385758a2f 100644
> --- a/drivers/pci/controller/plda/pcie-plda.h
> +++ b/drivers/pci/controller/plda/pcie-plda.h
> @@ -10,10 +10,20 @@
>  #define PLDA_MAX_NUM_MSI_IRQS			32
>  
>  /* PCIe Bridge Phy Regs */
> +#define GEN_SETTINGS				0x80
> +#define  RP_ENABLE				1
> +#define PCIE_PCI_IDS_DW1			0x9c
> +#define  IDS_CLASS_CODE_SHIFT			16
> +#define  REVISION_ID_MASK			GENMASK(7, 0)
> +#define  CLASS_CODE_ID_MASK			GENMASK(31, 8)
>  #define PCIE_PCI_IRQ_DW0			0xa8
>  #define  MSIX_CAP_MASK				BIT(31)
>  #define  NUM_MSI_MSGS_MASK			GENMASK(6, 4)
>  #define  NUM_MSI_MSGS_SHIFT			4
> +#define PCI_MISC				0xb4
> +#define  PHY_FUNCTION_DIS			BIT(15)
> +#define PCIE_WINROM				0xfc
> +#define  PREF_MEM_WIN_64_SUPPORT		BIT(3)
>  
>  #define IMASK_LOCAL				0x180
>  #define  DMA_END_ENGINE_0_MASK			0x00000000u
> @@ -65,6 +75,8 @@
>  #define ISTATUS_HOST				0x18c
>  #define IMSI_ADDR				0x190
>  #define ISTATUS_MSI				0x194
> +#define PMSG_SUPPORT_RX				0x3f0
> +#define  PMSG_LTR_SUPPORT			BIT(2)
>  
>  /* PCIe Master table init defines */
>  #define ATR0_PCIE_WIN0_SRCADDR_PARAM		0x600u
> @@ -86,6 +98,8 @@
>  #define  PCIE_TX_RX_INTERFACE			0x00000000u
>  #define  PCIE_CONFIG_INTERFACE			0x00000001u
>  
> +#define CONFIG_SPACE_ADDR_OFFSET		0x1000u
> +
>  #define ATR_ENTRY_SIZE				32
>  
>  enum plda_int_event {
> @@ -200,4 +214,59 @@ static inline void plda_set_default_msi(struct plda_msi *msi)
>  	msi->vector_phy = IMSI_ADDR;
>  	msi->num_vectors = PLDA_MAX_NUM_MSI_IRQS;
>  }
> -#endif
> +
> +static inline void plda_pcie_enable_root_port(struct plda_pcie_rp *plda)
> +{
> +	u32 value;
> +
> +	value = readl_relaxed(plda->bridge_addr + GEN_SETTINGS);
> +	value |= RP_ENABLE;
> +	writel_relaxed(value, plda->bridge_addr + GEN_SETTINGS);
> +}
> +
> +static inline void plda_pcie_set_standard_class(struct plda_pcie_rp *plda)
> +{
> +	u32 value;
> +
> +	/* set class code and reserve revision id */
> +	value = readl_relaxed(plda->bridge_addr + PCIE_PCI_IDS_DW1);
> +	value &= REVISION_ID_MASK;
> +	value |= (PCI_CLASS_BRIDGE_PCI << IDS_CLASS_CODE_SHIFT);
> +	writel_relaxed(value, plda->bridge_addr + PCIE_PCI_IDS_DW1);
> +}
> +
> +static inline void plda_pcie_set_pref_win_64bit(struct plda_pcie_rp *plda)
> +{
> +	u32 value;
> +
> +	value = readl_relaxed(plda->bridge_addr + PCIE_WINROM);
> +	value |= PREF_MEM_WIN_64_SUPPORT;
> +	writel_relaxed(value, plda->bridge_addr + PCIE_WINROM);
> +}

What does this do ?

> +static inline void plda_pcie_disable_ltr(struct plda_pcie_rp *plda)
> +{
> +	u32 value;
> +
> +	value = readl_relaxed(plda->bridge_addr + PMSG_SUPPORT_RX);
> +	value &= ~PMSG_LTR_SUPPORT;
> +	writel_relaxed(value, plda->bridge_addr + PMSG_SUPPORT_RX);
> +}
> +
> +static inline void plda_pcie_disable_func(struct plda_pcie_rp *plda)
> +{
> +	u32 value;
> +
> +	value = readl_relaxed(plda->bridge_addr + PCI_MISC);
> +	value |= PHY_FUNCTION_DIS;
> +	writel_relaxed(value, plda->bridge_addr + PCI_MISC);
> +}
> +
> +static inline void plda_pcie_write_rc_bar(struct plda_pcie_rp *plda, u64 val)
> +{
> +	void __iomem *addr = plda->bridge_addr + CONFIG_SPACE_ADDR_OFFSET;
> +
> +	writel_relaxed(lower_32_bits(val), addr + PCI_BASE_ADDRESS_0);
> +	writel_relaxed(upper_32_bits(val), addr + PCI_BASE_ADDRESS_1);
> +}
> +#endif /* _PCIE_PLDA_H */
> diff --git a/drivers/pci/controller/plda/pcie-starfive.c b/drivers/pci/controller/plda/pcie-starfive.c
> new file mode 100644
> index 000000000000..9bb9f0e29565
> --- /dev/null
> +++ b/drivers/pci/controller/plda/pcie-starfive.c
> @@ -0,0 +1,473 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * PCIe host controller driver for StarFive JH7110 Soc.
> + *
> + * Copyright (C) 2023 StarFive Technology Co., Ltd.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/clk.h>
> +#include <linux/delay.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/mfd/syscon.h>
> +#include <linux/module.h>
> +#include <linux/of_address.h>
> +#include <linux/of_irq.h>
> +#include <linux/of_pci.h>
> +#include <linux/pci.h>
> +#include <linux/phy/phy.h>
> +#include <linux/platform_device.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/regmap.h>
> +#include <linux/reset.h>
> +#include "../../pci.h"
> +
> +#include "pcie-plda.h"
> +
> +#define PCIE_FUNC_NUM			4
> +
> +/* system control */
> +#define STG_SYSCON_PCIE0_BASE			0x48
> +#define STG_SYSCON_PCIE1_BASE			0x1f8
> +
> +#define STG_SYSCON_AR_OFFSET			0x78
> +#define STG_SYSCON_AXI4_SLVL_AR_MASK		GENMASK(22, 8)
> +#define STG_SYSCON_AXI4_SLVL_PHY_AR(x)		FIELD_PREP(GENMASK(20, 17), x)
> +#define STG_SYSCON_AW_OFFSET			0x7c
> +#define STG_SYSCON_AXI4_SLVL_AW_MASK		GENMASK(14, 0)
> +#define STG_SYSCON_AXI4_SLVL_PHY_AW(x)		FIELD_PREP(GENMASK(12, 9), x)
> +#define STG_SYSCON_CLKREQ			BIT(22)
> +#define STG_SYSCON_CKREF_SRC_MASK		GENMASK(19, 18)
> +#define STG_SYSCON_RP_NEP_OFFSET		0xe8
> +#define STG_SYSCON_K_RP_NEP			BIT(8)
> +#define STG_SYSCON_LNKSTA_OFFSET		0x170
> +#define DATA_LINK_ACTIVE			BIT(5)
> +
> +/* Parameters for the waiting for link up routine */
> +#define LINK_WAIT_MAX_RETRIES	10
> +#define LINK_WAIT_USLEEP_MIN	90000
> +#define LINK_WAIT_USLEEP_MAX	100000
> +
> +struct starfive_jh7110_pcie {
> +	struct plda_pcie_rp plda;
> +	struct reset_control *resets;
> +	struct clk_bulk_data *clks;
> +	struct regmap *reg_syscon;
> +	struct gpio_desc *power_gpio;
> +	struct gpio_desc *reset_gpio;
> +	struct phy *phy;
> +
> +	unsigned int stg_pcie_base;
> +	int num_clks;
> +};
> +
> +/*
> + * The BAR0/1 of bridge should be hidden during enumeration to
> + * avoid the sizing and resource allocation by PCIe core.

Please explain. This is what's preventing this driver to re-use
the PCI_HOST_COMMON infrastructure and it is unclear why.

> + */
> +static bool starfive_pcie_hide_rc_bar(struct pci_bus *bus, unsigned int devfn,
> +				      int offset)
> +{
> +	if (pci_is_root_bus(bus) && !devfn &&
> +	    (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1))
> +		return true;
> +
> +	return false;
> +}
> +
> +static int starfive_pcie_config_write(struct pci_bus *bus, unsigned int devfn,
> +				      int where, int size, u32 value)
> +{
> +	if (starfive_pcie_hide_rc_bar(bus, devfn, where))
> +		return PCIBIOS_SUCCESSFUL;
> +
> +	return pci_generic_config_write(bus, devfn, where, size, value);
> +}
> +
> +static int starfive_pcie_config_read(struct pci_bus *bus, unsigned int devfn,
> +				     int where, int size, u32 *value)
> +{
> +	if (starfive_pcie_hide_rc_bar(bus, devfn, where)) {
> +		*value = 0;
> +		return PCIBIOS_SUCCESSFUL;
> +	}
> +
> +	return pci_generic_config_read(bus, devfn, where, size, value);
> +}
> +
> +static int starfive_pcie_parse_dt(struct starfive_jh7110_pcie *pcie,
> +				  struct device *dev)
> +{
> +	int domain_nr;
> +
> +	pcie->num_clks = devm_clk_bulk_get_all(dev, &pcie->clks);
> +	if (pcie->num_clks < 0)
> +		return dev_err_probe(dev, pcie->num_clks,
> +				     "failed to get pcie clocks\n");
> +
> +	pcie->resets = devm_reset_control_array_get_exclusive(dev);
> +	if (IS_ERR(pcie->resets))
> +		return dev_err_probe(dev, PTR_ERR(pcie->resets),
> +				     "failed to get pcie resets");
> +
> +	pcie->reg_syscon =
> +		syscon_regmap_lookup_by_phandle(dev->of_node,
> +						"starfive,stg-syscon");
> +
> +	if (IS_ERR(pcie->reg_syscon))
> +		return dev_err_probe(dev, PTR_ERR(pcie->reg_syscon),
> +				     "failed to parse starfive,stg-syscon\n");
> +
> +	pcie->phy = devm_phy_optional_get(dev, NULL);
> +	if (IS_ERR(pcie->phy))
> +		return dev_err_probe(dev, PTR_ERR(pcie->phy),
> +				     "failed to get pcie phy\n");
> +
> +	domain_nr = of_get_pci_domain_nr(dev->of_node);
> +
> +	if (domain_nr < 0 || domain_nr > 1)
> +		return dev_err_probe(dev, -ENODEV,
> +				     "failed to get valid pcie domain\n");
> +
> +	if (domain_nr == 0)
> +		pcie->stg_pcie_base = STG_SYSCON_PCIE0_BASE;
> +	else
> +		pcie->stg_pcie_base = STG_SYSCON_PCIE1_BASE;

I don't agree with this. The domain number is a software allocated
value attached to a controller. It is written nowhere that you can't
swap them and leave this code path broken.

> +	pcie->reset_gpio = devm_gpiod_get_optional(dev, "perst",
> +						   GPIOD_OUT_HIGH);
> +	if (IS_ERR(pcie->reset_gpio))
> +		return dev_err_probe(dev, PTR_ERR(pcie->reset_gpio),
> +				     "failed to get perst-gpio\n");
> +
> +	pcie->power_gpio = devm_gpiod_get_optional(dev, "enable",
> +						   GPIOD_OUT_LOW);
> +	if (IS_ERR(pcie->power_gpio))
> +		return dev_err_probe(dev, PTR_ERR(pcie->power_gpio),
> +				     "failed to get power-gpio\n");
> +
> +	return 0;
> +}
> +
> +static struct pci_ops starfive_pcie_ops = {
> +	.map_bus	= plda_pcie_map_bus,

It looks like this is an ECAM driver with the read/config quirks
for read/write in the method below. Please explain why they are needed.

> +	.read           = starfive_pcie_config_read,
> +	.write          = starfive_pcie_config_write,
> +};
> +
> +static int starfive_pcie_clk_rst_init(struct starfive_jh7110_pcie *pcie)
> +{
> +	struct device *dev = pcie->plda.dev;
> +	int ret;
> +
> +	ret = clk_bulk_prepare_enable(pcie->num_clks, pcie->clks);
> +	if (ret)
> +		return dev_err_probe(dev, ret, "failed to enable clocks\n");
> +
> +	ret = reset_control_deassert(pcie->resets);
> +	if (ret) {
> +		clk_bulk_disable_unprepare(pcie->num_clks, pcie->clks);
> +		dev_err_probe(dev, ret, "failed to deassert resets\n");
> +	}
> +
> +	return ret;
> +}
> +
> +static void starfive_pcie_clk_rst_deinit(struct starfive_jh7110_pcie *pcie)
> +{
> +	reset_control_assert(pcie->resets);
> +	clk_bulk_disable_unprepare(pcie->num_clks, pcie->clks);
> +}
> +
> +static bool starfive_pcie_link_up(struct plda_pcie_rp *plda)
> +{
> +	struct starfive_jh7110_pcie *pcie =
> +		container_of(plda, struct starfive_jh7110_pcie, plda);
> +	int ret;
> +	u32 stg_reg_val;
> +
> +	ret = regmap_read(pcie->reg_syscon,
> +			  pcie->stg_pcie_base + STG_SYSCON_LNKSTA_OFFSET,
> +			  &stg_reg_val);
> +	if (ret) {
> +		dev_err(pcie->plda.dev, "failed to read link status\n");
> +		return false;
> +	}
> +
> +	return !!(stg_reg_val & DATA_LINK_ACTIVE);
> +}
> +
> +static int starfive_pcie_host_wait_for_link(struct starfive_jh7110_pcie *pcie)
> +{
> +	int retries;
> +
> +	/* Check if the link is up or not */
> +	for (retries = 0; retries < LINK_WAIT_MAX_RETRIES; retries++) {
> +		if (starfive_pcie_link_up(&pcie->plda)) {
> +			dev_info(pcie->plda.dev, "port link up\n");
> +			return 0;
> +		}
> +		usleep_range(LINK_WAIT_USLEEP_MIN, LINK_WAIT_USLEEP_MAX);
> +	}
> +
> +	return -ETIMEDOUT;
> +}
> +
> +static int starfive_pcie_enable_phy(struct device *dev,
> +				    struct starfive_jh7110_pcie *pcie)
> +{
> +	int ret;
> +
> +	if (!pcie->phy)
> +		return 0;
> +
> +	ret = phy_init(pcie->phy);
> +	if (ret)
> +		return dev_err_probe(dev, ret,
> +				     "failed to initialize pcie phy\n");
> +
> +	ret = phy_set_mode(pcie->phy, PHY_MODE_PCIE);
> +	if (ret) {
> +		dev_err_probe(dev, ret, "failed to set pcie mode\n");
> +		goto err_phy_on;
> +	}
> +
> +	ret = phy_power_on(pcie->phy);
> +	if (ret) {
> +		dev_err_probe(dev, ret, "failed to power on pcie phy\n");
> +		goto err_phy_on;
> +	}
> +
> +	return 0;
> +
> +err_phy_on:
> +	phy_exit(pcie->phy);
> +	return ret;
> +}
> +
> +static void starfive_pcie_disable_phy(struct starfive_jh7110_pcie *pcie)
> +{
> +	phy_power_off(pcie->phy);
> +	phy_exit(pcie->phy);
> +}
> +
> +static void starfive_pcie_host_deinit(struct plda_pcie_rp *plda)
> +{
> +	struct starfive_jh7110_pcie *pcie =
> +		container_of(plda, struct starfive_jh7110_pcie, plda);
> +
> +	starfive_pcie_clk_rst_deinit(pcie);
> +	if (pcie->power_gpio)
> +		gpiod_set_value_cansleep(pcie->power_gpio, 0);
> +	starfive_pcie_disable_phy(pcie);
> +}
> +
> +static int starfive_pcie_host_init(struct plda_pcie_rp *plda)
> +{
> +	struct starfive_jh7110_pcie *pcie =
> +		container_of(plda, struct starfive_jh7110_pcie, plda);
> +	struct device *dev = plda->dev;
> +	int ret;
> +	int i;
> +
> +	ret = starfive_pcie_enable_phy(dev, pcie);
> +	if (ret)
> +		return ret;
> +
> +	regmap_update_bits(pcie->reg_syscon,
> +			   pcie->stg_pcie_base + STG_SYSCON_RP_NEP_OFFSET,
> +			   STG_SYSCON_K_RP_NEP, STG_SYSCON_K_RP_NEP);
> +
> +	regmap_update_bits(pcie->reg_syscon,
> +			   pcie->stg_pcie_base + STG_SYSCON_AW_OFFSET,
> +			   STG_SYSCON_CKREF_SRC_MASK,
> +			   FIELD_PREP(STG_SYSCON_CKREF_SRC_MASK, 2));
> +
> +	regmap_update_bits(pcie->reg_syscon,
> +			   pcie->stg_pcie_base + STG_SYSCON_AW_OFFSET,
> +			   STG_SYSCON_CLKREQ, STG_SYSCON_CLKREQ);
> +
> +	ret = starfive_pcie_clk_rst_init(pcie);
> +	if (ret)
> +		return ret;
> +
> +	if (pcie->power_gpio)
> +		gpiod_set_value_cansleep(pcie->power_gpio, 1);
> +
> +	if (pcie->reset_gpio)
> +		gpiod_set_value_cansleep(pcie->reset_gpio, 1);
> +
> +	/* Disable physical functions except #0 */
> +	for (i = 1; i < PCIE_FUNC_NUM; i++) {
> +		regmap_update_bits(pcie->reg_syscon,
> +				   pcie->stg_pcie_base + STG_SYSCON_AR_OFFSET,
> +				   STG_SYSCON_AXI4_SLVL_AR_MASK,
> +				   STG_SYSCON_AXI4_SLVL_PHY_AR(i));
> +
> +		regmap_update_bits(pcie->reg_syscon,
> +				   pcie->stg_pcie_base + STG_SYSCON_AW_OFFSET,
> +				   STG_SYSCON_AXI4_SLVL_AW_MASK,
> +				   STG_SYSCON_AXI4_SLVL_PHY_AW(i));
> +
> +		plda_pcie_disable_func(plda);
> +	}
> +
> +	regmap_update_bits(pcie->reg_syscon,
> +			   pcie->stg_pcie_base + STG_SYSCON_AR_OFFSET,
> +			   STG_SYSCON_AXI4_SLVL_AR_MASK, 0);
> +	regmap_update_bits(pcie->reg_syscon,
> +			   pcie->stg_pcie_base + STG_SYSCON_AW_OFFSET,
> +			   STG_SYSCON_AXI4_SLVL_AW_MASK, 0);
> +
> +	plda_pcie_enable_root_port(plda);
> +	plda_pcie_write_rc_bar(plda, 0);
> +
> +	/* PCIe PCI Standard Configuration Identification Settings. */
> +	plda_pcie_set_standard_class(plda);
> +
> +	/*
> +	 * The LTR message forwarding of PCIe Message Reception was set by core

"LTR message forwarding of PCIe Message Reception".

What is this referring to ?

Thanks,
Lorenzo

> +	 * as default, but the forward id & addr are also need to be reset.
> +	 * If we do not disable LTR message forwarding here, or set a legal
> +	 * forwarding address, the kernel will get stuck after the driver probe.
> +	 * To workaround, disable the LTR message forwarding support on
> +	 * PCIe Message Reception.
> +	 */
> +	plda_pcie_disable_ltr(plda);
> +
> +	/* Prefetchable memory window 64-bit addressing support */
> +	plda_pcie_set_pref_win_64bit(plda);
> +
> +	/*
> +	 * Ensure that PERST has been asserted for at least 100 ms,
> +	 * the sleep value is T_PVPERL from PCIe CEM spec r2.0 (Table 2-4)
> +	 */
> +	msleep(100);
> +	if (pcie->reset_gpio)
> +		gpiod_set_value_cansleep(pcie->reset_gpio, 0);
> +
> +	/*
> +	 * With a Downstream Port (<=5GT/s), software must wait a minimum
> +	 * of 100ms following exit from a conventional reset before
> +	 * sending a configuration request to the device.
> +	 */
> +	msleep(PCIE_RESET_CONFIG_DEVICE_WAIT_MS);
> +
> +	if (starfive_pcie_host_wait_for_link(pcie))
> +		dev_info(dev, "port link down\n");
> +
> +	return 0;
> +}
> +
> +static const struct plda_pcie_host_ops sf_host_ops = {
> +	.host_init = starfive_pcie_host_init,
> +	.host_deinit = starfive_pcie_host_deinit,
> +};
> +
> +static const struct plda_event stf_pcie_event = {
> +	.intx_event = EVENT_PM_MSI_INT_INTX,
> +	.msi_event  = EVENT_PM_MSI_INT_MSI
> +};
> +
> +static int starfive_pcie_probe(struct platform_device *pdev)
> +{
> +	struct starfive_jh7110_pcie *pcie;
> +	struct device *dev = &pdev->dev;
> +	struct plda_pcie_rp *plda;
> +	int ret;
> +
> +	pcie = devm_kzalloc(dev, sizeof(*pcie), GFP_KERNEL);
> +	if (!pcie)
> +		return -ENOMEM;
> +
> +	plda = &pcie->plda;
> +	plda->dev = dev;
> +
> +	ret = starfive_pcie_parse_dt(pcie, dev);
> +	if (ret)
> +		return ret;
> +
> +	plda->host_ops = &sf_host_ops;
> +	plda->num_events = PLDA_MAX_EVENT_NUM;
> +	/* mask doorbell event */
> +	plda->events_bitmap = GENMASK(PLDA_INT_EVENT_NUM - 1, 0)
> +			     & ~BIT(PLDA_AXI_DOORBELL)
> +			     & ~BIT(PLDA_PCIE_DOORBELL);
> +	plda->events_bitmap <<= PLDA_NUM_DMA_EVENTS;
> +	ret = plda_pcie_host_init(&pcie->plda, &starfive_pcie_ops,
> +				  &stf_pcie_event);
> +	if (ret)
> +		return ret;
> +
> +	pm_runtime_enable(&pdev->dev);
> +	pm_runtime_get_sync(&pdev->dev);
> +	platform_set_drvdata(pdev, pcie);
> +
> +	return 0;
> +}
> +
> +static void starfive_pcie_remove(struct platform_device *pdev)
> +{
> +	struct starfive_jh7110_pcie *pcie = platform_get_drvdata(pdev);
> +
> +	pm_runtime_put(&pdev->dev);
> +	pm_runtime_disable(&pdev->dev);
> +	plda_pcie_host_deinit(&pcie->plda);
> +	platform_set_drvdata(pdev, NULL);
> +}
> +
> +static int starfive_pcie_suspend_noirq(struct device *dev)
> +{
> +	struct starfive_jh7110_pcie *pcie = dev_get_drvdata(dev);
> +
> +	clk_bulk_disable_unprepare(pcie->num_clks, pcie->clks);
> +	starfive_pcie_disable_phy(pcie);
> +
> +	return 0;
> +}
> +
> +static int starfive_pcie_resume_noirq(struct device *dev)
> +{
> +	struct starfive_jh7110_pcie *pcie = dev_get_drvdata(dev);
> +	int ret;
> +
> +	ret = starfive_pcie_enable_phy(dev, pcie);
> +	if (ret)
> +		return ret;
> +
> +	ret = clk_bulk_prepare_enable(pcie->num_clks, pcie->clks);
> +	if (ret) {
> +		dev_err(dev, "failed to enable clocks\n");
> +		starfive_pcie_disable_phy(pcie);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct dev_pm_ops starfive_pcie_pm_ops = {
> +	NOIRQ_SYSTEM_SLEEP_PM_OPS(starfive_pcie_suspend_noirq,
> +				  starfive_pcie_resume_noirq)
> +};
> +
> +static const struct of_device_id starfive_pcie_of_match[] = {
> +	{ .compatible = "starfive,jh7110-pcie", },
> +	{ /* sentinel */ }
> +};
> +MODULE_DEVICE_TABLE(of, starfive_pcie_of_match);
> +
> +static struct platform_driver starfive_pcie_driver = {
> +	.driver = {
> +		.name = "pcie-starfive",
> +		.of_match_table = of_match_ptr(starfive_pcie_of_match),
> +		.pm = pm_sleep_ptr(&starfive_pcie_pm_ops),
> +	},
> +	.probe = starfive_pcie_probe,
> +	.remove_new = starfive_pcie_remove,
> +};
> +module_platform_driver(starfive_pcie_driver);
> +
> +MODULE_DESCRIPTION("StarFive JH7110 PCIe host driver");
> +MODULE_LICENSE("GPL v2");
> -- 
> 2.17.1
> 
> 
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers.

[Lorenzo Pieralisi]

On Tue, Feb 27, 2024 at 06:35:21PM +0800, Minda Chen wrote:
> From: Kevin Xie <kevin.xie@starfivetech.com>
> 
> As the Starfive JH7110 hardware can't keep two inbound post write in
> order all the time, such as MSI messages and NVMe completions. If the
> NVMe completion update later than the MSI, an NVMe IRQ handle will miss.

Please explain what the problem is and what "NVMe completions" means
given that you are talking about posted writes.

If you have a link to an erratum write-up it would certainly help.

This looks completely broken to me, if the controller can't guarantee
PCIe transactions ordering it is toast, there is not even a point
considering mainline merging.

> As a workaround, we will wait a while before going to the generic
> handle here.
> 
> Verified with NVMe SSD, USB SSD, R8169 NIC.
> The performance are stable and even higher after this patch.

I assume this is a joke even though it does not make me laugh.

Thanks,
Lorenzo

> 
> Signed-off-by: Kevin Xie <kevin.xie@starfivetech.com>
> Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> ---
>  drivers/pci/controller/plda/pcie-plda-host.c | 12 ++++++++++++
>  drivers/pci/controller/plda/pcie-plda.h      |  1 +
>  drivers/pci/controller/plda/pcie-starfive.c  |  1 +
>  3 files changed, 14 insertions(+)
> 
> diff --git a/drivers/pci/controller/plda/pcie-plda-host.c b/drivers/pci/controller/plda/pcie-plda-host.c
> index a18923d7cea6..9e077ddf45c0 100644
> --- a/drivers/pci/controller/plda/pcie-plda-host.c
> +++ b/drivers/pci/controller/plda/pcie-plda-host.c
> @@ -13,6 +13,7 @@
>  #include <linux/msi.h>
>  #include <linux/pci_regs.h>
>  #include <linux/pci-ecam.h>
> +#include <linux/delay.h>
>  
>  #include "pcie-plda.h"
>  
> @@ -44,6 +45,17 @@ static void plda_handle_msi(struct irq_desc *desc)
>  			       bridge_base_addr + ISTATUS_LOCAL);
>  		status = readl_relaxed(bridge_base_addr + ISTATUS_MSI);
>  		for_each_set_bit(bit, &status, msi->num_vectors) {
> +			/*
> +			 * As the Starfive JH7110 hardware can't keep two
> +			 * inbound post write in order all the time, such as
> +			 * MSI messages and NVMe completions.
> +			 * If the NVMe completion update later than the MSI,
> +			 * an NVMe IRQ handle will miss.
> +			 * As a workaround, we will wait a while before
> +			 * going to the generic handle here.
> +			 */
> +			if (port->msi_quirk_delay_us)
> +				udelay(port->msi_quirk_delay_us);
>  			ret = generic_handle_domain_irq(msi->dev_domain, bit);
>  			if (ret)
>  				dev_err_ratelimited(dev, "bad MSI IRQ %d\n",
> diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
> index 04e385758a2f..feccf285dfe8 100644
> --- a/drivers/pci/controller/plda/pcie-plda.h
> +++ b/drivers/pci/controller/plda/pcie-plda.h
> @@ -186,6 +186,7 @@ struct plda_pcie_rp {
>  	int msi_irq;
>  	int intx_irq;
>  	int num_events;
> +	u16 msi_quirk_delay_us;
>  };
>  
>  struct plda_event {
> diff --git a/drivers/pci/controller/plda/pcie-starfive.c b/drivers/pci/controller/plda/pcie-starfive.c
> index 9bb9f0e29565..5cfc30572b7f 100644
> --- a/drivers/pci/controller/plda/pcie-starfive.c
> +++ b/drivers/pci/controller/plda/pcie-starfive.c
> @@ -391,6 +391,7 @@ static int starfive_pcie_probe(struct platform_device *pdev)
>  
>  	plda->host_ops = &sf_host_ops;
>  	plda->num_events = PLDA_MAX_EVENT_NUM;
> +	plda->msi_quirk_delay_us = 1;
>  	/* mask doorbell event */
>  	plda->events_bitmap = GENMASK(PLDA_INT_EVENT_NUM - 1, 0)
>  			     & ~BIT(PLDA_AXI_DOORBELL)
> -- 
> 2.17.1
> 
> 
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH v15,RESEND 05/23] PCI: microchip: Rename two PCIe data structures

[Minda Chen]

> 
> On Tue, Feb 27, 2024 at 06:35:04PM +0800, Minda Chen wrote:
> > Add PLDA PCIe related data structures by rename data structure name
> > from
> > mc_* to plda_*.
> >
> > axi_base_addr is stayed in struct mc_pcie for it's microchip its own data.
> 
> Here in "for it's microchip" you mean "since it's microchip" own data.
> 
> Please fix this "for" usage in all commit logs, I noticed it is a recurrent mistake
> you make I can do it but it is tedious.
> 
> Don't resend another version yet, I am trying to see what I can take from this
> one.
> 
Thanks. I will change this. I know the Starfive codes need time to review. So previous 
I just request accept some refactoring patches first. It looks now this can't be achieved. 
I will send next version in 6.9-rc1. 
And Kevin will reply the Starfive patch(patch 21, patch 22).

> > The event interrupt codes is still using struct mc_pcie because the
> > event
> 
> "code", please not "codes". Please fix it everywhere in your patch series, same
> as above.
> 
> Lorenzo
> 
I will check all the commit message.

> > interrupt codes can not be re-used.
> >
> > Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> > Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
> > ---
> >  .../pci/controller/plda/pcie-microchip-host.c | 96
> > ++++++++++---------
> >  1 file changed, 53 insertions(+), 43 deletions(-)
> >
> > diff --git a/drivers/pci/controller/plda/pcie-microchip-host.c
> > b/drivers/pci/controller/plda/pcie-microchip-host.c
> > index c55ede80a6d0..df0736f688ce 100644
> > --- a/drivers/pci/controller/plda/pcie-microchip-host.c
> > +++ b/drivers/pci/controller/plda/pcie-microchip-host.c
> > @@ -22,7 +22,7 @@
> >  #include "pcie-plda.h"
> >
> >  /* Number of MSI IRQs */
> > -#define MC_MAX_NUM_MSI_IRQS			32
> > +#define PLDA_MAX_NUM_MSI_IRQS			32
> >
> >  /* PCIe Bridge Phy and Controller Phy offsets */
> >  #define MC_PCIE1_BRIDGE_ADDR			0x00008000u
> > @@ -179,25 +179,29 @@ struct event_map {
> >  	u32 event_bit;
> >  };
> >
> > -struct mc_msi {
> > +struct plda_msi {
> >  	struct mutex lock;		/* Protect used bitmap */
> >  	struct irq_domain *msi_domain;
> >  	struct irq_domain *dev_domain;
> >  	u32 num_vectors;
> >  	u64 vector_phy;
> > -	DECLARE_BITMAP(used, MC_MAX_NUM_MSI_IRQS);
> > +	DECLARE_BITMAP(used, PLDA_MAX_NUM_MSI_IRQS);
> >  };
> >
> > -struct mc_pcie {
> > -	void __iomem *axi_base_addr;
> > +struct plda_pcie_rp {
> >  	struct device *dev;
> >  	struct irq_domain *intx_domain;
> >  	struct irq_domain *event_domain;
> >  	raw_spinlock_t lock;
> > -	struct mc_msi msi;
> > +	struct plda_msi msi;
> >  	void __iomem *bridge_addr;
> >  };
> >
> > +struct mc_pcie {
> > +	struct plda_pcie_rp plda;
> > +	void __iomem *axi_base_addr;
> > +};
> > +
> >  struct cause {
> >  	const char *sym;
> >  	const char *str;
> > @@ -313,7 +317,7 @@ static struct mc_pcie *port;
> >
> >  static void mc_pcie_enable_msi(struct mc_pcie *port, void __iomem
> > *ecam)  {
> > -	struct mc_msi *msi = &port->msi;
> > +	struct plda_msi *msi = &port->plda.msi;
> >  	u16 reg;
> >  	u8 queue_size;
> >
> > @@ -336,10 +340,10 @@ static void mc_pcie_enable_msi(struct mc_pcie
> > *port, void __iomem *ecam)
> >
> >  static void mc_handle_msi(struct irq_desc *desc)  {
> > -	struct mc_pcie *port = irq_desc_get_handler_data(desc);
> > +	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
> >  	struct irq_chip *chip = irq_desc_get_chip(desc);
> >  	struct device *dev = port->dev;
> > -	struct mc_msi *msi = &port->msi;
> > +	struct plda_msi *msi = &port->msi;
> >  	void __iomem *bridge_base_addr = port->bridge_addr;
> >  	unsigned long status;
> >  	u32 bit;
> > @@ -364,7 +368,7 @@ static void mc_handle_msi(struct irq_desc *desc)
> >
> >  static void mc_msi_bottom_irq_ack(struct irq_data *data)  {
> > -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> > +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> >  	void __iomem *bridge_base_addr = port->bridge_addr;
> >  	u32 bitpos = data->hwirq;
> >
> > @@ -373,7 +377,7 @@ static void mc_msi_bottom_irq_ack(struct irq_data
> > *data)
> >
> >  static void mc_compose_msi_msg(struct irq_data *data, struct msi_msg
> > *msg)  {
> > -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> > +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> >  	phys_addr_t addr = port->msi.vector_phy;
> >
> >  	msg->address_lo = lower_32_bits(addr); @@ -400,8 +404,8 @@ static
> > struct irq_chip mc_msi_bottom_irq_chip = {  static int
> > mc_irq_msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
> >  				   unsigned int nr_irqs, void *args)  {
> > -	struct mc_pcie *port = domain->host_data;
> > -	struct mc_msi *msi = &port->msi;
> > +	struct plda_pcie_rp *port = domain->host_data;
> > +	struct plda_msi *msi = &port->msi;
> >  	unsigned long bit;
> >
> >  	mutex_lock(&msi->lock);
> > @@ -425,8 +429,8 @@ static void mc_irq_msi_domain_free(struct
> irq_domain *domain, unsigned int virq,
> >  				   unsigned int nr_irqs)
> >  {
> >  	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > -	struct mc_pcie *port = irq_data_get_irq_chip_data(d);
> > -	struct mc_msi *msi = &port->msi;
> > +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(d);
> > +	struct plda_msi *msi = &port->msi;
> >
> >  	mutex_lock(&msi->lock);
> >
> > @@ -456,11 +460,11 @@ static struct msi_domain_info
> mc_msi_domain_info = {
> >  	.chip = &mc_msi_irq_chip,
> >  };
> >
> > -static int mc_allocate_msi_domains(struct mc_pcie *port)
> > +static int mc_allocate_msi_domains(struct plda_pcie_rp *port)
> >  {
> >  	struct device *dev = port->dev;
> >  	struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
> > -	struct mc_msi *msi = &port->msi;
> > +	struct plda_msi *msi = &port->msi;
> >
> >  	mutex_init(&port->msi.lock);
> >
> > @@ -484,7 +488,7 @@ static int mc_allocate_msi_domains(struct mc_pcie
> > *port)
> >
> >  static void mc_handle_intx(struct irq_desc *desc)  {
> > -	struct mc_pcie *port = irq_desc_get_handler_data(desc);
> > +	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
> >  	struct irq_chip *chip = irq_desc_get_chip(desc);
> >  	struct device *dev = port->dev;
> >  	void __iomem *bridge_base_addr = port->bridge_addr; @@ -511,7 +515,7
> > @@ static void mc_handle_intx(struct irq_desc *desc)
> >
> >  static void mc_ack_intx_irq(struct irq_data *data)  {
> > -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> > +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> >  	void __iomem *bridge_base_addr = port->bridge_addr;
> >  	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT);
> >
> > @@ -520,7 +524,7 @@ static void mc_ack_intx_irq(struct irq_data *data)
> >
> >  static void mc_mask_intx_irq(struct irq_data *data)  {
> > -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> > +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> >  	void __iomem *bridge_base_addr = port->bridge_addr;
> >  	unsigned long flags;
> >  	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT); @@ -535,7
> > +539,7 @@ static void mc_mask_intx_irq(struct irq_data *data)
> >
> >  static void mc_unmask_intx_irq(struct irq_data *data)  {
> > -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> > +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> >  	void __iomem *bridge_base_addr = port->bridge_addr;
> >  	unsigned long flags;
> >  	u32 mask = BIT(data->hwirq + PM_MSI_INT_INTX_SHIFT); @@ -625,21
> > +629,22 @@ static u32 local_events(struct mc_pcie *port)
> >  	return val;
> >  }
> >
> > -static u32 get_events(struct mc_pcie *port)
> > +static u32 get_events(struct plda_pcie_rp *port)
> >  {
> > +	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
> >  	u32 events = 0;
> >
> > -	events |= pcie_events(port);
> > -	events |= sec_errors(port);
> > -	events |= ded_errors(port);
> > -	events |= local_events(port);
> > +	events |= pcie_events(mc_port);
> > +	events |= sec_errors(mc_port);
> > +	events |= ded_errors(mc_port);
> > +	events |= local_events(mc_port);
> >
> >  	return events;
> >  }
> >
> >  static irqreturn_t mc_event_handler(int irq, void *dev_id)  {
> > -	struct mc_pcie *port = dev_id;
> > +	struct plda_pcie_rp *port = dev_id;
> >  	struct device *dev = port->dev;
> >  	struct irq_data *data;
> >
> > @@ -655,7 +660,7 @@ static irqreturn_t mc_event_handler(int irq, void
> > *dev_id)
> >
> >  static void mc_handle_event(struct irq_desc *desc)  {
> > -	struct mc_pcie *port = irq_desc_get_handler_data(desc);
> > +	struct plda_pcie_rp *port = irq_desc_get_handler_data(desc);
> >  	unsigned long events;
> >  	u32 bit;
> >  	struct irq_chip *chip = irq_desc_get_chip(desc); @@ -672,12 +677,13
> > @@ static void mc_handle_event(struct irq_desc *desc)
> >
> >  static void mc_ack_event_irq(struct irq_data *data)  {
> > -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> > +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> > +	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
> >  	u32 event = data->hwirq;
> >  	void __iomem *addr;
> >  	u32 mask;
> >
> > -	addr = port->axi_base_addr + event_descs[event].base +
> > +	addr = mc_port->axi_base_addr + event_descs[event].base +
> >  		event_descs[event].offset;
> >  	mask = event_descs[event].mask;
> >  	mask |= event_descs[event].enb_mask; @@ -687,13 +693,14 @@ static
> > void mc_ack_event_irq(struct irq_data *data)
> >
> >  static void mc_mask_event_irq(struct irq_data *data)  {
> > -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> > +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> > +	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
> >  	u32 event = data->hwirq;
> >  	void __iomem *addr;
> >  	u32 mask;
> >  	u32 val;
> >
> > -	addr = port->axi_base_addr + event_descs[event].base +
> > +	addr = mc_port->axi_base_addr + event_descs[event].base +
> >  		event_descs[event].mask_offset;
> >  	mask = event_descs[event].mask;
> >  	if (event_descs[event].enb_mask) {
> > @@ -717,13 +724,14 @@ static void mc_mask_event_irq(struct irq_data
> > *data)
> >
> >  static void mc_unmask_event_irq(struct irq_data *data)  {
> > -	struct mc_pcie *port = irq_data_get_irq_chip_data(data);
> > +	struct plda_pcie_rp *port = irq_data_get_irq_chip_data(data);
> > +	struct mc_pcie *mc_port = container_of(port, struct mc_pcie, plda);
> >  	u32 event = data->hwirq;
> >  	void __iomem *addr;
> >  	u32 mask;
> >  	u32 val;
> >
> > -	addr = port->axi_base_addr + event_descs[event].base +
> > +	addr = mc_port->axi_base_addr + event_descs[event].base +
> >  		event_descs[event].mask_offset;
> >  	mask = event_descs[event].mask;
> >
> > @@ -811,7 +819,7 @@ static int mc_pcie_init_clks(struct device *dev)
> >  	return 0;
> >  }
> >
> > -static int mc_pcie_init_irq_domains(struct mc_pcie *port)
> > +static int mc_pcie_init_irq_domains(struct plda_pcie_rp *port)
> >  {
> >  	struct device *dev = port->dev;
> >  	struct device_node *node = dev->of_node; @@ -889,7 +897,7 @@ static
> > void mc_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
> > }
> >
> >  static int mc_pcie_setup_windows(struct platform_device *pdev,
> > -				 struct mc_pcie *port)
> > +				 struct plda_pcie_rp *port)
> >  {
> >  	void __iomem *bridge_base_addr = port->bridge_addr;
> >  	struct pci_host_bridge *bridge = platform_get_drvdata(pdev); @@
> > -970,7 +978,7 @@ static void mc_disable_interrupts(struct mc_pcie *port)
> >  	writel_relaxed(GENMASK(31, 0), bridge_base_addr + ISTATUS_HOST);  }
> >
> > -static int mc_init_interrupts(struct platform_device *pdev, struct
> > mc_pcie *port)
> > +static int mc_init_interrupts(struct platform_device *pdev, struct
> > +plda_pcie_rp *port)
> >  {
> >  	struct device *dev = &pdev->dev;
> >  	int irq;
> > @@ -1043,12 +1051,12 @@ static int mc_platform_init(struct
> pci_config_window *cfg)
> >  	mc_pcie_enable_msi(port, cfg->win);
> >
> >  	/* Configure non-config space outbound ranges */
> > -	ret = mc_pcie_setup_windows(pdev, port);
> > +	ret = mc_pcie_setup_windows(pdev, &port->plda);
> >  	if (ret)
> >  		return ret;
> >
> >  	/* Address translation is up; safe to enable interrupts */
> > -	ret = mc_init_interrupts(pdev, port);
> > +	ret = mc_init_interrupts(pdev, &port->plda);
> >  	if (ret)
> >  		return ret;
> >
> > @@ -1059,6 +1067,7 @@ static int mc_host_probe(struct platform_device
> > *pdev)  {
> >  	struct device *dev = &pdev->dev;
> >  	void __iomem *bridge_base_addr;
> > +	struct plda_pcie_rp *plda;
> >  	int ret;
> >  	u32 val;
> >
> > @@ -1066,7 +1075,8 @@ static int mc_host_probe(struct platform_device
> *pdev)
> >  	if (!port)
> >  		return -ENOMEM;
> >
> > -	port->dev = dev;
> > +	plda = &port->plda;
> > +	plda->dev = dev;
> >
> >  	port->axi_base_addr = devm_platform_ioremap_resource(pdev, 1);
> >  	if (IS_ERR(port->axi_base_addr))
> > @@ -1075,7 +1085,7 @@ static int mc_host_probe(struct platform_device
> *pdev)
> >  	mc_disable_interrupts(port);
> >
> >  	bridge_base_addr = port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
> > -	port->bridge_addr = bridge_base_addr;
> > +	plda->bridge_addr = bridge_base_addr;
> >
> >  	/* Allow enabling MSI by disabling MSI-X */
> >  	val = readl(bridge_base_addr + PCIE_PCI_IRQ_DW0); @@ -1087,10
> > +1097,10 @@ static int mc_host_probe(struct platform_device *pdev)
> >  	val &= NUM_MSI_MSGS_MASK;
> >  	val >>= NUM_MSI_MSGS_SHIFT;
> >
> > -	port->msi.num_vectors = 1 << val;
> > +	plda->msi.num_vectors = 1 << val;
> >
> >  	/* Pick vector address from design */
> > -	port->msi.vector_phy = readl_relaxed(bridge_base_addr + IMSI_ADDR);
> > +	plda->msi.vector_phy = readl_relaxed(bridge_base_addr + IMSI_ADDR);
> >
> >  	ret = mc_pcie_init_clks(dev);
> >  	if (ret) {
> > --
> > 2.17.1
> >
> >
> > _______________________________________________
> > linux-riscv mailing list
> > linux-riscv@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH v15,RESEND 21/23] PCI: starfive: Add JH7110 PCIe controller

[Kevin Xie]

> On Tue, Feb 27, 2024 at 06:35:20PM +0800, Minda Chen wrote:
> > Add StarFive JH7110 SoC PCIe controller platform driver codes, JH7110
> > with PLDA host PCIe core.
> >
> > Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> > Co-developed-by: Kevin Xie <kevin.xie@starfivetech.com>
> > Reviewed-by: Mason Huo <mason.huo@starfivetech.com>
> > ---
> >  MAINTAINERS                                 |   7 +
> >  drivers/pci/controller/plda/Kconfig         |  12 +
> >  drivers/pci/controller/plda/Makefile        |   1 +
> >  drivers/pci/controller/plda/pcie-plda.h     |  71 ++-
> >  drivers/pci/controller/plda/pcie-starfive.c | 473
> > ++++++++++++++++++++
> >  5 files changed, 563 insertions(+), 1 deletion(-)  create mode 100644
> > drivers/pci/controller/plda/pcie-starfive.c
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS index
> > fc9576e69a71..d3b0a0fb754f 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -17078,6 +17078,13 @@ S:	Maintained
> >  F:	Documentation/devicetree/bindings/pci/socionext,uniphier-pcie*
> >  F:	drivers/pci/controller/dwc/pcie-uniphier*
> >
> > +PCIE DRIVER FOR STARFIVE JH71x0
> > +M:	Kevin Xie <kevin.xie@starfivetech.com>
> > +L:	linux-pci@vger.kernel.org
> > +S:	Maintained
> > +F:	Documentation/devicetree/bindings/pci/starfive*
> > +F:	drivers/pci/controller/plda/pcie-starfive.c
> > +
> >  PCIE DRIVER FOR ST SPEAR13XX
> >  M:	Pratyush Anand <pratyush.anand@gmail.com>
> >  L:	linux-pci@vger.kernel.org

> > diff --git a/drivers/pci/controller/plda/pcie-plda.h
> > b/drivers/pci/controller/plda/pcie-plda.h
> > index 7b69891700a4..04e385758a2f 100644
> > --- a/drivers/pci/controller/plda/pcie-plda.h
> > +++ b/drivers/pci/controller/plda/pcie-plda.h
> > @@ -10,10 +10,20 @@
> >  #define PLDA_MAX_NUM_MSI_IRQS			32
> >
> >  /* PCIe Bridge Phy Regs */
> > +#define GEN_SETTINGS				0x80
> > +#define  RP_ENABLE				1
> > +#define PCIE_PCI_IDS_DW1			0x9c
> > +#define  IDS_CLASS_CODE_SHIFT			16
> > +#define  REVISION_ID_MASK			GENMASK(7, 0)
> > +#define  CLASS_CODE_ID_MASK			GENMASK(31, 8)
> >  #define PCIE_PCI_IRQ_DW0			0xa8
> >  #define  MSIX_CAP_MASK				BIT(31)
> >  #define  NUM_MSI_MSGS_MASK			GENMASK(6, 4)
> >  #define  NUM_MSI_MSGS_SHIFT			4
> > +#define PCI_MISC				0xb4
> > +#define  PHY_FUNCTION_DIS			BIT(15)
> > +#define PCIE_WINROM				0xfc
> > +#define  PREF_MEM_WIN_64_SUPPORT		BIT(3)
> >
> >  #define IMASK_LOCAL				0x180
> >  #define  DMA_END_ENGINE_0_MASK			0x00000000u
> > @@ -65,6 +75,8 @@
> >  #define ISTATUS_HOST				0x18c
> >  #define IMSI_ADDR				0x190
> >  #define ISTATUS_MSI				0x194
> > +#define PMSG_SUPPORT_RX				0x3f0
> > +#define  PMSG_LTR_SUPPORT			BIT(2)
> >
> >  /* PCIe Master table init defines */
> >  #define ATR0_PCIE_WIN0_SRCADDR_PARAM		0x600u
> > @@ -86,6 +98,8 @@
> >  #define  PCIE_TX_RX_INTERFACE			0x00000000u
> >  #define  PCIE_CONFIG_INTERFACE			0x00000001u
> >
> > +#define CONFIG_SPACE_ADDR_OFFSET		0x1000u
> > +
> >  #define ATR_ENTRY_SIZE				32
> >
> >  enum plda_int_event {
> > @@ -200,4 +214,59 @@ static inline void plda_set_default_msi(struct
> plda_msi *msi)
> >  	msi->vector_phy = IMSI_ADDR;
> >  	msi->num_vectors = PLDA_MAX_NUM_MSI_IRQS;  } -#endif
> > +
> > +static inline void plda_pcie_enable_root_port(struct plda_pcie_rp
> > +*plda) {
> > +	u32 value;
> > +
> > +	value = readl_relaxed(plda->bridge_addr + GEN_SETTINGS);
> > +	value |= RP_ENABLE;
> > +	writel_relaxed(value, plda->bridge_addr + GEN_SETTINGS); }
> > +
> > +static inline void plda_pcie_set_standard_class(struct plda_pcie_rp
> > +*plda) {
> > +	u32 value;
> > +
> > +	/* set class code and reserve revision id */
> > +	value = readl_relaxed(plda->bridge_addr + PCIE_PCI_IDS_DW1);
> > +	value &= REVISION_ID_MASK;
> > +	value |= (PCI_CLASS_BRIDGE_PCI << IDS_CLASS_CODE_SHIFT);
> > +	writel_relaxed(value, plda->bridge_addr + PCIE_PCI_IDS_DW1); }
> > +
> > +static inline void plda_pcie_set_pref_win_64bit(struct plda_pcie_rp
> > +*plda) {
> > +	u32 value;
> > +
> > +	value = readl_relaxed(plda->bridge_addr + PCIE_WINROM);
> > +	value |= PREF_MEM_WIN_64_SUPPORT;
> > +	writel_relaxed(value, plda->bridge_addr + PCIE_WINROM); }
> 
> What does this do ?
> 

This is an internal register to enable the prefetchable memory window
64-bit addressing support. ( disabled by default)
After this enable configuration, the address translation config in DTS
and ATU about 64-bit prefetchable memory are useful.

I will add the annotation here in the next version.

> > +static inline void plda_pcie_disable_ltr(struct plda_pcie_rp *plda) {
> > +	u32 value;
> > +
> > +	value = readl_relaxed(plda->bridge_addr + PMSG_SUPPORT_RX);
> > +	value &= ~PMSG_LTR_SUPPORT;
> > +	writel_relaxed(value, plda->bridge_addr + PMSG_SUPPORT_RX); }
> > +
> > +static inline void plda_pcie_disable_func(struct plda_pcie_rp *plda)
> > +{
> > +	u32 value;
> > +
> > +	value = readl_relaxed(plda->bridge_addr + PCI_MISC);
> > +	value |= PHY_FUNCTION_DIS;
> > +	writel_relaxed(value, plda->bridge_addr + PCI_MISC); }
> > +
> > +static inline void plda_pcie_write_rc_bar(struct plda_pcie_rp *plda,
> > +u64 val) {
> > +	void __iomem *addr = plda->bridge_addr +
> CONFIG_SPACE_ADDR_OFFSET;
> > +
> > +	writel_relaxed(lower_32_bits(val), addr + PCI_BASE_ADDRESS_0);
> > +	writel_relaxed(upper_32_bits(val), addr + PCI_BASE_ADDRESS_1); }
> > +#endif /* _PCIE_PLDA_H */
> > diff --git a/drivers/pci/controller/plda/pcie-starfive.c
> > b/drivers/pci/controller/plda/pcie-starfive.c
> > new file mode 100644
> > index 000000000000..9bb9f0e29565
> > --- /dev/null
> > +++ b/drivers/pci/controller/plda/pcie-starfive.c
> > @@ -0,0 +1,473 @@
> > +// SPDX-License-Identifier: GPL-2.0+
> > +/*
> > + * PCIe host controller driver for StarFive JH7110 Soc.
> > + *
> > + * Copyright (C) 2023 StarFive Technology Co., Ltd.
> > + */
> > +
> > +#include <linux/bitfield.h>
> > +#include <linux/clk.h>
> > +#include <linux/delay.h>
> > +#include <linux/gpio/consumer.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/kernel.h>
> > +#include <linux/mfd/syscon.h>
> > +#include <linux/module.h>
> > +#include <linux/of_address.h>
> > +#include <linux/of_irq.h>
> > +#include <linux/of_pci.h>
> > +#include <linux/pci.h>
> > +#include <linux/phy/phy.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/regmap.h>
> > +#include <linux/reset.h>
> > +#include "../../pci.h"
> > +
> > +#include "pcie-plda.h"
> > +
> > +#define PCIE_FUNC_NUM			4
> > +
> > +/* system control */
> > +#define STG_SYSCON_PCIE0_BASE			0x48
> > +#define STG_SYSCON_PCIE1_BASE			0x1f8
> > +
> > +#define STG_SYSCON_AR_OFFSET			0x78
> > +#define STG_SYSCON_AXI4_SLVL_AR_MASK		GENMASK(22, 8)
> > +#define STG_SYSCON_AXI4_SLVL_PHY_AR(x)
> 	FIELD_PREP(GENMASK(20, 17), x)
> > +#define STG_SYSCON_AW_OFFSET			0x7c
> > +#define STG_SYSCON_AXI4_SLVL_AW_MASK		GENMASK(14, 0)
> > +#define STG_SYSCON_AXI4_SLVL_PHY_AW(x)
> 	FIELD_PREP(GENMASK(12, 9), x)
> > +#define STG_SYSCON_CLKREQ			BIT(22)
> > +#define STG_SYSCON_CKREF_SRC_MASK		GENMASK(19, 18)
> > +#define STG_SYSCON_RP_NEP_OFFSET		0xe8
> > +#define STG_SYSCON_K_RP_NEP			BIT(8)
> > +#define STG_SYSCON_LNKSTA_OFFSET		0x170
> > +#define DATA_LINK_ACTIVE			BIT(5)
> > +
> > +/* Parameters for the waiting for link up routine */
> > +#define LINK_WAIT_MAX_RETRIES	10
> > +#define LINK_WAIT_USLEEP_MIN	90000
> > +#define LINK_WAIT_USLEEP_MAX	100000
> > +
> > +struct starfive_jh7110_pcie {
> > +	struct plda_pcie_rp plda;
> > +	struct reset_control *resets;
> > +	struct clk_bulk_data *clks;
> > +	struct regmap *reg_syscon;
> > +	struct gpio_desc *power_gpio;
> > +	struct gpio_desc *reset_gpio;
> > +	struct phy *phy;
> > +
> > +	unsigned int stg_pcie_base;
> > +	int num_clks;
> > +};
> > +
> > +/*
> > + * The BAR0/1 of bridge should be hidden during enumeration to
> > + * avoid the sizing and resource allocation by PCIe core.
> 
> Please explain. This is what's preventing this driver to re-use the
> PCI_HOST_COMMON infrastructure and it is unclear why.
> 

Sorry, we didn't explain it clearly enough here.
JH7110 PCIe port BAR0/1 is configured as 64-bit prefetchable memory
space, PCIe read and write requests targeting BAR0/1 are routed to so
called 'Bridge Configuration space' in PLDA datasheet, which contains
the bridge internal registers, such as interrupt, DMA & ATU registers...

JH7110 can access the Bridge Configuration space by local bus, and don`t
want the bridge internal registers accessed by the DMA from EP devices.
Thus, they are unimplemented and should be hidden.

I will modify the annotation in next version if you think it is ok.

> > + */
> > +static bool starfive_pcie_hide_rc_bar(struct pci_bus *bus, unsigned int
> devfn,
> > +				      int offset)
> > +{
> > +	if (pci_is_root_bus(bus) && !devfn &&
> > +	    (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1))
> > +		return true;
> > +
> > +	return false;
> > +}
> > +
> > +static int starfive_pcie_config_write(struct pci_bus *bus, unsigned int devfn,
> > +				      int where, int size, u32 value) {
> > +	if (starfive_pcie_hide_rc_bar(bus, devfn, where))
> > +		return PCIBIOS_SUCCESSFUL;
> > +
> > +	return pci_generic_config_write(bus, devfn, where, size, value); }
> > +
> > +static int starfive_pcie_config_read(struct pci_bus *bus, unsigned int devfn,
> > +				     int where, int size, u32 *value) {
> > +	if (starfive_pcie_hide_rc_bar(bus, devfn, where)) {
> > +		*value = 0;
> > +		return PCIBIOS_SUCCESSFUL;
> > +	}
> > +
> > +	return pci_generic_config_read(bus, devfn, where, size, value); }
> > +
> > +static int starfive_pcie_parse_dt(struct starfive_jh7110_pcie *pcie,
> > +				  struct device *dev)
> > +{
> > +	int domain_nr;
> > +
> > +	pcie->num_clks = devm_clk_bulk_get_all(dev, &pcie->clks);
> > +	if (pcie->num_clks < 0)
> > +		return dev_err_probe(dev, pcie->num_clks,
> > +				     "failed to get pcie clocks\n");
> > +
> > +	pcie->resets = devm_reset_control_array_get_exclusive(dev);
> > +	if (IS_ERR(pcie->resets))
> > +		return dev_err_probe(dev, PTR_ERR(pcie->resets),
> > +				     "failed to get pcie resets");
> > +
> > +	pcie->reg_syscon =
> > +		syscon_regmap_lookup_by_phandle(dev->of_node,
> > +						"starfive,stg-syscon");
> > +
> > +	if (IS_ERR(pcie->reg_syscon))
> > +		return dev_err_probe(dev, PTR_ERR(pcie->reg_syscon),
> > +				     "failed to parse starfive,stg-syscon\n");
> > +
> > +	pcie->phy = devm_phy_optional_get(dev, NULL);
> > +	if (IS_ERR(pcie->phy))
> > +		return dev_err_probe(dev, PTR_ERR(pcie->phy),
> > +				     "failed to get pcie phy\n");
> > +
> > +	domain_nr = of_get_pci_domain_nr(dev->of_node);
> > +
> > +	if (domain_nr < 0 || domain_nr > 1)
> > +		return dev_err_probe(dev, -ENODEV,
> > +				     "failed to get valid pcie domain\n");
> > +
> > +	if (domain_nr == 0)
> > +		pcie->stg_pcie_base = STG_SYSCON_PCIE0_BASE;
> > +	else
> > +		pcie->stg_pcie_base = STG_SYSCON_PCIE1_BASE;
> 
> I don't agree with this. The domain number is a software allocated value
> attached to a controller. It is written nowhere that you can't swap them and
> leave this code path broken.
> 

We set 'linux,pci-domain = <0>' in our pcie0 DTS node, and <1> for 
pcie1. The value should be static now.

As the STG system controller defines different bases in PCIe RP0 & RP1,
we have to identify which controller is doing the hardware initialization.
We discussed this before with other maintainers, do you have a better
suggestion?

> > +	pcie->reset_gpio = devm_gpiod_get_optional(dev, "perst",
> > +						   GPIOD_OUT_HIGH);
> > +	if (IS_ERR(pcie->reset_gpio))
> > +		return dev_err_probe(dev, PTR_ERR(pcie->reset_gpio),
> > +				     "failed to get perst-gpio\n");
> > +
> > +	pcie->power_gpio = devm_gpiod_get_optional(dev, "enable",
> > +						   GPIOD_OUT_LOW);
> > +	if (IS_ERR(pcie->power_gpio))
> > +		return dev_err_probe(dev, PTR_ERR(pcie->power_gpio),
> > +				     "failed to get power-gpio\n");
> > +
> > +	return 0;
> > +}
> > +
> > +static struct pci_ops starfive_pcie_ops = {
> > +	.map_bus	= plda_pcie_map_bus,
> 
> It looks like this is an ECAM driver with the read/config quirks for read/write in
> the method below. Please explain why they are needed.
> 

Yes, that's for the hide RC BAR operations mentioned above.

> > +	.read           = starfive_pcie_config_read,
> > +	.write          = starfive_pcie_config_write,
> > +};
> > +
> > +static int starfive_pcie_clk_rst_init(struct starfive_jh7110_pcie
> > +*pcie) {
> > +	struct device *dev = pcie->plda.dev;
> > +	int ret;
> > +
> > +	/* PCIe PCI Standard Configuration Identification Settings. */
> > +	plda_pcie_set_standard_class(plda);
> > +
> > +	/*
> > +	 * The LTR message forwarding of PCIe Message Reception was set by
> > +core
> 
> "LTR message forwarding of PCIe Message Reception".
> 
> What is this referring to ?
> 

Sorry, this sentence is ambiguous.
"PCIe Message Reception" is a register to enable the function about forwarding
specific PCIe messages to AXI application, such as ATS/LTR/OBFF messages...
Here we clear bit 2 to disable the LTR message forwarding to the AXI application.

I will modify the annotation in next version if you think it is ok.

Thank you for your valuable comments.
Kevin.

> Thanks,
> Lorenzo
> 
> > +	 * as default, but the forward id & addr are also need to be reset.
> > +	 * If we do not disable LTR message forwarding here, or set a legal
> > +	 * forwarding address, the kernel will get stuck after the driver probe.
> > +	 * To workaround, disable the LTR message forwarding support on
> > +	 * PCIe Message Reception.
> > +	 */
> > +	plda_pcie_disable_ltr(plda);
> > +
> > +	/* Prefetchable memory window 64-bit addressing support */
> > +	plda_pcie_set_pref_win_64bit(plda);
> > +
> >
> > _______________________________________________
> > linux-riscv mailing list
> > linux-riscv@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers.

[Palmer Dabbelt]

On Thu, 29 Feb 2024 07:08:43 PST (-0800), lpieralisi@kernel.org wrote:
> On Tue, Feb 27, 2024 at 06:35:21PM +0800, Minda Chen wrote:
>> From: Kevin Xie <kevin.xie@starfivetech.com>
>>
>> As the Starfive JH7110 hardware can't keep two inbound post write in
>> order all the time, such as MSI messages and NVMe completions. If the
>> NVMe completion update later than the MSI, an NVMe IRQ handle will miss.
>
> Please explain what the problem is and what "NVMe completions" means
> given that you are talking about posted writes.
>
> If you have a link to an erratum write-up it would certainly help.

I think we really need to see that errata document.  Our formal memory 
model doesn't account for device interactions so it's possible there's 
just some arch fence we can make stronger in order to get things ordered 
again -- we've had similar problems with some other RISC-V chips, and 
while it ends up being slow at least it's correct.

> This looks completely broken to me, if the controller can't guarantee
> PCIe transactions ordering it is toast, there is not even a point
> considering mainline merging.

I wouldn't be at all surprised if that's the case.  Without some 
concrete ISA mechanisms here we're sort of just stuck hoping the SOC 
vendors do the right thing, which is always terrifying.

I'm not really a PCIe person so this is all a bit vague, but IIRC we had 
a bunch of possible PCIe ordering violations in the SiFive memory system 
back when I was there and we never really got a scheme for making sure 
things were correct.

So I think we really do need to see that errata document to know what's 
possible here.  Folks have been able to come up with clever solutions to 
these problems before, maybe we'll get lucky again.

>> As a workaround, we will wait a while before going to the generic
>> handle here.
>>
>> Verified with NVMe SSD, USB SSD, R8169 NIC.
>> The performance are stable and even higher after this patch.
>
> I assume this is a joke even though it does not make me laugh.

So you're new to RISC-V, then?  It gets way worse than this ;)

> Thanks,
> Lorenzo
>
>>
>> Signed-off-by: Kevin Xie <kevin.xie@starfivetech.com>
>> Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
>> ---
>>  drivers/pci/controller/plda/pcie-plda-host.c | 12 ++++++++++++
>>  drivers/pci/controller/plda/pcie-plda.h      |  1 +
>>  drivers/pci/controller/plda/pcie-starfive.c  |  1 +
>>  3 files changed, 14 insertions(+)
>>
>> diff --git a/drivers/pci/controller/plda/pcie-plda-host.c b/drivers/pci/controller/plda/pcie-plda-host.c
>> index a18923d7cea6..9e077ddf45c0 100644
>> --- a/drivers/pci/controller/plda/pcie-plda-host.c
>> +++ b/drivers/pci/controller/plda/pcie-plda-host.c
>> @@ -13,6 +13,7 @@
>>  #include <linux/msi.h>
>>  #include <linux/pci_regs.h>
>>  #include <linux/pci-ecam.h>
>> +#include <linux/delay.h>
>>
>>  #include "pcie-plda.h"
>>
>> @@ -44,6 +45,17 @@ static void plda_handle_msi(struct irq_desc *desc)
>>  			       bridge_base_addr + ISTATUS_LOCAL);
>>  		status = readl_relaxed(bridge_base_addr + ISTATUS_MSI);
>>  		for_each_set_bit(bit, &status, msi->num_vectors) {
>> +			/*
>> +			 * As the Starfive JH7110 hardware can't keep two
>> +			 * inbound post write in order all the time, such as
>> +			 * MSI messages and NVMe completions.
>> +			 * If the NVMe completion update later than the MSI,
>> +			 * an NVMe IRQ handle will miss.
>> +			 * As a workaround, we will wait a while before
>> +			 * going to the generic handle here.
>> +			 */
>> +			if (port->msi_quirk_delay_us)
>> +				udelay(port->msi_quirk_delay_us);
>>  			ret = generic_handle_domain_irq(msi->dev_domain, bit);
>>  			if (ret)
>>  				dev_err_ratelimited(dev, "bad MSI IRQ %d\n",
>> diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
>> index 04e385758a2f..feccf285dfe8 100644
>> --- a/drivers/pci/controller/plda/pcie-plda.h
>> +++ b/drivers/pci/controller/plda/pcie-plda.h
>> @@ -186,6 +186,7 @@ struct plda_pcie_rp {
>>  	int msi_irq;
>>  	int intx_irq;
>>  	int num_events;
>> +	u16 msi_quirk_delay_us;
>>  };
>>
>>  struct plda_event {
>> diff --git a/drivers/pci/controller/plda/pcie-starfive.c b/drivers/pci/controller/plda/pcie-starfive.c
>> index 9bb9f0e29565..5cfc30572b7f 100644
>> --- a/drivers/pci/controller/plda/pcie-starfive.c
>> +++ b/drivers/pci/controller/plda/pcie-starfive.c
>> @@ -391,6 +391,7 @@ static int starfive_pcie_probe(struct platform_device *pdev)
>>
>>  	plda->host_ops = &sf_host_ops;
>>  	plda->num_events = PLDA_MAX_EVENT_NUM;
>> +	plda->msi_quirk_delay_us = 1;
>>  	/* mask doorbell event */
>>  	plda->events_bitmap = GENMASK(PLDA_INT_EVENT_NUM - 1, 0)
>>  			     & ~BIT(PLDA_AXI_DOORBELL)
>> --
>> 2.17.1
>>
>>
>> _______________________________________________
>> linux-riscv mailing list
>> linux-riscv@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers.

[Lorenzo Pieralisi]

On Mon, Mar 04, 2024 at 10:08:06AM -0800, Palmer Dabbelt wrote:
> On Thu, 29 Feb 2024 07:08:43 PST (-0800), lpieralisi@kernel.org wrote:
> > On Tue, Feb 27, 2024 at 06:35:21PM +0800, Minda Chen wrote:
> > > From: Kevin Xie <kevin.xie@starfivetech.com>
> > > 
> > > As the Starfive JH7110 hardware can't keep two inbound post write in
> > > order all the time, such as MSI messages and NVMe completions. If the
> > > NVMe completion update later than the MSI, an NVMe IRQ handle will miss.
> > 
> > Please explain what the problem is and what "NVMe completions" means
> > given that you are talking about posted writes.
> > 
> > If you have a link to an erratum write-up it would certainly help.
> 
> I think we really need to see that errata document.  Our formal memory model
> doesn't account for device interactions so it's possible there's just some
> arch fence we can make stronger in order to get things ordered again --
> we've had similar problems with some other RISC-V chips, and while it ends
> up being slow at least it's correct.
> 
> > This looks completely broken to me, if the controller can't guarantee
> > PCIe transactions ordering it is toast, there is not even a point
> > considering mainline merging.
> 
> I wouldn't be at all surprised if that's the case.  Without some concrete
> ISA mechanisms here we're sort of just stuck hoping the SOC vendors do the
> right thing, which is always terrifying.
> 
> I'm not really a PCIe person so this is all a bit vague, but IIRC we had a
> bunch of possible PCIe ordering violations in the SiFive memory system back
> when I was there and we never really got a scheme for making sure things
> were correct.
> 
> So I think we really do need to see that errata document to know what's
> possible here.  Folks have been able to come up with clever solutions to
> these problems before, maybe we'll get lucky again.
> 
> > > As a workaround, we will wait a while before going to the generic
> > > handle here.
> > > 
> > > Verified with NVMe SSD, USB SSD, R8169 NIC.
> > > The performance are stable and even higher after this patch.
> > 
> > I assume this is a joke even though it does not make me laugh.
> 
> So you're new to RISC-V, then?  It gets way worse than this ;)

To me this is just a PCI controller driver, arch does not matter.

What annoyed me is that we really can't state that this patch improves
performance, sorry, the patch itself is not acceptable, let's try
not to rub it in :)

Please post an erratum write-up and we shall see what can be done.

Thanks,
Lorenzo

> > Thanks,
> > Lorenzo
> > 
> > > 
> > > Signed-off-by: Kevin Xie <kevin.xie@starfivetech.com>
> > > Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> > > ---
> > >  drivers/pci/controller/plda/pcie-plda-host.c | 12 ++++++++++++
> > >  drivers/pci/controller/plda/pcie-plda.h      |  1 +
> > >  drivers/pci/controller/plda/pcie-starfive.c  |  1 +
> > >  3 files changed, 14 insertions(+)
> > > 
> > > diff --git a/drivers/pci/controller/plda/pcie-plda-host.c b/drivers/pci/controller/plda/pcie-plda-host.c
> > > index a18923d7cea6..9e077ddf45c0 100644
> > > --- a/drivers/pci/controller/plda/pcie-plda-host.c
> > > +++ b/drivers/pci/controller/plda/pcie-plda-host.c
> > > @@ -13,6 +13,7 @@
> > >  #include <linux/msi.h>
> > >  #include <linux/pci_regs.h>
> > >  #include <linux/pci-ecam.h>
> > > +#include <linux/delay.h>
> > > 
> > >  #include "pcie-plda.h"
> > > 
> > > @@ -44,6 +45,17 @@ static void plda_handle_msi(struct irq_desc *desc)
> > >  			       bridge_base_addr + ISTATUS_LOCAL);
> > >  		status = readl_relaxed(bridge_base_addr + ISTATUS_MSI);
> > >  		for_each_set_bit(bit, &status, msi->num_vectors) {
> > > +			/*
> > > +			 * As the Starfive JH7110 hardware can't keep two
> > > +			 * inbound post write in order all the time, such as
> > > +			 * MSI messages and NVMe completions.
> > > +			 * If the NVMe completion update later than the MSI,
> > > +			 * an NVMe IRQ handle will miss.
> > > +			 * As a workaround, we will wait a while before
> > > +			 * going to the generic handle here.
> > > +			 */
> > > +			if (port->msi_quirk_delay_us)
> > > +				udelay(port->msi_quirk_delay_us);
> > >  			ret = generic_handle_domain_irq(msi->dev_domain, bit);
> > >  			if (ret)
> > >  				dev_err_ratelimited(dev, "bad MSI IRQ %d\n",
> > > diff --git a/drivers/pci/controller/plda/pcie-plda.h b/drivers/pci/controller/plda/pcie-plda.h
> > > index 04e385758a2f..feccf285dfe8 100644
> > > --- a/drivers/pci/controller/plda/pcie-plda.h
> > > +++ b/drivers/pci/controller/plda/pcie-plda.h
> > > @@ -186,6 +186,7 @@ struct plda_pcie_rp {
> > >  	int msi_irq;
> > >  	int intx_irq;
> > >  	int num_events;
> > > +	u16 msi_quirk_delay_us;
> > >  };
> > > 
> > >  struct plda_event {
> > > diff --git a/drivers/pci/controller/plda/pcie-starfive.c b/drivers/pci/controller/plda/pcie-starfive.c
> > > index 9bb9f0e29565..5cfc30572b7f 100644
> > > --- a/drivers/pci/controller/plda/pcie-starfive.c
> > > +++ b/drivers/pci/controller/plda/pcie-starfive.c
> > > @@ -391,6 +391,7 @@ static int starfive_pcie_probe(struct platform_device *pdev)
> > > 
> > >  	plda->host_ops = &sf_host_ops;
> > >  	plda->num_events = PLDA_MAX_EVENT_NUM;
> > > +	plda->msi_quirk_delay_us = 1;
> > >  	/* mask doorbell event */
> > >  	plda->events_bitmap = GENMASK(PLDA_INT_EVENT_NUM - 1, 0)
> > >  			     & ~BIT(PLDA_AXI_DOORBELL)
> > > --
> > > 2.17.1
> > > 
> > > 
> > > _______________________________________________
> > > linux-riscv mailing list
> > > linux-riscv@lists.infradead.org
> > > http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers.

[Kevin Xie]

> Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout
> workaround to host drivers.
> 
> On Mon, Mar 04, 2024 at 10:08:06AM -0800, Palmer Dabbelt wrote:
> > On Thu, 29 Feb 2024 07:08:43 PST (-0800), lpieralisi@kernel.org wrote:
> > > On Tue, Feb 27, 2024 at 06:35:21PM +0800, Minda Chen wrote:
> > > > From: Kevin Xie <kevin.xie@starfivetech.com>
> > > >
> > > > As the Starfive JH7110 hardware can't keep two inbound post write
> > > > in order all the time, such as MSI messages and NVMe completions.
> > > > If the NVMe completion update later than the MSI, an NVMe IRQ handle
> will miss.
> > >
> > > Please explain what the problem is and what "NVMe completions" means
> > > given that you are talking about posted writes.

Sorry, we made a casual conclusion here.
Not any two of inbound post requests can`t be kept in order in JH7110 SoC, 
the only one case we found is NVMe completions with MSI interrupts.
To be more precise, they are the pending status in nvme_completion struct and
nvme_irq handler in nvme/host/pci.c.

We have shown the original workaround patch before:
https://lore.kernel.org/lkml/CAJM55Z9HtBSyCq7rDEDFdw644pOWCKJfPqhmi3SD1x6p3g2SLQ@mail.gmail.com/
We put it in our github branch and works fine for a long time.
Looking forward to better advices from someone familiar with NVMe drivers.

> > >
> > > If you have a link to an erratum write-up it would certainly help.
> >

That`s not a certain hardware issue with existing erratum, and we are doing
more investigation for it in these days.

The next version we will skip this workaround patch, and then summit separate
patch (with erratum) for the fix after more debug on the issue.

> > I think we really need to see that errata document.  Our formal memory
> > model doesn't account for device interactions so it's possible there's
> > just some arch fence we can make stronger in order to get things
> > ordered again -- we've had similar problems with some other RISC-V
> > chips, and while it ends up being slow at least it's correct.
> >
> > > This looks completely broken to me, if the controller can't
> > > guarantee PCIe transactions ordering it is toast, there is not even
> > > a point considering mainline merging.
> >
> > I wouldn't be at all surprised if that's the case.  Without some
> > concrete ISA mechanisms here we're sort of just stuck hoping the SOC
> > vendors do the right thing, which is always terrifying.
> >
> > I'm not really a PCIe person so this is all a bit vague, but IIRC we
> > had a bunch of possible PCIe ordering violations in the SiFive memory
> > system back when I was there and we never really got a scheme for
> > making sure things were correct.
> >
> > So I think we really do need to see that errata document to know
> > what's possible here.  Folks have been able to come up with clever
> > solutions to these problems before, maybe we'll get lucky again.
> >
> > > > As a workaround, we will wait a while before going to the generic
> > > > handle here.
> > > >
> > > > Verified with NVMe SSD, USB SSD, R8169 NIC.
> > > > The performance are stable and even higher after this patch.
> > >
> > > I assume this is a joke even though it does not make me laugh.
> >
> > So you're new to RISC-V, then?  It gets way worse than this ;)
> 
> To me this is just a PCI controller driver, arch does not matter.
> 
> What annoyed me is that we really can't state that this patch improves
> performance, sorry, the patch itself is not acceptable, let's try not to rub it in :)
> 

I'm sorry, the description here is confusing too.
The performance is compared with the version that hasn't been fixed.
It is reasonable that we can get stable performance when NVMe SSD works
without any timeouts, but that doesn't means it can improve the performance
for any other platform.

Thank you for your comments, Lorenzo & Palmer.
Kevin.

> Please post an erratum write-up and we shall see what can be done.
> 
> Thanks,
> Lorenzo
> 
> > > Thanks,
> > > Lorenzo
> > >
> > > >
> > > > Signed-off-by: Kevin Xie <kevin.xie@starfivetech.com>
> > > > Signed-off-by: Minda Chen <minda.chen@starfivetech.com>
> > > > ---
> > > >  drivers/pci/controller/plda/pcie-plda-host.c | 12 ++++++++++++
> > > >  drivers/pci/controller/plda/pcie-plda.h      |  1 +
> > > >  drivers/pci/controller/plda/pcie-starfive.c  |  1 +
> > > >  3 files changed, 14 insertions(+)
> > > >
> > > > diff --git a/drivers/pci/controller/plda/pcie-plda-host.c
> > > > b/drivers/pci/controller/plda/pcie-plda-host.c
> > > > index a18923d7cea6..9e077ddf45c0 100644
> > > > --- a/drivers/pci/controller/plda/pcie-plda-host.c
> > > > +++ b/drivers/pci/controller/plda/pcie-plda-host.c
> > > > @@ -13,6 +13,7 @@
> > > >  #include <linux/msi.h>
> > > >  #include <linux/pci_regs.h>
> > > >  #include <linux/pci-ecam.h>
> > > > +#include <linux/delay.h>
> > > >
> > > >  #include "pcie-plda.h"
> > > >
> > > > @@ -44,6 +45,17 @@ static void plda_handle_msi(struct irq_desc *desc)
> > > >  			       bridge_base_addr + ISTATUS_LOCAL);
> > > >  		status = readl_relaxed(bridge_base_addr + ISTATUS_MSI);
> > > >  		for_each_set_bit(bit, &status, msi->num_vectors) {
> > > > +			/*
> > > > +			 * As the Starfive JH7110 hardware can't keep two
> > > > +			 * inbound post write in order all the time, such as
> > > > +			 * MSI messages and NVMe completions.
> > > > +			 * If the NVMe completion update later than the MSI,
> > > > +			 * an NVMe IRQ handle will miss.
> > > > +			 * As a workaround, we will wait a while before
> > > > +			 * going to the generic handle here.
> > > > +			 */
> > > > +			if (port->msi_quirk_delay_us)
> > > > +				udelay(port->msi_quirk_delay_us);
> > > >  			ret = generic_handle_domain_irq(msi->dev_domain, bit);
> > > >  			if (ret)
> > > >  				dev_err_ratelimited(dev, "bad MSI IRQ %d\n", diff --git
> > > > a/drivers/pci/controller/plda/pcie-plda.h
> > > > b/drivers/pci/controller/plda/pcie-plda.h
> > > > index 04e385758a2f..feccf285dfe8 100644
> > > > --- a/drivers/pci/controller/plda/pcie-plda.h
> > > > +++ b/drivers/pci/controller/plda/pcie-plda.h
> > > > @@ -186,6 +186,7 @@ struct plda_pcie_rp {
> > > >  	int msi_irq;
> > > >  	int intx_irq;
> > > >  	int num_events;
> > > > +	u16 msi_quirk_delay_us;
> > > >  };
> > > >
> > > >  struct plda_event {
> > > > diff --git a/drivers/pci/controller/plda/pcie-starfive.c
> > > > b/drivers/pci/controller/plda/pcie-starfive.c
> > > > index 9bb9f0e29565..5cfc30572b7f 100644
> > > > --- a/drivers/pci/controller/plda/pcie-starfive.c
> > > > +++ b/drivers/pci/controller/plda/pcie-starfive.c
> > > > @@ -391,6 +391,7 @@ static int starfive_pcie_probe(struct
> > > > platform_device *pdev)
> > > >
> > > >  	plda->host_ops = &sf_host_ops;
> > > >  	plda->num_events = PLDA_MAX_EVENT_NUM;
> > > > +	plda->msi_quirk_delay_us = 1;
> > > >  	/* mask doorbell event */
> > > >  	plda->events_bitmap = GENMASK(PLDA_INT_EVENT_NUM - 1, 0)
> > > >  			     & ~BIT(PLDA_AXI_DOORBELL)
> > > > --
> > > > 2.17.1
> > > >
> > > >
> > > > _______________________________________________
> > > > linux-riscv mailing list
> > > > linux-riscv@lists.infradead.org
> > > > http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers.

[Keith Busch]

On Thu, Mar 14, 2024 at 02:18:38AM +0000, Kevin Xie wrote:
> > Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout
> > workaround to host drivers.
> > 
> > On Mon, Mar 04, 2024 at 10:08:06AM -0800, Palmer Dabbelt wrote:
> > > On Thu, 29 Feb 2024 07:08:43 PST (-0800), lpieralisi@kernel.org wrote:
> > > > On Tue, Feb 27, 2024 at 06:35:21PM +0800, Minda Chen wrote:
> > > > > From: Kevin Xie <kevin.xie@starfivetech.com>
> > > > >
> > > > > As the Starfive JH7110 hardware can't keep two inbound post write
> > > > > in order all the time, such as MSI messages and NVMe completions.
> > > > > If the NVMe completion update later than the MSI, an NVMe IRQ handle
> > will miss.
> > > >
> > > > Please explain what the problem is and what "NVMe completions" means
> > > > given that you are talking about posted writes.
> 
> Sorry, we made a casual conclusion here.
> Not any two of inbound post requests can`t be kept in order in JH7110 SoC, 
> the only one case we found is NVMe completions with MSI interrupts.
> To be more precise, they are the pending status in nvme_completion struct and
> nvme_irq handler in nvme/host/pci.c.
> 
> We have shown the original workaround patch before:
> https://lore.kernel.org/lkml/CAJM55Z9HtBSyCq7rDEDFdw644pOWCKJfPqhmi3SD1x6p3g2SLQ@mail.gmail.com/
> We put it in our github branch and works fine for a long time.
> Looking forward to better advices from someone familiar with NVMe drivers.

So this platform treats strictly ordered writes the same as if relaxed
ordering was enabled? I am not sure if we could reasonably work around
such behavior. An arbitrary delay is likely too long for most cases, and
too short for the worst case.

I suppose we could quirk a non-posted transaction in the interrupt
handler to force flush pending memory updates, but that will noticeably
harm your nvme performance. Maybe if you constrain such behavior to the
spurious IRQ_NONE condition, then it might be okay? I don't know.

Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers.

[Keith Busch]

On Wed, Mar 13, 2024 at 08:51:29PM -0600, Keith Busch wrote:
> I suppose we could quirk a non-posted transaction in the interrupt
> handler to force flush pending memory updates, but that will noticeably
> harm your nvme performance. Maybe if you constrain such behavior to the
> spurious IRQ_NONE condition, then it might be okay? I don't know.

Hm, that may not be good enough: if nvme completions can be reordered
with their msi's, then I assume data may reorder with their completion.
Your application will inevitably see stale and corrupted data, so it
sounds like you need some kind of barrier per completion. Ouch!

Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers.

[Bo Gan]

On 3/13/24 7:51 PM, Keith Busch wrote:
> On Thu, Mar 14, 2024 at 02:18:38AM +0000, Kevin Xie wrote:
>>> Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout
>>> workaround to host drivers.
>>>
>>> On Mon, Mar 04, 2024 at 10:08:06AM -0800, Palmer Dabbelt wrote:
>>>> On Thu, 29 Feb 2024 07:08:43 PST (-0800), lpieralisi@kernel.org wrote:
>>>>> On Tue, Feb 27, 2024 at 06:35:21PM +0800, Minda Chen wrote:
>>>>>> From: Kevin Xie <kevin.xie@starfivetech.com>
>>>>>>
>>>>>> As the Starfive JH7110 hardware can't keep two inbound post write
>>>>>> in order all the time, such as MSI messages and NVMe completions.
>>>>>> If the NVMe completion update later than the MSI, an NVMe IRQ handle
>>> will miss.
>>>>>
>>>>> Please explain what the problem is and what "NVMe completions" means
>>>>> given that you are talking about posted writes.

Echoing Keith here. Why are you treating NVMe completions + MSI as a special case?
What's special about this combination other than two posted writes? I own JH7110
visionfive 2 boards myself, and if I'm not mistaken, there are two identical PCIe
controllers in JH7110. The first one connects the onboard USB controller of vf2,
which also enables MSI interrupts. How come this exact problem not affecting the
USB controller? The commit message from Minda strongly suggests it does, and also
for R8169 NIC. Thus, why would you suggest the problem is confined to NVMe?

Bo

>>
>> Sorry, we made a casual conclusion here.
>> Not any two of inbound post requests can`t be kept in order in JH7110 SoC,
>> the only one case we found is NVMe completions with MSI interrupts.
>> To be more precise, they are the pending status in nvme_completion struct and
>> nvme_irq handler in nvme/host/pci.c.
>>
>> We have shown the original workaround patch before:
>> https://lore.kernel.org/lkml/CAJM55Z9HtBSyCq7rDEDFdw644pOWCKJfPqhmi3SD1x6p3g2SLQ@mail.gmail.com/
>> We put it in our github branch and works fine for a long time.
>> Looking forward to better advices from someone familiar with NVMe drivers.
> 
> So this platform treats strictly ordered writes the same as if relaxed
> ordering was enabled? I am not sure if we could reasonably work around
> such behavior. An arbitrary delay is likely too long for most cases, and
> too short for the worst case.
> 
> I suppose we could quirk a non-posted transaction in the interrupt
> handler to force flush pending memory updates, but that will noticeably
> harm your nvme performance. Maybe if you constrain such behavior to the
> spurious IRQ_NONE condition, then it might be okay? I don't know.
> 

Also copied Keith's latest reply below, and I also have the same doubt.

> Hm, that may not be good enough: if nvme completions can be reordered
> with their msi's, then I assume data may reorder with their completion.
> Your application will inevitably see stale and corrupted data, so it
> sounds like you need some kind of barrier per completion. Ouch!

Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers.

[Kevin Xie]

> On 3/13/24 7:51 PM, Keith Busch wrote:
> > On Thu, Mar 14, 2024 at 02:18:38AM +0000, Kevin Xie wrote:
> >>> Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout
> >>> workaround to host drivers.
> >>>
> >>> On Mon, Mar 04, 2024 at 10:08:06AM -0800, Palmer Dabbelt wrote:
> >>>> On Thu, 29 Feb 2024 07:08:43 PST (-0800), lpieralisi@kernel.org wrote:
> >>>>> On Tue, Feb 27, 2024 at 06:35:21PM +0800, Minda Chen wrote:
> >>>>>> From: Kevin Xie <kevin.xie@starfivetech.com>
> >>>>>>
> >>>>>> As the Starfive JH7110 hardware can't keep two inbound post write
> >>>>>> in order all the time, such as MSI messages and NVMe completions.
> >>>>>> If the NVMe completion update later than the MSI, an NVMe IRQ
> >>>>>> handle
> >>> will miss.
> >>>>>
> >>>>> Please explain what the problem is and what "NVMe completions"
> >>>>> means given that you are talking about posted writes.
> 
> Echoing Keith here. Why are you treating NVMe completions + MSI as a special
> case?
> What's special about this combination other than two posted writes? I own
> JH7110 visionfive 2 boards myself, and if I'm not mistaken, there are two
> identical PCIe controllers in JH7110. The first one connects the onboard USB
> controller of vf2, which also enables MSI interrupts. How come this exact
> problem not affecting the USB controller? The commit message from Minda
> strongly suggests it does, and also for R8169 NIC. Thus, why would you suggest
> the problem is confined to NVMe?
> 
> Bo
> 

Hi, Bo, 
Yes, we have two PCIe controller in JH7110 SoC, and the USB hub & NIC over
PCIe work fine no matter we apply this patch or not.

Let me explain in detail about the origin of this patch:
As described in the title, we fixed the timeout issue by a workaround in NVMe
driver, that may affects all other platforms. Thus, we wanted to offload the
workaround from NVMe driver to our PCIe controller platform driver.
After finished the offload patch, we wanted to test if it does harm to the other
PCIe devices, so you can see we tested with R8169 NIC in description.

MSI and NVMe completion are two inbound post requests from NVMe device
to JH7110.
We made the mistake of generalizing both of them as " two inbound post write", 
because we had been investigating the issue in the direction of inbound ordering.

Actually, the phenomenon of this problem is:
"JH7110 have a small probability of not getting the updated NVMe completion 
pending status in NVMe MSI handler, and we can get one after 1 to 3us delay."
Thus, that may related to the ordering, cache consistency or other reasons.
This issue is still under investigation.
 
> >>
> >> Sorry, we made a casual conclusion here.
> >> Not any two of inbound post requests can`t be kept in order in JH7110
> >> SoC, the only one case we found is NVMe completions with MSI interrupts.
> >> To be more precise, they are the pending status in nvme_completion
> >> struct and nvme_irq handler in nvme/host/pci.c.
> >>
> >> We have shown the original workaround patch before:
> >>
> https://lore.kernel.org/lkml/CAJM55Z9HtBSyCq7rDEDFdw644pOWCKJfPqhmi3
> S
> >> D1x6p3g2SLQ@mail.gmail.com/ We put it in our github branch and works
> >> fine for a long time.
> >> Looking forward to better advices from someone familiar with NVMe
> drivers.
> >
> > So this platform treats strictly ordered writes the same as if relaxed
> > ordering was enabled? I am not sure if we could reasonably work around
> > such behavior. An arbitrary delay is likely too long for most cases,
> > and too short for the worst case.
> >
> > I suppose we could quirk a non-posted transaction in the interrupt
> > handler to force flush pending memory updates, but that will
> > noticeably harm your nvme performance. Maybe if you constrain such
> > behavior to the spurious IRQ_NONE condition, then it might be okay? I don't
> know.
> >
> 
> Also copied Keith's latest reply below, and I also have the same doubt.
> 

Hi, Keith, sorry for the late reply.
We have tried to add a dummy non-post request( config read ) in the handler,
but it doesn't help.
Besides, we tried to add the mb() before checking the NVMe completion, 
and it doesn't help too.

> > Hm, that may not be good enough: if nvme completions can be reordered
> > with their msi's, then I assume data may reorder with their completion.
> > Your application will inevitably see stale and corrupted data, so it
> > sounds like you need some kind of barrier per completion. Ouch!

If we do not apply the patch, we might get the timeout warnings and waste
some time, the problem seems to be less serious than you described.
After applying the workaround, we can do tasks with NVMe SSD normally,
such as boot up, hibernation and saving data.