[PATCH v1 0/5] PolarFire SoC Icicle Reference Design PCIe ?support?/fixes

[PATCH v1 0/5] PolarFire SoC Icicle Reference Design PCIe ?support?/fixes

[Conor Dooley]

Hey all,

Here's some patches that add an Erratum to enable non-coherent DMA
support for PolarFire SoC. By nature of being an FPGA, and due to the
PCIe root ports being only 32-bit capable, many bitstreams configure the
FPGA fabric such that peripherals in the fabric become non-coherent. The
PCIe root ports on PolarFire SoC are connected to the core-complex via
the fabric, and therefore can be (and regularly are) made DMA
non-coherent. The Icicle Kit Reference Design has been configuring the
PCIe root port in this manner since late 2022 and in a way unsupported
by mainline since earlier that year. Adding this non-coherent DMA
support makes PCIe functional on those FPGA designs. Daire did almost all
the work to figure out how to support these kinds of designs, and this
series depends on his patches to introduce the required dma-ranges
handling for the root port driver:
https://lore.kernel.org/linux-pci/20240531085333.2501399-1-daire.mcnamara@microchip.com/

The final patch depends on:
https://lore.kernel.org/linux-pci/20240527-slather-backfire-db4605ae7cd7@wendy/

I'm not sure if an Erratum is really the right way to go about doing
this, but I didn't want to make ARCH_MICROCHIP depend on NONPORTABLE.
An alternative would be to add a menu under drivers/soc like Renesas
does and allow it to be toggled as an option there instead.

Thanks,
Conor.

CC: Paul Walmsley <paul.walmsley@sifive.com>
CC: Palmer Dabbelt <palmer@dabbelt.com>
CC: Conor Dooley <conor.dooley@microchip.com>
CC: Daire McNamara <daire.mcnamara@microchip.com>
CC: Rob Herring <robh@kernel.org>
CC: Krzysztof Kozlowski <krzk+dt@kernel.org>
CC: Samuel Holland <samuel.holland@sifive.com>
CC: linux-riscv@lists.infradead.org
CC: devicetree@vger.kernel.org

Conor Dooley (5):
  cache: ccache: allow building for PolarFire
  cache: ccache: add mpfs to nonstandard cache ops list
  RISC-V: Add an MPFS erratum for PCIe
  riscv: dts: microchip: modify memory map & add dma-ranges for pcie on
    icicle
  riscv: dts: microchip: update pcie reg properties

 arch/riscv/Kconfig.errata                     | 19 +++++
 .../dts/microchip/mpfs-icicle-kit-fabric.dtsi | 77 ++++++++++++-------
 .../boot/dts/microchip/mpfs-icicle-kit.dts    | 44 +++++++++--
 .../dts/microchip/mpfs-m100pfs-fabric.dtsi    |  6 +-
 .../dts/microchip/mpfs-polarberry-fabric.dtsi |  6 +-
 drivers/cache/Kconfig                         |  2 +-
 drivers/cache/sifive_ccache.c                 |  2 +
 7 files changed, 116 insertions(+), 40 deletions(-)

-- 
2.43.2

[PATCH v1 1/5] cache: ccache: allow building for PolarFire

[Conor Dooley]

There's a ccache on PolarFire SoC, so don't limit it to SIFIVE/STARFIVE
SoCs only.

Signed-off-by: Conor Dooley <conor.dooley@microchip.com>
---
 drivers/cache/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/cache/Kconfig b/drivers/cache/Kconfig
index 9345ce4976d76..b2d525e540212 100644
--- a/drivers/cache/Kconfig
+++ b/drivers/cache/Kconfig
@@ -10,7 +10,7 @@ config AX45MP_L2_CACHE
 
 config SIFIVE_CCACHE
 	bool "Sifive Composable Cache controller"
-	depends on ARCH_SIFIVE || ARCH_STARFIVE
+	depends on ARCH_MICROCHIP || ARCH_SIFIVE || ARCH_STARFIVE
 	help
 	  Support for the composable cache controller on SiFive platforms.
 
-- 
2.43.2

Re: [PATCH v1 1/5] cache: ccache: allow building for PolarFire

[Emil Renner Berthing]

Conor Dooley wrote:
> There's a ccache on PolarFire SoC, so don't limit it to SIFIVE/STARFIVE
> SoCs only.
>
> Signed-off-by: Conor Dooley <conor.dooley@microchip.com>

Reviewed-by: Emil Renner Berthing <emil.renner.berthing@canonical.com>

> ---
>  drivers/cache/Kconfig | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/cache/Kconfig b/drivers/cache/Kconfig
> index 9345ce4976d76..b2d525e540212 100644
> --- a/drivers/cache/Kconfig
> +++ b/drivers/cache/Kconfig
> @@ -10,7 +10,7 @@ config AX45MP_L2_CACHE
>
>  config SIFIVE_CCACHE
>  	bool "Sifive Composable Cache controller"
> -	depends on ARCH_SIFIVE || ARCH_STARFIVE
> +	depends on ARCH_MICROCHIP || ARCH_SIFIVE || ARCH_STARFIVE
>  	help
>  	  Support for the composable cache controller on SiFive platforms.
>
> --
> 2.43.2
>
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH v1 2/5] cache: ccache: add mpfs to nonstandard cache ops list

[Conor Dooley]

On PolarFire SoC, for performance reasons, we want to use non-coherent
DMA. Add it to the match table with the non-standard non-coherent
cache ops requirement.

Signed-off-by: Conor Dooley <conor.dooley@microchip.com>
---
 drivers/cache/sifive_ccache.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/cache/sifive_ccache.c b/drivers/cache/sifive_ccache.c
index 6874b72ec59d8..277e66a61efdc 100644
--- a/drivers/cache/sifive_ccache.c
+++ b/drivers/cache/sifive_ccache.c
@@ -122,6 +122,8 @@ static const struct of_device_id sifive_ccache_ids[] = {
 	{ .compatible = "sifive,fu740-c000-ccache" },
 	{ .compatible = "starfive,jh7100-ccache",
 	  .data = (void *)(QUIRK_NONSTANDARD_CACHE_OPS | QUIRK_BROKEN_DATA_UNCORR) },
+	{ .compatible = "microchip,mpfs-ccache",
+	  .data = (void *)(QUIRK_NONSTANDARD_CACHE_OPS) },
 	{ .compatible = "sifive,ccache0" },
 	{ /* end of table */ }
 };
-- 
2.43.2

Re: [PATCH v1 2/5] cache: ccache: add mpfs to nonstandard cache ops list

[Emil Renner Berthing]

Conor Dooley wrote:
> On PolarFire SoC, for performance reasons, we want to use non-coherent
> DMA. Add it to the match table with the non-standard non-coherent
> cache ops requirement.
>
> Signed-off-by: Conor Dooley <conor.dooley@microchip.com>

Reviewed-by: Emil Renner Berthing <emil.renner.berthing@canonical.com>

> ---
>  drivers/cache/sifive_ccache.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/drivers/cache/sifive_ccache.c b/drivers/cache/sifive_ccache.c
> index 6874b72ec59d8..277e66a61efdc 100644
> --- a/drivers/cache/sifive_ccache.c
> +++ b/drivers/cache/sifive_ccache.c
> @@ -122,6 +122,8 @@ static const struct of_device_id sifive_ccache_ids[] = {
>  	{ .compatible = "sifive,fu740-c000-ccache" },
>  	{ .compatible = "starfive,jh7100-ccache",
>  	  .data = (void *)(QUIRK_NONSTANDARD_CACHE_OPS | QUIRK_BROKEN_DATA_UNCORR) },
> +	{ .compatible = "microchip,mpfs-ccache",
> +	  .data = (void *)(QUIRK_NONSTANDARD_CACHE_OPS) },
>  	{ .compatible = "sifive,ccache0" },
>  	{ /* end of table */ }
>  };
> --
> 2.43.2
>
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH v1 3/5] RISC-V: Add an MPFS erratum for PCIe

[Conor Dooley]

On PolarFire SoC the PCIe root port is limited to 32-bit addressing
(both when acting as an AXI-Slave and as an AXI-Master) due to how it
is connected to the processor core complex via a Fabric Interface
Controller (FIC), rather than being part of the Microprocessor Subsystem
(MSS) like the other hard peripherals.
The AXI buses between the FIC and the MSS are true 64-bit buses, but the
PCIe root port is (effectively) only attached to the lower 32-bits of
each bus. As PolarFire SoC is an FPGA, several of Microchip's customers
that did not want to suffer the penalty of bounce buffering inserted a
"shim" in the FPGA fabric that would alter the address of the AXI-S
transaction on the return path to the MSS (conceptually moving the 4GiB
range that the root port can address via the AXI-S interface around
64-bit space to better cover where those customers had placed their
DDR). PolarFire SoC has a memory map with various apertures into the
physical DDR, and depending on the MSS' configuration, the FPGA design
can control what physical memory addresses back each aperture.

The main apertures, which are cached memory, lie at 0x10_0000_0000 and
0x8000_0000. There are also non-cached apertures at 0xC000_0000 and
0x14_0000_0000. The "default" configuration is to have each of apertures
overlap.

Some use-cases add a shim to the FPGA fabric that will re-route
transactions from the 0x10_0000_0000 region to avoid bounce buffering
for 64-bit addresses or because the aperture at 0x8000_0000 does not
correspond to usable memory (perhaps an AMP context is using it or the
designer chose not to place memory at the aperture's physical address).
Other user-cases re-route them to the non-cached region at
0x14_0000_0000. If the latter is done,
DMA for the PCIe root ports become non-coherent. Such a scheme is only
possible because the SiFive ccache on PolarFire SoC can perform flush
invalidation of the L1/L2 cache for these types of configuration.

Add an Erratum to support this second type of configuration. Like the JH7100's
non-coherent DMA option, this requires use of DMA_GLOBAL_POOL and is
therefore incompatible with Zicbom.

Signed-off-by: Conor Dooley <conor.dooley@microchip.com>
---
 arch/riscv/Kconfig.errata | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/arch/riscv/Kconfig.errata b/arch/riscv/Kconfig.errata
index 2acc7d876e1fb..0a9e13efbe6b5 100644
--- a/arch/riscv/Kconfig.errata
+++ b/arch/riscv/Kconfig.errata
@@ -21,6 +21,25 @@ config ERRATA_ANDES_CMO
 
 	  If you don't know what to do here, say "Y".
 
+config ERRATA_MICROCHIP
+	bool "Microchip errata"
+	depends on ARCH_MICROCHIP
+	help
+	  All Microchip errata Kconfig options depend on this option. Disabling
+	  this option will disable all Microchip errata.
+
+config ERRATA_POLARFIRE_SOC_DMA_NON_COHERENT
+	bool "Non-coherent DMA support for PolarFire SoC"
+	depends on NONPORTABLE
+	depends on ERRATA_MICROCHIP
+	select DMA_GLOBAL_POOL
+	select RISCV_NONSTANDARD_CACHE_OPS
+	help
+	  Enable support for non-coherent DMA on PolarFire SoC.
+	  This support is not required for any peripherals in the MSS, but may
+	  be required for peripherals in the FPGA fabric and is required for
+	  the PCI root port to operate correctly, due to addressing limitations.
+
 config ERRATA_SIFIVE
 	bool "SiFive errata"
 	depends on RISCV_ALTERNATIVE
-- 
2.43.2

Re: [PATCH v1 3/5] RISC-V: Add an MPFS erratum for PCIe

[Emil Renner Berthing]

Conor Dooley wrote:
> On PolarFire SoC the PCIe root port is limited to 32-bit addressing
> (both when acting as an AXI-Slave and as an AXI-Master) due to how it
> is connected to the processor core complex via a Fabric Interface
> Controller (FIC), rather than being part of the Microprocessor Subsystem
> (MSS) like the other hard peripherals.
> The AXI buses between the FIC and the MSS are true 64-bit buses, but the
> PCIe root port is (effectively) only attached to the lower 32-bits of
> each bus. As PolarFire SoC is an FPGA, several of Microchip's customers
> that did not want to suffer the penalty of bounce buffering inserted a
> "shim" in the FPGA fabric that would alter the address of the AXI-S
> transaction on the return path to the MSS (conceptually moving the 4GiB
> range that the root port can address via the AXI-S interface around
> 64-bit space to better cover where those customers had placed their
> DDR). PolarFire SoC has a memory map with various apertures into the
> physical DDR, and depending on the MSS' configuration, the FPGA design
> can control what physical memory addresses back each aperture.
>
> The main apertures, which are cached memory, lie at 0x10_0000_0000 and
> 0x8000_0000. There are also non-cached apertures at 0xC000_0000 and
> 0x14_0000_0000. The "default" configuration is to have each of apertures
> overlap.
>
> Some use-cases add a shim to the FPGA fabric that will re-route
> transactions from the 0x10_0000_0000 region to avoid bounce buffering
> for 64-bit addresses or because the aperture at 0x8000_0000 does not
> correspond to usable memory (perhaps an AMP context is using it or the
> designer chose not to place memory at the aperture's physical address).
> Other user-cases re-route them to the non-cached region at
> 0x14_0000_0000. If the latter is done,
> DMA for the PCIe root ports become non-coherent. Such a scheme is only
> possible because the SiFive ccache on PolarFire SoC can perform flush
> invalidation of the L1/L2 cache for these types of configuration.
>
> Add an Erratum to support this second type of configuration. Like the JH7100's
> non-coherent DMA option, this requires use of DMA_GLOBAL_POOL and is
> therefore incompatible with Zicbom.
>
> Signed-off-by: Conor Dooley <conor.dooley@microchip.com>

Reviewed-by: Emil Renner Berthing <emil.renner.berthing@canonical.com>

> ---
>  arch/riscv/Kconfig.errata | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
>
> diff --git a/arch/riscv/Kconfig.errata b/arch/riscv/Kconfig.errata
> index 2acc7d876e1fb..0a9e13efbe6b5 100644
> --- a/arch/riscv/Kconfig.errata
> +++ b/arch/riscv/Kconfig.errata
> @@ -21,6 +21,25 @@ config ERRATA_ANDES_CMO
>
>  	  If you don't know what to do here, say "Y".
>
> +config ERRATA_MICROCHIP
> +	bool "Microchip errata"
> +	depends on ARCH_MICROCHIP
> +	help
> +	  All Microchip errata Kconfig options depend on this option. Disabling
> +	  this option will disable all Microchip errata.
> +
> +config ERRATA_POLARFIRE_SOC_DMA_NON_COHERENT
> +	bool "Non-coherent DMA support for PolarFire SoC"
> +	depends on NONPORTABLE
> +	depends on ERRATA_MICROCHIP
> +	select DMA_GLOBAL_POOL
> +	select RISCV_NONSTANDARD_CACHE_OPS
> +	help
> +	  Enable support for non-coherent DMA on PolarFire SoC.
> +	  This support is not required for any peripherals in the MSS, but may
> +	  be required for peripherals in the FPGA fabric and is required for
> +	  the PCI root port to operate correctly, due to addressing limitations.
> +
>  config ERRATA_SIFIVE
>  	bool "SiFive errata"
>  	depends on RISCV_ALTERNATIVE
> --
> 2.43.2
>
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH v1 4/5] riscv: dts: microchip: modify memory map & add dma-ranges for pcie on icicle

[Conor Dooley]

Since 2022/02, the main reference design for the Icicle Kit uses a
non-coherent memory configuration for PCIe, and a shim in the FPGA
fabric, in response to customer requests. As a result, the PCIe root
port has not been usable in mainline since that point in time.

The memory apertures on the reference design have been configured so
that the base of each aperture is mapped to 0x0 in physical memory, a
completely "overlaid" approach. This enables the PCIe root port to
operate in a non-coherent manner (the CPUs/harts use the cached variant
of a particular actual DDR address, the PCIe root port uses the
non-cached variant of the same address. As an example, to access DDR
location 0, the FIC, to which the PCIe rootport is attached, would use
14'0000'0000 and the CPUs would use 10'0000'0000. In this example, the
FIC is responsible for the upper 32-bits of the AXI address and the
root-port (as it is limited to 32-bits on the AXI-S interface) is
responsible for the lower 32-bits of the AXI address. The FPGA designs
utilizing this approach use a simple "shim" to statically set the upper
32-bits of all AXI-S addresses to '000'0014'.

Describe some regions of non-cached memory (and immediately reserve them,
as PolarFire SoC does not support atomics in non-cached memory) so that
we can configure dma-ranges and dma-pools to support a "shim" that will
shift addresses in AXI transactions from the PCIe root port into the
0x14_0000_0000 non-cached region of memory.

The price paid for this is a reduction in the overall system memory.

Signed-off-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../dts/microchip/mpfs-icicle-kit-fabric.dtsi | 75 ++++++++++++-------
 .../boot/dts/microchip/mpfs-icicle-kit.dts    | 44 +++++++++--
 2 files changed, 84 insertions(+), 35 deletions(-)

diff --git a/arch/riscv/boot/dts/microchip/mpfs-icicle-kit-fabric.dtsi b/arch/riscv/boot/dts/microchip/mpfs-icicle-kit-fabric.dtsi
index 1069134f2e12a..33e76db965bbc 100644
--- a/arch/riscv/boot/dts/microchip/mpfs-icicle-kit-fabric.dtsi
+++ b/arch/riscv/boot/dts/microchip/mpfs-icicle-kit-fabric.dtsi
@@ -26,33 +26,54 @@ i2c2: i2c@40000200 {
 		status = "disabled";
 	};
 
-	pcie: pcie@3000000000 {
-		compatible = "microchip,pcie-host-1.0";
-		#address-cells = <0x3>;
-		#interrupt-cells = <0x1>;
-		#size-cells = <0x2>;
-		device_type = "pci";
-		reg = <0x30 0x0 0x0 0x8000000>, <0x0 0x43000000 0x0 0x10000>;
-		reg-names = "cfg", "apb";
-		bus-range = <0x0 0x7f>;
-		interrupt-parent = <&plic>;
-		interrupts = <119>;
-		interrupt-map = <0 0 0 1 &pcie_intc 0>,
-				<0 0 0 2 &pcie_intc 1>,
-				<0 0 0 3 &pcie_intc 2>,
-				<0 0 0 4 &pcie_intc 3>;
-		interrupt-map-mask = <0 0 0 7>;
-		clocks = <&ccc_nw CLK_CCC_PLL0_OUT1>, <&ccc_nw CLK_CCC_PLL0_OUT3>;
-		clock-names = "fic1", "fic3";
-		ranges = <0x3000000 0x0 0x8000000 0x30 0x8000000 0x0 0x80000000>;
-		dma-ranges = <0x02000000 0x0 0x00000000 0x0 0x00000000 0x1 0x00000000>;
-		msi-parent = <&pcie>;
-		msi-controller;
-		status = "disabled";
-		pcie_intc: interrupt-controller {
-			#address-cells = <0>;
-			#interrupt-cells = <1>;
-			interrupt-controller;
+	fabric-pcie-bus@3000000000 {
+		compatible = "simple-bus";
+		#address-cells = <2>;
+		#size-cells = <2>;
+		ranges = <0x0 0x40000000 0x0 0x40000000 0x0 0x20000000>,
+			 <0x30 0x0 0x30 0x0 0x10 0x0>;
+		dma-ranges = <0x0 0x0 0x0 0x80000000 0x0 0x6000000>,
+			     <0x0 0x6000000 0x0 0xc6000000 0x0 0x4000000>,
+			     <0x0 0xa000000 0x0 0x8a000000 0x0 0x8000000>,
+			     <0x0 0x12000000 0x14 0x12000000 0x0 0x10000000>,
+			     <0x0 0x22000000 0x10 0x22000000 0x0 0x5e000000>;
+
+		pcie: pcie@3000000000 {
+			compatible = "microchip,pcie-host-1.0";
+			#address-cells = <0x3>;
+			#interrupt-cells = <0x1>;
+			#size-cells = <0x2>;
+			device_type = "pci";
+			dma-noncoherent;
+			reg = <0x30 0x0 0x0 0x8000000>, <0x0 0x43000000 0x0 0x10000>;
+			reg-names = "cfg", "apb";
+			bus-range = <0x0 0x7f>;
+			interrupt-parent = <&plic>;
+			interrupts = <119>;
+			interrupt-map = <0 0 0 1 &pcie_intc 0>,
+					<0 0 0 2 &pcie_intc 1>,
+					<0 0 0 3 &pcie_intc 2>,
+					<0 0 0 4 &pcie_intc 3>;
+			interrupt-map-mask = <0 0 0 7>;
+			clocks = <&ccc_nw CLK_CCC_PLL0_OUT1>, <&ccc_nw CLK_CCC_PLL0_OUT3>;
+			clock-names = "fic1", "fic3";
+			ranges = <0x43000000 0x0 0x9000000 0x30 0x9000000 0x0 0xf000000>,
+				 <0x1000000 0x0 0x8000000 0x30 0x8000000 0x0 0x1000000>,
+				 <0x3000000 0x0 0x18000000 0x30 0x18000000 0x0 0x70000000>;
+			dma-ranges = <0x3000000 0x0 0x80000000 0x0 0x0 0x0 0x6000000>,
+				     <0x3000000 0x0 0x86000000 0x0 0x6000000 0x0 0x4000000>,
+				     <0x3000000 0x0 0x8a000000 0x0 0xa000000 0x0 0x8000000>,
+				     <0x3000000 0x0 0x92000000 0x0 0x12000000 0x0 0x10000000>,
+				     <0x3000000 0x0 0xa2000000 0x0 0x22000000 0x0 0x5e000000>;
+			msi-parent = <&pcie>;
+			msi-controller;
+			status = "disabled";
+
+			pcie_intc: interrupt-controller {
+				#address-cells = <0>;
+				#interrupt-cells = <1>;
+				interrupt-controller;
+			};
 		};
 	};
 
diff --git a/arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts b/arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts
index f80df225f72b4..6c9340992e9eb 100644
--- a/arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts
+++ b/arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dts
@@ -54,16 +54,29 @@ led-4 {
 		};
 	};
 
-	ddrc_cache_lo: memory@80000000 {
+	memory@80000000 {
 		device_type = "memory";
-		reg = <0x0 0x80000000 0x0 0x40000000>;
-		status = "okay";
+		reg = <0x0 0x80000000 0x0 0x6000000>;
 	};
 
-	ddrc_cache_hi: memory@1040000000 {
+	memory@8a000000 {
 		device_type = "memory";
-		reg = <0x10 0x40000000 0x0 0x40000000>;
-		status = "okay";
+		reg = <0x0 0x8a000000 0x0 0x8000000>;
+	};
+
+	memory@c4000000 {
+		device_type = "memory";
+		reg = <0x0 0xc6000000 0x0 0x4000000>;
+	};
+
+	memory@1022000000 {
+		device_type = "memory";
+		reg = <0x10 0x22000000 0x0 0x5e000000>;
+	};
+
+	memory@1412000000 {
+		device_type = "memory";
+		reg = <0x14 0x12000000 0x0 0x10000000>;
 	};
 
 	reserved-memory {
@@ -71,10 +84,25 @@ reserved-memory {
 		#size-cells = <2>;
 		ranges;
 
-		hss_payload: region@BFC00000 {
-			reg = <0x0 0xBFC00000 0x0 0x400000>;
+		hss_payload: region@103fc00000 {
+			reg = <0x10 0x3fc00000 0x0 0x400000>;
 			no-map;
 		};
+
+		non-cached-low-buffer {
+			compatible = "shared-dma-pool";
+			size = <0x0 0x4000000>;
+			no-map;
+			alloc-ranges = <0x0 0xc6000000 0x0 0x4000000>;
+		};
+
+		non-cached-high-buffer {
+			compatible = "shared-dma-pool";
+			size = <0x0 0x10000000>;
+			no-map;
+			linux,dma-default;
+			alloc-ranges = <0x14 0x12000000 0x0 0x10000000>;
+		};
 	};
 };
 
-- 
2.43.2

[PATCH v1 5/5] riscv: dts: microchip: update pcie reg properties

[Conor Dooley]

Split the "apb" regions of memory on PolarFire SoC devicetrees PCIe
nodes into two regions, so that it will be possible to distinguish
between which root port instance is in use. Currently the "apb" region
points to the base of the root port region and the Linux driver uses
hard-coded offsets to find the "control" and "bridge" regions. The new
method for describing these regions explicitly passes the base address
for the two regions of interest.

Signed-off-by: Conor Dooley <conor.dooley@microchip.com>
---
 arch/riscv/boot/dts/microchip/mpfs-icicle-kit-fabric.dtsi | 6 ++++--
 arch/riscv/boot/dts/microchip/mpfs-m100pfs-fabric.dtsi    | 6 ++++--
 arch/riscv/boot/dts/microchip/mpfs-polarberry-fabric.dtsi | 6 ++++--
 3 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/arch/riscv/boot/dts/microchip/mpfs-icicle-kit-fabric.dtsi b/arch/riscv/boot/dts/microchip/mpfs-icicle-kit-fabric.dtsi
index 33e76db965bbc..f151aa2606d7b 100644
--- a/arch/riscv/boot/dts/microchip/mpfs-icicle-kit-fabric.dtsi
+++ b/arch/riscv/boot/dts/microchip/mpfs-icicle-kit-fabric.dtsi
@@ -45,8 +45,10 @@ pcie: pcie@3000000000 {
 			#size-cells = <0x2>;
 			device_type = "pci";
 			dma-noncoherent;
-			reg = <0x30 0x0 0x0 0x8000000>, <0x0 0x43000000 0x0 0x10000>;
-			reg-names = "cfg", "apb";
+			reg = <0x30 0x0 0x0 0x8000000>,
+			      <0x0 0x43008000 0x0 0x00002000>,
+			      <0x0 0x4300a000 0x0 0x00002000>;
+			reg-names = "cfg", "bridge", "ctrl";
 			bus-range = <0x0 0x7f>;
 			interrupt-parent = <&plic>;
 			interrupts = <119>;
diff --git a/arch/riscv/boot/dts/microchip/mpfs-m100pfs-fabric.dtsi b/arch/riscv/boot/dts/microchip/mpfs-m100pfs-fabric.dtsi
index 8230f06ddf48a..f5036126f2654 100644
--- a/arch/riscv/boot/dts/microchip/mpfs-m100pfs-fabric.dtsi
+++ b/arch/riscv/boot/dts/microchip/mpfs-m100pfs-fabric.dtsi
@@ -20,8 +20,10 @@ pcie: pcie@2000000000 {
 		#interrupt-cells = <0x1>;
 		#size-cells = <0x2>;
 		device_type = "pci";
-		reg = <0x20 0x0 0x0 0x8000000>, <0x0 0x43000000 0x0 0x10000>;
-		reg-names = "cfg", "apb";
+		reg = <0x30 0x0 0x0 0x8000000>,
+		      <0x0 0x43008000 0x0 0x00002000>,
+		      <0x0 0x4300a000 0x0 0x00002000>;
+		reg-names = "cfg", "bridge", "ctrl";
 		bus-range = <0x0 0x7f>;
 		interrupt-parent = <&plic>;
 		interrupts = <119>;
diff --git a/arch/riscv/boot/dts/microchip/mpfs-polarberry-fabric.dtsi b/arch/riscv/boot/dts/microchip/mpfs-polarberry-fabric.dtsi
index 9a56de7b91d64..121b13f9c8bf4 100644
--- a/arch/riscv/boot/dts/microchip/mpfs-polarberry-fabric.dtsi
+++ b/arch/riscv/boot/dts/microchip/mpfs-polarberry-fabric.dtsi
@@ -20,8 +20,10 @@ pcie: pcie@2000000000 {
 		#interrupt-cells = <0x1>;
 		#size-cells = <0x2>;
 		device_type = "pci";
-		reg = <0x20 0x0 0x0 0x8000000>, <0x0 0x43000000 0x0 0x10000>;
-		reg-names = "cfg", "apb";
+		reg = <0x30 0x0 0x0 0x8000000>,
+		      <0x0 0x43008000 0x0 0x00002000>,
+		      <0x0 0x4300a000 0x0 0x00002000>;
+		reg-names = "cfg", "bridge", "ctrl";
 		bus-range = <0x0 0x7f>;
 		interrupt-parent = <&plic>;
 		interrupts = <119>;
-- 
2.43.2

Re: [PATCH v1 0/5] PolarFire SoC Icicle Reference Design PCIe ?support?/fixes

[Conor Dooley]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

On Mon, Jun 10, 2024 at 12:09:12PM +0100, Conor Dooley wrote:
> Hey all,
> 
> Here's some patches that add an Erratum to enable non-coherent DMA
> support for PolarFire SoC...

My local mail client shows that I sent two copies of this series, but I
only see one on lore. I accidentally ran the wrong command, and didn't
set an stmp server in `git send-email`, so I think that only one copy
actually went out and my mail client is confused.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v1 0/5] PolarFire SoC Icicle Reference Design PCIe ?support?/fixes

[Rob Herring (Arm)]

On Mon, 10 Jun 2024 12:09:12 +0100, Conor Dooley wrote:
> Hey all,
> 
> Here's some patches that add an Erratum to enable non-coherent DMA
> support for PolarFire SoC. By nature of being an FPGA, and due to the
> PCIe root ports being only 32-bit capable, many bitstreams configure the
> FPGA fabric such that peripherals in the fabric become non-coherent. The
> PCIe root ports on PolarFire SoC are connected to the core-complex via
> the fabric, and therefore can be (and regularly are) made DMA
> non-coherent. The Icicle Kit Reference Design has been configuring the
> PCIe root port in this manner since late 2022 and in a way unsupported
> by mainline since earlier that year. Adding this non-coherent DMA
> support makes PCIe functional on those FPGA designs. Daire did almost all
> the work to figure out how to support these kinds of designs, and this
> series depends on his patches to introduce the required dma-ranges
> handling for the root port driver:
> https://lore.kernel.org/linux-pci/20240531085333.2501399-1-daire.mcnamara@microchip.com/
> 
> The final patch depends on:
> https://lore.kernel.org/linux-pci/20240527-slather-backfire-db4605ae7cd7@wendy/
> 
> I'm not sure if an Erratum is really the right way to go about doing
> this, but I didn't want to make ARCH_MICROCHIP depend on NONPORTABLE.
> An alternative would be to add a menu under drivers/soc like Renesas
> does and allow it to be toggled as an option there instead.
> 
> Thanks,
> Conor.
> 
> CC: Paul Walmsley <paul.walmsley@sifive.com>
> CC: Palmer Dabbelt <palmer@dabbelt.com>
> CC: Conor Dooley <conor.dooley@microchip.com>
> CC: Daire McNamara <daire.mcnamara@microchip.com>
> CC: Rob Herring <robh@kernel.org>
> CC: Krzysztof Kozlowski <krzk+dt@kernel.org>
> CC: Samuel Holland <samuel.holland@sifive.com>
> CC: linux-riscv@lists.infradead.org
> CC: devicetree@vger.kernel.org
> 
> Conor Dooley (5):
>   cache: ccache: allow building for PolarFire
>   cache: ccache: add mpfs to nonstandard cache ops list
>   RISC-V: Add an MPFS erratum for PCIe
>   riscv: dts: microchip: modify memory map & add dma-ranges for pcie on
>     icicle
>   riscv: dts: microchip: update pcie reg properties
> 
>  arch/riscv/Kconfig.errata                     | 19 +++++
>  .../dts/microchip/mpfs-icicle-kit-fabric.dtsi | 77 ++++++++++++-------
>  .../boot/dts/microchip/mpfs-icicle-kit.dts    | 44 +++++++++--
>  .../dts/microchip/mpfs-m100pfs-fabric.dtsi    |  6 +-
>  .../dts/microchip/mpfs-polarberry-fabric.dtsi |  6 +-
>  drivers/cache/Kconfig                         |  2 +-
>  drivers/cache/sifive_ccache.c                 |  2 +
>  7 files changed, 116 insertions(+), 40 deletions(-)
> 
> --
> 2.43.2
> 
> 
> 


My bot found new DTB warnings on the .dts files added or changed in this
series.

Some warnings may be from an existing SoC .dtsi. Or perhaps the warnings
are fixed by another series. Ultimately, it is up to the platform
maintainer whether these warnings are acceptable or not. No need to reply
unless the platform maintainer has comments.

If you already ran DT checks and didn't see these error(s), then
make sure dt-schema is up to date:

  pip3 install dtschema --upgrade


New warnings running 'make CHECK_DTBS=y microchip/mpfs-icicle-kit.dtb' for 20240610-vertical-frugally-a92a55427dd9@wendy:

arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dtb: pcie@3000000000: reg: [[48, 0, 0, 134217728], [0, 1124106240, 0, 8192], [0, 1124114432, 0, 8192]] is too long
	from schema $id: http://devicetree.org/schemas/pci/microchip,pcie-host.yaml#
arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dtb: pcie@3000000000: reg-names:1: 'apb' was expected
	from schema $id: http://devicetree.org/schemas/pci/microchip,pcie-host.yaml#
arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dtb: pcie@3000000000: reg-names: ['cfg', 'bridge', 'ctrl'] is too long
	from schema $id: http://devicetree.org/schemas/pci/microchip,pcie-host.yaml#
arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dtb: pcie@3000000000: Unevaluated properties are not allowed ('#address-cells', '#interrupt-cells', '#size-cells', 'bus-range', 'device_type', 'dma-noncoherent', 'interrupt-controller', 'interrupt-map', 'interrupt-map-mask', 'interrupt-parent', 'interrupts', 'msi-parent', 'reg', 'reg-names' were unexpected)
	from schema $id: http://devicetree.org/schemas/pci/microchip,pcie-host.yaml#

Re: [PATCH v1 0/5] PolarFire SoC Icicle Reference Design PCIe ?support?/fixes

[Conor Dooley]

[-- Attachment #1: Type: text/plain, Size: 1518 bytes --]

On Mon, Jun 10, 2024 at 01:55:14PM -0600, Rob Herring (Arm) wrote:
> On Mon, 10 Jun 2024 12:09:12 +0100, Conor Dooley wrote:

> > The final patch depends on:
> > https://lore.kernel.org/linux-pci/20240527-slather-backfire-db4605ae7cd7@wendy/


> 
> New warnings running 'make CHECK_DTBS=y microchip/mpfs-icicle-kit.dtb' for 20240610-vertical-frugally-a92a55427dd9@wendy:
> 
> arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dtb: pcie@3000000000: reg: [[48, 0, 0, 134217728], [0, 1124106240, 0, 8192], [0, 1124114432, 0, 8192]] is too long
> 	from schema $id: http://devicetree.org/schemas/pci/microchip,pcie-host.yaml#
> arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dtb: pcie@3000000000: reg-names:1: 'apb' was expected
> 	from schema $id: http://devicetree.org/schemas/pci/microchip,pcie-host.yaml#
> arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dtb: pcie@3000000000: reg-names: ['cfg', 'bridge', 'ctrl'] is too long
> 	from schema $id: http://devicetree.org/schemas/pci/microchip,pcie-host.yaml#
> arch/riscv/boot/dts/microchip/mpfs-icicle-kit.dtb: pcie@3000000000: Unevaluated properties are not allowed ('#address-cells', '#interrupt-cells', '#size-cells', 'bus-range', 'device_type', 'dma-noncoherent', 'interrupt-controller', 'interrupt-map', 'interrupt-map-mask', 'interrupt-parent', 'interrupts', 'msi-parent', 'reg', 'reg-names' were unexpected)
> 	from schema $id: http://devicetree.org/schemas/pci/microchip,pcie-host.yaml#

Yah, not a problem. Patches for that are on the pci list atm.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v1 0/5] PolarFire SoC Icicle Reference Design PCIe ?support?/fixes

[Palmer Dabbelt]

On Mon, 10 Jun 2024 04:09:12 PDT (-0700), Conor Dooley wrote:
> Hey all,
>
> Here's some patches that add an Erratum to enable non-coherent DMA
> support for PolarFire SoC. By nature of being an FPGA, and due to the
> PCIe root ports being only 32-bit capable, many bitstreams configure the
> FPGA fabric such that peripherals in the fabric become non-coherent. The
> PCIe root ports on PolarFire SoC are connected to the core-complex via
> the fabric, and therefore can be (and regularly are) made DMA
> non-coherent. The Icicle Kit Reference Design has been configuring the
> PCIe root port in this manner since late 2022 and in a way unsupported
> by mainline since earlier that year. Adding this non-coherent DMA
> support makes PCIe functional on those FPGA designs. Daire did almost all
> the work to figure out how to support these kinds of designs, and this
> series depends on his patches to introduce the required dma-ranges
> handling for the root port driver:
> https://lore.kernel.org/linux-pci/20240531085333.2501399-1-daire.mcnamara@microchip.com/
>
> The final patch depends on:
> https://lore.kernel.org/linux-pci/20240527-slather-backfire-db4605ae7cd7@wendy/
>
> I'm not sure if an Erratum is really the right way to go about doing
> this, but I didn't want to make ARCH_MICROCHIP depend on NONPORTABLE.
> An alternative would be to add a menu under drivers/soc like Renesas
> does and allow it to be toggled as an option there instead.

Acked-by: Palmer Dabbelt <palmer@rivosinc.com>

if you want to take this via some other tree.

I don't see any issue calling it an erratum, if it lets us keep building 
generic kernels for these systems that seems like a good enough agrument 
to me.

>
> Thanks,
> Conor.
>
> CC: Paul Walmsley <paul.walmsley@sifive.com>
> CC: Palmer Dabbelt <palmer@dabbelt.com>
> CC: Conor Dooley <conor.dooley@microchip.com>
> CC: Daire McNamara <daire.mcnamara@microchip.com>
> CC: Rob Herring <robh@kernel.org>
> CC: Krzysztof Kozlowski <krzk+dt@kernel.org>
> CC: Samuel Holland <samuel.holland@sifive.com>
> CC: linux-riscv@lists.infradead.org
> CC: devicetree@vger.kernel.org
>
> Conor Dooley (5):
>   cache: ccache: allow building for PolarFire
>   cache: ccache: add mpfs to nonstandard cache ops list
>   RISC-V: Add an MPFS erratum for PCIe
>   riscv: dts: microchip: modify memory map & add dma-ranges for pcie on
>     icicle
>   riscv: dts: microchip: update pcie reg properties
>
>  arch/riscv/Kconfig.errata                     | 19 +++++
>  .../dts/microchip/mpfs-icicle-kit-fabric.dtsi | 77 ++++++++++++-------
>  .../boot/dts/microchip/mpfs-icicle-kit.dts    | 44 +++++++++--
>  .../dts/microchip/mpfs-m100pfs-fabric.dtsi    |  6 +-
>  .../dts/microchip/mpfs-polarberry-fabric.dtsi |  6 +-
>  drivers/cache/Kconfig                         |  2 +-
>  drivers/cache/sifive_ccache.c                 |  2 +
>  7 files changed, 116 insertions(+), 40 deletions(-)

Re: [PATCH v1 0/5] PolarFire SoC Icicle Reference Design PCIe ?support?/fixes

[Conor Dooley]

[-- Attachment #1: Type: text/plain, Size: 2069 bytes --]

On Wed, Jul 03, 2024 at 01:09:43PM -0700, Palmer Dabbelt wrote:
> On Mon, 10 Jun 2024 04:09:12 PDT (-0700), Conor Dooley wrote:
> > Hey all,
> > 
> > Here's some patches that add an Erratum to enable non-coherent DMA
> > support for PolarFire SoC. By nature of being an FPGA, and due to the
> > PCIe root ports being only 32-bit capable, many bitstreams configure the
> > FPGA fabric such that peripherals in the fabric become non-coherent. The
> > PCIe root ports on PolarFire SoC are connected to the core-complex via
> > the fabric, and therefore can be (and regularly are) made DMA
> > non-coherent. The Icicle Kit Reference Design has been configuring the
> > PCIe root port in this manner since late 2022 and in a way unsupported
> > by mainline since earlier that year. Adding this non-coherent DMA
> > support makes PCIe functional on those FPGA designs. Daire did almost all
> > the work to figure out how to support these kinds of designs, and this
> > series depends on his patches to introduce the required dma-ranges
> > handling for the root port driver:
> > https://lore.kernel.org/linux-pci/20240531085333.2501399-1-daire.mcnamara@microchip.com/
> > 
> > The final patch depends on:
> > https://lore.kernel.org/linux-pci/20240527-slather-backfire-db4605ae7cd7@wendy/
> > 
> > I'm not sure if an Erratum is really the right way to go about doing
> > this, but I didn't want to make ARCH_MICROCHIP depend on NONPORTABLE.
> > An alternative would be to add a menu under drivers/soc like Renesas
> > does and allow it to be toggled as an option there instead.
> 
> Acked-by: Palmer Dabbelt <palmer@rivosinc.com>
> 
> if you want to take this via some other tree.

Yeah, I was expecting to take it myself but hadn't gone fishing for an
ack because the PCI parts haven't landed yet & the copy in my k.org tree
had a KISMET issue reported against it.

> I don't see any issue calling it an erratum, if it lets us keep building
> generic kernels for these systems that seems like a good enough agrument to
> me.

Cool :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]