[RFC PATCH] riscv: dts: spacemit: k1: give each PCIe port its own DMA window

[RFC PATCH] riscv: dts: spacemit: k1: give each PCIe port its own DMA window

[Vlad Banea]

The dma-ranges of the shared pcie-bus skip bus addresses
0x8000_0000-0xb7ff_ffff, the union of the three ports' outbound memory
windows. CPU addresses 0x1_0000_0000-0x1_37ff_ffff are therefore
unreachable from all three ports.

SpacemiT's vendor device tree gives each port its own map that skips
only that port's window (dram_range2/6/7). Do the same here. On boards
with more than 4 GB of RAM, each port can then reach another 512-640 MB
without SWIOTLB bouncing.

Signed-off-by: Vlad Banea <montrealtango@gmail.com>
---

Notes:
    RFC: the per-port maps come from SpacemiT's Bianbu kernel (linux-6.6,
    k1-bl-v2.2.y, arch/riscv/boot/dts/spacemit/k1-x.dtsi). They imply that a
    port's inbound DMA to another port's outbound window reaches memory. I
    have not verified that on hardware. Yixun, Encrow, can SpacemiT confirm
    that this is how the K1 routes inbound PCIe traffic?
    
    On an OrangePi R2S (8 GB, RTL8125B NICs on pcie1 and pcie2), up to 87
    SWIOTLB slots were in use under load with the shared bus, and none with
    this patch. Receive pages were kept below 4 GB in that test, so only
    transmit buffers could use the new ranges.

 arch/riscv/boot/dts/spacemit/k1.dtsi | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/arch/riscv/boot/dts/spacemit/k1.dtsi b/arch/riscv/boot/dts/spacemit/k1.dtsi
index 3fabfd34114b..949843bb7789 100644
--- a/arch/riscv/boot/dts/spacemit/k1.dtsi
+++ b/arch/riscv/boot/dts/spacemit/k1.dtsi
@@ -1152,13 +1152,14 @@ eth1: ethernet@cac81000 {
 			};
 		};
 
-		pcie-bus {
+		pcie0-bus {
 			compatible = "simple-bus";
 			ranges;
 			#address-cells = <2>;
 			#size-cells = <2>;
+			/* skips pcie0's own window, bus 0x8000_0000-0x8fff_ffff */
 			dma-ranges = <0x0 0x00000000 0x0 0x00000000 0x0 0x80000000>,
-				     <0x0 0xb8000000 0x1 0x38000000 0x3 0x48000000>;
+				     <0x0 0x90000000 0x1 0x10000000 0x3 0x70000000>;
 
 			pcie0: pcie@ca000000 {
 				device_type = "pci";
@@ -1203,6 +1204,17 @@ pcie0_port: pcie@0 {
 					ranges;
 				};
 			};
+		};
+
+		pcie1-bus {
+			compatible = "simple-bus";
+			ranges;
+			#address-cells = <2>;
+			#size-cells = <2>;
+			/* skips pcie1's own window, bus 0x9000_0000-0x9fff_ffff */
+			dma-ranges = <0x0 0x00000000 0x0 0x00000000 0x0 0x80000000>,
+				     <0x0 0x80000000 0x1 0x00000000 0x0 0x10000000>,
+				     <0x0 0xa0000000 0x1 0x20000000 0x3 0x60000000>;
 
 			pcie1: pcie@ca400000 {
 				device_type = "pci";
@@ -1247,6 +1259,17 @@ pcie1_port: pcie@0 {
 					ranges;
 				};
 			};
+		};
+
+		pcie2-bus {
+			compatible = "simple-bus";
+			ranges;
+			#address-cells = <2>;
+			#size-cells = <2>;
+			/* skips pcie2's own window, bus 0xa000_0000-0xb7ff_ffff */
+			dma-ranges = <0x0 0x00000000 0x0 0x00000000 0x0 0x80000000>,
+				     <0x0 0x80000000 0x1 0x00000000 0x0 0x20000000>,
+				     <0x0 0xb8000000 0x1 0x38000000 0x3 0x48000000>;
 
 			pcie2: pcie@ca800000 {
 				device_type = "pci";

base-commit: d05c6236496071916e19012cda7149465f15e66e
-- 
2.53.0

Re: [RFC PATCH] riscv: dts: spacemit: k1: give each PCIe port its own DMA window

[Troy Mitchell]

[-- Attachment #1: Type: text/plain, Size: 2456 bytes --]

On Wed, 11 Nov 2026 20:51:47 -0500, Vlad Banea wrote:
> The dma-ranges of the shared pcie-bus skip bus addresses
> 0x8000_0000-0xb7ff_ffff, the union of the three ports' outbound memory
> windows. CPU addresses 0x1_0000_0000-0x1_37ff_ffff are therefore
> unreachable from all three ports.
>
> SpacemiT's vendor device tree gives each port its own map that skips
> only that port's window (dram_range2/6/7). Do the same here. On boards
> with more than 4 GB of RAM, each port can then reach another 512-640 MB
> without SWIOTLB bouncing.
This already applies above 2 GB: RAM past the first 2 GB starts at
0x1_0000_0000, so 4 GB boards hit this hole too. Please say "more
than 2 GB", give the hardware reason below instead of pointing at
the vendor tree, and drop the RFC tag.

> Signed-off-by: Vlad Banea <montrealtango@gmail.com>
> ---
>
> Notes:
>     RFC: the per-port maps come from SpacemiT's Bianbu kernel (linux-6.6,
>     k1-bl-v2.2.y, arch/riscv/boot/dts/spacemit/k1-x.dtsi). They imply that a
>     port's inbound DMA to another port's outbound window reaches memory. I
>     have not verified that on hardware. Yixun, Encrow, can SpacemiT confirm
>     that this is how the K1 routes inbound PCIe traffic?
Yes, it reaches memory. On K1 a PCIe master only sees DDR: bus 0-2G
is CPU 0-2G and bus 2G-16G is CPU 4G-18G. The 2G-4G I/O region with
the outbound windows is not reachable from PCIe.
Only the port's own window has to be skipped: it falls inside the
root port's memory window, so the root port does not forward it
upstream.

[...]
> diff --git a/arch/riscv/boot/dts/spacemit/k1.dtsi b/arch/riscv/boot/dts/spacemit/k1.dtsi
> index 3fabfd34114b..949843bb7789 100644
> --- a/arch/riscv/boot/dts/spacemit/k1.dtsi
> +++ b/arch/riscv/boot/dts/spacemit/k1.dtsi
> @@ -1152,13 +1152,14 @@ eth1: ethernet@cac81000 {
>  			};
>  		};
>
> -		pcie-bus {
> +		pcie0-bus {
>  			compatible = "simple-bus";
>  			ranges;
>  			#address-cells = <2>;
>  			#size-cells = <2>;
> +			/* skips pcie0's own window, bus 0x8000_0000-0x8fff_ffff */
>  			dma-ranges = <0x0 0x00000000 0x0 0x00000000 0x0 0x80000000>,
> -				     <0x0 0xb8000000 0x1 0x38000000 0x3 0x48000000>;
> +				     <0x0 0x90000000 0x1 0x10000000 0x3 0x70000000>;
The maps for all three ports are correct: each one skips only its
own window and maps the other two windows to DDR, which matches the
address map above.

-- 
Troy Mitchell

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 273 bytes --]