[PATCH] arm64: dts: rockchip: enable the mmu600_pcie IOMMU on the rk3588 SoC

[PATCH] arm64: dts: rockchip: enable the mmu600_pcie IOMMU on the rk3588 SoC

[Niklas Cassel]

Commit cd81d3a0695c ("arm64: dts: rockchip: add rk3588 pcie and php
IOMMUs") added the rk3588 SoC's pcie IOMMU and php IOMMU as disabled.

The mmu600_pcie is connected with the five PCIe controllers.
See 8.2 Block Diagram, in rk3588 TRM (Technical Reference Manual).

The five PCIe controllers are:
pcie3x4, pcie3x2, pcie2x1l0, pcie2x1l1, pcie2x1l2.

pcie3x4 can run in either Root Complex mode or Endpoint mode, the other
four PCIe controllers can only run in Root Complex mode. To describe this
we thus have six different device nodes in the device tree.

A PCIe controller in Root Complex mode needs to specify an iommu-map, such
that the device knows how to convert a Requester ID (PCI BDF) to an IOMMU
master ID (stream ID). (A PCIe controller in Endpoint mode should use the
iommus property, just like a regular device.)

If you look at the device tree bindings for msi-map and iommu-map, you can
see that the conversion from Requester ID to MSI-specifier data is the same
as the conversion from Requester ID to IOMMU specifier data. Thus it is
sensible to define the iommu-map property value similar to the msi-map,
such that the conversion will be identical.

Add the proper iommu device tree properties for these six device nodes
connected to the mmu600_pcie, so that we can enable the mmu600_pcie IOMMU.
(The mmu600_php IOMMU is not touched, so it is still disabled.)

Signed-off-by: Niklas Cassel <cassel@kernel.org>
---
Testing done:
-pcie2x1l2: Ethernet connected to the pcie2x1l2 works as expected.
-pcie3x4: A PCIe endpoint (running the PCI endpoint framework) connected to
 pcie3x4 can run pcitest.sh without any issues.
 Modifying the PCI endpoint to DMA to an invalid address gives IOMMU errors
 (as expected).
-pcie3x4_ep: A PCIe root complex (running Linux) connected to pcie3x4_ep can
 run pcitest.sh without any issues.
 Modifying the PCI endpoint's inbound address translation to point to a
 buffer that has not been mapped using the DMA API, results in IOMMU errors
 when the RC writes to the PCI BARs exposed by the endpoint (as expected).
-My board does not expose a convenient method to test the other PCIe
 controllers, but considering that the ones that I did test worked fine,
 I do not see any reason why the others should not do the same.

 arch/arm64/boot/dts/rockchip/rk3588-base.dtsi  | 3 ++-
 arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi | 4 ++++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/boot/dts/rockchip/rk3588-base.dtsi b/arch/arm64/boot/dts/rockchip/rk3588-base.dtsi
index d97d84b888375..f96e607db2857 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588-base.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588-base.dtsi
@@ -551,7 +551,6 @@ mmu600_pcie: iommu@fc900000 {
 			     <GIC_SPI 367 IRQ_TYPE_LEVEL_HIGH 0>;
 		interrupt-names = "eventq", "gerror", "priq", "cmdq-sync";
 		#iommu-cells = <1>;
-		status = "disabled";
 	};
 
 	mmu600_php: iommu@fcb00000 {
@@ -1641,6 +1640,7 @@ pcie2x1l1: pcie@fe180000 {
 		linux,pci-domain = <3>;
 		max-link-speed = <2>;
 		msi-map = <0x3000 &its0 0x3000 0x1000>;
+		iommu-map = <0x3000 &mmu600_pcie 0x3000 0x1000>;
 		num-lanes = <1>;
 		phys = <&combphy2_psu PHY_TYPE_PCIE>;
 		phy-names = "pcie-phy";
@@ -1692,6 +1692,7 @@ pcie2x1l2: pcie@fe190000 {
 		linux,pci-domain = <4>;
 		max-link-speed = <2>;
 		msi-map = <0x4000 &its0 0x4000 0x1000>;
+		iommu-map = <0x4000 &mmu600_pcie 0x4000 0x1000>;
 		num-lanes = <1>;
 		phys = <&combphy0_ps PHY_TYPE_PCIE>;
 		phy-names = "pcie-phy";
diff --git a/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi b/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi
index 0ce0934ec6b79..4a950907ea6f5 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi
@@ -162,6 +162,7 @@ pcie3x4: pcie@fe150000 {
 		linux,pci-domain = <0>;
 		max-link-speed = <3>;
 		msi-map = <0x0000 &its1 0x0000 0x1000>;
+		iommu-map = <0x0000 &mmu600_pcie 0x0000 0x1000>;
 		num-lanes = <4>;
 		phys = <&pcie30phy>;
 		phy-names = "pcie-phy";
@@ -212,6 +213,7 @@ pcie3x4_ep: pcie-ep@fe150000 {
 		interrupt-names = "sys", "pmc", "msg", "legacy", "err",
 				  "dma0", "dma1", "dma2", "dma3";
 		max-link-speed = <3>;
+		iommus = <&mmu600_pcie 0x0000>;
 		num-lanes = <4>;
 		phys = <&pcie30phy>;
 		phy-names = "pcie-phy";
@@ -248,6 +250,7 @@ pcie3x2: pcie@fe160000 {
 		linux,pci-domain = <1>;
 		max-link-speed = <3>;
 		msi-map = <0x1000 &its1 0x1000 0x1000>;
+		iommu-map = <0x1000 &mmu600_pcie 0x1000 0x1000>;
 		num-lanes = <2>;
 		phys = <&pcie30phy>;
 		phy-names = "pcie-phy";
@@ -297,6 +300,7 @@ pcie2x1l0: pcie@fe170000 {
 		linux,pci-domain = <2>;
 		max-link-speed = <2>;
 		msi-map = <0x2000 &its0 0x2000 0x1000>;
+		iommu-map = <0x2000 &mmu600_pcie 0x2000 0x1000>;
 		num-lanes = <1>;
 		phys = <&combphy1_ps PHY_TYPE_PCIE>;
 		phy-names = "pcie-phy";
-- 
2.47.0

Re: [PATCH] arm64: dts: rockchip: enable the mmu600_pcie IOMMU on the rk3588 SoC

[Niklas Cassel]

On Thu, Nov 07, 2024 at 01:37:33PM +0100, Niklas Cassel wrote:
> Commit cd81d3a0695c ("arm64: dts: rockchip: add rk3588 pcie and php
> IOMMUs") added the rk3588 SoC's pcie IOMMU and php IOMMU as disabled.
> 
> The mmu600_pcie is connected with the five PCIe controllers.
> See 8.2 Block Diagram, in rk3588 TRM (Technical Reference Manual).
> 
> The five PCIe controllers are:
> pcie3x4, pcie3x2, pcie2x1l0, pcie2x1l1, pcie2x1l2.
> 
> pcie3x4 can run in either Root Complex mode or Endpoint mode, the other
> four PCIe controllers can only run in Root Complex mode. To describe this
> we thus have six different device nodes in the device tree.
> 
> A PCIe controller in Root Complex mode needs to specify an iommu-map, such
> that the device knows how to convert a Requester ID (PCI BDF) to an IOMMU
> master ID (stream ID). (A PCIe controller in Endpoint mode should use the
> iommus property, just like a regular device.)
> 
> If you look at the device tree bindings for msi-map and iommu-map, you can
> see that the conversion from Requester ID to MSI-specifier data is the same
> as the conversion from Requester ID to IOMMU specifier data. Thus it is
> sensible to define the iommu-map property value similar to the msi-map,
> such that the conversion will be identical.
> 
> Add the proper iommu device tree properties for these six device nodes
> connected to the mmu600_pcie, so that we can enable the mmu600_pcie IOMMU.
> (The mmu600_php IOMMU is not touched, so it is still disabled.)
> 
> Signed-off-by: Niklas Cassel <cassel@kernel.org>
> ---

Hello Heiko,

Any chance of getting this picked up?

(If not now, then at least for 6.14.)


Kind regards,
Niklas

Re: [PATCH] arm64: dts: rockchip: enable the mmu600_pcie IOMMU on the rk3588 SoC

[Heiko Stübner]

Hi Niklas,

Am Montag, 18. November 2024, 12:52:23 CET schrieb Niklas Cassel:
> On Thu, Nov 07, 2024 at 01:37:33PM +0100, Niklas Cassel wrote:
> > Commit cd81d3a0695c ("arm64: dts: rockchip: add rk3588 pcie and php
> > IOMMUs") added the rk3588 SoC's pcie IOMMU and php IOMMU as disabled.
> > 
> > The mmu600_pcie is connected with the five PCIe controllers.
> > See 8.2 Block Diagram, in rk3588 TRM (Technical Reference Manual).
> > 
> > The five PCIe controllers are:
> > pcie3x4, pcie3x2, pcie2x1l0, pcie2x1l1, pcie2x1l2.
> > 
> > pcie3x4 can run in either Root Complex mode or Endpoint mode, the other
> > four PCIe controllers can only run in Root Complex mode. To describe this
> > we thus have six different device nodes in the device tree.
> > 
> > A PCIe controller in Root Complex mode needs to specify an iommu-map, such
> > that the device knows how to convert a Requester ID (PCI BDF) to an IOMMU
> > master ID (stream ID). (A PCIe controller in Endpoint mode should use the
> > iommus property, just like a regular device.)
> > 
> > If you look at the device tree bindings for msi-map and iommu-map, you can
> > see that the conversion from Requester ID to MSI-specifier data is the same
> > as the conversion from Requester ID to IOMMU specifier data. Thus it is
> > sensible to define the iommu-map property value similar to the msi-map,
> > such that the conversion will be identical.
> > 
> > Add the proper iommu device tree properties for these six device nodes
> > connected to the mmu600_pcie, so that we can enable the mmu600_pcie IOMMU.
> > (The mmu600_php IOMMU is not touched, so it is still disabled.)
> > 
> > Signed-off-by: Niklas Cassel <cassel@kernel.org>
> > ---
> 
> Hello Heiko,
> 
> Any chance of getting this picked up?
> 
> (If not now, then at least for 6.14.)

Oh, definitly :-) ... it's marked as to look at.

For 6.14 ... the patch arrived shortly before -rc7, with PCIe stuff in it.
IOMMUs + PCIe is sort of a topic I'm cautious about shortly before
the merge window ;-)


Heiko

Re: [PATCH] arm64: dts: rockchip: enable the mmu600_pcie IOMMU on the rk3588 SoC

[Heiko Stuebner]

On Thu, 07 Nov 2024 13:37:33 +0100, Niklas Cassel wrote:
> Commit cd81d3a0695c ("arm64: dts: rockchip: add rk3588 pcie and php
> IOMMUs") added the rk3588 SoC's pcie IOMMU and php IOMMU as disabled.
> 
> The mmu600_pcie is connected with the five PCIe controllers.
> See 8.2 Block Diagram, in rk3588 TRM (Technical Reference Manual).
> 
> The five PCIe controllers are:
> pcie3x4, pcie3x2, pcie2x1l0, pcie2x1l1, pcie2x1l2.
> 
> [...]

Applied, thanks!

[1/1] arm64: dts: rockchip: enable the mmu600_pcie IOMMU on the rk3588 SoC
      commit: da92d3dfc871e821a1bface3ba5afcf8cda19805

Best regards,
-- 
Heiko Stuebner <heiko@sntech.de>

Re: [PATCH] arm64: dts: rockchip: enable the mmu600_pcie IOMMU on the rk3588 SoC

[Heiko Stübner]

Hi Niklas,

Am Donnerstag, 7. November 2024, 13:37:33 Mitteleuropäische Sommerzeit schrieb Niklas Cassel:
> Commit cd81d3a0695c ("arm64: dts: rockchip: add rk3588 pcie and php
> IOMMUs") added the rk3588 SoC's pcie IOMMU and php IOMMU as disabled.
> 
> The mmu600_pcie is connected with the five PCIe controllers.
> See 8.2 Block Diagram, in rk3588 TRM (Technical Reference Manual).
> 
> The five PCIe controllers are:
> pcie3x4, pcie3x2, pcie2x1l0, pcie2x1l1, pcie2x1l2.
> 
> pcie3x4 can run in either Root Complex mode or Endpoint mode, the other
> four PCIe controllers can only run in Root Complex mode. To describe this
> we thus have six different device nodes in the device tree.
> 
> A PCIe controller in Root Complex mode needs to specify an iommu-map, such
> that the device knows how to convert a Requester ID (PCI BDF) to an IOMMU
> master ID (stream ID). (A PCIe controller in Endpoint mode should use the
> iommus property, just like a regular device.)
> 
> If you look at the device tree bindings for msi-map and iommu-map, you can
> see that the conversion from Requester ID to MSI-specifier data is the same
> as the conversion from Requester ID to IOMMU specifier data. Thus it is
> sensible to define the iommu-map property value similar to the msi-map,
> such that the conversion will be identical.
> 
> Add the proper iommu device tree properties for these six device nodes
> connected to the mmu600_pcie, so that we can enable the mmu600_pcie IOMMU.
> (The mmu600_php IOMMU is not touched, so it is still disabled.)
> 
> Signed-off-by: Niklas Cassel <cassel@kernel.org>

I've run into a bit of a problem with this patch and v6.16 today :-( .

It seems the SMMU currently does not play well, if has no users. With
this patch the pcie smmu is always probed, even if nothing is using it.
(or in fact the PCIe controller driver defers probing for whatever reason).

But in all cases the SMMU's shutdown callback will be called when needed.

So test-case:
- probe pcie-smmu, but do not probe PCIe ... hit reboot
  The smmu shutdown function will hang for a bit, and at least my two
  test boards  did reboot after hanging about 30 seconds
  (I did trace that down to the arm_smmu_shutdown function)
- probe pcie-smmu and probe PCIe - creating a user for the smmu
  Reboot happens without hang/delay

Likely the SMMU is missing either a clock or its power-domain.

Just adding the power-domain to the node, did keep the domain on, but
did not solve the hang for me.

Strangely enough, additionally forcing all clocks to stay permanently on,
also didn't help (or I messed up somewhere) .
I'm not sure if the PCIe controller does cause the SMMU to do some more
initialization, that would be missing otherwise.


Robin did point out that there is ongoing runtime-pm work for the
SMMU [0], but just adding the power-domain does keep it on, so this is
likely not related.


As you have spent so much time on PCIe topics, maybe you have a better
idea of what to poke :-)


Heiko


[0] https://lore.kernel.org/linux-iommu/20250616203149.2649118-1-praan@google.com/

Re: [PATCH] arm64: dts: rockchip: enable the mmu600_pcie IOMMU on the rk3588 SoC

[Niklas Cassel]

Hello Heiko,

On Wed, Aug 06, 2025 at 09:36:10PM +0200, Heiko Stübner wrote:
> 
> I've run into a bit of a problem with this patch and v6.16 today :-( .

Is the problem there also on older kernels?
(There as been a bunch of patches to the arm-smmu-v3 driver lately.)


> 
> It seems the SMMU currently does not play well, if has no users. With
> this patch the pcie smmu is always probed, even if nothing is using it.
> (or in fact the PCIe controller driver defers probing for whatever reason).

Well, the good thing is that the PCIe driver will probe successfully even
when there is nothing attached to the PCIe slot:
https://github.com/torvalds/linux/blob/v6.17-rc1/drivers/pci/controller/dwc/pcie-designware-host.c#L560-L561

So the only case where the driver will still be deferred after a normal
boot is if e.g. the user forgot to select the Kconfig for the PHY driver.

It does of course sound very bad that the board takes 30 seconds to reboot
in this case, but at least it reboots eventually.


> 
> But in all cases the SMMU's shutdown callback will be called when needed.
> 
> So test-case:
> - probe pcie-smmu, but do not probe PCIe ... hit reboot
>   The smmu shutdown function will hang for a bit, and at least my two
>   test boards  did reboot after hanging about 30 seconds
>   (I did trace that down to the arm_smmu_shutdown function)
> - probe pcie-smmu and probe PCIe - creating a user for the smmu
>   Reboot happens without hang/delay
> 
> Likely the SMMU is missing either a clock or its power-domain.
> 
> Just adding the power-domain to the node, did keep the domain on, but
> did not solve the hang for me.

Which power-domain did you add?

When I look at the PCIe controllers, they seem to use:
power-domains = <&power RK3588_PD_PCIE>;

So I assume that this is the power domain that you tried?

If I look at the RK3588 TRM Part1, 7.3.1 Domain Description,
page 1096, it does seem like the MMU600_PCIE is in power domain
PD_PHP and not PD_PCIE.

Perhaps you could something like:
power-domains = <&power RK3588_PD_PCIE>, <&power RK3588_PD_PHP>;

just to see if that improves things? (Ideally I would assume that we
should just need RK3588_PD_PHP.)

I do notice that RK3588_PD_PHP is defined/used in:
drivers/pmdomain/rockchip/pm-domains.c and
include/dt-bindings/power/rk3588-power.h

However, the RK3588_PD_PHP power domain does not seem to be defined in
rk3588-base.dtsi, like e.g. the RK3588_PD_PCIE power domain is:
https://github.com/torvalds/linux/blob/v6.17-rc1/arch/arm64/boot/dts/rockchip/rk3588-base.dtsi#L1115-L1121

I also cannot find the RK3588_PD_PHP power domain defined in the
downstream dtsi:
https://github.com/radxa/kernel/blob/linux-6.1-stan-rkr5.1/arch/arm64/boot/dts/rockchip/rk3588s.dtsi#L3422-L3427



Side note: from the TRM is seems that
pcie2x1l0, pcie2x1l1, pcie2x1l2 is also in the RK3588_PD_PHP power domain.
(only pcie3x4 and pcie3x2 is in the RK3588_PD_PCIE power domain.)

Yet, looking at rk3588-base.dtsi and rk3588-extra.dtsi, all five
PCIe controllers has:
power-domains = <&power RK3588_PD_PCIE>;

Looking at downstream, pcie3x4 and pcie3x2 has:
power-domains = <&power RK3588_PD_PCIE>;
but pcie2x1l0, pcie2x1l1, pcie2x1l2 has no power-domains property at all.

Perhaps a bug when Sebastian added the PCIe device tree nodes to upstream?


> Strangely enough, additionally forcing all clocks to stay permanently on,
> also didn't help (or I messed up somewhere) .
> I'm not sure if the PCIe controller does cause the SMMU to do some more
> initialization, that would be missing otherwise.

That does sound like the most likely explanation.

arm_smmu_device_shutdown() calls arm_smmu_device_disable() which simply
does a arm_smmu_write_reg_sync(smmu, 0, ARM_SMMU_CR0, ARM_SMMU_CR0ACK);

The ARM_SMMU_POLL_TIMEOUT_US is however 1s, so if your board reboots after
30 seconds, that is probably because of a watchdog or similar.


Kind regards,
Niklas