Re: [PATCH] arm64: dts: rockchip: enable the mmu600_pcie IOMMU on the rk3588 SoC

["Heiko Stübner" <heiko@sntech.de>]

Hi Niklas,

Am Donnerstag, 7. November 2024, 13:37:33 Mitteleuropäische Sommerzeit schrieb Niklas Cassel:
> Commit cd81d3a0695c ("arm64: dts: rockchip: add rk3588 pcie and php
> IOMMUs") added the rk3588 SoC's pcie IOMMU and php IOMMU as disabled.
> 
> The mmu600_pcie is connected with the five PCIe controllers.
> See 8.2 Block Diagram, in rk3588 TRM (Technical Reference Manual).
> 
> The five PCIe controllers are:
> pcie3x4, pcie3x2, pcie2x1l0, pcie2x1l1, pcie2x1l2.
> 
> pcie3x4 can run in either Root Complex mode or Endpoint mode, the other
> four PCIe controllers can only run in Root Complex mode. To describe this
> we thus have six different device nodes in the device tree.
> 
> A PCIe controller in Root Complex mode needs to specify an iommu-map, such
> that the device knows how to convert a Requester ID (PCI BDF) to an IOMMU
> master ID (stream ID). (A PCIe controller in Endpoint mode should use the
> iommus property, just like a regular device.)
> 
> If you look at the device tree bindings for msi-map and iommu-map, you can
> see that the conversion from Requester ID to MSI-specifier data is the same
> as the conversion from Requester ID to IOMMU specifier data. Thus it is
> sensible to define the iommu-map property value similar to the msi-map,
> such that the conversion will be identical.
> 
> Add the proper iommu device tree properties for these six device nodes
> connected to the mmu600_pcie, so that we can enable the mmu600_pcie IOMMU.
> (The mmu600_php IOMMU is not touched, so it is still disabled.)
> 
> Signed-off-by: Niklas Cassel <cassel@kernel.org>

I've run into a bit of a problem with this patch and v6.16 today :-( .

It seems the SMMU currently does not play well, if has no users. With
this patch the pcie smmu is always probed, even if nothing is using it.
(or in fact the PCIe controller driver defers probing for whatever reason).

But in all cases the SMMU's shutdown callback will be called when needed.

So test-case:
- probe pcie-smmu, but do not probe PCIe ... hit reboot
  The smmu shutdown function will hang for a bit, and at least my two
  test boards  did reboot after hanging about 30 seconds
  (I did trace that down to the arm_smmu_shutdown function)
- probe pcie-smmu and probe PCIe - creating a user for the smmu
  Reboot happens without hang/delay

Likely the SMMU is missing either a clock or its power-domain.

Just adding the power-domain to the node, did keep the domain on, but
did not solve the hang for me.

Strangely enough, additionally forcing all clocks to stay permanently on,
also didn't help (or I messed up somewhere) .
I'm not sure if the PCIe controller does cause the SMMU to do some more
initialization, that would be missing otherwise.


Robin did point out that there is ongoing runtime-pm work for the
SMMU [0], but just adding the power-domain does keep it on, so this is
likely not related.


As you have spent so much time on PCIe topics, maybe you have a better
idea of what to poke :-)


Heiko


[0] https://lore.kernel.org/linux-iommu/20250616203149.2649118-1-praan@google.com/