[PATCH 0/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[PATCH 0/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[Chukun Pan]

For a long time, rk3568's MSI-X had bugs and could only work on one node.
e.g. [    7.250882] r8125 0002:01:00.0: no MSI/MSI-X. Back to INTx.

Now the ITS of GICv3 on rk3568 has been fixed by commit b08e2f42e86b
("irqchip/gic-v3-its: Share ITS tables with a non-trusted hypervisor")
and commit 2d81e1bb6252 ("irqchip/gic-v3: Add Rockchip 3568002 erratum
workaround").

Following commit b956c9de9175 ("arm64: dts: rockchip: rk356x: Move
PCIe MSI to use GIC ITS instead of MBI"), change the PCIe3 controller's
MSI on rk3568 to use ITS, so that all MSI-X can work properly.

~# dmesg | grep -E 'GIC|ITS'
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] GIC: enabling workaround for GICv3: non-coherent attribute
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 320 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: MBI range [296:319]
[    0.000000] GICv3: Using MBI frame 0x00000000fd410000
[    0.000000] GICv3: GICv3 features: 16 PPIs
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000fd460000
[    0.000000] ITS [mem 0xfd440000-0xfd45ffff]
[    0.000000] GIC: enabling workaround for ITS: Rockchip erratum RK3568002
[    0.000000] GIC: enabling workaround for ITS: non-coherent attribute
[    0.000000] ITS@0x00000000fd440000: allocated 8192 Devices @210000 (indirect, esz 8, psz 64K, shr 0)
[    0.000000] ITS@0x00000000fd440000: allocated 32768 Interrupt Collections @220000 (flat, esz 2, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x0000000000230000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000000240000
[    0.013946] GICv3: CPU1: found redistributor 100 region 0:0x00000000fd480000
[    0.013968] GICv3: CPU1: using allocated LPI pending table @0x0000000000250000
[    0.014948] GICv3: CPU2: found redistributor 200 region 0:0x00000000fd4a0000
[    0.014968] GICv3: CPU2: using allocated LPI pending table @0x0000000000260000
[    0.015904] GICv3: CPU3: found redistributor 300 region 0:0x00000000fd4c0000
[    0.015923] GICv3: CPU3: using allocated LPI pending table @0x0000000000270000

~# lspci -v | grep MSI-X
        Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
        Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
        Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-

Chukun Pan (1):
  arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

 arch/arm64/boot/dts/rockchip/rk3568.dtsi | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

-- 
2.25.1


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

[PATCH 1/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[Chukun Pan]

Following commit b956c9de9175 ("arm64: dts: rockchip: rk356x: Move
PCIe MSI to use GIC ITS instead of MBI"), change the PCIe3 controller's
MSI on rk3568 to use ITS, so that all MSI-X can work properly.

Signed-off-by: Chukun Pan <amadeus@jmu.edu.cn>
---
 arch/arm64/boot/dts/rockchip/rk3568.dtsi | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/boot/dts/rockchip/rk3568.dtsi b/arch/arm64/boot/dts/rockchip/rk3568.dtsi
index 695cccbdab0f..e719a3df126c 100644
--- a/arch/arm64/boot/dts/rockchip/rk3568.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3568.dtsi
@@ -152,7 +152,7 @@ pcie3x1: pcie@fe270000 {
 		compatible = "rockchip,rk3568-pcie";
 		#address-cells = <3>;
 		#size-cells = <2>;
-		bus-range = <0x0 0xf>;
+		bus-range = <0x10 0x1f>;
 		clocks = <&cru ACLK_PCIE30X1_MST>, <&cru ACLK_PCIE30X1_SLV>,
 			 <&cru ACLK_PCIE30X1_DBI>, <&cru PCLK_PCIE30X1>,
 			 <&cru CLK_PCIE30X1_AUX_NDFT>;
@@ -175,7 +175,7 @@ pcie3x1: pcie@fe270000 {
 		num-ib-windows = <6>;
 		num-ob-windows = <2>;
 		max-link-speed = <3>;
-		msi-map = <0x0 &gic 0x1000 0x1000>;
+		msi-map = <0x1000 &its 0x1000 0x1000>;
 		num-lanes = <1>;
 		phys = <&pcie30phy>;
 		phy-names = "pcie-phy";
@@ -205,7 +205,7 @@ pcie3x2: pcie@fe280000 {
 		compatible = "rockchip,rk3568-pcie";
 		#address-cells = <3>;
 		#size-cells = <2>;
-		bus-range = <0x0 0xf>;
+		bus-range = <0x20 0x2f>;
 		clocks = <&cru ACLK_PCIE30X2_MST>, <&cru ACLK_PCIE30X2_SLV>,
 			 <&cru ACLK_PCIE30X2_DBI>, <&cru PCLK_PCIE30X2>,
 			 <&cru CLK_PCIE30X2_AUX_NDFT>;
@@ -228,7 +228,7 @@ pcie3x2: pcie@fe280000 {
 		num-ib-windows = <6>;
 		num-ob-windows = <2>;
 		max-link-speed = <3>;
-		msi-map = <0x0 &gic 0x2000 0x1000>;
+		msi-map = <0x2000 &its 0x2000 0x1000>;
 		num-lanes = <2>;
 		phys = <&pcie30phy>;
 		phy-names = "pcie-phy";
-- 
2.25.1


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [PATCH 1/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[kernel test robot]

Hi Chukun,

kernel test robot noticed the following build errors:

[auto build test ERROR on rockchip/for-next]
[also build test ERROR on linus/master v6.14-rc7]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Chukun-Pan/arm64-dts-rockchip-rk3568-Move-PCIe3-MSI-to-use-GIC-ITS/20250308-173422
base:   https://git.kernel.org/pub/scm/linux/kernel/git/mmind/linux-rockchip.git for-next
patch link:    https://lore.kernel.org/r/20250308093008.568437-2-amadeus%40jmu.edu.cn
patch subject: [PATCH 1/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS
config: arm64-randconfig-r071-20250321 (https://download.01.org/0day-ci/archive/20250322/202503221203.7E0Oa0zG-lkp@intel.com/config)
compiler: clang version 16.0.6 (https://github.com/llvm/llvm-project 7cbf1a2591520c2491aa35339f227775f4d3adf6)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250322/202503221203.7E0Oa0zG-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202503221203.7E0Oa0zG-lkp@intel.com/

All errors (new ones prefixed by >>):

   also defined at arch/arm64/boot/dts/rockchip/rk3568-bpi-r2-pro.dts:597.10-605.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
   also defined at arch/arm64/boot/dts/rockchip/rk3568-fastrhino-r66s.dtsi:368.10-373.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
   also defined at arch/arm64/boot/dts/rockchip/rk3568-lubancat-2.dts:518.10-522.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
   also defined at arch/arm64/boot/dts/rockchip/rk3568-mecsbc.dts:273.10-279.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
   also defined at arch/arm64/boot/dts/rockchip/rk3568-nanopi-r5c.dts:75.10-80.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
   also defined at arch/arm64/boot/dts/rockchip/rk3568-nanopi-r5s.dts:103.10-110.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
   also defined at arch/arm64/boot/dts/rockchip/rk3568-odroid-m1.dts:503.10-509.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
   also defined at arch/arm64/boot/dts/rockchip/rk3568-photonicat.dts:361.10-369.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
   also defined at arch/arm64/boot/dts/rockchip/rk3568-qnap-ts433.dts:551.10-556.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
--
   also defined at arch/arm64/boot/dts/rockchip/rk3568-radxa-e25.dts:133.10-140.3
>> ERROR: Input tree has errors, aborting (use -f to force output)
..

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [PATCH 1/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[Chukun Pan]

Hi,

> kernel test robot noticed the following build errors:
>
> [auto build test ERROR on rockchip/for-next]
> [also build test ERROR on linus/master v6.14-rc7]
> [If your patch is applied to the wrong git tree, kindly drop us a note.
> And when submitting patch, we suggest to use '--base' as documented in
> https://git-scm.com/docs/git-format-patch#_base_tree_information]

commit f15be3d4a0a5 ("arm64: dts: rockchip: rk356x: Add MSI controller node")
is already included in rockchip/for-next. Do I need to resend this patch?

Thanks,
Chukun

--
2.25.1


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [PATCH 1/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[Heiko Stuebner]

On Sat, 08 Mar 2025 17:30:08 +0800, Chukun Pan wrote:
> Following commit b956c9de9175 ("arm64: dts: rockchip: rk356x: Move
> PCIe MSI to use GIC ITS instead of MBI"), change the PCIe3 controller's
> MSI on rk3568 to use ITS, so that all MSI-X can work properly.
> 
> 

Applied, thanks!

[1/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS
      commit: fbea35a661ed100cee2f3bab8015fb0155508106

Best regards,
-- 
Heiko Stuebner <heiko@sntech.de>

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [PATCH 0/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[Diederik de Haas]

[-- Attachment #1.1: Type: text/plain, Size: 2006 bytes --]

Hi,

On Sat Mar 8, 2025 at 10:30 AM CET, Chukun Pan wrote:
> For a long time, rk3568's MSI-X had bugs and could only work on one node.
> e.g. [    7.250882] r8125 0002:01:00.0: no MSI/MSI-X. Back to INTx.
>
> Now the ITS of GICv3 on rk3568 has been fixed by commit b08e2f42e86b
> ("irqchip/gic-v3-its: Share ITS tables with a non-trusted hypervisor")
> and commit 2d81e1bb6252 ("irqchip/gic-v3: Add Rockchip 3568002 erratum
> workaround").
>
> Following commit b956c9de9175 ("arm64: dts: rockchip: rk356x: Move
> PCIe MSI to use GIC ITS instead of MBI"), change the PCIe3 controller's
> MSI on rk3568 to use ITS, so that all MSI-X can work properly.
>
> ~# dmesg | grep -E 'GIC|ITS'
> <snip>
>
> ~# lspci -v | grep MSI-X
>         Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
>         Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
>         Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
>         Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
>         Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-

I tested this patch on my NanoPi R5S with a 6.15-rc3 kernel + a number
of [vcc|phy]-supply patches that have been accepted for 6.16 (and a
small WIP LED patch).

With this patch I get the following kernel warnings:

pci 0001:10:00.0: Primary bus is hard wired to 0
pci 0002:20:00.0: Primary bus is hard wired to 0

If I 'unapply' this patch, I don't see those warnings.
The output of the above mentioned ``dmesg`` and ``lspci`` commands is
exactly the same in both cases.

The PCI IDs refer to:
PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)

It's possible that this patch only brought a(nother) problem to light,
but was I supposed to see an improvement and if so where/how?

Cheers,
  Diederik

> Chukun Pan (1):
>   arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS
>
>  arch/arm64/boot/dts/rockchip/rk3568.dtsi | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [PATCH 0/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[Diederik de Haas]

[-- Attachment #1.1: Type: text/plain, Size: 1364 bytes --]

Hi again,

On Fri May 9, 2025 at 5:34 PM CEST, Diederik de Haas wrote:
> On Sat Mar 8, 2025 at 10:30 AM CET, Chukun Pan wrote:
>> For a long time, rk3568's MSI-X had bugs and could only work on one node.
>> e.g. [    7.250882] r8125 0002:01:00.0: no MSI/MSI-X. Back to INTx.
>>
>> Following commit b956c9de9175 ("arm64: dts: rockchip: rk356x: Move
>> PCIe MSI to use GIC ITS instead of MBI"), change the PCIe3 controller's
>> MSI on rk3568 to use ITS, so that all MSI-X can work properly.
>>
> I tested this patch on my NanoPi R5S with a 6.15-rc3 kernel + a number
> of [vcc|phy]-supply patches that have been accepted for 6.16 (and a
> small WIP LED patch).
>
> With this patch I get the following kernel warnings:
>
> pci 0001:10:00.0: Primary bus is hard wired to 0
> pci 0002:20:00.0: Primary bus is hard wired to 0
>
> If I 'unapply' this patch, I don't see those warnings.

I was pretty sure I had seen those messages before, but couldn't find
them before. But now I have: on my rk3588-rock-5b.

> It's possible that this patch only brought a(nother) problem to light,

So it looks indeed to be this.

Cheers,
  Diederik

>> Chukun Pan (1):
>>   arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS
>>
>>  arch/arm64/boot/dts/rockchip/rk3568.dtsi | 8 ++++----
>>  1 file changed, 4 insertions(+), 4 deletions(-)


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [PATCH 0/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[Chukun Pan]

Hi,

> > With this patch I get the following kernel warnings:
> >
> > pci 0001:10:00.0: Primary bus is hard wired to 0
> > pci 0002:20:00.0: Primary bus is hard wired to 0
> >
> > If I 'unapply' this patch, I don't see those warnings.

> I was pretty sure I had seen those messages before, but couldn't find
> them before. But now I have: on my rk3588-rock-5b.

Thanks for the reminder, I didn't notice this before.
The BSP kernel also has this warning.

Before this patch:
[    2.997725] pci_bus 0001:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    3.009990] pci 0001:00:00.0: BAR 6: assigned [mem 0xf2200000-0xf220ffff pref]
[    3.018100] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
...
[    3.401416] pci_bus 0002:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
...
[    3.545459] pci 0002:00:00.0: PCI bridge to [bus 01-ff]

After this patch:
[    3.037779] pci 0001:10:00.0: Primary bus is hard wired to 0
[    3.044120] pci 0001:10:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
[    3.053362] pci_bus 0001:11: busn_res: [bus 11-1f] end is updated to 11
[    3.068920] pci 0001:10:00.0: PCI bridge to [bus 11]
...
[    3.451429] pci 0002:20:00.0: Primary bus is hard wired to 0
[    3.457793] pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
...
[    3.535794] pci_bus 0002:21: busn_res: [bus 21-2f] end is updated to 21
...
[    3.612893] pci 0002:20:00.0: PCI bridge to [bus 21]

Looks like a harmless warning.

Thanks,
Chukun

--
2.25.1


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [PATCH 0/1] arm64: dts: rockchip: rk3568: Move PCIe3 MSI to use GIC ITS

[Diederik de Haas]

[-- Attachment #1.1: Type: text/plain, Size: 3346 bytes --]

Hi,

Added linux-pci ML to "To".

On Mon May 12, 2025 at 9:00 AM CEST, Chukun Pan wrote:
>> > With this patch I get the following kernel warnings:
>> >
>> > pci 0001:10:00.0: Primary bus is hard wired to 0
>> > pci 0002:20:00.0: Primary bus is hard wired to 0
>> >
>> > If I 'unapply' this patch, I don't see those warnings.
>
>> I was pretty sure I had seen those messages before, but couldn't find
>> them before. But now I have: on my rk3588-rock-5b.
>
> Thanks for the reminder, I didn't notice this before.
> The BSP kernel also has this warning.
>
> Before this patch:
> [    2.997725] pci_bus 0001:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
> [    3.009990] pci 0001:00:00.0: BAR 6: assigned [mem 0xf2200000-0xf220ffff pref]
> [    3.018100] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
> ...
> [    3.401416] pci_bus 0002:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
> ...
> [    3.545459] pci 0002:00:00.0: PCI bridge to [bus 01-ff]
>
> After this patch:
> [    3.037779] pci 0001:10:00.0: Primary bus is hard wired to 0
> [    3.044120] pci 0001:10:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
> [    3.053362] pci_bus 0001:11: busn_res: [bus 11-1f] end is updated to 11
> [    3.068920] pci 0001:10:00.0: PCI bridge to [bus 11]
> ...
> [    3.451429] pci 0002:20:00.0: Primary bus is hard wired to 0
> [    3.457793] pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
> ...
> [    3.535794] pci_bus 0002:21: busn_res: [bus 21-2f] end is updated to 21
> ...
> [    3.612893] pci 0002:20:00.0: PCI bridge to [bus 21]
>
> Looks like a harmless warning.

I see various messages which look odd or suboptimal to me:
- (conflicts with (null) [bus 00-0f])
- bridge configuration invalid ([bus 01-ff]), reconfiguring

But those are informational messages, so I guess that is considered
normal. Looking a bit further and it does look that the severities in
``drivers/pci/probe.c`` are chosen deliberately. So even though my NVMe
drives seem to work, I'm not ready yet to ignore a WARNING.

In my view, a warning is something that should be fixed or if it's
indeed harmless, then its severity should be downgraded.

So I looked where that warning came from and found commit
71f6bd4a2313 ("PCI: workaround hard-wired bus number V2")

And its commit message does not make it clear to *me* if it's valid:

    Fixes PCI device detection on IBM xSeries IBM 3850 M2 / x3950 M2
    when using ACPI resources (_CRS).
    This is default, a manual workaround (without this patch)
    would be pci=nocrs boot param.

    V2: Add dev_warn if the workaround is hit. This should reveal
    how common such setups are (via google) and point to possible
    problems if things are still not working as expected.

This could be interpreted as "let's make it a warning so people will put
it in a search engine (and not just ignore it) and then we can find out
via that, if it's a common issue".

It would be helpful if the people (way) more familiar with the PCI
subsystem then me to tell me/us whether the severity is appropriate (and
thus should be fixed?) or if this should be an info or dbg level message
instead.

Cheers,
  Diederik

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip