In device tree, can't understand 'ranges' property of PCIe root complex node

In device tree, can't understand 'ranges' property of PCIe root complex node

[Chan Kim]

Hello all,

For example in linux-5.15.68's arch/arm64/boot/dts/ti/k3-am64-main.dtsi,  

    &cbass_main {

        ... skip ...

        pcie0_rc: pcie@f102000 {
            compatible = "ti,am64-pcie-host", "ti,j721e-pcie-host";
            reg = <0x00 0x0f102000 0x00 0x1000>,
                  <0x00 0x0f100000 0x00 0x400>,
                  <0x00 0x0d000000 0x00 0x00800000>,
                  <0x00 0x68000000 0x00 0x00001000>;

            reg-names = "intd_cfg", "user_cfg", "reg", "cfg";
      ... skip ...   

            #address-cells = <3>;
            #size-cells = <2>;

       ...skip ...

            msi-map = <0x0 &gic_its 0x0 0x10000>;
            ranges = <0x01000000 0x00 0x68001000  0x00 0x68001000  0x00
0x0010000>,
                 <0x02000000 0x00 0x68011000  0x00 0x68011000  0x00
0x7fef000>;
            dma-ranges = <0x02000000 0x0 0x0 0x0 0x0 0x00000010 0x0>;
        };
    };

the 'ranges' property says  
- IO space starting at address 0x68001000 with size 64KiB is mapped to the
parent bus address 0x68001000.    
- a 32 bit memory address space starting with address 0x68011000 and size
0x7fef000 is mapped to parent bus address 0x68011000.  

The device tree document says these addresses are all physical. 
What I can't understand is :   

I know the BIOS will set the BAR addresses during the enumeration process.
This is allocating the endpoint's function to a certain PCIe bus address and
I guess setting BAR can be done without this 'ranges' information(it will
assign free address range for that device according to the device's need).
By the time linux kernel later reads this 'ranges' property, the BIOS (or
bootloader) has already assigned different PCIe address to that device.
Then, how is this 'ranges' property used by the kernel?

Thank you.

Chan Kim





_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

RE: In device tree, can't understand 'ranges' property of PCIe root complex node

[Chan Kim]

Hello all,

I forgot that I asked this question here and I'll write what I have found
later.

I found in 'PCI bus bindings to : IEEE Std 1275-1994 standard for Boot
firmware' document, this 'range' property is for pci-to-pci bridge (and RC
contains bridge property I understand). So if an address range is behind
that bridge, that should be included in the 'ranges' property.

And '6. Probing PCI-to-PCI bridges' of the document says (after setting the
bridge parameters during the scan) "After setting the Memory Base, Memory
Limit, I/O Base and I/O Limit registers, create a "ranges" property that
represents the Memory and I/O mappings that have been established." . So it
looks like this parameter is set by the bus scanning entity like bootloader
or kernel.

So BAR is setting the PCIe address range for each device/function, and
'ranges' is for setting address ranges covering all the devices behind the
RC. I think the ranges can be merged or combined for some devices and it
provides also the translation relation(between PCIe address and host CPU
address). And I haven't check in the linux source but this 'ranges' property
should be set in the address translation unit inside the PCIe RC core. (For
example, in Synopsys PCIe core, there is iATU-internal address translation
unit- for which we can set address translation entries for some ranges.).
and I also guess when we do ioremap for a PCIe BAR address, it will consider
this address translation for that address too.

But still, I'm curious what happens in the existing 'ranges' property in the
device tree before the bus scanning. Does the bootloader/kernel respect the
value always (and allocates address within that ranges when allocating
BARs.)? or can it ignore and override the range assignment?

I would appreciate if someone could clear things up for me.

Thank you.
Chan Kim


>-----Original Message-----
>From: Chan Kim <ckim@etri.re.kr>
>Sent: Monday, February 6, 2023 9:07 PM
>To: kernelnewbies@kernelnewbies.org
>Subject: In device tree, can't understand 'ranges' property of PCIe root
>complex node
>
>Hello all,
>
>For example in linux-5.15.68's arch/arm64/boot/dts/ti/k3-am64-main.dtsi,
>
>    &cbass_main {
>
>        ... skip ...
>
>        pcie0_rc: pcie@f102000 {
>            compatible = "ti,am64-pcie-host", "ti,j721e-pcie-host";
>            reg = <0x00 0x0f102000 0x00 0x1000>,
>                  <0x00 0x0f100000 0x00 0x400>,
>                  <0x00 0x0d000000 0x00 0x00800000>,
>                  <0x00 0x68000000 0x00 0x00001000>;
>
>            reg-names = "intd_cfg", "user_cfg", "reg", "cfg";
>      ... skip ...
>
>            #address-cells = <3>;
>            #size-cells = <2>;
>
>       ...skip ...
>
>            msi-map = <0x0 &gic_its 0x0 0x10000>;
>            ranges = <0x01000000 0x00 0x68001000  0x00 0x68001000  0x00
>0x0010000>,
>                 <0x02000000 0x00 0x68011000  0x00 0x68011000  0x00
>0x7fef000>;
>            dma-ranges = <0x02000000 0x0 0x0 0x0 0x0 0x00000010 0x0>;
>        };
>    };
>
>the 'ranges' property says
>- IO space starting at address 0x68001000 with size 64KiB is mapped to the
>parent bus address 0x68001000.
>- a 32 bit memory address space starting with address 0x68011000 and size
>0x7fef000 is mapped to parent bus address 0x68011000.
>
>The device tree document says these addresses are all physical.
>What I can't understand is :
>
>I know the BIOS will set the BAR addresses during the enumeration process.
>This is allocating the endpoint's function to a certain PCIe bus address
and
>I guess setting BAR can be done without this 'ranges' information(it will
>assign free address range for that device according to the device's need).
>By the time linux kernel later reads this 'ranges' property, the BIOS (or
>bootloader) has already assigned different PCIe address to that device.
>Then, how is this 'ranges' property used by the kernel?
>
>Thank you.
>
>Chan Kim
>
>
>
>
>
>_______________________________________________
>Kernelnewbies mailing list
>Kernelnewbies@kernelnewbies.org
>https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies





_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies