Freescale P2020 / 85xx PCIe and Advance Error Reporting (AER) service problem

Freescale P2020 / 85xx PCIe and Advance Error Reporting (AER) service problem

[Eran Liberty]

Dear Penguins,

SHORT:
There is a BUG in the current code design / Freescale P2020/85xx PCIe design that prevent it from registering to the PCIe AER... or that I have missed something :) ..

LESS SHORT:
I am in the process of a Freescale P2020 based board bring up. P2020 is basically two 85xx processors and their peripherals share most features.

PCIe has a very extensive error reporting section and the Kernel already has a very nice looking Advanced Error Reporting driver.

I encounter difficulties trying to connect the P2020/85xx PCIe device to this AER service driver.

My technical findings follows:

 - pcie_portdrv_probe() will be called for every BRIDGE class PCI device. P2020 PCIe is a PCI-PCI BRIDGE class so no problem here. 
 - The code will continue to check that we have PCI_CAP_ID_EXP capability, which we have and continue to pcie_port_device_register().
 - Now ,the function pcie_port_device_register() will FAIL. It will fail because it will call assign_interrupt_mode(), return with PCIE_PORT_NO_IRQ, and giveup with a reasonable remark in the code
"/*
  * Don't use service devices that require interrupts if there is
  * no way to generate them.
  */"

So now the question is why calling assign_interrupt_mode() with the P2020 PCIe ROOT device return empty? Well...
 - First assign_interrupt_mode() will test for PCIE_PORT_MSIX_MODE. Freescale PCIe does not support this...
 - Second attampt is made to discover PCIE_PORT_MSI_MODE, which Freescale should support but the PCIe PCI_CAP_ID_MSI capability is published on the device side of the bridge and NOT on the PCIe ROOT device, which is the one probed and thus fails.
 - Last it attempts to look at "dev->pin" in order to set PCIE_PORT_INTx_MODE. On top of being the less recommended way (the old way), The Freescale PCIE ROOT device pin is not set anywhere.

Failing all those the probe fails and the AER service is not activated for the PCIE device.

QUESTION:
1. What am I missing?
2. Has anyone enabled the AER PCIe service for P2020/MPC85xx?
3. Should the PCIe ROOT end report MSI capabilities or should the device end report itself as bridge ???

-- Liberty

Re: Freescale P2020 / 85xx PCIe and Advance Error Reporting (AER) service problem

[Kumar Gala]

On Oct 7, 2010, at 7:30 AM, Eran Liberty wrote:

> Dear Penguins,
>=20
> SHORT:
> There is a BUG in the current code design / Freescale P2020/85xx PCIe =
design that prevent it from registering to the PCIe AER... or that I =
have missed something :) ..
>=20
> LESS SHORT:
> I am in the process of a Freescale P2020 based board bring up. P2020 =
is basically two 85xx processors and their peripherals share most =
features.
>=20
> PCIe has a very extensive error reporting section and the Kernel =
already has a very nice looking Advanced Error Reporting driver.
>=20
> I encounter difficulties trying to connect the P2020/85xx PCIe device =
to this AER service driver.
>=20
> My technical findings follows:
>=20
> - pcie_portdrv_probe() will be called for every BRIDGE class PCI =
device. P2020 PCIe is a PCI-PCI BRIDGE class so no problem here. - The =
code will continue to check that we have PCI_CAP_ID_EXP capability, =
which we have and continue to pcie_port_device_register().
> - Now ,the function pcie_port_device_register() will FAIL. It will =
fail because it will call assign_interrupt_mode(), return with =
PCIE_PORT_NO_IRQ, and giveup with a reasonable remark in the code
> "/*
> * Don't use service devices that require interrupts if there is
> * no way to generate them.
> */"
>=20
> So now the question is why calling assign_interrupt_mode() with the =
P2020 PCIe ROOT device return empty? Well...
> - First assign_interrupt_mode() will test for PCIE_PORT_MSIX_MODE. =
Freescale PCIe does not support this...
> - Second attampt is made to discover PCIE_PORT_MSI_MODE, which =
Freescale should support but the PCIe PCI_CAP_ID_MSI capability is =
published on the device side of the bridge and NOT on the PCIe ROOT =
device, which is the one probed and thus fails.
> - Last it attempts to look at "dev->pin" in order to set =
PCIE_PORT_INTx_MODE. On top of being the less recommended way (the old =
way), The Freescale PCIE ROOT device pin is not set anywhere.
>=20
> Failing all those the probe fails and the AER service is not activated =
for the PCIE device.
>=20
> QUESTION:
> 1. What am I missing?
> 2. Has anyone enabled the AER PCIe service for P2020/MPC85xx?
> 3. Should the PCIe ROOT end report MSI capabilities or should the =
device end report itself as bridge ???
>=20
> -- Liberty

Do you have some code that enables AER on P2020.  If so it might be =
easier to see what's going on.

- k

Re: Freescale P2020 / 85xx PCIe and Advance Error Reporting (AER) service problem

[Eran Liberty]

Kumar Gala wrote:
> On Oct 7, 2010, at 7:30 AM, Eran Liberty wrote:
>
>   
>> Dear Penguins,
>>
>> SHORT:
>> There is a BUG in the current code design / Freescale P2020/85xx PCIe design that prevent it from registering to the PCIe AER... or that I have missed something :) ..
>>
>> LESS SHORT:
>> I am in the process of a Freescale P2020 based board bring up. P2020 is basically two 85xx processors and their peripherals share most features.
>>
>> PCIe has a very extensive error reporting section and the Kernel already has a very nice looking Advanced Error Reporting driver.
>>
>> I encounter difficulties trying to connect the P2020/85xx PCIe device to this AER service driver.
>>
>> My technical findings follows:
>>
>> - pcie_portdrv_probe() will be called for every BRIDGE class PCI device. P2020 PCIe is a PCI-PCI BRIDGE class so no problem here. - The code will continue to check that we have PCI_CAP_ID_EXP capability, which we have and continue to pcie_port_device_register().
>> - Now ,the function pcie_port_device_register() will FAIL. It will fail because it will call assign_interrupt_mode(), return with PCIE_PORT_NO_IRQ, and giveup with a reasonable remark in the code
>> "/*
>> * Don't use service devices that require interrupts if there is
>> * no way to generate them.
>> */"
>>
>> So now the question is why calling assign_interrupt_mode() with the P2020 PCIe ROOT device return empty? Well...
>> - First assign_interrupt_mode() will test for PCIE_PORT_MSIX_MODE. Freescale PCIe does not support this...
>> - Second attampt is made to discover PCIE_PORT_MSI_MODE, which Freescale should support but the PCIe PCI_CAP_ID_MSI capability is published on the device side of the bridge and NOT on the PCIe ROOT device, which is the one probed and thus fails.
>> - Last it attempts to look at "dev->pin" in order to set PCIE_PORT_INTx_MODE. On top of being the less recommended way (the old way), The Freescale PCIE ROOT device pin is not set anywhere.
>>
>> Failing all those the probe fails and the AER service is not activated for the PCIE device.
>>
>> QUESTION:
>> 1. What am I missing?
>> 2. Has anyone enabled the AER PCIe service for P2020/MPC85xx?
>> 3. Should the PCIe ROOT end report MSI capabilities or should the device end report itself as bridge ???
>>
>> -- Liberty
>>     
>
> Do you have some code that enables AER on P2020.  If so it might be easier to see what's going on.
>
>
>   
> - k
>
>
The code that enables the AER is the problem. The kernel fails to probe 
my devices into the AER sevice. I can brutally hack my kernel to 
register the AER service. I copy the dev->irq and dev->pin from the 
child device to the bridge device. This will get the bridge device 
registered BUT it is not working and this hack is surly not a solution.

Here is a snap shot of my un-hacked devices, hopefully it will clarify 
the situation:

~ # ls -la /sys/bus/pci/devices/
drwxr-xr-x    2 root     root            0 Jan  3 17:25 .
drwxr-xr-x    5 root     root            0 Jan  3 17:25 ..
lrwxrwxrwx    1 root     root            0 Jan  3 17:25 0001:01:00.0 -> 
../../../devices/pci0001:01/0001:01:00.0
lrwxrwxrwx    1 root     root            0 Jan  3 17:25 0001:02:00.0 -> 
../../../devices/pci0001:01/0001:01:00.0/0001:02:00.0

"0001:01:00.0" is the PCIe PCI-PCI bridge side (Freescale's side)
~ # cat /sys/bus/pci/devices/0001\:01\:00.0/vendor
0x1957 (PCI_VENDOR_ID_FREESCALE)
~ # cat /sys/bus/pci/devices/0001\:01\:00.0/device
0x0071 (PCI_DEVICE_ID_P2020)
~ # cat /sys/bus/pci/devices/0001\:01\:00.0/class
0x060400 (PCI_CLASS_BRIDGE_PCI)
~ # cat /sys/bus/pci/devices/0001\:01\:00.0/enable
1
~ # cat /sys/bus/pci/devices/0001\:01\:00.0/irq
0
~ # cat /sys/bus/pci/devices/0001\:01\:00.0/msi_bus
1
~ # cat /sys/bus/pci/devices/0001\:01\:00.0/resource
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x000000000000ffff 0x0000000000000100
0x00000000c0000000 0x00000000dfffffff 0x0000000000000200
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
~ # hexdump  /sys/bus/pci/devices/0001\:01\:00.0/config
0000000 5719 7100 0601 1000 1000 200b 0800 0100 (PCI Compatible 
Configuration Headers)
0000010 0000 f0ff 0000 0000 0002 0200 0000 0000
0000020 00c0 f0df f1ff 0100 0000 0000 0000 0000
0000030 0000 0000 4400 0000 0000 0000 0000 0000
0000040 0000 0000 014c 02fe 0000 0000 1000 4100 (Power Mgmt 
Capabilities) , (0x4C: PCI Express Capabilities)
0000050 0100 0000 3e58 0000 41d4 0300 0800 4100
0000060 c007 0000 c003 4000 0000 0000 0000 0000
0000070 0000 0000 0000 0000 0000 0000 0000 0000
*
0000100 0100 0100 0000 0000 0000 0000 1020 0600 (Advanced Error 
Reporting Capability)
0000110 0000 0000 0000 0000 a000 0000 0000 0000
0000120 0000 0000 0000 0000 0000 0000 0000 0000
*
0000400 0000 0000 1600 0000 e204 0000 0000 0000 (PCI Express Controller 
Internal CSRs)
0000410 0800 0000 0000 0000 0000 0000 4040 0000
0000420 0000 0000 0000 0000 0000 0000 0000 0000
0000430 0000 0000 0000 0000 8300 6300 9ba0 0900
0000440 1000 0000 0000 0000 0000 0000 0000 0000
0000450 ced7 1400 201e fc01 0000 0000 5c0c 0000
0000460 0000 0000 0000 0000 0000 0000 0000 0000
0000470 5719 7100 1000 200b 0000 0000 0100 0000
0000480 443d 0000 0000 0000 f007 0000 0000 0000
0000490 c007 0000 0000 0000 0000 0000 0000 0000
00004a0 0000 0000 0000 0000 0000 0000 0000 0000
00004b0 0000 0000 0000 0000 2804 0180 8520 0000
00004c0 ff00 0000 0000 0000 0000 0000 1100 0000
00004d0 0000 0000 0000 0000 0000 0000 0000 0000
00004e0 0000 0000 0101 0000 0101 0000 0000 0000
00004f0 4a00 0001 0200 0004 0000 0000 0800 0000
0000500 0000 0000 0000 0000 0000 0000 0000 0000
0000510 0000 0000 0000 0000 8300 0000 0000 0000
0000520 0000 0000 0000 0000 0000 0000 0000 0000
*
0000540 0000 0000 0001 0202 0000 0000 0500 0000
0000550 0000 0000 0000 0000 0000 0000 0000 0000
*
0000590 d0cf 3200 0000 0000 0000 0000 0000 0000
00005a0 3f00 0000 0000 0000 0000 0000 0000 0000
00005b0 0000 0000 0000 0000 0000 0000 0000 0000
*
0001000

Now here is the other side of the tunnel, "0001:02:00.0" (our device)
~ # cat /sys/bus/pci/devices/0001\:02\:00.0/vendor
0x1234 (Just a number i picked up and told the HW dudes to put into the 
device... don't look it up :) )
~ # cat /sys/bus/pci/devices/0001\:02\:00.0/device
0x0002
~ # cat /sys/bus/pci/devices/0001\:02\:00.0/class
0xff0000
~ # cat /sys/bus/pci/devices/0001\:02\:00.0/enable
0
~ # cat /sys/bus/pci/devices/0001\:02\:00.0/irq
16
~ # cat /sys/bus/pci/devices/0001\:02\:00.0/msi_bus
~ # cat /sys/bus/pci/devices/0001\:02\:00.0/resource
0x00000000c0000000 0x00000000c007ffff 0x0000000000020200
0x00000000c0080000 0x00000000c00fffff 0x0000000000020200
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
~ # hexdump /sys/bus/pci/devices/0001\:02\:00.0/config
0000000 3412 0200 0600 1000 0100 00ff 0800 0000
0000010 0000 00c0 0000 08c0 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 3412 0200
0000030 0000 0000 5000 0000 0000 0000 0001 0000
0000040 0000 0000 6061 0002 0000 0000 0000 0000
0000050 0578 8400 0000 0000 0000 0000 0000 0000
0000060 0000 0000 0000 0000 1178 0000 0000 0000
0000070 0000 0000 0000 0000 0180 0300 0800 0000
0000080 1000 0100 0180 3c00 3058 0000 41f4 0301
0000090 0000 4110 0000 0400 c003 0000 0000 0000
00000a0 0000 0000 0000 0000 0000 0000 0000 0000
00000b0 0100 0100 0000 0000 0000 0000 0000 0000
00000c0 0000 0000 0000 0000 0000 0000 0000 0000
*
0000100 0200 0180 0000 0000 0000 0000 0000 0000
0000110 0000 0000 ff00 0080 0000 0000 0000 0000
0000120 0000 0000 0000 0000 0000 0000 0000 0000
*
0000800 0100 0100 0000 0000 0000 0000 3120 0600
0000810 0000 0000 0020 0000 0000 0000 0000 0000
0000820 0000 0000 0000 0000 0000 0000 0000 0000
*
0001000

NOW... As it is, under "/sys/bus/pci_express/devices/" you will find no 
devices. Debug prints which I have stuffed into pcie_portdrv_probe() 
shows that: "0001:01:00" is probed and fails to register for lacking a 
suitable interrupt mechanism as described by the previous mail, and 
"0001:02:00" fails to probe since it is not a PCI ROOT port.
As said, if I hack and slash and mainly cheat I can persuade the Kernel 
not to fail the probe and then I get these:

~ # ls -la /sys/bus/pci_express/devices/
drwxr-xr-x    2 root     root            0 Jan  3 17:49 .
drwxr-xr-x    4 root     root            0 Jan  3 17:49 ..
lrwxrwxrwx    1 root     root            0 Jan  3 17:49 
0001:01:00.0:pcie01 -> 
../../../devices/pci0001:01/0001:01:00.0/0001:01:00.0:pcie01
lrwxrwxrwx    1 root     root            0 Jan  3 17:49 
0001:01:00.0:pcie02 -> 
../../../devices/pci0001:01/0001:01:00.0/0001:01:00.0:pcie02

This is NOT functional as I have crippled the kernel on the way but I 
think this is  what I am missing and maybe my goal.

-- Liberty

Re: Freescale P2020 / 85xx PCIe and Advance Error Reporting (AER) service problem

[Benjamin Herrenschmidt]

>  - pcie_portdrv_probe() will be called for every BRIDGE class PCI device. P2020 PCIe is a PCI-PCI BRIDGE class so no problem here. 
>  - The code will continue to check that we have PCI_CAP_ID_EXP capability, which we have and continue to pcie_port_device_register().
>  - Now ,the function pcie_port_device_register() will FAIL. It will fail because it will call assign_interrupt_mode(), return with PCIE_PORT_NO_IRQ, and giveup with a reasonable remark in the code
> "/*
>   * Don't use service devices that require interrupts if there is
>   * no way to generate them.
>   */"
> 
> So now the question is why calling assign_interrupt_mode() with the P2020 PCIe ROOT device return empty? Well...
>  - First assign_interrupt_mode() will test for PCIE_PORT_MSIX_MODE. Freescale PCIe does not support this...
>  - Second attampt is made to discover PCIE_PORT_MSI_MODE, which Freescale should support but the PCIe PCI_CAP_ID_MSI capability is published on the device side of the bridge and NOT on the PCIe ROOT device, which is the one probed and thus fails.
>  - Last it attempts to look at "dev->pin" in order to set PCIE_PORT_INTx_MODE. On top of being the less recommended way (the old way), The Freescale PCIE ROOT device pin is not set anywhere.
> 
> Failing all those the probe fails and the AER service is not activated for the PCIE device.

So the question boils down to how does the bridge generate the AER
interrupts. This should be documented in the FSL docs no ? The MSI in
the child/device should be unrelated (it's your device MSI) no ? So the
question is where's the missing interrupt.

If it's a SoC interrupt, coming from the device-tree, then perhaps the
generic AER code should be extended to recognize those.

Cheers,
Ben.

Re: Freescale P2020 / 85xx PCIe and Advance Error Reporting (AER) service problem

[Eran Liberty]

Benjamin Herrenschmidt wrote:
>>  - pcie_portdrv_probe() will be called for every BRIDGE class PCI device. P2020 PCIe is a PCI-PCI BRIDGE class so no problem here. 
>>  - The code will continue to check that we have PCI_CAP_ID_EXP capability, which we have and continue to pcie_port_device_register().
>>  - Now ,the function pcie_port_device_register() will FAIL. It will fail because it will call assign_interrupt_mode(), return with PCIE_PORT_NO_IRQ, and giveup with a reasonable remark in the code
>> "/*
>>   * Don't use service devices that require interrupts if there is
>>   * no way to generate them.
>>   */"
>>
>> So now the question is why calling assign_interrupt_mode() with the P2020 PCIe ROOT device return empty? Well...
>>  - First assign_interrupt_mode() will test for PCIE_PORT_MSIX_MODE. Freescale PCIe does not support this...
>>  - Second attampt is made to discover PCIE_PORT_MSI_MODE, which Freescale should support but the PCIe PCI_CAP_ID_MSI capability is published on the device side of the bridge and NOT on the PCIe ROOT device, which is the one probed and thus fails.
>>  - Last it attempts to look at "dev->pin" in order to set PCIE_PORT_INTx_MODE. On top of being the less recommended way (the old way), The Freescale PCIE ROOT device pin is not set anywhere.
>>
>> Failing all those the probe fails and the AER service is not activated for the PCIE device.
>
> So the question boils down to how does the bridge generate the AER
> interrupts. This should be documented in the FSL docs no ? The MSI in
> the child/device should be unrelated (it's your device MSI) no ? So the
> question is where's the missing interrupt.
>
> If it's a SoC interrupt, coming from the device-tree, then perhaps the
> generic AER code should be extended to recognize those.
>
> Cheers,
> Ben.
>
I agree...

BUT if we take into consideration that:
1. Freescale is a serious dude in the hood and on the whole does a good 
job with its products and their Linux support.
2. The P2020 does state it has an MSI mechanism support (although one is 
not present as a PCIe capability header for some reason)
3. Errors in general and AER are major features in PCIe.
4. PCIe has been here quite a while and it is not new to Freescale or 
anyone else.
I am much more inclined to believe that I have missed something by a 
mile then that Freescale did. I just don't know what I am missing.

My device tree is a clone of "arch/ powerpc/ boot/ dts/ p2020rdb.dts"

It has a PCI node that looks like this:
----------------------------- snip -----------------------------
    pci0: pcie@ffe09000 {
        cell-index = <1>;
        compatible = "fsl,mpc8548-pcie";
        device_type = "pci";
        #interrupt-cells = <1>;
        #size-cells = <2>;
        #address-cells = <3>;
        reg = <0 0xffe09000 0 0x1000>;
        bus-range = <0 255>;
        ranges = <0x2000000 0x0 0xa0000000 0 0xa0000000 0x0 0x20000000
              0x1000000 0x0 0x00000000 0 0xffc30000 0x0 0x10000>;
        clock-frequency = <33333333>;
        interrupt-parent = <&mpic>;
        interrupts = <25 2>;
        interrupt-map-mask = <0xf800 0x0 0x0 0x7>;
        interrupt-map = <
            /* IDSEL 0x0 */
            0000 0x0 0x0 0x1 &mpic 0x4 0x1
            0000 0x0 0x0 0x2 &mpic 0x5 0x1
            0000 0x0 0x0 0x3 &mpic 0x6 0x1
            0000 0x0 0x0 0x4 &mpic 0x7 0x1
            >;
        pcie@0 {
            reg = <0x0 0x0 0x0 0x0 0x0>;
            #size-cells = <2>;
            #address-cells = <3>;
            device_type = "pci";
            ranges = <0x2000000 0x0 0xa0000000
                  0x2000000 0x0 0xa0000000
                  0x0 0x20000000

                  0x1000000 0x0 0x0
                  0x1000000 0x0 0x0
                  0x0 0x100000>;
        };
    };
----------------------------- snap -----------------------------

and under "soc" it has an MSI node that looks like that:
----------------------------- snip -----------------------------
        msi@41600 {
            compatible = "fsl,p2020-msi", "fsl,mpic-msi";
            reg = <0x41600 0x80>;
            msi-available-ranges = <0 0x100>;
            interrupts = <
                0xe0 0
                0xe1 0
                0xe2 0
                0xe3 0
                0xe4 0
                0xe5 0
                0xe6 0
                0xe7 0>;
            interrupt-parent = <&mpic>;
        };
----------------------------- snap -----------------------------

-- Liberty

Re: Freescale P2020 / 85xx PCIe and Advance Error Reporting (AER) service problem

[Benjamin Herrenschmidt]

> BUT if we take into consideration that:
> 1. Freescale is a serious dude in the hood and on the whole does a good 
> job with its products and their Linux support.

Sure but that's irrelevant to the technical problem at hand :-)

> 2. The P2020 does state it has an MSI mechanism support (although one is 
> not present as a PCIe capability header for some reason)

Then it's broken :-(

> 3. Errors in general and AER are major features in PCIe.
> 4. PCIe has been here quite a while and it is not new to Freescale or 
> anyone else.

Right but we don't do AER on ppc44x either, I know we should but for
some reason, AER hasn't been on anybody #1 priority list in embedded
world so far...

> I am much more inclined to believe that I have missed something by a 
> mile then that Freescale did. I just don't know what I am missing.

No, I think you haven't and we just need to fix it :-)

Cheers,
Ben.

> My device tree is a clone of "arch/ powerpc/ boot/ dts/ p2020rdb.dts"
> 
> It has a PCI node that looks like this:
> ----------------------------- snip -----------------------------
>     pci0: pcie@ffe09000 {
>         cell-index = <1>;
>         compatible = "fsl,mpc8548-pcie";
>         device_type = "pci";
>         #interrupt-cells = <1>;
>         #size-cells = <2>;
>         #address-cells = <3>;
>         reg = <0 0xffe09000 0 0x1000>;
>         bus-range = <0 255>;
>         ranges = <0x2000000 0x0 0xa0000000 0 0xa0000000 0x0 0x20000000
>               0x1000000 0x0 0x00000000 0 0xffc30000 0x0 0x10000>;
>         clock-frequency = <33333333>;
>         interrupt-parent = <&mpic>;
>         interrupts = <25 2>;
>         interrupt-map-mask = <0xf800 0x0 0x0 0x7>;
>         interrupt-map = <
>             /* IDSEL 0x0 */
>             0000 0x0 0x0 0x1 &mpic 0x4 0x1
>             0000 0x0 0x0 0x2 &mpic 0x5 0x1
>             0000 0x0 0x0 0x3 &mpic 0x6 0x1
>             0000 0x0 0x0 0x4 &mpic 0x7 0x1
>             >;
>         pcie@0 {
>             reg = <0x0 0x0 0x0 0x0 0x0>;
>             #size-cells = <2>;
>             #address-cells = <3>;
>             device_type = "pci";
>             ranges = <0x2000000 0x0 0xa0000000
>                   0x2000000 0x0 0xa0000000
>                   0x0 0x20000000
> 
>                   0x1000000 0x0 0x0
>                   0x1000000 0x0 0x0
>                   0x0 0x100000>;
>         };
>     };
> ----------------------------- snap -----------------------------
> 
> and under "soc" it has an MSI node that looks like that:
> ----------------------------- snip -----------------------------
>         msi@41600 {
>             compatible = "fsl,p2020-msi", "fsl,mpic-msi";
>             reg = <0x41600 0x80>;
>             msi-available-ranges = <0 0x100>;
>             interrupts = <
>                 0xe0 0
>                 0xe1 0
>                 0xe2 0
>                 0xe3 0
>                 0xe4 0
>                 0xe5 0
>                 0xe6 0
>                 0xe7 0>;
>             interrupt-parent = <&mpic>;
>         };
> ----------------------------- snap -----------------------------
> 
> -- Liberty
> 

Freescale P2020 CPU Freeze over PCIe abort signal

[Eran Liberty]

This should probably go to the Freescale support, as it feels like a 
hardware issue yet the end result is a very frozen Linux kernel so I 
post here first...

I have a programmable FPGA PCIe device connected to a Freescale's P2020 
PCIe port. As part of the bring-up tests, we are testing two faulty 
scenarios:
1. The FPGA totally ignores the PCIe transaction.
2. The FPGA return a transaction abort.

Both are plausible PCIe behavior and their should be outcome is 
documented in the PCIe spec. The first should be terminated by the 
transaction requestor timeout mechanism and raise an error, the second 
should abort the transaction and raise and error.

In P2020 if I do any of those the CPU is left hung over the transaction.

something like:
in_le32(addr)

is turned into:
7c 00 04 ac     sync   
7c 00 4c 2c     lwbrx   r0,0,r9
0c 00 00 00     twi     0,r0,0
4c 00 01 2c     isync

assembly code, where in r9 (in this example) hold an address which is 
physically mapped into the PCIe resource space.

The CPU will hang over the load instruction.

Just for the fun of it, I have wrote my own assembly function omitting 
everything but the load instruction; still freeze.
Replace "lwbrx" with a simple "lwz"; still freeze.

It looks like the CPU snoozes till the PCIe transaction is done with no 
timeouts, ignoring any abort signal.

I am going to:
A. Try to reach the Freescale support.
B. Asked the FPGA designed to give me a new behavior that will stall the 
PCIe transaction replay for 10 sec, but after those return ok.
C. report back here with either A or B.

If you have any ideas I would love to hear them.

-- Liberty

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Bin Meng]

On Mon, Oct 18, 2010 at 3:24 AM, Eran Liberty <liberty@extricom.com> wrote:
<snip>

> In P2020 if I do any of those the CPU is left hung over the transaction.
>
> something like:
> in_le32(addr)
>
> is turned into:
> 7c 00 04 ac =A0 =A0 sync =A0 7c 00 4c 2c =A0 =A0 lwbrx =A0 r0,0,r9
> 0c 00 00 00 =A0 =A0 twi =A0 =A0 0,r0,0
> 4c 00 01 2c =A0 =A0 isync
>
> assembly code, where in r9 (in this example) hold an address which is
> physically mapped into the PCIe resource space.
>
> The CPU will hang over the load instruction.
>
> Just for the fun of it, I have wrote my own assembly function omitting
> everything but the load instruction; still freeze.
> Replace "lwbrx" with a simple "lwz"; still freeze.
>
> It looks like the CPU snoozes till the PCIe transaction is done with no
> timeouts, ignoring any abort signal.
>

It sounds like a similar issue I got with the 83xx PCIe host controller.

If there is no valid link established under the PCIe host controller
(ie: no device connected), or the device under the PCIe host
controller does not respond the configuration access correctly,
the host will hang at the instruction of "lwz" when trying to access
the PCIe configuration space (in 83xx, it's memory mapped)

And I remember the same issue exists in 8548, not sure if it is fixed in P2=
020.

<snip>

Bin

> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
>

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[tiejun.chen]

Eran Liberty wrote:
> This should probably go to the Freescale support, as it feels like a
> hardware issue yet the end result is a very frozen Linux kernel so I
> post here first...
> 
> I have a programmable FPGA PCIe device connected to a Freescale's P2020
> PCIe port. As part of the bring-up tests, we are testing two faulty
> scenarios:
> 1. The FPGA totally ignores the PCIe transaction.
> 2. The FPGA return a transaction abort.
> 
> Both are plausible PCIe behavior and their should be outcome is
> documented in the PCIe spec. The first should be terminated by the
> transaction requestor timeout mechanism and raise an error, the second
> should abort the transaction and raise and error.
> 
> In P2020 if I do any of those the CPU is left hung over the transaction.
> 
> something like:
> in_le32(addr)
> 
> is turned into:
> 7c 00 04 ac     sync   7c 00 4c 2c     lwbrx   r0,0,r9
> 0c 00 00 00     twi     0,r0,0
> 4c 00 01 2c     isync
> 
> assembly code, where in r9 (in this example) hold an address which is
> physically mapped into the PCIe resource space.
> 
> The CPU will hang over the load instruction.
> 
> Just for the fun of it, I have wrote my own assembly function omitting
> everything but the load instruction; still freeze.
> Replace "lwbrx" with a simple "lwz"; still freeze.
> 
> It looks like the CPU snoozes till the PCIe transaction is done with no
> timeouts, ignoring any abort signal.
> 

AFAIK we can set one bit on PEX_ERR_DISR to detect PCI Express completion
time-out. If so one interrupt should be issued. But I'm not sure if this can fix
your issue.

Tiejun

> I am going to:
> A. Try to reach the Freescale support.
> B. Asked the FPGA designed to give me a new behavior that will stall the
> PCIe transaction replay for 10 sec, but after those return ok.
> C. report back here with either A or B.
> 
> If you have any ideas I would love to hear them.
> 
> -- Liberty
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
> 

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Eran Liberty]

tiejun.chen wrote:
> AFAIK we can set one bit on PEX_ERR_DISR to detect PCI Express completion
> time-out. If so one interrupt should be issued. But I'm not sure if this can fix
> your issue.
>
> Tiejun
>   
As I understand the problem, this will not help me as the CPU itself is 
on hold waiting for the assembly line to complete. It may give me enough 
to spot the faulty situation and crash the system... but I am aiming for 
an Enterprise grade product.  Crash/oops is not something I want to put 
into the system if I do not absolutely have to and I do have a hardware 
watch-dog that will pull me out if I do.

So just for the fun of it I am going to follow up on this PEX_ERR_DISR, 
but I do not see how it will help.

-- Liberty

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Eran Liberty]

Eran Liberty wrote:
> This should probably go to the Freescale support, as it feels like a 
> hardware issue yet the end result is a very frozen Linux kernel so I 
> post here first...
>
> I have a programmable FPGA PCIe device connected to a Freescale's 
> P2020 PCIe port. As part of the bring-up tests, we are testing two 
> faulty scenarios:
> 1. The FPGA totally ignores the PCIe transaction.
> 2. The FPGA return a transaction abort.
>
> Both are plausible PCIe behavior and their should be outcome is 
> documented in the PCIe spec. The first should be terminated by the 
> transaction requestor timeout mechanism and raise an error, the second 
> should abort the transaction and raise and error.
>
> In P2020 if I do any of those the CPU is left hung over the transaction.
>
> something like:
> in_le32(addr)
>
> is turned into:
> 7c 00 04 ac     sync   7c 00 4c 2c     lwbrx   r0,0,r9
> 0c 00 00 00     twi     0,r0,0
> 4c 00 01 2c     isync
>
> assembly code, where in r9 (in this example) hold an address which is 
> physically mapped into the PCIe resource space.
>
> The CPU will hang over the load instruction.
>
> Just for the fun of it, I have wrote my own assembly function omitting 
> everything but the load instruction; still freeze.
> Replace "lwbrx" with a simple "lwz"; still freeze.
>
> It looks like the CPU snoozes till the PCIe transaction is done with 
> no timeouts, ignoring any abort signal.
>
> I am going to:
> A. Try to reach the Freescale support.
> B. Asked the FPGA designed to give me a new behavior that will stall 
> the PCIe transaction replay for 10 sec, but after those return ok.
> C. report back here with either A or B.
>
> If you have any ideas I would love to hear them.
>
> -- Liberty
>
Some more info:

As said the the FPGA designer provided me a PCIe device that will stall 
its response to a variable amount of time. The CPU became un-frozen 
after this amount of time. More over, we have found that in that period 
till it un-froze the PCIe core did a retry to that transaction over and 
over every 40 ms. This gave me the bright idea to look for the word 
"retry" in the Freescale documentation which rewarded me with these 
registers:

------------------------------------------------------- snip 
-------------------------------------------------------
16.3.2.3        PCI Express Outbound Completion Timeout Register
                (PEX_OTB_CPL_TOR)
The PCI Express outbound completion timeout register, shown in Figure 
16-4, contains the maximum wait
time for a response to come back as a result of an outbound non-posted 
request before a timeout condition
occurs.
 Offset 
0x00C                                                                                                
Access: Read/Write
         0   1              5     7   
8                                                                                      
31
     R
        TD            
—                                                            TC
     W
 Reset 0     0  0  0   0   0   0  0   0   0   0   1    0  0   0  0    
1   1  1    1   1  1   1   1   1  1   1   1  1  1   1  1
            Figure 16-4. PCI Express Outbound Completion Timeout 
Register (PEX_OTB_CPL_TOR)
Table 16-6 describes the PCI Express outbound completion timeout 
register fields.
                                 Table 16-6. PEX_OTB_CPL_TOR Field 
Descriptions
  Bits     Name                                                     
Description
   0        TD     Timeout disable. This bit controls the 
enabling/disabling of the timeout function.
                   0 Enable completion timeout
                   1 Disable completion timeout
  1–7        —     Reserved
 8–31       TC     Timeout counter. This is the value that is used to 
load the response counter of the completion timeout.
                   One TC unit is 8× the PCI Express controller clock 
period; that is, one TC unit is 20 ns at 400 MHz, and 30
                   ns at 266.66 MHz.
                   The following are examples of timeout periods based 
on different TC settings:
                   0x00_0000 Reserved
                   0x10_FFFF 22.28 ms at 400 MHz controller clock; 33.34 
ms at 266.66 MHz controller clock
                   0xFF_FFFF 335.54 ms at 400 MHz controller clock; 
503.31 ms at 266.66 MHz controller clock


16.3.2.4       PCI Express Configuration Retry Timeout Register
               (PEX_CONF_RTY_TOR)
The PCI Express configuration retry timeout register, shown in Figure 
16-5, contains the maximum time
period during which retries of configuration transactions which resulted 
in a CRS response occur.
 Offset 
0x010                                                                               
Access: Read/Write
         0  1     3   
4                                                                                     
31
     R
        RD     —                                                 TC
     W
 Reset 0    0  0  0  0   1  0  0  0  0   0  0   0  0  0  0   1  1  1  1  
1  1   1   1 1 1   1  1   1 1   1  1
           Figure 16-5. PCI Express Configuration Retry Timeout Register 
(PEX_CONF_RTY_TOR)
                            QorIQ P2020 Integrated Processor Reference 
Manual, Rev. 0
16-12                                                                                   
Freescale Semiconductor
                                                                                                 
PCI Express Interface Controller
Table 16-7 describes the PCI Express configuration retry timeout 
register fields.
                            Table 16-7. PEX_CONF_RTY_TOR Field Descriptions
  Bits  Name                                                     Description
   0     RD    Retry disable. This bit disables the retry of a 
configuration transaction that receives a CRS status response
               packet.
               0 Enable retry of a configuration transaction in response 
to receiving a CRS status response until the timeout
                  counter (defined by the PEX_CONF_RTY_TOR[TC] field) 
has expired.
               1 Disable retry of a configuration transaction regardless 
of receiving a CRS status response.
  1–3     —    Reserved
 4–31    TC    Timeout counter. This is the value that is used to load 
the CRS response counter.
               One TC unit is 8× the PCI Express controller clock 
period; that is, one TC unit is 20 ns at 400 MHz and 30 ns
               at 266.66 MHz.
               Timeout period based on different TC settings:
               0x000_0000        Reserved
               0x400_FFFF        1.34 s at 400 MHz controller clock, 
2.02 s at 266.66 MHz controller clock
               0xFFF_FFFF        5.37 s at 400 MHz controller clock, 
8.05 s at 266.66 MHz controller clock
------------------------------------------------------- snap 
-------------------------------------------------------

Now this is all nice on the paper, but what the P2020 seems to be doing 
in reality is
1. never expire
2. do re-tries even in the non configuration access

I am going to try to disable completion timeout and see if I get better 
behavior.

-- Liberty

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Eran Liberty]

Eran Liberty wrote:
> Eran Liberty wrote:
>> This should probably go to the Freescale support, as it feels like a 
>> hardware issue yet the end result is a very frozen Linux kernel so I 
>> post here first...
>>
>> I have a programmable FPGA PCIe device connected to a Freescale's 
>> P2020 PCIe port. As part of the bring-up tests, we are testing two 
>> faulty scenarios:
>> 1. The FPGA totally ignores the PCIe transaction.
>> 2. The FPGA return a transaction abort.
>>
>> Both are plausible PCIe behavior and their should be outcome is 
>> documented in the PCIe spec. The first should be terminated by the 
>> transaction requestor timeout mechanism and raise an error, the 
>> second should abort the transaction and raise and error.
>>
>> In P2020 if I do any of those the CPU is left hung over the transaction.
>>
>> something like:
>> in_le32(addr)
>>
>> is turned into:
>> 7c 00 04 ac     sync   7c 00 4c 2c     lwbrx   r0,0,r9
>> 0c 00 00 00     twi     0,r0,0
>> 4c 00 01 2c     isync
>>
>> assembly code, where in r9 (in this example) hold an address which is 
>> physically mapped into the PCIe resource space.
>>
>> The CPU will hang over the load instruction.
>>
>> Just for the fun of it, I have wrote my own assembly function 
>> omitting everything but the load instruction; still freeze.
>> Replace "lwbrx" with a simple "lwz"; still freeze.
>>
>> It looks like the CPU snoozes till the PCIe transaction is done with 
>> no timeouts, ignoring any abort signal.
>>
>> I am going to:
>> A. Try to reach the Freescale support.
>> B. Asked the FPGA designed to give me a new behavior that will stall 
>> the PCIe transaction replay for 10 sec, but after those return ok.
>> C. report back here with either A or B.
>>
>> If you have any ideas I would love to hear them.
>>
>> -- Liberty
>>
> Some more info:
>
> As said the the FPGA designer provided me a PCIe device that will 
> stall its response to a variable amount of time. The CPU became 
> un-frozen after this amount of time. More over, we have found that in 
> that period till it un-froze the PCIe core did a retry to that 
> transaction over and over every 40 ms. This gave me the bright idea to 
> look for the word "retry" in the Freescale documentation which 
> rewarded me with these registers:
>
> ------------------------------------------------------- snip 
> -------------------------------------------------------
> 16.3.2.3        PCI Express Outbound Completion Timeout Register
>                (PEX_OTB_CPL_TOR)
> The PCI Express outbound completion timeout register, shown in Figure 
> 16-4, contains the maximum wait
> time for a response to come back as a result of an outbound non-posted 
> request before a timeout condition
> occurs.
> Offset 
> 0x00C                                                                                                
> Access: Read/Write
>         0   1              5     7   
> 8                                                                                      
> 31
>     R
>        TD            
> —                                                            TC
>     W
> Reset 0     0  0  0   0   0   0  0   0   0   0   1    0  0   0  0    
> 1   1  1    1   1  1   1   1   1  1   1   1  1  1   1  1
>            Figure 16-4. PCI Express Outbound Completion Timeout 
> Register (PEX_OTB_CPL_TOR)
> Table 16-6 describes the PCI Express outbound completion timeout 
> register fields.
>                                 Table 16-6. PEX_OTB_CPL_TOR Field 
> Descriptions
>  Bits     Name                                                     
> Description
>   0        TD     Timeout disable. This bit controls the 
> enabling/disabling of the timeout function.
>                   0 Enable completion timeout
>                   1 Disable completion timeout
>  1–7        —     Reserved
> 8–31       TC     Timeout counter. This is the value that is used to 
> load the response counter of the completion timeout.
>                   One TC unit is 8× the PCI Express controller clock 
> period; that is, one TC unit is 20 ns at 400 MHz, and 30
>                   ns at 266.66 MHz.
>                   The following are examples of timeout periods based 
> on different TC settings:
>                   0x00_0000 Reserved
>                   0x10_FFFF 22.28 ms at 400 MHz controller clock; 
> 33.34 ms at 266.66 MHz controller clock
>                   0xFF_FFFF 335.54 ms at 400 MHz controller clock; 
> 503.31 ms at 266.66 MHz controller clock
>
>
> 16.3.2.4       PCI Express Configuration Retry Timeout Register
>               (PEX_CONF_RTY_TOR)
> The PCI Express configuration retry timeout register, shown in Figure 
> 16-5, contains the maximum time
> period during which retries of configuration transactions which 
> resulted in a CRS response occur.
> Offset 
> 0x010                                                                               
> Access: Read/Write
>         0  1     3   
> 4                                                                                     
> 31
>     R
>        RD     —                                                 TC
>     W
> Reset 0    0  0  0  0   1  0  0  0  0   0  0   0  0  0  0   1  1  1  
> 1  1  1   1   1 1 1   1  1   1 1   1  1
>           Figure 16-5. PCI Express Configuration Retry Timeout 
> Register (PEX_CONF_RTY_TOR)
>                            QorIQ P2020 Integrated Processor Reference 
> Manual, Rev. 0
> 16-12                                                                                   
> Freescale Semiconductor
>                                                                                                 
> PCI Express Interface Controller
> Table 16-7 describes the PCI Express configuration retry timeout 
> register fields.
>                            Table 16-7. PEX_CONF_RTY_TOR Field 
> Descriptions
>  Bits  Name                                                     
> Description
>   0     RD    Retry disable. This bit disables the retry of a 
> configuration transaction that receives a CRS status response
>               packet.
>               0 Enable retry of a configuration transaction in 
> response to receiving a CRS status response until the timeout
>                  counter (defined by the PEX_CONF_RTY_TOR[TC] field) 
> has expired.
>               1 Disable retry of a configuration transaction 
> regardless of receiving a CRS status response.
>  1–3     —    Reserved
> 4–31    TC    Timeout counter. This is the value that is used to load 
> the CRS response counter.
>               One TC unit is 8× the PCI Express controller clock 
> period; that is, one TC unit is 20 ns at 400 MHz and 30 ns
>               at 266.66 MHz.
>               Timeout period based on different TC settings:
>               0x000_0000        Reserved
>               0x400_FFFF        1.34 s at 400 MHz controller clock, 
> 2.02 s at 266.66 MHz controller clock
>               0xFFF_FFFF        5.37 s at 400 MHz controller clock, 
> 8.05 s at 266.66 MHz controller clock
> ------------------------------------------------------- snap 
> -------------------------------------------------------
>
> Now this is all nice on the paper, but what the P2020 seems to be 
> doing in reality is
> 1. never expire
> 2. do re-tries even in the non configuration access
>
> I am going to try to disable completion timeout and see if I get 
> better behavior.
>
> -- Liberty
>
>
Disabling PEX_OTB_CPL_TOR,  PEX_CONF_RTY_TOR, or both yields the same 
behavior. The kernel freezes over the load command while the underlying 
hardware does PCIe transaction retries to infinity and beyond.

-- Liberty