Problems with PCI on 8280

Problems with PCI on 8280

[Rune Torgersen]

[-- Attachment #1: Type: text/plain, Size: 3108 bytes --]

Hi.
 We have e 8280 bioard with PCI. We have three memory ranges set up;
0x8000_0000 to 0x8fff_ffff is prefetchable memory
0x9000_0000 to 0x97ff_ffff is non-prefetchable memory
0x9800_0000 to 0x9800_ffff is IO.

our device tree has
ranges = <42000000 0 80000000 80000000 0 10000000    // Pre-fetch memory
	  02000000 0 90000000 90000000 0 08000000    // Normal Memory
	  01000000 0 98000000 98000000 0 00010000>;  // I/O

for the memory ranges in the PCI node (full node at the end)

Our problem is that we can only access the first 128MB of the prefetchable area.
an access to 0x87ff_ffff works (after ioremap), while an access to 0x8800_0000 causes an bus error.

trying to access 0x87FFFFFC -> 0x00000000
trying to access 0x88000000 ->Machine check in kernel mode.
Caused by (from SRR1=49030): Transfer error ack signal

It is like the prefetch area is not set up correctly for ioremap.

The PCI registers are set up correctly in u-boot, and we can access the whole prefetch area in u-boot without any problems.

pci node in device tree:

	pci@f0010800 {
		device_type = "pci";
		reg = <f0010800 10c f00101ac 8 f00101c4 8>;
		compatible = "fsl,mpc8280-pci", "apmax-pci", "fsl,pq2-pci";
		#interrupt-cells = <1>;
		#size-cells = <2>;
		#address-cells = <3>;
		clock-frequency = <d#49766400>;               // For our board.
		interrupt-map-mask = <f800 0 0 7>;            // Anded with the interrupt-map values
		interrupt-map = <
		                /* IDSEL 0x11 */
		                00008800 0 0 1 &PCI_INT 5>;   // 3 first numbers pci device specifier are: (buss << 16 | id_sel << 11) 0 0
		                                              //    buss = 0, id_sel = 11
		                                              // Next number is interrupt 1-4 mapped A-D (interrupt A for us)
		                                              // Last two numbers are host interrupt specifier (external interrupt 5)

		//interrupt-parent = <&PIC>;
		interrupt-parent = <&PCI_INT>; 
		// interrupts = <12 8>;
               // Mem type 0 Bus Add  Loc. Add 0 Length      <<--- Remember the mem type is little endian
		ranges = <42000000 0 80000000 80000000 0 10000000    // Pre-fetch memory
		          02000000 0 90000000 90000000 0 08000000    // Normal Memory
		          01000000 0 98000000 98000000 0 00010000>;  // I/O
		          
        bus-range = <0 ff>;
        pci@12 {
    		device_type = "pci";
            reg = <9000 0 0 0 0>;
    		#interrupt-cells = <1>;
    		#size-cells = <2>;
    		#address-cells = <3>;
    		interrupt-parent = <&PCI_INT>; 
    		interrupt-map-mask = <0f800 0 0 7>;
    		interrupt-map = <
    		                /* IDSEL 0x10 */
    		                00000000 0 0 1 &PCI_INT 1

    		                /* IDSEL 0x11 */
    		                00000800 0 0 1 &PCI_INT 2
    		                
    		                /* IDSEL 0x12 */
    		                00001000 0 0 1 &PCI_INT 3
    		                
    		                /* IDSEL 0x13 */
    		                00001800 0 0 1 &PCI_INT 4 >;
        };
	};





[-- Attachment #2: Type: text/html, Size: 9417 bytes --]

Re: Problems with PCI on 8280

[Scott Wood]

Rune Torgersen wrote:
> Our problem is that we can only access the first 128MB of the 
> prefetchable area.
> an access to 0x87ff_ffff works (after ioremap), while an access to 
> 0x8800_0000 causes an bus error.
> 
> trying to access 0x87FFFFFC -> 0x00000000
> trying to access 0x88000000 ->Machine check in kernel mode.
> Caused by (from SRR1=49030): Transfer error ack signal
> 
> It is like the prefetch area is not set up correctly for ioremap.
> 
> The PCI registers are set up correctly in u-boot, and we can access the 
> whole prefetch area in u-boot without any problems.

Are you using cuImage, or a regular uImage with a device-tree-aware 
u-boot?  If the former, try commenting out the call to fixup_pci() in 
arch/powerpc/boot/cuboot-pq2.c and let me know if that changes anything.

Otherwise, are you sure the kernel didn't move BARs around such that 
there's no longer a PCI device mapped at 0x88000000 (or that the device 
that is there has been disabled)?

Can you show the contents of the following big endian registers:
TESCR1 (0x10040), TESCR2 (0x10044), PCIBRx (0x101ac, 0x101b0), and 
PCIMSKx (0x101c4, 0x101c8)

and the following little endian registers:
POTARx (0x10800, 0x10818, 0x10830), POBARx (0x10808, 0x10820, 0x10838), 
POCMRx (0x10810, 0x10828, 0x10840), POESR (0x10084), PCI_EACR (0x10890), 
and PCI_ECCR (0x108a0)

after the machine check happens?

-Scott

RE: Problems with PCI on 8280

[Rune Torgersen]

[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]

>From: Scott Wood 
>Are you using cuImage, or a regular uImage with a device-tree-aware u-boot

cuImage for now.

>If the former, try commenting out the call to fixup_pci() in 
>arch/powerpc/boot/cuboot-pq2.c and let me know if that changes anything.

Did that, That made our PCI bus fail completely

Looking in fixup_pci, I see it sets up one outbound window to the size of prefechable and non-prefetcabe memory
and anoter one to the size of the IO.

We have our PCI set up to ONE 512MB outbound range doing all three. 
(I can see where you'd save on mem usage by doing two)

I do see a bug tho.
In our case prefetch is 256MB, memio is 128MB.
the calculation for outbound wiondow 1 sets the size to ~(pref+mmio-1). That only works if the resulting size is a power of 2.

If we comment out the rewrite of the outbound windows, we get PCI to work.
But since our u-boot sets everything up correctly (Including prefecable memory) we should not need this? or is it rewriting the device tree in some way?




[-- Attachment #2: Type: text/html, Size: 1509 bytes --]

Re: Problems with PCI on 8280

[Scott Wood]

Rune Torgersen wrote:
>  >From: Scott Wood
>  >Are you using cuImage, or a regular uImage with a device-tree-aware u-boot
> 
> cuImage for now.
> 
>  >If the former, try commenting out the call to fixup_pci() in
>  >arch/powerpc/boot/cuboot-pq2.c and let me know if that changes anything.
> 
> Did that, That made our PCI bus fail completely

Hmm...  that suggests that something in u-boot's setup is either 
incorrect, or (more likely in this case) doesn't match the device tree.

Or maybe it's the bus parking/arbiter tweaking at the end that u-boot is 
missing.

> We have our PCI set up to ONE 512MB outbound range doing all three.
> (I can see where you'd save on mem usage by doing two)
> 
> I do see a bug tho.
> In our case prefetch is 256MB, memio is 128MB.
> the calculation for outbound wiondow 1 sets the size to ~(pref+mmio-1). 
> That only works if the resulting size is a power of 2.

Ah.  Yes, I was assuming both windows would be the same size.  The code 
should be changed to check that, and if any conditions for setting 
PCIBRx/PCIMSKx fail, check to see if the firmware-provided values work 
(and if not, bail without touching anything).

> If we comment out the rewrite of the outbound windows, we get PCI to work.
> But since our u-boot sets everything up correctly (Including prefecable 
> memory) we should not need this? or is it rewriting the device tree in 
> some way?

You shouldn't need it...  I'd compare the register values u-boot sets 
with the values the corrected fixup_pci() sets (including the soc regs 
at the end).  Something must be different.

There is no device tree re-writing; it tries to program the hardware to 
match what's in the device tree.

-Scott

RE: Problems with PCI on 8280

[Rune Torgersen]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

From: Scott Wood

>Hmm...  that suggests that something in u-boot's setup is either 
>incorrect, or (more likely in this case) doesn't match the device tree.

Turns out u-boot was not setting u the inbound window size correctly.
It was hardcoded to 512MB and we have a gig of ram. Set it correctly, and CPI now works fully even without fixup_pci()

>Ah.  Yes, I was assuming both windows would be the same size.  The code 
>should be changed to check that, and if any conditions for setting 
>PCIBRx/PCIMSKx fail, check to see if the firmware-provided values work 
>(and if not, bail without touching anything).

yup. In our case we only need one window.

BTW. there is a bug in the inbound window size calculation.
The mem_log2 variable sould be the shift value, not 1<< shift.
and on the next line the window size mask sould be anded with 0x000f_ffff before or'ed with 0xa000_0000


[-- Attachment #2: Type: text/html, Size: 1386 bytes --]

Re: Problems with PCI on 8280

[Scott Wood]

On Thu, Jan 17, 2008 at 06:27:46PM -0600, Rune Torgersen wrote:
> BTW. there is a bug in the inbound window size calculation. The mem_log2
> variable sould be the shift value, not 1<< shift. and on the next line the
> window size mask sould be anded with 0x000f_ffff before or'ed with
> 0xa000_0000

D'oh.  Thanks for spotting it!

-Scott