PCI on 834x

PCI on 834x

[Gary Thomas]

Does anyone have experience setting up the PCI controller on
the MPC8349?  I have it running fine when my system has 128MB
or less of main memory, but when I have 256MB or more, it all
falls apart :-(

Any indication of the pertinent settings would be appreciated.

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PCI on 834x

[Scott Wood]

Gary Thomas wrote:
> Does anyone have experience setting up the PCI controller on
> the MPC8349?  I have it running fine when my system has 128MB
> or less of main memory, but when I have 256MB or more, it all
> falls apart :-(
> 
> Any indication of the pertinent settings would be appreciated.

Check the inbound PCI translation registers, especially PIWARn.

-Scott

Re: PCI on 834x

[Gary Thomas]

On 02/24/2010 11:48 AM, Scott Wood wrote:
> Gary Thomas wrote:
>> Does anyone have experience setting up the PCI controller on
>> the MPC8349? I have it running fine when my system has 128MB
>> or less of main memory, but when I have 256MB or more, it all
>> falls apart :-(
>>
>> Any indication of the pertinent settings would be appreciated.
>
> Check the inbound PCI translation registers, especially PIWARn.

Yes, I set it to 256MB (the size of RAM).
I also set PCI_AR0 to a 512MB window (when it's set to 256, I
get address [segmentation violation] errors when accessing my
devices that end up at the high end of the logical address space)
Finally, I set POCMR0 to a 512MB window (corresponding to PCI_AR0)

In this configuration, my device doesn't seem to be able to
access RAM.  If PIWAR1 is too small, the device just hangs.

=============================== 128MB setup =========================
PIWAR1 = 0xA0055000 | (27-1)
POCMR0 = 0x800F0000
PCIAR0 = 0x80000000 | (28-1)
00:00.0 Power PC: Unknown device 1957:0082 (rev 30)
         Flags: bus master, 66MHz, fast devsel, latency 0
         Memory at cc000000 (32-bit, non-prefetchable) [size=1M]
         Memory at c0000000 (64-bit, prefetchable) [size=128M]
         Memory at <unassigned> (64-bit, non-prefetchable)
         Capabilities: [48] #06 [0000]

00:0b.0 Mass storage controller: Silicon Image, Inc. SiI 3512 [SATALink/SATARaid] Serial ATA Controller (rev 01)
         Subsystem: Silicon Image, Inc. SiI 3512 SATALink Controller
         Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 22
         I/O ports at 1000 [size=8]
         I/O ports at 1008 [size=4]
         I/O ports at 1010 [size=8]
         I/O ports at 1018 [size=4]
         I/O ports at 1020 [size=16]
         Memory at cc180000 (32-bit, non-prefetchable) [size=512]
         [virtual] Expansion ROM at cc100000 [disabled] [size=512K]
         Capabilities: [60] Power Management version 2

00:0c.0 Display controller: Fujitsu Limited. Unknown device 2019 (rev 01)
         Flags: bus master, medium devsel, latency 0, IRQ 19
         Memory at c8000000 (32-bit, non-prefetchable) [size=64M]
=====================================================================

=============================== 256MB setup =========================
PIWAR1 = 0xA0055000 | (28-1)
POCMR0 = 0x800E0000
PCIAR0 = 0x80000000 | (29-1)
00:00.0 Power PC: Unknown device 1957:0082 (rev 30)
         Flags: bus master, 66MHz, fast devsel, latency 0
         Memory at d4000000 (32-bit, non-prefetchable) [size=1M]
         Memory at c0000000 (64-bit, prefetchable) [size=256M]
         Memory at <unassigned> (64-bit, non-prefetchable)
         Capabilities: [48] #06 [0000]

00:0b.0 Mass storage controller: Silicon Image, Inc. SiI 3512 [SATALink/SATARaid] Serial ATA Controller (rev 01)
         Subsystem: Silicon Image, Inc. SiI 3512 SATALink Controller
         Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 22
         I/O ports at 1000 [size=8]
         I/O ports at 1008 [size=4]
         I/O ports at 1010 [size=8]
         I/O ports at 1018 [size=4]
         I/O ports at 1020 [size=16]
         Memory at d4180000 (32-bit, non-prefetchable) [size=512]
         [virtual] Expansion ROM at d4100000 [disabled] [size=512K]
         Capabilities: [60] Power Management version 2

00:0c.0 Display controller: Fujitsu Limited. Unknown device 2019 (rev 01)
         Flags: bus master, medium devsel, latency 0, IRQ 19
         [virtual] Memory at d0000000 (32-bit, non-prefetchable) [size=64M]
=====================================================================

Any ideas what I might be missing?

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PCI on 834x

[Scott Wood]

Gary Thomas wrote:
> On 02/24/2010 11:48 AM, Scott Wood wrote:
>> Gary Thomas wrote:
>>> Does anyone have experience setting up the PCI controller on
>>> the MPC8349? I have it running fine when my system has 128MB
>>> or less of main memory, but when I have 256MB or more, it all
>>> falls apart :-(
>>>
>>> Any indication of the pertinent settings would be appreciated.
>>
>> Check the inbound PCI translation registers, especially PIWARn.
> 
> Yes, I set it to 256MB (the size of RAM).
> I also set PCI_AR0 to a 512MB window (when it's set to 256, I
> get address [segmentation violation] errors when accessing my
> devices that end up at the high end of the logical address space)
> Finally, I set POCMR0 to a 512MB window (corresponding to PCI_AR0)

I don't see what the outbound window size has to do with RAM size... are 
there any other differences between the 128MiB and 256MiB setups?  Same 
exact software?  What happens if you take the 256MiB setup and restrict 
the kernel to using the lower 128MiB ("mem=128M" on the kernel 
bootargs), with no other changes?  What happens if you use a 512MiB 
outbound window on the 128MiB setup?

Is this behavior consistent across multiple boards (i.e. could there be 
a hardware problem)?

Are there any errors indicated in the device, the PCI controller, the 
arbiter, etc?

-Scott

Re: PCI on 834x

[Gary Thomas]

On 02/24/2010 12:31 PM, Scott Wood wrote:
> Gary Thomas wrote:
>> On 02/24/2010 11:48 AM, Scott Wood wrote:
>>> Gary Thomas wrote:
>>>> Does anyone have experience setting up the PCI controller on
>>>> the MPC8349? I have it running fine when my system has 128MB
>>>> or less of main memory, but when I have 256MB or more, it all
>>>> falls apart :-(
>>>>
>>>> Any indication of the pertinent settings would be appreciated.
>>>
>>> Check the inbound PCI translation registers, especially PIWARn.
>>
>> Yes, I set it to 256MB (the size of RAM).
>> I also set PCI_AR0 to a 512MB window (when it's set to 256, I
>> get address [segmentation violation] errors when accessing my
>> devices that end up at the high end of the logical address space)
>> Finally, I set POCMR0 to a 512MB window (corresponding to PCI_AR0)
>
> I don't see what the outbound window size has to do with RAM size... are there any other differences between the 128MiB and 256MiB setups? Same exact software? What happens if you
> take the 256MiB setup and restrict the kernel to using the lower 128MiB ("mem=128M" on the kernel bootargs), with no other changes? What happens if you use a 512MiB outbound window
> on the 128MiB setup?

Yes, I'm using the exact same kernel with these two different PCI
setups (done by the boot loader).

Restricting the memory via mem=128M has no effect - the PCI layout is the same.

I think the outbound window size is required because of how the Linux PCI
remaps the space (note in my dumps that it put the MMIO of the boards starting
at 0xD0000000 when the inbound window is 0x10000000)

> Is this behavior consistent across multiple boards (i.e. could there be a hardware problem)?

I've had this same problem on many platforms before (834x and 8248 as well).  In the
past, I didn't have the time to spend fixing it, so I just worked around it by limiting
the system to 128MB  in the boot loader + PCI setup)

> Are there any errors indicated in the device, the PCI controller, the arbiter, etc?

None that are obvious to me.

To be clear, here are my observations:

* total memory 128MB (or less), inbound window 128MB, outbound window 128MB
   ==> device is happy
* total memory 256MB, inbound window 128MB, outbound window 128MB
   ==> device hangs, unable to access RAM (Linux handed it something higher than 128MB)
* total memory 256MB, inbound window 256MB, outbound window 128MB
   ==> segmentation violation when accessing device MMIO space
* total memory 256MB, inbound window 256MB, outbound window 256MB
   ==> access to MMIO space fails (Linux can't find device).  In this case, the SATA_SIL reports
          ata1: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
          ata2: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
       which implies that the driver get's junk when accessing the device MMIO space

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PCI on 834x

[Scott Wood]

Gary Thomas wrote:
> Yes, I'm using the exact same kernel with these two different PCI
> setups (done by the boot loader).
> 
> Restricting the memory via mem=128M has no effect - the PCI layout is 
> the same.
> 
> I think the outbound window size is required because of how the Linux PCI
> remaps the space (note in my dumps that it put the MMIO of the boards 
> starting
> at 0xD0000000 when the inbound window is 0x10000000)

I see where the amount of RAM is mattering -- Linux is assigning 
outbound I/O space to the PCI controller itself (device 00:00.0) and the 
amount that it asks for seems to differ based on memory size.  Linux 
ought to skip that device when assigning resources.  Some platforms do 
this (search for pci_exclude_device), but it seems to be missing on 83xx.

>> Are there any errors indicated in the device, the PCI controller, the 
>> arbiter, etc?
> 
> None that are obvious to me.
> 
> To be clear, here are my observations:
> 
> * total memory 128MB (or less), inbound window 128MB, outbound window 128MB
>   ==> device is happy
> * total memory 256MB, inbound window 128MB, outbound window 128MB
>   ==> device hangs, unable to access RAM (Linux handed it something 
> higher than 128MB)

Linux gets its idea of how much space there is to allocate from the 
ranges property of the pci node in the device tree -- this should be 
made to match what u-boot configures in PIWARn.

> * total memory 256MB, inbound window 256MB, outbound window 128MB
>   ==> segmentation violation when accessing device MMIO space
> * total memory 256MB, inbound window 256MB, outbound window 256MB
>   ==> access to MMIO space fails (Linux can't find device).  In this 
> case, the SATA_SIL reports
>          ata1: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
>          ata2: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)
>       which implies that the driver get's junk when accessing the device 
> MMIO space

Are there any other relevant messages earlier in boot, perhaps from the 
PCI code failing to assign a resource?

-Scott

Re: PCI on 834x

[Anton Vorontsov]

On Wed, Feb 24, 2010 at 02:26:20PM -0600, Scott Wood wrote:
> Gary Thomas wrote:
> >Yes, I'm using the exact same kernel with these two different PCI
> >setups (done by the boot loader).
> >
> >Restricting the memory via mem=128M has no effect - the PCI layout
> >is the same.
> >
> >I think the outbound window size is required because of how the Linux PCI
> >remaps the space (note in my dumps that it put the MMIO of the
> >boards starting
> >at 0xD0000000 when the inbound window is 0x10000000)
> 
> I see where the amount of RAM is mattering -- Linux is assigning
> outbound I/O space to the PCI controller itself (device 00:00.0) and
> the amount that it asks for seems to differ based on memory size.
> Linux ought to skip that device when assigning resources.  Some
> platforms do this (search for pci_exclude_device), but it seems to
> be missing on 83xx.

Actually, 83xx had these exclude_device hooks, but they were removed:

commit d8f1324a5063c833862328ceafabc53ac3cc4f71
Author: Kumar Gala <galak@kernel.crashing.org>
Date:   Wed Sep 12 22:14:10 2007 -0500

    [POWERPC] 83xx: Removed PCI exclude of PHB

    Now that the generic code doesn't assign resources for Freescale
    PHBs we dont have to explicitly exclude it.

    Signed-off-by: Kumar Gala <galak@kernel.crashing.org>


May be the generic code started to assign the resources again?

-- 
Anton Vorontsov
email: cbouatmailru@gmail.com
irc://irc.freenode.net/bd2

Re: PCI on 834x

[Gary Thomas]

On 02/24/2010 01:51 PM, Anton Vorontsov wrote:
> On Wed, Feb 24, 2010 at 02:26:20PM -0600, Scott Wood wrote:
>> Gary Thomas wrote:
>>> Yes, I'm using the exact same kernel with these two different PCI
>>> setups (done by the boot loader).
>>>
>>> Restricting the memory via mem=128M has no effect - the PCI layout
>>> is the same.
>>>
>>> I think the outbound window size is required because of how the Linux PCI
>>> remaps the space (note in my dumps that it put the MMIO of the
>>> boards starting
>>> at 0xD0000000 when the inbound window is 0x10000000)
>>
>> I see where the amount of RAM is mattering -- Linux is assigning
>> outbound I/O space to the PCI controller itself (device 00:00.0) and
>> the amount that it asks for seems to differ based on memory size.
>> Linux ought to skip that device when assigning resources.  Some
>> platforms do this (search for pci_exclude_device), but it seems to
>> be missing on 83xx.
>
> Actually, 83xx had these exclude_device hooks, but they were removed:
>
> commit d8f1324a5063c833862328ceafabc53ac3cc4f71
> Author: Kumar Gala<galak@kernel.crashing.org>
> Date:   Wed Sep 12 22:14:10 2007 -0500
>
>      [POWERPC] 83xx: Removed PCI exclude of PHB
>
>      Now that the generic code doesn't assign resources for Freescale
>      PHBs we dont have to explicitly exclude it.
>
>      Signed-off-by: Kumar Gala<galak@kernel.crashing.org>
>
>
> May be the generic code started to assign the resources again?
>

That cracked it; I re-enabled the exclusion of the bridge and now
it's all working fine.

Thanks for the help

Note: I'm working with a fairly old kernel, so these results would
have to be reworked against the latest.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PCI on 834x

[Kumar Gala]

On Feb 24, 2010, at 4:14 PM, Gary Thomas wrote:

> On 02/24/2010 01:51 PM, Anton Vorontsov wrote:
>> On Wed, Feb 24, 2010 at 02:26:20PM -0600, Scott Wood wrote:
>>> Gary Thomas wrote:
>>>> Yes, I'm using the exact same kernel with these two different PCI
>>>> setups (done by the boot loader).
>>>>=20
>>>> Restricting the memory via mem=3D128M has no effect - the PCI =
layout
>>>> is the same.
>>>>=20
>>>> I think the outbound window size is required because of how the =
Linux PCI
>>>> remaps the space (note in my dumps that it put the MMIO of the
>>>> boards starting
>>>> at 0xD0000000 when the inbound window is 0x10000000)
>>>=20
>>> I see where the amount of RAM is mattering -- Linux is assigning
>>> outbound I/O space to the PCI controller itself (device 00:00.0) and
>>> the amount that it asks for seems to differ based on memory size.
>>> Linux ought to skip that device when assigning resources.  Some
>>> platforms do this (search for pci_exclude_device), but it seems to
>>> be missing on 83xx.
>>=20
>> Actually, 83xx had these exclude_device hooks, but they were removed:
>>=20
>> commit d8f1324a5063c833862328ceafabc53ac3cc4f71
>> Author: Kumar Gala<galak@kernel.crashing.org>
>> Date:   Wed Sep 12 22:14:10 2007 -0500
>>=20
>>     [POWERPC] 83xx: Removed PCI exclude of PHB
>>=20
>>     Now that the generic code doesn't assign resources for Freescale
>>     PHBs we dont have to explicitly exclude it.
>>=20
>>     Signed-off-by: Kumar Gala<galak@kernel.crashing.org>
>>=20
>>=20
>> May be the generic code started to assign the resources again?
>>=20
>=20
> That cracked it; I re-enabled the exclusion of the bridge and now
> it's all working fine.
>=20
> Thanks for the help
>=20
> Note: I'm working with a fairly old kernel, so these results would
> have to be reworked against the latest.

Odd that the generic code isn't dealing with that for you.

- k=

Re: PCI on 834x

[Gary Thomas]

On 02/24/2010 03:25 PM, Kumar Gala wrote:
>
> On Feb 24, 2010, at 4:14 PM, Gary Thomas wrote:
>
>> On 02/24/2010 01:51 PM, Anton Vorontsov wrote:
>>> On Wed, Feb 24, 2010 at 02:26:20PM -0600, Scott Wood wrote:
>>>> Gary Thomas wrote:
>>>>> Yes, I'm using the exact same kernel with these two different PCI
>>>>> setups (done by the boot loader).
>>>>>
>>>>> Restricting the memory via mem=128M has no effect - the PCI layout
>>>>> is the same.
>>>>>
>>>>> I think the outbound window size is required because of how the Linux PCI
>>>>> remaps the space (note in my dumps that it put the MMIO of the
>>>>> boards starting
>>>>> at 0xD0000000 when the inbound window is 0x10000000)
>>>>
>>>> I see where the amount of RAM is mattering -- Linux is assigning
>>>> outbound I/O space to the PCI controller itself (device 00:00.0) and
>>>> the amount that it asks for seems to differ based on memory size.
>>>> Linux ought to skip that device when assigning resources.  Some
>>>> platforms do this (search for pci_exclude_device), but it seems to
>>>> be missing on 83xx.
>>>
>>> Actually, 83xx had these exclude_device hooks, but they were removed:
>>>
>>> commit d8f1324a5063c833862328ceafabc53ac3cc4f71
>>> Author: Kumar Gala<galak@kernel.crashing.org>
>>> Date:   Wed Sep 12 22:14:10 2007 -0500
>>>
>>>      [POWERPC] 83xx: Removed PCI exclude of PHB
>>>
>>>      Now that the generic code doesn't assign resources for Freescale
>>>      PHBs we dont have to explicitly exclude it.
>>>
>>>      Signed-off-by: Kumar Gala<galak@kernel.crashing.org>
>>>
>>>
>>> May be the generic code started to assign the resources again?
>>>
>>
>> That cracked it; I re-enabled the exclusion of the bridge and now
>> it's all working fine.
>>
>> Thanks for the help
>>
>> Note: I'm working with a fairly old kernel, so these results would
>> have to be reworked against the latest.
>
> Odd that the generic code isn't dealing with that for you.

Remember it's an old kernel (2.6.28), so who knows the status.
As I said, I'll revisit this when I move to a newer kernel.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PCI on 834x

[Gary Thomas]

On 02/24/2010 04:08 PM, Gary Thomas wrote:
> On 02/24/2010 03:25 PM, Kumar Gala wrote:
>>
>> On Feb 24, 2010, at 4:14 PM, Gary Thomas wrote:
>>
>>> On 02/24/2010 01:51 PM, Anton Vorontsov wrote:
>>>> On Wed, Feb 24, 2010 at 02:26:20PM -0600, Scott Wood wrote:
>>>>> Gary Thomas wrote:
>>>>>> Yes, I'm using the exact same kernel with these two different PCI
>>>>>> setups (done by the boot loader).
>>>>>>
>>>>>> Restricting the memory via mem=128M has no effect - the PCI layout
>>>>>> is the same.
>>>>>>
>>>>>> I think the outbound window size is required because of how the Linux PCI
>>>>>> remaps the space (note in my dumps that it put the MMIO of the
>>>>>> boards starting
>>>>>> at 0xD0000000 when the inbound window is 0x10000000)
>>>>>
>>>>> I see where the amount of RAM is mattering -- Linux is assigning
>>>>> outbound I/O space to the PCI controller itself (device 00:00.0) and
>>>>> the amount that it asks for seems to differ based on memory size.
>>>>> Linux ought to skip that device when assigning resources. Some
>>>>> platforms do this (search for pci_exclude_device), but it seems to
>>>>> be missing on 83xx.
>>>>
>>>> Actually, 83xx had these exclude_device hooks, but they were removed:
>>>>
>>>> commit d8f1324a5063c833862328ceafabc53ac3cc4f71
>>>> Author: Kumar Gala<galak@kernel.crashing.org>
>>>> Date: Wed Sep 12 22:14:10 2007 -0500
>>>>
>>>> [POWERPC] 83xx: Removed PCI exclude of PHB
>>>>
>>>> Now that the generic code doesn't assign resources for Freescale
>>>> PHBs we dont have to explicitly exclude it.
>>>>
>>>> Signed-off-by: Kumar Gala<galak@kernel.crashing.org>
>>>>
>>>>
>>>> May be the generic code started to assign the resources again?
>>>>
>>>
>>> That cracked it; I re-enabled the exclusion of the bridge and now
>>> it's all working fine.
>>>
>>> Thanks for the help
>>>
>>> Note: I'm working with a fairly old kernel, so these results would
>>> have to be reworked against the latest.
>>
>> Odd that the generic code isn't dealing with that for you.
>
> Remember it's an old kernel (2.6.28), so who knows the status.
> As I said, I'll revisit this when I move to a newer kernel.
>

I may have been too hasty pronouncing this fixed.  Indeed, the
SATA interface now works, but my video card (Fujitsu Coral-P)
does not work when it's mapped at the bottom of the PCI space :-(

With the bridge mapped, the video ends up at a non-zero address
(0xC8000000..0xCFFFFFFF).  If it gets mapped to 0xC0000000, it
fails to respond to MMIO accesses.

Any ideas how I might get around this?  Is there a way to force
the PCI allocator to start somewhere other than [relative] zero?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PCI on 834x

[Benjamin Herrenschmidt]

On Thu, 2010-02-25 at 07:25 -0700, Gary Thomas wrote:
> I may have been too hasty pronouncing this fixed.  Indeed, the
> SATA interface now works, but my video card (Fujitsu Coral-P)
> does not work when it's mapped at the bottom of the PCI space :-(
> 
> With the bridge mapped, the video ends up at a non-zero address
> (0xC8000000..0xCFFFFFFF).  If it gets mapped to 0xC0000000, it
> fails to respond to MMIO accesses.
> 
> Any ideas how I might get around this?  Is there a way to force
> the PCI allocator to start somewhere other than [relative] zero?

I'm not familiar with the way the FSL bridge works, but it would
be possible to invert MMIO and DMA on your PCI bus. IE. Have MMIO go
from 0....2G and DMA from 2G..4G for example. Provided the FSL bridge
can offset the DMA back down to 0 (memory). Can it ?

Cheers,
Ben.

Re: PCI on 834x

[Scott Wood]

Benjamin Herrenschmidt wrote:
> On Thu, 2010-02-25 at 07:25 -0700, Gary Thomas wrote:
>> I may have been too hasty pronouncing this fixed.  Indeed, the
>> SATA interface now works, but my video card (Fujitsu Coral-P)
>> does not work when it's mapped at the bottom of the PCI space :-(
>>
>> With the bridge mapped, the video ends up at a non-zero address
>> (0xC8000000..0xCFFFFFFF).  If it gets mapped to 0xC0000000, it
>> fails to respond to MMIO accesses.
>>
>> Any ideas how I might get around this?  Is there a way to force
>> the PCI allocator to start somewhere other than [relative] zero?
> 
> I'm not familiar with the way the FSL bridge works, but it would
> be possible to invert MMIO and DMA on your PCI bus. IE. Have MMIO go
> from 0....2G and DMA from 2G..4G for example. Provided the FSL bridge
> can offset the DMA back down to 0 (memory). Can it ?

It can, but I don't see how that would help, if the problem is that the 
video card doesn't like the low 30 bits of its MMIO address being zero.

Gary, can you check that the MMIO addresses are going to the PCI bus 
as-is, and aren't being translated down to zero?  I.e. POTARn should 
equal POBARn, and likewise in the device tree's pci node's ranges.

-Scott

Re: PCI on 834x

[Gary Thomas]

On 02/25/2010 02:03 PM, Scott Wood wrote:
> Benjamin Herrenschmidt wrote:
>> On Thu, 2010-02-25 at 07:25 -0700, Gary Thomas wrote:
>>> I may have been too hasty pronouncing this fixed. Indeed, the
>>> SATA interface now works, but my video card (Fujitsu Coral-P)
>>> does not work when it's mapped at the bottom of the PCI space :-(
>>>
>>> With the bridge mapped, the video ends up at a non-zero address
>>> (0xC8000000..0xCFFFFFFF). If it gets mapped to 0xC0000000, it
>>> fails to respond to MMIO accesses.
>>>
>>> Any ideas how I might get around this? Is there a way to force
>>> the PCI allocator to start somewhere other than [relative] zero?
>>
>> I'm not familiar with the way the FSL bridge works, but it would
>> be possible to invert MMIO and DMA on your PCI bus. IE. Have MMIO go
>> from 0....2G and DMA from 2G..4G for example. Provided the FSL bridge
>> can offset the DMA back down to 0 (memory). Can it ?
>
> It can, but I don't see how that would help, if the problem is that the video card doesn't like the low 30 bits of its MMIO address being zero.
>
> Gary, can you check that the MMIO addresses are going to the PCI bus as-is, and aren't being translated down to zero? I.e. POTARn should equal POBARn, and likewise in the device
> tree's pci node's ranges.

Hmm, that doesn't match with how I've always had this setup.  I have:
   POTAR0 = 0x00000000
   POTBR0 = 0x000C0000        (0xC0000000 >> 12)

My device tree mappings are:
   ranges = <0x02000000 0x0 0xC0000000 0xC0000000 0x0 0x10000000
             0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>

n.b. I don't run U-Boot on these platforms (being the author
of RedBoot and all... :-)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PCI on 834x

[Scott Wood]

Gary Thomas wrote:
> On 02/25/2010 02:03 PM, Scott Wood wrote:
>> Gary, can you check that the MMIO addresses are going to the PCI bus 
>> as-is, and aren't being translated down to zero? I.e. POTARn should 
>> equal POBARn, and likewise in the device
>> tree's pci node's ranges.
> 
> Hmm, that doesn't match with how I've always had this setup.  I have:
>   POTAR0 = 0x00000000
>   POTBR0 = 0x000C0000        (0xC0000000 >> 12)
> 
> My device tree mappings are:
>   ranges = <0x02000000 0x0 0xC0000000 0xC0000000 0x0 0x10000000
>             0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>

That ranges property says that host address 0xc0000000 maps to PCI 
address 0xc0000000, so Linux will program the BAR to 0xc0000000, but the 
actual accesses will go elsewhere because POTAR0 is zero.

Setting POTAR to zero is also a bad idea because it's aliasing your DMA 
window (it may work in certain situations based on who's initiating the 
transaction, but it seems like it's asking for trouble), plus it seems 
some cards just don't like address zero.

Try setting POTAR0 to 0x000c0000 and see what happens.

-Scott

Re: PCI on 834x

[Gary Thomas]

On 02/25/2010 02:24 PM, Scott Wood wrote:
> Gary Thomas wrote:
>> On 02/25/2010 02:03 PM, Scott Wood wrote:
>>> Gary, can you check that the MMIO addresses are going to the PCI bus as-is, and aren't being translated down to zero? I.e. POTARn should equal POBARn, and likewise in the device
>>> tree's pci node's ranges.
>>
>> Hmm, that doesn't match with how I've always had this setup. I have:
>> POTAR0 = 0x00000000
>> POTBR0 = 0x000C0000 (0xC0000000 >> 12)
>>
>> My device tree mappings are:
>> ranges = <0x02000000 0x0 0xC0000000 0xC0000000 0x0 0x10000000
>> 0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>
>
> That ranges property says that host address 0xc0000000 maps to PCI address 0xc0000000, so Linux will program the BAR to 0xc0000000, but the actual accesses will go elsewhere
> because POTAR0 is zero.
>
> Setting POTAR to zero is also a bad idea because it's aliasing your DMA window (it may work in certain situations based on who's initiating the transaction, but it seems like it's
> asking for trouble), plus it seems some cards just don't like address zero.
>
> Try setting POTAR0 to 0x000c0000 and see what happens.

Sadly, that doesn't work at all - neither RedBoot nor Linux
can talk to any of the PCI devices.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PCI on 834x

[Scott Wood]

Gary Thomas wrote:
> On 02/25/2010 02:24 PM, Scott Wood wrote:
>> Gary Thomas wrote:
>>> On 02/25/2010 02:03 PM, Scott Wood wrote:
>>>> Gary, can you check that the MMIO addresses are going to the PCI bus 
>>>> as-is, and aren't being translated down to zero? I.e. POTARn should 
>>>> equal POBARn, and likewise in the device
>>>> tree's pci node's ranges.
>>>
>>> Hmm, that doesn't match with how I've always had this setup. I have:
>>> POTAR0 = 0x00000000
>>> POTBR0 = 0x000C0000 (0xC0000000 >> 12)
>>>
>>> My device tree mappings are:
>>> ranges = <0x02000000 0x0 0xC0000000 0xC0000000 0x0 0x10000000
>>> 0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>
>>
>> That ranges property says that host address 0xc0000000 maps to PCI 
>> address 0xc0000000, so Linux will program the BAR to 0xc0000000, but 
>> the actual accesses will go elsewhere
>> because POTAR0 is zero.
>>
>> Setting POTAR to zero is also a bad idea because it's aliasing your 
>> DMA window (it may work in certain situations based on who's 
>> initiating the transaction, but it seems like it's
>> asking for trouble), plus it seems some cards just don't like address 
>> zero.
>>
>> Try setting POTAR0 to 0x000c0000 and see what happens.
> 
> Sadly, that doesn't work at all - neither RedBoot nor Linux
> can talk to any of the PCI devices.

Make sure that whatever is doing the PCI enumeration is allocating MMIO 
addresses starting at 0xc0000000.  That anything worked at all before 
indicates that something was probably allocating from zero, despite 
what's in the device tree.

-Scott

Re: PCI on 834x

[Benjamin Herrenschmidt]

On Thu, 2010-02-25 at 15:03 -0600, Scott Wood wrote:
> 
> It can, but I don't see how that would help, if the problem is that the 
> video card doesn't like the low 30 bits of its MMIO address being zero.

My idea was the video card, if it's an old piece of shit, may not be
decoding all address bits.

> Gary, can you check that the MMIO addresses are going to the PCI bus 
> as-is, and aren't being translated down to zero?  I.e. POTARn should 
> equal POBARn, and likewise in the device tree's pci node's ranges.

Cheers,
Ben.