Change in PCI behaviour

Change in PCI behaviour

[Gary Thomas]

I'm upgrading from 2.6.28 to 2.6.32 (yes, I know it's not the latest,
but it's the best I can do at the moment).  There seems to have been
a change in how the PCI bus is scanned/assigned which is causing me
some hardware problems.  My hardware is FSL MPC8347 and the problem
is likely specific to the FSL PCI code.

My system has 256MB RAM, fully mapped into the PCI export window.
There are an additional 2 devices on the PCI bus.  On 2.6.28, I
get this layout:

PCI: Probing PCI hardware
pci 0000:00:00.0: reg 10 32bit mmio: [0x000000-0x0fffff]
pci 0000:00:00.0: reg 18 64bit mmio: [0x000000-0xfffffff]
pci 0000:00:0b.0: reg 10 io port: [0x1000-0x1007]
pci 0000:00:0b.0: reg 14 io port: [0x1008-0x100b]
pci 0000:00:0b.0: reg 18 io port: [0x1010-0x1017]
pci 0000:00:0b.0: reg 1c io port: [0x1018-0x101b]
pci 0000:00:0b.0: reg 20 io port: [0x1020-0x102f]
pci 0000:00:0b.0: reg 24 32bit mmio: [0x100000-0x1001ff]
pci 0000:00:0b.0: reg 30 32bit mmio: [0x000000-0x07ffff]
pci 0000:00:0b.0: supports D1 D2
pci 0000:00:0c.0: reg 10 32bit mmio: [0x4000000-0x7ffffff]
PCI: Cannot allocate resource region 5 of device 0000:00:0b.0, will remap
PCI: Cannot allocate resource region 0 of device 0000:00:0c.0, will remap
bus: 00 index 0 io port: [0x00-0xfffff]
bus: 00 index 1 mmio: [0xc0000000-0xdfffffff]

There are no notices, but the key item is that device 0000:00:0c.0 gets
mapped at PCI address 0xD0000000.

On 2.6.32, I get this:

PCI: Probing PCI hardware
pci 0000:00:0b.0: reg 10: [io  0x1000-0x1007]
pci 0000:00:0b.0: reg 14: [io  0x1008-0x100b]
pci 0000:00:0b.0: reg 18: [io  0x1010-0x1017]
pci 0000:00:0b.0: reg 1c: [io  0x1018-0x101b]
pci 0000:00:0b.0: reg 20: [io  0x1020-0x102f]
pci 0000:00:0b.0: reg 24: [mem 0x00100000-0x001001ff]
pci 0000:00:0b.0: reg 30: [mem 0x00000000-0x0007ffff pref]
pci 0000:00:0b.0: supports D1 D2
pci 0000:00:0c.0: reg 10: [mem 0x04000000-0x07ffffff]
PCI: Cannot allocate resource region 5 of device 0000:00:0b.0, will remap
PCI: Cannot allocate resource region 0 of device 0000:00:0c.0, will remap
pci 0000:00:0c.0: BAR 0: assigned [mem 0xc0000000-0xc3ffffff]
pci 0000:00:0c.0: BAR 0: set to [mem 0xc0000000-0xc3ffffff] (PCI address [0xc0000000-0xc3ffffff]
pci 0000:00:0b.0: BAR 6: assigned [mem 0xc4000000-0xc407ffff pref]
pci 0000:00:0b.0: BAR 5: assigned [mem 0xc4080000-0xc40801ff]
pci 0000:00:0b.0: BAR 5: set to [mem 0xc4080000-0xc40801ff] (PCI address [0xc4080000-0xc40801ff]
pci_bus 0000:00: resource 0 [io  0x0000-0xfffff]
pci_bus 0000:00: resource 1 [mem 0xc0000000-0xdfffffff]

In this case, note that PCI device 0000:00:0c.0 is at 0xc0000000.
This causes problems because it's a truly stupid device that does
not work properly at PCI [relative] address 0x00000000.  It simply
does not respond at that address.  Pick anywhere else and it will
work fine!

On 2.6.28, I get this layout:
# ls -l /sys/bus/pci/devices
lrwxrwxrwx    1 root     root             0 Jan  1  1970 0000:00:00.0 -> ../../../devices/pci0000:00/0000:00:00.0
lrwxrwxrwx    1 root     root             0 Jan  1  1970 0000:00:0b.0 -> ../../../devices/pci0000:00/0000:00:0b.0
lrwxrwxrwx    1 root     root             0 Jan  1  1970 0000:00:0c.0 -> ../../../devices/pci0000:00/0000:00:0c.0
# cat /sys/bus/pci/devices/0000\:00\:0c.0/resource
0x00000000d0000000 0x00000000d3ffffff 0x0000000000020200
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000


On 2.6.32, the final layout looks like this:
# ls -l /sys/bus/pci/devices/
lrwxrwxrwx    1 root     root             0 Jan  1  1970 0000:00:0b.0 -> ../../../devices/pci0000:00/0000:00:0b.0
lrwxrwxrwx    1 root     root             0 Jan  1  1970 0000:00:0c.0 -> ../../../devices/pci0000:00/0000:00:0c.0
# cat /sys/bus/pci/devices/0000\:00\:0c.0/resource
0x00000000c0000000 0x00000000c3ffffff 0x0000000000020200
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000

Bottom line: how can I get this behaviour back (so as to get my
stupid graphics controller working again)??

Thanks for any ideas

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Change in PCI behaviour

[Benjamin Herrenschmidt]

On Fri, 2010-11-19 at 08:42 -0700, Gary Thomas wrote:
> In this case, note that PCI device 0000:00:0c.0 is at 0xc0000000.
> This causes problems because it's a truly stupid device that does
> not work properly at PCI [relative] address 0x00000000.  It simply
> does not respond at that address.  Pick anywhere else and it will
> work fine! 

Hrm, we used to have a trick avoid giving out the first meg of a bus to
avoid that sort of thing, I suppose it got lost. The rest is related to
the way you map your PCI I suppose in your dts. You can switch back to a
1:1 instead of 1:0 mapping I suppose.

One way to achieve the above result would be to, in your platform code,
reserve the mem region that corresponds to PCI 0...1M (c0000000...+1M)
before the device resources are assigned/allocated.

I though we had code to do that with the "legacy" regions somewhere...
oh well, no code at hand to check right now.

Cheers,
Ben.

Re: Change in PCI behaviour

[Gary Thomas]

On 11/19/2010 02:46 PM, Benjamin Herrenschmidt wrote:
> On Fri, 2010-11-19 at 08:42 -0700, Gary Thomas wrote:
>> In this case, note that PCI device 0000:00:0c.0 is at 0xc0000000.
>> This causes problems because it's a truly stupid device that does
>> not work properly at PCI [relative] address 0x00000000.  It simply
>> does not respond at that address.  Pick anywhere else and it will
>> work fine!
>
> Hrm, we used to have a trick avoid giving out the first meg of a bus to
> avoid that sort of thing, I suppose it got lost. The rest is related to
> the way you map your PCI I suppose in your dts. You can switch back to a
> 1:1 instead of 1:0 mapping I suppose.
>
> One way to achieve the above result would be to, in your platform code,
> reserve the mem region that corresponds to PCI 0...1M (c0000000...+1M)
> before the device resources are assigned/allocated.
>
> I though we had code to do that with the "legacy" regions somewhere...
> oh well, no code at hand to check right now.

Thanks, I found a combo of regions in my DTS that fixed this.

That went well and the system is now running, but it's not stable :-(
It will crash randomly, generally leaving no trace of what went wrong.
I've attached a BDI to it, but mostly all it can tell me is "it's dead"
The one thing that seems to pop up is it looks like it's jumping into
space (aka the wrong place) when doing rfi (this is a guess).  I've
seen things like the MSR ends up loaded with an address, or similar
strangeness.

Were there any system level changes during this period (I know it's
some time ago) that might have introduced such an instability?  It's
tough to scan through the diffs and get a feeling for any little details
like this.

Any ideas or hints greatly appreciated, thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Change in PCI behaviour

[Gary Thomas]

On 11/21/2010 10:59 AM, Gary Thomas wrote:
> On 11/19/2010 02:46 PM, Benjamin Herrenschmidt wrote:
>> On Fri, 2010-11-19 at 08:42 -0700, Gary Thomas wrote:
>>> In this case, note that PCI device 0000:00:0c.0 is at 0xc0000000.
>>> This causes problems because it's a truly stupid device that does
>>> not work properly at PCI [relative] address 0x00000000. It simply
>>> does not respond at that address. Pick anywhere else and it will
>>> work fine!
>>
>> Hrm, we used to have a trick avoid giving out the first meg of a bus to
>> avoid that sort of thing, I suppose it got lost. The rest is related to
>> the way you map your PCI I suppose in your dts. You can switch back to a
>> 1:1 instead of 1:0 mapping I suppose.
>>
>> One way to achieve the above result would be to, in your platform code,
>> reserve the mem region that corresponds to PCI 0...1M (c0000000...+1M)
>> before the device resources are assigned/allocated.
>>
>> I though we had code to do that with the "legacy" regions somewhere...
>> oh well, no code at hand to check right now.
>
> Thanks, I found a combo of regions in my DTS that fixed this.
>
> That went well and the system is now running, but it's not stable :-(
> It will crash randomly, generally leaving no trace of what went wrong.
> I've attached a BDI to it, but mostly all it can tell me is "it's dead"
> The one thing that seems to pop up is it looks like it's jumping into
> space (aka the wrong place) when doing rfi (this is a guess). I've
> seen things like the MSR ends up loaded with an address, or similar
> strangeness.
>
> Were there any system level changes during this period (I know it's
> some time ago) that might have introduced such an instability? It's
> tough to scan through the diffs and get a feeling for any little details
> like this.
>
> Any ideas or hints greatly appreciated, thanks
>

I have a bit more information on this.  I'm pretty sure that the failures
are only happening in my SCSI (SATA actually) code.  My board (8347ea) has
a PCI bus with a SIL SATA controller.  This combo works perfectly in 2.6.28.
In 2.6.32, it will run for a while (possibly quite a while), then timeout
trying to do a large block write - typically 256 blocks.  Once this timeout
happens, the SIL controller is stuck and accesses to it will eventually
cause the whole system to hang (as above).

Was there any major change in how PCI or DMA was handled between 2.6.28
and 2.6.32?  Given the ephemeral nature of these failures (multiple runs
all eventually fail, but never the same twice), my only hope of fixing it
will be to have some ideas what might have changed.

Thanks for any ideas

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Change in PCI behaviour

[Benjamin Herrenschmidt]

On Mon, 2010-11-22 at 03:01 -0700, Gary Thomas wrote:
> I have a bit more information on this.  I'm pretty sure that the failures
> are only happening in my SCSI (SATA actually) code.  My board (8347ea) has
> a PCI bus with a SIL SATA controller.  This combo works perfectly in 2.6.28.
> In 2.6.32, it will run for a while (possibly quite a while), then timeout
> trying to do a large block write - typically 256 blocks.  Once this timeout
> happens, the SIL controller is stuck and accesses to it will eventually
> cause the whole system to hang (as above).
> 
> Was there any major change in how PCI or DMA was handled between 2.6.28
> and 2.6.32?  Given the ephemeral nature of these failures (multiple runs
> all eventually fail, but never the same twice), my only hope of fixing it
> will be to have some ideas what might have changed.

Maybe the changes you did to the PCI outbound windows are now breaking
DMA ? Make sure the outbound and inbound don't overlap for example and
that all RAM is reachable for inbound.

Cheers,
Ben.

Re: Change in PCI behaviour

[Gary Thomas]

On 11/22/2010 01:26 PM, Benjamin Herrenschmidt wrote:
> On Mon, 2010-11-22 at 03:01 -0700, Gary Thomas wrote:
>> I have a bit more information on this.  I'm pretty sure that the failures
>> are only happening in my SCSI (SATA actually) code.  My board (8347ea) has
>> a PCI bus with a SIL SATA controller.  This combo works perfectly in 2.6.28.
>> In 2.6.32, it will run for a while (possibly quite a while), then timeout
>> trying to do a large block write - typically 256 blocks.  Once this timeout
>> happens, the SIL controller is stuck and accesses to it will eventually
>> cause the whole system to hang (as above).
>>
>> Was there any major change in how PCI or DMA was handled between 2.6.28
>> and 2.6.32?  Given the ephemeral nature of these failures (multiple runs
>> all eventually fail, but never the same twice), my only hope of fixing it
>> will be to have some ideas what might have changed.
>
> Maybe the changes you did to the PCI outbound windows are now breaking
> DMA ? Make sure the outbound and inbound don't overlap for example and
> that all RAM is reachable for inbound.

Here's what I did to work around this - in my DTS, I set up my PCI as
	ranges = <0x02000000 0x0 0xC4000000 0xC4000000 0x0 0x1C000000
	          0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>;
Before, I had it as
	ranges = <0x02000000 0x0 0xC0000000 0xC0000000 0x0 0x20000000
	          0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>;

I wasn't sure how to reserve the memory (based on your earlier suggestion),
so I just narrowed the window.  Note that I did not change the PCI hardware
registers (maybe the FSL code does?), so the outbound window should still
be the whole 512MB.

If this isn't viable, perhaps you could explain a bit more how to reserve
such a chunk of memory so that the PCI mappings remain the same.

Thanks again

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Change in PCI behaviour

[Gary Thomas]

On 11/23/2010 07:44 AM, Gary Thomas wrote:
> On 11/22/2010 01:26 PM, Benjamin Herrenschmidt wrote:
>> On Mon, 2010-11-22 at 03:01 -0700, Gary Thomas wrote:
>>> I have a bit more information on this. I'm pretty sure that the failures
>>> are only happening in my SCSI (SATA actually) code. My board (8347ea)
>>> has
>>> a PCI bus with a SIL SATA controller. This combo works perfectly in
>>> 2.6.28.
>>> In 2.6.32, it will run for a while (possibly quite a while), then
>>> timeout
>>> trying to do a large block write - typically 256 blocks. Once this
>>> timeout
>>> happens, the SIL controller is stuck and accesses to it will eventually
>>> cause the whole system to hang (as above).
>>>
>>> Was there any major change in how PCI or DMA was handled between 2.6.28
>>> and 2.6.32? Given the ephemeral nature of these failures (multiple runs
>>> all eventually fail, but never the same twice), my only hope of
>>> fixing it
>>> will be to have some ideas what might have changed.
>>
>> Maybe the changes you did to the PCI outbound windows are now breaking
>> DMA ? Make sure the outbound and inbound don't overlap for example and
>> that all RAM is reachable for inbound.
>
> Here's what I did to work around this - in my DTS, I set up my PCI as
> ranges = <0x02000000 0x0 0xC4000000 0xC4000000 0x0 0x1C000000
> 0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>;
> Before, I had it as
> ranges = <0x02000000 0x0 0xC0000000 0xC0000000 0x0 0x20000000
> 0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>;
>
> I wasn't sure how to reserve the memory (based on your earlier suggestion),
> so I just narrowed the window. Note that I did not change the PCI hardware
> registers (maybe the FSL code does?), so the outbound window should still
> be the whole 512MB.
>
> If this isn't viable, perhaps you could explain a bit more how to reserve
> such a chunk of memory so that the PCI mappings remain the same.

Any ideas on this?  I'm a bit lost as to how to reserve the memory like
you suggested and what I've tried so far has met little success.

Thanks again

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Change in PCI behaviour

[Benjamin Herrenschmidt]

On Sat, 2010-12-04 at 05:49 -0700, Gary Thomas wrote:
> On 11/23/2010 07:44 AM, Gary Thomas wrote:
> > On 11/22/2010 01:26 PM, Benjamin Herrenschmidt wrote:
> >> On Mon, 2010-11-22 at 03:01 -0700, Gary Thomas wrote:
> >>> I have a bit more information on this. I'm pretty sure that the failures
> >>> are only happening in my SCSI (SATA actually) code. My board (8347ea)
> >>> has
> >>> a PCI bus with a SIL SATA controller. This combo works perfectly in
> >>> 2.6.28.
> >>> In 2.6.32, it will run for a while (possibly quite a while), then
> >>> timeout
> >>> trying to do a large block write - typically 256 blocks. Once this
> >>> timeout
> >>> happens, the SIL controller is stuck and accesses to it will eventually
> >>> cause the whole system to hang (as above).
> >>>
> >>> Was there any major change in how PCI or DMA was handled between 2.6.28
> >>> and 2.6.32? Given the ephemeral nature of these failures (multiple runs
> >>> all eventually fail, but never the same twice), my only hope of
> >>> fixing it
> >>> will be to have some ideas what might have changed.
> >>
> >> Maybe the changes you did to the PCI outbound windows are now breaking
> >> DMA ? Make sure the outbound and inbound don't overlap for example and
> >> that all RAM is reachable for inbound.
> >
> > Here's what I did to work around this - in my DTS, I set up my PCI as
> > ranges = <0x02000000 0x0 0xC4000000 0xC4000000 0x0 0x1C000000
> > 0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>;
> > Before, I had it as
> > ranges = <0x02000000 0x0 0xC0000000 0xC0000000 0x0 0x20000000
> > 0x01000000 0x0 0x00000000 0xB8000000 0x0 0x00100000>;
> >
> > I wasn't sure how to reserve the memory (based on your earlier suggestion),
> > so I just narrowed the window. Note that I did not change the PCI hardware
> > registers (maybe the FSL code does?), so the outbound window should still
> > be the whole 512MB.
> >
> > If this isn't viable, perhaps you could explain a bit more how to reserve
> > such a chunk of memory so that the PCI mappings remain the same.
> 
> Any ideas on this?  I'm a bit lost as to how to reserve the memory like
> you suggested and what I've tried so far has met little success.
> 
> Thanks again
> 

Look at pcibios_reserve_legacy_regions() in
arch/powerpc/kernel/pci-common.c, it "reserves" the legacy IO and VGA
regions on host bridges. You can make it reserve whatever your
device is allergic too.

Cheers,
Ben.

Re: Change in PCI behaviour

[Gabriel Paubert]

On Fri, Nov 19, 2010 at 08:42:46AM -0700, Gary Thomas wrote:
> In this case, note that PCI device 0000:00:0c.0 is at 0xc0000000.
> This causes problems because it's a truly stupid device that does
> not work properly at PCI [relative] address 0x00000000.  It simply
> does not respond at that address.  Pick anywhere else and it will
> work fine!

Yes, but it was one upon a time in the PCI spec that setting the
a base register to 0 should disable the corresponding decoder.

I don't know whether this has changed (I actually never had the 
final PCI spec, only drafts). However I once had a device who
actually did not disable base addresses set to zero and this was 
described as a bug in its (numerous) errata. This also caused
a lot of mayhem since in some versions/configurations it used 
up to 64kB of PCI I/O space (especially fun on x86...). 

	Gabriel