Probing unconfigured PCI bridge resouces

Probing unconfigured PCI bridge resouces

[Benjamin Herrenschmidt]

Hi folks !

I was digging around chasing a different problem when I stumbled upon
this, I may be missing something but it still feels odd...

So from what I can tell, the only way that we set the flag for the 3
bridge resources of a P2P bridge (IORESOURCE_IO, IORESOURCE_MEM etc...)
is via pci_read_bridge_bases().

This will in turn only set those flags if the resource has a positive
size (limit >= start) ie. it wasn't closed by the firmware.

However what does that leave us in case where we either want to ignore
the firmware (reconfigure everything, which we do on various embedded
platforms) or if the bridge is plain uninitialized and happen to comes
up with closed resources ?

>From what I can tell, we don't set these, which means that we won't
either try to reallocate or reassign those windows later in the resource
assignment code. IE the code in setup-bus.c will use
find_free_bus_resource() which only works if the type has been set...

Or am I missing some important fact here ?

I suspect what's saved us so far on some of those embedded platforms is
that uninitialized bridges tend to come up "open" with a tiny window
0...2M (for memory, I think it's 4K for IO)... but I can see problems (I
had code to specifically re-open some windows on some Apple machines in
the past, which I think we lost over time and might need to be brought
back for example).

Cheers,
Ben.

Re: Probing unconfigured PCI bridge resouces

[Benjamin Herrenschmidt]

On Tue, 2012-01-24 at 15:25 +1100, Benjamin Herrenschmidt wrote:
 .../...

> Or am I missing some important fact here ?

Ok, I think I was missing pci_bridge_check_ranges(), gotta figure now
why this didn't kick in properly for me, probably some arch code
crackpot.

One thing I noticed is that we do clear IORESOURCE_UNSET in setup-res.c
when setting up a device resource but we don't clear it for bridges in
generic code ever, so it leaks all the way through from my early arch
code. Not a big deal but annoying.

Cheers,
Ben.

Re: Probing unconfigured PCI bridge resouces

[Jesse Barnes]

[-- Attachment #1: Type: text/plain, Size: 925 bytes --]

On Tue, 24 Jan 2012 15:50:33 +1100
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Tue, 2012-01-24 at 15:25 +1100, Benjamin Herrenschmidt wrote:
>  .../...
> 
> > Or am I missing some important fact here ?
> 
> Ok, I think I was missing pci_bridge_check_ranges(), gotta figure now
> why this didn't kick in properly for me, probably some arch code
> crackpot.
> 
> One thing I noticed is that we do clear IORESOURCE_UNSET in setup-res.c
> when setting up a device resource but we don't clear it for bridges in
> generic code ever, so it leaks all the way through from my early arch
> code. Not a big deal but annoying.

Maybe you can fix that when you unify your arch resource handling code
with the core one day. :)

There have been a lot of changes to handle re-allocation and such, and
I'm sure there's more duplication now...

-- 
Jesse Barnes, Intel Open Source Technology Center

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Probing unconfigured PCI bridge resouces

[Benjamin Herrenschmidt]

On Fri, 2012-01-27 at 09:36 -0800, Jesse Barnes wrote:

> > Ok, I think I was missing pci_bridge_check_ranges(), gotta figure now
> > why this didn't kick in properly for me, probably some arch code
> > crackpot.
> > 
> > One thing I noticed is that we do clear IORESOURCE_UNSET in setup-res.c
> > when setting up a device resource but we don't clear it for bridges in
> > generic code ever, so it leaks all the way through from my early arch
> > code. Not a big deal but annoying.
> 
> Maybe you can fix that when you unify your arch resource handling code
> with the core one day. :)
> 
> There have been a lot of changes to handle re-allocation and such, and
> I'm sure there's more duplication now...

There isn't much the my arch does that isn't just calling into core
nowadays on powerpc, look at my resource survey :-)

I have some slightly different (arguably better ?) logic to deal with
unassigned resources in the initial allocation pass before re-assigning
(or assigning unassigned) and flags to enable ignoring the initial state
of devices (which should be improved with core changes one of these
days, for example we still read all bridge bases even when we know we
are just going to re-assign everything etc...).

The one thing I cannot unify is actually new in 3.3, it's the code to
assign resources on the new "powernv" platform which requires its own
logic unfortunately and I'm not sure I can make the core do what I want
without major uglyness.

The main reason is that on p7ioc and similar recent IBM host bridges, we
have a domain mechanism which allows error isolation, dma isolation
etc... which also encompass MMIO (in order to properly assign MMIO
errors such as target aborts etc... with a domain).

The way this works is by segmenting the MMIO space, which means that all
BARs of a all devices in a given domain shall be assigned to a group of
segments. Now I -can- renumber 32-bit segments so in theory I could get
away with alignment tricks (tho nasty ones) only but I can't always
renumber 64-bit segments (tho in my current scheme I simply don't use
64-bit MMIO segments at all, in part due to that complication).

The consequence is that I need to pre-assign domains at boot based on
various constraints and policy decisions which unfortunately have to be
made by the kernel arbitrarily, and then tweak the MMIO resource
allocation to assign BARs of devices accordingly so they fit into
"segments" that can be made to correspond with the domains.

Cheers,
Ben.