[parisc-linux] FWD: PCI Patches update

[parisc-linux] FWD: PCI Patches update

[Grant Grundler]

Hi all,

I got email from Ivan Kokshaysky describing recent changes in 2.5 PCI
support and realized I should document one of the outstanding
"bugs" with PCI resource management under PARISC.

enjoy!
grant

---- Forwarded message from Grant Grundler <grundler@dsl2.external.hp.com> -----
On Mon, Mar 17, 2003 at 11:33:12PM +0300, Ivan Kokshaysky wrote:
> - windows overlapping with something else.

On parisc, this is one of the problems I've dodged.
I'm not looking for a solution right now - but if you know the
obvious answer, please tell me.  I just want to outline the problem.

The PCI controller (lba_pci.c) on newer PARISC machines can support
two MMIO ranges. The "Distributed" range is a "fixed" size and base
(1/8th of whatever the parent IO Controller (SBA) has).  The second
"Directed" Range can use any address otherwise not assigned to PCI
devices and directed to any PCI controller under the same SBA.

PARISC lba_pci.c currently only advertises Distributed ranges
and pretends Directed ranges don't exist. Things still work,
but the resource accounting in /proc/iomem is inaccurate.

Since Directed Ranges have precedence in the MMIO address routing, they
can overlap with Distributed ranges. Worse, The SBA can assign "duplicate"
MMIO address ranges by routing "unused" address space to a Directed range
under a different PCI controller.  I think both are very ugly scenarios
and will need alot of code to detect/handle properly.

Each Distributed Range is typically only 16MB in size.
Consumers of Directed Ranges are things like graphics cards or X.25
cards which want 16MB or greater in a single BAR.
Eg /proc/iomem on a C3000 running 2.4.20 says:
f0000000-f0ffffff : LBA PCI LMMIO
  f05d0000-f05d0000 : lcd_data
  f05d0008-f05d0008 : lcd_cmd
f1000000-f1ffffff : LBA PCI LMMIO
f4000000-f4ffffff : LBA PCI LMMIO
  f4007000-f4007fff : usb-ohci
  f4008000-f40083ff : tulip
  f4900000-f490007f : tulip
  f4901000-f490107f : tulip
  f4902000-f490207f : tulip
  f4903000-f490307f : tulip
f6000000-f6ffffff : LBA PCI LMMIO
  f6000000-f600ffff : tg3
fa100000-fa4fffff : stifb mmio
fb000000-fbffffff : stifb
...

In the same machine (C3000) as the above, lspci says:
...
01:04.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03)
(prog-if 00 [Normal decode])
        Flags: bus master, medium devsel, latency 255
	Bus: primary=01, secondary=02, subordinate=02, sec-latency=255
	I/O behind bridge: 00003000-00003fff
	Memory behind bridge: f4900000-f49fffff
	Prefetchable memory behind bridge: 00000000f0100000-00000000f0100000
	Capabilities: <available only to root>
...
03:01.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1010 66MHz Ultra3 SCSI Adapter (rev 01)
	Subsystem: Hewlett-Packard Company: Unknown device 1340
	Flags: bus master, 66Mhz, medium devsel, latency 255, IRQ 163
	I/O ports at 28000 [size=256]
	Memory at f6014000 (64-bit, non-prefetchable) [size=1K]
	Memory at f6010000 (64-bit, non-prefetchable) [size=8K]
	Expansion ROM at f6020000 [disabled] [size=128K]
	Capabilities: <available only to root>

03:03.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703 Gigabit Ethernet (rev 02)
        Subsystem: Broadcom Corporation NetXtreme BCM5703 1000Base-T
	Flags: bus master, 66Mhz, medium devsel, latency 248, IRQ 160
	Memory at f6000000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: <available only to root>

04:02.0 Display controller: Hewlett-Packard Company A4977A Visualize EG
(rev 03)
        Flags: 66Mhz, medium devsel
	Memory at fa000000 (32-bit, non-prefetchable) [size=32M]
	Expansion ROM at f7000000 [disabled] [size=64K]

[ I now see one additional bugs *sigh*: PCI-PCI bridge resources are
not properly accounted (4-port Tulip) card. I'll have to fix that. ]

sym53c8xx_2 driver is using IO port space and thus won't get reported
in /proc/iomem.

The "stifb" driver claims the upper 16MB of the 32MB BAR and it's all
routed to PCI Controller for slot 2 (PCI Bus 4). Since no other devices
share that PCI controller, I'm pretty sure the same PCI Controller also
gets 0xf100000 range. Add a second gfx card and it's gets uglier.
I watched Thomas Boegendorf have fun at LinuxTag by putting 4
similar "Vis-EG" cards in one B2600 and run xinerama. I now
wish I had captured the /proc/iomem and lspci -v output for
that one!

Anyway, I hope that was an interesting diversion.

cheers,
grant

----- End forwarded message -----

Re: [parisc-linux] FWD: PCI Patches update

[Matthew Wilcox]

On Mon, Mar 17, 2003 at 11:22:33PM -0700, Grant Grundler wrote:
> On parisc, this is one of the problems I've dodged.
> I'm not looking for a solution right now - but if you know the
> obvious answer, please tell me.  I just want to outline the problem.

Certainly!  ;-)

> The PCI controller (lba_pci.c) on newer PARISC machines can support
> two MMIO ranges. The "Distributed" range is a "fixed" size and base
> (1/8th of whatever the parent IO Controller (SBA) has).  The second
> "Directed" Range can use any address otherwise not assigned to PCI
> devices and directed to any PCI controller under the same SBA.

OK.

> PARISC lba_pci.c currently only advertises Distributed ranges
> and pretends Directed ranges don't exist. Things still work,
> but the resource accounting in /proc/iomem is inaccurate.

We can fix that!  ;-)

> Since Directed Ranges have precedence in the MMIO address routing, they
> can overlap with Distributed ranges. Worse, The SBA can assign "duplicate"
> MMIO address ranges by routing "unused" address space to a Directed range
> under a different PCI controller.  I think both are very ugly scenarios
> and will need alot of code to detect/handle properly.

I don't think it's such a big problem.  Well, I wouldn't, would I?  ;-)

> Each Distributed Range is typically only 16MB in size.
> Consumers of Directed Ranges are things like graphics cards or X.25
> cards which want 16MB or greater in a single BAR.
> Eg /proc/iomem on a C3000 running 2.4.20 says:
> f0000000-f0ffffff : LBA PCI LMMIO
>   f05d0000-f05d0000 : lcd_data
>   f05d0008-f05d0008 : lcd_cmd
> f1000000-f1ffffff : LBA PCI LMMIO
> f4000000-f4ffffff : LBA PCI LMMIO
>   f4007000-f4007fff : usb-ohci
>   f4008000-f40083ff : tulip
>   f4900000-f490007f : tulip
>   f4901000-f490107f : tulip
>   f4902000-f490207f : tulip
>   f4903000-f490307f : tulip
> f6000000-f6ffffff : LBA PCI LMMIO
>   f6000000-f600ffff : tg3
> fa100000-fa4fffff : stifb mmio
> fb000000-fbffffff : stifb
> ...

Thanks for posting the example, it makes life much easier.  To take some
experience from ccio-based systems, we might favour a /proc/iomem which
ends up looking something like this:

f0000000-f0ffffff : PCI bus [10/0/]
  f05d0000-f05d0000 : lcd_data
  f05d0008-f05d0008 : lcd_cmd
f1000000-f1ffffff : PCI bus [10/1/]
f4000000-f4ffffff : PCI bus [10/2/]
  f4007000-f4007fff : usb-ohci
  f4008000-f40083ff : tulip
  f4900000-f490007f : tulip
  f4901000-f490107f : tulip
  f4902000-f490207f : tulip 
  f4903000-f490307f : tulip
f6000000-f6ffffff : PCI bus [10/4/]
  f6000000-f600ffff : tg3
fa000000-fbffffff : PCI bus [10/1/]
  fa100000-fa4fffff : stifb mmio
  fb000000-fbffffff : stifb

So, how to accomplish this?  You're right it's going to be a little more
tricky than before since we may discover overlapping ranges in any order
and the directed range always takes priority over the distributed range.

Calling __request_resource() will return the address of a conflicting
resource (if there is one).  Only snag is that it's static, but I'm
sure no-one will scream too loudly if we unstatic it.  So if we're
requesting a directed range, we'll tweak the existing range to move it
out of our way.  If we're requesting a distributed range, we'll shrink
our range to avoid colliding.  In either case, we will request again.

> [ I now see one additional bugs *sigh*: PCI-PCI bridge resources are
> not properly accounted (4-port Tulip) card. I'll have to fix that. ]

iomem isn't a device hierarchy... it's not necessary to put _everything_
in.  Having said that, I think it'd be kind of nice to have it.

> The "stifb" driver claims the upper 16MB of the 32MB BAR and it's all
> routed to PCI Controller for slot 2 (PCI Bus 4). Since no other devices
> share that PCI controller, I'm pretty sure the same PCI Controller also
> gets 0xf100000 range. Add a second gfx card and it's gets uglier.
> I watched Thomas Boegendorf have fun at LinuxTag by putting 4
> similar "Vis-EG" cards in one B2600 and run xinerama. I now
> wish I had captured the /proc/iomem and lspci -v output for
> that one!

Hmm.. if I remember rightly, we put a large number of gfx cards in the
L2000 in Fort Collins once... wonder if they're still in there.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [parisc-linux] FWD: PCI Patches update

[Grant Grundler]

On Tue, Mar 18, 2003 at 02:06:37PM +0000, Matthew Wilcox wrote:
> > PARISC lba_pci.c currently only advertises Distributed ranges
> > and pretends Directed ranges don't exist. Things still work,
> > but the resource accounting in /proc/iomem is inaccurate.
> 
> We can fix that!  ;-)

yes. ivan also sent me some suggestions (which I suspect are for 2.5).
But I still care more about getting 2.4 working right.

> Thanks for posting the example, it makes life much easier.  To take some
> experience from ccio-based systems, we might favour a /proc/iomem which
> ends up looking something like this:
> 
> f0000000-f0ffffff : PCI bus [10/0/]

yes - adding the HW path would be a good idea.
It would help identify which LBA/ropes are involved.
But printing the PCI bus number (in addition?) makes more sense since
I can then correlate iomem output with lspci output.


> So, how to accomplish this?  You're right it's going to be a little more
> tricky than before since we may discover overlapping ranges in any order
> and the directed range always takes priority over the distributed range.

exactly. I'm still thinking about it and my current idea is to
read all SBA ranges first and then let LBA code resolve conflicts
as attemptes to claim ranges. Ivan got me started on that:
| First, you create the set of MMIO resources for SBA which represents
| all MMIO ranges. Then you can either shrink or completely remove
| those of resources for Distributed ranges which overlap with Directed
| range to avoid resource conflicts (this seems to be ok as you says that
| the latter has precedence).

The only problem I'm still looking at is Direct Ranges are described
by three registers which *only* can be read using 64-bit reads.
Tough to do from a (strictly) PA 1.1 built kernel. Still looking
at this issue before moving on.


> Calling __request_resource() will return the address of a conflicting
> resource (if there is one).  Only snag is that it's static, but I'm
> sure no-one will scream too loudly if we unstatic it.  So if we're
> requesting a directed range, we'll tweak the existing range to move it
> out of our way.  If we're requesting a distributed range, we'll shrink
> our range to avoid colliding.  In either case, we will request again.

yes, this is a good idea. I guess I had the same idea but hadn't thought
out how to implement it.

> > [ I now see one additional bugs *sigh*: PCI-PCI bridge resources are
> > not properly accounted (4-port Tulip) card. I'll have to fix that. ]
> 
> iomem isn't a device hierarchy... it's not necessary to put _everything_
> in.  Having said that, I think it'd be kind of nice to have it.

right. /proc/iomem *is* a resource hierarchy and part of the hierarchy
is missing in the case of PCI-PCI bridge.


> Hmm.. if I remember rightly, we put a large number of gfx cards in the
> L2000 in Fort Collins once... wonder if they're still in there.

but L2000 is a PAT PDC box.  We'll get different behavior than B2600.
Unless PAT PDC supports those as boot devices, none of them will get
enabled since lba_pci.c doesn't attempt to program Directed Ranges
(aka lba_dev->elmmio).

Anyway, personally, I don't care about gfx cards working under PAT PDC.
If someone has to make it work, I'm happy to advise and review patches.

thanks,
grant