Re: IO issues vs. multiple busses

Re: IO issues vs. multiple busses

[Grant Grundler]

Benjamin,
Here are my comments directly responding to your mail.

Benjamin Herrenschmidt wrote:
> Hi Grant !
> 
> My original mail is:
> 
> Here's the return of an oooold problem for which we really need a
> solution asap since it's now biting us in real life configurations...
> 
> So the problem happens when you have a machine with more than one PCI
> host bridge. This is typically the case of all new Apple machines as they
> have 3 host bridges in one chip (2 of them are relevant: the AGP and the
> PCI). I don't think the problem exist on x86 machines with real IO
> cycles, at least in that case, the problem is different.

Large systems have problems with I/O port space and legacy devices.
There just isn't enough I/O port space to support large configs
and ISA aliasing and all the other crud. That's why Intel is (a)
ditching all the legacy crap in IA64 and (b) strongly encouraging
people to use MMIO space on PCI.


> In order to generate IO cycles, the bridge provides us with a region in
> CPU physical memory space (a 16Mb region in our case) that translates
> accesses to IO cycles on the PCI bus. Our implementation of inb/outb
> currently relies on the kernel ioremap'ing one of these regions (the PCI
> one) and using the ioremap result as a base (offset) inside the inb/outb
> functions.

If you only support one type of bridge, you could avoid the indirect
function call (which parisc-linux uses) and encode the access method
directly in the inb/outb macros.

Just note that processor speed is so much faster (and getter faster)
than the ISA bus (and PCI-1X bus), that CPU overhead is mostly
irrelevant to the cost of accessing IO port space. On older x86
boxes it is relevant.

> So that mean that the current design won't allow access to IOs located on
> any bus but the one we arbitrarily choose (the PCI bus). That's fine in
> most case, until you decide to put a 3dfx or nvidia card in the AGP slot.
> Those cards require some IO accesses to be done to the legacy VGA
> addresses, and of course, our inb/outb functions can't do that.

parisc-linux has solved exactly that problem.

> Obviously, we can hack some driver specific thing that would use the
> arch-specific code to retreive the proper io base address for a given
> host bridge, but that's a hack. I'm looking for a solution that would
> cleanly apply to all archs that may potentially face this problem.

I don't believe such a solution exists which is "cleaner" than
what parisc-linux does and meets the same objectives. Right now,
it's important the install be easy in order to make it easy for
people to migrate from HPUX to parisc-linux. :^)

> The problem potentially exist also for any PCI card that has PCI IOs on
> anything but the main PCI bus. 
> 
> One possibility is to limit our IO space to 64k per bus (to avoid
> bloating) and then use a hacked ioremap to create a single virtually
> contiguous kernel region that appends all those IO spaces together.
> Accessing IOs on bus N would just be the matter of calculating an address
> of the type 64k*N+offset and doing normal inb/outb on the result.

This might work for other arches. I'm pretty sure it won't for parisc.
Again, the issue is the IO port space access method varies by HBA.

> The
> arch PCI code could then properly fixup PCI IO resources for PCI drivers,
> and we could add a function of the kind
>
>  unsigned long pci_bus_io_offset(int busno);
> 
> that would return the offset to add to inb/outb when accessing IOs on the
> N'th PCI bus.

Basically, parisc-linux does that but the arch support hides that
from the device drivers.


> If we want to go a bit further, and allow ISA drivers that don't have a
> pci_dev structure to work on legacy devices on any bus, we could provide
> a set of function of the type
> 
>  int isa_get_bus_count();
>  unsigned long isa_get_bus_io_offset(int busno);
> 
> and eventually
> 
>  int isa_bus_to_pci_bus(int isa_busno);
>  int pci_bus_to_isa_bus(int pci_busno);

I don't like this either.
Reserving bus 0 for E/ISA solves the problem.


> I'm, of course open to any comments about this (in fact, I'd really like
> some feedback). One thing is that we also need to find a way to pass
> those infos to userland. Currently, we implement an arch-specific syscall
> that allow to retreive the IO physical base of a given PCI bus. That may
> be enough, but we may also want something that match more closely what we
> do in the kernel.

I agree with davem on this.

But maybe for different reasons.
The issue with exporting IO port regions is the access method.
Access method varies by platform (for parisc arch) and I don't
want to see user apps encoding the access method for specific platforms
by default. If someone intentionally wants to build an app for a
specific platform, that's different.

grant

Grant Grundler
parisc-linux {PCI|IOMMU|SMP} hacker
+1.408.447.7253

Re: IO issues vs. multiple busses

[Benjamin Herrenschmidt]

>Here are my comments directly responding to your mail.

Hi ! Thanks for taking the time to respond in details.

>Large systems have problems with I/O port space and legacy devices.
>There just isn't enough I/O port space to support large configs
>and ISA aliasing and all the other crud. That's why Intel is (a)
>ditching all the legacy crap in IA64 and (b) strongly encouraging
>people to use MMIO space on PCI.

Right. We need to decourage use of IOs, definitely ;)
I now tend to think that we shouldn't care about making a whole
architecture to handle those IO problems, but the simplest possible thing
that would fit our needs. (still for in-kernel matters)

>If you only support one type of bridge, you could avoid the indirect
>function call (which parisc-linux uses) and encode the access method
>directly in the inb/outb macros.

We do that now, and support IOs on one bridge only. However, some PCI
cards still require IO access and we do have several busses, so...
The reason why I'm getting this problem on the public place (again ?)
is that we are now faced with people who want to put video cards in both
AGP & PCI busses, those cards requiring accesses to some legacy VGA IOs
on each of their busses.

>Just note that processor speed is so much faster (and getter faster)
>than the ISA bus (and PCI-1X bus), that CPU overhead is mostly
>irrelevant to the cost of accessing IO port space. On older x86
>boxes it is relevant.

Right. That's my opinion too. But it's difficult to make everybody agree
on it ;) Even the simple mecanism Paul Mackerras did so that IOs to
non-existent devices don't kill the kernel (very small overhead) caused
some barking ;)

>parisc-linux has solved exactly that problem.

I have to look in more details. It's my understanding that you use high
bits of the IO address to store the HBA number and then use that to call
the proper access functions. That would solve the PCI IO problem (PCI cards
requiring IOs to BAR-mapped regions), but I don't see how it can fix the
problem of a card accessing legacy VGA addresses, except if you hand-fixed
the video drivers to fill those high bits before doing IOs.

If I understand things correctly, that mean that each card, instead of
accessing the legacy VGA port 0xpp, would instead access 0x00bb00pp
(or whatever mangling you use to stuff the HBA number).

>From the driver point of view, it's exactly the same as passing an "offset"
that would be added to the legacy address. So both methods (the one I describe
that would fit well for us) and yours can end up with the same driver-side API
which is to get an "IO base" for the bus a given card reside on.

The question is then to decide is all ISA busses are on a matching PCI bus,
in which case a simple unsigned pci_get_bus_io_base(int bus_no) -like function
would be enough, or if we want a scheme that supports other ISA-like busses ?

We could eventually decide to support only PCI, and additionally declare a
fake PCI bus for an ISA bus not matched to a PCI bus, whose config ops would
return no device in any slot.

Do we agree on this ? 

>I don't believe such a solution exists which is "cleaner" than
>what parisc-linux does and meets the same objectives. Right now,
>it's important the install be easy in order to make it easy for
>people to migrate from HPUX to parisc-linux. :^)

Well, from the driver point of view, I think it _do_ exist. Basically, the
driver will do inb/outb & friends. Whatever those function do in reality is
arch-dependant.
But we agree on the fact that in order for those functions to know on which
bus to tap, an additional information must be "cooked" inside the IO address
passed to them. That's why I'm proposing this notion of "io base".

Additionally, the same problem is true for ISA memory, when it exist
obviously.
I would indeed like to see the same function for
pci_get_legacy_mem_base(int bus_no)-like, that is allowed to return something
like ffffffff for informing the driver that this specific machine won't
support
ISA memory.

With those two simple functions, we could at least

 - fix the the fbdev's that need access to VGA regions so that they work on
   multiple bus systems properly
 - Have vgacon disable itself when there's no ISA memory (that can be
handled by
   reserving the region and thus preventing request_region from working
too, well,
   but that scheme would also simplify the various more/less hacked
macros used
   on all non-x86 archs to access the VGA memory).
 - Eventually have vgacon work on "any" bus, possibly by providing a kernel
   option telling it on which bus to look for a legacy VGA device (and
defaulting
   to whatever VGA device the PCI will find first). This way, vgacon
would work
   properly in most cases without arch-specific hacking.

Additionally, I beleive it would help making other legacy drivers (if
any) work on
non-0 busses (I'm thinking about IDE cards using legacy addresses, those
do exist),
and whatever.

The only thing that's annoying me in the fact that we keep tied to PCI is
that in
various embedded system, there is one (or more) ISA-like bus hanging
around with
legacy devices and no PCI, and abstracting a bit the notion of IO bus
would have
helped. But it seem that now, more embedded systems are also going toward 
in-CPU IOs anyway, along with eventually a PCI bus for the most expensive
ones,
so it may finally not be an issue in the long term.

>This might work for other arches. I'm pretty sure it won't for parisc.
>Again, the issue is the IO port space access method varies by HBA.

Well, as I said above, this can still be an arch-dependant implementation
detail provided that the driver-side interface for getting the "offset"
or whatever we call it that is applies to ISA addresses is common.

>I agree with davem on this.

I do too ;)

>But maybe for different reasons.
>The issue with exporting IO port regions is the access method.
>Access method varies by platform (for parisc arch) and I don't
>want to see user apps encoding the access method for specific platforms
>by default. If someone intentionally wants to build an app for a
>specific platform, that's different.

Right. DaveM idea, as we discussed it some time ago, is to abstract the
mapping
using mmap, and an ioctl that would provide some access information (stepping,
access size maybe, etc...). If really specific access methods are still
needed,
then they should be written specifically in the userland app too, there's no
workaround for that I can imagine. But the fact that those are needed can
probably be returned to userland by this ioctl via a specific result code or
whatever.

Ben.

Re: IO issues vs. multiple busses

[Geert Uytterhoeven]

On Sat, 3 Mar 2001, Benjamin Herrenschmidt wrote:
> With those two simple functions, we could at least
> 
>  - Have vgacon disable itself when there's no ISA memory (that can be
                                                ^^^^^^^^^^
> handled by
>    reserving the region and thus preventing request_region from working
                                              ^^^^^^^^^^^^^^
Do you mean request_mem_region()?

> too, well,
>    but that scheme would also simplify the various more/less hacked
> macros used
>    on all non-x86 archs to access the VGA memory).

request_mem_region() for ISA memory is another problem point. The few drivers
that use it seem to assume that the ISA memory base is 0. This won't work on
non-PC machines, since ISA memory may be somewhere else in the address space,
and more important, there already may be something different at address 0,
which breaks request_mem_region(). On a PC the first 16 MB of RAM (with some
holes at e.g. 0xa0000) overlap with ISA memory space, but not on other
architectures.

For ioremap() we have a hack on PPC (PReP/CHRP) that adds isa_mem_base if the
bus address to map falls in the first 16 MB area, but this cannot work for
request_mem_region(). I do have my full memory map (RAM) marked in /proc/iomem.

So once again I vote for the introduction of
isa_{request,release}_mem_region(), just like we already have isa_readb() and
friends.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: IO issues vs. multiple busses

[Benjamin Herrenschmidt]

>So once again I vote for the introduction of
>isa_{request,release}_mem_region(), just like we already have isa_readb() and
>friends.

Well, it's the same problem as the IO, there may be more than one ISA mem
region,
especially when you put 2 video cards on 2 different PCI hosts (even without a
PCI-ISA bridge).

In fact, with a PCI-ISA bridge, I can imagine a config where you need 2 ISA IO
regions and 2 ISA mem regions on the same PCI bus if that bridge does address 
translation.

My concern for now is mostly to get video cards fixed, I don't care much about
legacy ISA hardware as in those case, I guess we can limit ourselves to a
single
ISA bus and inb/oub beeing happy to cope with it.

The problem is that we use the same macros (inb/outb) to access that ISA bus,
and to access any PCI IO bus. Well, I would suggest the following:

 - inb/outb without offset -> the ISA bus if any, or the IO space of the
   first PCI host
 - inb/outb with offset (or encoded HBA number) -> IO space of an other bus
 - pci_get_bus_io_base() returns the IO offset for accessing the Nth PCI
   bus IO space so that the fb devs can do VGA IOs on the bus that holds
   their card.
 - pci_get_bus_isa_mem_base() returns the base address at which isa mem
   is available for a given PCI bus (that is the address that generates
   mem cycles in the range 0->64k). This is a physical address, the driver
   still have to ioremap it. Some PCI cards can have a BAR mapping the
   VGA memory elsewhere, drivers for those cards should prefer the BAR
   mapping of course.

All IO ranges can be mapped via kernel VM tricks into a single contiguous
space
with the offset beeing something like a 64k increment, or we can have the
inb/outb
do a lookup of the host bus like on parisc. That's an arch implementation
detail.

Is that ok ? I know it's not perfect, but it would allow to solve the most
important problem for now. The PCI cards in need of IOs (like PCI IDE cards)
can have their resources fixed up by the arch code in order to tap the correct
bus. Only the real legacy ISA drivers will be limited to the fixed (default)
ISA bus.

Ben.

Re: IO issues vs. multiple busses

[Grant Grundler]

Benjamin Herrenschmidt wrote:
...
> The reason why I'm getting this problem on the public place (again ?)
> is that we are now faced with people who want to put video cards in both
> AGP & PCI busses, those cards requiring accesses to some legacy VGA IOs
> on each of their busses.

I don't see any way of getting around virtualizing the IO port space.
(by that I mean encoding more info in the upper IO port address bits)

...
> Right. That's my opinion too. But it's difficult to make everybody agree
> on it ;) Even the simple mechanism Paul Mackerras did so that IOs to
> non-existent devices don't kill the kernel (very small overhead) caused
> some barking ;)

Well, make it a CONFIG_XXX option for your arch and the people who
insist on doing complicated things will have to live with complicated
solutions. I don't have a better idea.

> It's my understanding that you use high
> bits of the IO address to store the HBA number and then use that to call
> the proper access functions.

Correct. For some reason, I thought alpha (or sparc? hose number?) did this
as well.

> That would solve the PCI IO problem (PCI cards
> requiring IOs to BAR-mapped regions), but I don't see how it can fix the
> problem of a card accessing legacy VGA addresses, except if you hand-fixed
> the video drivers to fill those high bits before doing IOs.

That's right. That happens after the bus walk but before drivers see
the device in pci_fixup_bus().

> If I understand things correctly, that mean that each card, instead of
> accessing the legacy VGA port 0xpp, would instead access 0x00bb00pp
> (or whatever mangling you use to stuff the HBA number).

right.

> The question is then to decide is all ISA busses are on a matching PCI bus,
> in which case a simple unsigned pci_get_bus_io_base(int bus_no) -like functio
>   n
> would be enough, or if we want a scheme that supports other ISA-like busses ?

I don't know enough about your arch to answer that.

> We could eventually decide to support only PCI, and additionally declare a
> fake PCI bus for an ISA bus not matched to a PCI bus, whose config ops would
> return no device in any slot.
> Do we agree on this ? 

In theory yes. But davem already wrote:
| There is no 'fake' ISA bus number you need. There is a 'real' one, 
| the one on which the PCI-->ISA bridge lives, why not use that one 
| :-) 
 

> Well, from the driver point of view, I think it _do_ exist. Basically, the
> driver will do inb/outb & friends. Whatever those function do in reality is
> arch-dependant.

Right. I meant the underlying implementation of inb/outb.

> But we agree on the fact that in order for those functions to know on which
> bus to tap, an additional information must be "cooked" inside the IO address
> passed to them.

yes.

> Additionally, the same problem is true for ISA memory, when it exist
> obviously.

Really? I expected ISA memory to look like reguler uncacheable memory
and the drivers would simply dereference the address. But I'm not an
expert on how ISA busses work...


> With those two simple functions, we could at least

[ deleted list ]

I have to defer to someone like Alan Cox or Davem or someone who has
a clue about VGA. I don't. Like sparc, parisc doesn't support VGA.
Alan is the only person I know of who's even considered plugging a VGA
card into a parisc box.


> The only thing that's annoying me in the fact that we keep tied to PCI
> is that in various embedded system, there is one (or more) ISA-like
> bus hanging around with legacy devices and no PCI, and abstracting a

In short, where the HW doesn't route transactions down the right bus
adapter, the SW has too.

> bit the notion of IO bus would have helped. But it seem that now, more
> embedded systems are also going toward in-CPU IOs anyway, along with
> eventually a PCI bus for the most expensive ones, so it may finally
> not be an issue in the long term.

It sounds like the HW will do (some of?) the routing.

grant

Grant Grundler
parisc-linux {PCI|IOMMU|SMP} hacker
+1.408.447.7253

Re: IO issues vs. multiple busses

[Geert Uytterhoeven]

On Sat, 3 Mar 2001, Grant Grundler wrote:
> Benjamin Herrenschmidt wrote:
> > Additionally, the same problem is true for ISA memory, when it exist
> > obviously.
> 
> Really? I expected ISA memory to look like reguler uncacheable memory
> and the drivers would simply dereference the address. But I'm not an
> expert on how ISA busses work...

Nope, it's like PCI memory, but different. You have to use isa_readb() and
friends to access it.

While ISA I/O space and PCI I/O space are the same on many machines with memory
mapped I/O (with a size limitation for ISA I/O), ISA memory space and PCI
memory space are different. Cfr. on my CHRP LongTrail:

| callisto$ cat /proc/iomem 
| 00000000-08000000 : RAM
| c0000000-f6ffffff : GG2 PCI mem
|   c0000000-c0000fff : ATI Technologies Inc 3D Rage I/II 215GT [Mach64 GT]
|   c1000000-c107ffff : Apple Computer Inc. Hydra Mac I/O
|   c1080000-c108007f : Digital Equipment Corporation DECchip 21041 [Tulip Pass 3]
|     c1080000-c108007f : eth0
|   c2000000-c2ffffff : ATI Technologies Inc 3D Rage I/II 215GT [Mach64 GT]
|     c2000000-c2ffffff : atyfb
|   c3000000-c30000ff : Symbios Logic Inc. (formerly NCR) 53c875
|   c3001000-c3001fff : Symbios Logic Inc. (formerly NCR) 53c875
|   c4000000-c7ffffff : S3 Inc. 86c764/765 [Trio32/64/64V+]
| f7000000-f7ffffff : GG2 ISA mem
|   f70e0000-f70e7fff : NVRAM
| f8000000-f8ffffff : GG2 PCI I/O
| fec00000-fec7ffff : GG2 PCI cfg
| ff000000-ff7fffff : ROM exp
| fff80000-ffffffff : Flash ROM
| callisto$ 

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: IO issues vs. multiple busses

[Gabriel Paubert]

On Sat, 3 Mar 2001, Benjamin Herrenschmidt wrote:

> Right. That's my opinion too. But it's difficult to make everybody agree
> on it ;) Even the simple mecanism Paul Mackerras did so that IOs to
> non-existent devices don't kill the kernel (very small overhead) caused
> some barking ;)

Because it is ostrich's policy. I for one prefer machine checks and a dead
kernel to know where the problems are rather than silently ignoring them.
Fortunately I can patch my kernels, thanks god for open-source ;)

Having a special set of macros to tell that you want to handle machine
checks and return an error code in this case would be fine, burying the
head in the sand on _every_ I/O access is wrong.

	Regards,
	Gabriel.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/