Re: [PATCH] add legacy I/O and memory access routines to /proc/bus/pci API

[Jesse Barnes <jbarnes@engr.sgi.com>]

On Tuesday, December 14, 2004 3:55 pm, Bjorn Helgaas wrote:
> > +/proc/bus/pci contains a list of PCI devices available on the system
> > +and can be used for driving PCI devices from userspace, getting a list
> > +of PCI devices, and hotplug support.  Example:
> > +
> > +  $ tree /proc/bus/pci
> > +  /proc/bus/pci/
> > +  |-- 01  directory for devices on bus 1
> > +  |   |-- 01.0  single fn device in slot 1
>
> What about multiple PCI domains?  Should /proc/bus/pci be extended
> somehow to deal with them?

This might look different on a machine with domains, I ran the tree command on 
a machine /wo them.

> HP machines are currently configured to compress all PCI bus numbers
> into one domain, but we plan to turn off that compression soon, which
> means we'll have duplicate bus numbers in different domains.

Sure, ok.  Domains seem useful in other ways too, so I'm looking forward to 
seeing that.

> > +read
> > +  reads from either PCI config space or legacy I/O space, using the
> > +  current file postion, depending on the current I/O mode setting.
>
> I think by "legacy I/O space", you mean specifically "legacy
> I/O *port* space", right?  Maybe there's no current use for it,
> but I can imagine supporting MMIO accesses this way, too.

Yeah, I could clarify that, legacy memory space is supported via direct mmap.

> > +lseek
> > +  Can be used to set the current file position.  Note that the file
> > +  size is limited to 64k as that's how big legacy I/O space is.
>
> On i386, anyway ;-)  But on ia64, we support multiple 64k I/O port
> spaces (one of them being the 0-64K space that corresponds to the
> i386 "legacy" space).  Shouldn't we be able to access them with this
> interface, too?

Yeah, we could do that, any suggestions?  If I split the ioctl commands into 
PCIIOC_LEGACY_IO and PCIIOC_CONFIG the former could take an argument for the 
domain, would that work?  Then again, isn't having the pci_dev enough?

> > +ioctl
> > +  ioctl is used to set the mode of a subsequent read, write or mmap
> > +  call.  Available ioctls (in linux/pci.h) include
> > +    PCIIOC_CONTROLLER - return PCI domain number
> > +    PCIIOC_MMAP_IS_IO - next mmap maps to I/O space
> > +    PCIIOC_MMAP_IS_MEM - next mmap maps to memory space
> > +    PCIIOC_WRITE_COMBINE - try to use write gathering for the new
> > +                           region if the ioctl argument is true,
> > +      otherwise disable write gathering
> > +    PCIIOC_LEGACY_IO - read/write legacy I/O space if ioctl argument
> > +                       is true, otherwise subsequent read/writes will
> > +         go to config space
>
> Wouldn't it be nicer to have PCIIOC_{IO,CONFIG_SPACE}?  Then
> PCIIOC_MMIO could be added someday.  Using PCIIOC_LEGACY_IO with a
> boolean kind of locks you into having only two choices, ever.  (Well,
> you could extend the boolean to an int, but the *name* still suggests
> an on/off switch.)

Yeah, proc.c was a little inconsistent here.  Having two separate ioctls is ok 
with me.

> > +    PCIIOC_MMAP_IS_LEGACY_MEM - next mmap maps to legacy memory space
>
> I'm always confused about exactly what "legacy memory space" refers
> to.  I guess (from below) that it's MMIO space between 0 and 1M?

Yep.

> This seems similar to PCIIOC_MMAP_IS_MEM, except that for MMAP_IS_MEM,
> the user-supplied address is a host address that must be in one of the
> device's memory resources.  For MMAP_IS_LEGACY_MEM, it looks like you
> get a mapping to something in the first megabyte of the memory aperture
> that's routed to the device.

Exactly.

> So for MMAP_IS_LEGACY_MEM, you have no way of knowing whether the
> device will respond to the region you're mapping, right?

Correct.

> The comment on ia64_pci_get_legacy_mem() says we want to map the
> first megabyte of bus address space for the device.  But I don't
> think we can generate a bus address between 0-1M on any arbitrary
> PCI bus.  For example, the MMIO apertures on zx1 are rather small
> (on the order of 128-256MB), and we typically map them to non-
> overlapping bus address space to make peer-to-peer transactions
> possible.

Right, you can return -EINVAL or -ENODEV in this case.

> > + /* Do a legacy read of 1 byte (inb) from port 0x3cc */
> > + int fd = open("/proc/bus/pci/01/01.0", O_RDONLY);
> > + ioctl(fd, PCIIOC_LEGACY_IO, 1); /* enable legacy I/O */
> > + lseek(fd, 0x3cc, SEEK_SET);
>
> In this example, the particular device ("01/01.0") you open
> makes no difference, right?  The I/O port routing is determined
> by the chipset, not by which /proc/bus/pci/... file you open.

But the chipset can be programmed to route things correctly or remap the 
correct legacy I/O port domain in the callback routine.

> HP chipsets allow you to change the routing of VGA-related
> MMIO and I/O port space.  X currently twiddles this by hand.
> I've always thought there should be an ioctl that says "please
> route VGA resources to this PCI device", so X could do this
> twiddling in a chipset-independent way.

I think this routine could do that.

> I'd be interested in seeing the X side of your work, too.
> There's currently an ugly mess of figuring out what chipset
> we've got, fiddling with VGA routing and soft-fail settings,
> etc.  Maybe seeing your X support will help me figure out
> how to take advantage of your new kernel hooks to clean
> things up for zx1 and sx1000.

Hopefully we'll be able to post those shortly to xorg@freedesktop.org.  We're 
working through some driver issues atm.

Thanks,
Jesse