RFC: rewrite Documentation/pci.txt v3

RFC: rewrite Documentation/pci.txt v3

[Grant Grundler]

Greg, et al,
I've appended a rewrite of pci.txt.
I'm calling it "v3" only because I've gone through it three
full times.

This is an RFC because it might result in changes to the programming
model compared to what we have now.

The primary goal is to introduce the steps a PCI device driver needs
to take during initialization and shutdown of a PCI device.
Corollary to this is it means pci.txt will set expectations of when
things like MWI, MMIO resources, or IRQs needs to be enabled or
assigned by generic PCI code.

Secondary goals are "keep it short" and "cover all topics".
Yes, they are at odds with each other.

Some notes:
o I started this because of a email thread in June or early July about
  when to call request_mmio_region()...and then let it rot for a few
  monthes.  I just rediscovered my changes earlier this week and the
  may not include discussions that happened since July.

o IMHO, pci.txt needs to be short.
  Original version is about 12k. Mine is now 20k.  Not horribly
  long but it's starting to compete with LDD and I don't see
  a need for that.  Double points for suggesting things I can
  remove _and_ get a few people to agree with you.

o suspend/resume and hotplug are only mentioned in passing.
  Is that sufficient?

o Other Documentation/*.txt or sources/examples I should be pointing at
  instead of providing lengthy descriptions in pci.txt.


BTW, thanks to willy for doing an initial quick pass through it and catching
some gross errors. I'm sure there are more and they are not willy's fault.

thanks,
grant



			 How To Write Linux PCI Drivers

		   by Martin Mares <mj@ucw.cz> on 07-Feb-2000
	updated by Grant Grundler <grundler@parisc-linux.org> on 26-Jul-2006

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The world of PCI is vast and it's full of (mostly unpleasant) surprises.
Since each CPU architecture implements different chipsets and PCI devices
have different requirements (erm, "features"), the result is the PCI support
in the Linux kernel is not as trivial as one would wish. This short paper
tries to introduce all potential driver authors to Linux APIs for
PCI device drivers.

A more complete resource is the third edition of "Linux Device Drivers"
by Jonathan Corbet, Alessandro Rubini, and Greg Kroah-Hartman.
LDD3 is available for free (under Creative Commons License) from:

	http://lwn.net/Kernel/LDD3/

However, this document and the LDD3 are both subject to "bit rot".
Refer to the source code if things are not working as described here.

Please send questions/comments/patches about Linux PCI API to the
"Linux PCI" <linux-pci@atrey.karlin.mff.cuni.cz> mailing list
if code or Documentation/ files are not clear.


0. Structure of PCI drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~
PCI drivers "discover" PCI devices in a system via pci_register_driver().
Actually, it's the other way around. The PCI generic code will notify the
driver at every the PCI device which match a "description" advertised
by the driver. Details on this below.

pci_register_driver() leaves most of the probing for devices to
the PCI layer and supports online insertion/removal of devices [thus
supporting PCI, hot-pluggable PCI and CardBus in a single driver].
pci_register_driver() call requires passing in a table of function
calls and thus dictates the high level structure of a driver.

Once the driver knows about a PCI device and takes ownership, the
driver generally needs to perform the following initialization:

	request MMIO/IOP resources
	Enable the device
	set the DMA mask size (for both coherent and streaming DMA)
	allocate and initialize shared control data (pci_allocate_coherent())
	Access device configuration space (if needed)
	register IRQ handler (request_irq()) 
	Initialize non-PCI (ie LAN/SCSI/etc parts of the chip)
	enable DMA/processing engines.

When done using the device, and perhaps the module needs to be unloaded,
the driver needs to take the follow steps:
	disable the device from generating IRQs
	release the IRQ (free_irq())
	stop all DMA activity
	release DMA buffers (both streaming and coherent)
	unregister from other subsystems (e.g. scsi or netdev)
	Disable the device

Most of these topics are covered in the following sections.
For the rest look at LDD3 or <linux/pci.h> .

If the PCI subsystem is not configured (CONFIG_PCI is not set), most of
the PCI functions described below are defined as inline functions either
completely empty or just returning an appropriate error codes to avoid
lots of ifdefs in the drivers.


1. pci_register_driver() call
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
New-style drivers call pci_register_driver() during their
initialization with a pointer to a structure describing the driver
(struct pci_driver):

	field name	Description
	----------	------------------------------------------------------
	id_table	Pointer to table of device ID's the driver is
			interested in.  Most drivers should export this
			table using MODULE_DEVICE_TABLE(pci,...).

	probe		This probing function gets called (during execution
			of pci_register_driver() for already existing
			devices or later if a new device gets inserted) for
			all PCI devices which match the ID table and are not
			"owned" by the other drivers yet. This function gets
			passed a "struct pci_dev *" for each device whose
			entry in the ID table matches the device. The probe
			function returns zero when the driver chooses to
			take "ownership" of the device or an error code
			(negative number) otherwise.
			The probe function always gets called from process
			context, so it can sleep.

	remove		The remove() function gets called whenever a device
			being handled by this driver is removed (either during
			deregistration of the driver or when it's manually
			pulled out of a hot-pluggable slot).
			The remove function always gets called from process
			context, so it can sleep.

	save_state	Save a device's state before it is suspended.

	suspend		Put device into low power state.

	resume		Wake device from low power state.

	enable_wake	Enable device to generate wake events from a low power
			state.

			(Please see Documentation/power/pci.txt for descriptions
			of PCI Power Management and the related functions)


The ID table is an array of struct pci_device_id ending with an all-zero entry.
Each entry consists of:

	vendor, device	Vendor and device ID to match (or PCI_ANY_ID)

	subvendor,	Subsystem vendor and device ID to match (or PCI_ANY_ID)
	subdevice,

	class		Device class, subclass, and "interface" to match.
			See Appendix D of the PCI Local Bus Spec or
			include/linux/pci_ids.h for a full list of classes.
			Most drivers do not need to specify class/class_mask
			as vendor/device is normally sufficient.

	class_mask	limit which sub-fields of the class field are compared.
			See drivers/scsi/sym53c8xx_2/ for example of usage.

	driver_data	Data private to the driver.
			Most drivers don't need to use driver_data field.
			Best practice is to use driver_data as an index
			into a static list of equivalent device types,
			instead of using it as a pointer.


Have a table entry {PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID}
to have probe() called for every PCI device known to the system.

New PCI IDs may be added to a device driver pci_ids table at runtime
as shown below:

echo "vendor device subvendor subdevice class class_mask driver_data" > \
  /sys/bus/pci/drivers/{driver}/new_id

All fields are passed in as hexadecimal values (no leading 0x).
Users need pass only as many fields as necessary:
	o vendor, device, subvendor, and subdevice fields default
	  to PCI_ANY_ID (FFFFFFFF),
	o class and classmask fields default to 0
	o driver_data defaults to 0UL.

Once added, the driver probe routine will be invoked for any unclaimed
PCI devices listed in it's (newly updated) pci_ids list.

Device drivers must initialize use_driver_data in the dynids struct
in their pci_driver struct prior to calling pci_register_driver in order
for the driver_data field to get passed to the driver. Otherwise, only a
0 is passed in that field.

When the driver exits, it just calls pci_unregister_driver() and the PCI layer
automatically calls the remove hook for all devices handled by the driver.

Please mark the initialization and cleanup functions where appropriate
(the corresponding macros are defined in <linux/init.h>):

	__init		Initialization code. Thrown away after the driver
			initializes.
	__exit		Exit code. Ignored for non-modular drivers.
	__devinit	Device initialization code. Identical to __init if
			the kernel is not compiled with CONFIG_HOTPLUG, normal
			function otherwise.
	__devexit	The same for __exit.

Tips on marks:
	o The module_init()/module_exit() functions (and all initialization
          functions called _only_ from these) should be marked __init/exit.

	o The struct pci_driver shouldn't be marked with any of these tags.

	o The ID table array should be marked __devinitdata.

	o The probe() and remove() functions (and all initialization
	  functions called only from these) should be marked __devinit/exit.

	o If the driver is not a hotplug driver then use only 
	  __init/exit and __initdata/exitdata.

        o Pointers to functions marked as __devexit must be created using
          __devexit_p(function_name).  That will generate the function
          name or NULL if the __devexit function will be discarded.


2. How to find PCI devices manually
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

PCI drivers should have a really good reason for not using the
pci_register_driver() interface to search for PCI devices.
A manual search may be performed using the following constructs.

Searching by vendor and device ID:

	struct pci_dev *dev = NULL;
	while (dev = pci_get_device(VENDOR_ID, DEVICE_ID, dev))
		configure_device(dev);

Searching by class ID (iterate in a similar way):

	pci_get_class(CLASS_ID, dev)

Searching by both vendor/device and subsystem vendor/device ID:

	pci_get_subsys(VENDOR_ID, DEVICE_ID, SUBSYS_VENDOR_ID, SUBSYS_DEVICE_ID, dev).

   You can use the constant PCI_ANY_ID as a wildcard replacement for
VENDOR_ID or DEVICE_ID.  This allows searching for any device from a
specific vendor, for example.

   These functions are hotplug-safe. They increment the reference count on
the pci_dev that they return. You must eventually (possibly at module unload)
decrement the reference count on these devices by calling pci_dev_put().


3. Device Initialization Steps
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As noted in introduction, most PCI drivers need the following steps
for device initialization:

	request MMIO/IOP resources
	Enable the device
	enable/program PCI Config capabilities (e.g. MSI)
	set the DMA mask size (for both coherent and streaming DMA)
	allocate and initialize shared control data (pci_allocate_coherent())
	register IRQ handler (request_irq()) 
	"other" device specific initialization (start DMA/Processing engine)

The driver can access PCI config space registers at any time.
(Well, almost. When running BIST, config space can go away...but
that will just result in a PCI Bus Master Abort).


3.1 Request MMIO/IOP resources
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Memory (MMIO), and I/O port addresses numbers should NOT be read directly
from the PCI device config space. Use the values in the pci_dev structure
as the PCI "bus address" might have been remapped to a "host physical"
address by the kernel.

See Documentation/IO-mapping.txt for how to access device registers
or device memory.

The device driver needs to call pci_request_region() to make sure
no other device is already using the same address resource. The driver
is expected to determine MMIO and IO Port resource availability
_before_ calling pci_enable_device().  Conversely, drivers should
call pci_release_region() AFTER calling pci_disable_device().
The idea is to prevent two devices colliding on the same address range.

Generic flavors of pci_request_region() are request_mem_region()
(for MMIO ranges) and request_region() (for IO Port ranges).
Use these for address resources that are not described by "normal" PCI
interfaces (e.g. BAR).


3.2 Enable the PCI device
~~~~~~~~~~~~~~~~~~~~~~~~~
   Before touching any device registers, you need to enable the PCI device
by calling pci_enable_device(). This will:
	o wakes up the device if it was in suspended state,
	o enable I/O and memory regions of the device,
	o allocates an IRQ (if BIOS did not),

NOTE: pci_enable_device() can fail! Check the return value.

   pci_set_master() will enable DMA by setting the bus master bit
in PCI_COMMAND register. It also fixes the latency timer value if
it's set to something bogus by the BIOS.

   If the PCI device can use the PCI Memory-Write-Invalidate transaction,
call pci_set_mwi().  This enables the PCI_COMMAND bit for Mem-Wr-Inval
and also ensures that the cache line size register is set correctly.
Check the return value of pci_set_mwi() as not all architectures
or chipsets may support Memory-Write-Invalidate.


3.2 Set the DMA mask size
~~~~~~~~~~~~~~~~~~~~~~~~~
[ If anything below doesn't make sense, it means you need to read
  Documentation/DMA-API.txt. This section is just a reminder that
  drivers need to indicate DMA capabilities of the device and is not
  an authoritative source for DMA interfaces. ]

While all drivers should explictly indicate the DMA capability
(e.g. 32 or 64 bit) of the PCI bus master, devices with more than
32-bit bus master capability for streaming data need the driver
to "register" this capability by calling pci_set_dma_mask() with
appropriate parameters.  In general this allows more efficient DMA
on systems where System RAM exists above 4G _physical_ address.

Drivers for all PCI-X and PCIe compliant devices must call
pci_set_dma_mask(). All PCI-X and PCIe compliant devices are 64-bit DMA
devices.

Similarly, drivers must also "register" this capability if the device
can directly address "consistent" in System RAM above 4G physical address.
Drivers for PCI-X and PCIe compliant devices should also call
pci_set_consistent_dma_mask().


3.3 Setup shared control data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once the DMA masks are set, the driver can allocate "consistent" (aka shared)
memory.  See Documentation/DMA-API.txt for a full description of
the DMA APIs. This section is just a reminder that it needs to be done
before enabling DMA on the device.


3.4 initialize device registers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Some drivers will need specific "capability" fields programmed
or other "vendor specific" register initialized or reset.
E.g. clearing pending interrupts.


3.5 register IRQ handler
~~~~~~~~~~~~~~~~~~~~~~~~
While calling request_irq() is the the last step describe here,
this is often just another intermediate step to initializing a device.
This step can often be deferred until the device is opened for use.

All interrupt handlers should be registered with IRQF_SHARED and use the
devid to map IRQs to devices (remember that all PCI interrupts are shared).

request_irq() will associate a interrupt handler and device handle
with an interrupt number. Historically interrupt numbers represent
IRQ lines which run from the PCI device to the Interrupt controller.
With MSI and MSI-X (more below) the interrupt number is a CPU "vector".

request_irq() also enables the interrupt. Make sure the device is
quiesced and does not have any interrupts pending before registering
the interrupt handler.

MSI and MSI-X are PCI capabilities. Both are "Message Signaled Interrupts"
which deliver interrupts to the CPU via a DMA write to a Local APIC.
The fundemental difference between MSI and MSI-X are how multiple
"vectors" get allocated. MSI requires contiguous blocks of vectors
while MSI-X can allocate several individual ones.

MSI capability can be enabled by calling pci_enable_msi() or
pci_enable_msix() before calling request_irq(). This causes
the PCI support to program CPU vector data into the PCI device
capability registers.

If your PCI device supports both, try to enable MSI-X first.
Only one can be enabled at a time.  Many architectures, chipsets,
or BIOSs do NOT support MSI or MSI-X and the call to pci_enable_msi/msix
will fail. This is important to note since many drivers have
two (or more) interrupt handlers: one for MSI/MSI-X and another for IRQs.
They choose which handler to register with request_irq() based on the
return value from pci_enable_msi/msix().

There are (at least) two really good reasons for using MSI:
1) MSI is an exclusive interrupt vector by definition.
   This means the interrupt handler doesn't have to verify that
   it's device caused the interrupt.

2) MSI avoids DMA/IRQ race conditions. DMA to host memory is guaranteed
   to be visible to the host CPU(s) when the MSI is delivered. This
   is important for both data coherency and avoiding stale control data.
   This guarantee allows the driver to omit MMIO reads to flush
   the DMA stream.
   
See drivers/infiniband/hw/mthca/ or drivers/net/tg3.c for examples
of MSI/MSI-X usage.


4  PCI device shutdown
~~~~~~~~~~~~~~~~~~~~~~
When a PCI device driver is being unloaded, most of the follow 
steps need to be performed:

	disable the device from generating IRQs
	release the IRQ (free_irq())
	stop all DMA activity
	release DMA buffers (both streaming and consistent)
	unregister from other subsystems (e.g. scsi or netdev)
	Disable device from responding to MMIO/IO Port addresses
	release MMIO/IO Port resource(s)


4.1 Stop IRQs on the device
~~~~~~~~~~~~~~~~~~~~~~~~~~~
How to do this is chip/device specific. If it's not done, it opens
the possibility of a "screaming interrupt" if (and only if)
the IRQ is shared with another device.

When the shared IRQ handler is "unhoooked", the remaining devices
using the same IIRQ still need the IRQ enabled. Thus if the "unhooked"
device asserts IRQ signal, the system wil respond assuming it was one
of the remaining devices asserted the IRQ line. Since none of the other
devices will handle the IRQ, the system will "hang" until it decides
the IRQ isn't going to get handled and masks the IRQ (100,000 iterations
later). Once the shared IRQ is masked, the remaining devices will
stop functioning properly. Not a nice situation.

This is another reason to use MSI or MSI-X if it's available.
MSI and MSI-X are defined to be exclusive interrupts and thus
are not susceptible to the "screaming interrupt" problem.


4.2 release the IRQ
~~~~~~~~~~~~~~~~~~~
Once the device is quiesced (no more IRQs), one can call free_irq().
This function will return control once any pending IRQs are handled,
"unhook" the drivers IRQ handler from that IRQ, and finally release
the IRQ if no one else is using it.


4.3 stop all DMA activity
~~~~~~~~~~~~~~~~~~~~~~~~~
It's extremely important to stop all DMA operations BEFORE attempting
to deallocate resources. Failure to do so can result in memory corruption,
hangs, and on some architectures with IOMMUs, hard crash.

Stopping DMA after stopping the IRQs helps avoid races where the
IRQ handler might restart DMA engines.

While this step sounds obvious and trivial, several "mature" drivers
didn't get this step right in the past.


4.4 release DMA buffers
~~~~~~~~~~~~~~~~~~~~~~~
Once DMA is stopped, clean up streaming DMA first.
ie. unmap data buffers and return buffers to "upstream"
owners if there is one.

Then clean up "consistent" buffers which contain the control data.

See Documentation/DMA-API.txt for details on unmapping interfaces.


4.5 unregister from other subsystems
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Most low level PCI device drivers support some other subsystem
like USB, ALSA, SCSI, NetDev, Infiniband, etc. Make sure your
driver isn't losing resources from that other subsystem.
If this happens, typically the symptom is an Oops (panic) when
the subsystem attempts to call into a driver that has been unloaded.


4.6 Disable device from responding to MMIO/IO Port addresses
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is just a call to pci_disable_device().
This is the symetric opposite of pci_enable_device().
Do not do anything with the device after calling pci_disable_device().


4.7 release MMIO/IO Port resource(s)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Call pci_release_region() to mark the MMIO or IO Port range as available.

Failure to do so usually results in the inability to reload the driver.


5. How to access PCI config space
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   You can use pci_(read|write)_config_(byte|word|dword) to access the config
space of a device represented by struct pci_dev *. All these functions return 0
when successful or an error code (PCIBIOS_...) which can be translated to a text
string by pcibios_strerror. Most drivers expect that accesses to valid PCI
devices don't fail.

   If you don't have a struct pci_dev available, you can call
pci_bus_(read|write)_config_(byte|word|dword) to access a given device
and function on that bus.

   If you access fields in the standard portion of the config header, please
use symbolic names of locations and bits declared in <linux/pci.h>.

   If you need to access Extended PCI Capability registers, just call
pci_find_capability() for the particular capability and it will find the
corresponding register block for you.


6. Other interesting functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pci_find_slot()			Find pci_dev corresponding to given bus and
				slot numbers.
pci_set_power_state()		Set PCI Power Management state (0=D0 ... 3=D3)
pci_find_capability()		Find specified capability in device's capability
				list.
pci_module_init()		Inline helper function for ensuring correct
				pci_driver initialization and error handling.
pci_resource_start()		Returns bus start address for a given PCI region
pci_resource_end()		Returns bus end address for a given PCI region
pci_resource_len()		Returns the byte length of a PCI region
pci_set_drvdata()		Set private driver data pointer for a pci_dev
pci_get_drvdata()		Return private driver data pointer for a pci_dev
pci_set_mwi()			Enable Memory-Write-Invalidate transactions.
pci_clear_mwi()			Disable Memory-Write-Invalidate transactions.


7. Miscellaneous hints
~~~~~~~~~~~~~~~~~~~~~~
When displaying PCI device names to the user (for example when a driver wants
to tell the user what card has it found), please use pci_name(pci_dev)
for this purpose.

Always refer to the PCI devices by a pointer to the pci_dev structure.
All PCI layer functions use this identification and it's the only
reasonable one. Don't use bus/slot/function numbers except for very
special purposes -- on systems with multiple primary buses their semantics
can be pretty complex.

Don't try to turn on Fast Back to Back writes in your driver.  All devices
on the bus need to be capable of doing it, so this is something which needs
to be handled by platform and generic code, not individual drivers.


8. Vendor and device identifications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please do not add new device ids to include/linux/pci_ids.h.
Use PCI_VENDOR_ID_xxx for vendors, and a hex constant for device ids.

PCI_VENDOR_ID_xxx constants are re-used. The device ids are arbitrary
hex numbers (vendor controlled) and normally used only in a single
location, the pci_device_id table.

Please DO submit new vendor/device ids to pciids.sourceforge.net project.


9. Obsolete functions
~~~~~~~~~~~~~~~~~~~~~
There are several functions which you might come across when trying to
port an old driver to the new PCI interface.  They are no longer present
in the kernel as they aren't compatible with hotplug or PCI domains or
having sane locking.

pci_find_device()		Superseded by pci_get_device()
pci_find_subsys()		Superseded by pci_get_subsys()
pci_find_slot()			Superseded by pci_get_slot()


The alternative is the traditional PCI device driver that walks PCI
device lists. This is still possible but discouraged.

Re: RFC: rewrite Documentation/pci.txt v3

[Andi Kleen]

On Saturday 21 October 2006 09:14, Grant Grundler wrote:

I think you should add a note somewhere that the driver
should be prepared to initialize the device from any
state it put it in before. That is important for kexec
which tries to reinit the drivers after a crash.

And add a reference to kerneldoc somewhere too? There
is some PCI documentation in there (although it probably
could need some overall editing) 

> 3.2 Set the DMA mask size
> ~~~~~~~~~~~~~~~~~~~~~~~~~
> [ If anything below doesn't make sense, it means you need to read
>   Documentation/DMA-API.txt. This section is just a reminder that
>   drivers need to indicate DMA capabilities of the device and is not
>   an authoritative source for DMA interfaces. ]
> 
> While all drivers should explictly indicate the DMA capability

I think it's ok to rely on the defaults if 0xffffffff is suitable

And perhaps one sentence what a consistent allocation is.

> 4  PCI device shutdown
> ~~~~~~~~~~~~~~~~~~~~~~
> When a PCI device driver is being unloaded, most of the follow 
> steps need to be performed:
> 
> 	disable the device from generating IRQs
> 	release the IRQ (free_irq())
> 	stop all DMA activity
> 	release DMA buffers (both streaming and consistent)
> 	unregister from other subsystems (e.g. scsi or netdev)
> 	Disable device from responding to MMIO/IO Port addresses
> 	release MMIO/IO Port resource(s)


put the pci_dev you got during explicit probing? For the normal
probing path the higher layer takes care of it I think.

 
>    If you don't have a struct pci_dev available, you can call
> pci_bus_(read|write)_config_(byte|word|dword) to access a given device
> and function on that bus.


This shouldn't happen in driver code right? I would remove it.

People who want to do such low level things can read the source
or kerneldoc.
 
>    If you access fields in the standard portion of the config header, please
> use symbolic names of locations and bits declared in <linux/pci.h>.
> 
>    If you need to access Extended PCI Capability registers, just call
> pci_find_capability() for the particular capability and it will find the
> corresponding register block for you.
> 
> 
> 6. Other interesting functions
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> pci_find_slot()			Find pci_dev corresponding to given bus and
> 				slot numbers.

Again i don't think that fits here.

> pci_set_power_state()		Set PCI Power Management state (0=D0 ... 3=D3)
> pci_find_capability()		Find specified capability in device's capability
> 				list.
> pci_module_init()		Inline helper function for ensuring correct
> 				pci_driver initialization and error handling.

Perhaps that should be described at the beginning?

> pci_resource_start()		Returns bus start address for a given PCI region
> pci_resource_end()		Returns bus end address for a given PCI region
> pci_resource_len()		Returns the byte length of a PCI region

And that in the section for resources?

-Andi

Re: RFC: rewrite Documentation/pci.txt v3

[Grant Grundler]

Andi,
Thanks for the feedback.

One contentious issue you didn't mention was ordering of
when to call pci_request_region() and  pci_enable_device().

I've been thinking about it and I think the API design is just
broken right now.  We shouldn't need pci_request_resource() at all.
If pci_enable_device() is expected to allocate requested resource
types (MMIO vs IO Port) when BIOS does not, then it should also
reserve resources when BIOS or some other code did the allocation.

For "normal" PCI BARs,  something like:
	pci_enable_device(pdev,resource_type_mask_to_enable);
	dev->base = ioremap_nocache(pdev, pci_resource(xxx));

should be sufficient. Or am I missing something?

I'm sure MMIO/IO Port resources NOT described by PCI config space can
be described by generic request_resource() (note the lack of "pci_").


On Sat, Oct 21, 2006 at 02:55:29PM +0200, Andi Kleen wrote:
> On Saturday 21 October 2006 09:14, Grant Grundler wrote:
> 
> I think you should add a note somewhere that the driver
> should be prepared to initialize the device from any
> state it put it in before. That is important for kexec
> which tries to reinit the drivers after a crash.

Yes, I agree - also so the driver can run under different
types of BIOS (different architectures).
I'll think about it more.

> And add a reference to kerneldoc somewhere too? There
> is some PCI documentation in there (although it probably
> could need some overall editing) 

I need more specific guidance on this.

> > 3.2 Set the DMA mask size
> > ~~~~~~~~~~~~~~~~~~~~~~~~~
> > While all drivers should explictly indicate the DMA capability
> 
> I think it's ok to rely on the defaults if 0xffffffff is suitable

I want the driver writer to think about this. I hope it
gets them to verify the device DMA capabilities and if it's
something other than 32-bit, get motivated to make use
of those features. 

E.g. lots of SATA/IDE devices are 64-bit DMA capable but it
looks like generic code is leaving them at 32-bit.
Not good if the box is > 4GB (and many are now).


> And perhaps one sentence what a consistent allocation is.

Hrm...ok. I'm not enthusiastic about this though.

I've read your other comments and no issues with them.

thanks again!
grant

Re: RFC: rewrite Documentation/pci.txt v3

[Andi Kleen]

On Sunday 22 October 2006 21:03, Grant Grundler wrote: 

>One contentious issue you didn't mention was ordering of
>when to call pci_request_region() and  pci_enable_device().

I don't have a opinion on that.

> > And add a reference to kerneldoc somewhere too? There
> > is some PCI documentation in there (although it probably
> > could need some overall editing) 
> 
> I need more specific guidance on this.

The /** function arguments can be turned into Documentation.
When you do make pdfdocs you get a .pdf in Documentation/DocBook/kernel-api.pdf

Unfortunately it is right now broken and generates illegal pdfs
(and overall throws lots of errors). But maybe someone will fix it.

make mandocs seems to work though

There's also htmldocs and a couple of others.

Anyways, it's the most uptodate reference documentation because it's in the code.

> 
> > > 3.2 Set the DMA mask size
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~
> > > While all drivers should explictly indicate the DMA capability
> > 
> > I think it's ok to rely on the defaults if 0xffffffff is suitable
> 
> I want the driver writer to think about this. I hope it
> gets them to verify the device DMA capabilities and if it's
> something other than 32-bit, get motivated to make use
> of those features. 
> 
> E.g. lots of SATA/IDE devices are 64-bit DMA capable but it
> looks like generic code is leaving them at 32-bit.
> Not good if the box is > 4GB (and many are now).

Not a good example imho, usually to get >4GB on those you need to use a 
special interface, a standard SATA/IDE driver cannot do it.
It's a lot more than just thinking about it.

-Andi

Re: RFC: rewrite Documentation/pci.txt v3

[Grant Grundler]

On Sun, Oct 22, 2006 at 11:44:39PM +0200, Andi Kleen wrote:
...
> The /** function arguments can be turned into Documentation.
> When you do make pdfdocs you get a .pdf in
> Documentation/DocBook/kernel-api.pdf

Ah ok. I haven't look at DocBook in a long time.
I'll take a look at it before making references to it.

...
> > E.g. lots of SATA/IDE devices are 64-bit DMA capable but it
> > looks like generic code is leaving them at 32-bit.
> > Not good if the box is > 4GB (and many are now).
> 
> Not a good example imho, usually to get >4GB on those you need to use a 
> special interface, a standard SATA/IDE driver cannot do it.
> It's a lot more than just thinking about it.

IDE subsystem is a bad example in that it predates PCI.
But IDE is a good example as any.  Changes to subsystems often happen
when at least two people "think about it": the driver writer and
the subsystem maintainer.

ide_build_dmatable() follows some standard which dictates (and HW implements)
an SG list structure with 32-bit addresses and 16-bit counters.
New PCI devices are certainly capable of more than that.

So I'm now certain I want to keep references to pci_set_dma_mask()
in this document since some SATA (not IDE) could do something different
if the SATA device driver would advertise it was 64-bit capable.
I know jgarzik is smart and I hope he's already provided some way
for libata users to indicate their devices are 64-bit capable.
(Maybe Docbook would tell me that - checking :)

thanks,
grant

Re: RFC: rewrite Documentation/pci.txt v3

[Randy Dunlap]

On Sat, 21 Oct 2006 01:14:36 -0600 Grant Grundler wrote:

> 0. Structure of PCI drivers
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> PCI drivers "discover" PCI devices in a system via pci_register_driver().
> Actually, it's the other way around. The PCI generic code will notify the
> driver at every the PCI device which match a "description" advertised
         whenever ?

> by the driver. Details on this below.
> 
> 1. pci_register_driver() call
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> New-style drivers call pci_register_driver() during their
> initialization with a pointer to a structure describing the driver
> (struct pci_driver):

Could/would you add all of this to include/linux/pci.h as
kernel-doc?
although that would require that _all_ struct fields be described.

> 	field name	Description
> 	----------	------------------------------------------------------

> 			(Please see Documentation/power/pci.txt for descriptions
> 			of PCI Power Management and the related functions)

end with .)


> The ID table is an array of struct pci_device_id ending with an all-zero entry.
                                                  ^entries ?

> Once added, the driver probe routine will be invoked for any unclaimed
> PCI devices listed in it's (newly updated) pci_ids list.
*                       its

> Tips on marks:
> 	o The module_init()/module_exit() functions (and all initialization
>           functions called _only_ from these) should be marked __init/exit.
> 
> 	o The struct pci_driver shouldn't be marked with any of these tags.
> 
> 	o The ID table array should be marked __devinitdata.
> 
> 	o The probe() and remove() functions (and all initialization
> 	  functions called only from these) should be marked __devinit/exit.
> 
> 	o If the driver is not a hotplug driver then use only 
> 	  __init/exit and __initdata/exitdata.

I'd prefer to see exit, exitdata, etc., spelled with leading __.

>         o Pointers to functions marked as __devexit must be created using
>           __devexit_p(function_name).  That will generate the function
>           name or NULL if the __devexit function will be discarded.

> 3. Device Initialization Steps
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> As noted in introduction, most PCI drivers need the following steps
*            ^the

> for device initialization:

> 3.1 Request MMIO/IOP resources
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Memory (MMIO), and I/O port addresses numbers should NOT be read directly

not both addresses and numbers...

> from the PCI device config space. Use the values in the pci_dev structure
> as the PCI "bus address" might have been remapped to a "host physical"
> address by the kernel.
> 

> 3.2 Enable the PCI device
> ~~~~~~~~~~~~~~~~~~~~~~~~~
>    Before touching any device registers, you need to enable the PCI device
s/you need/the driver needs/

> by calling pci_enable_device(). This will:
> 	o wakes up the device if it was in suspended state,
*         wake
> 	o enable I/O and memory regions of the device,
> 	o allocates an IRQ (if BIOS did not),
*         allocate
+ s/,/./

> NOTE: pci_enable_device() can fail! Check the return value.
> 
>    pci_set_master() will enable DMA by setting the bus master bit
> in PCI_COMMAND register. It also fixes the latency timer value if
*   ^the

> it's set to something bogus by the BIOS.
> 

> 3.2 Set the DMA mask size
> ~~~~~~~~~~~~~~~~~~~~~~~~~

> Similarly, drivers must also "register" this capability if the device
> can directly address "consistent" in System RAM above 4G physical address.
* "consistent" <what> ?

> Drivers for PCI-X and PCIe compliant devices should also call
> pci_set_consistent_dma_mask().

> 3.5 register IRQ handler
> ~~~~~~~~~~~~~~~~~~~~~~~~
> While calling request_irq() is the the last step describe here,
> this is often just another intermediate step to initializing a device.
> This step can often be deferred until the device is opened for use.
> 
> All interrupt handlers should be registered with IRQF_SHARED and use the
> devid to map IRQs to devices (remember that all PCI interrupts are shared).
* (excluding MSI)

> If your PCI device supports both, try to enable MSI-X first.
> Only one can be enabled at a time.  Many architectures, chipsets,
> or BIOSs do NOT support MSI or MSI-X and the call to pci_enable_msi/msix
* BIOSes ?

> will fail. This is important to note since many drivers have
> two (or more) interrupt handlers: one for MSI/MSI-X and another for IRQs.
> They choose which handler to register with request_irq() based on the
> return value from pci_enable_msi/msix().
> 
> There are (at least) two really good reasons for using MSI:
> 1) MSI is an exclusive interrupt vector by definition.
>    This means the interrupt handler doesn't have to verify that
>    it's device caused the interrupt.
*    its

> 4  PCI device shutdown
  4.

> ~~~~~~~~~~~~~~~~~~~~~~

> 4.1 Stop IRQs on the device
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> How to do this is chip/device specific. If it's not done, it opens
> the possibility of a "screaming interrupt" if (and only if)
> the IRQ is shared with another device.
> 
> When the shared IRQ handler is "unhoooked", the remaining devices
> using the same IIRQ still need the IRQ enabled. Thus if the "unhooked"

 IIRQ ?

> device asserts IRQ signal, the system wil respond assuming it was one
> of the remaining devices asserted the IRQ line. Since none of the other
> devices will handle the IRQ, the system will "hang" until it decides
> the IRQ isn't going to get handled and masks the IRQ (100,000 iterations
> later). Once the shared IRQ is masked, the remaining devices will
> stop functioning properly. Not a nice situation.
> 

> 4.4 release DMA buffers
> ~~~~~~~~~~~~~~~~~~~~~~~
> Once DMA is stopped, clean up streaming DMA first.
> ie. unmap data buffers and return buffers to "upstream"

  i.e.

> owners if there is one.
> 
> 4.6 Disable device from responding to MMIO/IO Port addresses
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> This is just a call to pci_disable_device().
> This is the symetric opposite of pci_enable_device().

              symmetric

> Do not do anything with the device after calling pci_disable_device().

HTH.
---
~Randy

Re: RFC: rewrite Documentation/pci.txt v3

[Grant Grundler]

Randy,
many thanks for the thorough review...lots of silly grammar things
that I should have but just didn't see.

On Mon, Oct 23, 2006 at 12:20:23PM -0700, Randy Dunlap wrote:
> On Sat, 21 Oct 2006 01:14:36 -0600 Grant Grundler wrote:
> 
> > 0. Structure of PCI drivers
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > PCI drivers "discover" PCI devices in a system via pci_register_driver().
> > Actually, it's the other way around. The PCI generic code will notify the
> > driver at every the PCI device which match a "description" advertised
>          whenever ?

Yes. :)
I've rephrased that sentence to be a bit clearer:
	When the PCI generic code discovers a new device, the driver
	with a matching "description" will be notified.

Remember, this document is an introduction, not a complete reference.

> > by the driver. Details on this below.
> > 
> > 1. pci_register_driver() call
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > New-style drivers call pci_register_driver() during their
> > initialization with a pointer to a structure describing the driver
> > (struct pci_driver):
> 
> Could/would you add all of this to include/linux/pci.h as
> kernel-doc?

Hrm...that's a good idea. I could work on that as time permits
but I've got several other equally important things queued up.

This would also be a good "intern" project if a kernel newbie is
looking for something useful to do. I'd be happy to advise/review
results as needed.

> although that would require that _all_ struct fields be described.
> 
> > 	field name	Description
> > 	----------	------------------------------------------------------
> 
> > 			(Please see Documentation/power/pci.txt for descriptions
> > 			of PCI Power Management and the related functions)
> 
> end with .)

fixed 

> > The ID table is an array of struct pci_device_id ending with an all-zero entry.
>                                                   ^entries ?

fixed

> > Once added, the driver probe routine will be invoked for any unclaimed
> > PCI devices listed in it's (newly updated) pci_ids list.
> *                       its

fixed (*ugh* Jes, can I borrow some brown paper bags from you? ;)

> > 	o If the driver is not a hotplug driver then use only 
> > 	  __init/exit and __initdata/exitdata.
> 
> I'd prefer to see exit, exitdata, etc., spelled with leading __.

I agree. Fixed several of those in than section. I think I got them all.

> 
> >         o Pointers to functions marked as __devexit must be created using
> >           __devexit_p(function_name).  That will generate the function
> >           name or NULL if the __devexit function will be discarded.
> 
> > 3. Device Initialization Steps
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 
> > As noted in introduction, most PCI drivers need the following steps
> *            ^the
> 
> > for device initialization:
> 
> > 3.1 Request MMIO/IOP resources
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 
> > Memory (MMIO), and I/O port addresses numbers should NOT be read directly
> 
> not both addresses and numbers...

Correct...just addresses.

> > from the PCI device config space. Use the values in the pci_dev structure
> > as the PCI "bus address" might have been remapped to a "host physical"
> > address by the kernel.
> > 
> 
> > 3.2 Enable the PCI device
> > ~~~~~~~~~~~~~~~~~~~~~~~~~
> >    Before touching any device registers, you need to enable the PCI device
> s/you need/the driver needs/
> 
> > by calling pci_enable_device(). This will:
> > 	o wakes up the device if it was in suspended state,
> *         wake
> > 	o enable I/O and memory regions of the device,
> > 	o allocates an IRQ (if BIOS did not),
> *         allocate
> + s/,/./

thanks - all three fixed

> > NOTE: pci_enable_device() can fail! Check the return value.
> > 
> >    pci_set_master() will enable DMA by setting the bus master bit
> > in PCI_COMMAND register. It also fixes the latency timer value if
> *   ^the

fixed

> 
> > it's set to something bogus by the BIOS.
> > 
> 
> > 3.2 Set the DMA mask size
> > ~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> > Similarly, drivers must also "register" this capability if the device
> > can directly address "consistent" in System RAM above 4G physical address.
> * "consistent" <what> ?

ah sorry: consistent memory.

> > Drivers for PCI-X and PCIe compliant devices should also call
> > pci_set_consistent_dma_mask().
> 
> > 3.5 register IRQ handler
> > ~~~~~~~~~~~~~~~~~~~~~~~~
> > While calling request_irq() is the the last step describe here,
> > this is often just another intermediate step to initializing a device.
> > This step can often be deferred until the device is opened for use.
> > 
> > All interrupt handlers should be registered with IRQF_SHARED and use the
> > devid to map IRQs to devices (remember that all PCI interrupts are shared).
> * (excluding MSI)

good catch. I believe this is part of the original text that
predates all the MSI support.

New text now reads:
	All interrupt handlers for IRQ lines should be registered with
	IRQF_SHAREDand use the devid to map IRQs to devices (remember
	that all PCI IRQ linescan be shared).

> 
> > If your PCI device supports both, try to enable MSI-X first.
> > Only one can be enabled at a time.  Many architectures, chipsets,
> > or BIOSs do NOT support MSI or MSI-X and the call to pci_enable_msi/msix
> * BIOSes ?

I don't know. I'll use BIOSes.

> 
> > will fail. This is important to note since many drivers have
> > two (or more) interrupt handlers: one for MSI/MSI-X and another for IRQs.
> > They choose which handler to register with request_irq() based on the
> > return value from pci_enable_msi/msix().
> > 
> > There are (at least) two really good reasons for using MSI:
> > 1) MSI is an exclusive interrupt vector by definition.
> >    This means the interrupt handler doesn't have to verify that
> >    it's device caused the interrupt.
> *    its
> 
> > 4  PCI device shutdown
>   4.

Both fixed - thanks

> > ~~~~~~~~~~~~~~~~~~~~~~
> 
> > 4.1 Stop IRQs on the device
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > How to do this is chip/device specific. If it's not done, it opens
> > the possibility of a "screaming interrupt" if (and only if)
> > the IRQ is shared with another device.
> > 
> > When the shared IRQ handler is "unhoooked", the remaining devices
> > using the same IIRQ still need the IRQ enabled. Thus if the "unhooked"
> 
>  IIRQ ?

IRQ - fixed.

> 
> > device asserts IRQ signal, the system wil respond assuming it was one
> > of the remaining devices asserted the IRQ line. Since none of the other
> > devices will handle the IRQ, the system will "hang" until it decides
> > the IRQ isn't going to get handled and masks the IRQ (100,000 iterations
> > later). Once the shared IRQ is masked, the remaining devices will
> > stop functioning properly. Not a nice situation.
> > 
> 
> > 4.4 release DMA buffers
> > ~~~~~~~~~~~~~~~~~~~~~~~
> > Once DMA is stopped, clean up streaming DMA first.
> > ie. unmap data buffers and return buffers to "upstream"
> 
>   i.e.
> 
> > owners if there is one.
> > 
> > 4.6 Disable device from responding to MMIO/IO Port addresses
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > This is just a call to pci_disable_device().
> > This is the symetric opposite of pci_enable_device().
> 
>               symmetric

thanks - I just mispelled it...I will run this through ispell before
submitting it again for review.

> 
> > Do not do anything with the device after calling pci_disable_device().
> 
> HTH.

yes - it did help.

many thanks,
grant