Re: understanding PCI enumeration for hot-plug

[Grant Grundler <grundler@parisc-linux.org>]

On Fri, Feb 12, 2010 at 09:52:59AM +0530, Rajat Jain wrote:
> 
> Hello,
> 
> I need to support hot-plug in a system where I know in advance the
> devices that can be hot-plugged into the system. 
> 
> I'm trying to understand the PCI enumeration / re-enumeration process in
> the context of hot-plug / unplug. I'm assuming the case where the
> firmware is really dumb and the kernel needs to manage / allocate all
> PCI resources. 
> 
> I'm going through the PCI specs (and PCI-to-PCI bridge specs) and here
> is what I think needs to be done when ever PCI / PCIe devices are added
> / removed from the system. I would appreciate if some one could please
> confirm my understanding and point out if I am missing something:

Interrupt routing. Parent Bridge will have some associated entries for IRQs.
If using MSI or MSI-X (preferred), can avoid IRQ lines.

> PCI-REOURCE / BUS-NUMBER MGMT
> ==============> 
> [IFF NECESSARY] PCI configuration space of all the bridges needs to be
> re-written, right from the immediate parent of the device being removed
> until the host bridge in order to ensure that:
> 
> a) Each parent bridge has secondary and subordinate bus range set so as
> to include all the bus numbers below it. This is required to forward
> configuration transactions.
> 
> b) Each parent bridge has IO base and IO limit set so as to include all
> the IO address space below it.
> 
> c) Each parent bridge has Memory base and Memory limit set so as to
> include all the Memory address space below it.
> 
> d) Each parent bridge has Pre-fetch Memory base and Memory limit set so
> as to include all the Pre-fetch Memory address space below it.
> 
> Note-1: The reason a/b/c/d above are marked "IFF NECESSARY" is that we
> can avoid all the above work if we can pre-allocate the above resources
> for future devices, and set these parent bridges ranges accordingly. 
> 
> IN other words, consider a system where we know in advance, the PCI
> device tree on the devices that can be hot-plugged into the system. Here
> we can set aside PCI bus numbers and IO / Mem / Prefetchable memory
> ranges for them in advance. And thus configure the parent bridges to
> already include those ranges. Thus later when the devices are added,
> none of the parent bridges will need to be re-programmed. Is my
> understanding correct?

I think so.

> DEVICE DETECTION & INITIALIZATION
> ================> 
> e) Upon detection of the device (By attempting to read its config
> space), the most important item is to program its BARs to appropriate
> address spaces as requested by the device. The BARs need to be
> programmed such that they are included in the appropriate Base / limit
> registers of all the bridges upstream. Correct?

Yes.

> Again, if we've used the strategy specified in Note-1, we can simply use
> the range we've already aside for this device.
> 
> 
> MY QUESTIONS
> ======> 1) Is my above understanding correct?
> 
> 2) Does anything else also needs to be done in order to make it work?
> 
> 3) As I said I'm trying to make it fast and optimize for the case where
> I know the devices [thus the PCI tree] that can be plugged in. Will my
> strategy specified in Note-1 work? So all I need to configure is my
> newly detected devices / bridges and not the existing ones...

It's a bit hackish, but it should work.

grant

> 
> Thanks in Advance,
> 
> Rajat Jain
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html