Re: Does my understanding correct?

[Don Dutile <ddutile@redhat.com>]

On 05/02/2012 10:59 AM, Bjorn Helgaas wrote:
> On Wed, May 2, 2012 at 12:24 AM, Richard Yang
> <weiyang@linux.vnet.ibm.com>  wrote:
>> On Mon, Apr 30, 2012 at 09:56:13AM -0600, Bjorn Helgaas wrote:
>>> On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
>>> <weiyang@linux.vnet.ibm.com>  wrote:
>>>
>> Thanks for your nice chart.
>>> I think assignments shown for the PCIe-to-PCI bridge are OK, although
>>> I would draw it like this because the bridge originates a single bus
>>> 02 that may have multiple devices attached to it (this side is PCI,
>>> not PCIe, so it really is a shared bus):
>>>
>>>                                  ^
>>>                                  |
>>>                         +--------+--------+
>>>                         |     00:02.0     |
>>>                         | PCIe-PCI bridge |
>>>                         |                 |
>>>                         +--------+--------+
>>>                                  |
>>>                                  |
>>>                       +---------------------+    Bus 02
>>>                       |                     |
>>>                       |                     |
>>>                       |                     |
>>>                  +----v----+           +----v----+
>>>                  | 02:00.0 |           | 02:01.0 |
>>>                  +---------+           +---------+
>>>
>> So for this case, there is not internal bus, while this is really a
>> physical shared bus, not a logical one.
>
> Yes.  The downstream side of the PCIe-PCI bridge is PCI.
>
>>> I think the PCIe switch part is incorrect.  Here's Figure 1-3 from sec
>>> 1.3.3 of the PCIe r3 spec:
>>
>>                                      ^
>>                                      |
>>      +-------------------------------|------------------------------+
>>      |                               |                              |
>>      |                          +----+----+                         |
>>      |                          | virtual |                         |
>>      |                          | PCI-PCI |                         |
>>      |                          | bridge  |                         |
>>      |                          +----+----+                         |
>>      |                               |                              |
>>      |                               |Bus#3                         |
>>      |                               |                              |
>>      |          +----------------------------------------+          |
>>      |          |                    |                   |          |
>>      |          |                    |                   |          |
>>      |          |03:00.0             |03:01.0            |03:02.0   |
>>      |     +----+----+          +----+----+         +----++---+     |
>>      |     | virtual |          | virtual |         | virtual |     |
>>      |     | PCI-PCI |          | PCI-PCI |         | PCI-PCI |     |
>>      |     | bridge  |          | bridge  |         | bridge  |     |
>>      |     +----+----+          +----+----+         +----+----+     |
>>      |          |Bus#4?              |                   |          |
>>      |     -----+-------             |                   |          |
>>      +----------|--------------------|-------------------|----------+
>>                 |                    |                   |
>>                 v                    v                   v
>>
>>
>>> A PCIe switch appears as two or more PCI-PCI bridges.  One is
>>> associated with the upstream port; the others with the downstream
>>> ports.
>>>
>>> A bridge always has a primary side and a secondary side.  In your
>>> diagram, the bridge associated with the upstream port would be 00:01.0
>>> (primary bus 00) and could have a secondary bus of 03 (since 02 is
>>> already consumed by the PCIe-PCI bridge).
>> Hmm... I am confused why is 03. 02 is used but 01 is not used.
>> Switch should be configured after PCIe2PCI bridge?
>
> It's likely that the PCIe switch would be configured first, since its
> device number is lower, but that is not a requirement.  The
> requirement is that the bus number ranges consumed by bridges be
> non-overlapping.  In this case (using your original topology plus my
> PCIe switch diagram), we'd have:
>
>    1. an endpoint at 00:00.0 -- consumes no additional buses
>    2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
>    3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
>
> Bus 03 is the internal PCIe switch bus that connects the upstream port
> to the downstream ports.  Buses 04, 05, and 06 are the links
> originating from the downstream ports.
>
>>> The bridges associated with the downstream ports are all logically on
>>> bus 03.  Their primary bus number would be 03; they might be 03:00.0,
>>> 03:01.0, 03:02.0, etc.  Each would have its own secondary bus number,
>>> for example 04, 05, 06.  That secondary bus number is for the
>>> downstream link from the corresponding downstream port.
>> Hmm, as you mentioned in previous letter, PCIe is an point-to-point
>> protocol, then the secondary bus should reside in the Switch?
>> Do you think my Bus#4 is correct?
>
> No.  Bus 04 is a PCIe link that connects the downstream port (03:00.0)
> to a single PCIe device.  That device (04:00) could be a PCIe
> endpoint, or it could be the upstream port of another PCIe switch.
> (If it is a switch, more bus numbers would be required.)
>
>>> The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0
>>> (or these could be the upstream ports of more PCIe switches).
>> So below the PCIe downstream port, there is only on PCIe device.
>> The whole bus is occupied by this device?
>> That is why the device could have upto 256 functions?
>
> Yes, if it supports ARI.
>
minor tweak: if the device *and* the downstream PCIe bridge support ARI...
both must exist.  I know of one such module, e.g., non-ARI PCIe switch w/PCIe endpoints)
and the endpoints have SRIOV/VF support... not a very smart hw combo. .. nuf said.

> Bjorn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html