Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

Sinan Kaya
2018-02-01T15:14:14Z

On 1/31/2018 10:37 PM, Bjorn Helgaas wrote:
> On Wed, Jan 31, 2018 at 07:13:56PM -0500, Sinan Kaya wrote:
>> On 1/31/2018 7:01 PM, Myron Stowe wrote:
>>>> I think from above examples:
>>>> 1. perf mode is moving devices to 256 MPS as it can.
>>>> 2. safe mode is setting to 128 MPS
>>>> 3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.
>>>> 4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
>>>> 5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.
>
>>> In Sinan's original posting, a reference to
>>> https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
>>> was provided. When I read that paper and got to the "Read
>>> Completion Boundary" section I thought to myself: "If RCB can only
>>> be 64 or 128 bytes then what's the point of MPS (or MRRS) as all
>>> TLP completions would be limited to 64 or 128 bytes? (see also the
>>> paper's 'Read Completions with the RCB Set to 64 Bytes' figure)".
>>> I brought this up to a colleague and they surmised that possibly
>>> only _lower end_ (a.k.a. lazy) chipset implementations would truly
>>> have RCB limited sized completions; higher end chipsets would of
>>> course have to comply with RCB when communicating with the memory
>>> controller but could then aggregate data into larger MPS (or MRRS)
>>> sized TLP completion packets. Perhaps this might explain why you
>>> always saw slot/switch/root values set at 128B?
>>
>> I have looked at that paper before. It is plain wrong. Read
>> Completion Boundary is about the alignment of addresses that an
>> endpoint is sending in memory read packets according to the spec. It
>> has nothing to do with the packet size.
>
> I hope this isn't being too pedantic, but I think RCB applies to the
> *completer*, not the requester. Typically an endpoint generates a
> memory read and a root port supplies data to complete it.
>
> I didn't see anything in the "Read Completion Boundary" section of the
> paper that I thought *contradicted* the spec. I did think this was
> speculation:
>
> Typically, most root complexes set the RCB at 64 bytes and return
> data in 64-byte completions instead of what might be allowed by the
> MPS.
>
> That would be legal per spec, but I don't have direct knowledge of
> whether root complexes actually do that. I tend to doubt it because
> it seems like people do observe performance differences based on the
> MPS setting.
>
> I agree that the "Read Transaction Throughput" section is inaccurate
> when it says:
>
> The size of the completion is determined by a completer's read
> completion boundary.
>
> RCB may limit the size of the first completion to less than MPS, but
> subsequent completions (except the final one) only need to be a
> multiple of RCB.
>
> RCB determines the boundaries at which a completer is allowed to end a
> TLP (PCIe r4.0, sec 2.3.1.1 has the rules). Let's work through an
> example:
>
> MPS=128
> RCB=64
>
> Endpoint generates a 256-byte read request for addresses 0x2-0x101
>
> In the absence of RCB, a root port could satisfy this request with two
> completions:
>
> completion 0: bytes 0x2- 0x81 (128 bytes) <-- illegal
> completion 0: bytes 0x82-0x101 (128 bytes)
>
> But with RCB, the first completion is illegal because it doesn't
> satisfy the entire request and it doesn't end on a multiple of RCB.
> Because of RCB the completer must respond with at least three
> completions. The best it can do is this:
>
> completion 0: bytes 0x2- 0x7f (126 bytes)
> completion 1: bytes 0x80- 0xff (128 bytes)
> completion 2: bytes 0x100-0x101 (2 bytes)
>
> It is *allowed* but not required to respond with more completions,
> e.g.,
>
> completion 0: bytes 0x2- 0x3f (62 bytes)
> completion 1: bytes 0x40- 0x7f (64 bytes)
> completion 2: bytes 0x80- 0xbf (64 bytes)
> completion 3: bytes 0xc0- 0xff (64 bytes)
> completion 4: bytes 0x100-0x101 (2 bytes)
>

Very good summary. Thanks for doing a detailed analysis.

--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.