Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya <okaya@codeaurora.org>]

On 1/31/2018 10:37 PM, Bjorn Helgaas wrote:
> On Wed, Jan 31, 2018 at 07:13:56PM -0500, Sinan Kaya wrote:
>> On 1/31/2018 7:01 PM, Myron Stowe wrote:
>>>> I think from above examples:
>>>> 1. perf mode is moving devices to 256 MPS as it can.
>>>> 2. safe mode is setting to 128 MPS
>>>> 3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.
>>>> 4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
>>>> 5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.
> 
>>> In Sinan's original posting, a reference to
>>> https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
>>> was provided.  When I read that paper and got to the "Read
>>> Completion Boundary" section I thought to myself: "If RCB can only
>>> be 64 or 128 bytes then what's the point of MPS (or MRRS) as all
>>> TLP completions would be limited to 64 or 128 bytes? (see also the
>>> paper's 'Read Completions with the RCB Set to 64 Bytes' figure)".
>>> I brought this up to a colleague and they surmised that possibly
>>> only _lower end_ (a.k.a. lazy) chipset implementations would truly
>>> have RCB limited sized completions; higher end chipsets would of
>>> course have to comply with RCB when communicating with the memory
>>> controller but could then aggregate data into larger MPS (or MRRS)
>>> sized TLP completion packets.  Perhaps this might explain why you
>>> always saw slot/switch/root values set at 128B?
>>
>> I have looked at that paper before. It is plain wrong. Read
>> Completion Boundary is about the alignment of addresses that an
>> endpoint is sending in memory read packets according to the spec. It
>> has nothing to do with the packet size.
> 
> I hope this isn't being too pedantic, but I think RCB applies to the
> *completer*, not the requester.  Typically an endpoint generates a
> memory read and a root port supplies data to complete it.
> 
> I didn't see anything in the "Read Completion Boundary" section of the
> paper that I thought *contradicted* the spec.  I did think this was
> speculation:
> 
>   Typically, most root complexes set the RCB at 64 bytes and return
>   data in 64-byte completions instead of what might be allowed by the
>   MPS.
> 
> That would be legal per spec, but I don't have direct knowledge of
> whether root complexes actually do that.  I tend to doubt it because
> it seems like people do observe performance differences based on the
> MPS setting.
> 
> I agree that the "Read Transaction Throughput" section is inaccurate
> when it says:
> 
>   The size of the completion is determined by a completer's read
>   completion boundary.
> 
> RCB may limit the size of the first completion to less than MPS, but
> subsequent completions (except the final one) only need to be a
> multiple of RCB.
> 
> RCB determines the boundaries at which a completer is allowed to end a
> TLP (PCIe r4.0, sec 2.3.1.1 has the rules).  Let's work through an
> example:
> 
>   MPS=128
>   RCB=64
> 
>   Endpoint generates a 256-byte read request for addresses 0x2-0x101
> 
> In the absence of RCB, a root port could satisfy this request with two
> completions:
> 
>   completion 0: bytes   0x2- 0x81  (128 bytes)  <-- illegal
>   completion 0: bytes  0x82-0x101  (128 bytes)
> 
> But with RCB, the first completion is illegal because it doesn't
> satisfy the entire request and it doesn't end on a multiple of RCB.
> Because of RCB the completer must respond with at least three
> completions.  The best it can do is this:
> 
>   completion 0: bytes   0x2- 0x7f  (126 bytes)
>   completion 1: bytes  0x80- 0xff  (128 bytes)
>   completion 2: bytes 0x100-0x101  (2 bytes)
> 
> It is *allowed* but not required to respond with more completions,
> e.g.,
> 
>   completion 0: bytes   0x2- 0x3f  (62 bytes)
>   completion 1: bytes  0x40- 0x7f  (64 bytes)
>   completion 2: bytes  0x80- 0xbf  (64 bytes)
>   completion 3: bytes  0xc0- 0xff  (64 bytes)
>   completion 4: bytes 0x100-0x101  (2 bytes)
> 

Very good summary. Thanks for doing a detailed analysis.


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.