Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

Sinan Kaya
2018-01-22T23:24:22Z

On 1/22/2018 5:51 PM, Bjorn Helgaas wrote:
> On Mon, Jan 22, 2018 at 05:04:03PM -0500, Sinan Kaya wrote:
>> On 1/22/2018 4:36 PM, Bjorn Helgaas wrote:

>>> Reducing MPS may be necessary if there are several devices in the
>>> hierarchy and one requires a smaller MPS than the others. That
>>> obviously reduces the maximum read and write performance.
>>>
>>> Reducing the MRRS may be useful to prevent one device from hogging
>>> a link, but of course, it reduces read performance for that device
>>> because we need more read requests.
>>
>> Maybe, a picture could help.
>>
>> root (MPS=256)
>> |
>> ------------------
>> / \
>> bridge0 (MPS=256) bridge1 (MPS=128)
>> / \
>> EP0 (MPS=256) EP1 (MPS=128)
>>
>> If I understood this right, code allows the configuration above with
>> the performance mode so that MPS doesn't have to be uniform across
>> the tree.
>
> Yes. In PERFORMANCE mode, we will set EP1's MRRS=128 and
> EP0's MRRS=256, just as you show.
>
>> It just needs to be consistent between the root port and endpoints.
>
> No, it doesn't need to be consistent. In PERFORMANCE mode, we'll set
> the root's MPS=256 and EP1's MPS=128.
>
> (I'm not actually 100% convinced that the PERFORMANCE mode approach of
> reducing MRRS is safe, necessary, and maintainable. I suspect that in
> many of the interesting cases, the device we care about is the only
> one below a Root Port, and we can get the performance we need by
> maximizing MPS and MRRS for that Root Port and its children,
> independent of the rest of the system.)

Maybe, I started seeing more and more NVMe devices behind a switch every
day. That's why, I'm concerned.

>
>> Why are we reducing MRRS in this case?
>
> We have to set EP1's MRRS=128 so it will never receive a completion
> larger than 128 bytes. If we set EP1's MRRS=256, it could receive
> 256-byte TLPs, which it would treat as malformed. (We also assume no
> peer-to-peer DMA that targets EP1.)

What if we were to keep root port MPS as 128? and not touch the BIOS
configured MRRS (4k) ?

Everybody should be happy, right?

I know there is a rule to check the completions against MPS. Root port
could generate transactions that is a multiple of 128 bytes for reads.

Is there any rule against checking incoming writes?

>
>> Are we assuming that root bus cannot handle more than 256 bytes and
>> bridge1 would be starved while root bus is passing the completions
>> to bridge0?
>
> We don't have to assume. Every device tells us via Dev Cap what size
> TLPs it can handle. In your example, I assume the root's Dev Cap
> tells us it supports 256-byte TLPs.
>
> PERFORMANCE mode reduces MRRS not because of a starvation issue, but
> because reducing EP1's MRRS allows EP0 to use a larger MPS.
>

--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.