Re: [nicholas.johnson-opensource@outlook.com.au: [PATCH v6 3/4] PCI: Fix bug resulting in double hpmemsize being assigned to MMIO window]

[Bjorn Helgaas <helgaas@kernel.org>]

On Thu, Jun 20, 2019 at 12:44:11AM +0000, Nicholas Johnson wrote:

> Correct me if I am wrong about assumptions about windows. My
> understanding cannot be perfect. As far as I know, 64-bit BARs
> should always be prefetchable, 

There's no requirement that a 64-bit BAR be prefetchable.

  - BARs of PCIe Functions must be prefetchable unless they have read
    side effects or can't tolerate write merging (PCIe r5.0, sec
    7.5.1.2.1).

  - BARs of PCIe Functions other than Legacy Endpoints must be 64-bit
    if they are prefetchable (sec 7.5.1.2.1).

  - Bridge non-prefetchable memory windows are limited to 32-bit
    (7.5.1.3.8).

  - There's some ambiguity in the spec about bridge prefetchable
    memory windows.  Current specs claim 64-bit addresses must be
    supported (sec 7.5.1.3.9), but also say the upper 32 bits are
    optional (sec 7.5.1.3.10).  Both 32- and 64-bit versions
    definitely exist.

> but I own the Aquantia AQC-107S NIC and it has three 64-bit non-pref
> BARs. It happens that they are assigned into the 32-bit window.

This is as it should be.  Non-prefetchable windows are 32 bits, and
in general non-prefetchable BARs must be placed there.

There is some wiggle room in pure PCIe systems because PCIe reads
always contain an explicit length, so in some cases it is safe to
put a non-prefetchable BAR in a prefetchable window (see the
implementation note in sec 7.5.1.2.1).  But I don't think Linux
currently implements this.