[QUESTION] Early Write Acknowledge for PCIe configuration space

Will Deacon
2017-01-06T11:42:32Z

On Fri, Jan 06, 2017 at 12:24:25PM +0100, Arnd Bergmann wrote:
> On Friday, January 6, 2017 11:15:22 AM CET John Garry wrote:
> > [apologies if this has been queried before]
> >
> > Hi ARM guys,
> >
> > I have a question about the device memory attributes we assign for PCIe
> > config space for arm64. Currently we use ioremap to map in the config
> > space; this uses nGnRE, which means we enable Early Write Acknowledge.
> >
> > The ARMv8 ARM states that "ARM recommend that No Early Write Acknowledge
> > Hint is used for PCIe configuration writes".
> >
> > I understand a problem with using E is in that configuration write is a
> > non-post operation, which means the RP requires to get the completion
> > ack from the EP The problem here is if CPU writes data to ECAM by E,
> > complete will go back to CPU directly, and maybe at this point the write
> > has not reached the EP.
> >
> > I believe that this may cause ordering issues in PCI read/write. In
> > practice we use non-relaxed readl/writel to access config space, which
> > include the synchronization barriers, which, *as I understand*, even if
> > for full system domain, may be negated by the E attribute for PCIe.
>
> I don't think the barriers in readl/writel are enough here, in particular
> the write barrier is *before* the access to synchronize DMAs
> on RAM with MMIO accesses, which is a bit different from what you
> have here.
>
> > So a question: why is the recommendation in the ARMv8 ARM ignored?
>
> Probably nobody thought about this properly in the Linux drivers. The
> ARMv8 ARM sounds correct here.

The ARMv8 ARM also says that the E attribute is a hint, so there's no
guarantee that it's actually honoured by the implementation. However,
now that it explicitly mentions PCI config space, the intention is clearly
that nE *is* honoured for systems using PCIe, so I agree that we should make
this change. I don't want to use the nE type for all ioremap invocations,
though.

Will