Re: [PATCH 13/18] arm64: ioremap: use nGnRnE mappings on platforms that require it

Hector Martin
2021-02-09T00:26:58Z

On 09/02/2021 08.20, Mark Kettenis wrote:
> It is only PCI mmio space that needs to be nGnRE. The PCI host
> controller register space itself needs nGnRnE just like all other
> integrated peripherals (or at least it works that way).

This is correct. Actually, as I just discovered, nGnRE writes to MMIO
are not silently blackholed, but rather raise an SError. A certain other
Linux loader masks those SErrors in a vendor register completely
unnecessarily, which is why this isn't apparent when you use it. I never
noticed this myself until now because when I first ran into it, it was
breaking the UART, so of course I'd never see the SErrors, and I didn't
try again after I learned more about the L2C SError control mechanism :-)

Testing now, it seems we can actually fairly neatly figure out where
nGnRE is allowed and where not, as writes that fail due to that raise a
SError with L2C_ERR_INF=3.

I probed writing to i<<28 for i = [0..255], using nGnRE. This reveals
that nGnRE writes are allowed (i.e. either succeed or error out
differently) in the following ranges:

0x400000000 - 0x4ffffffff (apciec0)
0x580000000 - 0x67fffffff (apciec1)
0x6a0000000 - 0x6ffffffff (apcie)

Which matches the `ranges` properties of the respective apcie devices in
the ADT. The first two are obviously the TB3 ports, amd have more
features (three ranges instead of two, presumably IO port ranges are
supported on those, there's some extra DMA stuff, etc).

So the hardware behavior is to block nGnRE everywhere except in those
ranges (i.e. the nGnRnE fault takes precedence over other errors, like
the address not existing at all).

--
Hector Martin (marcan@marcan.st)
Public Key: https://mrcn.st/pub