Re: [PATCH v2 00/24] ARM: zte: Add support for Sanechips ZX279128S and ZTE ZXHN H3600

[Agustin Santiago Isasmendi <isasmendi.agus@gmail.com>]

Hi Stefan,

On Thu, Nov 12, 2026 at 01:02, Stefan Dösinger wrote:
> Is it possible to program PCIe mmio areas as uncached in the level 2 cache or
> tell the cache to ignore a certain address range in some way? If it is
> possible, is it enough to prevent the crash?

The PCIe registers are already uncached.  The problem is that on this
chip every access from the CPUs goes through the L2 cache controller,
cached or not, and that controller is where it gets stuck.  This is how
it looks from my tests (there is no public documentation of the
chip's internals, so the picture is what the experiments show, not a
datasheet):

   CPU0      CPU1
     |         |
   +-------------+
   |     SCU     |   <- filter: can send a range to its 2nd port
   +-------------+
     |         |
   +-------------+
   |     L2C     |   <- both SCU ports end up here
   +-------------+
     |         |      <- filter: RAM on one port, the rest on the other
    RAM     PCIe, switch,
            other devices

How I got the picture:

- L2C filter: read back at boot, RAM goes out of one port and
  everything else out of the other.  Using it for the PCIe range moves
  RAM to the wrong port and the board doesn't boot.
- SCU 2nd port: RAM reads sent through it still show up in the L2C
  counters and return data that was only in the L2, so that port goes
  into the L2C too.  With the PCIe range on it, PCIe works but the
  board still hangs, 3 boots out of 3.

So both filters only change which port an access uses; neither lets it
skip the L2 cache controller.  Test code and logs are available if
anyone wants them.

> Do you have a guess what's special about the PCI address space that it
> triggers the issue? It sounds like it doesn't affect other hardware or RAM,
> otherwise ZTE would have disabled (or just fixed) the cache entirely.

It only hangs when three things happen together:

- one CPU reads a register of the PCIe card,
- the other CPU asks the L2 cache controller to do some work (the
  kernel does this all the time, e.g. on every writel()),
- the PCIe card is busy on its own (WiFi up).

With the WiFi down, 150 million PCIe reads did not hang it, even with
the other CPU keeping the L2 cache controller busy the whole time.
Why exactly these three block each other inside the chip I can't tell
without Sanechips' documentation.

This is a known problem.  Marvell had the same one in 2014, and ARM
(Catalin Marinas) said the rule is simply: don't access the L2 cache
controller and PCIe from the CPUs at the same time.

  https://lore.kernel.org/r/20140407121304.GD3360@arm.com

Marvell could avoid it because on their chip all devices share the
CPU caches, and they also turned off the extra L2 work the kernel does
on every writel(), so the kernel almost never has to talk to the L2
cache controller.  On our chip only the PCIe (and probably USB)
devices can share the caches, after setting a few registers that
ZTE's own kernel sets.  The Ethernet switch can't, so that extra work
on every writel() has to stay, and that alone is enough to hang it: I
ran ZTE's exact setup without its lock and the board hung the same
way.  That is why ZTE needs the lock.  Making PCIe share the caches
may be worth having on its own; I'll measure it and can send it
separately.

> What does "TRM" mean in this context?

Technical Reference Manual, ARM's documentation for the L2 cache
controller.

So my suggestion for the ZX279128S device tree is still to keep the L2
cache off (status = "disabled" on the L2 cache node, with the patch
below).  It is the only option that can't break with whatever drivers
come later, and the cost is acceptable for a router: forwarding with
the hardware offload is unchanged, only memory-heavy work on the CPU
gets slower.  The lock works but was found too invasive, and the
lock-free variants depend on every DMA user being coherent, which the
switch can't be.

The v2 of the patch, with the change Arnd agreed to:
https://lore.kernel.org/r/20261112072953.989664-1-isasmendi.agus@gmail.com

Agustin