[RFC PATCH 0/6] Cache coherency management subsystem

Jonathan Cameron
2025-03-20T17:43:30Z

Note that I've only a vague idea of who will care about this
so please do +CC others as needed.

On x86 there is the much loved WBINVD instruction that causes a write back
and invalidate of all caches in the system. It is expensive but it is
necessary in a few corner cases. These are cases where the contents of
Physical Memory may change without any writes from the host. Whilst there
are a few reasons this might happen, the one I care about here is when
we are adding or removing mappings on CXL. So typically going from
there being actual memory at a host Physical Address to nothing there
(reads as zero, writes dropped) or visa-versa. That involves the
reprogramming of address decoders (HDM Decoders); in the near future
it may also include the device offering dynamic capacity extents. The
thing that makes it very hard to handle with CPU flushes is that the
instructions are normally VA based and not guaranteed to reach beyond
the Point of Coherence or similar. You might be able to (ab)use
various flush operations intended to ensure persistence memory but
in general they don't work either.

So on other architectures such as ARM64 we have no instruction similar to
WBINVD but we may have device interfaces in the system that provide a way
to ensure a PA range undergoes the write back and invalidate action. This
RFC is to find a way to support those cache maintenance device interfaces.
The ones I know about are much more flexible than WBINVD, allowing
invalidation of particular PA ranges, or a much richer set of flush types
(not supported yet as not needed for upstream use cases).

To illustrate how a solution might work, I've taken both a HiSilicon
design (slight quirk as registers overlap with existing PMU driver)
and more controversially a firmware interface proposal from ARM
(wrapped up in made up ACPI) that was dropped from the released spec
but for which the alpha spec is still available.

Why drivers/cache?
- Mainly because it exists and smells like a reasonable place.
- Conor, you are maintainer for this currently do you mind us putting this
stuff in there?

Why not just register a singleton function pointer?
- Systems may include multiple cache control devices, responsible
for different parts of the PA address range (interleaving etc make
this complex). They may not all share a common hardware interface.
- A device class is more convenient than managing multiple homogeneous
device instances within a driver.
- Disadvantage is that we need this small class

Generalizing to more arch?
- I've started with ARM64, but if useful elsewhere the small amount
of arch code could be moved to a generic location.

QEMU emulation code at
http://gitlab.com/jic23/qemu cxl-2025-03-20

Why an RFC?
- I'm really just looking for feedback on whether the class approach
is the way to go at this stage. I'm not strongly attached to it but
it feels like the right balance of complexity and flexibility to me.
- I made up the ACPI spec - it's not documented, non official and
honestly needs work. I would however like to get feedback on whether
it is something we want to try and get through the ACPI Working group
as a much improved code first proposal? The potential justification
being to avoid the need for lots trivial drivers where maybe a bit
of DSDT interpreted code does the job better.

Jonathan Cameron (3):
cache: coherency device class
acpi: PoC of Cache control via ACPI0019 and _DSM
Hack: Pretend we have PSCI 1.2

Yicong Yang (3):
memregion: Support fine grained invalidate by
cpu_cache_invalidate_memregion()
arm64: Support ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION
cache: Support cache maintenance for HiSilicon SoC Hydra Home Agent

arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/cacheflush.h | 14 ++
arch/arm64/mm/flush.c | 42 ++++++
arch/x86/mm/pat/set_memory.c | 2 +-
drivers/acpi/Makefile | 1 +
drivers/cache/Kconfig | 26 ++++
drivers/cache/Makefile | 4 +
drivers/cache/acpi_cache_control.c | 157 ++++++++++++++++++++++
drivers/cache/coherency_core.c | 130 +++++++++++++++++++
drivers/cache/hisi_soc_hha.c | 193 ++++++++++++++++++++++++++++
drivers/cxl/core/region.c | 6 +-
drivers/firmware/psci/psci.c | 2 +
drivers/nvdimm/region.c | 3 +-
drivers/nvdimm/region_devs.c | 3 +-
include/linux/cache_coherency.h | 60 +++++++++
include/linux/memregion.h | 8 +-
16 files changed, 646 insertions(+), 6 deletions(-)
create mode 100644 drivers/cache/acpi_cache_control.c
create mode 100644 drivers/cache/coherency_core.c
create mode 100644 drivers/cache/hisi_soc_hha.c
create mode 100644 include/linux/cache_coherency.h

--
2.43.0