Re: [PATCH 5/5] cxl/region: Manage CPU caches relative to DPA invalidation events

[Dan Williams <dan.j.williams@intel.com>]

[ add linux-arm-kernel@lists.infradead.org ]

Background for ARM folks, CXL can dynamically reconfigure the target
devices that back a given physical memory region. When that happens the
CPU cache can be holding cache data from a previous configuration. The
mitigation for that scenario on x86 is wbinvd, ARM does not have an
equivalent. The result, dynamic region creation is disabled on ARM. In
the near term, most CXL is configured pre-boot, but going forward this
restriction is untenable.

Davidlohr Bueso wrote:
> On Thu, 01 Dec 2022, Dan Williams wrote:
> 
> >A "DPA invalidation event" is any scenario where the contents of a DPA
> >(Device Physical Address) is modified in a way that is incoherent with
> >CPU caches, or if the HPA (Host Physical Address) to DPA association
> >changes due to a remapping event.
> >
> >PMEM security events like Unlock and Passphrase Secure Erase already
> >manage caches through LIBNVDIMM,
> 
> Just to be clear, is this is why you get rid of the explicit flushing
> for the respective commands in security.c?

Correct, because those commands can only be executed through libnvdimm.

> 
> >so that leaves HPA to DPA remap events
> >that need cache management by the CXL core. Those only happen when the
> >boot time CXL configuration has changed. That event occurs when
> >userspace attaches an endpoint decoder to a region configuration, and
> >that region is subsequently activated.
> >
> >The implications of not invalidating caches between remap events is that
> >reads from the region at different points in time may return different
> >results due to stale cached data from the previous HPA to DPA mapping.
> >Without a guarantee that the region contents after cxl_region_probe()
> >are written before being read (a layering-violation assumption that
> >cxl_region_probe() can not make) the CXL subsystem needs to ensure that
> >reads that precede writes see consistent results.
> 
> Hmm where does this leave us remaping under arm64 which is doesn't have
> ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION?
> 
> Back when we were discussing this it was all related to the security stuff,
> which under arm it could just be easily discarded as not available feature.

I can throw out a few strawman options, but really need help from ARM
folks to decide where to go next.

1/ Map and loop cache flushing line by line. It works, but for Terabytes
   of CXL the cost is 10s of seconds of latency to reconfigure a region.
   That said, region configuration, outside of test scenarios, is typically
   a "once per bare metal provisioning" event.

2/ Set a configuration dependency that mandates that all CXL memory be
   routed through the page allocator where it is guaranteed that the memory
   will be written (zeroed) before use. This restricts some planned use
   cases for the "Dynamic Capacity Device" capability.

3/ Work with the CXL consortium to extend the back-invalidate concept
   for general purpose usage to make devices capable of invalidating caches
   for a new memory region they joined, and mandate it for ARM. This one
   has a long lead time and a gap for every device in flight currently.