Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset

[Jonathan Cameron <jonathan.cameron@huawei.com>]

On Wed, 21 Jan 2026 20:36:01 +0000
Vikram Sethi <vsethi@nvidia.com> wrote:

> Hi Jonathan,
> 
> From: Jonathan Cameron <jonathan.cameron@huawei.com>
> Date: Wednesday, January 21, 2026 at 5:20 AM
> To: Srirangan Madhavan <smadhavan@nvidia.com>
> Cc: dave@stgolabs.net <dave@stgolabs.net>, dave.jiang@intel.com <dave.jiang@intel.com>, alison.schofield@intel.com <alison.schofield@intel.com>, vishal.l.verma@intel.com <vishal.l.verma@intel.com>, ira.weiny@intel.com <ira.weiny@intel.com>, dan.j.williams@intel.com <dan.j.williams@intel.com>, bhelgaas@google.com <bhelgaas@google.com>, ming.li@zohomail.com <ming.li@zohomail.com>, rrichter@amd.com <rrichter@amd.com>, Smita.KoralahalliChannabasappa@amd.com <Smita.KoralahalliChannabasappa@amd.com>, huaisheng.ye@intel.com <huaisheng.ye@intel.com>, linux-cxl@vger.kernel.org <linux-cxl@vger.kernel.org>, linux-pci@vger.kernel.org <linux-pci@vger.kernel.org>, Vishal Aslot <vaslot@nvidia.com>, Vikram Sethi <vsethi@nvidia.com>, Shanker Donthineni <sdonthineni@nvidia.com>, Vidya Sagar <vidyas@nvidia.com>, Matt Ochs <mochs@nvidia.com>, Jason Sequeira <jsequeira@nvidia.com>
> Subject: Re: [PATCH v4 07/10] cxl: add host cache flush and multi-function reset
> 

Hi Vikram, Happy new year!

> On Tue, 20 Jan 2026 22:26:07 +0000
> smadhavan@nvidia.com wrote:
> 
> >> From: Srirangan Madhavan <smadhavan@nvidia.com>
> >>Flush host CPU caches for mapped HDM ranges after teardown and prepare
> >> sibling Type 2 functions on multi-function devices. The host cache
> > >maintenance uses wbinvd_on_all_cpus() on x86 and VA-based PoC clean+  
> 
> >That's not sufficient in general on arm64.  It might flush far enough
> >or there might be buffers beyond the point of coherence that are not
> >flushed.  
> snip
> 
> >Needs to some sort of arch_ call as well, not hidden in the cxl driver.  
> 
> Yes, and in fact it cannot be a cache flush by VA while PTE mappings
> to the memory are valid, as the core can prefetch the lines back
> right away, and later evictions post device reset can cause device
> errors as the device snoop filter isn’t aware of host having any
> lines.

Prefetching should (hopefully) not make any dirty lines. Hopefully
no one does clean writebacks (and there is a way to check if they
do in the ACPI tables).  You need to flush again after to force out
that stale stuff though before any demand fetches occur.

+ don't forget PA based prefetchers.  Doesn't even need to
be a mapping for fetches into distance caches to happen.  A given
platform might have restrictions on where those PA prefetchers will
go but we currently have no way to discover that.

I need to think a bit more on this as there are some scary
comments in the spec on CXL.reset such as all CXL.mem reads
are dropped (timeout fun). Mind you a device is permitted
to do that anyway before cxl.mem is enabled, so hopefully n
one times out if a prefetcher hits the device before that's on..

> The only way to do this correctly is via a SMC call to Arm
> trusted Firmware to handle in SOC specific way (after the memory has
> been offlined and removed from the PTEs), since there is no
> architectural way to flush by PA (set/way isn’t usable in the
> kernel). There was a SMC call defined for this in the PSCI spec 2
> years ago, but was removed for some reason. I had a discussion with
> some ARM folks maintaining the PSCI and SMCCC specifications and the
> direction was to bring the SMC call back in SMCCC specification. Like
> you say, that can be a separate patch series. Adding Souvik for SMCCC
> specification details once it is available.
> 

There are other possible paths which is where drivers/cache stuff
came from.+CC James Morse.

From a kernel point of view SMCCC needs to be just one option as a
bunch of hardware (I'll only point at ours as not sure what else is
public) provide MMIO accessible agents to do this stuff and going via
EL3 to talk to an engine the kernel can poke directly is silly.

I'm not against the PSCI thing coming back though if someone needs it.
Preferably without the CPU rendezvous stuff though -> or Linux can
just reject anyone who does that.

We could also revisit the approach of using an AML op region to issue
the SMCCC, thus allowing us to support vendor SMCCC calls (+ a general
one if ARM do bring that back) or whatever magic they want to use.
https://lore.kernel.org/all/20250820102950.175065-8-Jonathan.Cameron@huawei.com/

Advantage of that is you can wrap SMCCC, MMIO or whatever else you like
up and the kernel only needs one driver.  Disadvantage is need a spec
and only ACPI etc so we need a driver anyway if anyone cares on DT systems...

If you want to mess with that I can dig out the QEMU emulation and rebase
that driver.  I thought it was kind of cute, just a solution we didn't need
at the time!

Jonathan