Deep flush support for CXL pmem?

Deep flush support for CXL pmem?

[Huang, Ying]

Hi, All,

When reading the documentation of PMDK (https://github.com/pmem/pmdk), I
found that deep flush is used for data loss recovery in addition to GPF.
With that, we can identify that a file isn't affected by the GPF failure
because it wasn't opened during the GPF failure.

However, IIUC, the Linux kernel doesn't support deep flush (via
nvdimm_flush()) for CXL pmem at least for now.  For NVDIMM, we have WPQ,
but it appears that we don't have that for CXL.  Is deep flush for CXL
defined in some spec?

---
Best Regards,
Huang, Ying

Re: Deep flush support for CXL pmem?

[Gregory Price]

On Fri, Oct 17, 2025 at 02:11:16PM +0800, Huang, Ying wrote:
> Hi, All,
> 
> When reading the documentation of PMDK (https://github.com/pmem/pmdk), I
> found that deep flush is used for data loss recovery in addition to GPF.
> With that, we can identify that a file isn't affected by the GPF failure
> because it wasn't opened during the GPF failure.
> 
> However, IIUC, the Linux kernel doesn't support deep flush (via
> nvdimm_flush()) for CXL pmem at least for now.  For NVDIMM, we have WPQ,
> but it appears that we don't have that for CXL.  Is deep flush for CXL
> defined in some spec?
>

Should note that this is relevant not just for pmem, but also for shared
memory use cases.  Last I looked there's nothing defined / implemented.

When I reviewed the cxl.mem protocol about a year or so ago, I came away
thinking a deep flush was going to be problematic for shared devices
(multiple separate coherency domains / non-coherent memory).

~Gregory

Re: Deep flush support for CXL pmem?

[Jonathan Cameron]

On Fri, 17 Oct 2025 10:25:26 -0400
Gregory Price <gourry@gourry.net> wrote:

> On Fri, Oct 17, 2025 at 02:11:16PM +0800, Huang, Ying wrote:
> > Hi, All,
> > 
> > When reading the documentation of PMDK (https://github.com/pmem/pmdk), I
> > found that deep flush is used for data loss recovery in addition to GPF.
> > With that, we can identify that a file isn't affected by the GPF failure
> > because it wasn't opened during the GPF failure.
> > 
> > However, IIUC, the Linux kernel doesn't support deep flush (via
> > nvdimm_flush()) for CXL pmem at least for now.  For NVDIMM, we have WPQ,
> > but it appears that we don't have that for CXL.  Is deep flush for CXL
> > defined in some spec?
> >  
> 
> Should note that this is relevant not just for pmem, but also for shared
> memory use cases.  Last I looked there's nothing defined / implemented.
> 
> When I reviewed the cxl.mem protocol about a year or so ago, I came away
> thinking a deep flush was going to be problematic for shared devices
> (multiple separate coherency domains / non-coherent memory).

Two things to deal with.  Getting it to the device.  For that what happens
is CPU architecture / host system architecture dependent. Not all (possibly
none) of the architectures have this well defined today - on some it
just works (cross your fingers that this is always the case). In some systems
using the persistent flushes gets it out on the bus whether or not the
CXL device is persistent.

There are ways around that, such as 'device' like entities on particular
systems that do this.  We have one in our Hydra Home Agent that is handled
by the cache maintenance 'subsystem' series (I'll send v4 of that early
next week). You can use that to guarantee caches are written back to
the device. We don't currently use it outside of capacity additions / removal
(DCD or address decoder programming routes).

The other thing is getting the device to actually write it to persistent
memory rather than relying on local or remote power to keep things up long
enough for writes to complete.

So I understand that there are gaps in what is guaranteed to do the job for both
deep flushes to persist memory and for shared memory use cases.

Jonathan

> 
> ~Gregory

Re: Deep flush support for CXL pmem?

[dan.j.williams]

Huang, Ying wrote:
> Hi, All,
> 
> When reading the documentation of PMDK (https://github.com/pmem/pmdk), I
> found that deep flush is used for data loss recovery in addition to GPF.
> With that, we can identify that a file isn't affected by the GPF failure
> because it wasn't opened during the GPF failure.
> 
> However, IIUC, the Linux kernel doesn't support deep flush (via
> nvdimm_flush()) for CXL pmem at least for now.  For NVDIMM, we have WPQ,
> but it appears that we don't have that for CXL.  Is deep flush for CXL
> defined in some spec?

No, and I think a dynamic "deep flush" mechanism is a mistake that CXL
should not pursue. (personal opinion, not necessarily opinion of
$employer)

"All storage is a lie" applies here in that all a deep flush mechanism
does is maybe reduce the occurrence of dirty-shutdown failures. It can
not guarantee that some other event causes the write to be dropped.

CXL does have a Global Persistence Flush mechanism that fires as the
system is dying, but it is not something that can be triggered at run
time.

So, either the system has the reserve energy to make sure that all
globally visible writes make it to persistent storage or it does not. If
it does not then it had better arrange for those failures to be flagged
via a dirty shutdown mechanism.

If a platform fires dirty-shutdown events at a significantly higher rate
than say the capacitor-protected write-cache on an SSD, then that feels
like an "improve the hardware" problem, not a "teach the software to
chase writes with a deep flush and hope it helps" problem.

See discussions like this for the last time mapping pmem deep flush was
attempted to be mapped to storage semantics like FUA (Force Unit
Access):

http://lore.kernel.org/YtefnyIvY9OdrVU5@infradead.org

Re: Deep flush support for CXL pmem?

[Davidlohr Bueso]

On Fri, 17 Oct 2025, dan.j.williams@intel.com wrote:

>CXL does have a Global Persistence Flush mechanism that fires as the
>system is dying, but it is not something that can be triggered at run
>time.
>
>So, either the system has the reserve energy to make sure that all
>globally visible writes make it to persistent storage or it does not. If
>it does not then it had better arrange for those failures to be flagged
>via a dirty shutdown mechanism.

In addition, GPF is purposefully vague about the details of how this is
correctly handled (ie phase 2), which imo implies eADR-like semantics,
as there is no extra programming burden. So the scope is CPU caches + pmem
write-caches, and hw vendors need to budget power accordingly.

Thanks,
Davidlohr

Re: Deep flush support for CXL pmem?

[Huang, Ying]

Hi, Dan,

<dan.j.williams@intel.com> writes:

> Huang, Ying wrote:
>> Hi, All,
>> 
>> When reading the documentation of PMDK (https://github.com/pmem/pmdk), I
>> found that deep flush is used for data loss recovery in addition to GPF.
>> With that, we can identify that a file isn't affected by the GPF failure
>> because it wasn't opened during the GPF failure.
>> 
>> However, IIUC, the Linux kernel doesn't support deep flush (via
>> nvdimm_flush()) for CXL pmem at least for now.  For NVDIMM, we have WPQ,
>> but it appears that we don't have that for CXL.  Is deep flush for CXL
>> defined in some spec?
>
> No, and I think a dynamic "deep flush" mechanism is a mistake that CXL
> should not pursue. (personal opinion, not necessarily opinion of
> $employer)
>
> "All storage is a lie" applies here in that all a deep flush mechanism
> does is maybe reduce the occurrence of dirty-shutdown failures. It can
> not guarantee that some other event causes the write to be dropped.
>
> CXL does have a Global Persistence Flush mechanism that fires as the
> system is dying, but it is not something that can be triggered at run
> time.
>
> So, either the system has the reserve energy to make sure that all
> globally visible writes make it to persistent storage or it does not. If
> it does not then it had better arrange for those failures to be flagged
> via a dirty shutdown mechanism.
>
> If a platform fires dirty-shutdown events at a significantly higher rate
> than say the capacitor-protected write-cache on an SSD, then that feels
> like an "improve the hardware" problem, not a "teach the software to
> chase writes with a deep flush and hope it helps" problem.
>
> See discussions like this for the last time mapping pmem deep flush was
> attempted to be mapped to storage semantics like FUA (Force Unit
> Access):
>
> http://lore.kernel.org/YtefnyIvY9OdrVU5@infradead.org

Thanks a lot for the detailed explanation and reference to the previous
discussion.  It's very helpful.

As you said, in fact, we cannot trust "deep flush" either.  So, it
provides little value on top of the dirty shutdown count.

---
Best Regards,
Huang, Ying