[RFC] block/nvme: exploring asynchronous durability notification semantics

[RFC] block/nvme: exploring asynchronous durability notification semantics

[Esteban Cerutti]

Hi,

I would like to explore whether current NVMe completion semantics
unnecessarily conflate execution completion with durability, and
whether there is room for a more explicit, asynchronous durability
notification model between host and device.

Today, a successful write completion indicates command execution,
but not necessarily physical persistence to non-volatile media unless
FUA or Flush is used. This forces the kernel and filesystems to assume
worst-case durability behavior and rely on synchronous flushes and
barriers for safety.

The device internally knows when data is staged in volatile buffers
versus committed to NAND (or equivalent persistent media), but this
information is not exposed to the host.

This RFC explores a potential extension model with two components:

1) Multi-phase completion semantics

   - Normal completion continues to signal execution.
   - The device assigns a persistence token ID.
   - When the data is physically committed to non-volatile media,
     the device emits an asynchronous durability confirmation
     referencing that token.

This would decouple execution throughput from durability
confirmation and potentially allow filesystems to close journal
transactions only upon confirmed persistence, without forcing
synchronous flush fences.

2) Advisory write intent classification

   - Host-provided hints such as EPHEMERAL, STANDARD, or CRITICAL.
   - CRITICAL writes would request immediate durability.
   - EPHEMERAL writes could tolerate extended volatile staging.

Additionally, I am curious whether host power-state awareness
could be relevant in such a model. For example, if the kernel
can detect battery-backed operation or confirmed UPS
infrastructure, it could advertise a bounded persistence
relaxation window (e.g. guaranteed power for N ms), allowing
the device to safely extend volatile staging within that
window. This would be advisory and revocable, not a mandatory
trust model.

Questions for discussion:

- Has asynchronous durability acknowledgment been previously
  explored in NVMe or block-layer discussions?

- Are there fundamental architectural reasons why separating
  execution completion from durability confirmation would not
  be viable?

- Would such semantics belong strictly in NVMe specification
  work, or is there room for experimentation in the Linux NVMe
  driver as a prototype?

- Are there known workloads where this model would clearly fail
  or provide no measurable benefit?

This is not a proposal for immediate implementation, but an
attempt to identify whether the current binary durability model
(completion vs flush) leaves performance or efficiency on the
table due to lack of explicit state sharing between host and
device.

Feedback, criticism, or pointers to prior art are very welcome.

Thanks,
Esteban Cerutti

Re: [RFC] block/nvme: exploring asynchronous durability notification semantics

[Hannes Reinecke]

On 4/2/26 23:22, Esteban Cerutti wrote:
> Hi,
> 
> I would like to explore whether current NVMe completion semantics
> unnecessarily conflate execution completion with durability, and
> whether there is room for a more explicit, asynchronous durability
> notification model between host and device.
> 
> Today, a successful write completion indicates command execution,
> but not necessarily physical persistence to non-volatile media unless
> FUA or Flush is used. This forces the kernel and filesystems to assume
> worst-case durability behavior and rely on synchronous flushes and
> barriers for safety.
> 
Not necessarily. Synchronous flushes and barriers are really only
an issue for ext4 (which happens to rely on that). Other filesystems
like btrfs work happily without that.
It's more a design choice; explicit flushes guarantee global visibility,
something which you need if you have things like superblocks at fixed
positions. But if you haven't (like btrfs) than you are perfectly fine
using per-request flushes (eg to ensure the log has been written), and
then you don't need a global flush.

> The device internally knows when data is staged in volatile buffers
> versus committed to NAND (or equivalent persistent media), but this
> information is not exposed to the host.
> 
... and not available to the host.
I'm not aware of any command (or I/O completion information) which
would tell you if and when an I/O cached internally in the drive is
written to disk.

> This RFC explores a potential extension model with two components:
> 
> 1) Multi-phase completion semantics
> 
>     - Normal completion continues to signal execution.
>     - The device assigns a persistence token ID.
>     - When the data is physically committed to non-volatile media,
>       the device emits an asynchronous durability confirmation
>       referencing that token.
> 
> This would decouple execution throughput from durability
> confirmation and potentially allow filesystems to close journal
> transactions only upon confirmed persistence, without forcing
> synchronous flush fences.
> 
... or redesign your filesystem to _not_ rely on synchronous flushes.

> 2) Advisory write intent classification
> 
>     - Host-provided hints such as EPHEMERAL, STANDARD, or CRITICAL.
>     - CRITICAL writes would request immediate durability.
>     - EPHEMERAL writes could tolerate extended volatile staging.
> 
> Additionally, I am curious whether host power-state awareness
> could be relevant in such a model. For example, if the kernel
> can detect battery-backed operation or confirmed UPS
> infrastructure, it could advertise a bounded persistence
> relaxation window (e.g. guaranteed power for N ms), allowing
> the device to safely extend volatile staging within that
> window. This would be advisory and revocable, not a mandatory
> trust model.
> 
Been there, done that. Really hard to ensure.
Some RAID controllers try to do something here by not
advertising volatile cache if battery backup is not present,
but that's a really lame thing as you need to ensure that
you have enough power to keep the HDDs running after a
power outage until you have written out all data.
It's getting even harder for on-board HBAs where you have
no distinct power rails for the HDDs; you'd need to have
a battery large enough to keep the entire system running,
but that would be massively overprovisioned for 99% of the
time. And hence far too expensive.

Doubt it's worth it.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: [RFC] block/nvme: exploring asynchronous durability notification semantics

[Christoph Hellwig]

On Thu, Apr 02, 2026 at 06:22:36PM -0300, Esteban Cerutti wrote:
> Today, a successful write completion indicates command execution,
> but not necessarily physical persistence to non-volatile media unless
> FUA or Flush is used. This forces the kernel and filesystems to assume
> worst-case durability behavior and rely on synchronous flushes and
> barriers for safety.

Nothing relies on synchronous flushes, and we killed barriers a long
time ago.  FUA does as you say provide persistence notifications and
is heavily used for the (relatively rare) case where it matters.

>    - Normal completion continues to signal execution.
>    - The device assigns a persistence token ID.
>    - When the data is physically committed to non-volatile media,
>      the device emits an asynchronous durability confirmation
>      referencing that token.
> 
> This would decouple execution throughput from durability
> confirmation and potentially allow filesystems to close journal
> transactions only upon confirmed persistence, without forcing
> synchronous flush fences.

This is so complex that it's not going to work in practice.

You've also failed to explain where you think your model is actually
helping to improve clearly identifiable workloads.  Note that all of
this would be limited to consumer hardware anyway, as volatile write
caches aren't really a thing for higher end hardware.

[RFC] block/nvme: exploring asynchronous durability notification semantics

[Esteban Cerutti]

Hi Hannes, Christoph,

thank you both for the detailed replies. They were very helpful and
corrected several misconceptions on my side.

In particular, I now understand that:

- reliance on synchronous flushes is not universal and largely depends
  on filesystem design (e.g. ext4 vs btrfs),
- FUA already provides a per-request durability guarantee when needed,
- and that power-backed guarantees at the system level are much harder
  to make reliable than they may appear.

Based on your feedback, I took some time to read more about how this
problem is handled in practice, especially at the filesystem and
database level (e.g. journaling, WAL, group commit, batching of fsync).

It seems that the cost of durability in fsync-heavy workloads is indeed
well known, but is typically addressed at higher layers rather than in
the block layer or device interface.

So rather than defending my original idea, I would like to refine the
question.

Is there a fundamental reason why these optimizations are handled
exclusively at the filesystem / application level, instead of being
exposed (or assisted) at a lower level such as the block layer or
device interface?

For example, is it due to:

  - lack of reliable visibility into device-internal persistence state,
  - difficulty in defining correct semantics across different hardware,
  - or simply that higher layers already have enough information to
    optimize safely (making lower-level support unnecessary)?

I realize that my earlier proposal likely underestimated the complexity
involved, but I am still trying to understand whether there is a
meaningful limitation at the lower layers, or if this is primarily a
question of where the problem is best solved.

Thanks again for your time and for the insights — this has already been
very useful for understanding how these pieces fit together.

Best regards,
Esteban