Re: fall back from direct to buffered I/O when stable writes are required

[Christoph Hellwig <hch@lst.de>]

On Mon, Nov 03, 2025 at 12:14:06PM +0100, Jan Kara wrote:
> > Yes, it's pretty clear that the result in non-deterministic in what you
> > get.  But that result still does not result in corruption, because
> > there is a clear boundary ( either the sector size, or for NVMe
> > optionally even a larger bodunary) that designates the atomicy boundary.
> 
> Well, is that boundary really guaranteed? I mean if you modify the buffer
> under IO couldn't it happen that the DMA sees part of the sector new and
> part of the sector old? I agree the window is small but I think the real
> guarantee is architecture dependent and likely cacheline granularity or
> something like that.

If you actually modify it: yes.  But I think Keith' argument was just
about regular racing reads vs writes.

> > pretty clearly not an application bug.  It's also pretty clear that
> > at least some applications (qemu and other VMs) have been doings this
> > for 20+ years.
> 
> Well, I'm mostly of the opinion that modifying IO buffers in flight is an
> application bug (as much as most current storage stacks tolerate it) but on
> the other hand returning IO errors later or even corrupting RAID5 on resync
> is, in my opinion, not a sane error handling on the kernel side either so I
> think we need to do better.

Yes.  Also if you look at the man page which is about official as it gets
for the semantics you can't find anything requiring the buffers to be
stable (but all kinds of other odd rants).

> I also think the performance cost of the unconditional bounce buffering is
> so heavy that it's just a polite way of pushing the app to do proper IO
> buffer synchronization itself (assuming it cares about IO performance but
> given it bothered with direct IO it presumably does). 
>
> So the question is how to get out of this mess with the least disruption
> possible which IMO also means providing easy way for well-behaved apps to
> avoid the overhead.

Remember the cases where this matters is checksumming and parity, where
we touch all the cache lines anyway and consume the DRAM bandwidth,
although bounce buffering upgrades this from pure reads to also writes.
So the overhead is heavy, but if we handle it the right way, that is
doing the checksum/parity calculation while the cache line is still hot
it should not be prohibitive.  And getting this right in the direct
I/O code means that the low-level code could stop bounce buffering
for buffered I/O, providing a major speedup there.

I've been thinking a bit more on how to better get the copy close to the
checksumming at least for PI, and to avoid the extra copies for RAID5
buffered I/O. M maybe a better way is to mark a bio as trusted/untrusted
so that the checksumming/raid code can bounce buffer it, and I start to
like that idea.  A complication is that PI could relax that requirement
if we support PI passthrough from userspace (currently only for block
device, but I plan to add file system support), where the device checks
it, but we can't do that for parity RAID.