Re: [PATCH v11 2/3] NFSD: Implement NFSD_IO_DIRECT for NFS WRITE

[Mike Snitzer <snitzer@kernel.org>]

On Sat, Nov 08, 2025 at 10:42:27AM +1100, NeilBrown wrote:
> On Sat, 08 Nov 2025, Mike Snitzer wrote:
> > On Sat, Nov 08, 2025 at 08:58:56AM +1100, NeilBrown wrote:
> > > On Sat, 08 Nov 2025, Mike Snitzer wrote:
> > > > On Fri, Nov 07, 2025 at 03:08:11PM -0500, Chuck Lever wrote:
> > > > > On 11/7/25 3:05 PM, Mike Snitzer wrote:
> > > > > >> Agreed. The cover letter noted that this is still controversial.
> > > > > > Only see this, must be missing what you're referring to:
> > > > > > 
> > > > > >   Changes since v9:
> > > > > >   * Unaligned segments no longer use IOCB_DONTCACHE
> > > > > 
> > > > > From the v11 cover letter:
> > > > > 
> > > > > > One controversy remains: Whether to set DONTCACHE for the unaligned
> > > > > > segments.
> > > > 
> > > > Ha, blind as a bat...
> > > > 
> > > > hopefully the rest of my reply helps dispel the controversy
> > > > 
> > > 
> > > Unfortunately I don't think it does.  I don't think it even addresses
> > > it.
> > > 
> > > What Christoph said was:
> > > 
> > >    I'd like to sort out the discussion on why to set IOCB_DONTCACHE when
> > >    nothing is aligned, but not for the non-aligned parts as that is
> > >    extremely counter-intuitive.
> > > 
> > > You gave a lengthy (and probably valid) description on why "not for the
> > > non-aligned parts" but don't seem to address the "why to set
> > > IOCB_DONTCACHE when nothing is aligned" bit.
> > 
> > Because if the entire IO is so poorly formed that it cannot be issued
> > with DIO then at least we can provide the spirit of what was requested
> > from the admin having cared to enable NFSD_IO_DIRECT.
> 
> "so poorly formed"...  How can IO be poorly formed in a way that is
> worse than not being aligned?

I was talking about the no_dio case when iov_iter_bvec_offset() isn't
aligned (case 2 below). The entire WRITE must use buffered IO even
though NFSD_IO_DIRECT configured.

> > Those IOs could be quite large.. full WRITE payload.  I have a
> > reproducer if you'd like it!
> 
> Can they?  How does that work?
> From looking at the code I can only see that "Those IOs" are too small
> to have any whole pages in them.

Huh?  You're clearly confused by the 2 cases of NFSD_IO_DIRECT's
unaligned WRITE handling (as evidenced by you solidly mixing them).

1: WRITE is split, has DIO-aligned middle and prefix and/or suffix.

prefix/suffix are the subpage IOs of a misaligned WRITE that _does_
have a DIO-aligned middle, I covered that in detail in this email:
https://lore.kernel.org/linux-nfs/aQ5Q99Kvw0ZE09Th@kernel.org/
it left you wanting, thinking I missed Christoph's point (I didn't).

2: WRITE cannot be split such that middle is DIO-aligned.

The case you wanted me to elaborate on, I did so here:
https://lore.kernel.org/linux-nfs/aQ5xkjIzf6uU_zLa@kernel.org/
when entire WRITE is misaligned it makes all the sense in the world to
use DONTCACHE.  Chuck, Christoph and I all agree on this.

But just because it makes sense for case 2's (potentially large) WRITE
to not use cached buffered IO if NFSD_IO_DIRECT, it doesn't logically
follow that case 1's two individual pages for prefix/suffix must use
DONTCACHE buffered IO too.

Now do you see the so-called "controversy"?

Christoph expects case 1 to use DONTCACHE because it best reflects
O_DIRECT semantics (avoid using caching).

Here is another way to look at it:

Q: Case 2 uses DONTCACHE, so case 1 should too right?

A: NO. There is legit benefit to having case 1 use cached buffered IO
when issuing its 2 subpage IOs; and that benefit doesn't cause harm
because order-0 page management is not causing the MM problems that
NFSD_IO_DIRECT sets out to avoid (whereas higher order cached buffered
IO is exactly what both DONTCACHE and NFSD_IO_DIRECT aim to avoid.
Otherwise MM spins and spins trying to find adequate free pages,
cannot so does dirty writeback and reclaim which causes kswapd and
kcompactd to burn cpu, etc).

NFSD_IO_DIRECT and DONTCACHE both want to avoid the pathological worst
cases of the MM subsystem when its coping with large IOs.  But
NFSD_IO_DIRECT is doing so in terms of O_DIRECT whereas DONTCACHE is
approximating O_DIRECT (uses page cache with drop-behind semantics,
without concern for DIO alignment, etc).  Both are effective in their
own way, but NFSD_IO_DIRECT only uniformly benefits from using
DONTCACHE buffered IO for case 2.

I'm advocating we make an informed decision for NFSD_IO_DIRECT to use
cached buffered IO for case 1's simple usage of order-0 allocations.
Let's please not get hung up of intent of O_DIRECT because
NFSD_IO_DIRECT achieves that intent very well -- and by using cached
buffered IO it can avoid pathological performance problems (case 1's
handling of streaming misaligned WRITEs that _must_ do RMW).

> Maybe we need a clear description of the sort of IOs that are big enough
> that they impose a performance cost on the page-cache, but that somehow
> cannot be usefully split into a prefix/middle/suffix where prefix and
> suffix are less than a page each.

So defining case 2 again: entire WRITE payload is unaligned because
each aligned logical_block_size doesn't start on a dma_alignment
boundary in memory.  (Also the definition of "the entire IO is so
poorly formed that it cannot be issued with DIO".)

Mike