Re: [PATCH v4 0/10] NFSD: keep direct-mode I/O out of the page cache

[Mike Snitzer <snitzer@kernel.org>]

On Fri, Oct 16, 2026 at 03:23:27PM -0400, Chuck Lever wrote:
> On Thu, 15 Oct 2026 19:32:59 -0400, Mike Snitzer wrote:
> > 9/10 calls folio_mark_dropbehind() to keep the page two misaligned
> > WRITEs share until both have written it, with no state in NFSD: 704
> > device reads for 42895 WRITEs where there were 45144 for 45664
> > (47008-byte records, 32 interleaved writers, 4 KiB pages), and on a
> > four-server pNFS flexfiles rig 0.003 device reads per record instead of
> > 0.014-0.015, about 2.9 GiB of page-cache growth instead of 11 GiB, and
> > 2.4% to 4.9% more write throughput.
> 
> The direct-mode WRITE path started as one decision: aligned, go
> direct; otherwise split and go buffered for the ends. This series
> adds a page-boundary cut rule that depends on the relation between
> offset_align and PAGE_SIZE, a minimum middle size, a knob for
> whether the ends are DONTCACHE, a floor on READ size, a rule for
> when the sync is issued, and then the shared-page pairing in 9/10.
> Each is small and each has a measurement behind it. Taken together
> they make it hard to say, for a given WRITE, what I/O NFSD will
> issue and what the page cache will look like afterward, and that is
> the property we need to hold onto when a corruption or a performance
> report comes in. I'm not yet convinced that 1/10 through 6/10 buy
> enough to give that up.
> 
> 9/10 gives NFSD a semantic that DONTCACHE does not have: drop this
> page after the second writer, not the first. It builds that out of
> page-cache presence, with an unmarked folio standing in for "a
> partner is coming" and -EEXIST standing in for "the partner has
> been here".
> 
> Neither layer actually knows either of those things with certainty.
> mm cannot know another WRITE is on its way, and NFSD can only guess:
> a page left by a READ, by readahead, or by an earlier claim whose
> partner never arrived makes the first writer look like the second,
> and the description already lists the cases where a claimed page
> stays cached until reclaim because nobody comes to complete it.

I'll be posting a followup 11/10 patch, because your note ^ caused me
to revisit the handling of -EEXIST and I found concurrent receive of
the partner caused the code to be ineffective.  So yes, it wasn't
quite right.  More below.

> The 6000-byte numbers in 9/10 show the shared page is not a splitting
> problem. Those records are never split, and they still cost a read
> per record when the ends are DONTCACHE, because DONTCACHE drops the
> page after the first writer's writeback. So the choice is really
> between two things we already have: issue the ends and unsplit
> WRITEs DONTCACHE and pay a read-modify-write per boundary, or issue
> them as ordinary buffered I/O and let reclaim take the pages.

Agreed on the first part: the shared page is not a splitting problem.
Any buffered DONTCACHE write that covers a page only partly shares that
page with its neighbour, split or not, and 9/10 claims the partly
covered pages of an unsplit WRITE the same way, which is where the
6000-byte numbers come from.  The subject and opening paragraph of 9/10
frame it as a split-WRITE problem, and that is too narrow; v5 will say
what you said.

But I don't think the choice is only between those two.  Each pays for
one of the two things NFSD DIRECT is trying to avoid:

- DONTCACHE without 9/10 pays the read-modify-write: 45144 device reads
  for 45664 WRITEs of 47008-byte records, 196589 reads for the 6000-byte
  records.

- Ordinary buffered I/O (direct_misaligned_dontcache=N) avoids the
  reads, but every page the buffered segments touch stays in the page
  cache until reclaim.  On the pNFS rig in 9/10 that grew the page cache
  by about 630 GiB over the runs, against about 11 GiB with DONTCACHE
  ends.  For records too small to split, such as the 6000-byte ones,
  the whole WRITE is buffered, so every page written stays cached:
  NFSD DIRECT for those workloads becomes plain buffered I/O, which is
  what the DONTCACHE ends are there to avoid.

9/10 keeps each shared page only until both of its writers have written
it, and then drops it.  That gets the footprint of the first option and
the reads of the second together: 704 reads for the 47008-byte records,
385 for the 6000-byte ones, 81 of the file's 524066 pages left resident,
and on the rig 0.003 reads per record (from 0.014-0.015) with 2.9 GiB of
page-cache growth instead of 11 GiB, and write throughput up 2.4-4.9%.

More about patch 11/10 that bears on this: finding the folio already
in the page cache says only that the other WRITE has claimed the page,
not that it has written it.  Two WRITEs that arrive together, such as
the RPCs of one O_DIRECT write larger than the client's wsize, both
claim the page first, and the page could be dropped before the second
half was in it. With the fix, a page whose two writers arrive together
is simply not marked, so for that page alone 9/10 falls back to your
second option and reclaim takes it.  That is the cost I'd rather pay
where NFSD can't tell the order, but only there, not on every
boundary.

> If ordinary buffered ends turn out to cost throughput with flexfiles
> (or any NFSv3 I/O) other than just occupying memory, that is a
> reclaim or DONTCACHE result worth bringing to linux-mm on its own,
> and the right fix would be a DONTCACHE semantic that mm is willing
> to define, not special state in NFSD.
> 
> Consider splitting this into two series (which I already proposed
> for merge process-related reasons). 1-6 need careful scrutiny to
> understand the trade-offs between scalability gains and
> predictability. 7-9 need to be explored as an issue with DONTCACHE
> that should be resolved below NFSD, not within it.
> 
> I'll study 1-6 more.

Patch 10 too please, it improves tracing so we know precisely how each
segment is issued.

Thanks,
Mike