Re: [PATCH v10 7/7] NFSD: drop a shared boundary page only once both WRITEs have written it

[Mike Snitzer <snitzer@kernel.org>]

On Wed, Nov 04, 2026 at 10:15:43AM +0100, Christoph Hellwig wrote:
> Why does the write include an fsync?  Are these STABLE writes, and
> if yes, why?

Only when the client asks for it.  A WRITE with stable_how FILE_SYNC or
DATA_SYNC must be on stable storage before NFSD replies.  So NFSD runs
one vfs_fsync_range() over the whole WRITE after all of its segments.
An UNSTABLE WRITE has no fsync.  Its shared units are completed while
still dirty, and the writeback that cleans them drops them, whether the
flusher or the client's COMMIT starts it.

The Linux client sends an O_DIRECT write that fits in one WRITE as
FILE_SYNC (FLUSH_COND_STABLE in nfs_pgio_rpcsetup()).  That is the
common case for the workload below.

> Can you explain where this very odd number comes from.  Is that just a
> worst case?  Or for some reason actually relevant?

It is the transfer size of IO500's ior-hard: every rank writes
47008-byte records, interleaved with the other ranks, into one shared
file, with O_DIRECT.  It is meant as a hard case, and it is the
workload that showed the problem on our servers.  The same happens
with any record size that is not a multiple of the page size.  The
cover letter also has 6000-byte records.

> How does this compare to just not using DONTCACHE at all for unaligned
> tails (and maybe heads, if for some reason they I/O you care about
> reordered writes).

Jon measured that on four pNFS flexfiles data servers with ior-hard
from three clients.  In one run, cached heads and tails removed the
reads, 0.000 per record.  But the page cache on the four servers grew
by 622 GiB during a 318 s write phase.  In a later run of the same
workload with this series, it grew by 2.9 GiB, at 0.003 reads per
record.  Keeping the data out of the server's page cache is the point
of direct mode, so cached ends are not a fix for this case.  Using
buffered heads and tails are available (in cel/nfsd-testing): set the
NFSD debugfs knob /sys/kernel/debug/nfsd/direct_misaligned_dontcache
to N.

> And how does it compare to that plus manual
> sync_file_range calls to micro-manage the I/O?

Wasn't measured because sync_file_range() writes the folio back but
leaves it cached.  To get it out of the cache, NFSD would have to drop
it after writeback.  That is only safe once the neighbouring WRITE has
written its part of the folio.  Otherwise the neighbour reads it back
from disk, which is the problem in the first place. NFSD does not know
when that is without tracking the half-written folios of every file
itself.  An earlier version of NFSD kept one tail folio per open file
for the next sequential WRITE.  With interleaved writers the next
WRITE is rarely the neighbour, and it still read 0.81 blocks per
record.  The folio's uptodate state already records whether the
neighbour has written its part, and 5/7 uses that.

> In general I'm not in favour of so many core hacks for
> very nfsd-specific behavior.

Most of the series is not NFSD-specific.  1/7 to 5/7 fix RWF_DONTCACHE
itself, for any application.  The table in the cover letter is plain
pwritev2() on a local XFS file, with no NFSD.  It reads 0.97 blocks per
write at 831 MiB/s before 5/7, and 0.0005 at 2281 MiB/s after it.  2/7
also fixes a bug in v7.3 on its own.

Only 6/7 is for callers that sync once after writing several segments,
which is what NFSD does for a stable WRITE.  The NFS client's LOCALIO
path does the same.  Without 6/7, each stable WRITE wakes the
DONTCACHE writeback kick once per segment, 23,000 to 65,000 times a
run, for folios that its own fsync is about to write.

Thanks,
Mike