[PATCH cel/nfsd-testing replacement] NFSD: cut a misaligned direct-mode WRITE at folio boundaries

[PATCH cel/nfsd-testing replacement] NFSD: cut a misaligned direct-mode WRITE at folio boundaries

[Mike Snitzer]

nfsd_write_dio_iters_init() splits a misaligned direct-mode WRITE into a
direct middle and buffered ends.  Then as little of the WRITE as
possible goes through the page cache.  That works only if direct and
buffered I/O never meet inside a folio.  The direct middle invalidates
the page cache over its range.  So a middle that starts or stops inside
the folio of a buffered end drops that folio.  Then the second of the
two neighbouring WRITEs reads it back from the device.

The middle was cut at the file system's direct I/O offset alignment:

        prefix_end = round_up(offset, offset_align);
        middle_end = round_down(orig_end, offset_align);

Where that alignment is a multiple of the smallest folio the mapping
allows, the cut is a folio boundary, and the problem cannot occur.  This
is the case for 4 KiB pages over 4096-byte blocks.  It is broken where
the smallest folio is larger than the alignment.  Examples are 16 and
64 KiB pages over 4096-byte blocks, and 4 KiB pages over 512-byte
blocks.

Another is a block larger than the page, which the page cache holds as
one folio.  There a misaligned WRITE costs a read per shared folio on
top of its three I/Os.

Cut each end of a WRITE on its own:

- an end that is already an offset_align multiple needs no buffered
  segment, and the middle starts or stops there, inside a folio or not.
  A WRITE aligned to the device stays fully direct at every page size.
- a misaligned end needs a buffered segment in any case.  That segment
  now runs to the nearest boundary of the smallest folio inside the
  WRITE.

That is one rule: a misaligned end is cut at a multiple of the larger of
offset_align and mapping_min_folio_nrbytes().  Both are powers of two.

The segments still tile exactly the bytes of the WRITE: a buffered end
grows inward, at the middle's expense, never outward.  A misaligned
WRITE that holds no whole folio of the smallest size has no middle, and
is issued as one buffered segment.  Where offset_align is a multiple of
the smallest folio, nothing changes.

A folio larger than the smallest that the mapping allows can still span
the cut.  Readahead or an earlier buffered write can leave one, and only
the lookup that finds it learns its size.  The direct middle then
invalidates it as a whole, and the neighbouring WRITE reads its part
back once.  Direct I/O always has that read-back for a cached folio, and
the cut cannot prevent it.

nfsd-io-modes.rst said that DONTCACHE is not used for the misaligned
segments.  The write loop already contradicted that: every segment
without IOCB_DIRECT gets IOCB_DONTCACHE when the file system supports
it.  Replace that sentence with one that says so, next to one that
describes the new cut.

Fixes: 06c5c97293e3 ("NFSD: Implement NFSD_IO_DIRECT for NFS WRITE")
Cc: stable@vger.kernel.org
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
This replaces 80728157d1bd ("NFSD: cut a misaligned direct-mode WRITE at
page boundaries") in cel/nfsd-testing.  Changes from that commit:

- the cut is max_t(unsigned long, offset_align,
  mapping_min_folio_nrbytes(nf->nf_file->f_mapping)).  PAGE_SIZE is
  gone, and cut keeps the helper's unsigned long type.
- the subject, the commit message, the comment and nfsd-io-modes.rst
  name two bounds: the DIO offset alignment and the mapping's smallest
  folio.

It also takes the place of v8 1/7, which will not be in v9.

Heads-up for applying the commits after 80728157d1bd: of the 27 in
cel/nfsd-testing (6d13b7f74eec), only fcd8823db80e ("NFSD: only split a
direct-mode WRITE for a worthwhile direct middle") conflicts, in
nfsd-io-modes.rst.  Take its paragraph with "page" changed to "folio" in
"nearest page boundary" and "never shares a page".  Nothing conflicts in
code.  The other 26 apply cleanly, and the result differs from
cel/nfsd-testing only in this change.

 .../filesystems/nfs/nfsd-io-modes.rst         |  9 ++++++---
 fs/nfsd/vfs.c                                 | 20 +++++++++++++++++--
 2 files changed, 24 insertions(+), 5 deletions(-)

diff --git a/Documentation/filesystems/nfs/nfsd-io-modes.rst b/Documentation/filesystems/nfs/nfsd-io-modes.rst
index 0fd6e82478fe6..8f5e6872f1004 100644
--- a/Documentation/filesystems/nfs/nfsd-io-modes.rst
+++ b/Documentation/filesystems/nfs/nfsd-io-modes.rst
@@ -126,9 +126,12 @@ Misaligned WRITE:
     middle and end as needed. The large middle segment is DIO-aligned
     and the start and/or end are misaligned. Buffered IO is used for the
     misaligned segments and O_DIRECT is used for the middle DIO-aligned
-    segment. DONTCACHE buffered IO is _not_ used for the misaligned
-    segments because using normal buffered IO offers significant RMW
-    performance benefit when handling streaming misaligned WRITEs.
+    segment. A misaligned start or end segment runs to the nearest folio
+    boundary inside the WRITE, not just to the nearest DIO-aligned
+    offset, so the O_DIRECT segment never shares a folio with a buffered
+    one; a start or end that is already DIO-aligned gets no buffered
+    segment at all. If the filesystem supports FOP_DONTCACHE, the
+    misaligned segments are marked DONTCACHE.
 
 Tracing:
     The nfsd_read_direct trace event shows how NFSD expands any
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index c4b32e41b4c5e..f18309d4609b7 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -1396,6 +1396,7 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
 	size_t prefix, middle, suffix;
 	loff_t offset = iocb->ki_pos;
 	unsigned int nsegs = 0;
+	unsigned long cut;
 
 	/*
 	 * Check if direct I/O is feasible for this write request.
@@ -1406,9 +1407,24 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
 	    unlikely(total < max(offset_align, mem_align)))
 		goto no_dio;
 
-	prefix_end = round_up(offset, offset_align);
 	orig_end = offset + total;
-	middle_end = round_down(orig_end, offset_align);
+
+	/*
+	 * The direct middle invalidates the page cache over its range, so it
+	 * must not share a folio with a buffered end.  An aligned end needs no
+	 * buffered segment.  A misaligned one is cut on a boundary of the
+	 * smallest folio the mapping allows, even where the offset alignment
+	 * is finer.  A WRITE may end at OFFSET_MAX, so the prefix is sized as
+	 * a distance rather than by rounding @offset up, which could wrap.
+	 */
+	cut = max_t(unsigned long, offset_align,
+		    mapping_min_folio_nrbytes(nf->nf_file->f_mapping));
+	prefix_end = offset;
+	if (!IS_ALIGNED(offset, offset_align))
+		prefix_end += min_t(loff_t, cut - (offset & (cut - 1)), total);
+	middle_end = orig_end;
+	if (!IS_ALIGNED(orig_end, offset_align))
+		middle_end = max(round_down(orig_end, cut), prefix_end);
 
 	prefix = prefix_end - offset;
 	middle = middle_end - prefix_end;

base-commit: 4e84914d70a0194e18bba6d814cfe71b340aadc8
-- 
2.52.0