* [PATCH v5 1/6] NFSD: add tracing for how direct-mode READ and WRITE are serviced
2026-10-19 15:07 [PATCH v5 0/6] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
@ 2026-10-19 15:07 ` Mike Snitzer
2026-10-19 15:14 ` Chuck Lever
` (2 more replies)
2026-10-19 15:07 ` [PATCH v5 2/6] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats Mike Snitzer
` (4 subsequent siblings)
5 siblings, 3 replies; 11+ messages in thread
From: Mike Snitzer @ 2026-10-19 15:07 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, linux-mm,
linux-fsdevel
When io_cache_read or io_cache_write selects a direct I/O mode, a
request may still be serviced with DONTCACHE buffered or normal
buffered I/O, and a WRITE may be split into up to three segments that
each take a different path depending on the request's offset/length
alignment, the payload's memory alignment, the filesystem's
FOP_DONTCACHE support, direct_misaligned_min_bytes and
direct_misaligned_dontcache. Until now the only visibility was
nfsd_{read,write}_direct for O_DIRECT and nfsd_{read,write}_vector for
everything else, so the DONTCACHE and cached buffered cases were
indistinguishable and the reason a WRITE was not issued as direct I/O
was not recorded at all.
Add:
- nfsd_write_dio_split, emitted once per direct-mode WRITE before any
segment is issued. It records the advertised offset and memory
alignments, the payload's offset within its first page, the start,
middle and end segment sizes, a disposition naming the path taken
(direct, no_alignment, too_small, no_middle, mem_misaligned), and
whether buffered segments are issued IOCB_DONTCACHE.
- nfsd_write_dontcache and nfsd_read_dontcache, emitted for WRITE
segments and READs serviced with IOCB_DONTCACHE. nfsd_write_vector
and nfsd_read_vector now fire only for normal buffered I/O.
The disposition enum lives in vfs.h, which trace.h already includes.
Document the new events in nfsd-io-modes.rst, together with iomap's
iomap_dio_invalidate_fail event, which reports an O_DIRECT segment that
the filesystem serviced with buffered I/O instead.
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
.../filesystems/nfs/nfsd-io-modes.rst | 38 ++++++++-
fs/nfsd/trace.h | 77 +++++++++++++++++++
fs/nfsd/vfs.c | 47 ++++++++---
fs/nfsd/vfs.h | 9 +++
4 files changed, 156 insertions(+), 15 deletions(-)
diff --git a/Documentation/filesystems/nfs/nfsd-io-modes.rst b/Documentation/filesystems/nfs/nfsd-io-modes.rst
index 0a5f4ef14a8a..11da2cee76fc 100644
--- a/Documentation/filesystems/nfs/nfsd-io-modes.rst
+++ b/Documentation/filesystems/nfs/nfsd-io-modes.rst
@@ -171,21 +171,53 @@ Misaligned WRITE:
Tracing:
The nfsd_read_direct trace event shows how NFSD expands any
misaligned READ to the next DIO-aligned block (on either end of the
- original READ, as needed).
+ original READ, as needed). A READ that is serviced with buffered IO
+ instead emits nfsd_read_dontcache (DONTCACHE buffered IO) or
+ nfsd_read_vector (normal buffered IO).
This combination of trace events is useful for READs::
echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_vector/enable
+ echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_dontcache/enable
echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_direct/enable
echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_io_done/enable
echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_read/enable
- The nfsd_write_direct trace event shows how NFSD splits a given
- misaligned WRITE into a DIO-aligned middle segment.
+ The nfsd_write_dio_split trace event is emitted once per WRITE in a
+ DIRECT IO mode, before any IO is issued. It records the alignments
+ the filesystem advertised, the payload's offset in its first page,
+ the sizes of the start, middle and end segments, and a
+ disposition::
+
+ direct the aligned middle segment uses O_DIRECT
+ mem_misaligned payload memory is misaligned; not split
+ no_alignment the filesystem advertises no DIO alignment; not split
+ too_small smaller than the larger of the offset and memory
+ alignments; not split
+ no_middle no aligned middle, or a misaligned start or end
+ with a middle smaller than
+ direct_misaligned_min_bytes bytes; not split
+
+ dontcache=1 means any buffered segment is issued DONTCACHE: the start
+ and end segments of a split WRITE, or the whole of a WRITE that is
+ not split. It is 0 when direct_misaligned_dontcache is N or the
+ filesystem lacks FOP_DONTCACHE, and is reported even for a WRITE
+ that has no buffered segment.
+
+ Each segment then emits nfsd_write_direct (O_DIRECT),
+ nfsd_write_dontcache (DONTCACHE buffered IO) or nfsd_write_vector
+ (normal buffered IO).
This combination of trace events is useful for WRITEs::
echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_opened/enable
+ echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_dio_split/enable
echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_direct/enable
+ echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_dontcache/enable
+ echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_vector/enable
echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_io_done/enable
echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_write/enable
+ echo 1 > /sys/kernel/tracing/events/iomap/iomap_dio_invalidate_fail/enable
+
+ iomap_dio_invalidate_fail reports an O_DIRECT middle segment that
+ the filesystem serviced with buffered IO instead.
diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
index e0908cb21c4e..e2615cfdf6b7 100644
--- a/fs/nfsd/trace.h
+++ b/fs/nfsd/trace.h
@@ -501,6 +501,7 @@ DEFINE_EVENT(nfsd_io_class, nfsd_##name, \
DEFINE_NFSD_IO_EVENT(read_start);
DEFINE_NFSD_IO_EVENT(read_splice);
DEFINE_NFSD_IO_EVENT(read_vector);
+DEFINE_NFSD_IO_EVENT(read_dontcache);
DEFINE_NFSD_IO_EVENT(read_direct);
DEFINE_NFSD_IO_EVENT(read_io_done);
DEFINE_NFSD_IO_EVENT(read_done);
@@ -508,11 +509,87 @@ DEFINE_NFSD_IO_EVENT(write_start);
DEFINE_NFSD_IO_EVENT(write_opened);
DEFINE_NFSD_IO_EVENT(write_direct);
DEFINE_NFSD_IO_EVENT(write_vector);
+DEFINE_NFSD_IO_EVENT(write_dontcache);
DEFINE_NFSD_IO_EVENT(write_io_done);
DEFINE_NFSD_IO_EVENT(write_done);
DEFINE_NFSD_IO_EVENT(commit_start);
DEFINE_NFSD_IO_EVENT(commit_done);
+TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_DIRECT);
+TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_MEM_MISALIGNED);
+TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_NO_ALIGNMENT);
+TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_TOO_SMALL);
+TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_NO_MIDDLE);
+
+#define show_nfsd_write_dio_disposition(x) \
+ __print_symbolic(x, \
+ { NFSD_WRITE_DIO_DIRECT, "direct" }, \
+ { NFSD_WRITE_DIO_MEM_MISALIGNED, "mem_misaligned" }, \
+ { NFSD_WRITE_DIO_NO_ALIGNMENT, "no_alignment" }, \
+ { NFSD_WRITE_DIO_TOO_SMALL, "too_small" }, \
+ { NFSD_WRITE_DIO_NO_MIDDLE, "no_middle" })
+
+/*
+ * Emitted once per direct-mode WRITE, before any segment is issued.
+ * prefix, middle and suffix are zero when not computed. mem_offset is
+ * the offset of the payload within its first page.
+ */
+TRACE_EVENT(nfsd_write_dio_split,
+ TP_PROTO(struct svc_rqst *rqstp,
+ struct svc_fh *fhp,
+ u64 offset,
+ u32 len,
+ u32 offset_align,
+ u32 mem_align,
+ u32 mem_offset,
+ u32 prefix,
+ u32 middle,
+ u32 suffix,
+ enum nfsd_write_dio_disposition disposition,
+ bool dontcache),
+ TP_ARGS(rqstp, fhp, offset, len, offset_align, mem_align, mem_offset,
+ prefix, middle, suffix, disposition, dontcache),
+ TP_STRUCT__entry(
+ __field(u32, xid)
+ __field(u32, fh_hash)
+ __field(u64, offset)
+ __field(u32, len)
+ __field(u32, offset_align)
+ __field(u32, mem_align)
+ __field(u32, mem_offset)
+ __field(u32, prefix)
+ __field(u32, middle)
+ __field(u32, suffix)
+ __field(unsigned int, disposition)
+ __field(bool, dontcache)
+ ),
+ TP_fast_assign(
+ __entry->xid = be32_to_cpu(rqstp->rq_xid);
+ __entry->fh_hash = knfsd_fh_hash(&fhp->fh_handle);
+ __entry->offset = offset;
+ __entry->len = len;
+ __entry->offset_align = offset_align;
+ __entry->mem_align = mem_align;
+ __entry->mem_offset = mem_offset;
+ __entry->prefix = prefix;
+ __entry->middle = middle;
+ __entry->suffix = suffix;
+ __entry->disposition = disposition;
+ __entry->dontcache = dontcache;
+ ),
+ TP_printk("xid=0x%08x fh_hash=0x%08x offset=%llu len=%u "
+ "offset_align=%u mem_align=%u mem_offset=%u "
+ "prefix=%u middle=%u suffix=%u disposition=%s "
+ "dontcache=%u",
+ __entry->xid, __entry->fh_hash,
+ __entry->offset, __entry->len,
+ __entry->offset_align, __entry->mem_align,
+ __entry->mem_offset,
+ __entry->prefix, __entry->middle, __entry->suffix,
+ show_nfsd_write_dio_disposition(__entry->disposition),
+ __entry->dontcache)
+);
+
DECLARE_EVENT_CLASS(nfsd_err_class,
TP_PROTO(struct svc_rqst *rqstp,
struct svc_fh *fhp,
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index 32136cd32af9..0e937d116f39 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -1323,7 +1323,10 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
}
rqstp->rq_next_page = page;
- trace_nfsd_read_vector(rqstp, fhp, offset, *count - total);
+ if (kiocb.ki_flags & IOCB_DONTCACHE)
+ trace_nfsd_read_dontcache(rqstp, fhp, offset, *count - total);
+ else
+ trace_nfsd_read_vector(rqstp, fhp, offset, *count - total);
iov_iter_bvec(&iter, ITER_DEST, rqstp->rq_bvec, v, *count - total);
host_err = vfs_iocb_iter_read(file, &kiocb, &iter);
return nfsd_finish_read(rqstp, fhp, file, offset, count, eof, host_err);
@@ -1391,7 +1394,8 @@ nfsd_write_dio_seg_init(struct nfsd_write_dio_seg *segment,
}
static unsigned int
-nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
+nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
+ struct nfsd_file *nf, struct bio_vec *bvec,
unsigned int nvecs, struct kiocb *iocb,
unsigned long total,
struct nfsd_write_dio_seg segments[3])
@@ -1399,7 +1403,9 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
u32 offset_align = nf->nf_dio_offset_align;
loff_t prefix_end, orig_end, middle_end;
u32 mem_align = nf->nf_dio_mem_align;
- size_t prefix, middle, suffix;
+ size_t prefix = 0, middle = 0, suffix = 0;
+ enum nfsd_write_dio_disposition disposition;
+ u32 mem_offset = nvecs ? bvec->bv_offset : 0;
loff_t offset = iocb->ki_pos;
unsigned int dontcache_flags = 0;
unsigned int buffered_flags;
@@ -1415,9 +1421,14 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
* If alignments are not available, the write is too small,
* or no alignment can be found, fall back to buffered I/O.
*/
- if (unlikely(!mem_align || !offset_align) ||
- unlikely(total < max(offset_align, mem_align)))
+ if (unlikely(!mem_align || !offset_align)) {
+ disposition = NFSD_WRITE_DIO_NO_ALIGNMENT;
goto no_dio;
+ }
+ if (unlikely(total < max(offset_align, mem_align))) {
+ disposition = NFSD_WRITE_DIO_TOO_SMALL;
+ goto no_dio;
+ }
orig_end = offset + total;
@@ -1443,8 +1454,10 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
if (!middle ||
((prefix || suffix) &&
- middle < nfsd_direct_misaligned_min_bytes))
+ middle < nfsd_direct_misaligned_min_bytes)) {
+ disposition = NFSD_WRITE_DIO_NO_MIDDLE;
goto no_dio;
+ }
if (prefix) {
nfsd_write_dio_seg_init(&segments[nsegs], bvec,
@@ -1462,28 +1475,35 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
* the first bvec, all subsequent bvecs start at bv_offset zero
* (page-aligned). Therefore, only the first bvec is checked.
*/
- if (iov_iter_bvec_offset(&segments[nsegs].iter) & (mem_align - 1))
+ if (iov_iter_bvec_offset(&segments[nsegs].iter) & (mem_align - 1)) {
+ disposition = NFSD_WRITE_DIO_MEM_MISALIGNED;
goto no_dio;
+ }
/*
* IOCB_DONTCACHE matters only if the file system falls back to
* buffered I/O (-ENOTBLK).
*/
segments[nsegs++].flags |= IOCB_DIRECT | dontcache_flags;
+ disposition = NFSD_WRITE_DIO_DIRECT;
if (suffix) {
nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs, total,
prefix + middle, suffix, iocb);
segments[nsegs++].flags |= buffered_flags;
}
-
- return nsegs;
+ goto out;
no_dio:
/* Issue the whole WRITE as a single buffered segment. */
nfsd_write_dio_seg_init(&segments[0], bvec, nvecs, total, 0,
total, iocb);
segments[0].flags |= buffered_flags;
- return 1;
+ nsegs = 1;
+out:
+ trace_nfsd_write_dio_split(rqstp, fhp, offset, total, offset_align,
+ mem_align, mem_offset, prefix, middle,
+ suffix, disposition, !!buffered_flags);
+ return nsegs;
}
static noinline_for_stack int
@@ -1499,8 +1519,8 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
ssize_t host_err;
size_t expected;
- nsegs = nfsd_write_dio_iters_init(nf, rqstp->rq_bvec, nvecs,
- kiocb, *cnt, segments);
+ nsegs = nfsd_write_dio_iters_init(rqstp, fhp, nf, rqstp->rq_bvec,
+ nvecs, kiocb, *cnt, segments);
/*
* Persist a split synchronous WRITE once, after all of its
@@ -1517,6 +1537,9 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
if (kiocb->ki_flags & IOCB_DIRECT)
trace_nfsd_write_direct(rqstp, fhp, kiocb->ki_pos,
segments[i].iter.count);
+ else if (kiocb->ki_flags & IOCB_DONTCACHE)
+ trace_nfsd_write_dontcache(rqstp, fhp, kiocb->ki_pos,
+ segments[i].iter.count);
else
trace_nfsd_write_vector(rqstp, fhp, kiocb->ki_pos,
segments[i].iter.count);
diff --git a/fs/nfsd/vfs.h b/fs/nfsd/vfs.h
index 1f05c83a3ea6..e2a7c1c44949 100644
--- a/fs/nfsd/vfs.h
+++ b/fs/nfsd/vfs.h
@@ -230,4 +230,13 @@ __be32 nfsd_permission(struct svc_cred *cred, struct svc_export *exp,
void nfsd_filp_close(struct file *fp);
+/* The path nfsd_write_dio_iters_init() took for a direct-mode WRITE. */
+enum nfsd_write_dio_disposition {
+ NFSD_WRITE_DIO_DIRECT,
+ NFSD_WRITE_DIO_MEM_MISALIGNED,
+ NFSD_WRITE_DIO_NO_ALIGNMENT,
+ NFSD_WRITE_DIO_TOO_SMALL,
+ NFSD_WRITE_DIO_NO_MIDDLE,
+};
+
#endif /* LINUX_NFSD_VFS_H */
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH v5 1/6] NFSD: add tracing for how direct-mode READ and WRITE are serviced
2026-10-19 15:07 ` [PATCH v5 1/6] NFSD: add tracing for how direct-mode READ and WRITE are serviced Mike Snitzer
@ 2026-10-19 15:14 ` Chuck Lever
2026-10-19 17:10 ` Jeff Layton
2026-10-19 18:33 ` Chuck Lever
2 siblings, 0 replies; 11+ messages in thread
From: Chuck Lever @ 2026-10-19 15:14 UTC (permalink / raw)
To: Mike Snitzer, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox (Oracle), linux-mm,
linux-fsdevel
On Mon, Oct 19, 2026, at 11:07 AM, Mike Snitzer wrote:
> When io_cache_read or io_cache_write selects a direct I/O mode, a
> request may still be serviced with DONTCACHE buffered or normal
> buffered I/O, and a WRITE may be split into up to three segments that
> each take a different path depending on the request's offset/length
> alignment, the payload's memory alignment, the filesystem's
> FOP_DONTCACHE support, direct_misaligned_min_bytes and
> direct_misaligned_dontcache. Until now the only visibility was
> nfsd_{read,write}_direct for O_DIRECT and nfsd_{read,write}_vector for
> everything else, so the DONTCACHE and cached buffered cases were
> indistinguishable and the reason a WRITE was not issued as direct I/O
> was not recorded at all.
>
> Add:
>
> - nfsd_write_dio_split, emitted once per direct-mode WRITE before any
> segment is issued. It records the advertised offset and memory
> alignments, the payload's offset within its first page, the start,
> middle and end segment sizes, a disposition naming the path taken
> (direct, no_alignment, too_small, no_middle, mem_misaligned), and
> whether buffered segments are issued IOCB_DONTCACHE.
>
> - nfsd_write_dontcache and nfsd_read_dontcache, emitted for WRITE
> segments and READs serviced with IOCB_DONTCACHE. nfsd_write_vector
> and nfsd_read_vector now fire only for normal buffered I/O.
>
> The disposition enum lives in vfs.h, which trace.h already includes.
>
> Document the new events in nfsd-io-modes.rst, together with iomap's
> iomap_dio_invalidate_fail event, which reports an O_DIRECT segment that
> the filesystem serviced with buffered I/O instead.
>
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Thanks for sinking this one to the bottom of the series. That was on
my to-do list for today.
--
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH v5 1/6] NFSD: add tracing for how direct-mode READ and WRITE are serviced
2026-10-19 15:07 ` [PATCH v5 1/6] NFSD: add tracing for how direct-mode READ and WRITE are serviced Mike Snitzer
2026-10-19 15:14 ` Chuck Lever
@ 2026-10-19 17:10 ` Jeff Layton
2026-10-19 18:33 ` Chuck Lever
2 siblings, 0 replies; 11+ messages in thread
From: Jeff Layton @ 2026-10-19 17:10 UTC (permalink / raw)
To: Mike Snitzer, Chuck Lever
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, linux-mm,
linux-fsdevel
On Mon, 2026-10-19 at 11:07 -0400, Mike Snitzer wrote:
> When io_cache_read or io_cache_write selects a direct I/O mode, a
> request may still be serviced with DONTCACHE buffered or normal
> buffered I/O, and a WRITE may be split into up to three segments that
> each take a different path depending on the request's offset/length
> alignment, the payload's memory alignment, the filesystem's
> FOP_DONTCACHE support, direct_misaligned_min_bytes and
> direct_misaligned_dontcache. Until now the only visibility was
> nfsd_{read,write}_direct for O_DIRECT and nfsd_{read,write}_vector for
> everything else, so the DONTCACHE and cached buffered cases were
> indistinguishable and the reason a WRITE was not issued as direct I/O
> was not recorded at all.
>
> Add:
>
> - nfsd_write_dio_split, emitted once per direct-mode WRITE before any
> segment is issued. It records the advertised offset and memory
> alignments, the payload's offset within its first page, the start,
> middle and end segment sizes, a disposition naming the path taken
> (direct, no_alignment, too_small, no_middle, mem_misaligned), and
> whether buffered segments are issued IOCB_DONTCACHE.
>
> - nfsd_write_dontcache and nfsd_read_dontcache, emitted for WRITE
> segments and READs serviced with IOCB_DONTCACHE. nfsd_write_vector
> and nfsd_read_vector now fire only for normal buffered I/O.
>
> The disposition enum lives in vfs.h, which trace.h already includes.
>
> Document the new events in nfsd-io-modes.rst, together with iomap's
> iomap_dio_invalidate_fail event, which reports an O_DIRECT segment that
> the filesystem serviced with buffered I/O instead.
>
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
> .../filesystems/nfs/nfsd-io-modes.rst | 38 ++++++++-
> fs/nfsd/trace.h | 77 +++++++++++++++++++
> fs/nfsd/vfs.c | 47 ++++++++---
> fs/nfsd/vfs.h | 9 +++
> 4 files changed, 156 insertions(+), 15 deletions(-)
>
> diff --git a/Documentation/filesystems/nfs/nfsd-io-modes.rst b/Documentation/filesystems/nfs/nfsd-io-modes.rst
> index 0a5f4ef14a8a..11da2cee76fc 100644
> --- a/Documentation/filesystems/nfs/nfsd-io-modes.rst
> +++ b/Documentation/filesystems/nfs/nfsd-io-modes.rst
> @@ -171,21 +171,53 @@ Misaligned WRITE:
> Tracing:
> The nfsd_read_direct trace event shows how NFSD expands any
> misaligned READ to the next DIO-aligned block (on either end of the
> - original READ, as needed).
> + original READ, as needed). A READ that is serviced with buffered IO
> + instead emits nfsd_read_dontcache (DONTCACHE buffered IO) or
> + nfsd_read_vector (normal buffered IO).
>
> This combination of trace events is useful for READs::
>
> echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_vector/enable
> + echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_dontcache/enable
> echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_direct/enable
> echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_io_done/enable
> echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_read/enable
>
> - The nfsd_write_direct trace event shows how NFSD splits a given
> - misaligned WRITE into a DIO-aligned middle segment.
> + The nfsd_write_dio_split trace event is emitted once per WRITE in a
> + DIRECT IO mode, before any IO is issued. It records the alignments
> + the filesystem advertised, the payload's offset in its first page,
> + the sizes of the start, middle and end segments, and a
> + disposition::
> +
> + direct the aligned middle segment uses O_DIRECT
> + mem_misaligned payload memory is misaligned; not split
> + no_alignment the filesystem advertises no DIO alignment; not split
> + too_small smaller than the larger of the offset and memory
> + alignments; not split
> + no_middle no aligned middle, or a misaligned start or end
> + with a middle smaller than
> + direct_misaligned_min_bytes bytes; not split
> +
> + dontcache=1 means any buffered segment is issued DONTCACHE: the start
> + and end segments of a split WRITE, or the whole of a WRITE that is
> + not split. It is 0 when direct_misaligned_dontcache is N or the
> + filesystem lacks FOP_DONTCACHE, and is reported even for a WRITE
> + that has no buffered segment.
> +
> + Each segment then emits nfsd_write_direct (O_DIRECT),
> + nfsd_write_dontcache (DONTCACHE buffered IO) or nfsd_write_vector
> + (normal buffered IO).
>
> This combination of trace events is useful for WRITEs::
>
> echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_opened/enable
> + echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_dio_split/enable
> echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_direct/enable
> + echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_dontcache/enable
> + echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_vector/enable
> echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_io_done/enable
> echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_write/enable
> + echo 1 > /sys/kernel/tracing/events/iomap/iomap_dio_invalidate_fail/enable
> +
> + iomap_dio_invalidate_fail reports an O_DIRECT middle segment that
> + the filesystem serviced with buffered IO instead.
> diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
> index e0908cb21c4e..e2615cfdf6b7 100644
> --- a/fs/nfsd/trace.h
> +++ b/fs/nfsd/trace.h
> @@ -501,6 +501,7 @@ DEFINE_EVENT(nfsd_io_class, nfsd_##name, \
> DEFINE_NFSD_IO_EVENT(read_start);
> DEFINE_NFSD_IO_EVENT(read_splice);
> DEFINE_NFSD_IO_EVENT(read_vector);
> +DEFINE_NFSD_IO_EVENT(read_dontcache);
> DEFINE_NFSD_IO_EVENT(read_direct);
> DEFINE_NFSD_IO_EVENT(read_io_done);
> DEFINE_NFSD_IO_EVENT(read_done);
> @@ -508,11 +509,87 @@ DEFINE_NFSD_IO_EVENT(write_start);
> DEFINE_NFSD_IO_EVENT(write_opened);
> DEFINE_NFSD_IO_EVENT(write_direct);
> DEFINE_NFSD_IO_EVENT(write_vector);
> +DEFINE_NFSD_IO_EVENT(write_dontcache);
> DEFINE_NFSD_IO_EVENT(write_io_done);
> DEFINE_NFSD_IO_EVENT(write_done);
> DEFINE_NFSD_IO_EVENT(commit_start);
> DEFINE_NFSD_IO_EVENT(commit_done);
>
> +TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_DIRECT);
> +TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_MEM_MISALIGNED);
> +TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_NO_ALIGNMENT);
> +TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_TOO_SMALL);
> +TRACE_DEFINE_ENUM(NFSD_WRITE_DIO_NO_MIDDLE);
> +
> +#define show_nfsd_write_dio_disposition(x) \
> + __print_symbolic(x, \
> + { NFSD_WRITE_DIO_DIRECT, "direct" }, \
> + { NFSD_WRITE_DIO_MEM_MISALIGNED, "mem_misaligned" }, \
> + { NFSD_WRITE_DIO_NO_ALIGNMENT, "no_alignment" }, \
> + { NFSD_WRITE_DIO_TOO_SMALL, "too_small" }, \
> + { NFSD_WRITE_DIO_NO_MIDDLE, "no_middle" })
> +
> +/*
> + * Emitted once per direct-mode WRITE, before any segment is issued.
> + * prefix, middle and suffix are zero when not computed. mem_offset is
> + * the offset of the payload within its first page.
> + */
> +TRACE_EVENT(nfsd_write_dio_split,
> + TP_PROTO(struct svc_rqst *rqstp,
> + struct svc_fh *fhp,
> + u64 offset,
> + u32 len,
> + u32 offset_align,
> + u32 mem_align,
> + u32 mem_offset,
> + u32 prefix,
> + u32 middle,
> + u32 suffix,
> + enum nfsd_write_dio_disposition disposition,
> + bool dontcache),
> + TP_ARGS(rqstp, fhp, offset, len, offset_align, mem_align, mem_offset,
> + prefix, middle, suffix, disposition, dontcache),
> + TP_STRUCT__entry(
> + __field(u32, xid)
> + __field(u32, fh_hash)
> + __field(u64, offset)
> + __field(u32, len)
> + __field(u32, offset_align)
> + __field(u32, mem_align)
> + __field(u32, mem_offset)
> + __field(u32, prefix)
> + __field(u32, middle)
> + __field(u32, suffix)
> + __field(unsigned int, disposition)
> + __field(bool, dontcache)
> + ),
> + TP_fast_assign(
> + __entry->xid = be32_to_cpu(rqstp->rq_xid);
> + __entry->fh_hash = knfsd_fh_hash(&fhp->fh_handle);
> + __entry->offset = offset;
> + __entry->len = len;
> + __entry->offset_align = offset_align;
> + __entry->mem_align = mem_align;
> + __entry->mem_offset = mem_offset;
> + __entry->prefix = prefix;
> + __entry->middle = middle;
> + __entry->suffix = suffix;
> + __entry->disposition = disposition;
> + __entry->dontcache = dontcache;
> + ),
> + TP_printk("xid=0x%08x fh_hash=0x%08x offset=%llu len=%u "
> + "offset_align=%u mem_align=%u mem_offset=%u "
> + "prefix=%u middle=%u suffix=%u disposition=%s "
> + "dontcache=%u",
> + __entry->xid, __entry->fh_hash,
> + __entry->offset, __entry->len,
> + __entry->offset_align, __entry->mem_align,
> + __entry->mem_offset,
> + __entry->prefix, __entry->middle, __entry->suffix,
> + show_nfsd_write_dio_disposition(__entry->disposition),
> + __entry->dontcache)
> +);
> +
> DECLARE_EVENT_CLASS(nfsd_err_class,
> TP_PROTO(struct svc_rqst *rqstp,
> struct svc_fh *fhp,
> diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
> index 32136cd32af9..0e937d116f39 100644
> --- a/fs/nfsd/vfs.c
> +++ b/fs/nfsd/vfs.c
> @@ -1323,7 +1323,10 @@ __be32 nfsd_iter_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
> }
> rqstp->rq_next_page = page;
>
> - trace_nfsd_read_vector(rqstp, fhp, offset, *count - total);
> + if (kiocb.ki_flags & IOCB_DONTCACHE)
> + trace_nfsd_read_dontcache(rqstp, fhp, offset, *count - total);
> + else
> + trace_nfsd_read_vector(rqstp, fhp, offset, *count - total);
It would be nicer if we could embed the above conditional in something
that is only enabled when the tracepoints are enabled. What would you
think about making those conditional tracepoints, a'la
TRACE_EVENT_CONDITION()? Then the ki_flags tests are only done when the
tracepoints are enabled.
Alternately you could turn all of those into a single trace_nfsd_read
tracepoint that also records ki_flags.
> iov_iter_bvec(&iter, ITER_DEST, rqstp->rq_bvec, v, *count - total);
> host_err = vfs_iocb_iter_read(file, &kiocb, &iter);
> return nfsd_finish_read(rqstp, fhp, file, offset, count, eof, host_err);
> @@ -1391,7 +1394,8 @@ nfsd_write_dio_seg_init(struct nfsd_write_dio_seg *segment,
> }
>
> static unsigned int
> -nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
> +nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
> + struct nfsd_file *nf, struct bio_vec *bvec,
> unsigned int nvecs, struct kiocb *iocb,
> unsigned long total,
> struct nfsd_write_dio_seg segments[3])
> @@ -1399,7 +1403,9 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
> u32 offset_align = nf->nf_dio_offset_align;
> loff_t prefix_end, orig_end, middle_end;
> u32 mem_align = nf->nf_dio_mem_align;
> - size_t prefix, middle, suffix;
> + size_t prefix = 0, middle = 0, suffix = 0;
> + enum nfsd_write_dio_disposition disposition;
> + u32 mem_offset = nvecs ? bvec->bv_offset : 0;
> loff_t offset = iocb->ki_pos;
> unsigned int dontcache_flags = 0;
> unsigned int buffered_flags;
> @@ -1415,9 +1421,14 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
> * If alignments are not available, the write is too small,
> * or no alignment can be found, fall back to buffered I/O.
> */
> - if (unlikely(!mem_align || !offset_align) ||
> - unlikely(total < max(offset_align, mem_align)))
> + if (unlikely(!mem_align || !offset_align)) {
> + disposition = NFSD_WRITE_DIO_NO_ALIGNMENT;
> goto no_dio;
> + }
> + if (unlikely(total < max(offset_align, mem_align))) {
> + disposition = NFSD_WRITE_DIO_TOO_SMALL;
> + goto no_dio;
> + }
>
> orig_end = offset + total;
>
> @@ -1443,8 +1454,10 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
>
> if (!middle ||
> ((prefix || suffix) &&
> - middle < nfsd_direct_misaligned_min_bytes))
> + middle < nfsd_direct_misaligned_min_bytes)) {
> + disposition = NFSD_WRITE_DIO_NO_MIDDLE;
> goto no_dio;
> + }
>
> if (prefix) {
> nfsd_write_dio_seg_init(&segments[nsegs], bvec,
> @@ -1462,28 +1475,35 @@ nfsd_write_dio_iters_init(struct nfsd_file *nf, struct bio_vec *bvec,
> * the first bvec, all subsequent bvecs start at bv_offset zero
> * (page-aligned). Therefore, only the first bvec is checked.
> */
> - if (iov_iter_bvec_offset(&segments[nsegs].iter) & (mem_align - 1))
> + if (iov_iter_bvec_offset(&segments[nsegs].iter) & (mem_align - 1)) {
> + disposition = NFSD_WRITE_DIO_MEM_MISALIGNED;
> goto no_dio;
> + }
> /*
> * IOCB_DONTCACHE matters only if the file system falls back to
> * buffered I/O (-ENOTBLK).
> */
> segments[nsegs++].flags |= IOCB_DIRECT | dontcache_flags;
> + disposition = NFSD_WRITE_DIO_DIRECT;
>
> if (suffix) {
> nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs, total,
> prefix + middle, suffix, iocb);
> segments[nsegs++].flags |= buffered_flags;
> }
> -
> - return nsegs;
> + goto out;
>
> no_dio:
> /* Issue the whole WRITE as a single buffered segment. */
> nfsd_write_dio_seg_init(&segments[0], bvec, nvecs, total, 0,
> total, iocb);
> segments[0].flags |= buffered_flags;
> - return 1;
> + nsegs = 1;
> +out:
> + trace_nfsd_write_dio_split(rqstp, fhp, offset, total, offset_align,
> + mem_align, mem_offset, prefix, middle,
> + suffix, disposition, !!buffered_flags);
> + return nsegs;
> }
>
> static noinline_for_stack int
> @@ -1499,8 +1519,8 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
> ssize_t host_err;
> size_t expected;
>
> - nsegs = nfsd_write_dio_iters_init(nf, rqstp->rq_bvec, nvecs,
> - kiocb, *cnt, segments);
> + nsegs = nfsd_write_dio_iters_init(rqstp, fhp, nf, rqstp->rq_bvec,
> + nvecs, kiocb, *cnt, segments);
>
> /*
> * Persist a split synchronous WRITE once, after all of its
> @@ -1517,6 +1537,9 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
> if (kiocb->ki_flags & IOCB_DIRECT)
> trace_nfsd_write_direct(rqstp, fhp, kiocb->ki_pos,
> segments[i].iter.count);
> + else if (kiocb->ki_flags & IOCB_DONTCACHE)
> + trace_nfsd_write_dontcache(rqstp, fhp, kiocb->ki_pos,
> + segments[i].iter.count);
> else
> trace_nfsd_write_vector(rqstp, fhp, kiocb->ki_pos,
> segments[i].iter.count);
> diff --git a/fs/nfsd/vfs.h b/fs/nfsd/vfs.h
> index 1f05c83a3ea6..e2a7c1c44949 100644
> --- a/fs/nfsd/vfs.h
> +++ b/fs/nfsd/vfs.h
> @@ -230,4 +230,13 @@ __be32 nfsd_permission(struct svc_cred *cred, struct svc_export *exp,
>
> void nfsd_filp_close(struct file *fp);
>
> +/* The path nfsd_write_dio_iters_init() took for a direct-mode WRITE. */
> +enum nfsd_write_dio_disposition {
> + NFSD_WRITE_DIO_DIRECT,
> + NFSD_WRITE_DIO_MEM_MISALIGNED,
> + NFSD_WRITE_DIO_NO_ALIGNMENT,
> + NFSD_WRITE_DIO_TOO_SMALL,
> + NFSD_WRITE_DIO_NO_MIDDLE,
> +};
> +
> #endif /* LINUX_NFSD_VFS_H */
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH v5 1/6] NFSD: add tracing for how direct-mode READ and WRITE are serviced
2026-10-19 15:07 ` [PATCH v5 1/6] NFSD: add tracing for how direct-mode READ and WRITE are serviced Mike Snitzer
2026-10-19 15:14 ` Chuck Lever
2026-10-19 17:10 ` Jeff Layton
@ 2026-10-19 18:33 ` Chuck Lever
2 siblings, 0 replies; 11+ messages in thread
From: Chuck Lever @ 2026-10-19 18:33 UTC (permalink / raw)
To: Mike Snitzer
Cc: Jeff Layton, linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo,
Tom Talpey, Christoph Hellwig, Jens Axboe, Matthew Wilcox,
linux-mm, linux-fsdevel
On Mon, Oct 19, 2026, at 11:07 AM, Mike Snitzer wrote:
> Add:
>
> - nfsd_write_dio_split, emitted once per direct-mode WRITE before any
> segment is issued. It records the advertised offset and memory
> alignments, the payload's offset within its first page, the start,
> middle and end segment sizes, a disposition naming the path taken
> (direct, no_alignment, too_small, no_middle, mem_misaligned), and
> whether buffered segments are issued IOCB_DONTCACHE.
Let's define a single event class and then use DEFINE_EVENT per outcome,
rather than one event with a disposition field. You'll notice this
pattern is used elsewhere in trace.h.
So nfsd_write_dio_class with the alignment and segment-size fields, and
nfsd_write_dio_direct, _no_alignment, _too_small, _no_middle,
_mem_misaligned fired at the site that makes each decision. An admin
chasing "why isn't this direct" then enables only the decline events.
Also, let's give the READ side the same treatment so admins can see
the decisions that prevent reads from going direct.
With the per-outcome events in place, the dontcache field on the
per-WRITE event is redundant: nfsd_write_direct/_dontcache/_vector
already say how each segment was issued. As it stands it also
misreports the direct middle, which carries dontcache_flags
regardless of direct_misaligned_dontcache. Consider dropping that
field.
A few less critical items:
- nfsd_write_dontcache and nfsd_write_vector are emitted only from
nfsd_direct_write(). The io_cache_write=dontcache and =buffered
paths in nfsd_vfs_write() emit neither, so the new DONTCACHE event
never fires in DONTCACHE mode and the commit message's "now fires
only for normal buffered I/O" isn't right either. nfsd_iter_read()
covers all three modes; please make the WRITE side match.
- If any enum from vfs.h ends up in a TP_PROTO, note that trace.h's
own #include "vfs.h" is after the per-class events. It builds today
only because xdr4.h pulls vfs.h in first. Move the include up with
the other headers.
- mem_offset records bvec[0].bv_offset, but the mem_misaligned test
is on bv_offset + prefix. With prefix=100 the event shows
mem_offset=0 next to a mem_misaligned verdict. Record the quantity
that was actually tested.
- The doc says iomap_dio_invalidate_fail "reports an O_DIRECT middle
segment that the filesystem serviced with buffered IO instead". It
reports one cause of that, on iomap filesystems only. ext4's
fallbacks are silent. Let's narrow the wording.
- The doc talks about start/middle/end segments while the event
prints prefix/middle/suffix. Let's pick one vocabulary.
Jeff's comment about the ki_flags test in the hot path stands on its
own. I'll leave that between you and Jeff.
--
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v5 2/6] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats
2026-10-19 15:07 [PATCH v5 0/6] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
2026-10-19 15:07 ` [PATCH v5 1/6] NFSD: add tracing for how direct-mode READ and WRITE are serviced Mike Snitzer
@ 2026-10-19 15:07 ` Mike Snitzer
2026-10-19 16:57 ` Jeff Layton
2026-10-19 15:07 ` [PATCH v5 3/6] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it Mike Snitzer
` (3 subsequent siblings)
5 siblings, 1 reply; 11+ messages in thread
From: Mike Snitzer @ 2026-10-19 15:07 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, linux-mm,
linux-fsdevel
The per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the
flusher writes back in response to an IOCB_DONTCACHE kick, but it is
not reported anywhere, so the DONTCACHE dirty backlog can only be
inferred from MemFree/Cached movement.
Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats
and as WbDontcacheDirty in the per-wb stats, next to the
Reclaimable line. Use wb_stat_sum() rather than wb_stat() to fold in
the per-cpu batches: the flusher targets the summed value and small
backlogs would otherwise read as zero.
Fixes: 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: linux-mm@kvack.org
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
mm/backing-dev.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 18e999053bae..b8a2c27e07fa 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -46,6 +46,7 @@ struct wb_stats {
unsigned long nr_dirty_time;
unsigned long nr_writeback;
unsigned long nr_reclaimable;
+ unsigned long nr_dontcache_dirty;
unsigned long nr_dirtied;
unsigned long nr_written;
unsigned long dirty_thresh;
@@ -78,6 +79,12 @@ static void collect_wb_stats(struct wb_stats *stats,
stats->nr_writeback += wb_stat(wb, WB_WRITEBACK);
stats->nr_reclaimable += wb_stat(wb, WB_RECLAIMABLE);
+ /*
+ * Sum the per-cpu batches: the dontcache flusher targets this
+ * counter via wb_stat_sum() and small backlogs would otherwise
+ * read as zero here.
+ */
+ stats->nr_dontcache_dirty += wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
stats->nr_dirtied += wb_stat(wb, WB_DIRTIED);
stats->nr_written += wb_stat(wb, WB_WRITTEN);
stats->wb_thresh += wb_calc_thresh(wb, stats->dirty_thresh);
@@ -125,6 +132,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
seq_printf(m,
"BdiWriteback: %10lu kB\n"
"BdiReclaimable: %10lu kB\n"
+ "BdiDontcacheDirty: %10lu kB\n"
"BdiDirtyThresh: %10lu kB\n"
"DirtyThresh: %10lu kB\n"
"BackgroundThresh: %10lu kB\n"
@@ -139,6 +147,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
"state: %10lx\n",
K(stats.nr_writeback),
K(stats.nr_reclaimable),
+ K(stats.nr_dontcache_dirty),
K(stats.wb_thresh),
K(dirty_thresh),
K(background_thresh),
@@ -163,6 +172,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
"WbCgIno: %10lu\n"
"WbWriteback: %10lu kB\n"
"WbReclaimable: %10lu kB\n"
+ "WbDontcacheDirty: %10lu kB\n"
"WbDirtyThresh: %10lu kB\n"
"WbDirtied: %10lu kB\n"
"WbWritten: %10lu kB\n"
@@ -179,6 +189,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
#endif
K(stats->nr_writeback),
K(stats->nr_reclaimable),
+ K(stats->nr_dontcache_dirty),
K(stats->wb_thresh),
K(stats->nr_dirtied),
K(stats->nr_written),
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH v5 2/6] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats
2026-10-19 15:07 ` [PATCH v5 2/6] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats Mike Snitzer
@ 2026-10-19 16:57 ` Jeff Layton
0 siblings, 0 replies; 11+ messages in thread
From: Jeff Layton @ 2026-10-19 16:57 UTC (permalink / raw)
To: Mike Snitzer, Chuck Lever
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, linux-mm,
linux-fsdevel
On Mon, 2026-10-19 at 11:07 -0400, Mike Snitzer wrote:
> The per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the
> flusher writes back in response to an IOCB_DONTCACHE kick, but it is
> not reported anywhere, so the DONTCACHE dirty backlog can only be
> inferred from MemFree/Cached movement.
>
> Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats
> and as WbDontcacheDirty in the per-wb stats, next to the
> Reclaimable line. Use wb_stat_sum() rather than wb_stat() to fold in
> the per-cpu batches: the flusher targets the summed value and small
> backlogs would otherwise read as zero.
>
> Fixes: 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")
> Cc: Jeff Layton <jlayton@kernel.org>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
> Cc: linux-mm@kvack.org
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
> mm/backing-dev.c | 11 +++++++++++
> 1 file changed, 11 insertions(+)
>
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index 18e999053bae..b8a2c27e07fa 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -46,6 +46,7 @@ struct wb_stats {
> unsigned long nr_dirty_time;
> unsigned long nr_writeback;
> unsigned long nr_reclaimable;
> + unsigned long nr_dontcache_dirty;
> unsigned long nr_dirtied;
> unsigned long nr_written;
> unsigned long dirty_thresh;
> @@ -78,6 +79,12 @@ static void collect_wb_stats(struct wb_stats *stats,
>
> stats->nr_writeback += wb_stat(wb, WB_WRITEBACK);
> stats->nr_reclaimable += wb_stat(wb, WB_RECLAIMABLE);
> + /*
> + * Sum the per-cpu batches: the dontcache flusher targets this
> + * counter via wb_stat_sum() and small backlogs would otherwise
> + * read as zero here.
> + */
> + stats->nr_dontcache_dirty += wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
> stats->nr_dirtied += wb_stat(wb, WB_DIRTIED);
> stats->nr_written += wb_stat(wb, WB_WRITTEN);
> stats->wb_thresh += wb_calc_thresh(wb, stats->dirty_thresh);
> @@ -125,6 +132,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
> seq_printf(m,
> "BdiWriteback: %10lu kB\n"
> "BdiReclaimable: %10lu kB\n"
> + "BdiDontcacheDirty: %10lu kB\n"
> "BdiDirtyThresh: %10lu kB\n"
> "DirtyThresh: %10lu kB\n"
> "BackgroundThresh: %10lu kB\n"
> @@ -139,6 +147,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
> "state: %10lx\n",
> K(stats.nr_writeback),
> K(stats.nr_reclaimable),
> + K(stats.nr_dontcache_dirty),
> K(stats.wb_thresh),
> K(dirty_thresh),
> K(background_thresh),
> @@ -163,6 +172,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
> "WbCgIno: %10lu\n"
> "WbWriteback: %10lu kB\n"
> "WbReclaimable: %10lu kB\n"
> + "WbDontcacheDirty: %10lu kB\n"
> "WbDirtyThresh: %10lu kB\n"
> "WbDirtied: %10lu kB\n"
> "WbWritten: %10lu kB\n"
> @@ -179,6 +189,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
> #endif
> K(stats->nr_writeback),
> K(stats->nr_reclaimable),
> + K(stats->nr_dontcache_dirty),
> K(stats->wb_thresh),
> K(stats->nr_dirtied),
> K(stats->nr_written),
Since this is for debugfs, I guess we don't have to worry about ABI.
Reviewed-by: Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v5 3/6] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it
2026-10-19 15:07 [PATCH v5 0/6] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
2026-10-19 15:07 ` [PATCH v5 1/6] NFSD: add tracing for how direct-mode READ and WRITE are serviced Mike Snitzer
2026-10-19 15:07 ` [PATCH v5 2/6] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats Mike Snitzer
@ 2026-10-19 15:07 ` Mike Snitzer
2026-10-19 15:07 ` [PATCH v5 4/6] mm, iomap: keep a partly covered page until an uncached write completes it Mike Snitzer
` (2 subsequent siblings)
5 siblings, 0 replies; 11+ messages in thread
From: Mike Snitzer @ 2026-10-19 15:07 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, linux-mm,
linux-fsdevel
A folio allocated by an RWF_DONTCACHE write is marked dropbehind before
it is dirtied, so folio_account_dirtied() adds it to WB_DONTCACHE_DIRTY
and folio_clear_dirty_for_io() or folio_account_cleaned() subtracts it
when it is cleaned. Nothing marks a folio that already exists. A
caller that sets PG_dropbehind on a folio that is already dirty
unbalances the counter: the clean subtracts what was never added, and
once the counter goes negative, wb_stat_sum() reads it as zero and the
DONTCACHE writeback kick stops writing anything back for every writer
on that bdi_writeback.
Add folio_mark_dropbehind(), the inverse of the clear in
__filemap_get_folio_mpol(): with the folio locked, mark it and, if it is
dirty, add it to WB_DONTCACHE_DIRTY. The folio lock orders this
against dirtying and against folio_clear_dirty_for_io(). A concurrent
lookup without FGP_DONTCACHE that clears the flag again subtracts what
this added.
The name follows folio_set_dirty() and folio_mark_dirty(): the first
only sets PG_dirty, the second also accounts the dirty folio.
folio_set_dropbehind() likewise only sets PG_dropbehind, and
folio_mark_dropbehind() also accounts it in WB_DONTCACHE_DIRTY.
A later commit uses it in NFSD to drop a page shared by two misaligned
WRITEs once both have written it.
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: linux-mm@kvack.org
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
include/linux/pagemap.h | 1 +
mm/filemap.c | 31 +++++++++++++++++++++++++++++++
2 files changed, 32 insertions(+)
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0adfa6605653..3012bd95812a 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1310,6 +1310,7 @@ void end_page_writeback(struct page *page);
void folio_end_writeback(struct folio *folio);
void folio_end_writeback_no_dropbehind(struct folio *folio);
void folio_end_dropbehind(struct folio *folio);
+void folio_mark_dropbehind(struct folio *folio);
void folio_wait_stable(struct folio *folio);
void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
diff --git a/mm/filemap.c b/mm/filemap.c
index 00fd89cf6f55..ed7e4dbea653 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1644,6 +1644,37 @@ void folio_end_dropbehind(struct folio *folio)
}
EXPORT_SYMBOL_GPL(folio_end_dropbehind);
+/**
+ * folio_mark_dropbehind - mark an existing folio dropbehind
+ * @folio: the folio, locked by the caller
+ *
+ * Mark @folio so that the writeback that cleans it drops it. A folio that
+ * is already dirty is added to WB_DONTCACHE_DIRTY, as folio_account_dirtied()
+ * would have added it had it been marked when dirtied, so that the clean
+ * that subtracts it leaves the counter balanced.
+ */
+void folio_mark_dropbehind(struct folio *folio)
+{
+ struct address_space *mapping = folio->mapping;
+
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+
+ if (folio_test_dropbehind(folio))
+ return;
+ folio_set_dropbehind(folio);
+ if (mapping && folio_test_dirty(folio) &&
+ mapping_can_writeback(mapping)) {
+ struct inode *inode = mapping->host;
+ struct bdi_writeback *wb;
+ struct wb_lock_cookie cookie = {};
+
+ wb = unlocked_inode_to_wb_begin(inode, &cookie);
+ wb_stat_mod(wb, WB_DONTCACHE_DIRTY, folio_nr_pages(folio));
+ unlocked_inode_to_wb_end(inode, &cookie);
+ }
+}
+EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
+
/**
* folio_end_writeback_no_dropbehind - End writeback against a folio.
* @folio: The folio.
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v5 4/6] mm, iomap: keep a partly covered page until an uncached write completes it
2026-10-19 15:07 [PATCH v5 0/6] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
` (2 preceding siblings ...)
2026-10-19 15:07 ` [PATCH v5 3/6] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it Mike Snitzer
@ 2026-10-19 15:07 ` Mike Snitzer
2026-10-19 15:07 ` [PATCH v5 5/6] mm, iomap: let an uncached writer complete its shared pages itself Mike Snitzer
2026-10-19 15:07 ` [PATCH v5 6/6] NFSD: drop a shared boundary page only once both WRITEs have written it Mike Snitzer
5 siblings, 0 replies; 11+ messages in thread
From: Mike Snitzer @ 2026-10-19 15:07 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, linux-mm,
linux-fsdevel
An uncached (RWF_DONTCACHE) buffered write marks drop-behind every folio
it creates, so writeback drops each one as soon as this write's data is
on disk. A page the write covers only partly is usually shared with the
neighbouring write, the one that ends or starts in it. Once the first
of the two has been written back, the page is gone, and the second has
to read the block back from disk before it can write its part: one read
per misaligned write.
Measured locally on XFS (emulated 4Kn NVMe, 4 KiB pages), 32 threads
writing interleaved records to one file with pwritev2(), 1 GiB per run,
before and after this patch:
record flags 4 KiB device MiB/s
bytes reads per write
before after before after
47008 RWF_DONTCACHE 0.97 0.000 831 2281
47008 RWF_DONTCACHE|RWF_DSYNC 0.99 0.000 601 1227
6000 RWF_DONTCACHE 0.97 0.000 121 488
6000 RWF_DONTCACHE|RWF_DSYNC 0.99 0.000 90 206
Before this patch every write waited for its read, and because XFS holds
its IOLOCK exclusively across a buffered write, the other 31 writers
waited too: both record sizes ran at about 20 thousand writes a second,
one device read per write; after it nothing is read back. With
RWF_DSYNC each write still waits for its own sync, so the gain is
smaller: the uncached write now runs as fast as the same write without
RWF_DONTCACHE, 1237 and 209 MiB/s against 1227 and 206 in the table.
Records of 49152 bytes, which cover only whole pages, read nothing
before or after.
NFSD DIRECT issues the misaligned ends of a WRITE as buffered uncached
segments and gains the same way. Without any handling of the shared
page, 32 interleaved writers of 47008-byte records read 45144 blocks
for 45664 WRITEs; with this patch, and no NFSD code for the shared page,
0.0001 to 0.0008 blocks per WRITE whether each record is sent as one
FILE_SYNC WRITE, as two one after the other, as two together, or as two
UNSTABLE. The page cache then marks the completed page while it is
still dirty, so each segment of a stable WRITE wakes the DONTCACHE
writeback kick (23 to 65 thousand rounds a run); the next two patches
let NFSD mark it after its sync instead.
Let the write path say which pages a write covers only partly
(FGP_DONTCACHE_EDGE, set by iomap when the write starts inside a page or
ends before its end) and get such a page as a single page. A folio
created for it is not marked, so writing back this write's part does not
drop it. A folio found uptodate is the neighbour's, already written:
this write completes it, so mark it, under the folio lock and before its
own data goes in, and the writeback that cleans both writes' data drops
it. The folio lock orders the two writes, so the second always finds
the first's data in place.
A page whose neighbour never writes, such as the last page of a file,
stays cached until reclaim. A partly covered page another task had
cached and that is uptodate is now dropped after the write's writeback,
where before an uncached write never dropped a page it did not create.
Only iomap file systems say which pages a write covers partly so far;
write_begin_get_folio() users such as ext4 and the NFS client do not.
With blocks smaller than a page, a write that crosses an extent boundary
inside a page asks for it as two partly covered halves, so the page,
though one write covers it whole, stays cached until reclaim.
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
fs/iomap/buffered-io.c | 12 +++++++++++-
include/linux/pagemap.h | 5 +++++
mm/filemap.c | 19 ++++++++++++++++++-
3 files changed, 34 insertions(+), 2 deletions(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 0a5ebfda90f1..780a0186045d 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -802,8 +802,18 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
if (iter->flags & IOMAP_NOWAIT)
fgp |= FGP_NOWAIT;
- if (iter->flags & IOMAP_DONTCACHE)
+ if (iter->flags & IOMAP_DONTCACHE) {
fgp |= FGP_DONTCACHE;
+ /*
+ * A page this write covers only partly is shared with a
+ * neighbouring write: get it as a single page, which is
+ * kept until the write that completes it.
+ */
+ if (offset_in_page(pos) || len < PAGE_SIZE) {
+ fgp |= FGP_DONTCACHE_EDGE;
+ len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
+ }
+ }
fgp |= fgf_set_order(len);
return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 3012bd95812a..0c1d6bd09aa0 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -701,6 +701,10 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
* * %FGP_NOWAIT - Don't block on the folio lock.
* * %FGP_STABLE - Wait for the folio to be stable (finished writeback)
* * %FGP_DONTCACHE - Uncached buffered IO
+ * * %FGP_DONTCACHE_EDGE - With %FGP_DONTCACHE: the write covers this page
+ * only partly, and a neighbouring write may complete it. A folio created
+ * for it is not marked drop-behind; a folio found uptodate is, as the
+ * write that completes it.
* * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
* implementation.
*/
@@ -715,6 +719,7 @@ typedef unsigned int __bitwise fgf_t;
#define FGP_FOR_MMAP ((__force fgf_t)0x00000040)
#define FGP_STABLE ((__force fgf_t)0x00000080)
#define FGP_DONTCACHE ((__force fgf_t)0x00000100)
+#define FGP_DONTCACHE_EDGE ((__force fgf_t)0x00000200)
#define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */
#define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
diff --git a/mm/filemap.c b/mm/filemap.c
index ed7e4dbea653..0397ec771547 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2016,6 +2016,16 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
if (fgp_flags & FGP_STABLE)
folio_wait_stable(folio);
+
+ /*
+ * An uncached write that covers this folio only partly, finding it
+ * uptodate, is the write that completes it: the neighbouring write
+ * that left it here has written its part. Mark it now, under the
+ * folio lock, so the writeback that cleans both writes' data drops it.
+ */
+ if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
+ (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
+ folio_mark_dropbehind(folio);
no_page:
if (!folio && (fgp_flags & FGP_CREAT)) {
unsigned int min_order = mapping_min_folio_order(mapping);
@@ -2053,7 +2063,14 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
/* Init accessed so avoid atomic mark_page_accessed later */
if (fgp_flags & FGP_ACCESSED)
__folio_set_referenced(folio);
- if (fgp_flags & FGP_DONTCACHE)
+ /*
+ * A folio the write covers only partly is left
+ * unmarked, so that writing back this write's part
+ * does not drop it before a neighbouring write has
+ * written the rest.
+ */
+ if ((fgp_flags & FGP_DONTCACHE) &&
+ !(fgp_flags & FGP_DONTCACHE_EDGE))
__folio_set_dropbehind(folio);
err = filemap_add_folio(mapping, folio, index, gfp);
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v5 5/6] mm, iomap: let an uncached writer complete its shared pages itself
2026-10-19 15:07 [PATCH v5 0/6] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
` (3 preceding siblings ...)
2026-10-19 15:07 ` [PATCH v5 4/6] mm, iomap: keep a partly covered page until an uncached write completes it Mike Snitzer
@ 2026-10-19 15:07 ` Mike Snitzer
2026-10-19 15:07 ` [PATCH v5 6/6] NFSD: drop a shared boundary page only once both WRITEs have written it Mike Snitzer
5 siblings, 0 replies; 11+ messages in thread
From: Mike Snitzer @ 2026-10-19 15:07 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, linux-mm,
linux-fsdevel
The write that completes a page an uncached write covers only partly
marks it drop-behind as it writes, while the page is dirty, so the page
counts as DONTCACHE dirty and the DONTCACHE writeback kick writes it
back. A writer that syncs its write itself, once, after several
segments -- NFSD answering a stable WRITE, the NFS client's LOCALIO path
-- issues each segment without IOCB_DSYNC, so every segment's kick
writes back pages its own sync is about to write. Measured with NFSD
DIRECT and 32 writers of 47008-byte records: with the previous patch
alone, 23,000 to 65,000 kick rounds a run, where NFSD marking the pages
itself after its sync had taken 51 to 274; with this patch and NFSD
using it, a FILE_SYNC WRITE wakes the kick not at all.
Let such a writer complete those pages itself. IOCB_DONTCACHE_DEFER
(IOMAP_DONTCACHE_DEFER, FGP_DONTCACHE_DEFER) leaves a partly covered page
unmarked even when the write finds it uptodate. Right before writing such
a page the writer asks filemap_dontcache_edge_completes() whether it is
uptodate already, which means the neighbouring write has written its part
and this write completes it; once the write and its sync are done,
filemap_dontcache_edge_complete() marks it and, clean, drops it at once.
Two writes that arrive together may both find the page absent before
either has written; that page is then left cached until reclaim, which
the folio-lock ordering of the default path avoids.
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
fs/iomap/buffered-io.c | 4 +++
fs/iomap/trace.h | 3 ++-
include/linux/fs.h | 8 +++++-
include/linux/iomap.h | 1 +
include/linux/pagemap.h | 6 +++++
mm/filemap.c | 58 ++++++++++++++++++++++++++++++++++++++++-
6 files changed, 77 insertions(+), 3 deletions(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 780a0186045d..5ca82000e2ec 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -811,6 +811,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
*/
if (offset_in_page(pos) || len < PAGE_SIZE) {
fgp |= FGP_DONTCACHE_EDGE;
+ if (iter->flags & IOMAP_DONTCACHE_DEFER)
+ fgp |= FGP_DONTCACHE_DEFER;
len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
}
}
@@ -1318,6 +1320,8 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
iter.flags |= IOMAP_NOWAIT;
if (iocb->ki_flags & IOCB_DONTCACHE)
iter.flags |= IOMAP_DONTCACHE;
+ if (iocb->ki_flags & IOCB_DONTCACHE_DEFER)
+ iter.flags |= IOMAP_DONTCACHE_DEFER;
while ((ret = iomap_iter(&iter, ops)) > 0)
iter.status = iomap_write_iter(&iter, i, write_ops);
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e1ea8392cf47..4f0343eeed1b 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -104,7 +104,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
{ IOMAP_UNSHARE, "UNSHARE" }, \
{ IOMAP_DAX, "DAX" }, \
{ IOMAP_ATOMIC, "ATOMIC" }, \
- { IOMAP_DONTCACHE, "DONTCACHE" }
+ { IOMAP_DONTCACHE, "DONTCACHE" }, \
+ { IOMAP_DONTCACHE_DEFER, "DONTCACHE_DEFER" }
#define IOMAP_F_FLAGS_STRINGS \
{ IOMAP_F_NEW, "NEW" }, \
diff --git a/include/linux/fs.h b/include/linux/fs.h
index f9d1e05e8ae6..ad181b6ce6b4 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -357,6 +357,11 @@ struct readahead_control;
/* kiocb is a read or write operation submitted by fs/aio.c. */
#define IOCB_AIO_RW (1 << 22)
#define IOCB_HAS_METADATA (1 << 23)
+/*
+ * With IOCB_DONTCACHE: the caller completes the pages the write covers only
+ * partly itself, see filemap_dontcache_edge_complete().
+ */
+#define IOCB_DONTCACHE_DEFER (1 << 24)
/* for use in trace events */
#define TRACE_IOCB_STRINGS \
@@ -374,7 +379,8 @@ struct readahead_control;
{ IOCB_NOIO, "NOIO" }, \
{ IOCB_ALLOC_CACHE, "ALLOC_CACHE" }, \
{ IOCB_AIO_RW, "AIO_RW" }, \
- { IOCB_HAS_METADATA, "AIO_HAS_METADATA" }
+ { IOCB_HAS_METADATA, "AIO_HAS_METADATA" }, \
+ { IOCB_DONTCACHE_DEFER, "DONTCACHE_DEFER" }
struct kiocb {
struct file *ki_filp;
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index bc7ae6327dbf..f3509d495a92 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -212,6 +212,7 @@ struct iomap_write_ops {
#endif /* CONFIG_FS_DAX */
#define IOMAP_ATOMIC (1 << 9) /* torn-write protection */
#define IOMAP_DONTCACHE (1 << 10)
+#define IOMAP_DONTCACHE_DEFER (1 << 11) /* see IOCB_DONTCACHE_DEFER */
/*
* Return the existing mapping at pos, or reserve space starting at pos for up
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0c1d6bd09aa0..a0a67e4c940b 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -705,6 +705,9 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
* only partly, and a neighbouring write may complete it. A folio created
* for it is not marked drop-behind; a folio found uptodate is, as the
* write that completes it.
+ * * %FGP_DONTCACHE_DEFER - With %FGP_DONTCACHE_EDGE: the caller completes the
+ * page itself (filemap_dontcache_edge_complete()), so a folio found uptodate
+ * is not marked either.
* * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
* implementation.
*/
@@ -720,6 +723,7 @@ typedef unsigned int __bitwise fgf_t;
#define FGP_STABLE ((__force fgf_t)0x00000080)
#define FGP_DONTCACHE ((__force fgf_t)0x00000100)
#define FGP_DONTCACHE_EDGE ((__force fgf_t)0x00000200)
+#define FGP_DONTCACHE_DEFER ((__force fgf_t)0x00000400)
#define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */
#define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
@@ -1316,6 +1320,8 @@ void folio_end_writeback(struct folio *folio);
void folio_end_writeback_no_dropbehind(struct folio *folio);
void folio_end_dropbehind(struct folio *folio);
void folio_mark_dropbehind(struct folio *folio);
+bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos);
+void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos);
void folio_wait_stable(struct folio *folio);
void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
diff --git a/mm/filemap.c b/mm/filemap.c
index 0397ec771547..7d7c56a42643 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1675,6 +1675,61 @@ void folio_mark_dropbehind(struct folio *folio)
}
EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
+/**
+ * filemap_dontcache_edge_completes - will this write complete a shared page?
+ * @mapping: the file's mapping
+ * @pos: any byte offset within the page
+ *
+ * For an uncached writer that completes the pages it covers only partly
+ * itself (IOCB_DONTCACHE_DEFER). Call right before writing such a page: it
+ * is in the page cache and uptodate when the neighbouring write has already
+ * written its part, and this write then completes it. Two writes that
+ * arrive together may both find it absent; the page is then left cached,
+ * unmarked, until reclaim.
+ *
+ * Return: true if the caller is to call filemap_dontcache_edge_complete()
+ * once its write, including any sync of it, is done.
+ */
+bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos)
+{
+ struct folio *folio;
+ bool ret;
+
+ /* FGP_DONTCACHE: a plain lookup would clear a drop-behind mark. */
+ folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT, FGP_DONTCACHE, 0);
+ if (IS_ERR(folio))
+ return false;
+ ret = folio_test_uptodate(folio);
+ folio_put(folio);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(filemap_dontcache_edge_completes);
+
+/**
+ * filemap_dontcache_edge_complete - drop a shared page both writes have written
+ * @mapping: the file's mapping
+ * @pos: any byte offset within the page
+ *
+ * Mark the page drop-behind. If it is already written back, drop it now;
+ * otherwise the writeback that cleans it drops it. A dirty page marked here
+ * counts as DONTCACHE dirty and wakes the DONTCACHE writeback kick, so a
+ * caller that syncs its write calls this after the sync.
+ */
+void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos)
+{
+ struct folio *folio;
+
+ folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT,
+ FGP_LOCK | FGP_DONTCACHE, 0);
+ if (IS_ERR(folio))
+ return;
+ folio_mark_dropbehind(folio);
+ folio_unlock(folio);
+ folio_end_dropbehind(folio);
+ folio_put(folio);
+}
+EXPORT_SYMBOL_GPL(filemap_dontcache_edge_complete);
+
/**
* folio_end_writeback_no_dropbehind - End writeback against a folio.
* @folio: The folio.
@@ -2024,7 +2079,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
* folio lock, so the writeback that cleans both writes' data drops it.
*/
if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
- (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
+ !(fgp_flags & FGP_DONTCACHE_DEFER) && (fgp_flags & FGP_LOCK) &&
+ folio_test_uptodate(folio))
folio_mark_dropbehind(folio);
no_page:
if (!folio && (fgp_flags & FGP_CREAT)) {
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v5 6/6] NFSD: drop a shared boundary page only once both WRITEs have written it
2026-10-19 15:07 [PATCH v5 0/6] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
` (4 preceding siblings ...)
2026-10-19 15:07 ` [PATCH v5 5/6] mm, iomap: let an uncached writer complete its shared pages itself Mike Snitzer
@ 2026-10-19 15:07 ` Mike Snitzer
5 siblings, 0 replies; 11+ messages in thread
From: Mike Snitzer @ 2026-10-19 15:07 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, linux-mm,
linux-fsdevel, Jonathan Flynn
A buffered DONTCACHE segment of a direct-mode WRITE -- the misaligned
start or end of a split WRITE, or the whole of one too small to split --
covers its first and last page only partly when it does not start or end
on a page boundary. Such a page is shared with the neighbouring WRITE,
which may arrive before, after or at the same time, from any client.
The page cache keeps a page an uncached write covers only partly for the
write that completes it ("mm, iomap: keep a partly covered page until an
uncached write completes it"), so the second WRITE does not have to read
it back from disk. Left at that, the page cache marks the page drop-
behind while it is still dirty, which counts it as DONTCACHE dirty and
wakes the DONTCACHE writeback kick for every segment of a stable WRITE,
only to write back what NFSD's single fsync is about to write.
Issue the buffered segments IOCB_DONTCACHE_DEFER and mark the pages here
instead ("mm, iomap: let an uncached writer complete its shared pages
itself"). Immediately before writing a segment, NFSD asks
filemap_dontcache_edge_completes() whether each page it covers only
partly has already been written by the neighbouring WRITE; once the
whole WRITE, its fsync included, is complete, it calls
filemap_dontcache_edge_complete() for those pages, which drops a clean
page at once. An UNSTABLE WRITE marks them dirty, and the writeback
that cleans them, by the flusher or for the client's COMMIT, drops them.
A page whose neighbouring WRITE never arrives, such as the last page of
the file, or arrives at the same time, so that neither finds the other's
part, stays cached until it is reclaimed. Nothing changes when the
buffered segments are normal buffered I/O (direct_misaligned_dontcache
is N, or the file system lacks FOP_DONTCACHE); the page cache keeps
partly covered pages only for file systems that use iomap for buffered
writes.
Measured with 32 interleaved writers of 47008-byte records over an
emulated 4Kn NVMe with 4 KiB pages, NFSD DIRECT with
direct_misaligned_dontcache=Y, each run from a freshly made file system:
device reads per WRITE are 0.0001-0.0002 whether each record is one
FILE_SYNC WRITE, two sent one after the other, two sent together, or two
sent UNSTABLE, against 45144 reads for 45664 WRITEs with no shared-page
handling, and the DONTCACHE writeback kick does not run at all for the
FILE_SYNC WRITEs.
On a four-server pNFS flexfiles rig (x86_64, 4 KiB logical blocks), three
clients writing 47008-byte records for 240 s against 640 nfsd threads per
server, keeping the shared page took device reads from 0.014-0.015 per
record to 0.003, page-cache growth over the write phase from about
11 GiB to 2.9 GiB, and write throughput up by 2.4% to 4.9%, with read
throughput unchanged. Keeping every boundary page cached instead, with
direct_misaligned_dontcache=N, grows the page cache by about 630 GiB over
the same runs.
Reported-by: Jonathan Flynn <jonathan.flynn@hammerspace.com>
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
.../filesystems/nfs/nfsd-io-modes.rst | 17 ++++
fs/nfsd/vfs.c | 99 ++++++++++++++++---
2 files changed, 104 insertions(+), 12 deletions(-)
diff --git a/Documentation/filesystems/nfs/nfsd-io-modes.rst b/Documentation/filesystems/nfs/nfsd-io-modes.rst
index 11da2cee76fc..26e6dc18a151 100644
--- a/Documentation/filesystems/nfs/nfsd-io-modes.rst
+++ b/Documentation/filesystems/nfs/nfsd-io-modes.rst
@@ -168,6 +168,23 @@ Misaligned WRITE:
above. Set it to N for a workload that reads back or rewrites what
it just wrote.
+ The page holding a start or end segment is shared with the
+ neighbouring WRITE, which may arrive before, after or at the same
+ time, from any client. So that the second WRITE to such a page does
+ not have to read it back from disk, the page cache keeps a page an
+ uncached write covers only partly, unmarked, for the write that
+ completes it. NFSD asks, immediately before writing a start or end
+ segment, whether the neighbouring WRITE has already written its
+ part; if it has, NFSD marks the page DONTCACHE once its own WRITE,
+ including its fsync, is complete, and the page is dropped. A page
+ whose neighbouring WRITE never arrives, or arrives at the same time,
+ stays cached until it is reclaimed. The first and last pages of a
+ WRITE that is not split are handled the same way.
+
+ None of this applies when the start and end segments are normal
+ buffered IO, or for a filesystem that does not use iomap for
+ buffered writes.
+
Tracing:
The nfsd_read_direct trace event shows how NFSD expands any
misaligned READ to the next DIO-aligned block (on either end of the
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index 0e937d116f39..9b5e158dc7a8 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -1372,6 +1372,7 @@ static int wait_for_concurrent_writes(struct file *file)
struct nfsd_write_dio_seg {
struct iov_iter iter;
int flags;
+ bool claim; /* completes shared pages */
};
static unsigned long
@@ -1391,6 +1392,65 @@ nfsd_write_dio_seg_init(struct nfsd_write_dio_seg *segment,
iov_iter_advance(&segment->iter, start);
iov_iter_truncate(&segment->iter, len);
segment->flags = iocb->ki_flags;
+ segment->claim = false;
+}
+
+/* The shared pages a segment completes, and how much of it was written. */
+struct nfsd_write_dio_hold {
+ loff_t first, last;
+ bool complete_first, complete_last;
+ size_t written;
+};
+
+/*
+ * A page that a buffered DONTCACHE segment covers only partly is shared with
+ * the neighbouring WRITE. The page cache keeps it, unmarked, for the write
+ * that completes it; the segment is issued IOCB_DONTCACHE_DEFER so that this
+ * WRITE, not the page cache, marks it, once the whole WRITE, its fsync
+ * included, is complete. Ask whether the neighbour has already written its
+ * part immediately before writing @seg: asking earlier gives the neighbour
+ * time to write its part and find this WRITE's part missing.
+ */
+static void
+nfsd_write_dio_seg_hold(struct file *file, const struct nfsd_write_dio_seg *seg,
+ loff_t pos, size_t len, struct nfsd_write_dio_hold *hold)
+{
+ struct address_space *mapping = file->f_mapping;
+
+ hold->first = pos;
+ hold->last = pos + len - 1;
+ hold->complete_first = false;
+ hold->complete_last = false;
+ hold->written = 0;
+ if (seg->claim && len) {
+ bool one_page = (hold->first >> PAGE_SHIFT) ==
+ (hold->last >> PAGE_SHIFT);
+ bool head = hold->first & ~PAGE_MASK;
+ bool tail = (hold->last + 1) & ~PAGE_MASK;
+
+ /* Only a page the segment covers partly is shared. */
+ if (head || (one_page && tail))
+ hold->complete_first =
+ filemap_dontcache_edge_completes(mapping, hold->first);
+ if (!one_page && tail)
+ hold->complete_last =
+ filemap_dontcache_edge_completes(mapping, hold->last);
+ }
+}
+
+/* Drop the shared pages this WRITE was the second of two to write. */
+static void
+nfsd_write_dio_seg_release(struct file *file,
+ const struct nfsd_write_dio_hold *hold)
+{
+ struct address_space *mapping = file->f_mapping;
+
+ if (hold->complete_first && hold->written)
+ filemap_dontcache_edge_complete(mapping, hold->first);
+ /* A short write did not reach the last page. */
+ if (hold->complete_last &&
+ hold->written == hold->last - hold->first + 1)
+ filemap_dontcache_edge_complete(mapping, hold->last);
}
static unsigned int
@@ -1414,7 +1474,9 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
if (nf->nf_file->f_op->fop_flags & FOP_DONTCACHE)
dontcache_flags = IOCB_DONTCACHE;
- buffered_flags = nfsd_direct_misaligned_dontcache ? dontcache_flags : 0;
+ /* NFSD marks their shared pages itself, see nfsd_write_dio_seg_hold() */
+ buffered_flags = nfsd_direct_misaligned_dontcache && dontcache_flags ?
+ dontcache_flags | IOCB_DONTCACHE_DEFER : 0;
/*
* Check if direct I/O is feasible for this write request.
@@ -1462,7 +1524,8 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
if (prefix) {
nfsd_write_dio_seg_init(&segments[nsegs], bvec,
nvecs, total, 0, prefix, iocb);
- segments[nsegs++].flags |= buffered_flags;
+ segments[nsegs].flags |= buffered_flags;
+ segments[nsegs++].claim = !!buffered_flags;
}
nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs,
@@ -1489,7 +1552,8 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
if (suffix) {
nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs, total,
prefix + middle, suffix, iocb);
- segments[nsegs++].flags |= buffered_flags;
+ segments[nsegs].flags |= buffered_flags;
+ segments[nsegs++].claim = !!buffered_flags;
}
goto out;
@@ -1498,6 +1562,7 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
nfsd_write_dio_seg_init(&segments[0], bvec, nvecs, total, 0,
total, iocb);
segments[0].flags |= buffered_flags;
+ segments[0].claim = !!buffered_flags;
nsegs = 1;
out:
trace_nfsd_write_dio_split(rqstp, fhp, offset, total, offset_align,
@@ -1511,12 +1576,13 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
struct nfsd_file *nf, unsigned int nvecs,
unsigned long *cnt, struct kiocb *kiocb)
{
+ struct nfsd_write_dio_hold holds[3];
struct nfsd_write_dio_seg segments[3];
struct file *file = nf->nf_file;
loff_t start = kiocb->ki_pos;
- unsigned int nsegs, i;
+ unsigned int nsegs, nheld = 0, i;
+ ssize_t host_err = 0;
int sync_flags;
- ssize_t host_err;
size_t expected;
nsegs = nfsd_write_dio_iters_init(rqstp, fhp, nf, rqstp->rq_bvec,
@@ -1545,23 +1611,32 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
segments[i].iter.count);
expected = iov_iter_count(&segments[i].iter);
-
+ nfsd_write_dio_seg_hold(file, &segments[i], kiocb->ki_pos,
+ expected, &holds[nheld]);
host_err = vfs_iocb_iter_write(file, kiocb, &segments[i].iter);
if (host_err < 0)
- return host_err;
+ break;
+ holds[nheld++].written = host_err;
*cnt += host_err;
if (host_err < (ssize_t)expected)
break; /* partial write */
}
- if (sync_flags && *cnt) {
+ if (host_err >= 0 && sync_flags && *cnt)
host_err = vfs_fsync_range(file, start, start + *cnt - 1,
!(sync_flags & IOCB_SYNC));
- if (host_err < 0)
- return host_err;
- }
- return 0;
+ /*
+ * Mark the pages this WRITE shares with its neighbours for dropping
+ * only once the WRITE is complete, including its fsync if it has one.
+ * A page marked while still dirty is counted as DONTCACHE dirty and
+ * wakes the DONTCACHE writeback kick; a page marked once clean is
+ * simply dropped.
+ */
+ for (i = 0; i < nheld; i++)
+ nfsd_write_dio_seg_release(file, &holds[i]);
+
+ return host_err < 0 ? host_err : 0;
}
/**
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread