[PATCH v7 0/5] mm, iomap, NFSD: stop uncached writes reading back shared pages

[PATCH v7 0/5] mm, iomap, NFSD: stop uncached writes reading back shared pages

[Mike Snitzer]

Hi,

Please consider this v7 for the v7.5 development cycle, so that it can
go upstream in the v7.5 merge window.

An uncached (RWF_DONTCACHE) buffered write drops each page it writes
once that page is written back.  A page the write covers only partly is
shared with the neighbouring write, and once the first of the two has
been written back the second has to read it back from disk before it
can write its part.  32 threads writing interleaved records to one file
on XFS (emulated 4Kn NVMe, 4 KiB pages), before and after 3/5:

  record  pwritev2() flags          4 KiB device       MiB/s
  bytes                             reads per write
                                    before   after     before  after
  47008   RWF_DONTCACHE             0.97     0.000     831     2281
  47008   RWF_DONTCACHE|RWF_DSYNC   0.99     0.000     601     1227
  6000    RWF_DONTCACHE             0.97     0.000     121      488
  6000    RWF_DONTCACHE|RWF_DSYNC   0.99     0.000      90      206

3/5 has the page cache keep such a page for the write that completes
it.  4/5 lets a writer that syncs its own write, as NFSD does for a
stable WRITE, complete those pages itself once the sync is done.  5/5
is the NFSD change that uses it.  1/5 reports the DONTCACHE dirty count
in the bdi debugfs stats, and 2/5 adds folio_mark_dropbehind(), which
3/5 and 4/5 use.

The prerequisite NFSD changes are staged in cel/nfsd-testing, through
ad42983b887e ("NFSD: add tracing for how direct-mode READ and WRITE
are serviced"), which was v6's 1/6.  This series applies on top of it,
and each patch builds with W=1 with no warnings.

1/5 to 4/5 are mm, with small iomap changes in 3/5 and 4/5, and are Cc'd
to linux-mm and linux-fsdevel.  If the mm and iomap maintainers are
agreeable, their acks would let Chuck take the series through the nfsd
tree.

Changes since v6:
- v6 1/6 dropped: Chuck applied it to cel/nfsd-testing as ad42983b887e.
- 1/5 to 4/5 (v6 2/6 to 5/6) are unchanged; 5/5 (v6 6/6) only differs
  in context, from its rebase.

v6: https://lore.kernel.org/linux-nfs/atbfEbgMU6UGkn2X@kernel.org/
v5: https://lore.kernel.org/linux-nfs/20261019150725.1507-1-snitzer@kernel.org/
v4: https://lore.kernel.org/linux-nfs/20261015233309.42999-1-snitzer@kernel.org/
v3: https://lore.kernel.org/linux-nfs/20261001045502.48381-1-snitzer@kernel.org/
v2: https://lore.kernel.org/linux-nfs/20260929231329.22018-1-snitzer@kernel.org/
v1: https://lore.kernel.org/linux-nfs/20260929173423.16149-1-snitzer@kernel.org/

Thanks,
Mike

Mike Snitzer (5):
  mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats
  mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it
  mm, iomap: keep a partly covered page until an uncached write completes it
  mm, iomap: let an uncached writer complete its shared pages itself
  NFSD: drop a shared boundary page only once both WRITEs have written it

 .../filesystems/nfs/nfsd-io-modes.rst         |  17 +++
 fs/iomap/buffered-io.c                        |  16 ++-
 fs/iomap/trace.h                              |   3 +-
 fs/nfsd/vfs.c                                 |  99 ++++++++++++++--
 include/linux/fs.h                            |   8 +-
 include/linux/iomap.h                         |   1 +
 include/linux/pagemap.h                       |  12 ++
 mm/backing-dev.c                              |  11 ++
 mm/filemap.c                                  | 106 +++++++++++++++++-
 9 files changed, 257 insertions(+), 16 deletions(-)

-- 
2.52.0

[PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats

[Mike Snitzer]

The per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the
flusher writes back in response to an IOCB_DONTCACHE kick, but it is
not reported anywhere, so the DONTCACHE dirty backlog can only be
inferred from MemFree/Cached movement.

Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats
and as WbDontcacheDirty in the per-wb stats, next to the
Reclaimable line.  Use wb_stat_sum() rather than wb_stat() to fold in
the per-cpu batches: the flusher targets the summed value and small
backlogs would otherwise read as zero.

Fixes: 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: linux-mm@kvack.org
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
 mm/backing-dev.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 18e999053bae..b8a2c27e07fa 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -46,6 +46,7 @@ struct wb_stats {
 	unsigned long nr_dirty_time;
 	unsigned long nr_writeback;
 	unsigned long nr_reclaimable;
+	unsigned long nr_dontcache_dirty;
 	unsigned long nr_dirtied;
 	unsigned long nr_written;
 	unsigned long dirty_thresh;
@@ -78,6 +79,12 @@ static void collect_wb_stats(struct wb_stats *stats,
 
 	stats->nr_writeback += wb_stat(wb, WB_WRITEBACK);
 	stats->nr_reclaimable += wb_stat(wb, WB_RECLAIMABLE);
+	/*
+	 * Sum the per-cpu batches: the dontcache flusher targets this
+	 * counter via wb_stat_sum() and small backlogs would otherwise
+	 * read as zero here.
+	 */
+	stats->nr_dontcache_dirty += wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
 	stats->nr_dirtied += wb_stat(wb, WB_DIRTIED);
 	stats->nr_written += wb_stat(wb, WB_WRITTEN);
 	stats->wb_thresh += wb_calc_thresh(wb, stats->dirty_thresh);
@@ -125,6 +132,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
 	seq_printf(m,
 		   "BdiWriteback:       %10lu kB\n"
 		   "BdiReclaimable:     %10lu kB\n"
+		   "BdiDontcacheDirty:  %10lu kB\n"
 		   "BdiDirtyThresh:     %10lu kB\n"
 		   "DirtyThresh:        %10lu kB\n"
 		   "BackgroundThresh:   %10lu kB\n"
@@ -139,6 +147,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
 		   "state:              %10lx\n",
 		   K(stats.nr_writeback),
 		   K(stats.nr_reclaimable),
+		   K(stats.nr_dontcache_dirty),
 		   K(stats.wb_thresh),
 		   K(dirty_thresh),
 		   K(background_thresh),
@@ -163,6 +172,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
 		   "WbCgIno:           %10lu\n"
 		   "WbWriteback:       %10lu kB\n"
 		   "WbReclaimable:     %10lu kB\n"
+		   "WbDontcacheDirty:  %10lu kB\n"
 		   "WbDirtyThresh:     %10lu kB\n"
 		   "WbDirtied:         %10lu kB\n"
 		   "WbWritten:         %10lu kB\n"
@@ -179,6 +189,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
 #endif
 		   K(stats->nr_writeback),
 		   K(stats->nr_reclaimable),
+		   K(stats->nr_dontcache_dirty),
 		   K(stats->wb_thresh),
 		   K(stats->nr_dirtied),
 		   K(stats->nr_written),
-- 
2.52.0

[PATCH v7 2/5] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it

[Mike Snitzer]

A folio allocated by an RWF_DONTCACHE write is marked dropbehind before
it is dirtied, so folio_account_dirtied() adds it to WB_DONTCACHE_DIRTY
and folio_clear_dirty_for_io() or folio_account_cleaned() subtracts it
when it is cleaned.  Nothing marks a folio that already exists.  A
caller that sets PG_dropbehind on a folio that is already dirty
unbalances the counter: the clean subtracts what was never added, and
once the counter goes negative, wb_stat_sum() reads it as zero and the
DONTCACHE writeback kick stops writing anything back for every writer
on that bdi_writeback.

Add folio_mark_dropbehind(), the inverse of the clear in
__filemap_get_folio_mpol(): with the folio locked, mark it and, if it is
dirty, add it to WB_DONTCACHE_DIRTY.  The folio lock orders this
against dirtying and against folio_clear_dirty_for_io().  A concurrent
lookup without FGP_DONTCACHE that clears the flag again subtracts what
this added.

The name follows folio_set_dirty() and folio_mark_dirty(): the first
only sets PG_dirty, the second also accounts the dirty folio.
folio_set_dropbehind() likewise only sets PG_dropbehind, and
folio_mark_dropbehind() also accounts it in WB_DONTCACHE_DIRTY.

A later commit uses it in NFSD to drop a page shared by two misaligned
WRITEs once both have written it.

Cc: Jeff Layton <jlayton@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: linux-mm@kvack.org
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
 include/linux/pagemap.h |  1 +
 mm/filemap.c            | 31 +++++++++++++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0adfa6605653..3012bd95812a 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1310,6 +1310,7 @@ void end_page_writeback(struct page *page);
 void folio_end_writeback(struct folio *folio);
 void folio_end_writeback_no_dropbehind(struct folio *folio);
 void folio_end_dropbehind(struct folio *folio);
+void folio_mark_dropbehind(struct folio *folio);
 void folio_wait_stable(struct folio *folio);
 void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
 void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
diff --git a/mm/filemap.c b/mm/filemap.c
index 00fd89cf6f55..ed7e4dbea653 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1644,6 +1644,37 @@ void folio_end_dropbehind(struct folio *folio)
 }
 EXPORT_SYMBOL_GPL(folio_end_dropbehind);
 
+/**
+ * folio_mark_dropbehind - mark an existing folio dropbehind
+ * @folio: the folio, locked by the caller
+ *
+ * Mark @folio so that the writeback that cleans it drops it.  A folio that
+ * is already dirty is added to WB_DONTCACHE_DIRTY, as folio_account_dirtied()
+ * would have added it had it been marked when dirtied, so that the clean
+ * that subtracts it leaves the counter balanced.
+ */
+void folio_mark_dropbehind(struct folio *folio)
+{
+	struct address_space *mapping = folio->mapping;
+
+	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+
+	if (folio_test_dropbehind(folio))
+		return;
+	folio_set_dropbehind(folio);
+	if (mapping && folio_test_dirty(folio) &&
+	    mapping_can_writeback(mapping)) {
+		struct inode *inode = mapping->host;
+		struct bdi_writeback *wb;
+		struct wb_lock_cookie cookie = {};
+
+		wb = unlocked_inode_to_wb_begin(inode, &cookie);
+		wb_stat_mod(wb, WB_DONTCACHE_DIRTY, folio_nr_pages(folio));
+		unlocked_inode_to_wb_end(inode, &cookie);
+	}
+}
+EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
+
 /**
  * folio_end_writeback_no_dropbehind - End writeback against a folio.
  * @folio: The folio.
-- 
2.52.0

[PATCH v7 3/5] mm, iomap: keep a partly covered page until an uncached write completes it

[Mike Snitzer]

An uncached (RWF_DONTCACHE) buffered write marks drop-behind every folio
it creates, so writeback drops each one as soon as this write's data is
on disk.  A page the write covers only partly is usually shared with the
neighbouring write, the one that ends or starts in it.  Once the first
of the two has been written back, the page is gone, and the second has
to read the block back from disk before it can write its part: one read
per misaligned write.

Measured locally on XFS (emulated 4Kn NVMe, 4 KiB pages), 32 threads
writing interleaved records to one file with pwritev2(), 1 GiB per run,
before and after this patch:

  record  flags                     4 KiB device       MiB/s
  bytes                             reads per write
                                    before   after     before  after
  47008   RWF_DONTCACHE             0.97     0.000     831     2281
  47008   RWF_DONTCACHE|RWF_DSYNC   0.99     0.000     601     1227
  6000    RWF_DONTCACHE             0.97     0.000     121      488
  6000    RWF_DONTCACHE|RWF_DSYNC   0.99     0.000      90      206

Before this patch every write waited for its read, and because XFS holds
its IOLOCK exclusively across a buffered write, the other 31 writers
waited too: both record sizes ran at about 20 thousand writes a second,
one device read per write; after it nothing is read back.  With
RWF_DSYNC each write still waits for its own sync, so the gain is
smaller: the uncached write now runs as fast as the same write without
RWF_DONTCACHE, 1237 and 209 MiB/s against 1227 and 206 in the table.
Records of 49152 bytes, which cover only whole pages, read nothing
before or after.

NFSD DIRECT issues the misaligned ends of a WRITE as buffered uncached
segments and gains the same way.  Without any handling of the shared
page, 32 interleaved writers of 47008-byte records read 45144 blocks
for 45664 WRITEs; with this patch, and no NFSD code for the shared page,
0.0001 to 0.0008 blocks per WRITE whether each record is sent as one
FILE_SYNC WRITE, as two one after the other, as two together, or as two
UNSTABLE.  The page cache then marks the completed page while it is
still dirty, so each segment of a stable WRITE wakes the DONTCACHE
writeback kick (23 to 65 thousand rounds a run); the next two patches
let NFSD mark it after its sync instead.

Let the write path say which pages a write covers only partly
(FGP_DONTCACHE_EDGE, set by iomap when the write starts inside a page or
ends before its end) and get such a page as a single page.  A folio
created for it is not marked, so writing back this write's part does not
drop it.  A folio found uptodate is the neighbour's, already written:
this write completes it, so mark it, under the folio lock and before its
own data goes in, and the writeback that cleans both writes' data drops
it.  The folio lock orders the two writes, so the second always finds
the first's data in place.

A page whose neighbour never writes, such as the last page of a file,
stays cached until reclaim.  A partly covered page another task had
cached and that is uptodate is now dropped after the write's writeback,
where before an uncached write never dropped a page it did not create.

Only iomap file systems say which pages a write covers partly so far;
write_begin_get_folio() users such as ext4 and the NFS client do not.
With blocks smaller than a page, a write that crosses an extent boundary
inside a page asks for it as two partly covered halves, so the page,
though one write covers it whole, stays cached until reclaim.

Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
 fs/iomap/buffered-io.c  | 12 +++++++++++-
 include/linux/pagemap.h |  5 +++++
 mm/filemap.c            | 19 ++++++++++++++++++-
 3 files changed, 34 insertions(+), 2 deletions(-)

diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 0a5ebfda90f1..780a0186045d 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -802,8 +802,18 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
 
 	if (iter->flags & IOMAP_NOWAIT)
 		fgp |= FGP_NOWAIT;
-	if (iter->flags & IOMAP_DONTCACHE)
+	if (iter->flags & IOMAP_DONTCACHE) {
 		fgp |= FGP_DONTCACHE;
+		/*
+		 * A page this write covers only partly is shared with a
+		 * neighbouring write: get it as a single page, which is
+		 * kept until the write that completes it.
+		 */
+		if (offset_in_page(pos) || len < PAGE_SIZE) {
+			fgp |= FGP_DONTCACHE_EDGE;
+			len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
+		}
+	}
 	fgp |= fgf_set_order(len);
 
 	return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 3012bd95812a..0c1d6bd09aa0 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -701,6 +701,10 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
  * * %FGP_NOWAIT - Don't block on the folio lock.
  * * %FGP_STABLE - Wait for the folio to be stable (finished writeback)
  * * %FGP_DONTCACHE - Uncached buffered IO
+ * * %FGP_DONTCACHE_EDGE - With %FGP_DONTCACHE: the write covers this page
+ *   only partly, and a neighbouring write may complete it.  A folio created
+ *   for it is not marked drop-behind; a folio found uptodate is, as the
+ *   write that completes it.
  * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
  *   implementation.
  */
@@ -715,6 +719,7 @@ typedef unsigned int __bitwise fgf_t;
 #define FGP_FOR_MMAP		((__force fgf_t)0x00000040)
 #define FGP_STABLE		((__force fgf_t)0x00000080)
 #define FGP_DONTCACHE		((__force fgf_t)0x00000100)
+#define FGP_DONTCACHE_EDGE	((__force fgf_t)0x00000200)
 #define FGF_GET_ORDER(fgf)	(((__force unsigned)fgf) >> 26)	/* top 6 bits */
 
 #define FGP_WRITEBEGIN		(FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
diff --git a/mm/filemap.c b/mm/filemap.c
index ed7e4dbea653..0397ec771547 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2016,6 +2016,16 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
 
 	if (fgp_flags & FGP_STABLE)
 		folio_wait_stable(folio);
+
+	/*
+	 * An uncached write that covers this folio only partly, finding it
+	 * uptodate, is the write that completes it: the neighbouring write
+	 * that left it here has written its part.  Mark it now, under the
+	 * folio lock, so the writeback that cleans both writes' data drops it.
+	 */
+	if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
+	    (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
+		folio_mark_dropbehind(folio);
 no_page:
 	if (!folio && (fgp_flags & FGP_CREAT)) {
 		unsigned int min_order = mapping_min_folio_order(mapping);
@@ -2053,7 +2063,14 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
 			/* Init accessed so avoid atomic mark_page_accessed later */
 			if (fgp_flags & FGP_ACCESSED)
 				__folio_set_referenced(folio);
-			if (fgp_flags & FGP_DONTCACHE)
+			/*
+			 * A folio the write covers only partly is left
+			 * unmarked, so that writing back this write's part
+			 * does not drop it before a neighbouring write has
+			 * written the rest.
+			 */
+			if ((fgp_flags & FGP_DONTCACHE) &&
+			    !(fgp_flags & FGP_DONTCACHE_EDGE))
 				__folio_set_dropbehind(folio);
 
 			err = filemap_add_folio(mapping, folio, index, gfp);
-- 
2.52.0

[PATCH v7 4/5] mm, iomap: let an uncached writer complete its shared pages itself

[Mike Snitzer]

The write that completes a page an uncached write covers only partly
marks it drop-behind as it writes, while the page is dirty, so the page
counts as DONTCACHE dirty and the DONTCACHE writeback kick writes it
back.  A writer that syncs its write itself, once, after several
segments -- NFSD answering a stable WRITE, the NFS client's LOCALIO path
-- issues each segment without IOCB_DSYNC, so every segment's kick
writes back pages its own sync is about to write.  Measured with NFSD
DIRECT and 32 writers of 47008-byte records: with the previous patch
alone, 23,000 to 65,000 kick rounds a run, where NFSD marking the pages
itself after its sync had taken 51 to 274; with this patch and NFSD
using it, a FILE_SYNC WRITE wakes the kick not at all.

Let such a writer complete those pages itself.  IOCB_DONTCACHE_DEFER
(IOMAP_DONTCACHE_DEFER, FGP_DONTCACHE_DEFER) leaves a partly covered page
unmarked even when the write finds it uptodate.  Right before writing such
a page the writer asks filemap_dontcache_edge_completes() whether it is
uptodate already, which means the neighbouring write has written its part
and this write completes it; once the write and its sync are done,
filemap_dontcache_edge_complete() marks it and, clean, drops it at once.

Two writes that arrive together may both find the page absent before
either has written; that page is then left cached until reclaim, which
the folio-lock ordering of the default path avoids.

Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
 fs/iomap/buffered-io.c  |  4 +++
 fs/iomap/trace.h        |  3 ++-
 include/linux/fs.h      |  8 +++++-
 include/linux/iomap.h   |  1 +
 include/linux/pagemap.h |  6 +++++
 mm/filemap.c            | 58 ++++++++++++++++++++++++++++++++++++++++-
 6 files changed, 77 insertions(+), 3 deletions(-)

diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 780a0186045d..5ca82000e2ec 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -811,6 +811,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
 		 */
 		if (offset_in_page(pos) || len < PAGE_SIZE) {
 			fgp |= FGP_DONTCACHE_EDGE;
+			if (iter->flags & IOMAP_DONTCACHE_DEFER)
+				fgp |= FGP_DONTCACHE_DEFER;
 			len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
 		}
 	}
@@ -1318,6 +1320,8 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 		iter.flags |= IOMAP_NOWAIT;
 	if (iocb->ki_flags & IOCB_DONTCACHE)
 		iter.flags |= IOMAP_DONTCACHE;
+	if (iocb->ki_flags & IOCB_DONTCACHE_DEFER)
+		iter.flags |= IOMAP_DONTCACHE_DEFER;
 
 	while ((ret = iomap_iter(&iter, ops)) > 0)
 		iter.status = iomap_write_iter(&iter, i, write_ops);
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e1ea8392cf47..4f0343eeed1b 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -104,7 +104,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
 	{ IOMAP_UNSHARE,	"UNSHARE" }, \
 	{ IOMAP_DAX,		"DAX" }, \
 	{ IOMAP_ATOMIC,		"ATOMIC" }, \
-	{ IOMAP_DONTCACHE,	"DONTCACHE" }
+	{ IOMAP_DONTCACHE,	"DONTCACHE" }, \
+	{ IOMAP_DONTCACHE_DEFER, "DONTCACHE_DEFER" }
 
 #define IOMAP_F_FLAGS_STRINGS \
 	{ IOMAP_F_NEW,		"NEW" }, \
diff --git a/include/linux/fs.h b/include/linux/fs.h
index f9d1e05e8ae6..ad181b6ce6b4 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -357,6 +357,11 @@ struct readahead_control;
 /* kiocb is a read or write operation submitted by fs/aio.c. */
 #define IOCB_AIO_RW		(1 << 22)
 #define IOCB_HAS_METADATA	(1 << 23)
+/*
+ * With IOCB_DONTCACHE: the caller completes the pages the write covers only
+ * partly itself, see filemap_dontcache_edge_complete().
+ */
+#define IOCB_DONTCACHE_DEFER	(1 << 24)
 
 /* for use in trace events */
 #define TRACE_IOCB_STRINGS \
@@ -374,7 +379,8 @@ struct readahead_control;
 	{ IOCB_NOIO,		"NOIO" }, \
 	{ IOCB_ALLOC_CACHE,	"ALLOC_CACHE" }, \
 	{ IOCB_AIO_RW,		"AIO_RW" }, \
-	{ IOCB_HAS_METADATA,	"AIO_HAS_METADATA" }
+	{ IOCB_HAS_METADATA,	"AIO_HAS_METADATA" }, \
+	{ IOCB_DONTCACHE_DEFER,	"DONTCACHE_DEFER" }
 
 struct kiocb {
 	struct file		*ki_filp;
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index bc7ae6327dbf..f3509d495a92 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -212,6 +212,7 @@ struct iomap_write_ops {
 #endif /* CONFIG_FS_DAX */
 #define IOMAP_ATOMIC		(1 << 9) /* torn-write protection */
 #define IOMAP_DONTCACHE		(1 << 10)
+#define IOMAP_DONTCACHE_DEFER	(1 << 11) /* see IOCB_DONTCACHE_DEFER */
 
 /*
  * Return the existing mapping at pos, or reserve space starting at pos for up
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0c1d6bd09aa0..a0a67e4c940b 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -705,6 +705,9 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
  *   only partly, and a neighbouring write may complete it.  A folio created
  *   for it is not marked drop-behind; a folio found uptodate is, as the
  *   write that completes it.
+ * * %FGP_DONTCACHE_DEFER - With %FGP_DONTCACHE_EDGE: the caller completes the
+ *   page itself (filemap_dontcache_edge_complete()), so a folio found uptodate
+ *   is not marked either.
  * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
  *   implementation.
  */
@@ -720,6 +723,7 @@ typedef unsigned int __bitwise fgf_t;
 #define FGP_STABLE		((__force fgf_t)0x00000080)
 #define FGP_DONTCACHE		((__force fgf_t)0x00000100)
 #define FGP_DONTCACHE_EDGE	((__force fgf_t)0x00000200)
+#define FGP_DONTCACHE_DEFER	((__force fgf_t)0x00000400)
 #define FGF_GET_ORDER(fgf)	(((__force unsigned)fgf) >> 26)	/* top 6 bits */
 
 #define FGP_WRITEBEGIN		(FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
@@ -1316,6 +1320,8 @@ void folio_end_writeback(struct folio *folio);
 void folio_end_writeback_no_dropbehind(struct folio *folio);
 void folio_end_dropbehind(struct folio *folio);
 void folio_mark_dropbehind(struct folio *folio);
+bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos);
+void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos);
 void folio_wait_stable(struct folio *folio);
 void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
 void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
diff --git a/mm/filemap.c b/mm/filemap.c
index 0397ec771547..7d7c56a42643 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1675,6 +1675,61 @@ void folio_mark_dropbehind(struct folio *folio)
 }
 EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
 
+/**
+ * filemap_dontcache_edge_completes - will this write complete a shared page?
+ * @mapping: the file's mapping
+ * @pos: any byte offset within the page
+ *
+ * For an uncached writer that completes the pages it covers only partly
+ * itself (IOCB_DONTCACHE_DEFER).  Call right before writing such a page: it
+ * is in the page cache and uptodate when the neighbouring write has already
+ * written its part, and this write then completes it.  Two writes that
+ * arrive together may both find it absent; the page is then left cached,
+ * unmarked, until reclaim.
+ *
+ * Return: true if the caller is to call filemap_dontcache_edge_complete()
+ * once its write, including any sync of it, is done.
+ */
+bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos)
+{
+	struct folio *folio;
+	bool ret;
+
+	/* FGP_DONTCACHE: a plain lookup would clear a drop-behind mark. */
+	folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT, FGP_DONTCACHE, 0);
+	if (IS_ERR(folio))
+		return false;
+	ret = folio_test_uptodate(folio);
+	folio_put(folio);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(filemap_dontcache_edge_completes);
+
+/**
+ * filemap_dontcache_edge_complete - drop a shared page both writes have written
+ * @mapping: the file's mapping
+ * @pos: any byte offset within the page
+ *
+ * Mark the page drop-behind.  If it is already written back, drop it now;
+ * otherwise the writeback that cleans it drops it.  A dirty page marked here
+ * counts as DONTCACHE dirty and wakes the DONTCACHE writeback kick, so a
+ * caller that syncs its write calls this after the sync.
+ */
+void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos)
+{
+	struct folio *folio;
+
+	folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT,
+				    FGP_LOCK | FGP_DONTCACHE, 0);
+	if (IS_ERR(folio))
+		return;
+	folio_mark_dropbehind(folio);
+	folio_unlock(folio);
+	folio_end_dropbehind(folio);
+	folio_put(folio);
+}
+EXPORT_SYMBOL_GPL(filemap_dontcache_edge_complete);
+
 /**
  * folio_end_writeback_no_dropbehind - End writeback against a folio.
  * @folio: The folio.
@@ -2024,7 +2079,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
 	 * folio lock, so the writeback that cleans both writes' data drops it.
 	 */
 	if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
-	    (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
+	    !(fgp_flags & FGP_DONTCACHE_DEFER) && (fgp_flags & FGP_LOCK) &&
+	    folio_test_uptodate(folio))
 		folio_mark_dropbehind(folio);
 no_page:
 	if (!folio && (fgp_flags & FGP_CREAT)) {
-- 
2.52.0

[PATCH v7 5/5] NFSD: drop a shared boundary page only once both WRITEs have written it

[Mike Snitzer]

A buffered DONTCACHE segment of a direct-mode WRITE -- the misaligned
start or end of a split WRITE, or the whole of one too small to split --
covers its first and last page only partly when it does not start or end
on a page boundary.  Such a page is shared with the neighbouring WRITE,
which may arrive before, after or at the same time, from any client.

The page cache keeps a page an uncached write covers only partly for the
write that completes it ("mm, iomap: keep a partly covered page until an
uncached write completes it"), so the second WRITE does not have to read
it back from disk.  Left at that, the page cache marks the page drop-
behind while it is still dirty, which counts it as DONTCACHE dirty and
wakes the DONTCACHE writeback kick for every segment of a stable WRITE,
only to write back what NFSD's single fsync is about to write.

Issue the buffered segments IOCB_DONTCACHE_DEFER and mark the pages here
instead ("mm, iomap: let an uncached writer complete its shared pages
itself").  Immediately before writing a segment, NFSD asks
filemap_dontcache_edge_completes() whether each page it covers only
partly has already been written by the neighbouring WRITE; once the
whole WRITE, its fsync included, is complete, it calls
filemap_dontcache_edge_complete() for those pages, which drops a clean
page at once.  An UNSTABLE WRITE marks them dirty, and the writeback
that cleans them, by the flusher or for the client's COMMIT, drops them.

A page whose neighbouring WRITE never arrives, such as the last page of
the file, or arrives at the same time, so that neither finds the other's
part, stays cached until it is reclaimed.  Nothing changes when the
buffered segments are normal buffered I/O (direct_misaligned_dontcache
is N, or the file system lacks FOP_DONTCACHE); the page cache keeps
partly covered pages only for file systems that use iomap for buffered
writes.

Measured with 32 interleaved writers of 47008-byte records over an
emulated 4Kn NVMe with 4 KiB pages, NFSD DIRECT with
direct_misaligned_dontcache=Y, each run from a freshly made file system:
device reads per WRITE are 0.0001-0.0002 whether each record is one
FILE_SYNC WRITE, two sent one after the other, two sent together, or two
sent UNSTABLE, against 45144 reads for 45664 WRITEs with no shared-page
handling, and the DONTCACHE writeback kick does not run at all for the
FILE_SYNC WRITEs.

On a four-server pNFS flexfiles rig (x86_64, 4 KiB logical blocks), three
clients writing 47008-byte records for 240 s against 640 nfsd threads per
server, keeping the shared page took device reads from 0.014-0.015 per
record to 0.003, page-cache growth over the write phase from about
11 GiB to 2.9 GiB, and write throughput up by 2.4% to 4.9%, with read
throughput unchanged.  Keeping every boundary page cached instead, with
direct_misaligned_dontcache=N, grows the page cache by about 630 GiB over
the same runs.

Reported-by: Jonathan Flynn <jonathan.flynn@hammerspace.com>
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
 .../filesystems/nfs/nfsd-io-modes.rst         | 17 ++++
 fs/nfsd/vfs.c                                 | 99 ++++++++++++++++---
 2 files changed, 104 insertions(+), 12 deletions(-)

diff --git a/Documentation/filesystems/nfs/nfsd-io-modes.rst b/Documentation/filesystems/nfs/nfsd-io-modes.rst
index 0e69b5fdd675..9c663ce97c48 100644
--- a/Documentation/filesystems/nfs/nfsd-io-modes.rst
+++ b/Documentation/filesystems/nfs/nfsd-io-modes.rst
@@ -167,6 +167,23 @@ Misaligned WRITE:
     above. Set it to N for a workload that reads back or rewrites what
     it just wrote.
 
+    The page holding a start or end segment is shared with the
+    neighbouring WRITE, which may arrive before, after or at the same
+    time, from any client. So that the second WRITE to such a page does
+    not have to read it back from disk, the page cache keeps a page an
+    uncached write covers only partly, unmarked, for the write that
+    completes it. NFSD asks, immediately before writing a start or end
+    segment, whether the neighbouring WRITE has already written its
+    part; if it has, NFSD marks the page DONTCACHE once its own WRITE,
+    including its fsync, is complete, and the page is dropped. A page
+    whose neighbouring WRITE never arrives, or arrives at the same time,
+    stays cached until it is reclaimed. The first and last pages of a
+    WRITE that is not split are handled the same way.
+
+    None of this applies when the start and end segments are normal
+    buffered IO, or for a filesystem that does not use iomap for
+    buffered writes.
+
 Tracing:
     The nfsd_read_direct trace event marks a READ that NFSD issues as
     O_DIRECT. It records the offset and length the client requested.
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index 9219f8eb386d..3605ddf1ad1e 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -1387,6 +1387,7 @@ static int wait_for_concurrent_writes(struct file *file)
 struct nfsd_write_dio_seg {
 	struct iov_iter			iter;
 	int				flags;
+	bool				claim;		/* completes shared pages */
 };
 
 static unsigned long
@@ -1406,6 +1407,65 @@ nfsd_write_dio_seg_init(struct nfsd_write_dio_seg *segment,
 		iov_iter_advance(&segment->iter, start);
 	iov_iter_truncate(&segment->iter, len);
 	segment->flags = iocb->ki_flags;
+	segment->claim = false;
+}
+
+/* The shared pages a segment completes, and how much of it was written. */
+struct nfsd_write_dio_hold {
+	loff_t	first, last;
+	bool	complete_first, complete_last;
+	size_t	written;
+};
+
+/*
+ * A page that a buffered DONTCACHE segment covers only partly is shared with
+ * the neighbouring WRITE.  The page cache keeps it, unmarked, for the write
+ * that completes it; the segment is issued IOCB_DONTCACHE_DEFER so that this
+ * WRITE, not the page cache, marks it, once the whole WRITE, its fsync
+ * included, is complete.  Ask whether the neighbour has already written its
+ * part immediately before writing @seg: asking earlier gives the neighbour
+ * time to write its part and find this WRITE's part missing.
+ */
+static void
+nfsd_write_dio_seg_hold(struct file *file, const struct nfsd_write_dio_seg *seg,
+			loff_t pos, size_t len, struct nfsd_write_dio_hold *hold)
+{
+	struct address_space *mapping = file->f_mapping;
+
+	hold->first = pos;
+	hold->last = pos + len - 1;
+	hold->complete_first = false;
+	hold->complete_last = false;
+	hold->written = 0;
+	if (seg->claim && len) {
+		bool one_page = (hold->first >> PAGE_SHIFT) ==
+				(hold->last >> PAGE_SHIFT);
+		bool head = hold->first & ~PAGE_MASK;
+		bool tail = (hold->last + 1) & ~PAGE_MASK;
+
+		/* Only a page the segment covers partly is shared. */
+		if (head || (one_page && tail))
+			hold->complete_first =
+				filemap_dontcache_edge_completes(mapping, hold->first);
+		if (!one_page && tail)
+			hold->complete_last =
+				filemap_dontcache_edge_completes(mapping, hold->last);
+	}
+}
+
+/* Drop the shared pages this WRITE was the second of two to write. */
+static void
+nfsd_write_dio_seg_release(struct file *file,
+			   const struct nfsd_write_dio_hold *hold)
+{
+	struct address_space *mapping = file->f_mapping;
+
+	if (hold->complete_first && hold->written)
+		filemap_dontcache_edge_complete(mapping, hold->first);
+	/* A short write did not reach the last page. */
+	if (hold->complete_last &&
+	    hold->written == hold->last - hold->first + 1)
+		filemap_dontcache_edge_complete(mapping, hold->last);
 }
 
 static unsigned int
@@ -1428,7 +1488,9 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
 
 	if (nf->nf_file->f_op->fop_flags & FOP_DONTCACHE)
 		dontcache_flags = IOCB_DONTCACHE;
-	buffered_flags = nfsd_direct_misaligned_dontcache ? dontcache_flags : 0;
+	/* NFSD marks their shared pages itself, see nfsd_write_dio_seg_hold() */
+	buffered_flags = nfsd_direct_misaligned_dontcache && dontcache_flags ?
+			 dontcache_flags | IOCB_DONTCACHE_DEFER : 0;
 
 	/*
 	 * Check if direct I/O is feasible for this write request.
@@ -1483,7 +1545,8 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (prefix) {
 		nfsd_write_dio_seg_init(&segments[nsegs], bvec,
 					nvecs, total, 0, prefix, iocb);
-		segments[nsegs++].flags |= buffered_flags;
+		segments[nsegs].flags |= buffered_flags;
+		segments[nsegs++].claim = !!buffered_flags;
 	}
 
 	nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs,
@@ -1516,7 +1579,8 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (suffix) {
 		nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs, total,
 					prefix + middle, suffix, iocb);
-		segments[nsegs++].flags |= buffered_flags;
+		segments[nsegs].flags |= buffered_flags;
+		segments[nsegs++].claim = !!buffered_flags;
 	}
 	return nsegs;
 
@@ -1525,6 +1589,7 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	nfsd_write_dio_seg_init(&segments[0], bvec, nvecs, total, 0,
 				total, iocb);
 	segments[0].flags |= buffered_flags;
+	segments[0].claim = !!buffered_flags;
 	return 1;
 }
 
@@ -1533,12 +1598,13 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		  struct nfsd_file *nf, unsigned int nvecs,
 		  unsigned long *cnt, struct kiocb *kiocb)
 {
+	struct nfsd_write_dio_hold holds[3];
 	struct nfsd_write_dio_seg segments[3];
 	struct file *file = nf->nf_file;
 	loff_t start = kiocb->ki_pos;
-	unsigned int nsegs, i;
+	unsigned int nsegs, nheld = 0, i;
+	ssize_t host_err = 0;
 	int sync_flags;
-	ssize_t host_err;
 	size_t expected;
 
 	nsegs = nfsd_write_dio_iters_init(rqstp, fhp, nf, rqstp->rq_bvec,
@@ -1567,23 +1633,32 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 					kiocb->ki_flags);
 
 		expected = iov_iter_count(&segments[i].iter);
-
+		nfsd_write_dio_seg_hold(file, &segments[i], kiocb->ki_pos,
+					expected, &holds[nheld]);
 		host_err = vfs_iocb_iter_write(file, kiocb, &segments[i].iter);
 		if (host_err < 0)
-			return host_err;
+			break;
+		holds[nheld++].written = host_err;
 		*cnt += host_err;
 		if (host_err < (ssize_t)expected)
 			break;	/* partial write */
 	}
 
-	if (sync_flags && *cnt) {
+	if (host_err >= 0 && sync_flags && *cnt)
 		host_err = vfs_fsync_range(file, start, start + *cnt - 1,
 					   !(sync_flags & IOCB_SYNC));
-		if (host_err < 0)
-			return host_err;
-	}
 
-	return 0;
+	/*
+	 * Mark the pages this WRITE shares with its neighbours for dropping
+	 * only once the WRITE is complete, including its fsync if it has one.
+	 * A page marked while still dirty is counted as DONTCACHE dirty and
+	 * wakes the DONTCACHE writeback kick; a page marked once clean is
+	 * simply dropped.
+	 */
+	for (i = 0; i < nheld; i++)
+		nfsd_write_dio_seg_release(file, &holds[i]);
+
+	return host_err < 0 ? host_err : 0;
 }
 
 /**
-- 
2.52.0

Re: [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats

[Mike Snitzer]

On Tue, Oct 20, 2026 at 06:03:42PM -0400, Mike Snitzer wrote:
> The per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the
> flusher writes back in response to an IOCB_DONTCACHE kick, but it is
> not reported anywhere, so the DONTCACHE dirty backlog can only be
> inferred from MemFree/Cached movement.
> 
> Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats
> and as WbDontcacheDirty in the per-wb stats, next to the
> Reclaimable line.  Use wb_stat_sum() rather than wb_stat() to fold in
> the per-cpu batches: the flusher targets the summed value and small
> backlogs would otherwise read as zero.
> 
> Fixes: 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")
> Cc: Jeff Layton <jlayton@kernel.org>

FYI, I forgot to pick up Jeff's provided review tag for this patch:

Reviewed-by: Jeff Layton <jlayton@kernel.org>

see: https://lore.kernel.org/linux-nfs/2bee53783a2372b8dc715350b6f6d2ad5d8770ff.camel@kernel.org/

Thanks,
Mike

> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
> Cc: linux-mm@kvack.org
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
>  mm/backing-dev.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index 18e999053bae..b8a2c27e07fa 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -46,6 +46,7 @@ struct wb_stats {
>  	unsigned long nr_dirty_time;
>  	unsigned long nr_writeback;
>  	unsigned long nr_reclaimable;
> +	unsigned long nr_dontcache_dirty;
>  	unsigned long nr_dirtied;
>  	unsigned long nr_written;
>  	unsigned long dirty_thresh;
> @@ -78,6 +79,12 @@ static void collect_wb_stats(struct wb_stats *stats,
>  
>  	stats->nr_writeback += wb_stat(wb, WB_WRITEBACK);
>  	stats->nr_reclaimable += wb_stat(wb, WB_RECLAIMABLE);
> +	/*
> +	 * Sum the per-cpu batches: the dontcache flusher targets this
> +	 * counter via wb_stat_sum() and small backlogs would otherwise
> +	 * read as zero here.
> +	 */
> +	stats->nr_dontcache_dirty += wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
>  	stats->nr_dirtied += wb_stat(wb, WB_DIRTIED);
>  	stats->nr_written += wb_stat(wb, WB_WRITTEN);
>  	stats->wb_thresh += wb_calc_thresh(wb, stats->dirty_thresh);
> @@ -125,6 +132,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
>  	seq_printf(m,
>  		   "BdiWriteback:       %10lu kB\n"
>  		   "BdiReclaimable:     %10lu kB\n"
> +		   "BdiDontcacheDirty:  %10lu kB\n"
>  		   "BdiDirtyThresh:     %10lu kB\n"
>  		   "DirtyThresh:        %10lu kB\n"
>  		   "BackgroundThresh:   %10lu kB\n"
> @@ -139,6 +147,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
>  		   "state:              %10lx\n",
>  		   K(stats.nr_writeback),
>  		   K(stats.nr_reclaimable),
> +		   K(stats.nr_dontcache_dirty),
>  		   K(stats.wb_thresh),
>  		   K(dirty_thresh),
>  		   K(background_thresh),
> @@ -163,6 +172,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
>  		   "WbCgIno:           %10lu\n"
>  		   "WbWriteback:       %10lu kB\n"
>  		   "WbReclaimable:     %10lu kB\n"
> +		   "WbDontcacheDirty:  %10lu kB\n"
>  		   "WbDirtyThresh:     %10lu kB\n"
>  		   "WbDirtied:         %10lu kB\n"
>  		   "WbWritten:         %10lu kB\n"
> @@ -179,6 +189,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
>  #endif
>  		   K(stats->nr_writeback),
>  		   K(stats->nr_reclaimable),
> +		   K(stats->nr_dontcache_dirty),
>  		   K(stats->wb_thresh),
>  		   K(stats->nr_dirtied),
>  		   K(stats->nr_written),
> -- 
> 2.52.0
> 

Re: [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats

[Jeff Layton]

On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> The per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the
> flusher writes back in response to an IOCB_DONTCACHE kick, but it is
> not reported anywhere, so the DONTCACHE dirty backlog can only be
> inferred from MemFree/Cached movement.
> 
> Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats
> and as WbDontcacheDirty in the per-wb stats, next to the
> Reclaimable line.  Use wb_stat_sum() rather than wb_stat() to fold in
> the per-cpu batches: the flusher targets the summed value and small
> backlogs would otherwise read as zero.
> 
> Fixes: 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")
> Cc: Jeff Layton <jlayton@kernel.org>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
> Cc: linux-mm@kvack.org
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
>  mm/backing-dev.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index 18e999053bae..b8a2c27e07fa 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -46,6 +46,7 @@ struct wb_stats {
>  	unsigned long nr_dirty_time;
>  	unsigned long nr_writeback;
>  	unsigned long nr_reclaimable;
> +	unsigned long nr_dontcache_dirty;
>  	unsigned long nr_dirtied;
>  	unsigned long nr_written;
>  	unsigned long dirty_thresh;
> @@ -78,6 +79,12 @@ static void collect_wb_stats(struct wb_stats *stats,
>  
>  	stats->nr_writeback += wb_stat(wb, WB_WRITEBACK);
>  	stats->nr_reclaimable += wb_stat(wb, WB_RECLAIMABLE);
> +	/*
> +	 * Sum the per-cpu batches: the dontcache flusher targets this
> +	 * counter via wb_stat_sum() and small backlogs would otherwise
> +	 * read as zero here.
> +	 */

nit: I don't personally find this comment very helpful.

> +	stats->nr_dontcache_dirty += wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
>  	stats->nr_dirtied += wb_stat(wb, WB_DIRTIED);
>  	stats->nr_written += wb_stat(wb, WB_WRITTEN);
>  	stats->wb_thresh += wb_calc_thresh(wb, stats->dirty_thresh);
> @@ -125,6 +132,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
>  	seq_printf(m,
>  		   "BdiWriteback:       %10lu kB\n"
>  		   "BdiReclaimable:     %10lu kB\n"
> +		   "BdiDontcacheDirty:  %10lu kB\n"
>  		   "BdiDirtyThresh:     %10lu kB\n"
>  		   "DirtyThresh:        %10lu kB\n"
>  		   "BackgroundThresh:   %10lu kB\n"
> @@ -139,6 +147,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
>  		   "state:              %10lx\n",
>  		   K(stats.nr_writeback),
>  		   K(stats.nr_reclaimable),
> +		   K(stats.nr_dontcache_dirty),
>  		   K(stats.wb_thresh),
>  		   K(dirty_thresh),
>  		   K(background_thresh),
> @@ -163,6 +172,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
>  		   "WbCgIno:           %10lu\n"
>  		   "WbWriteback:       %10lu kB\n"
>  		   "WbReclaimable:     %10lu kB\n"
> +		   "WbDontcacheDirty:  %10lu kB\n"
>  		   "WbDirtyThresh:     %10lu kB\n"
>  		   "WbDirtied:         %10lu kB\n"
>  		   "WbWritten:         %10lu kB\n"
> @@ -179,6 +189,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
>  #endif
>  		   K(stats->nr_writeback),
>  		   K(stats->nr_reclaimable),
> +		   K(stats->nr_dontcache_dirty),
>  		   K(stats->wb_thresh),
>  		   K(stats->nr_dirtied),
>  		   K(stats->nr_written),

Patch looks fine though:

Reviewed-by: Jeff Layton <jlayton@kernel.org>

Re: [PATCH v7 2/5] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it

[Jeff Layton]

On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> A folio allocated by an RWF_DONTCACHE write is marked dropbehind before
> it is dirtied, so folio_account_dirtied() adds it to WB_DONTCACHE_DIRTY
> and folio_clear_dirty_for_io() or folio_account_cleaned() subtracts it
> when it is cleaned.  Nothing marks a folio that already exists.  A
> caller that sets PG_dropbehind on a folio that is already dirty
> unbalances the counter: the clean subtracts what was never added, and
> once the counter goes negative, wb_stat_sum() reads it as zero and the
> DONTCACHE writeback kick stops writing anything back for every writer
> on that bdi_writeback.
> 
> Add folio_mark_dropbehind(), the inverse of the clear in
> __filemap_get_folio_mpol(): with the folio locked, mark it and, if it is
> dirty, add it to WB_DONTCACHE_DIRTY.  The folio lock orders this
> against dirtying and against folio_clear_dirty_for_io().  A concurrent
> lookup without FGP_DONTCACHE that clears the flag again subtracts what
> this added.
> 
> The name follows folio_set_dirty() and folio_mark_dirty(): the first
> only sets PG_dirty, the second also accounts the dirty folio.
> folio_set_dropbehind() likewise only sets PG_dropbehind, and
> folio_mark_dropbehind() also accounts it in WB_DONTCACHE_DIRTY.
> 
> A later commit uses it in NFSD to drop a page shared by two misaligned
> WRITEs once both have written it.
> 
> Cc: Jeff Layton <jlayton@kernel.org>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
> Cc: linux-mm@kvack.org
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
>  include/linux/pagemap.h |  1 +
>  mm/filemap.c            | 31 +++++++++++++++++++++++++++++++
>  2 files changed, 32 insertions(+)
> 
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index 0adfa6605653..3012bd95812a 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -1310,6 +1310,7 @@ void end_page_writeback(struct page *page);
>  void folio_end_writeback(struct folio *folio);
>  void folio_end_writeback_no_dropbehind(struct folio *folio);
>  void folio_end_dropbehind(struct folio *folio);
> +void folio_mark_dropbehind(struct folio *folio);
>  void folio_wait_stable(struct folio *folio);
>  void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
>  void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
> diff --git a/mm/filemap.c b/mm/filemap.c
> index 00fd89cf6f55..ed7e4dbea653 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -1644,6 +1644,37 @@ void folio_end_dropbehind(struct folio *folio)
>  }
>  EXPORT_SYMBOL_GPL(folio_end_dropbehind);
>  
> +/**
> + * folio_mark_dropbehind - mark an existing folio dropbehind
> + * @folio: the folio, locked by the caller
> + *
> + * Mark @folio so that the writeback that cleans it drops it.  A folio that
> + * is already dirty is added to WB_DONTCACHE_DIRTY, as folio_account_dirtied()
> + * would have added it had it been marked when dirtied, so that the clean
> + * that subtracts it leaves the counter balanced.
> + */
> +void folio_mark_dropbehind(struct folio *folio)
> +{
> +	struct address_space *mapping = folio->mapping;
> +
> +	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
> +
> +	if (folio_test_dropbehind(folio))
> +		return;
> +	folio_set_dropbehind(folio);
> +	if (mapping && folio_test_dirty(folio) &&
> +	    mapping_can_writeback(mapping)) {
> +		struct inode *inode = mapping->host;
> +		struct bdi_writeback *wb;
> +		struct wb_lock_cookie cookie = {};
> +
> +		wb = unlocked_inode_to_wb_begin(inode, &cookie);
> +		wb_stat_mod(wb, WB_DONTCACHE_DIRTY, folio_nr_pages(folio));
> +		unlocked_inode_to_wb_end(inode, &cookie);
> +	}
> +}
> +EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
> +
>  /**
>   * folio_end_writeback_no_dropbehind - End writeback against a folio.
>   * @folio: The folio.

Chuck's sashiko had a comment on this one:

[Severity: Medium]
Could this overcount WB_DONTCACHE_DIRTY if folio_mark_dropbehind() races with
filemap_dirty_folio() in mm/page-writeback.c? The dirtying path can set
PG_dirty before folio_account_dirtied() checks PG_dropbehind, and it need not
hold the folio lock. If folio_mark_dropbehind() sets PG_dropbehind after that
dirty transition but before the accounting check, both paths increment the
counter, while folio_account_cleaned() subtracts it only once. The later NFSD
use of this helper retains the same interleaving. Could the flag update and
dirty accounting be serialized so the counter is incremented only once?
-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH v7 4/5] mm, iomap: let an uncached writer complete its shared pages itself

[Jeff Layton]

On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> The write that completes a page an uncached write covers only partly
> marks it drop-behind as it writes, while the page is dirty, so the page
> counts as DONTCACHE dirty and the DONTCACHE writeback kick writes it
> back.  A writer that syncs its write itself, once, after several
> segments -- NFSD answering a stable WRITE, the NFS client's LOCALIO path
> -- issues each segment without IOCB_DSYNC, so every segment's kick
> writes back pages its own sync is about to write.  Measured with NFSD
> DIRECT and 32 writers of 47008-byte records: with the previous patch
> alone, 23,000 to 65,000 kick rounds a run, where NFSD marking the pages
> itself after its sync had taken 51 to 274; with this patch and NFSD
> using it, a FILE_SYNC WRITE wakes the kick not at all.
> 
> Let such a writer complete those pages itself.  IOCB_DONTCACHE_DEFER
> (IOMAP_DONTCACHE_DEFER, FGP_DONTCACHE_DEFER) leaves a partly covered page
> unmarked even when the write finds it uptodate.  Right before writing such
> a page the writer asks filemap_dontcache_edge_completes() whether it is
> uptodate already, which means the neighbouring write has written its part
> and this write completes it; once the write and its sync are done,
> filemap_dontcache_edge_complete() marks it and, clean, drops it at once.
> 
> Two writes that arrive together may both find the page absent before
> either has written; that page is then left cached until reclaim, which
> the folio-lock ordering of the default path avoids.
> 
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
>  fs/iomap/buffered-io.c  |  4 +++
>  fs/iomap/trace.h        |  3 ++-
>  include/linux/fs.h      |  8 +++++-
>  include/linux/iomap.h   |  1 +
>  include/linux/pagemap.h |  6 +++++
>  mm/filemap.c            | 58 ++++++++++++++++++++++++++++++++++++++++-
>  6 files changed, 77 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
> index 780a0186045d..5ca82000e2ec 100644
> --- a/fs/iomap/buffered-io.c
> +++ b/fs/iomap/buffered-io.c
> @@ -811,6 +811,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
>  		 */
>  		if (offset_in_page(pos) || len < PAGE_SIZE) {
>  			fgp |= FGP_DONTCACHE_EDGE;
> +			if (iter->flags & IOMAP_DONTCACHE_DEFER)
> +				fgp |= FGP_DONTCACHE_DEFER;
>  			len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
>  		}
>  	}
> @@ -1318,6 +1320,8 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
>  		iter.flags |= IOMAP_NOWAIT;
>  	if (iocb->ki_flags & IOCB_DONTCACHE)
>  		iter.flags |= IOMAP_DONTCACHE;
> +	if (iocb->ki_flags & IOCB_DONTCACHE_DEFER)
> +		iter.flags |= IOMAP_DONTCACHE_DEFER;
>  
>  	while ((ret = iomap_iter(&iter, ops)) > 0)
>  		iter.status = iomap_write_iter(&iter, i, write_ops);
> diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
> index e1ea8392cf47..4f0343eeed1b 100644
> --- a/fs/iomap/trace.h
> +++ b/fs/iomap/trace.h
> @@ -104,7 +104,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
>  	{ IOMAP_UNSHARE,	"UNSHARE" }, \
>  	{ IOMAP_DAX,		"DAX" }, \
>  	{ IOMAP_ATOMIC,		"ATOMIC" }, \
> -	{ IOMAP_DONTCACHE,	"DONTCACHE" }
> +	{ IOMAP_DONTCACHE,	"DONTCACHE" }, \
> +	{ IOMAP_DONTCACHE_DEFER, "DONTCACHE_DEFER" }
>  
>  #define IOMAP_F_FLAGS_STRINGS \
>  	{ IOMAP_F_NEW,		"NEW" }, \
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index f9d1e05e8ae6..ad181b6ce6b4 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -357,6 +357,11 @@ struct readahead_control;
>  /* kiocb is a read or write operation submitted by fs/aio.c. */
>  #define IOCB_AIO_RW		(1 << 22)
>  #define IOCB_HAS_METADATA	(1 << 23)
> +/*
> + * With IOCB_DONTCACHE: the caller completes the pages the write covers only
> + * partly itself, see filemap_dontcache_edge_complete().
> + */
> +#define IOCB_DONTCACHE_DEFER	(1 << 24)
>  
>  /* for use in trace events */
>  #define TRACE_IOCB_STRINGS \
> @@ -374,7 +379,8 @@ struct readahead_control;
>  	{ IOCB_NOIO,		"NOIO" }, \
>  	{ IOCB_ALLOC_CACHE,	"ALLOC_CACHE" }, \
>  	{ IOCB_AIO_RW,		"AIO_RW" }, \
> -	{ IOCB_HAS_METADATA,	"AIO_HAS_METADATA" }
> +	{ IOCB_HAS_METADATA,	"AIO_HAS_METADATA" }, \
> +	{ IOCB_DONTCACHE_DEFER,	"DONTCACHE_DEFER" }
>  
>  struct kiocb {
>  	struct file		*ki_filp;
> diff --git a/include/linux/iomap.h b/include/linux/iomap.h
> index bc7ae6327dbf..f3509d495a92 100644
> --- a/include/linux/iomap.h
> +++ b/include/linux/iomap.h
> @@ -212,6 +212,7 @@ struct iomap_write_ops {
>  #endif /* CONFIG_FS_DAX */
>  #define IOMAP_ATOMIC		(1 << 9) /* torn-write protection */
>  #define IOMAP_DONTCACHE		(1 << 10)
> +#define IOMAP_DONTCACHE_DEFER	(1 << 11) /* see IOCB_DONTCACHE_DEFER */
>  
>  /*
>   * Return the existing mapping at pos, or reserve space starting at pos for up
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index 0c1d6bd09aa0..a0a67e4c940b 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -705,6 +705,9 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
>   *   only partly, and a neighbouring write may complete it.  A folio created
>   *   for it is not marked drop-behind; a folio found uptodate is, as the
>   *   write that completes it.
> + * * %FGP_DONTCACHE_DEFER - With %FGP_DONTCACHE_EDGE: the caller completes the
> + *   page itself (filemap_dontcache_edge_complete()), so a folio found uptodate
> + *   is not marked either.
>   * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
>   *   implementation.
>   */
> @@ -720,6 +723,7 @@ typedef unsigned int __bitwise fgf_t;
>  #define FGP_STABLE		((__force fgf_t)0x00000080)
>  #define FGP_DONTCACHE		((__force fgf_t)0x00000100)
>  #define FGP_DONTCACHE_EDGE	((__force fgf_t)0x00000200)
> +#define FGP_DONTCACHE_DEFER	((__force fgf_t)0x00000400)
>  #define FGF_GET_ORDER(fgf)	(((__force unsigned)fgf) >> 26)	/* top 6 bits */
>  
>  #define FGP_WRITEBEGIN		(FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
> @@ -1316,6 +1320,8 @@ void folio_end_writeback(struct folio *folio);
>  void folio_end_writeback_no_dropbehind(struct folio *folio);
>  void folio_end_dropbehind(struct folio *folio);
>  void folio_mark_dropbehind(struct folio *folio);
> +bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos);
> +void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos);
>  void folio_wait_stable(struct folio *folio);
>  void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
>  void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
> diff --git a/mm/filemap.c b/mm/filemap.c
> index 0397ec771547..7d7c56a42643 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -1675,6 +1675,61 @@ void folio_mark_dropbehind(struct folio *folio)
>  }
>  EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
>  
> +/**
> + * filemap_dontcache_edge_completes - will this write complete a shared page?
> + * @mapping: the file's mapping
> + * @pos: any byte offset within the page
> + *
> + * For an uncached writer that completes the pages it covers only partly
> + * itself (IOCB_DONTCACHE_DEFER).  Call right before writing such a page: it
> + * is in the page cache and uptodate when the neighbouring write has already
> + * written its part, and this write then completes it.  Two writes that
> + * arrive together may both find it absent; the page is then left cached,
> + * unmarked, until reclaim.
> + *
> + * Return: true if the caller is to call filemap_dontcache_edge_complete()
> + * once its write, including any sync of it, is done.
> + */
> +bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos)
> +{
> +	struct folio *folio;
> +	bool ret;
> +
> +	/* FGP_DONTCACHE: a plain lookup would clear a drop-behind mark. */
> +	folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT, FGP_DONTCACHE, 0);
> +	if (IS_ERR(folio))
> +		return false;
> +	ret = folio_test_uptodate(folio);
> +	folio_put(folio);
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(filemap_dontcache_edge_completes);
> +
> +/**
> + * filemap_dontcache_edge_complete - drop a shared page both writes have written
> + * @mapping: the file's mapping
> + * @pos: any byte offset within the page
> + *
> + * Mark the page drop-behind.  If it is already written back, drop it now;
> + * otherwise the writeback that cleans it drops it.  A dirty page marked here
> + * counts as DONTCACHE dirty and wakes the DONTCACHE writeback kick, so a
> + * caller that syncs its write calls this after the sync.
> + */
> +void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos)
> +{
> +	struct folio *folio;
> +
> +	folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT,
> +				    FGP_LOCK | FGP_DONTCACHE, 0);
> +	if (IS_ERR(folio))
> +		return;
> +	folio_mark_dropbehind(folio);
> +	folio_unlock(folio);
> +	folio_end_dropbehind(folio);
> +	folio_put(folio);
> +}
> +EXPORT_SYMBOL_GPL(filemap_dontcache_edge_complete);
> +
>  /**
>   * folio_end_writeback_no_dropbehind - End writeback against a folio.
>   * @folio: The folio.
> @@ -2024,7 +2079,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
>  	 * folio lock, so the writeback that cleans both writes' data drops it.
>  	 */
>  	if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
> -	    (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
> +	    !(fgp_flags & FGP_DONTCACHE_DEFER) && (fgp_flags & FGP_LOCK) &&
> +	    folio_test_uptodate(folio))
>  		folio_mark_dropbehind(folio);
>  no_page:
>  	if (!folio && (fgp_flags & FGP_CREAT)) {

2 comments from Chuck's sashiko on this one:

[Severity: Low]
Could filemap_dontcache_edge_complete() in mm/filemap.c avoid missing the
immediate drop when another task acquires the folio lock between
folio_unlock() and folio_end_dropbehind()? In that case folio_end_dropbehind()
fails its folio_trylock() and does not retry; if the folio is already clean,
there may be no later writeback completion to drop it, so it can remain cached
until unrelated activity or reclaim.

[Severity: Medium]
Could filemap_dontcache_edge_complete() in mm/filemap.c restrict marking to
the boundary page when the mapping returns a large folio? iomap_get_folio()
in fs/iomap/buffered-io.c limits a partial edge request to one page, but the
mapping's minimum folio order can still cause __filemap_get_folio_mpol() to
return a multi-page folio, or the lookup can find an existing large folio.
This helper then marks the entire folio for drop-behind. For example, if a
caller uses deferred completion on an XFS mapping with a minimum folio order,
marking one edge can account the whole dirty folio for DONTCACHE writeback or
invalidate the whole clean, unmapped folio. Could this path handle only the
page covered by the completed edge?

-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH v7 3/5] mm, iomap: keep a partly covered page until an uncached write completes it

[Jeff Layton]

On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> An uncached (RWF_DONTCACHE) buffered write marks drop-behind every folio
> it creates, so writeback drops each one as soon as this write's data is
> on disk.  A page the write covers only partly is usually shared with the
> neighbouring write, the one that ends or starts in it.  Once the first
> of the two has been written back, the page is gone, and the second has
> to read the block back from disk before it can write its part: one read
> per misaligned write.
> 
> Measured locally on XFS (emulated 4Kn NVMe, 4 KiB pages), 32 threads
> writing interleaved records to one file with pwritev2(), 1 GiB per run,
> before and after this patch:
> 
>   record  flags                     4 KiB device       MiB/s
>   bytes                             reads per write
>                                     before   after     before  after
>   47008   RWF_DONTCACHE             0.97     0.000     831     2281
>   47008   RWF_DONTCACHE|RWF_DSYNC   0.99     0.000     601     1227
>   6000    RWF_DONTCACHE             0.97     0.000     121      488
>   6000    RWF_DONTCACHE|RWF_DSYNC   0.99     0.000      90      206
> 
> Before this patch every write waited for its read, and because XFS holds
> its IOLOCK exclusively across a buffered write, the other 31 writers
> waited too: both record sizes ran at about 20 thousand writes a second,
> one device read per write; after it nothing is read back.  With
> RWF_DSYNC each write still waits for its own sync, so the gain is
> smaller: the uncached write now runs as fast as the same write without
> RWF_DONTCACHE, 1237 and 209 MiB/s against 1227 and 206 in the table.
> Records of 49152 bytes, which cover only whole pages, read nothing
> before or after.
> 
> NFSD DIRECT issues the misaligned ends of a WRITE as buffered uncached
> segments and gains the same way.  Without any handling of the shared
> page, 32 interleaved writers of 47008-byte records read 45144 blocks
> for 45664 WRITEs; with this patch, and no NFSD code for the shared page,
> 0.0001 to 0.0008 blocks per WRITE whether each record is sent as one
> FILE_SYNC WRITE, as two one after the other, as two together, or as two
> UNSTABLE.  The page cache then marks the completed page while it is
> still dirty, so each segment of a stable WRITE wakes the DONTCACHE
> writeback kick (23 to 65 thousand rounds a run); the next two patches
> let NFSD mark it after its sync instead.
> 
> Let the write path say which pages a write covers only partly
> (FGP_DONTCACHE_EDGE, set by iomap when the write starts inside a page or
> ends before its end) and get such a page as a single page.  A folio
> created for it is not marked, so writing back this write's part does not
> drop it.  A folio found uptodate is the neighbour's, already written:
> this write completes it, so mark it, under the folio lock and before its
> own data goes in, and the writeback that cleans both writes' data drops
> it.  The folio lock orders the two writes, so the second always finds
> the first's data in place.
> 
> A page whose neighbour never writes, such as the last page of a file,
> stays cached until reclaim.  A partly covered page another task had
> cached and that is uptodate is now dropped after the write's writeback,
> where before an uncached write never dropped a page it did not create.
> 
> Only iomap file systems say which pages a write covers partly so far;
> write_begin_get_folio() users such as ext4 and the NFS client do not.
> With blocks smaller than a page, a write that crosses an extent boundary
> inside a page asks for it as two partly covered halves, so the page,
> though one write covers it whole, stays cached until reclaim.
> 
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
>  fs/iomap/buffered-io.c  | 12 +++++++++++-
>  include/linux/pagemap.h |  5 +++++
>  mm/filemap.c            | 19 ++++++++++++++++++-
>  3 files changed, 34 insertions(+), 2 deletions(-)
> 

The nfsd case typically uses buffered or DONTCACHE for the unaligned
sub-page-sized "ends" when direct is in force, so mostly these ends get
submitted as small (< PAGE_SIZE or folio_size()) buffered I/Os.

Why do we need this special casing instead of just making those
unaligned ends use regular buffered I/O? Isn't it basically the same
thing?
 
> diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
> index 0a5ebfda90f1..780a0186045d 100644
> --- a/fs/iomap/buffered-io.c
> +++ b/fs/iomap/buffered-io.c
> @@ -802,8 +802,18 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
>  
>  	if (iter->flags & IOMAP_NOWAIT)
>  		fgp |= FGP_NOWAIT;
> -	if (iter->flags & IOMAP_DONTCACHE)
> +	if (iter->flags & IOMAP_DONTCACHE) {
>  		fgp |= FGP_DONTCACHE;
> +		/*
> +		 * A page this write covers only partly is shared with a
> +		 * neighbouring write: get it as a single page, which is
> +		 * kept until the write that completes it.
> +		 */
> +		if (offset_in_page(pos) || len < PAGE_SIZE) {
> +			fgp |= FGP_DONTCACHE_EDGE;
> +			len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
> +		}
> +	}
>  	fgp |= fgf_set_order(len);
>  
>  	return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index 3012bd95812a..0c1d6bd09aa0 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -701,6 +701,10 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
>   * * %FGP_NOWAIT - Don't block on the folio lock.
>   * * %FGP_STABLE - Wait for the folio to be stable (finished writeback)
>   * * %FGP_DONTCACHE - Uncached buffered IO
> + * * %FGP_DONTCACHE_EDGE - With %FGP_DONTCACHE: the write covers this page
> + *   only partly, and a neighbouring write may complete it.  A folio created
> + *   for it is not marked drop-behind; a folio found uptodate is, as the
> + *   write that completes it.
>   * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
>   *   implementation.
>   */
> @@ -715,6 +719,7 @@ typedef unsigned int __bitwise fgf_t;
>  #define FGP_FOR_MMAP		((__force fgf_t)0x00000040)
>  #define FGP_STABLE		((__force fgf_t)0x00000080)
>  #define FGP_DONTCACHE		((__force fgf_t)0x00000100)
> +#define FGP_DONTCACHE_EDGE	((__force fgf_t)0x00000200)
>  #define FGF_GET_ORDER(fgf)	(((__force unsigned)fgf) >> 26)	/* top 6 bits */
>  
>  #define FGP_WRITEBEGIN		(FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
> diff --git a/mm/filemap.c b/mm/filemap.c
> index ed7e4dbea653..0397ec771547 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -2016,6 +2016,16 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
>  
>  	if (fgp_flags & FGP_STABLE)
>  		folio_wait_stable(folio);
> +
> +	/*
> +	 * An uncached write that covers this folio only partly, finding it
> +	 * uptodate, is the write that completes it: the neighbouring write
> +	 * that left it here has written its part.  Mark it now, under the
> +	 * folio lock, so the writeback that cleans both writes' data drops it.
> +	 */
> +	if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
> +	    (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
> +		folio_mark_dropbehind(folio);
>  no_page:
>  	if (!folio && (fgp_flags & FGP_CREAT)) {
>  		unsigned int min_order = mapping_min_folio_order(mapping);
> @@ -2053,7 +2063,14 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
>  			/* Init accessed so avoid atomic mark_page_accessed later */
>  			if (fgp_flags & FGP_ACCESSED)
>  				__folio_set_referenced(folio);
> -			if (fgp_flags & FGP_DONTCACHE)
> +			/*
> +			 * A folio the write covers only partly is left
> +			 * unmarked, so that writing back this write's part
> +			 * does not drop it before a neighbouring write has
> +			 * written the rest.
> +			 */
> +			if ((fgp_flags & FGP_DONTCACHE) &&
> +			    !(fgp_flags & FGP_DONTCACHE_EDGE))
>  				__folio_set_dropbehind(folio);
>  
>  			err = filemap_add_folio(mapping, folio, index, gfp);

-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH v7 3/5] mm, iomap: keep a partly covered page until an uncached write completes it

[Mike Snitzer]

On Wed, Oct 21, 2026 at 10:36:07AM -0400, Jeff Layton wrote:
> On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> > An uncached (RWF_DONTCACHE) buffered write marks drop-behind every folio
> > it creates, so writeback drops each one as soon as this write's data is
> > on disk.  A page the write covers only partly is usually shared with the
> > neighbouring write, the one that ends or starts in it.  Once the first
> > of the two has been written back, the page is gone, and the second has
> > to read the block back from disk before it can write its part: one read
> > per misaligned write.
> > 
> > Measured locally on XFS (emulated 4Kn NVMe, 4 KiB pages), 32 threads
> > writing interleaved records to one file with pwritev2(), 1 GiB per run,
> > before and after this patch:
> > 
> >   record  flags                     4 KiB device       MiB/s
> >   bytes                             reads per write
> >                                     before   after     before  after
> >   47008   RWF_DONTCACHE             0.97     0.000     831     2281
> >   47008   RWF_DONTCACHE|RWF_DSYNC   0.99     0.000     601     1227
> >   6000    RWF_DONTCACHE             0.97     0.000     121      488
> >   6000    RWF_DONTCACHE|RWF_DSYNC   0.99     0.000      90      206
> > 
> > Before this patch every write waited for its read, and because XFS holds
> > its IOLOCK exclusively across a buffered write, the other 31 writers
> > waited too: both record sizes ran at about 20 thousand writes a second,
> > one device read per write; after it nothing is read back.  With
> > RWF_DSYNC each write still waits for its own sync, so the gain is
> > smaller: the uncached write now runs as fast as the same write without
> > RWF_DONTCACHE, 1237 and 209 MiB/s against 1227 and 206 in the table.
> > Records of 49152 bytes, which cover only whole pages, read nothing
> > before or after.
> > 
> > NFSD DIRECT issues the misaligned ends of a WRITE as buffered uncached
> > segments and gains the same way.  Without any handling of the shared
> > page, 32 interleaved writers of 47008-byte records read 45144 blocks
> > for 45664 WRITEs; with this patch, and no NFSD code for the shared page,
> > 0.0001 to 0.0008 blocks per WRITE whether each record is sent as one
> > FILE_SYNC WRITE, as two one after the other, as two together, or as two
> > UNSTABLE.  The page cache then marks the completed page while it is
> > still dirty, so each segment of a stable WRITE wakes the DONTCACHE
> > writeback kick (23 to 65 thousand rounds a run); the next two patches
> > let NFSD mark it after its sync instead.
> > 
> > Let the write path say which pages a write covers only partly
> > (FGP_DONTCACHE_EDGE, set by iomap when the write starts inside a page or
> > ends before its end) and get such a page as a single page.  A folio
> > created for it is not marked, so writing back this write's part does not
> > drop it.  A folio found uptodate is the neighbour's, already written:
> > this write completes it, so mark it, under the folio lock and before its
> > own data goes in, and the writeback that cleans both writes' data drops
> > it.  The folio lock orders the two writes, so the second always finds
> > the first's data in place.
> > 
> > A page whose neighbour never writes, such as the last page of a file,
> > stays cached until reclaim.  A partly covered page another task had
> > cached and that is uptodate is now dropped after the write's writeback,
> > where before an uncached write never dropped a page it did not create.
> > 
> > Only iomap file systems say which pages a write covers partly so far;
> > write_begin_get_folio() users such as ext4 and the NFS client do not.
> > With blocks smaller than a page, a write that crosses an extent boundary
> > inside a page asks for it as two partly covered halves, so the page,
> > though one write covers it whole, stays cached until reclaim.
> > 
> > Assisted-by: LLM
> > Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> > ---
> >  fs/iomap/buffered-io.c  | 12 +++++++++++-
> >  include/linux/pagemap.h |  5 +++++
> >  mm/filemap.c            | 19 ++++++++++++++++++-
> >  3 files changed, 34 insertions(+), 2 deletions(-)
> > 
> 
> The nfsd case typically uses buffered or DONTCACHE for the unaligned
> sub-page-sized "ends" when direct is in force, so mostly these ends get
> submitted as small (< PAGE_SIZE or folio_size()) buffered I/Os.

Yes.
 
> Why do we need this special casing instead of just making those
> unaligned ends use regular buffered I/O? Isn't it basically the same
> thing?

No.

For the first of the two writes that share a page it is the same
thing: 3/5 gets that page as a single page and doesn't mark it
drop-behind, just as a regular buffered write wouldn't.  The
difference is what happens after the second write completes the page.
With regular buffered I/O the page stays cached until reclaim; with
3/5 the second write marks it, and the writeback that cleans both
writes' data drops it.

That difference is the point of NFSD DIRECT: each misaligned WRITE
leaves one shared page behind, so with buffered ends the page cache
grows by about a page per WRITE and nothing ever reads it.  We
measured exactly the alternative you describe, with NFSD's
direct_misaligned_dontcache knob, on IOR-hard (47008-byte O_DIRECT
records, 192 ranks on 3 clients, 4 NFS servers, XFS on NVMe, 240 s):

  ends written as                write MiB/s  device reads  page cache
                                              per record    growth
  regular buffered               35,140       0.000         +653.7 GiB
  DONTCACHE, the shared page     34,463       0.015         +10.7 GiB
  kept until completed

Same rate, and neither reads back, but buffered ends filled ~650 GiB of
page cache in four minutes (about one 4 KiB page per 47008-byte record,
8.7% of everything written), which a longer run turns into reclaim.
The second row is an NFSD-only version of this (the shared page
claimed in nfsd itself); Chuck asked for it to live in mm, which is
patches 3/5 to 5/5 of this series -- which yields even less "device
reads per record" than the 0.015 result above (again, because mm is
the proper place for these changes, less unqualified work by NFSD).

And it isn't only NFSD's: any RWF_DONTCACHE writer whose writes share
pages reads each one back today (0.97 reads per write in the cover
letter's pwritev2() table, 831 -> 2281 MiB/s with 3/5), because
writeback drops the page after the first of the two writes.  Making
those writes use regular buffered I/O isn't an option for an
application that asked for RWF_DONTCACHE.

Thanks,
Mike

Re: [PATCH v7 0/5] mm, iomap, NFSD: stop uncached writes reading back shared pages

[Mike Snitzer]

On Tue, Oct 20, 2026 at 06:03:41PM -0400, Mike Snitzer wrote:
> Hi,
> 
> Please consider this v7 for the v7.5 development cycle, so that it can
> go upstream in the v7.5 merge window.
> 
> An uncached (RWF_DONTCACHE) buffered write drops each page it writes
> once that page is written back.  A page the write covers only partly is
> shared with the neighbouring write, and once the first of the two has
> been written back the second has to read it back from disk before it
> can write its part.  32 threads writing interleaved records to one file
> on XFS (emulated 4Kn NVMe, 4 KiB pages), before and after 3/5:
> 
>   record  pwritev2() flags          4 KiB device       MiB/s
>   bytes                             reads per write
>                                     before   after     before  after
>   47008   RWF_DONTCACHE             0.97     0.000     831     2281
>   47008   RWF_DONTCACHE|RWF_DSYNC   0.99     0.000     601     1227
>   6000    RWF_DONTCACHE             0.97     0.000     121      488
>   6000    RWF_DONTCACHE|RWF_DSYNC   0.99     0.000      90      206

For anyone who wants to reproduce the table, the two inlined files
(below) are what produced the data, with no NFS involved:

  dcwrite.c  N threads writing interleaved records to one file, thread
             r's record s at offset (s*N + r)*RECSZ, one pwritev2() per
             record with the given RWF_ flags.  Each record shares its
             first and last page with the records of two other threads.

  dc-ab.sh   for each record size and flag set: a fresh XFS on an
             emulated 4Kn NVMe namespace, dcwrite, then the device's
             4 KiB reads per write (from /sys/block/<dev>/stat around
             the write and a final sync), the file's resident pages
             after the sync (fincore), and the write rate.

The device is the in-kernel NVMe target's file backend on a tmpfs
image, attached over nvme-loop, so it has NVMe's queue limits and a
4096-byte logical block on a 4 KiB-page kernel, and every read the
page cache sends it is counted.  It needs root, CONFIG_NVME_TARGET and
CONFIG_NVME_TARGET_LOOP, nvme-cli, xfsprogs, util-linux's fincore, and
4 GiB free in /dev/shm:

  cc -O2 -pthread -o dcwrite dcwrite.c
  ./dc-ab.sh /tmp/dc-ab ./dcwrite

On kernel with this v7 patchset's patches 1-4:

rs     flags            | writes    4K-reads   reads/write READ-MiB  | resident-pages  | MiB/s
49152  none             | 21824     15         0.0007      0.1       | 261888/261888   | 318.6
49152  dontcache        | 21824     15         0.0007      0.1       | 0/261888        | 4433.3
49152  dsync            | 21824     15         0.0007      0.1       | 261888/261888   | 1074.9
49152  dontcache,dsync  | 21824     15         0.0007      0.1       | 0/261888        | 1475.3
47008  none             | 22816     15         0.0007      0.1       | 261850/261850   | 3379.0
47008  dontcache        | 22816     15         0.0007      0.1       | 1/261850        | 3053.8
47008  dsync            | 22816     15         0.0007      0.1       | 261850/261850   | 1205.3
47008  dontcache,dsync  | 22816     15         0.0007      0.1       | 1/261850        | 1173.4
6000   none             | 178944    15         0.0001      0.1       | 262125/262125   | 1923.5
6000   dontcache        | 178944    15         0.0001      0.1       | 0/262125        | 457.5
6000   dsync            | 178944    15         0.0001      0.1       | 262125/262125   | 204.8
6000   dontcache,dsync  | 178944    26         0.0001      0.1       | 11/262125       | 200.0

(The rates are from a 4 KiB-page arm64 VM and will differ elsewhere;
 the reads per write should not.)

The defaults are the cover letter's: 32 threads, 1 GiB per run, record
sizes 49152, 47008 and 6000 bytes, and flags none, dontcache, dsync and
dontcache,dsync (NT, TOTAL, RSS and FLAGSETS override them).  Run it
once on a kernel without the series and once with it.  What to look
for:

- 47008 and 6000 with dontcache, with or without dsync: about one read
  per write without the series, none with it.  These are the table's
  rows.
- 49152 with dontcache: the control.  Its records cover only whole
  pages, so it reads nothing either way.
- none and dsync: plain buffered writes.  They read nothing either way,
  and leave the whole file resident; the dontcache rows leave next to
  none.

Thanks,
Mike

/* dcwrite: interleaved misaligned buffered writers, no NFS.  N threads; thread
 * r's record s lands at (s*N + r)*RECSZ, the iorhard geometry, so each record
 * shares its first and last page with the records of two other threads.
 * Each record is one pwritev2() with FLAGS (RWF_DONTCACHE, RWF_DSYNC, ...).
 * With "direct" the file is opened O_DIRECT and the buffer is page aligned.
 * Reports the write phase alone and, after a final fsync(), "to stable".
 *
 * usage: dcwrite FILE NTHREADS RECSZ TOTALBYTES FLAGS
 *   FLAGS: comma list of none,dontcache,dsync,direct
 * Exit 1 at the first failed or short write, or a failed fsync(). */
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/uio.h>
#include <time.h>
#include <unistd.h>
#ifndef RWF_DONTCACHE
#define RWF_DONTCACHE 0x00000080
#endif

static int N, fd, flags, odirect; static long RECSZ, NREC; static volatile int failed;

static void *writer(void *arg)
{
	long r = (long)arg;
	char *buf = NULL;
	if (posix_memalign((void **)&buf, 4096, RECSZ)) { failed = 1; return NULL; }
	memset(buf, 'A' + r % 26, RECSZ);
	for (long s = 0; s < NREC && !failed; s++) {
		off_t off = (off_t)(s * N + r) * RECSZ;
		struct iovec iov = { buf, RECSZ };
		ssize_t n = pwritev2(fd, &iov, 1, off, flags);
		if (n != RECSZ) {
			fprintf(stderr, "thread %ld: pwritev2 at %lld: %s\n", r, (long long)off,
				n < 0 ? strerror(errno) : "short write");
			failed = 1;
		}
	}
	free(buf);
	return NULL;
}

int main(int argc, char **argv)
{
	if (argc != 6) { fprintf(stderr, "usage: %s FILE NTHREADS RECSZ TOTALBYTES FLAGS\n", argv[0]); return 2; }
	N = atoi(argv[2]); RECSZ = atol(argv[3]);
	if (N < 1 || N > 1024 || RECSZ < 1) { fprintf(stderr, "bad NTHREADS or RECSZ\n"); return 2; }
	NREC = atol(argv[4]) / RECSZ / N;
	for (char *t = strtok(argv[5], ","); t; t = strtok(NULL, ","))
		if (!strcmp(t, "dontcache")) flags |= RWF_DONTCACHE;
		else if (!strcmp(t, "dsync")) flags |= RWF_DSYNC;
		else if (!strcmp(t, "direct")) odirect = O_DIRECT;
		else if (strcmp(t, "none")) { fprintf(stderr, "bad flag %s\n", t); return 2; }
	fd = open(argv[1], O_WRONLY | O_CREAT | O_TRUNC | odirect, 0644);
	if (fd < 0) { perror(argv[1]); return 1; }
	pthread_t *t = calloc(N, sizeof(*t));
	struct timespec a, b; clock_gettime(CLOCK_MONOTONIC, &a);
	for (long r = 0; r < N; r++)
		if (pthread_create(&t[r], NULL, writer, (void *)r)) { perror("pthread_create"); return 1; }
	for (int r = 0; r < N; r++) pthread_join(t[r], NULL);
	clock_gettime(CLOCK_MONOTONIC, &b);
	int frc = fsync(fd);
	struct timespec c; clock_gettime(CLOCK_MONOTONIC, &c);
	double sec = (b.tv_sec - a.tv_sec) + (b.tv_nsec - a.tv_nsec) / 1e9;
	double ssec = (c.tv_sec - a.tv_sec) + (c.tv_nsec - a.tv_nsec) / 1e9;
	double mib = (double)NREC * N * RECSZ / 1048576;
	printf("dcwrite: %d threads x %ld records x %ld bytes = %.1f MiB in %.2fs (%.1f MiB/s) flags=0x%x%s; to stable %.2fs (%.1f MiB/s)\n",
	       N, NREC, RECSZ, mib, sec, mib / sec, flags, odirect ? " O_DIRECT" : "", ssec, mib / ssec);
	if (frc) perror("fsync");
	if (close(fd)) { perror("close"); frc = -1; }
	return (failed || frc) ? 1 : 0;
}

#!/bin/bash
# SPDX-License-Identifier: GPL-2.0
#
# dc-ab.sh OUTDIR [DCWRITE] - RWF_DONTCACHE shared-page test, no NFS.
#
# For each record size and flag set: a fresh XFS on an emulated 4Kn NVMe
# namespace (the in-kernel NVMe target's file backend on a tmpfs image,
# attached over nvme-loop, so the device has NVMe's queue limits and a
# 4096-byte logical block), then NT dcwrite threads writing TOTAL bytes as
# interleaved records.  It reports the device's 4 KiB reads per write, from
# /sys/block/<dev>/stat around the write and a final sync, the file's
# resident pages after the sync (fincore), and the write rate.
#
# Needs root, the nvmet and nvme-loop modules, nvme-cli, xfsprogs, util-linux
# (fincore), and SIZE (default 4G) free in /dev/shm.  Build dcwrite first:
#   cc -O2 -pthread -o dcwrite dcwrite.c
#
# Environment: NT (32), TOTAL (1 GiB), RSS ("49152 47008 6000"),
# FLAGSETS ("none dontcache dsync dontcache,dsync"), SIZE (4G),
# IMG (/dev/shm/dc-ab.img), MNT (/mnt/dc-ab), PORT (nvmet port id, 9).
set -u
OUT=${1:?usage: dc-ab.sh OUTDIR [DCWRITE]}; DCW=$(realpath "${2:-./dcwrite}")
NT=${NT:-32}; TOTAL=${TOTAL:-1073741824}; SIZE=${SIZE:-4G}
RSS=${RSS:-"49152 47008 6000"}
FLAGSETS=${FLAGSETS:-"none dontcache dsync dontcache,dsync"}
IMG=${IMG:-/dev/shm/dc-ab.img}; MNT=${MNT:-/mnt/dc-ab}; PORT=${PORT:-9}
NQN=dc-ab-test; CFS=/sys/kernel/config/nvmet
SUB=$CFS/subsystems/$NQN; PRT=$CFS/ports/$PORT

die() { echo "dc-ab.sh: $*" >&2; down; exit 2; }
nsdev() {
	local d
	for d in /sys/block/nvme*n*; do
		# not a multipath path node (nvmeXcYnZ), which has no /dev entry
		[[ $(basename "$d") =~ ^nvme[0-9]+n[0-9]+$ ]] || continue
		[ "$(cat "$d/device/subsysnqn" 2>/dev/null)" = $NQN ] &&
			{ basename "$d"; return; }
	done
}
up() {
	truncate -s "$SIZE" "$IMG" || die "cannot create $IMG"
	mkdir "$SUB" || die "cannot create nvmet subsystem $NQN"
	echo 1 > "$SUB/attr_allow_any_host"
	mkdir "$SUB/namespaces/1"
	echo -n "$IMG" > "$SUB/namespaces/1/device_path"
	# the image is in RAM already: no O_DIRECT below the emulated device
	echo 1 > "$SUB/namespaces/1/buffered_io"
	echo 1 > "$SUB/namespaces/1/enable" || die "cannot enable the namespace"
	mkdir "$PRT" || die "nvmet port $PORT is in use: set PORT"
	echo -n loop > "$PRT/addr_trtype"
	# nvme-loop sends a connect without an address to the first loop port,
	# which need not be this one
	echo -n $NQN > "$PRT/addr_traddr"
	ln -s "$SUB" "$PRT/subsystems/$NQN"
	nvme connect -t loop -a $NQN -n $NQN > /dev/null || die "nvme connect failed"
	local i
	for i in $(seq 20); do DEV=$(nsdev); [ -n "$DEV" ] && break; sleep 0.5; done
	[ -n "$DEV" ] || die "the namespace did not appear"
	udevadm settle 2> /dev/null
	mkfs.xfs -q -f "/dev/$DEV" || die "mkfs.xfs failed"
	mkdir -p "$MNT"
	mount -t xfs "/dev/$DEV" "$MNT" || die "mount failed"
}
down() {
	umount "$MNT" 2> /dev/null
	nvme disconnect -n $NQN > /dev/null 2>&1
	rm -f "$PRT/subsystems/$NQN" 2> /dev/null
	rmdir "$PRT" 2> /dev/null
	[ -d "$SUB" ] && echo 0 > "$SUB/namespaces/1/enable" 2> /dev/null
	rmdir "$SUB/namespaces/1" "$SUB" 2> /dev/null
	rm -f "$IMG"
}

[ "$(id -u)" = 0 ] || { echo "dc-ab.sh: run as root" >&2; exit 2; }
[ -x "$DCW" ] || { echo "dc-ab.sh: build dcwrite first: cc -O2 -pthread -o dcwrite dcwrite.c" >&2; exit 2; }
modprobe nvmet && modprobe nvme-loop || exit 2
[ -d "$SUB" ] && { echo "dc-ab.sh: $SUB exists: a run is in progress or was left behind" >&2; exit 2; }
mkdir -p "$OUT"
echo "kernel $(uname -r)" > "$OUT/env.txt"
printf '%-6s %-16s | %-9s %-10s %-11s %-9s | %-15s | %s\n' \
	rs flags writes 4K-reads reads/write READ-MiB resident-pages MiB/s > "$OUT/table.txt"
for rs in $RSS; do for fl in $FLAGSETS; do
	d=$OUT/$rs-${fl//,/+}; mkdir -p "$d"
	DEV=; up
	F=$MNT/dc.bin
	sync; echo 3 > /proc/sys/vm/drop_caches
	read -r _ _ s0 _ < "/sys/block/$DEV/stat"
	"$DCW" "$F" "$NT" "$rs" "$TOTAL" "$fl" > "$d/dcwrite.txt" 2>&1; rc=$?
	sync
	read -r _ _ s1 _ < "/sys/block/$DEV/stat"
	res=$(fincore -b -n -r -o RES "$F"); pages=$(( ($(stat -c %s "$F") + 4095) / 4096 ))
	nrec=$(( TOTAL / rs / NT * NT ))
	mbs=$(grep -o '([0-9.]* MiB/s' "$d/dcwrite.txt" | head -1 | tr -d '(' | awk '{print $1}')
	# data reads in 4 KiB blocks, from sectors read (a read request that
	# moves no sectors, as a flush can show up, adds nothing here)
	rd=$(( (s1 - s0) * 512 / 4096 ))
	rmib=$(awk -v s=$((s1 - s0)) 'BEGIN { printf "%.1f", s * 512 / 1048576 }')
	rpw=$(awk -v r=$rd -v w=$nrec 'BEGIN { printf "%.4f", r / w }')
	[ $rc -ne 0 ] && mbs="rc=$rc"
	printf '%-6s %-16s | %-9s %-10s %-11s %-9s | %-15s | %s\n' $rs $fl $nrec $rd \
		$rpw $rmib "$(( (res + 4095) / 4096 ))/$pages" "${mbs:-?}" >> "$OUT/table.txt"
	down
done; done
cat "$OUT/table.txt"

Re: [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats

[Mike Snitzer]

On Wed, Oct 21, 2026 at 06:50:47AM -0400, Jeff Layton wrote:
> On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> > The per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the
> > flusher writes back in response to an IOCB_DONTCACHE kick, but it is
> > not reported anywhere, so the DONTCACHE dirty backlog can only be
> > inferred from MemFree/Cached movement.
> > 
> > Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats
> > and as WbDontcacheDirty in the per-wb stats, next to the
> > Reclaimable line.  Use wb_stat_sum() rather than wb_stat() to fold in
> > the per-cpu batches: the flusher targets the summed value and small
> > backlogs would otherwise read as zero.
> > 
> > Fixes: 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")
> > Cc: Jeff Layton <jlayton@kernel.org>
> > Cc: Jens Axboe <axboe@kernel.dk>
> > Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
> > Cc: linux-mm@kvack.org
> > Assisted-by: LLM
> > Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> > ---
> >  mm/backing-dev.c | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> > 
> > diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> > index 18e999053bae..b8a2c27e07fa 100644
> > --- a/mm/backing-dev.c
> > +++ b/mm/backing-dev.c
> > @@ -46,6 +46,7 @@ struct wb_stats {
> >  	unsigned long nr_dirty_time;
> >  	unsigned long nr_writeback;
> >  	unsigned long nr_reclaimable;
> > +	unsigned long nr_dontcache_dirty;
> >  	unsigned long nr_dirtied;
> >  	unsigned long nr_written;
> >  	unsigned long dirty_thresh;
> > @@ -78,6 +79,12 @@ static void collect_wb_stats(struct wb_stats *stats,
> >  
> >  	stats->nr_writeback += wb_stat(wb, WB_WRITEBACK);
> >  	stats->nr_reclaimable += wb_stat(wb, WB_RECLAIMABLE);
> > +	/*
> > +	 * Sum the per-cpu batches: the dontcache flusher targets this
> > +	 * counter via wb_stat_sum() and small backlogs would otherwise
> > +	 * read as zero here.
> > +	 */
> 
> nit: I don't personally find this comment very helpful.

Meant to address "why not just use wb_stat()?". But will drop it (and
leave it to commit header to explain).

> > +	stats->nr_dontcache_dirty += wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
> >  	stats->nr_dirtied += wb_stat(wb, WB_DIRTIED);
> >  	stats->nr_written += wb_stat(wb, WB_WRITTEN);
> >  	stats->wb_thresh += wb_calc_thresh(wb, stats->dirty_thresh);
> > @@ -125,6 +132,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
> >  	seq_printf(m,
> >  		   "BdiWriteback:       %10lu kB\n"
> >  		   "BdiReclaimable:     %10lu kB\n"
> > +		   "BdiDontcacheDirty:  %10lu kB\n"
> >  		   "BdiDirtyThresh:     %10lu kB\n"
> >  		   "DirtyThresh:        %10lu kB\n"
> >  		   "BackgroundThresh:   %10lu kB\n"
> > @@ -139,6 +147,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
> >  		   "state:              %10lx\n",
> >  		   K(stats.nr_writeback),
> >  		   K(stats.nr_reclaimable),
> > +		   K(stats.nr_dontcache_dirty),
> >  		   K(stats.wb_thresh),
> >  		   K(dirty_thresh),
> >  		   K(background_thresh),
> > @@ -163,6 +172,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
> >  		   "WbCgIno:           %10lu\n"
> >  		   "WbWriteback:       %10lu kB\n"
> >  		   "WbReclaimable:     %10lu kB\n"
> > +		   "WbDontcacheDirty:  %10lu kB\n"
> >  		   "WbDirtyThresh:     %10lu kB\n"
> >  		   "WbDirtied:         %10lu kB\n"
> >  		   "WbWritten:         %10lu kB\n"
> > @@ -179,6 +189,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
> >  #endif
> >  		   K(stats->nr_writeback),
> >  		   K(stats->nr_reclaimable),
> > +		   K(stats->nr_dontcache_dirty),
> >  		   K(stats->wb_thresh),
> >  		   K(stats->nr_dirtied),
> >  		   K(stats->nr_written),
> 
> Patch looks fine though:
> 
> Reviewed-by: Jeff Layton <jlayton@kernel.org>

Thanks! 

Re: [PATCH v7 2/5] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it

[Mike Snitzer]

On Wed, Oct 21, 2026 at 06:52:59AM -0400, Jeff Layton wrote:
> 
> Chuck's sashiko had a comment on this one:
> 
> [Severity: Medium]
> Could this overcount WB_DONTCACHE_DIRTY if folio_mark_dropbehind() races with
> filemap_dirty_folio() in mm/page-writeback.c? The dirtying path can set
> PG_dirty before folio_account_dirtied() checks PG_dropbehind, and it need not
> hold the folio lock. If folio_mark_dropbehind() sets PG_dropbehind after that
> dirty transition but before the accounting check, both paths increment the
> counter, while folio_account_cleaned() subtracts it only once. The later NFSD
> use of this helper retains the same interleaving. Could the flag update and
> dirty accounting be serialized so the counter is incremented only once?

Yes, it can double count.  Every path that takes a folio off
WB_DONTCACHE_DIRTY holds the folio lock, as folio_mark_dropbehind()'s
callers do, but folio_account_dirtied() adds under the i_pages lock
after filemap_dirty_folio() has set PG_dirty without it.  So
PG_dirty is the wrong test: it is set before the folio is counted.
Reaching it needs a dirtier that doesn't hold the folio lock, such as a
zap of a dirty pte in a shared file mapping or bio_set_pages_dirty()
after a direct read into one; the cost is a counter the DONTCACHE kick
reads too high, not data.

v8 will test what folio_account_dirtied() leaves behind instead: the
dirty tag, which __folio_mark_dirty() sets in the same i_pages section
that counts the folio.  With the folio locked nothing can clean it, so
the tag is set exactly when the folio has been counted, and taking the
i_pages lock around setting the flag and testing the tag orders it
against that section:

        xa_lock_irqsave(&mapping->i_pages, flags);
        folio_set_dropbehind(folio);
        if (xa_get_mark(&mapping->i_pages, folio->index, PAGECACHE_TAG_DIRTY))
                wb_stat_mod(inode_to_wb(mapping->host), WB_DONTCACHE_DIRTY,
                            folio_nr_pages(folio));
        xa_unlock_irqrestore(&mapping->i_pages, flags);

A dirtier past folio_test_set_dirty() but not yet in that section finds
the flag set and counts the folio itself.

The same interleaving undercounts in the clear path that 88d6f128d06d
("mm: track DONTCACHE dirty pages per bdi_writeback") added to
__filemap_get_folio_mpol(): it clears the flag and subtracts if
PG_dirty is set, which a dirtier that has set PG_dirty but not yet
counted the folio makes it do for a folio that was never added.  The
same change, clearing instead of setting, fixes that; I will send it
as its own fix for 88d6f128d06d, at the start of this series for v8.

Thanks,
Mike

Re: [PATCH v7 4/5] mm, iomap: let an uncached writer complete its shared pages itself

[Mike Snitzer]

On Wed, Oct 21, 2026 at 06:54:14AM -0400, Jeff Layton wrote:
> 
> 2 comments from Chuck's sashiko on this one:
> 
> [Severity: Low]
> Could filemap_dontcache_edge_complete() in mm/filemap.c avoid missing the
> immediate drop when another task acquires the folio lock between
> folio_unlock() and folio_end_dropbehind()? In that case folio_end_dropbehind()
> fails its folio_trylock() and does not retry; if the folio is already clean,
> there may be no later writeback completion to drop it, so it can remain cached
> until unrelated activity or reclaim.

Yes.  There was no reason to let go of the lock first: the folio is
locked and we are in task context, so v8 drops it with
filemap_end_dropbehind() before the unlock, as folio_end_dropbehind()
does once its trylock succeeds.

> [Severity: Medium]
> Could filemap_dontcache_edge_complete() in mm/filemap.c restrict marking to
> the boundary page when the mapping returns a large folio? iomap_get_folio()
> in fs/iomap/buffered-io.c limits a partial edge request to one page, but the
> mapping's minimum folio order can still cause __filemap_get_folio_mpol() to
> return a multi-page folio, or the lookup can find an existing large folio.
> This helper then marks the entire folio for drop-behind. For example, if a
> caller uses deferred completion on an XFS mapping with a minimum folio order,
> marking one edge can account the whole dirty folio for DONTCACHE writeback or
> invalidate the whole clean, unmapped folio. Could this path handle only the
> page covered by the completed edge?

Also right, and it applied to 3/5 as well: the folio a write finds
uptodate there was marked whole in the same way.  3/5 only creates the
edge as a single page; a lookup can find a larger folio that readahead
or an earlier buffered write left, and dropping it drops everything in
it, including other edges whose second write hasn't come yet, which
then pay the read-back this series is meant to remove.  A folio can't
be dropped in part, so v8 marks a found folio only when it is the
smallest the mapping allows, folio_nr_pages(folio) ==
mapping_min_folio_nrpages(mapping): a single page, or a single block
where the block is larger than a page, which is then the unit two
writes share.  A larger folio is left unmarked, cached until reclaim
as it is without the series.  That is in 3/5's lookup and in both 4/5
helpers.

Looking at that turned up two more places that used the page where the
smallest folio is the unit, both fixed in v8:

- 3/5's iomap change decided that a write covers its first or last page
  only partly by page offset, so on a mapping whose blocks are larger
  than a page a write starting on a page boundary inside a block didn't
  ask for its block as an edge.  It now tests against
  mapping_min_folio_nrbytes().
- NFSD cuts a misaligned end of a direct-mode WRITE on a page boundary
  (80728157d1bd, "NFSD: cut a misaligned direct-mode WRITE at page
  boundaries", in Chuck's nfsd-testing), so there the direct middle and
  a buffered end could share a block.  v8 has a fix for that at the
  front, so the end is cut at the smallest folio as well.

Measured on XFS on an emulated 4Kn NVMe, 4 KiB pages, 32 threads writing
interleaved 47008-byte records, 1 GiB, 4 KiB reads per write from the
device, v7 against v8:

                                                v7        v8
  RWF_DONTCACHE over large folios readahead
    left (pages resident after)          0.94 (6%)  0.0026 (100%)
  RWF_DONTCACHE, 16 KiB blocks              1.62    0.0014
    (12288-byte records)                    2.89    0.0004
  NFS WRITEs through NFSD DIRECT,
    16 KiB blocks                           3.46    0.0016

(v7 in 128 MiB runs, v8 in 1 GiB runs; 16384-byte records, which cover
only whole blocks, read 0 either way.  On 16 KiB blocks one block read
back per write is 4.0 here.)

Thanks,
Mike