* [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats
2026-10-20 22:03 [PATCH v7 0/5] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
@ 2026-10-20 22:03 ` Mike Snitzer
2026-10-20 22:11 ` Mike Snitzer
2026-10-21 10:50 ` Jeff Layton
2026-10-20 22:03 ` [PATCH v7 2/5] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it Mike Snitzer
` (3 subsequent siblings)
4 siblings, 2 replies; 11+ messages in thread
From: Mike Snitzer @ 2026-10-20 22:03 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, Jonathan Corbet,
linux-mm, linux-fsdevel
The per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the
flusher writes back in response to an IOCB_DONTCACHE kick, but it is
not reported anywhere, so the DONTCACHE dirty backlog can only be
inferred from MemFree/Cached movement.
Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats
and as WbDontcacheDirty in the per-wb stats, next to the
Reclaimable line. Use wb_stat_sum() rather than wb_stat() to fold in
the per-cpu batches: the flusher targets the summed value and small
backlogs would otherwise read as zero.
Fixes: 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: linux-mm@kvack.org
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
mm/backing-dev.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 18e999053bae..b8a2c27e07fa 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -46,6 +46,7 @@ struct wb_stats {
unsigned long nr_dirty_time;
unsigned long nr_writeback;
unsigned long nr_reclaimable;
+ unsigned long nr_dontcache_dirty;
unsigned long nr_dirtied;
unsigned long nr_written;
unsigned long dirty_thresh;
@@ -78,6 +79,12 @@ static void collect_wb_stats(struct wb_stats *stats,
stats->nr_writeback += wb_stat(wb, WB_WRITEBACK);
stats->nr_reclaimable += wb_stat(wb, WB_RECLAIMABLE);
+ /*
+ * Sum the per-cpu batches: the dontcache flusher targets this
+ * counter via wb_stat_sum() and small backlogs would otherwise
+ * read as zero here.
+ */
+ stats->nr_dontcache_dirty += wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
stats->nr_dirtied += wb_stat(wb, WB_DIRTIED);
stats->nr_written += wb_stat(wb, WB_WRITTEN);
stats->wb_thresh += wb_calc_thresh(wb, stats->dirty_thresh);
@@ -125,6 +132,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
seq_printf(m,
"BdiWriteback: %10lu kB\n"
"BdiReclaimable: %10lu kB\n"
+ "BdiDontcacheDirty: %10lu kB\n"
"BdiDirtyThresh: %10lu kB\n"
"DirtyThresh: %10lu kB\n"
"BackgroundThresh: %10lu kB\n"
@@ -139,6 +147,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
"state: %10lx\n",
K(stats.nr_writeback),
K(stats.nr_reclaimable),
+ K(stats.nr_dontcache_dirty),
K(stats.wb_thresh),
K(dirty_thresh),
K(background_thresh),
@@ -163,6 +172,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
"WbCgIno: %10lu\n"
"WbWriteback: %10lu kB\n"
"WbReclaimable: %10lu kB\n"
+ "WbDontcacheDirty: %10lu kB\n"
"WbDirtyThresh: %10lu kB\n"
"WbDirtied: %10lu kB\n"
"WbWritten: %10lu kB\n"
@@ -179,6 +189,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
#endif
K(stats->nr_writeback),
K(stats->nr_reclaimable),
+ K(stats->nr_dontcache_dirty),
K(stats->wb_thresh),
K(stats->nr_dirtied),
K(stats->nr_written),
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats
2026-10-20 22:03 ` [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats Mike Snitzer
@ 2026-10-20 22:11 ` Mike Snitzer
2026-10-21 10:50 ` Jeff Layton
1 sibling, 0 replies; 11+ messages in thread
From: Mike Snitzer @ 2026-10-20 22:11 UTC (permalink / raw)
To: Mike Snitzer
Cc: Chuck Lever, Jeff Layton, linux-nfs, NeilBrown, Olga Kornievskaia,
Dai Ngo, Tom Talpey, Christoph Hellwig, Jens Axboe,
Matthew Wilcox, Jonathan Corbet, linux-mm, linux-fsdevel
On Tue, Oct 20, 2026 at 06:03:42PM -0400, Mike Snitzer wrote:
> The per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the
> flusher writes back in response to an IOCB_DONTCACHE kick, but it is
> not reported anywhere, so the DONTCACHE dirty backlog can only be
> inferred from MemFree/Cached movement.
>
> Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats
> and as WbDontcacheDirty in the per-wb stats, next to the
> Reclaimable line. Use wb_stat_sum() rather than wb_stat() to fold in
> the per-cpu batches: the flusher targets the summed value and small
> backlogs would otherwise read as zero.
>
> Fixes: 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")
> Cc: Jeff Layton <jlayton@kernel.org>
FYI, I forgot to pick up Jeff's provided review tag for this patch:
Reviewed-by: Jeff Layton <jlayton@kernel.org>
see: https://lore.kernel.org/linux-nfs/2bee53783a2372b8dc715350b6f6d2ad5d8770ff.camel@kernel.org/
Thanks,
Mike
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
> Cc: linux-mm@kvack.org
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
> mm/backing-dev.c | 11 +++++++++++
> 1 file changed, 11 insertions(+)
>
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index 18e999053bae..b8a2c27e07fa 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -46,6 +46,7 @@ struct wb_stats {
> unsigned long nr_dirty_time;
> unsigned long nr_writeback;
> unsigned long nr_reclaimable;
> + unsigned long nr_dontcache_dirty;
> unsigned long nr_dirtied;
> unsigned long nr_written;
> unsigned long dirty_thresh;
> @@ -78,6 +79,12 @@ static void collect_wb_stats(struct wb_stats *stats,
>
> stats->nr_writeback += wb_stat(wb, WB_WRITEBACK);
> stats->nr_reclaimable += wb_stat(wb, WB_RECLAIMABLE);
> + /*
> + * Sum the per-cpu batches: the dontcache flusher targets this
> + * counter via wb_stat_sum() and small backlogs would otherwise
> + * read as zero here.
> + */
> + stats->nr_dontcache_dirty += wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
> stats->nr_dirtied += wb_stat(wb, WB_DIRTIED);
> stats->nr_written += wb_stat(wb, WB_WRITTEN);
> stats->wb_thresh += wb_calc_thresh(wb, stats->dirty_thresh);
> @@ -125,6 +132,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
> seq_printf(m,
> "BdiWriteback: %10lu kB\n"
> "BdiReclaimable: %10lu kB\n"
> + "BdiDontcacheDirty: %10lu kB\n"
> "BdiDirtyThresh: %10lu kB\n"
> "DirtyThresh: %10lu kB\n"
> "BackgroundThresh: %10lu kB\n"
> @@ -139,6 +147,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
> "state: %10lx\n",
> K(stats.nr_writeback),
> K(stats.nr_reclaimable),
> + K(stats.nr_dontcache_dirty),
> K(stats.wb_thresh),
> K(dirty_thresh),
> K(background_thresh),
> @@ -163,6 +172,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
> "WbCgIno: %10lu\n"
> "WbWriteback: %10lu kB\n"
> "WbReclaimable: %10lu kB\n"
> + "WbDontcacheDirty: %10lu kB\n"
> "WbDirtyThresh: %10lu kB\n"
> "WbDirtied: %10lu kB\n"
> "WbWritten: %10lu kB\n"
> @@ -179,6 +189,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
> #endif
> K(stats->nr_writeback),
> K(stats->nr_reclaimable),
> + K(stats->nr_dontcache_dirty),
> K(stats->wb_thresh),
> K(stats->nr_dirtied),
> K(stats->nr_written),
> --
> 2.52.0
>
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats
2026-10-20 22:03 ` [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats Mike Snitzer
2026-10-20 22:11 ` Mike Snitzer
@ 2026-10-21 10:50 ` Jeff Layton
1 sibling, 0 replies; 11+ messages in thread
From: Jeff Layton @ 2026-10-21 10:50 UTC (permalink / raw)
To: Mike Snitzer, Chuck Lever
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, Jonathan Corbet,
linux-mm, linux-fsdevel
On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> The per-bdi_writeback WB_DONTCACHE_DIRTY counter drives how much the
> flusher writes back in response to an IOCB_DONTCACHE kick, but it is
> not reported anywhere, so the DONTCACHE dirty backlog can only be
> inferred from MemFree/Cached movement.
>
> Report it as BdiDontcacheDirty in /sys/kernel/debug/bdi/<dev>/stats
> and as WbDontcacheDirty in the per-wb stats, next to the
> Reclaimable line. Use wb_stat_sum() rather than wb_stat() to fold in
> the per-cpu batches: the flusher targets the summed value and small
> backlogs would otherwise read as zero.
>
> Fixes: 88d6f128d06d ("mm: track DONTCACHE dirty pages per bdi_writeback")
> Cc: Jeff Layton <jlayton@kernel.org>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
> Cc: linux-mm@kvack.org
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
> mm/backing-dev.c | 11 +++++++++++
> 1 file changed, 11 insertions(+)
>
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index 18e999053bae..b8a2c27e07fa 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -46,6 +46,7 @@ struct wb_stats {
> unsigned long nr_dirty_time;
> unsigned long nr_writeback;
> unsigned long nr_reclaimable;
> + unsigned long nr_dontcache_dirty;
> unsigned long nr_dirtied;
> unsigned long nr_written;
> unsigned long dirty_thresh;
> @@ -78,6 +79,12 @@ static void collect_wb_stats(struct wb_stats *stats,
>
> stats->nr_writeback += wb_stat(wb, WB_WRITEBACK);
> stats->nr_reclaimable += wb_stat(wb, WB_RECLAIMABLE);
> + /*
> + * Sum the per-cpu batches: the dontcache flusher targets this
> + * counter via wb_stat_sum() and small backlogs would otherwise
> + * read as zero here.
> + */
nit: I don't personally find this comment very helpful.
> + stats->nr_dontcache_dirty += wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
> stats->nr_dirtied += wb_stat(wb, WB_DIRTIED);
> stats->nr_written += wb_stat(wb, WB_WRITTEN);
> stats->wb_thresh += wb_calc_thresh(wb, stats->dirty_thresh);
> @@ -125,6 +132,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
> seq_printf(m,
> "BdiWriteback: %10lu kB\n"
> "BdiReclaimable: %10lu kB\n"
> + "BdiDontcacheDirty: %10lu kB\n"
> "BdiDirtyThresh: %10lu kB\n"
> "DirtyThresh: %10lu kB\n"
> "BackgroundThresh: %10lu kB\n"
> @@ -139,6 +147,7 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
> "state: %10lx\n",
> K(stats.nr_writeback),
> K(stats.nr_reclaimable),
> + K(stats.nr_dontcache_dirty),
> K(stats.wb_thresh),
> K(dirty_thresh),
> K(background_thresh),
> @@ -163,6 +172,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
> "WbCgIno: %10lu\n"
> "WbWriteback: %10lu kB\n"
> "WbReclaimable: %10lu kB\n"
> + "WbDontcacheDirty: %10lu kB\n"
> "WbDirtyThresh: %10lu kB\n"
> "WbDirtied: %10lu kB\n"
> "WbWritten: %10lu kB\n"
> @@ -179,6 +189,7 @@ static void wb_stats_show(struct seq_file *m, struct bdi_writeback *wb,
> #endif
> K(stats->nr_writeback),
> K(stats->nr_reclaimable),
> + K(stats->nr_dontcache_dirty),
> K(stats->wb_thresh),
> K(stats->nr_dirtied),
> K(stats->nr_written),
Patch looks fine though:
Reviewed-by: Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v7 2/5] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it
2026-10-20 22:03 [PATCH v7 0/5] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
2026-10-20 22:03 ` [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats Mike Snitzer
@ 2026-10-20 22:03 ` Mike Snitzer
2026-10-21 10:52 ` Jeff Layton
2026-10-20 22:03 ` [PATCH v7 3/5] mm, iomap: keep a partly covered page until an uncached write completes it Mike Snitzer
` (2 subsequent siblings)
4 siblings, 1 reply; 11+ messages in thread
From: Mike Snitzer @ 2026-10-20 22:03 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, Jonathan Corbet,
linux-mm, linux-fsdevel
A folio allocated by an RWF_DONTCACHE write is marked dropbehind before
it is dirtied, so folio_account_dirtied() adds it to WB_DONTCACHE_DIRTY
and folio_clear_dirty_for_io() or folio_account_cleaned() subtracts it
when it is cleaned. Nothing marks a folio that already exists. A
caller that sets PG_dropbehind on a folio that is already dirty
unbalances the counter: the clean subtracts what was never added, and
once the counter goes negative, wb_stat_sum() reads it as zero and the
DONTCACHE writeback kick stops writing anything back for every writer
on that bdi_writeback.
Add folio_mark_dropbehind(), the inverse of the clear in
__filemap_get_folio_mpol(): with the folio locked, mark it and, if it is
dirty, add it to WB_DONTCACHE_DIRTY. The folio lock orders this
against dirtying and against folio_clear_dirty_for_io(). A concurrent
lookup without FGP_DONTCACHE that clears the flag again subtracts what
this added.
The name follows folio_set_dirty() and folio_mark_dirty(): the first
only sets PG_dirty, the second also accounts the dirty folio.
folio_set_dropbehind() likewise only sets PG_dropbehind, and
folio_mark_dropbehind() also accounts it in WB_DONTCACHE_DIRTY.
A later commit uses it in NFSD to drop a page shared by two misaligned
WRITEs once both have written it.
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: linux-mm@kvack.org
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
include/linux/pagemap.h | 1 +
mm/filemap.c | 31 +++++++++++++++++++++++++++++++
2 files changed, 32 insertions(+)
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0adfa6605653..3012bd95812a 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1310,6 +1310,7 @@ void end_page_writeback(struct page *page);
void folio_end_writeback(struct folio *folio);
void folio_end_writeback_no_dropbehind(struct folio *folio);
void folio_end_dropbehind(struct folio *folio);
+void folio_mark_dropbehind(struct folio *folio);
void folio_wait_stable(struct folio *folio);
void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
diff --git a/mm/filemap.c b/mm/filemap.c
index 00fd89cf6f55..ed7e4dbea653 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1644,6 +1644,37 @@ void folio_end_dropbehind(struct folio *folio)
}
EXPORT_SYMBOL_GPL(folio_end_dropbehind);
+/**
+ * folio_mark_dropbehind - mark an existing folio dropbehind
+ * @folio: the folio, locked by the caller
+ *
+ * Mark @folio so that the writeback that cleans it drops it. A folio that
+ * is already dirty is added to WB_DONTCACHE_DIRTY, as folio_account_dirtied()
+ * would have added it had it been marked when dirtied, so that the clean
+ * that subtracts it leaves the counter balanced.
+ */
+void folio_mark_dropbehind(struct folio *folio)
+{
+ struct address_space *mapping = folio->mapping;
+
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+
+ if (folio_test_dropbehind(folio))
+ return;
+ folio_set_dropbehind(folio);
+ if (mapping && folio_test_dirty(folio) &&
+ mapping_can_writeback(mapping)) {
+ struct inode *inode = mapping->host;
+ struct bdi_writeback *wb;
+ struct wb_lock_cookie cookie = {};
+
+ wb = unlocked_inode_to_wb_begin(inode, &cookie);
+ wb_stat_mod(wb, WB_DONTCACHE_DIRTY, folio_nr_pages(folio));
+ unlocked_inode_to_wb_end(inode, &cookie);
+ }
+}
+EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
+
/**
* folio_end_writeback_no_dropbehind - End writeback against a folio.
* @folio: The folio.
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH v7 2/5] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it
2026-10-20 22:03 ` [PATCH v7 2/5] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it Mike Snitzer
@ 2026-10-21 10:52 ` Jeff Layton
0 siblings, 0 replies; 11+ messages in thread
From: Jeff Layton @ 2026-10-21 10:52 UTC (permalink / raw)
To: Mike Snitzer, Chuck Lever
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, Jonathan Corbet,
linux-mm, linux-fsdevel
On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> A folio allocated by an RWF_DONTCACHE write is marked dropbehind before
> it is dirtied, so folio_account_dirtied() adds it to WB_DONTCACHE_DIRTY
> and folio_clear_dirty_for_io() or folio_account_cleaned() subtracts it
> when it is cleaned. Nothing marks a folio that already exists. A
> caller that sets PG_dropbehind on a folio that is already dirty
> unbalances the counter: the clean subtracts what was never added, and
> once the counter goes negative, wb_stat_sum() reads it as zero and the
> DONTCACHE writeback kick stops writing anything back for every writer
> on that bdi_writeback.
>
> Add folio_mark_dropbehind(), the inverse of the clear in
> __filemap_get_folio_mpol(): with the folio locked, mark it and, if it is
> dirty, add it to WB_DONTCACHE_DIRTY. The folio lock orders this
> against dirtying and against folio_clear_dirty_for_io(). A concurrent
> lookup without FGP_DONTCACHE that clears the flag again subtracts what
> this added.
>
> The name follows folio_set_dirty() and folio_mark_dirty(): the first
> only sets PG_dirty, the second also accounts the dirty folio.
> folio_set_dropbehind() likewise only sets PG_dropbehind, and
> folio_mark_dropbehind() also accounts it in WB_DONTCACHE_DIRTY.
>
> A later commit uses it in NFSD to drop a page shared by two misaligned
> WRITEs once both have written it.
>
> Cc: Jeff Layton <jlayton@kernel.org>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
> Cc: linux-mm@kvack.org
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
> include/linux/pagemap.h | 1 +
> mm/filemap.c | 31 +++++++++++++++++++++++++++++++
> 2 files changed, 32 insertions(+)
>
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index 0adfa6605653..3012bd95812a 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -1310,6 +1310,7 @@ void end_page_writeback(struct page *page);
> void folio_end_writeback(struct folio *folio);
> void folio_end_writeback_no_dropbehind(struct folio *folio);
> void folio_end_dropbehind(struct folio *folio);
> +void folio_mark_dropbehind(struct folio *folio);
> void folio_wait_stable(struct folio *folio);
> void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
> void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
> diff --git a/mm/filemap.c b/mm/filemap.c
> index 00fd89cf6f55..ed7e4dbea653 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -1644,6 +1644,37 @@ void folio_end_dropbehind(struct folio *folio)
> }
> EXPORT_SYMBOL_GPL(folio_end_dropbehind);
>
> +/**
> + * folio_mark_dropbehind - mark an existing folio dropbehind
> + * @folio: the folio, locked by the caller
> + *
> + * Mark @folio so that the writeback that cleans it drops it. A folio that
> + * is already dirty is added to WB_DONTCACHE_DIRTY, as folio_account_dirtied()
> + * would have added it had it been marked when dirtied, so that the clean
> + * that subtracts it leaves the counter balanced.
> + */
> +void folio_mark_dropbehind(struct folio *folio)
> +{
> + struct address_space *mapping = folio->mapping;
> +
> + VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
> +
> + if (folio_test_dropbehind(folio))
> + return;
> + folio_set_dropbehind(folio);
> + if (mapping && folio_test_dirty(folio) &&
> + mapping_can_writeback(mapping)) {
> + struct inode *inode = mapping->host;
> + struct bdi_writeback *wb;
> + struct wb_lock_cookie cookie = {};
> +
> + wb = unlocked_inode_to_wb_begin(inode, &cookie);
> + wb_stat_mod(wb, WB_DONTCACHE_DIRTY, folio_nr_pages(folio));
> + unlocked_inode_to_wb_end(inode, &cookie);
> + }
> +}
> +EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
> +
> /**
> * folio_end_writeback_no_dropbehind - End writeback against a folio.
> * @folio: The folio.
Chuck's sashiko had a comment on this one:
[Severity: Medium]
Could this overcount WB_DONTCACHE_DIRTY if folio_mark_dropbehind() races with
filemap_dirty_folio() in mm/page-writeback.c? The dirtying path can set
PG_dirty before folio_account_dirtied() checks PG_dropbehind, and it need not
hold the folio lock. If folio_mark_dropbehind() sets PG_dropbehind after that
dirty transition but before the accounting check, both paths increment the
counter, while folio_account_cleaned() subtracts it only once. The later NFSD
use of this helper retains the same interleaving. Could the flag update and
dirty accounting be serialized so the counter is incremented only once?
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v7 3/5] mm, iomap: keep a partly covered page until an uncached write completes it
2026-10-20 22:03 [PATCH v7 0/5] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
2026-10-20 22:03 ` [PATCH v7 1/5] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats Mike Snitzer
2026-10-20 22:03 ` [PATCH v7 2/5] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it Mike Snitzer
@ 2026-10-20 22:03 ` Mike Snitzer
2026-10-21 14:36 ` Jeff Layton
2026-10-20 22:03 ` [PATCH v7 4/5] mm, iomap: let an uncached writer complete its shared pages itself Mike Snitzer
2026-10-20 22:03 ` [PATCH v7 5/5] NFSD: drop a shared boundary page only once both WRITEs have written it Mike Snitzer
4 siblings, 1 reply; 11+ messages in thread
From: Mike Snitzer @ 2026-10-20 22:03 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, Jonathan Corbet,
linux-mm, linux-fsdevel
An uncached (RWF_DONTCACHE) buffered write marks drop-behind every folio
it creates, so writeback drops each one as soon as this write's data is
on disk. A page the write covers only partly is usually shared with the
neighbouring write, the one that ends or starts in it. Once the first
of the two has been written back, the page is gone, and the second has
to read the block back from disk before it can write its part: one read
per misaligned write.
Measured locally on XFS (emulated 4Kn NVMe, 4 KiB pages), 32 threads
writing interleaved records to one file with pwritev2(), 1 GiB per run,
before and after this patch:
record flags 4 KiB device MiB/s
bytes reads per write
before after before after
47008 RWF_DONTCACHE 0.97 0.000 831 2281
47008 RWF_DONTCACHE|RWF_DSYNC 0.99 0.000 601 1227
6000 RWF_DONTCACHE 0.97 0.000 121 488
6000 RWF_DONTCACHE|RWF_DSYNC 0.99 0.000 90 206
Before this patch every write waited for its read, and because XFS holds
its IOLOCK exclusively across a buffered write, the other 31 writers
waited too: both record sizes ran at about 20 thousand writes a second,
one device read per write; after it nothing is read back. With
RWF_DSYNC each write still waits for its own sync, so the gain is
smaller: the uncached write now runs as fast as the same write without
RWF_DONTCACHE, 1237 and 209 MiB/s against 1227 and 206 in the table.
Records of 49152 bytes, which cover only whole pages, read nothing
before or after.
NFSD DIRECT issues the misaligned ends of a WRITE as buffered uncached
segments and gains the same way. Without any handling of the shared
page, 32 interleaved writers of 47008-byte records read 45144 blocks
for 45664 WRITEs; with this patch, and no NFSD code for the shared page,
0.0001 to 0.0008 blocks per WRITE whether each record is sent as one
FILE_SYNC WRITE, as two one after the other, as two together, or as two
UNSTABLE. The page cache then marks the completed page while it is
still dirty, so each segment of a stable WRITE wakes the DONTCACHE
writeback kick (23 to 65 thousand rounds a run); the next two patches
let NFSD mark it after its sync instead.
Let the write path say which pages a write covers only partly
(FGP_DONTCACHE_EDGE, set by iomap when the write starts inside a page or
ends before its end) and get such a page as a single page. A folio
created for it is not marked, so writing back this write's part does not
drop it. A folio found uptodate is the neighbour's, already written:
this write completes it, so mark it, under the folio lock and before its
own data goes in, and the writeback that cleans both writes' data drops
it. The folio lock orders the two writes, so the second always finds
the first's data in place.
A page whose neighbour never writes, such as the last page of a file,
stays cached until reclaim. A partly covered page another task had
cached and that is uptodate is now dropped after the write's writeback,
where before an uncached write never dropped a page it did not create.
Only iomap file systems say which pages a write covers partly so far;
write_begin_get_folio() users such as ext4 and the NFS client do not.
With blocks smaller than a page, a write that crosses an extent boundary
inside a page asks for it as two partly covered halves, so the page,
though one write covers it whole, stays cached until reclaim.
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
fs/iomap/buffered-io.c | 12 +++++++++++-
include/linux/pagemap.h | 5 +++++
mm/filemap.c | 19 ++++++++++++++++++-
3 files changed, 34 insertions(+), 2 deletions(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 0a5ebfda90f1..780a0186045d 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -802,8 +802,18 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
if (iter->flags & IOMAP_NOWAIT)
fgp |= FGP_NOWAIT;
- if (iter->flags & IOMAP_DONTCACHE)
+ if (iter->flags & IOMAP_DONTCACHE) {
fgp |= FGP_DONTCACHE;
+ /*
+ * A page this write covers only partly is shared with a
+ * neighbouring write: get it as a single page, which is
+ * kept until the write that completes it.
+ */
+ if (offset_in_page(pos) || len < PAGE_SIZE) {
+ fgp |= FGP_DONTCACHE_EDGE;
+ len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
+ }
+ }
fgp |= fgf_set_order(len);
return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 3012bd95812a..0c1d6bd09aa0 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -701,6 +701,10 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
* * %FGP_NOWAIT - Don't block on the folio lock.
* * %FGP_STABLE - Wait for the folio to be stable (finished writeback)
* * %FGP_DONTCACHE - Uncached buffered IO
+ * * %FGP_DONTCACHE_EDGE - With %FGP_DONTCACHE: the write covers this page
+ * only partly, and a neighbouring write may complete it. A folio created
+ * for it is not marked drop-behind; a folio found uptodate is, as the
+ * write that completes it.
* * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
* implementation.
*/
@@ -715,6 +719,7 @@ typedef unsigned int __bitwise fgf_t;
#define FGP_FOR_MMAP ((__force fgf_t)0x00000040)
#define FGP_STABLE ((__force fgf_t)0x00000080)
#define FGP_DONTCACHE ((__force fgf_t)0x00000100)
+#define FGP_DONTCACHE_EDGE ((__force fgf_t)0x00000200)
#define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */
#define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
diff --git a/mm/filemap.c b/mm/filemap.c
index ed7e4dbea653..0397ec771547 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2016,6 +2016,16 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
if (fgp_flags & FGP_STABLE)
folio_wait_stable(folio);
+
+ /*
+ * An uncached write that covers this folio only partly, finding it
+ * uptodate, is the write that completes it: the neighbouring write
+ * that left it here has written its part. Mark it now, under the
+ * folio lock, so the writeback that cleans both writes' data drops it.
+ */
+ if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
+ (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
+ folio_mark_dropbehind(folio);
no_page:
if (!folio && (fgp_flags & FGP_CREAT)) {
unsigned int min_order = mapping_min_folio_order(mapping);
@@ -2053,7 +2063,14 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
/* Init accessed so avoid atomic mark_page_accessed later */
if (fgp_flags & FGP_ACCESSED)
__folio_set_referenced(folio);
- if (fgp_flags & FGP_DONTCACHE)
+ /*
+ * A folio the write covers only partly is left
+ * unmarked, so that writing back this write's part
+ * does not drop it before a neighbouring write has
+ * written the rest.
+ */
+ if ((fgp_flags & FGP_DONTCACHE) &&
+ !(fgp_flags & FGP_DONTCACHE_EDGE))
__folio_set_dropbehind(folio);
err = filemap_add_folio(mapping, folio, index, gfp);
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH v7 3/5] mm, iomap: keep a partly covered page until an uncached write completes it
2026-10-20 22:03 ` [PATCH v7 3/5] mm, iomap: keep a partly covered page until an uncached write completes it Mike Snitzer
@ 2026-10-21 14:36 ` Jeff Layton
0 siblings, 0 replies; 11+ messages in thread
From: Jeff Layton @ 2026-10-21 14:36 UTC (permalink / raw)
To: Mike Snitzer, Chuck Lever
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, Jonathan Corbet,
linux-mm, linux-fsdevel
On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> An uncached (RWF_DONTCACHE) buffered write marks drop-behind every folio
> it creates, so writeback drops each one as soon as this write's data is
> on disk. A page the write covers only partly is usually shared with the
> neighbouring write, the one that ends or starts in it. Once the first
> of the two has been written back, the page is gone, and the second has
> to read the block back from disk before it can write its part: one read
> per misaligned write.
>
> Measured locally on XFS (emulated 4Kn NVMe, 4 KiB pages), 32 threads
> writing interleaved records to one file with pwritev2(), 1 GiB per run,
> before and after this patch:
>
> record flags 4 KiB device MiB/s
> bytes reads per write
> before after before after
> 47008 RWF_DONTCACHE 0.97 0.000 831 2281
> 47008 RWF_DONTCACHE|RWF_DSYNC 0.99 0.000 601 1227
> 6000 RWF_DONTCACHE 0.97 0.000 121 488
> 6000 RWF_DONTCACHE|RWF_DSYNC 0.99 0.000 90 206
>
> Before this patch every write waited for its read, and because XFS holds
> its IOLOCK exclusively across a buffered write, the other 31 writers
> waited too: both record sizes ran at about 20 thousand writes a second,
> one device read per write; after it nothing is read back. With
> RWF_DSYNC each write still waits for its own sync, so the gain is
> smaller: the uncached write now runs as fast as the same write without
> RWF_DONTCACHE, 1237 and 209 MiB/s against 1227 and 206 in the table.
> Records of 49152 bytes, which cover only whole pages, read nothing
> before or after.
>
> NFSD DIRECT issues the misaligned ends of a WRITE as buffered uncached
> segments and gains the same way. Without any handling of the shared
> page, 32 interleaved writers of 47008-byte records read 45144 blocks
> for 45664 WRITEs; with this patch, and no NFSD code for the shared page,
> 0.0001 to 0.0008 blocks per WRITE whether each record is sent as one
> FILE_SYNC WRITE, as two one after the other, as two together, or as two
> UNSTABLE. The page cache then marks the completed page while it is
> still dirty, so each segment of a stable WRITE wakes the DONTCACHE
> writeback kick (23 to 65 thousand rounds a run); the next two patches
> let NFSD mark it after its sync instead.
>
> Let the write path say which pages a write covers only partly
> (FGP_DONTCACHE_EDGE, set by iomap when the write starts inside a page or
> ends before its end) and get such a page as a single page. A folio
> created for it is not marked, so writing back this write's part does not
> drop it. A folio found uptodate is the neighbour's, already written:
> this write completes it, so mark it, under the folio lock and before its
> own data goes in, and the writeback that cleans both writes' data drops
> it. The folio lock orders the two writes, so the second always finds
> the first's data in place.
>
> A page whose neighbour never writes, such as the last page of a file,
> stays cached until reclaim. A partly covered page another task had
> cached and that is uptodate is now dropped after the write's writeback,
> where before an uncached write never dropped a page it did not create.
>
> Only iomap file systems say which pages a write covers partly so far;
> write_begin_get_folio() users such as ext4 and the NFS client do not.
> With blocks smaller than a page, a write that crosses an extent boundary
> inside a page asks for it as two partly covered halves, so the page,
> though one write covers it whole, stays cached until reclaim.
>
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
> fs/iomap/buffered-io.c | 12 +++++++++++-
> include/linux/pagemap.h | 5 +++++
> mm/filemap.c | 19 ++++++++++++++++++-
> 3 files changed, 34 insertions(+), 2 deletions(-)
>
The nfsd case typically uses buffered or DONTCACHE for the unaligned
sub-page-sized "ends" when direct is in force, so mostly these ends get
submitted as small (< PAGE_SIZE or folio_size()) buffered I/Os.
Why do we need this special casing instead of just making those
unaligned ends use regular buffered I/O? Isn't it basically the same
thing?
> diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
> index 0a5ebfda90f1..780a0186045d 100644
> --- a/fs/iomap/buffered-io.c
> +++ b/fs/iomap/buffered-io.c
> @@ -802,8 +802,18 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
>
> if (iter->flags & IOMAP_NOWAIT)
> fgp |= FGP_NOWAIT;
> - if (iter->flags & IOMAP_DONTCACHE)
> + if (iter->flags & IOMAP_DONTCACHE) {
> fgp |= FGP_DONTCACHE;
> + /*
> + * A page this write covers only partly is shared with a
> + * neighbouring write: get it as a single page, which is
> + * kept until the write that completes it.
> + */
> + if (offset_in_page(pos) || len < PAGE_SIZE) {
> + fgp |= FGP_DONTCACHE_EDGE;
> + len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
> + }
> + }
> fgp |= fgf_set_order(len);
>
> return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index 3012bd95812a..0c1d6bd09aa0 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -701,6 +701,10 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
> * * %FGP_NOWAIT - Don't block on the folio lock.
> * * %FGP_STABLE - Wait for the folio to be stable (finished writeback)
> * * %FGP_DONTCACHE - Uncached buffered IO
> + * * %FGP_DONTCACHE_EDGE - With %FGP_DONTCACHE: the write covers this page
> + * only partly, and a neighbouring write may complete it. A folio created
> + * for it is not marked drop-behind; a folio found uptodate is, as the
> + * write that completes it.
> * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
> * implementation.
> */
> @@ -715,6 +719,7 @@ typedef unsigned int __bitwise fgf_t;
> #define FGP_FOR_MMAP ((__force fgf_t)0x00000040)
> #define FGP_STABLE ((__force fgf_t)0x00000080)
> #define FGP_DONTCACHE ((__force fgf_t)0x00000100)
> +#define FGP_DONTCACHE_EDGE ((__force fgf_t)0x00000200)
> #define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */
>
> #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
> diff --git a/mm/filemap.c b/mm/filemap.c
> index ed7e4dbea653..0397ec771547 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -2016,6 +2016,16 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
>
> if (fgp_flags & FGP_STABLE)
> folio_wait_stable(folio);
> +
> + /*
> + * An uncached write that covers this folio only partly, finding it
> + * uptodate, is the write that completes it: the neighbouring write
> + * that left it here has written its part. Mark it now, under the
> + * folio lock, so the writeback that cleans both writes' data drops it.
> + */
> + if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
> + (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
> + folio_mark_dropbehind(folio);
> no_page:
> if (!folio && (fgp_flags & FGP_CREAT)) {
> unsigned int min_order = mapping_min_folio_order(mapping);
> @@ -2053,7 +2063,14 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
> /* Init accessed so avoid atomic mark_page_accessed later */
> if (fgp_flags & FGP_ACCESSED)
> __folio_set_referenced(folio);
> - if (fgp_flags & FGP_DONTCACHE)
> + /*
> + * A folio the write covers only partly is left
> + * unmarked, so that writing back this write's part
> + * does not drop it before a neighbouring write has
> + * written the rest.
> + */
> + if ((fgp_flags & FGP_DONTCACHE) &&
> + !(fgp_flags & FGP_DONTCACHE_EDGE))
> __folio_set_dropbehind(folio);
>
> err = filemap_add_folio(mapping, folio, index, gfp);
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v7 4/5] mm, iomap: let an uncached writer complete its shared pages itself
2026-10-20 22:03 [PATCH v7 0/5] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
` (2 preceding siblings ...)
2026-10-20 22:03 ` [PATCH v7 3/5] mm, iomap: keep a partly covered page until an uncached write completes it Mike Snitzer
@ 2026-10-20 22:03 ` Mike Snitzer
2026-10-21 10:54 ` Jeff Layton
2026-10-20 22:03 ` [PATCH v7 5/5] NFSD: drop a shared boundary page only once both WRITEs have written it Mike Snitzer
4 siblings, 1 reply; 11+ messages in thread
From: Mike Snitzer @ 2026-10-20 22:03 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, Jonathan Corbet,
linux-mm, linux-fsdevel
The write that completes a page an uncached write covers only partly
marks it drop-behind as it writes, while the page is dirty, so the page
counts as DONTCACHE dirty and the DONTCACHE writeback kick writes it
back. A writer that syncs its write itself, once, after several
segments -- NFSD answering a stable WRITE, the NFS client's LOCALIO path
-- issues each segment without IOCB_DSYNC, so every segment's kick
writes back pages its own sync is about to write. Measured with NFSD
DIRECT and 32 writers of 47008-byte records: with the previous patch
alone, 23,000 to 65,000 kick rounds a run, where NFSD marking the pages
itself after its sync had taken 51 to 274; with this patch and NFSD
using it, a FILE_SYNC WRITE wakes the kick not at all.
Let such a writer complete those pages itself. IOCB_DONTCACHE_DEFER
(IOMAP_DONTCACHE_DEFER, FGP_DONTCACHE_DEFER) leaves a partly covered page
unmarked even when the write finds it uptodate. Right before writing such
a page the writer asks filemap_dontcache_edge_completes() whether it is
uptodate already, which means the neighbouring write has written its part
and this write completes it; once the write and its sync are done,
filemap_dontcache_edge_complete() marks it and, clean, drops it at once.
Two writes that arrive together may both find the page absent before
either has written; that page is then left cached until reclaim, which
the folio-lock ordering of the default path avoids.
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
fs/iomap/buffered-io.c | 4 +++
fs/iomap/trace.h | 3 ++-
include/linux/fs.h | 8 +++++-
include/linux/iomap.h | 1 +
include/linux/pagemap.h | 6 +++++
mm/filemap.c | 58 ++++++++++++++++++++++++++++++++++++++++-
6 files changed, 77 insertions(+), 3 deletions(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 780a0186045d..5ca82000e2ec 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -811,6 +811,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
*/
if (offset_in_page(pos) || len < PAGE_SIZE) {
fgp |= FGP_DONTCACHE_EDGE;
+ if (iter->flags & IOMAP_DONTCACHE_DEFER)
+ fgp |= FGP_DONTCACHE_DEFER;
len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
}
}
@@ -1318,6 +1320,8 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
iter.flags |= IOMAP_NOWAIT;
if (iocb->ki_flags & IOCB_DONTCACHE)
iter.flags |= IOMAP_DONTCACHE;
+ if (iocb->ki_flags & IOCB_DONTCACHE_DEFER)
+ iter.flags |= IOMAP_DONTCACHE_DEFER;
while ((ret = iomap_iter(&iter, ops)) > 0)
iter.status = iomap_write_iter(&iter, i, write_ops);
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e1ea8392cf47..4f0343eeed1b 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -104,7 +104,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
{ IOMAP_UNSHARE, "UNSHARE" }, \
{ IOMAP_DAX, "DAX" }, \
{ IOMAP_ATOMIC, "ATOMIC" }, \
- { IOMAP_DONTCACHE, "DONTCACHE" }
+ { IOMAP_DONTCACHE, "DONTCACHE" }, \
+ { IOMAP_DONTCACHE_DEFER, "DONTCACHE_DEFER" }
#define IOMAP_F_FLAGS_STRINGS \
{ IOMAP_F_NEW, "NEW" }, \
diff --git a/include/linux/fs.h b/include/linux/fs.h
index f9d1e05e8ae6..ad181b6ce6b4 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -357,6 +357,11 @@ struct readahead_control;
/* kiocb is a read or write operation submitted by fs/aio.c. */
#define IOCB_AIO_RW (1 << 22)
#define IOCB_HAS_METADATA (1 << 23)
+/*
+ * With IOCB_DONTCACHE: the caller completes the pages the write covers only
+ * partly itself, see filemap_dontcache_edge_complete().
+ */
+#define IOCB_DONTCACHE_DEFER (1 << 24)
/* for use in trace events */
#define TRACE_IOCB_STRINGS \
@@ -374,7 +379,8 @@ struct readahead_control;
{ IOCB_NOIO, "NOIO" }, \
{ IOCB_ALLOC_CACHE, "ALLOC_CACHE" }, \
{ IOCB_AIO_RW, "AIO_RW" }, \
- { IOCB_HAS_METADATA, "AIO_HAS_METADATA" }
+ { IOCB_HAS_METADATA, "AIO_HAS_METADATA" }, \
+ { IOCB_DONTCACHE_DEFER, "DONTCACHE_DEFER" }
struct kiocb {
struct file *ki_filp;
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index bc7ae6327dbf..f3509d495a92 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -212,6 +212,7 @@ struct iomap_write_ops {
#endif /* CONFIG_FS_DAX */
#define IOMAP_ATOMIC (1 << 9) /* torn-write protection */
#define IOMAP_DONTCACHE (1 << 10)
+#define IOMAP_DONTCACHE_DEFER (1 << 11) /* see IOCB_DONTCACHE_DEFER */
/*
* Return the existing mapping at pos, or reserve space starting at pos for up
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0c1d6bd09aa0..a0a67e4c940b 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -705,6 +705,9 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
* only partly, and a neighbouring write may complete it. A folio created
* for it is not marked drop-behind; a folio found uptodate is, as the
* write that completes it.
+ * * %FGP_DONTCACHE_DEFER - With %FGP_DONTCACHE_EDGE: the caller completes the
+ * page itself (filemap_dontcache_edge_complete()), so a folio found uptodate
+ * is not marked either.
* * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
* implementation.
*/
@@ -720,6 +723,7 @@ typedef unsigned int __bitwise fgf_t;
#define FGP_STABLE ((__force fgf_t)0x00000080)
#define FGP_DONTCACHE ((__force fgf_t)0x00000100)
#define FGP_DONTCACHE_EDGE ((__force fgf_t)0x00000200)
+#define FGP_DONTCACHE_DEFER ((__force fgf_t)0x00000400)
#define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */
#define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
@@ -1316,6 +1320,8 @@ void folio_end_writeback(struct folio *folio);
void folio_end_writeback_no_dropbehind(struct folio *folio);
void folio_end_dropbehind(struct folio *folio);
void folio_mark_dropbehind(struct folio *folio);
+bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos);
+void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos);
void folio_wait_stable(struct folio *folio);
void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
diff --git a/mm/filemap.c b/mm/filemap.c
index 0397ec771547..7d7c56a42643 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1675,6 +1675,61 @@ void folio_mark_dropbehind(struct folio *folio)
}
EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
+/**
+ * filemap_dontcache_edge_completes - will this write complete a shared page?
+ * @mapping: the file's mapping
+ * @pos: any byte offset within the page
+ *
+ * For an uncached writer that completes the pages it covers only partly
+ * itself (IOCB_DONTCACHE_DEFER). Call right before writing such a page: it
+ * is in the page cache and uptodate when the neighbouring write has already
+ * written its part, and this write then completes it. Two writes that
+ * arrive together may both find it absent; the page is then left cached,
+ * unmarked, until reclaim.
+ *
+ * Return: true if the caller is to call filemap_dontcache_edge_complete()
+ * once its write, including any sync of it, is done.
+ */
+bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos)
+{
+ struct folio *folio;
+ bool ret;
+
+ /* FGP_DONTCACHE: a plain lookup would clear a drop-behind mark. */
+ folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT, FGP_DONTCACHE, 0);
+ if (IS_ERR(folio))
+ return false;
+ ret = folio_test_uptodate(folio);
+ folio_put(folio);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(filemap_dontcache_edge_completes);
+
+/**
+ * filemap_dontcache_edge_complete - drop a shared page both writes have written
+ * @mapping: the file's mapping
+ * @pos: any byte offset within the page
+ *
+ * Mark the page drop-behind. If it is already written back, drop it now;
+ * otherwise the writeback that cleans it drops it. A dirty page marked here
+ * counts as DONTCACHE dirty and wakes the DONTCACHE writeback kick, so a
+ * caller that syncs its write calls this after the sync.
+ */
+void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos)
+{
+ struct folio *folio;
+
+ folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT,
+ FGP_LOCK | FGP_DONTCACHE, 0);
+ if (IS_ERR(folio))
+ return;
+ folio_mark_dropbehind(folio);
+ folio_unlock(folio);
+ folio_end_dropbehind(folio);
+ folio_put(folio);
+}
+EXPORT_SYMBOL_GPL(filemap_dontcache_edge_complete);
+
/**
* folio_end_writeback_no_dropbehind - End writeback against a folio.
* @folio: The folio.
@@ -2024,7 +2079,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
* folio lock, so the writeback that cleans both writes' data drops it.
*/
if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
- (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
+ !(fgp_flags & FGP_DONTCACHE_DEFER) && (fgp_flags & FGP_LOCK) &&
+ folio_test_uptodate(folio))
folio_mark_dropbehind(folio);
no_page:
if (!folio && (fgp_flags & FGP_CREAT)) {
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH v7 4/5] mm, iomap: let an uncached writer complete its shared pages itself
2026-10-20 22:03 ` [PATCH v7 4/5] mm, iomap: let an uncached writer complete its shared pages itself Mike Snitzer
@ 2026-10-21 10:54 ` Jeff Layton
0 siblings, 0 replies; 11+ messages in thread
From: Jeff Layton @ 2026-10-21 10:54 UTC (permalink / raw)
To: Mike Snitzer, Chuck Lever
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, Jonathan Corbet,
linux-mm, linux-fsdevel
On Tue, 2026-10-20 at 18:03 -0400, Mike Snitzer wrote:
> The write that completes a page an uncached write covers only partly
> marks it drop-behind as it writes, while the page is dirty, so the page
> counts as DONTCACHE dirty and the DONTCACHE writeback kick writes it
> back. A writer that syncs its write itself, once, after several
> segments -- NFSD answering a stable WRITE, the NFS client's LOCALIO path
> -- issues each segment without IOCB_DSYNC, so every segment's kick
> writes back pages its own sync is about to write. Measured with NFSD
> DIRECT and 32 writers of 47008-byte records: with the previous patch
> alone, 23,000 to 65,000 kick rounds a run, where NFSD marking the pages
> itself after its sync had taken 51 to 274; with this patch and NFSD
> using it, a FILE_SYNC WRITE wakes the kick not at all.
>
> Let such a writer complete those pages itself. IOCB_DONTCACHE_DEFER
> (IOMAP_DONTCACHE_DEFER, FGP_DONTCACHE_DEFER) leaves a partly covered page
> unmarked even when the write finds it uptodate. Right before writing such
> a page the writer asks filemap_dontcache_edge_completes() whether it is
> uptodate already, which means the neighbouring write has written its part
> and this write completes it; once the write and its sync are done,
> filemap_dontcache_edge_complete() marks it and, clean, drops it at once.
>
> Two writes that arrive together may both find the page absent before
> either has written; that page is then left cached until reclaim, which
> the folio-lock ordering of the default path avoids.
>
> Assisted-by: LLM
> Signed-off-by: Mike Snitzer <snitzer@kernel.org>
> ---
> fs/iomap/buffered-io.c | 4 +++
> fs/iomap/trace.h | 3 ++-
> include/linux/fs.h | 8 +++++-
> include/linux/iomap.h | 1 +
> include/linux/pagemap.h | 6 +++++
> mm/filemap.c | 58 ++++++++++++++++++++++++++++++++++++++++-
> 6 files changed, 77 insertions(+), 3 deletions(-)
>
> diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
> index 780a0186045d..5ca82000e2ec 100644
> --- a/fs/iomap/buffered-io.c
> +++ b/fs/iomap/buffered-io.c
> @@ -811,6 +811,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
> */
> if (offset_in_page(pos) || len < PAGE_SIZE) {
> fgp |= FGP_DONTCACHE_EDGE;
> + if (iter->flags & IOMAP_DONTCACHE_DEFER)
> + fgp |= FGP_DONTCACHE_DEFER;
> len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos));
> }
> }
> @@ -1318,6 +1320,8 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
> iter.flags |= IOMAP_NOWAIT;
> if (iocb->ki_flags & IOCB_DONTCACHE)
> iter.flags |= IOMAP_DONTCACHE;
> + if (iocb->ki_flags & IOCB_DONTCACHE_DEFER)
> + iter.flags |= IOMAP_DONTCACHE_DEFER;
>
> while ((ret = iomap_iter(&iter, ops)) > 0)
> iter.status = iomap_write_iter(&iter, i, write_ops);
> diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
> index e1ea8392cf47..4f0343eeed1b 100644
> --- a/fs/iomap/trace.h
> +++ b/fs/iomap/trace.h
> @@ -104,7 +104,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
> { IOMAP_UNSHARE, "UNSHARE" }, \
> { IOMAP_DAX, "DAX" }, \
> { IOMAP_ATOMIC, "ATOMIC" }, \
> - { IOMAP_DONTCACHE, "DONTCACHE" }
> + { IOMAP_DONTCACHE, "DONTCACHE" }, \
> + { IOMAP_DONTCACHE_DEFER, "DONTCACHE_DEFER" }
>
> #define IOMAP_F_FLAGS_STRINGS \
> { IOMAP_F_NEW, "NEW" }, \
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index f9d1e05e8ae6..ad181b6ce6b4 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -357,6 +357,11 @@ struct readahead_control;
> /* kiocb is a read or write operation submitted by fs/aio.c. */
> #define IOCB_AIO_RW (1 << 22)
> #define IOCB_HAS_METADATA (1 << 23)
> +/*
> + * With IOCB_DONTCACHE: the caller completes the pages the write covers only
> + * partly itself, see filemap_dontcache_edge_complete().
> + */
> +#define IOCB_DONTCACHE_DEFER (1 << 24)
>
> /* for use in trace events */
> #define TRACE_IOCB_STRINGS \
> @@ -374,7 +379,8 @@ struct readahead_control;
> { IOCB_NOIO, "NOIO" }, \
> { IOCB_ALLOC_CACHE, "ALLOC_CACHE" }, \
> { IOCB_AIO_RW, "AIO_RW" }, \
> - { IOCB_HAS_METADATA, "AIO_HAS_METADATA" }
> + { IOCB_HAS_METADATA, "AIO_HAS_METADATA" }, \
> + { IOCB_DONTCACHE_DEFER, "DONTCACHE_DEFER" }
>
> struct kiocb {
> struct file *ki_filp;
> diff --git a/include/linux/iomap.h b/include/linux/iomap.h
> index bc7ae6327dbf..f3509d495a92 100644
> --- a/include/linux/iomap.h
> +++ b/include/linux/iomap.h
> @@ -212,6 +212,7 @@ struct iomap_write_ops {
> #endif /* CONFIG_FS_DAX */
> #define IOMAP_ATOMIC (1 << 9) /* torn-write protection */
> #define IOMAP_DONTCACHE (1 << 10)
> +#define IOMAP_DONTCACHE_DEFER (1 << 11) /* see IOCB_DONTCACHE_DEFER */
>
> /*
> * Return the existing mapping at pos, or reserve space starting at pos for up
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index 0c1d6bd09aa0..a0a67e4c940b 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -705,6 +705,9 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
> * only partly, and a neighbouring write may complete it. A folio created
> * for it is not marked drop-behind; a folio found uptodate is, as the
> * write that completes it.
> + * * %FGP_DONTCACHE_DEFER - With %FGP_DONTCACHE_EDGE: the caller completes the
> + * page itself (filemap_dontcache_edge_complete()), so a folio found uptodate
> + * is not marked either.
> * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin()
> * implementation.
> */
> @@ -720,6 +723,7 @@ typedef unsigned int __bitwise fgf_t;
> #define FGP_STABLE ((__force fgf_t)0x00000080)
> #define FGP_DONTCACHE ((__force fgf_t)0x00000100)
> #define FGP_DONTCACHE_EDGE ((__force fgf_t)0x00000200)
> +#define FGP_DONTCACHE_DEFER ((__force fgf_t)0x00000400)
> #define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */
>
> #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE)
> @@ -1316,6 +1320,8 @@ void folio_end_writeback(struct folio *folio);
> void folio_end_writeback_no_dropbehind(struct folio *folio);
> void folio_end_dropbehind(struct folio *folio);
> void folio_mark_dropbehind(struct folio *folio);
> +bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos);
> +void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos);
> void folio_wait_stable(struct folio *folio);
> void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
> void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
> diff --git a/mm/filemap.c b/mm/filemap.c
> index 0397ec771547..7d7c56a42643 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -1675,6 +1675,61 @@ void folio_mark_dropbehind(struct folio *folio)
> }
> EXPORT_SYMBOL_GPL(folio_mark_dropbehind);
>
> +/**
> + * filemap_dontcache_edge_completes - will this write complete a shared page?
> + * @mapping: the file's mapping
> + * @pos: any byte offset within the page
> + *
> + * For an uncached writer that completes the pages it covers only partly
> + * itself (IOCB_DONTCACHE_DEFER). Call right before writing such a page: it
> + * is in the page cache and uptodate when the neighbouring write has already
> + * written its part, and this write then completes it. Two writes that
> + * arrive together may both find it absent; the page is then left cached,
> + * unmarked, until reclaim.
> + *
> + * Return: true if the caller is to call filemap_dontcache_edge_complete()
> + * once its write, including any sync of it, is done.
> + */
> +bool filemap_dontcache_edge_completes(struct address_space *mapping, loff_t pos)
> +{
> + struct folio *folio;
> + bool ret;
> +
> + /* FGP_DONTCACHE: a plain lookup would clear a drop-behind mark. */
> + folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT, FGP_DONTCACHE, 0);
> + if (IS_ERR(folio))
> + return false;
> + ret = folio_test_uptodate(folio);
> + folio_put(folio);
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(filemap_dontcache_edge_completes);
> +
> +/**
> + * filemap_dontcache_edge_complete - drop a shared page both writes have written
> + * @mapping: the file's mapping
> + * @pos: any byte offset within the page
> + *
> + * Mark the page drop-behind. If it is already written back, drop it now;
> + * otherwise the writeback that cleans it drops it. A dirty page marked here
> + * counts as DONTCACHE dirty and wakes the DONTCACHE writeback kick, so a
> + * caller that syncs its write calls this after the sync.
> + */
> +void filemap_dontcache_edge_complete(struct address_space *mapping, loff_t pos)
> +{
> + struct folio *folio;
> +
> + folio = __filemap_get_folio(mapping, pos >> PAGE_SHIFT,
> + FGP_LOCK | FGP_DONTCACHE, 0);
> + if (IS_ERR(folio))
> + return;
> + folio_mark_dropbehind(folio);
> + folio_unlock(folio);
> + folio_end_dropbehind(folio);
> + folio_put(folio);
> +}
> +EXPORT_SYMBOL_GPL(filemap_dontcache_edge_complete);
> +
> /**
> * folio_end_writeback_no_dropbehind - End writeback against a folio.
> * @folio: The folio.
> @@ -2024,7 +2079,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
> * folio lock, so the writeback that cleans both writes' data drops it.
> */
> if ((fgp_flags & FGP_DONTCACHE) && (fgp_flags & FGP_DONTCACHE_EDGE) &&
> - (fgp_flags & FGP_LOCK) && folio_test_uptodate(folio))
> + !(fgp_flags & FGP_DONTCACHE_DEFER) && (fgp_flags & FGP_LOCK) &&
> + folio_test_uptodate(folio))
> folio_mark_dropbehind(folio);
> no_page:
> if (!folio && (fgp_flags & FGP_CREAT)) {
2 comments from Chuck's sashiko on this one:
[Severity: Low]
Could filemap_dontcache_edge_complete() in mm/filemap.c avoid missing the
immediate drop when another task acquires the folio lock between
folio_unlock() and folio_end_dropbehind()? In that case folio_end_dropbehind()
fails its folio_trylock() and does not retry; if the folio is already clean,
there may be no later writeback completion to drop it, so it can remain cached
until unrelated activity or reclaim.
[Severity: Medium]
Could filemap_dontcache_edge_complete() in mm/filemap.c restrict marking to
the boundary page when the mapping returns a large folio? iomap_get_folio()
in fs/iomap/buffered-io.c limits a partial edge request to one page, but the
mapping's minimum folio order can still cause __filemap_get_folio_mpol() to
return a multi-page folio, or the lookup can find an existing large folio.
This helper then marks the entire folio for drop-behind. For example, if a
caller uses deferred completion on an XFS mapping with a minimum folio order,
marking one edge can account the whole dirty folio for DONTCACHE writeback or
invalidate the whole clean, unmapped folio. Could this path handle only the
page covered by the completed edge?
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v7 5/5] NFSD: drop a shared boundary page only once both WRITEs have written it
2026-10-20 22:03 [PATCH v7 0/5] mm, iomap, NFSD: stop uncached writes reading back shared pages Mike Snitzer
` (3 preceding siblings ...)
2026-10-20 22:03 ` [PATCH v7 4/5] mm, iomap: let an uncached writer complete its shared pages itself Mike Snitzer
@ 2026-10-20 22:03 ` Mike Snitzer
4 siblings, 0 replies; 11+ messages in thread
From: Mike Snitzer @ 2026-10-20 22:03 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton
Cc: linux-nfs, NeilBrown, Olga Kornievskaia, Dai Ngo, Tom Talpey,
Christoph Hellwig, Jens Axboe, Matthew Wilcox, Jonathan Corbet,
linux-mm, linux-fsdevel, Jonathan Flynn
A buffered DONTCACHE segment of a direct-mode WRITE -- the misaligned
start or end of a split WRITE, or the whole of one too small to split --
covers its first and last page only partly when it does not start or end
on a page boundary. Such a page is shared with the neighbouring WRITE,
which may arrive before, after or at the same time, from any client.
The page cache keeps a page an uncached write covers only partly for the
write that completes it ("mm, iomap: keep a partly covered page until an
uncached write completes it"), so the second WRITE does not have to read
it back from disk. Left at that, the page cache marks the page drop-
behind while it is still dirty, which counts it as DONTCACHE dirty and
wakes the DONTCACHE writeback kick for every segment of a stable WRITE,
only to write back what NFSD's single fsync is about to write.
Issue the buffered segments IOCB_DONTCACHE_DEFER and mark the pages here
instead ("mm, iomap: let an uncached writer complete its shared pages
itself"). Immediately before writing a segment, NFSD asks
filemap_dontcache_edge_completes() whether each page it covers only
partly has already been written by the neighbouring WRITE; once the
whole WRITE, its fsync included, is complete, it calls
filemap_dontcache_edge_complete() for those pages, which drops a clean
page at once. An UNSTABLE WRITE marks them dirty, and the writeback
that cleans them, by the flusher or for the client's COMMIT, drops them.
A page whose neighbouring WRITE never arrives, such as the last page of
the file, or arrives at the same time, so that neither finds the other's
part, stays cached until it is reclaimed. Nothing changes when the
buffered segments are normal buffered I/O (direct_misaligned_dontcache
is N, or the file system lacks FOP_DONTCACHE); the page cache keeps
partly covered pages only for file systems that use iomap for buffered
writes.
Measured with 32 interleaved writers of 47008-byte records over an
emulated 4Kn NVMe with 4 KiB pages, NFSD DIRECT with
direct_misaligned_dontcache=Y, each run from a freshly made file system:
device reads per WRITE are 0.0001-0.0002 whether each record is one
FILE_SYNC WRITE, two sent one after the other, two sent together, or two
sent UNSTABLE, against 45144 reads for 45664 WRITEs with no shared-page
handling, and the DONTCACHE writeback kick does not run at all for the
FILE_SYNC WRITEs.
On a four-server pNFS flexfiles rig (x86_64, 4 KiB logical blocks), three
clients writing 47008-byte records for 240 s against 640 nfsd threads per
server, keeping the shared page took device reads from 0.014-0.015 per
record to 0.003, page-cache growth over the write phase from about
11 GiB to 2.9 GiB, and write throughput up by 2.4% to 4.9%, with read
throughput unchanged. Keeping every boundary page cached instead, with
direct_misaligned_dontcache=N, grows the page cache by about 630 GiB over
the same runs.
Reported-by: Jonathan Flynn <jonathan.flynn@hammerspace.com>
Assisted-by: LLM
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
.../filesystems/nfs/nfsd-io-modes.rst | 17 ++++
fs/nfsd/vfs.c | 99 ++++++++++++++++---
2 files changed, 104 insertions(+), 12 deletions(-)
diff --git a/Documentation/filesystems/nfs/nfsd-io-modes.rst b/Documentation/filesystems/nfs/nfsd-io-modes.rst
index 0e69b5fdd675..9c663ce97c48 100644
--- a/Documentation/filesystems/nfs/nfsd-io-modes.rst
+++ b/Documentation/filesystems/nfs/nfsd-io-modes.rst
@@ -167,6 +167,23 @@ Misaligned WRITE:
above. Set it to N for a workload that reads back or rewrites what
it just wrote.
+ The page holding a start or end segment is shared with the
+ neighbouring WRITE, which may arrive before, after or at the same
+ time, from any client. So that the second WRITE to such a page does
+ not have to read it back from disk, the page cache keeps a page an
+ uncached write covers only partly, unmarked, for the write that
+ completes it. NFSD asks, immediately before writing a start or end
+ segment, whether the neighbouring WRITE has already written its
+ part; if it has, NFSD marks the page DONTCACHE once its own WRITE,
+ including its fsync, is complete, and the page is dropped. A page
+ whose neighbouring WRITE never arrives, or arrives at the same time,
+ stays cached until it is reclaimed. The first and last pages of a
+ WRITE that is not split are handled the same way.
+
+ None of this applies when the start and end segments are normal
+ buffered IO, or for a filesystem that does not use iomap for
+ buffered writes.
+
Tracing:
The nfsd_read_direct trace event marks a READ that NFSD issues as
O_DIRECT. It records the offset and length the client requested.
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index 9219f8eb386d..3605ddf1ad1e 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -1387,6 +1387,7 @@ static int wait_for_concurrent_writes(struct file *file)
struct nfsd_write_dio_seg {
struct iov_iter iter;
int flags;
+ bool claim; /* completes shared pages */
};
static unsigned long
@@ -1406,6 +1407,65 @@ nfsd_write_dio_seg_init(struct nfsd_write_dio_seg *segment,
iov_iter_advance(&segment->iter, start);
iov_iter_truncate(&segment->iter, len);
segment->flags = iocb->ki_flags;
+ segment->claim = false;
+}
+
+/* The shared pages a segment completes, and how much of it was written. */
+struct nfsd_write_dio_hold {
+ loff_t first, last;
+ bool complete_first, complete_last;
+ size_t written;
+};
+
+/*
+ * A page that a buffered DONTCACHE segment covers only partly is shared with
+ * the neighbouring WRITE. The page cache keeps it, unmarked, for the write
+ * that completes it; the segment is issued IOCB_DONTCACHE_DEFER so that this
+ * WRITE, not the page cache, marks it, once the whole WRITE, its fsync
+ * included, is complete. Ask whether the neighbour has already written its
+ * part immediately before writing @seg: asking earlier gives the neighbour
+ * time to write its part and find this WRITE's part missing.
+ */
+static void
+nfsd_write_dio_seg_hold(struct file *file, const struct nfsd_write_dio_seg *seg,
+ loff_t pos, size_t len, struct nfsd_write_dio_hold *hold)
+{
+ struct address_space *mapping = file->f_mapping;
+
+ hold->first = pos;
+ hold->last = pos + len - 1;
+ hold->complete_first = false;
+ hold->complete_last = false;
+ hold->written = 0;
+ if (seg->claim && len) {
+ bool one_page = (hold->first >> PAGE_SHIFT) ==
+ (hold->last >> PAGE_SHIFT);
+ bool head = hold->first & ~PAGE_MASK;
+ bool tail = (hold->last + 1) & ~PAGE_MASK;
+
+ /* Only a page the segment covers partly is shared. */
+ if (head || (one_page && tail))
+ hold->complete_first =
+ filemap_dontcache_edge_completes(mapping, hold->first);
+ if (!one_page && tail)
+ hold->complete_last =
+ filemap_dontcache_edge_completes(mapping, hold->last);
+ }
+}
+
+/* Drop the shared pages this WRITE was the second of two to write. */
+static void
+nfsd_write_dio_seg_release(struct file *file,
+ const struct nfsd_write_dio_hold *hold)
+{
+ struct address_space *mapping = file->f_mapping;
+
+ if (hold->complete_first && hold->written)
+ filemap_dontcache_edge_complete(mapping, hold->first);
+ /* A short write did not reach the last page. */
+ if (hold->complete_last &&
+ hold->written == hold->last - hold->first + 1)
+ filemap_dontcache_edge_complete(mapping, hold->last);
}
static unsigned int
@@ -1428,7 +1488,9 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
if (nf->nf_file->f_op->fop_flags & FOP_DONTCACHE)
dontcache_flags = IOCB_DONTCACHE;
- buffered_flags = nfsd_direct_misaligned_dontcache ? dontcache_flags : 0;
+ /* NFSD marks their shared pages itself, see nfsd_write_dio_seg_hold() */
+ buffered_flags = nfsd_direct_misaligned_dontcache && dontcache_flags ?
+ dontcache_flags | IOCB_DONTCACHE_DEFER : 0;
/*
* Check if direct I/O is feasible for this write request.
@@ -1483,7 +1545,8 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
if (prefix) {
nfsd_write_dio_seg_init(&segments[nsegs], bvec,
nvecs, total, 0, prefix, iocb);
- segments[nsegs++].flags |= buffered_flags;
+ segments[nsegs].flags |= buffered_flags;
+ segments[nsegs++].claim = !!buffered_flags;
}
nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs,
@@ -1516,7 +1579,8 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
if (suffix) {
nfsd_write_dio_seg_init(&segments[nsegs], bvec, nvecs, total,
prefix + middle, suffix, iocb);
- segments[nsegs++].flags |= buffered_flags;
+ segments[nsegs].flags |= buffered_flags;
+ segments[nsegs++].claim = !!buffered_flags;
}
return nsegs;
@@ -1525,6 +1589,7 @@ nfsd_write_dio_iters_init(struct svc_rqst *rqstp, struct svc_fh *fhp,
nfsd_write_dio_seg_init(&segments[0], bvec, nvecs, total, 0,
total, iocb);
segments[0].flags |= buffered_flags;
+ segments[0].claim = !!buffered_flags;
return 1;
}
@@ -1533,12 +1598,13 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
struct nfsd_file *nf, unsigned int nvecs,
unsigned long *cnt, struct kiocb *kiocb)
{
+ struct nfsd_write_dio_hold holds[3];
struct nfsd_write_dio_seg segments[3];
struct file *file = nf->nf_file;
loff_t start = kiocb->ki_pos;
- unsigned int nsegs, i;
+ unsigned int nsegs, nheld = 0, i;
+ ssize_t host_err = 0;
int sync_flags;
- ssize_t host_err;
size_t expected;
nsegs = nfsd_write_dio_iters_init(rqstp, fhp, nf, rqstp->rq_bvec,
@@ -1567,23 +1633,32 @@ nfsd_direct_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
kiocb->ki_flags);
expected = iov_iter_count(&segments[i].iter);
-
+ nfsd_write_dio_seg_hold(file, &segments[i], kiocb->ki_pos,
+ expected, &holds[nheld]);
host_err = vfs_iocb_iter_write(file, kiocb, &segments[i].iter);
if (host_err < 0)
- return host_err;
+ break;
+ holds[nheld++].written = host_err;
*cnt += host_err;
if (host_err < (ssize_t)expected)
break; /* partial write */
}
- if (sync_flags && *cnt) {
+ if (host_err >= 0 && sync_flags && *cnt)
host_err = vfs_fsync_range(file, start, start + *cnt - 1,
!(sync_flags & IOCB_SYNC));
- if (host_err < 0)
- return host_err;
- }
- return 0;
+ /*
+ * Mark the pages this WRITE shares with its neighbours for dropping
+ * only once the WRITE is complete, including its fsync if it has one.
+ * A page marked while still dirty is counted as DONTCACHE dirty and
+ * wakes the DONTCACHE writeback kick; a page marked once clean is
+ * simply dropped.
+ */
+ for (i = 0; i < nheld; i++)
+ nfsd_write_dio_seg_release(file, &holds[i]);
+
+ return host_err < 0 ? host_err : 0;
}
/**
--
2.52.0
^ permalink raw reply related [flat|nested] 11+ messages in thread