Linux NFS development
 help / color / mirror / Atom feed
From: Mike Snitzer <snitzer@kernel.org>
To: Jan Kara <jack@suse.cz>
Cc: Matthew Wilcox <willy@infradead.org>,
	linux-mm@kvack.org, Christian Brauner <brauner@kernel.org>,
	linux-fsdevel@vger.kernel.org, Chuck Lever <cel@kernel.org>,
	Jeff Layton <jlayton@kernel.org>,
	linux-nfs@vger.kernel.org
Subject: Re: [PATCH v12 0/7] mm, iomap, NFSD: stop DONTCACHE writes reading back shared pages
Date: Mon, 16 Nov 2026 15:47:49 -0500	[thread overview]
Message-ID: <avtr9YaUNyjGHk4h@kernel.org> (raw)
In-Reply-To: <iushs6l3mjqkq65srej4gevaf2oeazy5j5miaipy6yxscetzjd@d4ti4qgbhman>

On Mon, Nov 16, 2026 at 12:02:20PM +0100, Jan Kara wrote:
> Hi!
> 
> On Thu 12-11-26 14:09:03, Mike Snitzer wrote:
> > An RWF_DONTCACHE write is buffered IO: its data goes through the page
> > cache, and writeback drops each folio once the folio is written back.
> > A write that starts or ends inside a folio covers that folio only
> > partially.  Usually another write covers the rest of the folio.  If
> > writeback drops the folio before the other write writes its part, the
> > other write must first read the folio back from disk.  32 threads
> > writing interleaved records to one XFS file, 4 KiB pages, before and
> > after 5/7:
> > 
> >   record  pwritev2() flags          4 KiB device       MiB/s
> >   bytes                             reads per write
> >                                     before   after     before  after
> >   47008   RWF_DONTCACHE             0.97     0.0005    831     2281
> >   47008   RWF_DONTCACHE|RWF_DSYNC   0.99     0.0005    601     1227
> >   6000    RWF_DONTCACHE             0.97     0.0001    121      488
> >   6000    RWF_DONTCACHE|RWF_DSYNC   0.99     0.0002     90      206
> > 
> > 47008 bytes is the transfer size of IO500's ior-hard, in which each
> > rank writes interleaved records to one shared file.  Any record size
> > that is not a multiple of the page size has the same problem.  6000
> > bytes is a second example.
> 
> OK, I understand the problem but frankly I hate the complexity this
> introduces. Patches 1-3 are good improvements, patch 4 is OK but 5 & 6 look
> too complex for too niche usecase to me. I didn't really pay attention to
> previous revisions of this patch set so I'm sorry if this was discussed but
> can't we just make iomap for unaligned RWF_DONTCACHE write ask for
> minimum-sized folios for the partial edges without FGP_DONTCACHE and
> use FGP_DONTCACHE only for folios fully within write? That will also solve
> the problem with the following write having to read again from the disk and
> it will be much simpler. Sure the edge folios will not be dropped after IO
> completion which may have smaller or larger effect (depending on the
> overall write size) but I'd say that's the cost of userspace making life
> hard for the kernel :-p

Thanks for looking, Jan.

On complexity: 5/7 and 6/7 add almost nothing to the fast path.  The
new code runs only at a partial edge.  Aligned IO, and IO that is not
RWF_DONTCACHE, pays for it with a few flag tests:

- IO that is not RWF_DONTCACHE (buffered, direct, reads): every new
  branch is behind IOMAP_DONTCACHE or FGP_DONTCACHE.  The cost is one
  flag test in the found-folio path of __filemap_get_folio(), one when
  a folio is created, and one per write call in
  iomap_file_buffered_write().
- An aligned RWF_DONTCACHE write: iomap_get_folio() tests whether the
  folio is partially covered, which is a mask and two compares per
  folio.  The test is false, so FGP_DONTCACHE_EDGE is not set, and the
  folio is created and marked dropbehind as before.
- No new locks, atomics, allocations, page flags or per-file state.
  The state of an edge is the uptodate state that the folio already
  has.  Writeback and reads do not change.
- folio_dontcache_edge_written() and the two 6/7 helpers run only at a
  partial edge, or for a caller that asks for IOCB_DONTCACHE_DEFER
  (NFSD).

So the complexity is all in the misaligned case, which needs it.  I'd
argue that if the kernel (MM) can handle userspace making life hard it
should, within reason.  I think my patchset nicely isolates the
complexity to userspace that makes life difficult.

The first part of what you describe is what 5/7 already does.  iomap
asks for a minimum-sized folio at each partial edge, and a new edge
folio is not marked dropbehind.  5/7 adds one more step: the write that
completes an edge marks it, so the writeback that cleans it drops it.

Without that step, every edge stays cached until reclaim.  For this
workload, that is a large part of what the writes put in the page
cache:

- With 4 KiB pages, each record leaves about one page cached.  Through
  LOCALIO with cached edges, 8.6% of the data stayed resident with
  47008-byte records, and 66% with 6000-byte records.  When the write
  that completes an edge drops it, 0% stayed resident.
- With 64 KiB pages, a 47008-byte record never covers a whole folio.
  So no folio is ever marked, and RWF_DONTCACHE caches as much as a
  normal buffered write.
- Jon Flynn measured cached edges through NFSD on four pNFS data servers
  with ior-hard.  The page cache grew by 622 GiB in a 318 s write phase.
  With this series it grew by 2.9 GiB.

So for this misaligned workload, cached edges keep from 9% to all of the
written data in the page cache, depending on the record size and the
page size. The reads stop, but the page cache fills until reclaim runs.

6/7 does not change what is cached.  A caller that syncs its own write
(NFSD, for a FILE_SYNC WRITE) uses it to mark the shared folios after
its sync.  Without it, each segment wakes the DONTCACHE writeback kick
for pages that the sync then writes.

Mike

      reply	other threads:[~2026-11-16 20:47 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12 19:09 [PATCH v12 0/7] mm, iomap, NFSD: stop DONTCACHE writes reading back shared pages Mike Snitzer
2026-11-12 19:09 ` [PATCH v12 1/7] mm: move clearing the dropbehind flag of a found folio into a helper Mike Snitzer
2026-11-16 10:47   ` Jan Kara
2026-11-12 19:09 ` [PATCH v12 2/7] mm: take a folio off WB_DONTCACHE_DIRTY only if it was counted there Mike Snitzer
2026-11-16 10:48   ` Jan Kara
2026-11-12 19:09 ` [PATCH v12 3/7] mm: expose WB_DONTCACHE_DIRTY in bdi debugfs stats Mike Snitzer
2026-11-16 10:48   ` Jan Kara
2026-11-12 19:09 ` [PATCH v12 4/7] mm: add folio_mark_dropbehind() to mark a folio dropbehind and account it Mike Snitzer
2026-11-12 19:09 ` [PATCH v12 5/7] mm, iomap: keep a partially covered page until a DONTCACHE write completes it Mike Snitzer
2026-11-12 19:09 ` [PATCH v12 6/7] mm, iomap: let a DONTCACHE writer complete its shared pages itself Mike Snitzer
2026-11-12 19:09 ` [PATCH v12 7/7] NFSD: drop a shared boundary page only once both WRITEs have written it Mike Snitzer
2026-11-16 11:02 ` [PATCH v12 0/7] mm, iomap, NFSD: stop DONTCACHE writes reading back shared pages Jan Kara
2026-11-16 20:47   ` Mike Snitzer [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=avtr9YaUNyjGHk4h@kernel.org \
    --to=snitzer@kernel.org \
    --cc=brauner@kernel.org \
    --cc=cel@kernel.org \
    --cc=jack@suse.cz \
    --cc=jlayton@kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=willy@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox