All of lore.kernel.org
 help / color / mirror / Atom feed
From: Benjamin Coddington <ben.coddington@hammerspace.com>
To: Trond Myklebust <trondmy@kernel.org>,
	Anna Schumaker <anna@kernel.org>,
	Mike Snitzer <snitzer@kernel.org>
Cc: linux-nfs@vger.kernel.org
Subject: [PATCH v3 0/5] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS
Date: Fri, 20 Nov 2026 11:09:51 -0500	[thread overview]
Message-ID: <cover.1795122899.git.bcodding@hammerspace.com> (raw)

From: Benjamin Coddington <bcodding@hammerspace.com>

When a file's flexfiles layouts carry FF_FLAGS_NO_IO_THRU_MDS, the client
keeps reads and writes off the MDS when it has no layout segment for
them.  Reads and writes handle that differently:

 - Reads test the inode's NFS_INO_NO_IO_THRU_MDS bit, wait a second
   between attempts, and honour io_maxretrans on soft mounts.

 - Writes test a bit on the layout header and return -EAGAIN.
   nfs_writepages() and the O_DIRECT reschedule path resend at once, so
   writers spin for as long as there is no layout.  nfs_wb_folio()
   returns the -EAGAIN, so a read(2) or write(2) that has to flush a
   folio another open context dirtied fails with EAGAIN.  Once the
   header is freed the bit is gone and writes go through the MDS, and
   ff_layout_pg_init_write() sends them there whenever
   pnfs_generic_pg_check_layout() drops an invalidated segment.

Neither direction uses the MDS when the server has said there is no
layout to give.  After NFS4ERR_LAYOUTUNAVAILABLE, RFC 8435 says the I/O
is to go to the MDS (section 5.1.1), and the flag is a hint (section
5.1.2).  Instead, reads wait and writes spin until the 120 second fail
bit expires, then send a LAYOUTGET that can fail the same way.

Patch 1 sends writeback of a single locked folio (nfs_wb_folio(),
reclaim, migration) through the MDS instead of refusing it.  Resending
with the folio locked could deadlock against a bulk recall, which can't
complete while nfs_writepages() holds a layout segment, and
nfs_writepages() would be waiting for that folio.  Patch 2 makes the
O_DIRECT write work a delayed_work.  Patch 3 has the remaining writes
test the inode bit.  After the folios and requests of a refused pass
are unlocked, data-integrity nfs_writepages() waits a second before it
resends, and the O_DIRECT reschedule queues its next pass a second
later; other writeback leaves the folios dirty.  Patch 4 removes the
header bit, which nothing tests after patch 3.  Patch 5 sends reads,
writes and the resend after a data server error through the MDS while
the fail bit for their iomode is set.

That last change is also the path for a data server the client can't
use, such as a version mismatch: the client reports the failure with
LAYOUTERROR or LAYOUTRETURN, and the server decides whether to answer
the next LAYOUTGET with NFS4ERR_LAYOUTUNAVAILABLE.  The missing-nfsv3
case is covered separately by Roberto's "NFSv4/flexfiles: couple back
with NFSv3".

These patches are independent of the series "pNFS: don't reply
NOMATCHING to a layout recall with I/O in flight", but both change
ff_layout_pg_get_mirror_count_write(), so whichever goes second will
need a small rebase.

Tested against a reffs MDS and DS with a six-way mirrored layout,
injecting LAYOUTGET errors after a file recall.  With
NFS4ERR_LAYOUTUNAVAILABLE, reads and writes go through the MDS at once,
and the data written through the MDS is on every mirror.  With a
retryable error (NFS4ERR_SERVERFAULT) held for 6 to 8 seconds, fsync
and the O_DIRECT reschedule retry once a second with no I/O to the MDS,
SIGKILL ends a writer blocked in fsync or an O_DIRECT write,
sync_file_range(SYNC_FILE_RANGE_WRITE) returns 0, and a soft mount with
io_maxretrans=3 returns EIO after about 3 seconds.  During the hold, a
read of a folio that another open context had dirtied flushes it
through the MDS and returns without EAGAIN.

The series is based on v7.4-rc3 with three flexfiles fixes that were
posted separately for stable:

  pNFS/flexfiles: read the layout hdr under i_lock in pg_get_mirror_count_write
  https://lore.kernel.org/linux-nfs/23435d398481f51ce2680053cd3a151fe673f91c.1795108831.git.bcodding@hammerspace.com/
  pNFS/flexfiles: don't delay or restart an rpc_task that LOCALIO never ran
  https://lore.kernel.org/linux-nfs/9c46fda0f2cd6d3843b198dac304b4722514b8e3.1795108831.git.bcodding@hammerspace.com/
  pNFS/flexfiles: restart a write that the DS error handler retries
  https://lore.kernel.org/linux-nfs/8ee107c9d0c68b6d0ee7305605ff7edbe0ffa160.1795108832.git.bcodding@hammerspace.com/

Patch 3 replaces the header bit test that the first of these fixes.

Changes since v2:
https://lore.kernel.org/linux-nfs/cover.1795117974.git.bcodding@hammerspace.com/
 - Patch 3: on a soft mount, fail the refused write once the next pass
   would exceed io_maxretrans, instead of waiting a second for a pass
   that fails anyway.  The first O_DIRECT pass queues its reschedule a
   second later too, instead of at once (sashiko).

Changes since v1:
https://lore.kernel.org/linux-nfs/cover.1795024981.git.bcodding@hammerspace.com/
 - Rebased onto the three fixes above.
 - Patch 1: shorter pg_allow_mds comment; the nfs_wb_folio() comment
   says the refused folio was redirtied (Anna).
 - New patch 2, so that patch 3 no longer sleeps on nfsiod.
 - Patch 3 (was 2): only data-integrity nfs_writepages() waits, and a
   fatal signal ends the wait; other writeback returns 0 instead of
   waiting a second per folio.  The O_DIRECT reschedule requeues after a
   second instead of sleeping.  The tracepoints that read the layout
   header without an lseg are emitted under i_lock.

Benjamin Coddington (5):
  pNFS/flexfiles: use the MDS for locked-folio writeback without a
    layout
  NFS: make the O_DIRECT write work a delayed_work
  pNFS/flexfiles: pace writes that have no layout and can't use the MDS
  pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit
  pNFS/flexfiles: use the MDS when the server has no layout to give

 fs/nfs/direct.c                        |  13 ++-
 fs/nfs/flexfilelayout/flexfilelayout.c | 138 ++++++++++++++++---------
 fs/nfs/flexfilelayout/flexfilelayout.h |  16 ---
 fs/nfs/internal.h                      |  14 +--
 fs/nfs/nfs4trace.c                     |   1 +
 fs/nfs/nfs4trace.h                     |   1 +
 fs/nfs/pagelist.c                      |   2 +
 fs/nfs/pnfs.c                          |  22 ++++
 fs/nfs/pnfs.h                          |   1 +
 fs/nfs/write.c                         |  31 +++++-
 include/linux/nfs_page.h               |   2 +
 11 files changed, 164 insertions(+), 77 deletions(-)


base-commit: aebb7274476403040d582c61478df8f4a7bf9cd3
prerequisite-patch-id: 71097d575f7ba74684462ab64b028ff8815dff63
prerequisite-patch-id: 5a73a1fb9d47a31d4ec337ec0264eadbcaa9fd4e
prerequisite-patch-id: 2663ab8e1949d6a11d957cc2784161bf2816cac2
-- 
2.53.0


             reply	other threads:[~2026-11-20 16:09 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 16:09 Benjamin Coddington [this message]
2026-11-20 16:09 ` [PATCH v3 1/5] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
2026-11-20 16:09 ` [PATCH v3 2/5] NFS: make the O_DIRECT write work a delayed_work Benjamin Coddington
2026-11-20 16:09 ` [PATCH v3 3/5] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
2026-11-20 16:09 ` [PATCH v3 4/5] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
2026-11-20 16:09 ` [PATCH v3 5/5] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1795122899.git.bcodding@hammerspace.com \
    --to=ben.coddington@hammerspace.com \
    --cc=anna@kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=snitzer@kernel.org \
    --cc=trondmy@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.