Linux NFS development
 help / color / mirror / Atom feed
* [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS
@ 2026-11-18 18:36 Benjamin Coddington
  2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
  To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs

From: Benjamin Coddington <bcodding@hammerspace.com>

When a file's flexfiles layouts carry FF_FLAGS_NO_IO_THRU_MDS, the client
keeps reads and writes off the MDS when it has no layout segment for
them.  Reads and writes handle that differently:

 - Reads test the inode's NFS_INO_NO_IO_THRU_MDS bit, wait a second
   between attempts, and honour io_maxretrans on soft mounts.

 - Writes test a bit on the layout header and return -EAGAIN.
   nfs_writepages() and the O_DIRECT reschedule path resend at once, so
   writers spin for as long as there is no layout.  nfs_wb_folio()
   returns the -EAGAIN, so a read(2) or write(2) that has to flush a
   folio another open context dirtied fails with EAGAIN.  Once the
   header is freed the bit is gone and writes go through the MDS, and
   ff_layout_pg_init_write() sends them there whenever
   pnfs_generic_pg_check_layout() drops an invalidated segment.

Neither direction uses the MDS when the server has said there is no
layout to give.  After NFS4ERR_LAYOUTUNAVAILABLE, RFC 8435 says the I/O
is to go to the MDS (section 5.1.1), and the flag is a hint (section
5.1.2).  Instead, reads wait and writes spin until the 120 second fail
bit expires, then send a LAYOUTGET that can fail the same way.

Patch 1 sends writeback of a single locked folio (nfs_wb_folio(),
reclaim, migration) through the MDS instead of refusing it.  Resending
with the folio locked could deadlock against a bulk recall, which can't
complete while nfs_writepages() holds a layout segment, and
nfs_writepages() would be waiting for that folio.  Patch 2 has the
remaining writes test the inode bit, and has nfs_writepages() and the
O_DIRECT reschedule path wait a second before resending a refused
write, after their folios and requests are unlocked.  Patch 3 removes
the header bit, which nothing tests after patch 2.  Patch 4 sends
reads, writes and the resend after a data server error through the MDS
while the fail bit for their iomode is set.

That last change is also the path for a data server the client can't
use, such as a version mismatch: the client reports the failure with
LAYOUTERROR or LAYOUTRETURN, and the server decides whether to answer
the next LAYOUTGET with NFS4ERR_LAYOUTUNAVAILABLE.  The missing-nfsv3
case is covered separately by Roberto's "NFSv4/flexfiles: couple back
with NFSv3".

These patches are independent of the series "pNFS: don't reply
NOMATCHING to a layout recall with I/O in flight", but both change
ff_layout_pg_get_mirror_count_write(), so whichever goes second will
need a small rebase.  The series is based on Anna's linux-next.

Tested against a reffs MDS and DS, injecting LAYOUTGET errors after a
file recall.  With NFS4ERR_LAYOUTUNAVAILABLE, reads and writes go through
the MDS at once.  With a retryable error (NFS4ERR_SERVERFAULT) held for
8 seconds, writeback sends one LAYOUTGET a second and no I/O to the MDS,
SIGKILL ends a writer blocked in fsync, and a soft mount with
io_maxretrans=3 returns EIO after about 4 seconds.  During the hold, a
read of a folio that another open context had dirtied flushes it
through the MDS and returns without EAGAIN.

Benjamin Coddington (4):
  pNFS/flexfiles: use the MDS for locked-folio writeback without a
    layout
  pNFS/flexfiles: pace writes that have no layout and can't use the MDS
  pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit
  pNFS/flexfiles: use the MDS when the server has no layout to give

 fs/nfs/direct.c                        |   2 +
 fs/nfs/flexfilelayout/flexfilelayout.c | 113 +++++++++++++++++--------
 fs/nfs/flexfilelayout/flexfilelayout.h |  16 ----
 fs/nfs/internal.h                      |  11 +--
 fs/nfs/nfs4trace.c                     |   1 +
 fs/nfs/nfs4trace.h                     |   1 +
 fs/nfs/pagelist.c                      |   2 +
 fs/nfs/pnfs.c                          |  22 +++++
 fs/nfs/pnfs.h                          |   1 +
 fs/nfs/write.c                         |  21 ++++-
 include/linux/nfs_page.h               |  10 +++
 11 files changed, 139 insertions(+), 61 deletions(-)

-- 
2.53.0


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout
  2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
  2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
  To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs

From: Benjamin Coddington <bcodding@hammerspace.com>

nfs_writepage_locked() writes back one folio that its caller holds
locked, for nfs_wb_folio(), nfs_wb_folio_reclaim() and migration.  When
there is no layout segment and the file's layout header carries
FF_FLAGS_NO_IO_THRU_MDS, ff_layout_pg_get_mirror_count_write() refuses
the write with -EAGAIN.  nfs_do_writepage() redirties the folio and
returns the -EAGAIN, and nfs_wb_folio() passes it to nfs_updatepage(),
nfs_flush_incompatible(), nfs_read_folio() and nfs_launder_folio().  A
read(2) or write(2) on a blocking file then fails with EAGAIN, for
example when another open context has dirtied the same folio while a
layout is being recalled.

Resending from nfs_wb_folio(), as nfs_writepages() does, would hold the
folio locked across the resends.  A bulk recall can't complete while
nfs_writepages() holds a layout segment, and nfs_writepages() would be
waiting for that folio's lock.

FF_FLAGS_NO_IO_THRU_MDS is a hint (RFC 8435, section 5.1.2).  Mark the
descriptor that nfs_writepage_locked() uses with a new pg_allow_mds
bit, and send its write through the MDS when there is no layout
segment.  When ff_layout_pg_init_write() can't use a data server, send
such a write to the MDS too, instead of retrying with the folio locked.

ff_layout_pg_init_write() switches a write to the MDS by resetting the
descriptor and returning -EAGAIN, which nfs_wb_folio() also passes up,
with or without the flag.  When nfs_wb_folio() gets -EAGAIN, wait for
any part of the folio already under writeback, as its loop does, and
write the folio once more with a descriptor that goes straight to the
MDS.  nfs_wb_folio_reclaim() still returns the error, and
nfs_release_folio() keeps the folio.

Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in pg_get_mirror_count_write")
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/flexfilelayout/flexfilelayout.c |  5 +++--
 fs/nfs/pagelist.c                      |  1 +
 fs/nfs/write.c                         | 19 +++++++++++++++----
 include/linux/nfs_page.h               |  5 +++++
 4 files changed, 24 insertions(+), 6 deletions(-)

diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 5c289737366f..2b0a0af469fb 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -1176,7 +1176,8 @@ ff_layout_pg_init_write(struct nfs_pageio_descriptor *pgio,
 					       mirror_ds, dss_id, OP_WRITE);
 		if (IS_ERR(ds)) {
 			nfs4_ff_layout_put_deviceid(mirror_ds);
-			if (!ff_layout_no_fallback_to_mds(pgio->pg_lseg))
+			if (pgio->pg_allow_mds ||
+			    !ff_layout_no_fallback_to_mds(pgio->pg_lseg))
 				goto out_mds;
 			pnfs_generic_pg_cleanup(pgio);
 			/* Sleep for 1 second before retrying */
@@ -1229,7 +1230,7 @@ ff_layout_pg_get_mirror_count_write(struct nfs_pageio_descriptor *pgio,
 			0, NFS4_MAX_UINT64, IOMODE_RW,
 			NFS_I(pgio->pg_inode)->layout,
 			pgio->pg_lseg);
-	if (NFS_I(pgio->pg_inode)->layout &&
+	if (!pgio->pg_allow_mds && NFS_I(pgio->pg_inode)->layout &&
 	    ff_layout_hdr_no_fallback_to_mds(NFS_I(pgio->pg_inode)->layout)) {
 		/*
 		 * FF_FLAGS_NO_IO_THRU_MDS: no current lseg but the server's
diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
index 71f0ce2bc4ea..aa6d7b658d79 100644
--- a/fs/nfs/pagelist.c
+++ b/fs/nfs/pagelist.c
@@ -871,6 +871,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc,
 		     int io_flags)
 {
 	desc->pg_moreio = 0;
+	desc->pg_allow_mds = 0;
 	desc->pg_inode = inode;
 	desc->pg_ops = pg_ops;
 	desc->pg_completion_ops = compl_ops;
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 0b42d31f7a75..ec6b771b8dbe 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -627,15 +627,16 @@ static int nfs_do_writepage(struct folio *folio, struct writeback_control *wbc,
  * Write an mmapped page to the server.
  */
 static int nfs_writepage_locked(struct folio *folio,
-				struct writeback_control *wbc)
+				struct writeback_control *wbc, bool force_mds)
 {
 	struct nfs_pageio_descriptor pgio;
 	struct inode *inode = folio->mapping->host;
 	int err;
 
 	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
-	nfs_pageio_init_write(&pgio, inode, 0, false,
+	nfs_pageio_init_write(&pgio, inode, 0, force_mds,
 			      &nfs_async_write_completion_ops);
+	pgio.pg_allow_mds = 1;
 	err = nfs_do_writepage(folio, wbc, &pgio);
 	pgio.pg_error = 0;
 	nfs_pageio_complete(&pgio);
@@ -2057,7 +2058,7 @@ int nfs_wb_folio_reclaim(struct inode *inode, struct folio *folio)
 		return -EBUSY;
 	if (folio_clear_dirty_for_io(folio)) {
 		trace_nfs_writeback_folio_reclaim(inode, range_start, len);
-		ret = nfs_writepage_locked(folio, &wbc);
+		ret = nfs_writepage_locked(folio, &wbc, false);
 		trace_nfs_writeback_folio_reclaim_done(inode, range_start, len,
 						       ret);
 		return ret;
@@ -2084,6 +2085,7 @@ int nfs_wb_folio(struct inode *inode, struct folio *folio)
 		.range_start = range_start,
 		.range_end = range_start + len - 1,
 	};
+	bool force_mds = false;
 	int ret;
 
 	trace_nfs_writeback_folio(inode, range_start, len);
@@ -2091,7 +2093,16 @@ int nfs_wb_folio(struct inode *inode, struct folio *folio)
 	for (;;) {
 		folio_wait_writeback(folio);
 		if (folio_clear_dirty_for_io(folio)) {
-			ret = nfs_writepage_locked(folio, &wbc);
+			ret = nfs_writepage_locked(folio, &wbc, force_mds);
+			/*
+			 * -EAGAIN asks for a resend, which can't wait for a
+			 * layout with the folio locked: resend it through the
+			 * MDS once any part of it already sent has completed.
+			 */
+			if (ret == -EAGAIN && !force_mds) {
+				force_mds = true;
+				continue;
+			}
 			if (ret < 0)
 				goto out_error;
 			continue;
diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
index c38e4b380be5..d64332afc374 100644
--- a/include/linux/nfs_page.h
+++ b/include/linux/nfs_page.h
@@ -120,6 +120,11 @@ struct nfs_pageio_descriptor {
 	u32			pg_mirror_idx;	/* current mirror */
 	unsigned short		pg_maxretrans;
 	unsigned char		pg_moreio : 1;
+	/*
+	 * Set when the caller holds the folio locked and can't wait for a
+	 * layout: the write goes through the MDS instead of waiting for one.
+	 */
+	unsigned char		pg_allow_mds : 1;
 };
 
 /* arbitrarily selected limit to number of mirrors */
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS
  2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
  2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
  2026-11-18 18:36 ` [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
  2026-11-18 18:36 ` [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington
  3 siblings, 0 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
  To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs

From: Benjamin Coddington <bcodding@hammerspace.com>

When a write has no layout segment and its file's layouts carry
FF_FLAGS_NO_IO_THRU_MDS, ff_layout_pg_get_mirror_count_write() returns
-EAGAIN.  nfs_writepages() and the O_DIRECT reschedule path resend it at
once, so writers spin for as long as pnfs_update_layout() has no layout
to give: while NFS_LAYOUT_BULK_RECALL is set, while LAYOUTGETs are
blocked, and for the 120 seconds after NFS4ERR_LAYOUTUNAVAILABLE.

The check also tests a bit on the layout header.  Once that header has
been freed, a new one starts without the bit and the write goes through
the MDS.  ff_layout_pg_init_write() has no check at all: when
pnfs_generic_pg_check_layout() drops an invalidated segment, the rest of
the descriptor's writes go through the MDS.

Handle both places the way ff_layout_pg_init_read() handles reads: test
the inode's NFS_INO_NO_IO_THRU_MDS bit, return -ERESTARTSYS to a killed
or exiting task, and honour io_maxretrans on soft mounts.  Mark a
refusal with a new pg_nolayout descriptor bit, and have nfs_writepages()
and nfs_direct_write_reschedule() wait a second before they resend,
after the pass that was refused has unlocked its folios and requests.
Writes marked pg_allow_mds by nfs_writepage_locked() still go through
the MDS.

The test is ff_layout_inode_no_fallback_to_mds(), which
ff_layout_pg_init_read() now uses too, and the refusal is
ff_layout_pg_refuse_write().  Emit the MDS-fallback tracepoint only
when the write does go through the MDS, and a new
pnfs_mds_fallback_write_refused tracepoint when it is refused.

Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in pg_get_mirror_count_write")
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/direct.c                        |  2 +
 fs/nfs/flexfilelayout/flexfilelayout.c | 66 ++++++++++++++++++++------
 fs/nfs/nfs4trace.c                     |  1 +
 fs/nfs/nfs4trace.h                     |  1 +
 fs/nfs/pagelist.c                      |  1 +
 fs/nfs/write.c                         |  2 +
 include/linux/nfs_page.h               |  5 ++
 7 files changed, 64 insertions(+), 14 deletions(-)

diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
index a3c6e8f4ea06..e6a2c6436b44 100644
--- a/fs/nfs/direct.c
+++ b/fs/nfs/direct.c
@@ -626,6 +626,8 @@ static void nfs_direct_write_reschedule(struct nfs_direct_req *dreq)
 		}
 	}
 
+	if (desc.pg_error == -EAGAIN && desc.pg_nolayout)
+		ssleep(1);
 	if (put_dreq(dreq))
 		nfs_direct_write_complete(dreq);
 }
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 2b0a0af469fb..4eb12bbb7838 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -982,6 +982,45 @@ static bool ff_layout_pg_retry_wait(struct nfs_pageio_descriptor *pgio)
 	return true;
 }
 
+/*
+ * Returns true if the inode's layouts forbid I/O through the MDS.
+ */
+static bool ff_layout_inode_no_fallback_to_mds(struct inode *inode, u32 iomode)
+{
+	return nfs_no_io_thru_mds(inode);
+}
+
+/*
+ * Refuse a write that has no layout segment and can't go through the MDS:
+ * -EAGAIN so that the caller resends it, or a hard error when the caller
+ * should give up.  pg_nolayout tells a caller that resends at once to
+ * wait first, once it has dropped its locks.
+ */
+static void ff_layout_pg_refuse_write(struct nfs_pageio_descriptor *pgio,
+				      struct nfs_page *req)
+{
+	struct nfs_server *server = NFS_SERVER(pgio->pg_inode);
+
+	trace_pnfs_mds_fallback_write_refused(pgio->pg_inode, req_offset(req),
+					      req->wb_bytes, IOMODE_RW,
+					      NFS_I(pgio->pg_inode)->layout,
+					      NULL);
+	if (server->flags & (NFS_MOUNT_SOFT | NFS_MOUNT_SOFTERR) &&
+	    io_maxretrans && req->wb_nio > io_maxretrans) {
+		if (server->flags & NFS_MOUNT_SOFTERR)
+			pgio->pg_error = -ETIMEDOUT;
+		else
+			pgio->pg_error = -EIO;
+		return;
+	}
+	if (fatal_signal_pending(current) || nfs_current_task_exiting()) {
+		pgio->pg_error = -ERESTARTSYS;
+		return;
+	}
+	pgio->pg_nolayout = 1;
+	pgio->pg_error = -EAGAIN;
+}
+
 static void
 ff_layout_pg_get_read(struct nfs_pageio_descriptor *pgio,
 		      struct nfs_page *req,
@@ -1108,7 +1147,7 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,
 	 * FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
 	 * this asks the inode and not the hdr.
 	 */
-	if (!nfs_no_io_thru_mds(pgio->pg_inode))
+	if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
 		goto out_mds;
 retry_nolseg:
 	/* Retry getting layout segment if lower layer returned -EAGAIN */
@@ -1156,9 +1195,13 @@ ff_layout_pg_init_write(struct nfs_pageio_descriptor *pgio,
 			return;
 		}
 	}
-	/* If no lseg, fall back to write through mds */
-	if (pgio->pg_lseg == NULL)
-		goto out_mds;
+	if (!pgio->pg_lseg) {
+		if (pgio->pg_allow_mds ||
+		    !ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_RW))
+			goto out_mds;
+		ff_layout_pg_refuse_write(pgio, req);
+		return;
+	}
 
 	/* Use a direct mapping of ds_idx to pgio mirror_idx */
 	if (pgio->pg_mirror_count != FF_LAYOUT_MIRROR_COUNT(pgio->pg_lseg))
@@ -1225,21 +1268,16 @@ ff_layout_pg_get_mirror_count_write(struct nfs_pageio_descriptor *pgio,
 	}
 	if (pgio->pg_lseg)
 		return FF_LAYOUT_MIRROR_COUNT(pgio->pg_lseg);
+	if (!pgio->pg_allow_mds &&
+	    ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_RW)) {
+		ff_layout_pg_refuse_write(pgio, req);
+		goto out;
+	}
 
 	trace_pnfs_mds_fallback_pg_get_mirror_count(pgio->pg_inode,
 			0, NFS4_MAX_UINT64, IOMODE_RW,
 			NFS_I(pgio->pg_inode)->layout,
 			pgio->pg_lseg);
-	if (!pgio->pg_allow_mds && NFS_I(pgio->pg_inode)->layout &&
-	    ff_layout_hdr_no_fallback_to_mds(NFS_I(pgio->pg_inode)->layout)) {
-		/*
-		 * FF_FLAGS_NO_IO_THRU_MDS: no current lseg but the server's
-		 * policy forbids MDS fallback.  Surface -EAGAIN so writeback
-		 * retries rather than silently issuing the WRITE via MDS.
-		 */
-		pgio->pg_error = -EAGAIN;
-		goto out;
-	}
 	/* no lseg means that pnfs is not in use, so no mirroring here */
 	nfs_pageio_reset_write_mds(pgio);
 out:
diff --git a/fs/nfs/nfs4trace.c b/fs/nfs/nfs4trace.c
index 3fdc013f56d8..93a84f546de8 100644
--- a/fs/nfs/nfs4trace.c
+++ b/fs/nfs/nfs4trace.c
@@ -25,6 +25,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_read_done);
 EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_done);
 EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_read_pagelist);
 EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_pagelist);
+EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_refused);
 EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_ds_connect);
 
 EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_read_error);
diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h
index b5c89eeef2bc..94ed6b833a28 100644
--- a/fs/nfs/nfs4trace.h
+++ b/fs/nfs/nfs4trace.h
@@ -2115,6 +2115,7 @@ DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_read_done);
 DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_done);
 DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_read_pagelist);
 DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_pagelist);
+DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_refused);
 
 DECLARE_EVENT_CLASS(nfs4_deviceid_event,
 		TP_PROTO(
diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
index aa6d7b658d79..9a8731de6f5b 100644
--- a/fs/nfs/pagelist.c
+++ b/fs/nfs/pagelist.c
@@ -872,6 +872,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc,
 {
 	desc->pg_moreio = 0;
 	desc->pg_allow_mds = 0;
+	desc->pg_nolayout = 0;
 	desc->pg_inode = inode;
 	desc->pg_ops = pg_ops;
 	desc->pg_completion_ops = compl_ops;
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index ec6b771b8dbe..005d4b795b80 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -693,6 +693,8 @@ int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc)
 		nfs_pageio_complete(&pgio);
 		if (err == -EAGAIN && mntflags & NFS_MOUNT_SOFTERR)
 			break;
+		if (err == -EAGAIN && pgio.pg_nolayout)
+			ssleep(1);
 	} while (err < 0 && !nfs_error_is_fatal(err));
 	nfs_io_completion_put(ioc);
 
diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
index d64332afc374..f9b22b990de7 100644
--- a/include/linux/nfs_page.h
+++ b/include/linux/nfs_page.h
@@ -125,6 +125,11 @@ struct nfs_pageio_descriptor {
 	 * layout: the write goes through the MDS instead of waiting for one.
 	 */
 	unsigned char		pg_allow_mds : 1;
+	/*
+	 * Set by a layout driver that refused a write with -EAGAIN for lack
+	 * of a layout, so that a caller that resends waits first.
+	 */
+	unsigned char		pg_nolayout : 1;
 };
 
 /* arbitrarily selected limit to number of mirrors */
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit
  2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
  2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
  2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
  2026-11-18 18:36 ` [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington
  3 siblings, 0 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
  To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs

From: Benjamin Coddington <bcodding@hammerspace.com>

Nothing tests NFS4_FF_HDR_NO_IO_THRU_MDS since writes moved to the
inode's NFS_INO_NO_IO_THRU_MDS bit.  Remove it, its helper, and
nfs4_flexfile_layout::flags, which held only that bit.

Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/flexfilelayout/flexfilelayout.c |  8 ++------
 fs/nfs/flexfilelayout/flexfilelayout.h | 16 ----------------
 fs/nfs/internal.h                      |  9 ++++-----
 3 files changed, 6 insertions(+), 27 deletions(-)

diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 4eb12bbb7838..a522c3fe49e5 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -641,12 +641,8 @@ ff_layout_alloc_lseg(struct pnfs_layout_hdr *lh,
 	if (!p)
 		goto out_sort_mirrors;
 	fls->flags = be32_to_cpup(p);
-	if (fls->flags & FF_FLAGS_NO_IO_THRU_MDS) {
-		set_bit(NFS4_FF_HDR_NO_IO_THRU_MDS,
-			&FF_LAYOUT_FROM_HDR(lh)->flags);
-		/* Outlives this layout hdr; see NFS_INO_NO_IO_THRU_MDS */
+	if (fls->flags & FF_FLAGS_NO_IO_THRU_MDS)
 		nfs_set_no_io_thru_mds(lh->plh_inode);
-	}
 
 	p = xdr_inline_decode(&stream, 4);
 	if (!p)
@@ -1145,7 +1141,7 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,
 	 * MDS there is no fallback to take, so wait for a layout on the
 	 * same terms as the -EAGAIN above.  The layout hdr that carried
 	 * FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
-	 * this asks the inode and not the hdr.
+	 * the flag is kept on the inode.
 	 */
 	if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
 		goto out_mds;
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.h b/fs/nfs/flexfilelayout/flexfilelayout.h
index 5d5587fded76..bb60537b2b93 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.h
+++ b/fs/nfs/flexfilelayout/flexfilelayout.h
@@ -113,16 +113,12 @@ struct nfs4_ff_layout_segment {
 	struct nfs4_ff_layout_mirror	*mirror_array[] __counted_by(mirror_array_cnt);
 };
 
-/* nfs4_flexfile_layout::flags bit indices */
-#define NFS4_FF_HDR_NO_IO_THRU_MDS  0   /* any lseg has had FF_FLAGS_NO_IO_THRU_MDS */
-
 struct nfs4_flexfile_layout {
 	struct pnfs_layout_hdr generic_hdr;
 	struct pnfs_ds_commit_info commit_info;
 	struct list_head	mirrors;
 	struct list_head	error_list; /* nfs4_ff_layout_ds_err */
 	ktime_t			last_report_time; /* Layoutstat report times */
-	unsigned long		flags;
 };
 
 /*
@@ -186,18 +182,6 @@ ff_layout_no_fallback_to_mds(struct pnfs_layout_segment *lseg)
 	return FF_LAYOUT_LSEG(lseg)->flags & FF_FLAGS_NO_IO_THRU_MDS;
 }
 
-/*
- * Sticky hdr-level mirror of FF_FLAGS_NO_IO_THRU_MDS so callers that have
- * no current lseg (e.g. between LAYOUTRETURN and the next LAYOUTGET) can
- * still honor the no-MDS-fallback policy.
- */
-static inline bool
-ff_layout_hdr_no_fallback_to_mds(struct pnfs_layout_hdr *lo)
-{
-	return test_bit(NFS4_FF_HDR_NO_IO_THRU_MDS,
-			&FF_LAYOUT_FROM_HDR(lo)->flags);
-}
-
 static inline bool
 ff_layout_no_read_on_rw(struct pnfs_layout_segment *lseg)
 {
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 5af6a032f145..5b40364fc38f 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -556,11 +556,10 @@ static inline bool nfs_file_io_is_buffered(struct nfs_inode *nfsi)
  * FF_FLAGS_NO_IO_THRU_MDS in ff_layout_alloc_lseg()).  The RFC 5661
  * mdsthreshold hint must not be acted on for such a file, since the only
  * thing pnfs_within_mdsthreshold() can ask for is the one thing the
- * layout forbids.  Servers are assumed to be consistent in their
- * no-fallback policy per file, the same assumption
- * ff_layout_hdr_no_fallback_to_mds() already makes; if one were not, the
- * only effect is that its mdsthreshold hint - a SHOULD - stops being
- * honored for an inode that is already in core.
+ * layout forbids.  Flexfiles also retries reads and writes that have no
+ * layout segment instead of sending them through the MDS, except
+ * writeback of a single locked folio (pg_allow_mds).  Servers are
+ * assumed to be consistent in their no-fallback policy per file.
  */
 #define NFS_INO_NO_IO_THRU_MDS	(30)
 
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give
  2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
                   ` (2 preceding siblings ...)
  2026-11-18 18:36 ` [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
  3 siblings, 0 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
  To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs

From: Benjamin Coddington <bcodding@hammerspace.com>

When LAYOUTGET fails with NFS4ERR_LAYOUTUNAVAILABLE, or with
NFS4ERR_TOOSMALL after the retry at the session's maximum reply size,
pnfs_update_layout() sets the layout's fail bit for that iomode and
returns no layout without sending another LAYOUTGET for the next 120
seconds.  If the file's layouts carried FF_FLAGS_NO_IO_THRU_MDS, reads
and writes wait for a layout the whole time, and then send a LAYOUTGET
that can fail the same way.  On a hard mount that repeats for as long
as the server keeps answering that way.

RFC 8435 says that after NFS4ERR_LAYOUTUNAVAILABLE "the I/O is to go to
the metadata server" (section 5.1.1), and that FF_FLAGS_NO_IO_THRU_MDS
is a hint (section 5.1.2).  Send the I/O through the MDS while the fail
bit is set, in ff_layout_inode_no_fallback_to_mds(), which reads,
writes and the resend after a data server error share.  This also lets
a server deal with a data server the client can't use: the client
reports the failure with LAYOUTERROR (or in LAYOUTRETURN) and returns
the layout, and the server can answer the next LAYOUTGET with
NFS4ERR_LAYOUTUNAVAILABLE.

Fixes: 2f007ba074ad ("pNFS/flexfiles: don't read through MDS when previous layout forbid it")
Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in pg_get_mirror_count_write")
Fixes: ec906cf7bee6 ("pNFS/flexfiles: don't reset to MDS for v4 error when previous layout forbid it")
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/flexfilelayout/flexfilelayout.c | 40 +++++++++++++++-----------
 fs/nfs/internal.h                      |  6 ++--
 fs/nfs/pnfs.c                          | 22 ++++++++++++++
 fs/nfs/pnfs.h                          |  1 +
 4 files changed, 50 insertions(+), 19 deletions(-)

diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index a522c3fe49e5..c4fae5408535 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -980,10 +980,14 @@ static bool ff_layout_pg_retry_wait(struct nfs_pageio_descriptor *pgio)
 
 /*
  * Returns true if the inode's layouts forbid I/O through the MDS.
+ * FF_FLAGS_NO_IO_THRU_MDS is a hint (RFC 8435, section 5.1.2), so set it
+ * aside while a failed LAYOUTGET for @iomode (NFS4ERR_LAYOUTUNAVAILABLE or
+ * NFS4ERR_TOOSMALL) keeps pnfs_update_layout() from sending another.
  */
 static bool ff_layout_inode_no_fallback_to_mds(struct inode *inode, u32 iomode)
 {
-	return nfs_no_io_thru_mds(inode);
+	return nfs_no_io_thru_mds(inode) &&
+	       !pnfs_layout_get_failed(inode, iomode);
 }
 
 /*
@@ -1135,13 +1139,12 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,
 	}
 	/*
 	 * No segment, and no error to report either: pnfs_update_layout()
-	 * simply has nothing to give (NFS_LAYOUT_BULK_RECALL, a failed
-	 * pnfs_layout_io_test, blocked LAYOUTGETs, a layout being
-	 * returned).  If the server forbids reading this file through the
-	 * MDS there is no fallback to take, so wait for a layout on the
-	 * same terms as the -EAGAIN above.  The layout hdr that carried
-	 * FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
-	 * the flag is kept on the inode.
+	 * simply has nothing to give (NFS_LAYOUT_BULK_RECALL, blocked
+	 * LAYOUTGETs, a layout being returned).  If the server forbids
+	 * reading this file through the MDS there is no fallback to take,
+	 * so wait for a layout on the same terms as the -EAGAIN above.  The
+	 * layout hdr that carried FF_FLAGS_NO_IO_THRU_MDS may itself be
+	 * gone by now, which is why the flag is kept on the inode.
 	 */
 	if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
 		goto out_mds;
@@ -1395,7 +1398,8 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
 					   struct nfs4_state *state,
 					   struct nfs_client *clp,
 					   struct pnfs_layout_segment *lseg,
-					   struct nfs4_deviceid_node *devid)
+					   struct nfs4_deviceid_node *devid,
+					   u32 iomode)
 {
 	struct pnfs_layout_hdr *lo = lseg->pls_layout;
 	struct inode *inode = lo->plh_inode;
@@ -1484,8 +1488,9 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
 		return -NFS4ERR_RESET_TO_PNFS;
 reset:
 	/*
-	 * FF_FLAGS_NO_IO_THRU_MDS: never resend through the MDS.  The
-	 * caller would do so with force_mds set (ff_layout_reset_read(),
+	 * FF_FLAGS_NO_IO_THRU_MDS: don't resend through the MDS (see
+	 * ff_layout_inode_no_fallback_to_mds()).  The caller would do so
+	 * with force_mds set (ff_layout_reset_read(),
 	 * ff_layout_reset_write(hdr, false) -> pnfs_*_done_resend_to_mds()),
 	 * which builds a descriptor out of the plain MDS page ops and so
 	 * consults no layout at all -- this is the last point at which the
@@ -1495,7 +1500,7 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
 	 * pnfs_destroy_layout().  Having done so they can no longer ask the
 	 * layout hdr about the flag, hence the inode.
 	 */
-	if (nfs_no_io_thru_mds(inode)) {
+	if (ff_layout_inode_no_fallback_to_mds(inode, iomode)) {
 		dprintk("%s Retry through pNFS, no MDS fallback. Error %d\n",
 			__func__, task->tk_status);
 		return -NFS4ERR_RESET_TO_PNFS;
@@ -1580,7 +1585,8 @@ static int ff_layout_async_handle_error(struct rpc_task *task,
 					struct nfs4_state *state,
 					struct nfs_client *clp,
 					struct pnfs_layout_segment *lseg,
-					struct nfs4_deviceid_node *devid)
+					struct nfs4_deviceid_node *devid,
+					u32 iomode)
 {
 	int vers = clp->cl_nfs_mod->rpc_vers->number;
 
@@ -1600,7 +1606,7 @@ static int ff_layout_async_handle_error(struct rpc_task *task,
 						       lseg, devid);
 	case 4:
 		return ff_layout_async_handle_error_v4(task, op_status, state,
-						       clp, lseg, devid);
+						       clp, lseg, devid, iomode);
 	default:
 		/* should never happen */
 		WARN_ON_ONCE(1);
@@ -1734,7 +1740,7 @@ static int ff_layout_read_done_cb(struct rpc_task *task,
 	err = ff_layout_async_handle_error(task, hdr->res.op_status,
 					   hdr->args.context->state,
 					   hdr->ds_clp, hdr->lseg,
-					   hdr->ds_dev);
+					   hdr->ds_dev, IOMODE_READ);
 
 	trace_nfs4_pnfs_read(hdr, err);
 	clear_bit(NFS_IOHDR_RESEND_PNFS, &hdr->flags);
@@ -1944,7 +1950,7 @@ static int ff_layout_write_done_cb(struct rpc_task *task,
 	err = ff_layout_async_handle_error(task, hdr->res.op_status,
 					   hdr->args.context->state,
 					   hdr->ds_clp, hdr->lseg,
-					   hdr->ds_dev);
+					   hdr->ds_dev, IOMODE_RW);
 
 	trace_nfs4_pnfs_write(hdr, err);
 	clear_bit(NFS_IOHDR_RESEND_PNFS, &hdr->flags);
@@ -1995,7 +2001,7 @@ static int ff_layout_commit_done_cb(struct rpc_task *task,
 
 	err = ff_layout_async_handle_error(task, data->res.op_status,
 					   NULL, data->ds_clp, data->lseg,
-					   data->ds_dev);
+					   data->ds_dev, IOMODE_RW);
 
 	trace_nfs4_pnfs_commit_ds(data, err);
 	switch (err) {
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 5b40364fc38f..fc2a27b5126a 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -558,8 +558,10 @@ static inline bool nfs_file_io_is_buffered(struct nfs_inode *nfsi)
  * thing pnfs_within_mdsthreshold() can ask for is the one thing the
  * layout forbids.  Flexfiles also retries reads and writes that have no
  * layout segment instead of sending them through the MDS, except
- * writeback of a single locked folio (pg_allow_mds).  Servers are
- * assumed to be consistent in their no-fallback policy per file.
+ * writeback of a single locked folio (pg_allow_mds), and unless a
+ * LAYOUTGET for that iomode has failed with NFS4ERR_LAYOUTUNAVAILABLE or
+ * NFS4ERR_TOOSMALL.  Servers are assumed to be consistent in their
+ * no-fallback policy per file.
  */
 #define NFS_INO_NO_IO_THRU_MDS	(30)
 
diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 14f4a4b12ffd..582fe81fc2d1 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -551,6 +551,28 @@ pnfs_layout_io_test_failed(struct pnfs_layout_hdr *lo, u32 iomode)
 	return true;
 }
 
+/*
+ * Returns true if a LAYOUTGET for @iomode failed with an error, such as
+ * NFS4ERR_LAYOUTUNAVAILABLE, in the last PNFS_LAYOUTGET_RETRY_TIMEOUT.
+ * pnfs_layout_io_test_failed() clears an expired fail bit only when
+ * pnfs_update_layout() gets that far, so check the time here too.
+ */
+bool pnfs_layout_get_failed(struct inode *inode, u32 iomode)
+{
+	struct pnfs_layout_hdr *lo;
+	unsigned long now = jiffies;
+	bool ret = false;
+
+	spin_lock(&inode->i_lock);
+	lo = NFS_I(inode)->layout;
+	if (lo && test_bit(pnfs_iomode_to_fail_bit(iomode), &lo->plh_flags))
+		ret = time_in_range(lo->plh_retry_timestamp,
+				    now - PNFS_LAYOUTGET_RETRY_TIMEOUT, now);
+	spin_unlock(&inode->i_lock);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(pnfs_layout_get_failed);
+
 static void
 pnfs_init_lseg(struct pnfs_layout_hdr *lo, struct pnfs_layout_segment *lseg,
 		const struct pnfs_layout_range *range,
diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
index 2774d4adf4c3..0007a4ca3da9 100644
--- a/fs/nfs/pnfs.h
+++ b/fs/nfs/pnfs.h
@@ -296,6 +296,7 @@ int pnfs_generic_pg_writepages(struct nfs_pageio_descriptor *desc);
 size_t pnfs_generic_pg_test(struct nfs_pageio_descriptor *pgio,
 			    struct nfs_page *prev, struct nfs_page *req);
 void pnfs_set_lo_fail(struct pnfs_layout_segment *lseg);
+bool pnfs_layout_get_failed(struct inode *inode, u32 iomode);
 struct pnfs_layout_segment *pnfs_layout_process(struct nfs4_layoutget *lgp);
 void pnfs_layoutget_free(struct nfs4_layoutget *lgp);
 void pnfs_free_lseg_list(struct list_head *tmp_list);
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-18 18:37 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
2026-11-18 18:36 ` [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
2026-11-18 18:36 ` [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox