* [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS
@ 2026-11-18 18:36 Benjamin Coddington
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
When a file's flexfiles layouts carry FF_FLAGS_NO_IO_THRU_MDS, the client
keeps reads and writes off the MDS when it has no layout segment for
them. Reads and writes handle that differently:
- Reads test the inode's NFS_INO_NO_IO_THRU_MDS bit, wait a second
between attempts, and honour io_maxretrans on soft mounts.
- Writes test a bit on the layout header and return -EAGAIN.
nfs_writepages() and the O_DIRECT reschedule path resend at once, so
writers spin for as long as there is no layout. nfs_wb_folio()
returns the -EAGAIN, so a read(2) or write(2) that has to flush a
folio another open context dirtied fails with EAGAIN. Once the
header is freed the bit is gone and writes go through the MDS, and
ff_layout_pg_init_write() sends them there whenever
pnfs_generic_pg_check_layout() drops an invalidated segment.
Neither direction uses the MDS when the server has said there is no
layout to give. After NFS4ERR_LAYOUTUNAVAILABLE, RFC 8435 says the I/O
is to go to the MDS (section 5.1.1), and the flag is a hint (section
5.1.2). Instead, reads wait and writes spin until the 120 second fail
bit expires, then send a LAYOUTGET that can fail the same way.
Patch 1 sends writeback of a single locked folio (nfs_wb_folio(),
reclaim, migration) through the MDS instead of refusing it. Resending
with the folio locked could deadlock against a bulk recall, which can't
complete while nfs_writepages() holds a layout segment, and
nfs_writepages() would be waiting for that folio. Patch 2 has the
remaining writes test the inode bit, and has nfs_writepages() and the
O_DIRECT reschedule path wait a second before resending a refused
write, after their folios and requests are unlocked. Patch 3 removes
the header bit, which nothing tests after patch 2. Patch 4 sends
reads, writes and the resend after a data server error through the MDS
while the fail bit for their iomode is set.
That last change is also the path for a data server the client can't
use, such as a version mismatch: the client reports the failure with
LAYOUTERROR or LAYOUTRETURN, and the server decides whether to answer
the next LAYOUTGET with NFS4ERR_LAYOUTUNAVAILABLE. The missing-nfsv3
case is covered separately by Roberto's "NFSv4/flexfiles: couple back
with NFSv3".
These patches are independent of the series "pNFS: don't reply
NOMATCHING to a layout recall with I/O in flight", but both change
ff_layout_pg_get_mirror_count_write(), so whichever goes second will
need a small rebase. The series is based on Anna's linux-next.
Tested against a reffs MDS and DS, injecting LAYOUTGET errors after a
file recall. With NFS4ERR_LAYOUTUNAVAILABLE, reads and writes go through
the MDS at once. With a retryable error (NFS4ERR_SERVERFAULT) held for
8 seconds, writeback sends one LAYOUTGET a second and no I/O to the MDS,
SIGKILL ends a writer blocked in fsync, and a soft mount with
io_maxretrans=3 returns EIO after about 4 seconds. During the hold, a
read of a folio that another open context had dirtied flushes it
through the MDS and returns without EAGAIN.
Benjamin Coddington (4):
pNFS/flexfiles: use the MDS for locked-folio writeback without a
layout
pNFS/flexfiles: pace writes that have no layout and can't use the MDS
pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit
pNFS/flexfiles: use the MDS when the server has no layout to give
fs/nfs/direct.c | 2 +
fs/nfs/flexfilelayout/flexfilelayout.c | 113 +++++++++++++++++--------
fs/nfs/flexfilelayout/flexfilelayout.h | 16 ----
fs/nfs/internal.h | 11 +--
fs/nfs/nfs4trace.c | 1 +
fs/nfs/nfs4trace.h | 1 +
fs/nfs/pagelist.c | 2 +
fs/nfs/pnfs.c | 22 +++++
fs/nfs/pnfs.h | 1 +
fs/nfs/write.c | 21 ++++-
include/linux/nfs_page.h | 10 +++
11 files changed, 139 insertions(+), 61 deletions(-)
--
2.53.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
nfs_writepage_locked() writes back one folio that its caller holds
locked, for nfs_wb_folio(), nfs_wb_folio_reclaim() and migration. When
there is no layout segment and the file's layout header carries
FF_FLAGS_NO_IO_THRU_MDS, ff_layout_pg_get_mirror_count_write() refuses
the write with -EAGAIN. nfs_do_writepage() redirties the folio and
returns the -EAGAIN, and nfs_wb_folio() passes it to nfs_updatepage(),
nfs_flush_incompatible(), nfs_read_folio() and nfs_launder_folio(). A
read(2) or write(2) on a blocking file then fails with EAGAIN, for
example when another open context has dirtied the same folio while a
layout is being recalled.
Resending from nfs_wb_folio(), as nfs_writepages() does, would hold the
folio locked across the resends. A bulk recall can't complete while
nfs_writepages() holds a layout segment, and nfs_writepages() would be
waiting for that folio's lock.
FF_FLAGS_NO_IO_THRU_MDS is a hint (RFC 8435, section 5.1.2). Mark the
descriptor that nfs_writepage_locked() uses with a new pg_allow_mds
bit, and send its write through the MDS when there is no layout
segment. When ff_layout_pg_init_write() can't use a data server, send
such a write to the MDS too, instead of retrying with the folio locked.
ff_layout_pg_init_write() switches a write to the MDS by resetting the
descriptor and returning -EAGAIN, which nfs_wb_folio() also passes up,
with or without the flag. When nfs_wb_folio() gets -EAGAIN, wait for
any part of the folio already under writeback, as its loop does, and
write the folio once more with a descriptor that goes straight to the
MDS. nfs_wb_folio_reclaim() still returns the error, and
nfs_release_folio() keeps the folio.
Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in pg_get_mirror_count_write")
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
fs/nfs/flexfilelayout/flexfilelayout.c | 5 +++--
fs/nfs/pagelist.c | 1 +
fs/nfs/write.c | 19 +++++++++++++++----
include/linux/nfs_page.h | 5 +++++
4 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 5c289737366f..2b0a0af469fb 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -1176,7 +1176,8 @@ ff_layout_pg_init_write(struct nfs_pageio_descriptor *pgio,
mirror_ds, dss_id, OP_WRITE);
if (IS_ERR(ds)) {
nfs4_ff_layout_put_deviceid(mirror_ds);
- if (!ff_layout_no_fallback_to_mds(pgio->pg_lseg))
+ if (pgio->pg_allow_mds ||
+ !ff_layout_no_fallback_to_mds(pgio->pg_lseg))
goto out_mds;
pnfs_generic_pg_cleanup(pgio);
/* Sleep for 1 second before retrying */
@@ -1229,7 +1230,7 @@ ff_layout_pg_get_mirror_count_write(struct nfs_pageio_descriptor *pgio,
0, NFS4_MAX_UINT64, IOMODE_RW,
NFS_I(pgio->pg_inode)->layout,
pgio->pg_lseg);
- if (NFS_I(pgio->pg_inode)->layout &&
+ if (!pgio->pg_allow_mds && NFS_I(pgio->pg_inode)->layout &&
ff_layout_hdr_no_fallback_to_mds(NFS_I(pgio->pg_inode)->layout)) {
/*
* FF_FLAGS_NO_IO_THRU_MDS: no current lseg but the server's
diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
index 71f0ce2bc4ea..aa6d7b658d79 100644
--- a/fs/nfs/pagelist.c
+++ b/fs/nfs/pagelist.c
@@ -871,6 +871,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc,
int io_flags)
{
desc->pg_moreio = 0;
+ desc->pg_allow_mds = 0;
desc->pg_inode = inode;
desc->pg_ops = pg_ops;
desc->pg_completion_ops = compl_ops;
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 0b42d31f7a75..ec6b771b8dbe 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -627,15 +627,16 @@ static int nfs_do_writepage(struct folio *folio, struct writeback_control *wbc,
* Write an mmapped page to the server.
*/
static int nfs_writepage_locked(struct folio *folio,
- struct writeback_control *wbc)
+ struct writeback_control *wbc, bool force_mds)
{
struct nfs_pageio_descriptor pgio;
struct inode *inode = folio->mapping->host;
int err;
nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
- nfs_pageio_init_write(&pgio, inode, 0, false,
+ nfs_pageio_init_write(&pgio, inode, 0, force_mds,
&nfs_async_write_completion_ops);
+ pgio.pg_allow_mds = 1;
err = nfs_do_writepage(folio, wbc, &pgio);
pgio.pg_error = 0;
nfs_pageio_complete(&pgio);
@@ -2057,7 +2058,7 @@ int nfs_wb_folio_reclaim(struct inode *inode, struct folio *folio)
return -EBUSY;
if (folio_clear_dirty_for_io(folio)) {
trace_nfs_writeback_folio_reclaim(inode, range_start, len);
- ret = nfs_writepage_locked(folio, &wbc);
+ ret = nfs_writepage_locked(folio, &wbc, false);
trace_nfs_writeback_folio_reclaim_done(inode, range_start, len,
ret);
return ret;
@@ -2084,6 +2085,7 @@ int nfs_wb_folio(struct inode *inode, struct folio *folio)
.range_start = range_start,
.range_end = range_start + len - 1,
};
+ bool force_mds = false;
int ret;
trace_nfs_writeback_folio(inode, range_start, len);
@@ -2091,7 +2093,16 @@ int nfs_wb_folio(struct inode *inode, struct folio *folio)
for (;;) {
folio_wait_writeback(folio);
if (folio_clear_dirty_for_io(folio)) {
- ret = nfs_writepage_locked(folio, &wbc);
+ ret = nfs_writepage_locked(folio, &wbc, force_mds);
+ /*
+ * -EAGAIN asks for a resend, which can't wait for a
+ * layout with the folio locked: resend it through the
+ * MDS once any part of it already sent has completed.
+ */
+ if (ret == -EAGAIN && !force_mds) {
+ force_mds = true;
+ continue;
+ }
if (ret < 0)
goto out_error;
continue;
diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
index c38e4b380be5..d64332afc374 100644
--- a/include/linux/nfs_page.h
+++ b/include/linux/nfs_page.h
@@ -120,6 +120,11 @@ struct nfs_pageio_descriptor {
u32 pg_mirror_idx; /* current mirror */
unsigned short pg_maxretrans;
unsigned char pg_moreio : 1;
+ /*
+ * Set when the caller holds the folio locked and can't wait for a
+ * layout: the write goes through the MDS instead of waiting for one.
+ */
+ unsigned char pg_allow_mds : 1;
};
/* arbitrarily selected limit to number of mirrors */
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
2026-11-18 18:36 ` [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
2026-11-18 18:36 ` [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington
3 siblings, 0 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
When a write has no layout segment and its file's layouts carry
FF_FLAGS_NO_IO_THRU_MDS, ff_layout_pg_get_mirror_count_write() returns
-EAGAIN. nfs_writepages() and the O_DIRECT reschedule path resend it at
once, so writers spin for as long as pnfs_update_layout() has no layout
to give: while NFS_LAYOUT_BULK_RECALL is set, while LAYOUTGETs are
blocked, and for the 120 seconds after NFS4ERR_LAYOUTUNAVAILABLE.
The check also tests a bit on the layout header. Once that header has
been freed, a new one starts without the bit and the write goes through
the MDS. ff_layout_pg_init_write() has no check at all: when
pnfs_generic_pg_check_layout() drops an invalidated segment, the rest of
the descriptor's writes go through the MDS.
Handle both places the way ff_layout_pg_init_read() handles reads: test
the inode's NFS_INO_NO_IO_THRU_MDS bit, return -ERESTARTSYS to a killed
or exiting task, and honour io_maxretrans on soft mounts. Mark a
refusal with a new pg_nolayout descriptor bit, and have nfs_writepages()
and nfs_direct_write_reschedule() wait a second before they resend,
after the pass that was refused has unlocked its folios and requests.
Writes marked pg_allow_mds by nfs_writepage_locked() still go through
the MDS.
The test is ff_layout_inode_no_fallback_to_mds(), which
ff_layout_pg_init_read() now uses too, and the refusal is
ff_layout_pg_refuse_write(). Emit the MDS-fallback tracepoint only
when the write does go through the MDS, and a new
pnfs_mds_fallback_write_refused tracepoint when it is refused.
Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in pg_get_mirror_count_write")
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
fs/nfs/direct.c | 2 +
fs/nfs/flexfilelayout/flexfilelayout.c | 66 ++++++++++++++++++++------
fs/nfs/nfs4trace.c | 1 +
fs/nfs/nfs4trace.h | 1 +
fs/nfs/pagelist.c | 1 +
fs/nfs/write.c | 2 +
include/linux/nfs_page.h | 5 ++
7 files changed, 64 insertions(+), 14 deletions(-)
diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
index a3c6e8f4ea06..e6a2c6436b44 100644
--- a/fs/nfs/direct.c
+++ b/fs/nfs/direct.c
@@ -626,6 +626,8 @@ static void nfs_direct_write_reschedule(struct nfs_direct_req *dreq)
}
}
+ if (desc.pg_error == -EAGAIN && desc.pg_nolayout)
+ ssleep(1);
if (put_dreq(dreq))
nfs_direct_write_complete(dreq);
}
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 2b0a0af469fb..4eb12bbb7838 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -982,6 +982,45 @@ static bool ff_layout_pg_retry_wait(struct nfs_pageio_descriptor *pgio)
return true;
}
+/*
+ * Returns true if the inode's layouts forbid I/O through the MDS.
+ */
+static bool ff_layout_inode_no_fallback_to_mds(struct inode *inode, u32 iomode)
+{
+ return nfs_no_io_thru_mds(inode);
+}
+
+/*
+ * Refuse a write that has no layout segment and can't go through the MDS:
+ * -EAGAIN so that the caller resends it, or a hard error when the caller
+ * should give up. pg_nolayout tells a caller that resends at once to
+ * wait first, once it has dropped its locks.
+ */
+static void ff_layout_pg_refuse_write(struct nfs_pageio_descriptor *pgio,
+ struct nfs_page *req)
+{
+ struct nfs_server *server = NFS_SERVER(pgio->pg_inode);
+
+ trace_pnfs_mds_fallback_write_refused(pgio->pg_inode, req_offset(req),
+ req->wb_bytes, IOMODE_RW,
+ NFS_I(pgio->pg_inode)->layout,
+ NULL);
+ if (server->flags & (NFS_MOUNT_SOFT | NFS_MOUNT_SOFTERR) &&
+ io_maxretrans && req->wb_nio > io_maxretrans) {
+ if (server->flags & NFS_MOUNT_SOFTERR)
+ pgio->pg_error = -ETIMEDOUT;
+ else
+ pgio->pg_error = -EIO;
+ return;
+ }
+ if (fatal_signal_pending(current) || nfs_current_task_exiting()) {
+ pgio->pg_error = -ERESTARTSYS;
+ return;
+ }
+ pgio->pg_nolayout = 1;
+ pgio->pg_error = -EAGAIN;
+}
+
static void
ff_layout_pg_get_read(struct nfs_pageio_descriptor *pgio,
struct nfs_page *req,
@@ -1108,7 +1147,7 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,
* FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
* this asks the inode and not the hdr.
*/
- if (!nfs_no_io_thru_mds(pgio->pg_inode))
+ if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
goto out_mds;
retry_nolseg:
/* Retry getting layout segment if lower layer returned -EAGAIN */
@@ -1156,9 +1195,13 @@ ff_layout_pg_init_write(struct nfs_pageio_descriptor *pgio,
return;
}
}
- /* If no lseg, fall back to write through mds */
- if (pgio->pg_lseg == NULL)
- goto out_mds;
+ if (!pgio->pg_lseg) {
+ if (pgio->pg_allow_mds ||
+ !ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_RW))
+ goto out_mds;
+ ff_layout_pg_refuse_write(pgio, req);
+ return;
+ }
/* Use a direct mapping of ds_idx to pgio mirror_idx */
if (pgio->pg_mirror_count != FF_LAYOUT_MIRROR_COUNT(pgio->pg_lseg))
@@ -1225,21 +1268,16 @@ ff_layout_pg_get_mirror_count_write(struct nfs_pageio_descriptor *pgio,
}
if (pgio->pg_lseg)
return FF_LAYOUT_MIRROR_COUNT(pgio->pg_lseg);
+ if (!pgio->pg_allow_mds &&
+ ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_RW)) {
+ ff_layout_pg_refuse_write(pgio, req);
+ goto out;
+ }
trace_pnfs_mds_fallback_pg_get_mirror_count(pgio->pg_inode,
0, NFS4_MAX_UINT64, IOMODE_RW,
NFS_I(pgio->pg_inode)->layout,
pgio->pg_lseg);
- if (!pgio->pg_allow_mds && NFS_I(pgio->pg_inode)->layout &&
- ff_layout_hdr_no_fallback_to_mds(NFS_I(pgio->pg_inode)->layout)) {
- /*
- * FF_FLAGS_NO_IO_THRU_MDS: no current lseg but the server's
- * policy forbids MDS fallback. Surface -EAGAIN so writeback
- * retries rather than silently issuing the WRITE via MDS.
- */
- pgio->pg_error = -EAGAIN;
- goto out;
- }
/* no lseg means that pnfs is not in use, so no mirroring here */
nfs_pageio_reset_write_mds(pgio);
out:
diff --git a/fs/nfs/nfs4trace.c b/fs/nfs/nfs4trace.c
index 3fdc013f56d8..93a84f546de8 100644
--- a/fs/nfs/nfs4trace.c
+++ b/fs/nfs/nfs4trace.c
@@ -25,6 +25,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_read_done);
EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_done);
EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_read_pagelist);
EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_pagelist);
+EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_refused);
EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_ds_connect);
EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_read_error);
diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h
index b5c89eeef2bc..94ed6b833a28 100644
--- a/fs/nfs/nfs4trace.h
+++ b/fs/nfs/nfs4trace.h
@@ -2115,6 +2115,7 @@ DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_read_done);
DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_done);
DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_read_pagelist);
DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_pagelist);
+DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_refused);
DECLARE_EVENT_CLASS(nfs4_deviceid_event,
TP_PROTO(
diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
index aa6d7b658d79..9a8731de6f5b 100644
--- a/fs/nfs/pagelist.c
+++ b/fs/nfs/pagelist.c
@@ -872,6 +872,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc,
{
desc->pg_moreio = 0;
desc->pg_allow_mds = 0;
+ desc->pg_nolayout = 0;
desc->pg_inode = inode;
desc->pg_ops = pg_ops;
desc->pg_completion_ops = compl_ops;
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index ec6b771b8dbe..005d4b795b80 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -693,6 +693,8 @@ int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc)
nfs_pageio_complete(&pgio);
if (err == -EAGAIN && mntflags & NFS_MOUNT_SOFTERR)
break;
+ if (err == -EAGAIN && pgio.pg_nolayout)
+ ssleep(1);
} while (err < 0 && !nfs_error_is_fatal(err));
nfs_io_completion_put(ioc);
diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
index d64332afc374..f9b22b990de7 100644
--- a/include/linux/nfs_page.h
+++ b/include/linux/nfs_page.h
@@ -125,6 +125,11 @@ struct nfs_pageio_descriptor {
* layout: the write goes through the MDS instead of waiting for one.
*/
unsigned char pg_allow_mds : 1;
+ /*
+ * Set by a layout driver that refused a write with -EAGAIN for lack
+ * of a layout, so that a caller that resends waits first.
+ */
+ unsigned char pg_nolayout : 1;
};
/* arbitrarily selected limit to number of mirrors */
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
2026-11-18 18:36 ` [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington
3 siblings, 0 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
Nothing tests NFS4_FF_HDR_NO_IO_THRU_MDS since writes moved to the
inode's NFS_INO_NO_IO_THRU_MDS bit. Remove it, its helper, and
nfs4_flexfile_layout::flags, which held only that bit.
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
fs/nfs/flexfilelayout/flexfilelayout.c | 8 ++------
fs/nfs/flexfilelayout/flexfilelayout.h | 16 ----------------
fs/nfs/internal.h | 9 ++++-----
3 files changed, 6 insertions(+), 27 deletions(-)
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 4eb12bbb7838..a522c3fe49e5 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -641,12 +641,8 @@ ff_layout_alloc_lseg(struct pnfs_layout_hdr *lh,
if (!p)
goto out_sort_mirrors;
fls->flags = be32_to_cpup(p);
- if (fls->flags & FF_FLAGS_NO_IO_THRU_MDS) {
- set_bit(NFS4_FF_HDR_NO_IO_THRU_MDS,
- &FF_LAYOUT_FROM_HDR(lh)->flags);
- /* Outlives this layout hdr; see NFS_INO_NO_IO_THRU_MDS */
+ if (fls->flags & FF_FLAGS_NO_IO_THRU_MDS)
nfs_set_no_io_thru_mds(lh->plh_inode);
- }
p = xdr_inline_decode(&stream, 4);
if (!p)
@@ -1145,7 +1141,7 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,
* MDS there is no fallback to take, so wait for a layout on the
* same terms as the -EAGAIN above. The layout hdr that carried
* FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
- * this asks the inode and not the hdr.
+ * the flag is kept on the inode.
*/
if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
goto out_mds;
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.h b/fs/nfs/flexfilelayout/flexfilelayout.h
index 5d5587fded76..bb60537b2b93 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.h
+++ b/fs/nfs/flexfilelayout/flexfilelayout.h
@@ -113,16 +113,12 @@ struct nfs4_ff_layout_segment {
struct nfs4_ff_layout_mirror *mirror_array[] __counted_by(mirror_array_cnt);
};
-/* nfs4_flexfile_layout::flags bit indices */
-#define NFS4_FF_HDR_NO_IO_THRU_MDS 0 /* any lseg has had FF_FLAGS_NO_IO_THRU_MDS */
-
struct nfs4_flexfile_layout {
struct pnfs_layout_hdr generic_hdr;
struct pnfs_ds_commit_info commit_info;
struct list_head mirrors;
struct list_head error_list; /* nfs4_ff_layout_ds_err */
ktime_t last_report_time; /* Layoutstat report times */
- unsigned long flags;
};
/*
@@ -186,18 +182,6 @@ ff_layout_no_fallback_to_mds(struct pnfs_layout_segment *lseg)
return FF_LAYOUT_LSEG(lseg)->flags & FF_FLAGS_NO_IO_THRU_MDS;
}
-/*
- * Sticky hdr-level mirror of FF_FLAGS_NO_IO_THRU_MDS so callers that have
- * no current lseg (e.g. between LAYOUTRETURN and the next LAYOUTGET) can
- * still honor the no-MDS-fallback policy.
- */
-static inline bool
-ff_layout_hdr_no_fallback_to_mds(struct pnfs_layout_hdr *lo)
-{
- return test_bit(NFS4_FF_HDR_NO_IO_THRU_MDS,
- &FF_LAYOUT_FROM_HDR(lo)->flags);
-}
-
static inline bool
ff_layout_no_read_on_rw(struct pnfs_layout_segment *lseg)
{
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 5af6a032f145..5b40364fc38f 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -556,11 +556,10 @@ static inline bool nfs_file_io_is_buffered(struct nfs_inode *nfsi)
* FF_FLAGS_NO_IO_THRU_MDS in ff_layout_alloc_lseg()). The RFC 5661
* mdsthreshold hint must not be acted on for such a file, since the only
* thing pnfs_within_mdsthreshold() can ask for is the one thing the
- * layout forbids. Servers are assumed to be consistent in their
- * no-fallback policy per file, the same assumption
- * ff_layout_hdr_no_fallback_to_mds() already makes; if one were not, the
- * only effect is that its mdsthreshold hint - a SHOULD - stops being
- * honored for an inode that is already in core.
+ * layout forbids. Flexfiles also retries reads and writes that have no
+ * layout segment instead of sending them through the MDS, except
+ * writeback of a single locked folio (pg_allow_mds). Servers are
+ * assumed to be consistent in their no-fallback policy per file.
*/
#define NFS_INO_NO_IO_THRU_MDS (30)
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
` (2 preceding siblings ...)
2026-11-18 18:36 ` [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
3 siblings, 0 replies; 5+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
When LAYOUTGET fails with NFS4ERR_LAYOUTUNAVAILABLE, or with
NFS4ERR_TOOSMALL after the retry at the session's maximum reply size,
pnfs_update_layout() sets the layout's fail bit for that iomode and
returns no layout without sending another LAYOUTGET for the next 120
seconds. If the file's layouts carried FF_FLAGS_NO_IO_THRU_MDS, reads
and writes wait for a layout the whole time, and then send a LAYOUTGET
that can fail the same way. On a hard mount that repeats for as long
as the server keeps answering that way.
RFC 8435 says that after NFS4ERR_LAYOUTUNAVAILABLE "the I/O is to go to
the metadata server" (section 5.1.1), and that FF_FLAGS_NO_IO_THRU_MDS
is a hint (section 5.1.2). Send the I/O through the MDS while the fail
bit is set, in ff_layout_inode_no_fallback_to_mds(), which reads,
writes and the resend after a data server error share. This also lets
a server deal with a data server the client can't use: the client
reports the failure with LAYOUTERROR (or in LAYOUTRETURN) and returns
the layout, and the server can answer the next LAYOUTGET with
NFS4ERR_LAYOUTUNAVAILABLE.
Fixes: 2f007ba074ad ("pNFS/flexfiles: don't read through MDS when previous layout forbid it")
Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in pg_get_mirror_count_write")
Fixes: ec906cf7bee6 ("pNFS/flexfiles: don't reset to MDS for v4 error when previous layout forbid it")
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
fs/nfs/flexfilelayout/flexfilelayout.c | 40 +++++++++++++++-----------
fs/nfs/internal.h | 6 ++--
fs/nfs/pnfs.c | 22 ++++++++++++++
fs/nfs/pnfs.h | 1 +
4 files changed, 50 insertions(+), 19 deletions(-)
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index a522c3fe49e5..c4fae5408535 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -980,10 +980,14 @@ static bool ff_layout_pg_retry_wait(struct nfs_pageio_descriptor *pgio)
/*
* Returns true if the inode's layouts forbid I/O through the MDS.
+ * FF_FLAGS_NO_IO_THRU_MDS is a hint (RFC 8435, section 5.1.2), so set it
+ * aside while a failed LAYOUTGET for @iomode (NFS4ERR_LAYOUTUNAVAILABLE or
+ * NFS4ERR_TOOSMALL) keeps pnfs_update_layout() from sending another.
*/
static bool ff_layout_inode_no_fallback_to_mds(struct inode *inode, u32 iomode)
{
- return nfs_no_io_thru_mds(inode);
+ return nfs_no_io_thru_mds(inode) &&
+ !pnfs_layout_get_failed(inode, iomode);
}
/*
@@ -1135,13 +1139,12 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,
}
/*
* No segment, and no error to report either: pnfs_update_layout()
- * simply has nothing to give (NFS_LAYOUT_BULK_RECALL, a failed
- * pnfs_layout_io_test, blocked LAYOUTGETs, a layout being
- * returned). If the server forbids reading this file through the
- * MDS there is no fallback to take, so wait for a layout on the
- * same terms as the -EAGAIN above. The layout hdr that carried
- * FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
- * the flag is kept on the inode.
+ * simply has nothing to give (NFS_LAYOUT_BULK_RECALL, blocked
+ * LAYOUTGETs, a layout being returned). If the server forbids
+ * reading this file through the MDS there is no fallback to take,
+ * so wait for a layout on the same terms as the -EAGAIN above. The
+ * layout hdr that carried FF_FLAGS_NO_IO_THRU_MDS may itself be
+ * gone by now, which is why the flag is kept on the inode.
*/
if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
goto out_mds;
@@ -1395,7 +1398,8 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
struct nfs4_state *state,
struct nfs_client *clp,
struct pnfs_layout_segment *lseg,
- struct nfs4_deviceid_node *devid)
+ struct nfs4_deviceid_node *devid,
+ u32 iomode)
{
struct pnfs_layout_hdr *lo = lseg->pls_layout;
struct inode *inode = lo->plh_inode;
@@ -1484,8 +1488,9 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
return -NFS4ERR_RESET_TO_PNFS;
reset:
/*
- * FF_FLAGS_NO_IO_THRU_MDS: never resend through the MDS. The
- * caller would do so with force_mds set (ff_layout_reset_read(),
+ * FF_FLAGS_NO_IO_THRU_MDS: don't resend through the MDS (see
+ * ff_layout_inode_no_fallback_to_mds()). The caller would do so
+ * with force_mds set (ff_layout_reset_read(),
* ff_layout_reset_write(hdr, false) -> pnfs_*_done_resend_to_mds()),
* which builds a descriptor out of the plain MDS page ops and so
* consults no layout at all -- this is the last point at which the
@@ -1495,7 +1500,7 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
* pnfs_destroy_layout(). Having done so they can no longer ask the
* layout hdr about the flag, hence the inode.
*/
- if (nfs_no_io_thru_mds(inode)) {
+ if (ff_layout_inode_no_fallback_to_mds(inode, iomode)) {
dprintk("%s Retry through pNFS, no MDS fallback. Error %d\n",
__func__, task->tk_status);
return -NFS4ERR_RESET_TO_PNFS;
@@ -1580,7 +1585,8 @@ static int ff_layout_async_handle_error(struct rpc_task *task,
struct nfs4_state *state,
struct nfs_client *clp,
struct pnfs_layout_segment *lseg,
- struct nfs4_deviceid_node *devid)
+ struct nfs4_deviceid_node *devid,
+ u32 iomode)
{
int vers = clp->cl_nfs_mod->rpc_vers->number;
@@ -1600,7 +1606,7 @@ static int ff_layout_async_handle_error(struct rpc_task *task,
lseg, devid);
case 4:
return ff_layout_async_handle_error_v4(task, op_status, state,
- clp, lseg, devid);
+ clp, lseg, devid, iomode);
default:
/* should never happen */
WARN_ON_ONCE(1);
@@ -1734,7 +1740,7 @@ static int ff_layout_read_done_cb(struct rpc_task *task,
err = ff_layout_async_handle_error(task, hdr->res.op_status,
hdr->args.context->state,
hdr->ds_clp, hdr->lseg,
- hdr->ds_dev);
+ hdr->ds_dev, IOMODE_READ);
trace_nfs4_pnfs_read(hdr, err);
clear_bit(NFS_IOHDR_RESEND_PNFS, &hdr->flags);
@@ -1944,7 +1950,7 @@ static int ff_layout_write_done_cb(struct rpc_task *task,
err = ff_layout_async_handle_error(task, hdr->res.op_status,
hdr->args.context->state,
hdr->ds_clp, hdr->lseg,
- hdr->ds_dev);
+ hdr->ds_dev, IOMODE_RW);
trace_nfs4_pnfs_write(hdr, err);
clear_bit(NFS_IOHDR_RESEND_PNFS, &hdr->flags);
@@ -1995,7 +2001,7 @@ static int ff_layout_commit_done_cb(struct rpc_task *task,
err = ff_layout_async_handle_error(task, data->res.op_status,
NULL, data->ds_clp, data->lseg,
- data->ds_dev);
+ data->ds_dev, IOMODE_RW);
trace_nfs4_pnfs_commit_ds(data, err);
switch (err) {
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 5b40364fc38f..fc2a27b5126a 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -558,8 +558,10 @@ static inline bool nfs_file_io_is_buffered(struct nfs_inode *nfsi)
* thing pnfs_within_mdsthreshold() can ask for is the one thing the
* layout forbids. Flexfiles also retries reads and writes that have no
* layout segment instead of sending them through the MDS, except
- * writeback of a single locked folio (pg_allow_mds). Servers are
- * assumed to be consistent in their no-fallback policy per file.
+ * writeback of a single locked folio (pg_allow_mds), and unless a
+ * LAYOUTGET for that iomode has failed with NFS4ERR_LAYOUTUNAVAILABLE or
+ * NFS4ERR_TOOSMALL. Servers are assumed to be consistent in their
+ * no-fallback policy per file.
*/
#define NFS_INO_NO_IO_THRU_MDS (30)
diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 14f4a4b12ffd..582fe81fc2d1 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -551,6 +551,28 @@ pnfs_layout_io_test_failed(struct pnfs_layout_hdr *lo, u32 iomode)
return true;
}
+/*
+ * Returns true if a LAYOUTGET for @iomode failed with an error, such as
+ * NFS4ERR_LAYOUTUNAVAILABLE, in the last PNFS_LAYOUTGET_RETRY_TIMEOUT.
+ * pnfs_layout_io_test_failed() clears an expired fail bit only when
+ * pnfs_update_layout() gets that far, so check the time here too.
+ */
+bool pnfs_layout_get_failed(struct inode *inode, u32 iomode)
+{
+ struct pnfs_layout_hdr *lo;
+ unsigned long now = jiffies;
+ bool ret = false;
+
+ spin_lock(&inode->i_lock);
+ lo = NFS_I(inode)->layout;
+ if (lo && test_bit(pnfs_iomode_to_fail_bit(iomode), &lo->plh_flags))
+ ret = time_in_range(lo->plh_retry_timestamp,
+ now - PNFS_LAYOUTGET_RETRY_TIMEOUT, now);
+ spin_unlock(&inode->i_lock);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(pnfs_layout_get_failed);
+
static void
pnfs_init_lseg(struct pnfs_layout_hdr *lo, struct pnfs_layout_segment *lseg,
const struct pnfs_layout_range *range,
diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
index 2774d4adf4c3..0007a4ca3da9 100644
--- a/fs/nfs/pnfs.h
+++ b/fs/nfs/pnfs.h
@@ -296,6 +296,7 @@ int pnfs_generic_pg_writepages(struct nfs_pageio_descriptor *desc);
size_t pnfs_generic_pg_test(struct nfs_pageio_descriptor *pgio,
struct nfs_page *prev, struct nfs_page *req);
void pnfs_set_lo_fail(struct pnfs_layout_segment *lseg);
+bool pnfs_layout_get_failed(struct inode *inode, u32 iomode);
struct pnfs_layout_segment *pnfs_layout_process(struct nfs4_layoutget *lgp);
void pnfs_layoutget_free(struct nfs4_layoutget *lgp);
void pnfs_free_lseg_list(struct list_head *tmp_list);
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-18 18:37 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
2026-11-18 18:36 ` [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
2026-11-18 18:36 ` [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox