* [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS
@ 2026-11-18 18:36 Benjamin Coddington
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
When a file's flexfiles layouts carry FF_FLAGS_NO_IO_THRU_MDS, the client
keeps reads and writes off the MDS when it has no layout segment for
them. Reads and writes handle that differently:
- Reads test the inode's NFS_INO_NO_IO_THRU_MDS bit, wait a second
between attempts, and honour io_maxretrans on soft mounts.
- Writes test a bit on the layout header and return -EAGAIN.
nfs_writepages() and the O_DIRECT reschedule path resend at once, so
writers spin for as long as there is no layout. nfs_wb_folio()
returns the -EAGAIN, so a read(2) or write(2) that has to flush a
folio another open context dirtied fails with EAGAIN. Once the
header is freed the bit is gone and writes go through the MDS, and
ff_layout_pg_init_write() sends them there whenever
pnfs_generic_pg_check_layout() drops an invalidated segment.
Neither direction uses the MDS when the server has said there is no
layout to give. After NFS4ERR_LAYOUTUNAVAILABLE, RFC 8435 says the I/O
is to go to the MDS (section 5.1.1), and the flag is a hint (section
5.1.2). Instead, reads wait and writes spin until the 120 second fail
bit expires, then send a LAYOUTGET that can fail the same way.
Patch 1 sends writeback of a single locked folio (nfs_wb_folio(),
reclaim, migration) through the MDS instead of refusing it. Resending
with the folio locked could deadlock against a bulk recall, which can't
complete while nfs_writepages() holds a layout segment, and
nfs_writepages() would be waiting for that folio. Patch 2 has the
remaining writes test the inode bit, and has nfs_writepages() and the
O_DIRECT reschedule path wait a second before resending a refused
write, after their folios and requests are unlocked. Patch 3 removes
the header bit, which nothing tests after patch 2. Patch 4 sends
reads, writes and the resend after a data server error through the MDS
while the fail bit for their iomode is set.
That last change is also the path for a data server the client can't
use, such as a version mismatch: the client reports the failure with
LAYOUTERROR or LAYOUTRETURN, and the server decides whether to answer
the next LAYOUTGET with NFS4ERR_LAYOUTUNAVAILABLE. The missing-nfsv3
case is covered separately by Roberto's "NFSv4/flexfiles: couple back
with NFSv3".
These patches are independent of the series "pNFS: don't reply
NOMATCHING to a layout recall with I/O in flight", but both change
ff_layout_pg_get_mirror_count_write(), so whichever goes second will
need a small rebase. The series is based on Anna's linux-next.
Tested against a reffs MDS and DS, injecting LAYOUTGET errors after a
file recall. With NFS4ERR_LAYOUTUNAVAILABLE, reads and writes go through
the MDS at once. With a retryable error (NFS4ERR_SERVERFAULT) held for
8 seconds, writeback sends one LAYOUTGET a second and no I/O to the MDS,
SIGKILL ends a writer blocked in fsync, and a soft mount with
io_maxretrans=3 returns EIO after about 4 seconds. During the hold, a
read of a folio that another open context had dirtied flushes it
through the MDS and returns without EAGAIN.
Benjamin Coddington (4):
pNFS/flexfiles: use the MDS for locked-folio writeback without a
layout
pNFS/flexfiles: pace writes that have no layout and can't use the MDS
pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit
pNFS/flexfiles: use the MDS when the server has no layout to give
fs/nfs/direct.c | 2 +
fs/nfs/flexfilelayout/flexfilelayout.c | 113 +++++++++++++++++--------
fs/nfs/flexfilelayout/flexfilelayout.h | 16 ----
fs/nfs/internal.h | 11 +--
fs/nfs/nfs4trace.c | 1 +
fs/nfs/nfs4trace.h | 1 +
fs/nfs/pagelist.c | 2 +
fs/nfs/pnfs.c | 22 +++++
fs/nfs/pnfs.h | 1 +
fs/nfs/write.c | 21 ++++-
include/linux/nfs_page.h | 10 +++
11 files changed, 139 insertions(+), 61 deletions(-)
--
2.53.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
2026-11-18 23:22 ` Anna Schumaker
2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
` (2 subsequent siblings)
3 siblings, 1 reply; 8+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
nfs_writepage_locked() writes back one folio that its caller holds
locked, for nfs_wb_folio(), nfs_wb_folio_reclaim() and migration. When
there is no layout segment and the file's layout header carries
FF_FLAGS_NO_IO_THRU_MDS, ff_layout_pg_get_mirror_count_write() refuses
the write with -EAGAIN. nfs_do_writepage() redirties the folio and
returns the -EAGAIN, and nfs_wb_folio() passes it to nfs_updatepage(),
nfs_flush_incompatible(), nfs_read_folio() and nfs_launder_folio(). A
read(2) or write(2) on a blocking file then fails with EAGAIN, for
example when another open context has dirtied the same folio while a
layout is being recalled.
Resending from nfs_wb_folio(), as nfs_writepages() does, would hold the
folio locked across the resends. A bulk recall can't complete while
nfs_writepages() holds a layout segment, and nfs_writepages() would be
waiting for that folio's lock.
FF_FLAGS_NO_IO_THRU_MDS is a hint (RFC 8435, section 5.1.2). Mark the
descriptor that nfs_writepage_locked() uses with a new pg_allow_mds
bit, and send its write through the MDS when there is no layout
segment. When ff_layout_pg_init_write() can't use a data server, send
such a write to the MDS too, instead of retrying with the folio locked.
ff_layout_pg_init_write() switches a write to the MDS by resetting the
descriptor and returning -EAGAIN, which nfs_wb_folio() also passes up,
with or without the flag. When nfs_wb_folio() gets -EAGAIN, wait for
any part of the folio already under writeback, as its loop does, and
write the folio once more with a descriptor that goes straight to the
MDS. nfs_wb_folio_reclaim() still returns the error, and
nfs_release_folio() keeps the folio.
Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in pg_get_mirror_count_write")
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
fs/nfs/flexfilelayout/flexfilelayout.c | 5 +++--
fs/nfs/pagelist.c | 1 +
fs/nfs/write.c | 19 +++++++++++++++----
include/linux/nfs_page.h | 5 +++++
4 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 5c289737366f..2b0a0af469fb 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -1176,7 +1176,8 @@ ff_layout_pg_init_write(struct nfs_pageio_descriptor *pgio,
mirror_ds, dss_id, OP_WRITE);
if (IS_ERR(ds)) {
nfs4_ff_layout_put_deviceid(mirror_ds);
- if (!ff_layout_no_fallback_to_mds(pgio->pg_lseg))
+ if (pgio->pg_allow_mds ||
+ !ff_layout_no_fallback_to_mds(pgio->pg_lseg))
goto out_mds;
pnfs_generic_pg_cleanup(pgio);
/* Sleep for 1 second before retrying */
@@ -1229,7 +1230,7 @@ ff_layout_pg_get_mirror_count_write(struct nfs_pageio_descriptor *pgio,
0, NFS4_MAX_UINT64, IOMODE_RW,
NFS_I(pgio->pg_inode)->layout,
pgio->pg_lseg);
- if (NFS_I(pgio->pg_inode)->layout &&
+ if (!pgio->pg_allow_mds && NFS_I(pgio->pg_inode)->layout &&
ff_layout_hdr_no_fallback_to_mds(NFS_I(pgio->pg_inode)->layout)) {
/*
* FF_FLAGS_NO_IO_THRU_MDS: no current lseg but the server's
diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
index 71f0ce2bc4ea..aa6d7b658d79 100644
--- a/fs/nfs/pagelist.c
+++ b/fs/nfs/pagelist.c
@@ -871,6 +871,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc,
int io_flags)
{
desc->pg_moreio = 0;
+ desc->pg_allow_mds = 0;
desc->pg_inode = inode;
desc->pg_ops = pg_ops;
desc->pg_completion_ops = compl_ops;
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 0b42d31f7a75..ec6b771b8dbe 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -627,15 +627,16 @@ static int nfs_do_writepage(struct folio *folio, struct writeback_control *wbc,
* Write an mmapped page to the server.
*/
static int nfs_writepage_locked(struct folio *folio,
- struct writeback_control *wbc)
+ struct writeback_control *wbc, bool force_mds)
{
struct nfs_pageio_descriptor pgio;
struct inode *inode = folio->mapping->host;
int err;
nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
- nfs_pageio_init_write(&pgio, inode, 0, false,
+ nfs_pageio_init_write(&pgio, inode, 0, force_mds,
&nfs_async_write_completion_ops);
+ pgio.pg_allow_mds = 1;
err = nfs_do_writepage(folio, wbc, &pgio);
pgio.pg_error = 0;
nfs_pageio_complete(&pgio);
@@ -2057,7 +2058,7 @@ int nfs_wb_folio_reclaim(struct inode *inode, struct folio *folio)
return -EBUSY;
if (folio_clear_dirty_for_io(folio)) {
trace_nfs_writeback_folio_reclaim(inode, range_start, len);
- ret = nfs_writepage_locked(folio, &wbc);
+ ret = nfs_writepage_locked(folio, &wbc, false);
trace_nfs_writeback_folio_reclaim_done(inode, range_start, len,
ret);
return ret;
@@ -2084,6 +2085,7 @@ int nfs_wb_folio(struct inode *inode, struct folio *folio)
.range_start = range_start,
.range_end = range_start + len - 1,
};
+ bool force_mds = false;
int ret;
trace_nfs_writeback_folio(inode, range_start, len);
@@ -2091,7 +2093,16 @@ int nfs_wb_folio(struct inode *inode, struct folio *folio)
for (;;) {
folio_wait_writeback(folio);
if (folio_clear_dirty_for_io(folio)) {
- ret = nfs_writepage_locked(folio, &wbc);
+ ret = nfs_writepage_locked(folio, &wbc, force_mds);
+ /*
+ * -EAGAIN asks for a resend, which can't wait for a
+ * layout with the folio locked: resend it through the
+ * MDS once any part of it already sent has completed.
+ */
+ if (ret == -EAGAIN && !force_mds) {
+ force_mds = true;
+ continue;
+ }
if (ret < 0)
goto out_error;
continue;
diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
index c38e4b380be5..d64332afc374 100644
--- a/include/linux/nfs_page.h
+++ b/include/linux/nfs_page.h
@@ -120,6 +120,11 @@ struct nfs_pageio_descriptor {
u32 pg_mirror_idx; /* current mirror */
unsigned short pg_maxretrans;
unsigned char pg_moreio : 1;
+ /*
+ * Set when the caller holds the folio locked and can't wait for a
+ * layout: the write goes through the MDS instead of waiting for one.
+ */
+ unsigned char pg_allow_mds : 1;
};
/* arbitrarily selected limit to number of mirrors */
--
2.53.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
2026-11-18 23:52 ` Anna Schumaker
2026-11-18 18:36 ` [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
2026-11-18 18:36 ` [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington
3 siblings, 1 reply; 8+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
When a write has no layout segment and its file's layouts carry
FF_FLAGS_NO_IO_THRU_MDS, ff_layout_pg_get_mirror_count_write() returns
-EAGAIN. nfs_writepages() and the O_DIRECT reschedule path resend it at
once, so writers spin for as long as pnfs_update_layout() has no layout
to give: while NFS_LAYOUT_BULK_RECALL is set, while LAYOUTGETs are
blocked, and for the 120 seconds after NFS4ERR_LAYOUTUNAVAILABLE.
The check also tests a bit on the layout header. Once that header has
been freed, a new one starts without the bit and the write goes through
the MDS. ff_layout_pg_init_write() has no check at all: when
pnfs_generic_pg_check_layout() drops an invalidated segment, the rest of
the descriptor's writes go through the MDS.
Handle both places the way ff_layout_pg_init_read() handles reads: test
the inode's NFS_INO_NO_IO_THRU_MDS bit, return -ERESTARTSYS to a killed
or exiting task, and honour io_maxretrans on soft mounts. Mark a
refusal with a new pg_nolayout descriptor bit, and have nfs_writepages()
and nfs_direct_write_reschedule() wait a second before they resend,
after the pass that was refused has unlocked its folios and requests.
Writes marked pg_allow_mds by nfs_writepage_locked() still go through
the MDS.
The test is ff_layout_inode_no_fallback_to_mds(), which
ff_layout_pg_init_read() now uses too, and the refusal is
ff_layout_pg_refuse_write(). Emit the MDS-fallback tracepoint only
when the write does go through the MDS, and a new
pnfs_mds_fallback_write_refused tracepoint when it is refused.
Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in pg_get_mirror_count_write")
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
fs/nfs/direct.c | 2 +
fs/nfs/flexfilelayout/flexfilelayout.c | 66 ++++++++++++++++++++------
fs/nfs/nfs4trace.c | 1 +
fs/nfs/nfs4trace.h | 1 +
fs/nfs/pagelist.c | 1 +
fs/nfs/write.c | 2 +
include/linux/nfs_page.h | 5 ++
7 files changed, 64 insertions(+), 14 deletions(-)
diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
index a3c6e8f4ea06..e6a2c6436b44 100644
--- a/fs/nfs/direct.c
+++ b/fs/nfs/direct.c
@@ -626,6 +626,8 @@ static void nfs_direct_write_reschedule(struct nfs_direct_req *dreq)
}
}
+ if (desc.pg_error == -EAGAIN && desc.pg_nolayout)
+ ssleep(1);
if (put_dreq(dreq))
nfs_direct_write_complete(dreq);
}
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 2b0a0af469fb..4eb12bbb7838 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -982,6 +982,45 @@ static bool ff_layout_pg_retry_wait(struct nfs_pageio_descriptor *pgio)
return true;
}
+/*
+ * Returns true if the inode's layouts forbid I/O through the MDS.
+ */
+static bool ff_layout_inode_no_fallback_to_mds(struct inode *inode, u32 iomode)
+{
+ return nfs_no_io_thru_mds(inode);
+}
+
+/*
+ * Refuse a write that has no layout segment and can't go through the MDS:
+ * -EAGAIN so that the caller resends it, or a hard error when the caller
+ * should give up. pg_nolayout tells a caller that resends at once to
+ * wait first, once it has dropped its locks.
+ */
+static void ff_layout_pg_refuse_write(struct nfs_pageio_descriptor *pgio,
+ struct nfs_page *req)
+{
+ struct nfs_server *server = NFS_SERVER(pgio->pg_inode);
+
+ trace_pnfs_mds_fallback_write_refused(pgio->pg_inode, req_offset(req),
+ req->wb_bytes, IOMODE_RW,
+ NFS_I(pgio->pg_inode)->layout,
+ NULL);
+ if (server->flags & (NFS_MOUNT_SOFT | NFS_MOUNT_SOFTERR) &&
+ io_maxretrans && req->wb_nio > io_maxretrans) {
+ if (server->flags & NFS_MOUNT_SOFTERR)
+ pgio->pg_error = -ETIMEDOUT;
+ else
+ pgio->pg_error = -EIO;
+ return;
+ }
+ if (fatal_signal_pending(current) || nfs_current_task_exiting()) {
+ pgio->pg_error = -ERESTARTSYS;
+ return;
+ }
+ pgio->pg_nolayout = 1;
+ pgio->pg_error = -EAGAIN;
+}
+
static void
ff_layout_pg_get_read(struct nfs_pageio_descriptor *pgio,
struct nfs_page *req,
@@ -1108,7 +1147,7 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,
* FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
* this asks the inode and not the hdr.
*/
- if (!nfs_no_io_thru_mds(pgio->pg_inode))
+ if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
goto out_mds;
retry_nolseg:
/* Retry getting layout segment if lower layer returned -EAGAIN */
@@ -1156,9 +1195,13 @@ ff_layout_pg_init_write(struct nfs_pageio_descriptor *pgio,
return;
}
}
- /* If no lseg, fall back to write through mds */
- if (pgio->pg_lseg == NULL)
- goto out_mds;
+ if (!pgio->pg_lseg) {
+ if (pgio->pg_allow_mds ||
+ !ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_RW))
+ goto out_mds;
+ ff_layout_pg_refuse_write(pgio, req);
+ return;
+ }
/* Use a direct mapping of ds_idx to pgio mirror_idx */
if (pgio->pg_mirror_count != FF_LAYOUT_MIRROR_COUNT(pgio->pg_lseg))
@@ -1225,21 +1268,16 @@ ff_layout_pg_get_mirror_count_write(struct nfs_pageio_descriptor *pgio,
}
if (pgio->pg_lseg)
return FF_LAYOUT_MIRROR_COUNT(pgio->pg_lseg);
+ if (!pgio->pg_allow_mds &&
+ ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_RW)) {
+ ff_layout_pg_refuse_write(pgio, req);
+ goto out;
+ }
trace_pnfs_mds_fallback_pg_get_mirror_count(pgio->pg_inode,
0, NFS4_MAX_UINT64, IOMODE_RW,
NFS_I(pgio->pg_inode)->layout,
pgio->pg_lseg);
- if (!pgio->pg_allow_mds && NFS_I(pgio->pg_inode)->layout &&
- ff_layout_hdr_no_fallback_to_mds(NFS_I(pgio->pg_inode)->layout)) {
- /*
- * FF_FLAGS_NO_IO_THRU_MDS: no current lseg but the server's
- * policy forbids MDS fallback. Surface -EAGAIN so writeback
- * retries rather than silently issuing the WRITE via MDS.
- */
- pgio->pg_error = -EAGAIN;
- goto out;
- }
/* no lseg means that pnfs is not in use, so no mirroring here */
nfs_pageio_reset_write_mds(pgio);
out:
diff --git a/fs/nfs/nfs4trace.c b/fs/nfs/nfs4trace.c
index 3fdc013f56d8..93a84f546de8 100644
--- a/fs/nfs/nfs4trace.c
+++ b/fs/nfs/nfs4trace.c
@@ -25,6 +25,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_read_done);
EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_done);
EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_read_pagelist);
EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_pagelist);
+EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_refused);
EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_ds_connect);
EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_read_error);
diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h
index b5c89eeef2bc..94ed6b833a28 100644
--- a/fs/nfs/nfs4trace.h
+++ b/fs/nfs/nfs4trace.h
@@ -2115,6 +2115,7 @@ DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_read_done);
DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_done);
DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_read_pagelist);
DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_pagelist);
+DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_refused);
DECLARE_EVENT_CLASS(nfs4_deviceid_event,
TP_PROTO(
diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
index aa6d7b658d79..9a8731de6f5b 100644
--- a/fs/nfs/pagelist.c
+++ b/fs/nfs/pagelist.c
@@ -872,6 +872,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc,
{
desc->pg_moreio = 0;
desc->pg_allow_mds = 0;
+ desc->pg_nolayout = 0;
desc->pg_inode = inode;
desc->pg_ops = pg_ops;
desc->pg_completion_ops = compl_ops;
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index ec6b771b8dbe..005d4b795b80 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -693,6 +693,8 @@ int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc)
nfs_pageio_complete(&pgio);
if (err == -EAGAIN && mntflags & NFS_MOUNT_SOFTERR)
break;
+ if (err == -EAGAIN && pgio.pg_nolayout)
+ ssleep(1);
} while (err < 0 && !nfs_error_is_fatal(err));
nfs_io_completion_put(ioc);
diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
index d64332afc374..f9b22b990de7 100644
--- a/include/linux/nfs_page.h
+++ b/include/linux/nfs_page.h
@@ -125,6 +125,11 @@ struct nfs_pageio_descriptor {
* layout: the write goes through the MDS instead of waiting for one.
*/
unsigned char pg_allow_mds : 1;
+ /*
+ * Set by a layout driver that refused a write with -EAGAIN for lack
+ * of a layout, so that a caller that resends waits first.
+ */
+ unsigned char pg_nolayout : 1;
};
/* arbitrarily selected limit to number of mirrors */
--
2.53.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
2026-11-18 18:36 ` [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington
3 siblings, 0 replies; 8+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
Nothing tests NFS4_FF_HDR_NO_IO_THRU_MDS since writes moved to the
inode's NFS_INO_NO_IO_THRU_MDS bit. Remove it, its helper, and
nfs4_flexfile_layout::flags, which held only that bit.
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
fs/nfs/flexfilelayout/flexfilelayout.c | 8 ++------
fs/nfs/flexfilelayout/flexfilelayout.h | 16 ----------------
fs/nfs/internal.h | 9 ++++-----
3 files changed, 6 insertions(+), 27 deletions(-)
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 4eb12bbb7838..a522c3fe49e5 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -641,12 +641,8 @@ ff_layout_alloc_lseg(struct pnfs_layout_hdr *lh,
if (!p)
goto out_sort_mirrors;
fls->flags = be32_to_cpup(p);
- if (fls->flags & FF_FLAGS_NO_IO_THRU_MDS) {
- set_bit(NFS4_FF_HDR_NO_IO_THRU_MDS,
- &FF_LAYOUT_FROM_HDR(lh)->flags);
- /* Outlives this layout hdr; see NFS_INO_NO_IO_THRU_MDS */
+ if (fls->flags & FF_FLAGS_NO_IO_THRU_MDS)
nfs_set_no_io_thru_mds(lh->plh_inode);
- }
p = xdr_inline_decode(&stream, 4);
if (!p)
@@ -1145,7 +1141,7 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,
* MDS there is no fallback to take, so wait for a layout on the
* same terms as the -EAGAIN above. The layout hdr that carried
* FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
- * this asks the inode and not the hdr.
+ * the flag is kept on the inode.
*/
if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
goto out_mds;
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.h b/fs/nfs/flexfilelayout/flexfilelayout.h
index 5d5587fded76..bb60537b2b93 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.h
+++ b/fs/nfs/flexfilelayout/flexfilelayout.h
@@ -113,16 +113,12 @@ struct nfs4_ff_layout_segment {
struct nfs4_ff_layout_mirror *mirror_array[] __counted_by(mirror_array_cnt);
};
-/* nfs4_flexfile_layout::flags bit indices */
-#define NFS4_FF_HDR_NO_IO_THRU_MDS 0 /* any lseg has had FF_FLAGS_NO_IO_THRU_MDS */
-
struct nfs4_flexfile_layout {
struct pnfs_layout_hdr generic_hdr;
struct pnfs_ds_commit_info commit_info;
struct list_head mirrors;
struct list_head error_list; /* nfs4_ff_layout_ds_err */
ktime_t last_report_time; /* Layoutstat report times */
- unsigned long flags;
};
/*
@@ -186,18 +182,6 @@ ff_layout_no_fallback_to_mds(struct pnfs_layout_segment *lseg)
return FF_LAYOUT_LSEG(lseg)->flags & FF_FLAGS_NO_IO_THRU_MDS;
}
-/*
- * Sticky hdr-level mirror of FF_FLAGS_NO_IO_THRU_MDS so callers that have
- * no current lseg (e.g. between LAYOUTRETURN and the next LAYOUTGET) can
- * still honor the no-MDS-fallback policy.
- */
-static inline bool
-ff_layout_hdr_no_fallback_to_mds(struct pnfs_layout_hdr *lo)
-{
- return test_bit(NFS4_FF_HDR_NO_IO_THRU_MDS,
- &FF_LAYOUT_FROM_HDR(lo)->flags);
-}
-
static inline bool
ff_layout_no_read_on_rw(struct pnfs_layout_segment *lseg)
{
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 5af6a032f145..5b40364fc38f 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -556,11 +556,10 @@ static inline bool nfs_file_io_is_buffered(struct nfs_inode *nfsi)
* FF_FLAGS_NO_IO_THRU_MDS in ff_layout_alloc_lseg()). The RFC 5661
* mdsthreshold hint must not be acted on for such a file, since the only
* thing pnfs_within_mdsthreshold() can ask for is the one thing the
- * layout forbids. Servers are assumed to be consistent in their
- * no-fallback policy per file, the same assumption
- * ff_layout_hdr_no_fallback_to_mds() already makes; if one were not, the
- * only effect is that its mdsthreshold hint - a SHOULD - stops being
- * honored for an inode that is already in core.
+ * layout forbids. Flexfiles also retries reads and writes that have no
+ * layout segment instead of sending them through the MDS, except
+ * writeback of a single locked folio (pg_allow_mds). Servers are
+ * assumed to be consistent in their no-fallback policy per file.
*/
#define NFS_INO_NO_IO_THRU_MDS (30)
--
2.53.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
` (2 preceding siblings ...)
2026-11-18 18:36 ` [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
@ 2026-11-18 18:36 ` Benjamin Coddington
3 siblings, 0 replies; 8+ messages in thread
From: Benjamin Coddington @ 2026-11-18 18:36 UTC (permalink / raw)
To: Trond Myklebust, Anna Schumaker, Mike Snitzer; +Cc: linux-nfs
From: Benjamin Coddington <bcodding@hammerspace.com>
When LAYOUTGET fails with NFS4ERR_LAYOUTUNAVAILABLE, or with
NFS4ERR_TOOSMALL after the retry at the session's maximum reply size,
pnfs_update_layout() sets the layout's fail bit for that iomode and
returns no layout without sending another LAYOUTGET for the next 120
seconds. If the file's layouts carried FF_FLAGS_NO_IO_THRU_MDS, reads
and writes wait for a layout the whole time, and then send a LAYOUTGET
that can fail the same way. On a hard mount that repeats for as long
as the server keeps answering that way.
RFC 8435 says that after NFS4ERR_LAYOUTUNAVAILABLE "the I/O is to go to
the metadata server" (section 5.1.1), and that FF_FLAGS_NO_IO_THRU_MDS
is a hint (section 5.1.2). Send the I/O through the MDS while the fail
bit is set, in ff_layout_inode_no_fallback_to_mds(), which reads,
writes and the resend after a data server error share. This also lets
a server deal with a data server the client can't use: the client
reports the failure with LAYOUTERROR (or in LAYOUTRETURN) and returns
the layout, and the server can answer the next LAYOUTGET with
NFS4ERR_LAYOUTUNAVAILABLE.
Fixes: 2f007ba074ad ("pNFS/flexfiles: don't read through MDS when previous layout forbid it")
Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in pg_get_mirror_count_write")
Fixes: ec906cf7bee6 ("pNFS/flexfiles: don't reset to MDS for v4 error when previous layout forbid it")
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
fs/nfs/flexfilelayout/flexfilelayout.c | 40 +++++++++++++++-----------
fs/nfs/internal.h | 6 ++--
fs/nfs/pnfs.c | 22 ++++++++++++++
fs/nfs/pnfs.h | 1 +
4 files changed, 50 insertions(+), 19 deletions(-)
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index a522c3fe49e5..c4fae5408535 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -980,10 +980,14 @@ static bool ff_layout_pg_retry_wait(struct nfs_pageio_descriptor *pgio)
/*
* Returns true if the inode's layouts forbid I/O through the MDS.
+ * FF_FLAGS_NO_IO_THRU_MDS is a hint (RFC 8435, section 5.1.2), so set it
+ * aside while a failed LAYOUTGET for @iomode (NFS4ERR_LAYOUTUNAVAILABLE or
+ * NFS4ERR_TOOSMALL) keeps pnfs_update_layout() from sending another.
*/
static bool ff_layout_inode_no_fallback_to_mds(struct inode *inode, u32 iomode)
{
- return nfs_no_io_thru_mds(inode);
+ return nfs_no_io_thru_mds(inode) &&
+ !pnfs_layout_get_failed(inode, iomode);
}
/*
@@ -1135,13 +1139,12 @@ ff_layout_pg_init_read(struct nfs_pageio_descriptor *pgio,
}
/*
* No segment, and no error to report either: pnfs_update_layout()
- * simply has nothing to give (NFS_LAYOUT_BULK_RECALL, a failed
- * pnfs_layout_io_test, blocked LAYOUTGETs, a layout being
- * returned). If the server forbids reading this file through the
- * MDS there is no fallback to take, so wait for a layout on the
- * same terms as the -EAGAIN above. The layout hdr that carried
- * FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
- * the flag is kept on the inode.
+ * simply has nothing to give (NFS_LAYOUT_BULK_RECALL, blocked
+ * LAYOUTGETs, a layout being returned). If the server forbids
+ * reading this file through the MDS there is no fallback to take,
+ * so wait for a layout on the same terms as the -EAGAIN above. The
+ * layout hdr that carried FF_FLAGS_NO_IO_THRU_MDS may itself be
+ * gone by now, which is why the flag is kept on the inode.
*/
if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
goto out_mds;
@@ -1395,7 +1398,8 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
struct nfs4_state *state,
struct nfs_client *clp,
struct pnfs_layout_segment *lseg,
- struct nfs4_deviceid_node *devid)
+ struct nfs4_deviceid_node *devid,
+ u32 iomode)
{
struct pnfs_layout_hdr *lo = lseg->pls_layout;
struct inode *inode = lo->plh_inode;
@@ -1484,8 +1488,9 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
return -NFS4ERR_RESET_TO_PNFS;
reset:
/*
- * FF_FLAGS_NO_IO_THRU_MDS: never resend through the MDS. The
- * caller would do so with force_mds set (ff_layout_reset_read(),
+ * FF_FLAGS_NO_IO_THRU_MDS: don't resend through the MDS (see
+ * ff_layout_inode_no_fallback_to_mds()). The caller would do so
+ * with force_mds set (ff_layout_reset_read(),
* ff_layout_reset_write(hdr, false) -> pnfs_*_done_resend_to_mds()),
* which builds a descriptor out of the plain MDS page ops and so
* consults no layout at all -- this is the last point at which the
@@ -1495,7 +1500,7 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
* pnfs_destroy_layout(). Having done so they can no longer ask the
* layout hdr about the flag, hence the inode.
*/
- if (nfs_no_io_thru_mds(inode)) {
+ if (ff_layout_inode_no_fallback_to_mds(inode, iomode)) {
dprintk("%s Retry through pNFS, no MDS fallback. Error %d\n",
__func__, task->tk_status);
return -NFS4ERR_RESET_TO_PNFS;
@@ -1580,7 +1585,8 @@ static int ff_layout_async_handle_error(struct rpc_task *task,
struct nfs4_state *state,
struct nfs_client *clp,
struct pnfs_layout_segment *lseg,
- struct nfs4_deviceid_node *devid)
+ struct nfs4_deviceid_node *devid,
+ u32 iomode)
{
int vers = clp->cl_nfs_mod->rpc_vers->number;
@@ -1600,7 +1606,7 @@ static int ff_layout_async_handle_error(struct rpc_task *task,
lseg, devid);
case 4:
return ff_layout_async_handle_error_v4(task, op_status, state,
- clp, lseg, devid);
+ clp, lseg, devid, iomode);
default:
/* should never happen */
WARN_ON_ONCE(1);
@@ -1734,7 +1740,7 @@ static int ff_layout_read_done_cb(struct rpc_task *task,
err = ff_layout_async_handle_error(task, hdr->res.op_status,
hdr->args.context->state,
hdr->ds_clp, hdr->lseg,
- hdr->ds_dev);
+ hdr->ds_dev, IOMODE_READ);
trace_nfs4_pnfs_read(hdr, err);
clear_bit(NFS_IOHDR_RESEND_PNFS, &hdr->flags);
@@ -1944,7 +1950,7 @@ static int ff_layout_write_done_cb(struct rpc_task *task,
err = ff_layout_async_handle_error(task, hdr->res.op_status,
hdr->args.context->state,
hdr->ds_clp, hdr->lseg,
- hdr->ds_dev);
+ hdr->ds_dev, IOMODE_RW);
trace_nfs4_pnfs_write(hdr, err);
clear_bit(NFS_IOHDR_RESEND_PNFS, &hdr->flags);
@@ -1995,7 +2001,7 @@ static int ff_layout_commit_done_cb(struct rpc_task *task,
err = ff_layout_async_handle_error(task, data->res.op_status,
NULL, data->ds_clp, data->lseg,
- data->ds_dev);
+ data->ds_dev, IOMODE_RW);
trace_nfs4_pnfs_commit_ds(data, err);
switch (err) {
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 5b40364fc38f..fc2a27b5126a 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -558,8 +558,10 @@ static inline bool nfs_file_io_is_buffered(struct nfs_inode *nfsi)
* thing pnfs_within_mdsthreshold() can ask for is the one thing the
* layout forbids. Flexfiles also retries reads and writes that have no
* layout segment instead of sending them through the MDS, except
- * writeback of a single locked folio (pg_allow_mds). Servers are
- * assumed to be consistent in their no-fallback policy per file.
+ * writeback of a single locked folio (pg_allow_mds), and unless a
+ * LAYOUTGET for that iomode has failed with NFS4ERR_LAYOUTUNAVAILABLE or
+ * NFS4ERR_TOOSMALL. Servers are assumed to be consistent in their
+ * no-fallback policy per file.
*/
#define NFS_INO_NO_IO_THRU_MDS (30)
diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 14f4a4b12ffd..582fe81fc2d1 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -551,6 +551,28 @@ pnfs_layout_io_test_failed(struct pnfs_layout_hdr *lo, u32 iomode)
return true;
}
+/*
+ * Returns true if a LAYOUTGET for @iomode failed with an error, such as
+ * NFS4ERR_LAYOUTUNAVAILABLE, in the last PNFS_LAYOUTGET_RETRY_TIMEOUT.
+ * pnfs_layout_io_test_failed() clears an expired fail bit only when
+ * pnfs_update_layout() gets that far, so check the time here too.
+ */
+bool pnfs_layout_get_failed(struct inode *inode, u32 iomode)
+{
+ struct pnfs_layout_hdr *lo;
+ unsigned long now = jiffies;
+ bool ret = false;
+
+ spin_lock(&inode->i_lock);
+ lo = NFS_I(inode)->layout;
+ if (lo && test_bit(pnfs_iomode_to_fail_bit(iomode), &lo->plh_flags))
+ ret = time_in_range(lo->plh_retry_timestamp,
+ now - PNFS_LAYOUTGET_RETRY_TIMEOUT, now);
+ spin_unlock(&inode->i_lock);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(pnfs_layout_get_failed);
+
static void
pnfs_init_lseg(struct pnfs_layout_hdr *lo, struct pnfs_layout_segment *lseg,
const struct pnfs_layout_range *range,
diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
index 2774d4adf4c3..0007a4ca3da9 100644
--- a/fs/nfs/pnfs.h
+++ b/fs/nfs/pnfs.h
@@ -296,6 +296,7 @@ int pnfs_generic_pg_writepages(struct nfs_pageio_descriptor *desc);
size_t pnfs_generic_pg_test(struct nfs_pageio_descriptor *pgio,
struct nfs_page *prev, struct nfs_page *req);
void pnfs_set_lo_fail(struct pnfs_layout_segment *lseg);
+bool pnfs_layout_get_failed(struct inode *inode, u32 iomode);
struct pnfs_layout_segment *pnfs_layout_process(struct nfs4_layoutget *lgp);
void pnfs_layoutget_free(struct nfs4_layoutget *lgp);
void pnfs_free_lseg_list(struct list_head *tmp_list);
--
2.53.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
@ 2026-11-18 23:22 ` Anna Schumaker
2026-11-18 23:56 ` Benjamin Coddington
0 siblings, 1 reply; 8+ messages in thread
From: Anna Schumaker @ 2026-11-18 23:22 UTC (permalink / raw)
To: Benjamin Coddington, Trond Myklebust, Mike Snitzer; +Cc: linux-nfs
Hi Ben,
On Wed, Nov 18, 2026, at 1:36 PM, Benjamin Coddington wrote:
> From: Benjamin Coddington <bcodding@hammerspace.com>
>
> nfs_writepage_locked() writes back one folio that its caller holds
> locked, for nfs_wb_folio(), nfs_wb_folio_reclaim() and migration. When
> there is no layout segment and the file's layout header carries
> FF_FLAGS_NO_IO_THRU_MDS, ff_layout_pg_get_mirror_count_write() refuses
> the write with -EAGAIN. nfs_do_writepage() redirties the folio and
> returns the -EAGAIN, and nfs_wb_folio() passes it to nfs_updatepage(),
> nfs_flush_incompatible(), nfs_read_folio() and nfs_launder_folio(). A
> read(2) or write(2) on a blocking file then fails with EAGAIN, for
> example when another open context has dirtied the same folio while a
> layout is being recalled.
>
> Resending from nfs_wb_folio(), as nfs_writepages() does, would hold the
> folio locked across the resends. A bulk recall can't complete while
> nfs_writepages() holds a layout segment, and nfs_writepages() would be
> waiting for that folio's lock.
>
> FF_FLAGS_NO_IO_THRU_MDS is a hint (RFC 8435, section 5.1.2). Mark the
> descriptor that nfs_writepage_locked() uses with a new pg_allow_mds
> bit, and send its write through the MDS when there is no layout
> segment. When ff_layout_pg_init_write() can't use a data server, send
> such a write to the MDS too, instead of retrying with the folio locked.
>
> ff_layout_pg_init_write() switches a write to the MDS by resetting the
> descriptor and returning -EAGAIN, which nfs_wb_folio() also passes up,
> with or without the flag. When nfs_wb_folio() gets -EAGAIN, wait for
> any part of the folio already under writeback, as its loop does, and
> write the folio once more with a descriptor that goes straight to the
> MDS. nfs_wb_folio_reclaim() still returns the error, and
> nfs_release_folio() keeps the folio.
>
> Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in
> pg_get_mirror_count_write")
> Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
> ---
> fs/nfs/flexfilelayout/flexfilelayout.c | 5 +++--
> fs/nfs/pagelist.c | 1 +
> fs/nfs/write.c | 19 +++++++++++++++----
> include/linux/nfs_page.h | 5 +++++
> 4 files changed, 24 insertions(+), 6 deletions(-)
>
> diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c
> b/fs/nfs/flexfilelayout/flexfilelayout.c
> index 5c289737366f..2b0a0af469fb 100644
> --- a/fs/nfs/flexfilelayout/flexfilelayout.c
> +++ b/fs/nfs/flexfilelayout/flexfilelayout.c
> @@ -1176,7 +1176,8 @@ ff_layout_pg_init_write(struct
> nfs_pageio_descriptor *pgio,
> mirror_ds, dss_id, OP_WRITE);
> if (IS_ERR(ds)) {
> nfs4_ff_layout_put_deviceid(mirror_ds);
> - if (!ff_layout_no_fallback_to_mds(pgio->pg_lseg))
> + if (pgio->pg_allow_mds ||
> + !ff_layout_no_fallback_to_mds(pgio->pg_lseg))
> goto out_mds;
> pnfs_generic_pg_cleanup(pgio);
> /* Sleep for 1 second before retrying */
> @@ -1229,7 +1230,7 @@ ff_layout_pg_get_mirror_count_write(struct
> nfs_pageio_descriptor *pgio,
> 0, NFS4_MAX_UINT64, IOMODE_RW,
> NFS_I(pgio->pg_inode)->layout,
> pgio->pg_lseg);
> - if (NFS_I(pgio->pg_inode)->layout &&
> + if (!pgio->pg_allow_mds && NFS_I(pgio->pg_inode)->layout &&
> ff_layout_hdr_no_fallback_to_mds(NFS_I(pgio->pg_inode)->layout)) {
> /*
> * FF_FLAGS_NO_IO_THRU_MDS: no current lseg but the server's
> diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> index 71f0ce2bc4ea..aa6d7b658d79 100644
> --- a/fs/nfs/pagelist.c
> +++ b/fs/nfs/pagelist.c
> @@ -871,6 +871,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
> *desc,
> int io_flags)
> {
> desc->pg_moreio = 0;
> + desc->pg_allow_mds = 0;
> desc->pg_inode = inode;
> desc->pg_ops = pg_ops;
> desc->pg_completion_ops = compl_ops;
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index 0b42d31f7a75..ec6b771b8dbe 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -627,15 +627,16 @@ static int nfs_do_writepage(struct folio *folio,
> struct writeback_control *wbc,
> * Write an mmapped page to the server.
> */
> static int nfs_writepage_locked(struct folio *folio,
> - struct writeback_control *wbc)
> + struct writeback_control *wbc, bool force_mds)
> {
> struct nfs_pageio_descriptor pgio;
> struct inode *inode = folio->mapping->host;
> int err;
>
> nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
> - nfs_pageio_init_write(&pgio, inode, 0, false,
> + nfs_pageio_init_write(&pgio, inode, 0, force_mds,
> &nfs_async_write_completion_ops);
> + pgio.pg_allow_mds = 1;
> err = nfs_do_writepage(folio, wbc, &pgio);
> pgio.pg_error = 0;
> nfs_pageio_complete(&pgio);
> @@ -2057,7 +2058,7 @@ int nfs_wb_folio_reclaim(struct inode *inode,
> struct folio *folio)
> return -EBUSY;
> if (folio_clear_dirty_for_io(folio)) {
> trace_nfs_writeback_folio_reclaim(inode, range_start, len);
> - ret = nfs_writepage_locked(folio, &wbc);
> + ret = nfs_writepage_locked(folio, &wbc, false);
> trace_nfs_writeback_folio_reclaim_done(inode, range_start, len,
> ret);
> return ret;
> @@ -2084,6 +2085,7 @@ int nfs_wb_folio(struct inode *inode, struct
> folio *folio)
> .range_start = range_start,
> .range_end = range_start + len - 1,
> };
> + bool force_mds = false;
> int ret;
>
> trace_nfs_writeback_folio(inode, range_start, len);
> @@ -2091,7 +2093,16 @@ int nfs_wb_folio(struct inode *inode, struct
> folio *folio)
> for (;;) {
> folio_wait_writeback(folio);
> if (folio_clear_dirty_for_io(folio)) {
> - ret = nfs_writepage_locked(folio, &wbc);
> + ret = nfs_writepage_locked(folio, &wbc, force_mds);
> + /*
> + * -EAGAIN asks for a resend, which can't wait for a
> + * layout with the folio locked: resend it through the
> + * MDS once any part of it already sent has completed.
> + */
> + if (ret == -EAGAIN && !force_mds) {
> + force_mds = true;
> + continue;
I think I've managed to confuse myself thinking about this. If ret = -EAGAIN,
we restart the loop with 'force_mds = true'. Won't folio_clear_dirty_for_io()
return 'false' the second time around and bypass the nfs_writepage_locked() call?
I would not be surprised if I'm misunderstanding how something works here.
> + }
> if (ret < 0)
> goto out_error;
> continue;
> diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> index c38e4b380be5..d64332afc374 100644
> --- a/include/linux/nfs_page.h
> +++ b/include/linux/nfs_page.h
> @@ -120,6 +120,11 @@ struct nfs_pageio_descriptor {
> u32 pg_mirror_idx; /* current mirror */
> unsigned short pg_maxretrans;
> unsigned char pg_moreio : 1;
> + /*
> + * Set when the caller holds the folio locked and can't wait for a
> + * layout: the write goes through the MDS instead of waiting for one.
> + */
Any way to whittle down this comment to just a few words so it matches the
style that we usually use? (See pg_mirror_idx a few lines above it).
Thanks,
Anna
> + unsigned char pg_allow_mds : 1;
> };
>
> /* arbitrarily selected limit to number of mirrors */
> --
> 2.53.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS
2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
@ 2026-11-18 23:52 ` Anna Schumaker
0 siblings, 0 replies; 8+ messages in thread
From: Anna Schumaker @ 2026-11-18 23:52 UTC (permalink / raw)
To: Benjamin Coddington, Trond Myklebust, Mike Snitzer; +Cc: linux-nfs
Hi Ben,
On Wed, Nov 18, 2026, at 1:36 PM, Benjamin Coddington wrote:
> From: Benjamin Coddington <bcodding@hammerspace.com>
>
> When a write has no layout segment and its file's layouts carry
> FF_FLAGS_NO_IO_THRU_MDS, ff_layout_pg_get_mirror_count_write() returns
> -EAGAIN. nfs_writepages() and the O_DIRECT reschedule path resend it at
> once, so writers spin for as long as pnfs_update_layout() has no layout
> to give: while NFS_LAYOUT_BULK_RECALL is set, while LAYOUTGETs are
> blocked, and for the 120 seconds after NFS4ERR_LAYOUTUNAVAILABLE.
>
> The check also tests a bit on the layout header. Once that header has
> been freed, a new one starts without the bit and the write goes through
> the MDS. ff_layout_pg_init_write() has no check at all: when
> pnfs_generic_pg_check_layout() drops an invalidated segment, the rest of
> the descriptor's writes go through the MDS.
>
> Handle both places the way ff_layout_pg_init_read() handles reads: test
> the inode's NFS_INO_NO_IO_THRU_MDS bit, return -ERESTARTSYS to a killed
> or exiting task, and honour io_maxretrans on soft mounts. Mark a
> refusal with a new pg_nolayout descriptor bit, and have nfs_writepages()
> and nfs_direct_write_reschedule() wait a second before they resend,
> after the pass that was refused has unlocked its folios and requests.
> Writes marked pg_allow_mds by nfs_writepage_locked() still go through
> the MDS.
>
> The test is ff_layout_inode_no_fallback_to_mds(), which
> ff_layout_pg_init_read() now uses too, and the refusal is
> ff_layout_pg_refuse_write(). Emit the MDS-fallback tracepoint only
> when the write does go through the MDS, and a new
> pnfs_mds_fallback_write_refused tracepoint when it is refused.
>
> Fixes: 1d62e659c0bf ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS in
> pg_get_mirror_count_write")
> Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
> ---
> fs/nfs/direct.c | 2 +
> fs/nfs/flexfilelayout/flexfilelayout.c | 66 ++++++++++++++++++++------
> fs/nfs/nfs4trace.c | 1 +
> fs/nfs/nfs4trace.h | 1 +
> fs/nfs/pagelist.c | 1 +
> fs/nfs/write.c | 2 +
> include/linux/nfs_page.h | 5 ++
> 7 files changed, 64 insertions(+), 14 deletions(-)
>
> diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
> index a3c6e8f4ea06..e6a2c6436b44 100644
> --- a/fs/nfs/direct.c
> +++ b/fs/nfs/direct.c
> @@ -626,6 +626,8 @@ static void nfs_direct_write_reschedule(struct
> nfs_direct_req *dreq)
> }
> }
>
> + if (desc.pg_error == -EAGAIN && desc.pg_nolayout)
> + ssleep(1);
Is there any possibility that the sleep length would be modified sometime
in the future? I ask, because the nfs_writepages() gets the same modification
with a hardcoded "1" here and my worry is that if we ever need to change this
length in the future, we might accidentally update one function and miss the
other. Maybe this whole if-statement would be better as an inline function that
takes the error and struct nfs_pageio_descriptor?
Alternatively, is there any way to handle the sleep directly in the flexfiles
code and not in the generic client since that's the only way to reach this point?
Thanks,
Anna
> if (put_dreq(dreq))
> nfs_direct_write_complete(dreq);
> }
> diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c
> b/fs/nfs/flexfilelayout/flexfilelayout.c
> index 2b0a0af469fb..4eb12bbb7838 100644
> --- a/fs/nfs/flexfilelayout/flexfilelayout.c
> +++ b/fs/nfs/flexfilelayout/flexfilelayout.c
> @@ -982,6 +982,45 @@ static bool ff_layout_pg_retry_wait(struct
> nfs_pageio_descriptor *pgio)
> return true;
> }
>
> +/*
> + * Returns true if the inode's layouts forbid I/O through the MDS.
> + */
> +static bool ff_layout_inode_no_fallback_to_mds(struct inode *inode,
> u32 iomode)
> +{
> + return nfs_no_io_thru_mds(inode);
> +}
> +
> +/*
> + * Refuse a write that has no layout segment and can't go through the
> MDS:
> + * -EAGAIN so that the caller resends it, or a hard error when the
> caller
> + * should give up. pg_nolayout tells a caller that resends at once to
> + * wait first, once it has dropped its locks.
> + */
> +static void ff_layout_pg_refuse_write(struct nfs_pageio_descriptor
> *pgio,
> + struct nfs_page *req)
> +{
> + struct nfs_server *server = NFS_SERVER(pgio->pg_inode);
> +
> + trace_pnfs_mds_fallback_write_refused(pgio->pg_inode, req_offset(req),
> + req->wb_bytes, IOMODE_RW,
> + NFS_I(pgio->pg_inode)->layout,
> + NULL);
> + if (server->flags & (NFS_MOUNT_SOFT | NFS_MOUNT_SOFTERR) &&
> + io_maxretrans && req->wb_nio > io_maxretrans) {
> + if (server->flags & NFS_MOUNT_SOFTERR)
> + pgio->pg_error = -ETIMEDOUT;
> + else
> + pgio->pg_error = -EIO;
> + return;
> + }
> + if (fatal_signal_pending(current) || nfs_current_task_exiting()) {
> + pgio->pg_error = -ERESTARTSYS;
> + return;
> + }
> + pgio->pg_nolayout = 1;
> + pgio->pg_error = -EAGAIN;
> +}
> +
> static void
> ff_layout_pg_get_read(struct nfs_pageio_descriptor *pgio,
> struct nfs_page *req,
> @@ -1108,7 +1147,7 @@ ff_layout_pg_init_read(struct
> nfs_pageio_descriptor *pgio,
> * FF_FLAGS_NO_IO_THRU_MDS may itself be gone by now, which is why
> * this asks the inode and not the hdr.
> */
> - if (!nfs_no_io_thru_mds(pgio->pg_inode))
> + if (!ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_READ))
> goto out_mds;
> retry_nolseg:
> /* Retry getting layout segment if lower layer returned -EAGAIN */
> @@ -1156,9 +1195,13 @@ ff_layout_pg_init_write(struct
> nfs_pageio_descriptor *pgio,
> return;
> }
> }
> - /* If no lseg, fall back to write through mds */
> - if (pgio->pg_lseg == NULL)
> - goto out_mds;
> + if (!pgio->pg_lseg) {
> + if (pgio->pg_allow_mds ||
> + !ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_RW))
> + goto out_mds;
> + ff_layout_pg_refuse_write(pgio, req);
> + return;
> + }
>
> /* Use a direct mapping of ds_idx to pgio mirror_idx */
> if (pgio->pg_mirror_count != FF_LAYOUT_MIRROR_COUNT(pgio->pg_lseg))
> @@ -1225,21 +1268,16 @@ ff_layout_pg_get_mirror_count_write(struct
> nfs_pageio_descriptor *pgio,
> }
> if (pgio->pg_lseg)
> return FF_LAYOUT_MIRROR_COUNT(pgio->pg_lseg);
> + if (!pgio->pg_allow_mds &&
> + ff_layout_inode_no_fallback_to_mds(pgio->pg_inode, IOMODE_RW)) {
> + ff_layout_pg_refuse_write(pgio, req);
> + goto out;
> + }
>
> trace_pnfs_mds_fallback_pg_get_mirror_count(pgio->pg_inode,
> 0, NFS4_MAX_UINT64, IOMODE_RW,
> NFS_I(pgio->pg_inode)->layout,
> pgio->pg_lseg);
> - if (!pgio->pg_allow_mds && NFS_I(pgio->pg_inode)->layout &&
> - ff_layout_hdr_no_fallback_to_mds(NFS_I(pgio->pg_inode)->layout)) {
> - /*
> - * FF_FLAGS_NO_IO_THRU_MDS: no current lseg but the server's
> - * policy forbids MDS fallback. Surface -EAGAIN so writeback
> - * retries rather than silently issuing the WRITE via MDS.
> - */
> - pgio->pg_error = -EAGAIN;
> - goto out;
> - }
> /* no lseg means that pnfs is not in use, so no mirroring here */
> nfs_pageio_reset_write_mds(pgio);
> out:
> diff --git a/fs/nfs/nfs4trace.c b/fs/nfs/nfs4trace.c
> index 3fdc013f56d8..93a84f546de8 100644
> --- a/fs/nfs/nfs4trace.c
> +++ b/fs/nfs/nfs4trace.c
> @@ -25,6 +25,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_read_done);
> EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_done);
> EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_read_pagelist);
> EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_pagelist);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_mds_fallback_write_refused);
> EXPORT_TRACEPOINT_SYMBOL_GPL(pnfs_ds_connect);
>
> EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_read_error);
> diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h
> index b5c89eeef2bc..94ed6b833a28 100644
> --- a/fs/nfs/nfs4trace.h
> +++ b/fs/nfs/nfs4trace.h
> @@ -2115,6 +2115,7 @@ DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_read_done);
> DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_done);
> DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_read_pagelist);
> DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_pagelist);
> +DEFINE_PNFS_LAYOUT_EVENT(pnfs_mds_fallback_write_refused);
>
> DECLARE_EVENT_CLASS(nfs4_deviceid_event,
> TP_PROTO(
> diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> index aa6d7b658d79..9a8731de6f5b 100644
> --- a/fs/nfs/pagelist.c
> +++ b/fs/nfs/pagelist.c
> @@ -872,6 +872,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor
> *desc,
> {
> desc->pg_moreio = 0;
> desc->pg_allow_mds = 0;
> + desc->pg_nolayout = 0;
> desc->pg_inode = inode;
> desc->pg_ops = pg_ops;
> desc->pg_completion_ops = compl_ops;
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index ec6b771b8dbe..005d4b795b80 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -693,6 +693,8 @@ int nfs_writepages(struct address_space *mapping,
> struct writeback_control *wbc)
> nfs_pageio_complete(&pgio);
> if (err == -EAGAIN && mntflags & NFS_MOUNT_SOFTERR)
> break;
> + if (err == -EAGAIN && pgio.pg_nolayout)
> + ssleep(1);
> } while (err < 0 && !nfs_error_is_fatal(err));
> nfs_io_completion_put(ioc);
>
> diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> index d64332afc374..f9b22b990de7 100644
> --- a/include/linux/nfs_page.h
> +++ b/include/linux/nfs_page.h
> @@ -125,6 +125,11 @@ struct nfs_pageio_descriptor {
> * layout: the write goes through the MDS instead of waiting for one.
> */
> unsigned char pg_allow_mds : 1;
> + /*
> + * Set by a layout driver that refused a write with -EAGAIN for lack
> + * of a layout, so that a caller that resends waits first.
> + */
> + unsigned char pg_nolayout : 1;
> };
>
> /* arbitrarily selected limit to number of mirrors */
> --
> 2.53.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout
2026-11-18 23:22 ` Anna Schumaker
@ 2026-11-18 23:56 ` Benjamin Coddington
0 siblings, 0 replies; 8+ messages in thread
From: Benjamin Coddington @ 2026-11-18 23:56 UTC (permalink / raw)
To: Anna Schumaker
Cc: Benjamin Coddington, Trond Myklebust, Mike Snitzer, linux-nfs
On 18 Nov 2026, at 15:22, Anna Schumaker wrote:
> Hi Ben,
>
> On Wed, Nov 18, 2026, at 1:36 PM, Benjamin Coddington wrote:
...
> trace_nfs_writeback_folio(inode, range_start, len);
>> @@ -2091,7 +2093,16 @@ int nfs_wb_folio(struct inode *inode, struct
>> folio *folio)
>> for (;;) {
>> folio_wait_writeback(folio);
>> if (folio_clear_dirty_for_io(folio)) {
>> - ret = nfs_writepage_locked(folio, &wbc);
>> + ret = nfs_writepage_locked(folio, &wbc, force_mds);
>> + /*
>> + * -EAGAIN asks for a resend, which can't wait for a
>> + * layout with the folio locked: resend it through the
>> + * MDS once any part of it already sent has completed.
>> + */
>> + if (ret == -EAGAIN && !force_mds) {
>> + force_mds = true;
>> + continue;
>
> I think I've managed to confuse myself thinking about this. If ret = -EAGAIN,
> we restart the loop with 'force_mds = true'. Won't folio_clear_dirty_for_io()
> return 'false' the second time around and bypass the nfs_writepage_locked() call?
>
> I would not be surprised if I'm misunderstanding how something works here.
Its not obvious, but nfs_writepage_locked() -> nfs_do_writepage() redirties
the folio before it returns the -EAGAIN. That same pattern is in
nfs_writepages() which relies on it for its resend.
>
>> + }
>> if (ret < 0)
>> goto out_error;
>> continue;
>> diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
>> index c38e4b380be5..d64332afc374 100644
>> --- a/include/linux/nfs_page.h
>> +++ b/include/linux/nfs_page.h
>> @@ -120,6 +120,11 @@ struct nfs_pageio_descriptor {
>> u32 pg_mirror_idx; /* current mirror */
>> unsigned short pg_maxretrans;
>> unsigned char pg_moreio : 1;
>> + /*
>> + * Set when the caller holds the folio locked and can't wait for a
>> + * layout: the write goes through the MDS instead of waiting for one.
>> + */
>
> Any way to whittle down this comment to just a few words so it matches the
> style that we usually use? (See pg_mirror_idx a few lines above it).
You bet, will do.
I'm holding v2 for a couple of fixes that should go in ahead of this series,
and I'll fold in the sashiko findings on 2/4 as well.
Thanks for the look at this!
Ben
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-11-18 23:56 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 18:36 [PATCH 0/4] pNFS/flexfiles: I/O without a layout under FF_FLAGS_NO_IO_THRU_MDS Benjamin Coddington
2026-11-18 18:36 ` [PATCH 1/4] pNFS/flexfiles: use the MDS for locked-folio writeback without a layout Benjamin Coddington
2026-11-18 23:22 ` Anna Schumaker
2026-11-18 23:56 ` Benjamin Coddington
2026-11-18 18:36 ` [PATCH 2/4] pNFS/flexfiles: pace writes that have no layout and can't use the MDS Benjamin Coddington
2026-11-18 23:52 ` Anna Schumaker
2026-11-18 18:36 ` [PATCH 3/4] pNFS/flexfiles: remove the layout-header NO_IO_THRU_MDS bit Benjamin Coddington
2026-11-18 18:36 ` [PATCH 4/4] pNFS/flexfiles: use the MDS when the server has no layout to give Benjamin Coddington
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.