[PATCH 0/6] NFS: size the LAYOUTGET reply buffer for wide flexfiles layouts

[PATCH 0/6] NFS: size the LAYOUTGET reply buffer for wide flexfiles layouts

[Benjamin Coddington]

The flexfiles layout driver caps its LAYOUTGET reply buffer at a single
page, which limits a striped layout segment to roughly 28 ff_data_server4
entries -- far below the 4096-stripe decode limit the client otherwise
advertises.  A server striping wider than that has no way to hand the
client a layout: it returns NFS4ERR_TOOSMALL, the client falls back to
I/O through the MDS, and pNFS never engages for those files.

This series first makes the client behave sanely when a layout does not
fit the size it advertised in loga_maxcount, and then lets the reply
buffer grow on demand up to the session's maximum response size.

Patch 1 is a standalone fix (Cc: stable).  A server's NFS4ERR_TOOSMALL
is mapped to -ETOOSMALL at decode and then handled nowhere, so the
client falls back to the MDS for that one request and re-sends a doomed
LAYOUTGET on every subsequent pageio attempt.  Suspend pNFS via the
layout fail bit instead, the way NFS4ERR_LAYOUTUNAVAILABLE already does.

Patches 2-5 make wide layouts work.  loga_maxcount is derived from the
reply buffer actually allocated rather than a fixed 4096 (block and SCSI
layouts were advertising 4KB against a session-sized buffer, so a server
whose extent list encodes larger than a page got a needless TOOSMALL).
A TOOSMALL LAYOUTGET is then retried once with the buffer raised to the
session's maximum response size -- the same bound GETDEVICEINFO already
uses -- and a non-conformant server that ignores loga_maxcount and
overruns the buffer outright takes that same recovery path instead of
today's -EINVAL.  Finally the escalated size is remembered per-server, so
subsequent opens skip the attempt that is known to fail; this is also
what lets the LAYOUTGET attached to OPEN succeed against a wide-striping
server, since that path is best-effort and has no retry of its own.

The common path is unchanged throughout: the first LAYOUTGET for a
layout still goes out with the layout driver's default reply buffer, and
larger buffers are only ever allocated against servers that actually
hand out wide layouts.

Patch 6 moves the decoded per-mirror stripe array to kvzalloc_objs(), so
that a wide stripe array does not depend on a high-order allocation
succeeding.

Wire-validated against reffs at stripe widths 2 and 64, exercising both
the conformant NFS4ERR_TOOSMALL path and the buffer-overrun path.

Two known gaps are deliberately left for follow-up work:

  - The decoded per-stripe footprint is heavy: nfs4_ff_layout_ds_stripe
    is roughly 300 bytes and embeds localio and layoutstats state that
    most stripes never use.  Both the structure and its array container
    want rework before very large stripe counts are comfortable.

  - The client decodes only the first logr_layout entry of a LAYOUTGET
    reply and silently discards the rest, so it can accept less coverage
    than it asked for in loga_minlength, and cannot amortize a
    multi-segment reply.

A related series, "NFS: flexfiles device notifications and caching for
wide striped layouts", handles CB_NOTIFY_DEVICEID and scales the device
caches for the layouts this one makes fetchable.  The two are
independent and apply cleanly in either order.

Benjamin Coddington (6):
  NFSv4.1/pnfs: suspend pNFS on NFS4ERR_TOOSMALL from LAYOUTGET
  NFSv4.1/pnfs: derive loga_maxcount from the LAYOUTGET reply buffer
  NFSv4.1/pnfs: retry LAYOUTGET with a larger reply buffer on
    NFS4ERR_TOOSMALL
  NFSv4.1/pnfs: treat an oversized LAYOUTGET reply as -EMSGSIZE
  NFSv4.1/pnfs: remember when a server needs a larger LAYOUTGET reply
    buffer
  NFSv4/flexfiles: allocate the per-mirror stripe array with
    kvzalloc_objs

 fs/nfs/flexfilelayout/flexfilelayout.c |  6 +--
 fs/nfs/nfs4proc.c                      |  9 +++++
 fs/nfs/nfs4xdr.c                       |  2 +-
 fs/nfs/pnfs.c                          | 54 +++++++++++++++++++++++---
 include/linux/nfs_fs_sb.h              |  4 ++
 5 files changed, 65 insertions(+), 10 deletions(-)

base-commit: db2ddb87143519e20a95aa36c60b36107b736a58
-- 
2.53.0

[PATCH 1/6] NFSv4.1/pnfs: suspend pNFS on NFS4ERR_TOOSMALL from LAYOUTGET

[Benjamin Coddington]

If the layout for the requested range is larger than the size the
client advertised in loga_maxcount, RFC 8881 Section 18.43.3 has the
metadata server return NFS4ERR_TOOSMALL.  The client caps
loga_maxcount at a single page, so a flexfiles server that stripes a
layout segment across several dozen data servers produces this error
today.

The client has no handling for it: nfs4_stat_to_errno() maps the
error to -ETOOSMALL during decode, which nothing in the layoutget
path recognizes and nfs_error_is_fatal() does not consider fatal, so
pnfs_update_layout() clears the layout fail bit and returns no
segment.  The I/O falls back to the MDS, but because no fail bit was
set, every subsequent pageio attempt sends another LAYOUTGET that is
doomed to the same NFS4ERR_TOOSMALL.  Files whose layouts do not fit
the reply buffer never use pNFS and pay an extra round trip on every
pageio.

Map -ETOOSMALL to -EMSGSIZE in the layoutget exception handler and
have pnfs_update_layout() treat it like NFS4ERR_LAYOUTUNAVAILABLE:
mark the layout mode as failed and fall back to I/O through the MDS.

Fixes: d600ad1f2bdb ("NFS41: pop some layoutget errors to application")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/nfs4proc.c | 9 +++++++++
 fs/nfs/pnfs.c     | 2 ++
 2 files changed, 11 insertions(+)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 5709c6fea85b..016e8b38b87b 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -9623,6 +9623,15 @@ nfs4_layoutget_handle_exception(struct rpc_task *task,
 	case -NFS4ERR_BADLAYOUT:
 		status = -EOVERFLOW;
 		goto out;
+	/*
+	 * NFS4ERR_TOOSMALL means the layout for the requested range
+	 * exceeds what the client advertised in loga_maxcount (see
+	 * RFC8881 section 18.43.3).  Note nfs4_stat_to_errno() has
+	 * already mapped it to -ETOOSMALL during decode.
+	 */
+	case -ETOOSMALL:
+		status = -EMSGSIZE;
+		goto out;
 	/*
 	 * NFS4ERR_LAYOUTTRYLATER is a conflict with another client
 	 * (or clients) writing to the same RAID stripe except when
diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 7715e2bd5871..10102bda6a38 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -2338,6 +2338,8 @@ pnfs_update_layout(struct inode *ino,
 			break;
 		case -ENODATA:
 			/* The server returned NFS4ERR_LAYOUTUNAVAILABLE */
+		case -EMSGSIZE:
+			/* The layout exceeded loga_maxcount (NFS4ERR_TOOSMALL) */
 			pnfs_layout_set_fail_bit(
 				lo, pnfs_iomode_to_fail_bit(iomode));
 			lseg = NULL;
-- 
2.53.0

[PATCH 2/6] NFSv4.1/pnfs: derive loga_maxcount from the LAYOUTGET reply buffer

[Benjamin Coddington]

The client advertises a fixed PNFS_LAYOUT_MAXSIZE (4096) in
loga_maxcount no matter what reply buffer it actually allocated.  The
two only happen to agree for layout drivers that cap
max_layoutget_response at a single page.  Block and SCSI layouts
allocate a session-sized reply buffer, yet still tell the server it
must not send more than 4KB of layout: a server whose extent list
encodes larger than a page returns NFS4ERR_TOOSMALL even though the
client has ample room to receive it.

Set loga_maxcount from the size of the page array allocated for the
reply.  The client never advertises more than it can receive, and any
future change to the reply buffer sizing automatically carries its
wire advertisement with it.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/pnfs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 10102bda6a38..88deefd45202 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -1209,7 +1209,7 @@ pnfs_alloc_init_layoutget_args(struct inode *ino,
 				lgp->args.minlength = i_size - range->offset;
 		}
 	}
-	lgp->args.maxcount = PNFS_LAYOUT_MAXSIZE;
+	lgp->args.maxcount = lgp->args.layout.pglen;
 	pnfs_copy_range(&lgp->args.range, range);
 	lgp->args.type = server->pnfs_curr_ld->id;
 	lgp->args.inode = ino;
-- 
2.53.0

[PATCH 3/6] NFSv4.1/pnfs: retry LAYOUTGET with a larger reply buffer on NFS4ERR_TOOSMALL

[Benjamin Coddington]

Layout drivers cap the LAYOUTGET reply buffer with
max_layoutget_response; the flexfiles driver caps it at a single page,
which limits a striped layout segment to roughly 28 stripes.  A server
striping wider than that returns NFS4ERR_TOOSMALL, and the client
falls back to I/O through the MDS -- pNFS never engages for those
files.

Instead of failing over to the MDS on the first NFS4ERR_TOOSMALL,
retry the LAYOUTGET once with the reply buffer raised to the session's
maximum response size, the same bound GETDEVICEINFO already uses.  The
server offers no size hint in the TOOSMALL error, and the page array
is transient (freed when the RPC completes), so a single jump to the
ceiling is preferred over incremental growth.  If the layout does not
fit even the session-sized buffer, fall back to the MDS as before.

The common path is unchanged: the first LAYOUTGET for a layout is
still sent with the driver's default reply buffer, and larger buffers
are only ever allocated against servers that actually hand out wide
layouts.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/pnfs.c | 45 ++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 38 insertions(+), 7 deletions(-)

diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 88deefd45202..b596a65c7ef2 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -1166,11 +1166,12 @@ pnfs_alloc_init_layoutget_args(struct inode *ino,
 	   struct nfs_open_context *ctx,
 	   const nfs4_stateid *stateid,
 	   const struct pnfs_layout_range *range,
-	   gfp_t gfp_flags)
+	   size_t min_reply_sz, gfp_t gfp_flags)
 {
 	struct nfs_server *server = pnfs_find_server(ino, ctx);
 	size_t max_reply_sz = server->pnfs_curr_ld->max_layoutget_response;
-	size_t max_pages = max_response_pages(server);
+	size_t session_pages = max_response_pages(server);
+	size_t max_pages = session_pages;
 	struct nfs4_layoutget *lgp;
 
 	dprintk("--> %s\n", __func__);
@@ -1185,6 +1186,17 @@ pnfs_alloc_init_layoutget_args(struct inode *ino,
 			max_pages = npages;
 	}
 
+	/*
+	 * A previous LAYOUTGET for this layout did not fit the reply
+	 * buffer: raise the layout driver's default up to the session's
+	 * maximum response size.
+	 */
+	if (min_reply_sz) {
+		size_t npages = (min_reply_sz + PAGE_SIZE - 1) >> PAGE_SHIFT;
+		if (npages > max_pages)
+			max_pages = min(npages, session_pages);
+	}
+
 	lgp->args.layout.pages = nfs4_alloc_pages(max_pages, gfp_flags);
 	if (!lgp->args.layout.pages) {
 		kfree(lgp);
@@ -2153,6 +2165,7 @@ pnfs_update_layout(struct inode *ino,
 		.inode = ino,
 	};
 	unsigned long giveup = jiffies + (clp->cl_lease_time << 1);
+	size_t reply_sz = 0;
 	bool first;
 
 	if (!pnfs_enabled_sb(NFS_SERVER(ino))) {
@@ -2311,7 +2324,8 @@ pnfs_update_layout(struct inode *ino,
 	if (arg.length != NFS4_MAX_UINT64)
 		arg.length = PAGE_ALIGN(arg.length);
 
-	lgp = pnfs_alloc_init_layoutget_args(ino, ctx, &stateid, &arg, gfp_flags);
+	lgp = pnfs_alloc_init_layoutget_args(ino, ctx, &stateid, &arg, reply_sz,
+					     gfp_flags);
 	if (!lgp) {
 		lseg = ERR_PTR(-ENOMEM);
 		trace_pnfs_update_layout(ino, pos, count, iomode, lo, NULL,
@@ -2338,12 +2352,29 @@ pnfs_update_layout(struct inode *ino,
 			break;
 		case -ENODATA:
 			/* The server returned NFS4ERR_LAYOUTUNAVAILABLE */
-		case -EMSGSIZE:
-			/* The layout exceeded loga_maxcount (NFS4ERR_TOOSMALL) */
 			pnfs_layout_set_fail_bit(
 				lo, pnfs_iomode_to_fail_bit(iomode));
 			lseg = NULL;
 			goto out_put_layout_hdr;
+		case -EMSGSIZE: {
+			/*
+			 * The layout exceeded loga_maxcount (NFS4ERR_TOOSMALL):
+			 * retry once with the reply buffer raised to the
+			 * session's maximum response size before falling back
+			 * to I/O through the MDS.
+			 */
+			size_t max = max_response_pages(server) << PAGE_SHIFT;
+
+			if (reply_sz < max) {
+				reply_sz = max;
+				exception.retry = 1;
+				break;
+			}
+			pnfs_layout_set_fail_bit(
+				lo, pnfs_iomode_to_fail_bit(iomode));
+			lseg = NULL;
+			goto out_put_layout_hdr;
+		}
 		default:
 			if (!nfs_error_is_fatal(PTR_ERR(lseg))) {
 				pnfs_layout_clear_fail_bit(lo, pnfs_iomode_to_fail_bit(iomode));
@@ -2456,7 +2487,7 @@ static void _lgopen_prepare_attached(struct nfs4_opendata *data,
 	lo = _pnfs_grab_empty_layout(ino, ctx);
 	if (!lo)
 		return;
-	lgp = pnfs_alloc_init_layoutget_args(ino, ctx, &current_stateid, &rng,
+	lgp = pnfs_alloc_init_layoutget_args(ino, ctx, &current_stateid, &rng, 0,
 					     nfs_io_gfp_mask());
 	if (!lgp) {
 		pnfs_clear_first_layoutget(lo);
@@ -2482,7 +2513,7 @@ static void _lgopen_prepare_floating(struct nfs4_opendata *data,
 	};
 	struct nfs4_layoutget *lgp;
 
-	lgp = pnfs_alloc_init_layoutget_args(ino, ctx, &current_stateid, &rng,
+	lgp = pnfs_alloc_init_layoutget_args(ino, ctx, &current_stateid, &rng, 0,
 					     nfs_io_gfp_mask());
 	if (!lgp)
 		return;
-- 
2.53.0

[PATCH 4/6] NFSv4.1/pnfs: treat an oversized LAYOUTGET reply as -EMSGSIZE

[Benjamin Coddington]

decode_layoutget() already detects a server that sends a layout body
larger than the reply buffer we provided ("server cheating in
layoutget reply") but maps it to -EINVAL, which pnfs_update_layout()
treats as a transient error: the client falls back to the MDS for
this I/O only and re-sends a doomed LAYOUTGET on every subsequent
pageio attempt.

A server that overruns the reply buffer has ignored loga_maxcount, but
the client can recover the same way it recovers from a conformant
server's NFS4ERR_TOOSMALL: return -EMSGSIZE from exactly this check so
that the layoutget path retries once with a session-sized reply buffer
and otherwise marks the layout mode failed and falls back to the MDS.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/nfs4xdr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index c23c2eee1b5c..6b46ed346a77 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -6154,7 +6154,7 @@ static int decode_layoutget(struct xdr_stream *xdr, struct rpc_rqst *req,
 		dprintk("NFS: server cheating in layoutget reply: "
 				"layout len %u > recvd %u\n",
 				res->layoutp->len, recvd);
-		status = -EINVAL;
+		status = -EMSGSIZE;
 		goto out;
 	}
 
-- 
2.53.0

[PATCH 5/6] NFSv4.1/pnfs: remember when a server needs a larger LAYOUTGET reply buffer

[Benjamin Coddington]

When a LAYOUTGET only succeeds after escalating the reply buffer, every
layout fetched from that server is likely to need the larger buffer:
remember the escalated size on the nfs_server and use it as the floor
for subsequent LAYOUTGET reply buffers, skipping the doomed attempt at
the layout driver's default size.

This also lets the LAYOUTGET attached to OPEN benefit: the lgopen path
is best-effort with no retry of its own, so without the learned size
it would fail with NFS4ERR_TOOSMALL at every open against a server
handing out wide layouts, and layouts would only ever be acquired by
the I/O path's retry.

The field is a hint: reads and writes are racy by design, the value
only ever grows toward the session's maximum response size, and a
stale-low read merely costs one escalation round trip.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/pnfs.c             | 15 ++++++++++++---
 include/linux/nfs_fs_sb.h |  4 ++++
 2 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index b596a65c7ef2..e9ec960ac22d 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -1187,10 +1187,12 @@ pnfs_alloc_init_layoutget_args(struct inode *ino,
 	}
 
 	/*
-	 * A previous LAYOUTGET for this layout did not fit the reply
-	 * buffer: raise the layout driver's default up to the session's
-	 * maximum response size.
+	 * A previous LAYOUTGET on this layout or on this server did not
+	 * fit the reply buffer: raise the layout driver's default up to
+	 * the session's maximum response size.
 	 */
+	if (!min_reply_sz)
+		min_reply_sz = READ_ONCE(server->lg_reply_sz);
 	if (min_reply_sz) {
 		size_t npages = (min_reply_sz + PAGE_SIZE - 1) >> PAGE_SHIFT;
 		if (npages > max_pages)
@@ -2393,6 +2395,13 @@ pnfs_update_layout(struct inode *ino,
 			goto lookup_again;
 		}
 	} else {
+		/*
+		 * A LAYOUTGET that only succeeded with an escalated reply
+		 * buffer: remember the size so that future LAYOUTGETs to
+		 * this server skip the attempt at the driver's default.
+		 */
+		if (reply_sz)
+			WRITE_ONCE(server->lg_reply_sz, reply_sz);
 		pnfs_layout_clear_fail_bit(lo, pnfs_iomode_to_fail_bit(iomode));
 	}
 
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 34d294774f8c..3e6bae7e5221 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -248,6 +248,10 @@ struct nfs_server {
 						   that are supported on this
 						   filesystem */
 	struct pnfs_layoutdriver_type  *pnfs_curr_ld; /* Active layout driver */
+	unsigned int		lg_reply_sz;	/* Learned LAYOUTGET reply
+						   buffer size, when the layout
+						   driver's default has proved
+						   too small */
 	struct rpc_wait_queue	roc_rpcwaitq;
 
 	/* the following fields are protected by nfs_client->cl_lock */
-- 
2.53.0

[PATCH 6/6] NFSv4/flexfiles: allocate the per-mirror stripe array with kvzalloc_objs

[Benjamin Coddington]

Each mirror's stripe array is a single contiguous allocation of
dss_count * sizeof(struct nfs4_ff_layout_ds_stripe) -- roughly 300
bytes per stripe.  With the LAYOUTGET reply buffer no longer capped at
a single page, a wide striped layout can push this well past the
high-order allocation comfort zone (a 2048-stripe mirror is a ~600KB
contiguous allocation) where it can fail under memory fragmentation.

Use kvzalloc_objs() so wide stripe arrays fall back to vmalloc.  Note
the vmalloc fallback is unavailable when the pageio path allocates
under memalloc_noio (swap over pNFS); that case simply behaves as
before.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
---
 fs/nfs/flexfilelayout/flexfilelayout.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index c4aa995026f6..d96d73d04f01 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -285,8 +285,8 @@ static struct nfs4_ff_layout_mirror *ff_layout_alloc_mirror(u32 dss_count,
 
 	mirror->dss_count = dss_count;
 	mirror->dss =
-		kzalloc_objs(struct nfs4_ff_layout_ds_stripe, dss_count,
-			     gfp_flags);
+		kvzalloc_objs(struct nfs4_ff_layout_ds_stripe, dss_count,
+			      gfp_flags);
 	if (mirror->dss == NULL) {
 		kfree(mirror);
 		return NULL;
@@ -315,7 +315,7 @@ static void ff_layout_free_mirror(struct nfs4_ff_layout_mirror *mirror)
 		nfs4_ff_layout_put_deviceid(mirror->dss[dss_id].mirror_ds);
 	}
 
-	kfree(mirror->dss);
+	kvfree(mirror->dss);
 	kfree(mirror);
 }
 
-- 
2.53.0