From: Benjamin Coddington <ben.coddington@hammerspace.com>
To: Trond Myklebust <trondmy@kernel.org>,
Anna Schumaker <anna@kernel.org>,
Mike Snitzer <snitzer@kernel.org>
Cc: linux-nfs@vger.kernel.org
Subject: [PATCH v2 0/4] pNFS/flexfiles: mirrored writes that fall back to the MDS
Date: Fri, 20 Nov 2026 11:10:46 -0500 [thread overview]
Message-ID: <cover.1795148523.git.bcodding@hammerspace.com> (raw)
From: Benjamin Coddington <bcodding@hammerspace.com>
ff_layout_pg_init_write() can fall back to the MDS from a pg_init in the
middle of a mirrored descriptor: when pnfs_generic_pg_check_layout() drops
the layout segment, or when a mirror's DS can't be prepared. pg_init runs
for each mirror whenever that mirror's list is empty, so by then the other
mirrors can already hold requests. nfs_pageio_reset_write_mds() replaces
pg_ops with nfs_pgio_rw_ops before it completes the descriptor, and
nfs_pgio_rw_ops has no pg_get_mirror, so the completion sees only mirror 0.
The requests on the other mirrors are never sent or cleaned up, the mirror
array they are linked on is freed, and their folios never leave writeback:
the next fsync hangs in folio_wait_writeback(). nfs_do_recoalesce() and
pnfs_write_through_mds() also write to the freed array.
Patch 4 keeps a mirrored descriptor that still holds requests off the MDS:
pg_init returns -EAGAIN, as it does for a mirror count change, and error
cleanup redirties or requeues the requests with the flexfiles ops still in
place. The next pg_init starts from an empty descriptor and can switch
safely. When a mirror's DS fails in ff_layout_write_pagelist(), it no
longer returns PNFS_NOT_ATTEMPTED for a mirrored layout. Instead it
returns the layout with the failing device and retries with a new one,
as RFC 8435 section 7 asks and as FF_FLAGS_NO_IO_THRU_MDS already does;
section 8.2.3 leaves it to the server to decide, in that layout, whether
the errant mirror stays. The write goes to the MDS only if the server
has no layout to give, or the DS can't be prepared again.
Patches 1 to 3 fix the generic code that path goes through. O_DIRECT
error cleanup releases -EAGAIN requests without writing them, so an
O_DIRECT write would report success with some mirrors left holding the
old data; patch 1 requeues them for the reschedule instead. Patch 2 makes
nfs_do_recoalesce() look its mirror up again after pg_init, and stops
nfs_pageio_complete() from sending the next mirror after a failed pg_init
released the layout segment. Patch 3 sends every mirror when one of them
can take no more, so the mirrors stay in step and the pg_init for the
next request, for example past the end of a layout segment, finds the
descriptor empty.
Sashiko found the bug while reviewing the FF_FLAGS_NO_IO_THRU_MDS series:
https://lore.kernel.org/linux-nfs/cover.1795024981.git.bcodding@hammerspace.com/
These patches don't depend on that series, and the two apply in either
order.
Tested against a reffs MDS and DS with a six-way mirrored layout, on a
KASAN kernel, with client fault injection that fails a mirror's pg_init
while other mirrors hold requests, fails the pg_init that a retry runs
during completion, fails a mirror's DS setup in write_pagelist, and ends
the layout segment partway through a write. Without the series, fsync
hangs in folio_wait_writeback() and KASAN reports use-after-free in
nfs_do_recoalesce() and pnfs_write_through_mds(). Without patch 1, an
O_DIRECT write returns success while five of the six mirrors keep the old
data. With v1 of the series, a write that crosses the end of a layout
segment loops in writeback. With this version, buffered and O_DIRECT
writes complete, a DS failure in write_pagelist is followed by a
LAYOUTRETURN and a new LAYOUTGET instead of an MDS write, the data is on
all six mirrors, and KASAN reports nothing.
The series is based on v7.4-rc3 with three flexfiles fixes that were
posted separately for stable, and also applies to v7.4-rc3 alone:
pNFS/flexfiles: read the layout hdr under i_lock in pg_get_mirror_count_write
https://lore.kernel.org/linux-nfs/23435d398481f51ce2680053cd3a151fe673f91c.1795108831.git.bcodding@hammerspace.com/
pNFS/flexfiles: don't delay or restart an rpc_task that LOCALIO never ran
https://lore.kernel.org/linux-nfs/9c46fda0f2cd6d3843b198dac304b4722514b8e3.1795108831.git.bcodding@hammerspace.com/
pNFS/flexfiles: restart a write that the DS error handler retries
https://lore.kernel.org/linux-nfs/8ee107c9d0c68b6d0ee7305605ff7edbe0ffa160.1795108832.git.bcodding@hammerspace.com/
Changes since v1:
https://lore.kernel.org/linux-nfs/cover.1795136323.git.bcodding@hammerspace.com/
- New patch 3: send every mirror when one of them can take no more.
With v1, a write crossing the end of a layout segment, where the next
range has no layout, looped: the busy check redirtied the previous
request every pass (sashiko).
- Patch 4 (was 3): a DS failure in write_pagelist returns the layout and
retries with a new one instead of marking the deviceid unavailable and
falling back to the MDS; pg_init no longer checks for an unavailable
DS. The mark affected every file on the client, and a marked device
made each new descriptor fail its first request until the mark expired
(sashiko). A fatal error from preparing the DS retries without
returning the layout.
- Patch 2: Fixes tag 862f35c94730, which made the free reachable, instead
of 03d5eb65b538.
- Patch 1: the changelog lists the other -EAGAIN cleanups that now
requeue.
Benjamin Coddington (4):
NFS: resend O_DIRECT write requests that error cleanup gets with
-EAGAIN
NFS: don't use a mirror after pg_init can free it
NFS: send every mirror when one of them can take no more
pNFS/flexfiles: don't switch a busy mirrored write descriptor to the
MDS
fs/nfs/direct.c | 48 +++++++++++++++----------
fs/nfs/flexfilelayout/flexfilelayout.c | 39 +++++++++++++++++---
fs/nfs/pagelist.c | 49 +++++++++++++++++++++++---
fs/nfs/read.c | 3 +-
fs/nfs/write.c | 5 +--
include/linux/nfs_xdr.h | 3 +-
6 files changed, 115 insertions(+), 32 deletions(-)
base-commit: aebb7274476403040d582c61478df8f4a7bf9cd3
prerequisite-patch-id: 71097d575f7ba74684462ab64b028ff8815dff63
prerequisite-patch-id: 5a73a1fb9d47a31d4ec337ec0264eadbcaa9fd4e
prerequisite-patch-id: 2663ab8e1949d6a11d957cc2784161bf2816cac2
--
2.53.0
next reply other threads:[~2026-11-20 16:11 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 16:10 Benjamin Coddington [this message]
2026-11-20 16:10 ` [PATCH v2 1/4] NFS: resend O_DIRECT write requests that error cleanup gets with -EAGAIN Benjamin Coddington
2026-11-20 16:10 ` [PATCH v2 2/4] NFS: don't use a mirror after pg_init can free it Benjamin Coddington
2026-11-20 16:10 ` [PATCH v2 3/4] NFS: send every mirror when one of them can take no more Benjamin Coddington
2026-11-20 16:10 ` [PATCH v2 4/4] pNFS/flexfiles: don't switch a busy mirrored write descriptor to the MDS Benjamin Coddington
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1795148523.git.bcodding@hammerspace.com \
--to=ben.coddington@hammerspace.com \
--cc=anna@kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=snitzer@kernel.org \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.