Linux NFS development
 help / color / mirror / Atom feed
From: Benjamin Coddington <ben.coddington@hammerspace.com>
To: Trond Myklebust <trondmy@kernel.org>,
	Anna Schumaker <anna@kernel.org>,
	Mike Snitzer <snitzer@kernel.org>
Cc: linux-nfs@vger.kernel.org
Subject: [PATCH 0/3] pNFS/flexfiles: mirrored writes that fall back to the MDS
Date: Thu, 19 Nov 2026 20:01:51 -0500	[thread overview]
Message-ID: <cover.1795136323.git.bcodding@hammerspace.com> (raw)

From: Benjamin Coddington <bcodding@hammerspace.com>

ff_layout_pg_init_write() can fall back to the MDS from a pg_init in the
middle of a mirrored descriptor: when pnfs_generic_pg_check_layout() drops
the layout segment, or when a mirror's DS can't be prepared.  pg_init runs
for each mirror whenever that mirror's list is empty, so by then the other
mirrors can already hold requests.  nfs_pageio_reset_write_mds() replaces
pg_ops with nfs_pgio_rw_ops before it completes the descriptor, and
nfs_pgio_rw_ops has no pg_get_mirror, so the completion sees only mirror 0.
The requests on the other mirrors are never sent or cleaned up, the mirror
array they are linked on is freed, and their folios never leave writeback:
the next fsync hangs in folio_wait_writeback().  nfs_do_recoalesce() and
pnfs_write_through_mds() also write to the freed array.

Patch 3 keeps a mirrored descriptor that still holds requests off the MDS:
pg_init returns -EAGAIN, as it does for a mirror count change, and error
cleanup redirties or requeues the requests with the flexfiles ops still in
place.  The next pg_init starts from an empty descriptor and can switch
safely.  ff_layout_write_pagelist() returns PNFS_TRY_AGAIN instead of
PNFS_NOT_ATTEMPTED for a mirrored layout and leaves the fallback to pg_init
of the retry.

Patches 1 and 2 fix the generic code that path goes through.  O_DIRECT
error cleanup releases -EAGAIN requests without writing them, so after
patch 3 an O_DIRECT write would report success with some mirrors left
holding the old data; patch 1 requeues them for the reschedule instead.
Patch 2 makes nfs_do_recoalesce() look its mirror up again after
pg_init, and stops nfs_pageio_complete() from sending the next mirror
after a failed pg_init released the layout segment.

Sashiko found the bug while reviewing the FF_FLAGS_NO_IO_THRU_MDS series:
https://lore.kernel.org/linux-nfs/cover.1795024981.git.bcodding@hammerspace.com/
These patches don't depend on that series, and the two apply in either
order.

Tested against a reffs MDS and DS with a six-way mirrored layout, on a
KASAN kernel, with client fault injection that fails a mirror's pg_init
while other mirrors hold requests, fails the pg_init that a retry runs
during completion, and fails a mirror's DS setup in write_pagelist.
Without the series, fsync hangs in folio_wait_writeback() and KASAN
reports use-after-free in nfs_do_recoalesce() and
pnfs_write_through_mds().  Without patch 1, an O_DIRECT write returns
success while five of the six mirrors keep the old data.  With the
series, buffered and O_DIRECT writes complete, the data is on all six
mirrors, and KASAN reports nothing.

The series is based on v7.4-rc3 with three flexfiles fixes that were
posted separately for stable, and also applies to v7.4-rc3 alone:

  pNFS/flexfiles: read the layout hdr under i_lock in pg_get_mirror_count_write
  https://lore.kernel.org/linux-nfs/23435d398481f51ce2680053cd3a151fe673f91c.1795108831.git.bcodding@hammerspace.com/
  pNFS/flexfiles: don't delay or restart an rpc_task that LOCALIO never ran
  https://lore.kernel.org/linux-nfs/9c46fda0f2cd6d3843b198dac304b4722514b8e3.1795108831.git.bcodding@hammerspace.com/
  pNFS/flexfiles: restart a write that the DS error handler retries
  https://lore.kernel.org/linux-nfs/8ee107c9d0c68b6d0ee7305605ff7edbe0ffa160.1795108832.git.bcodding@hammerspace.com/

Benjamin Coddington (3):
  NFS: resend O_DIRECT write requests that error cleanup gets with
    -EAGAIN
  NFS: don't use a mirror after pg_init can free it
  pNFS/flexfiles: don't switch a busy mirrored write descriptor to the
    MDS

 fs/nfs/direct.c                        | 48 ++++++++++++++++----------
 fs/nfs/flexfilelayout/flexfilelayout.c | 48 ++++++++++++++++++++++++--
 fs/nfs/pagelist.c                      | 15 +++++---
 fs/nfs/read.c                          |  3 +-
 fs/nfs/write.c                         |  5 +--
 include/linux/nfs_xdr.h                |  3 +-
 6 files changed, 93 insertions(+), 29 deletions(-)


base-commit: aebb7274476403040d582c61478df8f4a7bf9cd3
prerequisite-patch-id: 71097d575f7ba74684462ab64b028ff8815dff63
prerequisite-patch-id: 5a73a1fb9d47a31d4ec337ec0264eadbcaa9fd4e
prerequisite-patch-id: 2663ab8e1949d6a11d957cc2784161bf2816cac2
-- 
2.53.0


             reply	other threads:[~2026-11-20  1:01 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20  1:01 Benjamin Coddington [this message]
2026-11-20  1:01 ` [PATCH 1/3] NFS: resend O_DIRECT write requests that error cleanup gets with -EAGAIN Benjamin Coddington
2026-11-20  1:01 ` [PATCH 2/3] NFS: don't use a mirror after pg_init can free it Benjamin Coddington
2026-11-20  1:01 ` [PATCH 3/3] pNFS/flexfiles: don't switch a busy mirrored write descriptor to the MDS Benjamin Coddington

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1795136323.git.bcodding@hammerspace.com \
    --to=ben.coddington@hammerspace.com \
    --cc=anna@kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=snitzer@kernel.org \
    --cc=trondmy@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox