[PATCH v2 0/6] pNFS: don't reply NOMATCHING to a layout recall with I/O in flight

[Benjamin Coddington <ben.coddington@hammerspace.com>]

From: Benjamin Coddington <bcodding@hammerspace.com>

A client can reply NFS4ERR_NOMATCHING_LAYOUT to a CB_LAYOUTRECALL while
flexfiles DS I/O on that layout is still outstanding.  The server then
treats the layout as returned, and that I/O can reach the storage device
after the recall has completed.  We found two ways to get there:

 - A bulk recall (FSID or ALL) that finds segments pinned by I/O
   invalidates the layout stateid and replies NFS4ERR_DELAY.  The resent
   recall skips layouts with an invalid stateid, finds nothing, and gets
   NFS4ERR_NOMATCHING_LAYOUT.

 - If another path has already invalidated the stateid without a
   LAYOUTRETURN, for example pnfs_destroy_layout() after an ESTALE, then
   either kind of recall gets NFS4ERR_NOMATCHING_LAYOUT right away.

Patch 1 adds a tracepoint for bulk recall replies.  Patch 2 clears
NFS_LAYOUT_BULK_RECALL once an invalid layout's last segment is freed;
flexfiles marks every segment for return when its I/O completes, so the
flag otherwise stays set until the layout header is freed.  Patch 3 fixes
a leak of segment references when ->prepare_layoutcommit() fails, which
would otherwise leave the next two patches answering NFS4ERR_DELAY
forever.  Patch 4 has the bulk walker collect invalid layouts that still
have segments, so the resent recall gets NFS4ERR_DELAY until they drain.
Patch 5 does the same for a file recall, and sets NFS_LAYOUT_BULK_RECALL
so that I/O needing a new layout doesn't wait on a LAYOUTGET the server
won't grant while the recall is outstanding.  Patch 6 lets flexfiles
writes go through the MDS while that recall drains, even when the layout
carries FF_FLAGS_NO_IO_THRU_MDS, which RFC 8435 makes a hint; without it
those writers retry without any delay until the drain finishes.

Patch 4 was posted earlier on its own; its changelog is below the fold.

Tested by sending FSID, ALL and FILE recalls on demand, resent every
second after NFS4ERR_DELAY, while the client's DS traffic was held (DS
stopped, or dropped with iptables) during a 64MB O_DIRECT write:

 - Unpatched, an FSID recall gets NFS4ERR_DELAY, then
   NFS4ERR_NOMATCHING_LAYOUT on the first resend with the write still
   held.

 - Patched, FSID and ALL recalls get NFS4ERR_DELAY until the DS I/O
   completes or times out, then NFS4ERR_NOMATCHING_LAYOUT.  With no I/O
   outstanding the first reply is NFS4ERR_NOMATCHING_LAYOUT.

 - Patched, with the layout invalidated by NFS4ERR_STALE on an
   OPEN_DOWNGRADE, a file recall gets NFS4ERR_DELAY until the I/O
   drains.  With clora_changed set the I/O is cancelled and the resend
   gets NFS4ERR_NOMATCHING_LAYOUT.

 - With FF_FLAGS_NO_IO_THRU_MDS, a second O_DIRECT writer to a file
   whose recall was draining wrote its 8MB through the MDS in 23ms with
   patch 6.  Without it, the writer retried about 330,000 times in ten
   seconds until the hold was released.

Patch 2 was not exercised directly: in these runs the layout header was
freed once the I/O drained.

Changes in v2:
 - Patch 3: only reschedule the LAYOUTCOMMIT if the layout stateid is
   still valid.  Putting the segment references could free the layout
   header and leave NFS_INO_LAYOUTCOMMIT set, so the next
   pnfs_layoutcommit_inode() dereferenced a NULL nfsi->layout.

v1: https://lore.kernel.org/linux-nfs/cover.1793298149.git.bcodding@hammerspace.com/

Benjamin Coddington (6):
  NFSv4: add a tracepoint for bulk CB_LAYOUTRECALL replies
  pNFS: clear NFS_LAYOUT_BULK_RECALL when an invalid layout's last
    segment is freed
  pNFS: don't leak layout segments when prepare_layoutcommit fails
  pNFS: don't send _NOMATCHING_LAYOUT if still doing IO on a layout
  pNFS: answer a file recall with NFS4ERR_DELAY while I/O drains
  pNFS/flexfiles: write through the MDS while a recalled layout drains

 fs/nfs/callback_proc.c                 |  19 ++++-
 fs/nfs/flexfilelayout/flexfilelayout.c |   5 +-
 fs/nfs/nfs4trace.h                     |  49 ++++++++++++
 fs/nfs/pnfs.c                          | 105 +++++++++++++++++++++----
 fs/nfs/pnfs.h                          |   4 +
 5 files changed, 163 insertions(+), 19 deletions(-)


base-commit: ef30b3efb7e8bae94d790a6503a595e1051323e7
-- 
2.53.0