[PATCH v2 0/7] nfsd: layout recall and lease break fixes

[PATCH v2 0/7] nfsd: layout recall and lease break fixes

[Daejun Park via B4 Relay]

nfsd polls a client that has accepted a CB_LAYOUTRECALL by sending the
callback again every 10 ms, with the stateid of the first callback. A
Linux client from v6.8 on answers the repeat with NFS4ERR_OLD_STATEID,
and nfsd takes that as a failed recall. So a client that needs more
than 10 ms to return a layout has the layout revoked and the fence path
run. With the block layout that client loses data it has written and
synced, with the SCSI layout it is cut off from the device. Patch 5
lets nfsd keep polling instead.

A recall that takes longer makes the operation that started it wait
longer. When that operation is a WRITE, a SETATTR that changes the
size, an ALLOCATE, a DEALLOCATE, a CLONE or a COPY, the nfsd thread
serving it waits in XFS until the layouts are returned or the lease
break times out. As Chuck pointed out on v1, a few such requests can
take every nfsd thread, and the server then cannot serve the
LAYOUTRETURN that would release them. Patch 2 recalls the layouts
before the operation without waiting, gives the layouts of other
clients 30 ms to come back when it starts their recall, and otherwise
answers NFS4ERR_DELAY.

The cost of patch 2: a client that writes to a file through its layout
and, for writes that are not aligned, through the server recalls its
own layout with each such WRITE and has its LAYOUTGETs fail with
NFS4ERR_RECALLCONFLICT meanwhile. In test 2 it wrote 48, 46 and 4 1 MiB
blocks through its layout in 30 seconds instead of 534, 537 and 505;
before, 6 of 7, 4 of 4 and 17 of 20 recalls ended in the failure path
and 7, 4 and 20 blocks were lost. The series on top of this one, with
the NFS client fix it needs, brings that to 193, 192 and 184 blocks.

Patches 3 and 4 are the other two that patch 5 needs. Patch 3: the
limit of two lease periods on the poll has never triggered, and without
it the poll would have no end. Patch 4: for the block layout a lease
break gives up after fs.lease-break-time (45 s) without running
/sbin/nfsd-recall-failed. Until now the recall had failed long before
that; with patch 5 it is polled for two lease periods (180 s). Patch 4
fences the client at fs.lease-break-time, as the SCSI layout does since
v7.1; by default that is shorter than the one lease period RFC 8881
section 12.5.5 has the server wait after a recall.

Patch 1 fixes a lease break that hits a LAYOUTGET in progress and then
waits for a layout that is never recalled. It comes first because
patch 2 makes that common: the break of patch 2 runs outside the XFS
iolock, so it can land between the allocation of a layout stateid and
the insertion of its first layout.

Patches 6 and 7 deal with two answers that the polls handle badly. A
client that a LAYOUTRETURN has moved past the seqid of the recall
answers every poll NFS4ERR_OLD_STATEID and is never asked for the
layouts it still holds; patch 6 polls such a client, one that has not
answered NFS4_OK, with a new seqid. A client that answers
NFS4ERR_BAD_STATEID after returning its last layout is fenced; patch 7
looks for layouts first.

Patches 1 to 5 have to go together, also to stable kernels. Patch 6
builds on patches 1 and 5, and patch 7 stands on its own. If patch 2
should not go to stable, patches 3 to 5 should not either; patches 1
and 7 can go alone. Kernels before v7.1 do not have the lease break
timeout that patch 4 fixes; there an operation on the server that does
not come from nfsd waits for the recall for up to two lease periods, as
it already does, without a limit, for a client that keeps answering
NFS4ERR_DELAY. Patch 2 uses nfsd_thread_local_info and needs adapting
where that is missing.

Tests. QEMU guests on one host: an nvmet-tcp target, the server (nfsd,
XFS on the nvmet namespace, pnfs export) and up to three clients, all
on the same kernel with KASAN and lockdep, NFS v4.2, block layout
unless noted, O_DIRECT I/O. "v2" is this series on nfsd-testing
56589cdb5881, "v1" is v1 on it, and "before" is that commit unless
noted. No KASAN or lockdep report on the server in any run, nor on the
clients and the target in tests 2, 3 and 5, where they were checked.

1. nfsd threads (patch 2), 4 nfsd threads. A client holds a layout of
   a file and its path to the device is cut for 70 s. Another client,
   without pNFS, sends 4 KiB WRITEs to the file through the server, all
   at once, and reads 4 KiB of another file every second:

                                            before     v1        v2
   8 WRITEs, NFSv4.0: the WRITEs take       46.7 s     -         45.2 s
     longest read of the other file         46.7 s     -         0.031 s
   64 WRITEs, NFSv4.0: the WRITEs take      -          49.2 s    45.4 s
     longest read of the other file         -          49.1 s    0.22 s
   8 WRITEs, NFSv3: the WRITEs take         47.0 s     -         51.2 s
     longest read of the other file         46.9 s     -         0.021 s

   The WRITEs wait for the lease break, which ends with
   /sbin/nfsd-recall-failed (patch 4). NFSv3 WRITEs take longer with
   v2 because the client waits 5 s after NFS3ERR_JUKEBOX. In the run
   with 64 WRITEs before this series, the recall failed after 10 ms, as
   described above, and the WRITEs were done in 0.5 s.

2. What patch 2 costs. Two clients write 1 MiB blocks to a new file for
   30 s, one through its layout and one (NFSv4.0) through the server,
   three runs. "+ follow-up" adds the nfsd series that goes on top of
   this one and an NFS client fix that it needs.

                      through the     of those, read      through the
                      layout          back as zeroes      server
   before             416/200/186     28/131/111          120/325/323
   v2                 174/176/170     0/0/0               184/203/220
   v2 + follow-up     170/175/176     0/0/0               284/223/207

   The 30 ms wait is what keeps the writer through the server going:
   two earlier revisions of v2 + follow-up that differ only in it gave
   79/79/80 blocks through the server without the wait and 238/230/137
   with it.

   One client doing both, 1 MiB blocks through its layout and 512 byte
   writes through the server:

                      through the     of those, read      writes through
                      layout          back as zeroes      the server
   before             534/537/505     7/4/20              32/21/87
   v2                 48/46/4         0/0/0               3600/3262/4917
   v2 + follow-up     193/192/184     0/0/0               1082/1488/1669

   With v2 alone, each WRITE of that client through the server recalls
   its layout. About half of its LAYOUTGETs then meet that recall and
   get NFS4ERR_RECALLCONFLICT, as before; the other half are in
   progress when the WRITE breaks the lease, and patch 1 fails them.
   The Linux client waits up to 15 s between LAYOUTGETs that fail with
   NFS4ERR_RECALLCONFLICT. The follow-up hands out the layout of such a
   LAYOUTGET and recalls it at once. That needs the referring call the
   follow-up adds to CB_LAYOUTRECALL: tried in this series, the recall
   overtook the LAYOUTGET reply, and in all three runs 92 to 104 of
   about 260 blocks read back as zeroes.

   On 214e388464bf, before and v2 gave the same picture, except that in
   one of three runs of the one-client case with v2 one block read back
   as zeroes, through the recall race described under "Not changed".

   The series applies to nfsd-testing c32fef6c476d unchanged, and there
   before and v2 gave the same picture again, also for NFSv3, whose
   replies now come from xdrgen. In test 1 the longest read of the
   other file was 47.8 s before and 0.036 s with v2 over NFSv4.0, and
   48.3 s and 0.019 s over NFSv3, where the WRITEs took 48.4 s and
   51.3 s. In this test, with two clients, 48, 94 and 90 of 180, 185
   and 181 blocks through the layout read back as zeroes before, and
   none of 161, 177 and 172 with v2; with one client, 8, 0 and 3
   before and none with v2.

   Writes to a file without leases: fio, NFSv4.0, no KASAN or lockdep,
   10 runs of 30 s, medians without and with an earlier revision of v2
   (the same path for such a file): 1 MiB sequential 65.8 and 67.5
   MiB/s, 4 KiB random 262 and 263 IOPS.

3. Recall polling (patches 3 to 6). With v2 no recall ended in the
   failure path (0 of 9, 9, 63 and 277 in the arms of patch 5) and no
   block written to a new file was lost (0 of 545, 511, 477, 577 and
   36). Patch 5 has the runs before.
   On c32fef6c476d, four of those arms lost 4, 2, 18 and 9 blocks
   before, with 46 recall failures logged, and none with v2, with no
   recall failure.

4. One client writing through a block layout whose path to the device
   is cut while it can still talk to the server, and a 4 KiB write to
   the file on the server ("before" of the 15 s case is 32eb1a60b456).
   The client's write that is in flight stays stuck until the path is
   back.

                                                   before      v2
   path cut for 15 s: the server write takes       0.05 s      15.6 s
     the recall                                    fails       completes
     /sbin/nfsd-recall-failed is run               at once     no
   path cut for 70 s: the server write takes       0.05 s      47.2 s
     /sbin/nfsd-recall-failed is run               at once     at 47.1 s

   The same 70 s case with a script that expires the client it fences
   through /proc/fs/nfsd/clients/<clid>/ctl: with v1's fence worker,
   which runs the script itself, the script never finished and the
   server write went on only after the path was back, after 93 s. With
   v2 the script was done 36 ms after it started, and the server write
   took 46.1 s. With a lease time of 15 s, the recall reaches its limit
   (patch 3) after 30 s, before the lease break times out, and the
   recall failure path runs the script: there it hangs with v2 as well,
   the script in nfsd4_shutdown_callback() and rpciod in
   call_usermodehelper_exec().

5. A lease break during a LAYOUTGET (patch 1). With v2 the longest of
   424 writes on the server took 55 ms, and 108 ms with a delay of
   100 ms added to LAYOUTGET for the test. Patch 1 has the runs without
   it.

6. SCSI layout. The target runs a kernel with "nvmet: fix preempt of
   another registrant by the reservation holder" [1], so that the
   preempt removes the registration. With v2, the path of a client cut
   for 15 s: the server write takes 15.2 s and the client is not
   fenced; cut for 70 s: 47.6 s, and the fence worker fences the client
   (registrants 2 -> 1).

pynfs BLOCK1, BLOCK2, BLOCK4 and BLOCK5 pass before and with v2 (BLOCK3
fails in both with a TypeError in the test). The commit messages of
patches 6 and 7 have pynfs clients that answer as described there.

Not changed by these patches:

- A layout granted between the check of patch 2 and the break in XFS
  still makes the nfsd thread wait. No new layout lease is granted
  while a break is pending, so that needs a LAYOUTGET for a file that
  had no layout at the check. An asynchronous COPY runs in its own
  kthread and still waits there.

- The recall failure path still runs /sbin/nfsd-recall-failed from
  nfsd4_cb_layout_done(), in rpciod, while the callback is in flight.
  A script that expires the client it fences hangs there, because the
  teardown waits for that callback in nfsd4_shutdown_callback(); test 4
  shows it. Neither path looks at the exit status of the script, and
  the recall failure path does not look at the result of
  ->fence_client(): a fence that fails is taken as done there, and the
  layouts it drops end the retries of a fence worker that a lease break
  has started (patch 3 says so).

- nfsd still polls a client that has answered NFS4_OK with the seqid it
  has used, where RFC 8881 section 12.5.3 has the server increment the
  seqid in each request. Patch 5 only makes nfsd bear the answer
  (patch 6 gives a new seqid only to a client that has not answered
  NFS4_OK), and while it polls, nfsd sends a callback every 10 ms for
  up to two lease periods.

- CB_LAYOUTRECALL carries no referring call, as CB_RECALL does since
  "NFSD: Send referring calls with CB_RECALL". When a recall overtakes
  the reply to the LAYOUTGET that created the layout stateid, the Linux
  client answers NFS4ERR_NOMATCHING_LAYOUT, nfsd drops the layout and
  the client uses it afterwards. nfsd also trusts
  NFS4ERR_NOMATCHING_LAYOUT while it still has layouts on the layout
  stateid. The series on top of this one adds the referring call and
  resends the recall once before it trusts that answer, and with it
  recalls a layout handed out by a LAYOUTGET that a lease break met,
  instead of failing that LAYOUTGET (see test 2).

- Which client makes progress when several clients use one file: the
  client that holds a layout now keeps it while it does I/O through
  it, where before its layout was revoked after 10 ms. Nothing here
  limits how long one client can keep others out.

[1] https://lore.kernel.org/r/20261002065931epcms2p1cb2c16f98cf3b05f8c859ad92b0ad465@epcms2p1

---
Changes in v2:
- New patch 2, which recalls the layouts without waiting before an
  operation that modifies the file and answers NFS4ERR_DELAY, as Chuck
  asked:
  https://lore.kernel.org/r/63ef8d7c-086f-4cc8-84bb-72c1b23bf596@slotpi15m67
- v1 4/6 is now patch 1, ahead of patch 2, which makes its case common.
  v1 1/6 to 3/6 are patches 3 to 5, v1 5/6 and 6/6 are patches 6 and 7.
- Sashiko on v1:
  - 2/6, the fence script waiting for itself when it expires the client
    through /proc/fs/nfsd/clients/<clid>/ctl: patch 4 runs the script
    in a work item on a new workqueue, nfsd4_fence, and the fence
    worker checks every 100 ms whether it is done. Teardown waits only
    for the fence worker, which no longer waits for the script.
  - 2/6, sleeping fence workers filling system_dfl_wq: the same change;
    a script that runs long occupies nfsd4_fence.
  - 3/6, ktime_get() stops during suspend: patch 3 takes the time with
    ktime_get_boottime().
  - 5/6, si_generation written under sc_lock in ->done and read under
    ls_mutex only: patch 6 reads it under sc_lock in
    nfsd4_preprocess_layout_stateid() as well.
  - 1/6, the result of ->fence_client() not looked at on the recall
    failure path, which also ends the retries of the fence worker: not
    changed, said in patch 3. Patch 3 makes the path reachable after two
    lease periods for a client that keeps answering; before, a client
    from v6.8 on reached it after 10 ms.
  - 2/6, the exit status of the script not looked at: not changed.
    Retrying a failed script would block the file for good on a server
    that has no /sbin/nfsd-recall-failed. Said in patch 4.
  - 6/6, NFS4ERR_NOMATCHING_LAYOUT trusted while layouts are left:
    preexisting, not changed here (see the list above).
- Rebased onto nfsd-testing c32fef6c476d without changes, and tests 1
  to 3 run again there.
- Test numbers in the cover from v2; the commit messages of patches 1,
  4 and 5 keep their runs on nfsd-testing 32eb1a60b456 for the side
  without the patch. Commit messages with one space after a period.
- Link to v1: https://lore.kernel.org/r/20261006-nfsd-layout-recall-v1-0-b59b31caf342@samsung.com

---
Daejun Park (7):
      nfsd: fail a LAYOUTGET in progress when the layout lease is broken
      nfsd: answer NFS4ERR_DELAY instead of waiting for a layout recall
      nfsd: make the time limit of a layout recall work
      nfsd: run the fence script when a block layout lease break times out
      nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID
      nfsd: poll a layout recall with a new seqid when the client is past it
      nfsd: do not fence a client for a layout recall with no layouts left

 fs/nfsd/nfs4layouts.c | 301 ++++++++++++++++++++++++++++++++++++++++++++++----
 fs/nfsd/nfs4proc.c    |   8 ++
 fs/nfsd/pnfs.h        |  15 +++
 fs/nfsd/state.h       |   8 ++
 fs/nfsd/trace.h       |   2 +
 fs/nfsd/vfs.c         |  81 +++++++++++++-
 fs/nfsd/vfs.h         |   1 +
 7 files changed, 391 insertions(+), 25 deletions(-)
---
base-commit: c32fef6c476daa5830f600614657baee59c83339
change-id: 20261006-nfsd-layout-recall-ba868d33371e

Best regards,
-- 
Daejun Park <daejun7.park@samsung.com>

[PATCH v2 1/7] nfsd: fail a LAYOUTGET in progress when the layout lease is broken

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

A layout stateid takes its FL_LAYOUT lease when it is allocated, before
the first layout is added to it. When the lease is broken in between,
nfsd4_recall_file_layout() finds no layouts and does nothing. The
LAYOUTGET then hands out a layout that is not recalled: the lease is
already marked as breaking, so ->lm_break() is not called for it again.
The breaker waits until the client gives the layout up by itself or the
lease break time is over; with the SCSI layout the client is then
fenced although it was never sent a recall. While the breaker waits,
kernel_setlease() fails for new layout stateids and the LAYOUTGETs of
other clients get NFS4ERR_DELAY.

Commit c5c707f96fc9 ("nfsd: implement pNFS layout recalls") marked such
a layout stateid as recalled, and since nothing kept the LAYOUTGET from
adding its layout afterwards, that layout was never recalled. Commit
fb610c4dbc99 ("nfsd: fix race to check ls_layouts") therefore made the
recall skip a layout stateid without layouts, which left the case
above.

Let the lease break mark such a layout stateid as recalled again, and
make nfsd4_insert_layout() refuse to add layouts to a recalled layout
stateid, which is what was missing the first time. The LAYOUTGET fails
with NFS4ERR_RECALLCONFLICT, the layout stateid is freed and the lease
goes away. No callback is sent for it; the new
nfsd_layout_recall_empty tracepoint shows where fi_lo_recalls was
raised without one. NFS4ERR_RECALLCONFLICT is what nfsd already answers
to a LAYOUTGET for a file with a recall in progress, also when the
recall went to another client (RFC 8881 section 15.1.14.3: "a
conflicting recall operation that is currently in progress for that
object"). The check in nfsd4_insert_layout() also covers a
layout stateid with layouts that is recalled between the checks in
nfsd4_layoutget() and nfsd4_insert_layout().

Only a lease break does this. A LAYOUTGET that finds a layout stateid
without layouts of another client leaves it alone, as before.

The window is the time a LAYOUTGET needs from allocating the layout
stateid to inserting the layout.

Two clients writing to one file while the server writes 4 KiB to it
every 50 ms, for 30 seconds, block layout (the runs without this patch
on nfsd-testing 32eb1a60b456, the runs with it with this whole series).
Without this patch one write on the server took 9.7 seconds in one run
and 13.9 seconds in another, in both until the clients were done; in a
third run it did not happen. With this patch the longest write on the
server took 55 ms.

To see the new path at work, a delay of 100 ms was added to LAYOUTGET
between the two steps for one pair of runs. Without the patch the
second write on the server took 30 seconds, the whole run. With it the
longest write took 108 ms and nfsd_layout_recall_empty fired 16 times.
(With that delay every LAYOUTGET meets a write on the server, so the
clients got no layout in that run. It shows the path and nothing else.)

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 49 +++++++++++++++++++++++++++++++++++++++++++------
 fs/nfsd/trace.h       |  1 +
 2 files changed, 44 insertions(+), 6 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 898776c9d7..72b53eebd1 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -336,14 +336,15 @@ nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp,
 }
 
 static void
-nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)
+nfsd4_recall_file_layout_locked(struct nfs4_layout_stateid *ls)
 {
-	spin_lock(&ls->ls_lock);
+	lockdep_assert_held(&ls->ls_lock);
+
 	if (ls->ls_recalled)
-		goto out_unlock;
+		return;
 
 	if (list_empty(&ls->ls_layouts))
-		goto out_unlock;
+		return;
 
 	ls->ls_recalled = true;
 	atomic_inc(&ls->ls_stid.sc_file->fi_lo_recalls);
@@ -353,7 +354,34 @@ nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)
 		refcount_inc(&ls->ls_stid.sc_count);
 		nfsd4_run_cb(&ls->ls_recall);
 	}
-out_unlock:
+}
+
+static void
+nfsd4_recall_file_layout(struct nfs4_layout_stateid *ls)
+{
+	spin_lock(&ls->ls_lock);
+	nfsd4_recall_file_layout_locked(ls);
+	spin_unlock(&ls->ls_lock);
+}
+
+/*
+ * A lease break also has to stop a LAYOUTGET that is in progress. Its
+ * layout stateid has the lease and no layouts yet, so there is nothing to
+ * recall, and the breaker goes on once that layout stateid is gone. Mark it
+ * as recalled without sending a callback: nfsd4_insert_layout() then fails
+ * the LAYOUTGET instead of handing out a layout after the break.
+ */
+static void
+nfsd4_break_file_layout(struct nfs4_layout_stateid *ls)
+{
+	spin_lock(&ls->ls_lock);
+	if (!ls->ls_recalled && list_empty(&ls->ls_layouts)) {
+		ls->ls_recalled = true;
+		atomic_inc(&ls->ls_stid.sc_file->fi_lo_recalls);
+		trace_nfsd_layout_recall_empty(&ls->ls_stid.sc_stateid);
+	} else {
+		nfsd4_recall_file_layout_locked(ls);
+	}
 	spin_unlock(&ls->ls_lock);
 }
 
@@ -432,6 +460,10 @@ nfsd4_insert_layout(struct nfsd4_layoutget *lgp, struct nfs4_layout_stateid *ls)
 	if (nfserr)
 		goto out;
 	spin_lock(&ls->ls_lock);
+	if (ls->ls_recalled) {
+		nfserr = nfserr_recallconflict;
+		goto out_unlock_ls;
+	}
 	list_for_each_entry(lp, &ls->ls_layouts, lo_perstate) {
 		if (layouts_try_merge(&lp->lo_seg, seg))
 			goto done;
@@ -450,6 +482,10 @@ nfsd4_insert_layout(struct nfsd4_layoutget *lgp, struct nfs4_layout_stateid *ls)
 	if (nfserr)
 		goto out;
 	spin_lock(&ls->ls_lock);
+	if (ls->ls_recalled) {
+		nfserr = nfserr_recallconflict;
+		goto out_unlock_ls;
+	}
 	list_for_each_entry(lp, &ls->ls_layouts, lo_perstate) {
 		if (layouts_try_merge(&lp->lo_seg, seg))
 			goto done;
@@ -460,6 +496,7 @@ nfsd4_insert_layout(struct nfsd4_layoutget *lgp, struct nfs4_layout_stateid *ls)
 	new = NULL;
 done:
 	nfs4_inc_and_copy_stateid(&lgp->lg_sid, &ls->ls_stid);
+out_unlock_ls:
 	spin_unlock(&ls->ls_lock);
 out:
 	spin_unlock(&fp->fi_lock);
@@ -779,7 +816,7 @@ nfsd4_layout_lm_break(struct file_lease *fl)
 	 * Enforce break lease timeout to prevent NFSD
 	 * thread from hanging in __break_lease.
 	 */
-	nfsd4_recall_file_layout(fl->c.flc_owner);
+	nfsd4_break_file_layout(fl->c.flc_owner);
 	return false;
 }
 
diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
index e5d19c6f48..abfdbc637c 100644
--- a/fs/nfsd/trace.h
+++ b/fs/nfsd/trace.h
@@ -679,6 +679,7 @@ DEFINE_STATEID_EVENT(layout_get_lookup_fail);
 DEFINE_STATEID_EVENT(layout_commit_lookup_fail);
 DEFINE_STATEID_EVENT(layout_return_lookup_fail);
 DEFINE_STATEID_EVENT(layout_recall);
+DEFINE_STATEID_EVENT(layout_recall_empty);
 DEFINE_STATEID_EVENT(layout_recall_done);
 DEFINE_STATEID_EVENT(layout_recall_fail);
 DEFINE_STATEID_EVENT(layout_recall_release);

-- 
2.43.0

[PATCH v2 2/7] nfsd: answer NFS4ERR_DELAY instead of waiting for a layout recall

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

A WRITE, a SETATTR that changes the size (also an OPEN that truncates
and an NFSv2 or NFSv3 CREATE of an existing file with a size), an
ALLOCATE, a DEALLOCATE, a CLONE and a COPY make XFS recall the layouts
of the file and wait in __break_lease() until they are returned or the
lease break times out: xfs_break_leased_layouts(), and
xfs_iolock_two_inodes_and_break_layout() for both files of a clone. The
nfsd thread serving the request sleeps there too. The client decides
when it returns its layout, so a few such requests can occupy every nfsd
thread, and the server then cannot process the LAYOUTRETURN that would
release them.

Before calling into the filesystem, recall the layouts without waiting,
as nfsd_open_break_lease() does for delegations, and return
-EWOULDBLOCK while a layout is held, which nfserrno() maps to
NFS4ERR_DELAY (NFS3ERR_JUKEBOX for NFSv3), so that the client retries.
When this starts the recall of a layout of another client, first give
the layouts NFSD_LAYOUTRETURN_TIMEOUT to come back, the 30 ms that
nfsd_wait_for_delegreturn() allows a delegation: a client that is not
doing I/O through its layout returns it within a few milliseconds, and
the retry would cost the client of the request at least 100 ms. Do not
wait for a recall that is already under way, nor when the client of
the request holds a layout of the file itself: it returns that layout
only after its own I/O through it has completed. nfsd4_close_layout()
wakes the waiter when a layout lease is removed. For a COPY or a CLONE,
both recalls are started before the wait. For a SETATTR, the recall
comes before inode_lock, which the LAYOUTCOMMIT that a client sends
before its LAYOUTRETURN takes. A file without leases costs one lockless
check of its lease list; a file with other leases, such as
delegations, a walk of that list under flc_lock.

Answering NFS4ERR_DELAY here was discussed in 2023 (second Link).
Passing a non-blocking flag down to the break in the filesystem would
need VFS and XFS changes and a way through notify_change(),
vfs_fallocate() and the remap path; breaking before the call, as
nfsd_open_break_lease() does for delegations, needs neither.

An nfsd thread can still block in the filesystem when a LAYOUTGET
completes between this check and the recall in the filesystem. No
layout lease can be added while a recall is pending, so this takes a
file that had no layout when nfsd checked. A recall that meets a
LAYOUTGET in progress fails that LAYOUTGET (previous patch), so the
layout it would have handed out is not left unrecalled. An asynchronous
COPY runs in its own kthread and is not changed; OFFLOAD_CANCEL and
client teardown wait for that kthread.

The cost is on a request that conflicts with a layout. When the layout
does not come back within 30 ms, or the client of the request holds it,
the request completes only when the client sends it again. The Linux
client does that after at least 100 ms for NFSv4 (NFS4_POLL_RETRY_MIN)
and after 5 seconds for NFSv3 (NFS_JUKEBOX_RETRY_TIME), where it used to
wait only until the layout was returned. An NFSv2 WRITE has its reply
dropped, as for any NFSERR_JUKEBOX, and an NFSv2 SETATTR or CREATE is
answered as for a delegation conflict. The recall now also comes before
the checks that would fail the request: a COPY recalls the layouts of
its source as well, because the filesystem may do it as a clone, also
where it cannot, and a CLONE on a filesystem without reflink can get
NFS4ERR_DELAY before NFS4ERR_NOTSUPP. An NFSv2 or NFSv3 CREATE of an
existing file with a size does the SETATTR with the parent directory
locked and holds it during the wait.

With 4 nfsd threads, a client that holds a layout of a file and has
lost its path to the device, and another client (NFSv4.0) that sends 8
4 KiB WRITEs to that file through the metadata server while it reads
4 KiB of another file every second: the WRITEs take about 45 seconds
with and without this patch, as they wait for the lease break, and
without it the reads of the other file wait as long. With it the
longest of 84 reads takes 31 ms, and 0.22 seconds with 64 WRITEs. With
the second client on NFSv3, the longest read takes 21 ms instead of 47
seconds, and the WRITEs take 51 seconds instead of 47, as the client
waits 5 seconds after NFS3ERR_JUKEBOX.

A client that writes to a file both through its layout and through the
server recalls its own layout with each such WRITE. About half of its
LAYOUTGETs then meet that recall and get NFS4ERR_RECALLCONFLICT, as
before; the others are in progress when the WRITE breaks the lease, and
the previous patch fails them. The Linux client waits up to 15 seconds
between LAYOUTGETs that fail with NFS4ERR_RECALLCONFLICT. Writing 1 MiB
blocks through its layout and 512 bytes through the server for 30
seconds, such a client wrote 48, 46 and 4 blocks through the layout in
three runs, against 534, 537 and 505 without this series. There most
recalls of its layout ended in the failure path, and 7, 4 and 20 of the
blocks read back as zeroes after a successful fsync(); none did with
this series.

Writes to a file without leases are not changed: fio with O_DIRECT,
NFSv4.0, 10 runs of 30 seconds, without KASAN and lockdep, gave median
65.8 and 67.5 MiB/s for 1 MiB sequential writes without and with an
earlier revision of this series (the same path for such a file), and 262
and 263 IOPS for 4 KiB random writes.

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")
Suggested-by: Chuck Lever <cel@kernel.org>
Link: https://lore.kernel.org/r/63ef8d7c-086f-4cc8-84bb-72c1b23bf596@slotpi15m67
Link: https://lore.kernel.org/r/ZVdj2U+kJ7DlI22g@infradead.org
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 58 +++++++++++++++++++++++++++++++++++-
 fs/nfsd/nfs4proc.c    |  8 +++++
 fs/nfsd/pnfs.h        | 15 ++++++++++
 fs/nfsd/state.h       |  1 +
 fs/nfsd/vfs.c         | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 fs/nfsd/vfs.h         |  1 +
 6 files changed, 162 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 72b53eebd1..035ab79faa 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -153,13 +153,69 @@ void nfsd4_close_layout(struct nfs4_layout_stateid *ls)
 	spin_unlock(&ls->ls_stid.sc_file->fi_lock);
 
 	if (fl) {
-		if (!nfsd4_layout_ops[ls->ls_layout_type]->disable_recalls)
+		if (!nfsd4_layout_ops[ls->ls_layout_type]->disable_recalls) {
 			kernel_setlease(fl->nf_file, F_UNLCK, NULL,
 					(void **)&ls);
+			/*
+			 * For nfsd_break_layouts(). No barrier needed: its
+			 * waiter rechecks under flc_lock, which removing the
+			 * lease took.
+			 */
+			wake_up_var(file_inode(fl->nf_file));
+		}
 		nfsd_file_put(fl);
 	}
 }
 
+/**
+ * nfsd4_layout_holders - Report which clients hold layout leases on a file
+ * @inode: file whose layout leases are checked
+ *
+ * The client of the request being processed is the one of its compound.
+ * Without one, as for NFSv3, every layout counts as another client's.
+ *
+ * Context: Takes the flc_lock of @inode.
+ *
+ * Return: a mask of NFSD_LAYOUTS_OWN if that client holds a layout lease
+ * on @inode, NFSD_LAYOUTS_OTHER if another client holds one, and
+ * NFSD_LAYOUTS_IDLE if a lease of another client is not being broken yet.
+ */
+unsigned int nfsd4_layout_holders(struct inode *inode)
+{
+	struct file_lock_context *ctx = locks_inode_context(inode);
+	struct nfs4_client *clp = NULL;
+	unsigned int holders = 0;
+	struct svc_rqst *rqstp;
+	struct file_lease *fl;
+
+	if (!ctx || list_empty_careful(&ctx->flc_lease))
+		return 0;
+	rqstp = nfsd_current_rqst();
+	if (nfsd_v4client(rqstp)) {
+		struct nfsd_thread_local_info *ntli = rqstp->rq_private;
+
+		clp = *ntli->ntli_lease_breaker;
+	}
+
+	spin_lock(&ctx->flc_lock);
+	for_each_file_lock(fl, &ctx->flc_lease) {
+		struct nfs4_layout_stateid *ls = fl->c.flc_owner;
+
+		if (fl->fl_lmops != &nfsd4_layouts_lm_ops)
+			continue;
+		if (clp && ls->ls_stid.sc_client == clp) {
+			holders |= NFSD_LAYOUTS_OWN;
+			continue;
+		}
+		holders |= NFSD_LAYOUTS_OTHER;
+		if (!(fl->c.flc_flags &
+		      (FL_UNLOCK_PENDING | FL_DOWNGRADE_PENDING)))
+			holders |= NFSD_LAYOUTS_IDLE;
+	}
+	spin_unlock(&ctx->flc_lock);
+	return holders;
+}
+
 static void
 nfsd4_free_layout_stateid(struct nfs4_stid *stid)
 {
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 0848baad68..4c0788b450 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -2384,6 +2384,14 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 		spin_unlock(&async_copy->cp_copy.cp_clp->async_lock);
 		status = nfs_ok;
 	} else {
+		struct inode *src = file_inode(copy->nf_src->nf_file);
+		struct inode *dst = file_inode(copy->nf_dst->nf_file);
+
+		/* A copy can be done as a clone, which breaks both files */
+		status = nfserrno(nfsd_break_layouts(src,
+						     dst != src ? dst : NULL));
+		if (status)
+			goto out;
 		status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
 				       copy->nf_dst->nf_file, true);
 		if ((READ_ONCE(copy->nf_dst->nf_file->f_mode) &
diff --git a/fs/nfsd/pnfs.h b/fs/nfsd/pnfs.h
index 559df0e5b7..220ab25986 100644
--- a/fs/nfsd/pnfs.h
+++ b/fs/nfsd/pnfs.h
@@ -72,12 +72,22 @@ int nfsd4_set_deviceid(struct nfsd4_deviceid *id, const struct svc_fh *fhp,
 struct nfsd4_deviceid_map *nfsd4_find_devid_map(int idx);
 #endif /* CONFIG_NFSD_V4 */
 
+/*
+ * What nfsd4_layout_holders() returns: the client of the request holds a
+ * layout lease (OWN), another client holds one (OTHER), a lease of another
+ * client is not being broken yet (IDLE).
+ */
+#define NFSD_LAYOUTS_OWN	0x1
+#define NFSD_LAYOUTS_OTHER	0x2
+#define NFSD_LAYOUTS_IDLE	0x4
+
 #ifdef CONFIG_NFSD_PNFS
 void nfsd4_setup_layout_type(struct svc_export *exp);
 void nfsd4_return_all_client_layouts(struct nfs4_client *);
 void nfsd4_return_all_file_layouts(struct nfs4_client *clp,
 		struct nfs4_file *fp);
 void nfsd4_close_layout(struct nfs4_layout_stateid *ls);
+unsigned int nfsd4_layout_holders(struct inode *inode);
 void nfsd4_stop_layout_fence(struct nfs4_layout_stateid *ls);
 int nfsd4_init_pnfs(void);
 void nfsd4_exit_pnfs(void);
@@ -101,6 +111,11 @@ static inline void nfsd4_close_layout(struct nfs4_layout_stateid *ls)
 {
 }
 
+static inline unsigned int nfsd4_layout_holders(struct inode *inode)
+{
+	return 0;
+}
+
 static inline void nfsd4_stop_layout_fence(struct nfs4_layout_stateid *ls)
 {
 }
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 79dfa47b82..3df871d301 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -62,6 +62,7 @@
 #define	NFS4_CLIENTS_PER_GB		1024
 #define NFSD_DELEGRETURN_TIMEOUT	(HZ / 34)	/* 30ms */
 #define	NFSD_CB_GETATTR_TIMEOUT		NFSD_DELEGRETURN_TIMEOUT
+#define	NFSD_LAYOUTRETURN_TIMEOUT	NFSD_DELEGRETURN_TIMEOUT
 
 typedef struct {
 	u32             cl_boot;
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index f4069da853..a3303d896d 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -46,6 +46,7 @@
 #include "netns.h"
 #include "stats.h"
 #include "vfs.h"
+#include "pnfs.h"
 #include "filecache.h"
 #include "trace.h"
 
@@ -583,6 +584,15 @@ nfsd_setattr(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		err = nfsd_get_write_access(fhp);
 		if (err)
 			return err;
+		/*
+		 * Not under inode_lock: the LAYOUTCOMMIT that a client sends
+		 * before it returns its layout takes it.
+		 */
+		err = nfserrno(nfsd_break_layouts(inode, NULL));
+		if (err) {
+			put_write_access(inode);
+			return err;
+		}
 	}
 
 	inode_lock(inode);
@@ -721,6 +731,14 @@ int nfsd_clone_file_range(struct file *src, u64 src_pos, struct file *dst,
 			  u64 dst_pos, u64 count, errseq_t *since)
 {
 	loff_t cloned;
+	int ret;
+
+	/* The filesystem breaks the layouts of both files */
+	ret = nfsd_break_layouts(file_inode(src),
+				 file_inode(dst) != file_inode(src) ?
+				 file_inode(dst) : NULL);
+	if (ret)
+		return ret;
 
 	*since = READ_ONCE(dst->f_wb_err);
 	cloned = vfs_clone_file_range(src, src_pos, dst, dst_pos, count, 0);
@@ -794,7 +812,9 @@ __be32 nfsd4_vfs_fallocate(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (!S_ISREG(file_inode(file)->i_mode))
 		return nfserr_inval;
 
-	error = vfs_fallocate(file, flags, offset, len);
+	error = nfsd_break_layouts(file_inode(file), NULL);
+	if (!error)
+		error = vfs_fallocate(file, flags, offset, len);
 	if (!error)
 		error = commit_metadata(fhp);
 
@@ -881,6 +901,61 @@ int nfsd_open_break_lease(struct inode *inode, int access)
 	return break_lease(inode, mode | O_NONBLOCK);
 }
 
+static bool nfsd_other_layouts(struct inode *inode)
+{
+	return nfsd4_layout_holders(inode) & NFSD_LAYOUTS_OTHER;
+}
+
+/**
+ * nfsd_break_layouts - Recall the layouts of one or two files
+ * @inode: file that is about to be modified
+ * @inode2: the other file of a COPY or CLONE, or NULL
+ *
+ * A filesystem that hands out layouts recalls them before it modifies a
+ * file, and waits until they are returned or the lease break times out.
+ * The client decides how long that takes, so an nfsd thread must not wait
+ * there. Start the recalls here without waiting. When this starts the
+ * recall of layouts of other clients, give them a short time to come back,
+ * as nfsd_wait_for_delegreturn() does for delegations. Do not wait for a
+ * recall that is already under way, nor for the layouts of the client of
+ * this request, which returns them only once its I/O through them is done.
+ *
+ * Context: Process context. Sleeps for up to NFSD_LAYOUTRETURN_TIMEOUT.
+ *
+ * Return: zero if no layout is left, -EWOULDBLOCK, which nfserrno() turns
+ * into NFS4ERR_DELAY, or -ENOMEM
+ */
+int nfsd_break_layouts(struct inode *inode, struct inode *inode2)
+{
+	long timeout = NFSD_LAYOUTRETURN_TIMEOUT;
+	unsigned int holders;
+	int ret, ret2 = 0;
+
+	holders = nfsd4_layout_holders(inode);
+	if (inode2)
+		holders |= nfsd4_layout_holders(inode2);
+	if (!holders)
+		return 0;
+
+	ret = break_layout(inode, false);
+	if (inode2)
+		ret2 = break_layout(inode2, false);
+	if (ret != -EWOULDBLOCK && ret2 != -EWOULDBLOCK)
+		return ret ? ret : ret2;
+	if ((holders & NFSD_LAYOUTS_OWN) || !(holders & NFSD_LAYOUTS_IDLE))
+		return -EWOULDBLOCK;
+
+	timeout = wait_var_event_timeout(inode, !nfsd_other_layouts(inode),
+					 timeout);
+	if (inode2 && timeout)
+		wait_var_event_timeout(inode2, !nfsd_other_layouts(inode2),
+				       timeout);
+	ret = break_layout(inode, false);
+	if (!ret && inode2)
+		ret = break_layout(inode2, false);
+	return ret;
+}
+
 /*
  * Open an existing file or directory.
  * The may_flags argument indicates the type of open (read/write/lock)
@@ -1490,6 +1565,10 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp,
 		goto out_nfserr;
 	}
 
+	host_err = nfsd_break_layouts(file_inode(file), NULL);
+	if (host_err)
+		goto out_nfserr;
+
 	since = READ_ONCE(file->f_wb_err);
 	if (verf)
 		nfsd_copy_write_verifier(verf, nn);
diff --git a/fs/nfsd/vfs.h b/fs/nfsd/vfs.h
index eb75e4c823..1f05c83a3e 100644
--- a/fs/nfsd/vfs.h
+++ b/fs/nfsd/vfs.h
@@ -171,6 +171,7 @@ __be32		nfsd_setxattr(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			    char *name, void *buf, u32 len, u32 flags);
 #endif
 int 		nfsd_open_break_lease(struct inode *, int);
+int		nfsd_break_layouts(struct inode *inode, struct inode *inode2);
 __be32		nfsd_open(struct svc_rqst *, struct svc_fh *, umode_t,
 				int, struct file **);
 int		nfsd_open_verified(struct svc_fh *fhp, umode_t type, int may_flags,

-- 
2.43.0

[PATCH v2 3/7] nfsd: make the time limit of a layout recall work

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

nfsd4_cb_layout_done() polls a client that has answered CB_LAYOUTRECALL
and still holds layouts, and is meant to give up after two lease
periods: the recall then fails and the client is fenced. The limit is
computed from task->tk_start. The poll restarts the RPC task from
->rpc_call_done(), and rpc_exit_task() resets the statistics of a task
that was restarted there, tk_start included. So tk_start is the time
of the last poll, and the limit is never reached: a client that keeps
answering is polled every 10 ms for as long as it holds the layout.

Count the two lease periods from the first answer of the client, so that
the time in which a callback could not be delivered is not taken from
it, and take the time with ktime_get_boottime(), which goes on while the
server is suspended, as the lease times of nfsd do. A layout stateid is
recalled once in its life (ls_recalled is not cleared), so the time
needs no reset.

When the limit is reached, the existing failure path is taken. It does
not look at the result of ->fence_client(), so a SCSI layout client
whose fencing fails is treated as fenced there, and the layouts it drops
also end the retries of a fence worker that a lease break has started.
Before this patch a client from v6.8 on reached that path 10 ms after
the recall.

A Linux client before v6.8 answers every poll with NFS4_OK until its
LAYOUTRETURN is prepared (read from the code, not run). If such a
client does not return the layout, it was polled without end and is now
fenced after two lease periods, which is what commit 6b9b21073d3b
("nfsd: give up on CB_LAYOUTRECALLs after two lease periods") and
commit 851238a22f3b ("nfsd: fix error handling for clients that fail to
return the layout") set out to do.

Found by reading the code. The limit could only be tested together
with "nfsd: keep polling a layout recall answered with
NFS4ERR_OLD_STATEID", because a current Linux client does not keep
answering a recall with NFS4_OK or NFS4ERR_DELAY: a client writing
through a block layout whose path to the device is cut for 70 seconds
while it can still talk to the server, a lease time of 15 seconds, and
a write to the file on the server. The recall fails 30 seconds after
the first answer of the client and /sbin/nfsd-recall-failed is run.
With the default lease time the limit is 180 seconds; a recall that
comes from a lease break is not waited for that long: after
fs.lease-break-time the breaker has the client fenced.

Fixes: 6b9b21073d3b ("nfsd: give up on CB_LAYOUTRECALLs after two lease periods")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 15 +++++++++++----
 fs/nfsd/state.h       |  1 +
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 035ab79faa..f05e8f3c0f 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -313,6 +313,7 @@ nfsd4_alloc_layout_stateid(struct nfsd4_compound_state *cstate,
 		return NULL;
 	}
 
+	ls->ls_recall_start = 0;
 	ls->ls_fenced = false;
 	ls->ls_fence_inflight = false;
 	ls->ls_fence_stopped = false;
@@ -808,12 +809,18 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 			return 1;
 
 		/* Poll the client until it's done with the layout */
-		now = ktime_get();
+		now = ktime_get_boottime();
 		nn = net_generic(ls->ls_stid.sc_client->net, nfsd_net_id);
 
-		/* Client gets 2 lease periods to return it */
-		cutoff = ktime_add_ns(task->tk_start,
-					 (u64)nn->nfsd4_lease * NSEC_PER_SEC * 2);
+		/*
+		 * Client gets 2 lease periods to return it, counted from its
+		 * first answer. task->tk_start cannot be used for this, it
+		 * is set anew each time the callback is restarted.
+		 */
+		if (!ls->ls_recall_start)
+			ls->ls_recall_start = now;
+		cutoff = ktime_add_ns(ls->ls_recall_start,
+				      (u64)nn->nfsd4_lease * NSEC_PER_SEC * 2);
 
 		if (ktime_before(now, cutoff)) {
 			rpc_delay(task, HZ/100); /* 10 mili-seconds */
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 3df871d301..0920c83857 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -853,6 +853,7 @@ struct nfs4_layout_stateid {
 	struct nfsd_file		*ls_file;
 	struct nfsd4_callback		ls_recall;
 	stateid_t			ls_recall_sid;
+	ktime_t				ls_recall_start;
 	bool				ls_recalled;
 	struct mutex			ls_mutex;
 

-- 
2.43.0

[PATCH v2 4/7] nfsd: run the fence script when a block layout lease break times out

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

When a lease break has waited for fs.lease-break-time,
nfsd4_layout_lm_breaker_timedout() has the client that holds the layout
fenced before the breaker goes on. For a layout type without
->fence_client, which is the block layout, it returns true at once: the
lease is removed and the operation on the server goes on while the
client still holds the layout. /sbin/nfsd-recall-failed, which
Documentation/admin-guide/nfs/pnfs-block-server.rst names as the way to
fence such a client, is only run when the CB_LAYOUTRECALL fails.

This hits a client that keeps answering the recall with NFS4_OK or
NFS4ERR_DELAY for longer than fs.lease-break-time, 45 seconds by
default, for example because its path to the device is cut. With the
previous patch the recall is polled for two lease periods, 180 seconds
by default, so the lease break times out first. A Linux client before
v6.8 answers that way (read from the code, not run), and with the next
patch every Linux client is polled that way; until now a client from
v6.8 on has the recall fail after 10 ms.

Let the fence worker run the script for a layout type without
->fence_client, and give up the lease after that. As on the recall
failure path, the exit status of the script is not looked at. The
same document says that the server retries a failed fence and keeps
the file from other clients meanwhile; that is what the fence worker
does with ->fence_client(). Doing it for the script would block the
file for good on a server that has no /sbin/nfsd-recall-failed, so
this patch leaves it as the recall failure path has it.

As for the SCSI layout since the commit this fixes, the client is then
fenced after fs.lease-break-time. That is shorter than one lease
period by default, while RFC 8881 section 12.5.5 has the server wait
one lease period after a recall before it takes further action.
Setting fs.lease-break-time to at least the lease time gives that wait
for both layout types.

The script runs on its own workqueue, nfsd4_fence, in a work item that
keeps copies of the client address and the device name and nothing of
the layout stateid or the client. The fence worker checks every 100 ms
whether it is done. Tearing down the client waits for the fence worker
since "nfsd: drain pNFS fence work during state teardown", and the
script may be what tears the client down, through
/proc/fs/nfsd/clients/<clid>/ctl: run by the fence worker itself, such a
script would wait for itself. A script that does not exit keeps the
lease, so the operation that broke it waits as long, as it does while a
SCSI fence keeps failing, and unloading nfsd waits for it too.

The recall failure path is not changed. nfsd4_cb_layout_done() still
runs the script in rpciod while the callback is in flight, and tearing
down the client waits for that callback in nfsd4_shutdown_callback(), so
a script that expires the client it fences still hangs there, as it does
without this patch: in a test with a lease time of 15 seconds, where the
recall reaches its time limit before the lease break times out, the
script stayed in its write to the ctl file, in
nfsd4_shutdown_callback(), and rpciod in call_usermodehelper_exec().
After a lease break has fenced a client, that path can also run the
script a second time, if the client does not return the layout within
the time limit of the recall.

This makes a lease break wait longer, and the next patch more so. Apply
it only together with "nfsd: answer NFS4ERR_DELAY instead of waiting for
a layout recall", earlier in this series, which keeps nfsd threads out
of that wait.

Tested together with the next patch: a client writing through a block
layout whose path to the device is cut for 70 seconds while it can still
talk to the server, default lease and lease break times, and a 4 KiB
write to the file on the server. Without this patch (the time limit and
the next patch on nfsd-testing 32eb1a60b456) the write on the server
goes on after 47 seconds and /sbin/nfsd-recall-failed is not run. With
it the script is run after 47 seconds and the write goes on then. In
both runs the client returns the layout when its path is back, and the
recall completes. With a script that expires the client it fences
through its ctl file, the script was done 36 ms after it started, and
the write on the server went on 46 seconds after it started.

Fixes: f52792f484ba ("NFSD: Enforce timeout on layout recall and integrate lease manager fencing")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 126 +++++++++++++++++++++++++++++++++++++++++++++-----
 fs/nfsd/state.h       |   4 ++
 2 files changed, 119 insertions(+), 11 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index f05e8f3c0f..8bcf4d5380 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -24,6 +24,22 @@ struct nfs4_layout {
 
 static struct kmem_cache *nfs4_layout_cache;
 static struct kmem_cache *nfs4_layout_stateid_cache;
+static struct workqueue_struct *nfsd4_fence_wq;
+
+/*
+ * A fence script run for a layout stateid. It keeps copies of what the
+ * script needs and none of the stateid or the client, which tearing down
+ * the client frees.
+ */
+struct nfsd4_fence_script {
+	struct work_struct	work;
+	refcount_t		ref;
+	bool			done;
+	char			addr[INET6_ADDRSTRLEN];
+	char			s_id[sizeof_field(struct super_block, s_id)];
+};
+
+#define NFSD4_FENCE_SCRIPT_POLL	(HZ / 10)
 
 static const struct nfsd4_callback_ops nfsd4_cb_layout_ops;
 static const struct lease_manager_operations nfsd4_layouts_lm_ops;
@@ -741,10 +757,8 @@ nfsd4_return_all_file_layouts(struct nfs4_client *clp, struct nfs4_file *fp)
 }
 
 static void
-nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
+nfsd4_run_fence_script(char *addr_str, char *s_id)
 {
-	struct nfs4_client *clp = ls->ls_stid.sc_client;
-	char addr_str[INET6_ADDRSTRLEN];
 	static char const nfsd_recall_failed[] = "/sbin/nfsd-recall-failed";
 	static char *envp[] = {
 		"HOME=/",
@@ -755,15 +769,13 @@ nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
 	char *argv[8];
 	int error;
 
-	rpc_ntop((struct sockaddr *)&clp->cl_addr, addr_str, sizeof(addr_str));
-
 	printk(KERN_WARNING
 		"nfsd: client %s failed to respond to layout recall. "
 		"  Fencing..\n", addr_str);
 
 	argv[0] = (char *)nfsd_recall_failed;
 	argv[1] = addr_str;
-	argv[2] = file->nf_file->f_path.mnt->mnt_sb->s_id;
+	argv[2] = s_id;
 	argv[3] = NULL;
 
 	error = call_usermodehelper(nfsd_recall_failed, argv, envp,
@@ -774,6 +786,57 @@ nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
 	}
 }
 
+static void
+nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
+{
+	struct nfs4_client *clp = ls->ls_stid.sc_client;
+	char addr_str[INET6_ADDRSTRLEN];
+
+	rpc_ntop((struct sockaddr *)&clp->cl_addr, addr_str, sizeof(addr_str));
+	nfsd4_run_fence_script(addr_str,
+			       file->nf_file->f_path.mnt->mnt_sb->s_id);
+}
+
+static void nfsd4_fence_script_put(struct nfsd4_fence_script *fs)
+{
+	if (refcount_dec_and_test(&fs->ref))
+		kfree(fs);
+}
+
+static void nfsd4_fence_script_work(struct work_struct *work)
+{
+	struct nfsd4_fence_script *fs =
+		container_of(work, struct nfsd4_fence_script, work);
+
+	nfsd4_run_fence_script(fs->addr, fs->s_id);
+	/* Pairs with the smp_load_acquire() in nfsd4_layout_fence_worker() */
+	smp_store_release(&fs->done, true);
+	nfsd4_fence_script_put(fs);
+}
+
+/*
+ * Run the fence script on nfsd4_fence_wq rather than in the fence worker,
+ * which only checks whether it is done. Tearing down the client waits for
+ * the fence worker, and the script may be what tears the client down,
+ * through /proc/fs/nfsd/clients/<clid>/ctl.
+ */
+static struct nfsd4_fence_script *
+nfsd4_start_fence_script(struct nfs4_layout_stateid *ls, struct nfsd_file *nf)
+{
+	struct nfs4_client *clp = ls->ls_stid.sc_client;
+	struct nfsd4_fence_script *fs;
+
+	fs = kzalloc_obj(*fs);
+	if (!fs)
+		return NULL;
+	refcount_set(&fs->ref, 2);
+	rpc_ntop((struct sockaddr *)&clp->cl_addr, fs->addr, sizeof(fs->addr));
+	strscpy(fs->s_id, nf->nf_file->f_path.mnt->mnt_sb->s_id);
+	INIT_WORK(&fs->work, nfsd4_fence_script_work);
+	queue_work(nfsd4_fence_wq, &fs->work);
+	return fs;
+}
+
 static bool
 nfsd4_cb_layout_prepare(struct nfsd4_callback *cb)
 {
@@ -911,6 +974,13 @@ nfsd4_layout_lm_open_conflict(struct file *filp, int arg)
 
 static void nfsd4_layout_fence_done(struct nfs4_layout_stateid *ls)
 {
+	struct nfsd4_fence_script *fs = ls->ls_fence_script;
+
+	/* The script runs on, but nothing waits for it any more. */
+	ls->ls_fence_script = NULL;
+	if (fs)
+		nfsd4_fence_script_put(fs);
+
 	/* Unlock the lease so that tasks waiting on it can proceed. */
 	nfsd4_close_layout(ls);
 
@@ -954,6 +1024,7 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
 	struct delayed_work *dwork = to_delayed_work(work);
 	struct nfs4_layout_stateid *ls = container_of(dwork,
 			struct nfs4_layout_stateid, ls_fence_work);
+	const struct nfsd4_layout_ops *ops;
 	struct nfsd_file *nf;
 	struct block_device *bdev;
 	struct nfs4_client *clp;
@@ -977,7 +1048,33 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
 	clp = ls->ls_stid.sc_client;
 	nn = net_generic(clp->net, nfsd_net_id);
 	bdev = nf->nf_file->f_path.mnt->mnt_sb->s_bdev;
-	if (nfsd4_layout_ops[ls->ls_layout_type]->fence_client(ls, nf)) {
+	ops = nfsd4_layout_ops[ls->ls_layout_type];
+	if (!ops->fence_client) {
+		struct nfsd4_fence_script *fs = ls->ls_fence_script;
+
+		/*
+		 * The layout type cannot fence. Run the fence script, as a
+		 * failed recall does, and give the lease up once it is done.
+		 */
+		if (!fs) {
+			fs = nfsd4_start_fence_script(ls, nf);
+			ls->ls_fence_script = fs;
+		}
+		nfsd_file_put(nf);
+		/* Pairs with nfsd4_fence_script_work() */
+		if (fs && smp_load_acquire(&fs->done))
+			goto dispose;
+		spin_lock(&ls->ls_lock);
+		if (ls->ls_fence_stopped) {
+			spin_unlock(&ls->ls_lock);
+			goto dispose;
+		}
+		mod_delayed_work(system_dfl_wq, &ls->ls_fence_work,
+				 NFSD4_FENCE_SCRIPT_POLL);
+		spin_unlock(&ls->ls_lock);
+		return;
+	}
+	if (ops->fence_client(ls, nf)) {
 		/* fenced ok */
 		nfsd_file_put(nf);
 		pr_warn("%s: FENCED client[%pISpc] clid[%d] to device[%s]\n",
@@ -1029,8 +1126,9 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
  * nfsd4_layout_lm_breaker_timedout - The layout recall has timed out.
  * @fl: file to check
  *
- * If the layout type supports a fence operation, schedule a worker to
- * fence the client from accessing the block device.
+ * Schedule a worker to fence the client from accessing the block device.
+ * If the layout type has no fence operation, the worker runs the fence
+ * script instead.
  *
  * This function runs under the protection of the spin_lock flc_lock.
  * At this time, the file_lease associated with the layout stateid is
@@ -1052,8 +1150,6 @@ nfsd4_layout_lm_breaker_timedout(struct file_lease *fl)
 {
 	struct nfs4_layout_stateid *ls = fl->c.flc_owner;
 
-	if (!nfsd4_layout_ops[ls->ls_layout_type]->fence_client)
-		return true;
 	/*
 	 * Make sure layout has not been returned yet before
 	 * taking a reference count on the layout stateid. The
@@ -1107,6 +1203,13 @@ nfsd4_init_pnfs(void)
 		kmem_cache_destroy(nfs4_layout_cache);
 		return -ENOMEM;
 	}
+
+	nfsd4_fence_wq = alloc_workqueue("nfsd4_fence", WQ_UNBOUND, 0);
+	if (!nfsd4_fence_wq) {
+		kmem_cache_destroy(nfs4_layout_stateid_cache);
+		kmem_cache_destroy(nfs4_layout_cache);
+		return -ENOMEM;
+	}
 	return 0;
 }
 
@@ -1115,6 +1218,7 @@ nfsd4_exit_pnfs(void)
 {
 	int i;
 
+	destroy_workqueue(nfsd4_fence_wq);
 	kmem_cache_destroy(nfs4_layout_cache);
 	kmem_cache_destroy(nfs4_layout_stateid_cache);
 
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 0920c83857..45bd5c122c 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -843,6 +843,8 @@ static inline struct nfs4_ol_stateid *openlockstateid(struct nfs4_stid *s)
 	return container_of(s, struct nfs4_ol_stateid, st_stid);
 }
 
+struct nfsd4_fence_script;
+
 struct nfs4_layout_stateid {
 	struct nfs4_stid		ls_stid;
 	struct list_head		ls_perclnt;
@@ -863,6 +865,8 @@ struct nfs4_layout_stateid {
 	bool				ls_fenced;
 	bool				ls_fence_inflight;
 	bool				ls_fence_stopped;
+	/* The fence script the fence worker waits for */
+	struct nfsd4_fence_script	*ls_fence_script;
 };
 
 static inline struct nfs4_layout_stateid *layoutstateid(struct nfs4_stid *s)

-- 
2.43.0

[PATCH v2 5/7] nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

When a client answers CB_LAYOUTRECALL with NFS4_OK and still holds
layouts, nfsd4_cb_layout_done() sends the callback again after 10 ms to
see if the client is done. The repeat carries the stateid of the first
callback, although RFC 8881 section 12.5.3 has the server increment the
seqid in each CB_LAYOUTRECALL request.

The same section makes the seqid of a processed CB_LAYOUTRECALL the
current seqid of the client. Since commit dce72920c81b ("NFSv4.1: if
referring calls are complete, trust the stateid argument"), which is in
v6.8, the Linux client answers a recall whose seqid is not newer than
its own with NFS4ERR_OLD_STATEID, unless a LAYOUTRETURN is on the wire
(NFS4ERR_DELAY) or the layouts are gone (NFS4ERR_NOMATCHING_LAYOUT).
Before that commit it answered a repeat of the recall it was working
on with NFS4_OK until its LAYOUTRETURN was prepared, and with
NFS4ERR_OLD_STATEID from then until nfsd had processed the LAYOUTRETURN
(read from the code, not run). So a client from v6.8 on that needs
more than 10 ms to finish its I/O and to send LAYOUTRETURN answers the
second callback with NFS4ERR_OLD_STATEID, and an older client can
answer a later one that way.

nfsd4_cb_layout_done() takes that as a failed recall. With the block
layout it logs "client ... failed to respond to layout recall.
Fencing.." and runs /sbin/nfsd-recall-failed, with the SCSI layout it
fences the client with a persistent reservation preempt, and in both
cases nfsd4_cb_layout_release() drops the layouts. Section 12.5.5
requires the server to wait one lease period after a recall before it
takes further action. Here the client is still using the layout: its
I/O can be in flight when the conflicting operation goes on, and what
it has written to newly allocated blocks stays unwritten on the server,
because the layout is gone when the client wants to commit it.

Clients doing O_DIRECT I/O in 1 MiB requests on one file for 30 seconds,
block layout on XFS. "after" is this patch with "nfsd: make the time
limit of a layout recall work", on nfsd-testing 32eb1a60b456.

Recalls that ended in the failure path, out of the recalls sent:

                                                   before      after
  2 clients read (different halves of the file)    7 / 9       0 / 9
  2 clients write                                  8 / 9       0 / 9
  3 clients write                                  19 / 20     0 / 20
  2 clients write, the server writes 4 KiB to
  the file every 50 ms                             216 / 229   0 / 250

Blocks that read back as zeroes on the server, out of the 1 MiB blocks
that the clients wrote to a new file with O_DIRECT, followed by one
successful fsync():

                                                   before      after
  2 clients write                                  6 / 564     0 / 481
  3 clients write                                  11 / 527    0 / 569
  1 client writes, the server sets the file size
  every second                                     21 / 469    0 / 504
  2 clients write, file preallocated               8 / 601     0 / 602
  1 client writes, 1 client reads                  4 / 493     0 / 508

One client writing through a block layout whose path to the device is
cut for 15 seconds while it can still talk to the server, and a 4 KiB
write to the file on the server:

                                                   before      after
  the write on the server takes                    0.05 s      15.6 s
  the recall                                       fails       completes
  /sbin/nfsd-recall-failed is run                  yes         no

Before, the write of the client that was in flight when the layout was
recalled was stuck for 18.5 seconds and no WRITE was sent to the
server, so it reached the device about 15 seconds after the write on
the server had returned. After, the write on the server waits for it.

Two clients writing one file with the SCSI layout on an nvmet-tcp
namespace: before, both recalls sent ended in nfsd4_scsi_fence_client(),
the number of registrants on the namespace went from 4 to 2, and both
clients logged reservation conflicts and did the rest of the run
through the server. The same with a new file, after a fresh boot: both
clients were fenced, and one of the 630 blocks they had written read
back as zeroes. After, no client is fenced and no block is lost.

Handle NFS4ERR_OLD_STATEID like NFS4ERR_DELAY: the client has seen the
recall, so keep polling until the layouts are returned or the time limit
is reached. Apply this only together with three earlier patches in this
series: the limit only works with "nfsd: make the time limit of a
layout recall work", with "nfsd: run the fence script when a block
layout lease break times out" a block layout client is fenced when the
lease break gives up, and "nfsd: answer NFS4ERR_DELAY instead of
waiting for a layout recall" keeps nfsd threads out of the longer wait.
Kernels before v7.1 do not have the lease break timeout that the second
of them fixes. There, an operation on the server that does not come
from nfsd, and an nfsd thread that still ends up in the filesystem's
break, waits until the recall ends, for at most two lease periods. Such
a kernel already waits that way, without a limit, for a client that
keeps answering NFS4ERR_DELAY, and RFC 8881 section 12.5.5 asks the
server to wait at least one lease period before it fences.

The poll sends the callback every 10 ms for as long as the client
holds the layout, as it already does for a client that answers
NFS4ERR_DELAY. With a client that needs long to return the layout
that is about 100 callbacks a second, for up to two lease periods.

A client can also answer this way without having processed the recall,
when a LAYOUTRETURN that left layouts behind has moved its seqid past
the seqid of the recall. With this patch alone it is then polled until
the time limit and fenced, where it used to be fenced at once, and
LAYOUTGETs for the file fail with NFS4ERR_RECALLCONFLICT meanwhile;
"nfsd: poll a layout recall with a new seqid when the client is past
it", later in this series, gives such a client a new seqid.

This keeps the poll as it is for a client that has answered NFS4_OK.
Sending each repeat with a new seqid, or ending the callback on NFS4_OK
and letting the LAYOUTRETURN finish the recall, would follow the RFC;
both are larger changes than a fix for stable kernels should be.

Fixes: 6b9b21073d3b ("nfsd: give up on CB_LAYOUTRECALLs after two lease periods")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 8bcf4d5380..5692d0aee7 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -863,7 +863,15 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 	switch (task->tk_status) {
 	case 0:
 	case -NFS4ERR_DELAY:
+	case -NFS4ERR_OLD_STATEID:
 		/*
+		 * RFC 8881 section 12.5.3 wants a new seqid in each
+		 * CB_LAYOUTRECALL request. The poll below sends the request
+		 * again with the seqid of the first one. A client that has
+		 * processed that one is at its seqid or past it, and the
+		 * Linux client answers the repeat with NFS4ERR_OLD_STATEID
+		 * while it is returning the layouts.
+		 *
 		 * Anything left? If not, then call it done. Note that we don't
 		 * take the spinlock since this is an optimization and nothing
 		 * should get added until the cb counter goes to zero.

-- 
2.43.0

[PATCH v2 6/7] nfsd: poll a layout recall with a new seqid when the client is past it

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

nfsd4_cb_layout_done() polls a client that answers a CB_LAYOUTRECALL
with NFS4ERR_OLD_STATEID, as "nfsd: keep polling a layout recall
answered with NFS4ERR_OLD_STATEID" has it do, and each poll repeats the
seqid that ->prepare gave the recall. NFS4ERR_OLD_STATEID says that
the client is at or past that seqid. A client that has answered
NFS4_OK to the recall is returning its layouts, and the polls end when
it has. A client that has not is past the seqid without having taken
the recall up. A LAYOUTRETURN that returns some layouts and leaves
others gets a new seqid; if nfsd processes it after ->prepare, the
client takes in a seqid above the recall's from its reply. Such a
client answers every poll with NFS4ERR_OLD_STATEID. The recall goes on
for two lease periods, a LAYOUTGET for the file from another client
gets NFS4ERR_RECALLCONFLICT meanwhile, and the client is then fenced,
although it was never asked for the layouts it still holds.

RFC 8881 section 12.5.3 has a new seqid for each CB_LAYOUTRECALL
request. When a client that has not answered NFS4_OK answers
NFS4ERR_OLD_STATEID, give the next poll a new seqid, which the client
takes up. ->prepare clears ls_recall_accepted, and ->done sets it when
the client answers NFS4_OK. The new seqid is traced as
nfsd_layout_recall_new_seqid.

->done runs in rpciod and does not take ls_mutex, which could block it;
nfs4_inc_and_copy_stateid() takes sc_lock. So
nfsd4_preprocess_layout_stateid(), which checks under ls_mutex that the
seqid a client presents is not above the current one, now reads the
current seqid under sc_lock as well. While a recall is out, only a
LAYOUTRETURN changes the seqid besides, since "nfsd: fail a LAYOUTGET in
progress when the layout lease is broken" keeps LAYOUTGET from adding
layouts to a recalled layout stateid.

A pynfs client that holds a layout answers a recall whose seqid is not
above the first seqid of the recall with NFS4ERR_OLD_STATEID, and
returns the layout on a later one. Without this patch, nfsd sent the
recall 501 times in six seconds, all with the first seqid, and a
LAYOUTGET of another client still got NFS4ERR_RECALLCONFLICT. With it,
the second recall carried a new seqid, 10 ms later, the client returned
the layout, and the LAYOUTGET succeeded. The block layout runs of the
earlier patches give the same results with this one and the next: no
recall failed, and every block written to a new file was found.

This builds on "nfsd: keep polling a layout recall answered with
NFS4ERR_OLD_STATEID" and "nfsd: fail a LAYOUTGET in progress when the
layout lease is broken", earlier in this series, and should go to
stable kernels with them.

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 40 ++++++++++++++++++++++++++++++++++++----
 fs/nfsd/state.h       |  2 ++
 fs/nfsd/trace.h       |  1 +
 3 files changed, 39 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 5692d0aee7..f54515e7f6 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -363,6 +363,7 @@ nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp,
 	struct nfs4_stid *stid;
 	unsigned short typemask = SC_TYPE_LAYOUT;
 	__be32 status;
+	bool after;
 
 	if (create)
 		typemask |= (SC_TYPE_OPEN | SC_TYPE_LOCK | SC_TYPE_DELEG);
@@ -391,7 +392,15 @@ nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp,
 
 		status = nfserr_bad_stateid;
 		mutex_lock(&ls->ls_mutex);
-		if (nfsd4_stateid_generation_after(stateid, &stid->sc_stateid))
+		/*
+		 * The recall callback can move the seqid on without
+		 * ls_mutex, under sc_lock.
+		 */
+		spin_lock(&stid->sc_lock);
+		after = nfsd4_stateid_generation_after(stateid,
+						       &stid->sc_stateid);
+		spin_unlock(&stid->sc_lock);
+		if (after)
 			goto out_unlock_stid;
 		if (layout_type != ls->ls_layout_type)
 			goto out_unlock_stid;
@@ -846,6 +855,7 @@ nfsd4_cb_layout_prepare(struct nfsd4_callback *cb)
 	mutex_lock(&ls->ls_mutex);
 	nfs4_inc_and_copy_stateid(&ls->ls_recall_sid, &ls->ls_stid);
 	mutex_unlock(&ls->ls_mutex);
+	ls->ls_recall_accepted = false;
 	return true;
 }
 
@@ -861,13 +871,35 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 
 	trace_nfsd_cb_layout_done(&ls->ls_stid.sc_stateid, task);
 	switch (task->tk_status) {
-	case 0:
-	case -NFS4ERR_DELAY:
 	case -NFS4ERR_OLD_STATEID:
+		/*
+		 * The client is at or past the seqid of the request. Unless
+		 * it has answered NFS4_OK to this recall and is returning
+		 * layouts, it has not taken the recall up, for instance
+		 * because a LAYOUTRETURN moved it past the seqid. Poll with a
+		 * new seqid then (RFC 8881 section 12.5.3), which such a
+		 * client answers afresh.
+		 *
+		 * ->done runs in rpciod and does not take ls_mutex, which
+		 * could block it. While the recall is out, only a
+		 * LAYOUTRETURN changes the seqid besides, and the check of the
+		 * seqid a client presents only asks whether it is above the
+		 * current one.
+		 */
+		if (!ls->ls_recall_accepted) {
+			nfs4_inc_and_copy_stateid(&ls->ls_recall_sid,
+						  &ls->ls_stid);
+			trace_nfsd_layout_recall_new_seqid(&ls->ls_recall_sid);
+		}
+		fallthrough;
+	case -NFS4ERR_DELAY:
+	case 0:
+		if (!task->tk_status)
+			ls->ls_recall_accepted = true;
 		/*
 		 * RFC 8881 section 12.5.3 wants a new seqid in each
 		 * CB_LAYOUTRECALL request. The poll below sends the request
-		 * again with the seqid of the first one. A client that has
+		 * again with the seqid it last had. A client that has
 		 * processed that one is at its seqid or past it, and the
 		 * Linux client answers the repeat with NFS4ERR_OLD_STATEID
 		 * while it is returning the layouts.
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 45bd5c122c..173c22b880 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -856,6 +856,8 @@ struct nfs4_layout_stateid {
 	struct nfsd4_callback		ls_recall;
 	stateid_t			ls_recall_sid;
 	ktime_t				ls_recall_start;
+	/* The client has answered NFS4_OK to the recall */
+	bool				ls_recall_accepted;
 	bool				ls_recalled;
 	struct mutex			ls_mutex;
 
diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
index abfdbc637c..e0908cb21c 100644
--- a/fs/nfsd/trace.h
+++ b/fs/nfsd/trace.h
@@ -681,6 +681,7 @@ DEFINE_STATEID_EVENT(layout_return_lookup_fail);
 DEFINE_STATEID_EVENT(layout_recall);
 DEFINE_STATEID_EVENT(layout_recall_empty);
 DEFINE_STATEID_EVENT(layout_recall_done);
+DEFINE_STATEID_EVENT(layout_recall_new_seqid);
 DEFINE_STATEID_EVENT(layout_recall_fail);
 DEFINE_STATEID_EVENT(layout_recall_release);
 

-- 
2.43.0

[PATCH v2 7/7] nfsd: do not fence a client for a layout recall with no layouts left

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

nfsd4_cb_layout_done() fences the client for any answer to a
CB_LAYOUTRECALL that it does not handle, without looking at whether the
layout stateid still has layouts. A LAYOUTRETURN can return the last
layout while the recall is still out. The Linux client answers
NFS4ERR_NOMATCHING_LAYOUT then, which RFC 8881 section 20.3.3 allows
for a client that holds no layouts for the file. A client that answers
NFS4ERR_BAD_STATEID is right too, since section 12.5.3 makes a layout
stateid invalid once the client has no layouts for the file, but it is
fenced for nothing.

Check for layouts before fencing, as the polls already do before their
time limit, and end the recall when none are left. This does not depend
on the earlier patches in this series.

A pynfs client that returns its layout on the recall and then answers
it NFS4ERR_BAD_STATEID: without this patch, nfsd logged "failed to
respond to layout recall" and ran the fence for it; with it, the recall
ended without either.

Fixes: c5c707f96fc9 ("nfsd: implement pNFS layout recalls")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
 fs/nfsd/nfs4layouts.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index f54515e7f6..e7a4eae51e 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -931,6 +931,15 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
 		}
 		fallthrough;
 	default:
+		/*
+		 * Nothing to fence for once the layouts are gone: a
+		 * LAYOUTRETURN can return the last one while the recall is
+		 * still out.
+		 */
+		if (list_empty(&ls->ls_layouts)) {
+			trace_nfsd_layout_recall_done(&ls->ls_stid.sc_stateid);
+			return 1;
+		}
 		/*
 		 * Unknown error or non-responding client, we'll need to fence.
 		 */

-- 
2.43.0

Re: [PATCH v2 5/7] nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID

[Jeff Layton]

On Wed, 2026-10-14 at 16:30 +0900, Daejun Park via B4 Relay wrote:
> From: Daejun Park <daejun7.park@samsung.com>
> 
> When a client answers CB_LAYOUTRECALL with NFS4_OK and still holds
> layouts, nfsd4_cb_layout_done() sends the callback again after 10 ms to
> see if the client is done. The repeat carries the stateid of the first
> callback, although RFC 8881 section 12.5.3 has the server increment the
> seqid in each CB_LAYOUTRECALL request.
> 
> The same section makes the seqid of a processed CB_LAYOUTRECALL the
> current seqid of the client. Since commit dce72920c81b ("NFSv4.1: if
> referring calls are complete, trust the stateid argument"), which is in
> v6.8, the Linux client answers a recall whose seqid is not newer than
> its own with NFS4ERR_OLD_STATEID, unless a LAYOUTRETURN is on the wire
> (NFS4ERR_DELAY) or the layouts are gone (NFS4ERR_NOMATCHING_LAYOUT).
> Before that commit it answered a repeat of the recall it was working
> on with NFS4_OK until its LAYOUTRETURN was prepared, and with
> NFS4ERR_OLD_STATEID from then until nfsd had processed the LAYOUTRETURN
> (read from the code, not run). So a client from v6.8 on that needs
> more than 10 ms to finish its I/O and to send LAYOUTRETURN answers the
> second callback with NFS4ERR_OLD_STATEID, and an older client can
> answer a later one that way.
> 

What eventually breaks this cycle? The client has a newer layout, so it
returns NFS4ERR_OLD_STATEID to the CB_LAYOUTRECALL, and now the server
is going to continually retransmit that request. What makes the server
eventually give up?

Or am I misunderstanding what causes this?

> nfsd4_cb_layout_done() takes that as a failed recall. With the block
> layout it logs "client ... failed to respond to layout recall.
> Fencing.." and runs /sbin/nfsd-recall-failed, with the SCSI layout it
> fences the client with a persistent reservation preempt, and in both
> cases nfsd4_cb_layout_release() drops the layouts. Section 12.5.5
> requires the server to wait one lease period after a recall before it
> takes further action. Here the client is still using the layout: its
> I/O can be in flight when the conflicting operation goes on, and what
> it has written to newly allocated blocks stays unwritten on the server,
> because the layout is gone when the client wants to commit it.
> 
> Clients doing O_DIRECT I/O in 1 MiB requests on one file for 30 seconds,
> block layout on XFS. "after" is this patch with "nfsd: make the time
> limit of a layout recall work", on nfsd-testing 32eb1a60b456.
> 
> Recalls that ended in the failure path, out of the recalls sent:
> 
>                                                    before      after
>   2 clients read (different halves of the file)    7 / 9       0 / 9
>   2 clients write                                  8 / 9       0 / 9
>   3 clients write                                  19 / 20     0 / 20
>   2 clients write, the server writes 4 KiB to
>   the file every 50 ms                             216 / 229   0 / 250
> 
> Blocks that read back as zeroes on the server, out of the 1 MiB blocks
> that the clients wrote to a new file with O_DIRECT, followed by one
> successful fsync():
> 
>                                                    before      after
>   2 clients write                                  6 / 564     0 / 481
>   3 clients write                                  11 / 527    0 / 569
>   1 client writes, the server sets the file size
>   every second                                     21 / 469    0 / 504
>   2 clients write, file preallocated               8 / 601     0 / 602
>   1 client writes, 1 client reads                  4 / 493     0 / 508
> 
> One client writing through a block layout whose path to the device is
> cut for 15 seconds while it can still talk to the server, and a 4 KiB
> write to the file on the server:
> 
>                                                    before      after
>   the write on the server takes                    0.05 s      15.6 s
>   the recall                                       fails       completes
>   /sbin/nfsd-recall-failed is run                  yes         no
> 
> Before, the write of the client that was in flight when the layout was
> recalled was stuck for 18.5 seconds and no WRITE was sent to the
> server, so it reached the device about 15 seconds after the write on
> the server had returned. After, the write on the server waits for it.
> 
> Two clients writing one file with the SCSI layout on an nvmet-tcp
> namespace: before, both recalls sent ended in nfsd4_scsi_fence_client(),
> the number of registrants on the namespace went from 4 to 2, and both
> clients logged reservation conflicts and did the rest of the run
> through the server. The same with a new file, after a fresh boot: both
> clients were fenced, and one of the 630 blocks they had written read
> back as zeroes. After, no client is fenced and no block is lost.
> 
> Handle NFS4ERR_OLD_STATEID like NFS4ERR_DELAY: the client has seen the
> recall, so keep polling until the layouts are returned or the time limit
> is reached. Apply this only together with three earlier patches in this
> series: the limit only works with "nfsd: make the time limit of a
> layout recall work", with "nfsd: run the fence script when a block
> layout lease break times out" a block layout client is fenced when the
> lease break gives up, and "nfsd: answer NFS4ERR_DELAY instead of
> waiting for a layout recall" keeps nfsd threads out of the longer wait.
> Kernels before v7.1 do not have the lease break timeout that the second
> of them fixes. There, an operation on the server that does not come
> from nfsd, and an nfsd thread that still ends up in the filesystem's
> break, waits until the recall ends, for at most two lease periods. Such
> a kernel already waits that way, without a limit, for a client that
> keeps answering NFS4ERR_DELAY, and RFC 8881 section 12.5.5 asks the
> server to wait at least one lease period before it fences.
> 
> The poll sends the callback every 10 ms for as long as the client
> holds the layout, as it already does for a client that answers
> NFS4ERR_DELAY. With a client that needs long to return the layout
> that is about 100 callbacks a second, for up to two lease periods.
> 
> A client can also answer this way without having processed the recall,
> when a LAYOUTRETURN that left layouts behind has moved its seqid past
> the seqid of the recall. With this patch alone it is then polled until
> the time limit and fenced, where it used to be fenced at once, and
> LAYOUTGETs for the file fail with NFS4ERR_RECALLCONFLICT meanwhile;
> "nfsd: poll a layout recall with a new seqid when the client is past
> it", later in this series, gives such a client a new seqid.
> 
> This keeps the poll as it is for a client that has answered NFS4_OK.
> Sending each repeat with a new seqid, or ending the callback on NFS4_OK
> and letting the LAYOUTRETURN finish the recall, would follow the RFC;
> both are larger changes than a fix for stable kernels should be.
> 
> Fixes: 6b9b21073d3b ("nfsd: give up on CB_LAYOUTRECALLs after two lease periods")
> Signed-off-by: Daejun Park <daejun7.park@samsung.com>
> ---
>  fs/nfsd/nfs4layouts.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
> index 8bcf4d5380..5692d0aee7 100644
> --- a/fs/nfsd/nfs4layouts.c
> +++ b/fs/nfsd/nfs4layouts.c
> @@ -863,7 +863,15 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
>  	switch (task->tk_status) {
>  	case 0:
>  	case -NFS4ERR_DELAY:
> +	case -NFS4ERR_OLD_STATEID:
>  		/*
> +		 * RFC 8881 section 12.5.3 wants a new seqid in each
> +		 * CB_LAYOUTRECALL request. The poll below sends the request
> +		 * again with the seqid of the first one. A client that has
> +		 * processed that one is at its seqid or past it, and the
> +		 * Linux client answers the repeat with NFS4ERR_OLD_STATEID
> +		 * while it is returning the layouts.
> +		 *
>  		 * Anything left? If not, then call it done. Note that we don't
>  		 * take the spinlock since this is an optimization and nothing
>  		 * should get added until the cb counter goes to zero.

-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH v2 5/7] nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID

[Jeff Layton]

On Wed, 2026-10-14 at 07:06 -0400, Jeff Layton wrote:
> On Wed, 2026-10-14 at 16:30 +0900, Daejun Park via B4 Relay wrote:
> > From: Daejun Park <daejun7.park@samsung.com>
> > 
> > When a client answers CB_LAYOUTRECALL with NFS4_OK and still holds
> > layouts, nfsd4_cb_layout_done() sends the callback again after 10 ms to
> > see if the client is done. The repeat carries the stateid of the first
> > callback, although RFC 8881 section 12.5.3 has the server increment the
> > seqid in each CB_LAYOUTRECALL request.
> > 
> > The same section makes the seqid of a processed CB_LAYOUTRECALL the
> > current seqid of the client. Since commit dce72920c81b ("NFSv4.1: if
> > referring calls are complete, trust the stateid argument"), which is in
> > v6.8, the Linux client answers a recall whose seqid is not newer than
> > its own with NFS4ERR_OLD_STATEID, unless a LAYOUTRETURN is on the wire
> > (NFS4ERR_DELAY) or the layouts are gone (NFS4ERR_NOMATCHING_LAYOUT).
> > Before that commit it answered a repeat of the recall it was working
> > on with NFS4_OK until its LAYOUTRETURN was prepared, and with
> > NFS4ERR_OLD_STATEID from then until nfsd had processed the LAYOUTRETURN
> > (read from the code, not run). So a client from v6.8 on that needs
> > more than 10 ms to finish its I/O and to send LAYOUTRETURN answers the
> > second callback with NFS4ERR_OLD_STATEID, and an older client can
> > answer a later one that way.
> > 
> 
> What eventually breaks this cycle? The client has a newer layout, so it
> returns NFS4ERR_OLD_STATEID to the CB_LAYOUTRECALL, and now the server
> is going to continually retransmit that request. What makes the server
> eventually give up?
> 
> Or am I misunderstanding what causes this?
> 
> 

Oh, I think I see. The next patch makes it retransmit the
CB_LAYOUTRECALL with a more up-to-date stateid. That should eventually
resolve itself, I think.

It would make sense to squash 5 and 6 together so we don't end up with
this endless cycle if this happens on a bisect.
-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH v2 0/7] nfsd: layout recall and lease break fixes

[Jeff Layton]

On Wed, 2026-10-14 at 16:29 +0900, Daejun Park via B4 Relay wrote:
> nfsd polls a client that has accepted a CB_LAYOUTRECALL by sending the
> callback again every 10 ms, with the stateid of the first callback. A
> Linux client from v6.8 on answers the repeat with NFS4ERR_OLD_STATEID,
> and nfsd takes that as a failed recall. So a client that needs more
> than 10 ms to return a layout has the layout revoked and the fence path
> run. With the block layout that client loses data it has written and
> synced, with the SCSI layout it is cut off from the device. Patch 5
> lets nfsd keep polling instead.
> 
> A recall that takes longer makes the operation that started it wait
> longer. When that operation is a WRITE, a SETATTR that changes the
> size, an ALLOCATE, a DEALLOCATE, a CLONE or a COPY, the nfsd thread
> serving it waits in XFS until the layouts are returned or the lease
> break times out. As Chuck pointed out on v1, a few such requests can
> take every nfsd thread, and the server then cannot serve the
> LAYOUTRETURN that would release them. Patch 2 recalls the layouts
> before the operation without waiting, gives the layouts of other
> clients 30 ms to come back when it starts their recall, and otherwise
> answers NFS4ERR_DELAY.
> 
> The cost of patch 2: a client that writes to a file through its layout
> and, for writes that are not aligned, through the server recalls its
> own layout with each such WRITE and has its LAYOUTGETs fail with
> NFS4ERR_RECALLCONFLICT meanwhile. In test 2 it wrote 48, 46 and 4 1 MiB
> blocks through its layout in 30 seconds instead of 534, 537 and 505;
> before, 6 of 7, 4 of 4 and 17 of 20 recalls ended in the failure path
> and 7, 4 and 20 blocks were lost. The series on top of this one, with
> the NFS client fix it needs, brings that to 193, 192 and 184 blocks.
> 
> Patches 3 and 4 are the other two that patch 5 needs. Patch 3: the
> limit of two lease periods on the poll has never triggered, and without
> it the poll would have no end. Patch 4: for the block layout a lease
> break gives up after fs.lease-break-time (45 s) without running
> /sbin/nfsd-recall-failed. Until now the recall had failed long before
> that; with patch 5 it is polled for two lease periods (180 s). Patch 4
> fences the client at fs.lease-break-time, as the SCSI layout does since
> v7.1; by default that is shorter than the one lease period RFC 8881
> section 12.5.5 has the server wait after a recall.
> 
> Patch 1 fixes a lease break that hits a LAYOUTGET in progress and then
> waits for a layout that is never recalled. It comes first because
> patch 2 makes that common: the break of patch 2 runs outside the XFS
> iolock, so it can land between the allocation of a layout stateid and
> the insertion of its first layout.
> 
> Patches 6 and 7 deal with two answers that the polls handle badly. A
> client that a LAYOUTRETURN has moved past the seqid of the recall
> answers every poll NFS4ERR_OLD_STATEID and is never asked for the
> layouts it still holds; patch 6 polls such a client, one that has not
> answered NFS4_OK, with a new seqid. A client that answers
> NFS4ERR_BAD_STATEID after returning its last layout is fenced; patch 7
> looks for layouts first.
> 
> Patches 1 to 5 have to go together, also to stable kernels. Patch 6
> builds on patches 1 and 5, and patch 7 stands on its own. If patch 2
> should not go to stable, patches 3 to 5 should not either; patches 1
> and 7 can go alone. Kernels before v7.1 do not have the lease break
> timeout that patch 4 fixes; there an operation on the server that does
> not come from nfsd waits for the recall for up to two lease periods, as
> it already does, without a limit, for a client that keeps answering
> NFS4ERR_DELAY. Patch 2 uses nfsd_thread_local_info and needs adapting
> where that is missing.
> 
> Tests. QEMU guests on one host: an nvmet-tcp target, the server (nfsd,
> XFS on the nvmet namespace, pnfs export) and up to three clients, all
> on the same kernel with KASAN and lockdep, NFS v4.2, block layout
> unless noted, O_DIRECT I/O. "v2" is this series on nfsd-testing
> 56589cdb5881, "v1" is v1 on it, and "before" is that commit unless
> noted. No KASAN or lockdep report on the server in any run, nor on the
> clients and the target in tests 2, 3 and 5, where they were checked.
> 
> 1. nfsd threads (patch 2), 4 nfsd threads. A client holds a layout of
>    a file and its path to the device is cut for 70 s. Another client,
>    without pNFS, sends 4 KiB WRITEs to the file through the server, all
>    at once, and reads 4 KiB of another file every second:
> 
>                                             before     v1        v2
>    8 WRITEs, NFSv4.0: the WRITEs take       46.7 s     -         45.2 s
>      longest read of the other file         46.7 s     -         0.031 s
>    64 WRITEs, NFSv4.0: the WRITEs take      -          49.2 s    45.4 s
>      longest read of the other file         -          49.1 s    0.22 s
>    8 WRITEs, NFSv3: the WRITEs take         47.0 s     -         51.2 s
>      longest read of the other file         46.9 s     -         0.021 s
> 
>    The WRITEs wait for the lease break, which ends with
>    /sbin/nfsd-recall-failed (patch 4). NFSv3 WRITEs take longer with
>    v2 because the client waits 5 s after NFS3ERR_JUKEBOX. In the run
>    with 64 WRITEs before this series, the recall failed after 10 ms, as
>    described above, and the WRITEs were done in 0.5 s.
> 
> 2. What patch 2 costs. Two clients write 1 MiB blocks to a new file for
>    30 s, one through its layout and one (NFSv4.0) through the server,
>    three runs. "+ follow-up" adds the nfsd series that goes on top of
>    this one and an NFS client fix that it needs.
> 
>                       through the     of those, read      through the
>                       layout          back as zeroes      server
>    before             416/200/186     28/131/111          120/325/323
>    v2                 174/176/170     0/0/0               184/203/220
>    v2 + follow-up     170/175/176     0/0/0               284/223/207
> 
>    The 30 ms wait is what keeps the writer through the server going:
>    two earlier revisions of v2 + follow-up that differ only in it gave
>    79/79/80 blocks through the server without the wait and 238/230/137
>    with it.
> 
>    One client doing both, 1 MiB blocks through its layout and 512 byte
>    writes through the server:
> 
>                       through the     of those, read      writes through
>                       layout          back as zeroes      the server
>    before             534/537/505     7/4/20              32/21/87
>    v2                 48/46/4         0/0/0               3600/3262/4917
>    v2 + follow-up     193/192/184     0/0/0               1082/1488/1669
> 
>    With v2 alone, each WRITE of that client through the server recalls
>    its layout. About half of its LAYOUTGETs then meet that recall and
>    get NFS4ERR_RECALLCONFLICT, as before; the other half are in
>    progress when the WRITE breaks the lease, and patch 1 fails them.
>    The Linux client waits up to 15 s between LAYOUTGETs that fail with
>    NFS4ERR_RECALLCONFLICT. The follow-up hands out the layout of such a
>    LAYOUTGET and recalls it at once. That needs the referring call the
>    follow-up adds to CB_LAYOUTRECALL: tried in this series, the recall
>    overtook the LAYOUTGET reply, and in all three runs 92 to 104 of
>    about 260 blocks read back as zeroes.
> 
>    On 214e388464bf, before and v2 gave the same picture, except that in
>    one of three runs of the one-client case with v2 one block read back
>    as zeroes, through the recall race described under "Not changed".
> 
>    The series applies to nfsd-testing c32fef6c476d unchanged, and there
>    before and v2 gave the same picture again, also for NFSv3, whose
>    replies now come from xdrgen. In test 1 the longest read of the
>    other file was 47.8 s before and 0.036 s with v2 over NFSv4.0, and
>    48.3 s and 0.019 s over NFSv3, where the WRITEs took 48.4 s and
>    51.3 s. In this test, with two clients, 48, 94 and 90 of 180, 185
>    and 181 blocks through the layout read back as zeroes before, and
>    none of 161, 177 and 172 with v2; with one client, 8, 0 and 3
>    before and none with v2.
> 
>    Writes to a file without leases: fio, NFSv4.0, no KASAN or lockdep,
>    10 runs of 30 s, medians without and with an earlier revision of v2
>    (the same path for such a file): 1 MiB sequential 65.8 and 67.5
>    MiB/s, 4 KiB random 262 and 263 IOPS.
> 
> 3. Recall polling (patches 3 to 6). With v2 no recall ended in the
>    failure path (0 of 9, 9, 63 and 277 in the arms of patch 5) and no
>    block written to a new file was lost (0 of 545, 511, 477, 577 and
>    36). Patch 5 has the runs before.
>    On c32fef6c476d, four of those arms lost 4, 2, 18 and 9 blocks
>    before, with 46 recall failures logged, and none with v2, with no
>    recall failure.
> 
> 4. One client writing through a block layout whose path to the device
>    is cut while it can still talk to the server, and a 4 KiB write to
>    the file on the server ("before" of the 15 s case is 32eb1a60b456).
>    The client's write that is in flight stays stuck until the path is
>    back.
> 
>                                                    before      v2
>    path cut for 15 s: the server write takes       0.05 s      15.6 s
>      the recall                                    fails       completes
>      /sbin/nfsd-recall-failed is run               at once     no
>    path cut for 70 s: the server write takes       0.05 s      47.2 s
>      /sbin/nfsd-recall-failed is run               at once     at 47.1 s
> 
>    The same 70 s case with a script that expires the client it fences
>    through /proc/fs/nfsd/clients/<clid>/ctl: with v1's fence worker,
>    which runs the script itself, the script never finished and the
>    server write went on only after the path was back, after 93 s. With
>    v2 the script was done 36 ms after it started, and the server write
>    took 46.1 s. With a lease time of 15 s, the recall reaches its limit
>    (patch 3) after 30 s, before the lease break times out, and the
>    recall failure path runs the script: there it hangs with v2 as well,
>    the script in nfsd4_shutdown_callback() and rpciod in
>    call_usermodehelper_exec().
> 
> 5. A lease break during a LAYOUTGET (patch 1). With v2 the longest of
>    424 writes on the server took 55 ms, and 108 ms with a delay of
>    100 ms added to LAYOUTGET for the test. Patch 1 has the runs without
>    it.
> 
> 6. SCSI layout. The target runs a kernel with "nvmet: fix preempt of
>    another registrant by the reservation holder" [1], so that the
>    preempt removes the registration. With v2, the path of a client cut
>    for 15 s: the server write takes 15.2 s and the client is not
>    fenced; cut for 70 s: 47.6 s, and the fence worker fences the client
>    (registrants 2 -> 1).
> 
> pynfs BLOCK1, BLOCK2, BLOCK4 and BLOCK5 pass before and with v2 (BLOCK3
> fails in both with a TypeError in the test). The commit messages of
> patches 6 and 7 have pynfs clients that answer as described there.
> 
> Not changed by these patches:
> 
> - A layout granted between the check of patch 2 and the break in XFS
>   still makes the nfsd thread wait. No new layout lease is granted
>   while a break is pending, so that needs a LAYOUTGET for a file that
>   had no layout at the check. An asynchronous COPY runs in its own
>   kthread and still waits there.
> 
> - The recall failure path still runs /sbin/nfsd-recall-failed from
>   nfsd4_cb_layout_done(), in rpciod, while the callback is in flight.
>   A script that expires the client it fences hangs there, because the
>   teardown waits for that callback in nfsd4_shutdown_callback(); test 4
>   shows it. Neither path looks at the exit status of the script, and
>   the recall failure path does not look at the result of
>   ->fence_client(): a fence that fails is taken as done there, and the
>   layouts it drops end the retries of a fence worker that a lease break
>   has started (patch 3 says so).
> 
> - nfsd still polls a client that has answered NFS4_OK with the seqid it
>   has used, where RFC 8881 section 12.5.3 has the server increment the
>   seqid in each request. Patch 5 only makes nfsd bear the answer
>   (patch 6 gives a new seqid only to a client that has not answered
>   NFS4_OK), and while it polls, nfsd sends a callback every 10 ms for
>   up to two lease periods.
> 
> - CB_LAYOUTRECALL carries no referring call, as CB_RECALL does since
>   "NFSD: Send referring calls with CB_RECALL". When a recall overtakes
>   the reply to the LAYOUTGET that created the layout stateid, the Linux
>   client answers NFS4ERR_NOMATCHING_LAYOUT, nfsd drops the layout and
>   the client uses it afterwards. nfsd also trusts
>   NFS4ERR_NOMATCHING_LAYOUT while it still has layouts on the layout
>   stateid. The series on top of this one adds the referring call and
>   resends the recall once before it trusts that answer, and with it
>   recalls a layout handed out by a LAYOUTGET that a lease break met,
>   instead of failing that LAYOUTGET (see test 2).
> 
> - Which client makes progress when several clients use one file: the
>   client that holds a layout now keeps it while it does I/O through
>   it, where before its layout was revoked after 10 ms. Nothing here
>   limits how long one client can keep others out.
> 
> [1] https://lore.kernel.org/r/20261002065931epcms2p1cb2c16f98cf3b05f8c859ad92b0ad465@epcms2p1
> 
> ---
> Changes in v2:
> - New patch 2, which recalls the layouts without waiting before an
>   operation that modifies the file and answers NFS4ERR_DELAY, as Chuck
>   asked:
>   https://lore.kernel.org/r/63ef8d7c-086f-4cc8-84bb-72c1b23bf596@slotpi15m67
> - v1 4/6 is now patch 1, ahead of patch 2, which makes its case common.
>   v1 1/6 to 3/6 are patches 3 to 5, v1 5/6 and 6/6 are patches 6 and 7.
> - Sashiko on v1:
>   - 2/6, the fence script waiting for itself when it expires the client
>     through /proc/fs/nfsd/clients/<clid>/ctl: patch 4 runs the script
>     in a work item on a new workqueue, nfsd4_fence, and the fence
>     worker checks every 100 ms whether it is done. Teardown waits only
>     for the fence worker, which no longer waits for the script.
>   - 2/6, sleeping fence workers filling system_dfl_wq: the same change;
>     a script that runs long occupies nfsd4_fence.
>   - 3/6, ktime_get() stops during suspend: patch 3 takes the time with
>     ktime_get_boottime().
>   - 5/6, si_generation written under sc_lock in ->done and read under
>     ls_mutex only: patch 6 reads it under sc_lock in
>     nfsd4_preprocess_layout_stateid() as well.
>   - 1/6, the result of ->fence_client() not looked at on the recall
>     failure path, which also ends the retries of the fence worker: not
>     changed, said in patch 3. Patch 3 makes the path reachable after two
>     lease periods for a client that keeps answering; before, a client
>     from v6.8 on reached it after 10 ms.
>   - 2/6, the exit status of the script not looked at: not changed.
>     Retrying a failed script would block the file for good on a server
>     that has no /sbin/nfsd-recall-failed. Said in patch 4.
>   - 6/6, NFS4ERR_NOMATCHING_LAYOUT trusted while layouts are left:
>     preexisting, not changed here (see the list above).
> - Rebased onto nfsd-testing c32fef6c476d without changes, and tests 1
>   to 3 run again there.
> - Test numbers in the cover from v2; the commit messages of patches 1,
>   4 and 5 keep their runs on nfsd-testing 32eb1a60b456 for the side
>   without the patch. Commit messages with one space after a period.
> - Link to v1: https://lore.kernel.org/r/20261006-nfsd-layout-recall-v1-0-b59b31caf342@samsung.com
> 
> ---
> Daejun Park (7):
>       nfsd: fail a LAYOUTGET in progress when the layout lease is broken
>       nfsd: answer NFS4ERR_DELAY instead of waiting for a layout recall
>       nfsd: make the time limit of a layout recall work
>       nfsd: run the fence script when a block layout lease break times out
>       nfsd: keep polling a layout recall answered with NFS4ERR_OLD_STATEID
>       nfsd: poll a layout recall with a new seqid when the client is past it
>       nfsd: do not fence a client for a layout recall with no layouts left
> 
>  fs/nfsd/nfs4layouts.c | 301 ++++++++++++++++++++++++++++++++++++++++++++++----
>  fs/nfsd/nfs4proc.c    |   8 ++
>  fs/nfsd/pnfs.h        |  15 +++
>  fs/nfsd/state.h       |   8 ++
>  fs/nfsd/trace.h       |   2 +
>  fs/nfsd/vfs.c         |  81 +++++++++++++-
>  fs/nfsd/vfs.h         |   1 +
>  7 files changed, 391 insertions(+), 25 deletions(-)
> ---
> base-commit: c32fef6c476daa5830f600614657baee59c83339
> change-id: 20261006-nfsd-layout-recall-ba868d33371e
> 
> Best regards,

I think 5 and 6 should be squashed together, but the series itself
looks fine. Nice work!

Reviewed-by: Jeff Layton <jlayton@kernel.org>

Re: [PATCH v2 2/7] nfsd: answer NFS4ERR_DELAY instead of waiting for a layout recall

[Chuck Lever]

On 10/14/26, Daejun Park wrote:
> @@ -2384,6 +2384,14 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  		spin_unlock(&async_copy->cp_copy.cp_clp->async_lock);
>  		status = nfs_ok;
>  	} else {
> +		struct inode *src = file_inode(copy->nf_src->nf_file);
> +		struct inode *dst = file_inode(copy->nf_dst->nf_file);
> +
> +		/* A copy can be done as a clone, which breaks both files */
> +		status = nfserrno(nfsd_break_layouts(src,
> +						     dst != src ? dst : NULL));
> +		if (status)
> +			goto out;
>  		status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
>  				       copy->nf_dst->nf_file, true);

This recalls layouts only for a synchronous COPY. The asynchronous
branch above it hands the same work to nfsd4_do_async_copy(), which
calls nfsd4_do_copy() with no recall at all. An async COPY whose
destination is held under an RW block layout by another client then
writes through the MDS page cache while that client writes the same
extents directly to the device. XFS breaks layouts for truncate,
fallocate and remap, but not for a buffered or spliced write, so
nothing below NFSD catches it either.

The recall therefore belongs before the sync/async split, so both
paths will execute it.

Nit: See if you can cut down the verbiage in the new comments.
kdoc comments are meant to document API contracts. Operational
details can be described close to the code itself (concisely,
please).


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

Re: [PATCH v2 4/7] nfsd: run the fence script when a block layout lease break times out

[Chuck Lever]

On 10/14/26, Daejun Park wrote:
> @@ -1004,7 +1075,33 @@ static void nfsd4_layout_fence_worker(struct work_struct *work)
>  	clp = ls->ls_stid.sc_client;
>  	nn = net_generic(clp->net, nfsd_net_id);
>  	bdev = nf->nf_file->f_path.mnt->mnt_sb->s_bdev;
> -	if (nfsd4_layout_ops[ls->ls_layout_type]->fence_client(ls, nf)) {
> +	ops = nfsd4_layout_ops[ls->ls_layout_type];
> +	if (!ops->fence_client) {
> +		struct nfsd4_fence_script *fs = ls->ls_fence_script;
> +
> +		/*
> +		 * The layout type cannot fence. Run the fence script, as a
> +		 * failed recall does, and give the lease up once it is done.
> +		 */
> +		if (!fs) {
> +			fs = nfsd4_start_fence_script(ls, nf);
> +			ls->ls_fence_script = fs;
> +		}
> +		nfsd_file_put(nf);
> +		/* Pairs with nfsd4_fence_script_work() */
> +		if (fs && smp_load_acquire(&fs->done))
> +			goto dispose;
> +		spin_lock(&ls->ls_lock);
> +		if (ls->ls_fence_stopped) {
> +			spin_unlock(&ls->ls_lock);
> +			goto dispose;
> +		}
> +		mod_delayed_work(system_dfl_wq, &ls->ls_fence_work,
> +				 NFSD4_FENCE_SCRIPT_POLL);
> +		spin_unlock(&ls->ls_lock);
> +		return;
> +	}

When nfsd4_start_fence_script() returns NULL, fs stays NULL,
the done check is skipped, and the worker re-arms itself every
100 ms until an allocation succeeds. The fence_client failure
path right below backs off through ls_fence_delay and warns
once; this path does neither. A GFP_KERNEL allocation this
small rarely fails, so the practical exposure is tiny, but the
retry could reuse ls_fence_delay rather than the script poll
interval when fs is NULL.

Also, the fence script can run twice for one stateid. The
lease break times out first and this worker runs it. If
the client stays silent, the CB_LAYOUTRECALL later reaches its
two-lease-period cutoff and nfsd4_cb_layout_done() runs it
again through nfsd4_cb_layout_fail(), since that path checks
only whether layouts remain and never ls_fenced. For a dead
client both fire. nfsd4_cb_layout_done() could skip the fence
when ls_fenced is already set.

Related: nfsd4_cb_layout_fail() now builds the same address
string and s_id that nfsd4_start_fence_script() builds, and
hands them to the same helper. If the first became a call to
the second, the two fence triggers would share one argv
construction.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)