[PATCH 0/4] NFSD: CB_RECALL_ANY fixes and a meaningful keep count

[PATCH 0/4] NFSD: CB_RECALL_ANY fixes and a meaningful keep count

[Chuck Lever]

NFSD sends every CB_RECALL_ANY with craa_objects_to_keep set to
zero. RFC 8881 Section 20.6.3 reads that as a request to retain no
delegation at all, whether or not the delegation backs a file an
application still has open.

NFSD meant the callback as a signal to return unused delegations.
Only the Linux client has been tested against it, and that client
ignores craa_objects_to_keep and returns every unused delegation of
the named types, so the fixed zero never produced visible
misbehavior.

The NFSv4.0 recipient bug is fixed on its own so it can be
backported to LTS ahead of the rest. An NFSv4.0 callback service
rejects CB_RECALL_ANY, and nothing brings the callback channel back
up afterward (patch 1).

The keep count comes from a new per-client delegation count rather
than a walk of cl_delegations (patch 2). deleg_reaper() runs under
nn->client_lock, and cl_delegations is serialized by nn->deleg_lock,
which nests outside it, so the reaper cannot count the list. It
reads the counter without the lock and may not depend on the value.

Each callback asks a client to give up one delegation. A larger
request reaches delegations that applications still have open, and
both reaper callers re-arm while their condition lasts. A client
holding a single delegation is skipped rather than sent the old
zero. That strands one delegation per client, which this
best-effort design accepts for now.

NFSD sets no recall target and remembers nothing across callbacks.
RFC 8881 Section 20.6.4 prescribes CB_RECALL against specific
delegations once a client fails to return any. NFSD does not take
that step yet. CB_RECALL_ANY is asynchronous and reports no
completion, so NFSD treats it as advisory.

---
Chuck Lever (4):
      NFSD: Do not send CB_RECALL_ANY to NFSv4.0 clients
      NFSD: Count the delegations held by each client
      NFSD: Name directory delegations in the CB_RECALL_ANY type mask
      NFSD: Send a meaningful CB_RECALL_ANY keep count

 fs/nfsd/nfs4state.c | 27 +++++++++++++++++++++++----
 fs/nfsd/state.h     |  2 ++
 2 files changed, 25 insertions(+), 4 deletions(-)
---
base-commit: 1d479c6b53f684b27da84ec352b7efb97f7f115f
change-id: 20260810-recall-any-keep-count-f50c2ae1b792

Best regards,
--  
Chuck Lever <cel@kernel.org>

[PATCH 1/4] NFSD: Do not send CB_RECALL_ANY to NFSv4.0 clients

[Chuck Lever]

deleg_reaper() sends CB_RECALL_ANY to every ACTIVE client holding
delegations, but CB_RECALL_ANY is an NFSv4.1 operation. An NFSv4.0
client's callback service accepts only CB_GETATTR and CB_RECALL, so it
replies OP_ILLEGAL. The decoder maps the unexpected opnum to -EIO, and
nfsd4_cb_done() marks the client's callback channel down.

Nothing brings the channel back. nfsd4_run_cb_work() sets NFSD4_CB_UP
only for a minor version above zero, and the only nfsd4_probe_callback()
call site an NFSv4.0 client reaches is nfsd4_setclientid_confirm(). One
visit from the reaper therefore leaves the channel marked down until the
client re-establishes its clientid. RENEW then returns
NFS4ERR_CB_PATH_DOWN for as long as the client holds delegations.
nfsd4_cb_channel_good() stops returning true, so the client is granted
no further delegations.

Skip clients at minor version zero.

Fixes: 44df6f439a17 ("NFSD: add delegation reaper to react to low memory condition")
Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfsd/nfs4state.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 510380b6aa7a..09b1aa2914bc 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -7947,6 +7947,8 @@ deleg_reaper(struct nfsd_net *nn)
 	list_for_each_safe(pos, next, &nn->client_lru) {
 		clp = list_entry(pos, struct nfs4_client, cl_lru);
 
+		if (clp->cl_minorversion == 0)
+			continue;
 		if (clp->cl_state != NFSD4_ACTIVE)
 			continue;
 		if (list_empty(&clp->cl_delegations))

-- 
2.54.0

[PATCH 2/4] NFSD: Count the delegations held by each client

[Chuck Lever]

struct nfs4_client records the delegations it holds on cl_delegations
but keeps no count of them. deleg_reaper() walks nn->client_lru under
nn->client_lock, but cl_delegations is serialized by nn->deleg_lock,
which nests outside nn->client_lock. A caller there cannot take
nn->deleg_lock to count the list. The cost tells against the walk as
well: an O(n) count per client, on a pass that already visits every
client.

Add cl_deleg_count, maintained at the two sites that mutate
cl_delegations. Both hold nn->deleg_lock, so the counter is already
serialized against itself and needs no atomic of its own. The decrement
sits below the delegation_hashed() test, next to the list_del_init it
pairs with, so it runs only when the delegation really leaves the list.

A reader that holds only nn->client_lock is not synchronized against
either update site, so it can see a count that does not match the
list. Such a reader marks the access with data_race() and may not
depend on the value for correctness.

No functional change.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfsd/nfs4state.c | 2 ++
 fs/nfsd/state.h     | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 09b1aa2914bc..2ddc77ac7312 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1521,6 +1521,7 @@ hash_delegation_locked(struct nfs4_delegation *dp, struct nfs4_file *fp)
 	dp->dl_stid.sc_type = SC_TYPE_DELEG;
 	list_add(&dp->dl_perfile, &fp->fi_delegations);
 	list_add(&dp->dl_perclnt, &clp->cl_delegations);
+	clp->cl_deleg_count++;
 	return 0;
 }
 
@@ -1552,6 +1553,7 @@ unhash_delegation_locked(struct nfs4_delegation *dp, unsigned short statusmask)
 	++dp->dl_time;
 	spin_lock(&fp->fi_lock);
 	list_del_init(&dp->dl_perclnt);
+	dp->dl_stid.sc_client->cl_deleg_count--;
 	list_del_init(&dp->dl_recall_lru);
 	list_del_init(&dp->dl_perfile);
 	spin_unlock(&fp->fi_lock);
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index ff1c9fa731aa..10beeb851cf6 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -632,6 +632,8 @@ struct nfs4_client {
 
 	unsigned int		cl_state;
 	atomic_t		cl_delegs_in_recall;
+	/* Length of cl_delegations, updated under nn->deleg_lock */
+	unsigned int		cl_deleg_count;
 
 	struct nfsd4_cb_recall_any	*cl_ra;
 	time64_t		cl_ra_time;

-- 
2.54.0

[PATCH 3/4] NFSD: Name directory delegations in the CB_RECALL_ANY type mask

[Chuck Lever]

RFC 8881 Section 20.6.3 has craa_type_mask name the types of
recallable object in the resource pool that is over-utilized. The
section distinguishes a server that shares one pool among all
classes from one that keeps separate pools per class. NFSD shares
one. nfsd_get_dir_deleg() allocates through __alloc_init_deleg(), the
same helper the file delegation path uses. A directory delegation comes
from the same slab and is charged against the same num_delegations and
max_delegations budget. The state shrinker counts it.

deleg_reaper() nevertheless names only RCA4_TYPE_MASK_RDATA_DLG and
RCA4_TYPE_MASK_WDATA_DLG, describing a resource pool NFSD does not
have. Add RCA4_TYPE_MASK_DIR_DLG.

A client that implements directory delegations will start returning
them on a callback that never named them before. NFSD takes that
return on the path it already takes for any other: nfsd4_delegreturn()
resolves the stateid as SC_TYPE_DELEG and calls destroy_delegation()
with no branch on what the delegation covers.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfsd/nfs4state.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 2ddc77ac7312..3017a93261ff 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -7969,7 +7969,8 @@ deleg_reaper(struct nfsd_net *nn)
 		clp->cl_ra_time = ktime_get_boottime_seconds();
 		clp->cl_ra->ra_keep = 0;
 		clp->cl_ra->ra_bmval[0] = BIT(RCA4_TYPE_MASK_RDATA_DLG) |
-						BIT(RCA4_TYPE_MASK_WDATA_DLG);
+						BIT(RCA4_TYPE_MASK_WDATA_DLG) |
+						BIT(RCA4_TYPE_MASK_DIR_DLG);
 		trace_nfsd_cb_recall_any(clp->cl_ra);
 		nfsd4_run_cb(&clp->cl_ra->ra_cb);
 	}

-- 
2.54.0

[PATCH 4/4] NFSD: Send a meaningful CB_RECALL_ANY keep count

[Chuck Lever]

deleg_reaper() sets craa_objects_to_keep to zero on every
CB_RECALL_ANY. Per RFC 8881 Section 20.6.3, that asks the client to
retain no read or write delegation at all, whether or not the
delegation backs an open file.

The field names a count the client may keep. The client picks which
objects to return, because the server cannot read lack of recent
use as lack of usefulness. Zero leaves nothing to choose among. A
client that complies returns the delegations backing its open files
and reopens each one with CLAIM_DELEGATE_CUR, so NFSD trades a
delegation for an open stateid and recovers nothing. A client that
reads the zero as "unspecified" does nothing instead, and
nfsd4_cb_recall_any_done() inspects only the reply status, so NFSD
cannot tell the two apart.

Derive the keep count from cl_deleg_count and ask each client for a
single delegation. A larger request reaches delegations an
application still has open, and both callers re-arm while their
condition lasts. Skip a client holding one delegation rather than
send the zero again. That gate subsumes the list_empty() test above
it, and it belongs above the NFSD4_CALLBACK_RUNNING test_and_set. A
continue below that point latches the bit with no callback in
flight to clear it.

The Linux client ignores craa_objs_to_keep and returns unused
delegations from the type mask alone, so the count changes nothing
for it. The gate does. The reaper goes quiet for a client walked
down to one delegation.

Fixes: 44df6f439a17 ("NFSD: add delegation reaper to react to low memory condition")
Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfsd/nfs4state.c | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 3017a93261ff..e825ecb2ddfc 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -7944,6 +7944,7 @@ deleg_reaper(struct nfsd_net *nn)
 {
 	struct list_head *pos, *next;
 	struct nfs4_client *clp;
+	unsigned int count;
 
 	spin_lock(&nn->client_lock);
 	list_for_each_safe(pos, next, &nn->client_lru) {
@@ -7953,21 +7954,34 @@ deleg_reaper(struct nfsd_net *nn)
 			continue;
 		if (clp->cl_state != NFSD4_ACTIVE)
 			continue;
-		if (list_empty(&clp->cl_delegations))
-			continue;
 		if (atomic_read(&clp->cl_delegs_in_recall))
 			continue;
 		if (ktime_get_boottime_seconds() - clp->cl_ra_time < 5)
 			continue;
 		if (clp->cl_cb_state != NFSD4_CB_UP)
 			continue;
+		/*
+		 * This read races with hash_delegation_locked() and
+		 * unhash_delegation_locked() on other CPUs. A stale
+		 * value only defers this client to the next
+		 * laundromat pass.
+		 */
+		count = data_race(READ_ONCE(clp->cl_deleg_count));
+		if (count < 2)
+			continue;
 		if (test_and_set_bit(NFSD4_CALLBACK_RUNNING, &clp->cl_ra->ra_cb.cb_flags))
 			continue;
 
 		/* release in nfsd4_cb_recall_any_release */
 		kref_get(&clp->cl_nfsdfs.cl_ref);
 		clp->cl_ra_time = ktime_get_boottime_seconds();
-		clp->cl_ra->ra_keep = 0;
+		/*
+		 * Ask for one delegation at a time. A larger request
+		 * reaches delegations backing files that applications
+		 * still have open. Returning one of those trades a DELEG
+		 * stateid for an OPEN stateid and frees nothing.
+		 */
+		clp->cl_ra->ra_keep = count - 1;
 		clp->cl_ra->ra_bmval[0] = BIT(RCA4_TYPE_MASK_RDATA_DLG) |
 						BIT(RCA4_TYPE_MASK_WDATA_DLG) |
 						BIT(RCA4_TYPE_MASK_DIR_DLG);

-- 
2.54.0

Re: [PATCH 4/4] NFSD: Send a meaningful CB_RECALL_ANY keep count

[Cedric Blancher]

On Tue, 11 Aug 2026 at 21:57, Chuck Lever <cel@kernel.org> wrote:
>
> deleg_reaper() sets craa_objects_to_keep to zero on every
> CB_RECALL_ANY. Per RFC 8881 Section 20.6.3, that asks the client to
> retain no read or write delegation at all, whether or not the
> delegation backs an open file.
>
> The field names a count the client may keep. The client picks which
> objects to return, because the server cannot read lack of recent
> use as lack of usefulness. Zero leaves nothing to choose among. A
> client that complies returns the delegations backing its open files
> and reopens each one with CLAIM_DELEGATE_CUR, so NFSD trades a
> delegation for an open stateid and recovers nothing. A client that
> reads the zero as "unspecified" does nothing instead, and
> nfsd4_cb_recall_any_done() inspects only the reply status, so NFSD
> cannot tell the two apart.
>
> Derive the keep count from cl_deleg_count and ask each client for a
> single delegation. A larger request reaches delegations an
> application still has open, and both callers re-arm while their
> condition lasts. Skip a client holding one delegation rather than
> send the zero again. That gate subsumes the list_empty() test above
> it, and it belongs above the NFSD4_CALLBACK_RUNNING test_and_set. A
> continue below that point latches the bit with no callback in
> flight to clear it.
>
> The Linux client ignores craa_objs_to_keep and returns unused
> delegations from the type mask alone, so the count changes nothing
> for it. The gate does. The reaper goes quiet for a client walked
> down to one delegation.
>
> Fixes: 44df6f439a17 ("NFSD: add delegation reaper to react to low memory condition")
> Signed-off-by: Chuck Lever <cel@kernel.org>

ms-nfs41-client hit that bug when implementing CB_RECALL_ANY
(https://github.com/kofemann/ms-nfs41-client/commit/3abe73c8ab0d924450a1c1c480f0b14f963f9531)
with Linux 7.0 nfsd.
What should existing NFSv4.1 clients do if they encounter a
objects_to_keep value of 0? Right now it recalls ALL delegations,
which basically is a "reset" of all delegations.

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: [PATCH 4/4] NFSD: Send a meaningful CB_RECALL_ANY keep count

[Jeff Layton]

On Wed, 2026-08-12 at 08:17 +0200, Cedric Blancher wrote:
> On Tue, 11 Aug 2026 at 21:57, Chuck Lever <cel@kernel.org> wrote:
> > 
> > deleg_reaper() sets craa_objects_to_keep to zero on every
> > CB_RECALL_ANY. Per RFC 8881 Section 20.6.3, that asks the client to
> > retain no read or write delegation at all, whether or not the
> > delegation backs an open file.
> > 
> > The field names a count the client may keep. The client picks which
> > objects to return, because the server cannot read lack of recent
> > use as lack of usefulness. Zero leaves nothing to choose among. A
> > client that complies returns the delegations backing its open files
> > and reopens each one with CLAIM_DELEGATE_CUR, so NFSD trades a
> > delegation for an open stateid and recovers nothing. A client that
> > reads the zero as "unspecified" does nothing instead, and
> > nfsd4_cb_recall_any_done() inspects only the reply status, so NFSD
> > cannot tell the two apart.
> > 
> > Derive the keep count from cl_deleg_count and ask each client for a
> > single delegation. A larger request reaches delegations an
> > application still has open, and both callers re-arm while their
> > condition lasts. Skip a client holding one delegation rather than
> > send the zero again. That gate subsumes the list_empty() test above
> > it, and it belongs above the NFSD4_CALLBACK_RUNNING test_and_set. A
> > continue below that point latches the bit with no callback in
> > flight to clear it.
> > 
> > The Linux client ignores craa_objs_to_keep and returns unused
> > delegations from the type mask alone, so the count changes nothing
> > for it. The gate does. The reaper goes quiet for a client walked
> > down to one delegation.
> > 
> > Fixes: 44df6f439a17 ("NFSD: add delegation reaper to react to low memory condition")
> > Signed-off-by: Chuck Lever <cel@kernel.org>
> 
> ms-nfs41-client hit that bug when implementing CB_RECALL_ANY
> (https://github.com/kofemann/ms-nfs41-client/commit/3abe73c8ab0d924450a1c1c480f0b14f963f9531)
> with Linux 7.0 nfsd.
> What should existing NFSv4.1 clients do if they encounter a
> objects_to_keep value of 0? Right now it recalls ALL delegations,
> which basically is a "reset" of all delegations.
> 
> 

Like Chuck quotes above:

"Per RFC 8881 Section 20.6.3, that asks the client to retain no read or
write delegation at all, whether or not the delegation backs an open
file."

I think a compliant client would need to continue to return everything.

We should note that the Linux client apparently ignores
craa_objects_to_keep and just returns a single delegation, so you could
follow suit (but know that it's not following the spec).
-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH 4/4] NFSD: Send a meaningful CB_RECALL_ANY keep count

[Chuck Lever]

On Wed, Aug 12, 2026, at 2:17 AM, Cedric Blancher wrote:
> On Tue, 11 Aug 2026 at 21:57, Chuck Lever <cel@kernel.org> wrote:
>>
>> deleg_reaper() sets craa_objects_to_keep to zero on every
>> CB_RECALL_ANY. Per RFC 8881 Section 20.6.3, that asks the client to
>> retain no read or write delegation at all, whether or not the
>> delegation backs an open file.
>>
>> The field names a count the client may keep. The client picks which
>> objects to return, because the server cannot read lack of recent
>> use as lack of usefulness. Zero leaves nothing to choose among. A
>> client that complies returns the delegations backing its open files
>> and reopens each one with CLAIM_DELEGATE_CUR, so NFSD trades a
>> delegation for an open stateid and recovers nothing. A client that
>> reads the zero as "unspecified" does nothing instead, and
>> nfsd4_cb_recall_any_done() inspects only the reply status, so NFSD
>> cannot tell the two apart.
>>
>> Derive the keep count from cl_deleg_count and ask each client for a
>> single delegation. A larger request reaches delegations an
>> application still has open, and both callers re-arm while their
>> condition lasts. Skip a client holding one delegation rather than
>> send the zero again. That gate subsumes the list_empty() test above
>> it, and it belongs above the NFSD4_CALLBACK_RUNNING test_and_set. A
>> continue below that point latches the bit with no callback in
>> flight to clear it.
>>
>> The Linux client ignores craa_objs_to_keep and returns unused
>> delegations from the type mask alone, so the count changes nothing
>> for it. The gate does. The reaper goes quiet for a client walked
>> down to one delegation.
>>
>> Fixes: 44df6f439a17 ("NFSD: add delegation reaper to react to low memory condition")
>> Signed-off-by: Chuck Lever <cel@kernel.org>
>
> ms-nfs41-client hit that bug when implementing CB_RECALL_ANY
> (https://github.com/kofemann/ms-nfs41-client/commit/3abe73c8ab0d924450a1c1c480f0b14f963f9531)
> with Linux 7.0 nfsd.
> What should existing NFSv4.1 clients do if they encounter a
> objects_to_keep value of 0? Right now it recalls ALL delegations,
> which basically is a "reset" of all delegations.

After studying this issue for a few days... and sleeping on it a
bit...

RFC 8881 Section 20.6.3 does not normatively mandate any particular
client response to CB_RECALL_ANY other than returning NFS4ERR_INVAL
when the craa_type_mask bitmask is invalid. The client-facing verbs
in that section are "is to return", "chooses", and "it is the job
of". All descriptive language, no BCP14 keywords.

Thus, according to spec, there are no interoperability consequences
if a client ignores the value of the craa_objects_to_keep argument.
The spec gives client implementers considerable flexibility here.

A server has no visibility of which delegations are actively in use
on clients, since the point of delegation is to reduce client-to-
server traffic. That's why the client gets to choose which to
return.

A good quality client implementation, IMHO, should choose idle or
currently unused delegations, but protect state that is still in
active use. To return a DELEG stateid that is still in use, the
client would need to first ensure it has an OPEN stateid to
continue using. That would result in no real change in the server
memory footprint for that file, so there is no benefit.

AFAICS a craa_objects_to_keep value of zero is not a bug; it is
in fact a valid value for that argument. But NFSD shouldn't ask
clients to toss out their entire working set at the first sign of
memory pressure. The new behavior is a little more graceful.

(So I realize the patches don't actually say this much, but my
thinking is still evolving).

-- 
Chuck Lever

Re: [PATCH 0/4] NFSD: CB_RECALL_ANY fixes and a meaningful keep count

[Jeff Layton]

On Tue, 2026-08-11 at 15:52 -0400, Chuck Lever wrote:
> NFSD sends every CB_RECALL_ANY with craa_objects_to_keep set to
> zero. RFC 8881 Section 20.6.3 reads that as a request to retain no
> delegation at all, whether or not the delegation backs a file an
> application still has open.
> 
> NFSD meant the callback as a signal to return unused delegations.
> Only the Linux client has been tested against it, and that client
> ignores craa_objects_to_keep and returns every unused delegation of
> the named types, so the fixed zero never produced visible
> misbehavior.
> 
> The NFSv4.0 recipient bug is fixed on its own so it can be
> backported to LTS ahead of the rest. An NFSv4.0 callback service
> rejects CB_RECALL_ANY, and nothing brings the callback channel back
> up afterward (patch 1).
> 
> The keep count comes from a new per-client delegation count rather
> than a walk of cl_delegations (patch 2). deleg_reaper() runs under
> nn->client_lock, and cl_delegations is serialized by nn->deleg_lock,
> which nests outside it, so the reaper cannot count the list. It
> reads the counter without the lock and may not depend on the value.
> 
> Each callback asks a client to give up one delegation. A larger
> request reaches delegations that applications still have open, and
> both reaper callers re-arm while their condition lasts. A client
> holding a single delegation is skipped rather than sent the old
> zero. That strands one delegation per client, which this
> best-effort design accepts for now.
> 
> NFSD sets no recall target and remembers nothing across callbacks.
> RFC 8881 Section 20.6.4 prescribes CB_RECALL against specific
> delegations once a client fails to return any. NFSD does not take
> that step yet. CB_RECALL_ANY is asynchronous and reports no
> completion, so NFSD treats it as advisory.
> 
> ---
> Chuck Lever (4):
>       NFSD: Do not send CB_RECALL_ANY to NFSv4.0 clients
>       NFSD: Count the delegations held by each client
>       NFSD: Name directory delegations in the CB_RECALL_ANY type mask
>       NFSD: Send a meaningful CB_RECALL_ANY keep count
> 
>  fs/nfsd/nfs4state.c | 27 +++++++++++++++++++++++----
>  fs/nfsd/state.h     |  2 ++
>  2 files changed, 25 insertions(+), 4 deletions(-)
> ---
> base-commit: 1d479c6b53f684b27da84ec352b7efb97f7f115f
> change-id: 20260810-recall-any-keep-count-f50c2ae1b792
> 
> Best regards,
> --  
> Chuck Lever <cel@kernel.org>

Looks reasonable.

Reviewed-by: Jeff Layton <jlayton@kernel.org>