[PATCH v2] nfs/localio: pin clients during global invalidation

[PATCH v2] nfs/localio: pin clients during global invalidation

[Jinpyo Lee]

nfs_localio_invalidate_clients() moves UUID nodes embedded in nfs_client
objects to a private list, drops the namespace list lock, and walks the
private list without holding references on the clients that own the nodes.

Moving an embedded list node does not retain its containing object.
Concurrent client teardown can therefore drop the final cl_count reference
after the splice. List iteration can preload the next UUID node before
processing the current entry, allowing that node's nfs_client to be freed
before the iterator advances to it. Generic KASAN reported an eight-byte
use-after-free read from a freed nfs_client allocation in
nfs_localio_invalidate_clients().

Process one UUID at a time under the namespace list lock. Acquire a
cl_count reference before releasing the lock, disable LOCALIO, and then
drop the reference with nfs_put_client(). If the reference cannot be
acquired, yield and retry so final teardown can remove the dying client
from the list.

Expose the existing nfs_put_client() declaration in a public NFS header and
remove its duplicate declaration from the private NFS header.

The reproducer creates 16 clients, populates their LOCALIO state, and races
administrator-driven global file-cache invalidation with normal client
teardown. It demonstrates the lifetime error, but does not establish
controlled reuse, privilege escalation, or an unprivileged end-to-end
trigger.

On the current nfsd-testing head, the unpatched KASAN kernel reproduced
the use-after-free in nfs_localio_invalidate_clients(). With only this
patch applied, the same race completed without a KASAN report. A separate
run that kept all 16 clients live recorded one LOCALIO disable event for
each client.

Basic NFSv4.2 and NFSv3 read, write, and unmount smoke tests also passed. A
full x86_64 kernel and modules build with GCC 13.3 and CONFIG_WERROR=y
completed without warnings. A source reproducer and complete logs are
available privately on request.

The vulnerability research and validation were conducted by members of the
Tobabz team as part of the Best of the Best 15th program.

Fixes: 085804110aa1 ("nfs_common: track all open nfsd_files per LOCALIO nfs_client")
Assisted-by: LLM
Signed-off-by: Jinpyo Lee <bint4b13@gmail.com>
---
Changes in v2:

- Move the nfs_put_client() declaration to a public NFS header instead of
  duplicating it.
- Make the commit message self-contained and document current-tree runtime
  validation.
- Correct the author identity and DCO sign-off.

 fs/nfs/internal.h          |  1 -
 fs/nfs_common/nfslocalio.c | 27 ++++++++++++++++++++-------
 include/linux/nfs_fs.h     |  2 ++
 3 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index abc81f5ae5780..c377075a8b5b5 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -223,7 +223,6 @@ int nfs_init_server_rpcclient(struct nfs_server *, const struct rpc_timeout *t,
 struct nfs_server *nfs_alloc_server(void);
 void nfs_server_copy_userdata(struct nfs_server *, struct nfs_server *);
 
-extern void nfs_put_client(struct nfs_client *);
 extern void nfs_free_client(struct nfs_client *);
 void nfs_cb_idr_remove(struct nfs_client *clp);
 extern struct nfs_client *nfs4_find_client_ident(struct net *, int);
diff --git a/fs/nfs_common/nfslocalio.c b/fs/nfs_common/nfslocalio.c
index 85aa03a7b020c..00fcba467ee8d 100644
--- a/fs/nfs_common/nfslocalio.c
+++ b/fs/nfs_common/nfslocalio.c
@@ -226,18 +226,31 @@ EXPORT_SYMBOL_GPL(nfs_localio_disable_client);
 void nfs_localio_invalidate_clients(struct list_head *nn_local_clients,
 				    spinlock_t *nn_local_clients_lock)
 {
-	LIST_HEAD(local_clients);
-	nfs_uuid_t *nfs_uuid, *tmp;
+	nfs_uuid_t *nfs_uuid;
 	struct nfs_client *clp;
 
-	spin_lock(nn_local_clients_lock);
-	list_splice_init(nn_local_clients, &local_clients);
-	spin_unlock(nn_local_clients_lock);
-	list_for_each_entry_safe(nfs_uuid, tmp, &local_clients, list) {
-		if (WARN_ON(nfs_uuid->list_lock != nn_local_clients_lock))
+	for (;;) {
+		spin_lock(nn_local_clients_lock);
+		nfs_uuid = list_first_entry_or_null(nn_local_clients,
+						    nfs_uuid_t, list);
+		if (!nfs_uuid) {
+			spin_unlock(nn_local_clients_lock);
+			break;
+		}
+		if (WARN_ON(nfs_uuid->list_lock != nn_local_clients_lock)) {
+			spin_unlock(nn_local_clients_lock);
 			break;
+		}
 		clp = container_of(nfs_uuid, struct nfs_client, cl_uuid);
+		if (!refcount_inc_not_zero(&clp->cl_count)) {
+			spin_unlock(nn_local_clients_lock);
+			cond_resched();
+			continue;
+		}
+		spin_unlock(nn_local_clients_lock);
+
 		nfs_localio_disable_client(clp);
+		nfs_put_client(clp);
 	}
 }
 EXPORT_SYMBOL_GPL(nfs_localio_invalidate_clients);
diff --git a/include/linux/nfs_fs.h b/include/linux/nfs_fs.h
index b85a73ae7919f..8c1e2d2027c62 100644
--- a/include/linux/nfs_fs.h
+++ b/include/linux/nfs_fs.h
@@ -84,6 +84,8 @@ struct nfs_file_localio {
 	void __rcu *nfs_uuid; /* opaque pointer to 'nfs_uuid_t' */
 };
 
+void nfs_put_client(struct nfs_client *clp);
+
 static inline void nfs_localio_file_init(struct nfs_file_localio *nfl)
 {
 #if IS_ENABLED(CONFIG_NFS_LOCALIO)
-- 
2.43.0

Re: [PATCH v2] nfs/localio: pin clients during global invalidation

[Chuck Lever]

On Wed, Sep 30, 2026 at 02:50:53PM +0900, Jinpyo Lee wrote:
> Process one UUID at a time under the namespace list lock. Acquire a
> cl_count reference before releasing the lock, disable LOCALIO, and then
> drop the reference with nfs_put_client(). If the reference cannot be
> acquired, yield and retry so final teardown can remove the dying client
> from the list.

The reported use-after-free is confirmed.

> A full x86_64 kernel and modules build with GCC 13.3 and
> CONFIG_WERROR=y completed without warnings.

How was NFS_FS set for that build?

> +		spin_unlock(nn_local_clients_lock);
> +
>  		nfs_localio_disable_client(clp);
> +		nfs_put_client(clp);

nfs_put_client() is exported from nfs.ko. This call makes
nfs_localio depend on nfs.ko, and nfs.ko already depends on
nfs_localio for nfs_uuid_init(), nfs_uuid_begin(), nfs_uuid_end(),
and nfs_open_local_fh().

With NFS_FS=m and NFSD=m, NFS_COMMON_LOCALIO_SUPPORT is also "m", so
the two modules now require each other and neither can be loaded
first. With NFS_FS=m and NFSD=y, nfs_localio is built in and
references a symbol that lives in a module, so vmlinux cannot link.
Only NFS_FS=y avoids both.

nfs_common cannot call into nfs.ko. The pin has to be something
nfs_common owns, or the put has to go through a callback that the
NFS client provides.

Another issue: If client teardown drops its reference while this
loop holds the temporary one, the put above is the final one, and
nfs_free_client() or nfs4_free_client() then runs in the caller's
context. The callers here are the export flush path, NFSD file
cache shutdown, and nfsd_net_pre_exit(). For NFSv4 that means
nfs4_shutdown_client() and the callback teardown run while NFSD
is flushing or shutting down.

A pin that does not hold cl_count would avoid this as well as the
module dependency.


> +	for (;;) {
> +		spin_lock(nn_local_clients_lock);
> +		nfs_uuid = list_first_entry_or_null(nn_local_clients,
> +						    nfs_uuid_t, list);

This assumes nfs_localio_disable_client() always removes the head
entry from the list. It does not when another thread is already in
nfs_uuid_put() for that client.

nfs_uuid_put() clears ->net first, then drops nfs_uuid->lock around
each nfs_to_nfsd_file_put_local() call, and unlinks the uuid from
nn->local_clients only after the last file is closed. While that
thread is closing files, this loop finds the same uuid at the head
of the list, takes a reference, gets "false" back from
nfs_uuid_put() because ->net is already NULL, drops the reference,
and goes around again. That path has no cond_resched(), so it spins
until the other thread finishes. The other thread can sleep in
nfs_to_nfsd_file_put_local() or in wait_var_event_spinlock().

The splice in the old code moved such an entry off the list, so
the walk visited it once and went on. The new loop needs a way to
get past an entry that someone else is already disabling.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)