[PATCH v6 0/6] nfsd/sunrpc: convert nfsstat server-side interfaces to use netlink

[PATCH v6 0/6] nfsd/sunrpc: convert nfsstat server-side interfaces to use netlink

[Jeff Layton]

This version fixes a few problems that Chuck and Sashiko pointed
out in review.

The nfsstat tool currently scrapes /proc/net/rpc/nfsd for server
statistics. This procfs interface has several limitations: the
counters are global (not network-namespace-aware), the format is
fragile to parse, and it cannot be extended without breaking
existing parsers.

This series adds per-network-namespace procedure call counts to
the sunrpc layer and exposes them through a new netlink handler
in the nfsd generic netlink family, allowing nfsstat to retrieve
server statistics via netlink with a procfs fallback for older
kernels. The nfs-utils patch for that part is already merged.

Additionally, this version adds tracking of callback operation counts.

Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
Changes in v6:
- Emit the entire server-stats object in a single netlink dump message
- Count CB_SEQUENCE on v4.1+ callbacks, for parity with the forechannel SEQUENCE count
- Fix a use-after-free in nfsd4_run_cb() by snapshotting the opcode, net, and minorversion
- Link to v5: https://lore.kernel.org/r/20260618-exportd-netlink-v5-0-e9aef947af3d@kernel.org

Changes in v5:
- Add NFSv4 callback call counts to nfsstat upcall
- Link to v4: https://lore.kernel.org/r/20260616-exportd-netlink-v4-0-03505aee3883@kernel.org

Changes in v4:
- Ensure error is returned when nla_put_*() calls fail
- Link to v3: https://lore.kernel.org/r/20260609-exportd-netlink-v3-0-aa5508a5bb1d@kernel.org

Changes in v3:
- Only increment per-net stats for the "primary" program
- Link to v2: https://lore.kernel.org/r/20260525-exportd-netlink-v2-0-40003fed450c@kernel.org

---
Jeff Layton (6):
      sunrpc: add per-netns per-procedure call counts to svc_stat
      sunrpc: use per-net counts in svc_seq_show()
      nfsd: implement server-stats-get netlink handler
      sunrpc: remove unused svc_version vs_count field
      nfsd: count NFSv4 callback operations per netns
      nfsd: export NFSv4 callback op stats via netlink

 Documentation/netlink/specs/nfsd.yaml | 111 +++++++++++++++++
 fs/lockd/svc4proc.c                   |   4 -
 fs/lockd/svcproc.c                    |   7 --
 fs/nfs/callback_xdr.c                 |   6 -
 fs/nfsd/localio.c                     |   3 -
 fs/nfsd/netlink.c                     |   5 +
 fs/nfsd/netlink.h                     |   2 +
 fs/nfsd/netns.h                       |  12 +-
 fs/nfsd/nfs2acl.c                     |   3 -
 fs/nfsd/nfs3acl.c                     |   3 -
 fs/nfsd/nfs3proc.c                    |   3 -
 fs/nfsd/nfs4callback.c                |  22 +++-
 fs/nfsd/nfs4proc.c                    |   3 -
 fs/nfsd/nfs4state.c                   |   2 -
 fs/nfsd/nfsctl.c                      | 222 +++++++++++++++++++++++++++++++++-
 fs/nfsd/nfsproc.c                     |   3 -
 fs/nfsd/stats.c                       |   2 +-
 fs/nfsd/stats.h                       |   5 +-
 include/linux/sunrpc/stats.h          |   6 +
 include/linux/sunrpc/svc.h            |   1 -
 include/uapi/linux/nfsd_netlink.h     |  36 ++++++
 net/sunrpc/stats.c                    |   2 +-
 net/sunrpc/svc.c                      |  63 +++++++++-
 23 files changed, 479 insertions(+), 47 deletions(-)
---
base-commit: 6f90c7618528b5ca5887f8c6057f26dcc7a27a99
change-id: 20260316-exportd-netlink-1c9fb52536e3

Best regards,
-- 
Jeff Layton <jlayton@kernel.org>

[PATCH v6 1/6] sunrpc: add per-netns per-procedure call counts to svc_stat

[Jeff Layton]

The existing per-procedure call counts live in global
svc_version->vs_count[] arrays which are not network-namespace-aware.
Add per-netns equivalents in struct svc_stat so the upcoming netlink
stats interface can return namespace-scoped statistics.

Add a vs_count pointer array to struct svc_stat, along with
svc_stat_alloc_counts() and svc_stat_free_counts() helpers to manage
per-version percpu call count arrays.

Increment the per-net counter alongside the global one in
svc_generic_init_request(). Call the alloc/free helpers from
nfsd_net_init() and nfsd_net_exit().

Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
 fs/nfsd/nfsctl.c             |  8 +++++-
 include/linux/sunrpc/stats.h |  6 +++++
 net/sunrpc/svc.c             | 62 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+), 1 deletion(-)

diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index caf59421f8f4..601301e34fc7 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -2514,9 +2514,12 @@ static __net_init int nfsd_net_init(struct net *net)
 
 	memset(&nn->nfsd_svcstats, 0, sizeof(nn->nfsd_svcstats));
 	nn->nfsd_svcstats.program = &nfsd_programs[0];
+	retval = svc_stat_alloc_counts(&nn->nfsd_svcstats);
+	if (retval)
+		goto out_proc_error;
 	if (!nfsd_proc_stat_init(net)) {
 		retval = -ENOMEM;
-		goto out_proc_error;
+		goto out_svcstats_error;
 	}
 
 	for (i = 0; i < sizeof(nn->nfsd_versions); i++)
@@ -2534,6 +2537,8 @@ static __net_init int nfsd_net_init(struct net *net)
 #endif
 	return 0;
 
+out_svcstats_error:
+	svc_stat_free_counts(&nn->nfsd_svcstats);
 out_proc_error:
 	percpu_counter_destroy_many(nn->counter, NFSD_STATS_COUNTERS_NUM);
 out_repcache_error:
@@ -2574,6 +2579,7 @@ static __net_exit void nfsd_net_exit(struct net *net)
 	kfree_sensitive(nn->fh_key);
 	nfsd_net_cb_shutdown(nn);
 	nfsd_proc_stat_shutdown(net);
+	svc_stat_free_counts(&nn->nfsd_svcstats);
 	percpu_counter_destroy_many(nn->counter, NFSD_STATS_COUNTERS_NUM);
 	nfsd_idmap_shutdown(net);
 	nfsd_export_shutdown(net);
diff --git a/include/linux/sunrpc/stats.h b/include/linux/sunrpc/stats.h
index 3ce1550d1beb..087ade905e29 100644
--- a/include/linux/sunrpc/stats.h
+++ b/include/linux/sunrpc/stats.h
@@ -37,9 +37,15 @@ struct svc_stat {
 				rpcbadfmt,
 				rpcbadauth,
 				rpcbadclnt;
+
+	/* Per-version per-procedure call counts (per-cpu, per-netns) */
+	unsigned long __percpu	**vs_count;
 };
 
 struct net;
+int			svc_stat_alloc_counts(struct svc_stat *statp);
+void			svc_stat_free_counts(struct svc_stat *statp);
+
 #ifdef CONFIG_PROC_FS
 int			rpc_proc_init(struct net *);
 void			rpc_proc_exit(struct net *);
diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c
index dd80a2eaaa74..0cee079df1cb 100644
--- a/net/sunrpc/svc.c
+++ b/net/sunrpc/svc.c
@@ -1438,6 +1438,14 @@ svc_generic_init_request(struct svc_rqst *rqstp,
 	/* Bump per-procedure stats counter */
 	this_cpu_inc(versp->vs_count[rqstp->rq_proc]);
 
+	/* Bump per-net per-procedure stats counter */
+	if (rqstp->rq_server->sv_stats &&
+	    rqstp->rq_server->sv_stats->program == progp &&
+	    rqstp->rq_server->sv_stats->vs_count &&
+	    rqstp->rq_server->sv_stats->vs_count[rqstp->rq_vers])
+		this_cpu_inc(rqstp->rq_server->sv_stats->vs_count
+				[rqstp->rq_vers][rqstp->rq_proc]);
+
 	ret->dispatch = versp->vs_dispatch;
 	return rpc_success;
 err_bad_vers:
@@ -1449,6 +1457,60 @@ svc_generic_init_request(struct svc_rqst *rqstp,
 }
 EXPORT_SYMBOL_GPL(svc_generic_init_request);
 
+/**
+ * svc_stat_alloc_counts - allocate per-netns per-version call count arrays
+ * @statp: svc_stat whose vs_count arrays should be allocated
+ *
+ * statp->program must be set before calling this.
+ *
+ * Returns zero on success, or a negative errno otherwise.
+ */
+int svc_stat_alloc_counts(struct svc_stat *statp)
+{
+	struct svc_program *prog = statp->program;
+	unsigned int i;
+
+	statp->vs_count = kcalloc(prog->pg_nvers,
+				  sizeof(unsigned long __percpu *),
+				  GFP_KERNEL);
+	if (!statp->vs_count)
+		return -ENOMEM;
+
+	for (i = 0; i < prog->pg_nvers; i++) {
+		if (!prog->pg_vers[i])
+			continue;
+		statp->vs_count[i] = __alloc_percpu(prog->pg_vers[i]->vs_nproc *
+					    sizeof(unsigned long),
+					    sizeof(unsigned long));
+		if (!statp->vs_count[i])
+			goto err;
+	}
+	return 0;
+err:
+	svc_stat_free_counts(statp);
+	return -ENOMEM;
+}
+EXPORT_SYMBOL_GPL(svc_stat_alloc_counts);
+
+/**
+ * svc_stat_free_counts - free per-netns per-version call count arrays
+ * @statp: svc_stat whose vs_count arrays should be freed
+ */
+void svc_stat_free_counts(struct svc_stat *statp)
+{
+	struct svc_program *prog = statp->program;
+	unsigned int i;
+
+	if (!statp->vs_count)
+		return;
+
+	for (i = 0; i < prog->pg_nvers; i++)
+		free_percpu(statp->vs_count[i]);
+	kfree(statp->vs_count);
+	statp->vs_count = NULL;
+}
+EXPORT_SYMBOL_GPL(svc_stat_free_counts);
+
 /*
  * Common routine for processing the RPC request.
  */

-- 
2.54.0

[PATCH v6 2/6] sunrpc: use per-net counts in svc_seq_show()

[Jeff Layton]

Update svc_seq_show() to read from the per-netns
statp->vs_count[] arrays instead of the global
svc_version->vs_count[].

The only caller is nfsd, which always allocates vs_count via
svc_stat_alloc_counts() in nfsd_net_init(), so the per-netns
arrays are always available.

This makes /proc/net/rpc/nfsd report per-network-namespace
procedure call counts.

Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
 net/sunrpc/stats.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/sunrpc/stats.c b/net/sunrpc/stats.c
index 7093e18ac26c..d08711bee18e 100644
--- a/net/sunrpc/stats.c
+++ b/net/sunrpc/stats.c
@@ -108,7 +108,7 @@ void svc_seq_show(struct seq_file *seq, const struct svc_stat *statp)
 		for (j = 0; j < vers->vs_nproc; j++) {
 			count = 0;
 			for_each_possible_cpu(k)
-				count += per_cpu(vers->vs_count[j], k);
+				count += per_cpu(statp->vs_count[i][j], k);
 			seq_printf(seq, " %lu", count);
 		}
 		seq_putc(seq, '\n');

-- 
2.54.0

[PATCH v6 3/6] nfsd: implement server-stats-get netlink handler

[Jeff Layton]

Implement nfsd_nl_server_stats_get_dumpit() which exposes the
NFS server statistics currently available via /proc/net/rpc/nfsd
through the nfsd generic netlink family.

The handler uses a dump operation to stream statistics across
multiple netlink messages:

  - First message: all scalar stats (reply cache, filehandle,
    IO, network, RPC) plus per-version procedure counts
    (proc2/3/4-ops) using per-netns vs_count arrays.

  - Subsequent messages: NFSv4 per-operation counts
    (proc4ops-ops), one entry per message, using cb->args[0]
    to track the current operation index across dump calls.

This allows nfsstat to retrieve server statistics via netlink
with a procfs fallback for older kernels.

Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
 Documentation/netlink/specs/nfsd.yaml | 105 ++++++++++++++++++++
 fs/nfsd/netlink.c                     |   5 +
 fs/nfsd/netlink.h                     |   2 +
 fs/nfsd/nfsctl.c                      | 179 ++++++++++++++++++++++++++++++++++
 include/uapi/linux/nfsd_netlink.h     |  35 +++++++
 5 files changed, 326 insertions(+)

diff --git a/Documentation/netlink/specs/nfsd.yaml b/Documentation/netlink/specs/nfsd.yaml
index 8f36fadd68f7..2a89d355ee7b 100644
--- a/Documentation/netlink/specs/nfsd.yaml
+++ b/Documentation/netlink/specs/nfsd.yaml
@@ -330,6 +330,86 @@ attribute-sets:
           of which client holds the state. Intended for use after
           all clients have been unexported from a given path,
           enabling the underlying filesystem to be unmounted.
+  -
+    name: server-proc-entry
+    attributes:
+      -
+        name: op
+        type: u32
+      -
+        name: count
+        type: u64
+      -
+        name: pad
+        type: pad
+  -
+    name: server-stats
+    attributes:
+      -
+        name: rc-hits
+        type: u64
+      -
+        name: rc-misses
+        type: u64
+      -
+        name: rc-nocache
+        type: u64
+      -
+        name: pad
+        type: pad
+      -
+        name: fh-stale
+        type: u64
+      -
+        name: io-read
+        type: u64
+      -
+        name: io-write
+        type: u64
+      -
+        name: netcnt
+        type: u32
+      -
+        name: netudpcnt
+        type: u32
+      -
+        name: nettcpcnt
+        type: u32
+      -
+        name: nettcpconn
+        type: u32
+      -
+        name: rpccnt
+        type: u32
+      -
+        name: rpcbadfmt
+        type: u32
+      -
+        name: rpcbadauth
+        type: u32
+      -
+        name: rpcbadclnt
+        type: u32
+      -
+        name: proc2-ops
+        type: nest
+        nested-attributes: server-proc-entry
+        multi-attr: true
+      -
+        name: proc3-ops
+        type: nest
+        nested-attributes: server-proc-entry
+        multi-attr: true
+      -
+        name: proc4-ops
+        type: nest
+        nested-attributes: server-proc-entry
+        multi-attr: true
+      -
+        name: proc4ops-ops
+        type: nest
+        nested-attributes: server-proc-entry
+        multi-attr: true
 
 operations:
   list:
@@ -516,6 +596,31 @@ operations:
         request:
           attributes:
             - path
+    -
+      name: server-stats-get
+      doc: dump NFS server statistics
+      attribute-set: server-stats
+      dump:
+        reply:
+          attributes:
+            - rc-hits
+            - rc-misses
+            - rc-nocache
+            - fh-stale
+            - io-read
+            - io-write
+            - netcnt
+            - netudpcnt
+            - nettcpcnt
+            - nettcpconn
+            - rpccnt
+            - rpcbadfmt
+            - rpcbadauth
+            - rpcbadclnt
+            - proc2-ops
+            - proc3-ops
+            - proc4-ops
+            - proc4ops-ops
 
 mcast-groups:
   list:
diff --git a/fs/nfsd/netlink.c b/fs/nfsd/netlink.c
index fbee3676d253..eba8b353f412 100644
--- a/fs/nfsd/netlink.c
+++ b/fs/nfsd/netlink.c
@@ -225,6 +225,11 @@ static const struct genl_split_ops nfsd_nl_ops[] = {
 		.maxattr	= NFSD_A_UNLOCK_EXPORT_PATH,
 		.flags		= GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
 	},
+	{
+		.cmd	= NFSD_CMD_SERVER_STATS_GET,
+		.dumpit	= nfsd_nl_server_stats_get_dumpit,
+		.flags	= GENL_CMD_CAP_DUMP,
+	},
 };
 
 static const struct genl_multicast_group nfsd_nl_mcgrps[] = {
diff --git a/fs/nfsd/netlink.h b/fs/nfsd/netlink.h
index af41aa0d4a65..027e2953db26 100644
--- a/fs/nfsd/netlink.h
+++ b/fs/nfsd/netlink.h
@@ -42,6 +42,8 @@ int nfsd_nl_cache_flush_doit(struct sk_buff *skb, struct genl_info *info);
 int nfsd_nl_unlock_ip_doit(struct sk_buff *skb, struct genl_info *info);
 int nfsd_nl_unlock_filesystem_doit(struct sk_buff *skb, struct genl_info *info);
 int nfsd_nl_unlock_export_doit(struct sk_buff *skb, struct genl_info *info);
+int nfsd_nl_server_stats_get_dumpit(struct sk_buff *skb,
+				    struct netlink_callback *cb);
 
 enum {
 	NFSD_NLGRP_NONE,
diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index 601301e34fc7..f0514d8149cd 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -2329,6 +2329,185 @@ int nfsd_nl_cache_flush_doit(struct sk_buff *skb, struct genl_info *info)
 	return 0;
 }
 
+static int nfsd_nl_fill_proc_ops(struct sk_buff *skb, int attr,
+				 unsigned long __percpu *counts,
+				 unsigned int nproc)
+{
+	struct nlattr *nest;
+	unsigned int j;
+	int k;
+
+	for (j = 0; j < nproc; j++) {
+		unsigned long count = 0;
+
+		for_each_possible_cpu(k)
+			count += per_cpu(counts[j], k);
+
+		nest = nla_nest_start(skb, attr);
+		if (!nest)
+			return -EMSGSIZE;
+		if (nla_put_u32(skb, NFSD_A_SERVER_PROC_ENTRY_OP, j) ||
+		    nla_put_u64_64bit(skb, NFSD_A_SERVER_PROC_ENTRY_COUNT,
+				      count, NFSD_A_SERVER_PROC_ENTRY_PAD)) {
+			nla_nest_cancel(skb, nest);
+			return -EMSGSIZE;
+		}
+		nla_nest_end(skb, nest);
+	}
+
+	return 0;
+}
+
+/**
+ * nfsd_nl_server_stats_get_dumpit - dump NFS server statistics
+ * @skb: reply buffer
+ * @cb: netlink metadata and command arguments
+ *
+ * The entire server-stats object is emitted in a single netlink message on
+ * the first invocation. cb->args[0] is set to -1 afterwards so that the next
+ * invocation terminates the dump.
+ *
+ * Returns the size of the reply or a negative errno.
+ */
+int nfsd_nl_server_stats_get_dumpit(struct sk_buff *skb,
+				    struct netlink_callback *cb)
+{
+	struct net *net = sock_net(skb->sk);
+	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
+	struct svc_stat *statp = &nn->nfsd_svcstats;
+	struct svc_program *prog = statp->program;
+	int start = cb->args[0];
+	void *hdr;
+
+	/*
+	 * cb->args[0] == 0: first call, emit the full server-stats object
+	 * cb->args[0] < 0: dump already complete
+	 */
+	if (start < 0)
+		return 0;
+
+	if (start == 0) {
+		hdr = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
+				  cb->nlh->nlmsg_seq, &nfsd_nl_family,
+				  NLM_F_MULTI, NFSD_CMD_SERVER_STATS_GET);
+		if (!hdr)
+			return -ENOBUFS;
+
+		/* Reply cache stats */
+		{
+			u64 hits, misses, nocache;
+
+			hits = percpu_counter_sum_positive(&nn->counter[NFSD_STATS_RC_HITS]);
+			misses = percpu_counter_sum_positive(&nn->counter[NFSD_STATS_RC_MISSES]);
+			nocache = percpu_counter_sum_positive(&nn->counter[NFSD_STATS_RC_NOCACHE]);
+			if (nla_put_u64_64bit(skb, NFSD_A_SERVER_STATS_RC_HITS,
+					hits, NFSD_A_SERVER_STATS_PAD) ||
+			    nla_put_u64_64bit(skb, NFSD_A_SERVER_STATS_RC_MISSES,
+					misses, NFSD_A_SERVER_STATS_PAD) ||
+			    nla_put_u64_64bit(skb, NFSD_A_SERVER_STATS_RC_NOCACHE,
+					nocache, NFSD_A_SERVER_STATS_PAD))
+				goto err_cancel;
+		}
+
+		/* Filehandle stats */
+		if (nla_put_u64_64bit(skb, NFSD_A_SERVER_STATS_FH_STALE,
+				percpu_counter_sum_positive(&nn->counter[NFSD_STATS_FH_STALE]),
+				NFSD_A_SERVER_STATS_PAD))
+			goto err_cancel;
+
+		/* IO stats */
+		{
+			u64 rd, wr;
+
+			rd = percpu_counter_sum_positive(&nn->counter[NFSD_STATS_IO_READ]);
+			wr = percpu_counter_sum_positive(&nn->counter[NFSD_STATS_IO_WRITE]);
+			if (nla_put_u64_64bit(skb, NFSD_A_SERVER_STATS_IO_READ,
+					rd, NFSD_A_SERVER_STATS_PAD) ||
+			    nla_put_u64_64bit(skb, NFSD_A_SERVER_STATS_IO_WRITE,
+					wr, NFSD_A_SERVER_STATS_PAD))
+				goto err_cancel;
+		}
+
+		/* Network stats */
+		if (nla_put_u32(skb, NFSD_A_SERVER_STATS_NETCNT,
+				statp->netcnt) ||
+		    nla_put_u32(skb, NFSD_A_SERVER_STATS_NETUDPCNT,
+				statp->netudpcnt) ||
+		    nla_put_u32(skb, NFSD_A_SERVER_STATS_NETTCPCNT,
+				statp->nettcpcnt) ||
+		    nla_put_u32(skb, NFSD_A_SERVER_STATS_NETTCPCONN,
+				statp->nettcpconn))
+			goto err_cancel;
+
+		/* RPC stats */
+		if (nla_put_u32(skb, NFSD_A_SERVER_STATS_RPCCNT,
+				statp->rpccnt) ||
+		    nla_put_u32(skb, NFSD_A_SERVER_STATS_RPCBADFMT,
+				statp->rpcbadfmt) ||
+		    nla_put_u32(skb, NFSD_A_SERVER_STATS_RPCBADAUTH,
+				statp->rpcbadauth) ||
+		    nla_put_u32(skb, NFSD_A_SERVER_STATS_RPCBADCLNT,
+				statp->rpcbadclnt))
+			goto err_cancel;
+
+		/* Per-version procedure counts */
+		if (statp->vs_count) {
+			static const int proc_attrs[] = {
+				[2] = NFSD_A_SERVER_STATS_PROC2_OPS,
+				[3] = NFSD_A_SERVER_STATS_PROC3_OPS,
+				[4] = NFSD_A_SERVER_STATS_PROC4_OPS,
+			};
+			unsigned int i;
+
+			for (i = 0; i < prog->pg_nvers &&
+			     i < ARRAY_SIZE(proc_attrs); i++) {
+				if (!prog->pg_vers[i] ||
+				    !statp->vs_count[i])
+					continue;
+				if (!proc_attrs[i])
+					continue;
+				if (nfsd_nl_fill_proc_ops(skb,
+						proc_attrs[i],
+						statp->vs_count[i],
+						prog->pg_vers[i]->vs_nproc))
+					goto err_cancel;
+			}
+		}
+
+#ifdef CONFIG_NFSD_V4
+		/* NFSv4 individual operation counts */
+		for (int i = 0; i <= LAST_NFS4_OP; i++) {
+			struct nlattr *nest;
+			u64 cnt;
+
+			cnt = percpu_counter_sum_positive(
+				&nn->counter[NFSD_STATS_NFS4_OP(i)]);
+
+			nest = nla_nest_start(skb,
+					NFSD_A_SERVER_STATS_PROC4OPS_OPS);
+			if (!nest)
+				goto err_cancel;
+			if (nla_put_u32(skb, NFSD_A_SERVER_PROC_ENTRY_OP, i) ||
+			    nla_put_u64_64bit(skb, NFSD_A_SERVER_PROC_ENTRY_COUNT,
+					      cnt, NFSD_A_SERVER_PROC_ENTRY_PAD)) {
+				nla_nest_cancel(skb, nest);
+				goto err_cancel;
+			}
+			nla_nest_end(skb, nest);
+		}
+#endif
+
+		genlmsg_end(skb, hdr);
+	}
+
+	cb->args[0] = -1;
+	return skb->len;
+
+err_cancel:
+	genlmsg_cancel(skb, hdr);
+	return -EMSGSIZE;
+}
+
 int nfsd_cache_notify(struct cache_detail *cd, struct cache_head *h, u32 cache_type)
 {
 	struct genlmsghdr *hdr;
diff --git a/include/uapi/linux/nfsd_netlink.h b/include/uapi/linux/nfsd_netlink.h
index f5b75d5caba9..3d076d173b1d 100644
--- a/include/uapi/linux/nfsd_netlink.h
+++ b/include/uapi/linux/nfsd_netlink.h
@@ -225,6 +225,40 @@ enum {
 	NFSD_A_UNLOCK_EXPORT_MAX = (__NFSD_A_UNLOCK_EXPORT_MAX - 1)
 };
 
+enum {
+	NFSD_A_SERVER_PROC_ENTRY_OP = 1,
+	NFSD_A_SERVER_PROC_ENTRY_COUNT,
+	NFSD_A_SERVER_PROC_ENTRY_PAD,
+
+	__NFSD_A_SERVER_PROC_ENTRY_MAX,
+	NFSD_A_SERVER_PROC_ENTRY_MAX = (__NFSD_A_SERVER_PROC_ENTRY_MAX - 1)
+};
+
+enum {
+	NFSD_A_SERVER_STATS_RC_HITS = 1,
+	NFSD_A_SERVER_STATS_RC_MISSES,
+	NFSD_A_SERVER_STATS_RC_NOCACHE,
+	NFSD_A_SERVER_STATS_PAD,
+	NFSD_A_SERVER_STATS_FH_STALE,
+	NFSD_A_SERVER_STATS_IO_READ,
+	NFSD_A_SERVER_STATS_IO_WRITE,
+	NFSD_A_SERVER_STATS_NETCNT,
+	NFSD_A_SERVER_STATS_NETUDPCNT,
+	NFSD_A_SERVER_STATS_NETTCPCNT,
+	NFSD_A_SERVER_STATS_NETTCPCONN,
+	NFSD_A_SERVER_STATS_RPCCNT,
+	NFSD_A_SERVER_STATS_RPCBADFMT,
+	NFSD_A_SERVER_STATS_RPCBADAUTH,
+	NFSD_A_SERVER_STATS_RPCBADCLNT,
+	NFSD_A_SERVER_STATS_PROC2_OPS,
+	NFSD_A_SERVER_STATS_PROC3_OPS,
+	NFSD_A_SERVER_STATS_PROC4_OPS,
+	NFSD_A_SERVER_STATS_PROC4OPS_OPS,
+
+	__NFSD_A_SERVER_STATS_MAX,
+	NFSD_A_SERVER_STATS_MAX = (__NFSD_A_SERVER_STATS_MAX - 1)
+};
+
 enum {
 	NFSD_CMD_RPC_STATUS_GET = 1,
 	NFSD_CMD_THREADS_SET,
@@ -244,6 +278,7 @@ enum {
 	NFSD_CMD_UNLOCK_IP,
 	NFSD_CMD_UNLOCK_FILESYSTEM,
 	NFSD_CMD_UNLOCK_EXPORT,
+	NFSD_CMD_SERVER_STATS_GET,
 
 	__NFSD_CMD_MAX,
 	NFSD_CMD_MAX = (__NFSD_CMD_MAX - 1)

-- 
2.54.0

[PATCH v6 4/6] sunrpc: remove unused svc_version vs_count field

[Jeff Layton]

Now that svc_seq_show() and the nfsd netlink stats handler both use
the per-netns svc_stat vs_count arrays, the global per-version
vs_count percpu counters are no longer read by anything. Remove the
vs_count field from struct svc_version and all the associated
DEFINE_PER_CPU_ALIGNED arrays and initializers across nfsd, lockd,
and the NFS client callback service.

Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
 fs/lockd/svc4proc.c        | 4 ----
 fs/lockd/svcproc.c         | 7 -------
 fs/nfs/callback_xdr.c      | 6 ------
 fs/nfsd/localio.c          | 3 ---
 fs/nfsd/nfs2acl.c          | 3 ---
 fs/nfsd/nfs3acl.c          | 3 ---
 fs/nfsd/nfs3proc.c         | 3 ---
 fs/nfsd/nfs4proc.c         | 3 ---
 fs/nfsd/nfsproc.c          | 3 ---
 include/linux/sunrpc/svc.h | 1 -
 net/sunrpc/svc.c           | 3 ---
 11 files changed, 39 deletions(-)

diff --git a/fs/lockd/svc4proc.c b/fs/lockd/svc4proc.c
index 080dffce9d8e..c096c0c0cf60 100644
--- a/fs/lockd/svc4proc.c
+++ b/fs/lockd/svc4proc.c
@@ -1420,14 +1420,10 @@ union nlm4svc_xdrstore {
 	struct nlm4_shareres_wrapper	shareres;
 };
 
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      nlm4svc_call_counters[ARRAY_SIZE(nlm4svc_procedures)]);
-
 const struct svc_version nlmsvc_version4 = {
 	.vs_vers	= 4,
 	.vs_nproc	= ARRAY_SIZE(nlm4svc_procedures),
 	.vs_proc	= nlm4svc_procedures,
-	.vs_count	= nlm4svc_call_counters,
 	.vs_dispatch	= nlmsvc_dispatch,
 	.vs_xdrsize	= sizeof(union nlm4svc_xdrstore),
 };
diff --git a/fs/lockd/svcproc.c b/fs/lockd/svcproc.c
index dce6f6e3fd40..07f65a03f300 100644
--- a/fs/lockd/svcproc.c
+++ b/fs/lockd/svcproc.c
@@ -1433,25 +1433,18 @@ union nlmsvc_xdrstore {
  * NLMv1 defines only procedures 1 - 15. Linux lockd also implements
  * procedures 0 (NULL) and 16 (SM_NOTIFY).
  */
-static DEFINE_PER_CPU_ALIGNED(unsigned long, nlm1svc_call_counters[17]);
-
 const struct svc_version nlmsvc_version1 = {
 	.vs_vers	= 1,
 	.vs_nproc	= 17,
 	.vs_proc	= nlmsvc_procedures,
-	.vs_count	= nlm1svc_call_counters,
 	.vs_dispatch	= nlmsvc_dispatch,
 	.vs_xdrsize	= sizeof(union nlmsvc_xdrstore),
 };
 
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      nlm3svc_call_counters[ARRAY_SIZE(nlmsvc_procedures)]);
-
 const struct svc_version nlmsvc_version3 = {
 	.vs_vers	= 3,
 	.vs_nproc	= ARRAY_SIZE(nlmsvc_procedures),
 	.vs_proc	= nlmsvc_procedures,
-	.vs_count	= nlm3svc_call_counters,
 	.vs_dispatch	= nlmsvc_dispatch,
 	.vs_xdrsize	= sizeof(union nlmsvc_xdrstore),
 };
diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c
index 4382baddc9ee..eec6040556c9 100644
--- a/fs/nfs/callback_xdr.c
+++ b/fs/nfs/callback_xdr.c
@@ -1090,26 +1090,20 @@ static const struct svc_procedure nfs4_callback_procedures1[] = {
 	}
 };
 
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      nfs4_callback_count1[ARRAY_SIZE(nfs4_callback_procedures1)]);
 const struct svc_version nfs4_callback_version1 = {
 	.vs_vers = 1,
 	.vs_nproc = ARRAY_SIZE(nfs4_callback_procedures1),
 	.vs_proc = nfs4_callback_procedures1,
-	.vs_count = nfs4_callback_count1,
 	.vs_xdrsize = NFS4_CALLBACK_XDRSIZE,
 	.vs_dispatch = nfs_callback_dispatch,
 	.vs_hidden = true,
 	.vs_need_cong_ctrl = true,
 };
 
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      nfs4_callback_count4[ARRAY_SIZE(nfs4_callback_procedures1)]);
 const struct svc_version nfs4_callback_version4 = {
 	.vs_vers = 4,
 	.vs_nproc = ARRAY_SIZE(nfs4_callback_procedures1),
 	.vs_proc = nfs4_callback_procedures1,
-	.vs_count = nfs4_callback_count4,
 	.vs_xdrsize = NFS4_CALLBACK_XDRSIZE,
 	.vs_dispatch = nfs_callback_dispatch,
 	.vs_hidden = true,
diff --git a/fs/nfsd/localio.c b/fs/nfsd/localio.c
index c3eb0557b3e1..c458c01e9478 100644
--- a/fs/nfsd/localio.c
+++ b/fs/nfsd/localio.c
@@ -210,14 +210,11 @@ static const struct svc_procedure localio_procedures1[] = {
 };
 
 #define LOCALIO_NR_PROCEDURES ARRAY_SIZE(localio_procedures1)
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      localio_count[LOCALIO_NR_PROCEDURES]);
 const struct svc_version localio_version1 = {
 	.vs_vers	= 1,
 	.vs_nproc	= LOCALIO_NR_PROCEDURES,
 	.vs_proc	= localio_procedures1,
 	.vs_dispatch	= nfsd_dispatch,
-	.vs_count	= localio_count,
 	.vs_xdrsize	= XDR_QUADLEN(UUID_SIZE),
 	.vs_hidden	= true,
 };
diff --git a/fs/nfsd/nfs2acl.c b/fs/nfsd/nfs2acl.c
index 827f90194c43..0fd6d9def3b4 100644
--- a/fs/nfsd/nfs2acl.c
+++ b/fs/nfsd/nfs2acl.c
@@ -389,13 +389,10 @@ static const struct svc_procedure nfsd_acl_procedures2[5] = {
 	},
 };
 
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      nfsd_acl_count2[ARRAY_SIZE(nfsd_acl_procedures2)]);
 const struct svc_version nfsd_acl_version2 = {
 	.vs_vers	= 2,
 	.vs_nproc	= ARRAY_SIZE(nfsd_acl_procedures2),
 	.vs_proc	= nfsd_acl_procedures2,
-	.vs_count	= nfsd_acl_count2,
 	.vs_dispatch	= nfsd_dispatch,
 	.vs_xdrsize	= NFS3_SVC_XDRSIZE,
 };
diff --git a/fs/nfsd/nfs3acl.c b/fs/nfsd/nfs3acl.c
index a87f9d7f32be..6b6b289db636 100644
--- a/fs/nfsd/nfs3acl.c
+++ b/fs/nfsd/nfs3acl.c
@@ -278,13 +278,10 @@ static const struct svc_procedure nfsd_acl_procedures3[3] = {
 	},
 };
 
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      nfsd_acl_count3[ARRAY_SIZE(nfsd_acl_procedures3)]);
 const struct svc_version nfsd_acl_version3 = {
 	.vs_vers	= 3,
 	.vs_nproc	= ARRAY_SIZE(nfsd_acl_procedures3),
 	.vs_proc	= nfsd_acl_procedures3,
-	.vs_count	= nfsd_acl_count3,
 	.vs_dispatch	= nfsd_dispatch,
 	.vs_xdrsize	= NFS3_SVC_XDRSIZE,
 };
diff --git a/fs/nfsd/nfs3proc.c b/fs/nfsd/nfs3proc.c
index 617a70d13292..a547ffc9b3d1 100644
--- a/fs/nfsd/nfs3proc.c
+++ b/fs/nfsd/nfs3proc.c
@@ -1108,13 +1108,10 @@ static const struct svc_procedure nfsd_procedures3[22] = {
 	},
 };
 
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      nfsd_count3[ARRAY_SIZE(nfsd_procedures3)]);
 const struct svc_version nfsd_version3 = {
 	.vs_vers	= 3,
 	.vs_nproc	= ARRAY_SIZE(nfsd_procedures3),
 	.vs_proc	= nfsd_procedures3,
 	.vs_dispatch	= nfsd_dispatch,
-	.vs_count	= nfsd_count3,
 	.vs_xdrsize	= NFS3_SVC_XDRSIZE,
 };
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index c413ed0810b9..005a9ce7ceb8 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -4177,13 +4177,10 @@ static const struct svc_procedure nfsd_procedures4[2] = {
 	},
 };
 
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      nfsd_count4[ARRAY_SIZE(nfsd_procedures4)]);
 const struct svc_version nfsd_version4 = {
 	.vs_vers		= 4,
 	.vs_nproc		= ARRAY_SIZE(nfsd_procedures4),
 	.vs_proc		= nfsd_procedures4,
-	.vs_count		= nfsd_count4,
 	.vs_dispatch		= nfsd_dispatch,
 	.vs_xdrsize		= NFS4_SVC_XDRSIZE,
 	.vs_rpcb_optnl		= true,
diff --git a/fs/nfsd/nfsproc.c b/fs/nfsd/nfsproc.c
index a73d5c259cd9..b8421cde795a 100644
--- a/fs/nfsd/nfsproc.c
+++ b/fs/nfsd/nfsproc.c
@@ -845,13 +845,10 @@ static const struct svc_procedure nfsd_procedures2[18] = {
 	},
 };
 
-static DEFINE_PER_CPU_ALIGNED(unsigned long,
-			      nfsd_count2[ARRAY_SIZE(nfsd_procedures2)]);
 const struct svc_version nfsd_version2 = {
 	.vs_vers	= 2,
 	.vs_nproc	= ARRAY_SIZE(nfsd_procedures2),
 	.vs_proc	= nfsd_procedures2,
-	.vs_count	= nfsd_count2,
 	.vs_dispatch	= nfsd_dispatch,
 	.vs_xdrsize	= NFS2_SVC_XDRSIZE,
 };
diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h
index 3a0152d926fb..34490c0ba88c 100644
--- a/include/linux/sunrpc/svc.h
+++ b/include/linux/sunrpc/svc.h
@@ -408,7 +408,6 @@ struct svc_version {
 	u32			vs_vers;	/* version number */
 	u32			vs_nproc;	/* number of procedures */
 	const struct svc_procedure *vs_proc;	/* per-procedure info */
-	unsigned long __percpu	*vs_count;	/* call counts */
 	u32			vs_xdrsize;	/* xdrsize needed for this version */
 
 	/* Don't register with rpcbind */
diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c
index 0cee079df1cb..8be33ffb32c8 100644
--- a/net/sunrpc/svc.c
+++ b/net/sunrpc/svc.c
@@ -1435,9 +1435,6 @@ svc_generic_init_request(struct svc_rqst *rqstp,
 	memset(rqstp->rq_argp, 0, procp->pc_argzero);
 	memset(rqstp->rq_resp, 0, procp->pc_ressize);
 
-	/* Bump per-procedure stats counter */
-	this_cpu_inc(versp->vs_count[rqstp->rq_proc]);
-
 	/* Bump per-net per-procedure stats counter */
 	if (rqstp->rq_server->sv_stats &&
 	    rqstp->rq_server->sv_stats->program == progp &&

-- 
2.54.0

[PATCH v6 5/6] nfsd: count NFSv4 callback operations per netns

[Jeff Layton]

The NFS server tracks per-operation call counts for the forward channel
(proc4ops) but keeps no statistics for the NFSv4 backchannel (callback)
operations it sends to clients.

Add a per-netns array of percpu counters for callback operations, indexed
by RFC 8881 callback opcode (OP_CB_GETATTR..OP_CB_OFFLOAD), and bump the
relevant counter in nfsd4_run_cb(), which is hit exactly once per callback
that is actually queued.

CB_GETATTR is sent when a GETATTR conflicts with an outstanding write
delegation, which is roughly what the dedicated wdeleg_getattr counter
tracked. The two are not identical: the old counter incremented on every
such conflict, whereas the CB_GETATTR counter only counts callbacks that
are actually queued, so concurrent conflicts that coalesce onto an
already in-flight CB_GETATTR are now counted once rather than once per
conflict. Report the procfs "wdeleg_getattr" line from the CB_GETATTR
counter and drop the now-redundant NFSD_STATS_WDELEG_GETATTR counter, its
helper, and its increment site.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
 fs/nfsd/netns.h        | 12 +++++++++++-
 fs/nfsd/nfs4callback.c | 22 +++++++++++++++++++++-
 fs/nfsd/nfs4state.c    |  2 --
 fs/nfsd/nfsctl.c       | 14 ++++++++++++++
 fs/nfsd/stats.c        |  2 +-
 fs/nfsd/stats.h        |  5 +++--
 6 files changed, 50 insertions(+), 7 deletions(-)

diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h
index 03724bef10a7..37daba0d4747 100644
--- a/fs/nfsd/netns.h
+++ b/fs/nfsd/netns.h
@@ -53,11 +53,16 @@ enum {
 	NFSD_STATS_FIRST_NFS4_OP,	/* count of individual nfsv4 operations */
 	NFSD_STATS_LAST_NFS4_OP = NFSD_STATS_FIRST_NFS4_OP + LAST_NFS4_OP,
 #define NFSD_STATS_NFS4_OP(op)	(NFSD_STATS_FIRST_NFS4_OP + (op))
-	NFSD_STATS_WDELEG_GETATTR,	/* count of getattr conflict with wdeleg */
 #endif
 	NFSD_STATS_COUNTERS_NUM
 };
 
+/*
+ * Per-netns NFSv4 callback (backchannel) per-operation counters, indexed
+ * directly by RFC 8881 callback opcode (OP_CB_GETATTR..OP_CB_OFFLOAD).
+ */
+#define NFSD_STATS_CB_OPS_NUM	(OP_CB_OFFLOAD + 1)
+
 /*
  * Represents a nfsd "container". With respect to nfsv4 state tracking, the
  * fields of interest are the *_id_hashtbls and the *_name_tree. These track
@@ -198,6 +203,11 @@ struct nfsd_net {
 	/* Per-netns stats counters */
 	struct percpu_counter    counter[NFSD_STATS_COUNTERS_NUM];
 
+#ifdef CONFIG_NFSD_V4
+	/* Per-netns NFSv4 callback (backchannel) per-operation counters */
+	struct percpu_counter    cb_counter[NFSD_STATS_CB_OPS_NUM];
+#endif
+
 	/* sunrpc svc stats */
 	struct svc_stat          nfsd_svcstats;
 
diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
index 71dcb448fa0a..b171257da6ba 100644
--- a/fs/nfsd/nfs4callback.c
+++ b/fs/nfsd/nfs4callback.c
@@ -1921,12 +1921,32 @@ void nfsd4_init_cb(struct nfsd4_callback *cb, struct nfs4_client *clp,
 bool nfsd4_run_cb(struct nfsd4_callback *cb)
 {
 	struct nfs4_client *clp = cb->cb_clp;
+	struct nfsd_net *nn = net_generic(clp->net, nfsd_net_id);
+	const struct nfsd4_callback_ops *ops = cb->cb_ops;
+	u32 minorversion = clp->cl_minorversion;
 	bool queued;
 
+	/*
+	 * Snapshot the opcode, minorversion, and the per-net pointer before
+	 * queuing: once nfsd4_queue_cb() has queued the work, the callback (and
+	 * the object that embeds it) may be processed and freed concurrently, so
+	 * neither cb nor clp can be dereferenced afterward.
+	 */
 	nfsd41_cb_inflight_begin(clp);
 	queued = nfsd4_queue_cb(cb);
-	if (!queued)
+	if (queued) {
+		if (ops)
+			nfsd_stats_cb_op_inc(nn, ops->opcode);
+		/*
+		 * Minorversion > 0 callbacks prepend a CB_SEQUENCE op (see
+		 * encode_cb_sequence4args()); count it like the forechannel
+		 * counts SEQUENCE, so it isn't perpetually reported as zero.
+		 */
+		if (minorversion > 0)
+			nfsd_stats_cb_op_inc(nn, OP_CB_SEQUENCE);
+	} else {
 		nfsd41_cb_inflight_end(clp);
+	}
 	return queued;
 }
 
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index b830aed7ae39..8f8da1312231 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -9904,7 +9904,6 @@ __be32
 nfsd4_deleg_getattr_conflict(struct svc_rqst *rqstp, struct dentry *dentry,
 			     struct nfs4_delegation **pdp)
 {
-	struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
 	struct nfsd_thread_local_info *ntli = rqstp->rq_private;
 	struct file_lock_context *ctx;
 	struct nfs4_delegation *dp = NULL;
@@ -9944,7 +9943,6 @@ nfsd4_deleg_getattr_conflict(struct svc_rqst *rqstp, struct dentry *dentry,
 		return 0;
 	}
 
-	nfsd_stats_wdeleg_getattr_inc(nn);
 	refcount_inc(&dp->dl_stid.sc_count);
 	ncf = &dp->dl_cb_fattr;
 	nfs4_cb_getattr(&dp->dl_cb_fattr);
diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index f0514d8149cd..4710415790e2 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -2691,6 +2691,13 @@ static __net_init int nfsd_net_init(struct net *net)
 	if (retval)
 		goto out_repcache_error;
 
+#ifdef CONFIG_NFSD_V4
+	retval = percpu_counter_init_many(nn->cb_counter, 0, GFP_KERNEL,
+					  NFSD_STATS_CB_OPS_NUM);
+	if (retval)
+		goto out_cb_counter_error;
+#endif
+
 	memset(&nn->nfsd_svcstats, 0, sizeof(nn->nfsd_svcstats));
 	nn->nfsd_svcstats.program = &nfsd_programs[0];
 	retval = svc_stat_alloc_counts(&nn->nfsd_svcstats);
@@ -2719,6 +2726,10 @@ static __net_init int nfsd_net_init(struct net *net)
 out_svcstats_error:
 	svc_stat_free_counts(&nn->nfsd_svcstats);
 out_proc_error:
+#ifdef CONFIG_NFSD_V4
+	percpu_counter_destroy_many(nn->cb_counter, NFSD_STATS_CB_OPS_NUM);
+out_cb_counter_error:
+#endif
 	percpu_counter_destroy_many(nn->counter, NFSD_STATS_COUNTERS_NUM);
 out_repcache_error:
 	nfsd_idmap_shutdown(net);
@@ -2759,6 +2770,9 @@ static __net_exit void nfsd_net_exit(struct net *net)
 	nfsd_net_cb_shutdown(nn);
 	nfsd_proc_stat_shutdown(net);
 	svc_stat_free_counts(&nn->nfsd_svcstats);
+#ifdef CONFIG_NFSD_V4
+	percpu_counter_destroy_many(nn->cb_counter, NFSD_STATS_CB_OPS_NUM);
+#endif
 	percpu_counter_destroy_many(nn->counter, NFSD_STATS_COUNTERS_NUM);
 	nfsd_idmap_shutdown(net);
 	nfsd_export_shutdown(net);
diff --git a/fs/nfsd/stats.c b/fs/nfsd/stats.c
index f7eaf95e20fc..35d9a5571946 100644
--- a/fs/nfsd/stats.c
+++ b/fs/nfsd/stats.c
@@ -63,7 +63,7 @@ static int nfsd_show(struct seq_file *seq, void *v)
 			   percpu_counter_sum_positive(&nn->counter[NFSD_STATS_NFS4_OP(i)]));
 	}
 	seq_printf(seq, "\nwdeleg_getattr %lld",
-		percpu_counter_sum_positive(&nn->counter[NFSD_STATS_WDELEG_GETATTR]));
+		percpu_counter_sum_positive(&nn->cb_counter[OP_CB_GETATTR]));
 
 	seq_putc(seq, '\n');
 #endif
diff --git a/fs/nfsd/stats.h b/fs/nfsd/stats.h
index e4efb0e4e56d..f90cd565f7cb 100644
--- a/fs/nfsd/stats.h
+++ b/fs/nfsd/stats.h
@@ -68,9 +68,10 @@ static inline void nfsd_stats_drc_mem_usage_sub(struct nfsd_net *nn, s64 amount)
 }
 
 #ifdef CONFIG_NFSD_V4
-static inline void nfsd_stats_wdeleg_getattr_inc(struct nfsd_net *nn)
+static inline void nfsd_stats_cb_op_inc(struct nfsd_net *nn, u32 opcode)
 {
-	percpu_counter_inc(&nn->counter[NFSD_STATS_WDELEG_GETATTR]);
+	if (opcode >= OP_CB_GETATTR && opcode <= OP_CB_OFFLOAD)
+		percpu_counter_inc(&nn->cb_counter[opcode]);
 }
 #endif
 #endif /* _NFSD_STATS_H */

-- 
2.54.0

[PATCH v6 6/6] nfsd: export NFSv4 callback op stats via netlink

[Jeff Layton]

Add a proc4cb-ops nested attribute to the server-stats netlink dump,
reusing the existing server-proc-entry (op/count) layout. The first dump
message emits one entry per callback opcode (OP_CB_GETATTR..OP_CB_OFFLOAD)
from the per-netns callback counters; the set is small and fits alongside
the scalar stats, so no extra paging is needed.

This lets nfsstat report NFSv4 backchannel operation counts over netlink,
including CB_GETATTR which corresponds to the procfs wdeleg_getattr line.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
 Documentation/netlink/specs/nfsd.yaml |  6 ++++++
 fs/nfsd/nfsctl.c                      | 21 +++++++++++++++++++++
 include/uapi/linux/nfsd_netlink.h     |  1 +
 3 files changed, 28 insertions(+)

diff --git a/Documentation/netlink/specs/nfsd.yaml b/Documentation/netlink/specs/nfsd.yaml
index 2a89d355ee7b..642268819c6f 100644
--- a/Documentation/netlink/specs/nfsd.yaml
+++ b/Documentation/netlink/specs/nfsd.yaml
@@ -410,6 +410,11 @@ attribute-sets:
         type: nest
         nested-attributes: server-proc-entry
         multi-attr: true
+      -
+        name: proc4cb-ops
+        type: nest
+        nested-attributes: server-proc-entry
+        multi-attr: true
 
 operations:
   list:
@@ -621,6 +626,7 @@ operations:
             - proc3-ops
             - proc4-ops
             - proc4ops-ops
+            - proc4cb-ops
 
 mcast-groups:
   list:
diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index 4710415790e2..6e941d8d08e1 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -2475,6 +2475,27 @@ int nfsd_nl_server_stats_get_dumpit(struct sk_buff *skb,
 		}
 
 #ifdef CONFIG_NFSD_V4
+		/* NFSv4 callback (backchannel) per-operation counts */
+		for (int op = OP_CB_GETATTR; op <= OP_CB_OFFLOAD; op++) {
+			struct nlattr *nest;
+			u64 cnt;
+
+			cnt = percpu_counter_sum_positive(
+				&nn->cb_counter[op]);
+
+			nest = nla_nest_start(skb,
+					NFSD_A_SERVER_STATS_PROC4CB_OPS);
+			if (!nest)
+				goto err_cancel;
+			if (nla_put_u32(skb, NFSD_A_SERVER_PROC_ENTRY_OP, op) ||
+			    nla_put_u64_64bit(skb, NFSD_A_SERVER_PROC_ENTRY_COUNT,
+					      cnt, NFSD_A_SERVER_PROC_ENTRY_PAD)) {
+				nla_nest_cancel(skb, nest);
+				goto err_cancel;
+			}
+			nla_nest_end(skb, nest);
+		}
+
 		/* NFSv4 individual operation counts */
 		for (int i = 0; i <= LAST_NFS4_OP; i++) {
 			struct nlattr *nest;
diff --git a/include/uapi/linux/nfsd_netlink.h b/include/uapi/linux/nfsd_netlink.h
index 3d076d173b1d..87da1d0bb21e 100644
--- a/include/uapi/linux/nfsd_netlink.h
+++ b/include/uapi/linux/nfsd_netlink.h
@@ -254,6 +254,7 @@ enum {
 	NFSD_A_SERVER_STATS_PROC3_OPS,
 	NFSD_A_SERVER_STATS_PROC4_OPS,
 	NFSD_A_SERVER_STATS_PROC4OPS_OPS,
+	NFSD_A_SERVER_STATS_PROC4CB_OPS,
 
 	__NFSD_A_SERVER_STATS_MAX,
 	NFSD_A_SERVER_STATS_MAX = (__NFSD_A_SERVER_STATS_MAX - 1)

-- 
2.54.0

Re: [PATCH v6 3/6] nfsd: implement server-stats-get netlink handler

[Chuck Lever]

On Fri, Jun 19, 2026, at 11:26 AM, Jeff Layton wrote:
> Implement nfsd_nl_server_stats_get_dumpit() which exposes the
> NFS server statistics currently available via /proc/net/rpc/nfsd
> through the nfsd generic netlink family.

A couple of questions on the dump design and one on the commit message,
then some smaller items.

The commit message says:

>   - First message: all scalar stats (reply cache, filehandle,
>     IO, network, RPC) plus per-version procedure counts
>     (proc2/3/4-ops) using per-netns vs_count arrays.
>
>   - Subsequent messages: NFSv4 per-operation counts
>     (proc4ops-ops), one entry per message, using cb->args[0]
>     to track the current operation index across dump calls.

The code does not stream across subsequent messages.  The proc4ops-ops
nests are emitted in the first message, inside the same "if (start == 0)"
block as the scalars, and cb->args[0] only ever holds 0 or -1, never an
operation index.  The function's own kerneldoc agrees: "The entire
server-stats object is emitted in a single netlink message on the first
invocation."

Should the commit message be reworded to match the code, or should the code
move to the streaming design the message describes?  That second option ties
into the next question.

> diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
> index 601301e34fc7..f0514d8149cd 100644
> --- a/fs/nfsd/nfsctl.c
> +++ b/fs/nfsd/nfsctl.c
> @@ -2329,6 +2329,185 @@ int nfsd_nl_cache_flush_doit(struct sk_buff *skb, struct genl_info *info)
>  	return 0;
>  }
>
> +static int nfsd_nl_fill_proc_ops(struct sk_buff *skb, int attr,
> +				 unsigned long __percpu *counts,
> +				 unsigned int nproc)
> +{
> +	struct nlattr *nest;
> +	unsigned int j;
> +	int k;
> +
> +	for (j = 0; j < nproc; j++) {
> +		unsigned long count = 0;
> +
> +		for_each_possible_cpu(k)
> +			count += per_cpu(counts[j], k);
> +
> +		nest = nla_nest_start(skb, attr);
> +		if (!nest)
> +			return -EMSGSIZE;
> +		if (nla_put_u32(skb, NFSD_A_SERVER_PROC_ENTRY_OP, j) ||
> +		    nla_put_u64_64bit(skb, NFSD_A_SERVER_PROC_ENTRY_COUNT,
> +				      count, NFSD_A_SERVER_PROC_ENTRY_PAD)) {
> +			nla_nest_cancel(skb, nest);
> +			return -EMSGSIZE;
> +		}
> +		nla_nest_end(skb, nest);
> +	}
> +
> +	return 0;
> +}
> +
> +/**
> + * nfsd_nl_server_stats_get_dumpit - dump NFS server statistics
> + * @skb: reply buffer
> + * @cb: netlink metadata and command arguments
> + *
> + * The entire server-stats object is emitted in a single netlink message on
> + * the first invocation. cb->args[0] is set to -1 afterwards so that the next
> + * invocation terminates the dump.
> + *
> + * Returns the size of the reply or a negative errno.
> + */
> +int nfsd_nl_server_stats_get_dumpit(struct sk_buff *skb,
> +				    struct netlink_callback *cb)
> +{
> +	struct net *net = sock_net(skb->sk);
> +	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
> +	struct svc_stat *statp = &nn->nfsd_svcstats;
> +	struct svc_program *prog = statp->program;
> +	int start = cb->args[0];
> +	void *hdr;
> +
> +	/*
> +	 * cb->args[0] == 0: first call, emit the full server-stats object
> +	 * cb->args[0] < 0: dump already complete
> +	 */
> +	if (start < 0)
> +		return 0;
> +
> +	if (start == 0) {
> +		hdr = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
> +				  cb->nlh->nlmsg_seq, &nfsd_nl_family,
> +				  NLM_F_MULTI, NFSD_CMD_SERVER_STATS_GET);
> +		if (!hdr)
> +			return -ENOBUFS;
> +

This handler assembles the entire server-stats object -- all scalars, the
proc2/3/4-ops nests, and one proc4ops-ops nest per NFSv4 operation -- in one
message under "if (start == 0)", then sets cb->args[0] = -1 so the next call
ends the dump.

Does that hold up once the assembled message no longer fits the dump buffer?
A dump skb is allocated at max(cb->min_dump_alloc, NLMSG_GOODSIZE), and this
handler sets neither cb->min_dump_alloc nor a .start callback, so the floor
is NLMSG_GOODSIZE, which is SKB_WITH_OVERHEAD(PAGE_SIZE), roughly 3.7k on a
4k-page host.  A larger skb is allocated only when userspace has previously
grown its receive buffer.

With LAST_NFS4_OP at 75, the proc4ops-ops loop alone emits 76 nests of about
28 bytes each (~2.1k); the proc2 and proc3 nests plus the scalar block bring
the total to roughly 3.5k -- already within a few hundred bytes of the floor,
and growing by one nest for every NFSv4 operation added later.

[ ... scalar puts for reply cache, fh, io, net, rpc snipped ... ]

> +		/* Per-version procedure counts */
> +		if (statp->vs_count) {
> +			static const int proc_attrs[] = {
> +				[2] = NFSD_A_SERVER_STATS_PROC2_OPS,
> +				[3] = NFSD_A_SERVER_STATS_PROC3_OPS,
> +				[4] = NFSD_A_SERVER_STATS_PROC4_OPS,
> +			};
> +			unsigned int i;
> +
> +			for (i = 0; i < prog->pg_nvers &&
> +			     i < ARRAY_SIZE(proc_attrs); i++) {
> +				if (!prog->pg_vers[i] ||
> +				    !statp->vs_count[i])
> +					continue;
> +				if (!proc_attrs[i])
> +					continue;
> +				if (nfsd_nl_fill_proc_ops(skb,
> +						proc_attrs[i],
> +						statp->vs_count[i],
> +						prog->pg_vers[i]->vs_nproc))
> +					goto err_cancel;
> +			}
> +		}
> +
> +#ifdef CONFIG_NFSD_V4
> +		/* NFSv4 individual operation counts */
> +		for (int i = 0; i <= LAST_NFS4_OP; i++) {
> +			struct nlattr *nest;
> +			u64 cnt;
> +
> +			cnt = percpu_counter_sum_positive(
> +				&nn->counter[NFSD_STATS_NFS4_OP(i)]);
> +
> +			nest = nla_nest_start(skb,
> +					NFSD_A_SERVER_STATS_PROC4OPS_OPS);
> +			if (!nest)
> +				goto err_cancel;
> +			if (nla_put_u32(skb, NFSD_A_SERVER_PROC_ENTRY_OP, i) ||
> +			    nla_put_u64_64bit(skb, NFSD_A_SERVER_PROC_ENTRY_COUNT,
> +					      cnt, NFSD_A_SERVER_PROC_ENTRY_PAD)) {
> +				nla_nest_cancel(skb, nest);
> +				goto err_cancel;
> +			}
> +			nla_nest_end(skb, nest);
> +		}
> +#endif

This loop open-codes the same nest that nfsd_nl_fill_proc_ops() builds just
above -- nla_nest_start(), nla_put_u32(PROC_ENTRY_OP),
nla_put_u64_64bit(PROC_ENTRY_COUNT), nla_nest_end() -- into the same
NFSD_A_SERVER_STATS_PROC4OPS_OPS attribute.  Could the helper be generalized
to take the per-op counter source so this is not a second copy of the same
code?

The per-version block above skips empty versions:

	if (!prog->pg_vers[i] || !statp->vs_count[i])
		continue;

but this loop emits an entry for every op 0..LAST_NFS4_OP, zero-count ops
included.  Is that difference intentional?  Skipping zero counts here would
also trim the worst-case message size above.

There is also a counter that this dump does not emit.  /proc/net/rpc/nfsd
prints a wdeleg_getattr line after proc4ops:

	seq_printf(seq, "\nwdeleg_getattr %lld",
		percpu_counter_sum_positive(&nn->counter[NFSD_STATS_WDELEG_GETATTR]));

incremented by nfsd_stats_wdeleg_getattr_inc().  Since the goal is to expose
the statistics currently available via /proc/net/rpc/nfsd, should
wdeleg_getattr get an attribute here too, so nfsstat over netlink does not
drop it relative to the procfs path?

> +
> +		genlmsg_end(skb, hdr);
> +	}
> +
> +	cb->args[0] = -1;
> +	return skb->len;
> +
> +err_cancel:
> +	genlmsg_cancel(skb, hdr);
> +	return -EMSGSIZE;
> +}

Back to the buffer question: when an nla_put eventually fails, control
reaches err_cancel, and genlmsg_cancel() trims the partial message back out,
so skb->len is 0 on return.  In netlink_dump():

	if (nlk->dump_done_errno == -EMSGSIZE && skb->len)
		nlk->dump_done_errno = skb->len;

the skb->len test fails, so -EMSGSIZE is delivered to userspace as a
terminal error rather than being taken as "buffer full, call again".  With
everything emitted under "if (start == 0)" and no per-attribute cursor in
cb->args, a retry rebuilds the same oversized message, so the dump cannot
ever complete.

nfsd_nl_rpc_status_get_dumpit() already follows the streaming pattern the
commit message describes: one entry per message, saving the cursor in
cb->args and returning skb->len on EMSGSIZE.  Would building this dump the
same way be more robust than the single-message form?

-- 
Chuck Lever

Re: [PATCH v6 5/6] nfsd: count NFSv4 callback operations per netns

[Chuck Lever]

On Fri, Jun 19, 2026, at 11:26 AM, Jeff Layton wrote:
> The NFS server tracks per-operation call counts for the forward channel
> (proc4ops) but keeps no statistics for the NFSv4 backchannel (callback)
> operations it sends to clients.
>
> Add a per-netns array of percpu counters for callback operations, indexed
> by RFC 8881 callback opcode (OP_CB_GETATTR..OP_CB_OFFLOAD), and bump the
> relevant counter in nfsd4_run_cb(), which is hit exactly once per callback
> that is actually queued.

> diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
> index 71dcb448fa0a..b171257da6ba 100644
> --- a/fs/nfsd/nfs4callback.c
> +++ b/fs/nfsd/nfs4callback.c
> @@ -1921,12 +1921,32 @@ void nfsd4_init_cb(struct nfsd4_callback *cb, struct nfs4_client *clp,
>  bool nfsd4_run_cb(struct nfsd4_callback *cb)
>  {
>  	struct nfs4_client *clp = cb->cb_clp;
> +	struct nfsd_net *nn = net_generic(clp->net, nfsd_net_id);
> +	const struct nfsd4_callback_ops *ops = cb->cb_ops;
> +	u32 minorversion = clp->cl_minorversion;
>  	bool queued;
>
> +	/*
> +	 * Snapshot the opcode, minorversion, and the per-net pointer before
> +	 * queuing: once nfsd4_queue_cb() has queued the work, the callback (and
> +	 * the object that embeds it) may be processed and freed concurrently, so
> +	 * neither cb nor clp can be dereferenced afterward.
> +	 */

The comment says the opcode is snapshotted before queuing, but the code
captures the cb_ops pointer and reads ops->opcode after nfsd4_queue_cb()
returns:

	if (ops)
		nfsd_stats_cb_op_inc(nn, ops->opcode);

That read is safe because every cb_ops points at a file-scope
static const nfsd4_callback_ops (nfsd4_cb_recall_ops,
nfsd4_cb_getattr_ops, and so on), which outlives cb, unlike cb and clp
themselves.  Would the comment read more accurately as snapshotting the
cb_ops pointer rather than the opcode, perhaps noting that ops stays
valid afterward because it points at static data?

>  	nfsd41_cb_inflight_begin(clp);
>  	queued = nfsd4_queue_cb(cb);
> -	if (!queued)
> +	if (queued) {
> +		if (ops)
> +			nfsd_stats_cb_op_inc(nn, ops->opcode);
> +		/*
> +		 * Minorversion > 0 callbacks prepend a CB_SEQUENCE op (see
> +		 * encode_cb_sequence4args()); count it like the forechannel
> +		 * counts SEQUENCE, so it isn't perpetually reported as zero.
> +		 */
> +		if (minorversion > 0)
> +			nfsd_stats_cb_op_inc(nn, OP_CB_SEQUENCE);

Can this inflate the OP_CB_SEQUENCE count for callbacks that never send
a CB_SEQUENCE?  The increment fires whenever minorversion > 0, including
the ops == NULL case.  The only callback with cb_ops == NULL is the
null probe:

  nfsd4_init_cb(&clp->cl_cb_null, clp, NULL, NFSPROC4_CLNT_CB_NULL);

which nfsd4_probe_callback() and nfsd4_shutdown_callback() queue via
nfsd4_run_cb(&clp->cl_cb_null).  For a 4.1+ client, nfsd4_run_cb_work()
drops that work item without sending any RPC:

	/*
	 * Don't send probe messages for 4.1 or later.
	 */
	if (!cb->cb_ops && clp->cl_minorversion) {
		nfsd4_mark_cb_state(clp, NFSD4_CB_UP);
		nfsd41_destroy_cb(cb);
		return;
	}

So callback-channel probes, backchannel changes, and client teardown
each bump OP_CB_SEQUENCE for a minorversion > 0 client even though no
CB_SEQUENCE (and no callback at all) goes on the wire.

Would moving the CB_SEQUENCE increment inside the if (ops) block avoid
this?  A real callback on a 4.1+ client always prepends CB_SEQUENCE, and
the null probe carries cb_ops == NULL, so:

	if (queued) {
		if (ops) {
			nfsd_stats_cb_op_inc(nn, ops->opcode);
			if (minorversion > 0)
				nfsd_stats_cb_op_inc(nn, OP_CB_SEQUENCE);
		}
	} else {
		nfsd41_cb_inflight_end(clp);
	}

> +	} else {
>  		nfsd41_cb_inflight_end(clp);
> +	}
>  	return queued;
>  }


-- 
Chuck Lever