[PATCH 4/5] NFSD: Eliminate percpu counter contention in IO byte accounting

[Chuck Lever <cel@kernel.org>]

nfsd_stats_io_write_add() and nfsd_stats_io_read_add() accumulate
byte counts in per-net-namespace and per-export percpu_counters
using percpu_counter_add(), which applies the default batch
threshold of max(32, 2*nr_cpus).

For a 4 KB NFS WRITE, the amount (4096) always exceeds this
threshold, so percpu_counter_add_batch() acquires the counter's
global spinlock on every update. Each WRITE RPC updates two
counters (per-net and per-export), producing two global lock
acquisitions per operation. Profiling on a 10-CPU RDMA NFS
server shows 0.44% of total CPU cycles spent contending on
these locks during a small random write workload.

Switch to percpu_counter_add_local(), which batches with
INT_MAX so that updates always remain on the per-CPU fast
path regardless of the amount. All readers of these counters
already use percpu_counter_sum_positive(), which sums the
per-CPU deltas under the global lock, so read accuracy is
unaffected.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfsd/stats.h | 30 ++++++++++++++++++++++++++----
 1 file changed, 26 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/stats.h b/fs/nfsd/stats.h
index 598ea45d1722..c8d7e3934c33 100644
--- a/fs/nfsd/stats.h
+++ b/fs/nfsd/stats.h
@@ -59,20 +59,42 @@ static inline void nfsd_stats_fh_stale_inc(struct nfsd_net *nn,
 		percpu_counter_inc(&exp->ex_stats->counter[EXP_STATS_FH_STALE]);
 }
 
+/**
+ * nfsd_stats_io_read_add - Count number of bytes for an NFS READ
+ * @nn: target network namespace
+ * @exp: target export
+ * @amount: byte count
+ *
+ * These counters are updated on every READ request. Readers use
+ * percpu_counter_sum_positive(), so local batching does not affect
+ * read accuracy.
+ */
 static inline void nfsd_stats_io_read_add(struct nfsd_net *nn,
 					  struct svc_export *exp, s64 amount)
 {
-	percpu_counter_add(&nn->counter[NFSD_STATS_IO_READ], amount);
+	percpu_counter_add_local(&nn->counter[NFSD_STATS_IO_READ], amount);
 	if (exp && exp->ex_stats)
-		percpu_counter_add(&exp->ex_stats->counter[EXP_STATS_IO_READ], amount);
+		percpu_counter_add_local(&exp->ex_stats->counter[EXP_STATS_IO_READ],
+					 amount);
 }
 
+/**
+ * nfsd_stats_io_write_add - Count number of bytes for an NFS WRITE
+ * @nn: target network namespace
+ * @exp: target export
+ * @amount: byte count
+ *
+ * These counters are updated on every WRITE request. Readers use
+ * percpu_counter_sum_positive(), so local batching does not affect
+ * read accuracy.
+ */
 static inline void nfsd_stats_io_write_add(struct nfsd_net *nn,
 					   struct svc_export *exp, s64 amount)
 {
-	percpu_counter_add(&nn->counter[NFSD_STATS_IO_WRITE], amount);
+	percpu_counter_add_local(&nn->counter[NFSD_STATS_IO_WRITE], amount);
 	if (exp && exp->ex_stats)
-		percpu_counter_add(&exp->ex_stats->counter[EXP_STATS_IO_WRITE], amount);
+		percpu_counter_add_local(&exp->ex_stats->counter[EXP_STATS_IO_WRITE],
+					 amount);
 }
 
 static inline void nfsd_stats_payload_misses_inc(struct nfsd_net *nn)
-- 
2.54.0