[PATCH 3/5] NFSD: Eliminate percpu counter contention in reply cache statistics

[Chuck Lever <cel@kernel.org>]

Each RPC passes through nfsd_cache_lookup(), which increments one
of nfsd_stats_rc_hits_inc(), nfsd_stats_rc_misses_inc(), or
nfsd_stats_rc_nocache_inc(). These helpers update
per-net-namespace percpu_counters with percpu_counter_inc(),
which applies the default batch threshold of max(32, 2*nr_cpus).

Once a CPU's local delta reaches that threshold, the update folds
into the shared counter under its global spinlock. On a busy
multi-CPU server this produces lock traffic on a counter cacheline
shared across all CPUs, growing with the request rate.

Switch to percpu_counter_add_local(fbc, 1), which batches with
INT_MAX so that increments always remain on the per-CPU fast path.
This matches the treatment already applied to the IO byte and DRC
memory counters. All readers of these counters use
percpu_counter_sum_positive(), which sums the per-CPU deltas under
the global lock, so read accuracy is unaffected.

Signed-off-by: Chuck Lever <cel@kernel.org>
---
 fs/nfsd/stats.h | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/fs/nfsd/stats.h b/fs/nfsd/stats.h
index 15d30c045dc3..598ea45d1722 100644
--- a/fs/nfsd/stats.h
+++ b/fs/nfsd/stats.h
@@ -16,19 +16,39 @@
 struct proc_dir_entry *nfsd_proc_stat_init(struct net *net);
 void nfsd_proc_stat_shutdown(struct net *net);
 
+/**
+ * nfsd_stats_rc_hits_inc - Count a duplicate reply cache hit
+ * @nn: target network namespace
+ *
+ * These reply cache counters are updated once per RPC. Readers use
+ * percpu_counter_sum_positive(), so local batching does not affect
+ * read accuracy.
+ */
 static inline void nfsd_stats_rc_hits_inc(struct nfsd_net *nn)
 {
-	percpu_counter_inc(&nn->counter[NFSD_STATS_RC_HITS]);
+	percpu_counter_add_local(&nn->counter[NFSD_STATS_RC_HITS], 1);
 }
 
+/**
+ * nfsd_stats_rc_misses_inc - Count a duplicate reply cache miss
+ * @nn: target network namespace
+ *
+ * See nfsd_stats_rc_hits_inc() for batching rationale.
+ */
 static inline void nfsd_stats_rc_misses_inc(struct nfsd_net *nn)
 {
-	percpu_counter_inc(&nn->counter[NFSD_STATS_RC_MISSES]);
+	percpu_counter_add_local(&nn->counter[NFSD_STATS_RC_MISSES], 1);
 }
 
+/**
+ * nfsd_stats_rc_nocache_inc - Count a request not cached in the reply cache
+ * @nn: target network namespace
+ *
+ * See nfsd_stats_rc_hits_inc() for batching rationale.
+ */
 static inline void nfsd_stats_rc_nocache_inc(struct nfsd_net *nn)
 {
-	percpu_counter_inc(&nn->counter[NFSD_STATS_RC_NOCACHE]);
+	percpu_counter_add_local(&nn->counter[NFSD_STATS_RC_NOCACHE], 1);
 }
 
 static inline void nfsd_stats_fh_stale_inc(struct nfsd_net *nn,
-- 
2.54.0