[PATCH RFC 0/4] nfsd: per-client fair-queue dispatch

[Benjamin Coddington <ben.coddington@hammerspace.com>]

knfsd dispatches ready transports from a single per-pool FIFO, so a
client's share of nfsd service scales with the number of connections it
holds rather than being shared per client.  A client that opens many
connections (nconnect, or a farm of data movers) starves other clients
on the same server purely by out-numbering them in sockets.

I measured this with a load generator that pins each request to a fixed
service time and does no filesystem work, so that nfsd thread-time is the
only scarce resource (8 threads, 10ms/op, ~648 ops/s pool ceiling).  A
greedy client opens K connections alongside one single-connection
interactive client.

NFSv3, dispatch as it is today:

  greedy K   greedy share   interactive ops/s
     1           50%              241
     4           80%              129
     8           89%               72
    16           94%               38

The interactive client's share tracks 1/(K+1) and its throughput falls
roughly 6x while it does nothing different.  NFSv4.1 behaves identically
(89% greedy at K=8) even when the greedy connections are bound to a
single session, because the dispatch decision is below the NFS version.

The same NFSv4.1 workload with fair queueing enabled:

  greedy K   greedy share   interactive ops/s
     8           72%              182
    16           73%              177
    32           70%              193

The greedy client's share no longer climbs with its connection count and
the interactive client recovers (72 -> 182 ops/s at K=8).  Aggregate
throughput is unchanged: the T/D pool ceiling is the same with fair
queueing on and off.  The split does not reach 50/50 because a single
interactive connection is bounded by its request window and by XPT_BUSY
serialising one transport; with a deeper window it reaches ~59/41.

The approach:

  - sunrpc grows an opaque per-transport fairness key (patch 1), with a
    default derived from the source address (the source port is excluded
    so a client's several connections share one key), and an opt-in
    per-pool scheduler that buckets ready transports by that key and
    dispatches round-robin across keys (patch 2).  When it is disabled,
    which is the default, the existing lockless FIFO path is unchanged.

  - nfsd gains a "fairq" module parameter to turn it on (patch 3) and
    stamps the NFSv4.1 clientid as the key when a connection binds to a
    session (patch 4), so all of a client's connections share one key.
    NFSv3 uses the source-address default.

This is an RFC; a few questions for the list:

  - Unit of fairness: clientid (used here) or session?  Earlier
    discussion leaned toward exploring per-session.

  - Mechanism: a fixed bucket hash under a per-pool spinlock taken only
    on the opt-in path, versus a lockless or per-flow structure.

  - Would a per-client in-flight cap be preferable to proportional fair
    queueing?

The measurement used a debug-only filehandle-latency hook that is not
part of this series.

Benjamin Coddington (4):
  sunrpc: add a per-transport fairness key to svc_xprt
  sunrpc: dispatch ready transports fairly per client
  nfsd: add a fairq module parameter
  nfsd: key NFSv4.1 connections by clientid for fair queueing

 fs/nfsd/nfs4state.c             |  17 +++
 fs/nfsd/nfssvc.c                |  19 +++
 include/linux/sunrpc/svc.h      |   5 +
 include/linux/sunrpc/svc_xprt.h |  46 ++++++-
 net/sunrpc/svc.c                |   2 +
 net/sunrpc/svc_xprt.c           | 216 +++++++++++++++++++++++++++++++-
 6 files changed, 302 insertions(+), 3 deletions(-)


base-commit: e7ca66ba17f1b5e4ecbb29b9c3c4a31aa062bed0
-- 
2.53.0