Re: [PATCH] nfsd: bound GSS session fore-channel slots to the GSS replay window

["Chuck Lever" <cel@kernel.org>]

On Thu, Jun 18, 2026, at 9:35 PM, Vjaceslavs Klimovs wrote:
> An RPCSEC_GSS context authenticates each RPC with a per-context
> sequence number, and the server enforces a fixed 128-entry
> replay/reorder window per context (GSS_SEQ_WIN,
> net/sunrpc/auth_gss/svcauth_gss.c).  A client mounting with nconnect
>> 1 round-robins a single context's sequence numbers across several
> TCP connections that drain at different rates, and every retransmit
> consumes a fresh sequence number.  The spread between the oldest
> in-flight and the newest issued sequence number can therefore exceed
> the number of in-flight RPCs; once it passes the window the server
> drops the lagging request, the hard mount retransmits, and a krb5p
> streaming write collapses into a retransmit storm (a stall to ~0 in
> the field; an EIO under the krb5-nfs-perf reproducer).
>
> The NFSv4.1 fore-channel slot count caps how many RPCs the client
> keeps in flight, so it is the effective ceiling on the
> sequence-number spread.  The default Linux client negotiates 64
> fore-channel slots (NFS4_DEF_SLOT_TABLE_SIZE), and commit
> 60aa6564317d ("nfsd: allocate new session-based DRC slots on demand.")
> lets the server grow the table far beyond that -- both exceed the
> 128-entry window once nconnect reordering and retransmits are
> accounted for.
>
> Bound a GSS-authenticated session's fore-channel slots to a quarter
> of the window, at both the initial CREATE_SESSION grant and on-demand
> growth, so the sequence-number spread stays inside the replay window.
> A quarter (32) keeps a 4x margin against overrun yet still allows
> enough in-flight RPCs to keep the per-byte krb5p crypto (AES plus
> HMAC) saturated across many server CPUs, so it does not cap
> streaming-write throughput on large machines.  sec=sys has no
> per-context sequence window and keeps the full
> NFSD_MAX_SLOTS_PER_SESSION table.
>
> Signed-off-by: Vjaceslavs Klimovs <vklimovs@gmail.com>
> ---
>
> Symptom / impact
>   A production krb5p NFSv4.2 mount (vers=4.2, rsize=wsize=1M, proto=tcp,
>   nconnect=8, sec=krb5p, enctype aes256-cts-hmac-sha384-192) that used to
>   sustain ~1 GB/s on a 10 GbE link began stalling to ~0 after a 6.12 -> 6.18
>   upgrade.  sec=sys over the same path is unaffected, and nconnect=1 does not
>   trigger it.
>
> Bisect
>   Onset is 60aa6564317d ("nfsd: allocate new session-based DRC slots on
>   demand.", v6.14).  Its parent caps the session slot table; the commit lets
>   it grow on demand, so in-flight RPCs (already 64 from the client default)
>   climb past the 128-entry GSS window and storm.
>
> Reproduction / validation
>   Two-VM QEMU harness (one VM a KDC + nfsd server, the other a krb5p client;
>   nconnect=8; sequential write to a tmpfs-backed export):
>     unpatched 6.18.x : retransmit storm (hundreds of retransmits) -> EIO
>     patched          : clean, retrans 0
>   Confirmed at both 2 and 16 vCPU per VM.
>
> Why the cap is 32 (and why a server-side cap at all)
>   The krb5p throughput ceiling here is GSS crypto, not the slot count: under
>   load ~7-9 cores sit ~99% in-kernel on both ends (AES-NI plus HMAC-SHA384
>   over every byte), and throughput scales with cores (~0.6 GB/s at 2 vCPU ->
>   ~2.0 GB/s at 16).  The slot count is a *secondary* lever, because it caps
>   how many RPCs are in flight to feed that parallel crypto.  Measured at
>   16 vCPU, varying only the cap:
>     16 slots -> ~1.7 GB/s    32 -> ~2.0 GB/s    40 -> ~2.15 GB/s    sys -> ~2.3
>   40 starts to graze the window again; 32 was the largest value that stayed
>   clean (retrans 0) across the sweep, so it keeps a 4x window margin without
>   throttling streaming writes on many-core servers.  (16 only looked "free"
>   at a 2-vCPU test point, where cores -- not slots -- were the bottleneck.)
>
> Possible client-side follow-ups (not addressed by this patch)
>   This is a server-side defensive bound.  Two client behaviors look worth a
>   separate look:
>    - the default of 64 GSS fore-channel slots already over-commits a single
>      128-entry context window once spread across nconnect links; and
>    - the v6.16 RFC2203 seqno cache (08d6ee6d8a10 "sunrpc: implement rfc2203
>      rpcsec_gss seqnum cache", plus follow-ups) -- the intended mitigation
>      for this storm -- in our testing converts a recoverable window graze
>      into a hard EIO at the 64-slot floor rather than recovering.  Cc Nikhil
>      Jha, who wrote it for the same workload.
>
>  fs/nfsd/nfs4state.c | 4 ++++
>  fs/nfsd/state.h     | 8 ++++++++
>  2 files changed, 12 insertions(+)
>
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index c5dba49c9035..f077f1b8a780 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -3998,6 +3998,9 @@ nfsd4_create_session(struct svc_rqst *rqstp,
>         status = check_forechannel_attrs(&cr_ses->fore_channel, nn);
>         if (status)
>                 return status;
> +       if (rqstp->rq_authop->flavour == RPC_AUTH_GSS &&
> +           cr_ses->fore_channel.maxreqs > NFSD_GSS_MAX_SLOTS_PER_SESSION)
> +               cr_ses->fore_channel.maxreqs = NFSD_GSS_MAX_SLOTS_PER_SESSION;
>         status = check_backchannel_attrs(&cr_ses->back_channel);
>         if (status)
>                 goto out_err;
> @@ -4511,6 +4514,7 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct
> nfsd4_compound_state *cstate,
>          */
>         if (seq->slotid == session->se_fchannel.maxreqs - 1 &&
>             session->se_target_maxslots >= session->se_fchannel.maxreqs &&
> +           rqstp->rq_authop->flavour != RPC_AUTH_GSS &&
>             session->se_fchannel.maxreqs < NFSD_MAX_SLOTS_PER_SESSION) {
>                 int s = session->se_fchannel.maxreqs;
>                 int cnt = DIV_ROUND_UP(s, 5);
> diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
> index aaf513ed9104..4bec489ed26b 100644
> --- a/fs/nfsd/state.h
> +++ b/fs/nfsd/state.h
> @@ -278,6 +278,14 @@ static inline struct nfs4_delegation
> *delegstateid(struct nfs4_stid *s)
>   * A large number can be needed to get good throughput on high-latency servers.
>   */
>  #define NFSD_MAX_SLOTS_PER_SESSION     2048
> +/*
> + * Bound GSS sessions to a quarter of the 128-entry RPCSEC_GSS replay window
> + * (GSS_SEQ_WIN): nconnect spreads one context seqno space across links and
> + * each retransmit burns another, so cap both the initial grant and growth.
> + * 32 keeps a 4x margin against window overrun while still allowing enough
> + * in-flight RPCs to saturate the per-byte GSS crypto across many CPUs.
> + */
> +#define NFSD_GSS_MAX_SLOTS_PER_SESSION 32
>  /* Maximum  session per slot cache size */
>  #define NFSD_SLOT_CACHE_SIZE           2048
>  /* Maximum number of NFSD_SLOT_CACHE_SIZE slots per session */

First, a few process-related notes.

- The diff body is mangled and cannot be applied with the usual
  automated tool chains. You should consult the material under
  Documentation/process/ to see how to configure your email
  client so that patch content is preserved when you post.

- There is already a set of fixes in nfsd-testing that hopes
  to manage session slot table growth better, though it does
  not directly target GSS request reordering.

- The patch applies to linux-6.18.y, not to upstream, and not
  to nfsd-testing. So I'm assuming you are looking for review
  comments rather than asking for this to be merged.

Now, second, why I'm responding: krb5p request reordering is a
well-known issue, and the nconnect configuration you are using
is common. The combination of these two settings is going to
make things worse, so it deserves some discussion.

But to be blunt, the proposed approach is an operational hack,
not a long-term solution that closes the re-ordering window.

- krb5i is also affected, but since it's not as compute-
  intensive, it's not as frequent a problem.

- Other NFS versions are affected when nconnect > 1, but they
  don't have a session slot table so the proposed solution
  can't help them.

- Even when nconnect == 1 there is a non-zero chance that
  one request is delayed until it falls outside the GSS
  sequence number window. (More on this below)

- The cap should be relative to the GSS window size, not
  a fixed constant. It could be set at CREATE_SESSION
  time, but the server doesn't know how many transport
  connections the client might make at that time.

- The selection of 32 is arbitrary. My own studies of the
  reordering issue shows that there is no limit (other than
  1) that can make re-ordering impossible. (My testing
  increased the GSS window instead of limiting it).

The reason for that is structural: the GSS window bounds
relative lag, not concurrency, but the NFSv4.1 session slot
count controls concurrency.

Those are different quantities, which is why no ratio can
close that gap and prevent reordering.

The server window in svcauth_gss is a sliding bound: it
tracks sd_max, the highest GSS seqno it has accepted, and
rejects anything below (sd_max - GSS_SEQ_WIN). sd_max
advances every time any request makes forward progress. So
the quantity that kills a request is how far its seqno
has fallen behind the leading edge (its lag), not how many
requests are in flight.

Now hold one request still (slow link, queued behind a busy
nfsd thread, a dropped segment awaiting retransmit) and let
the other M-1 slots keep cycling. Each reuse issues a fresh,
higher seqno; the server accepts them; sd_max climbs. The
stalled request's seqno doesn't move. Its lag grows without
bound for as long as it stays stalled... until it crosses
128 and is dropped. This happens for any M ≥ 2. The only
genuinely safe value is M = 1 (strict serialization, lag is
always 0), which is useless. So there is no useful ratio,
exactly as you found.

What M actually buys is the rate the leading edge advances:
with M-1 slots cycling at ~one completion per RTT, the edge
moves ~(M-1) seqnos per RTT, so a stalled request survives
~128/(M-1) RTTs before it's fatal.

  Smaller M = slower edge = longer grace period =
     lower steady-state spread under systematic reorder.

That's all the 32 does: it widens the grace window and lowers
the typical spread enough that drops become rare under the
reproducer's reorder pattern. It is a probability knob, not
an invariant. "4x margin" is margin against the spread the
reproducer happens to produce, not against reordering.

---

Now I'm speaking generally here, not proposing a specific
fix.

Since lag is the controlled variable, only something that
bounds lag directly fixes it:

- Client-side spread throttling: the client knows the
  oldest unacked seqno and the newest issued; it can
  refuse to issue a seqno more than (GSS_SEQ_WIN - margin)
  ahead of the oldest outstanding. That bounds lag at the
  source regardless of nconnect or slot count. This is the
  structurally correct fix, and it's on the side that has
  the information.

- Per-connection (or per-slot) GSS contexts: give each
  transport its own seqno space so connections draining at
  different rates don't share one window. Removes the
  cross-connection reorder entirely; costs more context
  state and a spec/implementation change.

- Negotiated window size: make GSS_SEQ_WIN scale with the
  advertised slot count rather than being a fixed 128. The
  server can't shrink replay protection safely, but a
  larger window for high-slot sessions trades memory for
  headroom. Still not a guarantee (lag is unbounded), but it
  moves the knob to where the slots actually are.

The server slot cap is defensible as a stopgap the server
might deploy unilaterally without trusting clients... but
it's treating the symptom (spread) one step removed from
the cause (unbounded lag, controllable only where seqnos are
issued).


-- 
Chuck Lever