Re: [PATCH v3 00/12] Improve the scalability of NFSD's classic DRC

["Chuck Lever" <cel@kernel.org>]

On Fri, Sep 11, 2026, at 7:20 PM, NeilBrown wrote:
> On Sat, 12 Sep 2026, Chuck Lever wrote:
>> On Fri, 11 Sep 2026, NeilBrown wrote:
>> > On Thu, 10 Sep 2026, Chuck Lever wrote:
>> > > 1. The transport reports delivery outright where it can. TCP now
>> > >    reports when snd_una passes the reply's sequence number, and RDMA
>> > >    reports on each Send completion.
>> >
>> > As you note in the relevant patch, the TCP ack can arrive before the NFS
>> > client has seen the reply.  But I don't think you drop the cached reply
>> > immediately, so that probably don't matter.
>> 
>> Right. The ACK marks the entry, and the next prune of that bucket
>> evicts it.
>> 
>> 
>> > > 2. Where no report will come, a later request on the same connection
>> > >    stands in as an implied ACK, since a client on a live TCP or RDMA
>> > >    connection does not retransmit within it (RFC 1813 Section 4.5).
>> >
>> > I don't follow this.  Why would no report come?  TCP will always send
>> > and ACK.
>> 
>> The cover letter was unclear. TCP always ACKs, but svc_tcp_sendto()
>> does not always track the reply. The per-socket ring is fixed-size,
>> so a burst overflows it. A kTLS send that leaves part of the record
>> unsent is not recorded, and neither is a failed send. Each of those
>> is reported as untracked right away, so the DRC entry has no report
>> to wait for.
>
> I don't see the need for the per-socket ring.
> When a reply is sent - record the TCP sequence number in the DRC.
> When pruning the DRC, get the TCP ack number first and compare it
> against sequence numbers.
> Do this often enough that that entries are pruned before the sequence
> wraps past them.
>
> kTLS is clearly more subtle but there must be some way record an
> approximate seq number for an incomplete send ..  maybe current seq +
> header-size + queued size ??
>
> Unsent replies will never get an ack, but presumably the client will
> resend, be answered using the cached reply, and we can then get an ack
> on the new connection.  I think it would be wrong to drop an unsent
> reply before the timeout, unless it gets re-sent and transport-acked
> before then.
>
> I think the opaque cookie aspect of the design is a mistake.  Everything
> is fully sequenced over a network connection so a sequence number
> (clearly paired with xpt_id) makes more sense...
> Or does RDMA not sequence replies?  If so then it can only ack a single
> reply at a time, not a range of replies.  But that needn't be a barrier.
>
> When a message is transmitted the transport hands a seq-number (and
> xpt_id) to the cache.  When it gets an ack it hands a range of
> seq-numbers to the cache.  For a transport that had out-of-order acks,
> this range would always be of length one (that transport might need a
> private atomic_t to generate the sequence numbers).  For a transport
> that batches acks (like TCP) the range could be much longer.
>
> Also, I wonder if the xpt_id should be 64bit assigned sequentially with no
> xa keeping track of used one.  64bits never wraps.
>
> I think that with this design the implied ACK would add no value.

Thanks, all makes sense to me and reduces code complexity.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)