Question: net/mlx5e kTLS TX TIS reuse vs in-flight WQEs

Question: net/mlx5e kTLS TX TIS reuse vs in-flight WQEs

[Rishikesh Jethwani]

Hi Tariq,

Could a DUMP/SEND WQE from an old owner on SQ A read or update TIS
state after SQ B has reinitialised that TIS for a new owner?

The existing TIS pool predates live rekey. My TLS 1.3 TX KeyUpdate
series (v17 [1]; v18 in preparation) calls tls_dev_del() and then
tls_dev_add() on a live socket, and the question arises through this
sequence:

  1. A spurious RTO retransmit of old-key data is posted on SQ A.
  2. While that data remains unacknowledged, setsockopt(TLS_TX) starts
     rekey and installs the rekey validator, which stops admitting new
     plaintext to hardware offload. An ACK covering all remaining
     old-key data marks rekey ready before the driver observes the
     retransmit's SQ completion. A subsequent sendmsg() enters rekey
     completion.
  3. Rekey completion calls synchronize_net(), which waits for CPU
     transmit paths, not SQ completions, so a WQE on a stalled or
     backlogged SQ A can still be pending. mlx5e_ktls_del_tx() then
     releases the DEK and immediately calls pool_push(). DEK
     reclamation uses batched SYNC_CRYPTO; returning the TIS does not
     wait for synchronization covering its old DEK.
  4. Before re-adding the offload, TLS allocates the new SW-fallback
     AEAD. In that interval another connection's mlx5e_ktls_add_tx()
     can pop the returned TIS and use it on SQ B. The same socket can
     also reach SQ B on re-add: with the retransmit queue empty, TCP
     sets ooo_okay even while the retransmitted skb still holds
     sk_wmem_alloc, so queue selection may move the flow.
     mlx5e_ktls_handle_tx_skb() posts the new owner's initial
     parameters with skip_static_post=false, fence_first_post=false:
     static parameters unfenced, progress parameters fenced, then a
     fence NOP.

An unobserved completion does not by itself mean the WQE still accesses
the TIS; that is the hardware ordering question. The same question
applies on SQ A: resync fences its first parameter update, initial
setup does not, and I do not know what either guarantees against
earlier WQEs that reference the TIS.

The normal close path has additional ordering. tls_dev_del() runs
from work queued by tls_device_sk_destruct(), which itself runs from
__sk_free(), i.e. only once sk_wmem_alloc reaches zero. For
transmitted mlx5 skbs, SQ completion calls napi_consume_skb(), which
invokes tcp_wfree() to release that accounting, and parameter and DUMP
WQEs preceding a posted SEND on the same SQ are retired by that SEND's
completion. For successful transmissions, this gives close-time
ordering without an explicit drain in del_tx. WQEs posted without a
subsequent SEND, and SQ error/reset paths, need separate lifetime
guarantees. Live rekey does not wait for socket destruction.

Was the recycling fence relying on this socket-lifetime ordering, or
does it independently protect against outstanding work on another SQ?
c4dfe704f53f ("net/mlx5e: kTLS, Recycle objects of device-offloaded
TLS TX connections") says the fence ensures context modification does
not interfere with existing traffic. Does that cover old SEND, DUMP
and parameter WQEs on another SQ, including work the device has not
yet executed?

If reuse must wait, would retaining the TIS and DEK until all
referencing work has completed or been definitively stopped suffice,
including during SQ error/reset?

I have not reproduced a failure; the reuse is visible in the driver,
but I have not established the hardware ordering guarantee. A PRM
reference, or a statement of which completion must precede reuse,
would settle it.

Thanks,
Rishikesh

[1] https://lore.kernel.org/netdev/20260917224355.2288021-1-rjethwani@purestorage.com/