[PATCH net-next v17 15/15] tls: document TLS 1.3 hardware offload rekey handling

[Rishikesh Jethwani <rjethwani@purestorage.com>]

Document TLS 1.3 support in the hardware offload path.

In Documentation/networking/tls-offload.rst, replace the stale note that
offload does not support TLS 1.3 with a description of the KeyUpdate
handling on both TX and RX, including how old-key records in flight are
bridged in software: each record is classified relative to the boundary at
which the NIC stopped using the old key, and partially transformed (mixed)
records are re-encrypted with the old key. In a mixed record the fragments
the NIC transformed but could not authenticate carry skb->decrypt_failed; a
non-mixed record with that flag was not transformed and is decrypted
directly.

In Documentation/networking/tls.rst, add the per-namespace rekey counters:
TlsCurrTxRekey/TlsCurrRxRekey for sessions currently in a deferred rekey,
TlsTxRekeyFallback/TlsRxRekeyFallback for rekeys that could not be
offloaded and fell back to software cryptography, and
TlsTxRekeyAborted/TlsRxRekeyAborted for deferred rekeys still pending when
the socket was destroyed.

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 Documentation/networking/tls-offload.rst | 175 +++++++++++++++++++++--
 Documentation/networking/tls.rst         |  17 +++
 2 files changed, 183 insertions(+), 9 deletions(-)

diff --git a/Documentation/networking/tls-offload.rst b/Documentation/networking/tls-offload.rst
index e5802bcd4d22..cdf84f4b817a 100644
--- a/Documentation/networking/tls-offload.rst
+++ b/Documentation/networking/tls-offload.rst
@@ -99,9 +99,8 @@ at the end of kernel structures (see :c:member:`driver_state` members
 in ``include/net/tls.h``) to avoid additional allocations and pointer
 dereferences.
 
-When the offloaded connection is destroyed the core calls
-the :c:member:`tls_dev_del` callback so the driver can release per-direction
-state:
+The core calls the :c:member:`tls_dev_del` callback so the driver can release
+per-direction state:
 
 .. code-block:: c
 
@@ -109,7 +108,14 @@ state:
 			    struct tls_context *ctx,
 			    enum tls_offload_ctx_dir direction);
 
-``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is provided.
+``tls_dev_del`` is called either when the offloaded connection is destroyed or,
+for a TLS 1.3 connection, when the old key is retired during a rekey (see the
+`Rekey`_ section). It operates on a single ``direction``, so the driver must
+release only the state for that direction and must not free state shared
+between directions or the socket as a whole. After a rekey ``tls_dev_del``,
+``tls_dev_add`` may be called again for the same socket and direction to
+install the new key. ``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is
+provided.
 
 The third TLS device callback is :c:member:`tls_dev_resync`, called by the core
 to synchronize the TCP stream with the record boundaries:
@@ -205,7 +211,10 @@ Upon reception of a TLS offloaded packet, the driver sets
 the :c:member:`decrypted` mark in :c:type:`struct sk_buff <sk_buff>`
 corresponding to the segment. Networking stack makes sure decrypted
 and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer)
-and takes care of partial decryption.
+and takes care of partial decryption. A segment the device processed but
+could not authenticate may instead carry the :c:member:`decrypt_failed`
+mark; see the `Error handling`_ section for what the mark implies about
+the payload.
 
 Resync handling
 ===============
@@ -404,8 +413,121 @@ records, then after 4 records, after 8, after 16... up until every
 Rekey
 =====
 
-Offload does not currently support TLS 1.3, therefore key rotation
-is not a concern for offloaded connections at this point.
+TLS 1.3 allows traffic keys to be updated mid-connection using the
+KeyUpdate message. Offloaded TLS 1.3 connections must therefore switch
+keys without tearing down the offload. The device cannot simply be given
+the new key because records encrypted (TX) or transformed (RX) with the
+old key may still be in flight. The stack retains the necessary old-key
+state and bridges the transition in software.
+
+TX
+--
+
+On TX, the new key is installed in a temporary software context, and
+sendmsg is routed through the software path. If no hardware-offloaded
+records remain unacknowledged, the switch completes inline during
+setsockopt. Otherwise the rekey is left pending and is completed later,
+on the sender's next ``sendmsg()`` after all old-key records have been
+ACKed (see `Completing a deferred rekey`_). Completion calls
+:c:func:`tls_dev_del` for the old key and reinstalls hardware offload
+with the new key at the current TCP write sequence. If reinstallation
+fails, the connection keeps encrypting in software with the new key; the
+next KeyUpdate re-arms the transition and retries the hardware
+installation.
+
+Unlike the software path, a ``TLS_TX`` setsockopt on an offloaded
+connection first flushes the open and partially sent hardware records to
+TCP before installing the new key. It therefore behaves like a blocking
+``send()`` of that record: it may wait for send buffer space (bounded by
+``SO_SNDTIMEO``), and on a non-blocking socket it fails with ``-EAGAIN``
+and must be retried once the socket is writable. The new key is not
+installed until the call succeeds; the connection keeps using the old key
+in the meantime.
+
+Completing a deferred rekey
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A deferred rekey is completed by the sender, not by the ACK path. When
+the last old-key record is acknowledged the stack only marks the rekey
+as ready; the device is not touched. The switch itself,
+:c:func:`tls_dev_del` of the old key followed by :c:func:`tls_dev_add`
+of the new one, runs at the start of the next ``sendmsg()`` on the
+socket, and that ``sendmsg()`` is the first to be encrypted by hardware
+again. No other event completes it: ``splice_eof()``, write-space
+wakeups, retransmissions and pure ACKs all leave the connection on the
+software path.
+
+This is intentional. Completion has to flush the software context's
+open record to TCP and may sleep for send buffer space, which rules out
+the ACK and write-space paths. Beyond that, the stack only switches when
+it has new data to hand to the device: the software path is fully
+correct with the new key, so deferring the switch costs host CPU but
+nothing else, and it keeps the device from being programmed for a
+connection that may never send again.
+
+Two consequences follow. A connection that stops sending after a
+KeyUpdate stays in the deferred state until it is closed: it is
+encrypted in software with the new key, it is counted in
+``TlsCurrTxRekey``, and at close it is reported as
+``TlsTxRekeyAborted``. That counter therefore includes senders that
+simply had nothing more to send, not only sockets torn down
+mid-transition, and is not by itself an error indication. And the return
+to hardware is delayed by at least one ACK round trip after the last
+old-key record, plus however long the application waits before its next
+``sendmsg()``. A sender that wants the hardware path back promptly can
+issue a small ``sendmsg()`` once its old data has been acknowledged.
+
+Completion can fail transiently or permanently. If the software flush
+cannot get send buffer space (``-EAGAIN``, or a signal on a blocking
+socket) the rekey stays pending, the ``sendmsg()`` proceeds in software,
+and the next ``sendmsg()`` retries; the ``tls_device_complete_rekey_retry``
+tracepoint fires. A hard failure (:c:func:`tls_dev_add` rejected, or the
+netdev gone) is terminal for this KeyUpdate: the connection is pinned to
+software encryption with the new key, counted in ``TlsTxRekeyFallback``
+and moved from ``TlsCurrTxDevice`` to ``TlsCurrTxSw``; the
+``tls_device_complete_rekey_fail`` tracepoint fires. The next ``TLS_TX``
+setsockopt re-arms the transition and retries.
+
+The decision to defer is taken at the start of the ``TLS_TX``
+setsockopt, before the open hardware record is flushed to TCP. That
+flush may block for send buffer space, and old-key records acknowledged
+while it sleeps do not change the decision: the rekey is still deferred
+and completes on a following ``sendmsg()`` rather than inline. This is
+conservative, not a correctness issue. The boundary is fixed at the
+write sequence after the flush, so the acknowledgment of the flushed
+record itself arms completion; the cost is one more ACK round trip and
+one more ``sendmsg()``. Applications should not expect an inline switch
+whenever the socket has unacknowledged data at the time of the
+setsockopt.
+
+RX
+--
+
+On RX, the NIC may already have transformed in-flight records with the
+old key before the peer's KeyUpdate is parsed. When the KeyUpdate is
+decoded, the stack removes the old key from the NIC but retains the old
+AEAD, IV, and record sequence in the software offload context.
+
+Each record is classified by the TCP sequence of its first byte relative
+to the boundary at which the NIC stopped using the old key. Records
+starting after that boundary carry new-key wire encryption, so the old
+software AEAD state can be released. Records before the boundary that
+remain fully encrypted are passed to the software path. Records that
+were partially transformed by the NIC are re-encrypted with the old key
+to restore the new-key ciphertext, allowing the software AEAD to decrypt
+them with the new key.
+
+If old-key records are still queued, installation of the new key through
+:c:func:`tls_dev_add` is deferred until those records have been consumed;
+otherwise it occurs immediately. When the NIC cannot authenticate a record
+processed during the transition, the affected fragments are delivered with
+``skb->decrypt_failed`` set, following the contract described in the
+`Error handling`_ section. In a mixed record such a fragment was
+transformed (XORed) with the old key, and the re-encrypt path uses this to
+undo the transform on those fragments with the old key while leaving
+untouched fragments intact. A non-mixed record carrying
+``skb->decrypt_failed`` was not transformed; it is still wire ciphertext
+and is decrypted directly by the software AEAD under the new key.
 
 Error handling
 ==============
@@ -442,8 +564,43 @@ to the host's stack as it was on the wire (recovering original packet in the
 driver if device provides precise error is sufficient).
 
 The Linux networking stack does not provide a way of reporting per-packet
-decryption and authentication errors, packets with errors must simply not
-have the :c:member:`decrypted` mark set.
+decryption and authentication errors. A packet with errors must not have
+the :c:member:`decrypted` mark set. In addition, the driver may set the
+:c:member:`decrypt_failed` mark on a segment the device matched to an
+offloaded connection and processed but could not authenticate. The two
+marks are mutually exclusive.
+
+The stack interprets :c:member:`decrypt_failed` per record, relative to the
+:c:member:`decrypted` mark of the other segments making up the same record.
+Coalescing (GRO, socket layer) and record classification are keyed on
+:c:member:`decrypted` alone, so :c:member:`decrypt_failed` segments may be
+merged with unmarked ones. A driver setting the mark must therefore honour
+the following contract:
+
+ * In a record none of whose segments carry :c:member:`decrypted`, every
+   segment, including one with :c:member:`decrypt_failed` set, must hold
+   the payload exactly as it was on the wire. This is the general rule
+   above: if the device did not successfully decrypt any part of a record
+   it must hand the whole record over untouched. The stack passes such a
+   record to software decryption directly and does not consult
+   :c:member:`decrypt_failed`.
+
+ * In a record where some segments carry :c:member:`decrypted` (a mixed
+   record), a segment with :c:member:`decrypt_failed` set must hold payload
+   the device has already transformed (XORed with the cipher keystream) but
+   failed to authenticate, and a segment with neither mark must hold the
+   payload as it was on the wire. The stack re-encrypts the
+   :c:member:`decrypted` and :c:member:`decrypt_failed` segments to restore
+   the ciphertext, leaves the unmarked segments intact, and authenticates
+   the whole record in software.
+
+A transformed segment delivered without :c:member:`decrypt_failed`, or an
+untransformed segment of a mixed record delivered with it, is restored
+incorrectly and the record fails software authentication. A device which
+cannot tell the driver whether a failed segment was transformed must
+recover the original packet before handing it to the stack, as described
+above, and leave both marks clear. During a TLS 1.3 rekey the mark also
+tells the stack which key the device applied; see the `Rekey`_ section.
 
 A packet should also not be handled by the TLS offload if it contains
 incorrect checksums.
diff --git a/Documentation/networking/tls.rst b/Documentation/networking/tls.rst
index 980c442d7161..cf05543260d8 100644
--- a/Documentation/networking/tls.rst
+++ b/Documentation/networking/tls.rst
@@ -314,6 +314,11 @@ TLS implementation exposes the following per-namespace statistics
   number of TX and RX sessions currently installed where NIC handles
   cryptography
 
+- ``TlsCurrTxRekey``, ``TlsCurrRxRekey`` -
+  number of TX and RX sessions currently undergoing a deferred rekey,
+  i.e. a rekey which could not be applied immediately and is waiting for
+  in-flight records to drain before the new key is installed in hardware
+
 - ``TlsTxSw``, ``TlsRxSw`` -
   number of TX and RX sessions opened with host cryptography
 
@@ -344,3 +349,15 @@ TLS implementation exposes the following per-namespace statistics
 - ``TlsRxRekeyReceived`` -
   number of received KeyUpdate handshake messages, requiring userspace
   to provide a new RX key
+
+- ``TlsTxRekeyFallback``, ``TlsRxRekeyFallback`` -
+  number of rekeys on existing sessions for TX and RX which could not be
+  offloaded to the NIC and fell back to software cryptography
+
+- ``TlsTxRekeyAborted``, ``TlsRxRekeyAborted`` -
+  number of deferred rekeys for TX and RX which were still pending when
+  the socket was destroyed, and so never completed. For TX hardware
+  offload this includes senders that sent nothing further after the
+  KeyUpdate, since the switch back to hardware only happens on
+  ``sendmsg()`` (see the Rekey section of
+  Documentation/networking/tls-offload.rst)
-- 
2.50.1