[PATCH net-next v16 00/10] tls: Add TLS 1.3 hardware offload support

[PATCH net-next v16 00/10] tls: Add TLS 1.3 hardware offload support

[Rishikesh Jethwani]

Hi all,

  This series adds TLS 1.3 hardware offload support including KeyUpdate
  (rekey) and a selftest for validation.

  Patch 1: Reject TLS 1.3 offload in chcr_ktls and nfp drivers
    These drivers only support TLS 1.2; add explicit version check.

  Patch 2: mlx5e TLS 1.3 hardware offload
    Add TLS 1.3 TX/RX offload on ConnectX-6 Dx and newer.
    Handle 12-byte IV format and TLS_1_3 context type.

  Patch 3: tls: reject rekey attempts on an existing HW-offloaded connection
    Prep for the following patch ("tls: add TLS 1.3 hardware offload
    support")

  Patch 4: tls: add TLS 1.3 hardware offload support
    Extend tls_device.c for TLS 1.3 record format (content type
    appended before tag). Handle TLS 1.3 IV construction in fallback.

  Patch 5: Split tls_set_sw_offload into init/finalize

  Patch 6: Prep helpers and refactors for HW offload KeyUpdate

  Patch 7: TX KeyUpdate support

  Patch 8: RX KeyUpdate support

  Patch 9: Tracepoints for RX KeyUpdate path

  Patch 10: Selftest for hardware offload

Rishikesh

Changes in v16:
  - Addressed review comments

Rishikesh Jethwani (10):
  net: tls: reject TLS 1.3 offload in chcr_ktls and nfp drivers
  net/mlx5e: add TLS 1.3 hardware offload support
  tls: reject rekey attempts on an existing HW-offloaded connection
  tls: add TLS 1.3 hardware offload support
  tls: split tls_set_sw_offload into init and finalize stages
  tls: prep helpers and refactors for HW offload KeyUpdate
  tls: device: add TX KeyUpdate support
  tls: device: add RX KeyUpdate support
  tls: device: add tracepoints for the KeyUpdate path
  selftests: net: add TLS hardware offload test

 MAINTAINERS                                   |   2 +
 .../chelsio/inline_crypto/ch_ktls/chcr_ktls.c |   3 +
 .../mellanox/mlx5/core/en_accel/ktls.c        |  10 +
 .../mellanox/mlx5/core/en_accel/ktls.h        |   8 +-
 .../mellanox/mlx5/core/en_accel/ktls_txrx.c   |  14 +-
 .../net/ethernet/netronome/nfp/crypto/tls.c   |   3 +
 include/net/tls.h                             | 104 +-
 include/uapi/linux/snmp.h                     |   6 +
 net/tls/tls.h                                 |  24 +-
 net/tls/tls_device.c                          | 955 +++++++++++++++--
 net/tls/tls_device_fallback.c                 |  82 +-
 net/tls/tls_main.c                            |  77 +-
 net/tls/tls_proc.c                            |   6 +
 net/tls/tls_sw.c                              | 167 ++-
 net/tls/trace.h                               |  98 ++
 .../selftests/drivers/net/hw/.gitignore       |   1 +
 .../testing/selftests/drivers/net/hw/Makefile |   2 +
 .../selftests/drivers/net/hw/tls_hw_offload.c | 975 ++++++++++++++++++
 .../drivers/net/hw/tls_hw_offload.py          | 295 ++++++
 19 files changed, 2613 insertions(+), 219 deletions(-)
 create mode 100644 tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
 create mode 100755 tools/testing/selftests/drivers/net/hw/tls_hw_offload.py

-- 
2.25.1

[PATCH v16 01/10] net: tls: reject TLS 1.3 offload in chcr_ktls and nfp drivers

[Rishikesh Jethwani]

These drivers only support TLS 1.2. Return early when TLS 1.3
is requested to prevent unsupported hardware offload attempts.

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c | 3 +++
 drivers/net/ethernet/netronome/nfp/crypto/tls.c                | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
index f5acd4be1e69..29e108ce6764 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
+++ b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
@@ -431,6 +431,9 @@ static int chcr_ktls_dev_add(struct net_device *netdev, struct sock *sk,
 	atomic64_inc(&port_stats->ktls_tx_connection_open);
 	u_ctx = adap->uld[CXGB4_ULD_KTLS].handle;
 
+	if (crypto_info->version != TLS_1_2_VERSION)
+		goto out;
+
 	if (direction == TLS_OFFLOAD_CTX_DIR_RX) {
 		pr_err("not expecting for RX direction\n");
 		goto out;
diff --git a/drivers/net/ethernet/netronome/nfp/crypto/tls.c b/drivers/net/ethernet/netronome/nfp/crypto/tls.c
index 9983d7aa2b9c..13864c6a55dc 100644
--- a/drivers/net/ethernet/netronome/nfp/crypto/tls.c
+++ b/drivers/net/ethernet/netronome/nfp/crypto/tls.c
@@ -287,6 +287,9 @@ nfp_net_tls_add(struct net_device *netdev, struct sock *sk,
 	BUILD_BUG_ON(offsetof(struct nfp_net_tls_offload_ctx, rx_end) >
 		     TLS_DRIVER_STATE_SIZE_RX);
 
+	if (crypto_info->version != TLS_1_2_VERSION)
+		return -EOPNOTSUPP;
+
 	if (!nfp_net_cipher_supported(nn, crypto_info->cipher_type, direction))
 		return -EOPNOTSUPP;
 
-- 
2.25.1

[PATCH v16 02/10] net/mlx5e: add TLS 1.3 hardware offload support

[Rishikesh Jethwani]

Enable TLS 1.3 TX/RX hardware offload on ConnectX-6 Dx and newer
crypto-enabled adapters.
Key changes:
- Add TLS 1.3 capability checking and version validation
- Use MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3 (0x3) for crypto context
- Handle TLS 1.3 IV format: full 12-byte IV copied to gcm_iv +
  implicit_iv (vs TLS 1.2's 4-byte salt only)

Tested with TLS 1.3 AES-GCM-128 and AES-GCM-256 cipher suites.

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
Tested-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
 .../ethernet/mellanox/mlx5/core/en_accel/ktls.h    |  8 +++++++-
 .../mellanox/mlx5/core/en_accel/ktls_txrx.c        | 14 +++++++++++---
 2 files changed, 18 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
index 07a04a142a2e..0469ca6a0762 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
@@ -30,7 +30,9 @@ static inline bool mlx5e_is_ktls_device(struct mlx5_core_dev *mdev)
 		return false;
 
 	return (MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_128) ||
-		MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256));
+		MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256) ||
+		MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_128) ||
+		MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_256));
 }
 
 static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
@@ -40,10 +42,14 @@ static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
 	case TLS_CIPHER_AES_GCM_128:
 		if (crypto_info->version == TLS_1_2_VERSION)
 			return MLX5_CAP_TLS(mdev,  tls_1_2_aes_gcm_128);
+		else if (crypto_info->version == TLS_1_3_VERSION)
+			return MLX5_CAP_TLS(mdev,  tls_1_3_aes_gcm_128);
 		break;
 	case TLS_CIPHER_AES_GCM_256:
 		if (crypto_info->version == TLS_1_2_VERSION)
 			return MLX5_CAP_TLS(mdev,  tls_1_2_aes_gcm_256);
+		else if (crypto_info->version == TLS_1_3_VERSION)
+			return MLX5_CAP_TLS(mdev,  tls_1_3_aes_gcm_256);
 		break;
 	}
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
index 570a912dd6fa..f3f1be1d4034 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
@@ -6,6 +6,7 @@
 
 enum {
 	MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 = 0x2,
+	MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3 = 0x3,
 };
 
 enum {
@@ -15,8 +16,10 @@ enum {
 #define EXTRACT_INFO_FIELDS do { \
 	salt    = info->salt;    \
 	rec_seq = info->rec_seq; \
+	iv      = info->iv;      \
 	salt_sz    = sizeof(info->salt);    \
 	rec_seq_sz = sizeof(info->rec_seq); \
+	iv_sz      = sizeof(info->iv);      \
 } while (0)
 
 static void
@@ -24,9 +27,9 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
 		   union mlx5e_crypto_info *crypto_info,
 		   u32 key_id, u32 resync_tcp_sn)
 {
+	u16 salt_sz, rec_seq_sz, iv_sz;
+	char *salt, *rec_seq, *iv;
 	char *initial_rn, *gcm_iv;
-	u16 salt_sz, rec_seq_sz;
-	char *salt, *rec_seq;
 	u8 tls_version;
 	u8 *ctx;
 
@@ -59,7 +62,12 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
 	memcpy(gcm_iv,      salt,    salt_sz);
 	memcpy(initial_rn,  rec_seq, rec_seq_sz);
 
-	tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+	if (crypto_info->crypto_info.version == TLS_1_3_VERSION) {
+		memcpy(gcm_iv + salt_sz, iv, iv_sz);
+		tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3;
+	} else {
+		tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+	}
 
 	MLX5_SET(tls_static_params, ctx, tls_version, tls_version);
 	MLX5_SET(tls_static_params, ctx, const_1, 1);
-- 
2.25.1

[PATCH v16 03/10] tls: reject rekey attempts on an existing HW-offloaded connection

[Rishikesh Jethwani]

On a TLS 1.3 rekey, do_tls_setsockopt_conf() must not dispatch to
tls_set_device_offload{,_rx}(): reject the rekey with -EOPNOTSUPP for a
HW-offloaded connection (HW KeyUpdate is not supported yet, and we must
not fall back to software mid-connection), and for a SW-offloaded one
re-init the software crypto state via tls_set_sw_offload() directly.

Prep for the following patch ("tls: add TLS 1.3 hardware offload
support"), which drops the TLS_1_2_VERSION guards in those helpers.
Those guards currently also reject a (TLS 1.3) rekey; once gone, a rekey
reaching tls_set_device_offload_rx(),which has no other early guard,
would set up a brand-new device offload instead of a key update.

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 net/tls/tls_main.c | 85 +++++++++++++++++++++++++++-------------------
 1 file changed, 50 insertions(+), 35 deletions(-)

diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index fbb274287aa5..a62e94508c8b 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -710,49 +710,64 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
 	}
 
 	if (tx) {
-		rc = tls_set_device_offload(sk);
-		conf = TLS_HW;
-		if (!rc) {
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
-		} else {
-			rc = tls_set_sw_offload(sk, 1,
-						update ? crypto_info : NULL);
-			if (rc)
-				goto err_crypto_info;
-
-			if (update) {
-				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
-			} else {
-				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXSW);
-				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+		if (update && ctx->tx_conf == TLS_HW) {
+			rc = -EOPNOTSUPP;
+			goto err_crypto_info;
+		}
+
+		if (!update) {
+			rc = tls_set_device_offload(sk);
+			conf = TLS_HW;
+			if (!rc) {
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+				goto out;
 			}
-			conf = TLS_SW;
 		}
-	} else {
-		rc = tls_set_device_offload_rx(sk, ctx);
-		conf = TLS_HW;
-		if (!rc) {
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+
+		rc = tls_set_sw_offload(sk, 1, update ? crypto_info : NULL);
+		if (rc)
+			goto err_crypto_info;
+
+		if (update) {
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
 		} else {
-			rc = tls_set_sw_offload(sk, 0,
-						update ? crypto_info : NULL);
-			if (rc)
-				goto err_crypto_info;
-
-			if (update) {
-				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
-			} else {
-				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXSW);
-				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXSW);
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+		}
+		conf = TLS_SW;
+	} else {
+		if (update && ctx->rx_conf == TLS_HW) {
+			rc = -EOPNOTSUPP;
+			goto err_crypto_info;
+		}
+
+		if (!update) {
+			rc = tls_set_device_offload_rx(sk, ctx);
+			conf = TLS_HW;
+			if (!rc) {
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+				tls_sw_strparser_arm(sk, ctx);
+				goto out;
 			}
-			conf = TLS_SW;
 		}
-		if (!update)
+
+		rc = tls_set_sw_offload(sk, 0, update ? crypto_info : NULL);
+		if (rc)
+			goto err_crypto_info;
+
+		if (update) {
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+		} else {
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXSW);
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
 			tls_sw_strparser_arm(sk, ctx);
+		}
+		conf = TLS_SW;
 	}
 
+out:
 	if (tx)
 		ctx->tx_conf = conf;
 	else
-- 
2.25.1

[PATCH v16 04/10] tls: add TLS 1.3 hardware offload support

[Rishikesh Jethwani]

Add TLS 1.3 support to the kernel TLS hardware offload infrastructure,
enabling hardware acceleration for TLS 1.3 connections on capable NICs.
The dispatch-side restructure needed to make this safe on the rekey
path lands in a preceding patch ("tls: skip device-offload dispatch on
the rekey path"); this patch is limited to the device / device_fallback
side:

  - Drop the TLS_1_2_VERSION guards in tls_set_device_offload() and
    tls_set_device_offload_rx() so 1.3 crypto_info is accepted.

  - tls_device_record_close(): append TLS 1.3's content_type byte
    together with the tag as one tail segment; use the pre-populated
    dummy_page (identity-mapped byte values) as the fallback when
    pfrag allocation fails so the content_type byte lands on the
    dummy path too.

  - tls_device_reencrypt(): use prot->prepend_size instead of an
    open-coded TLS_HEADER_SIZE + iv, so the 1.3 prepend layout is
    handled correctly.

  - tls_device_fallback.c / tls_enc_record(): thread tls_context in
    to reach crypto_send for the 1.3 static IV; select IV source and
    length adjustment based on prot->version; XOR the IV with the
    record sequence via tls_xor_iv_with_seq() for 1.3; use
    prot->aad_size and prot->prepend_size instead of the 1.2-only
    constants. Inline the single caller of tls_init_aead_request().

  - tls_device_init(): pre-populate dummy_page with an identity byte
    map so any record_type used as a page offset yields the correct
    content_type byte on the fallback path (avoids a runtime range
    check).

Tested on Mellanox ConnectX-6 Dx (Crypto Enabled) with TLS 1.3
AES-GCM-128 and AES-GCM-256 cipher suites.

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 net/tls/tls_device.c          | 65 ++++++++++++++++++++---------------
 net/tls/tls_device_fallback.c | 58 +++++++++++++++++--------------
 2 files changed, 71 insertions(+), 52 deletions(-)

diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index 37bb06a8e8f5..bbb1aa733500 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -317,25 +317,34 @@ static void tls_device_record_close(struct sock *sk,
 				    unsigned char record_type)
 {
 	struct tls_prot_info *prot = &ctx->prot_info;
-	struct page_frag dummy_tag_frag;
-
-	/* append tag
-	 * device will fill in the tag, we just need to append a placeholder
-	 * use socket memory to improve coalescing (re-using a single buffer
-	 * increases frag count)
-	 * if we can't allocate memory now use the dummy page
+	int tail = prot->tag_size + prot->tail_size;
+
+	/* Append tail: tag for TLS 1.2, content_type + tag for TLS 1.3.
+	 * Device fills in the tag, we just need to append a placeholder.
+	 * Use socket memory to improve coalescing (re-using a single buffer
+	 * increases frag count); if allocation fails use dummy_page
+	 * (offset = record_type gives correct content_type byte via
+	 * identity mapping)
 	 */
-	if (unlikely(pfrag->size - pfrag->offset < prot->tag_size) &&
-	    !skb_page_frag_refill(prot->tag_size, pfrag, sk->sk_allocation)) {
-		dummy_tag_frag.page = dummy_page;
-		dummy_tag_frag.offset = 0;
-		pfrag = &dummy_tag_frag;
+	if (unlikely(pfrag->size - pfrag->offset < tail) &&
+	    !skb_page_frag_refill(tail, pfrag, sk->sk_allocation)) {
+		struct page_frag dummy_pfrag = {
+			.page = dummy_page,
+			.offset = record_type,
+		};
+		tls_append_frag(record, &dummy_pfrag, tail);
+	} else {
+		if (prot->tail_size) {
+			char *content_type_addr = page_address(pfrag->page) +
+						  pfrag->offset;
+			*content_type_addr = record_type;
+		}
+		tls_append_frag(record, pfrag, tail);
 	}
-	tls_append_frag(record, pfrag, prot->tag_size);
 
 	/* fill prepend */
 	tls_fill_prepend(ctx, skb_frag_address(&record->frags[0]),
-			 record->len - prot->overhead_size,
+			 record->len - prot->overhead_size + prot->tail_size,
 			 record_type);
 }
 
@@ -885,6 +894,7 @@ static int
 tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
 {
 	struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(tls_ctx);
+	struct tls_prot_info *prot = &tls_ctx->prot_info;
 	const struct tls_cipher_desc *cipher_desc;
 	int err, offset, copy, data_len, pos;
 	struct sk_buff *skb, *skb_iter;
@@ -896,7 +906,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
 	DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
 
 	rxm = strp_msg(tls_strp_msg(sw_ctx));
-	orig_buf = kmalloc(rxm->full_len + TLS_HEADER_SIZE + cipher_desc->iv,
+	orig_buf = kmalloc(rxm->full_len + prot->prepend_size,
 			   sk->sk_allocation);
 	if (!orig_buf)
 		return -ENOMEM;
@@ -911,9 +921,8 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
 	offset = rxm->offset;
 
 	sg_init_table(sg, 1);
-	sg_set_buf(&sg[0], buf,
-		   rxm->full_len + TLS_HEADER_SIZE + cipher_desc->iv);
-	err = skb_copy_bits(skb, offset, buf, TLS_HEADER_SIZE + cipher_desc->iv);
+	sg_set_buf(&sg[0], buf, rxm->full_len + prot->prepend_size);
+	err = skb_copy_bits(skb, offset, buf, prot->prepend_size);
 	if (err)
 		goto free_buf;
 
@@ -1091,11 +1100,6 @@ int tls_set_device_offload(struct sock *sk)
 	}
 
 	crypto_info = &ctx->crypto_send.info;
-	if (crypto_info->version != TLS_1_2_VERSION) {
-		rc = -EOPNOTSUPP;
-		goto release_netdev;
-	}
-
 	cipher_desc = get_cipher_desc(crypto_info->cipher_type);
 	if (!cipher_desc || !cipher_desc->offloadable) {
 		rc = -EINVAL;
@@ -1198,9 +1202,6 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
 	struct net_device *netdev;
 	int rc = 0;
 
-	if (ctx->crypto_recv.info.version != TLS_1_2_VERSION)
-		return -EOPNOTSUPP;
-
 	netdev = get_netdev_for_sock(sk);
 	if (!netdev) {
 		pr_err_ratelimited("%s: netdev not found\n", __func__);
@@ -1410,12 +1411,22 @@ static struct notifier_block tls_dev_notifier = {
 
 int __init tls_device_init(void)
 {
-	int err;
+	unsigned char *page_addr;
+	int err, i;
 
 	dummy_page = alloc_page(GFP_KERNEL);
 	if (!dummy_page)
 		return -ENOMEM;
 
+	/* Pre-populate dummy_page with identity mapping for all byte values.
+	 * This is used as fallback for TLS 1.3 content type when memory
+	 * allocation fails. By populating all 256 values, we avoid needing
+	 * to validate record_type at runtime.
+	 */
+	page_addr = page_address(dummy_page);
+	for (i = 0; i < 256; i++)
+		page_addr[i] = (unsigned char)i;
+
 	destruct_wq = alloc_workqueue("ktls_device_destruct", WQ_PERCPU, 0);
 	if (!destruct_wq) {
 		err = -ENOMEM;
diff --git a/net/tls/tls_device_fallback.c b/net/tls/tls_device_fallback.c
index 3b7d0ab2bcf1..1110f7ac6bcb 100644
--- a/net/tls/tls_device_fallback.c
+++ b/net/tls/tls_device_fallback.c
@@ -37,14 +37,15 @@
 
 #include "tls.h"
 
-static int tls_enc_record(struct aead_request *aead_req,
+static int tls_enc_record(struct tls_context *tls_ctx,
+			  struct aead_request *aead_req,
 			  struct crypto_aead *aead, char *aad,
 			  char *iv, __be64 rcd_sn,
 			  struct scatter_walk *in,
-			  struct scatter_walk *out, int *in_len,
-			  struct tls_prot_info *prot)
+			  struct scatter_walk *out, int *in_len)
 {
 	unsigned char buf[TLS_HEADER_SIZE + TLS_MAX_IV_SIZE];
+	struct tls_prot_info *prot = &tls_ctx->prot_info;
 	const struct tls_cipher_desc *cipher_desc;
 	struct scatterlist sg_in[3];
 	struct scatterlist sg_out[3];
@@ -55,7 +56,7 @@ static int tls_enc_record(struct aead_request *aead_req,
 	cipher_desc = get_cipher_desc(prot->cipher_type);
 	DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
 
-	buf_size = TLS_HEADER_SIZE + cipher_desc->iv;
+	buf_size = prot->prepend_size;
 	len = min_t(int, *in_len, buf_size);
 
 	memcpy_from_scatterwalk(buf, in, len);
@@ -66,16 +67,27 @@ static int tls_enc_record(struct aead_request *aead_req,
 		return 0;
 
 	len = buf[4] | (buf[3] << 8);
-	len -= cipher_desc->iv;
+	if (prot->version != TLS_1_3_VERSION)
+		len -= cipher_desc->iv;
 
 	tls_make_aad(aad, len - cipher_desc->tag, (char *)&rcd_sn, buf[0], prot);
 
-	memcpy(iv + cipher_desc->salt, buf + TLS_HEADER_SIZE, cipher_desc->iv);
+	if (prot->version == TLS_1_3_VERSION) {
+		void *iv_src = crypto_info_iv(&tls_ctx->crypto_send.info,
+					      cipher_desc);
+
+		memcpy(iv + cipher_desc->salt, iv_src, cipher_desc->iv);
+	} else {
+		memcpy(iv + cipher_desc->salt, buf + TLS_HEADER_SIZE,
+		       cipher_desc->iv);
+	}
+
+	tls_xor_iv_with_seq(prot, iv, (char *)&rcd_sn);
 
 	sg_init_table(sg_in, ARRAY_SIZE(sg_in));
 	sg_init_table(sg_out, ARRAY_SIZE(sg_out));
-	sg_set_buf(sg_in, aad, TLS_AAD_SPACE_SIZE);
-	sg_set_buf(sg_out, aad, TLS_AAD_SPACE_SIZE);
+	sg_set_buf(sg_in, aad, prot->aad_size);
+	sg_set_buf(sg_out, aad, prot->aad_size);
 	scatterwalk_get_sglist(in, sg_in + 1);
 	scatterwalk_get_sglist(out, sg_out + 1);
 
@@ -108,13 +120,6 @@ static int tls_enc_record(struct aead_request *aead_req,
 	return rc;
 }
 
-static void tls_init_aead_request(struct aead_request *aead_req,
-				  struct crypto_aead *aead)
-{
-	aead_request_set_tfm(aead_req, aead);
-	aead_request_set_ad(aead_req, TLS_AAD_SPACE_SIZE);
-}
-
 static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,
 						   gfp_t flags)
 {
@@ -124,14 +129,15 @@ static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,
 
 	aead_req = kzalloc(req_size, flags);
 	if (aead_req)
-		tls_init_aead_request(aead_req, aead);
+		aead_request_set_tfm(aead_req, aead);
 	return aead_req;
 }
 
-static int tls_enc_records(struct aead_request *aead_req,
+static int tls_enc_records(struct tls_context *tls_ctx,
+			   struct aead_request *aead_req,
 			   struct crypto_aead *aead, struct scatterlist *sg_in,
 			   struct scatterlist *sg_out, char *aad, char *iv,
-			   u64 rcd_sn, int len, struct tls_prot_info *prot)
+			   u64 rcd_sn, int len)
 {
 	struct scatter_walk out, in;
 	int rc;
@@ -140,8 +146,8 @@ static int tls_enc_records(struct aead_request *aead_req,
 	scatterwalk_start(&out, sg_out);
 
 	do {
-		rc = tls_enc_record(aead_req, aead, aad, iv,
-				    cpu_to_be64(rcd_sn), &in, &out, &len, prot);
+		rc = tls_enc_record(tls_ctx, aead_req, aead, aad, iv,
+				    cpu_to_be64(rcd_sn), &in, &out, &len);
 		rcd_sn++;
 
 	} while (rc == 0 && len);
@@ -314,7 +320,10 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
 	cipher_desc = get_cipher_desc(tls_ctx->crypto_send.info.cipher_type);
 	DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
 
-	buf_len = cipher_desc->salt + cipher_desc->iv + TLS_AAD_SPACE_SIZE +
+	aead_request_set_ad(aead_req, tls_ctx->prot_info.aad_size);
+
+	buf_len = cipher_desc->salt + cipher_desc->iv +
+		  tls_ctx->prot_info.aad_size +
 		  sync_size + cipher_desc->tag;
 	buf = kmalloc(buf_len, GFP_ATOMIC);
 	if (!buf)
@@ -324,7 +333,7 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
 	salt = crypto_info_salt(&tls_ctx->crypto_send.info, cipher_desc);
 	memcpy(iv, salt, cipher_desc->salt);
 	aad = buf + cipher_desc->salt + cipher_desc->iv;
-	dummy_buf = aad + TLS_AAD_SPACE_SIZE;
+	dummy_buf = aad + tls_ctx->prot_info.aad_size;
 
 	nskb = alloc_skb(skb_headroom(skb) + skb->len, GFP_ATOMIC);
 	if (!nskb)
@@ -335,9 +344,8 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
 	fill_sg_out(sg_out, buf, tls_ctx, nskb, tcp_payload_offset,
 		    payload_len, sync_size, dummy_buf);
 
-	if (tls_enc_records(aead_req, ctx->aead_send, sg_in, sg_out, aad, iv,
-			    rcd_sn, sync_size + payload_len,
-			    &tls_ctx->prot_info) < 0)
+	if (tls_enc_records(tls_ctx, aead_req, ctx->aead_send, sg_in, sg_out,
+			    aad, iv, rcd_sn, sync_size + payload_len) < 0)
 		goto free_nskb;
 
 	complete_skb(nskb, skb, tcp_payload_offset);
-- 
2.25.1

[PATCH v16 05/10] tls: split tls_set_sw_offload into init and finalize stages

[Rishikesh Jethwani]

Separate cipher context initialization from key material finalization
to support staged setup for hardware offload fallback paths.

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 net/tls/tls.h        |  4 +++
 net/tls/tls_device.c |  3 +-
 net/tls/tls_sw.c     | 77 +++++++++++++++++++++++++++++++-------------
 3 files changed, 61 insertions(+), 23 deletions(-)

diff --git a/net/tls/tls.h b/net/tls/tls.h
index 60a37bdaaa25..5a6ee1ea00f8 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -147,6 +147,10 @@ void tls_strp_abort_strp(struct tls_strparser *strp, int err);
 int init_prot_info(struct tls_prot_info *prot,
 		   const struct tls_crypto_info *crypto_info,
 		   const struct tls_cipher_desc *cipher_desc);
+int tls_sw_ctx_init(struct sock *sk, int tx,
+		    struct tls_crypto_info *new_crypto_info);
+void tls_sw_ctx_finalize(struct sock *sk, int tx,
+			 struct tls_crypto_info *new_crypto_info);
 int tls_set_sw_offload(struct sock *sk, int tx,
 		       struct tls_crypto_info *new_crypto_info);
 void tls_update_rx_zc_capable(struct tls_context *tls_ctx);
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index bbb1aa733500..cf67e1f6c5f4 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -1235,7 +1235,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
 	context->resync_nh_reset = 1;
 
 	ctx->priv_ctx_rx = context;
-	rc = tls_set_sw_offload(sk, 0, NULL);
+	rc = tls_sw_ctx_init(sk, 0, NULL);
 	if (rc)
 		goto release_ctx;
 
@@ -1249,6 +1249,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
 		goto free_sw_resources;
 
 	tls_device_attach(ctx, sk, netdev);
+	tls_sw_ctx_finalize(sk, 0, NULL);
 	up_read(&device_offload_lock);
 
 	dev_put(netdev);
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 62d46736e24b..63c83247f9a3 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -2517,20 +2517,19 @@ static void tls_finish_key_update(struct sock *sk, struct tls_context *tls_ctx)
 	ctx->saved_data_ready(sk);
 }
 
-int tls_set_sw_offload(struct sock *sk, int tx,
-		       struct tls_crypto_info *new_crypto_info)
+int tls_sw_ctx_init(struct sock *sk, int tx,
+		    struct tls_crypto_info *new_crypto_info)
 {
 	struct tls_crypto_info *crypto_info, *src_crypto_info;
 	struct tls_sw_context_tx *sw_ctx_tx = NULL;
 	struct tls_sw_context_rx *sw_ctx_rx = NULL;
 	const struct tls_cipher_desc *cipher_desc;
-	char *iv, *rec_seq, *key, *salt;
-	struct cipher_context *cctx;
 	struct tls_prot_info *prot;
 	struct crypto_aead **aead;
 	struct tls_context *ctx;
 	struct crypto_tfm *tfm;
 	int rc = 0;
+	char *key;
 
 	ctx = tls_get_ctx(sk);
 	prot = &ctx->prot_info;
@@ -2551,12 +2550,10 @@ int tls_set_sw_offload(struct sock *sk, int tx,
 	if (tx) {
 		sw_ctx_tx = ctx->priv_ctx_tx;
 		crypto_info = &ctx->crypto_send.info;
-		cctx = &ctx->tx;
 		aead = &sw_ctx_tx->aead_send;
 	} else {
 		sw_ctx_rx = ctx->priv_ctx_rx;
 		crypto_info = &ctx->crypto_recv.info;
-		cctx = &ctx->rx;
 		aead = &sw_ctx_rx->aead_recv;
 	}
 
@@ -2572,10 +2569,7 @@ int tls_set_sw_offload(struct sock *sk, int tx,
 	if (rc)
 		goto free_priv;
 
-	iv = crypto_info_iv(src_crypto_info, cipher_desc);
 	key = crypto_info_key(src_crypto_info, cipher_desc);
-	salt = crypto_info_salt(src_crypto_info, cipher_desc);
-	rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);
 
 	if (!*aead) {
 		*aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, 0);
@@ -2619,19 +2613,6 @@ int tls_set_sw_offload(struct sock *sk, int tx,
 			goto free_aead;
 	}
 
-	memcpy(cctx->iv, salt, cipher_desc->salt);
-	memcpy(cctx->iv + cipher_desc->salt, iv, cipher_desc->iv);
-	memcpy(cctx->rec_seq, rec_seq, cipher_desc->rec_seq);
-
-	if (new_crypto_info) {
-		unsafe_memcpy(crypto_info, new_crypto_info,
-			      cipher_desc->crypto_info,
-			      /* size was checked in do_tls_setsockopt_conf */);
-		memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
-		if (!tx)
-			tls_finish_key_update(sk, ctx);
-	}
-
 	goto out;
 
 free_aead:
@@ -2650,3 +2631,55 @@ int tls_set_sw_offload(struct sock *sk, int tx,
 out:
 	return rc;
 }
+
+void tls_sw_ctx_finalize(struct sock *sk, int tx,
+			 struct tls_crypto_info *new_crypto_info)
+{
+	struct tls_crypto_info *crypto_info, *src_crypto_info;
+	const struct tls_cipher_desc *cipher_desc;
+	struct tls_context *ctx = tls_get_ctx(sk);
+	struct cipher_context *cctx;
+	char *iv, *salt, *rec_seq;
+
+	if (tx) {
+		crypto_info = &ctx->crypto_send.info;
+		cctx = &ctx->tx;
+	} else {
+		crypto_info = &ctx->crypto_recv.info;
+		cctx = &ctx->rx;
+	}
+
+	src_crypto_info = new_crypto_info ?: crypto_info;
+	cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+
+	iv = crypto_info_iv(src_crypto_info, cipher_desc);
+	salt = crypto_info_salt(src_crypto_info, cipher_desc);
+	rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);
+
+	memcpy(cctx->iv, salt, cipher_desc->salt);
+	memcpy(cctx->iv + cipher_desc->salt, iv, cipher_desc->iv);
+	memcpy(cctx->rec_seq, rec_seq, cipher_desc->rec_seq);
+
+	if (new_crypto_info) {
+		unsafe_memcpy(crypto_info, new_crypto_info,
+			      cipher_desc->crypto_info,
+			      /* size was checked in do_tls_setsockopt_conf */);
+		memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
+
+		if (!tx)
+			tls_finish_key_update(sk, ctx);
+	}
+}
+
+int tls_set_sw_offload(struct sock *sk, int tx,
+		       struct tls_crypto_info *new_crypto_info)
+{
+	int rc;
+
+	rc = tls_sw_ctx_init(sk, tx, new_crypto_info);
+	if (rc)
+		return rc;
+
+	tls_sw_ctx_finalize(sk, tx, new_crypto_info);
+	return 0;
+}
-- 
2.25.1

[PATCH v16 06/10] tls: prep helpers and refactors for HW offload KeyUpdate

[Rishikesh Jethwani]

Preparatory refactors for TX and RX HW rekey support; no functional
change.

  - Hoist cipher_context / tls_crypto_context above
    tls_offload_context_tx so they can be embedded in offload
    contexts.

  - Add tls_tx_cipher_ctx() accessor and factor tls_sw_ctx_tx_init()
    so the TX path can redirect to a temporary SW context during
    rekey.

  - Split tls_set_device_offload() into a dispatcher and
    tls_set_device_offload_initial(); a _rekey() sibling follows.

  - Factor tls_device_dev_add_tx() and tls_device_commit_start_marker()
    so the rekey completion path can reuse them.

  - Move crypto_aead_setauthsize() into the !*aead block so a fresh
    AEAD is correctly configured when RX HW rekey allocates one.

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 include/net/tls.h    |  38 +++++++-----
 net/tls/tls.h        |   3 +
 net/tls/tls_device.c | 139 ++++++++++++++++++++++++++-----------------
 net/tls/tls_sw.c     |  67 ++++++++++++---------
 4 files changed, 149 insertions(+), 98 deletions(-)

diff --git a/include/net/tls.h b/include/net/tls.h
index e57bef58851e..eb258bcd62bc 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -155,6 +155,22 @@ struct tls_record_info {
 	skb_frag_t frags[MAX_SKB_FRAGS];
 };
 
+struct cipher_context {
+	char iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
+	char rec_seq[TLS_MAX_REC_SEQ_SIZE];
+};
+
+union tls_crypto_context {
+	struct tls_crypto_info info;
+	union {
+		struct tls12_crypto_info_aes_gcm_128 aes_gcm_128;
+		struct tls12_crypto_info_aes_gcm_256 aes_gcm_256;
+		struct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;
+		struct tls12_crypto_info_sm4_gcm sm4_gcm;
+		struct tls12_crypto_info_sm4_ccm sm4_ccm;
+	};
+};
+
 #define TLS_DRIVER_STATE_SIZE_TX	16
 struct tls_offload_context_tx {
 	struct crypto_aead *aead_send;
@@ -195,22 +211,6 @@ enum tls_context_flags {
 	TLS_RX_DEV_CLOSED = 2,
 };
 
-struct cipher_context {
-	char iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
-	char rec_seq[TLS_MAX_REC_SEQ_SIZE];
-};
-
-union tls_crypto_context {
-	struct tls_crypto_info info;
-	union {
-		struct tls12_crypto_info_aes_gcm_128 aes_gcm_128;
-		struct tls12_crypto_info_aes_gcm_256 aes_gcm_256;
-		struct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;
-		struct tls12_crypto_info_sm4_gcm sm4_gcm;
-		struct tls12_crypto_info_sm4_ccm sm4_ccm;
-	};
-};
-
 struct tls_prot_info {
 	u16 version;
 	u16 cipher_type;
@@ -392,6 +392,12 @@ static inline struct tls_sw_context_tx *tls_sw_ctx_tx(
 	return (struct tls_sw_context_tx *)tls_ctx->priv_ctx_tx;
 }
 
+static inline struct cipher_context *tls_tx_cipher_ctx(
+		const struct tls_context *tls_ctx)
+{
+	return (struct cipher_context *)&tls_ctx->tx;
+}
+
 static inline struct tls_offload_context_tx *
 tls_offload_ctx_tx(const struct tls_context *tls_ctx)
 {
diff --git a/net/tls/tls.h b/net/tls/tls.h
index 5a6ee1ea00f8..e8b552c2f4fb 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -157,7 +157,10 @@ void tls_update_rx_zc_capable(struct tls_context *tls_ctx);
 void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx);
 void tls_sw_strparser_done(struct tls_context *tls_ctx);
 int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx);
+int tls_sw_push_pending_record(struct sock *sk, int flags);
 void tls_sw_splice_eof(struct socket *sock);
+void tls_sw_splice_eof_locked(struct socket *sock);
 void tls_sw_cancel_work_tx(struct tls_context *tls_ctx);
 void tls_sw_release_resources_tx(struct sock *sk);
 void tls_sw_free_ctx_tx(struct tls_context *tls_ctx);
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index cf67e1f6c5f4..4f6b3ea1bde4 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -138,6 +138,41 @@ static struct net_device *get_netdev_for_sock(struct sock *sk)
 	return lowest_dev;
 }
 
+static int tls_device_dev_add_tx(struct sock *sk, struct net_device *netdev,
+				 struct tls_crypto_info *crypto_info,
+				 u32 write_seq)
+{
+	const struct tls_cipher_desc *cipher_desc;
+	char *rec_seq;
+	int rc;
+
+	cipher_desc = get_cipher_desc(crypto_info->cipher_type);
+	DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+	rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,
+					     crypto_info, write_seq);
+	rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+	trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,
+				     write_seq, rec_seq, rc);
+	return rc;
+}
+
+static void tls_device_commit_start_marker(struct sock *sk,
+					struct tls_offload_context_tx *offload_ctx,
+					struct tls_record_info *start_marker_record)
+{
+	start_marker_record->end_seq = tcp_sk(sk)->write_seq;
+	start_marker_record->len = 0;
+	start_marker_record->num_frags = 0;
+	list_add_tail_rcu(&start_marker_record->list, &offload_ctx->records_list);
+
+	/* TLS offload is greatly simplified if we don't send
+	 * SKBs where only part of the payload needs to be encrypted.
+	 * So mark the last skb in the write queue as end of record.
+	 */
+	tcp_write_collapse_fence(sk);
+}
+
 static void destroy_record(struct tls_record_info *record)
 {
 	int i;
@@ -1070,57 +1105,31 @@ static struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *c
 	return offload_ctx;
 }
 
-int tls_set_device_offload(struct sock *sk)
+static int tls_set_device_offload_initial(struct sock *sk,
+					  struct tls_context *ctx,
+					  struct net_device *netdev,
+					  struct tls_crypto_info *crypto_info,
+					  const struct tls_cipher_desc *cipher_desc)
 {
+	struct tls_prot_info *prot = &ctx->prot_info;
 	struct tls_record_info *start_marker_record;
 	struct tls_offload_context_tx *offload_ctx;
-	const struct tls_cipher_desc *cipher_desc;
-	struct tls_crypto_info *crypto_info;
-	struct tls_prot_info *prot;
-	struct net_device *netdev;
-	struct tls_context *ctx;
 	char *iv, *rec_seq;
 	int rc;
 
-	ctx = tls_get_ctx(sk);
-	prot = &ctx->prot_info;
-
-	if (ctx->priv_ctx_tx)
-		return -EEXIST;
-
-	netdev = get_netdev_for_sock(sk);
-	if (!netdev) {
-		pr_err_ratelimited("%s: netdev not found\n", __func__);
-		return -EINVAL;
-	}
-
-	if (!(netdev->features & NETIF_F_HW_TLS_TX)) {
-		rc = -EOPNOTSUPP;
-		goto release_netdev;
-	}
-
-	crypto_info = &ctx->crypto_send.info;
-	cipher_desc = get_cipher_desc(crypto_info->cipher_type);
-	if (!cipher_desc || !cipher_desc->offloadable) {
-		rc = -EINVAL;
-		goto release_netdev;
-	}
+	iv = crypto_info_iv(crypto_info, cipher_desc);
+	rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
 
 	rc = init_prot_info(prot, crypto_info, cipher_desc);
 	if (rc)
-		goto release_netdev;
-
-	iv = crypto_info_iv(crypto_info, cipher_desc);
-	rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+		return rc;
 
 	memcpy(ctx->tx.iv + cipher_desc->salt, iv, cipher_desc->iv);
 	memcpy(ctx->tx.rec_seq, rec_seq, cipher_desc->rec_seq);
 
 	start_marker_record = kmalloc_obj(*start_marker_record);
-	if (!start_marker_record) {
-		rc = -ENOMEM;
-		goto release_netdev;
-	}
+	if (!start_marker_record)
+		return -ENOMEM;
 
 	offload_ctx = alloc_offload_ctx_tx(ctx);
 	if (!offload_ctx) {
@@ -1132,20 +1141,11 @@ int tls_set_device_offload(struct sock *sk)
 	if (rc)
 		goto free_offload_ctx;
 
-	start_marker_record->end_seq = tcp_sk(sk)->write_seq;
-	start_marker_record->len = 0;
-	start_marker_record->num_frags = 0;
-	list_add_tail(&start_marker_record->list, &offload_ctx->records_list);
+	tls_device_commit_start_marker(sk, offload_ctx, start_marker_record);
 
 	clean_acked_data_enable(tcp_sk(sk), &tls_tcp_clean_acked);
 	ctx->push_pending_record = tls_device_push_pending_record;
 
-	/* TLS offload is greatly simplified if we don't send
-	 * SKBs where only part of the payload needs to be encrypted.
-	 * So mark the last skb in the write queue as end of record.
-	 */
-	tcp_write_collapse_fence(sk);
-
 	/* Avoid offloading if the device is down
 	 * We don't want to offload new flows after
 	 * the NETDEV_DOWN event
@@ -1161,11 +1161,8 @@ int tls_set_device_offload(struct sock *sk)
 	}
 
 	ctx->priv_ctx_tx = offload_ctx;
-	rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,
-					     &ctx->crypto_send.info,
-					     tcp_sk(sk)->write_seq);
-	trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,
-				     tcp_sk(sk)->write_seq, rec_seq, rc);
+	rc = tls_device_dev_add_tx(sk, netdev, crypto_info,
+				   tcp_sk(sk)->write_seq);
 	if (rc)
 		goto release_lock;
 
@@ -1177,7 +1174,6 @@ int tls_set_device_offload(struct sock *sk)
 	 * by the netdev's xmit function.
 	 */
 	smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);
-	dev_put(netdev);
 
 	return 0;
 
@@ -1190,6 +1186,43 @@ int tls_set_device_offload(struct sock *sk)
 	ctx->priv_ctx_tx = NULL;
 free_marker_record:
 	kfree(start_marker_record);
+	return rc;
+}
+
+int tls_set_device_offload(struct sock *sk)
+{
+	const struct tls_cipher_desc *cipher_desc;
+	struct tls_crypto_info *crypto_info;
+	struct net_device *netdev;
+	struct tls_context *ctx;
+	int rc;
+
+	ctx = tls_get_ctx(sk);
+
+	if (ctx->priv_ctx_tx)
+		return -EEXIST;
+
+	netdev = get_netdev_for_sock(sk);
+	if (!netdev) {
+		pr_err_ratelimited("%s: netdev not found\n", __func__);
+		return -EINVAL;
+	}
+
+	if (!(netdev->features & NETIF_F_HW_TLS_TX)) {
+		rc = -EOPNOTSUPP;
+		goto release_netdev;
+	}
+
+	crypto_info = &ctx->crypto_send.info;
+	cipher_desc = get_cipher_desc(crypto_info->cipher_type);
+	if (!cipher_desc || !cipher_desc->offloadable) {
+		rc = -EINVAL;
+		goto release_netdev;
+	}
+
+	rc = tls_set_device_offload_initial(sk, ctx, netdev, crypto_info,
+					    cipher_desc);
+
 release_netdev:
 	dev_put(netdev);
 	return rc;
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 63c83247f9a3..7b6ab992c86e 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -555,11 +555,11 @@ static int tls_do_encryption(struct sock *sk,
 		break;
 	}
 
-	memcpy(&rec->iv_data[iv_offset], tls_ctx->tx.iv,
+	memcpy(&rec->iv_data[iv_offset], tls_tx_cipher_ctx(tls_ctx)->iv,
 	       prot->iv_size + prot->salt_size);
 
 	tls_xor_iv_with_seq(prot, rec->iv_data + iv_offset,
-			    tls_ctx->tx.rec_seq);
+			    tls_tx_cipher_ctx(tls_ctx)->rec_seq);
 
 	sge->offset += prot->prepend_size;
 	sge->length -= prot->prepend_size;
@@ -610,7 +610,7 @@ static int tls_do_encryption(struct sock *sk,
 
 	/* Unhook the record from context if encryption is not failure */
 	ctx->open_rec = NULL;
-	tls_advance_record_sn(sk, prot, &tls_ctx->tx);
+	tls_advance_record_sn(sk, prot, tls_tx_cipher_ctx(tls_ctx));
 	return rc;
 }
 
@@ -676,7 +676,7 @@ static int tls_push_record(struct sock *sk, int flags,
 	sg_chain(rec->sg_aead_out, 2, &msg_en->sg.data[i]);
 
 	tls_make_aad(rec->aad_space, msg_pl->sg.size + prot->tail_size,
-		     tls_ctx->tx.rec_seq, record_type, prot);
+		     tls_tx_cipher_ctx(tls_ctx)->rec_seq, record_type, prot);
 
 	tls_fill_prepend(tls_ctx,
 			 page_address(sg_page(&msg_en->sg.data[i])) +
@@ -712,7 +712,7 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,
 	return err;
 }
 
-static int tls_sw_push_pending_record(struct sock *sk, int flags)
+int tls_sw_push_pending_record(struct sock *sk, int flags)
 {
 	struct tls_context *tls_ctx = tls_get_ctx(sk);
 	struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);
@@ -1028,7 +1028,7 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
 /*
  * Handle unexpected EOF during splice without SPLICE_F_MORE set.
  */
-void tls_sw_splice_eof(struct socket *sock)
+void tls_sw_splice_eof_locked(struct socket *sock)
 {
 	struct sock *sk = sock->sk;
 	struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -1039,21 +1039,15 @@ void tls_sw_splice_eof(struct socket *sock)
 	bool retrying = false;
 	int ret = 0;
 
-	if (!ctx->open_rec)
-		return;
-
-	mutex_lock(&tls_ctx->tx_lock);
-	lock_sock(sk);
-
 retry:
 	/* same checks as in tls_sw_push_pending_record() */
 	rec = ctx->open_rec;
 	if (!rec)
-		goto unlock;
+		return;
 
 	msg_pl = &rec->msg_plaintext;
 	if (msg_pl->sg.size == 0)
-		goto unlock;
+		return;
 
 	/* Perform transmission. */
 	ret = bpf_exec_tx_verdict(msg_pl, sk, TLS_RECORD_TYPE_DATA,
@@ -1062,26 +1056,38 @@ void tls_sw_splice_eof(struct socket *sock)
 	case 0:
 	case -EAGAIN:
 		if (retrying)
-			goto unlock;
+			return;
 		retrying = true;
 		goto retry;
 	case -EINPROGRESS:
 		break;
 	default:
-		goto unlock;
+		return;
 	}
 
 	/* Wait for pending encryptions to get completed */
 	if (tls_encrypt_async_wait(ctx))
-		goto unlock;
+		return;
 
 	/* Transmit if any encryptions have completed */
 	if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) {
 		cancel_delayed_work(&ctx->tx_work.work);
 		tls_tx_records(sk, 0);
 	}
+}
+
+void tls_sw_splice_eof(struct socket *sock)
+{
+	struct sock *sk = sock->sk;
+	struct tls_context *tls_ctx = tls_get_ctx(sk);
+	struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);
 
-unlock:
+	if (!ctx->open_rec)
+		return;
+
+	mutex_lock(&tls_ctx->tx_lock);
+	lock_sock(sk);
+	tls_sw_splice_eof_locked(sock);
 	release_sock(sk);
 	mutex_unlock(&tls_ctx->tx_lock);
 }
@@ -2396,6 +2402,15 @@ static void tx_work_handler(struct work_struct *work)
 	}
 }
 
+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx)
+{
+	crypto_init_wait(&sw_ctx->async_wait);
+	atomic_set(&sw_ctx->encrypt_pending, 1);
+	INIT_LIST_HEAD(&sw_ctx->tx_list);
+	INIT_DELAYED_WORK(&sw_ctx->tx_work.work, tx_work_handler);
+	sw_ctx->tx_work.sk = sk;
+}
+
 static bool tls_is_tx_ready(struct tls_sw_context_tx *ctx)
 {
 	struct tls_rec *rec;
@@ -2447,11 +2462,7 @@ static struct tls_sw_context_tx *init_ctx_tx(struct tls_context *ctx, struct soc
 		sw_ctx_tx = ctx->priv_ctx_tx;
 	}
 
-	crypto_init_wait(&sw_ctx_tx->async_wait);
-	atomic_set(&sw_ctx_tx->encrypt_pending, 1);
-	INIT_LIST_HEAD(&sw_ctx_tx->tx_list);
-	INIT_DELAYED_WORK(&sw_ctx_tx->tx_work.work, tx_work_handler);
-	sw_ctx_tx->tx_work.sk = sk;
+	tls_sw_ctx_tx_init(sk, sw_ctx_tx);
 
 	return sw_ctx_tx;
 }
@@ -2578,6 +2589,10 @@ int tls_sw_ctx_init(struct sock *sk, int tx,
 			*aead = NULL;
 			goto free_priv;
 		}
+
+		rc = crypto_aead_setauthsize(*aead, prot->tag_size);
+		if (rc)
+			goto free_aead;
 	}
 
 	ctx->push_pending_record = tls_sw_push_pending_record;
@@ -2594,12 +2609,6 @@ int tls_sw_ctx_init(struct sock *sk, int tx,
 			goto free_aead;
 	}
 
-	if (!new_crypto_info) {
-		rc = crypto_aead_setauthsize(*aead, prot->tag_size);
-		if (rc)
-			goto free_aead;
-	}
-
 	if (!tx && !new_crypto_info) {
 		tfm = crypto_aead_tfm(sw_ctx_rx->aead_recv);
 
-- 
2.25.1

[PATCH v16 07/10] tls: device: add TX KeyUpdate support

[Rishikesh Jethwani]

The NIC key cannot be replaced while HW-offloaded records
are still unacked. tls_device_start_rekey() installs a temporary SW
context with the new key and redirects sendmsg through
tls_sw_sendmsg_locked. If no records are pending,
tls_device_complete_rekey() runs inline during setsockopt; otherwise
tls_tcp_clean_acked sets REKEY_READY once all old-key records are ACKed
and the next sendmsg completes the rekey, flushing SW records and
reinstalling HW offload at the current write_seq. A KeyUpdate
arriving while one is pending re-keys the SW AEAD in place; if the
HW reinstall fails the socket stays in SW mode (REKEY_FAILED).

Tested on Mellanox ConnectX-6 Dx (Crypto Enabled) with multiple
TLS 1.3 TX KeyUpdate cycles.

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 include/net/tls.h             |  47 ++++
 include/uapi/linux/snmp.h     |   3 +
 net/tls/tls.h                 |   8 +-
 net/tls/tls_device.c          | 451 ++++++++++++++++++++++++++++++++--
 net/tls/tls_device_fallback.c |  24 ++
 net/tls/tls_main.c            |  90 +++++--
 net/tls/tls_proc.c            |   3 +
 net/tls/tls_sw.c              |  22 +-
 8 files changed, 602 insertions(+), 46 deletions(-)

diff --git a/include/net/tls.h b/include/net/tls.h
index eb258bcd62bc..7c2cf51dc887 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -185,6 +185,14 @@ struct tls_offload_context_tx {
 	void (*sk_destruct)(struct sock *sk);
 	struct work_struct destruct_work;
 	struct tls_context *ctx;
+
+	struct {
+		struct tls_sw_context_tx sw;	/* SW context for new key */
+		struct cipher_context tx;	/* IV, rec_seq for new key */
+		union tls_crypto_context crypto_send; /* Crypto for new key */
+		struct tls_record_info *start_marker;
+	} rekey;
+
 	/* The TLS layer reserves room for driver specific state
 	 * Currently the belief is that there is not enough
 	 * driver specific state to justify another layer of indirection
@@ -209,6 +217,21 @@ enum tls_context_flags {
 	 * tls_dev_del call in tls_device_down if it happens simultaneously.
 	 */
 	TLS_RX_DEV_CLOSED = 2,
+	/* Flag for TX HW context deleted during failed rekey.
+	 * Prevents double tls_dev_del in cleanup paths.
+	 */
+	TLS_TX_DEV_CLOSED = 3,
+	/* TX rekey is pending, waiting for old-key data to be ACKed.
+	 * While set, new data uses SW path with new key, HW keeps old key
+	 * for retransmissions.
+	 */
+	TLS_TX_REKEY_PENDING = 4,
+	/* All old-key data has been ACKed, ready to install new key in HW. */
+	TLS_TX_REKEY_READY = 5,
+	/* HW rekey failed, permanently stay in SW encrypt mode.
+	 * Prevents tls_tcp_clean_acked from re-setting TLS_TX_REKEY_READY.
+	 */
+	TLS_TX_REKEY_FAILED = 6,
 };
 
 struct tls_prot_info {
@@ -257,6 +280,17 @@ struct tls_context {
 			       */
 	unsigned long flags;
 
+	struct {
+		/* TCP sequence number boundary for pending rekey.
+		 * Packets with seq < this use old key, >= use new key.
+		 */
+		u32 boundary_seq;
+
+		/* Pointers to rekey contexts for SW encryption with new key */
+		struct tls_sw_context_tx *sw_ctx;
+		struct cipher_context *cipher_ctx;
+	} rekey;
+
 	/* cache cold stuff */
 	struct proto *sk_proto;
 	struct sock *sk;
@@ -389,12 +423,22 @@ static inline struct tls_sw_context_rx *tls_sw_ctx_rx(
 static inline struct tls_sw_context_tx *tls_sw_ctx_tx(
 		const struct tls_context *tls_ctx)
 {
+	struct tls_sw_context_tx *rekey_ctx = READ_ONCE(tls_ctx->rekey.sw_ctx);
+
+	if (unlikely(rekey_ctx))
+		return rekey_ctx;
+
 	return (struct tls_sw_context_tx *)tls_ctx->priv_ctx_tx;
 }
 
 static inline struct cipher_context *tls_tx_cipher_ctx(
 		const struct tls_context *tls_ctx)
 {
+	struct cipher_context *rekey_ctx = READ_ONCE(tls_ctx->rekey.cipher_ctx);
+
+	if (unlikely(rekey_ctx))
+		return rekey_ctx;
+
 	return (struct cipher_context *)&tls_ctx->tx;
 }
 
@@ -510,6 +554,9 @@ struct sk_buff *tls_encrypt_skb(struct sk_buff *skb);
 #ifdef CONFIG_TLS_DEVICE
 void tls_device_sk_destruct(struct sock *sk);
 void tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq);
+struct sk_buff *
+tls_validate_xmit_skb_rekey(struct sock *sk, struct net_device *dev,
+			    struct sk_buff *skb);
 
 static inline bool tls_is_sk_rx_device_offloaded(struct sock *sk)
 {
diff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h
index 49f5640092a0..119fbd49179c 100644
--- a/include/uapi/linux/snmp.h
+++ b/include/uapi/linux/snmp.h
@@ -369,6 +369,9 @@ enum
 	LINUX_MIB_TLSTXREKEYOK,			/* TlsTxRekeyOk */
 	LINUX_MIB_TLSTXREKEYERROR,		/* TlsTxRekeyError */
 	LINUX_MIB_TLSRXREKEYRECEIVED,		/* TlsRxRekeyReceived */
+	LINUX_MIB_TLSTXREKEYFALLBACK,		/* TlsTxRekeyFallback */
+	LINUX_MIB_TLSTXREKEYINPROGRESS,		/* TlsTxRekeyInProgress */
+	LINUX_MIB_TLSTXREKEYABORTED,		/* TlsTxRekeyAborted */
 	__LINUX_MIB_TLSMAX
 };
 
diff --git a/net/tls/tls.h b/net/tls/tls.h
index e8b552c2f4fb..ea8fdf198572 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -157,7 +157,10 @@ void tls_update_rx_zc_capable(struct tls_context *tls_ctx);
 void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx);
 void tls_sw_strparser_done(struct tls_context *tls_ctx);
 int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size);
 void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx);
+int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx);
+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx);
 int tls_sw_push_pending_record(struct sock *sk, int flags);
 void tls_sw_splice_eof(struct socket *sock);
 void tls_sw_splice_eof_locked(struct socket *sock);
@@ -237,7 +240,8 @@ static inline bool tls_strp_msg_mixed_decrypted(struct tls_sw_context_rx *ctx)
 #ifdef CONFIG_TLS_DEVICE
 int tls_device_init(void);
 void tls_device_cleanup(void);
-int tls_set_device_offload(struct sock *sk);
+int tls_set_device_offload(struct sock *sk,
+			   struct tls_crypto_info *crypto_info);
 void tls_device_free_resources_tx(struct sock *sk);
 int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx);
 void tls_device_offload_cleanup_rx(struct sock *sk);
@@ -248,7 +252,7 @@ static inline int tls_device_init(void) { return 0; }
 static inline void tls_device_cleanup(void) {}
 
 static inline int
-tls_set_device_offload(struct sock *sk)
+tls_set_device_offload(struct sock *sk, struct tls_crypto_info *crypto_info)
 {
 	return -EOPNOTSUPP;
 }
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index 4f6b3ea1bde4..85c74d0816b7 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -57,8 +57,15 @@ static struct page *dummy_page;
 
 static void tls_device_free_ctx(struct tls_context *ctx)
 {
-	if (ctx->tx_conf == TLS_HW)
-		kfree(tls_offload_ctx_tx(ctx));
+	if (ctx->tx_conf == TLS_HW) {
+		struct tls_offload_context_tx *offload_ctx =
+			tls_offload_ctx_tx(ctx);
+
+		kfree(offload_ctx->rekey.start_marker);
+		memzero_explicit(&offload_ctx->rekey,
+				 sizeof(offload_ctx->rekey));
+		kfree(offload_ctx);
+	}
 
 	if (ctx->rx_conf == TLS_HW)
 		kfree(tls_offload_ctx_rx(ctx));
@@ -79,7 +86,9 @@ static void tls_device_tx_del_task(struct work_struct *work)
 	netdev = rcu_dereference_protected(ctx->netdev,
 					   !refcount_read(&ctx->refcount));
 
-	netdev->tlsdev_ops->tls_dev_del(netdev, ctx, TLS_OFFLOAD_CTX_DIR_TX);
+	if (!test_bit(TLS_TX_DEV_CLOSED, &ctx->flags))
+		netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+						TLS_OFFLOAD_CTX_DIR_TX);
 	dev_put(netdev);
 	ctx->netdev = NULL;
 	tls_device_free_ctx(ctx);
@@ -161,10 +170,14 @@ static void tls_device_commit_start_marker(struct sock *sk,
 					struct tls_offload_context_tx *offload_ctx,
 					struct tls_record_info *start_marker_record)
 {
+	unsigned long flags;
+
+	spin_lock_irqsave(&offload_ctx->lock, flags);
 	start_marker_record->end_seq = tcp_sk(sk)->write_seq;
 	start_marker_record->len = 0;
 	start_marker_record->num_frags = 0;
 	list_add_tail_rcu(&start_marker_record->list, &offload_ctx->records_list);
+	spin_unlock_irqrestore(&offload_ctx->lock, flags);
 
 	/* TLS offload is greatly simplified if we don't send
 	 * SKBs where only part of the payload needs to be encrypted.
@@ -194,6 +207,24 @@ static void delete_all_records(struct tls_offload_context_tx *offload_ctx)
 	offload_ctx->retransmit_hint = NULL;
 }
 
+static bool tls_has_unacked_records(struct tls_offload_context_tx *offload_ctx)
+{
+	struct tls_record_info *info;
+	bool has_unacked = false;
+	unsigned long flags;
+
+	spin_lock_irqsave(&offload_ctx->lock, flags);
+	list_for_each_entry(info, &offload_ctx->records_list, list) {
+		if (!tls_record_is_start_marker(info)) {
+			has_unacked = true;
+			break;
+		}
+	}
+	spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+	return has_unacked;
+}
+
 static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)
 {
 	struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -222,6 +253,19 @@ static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)
 	}
 
 	ctx->unacked_record_sn += deleted_records;
+
+	/* Once all old-key HW records are ACKed, set REKEY_READY to
+	 * let sendmsg know it can finish the rekey and switch back
+	 * to HW offload.
+	 */
+	if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags) &&
+	    !test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags)) {
+		u32 boundary_seq = READ_ONCE(tls_ctx->rekey.boundary_seq);
+
+		if (!before(acked_seq, boundary_seq))
+			set_bit(TLS_TX_REKEY_READY, &tls_ctx->flags);
+	}
+
 	spin_unlock_irqrestore(&ctx->lock, flags);
 }
 
@@ -252,7 +296,15 @@ void tls_device_free_resources_tx(struct sock *sk)
 {
 	struct tls_context *tls_ctx = tls_get_ctx(sk);
 
-	tls_free_partial_record(sk, tls_ctx);
+	if (unlikely(tls_ctx->rekey.sw_ctx))
+		tls_sw_release_resources_tx(sk);
+	else
+		tls_free_partial_record(sk, tls_ctx);
+
+	if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags)) {
+		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYABORTED);
+		TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYINPROGRESS);
+	}
 }
 
 void tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)
@@ -462,6 +514,9 @@ static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i)
 	return 0;
 }
 
+static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,
+				     bool deferred);
+
 static int tls_push_data(struct sock *sk,
 			 struct iov_iter *iter,
 			 size_t size, int flags,
@@ -618,6 +673,19 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
 	mutex_lock(&tls_ctx->tx_lock);
 	lock_sock(sk);
 
+	/* Old-key records all ACKed; switch back to HW. */
+	if (test_bit(TLS_TX_REKEY_READY, &tls_ctx->flags))
+		tls_device_complete_rekey(sk, tls_ctx, true);
+
+	/* Use SW path if rekey is in progress (PENDING) or if HW rekey
+	 * failed (FAILED).
+	 */
+	if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags) ||
+	    test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags)) {
+		rc = tls_sw_sendmsg_locked(sk, msg, size);
+		goto out;
+	}
+
 	if (unlikely(msg->msg_controllen)) {
 		rc = tls_process_cmsg(sk, msg, &record_type);
 		if (rc)
@@ -646,8 +714,11 @@ void tls_device_splice_eof(struct socket *sock)
 	mutex_lock(&tls_ctx->tx_lock);
 	lock_sock(sk);
 
-	if (tls_is_partially_sent_record(tls_ctx) ||
-	    tls_is_pending_open_record(tls_ctx)) {
+	if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags) ||
+	    test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags)) {
+		tls_sw_splice_eof_locked(sock);
+	} else if (tls_is_partially_sent_record(tls_ctx) ||
+		   tls_is_pending_open_record(tls_ctx)) {
 		iov_iter_bvec(&iter, ITER_SOURCE, NULL, 0, 0);
 		tls_push_data(sk, &iter, 0, 0, TLS_RECORD_TYPE_DATA);
 	}
@@ -718,8 +789,13 @@ EXPORT_SYMBOL(tls_get_record);
 
 static int tls_device_push_pending_record(struct sock *sk, int flags)
 {
+	struct tls_context *tls_ctx = tls_get_ctx(sk);
 	struct iov_iter iter;
 
+	if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags) ||
+	    test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags))
+		return tls_sw_push_pending_record(sk, flags);
+
 	iov_iter_kvec(&iter, ITER_SOURCE, NULL, 0, 0);
 	return tls_push_data(sk, &iter, 0, flags, TLS_RECORD_TYPE_DATA);
 }
@@ -1105,6 +1181,334 @@ static struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *c
 	return offload_ctx;
 }
 
+static int tls_device_init_rekey_sw(struct sock *sk,
+				    struct tls_context *ctx,
+				    struct tls_offload_context_tx *offload_ctx,
+				    struct tls_crypto_info *new_crypto_info)
+{
+	struct tls_sw_context_tx *sw_ctx = &offload_ctx->rekey.sw;
+	const struct tls_cipher_desc *cipher_desc;
+	char *key;
+	int rc;
+
+	cipher_desc = get_cipher_desc(new_crypto_info->cipher_type);
+	DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+	memset(sw_ctx, 0, sizeof(*sw_ctx));
+	tls_sw_ctx_tx_init(sk, sw_ctx);
+
+	sw_ctx->aead_send = crypto_alloc_aead(cipher_desc->cipher_name, 0, 0);
+	if (IS_ERR(sw_ctx->aead_send)) {
+		rc = PTR_ERR(sw_ctx->aead_send);
+		sw_ctx->aead_send = NULL;
+		return rc;
+	}
+
+	key = crypto_info_key(new_crypto_info, cipher_desc);
+	rc = crypto_aead_setkey(sw_ctx->aead_send, key, cipher_desc->key);
+	if (rc)
+		goto free_aead;
+
+	rc = crypto_aead_setauthsize(sw_ctx->aead_send, cipher_desc->tag);
+	if (rc)
+		goto free_aead;
+
+	return 0;
+
+free_aead:
+	crypto_free_aead(sw_ctx->aead_send);
+	sw_ctx->aead_send = NULL;
+	return rc;
+}
+
+static int tls_device_start_rekey(struct sock *sk,
+				  struct tls_context *ctx,
+				  struct tls_offload_context_tx *offload_ctx,
+				  struct tls_crypto_info *new_crypto_info)
+{
+	bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+	bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+	const struct tls_cipher_desc *cipher_desc;
+	struct crypto_aead *new_aead, *old_aead;
+	char *key, *iv, *rec_seq, *salt;
+	unsigned long flags;
+	int rc;
+
+	cipher_desc = get_cipher_desc(new_crypto_info->cipher_type);
+	DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+	key = crypto_info_key(new_crypto_info, cipher_desc);
+	iv = crypto_info_iv(new_crypto_info, cipher_desc);
+	rec_seq = crypto_info_rec_seq(new_crypto_info, cipher_desc);
+	salt = crypto_info_salt(new_crypto_info, cipher_desc);
+
+	if (rekey_pending || rekey_failed) {
+		/* Wait for in-flight async encryptions submitted to this tfm
+		 * with the previous key before changing it.
+		 */
+		rc = tls_encrypt_async_wait(&offload_ctx->rekey.sw);
+		if (rc)
+			return rc;
+
+		/* Build the new key into a fresh tfm and swap it in only on
+		 * success. Re-keying the live tfm in place is not atomic: a
+		 * failed crypto_aead_setkey() leaves it with CRYPTO_TFM_NEED_KEY
+		 * set, destroying the previous key and rendering the SW fallback
+		 * path unable to encrypt.
+		 */
+		new_aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, 0);
+		if (IS_ERR(new_aead))
+			return PTR_ERR(new_aead);
+
+		rc = crypto_aead_setkey(new_aead, key, cipher_desc->key);
+		if (!rc)
+			rc = crypto_aead_setauthsize(new_aead, cipher_desc->tag);
+		if (rc) {
+			crypto_free_aead(new_aead);
+			return rc;
+		}
+
+		old_aead = offload_ctx->rekey.sw.aead_send;
+		offload_ctx->rekey.sw.aead_send = new_aead;
+		crypto_free_aead(old_aead);
+
+		memcpy(offload_ctx->rekey.tx.iv, salt, cipher_desc->salt);
+		memcpy(offload_ctx->rekey.tx.iv + cipher_desc->salt, iv,
+		       cipher_desc->iv);
+		memcpy(offload_ctx->rekey.tx.rec_seq, rec_seq,
+		       cipher_desc->rec_seq);
+
+		if (rekey_failed) {
+			set_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+			clear_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+			TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+		}
+	} else {
+		/* Flush any HW open_record before switching to SW */
+		if (tls_is_pending_open_record(ctx)) {
+			rc = ctx->push_pending_record(sk, 0);
+			if (rc < 0)
+				return rc;
+		}
+
+		rc = tls_device_init_rekey_sw(sk, ctx, offload_ctx,
+					      new_crypto_info);
+		if (rc)
+			return rc;
+
+		memcpy(offload_ctx->rekey.tx.iv, salt, cipher_desc->salt);
+		memcpy(offload_ctx->rekey.tx.iv + cipher_desc->salt, iv,
+		       cipher_desc->iv);
+		memcpy(offload_ctx->rekey.tx.rec_seq, rec_seq,
+		       cipher_desc->rec_seq);
+
+		/* Prevent a partial record straddling the SW/HW boundary. */
+		tcp_write_collapse_fence(sk);
+
+		WRITE_ONCE(ctx->rekey.sw_ctx, &offload_ctx->rekey.sw);
+		WRITE_ONCE(ctx->rekey.cipher_ctx, &offload_ctx->rekey.tx);
+
+		spin_lock_irqsave(&offload_ctx->lock, flags);
+		WRITE_ONCE(ctx->rekey.boundary_seq, tcp_sk(sk)->write_seq);
+		set_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+		spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+		/* Switch to rekey validator; new sends won't use HW offload */
+		smp_store_release(&sk->sk_validate_xmit_skb,
+				  tls_validate_xmit_skb_rekey);
+	}
+
+	unsafe_memcpy(&offload_ctx->rekey.crypto_send.info, new_crypto_info,
+		      cipher_desc->crypto_info,
+		      /* checked in do_tls_setsockopt_conf */);
+	memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
+
+	return 0;
+}
+
+static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,
+				     bool deferred)
+{
+	struct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);
+	const struct tls_cipher_desc *cipher_desc;
+	struct crypto_aead *new_aead, *old_aead;
+	struct net_device *netdev;
+	unsigned long flags;
+	__be64 rcd_sn;
+	char *key;
+	int rc;
+
+	cipher_desc = get_cipher_desc(offload_ctx->rekey.crypto_send.info.cipher_type);
+	DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+	DEBUG_NET_WARN_ON_ONCE(!offload_ctx->rekey.start_marker);
+
+	rc = tls_sw_drain_tx(sk, ctx);
+	if (rc)
+		return rc;
+
+	down_read(&device_offload_lock);
+
+	netdev = rcu_dereference_protected(ctx->netdev,
+					   lockdep_is_held(&device_offload_lock));
+	if (!netdev) {
+		rc = -ENODEV;
+		goto release_lock;
+	}
+
+	if (!test_bit(TLS_TX_DEV_CLOSED, &ctx->flags)) {
+		netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+						TLS_OFFLOAD_CTX_DIR_TX);
+		set_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+	}
+
+	/* Build the new SW-fallback key into a fresh tfm and swap it in only
+	 * on success. Re-keying the live aead_send in place is not atomic: a
+	 * failed crypto_aead_setkey() leaves it with CRYPTO_TFM_NEED_KEY set,
+	 * destroying the working key. Doing this while the HW context is torn
+	 * down (TLS_TX_DEV_CLOSED set) means a failure falls into rekey_fail
+	 * with HW off, so the SW fallback is coherent, same as a dev_add
+	 * failure.
+	 */
+	key = crypto_info_key(&offload_ctx->rekey.crypto_send.info, cipher_desc);
+	new_aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, CRYPTO_ALG_ASYNC);
+	if (IS_ERR(new_aead)) {
+		rc = PTR_ERR(new_aead);
+		goto release_lock;
+	}
+	rc = crypto_aead_setkey(new_aead, key, cipher_desc->key);
+	if (!rc)
+		rc = crypto_aead_setauthsize(new_aead, cipher_desc->tag);
+	if (rc) {
+		crypto_free_aead(new_aead);
+		goto release_lock;
+	}
+
+	memcpy(crypto_info_rec_seq(&offload_ctx->rekey.crypto_send.info, cipher_desc),
+	       offload_ctx->rekey.tx.rec_seq, cipher_desc->rec_seq);
+
+	rc = tls_device_dev_add_tx(sk, netdev, &offload_ctx->rekey.crypto_send.info,
+				   tcp_sk(sk)->write_seq);
+	if (rc) {
+		crypto_free_aead(new_aead);
+		goto release_lock;
+	}
+
+	/* Point of no return: HW is live with the new key. Swap in the new
+	 * fallback tfm and drop the old one; the remaining steps cannot fail.
+	 */
+	old_aead = offload_ctx->aead_send;
+	offload_ctx->aead_send = new_aead;
+	crypto_free_aead(old_aead);
+	clear_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+
+release_lock:
+	up_read(&device_offload_lock);
+
+	if (rc)
+		goto rekey_fail;
+
+	spin_lock_irqsave(&offload_ctx->lock, flags);
+	memcpy(&rcd_sn, offload_ctx->rekey.tx.rec_seq, sizeof(rcd_sn));
+	offload_ctx->unacked_record_sn = be64_to_cpu(rcd_sn) - 1;
+	spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+	memcpy(ctx->tx.iv, offload_ctx->rekey.tx.iv,
+	       cipher_desc->salt + cipher_desc->iv);
+	memcpy(ctx->tx.rec_seq, offload_ctx->rekey.tx.rec_seq,
+	       cipher_desc->rec_seq);
+	unsafe_memcpy(&ctx->crypto_send.info,
+		      &offload_ctx->rekey.crypto_send.info,
+		      cipher_desc->crypto_info,
+		      /* checked during rekey setup */);
+
+	/* Start marker: the NIC passes through everything before
+	 * write_seq unencrypted (already SW-encrypted during rekey),
+	 * same as during initial offload setup.
+	 */
+	tls_device_commit_start_marker(sk, offload_ctx,
+				       offload_ctx->rekey.start_marker);
+
+	spin_lock_irqsave(&offload_ctx->lock, flags);
+	clear_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+	clear_bit(TLS_TX_REKEY_READY, &ctx->flags);
+	clear_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+	spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+	/* Switch back to HW offload validator */
+	smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);
+
+	crypto_free_aead(tls_sw_ctx_tx(ctx)->aead_send);
+	WRITE_ONCE(ctx->rekey.sw_ctx, NULL);
+	WRITE_ONCE(ctx->rekey.cipher_ctx, NULL);
+	memzero_explicit(&offload_ctx->rekey, sizeof(offload_ctx->rekey));
+
+	if (deferred)
+		TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYINPROGRESS);
+	TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
+	return 0;
+
+rekey_fail:
+	kfree(offload_ctx->rekey.start_marker);
+	offload_ctx->rekey.start_marker = NULL;
+	spin_lock_irqsave(&offload_ctx->lock, flags);
+	set_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+	clear_bit(TLS_TX_REKEY_READY, &ctx->flags);
+	clear_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+	spin_unlock_irqrestore(&offload_ctx->lock, flags);
+	if (deferred)
+		TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYINPROGRESS);
+	TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYFALLBACK);
+	TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+	TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+
+	return 0;
+}
+
+static int tls_set_device_offload_rekey(struct sock *sk,
+					struct tls_context *ctx,
+					struct net_device *netdev,
+					struct tls_crypto_info *new_crypto_info)
+{
+	struct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);
+	bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+	bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+	bool defer = true;
+	int rc;
+
+	if (!rekey_pending && !rekey_failed)
+		defer = tls_has_unacked_records(offload_ctx) ||
+			tls_is_pending_open_record(ctx);
+
+	if (!offload_ctx->rekey.start_marker) {
+		offload_ctx->rekey.start_marker =
+			kmalloc_obj(*offload_ctx->rekey.start_marker);
+		if (!offload_ctx->rekey.start_marker)
+			return -ENOMEM;
+	}
+
+	down_read(&device_offload_lock);
+
+	rc = tls_device_start_rekey(sk, ctx, offload_ctx, new_crypto_info);
+	if (rc) {
+		up_read(&device_offload_lock);
+		return rc;
+	}
+
+	up_read(&device_offload_lock);
+
+	if (defer) {
+		if (!rekey_pending)
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYINPROGRESS);
+		else
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
+		return 0;
+	}
+
+	return tls_device_complete_rekey(sk, ctx, false);
+}
+
 static int tls_set_device_offload_initial(struct sock *sk,
 					  struct tls_context *ctx,
 					  struct net_device *netdev,
@@ -1189,18 +1593,23 @@ static int tls_set_device_offload_initial(struct sock *sk,
 	return rc;
 }
 
-int tls_set_device_offload(struct sock *sk)
+int tls_set_device_offload(struct sock *sk,
+			   struct tls_crypto_info *new_crypto_info)
 {
+	struct tls_crypto_info *crypto_info, *src_crypto_info;
 	const struct tls_cipher_desc *cipher_desc;
-	struct tls_crypto_info *crypto_info;
 	struct net_device *netdev;
 	struct tls_context *ctx;
 	int rc;
 
 	ctx = tls_get_ctx(sk);
 
-	if (ctx->priv_ctx_tx)
-		return -EEXIST;
+	/* Rekey is only supported for connections that are already
+	 * using HW offload. For SW offload connections, the caller
+	 * should fall back to tls_set_sw_offload() for rekey.
+	 */
+	if (new_crypto_info && ctx->tx_conf != TLS_HW)
+		return -EINVAL;
 
 	netdev = get_netdev_for_sock(sk);
 	if (!netdev) {
@@ -1214,14 +1623,20 @@ int tls_set_device_offload(struct sock *sk)
 	}
 
 	crypto_info = &ctx->crypto_send.info;
-	cipher_desc = get_cipher_desc(crypto_info->cipher_type);
+	src_crypto_info = new_crypto_info ?: crypto_info;
+	cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
 	if (!cipher_desc || !cipher_desc->offloadable) {
 		rc = -EINVAL;
 		goto release_netdev;
 	}
 
-	rc = tls_set_device_offload_initial(sk, ctx, netdev, crypto_info,
-					    cipher_desc);
+	if (new_crypto_info)
+		rc = tls_set_device_offload_rekey(sk, ctx, netdev,
+						  src_crypto_info);
+	else
+		rc = tls_set_device_offload_initial(sk, ctx, netdev,
+						    src_crypto_info,
+						    cipher_desc);
 
 release_netdev:
 	dev_put(netdev);
@@ -1354,7 +1769,10 @@ static int tls_device_down(struct net_device *netdev)
 		/* Stop offloaded TX and switch to the fallback.
 		 * tls_is_skb_tx_device_offloaded will return false.
 		 */
-		WRITE_ONCE(ctx->sk->sk_validate_xmit_skb, tls_validate_xmit_skb_sw);
+		if (!test_bit(TLS_TX_REKEY_PENDING, &ctx->flags) &&
+		    !test_bit(TLS_TX_REKEY_FAILED, &ctx->flags))
+			WRITE_ONCE(ctx->sk->sk_validate_xmit_skb,
+				   tls_validate_xmit_skb_sw);
 
 		/* Stop the RX and TX resync.
 		 * tls_dev_resync must not be called after tls_dev_del.
@@ -1371,9 +1789,12 @@ static int tls_device_down(struct net_device *netdev)
 		synchronize_net();
 
 		/* Release the offload context on the driver side. */
-		if (ctx->tx_conf == TLS_HW)
+		if (ctx->tx_conf == TLS_HW &&
+		    !test_bit(TLS_TX_DEV_CLOSED, &ctx->flags)) {
 			netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
 							TLS_OFFLOAD_CTX_DIR_TX);
+			set_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+		}
 		if (ctx->rx_conf == TLS_HW &&
 		    !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
 			netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
diff --git a/net/tls/tls_device_fallback.c b/net/tls/tls_device_fallback.c
index 1110f7ac6bcb..64ac4ef4012b 100644
--- a/net/tls/tls_device_fallback.c
+++ b/net/tls/tls_device_fallback.c
@@ -435,6 +435,30 @@ struct sk_buff *tls_validate_xmit_skb_sw(struct sock *sk,
 	return tls_sw_fallback(sk, skb);
 }
 
+struct sk_buff *tls_validate_xmit_skb_rekey(struct sock *sk,
+					    struct net_device *dev,
+					    struct sk_buff *skb)
+{
+	struct tls_context *tls_ctx = tls_get_ctx(sk);
+	u32 tcp_seq = ntohl(tcp_hdr(skb)->seq);
+	u32 boundary_seq;
+
+	if (test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags))
+		return skb;
+
+	/* If this packet is at or after the rekey boundary, it's already
+	 * SW-encrypted with the new key, pass through unchanged
+	 */
+	boundary_seq = READ_ONCE(tls_ctx->rekey.boundary_seq);
+	if (!before(tcp_seq, boundary_seq))
+		return skb;
+
+	/* Packet before boundary means retransmit of old data,
+	 * use SW fallback with the old key
+	 */
+	return tls_sw_fallback(sk, skb);
+}
+
 struct sk_buff *tls_encrypt_skb(struct sk_buff *skb)
 {
 	return tls_sw_fallback(skb->sk, skb);
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index a62e94508c8b..536f26d68d9d 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -347,8 +347,14 @@ static void tls_sk_proto_cleanup(struct sock *sk,
 		tls_sw_release_resources_tx(sk);
 		TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
 	} else if (ctx->tx_conf == TLS_HW) {
+		bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+
 		tls_device_free_resources_tx(sk);
-		TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+
+		if (rekey_failed)
+			TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+		else
+			TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
 	}
 
 	if (ctx->rx_conf == TLS_SW) {
@@ -369,6 +375,8 @@ static void tls_sk_proto_close(struct sock *sk, long timeout)
 
 	if (ctx->tx_conf == TLS_SW)
 		tls_sw_cancel_work_tx(ctx);
+	else if (ctx->tx_conf == TLS_HW && ctx->rekey.sw_ctx)
+		tls_sw_cancel_work_tx(ctx);
 
 	lock_sock(sk);
 	free_ctx = ctx->tx_conf != TLS_HW && ctx->rx_conf != TLS_HW;
@@ -445,8 +453,16 @@ static int do_tls_getsockopt_conf(struct sock *sk, sockopt_t *opt, int tx)
 
 	/* get user crypto info */
 	if (tx) {
-		crypto_info = &ctx->crypto_send.info;
-		cctx = &ctx->tx;
+		if (ctx->rekey.cipher_ctx) {
+			struct tls_offload_context_tx *offload_ctx =
+				tls_offload_ctx_tx(ctx);
+
+			crypto_info = &offload_ctx->rekey.crypto_send.info;
+			cctx = &offload_ctx->rekey.tx;
+		} else {
+			crypto_info = &ctx->crypto_send.info;
+			cctx = &ctx->tx;
+		}
 	} else {
 		crypto_info = &ctx->crypto_recv.info;
 		cctx = &ctx->rx;
@@ -710,32 +726,32 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
 	}
 
 	if (tx) {
-		if (update && ctx->tx_conf == TLS_HW) {
-			rc = -EOPNOTSUPP;
-			goto err_crypto_info;
-		}
-
-		if (!update) {
-			rc = tls_set_device_offload(sk);
-			conf = TLS_HW;
-			if (!rc) {
+		rc = tls_set_device_offload(sk, update ? crypto_info : NULL);
+		conf = TLS_HW;
+		if (!rc) {
+			if (!update) {
 				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
 				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
-				goto out;
 			}
-		}
-
-		rc = tls_set_sw_offload(sk, 1, update ? crypto_info : NULL);
-		if (rc)
+		} else if (update && ctx->tx_conf == TLS_HW) {
+			/* HW rekey failed - return the actual error.
+			 * Cannot fall back to SW for an existing HW connection.
+			 */
 			goto err_crypto_info;
-
-		if (update) {
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
 		} else {
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXSW);
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+			rc = tls_set_sw_offload(sk, 1,
+						update ? crypto_info : NULL);
+			if (rc)
+				goto err_crypto_info;
+
+			if (update) {
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
+			} else {
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXSW);
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+			}
+			conf = TLS_SW;
 		}
-		conf = TLS_SW;
 	} else {
 		if (update && ctx->rx_conf == TLS_HW) {
 			rc = -EOPNOTSUPP;
@@ -881,12 +897,34 @@ static int do_tls_setsockopt(struct sock *sk, int optname, sockptr_t optval,
 
 	switch (optname) {
 	case TLS_TX:
-	case TLS_RX:
+	case TLS_RX: {
+		/* A HW-offload TX rekey arrives mid-stream and races with an
+		 * in-flight tls_device_sendmsg(), which holds tx_lock across
+		 * the lock_sock drop in sk_stream_wait_memory() while an
+		 * open_record is half built. start_rekey() would flush that
+		 * record and flip the SW/HW boundary under the sender,
+		 * corrupting record framing. Serialize against the data path
+		 * with tx_lock for all TX setsockopt.
+		 *
+		 * Whether this call ends up a rekey is only known under
+		 * lock_sock (it depends on tx_conf, which transitions there),
+		 * so we cannot make taking tx_lock conditional on that without
+		 * racing a concurrent initial setup on the same socket. Always
+		 * take tx_lock for TLS_TX: it respects the tx_lock -> lock_sock
+		 * ordering used by the data path, and during initial setup
+		 * there is no concurrent sender, so the lock is uncontended.
+		 */
+		bool tx = optname == TLS_TX;
+
+		if (tx)
+			mutex_lock(&tls_get_ctx(sk)->tx_lock);
 		lock_sock(sk);
-		rc = do_tls_setsockopt_conf(sk, optval, optlen,
-					    optname == TLS_TX);
+		rc = do_tls_setsockopt_conf(sk, optval, optlen, tx);
 		release_sock(sk);
+		if (tx)
+			mutex_unlock(&tls_get_ctx(sk)->tx_lock);
 		break;
+	}
 	case TLS_TX_ZEROCOPY_RO:
 		lock_sock(sk);
 		rc = do_tls_setsockopt_tx_zc(sk, optval, optlen);
diff --git a/net/tls/tls_proc.c b/net/tls/tls_proc.c
index 4012c4372d4c..adca8561c7da 100644
--- a/net/tls/tls_proc.c
+++ b/net/tls/tls_proc.c
@@ -27,6 +27,9 @@ static const struct snmp_mib tls_mib_list[] = {
 	SNMP_MIB_ITEM("TlsTxRekeyOk", LINUX_MIB_TLSTXREKEYOK),
 	SNMP_MIB_ITEM("TlsTxRekeyError", LINUX_MIB_TLSTXREKEYERROR),
 	SNMP_MIB_ITEM("TlsRxRekeyReceived", LINUX_MIB_TLSRXREKEYRECEIVED),
+	SNMP_MIB_ITEM("TlsTxRekeyFallback", LINUX_MIB_TLSTXREKEYFALLBACK),
+	SNMP_MIB_ITEM("TlsTxRekeyInProgress", LINUX_MIB_TLSTXREKEYINPROGRESS),
+	SNMP_MIB_ITEM("TlsTxRekeyAborted", LINUX_MIB_TLSTXREKEYABORTED),
 };
 
 static int tls_statistics_seq_show(struct seq_file *seq, void *v)
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 7b6ab992c86e..89f60cedb142 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -522,7 +522,7 @@ static void tls_encrypt_done(void *data, int err)
 		complete(&ctx->async_wait.completion);
 }
 
-static int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
 {
 	if (!atomic_dec_and_test(&ctx->encrypt_pending))
 		crypto_wait_req(-EINPROGRESS, &ctx->async_wait);
@@ -763,8 +763,7 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg,
 	return 0;
 }
 
-static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg,
-				 size_t size)
+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
 {
 	long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
 	struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -2411,6 +2410,23 @@ void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx)
 	sw_ctx->tx_work.sk = sk;
 }
 
+int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx)
+{
+	struct tls_sw_context_tx *sw_ctx = tls_sw_ctx_tx(ctx);
+	int rc;
+
+	if (tls_is_pending_open_record(ctx))
+		tls_sw_push_pending_record(sk, 0);
+	tls_encrypt_async_wait(sw_ctx);
+	rc = tls_tx_records(sk, -1);
+	if (rc < 0 || tls_is_partially_sent_record(ctx) ||
+	    tls_is_pending_open_record(ctx))
+		return rc < 0 ? rc : -EAGAIN;
+
+	cancel_delayed_work_sync(&sw_ctx->tx_work.work);
+	return 0;
+}
+
 static bool tls_is_tx_ready(struct tls_sw_context_tx *ctx)
 {
 	struct tls_rec *rec;
-- 
2.25.1

[PATCH v16 08/10] tls: device: add RX KeyUpdate support

[Rishikesh Jethwani]

On RX, the NIC may have already decrypted in-flight records with
the old key before the peer's KeyUpdate is parsed, so the old
AEAD, IV and rec_seq are retained on tls_offload_context_rx.
tls_device_rx_del_key() is called from tls_check_pending_rekey()
when a KeyUpdate record is decoded; otherwise post-KeyUpdate records
(carrying new-key wire encryption) would be decrypted with the retired key.
tls_device_decrypted() classifies records by old_nic_boundary:

  - after the boundary: new-key record; drop the old key.
  - before, fully encrypted: advance old_rec_seq, let SW AEAD decrypt.
  - before, (partially) decrypted: reencrypt with the old key so SW
    AEAD can decrypt with the new key.

rec_start_seq is the TCP sequence of the record's first byte, used both
for the trace_tls_device_decrypted() tracepoint and the old_nic_boundary
classification above. Because copied_seq is advanced at different points
in the two strparser modes, the record start is computed differently: in
copy_mode the record has already been dequeued (tcp_read_done() in
tls_strp_msg_cow() advanced copied_seq past it), so full_len is
subtracted; in non-copy mode copied_seq still points at the record start
and is used directly. This also corrects the tracepoint's first argument,
which previously subtracted full_len unconditionally and was off by one
record on the non-copy path.

For mixed records the NIC may have cleared skb->decrypted on auth failure;
the optional tls_dev_rx_rekey_fixup callback re-asserts it,
then the record is reencrypted with the old key so the SW AEAD can
decrypt it with the new key.

The new key's tls_dev_add is deferred until the old key is fully
consumed: tls_set_device_offload_rx() sets dev_add_pending while
old_aead_recv is retained, and tls_device_deferred_dev_add_rx()
installs the new key once copied_seq crosses old_nic_boundary.

Tested on Mellanox ConnectX-6 Dx (Crypto Enabled) with multiple
TLS 1.3 RX KeyUpdate cycles.

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 .../mellanox/mlx5/core/en_accel/ktls.c        |  10 +
 include/net/tls.h                             |  19 +-
 include/uapi/linux/snmp.h                     |   3 +
 net/tls/tls.h                                 |   9 +-
 net/tls/tls_device.c                          | 326 ++++++++++++++++--
 net/tls/tls_main.c                            |  46 +--
 net/tls/tls_proc.c                            |   3 +
 net/tls/tls_sw.c                              |   1 +
 8 files changed, 363 insertions(+), 54 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
index 337201f12895..71c9d3ca947d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
@@ -86,10 +86,20 @@ static int mlx5e_ktls_resync(struct net_device *netdev,
 	return 0;
 }
 
+static void mlx5e_ktls_rx_rekey_fixup(struct sk_buff *skb)
+{
+	struct sk_buff *frag_iter;
+
+	skb->decrypted = 1;
+	skb_walk_frags(skb, frag_iter)
+		frag_iter->decrypted = 1;
+}
+
 static const struct tlsdev_ops mlx5e_ktls_ops = {
 	.tls_dev_add = mlx5e_ktls_add,
 	.tls_dev_del = mlx5e_ktls_del,
 	.tls_dev_resync = mlx5e_ktls_resync,
+	.tls_dev_rx_rekey_fixup = mlx5e_ktls_rx_rekey_fixup,
 };
 
 bool mlx5e_is_ktls_rx(struct mlx5_core_dev *mdev)
diff --git a/include/net/tls.h b/include/net/tls.h
index 7c2cf51dc887..de4e25b034d6 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -211,10 +211,11 @@ enum tls_context_flags {
 	 * to be atomic.
 	 */
 	TLS_TX_SYNC_SCHED = 1,
-	/* tls_dev_del was called for the RX side, device state was released,
-	 * but tls_ctx->netdev might still be kept, because TX-side driver
-	 * resources might not be released yet. Used to prevent the second
-	 * tls_dev_del call in tls_device_down if it happens simultaneously.
+	/* tls_dev_del was called for the RX side: the NIC currently holds no
+	 * RX offload context. Set whenever that context is released (socket
+	 * teardown, tls_device_down, or during a rekey before re-add) and
+	 * cleared when tls_dev_add re-establishes it. Readers use it to avoid
+	 * a double tls_dev_del and to suppress resync while the NIC has no key.
 	 */
 	TLS_RX_DEV_CLOSED = 2,
 	/* Flag for TX HW context deleted during failed rekey.
@@ -321,6 +322,7 @@ struct tlsdev_ops {
 	int (*tls_dev_resync)(struct net_device *netdev,
 			      struct sock *sk, u32 seq, u8 *rcd_sn,
 			      enum tls_offload_ctx_dir direction);
+	void (*tls_dev_rx_rekey_fixup)(struct sk_buff *skb);
 };
 
 enum tls_offload_sync_type {
@@ -349,6 +351,15 @@ struct tls_offload_context_rx {
 	u8 resync_nh_reset:1;
 	/* CORE_NEXT_HINT-only member, but use the hole here */
 	u8 resync_nh_do_now:1;
+	/* tls_dev_add deferred until old key is freed */
+	u8 dev_add_pending:1;
+	struct {
+		struct crypto_aead *old_aead_recv; /* old key AEAD cipher */
+		char old_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE]; /* old key IV */
+		char old_rec_seq[TLS_MAX_REC_SEQ_SIZE]; /* old key TLS record seq */
+		u32 old_nic_boundary; /* TCP seq: NIC switched to next key */
+		void (*rekey_fixup)(struct sk_buff *skb);
+	} rekey;
 	union {
 		/* TLS_OFFLOAD_SYNC_TYPE_DRIVER_REQ */
 		struct {
diff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h
index 119fbd49179c..284f879c92cc 100644
--- a/include/uapi/linux/snmp.h
+++ b/include/uapi/linux/snmp.h
@@ -370,8 +370,11 @@ enum
 	LINUX_MIB_TLSTXREKEYERROR,		/* TlsTxRekeyError */
 	LINUX_MIB_TLSRXREKEYRECEIVED,		/* TlsRxRekeyReceived */
 	LINUX_MIB_TLSTXREKEYFALLBACK,		/* TlsTxRekeyFallback */
+	LINUX_MIB_TLSRXREKEYFALLBACK,		/* TlsRxRekeyFallback */
 	LINUX_MIB_TLSTXREKEYINPROGRESS,		/* TlsTxRekeyInProgress */
+	LINUX_MIB_TLSRXREKEYINPROGRESS,		/* TlsRxRekeyInProgress */
 	LINUX_MIB_TLSTXREKEYABORTED,		/* TlsTxRekeyAborted */
+	LINUX_MIB_TLSRXREKEYABORTED,		/* TlsRxRekeyAborted */
 	__LINUX_MIB_TLSMAX
 };
 
diff --git a/net/tls/tls.h b/net/tls/tls.h
index ea8fdf198572..0e2d6baddb4a 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -243,8 +243,10 @@ void tls_device_cleanup(void);
 int tls_set_device_offload(struct sock *sk,
 			   struct tls_crypto_info *crypto_info);
 void tls_device_free_resources_tx(struct sock *sk);
-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx);
+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+			      struct tls_crypto_info *crypto_info);
 void tls_device_offload_cleanup_rx(struct sock *sk);
+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx);
 void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq);
 int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx);
 #else
@@ -260,13 +262,16 @@ tls_set_device_offload(struct sock *sk, struct tls_crypto_info *crypto_info)
 static inline void tls_device_free_resources_tx(struct sock *sk) {}
 
 static inline int
-tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
+tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+			  struct tls_crypto_info *crypto_info)
 {
 	return -EOPNOTSUPP;
 }
 
 static inline void tls_device_offload_cleanup_rx(struct sock *sk) {}
 static inline void
+tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx) {}
+static inline void
 tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq) {}
 
 static inline int
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index 85c74d0816b7..c34268cd9531 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -67,8 +67,18 @@ static void tls_device_free_ctx(struct tls_context *ctx)
 		kfree(offload_ctx);
 	}
 
-	if (ctx->rx_conf == TLS_HW)
-		kfree(tls_offload_ctx_rx(ctx));
+	if (ctx->rx_conf == TLS_HW) {
+		struct tls_offload_context_rx *offload_ctx =
+			tls_offload_ctx_rx(ctx);
+
+		/* Normally freed and NULLed in tls_device_offload_cleanup_rx();
+		 * free defensively here so a future path can't leak the tfm.
+		 */
+		crypto_free_aead(offload_ctx->rekey.old_aead_recv);
+		memzero_explicit(&offload_ctx->rekey,
+				 sizeof(offload_ctx->rekey));
+		kfree(offload_ctx);
+	}
 
 	tls_ctx_free(NULL, ctx);
 }
@@ -186,6 +196,82 @@ static void tls_device_commit_start_marker(struct sock *sk,
 	tcp_write_collapse_fence(sk);
 }
 
+static int tls_device_dev_add_rx(struct sock *sk, struct tls_context *tls_ctx,
+				 struct net_device *netdev,
+				 struct tls_crypto_info *crypto_info,
+				 u32 cur_seq, bool is_rekey)
+{
+	const struct tls_cipher_desc *cipher_desc;
+	char *rec_seq;
+	int rc;
+
+	cipher_desc = get_cipher_desc(crypto_info->cipher_type);
+	DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+	rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk,
+					     TLS_OFFLOAD_CTX_DIR_RX,
+					     crypto_info, cur_seq);
+	rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+	trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
+				     cur_seq, rec_seq, rc);
+	if (!rc) {
+		clear_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
+		clear_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
+		if (is_rekey)
+			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+	} else if (is_rekey) {
+		set_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
+		set_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
+		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYFALLBACK);
+	}
+	return rc;
+}
+
+static void tls_device_deferred_dev_add_rx(struct sock *sk,
+					   struct tls_context *tls_ctx,
+					   struct tls_offload_context_rx *ctx)
+{
+	struct net_device *netdev;
+
+	ctx->dev_add_pending = 0;
+
+	down_read(&device_offload_lock);
+	netdev = rcu_dereference_protected(tls_ctx->netdev,
+					   lockdep_is_held(&device_offload_lock));
+	if (netdev)
+		tls_device_dev_add_rx(sk, tls_ctx, netdev,
+				      &tls_ctx->crypto_recv.info,
+				      tcp_sk(sk)->copied_seq, true);
+	else
+		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYFALLBACK);
+	up_read(&device_offload_lock);
+	TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYINPROGRESS);
+}
+
+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx)
+{
+	struct net_device *netdev;
+
+	if (ctx->rx_conf != TLS_HW)
+		return;
+	if (test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
+		return;
+
+	down_read(&device_offload_lock);
+	netdev = rcu_dereference_protected(ctx->netdev,
+					   lockdep_is_held(&device_offload_lock));
+	if (!netdev) {
+		up_read(&device_offload_lock);
+		return;
+	}
+
+	set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+	synchronize_net();
+	netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+					TLS_OFFLOAD_CTX_DIR_RX);
+	up_read(&device_offload_lock);
+}
+
 static void destroy_record(struct tls_record_info *record)
 {
 	int i;
@@ -904,6 +990,8 @@ void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq)
 		return;
 	if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags)))
 		return;
+	if (unlikely(test_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags)))
+		return;
 
 	prot = &tls_ctx->prot_info;
 	rx_ctx = tls_offload_ctx_rx(tls_ctx);
@@ -1093,13 +1181,56 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
 	return err;
 }
 
+/*
+ * Temporarily swap in the old key, run
+ * tls_device_reencrypt(), then restore the current key.
+ */
+static int tls_device_reencrypt_old_key(struct sock *sk,
+					struct tls_offload_context_rx *ctx,
+					struct tls_sw_context_rx *sw_ctx,
+					struct tls_context *tls_ctx)
+{
+	struct crypto_aead *saved_aead = sw_ctx->aead_recv;
+	char saved_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
+	char saved_rec_seq[TLS_MAX_REC_SEQ_SIZE];
+	int ret;
+
+	memcpy(saved_iv, tls_ctx->rx.iv, sizeof(saved_iv));
+	memcpy(saved_rec_seq, tls_ctx->rx.rec_seq, sizeof(saved_rec_seq));
+
+	sw_ctx->aead_recv = ctx->rekey.old_aead_recv;
+	memcpy(tls_ctx->rx.iv, ctx->rekey.old_iv, sizeof(ctx->rekey.old_iv));
+	memcpy(tls_ctx->rx.rec_seq, ctx->rekey.old_rec_seq,
+	       sizeof(ctx->rekey.old_rec_seq));
+
+	ret = tls_device_reencrypt(sk, tls_ctx);
+
+	memcpy(ctx->rekey.old_rec_seq, tls_ctx->rx.rec_seq,
+	       sizeof(ctx->rekey.old_rec_seq));
+
+	sw_ctx->aead_recv = saved_aead;
+	memcpy(tls_ctx->rx.iv, saved_iv, sizeof(saved_iv));
+	memcpy(tls_ctx->rx.rec_seq, saved_rec_seq, sizeof(saved_rec_seq));
+
+	if (ret)
+		return ret;
+
+	tls_bigint_increment(ctx->rekey.old_rec_seq,
+			     tls_ctx->prot_info.rec_seq_size);
+	ctx->resync_nh_reset = 1;
+
+	return 0;
+}
+
 int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
 {
 	struct tls_offload_context_rx *ctx = tls_offload_ctx_rx(tls_ctx);
 	struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(tls_ctx);
 	struct sk_buff *skb = tls_strp_msg(sw_ctx);
+	u32 copied_seq = tcp_sk(sk)->copied_seq;
 	struct strp_msg *rxm = strp_msg(skb);
 	int is_decrypted, is_encrypted;
+	u32 rec_start_seq;
 
 	if (!tls_strp_msg_mixed_decrypted(sw_ctx)) {
 		is_decrypted = skb->decrypted;
@@ -1109,10 +1240,50 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
 		is_encrypted = 0;
 	}
 
-	trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,
+	rec_start_seq = sw_ctx->strp.copy_mode
+		? copied_seq - rxm->full_len
+		: copied_seq;
+
+	trace_tls_device_decrypted(sk, rec_start_seq,
 				   tls_ctx->rx.rec_seq, rxm->full_len,
 				   is_encrypted, is_decrypted);
 
+	if (unlikely(ctx->rekey.old_aead_recv)) {
+		bool before_nic_boundary =
+			before(rec_start_seq, ctx->rekey.old_nic_boundary);
+
+		if (before_nic_boundary) {
+			/* A non-mixed record with skb->decrypted clear is
+			 * untouched wire ciphertext: the NIC never XORed the
+			 * payload, so SW can decrypt it directly after advancing
+			 * old_rec_seq. Auth-failure clearing (see rekey_fixup
+			 * below) only ever produces mixed records, which are
+			 * classified as neither encrypted nor decrypted and fall
+			 * through to the reencrypt path.
+			 */
+			if (is_encrypted) {
+				tls_bigint_increment(ctx->rekey.old_rec_seq,
+						     tls_ctx->prot_info.rec_seq_size);
+				return 0;
+			}
+
+			/* rekey_fixup sets decrypted flags in case the NIC clears
+			 * decrypted flags on auth failure
+			 */
+			if (!is_decrypted && ctx->rekey.rekey_fixup)
+				ctx->rekey.rekey_fixup(skb);
+
+			return tls_device_reencrypt_old_key(sk, ctx,
+							    sw_ctx, tls_ctx);
+		}
+
+		crypto_free_aead(ctx->rekey.old_aead_recv);
+		ctx->rekey.old_aead_recv = NULL;
+
+		if (ctx->dev_add_pending)
+			tls_device_deferred_dev_add_rx(sk, tls_ctx, ctx);
+	}
+
 	if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags))) {
 		if (likely(is_encrypted || is_decrypted))
 			return is_decrypted;
@@ -1643,13 +1814,31 @@ int tls_set_device_offload(struct sock *sk,
 	return rc;
 }
 
-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+			      struct tls_crypto_info *new_crypto_info)
 {
-	struct tls12_crypto_info_aes_gcm_128 *info;
+	struct tls_crypto_info *crypto_info, *src_crypto_info;
+	const struct tls_cipher_desc *cipher_desc;
+	u32 copied_seq = tcp_sk(sk)->copied_seq;
 	struct tls_offload_context_rx *context;
 	struct net_device *netdev;
+	bool was_dev_add_pending;
+	bool moved_aead_recv = false;
 	int rc = 0;
 
+	/* Rekey is only supported for connections that are already
+	 * using HW offload. For SW offload connections, the caller
+	 * should fall back to tls_set_sw_offload() for rekey.
+	 */
+	if (new_crypto_info && ctx->rx_conf != TLS_HW)
+		return -EINVAL;
+
+	crypto_info = &ctx->crypto_recv.info;
+	src_crypto_info = new_crypto_info ?: crypto_info;
+	cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+	if (!cipher_desc || !cipher_desc->offloadable)
+		return -EINVAL;
+
 	netdev = get_netdev_for_sock(sk);
 	if (!netdev) {
 		pr_err_ratelimited("%s: netdev not found\n", __func__);
@@ -1675,29 +1864,88 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
 		goto release_lock;
 	}
 
-	context = kzalloc_obj(*context);
-	if (!context) {
-		rc = -ENOMEM;
-		goto release_lock;
+	if (!new_crypto_info) {
+		context = kzalloc_obj(*context);
+		if (!context) {
+			rc = -ENOMEM;
+			goto release_lock;
+		}
+		ctx->priv_ctx_rx = context;
+	} else {
+		context = tls_offload_ctx_rx(ctx);
 	}
+	was_dev_add_pending = context->dev_add_pending;
 	context->resync_nh_reset = 1;
 
-	ctx->priv_ctx_rx = context;
-	rc = tls_sw_ctx_init(sk, 0, NULL);
+	if (new_crypto_info) {
+		struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);
+
+		if (!test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
+			set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+			synchronize_net();
+			netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+							TLS_OFFLOAD_CTX_DIR_RX);
+		}
+
+		if (context->rekey.old_aead_recv &&
+		    before(copied_seq, context->rekey.old_nic_boundary)) {
+			/* Previous rekey still draining. Keep rekey.old_aead_recv,
+			 * it is the only key that can undo the NIC-XOR on queued
+			 * records. sw_ctx->aead_recv may be re-setkey'd by
+			 * tls_sw_ctx_init(); that intermediate key was never on
+			 * the NIC and its wire era is drained, so it is needed
+			 * for neither undo nor AEAD. Defer dev_add; the new key
+			 * is installed once copied_seq crosses rekey.old_nic_boundary.
+			 */
+			context->dev_add_pending = 1;
+		} else {
+			u32 rcv_nxt;
+
+			if (context->rekey.old_aead_recv) {
+				crypto_free_aead(context->rekey.old_aead_recv);
+				context->rekey.old_aead_recv = NULL;
+			}
+
+			/* flush the backlog so rcv_nxt is accurate */
+			__sk_flush_backlog(sk);
+			rcv_nxt = tcp_sk(sk)->rcv_nxt;
+
+			if (before(copied_seq, rcv_nxt)) {
+				context->rekey.old_aead_recv = sw_ctx->aead_recv;
+				sw_ctx->aead_recv = NULL;
+				moved_aead_recv = true;
+				memcpy(context->rekey.old_iv, ctx->rx.iv,
+				       sizeof(context->rekey.old_iv));
+				memcpy(context->rekey.old_rec_seq, ctx->rx.rec_seq,
+				       sizeof(context->rekey.old_rec_seq));
+				context->rekey.old_nic_boundary = rcv_nxt;
+				context->rekey.rekey_fixup =
+					netdev->tlsdev_ops->tls_dev_rx_rekey_fixup;
+				context->dev_add_pending = 1;
+			}
+		}
+	}
+
+	rc = tls_sw_ctx_init(sk, 0, new_crypto_info);
 	if (rc)
 		goto release_ctx;
 
-	rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_RX,
-					     &ctx->crypto_recv.info,
-					     tcp_sk(sk)->copied_seq);
-	info = (void *)&ctx->crypto_recv.info;
-	trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
-				     tcp_sk(sk)->copied_seq, info->rec_seq, rc);
-	if (rc)
-		goto free_sw_resources;
+	if (!context->dev_add_pending) {
+		rc = tls_device_dev_add_rx(sk, ctx, netdev, src_crypto_info,
+					   copied_seq, !!new_crypto_info);
+		if (!new_crypto_info) {
+			if (rc)
+				goto free_sw_resources;
+			tls_device_attach(ctx, sk, netdev);
+		}
+	} else if (!was_dev_add_pending) {
+		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYINPROGRESS);
+	} else {
+		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+	}
+
+	tls_sw_ctx_finalize(sk, 0, new_crypto_info);
 
-	tls_device_attach(ctx, sk, netdev);
-	tls_sw_ctx_finalize(sk, 0, NULL);
 	up_read(&device_offload_lock);
 
 	dev_put(netdev);
@@ -1709,7 +1957,18 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
 	tls_sw_free_resources_rx(sk);
 	down_read(&device_offload_lock);
 release_ctx:
-	ctx->priv_ctx_rx = NULL;
+	if (!new_crypto_info) {
+		ctx->priv_ctx_rx = NULL;
+	} else {
+		context->dev_add_pending = was_dev_add_pending;
+		if (moved_aead_recv) {
+			struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);
+
+			crypto_free_aead(sw_ctx->aead_recv);
+			sw_ctx->aead_recv = context->rekey.old_aead_recv;
+			context->rekey.old_aead_recv = NULL;
+		}
+	}
 release_lock:
 	up_read(&device_offload_lock);
 release_netdev:
@@ -1720,6 +1979,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
 void tls_device_offload_cleanup_rx(struct sock *sk)
 {
 	struct tls_context *tls_ctx = tls_get_ctx(sk);
+	struct tls_offload_context_rx *rx_ctx;
 	struct net_device *netdev;
 
 	down_read(&device_offload_lock);
@@ -1728,8 +1988,9 @@ void tls_device_offload_cleanup_rx(struct sock *sk)
 	if (!netdev)
 		goto out;
 
-	netdev->tlsdev_ops->tls_dev_del(netdev, tls_ctx,
-					TLS_OFFLOAD_CTX_DIR_RX);
+	if (!test_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags))
+		netdev->tlsdev_ops->tls_dev_del(netdev, tls_ctx,
+						TLS_OFFLOAD_CTX_DIR_RX);
 
 	if (tls_ctx->tx_conf != TLS_HW) {
 		dev_put(netdev);
@@ -1739,6 +2000,19 @@ void tls_device_offload_cleanup_rx(struct sock *sk)
 	}
 out:
 	up_read(&device_offload_lock);
+
+	rx_ctx = tls_offload_ctx_rx(tls_ctx);
+	if (rx_ctx && rx_ctx->rekey.old_aead_recv) {
+		crypto_free_aead(rx_ctx->rekey.old_aead_recv);
+		rx_ctx->rekey.old_aead_recv = NULL;
+	}
+
+	if (rx_ctx && rx_ctx->dev_add_pending) {
+		rx_ctx->dev_add_pending = 0;
+		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYABORTED);
+		TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYINPROGRESS);
+	}
+
 	tls_sw_release_resources_rx(sk);
 }
 
@@ -1796,9 +2070,11 @@ static int tls_device_down(struct net_device *netdev)
 			set_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
 		}
 		if (ctx->rx_conf == TLS_HW &&
-		    !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
+		    !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
 			netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
 							TLS_OFFLOAD_CTX_DIR_RX);
+			set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+		}
 
 		dev_put(netdev);
 
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index 536f26d68d9d..466117a1530f 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -753,37 +753,37 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
 			conf = TLS_SW;
 		}
 	} else {
-		if (update && ctx->rx_conf == TLS_HW) {
-			rc = -EOPNOTSUPP;
-			goto err_crypto_info;
-		}
-
-		if (!update) {
-			rc = tls_set_device_offload_rx(sk, ctx);
-			conf = TLS_HW;
-			if (!rc) {
+		rc = tls_set_device_offload_rx(sk, ctx,
+					       update ? crypto_info : NULL);
+		conf = TLS_HW;
+		if (!rc) {
+			if (!update) {
 				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
 				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
-				tls_sw_strparser_arm(sk, ctx);
-				goto out;
 			}
-		}
-
-		rc = tls_set_sw_offload(sk, 0, update ? crypto_info : NULL);
-		if (rc)
+		} else if (update && ctx->rx_conf == TLS_HW) {
+			/* HW rekey failed - return the actual error.
+			 * Cannot fall back to SW for an existing HW connection.
+			 */
 			goto err_crypto_info;
-
-		if (update) {
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
 		} else {
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXSW);
-			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
-			tls_sw_strparser_arm(sk, ctx);
+			rc = tls_set_sw_offload(sk, 0,
+						update ? crypto_info : NULL);
+			if (rc)
+				goto err_crypto_info;
+
+			if (update) {
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+			} else {
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXSW);
+				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+			}
+			conf = TLS_SW;
 		}
-		conf = TLS_SW;
+		if (!update)
+			tls_sw_strparser_arm(sk, ctx);
 	}
 
-out:
 	if (tx)
 		ctx->tx_conf = conf;
 	else
diff --git a/net/tls/tls_proc.c b/net/tls/tls_proc.c
index adca8561c7da..ae2322481c71 100644
--- a/net/tls/tls_proc.c
+++ b/net/tls/tls_proc.c
@@ -28,8 +28,11 @@ static const struct snmp_mib tls_mib_list[] = {
 	SNMP_MIB_ITEM("TlsTxRekeyError", LINUX_MIB_TLSTXREKEYERROR),
 	SNMP_MIB_ITEM("TlsRxRekeyReceived", LINUX_MIB_TLSRXREKEYRECEIVED),
 	SNMP_MIB_ITEM("TlsTxRekeyFallback", LINUX_MIB_TLSTXREKEYFALLBACK),
+	SNMP_MIB_ITEM("TlsRxRekeyFallback", LINUX_MIB_TLSRXREKEYFALLBACK),
 	SNMP_MIB_ITEM("TlsTxRekeyInProgress", LINUX_MIB_TLSTXREKEYINPROGRESS),
+	SNMP_MIB_ITEM("TlsRxRekeyInProgress", LINUX_MIB_TLSRXREKEYINPROGRESS),
 	SNMP_MIB_ITEM("TlsTxRekeyAborted", LINUX_MIB_TLSTXREKEYABORTED),
+	SNMP_MIB_ITEM("TlsRxRekeyAborted", LINUX_MIB_TLSRXREKEYABORTED),
 };
 
 static int tls_statistics_seq_show(struct seq_file *seq, void *v)
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 89f60cedb142..21a10a366b57 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -1556,6 +1556,7 @@ static int tls_check_pending_rekey(struct sock *sk, struct tls_context *ctx,
 	if (hs_type == TLS_HANDSHAKE_KEYUPDATE) {
 		struct tls_sw_context_rx *rx_ctx = ctx->priv_ctx_rx;
 
+		tls_device_rx_del_key(sk, ctx);
 		WRITE_ONCE(rx_ctx->key_update_pending, true);
 		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYRECEIVED);
 	}
-- 
2.25.1

[PATCH v16 09/10] tls: device: add tracepoints for the KeyUpdate path

[Rishikesh Jethwani]

Add four trace events covering the rekey state machine in
tls_device.c:

  tls_device_rekey_start         - rekey accepted; inflight=1 means old-key
                                   data is still queued, dev_add deferred
  tls_device_rekey_reencrypt     - old-key undo pass for a boundary record
  tls_device_rekey_done          - boundary crossed, old_aead_recv freed,
                                   deferred dev_add issued if pending
  tls_device_complete_rekey_fail - TX rekey completion failed in sendmsg;
                                   READY is left set and the next sendmsg
                                   retries

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 net/tls/tls_device.c | 17 +++++++-
 net/tls/trace.h      | 98 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 113 insertions(+), 2 deletions(-)

diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index c34268cd9531..9a4c121dbebb 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -760,8 +760,14 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
 	lock_sock(sk);
 
 	/* Old-key records all ACKed; switch back to HW. */
-	if (test_bit(TLS_TX_REKEY_READY, &tls_ctx->flags))
-		tls_device_complete_rekey(sk, tls_ctx, true);
+	if (test_bit(TLS_TX_REKEY_READY, &tls_ctx->flags)) {
+		rc = tls_device_complete_rekey(sk, tls_ctx, true);
+		/* On failure the READY bit is left set; the next sendmsg
+		 * retries.
+		 */
+		if (rc)
+			trace_tls_device_complete_rekey_fail(sk, rc);
+	}
 
 	/* Use SW path if rekey is in progress (PENDING) or if HW rekey
 	 * failed (FAILED).
@@ -1267,6 +1273,9 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
 				return 0;
 			}
 
+			trace_tls_device_rekey_reencrypt(sk, rec_start_seq,
+							 ctx->rekey.old_nic_boundary);
+
 			/* rekey_fixup sets decrypted flags in case the NIC clears
 			 * decrypted flags on auth failure
 			 */
@@ -1277,6 +1286,8 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
 							    sw_ctx, tls_ctx);
 		}
 
+		trace_tls_device_rekey_done(sk, rec_start_seq,
+					    ctx->rekey.old_nic_boundary);
 		crypto_free_aead(ctx->rekey.old_aead_recv);
 		ctx->rekey.old_aead_recv = NULL;
 
@@ -1923,6 +1934,8 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
 					netdev->tlsdev_ops->tls_dev_rx_rekey_fixup;
 				context->dev_add_pending = 1;
 			}
+			trace_tls_device_rekey_start(sk, copied_seq, rcv_nxt,
+						     before(copied_seq, rcv_nxt));
 		}
 	}
 
diff --git a/net/tls/trace.h b/net/tls/trace.h
index 2d8ce4ff3265..2a90b77d75e8 100644
--- a/net/tls/trace.h
+++ b/net/tls/trace.h
@@ -192,6 +192,104 @@ TRACE_EVENT(tls_device_tx_resync_send,
 	)
 );
 
+TRACE_EVENT(tls_device_rekey_start,
+
+	TP_PROTO(struct sock *sk, u32 copied_seq, u32 nic_boundary,
+		 bool inflight),
+
+	TP_ARGS(sk, copied_seq, nic_boundary, inflight),
+
+	TP_STRUCT__entry(
+		__field(	struct sock *,	sk		)
+		__field(	u32,		copied_seq	)
+		__field(	u32,		nic_boundary	)
+		__field(	bool,		inflight	)
+	),
+
+	TP_fast_assign(
+		__entry->sk = sk;
+		__entry->copied_seq = copied_seq;
+		__entry->nic_boundary = nic_boundary;
+		__entry->inflight = inflight;
+	),
+
+	TP_printk(
+		"sk=%p copied_seq=%u nic_boundary=%u inflight=%d",
+		__entry->sk, __entry->copied_seq, __entry->nic_boundary,
+		__entry->inflight
+	)
+);
+
+TRACE_EVENT(tls_device_rekey_reencrypt,
+
+	TP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary),
+
+	TP_ARGS(sk, tcp_seq, nic_boundary),
+
+	TP_STRUCT__entry(
+		__field(	struct sock *,	sk		)
+		__field(	u32,		tcp_seq		)
+		__field(	u32,		nic_boundary	)
+	),
+
+	TP_fast_assign(
+		__entry->sk = sk;
+		__entry->tcp_seq = tcp_seq;
+		__entry->nic_boundary = nic_boundary;
+	),
+
+	TP_printk(
+		"sk=%p tcp_seq=%u nic_boundary=%u",
+		__entry->sk, __entry->tcp_seq, __entry->nic_boundary
+	)
+);
+
+TRACE_EVENT(tls_device_rekey_done,
+
+	TP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary),
+
+	TP_ARGS(sk, tcp_seq, nic_boundary),
+
+	TP_STRUCT__entry(
+		__field(	struct sock *,	sk		)
+		__field(	u32,		tcp_seq		)
+		__field(	u32,		nic_boundary	)
+	),
+
+	TP_fast_assign(
+		__entry->sk = sk;
+		__entry->tcp_seq = tcp_seq;
+		__entry->nic_boundary = nic_boundary;
+	),
+
+	TP_printk(
+		"sk=%p tcp_seq=%u nic_boundary=%u",
+		__entry->sk, __entry->tcp_seq, __entry->nic_boundary
+	)
+);
+
+TRACE_EVENT(tls_device_complete_rekey_fail,
+
+	TP_PROTO(struct sock *sk, int rc),
+
+	TP_ARGS(sk, rc),
+
+	TP_STRUCT__entry(
+		__field(	struct sock *,	sk	)
+		__field(	int,		rc	)
+	),
+
+	TP_fast_assign(
+		__entry->sk = sk;
+		__entry->rc = rc;
+	),
+
+	TP_printk(
+		"sk=%p rc=%d",
+		__entry->sk, __entry->rc
+	)
+);
+
 #endif /* _TLS_TRACE_H_ */
 
 #undef TRACE_INCLUDE_PATH
-- 
2.25.1

[PATCH v16 10/10] selftests: net: add TLS hardware offload test

[Rishikesh Jethwani]

Two-node kTLS HW offload test using NetDrvEpEnv. A C helper binary
acts as TLS client or server; a Python harness drives it and verifies
TLS stat counters (RekeyOk, RekeyReceived, RekeyFallback,
RekeyInProgress, RekeyAborted, RekeyError, DecryptError).

Covers TLS 1.2/1.3 with AES-GCM-128/256, rekey with various buffer
sizes, and burst variants that stress TX rekey (temporary SW phase,
HW reinstall) and RX rekey (boundary tracking, old-key reencryption,
deferred dev_add).

Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
 MAINTAINERS                                   |   2 +
 .../selftests/drivers/net/hw/.gitignore       |   1 +
 .../testing/selftests/drivers/net/hw/Makefile |   2 +
 .../selftests/drivers/net/hw/tls_hw_offload.c | 975 ++++++++++++++++++
 .../drivers/net/hw/tls_hw_offload.py          | 295 ++++++
 5 files changed, 1275 insertions(+)
 create mode 100644 tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
 create mode 100755 tools/testing/selftests/drivers/net/hw/tls_hw_offload.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 08e43bc09735..6e119592b72a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19066,6 +19066,8 @@ F:	Documentation/networking/tls*
 F:	include/net/tls.h
 F:	include/uapi/linux/tls.h
 F:	net/tls/
+F:	tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
+F:	tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
 F:	tools/testing/selftests/net/tls.c
 
 NETWORKING [SOCKETS]
diff --git a/tools/testing/selftests/drivers/net/hw/.gitignore b/tools/testing/selftests/drivers/net/hw/.gitignore
index 46540468a775..f0a5d15b469b 100644
--- a/tools/testing/selftests/drivers/net/hw/.gitignore
+++ b/tools/testing/selftests/drivers/net/hw/.gitignore
@@ -2,3 +2,4 @@
 iou-zcrx
 ncdevmem
 toeplitz
+tls_hw_offload
diff --git a/tools/testing/selftests/drivers/net/hw/Makefile b/tools/testing/selftests/drivers/net/hw/Makefile
index 234db5c2c90c..c274c41f97e8 100644
--- a/tools/testing/selftests/drivers/net/hw/Makefile
+++ b/tools/testing/selftests/drivers/net/hw/Makefile
@@ -15,6 +15,7 @@ endif
 
 TEST_GEN_FILES := \
 	$(COND_GEN_FILES) \
+	tls_hw_offload \
 # end of TEST_GEN_FILES
 
 TEST_PROGS = \
@@ -46,6 +47,7 @@ TEST_PROGS = \
 	rss_drv.py \
 	rss_flow_label.py \
 	rss_input_xfrm.py \
+	tls_hw_offload.py \
 	toeplitz.py \
 	tso.py \
 	userns_devmem.py \
diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
new file mode 100644
index 000000000000..1882e2f073ad
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
@@ -0,0 +1,975 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * TLS Hardware Offload Two-Node Test
+ *
+ * Tests kTLS hardware offload between two physical nodes using
+ * hardcoded keys. Supports TLS 1.2/1.3, AES-GCM-128/256, and rekey.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <limits.h>
+#include <time.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+#include <arpa/inet.h>
+#include <linux/tls.h>
+
+#define TLS_RECORD_TYPE_HANDSHAKE		22
+#define TLS_HANDSHAKE_KEY_UPDATE		0x18
+
+/* Large enough for a TLS 1.3 KeyUpdate handshake record's plaintext. */
+#define MIN_BUF_SIZE   16
+
+/* Initial key material */
+static struct tls12_crypto_info_aes_gcm_128 tls_info_key0_128 = {
+	.info = {
+		.version = TLS_1_3_VERSION,
+		.cipher_type = TLS_CIPHER_AES_GCM_128,
+	},
+	.iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },
+	.key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+		 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10 },
+	.salt = { 0x01, 0x02, 0x03, 0x04 },
+	.rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
+};
+
+static struct tls12_crypto_info_aes_gcm_256 tls_info_key0_256 = {
+	.info = {
+		.version = TLS_1_3_VERSION,
+		.cipher_type = TLS_CIPHER_AES_GCM_256,
+	},
+	.iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },
+	.key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+		 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10,
+		 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18,
+		 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20 },
+	.salt = { 0x01, 0x02, 0x03, 0x04 },
+	.rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
+};
+
+static int num_rekeys;
+static int num_iterations = 100;
+static int cipher_type = TLS_CIPHER_AES_GCM_128;
+static int tls_version = TLS_1_3_VERSION;
+static int server_port = 4433;
+static char *server_ip;
+
+static int send_size = 16384;
+static int random_size_max;
+/* Burst mode: sender keeps pushing records without reading from the peer;
+ * receiver drains without echoing back. Only the client initiates rekey.
+ */
+static int burst_mode;
+static int zc_rx;
+
+/* XOR each byte with the generation so both endpoints derive the
+ * same per-generation key without a real KDF. Generation 0 leaves
+ * the base key unchanged.
+ */
+static void derive_key_fields(unsigned char *key, int key_size,
+			      unsigned char *iv, int iv_size,
+			      unsigned char *salt, int salt_size,
+			      unsigned char *rec_seq, int rec_seq_size,
+			      int generation)
+{
+	int i;
+
+	for (i = 0; i < key_size; i++)
+		key[i] ^= generation;
+	for (i = 0; i < iv_size; i++)
+		iv[i] ^= generation;
+	for (i = 0; i < salt_size; i++)
+		salt[i] ^= generation;
+	memset(rec_seq, 0, rec_seq_size);
+}
+
+static void derive_key_128(struct tls12_crypto_info_aes_gcm_128 *key,
+			   int generation)
+{
+	memcpy(key, &tls_info_key0_128, sizeof(*key));
+	key->info.version = tls_version;
+	derive_key_fields(key->key, TLS_CIPHER_AES_GCM_128_KEY_SIZE,
+			  key->iv, TLS_CIPHER_AES_GCM_128_IV_SIZE,
+			  key->salt, TLS_CIPHER_AES_GCM_128_SALT_SIZE,
+			  key->rec_seq, TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE,
+			  generation);
+}
+
+static void derive_key_256(struct tls12_crypto_info_aes_gcm_256 *key,
+			   int generation)
+{
+	memcpy(key, &tls_info_key0_256, sizeof(*key));
+	key->info.version = tls_version;
+	derive_key_fields(key->key, TLS_CIPHER_AES_GCM_256_KEY_SIZE,
+			  key->iv, TLS_CIPHER_AES_GCM_256_IV_SIZE,
+			  key->salt, TLS_CIPHER_AES_GCM_256_SALT_SIZE,
+			  key->rec_seq, TLS_CIPHER_AES_GCM_256_REC_SEQ_SIZE,
+			  generation);
+}
+
+static const char *cipher_name(int cipher)
+{
+	switch (cipher) {
+	case TLS_CIPHER_AES_GCM_128: return "AES-GCM-128";
+	case TLS_CIPHER_AES_GCM_256: return "AES-GCM-256";
+	default: return "unknown";
+	}
+}
+
+static const char *version_name(int version)
+{
+	switch (version) {
+	case TLS_1_2_VERSION: return "TLS 1.2";
+	case TLS_1_3_VERSION: return "TLS 1.3";
+	default: return "unknown";
+	}
+}
+
+static int setup_tls_ulp(int fd)
+{
+	int ret;
+
+	ret = setsockopt(fd, IPPROTO_TCP, TCP_ULP, "tls", sizeof("tls"));
+	if (ret < 0) {
+		printf("SETUP ERROR: TCP_ULP failed: %s\n", strerror(errno));
+		return -1;
+	}
+	return 0;
+}
+
+static int set_zc_rx(int fd)
+{
+	int val = 1;
+
+	if (setsockopt(fd, SOL_TLS, TLS_RX_EXPECT_NO_PAD, &val,
+		       sizeof(val)) < 0) {
+		printf("SETUP ERROR: TLS_RX_EXPECT_NO_PAD failed: %s\n",
+		       strerror(errno));
+		return -1;
+	}
+	return 0;
+}
+
+/* Send a TLS 1.3 KeyUpdate handshake record. The kernel only
+ * inspects the HandshakeType byte to detect KeyUpdate, so don't
+ * bother with the 3-byte length or request_update fields.
+ */
+static int send_tls_key_update(int fd)
+{
+	char cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];
+	unsigned char key_update_msg = TLS_HANDSHAKE_KEY_UPDATE;
+	struct msghdr msg = {0};
+	struct cmsghdr *cmsg;
+	struct iovec iov;
+
+	iov.iov_base = &key_update_msg;
+	iov.iov_len = sizeof(key_update_msg);
+
+	msg.msg_iov = &iov;
+	msg.msg_iovlen = 1;
+	msg.msg_control = cmsg_buf;
+	msg.msg_controllen = sizeof(cmsg_buf);
+
+	cmsg = CMSG_FIRSTHDR(&msg);
+	cmsg->cmsg_level = SOL_TLS;
+	cmsg->cmsg_type = TLS_SET_RECORD_TYPE;
+	cmsg->cmsg_len = CMSG_LEN(sizeof(unsigned char));
+	*CMSG_DATA(cmsg) = TLS_RECORD_TYPE_HANDSHAKE;
+	msg.msg_controllen = cmsg->cmsg_len;
+
+	if (sendmsg(fd, &msg, 0) < 0) {
+		printf("sendmsg KeyUpdate failed: %s\n", strerror(errno));
+		return -1;
+	}
+
+	printf("Sent TLS KeyUpdate handshake message\n");
+	return 0;
+}
+
+static int recv_tls_message(int fd, char *buf, size_t buflen, int *record_type,
+			    int flags)
+{
+	char cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];
+	struct msghdr msg = {0};
+	struct cmsghdr *cmsg;
+	struct iovec iov;
+	int ret;
+
+	iov.iov_base = buf;
+	iov.iov_len = buflen;
+
+	msg.msg_iov = &iov;
+	msg.msg_iovlen = 1;
+	msg.msg_control = cmsg_buf;
+	msg.msg_controllen = sizeof(cmsg_buf);
+
+	ret = recvmsg(fd, &msg, flags);
+	if (ret <= 0)
+		return ret;
+
+	cmsg = CMSG_FIRSTHDR(&msg);
+	if (cmsg && cmsg->cmsg_level == SOL_TLS &&
+	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE)
+		*record_type = *((unsigned char *)CMSG_DATA(cmsg));
+
+	return ret;
+}
+
+/* Confirm a handshake record starting with HandshakeType KeyUpdate. */
+static int check_keyupdate(const char *buf, int len, int record_type)
+{
+	if (record_type != TLS_RECORD_TYPE_HANDSHAKE) {
+		printf("Expected handshake record (0x%02x), got 0x%02x\n",
+		       TLS_RECORD_TYPE_HANDSHAKE, record_type);
+		return -1;
+	}
+	if (len < 1 || (unsigned char)buf[0] != TLS_HANDSHAKE_KEY_UPDATE) {
+		printf("Expected KeyUpdate (0x%02x), got 0x%02x\n",
+		       TLS_HANDSHAKE_KEY_UPDATE,
+		       len ? (unsigned char)buf[0] : 0);
+		return -1;
+	}
+	printf("Received TLS KeyUpdate\n");
+	return 0;
+}
+
+static int recv_tls_keyupdate(int fd)
+{
+	char buf[MIN_BUF_SIZE];
+	int record_type = 0;
+	int ret;
+
+	ret = recv_tls_message(fd, buf, sizeof(buf), &record_type, 0);
+	if (ret < 0) {
+		printf("recv_tls_message failed: %s\n", strerror(errno));
+		return -1;
+	}
+
+	return check_keyupdate(buf, ret, record_type);
+}
+
+static int check_ekeyexpired(int fd)
+{
+	char buf[MIN_BUF_SIZE];
+	int ret;
+
+	ret = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
+	if (ret == -1 && errno == EKEYEXPIRED) {
+		printf("recv() returned EKEYEXPIRED as expected\n");
+		return 0;
+	} else if (ret == -1 && errno == EAGAIN) {
+		printf("recv() returned EAGAIN (no pending data)\n");
+		return 0;
+	} else if (ret > 0) {
+		printf("FAIL: recv() returned %d bytes, expected EKEYEXPIRED\n",
+		       ret);
+		return -1;
+	} else {
+		printf("FAIL: recv() returned unexpected error: %s\n",
+		       strerror(errno));
+		return -1;
+	}
+}
+
+static int do_tls_rekey(int fd, int direction, int generation, int cipher)
+{
+	const char *dir = direction == TLS_TX ? "TX" : "RX";
+	int ret;
+
+	printf("%s TLS_%s %s gen %d...\n",
+	       generation ? "Rekeying" : "Installing",
+	       dir, cipher_name(cipher), generation);
+
+	if (cipher == TLS_CIPHER_AES_GCM_256) {
+		struct tls12_crypto_info_aes_gcm_256 key;
+
+		derive_key_256(&key, generation);
+		ret = setsockopt(fd, SOL_TLS, direction, &key, sizeof(key));
+	} else {
+		struct tls12_crypto_info_aes_gcm_128 key;
+
+		derive_key_128(&key, generation);
+		ret = setsockopt(fd, SOL_TLS, direction, &key, sizeof(key));
+	}
+
+	if (ret < 0) {
+		printf("%sTLS_%s %s gen %d failed: %s\n",
+		       generation ? "" : "SETUP ERROR: ", dir,
+		       cipher_name(cipher), generation, strerror(errno));
+		return -1;
+	}
+	printf("TLS_%s %s gen %d installed\n",
+	       dir, cipher_name(cipher), generation);
+	return 0;
+}
+
+/* Open a TCP connection to server_ip:server_port, switch to the TLS
+ * ULP, and install initial generation-0 TX/RX keys. Returns the fd on
+ * success, -1 on error (with the fd already closed).
+ */
+static int client_connect_tls(void)
+{
+	struct sockaddr_in sa;
+	int csk;
+
+	csk = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
+	if (csk < 0) {
+		printf("SETUP ERROR: failed to create socket: %s\n",
+		       strerror(errno));
+		return -1;
+	}
+
+	memset(&sa, 0, sizeof(sa));
+	sa.sin_family = AF_INET;
+	sa.sin_addr.s_addr = inet_addr(server_ip);
+	sa.sin_port = htons(server_port);
+	printf("Connecting to %s:%d...\n", server_ip, server_port);
+
+	if (connect(csk, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
+		printf("SETUP ERROR: connect failed: %s\n", strerror(errno));
+		goto err;
+	}
+	printf("Connected!\n");
+
+	if (setup_tls_ulp(csk) < 0)
+		goto err;
+
+	if (do_tls_rekey(csk, TLS_TX, 0, cipher_type) < 0 ||
+	    do_tls_rekey(csk, TLS_RX, 0, cipher_type) < 0)
+		goto err;
+
+	return csk;
+err:
+	close(csk);
+	return -1;
+}
+
+/* Drain `len` echoed bytes from the server and verify they match the
+ * payload we just sent.
+ */
+static int client_recv_echo(int fd, const char *sent, char *echo_buf,
+			    ssize_t len)
+{
+	ssize_t total = 0;
+	ssize_t n;
+
+	while (total < len) {
+		n = recv(fd, echo_buf + total, len - total, 0);
+		if (n < 0) {
+			printf("FAIL: Echo recv failed: %s\n", strerror(errno));
+			return -1;
+		}
+		if (n == 0) {
+			printf("FAIL: Connection closed during echo\n");
+			return -1;
+		}
+		total += n;
+	}
+
+	if (memcmp(sent, echo_buf, len) != 0) {
+		printf("FAIL: Echo data mismatch!\n");
+		return -1;
+	}
+	printf("Received echo %zd bytes (ok)\n", total);
+	return 0;
+}
+
+/* Client side of a rekey: send KeyUpdate and rotate TX. In echo mode
+ * also wait for the peer's KeyUpdate and rotate RX.
+ */
+static int client_rekey(int fd, int generation)
+{
+	if (send_tls_key_update(fd) < 0) {
+		printf("FAIL: send KeyUpdate\n");
+		return -1;
+	}
+
+	if (do_tls_rekey(fd, TLS_TX, generation, cipher_type) < 0)
+		return -1;
+
+	if (burst_mode)
+		return 0;
+
+	if (recv_tls_keyupdate(fd) < 0) {
+		printf("FAIL: recv KeyUpdate from server\n");
+		return -1;
+	}
+
+	if (check_ekeyexpired(fd) < 0)
+		return -1;
+
+	return do_tls_rekey(fd, TLS_RX, generation, cipher_type);
+}
+
+static int do_client(void)
+{
+	char *buf = NULL, *echo_buf = NULL;
+	int max_size, rekey_interval;
+	int csk = -1, i;
+	int test_result = -1;
+	int current_gen = 0;
+	int next_rekey_at;
+	ssize_t n;
+
+	max_size = random_size_max > 0 ? random_size_max : send_size;
+	if (max_size < MIN_BUF_SIZE)
+		max_size = MIN_BUF_SIZE;
+	buf = malloc(max_size);
+	if (!burst_mode)
+		echo_buf = malloc(max_size);
+	if (!buf || (!burst_mode && !echo_buf)) {
+		printf("SETUP ERROR: failed to allocate buffers\n");
+		goto out;
+	}
+
+	csk = client_connect_tls();
+	if (csk < 0)
+		goto out;
+
+	if (num_rekeys)
+		printf("TLS %s setup complete. Will perform %d rekey(s).\n",
+		       cipher_name(cipher_type), num_rekeys);
+	else
+		printf("TLS setup complete.\n");
+
+	if (random_size_max > 0)
+		printf("Sending %d messages of random size (1..%d bytes)...\n",
+		       num_iterations, random_size_max);
+	else
+		printf("Sending %d messages of %d bytes...\n",
+		       num_iterations, send_size);
+
+	rekey_interval = num_iterations / (num_rekeys + 1);
+	next_rekey_at = rekey_interval;
+
+	for (i = 1; i <= num_iterations; i++) {
+		int this_size;
+
+		if (random_size_max > 0)
+			this_size = (rand() % random_size_max) + 1;
+		else
+			this_size = send_size;
+
+		/* In burst mode, use a per-iteration fill pattern so the
+		 * receiver can detect any plaintext corruption without a
+		 * round-trip echo.
+		 */
+		if (burst_mode) {
+			memset(buf, i & 0xFF, this_size);
+		} else {
+			int j;
+
+			for (j = 0; j < this_size; j++)
+				buf[j] = rand() & 0xFF;
+		}
+
+		n = send(csk, buf, this_size, 0);
+		if (n != this_size) {
+			printf("FAIL: send failed: %s\n", strerror(errno));
+			goto out;
+		}
+
+		if (!burst_mode) {
+			printf("Sent %zd bytes (iteration %d)\n", n, i);
+			if (client_recv_echo(csk, buf, echo_buf, n) < 0)
+				goto out;
+		}
+
+		/* Rekey at intervals. In echo mode this is a full bidirectional
+		 * exchange; in burst mode the client only rotates its TX key
+		 * and sends KeyUpdate - the peer is expected to follow.
+		 */
+		if (num_rekeys && current_gen < num_rekeys &&
+		    i == next_rekey_at) {
+			current_gen++;
+			printf("\n=== Client Rekey gen %d ===\n", current_gen);
+
+			if (client_rekey(csk, current_gen) < 0)
+				goto out;
+
+			next_rekey_at += rekey_interval;
+			printf("=== Client Rekey gen %d Complete ===\n\n",
+			       current_gen);
+		}
+	}
+
+	test_result = 0;
+out:
+	if (num_rekeys)
+		printf("Rekeys completed: %d/%d\n", current_gen, num_rekeys);
+	if (csk >= 0)
+		close(csk);
+	free(buf);
+	free(echo_buf);
+	return test_result;
+}
+
+/* Bind/listen on server_port, accept one client, switch to the TLS ULP
+ * and install initial generation-0 keys (plus zc_rx if requested).
+ * Returns the connected fd on success and writes the listener fd to
+ * *lsk_out so the caller can close it. Returns -1 on error, with all
+ * intermediate fds already closed and *lsk_out left at -1.
+ */
+static int server_accept_tls(int *lsk_out)
+{
+	int lsk, csk, one = 1;
+	struct sockaddr_in sa;
+
+	*lsk_out = -1;
+
+	lsk = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
+	if (lsk < 0) {
+		printf("SETUP ERROR: failed to create socket: %s\n",
+		       strerror(errno));
+		return -1;
+	}
+
+	setsockopt(lsk, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
+
+	memset(&sa, 0, sizeof(sa));
+	sa.sin_family = AF_INET;
+	sa.sin_addr.s_addr = INADDR_ANY;
+	sa.sin_port = htons(server_port);
+
+	if (bind(lsk, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
+		printf("SETUP ERROR: bind failed: %s\n", strerror(errno));
+		close(lsk);
+		return -1;
+	}
+
+	if (listen(lsk, 1) < 0) {
+		printf("SETUP ERROR: listen failed: %s\n", strerror(errno));
+		close(lsk);
+		return -1;
+	}
+
+	printf("Server listening on 0.0.0.0:%d\n", server_port);
+	printf("Waiting for client connection...\n");
+
+	csk = accept(lsk, (struct sockaddr *)NULL, (socklen_t *)NULL);
+	if (csk < 0) {
+		printf("SETUP ERROR: accept failed: %s\n", strerror(errno));
+		close(lsk);
+		return -1;
+	}
+	printf("Client connected!\n");
+
+	if (setup_tls_ulp(csk) < 0)
+		goto err;
+
+	if (do_tls_rekey(csk, TLS_TX, 0, cipher_type) < 0 ||
+	    do_tls_rekey(csk, TLS_RX, 0, cipher_type) < 0)
+		goto err;
+
+	if (zc_rx && set_zc_rx(csk) < 0)
+		goto err;
+
+	*lsk_out = lsk;
+	return csk;
+err:
+	close(csk);
+	close(lsk);
+	return -1;
+}
+
+/* Server side of a rekey: drain any in-flight ciphertext that hit
+ * EKEYEXPIRED and rotate RX. In echo mode also send a KeyUpdate back
+ * and rotate TX.
+ */
+static int server_rekey(int fd, int generation)
+{
+	if (check_ekeyexpired(fd) < 0)
+		return -1;
+
+	if (do_tls_rekey(fd, TLS_RX, generation, cipher_type) < 0)
+		return -1;
+
+	if (burst_mode)
+		return 0;
+
+	if (send_tls_key_update(fd) < 0) {
+		printf("FAIL: send KeyUpdate\n");
+		return -1;
+	}
+
+	return do_tls_rekey(fd, TLS_TX, generation, cipher_type);
+}
+
+/* Burst mode: MSG_WAITALL gives us exactly one iteration's payload,
+ * filled with (send_iter & 0xff). Catches decrypt-succeeded-but-
+ * plaintext-corrupt bugs that AEAD counters alone would miss.
+ */
+static int server_verify_burst(int fd, char *buf, int buf_size,
+			       ssize_t n, int send_iter)
+{
+	unsigned char expect = send_iter & 0xFF;
+	int j;
+
+	if (n != send_size) {
+		int record_type = 0;
+		ssize_t n2;
+
+		/* MSG_WAITALL returned short; probe one more recv to
+		 * surface any pending socket error for the diagnostic.
+		 */
+		n2 = recv_tls_message(fd, buf, buf_size, &record_type, 0);
+		printf("FAIL: short recv in burst mode: got %zd, expected %d (iter %d)\n",
+		       n, send_size, send_iter);
+		printf("      follow-up recv: %zd errno=%s\n",
+		       n2, n2 < 0 ? strerror(errno) : "ok");
+		return -1;
+	}
+
+	for (j = 0; j < n; j++) {
+		if ((unsigned char)buf[j] != expect) {
+			printf("FAIL: data mismatch iter %d offset %d: expected 0x%02x got 0x%02x\n",
+			       send_iter, j, expect, (unsigned char)buf[j]);
+			return -1;
+		}
+	}
+	return 0;
+}
+
+static int server_echo_send(int fd, const char *buf, ssize_t n)
+{
+	ssize_t sent;
+	int ret;
+
+	for (sent = 0; sent < n; sent += ret) {
+		ret = send(fd, buf + sent, n - sent, 0);
+		if (ret < 0) {
+			printf("FAIL: Echo send failed: %s\n", strerror(errno));
+			return -1;
+		}
+	}
+	return 0;
+}
+
+static int do_server(void)
+{
+	int lsk = -1, csk = -1;
+	ssize_t n, total = 0;
+	int test_result = -1;
+	int current_gen = 0;
+	int recv_count = 0;
+	int send_iter = 1;
+	char *buf = NULL;
+	int record_type = 0;
+	int recv_flags;
+	int buf_size;
+
+	buf_size = send_size;
+	if (buf_size < MIN_BUF_SIZE)
+		buf_size = MIN_BUF_SIZE;
+	buf = malloc(buf_size);
+	if (!buf) {
+		printf("SETUP ERROR: failed to allocate buffer\n");
+		goto out;
+	}
+
+	csk = server_accept_tls(&lsk);
+	if (csk < 0)
+		goto out;
+
+	printf("TLS %s setup complete. Receiving...\n",
+	       cipher_name(cipher_type));
+
+	/* Burst mode: ask for a full iteration's worth of plaintext per
+	 * recv. kTLS accumulates across data records when MSG_WAITALL is
+	 * set (target == len), and breaks cleanly at control records, so
+	 * each recv returns exactly send_size data bytes or a small KU.
+	 */
+	recv_flags = burst_mode ? MSG_WAITALL : 0;
+
+	/* Main receive loop */
+	while (1) {
+		n = recv_tls_message(csk, buf, buf_size, &record_type,
+				     recv_flags);
+		if (n == 0) {
+			printf("Connection closed by client\n");
+			break;
+		}
+		if (n < 0) {
+			printf("FAIL: recv failed: %s\n", strerror(errno));
+			goto out;
+		}
+
+		/* Handle KeyUpdate. In echo mode the server mirrors the
+		 * rekey back to the peer; in burst mode it only rotates
+		 * its RX key and keeps draining.
+		 */
+		if (record_type == TLS_RECORD_TYPE_HANDSHAKE) {
+			if (check_keyupdate(buf, n, record_type) < 0)
+				goto out;
+			current_gen++;
+			printf("\n=== Server Rekey gen %d ===\n", current_gen);
+
+			if (server_rekey(csk, current_gen) < 0)
+				goto out;
+
+			printf("=== Server Rekey gen %d Complete ===\n\n",
+			       current_gen);
+			continue;
+		}
+
+		total += n;
+		recv_count++;
+
+		if (burst_mode) {
+			if (server_verify_burst(csk, buf, buf_size, n,
+						send_iter) < 0)
+				goto out;
+			send_iter++;
+			continue;
+		}
+
+		printf("Received %zd bytes (total: %zd, count: %d)\n",
+		       n, total, recv_count);
+
+		if (server_echo_send(csk, buf, n) < 0)
+			goto out;
+		printf("Echoed %zd bytes back to client\n", n);
+	}
+
+	test_result = 0;
+out:
+	printf("Connection closed. Total received: %zd bytes\n", total);
+	if (num_rekeys)
+		printf("Rekeys completed: %d\n", current_gen);
+
+	if (csk >= 0)
+		close(csk);
+	if (lsk >= 0)
+		close(lsk);
+	free(buf);
+	return test_result;
+}
+
+static int parse_int_arg(const char *arg, int min, int max,
+			 const char *name, int *out)
+{
+	char *endp;
+	long val;
+
+	errno = 0;
+	val = strtol(arg, &endp, 10);
+	if (errno || endp == arg || *endp != '\0' || val < min || val > max) {
+		if (max == INT_MAX)
+			printf("ERROR: Invalid %s '%s'. Must be >= %d.\n",
+			       name, arg, min);
+		else
+			printf("ERROR: Invalid %s '%s'. Must be %d..%d.\n",
+			       name, arg, min, max);
+		return -1;
+	}
+	*out = (int)val;
+	return 0;
+}
+
+static int parse_cipher_option(const char *arg)
+{
+	if (strcmp(arg, "128") == 0) {
+		cipher_type = TLS_CIPHER_AES_GCM_128;
+		return 0;
+	} else if (strcmp(arg, "256") == 0) {
+		cipher_type = TLS_CIPHER_AES_GCM_256;
+		return 0;
+	}
+	printf("ERROR: Invalid cipher '%s'. Must be 128 or 256.\n", arg);
+	return -1;
+}
+
+static int parse_version_option(const char *arg)
+{
+	if (strcmp(arg, "1.2") == 0) {
+		tls_version = TLS_1_2_VERSION;
+		return 0;
+	} else if (strcmp(arg, "1.3") == 0) {
+		tls_version = TLS_1_3_VERSION;
+		return 0;
+	}
+	printf("ERROR: Invalid TLS version '%s'. Must be 1.2 or 1.3.\n", arg);
+	return -1;
+}
+
+static void print_usage(const char *prog)
+{
+	printf("TLS Hardware Offload Two-Node Test\n\n");
+	printf("Usage:\n");
+	printf("  %s server [OPTIONS]\n", prog);
+	printf("  %s client -s <ip> [OPTIONS]\n", prog);
+	printf("\nOptions:\n");
+	printf("  -s <ip>       Server IPv4 address (client, required)\n");
+	printf("  -p <port>     Server port (default: 4433)\n");
+	printf("  -b <size>     Send buffer size in bytes (default: 16384)\n");
+	printf("  -r <max>      Use random send buffer sizes (1..<max>)\n");
+	printf("  -v <version>  TLS version: 1.2 or 1.3 (default: 1.3)\n");
+	printf("  -c <cipher>   Cipher: 128 or 256 (default: 128)\n");
+	printf("  -n <N>        Number of send/echo iterations (default: 100)\n");
+	printf("  -k <N>        Perform N rekeys (client only, TLS 1.3; N < iterations)\n");
+	printf("  -B            Burst mode: client sends continuously without echo;\n");
+	printf("                server drains and handles KeyUpdate without responding.\n");
+	printf("  -Z            Set TLS_RX_EXPECT_NO_PAD on the server: TLS 1.3\n");
+	printf("                opt-in to the zero-copy RX fast path. Not needed\n");
+	printf("                for TLS 1.2 (always eligible). Server only.\n");
+	printf("  -h            Show this help message\n");
+	printf("\nExample:\n");
+	printf("  Node A: %s server\n", prog);
+	printf("  Node B: %s client -s 192.168.20.2\n", prog);
+	printf("\nRekey Example (3 rekeys, TLS 1.3 only):\n");
+	printf("  Node A: %s server\n", prog);
+	printf("  Node B: %s client -s 192.168.20.2 -k 3\n", prog);
+	printf("\nBurst Mode Example (client stresses TX rekey under load):\n");
+	printf("  Node A: %s server -B\n", prog);
+	printf("  Node B: %s client -s 192.168.20.2 -B -k 3\n", prog);
+}
+
+int main(int argc, char *argv[])
+{
+	int send_size_set = 0;
+	int is_server;
+	int opt;
+
+	if (argc < 2 ||
+	    (strcmp(argv[1], "server") && strcmp(argv[1], "client"))) {
+		print_usage(argv[0]);
+		return 1;
+	}
+	is_server = !strcmp(argv[1], "server");
+
+	optind = 2; /* skip subcommand */
+	while ((opt = getopt(argc, argv, "s:p:b:r:c:v:k:n:BZh")) != -1) {
+		switch (opt) {
+		case 's':
+			server_ip = optarg;
+			break;
+		case 'B':
+			burst_mode = 1;
+			break;
+		case 'Z':
+			zc_rx = 1;
+			break;
+		case 'p':
+			if (parse_int_arg(optarg, 1, 65535, "port",
+					  &server_port) < 0)
+				return 1;
+			break;
+		case 'b':
+			if (parse_int_arg(optarg, 1, INT_MAX, "buffer size",
+					  &send_size) < 0)
+				return 1;
+			send_size_set = 1;
+			break;
+		case 'r':
+			if (parse_int_arg(optarg, 1, INT_MAX, "random size",
+					  &random_size_max) < 0)
+				return 1;
+			break;
+		case 'c':
+			if (parse_cipher_option(optarg) < 0)
+				return 1;
+			break;
+		case 'v':
+			if (parse_version_option(optarg) < 0)
+				return 1;
+			break;
+		case 'k':
+			if (parse_int_arg(optarg, 1, 255, "rekey count",
+					  &num_rekeys) < 0)
+				return 1;
+			break;
+		case 'n':
+			if (parse_int_arg(optarg, 1, INT_MAX, "iteration count",
+					  &num_iterations) < 0)
+				return 1;
+			break;
+		case 'h':
+			print_usage(argv[0]);
+			return 0;
+		default:
+			print_usage(argv[0]);
+			return 1;
+		}
+	}
+
+	if (send_size_set && random_size_max > 0) {
+		printf("ERROR: -b and -r are mutually exclusive\n");
+		return 1;
+	}
+
+	if (zc_rx && tls_version != TLS_1_3_VERSION) {
+		printf("ERROR: -Z (TLS_RX_EXPECT_NO_PAD) requires TLS 1.3\n");
+		return 1;
+	}
+
+	if (burst_mode && random_size_max > 0) {
+		printf("ERROR: -B and -r are mutually exclusive\n");
+		return 1;
+	}
+
+	if (burst_mode && send_size < MIN_BUF_SIZE) {
+		printf("ERROR: -b must be >= %d in burst mode (-B)\n",
+		       MIN_BUF_SIZE);
+		return 1;
+	}
+
+	if (is_server) {
+		if (server_ip) {
+			printf("warning: -s is ignored in server mode\n");
+			server_ip = NULL;
+		}
+		if (random_size_max > 0) {
+			printf("warning: -r is ignored in server mode\n");
+			random_size_max = 0;
+		}
+		if (num_rekeys) {
+			printf("warning: -k is ignored in server mode\n");
+			num_rekeys = 0;
+		}
+	} else {
+		if (!server_ip) {
+			printf("ERROR: Client requires -s <ip> option\n");
+			return 1;
+		}
+		if (tls_version == TLS_1_2_VERSION && num_rekeys) {
+			printf("ERROR: TLS 1.2 does not support rekey\n");
+			return 1;
+		}
+		if (num_rekeys >= num_iterations) {
+			printf("ERROR: num_rekeys (%d) must be < num_iterations (%d)\n",
+			       num_rekeys, num_iterations);
+			return 1;
+		}
+		if (zc_rx) {
+			printf("ERROR: -Z applies to the server (receiver) only\n");
+			return 1;
+		}
+	}
+
+	printf("TLS Version: %s\n", version_name(tls_version));
+	printf("Cipher: %s\n", cipher_name(cipher_type));
+	if (random_size_max > 0)
+		printf("Buffer size: random (1..%d)\n", random_size_max);
+	else
+		printf("Buffer size: %d\n", send_size);
+
+	if (num_rekeys)
+		printf("Rekey testing ENABLED: %d rekey(s)\n", num_rekeys);
+	if (burst_mode)
+		printf("Burst mode ENABLED\n");
+	if (zc_rx)
+		printf("TLS_RX_EXPECT_NO_PAD ENABLED\n");
+
+	srand(time(NULL));
+
+	if (is_server)
+		return do_server() ? 1 : 0;
+
+	return do_client() ? 1 : 0;
+}
diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
new file mode 100755
index 000000000000..b8f5a3314030
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
@@ -0,0 +1,295 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Test kTLS hardware offload using a C helper binary."""
+
+from collections import defaultdict
+
+from lib.py import ksft_run, ksft_exit, ksft_pr, KsftSkipEx, ksft_true
+from lib.py import ksft_variants, KsftNamedVariant
+from lib.py import NetDrvEpEnv
+from lib.py import cmd, bkg, wait_port_listen, rand_port
+from lib.py import CmdExitFailure
+
+# Burst variants push hundreds of MB and perform many rekeys; the
+# default cmd() timeout (5s) is too short.
+BURST_TIMEOUT_S = 180
+
+
+def check_tls_support(cfg):
+    try:
+        cmd("test -f /proc/net/tls_stat")
+        cmd("test -f /proc/net/tls_stat", host=cfg.remote)
+    except CmdExitFailure as e:
+        raise KsftSkipEx(f"kTLS not supported: {e}")
+
+    try:
+        features = cmd(f"ethtool -k {cfg.ifname}").stdout
+        if 'tls-hw-tx-offload: on' not in features:
+            raise KsftSkipEx("Device does not support TLS HW TX offload")
+        if 'tls-hw-rx-offload: on' not in features:
+            raise KsftSkipEx("Device does not support TLS HW RX offload")
+    except CmdExitFailure as e:
+        raise KsftSkipEx(f"Cannot determine TLS HW offload support: {e}")
+
+
+def read_tls_stats(host=None):
+    stats = defaultdict(int)
+    output = cmd("cat /proc/net/tls_stat", host=host)
+    for line in output.stdout.strip().split('\n'):
+        parts = line.split()
+        if len(parts) == 2:
+            stats[parts[0]] = int(parts[1])
+    return stats
+
+
+def stat_diff(before, after, key):
+    return after[key] - before[key]
+
+
+def check_path(before, after, direction, role, require_hw):
+    """On the DUT, require HW offload; on the remote, HW or SW is fine."""
+    dev = stat_diff(before, after, f'Tls{direction}Device')
+    sw = stat_diff(before, after, f'Tls{direction}Sw')
+    if require_hw:
+        if dev < 1:
+            ksft_pr(f"FAIL: {role} {direction}: HW offload not engaged "
+                    f"(Device={dev}, Sw={sw})")
+            return 1
+    elif dev < 1 and sw < 1:
+        ksft_pr(f"FAIL: {role} {direction}: no TLS activity "
+                f"(Device={dev}, Sw={sw})")
+        return 1
+    return 0
+
+
+def check_min(before, after, key, minimum, role):
+    diff = stat_diff(before, after, key)
+    if diff < minimum:
+        ksft_pr(f"FAIL: {role} {key}: expected >= {minimum}, got {diff}")
+        return 1
+    return 0
+
+
+def check_max(before, after, key, maximum, role):
+    diff = stat_diff(before, after, key)
+    if diff > maximum:
+        ksft_pr(f"FAIL: {role} {key}: expected <= {maximum}, got {diff}")
+        return 1
+    return 0
+
+
+def check_eq_sum(before, after, keys, expected, role):
+    """Verify the sum of `keys` increased by exactly `expected`."""
+    diff = sum(stat_diff(before, after, k) for k in keys)
+    if diff != expected:
+        ksft_pr(f"FAIL: {role} sum({'+'.join(keys)}): "
+                f"expected == {expected}, got {diff}")
+        return 1
+    return 0
+
+
+def check_zero(before, after, key, role):
+    diff = stat_diff(before, after, key)
+    if diff != 0:
+        ksft_pr(f"FAIL: {role} {key} changed by {diff}, expected 0")
+        return 1
+    return 0
+
+
+def verify_tls_counters(stats_before, stats_after, expected_rekeys,
+                        tls_role, is_dut, burst=False):
+    """Verify TLS counters on one side of the connection.
+
+    tls_role: 'client' or 'server' (TLS role this side played).
+    is_dut: True for the local DUT; requires HW offload counters.
+    burst: burst mode - only the TLS client rotates its TX key; the TLS
+           server only follows with an RX rotation on KeyUpdate receipt.
+    """
+    role = 'DUT' if is_dut else 'Peer'
+
+    # In burst mode the TLS client only TXs and the TLS server only RXs.
+    # In echo mode both sides drive both directions.
+    with_tx = not burst or tls_role == 'client'
+    with_rx = not burst or tls_role != 'client'
+
+    errors = 0
+    if with_tx:
+        errors += check_path(stats_before, stats_after, 'Tx', role,
+                             require_hw=is_dut)
+    if with_rx:
+        errors += check_path(stats_before, stats_after, 'Rx', role,
+                             require_hw=is_dut)
+
+    if expected_rekeys > 0:
+        if with_tx:
+            # At most one rekey can be PENDING at socket close (single
+            # TLS_TX_REKEY_PENDING bit), so at most one rekey lands in
+            # TlsTxRekeyAborted; the rest must complete as TlsTxRekeyOk.
+            # Together with the sum equality this pins:
+            #   Aborted=0 -> Ok=N, Aborted=1 -> Ok=N-1.
+            errors += check_max(stats_before, stats_after,
+                                'TlsTxRekeyAborted', 1, role)
+            errors += check_eq_sum(stats_before, stats_after,
+                                   ['TlsTxRekeyOk', 'TlsTxRekeyAborted'],
+                                   expected_rekeys, role)
+            errors += check_zero(stats_before, stats_after,
+                                 'TlsTxRekeyError', role)
+            errors += check_zero(stats_before, stats_after,
+                                 'TlsTxRekeyFallback', role)
+            errors += check_zero(stats_before, stats_after,
+                                 'TlsTxRekeyInProgress', role)
+        if with_rx:
+            # At most one rekey can be deferred (single dev_add_pending)
+            # at socket close, landing in TlsRxRekeyAborted; the rest
+            # complete as TlsRxRekeyOk. Together with the sum equality this
+            # pins: Aborted=0 -> Ok=N, Aborted=1 -> Ok=N-1.
+            errors += check_max(stats_before, stats_after,
+                                'TlsRxRekeyAborted', 1, role)
+            errors += check_eq_sum(stats_before, stats_after,
+                                   ['TlsRxRekeyOk', 'TlsRxRekeyAborted'],
+                                   expected_rekeys, role)
+            errors += check_min(stats_before, stats_after,
+                                'TlsRxRekeyReceived', expected_rekeys, role)
+            errors += check_zero(stats_before, stats_after,
+                                 'TlsRxRekeyError', role)
+            errors += check_zero(stats_before, stats_after,
+                                 'TlsRxRekeyFallback', role)
+            errors += check_zero(stats_before, stats_after,
+                                 'TlsRxRekeyInProgress', role)
+
+    errors += check_zero(stats_before, stats_after, 'TlsDecryptError', role)
+
+    return errors
+
+
+def run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=0,
+                 buffer_size=None, random_max=None, burst=False, zc=False,
+                 dut_role="client", num_iterations=None):
+    """Run the TLS offload test.
+
+    dut_role: 'client' (default) - DUT runs the TLS client, remote the server.
+              'server' - swap: DUT listens, remote connects. Used for burst_rx
+              so the DUT's RX path is the one under rekey pressure.
+
+    The DUT (local) is the kernel under test; the remote is just a traffic
+    source/sink and may run any kernel without HW offload. Both sides run
+    kTLS because TLS is pairwise, but verify_tls_counters() requires HW
+    offload only on the DUT (is_dut=True); the peer may use SW kTLS.
+    """
+    port = rand_port()
+    send_size = random_max or buffer_size
+
+    if dut_role == "client":
+        server_bin, server_host = cfg.bin_remote, cfg.remote
+        client_bin, client_host = cfg.bin_local, None
+        client_target = cfg.remote_addr_v['4']
+    else:
+        server_bin, server_host = cfg.bin_local, None
+        client_bin, client_host = cfg.bin_remote, cfg.remote
+        client_target = cfg.addr_v['4']
+
+    server_parts = [f"{server_bin} server -p {port} -c {cipher}",
+                    f"-v {tls_version}"]
+    if burst:
+        server_parts.append("-B")
+    if zc:
+        server_parts.append("-Z")
+    if send_size:
+        server_parts.append(f"-b {send_size}")
+    server_cmd = " ".join(server_parts)
+
+    client_parts = [f"{client_bin} client -s {client_target}",
+                    f"-p {port} -c {cipher} -v {tls_version}"]
+    if rekey:
+        client_parts.append(f"-k {rekey}")
+    if burst:
+        client_parts.append("-B")
+    if num_iterations:
+        client_parts.append(f"-n {num_iterations}")
+    if random_max:
+        client_parts.append(f"-r {random_max}")
+    elif buffer_size:
+        client_parts.append(f"-b {buffer_size}")
+    client_cmd = " ".join(client_parts)
+
+    cmd_timeout = BURST_TIMEOUT_S if burst else 10
+
+    stats_before_local = read_tls_stats()
+    stats_before_remote = read_tls_stats(host=cfg.remote)
+
+    with bkg(server_cmd, host=server_host, exit_wait=True):
+        wait_port_listen(port, host=server_host)
+        cmd(client_cmd, host=client_host, timeout=cmd_timeout)
+
+    stats_after_local = read_tls_stats()
+    stats_after_remote = read_tls_stats(host=cfg.remote)
+
+    peer_tls_role = 'server' if dut_role == 'client' else 'client'
+
+    dut_errors = verify_tls_counters(stats_before_local, stats_after_local,
+                                     rekey, dut_role, is_dut=True,
+                                     burst=burst)
+    peer_errors = verify_tls_counters(stats_before_remote, stats_after_remote,
+                                      rekey, peer_tls_role, is_dut=False,
+                                      burst=burst)
+
+    ksft_true(dut_errors == 0,
+              f"DUT TLS counters verified ({dut_errors} failures)")
+    ksft_true(peer_errors == 0,
+              f"Peer TLS counters verified ({peer_errors} failures)")
+
+
+@ksft_variants([
+    KsftNamedVariant("tls13_aes128", "128", "1.3"),
+    KsftNamedVariant("tls13_aes256", "256", "1.3"),
+    KsftNamedVariant("tls12_aes128", "128", "1.2"),
+    KsftNamedVariant("tls12_aes256", "256", "1.2"),
+])
+def test_tls_offload(cfg, cipher, tls_version):
+    run_tls_test(cfg, cipher=cipher, tls_version=tls_version)
+
+
+@ksft_variants([
+    KsftNamedVariant("single", 1),
+    KsftNamedVariant("multiple", 99),
+    KsftNamedVariant("small_buf", 30, 512),
+    KsftNamedVariant("large_buf", 10, 2097152),
+    KsftNamedVariant("random_buf", 20, None, 8192),
+])
+def test_tls_offload_rekey(cfg, rekey, buffer_size=None, random_max=None):
+    run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=rekey,
+                 buffer_size=buffer_size, random_max=random_max)
+
+
+# Columns:                                          dut_role  zc     interval rekeys buffer_size
+@ksft_variants([
+    KsftNamedVariant("burst_tx_rekey_every_1",        "client", False, 1,       50,    65536),
+    KsftNamedVariant("burst_tx_rekey_every_1000",     "client", False, 1000,    3,     65536),
+    KsftNamedVariant("burst_rx_rekey_every_10",       "server", False, 10,      20,    65536),
+    KsftNamedVariant("burst_rx_rekey_every_10000",    "server", False, 10000,   1,     32768),
+    KsftNamedVariant("burst_rx_zc_rekey_every_100",   "server", True,  100,     10,    65536),
+    KsftNamedVariant("burst_rx_zc_rekey_every_20000", "server", True,  20000,   1,     16384),
+])
+def test_tls_offload_burst(cfg, dut_role, zc, interval, rekeys, buffer_size):
+    run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=rekeys,
+                 buffer_size=buffer_size, burst=True, zc=zc, dut_role=dut_role,
+                 num_iterations=interval * (rekeys + 1))
+
+
+def main() -> None:
+    with NetDrvEpEnv(__file__, nsim_test=False) as cfg:
+        cfg.bin_local = cfg.test_dir / "tls_hw_offload"
+        if not cfg.bin_local.exists():
+            raise KsftSkipEx(f"tls_hw_offload binary not found at {cfg.bin_local}")
+        cfg.bin_remote = cfg.remote.deploy(cfg.bin_local)
+        cfg.require_ipver("4")
+        check_tls_support(cfg)
+
+        ksft_run([test_tls_offload, test_tls_offload_rekey,
+                  test_tls_offload_burst], args=(cfg, ))
+    ksft_exit()
+
+
+if __name__ == "__main__":
+    main()
-- 
2.25.1

Re: [PATCH v16 10/10] selftests: net: add TLS hardware offload test

[Jakub Kicinski]

On Fri,  7 Aug 2026 12:38:53 -0600 Rishikesh Jethwani wrote:
> Two-node kTLS HW offload test using NetDrvEpEnv. A C helper binary
> acts as TLS client or server; a Python harness drives it and verifies
> TLS stat counters (RekeyOk, RekeyReceived, RekeyFallback,
> RekeyInProgress, RekeyAborted, RekeyError, DecryptError).
> 
> Covers TLS 1.2/1.3 with AES-GCM-128/256, rekey with various buffer
> sizes, and burst variants that stress TX rekey (temporary SW phase,
> HW reinstall) and RX rekey (boundary tracking, old-key reencryption,
> deferred dev_add).
> 
> Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
> ---
>  MAINTAINERS                                   |   2 +
>  .../selftests/drivers/net/hw/.gitignore       |   1 +
>  .../testing/selftests/drivers/net/hw/Makefile |   2 +
>  .../selftests/drivers/net/hw/tls_hw_offload.c | 975 ++++++++++++++++++
>  .../drivers/net/hw/tls_hw_offload.py          | 295 ++++++
>  5 files changed, 1275 insertions(+)
>  create mode 100644 tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
>  create mode 100755 tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 08e43bc09735..6e119592b72a 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -19066,6 +19066,8 @@ F:	Documentation/networking/tls*
>  F:	include/net/tls.h
>  F:	include/uapi/linux/tls.h
>  F:	net/tls/
> +F:	tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
> +F:	tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
>  F:	tools/testing/selftests/net/tls.c
>  
>  NETWORKING [SOCKETS]
> diff --git a/tools/testing/selftests/drivers/net/hw/.gitignore b/tools/testing/selftests/drivers/net/hw/.gitignore
> index 46540468a775..f0a5d15b469b 100644
> --- a/tools/testing/selftests/drivers/net/hw/.gitignore
> +++ b/tools/testing/selftests/drivers/net/hw/.gitignore
> @@ -2,3 +2,4 @@
>  iou-zcrx
>  ncdevmem
>  toeplitz
> +tls_hw_offload

nit: alphabetic sort would put tls before toeplitz?

> +static int client_connect_tls(void)
> +{
> +	struct sockaddr_in sa;
> +	int csk;
> +
> +	csk = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);

IPv6 support is a must these days, should be pretty easy
with getaddrinfo() ? Ideally we'd support forcing IP version
using -4/-6 flags and appropriate plumbing on the Python side.
Run at least a basic test over both

> +	if (csk < 0) {
> +		printf("SETUP ERROR: failed to create socket: %s\n",
> +		       strerror(errno));
> +		return -1;
> +	}
> +
> +	memset(&sa, 0, sizeof(sa));
> +	sa.sin_family = AF_INET;
> +	sa.sin_addr.s_addr = inet_addr(server_ip);
> +	sa.sin_port = htons(server_port);
> +	printf("Connecting to %s:%d...\n", server_ip, server_port);
> +

> diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
> new file mode 100755
> index 000000000000..b8f5a3314030
> --- /dev/null
> +++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
> @@ -0,0 +1,295 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0
> +
> +"""Test kTLS hardware offload using a C helper binary."""
> +
> +from collections import defaultdict
> +
> +from lib.py import ksft_run, ksft_exit, ksft_pr, KsftSkipEx, ksft_true
> +from lib.py import ksft_variants, KsftNamedVariant
> +from lib.py import NetDrvEpEnv
> +from lib.py import cmd, bkg, wait_port_listen, rand_port
> +from lib.py import CmdExitFailure
> +
> +# Burst variants push hundreds of MB and perform many rekeys; the
> +# default cmd() timeout (5s) is too short.
> +BURST_TIMEOUT_S = 180
> +
> +
> +def check_tls_support(cfg):

please make sure pylint --disable=R passes cleanly on new Python files
docstring missing here (you can prefix the trivial local helpers with
_ to avoid that)

> +    try:
> +        cmd("test -f /proc/net/tls_stat")
> +        cmd("test -f /proc/net/tls_stat", host=cfg.remote)
> +    except CmdExitFailure as e:
> +        raise KsftSkipEx(f"kTLS not supported: {e}")

That's fine, but you also must update the 
tools/testing/selftests/drivers/net/hw/config
config to make sure that x86 defconfig + that file result in a build
with working TLS offload

> +    try:
> +        features = cmd(f"ethtool -k {cfg.ifname}").stdout
> +        if 'tls-hw-tx-offload: on' not in features:
> +            raise KsftSkipEx("Device does not support TLS HW TX offload")
> +        if 'tls-hw-rx-offload: on' not in features:
> +            raise KsftSkipEx("Device does not support TLS HW RX offload")
> +    except CmdExitFailure as e:
> +        raise KsftSkipEx(f"Cannot determine TLS HW offload support: {e}")
> +
> +
> +def read_tls_stats(host=None):
> +    stats = defaultdict(int)
> +    output = cmd("cat /proc/net/tls_stat", host=host)
> +    for line in output.stdout.strip().split('\n'):
> +        parts = line.split()
> +        if len(parts) == 2:
> +            stats[parts[0]] = int(parts[1])
> +    return stats
> +
> +
> +def stat_diff(before, after, key):
> +    return after[key] - before[key]
> +
> +
> +def check_path(before, after, direction, role, require_hw):
> +    """On the DUT, require HW offload; on the remote, HW or SW is fine."""
> +    dev = stat_diff(before, after, f'Tls{direction}Device')
> +    sw = stat_diff(before, after, f'Tls{direction}Sw')
> +    if require_hw:
> +        if dev < 1:
> +            ksft_pr(f"FAIL: {role} {direction}: HW offload not engaged "
> +                    f"(Device={dev}, Sw={sw})")

ksft_lt(..., comment="your string") ?
Please use the official check helpers

> +def main() -> None:
> +    with NetDrvEpEnv(__file__, nsim_test=False) as cfg:
> +        cfg.bin_local = cfg.test_dir / "tls_hw_offload"
> +        if not cfg.bin_local.exists():
> +            raise KsftSkipEx(f"tls_hw_offload binary not found at {cfg.bin_local}")
> +        cfg.bin_remote = cfg.remote.deploy(cfg.bin_local)
> +        cfg.require_ipver("4")
> +        check_tls_support(cfg)
> +
> +        ksft_run([test_tls_offload, test_tls_offload_rekey,
> +                  test_tls_offload_burst], args=(cfg, ))
> +    ksft_exit()
> +
> +
> +if __name__ == "__main__":
> +    main()

Re: [PATCH v16 04/10] tls: add TLS 1.3 hardware offload support

[Jakub Kicinski]

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
---
    tls: add TLS 1.3 hardware offload support

This drops the TLS_1_2_VERSION gates in tls_set_device_offload() and
tls_set_device_offload_rx(), teaches tls_device_record_close() to append
the TLS 1.3 content_type byte along with the tag, and switches the
device fallback encryption path to prot->prepend_size / prot->aad_size
so the 1.3 layouts work.  tls_device_init() also pre-populates
dummy_page with an identity byte map.

A few questions below.

> diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
> index 37bb06a8e8f5..bbb1aa733500 100644
> --- a/net/tls/tls_device.c
> +++ b/net/tls/tls_device.c
> @@ -317,25 +317,34 @@ static void tls_device_record_close(struct sock *sk,
>  				    unsigned char record_type)
>  {
>  	struct tls_prot_info *prot = &ctx->prot_info;
> -	struct page_frag dummy_tag_frag;
> -
> -	/* append tag
> -	 * device will fill in the tag, we just need to append a placeholder
> -	 * use socket memory to improve coalescing (re-using a single buffer
> -	 * increases frag count)
> -	 * if we can't allocate memory now use the dummy page
> +	int tail = prot->tag_size + prot->tail_size;
> +
> +	/* Append tail: tag for TLS 1.2, content_type + tag for TLS 1.3.
> +	 * Device fills in the tag, we just need to append a placeholder.
> +	 * Use socket memory to improve coalescing (re-using a single buffer
> +	 * increases frag count); if allocation fails use dummy_page
> +	 * (offset = record_type gives correct content_type byte via
> +	 * identity mapping)
>  	 */
> -	if (unlikely(pfrag->size - pfrag->offset < prot->tag_size) &&
> -	    !skb_page_frag_refill(prot->tag_size, pfrag, sk->sk_allocation)) {
> -		dummy_tag_frag.page = dummy_page;
> -		dummy_tag_frag.offset = 0;
> -		pfrag = &dummy_tag_frag;
> +	if (unlikely(pfrag->size - pfrag->offset < tail) &&
> +	    !skb_page_frag_refill(tail, pfrag, sk->sk_allocation)) {
> +		struct page_frag dummy_pfrag = {
> +			.page = dummy_page,
> +			.offset = record_type,
> +		};
> +		tls_append_frag(record, &dummy_pfrag, tail);

[Severity: Medium]
Here record_type is used directly as an offset into the shared global
dummy_page, and tail = prot->tag_size + prot->tail_size bytes are appended
starting from that offset.  tls_device_init() only writes the first 256
bytes of the page, and the page comes from a plain alloc_page(GFP_KERNEL),
so the rest is not zeroed.

With AES-GCM (tag_size 16) and TLS 1.3 (tail_size 1), a record_type of 240
or above makes the appended frag cover dummy_page bytes
[256, record_type + 17), which are never initialized.

record_type is taken verbatim from the user cmsg; tls_process_cmsg() in
tls_main.c checks only cmsg_len and MSG_MORE:

	case TLS_SET_RECORD_TYPE:
		if (cmsg->cmsg_len < CMSG_LEN(sizeof(*record_type)))
			return -EINVAL;
		if (msg->msg_flags & MSG_MORE)
			return -EINVAL;
		...
		*record_type = *(unsigned char *)CMSG_DATA(cmsg);

so all 256 offsets are reachable from an unprivileged process.  Can this
put uninitialized page contents into the tag placeholder area of the
record?  Would zeroing the page (__GFP_ZERO), initializing
256 + max tag bytes, or bounding the offset be preferable?

Related to this, the comment added in tls_device_init() says "By populating
all 256 values, we avoid needing to validate record_type at runtime", but
the identity map only covers the content_type byte, not the tag portion of
the tail that follows it.

> +	} else {
> +		if (prot->tail_size) {
> +			char *content_type_addr = page_address(pfrag->page) +
> +						  pfrag->offset;
> +			*content_type_addr = record_type;
> +		}
> +		tls_append_frag(record, pfrag, tail);
>  	}
> -	tls_append_frag(record, pfrag, prot->tag_size);

[Severity: High]
This isn't a bug introduced by this patch, but the else branch now
dereferences pfrag->page one step earlier than before, and pfrag->page is
never checked for NULL.

Consider the state skb_page_frag_refill() leaves behind on failure in
net/core/sock.c:

	pfrag->offset = 0;
	...
	pfrag->page = alloc_page(gfp);
	if (likely(pfrag->page)) {
		pfrag->size = PAGE_SIZE;
		return true;
	}
	return false;

offset is zeroed and page is NULL, but size keeps its previous non-zero
value, so pfrag->size - pfrag->offset is at least PAGE_SIZE and the
"< tail" guard here is false.  The dummy_page fallback is skipped and the
else branch runs.

Can tls_device_record_close() then be reached with that pfrag?  In
tls_push_data(), with an already-open record (record->len >
prot->prepend_size, e.g. after a previous MSG_MORE sendmsg),
tls_do_allocation() takes the sk_page_frag_refill(sk, pfrag) path; on
allocation failure it returns -ENOMEM, sk_stream_wait_memory() returns
non-zero (immediately -EAGAIN for MSG_DONTWAIT), and control does:

	} else if (record->len > prot->prepend_size) {
		goto last_record;
	}

which sets done = true and calls tls_device_record_close(sk, tls_ctx,
record, pfrag, record_type).

For TLS 1.3 (prot->tail_size == 1) that computes page_address(NULL) + 0 and
stores record_type through it, then tls_append_frag() does get_page(NULL).
The pre-patch code took the same false branch and crashed in
tls_append_frag(), so the NULL deref itself is older, but should this test
pfrag->page (or reset pfrag->size on refill failure) so the dummy_page
fallback that exists for exactly this situation is actually used?

>  
>  	/* fill prepend */
>  	tls_fill_prepend(ctx, skb_frag_address(&record->frags[0]),
> -			 record->len - prot->overhead_size,
> +			 record->len - prot->overhead_size + prot->tail_size,
>  			 record_type);
>  }
>  

[ ... ]

> @@ -1091,11 +1100,6 @@ int tls_set_device_offload(struct sock *sk)
>  	}
>  
>  	crypto_info = &ctx->crypto_send.info;
> -	if (crypto_info->version != TLS_1_2_VERSION) {
> -		rc = -EOPNOTSUPP;
> -		goto release_netdev;
> -	}
> -
>  	cipher_desc = get_cipher_desc(crypto_info->cipher_type);
>  	if (!cipher_desc || !cipher_desc->offloadable) {
>  		rc = -EINVAL;

[Severity: Low]
Should Documentation/networking/tls-offload.rst be updated along with this?
Its Rekey section still says:

	Offload does not currently support TLS 1.3, therefore key rotation
	is not a concern for offloaded connections at this point.

Removing the version gates here and in tls_set_device_offload_rx() makes
the first half untrue, and the later patches in the series ("tls: device:
add TX KeyUpdate support", "tls: device: add RX KeyUpdate support") make
the second half untrue.  No patch in the series touches that file, so the
sentence is still there at the end of the series.

>  
> @@ -1410,12 +1411,22 @@ static struct notifier_block tls_dev_notifier = {
>  
>  int __init tls_device_init(void)
>  {
> -	int err;
> +	unsigned char *page_addr;
> +	int err, i;
>  
>  	dummy_page = alloc_page(GFP_KERNEL);
>  	if (!dummy_page)
>  		return -ENOMEM;
>  
> +	/* Pre-populate dummy_page with identity mapping for all byte values.
> +	 * This is used as fallback for TLS 1.3 content type when memory
> +	 * allocation fails. By populating all 256 values, we avoid needing
> +	 * to validate record_type at runtime.
> +	 */
> +	page_addr = page_address(dummy_page);
> +	for (i = 0; i < 256; i++)
> +		page_addr[i] = (unsigned char)i;
> +
>  	destruct_wq = alloc_workqueue("ktls_device_destruct", WQ_PERCPU, 0);
>  	if (!destruct_wq) {
>  		err = -ENOMEM;
-- 
pw-bot: cr

Re: [PATCH v16 05/10] tls: split tls_set_sw_offload into init and finalize stages

[Jakub Kicinski]

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
---
tls: split tls_set_sw_offload into init and finalize stages

This splits tls_set_sw_offload() into tls_sw_ctx_init(), which allocates
the SW contexts and sets up the AEAD transform, and tls_sw_ctx_finalize(),
which copies the salt/IV/record sequence into the live cipher context.
tls_set_device_offload_rx() is converted to call the two halves separately.

A couple of questions below.

> diff --git a/net/tls/tls.h b/net/tls/tls.h
> index 60a37bdaaa25..5a6ee1ea00f8 100644
> --- a/net/tls/tls.h
> +++ b/net/tls/tls.h
> @@ -147,6 +147,10 @@ void tls_strp_abort_strp(struct tls_strparser *strp, int err);
>  int init_prot_info(struct tls_prot_info *prot,
>  		   const struct tls_crypto_info *crypto_info,
>  		   const struct tls_cipher_desc *cipher_desc);
> +int tls_sw_ctx_init(struct sock *sk, int tx,
> +		    struct tls_crypto_info *new_crypto_info);
> +void tls_sw_ctx_finalize(struct sock *sk, int tx,
> +			 struct tls_crypto_info *new_crypto_info);

[Severity: Low]
Would it be worth documenting the pairing contract of these two helpers
here?  As written, tls_sw_ctx_finalize() may only be called after a
tls_sw_ctx_init() that returned 0, with the same tx value, the same
new_crypto_info pointer, and an unchanged crypto_info->cipher_type.  Since
finalize() returns void it cannot report a violation, and the tx /
new_crypto_info selection logic is now duplicated in both halves.

>  int tls_set_sw_offload(struct sock *sk, int tx,
>  		       struct tls_crypto_info *new_crypto_info);
>  void tls_update_rx_zc_capable(struct tls_context *tls_ctx);
> diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
> index bbb1aa733500..cf67e1f6c5f4 100644
> --- a/net/tls/tls_device.c
> +++ b/net/tls/tls_device.c
> @@ -1235,7 +1235,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
>  	context->resync_nh_reset = 1;
>  
>  	ctx->priv_ctx_rx = context;
> -	rc = tls_set_sw_offload(sk, 0, NULL);
> +	rc = tls_sw_ctx_init(sk, 0, NULL);
>  	if (rc)
>  		goto release_ctx;
>  
> @@ -1249,6 +1249,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
>  		goto free_sw_resources;
>  
>  	tls_device_attach(ctx, sk, netdev);
> +	tls_sw_ctx_finalize(sk, 0, NULL);

[Severity: Low]
The commit message describes this as separating "cipher context
initialization from key material finalization", but this hunk also changes
the order of operations in tls_set_device_offload_rx().

Before the patch, tls_set_sw_offload(sk, 0, NULL) filled in ctx->rx.iv and
ctx->rx.rec_seq before netdev->tlsdev_ops->tls_dev_add() programmed the NIC
and before tls_device_attach() published the context on tls_device_list and
swapped sk->sk_destruct.  Now ctx->rx stays zeroed across both of those
calls and is only committed afterwards.

The split by itself could have been done with tls_sw_ctx_finalize() invoked
at exactly the point where the old copy happened, so the new ordering looks
like a deliberate choice.  Could the commit message say why finalize() has
to run after tls_dev_add() and tls_device_attach()?

For what it is worth, the window looks harmless at this commit: the whole
function runs under lock_sock() and down_read(&device_offload_lock), the RX
data path is only armed later by tls_sw_strparser_arm(), the RX tls_dev_add()
call is passed &ctx->crypto_recv.info rather than ctx->rx, and no in-tree
tls_dev_add() implementation reads the SW cipher_context.  A note in the
changelog would still help reviewers and anyone backporting this.

>  	up_read(&device_offload_lock);
>  
>  	dev_put(netdev);
> diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
> index 62d46736e24b..63c83247f9a3 100644
> --- a/net/tls/tls_sw.c
> +++ b/net/tls/tls_sw.c

[ ... ]

> @@ -2650,3 +2631,55 @@ int tls_set_sw_offload(struct sock *sk, int tx,
>  out:
>  	return rc;
>  }
> +
> +void tls_sw_ctx_finalize(struct sock *sk, int tx,
> +			 struct tls_crypto_info *new_crypto_info)
> +{
> +	struct tls_crypto_info *crypto_info, *src_crypto_info;
> +	const struct tls_cipher_desc *cipher_desc;
> +	struct tls_context *ctx = tls_get_ctx(sk);
> +	struct cipher_context *cctx;
> +	char *iv, *salt, *rec_seq;
> +
> +	if (tx) {
> +		crypto_info = &ctx->crypto_send.info;
> +		cctx = &ctx->tx;
> +	} else {
> +		crypto_info = &ctx->crypto_recv.info;
> +		cctx = &ctx->rx;
> +	}
> +
> +	src_crypto_info = new_crypto_info ?: crypto_info;
> +	cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
> +
> +	iv = crypto_info_iv(src_crypto_info, cipher_desc);
> +	salt = crypto_info_salt(src_crypto_info, cipher_desc);
> +	rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);
> +
> +	memcpy(cctx->iv, salt, cipher_desc->salt);
> +	memcpy(cctx->iv + cipher_desc->salt, iv, cipher_desc->iv);
> +	memcpy(cctx->rec_seq, rec_seq, cipher_desc->rec_seq);

[Severity: Low]
Should this repeated get_cipher_desc() lookup be NULL checked?

get_cipher_desc() returns NULL for a cipher_type outside
[TLS_CIPHER_MIN, TLS_CIPHER_MAX], and here the result is dereferenced
straight away by crypto_info_iv() / crypto_info_salt() /
crypto_info_rec_seq() (which read cipher_desc->*_offset) and by
cipher_desc->salt / ->iv / ->rec_seq in the memcpy() calls.

The sibling stage still guards the identical lookup:

net/tls/tls_sw.c:tls_sw_ctx_init() {
	...
	cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
	if (!cipher_desc) {
		rc = -EINVAL;
		goto free_priv;
	}
	...
}

Today both callers reach finalize() only after an init() that validated the
same cipher_type (tls_set_sw_offload() returns early on error, and
tls_set_device_offload_rx() checks rc from tls_sw_ctx_init()), and
do_tls_setsockopt_conf() rejects unknown cipher_type before that, so this
does not look reachable.  Since the helper is now exported module-wide as
void, though, would re-checking cipher_desc (or at least a comment stating
the assumption) be preferable?

> +
> +	if (new_crypto_info) {
> +		unsafe_memcpy(crypto_info, new_crypto_info,
> +			      cipher_desc->crypto_info,
> +			      /* size was checked in do_tls_setsockopt_conf */);

[Severity: Low]
The retained justification comment names only do_tls_setsockopt_conf(), but
tls_set_device_offload_rx() is now also a caller of this code.  Should the
comment be extended to cover the device path as well?

Separately, finalize() commits crypto_info, the IV and the record sequence
into the live socket context, and calls tls_finish_key_update() for RX,
without any indication that the crypto_aead_setkey() in the init stage
succeeded.  Is the caller expected to guarantee that ordering?

> +		memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
> +
> +		if (!tx)
> +			tls_finish_key_update(sk, ctx);
> +	}
> +}

Re: [PATCH v16 06/10] tls: prep helpers and refactors for HW offload KeyUpdate

[Jakub Kicinski]

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
---
tls: prep helpers and refactors for HW offload KeyUpdate

This patch hoists cipher_context / tls_crypto_context above
tls_offload_context_tx, adds a tls_tx_cipher_ctx() accessor plus a
tls_sw_ctx_tx_init() helper, splits tls_set_device_offload() and factors out
tls_device_dev_add_tx() / tls_device_commit_start_marker(), and moves
crypto_aead_setauthsize() into the !*aead block.  It is described as no
functional change.

No Link: tags present in the commit header.

> diff --git a/include/net/tls.h b/include/net/tls.h
> index e57bef58851e..eb258bcd62bc 100644
> --- a/include/net/tls.h
> +++ b/include/net/tls.h

[ ... ]

> @@ -392,6 +392,12 @@ static inline struct tls_sw_context_tx *tls_sw_ctx_tx(
>  	return (struct tls_sw_context_tx *)tls_ctx->priv_ctx_tx;
>  }
>  
> +static inline struct cipher_context *tls_tx_cipher_ctx(
> +		const struct tls_context *tls_ctx)
> +{
> +	return (struct cipher_context *)&tls_ctx->tx;
> +}
> +

[Severity: Low]

The commit message says this accessor exists "so the TX path can redirect to
a temporary SW context during rekey", but only some TX-side users are
converted here.  Is the remaining set intentional?

tls_fill_prepend() in net/tls/tls.h still reads the explicit nonce straight
from ctx->tx:

	if (prot->version != TLS_1_3_VERSION &&
	    prot->cipher_type != TLS_CIPHER_CHACHA20_POLY1305) {
		pkt_len += iv_size;

		memcpy(buf + TLS_NONCE_OFFSET,
		       ctx->tx.iv + prot->salt_size, iv_size);
	}

and on the HW TX side tls_device.c's tls_push_record() advances
&ctx->tx while tls_device_resync_tx() reads tls_ctx->tx.rec_seq directly.
tls_sw_ctx_finalize() and do_tls_getsockopt_conf() also still use &ctx->tx.

Walking the paths, none of these look reachable while the accessor points
somewhere else: do_tls_setsockopt_conf() rejects a second crypto info for
anything but TLS 1.3 with -EBUSY, so the TLS 1.2 nonce branch above cannot
run during a rekey, and the device record/resync paths are entered from
tls_push_data(), which the later series patch diverts to the SW helpers while
the rekey-pending flag is set.  So there does not appear to be a live
mismatch today.

Would it be worth a short comment on tls_fill_prepend()'s ctx->tx.iv use
noting that it is safe because rekey is TLS 1.3 only, so a future reader
does not have to redo this reasoning?

> diff --git a/net/tls/tls.h b/net/tls/tls.h
> index 5a6ee1ea00f8..e8b552c2f4fb 100644
> --- a/net/tls/tls.h
> +++ b/net/tls/tls.h
> @@ -157,7 +157,10 @@ void tls_update_rx_zc_capable(struct tls_context *tls_ctx);
>  void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx);
>  void tls_sw_strparser_done(struct tls_context *tls_ctx);
>  int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
> +void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx);
> +int tls_sw_push_pending_record(struct sock *sk, int flags);
>  void tls_sw_splice_eof(struct socket *sock);
> +void tls_sw_splice_eof_locked(struct socket *sock);

[Severity: Low]

The changelog lists five refactors, but two of the changes here are not among
them.  Could the commit message be extended to cover them?

tls_sw_push_pending_record() loses static and becomes a global symbol, and
tls_sw_splice_eof() is split into a new externally visible
tls_sw_splice_eof_locked() plus a locking wrapper.  Neither appears in the
bullet list.

tls_sw_splice_eof_locked() also gains an entry contract that is not written
down anywhere: the caller must already hold tls_ctx->tx_lock and the socket
lock, and is expected to have checked ctx->open_rec.  Would a comment above
it, or a lockdep assertion, be worth adding?

>  void tls_sw_cancel_work_tx(struct tls_context *tls_ctx);
>  void tls_sw_release_resources_tx(struct sock *sk);
>  void tls_sw_free_ctx_tx(struct tls_context *tls_ctx);

> diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
> index 63c83247f9a3..7b6ab992c86e 100644
> --- a/net/tls/tls_sw.c
> +++ b/net/tls/tls_sw.c
> @@ -555,11 +555,11 @@ static int tls_do_encryption(struct sock *sk,
>  		break;
>  	}
>  
> -	memcpy(&rec->iv_data[iv_offset], tls_ctx->tx.iv,
> +	memcpy(&rec->iv_data[iv_offset], tls_tx_cipher_ctx(tls_ctx)->iv,
>  	       prot->iv_size + prot->salt_size);
>  
>  	tls_xor_iv_with_seq(prot, rec->iv_data + iv_offset,
> -			    tls_ctx->tx.rec_seq);
> +			    tls_tx_cipher_ctx(tls_ctx)->rec_seq);
>  
>  	sge->offset += prot->prepend_size;
>  	sge->length -= prot->prepend_size;

[ ... ]

> @@ -676,7 +676,7 @@ static int tls_push_record(struct sock *sk, int flags,
>  	sg_chain(rec->sg_aead_out, 2, &msg_en->sg.data[i]);
>  
>  	tls_make_aad(rec->aad_space, msg_pl->sg.size + prot->tail_size,
> -		     tls_ctx->tx.rec_seq, record_type, prot);
> +		     tls_tx_cipher_ctx(tls_ctx)->rec_seq, record_type, prot);
>  
>  	tls_fill_prepend(tls_ctx,
>  			 page_address(sg_page(&msg_en->sg.data[i])) +

[ ... ]

> @@ -712,7 +712,7 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,
>  	return err;
>  }
>  
> -static int tls_sw_push_pending_record(struct sock *sk, int flags)
> +int tls_sw_push_pending_record(struct sock *sk, int flags)
>  {
>  	struct tls_context *tls_ctx = tls_get_ctx(sk);
>  	struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);

[ ... ]

> @@ -1028,7 +1028,7 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
>  /*
>   * Handle unexpected EOF during splice without SPLICE_F_MORE set.
>   */
> -void tls_sw_splice_eof(struct socket *sock)
> +void tls_sw_splice_eof_locked(struct socket *sock)
>  {
>  	struct sock *sk = sock->sk;
>  	struct tls_context *tls_ctx = tls_get_ctx(sk);
> @@ -1039,21 +1039,15 @@ void tls_sw_splice_eof(struct socket *sock)
>  	bool retrying = false;
>  	int ret = 0;
>  
> -	if (!ctx->open_rec)
> -		return;
> -
> -	mutex_lock(&tls_ctx->tx_lock);
> -	lock_sock(sk);
> -
>  retry:
>  	/* same checks as in tls_sw_push_pending_record() */
>  	rec = ctx->open_rec;

[ ... ]

>  	/* Transmit if any encryptions have completed */
>  	if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) {
>  		cancel_delayed_work(&ctx->tx_work.work);
>  		tls_tx_records(sk, 0);
>  	}
> +}
> +
> +void tls_sw_splice_eof(struct socket *sock)
> +{
> +	struct sock *sk = sock->sk;
> +	struct tls_context *tls_ctx = tls_get_ctx(sk);
> +	struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);
>  
> -unlock:
> +	if (!ctx->open_rec)
> +		return;
> +
> +	mutex_lock(&tls_ctx->tx_lock);
> +	lock_sock(sk);
> +	tls_sw_splice_eof_locked(sock);
>  	release_sock(sk);
>  	mutex_unlock(&tls_ctx->tx_lock);
>  }

Both observations are documentation level; the code changes themselves look
equivalent to the previous behaviour.

Re: [PATCH v16 07/10] tls: device: add TX KeyUpdate support

[Jakub Kicinski]

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
---
tls: device: add TX KeyUpdate support

This adds TX KeyUpdate for device-offloaded kTLS by installing a temporary
SW context with the new key, routing sendmsg through
tls_sw_sendmsg_locked() until all old-key HW records are ACKed, and then
reinstalling HW offload at the current write_seq. A few questions below
about the SW/HW transition, the record bookkeeping and the new counters.

> diff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h
> index 49f5640092a0..119fbd49179c 100644
> --- a/include/uapi/linux/snmp.h
> +++ b/include/uapi/linux/snmp.h
> @@ -369,6 +369,9 @@ enum
>  	LINUX_MIB_TLSTXREKEYOK,			/* TlsTxRekeyOk */
>  	LINUX_MIB_TLSTXREKEYERROR,		/* TlsTxRekeyError */
>  	LINUX_MIB_TLSRXREKEYRECEIVED,		/* TlsRxRekeyReceived */
> +	LINUX_MIB_TLSTXREKEYFALLBACK,		/* TlsTxRekeyFallback */
> +	LINUX_MIB_TLSTXREKEYINPROGRESS,		/* TlsTxRekeyInProgress */
> +	LINUX_MIB_TLSTXREKEYABORTED,		/* TlsTxRekeyAborted */
>  	__LINUX_MIB_TLSMAX
>  };

[Severity: Low]
These three names become visible in /proc/net/tls_stat through the new
SNMP_MIB_ITEM() entries in tls_mib_list[], but
Documentation/networking/tls.rst is not updated anywhere in this series,
even though it documents every other TLS counter including TlsTxRekeyOk,
TlsTxRekeyError, TlsRxRekeyReceived and the TlsCurrTx* gauges.

Should the statistics section of tls.rst gain these three?

Also, TlsTxRekeyInProgress is used as a gauge (incremented in
tls_set_device_offload_rekey() when the rekey is deferred, decremented in
tls_device_complete_rekey() and tls_device_free_resources_tx()), while
every other Tls*Rekey* counter is monotonic and the existing gauges use
the TlsCurr* prefix.

Would TlsCurrTxRekey (or similar) match the existing naming better?

> diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
> index 4f6b3ea1bde4..85c74d0816b7 100644
> --- a/net/tls/tls_device.c
> +++ b/net/tls/tls_device.c

[ ... ]

> @@ -252,7 +296,15 @@ void tls_device_free_resources_tx(struct sock *sk)
>  {
>  	struct tls_context *tls_ctx = tls_get_ctx(sk);
>  
> -	tls_free_partial_record(sk, tls_ctx);
> +	if (unlikely(tls_ctx->rekey.sw_ctx))
> +		tls_sw_release_resources_tx(sk);
> +	else
> +		tls_free_partial_record(sk, tls_ctx);

[Severity: Critical]
Can tls_ctx->partially_sent_record still belong to the device path here,
while rekey.sw_ctx points at a brand new SW context whose tx_list is
empty?

The device push path leaves it set when the record does not fit in the
send queue:

net/tls/tls_device.c:tls_push_record() {
	...
	return tls_push_sg(sk, tls_ctx, offload_ctx->sg_tx_data, 0, flags);
}

net/tls/tls_main.c:tls_push_sg() {
	...
		ctx->partially_sent_offset = offset;
		ctx->partially_sent_record = (void *)sg;
}

tls_device_start_rekey() only flushes a pending *open* record
(tls_is_pending_open_record()), which is false for an already closed but
partially sent record, and the defer decision in
tls_set_device_offload_rekey() also does not look at it:

	if (!rekey_pending && !rekey_failed)
		defer = tls_has_unacked_records(offload_ctx) ||
			tls_is_pending_open_record(ctx);

The SW paths assume partially_sent_record implies the head of tx_list is
the owning tls_rec:

net/tls/tls_sw.c:tls_sw_release_resources_tx() {
	if (tls_ctx->partially_sent_record) {
		tls_free_partial_record(sk, tls_ctx);
		rec = list_first_entry(&ctx->tx_list,
				       struct tls_rec, list);
		list_del(&rec->list);
		sk_msg_free(sk, &rec->msg_plaintext);
		kfree(rec);
	}

With the freshly initialized offload_ctx->rekey.sw, tx_list is empty, so
rec becomes container_of(&offload_ctx->rekey.sw.tx_list, struct tls_rec,
list). struct tls_rec starts with its list_head, so rec points into the
middle of the kmalloc'ed offload context; sk_msg_free() then walks an
sk_msg built out of the neighbouring rekey key/IV/rec_seq bytes and
kfree(rec) frees an interior pointer.

tls_sw_drain_tx() -> tls_tx_records() has the same list_first_entry()
assumption on the completion path.

Should tls_set_device_offload_rekey() also defer on
tls_is_partially_sent_record(), and should start_rekey() drain the device
partial record before publishing rekey.sw_ctx?

[ ... ]

> @@ -718,8 +789,13 @@ EXPORT_SYMBOL(tls_get_record);
>  
>  static int tls_device_push_pending_record(struct sock *sk, int flags)
>  {
> +	struct tls_context *tls_ctx = tls_get_ctx(sk);
>  	struct iov_iter iter;
>  
> +	if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags) ||
> +	    test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags))
> +		return tls_sw_push_pending_record(sk, flags);
> +
>  	iov_iter_kvec(&iter, ITER_SOURCE, NULL, 0, 0);
>  	return tls_push_data(sk, &iter, 0, flags, TLS_RECORD_TYPE_DATA);
>  }

[Severity: High]
sendmsg, splice_eof and push_pending_record are all made rekey aware, but
tls_write_space() is not, and tx_conf stays TLS_HW for the whole rekey:

net/tls/tls_main.c:tls_write_space() {
#ifdef CONFIG_TLS_DEVICE
	if (ctx->tx_conf == TLS_HW)
		tls_device_write_space(sk, ctx);
	else
#endif
		tls_sw_write_space(sk, ctx);
}

So while TLS_TX_REKEY_PENDING is set and all data is produced by
tls_sw_sendmsg_locked(), the device handler runs instead of the SW one.

Does that mean the SW tx_work is never rescheduled when send buffer space
frees up?  tls_sw_write_space() is the only place doing:

	if (tls_is_tx_ready(tx_ctx) &&
	    !test_and_set_bit(BIT_TX_SCHEDULED, &tx_ctx->tx_bitmask))
		schedule_delayed_work(&tx_ctx->tx_work.work, 0);

which would leave encrypted records sitting in
offload_ctx->rekey.sw.tx_list after an -EAGAIN from tls_push_sg(), and
tls_sw_drain_tx() returning -EAGAIN so the rekey never completes.

Can the WARN in tls_device_write_space() also fire now?  It has
WARN_ON_ONCE(sk->sk_write_pending), and tls_sw_sendmsg_locked() raises
sk_write_pending in sk_stream_wait_memory() with a SW partial record
outstanding.

Finally, tls_device_write_space() calls tls_push_partial_record() directly
on tls_ctx->partially_sent_record, which during the rekey belongs to the
SW tx_list.  Completing the push there clears partially_sent_record
without removing the owning head tls_rec, so the next tls_tx_records()
re-pushes the same record from msg_en->sg.data[msg_en->sg.curr].  Would
that duplicate ciphertext in the stream and put_page() the record pages a
second time?

> @@ -1105,6 +1181,334 @@ static struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *c
>  	return offload_ctx;
>  }
>  

[ ... ]

> +static int tls_device_start_rekey(struct sock *sk,
> +				  struct tls_context *ctx,
> +				  struct tls_offload_context_tx *offload_ctx,
> +				  struct tls_crypto_info *new_crypto_info)
> +{
> +	bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
> +	bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);

[ ... ]

> +	if (rekey_pending || rekey_failed) {
> +		/* Wait for in-flight async encryptions submitted to this tfm
> +		 * with the previous key before changing it.
> +		 */
> +		rc = tls_encrypt_async_wait(&offload_ctx->rekey.sw);
> +		if (rc)
> +			return rc;

[ ... ]

> +		old_aead = offload_ctx->rekey.sw.aead_send;
> +		offload_ctx->rekey.sw.aead_send = new_aead;
> +		crypto_free_aead(old_aead);
> +
> +		memcpy(offload_ctx->rekey.tx.iv, salt, cipher_desc->salt);
> +		memcpy(offload_ctx->rekey.tx.iv + cipher_desc->salt, iv,
> +		       cipher_desc->iv);
> +		memcpy(offload_ctx->rekey.tx.rec_seq, rec_seq,
> +		       cipher_desc->rec_seq);

[Severity: Medium]
The other branch below flushes a pending open record before switching
keys:

		/* Flush any HW open_record before switching to SW */
		if (tls_is_pending_open_record(ctx)) {
			rc = ctx->push_pending_record(sk, 0);

This branch swaps aead_send and rewrites rekey.tx.iv / rekey.tx.rec_seq
with no equivalent flush of offload_ctx->rekey.sw.open_rec.
tls_encrypt_async_wait() only waits for requests already submitted to the
tfm; compare the new tls_sw_drain_tx(), which pushes the pending open
record first:

	if (tls_is_pending_open_record(ctx))
		tls_sw_push_pending_record(sk, 0);
	tls_encrypt_async_wait(sw_ctx);

If a preceding sendmsg with MSG_MORE left plaintext in
offload_ctx->rekey.sw.open_rec, is that record then encrypted under the
newest key and newest rec_seq, instead of the key that was current when
the bytes were accepted?

> +
> +		if (rekey_failed) {
> +			set_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
> +			clear_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
> +			TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
> +			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
> +		}

[Severity: High]
This re-arms the boundary based validator and the READY logic without
refreshing ctx->rekey.boundary_seq; only the else branch below writes it:

		WRITE_ONCE(ctx->rekey.boundary_seq, tcp_sk(sk)->write_seq);

While TLS_TX_REKEY_FAILED is set the socket stays in SW mode and can send
an unbounded amount of data, so by the time a second KeyUpdate arrives
boundary_seq can be arbitrarily far behind write_seq.  Since before() is a
signed 32-bit comparison, once more than 2 GiB has been written past the
stale boundary:

net/tls/tls_device_fallback.c:tls_validate_xmit_skb_rekey() {
	boundary_seq = READ_ONCE(tls_ctx->rekey.boundary_seq);
	if (!before(tcp_seq, boundary_seq))
		return skb;

	return tls_sw_fallback(sk, skb);
}

would brand new, already SW-encrypted packets be handed to
tls_sw_fallback() and encrypted a second time?

And symmetrically in tls_tcp_clean_acked():

		if (!before(acked_seq, boundary_seq))
			set_bit(TLS_TX_REKEY_READY, &tls_ctx->flags);

with acked_seq more than 2 GiB past the stale boundary the READY bit stops
being set, so the second rekey can never complete.  Should the
rekey_failed path also update boundary_seq to the current write_seq?

> +	} else {
> +		/* Flush any HW open_record before switching to SW */
> +		if (tls_is_pending_open_record(ctx)) {
> +			rc = ctx->push_pending_record(sk, 0);
> +			if (rc < 0)
> +				return rc;
> +		}

[ ... ]

> +static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,
> +				     bool deferred)
> +{

[ ... ]

> +	rc = tls_device_dev_add_tx(sk, netdev, &offload_ctx->rekey.crypto_send.info,
> +				   tcp_sk(sk)->write_seq);
> +	if (rc) {
> +		crypto_free_aead(new_aead);
> +		goto release_lock;
> +	}
> +
> +	/* Point of no return: HW is live with the new key. Swap in the new
> +	 * fallback tfm and drop the old one; the remaining steps cannot fail.
> +	 */
> +	old_aead = offload_ctx->aead_send;
> +	offload_ctx->aead_send = new_aead;
> +	crypto_free_aead(old_aead);
> +	clear_bit(TLS_TX_DEV_CLOSED, &ctx->flags);

[Severity: High]
Is old_aead guaranteed to be unused at this point?  During the whole
PENDING window the installed validator routes every pre-boundary segment
into tls_sw_fallback(), which reads offload_ctx->aead_send locklessly from
qdisc dequeue on any CPU:

net/tls/tls_device_fallback.c:tls_enc_skb() {
	aead_req = tls_alloc_aead_request(ctx->aead_send, GFP_ATOMIC);
	...
	salt = crypto_info_salt(&tls_ctx->crypto_send.info, cipher_desc);
	...
	if (tls_enc_records(tls_ctx, aead_req, ctx->aead_send, sg_in, sg_out,

The memcpys just below overwrite ctx->tx.iv, ctx->tx.rec_seq and
ctx->crypto_send.info, which the same function reads.  There is no
synchronize_net(), RCU section or refcount between the free and those
readers, whereas tls_device_down() does exactly that before tearing TX
state down:

		rcu_assign_pointer(ctx->netdev, NULL);
		...
		/* Sync with inflight packets. ... */
		synchronize_net();

Can a fallback that already loaded old_aead (a queued pre-boundary
retransmission dequeued on another CPU) end up using a freed
crypto_aead, or re-encrypt an old record with the new key and salt?

> +
> +release_lock:
> +	up_read(&device_offload_lock);
> +
> +	if (rc)
> +		goto rekey_fail;
> +
> +	spin_lock_irqsave(&offload_ctx->lock, flags);
> +	memcpy(&rcd_sn, offload_ctx->rekey.tx.rec_seq, sizeof(rcd_sn));
> +	offload_ctx->unacked_record_sn = be64_to_cpu(rcd_sn) - 1;
> +	spin_unlock_irqrestore(&offload_ctx->lock, flags);

[Severity: High]
The "- 1" here encodes the invariant documented in alloc_offload_ctx_tx():

	/* start at rec_seq - 1 to account for the start marker record */
	memcpy(&rcd_sn, ctx->tx.rec_seq, sizeof(rcd_sn));
	offload_ctx->unacked_record_sn = be64_to_cpu(rcd_sn) - 1;

that is, exactly one start marker sits at the head of records_list.  On
the inline (non-deferred) path, can the previous start marker still be
linked?  tls_has_unacked_records() deliberately skips markers:

	list_for_each_entry(info, &offload_ctx->records_list, list) {
		if (!tls_record_is_start_marker(info)) {
			has_unacked = true;
			break;
		}
	}

so with records_list == [old_marker] the rekey is not deferred, and
tls_device_commit_start_marker() then does list_add_tail_rcu(), giving
[old_marker, new_marker, R1, ...] while the counter was rebased as if one
marker existed.  tls_get_record() increments record_sn per list entry, so
would every subsequent record report SN + 1, and would the skew become
permanent once tls_tcp_clean_acked() deletes both markers and adds
deleted_records on top of the rebased value?

The rebase and the records_list update are also in two separate critical
sections, so tls_tcp_clean_acked() can run in between and apply
ctx->unacked_record_sn += deleted_records to the freshly rebased value.

That SN is what drivers program into the NIC on TX resync and what
tls_enc_skb() uses to build the record nonce, so a mismatch means
retransmissions authenticated under the wrong sequence number.

> +
> +	memcpy(ctx->tx.iv, offload_ctx->rekey.tx.iv,
> +	       cipher_desc->salt + cipher_desc->iv);

[ ... ]

> +static int tls_set_device_offload_rekey(struct sock *sk,
> +					struct tls_context *ctx,
> +					struct net_device *netdev,
> +					struct tls_crypto_info *new_crypto_info)
> +{
> +	struct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);
> +	bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
> +	bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
> +	bool defer = true;
> +	int rc;
> +
> +	if (!rekey_pending && !rekey_failed)
> +		defer = tls_has_unacked_records(offload_ctx) ||
> +			tls_is_pending_open_record(ctx);

[ ... ]

> @@ -1354,7 +1769,10 @@ static int tls_device_down(struct net_device *netdev)
>  		/* Stop offloaded TX and switch to the fallback.
>  		 * tls_is_skb_tx_device_offloaded will return false.
>  		 */
> -		WRITE_ONCE(ctx->sk->sk_validate_xmit_skb, tls_validate_xmit_skb_sw);
> +		if (!test_bit(TLS_TX_REKEY_PENDING, &ctx->flags) &&
> +		    !test_bit(TLS_TX_REKEY_FAILED, &ctx->flags))
> +			WRITE_ONCE(ctx->sk->sk_validate_xmit_skb,
> +				   tls_validate_xmit_skb_sw);

[Severity: Medium]
This reads the rekey flags and writes sk_validate_xmit_skb under
down_write(&device_offload_lock), but tls_device_complete_rekey() clears
those flags and publishes the validator *after* up_read():

release_lock:
	up_read(&device_offload_lock);
	...
	clear_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
	...
	smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);

Can this decision be undone right after it is made?  CPU0 finishes
tls_device_dev_add_tx() and calls up_read(); CPU1 (NETDEV_DOWN) takes
down_write(), decides here, sets ctx->netdev = NULL, synchronize_net() and
tls_dev_del(); CPU0 then stores tls_validate_xmit_skb, overwriting CPU1's
choice.

The socket then reports HW TX offload again after the device context was
deleted, since tls_is_skb_tx_device_offloaded() is defined as
smp_load_acquire(&sk->sk_validate_xmit_skb) == &tls_validate_xmit_skb,
which is what the comment above is trying to avoid.

Should the flag clear and the validator store happen while
device_offload_lock is still held?

> diff --git a/net/tls/tls_device_fallback.c b/net/tls/tls_device_fallback.c
> index 1110f7ac6bcb..64ac4ef4012b 100644
> --- a/net/tls/tls_device_fallback.c
> +++ b/net/tls/tls_device_fallback.c
> @@ -435,6 +435,30 @@ struct sk_buff *tls_validate_xmit_skb_sw(struct sock *sk,
>  	return tls_sw_fallback(sk, skb);
>  }
>  
> +struct sk_buff *tls_validate_xmit_skb_rekey(struct sock *sk,
> +					    struct net_device *dev,
> +					    struct sk_buff *skb)
> +{
> +	struct tls_context *tls_ctx = tls_get_ctx(sk);
> +	u32 tcp_seq = ntohl(tcp_hdr(skb)->seq);
> +	u32 boundary_seq;
> +
> +	if (test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags))
> +		return skb;

[Severity: High]
This returns every packet unmodified, including pre-boundary ones, which
seems to contradict both the comment three lines below ("Packet before
boundary means retransmit of old data, use SW fallback with the old key")
and the flag's own documentation in include/net/tls.h ("HW rekey failed,
permanently stay in SW encrypt mode").

Pre-boundary packets are HW-era records carrying plaintext plus a
placeholder tag, and the NIC can no longer encrypt them because
tls_device_complete_rekey() already ran tls_dev_del() and set
TLS_TX_DEV_CLOSED before falling into rekey_fail.  On that path the newly
allocated tfm is freed and offload_ctx->aead_send still holds the usable
old key, so tls_sw_fallback() would work.

If an skb below boundary_seq is validated at qdisc dequeue just after
TLS_TX_REKEY_FAILED becomes visible, is it emitted in cleartext?  Should
the FAILED check come after the boundary comparison?

> +
> +	/* If this packet is at or after the rekey boundary, it's already
> +	 * SW-encrypted with the new key, pass through unchanged
> +	 */
> +	boundary_seq = READ_ONCE(tls_ctx->rekey.boundary_seq);
> +	if (!before(tcp_seq, boundary_seq))
> +		return skb;
> +
> +	/* Packet before boundary means retransmit of old data,
> +	 * use SW fallback with the old key
> +	 */
> +	return tls_sw_fallback(sk, skb);
> +}
> +
> diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
> index a62e94508c8b..536f26d68d9d 100644
> --- a/net/tls/tls_main.c
> +++ b/net/tls/tls_main.c
> @@ -445,8 +453,16 @@ static int do_tls_getsockopt_conf(struct sock *sk, sockopt_t *opt, int tx)
>  
>  	/* get user crypto info */
>  	if (tx) {
> -		crypto_info = &ctx->crypto_send.info;
> -		cctx = &ctx->tx;
> +		if (ctx->rekey.cipher_ctx) {
> +			struct tls_offload_context_tx *offload_ctx =
> +				tls_offload_ctx_tx(ctx);
> +
> +			crypto_info = &offload_ctx->rekey.crypto_send.info;
> +			cctx = &offload_ctx->rekey.tx;
> +		} else {
> +			crypto_info = &ctx->crypto_send.info;
> +			cctx = &ctx->tx;
> +		}
>  	} else {
>  		crypto_info = &ctx->crypto_recv.info;
>  		cctx = &ctx->rx;

[ ... ]

> @@ -881,12 +897,34 @@ static int do_tls_setsockopt(struct sock *sk, int optname, sockptr_t optval,
>  
>  	switch (optname) {
>  	case TLS_TX:
> -	case TLS_RX:
> +	case TLS_RX: {

[ ... ]

> +		bool tx = optname == TLS_TX;
> +
> +		if (tx)
> +			mutex_lock(&tls_get_ctx(sk)->tx_lock);
>  		lock_sock(sk);
> -		rc = do_tls_setsockopt_conf(sk, optval, optlen,
> -					    optname == TLS_TX);
> +		rc = do_tls_setsockopt_conf(sk, optval, optlen, tx);
>  		release_sock(sk);
> +		if (tx)
> +			mutex_unlock(&tls_get_ctx(sk)->tx_lock);
>  		break;
> +	}

[Severity: Medium]
The changelog describes only the TX rekey state machine, but two changes
here affect sockets that never do a KeyUpdate.

tx_lock is now taken for every TLS_TX setsockopt, including the initial
key installation and SW-only sockets, adding a tx_lock -> lock_sock
ordering on a path that previously took only lock_sock.  The reasoning
exists only as the comment in this hunk.

do_tls_getsockopt_conf() above now reports the *pending* key's
crypto_info, IV and rec_seq from tls_offload_ctx_tx(ctx)->rekey whenever
ctx->rekey.cipher_ctx is set, so getsockopt(SOL_TLS, TLS_TX) returns
different data than before while a rekey is pending.

Could the commit message mention both of these behaviour changes?

> diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
> index 7b6ab992c86e..89f60cedb142 100644
> --- a/net/tls/tls_sw.c
> +++ b/net/tls/tls_sw.c
> @@ -2411,6 +2410,23 @@ void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx)
>  	sw_ctx->tx_work.sk = sk;
>  }
>  
> +int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx)
> +{
> +	struct tls_sw_context_tx *sw_ctx = tls_sw_ctx_tx(ctx);
> +	int rc;
> +
> +	if (tls_is_pending_open_record(ctx))
> +		tls_sw_push_pending_record(sk, 0);
> +	tls_encrypt_async_wait(sw_ctx);
> +	rc = tls_tx_records(sk, -1);
> +	if (rc < 0 || tls_is_partially_sent_record(ctx) ||
> +	    tls_is_pending_open_record(ctx))
> +		return rc < 0 ? rc : -EAGAIN;
> +
> +	cancel_delayed_work_sync(&sw_ctx->tx_work.work);
> +	return 0;
> +}

[Severity: Medium]
Can this return 0 with records still linked on sw_ctx->tx_list?  The
return values of tls_sw_push_pending_record() and
tls_encrypt_async_wait() are discarded and list_empty(&sw_ctx->tx_list) is
never checked.

On an async encryption error (or sk->sk_err already set),
tls_encrypt_done() takes the error branch, sets rec = NULL and therefore
never marks the record tx_ready nor unlinks it:

	if (err || sk->sk_err) {
		rec = NULL;
		...
	}

	if (rec) {
		smp_store_mb(rec->tx_ready, true);

tls_tx_records() then hits the non-ready head, breaks and returns 0, so
tls_device_complete_rekey() proceeds and ends with:

	memzero_explicit(&offload_ctx->rekey, sizeof(offload_ctx->rekey));

wiping the tx_list head while the tls_rec is still linked to it.  Does
that leak the rec together with its msg_plaintext / msg_encrypted pages
and their sk_mem charges, and silently drop a record that sendmsg already
accepted?

The same applies to an orphaned open_rec: tls_push_record() clears
pending_open_record_frags before encryption, so tls_is_pending_open_record()
reads false while open_rec is still set.

Re: [PATCH v16 08/10] tls: device: add RX KeyUpdate support

[Jakub Kicinski]

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
---
tls: device: add RX KeyUpdate support

This adds RX KeyUpdate handling for TLS device offload by retaining the old
AEAD/IV/rec_seq on tls_offload_context_rx and classifying records against a
TCP sequence boundary, with the new key's tls_dev_add deferred until the old
key is drained. It also adds a new tls_dev_rx_rekey_fixup driver callback
(implemented for mlx5e) and three RX rekey statistics.

A few questions on the new paths below.

> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
> index 337201f12895..71c9d3ca947d 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
> @@ -86,10 +86,20 @@ static int mlx5e_ktls_resync(struct net_device *netdev,
>  	return 0;
>  }
>  
> +static void mlx5e_ktls_rx_rekey_fixup(struct sk_buff *skb)
> +{
> +	struct sk_buff *frag_iter;
> +
> +	skb->decrypted = 1;
> +	skb_walk_frags(skb, frag_iter)
> +		frag_iter->decrypted = 1;
> +}
> +

[Severity: High]
Should this be restricted to the fragments the device actually transformed?

tls_device_decrypted() invokes the callback for any pre-boundary record that
is neither fully encrypted nor fully decrypted, that is, for genuinely mixed
records:

	if (!is_decrypted && ctx->rekey.rekey_fixup)
		ctx->rekey.rekey_fixup(skb);

	return tls_device_reencrypt_old_key(sk, ctx, sw_ctx, tls_ctx);

and tls_device_reencrypt() decides per fragment whether to write the
transformed buffer back:

	if (skb_iter->decrypted) {
		err = skb_store_bits(skb_iter, frag_pos, buf, copy);

In mlx5e_ktls_handle_rx_skb() a cleared skb->decrypted is ambiguous: it is
left clear for CQE_TLS_OFFLOAD_ERROR (payload transformed, auth failed) and
also for CQE_TLS_OFFLOAD_RESYNC and non-offloaded packets (payload
untouched).

Since the new tls_device_rx_del_key() destroys the NIC RX context mid-stream
at KeyUpdate parse time, a record can straddle that point with leading
fragments transformed by the device and trailing fragments still carrying
wire ciphertext.

Does setting decrypted = 1 on the head skb and every fragment then make
tls_device_reencrypt() overwrite untouched ciphertext, so the following SW
AEAD decrypt of that record fails and tls_rx_one_record() aborts the
connection with -EBADMSG?

>  static const struct tlsdev_ops mlx5e_ktls_ops = {
>  	.tls_dev_add = mlx5e_ktls_add,
>  	.tls_dev_del = mlx5e_ktls_del,
>  	.tls_dev_resync = mlx5e_ktls_resync,
> +	.tls_dev_rx_rekey_fixup = mlx5e_ktls_rx_rekey_fixup,
>  };
>  
>  bool mlx5e_is_ktls_rx(struct mlx5_core_dev *mdev)
> diff --git a/include/net/tls.h b/include/net/tls.h
> index 7c2cf51dc887..de4e25b034d6 100644
> --- a/include/net/tls.h
> +++ b/include/net/tls.h
> @@ -211,10 +211,11 @@ enum tls_context_flags {
>  	 * to be atomic.
>  	 */
>  	TLS_TX_SYNC_SCHED = 1,
> -	/* tls_dev_del was called for the RX side, device state was released,
> -	 * but tls_ctx->netdev might still be kept, because TX-side driver
> -	 * resources might not be released yet. Used to prevent the second
> -	 * tls_dev_del call in tls_device_down if it happens simultaneously.
> +	/* tls_dev_del was called for the RX side: the NIC currently holds no
> +	 * RX offload context. Set whenever that context is released (socket
> +	 * teardown, tls_device_down, or during a rekey before re-add) and
> +	 * cleared when tls_dev_add re-establishes it. Readers use it to avoid
> +	 * a double tls_dev_del and to suppress resync while the NIC has no key.
>  	 */
>  	TLS_RX_DEV_CLOSED = 2,

[Severity: Low]
The reworded comment says the bit is set whenever the RX context is released,
including socket teardown, but tls_device_offload_cleanup_rx() still sets it
only in the tx_conf == TLS_HW branch:

	if (tls_ctx->tx_conf != TLS_HW) {
		dev_put(netdev);
		rcu_assign_pointer(tls_ctx->netdev, NULL);
	} else {
		set_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
	}

On the common tx_conf != TLS_HW teardown the RX context is released with the
flag left clear. Should the comment or the code be adjusted so readers of the
flag get the documented invariant?

>  	/* Flag for TX HW context deleted during failed rekey.
> @@ -321,6 +322,7 @@ struct tlsdev_ops {
>  	int (*tls_dev_resync)(struct net_device *netdev,
>  			      struct sock *sk, u32 seq, u8 *rcd_sn,
>  			      enum tls_offload_ctx_dir direction);
> +	void (*tls_dev_rx_rekey_fixup)(struct sk_buff *skb);
>  };

[Severity: Low]
This isn't a bug, but a new member of a driver-facing ops table is added with
no kernel-doc describing when the core calls it or what the driver must
guarantee, and mlx5e is the only implementer.

No other driver is reachable here today (nfp_net_tls_add() returns
-EOPNOTSUPP unless crypto_info->version == TLS_1_2_VERSION, and rekey is only
permitted for TLS 1.3, while chcr_ktls, cxgb4 and funeth are TX-only), so
could a short comment be added describing the expected semantics for future
RX-offload drivers?

>  
>  enum tls_offload_sync_type {

[ ... ]

> diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
> index 85c74d0816b7..c34268cd9531 100644
> --- a/net/tls/tls_device.c
> +++ b/net/tls/tls_device.c

[ ... ]

> @@ -186,6 +196,82 @@ static void tls_device_commit_start_marker(struct sock *sk,
>  	tcp_write_collapse_fence(sk);
>  }
>  
> +static int tls_device_dev_add_rx(struct sock *sk, struct tls_context *tls_ctx,
> +				 struct net_device *netdev,
> +				 struct tls_crypto_info *crypto_info,
> +				 u32 cur_seq, bool is_rekey)
> +{
> +	const struct tls_cipher_desc *cipher_desc;
> +	char *rec_seq;
> +	int rc;
> +
> +	cipher_desc = get_cipher_desc(crypto_info->cipher_type);
> +	DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
> +
> +	rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk,
> +					     TLS_OFFLOAD_CTX_DIR_RX,
> +					     crypto_info, cur_seq);
> +	rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
> +	trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
> +				     cur_seq, rec_seq, rc);
> +	if (!rc) {
> +		clear_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
> +		clear_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
> +		if (is_rekey)
> +			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
> +	} else if (is_rekey) {
> +		set_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
> +		set_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
> +		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYFALLBACK);
> +	}
> +	return rc;
> +}
> +
> +static void tls_device_deferred_dev_add_rx(struct sock *sk,
> +					   struct tls_context *tls_ctx,
> +					   struct tls_offload_context_rx *ctx)
> +{
> +	struct net_device *netdev;
> +
> +	ctx->dev_add_pending = 0;
> +
> +	down_read(&device_offload_lock);
> +	netdev = rcu_dereference_protected(tls_ctx->netdev,
> +					   lockdep_is_held(&device_offload_lock));
> +	if (netdev)
> +		tls_device_dev_add_rx(sk, tls_ctx, netdev,
> +				      &tls_ctx->crypto_recv.info,
> +				      tcp_sk(sk)->copied_seq, true);

[Severity: High]
Is the record number handed to the device here still valid?

crypto_recv.info holds the rec_seq userspace supplied at setsockopt time,
which is 0 for a TLS 1.3 KeyUpdate, and tls_sw_ctx_finalize() copies it in
once. The live counter is tls_ctx->rx.rec_seq, advanced per record by
tls_rx_one_record():

	tls_advance_record_sn(sk, prot, &tls_ctx->rx);

This deferred add only runs once the first post-boundary record is
classified, so at least one record has already been drained with the new key,
while cur_seq is the current tcp_sk(sk)->copied_seq.

On the driver side fill_static_params() does:

	initial_rn  = MLX5_ADDR_OF(tls_static_params, ctx, initial_record_number);
	memcpy(initial_rn,  rec_seq, rec_seq_sz);

and fill_progress_params() sets next_record_tcp_sn = start_offload_tcp_sn, so
the (TCP seq, record number) pair programmed into the NIC looks inconsistent
by the number of records already consumed. Should tls_ctx->rx.rec_seq be
passed instead? In copy_mode copied_seq is also already past the end of the
record just classified, which would add a further one-record skew.

> +	else
> +		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYFALLBACK);
> +	up_read(&device_offload_lock);
> +	TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYINPROGRESS);
> +}
> +
> +void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx)
> +{
> +	struct net_device *netdev;
> +
> +	if (ctx->rx_conf != TLS_HW)
> +		return;
> +	if (test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
> +		return;
> +
> +	down_read(&device_offload_lock);
> +	netdev = rcu_dereference_protected(ctx->netdev,
> +					   lockdep_is_held(&device_offload_lock));
> +	if (!netdev) {
> +		up_read(&device_offload_lock);
> +		return;
> +	}
> +
> +	set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
> +	synchronize_net();
> +	netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
> +					TLS_OFFLOAD_CTX_DIR_RX);
> +	up_read(&device_offload_lock);
> +}

[Severity: Medium]
This now runs from the per-record receive path:

tls_rx_one_record() -> tls_check_pending_rekey() -> tls_device_rx_del_key()

with lock_sock(sk) and the TLS RX reader lock held. Here we do
synchronize_net() and then ->tls_dev_del(), and mlx5e_ktls_del_rx() does a
second synchronize_net(), cancel_work_sync() and, when the cancel fails,
wait_for_completion(&priv_rx->add_ctx) waiting on NAPI.

Is holding the socket lock across two RCU grace periods plus an unbounded
wait acceptable in the receive path? A peer only has to send a handshake
record whose first byte is TLS_HANDSHAKE_KEYUPDATE, and the cost applies per
offloaded connection while softirq receive can only append to the
size-limited sk_backlog.

Related: device_offload_lock is now taken for read from the RX data path both
here and in tls_device_deferred_dev_add_rx() (which calls ->tls_dev_add()),
while tls_device_down() holds the same rwsem for write across a per-socket
teardown loop that itself does synchronize_net() and tls_dev_del() per
socket. Was that contention considered?

[Severity: Low]
Documentation/networking/tls-offload.rst still states that tls_dev_del is
called "When the offloaded connection is destroyed" and that "The third TLS
device callback is tls_dev_resync".

After this patch tls_dev_del runs on a live connection (from
tls_check_pending_rekey() and from tls_set_device_offload_rx() before
re-add), tls_dev_add is called again for the same socket and direction, and
there is now a fourth callback. Should that document be updated in the same
series so driver authors don't free per-socket state on tls_dev_del?

> +
>  static void destroy_record(struct tls_record_info *record)
>  {
>  	int i;

[ ... ]

> @@ -1093,13 +1181,56 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
>  	return err;
>  }
>  
> +/*
> + * Temporarily swap in the old key, run
> + * tls_device_reencrypt(), then restore the current key.
> + */
> +static int tls_device_reencrypt_old_key(struct sock *sk,
> +					struct tls_offload_context_rx *ctx,
> +					struct tls_sw_context_rx *sw_ctx,
> +					struct tls_context *tls_ctx)
> +{
> +	struct crypto_aead *saved_aead = sw_ctx->aead_recv;
> +	char saved_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
> +	char saved_rec_seq[TLS_MAX_REC_SEQ_SIZE];
> +	int ret;
> +
> +	memcpy(saved_iv, tls_ctx->rx.iv, sizeof(saved_iv));
> +	memcpy(saved_rec_seq, tls_ctx->rx.rec_seq, sizeof(saved_rec_seq));

[Severity: Low]
saved_iv holds the salt plus implicit IV and saved_rec_seq the record
sequence, and both are left resident on the kernel stack when the function
returns through either exit.

The rest of net/tls scrubs equivalent material with memzero_explicit()
(tls_ctx_free(), tls_sw_ctx_finalize(), tls_device_free_ctx()). Should these
two buffers be zeroized before returning?

> +
> +	sw_ctx->aead_recv = ctx->rekey.old_aead_recv;
> +	memcpy(tls_ctx->rx.iv, ctx->rekey.old_iv, sizeof(ctx->rekey.old_iv));
> +	memcpy(tls_ctx->rx.rec_seq, ctx->rekey.old_rec_seq,
> +	       sizeof(ctx->rekey.old_rec_seq));
> +
> +	ret = tls_device_reencrypt(sk, tls_ctx);
> +
> +	memcpy(ctx->rekey.old_rec_seq, tls_ctx->rx.rec_seq,
> +	       sizeof(ctx->rekey.old_rec_seq));
> +
> +	sw_ctx->aead_recv = saved_aead;
> +	memcpy(tls_ctx->rx.iv, saved_iv, sizeof(saved_iv));
> +	memcpy(tls_ctx->rx.rec_seq, saved_rec_seq, sizeof(saved_rec_seq));
> +
> +	if (ret)
> +		return ret;
> +
> +	tls_bigint_increment(ctx->rekey.old_rec_seq,
> +			     tls_ctx->prot_info.rec_seq_size);
> +	ctx->resync_nh_reset = 1;
> +
> +	return 0;
> +}
> +

[ ... ]

> @@ -1109,10 +1240,50 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
>  		is_encrypted = 0;
>  	}
>  
> -	trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,
> +	rec_start_seq = sw_ctx->strp.copy_mode
> +		? copied_seq - rxm->full_len
> +		: copied_seq;
> +
> +	trace_tls_device_decrypted(sk, rec_start_seq,
>  				   tls_ctx->rx.rec_seq, rxm->full_len,
>  				   is_encrypted, is_decrypted);
>  
> +	if (unlikely(ctx->rekey.old_aead_recv)) {
> +		bool before_nic_boundary =
> +			before(rec_start_seq, ctx->rekey.old_nic_boundary);
> +
> +		if (before_nic_boundary) {

[ ... ]

> +			if (is_encrypted) {
> +				tls_bigint_increment(ctx->rekey.old_rec_seq,
> +						     tls_ctx->prot_info.rec_seq_size);
> +				return 0;
> +			}
> +
> +			/* rekey_fixup sets decrypted flags in case the NIC clears
> +			 * decrypted flags on auth failure
> +			 */
> +			if (!is_decrypted && ctx->rekey.rekey_fixup)
> +				ctx->rekey.rekey_fixup(skb);
> +
> +			return tls_device_reencrypt_old_key(sk, ctx,
> +							    sw_ctx, tls_ctx);
> +		}
> +
> +		crypto_free_aead(ctx->rekey.old_aead_recv);
> +		ctx->rekey.old_aead_recv = NULL;
> +
> +		if (ctx->dev_add_pending)
> +			tls_device_deferred_dev_add_rx(sk, tls_ctx, ctx);
> +	}
> +
>  	if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags))) {
>  		if (likely(is_encrypted || is_decrypted))
>  			return is_decrypted;

[ ... ]

> @@ -1643,13 +1814,31 @@ int tls_set_device_offload(struct sock *sk,
>  	return rc;
>  }
>  
> -int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
> +int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
> +			      struct tls_crypto_info *new_crypto_info)
>  {
> -	struct tls12_crypto_info_aes_gcm_128 *info;
> +	struct tls_crypto_info *crypto_info, *src_crypto_info;
> +	const struct tls_cipher_desc *cipher_desc;
> +	u32 copied_seq = tcp_sk(sk)->copied_seq;
>  	struct tls_offload_context_rx *context;
>  	struct net_device *netdev;
> +	bool was_dev_add_pending;
> +	bool moved_aead_recv = false;
>  	int rc = 0;
>  
> +	/* Rekey is only supported for connections that are already
> +	 * using HW offload. For SW offload connections, the caller
> +	 * should fall back to tls_set_sw_offload() for rekey.
> +	 */
> +	if (new_crypto_info && ctx->rx_conf != TLS_HW)
> +		return -EINVAL;
> +
> +	crypto_info = &ctx->crypto_recv.info;
> +	src_crypto_info = new_crypto_info ?: crypto_info;
> +	cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
> +	if (!cipher_desc || !cipher_desc->offloadable)
> +		return -EINVAL;
> +
>  	netdev = get_netdev_for_sock(sk);
>  	if (!netdev) {
>  		pr_err_ratelimited("%s: netdev not found\n", __func__);
> @@ -1675,29 +1864,88 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
>  		goto release_lock;
>  	}
>  
> -	context = kzalloc_obj(*context);
> -	if (!context) {
> -		rc = -ENOMEM;
> -		goto release_lock;
> +	if (!new_crypto_info) {
> +		context = kzalloc_obj(*context);
> +		if (!context) {
> +			rc = -ENOMEM;
> +			goto release_lock;
> +		}
> +		ctx->priv_ctx_rx = context;
> +	} else {
> +		context = tls_offload_ctx_rx(ctx);
>  	}
> +	was_dev_add_pending = context->dev_add_pending;
>  	context->resync_nh_reset = 1;
>  
> -	ctx->priv_ctx_rx = context;
> -	rc = tls_sw_ctx_init(sk, 0, NULL);
> +	if (new_crypto_info) {
> +		struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);
> +
> +		if (!test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
> +			set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
> +			synchronize_net();
> +			netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
> +							TLS_OFFLOAD_CTX_DIR_RX);
> +		}

[Severity: Medium]
netdev here comes from get_netdev_for_sock(sk), which resolves
__sk_dst_get(sk) -> netdev_sk_get_lowest_dev(), not from
rcu_dereference_protected(ctx->netdev).

For an already-offloaded socket those can differ after a route change or a
bond/team failover, and tls_device_attach() is deliberately skipped on the
rekey path, so ctx->netdev is never updated and no reference is taken for the
device that ends up holding the HW context.

Can this call tls_dev_del() on a device that never held this socket's
context? mlx5e_ktls_del_rx() does priv = netdev_priv(netdev) and then
mlx5_ktls_destroy_key(priv->tls->dek_pool, priv_rx->dek) using the priv_rx
that a different device stored in ctx->driver_state.

And if the following tls_device_dev_add_rx() then succeeds on that other
device, is the context ever released? tls_device_offload_cleanup_rx() does
"if (!netdev) goto out;" on ctx->netdev and tls_device_down() matches
contexts by ctx->netdev.

Both tls_device_rx_del_key() and tls_device_deferred_dev_add_rx() use
rcu_dereference_protected(ctx->netdev) - should this path use the same
source?

> +
> +		if (context->rekey.old_aead_recv &&
> +		    before(copied_seq, context->rekey.old_nic_boundary)) {

[ ... ]

> +			context->dev_add_pending = 1;
> +		} else {
> +			u32 rcv_nxt;
> +
> +			if (context->rekey.old_aead_recv) {
> +				crypto_free_aead(context->rekey.old_aead_recv);
> +				context->rekey.old_aead_recv = NULL;
> +			}
> +
> +			/* flush the backlog so rcv_nxt is accurate */
> +			__sk_flush_backlog(sk);
> +			rcv_nxt = tcp_sk(sk)->rcv_nxt;
> +
> +			if (before(copied_seq, rcv_nxt)) {

[Severity: Medium]
This else branch frees rekey.old_aead_recv and sets it to NULL, and only sets
dev_add_pending inside "if (before(copied_seq, rcv_nxt))".

If a previous rekey already left dev_add_pending == 1 and there is no queued
data now, doesn't the function continue with dev_add_pending == 1 and
old_aead_recv == NULL? "if (!context->dev_add_pending)" below then skips
tls_device_dev_add_rx(), and the deferred completion is only reachable from
inside the old-key block of tls_device_decrypted():

	if (unlikely(ctx->rekey.old_aead_recv)) {
		...
		if (ctx->dev_add_pending)
			tls_device_deferred_dev_add_rx(sk, tls_ctx, ctx);
	}

Is the NIC RX key then never re-installed for the rest of the socket's
lifetime, with TLS_RX_DEV_CLOSED left set, TLSRXREKEYOK counted anyway and
TLSRXREKEYINPROGRESS elevated until close? Should this branch also clear
dev_add_pending?

The sequence looks reachable from peer timing alone: the first rekey with
queued data sets old_nic_boundary = rcv_nxt and dev_add_pending = 1, the
record straddling the boundary is classified by its start so it takes the
reencrypt path without running the deferred add, and if that record carries
the next KeyUpdate the second rekey lands in this else branch.

> +				context->rekey.old_aead_recv = sw_ctx->aead_recv;
> +				sw_ctx->aead_recv = NULL;
> +				moved_aead_recv = true;
> +				memcpy(context->rekey.old_iv, ctx->rx.iv,
> +				       sizeof(context->rekey.old_iv));
> +				memcpy(context->rekey.old_rec_seq, ctx->rx.rec_seq,
> +				       sizeof(context->rekey.old_rec_seq));
> +				context->rekey.old_nic_boundary = rcv_nxt;
> +				context->rekey.rekey_fixup =
> +					netdev->tlsdev_ops->tls_dev_rx_rekey_fixup;

[Severity: High]
This caches a driver function pointer, whose text lifetime is tied to the
driver module, into state that outlives the netdev reference.
tls_device_down() drops that reference without clearing it:

		rcu_assign_pointer(ctx->netdev, NULL);
		set_bit(TLS_RX_DEV_DEGRADED, &ctx->flags);
		synchronize_net();
		...
		dev_put(netdev);

rx_conf stays TLS_HW, so tls_device_decrypted() keeps running for records
already queued and calls ctx->rekey.rekey_fixup(skb) inside the
"if (unlikely(ctx->rekey.old_aead_recv))" block, which is before the
TLS_RX_DEV_DEGRADED bail-out.

If the driver module is unloaded while an RX rekey drain is still pending
(rmmod -> unregister_netdevice -> NETDEV_DOWN -> tls_device_down()), does
this indirect call land in freed module text? Should the pointer be cleared
when the netdev goes away, or fetched from ctx->netdev at call time instead?

> +				context->dev_add_pending = 1;
> +			}
> +		}
> +	}
> +
> +	rc = tls_sw_ctx_init(sk, 0, new_crypto_info);
>  	if (rc)
>  		goto release_ctx;

[Severity: Medium]
On the rekey path tls_dev_del(RX) has already run above, but this is the step
that can still fail: when the old AEAD was moved into rekey.old_aead_recv,
sw_ctx->aead_recv is NULL, so tls_sw_ctx_init() calls crypto_alloc_aead(),
which can fail with -ENOMEM (setauthsize/setkey can fail too).

The release_ctx path restores dev_add_pending and the old AEAD but performs
no hardware rollback, while mlx5e_ktls_del_rx() has already destroyed the
steering rule, the TIR and the DEK:

	if (priv_rx->rule.rule)
		mlx5e_accel_fs_del_sk(priv_rx->rule.rule);
	mlx5e_tir_destroy(&priv_rx->tir);
	mlx5_ktls_destroy_key(priv->tls->dek_pool, priv_rx->dek);

Does the socket then lose RX offload permanently while setsockopt returns an
error implying nothing changed? TLS_RX_DEV_CLOSED stays set, so later
tls_device_rx_del_key(), tls_device_offload_cleanup_rx() and
tls_device_down() all short-circuit and nothing calls tls_dev_add() again,
and no TLSRXREKEYFALLBACK is accounted. Would doing the teardown after the
last failable software step, or re-adding on this error path, work better?

>  
> -	rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_RX,
> -					     &ctx->crypto_recv.info,
> -					     tcp_sk(sk)->copied_seq);
> -	info = (void *)&ctx->crypto_recv.info;
> -	trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
> -				     tcp_sk(sk)->copied_seq, info->rec_seq, rc);
> -	if (rc)
> -		goto free_sw_resources;
> +	if (!context->dev_add_pending) {
> +		rc = tls_device_dev_add_rx(sk, ctx, netdev, src_crypto_info,
> +					   copied_seq, !!new_crypto_info);
> +		if (!new_crypto_info) {
> +			if (rc)
> +				goto free_sw_resources;
> +			tls_device_attach(ctx, sk, netdev);
> +		}
> +	} else if (!was_dev_add_pending) {
> +		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYINPROGRESS);
> +	} else {
> +		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
> +	}
> +
> +	tls_sw_ctx_finalize(sk, 0, new_crypto_info);
>  
> -	tls_device_attach(ctx, sk, netdev);
> -	tls_sw_ctx_finalize(sk, 0, NULL);
>  	up_read(&device_offload_lock);
>  
>  	dev_put(netdev);
> @@ -1709,7 +1957,18 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
>  	tls_sw_free_resources_rx(sk);
>  	down_read(&device_offload_lock);
>  release_ctx:
> -	ctx->priv_ctx_rx = NULL;
> +	if (!new_crypto_info) {
> +		ctx->priv_ctx_rx = NULL;
> +	} else {
> +		context->dev_add_pending = was_dev_add_pending;
> +		if (moved_aead_recv) {
> +			struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);
> +
> +			crypto_free_aead(sw_ctx->aead_recv);
> +			sw_ctx->aead_recv = context->rekey.old_aead_recv;
> +			context->rekey.old_aead_recv = NULL;
> +		}
> +	}
>  release_lock:
>  	up_read(&device_offload_lock);
>  release_netdev:

[ ... ]

> diff --git a/net/tls/tls_proc.c b/net/tls/tls_proc.c
> index adca8561c7da..ae2322481c71 100644
> --- a/net/tls/tls_proc.c
> +++ b/net/tls/tls_proc.c
> @@ -28,8 +28,11 @@ static const struct snmp_mib tls_mib_list[] = {
>  	SNMP_MIB_ITEM("TlsTxRekeyError", LINUX_MIB_TLSTXREKEYERROR),
>  	SNMP_MIB_ITEM("TlsRxRekeyReceived", LINUX_MIB_TLSRXREKEYRECEIVED),
>  	SNMP_MIB_ITEM("TlsTxRekeyFallback", LINUX_MIB_TLSTXREKEYFALLBACK),
> +	SNMP_MIB_ITEM("TlsRxRekeyFallback", LINUX_MIB_TLSRXREKEYFALLBACK),
>  	SNMP_MIB_ITEM("TlsTxRekeyInProgress", LINUX_MIB_TLSTXREKEYINPROGRESS),
> +	SNMP_MIB_ITEM("TlsRxRekeyInProgress", LINUX_MIB_TLSRXREKEYINPROGRESS),
>  	SNMP_MIB_ITEM("TlsTxRekeyAborted", LINUX_MIB_TLSTXREKEYABORTED),
> +	SNMP_MIB_ITEM("TlsRxRekeyAborted", LINUX_MIB_TLSRXREKEYABORTED),
>  };

[Severity: Low]
These three names become visible in /proc/net/tls_stat, but the "Statistics"
section of Documentation/networking/tls.rst, which enumerates every field of
that file, still ends at TlsRxRekeyReceived. Could the new counters be
documented there, including when TlsRxRekeyAborted and TlsRxRekeyFallback
increment? The TX-side siblings added earlier in this series are missing from
that list as well.

>  
>  static int tls_statistics_seq_show(struct seq_file *seq, void *v)

[ ... ]

Cross-instance finding from sashiko-gemini (af143e51c70a6781742cb38aa44153feb2fdbfb052af92dccb83139af70282d4):
[Severity: High]
Single-packet records bypass XOR-undo logic during KeyUpdate.

Re: [PATCH v16 09/10] tls: device: add tracepoints for the KeyUpdate path

[Jakub Kicinski]

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
---
tls: device: add tracepoints for the KeyUpdate path

This adds four trace events to net/tls/trace.h and wires them into the
RX and TX rekey paths in net/tls/tls_device.c. The events are intended to
let an operator follow the KeyUpdate state machine: rekey start, the
old-key reencrypt pass, boundary crossing, and TX rekey completion
failure in sendmsg.

> diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
> index c34268cd9531..9a4c121dbebb 100644
> --- a/net/tls/tls_device.c
> +++ b/net/tls/tls_device.c
> @@ -760,8 +760,14 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
>  	lock_sock(sk);
>  
>  	/* Old-key records all ACKed; switch back to HW. */
> -	if (test_bit(TLS_TX_REKEY_READY, &tls_ctx->flags))
> -		tls_device_complete_rekey(sk, tls_ctx, true);
> +	if (test_bit(TLS_TX_REKEY_READY, &tls_ctx->flags)) {
> +		rc = tls_device_complete_rekey(sk, tls_ctx, true);
> +		/* On failure the READY bit is left set; the next sendmsg
> +		 * retries.
> +		 */
> +		if (rc)
> +			trace_tls_device_complete_rekey_fail(sk, rc);
> +	}

[Severity: Low]
Can this event ever fire for an actual HW rekey failure?

tls_device_complete_rekey() has only one non-zero return, the drain at
the top:

net/tls/tls_device.c:tls_device_complete_rekey() {
	...
	rc = tls_sw_drain_tx(sk, ctx);
	if (rc)
		return rc;
	...
}

Every other failure (netdev gone, crypto_alloc_aead(),
crypto_aead_setkey(), crypto_aead_setauthsize(), tls_device_dev_add_tx())
goes to release_lock and then rekey_fail, which returns success:

net/tls/tls_device.c:tls_device_complete_rekey() {
rekey_fail:
	...
	set_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
	...
	return 0;
}

So the case that permanently drops the connection to SW encryption emits
nothing, while the case that does emit the event named _fail is
tls_sw_drain_tx() returning -EAGAIN:

net/tls/tls_sw.c:tls_sw_drain_tx() {
	if (rc < 0 || tls_is_partially_sent_record(ctx) ||
	    tls_is_pending_open_record(ctx))
		return rc < 0 ? rc : -EAGAIN;
}

which is a transient retry rather than a failure. Should the tracepoint
be emitted from the rekey_fail path instead, or in addition?

> @@ -1267,6 +1273,9 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
>  				return 0;
>  			}
>  
> +			trace_tls_device_rekey_reencrypt(sk, rec_start_seq,
> +							 ctx->rekey.old_nic_boundary);
> +
>  			/* rekey_fixup sets decrypted flags in case the NIC clears
>  			 * decrypted flags on auth failure
>  			 */
> @@ -1277,6 +1286,8 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
>  							    sw_ctx, tls_ctx);
>  		}
>  
> +		trace_tls_device_rekey_done(sk, rec_start_seq,
> +					    ctx->rekey.old_nic_boundary);
>  		crypto_free_aead(ctx->rekey.old_aead_recv);
>  		ctx->rekey.old_aead_recv = NULL;
>  
> @@ -1923,6 +1934,8 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
>  					netdev->tlsdev_ops->tls_dev_rx_rekey_fixup;
>  				context->dev_add_pending = 1;
>  			}
> +			trace_tls_device_rekey_start(sk, copied_seq, rcv_nxt,
> +						     before(copied_seq, rcv_nxt));
>  		}
>  	}

[Severity: Low]
Is this the only arm that should be traced?

The call sits inside the else arm of the RX rekey branch. The other arm
in tls_set_device_offload_rx() also accepts a rekey and defers dev_add,
and emits nothing:

net/tls/tls_device.c:tls_set_device_offload_rx() {
		if (context->rekey.old_aead_recv &&
		    before(copied_seq, context->rekey.old_nic_boundary)) {
			/* Previous rekey still draining. ... */
			context->dev_add_pending = 1;
		} else {
			...
			trace_tls_device_rekey_start(...);
		}
}

That first arm is the "old-key data is still queued, dev_add deferred"
state the commit message attributes to inflight=1. Since
tls_device_decrypted() emits trace_tls_device_rekey_done() for any
non-NULL ctx->rekey.old_aead_recv, wouldn't a second KeyUpdate arriving
while the previous one is still draining produce a rekey_done with no
preceding rekey_start?

Also, inside the traced arm the event fires even when
before(copied_seq, rcv_nxt) is false, in which case no old-key state is
created and rcv_nxt is never stored into ctx->rekey.old_nic_boundary.
Does the reported nic_boundary describe anything in that case?

[Severity: Low]
Are the start and done events meant to pair up?

trace_tls_device_rekey_start() is emitted here, before the rekey can
still be aborted. If the following tls_sw_ctx_init() fails, the
release_ctx path undoes the state that was just traced:

net/tls/tls_device.c:tls_set_device_offload_rx() {
release_ctx:
	...
		context->dev_add_pending = was_dev_add_pending;
		if (moved_aead_recv) {
			...
			sw_ctx->aead_recv = context->rekey.old_aead_recv;
			context->rekey.old_aead_recv = NULL;
		}
}

so a rekey_start is recorded for a rekey that did not take effect, with
no cancelling event.

In the other direction, rekey_done is emitted only at the free site in
tls_device_decrypted(). old_aead_recv is also freed because the boundary
was already crossed here:

net/tls/tls_set_device_offload_rx() {
			if (context->rekey.old_aead_recv) {
				crypto_free_aead(context->rekey.old_aead_recv);
				context->rekey.old_aead_recv = NULL;
			}
}

and on the RX teardown and context-free paths, none of which emit
rekey_done. Should those sites emit the event too, or should the commit
message avoid describing the events as a start/done pair?

> diff --git a/net/tls/trace.h b/net/tls/trace.h
> index 2d8ce4ff3265..2a90b77d75e8 100644
> --- a/net/tls/trace.h
> +++ b/net/tls/trace.h
> @@ -192,6 +192,104 @@ TRACE_EVENT(tls_device_tx_resync_send,
>  	)
>  );
>  
> +TRACE_EVENT(tls_device_rekey_start,
> +
> +	TP_PROTO(struct sock *sk, u32 copied_seq, u32 nic_boundary,
> +		 bool inflight),
> +
> +	TP_ARGS(sk, copied_seq, nic_boundary, inflight),
> +
> +	TP_STRUCT__entry(
> +		__field(	struct sock *,	sk		)
> +		__field(	u32,		copied_seq	)
> +		__field(	u32,		nic_boundary	)
> +		__field(	bool,		inflight	)
> +	),

[ ... ]

Re: [PATCH v16 10/10] selftests: net: add TLS hardware offload test

[Jakub Kicinski]

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
---
selftests: net: add TLS hardware offload test

This adds a two-node kTLS hardware offload selftest: a C helper acts as
either TLS client or server, and a Python harness drives it over
NetDrvEpEnv and checks the /proc/net/tls_stat counters.  It covers TLS
1.2/1.3, AES-GCM-128/256, rekey with several buffer sizes, and burst
variants that stress TX and RX rekey.

A few questions about the helper and the harness below.

> diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
> new file mode 100644
> index 000000000000..1882e2f073ad
> --- /dev/null
> +++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
> @@ -0,0 +1,975 @@

[ ... ]

> +static int do_client(void)
> +{

[ ... ]

> +		n = send(csk, buf, this_size, 0);
> +		if (n != this_size) {
> +			printf("FAIL: send failed: %s\n", strerror(errno));
> +			goto out;
> +		}
> +
> +		if (!burst_mode) {
> +			printf("Sent %zd bytes (iteration %d)\n", n, i);
> +			if (client_recv_echo(csk, buf, echo_buf, n) < 0)
> +				goto out;
> +		}

[Severity: Medium]
Can echo (non-burst) mode deadlock for large payloads?

do_client() issues one blocking send() of the whole payload and only
calls client_recv_echo() afterwards, so it does not read anything from
the socket while the send is outstanding.

On the other side, do_server() reads one record with recv_flags == 0 and
then blocks inside server_echo_send() before it returns to recv():

	for (sent = 0; sent < n; sent += ret) {
		ret = send(fd, buf + sent, n - sent, 0);

Once the echo direction fills up (server send buffer plus client receive
buffer, which cannot autotune because the client never copies data out),
the server blocks in send() and stops draining the forward direction.
The client then blocks in its own send().  Neither side sets
SO_SNDTIMEO/SO_RCVTIMEO, uses non-blocking I/O, or has a second reader,
so is there anything that breaks the cycle?

The large_buf rekey variant drives this path with 2 MiB per iteration:

    KsftNamedVariant("large_buf", 10, 2097152),

With initial tcp_wmem[1]/tcp_rmem[1] defaults only a few hundred KiB can
be in flight per direction, well below 2 MiB.  The client cmd() then
times out (cmd_timeout is 10 for non-burst runs) and the variant is
reported as a TLS offload counter failure rather than as a hang.  Would
a bounded per-iteration payload in echo mode, or a socket timeout, be
preferable here?

[Severity: Low]
Should this send() loop over the remainder instead of treating a short
count as fatal?  Every other transfer in this file loops:
client_recv_echo() uses "while (total < len)" and server_echo_send()
uses "for (sent = 0; sent < n; sent += ret)".

A blocking SOCK_STREAM send() can return a short count without setting
errno, so "FAIL: send failed: %s" would print a stale strerror(errno)
value.  In burst mode a short send also breaks the framing contract that
the server's MSG_WAITALL recv() relies on, since server_verify_burst()
expects exactly send_size bytes per iteration.

[ ... ]

> diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
> new file mode 100755
> index 000000000000..b8f5a3314030
> --- /dev/null
> +++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
> @@ -0,0 +1,295 @@

[ ... ]

> +# Burst variants push hundreds of MB and perform many rekeys; the
> +# default cmd() timeout (5s) is too short.
> +BURST_TIMEOUT_S = 180

[Severity: Low]
Is the "(5s)" in this comment correct?  tools/testing/selftests/net/lib/py/utils.py
declares the default as 20 s in both places:

    def __init__(self, comm, shell=None, fail=True, expect_fail=False, ns=None,
                 background=False, host=None, timeout=20, ksft_ready=None,
                 ksft_wait=None):
...
    def process(self, terminate=True, fail=None, expect_fail=False, timeout=20):

Following the 5 s premise, run_tls_test() sets "cmd_timeout =
BURST_TIMEOUT_S if burst else 10", which lowers the non-burst client
timeout below the library default.

Also, BURST_TIMEOUT_S is only passed to the client cmd().  bkg takes no
timeout argument and bkg.__exit__() calls self.process() without one, so
the background server that handles the same "hundreds of MB" is drained
with communicate(timeout=20).  If it needs longer after the client exits,
subprocess.TimeoutExpired escapes bkg.__exit__() rather than producing a
test failure.

> +def check_tls_support(cfg):
> +    try:
> +        cmd("test -f /proc/net/tls_stat")
> +        cmd("test -f /proc/net/tls_stat", host=cfg.remote)
> +    except CmdExitFailure as e:
> +        raise KsftSkipEx(f"kTLS not supported: {e}")

[Severity: Medium]
Does this check actually test for kTLS support, or only for the tls
module already being loaded?

/proc/net/tls_stat is created from the module's pernet registration path:

net/tls/tls_proc.c:tls_proc_init() {
	if (!proc_create_net_single("tls_stat", 0444, net->proc_net,
	...
}

reached via module_init(tls_register) -> register_pernet_subsys(&tls_proc_ops).
With the common CONFIG_TLS=m the module is autoloaded on the first
TCP_ULP="tls" setsockopt, so on a freshly booted DUT or peer the file
does not exist yet and check_tls_support() raises KsftSkipEx("kTLS not
supported") from main() before ksft_run(), skipping all variants on a
machine that fully supports offload.

Would a modprobe, or probing an actual TLS ULP socket first, be more
reliable?

> +def read_tls_stats(host=None):
> +    stats = defaultdict(int)
> +    output = cmd("cat /proc/net/tls_stat", host=host)

[Severity: Low]
These are per-netns counters (TLS_INC_STATS(sock_net(sk), ...) exposed
through net->mib.tls_statistics), and these tests run in the host
namespace on a real NIC via NetDrvEpEnv(__file__, nsim_test=False).  Any
other kTLS user in the same namespace during the before/after window
perturbs the deltas that check_eq_sum() and check_zero() require to
match exactly.

There is also a harness-internal path for this.  When
cmd(client_cmd, timeout=cmd_timeout) times out, utils.py does:

        if terminate:
            self.proc.terminate()
        stdout, stderr = self.proc.communicate(timeout=timeout)

For a foreground cmd() the child is not killed on TimeoutExpired, so the
orphaned client keeps running and its eventual socket teardown can bump
TlsTxRekeyAborted or TlsDecryptError inside a later variant's
measurement window.  Can one timeout (for example the large_buf hang
above) cascade into counter failures in unrelated variants?

> +def check_path(before, after, direction, role, require_hw):
> +    """On the DUT, require HW offload; on the remote, HW or SW is fine."""
> +    dev = stat_diff(before, after, f'Tls{direction}Device')
> +    sw = stat_diff(before, after, f'Tls{direction}Sw')
> +    if require_hw:
> +        if dev < 1:
> +            ksft_pr(f"FAIL: {role} {direction}: HW offload not engaged "
> +                    f"(Device={dev}, Sw={sw})")
> +            return 1

[Severity: Low]
Does a TlsTxDevice/TlsRxDevice delta of at least 1 show that the device
actually performed record crypto?

Those MIBs are bumped once per socket at setsockopt() time only:

net/tls/tls_main.c:do_tls_setsockopt_conf() {
		if (!rc) {
			if (!update) {
				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
				TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
			}
	...
}

so they record that tls_dev_add() accepted the connection, not that any
record was encrypted or decrypted in hardware.  A socket whose RX
offload later degrades to software still satisfies dev >= 1, since
TLS_RX_DEV_DEGRADED is set at runtime:

net/tls/tls_device.c:tls_device_down() {
		/* Start skipping the RX resync logic completely. */
		set_bit(TLS_RX_DEV_DEGRADED, &ctx->flags);
	...
}

For the non-rekey variants (expected_rekeys == 0) check_path() is the
only assertion that runs, so is the central premise of a test placed
under drivers/net/hw verified at all there?

[ ... ]

> +    if expected_rekeys > 0:
> +        if with_tx:

[ ... ]

> +            errors += check_eq_sum(stats_before, stats_after,
> +                                   ['TlsTxRekeyOk', 'TlsTxRekeyAborted'],
> +                                   expected_rekeys, role)
> +            errors += check_zero(stats_before, stats_after,
> +                                 'TlsTxRekeyError', role)
> +            errors += check_zero(stats_before, stats_after,
> +                                 'TlsTxRekeyFallback', role)

[Severity: Medium]
Should TlsTxRekeyFallback and TlsRxRekeyFallback be required to stay at
zero?  Those counters are the kernel's accounting for intentional,
graceful degradation to software when a device key (re)install fails,
and the fallback path returns success:

net/tls/tls_device.c:tls_device_complete_rekey() {
rekey_fail:
	...
	TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYFALLBACK);
	TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
	TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);

	return 0;
}

The RX side does the same on a failed rekey dev_add:

	} else if (is_rekey) {
		set_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
		set_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYFALLBACK);

A NIC has a bounded pool of TLS contexts and a rekey needs a fresh one
while the old is retired, and the burst variants rekey very aggressively
on a single connection (burst_tx_rekey_every_1 does 50 rekeys, one per
64 KiB send; burst_rx_rekey_every_10 does 20 RX rekeys).  If a device
falls back, check_zero('TlsTxRekeyFallback') fails and the exact
check_eq_sum(Ok + Aborted == N) fails at the same time, giving two
failures with nothing to distinguish "device out of contexts" from a
kernel rekey bug.  A TX fallback also leaves the socket in software for
good, so one transient failure affects every remaining rekey on that
connection.

> +            errors += check_zero(stats_before, stats_after,
> +                                 'TlsTxRekeyInProgress', role)
> +        if with_rx:

[ ... ]

> +            errors += check_eq_sum(stats_before, stats_after,
> +                                   ['TlsRxRekeyOk', 'TlsRxRekeyAborted'],
> +                                   expected_rekeys, role)
> +            errors += check_min(stats_before, stats_after,
> +                                'TlsRxRekeyReceived', expected_rekeys, role)

[Severity: Low]
These strict rekey assertions also run for the remote peer
(is_dut=False); only the hardware-vs-software path check is relaxed
there.  Rekey support and the Tls*Rekey* MIB names are recent additions,
and read_tls_stats() builds a defaultdict(int), so a peer kernel that
does not export them yields 0 and produces a hard failure such as
"expected == 1, got 0" instead of a skip.  The only peer-side gate is
the /proc/net/tls_stat existence check.

This also does not match the run_tls_test() docstring, which says the
remote "may run any kernel without HW offload".  Should the peer-side
rekey checks be gated on the counters being present?

[ ... ]