* Re: [PATCH net-next] selftests/net: packetdrill: add tcp_syncookies_ip6_9k
From: Neal Cardwell @ 2026-04-29 20:47 UTC (permalink / raw)
To: Eric Dumazet
Cc: Jakub Kicinski, David S . Miller, Paolo Abeni, Simon Horman,
Kuniyuki Iwashima, netdev, eric.dumazet
In-Reply-To: <CANn89iJrvSvw2XKvB7LfXc2egeaqC+seiCiKN2USMw9gicqM1w@mail.gmail.com>
On Wed, Apr 29, 2026 at 1:15 PM Eric Dumazet <edumazet@google.com> wrote:
>
> On Wed, Apr 29, 2026 at 10:09 AM Eric Dumazet <edumazet@google.com> wrote:
> >
> > On Wed, Apr 29, 2026 at 9:48 AM Jakub Kicinski <kuba@kernel.org> wrote:
> > >
> > > On Wed, 29 Apr 2026 08:00:36 +0000 Eric Dumazet wrote:
> > > > This test checks syncookie mode is able to reconstruct some
> > > > client options when TCP TS are used:
> > > >
> > > > - wscale option.
> > > > - sackOK.
> > > > - MSS (in a limited way).
> > > > - ECN (not tested, because of limited value).
> > >
> > > We're getting:
> > >
> > > # TAP version 13
> > > # 1..1
> > > # File "/tmp/code_oyWax9", line 223
> > > # assert tcpi_snd_wscale = 10, tcpi_snd_wscale
> > > # ^
> > > # SyntaxError: invalid syntax
> > > # tcp_syncookies_ip6_9k.pkt: error executing code: 'python3' returned non-zero status 1
> > >
> > > do we need to update packetdrill ?
> >
> > No, this should have been
> >
> > assert tcpi_snd_wscale == 10, tcpi_snd_wscale
> >
> > I initially had a value of 8, and decided going to 10 just to make it
> > different than the output wscale value.
> >
> > Sorry for the confusion.
>
> Neal, can you take care of adding TCPI_OPT_SACK to packetdrill ?
>
> Thanks!
>
> diff --git a/gtests/net/packetdrill/code.c b/gtests/net/packetdrill/code.c
> index d90bffabcfe6..6d8d20e64d8c 100644
> --- a/gtests/net/packetdrill/code.c
> +++ b/gtests/net/packetdrill/code.c
> @@ -129,6 +129,7 @@ static void write_symbols(struct code_state *code)
> emit_var(code, "TCPI_OPT_WSCALE", TCPI_OPT_WSCALE);
> emit_var(code, "TCPI_OPT_ECN", TCPI_OPT_ECN);
> emit_var(code, "TCPI_OPT_SYN_DATA", TCPI_OPT_SYN_DATA);
> + emit_var(code, "TCPI_OPT_SACK", TCPI_OPT_SACK);
> #endif /* linux */
> }
Sure! Done. I added all the missing bit values in the following commit:
https://github.com/google/packetdrill/commit/faa0dfb54065118625e169d3111ce09c65b20229
This can be picked up with the following, or similar:
git clone https://github.com/google/packetdrill.git
neal
^ permalink raw reply
* [PATCH iwl-next v3] libie: log more info when virtchnl fails
From: Li Li @ 2026-04-29 20:41 UTC (permalink / raw)
To: Tony Nguyen, Przemek Kitszel, David S. Miller, Jakub Kicinski,
Eric Dumazet, intel-wired-lan
Cc: netdev, linux-kernel, David Decotigny, Anjali Singhai,
Sridhar Samudrala, Brian Vazquez, Li Li, emil.s.tantilov
Virtchnl failures can be hard to debug without logs. Logging the details
of virtchnl transactions can be useful for debugging virtchnl-related
issues.
Tested: Built & booted on a test machine and synthetically produced a
virtual failure to produce the following log:
idpf 0000:01:00.0: Non-zero virtchnl ret val (msg op: 1, ret val: 6,
data_len: 8); xn id: 0, cookie: 0
idpf 0000:01:00.0: Transaction failed (op 1, xn state:
3, id: 0, cookie: 0, size: 8)
Signed-off-by: Li Li <boolli@google.com>
---
v3:
- Use dev_err_ratelimited in both logs.
- Move log placement to after virtchnl field validation.
- Remove redundant op/cookie fields since they were validated.
v2:
- Use dev_warn_ratelimited instead of dev_notice_ratelimited based on
reviewer feedback.
drivers/net/ethernet/intel/libie/controlq.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/drivers/net/ethernet/intel/libie/controlq.c b/drivers/net/ethernet/intel/libie/controlq.c
index ebc05355e39d..ceca8a076d79 100644
--- a/drivers/net/ethernet/intel/libie/controlq.c
+++ b/drivers/net/ethernet/intel/libie/controlq.c
@@ -766,6 +766,14 @@ libie_ctlq_xn_process_recv(struct libie_ctlq_xn_recv_params *params,
msg_cookie != xn->cookie)
return false;
+ if (ctlq_msg->chnl_retval) {
+ dev_err_ratelimited(
+ params->ctlq->dev,
+ "Non-zero virtchnl ret val (msg op: %u, ret val: %u, data_len: %u); xn id: %u, cookie: %u\n",
+ ctlq_msg->chnl_opcode, ctlq_msg->chnl_retval,
+ ctlq_msg->data_len, xn->index, xn->cookie);
+ }
+
spin_lock(&xn->xn_lock);
if (xn->state != LIBIE_CTLQ_XN_ASYNC &&
xn->state != LIBIE_CTLQ_XN_WAITING) {
@@ -1011,6 +1019,11 @@ int libie_ctlq_xn_send(struct libie_ctlq_xn_send_params *params)
params->recv_mem = xn->recv_mem;
break;
default:
+ dev_err_ratelimited(
+ params->ctlq->dev,
+ "Transaction failed (op %u, xn state: %d, id: %u, cookie: %u, size: %zu)\n",
+ params->chnl_opcode, xn->state, xn->index, xn->cookie,
+ xn->recv_mem.iov_len);
ret = -EBADMSG;
break;
}
--
2.54.0.545.g6539524ca2-goog
^ permalink raw reply related
* [syzbot] Monthly wpan report (Apr 2026)
From: syzbot @ 2026-04-29 20:32 UTC (permalink / raw)
To: alex.aring, linux-kernel, linux-wpan, miquel.raynal, netdev,
stefan, syzkaller-bugs
Hello wpan maintainers/developers,
This is a 31-day syzbot report for the wpan subsystem.
All related reports/information can be found at:
https://syzkaller.appspot.com/upstream/s/wpan
During the period, 0 new issues were detected and 0 were fixed.
In total, 5 issues are still open and 26 have already been fixed.
Some of the still happening issues:
Ref Crashes Repro Title
<1> 460 Yes KMSAN: uninit-value in ieee802154_hdr_push (2)
https://syzkaller.appspot.com/bug?extid=60a66d44892b66b56545
<2> 366 Yes WARNING in lowpan_xmit (2)
https://syzkaller.appspot.com/bug?extid=5b74e0e96f12e3728ec8
<3> 55 Yes WARNING in __dev_change_net_namespace (3)
https://syzkaller.appspot.com/bug?extid=3344d668bbbc12996d46
---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
To disable reminders for individual bugs, reply with the following command:
#syz set <Ref> no-reminders
To change bug's subsystems, reply with:
#syz set <Ref> subsystems: new-subsystem
You may send multiple commands in a single email message.
^ permalink raw reply
* [syzbot] Monthly netfilter report (Apr 2026)
From: syzbot @ 2026-04-29 20:32 UTC (permalink / raw)
To: fw, linux-kernel, netdev, netfilter-devel, pablo, syzkaller-bugs
Hello netfilter maintainers/developers,
This is a 31-day syzbot report for the netfilter subsystem.
All related reports/information can be found at:
https://syzkaller.appspot.com/upstream/s/netfilter
During the period, 0 new issues were detected and 0 were fixed.
In total, 14 issues are still open and 196 have already been fixed.
Some of the still happening issues:
Ref Crashes Repro Title
<1> 5008 Yes INFO: rcu detected stall in worker_thread (9)
https://syzkaller.appspot.com/bug?extid=225bfad78b079744fd5e
<2> 333 Yes INFO: rcu detected stall in gc_worker (3)
https://syzkaller.appspot.com/bug?extid=eec403943a2a2455adaa
<3> 127 Yes INFO: rcu detected stall in NF_HOOK (2)
https://syzkaller.appspot.com/bug?extid=34c2df040c6cfa15fdfe
<4> 73 No INFO: task hung in nfnetlink_rcv_msg (5)
https://syzkaller.appspot.com/bug?extid=c4b20b80ee6a7a2f5012
<5> 11 Yes KMSAN: uninit-value in ip6table_mangle_hook (3)
https://syzkaller.appspot.com/bug?extid=6023ea32e206eef7920a
---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
To disable reminders for individual bugs, reply with the following command:
#syz set <Ref> no-reminders
To change bug's subsystems, reply with:
#syz set <Ref> subsystems: new-subsystem
You may send multiple commands in a single email message.
^ permalink raw reply
* [syzbot] Monthly hams report (Apr 2026)
From: syzbot @ 2026-04-29 20:32 UTC (permalink / raw)
To: linux-hams, linux-kernel, netdev, syzkaller-bugs
Hello hams maintainers/developers,
This is a 31-day syzbot report for the hams subsystem.
All related reports/information can be found at:
https://syzkaller.appspot.com/upstream/s/hams
During the period, 3 new issues were detected and 1 were fixed.
In total, 20 issues are still open and 48 have already been fixed.
Some of the still happening issues:
Ref Crashes Repro Title
<1> 16483 Yes possible deadlock in nr_rt_device_down (3)
https://syzkaller.appspot.com/bug?extid=ccdfb85a561b973219c7
<2> 8461 Yes possible deadlock in nr_rt_ioctl (2)
https://syzkaller.appspot.com/bug?extid=14afda08dc3484d5db82
<3> 4371 Yes WARNING: ODEBUG bug in handle_softirqs
https://syzkaller.appspot.com/bug?extid=60db000b8468baeddbb1
<4> 1122 Yes possible deadlock in nr_remove_neigh (2)
https://syzkaller.appspot.com/bug?extid=8863ad36d31449b4dc17
<5> 803 Yes WARNING: ODEBUG bug in __run_timers (3)
https://syzkaller.appspot.com/bug?extid=7287222a6d88bdb559a7
<6> 188 Yes possible deadlock in nr_del_node (2)
https://syzkaller.appspot.com/bug?extid=6eb7834837cf6a8db75b
<7> 32 Yes WARNING: refcount bug in ax25_setsockopt
https://syzkaller.appspot.com/bug?extid=0ee4da32f91ae2a3f015
<8> 26 No KASAN: slab-use-after-free Write in rose_send_frame
https://syzkaller.appspot.com/bug?extid=9cc08d5dcb1fc8feee76
<9> 24 Yes KASAN: slab-use-after-free Read in nr_rt_ioctl
https://syzkaller.appspot.com/bug?extid=df52f4216bf7b4d768e7
<10> 23 No KASAN: slab-use-after-free Read in rose_t0timer_expiry
https://syzkaller.appspot.com/bug?extid=44cccec2c6a4501f6a45
---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
To disable reminders for individual bugs, reply with the following command:
#syz set <Ref> no-reminders
To change bug's subsystems, reply with:
#syz set <Ref> subsystems: new-subsystem
You may send multiple commands in a single email message.
^ permalink raw reply
* Re: [PATCH mlx5-next 0/4] mlx5-next updates 2026-04-28
From: Leon Romanovsky @ 2026-04-29 20:28 UTC (permalink / raw)
To: Jason Gunthorpe, Saeed Mahameed, Tariq Toukan
Cc: Eric Dumazet, Jakub Kicinski, Paolo Abeni, Andrew Lunn,
David S. Miller, Mark Bloch, Moshe Shemesh, Parav Pandit,
Shay Drori, Kees Cook, Daniel Jurgens, Or Har-Toov, Simon Horman,
Jiri Pirko, Adithya Jayachandran, linux-rdma, linux-kernel,
netdev, Gal Pressman, Dragos Tatulea
In-Reply-To: <20260428053851.220089-1-tariqt@nvidia.com>
On Tue, 28 Apr 2026 08:38:47 +0300, Tariq Toukan wrote:
> This series by Moshe contains mlx5 shared updates as preparation for
> upcoming features.
>
> Regards,
> Tariq
>
> Moshe Shemesh (4):
> mlx5: Rename the vport number enums for host PF and VF
> net/mlx5: Add function_id_type for enable/disable_hca cmds
> net/mlx5: Remove unused host_sf_enable field
> net/mlx5: Extend query_esw_functions output for multi-function support
>
> [...]
Applied, thanks!
[1/4] mlx5: Rename the vport number enums for host PF and VF
https://git.kernel.org/rdma/rdma/c/5ffd937ea4050e
[2/4] net/mlx5: Add function_id_type for enable/disable_hca cmds
https://git.kernel.org/rdma/rdma/c/a750f4674a6338
[3/4] net/mlx5: Remove unused host_sf_enable field
https://git.kernel.org/rdma/rdma/c/e2337517e127b7
[4/4] net/mlx5: Extend query_esw_functions output for multi-function support
https://git.kernel.org/rdma/rdma/c/02c54621e81ccd
Best regards,
--
Leon Romanovsky <leon@kernel.org>
^ permalink raw reply
* Re: [PATCH 0/9] Support for block device NVMEM providers
From: Loic Poulain @ 2026-04-29 20:20 UTC (permalink / raw)
To: Andrew Lunn
Cc: Ulf Hansson, Rob Herring, Krzysztof Kozlowski, Conor Dooley,
Bjorn Andersson, Konrad Dybcio, Jens Axboe, Johannes Berg,
Jeff Johnson, Bartosz Golaszewski, Marcel Holtmann,
Luiz Augusto von Dentz, Balakrishna Godavarthi, Rocky Liao,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Simon Horman, linux-mmc, devicetree, linux-kernel, linux-arm-msm,
linux-block, linux-wireless, ath10k, linux-bluetooth, netdev,
daniel
In-Reply-To: <b4de6c89-dd89-4e02-8d79-911e4f9f0813@lunn.ch>
Hi Andrew,
On Wed, Apr 29, 2026 at 3:06 AM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > Note that this is currently limited to eMMC-backed block devices, as
> > only the eMMC core associates a firmware node with the block device
> > (add_disk_fwnode). This can be easily extended in the future to
> > support additional block drivers.
>
> Would this be
>
> https://elixir.bootlin.com/linux/v7.0.1/source/drivers/mmc/core/block.c#L2641
>
> Looking at that function, mmc_blk_alloc_req() i don't see it doing
> anything different between an eMMC and MMC.
Yes, that’s correct, in the previous sentence I should have referred
to MMC-backed storage rather than specifically eMMC, since the MMC
scope also includes removable MMC/SD cards. That said, eMMC is more
likely to be used as an NVMEM area.
>
> An eMMC you don't expect to go away, since it is soldered
> down. However an MMC can be ejected. Is the code prepared for that?
I was about to say this wouldn’t be an issue, since the NVMEM device
would be unregistered along with the MMC card removal. However,
looking at nvmem_unregister(), it simply performs a kref_put(). This
means that if a consumer still holds a reference, e.g. through an
nvmem_cell, it might still be able to access the NVMEM reg_read
callback after the NVMEM device has been unregistered and the
NVMEM provider data (bnv) has been freed...
I guess I need to try and dig into this but at first glance it seems to be a
general NVMEM issue, since even though most NVMEM providers are
not hard-removable, there is nothing that fundamentally prevents them
from being logically removed/unbound at runtime.
Regards,
Loic
^ permalink raw reply
* [PATCH net-next V2 2/3] net/mlx5: add frag buf pools create/destroy paths
From: Tariq Toukan @ 2026-04-29 20:14 UTC (permalink / raw)
To: Eric Dumazet, Jakub Kicinski, Paolo Abeni, Andrew Lunn,
David S. Miller
Cc: Saeed Mahameed, Leon Romanovsky, Tariq Toukan, Mark Bloch, netdev,
linux-rdma, linux-kernel, Gal Pressman, Dragos Tatulea,
Moshe Shemesh, Nimrod Oren
In-Reply-To: <20260429201429.223809-1-tariqt@nvidia.com>
From: Nimrod Oren <noren@nvidia.com>
Introduce mlx5 DMA pool and pool-page data structures, and add the
creation and teardown paths.
Each NUMA node owns a set of mlx5_dma_pool instances, each one with a
different block size. The sizes are defined as all powers of two
starting from MLX5_ADAPTER_PAGE_SHIFT and up to PAGE_SHIFT. Since
mlx5_frag_bufs are used to back objects whose sizes are encoded relative
to MLX5_ADAPTER_PAGE_SHIFT, a smaller block_shift value cannot be used.
Requests larger than PAGE_SIZE continue to be handled as page-sized
fragments, as in the existing frag-buf allocation model.
Signed-off-by: Nimrod Oren <noren@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/alloc.c | 116 +++++++++++++++++-
include/linux/mlx5/driver.h | 7 +-
2 files changed, 119 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c
index cebb3559d2c9..fcc859c5f810 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c
@@ -37,10 +37,15 @@
#include <linux/bitmap.h>
#include <linux/dma-mapping.h>
#include <linux/vmalloc.h>
+#include <linux/nodemask.h>
#include <linux/mlx5/driver.h>
#include "mlx5_core.h"
+#define MLX5_FRAG_BUF_POOL_MIN_BLOCK_SHIFT MLX5_ADAPTER_PAGE_SHIFT
+#define MLX5_FRAG_BUF_POOLS_NUM \
+ (PAGE_SHIFT - MLX5_FRAG_BUF_POOL_MIN_BLOCK_SHIFT + 1)
+
struct mlx5_db_pgdir {
struct list_head list;
unsigned long *bitmap;
@@ -48,6 +53,27 @@ struct mlx5_db_pgdir {
dma_addr_t db_dma;
};
+struct mlx5_dma_pool {
+ /* Protects page_list and per-page allocation bitmaps. */
+ struct mutex lock;
+ struct list_head page_list;
+ struct mlx5_core_dev *dev;
+ int node;
+ u8 block_shift;
+};
+
+struct mlx5_dma_pool_page {
+ struct mlx5_dma_pool *pool;
+ struct list_head pool_link;
+ unsigned long *bitmap;
+ void *buf;
+ dma_addr_t dma;
+};
+
+struct mlx5_frag_buf_node_pools {
+ struct mlx5_dma_pool *pools[MLX5_FRAG_BUF_POOLS_NUM];
+};
+
/* Handling for queue buffers -- we allocate a bunch of memory and
* register it in a memory region at HCA virtual address 0.
*/
@@ -71,14 +97,100 @@ static void *mlx5_dma_zalloc_coherent_node(struct mlx5_core_dev *dev,
return cpu_handle;
}
-/* Implemented later in the series */
+static void mlx5_dma_pool_destroy(struct mlx5_dma_pool *pool)
+{
+ mutex_destroy(&pool->lock);
+ kfree(pool);
+}
+
+static struct mlx5_dma_pool *mlx5_dma_pool_create(struct mlx5_core_dev *dev,
+ int node, u8 block_shift)
+{
+ struct mlx5_dma_pool *pool;
+
+ pool = kzalloc_obj(*pool);
+ if (!pool)
+ return NULL;
+
+ INIT_LIST_HEAD(&pool->page_list);
+ mutex_init(&pool->lock);
+ pool->dev = dev;
+ pool->node = node;
+ pool->block_shift = block_shift;
+ return pool;
+}
+
+static void
+mlx5_frag_buf_node_pools_destroy(struct mlx5_frag_buf_node_pools *node_pools)
+{
+ for (int i = 0; i < MLX5_FRAG_BUF_POOLS_NUM; i++)
+ if (node_pools->pools[i])
+ mlx5_dma_pool_destroy(node_pools->pools[i]);
+ kfree(node_pools);
+}
+
+static struct mlx5_frag_buf_node_pools *
+mlx5_frag_buf_node_pools_create(struct mlx5_core_dev *dev, int node)
+{
+ struct mlx5_frag_buf_node_pools *node_pools;
+
+ node_pools = kzalloc_obj(*node_pools);
+ if (!node_pools)
+ return NULL;
+
+ for (int i = 0; i < MLX5_FRAG_BUF_POOLS_NUM; i++) {
+ u8 block_shift = MLX5_FRAG_BUF_POOL_MIN_BLOCK_SHIFT + i;
+
+ node_pools->pools[i] = mlx5_dma_pool_create(dev, node,
+ block_shift);
+ if (!node_pools->pools[i]) {
+ mlx5_frag_buf_node_pools_destroy(node_pools);
+ return NULL;
+ }
+ }
+
+ return node_pools;
+}
+
void mlx5_frag_buf_pools_cleanup(struct mlx5_core_dev *dev)
{
+ struct mlx5_priv *priv = &dev->priv;
+ int node;
+
+ for_each_node_state(node, N_POSSIBLE) {
+ struct mlx5_frag_buf_node_pools *node_pools;
+
+ node_pools = priv->frag_buf_node_pools[node];
+ if (!node_pools)
+ continue;
+ mlx5_frag_buf_node_pools_destroy(node_pools);
+ }
+
+ kfree(priv->frag_buf_node_pools);
+ priv->frag_buf_node_pools = NULL;
}
-/* Implemented later in the series */
int mlx5_frag_buf_pools_init(struct mlx5_core_dev *dev)
{
+ struct mlx5_priv *priv = &dev->priv;
+ int node;
+
+ priv->frag_buf_node_pools = kzalloc_objs(*priv->frag_buf_node_pools,
+ nr_node_ids);
+ if (!priv->frag_buf_node_pools)
+ return -ENOMEM;
+
+ for_each_node_state(node, N_POSSIBLE) {
+ struct mlx5_frag_buf_node_pools *node_pools;
+
+ node_pools = mlx5_frag_buf_node_pools_create(dev, node);
+ if (!node_pools) {
+ mlx5_frag_buf_pools_cleanup(dev);
+ return -ENOMEM;
+ }
+ priv->frag_buf_node_pools[node] = node_pools;
+ }
+
return 0;
}
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 04b96c5abb57..71f7615ab553 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -558,6 +558,7 @@ enum mlx5_func_type {
MLX5_FUNC_TYPE_NUM,
};
+struct mlx5_frag_buf_node_pools;
struct mlx5_ft_pool;
struct mlx5_priv {
/* IRQ table valid only for real pci devices PF or VF */
@@ -581,14 +582,16 @@ struct mlx5_priv {
struct mlx5_debugfs_entries dbg;
- /* start: alloc staff */
+ /* start: alloc stuff */
/* protect buffer allocation according to numa node */
struct mutex alloc_mutex;
int numa_node;
struct mutex pgdir_mutex;
struct list_head pgdir_list;
- /* end: alloc staff */
+
+ struct mlx5_frag_buf_node_pools **frag_buf_node_pools;
+ /* end: alloc stuff */
struct mlx5_adev **adev;
int adev_idx;
--
2.44.0
^ permalink raw reply related
* [PATCH net-next V2 3/3] net/mlx5: use internal dma pools for frag buf alloc
From: Tariq Toukan @ 2026-04-29 20:14 UTC (permalink / raw)
To: Eric Dumazet, Jakub Kicinski, Paolo Abeni, Andrew Lunn,
David S. Miller
Cc: Saeed Mahameed, Leon Romanovsky, Tariq Toukan, Mark Bloch, netdev,
linux-rdma, linux-kernel, Gal Pressman, Dragos Tatulea,
Moshe Shemesh, Nimrod Oren
In-Reply-To: <20260429201429.223809-1-tariqt@nvidia.com>
From: Nimrod Oren <noren@nvidia.com>
Add mlx5_dma_pool alloc/free paths, and wire mlx5_frag_buf allocation
and free paths to use them.
mlx5_frag_buf_alloc_node() now selects an mlx5_dma_pool to allocate
fragments from, instead of directly allocating full coherent pages.
mlx5_frag_buf_free() frees from the respective pool.
mlx5_dma_pool_alloc() keeps allocation fast by maintaining pages with
available indexes at the head of the list, so the common allocation path
can take a free index immediately. New backing pages are allocated only
when no free index is available.
mlx5_dma_pool_free() returns released indexes to the pool and frees a
backing page once all of its indexes become free. This avoids keeping
fully free pages for the lifetime of the pool and reduces coherent DMA
memory footprint.
Signed-off-by: Nimrod Oren <noren@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/alloc.c | 170 ++++++++++++++----
include/linux/mlx5/driver.h | 2 +
2 files changed, 140 insertions(+), 32 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c
index fcc859c5f810..f19644183828 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c
@@ -120,6 +120,111 @@ static struct mlx5_dma_pool *mlx5_dma_pool_create(struct mlx5_core_dev *dev,
return pool;
}
+static struct mlx5_dma_pool_page *
+mlx5_dma_pool_page_alloc(struct mlx5_dma_pool *pool)
+{
+ int blocks_per_page = BIT(PAGE_SHIFT - pool->block_shift);
+ struct mlx5_dma_pool_page *page;
+
+ page = kzalloc_obj(*page);
+ if (!page)
+ goto err_out;
+
+ page->pool = pool;
+ page->bitmap = bitmap_zalloc(blocks_per_page, GFP_KERNEL);
+ if (!page->bitmap)
+ goto err_free_page;
+
+ bitmap_fill(page->bitmap, blocks_per_page);
+ page->buf = mlx5_dma_zalloc_coherent_node(pool->dev, PAGE_SIZE,
+ &page->dma, pool->node);
+ if (!page->buf)
+ goto err_free_bitmap;
+
+ return page;
+
+err_free_bitmap:
+ bitmap_free(page->bitmap);
+err_free_page:
+ kfree(page);
+err_out:
+ return NULL;
+}
+
+static void mlx5_dma_pool_page_free(struct mlx5_core_dev *dev,
+ struct mlx5_dma_pool_page *page)
+{
+ dma_free_coherent(mlx5_core_dma_dev(dev), PAGE_SIZE, page->buf,
+ page->dma);
+ bitmap_free(page->bitmap);
+ kfree(page);
+}
+
+static int mlx5_dma_pool_alloc_from_page(struct mlx5_dma_pool *pool,
+ struct mlx5_dma_pool_page *page,
+ unsigned long *idx_out)
+{
+ int blocks_per_page = BIT(PAGE_SHIFT - pool->block_shift);
+
+ *idx_out = find_first_bit(page->bitmap, blocks_per_page);
+ if (*idx_out >= blocks_per_page)
+ return -ENOMEM;
+
+ __clear_bit(*idx_out, page->bitmap);
+
+ if (bitmap_empty(page->bitmap, blocks_per_page))
+ list_move_tail(&page->pool_link, &pool->page_list);
+
+ return 0;
+}
+
+static struct mlx5_dma_pool_page *
+mlx5_dma_pool_alloc(struct mlx5_dma_pool *pool, unsigned long *idx_out)
+{
+ struct mlx5_dma_pool_page *page;
+
+ mutex_lock(&pool->lock);
+
+ page = list_first_entry_or_null(&pool->page_list,
+ struct mlx5_dma_pool_page, pool_link);
+ if (page && !mlx5_dma_pool_alloc_from_page(pool, page, idx_out))
+ goto unlock; /* successfully allocated from existing page */
+
+ page = mlx5_dma_pool_page_alloc(pool);
+ if (!page)
+ goto unlock;
+
+ list_add(&page->pool_link, &pool->page_list);
+ mlx5_dma_pool_alloc_from_page(pool, page, idx_out);
+
+unlock:
+ mutex_unlock(&pool->lock);
+ return page;
+}
+
+static void mlx5_dma_pool_free(struct mlx5_dma_pool *pool,
+ struct mlx5_dma_pool_page *page,
+ unsigned long idx)
+{
+ int blocks_per_page = BIT(PAGE_SHIFT - pool->block_shift);
+ bool was_full;
+
+ mutex_lock(&pool->lock);
+ was_full = bitmap_empty(page->bitmap, blocks_per_page);
+ __set_bit(idx, page->bitmap);
+
+ if (bitmap_full(page->bitmap, blocks_per_page)) {
+ list_del(&page->pool_link);
+ mlx5_dma_pool_page_free(pool->dev, page);
+ } else {
+ memset((u8 *)page->buf + (idx << pool->block_shift), 0,
+ BIT(pool->block_shift));
+ if (was_full)
+ list_move(&page->pool_link, &pool->page_list);
+ }
+ mutex_unlock(&pool->lock);
+}
+
static void
mlx5_frag_buf_node_pools_destroy(struct mlx5_frag_buf_node_pools *node_pools)
{
@@ -197,56 +302,57 @@ int mlx5_frag_buf_pools_init(struct mlx5_core_dev *dev)
int mlx5_frag_buf_alloc_node(struct mlx5_core_dev *dev, int size,
struct mlx5_frag_buf *buf, int node)
{
- int i;
+ struct mlx5_dma_pool *pool;
+ int pool_idx;
+
+ node = node == NUMA_NO_NODE ? first_online_node : node;
buf->size = size;
buf->npages = DIV_ROUND_UP(size, PAGE_SIZE);
- buf->page_shift = PAGE_SHIFT;
- buf->frags = kzalloc_objs(struct mlx5_buf_list, buf->npages);
+ buf->page_shift = clamp_t(int, order_base_2(size),
+ MLX5_FRAG_BUF_POOL_MIN_BLOCK_SHIFT,
+ PAGE_SHIFT);
+ buf->frags = kcalloc_node(buf->npages, sizeof(*buf->frags),
+ GFP_KERNEL, node);
if (!buf->frags)
- goto err_out;
+ return -ENOMEM;
- for (i = 0; i < buf->npages; i++) {
+ pool_idx = buf->page_shift - MLX5_FRAG_BUF_POOL_MIN_BLOCK_SHIFT;
+ pool = dev->priv.frag_buf_node_pools[node]->pools[pool_idx];
+ for (int i = 0; i < buf->npages; i++) {
struct mlx5_buf_list *frag = &buf->frags[i];
- int frag_sz = min_t(int, size, PAGE_SIZE);
+ struct mlx5_dma_pool_page *page;
+ unsigned long idx;
- frag->buf = mlx5_dma_zalloc_coherent_node(dev, frag_sz,
- &frag->map, node);
- if (!frag->buf)
- goto err_free_buf;
- if (frag->map & ((1 << buf->page_shift) - 1)) {
- dma_free_coherent(mlx5_core_dma_dev(dev), frag_sz,
- buf->frags[i].buf, buf->frags[i].map);
- mlx5_core_warn(dev, "unexpected map alignment: %pad, page_shift=%d\n",
- &frag->map, buf->page_shift);
- goto err_free_buf;
+ page = mlx5_dma_pool_alloc(pool, &idx);
+ if (!page) {
+ mlx5_frag_buf_free(dev, buf);
+ return -ENOMEM;
}
- size -= frag_sz;
+ frag->buf = (u8 *)page->buf + (idx << pool->block_shift);
+ frag->map = page->dma + (idx << pool->block_shift);
+ frag->frag_page = page;
}
return 0;
-
-err_free_buf:
- while (i--)
- dma_free_coherent(mlx5_core_dma_dev(dev), PAGE_SIZE, buf->frags[i].buf,
- buf->frags[i].map);
- kfree(buf->frags);
-err_out:
- return -ENOMEM;
}
EXPORT_SYMBOL_GPL(mlx5_frag_buf_alloc_node);
void mlx5_frag_buf_free(struct mlx5_core_dev *dev, struct mlx5_frag_buf *buf)
{
- int size = buf->size;
- int i;
+ for (int i = 0; i < buf->npages; i++) {
+ struct mlx5_buf_list *frag = &buf->frags[i];
+ struct mlx5_dma_pool_page *page;
+ struct mlx5_dma_pool *pool;
+ unsigned long idx;
- for (i = 0; i < buf->npages; i++) {
- int frag_sz = min_t(int, size, PAGE_SIZE);
+ if (!frag->buf)
+ continue;
- dma_free_coherent(mlx5_core_dma_dev(dev), frag_sz, buf->frags[i].buf,
- buf->frags[i].map);
- size -= frag_sz;
+ page = frag->frag_page;
+ pool = page->pool;
+ idx = (frag->map - page->dma) >> pool->block_shift;
+ mlx5_dma_pool_free(pool, page, idx);
}
kfree(buf->frags);
}
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 71f7615ab553..531ce66fc8ef 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -343,9 +343,11 @@ struct mlx5_cmd_mailbox {
struct mlx5_cmd_mailbox *next;
};
+struct mlx5_dma_pool_page;
struct mlx5_buf_list {
void *buf;
dma_addr_t map;
+ struct mlx5_dma_pool_page *frag_page;
};
struct mlx5_frag_buf {
--
2.44.0
^ permalink raw reply related
* [PATCH net-next V2 1/3] net/mlx5: wire frag buf pools lifecycle hooks
From: Tariq Toukan @ 2026-04-29 20:14 UTC (permalink / raw)
To: Eric Dumazet, Jakub Kicinski, Paolo Abeni, Andrew Lunn,
David S. Miller
Cc: Saeed Mahameed, Leon Romanovsky, Tariq Toukan, Mark Bloch, netdev,
linux-rdma, linux-kernel, Gal Pressman, Dragos Tatulea,
Moshe Shemesh, Nimrod Oren
In-Reply-To: <20260429201429.223809-1-tariqt@nvidia.com>
From: Nimrod Oren <noren@nvidia.com>
Wire mlx5_frag_buf pools init/cleanup hooks into
mlx5_mdev_init()/uninit() and the init unwind path.
Keep temporary no-op stubs in alloc.c so lifecycle ordering is in place
before the coherent DMA sub-page allocator implementation is added in
follow-up patches.
Signed-off-by: Nimrod Oren <noren@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
drivers/net/ethernet/mellanox/mlx5/core/alloc.c | 11 +++++++++++
drivers/net/ethernet/mellanox/mlx5/core/main.c | 7 +++++++
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h | 2 ++
3 files changed, 20 insertions(+)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c
index 202feab1558a..cebb3559d2c9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c
@@ -71,6 +71,17 @@ static void *mlx5_dma_zalloc_coherent_node(struct mlx5_core_dev *dev,
return cpu_handle;
}
+/* Implemented later in the series */
+void mlx5_frag_buf_pools_cleanup(struct mlx5_core_dev *dev)
+{
+}
+
+/* Implemented later in the series */
+int mlx5_frag_buf_pools_init(struct mlx5_core_dev *dev)
+{
+ return 0;
+}
+
int mlx5_frag_buf_alloc_node(struct mlx5_core_dev *dev, int size,
struct mlx5_frag_buf *buf, int node)
{
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 74827e8ca125..b1b9ebfd3866 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -1817,6 +1817,10 @@ int mlx5_mdev_init(struct mlx5_core_dev *dev, int profile_idx)
priv->dbg.dbg_root = debugfs_create_dir(dev_name(dev->device),
mlx5_debugfs_root);
+ err = mlx5_frag_buf_pools_init(dev);
+ if (err)
+ goto err_frag_buf_pools_init;
+
INIT_LIST_HEAD(&priv->traps);
err = mlx5_cmd_init(dev);
@@ -1878,6 +1882,8 @@ int mlx5_mdev_init(struct mlx5_core_dev *dev, int profile_idx)
err_timeout_init:
mlx5_cmd_cleanup(dev);
err_cmd_init:
+ mlx5_frag_buf_pools_cleanup(dev);
+err_frag_buf_pools_init:
debugfs_remove(dev->priv.dbg.dbg_root);
mutex_destroy(&priv->pgdir_mutex);
mutex_destroy(&priv->alloc_mutex);
@@ -1902,6 +1908,7 @@ void mlx5_mdev_uninit(struct mlx5_core_dev *dev)
mlx5_health_cleanup(dev);
mlx5_tout_cleanup(dev);
mlx5_cmd_cleanup(dev);
+ mlx5_frag_buf_pools_cleanup(dev);
debugfs_remove_recursive(dev->priv.dbg.dbg_root);
mutex_destroy(&priv->pgdir_mutex);
mutex_destroy(&priv->alloc_mutex);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
index 1507e881d962..87f01c4e8d65 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
@@ -436,6 +436,8 @@ mlx5_sf_coredev_to_adev(struct mlx5_core_dev *mdev)
int mlx5_mdev_init(struct mlx5_core_dev *dev, int profile_idx);
void mlx5_mdev_uninit(struct mlx5_core_dev *dev);
+int mlx5_frag_buf_pools_init(struct mlx5_core_dev *dev);
+void mlx5_frag_buf_pools_cleanup(struct mlx5_core_dev *dev);
int mlx5_init_one(struct mlx5_core_dev *dev);
int mlx5_init_one_devl_locked(struct mlx5_core_dev *dev);
void mlx5_uninit_one(struct mlx5_core_dev *dev);
--
2.44.0
^ permalink raw reply related
* [PATCH net-next V2 0/3] net/mlx5: enable sub-page allocations for mlx5_frag_buf
From: Tariq Toukan @ 2026-04-29 20:14 UTC (permalink / raw)
To: Eric Dumazet, Jakub Kicinski, Paolo Abeni, Andrew Lunn,
David S. Miller
Cc: Saeed Mahameed, Leon Romanovsky, Tariq Toukan, Mark Bloch, netdev,
linux-rdma, linux-kernel, Gal Pressman, Dragos Tatulea,
Moshe Shemesh
Hi,
See detailed description by Nimrod below [1].
Regards,
Tariq
[1]
This series aims to improve memory utilization for DMA-coherent
fragmented-buffer allocations on systems with large PAGE_SIZE.
Before this change, such allocations were page-granular, as they were
backed by full pages. On large-page systems this caused significant
internal waste for small objects. For example, a single 4K request
consumed an entire 64K page.
The common kernel solution for sub-page coherent DMA allocations is the
DMA pool API. However, those pools do not return pages to the system
until teardown. That behavior is not a good fit for mlx5_frag_buf
allocations, since they back interface resources (WQs and CQs).
Interfaces may be removed dynamically, so their memory footprint should
reflect live usage to avoid situations where large amounts of memory
remain tied up in pools.
This series introduces a lightweight mlx5-local pool implementation for
sub-page coherent DMA allocations, which immediately returns free
backing pages. It wires mlx5_frag_buf allocations to use these internal
pools, while keeping the mechanism reusable for other mlx5-internal
coherent DMA allocation users in follow-up work.
V2:
- Removed defensive WARN_ONCE() checks per Leon's review.
- Link to V1:
https://lore.kernel.org/r/20260428052920.219201-1-tariqt@nvidia.com/
Nimrod Oren (3):
net/mlx5: wire frag buf pools lifecycle hooks
net/mlx5: add frag buf pools create/destroy paths
net/mlx5: use internal dma pools for frag buf alloc
.../net/ethernet/mellanox/mlx5/core/alloc.c | 293 ++++++++++++++++--
.../net/ethernet/mellanox/mlx5/core/main.c | 7 +
.../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 +
include/linux/mlx5/driver.h | 9 +-
4 files changed, 277 insertions(+), 34 deletions(-)
base-commit: 09942ddedcb960f9e78fd817ec33f501d1040c5b
--
2.44.0
^ permalink raw reply
* Re: [Intel-wired-lan] [PATCH iwl-next v2] libie: log more info when virtchnl fails
From: Li Li @ 2026-04-29 20:13 UTC (permalink / raw)
To: Loktionov, Aleksandr
Cc: Nguyen, Anthony L, Kitszel, Przemyslaw, David S. Miller,
Jakub Kicinski, Eric Dumazet, intel-wired-lan@lists.osuosl.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
David Decotigny, Singhai, Anjali, Samudrala, Sridhar,
Brian Vazquez, Tantilov, Emil S
In-Reply-To: <IA3PR11MB8986566C5F04D57DEC8DE45AE5362@IA3PR11MB8986.namprd11.prod.outlook.com>
On Mon, Apr 27, 2026 at 2:06 AM Loktionov, Aleksandr
<aleksandr.loktionov@intel.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf
> > Of Li Li via Intel-wired-lan
> > Sent: Saturday, April 25, 2026 3:49 AM
> > To: Nguyen, Anthony L <anthony.l.nguyen@intel.com>; Kitszel,
> > Przemyslaw <przemyslaw.kitszel@intel.com>; David S. Miller
> > <davem@davemloft.net>; Jakub Kicinski <kuba@kernel.org>; Eric Dumazet
> > <edumazet@google.com>; intel-wired-lan@lists.osuosl.org
> > Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; David
> > Decotigny <decot@google.com>; Singhai, Anjali
> > <anjali.singhai@intel.com>; Samudrala, Sridhar
> > <sridhar.samudrala@intel.com>; Brian Vazquez <brianvv@google.com>; Li
> > Li <boolli@google.com>; Tantilov, Emil S <emil.s.tantilov@intel.com>
> > Subject: [Intel-wired-lan] [PATCH iwl-next v2] libie: log more info
> > when virtchnl fails
> >
> > Virtchnl failures can be hard to debug without logs. Logging the
> > details of virtchnl transactions can be useful for debugging virtchnl-
> > related issues.
> >
> > Tested: Built and booted on a test machine.
> >
> > Signed-off-by: Li Li <boolli@google.com>
> > ---
> > v2:
> > - Use dev_warn_ratelimited instead of dev_notice_ratelimited based on
> > reviewer feedback.
> >
> > drivers/net/ethernet/intel/libie/controlq.c | 15 +++++++++++++++
> > 1 file changed, 15 insertions(+)
> >
> > diff --git a/drivers/net/ethernet/intel/libie/controlq.c
> > b/drivers/net/ethernet/intel/libie/controlq.c
> > index ebc05355e39d..bf200fea1e12 100644
> > --- a/drivers/net/ethernet/intel/libie/controlq.c
> > +++ b/drivers/net/ethernet/intel/libie/controlq.c
> > @@ -762,6 +762,16 @@ libie_ctlq_xn_process_recv(struct
> > libie_ctlq_xn_recv_params *params,
> > status = ctlq_msg->chnl_retval ? -EFAULT : 0;
> >
> > xn = &xnm->ring[xn_index];
> > +
> > + if (ctlq_msg->chnl_retval) {
> > + dev_err_ratelimited(
> > + params->ctlq->dev,
> > + "Non-zero virtchnl ret val (msg op: %u, ret val:
> > %u, msg_cookie: %u, data_len: %u); xn op: %u, id: %u, cookie: %u\n",
> > + ctlq_msg->chnl_opcode, ctlq_msg->chnl_retval,
> > + msg_cookie, ctlq_msg->data_len, xn-
> > >virtchnl_opcode,
> > + xn->index, xn->cookie);
> Dangerous! You add xn logging before it's fields validation.
I intentionally added the log before the field validation because I
wanted to surface corrupted or timed out virtchnl replies. If we log
it only after the field validation, we won't have the visibility into
the corrupted / timed-out virtchnl replies.
But so far the bugs we have seen in our internal OOT driver all
appeared after the field validation, so I'm fine with moving the log
to after the validation as well. I will do it in the v3 patch.
>
>
> > + }
> > +
> > if (ctlq_msg->chnl_opcode != xn->virtchnl_opcode ||
> > msg_cookie != xn->cookie)
> > return false;
> > @@ -1011,6 +1021,11 @@ int libie_ctlq_xn_send(struct
> > libie_ctlq_xn_send_params *params)
> > params->recv_mem = xn->recv_mem;
> > break;
> > default:
> > + dev_warn_ratelimited(
> > + params->ctlq->dev,
> > + "Transaction failed (op %u, xn state: %d, id: %u,
> > cookie: %u, size: %zu)\n",
> > + params->chnl_opcode, xn->state, xn->index, xn-
> > >cookie,
> > + xn->recv_mem.iov_len);
> > ret = -EBADMSG;
> > break;
> > }
> > --
> > 2.54.0.rc2.544.gc7ae2d5bb8-goog
>
^ permalink raw reply
* Re: [PATCH net-next v2 1/5] net/tcp-ao: Drop support for most non-RFC-specified algorithms
From: Eric Biggers @ 2026-04-29 19:44 UTC (permalink / raw)
To: Simon Horman
Cc: netdev, linux-crypto, linux-kernel, edumazet, ncardwell, kuniyu,
davem, dsahern, kuba, pabeni, ardb, Jason, herbert, 0x7f454c46
In-Reply-To: <20260429185828.1539480-2-horms@kernel.org>
On Wed, Apr 29, 2026 at 07:58:29PM +0100, Simon Horman wrote:
> Is this a uAPI regression for existing users of TCP_AO_ADD_KEY?
>
> Before this change, tcp_ao_add::alg_name accepted any crypto_ahash
> algorithm name. The commit message itself notes "arbitrary HMAC algorithms
> probably do work", and the selftest prior to this patch exercised
> hmac(sha512), hmac(sha384), hmac(sha224), hmac(sha3-512), hmac(rmd160) and
> hmac(md5), which demonstrates those configurations were functional in
> practice.
>
> A setsockopt call that previously succeeded with e.g. alg_name =
> "hmac(sha512)" now returns -ENOENT after a kernel upgrade, with no
> deprecation window, no Kconfig opt-in, and no runtime toggle. HMAC-SHA512
> in particular seems like a plausible choice that a deployment may already
> rely on.
>
> Would it make sense to keep hmac(sha512) (and possibly hmac(sha384)) in the
> accepted list, or to gate the removal behind a Kconfig/sysctl while warning
> for a few releases?
This is intentional and has already been explained and discussed
extensively.
> Also, -ENOENT is the same error returned when the crypto module for an
> otherwise-accepted algorithm is simply not built in. Should a distinct
> errno (for example -EOPNOTSUPP) be used here so userspace can tell
> "algorithm rejected by policy" apart from "algorithm module missing"?
This review seems a bit inconsistent, as now it's advocating *for* a
UAPI change. I suggest we just stay with ENOENT, as this patch does.
> > diff --git a/tools/testing/selftests/net/tcp_ao/config b/tools/testing/selftests/net/tcp_ao/config
> > index f22148512365..47228a7d0b90 100644
> > --- a/tools/testing/selftests/net/tcp_ao/config
> > +++ b/tools/testing/selftests/net/tcp_ao/config
> > @@ -1,6 +1,5 @@
> > CONFIG_CRYPTO_CMAC=y
> > CONFIG_CRYPTO_HMAC=y
> > -CONFIG_CRYPTO_RMD160=y
> > CONFIG_CRYPTO_SHA1=y
> > CONFIG_IPV6=y
> > CONFIG_IPV6_MULTIPLE_TABLES=y
>
> Should CONFIG_CRYPTO_SHA256=y (and possibly CONFIG_CRYPTO_AES=y) be added
> here?
This is an existing bug, which patch 2 fixes by making CONFIG_TCP_AO
select the algorithms it uses.
> > diff --git a/tools/testing/selftests/net/tcp_ao/key-management.c b/tools/testing/selftests/net/tcp_ao/key-management.c
> > index 69d9a7a05d5c..d86bb380b79f 100644
> > --- a/tools/testing/selftests/net/tcp_ao/key-management.c
> > +++ b/tools/testing/selftests/net/tcp_ao/key-management.c
>
> [ ... ]
>
> > -const char *test_algos[] = {
> > - "cmac(aes128)",
> > - "hmac(sha1)", "hmac(sha512)", "hmac(sha384)", "hmac(sha256)",
> > - "hmac(sha224)", "hmac(sha3-512)",
> > - /* only if !CONFIG_FIPS */
> > -#define TEST_NON_FIPS_ALGOS 2
> > - "hmac(rmd160)", "hmac(md5)"
> > -};
> > +const char *test_algos[] = { "cmac(aes128)", "hmac(sha1)", "hmac(sha256)" };
>
> With the FIPS/algorithm-probing logic removed and roughly one third of
> generated keys now using hmac(sha256), and with the kernel side now
> returning -ENOENT for non-whitelisted algorithms, tcp_sigpool_alloc_ahash()
> will fail for every hmac(sha256) key on a kernel built from this config
> unless CRYPTO_SHA256 is selected.
>
> CRYPTO_SHA256 is not implied by CRYPTO_HMAC or TCP_AO, so on a kernel built
> strictly from tools/testing/selftests/net/tcp_ao/config the tests would
> appear to fail rather than be skipped. Can the config fragment be updated
> to match the new required algorithm set?
Again, this is an existing bug, which patch 2 fixes by making
CONFIG_TCP_AO select the algorithms it uses.
> One more question, on the commit message and documentation rather than the
> diff: Documentation/networking/tcp_ao.rst still describes TCP-AO as "May
> support any hashing algorithm"
That "May support any hashing algorithm" statement has always been
incorrect, so I wouldn't pay much attention to it. It also appears in a
table describing TCP-AO as a protocol, not the kernel's implementation.
> and does not mention the newly enforced
> whitelist or the -ENOENT failure mode. Should tcp_ao.rst be updated in
> this patch to list the accepted algorithm strings and the rationale (e.g.
> the 20-byte TCP option MAC cap), so userspace has a documented contract?
As stated in the commit message, the list of MAC algorithms supported by
the kernel's implementation of TCP-AO has always been undocumented. It
should be documented, but I would suggest documentation improvements
belong in a separate patch.
- Eric
^ permalink raw reply
* Re: [PATCH net v4 8/8] xsk: fix u64 descriptor address truncation on 32-bit architectures
From: Jason Xing @ 2026-04-29 19:02 UTC (permalink / raw)
To: Stanislav Fomichev
Cc: davem, edumazet, kuba, pabeni, bjorn, magnus.karlsson,
maciej.fijalkowski, jonathan.lemon, sdf, ast, daniel, hawk,
john.fastabend, aleksander.lobakin, bpf, netdev, Jason Xing
In-Reply-To: <afIeuwJNfwVmk9Pn@devvm7509.cco0.facebook.com>
On Wed, Apr 29, 2026 at 6:14 PM Stanislav Fomichev <sdf.kernel@gmail.com> wrote:
>
> On 04/29, Jason Xing wrote:
> > On Wed, Apr 29, 2026 at 2:11 AM Stanislav Fomichev <sdf.kernel@gmail.com> wrote:
> > >
> > > On 04/24, Jason Xing wrote:
> > > > From: Jason Xing <kernelxing@tencent.com>
> > > >
> > > > In copy mode TX, xsk_skb_destructor_set_addr() stores the 64-bit
> > > > descriptor address into skb_shinfo(skb)->destructor_arg (void *) via a
> > > > uintptr_t cast:
> > > >
> > > > skb_shinfo(skb)->destructor_arg = (void *)((uintptr_t)addr | 0x1UL);
> > > >
> > > > On 32-bit architectures uintptr_t is 32 bits, so the upper 32 bits of
> > > > the descriptor address are silently dropped. In unaligned mode the chunk
> > > > offset is encoded in bits 48-63 of the descriptor address
> > > > (XSK_UNALIGNED_BUF_OFFSET_SHIFT = 48), meaning the offset is lost
> > > > entirely. The completion queue then returns a truncated address to
> > > > userspace, making buffer recycling impossible.
> > > >
> > > > Fix this by handling the 32-bit case in the destructor_arg helpers:
> > > >
> > > > - xsk_skb_destructor_set_addr(): on !CONFIG_64BIT, allocate an
> > > > xsk_addrs struct via kmem_cache_zalloc() to store the full u64
> > > > address. Leave num_descs as 0 (zalloc) so that the subsequent
> > > > xsk_inc_num_desc() brings it to the correct count of 1.
> > > >
> > > > - xsk_skb_destructor_is_addr(): on !CONFIG_64BIT, return true only
> > > > when destructor_arg is NULL (not yet set), false when it points to
> > > > an xsk_addrs struct.
> > > >
> > > > - xsk_skb_init_misc(): call xsk_skb_destructor_set_addr() first
> > > > before touching any other skb fields; on failure return early so
> > > > the skb destructor is never changed from sock_wfree.
> > > >
> > > > The existing xsk_consume_skb() already handles 32-bit correctly after
> > > > these changes: xsk_skb_destructor_is_addr() returns false for any
> > > > allocated xsk_addrs, so the kmem_cache_free path is always taken.
> > > >
> > > > The overhead is one extra kmem_cache_zalloc per first descriptor on
> > > > 32-bit only; 64-bit builds are completely unchanged.
> > > >
> > > > Closes: https://lore.kernel.org/all/20260419045824.D9E5EC2BCAF@smtp.kernel.org/
> > > > Fixes: 0ebc27a4c67d ("xsk: avoid data corruption on cq descriptor number")
> > > > Signed-off-by: Jason Xing <kernelxing@tencent.com>
> > > > ---
> > > > net/xdp/xsk.c | 38 +++++++++++++++++++++++++++++++-------
> > > > 1 file changed, 31 insertions(+), 7 deletions(-)
> > > >
> > > > diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> > > > index ed96f6ec8ff2..fe88f47741b5 100644
> > > > --- a/net/xdp/xsk.c
> > > > +++ b/net/xdp/xsk.c
> > > > @@ -558,7 +558,10 @@ static int xsk_cq_reserve_locked(struct xsk_buff_pool *pool)
> > > >
> > > > static bool xsk_skb_destructor_is_addr(struct sk_buff *skb)
> > > > {
> > > > - return (uintptr_t)skb_shinfo(skb)->destructor_arg & 0x1UL;
> > > > + if (IS_ENABLED(CONFIG_64BIT))
> > > > + return (uintptr_t)skb_shinfo(skb)->destructor_arg & 0x1UL;
> > > > + else
> > > > + return !skb_shinfo(skb)->destructor_arg;
> > >
> > > Don't understand why we need to special case CONFIG_64BIT here?
> > > Shouldn't the same existing condition work on 32bit?
> >
> > Because 0x1UL is the particular semantic applied on a 64-bit arch.
> > xsk_skb_destructor_set_addr() sets it while
> > xsk_skb_destructor_is_addr() recognizes it. They are a pair.
> >
> > As you noticed, one liner works but is not that appropriate: on a
> > 32-bit arch, this member should be either a NULL point or a valid
> > pointer pointing to a memory region. Testing if it's NULL can be
> > helpful as to the long term maintenance because of its readability and
> > robustness/safety.
> >
> > The error path in allocation of skb is really complex, which is why
> > I'm so cautious to take care of it :)
>
> Let's cleanup the error path instead of adding more complexity? Similar to what
> you do with your "xsk: fix xsk_addrs slab leak on multi-buffer error path",
> but maybe add a few NULL checks?
Good suggestion. I think I can cook a follow up patch to do such a
thing targetting net-next tree. This patch 8 belongs to net material
which means it will be backported to the older stable kernel as soon
as it gets merged. IIUC, the better way is to make it as simple as
possible?
>
> Instead of 32 vs 64, I'd like to reason about whether destructor_arg
> is an address or an allocated array (not whether we have 1 or >1
> descriptors). And we special case 32 bit by always allocating it.
>
> Haven't checked, but maybe this is all you need (besides your _set_addr
> changes)?
>
> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> index 6149f6a79897..03f217e85d31 100644
> --- a/net/xdp/xsk.c
> +++ b/net/xdp/xsk.c
> @@ -589,6 +589,8 @@ static u32 xsk_get_num_desc(struct sk_buff *skb)
> return 1;
>
> xsk_addr = (struct xsk_addrs *)skb_shinfo(skb)->destructor_arg;
> + if (!xsk_addr)
> + return 0;
>
> return xsk_addr->num_descs;
> }
Right, as I mentioned, how about posting a new cleanup patch with your
suggested-by tag?
>
> > I've noticed the status has been changed to 'changes requested'. Does
> > that mean one way or another I have to post a new version?
>
> That wasn't me :-) From my POW, patches 1-7 are good to go..
Great! Thanks for the review. My hope is to get this series merged
soon in the net tree.
>
> > >
> > > > }
> > > >
> > > > static u64 xsk_skb_destructor_get_addr(struct sk_buff *skb)
> > > > @@ -566,9 +569,21 @@ static u64 xsk_skb_destructor_get_addr(struct sk_buff *skb)
> > > > return (u64)((uintptr_t)skb_shinfo(skb)->destructor_arg & ~0x1UL);
> > > > }
> > > >
> > > > -static void xsk_skb_destructor_set_addr(struct sk_buff *skb, u64 addr)
> > > > +static int xsk_skb_destructor_set_addr(struct sk_buff *skb, u64 addr)
> > > > {
> > >
> > > [..]
> > >
> > > > + if (!IS_ENABLED(CONFIG_64BIT)) {
> > > > + struct xsk_addrs *xsk_addr;
> > > > +
> > > > + xsk_addr = kmem_cache_zalloc(xsk_tx_generic_cache, GFP_KERNEL);
> > > > + if (!xsk_addr)
> > > > + return -ENOMEM;
> > > > + xsk_addr->addrs[0] = addr;
> > > > + skb_shinfo(skb)->destructor_arg = (void *)xsk_addr;
> > > > + return 0;
> > > > + }
> > > > +
> > > > skb_shinfo(skb)->destructor_arg = (void *)((uintptr_t)addr | 0x1UL);
> > > > + return 0;
> > >
> > > I think this is gonna be a 3rd copy paste of the same logic? Let's
> > > move to a new helper and replace existing kmem_cache_zalloc places?
> > >
> > > xsk_skb_destructor_alloc_list(prev_addr) ?
>
> Any comments on this?
I didn't comment on this because I thought I was not that sure if we
needed to wrap it up in the stable kernels :)
Of course, it would be easier for me to work on the net-next tree to
make the code look
more neat and elegant.
Thanks,
Jason
^ permalink raw reply
* Re: [PATCH net-next v2 1/5] net/tcp-ao: Drop support for most non-RFC-specified algorithms
From: Simon Horman @ 2026-04-29 18:58 UTC (permalink / raw)
To: ebiggers
Cc: 'Simon Horman', netdev, linux-crypto, linux-kernel,
edumazet, ncardwell, kuniyu, davem, dsahern, kuba, pabeni, ardb,
Jason, herbert, 0x7f454c46
In-Reply-To: <20260427172727.9310-2-ebiggers@kernel.org>
From: 'Simon Horman' <horms@kernel.org>
This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
Full review at: https://sashiko.dev
---
Subject: net/tcp-ao: Drop support for most non-RFC-specified algorithms
This change restricts the algorithms accepted by setsockopt(TCP_AO_ADD_KEY)
to cmac(aes128)/cmac(aes)+keylen=16, hmac(sha1) and hmac(sha256), returning
-ENOENT for anything else. The selftest algorithm table and its Kconfig
fragment are updated to drop FIPS detection and rmd160.
A few questions inline below.
> diff --git a/net/ipv4/tcp_ao.c b/net/ipv4/tcp_ao.c
> index a97cdf3e6af4..b21bd69b4e82 100644
> --- a/net/ipv4/tcp_ao.c
> +++ b/net/ipv4/tcp_ao.c
> @@ -1563,6 +1563,10 @@ static struct tcp_ao_key *tcp_ao_key_alloc(struct sock *sk,
> /* RFC5926, 3.1.1.2. KDF_AES_128_CMAC */
> if (!strcmp("cmac(aes128)", algo))
> algo = "cmac(aes)";
> + else if (strcmp("hmac(sha1)", algo) &&
> + strcmp("hmac(sha256)", algo) &&
> + (strcmp("cmac(aes)", algo) || cmd->keylen != 16))
> + return ERR_PTR(-ENOENT);
Is this a uAPI regression for existing users of TCP_AO_ADD_KEY?
Before this change, tcp_ao_add::alg_name accepted any crypto_ahash
algorithm name. The commit message itself notes "arbitrary HMAC algorithms
probably do work", and the selftest prior to this patch exercised
hmac(sha512), hmac(sha384), hmac(sha224), hmac(sha3-512), hmac(rmd160) and
hmac(md5), which demonstrates those configurations were functional in
practice.
A setsockopt call that previously succeeded with e.g. alg_name =
"hmac(sha512)" now returns -ENOENT after a kernel upgrade, with no
deprecation window, no Kconfig opt-in, and no runtime toggle. HMAC-SHA512
in particular seems like a plausible choice that a deployment may already
rely on.
Would it make sense to keep hmac(sha512) (and possibly hmac(sha384)) in the
accepted list, or to gate the removal behind a Kconfig/sysctl while warning
for a few releases?
Also, -ENOENT is the same error returned when the crypto module for an
otherwise-accepted algorithm is simply not built in. Should a distinct
errno (for example -EOPNOTSUPP) be used here so userspace can tell
"algorithm rejected by policy" apart from "algorithm module missing"?
> diff --git a/tools/testing/selftests/net/tcp_ao/config b/tools/testing/selftests/net/tcp_ao/config
> index f22148512365..47228a7d0b90 100644
> --- a/tools/testing/selftests/net/tcp_ao/config
> +++ b/tools/testing/selftests/net/tcp_ao/config
> @@ -1,6 +1,5 @@
> CONFIG_CRYPTO_CMAC=y
> CONFIG_CRYPTO_HMAC=y
> -CONFIG_CRYPTO_RMD160=y
> CONFIG_CRYPTO_SHA1=y
> CONFIG_IPV6=y
> CONFIG_IPV6_MULTIPLE_TABLES=y
Should CONFIG_CRYPTO_SHA256=y (and possibly CONFIG_CRYPTO_AES=y) be added
here?
> diff --git a/tools/testing/selftests/net/tcp_ao/key-management.c b/tools/testing/selftests/net/tcp_ao/key-management.c
> index 69d9a7a05d5c..d86bb380b79f 100644
> --- a/tools/testing/selftests/net/tcp_ao/key-management.c
> +++ b/tools/testing/selftests/net/tcp_ao/key-management.c
[ ... ]
> -const char *test_algos[] = {
> - "cmac(aes128)",
> - "hmac(sha1)", "hmac(sha512)", "hmac(sha384)", "hmac(sha256)",
> - "hmac(sha224)", "hmac(sha3-512)",
> - /* only if !CONFIG_FIPS */
> -#define TEST_NON_FIPS_ALGOS 2
> - "hmac(rmd160)", "hmac(md5)"
> -};
> +const char *test_algos[] = { "cmac(aes128)", "hmac(sha1)", "hmac(sha256)" };
With the FIPS/algorithm-probing logic removed and roughly one third of
generated keys now using hmac(sha256), and with the kernel side now
returning -ENOENT for non-whitelisted algorithms, tcp_sigpool_alloc_ahash()
will fail for every hmac(sha256) key on a kernel built from this config
unless CRYPTO_SHA256 is selected.
CRYPTO_SHA256 is not implied by CRYPTO_HMAC or TCP_AO, so on a kernel built
strictly from tools/testing/selftests/net/tcp_ao/config the tests would
appear to fail rather than be skipped. Can the config fragment be updated
to match the new required algorithm set?
One more question, on the commit message and documentation rather than the
diff: Documentation/networking/tcp_ao.rst still describes TCP-AO as "May
support any hashing algorithm" and does not mention the newly enforced
whitelist or the -ENOENT failure mode. Should tcp_ao.rst be updated in
this patch to list the accepted algorithm strings and the rationale (e.g.
the 20-byte TCP option MAC cap), so userspace has a documented contract?
^ permalink raw reply
* Re: [PATCH v2 2/9] x86/extable: switch to using FIELD_GET_SIGNED()
From: Yury Norov @ 2026-04-29 19:00 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Andy Lutomirski, Jonathan Cameron, David Lechner,
Johannes Berg, David Laight, Nuno Sá, Andy Shevchenko,
Ping-Ke Shih, Richard Cochran, Andrew Lunn, David S. Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni, Alexandre Belloni,
Yury Norov, Rasmus Villemoes, Hans de Goede, Linus Walleij,
Sakari Ailus, Salah Triki, Achim Gratz, Ben Collins, x86,
linux-kernel, linux-iio, linux-wireless, netdev, linux-rtc
In-Reply-To: <20260428093905.GA1026330@noisy.programming.kicks-ass.net>
On Tue, Apr 28, 2026 at 11:39:05AM +0200, Peter Zijlstra wrote:
> On Mon, Apr 27, 2026 at 05:41:19PM -0400, Yury Norov wrote:
> > The EX_DATA register is laid out such that EX_DATA_IMM occupied MSB.
> > It's done to make sure that FIELD_GET() will sign-extend the IMM
> > field during extraction.
> >
> > To enforce that, all EX_DATA masks are made signed integers. This
> > works, but relies on the particular implementation of FIELD_GET(),
> > i.e. masking then shifting, not vice versa; and the particular
> > placement of the fields in the register.
> >
> > Switch to using the dedicated FIELD_GET_SIGNED(), and relax those
> > limitations.
> >
> > Signed-off-by: Yury Norov <ynorov@nvidia.com>
>
> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Alright, thanks Peter and everyone. Now that all the patches except #9
are reviewed, I'm taking the series into bitmap-for-next.
Thanks,
Yury
^ permalink raw reply
* Re: [Intel-wired-lan] [PATCH iwl-next] libie: log more info when virtchnl fails
From: Li Li @ 2026-04-29 19:00 UTC (permalink / raw)
To: Paul Menzel
Cc: Tony Nguyen, Przemek Kitszel, David S. Miller, Jakub Kicinski,
Eric Dumazet, intel-wired-lan, netdev, linux-kernel,
David Decotigny, Anjali Singhai, Sridhar Samudrala, Brian Vazquez,
emil.s.tantilov
In-Reply-To: <2d8bd703-f368-4405-9266-d0d8824455aa@molgen.mpg.de>
On Mon, Apr 27, 2026 at 3:40 AM Paul Menzel <pmenzel@molgen.mpg.de> wrote:
>
> Dear Li,
>
>
> Thank you for your patch.
>
> Am 24.04.26 um 05:15 schrieb Li Li via Intel-wired-lan:
> > Virtchnl failures can be hard to debug without logs. Logging the details
> > of virtchnl transactions can be useful for debugging virtchnl-related
> > issues.
>
> Why is the first added one added as error, and the second as info?
Originally I was thinking the second log doesn't warrant an err level
log, but on second thought, I think it makes more sense to use
dev_err_ratelimited() in both places. I will send out a v3 patch to
update it.
>
>
> Do you have a reproducer to get each of the new log messages?
We recently had an IOMMU subsystem bug in our internal kernel that
caused virtchnl failures, and this patch helped root cause the IOMMU
subsystem issue. However, the patch we used internally was applied to
the OOT driver, which is implemented slightly differently than the
upstream driver. Example log we saw in our OOT driver, with the patch
applied:
idpf 0000:43:00.0: Non-zero virtchnl ret val (msg: 523, ret val: 6,
data_len: 0) (xn 523, id: 254, salt: 40, iov len: 128)
The message looks slightly different from the log added in this patch
because the OOT driver uses different terminology.
>
> > Tested: Built and booted on a test machine.
>
> Please paste the new messages.
As mentioned above, I don't have a reproducer for a real virtchnl
error in the upstream kernel, but I synthetically produced an error by
changing ctlq_msg->chnl_retval, and the following message appeared:
[ 309.891790] idpf 0000:01:00.0: Non-zero virtchnl ret val (msg op:
1, ret val: 6, msg_cookie: 0, data_len: 8); xn op: 1, id: 0, cookie: 0
[ 309.892357] idpf 0000:01:00.0: Transaction failed (op 1, xn state:
3, id: 0, cookie: 0, size: 8)
I will mention it in the v3 patch as well.
>
> > Signed-off-by: Li Li <boolli@google.com>
> > ---
> > drivers/net/ethernet/intel/libie/controlq.c | 15 +++++++++++++++
> > 1 file changed, 15 insertions(+)
> >
> > diff --git a/drivers/net/ethernet/intel/libie/controlq.c b/drivers/net/ethernet/intel/libie/controlq.c
> > index ebc05355e39d..7eaa77413621 100644
> > --- a/drivers/net/ethernet/intel/libie/controlq.c
> > +++ b/drivers/net/ethernet/intel/libie/controlq.c
> > @@ -762,6 +762,16 @@ libie_ctlq_xn_process_recv(struct libie_ctlq_xn_recv_params *params,
> > status = ctlq_msg->chnl_retval ? -EFAULT : 0;
> >
> > xn = &xnm->ring[xn_index];
> > +
> > + if (ctlq_msg->chnl_retval) {
> > + dev_err_ratelimited(
> > + params->ctlq->dev,
> > + "Non-zero virtchnl ret val (msg op: %u, ret val: %u, msg_cookie: %u, data_len: %u); xn op: %u, id: %u, cookie: %u\n",
> > + ctlq_msg->chnl_opcode, ctlq_msg->chnl_retval,
> > + msg_cookie, ctlq_msg->data_len, xn->virtchnl_opcode,
> > + xn->index, xn->cookie);
> > + }
> > +
> > if (ctlq_msg->chnl_opcode != xn->virtchnl_opcode ||
> > msg_cookie != xn->cookie)
> > return false;
> > @@ -1011,6 +1021,11 @@ int libie_ctlq_xn_send(struct libie_ctlq_xn_send_params *params)
> > params->recv_mem = xn->recv_mem;
> > break;
> > default:
> > + dev_notice_ratelimited(
> > + params->ctlq->dev,
> > + "Transaction failed (op %u, xn state: %d, id: %u, cookie: %u, size: %zu)\n",
> > + params->chnl_opcode, xn->state, xn->index, xn->cookie,
> > + xn->recv_mem.iov_len);
> > ret = -EBADMSG;
> > break;
> > }
>
>
> Kind regards,
>
> Paul
^ permalink raw reply
* [PATCH net, v2] net: mana: Fix crash from unvalidated SHM offset read from BAR0 during FLR
From: Dipayaan Roy @ 2026-04-29 18:57 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, andrew+netdev, davem, edumazet,
kuba, pabeni, leon, longli, kotaranov, horms, shradhagupta,
ssengar, ernis, shirazsaleem, linux-hyperv, netdev, linux-kernel,
linux-rdma, stephen, jacob.e.keller, dipayanroy, leitao, kees,
john.fastabend, hawk, bpf, daniel, ast, sdf, yury.norov
During Function Level Reset recovery, the MANA driver reads
hardware BAR0 registers that may temporarily contain garbage values.
The SHM (Shared Memory) offset read from GDMA_REG_SHM_OFFSET is used
to compute gc->shm_base, which is later dereferenced via readl() in
mana_smc_poll_register(). If the hardware returns an unaligned or
out-of-range value, the driver must not blindly use it, as this would
propagate the hardware error into a kernel crash.
The following crash was observed on an arm64 Hyper-V guest running
kernel 6.17.0-3013-azure during VF reset recovery triggered by HWC
timeout.
[13291.785274] Unable to handle kernel paging request at virtual address ffff8000a200001b
[13291.785311] Mem abort info:
[13291.785332] ESR = 0x0000000096000021
[13291.785343] EC = 0x25: DABT (current EL), IL = 32 bits
[13291.785355] SET = 0, FnV = 0
[13291.785363] EA = 0, S1PTW = 0
[13291.785372] FSC = 0x21: alignment fault
[13291.785382] Data abort info:
[13291.785391] ISV = 0, ISS = 0x00000021, ISS2 = 0x00000000
[13291.785404] CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[13291.785412] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[13291.785421] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000014df3a1000
[13291.785432] [ffff8000a200001b] pgd=1000000100438403, p4d=1000000100438403, pud=1000000100439403, pmd=0068000fc2000711
[13291.785703] Internal error: Oops: 0000000096000021 [#1] SMP
[13291.830975] Modules linked in: tls qrtr mana_ib ib_uverbs ib_core xt_owner xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables cfg80211 8021q garp mrp stp llc binfmt_misc joydev serio_raw nls_iso8859_1 hid_generic aes_ce_blk aes_ce_cipher polyval_ce ghash_ce sm4_ce_gcm sm4_ce_ccm sm4_ce sm4_ce_cipher hid_hyperv sm4 sm3_ce sha3_ce hv_netvsc hid vmgenid hyperv_keyboard hyperv_drm sch_fq_codel nvme_fabrics efi_pstore dm_multipath nfnetlink vsock_loopback vmw_vsock_virtio_transport_common hv_sock vmw_vsock_vmci_transport vmw_vmci vsock dmi_sysfs ip_tables x_tables autofs4
[13291.862630] CPU: 122 UID: 0 PID: 61796 Comm: kworker/122:2 Tainted: G W 6.17.0-3013-azure #13-Ubuntu VOLUNTARY
[13291.869902] Tainted: [W]=WARN
[13291.871901] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 01/08/2026
[13291.878086] Workqueue: events mana_serv_func
[13291.880718] pstate: 62400005 (nZCv daif +PAN -UAO +TCO -DIT -SSBS BTYPE=--)
[13291.884835] pc : mana_smc_poll_register+0x48/0xb0
[13291.887902] lr : mana_smc_setup_hwc+0x70/0x1c0
[13291.890493] sp : ffff8000ab79bbb0
[13291.892364] x29: ffff8000ab79bbb0 x28: ffff00410c8b5900 x27: ffff00410d630680
[13291.896252] x26: ffff004171f9fd80 x25: 000000016ed55000 x24: 000000017f37e000
[13291.899990] x23: 0000000000000000 x22: 000000016ed55000 x21: 0000000000000000
[13291.904497] x20: ffff8000a200001b x19: 0000000000004e20 x18: ffff8000a6183050
[13291.908308] x17: 0000000000000000 x16: 0000000000000000 x15: 000000000000000a
[13291.912542] x14: 0000000000000004 x13: 0000000000000000 x12: 0000000000000000
[13291.916298] x11: 0000000000000000 x10: 0000000000000001 x9 : ffffc45006af1bd8
[13291.920945] x8 : ffff000151129000 x7 : 0000000000000000 x6 : 0000000000000000
[13291.925293] x5 : 000000015f214000 x4 : 000000017217a000 x3 : 000000016ed50000
[13291.930436] x2 : 000000016ed55000 x1 : 0000000000000000 x0 : ffff8000a1ffffff
[13291.934342] Call trace:
[13291.935736] mana_smc_poll_register+0x48/0xb0 (P)
[13291.938611] mana_smc_setup_hwc+0x70/0x1c0
[13291.941113] mana_hwc_create_channel+0x1a0/0x3a0
[13291.944283] mana_gd_setup+0x16c/0x398
[13291.946584] mana_gd_resume+0x24/0x70
[13291.948917] mana_do_service+0x13c/0x1d0
[13291.951583] mana_serv_func+0x34/0x68
[13291.953732] process_one_work+0x168/0x3d0
[13291.956745] worker_thread+0x2ac/0x480
[13291.959104] kthread+0xf8/0x110
[13291.961026] ret_from_fork+0x10/0x20
[13291.963560] Code: d2807d00 9417c551 71000673 54000220 (b9400281)
[13291.967299] ---[ end trace 0000000000000000 ]---
Disassembly of mana_smc_poll_register() around the crash site:
Disassembly of section .text:
00000000000047c8 <mana_smc_poll_register>:
47c8: d503201f nop
47cc: d503201f nop
47d0: d503233f paciasp
47d4: f800865e str x30, [x18], #8
47d8: a9bd7bfd stp x29, x30, [sp, #-48]!
47dc: 910003fd mov x29, sp
47e0: a90153f3 stp x19, x20, [sp, #16]
47e4: 91007014 add x20, x0, #0x1c
47e8: 5289c413 mov w19, #0x4e20
47ec: f90013f5 str x21, [sp, #32]
47f0: 12001c35 and w21, w1, #0xff
47f4: 14000008 b 4814 <mana_smc_poll_register+0x4c>
47f8: 36f801e1 tbz w1, #31, 4834 <mana_smc_poll_register+0x6c>
47fc: 52800042 mov w2, #0x2
4800: d280fa01 mov x1, #0x7d0
4804: d2807d00 mov x0, #0x3e8
4808: 94000000 bl 0 <usleep_range_state>
480c: 71000673 subs w19, w19, #0x1
4810: 54000200 b.eq 4850 <mana_smc_poll_register+0x88>
4814: b9400281 ldr w1, [x20] <-- **** CRASHED HERE *****
4818: d50331bf dmb oshld
481c: 2a0103e2 mov w2, w1
...
From the crash signature x20 = ffff8000a200001b, this address
ends in 0x1b which is not 4-byte aligned, so the 'ldr w1, [x20]'
instruction (readl) triggers the arm64 alignment fault (FSC = 0x21).
The root cause is in mana_gd_init_vf_regs(), which computes:
gc->shm_base = gc->bar0_va + mana_gd_r64(gc, GDMA_REG_SHM_OFFSET);
The offset is used without any validation. The same problem exists
in mana_gd_init_pf_regs() for sriov_base_off and sriov_shm_off.
Fix this by validating all offsets before use:
- VF: check shm_off is within BAR0, properly aligned to 4 bytes
(readl requirement), and leaves room for the full 256-bit
(32-byte) SMC aperture.
- PF: check sriov_base_off is within BAR0, aligned to 8 bytes
(readq requirement), and leaves room to safely read the
sriov_shm_off register at sriov_base_off + GDMA_PF_REG_SHM_OFF.
Then check sriov_shm_off leaves room for the full SMC aperture.
All arithmetic uses subtraction rather than addition to avoid
integer overflow on garbage firmware values.
without validating the offset read from hardware. If the register
returns a garbage value that is neither within bar 0 bounds nor aligned
to the 4-byte granularity, thus causing the alignment fault.
Define SMC_APERTURE_SIZE (32 bytes, derived from the 256-bit aperture
width)
Return -EPROTO on invalid values. The existing recovery path in
mana_serv_reset() already handles -EPROTO by falling through to PCI
device rescan, giving the hardware another chance to present valid
register values after reset.
Fixes: 9bf66036d686 ("net: mana: Handle hardware recovery events when probing the device")
Signed-off-by: Dipayaan Roy <dipayanroy@linux.microsoft.com>
---
Changes in v2:
- Fix sriov_base_off alignment check: sizeof(u32) to sizeof(u64), since
mana_gd_r64() (readq) requires 8-byte alignment on arm64.
- Fix sriov_base_off bounds: also verify enough space remains in BAR0
to safely read sriov_shm_off at offset GDMA_PF_REG_SHM_OFF + 8 bytes.
- Fix integer overflow: rewrite bounds checks using subtraction
(remaining = bar0_size - base) instead of addition.
- Fix SMC aperture size: add gc->bar0_size - shm_off < SMC_APERTURE_SIZE
checks in both VF and PF paths; previously only the start address was
validated, but mana_smc_poll_register() accesses up to shm_base + 0x1c
(28 bytes from base, 32 bytes total).
- Export SMC_APERTURE_SIZE to shm_channel.h.
---
.../net/ethernet/microsoft/mana/gdma_main.c | 40 ++++++++++++++++---
include/net/mana/shm_channel.h | 6 +++
2 files changed, 41 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c b/drivers/net/ethernet/microsoft/mana/gdma_main.c
index 098fbda0d128..d8e816882f02 100644
--- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
+++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
@@ -43,8 +43,9 @@ static u64 mana_gd_r64(struct gdma_context *g, u64 offset)
static int mana_gd_init_pf_regs(struct pci_dev *pdev)
{
struct gdma_context *gc = pci_get_drvdata(pdev);
- void __iomem *sriov_base_va;
+ u64 remaining_barsize;
u64 sriov_base_off;
+ u64 sriov_shm_off;
gc->db_page_size = mana_gd_r32(gc, GDMA_PF_REG_DB_PAGE_SIZE) & 0xFFFF;
@@ -73,10 +74,28 @@ static int mana_gd_init_pf_regs(struct pci_dev *pdev)
gc->phys_db_page_base = gc->bar0_pa + gc->db_page_off;
sriov_base_off = mana_gd_r64(gc, GDMA_SRIOV_REG_CFG_BASE_OFF);
+ if (sriov_base_off >= gc->bar0_size ||
+ gc->bar0_size - sriov_base_off <
+ GDMA_PF_REG_SHM_OFF + sizeof(u64) ||
+ !IS_ALIGNED(sriov_base_off, sizeof(u64))) {
+ dev_err(gc->dev,
+ "SRIOV base offset 0x%llx out of range or unaligned (BAR0 size 0x%llx)\n",
+ sriov_base_off, (u64)gc->bar0_size);
+ return -EPROTO;
+ }
- sriov_base_va = gc->bar0_va + sriov_base_off;
- gc->shm_base = sriov_base_va +
- mana_gd_r64(gc, sriov_base_off + GDMA_PF_REG_SHM_OFF);
+ remaining_barsize = gc->bar0_size - sriov_base_off;
+ sriov_shm_off = mana_gd_r64(gc, sriov_base_off + GDMA_PF_REG_SHM_OFF);
+ if (sriov_shm_off >= remaining_barsize ||
+ remaining_barsize - sriov_shm_off < SMC_APERTURE_SIZE ||
+ !IS_ALIGNED(sriov_shm_off, sizeof(u32))) {
+ dev_err(gc->dev,
+ "SRIOV SHM offset 0x%llx out of range or unaligned (BAR0 size 0x%llx)\n",
+ sriov_shm_off, (u64)gc->bar0_size);
+ return -EPROTO;
+ }
+
+ gc->shm_base = gc->bar0_va + sriov_base_off + sriov_shm_off;
return 0;
}
@@ -84,6 +103,7 @@ static int mana_gd_init_pf_regs(struct pci_dev *pdev)
static int mana_gd_init_vf_regs(struct pci_dev *pdev)
{
struct gdma_context *gc = pci_get_drvdata(pdev);
+ u64 shm_off;
gc->db_page_size = mana_gd_r32(gc, GDMA_REG_DB_PAGE_SIZE) & 0xFFFF;
@@ -111,7 +131,17 @@ static int mana_gd_init_vf_regs(struct pci_dev *pdev)
gc->db_page_base = gc->bar0_va + gc->db_page_off;
gc->phys_db_page_base = gc->bar0_pa + gc->db_page_off;
- gc->shm_base = gc->bar0_va + mana_gd_r64(gc, GDMA_REG_SHM_OFFSET);
+ shm_off = mana_gd_r64(gc, GDMA_REG_SHM_OFFSET);
+ if (shm_off >= gc->bar0_size ||
+ gc->bar0_size - shm_off < SMC_APERTURE_SIZE ||
+ !IS_ALIGNED(shm_off, sizeof(u32))) {
+ dev_err(gc->dev,
+ "SHM offset 0x%llx out of range or unaligned (BAR0 size 0x%llx)\n",
+ shm_off, (u64)gc->bar0_size);
+ return -EPROTO;
+ }
+
+ gc->shm_base = gc->bar0_va + shm_off;
return 0;
}
diff --git a/include/net/mana/shm_channel.h b/include/net/mana/shm_channel.h
index 5199b41497ff..dbabcfb95daf 100644
--- a/include/net/mana/shm_channel.h
+++ b/include/net/mana/shm_channel.h
@@ -4,6 +4,12 @@
#ifndef _SHM_CHANNEL_H
#define _SHM_CHANNEL_H
+#define SMC_APERTURE_BITS 256
+#define SMC_BASIC_UNIT (sizeof(u32))
+#define SMC_APERTURE_DWORDS (SMC_APERTURE_BITS / (SMC_BASIC_UNIT * 8))
+#define SMC_LAST_DWORD (SMC_APERTURE_DWORDS - 1)
+#define SMC_APERTURE_SIZE (SMC_APERTURE_BITS / 8)
+
struct shm_channel {
struct device *dev;
void __iomem *base;
--
2.43.0
^ permalink raw reply related
* Re: [PATCH net v2] strparser: Fix race condition in strp_done()
From: Hyunwoo Kim @ 2026-04-29 18:46 UTC (permalink / raw)
To: Sabrina Dubroca
Cc: davem, edumazet, kuba, pabeni, horms, Julia.Lawall, linux,
nate.karstens, netdev, imv4bel
In-Reply-To: <ab2bA-g2qnwhdWbq@v4bel>
On Sat, Mar 21, 2026 at 04:07:47AM +0900, Hyunwoo Kim wrote:
> On Wed, Mar 11, 2026 at 01:13:45PM +0900, Hyunwoo Kim wrote:
> > On Fri, Mar 06, 2026 at 08:41:02PM +0900, Hyunwoo Kim wrote:
> > > On Fri, Mar 06, 2026 at 11:13:19AM +0100, Sabrina Dubroca wrote:
> > > > 2026-03-06, 09:11:04 +0900, Hyunwoo Kim wrote:
> > > > > On Fri, Mar 06, 2026 at 12:35:48AM +0100, Sabrina Dubroca wrote:
> > > > > > Sorry for the delay, I wanted to think about the race condition a bit
> > > > > > more.
> > > > > >
> > > > > > 2026-03-03, 10:50:05 +0900, Hyunwoo Kim wrote:
> > > > > > > On Tue, Mar 03, 2026 at 12:10:33AM +0100, Sabrina Dubroca wrote:
> > > > > > > > 2026-02-27, 06:51:10 +0900, Hyunwoo Kim wrote:
> > > > > > > > > On Mon, Feb 23, 2026 at 06:20:58PM +0100, Sabrina Dubroca wrote:
> > > > > > > > > > 2026-02-20, 18:29:55 +0900, Hyunwoo Kim wrote:
> > > > > > > > > > "strp stopped" is not really enough, I think we'd also need to reset
> > > > > > > > > > the CBs, and then grab bh_lock_sock to make sure a previously-running
> > > > > > > > > > ->sk_data_ready has completed. This is what kcm does, at least.
> > > > > > > > >
> > > > > > > > > It seems that this is not something that should be handled inside strp itself,
> > > > > > > > > but rather something that each caller of strp_stop() is expected to take care
> > > > > > > > > of individually. Would that be the right direction?
> > > > > > > >
> > > > > > > > Agree.
> > > > > > > >
> > > > > > > > > It also appears that ovpn and kcm handle this by implementing their own callback
> > > > > > > > > restoration logic.
> > > > > > > >
> > > > > > > > Right. I tried to look at skmsg/psock (the other user of strp), but
> > > > > > > > didn't get far enough to verify if it's handling this correctly.
> > > > > > > >
> > > > > > > > > > Without that, if strp_recv runs in parallel (not from strp->work) with
> > > > > > > > > > strp_done, cleaning up skb_head in strp_done seems problematic.
> > > > > > > > >
> > > > > > > > > From the espintcp perspective, how about applying a patch along the following lines?
> > > > > > > >
> > > > > > > > This is what I was thinking about, yes.
> > > > > > >
> > > > > > > In my opinion, it might be cleaner to split the espintcp callback restoration work into
> > > > > > > a separate patch, rather than merging it into the strparser v3 patch. What do you think?
> > > > > >
> > > > > > Sure. But once espintcp is fixed in that way, can the original race
> > > > > > condition with strparser still occur? release_sock() will wait for any
> > > > >
> > > > > If the espintcp callback restoration patch is applied, the strparser
> > > > > race should no longer occur in espintcp.
> > > > >
> > > > > > espintcp_data_ready()/strp_data_ready() that's already running, and a
> > > > > > sk_data_ready that starts after we've changed the callbacks will not
> > > > > > end up in strp_data_ready() at all so it won't restart the works that
> > > > > > are being stopped by strp_done()?
> > > > > >
> > > > > > It's quite reasonable to use disable*_work_sync in strp_done, but I'm
> > > > > > not sure there's a bug other than espintcp not terminating itself
> > > > > > correctly on the socket.
> > > > >
> > > > > That said, the _cancel APIs in strparser still appear to carry some
> > > > > structural risk, so it might still make sense to switch to the _disable
> > > > > APIs for the benefit of other strp users or potential future callers.
> > > >
> > > > Not really. Every user of strp that is open to the strp_recv vs
> > > > cancel_* race is also open to the strp_recv vs free race, so switching
> > > > from cancel_* to disable_* is only a partial fix.
> > > >
> > > > But if we took and released the socket lock in strp_done, we would
> > > > solve the issue for all users even without resetting the callbacks?
> > >
> > > Looks good to me. With this change, it seems the issue can be resolved
> > > not only for espintcp but for all strp users.
> > >
> > > When strp_stop() runs first:
> > > ```
> > > cpu0 cpu1
> > >
> > > espintcp_close()
> > > strp_stop()
> > > strp->stopped = 1;
> > > espintcp_data_ready()
> > > strp_data_ready()
> > > if (unlikely(strp->stopped)) return;
> > > strp_done()
> > > lock_sock();
> > > release_sock();
> > > cancel_delayed_work_sync(&strp->msg_timer_work);
> > > kfree_skb(strp->skb_head);
> > > ```
> > >
> > > When strp_data_ready() runs first:
> > > ```
> > > cpu0 cpu1
> > >
> > > espintcp_data_ready()
> > > strp_data_ready()
> > > if (unlikely(strp->stopped)) return;
> > > espintcp_close()
> > > strp_stop()
> > > strp->stopped = 1;
> > > strp_done()
> > > lock_sock();
> > > strp_read_sock()
> > > tcp_read_sock()
> > > __tcp_read_sock()
> > > strp_recv()
> > > __strp_recv()
> > > head = strp->skb_head;
> > > strp_start_timer()
> > > mod_delayed_work(&strp->msg_timer_work);
> > > ...
> > > release_sock();
> > > cancel_delayed_work_sync(&strp->msg_timer_work);
> > > kfree_skb(strp->skb_head);
> > > ```
> > > In both cases, the race does not appear to cause any problem.
> > >
> > > >
> > > > @@ -503,6 +503,10 @@ void strp_done(struct strparser *strp)
> > > > {
> > > > WARN_ON(!strp->stopped);
> > > >
> > > > + lock_sock(strp->sk);
> > > > + /* sync with other code */
> > > > + release_sock(strp->sk);
> > > > +
> > > > cancel_delayed_work_sync(&strp->msg_timer_work);
> > > > cancel_work_sync(&strp->work);
> > > >
> > > >
> > > >
> > > > - strp->stopped so any new call into strp_data_ready will not do anything
> > > >
> > > > - lock/release need to take bh_lock_sock so any existing call to
> > > > strp_data_ready will have to complete before we move on to cancel*_work
> > > >
> > > >
> > > >
> > > > Or maybe the requirement should be that strp_stop has to be called
> > >
> > > From my perspective, adding lock_sock() inside strp_done(), as in the
> > > patch above, looks cleaner.
> > >
> > > > under lock_sock() (or even just bh_lock_sock), but again I can't
> > > > figure out if that's ok for sockmap.
> > >
> > > sockmap/psock has a more complex call stack compared to other strp
> > > users, so I'm also not entirely certain about that part.
> >
> > I looked into the sockmap/psock side. sk_psock_strp_data_ready() is protected
> > by read_lock_bh(&sk->sk_callback_lock), and during teardown sk_psock_drop()
> > performs callback restoration and strp_stop() under
> > write_lock_bh(&sk->sk_callback_lock), so sockmap/psock doesn't have this race
> > to begin with.
> >
> > As for introducing this patch, in sockmap/psock strp_done() is only called
> > from sk_psock_destroy(), which is scheduled via queue_rcu_work() and runs
> > on system_percpu_wq after an RCU GP, so no locks including lock_sock are held
> > at that point. And since lock_sock is released before cancel_work_sync,
> > there's no circular dependency with do_strp_work/strp_msg_timeout either.
> > So this patch shouldn't introduce any new issues for sockmap/psock.
>
> Hi Sabrina,
>
> Could you please provide an update on the status of this patch?
>
>
> Best regards,
> Hyunwoo Kim
Dear Sabrina,
This issue has been public for some time and seems to be gradually
fading from attention.
Could you please let me know the current status of the work on this?
Best regards,
Hyunwoo Kim
>
> >
> > >
> > > >
> > > >
> > > > > With that in mind, perhaps the direct fix for this race could be handled
> > > > > in the espintcp callback restoration patch. For the strparser patch, I
> > > > > could instead adjust the commit message to reflect that it removes a
> > > > > potential hazard by replacing the _cancel APIs with the _disable
> > > > > variants, and resubmit it in that form.
> > > >
> > > > I'm not going to nack a patch doing s/cancel_/disable_/ in strp_done,
> > > > but it doesn't fully solve the race condition if the caller isn't
> > > > doing the right thing, and it doesn't do anything if the strp user is
> > > > handling the teardown correctly.
> > >
> > > I agree with your point there. Still, after the core patches addressing
> > > this race are applied, I plan to resubmit the _disable patch with an
> > > updated commit message. I think applying that change is still beneficial.
> > >
> > >
> > > Best regards,
> > > Hyunwoo Kim
^ permalink raw reply
* Re: [PATCH 0/2] netfilter: fix NULL ops race in iptable lazy init
From: Phil Sutter @ 2026-04-29 18:17 UTC (permalink / raw)
To: Tristan Madani
Cc: Pablo Neira Ayuso, Florian Westphal, netfilter-devel, netdev,
stable, linux-kernel, Tristan Madani
In-Reply-To: <20260429175613.1459342-1-tristmd@gmail.com>
Hi,
On Wed, Apr 29, 2026 at 05:56:10PM +0000, Tristan Madani wrote:
> From: Tristan Madani <tristan@talencesecurity.com>
>
> ipt_register_table() and ip6t_register_table() call xt_register_table()
> which adds the new table to the per-netns list, making it visible to
> other code paths. Only afterwards do they allocate the per-net copy of
> hook ops via kmemdup_array(). This leaves a window where the table is
> findable via xt_find_table() but has ops=NULL.
>
> If cleanup_net runs during this window (racing namespace teardown against
> lazy table init), ipt_unregister_table_pre_exit() /
> ip6t_unregister_table_pre_exit() finds the table and passes the NULL ops
> pointer to nf_unregister_net_hooks(), causing a general protection fault.
>
> Fix both ip_tables.c and ip6_tables.c by moving the ops allocation
> before xt_register_table(), so the table is never in the list with a
> NULL ops pointer.
Is this true? Your patch moves the ops allocation, but new_table->ops is
still assigned after xt_register_table() has returned. AIUI, the race
window is just reduced, not eliminated.
First I thought you could assign to table->ops since xt_register_table()
calls kmemdup(), but 'table' is const.
I guess checking table->ops value in *_pre_exit() is nonsense as well
since *_register_table() still runs in parallel. Do we need
serialization between the two routines?
Cheers, Phil
^ permalink raw reply
* [PATCH v13 6/6] selftests: net: add TLS hardware offload test
From: Rishikesh Jethwani @ 2026-04-29 18:10 UTC (permalink / raw)
To: netdev
Cc: saeedm, tariqt, mbloch, borisp, john.fastabend, kuba, sd, davem,
pabeni, edumazet, leon, Rishikesh Jethwani
In-Reply-To: <20260429181016.3164935-1-rjethwani@purestorage.com>
Two-node kTLS hardware offload test using NetDrvEpEnv. Tests TLS
1.2/1.3 with AES-GCM-128/256, rekey operations, and various buffer
sizes.
Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
MAINTAINERS | 2 +
.../selftests/drivers/net/hw/.gitignore | 1 +
.../testing/selftests/drivers/net/hw/Makefile | 2 +
.../selftests/drivers/net/hw/tls_hw_offload.c | 887 ++++++++++++++++++
.../drivers/net/hw/tls_hw_offload.py | 256 +++++
5 files changed, 1148 insertions(+)
create mode 100644 tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
create mode 100755 tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
diff --git a/MAINTAINERS b/MAINTAINERS
index 2fb1c75afd16..aedf42890094 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18776,6 +18776,8 @@ F: Documentation/networking/tls*
F: include/net/tls.h
F: include/uapi/linux/tls.h
F: net/tls/
+F: tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
+F: tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
F: tools/testing/selftests/net/tls.c
NETWORKING [SOCKETS]
diff --git a/tools/testing/selftests/drivers/net/hw/.gitignore b/tools/testing/selftests/drivers/net/hw/.gitignore
index 46540468a775..f0a5d15b469b 100644
--- a/tools/testing/selftests/drivers/net/hw/.gitignore
+++ b/tools/testing/selftests/drivers/net/hw/.gitignore
@@ -2,3 +2,4 @@
iou-zcrx
ncdevmem
toeplitz
+tls_hw_offload
diff --git a/tools/testing/selftests/drivers/net/hw/Makefile b/tools/testing/selftests/drivers/net/hw/Makefile
index 85ca4d1ecf9e..2dd633619f40 100644
--- a/tools/testing/selftests/drivers/net/hw/Makefile
+++ b/tools/testing/selftests/drivers/net/hw/Makefile
@@ -15,6 +15,7 @@ endif
TEST_GEN_FILES := \
$(COND_GEN_FILES) \
+ tls_hw_offload \
# end of TEST_GEN_FILES
TEST_PROGS = \
@@ -43,6 +44,7 @@ TEST_PROGS = \
rss_drv.py \
rss_flow_label.py \
rss_input_xfrm.py \
+ tls_hw_offload.py \
toeplitz.py \
tso.py \
uso.py \
diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
new file mode 100644
index 000000000000..db7c61d8b4e7
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
@@ -0,0 +1,887 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * TLS Hardware Offload Two-Node Test
+ *
+ * Tests kTLS hardware offload between two physical nodes using
+ * hardcoded keys. Supports TLS 1.2/1.3, AES-GCM-128/256, and rekey.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <time.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+#include <arpa/inet.h>
+#include <linux/tls.h>
+
+#define TLS_RECORD_TYPE_HANDSHAKE 22
+#define TLS_HANDSHAKE_KEY_UPDATE 0x18
+
+#define MIN_BUF_SIZE 16
+
+/* Initial key material */
+static struct tls12_crypto_info_aes_gcm_128 tls_info_key0_128 = {
+ .info = {
+ .version = TLS_1_3_VERSION,
+ .cipher_type = TLS_CIPHER_AES_GCM_128,
+ },
+ .iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },
+ .key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+ 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10 },
+ .salt = { 0x01, 0x02, 0x03, 0x04 },
+ .rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
+};
+
+static struct tls12_crypto_info_aes_gcm_256 tls_info_key0_256 = {
+ .info = {
+ .version = TLS_1_3_VERSION,
+ .cipher_type = TLS_CIPHER_AES_GCM_256,
+ },
+ .iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },
+ .key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+ 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10,
+ 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18,
+ 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20 },
+ .salt = { 0x01, 0x02, 0x03, 0x04 },
+ .rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
+};
+
+static int num_rekeys;
+static int num_iterations = 100;
+static int cipher_type = TLS_CIPHER_AES_GCM_128;
+static int tls_version = TLS_1_3_VERSION;
+static int server_port = 4433;
+static char *server_ip;
+
+static int send_size = 16384;
+static int random_size_max;
+/* Burst mode: sender keeps pushing records without reading from the peer;
+ * receiver drains without echoing back. Only the client initiates rekey.
+ */
+static int burst_mode;
+static int zc_rx;
+
+/* XOR each byte with the generation so both endpoints derive the
+ * same per-generation key without a real KDF. Generation 0 leaves
+ * the base key unchanged.
+ */
+static void derive_key_fields(unsigned char *key, int key_size,
+ unsigned char *iv, int iv_size,
+ unsigned char *salt, int salt_size,
+ unsigned char *rec_seq, int rec_seq_size,
+ int generation)
+{
+ int i;
+
+ for (i = 0; i < key_size; i++)
+ key[i] ^= generation;
+ for (i = 0; i < iv_size; i++)
+ iv[i] ^= generation;
+ for (i = 0; i < salt_size; i++)
+ salt[i] ^= generation;
+ memset(rec_seq, 0, rec_seq_size);
+}
+
+static void derive_key_128(struct tls12_crypto_info_aes_gcm_128 *key,
+ int generation)
+{
+ memcpy(key, &tls_info_key0_128, sizeof(*key));
+ key->info.version = tls_version;
+ derive_key_fields(key->key, TLS_CIPHER_AES_GCM_128_KEY_SIZE,
+ key->iv, TLS_CIPHER_AES_GCM_128_IV_SIZE,
+ key->salt, TLS_CIPHER_AES_GCM_128_SALT_SIZE,
+ key->rec_seq, TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE,
+ generation);
+}
+
+static void derive_key_256(struct tls12_crypto_info_aes_gcm_256 *key,
+ int generation)
+{
+ memcpy(key, &tls_info_key0_256, sizeof(*key));
+ key->info.version = tls_version;
+ derive_key_fields(key->key, TLS_CIPHER_AES_GCM_256_KEY_SIZE,
+ key->iv, TLS_CIPHER_AES_GCM_256_IV_SIZE,
+ key->salt, TLS_CIPHER_AES_GCM_256_SALT_SIZE,
+ key->rec_seq, TLS_CIPHER_AES_GCM_256_REC_SEQ_SIZE,
+ generation);
+}
+
+static const char *cipher_name(int cipher)
+{
+ switch (cipher) {
+ case TLS_CIPHER_AES_GCM_128: return "AES-GCM-128";
+ case TLS_CIPHER_AES_GCM_256: return "AES-GCM-256";
+ default: return "unknown";
+ }
+}
+
+static const char *version_name(int version)
+{
+ switch (version) {
+ case TLS_1_2_VERSION: return "TLS 1.2";
+ case TLS_1_3_VERSION: return "TLS 1.3";
+ default: return "unknown";
+ }
+}
+
+static int setup_tls_ulp(int fd)
+{
+ int ret;
+
+ ret = setsockopt(fd, IPPROTO_TCP, TCP_ULP, "tls", sizeof("tls"));
+ if (ret < 0) {
+ printf("SETUP ERROR: TCP_ULP failed: %s\n", strerror(errno));
+ return -1;
+ }
+ return 0;
+}
+
+static int set_zc_rx(int fd)
+{
+ int val = 1;
+
+ if (setsockopt(fd, SOL_TLS, TLS_RX_EXPECT_NO_PAD, &val,
+ sizeof(val)) < 0) {
+ printf("SETUP ERROR: TLS_RX_EXPECT_NO_PAD failed: %s\n",
+ strerror(errno));
+ return -1;
+ }
+ return 0;
+}
+
+/* Send a TLS 1.3 KeyUpdate handshake record. The kernel only
+ * inspects the HandshakeType byte to detect KeyUpdate, so don't
+ * bother with the 3-byte length or request_update fields.
+ */
+static int send_tls_key_update(int fd)
+{
+ char cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];
+ unsigned char key_update_msg = TLS_HANDSHAKE_KEY_UPDATE;
+ struct msghdr msg = {0};
+ struct cmsghdr *cmsg;
+ struct iovec iov;
+
+ iov.iov_base = &key_update_msg;
+ iov.iov_len = sizeof(key_update_msg);
+
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+ msg.msg_control = cmsg_buf;
+ msg.msg_controllen = sizeof(cmsg_buf);
+
+ cmsg = CMSG_FIRSTHDR(&msg);
+ cmsg->cmsg_level = SOL_TLS;
+ cmsg->cmsg_type = TLS_SET_RECORD_TYPE;
+ cmsg->cmsg_len = CMSG_LEN(sizeof(unsigned char));
+ *CMSG_DATA(cmsg) = TLS_RECORD_TYPE_HANDSHAKE;
+ msg.msg_controllen = cmsg->cmsg_len;
+
+ if (sendmsg(fd, &msg, 0) < 0) {
+ printf("sendmsg KeyUpdate failed: %s\n", strerror(errno));
+ return -1;
+ }
+
+ printf("Sent TLS KeyUpdate handshake message\n");
+ return 0;
+}
+
+static int recv_tls_message(int fd, char *buf, size_t buflen, int *record_type)
+{
+ char cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];
+ struct msghdr msg = {0};
+ struct cmsghdr *cmsg;
+ struct iovec iov;
+ int ret;
+
+ iov.iov_base = buf;
+ iov.iov_len = buflen;
+
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+ msg.msg_control = cmsg_buf;
+ msg.msg_controllen = sizeof(cmsg_buf);
+
+ ret = recvmsg(fd, &msg, 0);
+ if (ret <= 0)
+ return ret;
+
+ cmsg = CMSG_FIRSTHDR(&msg);
+ if (cmsg && cmsg->cmsg_level == SOL_TLS &&
+ cmsg->cmsg_type == TLS_GET_RECORD_TYPE)
+ *record_type = *((unsigned char *)CMSG_DATA(cmsg));
+
+ return ret;
+}
+
+/* Confirm a handshake record starting with HandshakeType KeyUpdate. */
+static int check_keyupdate(const char *buf, int len, int record_type)
+{
+ if (record_type != TLS_RECORD_TYPE_HANDSHAKE) {
+ printf("Expected handshake record (0x%02x), got 0x%02x\n",
+ TLS_RECORD_TYPE_HANDSHAKE, record_type);
+ return -1;
+ }
+ if (len < 1 || (unsigned char)buf[0] != TLS_HANDSHAKE_KEY_UPDATE) {
+ printf("Expected KeyUpdate (0x%02x), got 0x%02x\n",
+ TLS_HANDSHAKE_KEY_UPDATE,
+ len ? (unsigned char)buf[0] : 0);
+ return -1;
+ }
+ printf("Received TLS KeyUpdate\n");
+ return 0;
+}
+
+static int recv_tls_keyupdate(int fd)
+{
+ char buf[MIN_BUF_SIZE];
+ int record_type = 0;
+ int ret;
+
+ ret = recv_tls_message(fd, buf, sizeof(buf), &record_type);
+ if (ret < 0) {
+ printf("recv_tls_message failed: %s\n", strerror(errno));
+ return -1;
+ }
+
+ return check_keyupdate(buf, ret, record_type);
+}
+
+static int check_ekeyexpired(int fd)
+{
+ char buf[MIN_BUF_SIZE];
+ int ret;
+
+ ret = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
+ if (ret == -1 && errno == EKEYEXPIRED) {
+ printf("recv() returned EKEYEXPIRED as expected\n");
+ return 0;
+ } else if (ret == -1 && errno == EAGAIN) {
+ printf("recv() returned EAGAIN (no pending data)\n");
+ return 0;
+ } else if (ret > 0) {
+ printf("FAIL: recv() returned %d bytes, expected EKEYEXPIRED\n",
+ ret);
+ return -1;
+ } else {
+ printf("FAIL: recv() returned unexpected error: %s\n",
+ strerror(errno));
+ return -1;
+ }
+}
+
+static int do_tls_rekey(int fd, int direction, int generation, int cipher)
+{
+ const char *dir = direction == TLS_TX ? "TX" : "RX";
+ int ret;
+
+ printf("%s TLS_%s %s gen %d...\n",
+ generation ? "Rekeying" : "Installing",
+ dir, cipher_name(cipher), generation);
+
+ if (cipher == TLS_CIPHER_AES_GCM_256) {
+ struct tls12_crypto_info_aes_gcm_256 key;
+
+ derive_key_256(&key, generation);
+ ret = setsockopt(fd, SOL_TLS, direction, &key, sizeof(key));
+ } else {
+ struct tls12_crypto_info_aes_gcm_128 key;
+
+ derive_key_128(&key, generation);
+ ret = setsockopt(fd, SOL_TLS, direction, &key, sizeof(key));
+ }
+
+ if (ret < 0) {
+ printf("%sTLS_%s %s gen %d failed: %s\n",
+ generation ? "" : "SETUP ERROR: ", dir,
+ cipher_name(cipher), generation, strerror(errno));
+ return -1;
+ }
+ printf("TLS_%s %s gen %d installed\n",
+ dir, cipher_name(cipher), generation);
+ return 0;
+}
+
+static int do_client(void)
+{
+ char *buf = NULL, *echo_buf = NULL;
+ int max_size, rekey_interval;
+ ssize_t echo_total, echo_n;
+ int csk = -1, ret, i, j;
+ struct sockaddr_in sa;
+ int test_result = -1;
+ int current_gen = 0;
+ int next_rekey_at;
+ ssize_t n;
+
+ max_size = random_size_max > 0 ? random_size_max : send_size;
+ if (max_size < MIN_BUF_SIZE)
+ max_size = MIN_BUF_SIZE;
+ buf = malloc(max_size);
+ if (!burst_mode)
+ echo_buf = malloc(max_size);
+ if (!buf || (!burst_mode && !echo_buf)) {
+ printf("SETUP ERROR: failed to allocate buffers\n");
+ goto out;
+ }
+
+ csk = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
+ if (csk < 0) {
+ printf("SETUP ERROR: failed to create socket: %s\n",
+ strerror(errno));
+ goto out;
+ }
+
+ memset(&sa, 0, sizeof(sa));
+ sa.sin_family = AF_INET;
+ sa.sin_addr.s_addr = inet_addr(server_ip);
+ sa.sin_port = htons(server_port);
+ printf("Connecting to %s:%d...\n", server_ip, server_port);
+
+ ret = connect(csk, (struct sockaddr *)&sa, sizeof(sa));
+ if (ret < 0) {
+ printf("SETUP ERROR: connect failed: %s\n", strerror(errno));
+ goto out;
+ }
+ printf("Connected!\n");
+
+ if (setup_tls_ulp(csk) < 0)
+ goto out;
+
+ if (do_tls_rekey(csk, TLS_TX, 0, cipher_type) < 0 ||
+ do_tls_rekey(csk, TLS_RX, 0, cipher_type) < 0)
+ goto out;
+
+ if (num_rekeys)
+ printf("TLS %s setup complete. Will perform %d rekey(s).\n",
+ cipher_name(cipher_type), num_rekeys);
+ else
+ printf("TLS setup complete.\n");
+
+ if (random_size_max > 0)
+ printf("Sending %d messages of random size (1..%d bytes)...\n",
+ num_iterations, random_size_max);
+ else
+ printf("Sending %d messages of %d bytes...\n",
+ num_iterations, send_size);
+
+ rekey_interval = num_iterations / (num_rekeys + 1);
+ next_rekey_at = rekey_interval;
+
+ for (i = 1; i <= num_iterations; i++) {
+ int this_size;
+
+ if (random_size_max > 0)
+ this_size = (rand() % random_size_max) + 1;
+ else
+ this_size = send_size;
+
+ /* In burst mode, use a per-iteration fill pattern so the
+ * receiver can detect any plaintext corruption without a
+ * round-trip echo.
+ */
+ if (burst_mode) {
+ memset(buf, i & 0xFF, this_size);
+ } else {
+ for (j = 0; j < this_size; j++)
+ buf[j] = rand() & 0xFF;
+ }
+
+ n = send(csk, buf, this_size, 0);
+ if (n != this_size) {
+ printf("FAIL: send failed: %s\n", strerror(errno));
+ goto out;
+ }
+ /* Throttle per-iteration progress lines on long burst runs so
+ * stdout over ssh doesn't become the bottleneck.
+ */
+ if (!burst_mode || num_iterations <= 1000 || (i % 1000) == 0 ||
+ i == num_iterations)
+ printf("Sent %zd bytes (iteration %d)\n", n, i);
+
+ if (!burst_mode) {
+ echo_total = 0;
+ while (echo_total < n) {
+ echo_n = recv(csk, echo_buf + echo_total,
+ n - echo_total, 0);
+ if (echo_n < 0) {
+ printf("FAIL: Echo recv failed: %s\n",
+ strerror(errno));
+ goto out;
+ }
+ if (echo_n == 0) {
+ printf("FAIL: Connection closed during echo\n");
+ goto out;
+ }
+ echo_total += echo_n;
+ }
+
+ if (memcmp(buf, echo_buf, n) != 0) {
+ printf("FAIL: Echo data mismatch!\n");
+ goto out;
+ }
+ printf("Received echo %zd bytes (ok)\n", echo_total);
+ }
+
+ /* Rekey at intervals. In echo mode this is a full bidirectional
+ * exchange; in burst mode the client only rotates its TX key
+ * and sends KeyUpdate - the peer is expected to follow.
+ */
+ if (num_rekeys && current_gen < num_rekeys &&
+ i == next_rekey_at) {
+ current_gen++;
+ printf("\n=== Client Rekey gen %d ===\n", current_gen);
+
+ ret = send_tls_key_update(csk);
+ if (ret < 0) {
+ printf("FAIL: send KeyUpdate\n");
+ goto out;
+ }
+
+ ret = do_tls_rekey(csk, TLS_TX, current_gen, cipher_type);
+ if (ret < 0)
+ goto out;
+
+ if (!burst_mode) {
+ if (recv_tls_keyupdate(csk) < 0) {
+ printf("FAIL: recv KeyUpdate from server\n");
+ goto out;
+ }
+
+ if (check_ekeyexpired(csk) < 0)
+ goto out;
+
+ ret = do_tls_rekey(csk, TLS_RX, current_gen,
+ cipher_type);
+ if (ret < 0)
+ goto out;
+ }
+
+ next_rekey_at += rekey_interval;
+ printf("=== Client Rekey gen %d Complete ===\n\n",
+ current_gen);
+ }
+ }
+
+ test_result = 0;
+out:
+ if (num_rekeys)
+ printf("Rekeys completed: %d/%d\n", current_gen, num_rekeys);
+ if (csk >= 0)
+ close(csk);
+ free(buf);
+ free(echo_buf);
+ return test_result;
+}
+
+static int do_server(void)
+{
+ int lsk = -1, csk = -1, ret;
+ ssize_t n, total = 0, sent;
+ /* Burst-mode data verification state: client fills each iteration's
+ * send_size-byte block with (send_iter & 0xff). A single recv may
+ * return part of a block or span iterations, so track position as
+ * (send_iter, remaining) - bytes left in the current block.
+ */
+ int send_iter = 1;
+ int remaining = send_size;
+ struct sockaddr_in sa;
+ int test_result = -1;
+ int current_gen = 0;
+ int recv_count = 0;
+ char *buf = NULL;
+ int record_type;
+ int buf_size;
+ int one = 1;
+
+ buf_size = send_size;
+ if (buf_size < MIN_BUF_SIZE)
+ buf_size = MIN_BUF_SIZE;
+ buf = malloc(buf_size);
+ if (!buf) {
+ printf("SETUP ERROR: failed to allocate buffer\n");
+ goto out;
+ }
+
+ lsk = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
+ if (lsk < 0) {
+ printf("SETUP ERROR: failed to create socket: %s\n",
+ strerror(errno));
+ goto out;
+ }
+
+ setsockopt(lsk, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
+
+ memset(&sa, 0, sizeof(sa));
+ sa.sin_family = AF_INET;
+ sa.sin_addr.s_addr = INADDR_ANY;
+ sa.sin_port = htons(server_port);
+
+ ret = bind(lsk, (struct sockaddr *)&sa, sizeof(sa));
+ if (ret < 0) {
+ printf("SETUP ERROR: bind failed: %s\n", strerror(errno));
+ goto out;
+ }
+
+ ret = listen(lsk, 1);
+ if (ret < 0) {
+ printf("SETUP ERROR: listen failed: %s\n", strerror(errno));
+ goto out;
+ }
+
+ printf("Server listening on 0.0.0.0:%d\n", server_port);
+ printf("Waiting for client connection...\n");
+
+ csk = accept(lsk, (struct sockaddr *)NULL, (socklen_t *)NULL);
+ if (csk < 0) {
+ printf("SETUP ERROR: accept failed: %s\n", strerror(errno));
+ goto out;
+ }
+ printf("Client connected!\n");
+
+ if (setup_tls_ulp(csk) < 0)
+ goto out;
+
+ if (do_tls_rekey(csk, TLS_TX, 0, cipher_type) < 0 ||
+ do_tls_rekey(csk, TLS_RX, 0, cipher_type) < 0)
+ goto out;
+
+ if (zc_rx && set_zc_rx(csk) < 0)
+ goto out;
+
+ printf("TLS %s setup complete. Receiving...\n",
+ cipher_name(cipher_type));
+
+ /* Main receive loop */
+ while (1) {
+ n = recv_tls_message(csk, buf, buf_size, &record_type);
+ if (n == 0) {
+ printf("Connection closed by client\n");
+ break;
+ }
+ if (n < 0) {
+ printf("FAIL: recv failed: %s\n", strerror(errno));
+ goto out;
+ }
+
+ /* Handle KeyUpdate. In echo mode the server mirrors the
+ * rekey back to the peer; in burst mode it only rotates
+ * its RX key and keeps draining.
+ */
+ if (record_type == TLS_RECORD_TYPE_HANDSHAKE) {
+ if (check_keyupdate(buf, n, record_type) < 0)
+ goto out;
+ current_gen++;
+ printf("\n=== Server Rekey gen %d ===\n", current_gen);
+
+ if (check_ekeyexpired(csk) < 0)
+ goto out;
+
+ ret = do_tls_rekey(csk, TLS_RX, current_gen, cipher_type);
+ if (ret < 0)
+ goto out;
+
+ if (!burst_mode) {
+ ret = send_tls_key_update(csk);
+ if (ret < 0) {
+ printf("FAIL: send KeyUpdate\n");
+ goto out;
+ }
+
+ ret = do_tls_rekey(csk, TLS_TX, current_gen,
+ cipher_type);
+ if (ret < 0)
+ goto out;
+ }
+
+ printf("=== Server Rekey gen %d Complete ===\n\n",
+ current_gen);
+ continue;
+ }
+
+ total += n;
+ recv_count++;
+ /* Throttle per-record progress lines on long burst runs. */
+ if (!burst_mode || (recv_count % 1000) == 0)
+ printf("Received %zd bytes (total: %zd, count: %d)\n",
+ n, total, recv_count);
+
+ /* Burst mode: verify payload matches the client's fill
+ * pattern. TLS record boundaries may differ from send()
+ * boundaries, so walk the received buffer in chunks that
+ * fit within the current iteration's remaining bytes.
+ * Catches decrypt-succeeded-but-plaintext-corrupt bugs
+ * that AEAD counters alone would miss.
+ */
+ if (burst_mode) {
+ int off = 0;
+
+ while (off < n) {
+ unsigned char expect = send_iter & 0xFF;
+ int chunk = n - off;
+ int j;
+
+ if (chunk > remaining)
+ chunk = remaining;
+
+ for (j = 0; j < chunk; j++) {
+ if ((unsigned char)buf[off + j] != expect) {
+ printf("FAIL: data mismatch recv #%d offset %d:"
+ " expected 0x%02x got 0x%02x"
+ " (iter %d, remaining %d)\n",
+ recv_count, off + j,
+ expect,
+ (unsigned char)buf[off + j],
+ send_iter, remaining);
+ goto out;
+ }
+ }
+
+ off += chunk;
+ remaining -= chunk;
+ if (remaining == 0) {
+ send_iter++;
+ remaining = send_size;
+ }
+ }
+ continue;
+ }
+
+ for (sent = 0; sent < n; sent += ret) {
+ ret = send(csk, buf + sent, n - sent, 0);
+ if (ret < 0) {
+ printf("FAIL: Echo send failed: %s\n",
+ strerror(errno));
+ goto out;
+ }
+ }
+ printf("Echoed %zd bytes back to client\n", n);
+ }
+
+ test_result = 0;
+out:
+ printf("Connection closed. Total received: %zd bytes\n", total);
+ if (num_rekeys)
+ printf("Rekeys completed: %d\n", current_gen);
+
+ if (csk >= 0)
+ close(csk);
+ if (lsk >= 0)
+ close(lsk);
+ free(buf);
+ return test_result;
+}
+
+static int parse_cipher_option(const char *arg)
+{
+ if (strcmp(arg, "128") == 0) {
+ cipher_type = TLS_CIPHER_AES_GCM_128;
+ return 0;
+ } else if (strcmp(arg, "256") == 0) {
+ cipher_type = TLS_CIPHER_AES_GCM_256;
+ return 0;
+ }
+ printf("ERROR: Invalid cipher '%s'. Must be 128 or 256.\n", arg);
+ return -1;
+}
+
+static int parse_version_option(const char *arg)
+{
+ if (strcmp(arg, "1.2") == 0) {
+ tls_version = TLS_1_2_VERSION;
+ return 0;
+ } else if (strcmp(arg, "1.3") == 0) {
+ tls_version = TLS_1_3_VERSION;
+ return 0;
+ }
+ printf("ERROR: Invalid TLS version '%s'. Must be 1.2 or 1.3.\n", arg);
+ return -1;
+}
+
+static void print_usage(const char *prog)
+{
+ printf("TLS Hardware Offload Two-Node Test\n\n");
+ printf("Usage:\n");
+ printf(" %s server [OPTIONS]\n", prog);
+ printf(" %s client -s <ip> [OPTIONS]\n", prog);
+ printf("\nOptions:\n");
+ printf(" -s <ip> Server IPv4 address (client, required)\n");
+ printf(" -p <port> Server port (default: 4433)\n");
+ printf(" -b <size> Send buffer size in bytes (default: 16384)\n");
+ printf(" -r <max> Use random send buffer sizes (1..<max>)\n");
+ printf(" -v <version> TLS version: 1.2 or 1.3 (default: 1.3)\n");
+ printf(" -c <cipher> Cipher: 128 or 256 (default: 128)\n");
+ printf(" -n <N> Number of send/echo iterations (default: 100)\n");
+ printf(" -k <N> Perform N rekeys (client only, TLS 1.3; N < iterations)\n");
+ printf(" -B Burst mode: client sends continuously without echo;\n");
+ printf(" server drains and handles KeyUpdate without responding.\n");
+ printf(" -Z Enable zero-copy RX (TLS_RX_EXPECT_NO_PAD);\n");
+ printf(" server only, TLS 1.3 only.\n");
+ printf(" -h Show this help message\n");
+ printf("\nExample:\n");
+ printf(" Node A: %s server\n", prog);
+ printf(" Node B: %s client -s 192.168.20.2\n", prog);
+ printf("\nRekey Example (3 rekeys, TLS 1.3 only):\n");
+ printf(" Node A: %s server\n", prog);
+ printf(" Node B: %s client -s 192.168.20.2 -k 3\n", prog);
+ printf("\nBurst Mode Example (client stresses TX rekey under load):\n");
+ printf(" Node A: %s server -B\n", prog);
+ printf(" Node B: %s client -s 192.168.20.2 -B -k 3\n", prog);
+}
+
+int main(int argc, char *argv[])
+{
+ int send_size_set = 0;
+ int is_server;
+ int opt;
+
+ if (argc < 2 ||
+ (strcmp(argv[1], "server") && strcmp(argv[1], "client"))) {
+ print_usage(argv[0]);
+ return 1;
+ }
+ is_server = !strcmp(argv[1], "server");
+
+ optind = 2; /* skip subcommand */
+ while ((opt = getopt(argc, argv, "s:p:b:r:c:v:k:n:BZh")) != -1) {
+ switch (opt) {
+ case 's':
+ server_ip = optarg;
+ break;
+ case 'B':
+ burst_mode = 1;
+ break;
+ case 'Z':
+ zc_rx = 1;
+ break;
+ case 'p':
+ server_port = atoi(optarg);
+ if (server_port < 1 || server_port > 65535) {
+ printf("ERROR: Invalid port '%s'. Must be 1..65535.\n",
+ optarg);
+ return 1;
+ }
+ break;
+ case 'b':
+ send_size = atoi(optarg);
+ if (send_size < 1) {
+ printf("ERROR: Invalid buffer size '%s'. Must be >= 1.\n",
+ optarg);
+ return 1;
+ }
+ send_size_set = 1;
+ break;
+ case 'r':
+ random_size_max = atoi(optarg);
+ if (random_size_max < 1) {
+ printf("ERROR: Invalid random size '%s'. Must be >= 1.\n",
+ optarg);
+ return 1;
+ }
+ break;
+ case 'c':
+ if (parse_cipher_option(optarg) < 0)
+ return 1;
+ break;
+ case 'v':
+ if (parse_version_option(optarg) < 0)
+ return 1;
+ break;
+ case 'k':
+ num_rekeys = atoi(optarg);
+ if (num_rekeys < 1) {
+ printf("ERROR: Invalid rekey count '%s'. Must be >= 1.\n",
+ optarg);
+ return 1;
+ }
+ break;
+ case 'n':
+ num_iterations = atoi(optarg);
+ if (num_iterations < 1) {
+ printf("ERROR: Invalid iteration count '%s'. Must be >= 1.\n",
+ optarg);
+ return 1;
+ }
+ break;
+ case 'h':
+ print_usage(argv[0]);
+ return 0;
+ default:
+ print_usage(argv[0]);
+ return 1;
+ }
+ }
+
+ if (send_size_set && random_size_max > 0) {
+ printf("ERROR: -b and -r are mutually exclusive\n");
+ return 1;
+ }
+
+ if (zc_rx && tls_version != TLS_1_3_VERSION) {
+ printf("ERROR: -Z (TLS_RX_EXPECT_NO_PAD) requires TLS 1.3\n");
+ return 1;
+ }
+
+ if (burst_mode && random_size_max > 0) {
+ printf("ERROR: -B and -r are mutually exclusive\n");
+ return 1;
+ }
+
+ if (is_server) {
+ if (server_ip) {
+ printf("warning: -s is ignored in server mode\n");
+ server_ip = NULL;
+ }
+ if (random_size_max > 0) {
+ printf("warning: -r is ignored in server mode\n");
+ random_size_max = 0;
+ }
+ if (num_rekeys) {
+ printf("warning: -k is ignored in server mode\n");
+ num_rekeys = 0;
+ }
+ } else {
+ if (!server_ip) {
+ printf("ERROR: Client requires -s <ip> option\n");
+ return 1;
+ }
+ if (tls_version == TLS_1_2_VERSION && num_rekeys) {
+ printf("ERROR: TLS 1.2 does not support rekey\n");
+ return 1;
+ }
+ if (num_rekeys >= num_iterations) {
+ printf("ERROR: num_rekeys (%d) must be < num_iterations (%d)\n",
+ num_rekeys, num_iterations);
+ return 1;
+ }
+ if (zc_rx) {
+ printf("ERROR: -Z applies to the server (receiver) only\n");
+ return 1;
+ }
+ }
+
+ printf("TLS Version: %s\n", version_name(tls_version));
+ printf("Cipher: %s\n", cipher_name(cipher_type));
+ if (random_size_max > 0)
+ printf("Buffer size: random (1..%d)\n", random_size_max);
+ else
+ printf("Buffer size: %d\n", send_size);
+
+ if (num_rekeys)
+ printf("Rekey testing ENABLED: %d rekey(s)\n", num_rekeys);
+ if (burst_mode)
+ printf("Burst mode ENABLED\n");
+ if (zc_rx)
+ printf("Zero-copy RX ENABLED\n");
+
+ srand(time(NULL));
+
+ if (is_server)
+ return do_server() ? 1 : 0;
+
+ return do_client() ? 1 : 0;
+}
diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
new file mode 100755
index 000000000000..f12da0e66afd
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
@@ -0,0 +1,256 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Test kTLS hardware offload using a C helper binary."""
+
+from collections import defaultdict
+
+from lib.py import ksft_run, ksft_exit, ksft_pr, KsftSkipEx, ksft_true
+from lib.py import ksft_variants, KsftNamedVariant
+from lib.py import NetDrvEpEnv
+from lib.py import cmd, bkg, wait_port_listen, rand_port
+
+
+def check_tls_support(cfg):
+ try:
+ cmd("test -f /proc/net/tls_stat")
+ cmd("test -f /proc/net/tls_stat", host=cfg.remote)
+ except Exception as e:
+ raise KsftSkipEx(f"kTLS not supported: {e}")
+
+
+def read_tls_stats(host=None):
+ stats = defaultdict(int)
+ output = cmd("cat /proc/net/tls_stat", host=host)
+ for line in output.stdout.strip().split('\n'):
+ parts = line.split()
+ if len(parts) == 2:
+ stats[parts[0]] = int(parts[1])
+ return stats
+
+
+def stat_diff(before, after, key):
+ return after[key] - before[key]
+
+
+def check_path(before, after, direction, role, require_hw):
+ """On the DUT, require HW offload; on the remote, HW or SW is fine."""
+ dev = stat_diff(before, after, f'Tls{direction}Device')
+ sw = stat_diff(before, after, f'Tls{direction}Sw')
+ if require_hw:
+ if dev < 1:
+ ksft_pr(f"FAIL: {role} {direction}: HW offload not engaged "
+ f"(Device={dev}, Sw={sw})")
+ return 1
+ elif dev < 1 and sw < 1:
+ ksft_pr(f"FAIL: {role} {direction}: no TLS activity "
+ f"(Device={dev}, Sw={sw})")
+ return 1
+ return 0
+
+
+def check_min(before, after, key, minimum, role):
+ diff = stat_diff(before, after, key)
+ if diff < minimum:
+ ksft_pr(f"FAIL: {role} {key}: expected >= {minimum}, got {diff}")
+ return 1
+ return 0
+
+
+def check_zero(before, after, key, role):
+ diff = stat_diff(before, after, key)
+ if diff > 0:
+ ksft_pr(f"FAIL: {role} {key} increased by {diff}")
+ return 1
+ return 0
+
+
+def verify_tls_counters(stats_before, stats_after, expected_rekeys,
+ tls_role, is_dut, burst=False):
+ """Verify TLS counters on one side of the connection.
+
+ tls_role: 'client' or 'server' (TLS role this side played).
+ is_dut: True for the local DUT; requires HW offload counters.
+ burst: burst mode - only the TLS client rotates its TX key; the TLS
+ server only follows with an RX rotation on KeyUpdate receipt.
+ """
+ errors = 0
+ role = 'DUT' if is_dut else 'Peer'
+
+ # In burst mode only one direction carries TLS traffic per side
+ # (TLS client sends, TLS server receives). Check HW offload only on
+ # the active direction(s); require HW on the DUT's active direction.
+ if burst:
+ if tls_role == 'client':
+ errors += check_path(stats_before, stats_after, 'Tx', role,
+ require_hw=is_dut)
+ else:
+ errors += check_path(stats_before, stats_after, 'Rx', role,
+ require_hw=is_dut)
+ else:
+ errors += check_path(stats_before, stats_after, 'Tx', role,
+ require_hw=is_dut)
+ errors += check_path(stats_before, stats_after, 'Rx', role,
+ require_hw=is_dut)
+
+ if expected_rekeys > 0:
+ if burst:
+ if tls_role == 'client':
+ errors += check_min(stats_before, stats_after,
+ 'TlsTxRekeyOk', expected_rekeys, role)
+ errors += check_zero(stats_before, stats_after,
+ 'TlsTxRekeyError', role)
+ else:
+ errors += check_min(stats_before, stats_after,
+ 'TlsRxRekeyOk', expected_rekeys, role)
+ errors += check_min(stats_before, stats_after,
+ 'TlsRxRekeyReceived', expected_rekeys,
+ role)
+ errors += check_zero(stats_before, stats_after,
+ 'TlsRxRekeyError', role)
+ else:
+ errors += check_min(stats_before, stats_after,
+ 'TlsTxRekeyOk', expected_rekeys, role)
+ errors += check_min(stats_before, stats_after,
+ 'TlsRxRekeyOk', expected_rekeys, role)
+ if tls_role == 'server':
+ errors += check_min(stats_before, stats_after,
+ 'TlsRxRekeyReceived', expected_rekeys,
+ role)
+ errors += check_zero(stats_before, stats_after,
+ 'TlsTxRekeyError', role)
+ errors += check_zero(stats_before, stats_after,
+ 'TlsRxRekeyError', role)
+
+ errors += check_zero(stats_before, stats_after, 'TlsDecryptError', role)
+
+ return errors == 0
+
+
+def run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=0,
+ buffer_size=None, random_max=None, burst=False, zc=False,
+ dut_role="client", num_iterations=None):
+ """Run the TLS offload test.
+
+ dut_role: 'client' (default) - DUT runs the TLS client, remote the server.
+ 'server' - swap: DUT listens, remote connects. Used for burst_rx
+ so the DUT's RX path is the one under rekey pressure.
+
+ The DUT (local) is the kernel under test; the remote is just a traffic
+ source/sink and may run any kernel without HW offload. Both sides run
+ kTLS because TLS is pairwise, but verify_tls_counters() requires HW
+ offload only on the DUT (is_dut=True); the peer may use SW kTLS.
+ """
+ port = rand_port()
+ send_size = random_max or buffer_size
+
+ if dut_role == "client":
+ server_bin, server_host = cfg.bin_remote, cfg.remote
+ client_bin, client_host = cfg.bin_local, None
+ client_target = cfg.remote_addr_v['4']
+ else:
+ server_bin, server_host = cfg.bin_local, None
+ client_bin, client_host = cfg.bin_remote, cfg.remote
+ client_target = cfg.addr_v['4']
+
+ server_cmd = f"{server_bin} server -p {port} -c {cipher} -v {tls_version}"
+ if burst:
+ server_cmd += " -B"
+ if zc:
+ server_cmd += " -Z"
+ if send_size:
+ server_cmd += f" -b {send_size}"
+
+ client_cmd = (f"{client_bin} client -s {client_target} "
+ f"-p {port} -c {cipher} -v {tls_version}")
+ if rekey:
+ client_cmd += f" -k {rekey}"
+ if burst:
+ client_cmd += " -B"
+ if num_iterations:
+ client_cmd += f" -n {num_iterations}"
+ if random_max:
+ client_cmd += f" -r {random_max}"
+ elif buffer_size:
+ client_cmd += f" -b {buffer_size}"
+
+ # Burst variants push hundreds of MB and perform many rekeys; the
+ # default cmd() timeout (5s) is too short.
+ cmd_timeout = 180 if burst else 5
+
+ stats_before_local = read_tls_stats()
+ stats_before_remote = read_tls_stats(host=cfg.remote)
+
+ with bkg(server_cmd, host=server_host, exit_wait=True):
+ wait_port_listen(port, host=server_host)
+ cmd(client_cmd, host=client_host, timeout=cmd_timeout)
+
+ stats_after_local = read_tls_stats()
+ stats_after_remote = read_tls_stats(host=cfg.remote)
+
+ dut_tls_role = dut_role
+ peer_tls_role = 'server' if dut_role == 'client' else 'client'
+
+ dut_ok = verify_tls_counters(stats_before_local, stats_after_local,
+ rekey, dut_tls_role, is_dut=True,
+ burst=burst)
+ peer_ok = verify_tls_counters(stats_before_remote, stats_after_remote,
+ rekey, peer_tls_role, is_dut=False,
+ burst=burst)
+
+ ksft_true(dut_ok, "DUT TLS counters verified")
+ ksft_true(peer_ok, "Peer TLS counters verified")
+
+
+@ksft_variants([
+ KsftNamedVariant("tls13_aes128", "128", "1.3"),
+ KsftNamedVariant("tls13_aes256", "256", "1.3"),
+ KsftNamedVariant("tls12_aes128", "128", "1.2"),
+ KsftNamedVariant("tls12_aes256", "256", "1.2"),
+])
+def test_tls_offload(cfg, cipher, tls_version):
+ run_tls_test(cfg, cipher=cipher, tls_version=tls_version)
+
+
+@ksft_variants([
+ KsftNamedVariant("single", 1),
+ KsftNamedVariant("multiple", 99),
+ KsftNamedVariant("small_buf", 30, 512),
+ KsftNamedVariant("large_buf", 10, 2097152),
+ KsftNamedVariant("random_buf", 20, None, 8192),
+])
+def test_tls_offload_rekey(cfg, rekey, buffer_size=None, random_max=None):
+ run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=rekey,
+ buffer_size=buffer_size, random_max=random_max)
+
+
+@ksft_variants([
+ KsftNamedVariant("burst_tx_rekey_every_1", "client", False, 1, 50, 65536),
+ KsftNamedVariant("burst_tx_rekey_every_1000", "client", False, 1000, 3, 65536),
+ KsftNamedVariant("burst_rx_rekey_every_10", "server", False, 10, 20, 65536),
+ KsftNamedVariant("burst_rx_rekey_every_10000", "server", False, 10000, 1, 32768),
+ KsftNamedVariant("burst_rx_zc_rekey_every_100", "server", True, 100, 10, 65536),
+ KsftNamedVariant("burst_rx_zc_rekey_every_20000", "server", True, 20000, 1, 16384),
+])
+def test_tls_offload_burst(cfg, dut_role, zc, interval, rekeys, buffer_size):
+ run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=rekeys,
+ buffer_size=buffer_size, burst=True, zc=zc, dut_role=dut_role,
+ num_iterations=interval * (rekeys + 1))
+
+
+def main() -> None:
+ with NetDrvEpEnv(__file__, nsim_test=False) as cfg:
+ cfg.bin_local = cfg.test_dir / "tls_hw_offload"
+ if not cfg.bin_local.exists():
+ raise KsftSkipEx(f"tls_hw_offload binary not found at {cfg.bin_local}")
+ cfg.bin_remote = cfg.remote.deploy(cfg.bin_local)
+ cfg.require_ipver("4")
+ check_tls_support(cfg)
+
+ ksft_run([test_tls_offload, test_tls_offload_rekey,
+ test_tls_offload_burst], args=(cfg, ))
+ ksft_exit()
+
+
+if __name__ == "__main__":
+ main()
--
2.25.1
^ permalink raw reply related
* [PATCH v13 5/6] tls: add hardware offload key update support
From: Rishikesh Jethwani @ 2026-04-29 18:10 UTC (permalink / raw)
To: netdev
Cc: saeedm, tariqt, mbloch, borisp, john.fastabend, kuba, sd, davem,
pabeni, edumazet, leon, Rishikesh Jethwani
In-Reply-To: <20260429181016.3164935-1-rjethwani@purestorage.com>
On TX, the NIC key cannot be replaced while HW-offloaded records
are still unacked. tls_dev_start_rekey() installs a temporary SW
context with the new key and redirects sendmsg through
tls_sw_sendmsg_locked. If no records are pending,
tls_dev_complete_rekey() runs inline during setsockopt; otherwise
clean_acked sets REKEY_READY once all old-key records are ACKed
and the next sendmsg completes the rekey, flushing SW records and
reinstalling HW offload at the current write_seq. A KeyUpdate
arriving while one is pending re-keys the SW AEAD in place; if the
HW reinstall fails the socket stays in SW mode (REKEY_FAILED).
On RX, the NIC may have already decrypted in-flight records with
the old key before the peer's KeyUpdate is parsed, so the old
AEAD, IV and rec_seq are retained on tls_offload_context_rx.
tls_check_pending_rekey() invokes tls_device_rx_del_key() to drop
the NIC key; otherwise post-KeyUpdate records (carrying new-key
wire encryption) would be XOR'd with the retired key.
tls_device_decrypted() classifies records by old_nic_boundary:
- after the boundary: new-key record; drop the old key.
- before, fully encrypted: advance old_rec_seq, let SW AEAD decrypt.
- before, (partially) decrypted: reencrypt with the old key so SW
AEAD can decrypt with the new key.
For mixed records skb->decrypted flags can be wrong (NIC clears
them on auth failure); on -EBADMSG, tls_rx_rekey_retry() toggles
those flags, decrements old_rec_seq to reuse the nonce, and
retries once (gated by old_key_reencrypted).
The new key's tls_dev_add is deferred until the old key is fully
consumed: tls_set_device_offload_rx() sets dev_add_pending while
old_aead_recv is retained, and tls_device_deferred_dev_add()
installs the new key once copied_seq crosses old_nic_boundary.
Tested on Mellanox ConnectX-6 Dx (Crypto Enabled) with multiple
TLS 1.3 TX and RX KeyUpdate cycles.
Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
include/net/tls.h | 84 +++-
include/uapi/linux/snmp.h | 2 +
net/tls/tls.h | 29 +-
net/tls/tls_device.c | 753 +++++++++++++++++++++++++++++++---
net/tls/tls_device_fallback.c | 24 ++
net/tls/tls_main.c | 92 +++--
net/tls/tls_proc.c | 2 +
net/tls/tls_sw.c | 76 +++-
net/tls/trace.h | 79 ++++
9 files changed, 992 insertions(+), 149 deletions(-)
diff --git a/include/net/tls.h b/include/net/tls.h
index ebd2550280ae..6891aa6b484c 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -151,6 +151,22 @@ struct tls_record_info {
skb_frag_t frags[MAX_SKB_FRAGS];
};
+struct cipher_context {
+ char iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
+ char rec_seq[TLS_MAX_REC_SEQ_SIZE];
+};
+
+union tls_crypto_context {
+ struct tls_crypto_info info;
+ union {
+ struct tls12_crypto_info_aes_gcm_128 aes_gcm_128;
+ struct tls12_crypto_info_aes_gcm_256 aes_gcm_256;
+ struct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;
+ struct tls12_crypto_info_sm4_gcm sm4_gcm;
+ struct tls12_crypto_info_sm4_ccm sm4_ccm;
+ };
+};
+
#define TLS_DRIVER_STATE_SIZE_TX 16
struct tls_offload_context_tx {
struct crypto_aead *aead_send;
@@ -165,6 +181,11 @@ struct tls_offload_context_tx {
void (*sk_destruct)(struct sock *sk);
struct work_struct destruct_work;
struct tls_context *ctx;
+
+ struct tls_sw_context_tx rekey_sw; /* SW context for new key */
+ struct cipher_context rekey_tx; /* IV, rec_seq for new key */
+ union tls_crypto_context rekey_crypto_send; /* Crypto for new key */
+
/* The TLS layer reserves room for driver specific state
* Currently the belief is that there is not enough
* driver specific state to justify another layer of indirection
@@ -189,22 +210,21 @@ enum tls_context_flags {
* tls_dev_del call in tls_device_down if it happens simultaneously.
*/
TLS_RX_DEV_CLOSED = 2,
-};
-
-struct cipher_context {
- char iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
- char rec_seq[TLS_MAX_REC_SEQ_SIZE];
-};
-
-union tls_crypto_context {
- struct tls_crypto_info info;
- union {
- struct tls12_crypto_info_aes_gcm_128 aes_gcm_128;
- struct tls12_crypto_info_aes_gcm_256 aes_gcm_256;
- struct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;
- struct tls12_crypto_info_sm4_gcm sm4_gcm;
- struct tls12_crypto_info_sm4_ccm sm4_ccm;
- };
+ /* Flag for TX HW context deleted during failed rekey.
+ * Prevents double tls_dev_del in cleanup paths.
+ */
+ TLS_TX_DEV_CLOSED = 3,
+ /* TX rekey is pending, waiting for old-key data to be ACKed.
+ * While set, new data uses SW path with new key, HW keeps old key
+ * for retransmissions.
+ */
+ TLS_TX_REKEY_PENDING = 4,
+ /* All old-key data has been ACKed, ready to install new key in HW. */
+ TLS_TX_REKEY_READY = 5,
+ /* HW rekey failed, permanently stay in SW encrypt mode.
+ * Prevents tls_tcp_clean_acked from re-setting TLS_TX_REKEY_READY.
+ */
+ TLS_TX_REKEY_FAILED = 6,
};
struct tls_prot_info {
@@ -253,6 +273,15 @@ struct tls_context {
*/
unsigned long flags;
+ /* TCP sequence number boundary for pending rekey.
+ * Packets with seq < this use old key, >= use new key.
+ */
+ u32 rekey_boundary_seq;
+
+ /* Pointers to rekey contexts for SW encryption with new key */
+ struct tls_sw_context_tx *rekey_sw_ctx;
+ struct cipher_context *rekey_cipher_ctx;
+
/* cache cold stuff */
struct proto *sk_proto;
struct sock *sk;
@@ -311,6 +340,14 @@ struct tls_offload_context_rx {
u8 resync_nh_reset:1;
/* CORE_NEXT_HINT-only member, but use the hole here */
u8 resync_nh_do_now:1;
+ /* retry reencrypt of mixed record during rekey */
+ u8 old_key_reencrypted:1;
+ /* tls_dev_add deferred until old key is freed */
+ u8 dev_add_pending:1;
+ struct crypto_aead *old_aead_recv; /* old key AEAD cipher */
+ char old_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE]; /* old key IV */
+ char old_rec_seq[TLS_MAX_REC_SEQ_SIZE]; /* old key TLS record seq */
+ u32 old_nic_boundary; /* TCP seq: NIC switched to next key */
union {
/* TLS_OFFLOAD_SYNC_TYPE_DRIVER_REQ */
struct {
@@ -385,9 +422,21 @@ static inline struct tls_sw_context_rx *tls_sw_ctx_rx(
static inline struct tls_sw_context_tx *tls_sw_ctx_tx(
const struct tls_context *tls_ctx)
{
+ if (unlikely(tls_ctx->rekey_sw_ctx))
+ return tls_ctx->rekey_sw_ctx;
+
return (struct tls_sw_context_tx *)tls_ctx->priv_ctx_tx;
}
+static inline struct cipher_context *tls_tx_cipher_ctx(
+ const struct tls_context *tls_ctx)
+{
+ if (unlikely(tls_ctx->rekey_cipher_ctx))
+ return tls_ctx->rekey_cipher_ctx;
+
+ return (struct cipher_context *)&tls_ctx->tx;
+}
+
static inline struct tls_offload_context_tx *
tls_offload_ctx_tx(const struct tls_context *tls_ctx)
{
@@ -500,6 +549,9 @@ struct sk_buff *tls_encrypt_skb(struct sk_buff *skb);
#ifdef CONFIG_TLS_DEVICE
void tls_device_sk_destruct(struct sock *sk);
void tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq);
+struct sk_buff *
+tls_validate_xmit_skb_rekey(struct sock *sk, struct net_device *dev,
+ struct sk_buff *skb);
static inline bool tls_is_sk_rx_device_offloaded(struct sock *sk)
{
diff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h
index 49f5640092a0..39fa48821faa 100644
--- a/include/uapi/linux/snmp.h
+++ b/include/uapi/linux/snmp.h
@@ -369,6 +369,8 @@ enum
LINUX_MIB_TLSTXREKEYOK, /* TlsTxRekeyOk */
LINUX_MIB_TLSTXREKEYERROR, /* TlsTxRekeyError */
LINUX_MIB_TLSRXREKEYRECEIVED, /* TlsRxRekeyReceived */
+ LINUX_MIB_TLSTXREKEYHWFAIL, /* TlsTxRekeyHwFail */
+ LINUX_MIB_TLSRXREKEYHWFAIL, /* TlsRxRekeyHwFail */
__LINUX_MIB_TLSMAX
};
diff --git a/net/tls/tls.h b/net/tls/tls.h
index a65cf9bab190..03d558e80f9a 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -157,6 +157,9 @@ void tls_update_rx_zc_capable(struct tls_context *tls_ctx);
void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx);
void tls_sw_strparser_done(struct tls_context *tls_ctx);
int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size);
+void tls_tx_work_handler(struct work_struct *work);
+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx);
void tls_sw_splice_eof(struct socket *sock);
void tls_sw_cancel_work_tx(struct tls_context *tls_ctx);
void tls_sw_release_resources_tx(struct sock *sk);
@@ -176,6 +179,8 @@ int tls_sw_read_sock(struct sock *sk, read_descriptor_t *desc,
int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
void tls_device_splice_eof(struct socket *sock);
int tls_tx_records(struct sock *sk, int flags);
+int tls_sw_push_pending_record(struct sock *sk, int flags);
+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx);
void tls_sw_write_space(struct sock *sk, struct tls_context *ctx);
void tls_device_write_space(struct sock *sk, struct tls_context *ctx);
@@ -233,10 +238,13 @@ static inline bool tls_strp_msg_mixed_decrypted(struct tls_sw_context_rx *ctx)
#ifdef CONFIG_TLS_DEVICE
int tls_device_init(void);
void tls_device_cleanup(void);
-int tls_set_device_offload(struct sock *sk);
+int tls_set_device_offload(struct sock *sk,
+ struct tls_crypto_info *crypto_info);
void tls_device_free_resources_tx(struct sock *sk);
-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx);
+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+ struct tls_crypto_info *crypto_info);
void tls_device_offload_cleanup_rx(struct sock *sk);
+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx);
void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq);
int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx);
#else
@@ -244,7 +252,7 @@ static inline int tls_device_init(void) { return 0; }
static inline void tls_device_cleanup(void) {}
static inline int
-tls_set_device_offload(struct sock *sk)
+tls_set_device_offload(struct sock *sk, struct tls_crypto_info *crypto_info)
{
return -EOPNOTSUPP;
}
@@ -252,13 +260,16 @@ tls_set_device_offload(struct sock *sk)
static inline void tls_device_free_resources_tx(struct sock *sk) {}
static inline int
-tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
+tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+ struct tls_crypto_info *crypto_info)
{
return -EOPNOTSUPP;
}
static inline void tls_device_offload_cleanup_rx(struct sock *sk) {}
static inline void
+tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx) {}
+static inline void
tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq) {}
static inline int
@@ -298,6 +309,16 @@ static inline bool tls_bigint_increment(unsigned char *seq, int len)
return (i == -1);
}
+static inline void tls_bigint_decrement(unsigned char *seq, int len)
+{
+ int i;
+
+ for (i = len - 1; i >= 0; i--) {
+ if (seq[i]-- != 0)
+ break;
+ }
+}
+
static inline void tls_bigint_subtract(unsigned char *seq, int n)
{
u64 rcd_sn;
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index cd26873e9063..51f1cc783336 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -79,7 +79,9 @@ static void tls_device_tx_del_task(struct work_struct *work)
netdev = rcu_dereference_protected(ctx->netdev,
!refcount_read(&ctx->refcount));
- netdev->tlsdev_ops->tls_dev_del(netdev, ctx, TLS_OFFLOAD_CTX_DIR_TX);
+ if (!test_bit(TLS_TX_DEV_CLOSED, &ctx->flags))
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_TX);
dev_put(netdev);
ctx->netdev = NULL;
tls_device_free_ctx(ctx);
@@ -159,6 +161,262 @@ static void delete_all_records(struct tls_offload_context_tx *offload_ctx)
offload_ctx->retransmit_hint = NULL;
}
+static bool tls_has_unacked_records(struct tls_offload_context_tx *offload_ctx)
+{
+ struct tls_record_info *info;
+ bool has_unacked = false;
+ unsigned long flags;
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ list_for_each_entry(info, &offload_ctx->records_list, list) {
+ if (!tls_record_is_start_marker(info)) {
+ has_unacked = true;
+ break;
+ }
+ }
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ return has_unacked;
+}
+
+static int tls_device_init_rekey_sw(struct sock *sk,
+ struct tls_context *ctx,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_sw_context_tx *sw_ctx = &offload_ctx->rekey_sw;
+ const struct tls_cipher_desc *cipher_desc;
+ char *key;
+ int rc;
+
+ cipher_desc = get_cipher_desc(new_crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ memset(sw_ctx, 0, sizeof(*sw_ctx));
+ tls_sw_ctx_tx_init(sk, sw_ctx);
+
+ sw_ctx->aead_send = crypto_alloc_aead(cipher_desc->cipher_name, 0, 0);
+ if (IS_ERR(sw_ctx->aead_send)) {
+ rc = PTR_ERR(sw_ctx->aead_send);
+ sw_ctx->aead_send = NULL;
+ return rc;
+ }
+
+ key = crypto_info_key(new_crypto_info, cipher_desc);
+ rc = crypto_aead_setkey(sw_ctx->aead_send, key, cipher_desc->key);
+ if (rc)
+ goto free_aead;
+
+ rc = crypto_aead_setauthsize(sw_ctx->aead_send, cipher_desc->tag);
+ if (rc)
+ goto free_aead;
+
+ return 0;
+
+free_aead:
+ crypto_free_aead(sw_ctx->aead_send);
+ sw_ctx->aead_send = NULL;
+ return rc;
+}
+
+static int tls_device_start_rekey(struct sock *sk,
+ struct tls_context *ctx,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ const struct tls_cipher_desc *cipher_desc;
+ char *key, *iv, *rec_seq, *salt;
+ int rc;
+
+ cipher_desc = get_cipher_desc(new_crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ key = crypto_info_key(new_crypto_info, cipher_desc);
+ iv = crypto_info_iv(new_crypto_info, cipher_desc);
+ rec_seq = crypto_info_rec_seq(new_crypto_info, cipher_desc);
+ salt = crypto_info_salt(new_crypto_info, cipher_desc);
+
+ if (rekey_pending || rekey_failed) {
+ rc = crypto_aead_setkey(offload_ctx->rekey_sw.aead_send,
+ key, cipher_desc->key);
+ if (rc)
+ return rc;
+
+ memcpy(offload_ctx->rekey_tx.iv, salt, cipher_desc->salt);
+ memcpy(offload_ctx->rekey_tx.iv + cipher_desc->salt, iv,
+ cipher_desc->iv);
+ memcpy(offload_ctx->rekey_tx.rec_seq, rec_seq,
+ cipher_desc->rec_seq);
+
+ if (rekey_failed) {
+ set_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ }
+ } else {
+ rc = tls_device_init_rekey_sw(sk, ctx, offload_ctx,
+ new_crypto_info);
+ if (rc)
+ return rc;
+
+ memcpy(offload_ctx->rekey_tx.iv, salt, cipher_desc->salt);
+ memcpy(offload_ctx->rekey_tx.iv + cipher_desc->salt, iv,
+ cipher_desc->iv);
+ memcpy(offload_ctx->rekey_tx.rec_seq, rec_seq,
+ cipher_desc->rec_seq);
+
+ WRITE_ONCE(ctx->rekey_boundary_seq, tcp_sk(sk)->write_seq);
+
+ /* Prevent a partial record straddling the SW/HW boundary. */
+ tcp_write_collapse_fence(sk);
+
+ ctx->rekey_sw_ctx = &offload_ctx->rekey_sw;
+ ctx->rekey_cipher_ctx = &offload_ctx->rekey_tx;
+
+ set_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+
+ /* Switch to rekey validator; new sends won't use HW offload */
+ smp_store_release(&sk->sk_validate_xmit_skb,
+ tls_validate_xmit_skb_rekey);
+ }
+
+ unsafe_memcpy(&offload_ctx->rekey_crypto_send.info, new_crypto_info,
+ cipher_desc->crypto_info,
+ /* checked in do_tls_setsockopt_conf */);
+ memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
+
+ return 0;
+}
+
+static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx)
+{
+ struct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);
+ struct tls_record_info *start_marker_record;
+ const struct tls_cipher_desc *cipher_desc;
+ struct net_device *netdev;
+ unsigned long flags;
+ __be64 rcd_sn;
+ char *key;
+ int rc;
+
+ cipher_desc = get_cipher_desc(offload_ctx->rekey_crypto_send.info.cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ /* Flush all pending SW data before switching back to HW:
+ * 1. Close any open_rec left by MSG_MORE and encrypt it.
+ * 2. Wait for async crypto completions.
+ * 3. Push all ready records into TCP.
+ * If the send buffer is full, bail out and retry next sendmsg.
+ */
+ if (tls_is_pending_open_record(ctx))
+ tls_sw_push_pending_record(sk, 0);
+ tls_encrypt_async_wait(tls_sw_ctx_tx(ctx));
+ rc = tls_tx_records(sk, -1);
+ if (rc < 0 || tls_is_partially_sent_record(ctx) ||
+ tls_is_pending_open_record(ctx))
+ return rc < 0 ? rc : -EAGAIN;
+
+ cancel_delayed_work_sync(&offload_ctx->rekey_sw.tx_work.work);
+
+ start_marker_record = kmalloc_obj(*start_marker_record);
+ if (!start_marker_record)
+ return -ENOMEM;
+
+ down_read(&device_offload_lock);
+
+ netdev = rcu_dereference_protected(ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
+ if (!netdev) {
+ rc = -ENODEV;
+ goto release_lock;
+ }
+
+ if (!test_bit(TLS_TX_DEV_CLOSED, &ctx->flags)) {
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_TX);
+ set_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+ }
+
+ memcpy(crypto_info_rec_seq(&offload_ctx->rekey_crypto_send.info, cipher_desc),
+ offload_ctx->rekey_tx.rec_seq, cipher_desc->rec_seq);
+
+ rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,
+ &offload_ctx->rekey_crypto_send.info,
+ tcp_sk(sk)->write_seq);
+ trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,
+ tcp_sk(sk)->write_seq,
+ offload_ctx->rekey_tx.rec_seq, rc);
+
+release_lock:
+ up_read(&device_offload_lock);
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ memcpy(&rcd_sn, offload_ctx->rekey_tx.rec_seq, sizeof(rcd_sn));
+ offload_ctx->unacked_record_sn = be64_to_cpu(rcd_sn) - 1;
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ memcpy(ctx->tx.iv, offload_ctx->rekey_tx.iv,
+ cipher_desc->salt + cipher_desc->iv);
+ memcpy(ctx->tx.rec_seq, offload_ctx->rekey_tx.rec_seq,
+ cipher_desc->rec_seq);
+ unsafe_memcpy(&ctx->crypto_send.info,
+ &offload_ctx->rekey_crypto_send.info,
+ cipher_desc->crypto_info,
+ /* checked during rekey setup */);
+
+ if (rc)
+ goto rekey_fail;
+
+ clear_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+
+ key = crypto_info_key(&offload_ctx->rekey_crypto_send.info, cipher_desc);
+ rc = crypto_aead_setkey(offload_ctx->aead_send, key, cipher_desc->key);
+ if (rc)
+ goto rekey_fail;
+
+ /* Start marker: the NIC passes through everything before
+ * write_seq unencrypted (already SW-encrypted during rekey),
+ * same as during initial offload setup.
+ */
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ start_marker_record->end_seq = tcp_sk(sk)->write_seq;
+ start_marker_record->len = 0;
+ start_marker_record->num_frags = 0;
+ list_add_tail_rcu(&start_marker_record->list,
+ &offload_ctx->records_list);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ /* Prevent a partial record straddling the SW/HW boundary. */
+ tcp_write_collapse_fence(sk);
+
+ /* PENDING before READY: prevents clean_acked from
+ * re-setting REKEY_READY after we clear it.
+ */
+ clear_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ smp_mb__after_atomic();
+ clear_bit(TLS_TX_REKEY_READY, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+
+ /* Switch back to HW offload validator */
+ smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);
+
+ crypto_free_aead(tls_sw_ctx_tx(ctx)->aead_send);
+ ctx->rekey_sw_ctx = NULL;
+ ctx->rekey_cipher_ctx = NULL;
+
+ return 0;
+
+rekey_fail:
+ kfree(start_marker_record);
+ set_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_READY, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYHWFAIL);
+
+ return 0;
+}
+
static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -187,6 +445,19 @@ static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)
}
ctx->unacked_record_sn += deleted_records;
+
+ /* Once all old-key HW records are ACKed, set REKEY_READY to
+ * let sendmsg know it can finish the rekey and switch back
+ * to HW offload.
+ */
+ if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags) &&
+ !test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags)) {
+ u32 boundary_seq = READ_ONCE(tls_ctx->rekey_boundary_seq);
+
+ if (!before(acked_seq, boundary_seq))
+ set_bit(TLS_TX_REKEY_READY, &tls_ctx->flags);
+ }
+
spin_unlock_irqrestore(&ctx->lock, flags);
}
@@ -218,6 +489,9 @@ void tls_device_free_resources_tx(struct sock *sk)
struct tls_context *tls_ctx = tls_get_ctx(sk);
tls_free_partial_record(sk, tls_ctx);
+
+ if (unlikely(tls_ctx->rekey_sw_ctx))
+ tls_sw_release_resources_tx(sk);
}
void tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)
@@ -589,6 +863,19 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
goto out;
}
+ /* Old-key records all ACKed; switch back to HW. */
+ if (test_bit(TLS_TX_REKEY_READY, &tls_ctx->flags))
+ tls_device_complete_rekey(sk, tls_ctx);
+
+ /* Use SW path if rekey is in progress (PENDING) or if HW rekey
+ * failed (FAILED).
+ */
+ if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags) ||
+ test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags)) {
+ rc = tls_sw_sendmsg_locked(sk, msg, size);
+ goto out;
+ }
+
rc = tls_push_data(sk, &msg->msg_iter, size, msg->msg_flags,
record_type);
@@ -791,6 +1078,8 @@ void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq)
return;
if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags)))
return;
+ if (unlikely(test_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags)))
+ return;
prot = &tls_ctx->prot_info;
rx_ctx = tls_offload_ctx_rx(tls_ctx);
@@ -980,13 +1269,144 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
return err;
}
+/*
+ * temporarily swap in the old key, run
+ * tls_device_reencrypt(), then restore the current key.
+ */
+static int tls_old_key_reencrypt(struct sock *sk,
+ struct tls_offload_context_rx *ctx,
+ struct tls_sw_context_rx *sw_ctx,
+ struct tls_context *tls_ctx)
+{
+ struct crypto_aead *saved_aead = sw_ctx->aead_recv;
+ char saved_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
+ char saved_rec_seq[TLS_MAX_REC_SEQ_SIZE];
+ int ret;
+
+ memcpy(saved_iv, tls_ctx->rx.iv, sizeof(saved_iv));
+ memcpy(saved_rec_seq, tls_ctx->rx.rec_seq, sizeof(saved_rec_seq));
+
+ sw_ctx->aead_recv = ctx->old_aead_recv;
+ memcpy(tls_ctx->rx.iv, ctx->old_iv, sizeof(ctx->old_iv));
+ memcpy(tls_ctx->rx.rec_seq, ctx->old_rec_seq,
+ sizeof(ctx->old_rec_seq));
+
+ ret = tls_device_reencrypt(sk, tls_ctx);
+
+ memcpy(ctx->old_rec_seq, tls_ctx->rx.rec_seq,
+ sizeof(ctx->old_rec_seq));
+
+ sw_ctx->aead_recv = saved_aead;
+ memcpy(tls_ctx->rx.iv, saved_iv, sizeof(saved_iv));
+ memcpy(tls_ctx->rx.rec_seq, saved_rec_seq, sizeof(saved_rec_seq));
+
+ return ret;
+}
+
+/* Undo old-key XOR so SW AEAD can decrypt with the new key. */
+static int tls_device_reencrypt_old_key(struct sock *sk,
+ struct tls_offload_context_rx *ctx,
+ struct tls_sw_context_rx *sw_ctx,
+ struct tls_context *tls_ctx)
+{
+ int ret;
+
+ ret = tls_old_key_reencrypt(sk, ctx, sw_ctx, tls_ctx);
+ if (ret)
+ return ret;
+
+ tls_bigint_increment(ctx->old_rec_seq,
+ tls_ctx->prot_info.rec_seq_size);
+ ctx->resync_nh_reset = 1;
+
+ return 0;
+}
+
+/* Tear down NIC offload on peer KeyUpdate so post-KU records
+ * (new-key wire encryption) are not NIC-XOR'd with the retired key.
+ * NIC stays keyless until tls_set_device_offload_rx installs the new key.
+ */
+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx)
+{
+ struct net_device *netdev;
+
+ if (ctx->rx_conf != TLS_HW)
+ return;
+ if (test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
+ return;
+
+ down_read(&device_offload_lock);
+ netdev = rcu_dereference_protected(ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
+ if (!netdev) {
+ up_read(&device_offload_lock);
+ return;
+ }
+
+ set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+ synchronize_net();
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_RX);
+ up_read(&device_offload_lock);
+}
+
+static int tls_device_dev_add(struct sock *sk, struct tls_context *tls_ctx,
+ struct net_device *netdev,
+ struct tls_crypto_info *crypto_info,
+ u32 cur_seq, bool is_rekey)
+{
+ const struct tls_cipher_desc *cipher_desc;
+ char *rec_seq;
+ int rc;
+
+ cipher_desc = get_cipher_desc(crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk,
+ TLS_OFFLOAD_CTX_DIR_RX,
+ crypto_info, cur_seq);
+ rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+ trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
+ cur_seq, rec_seq, rc);
+ if (!rc) {
+ clear_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
+ clear_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
+ } else if (is_rekey) {
+ set_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
+ set_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYHWFAIL);
+ }
+ return rc;
+}
+
+static void tls_device_deferred_dev_add(struct sock *sk,
+ struct tls_context *tls_ctx,
+ struct tls_offload_context_rx *ctx)
+{
+ struct net_device *netdev;
+
+ ctx->dev_add_pending = 0;
+
+ down_read(&device_offload_lock);
+ netdev = rcu_dereference_protected(tls_ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
+ if (netdev)
+ tls_device_dev_add(sk, tls_ctx, netdev,
+ &tls_ctx->crypto_recv.info,
+ tcp_sk(sk)->copied_seq, true);
+ up_read(&device_offload_lock);
+}
+
+
int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
{
struct tls_offload_context_rx *ctx = tls_offload_ctx_rx(tls_ctx);
struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(tls_ctx);
struct sk_buff *skb = tls_strp_msg(sw_ctx);
+ u32 copied_seq = tcp_sk(sk)->copied_seq;
struct strp_msg *rxm = strp_msg(skb);
int is_decrypted, is_encrypted;
+ u32 rec_start_seq;
if (!tls_strp_msg_mixed_decrypted(sw_ctx)) {
is_decrypted = skb->decrypted;
@@ -996,10 +1416,67 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
is_encrypted = 0;
}
- trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,
+ rec_start_seq = sw_ctx->strp.copy_mode
+ ? copied_seq - rxm->full_len
+ : copied_seq;
+
+ trace_tls_device_decrypted(sk, rec_start_seq,
tls_ctx->rx.rec_seq, rxm->full_len,
is_encrypted, is_decrypted);
+ if (unlikely(ctx->old_aead_recv)) {
+ bool before_nic_boundary =
+ before(rec_start_seq, ctx->old_nic_boundary);
+
+ /* Retry path: mixed record first-pass XOR-undo produced
+ * EBADMSG because per-fragment decrypted flags don't
+ * reflect which fragments were actually XOR'd (NIC auth
+ * failure clearing flags). Toggle decrypted flag and re-XOR,
+ * decrement old_rec_seq to reuse the same nonce.
+ */
+ if (ctx->old_key_reencrypted) {
+ struct sk_buff *frag_iter;
+
+ trace_tls_device_rekey_reencrypt(sk, rec_start_seq,
+ ctx->old_nic_boundary,
+ true);
+ skb->decrypted = !skb->decrypted;
+ skb_walk_frags(skb, frag_iter)
+ frag_iter->decrypted = !frag_iter->decrypted;
+
+ tls_bigint_decrement(ctx->old_rec_seq,
+ tls_ctx->prot_info.rec_seq_size);
+ return tls_device_reencrypt_old_key(sk, ctx,
+ sw_ctx, tls_ctx);
+ }
+
+ if (before_nic_boundary) {
+ if (is_encrypted) {
+ tls_bigint_increment(ctx->old_rec_seq,
+ tls_ctx->prot_info.rec_seq_size);
+ return 0;
+ }
+ /* For mixed records, first old key rencrypt and if
+ * SW AEAD fails then retry with decrypted flags toggled
+ */
+ trace_tls_device_rekey_reencrypt(sk, rec_start_seq,
+ ctx->old_nic_boundary,
+ false);
+ if (!is_decrypted)
+ ctx->old_key_reencrypted = 1;
+ return tls_device_reencrypt_old_key(sk, ctx,
+ sw_ctx, tls_ctx);
+ }
+
+ trace_tls_device_rekey_done(sk, rec_start_seq,
+ ctx->old_nic_boundary);
+ crypto_free_aead(ctx->old_aead_recv);
+ ctx->old_aead_recv = NULL;
+
+ if (ctx->dev_add_pending)
+ tls_device_deferred_dev_add(sk, tls_ctx, ctx);
+ }
+
if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags))) {
if (likely(is_encrypted || is_decrypted))
return is_decrypted;
@@ -1068,57 +1545,31 @@ static struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *c
return offload_ctx;
}
-int tls_set_device_offload(struct sock *sk)
+static int tls_set_device_offload_initial(struct sock *sk,
+ struct tls_context *ctx,
+ struct net_device *netdev,
+ struct tls_crypto_info *crypto_info,
+ const struct tls_cipher_desc *cipher_desc)
{
+ struct tls_prot_info *prot = &ctx->prot_info;
struct tls_record_info *start_marker_record;
struct tls_offload_context_tx *offload_ctx;
- const struct tls_cipher_desc *cipher_desc;
- struct tls_crypto_info *crypto_info;
- struct tls_prot_info *prot;
- struct net_device *netdev;
- struct tls_context *ctx;
char *iv, *rec_seq;
int rc;
- ctx = tls_get_ctx(sk);
- prot = &ctx->prot_info;
-
- if (ctx->priv_ctx_tx)
- return -EEXIST;
-
- netdev = get_netdev_for_sock(sk);
- if (!netdev) {
- pr_err_ratelimited("%s: netdev not found\n", __func__);
- return -EINVAL;
- }
-
- if (!(netdev->features & NETIF_F_HW_TLS_TX)) {
- rc = -EOPNOTSUPP;
- goto release_netdev;
- }
-
- crypto_info = &ctx->crypto_send.info;
- cipher_desc = get_cipher_desc(crypto_info->cipher_type);
- if (!cipher_desc || !cipher_desc->offloadable) {
- rc = -EINVAL;
- goto release_netdev;
- }
+ iv = crypto_info_iv(crypto_info, cipher_desc);
+ rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
rc = init_prot_info(prot, crypto_info, cipher_desc);
if (rc)
- goto release_netdev;
-
- iv = crypto_info_iv(crypto_info, cipher_desc);
- rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+ return rc;
memcpy(ctx->tx.iv + cipher_desc->salt, iv, cipher_desc->iv);
memcpy(ctx->tx.rec_seq, rec_seq, cipher_desc->rec_seq);
start_marker_record = kmalloc_obj(*start_marker_record);
- if (!start_marker_record) {
- rc = -ENOMEM;
- goto release_netdev;
- }
+ if (!start_marker_record)
+ return -ENOMEM;
offload_ctx = alloc_offload_ctx_tx(ctx);
if (!offload_ctx) {
@@ -1159,8 +1610,10 @@ int tls_set_device_offload(struct sock *sk)
}
ctx->priv_ctx_tx = offload_ctx;
- rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,
- &ctx->crypto_send.info,
+
+ rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk,
+ TLS_OFFLOAD_CTX_DIR_TX,
+ crypto_info,
tcp_sk(sk)->write_seq);
trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,
tcp_sk(sk)->write_seq, rec_seq, rc);
@@ -1175,7 +1628,6 @@ int tls_set_device_offload(struct sock *sk)
* by the netdev's xmit function.
*/
smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);
- dev_put(netdev);
return 0;
@@ -1188,18 +1640,112 @@ int tls_set_device_offload(struct sock *sk)
ctx->priv_ctx_tx = NULL;
free_marker_record:
kfree(start_marker_record);
+ return rc;
+}
+
+static int tls_set_device_offload_rekey(struct sock *sk,
+ struct tls_context *ctx,
+ struct net_device *netdev,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);
+ bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ bool defer = true;
+ int rc;
+
+ if (!rekey_pending && !rekey_failed)
+ defer = tls_has_unacked_records(offload_ctx);
+
+ down_read(&device_offload_lock);
+
+ rc = tls_device_start_rekey(sk, ctx, offload_ctx, new_crypto_info);
+ if (rc) {
+ up_read(&device_offload_lock);
+ return rc;
+ }
+
+ up_read(&device_offload_lock);
+
+ if (!defer)
+ rc = tls_device_complete_rekey(sk, ctx);
+
+ return rc;
+}
+
+int tls_set_device_offload(struct sock *sk,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_crypto_info *crypto_info, *src_crypto_info;
+ const struct tls_cipher_desc *cipher_desc;
+ struct net_device *netdev;
+ struct tls_context *ctx;
+ int rc;
+
+ ctx = tls_get_ctx(sk);
+
+ /* Rekey is only supported for connections that are already
+ * using HW offload. For SW offload connections, the caller
+ * should fall back to tls_set_sw_offload() for rekey.
+ */
+ if (new_crypto_info && ctx->tx_conf != TLS_HW)
+ return -EINVAL;
+
+ netdev = get_netdev_for_sock(sk);
+ if (!netdev) {
+ pr_err_ratelimited("%s: netdev not found\n", __func__);
+ return -EINVAL;
+ }
+
+ if (!(netdev->features & NETIF_F_HW_TLS_TX)) {
+ rc = -EOPNOTSUPP;
+ goto release_netdev;
+ }
+
+ crypto_info = &ctx->crypto_send.info;
+ src_crypto_info = new_crypto_info ?: crypto_info;
+ cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+ if (!cipher_desc || !cipher_desc->offloadable) {
+ rc = -EINVAL;
+ goto release_netdev;
+ }
+
+ if (new_crypto_info)
+ rc = tls_set_device_offload_rekey(sk, ctx, netdev,
+ src_crypto_info);
+ else
+ rc = tls_set_device_offload_initial(sk, ctx, netdev,
+ src_crypto_info,
+ cipher_desc);
+
release_netdev:
dev_put(netdev);
return rc;
}
-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+ struct tls_crypto_info *new_crypto_info)
{
- struct tls12_crypto_info_aes_gcm_128 *info;
+ struct tls_crypto_info *crypto_info, *src_crypto_info;
+ const struct tls_cipher_desc *cipher_desc;
+ u32 copied_seq = tcp_sk(sk)->copied_seq;
struct tls_offload_context_rx *context;
struct net_device *netdev;
int rc = 0;
+ /* Rekey is only supported for connections that are already
+ * using HW offload. For SW offload connections, the caller
+ * should fall back to tls_set_sw_offload() for rekey.
+ */
+ if (new_crypto_info && ctx->rx_conf != TLS_HW)
+ return -EINVAL;
+
+ crypto_info = &ctx->crypto_recv.info;
+ src_crypto_info = new_crypto_info ?: crypto_info;
+ cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+ if (!cipher_desc || !cipher_desc->offloadable)
+ return -EINVAL;
+
netdev = get_netdev_for_sock(sk);
if (!netdev) {
pr_err_ratelimited("%s: netdev not found\n", __func__);
@@ -1225,29 +1771,82 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
goto release_lock;
}
- context = kzalloc_obj(*context);
- if (!context) {
- rc = -ENOMEM;
- goto release_lock;
+ if (!new_crypto_info) {
+ context = kzalloc_obj(*context);
+ if (!context) {
+ rc = -ENOMEM;
+ goto release_lock;
+ }
+ ctx->priv_ctx_rx = context;
+ } else {
+ context = tls_offload_ctx_rx(ctx);
}
context->resync_nh_reset = 1;
- ctx->priv_ctx_rx = context;
- rc = tls_sw_ctx_init(sk, 0, NULL);
+ if (new_crypto_info) {
+ struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);
+
+ if (!test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
+ set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+ synchronize_net();
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_RX);
+ }
+
+ if (context->old_aead_recv &&
+ before(copied_seq, context->old_nic_boundary)) {
+ /* Previous rekey still draining. Keep old_aead_recv,
+ * it is the only key that can undo the NIC-XOR on queued
+ * records. sw_ctx->aead_recv may be re-setkey'd by
+ * tls_sw_ctx_init(); that intermediate key was never on
+ * the NIC and its wire era is drained, so it is needed
+ * for neither undo nor AEAD. Defer dev_add; the new key
+ * is installed once copied_seq crosses old_nic_boundary.
+ */
+ context->dev_add_pending = 1;
+ } else {
+ u32 rcv_nxt;
+
+ if (context->old_aead_recv) {
+ crypto_free_aead(context->old_aead_recv);
+ context->old_aead_recv = NULL;
+ }
+
+ /* flush the backlog so rcv_nxt is accurate */
+ __sk_flush_backlog(sk);
+ rcv_nxt = tcp_sk(sk)->rcv_nxt;
+
+ if (before(copied_seq, rcv_nxt)) {
+ context->old_aead_recv = sw_ctx->aead_recv;
+ sw_ctx->aead_recv = NULL;
+ memcpy(context->old_iv, ctx->rx.iv,
+ sizeof(context->old_iv));
+ memcpy(context->old_rec_seq, ctx->rx.rec_seq,
+ sizeof(context->old_rec_seq));
+ context->old_nic_boundary = rcv_nxt;
+ context->dev_add_pending = 1;
+ }
+ trace_tls_device_rekey_start(sk, copied_seq, rcv_nxt,
+ before(copied_seq, rcv_nxt));
+ }
+ }
+
+ rc = tls_sw_ctx_init(sk, 0, new_crypto_info);
if (rc)
goto release_ctx;
- rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_RX,
- &ctx->crypto_recv.info,
- tcp_sk(sk)->copied_seq);
- info = (void *)&ctx->crypto_recv.info;
- trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
- tcp_sk(sk)->copied_seq, info->rec_seq, rc);
- if (rc)
- goto free_sw_resources;
+ if (!context->dev_add_pending) {
+ rc = tls_device_dev_add(sk, ctx, netdev, src_crypto_info,
+ copied_seq, !!new_crypto_info);
+ if (!new_crypto_info) {
+ if (rc)
+ goto free_sw_resources;
+ tls_device_attach(ctx, sk, netdev);
+ }
+ }
+
+ tls_sw_ctx_finalize(sk, 0, new_crypto_info);
- tls_device_attach(ctx, sk, netdev);
- tls_sw_ctx_finalize(sk, 0, NULL);
up_read(&device_offload_lock);
dev_put(netdev);
@@ -1256,10 +1855,13 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
free_sw_resources:
up_read(&device_offload_lock);
- tls_sw_free_resources_rx(sk);
+ tls_sw_release_resources_rx(sk);
down_read(&device_offload_lock);
release_ctx:
- ctx->priv_ctx_rx = NULL;
+ if (!new_crypto_info) {
+ kfree(ctx->priv_ctx_rx);
+ ctx->priv_ctx_rx = NULL;
+ }
release_lock:
up_read(&device_offload_lock);
release_netdev:
@@ -1270,6 +1872,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
void tls_device_offload_cleanup_rx(struct sock *sk)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
+ struct tls_offload_context_rx *rx_ctx;
struct net_device *netdev;
down_read(&device_offload_lock);
@@ -1278,8 +1881,9 @@ void tls_device_offload_cleanup_rx(struct sock *sk)
if (!netdev)
goto out;
- netdev->tlsdev_ops->tls_dev_del(netdev, tls_ctx,
- TLS_OFFLOAD_CTX_DIR_RX);
+ if (!test_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags))
+ netdev->tlsdev_ops->tls_dev_del(netdev, tls_ctx,
+ TLS_OFFLOAD_CTX_DIR_RX);
if (tls_ctx->tx_conf != TLS_HW) {
dev_put(netdev);
@@ -1289,6 +1893,13 @@ void tls_device_offload_cleanup_rx(struct sock *sk)
}
out:
up_read(&device_offload_lock);
+
+ rx_ctx = tls_offload_ctx_rx(tls_ctx);
+ if (rx_ctx && rx_ctx->old_aead_recv) {
+ crypto_free_aead(rx_ctx->old_aead_recv);
+ rx_ctx->old_aead_recv = NULL;
+ }
+
tls_sw_release_resources_rx(sk);
}
@@ -1319,7 +1930,10 @@ static int tls_device_down(struct net_device *netdev)
/* Stop offloaded TX and switch to the fallback.
* tls_is_skb_tx_device_offloaded will return false.
*/
- WRITE_ONCE(ctx->sk->sk_validate_xmit_skb, tls_validate_xmit_skb_sw);
+ if (!test_bit(TLS_TX_REKEY_PENDING, &ctx->flags) &&
+ !test_bit(TLS_TX_REKEY_FAILED, &ctx->flags))
+ WRITE_ONCE(ctx->sk->sk_validate_xmit_skb,
+ tls_validate_xmit_skb_sw);
/* Stop the RX and TX resync.
* tls_dev_resync must not be called after tls_dev_del.
@@ -1336,13 +1950,18 @@ static int tls_device_down(struct net_device *netdev)
synchronize_net();
/* Release the offload context on the driver side. */
- if (ctx->tx_conf == TLS_HW)
+ if (ctx->tx_conf == TLS_HW &&
+ !test_bit(TLS_TX_DEV_CLOSED, &ctx->flags)) {
netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
TLS_OFFLOAD_CTX_DIR_TX);
+ set_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+ }
if (ctx->rx_conf == TLS_HW &&
- !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
+ !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
TLS_OFFLOAD_CTX_DIR_RX);
+ set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+ }
dev_put(netdev);
diff --git a/net/tls/tls_device_fallback.c b/net/tls/tls_device_fallback.c
index 1110f7ac6bcb..5be425a32c82 100644
--- a/net/tls/tls_device_fallback.c
+++ b/net/tls/tls_device_fallback.c
@@ -435,6 +435,30 @@ struct sk_buff *tls_validate_xmit_skb_sw(struct sock *sk,
return tls_sw_fallback(sk, skb);
}
+struct sk_buff *tls_validate_xmit_skb_rekey(struct sock *sk,
+ struct net_device *dev,
+ struct sk_buff *skb)
+{
+ struct tls_context *tls_ctx = tls_get_ctx(sk);
+ u32 tcp_seq = ntohl(tcp_hdr(skb)->seq);
+ u32 boundary_seq;
+
+ if (test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags))
+ return skb;
+
+ /* If this packet is at or after the rekey boundary, it's already
+ * SW-encrypted with the new key, pass through unchanged
+ */
+ boundary_seq = READ_ONCE(tls_ctx->rekey_boundary_seq);
+ if (!before(tcp_seq, boundary_seq))
+ return skb;
+
+ /* Packet before boundary means retransmit of old data,
+ * use SW fallback with the old key
+ */
+ return tls_sw_fallback(sk, skb);
+}
+
struct sk_buff *tls_encrypt_skb(struct sk_buff *skb)
{
return tls_sw_fallback(skb->sk, skb);
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index fd04857fa0ab..ab701f166b57 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -371,6 +371,8 @@ static void tls_sk_proto_close(struct sock *sk, long timeout)
if (ctx->tx_conf == TLS_SW)
tls_sw_cancel_work_tx(ctx);
+ else if (ctx->tx_conf == TLS_HW && ctx->rekey_sw_ctx)
+ tls_sw_cancel_work_tx(ctx);
lock_sock(sk);
free_ctx = ctx->tx_conf != TLS_HW && ctx->rx_conf != TLS_HW;
@@ -711,64 +713,68 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
}
if (tx) {
- if (update && ctx->tx_conf == TLS_HW) {
- rc = -EOPNOTSUPP;
- goto err_crypto_info;
- }
-
- if (!update) {
- rc = tls_set_device_offload(sk);
- conf = TLS_HW;
- if (!rc) {
+ rc = tls_set_device_offload(sk, update ? crypto_info : NULL);
+ conf = TLS_HW;
+ if (!rc) {
+ if (update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
+ } else {
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
- goto out;
}
- }
-
- rc = tls_set_sw_offload(sk, 1, update ? crypto_info : NULL);
- if (rc)
+ } else if (update && ctx->tx_conf == TLS_HW) {
+ /* HW rekey failed - return the actual error.
+ * Cannot fall back to SW for an existing HW connection.
+ */
goto err_crypto_info;
-
- if (update) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
} else {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXSW);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+ rc = tls_set_sw_offload(sk, 1,
+ update ? crypto_info : NULL);
+ if (rc)
+ goto err_crypto_info;
+
+ if (update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
+ } else {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXSW);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+ }
+ conf = TLS_SW;
}
- conf = TLS_SW;
} else {
- if (update && ctx->rx_conf == TLS_HW) {
- rc = -EOPNOTSUPP;
- goto err_crypto_info;
- }
-
- if (!update) {
- rc = tls_set_device_offload_rx(sk, ctx);
- conf = TLS_HW;
- if (!rc) {
+ rc = tls_set_device_offload_rx(sk, ctx,
+ update ? crypto_info : NULL);
+ conf = TLS_HW;
+ if (!rc) {
+ if (update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+ } else {
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
- tls_sw_strparser_arm(sk, ctx);
- goto out;
}
- }
-
- rc = tls_set_sw_offload(sk, 0, update ? crypto_info : NULL);
- if (rc)
+ } else if (update && ctx->rx_conf == TLS_HW) {
+ /* HW rekey failed - return the actual error.
+ * Cannot fall back to SW for an existing HW connection.
+ */
goto err_crypto_info;
-
- if (update) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
} else {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXSW);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
- tls_sw_strparser_arm(sk, ctx);
+ rc = tls_set_sw_offload(sk, 0,
+ update ? crypto_info : NULL);
+ if (rc)
+ goto err_crypto_info;
+
+ if (update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+ } else {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXSW);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+ }
+ conf = TLS_SW;
}
- conf = TLS_SW;
+ if (!update)
+ tls_sw_strparser_arm(sk, ctx);
}
-out:
if (tx)
ctx->tx_conf = conf;
else
diff --git a/net/tls/tls_proc.c b/net/tls/tls_proc.c
index 4012c4372d4c..5599af306aab 100644
--- a/net/tls/tls_proc.c
+++ b/net/tls/tls_proc.c
@@ -27,6 +27,8 @@ static const struct snmp_mib tls_mib_list[] = {
SNMP_MIB_ITEM("TlsTxRekeyOk", LINUX_MIB_TLSTXREKEYOK),
SNMP_MIB_ITEM("TlsTxRekeyError", LINUX_MIB_TLSTXREKEYERROR),
SNMP_MIB_ITEM("TlsRxRekeyReceived", LINUX_MIB_TLSRXREKEYRECEIVED),
+ SNMP_MIB_ITEM("TlsTxRekeyHwFail", LINUX_MIB_TLSTXREKEYHWFAIL),
+ SNMP_MIB_ITEM("TlsRxRekeyHwFail", LINUX_MIB_TLSRXREKEYHWFAIL),
};
static int tls_statistics_seq_show(struct seq_file *seq, void *v)
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 1412b3dcce4c..fc60e8c0f24c 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -522,7 +522,7 @@ static void tls_encrypt_done(void *data, int err)
complete(&ctx->async_wait.completion);
}
-static int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
{
if (!atomic_dec_and_test(&ctx->encrypt_pending))
crypto_wait_req(-EINPROGRESS, &ctx->async_wait);
@@ -555,11 +555,11 @@ static int tls_do_encryption(struct sock *sk,
break;
}
- memcpy(&rec->iv_data[iv_offset], tls_ctx->tx.iv,
+ memcpy(&rec->iv_data[iv_offset], tls_tx_cipher_ctx(tls_ctx)->iv,
prot->iv_size + prot->salt_size);
tls_xor_iv_with_seq(prot, rec->iv_data + iv_offset,
- tls_ctx->tx.rec_seq);
+ tls_tx_cipher_ctx(tls_ctx)->rec_seq);
sge->offset += prot->prepend_size;
sge->length -= prot->prepend_size;
@@ -610,7 +610,7 @@ static int tls_do_encryption(struct sock *sk,
/* Unhook the record from context if encryption is not failure */
ctx->open_rec = NULL;
- tls_advance_record_sn(sk, prot, &tls_ctx->tx);
+ tls_advance_record_sn(sk, prot, tls_tx_cipher_ctx(tls_ctx));
return rc;
}
@@ -817,7 +817,7 @@ static int tls_push_record(struct sock *sk, int flags,
sg_chain(rec->sg_aead_out, 2, &msg_en->sg.data[i]);
tls_make_aad(rec->aad_space, msg_pl->sg.size + prot->tail_size,
- tls_ctx->tx.rec_seq, record_type, prot);
+ tls_tx_cipher_ctx(tls_ctx)->rec_seq, record_type, prot);
tls_fill_prepend(tls_ctx,
page_address(sg_page(&msg_en->sg.data[i])) +
@@ -982,7 +982,7 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,
return err;
}
-static int tls_sw_push_pending_record(struct sock *sk, int flags)
+int tls_sw_push_pending_record(struct sock *sk, int flags)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);
@@ -1033,8 +1033,7 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg,
return 0;
}
-static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg,
- size_t size)
+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
{
long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -1802,6 +1801,8 @@ static int tls_check_pending_rekey(struct sock *sk, struct tls_context *ctx,
if (hs_type == TLS_HANDSHAKE_KEYUPDATE) {
struct tls_sw_context_rx *rx_ctx = ctx->priv_ctx_rx;
+ /* Stop NIC from XOR-ing post-KU records with the retired key */
+ tls_device_rx_del_key(sk, ctx);
WRITE_ONCE(rx_ctx->key_update_pending, true);
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYRECEIVED);
}
@@ -1809,6 +1810,36 @@ static int tls_check_pending_rekey(struct sock *sk, struct tls_context *ctx,
return 0;
}
+static int tls_rx_rekey_retry(struct sock *sk, struct msghdr *msg,
+ struct tls_context *tls_ctx,
+ struct tls_decrypt_arg *darg, int err)
+{
+ struct tls_offload_context_rx *rx_ctx = tls_offload_ctx_rx(tls_ctx);
+ struct tls_prot_info *prot = &tls_ctx->prot_info;
+
+ if (!rx_ctx->old_key_reencrypted)
+ return err;
+
+ if (err == -EBADMSG) {
+ if (darg->zc) {
+ struct tls_sw_context_rx *sw_ctx =
+ tls_sw_ctx_rx(tls_ctx);
+ struct strp_msg *rxm;
+
+ rxm = strp_msg(tls_strp_msg(sw_ctx));
+ iov_iter_revert(&msg->msg_iter,
+ rxm->full_len - prot->overhead_size);
+ }
+
+ err = tls_decrypt_device(sk, msg, tls_ctx, darg);
+ if (!err)
+ err = tls_decrypt_sw(sk, tls_ctx, msg, darg);
+ }
+
+ rx_ctx->old_key_reencrypted = 0;
+ return err;
+}
+
static int tls_rx_one_record(struct sock *sk, struct msghdr *msg,
struct tls_decrypt_arg *darg)
{
@@ -1820,6 +1851,10 @@ static int tls_rx_one_record(struct sock *sk, struct msghdr *msg,
err = tls_decrypt_device(sk, msg, tls_ctx, darg);
if (!err)
err = tls_decrypt_sw(sk, tls_ctx, msg, darg);
+
+ if (tls_ctx->rx_conf == TLS_HW)
+ err = tls_rx_rekey_retry(sk, msg, tls_ctx, darg, err);
+
if (err < 0)
return err;
@@ -2630,7 +2665,7 @@ void tls_sw_free_resources_rx(struct sock *sk)
}
/* The work handler to transmitt the encrypted records in tx_list */
-static void tx_work_handler(struct work_struct *work)
+void tls_tx_work_handler(struct work_struct *work)
{
struct delayed_work *delayed_work = to_delayed_work(work);
struct tx_work *tx_work = container_of(delayed_work,
@@ -2663,6 +2698,15 @@ static void tx_work_handler(struct work_struct *work)
}
}
+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx)
+{
+ crypto_init_wait(&sw_ctx->async_wait);
+ atomic_set(&sw_ctx->encrypt_pending, 1);
+ INIT_LIST_HEAD(&sw_ctx->tx_list);
+ INIT_DELAYED_WORK(&sw_ctx->tx_work.work, tls_tx_work_handler);
+ sw_ctx->tx_work.sk = sk;
+}
+
static bool tls_is_tx_ready(struct tls_sw_context_tx *ctx)
{
struct tls_rec *rec;
@@ -2714,11 +2758,7 @@ static struct tls_sw_context_tx *init_ctx_tx(struct tls_context *ctx, struct soc
sw_ctx_tx = ctx->priv_ctx_tx;
}
- crypto_init_wait(&sw_ctx_tx->async_wait);
- atomic_set(&sw_ctx_tx->encrypt_pending, 1);
- INIT_LIST_HEAD(&sw_ctx_tx->tx_list);
- INIT_DELAYED_WORK(&sw_ctx_tx->tx_work.work, tx_work_handler);
- sw_ctx_tx->tx_work.sk = sk;
+ tls_sw_ctx_tx_init(sk, sw_ctx_tx);
return sw_ctx_tx;
}
@@ -2861,11 +2901,9 @@ int tls_sw_ctx_init(struct sock *sk, int tx,
goto free_aead;
}
- if (!new_crypto_info) {
- rc = crypto_aead_setauthsize(*aead, prot->tag_size);
- if (rc)
- goto free_aead;
- }
+ rc = crypto_aead_setauthsize(*aead, prot->tag_size);
+ if (rc)
+ goto free_aead;
if (!tx && !new_crypto_info) {
tfm = crypto_aead_tfm(sw_ctx_rx->aead_recv);
diff --git a/net/tls/trace.h b/net/tls/trace.h
index 2d8ce4ff3265..56fcf95c5aaf 100644
--- a/net/tls/trace.h
+++ b/net/tls/trace.h
@@ -192,6 +192,85 @@ TRACE_EVENT(tls_device_tx_resync_send,
)
);
+TRACE_EVENT(tls_device_rekey_start,
+
+ TP_PROTO(struct sock *sk, u32 copied_seq, u32 nic_boundary,
+ bool inflight),
+
+ TP_ARGS(sk, copied_seq, nic_boundary, inflight),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( u32, copied_seq )
+ __field( u32, nic_boundary )
+ __field( bool, inflight )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->copied_seq = copied_seq;
+ __entry->nic_boundary = nic_boundary;
+ __entry->inflight = inflight;
+ ),
+
+ TP_printk(
+ "sk=%p copied_seq=%u nic_boundary=%u inflight=%d",
+ __entry->sk, __entry->copied_seq, __entry->nic_boundary,
+ __entry->inflight
+ )
+);
+
+TRACE_EVENT(tls_device_rekey_reencrypt,
+
+ TP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary, bool retry),
+
+ TP_ARGS(sk, tcp_seq, nic_boundary, retry),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( u32, tcp_seq )
+ __field( u32, nic_boundary )
+ __field( bool, retry )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->tcp_seq = tcp_seq;
+ __entry->nic_boundary = nic_boundary;
+ __entry->retry = retry;
+ ),
+
+ TP_printk(
+ "sk=%p tcp_seq=%u nic_boundary=%u retry=%d",
+ __entry->sk, __entry->tcp_seq, __entry->nic_boundary,
+ __entry->retry
+ )
+);
+
+TRACE_EVENT(tls_device_rekey_done,
+
+ TP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary),
+
+ TP_ARGS(sk, tcp_seq, nic_boundary),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( u32, tcp_seq )
+ __field( u32, nic_boundary )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->tcp_seq = tcp_seq;
+ __entry->nic_boundary = nic_boundary;
+ ),
+
+ TP_printk(
+ "sk=%p tcp_seq=%u nic_boundary=%u",
+ __entry->sk, __entry->tcp_seq, __entry->nic_boundary
+ )
+);
+
#endif /* _TLS_TRACE_H_ */
#undef TRACE_INCLUDE_PATH
--
2.25.1
^ permalink raw reply related
* [PATCH v13 4/6] tls: split tls_set_sw_offload into init and finalize stages
From: Rishikesh Jethwani @ 2026-04-29 18:10 UTC (permalink / raw)
To: netdev
Cc: saeedm, tariqt, mbloch, borisp, john.fastabend, kuba, sd, davem,
pabeni, edumazet, leon, Rishikesh Jethwani
In-Reply-To: <20260429181016.3164935-1-rjethwani@purestorage.com>
Separate cipher context initialization from key material finalization
to support staged setup for hardware offload fallback paths.
Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
net/tls/tls.h | 4 +++
net/tls/tls_device.c | 3 +-
net/tls/tls_sw.c | 77 +++++++++++++++++++++++++++++++-------------
3 files changed, 61 insertions(+), 23 deletions(-)
diff --git a/net/tls/tls.h b/net/tls/tls.h
index e8f81a006520..a65cf9bab190 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -147,6 +147,10 @@ void tls_strp_abort_strp(struct tls_strparser *strp, int err);
int init_prot_info(struct tls_prot_info *prot,
const struct tls_crypto_info *crypto_info,
const struct tls_cipher_desc *cipher_desc);
+int tls_sw_ctx_init(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info);
+void tls_sw_ctx_finalize(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info);
int tls_set_sw_offload(struct sock *sk, int tx,
struct tls_crypto_info *new_crypto_info);
void tls_update_rx_zc_capable(struct tls_context *tls_ctx);
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index 1321bf9b59b0..cd26873e9063 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -1233,7 +1233,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
context->resync_nh_reset = 1;
ctx->priv_ctx_rx = context;
- rc = tls_set_sw_offload(sk, 0, NULL);
+ rc = tls_sw_ctx_init(sk, 0, NULL);
if (rc)
goto release_ctx;
@@ -1247,6 +1247,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
goto free_sw_resources;
tls_device_attach(ctx, sk, netdev);
+ tls_sw_ctx_finalize(sk, 0, NULL);
up_read(&device_offload_lock);
dev_put(netdev);
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 94d2ae0daa8c..1412b3dcce4c 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -2784,20 +2784,19 @@ static void tls_finish_key_update(struct sock *sk, struct tls_context *tls_ctx)
ctx->saved_data_ready(sk);
}
-int tls_set_sw_offload(struct sock *sk, int tx,
- struct tls_crypto_info *new_crypto_info)
+int tls_sw_ctx_init(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info)
{
struct tls_crypto_info *crypto_info, *src_crypto_info;
struct tls_sw_context_tx *sw_ctx_tx = NULL;
struct tls_sw_context_rx *sw_ctx_rx = NULL;
const struct tls_cipher_desc *cipher_desc;
- char *iv, *rec_seq, *key, *salt;
- struct cipher_context *cctx;
struct tls_prot_info *prot;
struct crypto_aead **aead;
struct tls_context *ctx;
struct crypto_tfm *tfm;
int rc = 0;
+ char *key;
ctx = tls_get_ctx(sk);
prot = &ctx->prot_info;
@@ -2818,12 +2817,10 @@ int tls_set_sw_offload(struct sock *sk, int tx,
if (tx) {
sw_ctx_tx = ctx->priv_ctx_tx;
crypto_info = &ctx->crypto_send.info;
- cctx = &ctx->tx;
aead = &sw_ctx_tx->aead_send;
} else {
sw_ctx_rx = ctx->priv_ctx_rx;
crypto_info = &ctx->crypto_recv.info;
- cctx = &ctx->rx;
aead = &sw_ctx_rx->aead_recv;
}
@@ -2839,10 +2836,7 @@ int tls_set_sw_offload(struct sock *sk, int tx,
if (rc)
goto free_priv;
- iv = crypto_info_iv(src_crypto_info, cipher_desc);
key = crypto_info_key(src_crypto_info, cipher_desc);
- salt = crypto_info_salt(src_crypto_info, cipher_desc);
- rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);
if (!*aead) {
*aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, 0);
@@ -2886,19 +2880,6 @@ int tls_set_sw_offload(struct sock *sk, int tx,
goto free_aead;
}
- memcpy(cctx->iv, salt, cipher_desc->salt);
- memcpy(cctx->iv + cipher_desc->salt, iv, cipher_desc->iv);
- memcpy(cctx->rec_seq, rec_seq, cipher_desc->rec_seq);
-
- if (new_crypto_info) {
- unsafe_memcpy(crypto_info, new_crypto_info,
- cipher_desc->crypto_info,
- /* size was checked in do_tls_setsockopt_conf */);
- memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
- if (!tx)
- tls_finish_key_update(sk, ctx);
- }
-
goto out;
free_aead:
@@ -2917,3 +2898,55 @@ int tls_set_sw_offload(struct sock *sk, int tx,
out:
return rc;
}
+
+void tls_sw_ctx_finalize(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_crypto_info *crypto_info, *src_crypto_info;
+ const struct tls_cipher_desc *cipher_desc;
+ struct tls_context *ctx = tls_get_ctx(sk);
+ struct cipher_context *cctx;
+ char *iv, *salt, *rec_seq;
+
+ if (tx) {
+ crypto_info = &ctx->crypto_send.info;
+ cctx = &ctx->tx;
+ } else {
+ crypto_info = &ctx->crypto_recv.info;
+ cctx = &ctx->rx;
+ }
+
+ src_crypto_info = new_crypto_info ?: crypto_info;
+ cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+
+ iv = crypto_info_iv(src_crypto_info, cipher_desc);
+ salt = crypto_info_salt(src_crypto_info, cipher_desc);
+ rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);
+
+ memcpy(cctx->iv, salt, cipher_desc->salt);
+ memcpy(cctx->iv + cipher_desc->salt, iv, cipher_desc->iv);
+ memcpy(cctx->rec_seq, rec_seq, cipher_desc->rec_seq);
+
+ if (new_crypto_info) {
+ unsafe_memcpy(crypto_info, new_crypto_info,
+ cipher_desc->crypto_info,
+ /* size was checked in do_tls_setsockopt_conf */);
+ memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
+
+ if (!tx)
+ tls_finish_key_update(sk, ctx);
+ }
+}
+
+int tls_set_sw_offload(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ int rc;
+
+ rc = tls_sw_ctx_init(sk, tx, new_crypto_info);
+ if (rc)
+ return rc;
+
+ tls_sw_ctx_finalize(sk, tx, new_crypto_info);
+ return 0;
+}
--
2.25.1
^ permalink raw reply related
* [PATCH v13 3/6] tls: add TLS 1.3 hardware offload support
From: Rishikesh Jethwani @ 2026-04-29 18:10 UTC (permalink / raw)
To: netdev
Cc: saeedm, tariqt, mbloch, borisp, john.fastabend, kuba, sd, davem,
pabeni, edumazet, leon, Rishikesh Jethwani
In-Reply-To: <20260429181016.3164935-1-rjethwani@purestorage.com>
Add TLS 1.3 support to the kernel TLS hardware offload infrastructure,
enabling hardware acceleration for TLS 1.3 connections on capable NICs.
Tested on Mellanox ConnectX-6 Dx (Crypto Enabled) with TLS 1.3 AES-GCM-128
and AES-GCM-256 cipher suites.
Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
net/tls/tls_device.c | 65 ++++++++++++++++-----------
net/tls/tls_device_fallback.c | 58 +++++++++++++-----------
net/tls/tls_main.c | 85 ++++++++++++++++++++---------------
3 files changed, 121 insertions(+), 87 deletions(-)
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index 99c8eff9783e..1321bf9b59b0 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -317,25 +317,34 @@ static void tls_device_record_close(struct sock *sk,
unsigned char record_type)
{
struct tls_prot_info *prot = &ctx->prot_info;
- struct page_frag dummy_tag_frag;
-
- /* append tag
- * device will fill in the tag, we just need to append a placeholder
- * use socket memory to improve coalescing (re-using a single buffer
- * increases frag count)
- * if we can't allocate memory now use the dummy page
+ int tail = prot->tag_size + prot->tail_size;
+
+ /* Append tail: tag for TLS 1.2, content_type + tag for TLS 1.3.
+ * Device fills in the tag, we just need to append a placeholder.
+ * Use socket memory to improve coalescing (re-using a single buffer
+ * increases frag count); if allocation fails use dummy_page
+ * (offset = record_type gives correct content_type byte via
+ * identity mapping)
*/
- if (unlikely(pfrag->size - pfrag->offset < prot->tag_size) &&
- !skb_page_frag_refill(prot->tag_size, pfrag, sk->sk_allocation)) {
- dummy_tag_frag.page = dummy_page;
- dummy_tag_frag.offset = 0;
- pfrag = &dummy_tag_frag;
+ if (unlikely(pfrag->size - pfrag->offset < tail) &&
+ !skb_page_frag_refill(tail, pfrag, sk->sk_allocation)) {
+ struct page_frag dummy_pfrag = {
+ .page = dummy_page,
+ .offset = record_type,
+ };
+ tls_append_frag(record, &dummy_pfrag, tail);
+ } else {
+ if (prot->tail_size) {
+ char *content_type_addr = page_address(pfrag->page) +
+ pfrag->offset;
+ *content_type_addr = record_type;
+ }
+ tls_append_frag(record, pfrag, tail);
}
- tls_append_frag(record, pfrag, prot->tag_size);
/* fill prepend */
tls_fill_prepend(ctx, skb_frag_address(&record->frags[0]),
- record->len - prot->overhead_size,
+ record->len - prot->overhead_size + prot->tail_size,
record_type);
}
@@ -883,6 +892,7 @@ static int
tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
{
struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(tls_ctx);
+ struct tls_prot_info *prot = &tls_ctx->prot_info;
const struct tls_cipher_desc *cipher_desc;
int err, offset, copy, data_len, pos;
struct sk_buff *skb, *skb_iter;
@@ -894,7 +904,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
rxm = strp_msg(tls_strp_msg(sw_ctx));
- orig_buf = kmalloc(rxm->full_len + TLS_HEADER_SIZE + cipher_desc->iv,
+ orig_buf = kmalloc(rxm->full_len + prot->prepend_size,
sk->sk_allocation);
if (!orig_buf)
return -ENOMEM;
@@ -909,9 +919,8 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
offset = rxm->offset;
sg_init_table(sg, 1);
- sg_set_buf(&sg[0], buf,
- rxm->full_len + TLS_HEADER_SIZE + cipher_desc->iv);
- err = skb_copy_bits(skb, offset, buf, TLS_HEADER_SIZE + cipher_desc->iv);
+ sg_set_buf(&sg[0], buf, rxm->full_len + prot->prepend_size);
+ err = skb_copy_bits(skb, offset, buf, prot->prepend_size);
if (err)
goto free_buf;
@@ -1089,11 +1098,6 @@ int tls_set_device_offload(struct sock *sk)
}
crypto_info = &ctx->crypto_send.info;
- if (crypto_info->version != TLS_1_2_VERSION) {
- rc = -EOPNOTSUPP;
- goto release_netdev;
- }
-
cipher_desc = get_cipher_desc(crypto_info->cipher_type);
if (!cipher_desc || !cipher_desc->offloadable) {
rc = -EINVAL;
@@ -1196,9 +1200,6 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
struct net_device *netdev;
int rc = 0;
- if (ctx->crypto_recv.info.version != TLS_1_2_VERSION)
- return -EOPNOTSUPP;
-
netdev = get_netdev_for_sock(sk);
if (!netdev) {
pr_err_ratelimited("%s: netdev not found\n", __func__);
@@ -1409,12 +1410,22 @@ static struct notifier_block tls_dev_notifier = {
int __init tls_device_init(void)
{
- int err;
+ unsigned char *page_addr;
+ int err, i;
dummy_page = alloc_page(GFP_KERNEL);
if (!dummy_page)
return -ENOMEM;
+ /* Pre-populate dummy_page with identity mapping for all byte values.
+ * This is used as fallback for TLS 1.3 content type when memory
+ * allocation fails. By populating all 256 values, we avoid needing
+ * to validate record_type at runtime.
+ */
+ page_addr = page_address(dummy_page);
+ for (i = 0; i < 256; i++)
+ page_addr[i] = (unsigned char)i;
+
destruct_wq = alloc_workqueue("ktls_device_destruct", WQ_PERCPU, 0);
if (!destruct_wq) {
err = -ENOMEM;
diff --git a/net/tls/tls_device_fallback.c b/net/tls/tls_device_fallback.c
index 3b7d0ab2bcf1..1110f7ac6bcb 100644
--- a/net/tls/tls_device_fallback.c
+++ b/net/tls/tls_device_fallback.c
@@ -37,14 +37,15 @@
#include "tls.h"
-static int tls_enc_record(struct aead_request *aead_req,
+static int tls_enc_record(struct tls_context *tls_ctx,
+ struct aead_request *aead_req,
struct crypto_aead *aead, char *aad,
char *iv, __be64 rcd_sn,
struct scatter_walk *in,
- struct scatter_walk *out, int *in_len,
- struct tls_prot_info *prot)
+ struct scatter_walk *out, int *in_len)
{
unsigned char buf[TLS_HEADER_SIZE + TLS_MAX_IV_SIZE];
+ struct tls_prot_info *prot = &tls_ctx->prot_info;
const struct tls_cipher_desc *cipher_desc;
struct scatterlist sg_in[3];
struct scatterlist sg_out[3];
@@ -55,7 +56,7 @@ static int tls_enc_record(struct aead_request *aead_req,
cipher_desc = get_cipher_desc(prot->cipher_type);
DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
- buf_size = TLS_HEADER_SIZE + cipher_desc->iv;
+ buf_size = prot->prepend_size;
len = min_t(int, *in_len, buf_size);
memcpy_from_scatterwalk(buf, in, len);
@@ -66,16 +67,27 @@ static int tls_enc_record(struct aead_request *aead_req,
return 0;
len = buf[4] | (buf[3] << 8);
- len -= cipher_desc->iv;
+ if (prot->version != TLS_1_3_VERSION)
+ len -= cipher_desc->iv;
tls_make_aad(aad, len - cipher_desc->tag, (char *)&rcd_sn, buf[0], prot);
- memcpy(iv + cipher_desc->salt, buf + TLS_HEADER_SIZE, cipher_desc->iv);
+ if (prot->version == TLS_1_3_VERSION) {
+ void *iv_src = crypto_info_iv(&tls_ctx->crypto_send.info,
+ cipher_desc);
+
+ memcpy(iv + cipher_desc->salt, iv_src, cipher_desc->iv);
+ } else {
+ memcpy(iv + cipher_desc->salt, buf + TLS_HEADER_SIZE,
+ cipher_desc->iv);
+ }
+
+ tls_xor_iv_with_seq(prot, iv, (char *)&rcd_sn);
sg_init_table(sg_in, ARRAY_SIZE(sg_in));
sg_init_table(sg_out, ARRAY_SIZE(sg_out));
- sg_set_buf(sg_in, aad, TLS_AAD_SPACE_SIZE);
- sg_set_buf(sg_out, aad, TLS_AAD_SPACE_SIZE);
+ sg_set_buf(sg_in, aad, prot->aad_size);
+ sg_set_buf(sg_out, aad, prot->aad_size);
scatterwalk_get_sglist(in, sg_in + 1);
scatterwalk_get_sglist(out, sg_out + 1);
@@ -108,13 +120,6 @@ static int tls_enc_record(struct aead_request *aead_req,
return rc;
}
-static void tls_init_aead_request(struct aead_request *aead_req,
- struct crypto_aead *aead)
-{
- aead_request_set_tfm(aead_req, aead);
- aead_request_set_ad(aead_req, TLS_AAD_SPACE_SIZE);
-}
-
static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,
gfp_t flags)
{
@@ -124,14 +129,15 @@ static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,
aead_req = kzalloc(req_size, flags);
if (aead_req)
- tls_init_aead_request(aead_req, aead);
+ aead_request_set_tfm(aead_req, aead);
return aead_req;
}
-static int tls_enc_records(struct aead_request *aead_req,
+static int tls_enc_records(struct tls_context *tls_ctx,
+ struct aead_request *aead_req,
struct crypto_aead *aead, struct scatterlist *sg_in,
struct scatterlist *sg_out, char *aad, char *iv,
- u64 rcd_sn, int len, struct tls_prot_info *prot)
+ u64 rcd_sn, int len)
{
struct scatter_walk out, in;
int rc;
@@ -140,8 +146,8 @@ static int tls_enc_records(struct aead_request *aead_req,
scatterwalk_start(&out, sg_out);
do {
- rc = tls_enc_record(aead_req, aead, aad, iv,
- cpu_to_be64(rcd_sn), &in, &out, &len, prot);
+ rc = tls_enc_record(tls_ctx, aead_req, aead, aad, iv,
+ cpu_to_be64(rcd_sn), &in, &out, &len);
rcd_sn++;
} while (rc == 0 && len);
@@ -314,7 +320,10 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
cipher_desc = get_cipher_desc(tls_ctx->crypto_send.info.cipher_type);
DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
- buf_len = cipher_desc->salt + cipher_desc->iv + TLS_AAD_SPACE_SIZE +
+ aead_request_set_ad(aead_req, tls_ctx->prot_info.aad_size);
+
+ buf_len = cipher_desc->salt + cipher_desc->iv +
+ tls_ctx->prot_info.aad_size +
sync_size + cipher_desc->tag;
buf = kmalloc(buf_len, GFP_ATOMIC);
if (!buf)
@@ -324,7 +333,7 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
salt = crypto_info_salt(&tls_ctx->crypto_send.info, cipher_desc);
memcpy(iv, salt, cipher_desc->salt);
aad = buf + cipher_desc->salt + cipher_desc->iv;
- dummy_buf = aad + TLS_AAD_SPACE_SIZE;
+ dummy_buf = aad + tls_ctx->prot_info.aad_size;
nskb = alloc_skb(skb_headroom(skb) + skb->len, GFP_ATOMIC);
if (!nskb)
@@ -335,9 +344,8 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
fill_sg_out(sg_out, buf, tls_ctx, nskb, tcp_payload_offset,
payload_len, sync_size, dummy_buf);
- if (tls_enc_records(aead_req, ctx->aead_send, sg_in, sg_out, aad, iv,
- rcd_sn, sync_size + payload_len,
- &tls_ctx->prot_info) < 0)
+ if (tls_enc_records(tls_ctx, aead_req, ctx->aead_send, sg_in, sg_out,
+ aad, iv, rcd_sn, sync_size + payload_len) < 0)
goto free_nskb;
complete_skb(nskb, skb, tcp_payload_offset);
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index fd39acf41a61..fd04857fa0ab 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -711,49 +711,64 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
}
if (tx) {
- rc = tls_set_device_offload(sk);
- conf = TLS_HW;
- if (!rc) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
- } else {
- rc = tls_set_sw_offload(sk, 1,
- update ? crypto_info : NULL);
- if (rc)
- goto err_crypto_info;
-
- if (update) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
- } else {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXSW);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+ if (update && ctx->tx_conf == TLS_HW) {
+ rc = -EOPNOTSUPP;
+ goto err_crypto_info;
+ }
+
+ if (!update) {
+ rc = tls_set_device_offload(sk);
+ conf = TLS_HW;
+ if (!rc) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+ goto out;
}
- conf = TLS_SW;
}
- } else {
- rc = tls_set_device_offload_rx(sk, ctx);
- conf = TLS_HW;
- if (!rc) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+
+ rc = tls_set_sw_offload(sk, 1, update ? crypto_info : NULL);
+ if (rc)
+ goto err_crypto_info;
+
+ if (update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
} else {
- rc = tls_set_sw_offload(sk, 0,
- update ? crypto_info : NULL);
- if (rc)
- goto err_crypto_info;
-
- if (update) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
- } else {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXSW);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXSW);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+ }
+ conf = TLS_SW;
+ } else {
+ if (update && ctx->rx_conf == TLS_HW) {
+ rc = -EOPNOTSUPP;
+ goto err_crypto_info;
+ }
+
+ if (!update) {
+ rc = tls_set_device_offload_rx(sk, ctx);
+ conf = TLS_HW;
+ if (!rc) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+ tls_sw_strparser_arm(sk, ctx);
+ goto out;
}
- conf = TLS_SW;
}
- if (!update)
+
+ rc = tls_set_sw_offload(sk, 0, update ? crypto_info : NULL);
+ if (rc)
+ goto err_crypto_info;
+
+ if (update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+ } else {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXSW);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
tls_sw_strparser_arm(sk, ctx);
+ }
+ conf = TLS_SW;
}
+out:
if (tx)
ctx->tx_conf = conf;
else
--
2.25.1
^ permalink raw reply related
* [PATCH v13 2/6] net/mlx5e: add TLS 1.3 hardware offload support
From: Rishikesh Jethwani @ 2026-04-29 18:10 UTC (permalink / raw)
To: netdev
Cc: saeedm, tariqt, mbloch, borisp, john.fastabend, kuba, sd, davem,
pabeni, edumazet, leon, Rishikesh Jethwani
In-Reply-To: <20260429181016.3164935-1-rjethwani@purestorage.com>
Enable TLS 1.3 TX/RX hardware offload on ConnectX-6 Dx and newer
crypto-enabled adapters.
Key changes:
- Add TLS 1.3 capability checking and version validation
- Use MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3 (0x3) for crypto context
- Handle TLS 1.3 IV format: full 12-byte IV copied to gcm_iv +
implicit_iv (vs TLS 1.2's 4-byte salt only)
Tested with TLS 1.3 AES-GCM-128 and AES-GCM-256 cipher suites.
Signed-off-by: Rishikesh Jethwani <rjethwani@purestorage.com>
---
.../ethernet/mellanox/mlx5/core/en_accel/ktls.h | 8 +++++++-
.../mellanox/mlx5/core/en_accel/ktls_txrx.c | 14 +++++++++++---
2 files changed, 18 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
index 07a04a142a2e..0469ca6a0762 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
@@ -30,7 +30,9 @@ static inline bool mlx5e_is_ktls_device(struct mlx5_core_dev *mdev)
return false;
return (MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_128) ||
- MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256));
+ MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256) ||
+ MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_128) ||
+ MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_256));
}
static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
@@ -40,10 +42,14 @@ static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
case TLS_CIPHER_AES_GCM_128:
if (crypto_info->version == TLS_1_2_VERSION)
return MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_128);
+ else if (crypto_info->version == TLS_1_3_VERSION)
+ return MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_128);
break;
case TLS_CIPHER_AES_GCM_256:
if (crypto_info->version == TLS_1_2_VERSION)
return MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256);
+ else if (crypto_info->version == TLS_1_3_VERSION)
+ return MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_256);
break;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
index 570a912dd6fa..f3f1be1d4034 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
@@ -6,6 +6,7 @@
enum {
MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 = 0x2,
+ MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3 = 0x3,
};
enum {
@@ -15,8 +16,10 @@ enum {
#define EXTRACT_INFO_FIELDS do { \
salt = info->salt; \
rec_seq = info->rec_seq; \
+ iv = info->iv; \
salt_sz = sizeof(info->salt); \
rec_seq_sz = sizeof(info->rec_seq); \
+ iv_sz = sizeof(info->iv); \
} while (0)
static void
@@ -24,9 +27,9 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
union mlx5e_crypto_info *crypto_info,
u32 key_id, u32 resync_tcp_sn)
{
+ u16 salt_sz, rec_seq_sz, iv_sz;
+ char *salt, *rec_seq, *iv;
char *initial_rn, *gcm_iv;
- u16 salt_sz, rec_seq_sz;
- char *salt, *rec_seq;
u8 tls_version;
u8 *ctx;
@@ -59,7 +62,12 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
memcpy(gcm_iv, salt, salt_sz);
memcpy(initial_rn, rec_seq, rec_seq_sz);
- tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+ if (crypto_info->crypto_info.version == TLS_1_3_VERSION) {
+ memcpy(gcm_iv + salt_sz, iv, iv_sz);
+ tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3;
+ } else {
+ tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+ }
MLX5_SET(tls_static_params, ctx, tls_version, tls_version);
MLX5_SET(tls_static_params, ctx, const_1, 1);
--
2.25.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox