* RE: [RFC v3 2/3] lib: add fastmem library
From: Morten Brørup @ 2026-05-28 9:11 UTC (permalink / raw)
To: Bruce Richardson, Vipin Varghese, Mattias Rönnblom, dev
Cc: Konstantin Ananyev, Mattias Rönnblom, Yogaraj Baskaravel,
Stephen Hemminger
In-Reply-To: <20260527173042.93867-3-hofors@lysator.liu.se>
> +/**
> + * Pre-reserve backing memory.
> + *
> + * Ensures that at least @p size bytes of memzone-backed memory are
> + * available to the allocator on @p socket_id, reserving additional
> + * memzones from EAL as needed to reach that total. Subsequent
> + * allocations served from the pre-reserved memory do not incur
> + * memzone-reservation cost.
> + *
> + * The reservation is cumulative: repeated calls to
> + * rte_fastmem_reserve() with the same @p socket_id grow the
> + * reservation monotonically. Reserved memory is never returned to
> + * the system during the allocator's lifetime.
> + *
> + * A typical use is to call rte_fastmem_reserve() once at
> + * application startup, with a size chosen to cover the expected
> + * steady-state working set. Allocations and frees during
> + * steady-state operation then avoid memzone reservations entirely.
> + *
> + * @param size
> + * The minimum amount of backing memory, in bytes, to make
> + * available on @p socket_id. The allocator may reserve more than
> + * the requested amount due to internal rounding (e.g., to memzone
> + * or block granularity).
> + *
> + * @param socket_id
> + * The NUMA socket on which to reserve memory, or SOCKET_ID_ANY
> + * to leave the choice to the allocator. With SOCKET_ID_ANY, the
> + * allocator starts on the calling lcore's socket (or the first
> + * configured socket if the caller is not bound to one) and falls
> + * back to other sockets if the preferred socket cannot satisfy
> + * the reservation.
> + *
> + * @return
> + * - 0: Success.
> + * - -ENOMEM: Insufficient huge-page memory to satisfy the request.
> + * - -EINVAL: Invalid @p socket_id.
> + */
> +__rte_experimental
> +int
> +rte_fastmem_reserve(size_t size, int socket_id);
@Bruce,
I vaguely recall that we discussed something about busses and sockets a long time ago, but I cannot remember the details.
Is socket_id the right type (and parameter name) to identify a memory bus?
@Vipin,
You have been working on topology awareness. Same question to you:
Is socket_id the right type (and parameter name) to identify a memory bus?
^ permalink raw reply
* RE: [RFC v3 0/3] lib/fastmem: fast small-object allocator
From: Morten Brørup @ 2026-05-28 9:02 UTC (permalink / raw)
To: Mattias Rönnblom, dev
Cc: Konstantin Ananyev, Mattias Rönnblom, Yogaraj Baskaravel,
Stephen Hemminger, Bruce Richardson
In-Reply-To: <20260527173042.93867-1-hofors@lysator.liu.se>
> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se]
> Sent: Wednesday, 27 May 2026 19.31
>
>
> This RFC introduces fastmem, a general-purpose small-object allocator
> for DPDK. It is intended to replace per-type mempools with a single
> allocator that handles arbitrary sizes, grows on demand, and matches
> mempool-level performance on the hot path.
>
> Motivation
> ----------
>
> DPDK applications commonly maintain many mempools — one per object
> type (connections, sessions, timers, work items). Each must be sized
> up front, wastes memory when over-provisioned, and cannot serve
> objects of a different size. Fastmem eliminates this by accepting
> arbitrary sizes at runtime, backed by a slab allocator that
> repurposes memory across size classes as demand shifts.
>
> Design
> ------
>
> Three-layer architecture:
>
> 1. Backing memory: 128 MiB IOVA-contiguous memzones from EAL,
> reserved lazily (or pre-reserved for deterministic latency).
>
> 2. Slabs: 2 MiB, 2 MiB-aligned regions carved from memzones.
> The alignment enables O(1) slab lookup from any object pointer
> via bitmask — no radix tree or index structure. Slabs move
> freely between 18 power-of-2 size classes (8 B to 1 MiB).
>
> 3. Per-lcore caches: bounded LIFO stacks (no locks on the hot
> path). Cache misses trigger bulk transfers to/from the shared
> bin under a spinlock.
>
> Key properties:
>
> - Zero per-object metadata in the production build.
> - NUMA-aware, with per-socket bins and free-slab pools.
> - DMA-usable memory with O(1) virt-to-IOVA translation.
> - Bulk alloc/free with all-or-nothing semantics.
> - Backing memory never returned during lifetime (slabs recycled).
> - Non-EAL threads supported (bypass cache, take bin lock).
> - Secondary process support (lazy attach, no per-lcore caches).
>
> API surface
> -----------
>
> rte_fastmem_init / deinit
> rte_fastmem_reserve
> rte_fastmem_set_limit / get_limit
> rte_fastmem_alloc / alloc_socket
> rte_fastmem_realloc
> rte_fastmem_alloc_bulk / alloc_bulk_socket
> rte_fastmem_free / free_bulk
> rte_fastmem_hlookup / halloc / halloc_bulk / hfree / hfree_bulk
> rte_fastmem_virt2iova
> rte_fastmem_cache_flush
> rte_fastmem_max_size / classes
> rte_fastmem_stats / stats_class / stats_lcore / stats_lcore_class
> rte_fastmem_stats_reset
>
> All APIs are marked __rte_experimental.
>
> Performance
> -----------
>
> The single-object hot path is roughly 2–3× the cost of mempool
> and an order of magnitude faster than rte_malloc. Under
> multi-lcore contention, fastmem scales similarly to mempool,
> while rte_malloc collapses.
>
> Limitations
> -----------
>
> - Maximum allocation: 1 MiB. Larger requests should use rte_malloc.
> - Power-of-2 classes only; worst-case internal fragmentation ~50%.
> - Backing memory not reclaimable short of deinit.
>
> Future work
> -----------
>
> - Lcore-affine allocations (false-sharing-free by construction).
> - Mempool ops driver for transparent drop-in use.
Regarding mempool support.
As you already mentioned, some mempools hold fully or partially initialized objects.
Releasing such an object to the heap would require an ability to reconstruct it on allocation from the heap.
In some cases, object reconstruction might be possible through callbacks or some other means.
And in some cases, object reconstruction might be practically impossible.
Under all circumstances, object reconstruction has a performance cost, which needs to be weighed up against the memory savings by freeing the objects back to the heap. This consideration is specific to each mempool, the kind of objects it holds, and how the mempool is being used.
If we look specifically at the mbuf mempool, an mbuf comprises of metadata (struct rte_mbuf and possibly struct rte_mbuf_ext_shared_info) and the packet buffer itself.
The mbuf structure supports using external buffers for the packet buffer, which does not need reconstruction if dynamically allocated from the heap.
It seems viable to keep the metadata parts of the mbufs in a mempool, and dynamically allocate/free their packet buffers on mbuf allocation/free. A shim mempool ops driver could relatively easily implement this. It might require a few additions to the mbuf and/or mempool libraries too, but that would be acceptable.
<feature creep>
Another thing regarding mbuf packet data:
Some NICs require packet buffers of 2048 bytes, but we also allocate a headroom of default 128 bytes in front of it, so the default packet buffer size (RTE_MBUF_DEFAULT_BUF_SIZE) is not 2^N, but 2048+128=2176 [1].
[1]: https://elixir.bootlin.com/dpdk/v26.03/source/lib/mbuf/rte_mbuf_core.h#L408
Allocating fastmem buffers of 4 KiB and only use 2.1 KiB seems wasteful.
Could the fastmem library support a shortlist of magic object sizes that are not 2^N?
The magic sizes should be explicitly configured at run-time. (The mbuf library must inform the fastmem library of the requested data_room_size before it populates the mbuf mempool.)
The shortlist should have a fixed max length, maybe 4 as default, preferably build-time configurable.
Removing a magic size from the shortlist need not be supported. Only adding magic sizes is required.
The magic sizes will be relatively large (assume 512 bytes or more), so adding a fastlib object metadata structure to each magic-sized object is acceptable, if necessary.
From a fastmem library perspective, WDYT?
</feature creep>
> - Debug mode (cookies, double-free detection, poison-on-free).
> - Telemetry integration.
> - EAL integration, allowing EAL-internal subsystems to use
> fastmem for their small-object allocations.
>
> Changes in RFC v3:
> - Add rte_fastmem_realloc() with full test coverage.
> - Add __rte_malloc/__rte_dealloc compiler attributes; remove
> incorrect __rte_alloc_size/__rte_alloc_align.
> - Extract normalize_align() helper; remove redundant inline
> directives.
> - Merge lifecycle and functional test suites.
> - Add realloc subsection to programming guide.
>
> Changes in RFC v2:
> - Fix cross-socket deinit use-after-free.
> - Add secondary process support.
> - Add handle-based allocation API.
> - Fix clang warnings; misc cleanup.
>
>
> Mattias Rönnblom (3):
> doc: add fastmem programming guide
> lib: add fastmem library
> app/test: add fastmem test suite
>
> app/test/meson.build | 3 +
> app/test/test_fastmem.c | 1801 +++++++++++++++++++++++++
> app/test/test_fastmem_perf.c | 1040 ++++++++++++++
> app/test/test_fastmem_profile.c | 157 +++
> doc/api/doxy-api-index.md | 1 +
> doc/api/doxy-api.conf.in | 1 +
> doc/guides/prog_guide/fastmem_lib.rst | 328 +++++
> doc/guides/prog_guide/index.rst | 1 +
> lib/fastmem/meson.build | 6 +
> lib/fastmem/rte_fastmem.c | 1748 ++++++++++++++++++++++++
> lib/fastmem/rte_fastmem.h | 815 +++++++++++
> lib/meson.build | 1 +
> 12 files changed, 5902 insertions(+)
> create mode 100644 app/test/test_fastmem.c
> create mode 100644 app/test/test_fastmem_perf.c
> create mode 100644 app/test/test_fastmem_profile.c
> create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
> create mode 100644 lib/fastmem/meson.build
> create mode 100644 lib/fastmem/rte_fastmem.c
> create mode 100644 lib/fastmem/rte_fastmem.h
>
> --
> 2.43.0
^ permalink raw reply
* [PATCH] net/sxe2: fix 32-bit SSE build
From: Thomas Monjalon @ 2026-05-28 8:47 UTC (permalink / raw)
To: dev; +Cc: Jie Liu
Seen in OBS on i586 Debian:
from ../drivers/net/sxe2/sxe2_txrx_vec_sse.c:5:
In function ‘_mm_loadu_si128’,
inlined from ‘rte_memcpy’
inlined from ‘sxe2_rx_pkts_refactor’
at ../drivers/net/sxe2/sxe2_txrx_vec_common.h:233:2:
/usr/lib/gcc/i686-linux-gnu/12/include/emmintrin.h:703:10: error:
array subscript 8 is outside array bounds of ‘struct rte_mbuf *[32]’
The important options to reproduce are "-m32 -O2 -march=corei7".
In 32-bit build the pointer array done_pkts[32] is smaller:
32 * 4 = 128 bytes
so an SSE access would be outside the bound.
The libc memcpy does not trigger such warning
and is a good choice to copy an array of pointers.
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
drivers/net/sxe2/sxe2_txrx_vec_common.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/sxe2/sxe2_txrx_vec_common.h b/drivers/net/sxe2/sxe2_txrx_vec_common.h
index 99e1663f03..6b1649c390 100644
--- a/drivers/net/sxe2/sxe2_txrx_vec_common.h
+++ b/drivers/net/sxe2/sxe2_txrx_vec_common.h
@@ -230,7 +230,8 @@ sxe2_rx_pkts_refactor(struct sxe2_rx_queue *rxq,
}
rxq->pkt_first_seg = first_seg;
rxq->pkt_last_seg = last_seg;
- rte_memcpy(mbuf_bufs, done_pkts, done_num * (sizeof(struct rte_mbuf *)));
+ memcpy(mbuf_bufs, done_pkts, done_num * sizeof(*done_pkts));
return done_num;
}
+
#endif /* SXE2_TXRX_VEC_COMMON_H */
--
2.54.0
^ permalink raw reply related
* [v2] crypto/openssl: update to OpenSSL 3.0 minimum version
From: Emma Finn @ 2026-05-28 8:03 UTC (permalink / raw)
To: Kai Ji; +Cc: dev, Emma Finn
In-Reply-To: <20260527110227.516868-1-emma.finn@intel.com>
Update the OpenSSL PMD to require OpenSSL 3.0.0 as the minimum
supported version, removing all compatibility code for earlier
versions (1.0.1, 1.1.0, 1.1.1).
Signed-off-by: Emma Finn <emma.finn@intel.com>
---
*v2: skip build if openssl v3.0 dependency is not met.
---
doc/guides/cryptodevs/openssl.rst | 4 +-
doc/guides/rel_notes/release_26_07.rst | 5 +
drivers/crypto/openssl/compat.h | 203 ------
drivers/crypto/openssl/meson.build | 4 +-
drivers/crypto/openssl/openssl_pmd_private.h | 30 -
drivers/crypto/openssl/rte_openssl_pmd.c | 648 +------------------
drivers/crypto/openssl/rte_openssl_pmd_ops.c | 206 ------
7 files changed, 21 insertions(+), 1079 deletions(-)
diff --git a/doc/guides/cryptodevs/openssl.rst b/doc/guides/cryptodevs/openssl.rst
index 9d94668a9a..b4e2a014e2 100644
--- a/doc/guides/cryptodevs/openssl.rst
+++ b/doc/guides/cryptodevs/openssl.rst
@@ -74,9 +74,9 @@ To compile the OpenSSL PMD the openssl library must be installed. It will
then be picked up by the Meson/Ninja build system.
To ensure that you have the latest security fixes it is recommended that you
-use version 1.1.1g or newer.
+use the latest stable version of OpenSSL 3.x.
-* 1.1.1g, 2020-Apr-21. https://www.openssl.org/source/
+* OpenSSL 3.0.0 or newer: https://www.openssl.org/source/
Initialization
--------------
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 58d782f77e..989d54f7b7 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -90,6 +90,11 @@ Removed Items
Also, make sure to start the actual text at the margin.
=======================================================
+* crypto/openssl: Removed support for OpenSSL 1.x versions from the OpenSSL crypto PMD.
+
+ The OpenSSL crypto PMD now requires OpenSSL 3.0 as the minimum version,
+ and all compatibility code for OpenSSL 1.0.1, 1.1.0, and 1.1.1 versions has been removed.
+
API Changes
-----------
diff --git a/drivers/crypto/openssl/compat.h b/drivers/crypto/openssl/compat.h
index e1814fea8c..14104dbf2e 100644
--- a/drivers/crypto/openssl/compat.h
+++ b/drivers/crypto/openssl/compat.h
@@ -5,7 +5,6 @@
#ifndef __RTA_COMPAT_H__
#define __RTA_COMPAT_H__
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
static __rte_always_inline void
free_hmac_ctx(EVP_MAC_CTX *ctx)
{
@@ -17,120 +16,7 @@ free_cmac_ctx(EVP_MAC_CTX *ctx)
{
EVP_MAC_CTX_free(ctx);
}
-#else
-static __rte_always_inline void
-free_hmac_ctx(HMAC_CTX *ctx)
-{
- HMAC_CTX_free(ctx);
-}
-
-static __rte_always_inline void
-free_cmac_ctx(CMAC_CTX *ctx)
-{
- CMAC_CTX_free(ctx);
-}
-#endif
-
-#if (OPENSSL_VERSION_NUMBER < 0x10100000L)
-
-static __rte_always_inline int
-set_rsa_params(RSA *rsa, BIGNUM *p, BIGNUM *q)
-{
- rsa->p = p;
- rsa->q = q;
- return 0;
-}
-
-static __rte_always_inline int
-set_rsa_crt_params(RSA *rsa, BIGNUM *dmp1, BIGNUM *dmq1, BIGNUM *iqmp)
-{
- rsa->dmp1 = dmp1;
- rsa->dmq1 = dmq1;
- rsa->iqmp = iqmp;
- return 0;
-}
-
-static __rte_always_inline int
-set_rsa_keys(RSA *rsa, BIGNUM *n, BIGNUM *e, BIGNUM *d)
-{
- rsa->n = n;
- rsa->e = e;
- rsa->d = d;
- return 0;
-}
-
-static __rte_always_inline int
-set_dh_params(DH *dh, BIGNUM *p, BIGNUM *g)
-{
- dh->p = p;
- dh->q = NULL;
- dh->g = g;
- return 0;
-}
-
-static __rte_always_inline int
-set_dh_priv_key(DH *dh, BIGNUM *priv_key)
-{
- dh->priv_key = priv_key;
- return 0;
-}
-
-static __rte_always_inline int
-set_dsa_params(DSA *dsa, BIGNUM *p, BIGNUM *q, BIGNUM *g)
-{
- dsa->p = p;
- dsa->q = q;
- dsa->g = g;
- return 0;
-}
-
-static __rte_always_inline void
-get_dh_pub_key(DH *dh, const BIGNUM **pub_key)
-{
- *pub_key = dh->pub_key;
-}
-
-static __rte_always_inline void
-get_dh_priv_key(DH *dh, const BIGNUM **priv_key)
-{
- *priv_key = dh->priv_key;
-}
-
-static __rte_always_inline void
-set_dsa_sign(DSA_SIG *sign, BIGNUM *r, BIGNUM *s)
-{
- sign->r = r;
- sign->s = s;
-}
-
-static __rte_always_inline void
-get_dsa_sign(DSA_SIG *sign, const BIGNUM **r, const BIGNUM **s)
-{
- *r = sign->r;
- *s = sign->s;
-}
-
-static __rte_always_inline int
-set_dsa_keys(DSA *dsa, BIGNUM *pub, BIGNUM *priv)
-{
- dsa->pub_key = pub;
- dsa->priv_key = priv;
- return 0;
-}
-
-static __rte_always_inline void
-set_dsa_pub_key(DSA *dsa, BIGNUM *pub)
-{
- dsa->pub_key = pub;
-}
-
-static __rte_always_inline void
-get_dsa_priv_key(DSA *dsa, BIGNUM **priv_key)
-{
- *priv_key = dsa->priv_key;
-}
-#elif (OPENSSL_VERSION_NUMBER >= 0x30000000L)
static __rte_always_inline void
set_dsa_sign(DSA_SIG *sign, BIGNUM *r, BIGNUM *s)
{
@@ -142,94 +28,5 @@ get_dsa_sign(DSA_SIG *sign, const BIGNUM **r, const BIGNUM **s)
{
DSA_SIG_get0(sign, r, s);
}
-#else
-
-static __rte_always_inline int
-set_rsa_params(RSA *rsa, BIGNUM *p, BIGNUM *q)
-{
- return !(RSA_set0_factors(rsa, p, q));
-}
-
-static __rte_always_inline int
-set_rsa_crt_params(RSA *rsa, BIGNUM *dmp1, BIGNUM *dmq1, BIGNUM *iqmp)
-{
- return !(RSA_set0_crt_params(rsa, dmp1, dmq1, iqmp));
-}
-
-/* n, e must be non-null, d can be NULL */
-
-static __rte_always_inline int
-set_rsa_keys(RSA *rsa, BIGNUM *n, BIGNUM *e, BIGNUM *d)
-{
- return !(RSA_set0_key(rsa, n, e, d));
-}
-
-static __rte_always_inline int
-set_dh_params(DH *dh, BIGNUM *p, BIGNUM *g)
-{
- return !(DH_set0_pqg(dh, p, NULL, g));
-}
-
-static __rte_always_inline int
-set_dh_priv_key(DH *dh, BIGNUM *priv_key)
-{
- return !(DH_set0_key(dh, NULL, priv_key));
-}
-
-static __rte_always_inline void
-get_dh_pub_key(DH *dh_key, const BIGNUM **pub_key)
-{
- DH_get0_key(dh_key, pub_key, NULL);
-}
-
-static __rte_always_inline void
-get_dh_priv_key(DH *dh_key, const BIGNUM **priv_key)
-{
- DH_get0_key(dh_key, NULL, priv_key);
-}
-
-static __rte_always_inline int
-set_dsa_params(DSA *dsa, BIGNUM *p, BIGNUM *q, BIGNUM *g)
-{
- return !(DSA_set0_pqg(dsa, p, q, g));
-}
-
-static __rte_always_inline void
-set_dsa_priv_key(DSA *dsa, BIGNUM *priv_key)
-{
- DSA_set0_key(dsa, NULL, priv_key);
-}
-
-static __rte_always_inline void
-set_dsa_sign(DSA_SIG *sign, BIGNUM *r, BIGNUM *s)
-{
- DSA_SIG_set0(sign, r, s);
-}
-
-static __rte_always_inline void
-get_dsa_sign(DSA_SIG *sign, const BIGNUM **r, const BIGNUM **s)
-{
- DSA_SIG_get0(sign, r, s);
-}
-
-static __rte_always_inline int
-set_dsa_keys(DSA *dsa, BIGNUM *pub, BIGNUM *priv)
-{
- return !(DSA_set0_key(dsa, pub, priv));
-}
-
-static __rte_always_inline void
-set_dsa_pub_key(DSA *dsa, BIGNUM *pub_key)
-{
- DSA_set0_key(dsa, pub_key, NULL);
-}
-
-static __rte_always_inline void
-get_dsa_priv_key(DSA *dsa, const BIGNUM **priv_key)
-{
- DSA_get0_key(dsa, NULL, priv_key);
-}
-
-#endif /* version < 10100000 */
#endif /* __RTA_COMPAT_H__ */
diff --git a/drivers/crypto/openssl/meson.build b/drivers/crypto/openssl/meson.build
index af469a9827..0d82c42764 100644
--- a/drivers/crypto/openssl/meson.build
+++ b/drivers/crypto/openssl/meson.build
@@ -7,10 +7,10 @@ if is_windows
subdir_done()
endif
-dep = dependency('libcrypto', required: false, method: 'pkg-config')
+dep = dependency('libcrypto', required: false, method: 'pkg-config', version: '>= 3.0.0')
if not dep.found()
build = false
- reason = 'missing dependency, "libcrypto"'
+ reason = 'missing dependency, "libcrypto >= 3.0.0"'
endif
deps += 'bus_vdev'
sources = files('rte_openssl_pmd.c', 'rte_openssl_pmd_ops.c')
diff --git a/drivers/crypto/openssl/openssl_pmd_private.h b/drivers/crypto/openssl/openssl_pmd_private.h
index d5a751600a..ab40012d61 100644
--- a/drivers/crypto/openssl/openssl_pmd_private.h
+++ b/drivers/crypto/openssl/openssl_pmd_private.h
@@ -13,10 +13,8 @@
#include <openssl/dh.h>
#include <openssl/dsa.h>
#include <openssl/ec.h>
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
#include <openssl/provider.h>
#include <openssl/core_names.h>
-#endif
#define CRYPTODEV_NAME_OPENSSL_PMD crypto_openssl
/**< Open SSL Crypto PMD device name */
@@ -84,13 +82,8 @@ struct evp_ctx_pair {
EVP_CIPHER_CTX *cipher;
union {
EVP_MD_CTX *auth;
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
EVP_MAC_CTX *hmac;
EVP_MAC_CTX *cmac;
-#else
- HMAC_CTX *hmac;
- CMAC_CTX *cmac;
-#endif
};
};
@@ -153,24 +146,13 @@ struct __rte_cache_aligned openssl_session {
/**< pointer to EVP key */
const EVP_MD *evp_algo;
/**< pointer to EVP algorithm function */
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
EVP_MAC_CTX * ctx;
-# else
- HMAC_CTX *ctx;
-# endif
/**< pointer to EVP context structure */
} hmac;
struct {
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
EVP_MAC_CTX * ctx;
/**< pointer to EVP context structure */
-# else
- const EVP_CIPHER * evp_algo;
- /**< pointer to EVP algorithm function */
- CMAC_CTX *ctx;
- /**< pointer to EVP context structure */
-# endif
} cmac;
};
@@ -198,9 +180,7 @@ struct __rte_cache_aligned openssl_asym_session {
struct rsa {
RSA *rsa;
uint32_t pad;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
EVP_PKEY_CTX * ctx;
-#endif
} r;
struct exp {
BIGNUM *exp;
@@ -216,38 +196,28 @@ struct __rte_cache_aligned openssl_asym_session {
uint32_t key_op;
BIGNUM *p;
BIGNUM *g;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD * param_bld;
OSSL_PARAM_BLD *param_bld_peer;
-#endif
} dh;
struct {
DSA *dsa;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD * param_bld;
BIGNUM *p;
BIGNUM *g;
BIGNUM *q;
BIGNUM *priv_key;
-#endif
} s;
struct {
uint8_t curve_id;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
EC_GROUP * group;
BIGNUM *priv_key;
-#endif
} ec;
struct {
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM * params;
-#endif
} sm2;
struct {
uint8_t curve_id;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM * params;
-#endif
} eddsa;
struct {
uint8_t type;
diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index c34efb8ad0..8748ef6195 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -19,35 +19,14 @@
#include "openssl_pmd_private.h"
#include "compat.h"
-#define DES_BLOCK_SIZE 8
-
-static uint8_t cryptodev_driver_id;
-
-#if (OPENSSL_VERSION_NUMBER < 0x10100000L)
-static HMAC_CTX *HMAC_CTX_new(void)
-{
- HMAC_CTX *ctx = OPENSSL_malloc(sizeof(*ctx));
-
- if (ctx != NULL)
- HMAC_CTX_init(ctx);
- return ctx;
-}
-
-static void HMAC_CTX_free(HMAC_CTX *ctx)
-{
- if (ctx != NULL) {
- HMAC_CTX_cleanup(ctx);
- OPENSSL_free(ctx);
- }
-}
-#endif
-
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
-
#include <openssl/provider.h>
#include <openssl/core_names.h>
#include <openssl/param_build.h>
+#define DES_BLOCK_SIZE 8
+
+static uint8_t cryptodev_driver_id;
+
#define MAX_OSSL_ALGO_NAME_SIZE 16
OSSL_PROVIDER *legacy;
@@ -104,7 +83,6 @@ digest_name_get(enum rte_crypto_auth_algorithm algo)
return NULL;
}
}
-#endif
static int cryptodev_openssl_remove(struct rte_vdev_device *vdev);
@@ -306,14 +284,12 @@ get_auth_algo(enum rte_crypto_auth_algorithm sessalgo,
case RTE_CRYPTO_AUTH_SHA3_512_HMAC:
*algo = EVP_sha3_512();
break;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
case RTE_CRYPTO_AUTH_SHAKE_128:
*algo = EVP_shake128();
break;
case RTE_CRYPTO_AUTH_SHAKE_256:
*algo = EVP_shake256();
break;
-#endif
default:
res = -EINVAL;
break;
@@ -659,12 +635,10 @@ static int
openssl_set_session_auth_parameters(struct openssl_session *sess,
const struct rte_crypto_sym_xform *xform)
{
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
char algo_name[MAX_OSSL_ALGO_NAME_SIZE];
OSSL_PARAM params[2];
const char *algo;
EVP_MAC *mac;
-# endif
/* Select auth generate/verify */
sess->auth.operation = xform->auth.op;
sess->auth.algo = xform->auth.algo;
@@ -708,10 +682,8 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
case RTE_CRYPTO_AUTH_SHA3_256:
case RTE_CRYPTO_AUTH_SHA3_384:
case RTE_CRYPTO_AUTH_SHA3_512:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
case RTE_CRYPTO_AUTH_SHAKE_128:
case RTE_CRYPTO_AUTH_SHAKE_256:
-#endif
sess->auth.mode = OPENSSL_AUTH_AS_AUTH;
if (get_auth_algo(xform->auth.algo,
&sess->auth.auth.evp_algo) != 0)
@@ -720,7 +692,6 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
break;
case RTE_CRYPTO_AUTH_AES_CMAC:
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
if (xform->auth.key.length == 16)
algo = SN_aes_128_cbc;
else if (xform->auth.key.length == 24)
@@ -745,22 +716,8 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
xform->auth.key.length,
params) != 1)
return -EINVAL;
-# else
- sess->auth.mode = OPENSSL_AUTH_AS_CMAC;
- sess->auth.cmac.ctx = CMAC_CTX_new();
- if (get_cipher_algo(RTE_CRYPTO_CIPHER_AES_CBC,
- xform->auth.key.length,
- &sess->auth.cmac.evp_algo) != 0)
- return -EINVAL;
- if (CMAC_Init(sess->auth.cmac.ctx,
- xform->auth.key.data,
- xform->auth.key.length,
- sess->auth.cmac.evp_algo, NULL) != 1)
- return -EINVAL;
-# endif
break;
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
case RTE_CRYPTO_AUTH_MD5_HMAC:
case RTE_CRYPTO_AUTH_SHA1_HMAC:
case RTE_CRYPTO_AUTH_SHA224_HMAC:
@@ -794,30 +751,6 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
params) != 1)
return -EINVAL;
break;
-# else
- case RTE_CRYPTO_AUTH_MD5_HMAC:
- case RTE_CRYPTO_AUTH_SHA1_HMAC:
- case RTE_CRYPTO_AUTH_SHA224_HMAC:
- case RTE_CRYPTO_AUTH_SHA256_HMAC:
- case RTE_CRYPTO_AUTH_SHA384_HMAC:
- case RTE_CRYPTO_AUTH_SHA512_HMAC:
- case RTE_CRYPTO_AUTH_SHA3_224_HMAC:
- case RTE_CRYPTO_AUTH_SHA3_256_HMAC:
- case RTE_CRYPTO_AUTH_SHA3_384_HMAC:
- case RTE_CRYPTO_AUTH_SHA3_512_HMAC:
- sess->auth.mode = OPENSSL_AUTH_AS_HMAC;
- sess->auth.hmac.ctx = HMAC_CTX_new();
- if (get_auth_algo(xform->auth.algo,
- &sess->auth.hmac.evp_algo) != 0)
- return -EINVAL;
-
- if (HMAC_Init_ex(sess->auth.hmac.ctx,
- xform->auth.key.data,
- xform->auth.key.length,
- sess->auth.hmac.evp_algo, NULL) != 1)
- return -EINVAL;
- break;
-# endif
default:
return -ENOTSUP;
}
@@ -1295,10 +1228,6 @@ process_openssl_auth_encryption_gcm(struct rte_mbuf *mbuf_src, int offset,
uint8_t *dst, uint8_t *tag, EVP_CIPHER_CTX *ctx)
{
int len = 0;
-#if OPENSSL_VERSION_NUMBER < 0x10100000L
- int unused = 0;
- uint8_t empty[] = {};
-#endif
if (EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv) <= 0)
goto process_auth_encryption_gcm_err;
@@ -1312,12 +1241,6 @@ process_openssl_auth_encryption_gcm(struct rte_mbuf *mbuf_src, int offset,
srclen, ctx, 0))
goto process_auth_encryption_gcm_err;
-#if OPENSSL_VERSION_NUMBER < 0x10100000L
- /* Workaround open ssl bug in version less then 1.0.1f */
- if (EVP_EncryptUpdate(ctx, empty, &unused, empty, 0) <= 0)
- goto process_auth_encryption_gcm_err;
-#endif
-
if (EVP_EncryptFinal_ex(ctx, dst, &len) <= 0)
goto process_auth_encryption_gcm_err;
@@ -1379,10 +1302,6 @@ process_openssl_auth_decryption_gcm(struct rte_mbuf *mbuf_src, int offset,
uint8_t *dst, uint8_t *tag, EVP_CIPHER_CTX *ctx)
{
int len = 0;
-#if OPENSSL_VERSION_NUMBER < 0x10100000L
- int unused = 0;
- uint8_t empty[] = {};
-#endif
if (EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, 16, tag) <= 0)
goto process_auth_decryption_gcm_err;
@@ -1399,12 +1318,6 @@ process_openssl_auth_decryption_gcm(struct rte_mbuf *mbuf_src, int offset,
srclen, ctx, 0))
goto process_auth_decryption_gcm_err;
-#if OPENSSL_VERSION_NUMBER < 0x10100000L
- /* Workaround open ssl bug in version less then 1.0.1f */
- if (EVP_DecryptUpdate(ctx, empty, &unused, empty, 0) <= 0)
- goto process_auth_decryption_gcm_err;
-#endif
-
if (EVP_DecryptFinal_ex(ctx, dst, &len) <= 0)
return -EFAULT;
@@ -1500,17 +1413,11 @@ process_openssl_auth(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
process_auth_final:
/* SHAKE algorithms are XOFs and require EVP_DigestFinalXOF */
if (algo == EVP_shake128() || algo == EVP_shake256()) {
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
/* Set XOF output length before calling EVP_DigestFinalXOF */
if (EVP_MD_CTX_ctrl(ctx, EVP_MD_CTRL_XOF_LEN, digest_length, NULL) <= 0)
goto process_auth_err;
if (EVP_DigestFinalXOF(ctx, dst, digest_length) <= 0)
goto process_auth_err;
-#else
- RTE_SET_USED(digest_length);
- OPENSSL_LOG(ERR, "SHAKE algorithms require OpenSSL 3.0+");
- goto process_auth_err;
-#endif
} else {
if (EVP_DigestFinal_ex(ctx, dst, (unsigned int *)&dstlen) <= 0)
goto process_auth_err;
@@ -1523,7 +1430,6 @@ process_openssl_auth(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
return -EINVAL;
}
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
/** Process standard openssl auth algorithms with hmac/cmac */
static int
process_openssl_auth_mac(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
@@ -1576,109 +1482,6 @@ process_openssl_auth_mac(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
OPENSSL_LOG(ERR, "Process openssl auth failed");
return -EINVAL;
}
-# else
-/** Process standard openssl auth algorithms with hmac */
-static int
-process_openssl_auth_hmac(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
- int srclen, HMAC_CTX *ctx)
-{
- unsigned int dstlen;
- struct rte_mbuf *m;
- int l, n = srclen;
- uint8_t *src;
-
- for (m = mbuf_src; m != NULL && offset > rte_pktmbuf_data_len(m);
- m = m->next)
- offset -= rte_pktmbuf_data_len(m);
-
- if (m == 0)
- goto process_auth_err;
-
- src = rte_pktmbuf_mtod_offset(m, uint8_t *, offset);
-
- l = rte_pktmbuf_data_len(m) - offset;
- if (srclen <= l) {
- if (HMAC_Update(ctx, (unsigned char *)src, srclen) != 1)
- goto process_auth_err;
- goto process_auth_final;
- }
-
- if (HMAC_Update(ctx, (unsigned char *)src, l) != 1)
- goto process_auth_err;
-
- n -= l;
-
- for (m = m->next; (m != NULL) && (n > 0); m = m->next) {
- src = rte_pktmbuf_mtod(m, uint8_t *);
- l = rte_pktmbuf_data_len(m) < n ? rte_pktmbuf_data_len(m) : n;
- if (HMAC_Update(ctx, (unsigned char *)src, l) != 1)
- goto process_auth_err;
- n -= l;
- }
-
-process_auth_final:
- if (HMAC_Final(ctx, dst, &dstlen) != 1)
- goto process_auth_err;
-
- if (unlikely(HMAC_Init_ex(ctx, NULL, 0, NULL, NULL) != 1))
- goto process_auth_err;
-
- return 0;
-
-process_auth_err:
- OPENSSL_LOG(ERR, "Process openssl auth failed");
- return -EINVAL;
-}
-
-/** Process standard openssl auth algorithms with cmac */
-static int
-process_openssl_auth_cmac(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
- int srclen, CMAC_CTX *ctx)
-{
- unsigned int dstlen;
- struct rte_mbuf *m;
- int l, n = srclen;
- uint8_t *src;
-
- for (m = mbuf_src; m != NULL && offset > rte_pktmbuf_data_len(m);
- m = m->next)
- offset -= rte_pktmbuf_data_len(m);
-
- if (m == 0)
- goto process_auth_err;
-
- src = rte_pktmbuf_mtod_offset(m, uint8_t *, offset);
-
- l = rte_pktmbuf_data_len(m) - offset;
- if (srclen <= l) {
- if (CMAC_Update(ctx, (unsigned char *)src, srclen) != 1)
- goto process_auth_err;
- goto process_auth_final;
- }
-
- if (CMAC_Update(ctx, (unsigned char *)src, l) != 1)
- goto process_auth_err;
-
- n -= l;
-
- for (m = m->next; (m != NULL) && (n > 0); m = m->next) {
- src = rte_pktmbuf_mtod(m, uint8_t *);
- l = rte_pktmbuf_data_len(m) < n ? rte_pktmbuf_data_len(m) : n;
- if (CMAC_Update(ctx, (unsigned char *)src, l) != 1)
- goto process_auth_err;
- n -= l;
- }
-
-process_auth_final:
- if (CMAC_Final(ctx, dst, (size_t *)&dstlen) != 1)
- goto process_auth_err;
- return 0;
-
-process_auth_err:
- OPENSSL_LOG(ERR, "Process openssl cmac auth failed");
- return -EINVAL;
-}
-# endif
/*----------------------------------------------------------------------------*/
static inline EVP_CIPHER_CTX *
@@ -1695,7 +1498,7 @@ get_local_cipher_ctx(struct openssl_session *sess, struct openssl_qp *qp)
/* EVP_CIPHER_CTX_dup() added in OSSL 3.2 */
*lctx = EVP_CIPHER_CTX_dup(sess->cipher.ctx);
return *lctx;
-#elif OPENSSL_VERSION_NUMBER >= 0x30000000L
+#else
if (sess->chain_order == OPENSSL_CHAIN_COMBINED) {
/* AESNI special-cased to use openssl_aesni_ctx_clone()
* to allow for working around lack of
@@ -1706,10 +1509,10 @@ get_local_cipher_ctx(struct openssl_session *sess, struct openssl_qp *qp)
*lctx = NULL;
return *lctx;
}
-#endif
*lctx = EVP_CIPHER_CTX_new();
EVP_CIPHER_CTX_copy(*lctx, sess->cipher.ctx);
+#endif
}
return *lctx;
@@ -1737,11 +1540,7 @@ get_local_auth_ctx(struct openssl_session *sess, struct openssl_qp *qp)
return *lctx;
}
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
static inline EVP_MAC_CTX *
-#else
-static inline HMAC_CTX *
-#endif
get_local_hmac_ctx(struct openssl_session *sess, struct openssl_qp *qp)
{
#if (OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x30003000L)
@@ -1759,31 +1558,16 @@ get_local_hmac_ctx(struct openssl_session *sess, struct openssl_qp *qp)
if (sess->ctx_copies_len == 0)
return sess->auth.hmac.ctx;
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
- EVP_MAC_CTX **lctx =
-#else
- HMAC_CTX **lctx =
-#endif
- &sess->qp_ctx[qp->id].hmac;
+ EVP_MAC_CTX **lctx = &sess->qp_ctx[qp->id].hmac;
- if (unlikely(*lctx == NULL)) {
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
+ if (unlikely(*lctx == NULL))
*lctx = EVP_MAC_CTX_dup(sess->auth.hmac.ctx);
-#else
- *lctx = HMAC_CTX_new();
- HMAC_CTX_copy(*lctx, sess->auth.hmac.ctx);
-#endif
- }
return *lctx;
#endif
}
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
static inline EVP_MAC_CTX *
-#else
-static inline CMAC_CTX *
-#endif
get_local_cmac_ctx(struct openssl_session *sess, struct openssl_qp *qp)
{
#if (OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x30003000L)
@@ -1801,21 +1585,10 @@ get_local_cmac_ctx(struct openssl_session *sess, struct openssl_qp *qp)
if (sess->ctx_copies_len == 0)
return sess->auth.cmac.ctx;
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
- EVP_MAC_CTX **lctx =
-#else
- CMAC_CTX **lctx =
-#endif
- &sess->qp_ctx[qp->id].cmac;
+ EVP_MAC_CTX **lctx = &sess->qp_ctx[qp->id].cmac;
- if (unlikely(*lctx == NULL)) {
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
+ if (unlikely(*lctx == NULL))
*lctx = EVP_MAC_CTX_dup(sess->auth.cmac.ctx);
-#else
- *lctx = CMAC_CTX_new();
- CMAC_CTX_copy(*lctx, sess->auth.cmac.ctx);
-#endif
- }
return *lctx;
#endif
@@ -2055,13 +1828,8 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op,
uint8_t *dst;
int srclen, status;
EVP_MD_CTX *ctx_a;
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
EVP_MAC_CTX *ctx_h;
EVP_MAC_CTX *ctx_c;
-# else
- HMAC_CTX *ctx_h;
- CMAC_CTX *ctx_c;
-# endif
srclen = op->sym->auth.data.length;
@@ -2076,30 +1844,18 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op,
break;
case OPENSSL_AUTH_AS_HMAC:
ctx_h = get_local_hmac_ctx(sess, qp);
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
status = process_openssl_auth_mac(mbuf_src, dst,
op->sym->auth.data.offset, srclen,
ctx_h);
-# else
- status = process_openssl_auth_hmac(mbuf_src, dst,
- op->sym->auth.data.offset, srclen,
- ctx_h);
-# endif
#if (OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x30003000L)
EVP_MAC_CTX_free(ctx_h);
#endif
break;
case OPENSSL_AUTH_AS_CMAC:
ctx_c = get_local_cmac_ctx(sess, qp);
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
status = process_openssl_auth_mac(mbuf_src, dst,
op->sym->auth.data.offset, srclen,
ctx_c);
-# else
- status = process_openssl_auth_cmac(mbuf_src, dst,
- op->sym->auth.data.offset, srclen,
- ctx_c);
-# endif
#if (OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x30003000L)
EVP_MAC_CTX_free(ctx_c);
#endif
@@ -2130,7 +1886,6 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op,
}
/* process dsa sign operation */
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
static int
process_openssl_dsa_sign_op_evp(struct rte_crypto_op *cop,
struct openssl_asym_session *sess)
@@ -2296,92 +2051,8 @@ process_openssl_dsa_verify_op_evp(struct rte_crypto_op *cop,
return ret;
}
-#else
-static int
-process_openssl_dsa_sign_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- struct rte_crypto_dsa_op_param *op = &cop->asym->dsa;
- DSA *dsa = sess->u.s.dsa;
- DSA_SIG *sign = NULL;
-
- sign = DSA_do_sign(op->message.data,
- op->message.length,
- dsa);
-
- if (sign == NULL) {
- OPENSSL_LOG(ERR, "%s:%d", __func__, __LINE__);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- } else {
- const BIGNUM *r = NULL, *s = NULL;
- get_dsa_sign(sign, &r, &s);
-
- op->r.length = BN_bn2bin(r, op->r.data);
- op->s.length = BN_bn2bin(s, op->s.data);
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
- }
-
- DSA_SIG_free(sign);
-
- return 0;
-}
-
-/* process dsa verify operation */
-static int
-process_openssl_dsa_verify_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- struct rte_crypto_dsa_op_param *op = &cop->asym->dsa;
- DSA *dsa = sess->u.s.dsa;
- int ret;
- DSA_SIG *sign = DSA_SIG_new();
- BIGNUM *r = NULL, *s = NULL;
- BIGNUM *pub_key = NULL;
-
- if (sign == NULL) {
- OPENSSL_LOG(ERR, " %s:%d", __func__, __LINE__);
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
-
- r = BN_bin2bn(op->r.data,
- op->r.length,
- r);
- s = BN_bin2bn(op->s.data,
- op->s.length,
- s);
- pub_key = BN_bin2bn(op->y.data,
- op->y.length,
- pub_key);
- if (!r || !s || !pub_key) {
- BN_free(r);
- BN_free(s);
- BN_free(pub_key);
-
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
- set_dsa_sign(sign, r, s);
- set_dsa_pub_key(dsa, pub_key);
-
- ret = DSA_do_verify(op->message.data,
- op->message.length,
- sign,
- dsa);
-
- if (ret != 1)
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- else
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
-
- DSA_SIG_free(sign);
-
- return 0;
-}
-#endif
/* process dh operation */
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
static int
process_openssl_dh_op_evp(struct rte_crypto_op *cop,
struct openssl_asym_session *sess)
@@ -2555,141 +2226,6 @@ process_openssl_dh_op_evp(struct rte_crypto_op *cop,
return ret;
}
-#else
-static int
-process_openssl_dh_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- struct rte_crypto_dh_op_param *op = &cop->asym->dh;
- struct rte_crypto_asym_op *asym_op = cop->asym;
- DH *dh_key = sess->u.dh.dh_key;
- BIGNUM *priv_key = NULL;
- int ret = 0;
-
- if (asym_op->dh.ke_type == RTE_CRYPTO_ASYM_KE_SHARED_SECRET_COMPUTE) {
- /* compute shared secret using peer public key
- * and current private key
- * shared secret = peer_key ^ priv_key mod p
- */
- BIGNUM *peer_key = NULL;
-
- /* copy private key and peer key and compute shared secret */
- peer_key = BN_bin2bn(op->pub_key.data,
- op->pub_key.length,
- peer_key);
- if (peer_key == NULL) {
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
- priv_key = BN_bin2bn(op->priv_key.data,
- op->priv_key.length,
- priv_key);
- if (priv_key == NULL) {
- BN_free(peer_key);
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
- ret = set_dh_priv_key(dh_key, priv_key);
- if (ret) {
- OPENSSL_LOG(ERR, "Failed to set private key");
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- BN_free(peer_key);
- BN_free(priv_key);
- return 0;
- }
-
- ret = DH_compute_key(
- op->shared_secret.data,
- peer_key, dh_key);
- if (ret < 0) {
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- BN_free(peer_key);
- /* priv key is already loaded into dh,
- * let's not free that directly here.
- * DH_free() will auto free it later.
- */
- return 0;
- }
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
- op->shared_secret.length = ret;
- BN_free(peer_key);
- return 0;
- }
-
- /*
- * other options are public and private key generations.
- *
- * if user provides private key,
- * then first set DH with user provided private key
- */
- if (asym_op->dh.ke_type == RTE_CRYPTO_ASYM_KE_PUB_KEY_GENERATE &&
- op->priv_key.length) {
- /* generate public key using user-provided private key
- * pub_key = g ^ priv_key mod p
- */
-
- /* load private key into DH */
- priv_key = BN_bin2bn(op->priv_key.data,
- op->priv_key.length,
- priv_key);
- if (priv_key == NULL) {
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
- ret = set_dh_priv_key(dh_key, priv_key);
- if (ret) {
- OPENSSL_LOG(ERR, "Failed to set private key");
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- BN_free(priv_key);
- return 0;
- }
- }
-
- /* generate public and private key pair.
- *
- * if private key already set, generates only public key.
- *
- * if private key is not already set, then set it to random value
- * and update internal private key.
- */
- if (!DH_generate_key(dh_key)) {
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return 0;
- }
-
- if (asym_op->dh.ke_type == RTE_CRYPTO_ASYM_KE_PUB_KEY_GENERATE) {
- const BIGNUM *pub_key = NULL;
-
- OPENSSL_LOG(DEBUG, "%s:%d update public key",
- __func__, __LINE__);
-
- /* get the generated keys */
- get_dh_pub_key(dh_key, &pub_key);
-
- /* output public key */
- op->pub_key.length = BN_bn2bin(pub_key,
- op->pub_key.data);
- }
-
- if (asym_op->dh.ke_type == RTE_CRYPTO_ASYM_KE_PRIV_KEY_GENERATE) {
- const BIGNUM *priv_key = NULL;
-
- OPENSSL_LOG(DEBUG, "%s:%d updated priv key",
- __func__, __LINE__);
-
- /* get the generated keys */
- get_dh_priv_key(dh_key, &priv_key);
-
- /* provide generated private key back to user */
- op->priv_key.length = BN_bn2bin(priv_key,
- op->priv_key.data);
- }
-
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
-
- return 0;
-}
-#endif
/* process modinv operation */
static int
@@ -2757,7 +2293,6 @@ process_openssl_modexp_op(struct rte_crypto_op *cop,
}
/* process rsa operations */
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
static int
process_openssl_rsa_op_evp(struct rte_crypto_op *cop,
struct openssl_asym_session *sess)
@@ -3333,133 +2868,7 @@ process_openssl_eddsa_op_evp(struct rte_crypto_op *cop,
return ret;
}
-#else
-static int
-process_openssl_rsa_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- int ret = 0;
- struct rte_crypto_asym_op *op = cop->asym;
- RSA *rsa = sess->u.r.rsa;
- uint32_t pad = sess->u.r.pad;
- uint8_t *tmp;
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
-
- switch (pad) {
- case RTE_CRYPTO_RSA_PADDING_PKCS1_5:
- pad = RSA_PKCS1_PADDING;
- break;
- case RTE_CRYPTO_RSA_PADDING_NONE:
- pad = RSA_NO_PADDING;
- break;
- default:
- cop->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
- OPENSSL_LOG(ERR,
- "rsa pad type not supported %d", pad);
- return 0;
- }
-
- switch (op->rsa.op_type) {
- case RTE_CRYPTO_ASYM_OP_ENCRYPT:
- ret = RSA_public_encrypt(op->rsa.message.length,
- op->rsa.message.data,
- op->rsa.cipher.data,
- rsa,
- pad);
-
- if (ret > 0)
- op->rsa.cipher.length = ret;
- OPENSSL_LOG(DEBUG,
- "length of encrypted text %d", ret);
- break;
-
- case RTE_CRYPTO_ASYM_OP_DECRYPT:
- ret = RSA_private_decrypt(op->rsa.cipher.length,
- op->rsa.cipher.data,
- op->rsa.message.data,
- rsa,
- pad);
- if (ret > 0)
- op->rsa.message.length = ret;
- break;
-
- case RTE_CRYPTO_ASYM_OP_SIGN:
- ret = RSA_private_encrypt(op->rsa.message.length,
- op->rsa.message.data,
- op->rsa.sign.data,
- rsa,
- pad);
- if (ret > 0)
- op->rsa.sign.length = ret;
- break;
-
- case RTE_CRYPTO_ASYM_OP_VERIFY:
- tmp = rte_malloc(NULL, op->rsa.sign.length, 0);
- if (tmp == NULL) {
- OPENSSL_LOG(ERR, "Memory allocation failed");
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- break;
- }
- ret = RSA_public_decrypt(op->rsa.sign.length,
- op->rsa.sign.data,
- tmp,
- rsa,
- pad);
-
- OPENSSL_LOG(DEBUG,
- "Length of public_decrypt %d "
- "length of message %zd",
- ret, op->rsa.message.length);
- if ((ret <= 0) || (CRYPTO_memcmp(tmp, op->rsa.message.data,
- op->rsa.message.length))) {
- OPENSSL_LOG(ERR, "RSA sign Verification failed");
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- }
- rte_free(tmp);
- break;
-
- default:
- /* allow ops with invalid args to be pushed to
- * completion queue
- */
- cop->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
- break;
- }
-
- if (ret < 0)
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
-
- return 0;
-}
-
-static int
-process_openssl_ecfpm_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- RTE_SET_USED(cop);
- RTE_SET_USED(sess);
- return -ENOTSUP;
-}
-
-static int
-process_openssl_sm2_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- RTE_SET_USED(cop);
- RTE_SET_USED(sess);
- return -ENOTSUP;
-}
-
-static int
-process_openssl_eddsa_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- RTE_SET_USED(cop);
- RTE_SET_USED(sess);
- return -ENOTSUP;
-}
-#endif
#if (OPENSSL_VERSION_NUMBER >= 0x30500000L)
static int
@@ -4085,14 +3494,12 @@ mldsa_sign_op_evp(struct rte_crypto_op *cop,
case RTE_CRYPTO_AUTH_SHA3_512:
check_md = EVP_sha3_512();
break;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
case RTE_CRYPTO_AUTH_SHAKE_128:
check_md = EVP_shake128();
break;
case RTE_CRYPTO_AUTH_SHAKE_256:
check_md = EVP_shake256();
break;
-#endif
default:
break;
}
@@ -4328,11 +3735,7 @@ process_asym_op(struct openssl_qp *qp, struct rte_crypto_op *op,
switch (sess->xfrm_type) {
case RTE_CRYPTO_ASYM_XFORM_RSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_rsa_op_evp(op, sess);
-# else
- retval = process_openssl_rsa_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_MODEX:
retval = process_openssl_modexp_op(op, sess);
@@ -4341,51 +3744,26 @@ process_asym_op(struct openssl_qp *qp, struct rte_crypto_op *op,
retval = process_openssl_modinv_op(op, sess);
break;
case RTE_CRYPTO_ASYM_XFORM_DH:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_dh_op_evp(op, sess);
-# else
- retval = process_openssl_dh_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_DSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
if (op->asym->dsa.op_type == RTE_CRYPTO_ASYM_OP_SIGN)
retval = process_openssl_dsa_sign_op_evp(op, sess);
else if (op->asym->dsa.op_type ==
RTE_CRYPTO_ASYM_OP_VERIFY)
retval =
process_openssl_dsa_verify_op_evp(op, sess);
-#else
- if (op->asym->dsa.op_type == RTE_CRYPTO_ASYM_OP_SIGN)
- retval = process_openssl_dsa_sign_op(op, sess);
- else if (op->asym->dsa.op_type ==
- RTE_CRYPTO_ASYM_OP_VERIFY)
- retval =
- process_openssl_dsa_verify_op(op, sess);
else
op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_ECFPM:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_ecfpm_op_evp(op, sess);
-#else
- retval = process_openssl_ecfpm_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_SM2:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_sm2_op_evp(op, sess);
-#else
- retval = process_openssl_sm2_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_EDDSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_eddsa_op_evp(op, sess);
-#else
- retval = process_openssl_eddsa_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_ML_KEM:
#if (OPENSSL_VERSION_NUMBER >= 0x30500000L)
@@ -4590,13 +3968,12 @@ cryptodev_openssl_create(const char *name,
rte_cryptodev_pmd_probing_finish(dev);
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
/* Load legacy provider
* Some algorithms are no longer available in earlier version of openssl,
* unless the legacy provider explicitly loaded. e.g. DES
*/
ossl_legacy_provider_load();
-# endif
+
return 0;
init_error:
@@ -4645,9 +4022,8 @@ cryptodev_openssl_remove(struct rte_vdev_device *vdev)
if (cryptodev == NULL)
return -ENODEV;
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
ossl_legacy_provider_unload();
-# endif
+
return rte_cryptodev_pmd_destroy(cryptodev);
}
diff --git a/drivers/crypto/openssl/rte_openssl_pmd_ops.c b/drivers/crypto/openssl/rte_openssl_pmd_ops.c
index 4e5fb07bb2..d927cc5228 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd_ops.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd_ops.c
@@ -10,11 +10,9 @@
#include "openssl_pmd_private.h"
#include "compat.h"
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
#include <openssl/provider.h>
#include <openssl/core_names.h>
#include <openssl/param_build.h>
-#endif
static const struct rte_cryptodev_capabilities openssl_pmd_capabilities[] = {
{ /* MD5 HMAC */
@@ -457,7 +455,6 @@ static const struct rte_cryptodev_capabilities openssl_pmd_capabilities[] = {
}, }
}, }
},
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
{ /* SHAKE_128 */
.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
{.sym = {
@@ -500,7 +497,6 @@ static const struct rte_cryptodev_capabilities openssl_pmd_capabilities[] = {
}, }
}, }
},
-#endif
{ /* AES CBC */
.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
{.sym = {
@@ -1222,7 +1218,6 @@ static int openssl_set_asym_session_parameters(
goto err_rsa;
asym_session->u.r.pad = xform->rsa.padding.type;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD * param_bld = OSSL_PARAM_BLD_new();
if (!param_bld) {
OPENSSL_LOG(ERR, "failed to allocate resources");
@@ -1323,79 +1318,7 @@ static int openssl_set_asym_session_parameters(
OSSL_PARAM_BLD_free(param_bld);
OSSL_PARAM_free(params);
ret = 0;
-#else
- RSA *rsa = RSA_new();
- if (rsa == NULL)
- goto err_rsa;
-
- if (xform->rsa.d.length > 0) {
- d = BN_bin2bn(
- (const unsigned char *)xform->rsa.d.data,
- xform->rsa.d.length,
- d);
- if (!d) {
- RSA_free(rsa);
- goto err_rsa;
- }
- }
-
- if (xform->rsa.key_type == RTE_RSA_KEY_TYPE_QT) {
- p = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.p.data,
- xform->rsa.qt.p.length,
- p);
- q = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.q.data,
- xform->rsa.qt.q.length,
- q);
- dmp1 = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.dP.data,
- xform->rsa.qt.dP.length,
- dmp1);
- dmq1 = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.dQ.data,
- xform->rsa.qt.dQ.length,
- dmq1);
- iqmp = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.qInv.data,
- xform->rsa.qt.qInv.length,
- iqmp);
- if (!p || !q || !dmp1 || !dmq1 || !iqmp) {
- RSA_free(rsa);
- goto err_rsa;
- }
- ret = set_rsa_params(rsa, p, q);
- if (ret) {
- OPENSSL_LOG(ERR,
- "failed to set rsa params");
- RSA_free(rsa);
- goto err_rsa;
- }
- ret = set_rsa_crt_params(rsa, dmp1, dmq1, iqmp);
- if (ret) {
- OPENSSL_LOG(ERR,
- "failed to set crt params");
- RSA_free(rsa);
- /*
- * set already populated params to NULL
- * as its freed by call to RSA_free
- */
- p = q = NULL;
- goto err_rsa;
- }
- }
-
- ret = set_rsa_keys(rsa, n, e, d);
- if (ret) {
- OPENSSL_LOG(ERR, "Failed to load rsa keys");
- RSA_free(rsa);
- return ret;
- }
- asym_session->u.r.rsa = rsa;
- asym_session->xfrm_type = RTE_CRYPTO_ASYM_XFORM_RSA;
- break;
-#endif
err_rsa:
BN_clear_free(n);
BN_clear_free(e);
@@ -1469,7 +1392,6 @@ static int openssl_set_asym_session_parameters(
case RTE_CRYPTO_ASYM_XFORM_DH:
{
DH *dh = NULL;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
BIGNUM **p = &asym_session->u.dh.p;
BIGNUM **g = &asym_session->u.dh.g;
@@ -1520,51 +1442,18 @@ static int openssl_set_asym_session_parameters(
asym_session->u.dh.param_bld = param_bld;
asym_session->u.dh.param_bld_peer = param_bld_peer;
-#else
- BIGNUM *p = NULL;
- BIGNUM *g = NULL;
-
- p = BN_bin2bn((const unsigned char *)
- xform->dh.p.data,
- xform->dh.p.length,
- p);
- g = BN_bin2bn((const unsigned char *)
- xform->dh.g.data,
- xform->dh.g.length,
- g);
- if (!p || !g)
- goto err_dh;
-
- dh = DH_new();
- if (dh == NULL) {
- OPENSSL_LOG(ERR,
- "failed to allocate resources");
- goto err_dh;
- }
- ret = set_dh_params(dh, p, g);
- if (ret) {
- DH_free(dh);
- goto err_dh;
- }
-#endif
asym_session->u.dh.dh_key = dh;
asym_session->xfrm_type = RTE_CRYPTO_ASYM_XFORM_DH;
break;
err_dh:
OPENSSL_LOG(ERR, " failed to set dh params");
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
BN_free(*p);
BN_free(*g);
-#else
- BN_free(p);
- BN_free(g);
-#endif
return -1;
}
case RTE_CRYPTO_ASYM_XFORM_DSA:
{
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
BIGNUM **p = &asym_session->u.s.p;
BIGNUM **g = &asym_session->u.s.g;
BIGNUM **q = &asym_session->u.s.q;
@@ -1615,85 +1504,16 @@ static int openssl_set_asym_session_parameters(
asym_session->u.s.param_bld = param_bld;
break;
-#else
- BIGNUM *p = NULL, *g = NULL;
- BIGNUM *q = NULL, *priv_key = NULL;
- BIGNUM *pub_key = BN_new();
- BN_zero(pub_key);
-
- p = BN_bin2bn((const unsigned char *)
- xform->dsa.p.data,
- xform->dsa.p.length,
- p);
-
- g = BN_bin2bn((const unsigned char *)
- xform->dsa.g.data,
- xform->dsa.g.length,
- g);
-
- q = BN_bin2bn((const unsigned char *)
- xform->dsa.q.data,
- xform->dsa.q.length,
- q);
- if (!p || !q || !g)
- goto err_dsa;
-
- priv_key = BN_bin2bn((const unsigned char *)
- xform->dsa.x.data,
- xform->dsa.x.length,
- priv_key);
- if (priv_key == NULL)
- goto err_dsa;
-
- DSA *dsa = DSA_new();
- if (dsa == NULL) {
- OPENSSL_LOG(ERR,
- " failed to allocate resources");
- goto err_dsa;
- }
-
- ret = set_dsa_params(dsa, p, q, g);
- if (ret) {
- DSA_free(dsa);
- OPENSSL_LOG(ERR, "Failed to dsa params");
- goto err_dsa;
- }
-
- /*
- * openssl 1.1.0 mandate that public key can't be
- * NULL in very first call. so set a dummy pub key.
- * to keep consistency, lets follow same approach for
- * both versions
- */
- /* just set dummy public for very 1st call */
- ret = set_dsa_keys(dsa, pub_key, priv_key);
- if (ret) {
- DSA_free(dsa);
- OPENSSL_LOG(ERR, "Failed to set keys");
- goto err_dsa;
- }
- asym_session->u.s.dsa = dsa;
- asym_session->xfrm_type = RTE_CRYPTO_ASYM_XFORM_DSA;
- break;
-#endif
err_dsa:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
BN_free(*p);
BN_free(*q);
BN_free(*g);
BN_free(*priv_key);
-#else
- BN_free(p);
- BN_free(q);
- BN_free(g);
- BN_free(priv_key);
-#endif
BN_free(pub_key);
return -1;
}
case RTE_CRYPTO_ASYM_XFORM_ECFPM:
{
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
EC_GROUP *ecgrp = NULL;
asym_session->xfrm_type = xform->xform_type;
@@ -1727,14 +1547,9 @@ static int openssl_set_asym_session_parameters(
asym_session->u.ec.curve_id = xform->ec.curve_id;
asym_session->u.ec.group = ecgrp;
break;
-#else
- OPENSSL_LOG(WARNING, "ECFPM unsupported for OpenSSL Version < 3.0");
- return -ENOTSUP;
-#endif
}
case RTE_CRYPTO_ASYM_XFORM_SM2:
{
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
#ifndef OPENSSL_NO_SM2
OSSL_PARAM_BLD *param_bld = NULL;
OSSL_PARAM *params = NULL;
@@ -1818,10 +1633,6 @@ static int openssl_set_asym_session_parameters(
#else
OPENSSL_LOG(WARNING, "SM2 unsupported in current OpenSSL Version");
return -ENOTSUP;
-#endif
-#else
- OPENSSL_LOG(WARNING, "SM2 unsupported for OpenSSL Version < 3.0");
- return -ENOTSUP;
#endif
}
case RTE_CRYPTO_ASYM_XFORM_EDDSA:
@@ -1983,12 +1794,7 @@ static void openssl_reset_asym_session(struct openssl_asym_session *sess)
{
switch (sess->xfrm_type) {
case RTE_CRYPTO_ASYM_XFORM_RSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
EVP_PKEY_CTX_free(sess->u.r.ctx);
-#else
- if (sess->u.r.rsa)
- RSA_free(sess->u.r.rsa);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_MODEX:
if (sess->u.e.ctx) {
@@ -2003,35 +1809,23 @@ static void openssl_reset_asym_session(struct openssl_asym_session *sess)
}
break;
case RTE_CRYPTO_ASYM_XFORM_DH:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD_free(sess->u.dh.param_bld);
OSSL_PARAM_BLD_free(sess->u.dh.param_bld_peer);
sess->u.dh.param_bld = NULL;
sess->u.dh.param_bld_peer = NULL;
-#else
- if (sess->u.dh.dh_key)
- DH_free(sess->u.dh.dh_key);
-#endif
BN_clear_free(sess->u.dh.p);
BN_clear_free(sess->u.dh.g);
break;
case RTE_CRYPTO_ASYM_XFORM_DSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD_free(sess->u.s.param_bld);
sess->u.s.param_bld = NULL;
BN_clear_free(sess->u.s.p);
BN_clear_free(sess->u.s.q);
BN_clear_free(sess->u.s.g);
BN_clear_free(sess->u.s.priv_key);
-#else
- if (sess->u.s.dsa)
- DSA_free(sess->u.s.dsa);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_SM2:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_free(sess->u.sm2.params);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_EDDSA:
#if (OPENSSL_VERSION_NUMBER >= 0x30300000L)
--
2.43.0
^ permalink raw reply related
* [PATCH] app/crypto-perf: support ML DSA
From: Pratik Senapati @ 2026-05-28 7:58 UTC (permalink / raw)
To: dev; +Cc: gakhil, anoobj, gmuthukrishn, kai.ji
In-Reply-To: <20260528075853.2206573-1-psenapati@marvell.com>
Add ML-DSA44 support to test-crypto-perf.
Signed-off-by: Pratik Senapati <psenapati@marvell.com>
---
app/test-crypto-perf/cperf_ops.c | 70 +
app/test-crypto-perf/cperf_options.h | 2 +
app/test-crypto-perf/cperf_options_parsing.c | 20 +-
app/test-crypto-perf/cperf_test_common.c | 3 +-
app/test-crypto-perf/cperf_test_vectors.c | 1661 ++++++++++++++++++
app/test-crypto-perf/cperf_test_vectors.h | 37 +
app/test-crypto-perf/main.c | 17 +
7 files changed, 1808 insertions(+), 2 deletions(-)
diff --git a/app/test-crypto-perf/cperf_ops.c b/app/test-crypto-perf/cperf_ops.c
index 806265f7bf..7d3a74b505 100644
--- a/app/test-crypto-perf/cperf_ops.c
+++ b/app/test-crypto-perf/cperf_ops.c
@@ -251,6 +251,58 @@ cperf_set_ops_asym_mlkem(struct rte_crypto_op **ops,
}
}
+static void
+cperf_set_ops_asym_mldsa(struct rte_crypto_op **ops,
+ uint32_t src_buf_offset __rte_unused,
+ uint32_t dst_buf_offset __rte_unused, uint16_t nb_ops,
+ void *sess,
+ const struct cperf_options *options,
+ const struct cperf_test_vector *test_vector __rte_unused,
+ uint16_t iv_offset __rte_unused,
+ uint32_t *imix_idx __rte_unused,
+ uint64_t *tsc_start __rte_unused)
+{
+ uint16_t i;
+
+ for (i = 0; i < nb_ops; i++) {
+ struct rte_crypto_asym_op *asym_op = ops[i]->asym;
+
+ ops[i]->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
+ rte_crypto_op_attach_asym_session(ops[i], sess);
+
+ if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_SIGN) {
+ asym_op->mldsa.op = RTE_CRYPTO_ML_DSA_OP_SIGN;
+ asym_op->mldsa.siggen.privkey.data = options->mldsa_data->privkey.data;
+ asym_op->mldsa.siggen.privkey.length = options->mldsa_data->privkey.length;
+ asym_op->mldsa.siggen.message.data = options->mldsa_data->message.data;
+ asym_op->mldsa.siggen.message.length = options->mldsa_data->message.length;
+ asym_op->mldsa.siggen.sign.data = options->mldsa_data->sign.data;
+ asym_op->mldsa.siggen.sign.length = options->mldsa_data->sign.length;
+ asym_op->mldsa.siggen.ctx.data = options->mldsa_data->ctx.data;
+ asym_op->mldsa.siggen.ctx.length = options->mldsa_data->ctx.length;
+ asym_op->mldsa.siggen.seed.data = options->mldsa_data->seed.data;
+ asym_op->mldsa.siggen.seed.length = options->mldsa_data->seed.length;
+ asym_op->mldsa.siggen.mu.data = options->mldsa_data->mu.data;
+ asym_op->mldsa.siggen.mu.length = options->mldsa_data->mu.length;
+ asym_op->mldsa.siggen.hash = options->mldsa_data->hash;
+ } else if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_VERIFY) {
+ asym_op->mldsa.op = RTE_CRYPTO_ML_DSA_OP_VERIFY;
+ asym_op->mldsa.sigver.pubkey.data = options->mldsa_data->pubkey.data;
+ asym_op->mldsa.sigver.pubkey.length = options->mldsa_data->pubkey.length;
+ asym_op->mldsa.sigver.message.data = options->mldsa_data->message.data;
+ asym_op->mldsa.sigver.message.length = options->mldsa_data->message.length;
+ asym_op->mldsa.sigver.sign.data = options->mldsa_data->sign.data;
+ asym_op->mldsa.sigver.sign.length = options->mldsa_data->sign.length;
+ asym_op->mldsa.sigver.ctx.data = options->mldsa_data->ctx.data;
+ asym_op->mldsa.sigver.ctx.length = options->mldsa_data->ctx.length;
+ asym_op->mldsa.sigver.mu.data = options->mldsa_data->mu.data;
+ asym_op->mldsa.sigver.mu.length = options->mldsa_data->mu.length;
+ asym_op->mldsa.sigver.hash = options->mldsa_data->hash;
+ } else {
+ rte_panic("Unsupported ML-DSA operation type %d\n", options->asym_op_type);
+ }
+ }
+}
#ifdef RTE_LIB_SECURITY
static void
@@ -1269,6 +1321,21 @@ cperf_create_session(struct rte_mempool *sess_mp,
return asym_sess;
}
+ if (options->op_type == CPERF_ASYM_MLDSA44) {
+ xform.next = NULL;
+ xform.xform_type = RTE_CRYPTO_ASYM_XFORM_ML_DSA;
+ xform.mldsa.type = RTE_CRYPTO_ML_DSA_44;
+ xform.mldsa.sign_deterministic =
+ options->mldsa_data->sign_deterministic;
+ xform.mldsa.sign_prehash = false;
+
+ ret = rte_cryptodev_asym_session_create(dev_id, &xform, sess_mp, &asym_sess);
+ if (ret < 0) {
+ RTE_LOG(ERR, USER1, "ML-DSA Asym session create failed\n");
+ return NULL;
+ }
+ return asym_sess;
+ }
if (options->op_type == CPERF_ASYM_MLKEM512) {
xform.next = NULL;
@@ -1600,6 +1667,9 @@ cperf_get_op_functions(const struct cperf_options *options,
case CPERF_ASYM_SM2:
op_fns->populate_ops = cperf_set_ops_asym_sm2;
break;
+ case CPERF_ASYM_MLDSA44:
+ op_fns->populate_ops = cperf_set_ops_asym_mldsa;
+ break;
case CPERF_ASYM_MLKEM512:
op_fns->populate_ops = cperf_set_ops_asym_mlkem;
break;
diff --git a/app/test-crypto-perf/cperf_options.h b/app/test-crypto-perf/cperf_options.h
index 98b8eeec3e..d8ceca2424 100644
--- a/app/test-crypto-perf/cperf_options.h
+++ b/app/test-crypto-perf/cperf_options.h
@@ -101,6 +101,7 @@ enum cperf_op_type {
CPERF_ASYM_SECP521R1,
CPERF_ASYM_ED25519,
CPERF_ASYM_SM2,
+ CPERF_ASYM_MLDSA44,
CPERF_ASYM_MLKEM512,
CPERF_TLS,
};
@@ -189,6 +190,7 @@ struct cperf_options {
struct cperf_eddsa_test_data *eddsa_data;
struct cperf_sm2_test_data *sm2_data;
struct cperf_mlkem_test_data *mlkem_data;
+ struct cperf_mldsa_test_data *mldsa_data;
enum rte_crypto_asym_op_type asym_op_type;
enum rte_crypto_auth_algorithm asym_hash_alg;
struct cperf_rsa_test_data *rsa_data;
diff --git a/app/test-crypto-perf/cperf_options_parsing.c b/app/test-crypto-perf/cperf_options_parsing.c
index 34afa938c8..44c2a7173a 100644
--- a/app/test-crypto-perf/cperf_options_parsing.c
+++ b/app/test-crypto-perf/cperf_options_parsing.c
@@ -41,7 +41,7 @@ usage(char *progname)
" --optype cipher-only / auth-only / cipher-then-auth / auth-then-cipher /\n"
" aead / pdcp / docsis / ipsec / modex / rsa / secp192r1 /\n"
" secp224r1 / secp256r1 / secp384r1 / secp521r1 / eddsa / sm2 /\n"
- " mlkem_512 /\n"
+ " mlkem_512 / mldsa_44 /\n"
" tls-record : set operation type\n"
" --sessionless: enable session-less crypto operations\n"
" --shared-session: share 1 session across all queue pairs on crypto device\n"
@@ -560,6 +560,10 @@ parse_op_type(struct cperf_options *opts, const char *arg)
cperf_op_type_strs[CPERF_ASYM_SM2],
CPERF_ASYM_SM2
},
+ {
+ cperf_op_type_strs[CPERF_ASYM_MLDSA44],
+ CPERF_ASYM_MLDSA44
+ },
{
cperf_op_type_strs[CPERF_ASYM_MLKEM512],
CPERF_ASYM_MLKEM512
@@ -1180,6 +1184,7 @@ cperf_options_default(struct cperf_options *opts)
opts->eddsa_data = &ed25519_perf_data;
opts->sm2_data = &sm2_perf_data;
opts->mlkem_data = &mlkem_encap_perf_data[0];
+ opts->mldsa_data = &mldsa_sign_perf_data[0];
opts->asym_op_type = RTE_CRYPTO_ASYM_OP_ENCRYPT;
}
@@ -1709,6 +1714,17 @@ cperf_options_check(struct cperf_options *options)
}
#endif
+ if (options->op_type == CPERF_ASYM_MLDSA44) {
+ if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_SIGN)
+ options->mldsa_data = &mldsa_sign_perf_data[0];
+ else if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_VERIFY)
+ options->mldsa_data = &mldsa_verify_perf_data[0];
+ else {
+ RTE_LOG(ERR, USER1, "ML-DSA only supports sign and verify operations\n");
+ return -EINVAL;
+ }
+ }
+
return 0;
}
@@ -1759,6 +1775,8 @@ cperf_options_dump(struct cperf_options *opts)
rte_crypto_asym_op_strings[opts->asym_op_type]);
if (opts->op_type == CPERF_ASYM_RSA)
printf("# rsa test name: %s\n", opts->rsa_data->name);
+ if (opts->op_type == CPERF_ASYM_MLDSA44)
+ printf("# mldsa test name: %s\n", opts->mldsa_data->name);
if (opts->op_type == CPERF_ASYM_MLKEM512)
printf("# mlkem test name: %s\n", opts->mlkem_data->name);
}
diff --git a/app/test-crypto-perf/cperf_test_common.c b/app/test-crypto-perf/cperf_test_common.c
index 383d4bd940..0bcaa6dfd8 100644
--- a/app/test-crypto-perf/cperf_test_common.c
+++ b/app/test-crypto-perf/cperf_test_common.c
@@ -314,7 +314,8 @@ cperf_is_asym_test(const struct cperf_options *options)
options->op_type == CPERF_ASYM_SECP521R1 ||
options->op_type == CPERF_ASYM_ED25519 ||
options->op_type == CPERF_ASYM_SM2 ||
- options->op_type == CPERF_ASYM_MLKEM512)
+ options->op_type == CPERF_ASYM_MLKEM512 ||
+ options->op_type == CPERF_ASYM_MLDSA44)
return true;
return false;
diff --git a/app/test-crypto-perf/cperf_test_vectors.c b/app/test-crypto-perf/cperf_test_vectors.c
index 6e2435b004..a3b050af11 100644
--- a/app/test-crypto-perf/cperf_test_vectors.c
+++ b/app/test-crypto-perf/cperf_test_vectors.c
@@ -3934,6 +3934,1624 @@ uint8_t mlkem_512_sk[] = {
0x5E, 0x7A, 0xB9, 0x3A, 0x40, 0xAD, 0x38, 0x6A
};
+uint8_t mldsa_44_seed[] = {
+ 0xf4, 0x41, 0xc9, 0x62, 0x17, 0x32, 0x2c, 0xbc,
+ 0xcf, 0x05, 0xf7, 0x5b, 0xd5, 0xfb, 0x0a, 0x0a,
+ 0x78, 0xbf, 0xad, 0xab, 0x89, 0x6e, 0x88, 0xdb,
+ 0xb4, 0x08, 0x90, 0xe0, 0x16, 0x9b, 0xb6, 0x66,
+};
+
+uint8_t mldsa_44_privkey[] = {
+ 0xcc, 0x2c, 0x93, 0xce, 0xfc, 0x0b, 0xf6, 0x74,
+ 0x93, 0x28, 0x95, 0xe8, 0xc0, 0xc8, 0x83, 0xb8,
+ 0xc9, 0x0c, 0x9a, 0x5c, 0x18, 0xd2, 0x79, 0x5c,
+ 0xf3, 0x58, 0xd8, 0x02, 0x6c, 0x5e, 0xad, 0x79,
+ 0xc1, 0xae, 0x0a, 0x97, 0x90, 0x41, 0x10, 0x32,
+ 0x59, 0x46, 0x12, 0x27, 0x0a, 0xa3, 0xc7, 0xf0,
+ 0x74, 0x54, 0x3d, 0xb2, 0x24, 0xc2, 0xa2, 0xec,
+ 0xaf, 0x31, 0xa0, 0xf4, 0x78, 0x97, 0x7c, 0x6a,
+ 0xdc, 0x91, 0x87, 0x60, 0x01, 0xa9, 0x19, 0x2d,
+ 0x0d, 0xca, 0xb4, 0x14, 0x9d, 0x41, 0xe3, 0x18,
+ 0x4b, 0x99, 0x31, 0xfa, 0xf4, 0xfa, 0xc4, 0x75,
+ 0x93, 0x87, 0xc7, 0x7a, 0x8f, 0x0e, 0xc6, 0xb2,
+ 0xfa, 0xf6, 0x48, 0x84, 0x17, 0x42, 0x01, 0xcb,
+ 0xd2, 0x55, 0x40, 0x2f, 0x23, 0x21, 0x91, 0x11,
+ 0x40, 0x00, 0xdc, 0x91, 0x3d, 0xf5, 0x96, 0xdb,
+ 0xb3, 0x6d, 0xd5, 0xf4, 0xc5, 0x11, 0x96, 0x8f,
+ 0x13, 0x25, 0x4a, 0x24, 0xb3, 0x90, 0x9b, 0x16,
+ 0x30, 0x20, 0x96, 0x10, 0x10, 0x97, 0x6c, 0xd0,
+ 0x26, 0x26, 0x61, 0x84, 0x85, 0x1b, 0xb2, 0x81,
+ 0x5c, 0x10, 0x88, 0x48, 0xc8, 0x8d, 0x1c, 0x42,
+ 0x61, 0x19, 0x32, 0x04, 0x81, 0xa6, 0x89, 0x98,
+ 0x42, 0x62, 0x14, 0xa3, 0x90, 0xc8, 0xc2, 0x4c,
+ 0x41, 0x04, 0x2c, 0x5b, 0x26, 0x2a, 0xd1, 0x30,
+ 0x12, 0x18, 0x40, 0x71, 0x08, 0x98, 0x70, 0x60,
+ 0x40, 0x09, 0x4a, 0x44, 0x0a, 0x24, 0x43, 0x92,
+ 0x04, 0x20, 0x06, 0x80, 0xa2, 0x01, 0x18, 0x47,
+ 0x12, 0x23, 0xc7, 0x6c, 0x50, 0x32, 0x61, 0xda,
+ 0x26, 0x40, 0x84, 0x98, 0x90, 0xd3, 0x04, 0x66,
+ 0x52, 0xb8, 0x70, 0x99, 0x02, 0x6c, 0x64, 0xb2,
+ 0x00, 0x1b, 0x35, 0x90, 0x23, 0x00, 0x92, 0x01,
+ 0x02, 0x20, 0x12, 0x89, 0x25, 0x62, 0xa6, 0x4d,
+ 0x1b, 0x29, 0x08, 0x08, 0x46, 0x72, 0xa1, 0x18,
+ 0x11, 0x03, 0x80, 0x60, 0xd4, 0x38, 0x80, 0x64,
+ 0x36, 0x4e, 0x23, 0x30, 0x8e, 0x23, 0x81, 0x60,
+ 0xd1, 0x84, 0x89, 0xe2, 0xa8, 0x2d, 0x80, 0x92,
+ 0x61, 0x8a, 0x22, 0x28, 0x0a, 0xc9, 0x50, 0xe3,
+ 0x96, 0x40, 0x84, 0xa0, 0x0d, 0xd8, 0xc2, 0x2c,
+ 0xd1, 0x30, 0x10, 0x18, 0xa0, 0x48, 0xe2, 0xb2,
+ 0x04, 0x00, 0xa8, 0x29, 0x19, 0x42, 0x68, 0xa3,
+ 0xa2, 0x71, 0xd9, 0xc0, 0x45, 0xa2, 0xc6, 0x4c,
+ 0x51, 0x36, 0x71, 0xc1, 0x40, 0x02, 0x08, 0x12,
+ 0x72, 0x90, 0x04, 0x2e, 0x80, 0x30, 0x24, 0x8b,
+ 0x90, 0x44, 0x82, 0x82, 0x68, 0xa2, 0x84, 0x2d,
+ 0x08, 0x00, 0x61, 0x23, 0x82, 0x48, 0x00, 0x33,
+ 0x29, 0x58, 0x48, 0x86, 0x01, 0x01, 0x21, 0x62,
+ 0xa8, 0x91, 0xe3, 0x10, 0x6c, 0x58, 0x24, 0x09,
+ 0x0c, 0x14, 0x02, 0x63, 0x22, 0x20, 0x12, 0x48,
+ 0x70, 0x81, 0x26, 0x6e, 0x00, 0x86, 0x31, 0x1a,
+ 0x22, 0x65, 0x40, 0x38, 0x00, 0x02, 0x47, 0x09,
+ 0xc9, 0x82, 0x8c, 0x82, 0xa0, 0x2c, 0xd3, 0x04,
+ 0x45, 0x50, 0xb0, 0x28, 0xd0, 0x10, 0x8d, 0x1c,
+ 0x10, 0x52, 0x4b, 0x00, 0x66, 0xc8, 0xc0, 0x08,
+ 0x9a, 0xa8, 0x2d, 0xd4, 0x02, 0x8a, 0x1b, 0x35,
+ 0x65, 0x04, 0xc7, 0x28, 0x84, 0x38, 0x50, 0x18,
+ 0x20, 0x4a, 0x91, 0x18, 0x8c, 0x9b, 0x20, 0x09,
+ 0x94, 0x08, 0x71, 0x64, 0x06, 0x25, 0x20, 0x23,
+ 0x09, 0xd8, 0x82, 0x68, 0x5a, 0xa6, 0x31, 0x63,
+ 0x36, 0x32, 0x5c, 0x12, 0x62, 0xdb, 0x32, 0x6e,
+ 0x14, 0x93, 0x68, 0x9a, 0x34, 0x80, 0x83, 0x44,
+ 0x4a, 0x18, 0x13, 0x49, 0x43, 0xb8, 0x0d, 0xda,
+ 0x92, 0x41, 0x42, 0xb8, 0x84, 0x52, 0x94, 0x91,
+ 0x04, 0x05, 0x4d, 0xa2, 0x38, 0x64, 0x01, 0x19,
+ 0x05, 0x18, 0xa2, 0x4d, 0x90, 0x34, 0x26, 0x91,
+ 0x30, 0x68, 0x1c, 0x14, 0x44, 0x4b, 0x18, 0x22,
+ 0x12, 0x90, 0x91, 0x42, 0x96, 0x91, 0x19, 0x48,
+ 0x4c, 0x24, 0x46, 0x80, 0xc3, 0xb8, 0x8c, 0x1c,
+ 0x80, 0x8d, 0x18, 0x45, 0x88, 0xd8, 0x94, 0x4c,
+ 0x13, 0x02, 0x65, 0xd3, 0x22, 0x29, 0x49, 0x04,
+ 0x82, 0x04, 0xc2, 0x40, 0x04, 0x34, 0x21, 0xe1,
+ 0x34, 0x46, 0x20, 0x90, 0x44, 0x93, 0x14, 0x44,
+ 0xdb, 0xc0, 0x4d, 0x1b, 0xa2, 0x24, 0x5a, 0x24,
+ 0x68, 0x10, 0x46, 0x66, 0x11, 0x43, 0x29, 0xd0,
+ 0x90, 0x0d, 0x49, 0x12, 0x4c, 0xc1, 0x96, 0x2c,
+ 0xdb, 0x22, 0x65, 0xe4, 0x16, 0x24, 0xd1, 0x24,
+ 0x26, 0x53, 0xb8, 0x45, 0x8a, 0xb4, 0x11, 0x04,
+ 0x96, 0x28, 0x80, 0x22, 0x8e, 0xd1, 0x10, 0x05,
+ 0x44, 0x44, 0x8a, 0x5c, 0xb4, 0x50, 0xdc, 0x10,
+ 0x02, 0xc1, 0xa2, 0x4c, 0x11, 0x05, 0x2c, 0x21,
+ 0x39, 0x6a, 0x64, 0x36, 0x01, 0x09, 0xc3, 0x0c,
+ 0x9a, 0x24, 0x52, 0x92, 0x20, 0x0a, 0xd2, 0xa8,
+ 0x49, 0x99, 0xc6, 0x8d, 0x50, 0x26, 0x62, 0x09,
+ 0x42, 0x05, 0x5b, 0xc0, 0x41, 0xd0, 0x22, 0x25,
+ 0x09, 0xb4, 0x2d, 0x98, 0xa2, 0x24, 0x12, 0x84,
+ 0x0d, 0xca, 0x98, 0x11, 0x22, 0x80, 0x21, 0x1b,
+ 0x00, 0x44, 0x84, 0xc6, 0x89, 0x92, 0x48, 0x42,
+ 0x50, 0xc8, 0x60, 0x24, 0x15, 0x69, 0xa2, 0x48,
+ 0x06, 0xc4, 0xb0, 0x24, 0x63, 0x14, 0x40, 0x12,
+ 0xc4, 0x61, 0x82, 0x30, 0x60, 0x88, 0x96, 0x89,
+ 0xc9, 0xc0, 0x89, 0x12, 0x36, 0x82, 0x8c, 0xc0,
+ 0x09, 0xc0, 0x88, 0x91, 0xdb, 0x10, 0x64, 0x08,
+ 0x36, 0x71, 0x94, 0x10, 0x46, 0x20, 0x83, 0x10,
+ 0x13, 0x13, 0x4c, 0x08, 0x28, 0x45, 0x24, 0xb9,
+ 0x64, 0x09, 0xb0, 0x48, 0x1c, 0x25, 0x50, 0x23,
+ 0x40, 0x90, 0xe0, 0x96, 0x50, 0x9a, 0x38, 0x41,
+ 0xcc, 0xb2, 0x2c, 0xdb, 0x04, 0x66, 0x61, 0x48,
+ 0x30, 0x02, 0x34, 0x4e, 0xd1, 0xc2, 0x85, 0xc9,
+ 0x06, 0x41, 0x19, 0x30, 0x69, 0xc8, 0x94, 0x68,
+ 0x1c, 0x83, 0x70, 0x03, 0x10, 0x4d, 0xc2, 0x16,
+ 0x4e, 0x64, 0x92, 0x30, 0x60, 0x16, 0x84, 0xa2,
+ 0xa8, 0x90, 0xa4, 0x38, 0x11, 0x0a, 0x88, 0x05,
+ 0x10, 0x93, 0x88, 0x43, 0x06, 0x10, 0x18, 0x10,
+ 0x70, 0x03, 0x28, 0x42, 0x89, 0x28, 0x8d, 0x8a,
+ 0xa8, 0x09, 0x94, 0x42, 0x85, 0xa3, 0x10, 0x61,
+ 0x0c, 0x45, 0x50, 0x04, 0x31, 0x4c, 0xd1, 0x10,
+ 0x12, 0x8b, 0x96, 0x05, 0x03, 0xa0, 0x89, 0x21,
+ 0xc4, 0x4d, 0x61, 0xc8, 0x61, 0x09, 0x99, 0x10,
+ 0x59, 0x96, 0x49, 0x00, 0x89, 0x08, 0x23, 0x00,
+ 0x10, 0xcc, 0xa4, 0x61, 0x01, 0x25, 0x80, 0x21,
+ 0x31, 0x09, 0x13, 0x28, 0x0d, 0x1c, 0x05, 0x85,
+ 0x62, 0x92, 0x44, 0x09, 0x10, 0x8a, 0x5a, 0xa6,
+ 0x20, 0x89, 0xb0, 0x20, 0x0c, 0xb3, 0x01, 0xc9,
+ 0x26, 0x61, 0x0b, 0x34, 0x6a, 0x8b, 0x44, 0x0e,
+ 0xf2, 0x47, 0x18, 0xf8, 0x4a, 0xfa, 0x45, 0x60,
+ 0x87, 0xeb, 0x0f, 0xff, 0x75, 0xa6, 0xba, 0x18,
+ 0x9a, 0x7d, 0x37, 0xc4, 0xc7, 0x7e, 0x1a, 0xc9,
+ 0x0d, 0x06, 0x53, 0xca, 0xf0, 0x2c, 0xe9, 0xe9,
+ 0x4b, 0x5f, 0x8b, 0xbc, 0x77, 0xc6, 0x71, 0x68,
+ 0x4f, 0x8c, 0x3b, 0x30, 0x59, 0x91, 0xf0, 0xb5,
+ 0x6d, 0xa6, 0x28, 0x2c, 0xd4, 0x0a, 0x00, 0x27,
+ 0x6a, 0x39, 0x17, 0x88, 0xa9, 0x93, 0xbb, 0x6e,
+ 0xda, 0x3e, 0x4b, 0xb7, 0x1b, 0x6a, 0xb2, 0xe2,
+ 0xc4, 0x6f, 0x21, 0x78, 0x74, 0xff, 0xa2, 0x02,
+ 0x65, 0x68, 0xa4, 0xce, 0xb4, 0xfe, 0x0c, 0x18,
+ 0x59, 0xb9, 0xc8, 0x33, 0x41, 0x27, 0xa1, 0x4b,
+ 0xcf, 0xe9, 0xa9, 0xd7, 0x2d, 0xf9, 0xcd, 0xc6,
+ 0xde, 0x4e, 0x1d, 0x7d, 0x07, 0xf4, 0xda, 0xed,
+ 0x89, 0xa1, 0xef, 0xbc, 0xec, 0x1b, 0xc3, 0x07,
+ 0xfd, 0xf5, 0x96, 0xa3, 0x6c, 0x70, 0xc8, 0xb9,
+ 0x9f, 0x46, 0x0a, 0xad, 0x46, 0x01, 0xa1, 0x92,
+ 0x7c, 0xd1, 0x12, 0x78, 0x89, 0x89, 0x98, 0x68,
+ 0x9e, 0xdd, 0xd9, 0x7b, 0x9f, 0x2b, 0x29, 0xc1,
+ 0x0e, 0x29, 0xb0, 0x44, 0x7c, 0x22, 0xd7, 0x67,
+ 0x47, 0x91, 0xf0, 0xf8, 0x56, 0x79, 0x89, 0x42,
+ 0x95, 0x60, 0x53, 0x32, 0xc1, 0x14, 0x48, 0x56,
+ 0x67, 0x00, 0x80, 0x86, 0xcd, 0x80, 0x21, 0x75,
+ 0xe8, 0xe2, 0x41, 0x6a, 0x5d, 0xb2, 0x12, 0x57,
+ 0x1e, 0x83, 0x34, 0xa4, 0x95, 0x4e, 0x0e, 0x71,
+ 0xac, 0xcb, 0x09, 0xef, 0xea, 0x4e, 0xe0, 0x4d,
+ 0x44, 0xb2, 0x67, 0xd2, 0x6e, 0x78, 0x01, 0xb5,
+ 0x09, 0x64, 0xa7, 0xdf, 0xe6, 0x16, 0x72, 0x6b,
+ 0x19, 0xe5, 0x88, 0x8a, 0x06, 0xae, 0x0a, 0x93,
+ 0xdb, 0xee, 0x0e, 0xf6, 0x52, 0x84, 0x95, 0x2b,
+ 0xf8, 0xf4, 0xcd, 0x5d, 0x8b, 0x3a, 0x86, 0x87,
+ 0x8e, 0x8b, 0x97, 0x65, 0xb1, 0xd6, 0x50, 0xbc,
+ 0xa7, 0xf0, 0x32, 0xb5, 0x39, 0x1e, 0x10, 0x7d,
+ 0x96, 0x11, 0xc1, 0xe8, 0x47, 0xfa, 0xdd, 0x24,
+ 0x22, 0xb6, 0x43, 0xb2, 0x41, 0x95, 0x98, 0x1a,
+ 0x3f, 0x0f, 0x9a, 0xd3, 0xc4, 0x10, 0x19, 0x85,
+ 0x0a, 0x0d, 0xde, 0xb7, 0x76, 0xec, 0xe3, 0x57,
+ 0x2e, 0x38, 0x4e, 0xb1, 0x2e, 0x4a, 0xaf, 0xc8,
+ 0x78, 0xa7, 0xb3, 0x1e, 0xaf, 0xe0, 0xe4, 0xbb,
+ 0xc8, 0x0b, 0x24, 0x04, 0x08, 0xd2, 0xbc, 0x91,
+ 0xca, 0x8e, 0x4e, 0x85, 0x39, 0x76, 0x20, 0xc9,
+ 0xa7, 0x47, 0x8b, 0x6a, 0xd9, 0xc8, 0xb7, 0xb0,
+ 0xa5, 0xa2, 0x85, 0x48, 0x4c, 0xcb, 0x16, 0xaf,
+ 0x45, 0x48, 0x00, 0xbe, 0x67, 0xef, 0x67, 0x65,
+ 0xa3, 0x9d, 0x43, 0x07, 0x59, 0x88, 0x9a, 0xad,
+ 0xe5, 0xad, 0x1e, 0xeb, 0xd5, 0xd4, 0xc7, 0xa4,
+ 0xb9, 0x40, 0x70, 0x2c, 0xe1, 0xcb, 0x5c, 0x3f,
+ 0x49, 0x6b, 0x51, 0xbf, 0xd3, 0xb7, 0xea, 0x33,
+ 0x52, 0x73, 0xcf, 0x26, 0x19, 0x54, 0x1d, 0xb6,
+ 0x33, 0x5d, 0x1a, 0x88, 0xdb, 0xd6, 0xfe, 0xae,
+ 0xf4, 0x45, 0x67, 0x80, 0x8a, 0xfc, 0xc4, 0xbe,
+ 0xfb, 0x6a, 0x4a, 0x75, 0x88, 0x35, 0x26, 0x1f,
+ 0x90, 0x65, 0x26, 0x5e, 0xb8, 0x0d, 0x15, 0x9d,
+ 0x70, 0x76, 0xf2, 0xfd, 0xed, 0x55, 0x71, 0x2b,
+ 0xe8, 0x2d, 0xc0, 0x28, 0xee, 0x53, 0x7c, 0xa1,
+ 0x3e, 0x86, 0x74, 0xa2, 0xd3, 0x9e, 0x2c, 0x70,
+ 0x15, 0x3a, 0x90, 0xfd, 0x1d, 0xac, 0x81, 0x2d,
+ 0xa6, 0x6a, 0xe6, 0xd7, 0x7f, 0xd9, 0x97, 0xc9,
+ 0x47, 0x7c, 0x07, 0xe6, 0x35, 0x45, 0xc9, 0x1f,
+ 0x08, 0xbc, 0x14, 0xdb, 0x12, 0x84, 0xc3, 0xc8,
+ 0xe2, 0x86, 0xa3, 0xa2, 0x1e, 0xfb, 0x3f, 0xca,
+ 0x92, 0x04, 0xe0, 0x6b, 0x29, 0xe0, 0xe0, 0x23,
+ 0x51, 0x40, 0x8e, 0xec, 0xf8, 0x87, 0xa2, 0x7a,
+ 0x2d, 0x8b, 0xc2, 0x62, 0x8c, 0x29, 0x82, 0x4b,
+ 0x86, 0x1a, 0xa7, 0xc6, 0xaa, 0x24, 0x5f, 0x24,
+ 0x03, 0x20, 0xa8, 0x3c, 0xa3, 0xcb, 0xd1, 0x60,
+ 0x64, 0x7d, 0x22, 0x9b, 0x07, 0x46, 0x1e, 0xff,
+ 0x43, 0xa9, 0x38, 0x34, 0x0f, 0x01, 0x00, 0x7f,
+ 0xc6, 0xcb, 0x0a, 0xed, 0x73, 0xc0, 0xfb, 0x77,
+ 0x80, 0x8c, 0x2c, 0x4c, 0xa5, 0x20, 0xe8, 0xcf,
+ 0x37, 0xf9, 0x00, 0x12, 0xe1, 0x6c, 0xb5, 0x9a,
+ 0x91, 0xbe, 0x1f, 0xaf, 0xf1, 0x8c, 0x6e, 0x09,
+ 0xea, 0xcf, 0xac, 0x43, 0x6e, 0x32, 0xd0, 0xf4,
+ 0x89, 0x4e, 0x42, 0xc2, 0xe7, 0x84, 0x7a, 0x66,
+ 0xb5, 0x2b, 0x5c, 0x7d, 0x83, 0x4b, 0xd4, 0x3d,
+ 0x81, 0x6d, 0x68, 0xb9, 0x72, 0xe5, 0xd5, 0xe5,
+ 0xf5, 0x9c, 0x4d, 0x95, 0xcd, 0xcd, 0x4d, 0x9a,
+ 0x4e, 0x69, 0xb9, 0x8a, 0x1e, 0x96, 0x69, 0xc0,
+ 0xe0, 0x39, 0x02, 0x21, 0x1c, 0x82, 0xd9, 0x5b,
+ 0xe5, 0xde, 0x34, 0xff, 0xf0, 0x15, 0xc7, 0x9a,
+ 0x62, 0x83, 0xca, 0xa3, 0x70, 0x15, 0xeb, 0x8f,
+ 0x73, 0x0e, 0xa9, 0x9c, 0xe9, 0xe3, 0xe7, 0xfc,
+ 0xb4, 0xcb, 0xac, 0xdf, 0x61, 0x91, 0xf3, 0x4b,
+ 0x87, 0xa6, 0x29, 0xea, 0x36, 0x81, 0x07, 0x3e,
+ 0xae, 0x69, 0xee, 0x87, 0x2a, 0x30, 0x1c, 0x97,
+ 0x96, 0x40, 0x08, 0x48, 0xcf, 0x5f, 0xfe, 0x57,
+ 0x23, 0xfd, 0xb7, 0x9a, 0x69, 0xef, 0xf5, 0x6e,
+ 0xc9, 0xdd, 0x95, 0x2b, 0x17, 0xd5, 0x2c, 0xad,
+ 0xa4, 0x49, 0x78, 0xb1, 0xd1, 0x00, 0xe7, 0xca,
+ 0xcb, 0x20, 0x9e, 0xba, 0xea, 0xc8, 0xd4, 0x9b,
+ 0x84, 0xac, 0x0b, 0xf0, 0x3b, 0x2c, 0xf0, 0x95,
+ 0x3b, 0x5b, 0xf8, 0xfe, 0xe5, 0xc6, 0xee, 0x7c,
+ 0x9e, 0x41, 0xf0, 0xef, 0xcf, 0x4d, 0xdd, 0xe8,
+ 0x63, 0x6b, 0xcf, 0xe9, 0xd2, 0x80, 0x98, 0xf9,
+ 0x7d, 0x22, 0x7e, 0xfc, 0xdd, 0x50, 0x6b, 0xa3,
+ 0x6d, 0x7c, 0xed, 0x35, 0xbc, 0x28, 0x64, 0x03,
+ 0xe6, 0x01, 0xac, 0x7c, 0xe1, 0x9b, 0x08, 0x6f,
+ 0xd8, 0xdc, 0xaf, 0xb6, 0x9d, 0x4c, 0xdf, 0xb1,
+ 0xde, 0xf4, 0x90, 0x5b, 0x46, 0xee, 0x7d, 0x28,
+ 0xfb, 0xaa, 0xa9, 0x69, 0x47, 0xf5, 0x52, 0x74,
+ 0x97, 0x9f, 0x8b, 0xbb, 0x26, 0x8f, 0x0d, 0x89,
+ 0x2f, 0xc9, 0x57, 0xac, 0x2f, 0x88, 0x12, 0xdd,
+ 0x12, 0x2f, 0x75, 0x97, 0xc7, 0x7b, 0x45, 0x46,
+ 0x3a, 0x52, 0x4b, 0xf6, 0x76, 0xe0, 0x99, 0xef,
+ 0xbe, 0xbf, 0xed, 0x0d, 0x32, 0xed, 0xc6, 0x5e,
+ 0x6a, 0xe6, 0x0e, 0x12, 0xd4, 0xce, 0xf8, 0x0c,
+ 0x7b, 0x4f, 0x1d, 0x3b, 0xf8, 0xaf, 0xdd, 0xd1,
+ 0x78, 0x74, 0x59, 0xc7, 0xe4, 0xbc, 0xe3, 0xdd,
+ 0x4a, 0xa7, 0x77, 0xa5, 0xcd, 0x80, 0x7c, 0x6b,
+ 0x19, 0x46, 0x3d, 0xbd, 0x0a, 0x75, 0x48, 0xfa,
+ 0x1c, 0xee, 0x02, 0xe9, 0x79, 0x10, 0x6c, 0xb4,
+ 0xb3, 0xb2, 0xbe, 0xc8, 0x5c, 0xd1, 0xa1, 0xe2,
+ 0xd5, 0x0a, 0xb5, 0xf3, 0x0c, 0x3f, 0x03, 0xb9,
+ 0x2f, 0x61, 0x44, 0x95, 0xe1, 0x16, 0xcb, 0xc1,
+ 0xee, 0xb3, 0xb8, 0x85, 0xd6, 0x1c, 0xf4, 0xfa,
+ 0x86, 0x73, 0xc5, 0xf3, 0xba, 0xb5, 0xe0, 0x95,
+ 0x34, 0x95, 0x72, 0xfa, 0xa7, 0xb8, 0x50, 0xba,
+ 0xfb, 0xa6, 0x3c, 0x0f, 0x3c, 0xe3, 0xaa, 0x56,
+ 0x05, 0xaf, 0x08, 0xec, 0x50, 0x12, 0x73, 0x25,
+ 0x29, 0x72, 0xf5, 0xf6, 0xee, 0x06, 0xe4, 0x05,
+ 0x1a, 0xe1, 0x1a, 0x13, 0x3f, 0xa3, 0x55, 0x16,
+ 0x12, 0x17, 0x88, 0xd2, 0xa7, 0xc1, 0xd0, 0xf2,
+ 0x7d, 0x51, 0x46, 0x36, 0x2e, 0x77, 0xa8, 0x02,
+ 0x67, 0xad, 0xe7, 0x63, 0xaa, 0xbf, 0xe5, 0xbf,
+ 0xa1, 0xab, 0xc4, 0xf6, 0x72, 0xae, 0xdf, 0x79,
+ 0x0e, 0x82, 0x47, 0xb5, 0xf4, 0x48, 0x39, 0x22,
+ 0xe3, 0x0a, 0x94, 0x5e, 0x3a, 0x4b, 0xa6, 0xa7,
+ 0x5c, 0xb8, 0xec, 0x3e, 0x10, 0xae, 0x94, 0x34,
+ 0x2c, 0xc0, 0x4b, 0xdd, 0xca, 0x4d, 0xc9, 0x7c,
+ 0xf8, 0x2e, 0x32, 0x4e, 0x00, 0x2c, 0x34, 0x71,
+ 0x28, 0xad, 0xd5, 0xcb, 0x19, 0xb8, 0x1d, 0xaa,
+ 0x85, 0xac, 0xec, 0x16, 0xb6, 0x1f, 0x95, 0xc5,
+ 0xa9, 0x4b, 0x8a, 0xfc, 0x70, 0xb7, 0x34, 0xb0,
+ 0x63, 0x55, 0xff, 0x15, 0xc2, 0x72, 0x74, 0xe0,
+ 0x6d, 0x01, 0x3d, 0xf0, 0x6a, 0xab, 0x4b, 0x8a,
+ 0x72, 0xdc, 0x2b, 0x0a, 0x8a, 0xf4, 0x02, 0x1b,
+ 0x0b, 0x51, 0xd8, 0xf7, 0x17, 0xcf, 0x48, 0xf0,
+ 0x97, 0xa0, 0xc2, 0x7a, 0xc4, 0x64, 0x59, 0x2a,
+ 0xa3, 0x27, 0xbe, 0x3b, 0xf7, 0xc5, 0x6d, 0x51,
+ 0x29, 0xe2, 0x68, 0x87, 0x75, 0x71, 0x72, 0xac,
+ 0xc8, 0xc1, 0xc8, 0xe5, 0xf6, 0x8a, 0x27, 0x8c,
+ 0x45, 0x03, 0x10, 0xf7, 0x1a, 0xff, 0x1b, 0x04,
+ 0x1c, 0xaf, 0x27, 0x7a, 0x57, 0xac, 0x0f, 0x57,
+ 0x2a, 0x74, 0xd0, 0x46, 0x64, 0xbf, 0xc9, 0xe8,
+ 0x35, 0x6b, 0x5c, 0x79, 0x9a, 0x51, 0xeb, 0xd1,
+ 0xb3, 0x1d, 0xfd, 0x2f, 0x83, 0x2e, 0x24, 0x95,
+ 0x94, 0xc7, 0x9d, 0x9f, 0x5d, 0x5c, 0x8e, 0x6c,
+ 0x8f, 0xfc, 0xce, 0x89, 0x67, 0x9b, 0x38, 0x4b,
+ 0x0d, 0x44, 0x9a, 0xb5, 0x74, 0xbf, 0x88, 0x90,
+ 0xa1, 0xf2, 0xc4, 0xd2, 0xb3, 0x07, 0xe5, 0x34,
+ 0x62, 0xc0, 0xfc, 0x3f, 0xdb, 0x77, 0x8e, 0x84,
+ 0x61, 0xc1, 0x15, 0xf6, 0x5f, 0x5d, 0x73, 0xbc,
+ 0x0c, 0x70, 0x32, 0xeb, 0x2c, 0xd3, 0x19, 0x83,
+ 0x22, 0xfa, 0x5b, 0xcc, 0x5f, 0xbf, 0xbe, 0xfc,
+ 0x3c, 0x28, 0x74, 0x2d, 0x33, 0x8f, 0x74, 0xe5,
+ 0xca, 0xc9, 0x84, 0x8f, 0xba, 0x48, 0x11, 0x84,
+ 0x33, 0xf0, 0x7d, 0x5c, 0xad, 0x47, 0x36, 0xf7,
+ 0x33, 0xb0, 0xbb, 0x44, 0x65, 0x72, 0x45, 0xd1,
+ 0x40, 0x4c, 0x39, 0x8b, 0x16, 0x90, 0x74, 0xa3,
+ 0xd2, 0x3c, 0xc6, 0x7a, 0xa4, 0x4f, 0x0d, 0x9a,
+ 0xe1, 0xfd, 0x49, 0x82, 0xc6, 0x69, 0xf7, 0xe7,
+ 0x3d, 0xeb, 0x2b, 0xbe, 0xfd, 0x1a, 0x73, 0x2c,
+ 0xf3, 0xbc, 0xd6, 0xaa, 0xaa, 0x54, 0x22, 0x5c,
+ 0x54, 0x92, 0x73, 0xe3, 0x79, 0x0a, 0x87, 0x20,
+ 0x82, 0xdb, 0x02, 0x42, 0x81, 0xb1, 0xf7, 0x0d,
+ 0xf0, 0xd8, 0x56, 0x0f, 0x74, 0x0c, 0x65, 0x7c,
+ 0x8c, 0x96, 0xb9, 0x9f, 0xfa, 0xb6, 0x48, 0x66,
+ 0xf1, 0xbe, 0xf8, 0x46, 0x70, 0xe5, 0x26, 0xc1,
+ 0x68, 0x7f, 0x81, 0x72, 0x14, 0x2f, 0x96, 0x22,
+ 0x05, 0xd1, 0xc0, 0xb8, 0x93, 0x9d, 0x02, 0x8b,
+ 0xff, 0x0b, 0x2d, 0xb9, 0x01, 0xaa, 0x61, 0x09,
+ 0x9d, 0x55, 0x21, 0xdd, 0xda, 0xec, 0x44, 0xce,
+ 0x00, 0x98, 0xf8, 0x34, 0x86, 0x49, 0xd4, 0x72,
+ 0x90, 0xfb, 0xd7, 0xb1, 0x33, 0xb9, 0xe5, 0xd3,
+ 0xc0, 0x27, 0x88, 0x4b, 0x6d, 0x84, 0x34, 0x7c,
+ 0x77, 0x7e, 0xcf, 0x27, 0x4d, 0x5f, 0x1d, 0xb4,
+ 0xb4, 0xac, 0xcc, 0x7d, 0x83, 0xc2, 0x89, 0x3d,
+ 0xeb, 0xb6, 0xa3, 0x57, 0x30, 0xbc, 0x94, 0x9f,
+ 0xe9, 0x63, 0xca, 0x7d, 0xa3, 0x9d, 0x0a, 0x43,
+ 0xed, 0x4a, 0xfd, 0xd3, 0x5e, 0x23, 0x97, 0x75,
+ 0x28, 0x97, 0xaa, 0x0c, 0xf9, 0xa1, 0x03, 0xb1,
+ 0x55, 0x82, 0xec, 0xcc, 0x91, 0x27, 0xf4, 0xcf,
+ 0x63, 0x10, 0xf3, 0xd3, 0x8b, 0x5c, 0xa8, 0xc2,
+ 0x58, 0x4d, 0xf5, 0x67, 0xe9, 0xe1, 0xc2, 0xf2,
+ 0x11, 0x5e, 0xf2, 0xe1, 0x1c, 0xf2, 0x0c, 0x50,
+ 0x71, 0x9f, 0xd1, 0x1c, 0xb8, 0x82, 0x7d, 0x3b,
+ 0x23, 0x90, 0x4e, 0x08, 0xbf, 0x51, 0xa1, 0xc9,
+ 0xe9, 0x90, 0x07, 0x4e, 0xb3, 0xcc, 0x72, 0xbf,
+ 0x2c, 0x87, 0x18, 0x3b, 0xc7, 0x29, 0x8a, 0x3b,
+ 0x0b, 0x4c, 0x55, 0x20, 0x08, 0xb0, 0xa3, 0x04,
+ 0xf9, 0x6b, 0x53, 0x84, 0x4c, 0xef, 0xc1, 0xc6,
+ 0xa4, 0xa9, 0xba, 0xe6, 0xb8, 0x72, 0x60, 0x6b,
+ 0x2d, 0x25, 0x08, 0xc1, 0xd4, 0x4e, 0x0b, 0xcb,
+ 0xe9, 0x71, 0x9b, 0x5a, 0x9a, 0x1e, 0x21, 0x9a,
+ 0x0a, 0xd0, 0x52, 0x82, 0xe9, 0x31, 0x45, 0x9e,
+ 0xe4, 0x85, 0x6b, 0xac, 0x8e, 0x7c, 0x73, 0x0c,
+ 0x0d, 0xd6, 0x54, 0x27, 0x5a, 0xce, 0xc8, 0x07,
+ 0x19, 0x84, 0x8b, 0xee, 0x0f, 0x6f, 0x8b, 0xbc,
+ 0x0c, 0x1b, 0xe0, 0x08, 0x64, 0xa6, 0xe6, 0xe1,
+ 0xf8, 0x56, 0x5e, 0xe8, 0x27, 0x7d, 0x26, 0xee,
+ 0x20, 0x5e, 0x3d, 0x2e, 0x84, 0xdc, 0x50, 0xb0,
+ 0x16, 0x3f, 0x8c, 0xec, 0xa9, 0xd3, 0x61, 0x85,
+ 0x5a, 0xba, 0x02, 0x82, 0x65, 0x30, 0xdb, 0x23,
+ 0x1c, 0xcb, 0xc3, 0xe6, 0x80, 0x5e, 0x4f, 0x6e,
+ 0xd8, 0x3b, 0x5e, 0xce, 0x67, 0xbb, 0xdb, 0x22,
+ 0xac, 0xf1, 0x32, 0x53, 0x6a, 0xd6, 0x43, 0x1a,
+ 0xf0, 0x11, 0xc5, 0x0f, 0xc5, 0x12, 0xdf, 0xe6,
+ 0x91, 0x8b, 0x15, 0xf4, 0x41, 0xd9, 0xe3, 0x51,
+};
+
+uint8_t mldsa_44_pubkey[] = {
+ 0xcc, 0x2c, 0x93, 0xce, 0xfc, 0x0b, 0xf6, 0x74,
+ 0x93, 0x28, 0x95, 0xe8, 0xc0, 0xc8, 0x83, 0xb8,
+ 0xc9, 0x0c, 0x9a, 0x5c, 0x18, 0xd2, 0x79, 0x5c,
+ 0xf3, 0x58, 0xd8, 0x02, 0x6c, 0x5e, 0xad, 0x79,
+ 0xde, 0x77, 0x1a, 0xd3, 0x49, 0xc9, 0x12, 0xc9,
+ 0xff, 0xcf, 0xa2, 0x6c, 0x1d, 0x37, 0x91, 0xa2,
+ 0xf5, 0x43, 0xbc, 0xdb, 0x9e, 0xdb, 0xa5, 0xdb,
+ 0xd9, 0x87, 0xc5, 0xf2, 0xf7, 0x79, 0x58, 0x4c,
+ 0x08, 0x9f, 0x9c, 0xca, 0x9c, 0xef, 0xa8, 0x80,
+ 0x27, 0x31, 0xf1, 0xc5, 0x0e, 0x2b, 0x6d, 0xa1,
+ 0x53, 0x9e, 0x99, 0x1e, 0xc8, 0x75, 0x4a, 0x59,
+ 0x27, 0x33, 0x41, 0xdd, 0x39, 0xff, 0x37, 0xdf,
+ 0x5b, 0xb9, 0xe4, 0x56, 0x54, 0x28, 0x8e, 0xb1,
+ 0xfe, 0xd8, 0x4e, 0x60, 0xcd, 0x22, 0xd7, 0x30,
+ 0xe5, 0x73, 0xe4, 0xb1, 0x08, 0x0f, 0x0a, 0xbe,
+ 0x9a, 0x44, 0xfd, 0xb1, 0xed, 0xb1, 0x8e, 0x0d,
+ 0x7c, 0x3c, 0x3d, 0x04, 0x52, 0x4b, 0x93, 0xf4,
+ 0xa6, 0xce, 0x8d, 0xb0, 0xe4, 0xf6, 0xb1, 0x09,
+ 0xfc, 0xc3, 0x42, 0x3d, 0xff, 0x4c, 0x55, 0x3c,
+ 0x73, 0x75, 0x35, 0x9d, 0xe8, 0x68, 0x42, 0x09,
+ 0x14, 0x36, 0x91, 0x63, 0xfc, 0xc6, 0x23, 0x7c,
+ 0x25, 0x81, 0xd5, 0xc1, 0xfe, 0xca, 0xf6, 0x71,
+ 0x51, 0x8e, 0xab, 0x29, 0xa1, 0x86, 0xbb, 0x45,
+ 0x43, 0x67, 0x7f, 0xdf, 0x7e, 0x92, 0xff, 0x35,
+ 0x38, 0xd3, 0xea, 0x94, 0xc9, 0xa3, 0x0f, 0x46,
+ 0x25, 0xa6, 0x1e, 0x00, 0x60, 0x7b, 0xc0, 0xbc,
+ 0xe9, 0x5e, 0x16, 0x0e, 0x81, 0xf5, 0x4e, 0x98,
+ 0xa1, 0x64, 0xb0, 0xb7, 0x02, 0xec, 0x73, 0xad,
+ 0xf8, 0xc1, 0xce, 0x8b, 0x8f, 0xbd, 0x89, 0xbf,
+ 0x0f, 0x42, 0x31, 0x6d, 0x75, 0x42, 0xd7, 0x59,
+ 0x64, 0xad, 0x09, 0xd3, 0x7a, 0x00, 0x7b, 0xdd,
+ 0x12, 0x76, 0xb0, 0x00, 0x73, 0x5b, 0xbf, 0x44,
+ 0x54, 0x6e, 0x56, 0x26, 0xa5, 0x27, 0x4d, 0xff,
+ 0xe5, 0x8a, 0x04, 0x73, 0xd9, 0x75, 0x8c, 0xf7,
+ 0x06, 0x64, 0xfb, 0xa5, 0x00, 0x27, 0x39, 0x0e,
+ 0x48, 0x8f, 0x73, 0x29, 0x61, 0x5f, 0x15, 0xf5,
+ 0x08, 0x15, 0x33, 0xd1, 0x76, 0xba, 0xf3, 0x3e,
+ 0x28, 0xb8, 0x57, 0xcd, 0x9d, 0x61, 0x1f, 0xca,
+ 0xd7, 0xc3, 0x10, 0xdb, 0x68, 0xeb, 0xa4, 0x15,
+ 0x40, 0xd8, 0xe7, 0xa0, 0xd2, 0xd1, 0xd6, 0xb1,
+ 0xd3, 0x75, 0x8e, 0xc1, 0x60, 0x02, 0xb5, 0x69,
+ 0xf2, 0x8c, 0xf3, 0xc6, 0x2e, 0x9d, 0xf7, 0x8d,
+ 0xed, 0xb0, 0x01, 0xba, 0xb5, 0x62, 0x7e, 0x8f,
+ 0x91, 0xbd, 0x73, 0xc6, 0x35, 0xf9, 0xdb, 0xea,
+ 0x28, 0xce, 0x8a, 0x47, 0x74, 0x7c, 0x3d, 0x85,
+ 0x4d, 0x1e, 0x11, 0x7e, 0xa2, 0xaa, 0x04, 0x30,
+ 0x12, 0xce, 0xea, 0xc1, 0xc5, 0x1e, 0x62, 0x7a,
+ 0x21, 0x6a, 0xd5, 0xb8, 0x3c, 0xa0, 0xd6, 0xca,
+ 0xb4, 0x8d, 0xff, 0xc3, 0xfc, 0xf2, 0xf1, 0x3c,
+ 0x87, 0x23, 0xef, 0x68, 0xe4, 0x37, 0x96, 0xb0,
+ 0xbb, 0x78, 0xa3, 0x3a, 0xe0, 0xd0, 0x41, 0xeb,
+ 0x4c, 0x80, 0x55, 0x9b, 0x66, 0x5e, 0x33, 0xea,
+ 0x33, 0x35, 0x79, 0xda, 0x04, 0x81, 0x19, 0x22,
+ 0xb4, 0x4c, 0x7a, 0xef, 0x51, 0x50, 0x20, 0x6c,
+ 0xf8, 0xf7, 0x5f, 0x03, 0x74, 0x72, 0xda, 0x1f,
+ 0xe3, 0x0b, 0x82, 0x71, 0x50, 0x3d, 0x02, 0x61,
+ 0x16, 0x44, 0xeb, 0xcf, 0xe3, 0x23, 0x9d, 0xef,
+ 0xec, 0xce, 0xb3, 0xd0, 0xd2, 0x19, 0xd1, 0xe0,
+ 0x2e, 0x33, 0xde, 0x28, 0x4b, 0xde, 0x85, 0xc1,
+ 0xb2, 0x8a, 0xa5, 0x82, 0x0c, 0xa6, 0x9f, 0x23,
+ 0x53, 0xea, 0xef, 0x78, 0x60, 0x24, 0x3a, 0x40,
+ 0xc0, 0xc1, 0xac, 0xc9, 0x64, 0xd4, 0x03, 0x5f,
+ 0x61, 0xa5, 0xdf, 0x6f, 0x6b, 0x4e, 0xb2, 0xe7,
+ 0x1a, 0xc6, 0x69, 0x69, 0xd9, 0xc6, 0x6e, 0x5d,
+ 0xea, 0xd8, 0xc0, 0x29, 0xcc, 0x8f, 0x35, 0x7b,
+ 0xcb, 0x48, 0x3a, 0xfd, 0xba, 0x7a, 0x90, 0xc7,
+ 0x65, 0x55, 0xfc, 0x90, 0x98, 0x6c, 0x15, 0x38,
+ 0x0e, 0x88, 0x6e, 0x08, 0x49, 0x85, 0xc6, 0x6d,
+ 0x8f, 0x30, 0x37, 0xa1, 0x64, 0x79, 0xa1, 0x65,
+ 0x2d, 0x07, 0x38, 0x3e, 0x1c, 0xa6, 0x12, 0xee,
+ 0xc5, 0x16, 0x68, 0x5c, 0x93, 0x14, 0xda, 0x33,
+ 0x3d, 0x74, 0x62, 0xf1, 0xcf, 0x0e, 0x81, 0x80,
+ 0xda, 0xea, 0xa7, 0x08, 0xde, 0xf3, 0xdc, 0xfe,
+ 0x3a, 0xf5, 0x60, 0xe9, 0x35, 0x95, 0x52, 0x6c,
+ 0x7f, 0xef, 0x8b, 0xa7, 0x55, 0x3d, 0x42, 0xfd,
+ 0x39, 0x24, 0xd2, 0xfb, 0x28, 0x08, 0x83, 0x2b,
+ 0x44, 0xf1, 0x21, 0x73, 0x13, 0x88, 0xa7, 0xba,
+ 0x2d, 0xd6, 0xe9, 0x4e, 0xcb, 0xfe, 0x18, 0xe4,
+ 0xc7, 0x85, 0xf3, 0xb8, 0xab, 0xd0, 0xb9, 0xbe,
+ 0x5e, 0x8a, 0x62, 0x0e, 0x61, 0xa3, 0x29, 0x1d,
+ 0xef, 0x46, 0xe7, 0x70, 0x14, 0xb1, 0x9f, 0x07,
+ 0xbf, 0x3f, 0x1f, 0xc8, 0x11, 0xe4, 0x2c, 0x39,
+ 0x48, 0x0e, 0xcf, 0x92, 0x62, 0x4b, 0x1f, 0x0c,
+ 0x2e, 0x29, 0x61, 0xef, 0x29, 0x67, 0x7c, 0x0f,
+ 0xac, 0x99, 0xb2, 0x9a, 0x7d, 0xac, 0x7c, 0x54,
+ 0xae, 0xed, 0x4b, 0xf5, 0x4a, 0x4b, 0xb4, 0x14,
+ 0xad, 0x6d, 0xce, 0xce, 0xcc, 0x3f, 0xb8, 0x59,
+ 0x71, 0x7c, 0xd4, 0x76, 0x7c, 0xe4, 0x68, 0xc6,
+ 0x76, 0xa8, 0xe3, 0x03, 0x48, 0xb6, 0xe4, 0x11,
+ 0x6c, 0x5f, 0xd4, 0x76, 0x80, 0xdf, 0x2c, 0x67,
+ 0x00, 0x4c, 0xce, 0x22, 0x21, 0xff, 0xb0, 0x9b,
+ 0xcb, 0x72, 0xd1, 0xe3, 0xa2, 0x32, 0x5c, 0x12,
+ 0xda, 0x7c, 0x95, 0x5d, 0xb2, 0xae, 0x1b, 0x25,
+ 0x2e, 0x5f, 0xe6, 0xff, 0x4c, 0xbf, 0x00, 0x16,
+ 0xd0, 0x3c, 0xbf, 0x44, 0x92, 0x07, 0xa0, 0x0d,
+ 0x13, 0xba, 0xb8, 0x62, 0x66, 0x0b, 0x14, 0x0f,
+ 0x5a, 0x9d, 0x42, 0x19, 0xd7, 0xbe, 0xe3, 0x33,
+ 0x53, 0xa2, 0x60, 0x2d, 0xde, 0x43, 0x85, 0x03,
+ 0x96, 0x50, 0x83, 0x83, 0x25, 0x3c, 0x19, 0x20,
+ 0x44, 0x5f, 0x3b, 0x5a, 0xc2, 0x90, 0xe1, 0x2d,
+ 0x47, 0x53, 0x50, 0xeb, 0xf7, 0xc7, 0x46, 0x4c,
+ 0xf2, 0xad, 0xb2, 0x8f, 0x03, 0x4e, 0x82, 0x5d,
+ 0x0f, 0xa0, 0x3d, 0xde, 0x4d, 0x93, 0x34, 0xd0,
+ 0x1a, 0xe2, 0x5d, 0xbc, 0xe5, 0x8d, 0xfb, 0x5a,
+ 0xba, 0xe2, 0x24, 0xec, 0xc8, 0xd2, 0xa3, 0x91,
+ 0xa2, 0xa9, 0x0e, 0x0e, 0xf5, 0xfb, 0x9f, 0xed,
+ 0x69, 0x21, 0x9d, 0x00, 0x92, 0xb5, 0x94, 0x0f,
+ 0x38, 0x94, 0x29, 0xf9, 0xe7, 0xaa, 0xf1, 0xf7,
+ 0x20, 0x10, 0xf0, 0xeb, 0x26, 0x96, 0x5a, 0x0e,
+ 0x99, 0x8d, 0x71, 0xef, 0xb2, 0xa0, 0xf3, 0x38,
+ 0xce, 0xf9, 0x99, 0x6f, 0x96, 0xbb, 0xa5, 0x55,
+ 0x27, 0x5c, 0xf5, 0xf8, 0x63, 0xf7, 0xf8, 0x0a,
+ 0x31, 0xee, 0x01, 0xa8, 0xc9, 0x0c, 0xbd, 0x73,
+ 0x62, 0x21, 0xa7, 0x1c, 0xd1, 0x62, 0xe5, 0xdd,
+ 0x43, 0x95, 0x4d, 0x60, 0x11, 0x65, 0x6c, 0xfa,
+ 0x67, 0x9a, 0x2f, 0x24, 0x1e, 0xbd, 0x10, 0xc4,
+ 0xe5, 0x23, 0x6f, 0x02, 0x76, 0x51, 0xd6, 0xe3,
+ 0xad, 0x88, 0xda, 0xc4, 0xd9, 0x6a, 0x8f, 0xf7,
+ 0xd2, 0x50, 0xdf, 0x9a, 0xad, 0x21, 0xc7, 0x5b,
+ 0x47, 0x83, 0xf6, 0xc8, 0xcb, 0x0a, 0xd5, 0x28,
+ 0x91, 0x3f, 0x18, 0x44, 0x62, 0x81, 0xfb, 0xcf,
+ 0x51, 0x5d, 0xa1, 0xf5, 0x46, 0x74, 0x0d, 0x32,
+ 0x21, 0x55, 0x1b, 0x8a, 0xf3, 0xd0, 0x4f, 0x41,
+ 0x86, 0xc3, 0x55, 0x0b, 0x1c, 0xd9, 0x56, 0x2f,
+ 0xcf, 0xe7, 0x9f, 0x06, 0x73, 0x82, 0xc2, 0x30,
+ 0x7a, 0xc2, 0x01, 0xa2, 0x1e, 0xeb, 0x5d, 0x77,
+ 0x7f, 0xbe, 0xee, 0xf4, 0x8a, 0x1b, 0x3b, 0xe3,
+ 0x2d, 0xd5, 0x3e, 0x3b, 0x40, 0x82, 0xd6, 0x60,
+ 0x86, 0x2f, 0xe9, 0xac, 0xc3, 0x65, 0xa5, 0xa3,
+ 0x8c, 0x54, 0x51, 0x1f, 0x8e, 0x89, 0x19, 0xdd,
+ 0xc5, 0x4a, 0xc0, 0xa2, 0x8b, 0xe5, 0x80, 0xfd,
+ 0xa1, 0xd8, 0xe9, 0x15, 0xa0, 0x94, 0xd7, 0x9e,
+ 0xe2, 0xf1, 0x83, 0x52, 0xf3, 0x0e, 0x34, 0x2f,
+ 0x85, 0x67, 0x49, 0x73, 0x67, 0xb9, 0xb5, 0xd5,
+ 0x0c, 0x1d, 0x03, 0x8e, 0x68, 0x57, 0x8d, 0xd2,
+ 0x33, 0x44, 0x94, 0x28, 0x5b, 0xfa, 0xe0, 0x2b,
+ 0x80, 0xd9, 0x96, 0x85, 0x6e, 0x0a, 0x2a, 0xdd,
+ 0x9b, 0x5b, 0x55, 0x5a, 0x31, 0xb6, 0xfe, 0x3e,
+ 0x0a, 0x41, 0x28, 0x20, 0xa7, 0xde, 0x31, 0xbe,
+ 0x73, 0xcb, 0xbc, 0xbb, 0x50, 0xf8, 0x58, 0x74,
+ 0xb1, 0x6b, 0x3f, 0x9a, 0x6c, 0x5e, 0x02, 0x87,
+ 0x8d, 0x19, 0xa1, 0xd3, 0x6a, 0xc2, 0x91, 0x82,
+ 0x3e, 0x0c, 0x90, 0xb8, 0xa6, 0x95, 0x23, 0x4e,
+ 0xb9, 0x2d, 0x4d, 0x6c, 0xfb, 0xd7, 0x5e, 0xdf,
+ 0xca, 0x06, 0x9b, 0x94, 0xb7, 0xfc, 0xbc, 0xad,
+ 0x39, 0x5d, 0x43, 0xcb, 0x1c, 0x7d, 0x3c, 0x95,
+ 0x3f, 0xf4, 0x47, 0x04, 0x1f, 0xfc, 0x8b, 0x23,
+ 0xe7, 0xcf, 0x24, 0x6f, 0x40, 0x9b, 0xa5, 0x56,
+ 0xcd, 0x69, 0x9e, 0x1f, 0x5a, 0xa0, 0x3b, 0x8f,
+ 0x3e, 0x1c, 0xe7, 0x42, 0xc9, 0x66, 0xee, 0x99,
+ 0x13, 0xb1, 0x4f, 0x65, 0x86, 0xba, 0xf0, 0x87,
+ 0x81, 0x6f, 0x44, 0x08, 0xb4, 0x18, 0xac, 0xb6,
+ 0xd3, 0x96, 0x68, 0xd4, 0x88, 0x89, 0xf2, 0xa2,
+ 0xd6, 0x2f, 0xfe, 0x7e, 0x49, 0xb3, 0x9c, 0xc9,
+ 0xaf, 0xd9, 0x28, 0xa8, 0x96, 0xc9, 0x2d, 0x26,
+ 0x5b, 0xc7, 0xef, 0x66, 0x08, 0xd2, 0x38, 0x4f,
+ 0x10, 0xb3, 0x83, 0x15, 0xf6, 0x00, 0x83, 0x3f,
+ 0x20, 0xfe, 0xa8, 0x44, 0x6b, 0x62, 0x3b, 0x17,
+ 0x39, 0x2e, 0xec, 0x5e, 0x78, 0xbe, 0xec, 0x16,
+ 0x29, 0xa6, 0x79, 0x4c, 0x08, 0x75, 0xc8, 0x78,
+ 0x3e, 0xc0, 0x05, 0xe9, 0xbb, 0x47, 0x94, 0xae,
+ 0xaf, 0xa5, 0xbb, 0xb8, 0x47, 0x31, 0xf7, 0xe9,
+ 0xb8, 0x1e, 0x6c, 0xda, 0x26, 0xc5, 0xf3, 0x26,
+ 0x89, 0xd5, 0x04, 0x23, 0x0b, 0x11, 0x84, 0x8f,
+};
+
+uint8_t mldsa_44_sign[] = {
+ 0x3E, 0xDD, 0xD2, 0x34, 0x62, 0x78, 0xEC, 0x19,
+ 0x9F, 0xAA, 0xEC, 0x89, 0x99, 0x78, 0x31, 0xCB,
+ 0x82, 0x0B, 0xE6, 0x46, 0x8D, 0x24, 0x9A, 0xD3,
+ 0x69, 0xB7, 0x01, 0x85, 0x93, 0x75, 0xD0, 0xBE,
+ 0xEA, 0x1F, 0x76, 0xFB, 0xB4, 0x6D, 0xC9, 0x64,
+ 0xD4, 0x44, 0x68, 0x5D, 0xAD, 0x09, 0x69, 0xFC,
+ 0x31, 0x2F, 0xD1, 0xFA, 0xEF, 0xF2, 0x1D, 0x2E,
+ 0xB6, 0xA5, 0xA9, 0xFD, 0x31, 0x04, 0x68, 0x1E,
+ 0x0B, 0xEC, 0xF4, 0x23, 0x4C, 0x9F, 0xC9, 0xD6,
+ 0x27, 0x8D, 0xE1, 0xAA, 0x29, 0x38, 0x16, 0x65,
+ 0x7E, 0x38, 0x5A, 0x30, 0xFC, 0xD9, 0xBF, 0x63,
+ 0x3B, 0x82, 0xFE, 0x0D, 0x68, 0xD4, 0x52, 0x55,
+ 0xBE, 0x86, 0x69, 0xCF, 0x75, 0x26, 0x2C, 0xB9,
+ 0x1D, 0x66, 0x39, 0x4C, 0x89, 0xAC, 0x36, 0xBF,
+ 0x34, 0x27, 0xCC, 0x7E, 0x6C, 0xC5, 0xBB, 0xFB,
+ 0x78, 0x03, 0x39, 0x61, 0xD8, 0x76, 0x63, 0x6E,
+ 0x6B, 0x68, 0x02, 0x43, 0x44, 0x57, 0x2E, 0x39,
+ 0x9E, 0x9D, 0x64, 0x77, 0x8E, 0x8B, 0x79, 0x36,
+ 0xCE, 0xE6, 0xBC, 0x6D, 0x80, 0xC8, 0x04, 0x81,
+ 0x2A, 0x04, 0xD4, 0xEF, 0x63, 0xE1, 0x3F, 0xC4,
+ 0xC1, 0x54, 0xD5, 0xAB, 0xE4, 0xEC, 0x65, 0xB0,
+ 0xF0, 0x1A, 0xB2, 0x32, 0x9A, 0xB8, 0x51, 0xD4,
+ 0x43, 0xFE, 0x81, 0x4F, 0xBD, 0x5D, 0xEE, 0xDE,
+ 0x24, 0xAC, 0xBC, 0x22, 0x9B, 0x80, 0xB7, 0xE8,
+ 0x22, 0x4B, 0x1B, 0x23, 0x89, 0x8C, 0xFE, 0xE3,
+ 0x30, 0x35, 0xA2, 0x0B, 0x4E, 0x66, 0x64, 0xFE,
+ 0x57, 0x68, 0xCF, 0xF5, 0xE5, 0x11, 0xB9, 0xB6,
+ 0x3A, 0x2B, 0x15, 0x0D, 0xA4, 0x11, 0xE1, 0x01,
+ 0x96, 0x06, 0x5D, 0x47, 0xCC, 0x04, 0x63, 0xB3,
+ 0xC7, 0xDD, 0x0F, 0x4A, 0x0A, 0x90, 0x9C, 0x0C,
+ 0x61, 0x1D, 0x4C, 0x21, 0x32, 0xD6, 0xE9, 0xDD,
+ 0x0F, 0x91, 0xA4, 0xD1, 0x30, 0x14, 0x1C, 0x48,
+ 0xEC, 0xF4, 0x4F, 0x02, 0x7B, 0x1E, 0x25, 0x3A,
+ 0x7C, 0x6B, 0x42, 0x13, 0xF7, 0xBC, 0xB5, 0x02,
+ 0xA9, 0x20, 0x85, 0x21, 0x01, 0x67, 0xC3, 0xDD,
+ 0x6C, 0x6D, 0xD3, 0xC9, 0x6F, 0x13, 0x75, 0xDD,
+ 0x1D, 0xD7, 0xE7, 0xF3, 0x34, 0x17, 0x37, 0xFF,
+ 0xE6, 0x3B, 0xB5, 0x1F, 0xEE, 0x51, 0x73, 0x6D,
+ 0x9E, 0xB7, 0xE2, 0xE7, 0xA1, 0x65, 0xE4, 0x29,
+ 0x8E, 0xBF, 0x66, 0xCE, 0x5E, 0xD4, 0xBA, 0x0C,
+ 0x18, 0x84, 0xBE, 0xAE, 0x9A, 0x17, 0x0D, 0xAE,
+ 0x55, 0x90, 0x7F, 0x72, 0x73, 0xAB, 0x9F, 0x87,
+ 0xCC, 0x3D, 0xCB, 0xE4, 0x38, 0x66, 0x92, 0xEE,
+ 0x6D, 0xE9, 0x0B, 0x8A, 0xE2, 0x5F, 0x68, 0x9D,
+ 0x06, 0xFD, 0xF3, 0x77, 0x4B, 0x50, 0xCD, 0x0E,
+ 0x2B, 0xE1, 0xD3, 0xB4, 0xF4, 0x02, 0xF5, 0x9B,
+ 0x5F, 0x3E, 0x59, 0xD7, 0x57, 0x9D, 0x87, 0x80,
+ 0x60, 0xEB, 0x70, 0xF5, 0x34, 0x56, 0x46, 0x5B,
+ 0xBA, 0x8F, 0x90, 0xAE, 0x9F, 0x6B, 0x43, 0x8C,
+ 0x51, 0x45, 0xD2, 0x16, 0x4C, 0xBA, 0x86, 0xF0,
+ 0xF4, 0xD5, 0x34, 0x6A, 0x3E, 0x5F, 0xAE, 0xBE,
+ 0x95, 0x40, 0xFE, 0x26, 0x4D, 0x5E, 0x60, 0x4E,
+ 0xD9, 0xEB, 0x47, 0x7D, 0x43, 0x63, 0x5F, 0x4B,
+ 0xB1, 0xCE, 0x7E, 0xA2, 0xF0, 0xC9, 0x30, 0x0C,
+ 0xB7, 0x13, 0x43, 0xC0, 0xF8, 0x02, 0x6C, 0xD2,
+ 0x5F, 0xCE, 0x25, 0xF3, 0xAE, 0xC3, 0x2D, 0xC3,
+ 0x13, 0xAE, 0x49, 0x8B, 0x82, 0x82, 0x44, 0xD0,
+ 0x50, 0xF8, 0x00, 0x5E, 0xAC, 0xDE, 0x4B, 0x88,
+ 0xC2, 0x38, 0x5C, 0xC3, 0x65, 0xF4, 0x25, 0x63,
+ 0x29, 0xC3, 0xB0, 0x7A, 0x45, 0x5D, 0x43, 0x89,
+ 0xCA, 0x5A, 0x12, 0x61, 0xD8, 0x92, 0x65, 0x1B,
+ 0x0F, 0xBF, 0x62, 0xB9, 0xA6, 0xF2, 0xA8, 0xA7,
+ 0x06, 0xAC, 0x02, 0xA4, 0xC6, 0x25, 0xD6, 0xC5,
+ 0xD2, 0xDE, 0x87, 0x88, 0x11, 0xBD, 0x7C, 0x87,
+ 0x91, 0xA9, 0x91, 0x60, 0x1F, 0x0F, 0xF0, 0x24,
+ 0xFE, 0xA0, 0xEB, 0xDC, 0x89, 0x68, 0x84, 0x10,
+ 0xD7, 0x55, 0xC1, 0x3A, 0xCA, 0x1F, 0xFF, 0x77,
+ 0x41, 0xA1, 0xDB, 0x13, 0x31, 0xEF, 0x9C, 0xA3,
+ 0xA2, 0x3A, 0x37, 0x1B, 0xF9, 0x46, 0xF8, 0x51,
+ 0x15, 0xB3, 0x0A, 0x12, 0x64, 0x3F, 0xE5, 0xA8,
+ 0x07, 0xA2, 0x57, 0x87, 0x2A, 0x3F, 0xD2, 0x87,
+ 0xDB, 0xC0, 0x33, 0xAC, 0xF5, 0x28, 0x42, 0xC5,
+ 0xD4, 0x20, 0x27, 0xAE, 0x57, 0xAF, 0x6C, 0x74,
+ 0x8A, 0xA0, 0x90, 0xEE, 0x34, 0xE9, 0x19, 0xB9,
+ 0x0E, 0xDF, 0x4D, 0xDE, 0x77, 0x66, 0xC4, 0xBC,
+ 0x99, 0x59, 0x14, 0xCB, 0x9D, 0xBF, 0xBC, 0x5F,
+ 0xFF, 0x2B, 0xA7, 0xED, 0x29, 0x49, 0x13, 0x6C,
+ 0x2B, 0x71, 0x58, 0xC2, 0xC4, 0x67, 0xCB, 0x6A,
+ 0x18, 0x6F, 0x4E, 0x88, 0xB9, 0x76, 0xC8, 0x6B,
+ 0xDC, 0x6A, 0x05, 0xA5, 0x22, 0x31, 0x15, 0xCE,
+ 0x54, 0xDA, 0xE8, 0x0E, 0xED, 0xDF, 0x46, 0x90,
+ 0x93, 0x9A, 0xE6, 0x2B, 0x45, 0xA4, 0x51, 0x42,
+ 0x29, 0x05, 0xA9, 0xFF, 0x29, 0xBE, 0x4F, 0x6E,
+ 0xE7, 0x52, 0x2C, 0x16, 0x3F, 0x95, 0x94, 0x7B,
+ 0xE6, 0xE1, 0xF7, 0x9F, 0x36, 0x1F, 0xEE, 0x46,
+ 0xA9, 0xE3, 0x71, 0x37, 0x08, 0xF7, 0x63, 0xAF,
+ 0x16, 0xB9, 0x43, 0x86, 0xAC, 0xC7, 0x5D, 0x5B,
+ 0x73, 0x38, 0x08, 0x2C, 0xC6, 0x65, 0x02, 0xC8,
+ 0x70, 0x71, 0x01, 0xA7, 0xD3, 0xA6, 0xB9, 0x74,
+ 0xAA, 0x71, 0x3B, 0x1B, 0xEF, 0x84, 0xA9, 0x77,
+ 0x91, 0x82, 0x7F, 0xB7, 0x3D, 0x2E, 0x8B, 0xD5,
+ 0x4D, 0xAD, 0x29, 0xAC, 0x51, 0x9E, 0xDB, 0xF0,
+ 0x4A, 0x1D, 0x29, 0x82, 0x04, 0x9E, 0x03, 0x8C,
+ 0x74, 0x66, 0x12, 0x75, 0xD0, 0x77, 0x07, 0xCC,
+ 0x7E, 0x8D, 0x19, 0x2F, 0x42, 0xFA, 0xB7, 0xBD,
+ 0x7E, 0x77, 0xEF, 0xF1, 0x35, 0xAF, 0x7C, 0xFF,
+ 0x52, 0x5F, 0xDB, 0x03, 0x99, 0x0D, 0x89, 0x8A,
+ 0x60, 0x61, 0x42, 0xC2, 0xE7, 0x33, 0x58, 0xE7,
+ 0x40, 0x90, 0x46, 0x84, 0xD5, 0x4B, 0x30, 0x88,
+ 0xA7, 0xBB, 0x25, 0xDE, 0x02, 0xFE, 0x57, 0x93,
+ 0xFD, 0xD0, 0x6E, 0xA2, 0xFE, 0x38, 0xA3, 0x5E,
+ 0x6C, 0x35, 0x06, 0xC4, 0xC6, 0x8C, 0x9C, 0x37,
+ 0x3B, 0x6A, 0x1D, 0x91, 0xCB, 0x84, 0x03, 0x47,
+ 0x85, 0xC7, 0x1D, 0x82, 0xEE, 0xB5, 0xF8, 0xEC,
+ 0x29, 0xDE, 0x30, 0x76, 0x32, 0x08, 0x6A, 0x75,
+ 0x43, 0x6A, 0xB8, 0x4C, 0x5D, 0x5A, 0x54, 0xC2,
+ 0xE3, 0x58, 0x43, 0x6E, 0x90, 0xCE, 0x51, 0xEB,
+ 0xE2, 0xFD, 0x68, 0xAE, 0xB3, 0xBA, 0x80, 0xB4,
+ 0x03, 0x84, 0x26, 0x4E, 0xAB, 0x7B, 0x15, 0x2D,
+ 0xDE, 0xB0, 0x7B, 0xF3, 0x4D, 0xFB, 0x6A, 0xFC,
+ 0xE3, 0x47, 0xFC, 0x9C, 0x34, 0xC8, 0xB2, 0x10,
+ 0x46, 0xCC, 0x5F, 0x0A, 0xC6, 0xEB, 0x48, 0x47,
+ 0xFF, 0x73, 0x75, 0xD0, 0x7C, 0xBF, 0x44, 0xCB,
+ 0xAE, 0x8A, 0x30, 0x0B, 0x3F, 0x58, 0x11, 0x51,
+ 0x32, 0xD5, 0x7C, 0xCC, 0xB5, 0xDB, 0xD9, 0x21,
+ 0xD0, 0x6C, 0x29, 0xE2, 0xD0, 0xE1, 0xB2, 0xC8,
+ 0x06, 0x29, 0xC0, 0xC8, 0x5B, 0xB2, 0x3A, 0x66,
+ 0x7A, 0x25, 0x13, 0x57, 0x80, 0x14, 0x5B, 0x31,
+ 0xC4, 0x7B, 0x21, 0xB5, 0x50, 0x9C, 0x76, 0x6B,
+ 0x8C, 0x16, 0xF7, 0x95, 0xA1, 0x79, 0x6B, 0xA8,
+ 0x22, 0x13, 0xD5, 0x32, 0x15, 0x3E, 0xFC, 0x55,
+ 0x80, 0xD2, 0x71, 0xD1, 0x59, 0x15, 0xDA, 0xFC,
+ 0x55, 0x75, 0xE7, 0x7E, 0x15, 0x48, 0xD9, 0x73,
+ 0x0D, 0x6A, 0x31, 0x8A, 0x1C, 0x86, 0x4A, 0x31,
+ 0x57, 0x5B, 0x0A, 0x9D, 0xE4, 0x8C, 0x80, 0x7F,
+ 0x0E, 0x07, 0x36, 0x38, 0x7C, 0xEA, 0x3C, 0xBD,
+ 0xBD, 0xD1, 0xAA, 0xE7, 0xE4, 0x43, 0xB2, 0xFC,
+ 0x26, 0x39, 0x4F, 0xCC, 0xE6, 0xB8, 0xD7, 0x90,
+ 0xC3, 0x5F, 0xAA, 0xEA, 0x78, 0xC6, 0xD7, 0x58,
+ 0x15, 0x7D, 0x27, 0xD0, 0x7E, 0x0F, 0x13, 0x0D,
+ 0x47, 0x49, 0x28, 0x5B, 0xD7, 0xBC, 0x41, 0xB5,
+ 0x66, 0x38, 0x29, 0xB4, 0x71, 0xEC, 0xA2, 0xDE,
+ 0xE3, 0xE0, 0x4C, 0x27, 0xB4, 0x2C, 0xEE, 0xF1,
+ 0x5F, 0x37, 0x81, 0xCE, 0x31, 0x42, 0x87, 0x44,
+ 0x39, 0x7B, 0x35, 0xCF, 0xE8, 0x7D, 0x5E, 0xD7,
+ 0xA3, 0x3D, 0xB9, 0x92, 0x95, 0x8C, 0x25, 0xC6,
+ 0xC9, 0xBC, 0x46, 0x4E, 0x03, 0x70, 0x29, 0x4B,
+ 0x79, 0xB8, 0xEF, 0x54, 0x98, 0x8B, 0x9A, 0x45,
+ 0x09, 0x8C, 0x43, 0xD1, 0x9B, 0x29, 0xEA, 0xDE,
+ 0xF2, 0x25, 0x10, 0xA6, 0xF9, 0x2C, 0xC8, 0x90,
+ 0x49, 0xA5, 0x3C, 0xDC, 0xCE, 0xA3, 0x98, 0xF1,
+ 0x4C, 0xC6, 0x3E, 0xE0, 0x21, 0x58, 0x1A, 0x39,
+ 0xDA, 0x50, 0x2A, 0x6A, 0x18, 0x49, 0xC9, 0xA1,
+ 0x9D, 0xF3, 0xF4, 0xFB, 0xDD, 0x6F, 0x8D, 0xF4,
+ 0xFE, 0x61, 0xA0, 0xC6, 0xF5, 0x58, 0x89, 0xAE,
+ 0xEC, 0xC6, 0xE0, 0x88, 0x4A, 0x07, 0x6F, 0x11,
+ 0x72, 0x5A, 0x6D, 0x3E, 0x08, 0x64, 0x3E, 0x23,
+ 0x7D, 0x9A, 0x74, 0xB3, 0xC3, 0xDA, 0xA7, 0x29,
+ 0x0E, 0xC1, 0x37, 0xB0, 0x04, 0x42, 0x47, 0x01,
+ 0x80, 0x4B, 0xC0, 0x35, 0x49, 0x19, 0xB1, 0xDB,
+ 0x51, 0x51, 0x09, 0x90, 0x08, 0xB2, 0xC4, 0x5C,
+ 0xA0, 0xD6, 0x6E, 0x09, 0xFF, 0xA0, 0xC6, 0x4F,
+ 0x32, 0x95, 0x84, 0xBE, 0xAA, 0x4A, 0x0A, 0x36,
+ 0x83, 0xD3, 0x0C, 0xB7, 0xE0, 0xD0, 0x24, 0xE9,
+ 0x0A, 0x2F, 0x0F, 0xB4, 0x3A, 0x2F, 0xCE, 0x34,
+ 0xE5, 0xF0, 0x67, 0x07, 0x72, 0x52, 0x24, 0xE9,
+ 0x9F, 0xBF, 0xDE, 0x40, 0x95, 0xAD, 0x74, 0x41,
+ 0x4A, 0x00, 0x07, 0xEB, 0x14, 0xDD, 0xC4, 0xA2,
+ 0x37, 0x72, 0x83, 0xE7, 0xD6, 0x20, 0xF5, 0xC6,
+ 0x86, 0x97, 0xC5, 0x64, 0x58, 0x1B, 0x57, 0x5F,
+ 0x9D, 0x06, 0x1D, 0xB5, 0x2D, 0x26, 0xBA, 0xE4,
+ 0x49, 0x71, 0x08, 0xBD, 0x21, 0xC4, 0xA9, 0xF3,
+ 0x26, 0x7A, 0x28, 0xD6, 0x81, 0x20, 0xCD, 0x9C,
+ 0xED, 0xB8, 0xDC, 0x45, 0x33, 0xC0, 0x54, 0x5C,
+ 0x3A, 0xF5, 0xC5, 0x1B, 0x19, 0x10, 0xCA, 0xDF,
+ 0x99, 0xD4, 0xEE, 0x25, 0xD6, 0x71, 0x24, 0x4D,
+ 0xFF, 0x14, 0x85, 0x58, 0x07, 0xC8, 0x3F, 0xD6,
+ 0x55, 0x1C, 0xBF, 0xF8, 0x94, 0x31, 0x04, 0x83,
+ 0xEC, 0xD1, 0xF5, 0x20, 0x72, 0xC3, 0xE9, 0xB1,
+ 0x8A, 0x00, 0x0B, 0x94, 0x53, 0x4C, 0x01, 0x8C,
+ 0xDE, 0x80, 0x59, 0x66, 0x72, 0x2D, 0xC5, 0x78,
+ 0xAC, 0x4A, 0xAD, 0x4F, 0x14, 0xC1, 0x78, 0x87,
+ 0x5F, 0xDF, 0xF2, 0x95, 0xCF, 0x8F, 0x3F, 0xA6,
+ 0xCC, 0xAA, 0x1E, 0xD8, 0xA8, 0x37, 0xAD, 0x5B,
+ 0xA5, 0x6D, 0xBC, 0x81, 0xAC, 0xCD, 0xFD, 0x56,
+ 0x9E, 0x8B, 0xB9, 0xEC, 0x7E, 0x82, 0x4E, 0x55,
+ 0x05, 0xFD, 0x9F, 0xDC, 0x0A, 0xF7, 0xBD, 0x3F,
+ 0xEA, 0x42, 0x9D, 0x7E, 0xAB, 0x09, 0xDF, 0x3D,
+ 0xA3, 0x8C, 0x63, 0x78, 0x81, 0x8F, 0x46, 0x9F,
+ 0x12, 0xB8, 0x8A, 0xA5, 0xE4, 0xE9, 0x73, 0x95,
+ 0x37, 0xD3, 0x39, 0xFE, 0x9B, 0x69, 0x58, 0x05,
+ 0x5B, 0x30, 0x09, 0x31, 0x8B, 0xD9, 0xDB, 0xAE,
+ 0x96, 0x93, 0x96, 0xDF, 0xC0, 0x70, 0x1D, 0xFF,
+ 0x5D, 0x60, 0x72, 0x5A, 0xDD, 0xC2, 0x3D, 0xCF,
+ 0x13, 0xA0, 0x36, 0x38, 0xF4, 0x3A, 0x03, 0x15,
+ 0x48, 0xF0, 0xB1, 0x4A, 0x12, 0x72, 0xBA, 0x6B,
+ 0xE2, 0xD6, 0x4E, 0x57, 0x22, 0x87, 0x2F, 0xE7,
+ 0x40, 0x1B, 0x22, 0xF3, 0x4A, 0xE7, 0xB1, 0xA4,
+ 0x28, 0xC7, 0xBB, 0x17, 0x5E, 0x0C, 0x03, 0xE3,
+ 0xF4, 0x67, 0x25, 0xB2, 0xBB, 0xE6, 0x4E, 0xCD,
+ 0x8A, 0x39, 0xC6, 0x05, 0xB7, 0x14, 0x10, 0xF4,
+ 0x36, 0xE9, 0x8A, 0xC3, 0x41, 0xD3, 0x30, 0xAD,
+ 0x79, 0x56, 0xE2, 0xC3, 0x55, 0xFC, 0x05, 0x3B,
+ 0xD0, 0x83, 0x31, 0xE7, 0xA1, 0x85, 0xBE, 0x6C,
+ 0xAF, 0x9D, 0xA6, 0x26, 0xC4, 0x7A, 0x4B, 0xD8,
+ 0x26, 0xA7, 0x59, 0xBB, 0x8D, 0x6E, 0x15, 0x96,
+ 0x84, 0x08, 0x0B, 0xDF, 0x29, 0x2D, 0x74, 0xD9,
+ 0xAD, 0xFE, 0xC1, 0x3A, 0x84, 0x4E, 0xCE, 0x8F,
+ 0x00, 0x9A, 0x50, 0xB8, 0x6F, 0x02, 0xE6, 0xEC,
+ 0xA9, 0xE6, 0x1F, 0x71, 0x6A, 0x48, 0x98, 0x61,
+ 0x3B, 0xC1, 0x37, 0x5A, 0x2E, 0xF3, 0xE0, 0xD6,
+ 0x11, 0x0C, 0x15, 0x39, 0x7B, 0xFB, 0x37, 0xAF,
+ 0x7A, 0xCA, 0xD6, 0x10, 0xDE, 0x95, 0x6A, 0xC6,
+ 0x19, 0xEC, 0x21, 0x51, 0xCE, 0xAE, 0x8F, 0x5A,
+ 0xEE, 0xEF, 0xB2, 0x9C, 0xF7, 0x4F, 0x5A, 0xB4,
+ 0x90, 0xED, 0xB0, 0x64, 0x59, 0x95, 0xC5, 0x32,
+ 0xC1, 0x85, 0xD7, 0x7E, 0x6C, 0xC6, 0x76, 0x11,
+ 0x4B, 0xB5, 0x17, 0x1D, 0xEE, 0x15, 0xFF, 0xD3,
+ 0xC7, 0xBB, 0xAA, 0x3C, 0x5D, 0x4D, 0x03, 0x82,
+ 0xC0, 0xC7, 0xEA, 0xD0, 0xD8, 0x1B, 0xFF, 0x3C,
+ 0x1D, 0x5A, 0x3F, 0xBD, 0x81, 0x66, 0x62, 0x6E,
+ 0xB3, 0xF5, 0x5F, 0xF0, 0x43, 0x90, 0x01, 0x71,
+ 0xDD, 0xB6, 0x0F, 0x60, 0xCE, 0xFB, 0x17, 0x21,
+ 0x5A, 0x7F, 0x0C, 0x69, 0x82, 0x9D, 0x4C, 0xF3,
+ 0x30, 0x1A, 0xF7, 0x1E, 0x85, 0x1A, 0x89, 0x84,
+ 0xF3, 0x4E, 0x8F, 0x15, 0x60, 0x43, 0x6D, 0x3A,
+ 0x5B, 0x07, 0xC0, 0x78, 0x6A, 0x02, 0xB4, 0x98,
+ 0x3D, 0xAB, 0xAC, 0x25, 0x55, 0xC8, 0x49, 0x7E,
+ 0xC9, 0x04, 0x73, 0xAF, 0x0D, 0x17, 0x1A, 0xA6,
+ 0xBE, 0xD8, 0x11, 0x69, 0x4F, 0x17, 0x39, 0xF3,
+ 0x57, 0x53, 0x1F, 0xD5, 0x1B, 0x15, 0x89, 0x53,
+ 0x54, 0xB1, 0x9F, 0xFD, 0x52, 0x92, 0xA8, 0x98,
+ 0xD4, 0x7B, 0xEE, 0x43, 0xC6, 0x31, 0xDE, 0xEC,
+ 0xDC, 0xE5, 0x1D, 0x90, 0x37, 0x63, 0xA1, 0xF8,
+ 0x42, 0x52, 0x8A, 0x73, 0x8B, 0x3D, 0x42, 0x85,
+ 0x64, 0x5C, 0xA9, 0xCC, 0xA1, 0xCD, 0xEB, 0x9E,
+ 0x2A, 0xF5, 0x21, 0x9E, 0x81, 0xC6, 0x2D, 0xD7,
+ 0x22, 0xC2, 0xEE, 0x47, 0x7E, 0xDA, 0x60, 0x26,
+ 0xFA, 0xE9, 0xB1, 0x81, 0x01, 0x77, 0xD3, 0x42,
+ 0xDD, 0x03, 0x65, 0x74, 0x2B, 0x85, 0x49, 0x9A,
+ 0xA5, 0x27, 0x76, 0x49, 0xF1, 0x72, 0x5E, 0xDF,
+ 0xB7, 0xAD, 0x19, 0x1A, 0x58, 0x89, 0x56, 0x5C,
+ 0x61, 0x03, 0xC7, 0x42, 0x84, 0xF2, 0x9E, 0x9E,
+ 0x09, 0x2A, 0x2D, 0x3B, 0xAE, 0x6A, 0x8E, 0x9A,
+ 0x87, 0x70, 0x16, 0x0A, 0xF1, 0xA9, 0xEA, 0xD2,
+ 0xF3, 0x2A, 0xA3, 0xB0, 0x32, 0xD3, 0x28, 0x92,
+ 0x19, 0x02, 0x77, 0xB2, 0x26, 0x1B, 0xC0, 0x59,
+ 0x1A, 0x67, 0x4B, 0x51, 0x04, 0x83, 0x25, 0xF6,
+ 0x4E, 0xA3, 0x92, 0x55, 0xCD, 0x74, 0xF5, 0x23,
+ 0x98, 0xB7, 0xB9, 0x75, 0xA6, 0xDC, 0x4E, 0xD2,
+ 0x93, 0x39, 0x94, 0xA3, 0xCD, 0xF8, 0xD5, 0xA4,
+ 0xA9, 0x3C, 0x97, 0xED, 0x18, 0x54, 0xBB, 0x59,
+ 0x7E, 0x75, 0xE3, 0xEE, 0x5A, 0x36, 0x53, 0x3D,
+ 0x02, 0xFB, 0x47, 0xE6, 0x10, 0x1D, 0xDF, 0x21,
+ 0xDA, 0x44, 0x66, 0x05, 0x81, 0xC4, 0xE9, 0x3B,
+ 0xD5, 0x56, 0xC5, 0xAD, 0x50, 0x7A, 0xE1, 0xAA,
+ 0x91, 0x7F, 0x7C, 0x4E, 0x85, 0xCC, 0xDC, 0x4A,
+ 0xBD, 0x72, 0x1F, 0x3F, 0x21, 0x93, 0x98, 0x21,
+ 0xB8, 0xCC, 0x21, 0x00, 0x24, 0x13, 0xAB, 0x63,
+ 0xC1, 0x31, 0xCA, 0x1C, 0x67, 0x0C, 0x0E, 0x5A,
+ 0x62, 0xC3, 0x1E, 0x6D, 0x63, 0x9B, 0xDA, 0x7F,
+ 0x92, 0xC4, 0x64, 0x8B, 0xC7, 0x80, 0x04, 0x05,
+ 0x36, 0x06, 0xCE, 0xF7, 0x37, 0x3A, 0xA8, 0xDF,
+ 0x2C, 0x61, 0x46, 0x48, 0x2A, 0x5C, 0xDC, 0xBB,
+ 0x34, 0x16, 0xDA, 0x59, 0x0D, 0x5B, 0xE5, 0x08,
+ 0x05, 0xAF, 0x59, 0x86, 0x34, 0x56, 0x88, 0x4F,
+ 0xAF, 0xA0, 0x28, 0xC4, 0xF2, 0x97, 0xC6, 0xF4,
+ 0x7C, 0xFD, 0xA4, 0x20, 0xE4, 0x20, 0x2C, 0xDA,
+ 0x98, 0xAD, 0x4A, 0x96, 0x9F, 0x62, 0xBB, 0x1C,
+ 0xB6, 0x28, 0xB0, 0xD0, 0xEA, 0x3B, 0xB8, 0x3C,
+ 0x8D, 0x4E, 0x97, 0xEA, 0x9E, 0x63, 0xAA, 0xAA,
+ 0x8D, 0x22, 0x61, 0x29, 0x1A, 0x82, 0x29, 0x86,
+ 0x87, 0x62, 0xA4, 0xE4, 0xC0, 0x03, 0x93, 0x55,
+ 0x7D, 0x40, 0xE9, 0x43, 0x14, 0x9C, 0xE2, 0xDA,
+ 0x0A, 0x00, 0x37, 0xEA, 0x80, 0xB8, 0x31, 0x66,
+ 0xAE, 0xD7, 0xF8, 0xB4, 0x24, 0xC7, 0x78, 0xDB,
+ 0x23, 0x66, 0x61, 0xA5, 0x47, 0x10, 0x4A, 0xE8,
+ 0x79, 0xB6, 0xBD, 0x6A, 0xB6, 0x22, 0x6C, 0x7B,
+ 0x19, 0xF3, 0x80, 0x35, 0x9A, 0x15, 0x36, 0x89,
+ 0x2A, 0x2C, 0xED, 0x26, 0xF1, 0x96, 0xFF, 0x30,
+ 0x4D, 0x03, 0x82, 0x2C, 0x69, 0x31, 0x40, 0x40,
+ 0x0C, 0xD1, 0x40, 0x3E, 0xE0, 0xB5, 0x37, 0xA8,
+ 0x6D, 0x68, 0x68, 0x8F, 0x9E, 0xC5, 0x12, 0x05,
+ 0xF5, 0x36, 0x85, 0x5F, 0x8A, 0x52, 0x7B, 0x60,
+ 0xA2, 0xC2, 0x39, 0xAF, 0x87, 0xC8, 0xC4, 0xF2,
+ 0x53, 0x1D, 0xF0, 0x84, 0x32, 0x8A, 0x04, 0x42,
+ 0xBF, 0xCA, 0x1A, 0x08, 0x89, 0x71, 0xCA, 0xC8,
+ 0x11, 0xA6, 0xDC, 0xED, 0x31, 0xFC, 0x41, 0xFC,
+ 0x84, 0x75, 0xA5, 0x98, 0x67, 0x9E, 0xE7, 0x69,
+ 0x9F, 0x3C, 0x42, 0x94, 0x69, 0xA5, 0x1D, 0xC2,
+ 0x55, 0x9D, 0x2E, 0xC9, 0x13, 0x3C, 0x68, 0x94,
+ 0xA9, 0x7E, 0x6A, 0xFC, 0x11, 0xE3, 0x97, 0x0D,
+ 0x59, 0x8F, 0x8D, 0x05, 0x01, 0x4A, 0x74, 0xB6,
+ 0x96, 0x6D, 0x1E, 0x7F, 0x63, 0xA5, 0x54, 0x39,
+ 0x5A, 0x6C, 0xB9, 0x9B, 0xC3, 0x79, 0xCB, 0xD3,
+ 0xC9, 0xE9, 0x37, 0x9E, 0xDC, 0x3B, 0xE2, 0xD2,
+ 0xE6, 0x07, 0x77, 0xFA, 0xD5, 0xCF, 0xD6, 0x42,
+ 0x45, 0x46, 0xFE, 0x06, 0x28, 0x0E, 0x4D, 0xA6,
+ 0x72, 0xE6, 0x6F, 0x39, 0xCD, 0xB9, 0xBF, 0x43,
+ 0xF7, 0xD9, 0x99, 0x9B, 0x46, 0xF1, 0x80, 0xBA,
+ 0x07, 0x4A, 0x0A, 0xC3, 0x9F, 0xD3, 0xCB, 0xA5,
+ 0x85, 0xF7, 0xEE, 0x9F, 0x6D, 0xC0, 0x52, 0xD7,
+ 0x17, 0x27, 0x81, 0x92, 0x98, 0xA0, 0xBB, 0xC9,
+ 0xD7, 0xE4, 0xF0, 0xF8, 0x25, 0x2C, 0x38, 0x3B,
+ 0x44, 0x4E, 0x55, 0x57, 0x5F, 0x86, 0x9A, 0xA1,
+ 0xB2, 0xBB, 0xBF, 0xE4, 0xF4, 0x04, 0x0D, 0x16,
+ 0x34, 0x48, 0x5A, 0x68, 0x72, 0x88, 0x8B, 0x97,
+ 0xA5, 0xAC, 0xBC, 0xC4, 0xC9, 0xCC, 0xF3, 0x03,
+ 0x12, 0x14, 0x2C, 0x3B, 0x3E, 0x3F, 0x4D, 0x52,
+ 0x79, 0x7A, 0x91, 0x96, 0xA8, 0xB3, 0xBB, 0xCC,
+ 0xCF, 0xD7, 0xDF, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x0C, 0x1D, 0x2F, 0x43,
+};
+
+uint8_t mldsa_44_sign_dtrm[] = {
+ 0xC7, 0x9B, 0x98, 0x33, 0xEA, 0xCC, 0xCC, 0x47,
+ 0x51, 0x8F, 0xE1, 0xC0, 0xDA, 0x82, 0x9B, 0xAA,
+ 0x74, 0x0A, 0x14, 0x25, 0x73, 0x53, 0x27, 0x7C,
+ 0x4C, 0x4A, 0xD6, 0xB7, 0x7C, 0x71, 0x25, 0xD8,
+ 0xC5, 0xBA, 0xA8, 0x18, 0x59, 0xC7, 0x64, 0x2E,
+ 0x9A, 0x65, 0xE9, 0x63, 0x83, 0x3D, 0xB8, 0x2B,
+ 0x42, 0x0A, 0x6D, 0x7D, 0xCE, 0x6A, 0xA3, 0x11,
+ 0xD6, 0xB6, 0x54, 0x82, 0x36, 0x38, 0xF1, 0x84,
+ 0x01, 0x3E, 0x26, 0x66, 0xC7, 0x38, 0x6F, 0x41,
+ 0x4C, 0xC2, 0x42, 0xB5, 0xBF, 0xCE, 0x7B, 0x8B,
+ 0xD5, 0x4F, 0x9B, 0x9F, 0x0E, 0xFE, 0x52, 0xAF,
+ 0x6D, 0x39, 0xD8, 0x0A, 0x27, 0x4D, 0xDB, 0x9F,
+ 0x04, 0x25, 0x8C, 0xF5, 0x89, 0x5B, 0xEB, 0xDC,
+ 0x70, 0xBF, 0x23, 0x1D, 0x14, 0x73, 0x9E, 0xFC,
+ 0x41, 0xA0, 0xA2, 0xA5, 0xAA, 0xAB, 0x8E, 0xCD,
+ 0x5E, 0x29, 0xC5, 0x3E, 0xEC, 0x0A, 0xD9, 0x06,
+ 0xEB, 0x8C, 0x18, 0x41, 0x80, 0x91, 0x5D, 0xC9,
+ 0x28, 0xCE, 0xDB, 0x1C, 0x0E, 0x8F, 0xE3, 0xC0,
+ 0xE5, 0xB1, 0x0A, 0x38, 0x06, 0x43, 0x36, 0x0D,
+ 0xF8, 0xE6, 0x2E, 0xCB, 0xEB, 0x51, 0xC8, 0x82,
+ 0xE0, 0x3B, 0xCB, 0xCE, 0x62, 0xC8, 0xE0, 0x8A,
+ 0x2C, 0xE4, 0xC7, 0xD7, 0xC1, 0x8A, 0x64, 0x0D,
+ 0xE8, 0x5A, 0x52, 0x83, 0x58, 0x93, 0x2A, 0x25,
+ 0x92, 0x65, 0x19, 0x4F, 0x0F, 0x44, 0x1C, 0xCE,
+ 0x56, 0x91, 0xB4, 0xC9, 0x8D, 0x9E, 0x6A, 0xD5,
+ 0xCA, 0xE3, 0xA0, 0xE3, 0x46, 0x45, 0x0D, 0x59,
+ 0x14, 0x90, 0x72, 0x67, 0xC5, 0x95, 0xAC, 0x2A,
+ 0xC4, 0x70, 0xD2, 0xCA, 0x40, 0x1D, 0x62, 0x0E,
+ 0x2A, 0xA4, 0x3E, 0xAC, 0x45, 0x00, 0x96, 0xA4,
+ 0xF6, 0xAC, 0xEE, 0x01, 0x19, 0xEB, 0xCC, 0x8C,
+ 0x62, 0xE3, 0x26, 0x59, 0xB7, 0xB9, 0xBF, 0xBB,
+ 0x8D, 0x72, 0xAD, 0x1A, 0xA2, 0x8D, 0xB1, 0x1B,
+ 0xC1, 0xC1, 0xD8, 0xFF, 0x03, 0xAF, 0x74, 0xDF,
+ 0xC6, 0x3C, 0xCE, 0xBD, 0xA8, 0x12, 0xFA, 0xAC,
+ 0x83, 0x8C, 0x7E, 0x26, 0x8E, 0x0D, 0x03, 0xF3,
+ 0x02, 0x17, 0x23, 0x13, 0x25, 0x01, 0x1F, 0xB9,
+ 0x3B, 0x1C, 0x3E, 0xA7, 0xD9, 0xE4, 0xE6, 0xCA,
+ 0xD6, 0x0C, 0x91, 0xAC, 0xEE, 0xD4, 0x42, 0xA7,
+ 0xB1, 0x62, 0x70, 0xC8, 0xC9, 0x2F, 0x0A, 0xE0,
+ 0xA8, 0xC7, 0x22, 0xE5, 0x2C, 0x06, 0x1C, 0x09,
+ 0x4B, 0x45, 0x25, 0x02, 0x07, 0x8A, 0x86, 0xCC,
+ 0xF8, 0xF9, 0x36, 0x11, 0x67, 0x75, 0xEE, 0xD2,
+ 0xD4, 0x67, 0x09, 0xFD, 0x37, 0xB4, 0x3F, 0x7B,
+ 0x4A, 0x2B, 0x4E, 0x05, 0x37, 0x5E, 0xFD, 0x97,
+ 0x79, 0x95, 0x13, 0x95, 0x0D, 0x75, 0x15, 0x94,
+ 0x27, 0x36, 0xB3, 0x97, 0x03, 0xBE, 0x38, 0x14,
+ 0xCB, 0x4D, 0x16, 0x49, 0xFF, 0x36, 0xFA, 0xED,
+ 0x4D, 0x97, 0x45, 0xE4, 0xFB, 0xF9, 0x61, 0x7D,
+ 0x1F, 0xCE, 0xC3, 0xBF, 0xB3, 0xEA, 0x3C, 0xBC,
+ 0x13, 0x05, 0xBB, 0xAB, 0x44, 0x84, 0x80, 0x0F,
+ 0xB9, 0x1F, 0x6E, 0x88, 0x27, 0x4A, 0xFA, 0x17,
+ 0x83, 0x76, 0x5F, 0xC5, 0xE8, 0x85, 0xBC, 0xFF,
+ 0xD8, 0x48, 0x4A, 0x9F, 0xEB, 0xC3, 0xF8, 0x41,
+ 0x86, 0x4D, 0xC2, 0x75, 0x7B, 0x74, 0x71, 0x68,
+ 0x9C, 0xF8, 0x05, 0x8B, 0xB7, 0xFA, 0xE0, 0xA7,
+ 0xFA, 0xEC, 0x53, 0x91, 0xD0, 0xB4, 0x44, 0x7A,
+ 0x42, 0xAD, 0xD2, 0x17, 0x86, 0x8F, 0x96, 0xF0,
+ 0xF1, 0xD6, 0xC0, 0x26, 0x34, 0x83, 0x2E, 0x62,
+ 0x0D, 0x3F, 0x13, 0x5A, 0x8C, 0xBC, 0x6D, 0x8D,
+ 0x39, 0x05, 0x91, 0x10, 0x32, 0xBE, 0x17, 0x8A,
+ 0xC7, 0x23, 0xBE, 0x3F, 0xC5, 0x8A, 0xE0, 0x81,
+ 0xFC, 0x0D, 0x4C, 0x62, 0xE8, 0x67, 0xBA, 0x8B,
+ 0xBD, 0xA5, 0xCB, 0xE9, 0x34, 0x3D, 0x8D, 0x84,
+ 0x9D, 0x65, 0xBF, 0x9E, 0xC4, 0xB3, 0x16, 0x2E,
+ 0x12, 0xB9, 0x1F, 0x82, 0xA1, 0x46, 0x26, 0xD5,
+ 0xAB, 0x7D, 0x80, 0x4D, 0x9C, 0xD3, 0xEB, 0x77,
+ 0xFB, 0x67, 0xEA, 0xEE, 0xA8, 0x7B, 0x17, 0x02,
+ 0x2E, 0x6E, 0xF5, 0x11, 0x44, 0xFC, 0x31, 0x01,
+ 0xCC, 0x03, 0x94, 0x0A, 0xCF, 0x8D, 0x12, 0xCD,
+ 0x87, 0x66, 0x38, 0x62, 0x3C, 0xBB, 0x98, 0x2E,
+ 0xC5, 0x58, 0xD0, 0xF3, 0x06, 0xBD, 0x49, 0x7E,
+ 0x18, 0x7D, 0x64, 0xEF, 0x37, 0x71, 0x87, 0xD4,
+ 0xE6, 0x2C, 0xCC, 0x32, 0x59, 0x65, 0x05, 0x45,
+ 0xDE, 0xF3, 0x17, 0x62, 0x2C, 0x25, 0x62, 0x9C,
+ 0x67, 0xBD, 0x74, 0xF1, 0x54, 0x0E, 0x0B, 0x4A,
+ 0x01, 0x9F, 0x5E, 0x3B, 0x08, 0x57, 0xFA, 0x09,
+ 0x99, 0x11, 0x2C, 0x3E, 0x1D, 0x2D, 0xF7, 0x91,
+ 0x12, 0x14, 0xFF, 0x95, 0x6F, 0xCA, 0xBD, 0x25,
+ 0xEC, 0x3C, 0x98, 0xA7, 0xAB, 0x98, 0x0E, 0xFB,
+ 0x01, 0xE9, 0x5E, 0x58, 0x5A, 0x49, 0x3F, 0xB2,
+ 0xC4, 0xBC, 0xF3, 0xEC, 0xCA, 0x4C, 0x54, 0x27,
+ 0x66, 0x10, 0x60, 0x76, 0x5C, 0xFB, 0x9E, 0xAC,
+ 0xD2, 0x09, 0x75, 0xB3, 0x2B, 0x65, 0xE7, 0x93,
+ 0xAD, 0x2B, 0x08, 0x6F, 0x87, 0x73, 0x45, 0x2A,
+ 0x1B, 0xB1, 0x75, 0xE7, 0x02, 0x9B, 0xCB, 0x56,
+ 0x87, 0x6F, 0x66, 0x44, 0xED, 0xDC, 0x03, 0x92,
+ 0xD4, 0xF1, 0x07, 0x40, 0xD6, 0x86, 0xD5, 0x98,
+ 0x3D, 0x9B, 0x03, 0xD9, 0x0A, 0x58, 0x59, 0x54,
+ 0x06, 0x10, 0xFD, 0x15, 0xFB, 0xE0, 0x5B, 0x81,
+ 0x8D, 0x0C, 0xC0, 0xA6, 0x15, 0x94, 0x72, 0x5E,
+ 0xC8, 0x4A, 0x73, 0x59, 0x63, 0xF6, 0xCD, 0x8D,
+ 0xA4, 0xC2, 0xAF, 0x2E, 0xAE, 0xC1, 0x93, 0x89,
+ 0x08, 0x81, 0x10, 0xEC, 0x44, 0x81, 0x33, 0x96,
+ 0x88, 0xDC, 0x19, 0xDF, 0xD4, 0x2E, 0x5D, 0x52,
+ 0xD7, 0x6A, 0x02, 0x64, 0x11, 0x6B, 0x24, 0xCF,
+ 0x51, 0xD4, 0xC1, 0x85, 0x7D, 0xA3, 0x24, 0x6A,
+ 0x05, 0x13, 0x55, 0xC3, 0x64, 0x5A, 0xDF, 0x9A,
+ 0x23, 0x73, 0xEA, 0xD9, 0x48, 0x98, 0xBE, 0x1C,
+ 0xBD, 0x3A, 0x5D, 0x37, 0xDC, 0x18, 0xA8, 0x39,
+ 0x71, 0x67, 0xC4, 0x12, 0xF1, 0xE3, 0x53, 0x73,
+ 0xF1, 0xD8, 0xC4, 0x8B, 0xB3, 0x64, 0x03, 0xDD,
+ 0x6E, 0xFB, 0x4A, 0xEF, 0xA4, 0x37, 0xA8, 0xA5,
+ 0x2E, 0x86, 0x40, 0x73, 0x16, 0xBA, 0x3A, 0xF5,
+ 0x5E, 0x47, 0xD8, 0x13, 0x58, 0xB7, 0x73, 0xA2,
+ 0xDA, 0x00, 0x9F, 0x90, 0xE4, 0xF8, 0x0B, 0xB3,
+ 0x26, 0x0A, 0xD8, 0x46, 0x8A, 0x09, 0x55, 0xCB,
+ 0xCB, 0xA4, 0x89, 0x84, 0xF8, 0x35, 0xF0, 0xDF,
+ 0x45, 0x5B, 0x6A, 0xAF, 0xA8, 0x5F, 0xE4, 0xC2,
+ 0x03, 0x9B, 0x5B, 0x11, 0x21, 0xA7, 0xDE, 0xC5,
+ 0x71, 0x3D, 0x00, 0xF8, 0x6F, 0x13, 0x75, 0x4E,
+ 0x7B, 0x6B, 0x32, 0x95, 0xD3, 0x77, 0x82, 0xA2,
+ 0x70, 0x6A, 0x80, 0xC3, 0x0D, 0x53, 0xC4, 0x1C,
+ 0xF0, 0x1C, 0x5A, 0x54, 0xA4, 0xB2, 0x7A, 0x90,
+ 0x6E, 0x48, 0x39, 0x19, 0x4A, 0x70, 0x8E, 0xEB,
+ 0xF2, 0x94, 0x07, 0x45, 0xC0, 0x4B, 0xC1, 0x80,
+ 0xED, 0x87, 0x39, 0x6B, 0x5F, 0x7E, 0x63, 0x95,
+ 0x38, 0xFB, 0xF4, 0x6E, 0x3A, 0x6A, 0x78, 0x59,
+ 0x1E, 0x51, 0x45, 0x02, 0x14, 0xAC, 0xA1, 0xD1,
+ 0xC8, 0x44, 0x03, 0xB6, 0x3D, 0xA4, 0xB3, 0xC1,
+ 0xE0, 0x1C, 0x39, 0xA1, 0x87, 0x21, 0x7C, 0xBA,
+ 0x9A, 0x92, 0xA3, 0xAF, 0x37, 0xE8, 0x27, 0xA1,
+ 0xBE, 0x2D, 0x83, 0x0A, 0x46, 0xE7, 0x68, 0x44,
+ 0x9D, 0xB1, 0x57, 0x33, 0x9B, 0x57, 0xE3, 0x0C,
+ 0xE7, 0xDF, 0x7F, 0x15, 0xC0, 0xB5, 0x3A, 0x08,
+ 0x6C, 0xEF, 0xB3, 0xD3, 0xA2, 0x3C, 0x84, 0xC8,
+ 0x77, 0x46, 0xB8, 0xD0, 0x1E, 0x6D, 0x7A, 0x0B,
+ 0xD7, 0x25, 0x77, 0x74, 0xF6, 0xD8, 0xF4, 0x75,
+ 0x54, 0x6A, 0x91, 0x48, 0x98, 0xAC, 0x52, 0xA6,
+ 0x6C, 0xBC, 0x05, 0xA4, 0x92, 0x8F, 0x52, 0xAA,
+ 0x97, 0xCC, 0x23, 0xD8, 0x69, 0x01, 0xB0, 0xD2,
+ 0x1F, 0x22, 0xE8, 0xD1, 0xF7, 0x89, 0x8E, 0x40,
+ 0xF1, 0xD8, 0xCE, 0xB5, 0x26, 0x82, 0xBD, 0xD7,
+ 0x7C, 0x88, 0xB2, 0x6F, 0x16, 0x07, 0x67, 0x51,
+ 0x45, 0x6C, 0x90, 0x63, 0xBA, 0x58, 0xB6, 0x71,
+ 0xB8, 0x54, 0xA8, 0x4C, 0x54, 0xB3, 0xD7, 0x2B,
+ 0x9F, 0xEB, 0x1C, 0xC6, 0x9F, 0xFA, 0xD6, 0xD3,
+ 0xED, 0xA0, 0x99, 0x18, 0x99, 0x23, 0xB0, 0x1A,
+ 0xAA, 0x3A, 0x42, 0xCB, 0x4A, 0xFC, 0x54, 0x30,
+ 0x86, 0xD0, 0xFF, 0x44, 0x62, 0xAB, 0x94, 0xBD,
+ 0xA9, 0x95, 0x27, 0x22, 0xD6, 0x57, 0xE7, 0x26,
+ 0x94, 0x18, 0x0C, 0xDE, 0x10, 0x44, 0xE4, 0x1A,
+ 0x55, 0x15, 0x07, 0xDC, 0x33, 0xCF, 0x4D, 0x27,
+ 0x9D, 0xDD, 0x19, 0xF7, 0x81, 0xD5, 0x2A, 0xFA,
+ 0x52, 0xC5, 0xA3, 0xF9, 0x6A, 0x78, 0xE8, 0x95,
+ 0xE5, 0x00, 0x98, 0x46, 0xD0, 0xA5, 0x55, 0x23,
+ 0x0D, 0xED, 0xA3, 0x84, 0x87, 0xAF, 0xB5, 0xBC,
+ 0x10, 0x8B, 0x1F, 0x0A, 0x5E, 0x08, 0x5B, 0xC8,
+ 0x78, 0xBD, 0xBC, 0xB5, 0x29, 0x56, 0x3C, 0xFB,
+ 0x95, 0xC7, 0xFD, 0xDD, 0xB6, 0x2A, 0x66, 0x5A,
+ 0x2B, 0x86, 0xA2, 0x74, 0x33, 0x67, 0xA3, 0x48,
+ 0xE7, 0xFA, 0xCB, 0x08, 0x6B, 0x9D, 0x85, 0x8E,
+ 0x38, 0x14, 0xCB, 0xC2, 0xDA, 0xA7, 0x06, 0x3A,
+ 0x23, 0xE6, 0x80, 0x7D, 0x01, 0xA2, 0xA4, 0x1B,
+ 0x3C, 0xCF, 0x9C, 0xE8, 0x62, 0x4D, 0xF5, 0x81,
+ 0x8C, 0xB9, 0x11, 0x8A, 0x12, 0x07, 0x93, 0xDD,
+ 0xB9, 0x40, 0x32, 0xB4, 0x60, 0x6A, 0xC8, 0x76,
+ 0x95, 0x54, 0x0A, 0x48, 0xE2, 0xFF, 0x8A, 0x3C,
+ 0xA2, 0x17, 0x4A, 0xCF, 0xDF, 0x6A, 0xAF, 0xBE,
+ 0x7A, 0x62, 0x7A, 0x3A, 0x60, 0x0B, 0x64, 0x2D,
+ 0xF0, 0xF9, 0x9C, 0x0B, 0xB2, 0x1C, 0xFD, 0xB6,
+ 0x3F, 0x86, 0x9F, 0x4A, 0xA7, 0xA3, 0x8C, 0xC1,
+ 0xAD, 0xF3, 0xFA, 0x86, 0xEF, 0x3F, 0xD7, 0x86,
+ 0x05, 0xF0, 0x8D, 0xA6, 0xD1, 0xFE, 0xE0, 0xB4,
+ 0x12, 0xC6, 0x35, 0x88, 0xFE, 0x77, 0xDE, 0x4E,
+ 0x36, 0x4C, 0x8A, 0x81, 0x62, 0xC0, 0x30, 0x95,
+ 0xC9, 0x2E, 0xB5, 0xCD, 0x09, 0x8D, 0x14, 0xDF,
+ 0xED, 0x2E, 0x2D, 0xCE, 0x8A, 0x94, 0x7A, 0xC7,
+ 0x12, 0x51, 0x8B, 0xEF, 0x45, 0xE9, 0x7A, 0x5E,
+ 0x1E, 0x51, 0x73, 0x34, 0x51, 0x2B, 0xCB, 0x45,
+ 0x22, 0xE6, 0x6F, 0x62, 0x20, 0xBA, 0xD2, 0x4E,
+ 0xDC, 0x1F, 0xEE, 0x25, 0xFD, 0xA5, 0xD8, 0xD6,
+ 0x4A, 0x24, 0xEA, 0x5E, 0x36, 0xF5, 0x76, 0x76,
+ 0x27, 0x53, 0x99, 0xA1, 0xAF, 0x9E, 0x3C, 0x2A,
+ 0x4C, 0xB2, 0x26, 0x8C, 0xC4, 0x1D, 0x38, 0x0C,
+ 0x3F, 0xC6, 0x34, 0xA8, 0x5B, 0x96, 0x46, 0x4B,
+ 0xE4, 0x17, 0xB8, 0x91, 0xD8, 0x28, 0x43, 0x99,
+ 0xD8, 0xB5, 0x1D, 0x87, 0x66, 0x29, 0xC6, 0x8E,
+ 0x43, 0x07, 0xAE, 0xBC, 0x05, 0xBF, 0xE2, 0xED,
+ 0xDD, 0xCD, 0xDA, 0x65, 0xC9, 0x87, 0x95, 0xB6,
+ 0x4D, 0x64, 0x64, 0x23, 0x5D, 0x19, 0x9D, 0x47,
+ 0xC0, 0xED, 0x36, 0x03, 0x8C, 0x1E, 0xBA, 0xDE,
+ 0xB7, 0x5D, 0xB8, 0x2B, 0x10, 0x35, 0x6F, 0xCD,
+ 0x8E, 0xF8, 0xFA, 0xC0, 0x95, 0x34, 0x5C, 0x6A,
+ 0x49, 0xDB, 0x30, 0x4C, 0xEB, 0x61, 0x27, 0xF5,
+ 0x83, 0x5B, 0xBF, 0x05, 0x1F, 0x56, 0xF0, 0x4B,
+ 0x1C, 0x25, 0x29, 0xA2, 0xC5, 0xF2, 0x46, 0xF6,
+ 0xE9, 0x04, 0xDB, 0x7D, 0x01, 0xDE, 0xE0, 0x76,
+ 0xEC, 0xB0, 0x24, 0x61, 0x41, 0xA4, 0x53, 0x84,
+ 0x90, 0x1B, 0x96, 0x72, 0x00, 0x14, 0x26, 0xED,
+ 0x20, 0x31, 0x5A, 0x24, 0xC9, 0xB2, 0x2A, 0x75,
+ 0xD3, 0x94, 0xC1, 0xFE, 0xCE, 0xC1, 0x57, 0xDE,
+ 0x5E, 0xCD, 0xBE, 0xCB, 0x85, 0x4A, 0x6B, 0x85,
+ 0x28, 0xC3, 0x7B, 0x12, 0x7F, 0x61, 0xDD, 0x22,
+ 0x07, 0x37, 0xFF, 0x70, 0x7A, 0xA1, 0x56, 0xD4,
+ 0x5A, 0x3A, 0x59, 0xAB, 0xC3, 0x65, 0x09, 0x0E,
+ 0x3C, 0x9C, 0x52, 0x34, 0xB2, 0x5D, 0x89, 0x62,
+ 0x89, 0x3E, 0x22, 0x07, 0x56, 0xA2, 0x4D, 0x95,
+ 0xE8, 0x78, 0x15, 0x44, 0x6F, 0xA3, 0xB2, 0x2C,
+ 0x24, 0xF8, 0xA0, 0x54, 0x9E, 0x73, 0x25, 0x0C,
+ 0xB3, 0x2C, 0xFF, 0x2D, 0x9C, 0x2E, 0xE1, 0xD8,
+ 0x70, 0xF1, 0xCA, 0xA4, 0xB6, 0x6E, 0xDF, 0xE5,
+ 0xB4, 0x0F, 0xF7, 0x00, 0x8E, 0x4F, 0x6F, 0xBA,
+ 0x0D, 0x1F, 0x82, 0xF0, 0x7D, 0xC4, 0xED, 0x1E,
+ 0xB6, 0x61, 0xE2, 0x3C, 0xBF, 0x5F, 0x88, 0xE1,
+ 0x37, 0x5E, 0xE7, 0x7D, 0xE2, 0x13, 0xDF, 0x9A,
+ 0x59, 0x6D, 0x85, 0x87, 0x7D, 0xCD, 0x8A, 0x01,
+ 0xA1, 0xA7, 0x10, 0x63, 0xB5, 0xA4, 0xC4, 0xE0,
+ 0x1A, 0x45, 0x4E, 0x91, 0x07, 0x54, 0x62, 0xD6,
+ 0xC2, 0x3F, 0x96, 0x40, 0xF6, 0x8B, 0x12, 0x59,
+ 0xEF, 0x20, 0x9C, 0x35, 0x4E, 0x37, 0x1F, 0xAA,
+ 0x9C, 0x01, 0xA5, 0x59, 0x85, 0x37, 0x9E, 0x4F,
+ 0xDF, 0xFF, 0x40, 0x6E, 0xC2, 0xE6, 0xC6, 0x09,
+ 0x70, 0x1C, 0xC2, 0xCD, 0x11, 0x6B, 0x84, 0xC8,
+ 0xEE, 0x2B, 0xAE, 0xD5, 0x36, 0xFC, 0x83, 0x30,
+ 0xDD, 0x5E, 0x13, 0x6C, 0x49, 0xDB, 0x44, 0xDD,
+ 0x39, 0x58, 0xE7, 0x83, 0xD6, 0x1A, 0x5F, 0xE2,
+ 0xF7, 0x82, 0xDC, 0x1E, 0x80, 0x44, 0xDB, 0x48,
+ 0x62, 0x32, 0x75, 0x3E, 0x6A, 0x98, 0x1A, 0xB2,
+ 0x36, 0xBF, 0xDE, 0x35, 0x6B, 0x20, 0x9F, 0x83,
+ 0xB5, 0xA2, 0x1C, 0x0B, 0x99, 0x9B, 0x49, 0x43,
+ 0xEB, 0x5B, 0x67, 0x03, 0x8A, 0x24, 0x45, 0x8C,
+ 0x15, 0xEE, 0xD2, 0xB2, 0x35, 0xEB, 0xAF, 0x69,
+ 0x0E, 0x69, 0x3B, 0x4D, 0x80, 0x9B, 0x75, 0x43,
+ 0xB4, 0x40, 0x30, 0xCF, 0xD9, 0x76, 0xD4, 0x58,
+ 0xAC, 0x2B, 0x10, 0xE3, 0xF3, 0x48, 0x38, 0xAA,
+ 0x69, 0x18, 0x51, 0x5C, 0xD7, 0xB0, 0x9E, 0x5B,
+ 0x1E, 0x18, 0xD7, 0x59, 0x67, 0x80, 0x4F, 0xCB,
+ 0x40, 0x49, 0xB0, 0x27, 0xCD, 0x15, 0xF7, 0x56,
+ 0x34, 0x0F, 0x5D, 0xD9, 0xC4, 0xCB, 0xF7, 0x83,
+ 0x0D, 0x8B, 0xCF, 0x21, 0x13, 0x1A, 0x49, 0xE6,
+ 0x21, 0x49, 0x0F, 0x67, 0xFF, 0xE6, 0xE7, 0xF4,
+ 0xD7, 0x2C, 0xDA, 0xFD, 0xBD, 0x86, 0xC2, 0x40,
+ 0x6E, 0xB5, 0x05, 0xFC, 0x3E, 0x88, 0xB4, 0xB1,
+ 0xC1, 0x5C, 0x6D, 0x0F, 0x0F, 0x17, 0xE6, 0xE0,
+ 0x96, 0x4B, 0x45, 0xBD, 0x07, 0x20, 0x00, 0x3F,
+ 0xF0, 0xF2, 0xB9, 0xA1, 0x35, 0x01, 0x47, 0x81,
+ 0x1F, 0xA2, 0x0A, 0x66, 0xDB, 0xD7, 0x58, 0x51,
+ 0x88, 0x78, 0x24, 0x6F, 0x7E, 0x68, 0x46, 0xBD,
+ 0xEB, 0x5C, 0xA9, 0xDE, 0x30, 0x34, 0x2B, 0xA3,
+ 0xCF, 0x93, 0x82, 0x14, 0x3D, 0x96, 0x1E, 0xD7,
+ 0x6E, 0x9A, 0x2E, 0x72, 0xD0, 0x49, 0xDB, 0x24,
+ 0xA0, 0x6E, 0x8E, 0xBA, 0x4A, 0x36, 0xCB, 0xF6,
+ 0x9A, 0x46, 0x24, 0xEF, 0x18, 0xA8, 0xD9, 0xD5,
+ 0x93, 0xF0, 0x2B, 0xEC, 0x60, 0x0A, 0x8F, 0x1C,
+ 0xE8, 0x69, 0x56, 0x14, 0x6C, 0x8E, 0x30, 0xFB,
+ 0x8D, 0xCF, 0xE7, 0x92, 0x59, 0x2E, 0x32, 0xEB,
+ 0xAE, 0xA3, 0x99, 0x87, 0x98, 0x49, 0x03, 0xCD,
+ 0x5E, 0xBD, 0xDE, 0xB9, 0xF5, 0xD0, 0xB3, 0xB1,
+ 0x65, 0xAC, 0xC7, 0x42, 0xB4, 0xCF, 0x6A, 0x00,
+ 0x3E, 0x2E, 0xC1, 0x13, 0xDF, 0x5E, 0x2E, 0xEC,
+ 0x30, 0x00, 0xE2, 0xAB, 0xFD, 0x74, 0x06, 0x55,
+ 0x88, 0x07, 0xB0, 0xEF, 0x6D, 0x43, 0x81, 0x15,
+ 0xFC, 0xF6, 0x96, 0x91, 0x3D, 0x2F, 0xFE, 0xBD,
+ 0xB9, 0xC8, 0xB6, 0x81, 0x63, 0xE0, 0xA9, 0x20,
+ 0x93, 0xEC, 0x4B, 0x1E, 0xB9, 0xC2, 0xD0, 0x8C,
+ 0x2E, 0xCB, 0x18, 0x7D, 0x1F, 0x66, 0xA6, 0x96,
+ 0xA9, 0xB8, 0x66, 0x4F, 0xCF, 0xC9, 0xDA, 0x97,
+ 0xCA, 0x07, 0xF1, 0xC8, 0xFB, 0x56, 0x60, 0xA9,
+ 0x25, 0xDB, 0xFC, 0x2D, 0xB6, 0x0A, 0x42, 0x6A,
+ 0x7F, 0xCE, 0x70, 0x91, 0xFF, 0x3B, 0xAC, 0xAB,
+ 0xF2, 0x23, 0x4E, 0x50, 0xE9, 0xE6, 0x2F, 0xCB,
+ 0x98, 0xBA, 0x7D, 0xD2, 0x8A, 0xDE, 0x6E, 0x80,
+ 0x0C, 0xC6, 0xEB, 0xD9, 0x64, 0xD4, 0x59, 0xD5,
+ 0x75, 0x00, 0x18, 0x7F, 0xD6, 0x86, 0xC2, 0x25,
+ 0x98, 0xA9, 0x28, 0x1C, 0x40, 0x46, 0xC6, 0xA5,
+ 0xAF, 0x6E, 0x5D, 0x25, 0x3C, 0x77, 0xF9, 0x43,
+ 0xF3, 0x20, 0xFC, 0x43, 0xA0, 0x5E, 0xF1, 0x65,
+ 0x5D, 0x8B, 0x33, 0x5A, 0x94, 0xF0, 0x6F, 0xB6,
+ 0xD4, 0x4A, 0x48, 0x64, 0x31, 0x73, 0x17, 0x4E,
+ 0x88, 0x90, 0xA8, 0xE4, 0x7F, 0xCE, 0xD5, 0x7D,
+ 0xE3, 0x84, 0x3F, 0x38, 0x71, 0x4C, 0xB1, 0x18,
+ 0xAA, 0x46, 0xF2, 0xCF, 0x99, 0xF0, 0x24, 0xD7,
+ 0xFD, 0xFC, 0x2C, 0x81, 0x25, 0xC1, 0x8E, 0x0B,
+ 0x82, 0x4C, 0x14, 0x6C, 0x44, 0xB1, 0x78, 0x67,
+ 0x14, 0x47, 0xCA, 0x70, 0x0D, 0x13, 0xD1, 0xA8,
+ 0x73, 0xBC, 0x4E, 0xAA, 0x1E, 0xB7, 0x59, 0xC0,
+ 0xAC, 0xE3, 0x21, 0x2B, 0x55, 0x22, 0x6A, 0x53,
+ 0xA6, 0xF0, 0xE9, 0x56, 0x37, 0x3B, 0xD6, 0x1B,
+ 0x2E, 0x57, 0x98, 0x4D, 0x6A, 0x7E, 0xEB, 0x2E,
+ 0x9B, 0xB8, 0xE2, 0x27, 0x55, 0xE2, 0x2E, 0xFA,
+ 0x3C, 0xD2, 0x7A, 0xCC, 0xDB, 0x5C, 0x45, 0x85,
+ 0xA6, 0x92, 0x49, 0x79, 0x9D, 0x18, 0x20, 0x50,
+ 0x7B, 0xF4, 0x0F, 0x43, 0x2F, 0x7B, 0x3E, 0x90,
+ 0xEF, 0xF3, 0x29, 0x66, 0xDF, 0xD1, 0xE9, 0x44,
+ 0xC6, 0x28, 0xAA, 0x48, 0x00, 0x5F, 0x12, 0xEB,
+ 0xEC, 0x26, 0x7F, 0xB3, 0x83, 0xFB, 0x50, 0x5A,
+ 0x5F, 0x5A, 0x8E, 0x08, 0x4B, 0xF7, 0x50, 0x10,
+ 0xC8, 0x73, 0x8F, 0x9C, 0xF6, 0xCB, 0xDC, 0xEA,
+ 0x78, 0x07, 0x2C, 0x59, 0xAD, 0x66, 0x48, 0xFF,
+ 0x56, 0x7D, 0x99, 0xC9, 0xEA, 0x64, 0xE7, 0xD0,
+ 0x46, 0x88, 0x58, 0x46, 0x93, 0x37, 0x70, 0xC1,
+ 0x91, 0x40, 0x02, 0x46, 0x7F, 0x8D, 0xFE, 0x66,
+ 0x07, 0x1F, 0x3B, 0x56, 0x6A, 0x6E, 0x88, 0xB2,
+ 0xBB, 0xC0, 0xC8, 0xCD, 0xD9, 0xDC, 0x0A, 0x0C,
+ 0x1A, 0x27, 0x35, 0x48, 0x4C, 0x5B, 0x5D, 0x90,
+ 0xA0, 0xB2, 0xB4, 0xCE, 0xD6, 0xDD, 0x07, 0x13,
+ 0x7C, 0x93, 0x99, 0xA2, 0xA9, 0xAC, 0xC3, 0xC8,
+ 0xEF, 0x1D, 0x33, 0x3D, 0x3F, 0x42, 0x52, 0x57,
+ 0x6D, 0x72, 0x87, 0x8E, 0x96, 0xA8, 0xC6, 0xC8,
+ 0xD0, 0xD6, 0xE8, 0xEB, 0xF2, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x0E, 0x1E, 0x29, 0x3D,
+};
+
+uint8_t mldsa_44_message[] = {
+ 0x54, 0x68, 0x69, 0x73, 0x20, 0x69, 0x73, 0x20,
+ 0x74, 0x68, 0x65, 0x20, 0x6D, 0x65, 0x73, 0x73,
+ 0x61, 0x67, 0x65, 0x20, 0x74, 0x6F, 0x20, 0x62,
+ 0x65, 0x20, 0x73, 0x69, 0x67, 0x6E, 0x65, 0x64,
+ 0x2E, 0x0A,
+};
+
+uint8_t mldsa_44_privkey_noseed[] = {
+ 0xBA, 0x71, 0xF9, 0xF6, 0x4E, 0x11, 0xBA, 0xEB,
+ 0x58, 0xFA, 0x9C, 0x6F, 0xBB, 0x6E, 0x14, 0xE6,
+ 0x1F, 0x18, 0x64, 0x3D, 0xAB, 0x49, 0x5B, 0x47,
+ 0x53, 0x9A, 0x91, 0x66, 0xCA, 0x01, 0x98, 0x13,
+ 0xC5, 0xC1, 0x61, 0x0A, 0x40, 0x77, 0x4F, 0x0E,
+ 0xBA, 0x33, 0x34, 0xF8, 0xB5, 0xBE, 0x56, 0xE8,
+ 0x78, 0x71, 0xB3, 0xC3, 0xA7, 0x72, 0xC0, 0x72,
+ 0x0F, 0xA3, 0x76, 0x66, 0xAE, 0x17, 0x35, 0xFD,
+ 0xE6, 0xBC, 0x38, 0xA1, 0xC3, 0x5F, 0x8C, 0xF0,
+ 0x8E, 0x44, 0x09, 0x24, 0xC9, 0x03, 0x71, 0x97,
+ 0xBB, 0x87, 0xFD, 0xC4, 0x64, 0x6B, 0x86, 0xDA,
+ 0x5A, 0x05, 0x89, 0xA3, 0x26, 0xCC, 0x0C, 0x0D,
+ 0x95, 0x0F, 0xF8, 0xB5, 0xA9, 0xEA, 0x41, 0x35,
+ 0xEA, 0xB8, 0xA9, 0x3F, 0x80, 0xF0, 0x92, 0x7E,
+ 0x12, 0x40, 0x46, 0xE2, 0x5B, 0x23, 0x66, 0xAE,
+ 0xA2, 0x5A, 0x6D, 0x1D, 0x0F, 0xEF, 0x98, 0x21,
+ 0x04, 0xB8, 0x09, 0xDA, 0x12, 0x48, 0xE1, 0x40,
+ 0x4C, 0x11, 0x03, 0x85, 0x99, 0xB0, 0x4D, 0x61,
+ 0x04, 0x0A, 0xC0, 0x34, 0x28, 0x23, 0x16, 0x80,
+ 0x08, 0x12, 0x45, 0x11, 0x49, 0x0C, 0xA0, 0x96,
+ 0x0D, 0xC2, 0xA8, 0x01, 0x5A, 0xA8, 0x65, 0x0B,
+ 0x30, 0x25, 0xD4, 0x94, 0x44, 0x8B, 0x38, 0x8C,
+ 0x10, 0x24, 0x41, 0x22, 0x40, 0x69, 0x8A, 0x04,
+ 0x04, 0x0C, 0x26, 0x90, 0xE3, 0x34, 0x69, 0xE2,
+ 0x24, 0x68, 0xD2, 0x44, 0x00, 0x0B, 0x84, 0x90,
+ 0xD9, 0x42, 0x30, 0x21, 0x91, 0x8D, 0x09, 0x99,
+ 0x64, 0x20, 0x39, 0x60, 0x82, 0x36, 0x21, 0x90,
+ 0xB8, 0x4C, 0x02, 0x12, 0x6E, 0x24, 0xB9, 0x70,
+ 0x84, 0x22, 0x01, 0xE1, 0xA8, 0x64, 0x44, 0x26,
+ 0x0E, 0x01, 0x17, 0x0E, 0x82, 0x40, 0x6C, 0x9B,
+ 0x38, 0x6D, 0x0A, 0x93, 0x50, 0x52, 0x22, 0x25,
+ 0xD2, 0xB8, 0x2C, 0x1A, 0x25, 0x70, 0x90, 0x12,
+ 0x22, 0xD1, 0xB2, 0x80, 0x22, 0x46, 0x52, 0x01,
+ 0x34, 0x80, 0x40, 0x34, 0x21, 0x41, 0x06, 0x60,
+ 0x41, 0x32, 0x82, 0x09, 0x45, 0x90, 0x52, 0x32,
+ 0x29, 0x83, 0xB8, 0x44, 0x93, 0x12, 0x21, 0x0A,
+ 0xC1, 0x89, 0x60, 0xB2, 0x71, 0x9C, 0x90, 0x69,
+ 0x90, 0x82, 0x90, 0x89, 0x12, 0x49, 0x40, 0x40,
+ 0x8A, 0x5C, 0x10, 0x48, 0x01, 0x87, 0x91, 0x03,
+ 0xB0, 0x40, 0xD8, 0x44, 0x05, 0xE4, 0xB0, 0x29,
+ 0x0A, 0x27, 0x90, 0x1C, 0xC9, 0x50, 0x58, 0x02,
+ 0x8C, 0x0C, 0x49, 0x72, 0x49, 0x84, 0x6D, 0x19,
+ 0xA7, 0x85, 0xE3, 0x00, 0x30, 0xCB, 0xB2, 0x21,
+ 0xE2, 0x08, 0x10, 0x4C, 0xA6, 0x64, 0x11, 0x43,
+ 0x68, 0x52, 0x94, 0x69, 0x09, 0x39, 0x32, 0x91,
+ 0xA2, 0x6D, 0xCA, 0x14, 0x84, 0xA3, 0x92, 0x11,
+ 0xDB, 0x94, 0x2C, 0x01, 0xC6, 0x64, 0x14, 0x24,
+ 0x8C, 0x1A, 0x15, 0x31, 0x0A, 0x23, 0x0E, 0xD8,
+ 0x14, 0x20, 0xA4, 0x26, 0x8D, 0x43, 0x40, 0x90,
+ 0xC1, 0x42, 0x86, 0xA4, 0x10, 0x85, 0x0C, 0x32,
+ 0x2D, 0xD0, 0x28, 0x60, 0x1B, 0xB5, 0x0C, 0x52,
+ 0x44, 0x61, 0x21, 0x15, 0x24, 0x21, 0xC6, 0x04,
+ 0xD8, 0x34, 0x31, 0xD9, 0x12, 0x71, 0xA2, 0x34,
+ 0x52, 0x03, 0x30, 0x89, 0x54, 0x32, 0x10, 0x83,
+ 0x94, 0x08, 0xA2, 0xA4, 0x70, 0xD2, 0x44, 0x8E,
+ 0x84, 0x02, 0x8D, 0x91, 0x46, 0x62, 0x0B, 0x00,
+ 0x0E, 0xA0, 0x80, 0x70, 0x22, 0xB5, 0x0D, 0x0B,
+ 0xC9, 0x01, 0x00, 0x26, 0x4D, 0xE2, 0x44, 0x80,
+ 0x93, 0x20, 0x65, 0xC9, 0x96, 0x8D, 0xDA, 0xA4,
+ 0x4C, 0x42, 0xA8, 0x00, 0x1A, 0x27, 0x6C, 0xC3,
+ 0x94, 0x4D, 0xA0, 0x30, 0x2C, 0xDA, 0xA8, 0x04,
+ 0x23, 0xA3, 0x8C, 0x63, 0x04, 0x0E, 0x59, 0xC8,
+ 0x00, 0x20, 0xA9, 0x6C, 0xC4, 0x06, 0x50, 0x82,
+ 0x00, 0x2E, 0x5C, 0xB2, 0x44, 0x61, 0x46, 0x51,
+ 0x0C, 0x18, 0x80, 0x18, 0x38, 0x82, 0x02, 0x25,
+ 0x90, 0xA2, 0xB2, 0x60, 0x21, 0xA0, 0x40, 0x20,
+ 0x49, 0x2A, 0x52, 0x28, 0x86, 0x93, 0xA6, 0x28,
+ 0x9B, 0x94, 0x71, 0x02, 0xB9, 0x6D, 0x49, 0xA0,
+ 0x2C, 0x09, 0x25, 0x28, 0x18, 0x93, 0x09, 0x10,
+ 0x49, 0x4C, 0x13, 0xC9, 0x91, 0xA3, 0x88, 0x2D,
+ 0x50, 0x12, 0x31, 0x63, 0xB4, 0x00, 0xA4, 0x00,
+ 0x4E, 0x04, 0xB6, 0x68, 0x12, 0x14, 0x26, 0x00,
+ 0x17, 0x00, 0x9B, 0x92, 0x0C, 0x51, 0xA8, 0x65,
+ 0x11, 0x44, 0x8D, 0x44, 0x42, 0x69, 0x00, 0xA5,
+ 0x6D, 0x04, 0xA9, 0x69, 0x43, 0x94, 0x69, 0x4C,
+ 0xB8, 0x60, 0x52, 0x26, 0x2D, 0xA1, 0x32, 0x11,
+ 0xCA, 0xB0, 0x11, 0x1B, 0x81, 0x90, 0x12, 0x11,
+ 0x64, 0x42, 0x86, 0x89, 0x9C, 0x12, 0x51, 0xDC,
+ 0xC0, 0x41, 0x13, 0x47, 0x68, 0x9B, 0x20, 0x92,
+ 0xD1, 0x26, 0x91, 0x23, 0x96, 0x50, 0x13, 0xB3,
+ 0x4C, 0x12, 0x45, 0x69, 0x21, 0xA1, 0x05, 0x0C,
+ 0x46, 0x65, 0x99, 0x38, 0x30, 0x99, 0x26, 0x2E,
+ 0x19, 0x15, 0x72, 0x11, 0x39, 0x8D, 0x89, 0x10,
+ 0x92, 0x50, 0x28, 0x11, 0x48, 0xA6, 0x49, 0x52,
+ 0xB6, 0x2D, 0xC9, 0x30, 0x4D, 0xD9, 0x32, 0x91,
+ 0x91, 0xA6, 0x44, 0x14, 0x34, 0x0E, 0x13, 0x39,
+ 0x41, 0x98, 0x06, 0x0C, 0x22, 0x25, 0x6C, 0x14,
+ 0x47, 0x70, 0x21, 0xB7, 0x60, 0x23, 0x98, 0x25,
+ 0xE4, 0x24, 0x6E, 0xD1, 0xB6, 0x44, 0x51, 0x44,
+ 0x2C, 0x41, 0x42, 0x85, 0xE0, 0x44, 0x42, 0x89,
+ 0x42, 0x42, 0xC2, 0x24, 0x2C, 0xCC, 0xC6, 0x68,
+ 0x13, 0x12, 0x69, 0x98, 0x30, 0x49, 0x23, 0x28,
+ 0x6A, 0x14, 0xC8, 0x11, 0x60, 0x92, 0x30, 0x11,
+ 0x26, 0x52, 0x92, 0xB0, 0x00, 0x5C, 0x84, 0x2C,
+ 0x4A, 0x14, 0x29, 0x0A, 0x95, 0x40, 0xE3, 0xC4,
+ 0x05, 0x0C, 0xC8, 0x89, 0xA1, 0x26, 0x62, 0x44,
+ 0x10, 0x04, 0x9A, 0xC8, 0x70, 0x0C, 0x93, 0x60,
+ 0x90, 0xB6, 0x30, 0x12, 0x17, 0x71, 0x23, 0x18,
+ 0x4E, 0xD4, 0x10, 0x85, 0x84, 0x20, 0x82, 0x5C,
+ 0x14, 0x48, 0x90, 0xB2, 0x51, 0xA0, 0x18, 0x92,
+ 0x93, 0x94, 0x80, 0x1A, 0x49, 0x72, 0xC0, 0x02,
+ 0x06, 0xE1, 0xA6, 0x8C, 0xA1, 0x80, 0x4D, 0x4A,
+ 0x26, 0x08, 0x18, 0x07, 0x62, 0x42, 0x12, 0x40,
+ 0x10, 0x41, 0x64, 0x44, 0x42, 0x4E, 0x94, 0x86,
+ 0x40, 0x9A, 0x84, 0x41, 0x21, 0xC5, 0x71, 0x01,
+ 0x96, 0x29, 0x19, 0x40, 0x4E, 0x94, 0x36, 0x22,
+ 0x24, 0xC3, 0x20, 0x22, 0x88, 0x71, 0x84, 0x20,
+ 0x4C, 0x64, 0x20, 0x69, 0xDB, 0x84, 0x24, 0x24,
+ 0x25, 0x46, 0x21, 0xC4, 0x71, 0x8A, 0x14, 0x64,
+ 0x01, 0x20, 0x65, 0x4C, 0x84, 0x24, 0x44, 0x24,
+ 0x30, 0x8A, 0x22, 0x70, 0xD3, 0x86, 0x30, 0x0B,
+ 0xB4, 0x6C, 0x21, 0x23, 0x0A, 0x8B, 0xB8, 0x4D,
+ 0x5C, 0xF7, 0xD7, 0xE6, 0x89, 0x30, 0x2B, 0xED,
+ 0xB1, 0xC5, 0x86, 0x7E, 0x7D, 0x26, 0x9B, 0x1C,
+ 0xDB, 0x07, 0xF8, 0x25, 0x64, 0x10, 0x82, 0xE1,
+ 0x9A, 0x8D, 0xA2, 0xF9, 0x30, 0x77, 0xE8, 0xB1,
+ 0xFC, 0x3D, 0x4E, 0x6B, 0x2D, 0x32, 0x58, 0x33,
+ 0x6B, 0x4F, 0x9C, 0x64, 0x55, 0x15, 0x3A, 0xC0,
+ 0x40, 0xA8, 0x47, 0xFB, 0x64, 0x7F, 0xBB, 0x6B,
+ 0x55, 0x2A, 0x40, 0x00, 0x71, 0xFE, 0x17, 0x72,
+ 0x48, 0x5B, 0x7A, 0x9D, 0x1F, 0x0D, 0x14, 0x7B,
+ 0xF3, 0x38, 0x8C, 0x56, 0x54, 0x71, 0xE4, 0xE6,
+ 0x2C, 0xC3, 0xCE, 0x0D, 0x0C, 0x0F, 0xC3, 0x60,
+ 0xDF, 0x92, 0x89, 0xED, 0x99, 0x18, 0x37, 0x6B,
+ 0x8B, 0x8B, 0x93, 0x14, 0x50, 0x47, 0xF8, 0xFE,
+ 0xA2, 0x98, 0x60, 0x07, 0xC2, 0xAA, 0x89, 0x92,
+ 0x2F, 0x69, 0xEB, 0x47, 0x5B, 0x59, 0x7B, 0x2B,
+ 0xBA, 0x23, 0x7B, 0x9C, 0x84, 0x2E, 0x3F, 0xF1,
+ 0xD3, 0x25, 0xE8, 0x2A, 0x1F, 0x23, 0xE9, 0x49,
+ 0x89, 0xD0, 0x06, 0xBC, 0x7C, 0xE4, 0x94, 0x6F,
+ 0x2E, 0x8B, 0x77, 0xE1, 0x08, 0x48, 0x46, 0x3C,
+ 0x47, 0xFE, 0x7B, 0x20, 0x9E, 0x2A, 0x61, 0x7D,
+ 0xDD, 0x41, 0x79, 0x6A, 0xE6, 0x14, 0x5E, 0x70,
+ 0x9C, 0xDA, 0x94, 0x06, 0xF2, 0x26, 0x12, 0x57,
+ 0xC2, 0x13, 0xB4, 0xB3, 0x0D, 0xA3, 0x0A, 0xC2,
+ 0x5B, 0x0D, 0x06, 0xCF, 0x79, 0xA8, 0x12, 0xC5,
+ 0xFC, 0xB0, 0xEF, 0x11, 0xD9, 0xFE, 0xDF, 0xE0,
+ 0x99, 0x4A, 0xFE, 0x3B, 0x69, 0xB0, 0x6A, 0x29,
+ 0x16, 0xCF, 0x69, 0x2B, 0x9D, 0xA7, 0x60, 0x28,
+ 0xE5, 0xF3, 0xA0, 0x48, 0x79, 0xE6, 0x96, 0xD2,
+ 0x1F, 0x73, 0x5C, 0x37, 0x83, 0x15, 0x36, 0x4D,
+ 0xB0, 0xA4, 0xE0, 0xAB, 0x6B, 0x53, 0xD3, 0x1E,
+ 0xFA, 0xF3, 0x0D, 0x65, 0xE3, 0x7A, 0x1B, 0x6A,
+ 0x77, 0x04, 0x6F, 0x04, 0xC6, 0x4B, 0xA1, 0x07,
+ 0x2A, 0x97, 0x80, 0xE0, 0xC5, 0x66, 0xC9, 0x43,
+ 0x39, 0xA4, 0xD1, 0x9D, 0x00, 0x68, 0xC5, 0x7D,
+ 0x6E, 0x6F, 0x0B, 0x51, 0x2D, 0xB7, 0x13, 0x4A,
+ 0x95, 0x0E, 0xAF, 0x4F, 0x7B, 0x01, 0xA5, 0xFD,
+ 0xD0, 0x65, 0xB9, 0x1B, 0xFA, 0x29, 0xE4, 0x42,
+ 0x36, 0x79, 0xCD, 0xE7, 0x4B, 0xC6, 0xA8, 0xF1,
+ 0xC8, 0x4C, 0x4D, 0xF7, 0x83, 0x87, 0x23, 0x1D,
+ 0xC8, 0x5C, 0xE3, 0x26, 0x70, 0x44, 0x59, 0x03,
+ 0xC4, 0xBE, 0xBE, 0xE3, 0xF5, 0x0C, 0x43, 0xE5,
+ 0x04, 0x49, 0x49, 0x69, 0x11, 0xAA, 0x93, 0xE7,
+ 0xE3, 0x95, 0x78, 0x74, 0x14, 0xD3, 0x17, 0x68,
+ 0xD9, 0x91, 0x25, 0x20, 0xF8, 0x3C, 0x02, 0xFF,
+ 0x01, 0x12, 0x4D, 0xCF, 0x0E, 0x12, 0x5F, 0xAF,
+ 0xD5, 0xB9, 0xD7, 0xE7, 0xDD, 0xA4, 0xF5, 0xB5,
+ 0x0C, 0x70, 0xAE, 0xBB, 0x85, 0x99, 0xA2, 0xE4,
+ 0x47, 0x6A, 0x0D, 0xE5, 0x31, 0xB0, 0x40, 0x26,
+ 0x72, 0xDF, 0x75, 0x75, 0x14, 0x2D, 0x86, 0x01,
+ 0x60, 0x5C, 0x94, 0x01, 0x79, 0x23, 0xF6, 0x4A,
+ 0xC5, 0x77, 0xC4, 0xBE, 0xD8, 0xD8, 0xE8, 0x9A,
+ 0x74, 0xCA, 0x9F, 0x38, 0x19, 0xCB, 0xF1, 0x42,
+ 0xA7, 0x2D, 0xEB, 0xE7, 0x7C, 0x4E, 0xFB, 0x71,
+ 0x27, 0xE2, 0xD8, 0xC1, 0xB7, 0xBF, 0xB6, 0x42,
+ 0x86, 0xC0, 0xBD, 0x52, 0x23, 0x3F, 0x43, 0xC6,
+ 0x7D, 0x57, 0x17, 0xF9, 0x7A, 0xD8, 0x28, 0x54,
+ 0x87, 0x3D, 0xDC, 0x7F, 0x71, 0xD6, 0x56, 0xAA,
+ 0xA6, 0xEF, 0x70, 0x70, 0x60, 0xAF, 0x28, 0x0B,
+ 0x9F, 0x45, 0x4B, 0x4F, 0xED, 0xB4, 0x77, 0x6E,
+ 0x83, 0xB2, 0xFD, 0xBA, 0x20, 0xA4, 0x5A, 0xEF,
+ 0xEB, 0x54, 0x9A, 0x1E, 0xD0, 0x38, 0x20, 0x21,
+ 0x89, 0x3C, 0xA9, 0xA6, 0xE7, 0x4C, 0xCC, 0x30,
+ 0xA2, 0x55, 0x39, 0x37, 0xCC, 0xEF, 0x34, 0x38,
+ 0x99, 0xB5, 0x02, 0xCF, 0x46, 0xDD, 0xB8, 0xDD,
+ 0x1D, 0x95, 0xFE, 0xFB, 0x60, 0xC9, 0xB2, 0x04,
+ 0x69, 0xA1, 0x50, 0x3B, 0x2A, 0x68, 0x75, 0x87,
+ 0x83, 0x0D, 0x33, 0xCE, 0xE9, 0xA7, 0x2D, 0x79,
+ 0x8F, 0xCF, 0x4A, 0x9B, 0x45, 0x2C, 0x85, 0x49,
+ 0xF5, 0x59, 0xC5, 0xD9, 0xFC, 0x6B, 0xFE, 0x08,
+ 0x3F, 0x44, 0x6C, 0x2D, 0x90, 0x39, 0x81, 0xD9,
+ 0xF2, 0x64, 0x92, 0x48, 0x3A, 0xB4, 0x52, 0xEA,
+ 0x5B, 0xB1, 0x00, 0x8F, 0xFE, 0xAC, 0x97, 0x5D,
+ 0xA0, 0x27, 0x59, 0x59, 0x3E, 0x7E, 0x06, 0x63,
+ 0x61, 0x07, 0x3A, 0x83, 0xB2, 0x7B, 0x53, 0x1A,
+ 0x3D, 0x0D, 0xDA, 0x51, 0x7C, 0xA9, 0x90, 0xEA,
+ 0x32, 0x35, 0xD1, 0xD7, 0xB5, 0xE0, 0x9D, 0xA5,
+ 0xF0, 0x2D, 0xC1, 0x52, 0x5B, 0x1D, 0xA6, 0x85,
+ 0x96, 0x5B, 0x54, 0xFC, 0x2A, 0x3A, 0x73, 0xA1,
+ 0x79, 0x0E, 0x0E, 0xFB, 0x69, 0xE7, 0x0A, 0x78,
+ 0xFA, 0x55, 0x03, 0x44, 0xEA, 0x8C, 0x75, 0x3D,
+ 0xBF, 0x18, 0x63, 0x9B, 0xAA, 0x8C, 0xB1, 0x25,
+ 0x9A, 0xA7, 0x4F, 0x68, 0xF9, 0x2A, 0xBA, 0x80,
+ 0x07, 0xC6, 0x18, 0xCC, 0xB6, 0xF5, 0x06, 0x9F,
+ 0xF4, 0x6B, 0x97, 0x51, 0xBB, 0xFF, 0xF3, 0x7D,
+ 0xF3, 0x21, 0x36, 0x0F, 0x0F, 0x5C, 0x0E, 0x7F,
+ 0x56, 0x26, 0xDD, 0x12, 0x9A, 0xE3, 0xAE, 0x2A,
+ 0x7C, 0x56, 0xCD, 0xB6, 0x11, 0xED, 0xA4, 0xC9,
+ 0x8F, 0xEC, 0x83, 0x16, 0x3C, 0xD5, 0x11, 0x68,
+ 0x78, 0xC1, 0xA9, 0x3E, 0xBA, 0xA2, 0x6D, 0xB4,
+ 0x05, 0xEA, 0xF4, 0xA7, 0xAB, 0xA2, 0x77, 0x83,
+ 0x7D, 0xE9, 0xA5, 0x15, 0x04, 0x70, 0x76, 0x24,
+ 0xEF, 0x2E, 0x1B, 0xBB, 0xCA, 0x29, 0x24, 0x11,
+ 0x16, 0x7F, 0x2E, 0x3D, 0x39, 0x0C, 0x0E, 0x51,
+ 0xF8, 0x4A, 0x2F, 0x13, 0x83, 0x90, 0xE3, 0x3F,
+ 0x85, 0x83, 0x5D, 0x38, 0xA9, 0x4D, 0xBB, 0xE7,
+ 0x1E, 0x6C, 0x82, 0x1E, 0x86, 0xB1, 0x1F, 0xFD,
+ 0x89, 0xEF, 0xF4, 0xBF, 0xE2, 0x08, 0xD6, 0x00,
+ 0x5D, 0x28, 0xF7, 0x04, 0xBA, 0xEA, 0xD1, 0xF2,
+ 0x5D, 0xE0, 0xEB, 0x24, 0x1B, 0x18, 0xFC, 0x7F,
+ 0xA0, 0xDD, 0xD9, 0x0D, 0xC1, 0x39, 0xBE, 0x7F,
+ 0xCB, 0xEB, 0x97, 0x30, 0xFA, 0xE4, 0xB5, 0xD1,
+ 0x72, 0x70, 0xCE, 0x4C, 0x67, 0x0C, 0x42, 0x57,
+ 0x0A, 0x9C, 0xF2, 0x5B, 0xC4, 0xFA, 0xE5, 0xCD,
+ 0x31, 0xE5, 0xD5, 0x5A, 0xD0, 0x22, 0x6A, 0x94,
+ 0xBE, 0x52, 0x94, 0x8C, 0x67, 0x02, 0xA9, 0x86,
+ 0xA0, 0xAD, 0xBF, 0xCD, 0x3A, 0xC4, 0x82, 0xBB,
+ 0x12, 0xAB, 0xBB, 0x79, 0xA2, 0xF6, 0x60, 0x28,
+ 0x42, 0x15, 0x3B, 0x2F, 0x82, 0xA3, 0xB3, 0xCD,
+ 0x16, 0x88, 0xE7, 0x4D, 0x36, 0x53, 0x4B, 0xFF,
+ 0x8C, 0x48, 0xD3, 0xC4, 0x51, 0xEB, 0x2C, 0x5F,
+ 0x98, 0xFE, 0xB9, 0xE7, 0x86, 0x4D, 0x60, 0xAF,
+ 0x96, 0xE8, 0x3B, 0x21, 0x62, 0x46, 0x74, 0x82,
+ 0xF0, 0x58, 0x63, 0x9C, 0x86, 0xA7, 0x85, 0xA9,
+ 0xA1, 0xD4, 0xB6, 0x9B, 0xC3, 0x0E, 0x77, 0xA6,
+ 0x4C, 0x3B, 0xBC, 0xD7, 0xDE, 0xB4, 0xE3, 0xD3,
+ 0x0F, 0x1A, 0x67, 0x21, 0x20, 0x3D, 0x87, 0xA8,
+ 0x8A, 0xB8, 0x5E, 0x02, 0x7A, 0x97, 0x42, 0xFC,
+ 0x68, 0x8F, 0x0A, 0xDF, 0x15, 0x72, 0x8E, 0x59,
+ 0x7E, 0x91, 0x0C, 0xFE, 0x5D, 0xF3, 0x3C, 0x56,
+ 0xA1, 0x36, 0xEF, 0x39, 0xC7, 0xCA, 0x5D, 0x65,
+ 0x0C, 0x2B, 0x9F, 0x90, 0x1C, 0x9B, 0x89, 0xE1,
+ 0xE0, 0x93, 0x54, 0x93, 0x61, 0xF3, 0x03, 0xBE,
+ 0x88, 0x39, 0xD1, 0x45, 0x4C, 0xCE, 0xB5, 0xFB,
+ 0xC4, 0x43, 0x5F, 0xA0, 0xDA, 0xB5, 0x8A, 0x8F,
+ 0xC2, 0x85, 0x36, 0x0E, 0xEA, 0x49, 0x1C, 0xA0,
+ 0x77, 0x96, 0x1C, 0x4A, 0xAA, 0x3E, 0x96, 0xDE,
+ 0x99, 0x71, 0xB9, 0x4F, 0xDF, 0xA5, 0x20, 0x7C,
+ 0xCF, 0x0D, 0x9D, 0xAB, 0x2C, 0x48, 0x96, 0xF0,
+ 0x7E, 0xB6, 0x77, 0x1A, 0x38, 0x3C, 0x65, 0x12,
+ 0xF4, 0x1E, 0xA2, 0x8D, 0xEE, 0xE4, 0x07, 0xFD,
+ 0xAE, 0x3C, 0x57, 0x4F, 0x5D, 0x41, 0x6A, 0x89,
+ 0x7A, 0x27, 0xEF, 0x7C, 0xF5, 0x96, 0xF0, 0x43,
+ 0x2D, 0x62, 0x4A, 0x2C, 0x4E, 0xAC, 0xE5, 0x2F,
+ 0x3C, 0xBF, 0x2C, 0x63, 0x31, 0xB8, 0x0C, 0x9C,
+ 0x91, 0x65, 0xBF, 0x13, 0x34, 0x24, 0x69, 0x32,
+ 0x02, 0x4E, 0xC0, 0xBE, 0x44, 0xB3, 0x21, 0x36,
+ 0xB4, 0xE4, 0x34, 0x27, 0x91, 0x35, 0x85, 0x03,
+ 0x64, 0xC7, 0x57, 0xF1, 0xDC, 0xFA, 0x63, 0x85,
+ 0xE2, 0x56, 0x33, 0x12, 0xC5, 0xF5, 0x53, 0xF0,
+ 0xC8, 0x44, 0xEA, 0xBB, 0x79, 0x11, 0xCE, 0xE7,
+ 0x60, 0xCA, 0xEB, 0x3E, 0x19, 0x3B, 0xF3, 0xA9,
+ 0xC3, 0x81, 0x14, 0x87, 0x23, 0x9A, 0xD2, 0xE0,
+ 0x14, 0x78, 0xF4, 0x6E, 0x41, 0x8A, 0x5D, 0xE5,
+ 0x6B, 0x7F, 0x17, 0x55, 0xBA, 0x68, 0xF9, 0xA3,
+ 0x74, 0x61, 0x3B, 0x5D, 0xE2, 0xED, 0x26, 0xC5,
+ 0x80, 0xC7, 0x72, 0xDB, 0xDB, 0xFA, 0xB1, 0xF7,
+ 0xE3, 0xF5, 0x7D, 0x94, 0xF8, 0x4E, 0x30, 0xDE,
+ 0xB2, 0x9D, 0x70, 0xA9, 0x1D, 0xF2, 0x88, 0xFC,
+ 0x43, 0xA2, 0x76, 0xDF, 0xED, 0x58, 0xE2, 0xB0,
+ 0xDB, 0x53, 0x83, 0xE5, 0x32, 0xB6, 0xEE, 0xDF,
+ 0xB3, 0x92, 0xE4, 0x3D, 0xC3, 0xDA, 0x72, 0x01,
+ 0xA0, 0x68, 0xF5, 0x23, 0x1E, 0xE5, 0x22, 0x09,
+ 0x8D, 0x68, 0x59, 0xB2, 0xD5, 0x64, 0x63, 0xA8,
+ 0x91, 0x7B, 0x3C, 0x25, 0x61, 0x65, 0x79, 0x66,
+ 0xDB, 0xC4, 0x78, 0x56, 0xB6, 0xFF, 0xC8, 0x2B,
+ 0xCC, 0x37, 0x9F, 0xFD, 0x08, 0xB2, 0x59, 0xF3,
+ 0xD9, 0xD7, 0x87, 0x3B, 0xA8, 0xFC, 0xBE, 0x4C,
+ 0x94, 0x13, 0xB6, 0x01, 0x15, 0x91, 0x60, 0x70,
+ 0x1D, 0xF0, 0x04, 0x70, 0xB1, 0x49, 0xBD, 0xF3,
+ 0x2F, 0x4D, 0x3C, 0xFC, 0xFB, 0x9D, 0xEB, 0xC7,
+ 0x72, 0x41, 0x71, 0x7D, 0x13, 0x06, 0x7A, 0xAE,
+ 0xD2, 0x3C, 0x7A, 0x26, 0x51, 0x18, 0x51, 0x69,
+ 0xF1, 0x26, 0x70, 0x61, 0xFB, 0x6B, 0x30, 0xE4,
+ 0xFE, 0xA7, 0x3F, 0x66, 0xF4, 0xF9, 0x27, 0x56,
+ 0xAC, 0x26, 0x23, 0x41, 0x8A, 0xF8, 0xB2, 0xA3,
+ 0x98, 0x71, 0x1B, 0x7C, 0x68, 0x07, 0xB4, 0x34,
+ 0x25, 0xE1, 0xD9, 0x9B, 0xFD, 0xCD, 0x5D, 0xF5,
+ 0x31, 0x95, 0x28, 0x79, 0x06, 0xA3, 0x32, 0xF5,
+ 0x99, 0x71, 0xA0, 0xC3, 0x43, 0x97, 0x5F, 0xC3,
+ 0x20, 0xAD, 0x13, 0x7C, 0x9E, 0x34, 0xCE, 0x7C,
+ 0xE8, 0x55, 0x20, 0xB2, 0x6C, 0xA1, 0x97, 0xA1,
+ 0xFA, 0x2D, 0xF2, 0xEC, 0xD4, 0xE3, 0xFA, 0x83,
+ 0x3B, 0x3B, 0xD2, 0xC2, 0x44, 0x82, 0x80, 0x42,
+ 0x52, 0xCF, 0x1D, 0xF6, 0xAD, 0xC6, 0x39, 0x8F,
+ 0x35, 0xE9, 0x8A, 0xB1, 0x87, 0x10, 0x40, 0x76,
+ 0x80, 0xC9, 0xC1, 0xDB, 0xAC, 0x8C, 0x7E, 0xDC,
+ 0x86, 0x46, 0xB9, 0x70, 0x82, 0xE2, 0xE1, 0x21,
+ 0xFA, 0xA7, 0xFA, 0xC2, 0x1E, 0x4A, 0x33, 0x83,
+ 0x84, 0xCB, 0x92, 0x20, 0x2C, 0x61, 0xBD, 0x12,
+ 0x6C, 0x5D, 0xDD, 0x45, 0x8B, 0x32, 0x7A, 0x18,
+ 0xBD, 0x71, 0x6F, 0x14, 0x2C, 0xA5, 0xCD, 0xB3,
+ 0xE4, 0x9D, 0x7E, 0xB8, 0xD9, 0x62, 0xB5, 0xB8,
+ 0x5A, 0x88, 0xF7, 0x99, 0xB6, 0x9A, 0x6A, 0x66,
+ 0xC7, 0xBD, 0x62, 0x9F, 0x56, 0xB4, 0x3C, 0x02,
+ 0x90, 0x62, 0x9B, 0x5E, 0x27, 0x4C, 0xDE, 0xC7,
+ 0xA0, 0x72, 0x29, 0xE7, 0x93, 0x9A, 0x77, 0xD3,
+ 0x2E, 0x8E, 0xF7, 0x30, 0xFC, 0xCE, 0xAD, 0x9C,
+ 0x4E, 0x06, 0x77, 0xA8, 0x3A, 0x03, 0x30, 0xAB,
+ 0x76, 0x5D, 0x33, 0x6D, 0xD2, 0xAA, 0x15, 0x5D,
+ 0xCD, 0x2A, 0xC7, 0xF3, 0x15, 0x29, 0x77, 0x4F,
+ 0x49, 0x36, 0xB0, 0x5D, 0x0B, 0x14, 0xB4, 0x8F,
+ 0xAA, 0x1E, 0x8C, 0xD4, 0x50, 0x56, 0xE5, 0x6C,
+ 0x13, 0x9B, 0x17, 0xF8, 0x90, 0x71, 0x5A, 0xD6,
+ 0x3D, 0x6C, 0x4A, 0x9F, 0x2D, 0x97, 0x6C, 0x8B,
+ 0x63, 0x5B, 0xDF, 0xE5, 0x86, 0x02, 0x81, 0x6F,
+ 0x61, 0x2C, 0x6E, 0x4B, 0x22, 0x53, 0x67, 0xCB,
+ 0x9A, 0x7B, 0xB7, 0x9C, 0x01, 0x8F, 0x1B, 0x8C,
+ 0x53, 0x15, 0x18, 0x0A, 0xAD, 0xBE, 0x3A, 0xB7,
+ 0x5A, 0xC3, 0x56, 0x20, 0x6F, 0xE2, 0x7C, 0x12,
+ 0xDF, 0x3B, 0x56, 0x97, 0x84, 0xE3, 0xA5, 0x38,
+ 0xFB, 0x05, 0x24, 0x18, 0x26, 0x6E, 0x72, 0xDB,
+ 0x40, 0x0D, 0x6F, 0x32, 0xC4, 0x29, 0x7F, 0x34,
+ 0xF9, 0xF1, 0xAF, 0x18, 0x6C, 0x37, 0x65, 0x65,
+ 0x5F, 0x11, 0xB3, 0xE5, 0xA3, 0xC8, 0x04, 0x9B,
+ 0x7D, 0xF1, 0x40, 0x11, 0xFF, 0x21, 0x5F, 0xBF,
+ 0x17, 0xBF, 0x89, 0xEE, 0x97, 0x6C, 0xF0, 0xDB,
+ 0xAB, 0x62, 0x70, 0x10, 0x4E, 0x7E, 0x31, 0x9D,
+ 0x1F, 0x64, 0xC5, 0x9E, 0x20, 0x9E, 0x35, 0x82,
+};
+
+uint8_t mldsa_44_pubkey_noseed[] = {
+ 0xBA, 0x71, 0xF9, 0xF6, 0x4E, 0x11, 0xBA, 0xEB,
+ 0x58, 0xFA, 0x9C, 0x6F, 0xBB, 0x6E, 0x14, 0xE6,
+ 0x1F, 0x18, 0x64, 0x3D, 0xAB, 0x49, 0x5B, 0x47,
+ 0x53, 0x9A, 0x91, 0x66, 0xCA, 0x01, 0x98, 0x13,
+ 0x1C, 0x44, 0xF8, 0x26, 0xBB, 0xD5, 0x6E, 0x34,
+ 0xE5, 0x5D, 0xB5, 0xE5, 0xE2, 0xD7, 0x33, 0x48,
+ 0x5E, 0x39, 0xEA, 0x26, 0x0F, 0xC6, 0x00, 0x0C,
+ 0x5E, 0xA4, 0xBA, 0x80, 0xD3, 0x45, 0x5C, 0xDE,
+ 0x53, 0xB4, 0x6F, 0x34, 0x48, 0x2A, 0xED, 0xFD,
+ 0x54, 0x50, 0xFC, 0x2E, 0x1B, 0xA4, 0xF2, 0x5D,
+ 0x15, 0xF9, 0xC1, 0x44, 0x24, 0x2F, 0xB3, 0x9B,
+ 0xB5, 0x22, 0x87, 0x18, 0x90, 0x30, 0xC5, 0x04,
+ 0x98, 0xE1, 0x71, 0x7B, 0x7C, 0x75, 0x8B, 0x19,
+ 0x0A, 0x67, 0x48, 0xEA, 0x9A, 0xA3, 0xF7, 0xAC,
+ 0xAA, 0xF2, 0xC7, 0xCB, 0x52, 0x6E, 0xD7, 0x17,
+ 0xC9, 0xF7, 0x9A, 0xEB, 0x84, 0x21, 0x4F, 0xA5,
+ 0xCD, 0x8D, 0xED, 0x92, 0xA0, 0xC3, 0xFA, 0x15,
+ 0x58, 0x81, 0x0F, 0x12, 0xC7, 0x05, 0x0A, 0x36,
+ 0x77, 0x08, 0xD1, 0x96, 0xCD, 0x24, 0xE5, 0xAF,
+ 0x97, 0x49, 0x04, 0xAE, 0xD8, 0xE4, 0xCE, 0x88,
+ 0x72, 0xE8, 0x69, 0x6B, 0x0B, 0x7B, 0xCA, 0x50,
+ 0xE4, 0x52, 0xCD, 0x7D, 0x30, 0xEA, 0x9A, 0x4A,
+ 0xDA, 0xC0, 0x31, 0x1D, 0x67, 0x2C, 0x6B, 0xDE,
+ 0x84, 0x96, 0x24, 0x0B, 0x07, 0x43, 0x14, 0x63,
+ 0x70, 0x88, 0x95, 0xCD, 0x9B, 0xAF, 0xC3, 0x16,
+ 0x32, 0xD7, 0x39, 0x76, 0x49, 0x38, 0x8F, 0xDA,
+ 0xFC, 0xBF, 0x7D, 0x30, 0x5A, 0x3D, 0xE9, 0xA4,
+ 0x95, 0xEC, 0xA7, 0x43, 0x3A, 0x8F, 0x83, 0xBA,
+ 0x0F, 0x0B, 0x25, 0xC4, 0x13, 0xC6, 0xE3, 0x9C,
+ 0x96, 0xEB, 0x7D, 0x69, 0x1B, 0x34, 0xD3, 0x7C,
+ 0xE3, 0x7F, 0x1E, 0xEA, 0xD1, 0xCF, 0x21, 0x7E,
+ 0x25, 0xEF, 0x34, 0xEE, 0xCF, 0x3F, 0x7C, 0x60,
+ 0xF8, 0x4B, 0x8E, 0xDF, 0xDD, 0xE8, 0x40, 0x5D,
+ 0x4F, 0x83, 0x25, 0x76, 0xC6, 0x1E, 0xF9, 0x8E,
+ 0x0A, 0x2F, 0x28, 0xDA, 0x18, 0x77, 0x00, 0x95,
+ 0x39, 0x24, 0xF6, 0x86, 0xB9, 0x46, 0x14, 0x70,
+ 0x5B, 0xCF, 0x53, 0xD3, 0x3F, 0xED, 0xD4, 0x34,
+ 0x8E, 0xDD, 0xDB, 0xDF, 0x28, 0xB5, 0x06, 0x5E,
+ 0x1F, 0x20, 0x77, 0x50, 0x43, 0xE8, 0x5C, 0xF9,
+ 0x31, 0xF8, 0x29, 0x17, 0x93, 0x63, 0xA1, 0xA7,
+ 0xE7, 0x40, 0x4A, 0x83, 0x8E, 0xC0, 0x00, 0x86,
+ 0xB0, 0x97, 0x63, 0x86, 0xFE, 0x63, 0x7C, 0x98,
+ 0x24, 0x47, 0x57, 0xE3, 0xF7, 0x69, 0xDD, 0xD4,
+ 0x46, 0x74, 0x71, 0xBF, 0xAD, 0x67, 0x0F, 0x9A,
+ 0x05, 0xF8, 0x24, 0x6E, 0xE5, 0x0A, 0x7B, 0x1E,
+ 0xAF, 0x87, 0xFC, 0x40, 0x69, 0xC3, 0xAE, 0x2A,
+ 0xA2, 0x03, 0x32, 0x58, 0x11, 0x77, 0x92, 0xF0,
+ 0xBC, 0xD4, 0x9E, 0x08, 0x3F, 0xD1, 0xBC, 0x74,
+ 0x96, 0xAB, 0xFF, 0x29, 0xCC, 0x94, 0xE4, 0x86,
+ 0x8B, 0x21, 0x21, 0x4E, 0xD3, 0x16, 0x52, 0x53,
+ 0x99, 0xA6, 0x10, 0xFB, 0xDD, 0x4A, 0x80, 0xE7,
+ 0xC8, 0x07, 0x15, 0xF2, 0x95, 0x78, 0xE2, 0xA8,
+ 0x4B, 0xB4, 0x0B, 0xDD, 0xDB, 0xD9, 0xF4, 0x7A,
+ 0x11, 0xB6, 0xE7, 0xDA, 0x11, 0x8A, 0x1B, 0x65,
+ 0x8D, 0x35, 0x9E, 0x8A, 0xEF, 0x55, 0xEB, 0x46,
+ 0xB5, 0x37, 0x6B, 0x5B, 0x65, 0x59, 0x79, 0x98,
+ 0x4A, 0x92, 0x2B, 0xEE, 0xBF, 0xC5, 0x9B, 0xCD,
+ 0x60, 0x0D, 0x53, 0x09, 0xDC, 0xCD, 0x72, 0xDB,
+ 0xF0, 0x78, 0x7D, 0xB8, 0xBA, 0x75, 0x7B, 0x53,
+ 0x7C, 0x1E, 0xAF, 0xD5, 0xC0, 0xF5, 0x0E, 0xA4,
+ 0xBC, 0x95, 0x83, 0x54, 0x9E, 0x28, 0x29, 0xA4,
+ 0x2C, 0x28, 0xCA, 0xC2, 0x48, 0xC9, 0x6D, 0x78,
+ 0x12, 0x4C, 0x47, 0x15, 0x9B, 0x18, 0xAE, 0xDD,
+ 0x75, 0x4A, 0xBA, 0x17, 0xB1, 0x9D, 0x43, 0x0F,
+ 0xB7, 0x8F, 0x63, 0x3E, 0xA9, 0xD2, 0x6F, 0x54,
+ 0xA9, 0xBD, 0x50, 0xF8, 0xD8, 0xF6, 0xB7, 0x35,
+ 0x94, 0xF8, 0x28, 0x97, 0x6E, 0x7E, 0xA0, 0x9C,
+ 0x53, 0xBB, 0xB9, 0xF1, 0x1A, 0x56, 0xC9, 0x50,
+ 0x7F, 0xB8, 0x9B, 0x9A, 0x5E, 0xBC, 0x03, 0x7A,
+ 0x37, 0x26, 0x7A, 0x95, 0xF8, 0x5B, 0x8D, 0x64,
+ 0xCA, 0x97, 0x19, 0x2B, 0x10, 0xA6, 0x6F, 0x41,
+ 0x7B, 0x3F, 0x61, 0xFE, 0x9C, 0xA5, 0x71, 0x30,
+ 0xA4, 0x8F, 0xD9, 0x25, 0xEA, 0xE2, 0xAB, 0x55,
+ 0x02, 0xD5, 0x71, 0xC8, 0xA5, 0x19, 0x03, 0xC1,
+ 0xD3, 0x98, 0xF4, 0xC1, 0xF7, 0x6A, 0x7E, 0x11,
+ 0x74, 0x39, 0x76, 0xAF, 0xDB, 0xC6, 0x97, 0xF2,
+ 0x30, 0x94, 0xA3, 0xCD, 0x76, 0x1F, 0xF9, 0x68,
+ 0x5D, 0xE3, 0x2E, 0x09, 0xFB, 0x3C, 0x28, 0xAD,
+ 0xD4, 0x53, 0x49, 0x03, 0x00, 0xBC, 0x7C, 0x89,
+ 0xDC, 0x01, 0x78, 0x00, 0x96, 0x07, 0x17, 0x22,
+ 0x94, 0x57, 0x75, 0xF2, 0x64, 0xE1, 0xB0, 0x62,
+ 0x3B, 0xCF, 0x46, 0x19, 0xC7, 0x12, 0xC8, 0x38,
+ 0x76, 0x12, 0x05, 0xD8, 0x76, 0x91, 0xB7, 0x5E,
+ 0xF3, 0x60, 0x19, 0x6C, 0xBB, 0x9E, 0x9B, 0x92,
+ 0xA0, 0xD4, 0xC4, 0xED, 0x62, 0x32, 0x6E, 0x50,
+ 0x24, 0xD7, 0x75, 0x10, 0xB8, 0xEE, 0x2C, 0x74,
+ 0x26, 0xCC, 0x22, 0xEA, 0xE2, 0x09, 0xDC, 0x9F,
+ 0x13, 0xBD, 0xE6, 0xBF, 0x08, 0xF5, 0xE7, 0x18,
+ 0x1B, 0xD3, 0xB4, 0x59, 0x45, 0x0B, 0x45, 0x1A,
+ 0x51, 0x53, 0x9A, 0x71, 0x5C, 0x21, 0xD6, 0x7D,
+ 0xD3, 0x30, 0xEB, 0x59, 0x70, 0xDB, 0x00, 0xD9,
+ 0xED, 0xBF, 0xB2, 0x82, 0x2B, 0x03, 0x6F, 0xA1,
+ 0x3B, 0xAF, 0xEB, 0x86, 0xD8, 0xDC, 0x78, 0x86,
+ 0x6E, 0x3F, 0x8D, 0x43, 0xE5, 0x3D, 0x78, 0xCC,
+ 0xA5, 0x59, 0x5A, 0x6F, 0xAF, 0x88, 0x6B, 0x5D,
+ 0xC1, 0x12, 0xF1, 0xCF, 0x4A, 0xDC, 0xFA, 0x87,
+ 0x58, 0x00, 0xD9, 0x0B, 0x48, 0x88, 0x3A, 0xF9,
+ 0x73, 0x16, 0xFE, 0x15, 0x06, 0x87, 0x3F, 0xC1,
+ 0x57, 0xE5, 0x70, 0xEA, 0xCB, 0xFD, 0x22, 0x28,
+ 0x68, 0xD1, 0x42, 0x34, 0x10, 0x19, 0x66, 0xAF,
+ 0xB6, 0xBF, 0x99, 0x40, 0x82, 0x92, 0x53, 0xA9,
+ 0x53, 0xAD, 0xA8, 0x9F, 0xC7, 0x56, 0xB6, 0xA8,
+ 0x49, 0xF7, 0x0A, 0xCB, 0x98, 0x38, 0xE6, 0x9F,
+ 0xAA, 0x50, 0xBB, 0xA7, 0x5E, 0x3E, 0x89, 0xC2,
+ 0xAD, 0xB5, 0x7E, 0x86, 0xD0, 0x88, 0xAB, 0x9B,
+ 0x04, 0xA2, 0x8E, 0x67, 0x07, 0x09, 0x17, 0x22,
+ 0x43, 0xEC, 0x5E, 0x00, 0x08, 0xA5, 0xCE, 0xAF,
+ 0x3F, 0x87, 0x22, 0xF4, 0x87, 0x30, 0x25, 0x96,
+ 0xFF, 0xD7, 0x55, 0xAD, 0x1B, 0x82, 0xA4, 0x9C,
+ 0x34, 0xB3, 0x46, 0x95, 0x15, 0xB4, 0x6A, 0xA2,
+ 0x90, 0xCD, 0x86, 0xEE, 0x38, 0xEA, 0x7A, 0x9B,
+ 0xE3, 0xF1, 0x03, 0x61, 0x03, 0x35, 0xB5, 0x31,
+ 0xCC, 0xA3, 0x33, 0xDD, 0xFE, 0x32, 0xB1, 0x45,
+ 0x10, 0xF4, 0xB0, 0x7E, 0xF9, 0x5F, 0xC6, 0x68,
+ 0x4E, 0x8C, 0x45, 0x4A, 0x92, 0xC1, 0x0D, 0xBB,
+ 0x5D, 0x59, 0xC7, 0xA7, 0xC6, 0x3F, 0xB3, 0x05,
+ 0xFE, 0x88, 0x19, 0x67, 0xD9, 0x9E, 0x66, 0x9E,
+ 0xB6, 0x32, 0x84, 0x05, 0x82, 0x56, 0x0B, 0xB4,
+ 0x03, 0x43, 0x1D, 0x40, 0xF7, 0x5A, 0x49, 0x54,
+ 0x90, 0x84, 0x82, 0x27, 0x82, 0x92, 0x82, 0x1F,
+ 0x4E, 0xA9, 0x1E, 0x42, 0xE7, 0x8F, 0xA4, 0x8C,
+ 0xAE, 0xE3, 0xC8, 0x36, 0x14, 0x6D, 0xCF, 0xD7,
+ 0x38, 0xD1, 0x17, 0xE9, 0x2E, 0x9A, 0x15, 0x13,
+ 0x7D, 0x28, 0xE8, 0xE6, 0xA4, 0xB4, 0x62, 0x26,
+ 0x50, 0xCB, 0x41, 0x35, 0x04, 0xCB, 0x3A, 0x33,
+ 0x5D, 0x44, 0xBE, 0xEC, 0x57, 0x46, 0xC1, 0xC2,
+ 0x94, 0xB1, 0xE8, 0xCB, 0x99, 0xCB, 0x60, 0x8D,
+ 0x92, 0x8F, 0x8C, 0xE3, 0x56, 0x36, 0x32, 0xC5,
+ 0x21, 0xF2, 0x3D, 0x13, 0xC6, 0x1A, 0x8F, 0x61,
+ 0xC0, 0x1D, 0xF8, 0xC9, 0x6C, 0x73, 0x60, 0xDB,
+ 0x4F, 0x3C, 0x68, 0xAA, 0x5D, 0x2F, 0xDD, 0x34,
+ 0x2A, 0x62, 0xFF, 0x34, 0x59, 0xC1, 0x16, 0x38,
+ 0x94, 0x21, 0xAB, 0x43, 0xE8, 0x58, 0x4C, 0x45,
+ 0x88, 0x2B, 0x50, 0xE6, 0xE4, 0xE9, 0x6D, 0xB6,
+ 0xF0, 0xB8, 0xFD, 0xE8, 0x90, 0xD5, 0xDB, 0xFA,
+ 0xDC, 0xD8, 0x86, 0x90, 0xB4, 0x49, 0xE6, 0x42,
+ 0x40, 0xDD, 0xB2, 0x02, 0x37, 0x47, 0xF3, 0x08,
+ 0x36, 0x3E, 0x30, 0x1A, 0xA7, 0x77, 0x57, 0x16,
+ 0x9F, 0xC6, 0x15, 0x06, 0x28, 0xD5, 0x92, 0x0B,
+ 0x5A, 0xA1, 0xAB, 0x1C, 0x8C, 0xBF, 0x44, 0xCB,
+ 0x00, 0xE0, 0x25, 0xD7, 0x87, 0x9D, 0x72, 0xB4,
+ 0x79, 0xE3, 0xAF, 0x53, 0x11, 0xC7, 0x85, 0x72,
+ 0x55, 0x90, 0xDA, 0x9C, 0x89, 0xB9, 0xFC, 0x3B,
+ 0x84, 0x50, 0x76, 0x95, 0x54, 0xEB, 0x44, 0xD2,
+ 0x03, 0xEB, 0xA2, 0xBB, 0xAE, 0xF9, 0xCA, 0xD2,
+ 0x23, 0x70, 0x11, 0xC2, 0xEA, 0x44, 0xEF, 0xF0,
+ 0x0F, 0x29, 0x9A, 0x48, 0xFF, 0xE2, 0x8C, 0xA9,
+ 0x3D, 0xDF, 0x85, 0xF7, 0x66, 0x08, 0x24, 0x2E,
+ 0xF8, 0xD6, 0xCC, 0x24, 0x61, 0x0A, 0x1E, 0x20,
+ 0x78, 0xFC, 0xAC, 0x4F, 0x93, 0x85, 0xC3, 0x14,
+ 0x90, 0x5E, 0xCA, 0xA8, 0x2E, 0x55, 0x39, 0x16,
+ 0xD9, 0x4D, 0x1A, 0x7C, 0x1E, 0xC6, 0x52, 0xAA,
+ 0x08, 0x89, 0x70, 0x83, 0xDA, 0xA2, 0xEB, 0xB1,
+ 0x77, 0x5F, 0xBC, 0x47, 0x1A, 0xE2, 0x77, 0x77,
+ 0xD7, 0x90, 0x4E, 0xA9, 0xF1, 0xB9, 0x2B, 0xCA,
+ 0xC3, 0xD8, 0xA3, 0x15, 0x84, 0x26, 0x08, 0x7B,
+ 0x64, 0x5B, 0x11, 0x08, 0xF0, 0xD6, 0x5F, 0xEC,
+ 0x93, 0x78, 0x9C, 0x05, 0x37, 0x43, 0xCA, 0x14,
+ 0xFD, 0x63, 0xD0, 0x5E, 0x98, 0xB6, 0x52, 0xDF,
+ 0x2B, 0x9C, 0x2F, 0xF9, 0xCE, 0x05, 0xF1, 0x94,
+ 0x07, 0x03, 0xFF, 0xB2, 0x73, 0xF8, 0x0E, 0x0E,
+ 0x27, 0x32, 0xEC, 0xA9, 0x96, 0x0D, 0x98, 0x1B,
+ 0x4C, 0xFD, 0x3B, 0x7B, 0xB8, 0x04, 0x5B, 0x3C,
+ 0x38, 0x30, 0x54, 0x6B, 0x9D, 0xD8, 0xDB, 0x0D,
+};
+
+uint8_t mldsa_44_extmu[] = {
+ 0x1B, 0x2A, 0xA5, 0xD5, 0xE3, 0xF3, 0xC1, 0xD4,
+ 0x8D, 0x7C, 0xE6, 0x8F, 0xE9, 0xE3, 0xD1, 0xF5,
+ 0x2E, 0x4B, 0x7C, 0xD6, 0x2A, 0xF4, 0x2E, 0xE5,
+ 0x8F, 0xC3, 0xB1, 0xA4, 0xB9, 0xD1, 0x2C, 0x8B,
+ 0x9E, 0x4D, 0x7A, 0xA9, 0x7C, 0xD3, 0xE6, 0x5A,
+ 0xF4, 0x1B, 0x2A, 0xA5, 0xD5, 0xE3, 0xF3, 0xC1,
+ 0xD4, 0x8D, 0x7C, 0xE6, 0x8F, 0xE9, 0xE3, 0xD1,
+ 0xF5, 0x2E, 0x4B, 0x7C, 0xD6, 0x2A, 0xF4, 0x2E,
+};
+
struct
cperf_rsa_test_data rsa_qt_perf_data[4] = {
{
@@ -4292,6 +5910,49 @@ struct cperf_mlkem_test_data mlkem_decap_perf_data[] = {
}
};
+struct cperf_mldsa_test_data mldsa_sign_perf_data[] = {
+ {
+ .name = "mldsa_44_sign (deterministic)",
+ .type = RTE_CRYPTO_ML_DSA_44,
+ .privkey = {
+ .data = mldsa_44_privkey_noseed,
+ .length = sizeof(mldsa_44_privkey_noseed),
+ },
+ .pubkey = {
+ .data = mldsa_44_pubkey_noseed,
+ .length = sizeof(mldsa_44_pubkey_noseed),
+ },
+ .message = {
+ .data = mldsa_44_message,
+ .length = sizeof(mldsa_44_message),
+ },
+ .sign = {
+ .data = mldsa_44_sign_dtrm,
+ .length = sizeof(mldsa_44_sign_dtrm),
+ },
+ .sign_deterministic = true,
+ },
+};
+
+struct cperf_mldsa_test_data mldsa_verify_perf_data[] = {
+ {
+ .name = "mldsa_44_verify",
+ .type = RTE_CRYPTO_ML_DSA_44,
+ .pubkey = {
+ .data = mldsa_44_pubkey_noseed,
+ .length = sizeof(mldsa_44_pubkey_noseed),
+ },
+ .message = {
+ .data = mldsa_44_message,
+ .length = sizeof(mldsa_44_message),
+ },
+ .sign = {
+ .data = mldsa_44_sign_dtrm,
+ .length = sizeof(mldsa_44_sign_dtrm),
+ },
+ },
+};
+
struct cperf_test_vector*
cperf_test_vector_get_dummy(struct cperf_options *options)
{
diff --git a/app/test-crypto-perf/cperf_test_vectors.h b/app/test-crypto-perf/cperf_test_vectors.h
index e498196ae3..6b4b4f22b4 100644
--- a/app/test-crypto-perf/cperf_test_vectors.h
+++ b/app/test-crypto-perf/cperf_test_vectors.h
@@ -188,6 +188,41 @@ struct cperf_sm2_test_data {
int curve;
};
+struct cperf_mldsa_test_data {
+ const char *name;
+ enum rte_crypto_ml_dsa_type type;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } privkey;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } pubkey;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } message;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } sign;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } ctx;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } seed;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } mu;
+ enum rte_crypto_auth_algorithm hash;
+ bool sign_deterministic;
+};
+
struct cperf_test_vector*
cperf_test_vector_get_dummy(struct cperf_options *options);
@@ -215,5 +250,7 @@ extern struct cperf_rsa_test_data rsa_qt_perf_data[4];
extern struct cperf_rsa_plaintext rsa_plaintext;
extern struct cperf_mlkem_test_data mlkem_encap_perf_data[];
extern struct cperf_mlkem_test_data mlkem_decap_perf_data[];
+extern struct cperf_mldsa_test_data mldsa_sign_perf_data[];
+extern struct cperf_mldsa_test_data mldsa_verify_perf_data[];
#endif
diff --git a/app/test-crypto-perf/main.c b/app/test-crypto-perf/main.c
index 2e228201ab..c4d844d225 100644
--- a/app/test-crypto-perf/main.c
+++ b/app/test-crypto-perf/main.c
@@ -55,6 +55,7 @@ const char *cperf_op_type_strs[] = {
[CPERF_ASYM_SM2] = "sm2",
[CPERF_TLS] = "tls-record",
[CPERF_ASYM_MLKEM512] = "mlkem_512",
+ [CPERF_ASYM_MLDSA44] = "mldsa_44",
};
const char *cperf_rsa_priv_keytype_strs[] = {
@@ -247,6 +248,7 @@ cperf_initialize_cryptodev(struct cperf_options *opts, uint8_t *enabled_cdevs)
case CPERF_ASYM_SM2:
case CPERF_ASYM_RSA:
case CPERF_ASYM_MLKEM512:
+ case CPERF_ASYM_MLDSA44:
case CPERF_ASYM_MODEX:
conf.ff_disable |= (RTE_CRYPTODEV_FF_SECURITY |
RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO);
@@ -516,6 +518,21 @@ cperf_verify_devices_capabilities(struct cperf_options *opts,
opts->sm2_data->sign_s.length = sm2_perf_data.sign_s.length;
}
}
+ if (opts->op_type == CPERF_ASYM_MLDSA44) {
+ asym_cap_idx.type = RTE_CRYPTO_ASYM_XFORM_ML_DSA;
+ asym_capability = rte_cryptodev_asym_capability_get(cdev_id, &asym_cap_idx);
+ if (asym_capability == NULL)
+ return -1;
+
+ if (opts->asym_op_type == RTE_CRYPTO_ASYM_OP_SIGN)
+ opts->mldsa_data = &mldsa_sign_perf_data[0];
+ else if (opts->asym_op_type == RTE_CRYPTO_ASYM_OP_VERIFY)
+ opts->mldsa_data = &mldsa_verify_perf_data[0];
+ else {
+ RTE_LOG(ERR, USER1, "Unsupported MLDSA operation type\n");
+ return -ENOTSUP;
+ }
+ }
if (opts->op_type == CPERF_ASYM_MLKEM512) {
asym_cap_idx.type = RTE_CRYPTO_ASYM_XFORM_ML_KEM;
--
2.43.0
^ permalink raw reply related
* [PATCH] app/crypto-perf: support ML KEM
From: Pratik Senapati @ 2026-05-28 7:58 UTC (permalink / raw)
To: dev; +Cc: gakhil, anoobj, gmuthukrishn, kai.ji
Add ML-KEM512 support to test-crypto-perf.
Signed-off-by: Pratik Senapati <psenapati@marvell.com>
---
app/test-crypto-perf/cperf_ops.c | 61 +++
app/test-crypto-perf/cperf_options.h | 2 +
app/test-crypto-perf/cperf_options_parsing.c | 20 +
app/test-crypto-perf/cperf_test_common.c | 3 +-
app/test-crypto-perf/cperf_test_vectors.c | 470 +++++++++++++++++++
app/test-crypto-perf/cperf_test_vectors.h | 27 ++
app/test-crypto-perf/main.c | 21 +-
7 files changed, 602 insertions(+), 2 deletions(-)
diff --git a/app/test-crypto-perf/cperf_ops.c b/app/test-crypto-perf/cperf_ops.c
index 9297b4a3d3..806265f7bf 100644
--- a/app/test-crypto-perf/cperf_ops.c
+++ b/app/test-crypto-perf/cperf_ops.c
@@ -208,6 +208,49 @@ cperf_set_ops_asym_sm2(struct rte_crypto_op **ops,
}
}
+static void
+cperf_set_ops_asym_mlkem(struct rte_crypto_op **ops,
+ uint32_t src_buf_offset __rte_unused,
+ uint32_t dst_buf_offset __rte_unused, uint16_t nb_ops,
+ void *sess,
+ const struct cperf_options *options,
+ const struct cperf_test_vector *test_vector __rte_unused,
+ uint16_t iv_offset __rte_unused,
+ uint32_t *imix_idx __rte_unused,
+ uint64_t *tsc_start __rte_unused)
+{
+ uint16_t i;
+
+ for (i = 0; i < nb_ops; i++) {
+ struct rte_crypto_asym_op *asym_op = ops[i]->asym;
+
+ ops[i]->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
+ rte_crypto_op_attach_asym_session(ops[i], sess);
+
+ if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_ENCRYPT) {
+ asym_op->mlkem.op = RTE_CRYPTO_ML_KEM_OP_ENCAP;
+ asym_op->mlkem.encap.ek.data = options->mlkem_data->ek.data;
+ asym_op->mlkem.encap.ek.length = options->mlkem_data->ek.length;
+ asym_op->mlkem.encap.cipher.data = options->mlkem_data->cipher.data;
+ asym_op->mlkem.encap.cipher.length = options->mlkem_data->cipher.length;
+ asym_op->mlkem.encap.sk.data = options->mlkem_data->sk.data;
+ asym_op->mlkem.encap.sk.length = options->mlkem_data->sk.length;
+ asym_op->mlkem.encap.message.data = options->mlkem_data->message.data;
+ asym_op->mlkem.encap.message.length = options->mlkem_data->message.length;
+ } else if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_DECRYPT) {
+ asym_op->mlkem.op = RTE_CRYPTO_ML_KEM_OP_DECAP;
+ asym_op->mlkem.decap.dk.data = options->mlkem_data->dk.data;
+ asym_op->mlkem.decap.dk.length = options->mlkem_data->dk.length;
+ asym_op->mlkem.decap.cipher.data = options->mlkem_data->cipher.data;
+ asym_op->mlkem.decap.cipher.length = options->mlkem_data->cipher.length;
+ asym_op->mlkem.decap.sk.data = options->mlkem_data->sk.data;
+ asym_op->mlkem.decap.sk.length = options->mlkem_data->sk.length;
+ } else {
+ rte_panic("Unsupported ML-KEM operation type %d\n", options->asym_op_type);
+ }
+ }
+}
+
#ifdef RTE_LIB_SECURITY
static void
@@ -1226,6 +1269,21 @@ cperf_create_session(struct rte_mempool *sess_mp,
return asym_sess;
}
+
+ if (options->op_type == CPERF_ASYM_MLKEM512) {
+ xform.next = NULL;
+ xform.xform_type = RTE_CRYPTO_ASYM_XFORM_ML_KEM;
+ xform.mlkem.type = RTE_CRYPTO_ML_KEM_512;
+
+ ret = rte_cryptodev_asym_session_create(dev_id, &xform, sess_mp, &asym_sess);
+ if (ret < 0) {
+ RTE_LOG(ERR, USER1, "ML-KEM Asym session create failed\n");
+ return NULL;
+ }
+ return asym_sess;
+ }
+
+
#ifdef RTE_LIB_SECURITY
/*
* security only
@@ -1542,6 +1600,9 @@ cperf_get_op_functions(const struct cperf_options *options,
case CPERF_ASYM_SM2:
op_fns->populate_ops = cperf_set_ops_asym_sm2;
break;
+ case CPERF_ASYM_MLKEM512:
+ op_fns->populate_ops = cperf_set_ops_asym_mlkem;
+ break;
#ifdef RTE_LIB_SECURITY
case CPERF_PDCP:
case CPERF_DOCSIS:
diff --git a/app/test-crypto-perf/cperf_options.h b/app/test-crypto-perf/cperf_options.h
index 428e8cb4f1..98b8eeec3e 100644
--- a/app/test-crypto-perf/cperf_options.h
+++ b/app/test-crypto-perf/cperf_options.h
@@ -101,6 +101,7 @@ enum cperf_op_type {
CPERF_ASYM_SECP521R1,
CPERF_ASYM_ED25519,
CPERF_ASYM_SM2,
+ CPERF_ASYM_MLKEM512,
CPERF_TLS,
};
@@ -187,6 +188,7 @@ struct cperf_options {
struct cperf_ecdsa_test_data *secp521r1_data;
struct cperf_eddsa_test_data *eddsa_data;
struct cperf_sm2_test_data *sm2_data;
+ struct cperf_mlkem_test_data *mlkem_data;
enum rte_crypto_asym_op_type asym_op_type;
enum rte_crypto_auth_algorithm asym_hash_alg;
struct cperf_rsa_test_data *rsa_data;
diff --git a/app/test-crypto-perf/cperf_options_parsing.c b/app/test-crypto-perf/cperf_options_parsing.c
index 14e731586b..34afa938c8 100644
--- a/app/test-crypto-perf/cperf_options_parsing.c
+++ b/app/test-crypto-perf/cperf_options_parsing.c
@@ -41,6 +41,7 @@ usage(char *progname)
" --optype cipher-only / auth-only / cipher-then-auth / auth-then-cipher /\n"
" aead / pdcp / docsis / ipsec / modex / rsa / secp192r1 /\n"
" secp224r1 / secp256r1 / secp384r1 / secp521r1 / eddsa / sm2 /\n"
+ " mlkem_512 /\n"
" tls-record : set operation type\n"
" --sessionless: enable session-less crypto operations\n"
" --shared-session: share 1 session across all queue pairs on crypto device\n"
@@ -559,6 +560,10 @@ parse_op_type(struct cperf_options *opts, const char *arg)
cperf_op_type_strs[CPERF_ASYM_SM2],
CPERF_ASYM_SM2
},
+ {
+ cperf_op_type_strs[CPERF_ASYM_MLKEM512],
+ CPERF_ASYM_MLKEM512
+ },
{
cperf_op_type_strs[CPERF_TLS],
CPERF_TLS
@@ -1174,6 +1179,7 @@ cperf_options_default(struct cperf_options *opts)
opts->secp521r1_data = &secp521r1_perf_data;
opts->eddsa_data = &ed25519_perf_data;
opts->sm2_data = &sm2_perf_data;
+ opts->mlkem_data = &mlkem_encap_perf_data[0];
opts->asym_op_type = RTE_CRYPTO_ASYM_OP_ENCRYPT;
}
@@ -1669,6 +1675,18 @@ cperf_options_check(struct cperf_options *options)
}
}
+ if (options->op_type == CPERF_ASYM_MLKEM512) {
+ if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_ENCRYPT)
+ options->mlkem_data = &mlkem_encap_perf_data[0];
+ else if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_DECRYPT)
+ options->mlkem_data = &mlkem_decap_perf_data[0];
+ else {
+ RTE_LOG(ERR, USER1,
+ "ML-KEM operations only support encrypt (encapsulate) and decrypt (decapsulate)\n");
+ return -EINVAL;
+ }
+ }
+
#ifdef RTE_LIB_SECURITY
if (options->op_type == CPERF_DOCSIS) {
if (check_docsis_buffer_length(options) < 0)
@@ -1741,6 +1759,8 @@ cperf_options_dump(struct cperf_options *opts)
rte_crypto_asym_op_strings[opts->asym_op_type]);
if (opts->op_type == CPERF_ASYM_RSA)
printf("# rsa test name: %s\n", opts->rsa_data->name);
+ if (opts->op_type == CPERF_ASYM_MLKEM512)
+ printf("# mlkem test name: %s\n", opts->mlkem_data->name);
}
printf("# sessionless: %s\n", opts->sessionless ? "yes" : "no");
printf("# shared session: %s\n", opts->shared_session ? "yes" : "no");
diff --git a/app/test-crypto-perf/cperf_test_common.c b/app/test-crypto-perf/cperf_test_common.c
index caf429b4d9..383d4bd940 100644
--- a/app/test-crypto-perf/cperf_test_common.c
+++ b/app/test-crypto-perf/cperf_test_common.c
@@ -313,7 +313,8 @@ cperf_is_asym_test(const struct cperf_options *options)
options->op_type == CPERF_ASYM_SECP384R1 ||
options->op_type == CPERF_ASYM_SECP521R1 ||
options->op_type == CPERF_ASYM_ED25519 ||
- options->op_type == CPERF_ASYM_SM2)
+ options->op_type == CPERF_ASYM_SM2 ||
+ options->op_type == CPERF_ASYM_MLKEM512)
return true;
return false;
diff --git a/app/test-crypto-perf/cperf_test_vectors.c b/app/test-crypto-perf/cperf_test_vectors.c
index f4f856ff69..6e2435b004 100644
--- a/app/test-crypto-perf/cperf_test_vectors.c
+++ b/app/test-crypto-perf/cperf_test_vectors.c
@@ -3510,6 +3510,430 @@ uint8_t rsa_8192_cipher[] = {
0xab, 0x45, 0xa7, 0x55, 0xf5, 0xc1, 0x33, 0x3e,
};
+uint8_t mlkem_512_dk[] = {
+ 0xd8, 0xc8, 0x4d, 0x74, 0x57, 0x6d, 0x43, 0x65,
+ 0x84, 0xe0, 0x82, 0x87, 0x2d, 0xd0, 0x0b, 0x39,
+ 0x08, 0x41, 0xc8, 0x85, 0x15, 0x26, 0xfb, 0x85,
+ 0x68, 0xe1, 0x30, 0x98, 0xf0, 0xb1, 0x3d, 0xea,
+ 0xa5, 0x4c, 0xc4, 0x59, 0x81, 0xfb, 0x33, 0xed,
+ 0x20, 0x3a, 0x7f, 0x10, 0x82, 0x3d, 0x85, 0x18,
+ 0x69, 0x56, 0x6b, 0x5b, 0x7c, 0x92, 0x0f, 0x57,
+ 0x37, 0xf2, 0x93, 0x38, 0x24, 0xd3, 0x1e, 0xe1,
+ 0x84, 0x16, 0x39, 0x9a, 0x0d, 0xbd, 0xba, 0x22,
+ 0xaa, 0xc8, 0xa1, 0xd9, 0xd8, 0x2a, 0xc7, 0xc1,
+ 0x78, 0x21, 0x01, 0x85, 0x38, 0x54, 0x39, 0x34,
+ 0x25, 0x8c, 0xce, 0xda, 0xa3, 0x0c, 0x66, 0x99,
+ 0x39, 0xf7, 0x63, 0xb6, 0x86, 0x80, 0xe0, 0xe4,
+ 0x64, 0x2a, 0xfc, 0xc2, 0xe0, 0x71, 0x6c, 0x59,
+ 0x7c, 0x9a, 0x03, 0xac, 0x6f, 0x16, 0x08, 0x64,
+ 0x3d, 0x4b, 0xa5, 0x83, 0xc5, 0x34, 0x27, 0x2a,
+ 0x20, 0x18, 0xb8, 0x2d, 0x28, 0x61, 0x2a, 0x94,
+ 0x55, 0xc7, 0xc3, 0x93, 0x91, 0x71, 0x43, 0x7f,
+ 0x74, 0xb2, 0x6b, 0x55, 0xb1, 0x9d, 0xc8, 0x08,
+ 0x53, 0xda, 0xb6, 0x19, 0x46, 0x5c, 0x20, 0xea,
+ 0x31, 0x31, 0xfc, 0x41, 0x02, 0xe6, 0x3c, 0x3e,
+ 0xd8, 0x68, 0x6f, 0xa6, 0x66, 0x1e, 0xc7, 0xc6,
+ 0x0a, 0xf8, 0x02, 0x1e, 0x1e, 0xfa, 0x9a, 0x30,
+ 0x18, 0x64, 0xdc, 0x30, 0x20, 0x9a, 0xb6, 0x6e,
+ 0x85, 0xe8, 0xb0, 0xb3, 0xe1, 0x24, 0x4e, 0xf0,
+ 0x4a, 0x9c, 0xb3, 0x0e, 0xe0, 0xd2, 0x6a, 0x6e,
+ 0xc7, 0xc3, 0x15, 0x35, 0x49, 0x77, 0x07, 0x7e,
+ 0x33, 0x7c, 0x97, 0x6e, 0xeb, 0xb3, 0x7c, 0x86,
+ 0xb2, 0xa0, 0xe3, 0x8a, 0x13, 0x81, 0x99, 0xc6,
+ 0x75, 0x41, 0xa6, 0x2a, 0x6f, 0x16, 0xe2, 0x4e,
+ 0x52, 0x14, 0x0d, 0xcb, 0x74, 0x8d, 0xb4, 0x89,
+ 0xcf, 0xa8, 0x4c, 0x57, 0xaa, 0x18, 0x7d, 0xd7,
+ 0x19, 0x5c, 0x11, 0x04, 0xc6, 0xe2, 0x2c, 0x09,
+ 0x4d, 0x78, 0x7e, 0x9e, 0x1a, 0xa2, 0xab, 0x60,
+ 0x59, 0xe4, 0x30, 0xa5, 0x6d, 0x11, 0x63, 0xf9,
+ 0xb1, 0xa0, 0x98, 0xe2, 0x81, 0x34, 0xa6, 0x57,
+ 0xf7, 0x18, 0x3e, 0x4c, 0x36, 0x02, 0x4d, 0xda,
+ 0x6e, 0xac, 0x39, 0x6c, 0x29, 0x0b, 0x8c, 0xd7,
+ 0x32, 0x8e, 0x45, 0x67, 0x36, 0x41, 0x6c, 0xab,
+ 0xbb, 0x04, 0x2e, 0xae, 0xd3, 0x9f, 0xee, 0xfa,
+ 0x5f, 0x4a, 0x04, 0x85, 0xf7, 0xfb, 0x97, 0x95,
+ 0x78, 0xc4, 0x08, 0x39, 0xb8, 0xc5, 0x17, 0x9d,
+ 0x2e, 0x6c, 0x14, 0xcd, 0xf2, 0xc1, 0xc5, 0x05,
+ 0x9d, 0x34, 0x12, 0xcd, 0x86, 0xbc, 0x9e, 0x1c,
+ 0xf9, 0x1c, 0x05, 0xc2, 0xc3, 0x29, 0xf2, 0x9d,
+ 0xf4, 0xa9, 0xcc, 0x0d, 0x38, 0xce, 0x6f, 0xb4,
+ 0x06, 0x68, 0xc0, 0xcd, 0xb2, 0xb6, 0x5d, 0xdf,
+ 0x89, 0x58, 0x99, 0x70, 0xbe, 0x37, 0xa8, 0xb2,
+ 0xc3, 0x6c, 0x2c, 0x13, 0x8c, 0xa7, 0x8a, 0x38,
+ 0x36, 0x36, 0xe0, 0x77, 0x79, 0xf6, 0x30, 0xc5,
+ 0x9c, 0x8e, 0x10, 0xc0, 0x1b, 0xb4, 0xe0, 0x74,
+ 0x36, 0x6c, 0x9f, 0x80, 0x73, 0x3e, 0x57, 0x69,
+ 0x5a, 0x79, 0x6a, 0x44, 0xb8, 0xe8, 0x25, 0xbe,
+ 0x96, 0xae, 0xd7, 0x87, 0x30, 0x7e, 0x76, 0x8e,
+ 0x1c, 0x09, 0x99, 0xd9, 0x4c, 0xb3, 0xf7, 0x45,
+ 0x92, 0xfa, 0x77, 0x9e, 0x18, 0x72, 0x1a, 0x23,
+ 0x9c, 0x4e, 0x5e, 0xd9, 0x22, 0x2e, 0xb0, 0x16,
+ 0x9d, 0x03, 0x6a, 0x0e, 0x49, 0x94, 0x97, 0xa3,
+ 0x19, 0x94, 0xa1, 0x7e, 0x9d, 0x7b, 0x87, 0xc1,
+ 0x6a, 0x12, 0x42, 0xc9, 0x22, 0xbe, 0xeb, 0x38,
+ 0xbe, 0xfa, 0x0b, 0xac, 0x25, 0xb9, 0x78, 0xec,
+ 0x33, 0xdf, 0x86, 0xcb, 0xce, 0xe1, 0xa6, 0x69,
+ 0x28, 0x39, 0xd5, 0x68, 0x73, 0xcb, 0x8a, 0xcc,
+ 0x3d, 0xfa, 0x0e, 0x79, 0x71, 0x3e, 0x44, 0x19,
+ 0x9a, 0xac, 0xf4, 0x43, 0xa9, 0xba, 0x10, 0x7e,
+ 0x49, 0xc8, 0x8f, 0xf9, 0xac, 0xd2, 0x07, 0xb1,
+ 0x4b, 0x2b, 0x15, 0x65, 0x05, 0x12, 0xc3, 0xcc,
+ 0x36, 0x69, 0x17, 0x78, 0xa3, 0x00, 0x6d, 0xb7,
+ 0xe9, 0x03, 0x78, 0xb3, 0x33, 0xe5, 0xcc, 0x22,
+ 0xc3, 0x91, 0x8f, 0x47, 0x79, 0xaf, 0x2e, 0x19,
+ 0x91, 0x05, 0xab, 0x5e, 0x3e, 0x43, 0xa4, 0x34,
+ 0x0b, 0x8c, 0x89, 0x75, 0xc0, 0x08, 0x44, 0x07,
+ 0xdd, 0x17, 0x83, 0xc6, 0x48, 0x09, 0x12, 0xe6,
+ 0x25, 0xde, 0x99, 0x00, 0x51, 0xc1, 0x65, 0x8a,
+ 0x85, 0x02, 0xa8, 0xd5, 0x72, 0x41, 0x67, 0x66,
+ 0x08, 0xe9, 0x10, 0xa3, 0xa2, 0x07, 0xc8, 0x06,
+ 0xc6, 0x0c, 0xc6, 0x47, 0xf7, 0x35, 0x20, 0x40,
+ 0x69, 0x04, 0x3c, 0xba, 0x4b, 0xd3, 0x78, 0xa2,
+ 0xa4, 0x71, 0xc7, 0x83, 0x2b, 0x69, 0xf9, 0xa2,
+ 0x8b, 0xf5, 0xa6, 0x87, 0x46, 0xb3, 0x9b, 0x7a,
+ 0x35, 0x97, 0x0b, 0xf4, 0x68, 0x15, 0x4a, 0x87,
+ 0xeb, 0x0b, 0x2b, 0xa3, 0xba, 0xcd, 0xb3, 0x28,
+ 0x80, 0x38, 0x19, 0x17, 0x1f, 0x44, 0x6f, 0x7f,
+ 0x50, 0x75, 0x93, 0x00, 0x4c, 0x36, 0x07, 0x07,
+ 0x11, 0xec, 0x91, 0xc4, 0x99, 0x5d, 0xf8, 0xe1,
+ 0x53, 0x84, 0x01, 0xca, 0x6a, 0x64, 0x8c, 0xef,
+ 0x17, 0xce, 0x81, 0xf5, 0xce, 0xaa, 0x80, 0xc1,
+ 0x85, 0xc6, 0x6b, 0xe2, 0xc9, 0x43, 0xc9, 0xf5,
+ 0xc3, 0x3a, 0xdb, 0xcf, 0xb9, 0x29, 0x64, 0x8d,
+ 0xe7, 0x98, 0x0f, 0x8b, 0xb0, 0xd2, 0x84, 0x25,
+ 0x37, 0xd1, 0x80, 0x38, 0x81, 0xb9, 0xe8, 0xb4,
+ 0xa7, 0xc8, 0x3a, 0x27, 0x77, 0x9b, 0x81, 0xbb,
+ 0x61, 0xae, 0x59, 0x06, 0x57, 0x40, 0x69, 0x11,
+ 0xb2, 0x41, 0x68, 0x2d, 0x26, 0x62, 0x93, 0x40,
+ 0x4d, 0x97, 0x1c, 0xb4, 0x63, 0x65, 0x04, 0xf8,
+ 0x8c, 0x2a, 0x01, 0xb9, 0x6d, 0x3d, 0x87, 0x30,
+ 0x28, 0x70, 0x80, 0x1d, 0xb2, 0x24, 0xb0, 0x5c,
+ 0x90, 0x95, 0xa3, 0x6c, 0xdc, 0xa6, 0x66, 0x15,
+ 0x33, 0x37, 0x8c, 0xa8, 0x9f, 0x24, 0xa3, 0x52,
+ 0x21, 0x06, 0xb6, 0xaf, 0x51, 0xc0, 0x1d, 0x42,
+ 0xa6, 0x55, 0x93, 0x6b, 0x4a, 0x32, 0x4e, 0xec,
+ 0x46, 0xbe, 0xfa, 0x4a, 0x13, 0x5a, 0xca, 0x11,
+ 0x39, 0xa1, 0x81, 0x8b, 0xc4, 0x07, 0x1b, 0x43,
+ 0x1e, 0xa0, 0x5b, 0x34, 0x2d, 0x10, 0x6c, 0xdd,
+ 0x3b, 0x18, 0x78, 0xe1, 0x0d, 0x5b, 0x61, 0x3d,
+ 0xfd, 0x51, 0x6a, 0x78, 0x88, 0x2f, 0xa7, 0x84,
+ 0xa3, 0x4f, 0x30, 0x3e, 0xb0, 0xdc, 0x96, 0xa8,
+ 0xd7, 0xbb, 0x4c, 0xa6, 0x95, 0xb9, 0x27, 0x3f,
+ 0x70, 0x46, 0xa3, 0xd8, 0x36, 0x22, 0x3b, 0x4b,
+ 0x91, 0xa4, 0xe6, 0x31, 0x0d, 0xbb, 0x27, 0x2f,
+ 0x15, 0x2c, 0xa7, 0x44, 0x19, 0xa9, 0xc1, 0xc8,
+ 0xb2, 0x77, 0x1a, 0xd5, 0x14, 0x29, 0xa1, 0x76,
+ 0xcc, 0x81, 0x88, 0x47, 0x3c, 0xd9, 0x9e, 0xba,
+ 0x99, 0x26, 0xbb, 0xf6, 0x5e, 0xbc, 0x2a, 0x8e,
+ 0xa0, 0x32, 0x32, 0x2e, 0x24, 0xa1, 0xc3, 0x07,
+ 0xa5, 0xd4, 0xbb, 0x95, 0x03, 0x29, 0xb5, 0x59,
+ 0x5a, 0x2b, 0x6c, 0x95, 0x15, 0xba, 0xb0, 0xaf,
+ 0x27, 0xa2, 0xcb, 0xef, 0x46, 0xb6, 0xb2, 0x25,
+ 0x21, 0x23, 0xf8, 0x18, 0x1c, 0xb0, 0x54, 0xad,
+ 0x48, 0x38, 0xf6, 0x33, 0xa2, 0x55, 0xc8, 0x63,
+ 0xce, 0x56, 0x34, 0x86, 0x6a, 0x61, 0x75, 0x75,
+ 0x7f, 0x75, 0x49, 0x2a, 0x57, 0x72, 0x76, 0x3e,
+ 0xa7, 0x08, 0x56, 0x28, 0x0b, 0x44, 0xa9, 0x00,
+ 0xc2, 0x88, 0x28, 0x02, 0x55, 0x33, 0x55, 0x43,
+ 0x97, 0x97, 0x42, 0xa4, 0xfe, 0xb1, 0x95, 0xee,
+ 0x9b, 0x5d, 0xa9, 0x18, 0xc1, 0x90, 0xa5, 0x65,
+ 0x20, 0x99, 0x39, 0xd2, 0xea, 0x08, 0x1c, 0xda,
+ 0xa8, 0xfa, 0xa3, 0x3b, 0x56, 0xf0, 0x59, 0x17,
+ 0x10, 0x0b, 0x45, 0x4a, 0x86, 0x88, 0x96, 0x49,
+ 0x68, 0x98, 0x26, 0x55, 0x40, 0xb7, 0x5a, 0x29,
+ 0x82, 0x93, 0xa1, 0x0e, 0x7b, 0x4b, 0xa2, 0x49,
+ 0x61, 0x2b, 0xaf, 0xb4, 0x2d, 0xa1, 0xdc, 0xae,
+ 0x79, 0x44, 0x66, 0x20, 0x72, 0xc5, 0x01, 0x36,
+ 0x10, 0xd8, 0x81, 0x36, 0xb1, 0xea, 0xb1, 0x0d,
+ 0xc2, 0x05, 0x75, 0xd8, 0xad, 0xe0, 0x73, 0xc6,
+ 0x82, 0x8c, 0x7e, 0x97, 0x2b, 0x87, 0xb5, 0xc9,
+ 0x1e, 0xac, 0xb0, 0x2a, 0x16, 0x20, 0x88, 0xb9,
+ 0x41, 0x2f, 0xc8, 0x4b, 0xce, 0xd8, 0xc4, 0xa8,
+ 0xe2, 0x68, 0x35, 0xd9, 0xd0, 0xc4, 0x7a, 0xf2,
+ 0x25, 0xca, 0xb8, 0x06, 0x42, 0xcb, 0xc9, 0x8f,
+ 0x0a, 0x60, 0xde, 0x19, 0xa8, 0xcb, 0x90, 0x85,
+ 0x74, 0x42, 0x03, 0x89, 0xf4, 0x97, 0x8e, 0x41,
+ 0xc5, 0x72, 0xe3, 0x52, 0x55, 0x81, 0x73, 0xe1,
+ 0x71, 0x22, 0xa2, 0xfb, 0x8a, 0x36, 0xea, 0xb3,
+ 0xc3, 0x33, 0x4f, 0x23, 0x55, 0x03, 0x1c, 0xa1,
+ 0x84, 0x74, 0xe1, 0x69, 0x4d, 0xb3, 0xc2, 0x1f,
+ 0xbc, 0x62, 0xeb, 0xf0, 0x50, 0x45, 0xb7, 0x83,
+ 0xd3, 0x28, 0x37, 0x46, 0x35, 0x79, 0x04, 0x38,
+ 0x16, 0xbd, 0x72, 0x2f, 0xbc, 0xf9, 0x7d, 0xb8,
+ 0x20, 0x12, 0x87, 0xa7, 0xb0, 0x82, 0x7c, 0x3e,
+ 0x4d, 0xa7, 0x18, 0x04, 0x54, 0x75, 0x19, 0x52,
+ 0x61, 0x80, 0x57, 0xbe, 0x06, 0x37, 0x9c, 0xde,
+ 0xb0, 0x93, 0x70, 0x54, 0x3f, 0x15, 0x83, 0xb2,
+ 0xd3, 0x88, 0xa6, 0xd8, 0x56, 0x9b, 0x91, 0xf3,
+ 0x00, 0x01, 0x00, 0xcd, 0xcb, 0x97, 0x16, 0x27,
+ 0xc7, 0xaf, 0xb1, 0xd5, 0xba, 0x90, 0xa1, 0x24,
+ 0x33, 0x26, 0x95, 0xba, 0xcb, 0x63, 0xa1, 0x56,
+ 0xb0, 0x23, 0x09, 0x4e, 0xa8, 0x0a, 0x75, 0x02,
+ 0xd8, 0x3b, 0xc1, 0x05, 0xa0, 0x73, 0x25, 0x26,
+ 0x3e, 0xb8, 0x5b, 0xca, 0x6a, 0xae, 0x41, 0x22,
+ 0xc5, 0x9d, 0x88, 0x3f, 0x37, 0x8c, 0x50, 0xcd,
+ 0x09, 0x32, 0x34, 0x81, 0x98, 0xc1, 0x17, 0xa5,
+ 0x62, 0x94, 0x8d, 0xcd, 0xd0, 0xc3, 0x08, 0x07,
+ 0x2f, 0x8d, 0x52, 0x7b, 0xee, 0x81, 0xc5, 0x95,
+ 0x2c, 0x8b, 0x46, 0x45, 0x5d, 0xf2, 0x62, 0x4d,
+ 0x2e, 0x22, 0x7f, 0x0d, 0xa5, 0xc8, 0xba, 0x72,
+ 0xa0, 0x8f, 0xd0, 0x21, 0x45, 0x28, 0xb9, 0xad,
+ 0xac, 0x96, 0xa6, 0xc7, 0x33, 0x6d, 0xf3, 0x17,
+ 0xcb, 0x7a, 0x32, 0x4e, 0x4c, 0x83, 0xa7, 0x15,
+ 0x55, 0x6a, 0xca, 0xc1, 0xff, 0xd8, 0x90, 0xab,
+ 0xaa, 0xa4, 0x65, 0xb0, 0x25, 0xf7, 0x47, 0x00,
+ 0x2d, 0x20, 0x19, 0x23, 0xb5, 0x02, 0x39, 0x5a,
+ 0x69, 0x1c, 0xc4, 0xbc, 0xa3, 0x80, 0xcc, 0x82,
+ 0xd8, 0x66, 0x63, 0xc5, 0x7e, 0xa3, 0x6c, 0x2a,
+ 0xb0, 0x7a, 0x02, 0xc6, 0xb3, 0xb7, 0xb0, 0x29,
+ 0x3f, 0xa3, 0x99, 0xbd, 0x28, 0xf2, 0x8e, 0xd6,
+ 0x7b, 0xb3, 0xd5, 0x24, 0x12, 0xb9, 0xbb, 0xa5,
+ 0x68, 0xba, 0x9d, 0xe3, 0x44, 0xa3, 0xeb, 0x64,
+ 0x13, 0xb8, 0x8a, 0x5e, 0x72, 0x60, 0xad, 0x95,
+ 0xcb, 0x9d, 0xfb, 0x54, 0x6e, 0x0e, 0x66, 0xb5,
+ 0xf4, 0x68, 0x25, 0x91, 0xc5, 0x1a, 0x4f, 0x71,
+ 0x61, 0x50, 0xc8, 0xcb, 0x34, 0xc5, 0xc2, 0x84,
+ 0xbc, 0x4e, 0x5f, 0x28, 0x6f, 0xbf, 0xbb, 0xc8,
+ 0xb1, 0xc4, 0x4a, 0x4b, 0xb7, 0x1a, 0x00, 0xa0,
+ 0x2d, 0xe9, 0x07, 0x39, 0x93, 0xe9, 0x81, 0x51,
+ 0xe1, 0x34, 0x83, 0xf5, 0x82, 0x9d, 0xc7, 0x47,
+ 0xf1, 0x61, 0xc8, 0x29, 0x52, 0x4d, 0x51, 0xb7,
+ 0xa1, 0xeb, 0x49, 0x50, 0x6e, 0xda, 0x72, 0x5d,
+ 0x69, 0x27, 0x83, 0xca, 0x4c, 0x3b, 0xd4, 0xcc,
+ 0x7f, 0x26, 0x44, 0x2b, 0x76, 0x41, 0xe1, 0x50,
+ 0xce, 0x0f, 0x40, 0x0d, 0x8f, 0x00, 0x6e, 0x27,
+ 0x95, 0x6e, 0x92, 0x1b, 0xaf, 0xe4, 0x9b, 0x5e,
+ 0x96, 0xec, 0xc7, 0xbf, 0x64, 0x77, 0x75, 0x6d,
+ 0xda, 0xa8, 0x9c, 0x48, 0xf3, 0x79, 0x80, 0x4e,
+ 0xed, 0x6f, 0xe0, 0x20, 0x1a, 0x2b, 0x7c, 0xa1,
+ 0xac, 0x74, 0x42, 0x49, 0xd7, 0xe7, 0x7f, 0x57,
+ 0x5a, 0xdb, 0x69, 0xb5, 0xe2, 0x7e, 0x81, 0x8d,
+ 0xa9, 0x22, 0x5c, 0xda, 0xf3, 0xad, 0xbc, 0x3e,
+ 0x0b, 0xd4, 0x9d, 0x63, 0xef, 0x40, 0x34, 0x66,
+ 0x80, 0x8c, 0x2d, 0x31, 0x13, 0xdb, 0x31, 0x03,
+ 0xdf, 0x29, 0x92, 0x4a, 0x81, 0x76, 0x4f, 0x4d,
+ 0x5a, 0x64, 0x79, 0xf2, 0x8b, 0xc8, 0x3b, 0xc9,
+ 0xf3, 0x35, 0xc7, 0x18, 0xcb, 0xb1, 0xa8, 0xd2,
+ 0x24, 0xdb, 0x4d, 0x6a, 0x6a, 0xf1, 0xeb, 0xd2,
+};
+
+uint8_t mlkem_512_ek[] = {
+ 0x28, 0x70, 0x80, 0x1d, 0xb2, 0x24, 0xb0, 0x5c,
+ 0x90, 0x95, 0xa3, 0x6c, 0xdc, 0xa6, 0x66, 0x15,
+ 0x33, 0x37, 0x8c, 0xa8, 0x9f, 0x24, 0xa3, 0x52,
+ 0x21, 0x06, 0xb6, 0xaf, 0x51, 0xc0, 0x1d, 0x42,
+ 0xa6, 0x55, 0x93, 0x6b, 0x4a, 0x32, 0x4e, 0xec,
+ 0x46, 0xbe, 0xfa, 0x4a, 0x13, 0x5a, 0xca, 0x11,
+ 0x39, 0xa1, 0x81, 0x8b, 0xc4, 0x07, 0x1b, 0x43,
+ 0x1e, 0xa0, 0x5b, 0x34, 0x2d, 0x10, 0x6c, 0xdd,
+ 0x3b, 0x18, 0x78, 0xe1, 0x0d, 0x5b, 0x61, 0x3d,
+ 0xfd, 0x51, 0x6a, 0x78, 0x88, 0x2f, 0xa7, 0x84,
+ 0xa3, 0x4f, 0x30, 0x3e, 0xb0, 0xdc, 0x96, 0xa8,
+ 0xd7, 0xbb, 0x4c, 0xa6, 0x95, 0xb9, 0x27, 0x3f,
+ 0x70, 0x46, 0xa3, 0xd8, 0x36, 0x22, 0x3b, 0x4b,
+ 0x91, 0xa4, 0xe6, 0x31, 0x0d, 0xbb, 0x27, 0x2f,
+ 0x15, 0x2c, 0xa7, 0x44, 0x19, 0xa9, 0xc1, 0xc8,
+ 0xb2, 0x77, 0x1a, 0xd5, 0x14, 0x29, 0xa1, 0x76,
+ 0xcc, 0x81, 0x88, 0x47, 0x3c, 0xd9, 0x9e, 0xba,
+ 0x99, 0x26, 0xbb, 0xf6, 0x5e, 0xbc, 0x2a, 0x8e,
+ 0xa0, 0x32, 0x32, 0x2e, 0x24, 0xa1, 0xc3, 0x07,
+ 0xa5, 0xd4, 0xbb, 0x95, 0x03, 0x29, 0xb5, 0x59,
+ 0x5a, 0x2b, 0x6c, 0x95, 0x15, 0xba, 0xb0, 0xaf,
+ 0x27, 0xa2, 0xcb, 0xef, 0x46, 0xb6, 0xb2, 0x25,
+ 0x21, 0x23, 0xf8, 0x18, 0x1c, 0xb0, 0x54, 0xad,
+ 0x48, 0x38, 0xf6, 0x33, 0xa2, 0x55, 0xc8, 0x63,
+ 0xce, 0x56, 0x34, 0x86, 0x6a, 0x61, 0x75, 0x75,
+ 0x7f, 0x75, 0x49, 0x2a, 0x57, 0x72, 0x76, 0x3e,
+ 0xa7, 0x08, 0x56, 0x28, 0x0b, 0x44, 0xa9, 0x00,
+ 0xc2, 0x88, 0x28, 0x02, 0x55, 0x33, 0x55, 0x43,
+ 0x97, 0x97, 0x42, 0xa4, 0xfe, 0xb1, 0x95, 0xee,
+ 0x9b, 0x5d, 0xa9, 0x18, 0xc1, 0x90, 0xa5, 0x65,
+ 0x20, 0x99, 0x39, 0xd2, 0xea, 0x08, 0x1c, 0xda,
+ 0xa8, 0xfa, 0xa3, 0x3b, 0x56, 0xf0, 0x59, 0x17,
+ 0x10, 0x0b, 0x45, 0x4a, 0x86, 0x88, 0x96, 0x49,
+ 0x68, 0x98, 0x26, 0x55, 0x40, 0xb7, 0x5a, 0x29,
+ 0x82, 0x93, 0xa1, 0x0e, 0x7b, 0x4b, 0xa2, 0x49,
+ 0x61, 0x2b, 0xaf, 0xb4, 0x2d, 0xa1, 0xdc, 0xae,
+ 0x79, 0x44, 0x66, 0x20, 0x72, 0xc5, 0x01, 0x36,
+ 0x10, 0xd8, 0x81, 0x36, 0xb1, 0xea, 0xb1, 0x0d,
+ 0xc2, 0x05, 0x75, 0xd8, 0xad, 0xe0, 0x73, 0xc6,
+ 0x82, 0x8c, 0x7e, 0x97, 0x2b, 0x87, 0xb5, 0xc9,
+ 0x1e, 0xac, 0xb0, 0x2a, 0x16, 0x20, 0x88, 0xb9,
+ 0x41, 0x2f, 0xc8, 0x4b, 0xce, 0xd8, 0xc4, 0xa8,
+ 0xe2, 0x68, 0x35, 0xd9, 0xd0, 0xc4, 0x7a, 0xf2,
+ 0x25, 0xca, 0xb8, 0x06, 0x42, 0xcb, 0xc9, 0x8f,
+ 0x0a, 0x60, 0xde, 0x19, 0xa8, 0xcb, 0x90, 0x85,
+ 0x74, 0x42, 0x03, 0x89, 0xf4, 0x97, 0x8e, 0x41,
+ 0xc5, 0x72, 0xe3, 0x52, 0x55, 0x81, 0x73, 0xe1,
+ 0x71, 0x22, 0xa2, 0xfb, 0x8a, 0x36, 0xea, 0xb3,
+ 0xc3, 0x33, 0x4f, 0x23, 0x55, 0x03, 0x1c, 0xa1,
+ 0x84, 0x74, 0xe1, 0x69, 0x4d, 0xb3, 0xc2, 0x1f,
+ 0xbc, 0x62, 0xeb, 0xf0, 0x50, 0x45, 0xb7, 0x83,
+ 0xd3, 0x28, 0x37, 0x46, 0x35, 0x79, 0x04, 0x38,
+ 0x16, 0xbd, 0x72, 0x2f, 0xbc, 0xf9, 0x7d, 0xb8,
+ 0x20, 0x12, 0x87, 0xa7, 0xb0, 0x82, 0x7c, 0x3e,
+ 0x4d, 0xa7, 0x18, 0x04, 0x54, 0x75, 0x19, 0x52,
+ 0x61, 0x80, 0x57, 0xbe, 0x06, 0x37, 0x9c, 0xde,
+ 0xb0, 0x93, 0x70, 0x54, 0x3f, 0x15, 0x83, 0xb2,
+ 0xd3, 0x88, 0xa6, 0xd8, 0x56, 0x9b, 0x91, 0xf3,
+ 0x00, 0x01, 0x00, 0xcd, 0xcb, 0x97, 0x16, 0x27,
+ 0xc7, 0xaf, 0xb1, 0xd5, 0xba, 0x90, 0xa1, 0x24,
+ 0x33, 0x26, 0x95, 0xba, 0xcb, 0x63, 0xa1, 0x56,
+ 0xb0, 0x23, 0x09, 0x4e, 0xa8, 0x0a, 0x75, 0x02,
+ 0xd8, 0x3b, 0xc1, 0x05, 0xa0, 0x73, 0x25, 0x26,
+ 0x3e, 0xb8, 0x5b, 0xca, 0x6a, 0xae, 0x41, 0x22,
+ 0xc5, 0x9d, 0x88, 0x3f, 0x37, 0x8c, 0x50, 0xcd,
+ 0x09, 0x32, 0x34, 0x81, 0x98, 0xc1, 0x17, 0xa5,
+ 0x62, 0x94, 0x8d, 0xcd, 0xd0, 0xc3, 0x08, 0x07,
+ 0x2f, 0x8d, 0x52, 0x7b, 0xee, 0x81, 0xc5, 0x95,
+ 0x2c, 0x8b, 0x46, 0x45, 0x5d, 0xf2, 0x62, 0x4d,
+ 0x2e, 0x22, 0x7f, 0x0d, 0xa5, 0xc8, 0xba, 0x72,
+ 0xa0, 0x8f, 0xd0, 0x21, 0x45, 0x28, 0xb9, 0xad,
+ 0xac, 0x96, 0xa6, 0xc7, 0x33, 0x6d, 0xf3, 0x17,
+ 0xcb, 0x7a, 0x32, 0x4e, 0x4c, 0x83, 0xa7, 0x15,
+ 0x55, 0x6a, 0xca, 0xc1, 0xff, 0xd8, 0x90, 0xab,
+ 0xaa, 0xa4, 0x65, 0xb0, 0x25, 0xf7, 0x47, 0x00,
+ 0x2d, 0x20, 0x19, 0x23, 0xb5, 0x02, 0x39, 0x5a,
+ 0x69, 0x1c, 0xc4, 0xbc, 0xa3, 0x80, 0xcc, 0x82,
+ 0xd8, 0x66, 0x63, 0xc5, 0x7e, 0xa3, 0x6c, 0x2a,
+ 0xb0, 0x7a, 0x02, 0xc6, 0xb3, 0xb7, 0xb0, 0x29,
+ 0x3f, 0xa3, 0x99, 0xbd, 0x28, 0xf2, 0x8e, 0xd6,
+ 0x7b, 0xb3, 0xd5, 0x24, 0x12, 0xb9, 0xbb, 0xa5,
+ 0x68, 0xba, 0x9d, 0xe3, 0x44, 0xa3, 0xeb, 0x64,
+ 0x13, 0xb8, 0x8a, 0x5e, 0x72, 0x60, 0xad, 0x95,
+ 0xcb, 0x9d, 0xfb, 0x54, 0x6e, 0x0e, 0x66, 0xb5,
+ 0xf4, 0x68, 0x25, 0x91, 0xc5, 0x1a, 0x4f, 0x71,
+ 0x61, 0x50, 0xc8, 0xcb, 0x34, 0xc5, 0xc2, 0x84,
+ 0xbc, 0x4e, 0x5f, 0x28, 0x6f, 0xbf, 0xbb, 0xc8,
+ 0xb1, 0xc4, 0x4a, 0x4b, 0xb7, 0x1a, 0x00, 0xa0,
+ 0x2d, 0xe9, 0x07, 0x39, 0x93, 0xe9, 0x81, 0x51,
+ 0xe1, 0x34, 0x83, 0xf5, 0x82, 0x9d, 0xc7, 0x47,
+ 0xf1, 0x61, 0xc8, 0x29, 0x52, 0x4d, 0x51, 0xb7,
+ 0xa1, 0xeb, 0x49, 0x50, 0x6e, 0xda, 0x72, 0x5d,
+ 0x69, 0x27, 0x83, 0xca, 0x4c, 0x3b, 0xd4, 0xcc,
+ 0x7f, 0x26, 0x44, 0x2b, 0x76, 0x41, 0xe1, 0x50,
+ 0xce, 0x0f, 0x40, 0x0d, 0x8f, 0x00, 0x6e, 0x27,
+ 0x95, 0x6e, 0x92, 0x1b, 0xaf, 0xe4, 0x9b, 0x5e,
+ 0x96, 0xec, 0xc7, 0xbf, 0x64, 0x77, 0x75, 0x6d,
+ 0xda, 0xa8, 0x9c, 0x48, 0xf3, 0x79, 0x80, 0x4e,
+ 0xed, 0x6f, 0xe0, 0x20, 0x1a, 0x2b, 0x7c, 0xa1,
+ 0xac, 0x74, 0x42, 0x49, 0xd7, 0xe7, 0x7f, 0x57,
+};
+
+uint8_t mlkem_512_message[] = {
+ 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x20,
+ 0x66, 0x72, 0x6f, 0x6d, 0x20, 0x4d, 0x4c, 0x2d,
+ 0x4b, 0x45, 0x4d, 0x20, 0x74, 0x65, 0x73, 0x74,
+ 0x20, 0x76, 0x65, 0x63, 0x74, 0x6f, 0x72, 0x73,
+ 0x2e
+};
+
+uint8_t mlkem_512_cipher[] = {
+ 0x92, 0x98, 0x11, 0x65, 0x10, 0x78, 0xE3, 0xA4,
+ 0x34, 0x94, 0xD1, 0x22, 0x56, 0xD6, 0xB1, 0xF6,
+ 0xF7, 0x51, 0x2A, 0xCA, 0x8E, 0x04, 0x1D, 0x89,
+ 0xBB, 0xA7, 0x84, 0xEF, 0xAA, 0x70, 0x11, 0x75,
+ 0xFC, 0x48, 0xA1, 0x7E, 0x4C, 0x99, 0xBD, 0xB8,
+ 0x05, 0x01, 0xA6, 0xF0, 0x0E, 0x18, 0x0D, 0x75,
+ 0xF0, 0xBB, 0xBD, 0x6F, 0x28, 0xF4, 0xDC, 0x3A,
+ 0x58, 0x3D, 0x83, 0x1D, 0x78, 0xF9, 0x0E, 0x06,
+ 0x5C, 0x0B, 0x77, 0x4C, 0x28, 0x82, 0xE1, 0xA4,
+ 0x77, 0xAD, 0x14, 0x71, 0x3D, 0x5D, 0x53, 0x41,
+ 0xCE, 0x4F, 0x36, 0x32, 0x24, 0xD7, 0x09, 0xF0,
+ 0xB8, 0xF2, 0x95, 0x68, 0x84, 0x5C, 0x62, 0xC3,
+ 0x77, 0x9A, 0x74, 0xD3, 0x29, 0xC1, 0x88, 0x8B,
+ 0xBF, 0x05, 0x73, 0x43, 0x30, 0x20, 0xB4, 0xB0,
+ 0xB3, 0xE1, 0xB5, 0x91, 0xA8, 0xBD, 0x4B, 0x7F,
+ 0x2E, 0x53, 0x20, 0x55, 0xDE, 0x4D, 0xDB, 0x91,
+ 0x73, 0xF9, 0x2F, 0x21, 0xF6, 0xAE, 0x6D, 0xEE,
+ 0x67, 0x37, 0x82, 0x11, 0xE3, 0x0A, 0x5A, 0xB8,
+ 0x68, 0xC7, 0x22, 0x34, 0x2A, 0xD4, 0x37, 0xAF,
+ 0x50, 0xE0, 0xDB, 0xED, 0x89, 0x6D, 0x97, 0x0C,
+ 0xF6, 0x0D, 0x6D, 0x9B, 0x1D, 0x5B, 0xE1, 0xF6,
+ 0x45, 0x48, 0xF0, 0x75, 0x38, 0xE0, 0x2E, 0x11,
+ 0x88, 0xEC, 0x8B, 0x51, 0x47, 0x21, 0xF6, 0x6F,
+ 0x13, 0x3E, 0xA4, 0x46, 0xFE, 0xB0, 0x15, 0x81,
+ 0x68, 0x9C, 0x0F, 0x15, 0xCB, 0x64, 0x6D, 0x0B,
+ 0x04, 0x63, 0xBA, 0x67, 0x6C, 0x86, 0xE8, 0xAC,
+ 0xB3, 0xE5, 0x47, 0x88, 0x27, 0x4A, 0xBC, 0x58,
+ 0x13, 0x4D, 0x6F, 0xA6, 0x2C, 0xEC, 0x83, 0xB0,
+ 0x6E, 0x93, 0x7C, 0xC7, 0x6E, 0xBB, 0xBF, 0xE1,
+ 0x9F, 0xD6, 0x6E, 0xD8, 0x7A, 0xC5, 0xA2, 0x4D,
+ 0xA2, 0x1C, 0xDA, 0x55, 0x1E, 0xF2, 0x1C, 0x71,
+ 0x3E, 0x82, 0xD7, 0x92, 0x6F, 0x89, 0x1E, 0x53,
+ 0x48, 0x00, 0x9E, 0x93, 0xCB, 0xE3, 0x79, 0xB5,
+ 0x87, 0xDE, 0xF7, 0x84, 0xD6, 0x25, 0x57, 0x07,
+ 0x47, 0xEE, 0xDB, 0xA8, 0xC2, 0x7E, 0xE8, 0xF9,
+ 0x56, 0x22, 0x34, 0xE9, 0xD0, 0xC8, 0x3F, 0xF2,
+ 0xCB, 0xE7, 0x92, 0xE1, 0x03, 0xF4, 0x38, 0x88,
+ 0xE4, 0xC6, 0xB2, 0x42, 0x6F, 0x08, 0x61, 0x77,
+ 0x7F, 0x8A, 0xCD, 0x9D, 0x21, 0x3E, 0xE2, 0x18,
+ 0x52, 0x15, 0x02, 0x62, 0x2F, 0xCA, 0xEE, 0x44,
+ 0x60, 0x15, 0x6B, 0x76, 0x7D, 0x12, 0xC7, 0x0F,
+ 0xA0, 0x6C, 0xA1, 0x7B, 0xAE, 0xD6, 0x40, 0x1B,
+ 0x61, 0x46, 0xB4, 0x47, 0x97, 0x6C, 0xE1, 0x4C,
+ 0x47, 0x35, 0x29, 0x02, 0x6D, 0x69, 0x5E, 0x32,
+ 0xF7, 0x7F, 0x9C, 0xFA, 0x87, 0xD3, 0x7D, 0x33,
+ 0x85, 0x53, 0xF3, 0x8F, 0x0A, 0x41, 0x3C, 0x84,
+ 0x00, 0x14, 0x04, 0x63, 0xA8, 0xC3, 0xCB, 0xEB,
+ 0xAF, 0xCB, 0xE1, 0x78, 0xB2, 0x30, 0xC0, 0xA6,
+ 0x9D, 0x7A, 0x28, 0x0A, 0x60, 0x5F, 0x45, 0x74,
+ 0x24, 0xBE, 0xD8, 0x0D, 0xCB, 0x3A, 0x1B, 0xD1,
+ 0x9C, 0x2A, 0x4E, 0x9A, 0xD1, 0x27, 0xCB, 0xAB,
+ 0x08, 0x72, 0x89, 0xD9, 0xBA, 0x15, 0x01, 0x92,
+ 0x4F, 0xFA, 0x7A, 0xCA, 0x32, 0xD3, 0x7E, 0xB1,
+ 0xEB, 0xE7, 0xDF, 0x21, 0x33, 0x4F, 0xBD, 0x5F,
+ 0xCF, 0x64, 0x5D, 0x36, 0x21, 0x63, 0x34, 0x68,
+ 0x90, 0x81, 0x15, 0xC7, 0xF7, 0x28, 0x46, 0x42,
+ 0xC9, 0x78, 0xC0, 0xA3, 0x5F, 0xF8, 0x86, 0x01,
+ 0xF4, 0x1C, 0x6A, 0x65, 0xA2, 0x0E, 0x01, 0x5B,
+ 0x29, 0xF4, 0xF8, 0x9E, 0xA5, 0x5C, 0xC0, 0x4B,
+ 0x74, 0x2C, 0x88, 0xA3, 0x6B, 0xC2, 0xC4, 0xDA,
+ 0xEF, 0xF2, 0x02, 0xCD, 0x89, 0x0C, 0x42, 0xDA,
+ 0xE6, 0xBB, 0x67, 0x51, 0x29, 0x4A, 0x32, 0x99,
+ 0xAC, 0xCC, 0xDC, 0xB6, 0x35, 0x8B, 0xCC, 0xE8,
+ 0x04, 0x78, 0x69, 0xA8, 0x19, 0x0E, 0xDC, 0x1E,
+ 0x6C, 0xFF, 0x46, 0xDF, 0x4E, 0x7A, 0x51, 0xC5,
+ 0x9F, 0x39, 0x6F, 0xC7, 0x96, 0xFE, 0x54, 0xF4,
+ 0xE2, 0xD8, 0xEB, 0x5C, 0xA1, 0x28, 0x84, 0xB4,
+ 0xEA, 0x34, 0xA8, 0x18, 0x86, 0x93, 0x8B, 0xE9,
+ 0xA7, 0xAB, 0x5E, 0xD9, 0x1E, 0x3E, 0x89, 0x8F,
+ 0x83, 0xB4, 0xEF, 0x8E, 0x21, 0x30, 0xB4, 0xAD,
+ 0x07, 0xFE, 0x8A, 0x3B, 0x34, 0xAE, 0xDE, 0x5C,
+ 0xCA, 0xBA, 0x9E, 0xFB, 0x1B, 0x81, 0xEC, 0xD3,
+ 0x0E, 0x40, 0xED, 0x65, 0x77, 0xF3, 0xC0, 0x56,
+ 0xDC, 0xD8, 0x46, 0xDB, 0x4F, 0x51, 0x4B, 0x24,
+ 0x42, 0x3B, 0xC7, 0xE2, 0x32, 0xBA, 0xA7, 0xC0,
+ 0xA4, 0xFA, 0x14, 0xC4, 0xB7, 0x1B, 0x84, 0x0D,
+ 0xC9, 0xD1, 0x8E, 0x1C, 0x50, 0xF1, 0xBB, 0xA6,
+ 0x14, 0xD4, 0x88, 0xD8, 0x6C, 0xCE, 0x44, 0x76,
+ 0x3A, 0x29, 0xF8, 0x69, 0xBD, 0x5B, 0x53, 0x4B,
+ 0x65, 0xB8, 0x76, 0x5F, 0xD2, 0x01, 0xE9, 0x55,
+ 0x83, 0x47, 0x2C, 0xD3, 0x9B, 0xCA, 0x85, 0x75,
+ 0x60, 0x14, 0x1B, 0x5D, 0x48, 0x97, 0x67, 0x18,
+ 0x9F, 0x8E, 0x8C, 0x86, 0x31, 0xCF, 0x07, 0x7D,
+ 0x9E, 0xEB, 0xDE, 0x06, 0x14, 0xFE, 0x00, 0x3F,
+ 0x9C, 0x09, 0xDF, 0x2B, 0x99, 0x08, 0x7A, 0x4C,
+ 0xD9, 0xC8, 0x3B, 0x54, 0xF0, 0x34, 0x07, 0x19,
+ 0xC5, 0x4A, 0x34, 0xBF, 0xB8, 0x31, 0xBF, 0x1C,
+ 0x6F, 0x5C, 0x07, 0x37, 0x76, 0xD9, 0xBD, 0x3B,
+ 0xB6, 0xB6, 0x8D, 0x6A, 0x1A, 0xE6, 0xE9, 0xCB,
+ 0x4D, 0xC3, 0xDC, 0x76, 0x91, 0xE7, 0x6F, 0x11,
+ 0x66, 0xF7, 0x76, 0xB3, 0x40, 0xAA, 0x51, 0x09,
+ 0x7C, 0xFE, 0xA2, 0x37, 0xBF, 0xC3, 0x92, 0xFA,
+ 0x75, 0x44, 0x76, 0xC7, 0x54, 0xC8, 0x91, 0x89,
+ 0x65, 0x84, 0x03, 0x3E, 0x46, 0x92, 0x1E, 0x67,
+ 0x8D, 0x8F, 0x52, 0x0D, 0x06, 0x22, 0x18, 0xFC,
+ 0x0B, 0x92, 0xCD, 0x94, 0xC2, 0x0A, 0x3F, 0x41,
+};
+
+uint8_t mlkem_512_sk[] = {
+ 0x26, 0xCD, 0x28, 0x15, 0xEB, 0x3E, 0x16, 0x56,
+ 0x84, 0x2D, 0x15, 0xAC, 0x32, 0x33, 0xDA, 0x01,
+ 0x71, 0x21, 0x82, 0x1B, 0xC7, 0xE3, 0x44, 0xF9,
+ 0x5E, 0x7A, 0xB9, 0x3A, 0x40, 0xAD, 0x38, 0x6A
+};
+
struct
cperf_rsa_test_data rsa_qt_perf_data[4] = {
{
@@ -3822,6 +4246,52 @@ cperf_rsa_test_data rsa_pub_perf_data[4] = {
}
};
+struct cperf_mlkem_test_data mlkem_encap_perf_data[] = {
+ {
+ .name = "mlkem_512_encap (deterministic)",
+ .type = RTE_CRYPTO_ML_KEM_512,
+ .dk = {
+ .data = mlkem_512_dk,
+ .length = sizeof(mlkem_512_dk),
+ },
+ .ek = {
+ .data = mlkem_512_ek,
+ .length = sizeof(mlkem_512_ek),
+ },
+ .message = {
+ .data = mlkem_512_message,
+ .length = 32,
+ },
+ .cipher = {
+ .data = mlkem_512_cipher,
+ .length = sizeof(mlkem_512_cipher),
+ },
+ .sk = {
+ .data = mlkem_512_sk,
+ .length = sizeof(mlkem_512_sk),
+ },
+ },
+};
+
+struct cperf_mlkem_test_data mlkem_decap_perf_data[] = {
+ {
+ .name = "mlkem_512_decap",
+ .type = RTE_CRYPTO_ML_KEM_512,
+ .cipher = {
+ .data = mlkem_512_cipher,
+ .length = sizeof(mlkem_512_cipher),
+ },
+ .dk = {
+ .data = mlkem_512_dk,
+ .length = sizeof(mlkem_512_dk),
+ },
+ .sk = {
+ .data = mlkem_512_sk,
+ .length = sizeof(mlkem_512_sk),
+ },
+ }
+};
+
struct cperf_test_vector*
cperf_test_vector_get_dummy(struct cperf_options *options)
{
diff --git a/app/test-crypto-perf/cperf_test_vectors.h b/app/test-crypto-perf/cperf_test_vectors.h
index d6f18268c4..e498196ae3 100644
--- a/app/test-crypto-perf/cperf_test_vectors.h
+++ b/app/test-crypto-perf/cperf_test_vectors.h
@@ -107,6 +107,31 @@ struct cperf_modex_test_data {
} result;
};
+struct cperf_mlkem_test_data {
+ const char *name;
+ enum rte_crypto_ml_kem_type type;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } dk;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } ek;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } message;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } cipher;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } sk;
+};
+
#define TEST_DATA_SIZE 4096
struct cperf_rsa_plaintext {
uint8_t data[TEST_DATA_SIZE];
@@ -188,5 +213,7 @@ extern struct cperf_rsa_test_data rsa_pub_perf_data[4];
extern struct cperf_rsa_test_data rsa_exp_perf_data[4];
extern struct cperf_rsa_test_data rsa_qt_perf_data[4];
extern struct cperf_rsa_plaintext rsa_plaintext;
+extern struct cperf_mlkem_test_data mlkem_encap_perf_data[];
+extern struct cperf_mlkem_test_data mlkem_decap_perf_data[];
#endif
diff --git a/app/test-crypto-perf/main.c b/app/test-crypto-perf/main.c
index f63e892413..2e228201ab 100644
--- a/app/test-crypto-perf/main.c
+++ b/app/test-crypto-perf/main.c
@@ -53,7 +53,8 @@ const char *cperf_op_type_strs[] = {
[CPERF_ASYM_SECP521R1] = "ecdsa_p521r1",
[CPERF_ASYM_ED25519] = "eddsa_25519",
[CPERF_ASYM_SM2] = "sm2",
- [CPERF_TLS] = "tls-record"
+ [CPERF_TLS] = "tls-record",
+ [CPERF_ASYM_MLKEM512] = "mlkem_512",
};
const char *cperf_rsa_priv_keytype_strs[] = {
@@ -245,6 +246,7 @@ cperf_initialize_cryptodev(struct cperf_options *opts, uint8_t *enabled_cdevs)
case CPERF_ASYM_ED25519:
case CPERF_ASYM_SM2:
case CPERF_ASYM_RSA:
+ case CPERF_ASYM_MLKEM512:
case CPERF_ASYM_MODEX:
conf.ff_disable |= (RTE_CRYPTODEV_FF_SECURITY |
RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO);
@@ -515,6 +517,23 @@ cperf_verify_devices_capabilities(struct cperf_options *opts,
}
}
+ if (opts->op_type == CPERF_ASYM_MLKEM512) {
+ asym_cap_idx.type = RTE_CRYPTO_ASYM_XFORM_ML_KEM;
+ asym_capability = rte_cryptodev_asym_capability_get(cdev_id, &asym_cap_idx);
+ if (asym_capability == NULL)
+ return -1;
+
+ if (opts->asym_op_type == RTE_CRYPTO_ASYM_OP_ENCRYPT)
+ opts->mlkem_data = &mlkem_encap_perf_data[0];
+ else if (opts->asym_op_type == RTE_CRYPTO_ASYM_OP_DECRYPT)
+ opts->mlkem_data = &mlkem_decap_perf_data[0];
+ else {
+ RTE_LOG(ERR, USER1, "Unsupported MLKEM operation type\n");
+ return -ENOTSUP;
+ }
+
+ }
+
if (opts->op_type == CPERF_AUTH_ONLY ||
opts->op_type == CPERF_CIPHER_THEN_AUTH ||
opts->op_type == CPERF_AUTH_THEN_CIPHER) {
--
2.43.0
^ permalink raw reply related
* [PATCH] crypto/openssl: fix use-after-free bug and cleanup
From: Pratik Senapati @ 2026-05-28 7:58 UTC (permalink / raw)
To: dev; +Cc: gakhil, anoobj, gmuthukrishn, kai.ji, stable
params is freed before it is used by
EVP_PKEY_decapsulate_init() causing a
use-after-free issue. Pass NULL to
EVP_PKEY_decapsulate_init() instead of params
to avoid it.
Add resource cleanup for all error paths in the ML-KEM
decapsulate handler and consolidate cleanup into
two goto labels err_pkey and err_decap.
Fixes: 5f761d7b60 ("crypto/openssl: support ML-KEM and ML-DSA")
Cc: stable@dpdk.org
Signed-off-by: Pratik Senapati <psenapati@marvell.com>
---
.mailmap | 1 +
drivers/crypto/openssl/rte_openssl_pmd.c | 30 +++++++++++-------------
2 files changed, 15 insertions(+), 16 deletions(-)
diff --git a/.mailmap b/.mailmap
index 4f93307aed..031becba8c 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1929,3 +1929,4 @@ Zoltan Kiss <zoltan.kiss@schaman.hu> <zoltan.kiss@linaro.org>
Zorik Machulsky <zorik@amazon.com>
Zyta Szpak <zyta@marvell.com> <zr@semihalf.com>
Zyta Szpak <zyta@marvell.com> <zyta.szpak@semihalf.com>
+Pratik Senapati <psenapati@marvell.com>
diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index 4f171f48cc..5bc51b8f0f 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -3683,38 +3683,29 @@ mlkem_decap_op_evp(struct rte_crypto_op *cop,
}
cctx = EVP_PKEY_CTX_new_from_pkey(NULL, pkey, NULL);
- if (cctx == NULL) {
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+ if (cctx == NULL)
+ goto err_pkey;
- if (EVP_PKEY_decapsulate_init(cctx, params) != 1) {
+ if (EVP_PKEY_decapsulate_init(cctx, NULL) != 1) {
cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_decap;
}
if (EVP_PKEY_decapsulate(cctx, NULL, &keylen,
op->decap.cipher.data, op->decap.cipher.length) != 1) {
OPENSSL_LOG(ERR, "Failed to determine output length");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_decap;
}
if (keylen > op->decap.sk.length) {
OPENSSL_LOG(ERR, "Insufficient buffer for shared key");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_decap;
}
if (EVP_PKEY_decapsulate(cctx, op->decap.sk.data, &keylen,
op->decap.cipher.data, op->decap.cipher.length) != 1) {
OPENSSL_LOG(ERR, "Failed to decapsulate");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_decap;
}
op->decap.sk.length = keylen;
@@ -3724,6 +3715,13 @@ mlkem_decap_op_evp(struct rte_crypto_op *cop,
ret = 0;
cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
return ret;
+
+err_decap:
+ EVP_PKEY_CTX_free(cctx);
+err_pkey:
+ EVP_PKEY_free(pkey);
+ cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
+ return -1;
}
static int
--
2.43.0
^ permalink raw reply related
* [v2] crypto/openssl: update to OpenSSL 3.0 minimum version
From: Emma Finn @ 2026-05-28 8:00 UTC (permalink / raw)
To: Kai Ji; +Cc: dev, Emma Finn
Update the OpenSSL PMD to require OpenSSL 3.0.0 as the minimum
supported version, removing all compatibility code for earlier
versions (1.0.1, 1.1.0, 1.1.1).
Signed-off-by: Emma Finn <emma.finn@intel.com>
---
*v2: skip build if openssl v3.0 dependency is not met.
---
doc/guides/cryptodevs/openssl.rst | 4 +-
doc/guides/rel_notes/release_26_07.rst | 5 +
drivers/crypto/openssl/compat.h | 203 ------
drivers/crypto/openssl/meson.build | 4 +-
drivers/crypto/openssl/openssl_pmd_private.h | 30 -
drivers/crypto/openssl/rte_openssl_pmd.c | 648 +------------------
drivers/crypto/openssl/rte_openssl_pmd_ops.c | 206 ------
7 files changed, 21 insertions(+), 1079 deletions(-)
diff --git a/doc/guides/cryptodevs/openssl.rst b/doc/guides/cryptodevs/openssl.rst
index 9d94668a9a..b4e2a014e2 100644
--- a/doc/guides/cryptodevs/openssl.rst
+++ b/doc/guides/cryptodevs/openssl.rst
@@ -74,9 +74,9 @@ To compile the OpenSSL PMD the openssl library must be installed. It will
then be picked up by the Meson/Ninja build system.
To ensure that you have the latest security fixes it is recommended that you
-use version 1.1.1g or newer.
+use the latest stable version of OpenSSL 3.x.
-* 1.1.1g, 2020-Apr-21. https://www.openssl.org/source/
+* OpenSSL 3.0.0 or newer: https://www.openssl.org/source/
Initialization
--------------
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 58d782f77e..989d54f7b7 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -90,6 +90,11 @@ Removed Items
Also, make sure to start the actual text at the margin.
=======================================================
+* crypto/openssl: Removed support for OpenSSL 1.x versions from the OpenSSL crypto PMD.
+
+ The OpenSSL crypto PMD now requires OpenSSL 3.0 as the minimum version,
+ and all compatibility code for OpenSSL 1.0.1, 1.1.0, and 1.1.1 versions has been removed.
+
API Changes
-----------
diff --git a/drivers/crypto/openssl/compat.h b/drivers/crypto/openssl/compat.h
index e1814fea8c..14104dbf2e 100644
--- a/drivers/crypto/openssl/compat.h
+++ b/drivers/crypto/openssl/compat.h
@@ -5,7 +5,6 @@
#ifndef __RTA_COMPAT_H__
#define __RTA_COMPAT_H__
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
static __rte_always_inline void
free_hmac_ctx(EVP_MAC_CTX *ctx)
{
@@ -17,120 +16,7 @@ free_cmac_ctx(EVP_MAC_CTX *ctx)
{
EVP_MAC_CTX_free(ctx);
}
-#else
-static __rte_always_inline void
-free_hmac_ctx(HMAC_CTX *ctx)
-{
- HMAC_CTX_free(ctx);
-}
-
-static __rte_always_inline void
-free_cmac_ctx(CMAC_CTX *ctx)
-{
- CMAC_CTX_free(ctx);
-}
-#endif
-
-#if (OPENSSL_VERSION_NUMBER < 0x10100000L)
-
-static __rte_always_inline int
-set_rsa_params(RSA *rsa, BIGNUM *p, BIGNUM *q)
-{
- rsa->p = p;
- rsa->q = q;
- return 0;
-}
-
-static __rte_always_inline int
-set_rsa_crt_params(RSA *rsa, BIGNUM *dmp1, BIGNUM *dmq1, BIGNUM *iqmp)
-{
- rsa->dmp1 = dmp1;
- rsa->dmq1 = dmq1;
- rsa->iqmp = iqmp;
- return 0;
-}
-
-static __rte_always_inline int
-set_rsa_keys(RSA *rsa, BIGNUM *n, BIGNUM *e, BIGNUM *d)
-{
- rsa->n = n;
- rsa->e = e;
- rsa->d = d;
- return 0;
-}
-
-static __rte_always_inline int
-set_dh_params(DH *dh, BIGNUM *p, BIGNUM *g)
-{
- dh->p = p;
- dh->q = NULL;
- dh->g = g;
- return 0;
-}
-
-static __rte_always_inline int
-set_dh_priv_key(DH *dh, BIGNUM *priv_key)
-{
- dh->priv_key = priv_key;
- return 0;
-}
-
-static __rte_always_inline int
-set_dsa_params(DSA *dsa, BIGNUM *p, BIGNUM *q, BIGNUM *g)
-{
- dsa->p = p;
- dsa->q = q;
- dsa->g = g;
- return 0;
-}
-
-static __rte_always_inline void
-get_dh_pub_key(DH *dh, const BIGNUM **pub_key)
-{
- *pub_key = dh->pub_key;
-}
-
-static __rte_always_inline void
-get_dh_priv_key(DH *dh, const BIGNUM **priv_key)
-{
- *priv_key = dh->priv_key;
-}
-
-static __rte_always_inline void
-set_dsa_sign(DSA_SIG *sign, BIGNUM *r, BIGNUM *s)
-{
- sign->r = r;
- sign->s = s;
-}
-
-static __rte_always_inline void
-get_dsa_sign(DSA_SIG *sign, const BIGNUM **r, const BIGNUM **s)
-{
- *r = sign->r;
- *s = sign->s;
-}
-
-static __rte_always_inline int
-set_dsa_keys(DSA *dsa, BIGNUM *pub, BIGNUM *priv)
-{
- dsa->pub_key = pub;
- dsa->priv_key = priv;
- return 0;
-}
-
-static __rte_always_inline void
-set_dsa_pub_key(DSA *dsa, BIGNUM *pub)
-{
- dsa->pub_key = pub;
-}
-
-static __rte_always_inline void
-get_dsa_priv_key(DSA *dsa, BIGNUM **priv_key)
-{
- *priv_key = dsa->priv_key;
-}
-#elif (OPENSSL_VERSION_NUMBER >= 0x30000000L)
static __rte_always_inline void
set_dsa_sign(DSA_SIG *sign, BIGNUM *r, BIGNUM *s)
{
@@ -142,94 +28,5 @@ get_dsa_sign(DSA_SIG *sign, const BIGNUM **r, const BIGNUM **s)
{
DSA_SIG_get0(sign, r, s);
}
-#else
-
-static __rte_always_inline int
-set_rsa_params(RSA *rsa, BIGNUM *p, BIGNUM *q)
-{
- return !(RSA_set0_factors(rsa, p, q));
-}
-
-static __rte_always_inline int
-set_rsa_crt_params(RSA *rsa, BIGNUM *dmp1, BIGNUM *dmq1, BIGNUM *iqmp)
-{
- return !(RSA_set0_crt_params(rsa, dmp1, dmq1, iqmp));
-}
-
-/* n, e must be non-null, d can be NULL */
-
-static __rte_always_inline int
-set_rsa_keys(RSA *rsa, BIGNUM *n, BIGNUM *e, BIGNUM *d)
-{
- return !(RSA_set0_key(rsa, n, e, d));
-}
-
-static __rte_always_inline int
-set_dh_params(DH *dh, BIGNUM *p, BIGNUM *g)
-{
- return !(DH_set0_pqg(dh, p, NULL, g));
-}
-
-static __rte_always_inline int
-set_dh_priv_key(DH *dh, BIGNUM *priv_key)
-{
- return !(DH_set0_key(dh, NULL, priv_key));
-}
-
-static __rte_always_inline void
-get_dh_pub_key(DH *dh_key, const BIGNUM **pub_key)
-{
- DH_get0_key(dh_key, pub_key, NULL);
-}
-
-static __rte_always_inline void
-get_dh_priv_key(DH *dh_key, const BIGNUM **priv_key)
-{
- DH_get0_key(dh_key, NULL, priv_key);
-}
-
-static __rte_always_inline int
-set_dsa_params(DSA *dsa, BIGNUM *p, BIGNUM *q, BIGNUM *g)
-{
- return !(DSA_set0_pqg(dsa, p, q, g));
-}
-
-static __rte_always_inline void
-set_dsa_priv_key(DSA *dsa, BIGNUM *priv_key)
-{
- DSA_set0_key(dsa, NULL, priv_key);
-}
-
-static __rte_always_inline void
-set_dsa_sign(DSA_SIG *sign, BIGNUM *r, BIGNUM *s)
-{
- DSA_SIG_set0(sign, r, s);
-}
-
-static __rte_always_inline void
-get_dsa_sign(DSA_SIG *sign, const BIGNUM **r, const BIGNUM **s)
-{
- DSA_SIG_get0(sign, r, s);
-}
-
-static __rte_always_inline int
-set_dsa_keys(DSA *dsa, BIGNUM *pub, BIGNUM *priv)
-{
- return !(DSA_set0_key(dsa, pub, priv));
-}
-
-static __rte_always_inline void
-set_dsa_pub_key(DSA *dsa, BIGNUM *pub_key)
-{
- DSA_set0_key(dsa, pub_key, NULL);
-}
-
-static __rte_always_inline void
-get_dsa_priv_key(DSA *dsa, const BIGNUM **priv_key)
-{
- DSA_get0_key(dsa, NULL, priv_key);
-}
-
-#endif /* version < 10100000 */
#endif /* __RTA_COMPAT_H__ */
diff --git a/drivers/crypto/openssl/meson.build b/drivers/crypto/openssl/meson.build
index af469a9827..0d82c42764 100644
--- a/drivers/crypto/openssl/meson.build
+++ b/drivers/crypto/openssl/meson.build
@@ -7,10 +7,10 @@ if is_windows
subdir_done()
endif
-dep = dependency('libcrypto', required: false, method: 'pkg-config')
+dep = dependency('libcrypto', required: false, method: 'pkg-config', version: '>= 3.0.0')
if not dep.found()
build = false
- reason = 'missing dependency, "libcrypto"'
+ reason = 'missing dependency, "libcrypto >= 3.0.0"'
endif
deps += 'bus_vdev'
sources = files('rte_openssl_pmd.c', 'rte_openssl_pmd_ops.c')
diff --git a/drivers/crypto/openssl/openssl_pmd_private.h b/drivers/crypto/openssl/openssl_pmd_private.h
index d5a751600a..ab40012d61 100644
--- a/drivers/crypto/openssl/openssl_pmd_private.h
+++ b/drivers/crypto/openssl/openssl_pmd_private.h
@@ -13,10 +13,8 @@
#include <openssl/dh.h>
#include <openssl/dsa.h>
#include <openssl/ec.h>
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
#include <openssl/provider.h>
#include <openssl/core_names.h>
-#endif
#define CRYPTODEV_NAME_OPENSSL_PMD crypto_openssl
/**< Open SSL Crypto PMD device name */
@@ -84,13 +82,8 @@ struct evp_ctx_pair {
EVP_CIPHER_CTX *cipher;
union {
EVP_MD_CTX *auth;
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
EVP_MAC_CTX *hmac;
EVP_MAC_CTX *cmac;
-#else
- HMAC_CTX *hmac;
- CMAC_CTX *cmac;
-#endif
};
};
@@ -153,24 +146,13 @@ struct __rte_cache_aligned openssl_session {
/**< pointer to EVP key */
const EVP_MD *evp_algo;
/**< pointer to EVP algorithm function */
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
EVP_MAC_CTX * ctx;
-# else
- HMAC_CTX *ctx;
-# endif
/**< pointer to EVP context structure */
} hmac;
struct {
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
EVP_MAC_CTX * ctx;
/**< pointer to EVP context structure */
-# else
- const EVP_CIPHER * evp_algo;
- /**< pointer to EVP algorithm function */
- CMAC_CTX *ctx;
- /**< pointer to EVP context structure */
-# endif
} cmac;
};
@@ -198,9 +180,7 @@ struct __rte_cache_aligned openssl_asym_session {
struct rsa {
RSA *rsa;
uint32_t pad;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
EVP_PKEY_CTX * ctx;
-#endif
} r;
struct exp {
BIGNUM *exp;
@@ -216,38 +196,28 @@ struct __rte_cache_aligned openssl_asym_session {
uint32_t key_op;
BIGNUM *p;
BIGNUM *g;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD * param_bld;
OSSL_PARAM_BLD *param_bld_peer;
-#endif
} dh;
struct {
DSA *dsa;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD * param_bld;
BIGNUM *p;
BIGNUM *g;
BIGNUM *q;
BIGNUM *priv_key;
-#endif
} s;
struct {
uint8_t curve_id;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
EC_GROUP * group;
BIGNUM *priv_key;
-#endif
} ec;
struct {
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM * params;
-#endif
} sm2;
struct {
uint8_t curve_id;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM * params;
-#endif
} eddsa;
struct {
uint8_t type;
diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index c34efb8ad0..8748ef6195 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -19,35 +19,14 @@
#include "openssl_pmd_private.h"
#include "compat.h"
-#define DES_BLOCK_SIZE 8
-
-static uint8_t cryptodev_driver_id;
-
-#if (OPENSSL_VERSION_NUMBER < 0x10100000L)
-static HMAC_CTX *HMAC_CTX_new(void)
-{
- HMAC_CTX *ctx = OPENSSL_malloc(sizeof(*ctx));
-
- if (ctx != NULL)
- HMAC_CTX_init(ctx);
- return ctx;
-}
-
-static void HMAC_CTX_free(HMAC_CTX *ctx)
-{
- if (ctx != NULL) {
- HMAC_CTX_cleanup(ctx);
- OPENSSL_free(ctx);
- }
-}
-#endif
-
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
-
#include <openssl/provider.h>
#include <openssl/core_names.h>
#include <openssl/param_build.h>
+#define DES_BLOCK_SIZE 8
+
+static uint8_t cryptodev_driver_id;
+
#define MAX_OSSL_ALGO_NAME_SIZE 16
OSSL_PROVIDER *legacy;
@@ -104,7 +83,6 @@ digest_name_get(enum rte_crypto_auth_algorithm algo)
return NULL;
}
}
-#endif
static int cryptodev_openssl_remove(struct rte_vdev_device *vdev);
@@ -306,14 +284,12 @@ get_auth_algo(enum rte_crypto_auth_algorithm sessalgo,
case RTE_CRYPTO_AUTH_SHA3_512_HMAC:
*algo = EVP_sha3_512();
break;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
case RTE_CRYPTO_AUTH_SHAKE_128:
*algo = EVP_shake128();
break;
case RTE_CRYPTO_AUTH_SHAKE_256:
*algo = EVP_shake256();
break;
-#endif
default:
res = -EINVAL;
break;
@@ -659,12 +635,10 @@ static int
openssl_set_session_auth_parameters(struct openssl_session *sess,
const struct rte_crypto_sym_xform *xform)
{
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
char algo_name[MAX_OSSL_ALGO_NAME_SIZE];
OSSL_PARAM params[2];
const char *algo;
EVP_MAC *mac;
-# endif
/* Select auth generate/verify */
sess->auth.operation = xform->auth.op;
sess->auth.algo = xform->auth.algo;
@@ -708,10 +682,8 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
case RTE_CRYPTO_AUTH_SHA3_256:
case RTE_CRYPTO_AUTH_SHA3_384:
case RTE_CRYPTO_AUTH_SHA3_512:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
case RTE_CRYPTO_AUTH_SHAKE_128:
case RTE_CRYPTO_AUTH_SHAKE_256:
-#endif
sess->auth.mode = OPENSSL_AUTH_AS_AUTH;
if (get_auth_algo(xform->auth.algo,
&sess->auth.auth.evp_algo) != 0)
@@ -720,7 +692,6 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
break;
case RTE_CRYPTO_AUTH_AES_CMAC:
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
if (xform->auth.key.length == 16)
algo = SN_aes_128_cbc;
else if (xform->auth.key.length == 24)
@@ -745,22 +716,8 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
xform->auth.key.length,
params) != 1)
return -EINVAL;
-# else
- sess->auth.mode = OPENSSL_AUTH_AS_CMAC;
- sess->auth.cmac.ctx = CMAC_CTX_new();
- if (get_cipher_algo(RTE_CRYPTO_CIPHER_AES_CBC,
- xform->auth.key.length,
- &sess->auth.cmac.evp_algo) != 0)
- return -EINVAL;
- if (CMAC_Init(sess->auth.cmac.ctx,
- xform->auth.key.data,
- xform->auth.key.length,
- sess->auth.cmac.evp_algo, NULL) != 1)
- return -EINVAL;
-# endif
break;
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
case RTE_CRYPTO_AUTH_MD5_HMAC:
case RTE_CRYPTO_AUTH_SHA1_HMAC:
case RTE_CRYPTO_AUTH_SHA224_HMAC:
@@ -794,30 +751,6 @@ openssl_set_session_auth_parameters(struct openssl_session *sess,
params) != 1)
return -EINVAL;
break;
-# else
- case RTE_CRYPTO_AUTH_MD5_HMAC:
- case RTE_CRYPTO_AUTH_SHA1_HMAC:
- case RTE_CRYPTO_AUTH_SHA224_HMAC:
- case RTE_CRYPTO_AUTH_SHA256_HMAC:
- case RTE_CRYPTO_AUTH_SHA384_HMAC:
- case RTE_CRYPTO_AUTH_SHA512_HMAC:
- case RTE_CRYPTO_AUTH_SHA3_224_HMAC:
- case RTE_CRYPTO_AUTH_SHA3_256_HMAC:
- case RTE_CRYPTO_AUTH_SHA3_384_HMAC:
- case RTE_CRYPTO_AUTH_SHA3_512_HMAC:
- sess->auth.mode = OPENSSL_AUTH_AS_HMAC;
- sess->auth.hmac.ctx = HMAC_CTX_new();
- if (get_auth_algo(xform->auth.algo,
- &sess->auth.hmac.evp_algo) != 0)
- return -EINVAL;
-
- if (HMAC_Init_ex(sess->auth.hmac.ctx,
- xform->auth.key.data,
- xform->auth.key.length,
- sess->auth.hmac.evp_algo, NULL) != 1)
- return -EINVAL;
- break;
-# endif
default:
return -ENOTSUP;
}
@@ -1295,10 +1228,6 @@ process_openssl_auth_encryption_gcm(struct rte_mbuf *mbuf_src, int offset,
uint8_t *dst, uint8_t *tag, EVP_CIPHER_CTX *ctx)
{
int len = 0;
-#if OPENSSL_VERSION_NUMBER < 0x10100000L
- int unused = 0;
- uint8_t empty[] = {};
-#endif
if (EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv) <= 0)
goto process_auth_encryption_gcm_err;
@@ -1312,12 +1241,6 @@ process_openssl_auth_encryption_gcm(struct rte_mbuf *mbuf_src, int offset,
srclen, ctx, 0))
goto process_auth_encryption_gcm_err;
-#if OPENSSL_VERSION_NUMBER < 0x10100000L
- /* Workaround open ssl bug in version less then 1.0.1f */
- if (EVP_EncryptUpdate(ctx, empty, &unused, empty, 0) <= 0)
- goto process_auth_encryption_gcm_err;
-#endif
-
if (EVP_EncryptFinal_ex(ctx, dst, &len) <= 0)
goto process_auth_encryption_gcm_err;
@@ -1379,10 +1302,6 @@ process_openssl_auth_decryption_gcm(struct rte_mbuf *mbuf_src, int offset,
uint8_t *dst, uint8_t *tag, EVP_CIPHER_CTX *ctx)
{
int len = 0;
-#if OPENSSL_VERSION_NUMBER < 0x10100000L
- int unused = 0;
- uint8_t empty[] = {};
-#endif
if (EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, 16, tag) <= 0)
goto process_auth_decryption_gcm_err;
@@ -1399,12 +1318,6 @@ process_openssl_auth_decryption_gcm(struct rte_mbuf *mbuf_src, int offset,
srclen, ctx, 0))
goto process_auth_decryption_gcm_err;
-#if OPENSSL_VERSION_NUMBER < 0x10100000L
- /* Workaround open ssl bug in version less then 1.0.1f */
- if (EVP_DecryptUpdate(ctx, empty, &unused, empty, 0) <= 0)
- goto process_auth_decryption_gcm_err;
-#endif
-
if (EVP_DecryptFinal_ex(ctx, dst, &len) <= 0)
return -EFAULT;
@@ -1500,17 +1413,11 @@ process_openssl_auth(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
process_auth_final:
/* SHAKE algorithms are XOFs and require EVP_DigestFinalXOF */
if (algo == EVP_shake128() || algo == EVP_shake256()) {
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
/* Set XOF output length before calling EVP_DigestFinalXOF */
if (EVP_MD_CTX_ctrl(ctx, EVP_MD_CTRL_XOF_LEN, digest_length, NULL) <= 0)
goto process_auth_err;
if (EVP_DigestFinalXOF(ctx, dst, digest_length) <= 0)
goto process_auth_err;
-#else
- RTE_SET_USED(digest_length);
- OPENSSL_LOG(ERR, "SHAKE algorithms require OpenSSL 3.0+");
- goto process_auth_err;
-#endif
} else {
if (EVP_DigestFinal_ex(ctx, dst, (unsigned int *)&dstlen) <= 0)
goto process_auth_err;
@@ -1523,7 +1430,6 @@ process_openssl_auth(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
return -EINVAL;
}
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
/** Process standard openssl auth algorithms with hmac/cmac */
static int
process_openssl_auth_mac(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
@@ -1576,109 +1482,6 @@ process_openssl_auth_mac(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
OPENSSL_LOG(ERR, "Process openssl auth failed");
return -EINVAL;
}
-# else
-/** Process standard openssl auth algorithms with hmac */
-static int
-process_openssl_auth_hmac(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
- int srclen, HMAC_CTX *ctx)
-{
- unsigned int dstlen;
- struct rte_mbuf *m;
- int l, n = srclen;
- uint8_t *src;
-
- for (m = mbuf_src; m != NULL && offset > rte_pktmbuf_data_len(m);
- m = m->next)
- offset -= rte_pktmbuf_data_len(m);
-
- if (m == 0)
- goto process_auth_err;
-
- src = rte_pktmbuf_mtod_offset(m, uint8_t *, offset);
-
- l = rte_pktmbuf_data_len(m) - offset;
- if (srclen <= l) {
- if (HMAC_Update(ctx, (unsigned char *)src, srclen) != 1)
- goto process_auth_err;
- goto process_auth_final;
- }
-
- if (HMAC_Update(ctx, (unsigned char *)src, l) != 1)
- goto process_auth_err;
-
- n -= l;
-
- for (m = m->next; (m != NULL) && (n > 0); m = m->next) {
- src = rte_pktmbuf_mtod(m, uint8_t *);
- l = rte_pktmbuf_data_len(m) < n ? rte_pktmbuf_data_len(m) : n;
- if (HMAC_Update(ctx, (unsigned char *)src, l) != 1)
- goto process_auth_err;
- n -= l;
- }
-
-process_auth_final:
- if (HMAC_Final(ctx, dst, &dstlen) != 1)
- goto process_auth_err;
-
- if (unlikely(HMAC_Init_ex(ctx, NULL, 0, NULL, NULL) != 1))
- goto process_auth_err;
-
- return 0;
-
-process_auth_err:
- OPENSSL_LOG(ERR, "Process openssl auth failed");
- return -EINVAL;
-}
-
-/** Process standard openssl auth algorithms with cmac */
-static int
-process_openssl_auth_cmac(struct rte_mbuf *mbuf_src, uint8_t *dst, int offset,
- int srclen, CMAC_CTX *ctx)
-{
- unsigned int dstlen;
- struct rte_mbuf *m;
- int l, n = srclen;
- uint8_t *src;
-
- for (m = mbuf_src; m != NULL && offset > rte_pktmbuf_data_len(m);
- m = m->next)
- offset -= rte_pktmbuf_data_len(m);
-
- if (m == 0)
- goto process_auth_err;
-
- src = rte_pktmbuf_mtod_offset(m, uint8_t *, offset);
-
- l = rte_pktmbuf_data_len(m) - offset;
- if (srclen <= l) {
- if (CMAC_Update(ctx, (unsigned char *)src, srclen) != 1)
- goto process_auth_err;
- goto process_auth_final;
- }
-
- if (CMAC_Update(ctx, (unsigned char *)src, l) != 1)
- goto process_auth_err;
-
- n -= l;
-
- for (m = m->next; (m != NULL) && (n > 0); m = m->next) {
- src = rte_pktmbuf_mtod(m, uint8_t *);
- l = rte_pktmbuf_data_len(m) < n ? rte_pktmbuf_data_len(m) : n;
- if (CMAC_Update(ctx, (unsigned char *)src, l) != 1)
- goto process_auth_err;
- n -= l;
- }
-
-process_auth_final:
- if (CMAC_Final(ctx, dst, (size_t *)&dstlen) != 1)
- goto process_auth_err;
- return 0;
-
-process_auth_err:
- OPENSSL_LOG(ERR, "Process openssl cmac auth failed");
- return -EINVAL;
-}
-# endif
/*----------------------------------------------------------------------------*/
static inline EVP_CIPHER_CTX *
@@ -1695,7 +1498,7 @@ get_local_cipher_ctx(struct openssl_session *sess, struct openssl_qp *qp)
/* EVP_CIPHER_CTX_dup() added in OSSL 3.2 */
*lctx = EVP_CIPHER_CTX_dup(sess->cipher.ctx);
return *lctx;
-#elif OPENSSL_VERSION_NUMBER >= 0x30000000L
+#else
if (sess->chain_order == OPENSSL_CHAIN_COMBINED) {
/* AESNI special-cased to use openssl_aesni_ctx_clone()
* to allow for working around lack of
@@ -1706,10 +1509,10 @@ get_local_cipher_ctx(struct openssl_session *sess, struct openssl_qp *qp)
*lctx = NULL;
return *lctx;
}
-#endif
*lctx = EVP_CIPHER_CTX_new();
EVP_CIPHER_CTX_copy(*lctx, sess->cipher.ctx);
+#endif
}
return *lctx;
@@ -1737,11 +1540,7 @@ get_local_auth_ctx(struct openssl_session *sess, struct openssl_qp *qp)
return *lctx;
}
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
static inline EVP_MAC_CTX *
-#else
-static inline HMAC_CTX *
-#endif
get_local_hmac_ctx(struct openssl_session *sess, struct openssl_qp *qp)
{
#if (OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x30003000L)
@@ -1759,31 +1558,16 @@ get_local_hmac_ctx(struct openssl_session *sess, struct openssl_qp *qp)
if (sess->ctx_copies_len == 0)
return sess->auth.hmac.ctx;
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
- EVP_MAC_CTX **lctx =
-#else
- HMAC_CTX **lctx =
-#endif
- &sess->qp_ctx[qp->id].hmac;
+ EVP_MAC_CTX **lctx = &sess->qp_ctx[qp->id].hmac;
- if (unlikely(*lctx == NULL)) {
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
+ if (unlikely(*lctx == NULL))
*lctx = EVP_MAC_CTX_dup(sess->auth.hmac.ctx);
-#else
- *lctx = HMAC_CTX_new();
- HMAC_CTX_copy(*lctx, sess->auth.hmac.ctx);
-#endif
- }
return *lctx;
#endif
}
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
static inline EVP_MAC_CTX *
-#else
-static inline CMAC_CTX *
-#endif
get_local_cmac_ctx(struct openssl_session *sess, struct openssl_qp *qp)
{
#if (OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x30003000L)
@@ -1801,21 +1585,10 @@ get_local_cmac_ctx(struct openssl_session *sess, struct openssl_qp *qp)
if (sess->ctx_copies_len == 0)
return sess->auth.cmac.ctx;
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
- EVP_MAC_CTX **lctx =
-#else
- CMAC_CTX **lctx =
-#endif
- &sess->qp_ctx[qp->id].cmac;
+ EVP_MAC_CTX **lctx = &sess->qp_ctx[qp->id].cmac;
- if (unlikely(*lctx == NULL)) {
-#if OPENSSL_VERSION_NUMBER >= 0x30000000L
+ if (unlikely(*lctx == NULL))
*lctx = EVP_MAC_CTX_dup(sess->auth.cmac.ctx);
-#else
- *lctx = CMAC_CTX_new();
- CMAC_CTX_copy(*lctx, sess->auth.cmac.ctx);
-#endif
- }
return *lctx;
#endif
@@ -2055,13 +1828,8 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op,
uint8_t *dst;
int srclen, status;
EVP_MD_CTX *ctx_a;
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
EVP_MAC_CTX *ctx_h;
EVP_MAC_CTX *ctx_c;
-# else
- HMAC_CTX *ctx_h;
- CMAC_CTX *ctx_c;
-# endif
srclen = op->sym->auth.data.length;
@@ -2076,30 +1844,18 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op,
break;
case OPENSSL_AUTH_AS_HMAC:
ctx_h = get_local_hmac_ctx(sess, qp);
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
status = process_openssl_auth_mac(mbuf_src, dst,
op->sym->auth.data.offset, srclen,
ctx_h);
-# else
- status = process_openssl_auth_hmac(mbuf_src, dst,
- op->sym->auth.data.offset, srclen,
- ctx_h);
-# endif
#if (OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x30003000L)
EVP_MAC_CTX_free(ctx_h);
#endif
break;
case OPENSSL_AUTH_AS_CMAC:
ctx_c = get_local_cmac_ctx(sess, qp);
-# if OPENSSL_VERSION_NUMBER >= 0x30000000L
status = process_openssl_auth_mac(mbuf_src, dst,
op->sym->auth.data.offset, srclen,
ctx_c);
-# else
- status = process_openssl_auth_cmac(mbuf_src, dst,
- op->sym->auth.data.offset, srclen,
- ctx_c);
-# endif
#if (OPENSSL_VERSION_NUMBER >= 0x30000000L && OPENSSL_VERSION_NUMBER < 0x30003000L)
EVP_MAC_CTX_free(ctx_c);
#endif
@@ -2130,7 +1886,6 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op,
}
/* process dsa sign operation */
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
static int
process_openssl_dsa_sign_op_evp(struct rte_crypto_op *cop,
struct openssl_asym_session *sess)
@@ -2296,92 +2051,8 @@ process_openssl_dsa_verify_op_evp(struct rte_crypto_op *cop,
return ret;
}
-#else
-static int
-process_openssl_dsa_sign_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- struct rte_crypto_dsa_op_param *op = &cop->asym->dsa;
- DSA *dsa = sess->u.s.dsa;
- DSA_SIG *sign = NULL;
-
- sign = DSA_do_sign(op->message.data,
- op->message.length,
- dsa);
-
- if (sign == NULL) {
- OPENSSL_LOG(ERR, "%s:%d", __func__, __LINE__);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- } else {
- const BIGNUM *r = NULL, *s = NULL;
- get_dsa_sign(sign, &r, &s);
-
- op->r.length = BN_bn2bin(r, op->r.data);
- op->s.length = BN_bn2bin(s, op->s.data);
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
- }
-
- DSA_SIG_free(sign);
-
- return 0;
-}
-
-/* process dsa verify operation */
-static int
-process_openssl_dsa_verify_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- struct rte_crypto_dsa_op_param *op = &cop->asym->dsa;
- DSA *dsa = sess->u.s.dsa;
- int ret;
- DSA_SIG *sign = DSA_SIG_new();
- BIGNUM *r = NULL, *s = NULL;
- BIGNUM *pub_key = NULL;
-
- if (sign == NULL) {
- OPENSSL_LOG(ERR, " %s:%d", __func__, __LINE__);
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
-
- r = BN_bin2bn(op->r.data,
- op->r.length,
- r);
- s = BN_bin2bn(op->s.data,
- op->s.length,
- s);
- pub_key = BN_bin2bn(op->y.data,
- op->y.length,
- pub_key);
- if (!r || !s || !pub_key) {
- BN_free(r);
- BN_free(s);
- BN_free(pub_key);
-
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
- set_dsa_sign(sign, r, s);
- set_dsa_pub_key(dsa, pub_key);
-
- ret = DSA_do_verify(op->message.data,
- op->message.length,
- sign,
- dsa);
-
- if (ret != 1)
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- else
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
-
- DSA_SIG_free(sign);
-
- return 0;
-}
-#endif
/* process dh operation */
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
static int
process_openssl_dh_op_evp(struct rte_crypto_op *cop,
struct openssl_asym_session *sess)
@@ -2555,141 +2226,6 @@ process_openssl_dh_op_evp(struct rte_crypto_op *cop,
return ret;
}
-#else
-static int
-process_openssl_dh_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- struct rte_crypto_dh_op_param *op = &cop->asym->dh;
- struct rte_crypto_asym_op *asym_op = cop->asym;
- DH *dh_key = sess->u.dh.dh_key;
- BIGNUM *priv_key = NULL;
- int ret = 0;
-
- if (asym_op->dh.ke_type == RTE_CRYPTO_ASYM_KE_SHARED_SECRET_COMPUTE) {
- /* compute shared secret using peer public key
- * and current private key
- * shared secret = peer_key ^ priv_key mod p
- */
- BIGNUM *peer_key = NULL;
-
- /* copy private key and peer key and compute shared secret */
- peer_key = BN_bin2bn(op->pub_key.data,
- op->pub_key.length,
- peer_key);
- if (peer_key == NULL) {
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
- priv_key = BN_bin2bn(op->priv_key.data,
- op->priv_key.length,
- priv_key);
- if (priv_key == NULL) {
- BN_free(peer_key);
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
- ret = set_dh_priv_key(dh_key, priv_key);
- if (ret) {
- OPENSSL_LOG(ERR, "Failed to set private key");
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- BN_free(peer_key);
- BN_free(priv_key);
- return 0;
- }
-
- ret = DH_compute_key(
- op->shared_secret.data,
- peer_key, dh_key);
- if (ret < 0) {
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- BN_free(peer_key);
- /* priv key is already loaded into dh,
- * let's not free that directly here.
- * DH_free() will auto free it later.
- */
- return 0;
- }
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
- op->shared_secret.length = ret;
- BN_free(peer_key);
- return 0;
- }
-
- /*
- * other options are public and private key generations.
- *
- * if user provides private key,
- * then first set DH with user provided private key
- */
- if (asym_op->dh.ke_type == RTE_CRYPTO_ASYM_KE_PUB_KEY_GENERATE &&
- op->priv_key.length) {
- /* generate public key using user-provided private key
- * pub_key = g ^ priv_key mod p
- */
-
- /* load private key into DH */
- priv_key = BN_bin2bn(op->priv_key.data,
- op->priv_key.length,
- priv_key);
- if (priv_key == NULL) {
- cop->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
- return -1;
- }
- ret = set_dh_priv_key(dh_key, priv_key);
- if (ret) {
- OPENSSL_LOG(ERR, "Failed to set private key");
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- BN_free(priv_key);
- return 0;
- }
- }
-
- /* generate public and private key pair.
- *
- * if private key already set, generates only public key.
- *
- * if private key is not already set, then set it to random value
- * and update internal private key.
- */
- if (!DH_generate_key(dh_key)) {
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return 0;
- }
-
- if (asym_op->dh.ke_type == RTE_CRYPTO_ASYM_KE_PUB_KEY_GENERATE) {
- const BIGNUM *pub_key = NULL;
-
- OPENSSL_LOG(DEBUG, "%s:%d update public key",
- __func__, __LINE__);
-
- /* get the generated keys */
- get_dh_pub_key(dh_key, &pub_key);
-
- /* output public key */
- op->pub_key.length = BN_bn2bin(pub_key,
- op->pub_key.data);
- }
-
- if (asym_op->dh.ke_type == RTE_CRYPTO_ASYM_KE_PRIV_KEY_GENERATE) {
- const BIGNUM *priv_key = NULL;
-
- OPENSSL_LOG(DEBUG, "%s:%d updated priv key",
- __func__, __LINE__);
-
- /* get the generated keys */
- get_dh_priv_key(dh_key, &priv_key);
-
- /* provide generated private key back to user */
- op->priv_key.length = BN_bn2bin(priv_key,
- op->priv_key.data);
- }
-
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
-
- return 0;
-}
-#endif
/* process modinv operation */
static int
@@ -2757,7 +2293,6 @@ process_openssl_modexp_op(struct rte_crypto_op *cop,
}
/* process rsa operations */
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
static int
process_openssl_rsa_op_evp(struct rte_crypto_op *cop,
struct openssl_asym_session *sess)
@@ -3333,133 +2868,7 @@ process_openssl_eddsa_op_evp(struct rte_crypto_op *cop,
return ret;
}
-#else
-static int
-process_openssl_rsa_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- int ret = 0;
- struct rte_crypto_asym_op *op = cop->asym;
- RSA *rsa = sess->u.r.rsa;
- uint32_t pad = sess->u.r.pad;
- uint8_t *tmp;
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
-
- switch (pad) {
- case RTE_CRYPTO_RSA_PADDING_PKCS1_5:
- pad = RSA_PKCS1_PADDING;
- break;
- case RTE_CRYPTO_RSA_PADDING_NONE:
- pad = RSA_NO_PADDING;
- break;
- default:
- cop->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
- OPENSSL_LOG(ERR,
- "rsa pad type not supported %d", pad);
- return 0;
- }
-
- switch (op->rsa.op_type) {
- case RTE_CRYPTO_ASYM_OP_ENCRYPT:
- ret = RSA_public_encrypt(op->rsa.message.length,
- op->rsa.message.data,
- op->rsa.cipher.data,
- rsa,
- pad);
-
- if (ret > 0)
- op->rsa.cipher.length = ret;
- OPENSSL_LOG(DEBUG,
- "length of encrypted text %d", ret);
- break;
-
- case RTE_CRYPTO_ASYM_OP_DECRYPT:
- ret = RSA_private_decrypt(op->rsa.cipher.length,
- op->rsa.cipher.data,
- op->rsa.message.data,
- rsa,
- pad);
- if (ret > 0)
- op->rsa.message.length = ret;
- break;
-
- case RTE_CRYPTO_ASYM_OP_SIGN:
- ret = RSA_private_encrypt(op->rsa.message.length,
- op->rsa.message.data,
- op->rsa.sign.data,
- rsa,
- pad);
- if (ret > 0)
- op->rsa.sign.length = ret;
- break;
-
- case RTE_CRYPTO_ASYM_OP_VERIFY:
- tmp = rte_malloc(NULL, op->rsa.sign.length, 0);
- if (tmp == NULL) {
- OPENSSL_LOG(ERR, "Memory allocation failed");
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- break;
- }
- ret = RSA_public_decrypt(op->rsa.sign.length,
- op->rsa.sign.data,
- tmp,
- rsa,
- pad);
-
- OPENSSL_LOG(DEBUG,
- "Length of public_decrypt %d "
- "length of message %zd",
- ret, op->rsa.message.length);
- if ((ret <= 0) || (CRYPTO_memcmp(tmp, op->rsa.message.data,
- op->rsa.message.length))) {
- OPENSSL_LOG(ERR, "RSA sign Verification failed");
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- }
- rte_free(tmp);
- break;
-
- default:
- /* allow ops with invalid args to be pushed to
- * completion queue
- */
- cop->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
- break;
- }
-
- if (ret < 0)
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
-
- return 0;
-}
-
-static int
-process_openssl_ecfpm_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- RTE_SET_USED(cop);
- RTE_SET_USED(sess);
- return -ENOTSUP;
-}
-
-static int
-process_openssl_sm2_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- RTE_SET_USED(cop);
- RTE_SET_USED(sess);
- return -ENOTSUP;
-}
-
-static int
-process_openssl_eddsa_op(struct rte_crypto_op *cop,
- struct openssl_asym_session *sess)
-{
- RTE_SET_USED(cop);
- RTE_SET_USED(sess);
- return -ENOTSUP;
-}
-#endif
#if (OPENSSL_VERSION_NUMBER >= 0x30500000L)
static int
@@ -4085,14 +3494,12 @@ mldsa_sign_op_evp(struct rte_crypto_op *cop,
case RTE_CRYPTO_AUTH_SHA3_512:
check_md = EVP_sha3_512();
break;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
case RTE_CRYPTO_AUTH_SHAKE_128:
check_md = EVP_shake128();
break;
case RTE_CRYPTO_AUTH_SHAKE_256:
check_md = EVP_shake256();
break;
-#endif
default:
break;
}
@@ -4328,11 +3735,7 @@ process_asym_op(struct openssl_qp *qp, struct rte_crypto_op *op,
switch (sess->xfrm_type) {
case RTE_CRYPTO_ASYM_XFORM_RSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_rsa_op_evp(op, sess);
-# else
- retval = process_openssl_rsa_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_MODEX:
retval = process_openssl_modexp_op(op, sess);
@@ -4341,51 +3744,26 @@ process_asym_op(struct openssl_qp *qp, struct rte_crypto_op *op,
retval = process_openssl_modinv_op(op, sess);
break;
case RTE_CRYPTO_ASYM_XFORM_DH:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_dh_op_evp(op, sess);
-# else
- retval = process_openssl_dh_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_DSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
if (op->asym->dsa.op_type == RTE_CRYPTO_ASYM_OP_SIGN)
retval = process_openssl_dsa_sign_op_evp(op, sess);
else if (op->asym->dsa.op_type ==
RTE_CRYPTO_ASYM_OP_VERIFY)
retval =
process_openssl_dsa_verify_op_evp(op, sess);
-#else
- if (op->asym->dsa.op_type == RTE_CRYPTO_ASYM_OP_SIGN)
- retval = process_openssl_dsa_sign_op(op, sess);
- else if (op->asym->dsa.op_type ==
- RTE_CRYPTO_ASYM_OP_VERIFY)
- retval =
- process_openssl_dsa_verify_op(op, sess);
else
op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_ECFPM:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_ecfpm_op_evp(op, sess);
-#else
- retval = process_openssl_ecfpm_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_SM2:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_sm2_op_evp(op, sess);
-#else
- retval = process_openssl_sm2_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_EDDSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
retval = process_openssl_eddsa_op_evp(op, sess);
-#else
- retval = process_openssl_eddsa_op(op, sess);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_ML_KEM:
#if (OPENSSL_VERSION_NUMBER >= 0x30500000L)
@@ -4590,13 +3968,12 @@ cryptodev_openssl_create(const char *name,
rte_cryptodev_pmd_probing_finish(dev);
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
/* Load legacy provider
* Some algorithms are no longer available in earlier version of openssl,
* unless the legacy provider explicitly loaded. e.g. DES
*/
ossl_legacy_provider_load();
-# endif
+
return 0;
init_error:
@@ -4645,9 +4022,8 @@ cryptodev_openssl_remove(struct rte_vdev_device *vdev)
if (cryptodev == NULL)
return -ENODEV;
-# if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
ossl_legacy_provider_unload();
-# endif
+
return rte_cryptodev_pmd_destroy(cryptodev);
}
diff --git a/drivers/crypto/openssl/rte_openssl_pmd_ops.c b/drivers/crypto/openssl/rte_openssl_pmd_ops.c
index 4e5fb07bb2..d927cc5228 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd_ops.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd_ops.c
@@ -10,11 +10,9 @@
#include "openssl_pmd_private.h"
#include "compat.h"
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
#include <openssl/provider.h>
#include <openssl/core_names.h>
#include <openssl/param_build.h>
-#endif
static const struct rte_cryptodev_capabilities openssl_pmd_capabilities[] = {
{ /* MD5 HMAC */
@@ -457,7 +455,6 @@ static const struct rte_cryptodev_capabilities openssl_pmd_capabilities[] = {
}, }
}, }
},
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
{ /* SHAKE_128 */
.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
{.sym = {
@@ -500,7 +497,6 @@ static const struct rte_cryptodev_capabilities openssl_pmd_capabilities[] = {
}, }
}, }
},
-#endif
{ /* AES CBC */
.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
{.sym = {
@@ -1222,7 +1218,6 @@ static int openssl_set_asym_session_parameters(
goto err_rsa;
asym_session->u.r.pad = xform->rsa.padding.type;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD * param_bld = OSSL_PARAM_BLD_new();
if (!param_bld) {
OPENSSL_LOG(ERR, "failed to allocate resources");
@@ -1323,79 +1318,7 @@ static int openssl_set_asym_session_parameters(
OSSL_PARAM_BLD_free(param_bld);
OSSL_PARAM_free(params);
ret = 0;
-#else
- RSA *rsa = RSA_new();
- if (rsa == NULL)
- goto err_rsa;
-
- if (xform->rsa.d.length > 0) {
- d = BN_bin2bn(
- (const unsigned char *)xform->rsa.d.data,
- xform->rsa.d.length,
- d);
- if (!d) {
- RSA_free(rsa);
- goto err_rsa;
- }
- }
-
- if (xform->rsa.key_type == RTE_RSA_KEY_TYPE_QT) {
- p = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.p.data,
- xform->rsa.qt.p.length,
- p);
- q = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.q.data,
- xform->rsa.qt.q.length,
- q);
- dmp1 = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.dP.data,
- xform->rsa.qt.dP.length,
- dmp1);
- dmq1 = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.dQ.data,
- xform->rsa.qt.dQ.length,
- dmq1);
- iqmp = BN_bin2bn((const unsigned char *)
- xform->rsa.qt.qInv.data,
- xform->rsa.qt.qInv.length,
- iqmp);
- if (!p || !q || !dmp1 || !dmq1 || !iqmp) {
- RSA_free(rsa);
- goto err_rsa;
- }
- ret = set_rsa_params(rsa, p, q);
- if (ret) {
- OPENSSL_LOG(ERR,
- "failed to set rsa params");
- RSA_free(rsa);
- goto err_rsa;
- }
- ret = set_rsa_crt_params(rsa, dmp1, dmq1, iqmp);
- if (ret) {
- OPENSSL_LOG(ERR,
- "failed to set crt params");
- RSA_free(rsa);
- /*
- * set already populated params to NULL
- * as its freed by call to RSA_free
- */
- p = q = NULL;
- goto err_rsa;
- }
- }
-
- ret = set_rsa_keys(rsa, n, e, d);
- if (ret) {
- OPENSSL_LOG(ERR, "Failed to load rsa keys");
- RSA_free(rsa);
- return ret;
- }
- asym_session->u.r.rsa = rsa;
- asym_session->xfrm_type = RTE_CRYPTO_ASYM_XFORM_RSA;
- break;
-#endif
err_rsa:
BN_clear_free(n);
BN_clear_free(e);
@@ -1469,7 +1392,6 @@ static int openssl_set_asym_session_parameters(
case RTE_CRYPTO_ASYM_XFORM_DH:
{
DH *dh = NULL;
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
BIGNUM **p = &asym_session->u.dh.p;
BIGNUM **g = &asym_session->u.dh.g;
@@ -1520,51 +1442,18 @@ static int openssl_set_asym_session_parameters(
asym_session->u.dh.param_bld = param_bld;
asym_session->u.dh.param_bld_peer = param_bld_peer;
-#else
- BIGNUM *p = NULL;
- BIGNUM *g = NULL;
-
- p = BN_bin2bn((const unsigned char *)
- xform->dh.p.data,
- xform->dh.p.length,
- p);
- g = BN_bin2bn((const unsigned char *)
- xform->dh.g.data,
- xform->dh.g.length,
- g);
- if (!p || !g)
- goto err_dh;
-
- dh = DH_new();
- if (dh == NULL) {
- OPENSSL_LOG(ERR,
- "failed to allocate resources");
- goto err_dh;
- }
- ret = set_dh_params(dh, p, g);
- if (ret) {
- DH_free(dh);
- goto err_dh;
- }
-#endif
asym_session->u.dh.dh_key = dh;
asym_session->xfrm_type = RTE_CRYPTO_ASYM_XFORM_DH;
break;
err_dh:
OPENSSL_LOG(ERR, " failed to set dh params");
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
BN_free(*p);
BN_free(*g);
-#else
- BN_free(p);
- BN_free(g);
-#endif
return -1;
}
case RTE_CRYPTO_ASYM_XFORM_DSA:
{
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
BIGNUM **p = &asym_session->u.s.p;
BIGNUM **g = &asym_session->u.s.g;
BIGNUM **q = &asym_session->u.s.q;
@@ -1615,85 +1504,16 @@ static int openssl_set_asym_session_parameters(
asym_session->u.s.param_bld = param_bld;
break;
-#else
- BIGNUM *p = NULL, *g = NULL;
- BIGNUM *q = NULL, *priv_key = NULL;
- BIGNUM *pub_key = BN_new();
- BN_zero(pub_key);
-
- p = BN_bin2bn((const unsigned char *)
- xform->dsa.p.data,
- xform->dsa.p.length,
- p);
-
- g = BN_bin2bn((const unsigned char *)
- xform->dsa.g.data,
- xform->dsa.g.length,
- g);
-
- q = BN_bin2bn((const unsigned char *)
- xform->dsa.q.data,
- xform->dsa.q.length,
- q);
- if (!p || !q || !g)
- goto err_dsa;
-
- priv_key = BN_bin2bn((const unsigned char *)
- xform->dsa.x.data,
- xform->dsa.x.length,
- priv_key);
- if (priv_key == NULL)
- goto err_dsa;
-
- DSA *dsa = DSA_new();
- if (dsa == NULL) {
- OPENSSL_LOG(ERR,
- " failed to allocate resources");
- goto err_dsa;
- }
-
- ret = set_dsa_params(dsa, p, q, g);
- if (ret) {
- DSA_free(dsa);
- OPENSSL_LOG(ERR, "Failed to dsa params");
- goto err_dsa;
- }
-
- /*
- * openssl 1.1.0 mandate that public key can't be
- * NULL in very first call. so set a dummy pub key.
- * to keep consistency, lets follow same approach for
- * both versions
- */
- /* just set dummy public for very 1st call */
- ret = set_dsa_keys(dsa, pub_key, priv_key);
- if (ret) {
- DSA_free(dsa);
- OPENSSL_LOG(ERR, "Failed to set keys");
- goto err_dsa;
- }
- asym_session->u.s.dsa = dsa;
- asym_session->xfrm_type = RTE_CRYPTO_ASYM_XFORM_DSA;
- break;
-#endif
err_dsa:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
BN_free(*p);
BN_free(*q);
BN_free(*g);
BN_free(*priv_key);
-#else
- BN_free(p);
- BN_free(q);
- BN_free(g);
- BN_free(priv_key);
-#endif
BN_free(pub_key);
return -1;
}
case RTE_CRYPTO_ASYM_XFORM_ECFPM:
{
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
EC_GROUP *ecgrp = NULL;
asym_session->xfrm_type = xform->xform_type;
@@ -1727,14 +1547,9 @@ static int openssl_set_asym_session_parameters(
asym_session->u.ec.curve_id = xform->ec.curve_id;
asym_session->u.ec.group = ecgrp;
break;
-#else
- OPENSSL_LOG(WARNING, "ECFPM unsupported for OpenSSL Version < 3.0");
- return -ENOTSUP;
-#endif
}
case RTE_CRYPTO_ASYM_XFORM_SM2:
{
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
#ifndef OPENSSL_NO_SM2
OSSL_PARAM_BLD *param_bld = NULL;
OSSL_PARAM *params = NULL;
@@ -1818,10 +1633,6 @@ static int openssl_set_asym_session_parameters(
#else
OPENSSL_LOG(WARNING, "SM2 unsupported in current OpenSSL Version");
return -ENOTSUP;
-#endif
-#else
- OPENSSL_LOG(WARNING, "SM2 unsupported for OpenSSL Version < 3.0");
- return -ENOTSUP;
#endif
}
case RTE_CRYPTO_ASYM_XFORM_EDDSA:
@@ -1983,12 +1794,7 @@ static void openssl_reset_asym_session(struct openssl_asym_session *sess)
{
switch (sess->xfrm_type) {
case RTE_CRYPTO_ASYM_XFORM_RSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
EVP_PKEY_CTX_free(sess->u.r.ctx);
-#else
- if (sess->u.r.rsa)
- RSA_free(sess->u.r.rsa);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_MODEX:
if (sess->u.e.ctx) {
@@ -2003,35 +1809,23 @@ static void openssl_reset_asym_session(struct openssl_asym_session *sess)
}
break;
case RTE_CRYPTO_ASYM_XFORM_DH:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD_free(sess->u.dh.param_bld);
OSSL_PARAM_BLD_free(sess->u.dh.param_bld_peer);
sess->u.dh.param_bld = NULL;
sess->u.dh.param_bld_peer = NULL;
-#else
- if (sess->u.dh.dh_key)
- DH_free(sess->u.dh.dh_key);
-#endif
BN_clear_free(sess->u.dh.p);
BN_clear_free(sess->u.dh.g);
break;
case RTE_CRYPTO_ASYM_XFORM_DSA:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_BLD_free(sess->u.s.param_bld);
sess->u.s.param_bld = NULL;
BN_clear_free(sess->u.s.p);
BN_clear_free(sess->u.s.q);
BN_clear_free(sess->u.s.g);
BN_clear_free(sess->u.s.priv_key);
-#else
- if (sess->u.s.dsa)
- DSA_free(sess->u.s.dsa);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_SM2:
-#if (OPENSSL_VERSION_NUMBER >= 0x30000000L)
OSSL_PARAM_free(sess->u.sm2.params);
-#endif
break;
case RTE_CRYPTO_ASYM_XFORM_EDDSA:
#if (OPENSSL_VERSION_NUMBER >= 0x30300000L)
--
2.43.0
^ permalink raw reply related
* RE: [EXTERNAL] Re: [PATCH v3 0/2] net/mana: add device reset support
From: Wei Hu @ 2026-05-28 7:30 UTC (permalink / raw)
To: Stephen Hemminger, Wei Hu; +Cc: dev@dpdk.org, Long Li
In-Reply-To: <20260526123747.46871467@phoenix.local>
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday, May 27, 2026 3:38 AM
> To: Wei Hu <weh@linux.microsoft.com>
> Cc: dev@dpdk.org; Long Li <longli@microsoft.com>; Wei Hu
> <weh@microsoft.com>
> Subject: [EXTERNAL] Re: [PATCH v3 0/2] net/mana: add device reset support
>
>
> Warnings
>
> 3. Secondary process: qsbr does not actually quiesce secondary lcores.
>
> rte_rcu_qsbr_thread_register is only called from
> mana_dev_configure, which the secondary never runs. The
> secondary's rx_burst/tx_burst still call thread_online/offline
> against the shared qsv, but the secondary tids are unregistered,
> so they are invisible to rte_rcu_qsbr_check in the primary, and
> the secondary MP handler (mana_mp_reset_enter) does not call
> qsbr_check at all -- it just sets db_page = NULL and munmaps.
>
> The dev_state check at the top of secondary tx_burst is racy:
> the page can be munmapped after the in-loop read of db_page but
> before the doorbell write at the bottom. The "All secondary
> threads are quiescent" log line in mana_mp_reset_enter is not
> true.
>
> The secondary needs a real reader-side primitive -- its own
> qsbr with secondary lcore registration, or an rwlock the MP
> handler takes before munmap.
>
Thanks for the v3 review, @Stephen. I will send out v4 to incorporate most
of the review comments except for this one.
The review on this point is not correct. Here I am providing analysis from
AI and my own test results to show why.
The concern is that "rte_rcu_qsbr_thread_register is only called
from mana_dev_configure, which the secondary never runs", so
secondary tids are unregistered and invisible to rte_rcu_qsbr_check.
This is incorrect. The RCU thread IDs are per-queue, not per-process.
The tid used in rte_rcu_qsbr_thread_online/offline is:
- RX path: tid = rxq->rxq_idx (0 to num_queues-1)
- TX path: tid = num_queues + txq_idx (num_queues to 2*num_queues-1)
These tids are registered once by primary in mana_dev_configure
(mana.c:117), which calls rte_rcu_qsbr_thread_register for IDs
0..2*num_queues-1. The QSV (priv->dev_state_qsv) is allocated
with rte_zmalloc_socket in shared hugepage memory (mana.c:2207).
The queue structures (rxq, txq) also live in shared memory via
dev->data->rx_queues[] / tx_queues[].
When a secondary lcore polls a queue, it calls
rte_rcu_qsbr_thread_online(qsv, tid) with the SAME shared QSV
and SAME queue-based tid. DPDK's fundamental model requires that
a given queue is polled by exactly one lcore at a time (queues
are not thread-safe), so the tid maps 1:1 to a lcore regardless
of which process that lcore belongs to. The tid IS registered ―
registration is a property of the QSV structure, not of the process.
The reset synchronization flow (mana_reset_enter, mana.c:1458-1488):
1. Primary: dev_state = MANA_DEV_RESET_ENTER (release store)
2. Primary: rte_rcu_qsbr_start + rte_rcu_qsbr_check ― blocks
until ALL registered tids have passed through thread_offline
3. Secondary lcores in rx_burst/tx_burst eventually call
rte_rcu_qsbr_thread_offline, and on re-entry see
dev_state != ACTIVE → return 0 without touching db_page
4. Only after QSBR check passes (ALL tids quiescent, including
secondary's) does primary proceed to mana_dev_stop (line 1472)
5. Primary sends MANA_MP_REQ_RESET_ENTER IPC (line 1481)
6. Secondary MP handler: sets db_page = NULL, munmaps
The alleged race ― "db_page can be munmapped after the in-loop
read but before the doorbell write" ― cannot happen because:
- db_page is munmapped at step 6, which is AFTER step 2 (QSBR).
- After QSBR passes, any secondary re-entry into tx_burst/rx_burst
reads dev_state with acquire ordering (tx.c:214, rx.c:468),
sees RESET_ENTER, and returns immediately without using the
local db_page copy.
- The memory ordering is sound: primary's release store of
dev_state (step 1) happens-before QSBR start (step 2).
Secondary's thread_offline (release) synchronizes-with
primary's QSBR check (acquire). On secondary re-entry,
thread_online + acquire load of dev_state sees the update.
The secondary MP handler does NOT need its own QSBR check ―
the primary's check at step 2 already covers all data path
threads including secondary's.
Note: The log "All secondary threads are quiescent" in
mana_mp_reset_enter is misleading and has been removed.
The actual quiescence is enforced by the primary's QSBR
check before IPC is sent.
Followings are from my own test. I added the debug log in mana_tx_burst() call.
diff --git a/drivers/net/mana/mana.h b/drivers/net/mana/mana.h
index 115bc722f4..038a3d9e00 100644
--- a/drivers/net/mana/mana.h
+++ b/drivers/net/mana/mana.h
@@ -454,6 +454,9 @@ struct mana_txq {
struct mana_stats stats;
unsigned int socket;
unsigned int txq_idx;
+#if 1
+ unsigned int call_cnt;
+#endif
};
struct mana_rxq {
diff --git a/drivers/net/mana/tx.c b/drivers/net/mana/tx.c
index 3b07a8b9a6..3ae825190e 100644
--- a/drivers/net/mana/tx.c
+++ b/drivers/net/mana/tx.c
@@ -210,6 +210,14 @@ mana_tx_burst(void *dpdk_txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
}
rte_rcu_qsbr_thread_online(dstate_qsv, tid);
+#if 1
+ if (txq->call_cnt++ % 1000000 == 0) {
+ if (rte_eal_process_type() == RTE_PROC_SECONDARY)
+ DP_LOG(ERR, "secondary tid %u online", tid);
+ else
+ DP_LOG(ERR, "primary tid %u online", tid);
+ }
+#endif
if (unlikely(rte_atomic_load_explicit(&priv->dev_state,
rte_memory_order_acquire) != MANA_DEV_ACTIVE || !db_page)) {
Primary command:
dpdk-testpmd --log-level='.*,8' -l 2-4 ... --proc-type=primary -- --nb-cores 2
--forward-mode=txonly ... --txq=4 --rxq=4 ... --num-procs=2 --proc-id 0
Secondary command:
dpdk-testpmd --log-level='.*,8' -l 6-8 ... --proc-type=secondary -- --nb-cores 2
--forward-mode=txonly ... --txq=4 --rxq=4 ... --num-procs=2 --proc-id 1
Here in both commands, I specify the number of txq and rxq to be 4.
After primary start, the log shows 8 threads total were registered in mana_dev_configure
and their tids:
Configuring Port 0 (socket 0)
MANA_DRIVER: mana_dev_configure(): priv 0x1003cdc80, port 0, dev port 1, num_queues: 4
MANA_DRIVER: mana_dev_configure(): Register thread 0x0 for priv 0x1003cdc80, port 0
MANA_DRIVER: mana_dev_configure(): Register thread 0x1 for priv 0x1003cdc80, port 0
MANA_DRIVER: mana_dev_configure(): Register thread 0x2 for priv 0x1003cdc80, port 0
MANA_DRIVER: mana_dev_configure(): Register thread 0x3 for priv 0x1003cdc80, port 0
MANA_DRIVER: mana_dev_configure(): Register thread 0x4 for priv 0x1003cdc80, port 0
MANA_DRIVER: mana_dev_configure(): Register thread 0x5 for priv 0x1003cdc80, port 0
MANA_DRIVER: mana_dev_configure(): Register thread 0x6 for priv 0x1003cdc80, port 0
MANA_DRIVER: mana_dev_configure(): Register thread 0x7 for priv 0x1003cdc80, port 0
ETHDEV: Port 0 Rx offload RSS_HASH is not requested but enabled
Note tid 0x0 to 0x3 are rx threads, 0x4 to 0x7 are tx threads. Then the primary
only printed out the tid 4 and 5 sere online:
MANA_DRIVER: primary tid 4 online
MANA_DRIVER: primary tid 5 online
Then I started the secondary and saw tid 6 and 7 were online in its mana_tx_burst call:
MANA_DRIVER: secondary tid 7 online
MANA_DRIVER: secondary tid 6 online
MANA_DRIVER: secondary tid 7 online
This simply means primary has registered all threads which would be up in both
primary and secondary in mana_ddev_configure. Primary only starts half of the
threads since --num-procs=2 passed in already tells primary that there will be
a secondary to start.
When the secondary starts, it takes the rest two tx and rx threads. All these threads
have already been registered in mana_dev_configure() by calling
rte_rcu_qsbr_thread_register() in the primary.
So, the review comments on this point is incorrect. All threads are covered and
there is no race at all.
Thanks,
Wei
^ permalink raw reply related
* RE: [PATCH v5 0/3] eal/topology: introduce topology-aware lcore grouping
From: Varghese, Vipin @ 2026-05-28 7:17 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev@dpdk.org, Tummala, Sivaprasad, konstantin.ananyev@huawei.com,
wathsala.vithanage@arm.com, bruce.richardson@intel.com,
viktorin@cesnet.cz, mb@smartsharesystems.com
In-Reply-To: <20260414132212.1726f08d@phoenix.local>
AMD General
Sharing v6 with fixes and cache-id fetch shortly.
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday, April 15, 2026 1:52 AM
> To: Varghese, Vipin <Vipin.Varghese@amd.com>
> Cc: dev@dpdk.org; Tummala, Sivaprasad <Sivaprasad.Tummala@amd.com>;
> konstantin.ananyev@huawei.com; wathsala.vithanage@arm.com;
> bruce.richardson@intel.com; viktorin@cesnet.cz; mb@smartsharesystems.com
> Subject: Re: [PATCH v5 0/3] eal/topology: introduce topology-aware lcore grouping
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Wed, 15 Apr 2026 01:08:18 +0530
> Vipin Varghese <vipin.varghese@amd.com> wrote:
>
> > This series introduces a topology library that groups DPDK lcores
> > based on CPU cache hierarchy and NUMA topology. The goal is to provide
> > a stable and explicit API that allows applications to select lcores
> > with better locality and cache sharing characteristics.
> >
> > The series includes:
> > - EAL support for topology discovery using hwloc and domain-based lcore
> > grouping (L1/L2/L3/L4/NUMA)
> > - Topology-aware test cases validating API behavior and edge conditions
> > - Programmer’s guide describing the topology library and APIs
> >
> > The API is marked experimental and does not change existing lcore
> > behavior unless explicitly used by the application.
> >
> > Changes in v5:
> > - Addressed review comments from v4
> > - Fixed ARM cross-compilation issues
> > - Cleaned up domain iteration and error handling
> > - Updated tests to cover domain edge cases
> > - Documentation refinements and API usage clarification
> >
> > Changes in v4:
> > - Corrected domain selection semantics
> > - Updated example usage
> > - Fixed naming and typo issues
> >
> > Changes in v3:
> > - Fixed macro naming (USE_NO_TOPOLOGY)
> > - Minor cleanups based on early feedback
> >
> > Tested on:
> > - AMD EPYC (Milan, Genoa, Siena, Turin, Turin-Dense, Sorano)
> > - Intel Xeon (SPR-SP, GNR-SP)
> > - ARM Ampere
> > - NVIDIA Grace Superchip
> >
> > Dependencies:
> > - hwloc-dev (tested with 2.10.0)
> >
> > Patch breakdown:
> > 1/3 eal/topology: add topology grouping for lcores
> > 2/3 app: add topology-aware test cases
> > 3/3 doc: add topology library documentation
> >
> > Future Work:
> > - integrate into examples
> > -- hellowrld: ready
> > -- pkt-distributor: in-progress
> > -- l2fwd: ready
> > -- l3fwd: to start
> > -- eventdevpipeline: PoC ready
> > - integrate topology test
> > -- crypto: yet to start
> > -- compression: yet to start
> > -- dma: PoC ready
> > - add new features for
> > -- PQoS: yet to start
> > -- Data Injection: PoC with BRDCM Thor-2 ready
> >
> > Tested OS: Linux only, need help with BSD and Windows
> >
> > Tested with and without hwloc-dev library for
> > - Ampere, aarch64, Neoverse-N1, NUMA-2, 256 CPU threads
> > - Grace superchip, aarch64, Neoverse-V2, NUMA-2, 144 CPU threads
> > - Intel GNR-SP, 6767P, NUMA-2, 256 Threads
> > - AMD EPYC Siena, 8534P, NUMA-1, 128 Threads
> > - AMD EPYC Sorano, 8635P, NUMA-1, 168 Threads
> >
> > Signed-off-by: Vipin Varghese <vipin.varghese@amd.com> ``
> >
> > Vipin Varghese (3):
> > eal/topology: add Topology grouping for lcores
> > app: add topology aware test case
> > doc: add new section topology
> >
> > app/test/meson.build | 1 +
> > app/test/test_ring_perf.c | 416 +++++++++++++-
> > app/test/test_stack_perf.c | 409 ++++++++++++++
> > app/test/test_topology.c | 676 ++++++++++++++++++++++
> > config/meson.build | 18 +
> > doc/api/doxy-api-index.md | 1 +
> > doc/guides/prog_guide/index.rst | 3 +-
> > doc/guides/prog_guide/topology_lib.rst | 155 +++++
> > lib/eal/common/eal_private.h | 74 +++
> > lib/eal/common/eal_topology.c | 747 +++++++++++++++++++++++++
> > lib/eal/common/meson.build | 1 +
> > lib/eal/freebsd/eal.c | 10 +-
> > lib/eal/include/meson.build | 1 +
> > lib/eal/include/rte_topology.h | 255 +++++++++
> > lib/eal/linux/eal.c | 7 +
> > lib/eal/meson.build | 4 +
> > 16 files changed, 2773 insertions(+), 5 deletions(-) create mode
> > 100644 app/test/test_topology.c create mode 100644
> > doc/guides/prog_guide/topology_lib.rst
> > create mode 100644 lib/eal/common/eal_topology.c create mode 100644
> > lib/eal/include/rte_topology.h
> >
>
> AI review reported lots of possible correctness problems:
>
>
> # DPDK Patch Review - bundle-1843-topology-v5.mbox
>
> ## Overview
>
> This patch series introduces topology awareness to DPDK, allowing applications to
> select logical cores based on CPU cache and I/O topology. The series adds a new
> `rte_topology` library with APIs to query and enumerate logical cores within topology
> domains (L1/L2/L3/L4/NUMA).
>
> ---
>
> ## Correctness Bugs (HIGHEST PRIORITY - report at >=50% confidence)
>
> ### Patch 1/3: lib/eal/common/eal_topology.c
>
> #### **Error: Resource leak on `eal_topology_map_layer()` failure**
>
> In `rte_eal_topology_init()`, if any call to `eal_topology_map_layer()` fails (returns <
> 0), the code calls `rte_eal_topology_release()` which frees previously allocated
> layers. However, the `hwloc_topology_t` handle `topo_cnfg.topology` is **NOT**
> destroyed before returning. This leaks the hwloc topology handle.
>
> **Location:** `lib/eal/common/eal_topology.c:685-712`
>
> ```c
> for (int i = 0; i < 5; i++) {
> *layers[i].count = hwloc_get_nbobjs_by_depth(topo_cnfg.topology,
> layers[i].depth);
> if (eal_topology_map_layer(topo_cnfg.topology, layers[i].depth, layers[i].count,
> layers[i].ptr, layers[i].total_cores, layers[i].name) < 0) {
> rte_eal_topology_release(); /* frees layer memory */
> return -1; /* BUG: topo_cnfg.topology NOT destroyed here */
> }
> }
>
> hwloc_topology_destroy(topo_cnfg.topology); /* only reached on success */
> topo_cnfg.topology = NULL; ```
>
> **Fix:** Destroy the topology before returning on error:
>
> ```c
> if (eal_topology_map_layer(...) < 0) {
> hwloc_topology_destroy(topo_cnfg.topology);
> topo_cnfg.topology = NULL;
> rte_eal_topology_release();
> return -1;
> }
> ```
>
> ---
>
> #### **Error: Potential use-after-free in `eal_topology_map_layer()` on partial
> allocation failure**
>
> In `eal_topology_map_layer()`, the code allocates `dm->cores` for each domain. If a
> later allocation in the same loop iteration fails (e.g., for domain `j+1`), the function
> returns -1 immediately **without freeing `dm->cores` already allocated for earlier
> domains**. The caller (`rte_eal_topology_init()`) then calls
> `rte_eal_topology_release()`, which expects `layer_ptr[j]` to be non-NULL but
> `layer_ptr[j]->cores` may be uninitialized or garbage if the allocation failed before
> that point. This can cause a use-after-free or double-free when
> `rte_eal_topology_release()` calls `rte_free(d->map[i]->cores)`.
>
> **Location:** `lib/eal/common/eal_topology.c:537-557`
>
> ```c
> for (uint16_t j = 0; j < *layer_cnt; j++) {
> hwloc_obj_t obj = hwloc_get_obj_by_depth(topology, depth, j);
> int cpu_count = hwloc_bitmap_weight(obj->cpuset);
> if (cpu_count == -1)
> continue;
>
> struct core_domain_mapping *dm =
> rte_zmalloc(NULL, sizeof(struct core_domain_mapping), 0);
> if (!dm)
> return -1; /* BUG: leaks layer_ptr array allocated by caller */
>
> (*layer_ptr)[j] = dm;
> CPU_ZERO(&dm->core_set);
> dm->core_count = 0;
>
> dm->cores = rte_malloc(NULL, sizeof(uint16_t) * cpu_count, 0);
> if (!dm->cores)
> return -1; /* BUG: leaks dm (just allocated) and previous entries */ } ```
>
> **Fix:** On allocation failure, free all previously allocated entries before returning:
>
> ```c
> if (!dm) {
> /* Free all previously allocated entries */
> for (uint16_t k = 0; k < j; k++) {
> if ((*layer_ptr)[k]) {
> rte_free((*layer_ptr)[k]->cores);
> rte_free((*layer_ptr)[k]);
> }
> }
> rte_free(*layer_ptr);
> return -1;
> }
>
> /* Same cleanup for dm->cores allocation failure */ if (!dm->cores) {
> rte_free(dm);
> for (uint16_t k = 0; k < j; k++) {
> if ((*layer_ptr)[k]) {
> rte_free((*layer_ptr)[k]->cores);
> rte_free((*layer_ptr)[k]);
> }
> }
> rte_free(*layer_ptr);
> return -1;
> }
> ```
>
> ---
>
> #### **Error: NULL pointer dereference in `eal_topology_map_layer()` when
> `hwloc_get_obj_by_depth()` returns NULL**
>
> In the second loop of `eal_topology_map_layer()` (lines 560-620), the code calls
> `hwloc_get_obj_by_depth(topology, depth, j)` and checks `if (!obj ||
> hwloc_bitmap_iszero(obj->cpuset))` to skip NULL objects. However, in the **inner
> loop** (lines 579-618), the code calls `hwloc_get_obj_by_depth(topology, depth, k)`
> and assigns it to `obj_core` but does **NOT** check if `obj_core` is NULL before
> calling `hwloc_bitmap_weight(obj_core->cpuset)` on line 581. If
> `hwloc_get_obj_by_depth()` returns NULL for domain `k`, this will dereference a
> NULL pointer.
>
> **Location:** `lib/eal/common/eal_topology.c:579-582`
>
> ```c
> for (uint16_t k = 0; k < *layer_cnt; k++) {
> hwloc_obj_t obj_core = hwloc_get_obj_by_depth(topology, depth, k);
> int cpu_count_core = hwloc_bitmap_weight(obj_core->cpuset); /* NULL deref if
> obj_core == NULL */
> if (cpu_count_core == -1)
> continue;
> ```
>
> **Fix:** Check `obj_core` before dereferencing:
>
> ```c
> hwloc_obj_t obj_core = hwloc_get_obj_by_depth(topology, depth, k); if (!obj_core)
> continue;
> int cpu_count_core = hwloc_bitmap_weight(obj_core->cpuset);
> if (cpu_count_core == -1)
> continue;
> ```
>
> ---
>
> #### **Error: Incorrect second argument to
> `rte_topo_get_nth_lcore_from_domain()` in `get_same_l1_domains()`
> (test_ring_perf.c and test_stack_perf.c)**
>
> In both `app/test/test_ring_perf.c:290` and `app/test/test_stack_perf.c:258`, the
> function `get_same_l1_domains()` calls:
>
> ```c
> id2 = rte_topo_get_nth_lcore_from_domain(domain, 0, 0,
> RTE_TOPO_DOMAIN_L1); ```
>
> The second argument (`lcore_pos`) is `0`, which is the same as for `id1`. This will
> assign **the same lcore** to both `id1` and `id2`, causing the subsequent check `if
> (id1 == id2) return 3;` to always trigger. This is a logic error: the intent is clearly to
> get two **different** lcores from the same domain.
>
> **Location:** `app/test/test_ring_perf.c:287-290` and
> `app/test/test_stack_perf.c:255-258`
>
> **Fix:** Use position `1` for the second lcore:
>
> ```c
> id1 = rte_topo_get_nth_lcore_from_domain(domain, 0, 0,
> RTE_TOPO_DOMAIN_L1);
> id2 = rte_topo_get_nth_lcore_from_domain(domain, 1, 0,
> RTE_TOPO_DOMAIN_L1); ```
>
> ---
>
> #### **Error: Iteration condition in `test_main_lcore_in_domain()` uses wrong
> domain type for lookup**
>
> In `app/test/test_topology.c:211`, the loop iterates over `domain_count` for
> `domain_types[d]`, but the call to `rte_topo_is_main_lcore_in_domain()` uses
> `RTE_TOPO_DOMAIN_NUMA` instead of `domain_types[d]`. This means the test
> only checks the NUMA domain regardless of which domain type `d` selects
> (L1/L2/L3/L4).
>
> **Location:** `app/test/test_topology.c:206-216`
>
> ```c
> for (unsigned int d = 0; d < RTE_DIM(domain_types); d++) {
> bool main_lcore_found = false;
> unsigned int domain_count = rte_topo_get_domain_count(domain_types[d]);
> for (unsigned int dmn_idx = 0; dmn_idx < domain_count; dmn_idx++) {
> main_lcore_found =
> rte_topo_is_main_lcore_in_domain(RTE_TOPO_DOMAIN_NUMA, /* BUG: should
> be domain_types[d] */
> dmn_idx);
> if (main_lcore_found)
> break;
> }
> ```
>
> **Fix:**
>
> ```c
> main_lcore_found = rte_topo_is_main_lcore_in_domain(domain_types[d], dmn_idx);
> ```
>
> ---
>
> #### **Error: Infinite loop risk in `rte_topo_get_nth_lcore_from_domain()` when `ptr-
> >core_count` is 0**
>
> In `lib/eal/common/eal_topology.c:296-318`, the function enters a `while (1)` loop
> that increments `new_lcore_pos`. If `ptr->core_count` is 0 (which the code checks
> earlier but does not return immediately), the loop will wrap `new_lcore_pos` back to
> 0 indefinitely, never breaking. While the function returns `RTE_MAX_LCORE` if `ptr-
> >core_count == 0` before the loop, the logic flow is unclear and the loop body does
> not have a clear termination condition if the core count is 0.
>
> **Location:** `lib/eal/common/eal_topology.c:283-318`
>
> **Fix:** Add a sanity check inside the loop to prevent infinite iteration:
>
> ```c
> unsigned int iterations = 0;
> while (1) {
> if (iterations++ > ptr->core_count * 2) /* safety limit */
> return RTE_MAX_LCORE;
> /* ... rest of loop ... */
> }
> ```
>
> However, the real issue is that the code already returns `RTE_MAX_LCORE` if
> `ptr->core_count == 0` on line 287, so this is more of a defensive-programming
> note. The function should be refactored for clarity.
>
> ---
>
> #### **Error: Missing NULL check after `get_domain_lcore_mapping()` in
> `rte_topo_get_next_lcore()`**
>
> In `rte_topo_get_next_lcore()`, the code calls `get_domain_lcore_mapping(flag,
> lcore_domain)` and checks if `ptr` is NULL on line 350. However, if `ptr` is NULL,
> the function returns `RTE_MAX_LCORE`. This is correct, but the subsequent logic
> on line 381 calls `rte_topo_is_main_lcore_in_domain(flag, lcore_domain)`, which
> internally may call `get_domain_lcore_mapping()` again. If that call also returns
> NULL (which it will if the domain is invalid), the function
> `rte_topo_is_main_lcore_in_domain()` will return `false`, which is safe. However, the
> logic is fragile and should explicitly handle the NULL case to avoid relying on
> transitive safety.
>
> **Location:** `lib/eal/common/eal_topology.c:381`
>
> **Recommendation:** The code is technically safe but could be clearer. No change
> required, but consider restructuring for maintainability.
>
> ---
>
> ### Patch 2/3: app/test Topology Tests
>
> #### **Error: Macro `RTE_TOPO_FOREACH_WORKER_LCORE_IN_DOMAIN`
> declares variable in macro expansion (shadowing risk)**
>
> In `lib/eal/include/rte_topology.h:243-248`, the macro
> `RTE_TOPO_FOREACH_WORKER_LCORE_IN_DOMAIN` declares a local
> variable `main_lcore` inside the macro expansion:
>
> ```c
> #define RTE_TOPO_FOREACH_WORKER_LCORE_IN_DOMAIN(lcore,
> domain_indx, flag) \
> lcore = rte_topo_get_nth_lcore_from_domain(domain, 0, 0, flag); \
> uint16_t main_lcore = rte_get_main_lcore(); \
> for (lcore = (lcore != main_lcore) ? \
> lcore : rte_topo_get_next_lcore(lcore, 1, 0, flag); \
> lcore < RTE_MAX_LCORE; \
> lcore = rte_topo_get_next_lcore(lcore, 1, 0, flag)) ```
>
> This can cause a compiler error or shadowing if the caller already has a variable
> named `main_lcore` in scope. Additionally, the macro uses `domain` (line 244) but
> the parameter is `domain_indx`, which is a typo and will cause a compilation error.
>
> **Location:** `lib/eal/include/rte_topology.h:243-248`
>
> **Fix:** Wrap in a `do { } while (0)` and use a uniquely-named variable, or document
> that the macro must not be used if `main_lcore` is already declared. Also fix the
> typo:
>
> ```c
> #define RTE_TOPO_FOREACH_WORKER_LCORE_IN_DOMAIN(lcore,
> domain_indx, flag) \
> lcore = rte_topo_get_nth_lcore_from_domain(domain_indx, 0, 0, flag); \
> uint16_t __topo_main_lcore = rte_get_main_lcore(); \
> for (lcore = (lcore != __topo_main_lcore) ? \
> lcore : rte_topo_get_next_lcore(lcore, 1, 0, flag); \
> lcore < RTE_MAX_LCORE; \
> lcore = rte_topo_get_next_lcore(lcore, 1, 0, flag)) ```
>
> ---
>
> ### Summary of Correctness Bugs
>
> 1. **hwloc topology leak** on `eal_topology_map_layer()` failure 2. **Resource
> leak** in `eal_topology_map_layer()` on partial allocation failure 3. **NULL pointer
> dereference** in `eal_topology_map_layer()` inner loop 4. **Logic error** in
> `get_same_l1_domains()` (same lcore assigned to `id1` and `id2`) 5. **Wrong
> domain type** in `test_main_lcore_in_domain()` (uses
> `RTE_TOPO_DOMAIN_NUMA` instead of `domain_types[d]`) 6. **Macro typo** in
> `RTE_TOPO_FOREACH_WORKER_LCORE_IN_DOMAIN` (uses `domain`
> instead of `domain_indx`) 7. **Potential infinite loop** in
> `rte_topo_get_nth_lcore_from_domain()` if `ptr->core_count == 0` (mitigated by
> early
^ permalink raw reply
* Re: [v1] dts: add default values for queue info
From: Patrick Robb @ 2026-05-28 1:34 UTC (permalink / raw)
To: abailey; +Cc: dev, dmarx, luca.vizzarro, probb, Patrick Robb
In-Reply-To: <20260310184416.216482-1-abailey@iol.unh.edu>
Reviewed-by: Patrick Robb <patrickrobb1997@gmail.com>
^ permalink raw reply
* [RFC v3 3/3] app/test: add fastmem test suite
From: Mattias Rönnblom @ 2026-05-27 17:30 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Bruce Richardson,
Mattias Rönnblom
In-Reply-To: <20260527173042.93867-1-hofors@lysator.liu.se>
Add functional, performance, and profiling test suites for the
fastmem library.
--
RFC v3:
* Add realloc test cases (same class, grow, shrink, NULL ptr,
zero size, too big, invalid align).
* Merge lifecycle and functional test suites into one.
* Suppress -Wuse-after-free in test_alloc_reuse (intentional
pointer comparison after free).
RFC v2:
* Add test_alloc_cross_socket_deinit exercising cross-socket
teardown path.
* Remove trailing double blank lines in test_fastmem.c.
Signed-off-by: Mattias Rönnblom <hofors@lysator.liu.se>
---
app/test/meson.build | 3 +
app/test/test_fastmem.c | 1801 +++++++++++++++++++++++++++++++
app/test/test_fastmem_perf.c | 1040 ++++++++++++++++++
app/test/test_fastmem_profile.c | 157 +++
4 files changed, 3001 insertions(+)
create mode 100644 app/test/test_fastmem.c
create mode 100644 app/test/test_fastmem_perf.c
create mode 100644 app/test/test_fastmem_profile.c
diff --git a/app/test/meson.build b/app/test/meson.build
index 7d458f9c07..d11c63be6f 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -82,6 +82,9 @@ source_file_deps = {
'test_event_vector_adapter.c': ['eventdev', 'bus_vdev'],
'test_eventdev.c': ['eventdev', 'bus_vdev'],
'test_external_mem.c': [],
+ 'test_fastmem.c': ['fastmem'],
+ 'test_fastmem_perf.c': ['fastmem', 'mempool'],
+ 'test_fastmem_profile.c': ['fastmem'],
'test_fbarray.c': [],
'test_fib.c': ['net', 'fib'],
'test_fib6.c': ['rib', 'fib'],
diff --git a/app/test/test_fastmem.c b/app/test/test_fastmem.c
new file mode 100644
index 0000000000..3ec9022b2d
--- /dev/null
+++ b/app/test/test_fastmem.c
@@ -0,0 +1,1801 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#include <errno.h>
+#include <inttypes.h>
+#include <stdalign.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_lcore.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_thread.h>
+
+#include <rte_fastmem.h>
+
+#include "test.h"
+
+#define FASTMEM_MEMZONE_SIZE (128U << 20)
+
+/*
+ * Count memzones whose names begin with the fastmem prefix.
+ * Used to verify that rte_fastmem_reserve() really did reserve
+ * backing memzones.
+ */
+static int fastmem_memzone_count;
+
+static void
+count_fastmem_memzones_walk(const struct rte_memzone *mz, void *arg)
+{
+ RTE_SET_USED(arg);
+
+ if (strncmp(mz->name, "fastmem_", strlen("fastmem_")) == 0)
+ fastmem_memzone_count++;
+}
+
+static unsigned int
+count_fastmem_memzones(void)
+{
+ fastmem_memzone_count = 0;
+ rte_memzone_walk(count_fastmem_memzones_walk, NULL);
+ return fastmem_memzone_count;
+}
+
+static int
+test_init_deinit(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_init() failed: %d", rc);
+
+ rte_fastmem_deinit();
+
+ /* A subsequent init/deinit cycle must succeed. */
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, 0, "second rte_fastmem_init() failed: %d", rc);
+
+ rte_fastmem_deinit();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_init_is_not_idempotent(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_init() failed: %d", rc);
+
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, -EBUSY,
+ "expected -EBUSY on re-init, got %d", rc);
+
+ rte_fastmem_deinit();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_deinit_without_init(void)
+{
+ /* Must be a no-op, not a crash. */
+ rte_fastmem_deinit();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_max_size(void)
+{
+ size_t max;
+
+ max = rte_fastmem_max_size();
+ TEST_ASSERT(max >= (1U << 20),
+ "max_size=%zu below required 1 MiB minimum", max);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_small(void)
+{
+ int socket_id;
+ unsigned int before, after;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ before = count_fastmem_memzones();
+
+ /*
+ * A small reserve request (1 byte) must result in exactly
+ * one memzone reservation: the internal rounding is to
+ * memzone granularity.
+ */
+ rc = rte_fastmem_reserve(1, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_reserve() failed: %d", rc);
+
+ after = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after - before, 1,
+ "expected 1 new memzone, got %u", after - before);
+
+ rte_fastmem_deinit();
+
+ /* After deinit the memzones must be released. */
+ TEST_ASSERT_EQUAL(count_fastmem_memzones(), 0,
+ "%u fastmem memzones leaked after deinit",
+ count_fastmem_memzones());
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_multiple_memzones(void)
+{
+ int socket_id;
+ unsigned int before, after;
+ size_t reserve_size;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ before = count_fastmem_memzones();
+
+ /*
+ * Request just over one memzone's worth; this must force
+ * a second memzone to be reserved.
+ */
+ reserve_size = FASTMEM_MEMZONE_SIZE + 1;
+ rc = rte_fastmem_reserve(reserve_size, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_reserve(%zu) failed: %d",
+ reserve_size, rc);
+
+ after = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after - before, 2,
+ "expected 2 new memzones for %zu-byte reserve, got %u",
+ reserve_size, after - before);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_cumulative(void)
+{
+ int socket_id;
+ unsigned int after_first, after_second;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ rc = rte_fastmem_reserve(FASTMEM_MEMZONE_SIZE, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "first reserve failed: %d", rc);
+
+ after_first = count_fastmem_memzones();
+
+ /*
+ * A second call requesting the same amount that's already
+ * reserved must not trigger any new memzone reservation.
+ */
+ rc = rte_fastmem_reserve(FASTMEM_MEMZONE_SIZE, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "second reserve failed: %d", rc);
+
+ after_second = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after_first, after_second,
+ "reserve of already-reserved amount added memzones (%u -> %u)",
+ after_first, after_second);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_invalid_socket(void)
+{
+ int rc;
+
+ rc = rte_fastmem_reserve(1, RTE_MAX_NUMA_NODES);
+ TEST_ASSERT_EQUAL(rc, -EINVAL,
+ "expected -EINVAL for out-of-range socket, got %d", rc);
+
+ rc = rte_fastmem_reserve(1, -2);
+ TEST_ASSERT_EQUAL(rc, -EINVAL,
+ "expected -EINVAL for negative socket, got %d", rc);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_without_init(void)
+{
+ int rc;
+
+ rc = rte_fastmem_reserve(1, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0,
+ "expected failure without init, got %d", rc);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_any_socket(void)
+{
+ unsigned int before, after;
+ int rc;
+
+ before = count_fastmem_memzones();
+
+ /*
+ * SOCKET_ID_ANY should succeed on any system with at least
+ * one configured socket. The allocator picks the caller's
+ * socket first and falls back to other sockets if needed.
+ */
+ rc = rte_fastmem_reserve(1, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0,
+ "rte_fastmem_reserve(SOCKET_ID_ANY) failed: %d", rc);
+
+ after = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after - before, 1,
+ "expected 1 new memzone, got %u", after - before);
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Stage 2 tests: allocation and free.
+ */
+
+static int
+test_alloc_too_big(void)
+{
+ void *p;
+ rte_errno = 0;
+ p = rte_fastmem_alloc(rte_fastmem_max_size() + 1, 0, 0);
+ TEST_ASSERT_NULL(p, "alloc above max_size returned non-NULL");
+ TEST_ASSERT_EQUAL(rte_errno, E2BIG,
+ "expected rte_errno=E2BIG, got %d", rte_errno);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_invalid_align(void)
+{
+ void *p;
+ rte_errno = 0;
+ p = rte_fastmem_alloc(16, 3, 0); /* 3 is not a power of 2 */
+ TEST_ASSERT_NULL(p, "alloc with align=3 returned non-NULL");
+ TEST_ASSERT_EQUAL(rte_errno, EINVAL,
+ "expected rte_errno=EINVAL, got %d", rte_errno);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_free_small(void)
+{
+ void *p;
+ p = rte_fastmem_alloc(8, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc(8) failed: rte_errno=%d", rte_errno);
+
+ /* Writing into the object must not crash. */
+ memset(p, 0xa5, 8);
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_free_various_sizes(void)
+{
+ static const size_t sizes[] = {
+ 1, 8, 16, 17, 63, 64, 128, 1024, 4096,
+ 64 * 1024, 256 * 1024, 1024 * 1024,
+ };
+ void *ptrs[RTE_DIM(sizes)];
+ unsigned int i;
+ for (i = 0; i < RTE_DIM(sizes); i++) {
+ ptrs[i] = rte_fastmem_alloc(sizes[i], 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "alloc(%zu) failed: rte_errno=%d",
+ sizes[i], rte_errno);
+ memset(ptrs[i], 0x5a, sizes[i]);
+ }
+
+ for (i = 0; i < RTE_DIM(sizes); i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_alignment(void)
+{
+ static const size_t aligns[] = {
+ 8, 16, 64, 256, 4096, 65536,
+ };
+ unsigned int i;
+ for (i = 0; i < RTE_DIM(aligns); i++) {
+ void *p = rte_fastmem_alloc(1, aligns[i], 0);
+
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc(1, align=%zu) failed: rte_errno=%d",
+ aligns[i], rte_errno);
+ TEST_ASSERT((uintptr_t)p % aligns[i] == 0,
+ "pointer %p not aligned on %zu",
+ p, aligns[i]);
+ rte_fastmem_free(p);
+ }
+
+ /* Default (align=0) gives at least RTE_CACHE_LINE_SIZE. */
+ {
+ void *p = rte_fastmem_alloc(1, 0, 0);
+
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc(1, align=0) failed: rte_errno=%d", rte_errno);
+ TEST_ASSERT((uintptr_t)p % RTE_CACHE_LINE_SIZE == 0,
+ "default-align pointer %p not cache-line aligned",
+ p);
+ rte_fastmem_free(p);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_zero_flag(void)
+{
+ uint8_t *p;
+ unsigned int i;
+ bool all_zero = true;
+
+ /*
+ * Dirty a slab first by allocating without F_ZERO, writing
+ * a non-zero pattern, and freeing. A subsequent F_ZERO
+ * allocation on the same slab must return zeroed memory.
+ */
+ p = rte_fastmem_alloc(128, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "priming alloc failed");
+ memset(p, 0xff, 128);
+ rte_fastmem_free(p);
+
+ p = rte_fastmem_alloc(128, 0, RTE_FASTMEM_F_ZERO);
+ TEST_ASSERT_NOT_NULL(p, "F_ZERO alloc failed");
+ for (i = 0; i < 128; i++) {
+ if (p[i] != 0) {
+ all_zero = false;
+ break;
+ }
+ }
+ TEST_ASSERT(all_zero, "F_ZERO returned non-zero byte at offset %u", i);
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+#if defined(__GNUC__) && !defined(__clang__)
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wuse-after-free"
+#endif
+static int
+test_alloc_reuse(void)
+{
+ void *first, *second;
+
+ first = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(first, "first alloc failed");
+ rte_fastmem_free(first);
+
+ second = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(second, "second alloc failed");
+
+ /*
+ * The slab's free list is LIFO, so the most recently freed
+ * object is at the head of the list. A subsequent alloc in
+ * the same class returns it.
+ */
+ TEST_ASSERT_EQUAL(first, second,
+ "free + alloc did not reuse: first=%p second=%p",
+ first, second);
+
+ rte_fastmem_free(second);
+
+ return TEST_SUCCESS;
+}
+#if defined(__GNUC__) && !defined(__clang__)
+#pragma GCC diagnostic pop
+#endif
+
+static int
+test_alloc_many_in_class(void)
+{
+ /*
+ * Allocate more objects in one class than fit in a single
+ * slab, forcing the bin to pull a second block. This
+ * exercises the partial->full transition and the cross-slab
+ * allocation path.
+ */
+ enum { CLASS_SIZE = 8, COUNT = 300000 };
+ void **ptrs;
+ unsigned int i;
+
+ ptrs = calloc(COUNT, sizeof(*ptrs));
+ TEST_ASSERT_NOT_NULL(ptrs, "calloc for test ptrs failed");
+
+ for (i = 0; i < COUNT; i++) {
+ ptrs[i] = rte_fastmem_alloc(CLASS_SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "alloc[%u] failed: rte_errno=%d",
+ i, rte_errno);
+ }
+
+ for (i = 0; i < COUNT; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ free(ptrs);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_socket(void)
+{
+ void *p;
+ int socket_id;
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ p = rte_fastmem_alloc_socket(64, 0, 0, socket_id);
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc_socket(%d) failed: rte_errno=%d",
+ socket_id, rte_errno);
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_block_repurposing(void)
+{
+ void *small, *large;
+
+ /*
+ * Allocate and free a small object, forcing a block to be
+ * assigned to the small class and then returned to the
+ * free-block pool. A subsequent allocation in a different
+ * class must be able to reuse that block.
+ */
+ small = rte_fastmem_alloc(8, 0, 0);
+ TEST_ASSERT_NOT_NULL(small, "small alloc failed");
+ rte_fastmem_free(small);
+
+ large = rte_fastmem_alloc(256 * 1024, 0, 0);
+ TEST_ASSERT_NOT_NULL(large, "large alloc failed");
+ rte_fastmem_free(large);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_block_repurposing_no_growth(void)
+{
+ struct rte_fastmem_stats stats;
+ void *small, *large;
+ uint64_t after_small;
+ int rc;
+
+ /*
+ * Stronger version of test_alloc_block_repurposing: assert
+ * that the cross-class allocation does not grow the
+ * backing memory (bytes_backing stays flat). Because the
+ * free-block pool is shared across size classes — not
+ * partitioned per class — the block freed from the small
+ * class must serve the large allocation without triggering
+ * a new memzone reservation.
+ */
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_stats() failed: %d", rc);
+ TEST_ASSERT_EQUAL(stats.bytes_backing, (uint64_t)0,
+ "unexpected pre-alloc bytes_backing: %" PRIu64,
+ stats.bytes_backing);
+
+ small = rte_fastmem_alloc(8, 0, 0);
+ TEST_ASSERT_NOT_NULL(small, "small alloc failed");
+
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_stats() failed: %d", rc);
+ TEST_ASSERT(stats.bytes_backing > 0,
+ "bytes_backing did not grow on first alloc");
+ after_small = stats.bytes_backing;
+
+ rte_fastmem_free(small);
+ rte_fastmem_cache_flush();
+
+ large = rte_fastmem_alloc(256 * 1024, 0, 0);
+ TEST_ASSERT_NOT_NULL(large,
+ "large alloc failed: rte_errno=%d", rte_errno);
+
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_stats() failed: %d", rc);
+ TEST_ASSERT_EQUAL(stats.bytes_backing, after_small,
+ "cross-class alloc grew backing memory from %" PRIu64
+ " to %" PRIu64,
+ after_small, stats.bytes_backing);
+
+ rte_fastmem_free(large);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_free_null(void)
+{
+ /* Must be a no-op, not a crash. */
+ rte_fastmem_free(NULL);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_content_integrity(void)
+{
+ /*
+ * Allocate a batch of objects, fill each with a distinct
+ * byte pattern, then verify none of the patterns overlap.
+ * This catches header overwrites (slab header corrupted by
+ * object access) and slot-overlap bugs (two pointers pointing
+ * at overlapping slots).
+ */
+ enum { N = 256, SIZE = 128 };
+ uint8_t *ptrs[N];
+ unsigned int i, j;
+ for (i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ memset(ptrs[i], (int)i, SIZE);
+ }
+
+ for (i = 0; i < N; i++)
+ for (j = 0; j < SIZE; j++)
+ TEST_ASSERT_EQUAL(ptrs[i][j], (uint8_t)i,
+ "corruption at ptrs[%u][%u]: got 0x%x, want 0x%x",
+ i, j, ptrs[i][j], (uint8_t)i);
+
+ for (i = 0; i < N; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_align_too_big(void)
+{
+ void *p;
+ /*
+ * A small size with an alignment larger than the maximum
+ * size class cannot be served. The class selected must be
+ * large enough for the alignment, but no such class exists.
+ */
+ rte_errno = 0;
+ p = rte_fastmem_alloc(1, rte_fastmem_max_size() * 2, 0);
+ TEST_ASSERT_NULL(p,
+ "alloc with align>max_size returned non-NULL");
+ TEST_ASSERT_EQUAL(rte_errno, E2BIG,
+ "expected rte_errno=E2BIG, got %d", rte_errno);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_align_one(void)
+{
+ void *p;
+ /* align=1 is a valid power of 2 and must be accepted. */
+ p = rte_fastmem_alloc(8, 1, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc(8, 1) failed: rte_errno=%d",
+ rte_errno);
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_socket_numa_placement(void)
+{
+ void *p;
+ int socket_id;
+ struct rte_memseg *ms;
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ p = rte_fastmem_alloc_socket(64, 0, 0, socket_id);
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc_socket(%d) failed: rte_errno=%d",
+ socket_id, rte_errno);
+
+ /*
+ * Walk the memory to find the memseg for this pointer and
+ * verify its socket. Skip the check if lookup fails (e.g.,
+ * --no-huge mode may not populate memsegs for fastmem's
+ * allocations in a way that rte_mem_virt2memseg can find).
+ */
+ ms = rte_mem_virt2memseg(p, NULL);
+ if (ms != NULL) {
+ TEST_ASSERT_EQUAL(ms->socket_id, socket_id,
+ "alloc on socket %d landed on socket %d",
+ socket_id, ms->socket_id);
+ }
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Allocate from a socket different from the calling lcore's socket,
+ * triggering a cross-socket cache allocation. Then deinit to exercise
+ * the teardown path where a cache's backing memory lives on a
+ * different socket than the one it serves.
+ */
+static int
+test_alloc_cross_socket_deinit(void)
+{
+ int local_sid, remote_sid;
+ unsigned int i, n_sockets;
+ void *p;
+
+ local_sid = (int)rte_socket_id();
+ if (local_sid < 0 || (unsigned int)local_sid >= RTE_MAX_NUMA_NODES)
+ local_sid = rte_socket_id_by_idx(0);
+
+ n_sockets = rte_socket_count();
+ if (n_sockets < 2)
+ return TEST_SKIPPED;
+
+ /* Find a socket different from the local one. */
+ remote_sid = -1;
+ for (i = 0; i < n_sockets; i++) {
+ int sid = rte_socket_id_by_idx(i);
+ if (sid >= 0 && sid != local_sid) {
+ remote_sid = sid;
+ break;
+ }
+ }
+ if (remote_sid < 0)
+ return TEST_SKIPPED;
+
+ p = rte_fastmem_alloc_socket(64, 0, 0, remote_sid);
+ TEST_ASSERT_NOT_NULL(p,
+ "cross-socket alloc(socket %d) failed: rte_errno=%d",
+ remote_sid, rte_errno);
+
+ rte_fastmem_free(p);
+
+ /* Teardown and re-init to exercise the deinit path with
+ * cross-socket caches.
+ */
+ rte_fastmem_deinit();
+
+ TEST_ASSERT_EQUAL(rte_fastmem_init(), 0,
+ "re-init after cross-socket deinit failed");
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Stage 3 tests: per-lcore caches.
+ */
+
+static int
+test_cache_flush(void)
+{
+ void *p;
+ /*
+ * Alloc and free one object, leaving it in the cache. Then
+ * flush and verify that a subsequent alloc may or may not
+ * return the same pointer (not asserting same/different —
+ * just checking that flush does not crash and a follow-up
+ * alloc still works).
+ */
+ p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "first alloc failed");
+ rte_fastmem_free(p);
+
+ rte_fastmem_cache_flush();
+
+ /* Flush again — must be idempotent. */
+ rte_fastmem_cache_flush();
+
+ p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "post-flush alloc failed");
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_cache_flush_without_init(void)
+{
+ /* Must be a no-op, not a crash. */
+ rte_fastmem_cache_flush();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_cache_exceeds_capacity(void)
+{
+ /*
+ * Free more objects at a single size class than the cache
+ * capacity (64 for classes <= 4 KiB). This forces the
+ * cache-drain slow path and verifies no corruption.
+ */
+ enum { COUNT = 200, SIZE = 64 };
+ void *ptrs[COUNT];
+ unsigned int i;
+
+ for (i = 0; i < COUNT; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "alloc[%u] failed: rte_errno=%d", i, rte_errno);
+ }
+
+ for (i = 0; i < COUNT; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ /* Re-alloc the same count should still work. */
+ for (i = 0; i < COUNT; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "re-alloc[%u] failed: rte_errno=%d", i, rte_errno);
+ }
+
+ for (i = 0; i < COUNT; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+struct non_eal_args {
+ int ok;
+ char pad[64];
+};
+
+static uint32_t
+non_eal_thread_main(void *arg)
+{
+ struct non_eal_args *args = arg;
+ uint8_t *p;
+
+ p = rte_fastmem_alloc(128, 0, 0);
+ if (p == NULL)
+ return 1;
+
+ memset(p, 0x7e, 128);
+
+ rte_fastmem_free(p);
+
+ args->ok = 1;
+ return 0;
+}
+
+static int
+test_non_eal_thread(void)
+{
+ rte_thread_t thread_id;
+ struct non_eal_args args = { 0 };
+ int rc;
+
+ rc = rte_thread_create(&thread_id, NULL, non_eal_thread_main, &args);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_thread_create() failed: %d", rc);
+
+ rc = rte_thread_join(thread_id, NULL);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_thread_join() failed: %d", rc);
+
+ TEST_ASSERT_EQUAL(args.ok, 1,
+ "non-EAL thread did not complete alloc/free successfully");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_cache_flush_returns_memory(void)
+{
+ /*
+ * When an entire slab's worth of objects is freed, the
+ * slab's block is returned to the free-block pool and can
+ * be reassigned to another size class. Verify the cache
+ * does not permanently hold objects that prevent this.
+ *
+ * Allocate enough objects in one class to force multiple
+ * slabs, free them all, then flush the cache. After the
+ * flush, all cached objects are drained to their bins and
+ * empty slabs are returned to the block pool.
+ */
+ enum { N = 200, SIZE = 64 };
+ void *ptrs[N];
+ unsigned int i;
+
+ for (i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ }
+ for (i = 0; i < N; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ rte_fastmem_cache_flush();
+
+ /*
+ * An allocation in a completely different class should
+ * succeed now, having access to any blocks freed by the
+ * flush.
+ */
+ {
+ void *other = rte_fastmem_alloc(65536, 0, 0);
+
+ TEST_ASSERT_NOT_NULL(other,
+ "post-flush cross-class alloc failed");
+ rte_fastmem_free(other);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_basic(void)
+{
+ enum { N = 32 };
+ void *ptrs[N];
+ int rc;
+
+ rc = rte_fastmem_alloc_bulk(ptrs, N, 64, 0, 0);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk failed: %d", rc);
+
+ /* Verify all pointers are non-NULL and distinct. */
+ for (unsigned int i = 0; i < N; i++) {
+ TEST_ASSERT_NOT_NULL(ptrs[i], "ptrs[%u] is NULL", i);
+ for (unsigned int j = 0; j < i; j++)
+ TEST_ASSERT(ptrs[i] != ptrs[j],
+ "ptrs[%u] == ptrs[%u]", i, j);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_zero_flag(void)
+{
+ enum { N = 8, SIZE = 128 };
+ void *ptrs[N];
+ int rc;
+
+ rc = rte_fastmem_alloc_bulk(ptrs, N, SIZE, 0, RTE_FASTMEM_F_ZERO);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk failed: %d", rc);
+
+ for (unsigned int i = 0; i < N; i++) {
+ uint8_t *p = ptrs[i];
+
+ for (unsigned int b = 0; b < SIZE; b++)
+ TEST_ASSERT_EQUAL(p[b], 0,
+ "ptrs[%u][%u] != 0", i, b);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_exceeds_cache(void)
+{
+ /* Allocate more than cache capacity (64) in one bulk call. */
+ enum { N = 128 };
+ void *ptrs[N];
+ int rc;
+
+ rc = rte_fastmem_alloc_bulk(ptrs, N, 64, 0, 0);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk(%u) failed: %d", N, rc);
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_socket(void)
+{
+ enum { N = 16 };
+ void *ptrs[N];
+ int socket_id;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no sockets");
+
+ rc = rte_fastmem_alloc_bulk_socket(ptrs, N, 64, 0, 0, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk_socket failed: %d", rc);
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ /* SOCKET_ID_ANY */
+ rc = rte_fastmem_alloc_bulk_socket(ptrs, N, 64, 0, 0, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk_socket(ANY) failed: %d", rc);
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_free_bulk(void)
+{
+ enum { N = 64 };
+ void *ptrs[N];
+ /* Allocate individually, free in bulk. */
+ for (unsigned int i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ /* Verify memory is reusable. */
+ for (unsigned int i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "re-alloc[%u] failed", i);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_classes(void)
+{
+ size_t sizes[32];
+ unsigned int n;
+
+ n = rte_fastmem_classes(NULL);
+ TEST_ASSERT_EQUAL(n, 18u, "expected 18 classes, got %u", n);
+
+ n = rte_fastmem_classes(sizes);
+ TEST_ASSERT_EQUAL(n, 18u, "expected 18 classes, got %u", n);
+ TEST_ASSERT_EQUAL(sizes[0], (size_t)8, "class 0 != 8");
+ TEST_ASSERT_EQUAL(sizes[n - 1], (size_t)(1 << 20),
+ "last class != 1 MiB");
+
+ for (unsigned int i = 0; i < n; i++) {
+ TEST_ASSERT(sizes[i] != 0 && (sizes[i] & (sizes[i] - 1)) == 0,
+ "class %u size %zu not power of 2", i, sizes[i]);
+ if (i > 0)
+ TEST_ASSERT(sizes[i] > sizes[i - 1],
+ "classes not ascending at %u", i);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_class(void)
+{
+ enum { N = 10 };
+ struct rte_fastmem_class_stats cs;
+ void *ptrs[N];
+ int rc;
+
+ for (unsigned int i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ }
+
+ rc = rte_fastmem_stats_class(64, &cs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_class failed: %d", rc);
+ TEST_ASSERT_EQUAL(cs.class_size, (size_t)64, "wrong class_size");
+ TEST_ASSERT(cs.alloc_cache_hits + cs.alloc_cache_misses == N,
+ "alloc count != N: hits=%" PRIu64 " misses=%" PRIu64,
+ cs.alloc_cache_hits, cs.alloc_cache_misses);
+ TEST_ASSERT_EQUAL(cs.in_use, (uint64_t)N, "in_use != N");
+
+ for (unsigned int i = 0; i < N; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ rc = rte_fastmem_stats_class(64, &cs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_class after free failed: %d", rc);
+ TEST_ASSERT_EQUAL(cs.in_use, (uint64_t)0, "in_use != 0 after free");
+
+ /* Invalid class size. */
+ rc = rte_fastmem_stats_class(13, &cs);
+ TEST_ASSERT_EQUAL(rc, -EINVAL, "expected -EINVAL for bad size");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_lcore(void)
+{
+ struct rte_fastmem_lcore_stats ls;
+ void *ptr;
+ int rc;
+
+ ptr = rte_fastmem_alloc(128, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ rc = rte_fastmem_stats_lcore(rte_lcore_id(), &ls);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_lcore failed: %d", rc);
+ TEST_ASSERT(ls.alloc_cache_hits + ls.alloc_cache_misses > 0,
+ "no alloc activity on this lcore");
+
+ rte_fastmem_free(ptr);
+
+ rc = rte_fastmem_stats_lcore(rte_lcore_id(), &ls);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_lcore after free failed: %d", rc);
+ TEST_ASSERT(ls.free_cache_hits + ls.free_cache_misses > 0,
+ "no free activity on this lcore");
+
+ /* Invalid lcore. */
+ rc = rte_fastmem_stats_lcore(RTE_MAX_LCORE, &ls);
+ TEST_ASSERT_EQUAL(rc, -EINVAL, "expected -EINVAL for bad lcore");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_lcore_class(void)
+{
+ struct rte_fastmem_lcore_class_stats lcs;
+ void *ptr;
+ int rc;
+
+ ptr = rte_fastmem_alloc(256, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ rc = rte_fastmem_stats_lcore_class(rte_lcore_id(), 256, &lcs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_lcore_class failed: %d", rc);
+ TEST_ASSERT_EQUAL(lcs.class_size, (size_t)256, "wrong class_size");
+ TEST_ASSERT(lcs.alloc_cache_hits + lcs.alloc_cache_misses > 0,
+ "no alloc activity");
+
+ rte_fastmem_free(ptr);
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_reset(void)
+{
+ struct rte_fastmem_stats gs;
+ void *ptr;
+ int rc;
+
+ ptr = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+ rte_fastmem_free(ptr);
+
+ rte_fastmem_stats_reset();
+
+ rc = rte_fastmem_stats(&gs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats failed: %d", rc);
+ TEST_ASSERT_EQUAL(gs.alloc_total, (uint64_t)0,
+ "alloc_total not zero after reset");
+ TEST_ASSERT_EQUAL(gs.free_total, (uint64_t)0,
+ "free_total not zero after reset");
+
+ return TEST_SUCCESS;
+}
+
+
+#define MIXED_LONG_LIVED_COUNT 25
+#define MIXED_SHORT_LIVED_ITERS 1000
+#define MIXED_MIN_LCORES 3
+
+static const size_t mixed_long_sizes[] = { 64, 256, 4096 };
+static const size_t mixed_short_sizes[] = { 8, 16, 32, 64, 128, 256, 512, 1024 };
+
+struct mixed_worker_args {
+ uint32_t seed;
+ int result;
+};
+
+static uint32_t
+xorshift32(uint32_t *state)
+{
+ uint32_t x = *state;
+
+ x ^= x << 13;
+ x ^= x >> 17;
+ x ^= x << 5;
+ *state = x;
+ return x;
+}
+
+static int
+mixed_worker(void *arg)
+{
+ struct mixed_worker_args *args = arg;
+ uint32_t seed = args->seed;
+ void *long_lived[MIXED_LONG_LIVED_COUNT];
+ size_t long_sizes[MIXED_LONG_LIVED_COUNT];
+ unsigned int i;
+
+ /* Allocate long-lived objects of mixed sizes. */
+ for (i = 0; i < MIXED_LONG_LIVED_COUNT; i++) {
+ long_sizes[i] = mixed_long_sizes[i % RTE_DIM(mixed_long_sizes)];
+ long_lived[i] = rte_fastmem_alloc(long_sizes[i], 0, 0);
+ if (long_lived[i] == NULL) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ memset(long_lived[i], (int)(i + 1), long_sizes[i]);
+ }
+
+ /* Rapidly cycle short-lived objects. */
+ for (i = 0; i < MIXED_SHORT_LIVED_ITERS; i++) {
+ size_t sz = mixed_short_sizes[xorshift32(&seed) %
+ RTE_DIM(mixed_short_sizes)];
+ uint8_t pattern = (uint8_t)(i & 0xff);
+ uint8_t *p;
+
+ p = rte_fastmem_alloc(sz, 0, 0);
+ if (p == NULL) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ memset(p, pattern, sz);
+
+ /* Verify before freeing. */
+ for (size_t j = 0; j < sz; j++) {
+ if (p[j] != pattern) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ }
+ rte_fastmem_free(p);
+ }
+
+ /* Verify long-lived objects are still intact. */
+ for (i = 0; i < MIXED_LONG_LIVED_COUNT; i++) {
+ uint8_t *bytes = long_lived[i];
+ uint8_t expected = (uint8_t)(i + 1);
+
+ for (size_t j = 0; j < long_sizes[i]; j++) {
+ if (bytes[j] != expected) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ }
+ rte_fastmem_free(long_lived[i]);
+ }
+
+ args->result = TEST_SUCCESS;
+ return 0;
+}
+
+static int
+test_mixed_lifetimes_multi_lcore(void)
+{
+ struct mixed_worker_args args[RTE_MAX_LCORE];
+ unsigned int lcore_id;
+ unsigned int count = 0;
+ struct rte_fastmem_stats stats;
+ int rc;
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id)
+ count++;
+
+ if (count < MIXED_MIN_LCORES) {
+ printf("Not enough worker lcores (%u < %u), skipping\n",
+ count, MIXED_MIN_LCORES);
+ return TEST_SKIPPED;
+ }
+
+ /* Launch workers with distinct seeds. */
+ uint32_t seed = 0xdeadbeef;
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ args[lcore_id].seed = seed;
+ args[lcore_id].result = TEST_FAILED;
+ seed += 0x12345678;
+ rte_eal_remote_launch(mixed_worker, &args[lcore_id], lcore_id);
+ }
+
+ rte_eal_mp_wait_lcore();
+
+ /* Check all workers succeeded. */
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ TEST_ASSERT_EQUAL(args[lcore_id].result, TEST_SUCCESS,
+ "worker on lcore %u failed", lcore_id);
+ }
+
+ /* Verify no memory leak. */
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "stats failed: %d", rc);
+ TEST_ASSERT_EQUAL(stats.bytes_in_use, (uint64_t)0,
+ "bytes_in_use not zero after test: %" PRIu64,
+ stats.bytes_in_use);
+
+ return TEST_SUCCESS;
+}
+
+
+/*
+ * Memory limit tests.
+ *
+ * FASTMEM_MEMZONE_SIZE is 128 MiB. We use a limit of 128 MiB
+ * (one memzone) for most tests, and large objects (256 KiB) to
+ * exhaust slabs quickly.
+ */
+
+#define LIMIT_ONE_MZ ((size_t)128 << 20)
+#define LIMIT_OBJ_SIZE ((size_t)256 * 1024)
+
+static int
+test_memory_limit_basic(void)
+{
+ int rc;
+
+ rc = rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+ TEST_ASSERT_EQUAL(rc, 0, "set_memory_limit failed: %d", rc);
+
+ const size_t got = rte_fastmem_get_limit(0);
+ TEST_ASSERT_EQUAL(got, LIMIT_ONE_MZ,
+ "get_memory_limit mismatch: %zu", got);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "first reserve failed: %d", rc);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ + 1, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0, "second reserve should have failed");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_alloc_exhaustion(void)
+{
+ const unsigned int max_ptrs = 1024;
+ void *ptrs[max_ptrs];
+ unsigned int count = 0;
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (count = 0; count < max_ptrs; count++) {
+ ptrs[count] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (ptrs[count] == NULL)
+ break;
+ }
+
+ TEST_ASSERT(count > 0, "should have allocated at least one");
+ TEST_ASSERT(count < max_ptrs, "should have hit the limit");
+ TEST_ASSERT_EQUAL(rte_errno, ENOMEM, "expected ENOMEM, got %d", rte_errno);
+
+ rte_fastmem_free(ptrs[count - 1]);
+ void *p = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc after free should succeed");
+ rte_fastmem_free(p);
+
+ for (unsigned int i = 0; i < count - 1; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_zero_blocks_growth(void)
+{
+ int rc;
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 0);
+
+ rc = rte_fastmem_reserve(1, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0, "reserve with limit=0 should fail");
+
+ void *p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NULL(p, "alloc with limit=0 should fail");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_below_current(void)
+{
+ int rc;
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "reserve failed: %d", rc);
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 1);
+
+ void *p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc from existing backing should work");
+ rte_fastmem_free(p);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ * 2, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0, "growth beyond limit should fail");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_socket_id_any(void)
+{
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 42);
+
+ for (unsigned int i = 0; i < rte_socket_count(); i++) {
+ const int sid = rte_socket_id_by_idx(i);
+ const size_t lim = rte_fastmem_get_limit(sid);
+
+ TEST_ASSERT_EQUAL(lim, (size_t)42,
+ "socket %d limit mismatch: %zu", sid, lim);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_unlimited(void)
+{
+ int rc;
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 0);
+ rte_fastmem_set_limit(SOCKET_ID_ANY, SIZE_MAX);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "reserve after reset failed: %d", rc);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_alloc_integrity_under_oom(void)
+{
+ const unsigned int n = 128;
+ const size_t obj_size = 1024;
+ uint8_t *ptrs[n];
+ const unsigned int extra_max = 1024;
+ void *extra[extra_max];
+ unsigned int n_extra = 0;
+ unsigned int i;
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (i = 0; i < n; i++) {
+ ptrs[i] = rte_fastmem_alloc(obj_size, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ memset(ptrs[i], (int)(i & 0xff), obj_size);
+ }
+
+ /* Exhaust remaining backing with large objects. */
+ for (n_extra = 0; n_extra < extra_max; n_extra++) {
+ extra[n_extra] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (extra[n_extra] == NULL)
+ break;
+ }
+
+ /* Verify original objects are intact. */
+ for (i = 0; i < n; i++) {
+ const uint8_t expected = (uint8_t)(i & 0xff);
+ for (unsigned int j = 0; j < obj_size; j++)
+ TEST_ASSERT_EQUAL(ptrs[i][j], expected,
+ "corruption at [%u][%u]", i, j);
+ }
+
+ for (i = 0; i < n; i++)
+ rte_fastmem_free(ptrs[i]);
+ for (i = 0; i < n_extra; i++)
+ rte_fastmem_free(extra[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_bulk_alloc_oom(void)
+{
+ const unsigned int bulk_n = 64;
+ const unsigned int drain_max = 512;
+ void *ptrs[bulk_n];
+ void *drain[drain_max];
+ unsigned int drained = 0;
+ int rc;
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (drained = 0; drained < drain_max; drained++) {
+ drain[drained] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (drain[drained] == NULL)
+ break;
+ }
+
+ /* Free a few — enough for some but not bulk_n objects. */
+ const unsigned int freed = RTE_MIN(drained, 4u);
+ for (unsigned int i = 0; i < freed; i++)
+ rte_fastmem_free(drain[--drained]);
+
+ rc = rte_fastmem_alloc_bulk(ptrs, bulk_n, LIMIT_OBJ_SIZE, 0, 0);
+ TEST_ASSERT(rc < 0, "bulk alloc should fail");
+
+ for (unsigned int i = 0; i < drained; i++)
+ rte_fastmem_free(drain[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_recovery_after_free(void)
+{
+ const unsigned int max_ptrs = 512;
+ void *ptrs[max_ptrs];
+ unsigned int count = 0;
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (count = 0; count < max_ptrs; count++) {
+ ptrs[count] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (ptrs[count] == NULL)
+ break;
+ }
+ TEST_ASSERT(count > 0 && count < max_ptrs,
+ "expected partial fill, got %u", count);
+
+ const unsigned int half = count / 2;
+ for (unsigned int i = 0; i < half; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ for (unsigned int i = 0; i < half; i++) {
+ ptrs[i] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "recovery alloc[%u] failed", i);
+ }
+
+ for (unsigned int i = 0; i < count; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+struct limit_worker_args {
+ unsigned int alloc_count;
+ int result;
+};
+
+static int
+limit_worker(void *arg)
+{
+ struct limit_worker_args *args = arg;
+ const unsigned int max_ptrs = 128;
+ void *ptrs[max_ptrs];
+ unsigned int i;
+
+ args->alloc_count = 0;
+
+ for (i = 0; i < max_ptrs; i++) {
+ ptrs[i] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (ptrs[i] == NULL)
+ break;
+ memset(ptrs[i], 0xab, LIMIT_OBJ_SIZE);
+ args->alloc_count++;
+ }
+
+ for (unsigned int j = 0; j < args->alloc_count; j++) {
+ uint8_t *bytes = ptrs[j];
+ for (size_t k = 0; k < LIMIT_OBJ_SIZE; k++) {
+ if (bytes[k] != 0xab) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ }
+ rte_fastmem_free(ptrs[j]);
+ }
+
+ args->result = TEST_SUCCESS;
+ return 0;
+}
+
+static int
+test_memory_limit_multi_lcore_oom(void)
+{
+ struct limit_worker_args args[RTE_MAX_LCORE];
+ unsigned int lcore_id;
+ unsigned int worker_count = 0;
+ RTE_LCORE_FOREACH_WORKER(lcore_id)
+ worker_count++;
+
+ if (worker_count < 2) {
+ printf("Not enough workers (%u < 2), skipping\n", worker_count);
+ return TEST_SKIPPED;
+ }
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ args[lcore_id].result = TEST_FAILED;
+ rte_eal_remote_launch(limit_worker, &args[lcore_id], lcore_id);
+ }
+
+ rte_eal_mp_wait_lcore();
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ TEST_ASSERT_EQUAL(args[lcore_id].result, TEST_SUCCESS,
+ "worker on lcore %u failed", lcore_id);
+ }
+
+ struct rte_fastmem_stats stats;
+ rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(stats.bytes_in_use, (uint64_t)0,
+ "bytes_in_use not zero: %" PRIu64, stats.bytes_in_use);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_realloc_same_class(void)
+{
+ void *ptr = rte_fastmem_alloc(32, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ /* Realloc to a smaller size within the same class (64 B class). */
+ void *ptr2 = rte_fastmem_realloc(ptr, 33, 0);
+ TEST_ASSERT_NOT_NULL(ptr2, "realloc failed");
+ TEST_ASSERT_EQUAL(ptr, ptr2,
+ "realloc returned different pointer for same class");
+
+ /* Realloc to exact class boundary — still same class. */
+ void *ptr3 = rte_fastmem_realloc(ptr2, 64, 0);
+ TEST_ASSERT_NOT_NULL(ptr3, "realloc failed");
+ TEST_ASSERT_EQUAL(ptr2, ptr3,
+ "realloc returned different pointer for same class");
+
+ rte_fastmem_free(ptr3);
+ return TEST_SUCCESS;
+}
+
+static int
+test_realloc_grow(void)
+{
+ const uint8_t pattern = 0xab;
+ void *ptr = rte_fastmem_alloc(16, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ memset(ptr, pattern, 16);
+
+ /* Grow beyond current class. */
+ void *ptr2 = rte_fastmem_realloc(ptr, 128, 0);
+ TEST_ASSERT_NOT_NULL(ptr2, "realloc grow failed");
+
+ /* Verify contents preserved. */
+ uint8_t *bytes = ptr2;
+ for (unsigned int i = 0; i < 16; i++)
+ TEST_ASSERT_EQUAL(bytes[i], pattern,
+ "content corrupted at byte %u", i);
+
+ rte_fastmem_free(ptr2);
+ return TEST_SUCCESS;
+}
+
+static int
+test_realloc_shrink(void)
+{
+ const uint8_t pattern = 0xcd;
+ void *ptr = rte_fastmem_alloc(256, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ memset(ptr, pattern, 256);
+
+ /* Shrink to a smaller class. */
+ void *ptr2 = rte_fastmem_realloc(ptr, 16, 0);
+ TEST_ASSERT_NOT_NULL(ptr2, "realloc shrink failed");
+
+ /* Verify contents preserved up to new size. */
+ uint8_t *bytes = ptr2;
+ for (unsigned int i = 0; i < 16; i++)
+ TEST_ASSERT_EQUAL(bytes[i], pattern,
+ "content corrupted at byte %u", i);
+
+ rte_fastmem_free(ptr2);
+ return TEST_SUCCESS;
+}
+
+static int
+test_realloc_null_ptr(void)
+{
+ /* NULL ptr should behave like alloc. */
+ void *ptr = rte_fastmem_realloc(NULL, 64, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "realloc(NULL) failed");
+
+ rte_fastmem_free(ptr);
+ return TEST_SUCCESS;
+}
+
+static int
+test_realloc_zero_size(void)
+{
+ void *ptr = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ /* size 0 should free and return NULL. */
+ void *ptr2 = rte_fastmem_realloc(ptr, 0, 0);
+ TEST_ASSERT_NULL(ptr2, "realloc(size=0) should return NULL");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_realloc_too_big(void)
+{
+ void *ptr = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ void *ptr2 = rte_fastmem_realloc(ptr, rte_fastmem_max_size() + 1, 0);
+ TEST_ASSERT_NULL(ptr2, "realloc should fail for oversized request");
+ TEST_ASSERT_EQUAL(rte_errno, E2BIG, "expected E2BIG");
+
+ /* Original pointer should still be valid. */
+ rte_fastmem_free(ptr);
+ return TEST_SUCCESS;
+}
+
+static int
+test_realloc_invalid_align(void)
+{
+ void *ptr = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ void *ptr2 = rte_fastmem_realloc(ptr, 64, 3);
+ TEST_ASSERT_NULL(ptr2, "realloc should fail for non-power-of-2 align");
+ TEST_ASSERT_EQUAL(rte_errno, EINVAL, "expected EINVAL");
+
+ rte_fastmem_free(ptr);
+ return TEST_SUCCESS;
+}
+
+static int
+fastmem_setup(void)
+{
+ return rte_fastmem_init();
+}
+
+static void
+fastmem_teardown(void)
+{
+ rte_fastmem_deinit();
+}
+
+static struct unit_test_suite fastmem_testsuite = {
+ .suite_name = "fastmem tests",
+ .setup = NULL,
+ .teardown = NULL,
+ .unit_test_cases = {
+ TEST_CASE(test_init_deinit),
+ TEST_CASE(test_init_is_not_idempotent),
+ TEST_CASE(test_deinit_without_init),
+ TEST_CASE(test_max_size),
+ TEST_CASE(test_reserve_without_init),
+ TEST_CASE(test_cache_flush_without_init),
+ TEST_CASE(test_classes),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_small),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_multiple_memzones),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_cumulative),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_invalid_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_any_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_too_big),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_invalid_align),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_free_small),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_free_various_sizes),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_alignment),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_zero_flag),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_reuse),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_many_in_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_block_repurposing),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_block_repurposing_no_growth),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_free_null),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_content_integrity),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_align_too_big),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_align_one),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_socket_numa_placement),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_cross_socket_deinit),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_cache_flush),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_cache_exceeds_capacity),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_non_eal_thread),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_cache_flush_returns_memory),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_basic),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_zero_flag),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_exceeds_cache),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_free_bulk),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_lcore),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_lcore_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_reset),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_mixed_lifetimes_multi_lcore),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_basic),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_alloc_exhaustion),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_zero_blocks_growth),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_below_current),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_socket_id_any),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_unlimited),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_alloc_integrity_under_oom),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_bulk_alloc_oom),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_recovery_after_free),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_multi_lcore_oom),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_realloc_same_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_realloc_grow),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_realloc_shrink),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_realloc_null_ptr),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_realloc_zero_size),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_realloc_too_big),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_realloc_invalid_align),
+ TEST_CASES_END()
+ }
+};
+
+static int
+test_fastmem(void)
+{
+ return unit_test_suite_runner(&fastmem_testsuite);
+}
+
+REGISTER_FAST_TEST(fastmem_autotest, NOHUGE_SKIP, ASAN_OK, test_fastmem);
diff --git a/app/test/test_fastmem_perf.c b/app/test/test_fastmem_perf.c
new file mode 100644
index 0000000000..73c0a4c6ce
--- /dev/null
+++ b/app/test/test_fastmem_perf.c
@@ -0,0 +1,1040 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_launch.h>
+#include <rte_lcore.h>
+#include <rte_malloc.h>
+#include <rte_mempool.h>
+#include <rte_stdatomic.h>
+
+#include <rte_fastmem.h>
+
+#include "test.h"
+
+#define TEST_LOG(...) printf(__VA_ARGS__)
+
+static const size_t SIZES[] = { 8, 64, 256, 1024, 4096 };
+#define N_SIZES RTE_DIM(SIZES)
+
+/* Number of ops for warmup and measurement. */
+#define WARMUP_OPS 20000u
+#define MEASURE_OPS 2000000u
+
+/* Buffer for scenarios that allocate N then free N. */
+#define BATCH_N 256
+
+/*
+ * Allocator vtable: a thin adapter exposing alloc / free /
+ * per-allocator setup/teardown. Each scenario calls these
+ * indirectly so the same timing loop serves all allocators.
+ */
+struct allocator {
+ const char *name;
+ int (*setup)(size_t size, unsigned int n_max);
+ void (*teardown)(void);
+ void *(*alloc)(void);
+ void (*free_obj)(void *ptr);
+ int (*alloc_bulk)(void **ptrs, unsigned int n);
+ void (*free_bulk)(void **ptrs, unsigned int n);
+};
+
+/* Fastmem adapter -------------------------------------------------- */
+
+static size_t fastmem_size;
+
+static int
+fastmem_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ fastmem_size = size;
+ return 0;
+}
+
+static void
+fastmem_teardown(void)
+{
+ rte_fastmem_cache_flush();
+}
+
+static void * __rte_noinline
+fastmem_alloc(void)
+{
+ return rte_fastmem_alloc(fastmem_size, 0, 0);
+}
+
+static void __rte_noinline
+fastmem_free(void *ptr)
+{
+ rte_fastmem_free(ptr);
+}
+
+/* Mempool adapter -------------------------------------------------- */
+
+static struct rte_mempool *mempool_pool;
+
+static int
+mempool_setup(size_t size, unsigned int n_max)
+{
+ char name[RTE_MEMPOOL_NAMESIZE];
+ unsigned int cache_size;
+
+ /*
+ * Pool size must accommodate the full batch burst plus
+ * per-lcore cache capacity. Use mempool's default cache
+ * size so we're measuring its standard hot path.
+ */
+ cache_size = RTE_MEMPOOL_CACHE_MAX_SIZE;
+
+ snprintf(name, sizeof(name), "fmperf_mp_%zu", size);
+ mempool_pool = rte_mempool_create(name, n_max + cache_size * 2,
+ size, cache_size, 0, NULL, NULL, NULL, NULL,
+ SOCKET_ID_ANY, 0);
+ if (mempool_pool == NULL) {
+ TEST_LOG("mempool_create(%zu) failed\n", size);
+ return -1;
+ }
+
+ return 0;
+}
+
+static void
+mempool_teardown(void)
+{
+ rte_mempool_free(mempool_pool);
+ mempool_pool = NULL;
+}
+
+static void * __rte_noinline
+mempool_alloc_one(void)
+{
+ void *obj = NULL;
+
+ if (rte_mempool_get(mempool_pool, &obj) < 0)
+ return NULL;
+ return obj;
+}
+
+static void __rte_noinline
+mempool_free_one(void *ptr)
+{
+ rte_mempool_put(mempool_pool, ptr);
+}
+
+/* rte_malloc adapter ----------------------------------------------- */
+
+static size_t malloc_size;
+
+static int
+malloc_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ malloc_size = size;
+ return 0;
+}
+
+static void
+malloc_teardown(void)
+{
+}
+
+static void * __rte_noinline
+malloc_alloc(void)
+{
+ return rte_malloc(NULL, malloc_size, 0);
+}
+
+static void __rte_noinline
+malloc_free(void *ptr)
+{
+ rte_free(ptr);
+}
+
+/* libc (glibc) malloc adapter -------------------------------------- */
+
+static size_t libc_size;
+
+static int
+libc_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ /*
+ * Round up to cache-line alignment to match the other
+ * allocators' default alignment guarantees and keep the
+ * comparison honest. aligned_alloc() requires size to be
+ * a multiple of the alignment.
+ */
+ libc_size = RTE_ALIGN_CEIL(size, RTE_CACHE_LINE_SIZE);
+ return 0;
+}
+
+static void
+libc_teardown(void)
+{
+}
+
+static void * __rte_noinline
+libc_alloc(void)
+{
+ return aligned_alloc(RTE_CACHE_LINE_SIZE, libc_size);
+}
+
+static void __rte_noinline
+libc_free(void *ptr)
+{
+ free(ptr);
+}
+
+/* Bulk adapters ---------------------------------------------------- */
+
+static int __rte_noinline
+fastmem_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return rte_fastmem_alloc_bulk(ptrs, n, fastmem_size, 0, 0);
+}
+
+static void __rte_noinline
+fastmem_free_bulk(void **ptrs, unsigned int n)
+{
+ rte_fastmem_free_bulk(ptrs, n);
+}
+
+/* Fastmem handle adapter ------------------------------------------- */
+
+static rte_fastmem_handle_t fastmem_handle;
+
+static int
+fastmem_h_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ return rte_fastmem_hlookup(size, 0, rte_socket_id(), &fastmem_handle);
+}
+
+static void
+fastmem_h_teardown(void)
+{
+ rte_fastmem_cache_flush();
+}
+
+static void * __rte_noinline
+fastmem_h_alloc(void)
+{
+ return rte_fastmem_halloc(fastmem_handle, 0);
+}
+
+static void __rte_noinline
+fastmem_h_free(void *ptr)
+{
+ rte_fastmem_hfree(fastmem_handle, ptr);
+}
+
+static int __rte_noinline
+fastmem_h_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return rte_fastmem_halloc_bulk(fastmem_handle, ptrs, n, 0);
+}
+
+static void __rte_noinline
+fastmem_h_free_bulk(void **ptrs, unsigned int n)
+{
+ rte_fastmem_hfree_bulk(fastmem_handle, ptrs, n);
+}
+
+/* Mempool adapter -------------------------------------------------- */
+
+static int __rte_noinline
+mempool_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return rte_mempool_get_bulk(mempool_pool, ptrs, n);
+}
+
+static void __rte_noinline
+mempool_free_bulk(void **ptrs, unsigned int n)
+{
+ rte_mempool_put_bulk(mempool_pool, ptrs, n);
+}
+
+static int __rte_noinline
+generic_alloc_bulk(void **ptrs, unsigned int n, void *(*alloc_fn)(void))
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++) {
+ ptrs[i] = alloc_fn();
+ if (ptrs[i] == NULL)
+ return -1;
+ }
+ return 0;
+}
+
+static int __rte_noinline
+malloc_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return generic_alloc_bulk(ptrs, n, malloc_alloc);
+}
+
+static void __rte_noinline
+malloc_free_bulk(void **ptrs, unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ malloc_free(ptrs[i]);
+}
+
+static int __rte_noinline
+libc_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return generic_alloc_bulk(ptrs, n, libc_alloc);
+}
+
+static void __rte_noinline
+libc_free_bulk(void **ptrs, unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ libc_free(ptrs[i]);
+}
+
+/* Adapter table ---------------------------------------------------- */
+
+static const struct allocator allocators[] = {
+ { "fastmem", fastmem_setup, fastmem_teardown, fastmem_alloc, fastmem_free, fastmem_alloc_bulk, fastmem_free_bulk },
+ { "fastmem_h", fastmem_h_setup, fastmem_h_teardown, fastmem_h_alloc, fastmem_h_free, fastmem_h_alloc_bulk, fastmem_h_free_bulk },
+ { "mempool", mempool_setup, mempool_teardown, mempool_alloc_one, mempool_free_one, mempool_alloc_bulk, mempool_free_bulk },
+ { "rte_malloc", malloc_setup, malloc_teardown, malloc_alloc, malloc_free, malloc_alloc_bulk, malloc_free_bulk },
+ { "libc", libc_setup, libc_teardown, libc_alloc, libc_free, libc_alloc_bulk, libc_free_bulk },
+};
+#define N_ALLOCATORS RTE_DIM(allocators)
+
+/*
+ * Scenario 1: tight alloc+free loop. A single object is cycled
+ * repeatedly. The LIFO path keeps the same pointer hot, giving
+ * a best-case measurement.
+ */
+static double
+run_tight(const struct allocator *alloc, size_t size)
+{
+ void *p;
+ uint64_t tsc;
+ unsigned int i;
+
+ if (alloc->setup(size, 1) < 0)
+ return -1.0;
+
+ /* Warmup. */
+ for (i = 0; i < WARMUP_OPS; i++) {
+ p = alloc->alloc();
+ if (p == NULL)
+ goto err;
+ alloc->free_obj(p);
+ }
+
+ tsc = rte_rdtsc_precise();
+ for (i = 0; i < MEASURE_OPS; i++) {
+ p = alloc->alloc();
+ if (p == NULL)
+ goto err;
+ alloc->free_obj(p);
+ }
+ tsc = rte_rdtsc_precise() - tsc;
+
+ alloc->teardown();
+
+ return (double)tsc / MEASURE_OPS;
+err:
+ alloc->teardown();
+ return -1.0;
+}
+
+/*
+ * Scenario 2: allocate N, free N (FIFO free order). Exercises
+ * cache refill and drain paths when N exceeds cache capacity.
+ */
+static void
+run_batch(const struct allocator *alloc, size_t size,
+ double *cycles_alloc, double *cycles_free)
+{
+ void *ptrs[BATCH_N];
+ uint64_t tsc_alloc = 0, tsc_free = 0;
+ unsigned int iter, i;
+ unsigned int iters;
+
+ *cycles_alloc = -1.0;
+ *cycles_free = -1.0;
+
+ if (alloc->setup(size, BATCH_N) < 0)
+ return;
+
+ /* Pick iteration count so total ops ~= MEASURE_OPS. */
+ iters = MEASURE_OPS / BATCH_N;
+
+ /* Warmup. */
+ for (iter = 0; iter < WARMUP_OPS / BATCH_N; iter++) {
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ for (i = 0; i < BATCH_N; i++)
+ alloc->free_obj(ptrs[i]);
+ }
+
+ for (iter = 0; iter < iters; iter++) {
+ uint64_t t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ tsc_alloc += rte_rdtsc_precise() - t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < BATCH_N; i++)
+ alloc->free_obj(ptrs[i]);
+ tsc_free += rte_rdtsc_precise() - t0;
+ }
+
+ alloc->teardown();
+
+ *cycles_alloc = (double)tsc_alloc / (iters * BATCH_N);
+ *cycles_free = (double)tsc_free / (iters * BATCH_N);
+ return;
+err:
+ alloc->teardown();
+}
+
+/*
+ * Scenario 3: allocate N, free N in reverse order.
+ */
+static void
+run_batch_reverse(const struct allocator *alloc, size_t size,
+ double *cycles_alloc, double *cycles_free)
+{
+ void *ptrs[BATCH_N];
+ uint64_t tsc_alloc = 0, tsc_free = 0;
+ unsigned int iter, i;
+ unsigned int iters;
+
+ *cycles_alloc = -1.0;
+ *cycles_free = -1.0;
+
+ if (alloc->setup(size, BATCH_N) < 0)
+ return;
+
+ iters = MEASURE_OPS / BATCH_N;
+
+ for (iter = 0; iter < WARMUP_OPS / BATCH_N; iter++) {
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ for (i = BATCH_N; i > 0; i--)
+ alloc->free_obj(ptrs[i - 1]);
+ }
+
+ for (iter = 0; iter < iters; iter++) {
+ uint64_t t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ tsc_alloc += rte_rdtsc_precise() - t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = BATCH_N; i > 0; i--)
+ alloc->free_obj(ptrs[i - 1]);
+ tsc_free += rte_rdtsc_precise() - t0;
+ }
+
+ alloc->teardown();
+
+ *cycles_alloc = (double)tsc_alloc / (iters * BATCH_N);
+ *cycles_free = (double)tsc_free / (iters * BATCH_N);
+ return;
+err:
+ alloc->teardown();
+}
+
+/*
+ * Scenario 4: multi-lcore alloc/work/free with a dummy-work
+ * baseline. Each worker runs a tight alloc → touch → free loop
+ * on its own lcore. A second run with the same dummy work but
+ * no allocator traffic establishes a baseline; the per-op
+ * allocator cost is reported as (alloc_run - baseline_run).
+ *
+ * Fixed size class and a fixed amount of dummy work per op —
+ * this scenario sweeps lcore count rather than size.
+ */
+#define MULTI_SIZE 256u
+#define MULTI_WORK_BYTES 64u
+#define MULTI_WORK_PASSES 8u /* RMW passes over the work region. */
+#define MULTI_OPS 200000u
+#define MULTI_WARMUP 2000u
+#define MAX_MULTI_LCORES 32u
+
+/*
+ * Per-worker volatile sink. Each worker writes to its own
+ * slot, preventing dead-code elimination of touch_buffer() and
+ * avoiding cross-lcore cache-line sharing on the hot path.
+ * Padded to cache-line stride to prevent false sharing between
+ * neighboring workers' slots.
+ */
+struct worker_sink {
+ volatile uint64_t value;
+} __rte_cache_aligned;
+
+static struct worker_sink worker_sinks[RTE_MAX_LCORE];
+
+/*
+ * Out-of-line dummy workload: run MULTI_WORK_PASSES
+ * read-modify-write passes over the first 'bytes' of the
+ * buffer. Each pass reads what the previous pass wrote, so the
+ * compiler cannot unroll or parallelize across passes — the
+ * work scales linearly with MULTI_WORK_PASSES. Returns an
+ * accumulator so the caller can feed it into a volatile sink;
+ * without that, the compiler could elide the whole function.
+ *
+ * __rte_noinline so it looks identical to the compiler in both
+ * the baseline (pre-allocated scratch buffer) and alloc-path
+ * runs, making the cycle-delta subtraction valid.
+ *
+ * The purpose of this being tunably expensive is to keep
+ * worker-per-iteration cost high relative to the allocator's
+ * critical section, so that even serialized allocators like
+ * rte_malloc spend most of their time outside the lock and the
+ * measured per-op allocator cost reflects its own work rather
+ * than its contention queue.
+ */
+static uint64_t __rte_noinline
+touch_buffer(void *buf, size_t bytes)
+{
+ uint64_t *p = buf;
+ size_t n = bytes / sizeof(uint64_t);
+ uint64_t acc = 0;
+ unsigned int pass;
+ size_t i;
+
+ /* Prime the buffer with a known pattern. */
+ for (i = 0; i < n; i++)
+ p[i] = i * 0x9E3779B97F4A7C15ULL;
+
+ /*
+ * Dependent RMW passes: each pass reads p[i] written by
+ * the previous pass, mixes the pass index in, and writes
+ * back. The XOR into acc keeps the chain live.
+ */
+ for (pass = 0; pass < MULTI_WORK_PASSES; pass++) {
+ for (i = 0; i < n; i++) {
+ uint64_t v = p[i];
+
+ v = v * 0xC2B2AE3D27D4EB4FULL + pass;
+ v ^= v >> 33;
+ p[i] = v;
+ acc ^= v;
+ }
+ }
+
+ return acc;
+}
+
+struct worker_args {
+ const struct allocator *alloc;
+ void *scratch; /* baseline only; NULL => alloc path */
+ unsigned int iters;
+ unsigned int warmup;
+ unsigned int bulk_n; /* 0 = single-object, >0 = bulk */
+ RTE_ATOMIC(bool) start_flag; /* barrier at worker entry */
+ uint64_t cycles; /* out */
+ unsigned int ops; /* out */
+ int err; /* out */
+};
+
+static int
+worker_run(void *arg)
+{
+ struct worker_args *wa = arg;
+ unsigned int lcore = rte_lcore_id();
+ uint64_t acc = 0;
+ uint64_t t0;
+ unsigned int i;
+
+ wa->err = 0;
+ wa->ops = 0;
+ wa->cycles = 0;
+
+ /* Wait for start flag (spin-barrier set by main). */
+ while (!rte_atomic_load_explicit(&wa->start_flag,
+ rte_memory_order_acquire))
+ rte_pause();
+
+ /* Warmup. */
+ for (i = 0; i < wa->warmup; i++) {
+ void *p;
+
+ if (wa->scratch != NULL)
+ p = wa->scratch;
+ else {
+ p = wa->alloc->alloc();
+ if (p == NULL) {
+ wa->err = -1;
+ return -1;
+ }
+ }
+ acc ^= touch_buffer(p, MULTI_WORK_BYTES);
+ if (wa->scratch == NULL)
+ wa->alloc->free_obj(p);
+ }
+
+ /* Measured loop. */
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < wa->iters; i++) {
+ void *p;
+
+ if (wa->scratch != NULL)
+ p = wa->scratch;
+ else {
+ p = wa->alloc->alloc();
+ if (p == NULL) {
+ wa->err = -1;
+ break;
+ }
+ }
+ acc ^= touch_buffer(p, MULTI_WORK_BYTES);
+ if (wa->scratch == NULL)
+ wa->alloc->free_obj(p);
+ }
+ wa->cycles = rte_rdtsc_precise() - t0;
+ wa->ops = i;
+
+ /* Publish accumulator to defeat dead-code elimination. */
+ worker_sinks[lcore].value ^= acc;
+
+ return 0;
+}
+
+static int
+worker_run_bulk(void *arg)
+{
+ struct worker_args *wa = arg;
+ unsigned int lcore = rte_lcore_id();
+ void *ptrs[BATCH_N];
+ uint64_t acc = 0;
+ uint64_t t0;
+ unsigned int i, j;
+ unsigned int bulk_n = wa->bulk_n;
+
+ wa->err = 0;
+ wa->ops = 0;
+ wa->cycles = 0;
+
+ while (!rte_atomic_load_explicit(&wa->start_flag,
+ rte_memory_order_acquire))
+ rte_pause();
+
+ /* Warmup. */
+ for (i = 0; i < wa->warmup; i++) {
+ if (wa->alloc->alloc_bulk(ptrs, bulk_n) < 0) {
+ wa->err = -1;
+ return -1;
+ }
+ for (j = 0; j < bulk_n; j++)
+ acc ^= touch_buffer(ptrs[j], MULTI_WORK_BYTES);
+ wa->alloc->free_bulk(ptrs, bulk_n);
+ }
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < wa->iters; i++) {
+ if (wa->alloc->alloc_bulk(ptrs, bulk_n) < 0) {
+ wa->err = -1;
+ break;
+ }
+ for (j = 0; j < bulk_n; j++)
+ acc ^= touch_buffer(ptrs[j], MULTI_WORK_BYTES);
+ wa->alloc->free_bulk(ptrs, bulk_n);
+ }
+ wa->cycles = rte_rdtsc_precise() - t0;
+ wa->ops = i * bulk_n;
+
+ worker_sinks[lcore].value ^= acc;
+
+ return 0;
+}
+
+/*
+ * Launch workers on the first 'n_workers' worker lcores, run
+ * either the baseline (scratch != NULL) or the alloc path
+ * (scratch == NULL), and return the mean per-op cycle cost
+ * averaged across participating workers.
+ *
+ * On any worker error, returns -1.0.
+ */
+static double
+run_multi_workers(const struct allocator *alloc, unsigned int n_workers,
+ void *const *scratches, unsigned int bulk_n)
+{
+ struct worker_args wargs[RTE_MAX_LCORE];
+ unsigned int worker_lcores[MAX_MULTI_LCORES];
+ unsigned int n = 0;
+ unsigned int lcore_id;
+ unsigned int i;
+ lcore_function_t *fn = bulk_n > 0 ? worker_run_bulk : worker_run;
+
+ /* Collect the first n_workers worker lcores. */
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ if (n >= n_workers)
+ break;
+ worker_lcores[n++] = lcore_id;
+ }
+ if (n < n_workers)
+ return -1.0;
+
+ /* Prepare per-worker args. */
+ for (i = 0; i < n_workers; i++) {
+ struct worker_args *wa = &wargs[worker_lcores[i]];
+
+ wa->alloc = alloc;
+ wa->scratch = scratches != NULL ? scratches[i] : NULL;
+ wa->iters = MULTI_OPS;
+ wa->warmup = MULTI_WARMUP;
+ wa->bulk_n = bulk_n;
+ rte_atomic_store_explicit(&wa->start_flag, false,
+ rte_memory_order_relaxed);
+ }
+
+ /* Launch workers. They spin on start_flag until released. */
+ for (i = 0; i < n_workers; i++)
+ rte_eal_remote_launch(fn, &wargs[worker_lcores[i]],
+ worker_lcores[i]);
+
+ /* Release all workers roughly simultaneously. */
+ for (i = 0; i < n_workers; i++)
+ rte_atomic_store_explicit(
+ &wargs[worker_lcores[i]].start_flag, true,
+ rte_memory_order_release);
+
+ /* Wait for completion. */
+ for (i = 0; i < n_workers; i++)
+ rte_eal_wait_lcore(worker_lcores[i]);
+
+ /* Aggregate: mean cycles per op across workers. */
+ {
+ double sum_cycles_per_op = 0.0;
+ unsigned int n_ok = 0;
+
+ for (i = 0; i < n_workers; i++) {
+ struct worker_args *wa = &wargs[worker_lcores[i]];
+
+ if (wa->err != 0 || wa->ops == 0)
+ return -1.0;
+ sum_cycles_per_op +=
+ (double)wa->cycles / (double)wa->ops;
+ n_ok++;
+ }
+ return sum_cycles_per_op / n_ok;
+ }
+}
+
+/*
+ * One sub-run of Scenario 4: given an allocator and a worker
+ * count, return (baseline, alloc_path) mean cycles per op.
+ */
+static void
+run_multi_lcore(const struct allocator *alloc, unsigned int n_workers,
+ unsigned int bulk_n, double *baseline, double *alloc_path)
+{
+ void *scratches[MAX_MULTI_LCORES] = {0};
+ unsigned int n_alloced = 0;
+ unsigned int i;
+
+ *baseline = -1.0;
+ *alloc_path = -1.0;
+
+ if (alloc->setup(MULTI_SIZE, n_workers * 64) < 0)
+ return;
+
+ /* Baseline: pre-allocate one scratch per worker. */
+ for (i = 0; i < n_workers; i++) {
+ scratches[i] = alloc->alloc();
+ if (scratches[i] == NULL)
+ goto err;
+ n_alloced++;
+ }
+
+ *baseline = run_multi_workers(alloc, n_workers, scratches, 0);
+
+ for (i = 0; i < n_alloced; i++)
+ alloc->free_obj(scratches[i]);
+ n_alloced = 0;
+
+ /* Alloc path: workers alloc+free each iter. */
+ *alloc_path = run_multi_workers(alloc, n_workers, NULL, bulk_n);
+
+ alloc->teardown();
+ return;
+err:
+ for (i = 0; i < n_alloced; i++)
+ alloc->free_obj(scratches[i]);
+ alloc->teardown();
+}
+
+/* Reporting -------------------------------------------------------- */
+
+static void
+print_header(const char *title)
+{
+ size_t i;
+
+ TEST_LOG("\n=== %s ===\n", title);
+ TEST_LOG("%-12s", "allocator");
+ for (i = 0; i < N_SIZES; i++)
+ TEST_LOG(" %10zu B", SIZES[i]);
+ TEST_LOG("\n");
+}
+
+static void
+print_row(const char *name, const double *values)
+{
+ size_t i;
+
+ TEST_LOG("%-12s", name);
+ for (i = 0; i < N_SIZES; i++) {
+ if (values[i] < 0)
+ TEST_LOG(" %12s", "--");
+ else
+ TEST_LOG(" %12.1f", values[i]);
+ }
+ TEST_LOG("\n");
+}
+
+static void
+print_multi_header(const char *title, const unsigned int *lcore_counts,
+ unsigned int n_counts)
+{
+ unsigned int i;
+
+ TEST_LOG("\n=== %s ===\n", title);
+ TEST_LOG("%-12s", "allocator");
+ for (i = 0; i < n_counts; i++)
+ TEST_LOG(" %8u lcore%c", lcore_counts[i],
+ lcore_counts[i] == 1 ? ' ' : 's');
+ TEST_LOG("\n");
+}
+
+static void
+print_multi_row(const char *name, const double *values, unsigned int n_counts)
+{
+ unsigned int i;
+
+ TEST_LOG("%-12s", name);
+ for (i = 0; i < n_counts; i++) {
+ if (values[i] < 0)
+ TEST_LOG(" %14s", "--");
+ else
+ TEST_LOG(" %14.1f", values[i]);
+ }
+ TEST_LOG("\n");
+}
+
+/* Driver ----------------------------------------------------------- */
+
+static int
+test_fastmem_perf(void)
+{
+ size_t i;
+ size_t a;
+ int rc;
+
+ rc = rte_fastmem_init();
+ if (rc < 0) {
+ TEST_LOG("rte_fastmem_init() failed: %d\n", rc);
+ return -1;
+ }
+
+ rc = rte_fastmem_reserve(128 * 1024 * 1024, SOCKET_ID_ANY);
+ if (rc < 0) {
+ TEST_LOG("rte_fastmem_reserve() failed: %d\n", rc);
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ TEST_LOG("\nfastmem performance — single-lcore, fixed-size\n");
+ TEST_LOG("All numbers are TSC cycles.\n");
+
+ /* Scenario 1: tight alloc+free. */
+ print_header("Scenario 1: Single-object hot path — cycles per (alloc + free)");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ vals[i] = run_tight(&allocators[a], SIZES[i]);
+ print_row(allocators[a].name, vals);
+ }
+
+ /* Scenario 2: batched, FIFO free. */
+ print_header("Scenario 2: Batch alloc, FIFO free — cycles per alloc");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_alloc);
+ }
+ print_header("Scenario 2: Batch alloc, FIFO free — cycles per free");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_free);
+ }
+
+ /* Scenario 3: batched, reverse free. */
+ print_header("Scenario 3: Batch alloc, LIFO free — cycles per alloc");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch_reverse(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_alloc);
+ }
+ print_header("Scenario 3: Batch alloc, LIFO free — cycles per free");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch_reverse(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_free);
+ }
+
+ /* Scenario 4: multi-lcore alloc/work/free with baseline. */
+ {
+ unsigned int max_workers = rte_lcore_count() - 1;
+ unsigned int lcore_counts[8];
+ unsigned int n_counts = 0;
+ unsigned int w;
+ double base_vals[N_ALLOCATORS][8];
+ double alloc_vals[N_ALLOCATORS][8];
+ double delta_vals[N_ALLOCATORS][8];
+
+ if (max_workers > MAX_MULTI_LCORES)
+ max_workers = MAX_MULTI_LCORES;
+
+ /* Sweep lcore counts: 1, 2, 4, 8, ... up to max_workers. */
+ for (w = 1; w <= max_workers && n_counts < RTE_DIM(lcore_counts); w *= 2)
+ lcore_counts[n_counts++] = w;
+ /* Ensure max_workers is the final column if not power of two. */
+ if (n_counts > 0 && lcore_counts[n_counts - 1] != max_workers &&
+ n_counts < RTE_DIM(lcore_counts) && max_workers >= 1)
+ lcore_counts[n_counts++] = max_workers;
+
+ if (n_counts == 0) {
+ TEST_LOG("\nScenario 4 (Multi-lcore contention) skipped: no worker lcores available.\n");
+ } else {
+ TEST_LOG("\nScenario 4 parameters: size=%u B\n",
+ MULTI_SIZE);
+
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ unsigned int c;
+
+ for (c = 0; c < n_counts; c++)
+ run_multi_lcore(&allocators[a], lcore_counts[c],
+ 0, &base_vals[a][c],
+ &alloc_vals[a][c]);
+ for (c = 0; c < n_counts; c++) {
+ if (base_vals[a][c] < 0 || alloc_vals[a][c] < 0)
+ delta_vals[a][c] = -1.0;
+ else
+ delta_vals[a][c] = alloc_vals[a][c] -
+ base_vals[a][c];
+ }
+ }
+
+ TEST_LOG("Baseline (domain logic only): %.1f cycles/op\n",
+ base_vals[0][0]);
+
+ print_multi_header("Scenario 4: Multi-lcore contention — allocator overhead (cycles/op)",
+ lcore_counts, n_counts);
+ for (a = 0; a < N_ALLOCATORS; a++)
+ print_multi_row(allocators[a].name,
+ delta_vals[a], n_counts);
+ }
+ }
+
+ /* Scenario 5: multi-lcore bulk alloc/work/free. */
+ {
+ unsigned int max_workers = rte_lcore_count() - 1;
+ unsigned int lcore_counts[8];
+ unsigned int n_counts = 0;
+ unsigned int w;
+ double base_vals[N_ALLOCATORS][8];
+ double alloc_vals[N_ALLOCATORS][8];
+ double delta_vals[N_ALLOCATORS][8];
+ unsigned int bulk_n = 8;
+
+ if (max_workers > MAX_MULTI_LCORES)
+ max_workers = MAX_MULTI_LCORES;
+
+ for (w = 1; w <= max_workers && n_counts < RTE_DIM(lcore_counts); w *= 2)
+ lcore_counts[n_counts++] = w;
+ if (n_counts > 0 && lcore_counts[n_counts - 1] != max_workers &&
+ n_counts < RTE_DIM(lcore_counts) && max_workers >= 1)
+ lcore_counts[n_counts++] = max_workers;
+
+ if (n_counts == 0) {
+ TEST_LOG("\nScenario 5 (Multi-lcore bulk contention) skipped: no worker lcores available.\n");
+ } else {
+ TEST_LOG("\nScenario 5 parameters: size=%u B, "
+ "bulk=%u\n",
+ MULTI_SIZE, bulk_n);
+
+ for (size_t a = 0; a < N_ALLOCATORS; a++) {
+ unsigned int c;
+
+ for (c = 0; c < n_counts; c++)
+ run_multi_lcore(&allocators[a],
+ lcore_counts[c], bulk_n,
+ &base_vals[a][c],
+ &alloc_vals[a][c]);
+ for (c = 0; c < n_counts; c++) {
+ if (base_vals[a][c] < 0 || alloc_vals[a][c] < 0)
+ delta_vals[a][c] = -1.0;
+ else
+ delta_vals[a][c] = alloc_vals[a][c] -
+ base_vals[a][c];
+ }
+ }
+
+ TEST_LOG("Baseline (domain logic only): %.1f cycles/op\n",
+ base_vals[0][0]);
+
+ print_multi_header("Scenario 5: Multi-lcore bulk contention — allocator overhead (cycles/op)",
+ lcore_counts, n_counts);
+ for (size_t a = 0; a < N_ALLOCATORS; a++)
+ print_multi_row(allocators[a].name,
+ delta_vals[a], n_counts);
+ }
+ }
+
+ TEST_LOG("\n");
+ rte_fastmem_deinit();
+ return 0;
+}
+
+REGISTER_PERF_TEST(fastmem_perf_autotest, test_fastmem_perf);
diff --git a/app/test/test_fastmem_profile.c b/app/test/test_fastmem_profile.c
new file mode 100644
index 0000000000..9a5dc94018
--- /dev/null
+++ b/app/test/test_fastmem_profile.c
@@ -0,0 +1,157 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+/*
+ * A minimal fastmem workload intended for use with perf record /
+ * perf report. Runs a tight alloc/free loop for a fixed duration
+ * so that sampling profilers can attribute cycles to individual
+ * functions and instructions within the fastmem hot path.
+ *
+ * Usage:
+ * perf record -g -- dpdk-test --no-huge --no-pci -m 8192 \
+ * -l 0 <<< fastmem_profile_autotest
+ * perf report
+ */
+
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_lcore.h>
+#include <rte_memory.h>
+
+#include <rte_fastmem.h>
+
+#include "test.h"
+
+/* Duration of each sub-test in TSC cycles (~3 seconds at 3 GHz). */
+#define PROFILE_DURATION_CYCLES (3ULL * rte_get_tsc_hz())
+
+/* Allocation size for the profiling workload. */
+#define PROFILE_SIZE 256u
+
+/*
+ * Sub-test 1: tight alloc+free, exercises only the per-lcore
+ * cache (no bin interaction after warmup).
+ */
+static int
+profile_cache_hit(void)
+{
+ uint64_t deadline;
+ uint64_t ops = 0;
+
+ deadline = rte_rdtsc() + PROFILE_DURATION_CYCLES;
+
+ while (rte_rdtsc() < deadline) {
+ void *p = rte_fastmem_alloc(PROFILE_SIZE, 0, 0);
+
+ if (p == NULL)
+ return -1;
+ rte_fastmem_free(p);
+ ops++;
+ }
+
+ printf(" cache_hit: %" PRIu64 " ops\n", ops);
+ return 0;
+}
+
+/*
+ * Sub-test 2: alloc N then free N, where N exceeds the cache
+ * capacity. This forces repeated cache refills and drains,
+ * exercising the bin lock and slab free-list traversal.
+ */
+#define PROFILE_BATCH 256u
+
+static int
+profile_cache_miss(void)
+{
+ void *ptrs[PROFILE_BATCH];
+ uint64_t deadline;
+ uint64_t ops = 0;
+ unsigned int i;
+
+ deadline = rte_rdtsc() + PROFILE_DURATION_CYCLES;
+
+ while (rte_rdtsc() < deadline) {
+ for (i = 0; i < PROFILE_BATCH; i++) {
+ ptrs[i] = rte_fastmem_alloc(PROFILE_SIZE, 0, 0);
+ if (ptrs[i] == NULL)
+ return -1;
+ }
+ for (i = 0; i < PROFILE_BATCH; i++)
+ rte_fastmem_free(ptrs[i]);
+ ops += PROFILE_BATCH;
+ }
+
+ printf(" cache_miss: %" PRIu64 " ops\n", ops);
+ return 0;
+}
+
+static int
+test_fastmem_profile_cache_hit(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ if (rc < 0) {
+ printf("rte_fastmem_init() failed: %d\n", rc);
+ return -1;
+ }
+
+ rc = rte_fastmem_reserve(128 * 1024 * 1024, SOCKET_ID_ANY);
+ if (rc < 0) {
+ printf("rte_fastmem_reserve() failed: %d\n", rc);
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ printf("fastmem profile: cache-hit workload (size=%u, ~%u s)\n",
+ PROFILE_SIZE, 3);
+
+ if (profile_cache_hit() < 0) {
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ rte_fastmem_deinit();
+ return 0;
+}
+
+static int
+test_fastmem_profile_cache_miss(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ if (rc < 0) {
+ printf("rte_fastmem_init() failed: %d\n", rc);
+ return -1;
+ }
+
+ rc = rte_fastmem_reserve(128 * 1024 * 1024, SOCKET_ID_ANY);
+ if (rc < 0) {
+ printf("rte_fastmem_reserve() failed: %d\n", rc);
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ printf("fastmem profile: cache-miss workload (size=%u, ~%u s)\n",
+ PROFILE_SIZE, 3);
+
+ if (profile_cache_miss() < 0) {
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ rte_fastmem_deinit();
+ return 0;
+}
+
+REGISTER_PERF_TEST(fastmem_profile_cache_hit_autotest,
+ test_fastmem_profile_cache_hit);
+REGISTER_PERF_TEST(fastmem_profile_cache_miss_autotest,
+ test_fastmem_profile_cache_miss);
--
2.43.0
^ permalink raw reply related
* [RFC v3 2/3] lib: add fastmem library
From: Mattias Rönnblom @ 2026-05-27 17:30 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Bruce Richardson,
Mattias Rönnblom
In-Reply-To: <20260527173042.93867-1-hofors@lysator.liu.se>
Introduce fastmem, a fast general-purpose small-object allocator
for DPDK applications. It allows an application to replace its
many per-type mempools with a single allocator that handles
arbitrary sizes, grows on demand, and offers mempool-level
performance on the hot path.
Applications that manage many object types (connections, sessions,
work items, timers) currently maintain a separate mempool for each,
requiring upfront sizing and wasting memory on over-provisioned
pools. Fastmem removes both constraints.
Key properties:
* Huge-page-backed, NUMA-aware, DMA-usable.
* Per-lcore caches for lock-free alloc/free on EAL threads.
* Bulk alloc and free APIs.
* Power-of-two size classes from 8 B to 1 MiB.
* Backing memory grows lazily; rte_fastmem_reserve() allows
upfront reservation to avoid latency spikes.
* Always-on per-lcore and per-class statistics.
Bounded to small objects; requests above rte_fastmem_max_size()
are rejected. Replacing rte_malloc is currently not a goal.
--
RFC v3:
* Add rte_fastmem_realloc().
* Add __rte_malloc and __rte_dealloc attributes to allocation functions.
* Remove __rte_alloc_size and __rte_alloc_align attributes.
These told the compiler the object is exactly the requested
size, but fastmem rounds up to the size class and the caller
may use the full class size. The mismatch caused false
_FORTIFY_SOURCE buffer-overflow aborts.
* Extract normalize_align() helper replacing repeated inline
alignment validation logic.
* Remove inline directives from static functions (redundant;
both GCC and clang inline them at -O2 regardless).
RFC v2:
* Fix use-after-free in rte_fastmem_deinit() when caches were
allocated cross-socket. Restructured teardown into three phases.
* Add defensive bounds check to local_socket_id() final fallback.
* Add secondary process support. Shared state is discovered lazily
on first allocation; secondaries operate without per-lcore caches.
* Add handle-based allocation API (rte_fastmem_hlookup,
rte_fastmem_halloc, rte_fastmem_halloc_bulk).
* Fix clang -Wthread-safety-analysis warnings.
* Move fastmem to alphabetical position in lib/meson.build.
Signed-off-by: Mattias Rönnblom <hofors@lysator.liu.se>
---
doc/api/doxy-api-index.md | 1 +
doc/api/doxy-api.conf.in | 1 +
lib/fastmem/meson.build | 6 +
lib/fastmem/rte_fastmem.c | 1748 +++++++++++++++++++++++++++++++++++++
lib/fastmem/rte_fastmem.h | 815 +++++++++++++++++
lib/meson.build | 1 +
6 files changed, 2572 insertions(+)
create mode 100644 lib/fastmem/meson.build
create mode 100644 lib/fastmem/rte_fastmem.c
create mode 100644 lib/fastmem/rte_fastmem.h
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 9296042119..7ebf1201ce 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -70,6 +70,7 @@ The public API headers are grouped by topics:
[memzone](@ref rte_memzone.h),
[mempool](@ref rte_mempool.h),
[malloc](@ref rte_malloc.h),
+ [fastmem](@ref rte_fastmem.h),
[memcpy](@ref rte_memcpy.h)
- **timers**:
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index bedd944681..4355e9fb2d 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -43,6 +43,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \
@TOPDIR@/lib/efd \
@TOPDIR@/lib/ethdev \
@TOPDIR@/lib/eventdev \
+ @TOPDIR@/lib/fastmem \
@TOPDIR@/lib/fib \
@TOPDIR@/lib/gpudev \
@TOPDIR@/lib/graph \
diff --git a/lib/fastmem/meson.build b/lib/fastmem/meson.build
new file mode 100644
index 0000000000..6c7834608f
--- /dev/null
+++ b/lib/fastmem/meson.build
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2026 Ericsson AB
+
+sources = files('rte_fastmem.c')
+headers = files('rte_fastmem.h')
+deps += ['eal']
diff --git a/lib/fastmem/rte_fastmem.c b/lib/fastmem/rte_fastmem.c
new file mode 100644
index 0000000000..5eff2ff693
--- /dev/null
+++ b/lib/fastmem/rte_fastmem.c
@@ -0,0 +1,1748 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/queue.h>
+
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_eal.h>
+#include <rte_errno.h>
+#include <rte_lcore.h>
+#include <rte_log.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_spinlock.h>
+
+#include <rte_fastmem.h>
+
+#include <eal_export.h>
+
+RTE_LOG_REGISTER_DEFAULT(fastmem_logtype, NOTICE);
+
+#define RTE_LOGTYPE_FASTMEM fastmem_logtype
+
+#define FASTMEM_LOG(level, ...) \
+ RTE_LOG_LINE(level, FASTMEM, "" __VA_ARGS__)
+
+#define FASTMEM_MEMZONE_SIZE_LOG2 27 /* 128 MiB */
+#define FASTMEM_MEMZONE_SIZE ((size_t)1 << FASTMEM_MEMZONE_SIZE_LOG2)
+
+#define FASTMEM_SLAB_SIZE_LOG2 21 /* 2 MiB */
+#define FASTMEM_SLAB_SIZE ((size_t)1 << FASTMEM_SLAB_SIZE_LOG2)
+#define FASTMEM_SLAB_MASK (FASTMEM_SLAB_SIZE - 1)
+
+#define FASTMEM_SLABS_PER_MEMZONE (FASTMEM_MEMZONE_SIZE / FASTMEM_SLAB_SIZE)
+
+#define FASTMEM_MAX_MEMZONES_PER_SOCKET 64
+
+#define FASTMEM_MIN_CLASS_LOG2 3 /* 8 B */
+#define FASTMEM_MAX_CLASS_LOG2 20 /* 1 MiB */
+#define FASTMEM_N_CLASSES (FASTMEM_MAX_CLASS_LOG2 - FASTMEM_MIN_CLASS_LOG2 + 1)
+
+#define FASTMEM_MIN_SIZE ((size_t)1 << FASTMEM_MIN_CLASS_LOG2)
+#define FASTMEM_MAX_ALLOC_SIZE ((size_t)1 << FASTMEM_MAX_CLASS_LOG2)
+
+#define FASTMEM_SLAB_HEADER_SIZE RTE_CACHE_LINE_SIZE
+
+#define FASTMEM_CACHE_BASE_CAPACITY 64
+#define FASTMEM_CACHE_FLOOR_CAPACITY 4
+#define FASTMEM_CACHE_BASE_CLASS_LOG2 12 /* 4 KiB */
+
+struct fastmem_bin;
+
+/*
+ * Slab header at offset 0 of each 2 MiB slab. Either free (linked
+ * via next_free) or assigned to a bin (linked via list).
+ */
+struct fastmem_slab {
+ struct fastmem_bin *bin;
+ void *free_head;
+ uint32_t free_count;
+ uint32_t n_slots;
+ struct fastmem_slab *next_free;
+ TAILQ_ENTRY(fastmem_slab) list;
+ rte_iova_t iova_base;
+} __rte_aligned(FASTMEM_SLAB_HEADER_SIZE);
+
+TAILQ_HEAD(fastmem_slab_list, fastmem_slab);
+
+struct fastmem_bin {
+ rte_spinlock_t lock;
+ uint32_t slot_size;
+ uint32_t slots_per_slab;
+ uint32_t class_idx;
+ struct fastmem_slab_list partial;
+ struct fastmem_slab_list full;
+ int socket_id;
+ uint64_t slab_acquires;
+ uint64_t slab_releases;
+ uint32_t slabs_partial;
+ uint32_t slabs_full;
+};
+
+/* Per-(lcore, class, socket) bounded LIFO of free object pointers. */
+struct fastmem_cache {
+ uint32_t count;
+ uint32_t capacity;
+ uint32_t target;
+ uint64_t alloc_cache_hits;
+ uint64_t alloc_cache_misses;
+ uint64_t alloc_nomem;
+ uint64_t free_cache_hits;
+ uint64_t free_cache_misses;
+ void *objs[];
+} __rte_cache_aligned;
+
+struct fastmem_socket_state {
+ rte_spinlock_t lock;
+ struct fastmem_slab *free_head;
+ size_t reserved_bytes;
+ size_t memory_limit;
+ unsigned int n_memzones;
+ unsigned int memzone_seq;
+ const struct rte_memzone *memzones[FASTMEM_MAX_MEMZONES_PER_SOCKET];
+ struct fastmem_bin bins[FASTMEM_N_CLASSES];
+ struct fastmem_cache *caches[RTE_MAX_LCORE][FASTMEM_N_CLASSES];
+};
+
+struct fastmem {
+ struct fastmem_socket_state sockets[RTE_MAX_NUMA_NODES];
+};
+
+static struct fastmem *fastmem;
+static const struct rte_memzone *fastmem_mz;
+static bool fastmem_is_primary; /* cached; avoids function call on hot path */
+
+static struct fastmem *
+fastmem_get(void)
+{
+ const struct rte_memzone *mz;
+
+ if (likely(fastmem != NULL))
+ return fastmem;
+
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ rte_errno = ENODEV;
+ return NULL;
+ }
+
+ mz = rte_memzone_lookup("fastmem_state");
+ if (mz == NULL) {
+ rte_errno = ENODEV;
+ return NULL;
+ }
+
+ fastmem_mz = mz;
+ fastmem = mz->addr;
+ return fastmem;
+}
+
+static unsigned int
+size_to_class(size_t size, size_t align)
+{
+ size_t effective;
+ unsigned int log2;
+
+ effective = size < FASTMEM_MIN_SIZE ? FASTMEM_MIN_SIZE : size;
+ if (align > effective)
+ effective = align;
+
+ log2 = 64u - rte_clz64(effective - 1);
+
+ if (log2 < FASTMEM_MIN_CLASS_LOG2)
+ log2 = FASTMEM_MIN_CLASS_LOG2;
+ if (log2 > FASTMEM_MAX_CLASS_LOG2)
+ return FASTMEM_N_CLASSES;
+
+ return log2 - FASTMEM_MIN_CLASS_LOG2;
+}
+
+static size_t
+class_size(unsigned int class_idx)
+{
+ return (size_t)1 << (class_idx + FASTMEM_MIN_CLASS_LOG2);
+}
+
+/**
+ * Normalize and validate the alignment argument.
+ * Returns true on success (align updated in place), false on invalid input.
+ */
+static bool
+normalize_align(size_t *align)
+{
+ if (*align == 0) {
+ *align = RTE_CACHE_LINE_SIZE;
+ return true;
+ }
+ return rte_is_power_of_2(*align);
+}
+
+static_assert(sizeof(struct fastmem_slab) == FASTMEM_SLAB_HEADER_SIZE,
+ "fastmem slab header must fit in exactly one cache line");
+static_assert(sizeof(struct fastmem_slab) <= FASTMEM_SLAB_SIZE,
+ "slab header larger than a slab makes no sense");
+
+static struct fastmem_slab *
+slab_of(void *obj)
+{
+ return (struct fastmem_slab *)
+ ((uintptr_t)obj & ~(uintptr_t)FASTMEM_SLAB_MASK);
+}
+
+static size_t
+slab_slot0_offset(size_t class_size)
+{
+ return class_size < FASTMEM_SLAB_HEADER_SIZE ?
+ FASTMEM_SLAB_HEADER_SIZE : class_size;
+}
+
+static uint32_t
+slab_slot_count(size_t class_size)
+{
+ size_t offset = slab_slot0_offset(class_size);
+
+ return (uint32_t)((FASTMEM_SLAB_SIZE - offset) / class_size);
+}
+
+/* Must be called with bin->lock held. */
+static void
+slab_init(struct fastmem_bin *bin, struct fastmem_slab *slab)
+{
+ size_t slot_size = bin->slot_size;
+ size_t offset = slab_slot0_offset(slot_size);
+ uint32_t n = bin->slots_per_slab;
+ void *prev = NULL;
+ uint32_t i;
+
+ slab->bin = bin;
+ slab->n_slots = n;
+ slab->free_count = n;
+
+ /* Build in reverse so pops yield sequential addresses. */
+ for (i = 0; i < n; i++) {
+ void *slot = RTE_PTR_ADD(slab, offset + i * slot_size);
+ *(void **)slot = prev;
+ prev = slot;
+ }
+ slab->free_head = prev;
+}
+
+static int
+grow_socket(struct fastmem_socket_state *socket, int socket_id)
+{
+ char name[RTE_MEMZONE_NAMESIZE];
+ const struct rte_memzone *mz;
+ unsigned int i;
+
+ if (socket->reserved_bytes + FASTMEM_MEMZONE_SIZE > socket->memory_limit) {
+ FASTMEM_LOG(ERR,
+ "reserve would exceed memory_limit (%zu) on socket %d",
+ socket->memory_limit, socket_id);
+ return -ENOMEM;
+ }
+
+ if (socket->n_memzones == FASTMEM_MAX_MEMZONES_PER_SOCKET) {
+ FASTMEM_LOG(ERR,
+ "reached per-socket memzone cap (%u) on socket %d",
+ FASTMEM_MAX_MEMZONES_PER_SOCKET, socket_id);
+ return -ENOMEM;
+ }
+
+ snprintf(name, sizeof(name), "fastmem_%d_%u", socket_id,
+ socket->memzone_seq++);
+
+ mz = rte_memzone_reserve_aligned(name, FASTMEM_MEMZONE_SIZE,
+ socket_id, RTE_MEMZONE_IOVA_CONTIG,
+ FASTMEM_SLAB_SIZE);
+ if (mz == NULL) {
+ FASTMEM_LOG(ERR,
+ "failed to reserve %zu-byte memzone '%s' on socket %d: %s",
+ (size_t)FASTMEM_MEMZONE_SIZE, name, socket_id,
+ rte_strerror(rte_errno));
+ return -ENOMEM;
+ }
+
+ socket->memzones[socket->n_memzones++] = mz;
+ socket->reserved_bytes += FASTMEM_MEMZONE_SIZE;
+
+ for (i = 0; i < FASTMEM_SLABS_PER_MEMZONE; i++) {
+ struct fastmem_slab *slab = RTE_PTR_ADD(mz->addr,
+ i * FASTMEM_SLAB_SIZE);
+
+ slab->iova_base = mz->iova + i * FASTMEM_SLAB_SIZE;
+ slab->next_free = socket->free_head;
+ socket->free_head = slab;
+ }
+
+ FASTMEM_LOG(DEBUG,
+ "reserved memzone '%s' (%zu bytes) on socket %d; %zu slabs added",
+ name, (size_t)FASTMEM_MEMZONE_SIZE, socket_id,
+ (size_t)FASTMEM_SLABS_PER_MEMZONE);
+
+ return 0;
+}
+
+static struct fastmem_slab *
+slab_acquire(struct fastmem_socket_state *socket, int socket_id)
+{
+ struct fastmem_slab *slab;
+
+ rte_spinlock_lock(&socket->lock);
+
+ if (socket->free_head == NULL) {
+ int rc = grow_socket(socket, socket_id);
+
+ if (rc < 0) {
+ rte_spinlock_unlock(&socket->lock);
+ return NULL;
+ }
+ }
+
+ slab = socket->free_head;
+ socket->free_head = slab->next_free;
+ slab->next_free = NULL;
+
+ rte_spinlock_unlock(&socket->lock);
+
+ return slab;
+}
+
+static void
+slab_release(struct fastmem_socket_state *socket,
+ struct fastmem_slab *slab)
+{
+ rte_spinlock_lock(&socket->lock);
+
+ slab->next_free = socket->free_head;
+ socket->free_head = slab;
+
+ rte_spinlock_unlock(&socket->lock);
+}
+
+static void
+bin_init(struct fastmem_bin *bin, unsigned int class_idx, int socket_id)
+{
+ size_t slot_size = class_size(class_idx);
+
+ rte_spinlock_init(&bin->lock);
+ bin->slot_size = (uint32_t)slot_size;
+ bin->slots_per_slab = slab_slot_count(slot_size);
+ bin->class_idx = class_idx;
+ TAILQ_INIT(&bin->partial);
+ TAILQ_INIT(&bin->full);
+ bin->socket_id = socket_id;
+ bin->slab_acquires = 0;
+ bin->slab_releases = 0;
+ bin->slabs_partial = 0;
+ bin->slabs_full = 0;
+}
+
+static void
+bin_release(struct fastmem_bin *bin, struct fastmem_socket_state *socket)
+{
+ struct fastmem_slab *slab;
+
+ while ((slab = TAILQ_FIRST(&bin->partial)) != NULL) {
+ TAILQ_REMOVE(&bin->partial, slab, list);
+ slab_release(socket, slab);
+ }
+ while ((slab = TAILQ_FIRST(&bin->full)) != NULL) {
+ TAILQ_REMOVE(&bin->full, slab, list);
+ slab_release(socket, slab);
+ }
+}
+
+static unsigned int
+bin_pop_locked(struct fastmem_bin *bin, void **objs, unsigned int n)
+{
+ unsigned int got = 0;
+
+ while (got < n) {
+ struct fastmem_slab *slab = TAILQ_FIRST(&bin->partial);
+ void *obj;
+
+ if (slab == NULL)
+ break;
+
+ obj = slab->free_head;
+ slab->free_head = *(void **)obj;
+ slab->free_count--;
+ objs[got++] = obj;
+
+ if (slab->free_count == 0) {
+ TAILQ_REMOVE(&bin->partial, slab, list);
+ TAILQ_INSERT_HEAD(&bin->full, slab, list);
+ bin->slabs_partial--;
+ bin->slabs_full++;
+ }
+ }
+
+ return got;
+}
+
+/*
+ * Fully-drained slabs are accumulated in @p to_release for the
+ * caller to return after dropping the lock.
+ */
+static unsigned int
+bin_push_locked(struct fastmem_bin *bin, void **objs, unsigned int n,
+ struct fastmem_slab **to_release)
+{
+ unsigned int n_release = 0;
+ unsigned int i;
+
+ for (i = 0; i < n; i++) {
+ void *obj = objs[i];
+ struct fastmem_slab *slab = (struct fastmem_slab *)
+ ((uintptr_t)obj & ~(uintptr_t)FASTMEM_SLAB_MASK);
+ bool was_full = slab->free_count == 0;
+
+ *(void **)obj = slab->free_head;
+ slab->free_head = obj;
+ slab->free_count++;
+
+ if (was_full) {
+ TAILQ_REMOVE(&bin->full, slab, list);
+ TAILQ_INSERT_HEAD(&bin->partial, slab, list);
+ bin->slabs_full--;
+ bin->slabs_partial++;
+ }
+
+ if (slab->free_count == slab->n_slots) {
+ TAILQ_REMOVE(&bin->partial, slab, list);
+ bin->slabs_partial--;
+ bin->slab_releases++;
+ to_release[n_release++] = slab;
+ }
+ }
+
+ return n_release;
+}
+
+static void *
+bin_alloc_one(struct fastmem_bin *bin)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[bin->socket_id];
+ void *obj;
+
+ rte_spinlock_lock(&bin->lock);
+
+ while (bin_pop_locked(bin, &obj, 1) == 0) {
+ struct fastmem_slab *slab;
+
+ if (TAILQ_FIRST(&bin->partial) != NULL)
+ continue;
+
+ rte_spinlock_unlock(&bin->lock);
+
+ slab = slab_acquire(socket, bin->socket_id);
+ if (slab == NULL) {
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ rte_spinlock_lock(&bin->lock);
+
+ if (unlikely(TAILQ_FIRST(&bin->partial) != NULL)) {
+ /* Release surplus slab without holding bin->lock. */
+ rte_spinlock_unlock(&bin->lock);
+ slab_release(socket, slab);
+ rte_spinlock_lock(&bin->lock);
+ } else {
+ slab_init(bin, slab);
+ TAILQ_INSERT_HEAD(&bin->partial, slab, list);
+ bin->slabs_partial++;
+ bin->slab_acquires++;
+ }
+ }
+
+ rte_spinlock_unlock(&bin->lock);
+
+ return obj;
+}
+
+static unsigned int
+bin_alloc_bulk(struct fastmem_bin *bin, void **objs, unsigned int n)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[bin->socket_id];
+ unsigned int got = 0;
+
+ rte_spinlock_lock(&bin->lock);
+
+ while (got < n) {
+ struct fastmem_slab *slab;
+
+ got += bin_pop_locked(bin, objs + got, n - got);
+ if (got == n)
+ break;
+
+ if (TAILQ_FIRST(&bin->partial) != NULL)
+ continue;
+
+ rte_spinlock_unlock(&bin->lock);
+
+ slab = slab_acquire(socket, bin->socket_id);
+ if (slab == NULL) {
+ rte_spinlock_lock(&bin->lock);
+ break;
+ }
+
+ rte_spinlock_lock(&bin->lock);
+
+ if (unlikely(TAILQ_FIRST(&bin->partial) != NULL)) {
+ /* Release surplus slab without holding bin->lock. */
+ rte_spinlock_unlock(&bin->lock);
+ slab_release(socket, slab);
+ rte_spinlock_lock(&bin->lock);
+ } else {
+ slab_init(bin, slab);
+ TAILQ_INSERT_HEAD(&bin->partial, slab, list);
+ bin->slabs_partial++;
+ bin->slab_acquires++;
+ }
+ }
+
+ rte_spinlock_unlock(&bin->lock);
+
+ return got;
+}
+
+static void
+bin_free_one(struct fastmem_bin *bin, void *obj)
+{
+ unsigned int n_release;
+ struct fastmem_slab *slab_to_release = NULL;
+ struct fastmem_socket_state *socket;
+
+ rte_spinlock_lock(&bin->lock);
+ n_release = bin_push_locked(bin, &obj, 1, &slab_to_release);
+ rte_spinlock_unlock(&bin->lock);
+
+ if (n_release > 0) {
+ socket = &fastmem->sockets[bin->socket_id];
+ slab_release(socket, slab_to_release);
+ }
+}
+
+static void
+bin_free_bulk(struct fastmem_bin *bin, void **objs, unsigned int n)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[bin->socket_id];
+ struct fastmem_slab *to_release[FASTMEM_CACHE_BASE_CAPACITY];
+ unsigned int n_release;
+ unsigned int i;
+
+ RTE_VERIFY(n <= RTE_DIM(to_release));
+
+ rte_spinlock_lock(&bin->lock);
+ n_release = bin_push_locked(bin, objs, n, to_release);
+ rte_spinlock_unlock(&bin->lock);
+
+ for (i = 0; i < n_release; i++)
+ slab_release(socket, to_release[i]);
+}
+
+static unsigned int
+cache_capacity(unsigned int class_idx)
+{
+ unsigned int class_log2 = class_idx + FASTMEM_MIN_CLASS_LOG2;
+ unsigned int shift;
+ unsigned int cap;
+
+ if (class_log2 <= FASTMEM_CACHE_BASE_CLASS_LOG2)
+ return FASTMEM_CACHE_BASE_CAPACITY;
+
+ shift = class_log2 - FASTMEM_CACHE_BASE_CLASS_LOG2;
+ cap = FASTMEM_CACHE_BASE_CAPACITY >> shift;
+
+ return cap < FASTMEM_CACHE_FLOOR_CAPACITY ?
+ FASTMEM_CACHE_FLOOR_CAPACITY : cap;
+}
+
+static struct fastmem_cache **
+cache_slot(struct fastmem_socket_state *socket, unsigned int class_idx,
+ unsigned int lcore_id)
+{
+ if (lcore_id >= RTE_MAX_LCORE)
+ return NULL;
+ return &socket->caches[lcore_id][class_idx];
+}
+
+static struct fastmem_cache *
+cache_create(struct fastmem_socket_state *socket,
+ unsigned int class_idx, unsigned int lcore_id)
+{
+ struct fastmem_cache **slot = cache_slot(socket, class_idx, lcore_id);
+ struct fastmem_cache *cache;
+ unsigned int capacity;
+ size_t cache_size;
+ unsigned int cache_class;
+ unsigned int own_socket;
+ struct fastmem_socket_state *alloc_socket;
+
+ if (slot == NULL)
+ return NULL;
+
+ cache = *slot;
+ if (cache != NULL)
+ return cache;
+
+ capacity = cache_capacity(class_idx);
+ cache_size = sizeof(*cache) + capacity * sizeof(void *);
+
+ /*
+ * Allocate the cache struct from fastmem on the calling
+ * lcore's socket (NUMA-local to the writer). Bypasses the
+ * cache layer to avoid recursion.
+ */
+ cache_class = size_to_class(cache_size, RTE_CACHE_LINE_SIZE);
+ own_socket = rte_socket_id();
+
+ if (cache_class >= FASTMEM_N_CLASSES) {
+ FASTMEM_LOG(ERR,
+ "cache size %zu exceeds max size class",
+ cache_size);
+ return NULL;
+ }
+
+ if (own_socket >= RTE_MAX_NUMA_NODES)
+ own_socket = (unsigned int)socket->bins[0].socket_id;
+
+ alloc_socket = &fastmem->sockets[own_socket];
+
+ cache = bin_alloc_one(&alloc_socket->bins[cache_class]);
+ if (cache == NULL) {
+ FASTMEM_LOG(ERR,
+ "failed to allocate cache for class %u on socket %u",
+ class_idx, own_socket);
+ return NULL;
+ }
+
+ cache->count = 0;
+ cache->capacity = capacity;
+ cache->target = capacity / 2;
+ cache->alloc_cache_hits = 0;
+ cache->alloc_cache_misses = 0;
+ cache->alloc_nomem = 0;
+ cache->free_cache_hits = 0;
+ cache->free_cache_misses = 0;
+
+ *slot = cache;
+
+ return cache;
+}
+
+static struct fastmem_cache *
+cache_get(struct fastmem_socket_state *socket, unsigned int class_idx,
+ unsigned int lcore_id)
+{
+ struct fastmem_cache **slot;
+ struct fastmem_cache *cache;
+
+ if (unlikely(!fastmem_is_primary))
+ return NULL;
+
+ slot = cache_slot(socket, class_idx, lcore_id);
+
+ if (slot == NULL)
+ return NULL;
+
+ cache = *slot;
+ if (cache != NULL)
+ return cache;
+
+ return cache_create(socket, class_idx, lcore_id);
+}
+
+static void *
+cache_pop(struct fastmem_cache *cache, struct fastmem_bin *bin)
+{
+ if (cache->count > 0) {
+ cache->alloc_cache_hits++;
+ return cache->objs[--cache->count];
+ }
+
+ cache->count = bin_alloc_bulk(bin, cache->objs, cache->target);
+ if (cache->count == 0)
+ return NULL;
+
+ cache->alloc_cache_misses++;
+ return cache->objs[--cache->count];
+}
+
+static void
+cache_push(struct fastmem_cache *cache, struct fastmem_bin *bin, void *obj)
+{
+ unsigned int drain;
+
+ if (cache->count < cache->capacity) {
+ cache->free_cache_hits++;
+ cache->objs[cache->count++] = obj;
+ return;
+ }
+
+ cache->free_cache_misses++;
+
+ /*
+ * Drain the oldest (bottom) half to the bin, keeping the
+ * newest (top) half for temporal reuse.
+ */
+ drain = cache->count - cache->target;
+ bin_free_bulk(bin, cache->objs, drain);
+ memmove(cache->objs, cache->objs + drain,
+ cache->target * sizeof(cache->objs[0]));
+ cache->count = cache->target;
+
+ cache->objs[cache->count++] = obj;
+}
+
+static void
+socket_release_caches(struct fastmem_socket_state *socket)
+{
+ unsigned int lcore;
+ unsigned int c;
+
+ for (lcore = 0; lcore < RTE_MAX_LCORE; lcore++) {
+ for (c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_cache *cache = socket->caches[lcore][c];
+ struct fastmem_slab *cache_slab;
+
+ if (cache == NULL)
+ continue;
+
+ if (cache->count > 0) {
+ bin_free_bulk(&socket->bins[c],
+ cache->objs, cache->count);
+ cache->count = 0;
+ }
+
+ cache_slab = slab_of(cache);
+ bin_free_one(cache_slab->bin, cache);
+
+ socket->caches[lcore][c] = NULL;
+ }
+ }
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_init, 24.11)
+rte_fastmem_init(void)
+{
+ unsigned int s, c;
+
+ if (fastmem != NULL)
+ return -EBUSY;
+
+ fastmem_mz = rte_memzone_reserve_aligned("fastmem_state",
+ sizeof(*fastmem), SOCKET_ID_ANY, 0,
+ RTE_CACHE_LINE_SIZE);
+ if (fastmem_mz == NULL)
+ return -ENOMEM;
+
+ fastmem = fastmem_mz->addr;
+ fastmem_is_primary = true;
+ memset(fastmem, 0, sizeof(*fastmem));
+
+ for (s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ rte_spinlock_init(&socket->lock);
+ socket->memory_limit = SIZE_MAX;
+
+ for (c = 0; c < FASTMEM_N_CLASSES; c++)
+ bin_init(&socket->bins[c], c, (int)s);
+ }
+
+ return 0;
+}
+
+static void
+release_socket_caches(struct fastmem_socket_state *socket)
+{
+ socket_release_caches(socket);
+}
+
+static void
+release_socket_bins(struct fastmem_socket_state *socket)
+{
+ unsigned int c;
+
+ for (c = 0; c < FASTMEM_N_CLASSES; c++)
+ bin_release(&socket->bins[c], socket);
+}
+
+static void
+release_socket_memzones(struct fastmem_socket_state *socket)
+{
+ unsigned int i;
+
+ for (i = 0; i < socket->n_memzones; i++)
+ rte_memzone_free(socket->memzones[i]);
+
+ socket->free_head = NULL;
+ socket->reserved_bytes = 0;
+ socket->n_memzones = 0;
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_deinit, 24.11)
+rte_fastmem_deinit(void)
+{
+ unsigned int i;
+
+ if (fastmem == NULL)
+ return;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ fastmem = NULL;
+ fastmem_mz = NULL;
+ return;
+ }
+
+ for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ release_socket_caches(&fastmem->sockets[i]);
+
+ for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ release_socket_bins(&fastmem->sockets[i]);
+
+ for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ release_socket_memzones(&fastmem->sockets[i]);
+
+ rte_memzone_free(fastmem_mz);
+ fastmem_mz = NULL;
+ fastmem = NULL;
+}
+
+/* Same resolution order as rte_malloc's malloc_get_numa_socket(). */
+static unsigned int
+local_socket_id(void)
+{
+ int sid = (int)rte_socket_id();
+
+ if (likely(sid >= 0 && sid < RTE_MAX_NUMA_NODES))
+ return sid;
+
+ sid = (int)rte_lcore_to_socket_id(rte_get_main_lcore());
+ if (likely(sid >= 0 && sid < RTE_MAX_NUMA_NODES))
+ return sid;
+
+ sid = rte_socket_id_by_idx(0);
+ if (likely(sid >= 0 && sid < RTE_MAX_NUMA_NODES))
+ return sid;
+
+ return 0;
+}
+
+static int
+reserve_on_socket(int sid, size_t size)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[sid];
+ int rc = 0;
+
+ rte_spinlock_lock(&socket->lock);
+
+ while (socket->reserved_bytes < size) {
+ rc = grow_socket(socket, sid);
+ if (rc < 0)
+ break;
+ }
+
+ rte_spinlock_unlock(&socket->lock);
+
+ return rc;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_reserve, 24.11)
+rte_fastmem_reserve(size_t size, int socket_id)
+{
+ unsigned int i;
+ int rc;
+
+ if (fastmem == NULL)
+ return -EINVAL;
+
+ if (socket_id != SOCKET_ID_ANY) {
+ if (socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return -EINVAL;
+ return reserve_on_socket(socket_id, size);
+ }
+
+ rc = reserve_on_socket(local_socket_id(), size);
+ if (rc == 0)
+ return 0;
+
+ for (i = 0; i < rte_socket_count(); i++) {
+ int sid = rte_socket_id_by_idx(i);
+
+ if (sid < 0 || (unsigned int)sid == local_socket_id())
+ continue;
+
+ rc = reserve_on_socket(sid, size);
+ if (rc == 0)
+ return 0;
+ }
+
+ return rc;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_set_limit, 24.11)
+rte_fastmem_set_limit(int socket_id, size_t max_bytes)
+{
+ if (fastmem == NULL)
+ return -EINVAL;
+
+ if (socket_id == SOCKET_ID_ANY) {
+ for (unsigned int i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ fastmem->sockets[i].memory_limit = max_bytes;
+ return 0;
+ }
+
+ if (socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return -EINVAL;
+
+ fastmem->sockets[socket_id].memory_limit = max_bytes;
+ return 0;
+}
+
+size_t
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_get_limit, 24.11)
+rte_fastmem_get_limit(int socket_id)
+{
+ if (fastmem == NULL || socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return 0;
+
+ return fastmem->sockets[socket_id].memory_limit;
+}
+
+size_t
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_max_size, 24.11)
+rte_fastmem_max_size(void)
+{
+ return FASTMEM_MAX_ALLOC_SIZE;
+}
+
+static void *
+alloc_from_socket(struct fastmem_socket_state *socket,
+ unsigned int class_idx, unsigned int lcore_id)
+{
+ struct fastmem_cache *cache;
+
+ cache = cache_get(socket, class_idx, lcore_id);
+ if (likely(cache != NULL))
+ return cache_pop(cache, &socket->bins[class_idx]);
+ return bin_alloc_one(&socket->bins[class_idx]);
+}
+
+static void
+do_free(void *ptr)
+{
+ struct fastmem_slab *slab;
+ struct fastmem_bin *bin;
+ struct fastmem_socket_state *socket;
+ unsigned int lcore_id;
+ struct fastmem_cache *cache;
+
+ slab = slab_of(ptr);
+ bin = slab->bin;
+ socket = &fastmem->sockets[bin->socket_id];
+
+ lcore_id = rte_lcore_id();
+ cache = cache_get(socket, bin->class_idx, lcore_id);
+ if (likely(cache != NULL))
+ cache_push(cache, bin, ptr);
+ else
+ bin_free_one(bin, ptr);
+}
+
+static int
+do_alloc_bulk(void **ptrs, unsigned int n, size_t size, size_t align,
+ unsigned int flags, unsigned int lcore_id,
+ int socket_id, bool fallback)
+{
+ unsigned int class_idx;
+ struct fastmem_socket_state *socket;
+ struct fastmem_cache *cache;
+ unsigned int got = 0;
+
+ if (unlikely(fastmem_get() == NULL))
+ return -rte_errno;
+
+ if (unlikely(!normalize_align(&align))) {
+ rte_errno = EINVAL;
+ return -EINVAL;
+ }
+
+ class_idx = size_to_class(size, align);
+ if (unlikely(class_idx >= FASTMEM_N_CLASSES)) {
+ rte_errno = E2BIG;
+ return -E2BIG;
+ }
+
+ socket = &fastmem->sockets[socket_id];
+ cache = cache_get(socket, class_idx, lcore_id);
+
+ if (likely(cache != NULL)) {
+ /* Drain from cache. */
+ unsigned int avail = RTE_MIN(cache->count, n);
+
+ cache->count -= avail;
+ memcpy(ptrs, &cache->objs[cache->count],
+ avail * sizeof(void *));
+ got = avail;
+ cache->alloc_cache_hits += avail;
+
+ if (got < n) {
+ unsigned int need = n - got;
+ unsigned int want = RTE_MAX(need, cache->target);
+ unsigned int filled;
+
+ if (want <= cache->capacity) {
+ /* Refill into cache, give caller their share. */
+ filled = bin_alloc_bulk(
+ &socket->bins[class_idx],
+ cache->objs, want);
+ if (filled > 0) {
+ cache->alloc_cache_misses += RTE_MIN(filled, need);
+ }
+ if (filled >= need) {
+ memcpy(ptrs + got,
+ cache->objs + filled - need,
+ need * sizeof(void *));
+ cache->count = filled - need;
+ got = n;
+ } else {
+ memcpy(ptrs + got, cache->objs,
+ filled * sizeof(void *));
+ got += filled;
+ cache->count = 0;
+ }
+ } else {
+ /* n exceeds cache capacity; pull directly. */
+ unsigned int direct = bin_alloc_bulk(
+ &socket->bins[class_idx],
+ ptrs + got, need);
+ if (direct > 0)
+ cache->alloc_cache_misses += direct;
+ got += direct;
+ }
+ }
+ } else {
+ got = bin_alloc_bulk(&socket->bins[class_idx], ptrs, n);
+ }
+
+ if (unlikely(got < n) && fallback) {
+ unsigned int i;
+
+ for (i = 0; i < rte_socket_count() && got < n; i++) {
+ int sid = rte_socket_id_by_idx(i);
+
+ if (sid < 0 || sid == socket_id)
+ continue;
+
+ socket = &fastmem->sockets[sid];
+ cache = cache_get(socket, class_idx, lcore_id);
+ if (likely(cache != NULL)) {
+ unsigned int avail =
+ RTE_MIN(cache->count, n - got);
+ cache->count -= avail;
+ memcpy(ptrs + got,
+ &cache->objs[cache->count],
+ avail * sizeof(void *));
+ cache->alloc_cache_hits += avail;
+ got += avail;
+ }
+ if (got < n) {
+ unsigned int direct = bin_alloc_bulk(
+ &socket->bins[class_idx],
+ ptrs + got, n - got);
+ if (direct > 0 && cache != NULL)
+ cache->alloc_cache_misses += direct;
+ got += direct;
+ }
+ }
+ }
+
+ if (unlikely(got < n)) {
+ /* All-or-nothing: return what we got. */
+ struct fastmem_cache **slot;
+ unsigned int i;
+
+ for (i = 0; i < got; i++)
+ do_free(ptrs[i]);
+
+ slot = cache_slot(
+ &fastmem->sockets[socket_id], class_idx,
+ lcore_id);
+ if (slot != NULL && *slot != NULL)
+ (*slot)->alloc_nomem++;
+ rte_errno = ENOMEM;
+ return -ENOMEM;
+ }
+
+ if (flags & RTE_FASTMEM_F_ZERO) {
+ size_t cs = class_size(class_idx);
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ memset(ptrs[i], 0, cs);
+ }
+
+ return 0;
+}
+
+static void *
+do_alloc(size_t size, size_t align, unsigned int flags,
+ unsigned int lcore_id, int socket_id, bool fallback)
+{
+ unsigned int class_idx;
+ struct fastmem_cache **slot;
+ void *obj;
+
+ if (unlikely(fastmem_get() == NULL))
+ return NULL;
+
+ if (unlikely(!normalize_align(&align))) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ class_idx = size_to_class(size, align);
+ if (unlikely(class_idx >= FASTMEM_N_CLASSES)) {
+ rte_errno = E2BIG;
+ return NULL;
+ }
+
+ obj = alloc_from_socket(&fastmem->sockets[socket_id],
+ class_idx, lcore_id);
+
+ if (likely(obj != NULL))
+ goto out;
+
+ if (fallback) {
+ unsigned int i;
+
+ for (i = 0; i < rte_socket_count(); i++) {
+ int sid = rte_socket_id_by_idx(i);
+
+ if (sid < 0 || sid == socket_id)
+ continue;
+
+ obj = alloc_from_socket(&fastmem->sockets[sid],
+ class_idx, lcore_id);
+ if (obj != NULL)
+ goto out;
+ }
+ }
+
+ slot = cache_slot(
+ &fastmem->sockets[socket_id], class_idx, lcore_id);
+ if (slot != NULL && *slot != NULL)
+ (*slot)->alloc_nomem++;
+ rte_errno = ENOMEM;
+ return NULL;
+
+out:
+ if (flags & RTE_FASTMEM_F_ZERO)
+ memset(obj, 0, class_size(class_idx));
+
+ return obj;
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc, 24.11)
+rte_fastmem_alloc(size_t size, size_t align, unsigned int flags)
+{
+ return do_alloc(size, align, flags, rte_lcore_id(),
+ local_socket_id(), false);
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc_socket, 24.11)
+rte_fastmem_alloc_socket(size_t size, size_t align, unsigned int flags,
+ int socket_id)
+{
+ if (socket_id == SOCKET_ID_ANY)
+ return do_alloc(size, align, flags, rte_lcore_id(),
+ local_socket_id(), true);
+
+ if (unlikely(socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ return do_alloc(size, align, flags, rte_lcore_id(), socket_id, false);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_free, 24.11)
+rte_fastmem_free(void *ptr)
+{
+ if (unlikely(ptr == NULL))
+ return;
+
+ do_free(ptr);
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_realloc, 24.11)
+rte_fastmem_realloc(void *ptr, size_t size, size_t align)
+{
+ struct fastmem_slab *slab;
+ unsigned int old_class, new_class;
+ size_t old_size;
+ void *new_ptr;
+
+ if (ptr == NULL)
+ return rte_fastmem_alloc(size, align, 0);
+
+ if (size == 0) {
+ rte_fastmem_free(ptr);
+ return NULL;
+ }
+
+ if (unlikely(!normalize_align(&align))) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ new_class = size_to_class(size, align);
+ if (unlikely(new_class >= FASTMEM_N_CLASSES)) {
+ rte_errno = E2BIG;
+ return NULL;
+ }
+
+ slab = slab_of(ptr);
+ old_class = slab->bin->class_idx;
+
+ if (new_class == old_class)
+ return ptr;
+
+ new_ptr = rte_fastmem_alloc(size, align, 0);
+ if (unlikely(new_ptr == NULL))
+ return NULL;
+
+ old_size = class_size(old_class);
+ memcpy(new_ptr, ptr, RTE_MIN(old_size, size));
+ rte_fastmem_free(ptr);
+
+ return new_ptr;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc_bulk, 24.11)
+rte_fastmem_alloc_bulk(void **ptrs, unsigned int n, size_t size, size_t align,
+ unsigned int flags)
+{
+ return do_alloc_bulk(ptrs, n, size, align, flags,
+ rte_lcore_id(), local_socket_id(), false);
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc_bulk_socket, 24.11)
+rte_fastmem_alloc_bulk_socket(void **ptrs, unsigned int n, size_t size,
+ size_t align, unsigned int flags, int socket_id)
+{
+ if (socket_id == SOCKET_ID_ANY)
+ return do_alloc_bulk(ptrs, n, size, align, flags,
+ rte_lcore_id(), local_socket_id(), true);
+
+ if (unlikely(socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)) {
+ rte_errno = EINVAL;
+ return -EINVAL;
+ }
+
+ return do_alloc_bulk(ptrs, n, size, align, flags,
+ rte_lcore_id(), socket_id, false);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_free_bulk, 24.11)
+rte_fastmem_free_bulk(void **ptrs, unsigned int n)
+{
+ unsigned int lcore_id;
+ struct fastmem_slab *slab;
+ struct fastmem_bin *bin;
+ struct fastmem_socket_state *socket;
+ struct fastmem_cache *cache;
+ unsigned int space;
+ unsigned int i;
+
+ if (unlikely(n == 0))
+ return;
+
+ lcore_id = rte_lcore_id();
+
+ /* Fast path: check if first object gives us the bin. */
+ slab = slab_of(ptrs[0]);
+ bin = slab->bin;
+ socket = &fastmem->sockets[bin->socket_id];
+ cache = cache_get(socket, bin->class_idx, lcore_id);
+
+ if (unlikely(cache == NULL)) {
+ for (i = 0; i < n; i++)
+ do_free(ptrs[i]);
+ return;
+ }
+
+ /*
+ * Try to push all objects into the cache in one memcpy.
+ * If any object belongs to a different bin, fall back to
+ * per-object free for the remainder.
+ */
+ space = cache->capacity - cache->count;
+ if (likely(n <= space)) {
+ /* Verify all same bin (common case). */
+ for (i = 1; i < n; i++) {
+ if (slab_of(ptrs[i])->bin != bin)
+ goto slow;
+ }
+ cache->free_cache_hits += n;
+ memcpy(&cache->objs[cache->count], ptrs,
+ n * sizeof(void *));
+ cache->count += n;
+ return;
+ }
+
+ /* Would overflow cache — drain first, then push. */
+ if (n <= cache->capacity) {
+ unsigned int drain;
+
+ for (i = 1; i < n; i++) {
+ if (slab_of(ptrs[i])->bin != bin)
+ goto slow;
+ }
+
+ cache->free_cache_misses += n;
+ drain = cache->count - cache->target + n;
+ if (drain > cache->count)
+ drain = cache->count;
+ if (drain > 0) {
+ bin_free_bulk(bin, cache->objs, drain);
+ cache->count -= drain;
+ memmove(cache->objs, cache->objs + drain,
+ cache->count * sizeof(cache->objs[0]));
+ }
+ memcpy(&cache->objs[cache->count], ptrs,
+ n * sizeof(void *));
+ cache->count += n;
+ return;
+ }
+
+slow:
+ for (i = 0; i < n; i++)
+ do_free(ptrs[i]);
+}
+
+#define fastmem_handle_class_BITS 8
+
+static rte_fastmem_handle_t
+fastmem_handle_pack(unsigned int class_idx, int socket_id)
+{
+ return (uint32_t)class_idx |
+ ((uint32_t)socket_id << fastmem_handle_class_BITS);
+}
+
+static unsigned int
+fastmem_handle_class(rte_fastmem_handle_t h)
+{
+ return h & ((1U << fastmem_handle_class_BITS) - 1);
+}
+
+static int
+fastmem_handle_socket(rte_fastmem_handle_t h)
+{
+ return (int)(h >> fastmem_handle_class_BITS);
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_hlookup, 24.11)
+rte_fastmem_hlookup(size_t size, size_t align, int socket_id,
+ rte_fastmem_handle_t *handle)
+{
+ unsigned int class_idx;
+ struct fastmem_socket_state *socket;
+
+ if (handle == NULL)
+ return -EINVAL;
+
+ if (!normalize_align(&align))
+ return -EINVAL;
+
+ if (socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return -EINVAL;
+
+ class_idx = size_to_class(size, align);
+ if (class_idx >= FASTMEM_N_CLASSES)
+ return -E2BIG;
+
+ /* Pre-create the cache for the calling lcore. */
+ socket = &fastmem->sockets[socket_id];
+ cache_create(socket, class_idx, rte_lcore_id());
+
+ *handle = fastmem_handle_pack(class_idx, socket_id);
+ return 0;
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_halloc, 24.11)
+rte_fastmem_halloc(rte_fastmem_handle_t handle, unsigned int flags)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+ unsigned int lcore_id = rte_lcore_id();
+ struct fastmem_socket_state *socket = &fastmem->sockets[socket_id];
+ struct fastmem_bin *bin = &socket->bins[class_idx];
+ struct fastmem_cache *cache;
+ void *obj;
+
+ RTE_ASSERT(fastmem != NULL);
+ RTE_ASSERT(lcore_id < RTE_MAX_LCORE);
+
+ cache = socket->caches[lcore_id][class_idx];
+ RTE_ASSERT(cache != NULL);
+
+ obj = cache_pop(cache, bin);
+ if (unlikely(obj == NULL)) {
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ if (flags & RTE_FASTMEM_F_ZERO)
+ memset(obj, 0, class_size(class_idx));
+
+ return obj;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_halloc_bulk, 24.11)
+rte_fastmem_halloc_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n, unsigned int flags)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+
+ return do_alloc_bulk(ptrs, n, class_size(class_idx),
+ RTE_CACHE_LINE_SIZE, flags, rte_lcore_id(),
+ socket_id, false);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_hfree, 24.11)
+rte_fastmem_hfree(rte_fastmem_handle_t handle, void *ptr)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+ struct fastmem_socket_state *socket = &fastmem->sockets[socket_id];
+ struct fastmem_bin *bin = &socket->bins[class_idx];
+ unsigned int lcore_id = rte_lcore_id();
+ struct fastmem_cache *cache;
+
+ if (unlikely(ptr == NULL))
+ return;
+
+ RTE_ASSERT(lcore_id < RTE_MAX_LCORE);
+
+ cache = socket->caches[lcore_id][class_idx];
+ RTE_ASSERT(cache != NULL);
+
+ cache_push(cache, bin, ptr);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_hfree_bulk, 24.11)
+rte_fastmem_hfree_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+ struct fastmem_socket_state *socket = &fastmem->sockets[socket_id];
+ struct fastmem_bin *bin = &socket->bins[class_idx];
+ unsigned int lcore_id;
+ struct fastmem_cache *cache;
+ unsigned int i;
+
+ if (unlikely(n == 0))
+ return;
+
+ lcore_id = rte_lcore_id();
+ cache = cache_get(socket, class_idx, lcore_id);
+
+ if (likely(cache != NULL)) {
+ for (i = 0; i < n; i++)
+ cache_push(cache, bin, ptrs[i]);
+ } else {
+ for (i = 0; i < n; i++)
+ bin_free_one(bin, ptrs[i]);
+ }
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_virt2iova, 24.11)
+rte_iova_t
+rte_fastmem_virt2iova(const void *ptr)
+{
+ struct fastmem_slab *slab;
+
+ RTE_ASSERT(fastmem != NULL);
+
+ slab = slab_of((void *)(uintptr_t)ptr);
+
+ return slab->iova_base + ((uintptr_t)ptr - (uintptr_t)slab);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_cache_flush, 24.11)
+rte_fastmem_cache_flush(void)
+{
+ unsigned int lcore_id;
+ unsigned int s, c;
+
+ if (fastmem == NULL)
+ return;
+
+ lcore_id = rte_lcore_id();
+ if (lcore_id >= RTE_MAX_LCORE)
+ return;
+
+ for (s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ for (c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_cache *cache =
+ socket->caches[lcore_id][c];
+ struct fastmem_slab *cache_slab;
+
+ if (cache == NULL)
+ continue;
+
+ if (cache->count > 0) {
+ bin_free_bulk(&socket->bins[c],
+ cache->objs, cache->count);
+ cache->count = 0;
+ }
+
+ cache_slab = slab_of(cache);
+ bin_free_one(cache_slab->bin, cache);
+
+ socket->caches[lcore_id][c] = NULL;
+ }
+ }
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats, 24.11)
+rte_fastmem_stats(struct rte_fastmem_stats *stats)
+{
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_stats){0};
+ stats->n_classes = FASTMEM_N_CLASSES;
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ stats->bytes_backing += socket->reserved_bytes;
+
+ for (unsigned int c = 0; c < FASTMEM_N_CLASSES; c++) {
+ uint64_t class_allocs = 0, class_frees = 0;
+
+ for (unsigned int l = 0; l < RTE_MAX_LCORE; l++) {
+ struct fastmem_cache *cache =
+ socket->caches[l][c];
+ if (cache == NULL)
+ continue;
+ class_allocs += cache->alloc_cache_hits +
+ cache->alloc_cache_misses;
+ class_frees += cache->free_cache_hits +
+ cache->free_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ }
+ stats->alloc_total += class_allocs;
+ stats->free_total += class_frees;
+ if (class_allocs > class_frees)
+ stats->bytes_in_use += class_size(c) *
+ (class_allocs - class_frees);
+ }
+ }
+
+ return 0;
+}
+
+static unsigned int
+exact_class_idx(size_t sz)
+{
+ unsigned int log2;
+
+ if (sz < FASTMEM_MIN_SIZE || sz > FASTMEM_MAX_ALLOC_SIZE)
+ return FASTMEM_N_CLASSES;
+ if ((sz & (sz - 1)) != 0)
+ return FASTMEM_N_CLASSES;
+
+ log2 = (unsigned int)rte_ctz64(sz);
+ if (log2 < FASTMEM_MIN_CLASS_LOG2 || log2 > FASTMEM_MAX_CLASS_LOG2)
+ return FASTMEM_N_CLASSES;
+
+ return log2 - FASTMEM_MIN_CLASS_LOG2;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_class, 24.11)
+rte_fastmem_stats_class(size_t class_size_arg,
+ struct rte_fastmem_class_stats *stats)
+{
+ unsigned int c;
+ uint64_t allocs, frees;
+
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+
+ c = exact_class_idx(class_size_arg);
+ if (c >= FASTMEM_N_CLASSES)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_class_stats){0};
+ stats->class_size = class_size(c);
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+ struct fastmem_bin *bin = &socket->bins[c];
+
+ for (unsigned int l = 0; l < RTE_MAX_LCORE; l++) {
+ struct fastmem_cache *cache = socket->caches[l][c];
+ if (cache == NULL)
+ continue;
+ stats->alloc_cache_hits += cache->alloc_cache_hits;
+ stats->alloc_cache_misses += cache->alloc_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ stats->free_cache_hits += cache->free_cache_hits;
+ stats->free_cache_misses += cache->free_cache_misses;
+ }
+
+ stats->slab_acquires += bin->slab_acquires;
+ stats->slab_releases += bin->slab_releases;
+ stats->slabs_partial += bin->slabs_partial;
+ stats->slabs_full += bin->slabs_full;
+ }
+
+ allocs = stats->alloc_cache_hits + stats->alloc_cache_misses;
+ frees = stats->free_cache_hits + stats->free_cache_misses;
+ if (allocs > frees)
+ stats->in_use = allocs - frees;
+
+ return 0;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_lcore, 24.11)
+rte_fastmem_stats_lcore(unsigned int lcore_id,
+ struct rte_fastmem_lcore_stats *stats)
+{
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+ if (lcore_id >= RTE_MAX_LCORE)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_lcore_stats){0};
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ for (unsigned int c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_cache *cache =
+ socket->caches[lcore_id][c];
+ if (cache == NULL)
+ continue;
+ stats->alloc_cache_hits += cache->alloc_cache_hits;
+ stats->alloc_cache_misses += cache->alloc_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ stats->free_cache_hits += cache->free_cache_hits;
+ stats->free_cache_misses += cache->free_cache_misses;
+ }
+ }
+
+ return 0;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_lcore_class, 24.11)
+rte_fastmem_stats_lcore_class(unsigned int lcore_id, size_t class_size_arg,
+ struct rte_fastmem_lcore_class_stats *stats)
+{
+ unsigned int c;
+
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+ if (lcore_id >= RTE_MAX_LCORE)
+ return -EINVAL;
+
+ c = exact_class_idx(class_size_arg);
+ if (c >= FASTMEM_N_CLASSES)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_lcore_class_stats){0};
+ stats->class_size = class_size(c);
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_cache *cache =
+ fastmem->sockets[s].caches[lcore_id][c];
+ if (cache == NULL)
+ continue;
+ stats->alloc_cache_hits += cache->alloc_cache_hits;
+ stats->alloc_cache_misses += cache->alloc_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ stats->free_cache_hits += cache->free_cache_hits;
+ stats->free_cache_misses += cache->free_cache_misses;
+ }
+
+ return 0;
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_reset, 24.11)
+rte_fastmem_stats_reset(void)
+{
+ if (fastmem == NULL)
+ return;
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ for (unsigned int c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_bin *bin = &socket->bins[c];
+
+ bin->slab_acquires = 0;
+ bin->slab_releases = 0;
+
+ for (unsigned int l = 0; l < RTE_MAX_LCORE; l++) {
+ struct fastmem_cache *cache =
+ socket->caches[l][c];
+ if (cache == NULL)
+ continue;
+ cache->alloc_cache_hits = 0;
+ cache->alloc_cache_misses = 0;
+ cache->alloc_nomem = 0;
+ cache->free_cache_hits = 0;
+ cache->free_cache_misses = 0;
+ }
+ }
+ }
+}
+
+unsigned int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_classes, 24.11)
+rte_fastmem_classes(size_t *sizes)
+{
+ if (sizes != NULL)
+ for (unsigned int i = 0; i < FASTMEM_N_CLASSES; i++)
+ sizes[i] = class_size(i);
+ return FASTMEM_N_CLASSES;
+}
diff --git a/lib/fastmem/rte_fastmem.h b/lib/fastmem/rte_fastmem.h
new file mode 100644
index 0000000000..1d74660da1
--- /dev/null
+++ b/lib/fastmem/rte_fastmem.h
@@ -0,0 +1,815 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#ifndef _RTE_FASTMEM_H_
+#define _RTE_FASTMEM_H_
+
+/**
+ * @file
+ *
+ * RTE Fastmem
+ *
+ * @warning
+ * @b EXPERIMENTAL:
+ * All functions in this file may be changed or removed without prior notice.
+ *
+ * The fastmem library is a fast, general-purpose small-object
+ * allocator for DPDK applications. It is intended to allow an
+ * application to replace its many per-type mempools — each sized
+ * for a single object type (a connection, a session, a work item,
+ * a timer, etc.) — with a single allocator that handles arbitrary
+ * object sizes, grows on demand, and offers mempool-level
+ * performance for the common allocation and free paths.
+ *
+ * Like mempool, fastmem is backed by huge pages, is NUMA-aware,
+ * supports bulk operations, and uses per-lcore caches to reduce
+ * shared-state contention. Unlike mempool, it does not require the
+ * caller to declare object sizes or counts up front.
+ *
+ * There is a single, global fastmem instance per process. The
+ * instance is brought up with rte_fastmem_init() and torn down with
+ * rte_fastmem_deinit(). Allocations are made with
+ * rte_fastmem_alloc() and freed with rte_fastmem_free().
+ *
+ * The allocator is bounded to small-object allocations. Requests
+ * larger than rte_fastmem_max_size() are rejected; callers with
+ * such needs should use rte_malloc() directly.
+ *
+ * Backing memory is reserved from DPDK memzones. Once reserved,
+ * backing memory is not returned to the system during the
+ * allocator's lifetime. Callers that need predictable latency may
+ * pre-reserve backing memory up front using rte_fastmem_reserve(),
+ * avoiding memzone-reservation overhead during steady-state
+ * operation.
+ *
+ * Alignment argument, @c align:
+ * If non-zero, @c align specifies an exact minimum alignment and
+ * must be a power of 2. If zero, the default alignment is
+ * @c RTE_CACHE_LINE_SIZE, so that objects obtained from distinct
+ * calls cannot false-share a cache line.
+ *
+ * Threads and per-lcore caches:
+ * Allocate and free calls from EAL threads are served through a
+ * per-lcore cache, which makes the common path lock-free.
+ * Unregistered non-EAL threads do not use a cache; their
+ * allocate and free calls go directly to shared state, take an
+ * internal lock, and cost more per call.
+ *
+ * Non-preemptible caller:
+ * Callers should not be preemptible while inside a fastmem call.
+ * Fastmem uses internal spinlocks; if a caller is preempted
+ * while holding one, any other thread that subsequently needs
+ * the same lock stalls until the preempted caller resumes.
+ */
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <rte_bitops.h>
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Flag for rte_fastmem_alloc() and its variants: initialize the
+ * returned memory to zero before returning it to the caller.
+ */
+#define RTE_FASTMEM_F_ZERO RTE_BIT32(0)
+
+/**
+ * Initialize the fastmem allocator.
+ *
+ * Sets up the library's internal state. Must be called before any
+ * allocation call. Typically called once per process, after
+ * rte_eal_init() and before the application's worker threads begin
+ * making allocations.
+ *
+ * Initialization does not pre-reserve any backing memory; memzones
+ * are reserved lazily as allocations require. An application that
+ * wants to avoid memzone-reservation latency on the allocation
+ * path should follow rte_fastmem_init() with one or more calls to
+ * rte_fastmem_reserve().
+ *
+ * This function is not thread-safe and must not be called
+ * concurrently with any other fastmem function.
+ *
+ * @return
+ * - 0: Success.
+ * - -EBUSY: The allocator is already initialized.
+ * - -ENOMEM: Unable to allocate internal state.
+ */
+__rte_experimental
+int
+rte_fastmem_init(void);
+
+/**
+ * Tear down the fastmem allocator.
+ *
+ * Releases the library's internal state and frees all backing
+ * memzones. After this call, no fastmem allocations or frees may
+ * be made until rte_fastmem_init() is called again.
+ *
+ * The caller is responsible for ensuring that no fastmem-allocated
+ * objects remain in use. Outstanding allocations at deinit time
+ * result in undefined behavior.
+ *
+ * This function is not thread-safe and must not be called
+ * concurrently with any other fastmem function.
+ */
+__rte_experimental
+void
+rte_fastmem_deinit(void);
+
+/**
+ * Pre-reserve backing memory.
+ *
+ * Ensures that at least @p size bytes of memzone-backed memory are
+ * available to the allocator on @p socket_id, reserving additional
+ * memzones from EAL as needed to reach that total. Subsequent
+ * allocations served from the pre-reserved memory do not incur
+ * memzone-reservation cost.
+ *
+ * The reservation is cumulative: repeated calls to
+ * rte_fastmem_reserve() with the same @p socket_id grow the
+ * reservation monotonically. Reserved memory is never returned to
+ * the system during the allocator's lifetime.
+ *
+ * A typical use is to call rte_fastmem_reserve() once at
+ * application startup, with a size chosen to cover the expected
+ * steady-state working set. Allocations and frees during
+ * steady-state operation then avoid memzone reservations entirely.
+ *
+ * @param size
+ * The minimum amount of backing memory, in bytes, to make
+ * available on @p socket_id. The allocator may reserve more than
+ * the requested amount due to internal rounding (e.g., to memzone
+ * or block granularity).
+ *
+ * @param socket_id
+ * The NUMA socket on which to reserve memory, or SOCKET_ID_ANY
+ * to leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the reservation.
+ *
+ * @return
+ * - 0: Success.
+ * - -ENOMEM: Insufficient huge-page memory to satisfy the request.
+ * - -EINVAL: Invalid @p socket_id.
+ */
+__rte_experimental
+int
+rte_fastmem_reserve(size_t size, int socket_id);
+
+/**
+ * Set the maximum backing memory that may be reserved on a socket.
+ *
+ * Once the limit is reached, allocations that would require new
+ * backing memory on the constrained socket fail with ENOMEM.
+ * Already-reserved memory is not released.
+ *
+ * Setting a limit below the current reserved amount is allowed and
+ * prevents further growth.
+ *
+ * @param socket_id
+ * The NUMA socket to constrain, or SOCKET_ID_ANY to apply the
+ * limit to all sockets.
+ * @param max_bytes
+ * Maximum backing memory in bytes, or SIZE_MAX for unlimited (the default).
+ * @return
+ * - 0: Success.
+ * - -EINVAL: Fastmem not initialized, or invalid @p socket_id.
+ */
+__rte_experimental
+int
+rte_fastmem_set_limit(int socket_id, size_t max_bytes);
+
+/**
+ * Get the maximum backing memory limit for a socket.
+ *
+ * @param socket_id
+ * The NUMA socket to query.
+ * @return
+ * The limit in bytes, or SIZE_MAX if unlimited.
+ */
+__rte_experimental
+size_t
+rte_fastmem_get_limit(int socket_id);
+
+/**
+ * Retrieve the largest allocation size the allocator supports.
+ *
+ * Requests larger than this size are rejected by the allocation
+ * functions. The returned value is a property of the allocator
+ * implementation and does not change across the lifetime of the
+ * process.
+ *
+ * @return
+ * The largest supported allocation size, in bytes.
+ */
+__rte_experimental
+size_t
+rte_fastmem_max_size(void);
+
+/* Forward declaration for __rte_dealloc attribute. */
+void rte_fastmem_free(void *ptr);
+
+/**
+ * Allocate an object from the fastmem allocator.
+ *
+ * Allocates at least @p size bytes, aligned to at least @p align
+ * bytes. The returned memory is backed by huge pages and is
+ * DMA-usable; its IOVA can be obtained via rte_fastmem_virt2iova().
+ *
+ * On NUMA systems, the memory is allocated on the socket of the
+ * calling lcore. Use rte_fastmem_alloc_socket() to target a
+ * specific socket.
+ *
+ * The allocated memory must be freed with rte_fastmem_free(). An
+ * allocation may be freed from any lcore, not only the lcore that
+ * made the allocation.
+ *
+ * This function is MT-safe.
+ *
+ * @param size
+ * Requested allocation size, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, the returned pointer will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, the returned pointer will
+ * be aligned on a multiple of @p align, which must be a power of
+ * 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags. Use
+ * RTE_FASTMEM_F_ZERO to obtain zero-initialized memory.
+ *
+ * @return
+ * - A pointer to the allocated object on success.
+ * - NULL on failure, with @c rte_errno set:
+ * - E2BIG: @p size exceeds rte_fastmem_max_size().
+ * - EINVAL: Invalid @p align (not a power of two).
+ * - ENOMEM: Allocation could not be served from existing
+ * backing memory and no additional memzone could be reserved.
+ */
+__rte_experimental
+void *
+rte_fastmem_alloc(size_t size, size_t align, unsigned int flags)
+ __rte_malloc __rte_dealloc(rte_fastmem_free, 1);
+
+/**
+ * Allocate an object on a specific NUMA socket.
+ *
+ * Like rte_fastmem_alloc(), but targets the specified NUMA socket
+ * rather than the socket of the calling lcore. Use this variant
+ * when the lifetime or access pattern of the allocation is not
+ * tied to the calling lcore's socket.
+ *
+ * This function is MT-safe.
+ *
+ * @param size
+ * Requested allocation size, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, the returned pointer will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, the returned pointer will
+ * be aligned on a multiple of @p align, which must be a power of
+ * 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags.
+ *
+ * @param socket_id
+ * The NUMA socket on which to allocate, or SOCKET_ID_ANY to
+ * leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the request.
+ *
+ * @return
+ * - A pointer to the allocated object on success.
+ * - NULL on failure, with @c rte_errno set (see rte_fastmem_alloc()).
+ */
+__rte_experimental
+void *
+rte_fastmem_alloc_socket(size_t size, size_t align, unsigned int flags,
+ int socket_id)
+ __rte_malloc __rte_dealloc(rte_fastmem_free, 1);
+
+/**
+ * Resize a fastmem allocation, preserving existing contents.
+ *
+ * If @p ptr is NULL, equivalent to rte_fastmem_alloc(size, align, 0).
+ * If @p size is 0, frees @p ptr and returns NULL.
+ *
+ * If the existing allocation can already satisfy the new size and
+ * alignment, the original pointer may be returned unchanged.
+ * Otherwise, a new allocation is made, the contents are copied
+ * (up to the minimum of old and new sizes), and the old allocation
+ * is freed.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptr
+ * Pointer to an existing fastmem allocation, or NULL.
+ *
+ * @param size
+ * New requested size in bytes. If 0, the allocation is freed.
+ *
+ * @param align
+ * If 0, alignment is at least @c RTE_CACHE_LINE_SIZE. Otherwise,
+ * must be a power of 2.
+ *
+ * @return
+ * - A pointer to the resized allocation on success.
+ * - NULL on failure, with @c rte_errno set:
+ * - E2BIG: @p size exceeds rte_fastmem_max_size().
+ * - EINVAL: Invalid @p align.
+ * - ENOMEM: Allocation could not be served.
+ * On failure, the original allocation at @p ptr remains valid.
+ */
+__rte_experimental
+void *
+rte_fastmem_realloc(void *ptr, size_t size, size_t align)
+ __rte_dealloc(rte_fastmem_free, 1);
+
+/**
+ * Free an object previously allocated by the fastmem allocator.
+ *
+ * @p ptr must have been returned by a prior call to any fastmem
+ * allocation function, or be NULL. If @p ptr is NULL, no operation
+ * is performed.
+ *
+ * Free may be called from any lcore, regardless of which lcore
+ * made the original allocation.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptr
+ * Pointer to an object previously allocated by fastmem, or NULL.
+ */
+__rte_experimental
+void
+rte_fastmem_free(void *ptr);
+
+/**
+ * Allocate multiple objects in bulk.
+ *
+ * Allocates @p n objects, each of size at least @p size and aligned
+ * to at least @p align bytes, and stores the resulting pointers
+ * into @p ptrs. All @p n objects have the same size and alignment.
+ *
+ * On NUMA systems, the memory is allocated on the socket of the
+ * calling lcore. Use rte_fastmem_alloc_bulk_socket() to target a
+ * specific socket.
+ *
+ * The bulk path amortizes per-object overhead and is typically
+ * faster than @p n individual calls to rte_fastmem_alloc().
+ *
+ * On failure no objects are allocated and @p ptrs is left
+ * untouched.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptrs
+ * An array of at least @p n pointers into which the newly
+ * allocated object pointers are written.
+ *
+ * @param n
+ * The number of objects to allocate.
+ *
+ * @param size
+ * Requested size of each object, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, returned pointers will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, returned pointers will be
+ * aligned on a multiple of @p align, which must be a power of 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags.
+ *
+ * @return
+ * - 0: All @p n objects were allocated and stored in @p ptrs.
+ * - -E2BIG: @p size exceeds rte_fastmem_max_size().
+ * - -EINVAL: Invalid @p align.
+ * - -ENOMEM: Not enough objects could be allocated to fill the
+ * request.
+ */
+__rte_experimental
+int
+rte_fastmem_alloc_bulk(void **ptrs, unsigned int n, size_t size, size_t align,
+ unsigned int flags);
+
+/**
+ * Allocate multiple objects in bulk on a specific NUMA socket.
+ *
+ * Like rte_fastmem_alloc_bulk(), but targets the specified NUMA
+ * socket rather than the socket of the calling lcore.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptrs
+ * An array of at least @p n pointers into which the newly
+ * allocated object pointers are written.
+ *
+ * @param n
+ * The number of objects to allocate.
+ *
+ * @param size
+ * Requested size of each object, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, returned pointers will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, returned pointers will be
+ * aligned on a multiple of @p align, which must be a power of 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags.
+ *
+ * @param socket_id
+ * The NUMA socket on which to allocate, or SOCKET_ID_ANY to
+ * leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the request.
+ *
+ * @return
+ * - 0: All @p n objects were allocated and stored in @p ptrs.
+ * - Negative errno on failure (see rte_fastmem_alloc_bulk()).
+ */
+__rte_experimental
+int
+rte_fastmem_alloc_bulk_socket(void **ptrs, unsigned int n, size_t size,
+ size_t align, unsigned int flags, int socket_id);
+
+/**
+ * Free multiple objects in bulk.
+ *
+ * Frees the @p n objects pointed to by @p ptrs. Each pointer in
+ * the array must have been returned by a prior fastmem allocation
+ * call and must not have been freed. The objects need not have
+ * the same size, alignment, or socket.
+ *
+ * The bulk path amortizes per-object overhead and is typically
+ * faster than @p n individual calls to rte_fastmem_free().
+ *
+ * This function is MT-safe.
+ *
+ * @param ptrs
+ * An array of @p n pointers to fastmem-allocated objects.
+ *
+ * @param n
+ * The number of objects to free.
+ */
+__rte_experimental
+void
+rte_fastmem_free_bulk(void **ptrs, unsigned int n);
+
+/**
+ * Opaque handle encoding a (size class, NUMA socket) pair.
+ *
+ * Obtained via rte_fastmem_hlookup(). Passing a handle to
+ * rte_fastmem_halloc() avoids the per-call size-class
+ * lookup and socket resolution, improving allocation throughput
+ * for fixed-size objects.
+ */
+typedef uint32_t rte_fastmem_handle_t;
+
+/**
+ * Look up a handle for a given object size and NUMA socket.
+ *
+ * The returned handle encodes the size class and socket, and can
+ * be passed to rte_fastmem_halloc() to allocate objects
+ * without repeating the class lookup.
+ *
+ * @param size
+ * Object size in bytes. Must not exceed rte_fastmem_max_size().
+ *
+ * @param align
+ * Alignment requirement (power of two), or 0 for the default
+ * (RTE_CACHE_LINE_SIZE).
+ *
+ * @param socket_id
+ * NUMA socket to allocate from.
+ *
+ * @param[out] handle
+ * On success, set to the resolved handle.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: Invalid alignment or socket_id.
+ * - -E2BIG: @p size exceeds rte_fastmem_max_size().
+ */
+__rte_experimental
+int
+rte_fastmem_hlookup(size_t size, size_t align, int socket_id,
+ rte_fastmem_handle_t *handle);
+
+/**
+ * Allocate an object using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_alloc() but skips the size-class
+ * lookup and socket resolution, using the pre-resolved handle
+ * instead.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param flags
+ * Allocation flags (e.g., RTE_FASTMEM_F_ZERO).
+ *
+ * @return
+ * A pointer to the allocated object, or NULL on failure
+ * (rte_errno is set).
+ */
+__rte_experimental
+void *
+rte_fastmem_halloc(rte_fastmem_handle_t handle, unsigned int flags)
+ __rte_malloc __rte_dealloc(rte_fastmem_free, 1);
+
+/**
+ * Bulk-allocate objects using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_alloc_bulk() but uses a pre-resolved
+ * handle. All-or-nothing semantics apply.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param[out] ptrs
+ * Array to receive @p n allocated pointers.
+ *
+ * @param n
+ * Number of objects to allocate.
+ *
+ * @param flags
+ * Allocation flags (e.g., RTE_FASTMEM_F_ZERO).
+ *
+ * @return
+ * - 0: All @p n objects allocated successfully.
+ * - -ENOMEM: Allocation failed; no objects were allocated.
+ */
+__rte_experimental
+int
+rte_fastmem_halloc_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n, unsigned int flags);
+
+/**
+ * Free an object using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_free() but skips the slab-header
+ * lookup by using the class and socket encoded in the handle.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param ptr
+ * A pointer previously returned by a fastmem allocation function.
+ * Must belong to the same size class and socket as @p handle.
+ * NULL is permitted (no-op).
+ */
+__rte_experimental
+void
+rte_fastmem_hfree(rte_fastmem_handle_t handle, void *ptr);
+
+/**
+ * Bulk-free objects using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_free_bulk() but skips per-object
+ * slab-header lookups.
+ *
+ * All objects must belong to the same size class and socket as
+ * @p handle.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param ptrs
+ * An array of @p n pointers to fastmem-allocated objects.
+ *
+ * @param n
+ * The number of objects to free.
+ */
+__rte_experimental
+void
+rte_fastmem_hfree_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n);
+
+/**
+ * Obtain the IOVA for a fastmem-allocated pointer.
+ *
+ * Translates a virtual address returned by a fastmem allocation
+ * function into the corresponding IOVA, suitable for use in device
+ * DMA descriptors.
+ *
+ * The returned IOVA is valid for the lifetime of the allocation.
+ *
+ * @p ptr must have been returned by a prior fastmem allocation
+ * function. Passing any other pointer results in undefined
+ * behavior.
+ *
+ * @param ptr
+ * A pointer previously returned by a fastmem allocation
+ * function.
+ *
+ * @return
+ * The IOVA corresponding to @p ptr.
+ */
+__rte_experimental
+rte_iova_t
+rte_fastmem_virt2iova(const void *ptr);
+
+/**
+ * Flush the calling lcore's per-lcore caches.
+ *
+ * Drains every cached object from the calling lcore's
+ * per-(size class, NUMA socket) caches back to their shared
+ * bins, and releases the cache state itself. A subsequent
+ * allocation or free on this lcore lazily recreates any caches
+ * it needs.
+ *
+ * This is useful in applications that have finished a bursty
+ * phase and want to release memory that would otherwise sit idle
+ * in caches. It is also useful in tests that want to observe
+ * bin-level state without per-lcore caching hiding activity.
+ *
+ * The call has no effect when invoked from a non-EAL thread.
+ *
+ * This function is not thread-safe with respect to concurrent
+ * allocations or frees on the calling lcore; call it only when
+ * the calling lcore is not making other fastmem calls.
+ */
+__rte_experimental
+void
+rte_fastmem_cache_flush(void);
+
+/**
+ * Global summary statistics.
+ */
+struct rte_fastmem_stats {
+ uint64_t bytes_backing; /**< Bytes of backing memory (memzones) reserved from EAL. */
+ uint64_t bytes_in_use; /**< Approximate bytes in live objects. */
+ uint64_t alloc_total; /**< Total successful alloc operations (hits + misses). */
+ uint64_t free_total; /**< Total free operations (hits + misses). */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ unsigned int n_classes; /**< Number of size classes. */
+};
+
+/**
+ * Per-size-class statistics (aggregated across all lcores).
+ *
+ * Allocation and free counters count individual objects, not
+ * operations. A bulk allocation of 32 objects that hits the cache
+ * increments alloc_cache_hits by 32.
+ */
+struct rte_fastmem_class_stats {
+ size_t class_size; /**< Usable size of this class (bytes). */
+ uint64_t in_use; /**< Objects currently live (allocs - frees). */
+ uint64_t alloc_cache_hits; /**< Allocs served from a per-lcore cache. */
+ uint64_t alloc_cache_misses; /**< Allocs that triggered a bin refill. */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ uint64_t free_cache_hits; /**< Frees absorbed by a per-lcore cache. */
+ uint64_t free_cache_misses; /**< Frees that triggered a bin drain. */
+ uint64_t slab_acquires; /**< Slabs pulled from the free pool. */
+ uint64_t slab_releases; /**< Slabs returned to the free pool. */
+ uint32_t slabs_partial; /**< Current partial slab count. */
+ uint32_t slabs_full; /**< Current full slab count. */
+};
+
+/**
+ * Per-lcore statistics (aggregated across all classes).
+ */
+struct rte_fastmem_lcore_stats {
+ uint64_t alloc_cache_hits; /**< Allocs served from this lcore's caches. */
+ uint64_t alloc_cache_misses; /**< Allocs that missed this lcore's caches. */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ uint64_t free_cache_hits; /**< Frees absorbed by this lcore's caches. */
+ uint64_t free_cache_misses; /**< Frees that bypassed this lcore's caches. */
+};
+
+/**
+ * Per-lcore, per-class statistics (no aggregation).
+ */
+struct rte_fastmem_lcore_class_stats {
+ size_t class_size; /**< Usable size of this class (bytes). */
+ uint64_t alloc_cache_hits; /**< Allocs served from cache. */
+ uint64_t alloc_cache_misses; /**< Allocs that triggered a bin refill. */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ uint64_t free_cache_hits; /**< Frees absorbed by cache. */
+ uint64_t free_cache_misses; /**< Frees that triggered a bin drain. */
+};
+
+/**
+ * Get the number of size classes and optionally their sizes.
+ *
+ * @param[out] sizes
+ * If non-NULL, filled with the size (in bytes) of each class.
+ * The caller must provide space for at least the returned number
+ * of entries.
+ *
+ * @return
+ * The number of size classes.
+ */
+__rte_experimental
+unsigned int
+rte_fastmem_classes(size_t *sizes);
+
+/**
+ * Retrieve global summary statistics.
+ *
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL or fastmem is not initialized.
+ */
+__rte_experimental
+int
+rte_fastmem_stats(struct rte_fastmem_stats *stats);
+
+/**
+ * Retrieve statistics for a single size class.
+ *
+ * @param class_size
+ * Exact size of the class to query (must match one of the values
+ * returned by rte_fastmem_classes()).
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL, fastmem is not initialized, or
+ * @p class_size does not match any size class.
+ */
+__rte_experimental
+int
+rte_fastmem_stats_class(size_t class_size,
+ struct rte_fastmem_class_stats *stats);
+
+/**
+ * Retrieve per-lcore statistics (aggregated across all classes).
+ *
+ * @param lcore_id
+ * The lcore to query.
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL, fastmem is not initialized, or
+ * @p lcore_id is invalid.
+ */
+__rte_experimental
+int
+rte_fastmem_stats_lcore(unsigned int lcore_id,
+ struct rte_fastmem_lcore_stats *stats);
+
+/**
+ * Retrieve per-lcore, per-class statistics.
+ *
+ * @param lcore_id
+ * The lcore to query.
+ * @param class_size
+ * Exact size of the class to query.
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL, fastmem is not initialized,
+ * @p lcore_id is invalid, or @p class_size does not match any
+ * size class.
+ */
+__rte_experimental
+int
+rte_fastmem_stats_lcore_class(unsigned int lcore_id, size_t class_size,
+ struct rte_fastmem_lcore_class_stats *stats);
+
+/**
+ * Reset all statistics counters to zero.
+ *
+ * Zeroes per-lcore cache counters and per-bin counters. Does not
+ * affect the allocator's operational state.
+ */
+__rte_experimental
+void
+rte_fastmem_stats_reset(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_FASTMEM_H_ */
diff --git a/lib/meson.build b/lib/meson.build
index 8f5cfd28a5..10906d4d53 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -38,6 +38,7 @@ libraries = [
'distributor',
'dmadev', # eventdev depends on this
'efd',
+ 'fastmem',
'eventdev',
'dispatcher', # dispatcher depends on eventdev
'gpudev',
--
2.43.0
^ permalink raw reply related
* [RFC v3 1/3] doc: add fastmem programming guide
From: Mattias Rönnblom @ 2026-05-27 17:30 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Bruce Richardson,
Mattias Rönnblom
In-Reply-To: <20260527173042.93867-1-hofors@lysator.liu.se>
Add a programming guide for the fastmem library covering usage,
API overview, design, and implementation details.
--
RFC v3:
* Add realloc subsection to Allocation and free section.
Signed-off-by: Mattias Rönnblom <hofors@lysator.liu.se>
---
doc/guides/prog_guide/fastmem_lib.rst | 328 ++++++++++++++++++++++++++
doc/guides/prog_guide/index.rst | 1 +
2 files changed, 329 insertions(+)
create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
diff --git a/doc/guides/prog_guide/fastmem_lib.rst b/doc/guides/prog_guide/fastmem_lib.rst
new file mode 100644
index 0000000000..cbc3bcf191
--- /dev/null
+++ b/doc/guides/prog_guide/fastmem_lib.rst
@@ -0,0 +1,328 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2026 Ericsson AB
+
+Fastmem Library
+===============
+
+The fastmem library is a fast, general-purpose small-object
+allocator for DPDK applications. It lets an application replace
+its many per-type mempools — each sized for a single object type
+— with a single allocator that handles arbitrary object sizes,
+grows on demand, and offers mempool-level performance for the
+common allocation and free paths.
+
+Like mempool, fastmem is backed by huge pages, is NUMA-aware,
+supports bulk operations, and uses per-lcore caches to reduce
+shared-state contention. Unlike mempool, it does not require the
+caller to declare object sizes or counts up front.
+
+
+When to use fastmem
+-------------------
+
+Use fastmem when:
+
+* Small objects (up to 1 MiB) are allocated and freed on the
+ data path with low, predictable latency requirements.
+
+* Many object types of varying sizes exist and maintaining a
+ separate mempool for each is impractical.
+
+* DMA-usable memory with efficient virtual-to-IOVA translation
+ is needed.
+
+Do not use fastmem for allocations larger than 1 MiB. Use
+``rte_malloc()`` instead.
+
+
+Initialization and teardown
+----------------------------
+
+.. code-block:: c
+
+ /* At startup, after rte_eal_init(). */
+ rte_fastmem_init();
+
+ /* Optional: pre-reserve backing memory to avoid latency
+ * spikes from on-demand memzone reservation. */
+ rte_fastmem_reserve(64 * 1024 * 1024, SOCKET_ID_ANY);
+
+ /* ... application runs ... */
+
+ /* At shutdown, after all allocations have been freed. */
+ rte_fastmem_deinit();
+
+Neither ``rte_fastmem_init()`` nor ``rte_fastmem_deinit()`` is
+thread-safe; call them from the main lcore during startup and
+shutdown.
+
+
+Allocation and free
+-------------------
+
+.. code-block:: c
+
+ void *obj = rte_fastmem_alloc(128, 0, 0);
+ /* Use obj... */
+ rte_fastmem_free(obj);
+
+``rte_fastmem_alloc()`` allocates on the calling lcore's NUMA
+socket. Use ``rte_fastmem_alloc_socket()`` to target a specific
+socket or to enable cross-socket fallback with ``SOCKET_ID_ANY``.
+
+Realloc
+~~~~~~~
+
+.. code-block:: c
+
+ obj = rte_fastmem_realloc(obj, 256, 0);
+
+``rte_fastmem_realloc()`` resizes an allocation, preserving its
+contents. If the existing allocation already satisfies the new
+size, the original pointer may be returned unchanged. Otherwise a
+new allocation is made, contents are copied, and the old
+allocation is freed. On failure, the original allocation remains
+valid.
+
+Alignment
+~~~~~~~~~
+
+When ``align`` is 0, the returned pointer is aligned to at least
+``RTE_CACHE_LINE_SIZE``. A non-zero ``align`` must be a power of
+two. Specifying an alignment smaller than ``RTE_CACHE_LINE_SIZE``
+is permitted but the returned object may then share a cache line
+with an adjacent allocation, risking false sharing.
+
+Zeroing
+~~~~~~~
+
+Pass ``RTE_FASTMEM_F_ZERO`` to receive zero-initialized memory:
+
+.. code-block:: c
+
+ void *obj = rte_fastmem_alloc(256, 0, RTE_FASTMEM_F_ZERO);
+
+
+Bulk allocation and free
+-------------------------
+
+.. code-block:: c
+
+ void *ptrs[32];
+
+ if (rte_fastmem_alloc_bulk(ptrs, 32, 64, 0, 0) < 0)
+ /* handle error */;
+
+ /* Use objects... */
+
+ rte_fastmem_free_bulk(ptrs, 32);
+
+Bulk allocation has all-or-nothing semantics: either all
+requested objects are returned, or none are (and ``rte_errno``
+is set to ``ENOMEM``).
+
+Bulk free is most efficient when all objects belong to the same
+size class; in that case the objects are pushed into the
+per-lcore cache in a single operation.
+
+
+IOVA translation
+----------------
+
+Memory returned by fastmem is DMA-usable. To obtain the IOVA
+for use in device descriptors:
+
+.. code-block:: c
+
+ rte_iova_t iova = rte_fastmem_virt2iova(obj);
+
+The translation is O(1). The returned IOVA is valid for the
+lifetime of the allocation.
+
+
+NUMA awareness
+--------------
+
+``rte_fastmem_alloc()`` allocates on the calling lcore's socket.
+``rte_fastmem_alloc_socket()`` accepts an explicit socket ID or
+``SOCKET_ID_ANY``:
+
+* Explicit socket: allocate only from that socket; fail with
+ ``ENOMEM`` if exhausted.
+
+* ``SOCKET_ID_ANY``: try the caller's local socket first, then
+ fall back to other sockets.
+
+
+Per-lcore caches
+----------------
+
+Each EAL thread has a private cache per size class. The common
+allocation and free paths operate entirely within this cache,
+avoiding locks. Cache misses (empty on alloc, full on free)
+trigger a bulk transfer to/from the shared bin under a lock.
+
+Non-EAL threads bypass the cache and take the bin lock on every
+operation.
+
+``rte_fastmem_cache_flush()`` drains the calling lcore's caches
+back to the shared bins. This is useful after bursty phases to
+release idle cached memory.
+
+
+Threading
+---------
+
+All allocation and free functions are thread-safe and may be
+called from any thread. An allocation made on one thread may be
+freed on any other.
+
+Fastmem uses internal spinlocks. A thread preempted while
+holding one delays other threads contending for the same lock
+(correctness is not affected, only latency).
+
+
+Pre-reserving memory
+--------------------
+
+By default, fastmem reserves backing memory lazily on first
+allocation. ``rte_fastmem_reserve(size, socket_id)`` forces
+reservation up front, ensuring subsequent allocations do not
+incur memzone-reservation latency:
+
+.. code-block:: c
+
+ /* Reserve 128 MiB on socket 0. */
+ rte_fastmem_reserve(128 * 1024 * 1024, 0);
+
+Once reserved, backing memory is never returned to the system
+during the allocator's lifetime.
+
+Memory limits
+~~~~~~~~~~~~~
+
+``rte_fastmem_set_limit(socket_id, max_bytes)`` caps how much
+backing memory may be reserved on a given socket. Once the limit is
+reached, allocations that would require new backing memory fail with
+``ENOMEM``. The default is ``SIZE_MAX`` (unlimited).
+``rte_fastmem_get_limit()`` returns the current limit for a socket.
+
+.. code-block:: c
+
+ /* Allow at most 256 MiB on socket 0. */
+ rte_fastmem_set_limit(0, 256 * 1024 * 1024);
+
+ /* Block all growth on socket 1. */
+ rte_fastmem_set_limit(1, 0);
+
+Pass ``SOCKET_ID_ANY`` to apply the same limit to all sockets.
+
+
+Size classes
+------------
+
+Fastmem uses power-of-two size classes from 8 bytes to 1 MiB
+(18 classes). A request for N bytes is served from the smallest
+class >= N. The maximum supported size is queryable via
+``rte_fastmem_max_size()``.
+
+With power-of-two classes, worst-case internal fragmentation is
+just under 50% (e.g., a 33-byte request occupies a 64-byte
+slot). Assuming a uniform distribution of request sizes, the
+average waste is 25%. In practice, DPDK workloads tend to
+cluster at or near powers of two, so typical waste is lower.
+
+Requests exceeding the maximum are rejected with ``E2BIG``.
+
+
+Implementation
+--------------
+
+Fastmem organizes memory in three layers: backing memzones, slabs,
+and per-lcore caches.
+
+Backing memory and slabs
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Backing memory is obtained from EAL as 128 MiB IOVA-contiguous
+memzones, each aligned to 2 MiB. A memzone is partitioned into
+64 fixed-size, 2 MiB **slabs**. Slabs are the unit of memory
+that moves between size classes: a free slab can be assigned to
+any bin on demand, and an empty slab (all objects freed) returns
+to the free-slab pool for reuse by another size class.
+
+The 2 MiB slab alignment is the key structural property. Given
+any object pointer, the allocator recovers the owning slab by
+masking off the low 21 bits — no radix tree, hash table, or
+memzone lookup is needed. This makes the free path fast: a
+single pointer-mask load reaches the slab header, which
+identifies the size class and bin.
+
+Each slab reserves 64 bytes at offset 0 for its header. The
+remaining space is divided into fixed-size slots equal to the
+size class. Allocated objects carry no per-object metadata; the
+full slot is available to the caller.
+
+Three-level allocation hierarchy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. **Per-lcore cache** — a bounded LIFO stack of free object
+ pointers, one per (lcore, size class, socket). Allocation
+ pops; free pushes. No lock is needed because only the owning
+ lcore accesses its cache.
+
+2. **Bin** — one per (size class, socket). Owns the partial and
+ full slab lists. A spinlock serializes bulk transfers between
+ the bin and per-lcore caches. Most traffic is absorbed by the
+ caches, so bin-lock contention is low.
+
+3. **Free-slab pool** — one per socket. A spinlock protects slab
+ acquisition and release. These events are rare relative to
+ object-level operations (a single small-object slab serves
+ thousands of allocations).
+
+On a cache miss (empty on alloc, full on free), the cache
+exchanges objects with the bin in bulk, targeting half-full to
+maximize headroom in both directions.
+
+Cache sizing
+~~~~~~~~~~~~
+
+Cache capacity varies by size class to bound per-lcore memory
+footprint:
+
+* Classes 8 B through 4 KiB: capacity 64.
+* Larger classes: capacity halves per class (32, 16, 8, 4),
+ flooring at 4.
+
+Even the largest classes remain cached. The capacity curve
+ensures that small, frequent allocations get the highest cache
+hit rate, while large allocations still avoid the bin lock on
+most operations.
+
+
+Statistics
+----------
+
+Fastmem maintains always-on, per-lcore counters that track
+allocation and free activity. Statistics are queryable at four
+levels of granularity: global summary, per size class, per lcore,
+and per lcore per class.
+
+``rte_fastmem_classes()`` returns the number of size classes and
+optionally fills an array with their sizes.
+
+See ``rte_fastmem.h`` for the full statistics API.
+
+
+Secondary Processes
+-------------------
+
+Fastmem works transparently in DPDK secondary processes. The shared
+state is discovered automatically on first allocation.
+
+Secondary processes do not use per-lcore caches; every allocation and
+free acquires the bin spinlock directly. This is acceptable for
+control-plane secondaries with low allocation rates. The primary
+process should pre-reserve sufficient backing memory with
+``rte_fastmem_reserve()`` since secondaries cannot grow the pool.
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index e6f24945b0..c85196c85e 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -28,6 +28,7 @@ Memory Management
mempool_lib
mbuf_lib
multi_proc_support
+ fastmem_lib
CPU Management
--
2.43.0
^ permalink raw reply related
* [RFC v3 0/3] lib/fastmem: fast small-object allocator
From: Mattias Rönnblom @ 2026-05-27 17:30 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Bruce Richardson,
Mattias Rönnblom
In-Reply-To: <20260526085743.64396-4-hofors@lysator.liu.se>
This RFC introduces fastmem, a general-purpose small-object allocator
for DPDK. It is intended to replace per-type mempools with a single
allocator that handles arbitrary sizes, grows on demand, and matches
mempool-level performance on the hot path.
Motivation
----------
DPDK applications commonly maintain many mempools — one per object
type (connections, sessions, timers, work items). Each must be sized
up front, wastes memory when over-provisioned, and cannot serve
objects of a different size. Fastmem eliminates this by accepting
arbitrary sizes at runtime, backed by a slab allocator that
repurposes memory across size classes as demand shifts.
Design
------
Three-layer architecture:
1. Backing memory: 128 MiB IOVA-contiguous memzones from EAL,
reserved lazily (or pre-reserved for deterministic latency).
2. Slabs: 2 MiB, 2 MiB-aligned regions carved from memzones.
The alignment enables O(1) slab lookup from any object pointer
via bitmask — no radix tree or index structure. Slabs move
freely between 18 power-of-2 size classes (8 B to 1 MiB).
3. Per-lcore caches: bounded LIFO stacks (no locks on the hot
path). Cache misses trigger bulk transfers to/from the shared
bin under a spinlock.
Key properties:
- Zero per-object metadata in the production build.
- NUMA-aware, with per-socket bins and free-slab pools.
- DMA-usable memory with O(1) virt-to-IOVA translation.
- Bulk alloc/free with all-or-nothing semantics.
- Backing memory never returned during lifetime (slabs recycled).
- Non-EAL threads supported (bypass cache, take bin lock).
- Secondary process support (lazy attach, no per-lcore caches).
API surface
-----------
rte_fastmem_init / deinit
rte_fastmem_reserve
rte_fastmem_set_limit / get_limit
rte_fastmem_alloc / alloc_socket
rte_fastmem_realloc
rte_fastmem_alloc_bulk / alloc_bulk_socket
rte_fastmem_free / free_bulk
rte_fastmem_hlookup / halloc / halloc_bulk / hfree / hfree_bulk
rte_fastmem_virt2iova
rte_fastmem_cache_flush
rte_fastmem_max_size / classes
rte_fastmem_stats / stats_class / stats_lcore / stats_lcore_class
rte_fastmem_stats_reset
All APIs are marked __rte_experimental.
Performance
-----------
The single-object hot path is roughly 2–3× the cost of mempool
and an order of magnitude faster than rte_malloc. Under
multi-lcore contention, fastmem scales similarly to mempool,
while rte_malloc collapses.
Limitations
-----------
- Maximum allocation: 1 MiB. Larger requests should use rte_malloc.
- Power-of-2 classes only; worst-case internal fragmentation ~50%.
- Backing memory not reclaimable short of deinit.
Future work
-----------
- Lcore-affine allocations (false-sharing-free by construction).
- Mempool ops driver for transparent drop-in use.
- Debug mode (cookies, double-free detection, poison-on-free).
- Telemetry integration.
- EAL integration, allowing EAL-internal subsystems to use
fastmem for their small-object allocations.
Changes in RFC v3:
- Add rte_fastmem_realloc() with full test coverage.
- Add __rte_malloc/__rte_dealloc compiler attributes; remove
incorrect __rte_alloc_size/__rte_alloc_align.
- Extract normalize_align() helper; remove redundant inline
directives.
- Merge lifecycle and functional test suites.
- Add realloc subsection to programming guide.
Changes in RFC v2:
- Fix cross-socket deinit use-after-free.
- Add secondary process support.
- Add handle-based allocation API.
- Fix clang warnings; misc cleanup.
Mattias Rönnblom (3):
doc: add fastmem programming guide
lib: add fastmem library
app/test: add fastmem test suite
app/test/meson.build | 3 +
app/test/test_fastmem.c | 1801 +++++++++++++++++++++++++
app/test/test_fastmem_perf.c | 1040 ++++++++++++++
app/test/test_fastmem_profile.c | 157 +++
doc/api/doxy-api-index.md | 1 +
doc/api/doxy-api.conf.in | 1 +
doc/guides/prog_guide/fastmem_lib.rst | 328 +++++
doc/guides/prog_guide/index.rst | 1 +
lib/fastmem/meson.build | 6 +
lib/fastmem/rte_fastmem.c | 1748 ++++++++++++++++++++++++
lib/fastmem/rte_fastmem.h | 815 +++++++++++
lib/meson.build | 1 +
12 files changed, 5902 insertions(+)
create mode 100644 app/test/test_fastmem.c
create mode 100644 app/test/test_fastmem_perf.c
create mode 100644 app/test/test_fastmem_profile.c
create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
create mode 100644 lib/fastmem/meson.build
create mode 100644 lib/fastmem/rte_fastmem.c
create mode 100644 lib/fastmem/rte_fastmem.h
--
2.43.0
^ permalink raw reply
* Re: [RFC 2/3] lib: add fastmem library
From: Mattias Rönnblom @ 2026-05-27 17:25 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260527072251.7c53719d@phoenix.local>
On 5/27/26 16:22, Stephen Hemminger wrote:
> On Mon, 25 May 2026 12:36:41 +0200
> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>
>> +
>> +static __rte_always_inline struct fastmem_cache *
>> +cache_get(struct fastmem_socket_state *socket, unsigned int class_idx,
>> + unsigned int lcore_id)
>
> Do not use always_inline. With current compilers using always inline
> makes the optimizer generate worse code. The only exceptions would
> be where inline is required to make assembly work or you have good benchmark
> data that proves that always_inline generates > 1% performance gain.
>
> To much of DPDK use __rte_always_inline as "cargo cult" it is faster setting.
__rte_always_inline is still useful, but it is rare. For example, it may
be required in certain situations to force constant propagation to
actually occur.
I'm removing both inline and always_inline. Doesn't make a difference,
so noise.
^ permalink raw reply
* RE: [PATCH v4 04/27] bpf: use C11 atomics in BPF_ST_ATOMIC_REG
From: Marat Khalili @ 2026-05-27 16:52 UTC (permalink / raw)
To: Stephen Hemminger, dev@dpdk.org; +Cc: Konstantin Ananyev
In-Reply-To: <20260526232542.620966-5-stephen@networkplumber.org>
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday 27 May 2026 00:24
> To: dev@dpdk.org
> Cc: Stephen Hemminger <stephen@networkplumber.org>; Konstantin Ananyev <konstantin.ananyev@huawei.com>;
> Marat Khalili <marat.khalili@huawei.com>
> Subject: [PATCH v4 04/27] bpf: use C11 atomics in BPF_ST_ATOMIC_REG
>
> The BPF_ST_ATOMIC_REG macro generated code with deprecated
> rte_atomicNN_add and rte_atomicNN_exchange.
>
> Replace this with the equivalent rte_stdatomic definitions.
> Use memory order seq_cst to preserve the previous behavior of
> rte_atomicNN_add() / rte_atomicNN_exchange() and matches
> the Linux kernel BPF interpreter for these opcodes.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> lib/bpf/bpf_exec.c | 13 +++++++------
> 1 file changed, 7 insertions(+), 6 deletions(-)
>
> diff --git a/lib/bpf/bpf_exec.c b/lib/bpf/bpf_exec.c
> index 18013753b1..ee6ec7516f 100644
> --- a/lib/bpf/bpf_exec.c
> +++ b/lib/bpf/bpf_exec.c
> @@ -10,6 +10,7 @@
> #include <rte_log.h>
> #include <rte_debug.h>
> #include <rte_byteorder.h>
> +#include <rte_stdatomic.h>
>
> #include "bpf_impl.h"
>
> @@ -65,16 +66,16 @@
> (type)(reg)[(ins)->src_reg])
>
> #define BPF_ST_ATOMIC_REG(reg, ins, tp) do { \
> + RTE_ATOMIC(uint##tp##_t) *dst = (RTE_ATOMIC(uint##tp##_t) *) \
> + (uintptr_t)((reg)[(ins)->dst_reg] + (ins)->off); \
> switch (ins->imm) { \
> case BPF_ATOMIC_ADD: \
> - rte_atomic##tp##_add((rte_atomic##tp##_t *) \
> - (uintptr_t)((reg)[(ins)->dst_reg] + (ins)->off), \
> - (reg)[(ins)->src_reg]); \
> + rte_atomic_fetch_add_explicit(dst, \
> + (reg)[(ins)->src_reg], rte_memory_order_seq_cst); \
> break; \
> case BPF_ATOMIC_XCHG: \
> - (reg)[(ins)->src_reg] = rte_atomic##tp##_exchange((uint##tp##_t *) \
> - (uintptr_t)((reg)[(ins)->dst_reg] + (ins)->off), \
> - (reg)[(ins)->src_reg]); \
> + (reg)[(ins)->src_reg] = rte_atomic_exchange_explicit(dst, \
> + (reg)[(ins)->src_reg], rte_memory_order_seq_cst); \
> break; \
> default: \
> /* this should be caught by validator and never reach here */ \
> --
> 2.53.0
Reviewed-by: Marat Khalili <marat.khalili@huawei.com>
nit: in the last sentence of the commit message `s` is not needed in `matches`,
and some word like `behavior` can be added for clarity after `interpreter`.
FWIW whole patchset builds on our amd64 and arm64 machines and passes our subset of tests.
^ permalink raw reply
* [RFC] doc, devtools: discourage new __rte_always_inline
From: Stephen Hemminger @ 2026-05-27 16:37 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger, Thomas Monjalon
Modern compilers at -O2 make good inlining decisions for small
static inline functions; forced inlining via __rte_always_inline
should be reserved for cases where it is required for correctness
or for documented measured performance reasons.
Document the policy in the coding style guide and add a
checkpatches.sh entry that warns when new uses of the attribute
are introduced.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
devtools/checkpatches.sh | 8 ++++++++
doc/guides/contributing/coding_style.rst | 25 +++++++++++++++++++++++-
2 files changed, 32 insertions(+), 1 deletion(-)
diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh
index f5dd77443f..2a3d364178 100755
--- a/devtools/checkpatches.sh
+++ b/devtools/checkpatches.sh
@@ -137,6 +137,14 @@ check_forbidden_additions() { # <patch>
-f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
"$1" || res=1
+ # forbid new use of __rte_always_inline
+ awk -v FOLDERS="lib drivers app examples" \
+ -v EXPRESSIONS='\\<__rte_always_inline\\>' \
+ -v RET_ON_FAIL=1 \
+ -v MESSAGE='Adding __rte_always_inline; prefer plain inline' \
+ -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
+ "$1" || res=1
+
# refrain from using compiler __rte_atomic_thread_fence()
# It should be avoided on x86 for SMP case.
awk -v FOLDERS="lib drivers app examples" \
diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index 243a3c2959..97e459853c 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -747,11 +747,34 @@ Static Variables and Functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* All functions and variables that are local to a file must be declared as ``static`` because it can often help the compiler to do some optimizations (such as, inlining the code).
-* Functions that should be inlined should to be declared as ``static inline`` and can be defined in a .c or a .h file.
+* Functions that should be inlined should be declared as ``static inline`` and can be defined in a .c or a .h file.
.. note::
Static functions defined in a header file must be declared as ``static inline`` in order to prevent compiler warnings about the function being unused.
+Use of ``__rte_always_inline``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``__rte_always_inline`` attribute forces the compiler to inline a function regardless of its size or call-graph heuristics.
+Prefer plain ``inline`` (or no annotation at all for static functions) and let the compiler decide.
+Modern compilers at ``-O2`` make good inlining decisions for small ``static inline`` functions in headers,
+and forced inlining can hurt performance by inflating function bodies, increasing register pressure, and overriding profile-guided optimization.
+
+``__rte_always_inline`` should only be used when one of the following applies:
+
+* The function contains ``__builtin_constant_p`` checks that gate a constant-folded fast path, and the optimization is lost if the function is not inlined into the caller.
+ Examples include byte-order helpers and length-dispatched copy/compare routines.
+
+* The function wraps inline assembly or a compiler intrinsic whose correctness depends on being inlined into the caller's register context (for example, intrinsics requiring a compile-time constant argument).
+
+* Measurement on a representative workload shows that the annotation is required to retain performance, and the reason is documented in the commit message that introduces it.
+
+Each use must be justified at the point it is introduced. Adding ``__rte_always_inline`` because nearby code uses it is not a justification;
+if the constant or intrinsic that requires inlining is several call levels up the call chain,
+restructure the code rather than annotating the entire chain.
+
+The complementary attribute ``__rte_noinline`` is useful for explicitly marking cold paths (error handling, initialization, slow-path fallbacks) where outlining the function can reduce instruction-cache pressure on the hot path.
+
Const Attribute
~~~~~~~~~~~~~~~
--
2.53.0
^ permalink raw reply related
* Re: [PATCH 3/3] net/bonding: remove redundant function names from log
From: Bruce Richardson @ 2026-05-27 16:28 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, Chas Williams, Min Hu (Connor)
In-Reply-To: <20260417165530.653328-4-stephen@networkplumber.org>
On Fri, Apr 17, 2026 at 09:51:37AM -0700, Stephen Hemminger wrote:
> The function name is already printed as part of RTE_BOND_LOG().
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> drivers/net/bonding/rte_eth_bond_api.c | 4 ++--
> drivers/net/bonding/rte_eth_bond_pmd.c | 23 ++++++++++-------------
> 2 files changed, 12 insertions(+), 15 deletions(-)
>
> diff --git a/drivers/net/bonding/rte_eth_bond_api.c b/drivers/net/bonding/rte_eth_bond_api.c
> index 9e5df67c18..78361e73d4 100644
> --- a/drivers/net/bonding/rte_eth_bond_api.c
> +++ b/drivers/net/bonding/rte_eth_bond_api.c
> @@ -485,8 +485,8 @@ __eth_bond_member_add_lock_free(uint16_t bonding_port_id, uint16_t member_port_i
> ret = rte_eth_dev_info_get(member_port_id, &dev_info);
> if (ret != 0) {
> RTE_BOND_LOG(ERR,
> - "%s: Error during getting device (port %u) info: %s",
> - __func__, member_port_id, strerror(-ret));
> + "Error during getting device (port %u) info: %s",
While updating, I think it would be good to update the log messages to be
more natural-sounding English. s/during getting/getting/
> + member_port_id, strerror(-ret));
>
> return ret;
> }
> diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
> index 6a42257d2b..e2819c931b 100644
> --- a/drivers/net/bonding/rte_eth_bond_pmd.c
> +++ b/drivers/net/bonding/rte_eth_bond_pmd.c
> @@ -210,8 +210,8 @@ bond_ethdev_8023ad_flow_verify(struct rte_eth_dev *bond_dev,
> int ret = rte_flow_validate(member_port, &flow_attr_8023ad,
> flow_item_8023ad, actions, &error);
> if (ret < 0) {
> - RTE_BOND_LOG(ERR, "%s: %s (member_port=%d queue_id=%d)",
> - __func__, error.message, member_port,
> + RTE_BOND_LOG(ERR, "%s (member_port=%u queue_id=%u)",
> + error.message, member_port,
> internals->mode4.dedicated_queues.rx_qid);
> return -1;
> }
> @@ -219,8 +219,8 @@ bond_ethdev_8023ad_flow_verify(struct rte_eth_dev *bond_dev,
> ret = rte_eth_dev_info_get(member_port, &member_info);
> if (ret != 0) {
> RTE_BOND_LOG(ERR,
> - "%s: Error during getting device (port %u) info: %s",
> - __func__, member_port, strerror(-ret));
> + "Error during getting device (port %u) info: %s",
> + member_port, strerror(-ret));
>
> return ret;
> }
> @@ -228,8 +228,8 @@ bond_ethdev_8023ad_flow_verify(struct rte_eth_dev *bond_dev,
> if (member_info.max_rx_queues < bond_dev->data->nb_rx_queues ||
> member_info.max_tx_queues < bond_dev->data->nb_tx_queues) {
> RTE_BOND_LOG(ERR,
> - "%s: Member %d capabilities doesn't allow allocating additional queues",
> - __func__, member_port);
> + "Member %u capabilities doesn't allow allocating additional queues",
> + member_port);
> return -1;
> }
>
> @@ -249,9 +249,8 @@ bond_8023ad_slow_pkt_hw_filter_supported(uint16_t port_id) {
> ret = rte_eth_dev_info_get(bond_dev->data->port_id, &bond_info);
> if (ret != 0) {
> RTE_BOND_LOG(ERR,
> - "%s: Error during getting device (port %u) info: %s",
> - __func__, bond_dev->data->port_id,
> - strerror(-ret));
> + "Error during getting device (port %u) info: %s",
> + bond_dev->data->port_id, strerror(-ret));
>
> return ret;
> }
> @@ -2347,10 +2346,8 @@ bond_ethdev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
> ret = rte_eth_dev_info_get(member.port_id, &member_info);
> if (ret != 0) {
> RTE_BOND_LOG(ERR,
> - "%s: Error during getting device (port %u) info: %s",
> - __func__,
> - member.port_id,
> - strerror(-ret));
> + "Error during getting device (port %u) info: %s",
> + member.port_id, strerror(-ret));
>
> return ret;
> }
^ permalink raw reply
* Re: [PATCH 2/3] net/bonding: prevent crash on Rx/Tx from secondary process
From: Bruce Richardson @ 2026-05-27 16:24 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, stable, Chas Williams, Min Hu (Connor), Anatoly Burakov,
Qi Zhang
In-Reply-To: <20260417165530.653328-3-stephen@networkplumber.org>
On Fri, Apr 17, 2026 at 09:51:36AM -0700, Stephen Hemminger wrote:
> The bonding PMD's secondary process attach path registered the
> ethdev but never installed rx_pkt_burst or tx_pkt_burst, leaving
> both as NULL. Any rx_burst or tx_burst call from a secondary
> process therefore crashed with a NULL pointer dereference.
>
> Fully sharing bonding state across processes would be
> overly complex. Instead, install blackhole burst functions
> in the secondary so the data path is safe by default. Rx returns 0
> and Tx frees the mbufs and reports them as transmitted,
> matching /dev/null semantics so applications do not spin
> retrying. Each stub logs once at NOTICE level on first use.
>
> Also reject bond mode changes from a secondary process.
> rte_eth_bond_mode_set() is callable from secondary, and
> without this guard it would overwrite the secondary's safe
> burst stubs with real per-mode functions whose internal
> state is not valid outside the primary, reintroducing the
> crash by another path.
>
> This keeps secondary support available for the more
> common use cases of procinfo and packet capture.
>
> Bugzilla ID: 1698
> Fixes: 4852aa8f6e21 ("drivers/net: enable hotplug on secondary process")
> Cc: stable@dpdk.org
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
However, see comment below.
> drivers/net/bonding/rte_eth_bond_pmd.c | 43 +++++++++++++++++++++++++-
> 1 file changed, 42 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
> index 96725071da..6a42257d2b 100644
> --- a/drivers/net/bonding/rte_eth_bond_pmd.c
> +++ b/drivers/net/bonding/rte_eth_bond_pmd.c
> @@ -56,6 +56,33 @@ get_vlan_offset(struct rte_ether_hdr *eth_hdr, uint16_t *proto)
> return vlan_offset;
> }
>
> +static uint16_t
> +bond_ethdev_rx_secondary(void *queue __rte_unused,
> + struct rte_mbuf **bufs __rte_unused, uint16_t nb_pkts __rte_unused)
> +{
> + static bool once = true;
> +
> + if (once) {
> + /* once per process is enough of a notice */
> + RTE_BOND_LOG(NOTICE, "receive not supported in secondary");
Since this does nothing, I think that an ERROR level warning is more
appropriate than notice. If not for Rx, certainly for Tx which just
silently drops packets!
> + once = false;
> + }
> + return 0;
> +}
> +
> +static uint16_t
> +bond_ethdev_tx_secondary(void *queue __rte_unused, struct rte_mbuf **bufs, uint16_t nb_pkts)
> +{
> + static bool once = true;
> +
> + if (once) {
> + RTE_BOND_LOG(NOTICE, "transmit not supported in secondary");
> + once = false;
> + }
> + rte_pktmbuf_free_bulk(bufs, nb_pkts);
> + return nb_pkts;
> +}
> +
> static uint16_t
> bond_ethdev_rx_burst(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
> {
> @@ -1599,6 +1626,11 @@ bond_ethdev_mode_set(struct rte_eth_dev *eth_dev, uint8_t mode)
> {
> struct bond_dev_private *internals;
>
> + if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
> + RTE_BOND_LOG(ERR, "Setting mode in secondary not allowed");
> + return -1;
> + }
> +
> internals = eth_dev->data->dev_private;
>
> switch (mode) {
> @@ -3799,9 +3831,18 @@ bond_probe(struct rte_vdev_device *dev)
> RTE_BOND_LOG(ERR, "Failed to probe %s", name);
> return -1;
> }
> - /* TODO: request info from primary to set up Rx and Tx */
> +
> eth_dev->dev_ops = &default_dev_ops;
> eth_dev->device = &dev->device;
> +
> + /*
> + * Propagation of bond mode would require adding
> + * MP client/server support and lots of error handling.
> + *
> + * For now just install a black hole.
> + */
> + eth_dev->tx_pkt_burst = bond_ethdev_tx_secondary;
> + eth_dev->rx_pkt_burst = bond_ethdev_rx_secondary;
> rte_eth_dev_probing_finish(eth_dev);
> return 0;
> }
> --
> 2.53.0
>
^ permalink raw reply
* Re: [PATCH 1/3] net/bonding: restore dedicated queue state on mode set error
From: Bruce Richardson @ 2026-05-27 16:19 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, stable, Chas Williams, Min Hu (Connor), Tomasz Kulasek,
Declan Doherty
In-Reply-To: <20260417165530.653328-2-stephen@networkplumber.org>
On Fri, Apr 17, 2026 at 09:51:35AM -0700, Stephen Hemminger wrote:
> The calls to enable and disable dedicated queues are missing
> proper error handling. If setting bonding mode fails,
> restore original state and propagate the error return value.
>
> Fixes: 112891cd27e5 ("net/bonding: add dedicated HW queues for LACP control")
> Cc: stable@dpdk.org
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> drivers/net/bonding/rte_eth_bond_8023ad.c | 19 ++++++++++++-------
> 1 file changed, 12 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/net/bonding/rte_eth_bond_8023ad.c b/drivers/net/bonding/rte_eth_bond_8023ad.c
> index ba88f6d261..a74b0059ac 100644
> --- a/drivers/net/bonding/rte_eth_bond_8023ad.c
> +++ b/drivers/net/bonding/rte_eth_bond_8023ad.c
> @@ -1727,7 +1727,7 @@ RTE_EXPORT_SYMBOL(rte_eth_bond_8023ad_dedicated_queues_enable)
> int
> rte_eth_bond_8023ad_dedicated_queues_enable(uint16_t port)
> {
> - int retval = 0;
> + int ret;
Any particular reason for the variable rename? Not that it's a problem, but
it does expand the diff.
> struct rte_eth_dev *dev;
> struct bond_dev_private *internals;
>
> @@ -1744,17 +1744,20 @@ rte_eth_bond_8023ad_dedicated_queues_enable(uint16_t port)
> if (dev->data->dev_started)
> return -1;
>
> + uint8_t old = internals->mode4.dedicated_queues.enabled;
Should you add a check that old != 1, and just return ok if so?
> internals->mode4.dedicated_queues.enabled = 1;
> + ret = bond_ethdev_mode_set(dev, internals->mode);
> + if (ret != 0)
> + internals->mode4.dedicated_queues.enabled = old;
>
Looking through the code, for 8023ad mode, I don't see any way of failure,
so this failure handling seems pointless. The function
bond_mode_8023ad_enable() loops through all the devices in the bond but
never checks for error for any of them, so always returns zero.
What is missing here is a check that the internals->mode is actually set to
8023ad.
Similar feedback applies to the disable function below.
/Bruce
^ permalink raw reply
* Re: [PATCH v2] app/test-pmd: add generic PROG action parser support
From: Stephen Hemminger @ 2026-05-27 15:41 UTC (permalink / raw)
To: Megha Ajmera; +Cc: bruce.richardson, cristian.dumitrescu, praveen.shetty, dev
In-Reply-To: <20260521101354.726240-1-megha.ajmera@intel.com>
On Thu, 21 May 2026 15:43:54 +0530
Megha Ajmera <megha.ajmera@intel.com> wrote:
> Add parser support for a generic PROG flow action in testpmd.
>
> The update adds CLI tokens and parsing logic for program name and
> argument tuples (name, size, value), enabling programmable action
> configuration through the flow command interface.
>
> Example flow rule:
> flow create 0 ingress pattern eth / end actions prog name my_prog
> argument name arg0 size 4 value 10 / end
>
> Signed-off-by: Megha Ajmera <megha.ajmera@intel.com>
> Signed-off-by: Praveen Shetty <praveen.shetty@intel.com>
> ---
Ran AI review on this manually since that gets better context.
The feedback was:
On Thu, 21 May 2026 15:43:54 +0530
Megha Ajmera <megha.ajmera@intel.com> wrote:
> Add parser support for a generic PROG flow action in testpmd.
>
> The update adds CLI tokens and parsing logic for program name and
> argument tuples (name, size, value), enabling programmable action
> configuration through the flow command interface.
Errors
1. CREATE proceeds with partially-converted action data on conversion
failure.
In cmd_flow_parsed() the CREATE case prints a warning if
convert_action_prog_to_rte_flow() returns negative, then
unconditionally calls port_flow_create(). Inside
convert_action_prog_to_rte_flow(), each per-PROG failure does
"continue" rather than aborting, so the function returns negative
while some actions[i].conf pointers have been replaced with
rte_flow_action_prog and others still point to the parser-internal
action_prog_data (a completely different layout). The PMD then
dereferences a mix of two unrelated structure types. Either abort
the create on negative ret, or make convert_action_prog_to_rte_flow()
fail-fast: free everything converted so far, restore conf pointers
(or never overwrite them until the whole pass succeeds), and return
without touching port_flow_create().
2. Argument value is sent in host byte order, but the API requires
network byte order.
The doc for struct rte_flow_action_prog_argument in
lib/ethdev/rte_flow.h says the value array must be in network byte
order. The conversion code does:
memcpy(value, &prog_data->args[j].value, args[j].size);
prog_data->args[j].value is a host-order uint64_t populated by
parse_int. For "size 4 value 10" on little-endian this produces
{0x0a,0x00,0x00,0x00}; on big-endian it produces {0x00,0x00,0x00,0x00}
-- the leading zero bytes of the 8-byte value, so the user always
sees zero. Either convert with rte_cpu_to_be_32/64 before memcpy,
or have the parser accept and store the value as a network-order
byte array of the declared size.
3. Only the CREATE path is converted; VALIDATE and the async/template
paths still pass action_prog_data to the PMD.
cmd_flow_parsed() routes the same in->args.vc.actions array to
VALIDATE (port_flow_validate), and to multiple async/template paths
(port_queue_flow_create, port_flow_template_table_create,
port_action_handle_create, etc.). None of these are converted by
this patch, so any PROG action used with "flow validate",
"flow queue ... create", indirect-action create, or pattern/action
templates passes the parser-internal action_prog_data to the PMD
with the wrong layout. Conversion needs to be factored into a
helper invoked from every code path that hands actions to the
ethdev API, not bolted onto CREATE only.
4. Unbounded scan for END in convert_action_prog_to_rte_flow().
while (actions[i].type != RTE_FLOW_ACTION_TYPE_END)
i++;
If a caller ever passes an actions array without a terminating END,
this walks off the end. The caller already knows
in->args.vc.actions_n; pass that as the bound and remove the
dual-mode behavior (count-or-not). The function is only ever called
with prog_action_count == 0, so the other branch is dead code
anyway.
5. size == 0 argument is silently accepted but violates the API.
The rte_flow_action_prog_argument doc states "its size must be
non-zero and its value must point to a valid array of size bytes".
The convert code does:
args[j].size = prog_data->args[j].size;
if (args[j].size == 0)
continue;
This leaves args[j].value == NULL while args[j].size == 0 and hands
that to the PMD. Either reject size == 0 at parse time, or treat it
as an error during conversion. Letting it through means
valid-looking input produces an API-contract violation that the PMD
has no obligation to handle.
Warnings
6. Identifiers with a leading double underscore are reserved.
__prog_argument_name_args_push, __prog_argument_size_args_push, and
__prog_argument_value_args_push use a "__" prefix. The C standard
reserves names starting with two underscores for the implementation.
Rename to e.g. prog_argument_name_args_push.
7. RTE_SET_USED on parameters that are used.
parse_vc_action_prog_argument() marks token and str as unused, then
passes both to parse_default(). parse_vc_action_prog_argument_value()
marks size as unused, then passes it to parse_vc_conf(). The v2
changelog says "Fixed compilation warning"; these spurious
RTE_SET_USED calls look like that fix. Remove them -- they mislead
future readers and can hide a real unused parameter if one is added
later.
8. Doxygen parameter name mismatch.
The comment block above convert_action_prog_to_rte_flow() documents
"@param action_count" but the actual parameter is named
prog_action_count. The same comment describes a dual-mode interface
("If 0, will count until END action") that is never exercised --
fix the name or simplify the interface.
9. No testpmd user-guide update.
This adds new CLI syntax
actions prog name <name> argument name <n> size <s> value <v>
but doc/guides/testpmd_app_ug/testpmd_funcs.rst is not updated.
Users have no documented way to learn the syntax.
10. No release notes update.
A new user-visible testpmd feature should have a one-line entry in
the current release notes.
Info
11. The static "struct arg arg[ACTION_PROG_MAX_ARGS]" arrays inside the
three __prog_argument_*_args_push helpers can simply be a single
non-static "struct arg" (or three) on the stack. push_args()
consumes the entries during the current parse_vc_conf() call;
persistent storage across CLI invocations is not needed and
obscures the lifetime.
12. action_prog_data.length is written by the COMMON_STRING parser for
the prog name field but never read afterwards (convert uses strdup,
relying on NUL termination). It is required as a destination for
the COMMON_STRING args triple, but a brief comment would prevent
the next reader from thinking it is meaningful state.
^ permalink raw reply
* Re: [PATCH 1/2] ethdev: fix out-of-bounds write in GENEVE option conversion
From: Stephen Hemminger @ 2026-05-27 15:38 UTC (permalink / raw)
To: James Raphael Tiovalen; +Cc: orika, thomas, andrew.rybchenko, dev, stable
In-Reply-To: <20260526181159.287633-2-jamestiotio@gmail.com>
On Wed, 27 May 2026 02:11:58 +0800
James Raphael Tiovalen <jamestiotio@gmail.com> wrote:
> rte_flow_conv_item_spec() is documented to truncate output to the
> caller-supplied buffer size. For RTE_FLOW_ITEM_TYPE_GENEVE_OPT, the
> deep-copy of the variable-length option data was gated on `size > 0`
> instead of `size >= off + tmp`, the form used by the sibling RAW
> branch. A caller passing a buffer just large enough for the header
> struct had adjacent memory clobbered by up to `option_len * 4` bytes of
> option payload.
>
> Align the GENEVE_OPT guard with the RAW one.
>
> Fixes: 841a0445442d ("ethdev: fix GENEVE option item conversion")
> Cc: stable@dpdk.org
>
> Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com>
> ---
Does not apply to current main branch.
There were recent fixes in this area.
Rebase and resubmit please.
^ permalink raw reply
* Re: [PATCH v5 00/21] Wangxun Fixes
From: Stephen Hemminger @ 2026-05-27 15:22 UTC (permalink / raw)
To: Zaiyu Wang; +Cc: dev
In-Reply-To: <20260527130222.24348-1-zaiyuwang@trustnetic.com>
On Wed, 27 May 2026 21:02:00 +0800
Zaiyu Wang <zaiyuwang@trustnetic.com> wrote:
> This series fixes several issues found on Wangxun Emerald, Sapphire and
> Amber-lite NICs, with a focus on link-related problems.
> ---
> v5:
> - Fixed issues identified by AI review
> ---
> v4:
> - Fixed issues identified by devtools scripts
> ---
> v3:
> - Addressed Stephen's comments
> ---
> v2:
> - Fixed compilation error and code style issues
> ---
>
> Zaiyu Wang (21):
> net/txgbe: remove duplicate xstats counters
> net/ngbe: remove duplicate xstats counters
> net/ngbe: add missing CDR config for YT PHY
> net/ngbe: fix VF promiscuous and allmulticast
> net/txgbe: fix inaccuracy in Tx rate limiting
> net/txgbe: fix link status check condition
> net/txgbe: fix Tx desc free logic
> net/txgbe: fix link flow control registers for Amber-Lite
> net/txgbe: fix link flow control config for Sapphire
> net/txgbe: fix a mass of unknown interrupts
> net/txgbe: fix traffic class priority configuration
> net/txgbe: fix link stability for 25G NIC
> net/txgbe: fix link stability for 40G NIC
> net/txgbe: fix link stability for Amber-Lite backplane mode
> net/txgbe: fix FEC mode configuration on 25G NIC
> net/txgbe: fix SFP module identification
> net/txgbe: fix get module info operation
> net/txgbe: fix get EEPROM operation
> net/txgbe: fix to reset Tx write-back pointer
> net/txgbe: fix to enable Tx desc check
> net/txgbe: fix temperature track for AML NIC
>
> drivers/net/ngbe/base/ngbe_phy_yt.c | 3 +
> drivers/net/ngbe/ngbe_ethdev.c | 5 -
> drivers/net/ngbe/ngbe_ethdev_vf.c | 11 +-
> drivers/net/txgbe/base/meson.build | 2 +
> drivers/net/txgbe/base/txgbe.h | 2 +
> drivers/net/txgbe/base/txgbe_aml.c | 185 +-
> drivers/net/txgbe/base/txgbe_aml.h | 6 +-
> drivers/net/txgbe/base/txgbe_aml40.c | 114 +-
> drivers/net/txgbe/base/txgbe_aml40.h | 6 +-
> drivers/net/txgbe/base/txgbe_dcb_hw.c | 2 +-
> drivers/net/txgbe/base/txgbe_e56.c | 3773 +++++++++++++++++++++
> drivers/net/txgbe/base/txgbe_e56.h | 1753 ++++++++++
> drivers/net/txgbe/base/txgbe_e56_bp.c | 2597 ++++++++++++++
> drivers/net/txgbe/base/txgbe_e56_bp.h | 282 ++
> drivers/net/txgbe/base/txgbe_hw.c | 54 +-
> drivers/net/txgbe/base/txgbe_hw.h | 4 +-
> drivers/net/txgbe/base/txgbe_osdep.h | 4 +
> drivers/net/txgbe/base/txgbe_phy.c | 362 +-
> drivers/net/txgbe/base/txgbe_phy.h | 45 +-
> drivers/net/txgbe/base/txgbe_regs.h | 13 +-
> drivers/net/txgbe/base/txgbe_type.h | 43 +-
> drivers/net/txgbe/txgbe_ethdev.c | 458 ++-
> drivers/net/txgbe/txgbe_ethdev.h | 7 +-
> drivers/net/txgbe/txgbe_rxtx.c | 107 +-
> drivers/net/txgbe/txgbe_rxtx.h | 36 +
> drivers/net/txgbe/txgbe_rxtx_vec_common.h | 16 +-
> 26 files changed, 9445 insertions(+), 445 deletions(-)
> create mode 100644 drivers/net/txgbe/base/txgbe_e56.c
> create mode 100644 drivers/net/txgbe/base/txgbe_e56.h
> create mode 100644 drivers/net/txgbe/base/txgbe_e56_bp.c
> create mode 100644 drivers/net/txgbe/base/txgbe_e56_bp.h
>
More AI review feedback summary:
18/21 — get EEPROM (Error): page is declared once before the for loop
and never reset, so after crossing the 256-byte boundary every
subsequent iteration walks one page further into the module than it
should. Plus two related issues: data[0x2] reads the output buffer not
the module byte, and the skip branch leaves stale databyte carrying
over from the previous iteration.
20/21 — Tx desc check (Error): #ifdef RTE_LIBRTE_SECURITY uses the
pre-2020 macro name; the modern name is RTE_LIB_SECURITY (and the
surrounding code in the same file uses it correctly). The IPsec guard
is therefore always compiled out and the wr32m runs for every queue,
including IPsec ones.
16/21 & 17/21 — whitespace: \t if (tab-then-space) on two lines. Checkpatch will catch it, but worth a pass.
17/21 — TXGBE_SFF_DDM_IMPLEMENTED: value 0x40 is correct as a bit-6
mask of byte 0x5C, but it is defined next to register-offset macros
which makes it read as an offset.
14/21 — kr_read_poll macro: uses usleep() directly; everything else in the txgbe base layer uses usec_delay().
7/21 — type pun: the atomic load casts a volatile uint32_t * to
volatile uint16_t *. Works on all DPDK platforms but is
strict-aliasing-iffy; a u32 load with a u16 cast at the use site would
be cleaner.
Longer full review:
Review of [PATCH v5 00/21] net/{txgbe,ngbe} fixes from Zaiyu Wang
This revision addresses the substantive issues raised on v4:
- 07/21: Tx desc free now uses a documented helper
txgbe_tx_headwb_desc_done() that correctly handles the head==next
boundary, and switches the headwb_mem read from a plain volatile
access to rte_atomic_load_explicit(... acquire).
- 08/21: AML xon/xoff stats no longer use plain assignment. The
counter now goes through the new TXGBE_UPDATE_COUNTER_32BIT_GENERIC
macro with offset tracking, and txgbe_clear_hw_cntrs() write-clears
the AML registers and zeroes hw->last_stats on reset.
- 11/21: traffic class priority is consistent across all three
callers. TXGBE_RPUP2TC_UP_SHIFT is bumped to 4, TXGBE_DCBUP2TC_MAP
is updated to match, txgbe_vmdq_dcb_configure() uses the macros
instead of a hardcoded *3 shift, and the unused TXGBE_DCBUP2TC_DEC
is removed. The bonus fix of redirecting
txgbe_dcb_config_tx_data_arbiter_raptor() to TXGBE_PBTXUP2TC instead
of TXGBE_PBRXUP2TC is welcome.
- 12/21: txgbe_setup_phy_link_aml() now sets link_up = false before
'goto out' on TXGBE_ERR_TIMEOUT, so the out: block correctly routes
to *need_reset = true. The generic 'compare' qsort helper has been
renamed to txgbe_e56_int_cmp().
Remaining findings on v5 below.
----------------------------------------------------------------
Patch 18/21 (net/txgbe: fix get EEPROM operation)
Error: page accumulation across loop iterations in
txgbe_get_module_eeprom() will return wrong bytes for any QSFP read
that crosses the 256-byte page boundary.
+ uint8_t page = 0;
...
+ for (i = info->offset; i < info->offset + info->length; i++) {
+ if (is_sfp) {
...
+ } else {
+ offset = i;
+ while (offset >= RTE_ETH_MODULE_SFF_8436_LEN) {
+ offset -= RTE_ETH_MODULE_SFF_8436_LEN / 2;
+ page++;
+ }
+ if (page == 0 || !(data[0x2] & 0x4)) {
+ status = hw->phy.read_i2c_sff8636(hw, page, offset,
+ &databyte);
'page' is declared once before the for loop and never reset, but the
while loop only increments it. For i = 256 the math is correct
(page=0 entering, page=1 leaving, offset=128). For i = 257 the loop
enters with page=1 already set from the previous iteration and ends
with page=2, offset=129 - it should still be page=1. Every subsequent
iteration adds one more page, so the function reads bytes from
ever-higher pages instead of staying on page 1, 2, 3, ...
Reset page (and rebuild offset) at each iteration:
uint16_t addr;
uint8_t page;
for (i = info->offset; i < info->offset + info->length; i++) {
...
} else {
page = 0;
addr = i;
while (addr >= RTE_ETH_MODULE_SFF_8436_LEN) {
addr -= RTE_ETH_MODULE_SFF_8436_LEN / 2;
page++;
}
...
}
}
Two related concerns in the same block:
- The flat-memory check '!(data[0x2] & 0x4)' inspects byte 2 of the
caller's output buffer rather than module byte 2. If info->offset
> 2 the byte was never read, so the test reads the value left by
memset (zero) and always evaluates to true. Read module byte 2
explicitly before the loop and stash it in a local.
- When the skip branch is taken (paged read on flat memory), the
loop still does 'data[i - info->offset] = databyte', so the byte
written is whatever databyte held from the previous iteration.
Set databyte to 0 (or 0xff) before each iteration, or set the
output byte directly inside the skip branch.
Patch 20/21 (net/txgbe: fix to enable Tx desc check)
Error: the new IPsec guard uses the wrong macro name and is always
compiled out.
+#ifdef RTE_LIBRTE_SECURITY
+ if (!txq->using_ipsec)
+#endif
+ wr32m(hw, TXGBE_TDM_DESC_CHK(txq->reg_idx / 32),
+ RTE_BIT32(txq->reg_idx % 32), RTE_BIT32(txq->reg_idx % 32));
The DPDK build macro is RTE_LIB_SECURITY; RTE_LIBRTE_SECURITY is the
pre-2020 name and is no longer defined anywhere in the tree (the rest
of this same driver uses RTE_LIB_SECURITY in the surrounding code,
including the deleted block this patch replaces). Result: the
'if (!txq->using_ipsec)' line is never preprocessed in, the wr32m
runs for every queue unconditionally, and IPsec-enabled queues get
the desc-check bit set even though the existing intent was to skip
them.
Replace with RTE_LIB_SECURITY. Matches the existing pattern
elsewhere in txgbe_rxtx.c (txgbe_rxtx.c:64, :444, :882, ...).
Patch 16/21 (net/txgbe: fix SFP module identification)
Warning: stray space in indentation of txgbe_read_i2c_sff8636() body.
+ s32 err = hw->phy.write_i2c_byte(hw, TXGBE_SFF_QSFP_PAGE_SELECT,
+ TXGBE_I2C_EEPROM_DEV_ADDR,
+ page);
+ if (err != 0)
+ return err;
The 'if' line is indented with tab+space instead of a single tab.
checkpatch will flag this.
Info: this patch refactors away phy.read_i2c_byte_unlocked and
phy.write_i2c_byte_unlocked and merges them into the existing
phy.read_i2c_byte / phy.write_i2c_byte slots, which now no longer
acquire the swfw semaphore. Patches 17 and 18 add explicit
acquire/release around their own callers, which is correct, but it is
worth double-checking that no other in-tree caller of
phy.read_i2c_eeprom / phy.read_i2c_sff8472 / phy.read_i2c_byte runs
without holding TXGBE_MNGSEM_SWPHY after this patch. The lock change
and the callsite updates would arguably read better squashed together
or at least kept adjacent in the series.
Patch 17/21 (net/txgbe: fix get module info operation)
Warning: stray space in indentation - same checkpatch issue as above.
+ if (hw->mac.type == txgbe_mac_aml) {
+ value = rd32(hw, TXGBE_GPIOEXT);
+ if (value & TXGBE_SFP1_MOD_ABS_LS) {
'if' is tab+space indented; should be two tabs.
Info: TXGBE_SFF_DDM_IMPLEMENTED is added next to the SFP register
offset definitions, but is then used as a bit mask of byte 0x5C:
#define TXGBE_SFF_DDM_IMPLEMENTED 0x40
#define TXGBE_SFF_SFF_8472_SWAP 0x5C
...
if (sff8472_rev == TXGBE_SFF_SFF_8472_UNSUP || page_swap ||
!(addr_mode & TXGBE_SFF_DDM_IMPLEMENTED)) {
The value 0x40 is correct as bit 6 of the diagnostic-monitoring-type
byte at offset 0x5C, but placing it alongside register offsets makes
the macro look like an offset. Consider moving it into a bit-mask
group or renaming the offset/mask pair to make the intent obvious
(e.g. _ADDR vs _MASK suffix).
Patch 14/21 (net/txgbe: fix link stability for Amber-Lite)
Warning: the new kr_read_poll() macro in txgbe_phy.h uses usleep()
directly, while the rest of the txgbe base layer uses the
usec_delay() abstraction (which expands to rte_delay_us_block() on
DPDK). Mixing the two is inconsistent and pulls in a POSIX dependency
in a base/ file:
+#define kr_read_poll(op, val, cond, sleep_us, \
+ times, args...) \
+({ \
...
+ usleep(__sleep_us);\
...
+})
Switch to usec_delay() to match txgbe_eeprom.c, txgbe_hw.c, and the
rest of the same file.
Patch 7/21 (net/txgbe: fix Tx desc free logic)
Info: txq->headwb_mem is declared 'volatile uint32_t *', but the new
atomic read reads it as 'volatile uint16_t *':
+ const uint16_t head = rte_atomic_load_explicit((volatile uint16_t *)txq->headwb_mem,
+ rte_memory_order_acquire);
This works on every architecture DPDK supports (lower 16 bits of an
aligned 32-bit object are accessible as a 16-bit atomic and head fits
in 16 bits), but it is a type pun on a hardware-written object and a
strict-aliasing violation in pure C. A 32-bit atomic load with an
explicit cast at use site would be cleaner:
uint32_t h = rte_atomic_load_explicit(txq->headwb_mem,
rte_memory_order_acquire);
const uint16_t head = (uint16_t)h;
Stephen Hemminger <stephen@networkplumber.org>
^ permalink raw reply
* [PATCH] ethdev: promote experimental API's to stable
From: Stephen Hemminger @ 2026-05-27 14:44 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger, Thomas Monjalon, Andrew Rybchenko
Several API's in ethdev have been marked as experimental
(some as old as 19.11). There is no firm policy but any API
that has been experimental for since the last LTS is good candidate
to be stable.
APIs added in 25.11 or later are left as experimental for
another release cycle.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
doc/guides/rel_notes/release_26_07.rst | 55 ++++++
lib/ethdev/rte_ethdev.c | 88 +++++-----
lib/ethdev/rte_ethdev.h | 233 -------------------------
lib/ethdev/rte_ethdev_cman.c | 8 +-
4 files changed, 103 insertions(+), 281 deletions(-)
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 1eac84895b..d1b1c0abc5 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -111,6 +111,61 @@ API Changes
Also, make sure to start the actual text at the margin.
=======================================================
+* **ethdev: promoted several APIs from experimental to stable.**
+
+ The following ethdev APIs are no longer marked experimental:
+
+ * ``rte_eth_buffer_split_get_supported_hdr_ptypes``
+ * ``rte_eth_cman_config_get``
+ * ``rte_eth_cman_config_init``
+ * ``rte_eth_cman_config_set``
+ * ``rte_eth_cman_info_get``
+ * ``rte_eth_dev_capability_name``
+ * ``rte_eth_dev_conf_get``
+ * ``rte_eth_dev_count_aggr_ports``
+ * ``rte_eth_dev_get_module_eeprom``
+ * ``rte_eth_dev_get_module_info``
+ * ``rte_eth_dev_get_reg_info_ext``
+ * ``rte_eth_dev_hairpin_capability_get``
+ * ``rte_eth_dev_map_aggr_tx_affinity``
+ * ``rte_eth_dev_priority_flow_ctrl_queue_configure``
+ * ``rte_eth_dev_priority_flow_ctrl_queue_info_get``
+ * ``rte_eth_dev_priv_dump``
+ * ``rte_eth_dev_rss_algo_name``
+ * ``rte_eth_fec_get``
+ * ``rte_eth_fec_get_capability``
+ * ``rte_eth_fec_set``
+ * ``rte_eth_find_rss_algo``
+ * ``rte_eth_get_monitor_addr``
+ * ``rte_eth_hairpin_bind``
+ * ``rte_eth_hairpin_get_peer_ports``
+ * ``rte_eth_hairpin_unbind``
+ * ``rte_eth_ip_reassembly_capability_get``
+ * ``rte_eth_ip_reassembly_conf_get``
+ * ``rte_eth_ip_reassembly_conf_set``
+ * ``rte_eth_link_speed_to_str``
+ * ``rte_eth_link_to_str``
+ * ``rte_eth_macaddrs_get``
+ * ``rte_eth_read_clock``
+ * ``rte_eth_recycle_mbufs``
+ * ``rte_eth_recycle_rx_queue_info_get``
+ * ``rte_eth_representor_info_get``
+ * ``rte_eth_rx_avail_thresh_query``
+ * ``rte_eth_rx_avail_thresh_set``
+ * ``rte_eth_rx_descriptor_dump``
+ * ``rte_eth_rx_hairpin_queue_setup``
+ * ``rte_eth_rx_queue_is_valid``
+ * ``rte_eth_speed_lanes_get``
+ * ``rte_eth_speed_lanes_get_capability``
+ * ``rte_eth_speed_lanes_set``
+ * ``rte_eth_timesync_adjust_freq``
+ * ``rte_eth_tx_descriptor_dump``
+ * ``rte_eth_tx_hairpin_queue_setup``
+ * ``rte_eth_tx_queue_count``
+ * ``rte_eth_tx_queue_is_valid``
+ * ``rte_eth_xstats_query_state``
+ * ``rte_eth_xstats_set_counter``
+
ABI Changes
-----------
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index 2edc7a362e..9ba84a6e5e 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -839,7 +839,7 @@ eth_dev_validate_tx_queue(const struct rte_eth_dev *dev, uint16_t tx_queue_id)
return 0;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_rx_queue_is_valid, 23.07)
+RTE_EXPORT_SYMBOL(rte_eth_rx_queue_is_valid)
int
rte_eth_rx_queue_is_valid(uint16_t port_id, uint16_t queue_id)
{
@@ -851,7 +851,7 @@ rte_eth_rx_queue_is_valid(uint16_t port_id, uint16_t queue_id)
return eth_dev_validate_rx_queue(dev, queue_id);
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_tx_queue_is_valid, 23.07)
+RTE_EXPORT_SYMBOL(rte_eth_tx_queue_is_valid)
int
rte_eth_tx_queue_is_valid(uint16_t port_id, uint16_t queue_id)
{
@@ -1171,7 +1171,7 @@ eth_dev_offload_names(uint64_t bitmask, char *buf, size_t size,
return buf;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_capability_name, 21.11)
+RTE_EXPORT_SYMBOL(rte_eth_dev_capability_name)
const char *
rte_eth_dev_capability_name(uint64_t capability)
{
@@ -1929,7 +1929,7 @@ rte_eth_dev_set_link_down(uint16_t port_id)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_speed_lanes_get, 24.11)
+RTE_EXPORT_SYMBOL(rte_eth_speed_lanes_get)
int
rte_eth_speed_lanes_get(uint16_t port_id, uint32_t *lane)
{
@@ -1943,7 +1943,7 @@ rte_eth_speed_lanes_get(uint16_t port_id, uint32_t *lane)
return eth_err(port_id, dev->dev_ops->speed_lanes_get(dev, lane));
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_speed_lanes_get_capability, 24.11)
+RTE_EXPORT_SYMBOL(rte_eth_speed_lanes_get_capability)
int
rte_eth_speed_lanes_get_capability(uint16_t port_id,
struct rte_eth_speed_lanes_capa *speed_lanes_capa,
@@ -1970,7 +1970,7 @@ rte_eth_speed_lanes_get_capability(uint16_t port_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_speed_lanes_set, 24.11)
+RTE_EXPORT_SYMBOL(rte_eth_speed_lanes_set)
int
rte_eth_speed_lanes_set(uint16_t port_id, uint32_t speed_lanes_capa)
{
@@ -2498,7 +2498,7 @@ rte_eth_rx_queue_setup(uint16_t port_id, uint16_t rx_queue_id,
return eth_err(port_id, ret);
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_rx_hairpin_queue_setup, 19.11)
+RTE_EXPORT_SYMBOL(rte_eth_rx_hairpin_queue_setup)
int
rte_eth_rx_hairpin_queue_setup(uint16_t port_id, uint16_t rx_queue_id,
uint16_t nb_rx_desc,
@@ -2716,7 +2716,7 @@ rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id,
tx_queue_id, nb_tx_desc, socket_id, &local_conf));
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_tx_hairpin_queue_setup, 19.11)
+RTE_EXPORT_SYMBOL(rte_eth_tx_hairpin_queue_setup)
int
rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id,
uint16_t nb_tx_desc,
@@ -2816,7 +2816,7 @@ rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_hairpin_bind, 20.11)
+RTE_EXPORT_SYMBOL(rte_eth_hairpin_bind)
int
rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)
{
@@ -2844,7 +2844,7 @@ rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_hairpin_unbind, 20.11)
+RTE_EXPORT_SYMBOL(rte_eth_hairpin_unbind)
int
rte_eth_hairpin_unbind(uint16_t tx_port, uint16_t rx_port)
{
@@ -2872,7 +2872,7 @@ rte_eth_hairpin_unbind(uint16_t tx_port, uint16_t rx_port)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_hairpin_get_peer_ports, 20.11)
+RTE_EXPORT_SYMBOL(rte_eth_hairpin_get_peer_ports)
int
rte_eth_hairpin_get_peer_ports(uint16_t port_id, uint16_t *peer_ports,
size_t len, uint32_t direction)
@@ -3200,7 +3200,7 @@ rte_eth_link_get_nowait(uint16_t port_id, struct rte_eth_link *eth_link)
return 0;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_link_speed_to_str, 20.11)
+RTE_EXPORT_SYMBOL(rte_eth_link_speed_to_str)
const char *
rte_eth_link_speed_to_str(uint32_t link_speed)
{
@@ -3267,7 +3267,7 @@ rte_eth_link_speed_to_str(uint32_t link_speed)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_link_to_str, 20.11)
+RTE_EXPORT_SYMBOL(rte_eth_link_to_str)
int
rte_eth_link_to_str(char *str, size_t len, const struct rte_eth_link *eth_link)
{
@@ -3946,7 +3946,7 @@ rte_eth_xstats_reset(uint16_t port_id)
return rte_eth_stats_reset(port_id);
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_xstats_set_counter, 25.03)
+RTE_EXPORT_SYMBOL(rte_eth_xstats_set_counter)
int
rte_eth_xstats_set_counter(uint16_t port_id, uint64_t id, int on_off)
{
@@ -3976,7 +3976,7 @@ rte_eth_xstats_set_counter(uint16_t port_id, uint64_t id, int on_off)
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_xstats_query_state, 25.03)
+RTE_EXPORT_SYMBOL(rte_eth_xstats_query_state)
int
rte_eth_xstats_query_state(uint16_t port_id, uint64_t id)
{
@@ -4145,7 +4145,7 @@ rte_eth_dev_info_get(uint16_t port_id, struct rte_eth_dev_info *dev_info)
return 0;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_conf_get, 21.11)
+RTE_EXPORT_SYMBOL(rte_eth_dev_conf_get)
int
rte_eth_dev_conf_get(uint16_t port_id, struct rte_eth_conf *dev_conf)
{
@@ -4306,7 +4306,7 @@ rte_eth_dev_set_ptypes(uint16_t port_id, uint32_t ptype_mask,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_macaddrs_get, 21.11)
+RTE_EXPORT_SYMBOL(rte_eth_macaddrs_get)
int
rte_eth_macaddrs_get(uint16_t port_id, struct rte_ether_addr *ma,
unsigned int num)
@@ -4805,7 +4805,7 @@ validate_tx_pause_config(struct rte_eth_dev_info *dev_info, uint8_t tc_max,
return 0;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_priority_flow_ctrl_queue_info_get, 22.03)
+RTE_EXPORT_SYMBOL(rte_eth_dev_priority_flow_ctrl_queue_info_get)
int
rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
struct rte_eth_pfc_queue_info *pfc_queue_info)
@@ -4833,7 +4833,7 @@ rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_priority_flow_ctrl_queue_configure, 22.03)
+RTE_EXPORT_SYMBOL(rte_eth_dev_priority_flow_ctrl_queue_configure)
int
rte_eth_dev_priority_flow_ctrl_queue_configure(uint16_t port_id,
struct rte_eth_pfc_queue_conf *pfc_queue_conf)
@@ -5147,7 +5147,7 @@ rte_eth_dev_rss_hash_conf_get(uint16_t port_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_rss_algo_name, 23.11)
+RTE_EXPORT_SYMBOL(rte_eth_dev_rss_algo_name)
const char *
rte_eth_dev_rss_algo_name(enum rte_eth_hash_function rss_algo)
{
@@ -5162,7 +5162,7 @@ rte_eth_dev_rss_algo_name(enum rte_eth_hash_function rss_algo)
return name;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_find_rss_algo, 24.03)
+RTE_EXPORT_SYMBOL(rte_eth_find_rss_algo)
int
rte_eth_find_rss_algo(const char *name, uint32_t *algo)
{
@@ -5280,7 +5280,7 @@ rte_eth_led_off(uint16_t port_id)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_fec_get_capability, 20.11)
+RTE_EXPORT_SYMBOL(rte_eth_fec_get_capability)
int
rte_eth_fec_get_capability(uint16_t port_id,
struct rte_eth_fec_capa *speed_fec_capa,
@@ -5308,7 +5308,7 @@ rte_eth_fec_get_capability(uint16_t port_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_fec_get, 20.11)
+RTE_EXPORT_SYMBOL(rte_eth_fec_get)
int
rte_eth_fec_get(uint16_t port_id, uint32_t *fec_capa)
{
@@ -5334,7 +5334,7 @@ rte_eth_fec_get(uint16_t port_id, uint32_t *fec_capa)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_fec_set, 20.11)
+RTE_EXPORT_SYMBOL(rte_eth_fec_set)
int
rte_eth_fec_set(uint16_t port_id, uint32_t fec_capa)
{
@@ -5694,7 +5694,7 @@ int rte_eth_set_queue_rate_limit(uint16_t port_id, uint16_t queue_idx,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_rx_avail_thresh_set, 22.07)
+RTE_EXPORT_SYMBOL(rte_eth_rx_avail_thresh_set)
int rte_eth_rx_avail_thresh_set(uint16_t port_id, uint16_t queue_id,
uint8_t avail_thresh)
{
@@ -5727,7 +5727,7 @@ int rte_eth_rx_avail_thresh_set(uint16_t port_id, uint16_t queue_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_rx_avail_thresh_query, 22.07)
+RTE_EXPORT_SYMBOL(rte_eth_rx_avail_thresh_query)
int rte_eth_rx_avail_thresh_query(uint16_t port_id, uint16_t *queue_id,
uint8_t *avail_thresh)
{
@@ -6413,7 +6413,7 @@ rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,
return 0;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_recycle_rx_queue_info_get, 23.11)
+RTE_EXPORT_SYMBOL(rte_eth_recycle_rx_queue_info_get)
int
rte_eth_recycle_rx_queue_info_get(uint16_t port_id, uint16_t queue_id,
struct rte_eth_recycle_rxq_info *recycle_rxq_info)
@@ -6504,7 +6504,7 @@ rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_get_monitor_addr, 21.02)
+RTE_EXPORT_SYMBOL(rte_eth_get_monitor_addr)
int
rte_eth_get_monitor_addr(uint16_t port_id, uint16_t queue_id,
struct rte_power_monitor_cond *pmc)
@@ -6675,7 +6675,7 @@ rte_eth_timesync_adjust_time(uint16_t port_id, int64_t delta)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_timesync_adjust_freq, 24.11)
+RTE_EXPORT_SYMBOL(rte_eth_timesync_adjust_freq)
int
rte_eth_timesync_adjust_freq(uint16_t port_id, int64_t ppm)
{
@@ -6746,7 +6746,7 @@ rte_eth_timesync_write_time(uint16_t port_id, const struct timespec *timestamp)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_read_clock, 19.08)
+RTE_EXPORT_SYMBOL(rte_eth_read_clock)
int
rte_eth_read_clock(uint16_t port_id, uint64_t *clock)
{
@@ -6802,7 +6802,7 @@ rte_eth_dev_get_reg_info(uint16_t port_id, struct rte_dev_reg_info *info)
return 0;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_get_reg_info_ext, 24.11)
+RTE_EXPORT_SYMBOL(rte_eth_dev_get_reg_info_ext)
int
rte_eth_dev_get_reg_info_ext(uint16_t port_id, struct rte_dev_reg_info *info)
{
@@ -6909,7 +6909,7 @@ rte_eth_dev_set_eeprom(uint16_t port_id, struct rte_dev_eeprom_info *info)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_get_module_info, 18.05)
+RTE_EXPORT_SYMBOL(rte_eth_dev_get_module_info)
int
rte_eth_dev_get_module_info(uint16_t port_id,
struct rte_eth_dev_module_info *modinfo)
@@ -6936,7 +6936,7 @@ rte_eth_dev_get_module_info(uint16_t port_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_get_module_eeprom, 18.05)
+RTE_EXPORT_SYMBOL(rte_eth_dev_get_module_eeprom)
int
rte_eth_dev_get_module_eeprom(uint16_t port_id,
struct rte_dev_eeprom_info *info)
@@ -7051,7 +7051,7 @@ rte_eth_dev_adjust_nb_rx_tx_desc(uint16_t port_id,
return 0;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_hairpin_capability_get, 19.11)
+RTE_EXPORT_SYMBOL(rte_eth_dev_hairpin_capability_get)
int
rte_eth_dev_hairpin_capability_get(uint16_t port_id,
struct rte_eth_hairpin_cap *cap)
@@ -7106,7 +7106,7 @@ rte_eth_dev_pool_ops_supported(uint16_t port_id, const char *pool)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_representor_info_get, 21.05)
+RTE_EXPORT_SYMBOL(rte_eth_representor_info_get)
int
rte_eth_representor_info_get(uint16_t port_id,
struct rte_eth_representor_info *info)
@@ -7162,7 +7162,7 @@ rte_eth_rx_metadata_negotiate(uint16_t port_id, uint64_t *features)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_ip_reassembly_capability_get, 22.03)
+RTE_EXPORT_SYMBOL(rte_eth_ip_reassembly_capability_get)
int
rte_eth_ip_reassembly_capability_get(uint16_t port_id,
struct rte_eth_ip_reassembly_params *reassembly_capa)
@@ -7198,7 +7198,7 @@ rte_eth_ip_reassembly_capability_get(uint16_t port_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_ip_reassembly_conf_get, 22.03)
+RTE_EXPORT_SYMBOL(rte_eth_ip_reassembly_conf_get)
int
rte_eth_ip_reassembly_conf_get(uint16_t port_id,
struct rte_eth_ip_reassembly_params *conf)
@@ -7232,7 +7232,7 @@ rte_eth_ip_reassembly_conf_get(uint16_t port_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_ip_reassembly_conf_set, 22.03)
+RTE_EXPORT_SYMBOL(rte_eth_ip_reassembly_conf_set)
int
rte_eth_ip_reassembly_conf_set(uint16_t port_id,
const struct rte_eth_ip_reassembly_params *conf)
@@ -7273,7 +7273,7 @@ rte_eth_ip_reassembly_conf_set(uint16_t port_id,
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_priv_dump, 22.03)
+RTE_EXPORT_SYMBOL(rte_eth_dev_priv_dump)
int
rte_eth_dev_priv_dump(uint16_t port_id, FILE *file)
{
@@ -7292,7 +7292,7 @@ rte_eth_dev_priv_dump(uint16_t port_id, FILE *file)
return eth_err(port_id, dev->dev_ops->eth_dev_priv_dump(dev, file));
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_rx_descriptor_dump, 22.11)
+RTE_EXPORT_SYMBOL(rte_eth_rx_descriptor_dump)
int
rte_eth_rx_descriptor_dump(uint16_t port_id, uint16_t queue_id,
uint16_t offset, uint16_t num, FILE *file)
@@ -7319,7 +7319,7 @@ rte_eth_rx_descriptor_dump(uint16_t port_id, uint16_t queue_id,
dev->dev_ops->eth_rx_descriptor_dump(dev, queue_id, offset, num, file));
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_tx_descriptor_dump, 22.11)
+RTE_EXPORT_SYMBOL(rte_eth_tx_descriptor_dump)
int
rte_eth_tx_descriptor_dump(uint16_t port_id, uint16_t queue_id,
uint16_t offset, uint16_t num, FILE *file)
@@ -7346,7 +7346,7 @@ rte_eth_tx_descriptor_dump(uint16_t port_id, uint16_t queue_id,
dev->dev_ops->eth_tx_descriptor_dump(dev, queue_id, offset, num, file));
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_buffer_split_get_supported_hdr_ptypes, 22.11)
+RTE_EXPORT_SYMBOL(rte_eth_buffer_split_get_supported_hdr_ptypes)
int
rte_eth_buffer_split_get_supported_hdr_ptypes(uint16_t port_id, uint32_t *ptypes, int num)
{
@@ -7386,7 +7386,7 @@ rte_eth_buffer_split_get_supported_hdr_ptypes(uint16_t port_id, uint32_t *ptypes
return j;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_count_aggr_ports, 23.03)
+RTE_EXPORT_SYMBOL(rte_eth_dev_count_aggr_ports)
int rte_eth_dev_count_aggr_ports(uint16_t port_id)
{
struct rte_eth_dev *dev;
@@ -7404,7 +7404,7 @@ int rte_eth_dev_count_aggr_ports(uint16_t port_id)
return ret;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_map_aggr_tx_affinity, 23.03)
+RTE_EXPORT_SYMBOL(rte_eth_dev_map_aggr_tx_affinity)
int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id,
uint8_t affinity)
{
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index 0d8e2d0236..0381f8317a 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -1061,9 +1061,6 @@ struct rte_eth_txmode {
};
/**
- * @warning
- * @b EXPERIMENTAL: this structure may change without prior notice.
- *
* A structure used to configure an Rx packet segment to split.
*
* If RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT flag is set in offloads field,
@@ -1139,9 +1136,6 @@ struct rte_eth_rxseg_split {
};
/**
- * @warning
- * @b EXPERIMENTAL: this structure may change without prior notice.
- *
* A common structure used to describe Rx packet segment properties.
*/
union rte_eth_rxseg {
@@ -1230,9 +1224,6 @@ struct rte_eth_txconf {
};
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* A structure used to return the Tx or Rx hairpin queue capabilities.
*/
struct rte_eth_hairpin_queue_cap {
@@ -1252,9 +1243,6 @@ struct rte_eth_hairpin_queue_cap {
};
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* A structure used to return the hairpin capabilities that are supported.
*/
struct rte_eth_hairpin_cap {
@@ -1272,9 +1260,6 @@ struct rte_eth_hairpin_cap {
#define RTE_ETH_MAX_HAIRPIN_PEERS 32
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* A structure used to hold hairpin peer data.
*/
struct rte_eth_hairpin_peer {
@@ -1283,9 +1268,6 @@ struct rte_eth_hairpin_peer {
};
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* A structure used to configure hairpin binding.
*/
struct rte_eth_hairpin_conf {
@@ -1426,9 +1408,6 @@ struct rte_eth_pfc_conf {
};
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* A structure used to retrieve information of queue based PFC.
*/
struct rte_eth_pfc_queue_info {
@@ -1441,9 +1420,6 @@ struct rte_eth_pfc_queue_info {
};
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* A structure used to configure Ethernet priority flow control parameters for
* ethdev queues.
*
@@ -1748,9 +1724,6 @@ struct rte_eth_switch_info {
};
/**
- * @warning
- * @b EXPERIMENTAL: this structure may change without prior notice.
- *
* Ethernet device Rx buffer segmentation capabilities.
*/
struct rte_eth_rxseg_capa {
@@ -1777,9 +1750,6 @@ enum rte_eth_representor_type {
};
/**
- * @warning
- * @b EXPERIMENTAL: this enumeration may change without prior notice.
- *
* Ethernet device error handling mode.
*/
enum rte_eth_err_handle_mode {
@@ -1918,9 +1888,6 @@ struct __rte_cache_min_aligned rte_eth_txq_info {
};
/**
- * @warning
- * @b EXPERIMENTAL: this structure may change without prior notice.
- *
* Ethernet device Rx queue information structure for recycling mbufs.
* Used to retrieve Rx queue information when Tx queue reusing mbufs and moving
* them into Rx mbuf ring.
@@ -2384,9 +2351,6 @@ const char *rte_eth_dev_rx_offload_name(uint64_t offload);
const char *rte_eth_dev_tx_offload_name(uint64_t offload);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Get RTE_ETH_DEV_CAPA_* flag name.
*
* @param capability
@@ -2394,7 +2358,6 @@ const char *rte_eth_dev_tx_offload_name(uint64_t offload);
* @return
* Capability name or 'UNKNOWN' if the flag cannot be recognized.
*/
-__rte_experimental
const char *rte_eth_dev_capability_name(uint64_t capability);
/**
@@ -2518,9 +2481,6 @@ int rte_eth_rx_queue_setup(uint16_t port_id, uint16_t rx_queue_id,
struct rte_mempool *mb_pool);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Allocate and set up a hairpin receive queue for an Ethernet device.
*
* The function set up the selected queue to be used in hairpin.
@@ -2544,7 +2504,6 @@ int rte_eth_rx_queue_setup(uint16_t port_id, uint16_t rx_queue_id,
* - (-EINVAL) if bad parameter.
* - (-ENOMEM) if unable to allocate the resources.
*/
-__rte_experimental
int rte_eth_rx_hairpin_queue_setup
(uint16_t port_id, uint16_t rx_queue_id, uint16_t nb_rx_desc,
const struct rte_eth_hairpin_conf *conf);
@@ -2602,9 +2561,6 @@ int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id,
const struct rte_eth_txconf *tx_conf);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Allocate and set up a transmit hairpin queue for an Ethernet device.
*
* @param port_id
@@ -2626,15 +2582,11 @@ int rte_eth_tx_queue_setup(uint16_t port_id, uint16_t tx_queue_id,
* - (-EINVAL) if bad parameter.
* - (-ENOMEM) if unable to allocate the resources.
*/
-__rte_experimental
int rte_eth_tx_hairpin_queue_setup
(uint16_t port_id, uint16_t tx_queue_id, uint16_t nb_tx_desc,
const struct rte_eth_hairpin_conf *conf);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Get all the hairpin peer Rx / Tx ports of the current port.
* The caller should ensure that the array is large enough to save the ports
* list.
@@ -2657,14 +2609,10 @@ int rte_eth_tx_hairpin_queue_setup
* - (-ENOTSUP) if hardware doesn't support.
* - Others detailed errors from PMDs.
*/
-__rte_experimental
int rte_eth_hairpin_get_peer_ports(uint16_t port_id, uint16_t *peer_ports,
size_t len, uint32_t direction);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Bind all hairpin Tx queues of one port to the Rx queues of the peer port.
* It is only allowed to call this function after all hairpin queues are
* configured properly and the devices are in started state.
@@ -2683,13 +2631,9 @@ int rte_eth_hairpin_get_peer_ports(uint16_t port_id, uint16_t *peer_ports,
* - (-ENOTSUP) if hardware doesn't support.
* - Others detailed errors from PMDs.
*/
-__rte_experimental
int rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Unbind all hairpin Tx queues of one port from the Rx queues of the peer port.
* This should be called before closing the Tx or Rx devices, if the bind
* function is called before.
@@ -2710,13 +2654,9 @@ int rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port);
* - (-ENOTSUP) if hardware doesn't support.
* - Others detailed errors from PMDs.
*/
-__rte_experimental
int rte_eth_hairpin_unbind(uint16_t tx_port, uint16_t rx_port);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Get the number of aggregated ports of the DPDK port (specified with port_id).
* It is used when multiple ports are aggregated into a single one.
*
@@ -2728,13 +2668,9 @@ int rte_eth_hairpin_unbind(uint16_t tx_port, uint16_t rx_port);
* @return
* - (>=0) the number of aggregated port if success.
*/
-__rte_experimental
int rte_eth_dev_count_aggr_ports(uint16_t port_id);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Map a Tx queue with an aggregated port of the DPDK port (specified with port_id).
* When multiple ports are aggregated into a single one,
* it allows to choose which port to use for Tx via a queue.
@@ -2758,7 +2694,6 @@ int rte_eth_dev_count_aggr_ports(uint16_t port_id);
* @return
* Zero if successful. Non-zero otherwise.
*/
-__rte_experimental
int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id,
uint8_t affinity);
@@ -2788,9 +2723,6 @@ int rte_eth_dev_socket_id(uint16_t port_id);
int rte_eth_dev_is_valid_port(uint16_t port_id);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
- *
* Check if Rx queue is valid.
* If the queue has been setup, it is considered valid.
*
@@ -2803,13 +2735,9 @@ int rte_eth_dev_is_valid_port(uint16_t port_id);
* - -EINVAL: if queue_id is out of range or queue has not been setup.
* - 0 if Rx queue is valid.
*/
-__rte_experimental
int rte_eth_rx_queue_is_valid(uint16_t port_id, uint16_t queue_id);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
- *
* Check if Tx queue is valid.
* If the queue has been setup, it is considered valid.
*
@@ -2822,7 +2750,6 @@ int rte_eth_rx_queue_is_valid(uint16_t port_id, uint16_t queue_id);
* - -EINVAL: if queue_id is out of range or queue has not been setup.
* - 0 if Tx queue is valid.
*/
-__rte_experimental
int rte_eth_tx_queue_is_valid(uint16_t port_id, uint16_t queue_id);
/**
@@ -3129,9 +3056,6 @@ int rte_eth_link_get_nowait(uint16_t port_id, struct rte_eth_link *link)
__rte_warn_unused_result;
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* The function converts a link_speed to a string. It handles all special
* values like unknown or none speed.
*
@@ -3141,13 +3065,9 @@ int rte_eth_link_get_nowait(uint16_t port_id, struct rte_eth_link *link)
* Link speed in textual format. It's pointer to immutable memory.
* No free is required.
*/
-__rte_experimental
const char *rte_eth_link_speed_to_str(uint32_t link_speed);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* This function converts an Ethernet link type to a string.
*
* @param link_connector
@@ -3159,9 +3079,6 @@ __rte_experimental
const char *rte_eth_link_connector_to_str(enum rte_eth_link_connector link_connector);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* The function converts a rte_eth_link struct representing a link status to
* a string.
*
@@ -3176,14 +3093,10 @@ const char *rte_eth_link_connector_to_str(enum rte_eth_link_connector link_conne
* @return
* Number of bytes written to str array or -EINVAL if bad parameter.
*/
-__rte_experimental
int rte_eth_link_to_str(char *str, size_t len,
const struct rte_eth_link *eth_link);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Get Active lanes.
*
* @param port_id
@@ -3200,13 +3113,9 @@ int rte_eth_link_to_str(char *str, size_t len,
* - (-EIO) if device is removed.
* - (-ENODEV) if *port_id* invalid.
*/
-__rte_experimental
int rte_eth_speed_lanes_get(uint16_t port_id, uint32_t *lanes);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Set speed lanes supported by the NIC.
*
* @param port_id
@@ -3224,13 +3133,9 @@ int rte_eth_speed_lanes_get(uint16_t port_id, uint32_t *lanes);
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if *lanes* count not in speeds capability list.
*/
-__rte_experimental
int rte_eth_speed_lanes_set(uint16_t port_id, uint32_t speed_lanes);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Get speed lanes supported by the NIC.
*
* @param port_id
@@ -3249,7 +3154,6 @@ int rte_eth_speed_lanes_set(uint16_t port_id, uint32_t speed_lanes);
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if *speed_lanes* invalid
*/
-__rte_experimental
int rte_eth_speed_lanes_get_capability(uint16_t port_id,
struct rte_eth_speed_lanes_capa *speed_lanes_capa,
unsigned int num);
@@ -3447,7 +3351,6 @@ int rte_eth_xstats_get_id_by_name(uint16_t port_id, const char *xstat_name,
* - (-EPERM) enabling this counter is not permitted
* - (-ENOSPC) no resources
*/
-__rte_experimental
int rte_eth_xstats_set_counter(uint16_t port_id, uint64_t id, int on_off);
/**
@@ -3461,7 +3364,6 @@ int rte_eth_xstats_set_counter(uint16_t port_id, uint64_t id, int on_off);
* - (-ENOTSUP) enable/disabling is not implemented
* - (-EINVAL) xstat id is invalid
*/
-__rte_experimental
int rte_eth_xstats_query_state(uint16_t port_id, uint64_t id);
/**
@@ -3539,9 +3441,6 @@ int rte_eth_dev_set_rx_queue_stats_mapping(uint16_t port_id,
int rte_eth_macaddr_get(uint16_t port_id, struct rte_ether_addr *mac_addr);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice
- *
* Retrieve the Ethernet addresses of an Ethernet device.
*
* @param port_id
@@ -3558,7 +3457,6 @@ int rte_eth_macaddr_get(uint16_t port_id, struct rte_ether_addr *mac_addr);
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if bad parameter.
*/
-__rte_experimental
int rte_eth_macaddrs_get(uint16_t port_id, struct rte_ether_addr *ma,
unsigned int num);
@@ -3585,9 +3483,6 @@ int rte_eth_dev_info_get(uint16_t port_id, struct rte_eth_dev_info *dev_info)
__rte_warn_unused_result;
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Retrieve the configuration of an Ethernet device.
*
* @param port_id
@@ -3599,7 +3494,6 @@ int rte_eth_dev_info_get(uint16_t port_id, struct rte_eth_dev_info *dev_info)
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if bad parameter.
*/
-__rte_experimental
int rte_eth_dev_conf_get(uint16_t port_id, struct rte_eth_conf *dev_conf)
__rte_warn_unused_result;
@@ -3848,9 +3742,6 @@ int rte_eth_dev_get_vlan_offload(uint16_t port_id);
int rte_eth_dev_set_vlan_pvid(uint16_t port_id, uint16_t pvid, int on);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Set Rx queue available descriptors threshold.
*
* @param port_id
@@ -3872,14 +3763,10 @@ int rte_eth_dev_set_vlan_pvid(uint16_t port_id, uint16_t pvid, int on);
* - (-ENOTSUP) if available Rx descriptors threshold is not supported.
* - (-EIO) if device is removed.
*/
-__rte_experimental
int rte_eth_rx_avail_thresh_set(uint16_t port_id, uint16_t queue_id,
uint8_t avail_thresh);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Find Rx queue with RTE_ETH_EVENT_RX_AVAIL_THRESH event pending.
*
* @param port_id
@@ -3902,7 +3789,6 @@ int rte_eth_rx_avail_thresh_set(uint16_t port_id, uint16_t queue_id,
* - (-ENOTSUP) if operation is not supported.
* - (-EIO) if device is removed.
*/
-__rte_experimental
int rte_eth_rx_avail_thresh_query(uint16_t port_id, uint16_t *queue_id,
uint8_t *avail_thresh);
@@ -4474,9 +4360,6 @@ int rte_eth_led_on(uint16_t port_id);
int rte_eth_led_off(uint16_t port_id);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Get Forward Error Correction(FEC) capability.
*
* @param port_id
@@ -4501,15 +4384,11 @@ int rte_eth_led_off(uint16_t port_id);
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if *num* or *speed_fec_capa* invalid
*/
-__rte_experimental
int rte_eth_fec_get_capability(uint16_t port_id,
struct rte_eth_fec_capa *speed_fec_capa,
unsigned int num);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Get current Forward Error Correction(FEC) mode.
* If link is down and AUTO is enabled, AUTO is returned, otherwise,
* configured FEC mode is returned.
@@ -4526,13 +4405,9 @@ int rte_eth_fec_get_capability(uint16_t port_id,
* - (-EIO) if device is removed.
* - (-ENODEV) if *port_id* invalid.
*/
-__rte_experimental
int rte_eth_fec_get(uint16_t port_id, uint32_t *fec_capa);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Set Forward Error Correction(FEC) mode.
*
* @param port_id
@@ -4552,7 +4427,6 @@ int rte_eth_fec_get(uint16_t port_id, uint32_t *fec_capa);
* - (-EIO) if device is removed.
* - (-ENODEV) if *port_id* invalid.
*/
-__rte_experimental
int rte_eth_fec_set(uint16_t port_id, uint32_t fec_capa);
/**
@@ -4629,9 +4503,6 @@ int rte_eth_dev_mac_addr_add(uint16_t port_id, struct rte_ether_addr *mac_addr,
uint32_t pool);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Retrieve the information for queue based PFC.
*
* @param port_id
@@ -4645,14 +4516,10 @@ int rte_eth_dev_mac_addr_add(uint16_t port_id, struct rte_ether_addr *mac_addr,
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if bad parameter.
*/
-__rte_experimental
int rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
struct rte_eth_pfc_queue_info *pfc_queue_info);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Configure the queue based priority flow control for a given queue
* for Ethernet device.
*
@@ -4672,7 +4539,6 @@ int rte_eth_dev_priority_flow_ctrl_queue_info_get(uint16_t port_id,
* - (-EINVAL) if bad parameter
* - (-EIO) if flow control setup queue failure
*/
-__rte_experimental
int rte_eth_dev_priority_flow_ctrl_queue_configure(uint16_t port_id,
struct rte_eth_pfc_queue_conf *pfc_queue_conf);
@@ -4854,9 +4720,6 @@ rte_eth_dev_rss_hash_conf_get(uint16_t port_id,
struct rte_eth_rss_conf *rss_conf);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
- *
* Get the name of RSS hash algorithm.
*
* @param rss_algo
@@ -4865,14 +4728,10 @@ rte_eth_dev_rss_hash_conf_get(uint16_t port_id,
* @return
* Hash algorithm name or 'UNKNOWN' if the rss_algo cannot be recognized.
*/
-__rte_experimental
const char *
rte_eth_dev_rss_algo_name(enum rte_eth_hash_function rss_algo);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
- *
* Get RSS hash algorithm by its name.
*
* @param name
@@ -4885,7 +4744,6 @@ rte_eth_dev_rss_algo_name(enum rte_eth_hash_function rss_algo);
* - (0) if successful.
* - (-EINVAL) if not found.
*/
-__rte_experimental
int
rte_eth_find_rss_algo(const char *name, uint32_t *algo);
@@ -5164,9 +5022,6 @@ int rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,
struct rte_eth_txq_info *qinfo);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Retrieve information about given ports's Rx queue for recycling mbufs.
*
* @param port_id
@@ -5183,7 +5038,6 @@ int rte_eth_tx_queue_info_get(uint16_t port_id, uint16_t queue_id,
* - -ENOTSUP: routine is not supported by the device PMD.
* - -EINVAL: The queue_id is out of range.
*/
-__rte_experimental
int rte_eth_recycle_rx_queue_info_get(uint16_t port_id,
uint16_t queue_id,
struct rte_eth_recycle_rxq_info *recycle_rxq_info);
@@ -5231,9 +5085,6 @@ int rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
struct rte_eth_burst_mode *mode);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Retrieve the monitor condition for a given receive queue.
*
* @param port_id
@@ -5250,7 +5101,6 @@ int rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
* -EINVAL: Invalid parameters.
* -ENODEV: Invalid port ID.
*/
-__rte_experimental
int rte_eth_get_monitor_addr(uint16_t port_id, uint16_t queue_id,
struct rte_power_monitor_cond *pmc);
@@ -5280,7 +5130,6 @@ int rte_eth_get_monitor_addr(uint16_t port_id, uint16_t queue_id,
* - (-EIO) if device is removed.
* - others depends on the specific operations implementation.
*/
-__rte_experimental
int rte_eth_dev_get_reg_info_ext(uint16_t port_id, struct rte_dev_reg_info *info);
/**
@@ -5355,9 +5204,6 @@ int rte_eth_dev_get_eeprom(uint16_t port_id, struct rte_dev_eeprom_info *info);
int rte_eth_dev_set_eeprom(uint16_t port_id, struct rte_dev_eeprom_info *info);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Retrieve the type and size of plugin module EEPROM
*
* @param port_id
@@ -5372,15 +5218,11 @@ int rte_eth_dev_set_eeprom(uint16_t port_id, struct rte_dev_eeprom_info *info);
* - (-EIO) if device is removed.
* - others depends on the specific operations implementation.
*/
-__rte_experimental
int
rte_eth_dev_get_module_info(uint16_t port_id, struct rte_eth_dev_module_info *modinfo)
__rte_warn_unused_result;
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Retrieve the data of plugin module EEPROM
*
* @param port_id
@@ -5396,7 +5238,6 @@ rte_eth_dev_get_module_info(uint16_t port_id, struct rte_eth_dev_module_info *mo
* - (-EIO) if device is removed.
* - others depends on the specific operations implementation.
*/
-__rte_experimental
int
rte_eth_dev_get_module_eeprom(uint16_t port_id, struct rte_dev_eeprom_info *info)
__rte_warn_unused_result;
@@ -5550,7 +5391,6 @@ int rte_eth_timesync_adjust_time(uint16_t port_id, int64_t delta);
* - -EIO: if device is removed.
* - -ENOTSUP: The function is not supported by the Ethernet driver.
*/
-__rte_experimental
int rte_eth_timesync_adjust_freq(uint16_t port_id, int64_t ppm);
/**
@@ -5591,9 +5431,6 @@ int rte_eth_timesync_read_time(uint16_t port_id, struct timespec *time);
int rte_eth_timesync_write_time(uint16_t port_id, const struct timespec *time);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice.
- *
* Read the current clock counter of an Ethernet device
*
* This returns the current raw clock value of an Ethernet device. It is
@@ -5635,7 +5472,6 @@ int rte_eth_timesync_write_time(uint16_t port_id, const struct timespec *time);
* - -ENOTSUP: The function is not supported by the Ethernet driver.
* - -EINVAL: if bad parameter.
*/
-__rte_experimental
int
rte_eth_read_clock(uint16_t port_id, uint64_t *clock);
@@ -5726,9 +5562,6 @@ void *
rte_eth_dev_get_sec_ctx(uint16_t port_id);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Query the device hairpin capabilities.
*
* @param port_id
@@ -5740,14 +5573,10 @@ rte_eth_dev_get_sec_ctx(uint16_t port_id);
* - (-ENOTSUP) if hardware doesn't support.
* - (-EINVAL) if bad parameter.
*/
-__rte_experimental
int rte_eth_dev_hairpin_capability_get(uint16_t port_id,
struct rte_eth_hairpin_cap *cap);
/**
- * @warning
- * @b EXPERIMENTAL: this structure may change without prior notice.
- *
* Ethernet device representor ID range entry
*/
struct rte_eth_representor_range {
@@ -5765,9 +5594,6 @@ struct rte_eth_representor_range {
};
/**
- * @warning
- * @b EXPERIMENTAL: this structure may change without prior notice.
- *
* Ethernet device representor information
*/
struct rte_eth_representor_info {
@@ -5801,7 +5627,6 @@ struct rte_eth_representor_info {
* - (-EIO) if device is removed.
* - (>=0) number of available representor range entries.
*/
-__rte_experimental
int rte_eth_representor_info_get(uint16_t port_id,
struct rte_eth_representor_info *info);
@@ -5881,9 +5706,6 @@ struct rte_eth_ip_reassembly_params {
};
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice
- *
* Get IP reassembly capabilities supported by the PMD. This is the first API
* to be called for enabling the IP reassembly offload feature. PMD will return
* the maximum values of parameters that PMD can support and user can call
@@ -5900,14 +5722,10 @@ struct rte_eth_ip_reassembly_params {
* - (-EINVAL) if device is not configured or *capa* passed is NULL.
* - (0) on success.
*/
-__rte_experimental
int rte_eth_ip_reassembly_capability_get(uint16_t port_id,
struct rte_eth_ip_reassembly_params *capa);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice
- *
* Get IP reassembly configuration parameters currently set in PMD.
* The API will return error if the configuration is not already
* set using rte_eth_ip_reassembly_conf_set() before calling this API or if
@@ -5925,14 +5743,10 @@ int rte_eth_ip_reassembly_capability_get(uint16_t port_id,
* configuration is not set using rte_eth_ip_reassembly_conf_set().
* - (0) on success.
*/
-__rte_experimental
int rte_eth_ip_reassembly_conf_get(uint16_t port_id,
struct rte_eth_ip_reassembly_params *conf);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice
- *
* Set IP reassembly configuration parameters if the PMD supports IP reassembly
* offload. User should first call rte_eth_ip_reassembly_capability_get() to
* check the maximum values supported by the PMD before setting the
@@ -5958,7 +5772,6 @@ int rte_eth_ip_reassembly_conf_get(uint16_t port_id,
* successfully by the PMD.
* - (0) on success.
*/
-__rte_experimental
int rte_eth_ip_reassembly_conf_set(uint16_t port_id,
const struct rte_eth_ip_reassembly_params *conf);
@@ -5982,9 +5795,6 @@ typedef struct {
} rte_eth_ip_reassembly_dynfield_t;
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Dump private info from device to a file. Provided data and the order depends
* on the PMD.
*
@@ -5999,13 +5809,9 @@ typedef struct {
* - (-ENOTSUP) if the device does not support this function.
* - (-EIO) if device is removed.
*/
-__rte_experimental
int rte_eth_dev_priv_dump(uint16_t port_id, FILE *file);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Dump ethdev Rx descriptor info to a file.
*
* This API is used for debugging, not a dataplane API.
@@ -6025,14 +5831,10 @@ int rte_eth_dev_priv_dump(uint16_t port_id, FILE *file);
* - On success, zero.
* - On failure, a negative value.
*/
-__rte_experimental
int rte_eth_rx_descriptor_dump(uint16_t port_id, uint16_t queue_id,
uint16_t offset, uint16_t num, FILE *file);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Dump ethdev Tx descriptor info to a file.
*
* This API is used for debugging, not a dataplane API.
@@ -6052,7 +5854,6 @@ int rte_eth_rx_descriptor_dump(uint16_t port_id, uint16_t queue_id,
* - On success, zero.
* - On failure, a negative value.
*/
-__rte_experimental
int rte_eth_tx_descriptor_dump(uint16_t port_id, uint16_t queue_id,
uint16_t offset, uint16_t num, FILE *file);
@@ -6071,9 +5872,6 @@ enum rte_eth_cman_obj {
};
/**
- * @warning
- * @b EXPERIMENTAL: this structure may change, or be removed, without prior notice
- *
* A structure used to retrieve information of ethdev congestion management.
*/
struct rte_eth_cman_info {
@@ -6095,9 +5893,6 @@ struct rte_eth_cman_info {
};
/**
- * @warning
- * @b EXPERIMENTAL: this structure may change, or be removed, without prior notice
- *
* A structure used to configure the ethdev congestion management.
*/
struct rte_eth_cman_config {
@@ -6139,9 +5934,6 @@ struct rte_eth_cman_config {
};
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Retrieve the information for ethdev congestion management
*
* @param port_id
@@ -6155,13 +5947,9 @@ struct rte_eth_cman_config {
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if bad parameter.
*/
-__rte_experimental
int rte_eth_cman_info_get(uint16_t port_id, struct rte_eth_cman_info *info);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Initialize the ethdev congestion management configuration structure with default values.
*
* @param port_id
@@ -6175,13 +5963,9 @@ int rte_eth_cman_info_get(uint16_t port_id, struct rte_eth_cman_info *info);
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if bad parameter.
*/
-__rte_experimental
int rte_eth_cman_config_init(uint16_t port_id, struct rte_eth_cman_config *config);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Configure ethdev congestion management
*
* @param port_id
@@ -6194,13 +5978,9 @@ int rte_eth_cman_config_init(uint16_t port_id, struct rte_eth_cman_config *confi
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if bad parameter.
*/
-__rte_experimental
int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *config);
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Retrieve the applied ethdev congestion management parameters for the given port.
*
* @param port_id
@@ -6217,7 +5997,6 @@ int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if bad parameter.
*/
-__rte_experimental
int rte_eth_cman_config_get(uint16_t port_id, struct rte_eth_cman_config *config);
#ifdef __cplusplus
@@ -6949,9 +6728,6 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
}
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
* Recycle used mbufs from a transmit queue of an Ethernet device, and move
* these mbufs into a mbuf ring for a receive queue of an Ethernet device.
* This can bypass mempool path to save CPU cycles.
@@ -7001,7 +6777,6 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
* @return
* The number of recycling mbufs.
*/
-__rte_experimental
static inline uint16_t
rte_eth_recycle_mbufs(uint16_t rx_port_id, uint16_t rx_queue_id,
uint16_t tx_port_id, uint16_t tx_queue_id,
@@ -7076,9 +6851,6 @@ rte_eth_recycle_mbufs(uint16_t rx_port_id, uint16_t rx_queue_id,
}
/**
- * @warning
- * @b EXPERIMENTAL: this API may change without prior notice
- *
* Get supported header protocols to split on Rx.
*
* When a packet type is announced to be split,
@@ -7103,14 +6875,10 @@ rte_eth_recycle_mbufs(uint16_t rx_port_id, uint16_t rx_queue_id,
* - (-ENODEV) if *port_id* invalid.
* - (-EINVAL) if bad parameter.
*/
-__rte_experimental
int rte_eth_buffer_split_get_supported_hdr_ptypes(uint16_t port_id, uint32_t *ptypes, int num)
__rte_warn_unused_result;
/**
- * @warning
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice.
- *
* Get the number of used descriptors of a Tx queue.
*
* This function retrieves the number of used descriptors of a transmit queue.
@@ -7141,7 +6909,6 @@ int rte_eth_buffer_split_get_supported_hdr_ptypes(uint16_t port_id, uint32_t *pt
* If the use case only involves checking the status of a specific descriptor slot,
* opt for rte_eth_tx_descriptor_status() instead.
*/
-__rte_experimental
static inline int
rte_eth_tx_queue_count(uint16_t port_id, uint16_t queue_id)
{
diff --git a/lib/ethdev/rte_ethdev_cman.c b/lib/ethdev/rte_ethdev_cman.c
index a8460e6977..3a62432bf0 100644
--- a/lib/ethdev/rte_ethdev_cman.c
+++ b/lib/ethdev/rte_ethdev_cman.c
@@ -12,7 +12,7 @@
#include "ethdev_trace.h"
/* Get congestion management information for a port */
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_cman_info_get, 22.11)
+RTE_EXPORT_SYMBOL(rte_eth_cman_info_get)
int
rte_eth_cman_info_get(uint16_t port_id, struct rte_eth_cman_info *info)
{
@@ -41,7 +41,7 @@ rte_eth_cman_info_get(uint16_t port_id, struct rte_eth_cman_info *info)
}
/* Initialize congestion management structure with default values */
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_cman_config_init, 22.11)
+RTE_EXPORT_SYMBOL(rte_eth_cman_config_init)
int
rte_eth_cman_config_init(uint16_t port_id, struct rte_eth_cman_config *config)
{
@@ -70,7 +70,7 @@ rte_eth_cman_config_init(uint16_t port_id, struct rte_eth_cman_config *config)
}
/* Configure congestion management on a port */
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_cman_config_set, 22.11)
+RTE_EXPORT_SYMBOL(rte_eth_cman_config_set)
int
rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *config)
{
@@ -98,7 +98,7 @@ rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *conf
}
/* Retrieve congestion management configuration of a port */
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_cman_config_get, 22.11)
+RTE_EXPORT_SYMBOL(rte_eth_cman_config_get)
int
rte_eth_cman_config_get(uint16_t port_id, struct rte_eth_cman_config *config)
{
--
2.53.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox