* Re: [PATCH 1/1] buildtools: remove absolute paths from pc file
From: Bruce Richardson @ 2026-06-09 8:09 UTC (permalink / raw)
To: Srikanth Yalavarthi; +Cc: dev
In-Reply-To: <aifFFS9gvRn9JYKg@bricha3-mobl1.ger.corp.intel.com>
On Tue, Jun 09, 2026 at 08:47:33AM +0100, Bruce Richardson wrote:
> On Tue, Jun 09, 2026 at 10:02:20AM +0530, Srikanth Yalavarthi wrote:
> > When linking with non-versioned libraries, absolute paths
> > of the libraries are added to libdpdk.pc. This patch replaces
> > the absolute path with correct linker flags, -l<libname>.
> >
>
> Do we need to check if we need a "-L" flag for some of these paths that are
> stripped?
>
> > https://github.com/mesonbuild/meson/issues/7766
> >
The bug report indicates that this issue only occurs for some libs which
are missing an SONAME. Checking the .pc files generated on my own system, I
don't see any libs listed with full paths. Out of interest, what libraries
are causing problems? (Alternative question - how can we reproduce this
issue to check the fix :-) )
Thanks,
/Bruce
^ permalink raw reply
* Re: [PATCH v4 5/5] eal: avoid deadlock in async IPC alarm callback
From: Burakov, Anatoly @ 2026-06-09 8:04 UTC (permalink / raw)
To: dev, Jianfeng Tan
In-Reply-To: <c5284e2f5e9fb7d664ab65115cac2634c93aa510.1780669755.git.anatoly.burakov@intel.com>
On 6/5/2026 4:29 PM, Anatoly Burakov wrote:
> async_reply_handle_thread_unsafe() can run while holding
> pending_requests.lock and currently calls rte_eal_alarm_cancel().
>
> rte_eal_alarm_cancel() may spin-wait for an executing callback, which can
> deadlock if that callback is blocked on the same lock.
>
> Remove callback-side alarm cancellation. It is safe to do so, because any
> callback triggered without a pending request becomes a noop.
>
> Fixes: daf9bfca717e ("ipc: remove thread for async requests")
> Cc: stable@dpdk.org
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
Okay, the AI review seems to keep flagging issues that are technically
true in the patches, but are intentional and do get better once the
complete patchset is applied.
Looks like I need to merge some of the patches or rethink the order in
which the fixes are applied to avoid these issues.
--
Thanks,
Anatoly
^ permalink raw reply
* Re: [PATCH v16 0/5] Support add/remove memory region and get-max-slots
From: Maxime Coquelin @ 2026-06-09 7:58 UTC (permalink / raw)
To: pravin.bathija; +Cc: dev, stephen, fengchengwen, thomas
In-Reply-To: <20260606025211.1082615-1-pravin.bathija@dell.com>
On Sat, Jun 6, 2026 at 4:52 AM <pravin.bathija@dell.com> wrote:
>
> From: Pravin M Bathija <pravin.bathija@dell.com>
>
> This is version v16 of the patchset and it incorporates the
> recommendations made by Stephen Hemminger.
>
> Patch 4/5
> - Fixed use-after-free in vhost_user_add_mem_reg(): reg pointed into
> dev->mem->regions[] but dev_invalidate_vrings() ->
> translate_ring_addresses() -> numa_realloc() can relocate dev->mem ;
> re-drive the region pointer from the refreshed dev before reading
> host_user_addr for the reply.
> - Gated the ADD_MEM_REG mapping-address reply on dev->postcopy_listening
> per the vhost-user spec (reply is postcopy-only); return
> RTE_VHOST_MSG_RESULT_OK in non-postcopy mode, matching SET_MEM_TABLE,
> to avoid desyncing the channel.
>
> This implementation has been extensively tested by doing Read/Write I/O
> from multiple instances of fio + libblkio (front-end) talking to
> spdk/dpdk (back-end) based drives. Tested with qemu front-end talking to
> dpdk testpmd (back-end) performing add/removal of memory regions. Also
> tested post-copy live migration after doing add_memory_region.
>
> Version Log:
> Version v16 (Current version): Incorporate code review suggestions from
> Stephen Hemminger.
>
> Version v15: Incorporate review suggestions from Maxime Coquelin.
> Patch 4/5
> - Changed VHOST_USER_REM_MEM_REG handler declaration from
> accepts_fd=true to accepts_fd=false, as the remove request does not
> expect FDs in ancillary data.
> - Removed all close_msg_fds(ctx) calls from vhost_user_rem_mem_reg(), no
> longer needed since the handler is declared as not accepting FDs.
> - Removed validate_msg_fds(dev, ctx, 0) check from
> vhost_user_rem_mem_reg(), as FD validation is now handled generically
> by the framework.
> - Added targeted IOTLB cache invalidation in vhost_user_rem_mem_reg()
> using vhost_user_iotlb_cache_remove() for the removed region's GPA
> range, instead of the nuclear iotlb_flush_all() used by set_mem_table.
>
> Version v14: Incorporate code review suggestions from Stephen Hemminger
> and Fengcheng Wen.
> Changes from Fengcheng Wen review:
> Patch 3/5
> - Moved free_all_mem_regions() call sites in vhost_user_set_mem_table()
> from patch 4/5 to patch 3/5 so each commit compiles independently
> Patch 4/5
> - Renamed _dev_invalidate_vrings() to vhost_user_invalidate_vrings() to
> follow vhost naming convention
> - Added comment explaining *pdev propagation through
> translate_ring_addresses / numa_realloc()
> - Reordered local variables in vhost_user_add_mem_reg() and
> vhost_user_rem_mem_reg() by descending line length
> - Shortened overlap check variable names (current_region_guest_start/end
> --> cur_start/end, proposed_region_guest_start/end -> new_start/end)
> - Fixed DMA error path in vhost_user_add_mem_reg(): added
> free_new_region_no_dma label so async_dma_map_region(false) is not
> called when the map itself failed.
> Changes from Stephen Hemminger review:
> Patch 4/5
> - vhost_user_add_mem_reg() now constructs a reply with the back-end's
> host mapping address in userspace_addr and returns
> RTE_VHOST_MSG_RESULT_REPLY per the vhost-user spec
> - Added validate_msg_fds(dev, ctx, 0) in vhost_user_rem_mem_reg() to
> reject malformed messages with unexpected file descriptors
> - Dropped unnecessary (uint64_t) cast in vhost_user_get_max_mem_slots()
>
> Version v13: Incorporate code review suggestions from Fengcheng Wen
> Patch 2/5
> Renamed VhostUserSingleMemReg to VhostUserMemRegMsg and memory_single
> to memreg
> Patches 3/5 and 4/5
> Relocated function remove_guest_pages from patch 3/5 to 4/5
>
> Version v12: Incorporate code review suggestions from Maxime Coquelin
> and ai-code-review.
> Patch 3/5
> Refactored async_dma_map() to delegate to async_dma_map_region(),
> eliminating code duplication between the two functions.
> Restored original comments in async_dma_map_region() explaining why
> ENODEV and EINVAL errors are ignored (these were stripped in v10)
> Reverted unnecessary changes to vhost_user_postcopy_register() --
> removed the host_user_addr == 0 checks and reg_msg_index indirection
> that were added in v10, since this function is only called from
> vhost_user_set_mem_table() where regions are always contiguous.
>
> Version v11: Incorporate code review suggestions from Stephen Hemminger.
> Patch 4/5
> Fix incomplete cleanup in vhost_user_add_mem_reg() when
> vhost_user_mmap_region() fails after the mmap succeeds (e.g.
> add_guest_pages() realloc failure) realloc failure). The error path now
> calls remove_guest_pages() and free_mem_region() to undo the mapping
> and stale guest-page entries, preventing a leaked mmap and slot reuse
> corruption. The plain close(fd) path is kept for pre-mmap failures.
>
> Version v10: Incorporate code review suggestions from Stephen Hemminger.
> Patch 4/5
> Moved dev_invalidate_vrings after free_mem_region, array compaction, and
> nregions decrement. This ensures translate_ring_addresses only sees
> surviving memory regions, preventing vring pointers from resolving into
> a region that is about to be unmapped.
>
> Version v9: Incorporate code review suggestions from Stephen Hemminger.
> Patch 3/5
> Restored max_guest_pages initial value to hardcoded 8 instead of
> VHOST_MEMORY_MAX_NREGIONS, matching upstream semantics.
> Patch 4/5
> Added close(reg->fd) and reg->fd = -1 before goto close_msg_fds in the
> mmap failure path to fix fd leak after fd was moved from ctx->fds[0].
> Converted dev_invalidate_vrings from a plain function to a macro +
> implementation function pair, accepting message ID as a parameter so
> the static_assert reports the correct handler at each call site.
> Updated dev_invalidate_vrings call in add_mem_reg to pass
> VHOST_USER_ADD_MEM_REG as message ID.
> Updated dev_invalidate_vrings call in rem_mem_reg to pass
> VHOST_USER_REM_MEM_REG as message ID.
>
> Version v8: Incorporate code review suggestions from Stephen Hemminger.
> rewrite async_dma_map_region function to iterate guest pages by host
> address range matching
> change function dev_invalidate_vrings to accept a double pointer to
> propagate pointer updates
> new function remove_guest_pages was added
> add_mem_reg error path was narrowed to only clean up the single failed
> region instead of destroting all existing regions
>
> Version v7: Incorporate code review suggestions from Maxime Coquelin.
> Add debug messages to vhost_postcopy_register function.
>
> Version v6: Added the enablement of this feature as a final patch in
> this patch-set and other code optimizations as suggested by Maxime
> Coquelin.
>
> Version v5: removed the patch that increased the number of memory regions
> from 8 to 128. This will be submitted as a separate feature at a later
> point after incorporating additional optimizations. Also includes code
> optimizations as suggested by Feng Cheng Wen.
>
> Version v4: code optimizations as suggested by Feng Cheng Wen.
>
> Version v3: code optimizations as suggested by Maxime Coquelin
> and Thomas Monjalon.
>
> Version v2: code optimizations as suggested by Maxime Coquelin.
>
> Version v1: Initial patch set.
>
> Pravin M Bathija (5):
> vhost: add user to mailmap and define to vhost hdr
> vhost: header defines for add/rem mem region
> vhost: refactor memory helper functions
> vhost: add mem region add/remove handlers
> vhost: enable configure memory slots
>
> .mailmap | 1 +
> lib/vhost/rte_vhost.h | 4 +
> lib/vhost/vhost_user.c | 438 +++++++++++++++++++++++++++++++++++------
> lib/vhost/vhost_user.h | 10 +
> 4 files changed, 391 insertions(+), 62 deletions(-)
>
> --
> 2.43.0
>
Applied to next-virtio/for-next-net.
Thanks,
Maxime
^ permalink raw reply
* [PATCH] net/crc: add 4x folding loop for x86 SSE implementation
From: Shreesh Adiga @ 2026-06-09 7:57 UTC (permalink / raw)
To: Jasvinder Singh, Bruce Richardson, Konstantin Ananyev; +Cc: dev
Add a 64-byte loop that maintains 4 fold registers and processes
64 bytes at a time. The 4x fold registers is then reduced to 16 byte
single fold, similar to AVX512 implementation. This technique is
described in the paper by Intel:
"Fast CRC Computation for Generic Polynomials Using PCLMULQDQ Instruction"
This results in roughly 50% performance improvement due to better ILP
for large input sizes like 1024.
Signed-off-by: Shreesh Adiga <16567adigashreesh@gmail.com>
---
lib/net/net_crc_sse.c | 59 +++++++++++++++++++++++++++++++++++--------
1 file changed, 48 insertions(+), 11 deletions(-)
diff --git a/lib/net/net_crc_sse.c b/lib/net/net_crc_sse.c
index 3b6fbfecac..dfef8ecc59 100644
--- a/lib/net/net_crc_sse.c
+++ b/lib/net/net_crc_sse.c
@@ -14,6 +14,7 @@
/** PCLMULQDQ CRC computation context structure */
struct crc_pclmulqdq_ctx {
__m128i rk1_rk2;
+ __m128i rk3_rk4;
__m128i rk5_rk6;
__m128i rk7_rk8;
};
@@ -150,9 +151,36 @@ crc32_eth_calc_pclmulqdq(
temp = _mm_insert_epi32(_mm_setzero_si128(), crc, 0);
/**
- * Folding all data into single 16 byte data block
- * Assumes: fold holds first 16 bytes of data
+ * Folding all data into 4 parallel 16 byte data block
+ * Later folds 4 parallel blocks into single fold block
*/
+ if (likely(data_len >= 64)) {
+ __m128i fold1, fold2, fold3, fold4;
+ __m128i temp1, temp2, temp3, temp4;
+ fold1 = _mm_loadu_si128((const __m128i *)(data + 0));
+ fold2 = _mm_loadu_si128((const __m128i *)(data + 16));
+ fold3 = _mm_loadu_si128((const __m128i *)(data + 32));
+ fold4 = _mm_loadu_si128((const __m128i *)(data + 48));
+ fold1 = _mm_xor_si128(fold1, temp);
+ k = params->rk1_rk2;
+
+ for (n = 64; (n + 64) <= data_len; n += 64) {
+ temp1 = _mm_loadu_si128((const __m128i *)&data[n]);
+ temp2 = _mm_loadu_si128((const __m128i *)&data[n + 16]);
+ temp3 = _mm_loadu_si128((const __m128i *)&data[n + 32]);
+ temp4 = _mm_loadu_si128((const __m128i *)&data[n + 48]);
+ fold1 = crcr32_folding_round(temp1, k, fold1);
+ fold2 = crcr32_folding_round(temp2, k, fold2);
+ fold3 = crcr32_folding_round(temp3, k, fold3);
+ fold4 = crcr32_folding_round(temp4, k, fold4);
+ }
+
+ k = params->rk3_rk4;
+ fold1 = crcr32_folding_round(fold2, k, fold1);
+ fold1 = crcr32_folding_round(fold3, k, fold1);
+ fold = crcr32_folding_round(fold4, k, fold1);
+ goto single_fold_loop;
+ }
if (unlikely(data_len < 32)) {
if (unlikely(data_len == 16)) {
@@ -182,7 +210,7 @@ crc32_eth_calc_pclmulqdq(
fold = _mm_loadu_si128((const __m128i *)data);
fold = _mm_xor_si128(fold, temp);
n = 16;
- k = params->rk1_rk2;
+ k = params->rk3_rk4;
goto partial_bytes;
}
@@ -191,9 +219,12 @@ crc32_eth_calc_pclmulqdq(
fold = _mm_loadu_si128((const __m128i *)data);
fold = _mm_xor_si128(fold, temp);
- /** Main folding loop - the last 16 bytes is processed separately */
- k = params->rk1_rk2;
- for (n = 16; (n + 16) <= data_len; n += 16) {
+ /** Single folding loop - the last 16 bytes is processed separately */
+ k = params->rk3_rk4;
+ n = 16;
+
+single_fold_loop:
+ for (; (n + 16) <= data_len; n += 16) {
temp = _mm_loadu_si128((const __m128i *)&data[n]);
fold = crcr32_folding_round(temp, k, fold);
}
@@ -236,12 +267,14 @@ crc32_eth_calc_pclmulqdq(
void
rte_net_crc_sse42_init(void)
{
- uint64_t k1, k2, k5, k6;
+ uint64_t k1, k2, k3, k4, k5, k6;
uint64_t p = 0, q = 0;
/** Initialize CRC16 data */
- k1 = 0x189aeLLU;
- k2 = 0x8e10LLU;
+ k1 = 0x14ff2LLU;
+ k2 = 0x19a3cLLU;
+ k3 = 0x189aeLLU;
+ k4 = 0x8e10LLU;
k5 = 0x189aeLLU;
k6 = 0x114aaLLU;
q = 0x11c581910LLU;
@@ -249,12 +282,15 @@ rte_net_crc_sse42_init(void)
/** Save the params in context structure */
crc16_ccitt_pclmulqdq.rk1_rk2 = _mm_set_epi64x(k2, k1);
+ crc16_ccitt_pclmulqdq.rk3_rk4 = _mm_set_epi64x(k4, k3);
crc16_ccitt_pclmulqdq.rk5_rk6 = _mm_set_epi64x(k6, k5);
crc16_ccitt_pclmulqdq.rk7_rk8 = _mm_set_epi64x(p, q);
/** Initialize CRC32 data */
- k1 = 0xccaa009eLLU;
- k2 = 0x1751997d0LLU;
+ k1 = 0x1c6e41596LLU;
+ k2 = 0x154442bd4LLU;
+ k3 = 0xccaa009eLLU;
+ k4 = 0x1751997d0LLU;
k5 = 0xccaa009eLLU;
k6 = 0x163cd6124LLU;
q = 0x1f7011640LLU;
@@ -262,6 +298,7 @@ rte_net_crc_sse42_init(void)
/** Save the params in context structure */
crc32_eth_pclmulqdq.rk1_rk2 = _mm_set_epi64x(k2, k1);
+ crc32_eth_pclmulqdq.rk3_rk4 = _mm_set_epi64x(k4, k3);
crc32_eth_pclmulqdq.rk5_rk6 = _mm_set_epi64x(k6, k5);
crc32_eth_pclmulqdq.rk7_rk8 = _mm_set_epi64x(p, q);
}
--
2.53.0
^ permalink raw reply related
* [PATCH v1 1/1] net/nbl: fix multicast reception in promiscuous mode
From: Dimon Zhao @ 2026-06-09 7:51 UTC (permalink / raw)
To: dev; +Cc: Dimon Zhao, stable, Leon Yu, Sam Chen
In-Reply-To: <20260609075143.32695-1-dimon.zhao@nebula-matrix.com>
When promiscuous mode is enabled on NBL PMD,
the hardware does not forward multicast frames to the host,
causing the driver to fail receiving multicast packets.
This patch fixes the issue.
Fixes: 80bd3cad22c8 ("net/nbl: support promiscuous mode")
Cc: stable@dpdk.org
Signed-off-by: Dimon Zhao <dimon.zhao@nebula-matrix.com>
---
drivers/net/nbl/nbl_dev/nbl_dev.c | 5 +++
drivers/net/nbl/nbl_dispatch.c | 45 +++++++++++++++++++
drivers/net/nbl/nbl_include/nbl_def_channel.h | 5 +++
.../net/nbl/nbl_include/nbl_def_resource.h | 2 +
4 files changed, 57 insertions(+)
diff --git a/drivers/net/nbl/nbl_dev/nbl_dev.c b/drivers/net/nbl/nbl_dev/nbl_dev.c
index 35485ea691..3c0e98d687 100644
--- a/drivers/net/nbl/nbl_dev/nbl_dev.c
+++ b/drivers/net/nbl/nbl_dev/nbl_dev.c
@@ -784,6 +784,8 @@ int nbl_promiscuous_enable(struct rte_eth_dev *eth_dev)
struct nbl_common_info *common = &adapter->common;
if (!common->is_vf) {
+ disp_ops->cfg_multi_mcast(NBL_DEV_MGT_TO_DISP_PRIV(dev_mgt),
+ dev_mgt->net_dev->vsi_id, 1);
disp_ops->set_promisc_mode(NBL_DEV_MGT_TO_DISP_PRIV(dev_mgt),
dev_mgt->net_dev->vsi_id, 1);
dev_mgt->net_dev->promisc = 1;
@@ -802,6 +804,8 @@ int nbl_promiscuous_disable(struct rte_eth_dev *eth_dev)
struct nbl_common_info *common = &adapter->common;
if (!common->is_vf) {
+ disp_ops->cfg_multi_mcast(NBL_DEV_MGT_TO_DISP_PRIV(dev_mgt),
+ dev_mgt->net_dev->vsi_id, 0);
disp_ops->set_promisc_mode(NBL_DEV_MGT_TO_DISP_PRIV(dev_mgt),
dev_mgt->net_dev->vsi_id, 0);
dev_mgt->net_dev->promisc = 0;
@@ -1012,6 +1016,7 @@ static void nbl_dev_leonis_stop(void *p)
disp_ops->del_multi_rule(NBL_DEV_MGT_TO_DISP_PRIV(dev_mgt), net_dev->vsi_id);
disp_ops->del_macvlan(NBL_DEV_MGT_TO_DISP_PRIV(dev_mgt), mac, 0, net_dev->vsi_id);
}
+ disp_ops->cfg_multi_mcast(NBL_DEV_MGT_TO_DISP_PRIV(dev_mgt), net_dev->vsi_id, 0);
}
static void nbl_dev_remove_ops(struct nbl_dev_ops_tbl **dev_ops_tbl)
diff --git a/drivers/net/nbl/nbl_dispatch.c b/drivers/net/nbl/nbl_dispatch.c
index 52d37ba7fe..f4f1ee4842 100644
--- a/drivers/net/nbl/nbl_dispatch.c
+++ b/drivers/net/nbl/nbl_dispatch.c
@@ -698,6 +698,47 @@ static void nbl_disp_chan_del_multi_rule_req(void *priv, u16 vsi)
chan_ops->send_msg(NBL_DISP_MGT_TO_CHAN_PRIV(disp_mgt), &chan_send);
}
+static int nbl_disp_cfg_multi_mcast(void *priv, u16 vsi, u16 enable)
+{
+ struct nbl_dispatch_mgt *disp_mgt = (struct nbl_dispatch_mgt *)priv;
+ struct nbl_resource_ops *res_ops = NBL_DISP_MGT_TO_RES_OPS(disp_mgt);
+ int ret = 0;
+
+ if (enable)
+ ret = NBL_OPS_CALL(res_ops->add_multi_mcast,
+ (NBL_DISP_MGT_TO_RES_PRIV(disp_mgt), vsi));
+ else
+ NBL_OPS_CALL(res_ops->del_multi_mcast, (NBL_DISP_MGT_TO_RES_PRIV(disp_mgt), vsi));
+
+ return ret;
+}
+
+static int nbl_disp_chan_cfg_multi_mcast_req(void *priv, u16 vsi, u16 enable)
+{
+ struct nbl_dispatch_mgt *disp_mgt = (struct nbl_dispatch_mgt *)priv;
+ const struct nbl_channel_ops *chan_ops = NBL_DISP_MGT_TO_CHAN_OPS(disp_mgt);
+ struct nbl_common_info *common = NBL_DISP_MGT_TO_COMMON(disp_mgt);
+ struct nbl_chan_param_cfg_multi_mcast param = {0};
+ struct nbl_chan_send_info chan_send;
+ int ret = 0;
+
+ if (NBL_IS_COEXISTENCE(common)) {
+ ret = ioctl(common->devfd, NBL_DEV_USER_SET_MCAST_MODE, &enable);
+ if (ret) {
+ NBL_LOG(ERR, "userspace send cfg_multi_mcast ioctl msg failed ret %d",
+ ret);
+ return ret;
+ }
+ return 0;
+ }
+
+ param.vsi = vsi;
+ param.enable = enable;
+ NBL_CHAN_SEND(chan_send, 0, NBL_CHAN_MSG_CFG_MULTI_MCAST_RULE,
+ ¶m, sizeof(param), NULL, 0, 1);
+ return chan_ops->send_msg(NBL_DISP_MGT_TO_CHAN_PRIV(disp_mgt), &chan_send);
+}
+
static int nbl_disp_cfg_dsch(void *priv, u16 vsi_id, bool vld)
{
struct nbl_dispatch_mgt *disp_mgt = (struct nbl_dispatch_mgt *)priv;
@@ -1145,6 +1186,10 @@ do { \
NBL_DISP_CTRL_LVL_MGT, \
NBL_CHAN_MSG_DEL_MULTI_RULE, \
nbl_disp_chan_del_multi_rule_req, NULL); \
+ NBL_DISP_SET_OPS(cfg_multi_mcast, nbl_disp_cfg_multi_mcast, \
+ NBL_DISP_CTRL_LVL_MGT, \
+ NBL_CHAN_MSG_CFG_MULTI_MCAST_RULE, \
+ nbl_disp_chan_cfg_multi_mcast_req, NULL); \
NBL_DISP_SET_OPS(cfg_dsch, nbl_disp_cfg_dsch, \
NBL_DISP_CTRL_LVL_MGT, NBL_CHAN_MSG_CFG_DSCH, \
nbl_disp_chan_cfg_dsch_req, NULL); \
diff --git a/drivers/net/nbl/nbl_include/nbl_def_channel.h b/drivers/net/nbl/nbl_include/nbl_def_channel.h
index 55880737f1..257c81e01e 100644
--- a/drivers/net/nbl/nbl_include/nbl_def_channel.h
+++ b/drivers/net/nbl/nbl_include/nbl_def_channel.h
@@ -371,6 +371,11 @@ struct nbl_chan_param_del_multi_rule {
u16 vsi;
};
+struct nbl_chan_param_cfg_multi_mcast {
+ u16 vsi;
+ u16 enable;
+};
+
struct nbl_chan_param_cfg_dsch {
u16 vsi_id;
bool vld;
diff --git a/drivers/net/nbl/nbl_include/nbl_def_resource.h b/drivers/net/nbl/nbl_include/nbl_def_resource.h
index 6935598789..a803c59dc0 100644
--- a/drivers/net/nbl/nbl_include/nbl_def_resource.h
+++ b/drivers/net/nbl/nbl_include/nbl_def_resource.h
@@ -77,6 +77,8 @@ struct nbl_resource_ops {
int (*add_multi_rule)(void *priv, u16 vsi_id);
void (*del_multi_rule)(void *priv, u16 vsi_id);
int (*cfg_multi_mcast)(void *priv, u16 vsi_id, u16 enable);
+ int (*add_multi_mcast)(void *priv, u16 vsi);
+ void (*del_multi_mcast)(void *priv, u16 vsi);
void (*clear_flow)(void *priv, u16 vsi_id);
int (*cfg_dsch)(void *priv, u16 vsi_id, bool vld);
int (*setup_cqs)(void *priv, u16 vsi_id, u16 real_qps, bool rss_indir_set);
--
2.34.1
^ permalink raw reply related
* [PATCH v1 0/1] fix NBL PMD multicast reception in promiscuous mode
From: Dimon Zhao @ 2026-06-09 7:51 UTC (permalink / raw)
To: dev; +Cc: Dimon Zhao
When promiscuous mode is enabled on NBL PMD,
the hardware does not forward multicast frames to the host,
causing the driver to fail receiving multicast packets.
This patch fixes the issue.
Dimon Zhao (1):
net/nbl: fix multicast reception in promiscuous mode
drivers/net/nbl/nbl_dev/nbl_dev.c | 5 +++
drivers/net/nbl/nbl_dispatch.c | 45 +++++++++++++++++++
drivers/net/nbl/nbl_include/nbl_def_channel.h | 5 +++
.../net/nbl/nbl_include/nbl_def_resource.h | 2 +
4 files changed, 57 insertions(+)
--
2.34.1
^ permalink raw reply
* Re: [PATCH 1/1] buildtools: remove absolute paths from pc file
From: Bruce Richardson @ 2026-06-09 7:47 UTC (permalink / raw)
To: Srikanth Yalavarthi; +Cc: dev
In-Reply-To: <20260609043220.930405-1-syalavarthi@marvell.com>
On Tue, Jun 09, 2026 at 10:02:20AM +0530, Srikanth Yalavarthi wrote:
> When linking with non-versioned libraries, absolute paths
> of the libraries are added to libdpdk.pc. This patch replaces
> the absolute path with correct linker flags, -l<libname>.
>
Do we need to check if we need a "-L" flag for some of these paths that are
stripped?
> https://github.com/mesonbuild/meson/issues/7766
>
> Signed-off-by: Srikanth Yalavarthi <syalavarthi@marvell.com>
> ---
> buildtools/pkg-config/set-static-linker-flags.py | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/buildtools/pkg-config/set-static-linker-flags.py b/buildtools/pkg-config/set-static-linker-flags.py
> index 2745db34c29..bd217b07fc8 100644
> --- a/buildtools/pkg-config/set-static-linker-flags.py
> +++ b/buildtools/pkg-config/set-static-linker-flags.py
> @@ -6,9 +6,16 @@
> # Should be called from meson build itself
> import os
> import sys
> +import re
>
>
> def fix_ldflag(f):
> + if f.startswith('/'):
> + libfile = os.path.basename(f)
> + m = re.match(r'^lib(.+?)\.(a|so(?:\..*)?)$', libfile)
> + if m:
> + return '-l' + m.group(1)
> +
> if not f.startswith('-lrte_'):
> return f
> return '-l:lib' + f[2:] + '.a'
> --
> 2.34.1
>
^ permalink raw reply
* [DPDK/ethdev Bug 1953] net/iavf: TX VLAN insertion offload has no effect on E810 in Double-VLAN-Mode (kernel iavf inserts fine on same PF)
From: bugzilla @ 2026-06-09 7:49 UTC (permalink / raw)
To: dev
http://bugs.dpdk.org/show_bug.cgi?id=1953
Bug ID: 1953
Summary: net/iavf: TX VLAN insertion offload has no effect on
E810 in Double-VLAN-Mode (kernel iavf inserts fine on
same PF)
Product: DPDK
Version: 26.03
Hardware: x86
OS: Linux
Status: UNCONFIRMED
Severity: critical
Priority: Normal
Component: ethdev
Assignee: dev@dpdk.org
Reporter: g.navinsrinivas@gmail.com
Target Milestone: ---
Created attachment 354
--> http://bugs.dpdk.org/attachment.cgi?id=354&action=edit
The logs, pcap for both DPDK 25.11 and 26.03 and behaviour difference with
Kernel vlan tagging
On an Intel E810 with the PF in Double‑VLAN‑Mode (DVM), an iavf VF bound to
vfio-pci does not insert a TX VLAN tag — frames egress untagged despite
tx_offload vlan_insert on + tx_vlan set (and with
insertion enabled / insertion_cap: L2TAG2). The in-tree kernel iavf on the same
PF / adjacent VF inserts the single C‑VLAN correctly (802.1Q on the wire).
Forcing inner/L2TAG1 in the PMD doesn't help
either. Reproduced with vanilla dpdk-testpmd on both DPDK 25.11 and 26.03.
Env: E810 fw 7.9.1 / nvm 4.90 / DDP "ICE OS Default Package 1.3.49.0"; host ice
on kernel 6.8.12‑pve; guest iavf 6.17. VF VLAN_V2 caps: ins_outer=0xc0000207
(TOGGLE|XOR|L2TAG2|{8100,88A8,9100}),
ins_inner=0x80000101 (TOGGLE|L2TAG1|8100), VF_OFFLOAD_VLAN(v1)=0. Logs/pcaps
attached (per-version + kernel contrast).
On an Intel E810 with the PF in Double-VLAN-Mode (DVM), an iavf VF bound to
vfio-pci does not insert a TX VLAN tag — frames egress untagged despite
tx_offload vlan_insert on + tx_vlan set (and with insertion enabled /
insertion_cap: L2TAG2). The in-tree kernel iavf on the same PF / adjacent VF
inserts the single C-VLAN correctly (802.1Q on the wire). Forcing inner/L2TAG1
in the PMD doesn't help either. Reproduced with vanilla dpdk-testpmd on both
DPDK 25.11 and 26.03.
Reproduction (testpmd):
dpdk-testpmd -a <vf_pci> --file-prefix=tp -l <cores> -- -i
--eth-peer=0,<peer_mac>
port stop 0
port config 0 tx_offload vlan_insert on
tx_vlan set 0 20
port start 0
set fwd txonly
start
- tcpdump on the peer: frames arrive as plain ethertype IPv4 — no 802.1Q / vlan
20.
- show port xstats: tx_good counts up (frames do leave the VF; no Tx error).
Expected: egress frames carry 802.1Q vlan 20.
Actual: egress frames are untagged; tx_good increments, no Tx error reported.
Ruled out: VF was trust on, spoofchk off, and the VLAN filter added via
rte_eth_dev_vlan_filter() (vlan offload: filter on) — no change. Same result on
25.11 and 26.03, and with the PMD forced to inner/L2TAG1. The kernel iavf on
the same PF (adjacent VF) inserts the tag correctly, so the PF/HW and VF config
are fine — only the DPDK iavf PMD insertion path produces no tag.
Environment: E810 fw 7.9.1 / nvm 4.90 / DDP "ICE OS Default Package 1.3.49.0";
host ice on kernel 6.8.12-pve; guest iavf 6.17. VF VLAN_V2 caps:
ins_outer=0xc0000207 (TOGGLE|XOR|L2TAG2|{8100,88A8,9100}), ins_inner=0x80000101
(TOGGLE|L2TAG1|8100), VF_OFFLOAD_VLAN(v1)=0. Logs/pcaps attached (per-version +
kernel contrast).
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply
* [PATCH v2 1/2] app/crypto-perf: support ML KEM
From: Pratik Senapati @ 2026-06-09 5:53 UTC (permalink / raw)
To: dev; +Cc: gakhil, kai.ji
Add ML-KEM512 support to test-crypto-perf.
Signed-off-by: Pratik Senapati <psenapati@marvell.com>
---
app/test-crypto-perf/cperf_ops.c | 61 +++
app/test-crypto-perf/cperf_options.h | 2 +
app/test-crypto-perf/cperf_options_parsing.c | 20 +
app/test-crypto-perf/cperf_test_common.c | 3 +-
app/test-crypto-perf/cperf_test_vectors.c | 470 +++++++++++++++++++
app/test-crypto-perf/cperf_test_vectors.h | 27 ++
app/test-crypto-perf/main.c | 21 +-
doc/guides/tools/cryptoperf.rst | 1 +
8 files changed, 603 insertions(+), 2 deletions(-)
diff --git a/app/test-crypto-perf/cperf_ops.c b/app/test-crypto-perf/cperf_ops.c
index 9297b4a3d3..806265f7bf 100644
--- a/app/test-crypto-perf/cperf_ops.c
+++ b/app/test-crypto-perf/cperf_ops.c
@@ -208,6 +208,49 @@ cperf_set_ops_asym_sm2(struct rte_crypto_op **ops,
}
}
+static void
+cperf_set_ops_asym_mlkem(struct rte_crypto_op **ops,
+ uint32_t src_buf_offset __rte_unused,
+ uint32_t dst_buf_offset __rte_unused, uint16_t nb_ops,
+ void *sess,
+ const struct cperf_options *options,
+ const struct cperf_test_vector *test_vector __rte_unused,
+ uint16_t iv_offset __rte_unused,
+ uint32_t *imix_idx __rte_unused,
+ uint64_t *tsc_start __rte_unused)
+{
+ uint16_t i;
+
+ for (i = 0; i < nb_ops; i++) {
+ struct rte_crypto_asym_op *asym_op = ops[i]->asym;
+
+ ops[i]->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
+ rte_crypto_op_attach_asym_session(ops[i], sess);
+
+ if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_ENCRYPT) {
+ asym_op->mlkem.op = RTE_CRYPTO_ML_KEM_OP_ENCAP;
+ asym_op->mlkem.encap.ek.data = options->mlkem_data->ek.data;
+ asym_op->mlkem.encap.ek.length = options->mlkem_data->ek.length;
+ asym_op->mlkem.encap.cipher.data = options->mlkem_data->cipher.data;
+ asym_op->mlkem.encap.cipher.length = options->mlkem_data->cipher.length;
+ asym_op->mlkem.encap.sk.data = options->mlkem_data->sk.data;
+ asym_op->mlkem.encap.sk.length = options->mlkem_data->sk.length;
+ asym_op->mlkem.encap.message.data = options->mlkem_data->message.data;
+ asym_op->mlkem.encap.message.length = options->mlkem_data->message.length;
+ } else if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_DECRYPT) {
+ asym_op->mlkem.op = RTE_CRYPTO_ML_KEM_OP_DECAP;
+ asym_op->mlkem.decap.dk.data = options->mlkem_data->dk.data;
+ asym_op->mlkem.decap.dk.length = options->mlkem_data->dk.length;
+ asym_op->mlkem.decap.cipher.data = options->mlkem_data->cipher.data;
+ asym_op->mlkem.decap.cipher.length = options->mlkem_data->cipher.length;
+ asym_op->mlkem.decap.sk.data = options->mlkem_data->sk.data;
+ asym_op->mlkem.decap.sk.length = options->mlkem_data->sk.length;
+ } else {
+ rte_panic("Unsupported ML-KEM operation type %d\n", options->asym_op_type);
+ }
+ }
+}
+
#ifdef RTE_LIB_SECURITY
static void
@@ -1226,6 +1269,21 @@ cperf_create_session(struct rte_mempool *sess_mp,
return asym_sess;
}
+
+ if (options->op_type == CPERF_ASYM_MLKEM512) {
+ xform.next = NULL;
+ xform.xform_type = RTE_CRYPTO_ASYM_XFORM_ML_KEM;
+ xform.mlkem.type = RTE_CRYPTO_ML_KEM_512;
+
+ ret = rte_cryptodev_asym_session_create(dev_id, &xform, sess_mp, &asym_sess);
+ if (ret < 0) {
+ RTE_LOG(ERR, USER1, "ML-KEM Asym session create failed\n");
+ return NULL;
+ }
+ return asym_sess;
+ }
+
+
#ifdef RTE_LIB_SECURITY
/*
* security only
@@ -1542,6 +1600,9 @@ cperf_get_op_functions(const struct cperf_options *options,
case CPERF_ASYM_SM2:
op_fns->populate_ops = cperf_set_ops_asym_sm2;
break;
+ case CPERF_ASYM_MLKEM512:
+ op_fns->populate_ops = cperf_set_ops_asym_mlkem;
+ break;
#ifdef RTE_LIB_SECURITY
case CPERF_PDCP:
case CPERF_DOCSIS:
diff --git a/app/test-crypto-perf/cperf_options.h b/app/test-crypto-perf/cperf_options.h
index 428e8cb4f1..98b8eeec3e 100644
--- a/app/test-crypto-perf/cperf_options.h
+++ b/app/test-crypto-perf/cperf_options.h
@@ -101,6 +101,7 @@ enum cperf_op_type {
CPERF_ASYM_SECP521R1,
CPERF_ASYM_ED25519,
CPERF_ASYM_SM2,
+ CPERF_ASYM_MLKEM512,
CPERF_TLS,
};
@@ -187,6 +188,7 @@ struct cperf_options {
struct cperf_ecdsa_test_data *secp521r1_data;
struct cperf_eddsa_test_data *eddsa_data;
struct cperf_sm2_test_data *sm2_data;
+ struct cperf_mlkem_test_data *mlkem_data;
enum rte_crypto_asym_op_type asym_op_type;
enum rte_crypto_auth_algorithm asym_hash_alg;
struct cperf_rsa_test_data *rsa_data;
diff --git a/app/test-crypto-perf/cperf_options_parsing.c b/app/test-crypto-perf/cperf_options_parsing.c
index 14e731586b..34afa938c8 100644
--- a/app/test-crypto-perf/cperf_options_parsing.c
+++ b/app/test-crypto-perf/cperf_options_parsing.c
@@ -41,6 +41,7 @@ usage(char *progname)
" --optype cipher-only / auth-only / cipher-then-auth / auth-then-cipher /\n"
" aead / pdcp / docsis / ipsec / modex / rsa / secp192r1 /\n"
" secp224r1 / secp256r1 / secp384r1 / secp521r1 / eddsa / sm2 /\n"
+ " mlkem_512 /\n"
" tls-record : set operation type\n"
" --sessionless: enable session-less crypto operations\n"
" --shared-session: share 1 session across all queue pairs on crypto device\n"
@@ -559,6 +560,10 @@ parse_op_type(struct cperf_options *opts, const char *arg)
cperf_op_type_strs[CPERF_ASYM_SM2],
CPERF_ASYM_SM2
},
+ {
+ cperf_op_type_strs[CPERF_ASYM_MLKEM512],
+ CPERF_ASYM_MLKEM512
+ },
{
cperf_op_type_strs[CPERF_TLS],
CPERF_TLS
@@ -1174,6 +1179,7 @@ cperf_options_default(struct cperf_options *opts)
opts->secp521r1_data = &secp521r1_perf_data;
opts->eddsa_data = &ed25519_perf_data;
opts->sm2_data = &sm2_perf_data;
+ opts->mlkem_data = &mlkem_encap_perf_data[0];
opts->asym_op_type = RTE_CRYPTO_ASYM_OP_ENCRYPT;
}
@@ -1669,6 +1675,18 @@ cperf_options_check(struct cperf_options *options)
}
}
+ if (options->op_type == CPERF_ASYM_MLKEM512) {
+ if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_ENCRYPT)
+ options->mlkem_data = &mlkem_encap_perf_data[0];
+ else if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_DECRYPT)
+ options->mlkem_data = &mlkem_decap_perf_data[0];
+ else {
+ RTE_LOG(ERR, USER1,
+ "ML-KEM operations only support encrypt (encapsulate) and decrypt (decapsulate)\n");
+ return -EINVAL;
+ }
+ }
+
#ifdef RTE_LIB_SECURITY
if (options->op_type == CPERF_DOCSIS) {
if (check_docsis_buffer_length(options) < 0)
@@ -1741,6 +1759,8 @@ cperf_options_dump(struct cperf_options *opts)
rte_crypto_asym_op_strings[opts->asym_op_type]);
if (opts->op_type == CPERF_ASYM_RSA)
printf("# rsa test name: %s\n", opts->rsa_data->name);
+ if (opts->op_type == CPERF_ASYM_MLKEM512)
+ printf("# mlkem test name: %s\n", opts->mlkem_data->name);
}
printf("# sessionless: %s\n", opts->sessionless ? "yes" : "no");
printf("# shared session: %s\n", opts->shared_session ? "yes" : "no");
diff --git a/app/test-crypto-perf/cperf_test_common.c b/app/test-crypto-perf/cperf_test_common.c
index caf429b4d9..383d4bd940 100644
--- a/app/test-crypto-perf/cperf_test_common.c
+++ b/app/test-crypto-perf/cperf_test_common.c
@@ -313,7 +313,8 @@ cperf_is_asym_test(const struct cperf_options *options)
options->op_type == CPERF_ASYM_SECP384R1 ||
options->op_type == CPERF_ASYM_SECP521R1 ||
options->op_type == CPERF_ASYM_ED25519 ||
- options->op_type == CPERF_ASYM_SM2)
+ options->op_type == CPERF_ASYM_SM2 ||
+ options->op_type == CPERF_ASYM_MLKEM512)
return true;
return false;
diff --git a/app/test-crypto-perf/cperf_test_vectors.c b/app/test-crypto-perf/cperf_test_vectors.c
index f4f856ff69..6e2435b004 100644
--- a/app/test-crypto-perf/cperf_test_vectors.c
+++ b/app/test-crypto-perf/cperf_test_vectors.c
@@ -3510,6 +3510,430 @@ uint8_t rsa_8192_cipher[] = {
0xab, 0x45, 0xa7, 0x55, 0xf5, 0xc1, 0x33, 0x3e,
};
+uint8_t mlkem_512_dk[] = {
+ 0xd8, 0xc8, 0x4d, 0x74, 0x57, 0x6d, 0x43, 0x65,
+ 0x84, 0xe0, 0x82, 0x87, 0x2d, 0xd0, 0x0b, 0x39,
+ 0x08, 0x41, 0xc8, 0x85, 0x15, 0x26, 0xfb, 0x85,
+ 0x68, 0xe1, 0x30, 0x98, 0xf0, 0xb1, 0x3d, 0xea,
+ 0xa5, 0x4c, 0xc4, 0x59, 0x81, 0xfb, 0x33, 0xed,
+ 0x20, 0x3a, 0x7f, 0x10, 0x82, 0x3d, 0x85, 0x18,
+ 0x69, 0x56, 0x6b, 0x5b, 0x7c, 0x92, 0x0f, 0x57,
+ 0x37, 0xf2, 0x93, 0x38, 0x24, 0xd3, 0x1e, 0xe1,
+ 0x84, 0x16, 0x39, 0x9a, 0x0d, 0xbd, 0xba, 0x22,
+ 0xaa, 0xc8, 0xa1, 0xd9, 0xd8, 0x2a, 0xc7, 0xc1,
+ 0x78, 0x21, 0x01, 0x85, 0x38, 0x54, 0x39, 0x34,
+ 0x25, 0x8c, 0xce, 0xda, 0xa3, 0x0c, 0x66, 0x99,
+ 0x39, 0xf7, 0x63, 0xb6, 0x86, 0x80, 0xe0, 0xe4,
+ 0x64, 0x2a, 0xfc, 0xc2, 0xe0, 0x71, 0x6c, 0x59,
+ 0x7c, 0x9a, 0x03, 0xac, 0x6f, 0x16, 0x08, 0x64,
+ 0x3d, 0x4b, 0xa5, 0x83, 0xc5, 0x34, 0x27, 0x2a,
+ 0x20, 0x18, 0xb8, 0x2d, 0x28, 0x61, 0x2a, 0x94,
+ 0x55, 0xc7, 0xc3, 0x93, 0x91, 0x71, 0x43, 0x7f,
+ 0x74, 0xb2, 0x6b, 0x55, 0xb1, 0x9d, 0xc8, 0x08,
+ 0x53, 0xda, 0xb6, 0x19, 0x46, 0x5c, 0x20, 0xea,
+ 0x31, 0x31, 0xfc, 0x41, 0x02, 0xe6, 0x3c, 0x3e,
+ 0xd8, 0x68, 0x6f, 0xa6, 0x66, 0x1e, 0xc7, 0xc6,
+ 0x0a, 0xf8, 0x02, 0x1e, 0x1e, 0xfa, 0x9a, 0x30,
+ 0x18, 0x64, 0xdc, 0x30, 0x20, 0x9a, 0xb6, 0x6e,
+ 0x85, 0xe8, 0xb0, 0xb3, 0xe1, 0x24, 0x4e, 0xf0,
+ 0x4a, 0x9c, 0xb3, 0x0e, 0xe0, 0xd2, 0x6a, 0x6e,
+ 0xc7, 0xc3, 0x15, 0x35, 0x49, 0x77, 0x07, 0x7e,
+ 0x33, 0x7c, 0x97, 0x6e, 0xeb, 0xb3, 0x7c, 0x86,
+ 0xb2, 0xa0, 0xe3, 0x8a, 0x13, 0x81, 0x99, 0xc6,
+ 0x75, 0x41, 0xa6, 0x2a, 0x6f, 0x16, 0xe2, 0x4e,
+ 0x52, 0x14, 0x0d, 0xcb, 0x74, 0x8d, 0xb4, 0x89,
+ 0xcf, 0xa8, 0x4c, 0x57, 0xaa, 0x18, 0x7d, 0xd7,
+ 0x19, 0x5c, 0x11, 0x04, 0xc6, 0xe2, 0x2c, 0x09,
+ 0x4d, 0x78, 0x7e, 0x9e, 0x1a, 0xa2, 0xab, 0x60,
+ 0x59, 0xe4, 0x30, 0xa5, 0x6d, 0x11, 0x63, 0xf9,
+ 0xb1, 0xa0, 0x98, 0xe2, 0x81, 0x34, 0xa6, 0x57,
+ 0xf7, 0x18, 0x3e, 0x4c, 0x36, 0x02, 0x4d, 0xda,
+ 0x6e, 0xac, 0x39, 0x6c, 0x29, 0x0b, 0x8c, 0xd7,
+ 0x32, 0x8e, 0x45, 0x67, 0x36, 0x41, 0x6c, 0xab,
+ 0xbb, 0x04, 0x2e, 0xae, 0xd3, 0x9f, 0xee, 0xfa,
+ 0x5f, 0x4a, 0x04, 0x85, 0xf7, 0xfb, 0x97, 0x95,
+ 0x78, 0xc4, 0x08, 0x39, 0xb8, 0xc5, 0x17, 0x9d,
+ 0x2e, 0x6c, 0x14, 0xcd, 0xf2, 0xc1, 0xc5, 0x05,
+ 0x9d, 0x34, 0x12, 0xcd, 0x86, 0xbc, 0x9e, 0x1c,
+ 0xf9, 0x1c, 0x05, 0xc2, 0xc3, 0x29, 0xf2, 0x9d,
+ 0xf4, 0xa9, 0xcc, 0x0d, 0x38, 0xce, 0x6f, 0xb4,
+ 0x06, 0x68, 0xc0, 0xcd, 0xb2, 0xb6, 0x5d, 0xdf,
+ 0x89, 0x58, 0x99, 0x70, 0xbe, 0x37, 0xa8, 0xb2,
+ 0xc3, 0x6c, 0x2c, 0x13, 0x8c, 0xa7, 0x8a, 0x38,
+ 0x36, 0x36, 0xe0, 0x77, 0x79, 0xf6, 0x30, 0xc5,
+ 0x9c, 0x8e, 0x10, 0xc0, 0x1b, 0xb4, 0xe0, 0x74,
+ 0x36, 0x6c, 0x9f, 0x80, 0x73, 0x3e, 0x57, 0x69,
+ 0x5a, 0x79, 0x6a, 0x44, 0xb8, 0xe8, 0x25, 0xbe,
+ 0x96, 0xae, 0xd7, 0x87, 0x30, 0x7e, 0x76, 0x8e,
+ 0x1c, 0x09, 0x99, 0xd9, 0x4c, 0xb3, 0xf7, 0x45,
+ 0x92, 0xfa, 0x77, 0x9e, 0x18, 0x72, 0x1a, 0x23,
+ 0x9c, 0x4e, 0x5e, 0xd9, 0x22, 0x2e, 0xb0, 0x16,
+ 0x9d, 0x03, 0x6a, 0x0e, 0x49, 0x94, 0x97, 0xa3,
+ 0x19, 0x94, 0xa1, 0x7e, 0x9d, 0x7b, 0x87, 0xc1,
+ 0x6a, 0x12, 0x42, 0xc9, 0x22, 0xbe, 0xeb, 0x38,
+ 0xbe, 0xfa, 0x0b, 0xac, 0x25, 0xb9, 0x78, 0xec,
+ 0x33, 0xdf, 0x86, 0xcb, 0xce, 0xe1, 0xa6, 0x69,
+ 0x28, 0x39, 0xd5, 0x68, 0x73, 0xcb, 0x8a, 0xcc,
+ 0x3d, 0xfa, 0x0e, 0x79, 0x71, 0x3e, 0x44, 0x19,
+ 0x9a, 0xac, 0xf4, 0x43, 0xa9, 0xba, 0x10, 0x7e,
+ 0x49, 0xc8, 0x8f, 0xf9, 0xac, 0xd2, 0x07, 0xb1,
+ 0x4b, 0x2b, 0x15, 0x65, 0x05, 0x12, 0xc3, 0xcc,
+ 0x36, 0x69, 0x17, 0x78, 0xa3, 0x00, 0x6d, 0xb7,
+ 0xe9, 0x03, 0x78, 0xb3, 0x33, 0xe5, 0xcc, 0x22,
+ 0xc3, 0x91, 0x8f, 0x47, 0x79, 0xaf, 0x2e, 0x19,
+ 0x91, 0x05, 0xab, 0x5e, 0x3e, 0x43, 0xa4, 0x34,
+ 0x0b, 0x8c, 0x89, 0x75, 0xc0, 0x08, 0x44, 0x07,
+ 0xdd, 0x17, 0x83, 0xc6, 0x48, 0x09, 0x12, 0xe6,
+ 0x25, 0xde, 0x99, 0x00, 0x51, 0xc1, 0x65, 0x8a,
+ 0x85, 0x02, 0xa8, 0xd5, 0x72, 0x41, 0x67, 0x66,
+ 0x08, 0xe9, 0x10, 0xa3, 0xa2, 0x07, 0xc8, 0x06,
+ 0xc6, 0x0c, 0xc6, 0x47, 0xf7, 0x35, 0x20, 0x40,
+ 0x69, 0x04, 0x3c, 0xba, 0x4b, 0xd3, 0x78, 0xa2,
+ 0xa4, 0x71, 0xc7, 0x83, 0x2b, 0x69, 0xf9, 0xa2,
+ 0x8b, 0xf5, 0xa6, 0x87, 0x46, 0xb3, 0x9b, 0x7a,
+ 0x35, 0x97, 0x0b, 0xf4, 0x68, 0x15, 0x4a, 0x87,
+ 0xeb, 0x0b, 0x2b, 0xa3, 0xba, 0xcd, 0xb3, 0x28,
+ 0x80, 0x38, 0x19, 0x17, 0x1f, 0x44, 0x6f, 0x7f,
+ 0x50, 0x75, 0x93, 0x00, 0x4c, 0x36, 0x07, 0x07,
+ 0x11, 0xec, 0x91, 0xc4, 0x99, 0x5d, 0xf8, 0xe1,
+ 0x53, 0x84, 0x01, 0xca, 0x6a, 0x64, 0x8c, 0xef,
+ 0x17, 0xce, 0x81, 0xf5, 0xce, 0xaa, 0x80, 0xc1,
+ 0x85, 0xc6, 0x6b, 0xe2, 0xc9, 0x43, 0xc9, 0xf5,
+ 0xc3, 0x3a, 0xdb, 0xcf, 0xb9, 0x29, 0x64, 0x8d,
+ 0xe7, 0x98, 0x0f, 0x8b, 0xb0, 0xd2, 0x84, 0x25,
+ 0x37, 0xd1, 0x80, 0x38, 0x81, 0xb9, 0xe8, 0xb4,
+ 0xa7, 0xc8, 0x3a, 0x27, 0x77, 0x9b, 0x81, 0xbb,
+ 0x61, 0xae, 0x59, 0x06, 0x57, 0x40, 0x69, 0x11,
+ 0xb2, 0x41, 0x68, 0x2d, 0x26, 0x62, 0x93, 0x40,
+ 0x4d, 0x97, 0x1c, 0xb4, 0x63, 0x65, 0x04, 0xf8,
+ 0x8c, 0x2a, 0x01, 0xb9, 0x6d, 0x3d, 0x87, 0x30,
+ 0x28, 0x70, 0x80, 0x1d, 0xb2, 0x24, 0xb0, 0x5c,
+ 0x90, 0x95, 0xa3, 0x6c, 0xdc, 0xa6, 0x66, 0x15,
+ 0x33, 0x37, 0x8c, 0xa8, 0x9f, 0x24, 0xa3, 0x52,
+ 0x21, 0x06, 0xb6, 0xaf, 0x51, 0xc0, 0x1d, 0x42,
+ 0xa6, 0x55, 0x93, 0x6b, 0x4a, 0x32, 0x4e, 0xec,
+ 0x46, 0xbe, 0xfa, 0x4a, 0x13, 0x5a, 0xca, 0x11,
+ 0x39, 0xa1, 0x81, 0x8b, 0xc4, 0x07, 0x1b, 0x43,
+ 0x1e, 0xa0, 0x5b, 0x34, 0x2d, 0x10, 0x6c, 0xdd,
+ 0x3b, 0x18, 0x78, 0xe1, 0x0d, 0x5b, 0x61, 0x3d,
+ 0xfd, 0x51, 0x6a, 0x78, 0x88, 0x2f, 0xa7, 0x84,
+ 0xa3, 0x4f, 0x30, 0x3e, 0xb0, 0xdc, 0x96, 0xa8,
+ 0xd7, 0xbb, 0x4c, 0xa6, 0x95, 0xb9, 0x27, 0x3f,
+ 0x70, 0x46, 0xa3, 0xd8, 0x36, 0x22, 0x3b, 0x4b,
+ 0x91, 0xa4, 0xe6, 0x31, 0x0d, 0xbb, 0x27, 0x2f,
+ 0x15, 0x2c, 0xa7, 0x44, 0x19, 0xa9, 0xc1, 0xc8,
+ 0xb2, 0x77, 0x1a, 0xd5, 0x14, 0x29, 0xa1, 0x76,
+ 0xcc, 0x81, 0x88, 0x47, 0x3c, 0xd9, 0x9e, 0xba,
+ 0x99, 0x26, 0xbb, 0xf6, 0x5e, 0xbc, 0x2a, 0x8e,
+ 0xa0, 0x32, 0x32, 0x2e, 0x24, 0xa1, 0xc3, 0x07,
+ 0xa5, 0xd4, 0xbb, 0x95, 0x03, 0x29, 0xb5, 0x59,
+ 0x5a, 0x2b, 0x6c, 0x95, 0x15, 0xba, 0xb0, 0xaf,
+ 0x27, 0xa2, 0xcb, 0xef, 0x46, 0xb6, 0xb2, 0x25,
+ 0x21, 0x23, 0xf8, 0x18, 0x1c, 0xb0, 0x54, 0xad,
+ 0x48, 0x38, 0xf6, 0x33, 0xa2, 0x55, 0xc8, 0x63,
+ 0xce, 0x56, 0x34, 0x86, 0x6a, 0x61, 0x75, 0x75,
+ 0x7f, 0x75, 0x49, 0x2a, 0x57, 0x72, 0x76, 0x3e,
+ 0xa7, 0x08, 0x56, 0x28, 0x0b, 0x44, 0xa9, 0x00,
+ 0xc2, 0x88, 0x28, 0x02, 0x55, 0x33, 0x55, 0x43,
+ 0x97, 0x97, 0x42, 0xa4, 0xfe, 0xb1, 0x95, 0xee,
+ 0x9b, 0x5d, 0xa9, 0x18, 0xc1, 0x90, 0xa5, 0x65,
+ 0x20, 0x99, 0x39, 0xd2, 0xea, 0x08, 0x1c, 0xda,
+ 0xa8, 0xfa, 0xa3, 0x3b, 0x56, 0xf0, 0x59, 0x17,
+ 0x10, 0x0b, 0x45, 0x4a, 0x86, 0x88, 0x96, 0x49,
+ 0x68, 0x98, 0x26, 0x55, 0x40, 0xb7, 0x5a, 0x29,
+ 0x82, 0x93, 0xa1, 0x0e, 0x7b, 0x4b, 0xa2, 0x49,
+ 0x61, 0x2b, 0xaf, 0xb4, 0x2d, 0xa1, 0xdc, 0xae,
+ 0x79, 0x44, 0x66, 0x20, 0x72, 0xc5, 0x01, 0x36,
+ 0x10, 0xd8, 0x81, 0x36, 0xb1, 0xea, 0xb1, 0x0d,
+ 0xc2, 0x05, 0x75, 0xd8, 0xad, 0xe0, 0x73, 0xc6,
+ 0x82, 0x8c, 0x7e, 0x97, 0x2b, 0x87, 0xb5, 0xc9,
+ 0x1e, 0xac, 0xb0, 0x2a, 0x16, 0x20, 0x88, 0xb9,
+ 0x41, 0x2f, 0xc8, 0x4b, 0xce, 0xd8, 0xc4, 0xa8,
+ 0xe2, 0x68, 0x35, 0xd9, 0xd0, 0xc4, 0x7a, 0xf2,
+ 0x25, 0xca, 0xb8, 0x06, 0x42, 0xcb, 0xc9, 0x8f,
+ 0x0a, 0x60, 0xde, 0x19, 0xa8, 0xcb, 0x90, 0x85,
+ 0x74, 0x42, 0x03, 0x89, 0xf4, 0x97, 0x8e, 0x41,
+ 0xc5, 0x72, 0xe3, 0x52, 0x55, 0x81, 0x73, 0xe1,
+ 0x71, 0x22, 0xa2, 0xfb, 0x8a, 0x36, 0xea, 0xb3,
+ 0xc3, 0x33, 0x4f, 0x23, 0x55, 0x03, 0x1c, 0xa1,
+ 0x84, 0x74, 0xe1, 0x69, 0x4d, 0xb3, 0xc2, 0x1f,
+ 0xbc, 0x62, 0xeb, 0xf0, 0x50, 0x45, 0xb7, 0x83,
+ 0xd3, 0x28, 0x37, 0x46, 0x35, 0x79, 0x04, 0x38,
+ 0x16, 0xbd, 0x72, 0x2f, 0xbc, 0xf9, 0x7d, 0xb8,
+ 0x20, 0x12, 0x87, 0xa7, 0xb0, 0x82, 0x7c, 0x3e,
+ 0x4d, 0xa7, 0x18, 0x04, 0x54, 0x75, 0x19, 0x52,
+ 0x61, 0x80, 0x57, 0xbe, 0x06, 0x37, 0x9c, 0xde,
+ 0xb0, 0x93, 0x70, 0x54, 0x3f, 0x15, 0x83, 0xb2,
+ 0xd3, 0x88, 0xa6, 0xd8, 0x56, 0x9b, 0x91, 0xf3,
+ 0x00, 0x01, 0x00, 0xcd, 0xcb, 0x97, 0x16, 0x27,
+ 0xc7, 0xaf, 0xb1, 0xd5, 0xba, 0x90, 0xa1, 0x24,
+ 0x33, 0x26, 0x95, 0xba, 0xcb, 0x63, 0xa1, 0x56,
+ 0xb0, 0x23, 0x09, 0x4e, 0xa8, 0x0a, 0x75, 0x02,
+ 0xd8, 0x3b, 0xc1, 0x05, 0xa0, 0x73, 0x25, 0x26,
+ 0x3e, 0xb8, 0x5b, 0xca, 0x6a, 0xae, 0x41, 0x22,
+ 0xc5, 0x9d, 0x88, 0x3f, 0x37, 0x8c, 0x50, 0xcd,
+ 0x09, 0x32, 0x34, 0x81, 0x98, 0xc1, 0x17, 0xa5,
+ 0x62, 0x94, 0x8d, 0xcd, 0xd0, 0xc3, 0x08, 0x07,
+ 0x2f, 0x8d, 0x52, 0x7b, 0xee, 0x81, 0xc5, 0x95,
+ 0x2c, 0x8b, 0x46, 0x45, 0x5d, 0xf2, 0x62, 0x4d,
+ 0x2e, 0x22, 0x7f, 0x0d, 0xa5, 0xc8, 0xba, 0x72,
+ 0xa0, 0x8f, 0xd0, 0x21, 0x45, 0x28, 0xb9, 0xad,
+ 0xac, 0x96, 0xa6, 0xc7, 0x33, 0x6d, 0xf3, 0x17,
+ 0xcb, 0x7a, 0x32, 0x4e, 0x4c, 0x83, 0xa7, 0x15,
+ 0x55, 0x6a, 0xca, 0xc1, 0xff, 0xd8, 0x90, 0xab,
+ 0xaa, 0xa4, 0x65, 0xb0, 0x25, 0xf7, 0x47, 0x00,
+ 0x2d, 0x20, 0x19, 0x23, 0xb5, 0x02, 0x39, 0x5a,
+ 0x69, 0x1c, 0xc4, 0xbc, 0xa3, 0x80, 0xcc, 0x82,
+ 0xd8, 0x66, 0x63, 0xc5, 0x7e, 0xa3, 0x6c, 0x2a,
+ 0xb0, 0x7a, 0x02, 0xc6, 0xb3, 0xb7, 0xb0, 0x29,
+ 0x3f, 0xa3, 0x99, 0xbd, 0x28, 0xf2, 0x8e, 0xd6,
+ 0x7b, 0xb3, 0xd5, 0x24, 0x12, 0xb9, 0xbb, 0xa5,
+ 0x68, 0xba, 0x9d, 0xe3, 0x44, 0xa3, 0xeb, 0x64,
+ 0x13, 0xb8, 0x8a, 0x5e, 0x72, 0x60, 0xad, 0x95,
+ 0xcb, 0x9d, 0xfb, 0x54, 0x6e, 0x0e, 0x66, 0xb5,
+ 0xf4, 0x68, 0x25, 0x91, 0xc5, 0x1a, 0x4f, 0x71,
+ 0x61, 0x50, 0xc8, 0xcb, 0x34, 0xc5, 0xc2, 0x84,
+ 0xbc, 0x4e, 0x5f, 0x28, 0x6f, 0xbf, 0xbb, 0xc8,
+ 0xb1, 0xc4, 0x4a, 0x4b, 0xb7, 0x1a, 0x00, 0xa0,
+ 0x2d, 0xe9, 0x07, 0x39, 0x93, 0xe9, 0x81, 0x51,
+ 0xe1, 0x34, 0x83, 0xf5, 0x82, 0x9d, 0xc7, 0x47,
+ 0xf1, 0x61, 0xc8, 0x29, 0x52, 0x4d, 0x51, 0xb7,
+ 0xa1, 0xeb, 0x49, 0x50, 0x6e, 0xda, 0x72, 0x5d,
+ 0x69, 0x27, 0x83, 0xca, 0x4c, 0x3b, 0xd4, 0xcc,
+ 0x7f, 0x26, 0x44, 0x2b, 0x76, 0x41, 0xe1, 0x50,
+ 0xce, 0x0f, 0x40, 0x0d, 0x8f, 0x00, 0x6e, 0x27,
+ 0x95, 0x6e, 0x92, 0x1b, 0xaf, 0xe4, 0x9b, 0x5e,
+ 0x96, 0xec, 0xc7, 0xbf, 0x64, 0x77, 0x75, 0x6d,
+ 0xda, 0xa8, 0x9c, 0x48, 0xf3, 0x79, 0x80, 0x4e,
+ 0xed, 0x6f, 0xe0, 0x20, 0x1a, 0x2b, 0x7c, 0xa1,
+ 0xac, 0x74, 0x42, 0x49, 0xd7, 0xe7, 0x7f, 0x57,
+ 0x5a, 0xdb, 0x69, 0xb5, 0xe2, 0x7e, 0x81, 0x8d,
+ 0xa9, 0x22, 0x5c, 0xda, 0xf3, 0xad, 0xbc, 0x3e,
+ 0x0b, 0xd4, 0x9d, 0x63, 0xef, 0x40, 0x34, 0x66,
+ 0x80, 0x8c, 0x2d, 0x31, 0x13, 0xdb, 0x31, 0x03,
+ 0xdf, 0x29, 0x92, 0x4a, 0x81, 0x76, 0x4f, 0x4d,
+ 0x5a, 0x64, 0x79, 0xf2, 0x8b, 0xc8, 0x3b, 0xc9,
+ 0xf3, 0x35, 0xc7, 0x18, 0xcb, 0xb1, 0xa8, 0xd2,
+ 0x24, 0xdb, 0x4d, 0x6a, 0x6a, 0xf1, 0xeb, 0xd2,
+};
+
+uint8_t mlkem_512_ek[] = {
+ 0x28, 0x70, 0x80, 0x1d, 0xb2, 0x24, 0xb0, 0x5c,
+ 0x90, 0x95, 0xa3, 0x6c, 0xdc, 0xa6, 0x66, 0x15,
+ 0x33, 0x37, 0x8c, 0xa8, 0x9f, 0x24, 0xa3, 0x52,
+ 0x21, 0x06, 0xb6, 0xaf, 0x51, 0xc0, 0x1d, 0x42,
+ 0xa6, 0x55, 0x93, 0x6b, 0x4a, 0x32, 0x4e, 0xec,
+ 0x46, 0xbe, 0xfa, 0x4a, 0x13, 0x5a, 0xca, 0x11,
+ 0x39, 0xa1, 0x81, 0x8b, 0xc4, 0x07, 0x1b, 0x43,
+ 0x1e, 0xa0, 0x5b, 0x34, 0x2d, 0x10, 0x6c, 0xdd,
+ 0x3b, 0x18, 0x78, 0xe1, 0x0d, 0x5b, 0x61, 0x3d,
+ 0xfd, 0x51, 0x6a, 0x78, 0x88, 0x2f, 0xa7, 0x84,
+ 0xa3, 0x4f, 0x30, 0x3e, 0xb0, 0xdc, 0x96, 0xa8,
+ 0xd7, 0xbb, 0x4c, 0xa6, 0x95, 0xb9, 0x27, 0x3f,
+ 0x70, 0x46, 0xa3, 0xd8, 0x36, 0x22, 0x3b, 0x4b,
+ 0x91, 0xa4, 0xe6, 0x31, 0x0d, 0xbb, 0x27, 0x2f,
+ 0x15, 0x2c, 0xa7, 0x44, 0x19, 0xa9, 0xc1, 0xc8,
+ 0xb2, 0x77, 0x1a, 0xd5, 0x14, 0x29, 0xa1, 0x76,
+ 0xcc, 0x81, 0x88, 0x47, 0x3c, 0xd9, 0x9e, 0xba,
+ 0x99, 0x26, 0xbb, 0xf6, 0x5e, 0xbc, 0x2a, 0x8e,
+ 0xa0, 0x32, 0x32, 0x2e, 0x24, 0xa1, 0xc3, 0x07,
+ 0xa5, 0xd4, 0xbb, 0x95, 0x03, 0x29, 0xb5, 0x59,
+ 0x5a, 0x2b, 0x6c, 0x95, 0x15, 0xba, 0xb0, 0xaf,
+ 0x27, 0xa2, 0xcb, 0xef, 0x46, 0xb6, 0xb2, 0x25,
+ 0x21, 0x23, 0xf8, 0x18, 0x1c, 0xb0, 0x54, 0xad,
+ 0x48, 0x38, 0xf6, 0x33, 0xa2, 0x55, 0xc8, 0x63,
+ 0xce, 0x56, 0x34, 0x86, 0x6a, 0x61, 0x75, 0x75,
+ 0x7f, 0x75, 0x49, 0x2a, 0x57, 0x72, 0x76, 0x3e,
+ 0xa7, 0x08, 0x56, 0x28, 0x0b, 0x44, 0xa9, 0x00,
+ 0xc2, 0x88, 0x28, 0x02, 0x55, 0x33, 0x55, 0x43,
+ 0x97, 0x97, 0x42, 0xa4, 0xfe, 0xb1, 0x95, 0xee,
+ 0x9b, 0x5d, 0xa9, 0x18, 0xc1, 0x90, 0xa5, 0x65,
+ 0x20, 0x99, 0x39, 0xd2, 0xea, 0x08, 0x1c, 0xda,
+ 0xa8, 0xfa, 0xa3, 0x3b, 0x56, 0xf0, 0x59, 0x17,
+ 0x10, 0x0b, 0x45, 0x4a, 0x86, 0x88, 0x96, 0x49,
+ 0x68, 0x98, 0x26, 0x55, 0x40, 0xb7, 0x5a, 0x29,
+ 0x82, 0x93, 0xa1, 0x0e, 0x7b, 0x4b, 0xa2, 0x49,
+ 0x61, 0x2b, 0xaf, 0xb4, 0x2d, 0xa1, 0xdc, 0xae,
+ 0x79, 0x44, 0x66, 0x20, 0x72, 0xc5, 0x01, 0x36,
+ 0x10, 0xd8, 0x81, 0x36, 0xb1, 0xea, 0xb1, 0x0d,
+ 0xc2, 0x05, 0x75, 0xd8, 0xad, 0xe0, 0x73, 0xc6,
+ 0x82, 0x8c, 0x7e, 0x97, 0x2b, 0x87, 0xb5, 0xc9,
+ 0x1e, 0xac, 0xb0, 0x2a, 0x16, 0x20, 0x88, 0xb9,
+ 0x41, 0x2f, 0xc8, 0x4b, 0xce, 0xd8, 0xc4, 0xa8,
+ 0xe2, 0x68, 0x35, 0xd9, 0xd0, 0xc4, 0x7a, 0xf2,
+ 0x25, 0xca, 0xb8, 0x06, 0x42, 0xcb, 0xc9, 0x8f,
+ 0x0a, 0x60, 0xde, 0x19, 0xa8, 0xcb, 0x90, 0x85,
+ 0x74, 0x42, 0x03, 0x89, 0xf4, 0x97, 0x8e, 0x41,
+ 0xc5, 0x72, 0xe3, 0x52, 0x55, 0x81, 0x73, 0xe1,
+ 0x71, 0x22, 0xa2, 0xfb, 0x8a, 0x36, 0xea, 0xb3,
+ 0xc3, 0x33, 0x4f, 0x23, 0x55, 0x03, 0x1c, 0xa1,
+ 0x84, 0x74, 0xe1, 0x69, 0x4d, 0xb3, 0xc2, 0x1f,
+ 0xbc, 0x62, 0xeb, 0xf0, 0x50, 0x45, 0xb7, 0x83,
+ 0xd3, 0x28, 0x37, 0x46, 0x35, 0x79, 0x04, 0x38,
+ 0x16, 0xbd, 0x72, 0x2f, 0xbc, 0xf9, 0x7d, 0xb8,
+ 0x20, 0x12, 0x87, 0xa7, 0xb0, 0x82, 0x7c, 0x3e,
+ 0x4d, 0xa7, 0x18, 0x04, 0x54, 0x75, 0x19, 0x52,
+ 0x61, 0x80, 0x57, 0xbe, 0x06, 0x37, 0x9c, 0xde,
+ 0xb0, 0x93, 0x70, 0x54, 0x3f, 0x15, 0x83, 0xb2,
+ 0xd3, 0x88, 0xa6, 0xd8, 0x56, 0x9b, 0x91, 0xf3,
+ 0x00, 0x01, 0x00, 0xcd, 0xcb, 0x97, 0x16, 0x27,
+ 0xc7, 0xaf, 0xb1, 0xd5, 0xba, 0x90, 0xa1, 0x24,
+ 0x33, 0x26, 0x95, 0xba, 0xcb, 0x63, 0xa1, 0x56,
+ 0xb0, 0x23, 0x09, 0x4e, 0xa8, 0x0a, 0x75, 0x02,
+ 0xd8, 0x3b, 0xc1, 0x05, 0xa0, 0x73, 0x25, 0x26,
+ 0x3e, 0xb8, 0x5b, 0xca, 0x6a, 0xae, 0x41, 0x22,
+ 0xc5, 0x9d, 0x88, 0x3f, 0x37, 0x8c, 0x50, 0xcd,
+ 0x09, 0x32, 0x34, 0x81, 0x98, 0xc1, 0x17, 0xa5,
+ 0x62, 0x94, 0x8d, 0xcd, 0xd0, 0xc3, 0x08, 0x07,
+ 0x2f, 0x8d, 0x52, 0x7b, 0xee, 0x81, 0xc5, 0x95,
+ 0x2c, 0x8b, 0x46, 0x45, 0x5d, 0xf2, 0x62, 0x4d,
+ 0x2e, 0x22, 0x7f, 0x0d, 0xa5, 0xc8, 0xba, 0x72,
+ 0xa0, 0x8f, 0xd0, 0x21, 0x45, 0x28, 0xb9, 0xad,
+ 0xac, 0x96, 0xa6, 0xc7, 0x33, 0x6d, 0xf3, 0x17,
+ 0xcb, 0x7a, 0x32, 0x4e, 0x4c, 0x83, 0xa7, 0x15,
+ 0x55, 0x6a, 0xca, 0xc1, 0xff, 0xd8, 0x90, 0xab,
+ 0xaa, 0xa4, 0x65, 0xb0, 0x25, 0xf7, 0x47, 0x00,
+ 0x2d, 0x20, 0x19, 0x23, 0xb5, 0x02, 0x39, 0x5a,
+ 0x69, 0x1c, 0xc4, 0xbc, 0xa3, 0x80, 0xcc, 0x82,
+ 0xd8, 0x66, 0x63, 0xc5, 0x7e, 0xa3, 0x6c, 0x2a,
+ 0xb0, 0x7a, 0x02, 0xc6, 0xb3, 0xb7, 0xb0, 0x29,
+ 0x3f, 0xa3, 0x99, 0xbd, 0x28, 0xf2, 0x8e, 0xd6,
+ 0x7b, 0xb3, 0xd5, 0x24, 0x12, 0xb9, 0xbb, 0xa5,
+ 0x68, 0xba, 0x9d, 0xe3, 0x44, 0xa3, 0xeb, 0x64,
+ 0x13, 0xb8, 0x8a, 0x5e, 0x72, 0x60, 0xad, 0x95,
+ 0xcb, 0x9d, 0xfb, 0x54, 0x6e, 0x0e, 0x66, 0xb5,
+ 0xf4, 0x68, 0x25, 0x91, 0xc5, 0x1a, 0x4f, 0x71,
+ 0x61, 0x50, 0xc8, 0xcb, 0x34, 0xc5, 0xc2, 0x84,
+ 0xbc, 0x4e, 0x5f, 0x28, 0x6f, 0xbf, 0xbb, 0xc8,
+ 0xb1, 0xc4, 0x4a, 0x4b, 0xb7, 0x1a, 0x00, 0xa0,
+ 0x2d, 0xe9, 0x07, 0x39, 0x93, 0xe9, 0x81, 0x51,
+ 0xe1, 0x34, 0x83, 0xf5, 0x82, 0x9d, 0xc7, 0x47,
+ 0xf1, 0x61, 0xc8, 0x29, 0x52, 0x4d, 0x51, 0xb7,
+ 0xa1, 0xeb, 0x49, 0x50, 0x6e, 0xda, 0x72, 0x5d,
+ 0x69, 0x27, 0x83, 0xca, 0x4c, 0x3b, 0xd4, 0xcc,
+ 0x7f, 0x26, 0x44, 0x2b, 0x76, 0x41, 0xe1, 0x50,
+ 0xce, 0x0f, 0x40, 0x0d, 0x8f, 0x00, 0x6e, 0x27,
+ 0x95, 0x6e, 0x92, 0x1b, 0xaf, 0xe4, 0x9b, 0x5e,
+ 0x96, 0xec, 0xc7, 0xbf, 0x64, 0x77, 0x75, 0x6d,
+ 0xda, 0xa8, 0x9c, 0x48, 0xf3, 0x79, 0x80, 0x4e,
+ 0xed, 0x6f, 0xe0, 0x20, 0x1a, 0x2b, 0x7c, 0xa1,
+ 0xac, 0x74, 0x42, 0x49, 0xd7, 0xe7, 0x7f, 0x57,
+};
+
+uint8_t mlkem_512_message[] = {
+ 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x20,
+ 0x66, 0x72, 0x6f, 0x6d, 0x20, 0x4d, 0x4c, 0x2d,
+ 0x4b, 0x45, 0x4d, 0x20, 0x74, 0x65, 0x73, 0x74,
+ 0x20, 0x76, 0x65, 0x63, 0x74, 0x6f, 0x72, 0x73,
+ 0x2e
+};
+
+uint8_t mlkem_512_cipher[] = {
+ 0x92, 0x98, 0x11, 0x65, 0x10, 0x78, 0xE3, 0xA4,
+ 0x34, 0x94, 0xD1, 0x22, 0x56, 0xD6, 0xB1, 0xF6,
+ 0xF7, 0x51, 0x2A, 0xCA, 0x8E, 0x04, 0x1D, 0x89,
+ 0xBB, 0xA7, 0x84, 0xEF, 0xAA, 0x70, 0x11, 0x75,
+ 0xFC, 0x48, 0xA1, 0x7E, 0x4C, 0x99, 0xBD, 0xB8,
+ 0x05, 0x01, 0xA6, 0xF0, 0x0E, 0x18, 0x0D, 0x75,
+ 0xF0, 0xBB, 0xBD, 0x6F, 0x28, 0xF4, 0xDC, 0x3A,
+ 0x58, 0x3D, 0x83, 0x1D, 0x78, 0xF9, 0x0E, 0x06,
+ 0x5C, 0x0B, 0x77, 0x4C, 0x28, 0x82, 0xE1, 0xA4,
+ 0x77, 0xAD, 0x14, 0x71, 0x3D, 0x5D, 0x53, 0x41,
+ 0xCE, 0x4F, 0x36, 0x32, 0x24, 0xD7, 0x09, 0xF0,
+ 0xB8, 0xF2, 0x95, 0x68, 0x84, 0x5C, 0x62, 0xC3,
+ 0x77, 0x9A, 0x74, 0xD3, 0x29, 0xC1, 0x88, 0x8B,
+ 0xBF, 0x05, 0x73, 0x43, 0x30, 0x20, 0xB4, 0xB0,
+ 0xB3, 0xE1, 0xB5, 0x91, 0xA8, 0xBD, 0x4B, 0x7F,
+ 0x2E, 0x53, 0x20, 0x55, 0xDE, 0x4D, 0xDB, 0x91,
+ 0x73, 0xF9, 0x2F, 0x21, 0xF6, 0xAE, 0x6D, 0xEE,
+ 0x67, 0x37, 0x82, 0x11, 0xE3, 0x0A, 0x5A, 0xB8,
+ 0x68, 0xC7, 0x22, 0x34, 0x2A, 0xD4, 0x37, 0xAF,
+ 0x50, 0xE0, 0xDB, 0xED, 0x89, 0x6D, 0x97, 0x0C,
+ 0xF6, 0x0D, 0x6D, 0x9B, 0x1D, 0x5B, 0xE1, 0xF6,
+ 0x45, 0x48, 0xF0, 0x75, 0x38, 0xE0, 0x2E, 0x11,
+ 0x88, 0xEC, 0x8B, 0x51, 0x47, 0x21, 0xF6, 0x6F,
+ 0x13, 0x3E, 0xA4, 0x46, 0xFE, 0xB0, 0x15, 0x81,
+ 0x68, 0x9C, 0x0F, 0x15, 0xCB, 0x64, 0x6D, 0x0B,
+ 0x04, 0x63, 0xBA, 0x67, 0x6C, 0x86, 0xE8, 0xAC,
+ 0xB3, 0xE5, 0x47, 0x88, 0x27, 0x4A, 0xBC, 0x58,
+ 0x13, 0x4D, 0x6F, 0xA6, 0x2C, 0xEC, 0x83, 0xB0,
+ 0x6E, 0x93, 0x7C, 0xC7, 0x6E, 0xBB, 0xBF, 0xE1,
+ 0x9F, 0xD6, 0x6E, 0xD8, 0x7A, 0xC5, 0xA2, 0x4D,
+ 0xA2, 0x1C, 0xDA, 0x55, 0x1E, 0xF2, 0x1C, 0x71,
+ 0x3E, 0x82, 0xD7, 0x92, 0x6F, 0x89, 0x1E, 0x53,
+ 0x48, 0x00, 0x9E, 0x93, 0xCB, 0xE3, 0x79, 0xB5,
+ 0x87, 0xDE, 0xF7, 0x84, 0xD6, 0x25, 0x57, 0x07,
+ 0x47, 0xEE, 0xDB, 0xA8, 0xC2, 0x7E, 0xE8, 0xF9,
+ 0x56, 0x22, 0x34, 0xE9, 0xD0, 0xC8, 0x3F, 0xF2,
+ 0xCB, 0xE7, 0x92, 0xE1, 0x03, 0xF4, 0x38, 0x88,
+ 0xE4, 0xC6, 0xB2, 0x42, 0x6F, 0x08, 0x61, 0x77,
+ 0x7F, 0x8A, 0xCD, 0x9D, 0x21, 0x3E, 0xE2, 0x18,
+ 0x52, 0x15, 0x02, 0x62, 0x2F, 0xCA, 0xEE, 0x44,
+ 0x60, 0x15, 0x6B, 0x76, 0x7D, 0x12, 0xC7, 0x0F,
+ 0xA0, 0x6C, 0xA1, 0x7B, 0xAE, 0xD6, 0x40, 0x1B,
+ 0x61, 0x46, 0xB4, 0x47, 0x97, 0x6C, 0xE1, 0x4C,
+ 0x47, 0x35, 0x29, 0x02, 0x6D, 0x69, 0x5E, 0x32,
+ 0xF7, 0x7F, 0x9C, 0xFA, 0x87, 0xD3, 0x7D, 0x33,
+ 0x85, 0x53, 0xF3, 0x8F, 0x0A, 0x41, 0x3C, 0x84,
+ 0x00, 0x14, 0x04, 0x63, 0xA8, 0xC3, 0xCB, 0xEB,
+ 0xAF, 0xCB, 0xE1, 0x78, 0xB2, 0x30, 0xC0, 0xA6,
+ 0x9D, 0x7A, 0x28, 0x0A, 0x60, 0x5F, 0x45, 0x74,
+ 0x24, 0xBE, 0xD8, 0x0D, 0xCB, 0x3A, 0x1B, 0xD1,
+ 0x9C, 0x2A, 0x4E, 0x9A, 0xD1, 0x27, 0xCB, 0xAB,
+ 0x08, 0x72, 0x89, 0xD9, 0xBA, 0x15, 0x01, 0x92,
+ 0x4F, 0xFA, 0x7A, 0xCA, 0x32, 0xD3, 0x7E, 0xB1,
+ 0xEB, 0xE7, 0xDF, 0x21, 0x33, 0x4F, 0xBD, 0x5F,
+ 0xCF, 0x64, 0x5D, 0x36, 0x21, 0x63, 0x34, 0x68,
+ 0x90, 0x81, 0x15, 0xC7, 0xF7, 0x28, 0x46, 0x42,
+ 0xC9, 0x78, 0xC0, 0xA3, 0x5F, 0xF8, 0x86, 0x01,
+ 0xF4, 0x1C, 0x6A, 0x65, 0xA2, 0x0E, 0x01, 0x5B,
+ 0x29, 0xF4, 0xF8, 0x9E, 0xA5, 0x5C, 0xC0, 0x4B,
+ 0x74, 0x2C, 0x88, 0xA3, 0x6B, 0xC2, 0xC4, 0xDA,
+ 0xEF, 0xF2, 0x02, 0xCD, 0x89, 0x0C, 0x42, 0xDA,
+ 0xE6, 0xBB, 0x67, 0x51, 0x29, 0x4A, 0x32, 0x99,
+ 0xAC, 0xCC, 0xDC, 0xB6, 0x35, 0x8B, 0xCC, 0xE8,
+ 0x04, 0x78, 0x69, 0xA8, 0x19, 0x0E, 0xDC, 0x1E,
+ 0x6C, 0xFF, 0x46, 0xDF, 0x4E, 0x7A, 0x51, 0xC5,
+ 0x9F, 0x39, 0x6F, 0xC7, 0x96, 0xFE, 0x54, 0xF4,
+ 0xE2, 0xD8, 0xEB, 0x5C, 0xA1, 0x28, 0x84, 0xB4,
+ 0xEA, 0x34, 0xA8, 0x18, 0x86, 0x93, 0x8B, 0xE9,
+ 0xA7, 0xAB, 0x5E, 0xD9, 0x1E, 0x3E, 0x89, 0x8F,
+ 0x83, 0xB4, 0xEF, 0x8E, 0x21, 0x30, 0xB4, 0xAD,
+ 0x07, 0xFE, 0x8A, 0x3B, 0x34, 0xAE, 0xDE, 0x5C,
+ 0xCA, 0xBA, 0x9E, 0xFB, 0x1B, 0x81, 0xEC, 0xD3,
+ 0x0E, 0x40, 0xED, 0x65, 0x77, 0xF3, 0xC0, 0x56,
+ 0xDC, 0xD8, 0x46, 0xDB, 0x4F, 0x51, 0x4B, 0x24,
+ 0x42, 0x3B, 0xC7, 0xE2, 0x32, 0xBA, 0xA7, 0xC0,
+ 0xA4, 0xFA, 0x14, 0xC4, 0xB7, 0x1B, 0x84, 0x0D,
+ 0xC9, 0xD1, 0x8E, 0x1C, 0x50, 0xF1, 0xBB, 0xA6,
+ 0x14, 0xD4, 0x88, 0xD8, 0x6C, 0xCE, 0x44, 0x76,
+ 0x3A, 0x29, 0xF8, 0x69, 0xBD, 0x5B, 0x53, 0x4B,
+ 0x65, 0xB8, 0x76, 0x5F, 0xD2, 0x01, 0xE9, 0x55,
+ 0x83, 0x47, 0x2C, 0xD3, 0x9B, 0xCA, 0x85, 0x75,
+ 0x60, 0x14, 0x1B, 0x5D, 0x48, 0x97, 0x67, 0x18,
+ 0x9F, 0x8E, 0x8C, 0x86, 0x31, 0xCF, 0x07, 0x7D,
+ 0x9E, 0xEB, 0xDE, 0x06, 0x14, 0xFE, 0x00, 0x3F,
+ 0x9C, 0x09, 0xDF, 0x2B, 0x99, 0x08, 0x7A, 0x4C,
+ 0xD9, 0xC8, 0x3B, 0x54, 0xF0, 0x34, 0x07, 0x19,
+ 0xC5, 0x4A, 0x34, 0xBF, 0xB8, 0x31, 0xBF, 0x1C,
+ 0x6F, 0x5C, 0x07, 0x37, 0x76, 0xD9, 0xBD, 0x3B,
+ 0xB6, 0xB6, 0x8D, 0x6A, 0x1A, 0xE6, 0xE9, 0xCB,
+ 0x4D, 0xC3, 0xDC, 0x76, 0x91, 0xE7, 0x6F, 0x11,
+ 0x66, 0xF7, 0x76, 0xB3, 0x40, 0xAA, 0x51, 0x09,
+ 0x7C, 0xFE, 0xA2, 0x37, 0xBF, 0xC3, 0x92, 0xFA,
+ 0x75, 0x44, 0x76, 0xC7, 0x54, 0xC8, 0x91, 0x89,
+ 0x65, 0x84, 0x03, 0x3E, 0x46, 0x92, 0x1E, 0x67,
+ 0x8D, 0x8F, 0x52, 0x0D, 0x06, 0x22, 0x18, 0xFC,
+ 0x0B, 0x92, 0xCD, 0x94, 0xC2, 0x0A, 0x3F, 0x41,
+};
+
+uint8_t mlkem_512_sk[] = {
+ 0x26, 0xCD, 0x28, 0x15, 0xEB, 0x3E, 0x16, 0x56,
+ 0x84, 0x2D, 0x15, 0xAC, 0x32, 0x33, 0xDA, 0x01,
+ 0x71, 0x21, 0x82, 0x1B, 0xC7, 0xE3, 0x44, 0xF9,
+ 0x5E, 0x7A, 0xB9, 0x3A, 0x40, 0xAD, 0x38, 0x6A
+};
+
struct
cperf_rsa_test_data rsa_qt_perf_data[4] = {
{
@@ -3822,6 +4246,52 @@ cperf_rsa_test_data rsa_pub_perf_data[4] = {
}
};
+struct cperf_mlkem_test_data mlkem_encap_perf_data[] = {
+ {
+ .name = "mlkem_512_encap (deterministic)",
+ .type = RTE_CRYPTO_ML_KEM_512,
+ .dk = {
+ .data = mlkem_512_dk,
+ .length = sizeof(mlkem_512_dk),
+ },
+ .ek = {
+ .data = mlkem_512_ek,
+ .length = sizeof(mlkem_512_ek),
+ },
+ .message = {
+ .data = mlkem_512_message,
+ .length = 32,
+ },
+ .cipher = {
+ .data = mlkem_512_cipher,
+ .length = sizeof(mlkem_512_cipher),
+ },
+ .sk = {
+ .data = mlkem_512_sk,
+ .length = sizeof(mlkem_512_sk),
+ },
+ },
+};
+
+struct cperf_mlkem_test_data mlkem_decap_perf_data[] = {
+ {
+ .name = "mlkem_512_decap",
+ .type = RTE_CRYPTO_ML_KEM_512,
+ .cipher = {
+ .data = mlkem_512_cipher,
+ .length = sizeof(mlkem_512_cipher),
+ },
+ .dk = {
+ .data = mlkem_512_dk,
+ .length = sizeof(mlkem_512_dk),
+ },
+ .sk = {
+ .data = mlkem_512_sk,
+ .length = sizeof(mlkem_512_sk),
+ },
+ }
+};
+
struct cperf_test_vector*
cperf_test_vector_get_dummy(struct cperf_options *options)
{
diff --git a/app/test-crypto-perf/cperf_test_vectors.h b/app/test-crypto-perf/cperf_test_vectors.h
index d6f18268c4..e498196ae3 100644
--- a/app/test-crypto-perf/cperf_test_vectors.h
+++ b/app/test-crypto-perf/cperf_test_vectors.h
@@ -107,6 +107,31 @@ struct cperf_modex_test_data {
} result;
};
+struct cperf_mlkem_test_data {
+ const char *name;
+ enum rte_crypto_ml_kem_type type;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } dk;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } ek;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } message;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } cipher;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } sk;
+};
+
#define TEST_DATA_SIZE 4096
struct cperf_rsa_plaintext {
uint8_t data[TEST_DATA_SIZE];
@@ -188,5 +213,7 @@ extern struct cperf_rsa_test_data rsa_pub_perf_data[4];
extern struct cperf_rsa_test_data rsa_exp_perf_data[4];
extern struct cperf_rsa_test_data rsa_qt_perf_data[4];
extern struct cperf_rsa_plaintext rsa_plaintext;
+extern struct cperf_mlkem_test_data mlkem_encap_perf_data[];
+extern struct cperf_mlkem_test_data mlkem_decap_perf_data[];
#endif
diff --git a/app/test-crypto-perf/main.c b/app/test-crypto-perf/main.c
index f63e892413..2e228201ab 100644
--- a/app/test-crypto-perf/main.c
+++ b/app/test-crypto-perf/main.c
@@ -53,7 +53,8 @@ const char *cperf_op_type_strs[] = {
[CPERF_ASYM_SECP521R1] = "ecdsa_p521r1",
[CPERF_ASYM_ED25519] = "eddsa_25519",
[CPERF_ASYM_SM2] = "sm2",
- [CPERF_TLS] = "tls-record"
+ [CPERF_TLS] = "tls-record",
+ [CPERF_ASYM_MLKEM512] = "mlkem_512",
};
const char *cperf_rsa_priv_keytype_strs[] = {
@@ -245,6 +246,7 @@ cperf_initialize_cryptodev(struct cperf_options *opts, uint8_t *enabled_cdevs)
case CPERF_ASYM_ED25519:
case CPERF_ASYM_SM2:
case CPERF_ASYM_RSA:
+ case CPERF_ASYM_MLKEM512:
case CPERF_ASYM_MODEX:
conf.ff_disable |= (RTE_CRYPTODEV_FF_SECURITY |
RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO);
@@ -515,6 +517,23 @@ cperf_verify_devices_capabilities(struct cperf_options *opts,
}
}
+ if (opts->op_type == CPERF_ASYM_MLKEM512) {
+ asym_cap_idx.type = RTE_CRYPTO_ASYM_XFORM_ML_KEM;
+ asym_capability = rte_cryptodev_asym_capability_get(cdev_id, &asym_cap_idx);
+ if (asym_capability == NULL)
+ return -1;
+
+ if (opts->asym_op_type == RTE_CRYPTO_ASYM_OP_ENCRYPT)
+ opts->mlkem_data = &mlkem_encap_perf_data[0];
+ else if (opts->asym_op_type == RTE_CRYPTO_ASYM_OP_DECRYPT)
+ opts->mlkem_data = &mlkem_decap_perf_data[0];
+ else {
+ RTE_LOG(ERR, USER1, "Unsupported MLKEM operation type\n");
+ return -ENOTSUP;
+ }
+
+ }
+
if (opts->op_type == CPERF_AUTH_ONLY ||
opts->op_type == CPERF_CIPHER_THEN_AUTH ||
opts->op_type == CPERF_AUTH_THEN_CIPHER) {
diff --git a/doc/guides/tools/cryptoperf.rst b/doc/guides/tools/cryptoperf.rst
index d7ee468e17..13f9db66f4 100644
--- a/doc/guides/tools/cryptoperf.rst
+++ b/doc/guides/tools/cryptoperf.rst
@@ -183,6 +183,7 @@ The following are the application command-line options:
eddsa_25519
rsa
sm2
+ mlkem_512
ipsec
tls-record
--
2.43.0
^ permalink raw reply related
* [PATCH v2] crypto/openssl: fix use-after-free bug and cleanup
From: Pratik Senapati @ 2026-06-09 5:53 UTC (permalink / raw)
To: dev; +Cc: gakhil, kai.ji, stable
params is freed before it is used by
EVP_PKEY_decapsulate_init()
causing a use-after-free issue.
Pass NULL to EVP_PKEY_decapsulate_init()
instead of params to avoid it.
Add resource cleanup for all error paths in the
ML-KEM decapsulate and encapsulate handlers.
Consolidate cleanup into multiple goto labels;
err_decap, err_pkey, err_params for decap and
err_encap, err_pkey, err_params for encap.
Fixes: 5f761d7b605e ("crypto/openssl: support ML-KEM and ML-DSA")
Cc: stable@dpdk.org
Signed-off-by: Pratik Senapati <psenapati@marvell.com>
---
.mailmap | 1 +
drivers/crypto/openssl/rte_openssl_pmd.c | 124 +++++++++--------------
2 files changed, 47 insertions(+), 78 deletions(-)
diff --git a/.mailmap b/.mailmap
index 4f93307aed..b6f47c10b9 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1287,6 +1287,7 @@ Prashant Gupta <prashant.gupta_3@nxp.com>
Prashant Upadhyaya <prashant.upadhyaya@aricent.com> <praupadhyaya@gmail.com>
Prateek Agarwal <prateekag@cse.iitb.ac.in>
Prathisna Padmasanan <prathisna.padmasanan@intel.com>
+Pratik Senapati <psenapati@marvell.com>
Praveen Kaligineedi <pkaligineedi@google.com>
Praveen Shetty <praveen.shetty@intel.com>
Pravin Pathak <pravin.pathak.dev@gmail.com> <pravin.pathak@intel.com>
diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c
index 4f171f48cc..7464884fb2 100644
--- a/drivers/crypto/openssl/rte_openssl_pmd.c
+++ b/drivers/crypto/openssl/rte_openssl_pmd.c
@@ -3537,35 +3537,24 @@ mlkem_encap_op_evp(struct rte_crypto_op *cop,
return -1;
}
- if (EVP_PKEY_fromdata_init(pctx) != 1) {
- OSSL_PARAM_free(params);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+ cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- if (EVP_PKEY_fromdata(pctx, &pkey, EVP_PKEY_PUBLIC_KEY, params) != 1) {
- OSSL_PARAM_free(params);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
- OSSL_PARAM_free(params);
+ if (EVP_PKEY_fromdata_init(pctx) != 1)
+ goto err_params;
- if (pkey == NULL) {
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+ if (EVP_PKEY_fromdata(pctx, &pkey, EVP_PKEY_PUBLIC_KEY, params) != 1)
+ goto err_params;
+
+ if (pkey == NULL)
+ goto err_params;
cctx = EVP_PKEY_CTX_new_from_pkey(NULL, pkey, NULL);
- if (cctx == NULL) {
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+ if (cctx == NULL)
+ goto err_pkey;
- if (EVP_PKEY_encapsulate_init(cctx, NULL) != 1) {
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+
+ if (EVP_PKEY_encapsulate_init(cctx, NULL) != 1)
+ goto err_encap;
if (op->encap.message.length) {
const OSSL_PARAM kem_params[] = {
@@ -3574,49 +3563,42 @@ mlkem_encap_op_evp(struct rte_crypto_op *cop,
OSSL_PARAM_END
};
- if (EVP_PKEY_encapsulate_init(cctx, kem_params) != 1) {
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+ if (EVP_PKEY_encapsulate_init(cctx, kem_params) != 1)
+ goto err_encap;
}
if (EVP_PKEY_encapsulate(cctx, NULL, &outlen, NULL, &keylen) != 1) {
OPENSSL_LOG(ERR, "Failed to determine output length");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_encap;
}
if (outlen > op->encap.cipher.length) {
OPENSSL_LOG(ERR, "Insufficient buffer for cipher text");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_encap;
}
if (keylen > op->encap.sk.length) {
OPENSSL_LOG(ERR, "Insufficient buffer for shared key");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_encap;
}
if (EVP_PKEY_encapsulate(cctx, op->encap.cipher.data, &outlen,
op->encap.sk.data, &keylen) != 1) {
OPENSSL_LOG(ERR, "Failed to encapculate");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_encap;
}
op->encap.cipher.length = outlen;
op->encap.sk.length = keylen;
+ ret = 0;
+ cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
+err_encap:
EVP_PKEY_CTX_free(cctx);
+err_pkey:
EVP_PKEY_free(pkey);
- ret = 0;
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
+err_params:
+ OSSL_PARAM_free(params);
return ret;
}
@@ -3664,65 +3646,51 @@ mlkem_decap_op_evp(struct rte_crypto_op *cop,
return -1;
}
- if (EVP_PKEY_fromdata_init(pctx) != 1) {
- OSSL_PARAM_free(params);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+ cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- if (EVP_PKEY_fromdata(pctx, &pkey, EVP_PKEY_PRIVATE_KEY, params) != 1) {
- OSSL_PARAM_free(params);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
- OSSL_PARAM_free(params);
+ if (EVP_PKEY_fromdata_init(pctx) != 1)
+ goto err_params;
- if (pkey == NULL) {
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+ if (EVP_PKEY_fromdata(pctx, &pkey, EVP_PKEY_PRIVATE_KEY, params) != 1)
+ goto err_params;
+
+ if (pkey == NULL)
+ goto err_params;
cctx = EVP_PKEY_CTX_new_from_pkey(NULL, pkey, NULL);
- if (cctx == NULL) {
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+ if (cctx == NULL)
+ goto err_pkey;
- if (EVP_PKEY_decapsulate_init(cctx, params) != 1) {
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
- }
+ if (EVP_PKEY_decapsulate_init(cctx, NULL) != 1)
+ goto err_decap;
if (EVP_PKEY_decapsulate(cctx, NULL, &keylen,
op->decap.cipher.data, op->decap.cipher.length) != 1) {
OPENSSL_LOG(ERR, "Failed to determine output length");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_decap;
}
if (keylen > op->decap.sk.length) {
OPENSSL_LOG(ERR, "Insufficient buffer for shared key");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_decap;
}
if (EVP_PKEY_decapsulate(cctx, op->decap.sk.data, &keylen,
op->decap.cipher.data, op->decap.cipher.length) != 1) {
OPENSSL_LOG(ERR, "Failed to decapsulate");
- EVP_PKEY_free(pkey);
- cop->status = RTE_CRYPTO_OP_STATUS_ERROR;
- return -1;
+ goto err_decap;
}
op->decap.sk.length = keylen;
+ ret = 0;
+ cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
+err_decap:
EVP_PKEY_CTX_free(cctx);
+err_pkey:
EVP_PKEY_free(pkey);
- ret = 0;
- cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
+err_params:
+ OSSL_PARAM_free(params);
return ret;
}
--
2.43.0
^ permalink raw reply related
* [PATCH v2 2/2] app/crypto-perf: support ML DSA
From: Pratik Senapati @ 2026-06-09 5:53 UTC (permalink / raw)
To: dev; +Cc: gakhil, kai.ji
In-Reply-To: <20260609055315.1539233-1-psenapati@marvell.com>
Add ML-DSA44 support to test-crypto-perf.
Signed-off-by: Pratik Senapati <psenapati@marvell.com>
---
app/test-crypto-perf/cperf_ops.c | 70 +
app/test-crypto-perf/cperf_options.h | 2 +
app/test-crypto-perf/cperf_options_parsing.c | 20 +-
app/test-crypto-perf/cperf_test_common.c | 3 +-
app/test-crypto-perf/cperf_test_vectors.c | 1661 ++++++++++++++++++
app/test-crypto-perf/cperf_test_vectors.h | 37 +
app/test-crypto-perf/main.c | 17 +
doc/guides/tools/cryptoperf.rst | 1 +
8 files changed, 1809 insertions(+), 2 deletions(-)
diff --git a/app/test-crypto-perf/cperf_ops.c b/app/test-crypto-perf/cperf_ops.c
index 806265f7bf..7d3a74b505 100644
--- a/app/test-crypto-perf/cperf_ops.c
+++ b/app/test-crypto-perf/cperf_ops.c
@@ -251,6 +251,58 @@ cperf_set_ops_asym_mlkem(struct rte_crypto_op **ops,
}
}
+static void
+cperf_set_ops_asym_mldsa(struct rte_crypto_op **ops,
+ uint32_t src_buf_offset __rte_unused,
+ uint32_t dst_buf_offset __rte_unused, uint16_t nb_ops,
+ void *sess,
+ const struct cperf_options *options,
+ const struct cperf_test_vector *test_vector __rte_unused,
+ uint16_t iv_offset __rte_unused,
+ uint32_t *imix_idx __rte_unused,
+ uint64_t *tsc_start __rte_unused)
+{
+ uint16_t i;
+
+ for (i = 0; i < nb_ops; i++) {
+ struct rte_crypto_asym_op *asym_op = ops[i]->asym;
+
+ ops[i]->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;
+ rte_crypto_op_attach_asym_session(ops[i], sess);
+
+ if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_SIGN) {
+ asym_op->mldsa.op = RTE_CRYPTO_ML_DSA_OP_SIGN;
+ asym_op->mldsa.siggen.privkey.data = options->mldsa_data->privkey.data;
+ asym_op->mldsa.siggen.privkey.length = options->mldsa_data->privkey.length;
+ asym_op->mldsa.siggen.message.data = options->mldsa_data->message.data;
+ asym_op->mldsa.siggen.message.length = options->mldsa_data->message.length;
+ asym_op->mldsa.siggen.sign.data = options->mldsa_data->sign.data;
+ asym_op->mldsa.siggen.sign.length = options->mldsa_data->sign.length;
+ asym_op->mldsa.siggen.ctx.data = options->mldsa_data->ctx.data;
+ asym_op->mldsa.siggen.ctx.length = options->mldsa_data->ctx.length;
+ asym_op->mldsa.siggen.seed.data = options->mldsa_data->seed.data;
+ asym_op->mldsa.siggen.seed.length = options->mldsa_data->seed.length;
+ asym_op->mldsa.siggen.mu.data = options->mldsa_data->mu.data;
+ asym_op->mldsa.siggen.mu.length = options->mldsa_data->mu.length;
+ asym_op->mldsa.siggen.hash = options->mldsa_data->hash;
+ } else if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_VERIFY) {
+ asym_op->mldsa.op = RTE_CRYPTO_ML_DSA_OP_VERIFY;
+ asym_op->mldsa.sigver.pubkey.data = options->mldsa_data->pubkey.data;
+ asym_op->mldsa.sigver.pubkey.length = options->mldsa_data->pubkey.length;
+ asym_op->mldsa.sigver.message.data = options->mldsa_data->message.data;
+ asym_op->mldsa.sigver.message.length = options->mldsa_data->message.length;
+ asym_op->mldsa.sigver.sign.data = options->mldsa_data->sign.data;
+ asym_op->mldsa.sigver.sign.length = options->mldsa_data->sign.length;
+ asym_op->mldsa.sigver.ctx.data = options->mldsa_data->ctx.data;
+ asym_op->mldsa.sigver.ctx.length = options->mldsa_data->ctx.length;
+ asym_op->mldsa.sigver.mu.data = options->mldsa_data->mu.data;
+ asym_op->mldsa.sigver.mu.length = options->mldsa_data->mu.length;
+ asym_op->mldsa.sigver.hash = options->mldsa_data->hash;
+ } else {
+ rte_panic("Unsupported ML-DSA operation type %d\n", options->asym_op_type);
+ }
+ }
+}
#ifdef RTE_LIB_SECURITY
static void
@@ -1269,6 +1321,21 @@ cperf_create_session(struct rte_mempool *sess_mp,
return asym_sess;
}
+ if (options->op_type == CPERF_ASYM_MLDSA44) {
+ xform.next = NULL;
+ xform.xform_type = RTE_CRYPTO_ASYM_XFORM_ML_DSA;
+ xform.mldsa.type = RTE_CRYPTO_ML_DSA_44;
+ xform.mldsa.sign_deterministic =
+ options->mldsa_data->sign_deterministic;
+ xform.mldsa.sign_prehash = false;
+
+ ret = rte_cryptodev_asym_session_create(dev_id, &xform, sess_mp, &asym_sess);
+ if (ret < 0) {
+ RTE_LOG(ERR, USER1, "ML-DSA Asym session create failed\n");
+ return NULL;
+ }
+ return asym_sess;
+ }
if (options->op_type == CPERF_ASYM_MLKEM512) {
xform.next = NULL;
@@ -1600,6 +1667,9 @@ cperf_get_op_functions(const struct cperf_options *options,
case CPERF_ASYM_SM2:
op_fns->populate_ops = cperf_set_ops_asym_sm2;
break;
+ case CPERF_ASYM_MLDSA44:
+ op_fns->populate_ops = cperf_set_ops_asym_mldsa;
+ break;
case CPERF_ASYM_MLKEM512:
op_fns->populate_ops = cperf_set_ops_asym_mlkem;
break;
diff --git a/app/test-crypto-perf/cperf_options.h b/app/test-crypto-perf/cperf_options.h
index 98b8eeec3e..d8ceca2424 100644
--- a/app/test-crypto-perf/cperf_options.h
+++ b/app/test-crypto-perf/cperf_options.h
@@ -101,6 +101,7 @@ enum cperf_op_type {
CPERF_ASYM_SECP521R1,
CPERF_ASYM_ED25519,
CPERF_ASYM_SM2,
+ CPERF_ASYM_MLDSA44,
CPERF_ASYM_MLKEM512,
CPERF_TLS,
};
@@ -189,6 +190,7 @@ struct cperf_options {
struct cperf_eddsa_test_data *eddsa_data;
struct cperf_sm2_test_data *sm2_data;
struct cperf_mlkem_test_data *mlkem_data;
+ struct cperf_mldsa_test_data *mldsa_data;
enum rte_crypto_asym_op_type asym_op_type;
enum rte_crypto_auth_algorithm asym_hash_alg;
struct cperf_rsa_test_data *rsa_data;
diff --git a/app/test-crypto-perf/cperf_options_parsing.c b/app/test-crypto-perf/cperf_options_parsing.c
index 34afa938c8..44c2a7173a 100644
--- a/app/test-crypto-perf/cperf_options_parsing.c
+++ b/app/test-crypto-perf/cperf_options_parsing.c
@@ -41,7 +41,7 @@ usage(char *progname)
" --optype cipher-only / auth-only / cipher-then-auth / auth-then-cipher /\n"
" aead / pdcp / docsis / ipsec / modex / rsa / secp192r1 /\n"
" secp224r1 / secp256r1 / secp384r1 / secp521r1 / eddsa / sm2 /\n"
- " mlkem_512 /\n"
+ " mlkem_512 / mldsa_44 /\n"
" tls-record : set operation type\n"
" --sessionless: enable session-less crypto operations\n"
" --shared-session: share 1 session across all queue pairs on crypto device\n"
@@ -560,6 +560,10 @@ parse_op_type(struct cperf_options *opts, const char *arg)
cperf_op_type_strs[CPERF_ASYM_SM2],
CPERF_ASYM_SM2
},
+ {
+ cperf_op_type_strs[CPERF_ASYM_MLDSA44],
+ CPERF_ASYM_MLDSA44
+ },
{
cperf_op_type_strs[CPERF_ASYM_MLKEM512],
CPERF_ASYM_MLKEM512
@@ -1180,6 +1184,7 @@ cperf_options_default(struct cperf_options *opts)
opts->eddsa_data = &ed25519_perf_data;
opts->sm2_data = &sm2_perf_data;
opts->mlkem_data = &mlkem_encap_perf_data[0];
+ opts->mldsa_data = &mldsa_sign_perf_data[0];
opts->asym_op_type = RTE_CRYPTO_ASYM_OP_ENCRYPT;
}
@@ -1709,6 +1714,17 @@ cperf_options_check(struct cperf_options *options)
}
#endif
+ if (options->op_type == CPERF_ASYM_MLDSA44) {
+ if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_SIGN)
+ options->mldsa_data = &mldsa_sign_perf_data[0];
+ else if (options->asym_op_type == RTE_CRYPTO_ASYM_OP_VERIFY)
+ options->mldsa_data = &mldsa_verify_perf_data[0];
+ else {
+ RTE_LOG(ERR, USER1, "ML-DSA only supports sign and verify operations\n");
+ return -EINVAL;
+ }
+ }
+
return 0;
}
@@ -1759,6 +1775,8 @@ cperf_options_dump(struct cperf_options *opts)
rte_crypto_asym_op_strings[opts->asym_op_type]);
if (opts->op_type == CPERF_ASYM_RSA)
printf("# rsa test name: %s\n", opts->rsa_data->name);
+ if (opts->op_type == CPERF_ASYM_MLDSA44)
+ printf("# mldsa test name: %s\n", opts->mldsa_data->name);
if (opts->op_type == CPERF_ASYM_MLKEM512)
printf("# mlkem test name: %s\n", opts->mlkem_data->name);
}
diff --git a/app/test-crypto-perf/cperf_test_common.c b/app/test-crypto-perf/cperf_test_common.c
index 383d4bd940..0bcaa6dfd8 100644
--- a/app/test-crypto-perf/cperf_test_common.c
+++ b/app/test-crypto-perf/cperf_test_common.c
@@ -314,7 +314,8 @@ cperf_is_asym_test(const struct cperf_options *options)
options->op_type == CPERF_ASYM_SECP521R1 ||
options->op_type == CPERF_ASYM_ED25519 ||
options->op_type == CPERF_ASYM_SM2 ||
- options->op_type == CPERF_ASYM_MLKEM512)
+ options->op_type == CPERF_ASYM_MLKEM512 ||
+ options->op_type == CPERF_ASYM_MLDSA44)
return true;
return false;
diff --git a/app/test-crypto-perf/cperf_test_vectors.c b/app/test-crypto-perf/cperf_test_vectors.c
index 6e2435b004..a3b050af11 100644
--- a/app/test-crypto-perf/cperf_test_vectors.c
+++ b/app/test-crypto-perf/cperf_test_vectors.c
@@ -3934,6 +3934,1624 @@ uint8_t mlkem_512_sk[] = {
0x5E, 0x7A, 0xB9, 0x3A, 0x40, 0xAD, 0x38, 0x6A
};
+uint8_t mldsa_44_seed[] = {
+ 0xf4, 0x41, 0xc9, 0x62, 0x17, 0x32, 0x2c, 0xbc,
+ 0xcf, 0x05, 0xf7, 0x5b, 0xd5, 0xfb, 0x0a, 0x0a,
+ 0x78, 0xbf, 0xad, 0xab, 0x89, 0x6e, 0x88, 0xdb,
+ 0xb4, 0x08, 0x90, 0xe0, 0x16, 0x9b, 0xb6, 0x66,
+};
+
+uint8_t mldsa_44_privkey[] = {
+ 0xcc, 0x2c, 0x93, 0xce, 0xfc, 0x0b, 0xf6, 0x74,
+ 0x93, 0x28, 0x95, 0xe8, 0xc0, 0xc8, 0x83, 0xb8,
+ 0xc9, 0x0c, 0x9a, 0x5c, 0x18, 0xd2, 0x79, 0x5c,
+ 0xf3, 0x58, 0xd8, 0x02, 0x6c, 0x5e, 0xad, 0x79,
+ 0xc1, 0xae, 0x0a, 0x97, 0x90, 0x41, 0x10, 0x32,
+ 0x59, 0x46, 0x12, 0x27, 0x0a, 0xa3, 0xc7, 0xf0,
+ 0x74, 0x54, 0x3d, 0xb2, 0x24, 0xc2, 0xa2, 0xec,
+ 0xaf, 0x31, 0xa0, 0xf4, 0x78, 0x97, 0x7c, 0x6a,
+ 0xdc, 0x91, 0x87, 0x60, 0x01, 0xa9, 0x19, 0x2d,
+ 0x0d, 0xca, 0xb4, 0x14, 0x9d, 0x41, 0xe3, 0x18,
+ 0x4b, 0x99, 0x31, 0xfa, 0xf4, 0xfa, 0xc4, 0x75,
+ 0x93, 0x87, 0xc7, 0x7a, 0x8f, 0x0e, 0xc6, 0xb2,
+ 0xfa, 0xf6, 0x48, 0x84, 0x17, 0x42, 0x01, 0xcb,
+ 0xd2, 0x55, 0x40, 0x2f, 0x23, 0x21, 0x91, 0x11,
+ 0x40, 0x00, 0xdc, 0x91, 0x3d, 0xf5, 0x96, 0xdb,
+ 0xb3, 0x6d, 0xd5, 0xf4, 0xc5, 0x11, 0x96, 0x8f,
+ 0x13, 0x25, 0x4a, 0x24, 0xb3, 0x90, 0x9b, 0x16,
+ 0x30, 0x20, 0x96, 0x10, 0x10, 0x97, 0x6c, 0xd0,
+ 0x26, 0x26, 0x61, 0x84, 0x85, 0x1b, 0xb2, 0x81,
+ 0x5c, 0x10, 0x88, 0x48, 0xc8, 0x8d, 0x1c, 0x42,
+ 0x61, 0x19, 0x32, 0x04, 0x81, 0xa6, 0x89, 0x98,
+ 0x42, 0x62, 0x14, 0xa3, 0x90, 0xc8, 0xc2, 0x4c,
+ 0x41, 0x04, 0x2c, 0x5b, 0x26, 0x2a, 0xd1, 0x30,
+ 0x12, 0x18, 0x40, 0x71, 0x08, 0x98, 0x70, 0x60,
+ 0x40, 0x09, 0x4a, 0x44, 0x0a, 0x24, 0x43, 0x92,
+ 0x04, 0x20, 0x06, 0x80, 0xa2, 0x01, 0x18, 0x47,
+ 0x12, 0x23, 0xc7, 0x6c, 0x50, 0x32, 0x61, 0xda,
+ 0x26, 0x40, 0x84, 0x98, 0x90, 0xd3, 0x04, 0x66,
+ 0x52, 0xb8, 0x70, 0x99, 0x02, 0x6c, 0x64, 0xb2,
+ 0x00, 0x1b, 0x35, 0x90, 0x23, 0x00, 0x92, 0x01,
+ 0x02, 0x20, 0x12, 0x89, 0x25, 0x62, 0xa6, 0x4d,
+ 0x1b, 0x29, 0x08, 0x08, 0x46, 0x72, 0xa1, 0x18,
+ 0x11, 0x03, 0x80, 0x60, 0xd4, 0x38, 0x80, 0x64,
+ 0x36, 0x4e, 0x23, 0x30, 0x8e, 0x23, 0x81, 0x60,
+ 0xd1, 0x84, 0x89, 0xe2, 0xa8, 0x2d, 0x80, 0x92,
+ 0x61, 0x8a, 0x22, 0x28, 0x0a, 0xc9, 0x50, 0xe3,
+ 0x96, 0x40, 0x84, 0xa0, 0x0d, 0xd8, 0xc2, 0x2c,
+ 0xd1, 0x30, 0x10, 0x18, 0xa0, 0x48, 0xe2, 0xb2,
+ 0x04, 0x00, 0xa8, 0x29, 0x19, 0x42, 0x68, 0xa3,
+ 0xa2, 0x71, 0xd9, 0xc0, 0x45, 0xa2, 0xc6, 0x4c,
+ 0x51, 0x36, 0x71, 0xc1, 0x40, 0x02, 0x08, 0x12,
+ 0x72, 0x90, 0x04, 0x2e, 0x80, 0x30, 0x24, 0x8b,
+ 0x90, 0x44, 0x82, 0x82, 0x68, 0xa2, 0x84, 0x2d,
+ 0x08, 0x00, 0x61, 0x23, 0x82, 0x48, 0x00, 0x33,
+ 0x29, 0x58, 0x48, 0x86, 0x01, 0x01, 0x21, 0x62,
+ 0xa8, 0x91, 0xe3, 0x10, 0x6c, 0x58, 0x24, 0x09,
+ 0x0c, 0x14, 0x02, 0x63, 0x22, 0x20, 0x12, 0x48,
+ 0x70, 0x81, 0x26, 0x6e, 0x00, 0x86, 0x31, 0x1a,
+ 0x22, 0x65, 0x40, 0x38, 0x00, 0x02, 0x47, 0x09,
+ 0xc9, 0x82, 0x8c, 0x82, 0xa0, 0x2c, 0xd3, 0x04,
+ 0x45, 0x50, 0xb0, 0x28, 0xd0, 0x10, 0x8d, 0x1c,
+ 0x10, 0x52, 0x4b, 0x00, 0x66, 0xc8, 0xc0, 0x08,
+ 0x9a, 0xa8, 0x2d, 0xd4, 0x02, 0x8a, 0x1b, 0x35,
+ 0x65, 0x04, 0xc7, 0x28, 0x84, 0x38, 0x50, 0x18,
+ 0x20, 0x4a, 0x91, 0x18, 0x8c, 0x9b, 0x20, 0x09,
+ 0x94, 0x08, 0x71, 0x64, 0x06, 0x25, 0x20, 0x23,
+ 0x09, 0xd8, 0x82, 0x68, 0x5a, 0xa6, 0x31, 0x63,
+ 0x36, 0x32, 0x5c, 0x12, 0x62, 0xdb, 0x32, 0x6e,
+ 0x14, 0x93, 0x68, 0x9a, 0x34, 0x80, 0x83, 0x44,
+ 0x4a, 0x18, 0x13, 0x49, 0x43, 0xb8, 0x0d, 0xda,
+ 0x92, 0x41, 0x42, 0xb8, 0x84, 0x52, 0x94, 0x91,
+ 0x04, 0x05, 0x4d, 0xa2, 0x38, 0x64, 0x01, 0x19,
+ 0x05, 0x18, 0xa2, 0x4d, 0x90, 0x34, 0x26, 0x91,
+ 0x30, 0x68, 0x1c, 0x14, 0x44, 0x4b, 0x18, 0x22,
+ 0x12, 0x90, 0x91, 0x42, 0x96, 0x91, 0x19, 0x48,
+ 0x4c, 0x24, 0x46, 0x80, 0xc3, 0xb8, 0x8c, 0x1c,
+ 0x80, 0x8d, 0x18, 0x45, 0x88, 0xd8, 0x94, 0x4c,
+ 0x13, 0x02, 0x65, 0xd3, 0x22, 0x29, 0x49, 0x04,
+ 0x82, 0x04, 0xc2, 0x40, 0x04, 0x34, 0x21, 0xe1,
+ 0x34, 0x46, 0x20, 0x90, 0x44, 0x93, 0x14, 0x44,
+ 0xdb, 0xc0, 0x4d, 0x1b, 0xa2, 0x24, 0x5a, 0x24,
+ 0x68, 0x10, 0x46, 0x66, 0x11, 0x43, 0x29, 0xd0,
+ 0x90, 0x0d, 0x49, 0x12, 0x4c, 0xc1, 0x96, 0x2c,
+ 0xdb, 0x22, 0x65, 0xe4, 0x16, 0x24, 0xd1, 0x24,
+ 0x26, 0x53, 0xb8, 0x45, 0x8a, 0xb4, 0x11, 0x04,
+ 0x96, 0x28, 0x80, 0x22, 0x8e, 0xd1, 0x10, 0x05,
+ 0x44, 0x44, 0x8a, 0x5c, 0xb4, 0x50, 0xdc, 0x10,
+ 0x02, 0xc1, 0xa2, 0x4c, 0x11, 0x05, 0x2c, 0x21,
+ 0x39, 0x6a, 0x64, 0x36, 0x01, 0x09, 0xc3, 0x0c,
+ 0x9a, 0x24, 0x52, 0x92, 0x20, 0x0a, 0xd2, 0xa8,
+ 0x49, 0x99, 0xc6, 0x8d, 0x50, 0x26, 0x62, 0x09,
+ 0x42, 0x05, 0x5b, 0xc0, 0x41, 0xd0, 0x22, 0x25,
+ 0x09, 0xb4, 0x2d, 0x98, 0xa2, 0x24, 0x12, 0x84,
+ 0x0d, 0xca, 0x98, 0x11, 0x22, 0x80, 0x21, 0x1b,
+ 0x00, 0x44, 0x84, 0xc6, 0x89, 0x92, 0x48, 0x42,
+ 0x50, 0xc8, 0x60, 0x24, 0x15, 0x69, 0xa2, 0x48,
+ 0x06, 0xc4, 0xb0, 0x24, 0x63, 0x14, 0x40, 0x12,
+ 0xc4, 0x61, 0x82, 0x30, 0x60, 0x88, 0x96, 0x89,
+ 0xc9, 0xc0, 0x89, 0x12, 0x36, 0x82, 0x8c, 0xc0,
+ 0x09, 0xc0, 0x88, 0x91, 0xdb, 0x10, 0x64, 0x08,
+ 0x36, 0x71, 0x94, 0x10, 0x46, 0x20, 0x83, 0x10,
+ 0x13, 0x13, 0x4c, 0x08, 0x28, 0x45, 0x24, 0xb9,
+ 0x64, 0x09, 0xb0, 0x48, 0x1c, 0x25, 0x50, 0x23,
+ 0x40, 0x90, 0xe0, 0x96, 0x50, 0x9a, 0x38, 0x41,
+ 0xcc, 0xb2, 0x2c, 0xdb, 0x04, 0x66, 0x61, 0x48,
+ 0x30, 0x02, 0x34, 0x4e, 0xd1, 0xc2, 0x85, 0xc9,
+ 0x06, 0x41, 0x19, 0x30, 0x69, 0xc8, 0x94, 0x68,
+ 0x1c, 0x83, 0x70, 0x03, 0x10, 0x4d, 0xc2, 0x16,
+ 0x4e, 0x64, 0x92, 0x30, 0x60, 0x16, 0x84, 0xa2,
+ 0xa8, 0x90, 0xa4, 0x38, 0x11, 0x0a, 0x88, 0x05,
+ 0x10, 0x93, 0x88, 0x43, 0x06, 0x10, 0x18, 0x10,
+ 0x70, 0x03, 0x28, 0x42, 0x89, 0x28, 0x8d, 0x8a,
+ 0xa8, 0x09, 0x94, 0x42, 0x85, 0xa3, 0x10, 0x61,
+ 0x0c, 0x45, 0x50, 0x04, 0x31, 0x4c, 0xd1, 0x10,
+ 0x12, 0x8b, 0x96, 0x05, 0x03, 0xa0, 0x89, 0x21,
+ 0xc4, 0x4d, 0x61, 0xc8, 0x61, 0x09, 0x99, 0x10,
+ 0x59, 0x96, 0x49, 0x00, 0x89, 0x08, 0x23, 0x00,
+ 0x10, 0xcc, 0xa4, 0x61, 0x01, 0x25, 0x80, 0x21,
+ 0x31, 0x09, 0x13, 0x28, 0x0d, 0x1c, 0x05, 0x85,
+ 0x62, 0x92, 0x44, 0x09, 0x10, 0x8a, 0x5a, 0xa6,
+ 0x20, 0x89, 0xb0, 0x20, 0x0c, 0xb3, 0x01, 0xc9,
+ 0x26, 0x61, 0x0b, 0x34, 0x6a, 0x8b, 0x44, 0x0e,
+ 0xf2, 0x47, 0x18, 0xf8, 0x4a, 0xfa, 0x45, 0x60,
+ 0x87, 0xeb, 0x0f, 0xff, 0x75, 0xa6, 0xba, 0x18,
+ 0x9a, 0x7d, 0x37, 0xc4, 0xc7, 0x7e, 0x1a, 0xc9,
+ 0x0d, 0x06, 0x53, 0xca, 0xf0, 0x2c, 0xe9, 0xe9,
+ 0x4b, 0x5f, 0x8b, 0xbc, 0x77, 0xc6, 0x71, 0x68,
+ 0x4f, 0x8c, 0x3b, 0x30, 0x59, 0x91, 0xf0, 0xb5,
+ 0x6d, 0xa6, 0x28, 0x2c, 0xd4, 0x0a, 0x00, 0x27,
+ 0x6a, 0x39, 0x17, 0x88, 0xa9, 0x93, 0xbb, 0x6e,
+ 0xda, 0x3e, 0x4b, 0xb7, 0x1b, 0x6a, 0xb2, 0xe2,
+ 0xc4, 0x6f, 0x21, 0x78, 0x74, 0xff, 0xa2, 0x02,
+ 0x65, 0x68, 0xa4, 0xce, 0xb4, 0xfe, 0x0c, 0x18,
+ 0x59, 0xb9, 0xc8, 0x33, 0x41, 0x27, 0xa1, 0x4b,
+ 0xcf, 0xe9, 0xa9, 0xd7, 0x2d, 0xf9, 0xcd, 0xc6,
+ 0xde, 0x4e, 0x1d, 0x7d, 0x07, 0xf4, 0xda, 0xed,
+ 0x89, 0xa1, 0xef, 0xbc, 0xec, 0x1b, 0xc3, 0x07,
+ 0xfd, 0xf5, 0x96, 0xa3, 0x6c, 0x70, 0xc8, 0xb9,
+ 0x9f, 0x46, 0x0a, 0xad, 0x46, 0x01, 0xa1, 0x92,
+ 0x7c, 0xd1, 0x12, 0x78, 0x89, 0x89, 0x98, 0x68,
+ 0x9e, 0xdd, 0xd9, 0x7b, 0x9f, 0x2b, 0x29, 0xc1,
+ 0x0e, 0x29, 0xb0, 0x44, 0x7c, 0x22, 0xd7, 0x67,
+ 0x47, 0x91, 0xf0, 0xf8, 0x56, 0x79, 0x89, 0x42,
+ 0x95, 0x60, 0x53, 0x32, 0xc1, 0x14, 0x48, 0x56,
+ 0x67, 0x00, 0x80, 0x86, 0xcd, 0x80, 0x21, 0x75,
+ 0xe8, 0xe2, 0x41, 0x6a, 0x5d, 0xb2, 0x12, 0x57,
+ 0x1e, 0x83, 0x34, 0xa4, 0x95, 0x4e, 0x0e, 0x71,
+ 0xac, 0xcb, 0x09, 0xef, 0xea, 0x4e, 0xe0, 0x4d,
+ 0x44, 0xb2, 0x67, 0xd2, 0x6e, 0x78, 0x01, 0xb5,
+ 0x09, 0x64, 0xa7, 0xdf, 0xe6, 0x16, 0x72, 0x6b,
+ 0x19, 0xe5, 0x88, 0x8a, 0x06, 0xae, 0x0a, 0x93,
+ 0xdb, 0xee, 0x0e, 0xf6, 0x52, 0x84, 0x95, 0x2b,
+ 0xf8, 0xf4, 0xcd, 0x5d, 0x8b, 0x3a, 0x86, 0x87,
+ 0x8e, 0x8b, 0x97, 0x65, 0xb1, 0xd6, 0x50, 0xbc,
+ 0xa7, 0xf0, 0x32, 0xb5, 0x39, 0x1e, 0x10, 0x7d,
+ 0x96, 0x11, 0xc1, 0xe8, 0x47, 0xfa, 0xdd, 0x24,
+ 0x22, 0xb6, 0x43, 0xb2, 0x41, 0x95, 0x98, 0x1a,
+ 0x3f, 0x0f, 0x9a, 0xd3, 0xc4, 0x10, 0x19, 0x85,
+ 0x0a, 0x0d, 0xde, 0xb7, 0x76, 0xec, 0xe3, 0x57,
+ 0x2e, 0x38, 0x4e, 0xb1, 0x2e, 0x4a, 0xaf, 0xc8,
+ 0x78, 0xa7, 0xb3, 0x1e, 0xaf, 0xe0, 0xe4, 0xbb,
+ 0xc8, 0x0b, 0x24, 0x04, 0x08, 0xd2, 0xbc, 0x91,
+ 0xca, 0x8e, 0x4e, 0x85, 0x39, 0x76, 0x20, 0xc9,
+ 0xa7, 0x47, 0x8b, 0x6a, 0xd9, 0xc8, 0xb7, 0xb0,
+ 0xa5, 0xa2, 0x85, 0x48, 0x4c, 0xcb, 0x16, 0xaf,
+ 0x45, 0x48, 0x00, 0xbe, 0x67, 0xef, 0x67, 0x65,
+ 0xa3, 0x9d, 0x43, 0x07, 0x59, 0x88, 0x9a, 0xad,
+ 0xe5, 0xad, 0x1e, 0xeb, 0xd5, 0xd4, 0xc7, 0xa4,
+ 0xb9, 0x40, 0x70, 0x2c, 0xe1, 0xcb, 0x5c, 0x3f,
+ 0x49, 0x6b, 0x51, 0xbf, 0xd3, 0xb7, 0xea, 0x33,
+ 0x52, 0x73, 0xcf, 0x26, 0x19, 0x54, 0x1d, 0xb6,
+ 0x33, 0x5d, 0x1a, 0x88, 0xdb, 0xd6, 0xfe, 0xae,
+ 0xf4, 0x45, 0x67, 0x80, 0x8a, 0xfc, 0xc4, 0xbe,
+ 0xfb, 0x6a, 0x4a, 0x75, 0x88, 0x35, 0x26, 0x1f,
+ 0x90, 0x65, 0x26, 0x5e, 0xb8, 0x0d, 0x15, 0x9d,
+ 0x70, 0x76, 0xf2, 0xfd, 0xed, 0x55, 0x71, 0x2b,
+ 0xe8, 0x2d, 0xc0, 0x28, 0xee, 0x53, 0x7c, 0xa1,
+ 0x3e, 0x86, 0x74, 0xa2, 0xd3, 0x9e, 0x2c, 0x70,
+ 0x15, 0x3a, 0x90, 0xfd, 0x1d, 0xac, 0x81, 0x2d,
+ 0xa6, 0x6a, 0xe6, 0xd7, 0x7f, 0xd9, 0x97, 0xc9,
+ 0x47, 0x7c, 0x07, 0xe6, 0x35, 0x45, 0xc9, 0x1f,
+ 0x08, 0xbc, 0x14, 0xdb, 0x12, 0x84, 0xc3, 0xc8,
+ 0xe2, 0x86, 0xa3, 0xa2, 0x1e, 0xfb, 0x3f, 0xca,
+ 0x92, 0x04, 0xe0, 0x6b, 0x29, 0xe0, 0xe0, 0x23,
+ 0x51, 0x40, 0x8e, 0xec, 0xf8, 0x87, 0xa2, 0x7a,
+ 0x2d, 0x8b, 0xc2, 0x62, 0x8c, 0x29, 0x82, 0x4b,
+ 0x86, 0x1a, 0xa7, 0xc6, 0xaa, 0x24, 0x5f, 0x24,
+ 0x03, 0x20, 0xa8, 0x3c, 0xa3, 0xcb, 0xd1, 0x60,
+ 0x64, 0x7d, 0x22, 0x9b, 0x07, 0x46, 0x1e, 0xff,
+ 0x43, 0xa9, 0x38, 0x34, 0x0f, 0x01, 0x00, 0x7f,
+ 0xc6, 0xcb, 0x0a, 0xed, 0x73, 0xc0, 0xfb, 0x77,
+ 0x80, 0x8c, 0x2c, 0x4c, 0xa5, 0x20, 0xe8, 0xcf,
+ 0x37, 0xf9, 0x00, 0x12, 0xe1, 0x6c, 0xb5, 0x9a,
+ 0x91, 0xbe, 0x1f, 0xaf, 0xf1, 0x8c, 0x6e, 0x09,
+ 0xea, 0xcf, 0xac, 0x43, 0x6e, 0x32, 0xd0, 0xf4,
+ 0x89, 0x4e, 0x42, 0xc2, 0xe7, 0x84, 0x7a, 0x66,
+ 0xb5, 0x2b, 0x5c, 0x7d, 0x83, 0x4b, 0xd4, 0x3d,
+ 0x81, 0x6d, 0x68, 0xb9, 0x72, 0xe5, 0xd5, 0xe5,
+ 0xf5, 0x9c, 0x4d, 0x95, 0xcd, 0xcd, 0x4d, 0x9a,
+ 0x4e, 0x69, 0xb9, 0x8a, 0x1e, 0x96, 0x69, 0xc0,
+ 0xe0, 0x39, 0x02, 0x21, 0x1c, 0x82, 0xd9, 0x5b,
+ 0xe5, 0xde, 0x34, 0xff, 0xf0, 0x15, 0xc7, 0x9a,
+ 0x62, 0x83, 0xca, 0xa3, 0x70, 0x15, 0xeb, 0x8f,
+ 0x73, 0x0e, 0xa9, 0x9c, 0xe9, 0xe3, 0xe7, 0xfc,
+ 0xb4, 0xcb, 0xac, 0xdf, 0x61, 0x91, 0xf3, 0x4b,
+ 0x87, 0xa6, 0x29, 0xea, 0x36, 0x81, 0x07, 0x3e,
+ 0xae, 0x69, 0xee, 0x87, 0x2a, 0x30, 0x1c, 0x97,
+ 0x96, 0x40, 0x08, 0x48, 0xcf, 0x5f, 0xfe, 0x57,
+ 0x23, 0xfd, 0xb7, 0x9a, 0x69, 0xef, 0xf5, 0x6e,
+ 0xc9, 0xdd, 0x95, 0x2b, 0x17, 0xd5, 0x2c, 0xad,
+ 0xa4, 0x49, 0x78, 0xb1, 0xd1, 0x00, 0xe7, 0xca,
+ 0xcb, 0x20, 0x9e, 0xba, 0xea, 0xc8, 0xd4, 0x9b,
+ 0x84, 0xac, 0x0b, 0xf0, 0x3b, 0x2c, 0xf0, 0x95,
+ 0x3b, 0x5b, 0xf8, 0xfe, 0xe5, 0xc6, 0xee, 0x7c,
+ 0x9e, 0x41, 0xf0, 0xef, 0xcf, 0x4d, 0xdd, 0xe8,
+ 0x63, 0x6b, 0xcf, 0xe9, 0xd2, 0x80, 0x98, 0xf9,
+ 0x7d, 0x22, 0x7e, 0xfc, 0xdd, 0x50, 0x6b, 0xa3,
+ 0x6d, 0x7c, 0xed, 0x35, 0xbc, 0x28, 0x64, 0x03,
+ 0xe6, 0x01, 0xac, 0x7c, 0xe1, 0x9b, 0x08, 0x6f,
+ 0xd8, 0xdc, 0xaf, 0xb6, 0x9d, 0x4c, 0xdf, 0xb1,
+ 0xde, 0xf4, 0x90, 0x5b, 0x46, 0xee, 0x7d, 0x28,
+ 0xfb, 0xaa, 0xa9, 0x69, 0x47, 0xf5, 0x52, 0x74,
+ 0x97, 0x9f, 0x8b, 0xbb, 0x26, 0x8f, 0x0d, 0x89,
+ 0x2f, 0xc9, 0x57, 0xac, 0x2f, 0x88, 0x12, 0xdd,
+ 0x12, 0x2f, 0x75, 0x97, 0xc7, 0x7b, 0x45, 0x46,
+ 0x3a, 0x52, 0x4b, 0xf6, 0x76, 0xe0, 0x99, 0xef,
+ 0xbe, 0xbf, 0xed, 0x0d, 0x32, 0xed, 0xc6, 0x5e,
+ 0x6a, 0xe6, 0x0e, 0x12, 0xd4, 0xce, 0xf8, 0x0c,
+ 0x7b, 0x4f, 0x1d, 0x3b, 0xf8, 0xaf, 0xdd, 0xd1,
+ 0x78, 0x74, 0x59, 0xc7, 0xe4, 0xbc, 0xe3, 0xdd,
+ 0x4a, 0xa7, 0x77, 0xa5, 0xcd, 0x80, 0x7c, 0x6b,
+ 0x19, 0x46, 0x3d, 0xbd, 0x0a, 0x75, 0x48, 0xfa,
+ 0x1c, 0xee, 0x02, 0xe9, 0x79, 0x10, 0x6c, 0xb4,
+ 0xb3, 0xb2, 0xbe, 0xc8, 0x5c, 0xd1, 0xa1, 0xe2,
+ 0xd5, 0x0a, 0xb5, 0xf3, 0x0c, 0x3f, 0x03, 0xb9,
+ 0x2f, 0x61, 0x44, 0x95, 0xe1, 0x16, 0xcb, 0xc1,
+ 0xee, 0xb3, 0xb8, 0x85, 0xd6, 0x1c, 0xf4, 0xfa,
+ 0x86, 0x73, 0xc5, 0xf3, 0xba, 0xb5, 0xe0, 0x95,
+ 0x34, 0x95, 0x72, 0xfa, 0xa7, 0xb8, 0x50, 0xba,
+ 0xfb, 0xa6, 0x3c, 0x0f, 0x3c, 0xe3, 0xaa, 0x56,
+ 0x05, 0xaf, 0x08, 0xec, 0x50, 0x12, 0x73, 0x25,
+ 0x29, 0x72, 0xf5, 0xf6, 0xee, 0x06, 0xe4, 0x05,
+ 0x1a, 0xe1, 0x1a, 0x13, 0x3f, 0xa3, 0x55, 0x16,
+ 0x12, 0x17, 0x88, 0xd2, 0xa7, 0xc1, 0xd0, 0xf2,
+ 0x7d, 0x51, 0x46, 0x36, 0x2e, 0x77, 0xa8, 0x02,
+ 0x67, 0xad, 0xe7, 0x63, 0xaa, 0xbf, 0xe5, 0xbf,
+ 0xa1, 0xab, 0xc4, 0xf6, 0x72, 0xae, 0xdf, 0x79,
+ 0x0e, 0x82, 0x47, 0xb5, 0xf4, 0x48, 0x39, 0x22,
+ 0xe3, 0x0a, 0x94, 0x5e, 0x3a, 0x4b, 0xa6, 0xa7,
+ 0x5c, 0xb8, 0xec, 0x3e, 0x10, 0xae, 0x94, 0x34,
+ 0x2c, 0xc0, 0x4b, 0xdd, 0xca, 0x4d, 0xc9, 0x7c,
+ 0xf8, 0x2e, 0x32, 0x4e, 0x00, 0x2c, 0x34, 0x71,
+ 0x28, 0xad, 0xd5, 0xcb, 0x19, 0xb8, 0x1d, 0xaa,
+ 0x85, 0xac, 0xec, 0x16, 0xb6, 0x1f, 0x95, 0xc5,
+ 0xa9, 0x4b, 0x8a, 0xfc, 0x70, 0xb7, 0x34, 0xb0,
+ 0x63, 0x55, 0xff, 0x15, 0xc2, 0x72, 0x74, 0xe0,
+ 0x6d, 0x01, 0x3d, 0xf0, 0x6a, 0xab, 0x4b, 0x8a,
+ 0x72, 0xdc, 0x2b, 0x0a, 0x8a, 0xf4, 0x02, 0x1b,
+ 0x0b, 0x51, 0xd8, 0xf7, 0x17, 0xcf, 0x48, 0xf0,
+ 0x97, 0xa0, 0xc2, 0x7a, 0xc4, 0x64, 0x59, 0x2a,
+ 0xa3, 0x27, 0xbe, 0x3b, 0xf7, 0xc5, 0x6d, 0x51,
+ 0x29, 0xe2, 0x68, 0x87, 0x75, 0x71, 0x72, 0xac,
+ 0xc8, 0xc1, 0xc8, 0xe5, 0xf6, 0x8a, 0x27, 0x8c,
+ 0x45, 0x03, 0x10, 0xf7, 0x1a, 0xff, 0x1b, 0x04,
+ 0x1c, 0xaf, 0x27, 0x7a, 0x57, 0xac, 0x0f, 0x57,
+ 0x2a, 0x74, 0xd0, 0x46, 0x64, 0xbf, 0xc9, 0xe8,
+ 0x35, 0x6b, 0x5c, 0x79, 0x9a, 0x51, 0xeb, 0xd1,
+ 0xb3, 0x1d, 0xfd, 0x2f, 0x83, 0x2e, 0x24, 0x95,
+ 0x94, 0xc7, 0x9d, 0x9f, 0x5d, 0x5c, 0x8e, 0x6c,
+ 0x8f, 0xfc, 0xce, 0x89, 0x67, 0x9b, 0x38, 0x4b,
+ 0x0d, 0x44, 0x9a, 0xb5, 0x74, 0xbf, 0x88, 0x90,
+ 0xa1, 0xf2, 0xc4, 0xd2, 0xb3, 0x07, 0xe5, 0x34,
+ 0x62, 0xc0, 0xfc, 0x3f, 0xdb, 0x77, 0x8e, 0x84,
+ 0x61, 0xc1, 0x15, 0xf6, 0x5f, 0x5d, 0x73, 0xbc,
+ 0x0c, 0x70, 0x32, 0xeb, 0x2c, 0xd3, 0x19, 0x83,
+ 0x22, 0xfa, 0x5b, 0xcc, 0x5f, 0xbf, 0xbe, 0xfc,
+ 0x3c, 0x28, 0x74, 0x2d, 0x33, 0x8f, 0x74, 0xe5,
+ 0xca, 0xc9, 0x84, 0x8f, 0xba, 0x48, 0x11, 0x84,
+ 0x33, 0xf0, 0x7d, 0x5c, 0xad, 0x47, 0x36, 0xf7,
+ 0x33, 0xb0, 0xbb, 0x44, 0x65, 0x72, 0x45, 0xd1,
+ 0x40, 0x4c, 0x39, 0x8b, 0x16, 0x90, 0x74, 0xa3,
+ 0xd2, 0x3c, 0xc6, 0x7a, 0xa4, 0x4f, 0x0d, 0x9a,
+ 0xe1, 0xfd, 0x49, 0x82, 0xc6, 0x69, 0xf7, 0xe7,
+ 0x3d, 0xeb, 0x2b, 0xbe, 0xfd, 0x1a, 0x73, 0x2c,
+ 0xf3, 0xbc, 0xd6, 0xaa, 0xaa, 0x54, 0x22, 0x5c,
+ 0x54, 0x92, 0x73, 0xe3, 0x79, 0x0a, 0x87, 0x20,
+ 0x82, 0xdb, 0x02, 0x42, 0x81, 0xb1, 0xf7, 0x0d,
+ 0xf0, 0xd8, 0x56, 0x0f, 0x74, 0x0c, 0x65, 0x7c,
+ 0x8c, 0x96, 0xb9, 0x9f, 0xfa, 0xb6, 0x48, 0x66,
+ 0xf1, 0xbe, 0xf8, 0x46, 0x70, 0xe5, 0x26, 0xc1,
+ 0x68, 0x7f, 0x81, 0x72, 0x14, 0x2f, 0x96, 0x22,
+ 0x05, 0xd1, 0xc0, 0xb8, 0x93, 0x9d, 0x02, 0x8b,
+ 0xff, 0x0b, 0x2d, 0xb9, 0x01, 0xaa, 0x61, 0x09,
+ 0x9d, 0x55, 0x21, 0xdd, 0xda, 0xec, 0x44, 0xce,
+ 0x00, 0x98, 0xf8, 0x34, 0x86, 0x49, 0xd4, 0x72,
+ 0x90, 0xfb, 0xd7, 0xb1, 0x33, 0xb9, 0xe5, 0xd3,
+ 0xc0, 0x27, 0x88, 0x4b, 0x6d, 0x84, 0x34, 0x7c,
+ 0x77, 0x7e, 0xcf, 0x27, 0x4d, 0x5f, 0x1d, 0xb4,
+ 0xb4, 0xac, 0xcc, 0x7d, 0x83, 0xc2, 0x89, 0x3d,
+ 0xeb, 0xb6, 0xa3, 0x57, 0x30, 0xbc, 0x94, 0x9f,
+ 0xe9, 0x63, 0xca, 0x7d, 0xa3, 0x9d, 0x0a, 0x43,
+ 0xed, 0x4a, 0xfd, 0xd3, 0x5e, 0x23, 0x97, 0x75,
+ 0x28, 0x97, 0xaa, 0x0c, 0xf9, 0xa1, 0x03, 0xb1,
+ 0x55, 0x82, 0xec, 0xcc, 0x91, 0x27, 0xf4, 0xcf,
+ 0x63, 0x10, 0xf3, 0xd3, 0x8b, 0x5c, 0xa8, 0xc2,
+ 0x58, 0x4d, 0xf5, 0x67, 0xe9, 0xe1, 0xc2, 0xf2,
+ 0x11, 0x5e, 0xf2, 0xe1, 0x1c, 0xf2, 0x0c, 0x50,
+ 0x71, 0x9f, 0xd1, 0x1c, 0xb8, 0x82, 0x7d, 0x3b,
+ 0x23, 0x90, 0x4e, 0x08, 0xbf, 0x51, 0xa1, 0xc9,
+ 0xe9, 0x90, 0x07, 0x4e, 0xb3, 0xcc, 0x72, 0xbf,
+ 0x2c, 0x87, 0x18, 0x3b, 0xc7, 0x29, 0x8a, 0x3b,
+ 0x0b, 0x4c, 0x55, 0x20, 0x08, 0xb0, 0xa3, 0x04,
+ 0xf9, 0x6b, 0x53, 0x84, 0x4c, 0xef, 0xc1, 0xc6,
+ 0xa4, 0xa9, 0xba, 0xe6, 0xb8, 0x72, 0x60, 0x6b,
+ 0x2d, 0x25, 0x08, 0xc1, 0xd4, 0x4e, 0x0b, 0xcb,
+ 0xe9, 0x71, 0x9b, 0x5a, 0x9a, 0x1e, 0x21, 0x9a,
+ 0x0a, 0xd0, 0x52, 0x82, 0xe9, 0x31, 0x45, 0x9e,
+ 0xe4, 0x85, 0x6b, 0xac, 0x8e, 0x7c, 0x73, 0x0c,
+ 0x0d, 0xd6, 0x54, 0x27, 0x5a, 0xce, 0xc8, 0x07,
+ 0x19, 0x84, 0x8b, 0xee, 0x0f, 0x6f, 0x8b, 0xbc,
+ 0x0c, 0x1b, 0xe0, 0x08, 0x64, 0xa6, 0xe6, 0xe1,
+ 0xf8, 0x56, 0x5e, 0xe8, 0x27, 0x7d, 0x26, 0xee,
+ 0x20, 0x5e, 0x3d, 0x2e, 0x84, 0xdc, 0x50, 0xb0,
+ 0x16, 0x3f, 0x8c, 0xec, 0xa9, 0xd3, 0x61, 0x85,
+ 0x5a, 0xba, 0x02, 0x82, 0x65, 0x30, 0xdb, 0x23,
+ 0x1c, 0xcb, 0xc3, 0xe6, 0x80, 0x5e, 0x4f, 0x6e,
+ 0xd8, 0x3b, 0x5e, 0xce, 0x67, 0xbb, 0xdb, 0x22,
+ 0xac, 0xf1, 0x32, 0x53, 0x6a, 0xd6, 0x43, 0x1a,
+ 0xf0, 0x11, 0xc5, 0x0f, 0xc5, 0x12, 0xdf, 0xe6,
+ 0x91, 0x8b, 0x15, 0xf4, 0x41, 0xd9, 0xe3, 0x51,
+};
+
+uint8_t mldsa_44_pubkey[] = {
+ 0xcc, 0x2c, 0x93, 0xce, 0xfc, 0x0b, 0xf6, 0x74,
+ 0x93, 0x28, 0x95, 0xe8, 0xc0, 0xc8, 0x83, 0xb8,
+ 0xc9, 0x0c, 0x9a, 0x5c, 0x18, 0xd2, 0x79, 0x5c,
+ 0xf3, 0x58, 0xd8, 0x02, 0x6c, 0x5e, 0xad, 0x79,
+ 0xde, 0x77, 0x1a, 0xd3, 0x49, 0xc9, 0x12, 0xc9,
+ 0xff, 0xcf, 0xa2, 0x6c, 0x1d, 0x37, 0x91, 0xa2,
+ 0xf5, 0x43, 0xbc, 0xdb, 0x9e, 0xdb, 0xa5, 0xdb,
+ 0xd9, 0x87, 0xc5, 0xf2, 0xf7, 0x79, 0x58, 0x4c,
+ 0x08, 0x9f, 0x9c, 0xca, 0x9c, 0xef, 0xa8, 0x80,
+ 0x27, 0x31, 0xf1, 0xc5, 0x0e, 0x2b, 0x6d, 0xa1,
+ 0x53, 0x9e, 0x99, 0x1e, 0xc8, 0x75, 0x4a, 0x59,
+ 0x27, 0x33, 0x41, 0xdd, 0x39, 0xff, 0x37, 0xdf,
+ 0x5b, 0xb9, 0xe4, 0x56, 0x54, 0x28, 0x8e, 0xb1,
+ 0xfe, 0xd8, 0x4e, 0x60, 0xcd, 0x22, 0xd7, 0x30,
+ 0xe5, 0x73, 0xe4, 0xb1, 0x08, 0x0f, 0x0a, 0xbe,
+ 0x9a, 0x44, 0xfd, 0xb1, 0xed, 0xb1, 0x8e, 0x0d,
+ 0x7c, 0x3c, 0x3d, 0x04, 0x52, 0x4b, 0x93, 0xf4,
+ 0xa6, 0xce, 0x8d, 0xb0, 0xe4, 0xf6, 0xb1, 0x09,
+ 0xfc, 0xc3, 0x42, 0x3d, 0xff, 0x4c, 0x55, 0x3c,
+ 0x73, 0x75, 0x35, 0x9d, 0xe8, 0x68, 0x42, 0x09,
+ 0x14, 0x36, 0x91, 0x63, 0xfc, 0xc6, 0x23, 0x7c,
+ 0x25, 0x81, 0xd5, 0xc1, 0xfe, 0xca, 0xf6, 0x71,
+ 0x51, 0x8e, 0xab, 0x29, 0xa1, 0x86, 0xbb, 0x45,
+ 0x43, 0x67, 0x7f, 0xdf, 0x7e, 0x92, 0xff, 0x35,
+ 0x38, 0xd3, 0xea, 0x94, 0xc9, 0xa3, 0x0f, 0x46,
+ 0x25, 0xa6, 0x1e, 0x00, 0x60, 0x7b, 0xc0, 0xbc,
+ 0xe9, 0x5e, 0x16, 0x0e, 0x81, 0xf5, 0x4e, 0x98,
+ 0xa1, 0x64, 0xb0, 0xb7, 0x02, 0xec, 0x73, 0xad,
+ 0xf8, 0xc1, 0xce, 0x8b, 0x8f, 0xbd, 0x89, 0xbf,
+ 0x0f, 0x42, 0x31, 0x6d, 0x75, 0x42, 0xd7, 0x59,
+ 0x64, 0xad, 0x09, 0xd3, 0x7a, 0x00, 0x7b, 0xdd,
+ 0x12, 0x76, 0xb0, 0x00, 0x73, 0x5b, 0xbf, 0x44,
+ 0x54, 0x6e, 0x56, 0x26, 0xa5, 0x27, 0x4d, 0xff,
+ 0xe5, 0x8a, 0x04, 0x73, 0xd9, 0x75, 0x8c, 0xf7,
+ 0x06, 0x64, 0xfb, 0xa5, 0x00, 0x27, 0x39, 0x0e,
+ 0x48, 0x8f, 0x73, 0x29, 0x61, 0x5f, 0x15, 0xf5,
+ 0x08, 0x15, 0x33, 0xd1, 0x76, 0xba, 0xf3, 0x3e,
+ 0x28, 0xb8, 0x57, 0xcd, 0x9d, 0x61, 0x1f, 0xca,
+ 0xd7, 0xc3, 0x10, 0xdb, 0x68, 0xeb, 0xa4, 0x15,
+ 0x40, 0xd8, 0xe7, 0xa0, 0xd2, 0xd1, 0xd6, 0xb1,
+ 0xd3, 0x75, 0x8e, 0xc1, 0x60, 0x02, 0xb5, 0x69,
+ 0xf2, 0x8c, 0xf3, 0xc6, 0x2e, 0x9d, 0xf7, 0x8d,
+ 0xed, 0xb0, 0x01, 0xba, 0xb5, 0x62, 0x7e, 0x8f,
+ 0x91, 0xbd, 0x73, 0xc6, 0x35, 0xf9, 0xdb, 0xea,
+ 0x28, 0xce, 0x8a, 0x47, 0x74, 0x7c, 0x3d, 0x85,
+ 0x4d, 0x1e, 0x11, 0x7e, 0xa2, 0xaa, 0x04, 0x30,
+ 0x12, 0xce, 0xea, 0xc1, 0xc5, 0x1e, 0x62, 0x7a,
+ 0x21, 0x6a, 0xd5, 0xb8, 0x3c, 0xa0, 0xd6, 0xca,
+ 0xb4, 0x8d, 0xff, 0xc3, 0xfc, 0xf2, 0xf1, 0x3c,
+ 0x87, 0x23, 0xef, 0x68, 0xe4, 0x37, 0x96, 0xb0,
+ 0xbb, 0x78, 0xa3, 0x3a, 0xe0, 0xd0, 0x41, 0xeb,
+ 0x4c, 0x80, 0x55, 0x9b, 0x66, 0x5e, 0x33, 0xea,
+ 0x33, 0x35, 0x79, 0xda, 0x04, 0x81, 0x19, 0x22,
+ 0xb4, 0x4c, 0x7a, 0xef, 0x51, 0x50, 0x20, 0x6c,
+ 0xf8, 0xf7, 0x5f, 0x03, 0x74, 0x72, 0xda, 0x1f,
+ 0xe3, 0x0b, 0x82, 0x71, 0x50, 0x3d, 0x02, 0x61,
+ 0x16, 0x44, 0xeb, 0xcf, 0xe3, 0x23, 0x9d, 0xef,
+ 0xec, 0xce, 0xb3, 0xd0, 0xd2, 0x19, 0xd1, 0xe0,
+ 0x2e, 0x33, 0xde, 0x28, 0x4b, 0xde, 0x85, 0xc1,
+ 0xb2, 0x8a, 0xa5, 0x82, 0x0c, 0xa6, 0x9f, 0x23,
+ 0x53, 0xea, 0xef, 0x78, 0x60, 0x24, 0x3a, 0x40,
+ 0xc0, 0xc1, 0xac, 0xc9, 0x64, 0xd4, 0x03, 0x5f,
+ 0x61, 0xa5, 0xdf, 0x6f, 0x6b, 0x4e, 0xb2, 0xe7,
+ 0x1a, 0xc6, 0x69, 0x69, 0xd9, 0xc6, 0x6e, 0x5d,
+ 0xea, 0xd8, 0xc0, 0x29, 0xcc, 0x8f, 0x35, 0x7b,
+ 0xcb, 0x48, 0x3a, 0xfd, 0xba, 0x7a, 0x90, 0xc7,
+ 0x65, 0x55, 0xfc, 0x90, 0x98, 0x6c, 0x15, 0x38,
+ 0x0e, 0x88, 0x6e, 0x08, 0x49, 0x85, 0xc6, 0x6d,
+ 0x8f, 0x30, 0x37, 0xa1, 0x64, 0x79, 0xa1, 0x65,
+ 0x2d, 0x07, 0x38, 0x3e, 0x1c, 0xa6, 0x12, 0xee,
+ 0xc5, 0x16, 0x68, 0x5c, 0x93, 0x14, 0xda, 0x33,
+ 0x3d, 0x74, 0x62, 0xf1, 0xcf, 0x0e, 0x81, 0x80,
+ 0xda, 0xea, 0xa7, 0x08, 0xde, 0xf3, 0xdc, 0xfe,
+ 0x3a, 0xf5, 0x60, 0xe9, 0x35, 0x95, 0x52, 0x6c,
+ 0x7f, 0xef, 0x8b, 0xa7, 0x55, 0x3d, 0x42, 0xfd,
+ 0x39, 0x24, 0xd2, 0xfb, 0x28, 0x08, 0x83, 0x2b,
+ 0x44, 0xf1, 0x21, 0x73, 0x13, 0x88, 0xa7, 0xba,
+ 0x2d, 0xd6, 0xe9, 0x4e, 0xcb, 0xfe, 0x18, 0xe4,
+ 0xc7, 0x85, 0xf3, 0xb8, 0xab, 0xd0, 0xb9, 0xbe,
+ 0x5e, 0x8a, 0x62, 0x0e, 0x61, 0xa3, 0x29, 0x1d,
+ 0xef, 0x46, 0xe7, 0x70, 0x14, 0xb1, 0x9f, 0x07,
+ 0xbf, 0x3f, 0x1f, 0xc8, 0x11, 0xe4, 0x2c, 0x39,
+ 0x48, 0x0e, 0xcf, 0x92, 0x62, 0x4b, 0x1f, 0x0c,
+ 0x2e, 0x29, 0x61, 0xef, 0x29, 0x67, 0x7c, 0x0f,
+ 0xac, 0x99, 0xb2, 0x9a, 0x7d, 0xac, 0x7c, 0x54,
+ 0xae, 0xed, 0x4b, 0xf5, 0x4a, 0x4b, 0xb4, 0x14,
+ 0xad, 0x6d, 0xce, 0xce, 0xcc, 0x3f, 0xb8, 0x59,
+ 0x71, 0x7c, 0xd4, 0x76, 0x7c, 0xe4, 0x68, 0xc6,
+ 0x76, 0xa8, 0xe3, 0x03, 0x48, 0xb6, 0xe4, 0x11,
+ 0x6c, 0x5f, 0xd4, 0x76, 0x80, 0xdf, 0x2c, 0x67,
+ 0x00, 0x4c, 0xce, 0x22, 0x21, 0xff, 0xb0, 0x9b,
+ 0xcb, 0x72, 0xd1, 0xe3, 0xa2, 0x32, 0x5c, 0x12,
+ 0xda, 0x7c, 0x95, 0x5d, 0xb2, 0xae, 0x1b, 0x25,
+ 0x2e, 0x5f, 0xe6, 0xff, 0x4c, 0xbf, 0x00, 0x16,
+ 0xd0, 0x3c, 0xbf, 0x44, 0x92, 0x07, 0xa0, 0x0d,
+ 0x13, 0xba, 0xb8, 0x62, 0x66, 0x0b, 0x14, 0x0f,
+ 0x5a, 0x9d, 0x42, 0x19, 0xd7, 0xbe, 0xe3, 0x33,
+ 0x53, 0xa2, 0x60, 0x2d, 0xde, 0x43, 0x85, 0x03,
+ 0x96, 0x50, 0x83, 0x83, 0x25, 0x3c, 0x19, 0x20,
+ 0x44, 0x5f, 0x3b, 0x5a, 0xc2, 0x90, 0xe1, 0x2d,
+ 0x47, 0x53, 0x50, 0xeb, 0xf7, 0xc7, 0x46, 0x4c,
+ 0xf2, 0xad, 0xb2, 0x8f, 0x03, 0x4e, 0x82, 0x5d,
+ 0x0f, 0xa0, 0x3d, 0xde, 0x4d, 0x93, 0x34, 0xd0,
+ 0x1a, 0xe2, 0x5d, 0xbc, 0xe5, 0x8d, 0xfb, 0x5a,
+ 0xba, 0xe2, 0x24, 0xec, 0xc8, 0xd2, 0xa3, 0x91,
+ 0xa2, 0xa9, 0x0e, 0x0e, 0xf5, 0xfb, 0x9f, 0xed,
+ 0x69, 0x21, 0x9d, 0x00, 0x92, 0xb5, 0x94, 0x0f,
+ 0x38, 0x94, 0x29, 0xf9, 0xe7, 0xaa, 0xf1, 0xf7,
+ 0x20, 0x10, 0xf0, 0xeb, 0x26, 0x96, 0x5a, 0x0e,
+ 0x99, 0x8d, 0x71, 0xef, 0xb2, 0xa0, 0xf3, 0x38,
+ 0xce, 0xf9, 0x99, 0x6f, 0x96, 0xbb, 0xa5, 0x55,
+ 0x27, 0x5c, 0xf5, 0xf8, 0x63, 0xf7, 0xf8, 0x0a,
+ 0x31, 0xee, 0x01, 0xa8, 0xc9, 0x0c, 0xbd, 0x73,
+ 0x62, 0x21, 0xa7, 0x1c, 0xd1, 0x62, 0xe5, 0xdd,
+ 0x43, 0x95, 0x4d, 0x60, 0x11, 0x65, 0x6c, 0xfa,
+ 0x67, 0x9a, 0x2f, 0x24, 0x1e, 0xbd, 0x10, 0xc4,
+ 0xe5, 0x23, 0x6f, 0x02, 0x76, 0x51, 0xd6, 0xe3,
+ 0xad, 0x88, 0xda, 0xc4, 0xd9, 0x6a, 0x8f, 0xf7,
+ 0xd2, 0x50, 0xdf, 0x9a, 0xad, 0x21, 0xc7, 0x5b,
+ 0x47, 0x83, 0xf6, 0xc8, 0xcb, 0x0a, 0xd5, 0x28,
+ 0x91, 0x3f, 0x18, 0x44, 0x62, 0x81, 0xfb, 0xcf,
+ 0x51, 0x5d, 0xa1, 0xf5, 0x46, 0x74, 0x0d, 0x32,
+ 0x21, 0x55, 0x1b, 0x8a, 0xf3, 0xd0, 0x4f, 0x41,
+ 0x86, 0xc3, 0x55, 0x0b, 0x1c, 0xd9, 0x56, 0x2f,
+ 0xcf, 0xe7, 0x9f, 0x06, 0x73, 0x82, 0xc2, 0x30,
+ 0x7a, 0xc2, 0x01, 0xa2, 0x1e, 0xeb, 0x5d, 0x77,
+ 0x7f, 0xbe, 0xee, 0xf4, 0x8a, 0x1b, 0x3b, 0xe3,
+ 0x2d, 0xd5, 0x3e, 0x3b, 0x40, 0x82, 0xd6, 0x60,
+ 0x86, 0x2f, 0xe9, 0xac, 0xc3, 0x65, 0xa5, 0xa3,
+ 0x8c, 0x54, 0x51, 0x1f, 0x8e, 0x89, 0x19, 0xdd,
+ 0xc5, 0x4a, 0xc0, 0xa2, 0x8b, 0xe5, 0x80, 0xfd,
+ 0xa1, 0xd8, 0xe9, 0x15, 0xa0, 0x94, 0xd7, 0x9e,
+ 0xe2, 0xf1, 0x83, 0x52, 0xf3, 0x0e, 0x34, 0x2f,
+ 0x85, 0x67, 0x49, 0x73, 0x67, 0xb9, 0xb5, 0xd5,
+ 0x0c, 0x1d, 0x03, 0x8e, 0x68, 0x57, 0x8d, 0xd2,
+ 0x33, 0x44, 0x94, 0x28, 0x5b, 0xfa, 0xe0, 0x2b,
+ 0x80, 0xd9, 0x96, 0x85, 0x6e, 0x0a, 0x2a, 0xdd,
+ 0x9b, 0x5b, 0x55, 0x5a, 0x31, 0xb6, 0xfe, 0x3e,
+ 0x0a, 0x41, 0x28, 0x20, 0xa7, 0xde, 0x31, 0xbe,
+ 0x73, 0xcb, 0xbc, 0xbb, 0x50, 0xf8, 0x58, 0x74,
+ 0xb1, 0x6b, 0x3f, 0x9a, 0x6c, 0x5e, 0x02, 0x87,
+ 0x8d, 0x19, 0xa1, 0xd3, 0x6a, 0xc2, 0x91, 0x82,
+ 0x3e, 0x0c, 0x90, 0xb8, 0xa6, 0x95, 0x23, 0x4e,
+ 0xb9, 0x2d, 0x4d, 0x6c, 0xfb, 0xd7, 0x5e, 0xdf,
+ 0xca, 0x06, 0x9b, 0x94, 0xb7, 0xfc, 0xbc, 0xad,
+ 0x39, 0x5d, 0x43, 0xcb, 0x1c, 0x7d, 0x3c, 0x95,
+ 0x3f, 0xf4, 0x47, 0x04, 0x1f, 0xfc, 0x8b, 0x23,
+ 0xe7, 0xcf, 0x24, 0x6f, 0x40, 0x9b, 0xa5, 0x56,
+ 0xcd, 0x69, 0x9e, 0x1f, 0x5a, 0xa0, 0x3b, 0x8f,
+ 0x3e, 0x1c, 0xe7, 0x42, 0xc9, 0x66, 0xee, 0x99,
+ 0x13, 0xb1, 0x4f, 0x65, 0x86, 0xba, 0xf0, 0x87,
+ 0x81, 0x6f, 0x44, 0x08, 0xb4, 0x18, 0xac, 0xb6,
+ 0xd3, 0x96, 0x68, 0xd4, 0x88, 0x89, 0xf2, 0xa2,
+ 0xd6, 0x2f, 0xfe, 0x7e, 0x49, 0xb3, 0x9c, 0xc9,
+ 0xaf, 0xd9, 0x28, 0xa8, 0x96, 0xc9, 0x2d, 0x26,
+ 0x5b, 0xc7, 0xef, 0x66, 0x08, 0xd2, 0x38, 0x4f,
+ 0x10, 0xb3, 0x83, 0x15, 0xf6, 0x00, 0x83, 0x3f,
+ 0x20, 0xfe, 0xa8, 0x44, 0x6b, 0x62, 0x3b, 0x17,
+ 0x39, 0x2e, 0xec, 0x5e, 0x78, 0xbe, 0xec, 0x16,
+ 0x29, 0xa6, 0x79, 0x4c, 0x08, 0x75, 0xc8, 0x78,
+ 0x3e, 0xc0, 0x05, 0xe9, 0xbb, 0x47, 0x94, 0xae,
+ 0xaf, 0xa5, 0xbb, 0xb8, 0x47, 0x31, 0xf7, 0xe9,
+ 0xb8, 0x1e, 0x6c, 0xda, 0x26, 0xc5, 0xf3, 0x26,
+ 0x89, 0xd5, 0x04, 0x23, 0x0b, 0x11, 0x84, 0x8f,
+};
+
+uint8_t mldsa_44_sign[] = {
+ 0x3E, 0xDD, 0xD2, 0x34, 0x62, 0x78, 0xEC, 0x19,
+ 0x9F, 0xAA, 0xEC, 0x89, 0x99, 0x78, 0x31, 0xCB,
+ 0x82, 0x0B, 0xE6, 0x46, 0x8D, 0x24, 0x9A, 0xD3,
+ 0x69, 0xB7, 0x01, 0x85, 0x93, 0x75, 0xD0, 0xBE,
+ 0xEA, 0x1F, 0x76, 0xFB, 0xB4, 0x6D, 0xC9, 0x64,
+ 0xD4, 0x44, 0x68, 0x5D, 0xAD, 0x09, 0x69, 0xFC,
+ 0x31, 0x2F, 0xD1, 0xFA, 0xEF, 0xF2, 0x1D, 0x2E,
+ 0xB6, 0xA5, 0xA9, 0xFD, 0x31, 0x04, 0x68, 0x1E,
+ 0x0B, 0xEC, 0xF4, 0x23, 0x4C, 0x9F, 0xC9, 0xD6,
+ 0x27, 0x8D, 0xE1, 0xAA, 0x29, 0x38, 0x16, 0x65,
+ 0x7E, 0x38, 0x5A, 0x30, 0xFC, 0xD9, 0xBF, 0x63,
+ 0x3B, 0x82, 0xFE, 0x0D, 0x68, 0xD4, 0x52, 0x55,
+ 0xBE, 0x86, 0x69, 0xCF, 0x75, 0x26, 0x2C, 0xB9,
+ 0x1D, 0x66, 0x39, 0x4C, 0x89, 0xAC, 0x36, 0xBF,
+ 0x34, 0x27, 0xCC, 0x7E, 0x6C, 0xC5, 0xBB, 0xFB,
+ 0x78, 0x03, 0x39, 0x61, 0xD8, 0x76, 0x63, 0x6E,
+ 0x6B, 0x68, 0x02, 0x43, 0x44, 0x57, 0x2E, 0x39,
+ 0x9E, 0x9D, 0x64, 0x77, 0x8E, 0x8B, 0x79, 0x36,
+ 0xCE, 0xE6, 0xBC, 0x6D, 0x80, 0xC8, 0x04, 0x81,
+ 0x2A, 0x04, 0xD4, 0xEF, 0x63, 0xE1, 0x3F, 0xC4,
+ 0xC1, 0x54, 0xD5, 0xAB, 0xE4, 0xEC, 0x65, 0xB0,
+ 0xF0, 0x1A, 0xB2, 0x32, 0x9A, 0xB8, 0x51, 0xD4,
+ 0x43, 0xFE, 0x81, 0x4F, 0xBD, 0x5D, 0xEE, 0xDE,
+ 0x24, 0xAC, 0xBC, 0x22, 0x9B, 0x80, 0xB7, 0xE8,
+ 0x22, 0x4B, 0x1B, 0x23, 0x89, 0x8C, 0xFE, 0xE3,
+ 0x30, 0x35, 0xA2, 0x0B, 0x4E, 0x66, 0x64, 0xFE,
+ 0x57, 0x68, 0xCF, 0xF5, 0xE5, 0x11, 0xB9, 0xB6,
+ 0x3A, 0x2B, 0x15, 0x0D, 0xA4, 0x11, 0xE1, 0x01,
+ 0x96, 0x06, 0x5D, 0x47, 0xCC, 0x04, 0x63, 0xB3,
+ 0xC7, 0xDD, 0x0F, 0x4A, 0x0A, 0x90, 0x9C, 0x0C,
+ 0x61, 0x1D, 0x4C, 0x21, 0x32, 0xD6, 0xE9, 0xDD,
+ 0x0F, 0x91, 0xA4, 0xD1, 0x30, 0x14, 0x1C, 0x48,
+ 0xEC, 0xF4, 0x4F, 0x02, 0x7B, 0x1E, 0x25, 0x3A,
+ 0x7C, 0x6B, 0x42, 0x13, 0xF7, 0xBC, 0xB5, 0x02,
+ 0xA9, 0x20, 0x85, 0x21, 0x01, 0x67, 0xC3, 0xDD,
+ 0x6C, 0x6D, 0xD3, 0xC9, 0x6F, 0x13, 0x75, 0xDD,
+ 0x1D, 0xD7, 0xE7, 0xF3, 0x34, 0x17, 0x37, 0xFF,
+ 0xE6, 0x3B, 0xB5, 0x1F, 0xEE, 0x51, 0x73, 0x6D,
+ 0x9E, 0xB7, 0xE2, 0xE7, 0xA1, 0x65, 0xE4, 0x29,
+ 0x8E, 0xBF, 0x66, 0xCE, 0x5E, 0xD4, 0xBA, 0x0C,
+ 0x18, 0x84, 0xBE, 0xAE, 0x9A, 0x17, 0x0D, 0xAE,
+ 0x55, 0x90, 0x7F, 0x72, 0x73, 0xAB, 0x9F, 0x87,
+ 0xCC, 0x3D, 0xCB, 0xE4, 0x38, 0x66, 0x92, 0xEE,
+ 0x6D, 0xE9, 0x0B, 0x8A, 0xE2, 0x5F, 0x68, 0x9D,
+ 0x06, 0xFD, 0xF3, 0x77, 0x4B, 0x50, 0xCD, 0x0E,
+ 0x2B, 0xE1, 0xD3, 0xB4, 0xF4, 0x02, 0xF5, 0x9B,
+ 0x5F, 0x3E, 0x59, 0xD7, 0x57, 0x9D, 0x87, 0x80,
+ 0x60, 0xEB, 0x70, 0xF5, 0x34, 0x56, 0x46, 0x5B,
+ 0xBA, 0x8F, 0x90, 0xAE, 0x9F, 0x6B, 0x43, 0x8C,
+ 0x51, 0x45, 0xD2, 0x16, 0x4C, 0xBA, 0x86, 0xF0,
+ 0xF4, 0xD5, 0x34, 0x6A, 0x3E, 0x5F, 0xAE, 0xBE,
+ 0x95, 0x40, 0xFE, 0x26, 0x4D, 0x5E, 0x60, 0x4E,
+ 0xD9, 0xEB, 0x47, 0x7D, 0x43, 0x63, 0x5F, 0x4B,
+ 0xB1, 0xCE, 0x7E, 0xA2, 0xF0, 0xC9, 0x30, 0x0C,
+ 0xB7, 0x13, 0x43, 0xC0, 0xF8, 0x02, 0x6C, 0xD2,
+ 0x5F, 0xCE, 0x25, 0xF3, 0xAE, 0xC3, 0x2D, 0xC3,
+ 0x13, 0xAE, 0x49, 0x8B, 0x82, 0x82, 0x44, 0xD0,
+ 0x50, 0xF8, 0x00, 0x5E, 0xAC, 0xDE, 0x4B, 0x88,
+ 0xC2, 0x38, 0x5C, 0xC3, 0x65, 0xF4, 0x25, 0x63,
+ 0x29, 0xC3, 0xB0, 0x7A, 0x45, 0x5D, 0x43, 0x89,
+ 0xCA, 0x5A, 0x12, 0x61, 0xD8, 0x92, 0x65, 0x1B,
+ 0x0F, 0xBF, 0x62, 0xB9, 0xA6, 0xF2, 0xA8, 0xA7,
+ 0x06, 0xAC, 0x02, 0xA4, 0xC6, 0x25, 0xD6, 0xC5,
+ 0xD2, 0xDE, 0x87, 0x88, 0x11, 0xBD, 0x7C, 0x87,
+ 0x91, 0xA9, 0x91, 0x60, 0x1F, 0x0F, 0xF0, 0x24,
+ 0xFE, 0xA0, 0xEB, 0xDC, 0x89, 0x68, 0x84, 0x10,
+ 0xD7, 0x55, 0xC1, 0x3A, 0xCA, 0x1F, 0xFF, 0x77,
+ 0x41, 0xA1, 0xDB, 0x13, 0x31, 0xEF, 0x9C, 0xA3,
+ 0xA2, 0x3A, 0x37, 0x1B, 0xF9, 0x46, 0xF8, 0x51,
+ 0x15, 0xB3, 0x0A, 0x12, 0x64, 0x3F, 0xE5, 0xA8,
+ 0x07, 0xA2, 0x57, 0x87, 0x2A, 0x3F, 0xD2, 0x87,
+ 0xDB, 0xC0, 0x33, 0xAC, 0xF5, 0x28, 0x42, 0xC5,
+ 0xD4, 0x20, 0x27, 0xAE, 0x57, 0xAF, 0x6C, 0x74,
+ 0x8A, 0xA0, 0x90, 0xEE, 0x34, 0xE9, 0x19, 0xB9,
+ 0x0E, 0xDF, 0x4D, 0xDE, 0x77, 0x66, 0xC4, 0xBC,
+ 0x99, 0x59, 0x14, 0xCB, 0x9D, 0xBF, 0xBC, 0x5F,
+ 0xFF, 0x2B, 0xA7, 0xED, 0x29, 0x49, 0x13, 0x6C,
+ 0x2B, 0x71, 0x58, 0xC2, 0xC4, 0x67, 0xCB, 0x6A,
+ 0x18, 0x6F, 0x4E, 0x88, 0xB9, 0x76, 0xC8, 0x6B,
+ 0xDC, 0x6A, 0x05, 0xA5, 0x22, 0x31, 0x15, 0xCE,
+ 0x54, 0xDA, 0xE8, 0x0E, 0xED, 0xDF, 0x46, 0x90,
+ 0x93, 0x9A, 0xE6, 0x2B, 0x45, 0xA4, 0x51, 0x42,
+ 0x29, 0x05, 0xA9, 0xFF, 0x29, 0xBE, 0x4F, 0x6E,
+ 0xE7, 0x52, 0x2C, 0x16, 0x3F, 0x95, 0x94, 0x7B,
+ 0xE6, 0xE1, 0xF7, 0x9F, 0x36, 0x1F, 0xEE, 0x46,
+ 0xA9, 0xE3, 0x71, 0x37, 0x08, 0xF7, 0x63, 0xAF,
+ 0x16, 0xB9, 0x43, 0x86, 0xAC, 0xC7, 0x5D, 0x5B,
+ 0x73, 0x38, 0x08, 0x2C, 0xC6, 0x65, 0x02, 0xC8,
+ 0x70, 0x71, 0x01, 0xA7, 0xD3, 0xA6, 0xB9, 0x74,
+ 0xAA, 0x71, 0x3B, 0x1B, 0xEF, 0x84, 0xA9, 0x77,
+ 0x91, 0x82, 0x7F, 0xB7, 0x3D, 0x2E, 0x8B, 0xD5,
+ 0x4D, 0xAD, 0x29, 0xAC, 0x51, 0x9E, 0xDB, 0xF0,
+ 0x4A, 0x1D, 0x29, 0x82, 0x04, 0x9E, 0x03, 0x8C,
+ 0x74, 0x66, 0x12, 0x75, 0xD0, 0x77, 0x07, 0xCC,
+ 0x7E, 0x8D, 0x19, 0x2F, 0x42, 0xFA, 0xB7, 0xBD,
+ 0x7E, 0x77, 0xEF, 0xF1, 0x35, 0xAF, 0x7C, 0xFF,
+ 0x52, 0x5F, 0xDB, 0x03, 0x99, 0x0D, 0x89, 0x8A,
+ 0x60, 0x61, 0x42, 0xC2, 0xE7, 0x33, 0x58, 0xE7,
+ 0x40, 0x90, 0x46, 0x84, 0xD5, 0x4B, 0x30, 0x88,
+ 0xA7, 0xBB, 0x25, 0xDE, 0x02, 0xFE, 0x57, 0x93,
+ 0xFD, 0xD0, 0x6E, 0xA2, 0xFE, 0x38, 0xA3, 0x5E,
+ 0x6C, 0x35, 0x06, 0xC4, 0xC6, 0x8C, 0x9C, 0x37,
+ 0x3B, 0x6A, 0x1D, 0x91, 0xCB, 0x84, 0x03, 0x47,
+ 0x85, 0xC7, 0x1D, 0x82, 0xEE, 0xB5, 0xF8, 0xEC,
+ 0x29, 0xDE, 0x30, 0x76, 0x32, 0x08, 0x6A, 0x75,
+ 0x43, 0x6A, 0xB8, 0x4C, 0x5D, 0x5A, 0x54, 0xC2,
+ 0xE3, 0x58, 0x43, 0x6E, 0x90, 0xCE, 0x51, 0xEB,
+ 0xE2, 0xFD, 0x68, 0xAE, 0xB3, 0xBA, 0x80, 0xB4,
+ 0x03, 0x84, 0x26, 0x4E, 0xAB, 0x7B, 0x15, 0x2D,
+ 0xDE, 0xB0, 0x7B, 0xF3, 0x4D, 0xFB, 0x6A, 0xFC,
+ 0xE3, 0x47, 0xFC, 0x9C, 0x34, 0xC8, 0xB2, 0x10,
+ 0x46, 0xCC, 0x5F, 0x0A, 0xC6, 0xEB, 0x48, 0x47,
+ 0xFF, 0x73, 0x75, 0xD0, 0x7C, 0xBF, 0x44, 0xCB,
+ 0xAE, 0x8A, 0x30, 0x0B, 0x3F, 0x58, 0x11, 0x51,
+ 0x32, 0xD5, 0x7C, 0xCC, 0xB5, 0xDB, 0xD9, 0x21,
+ 0xD0, 0x6C, 0x29, 0xE2, 0xD0, 0xE1, 0xB2, 0xC8,
+ 0x06, 0x29, 0xC0, 0xC8, 0x5B, 0xB2, 0x3A, 0x66,
+ 0x7A, 0x25, 0x13, 0x57, 0x80, 0x14, 0x5B, 0x31,
+ 0xC4, 0x7B, 0x21, 0xB5, 0x50, 0x9C, 0x76, 0x6B,
+ 0x8C, 0x16, 0xF7, 0x95, 0xA1, 0x79, 0x6B, 0xA8,
+ 0x22, 0x13, 0xD5, 0x32, 0x15, 0x3E, 0xFC, 0x55,
+ 0x80, 0xD2, 0x71, 0xD1, 0x59, 0x15, 0xDA, 0xFC,
+ 0x55, 0x75, 0xE7, 0x7E, 0x15, 0x48, 0xD9, 0x73,
+ 0x0D, 0x6A, 0x31, 0x8A, 0x1C, 0x86, 0x4A, 0x31,
+ 0x57, 0x5B, 0x0A, 0x9D, 0xE4, 0x8C, 0x80, 0x7F,
+ 0x0E, 0x07, 0x36, 0x38, 0x7C, 0xEA, 0x3C, 0xBD,
+ 0xBD, 0xD1, 0xAA, 0xE7, 0xE4, 0x43, 0xB2, 0xFC,
+ 0x26, 0x39, 0x4F, 0xCC, 0xE6, 0xB8, 0xD7, 0x90,
+ 0xC3, 0x5F, 0xAA, 0xEA, 0x78, 0xC6, 0xD7, 0x58,
+ 0x15, 0x7D, 0x27, 0xD0, 0x7E, 0x0F, 0x13, 0x0D,
+ 0x47, 0x49, 0x28, 0x5B, 0xD7, 0xBC, 0x41, 0xB5,
+ 0x66, 0x38, 0x29, 0xB4, 0x71, 0xEC, 0xA2, 0xDE,
+ 0xE3, 0xE0, 0x4C, 0x27, 0xB4, 0x2C, 0xEE, 0xF1,
+ 0x5F, 0x37, 0x81, 0xCE, 0x31, 0x42, 0x87, 0x44,
+ 0x39, 0x7B, 0x35, 0xCF, 0xE8, 0x7D, 0x5E, 0xD7,
+ 0xA3, 0x3D, 0xB9, 0x92, 0x95, 0x8C, 0x25, 0xC6,
+ 0xC9, 0xBC, 0x46, 0x4E, 0x03, 0x70, 0x29, 0x4B,
+ 0x79, 0xB8, 0xEF, 0x54, 0x98, 0x8B, 0x9A, 0x45,
+ 0x09, 0x8C, 0x43, 0xD1, 0x9B, 0x29, 0xEA, 0xDE,
+ 0xF2, 0x25, 0x10, 0xA6, 0xF9, 0x2C, 0xC8, 0x90,
+ 0x49, 0xA5, 0x3C, 0xDC, 0xCE, 0xA3, 0x98, 0xF1,
+ 0x4C, 0xC6, 0x3E, 0xE0, 0x21, 0x58, 0x1A, 0x39,
+ 0xDA, 0x50, 0x2A, 0x6A, 0x18, 0x49, 0xC9, 0xA1,
+ 0x9D, 0xF3, 0xF4, 0xFB, 0xDD, 0x6F, 0x8D, 0xF4,
+ 0xFE, 0x61, 0xA0, 0xC6, 0xF5, 0x58, 0x89, 0xAE,
+ 0xEC, 0xC6, 0xE0, 0x88, 0x4A, 0x07, 0x6F, 0x11,
+ 0x72, 0x5A, 0x6D, 0x3E, 0x08, 0x64, 0x3E, 0x23,
+ 0x7D, 0x9A, 0x74, 0xB3, 0xC3, 0xDA, 0xA7, 0x29,
+ 0x0E, 0xC1, 0x37, 0xB0, 0x04, 0x42, 0x47, 0x01,
+ 0x80, 0x4B, 0xC0, 0x35, 0x49, 0x19, 0xB1, 0xDB,
+ 0x51, 0x51, 0x09, 0x90, 0x08, 0xB2, 0xC4, 0x5C,
+ 0xA0, 0xD6, 0x6E, 0x09, 0xFF, 0xA0, 0xC6, 0x4F,
+ 0x32, 0x95, 0x84, 0xBE, 0xAA, 0x4A, 0x0A, 0x36,
+ 0x83, 0xD3, 0x0C, 0xB7, 0xE0, 0xD0, 0x24, 0xE9,
+ 0x0A, 0x2F, 0x0F, 0xB4, 0x3A, 0x2F, 0xCE, 0x34,
+ 0xE5, 0xF0, 0x67, 0x07, 0x72, 0x52, 0x24, 0xE9,
+ 0x9F, 0xBF, 0xDE, 0x40, 0x95, 0xAD, 0x74, 0x41,
+ 0x4A, 0x00, 0x07, 0xEB, 0x14, 0xDD, 0xC4, 0xA2,
+ 0x37, 0x72, 0x83, 0xE7, 0xD6, 0x20, 0xF5, 0xC6,
+ 0x86, 0x97, 0xC5, 0x64, 0x58, 0x1B, 0x57, 0x5F,
+ 0x9D, 0x06, 0x1D, 0xB5, 0x2D, 0x26, 0xBA, 0xE4,
+ 0x49, 0x71, 0x08, 0xBD, 0x21, 0xC4, 0xA9, 0xF3,
+ 0x26, 0x7A, 0x28, 0xD6, 0x81, 0x20, 0xCD, 0x9C,
+ 0xED, 0xB8, 0xDC, 0x45, 0x33, 0xC0, 0x54, 0x5C,
+ 0x3A, 0xF5, 0xC5, 0x1B, 0x19, 0x10, 0xCA, 0xDF,
+ 0x99, 0xD4, 0xEE, 0x25, 0xD6, 0x71, 0x24, 0x4D,
+ 0xFF, 0x14, 0x85, 0x58, 0x07, 0xC8, 0x3F, 0xD6,
+ 0x55, 0x1C, 0xBF, 0xF8, 0x94, 0x31, 0x04, 0x83,
+ 0xEC, 0xD1, 0xF5, 0x20, 0x72, 0xC3, 0xE9, 0xB1,
+ 0x8A, 0x00, 0x0B, 0x94, 0x53, 0x4C, 0x01, 0x8C,
+ 0xDE, 0x80, 0x59, 0x66, 0x72, 0x2D, 0xC5, 0x78,
+ 0xAC, 0x4A, 0xAD, 0x4F, 0x14, 0xC1, 0x78, 0x87,
+ 0x5F, 0xDF, 0xF2, 0x95, 0xCF, 0x8F, 0x3F, 0xA6,
+ 0xCC, 0xAA, 0x1E, 0xD8, 0xA8, 0x37, 0xAD, 0x5B,
+ 0xA5, 0x6D, 0xBC, 0x81, 0xAC, 0xCD, 0xFD, 0x56,
+ 0x9E, 0x8B, 0xB9, 0xEC, 0x7E, 0x82, 0x4E, 0x55,
+ 0x05, 0xFD, 0x9F, 0xDC, 0x0A, 0xF7, 0xBD, 0x3F,
+ 0xEA, 0x42, 0x9D, 0x7E, 0xAB, 0x09, 0xDF, 0x3D,
+ 0xA3, 0x8C, 0x63, 0x78, 0x81, 0x8F, 0x46, 0x9F,
+ 0x12, 0xB8, 0x8A, 0xA5, 0xE4, 0xE9, 0x73, 0x95,
+ 0x37, 0xD3, 0x39, 0xFE, 0x9B, 0x69, 0x58, 0x05,
+ 0x5B, 0x30, 0x09, 0x31, 0x8B, 0xD9, 0xDB, 0xAE,
+ 0x96, 0x93, 0x96, 0xDF, 0xC0, 0x70, 0x1D, 0xFF,
+ 0x5D, 0x60, 0x72, 0x5A, 0xDD, 0xC2, 0x3D, 0xCF,
+ 0x13, 0xA0, 0x36, 0x38, 0xF4, 0x3A, 0x03, 0x15,
+ 0x48, 0xF0, 0xB1, 0x4A, 0x12, 0x72, 0xBA, 0x6B,
+ 0xE2, 0xD6, 0x4E, 0x57, 0x22, 0x87, 0x2F, 0xE7,
+ 0x40, 0x1B, 0x22, 0xF3, 0x4A, 0xE7, 0xB1, 0xA4,
+ 0x28, 0xC7, 0xBB, 0x17, 0x5E, 0x0C, 0x03, 0xE3,
+ 0xF4, 0x67, 0x25, 0xB2, 0xBB, 0xE6, 0x4E, 0xCD,
+ 0x8A, 0x39, 0xC6, 0x05, 0xB7, 0x14, 0x10, 0xF4,
+ 0x36, 0xE9, 0x8A, 0xC3, 0x41, 0xD3, 0x30, 0xAD,
+ 0x79, 0x56, 0xE2, 0xC3, 0x55, 0xFC, 0x05, 0x3B,
+ 0xD0, 0x83, 0x31, 0xE7, 0xA1, 0x85, 0xBE, 0x6C,
+ 0xAF, 0x9D, 0xA6, 0x26, 0xC4, 0x7A, 0x4B, 0xD8,
+ 0x26, 0xA7, 0x59, 0xBB, 0x8D, 0x6E, 0x15, 0x96,
+ 0x84, 0x08, 0x0B, 0xDF, 0x29, 0x2D, 0x74, 0xD9,
+ 0xAD, 0xFE, 0xC1, 0x3A, 0x84, 0x4E, 0xCE, 0x8F,
+ 0x00, 0x9A, 0x50, 0xB8, 0x6F, 0x02, 0xE6, 0xEC,
+ 0xA9, 0xE6, 0x1F, 0x71, 0x6A, 0x48, 0x98, 0x61,
+ 0x3B, 0xC1, 0x37, 0x5A, 0x2E, 0xF3, 0xE0, 0xD6,
+ 0x11, 0x0C, 0x15, 0x39, 0x7B, 0xFB, 0x37, 0xAF,
+ 0x7A, 0xCA, 0xD6, 0x10, 0xDE, 0x95, 0x6A, 0xC6,
+ 0x19, 0xEC, 0x21, 0x51, 0xCE, 0xAE, 0x8F, 0x5A,
+ 0xEE, 0xEF, 0xB2, 0x9C, 0xF7, 0x4F, 0x5A, 0xB4,
+ 0x90, 0xED, 0xB0, 0x64, 0x59, 0x95, 0xC5, 0x32,
+ 0xC1, 0x85, 0xD7, 0x7E, 0x6C, 0xC6, 0x76, 0x11,
+ 0x4B, 0xB5, 0x17, 0x1D, 0xEE, 0x15, 0xFF, 0xD3,
+ 0xC7, 0xBB, 0xAA, 0x3C, 0x5D, 0x4D, 0x03, 0x82,
+ 0xC0, 0xC7, 0xEA, 0xD0, 0xD8, 0x1B, 0xFF, 0x3C,
+ 0x1D, 0x5A, 0x3F, 0xBD, 0x81, 0x66, 0x62, 0x6E,
+ 0xB3, 0xF5, 0x5F, 0xF0, 0x43, 0x90, 0x01, 0x71,
+ 0xDD, 0xB6, 0x0F, 0x60, 0xCE, 0xFB, 0x17, 0x21,
+ 0x5A, 0x7F, 0x0C, 0x69, 0x82, 0x9D, 0x4C, 0xF3,
+ 0x30, 0x1A, 0xF7, 0x1E, 0x85, 0x1A, 0x89, 0x84,
+ 0xF3, 0x4E, 0x8F, 0x15, 0x60, 0x43, 0x6D, 0x3A,
+ 0x5B, 0x07, 0xC0, 0x78, 0x6A, 0x02, 0xB4, 0x98,
+ 0x3D, 0xAB, 0xAC, 0x25, 0x55, 0xC8, 0x49, 0x7E,
+ 0xC9, 0x04, 0x73, 0xAF, 0x0D, 0x17, 0x1A, 0xA6,
+ 0xBE, 0xD8, 0x11, 0x69, 0x4F, 0x17, 0x39, 0xF3,
+ 0x57, 0x53, 0x1F, 0xD5, 0x1B, 0x15, 0x89, 0x53,
+ 0x54, 0xB1, 0x9F, 0xFD, 0x52, 0x92, 0xA8, 0x98,
+ 0xD4, 0x7B, 0xEE, 0x43, 0xC6, 0x31, 0xDE, 0xEC,
+ 0xDC, 0xE5, 0x1D, 0x90, 0x37, 0x63, 0xA1, 0xF8,
+ 0x42, 0x52, 0x8A, 0x73, 0x8B, 0x3D, 0x42, 0x85,
+ 0x64, 0x5C, 0xA9, 0xCC, 0xA1, 0xCD, 0xEB, 0x9E,
+ 0x2A, 0xF5, 0x21, 0x9E, 0x81, 0xC6, 0x2D, 0xD7,
+ 0x22, 0xC2, 0xEE, 0x47, 0x7E, 0xDA, 0x60, 0x26,
+ 0xFA, 0xE9, 0xB1, 0x81, 0x01, 0x77, 0xD3, 0x42,
+ 0xDD, 0x03, 0x65, 0x74, 0x2B, 0x85, 0x49, 0x9A,
+ 0xA5, 0x27, 0x76, 0x49, 0xF1, 0x72, 0x5E, 0xDF,
+ 0xB7, 0xAD, 0x19, 0x1A, 0x58, 0x89, 0x56, 0x5C,
+ 0x61, 0x03, 0xC7, 0x42, 0x84, 0xF2, 0x9E, 0x9E,
+ 0x09, 0x2A, 0x2D, 0x3B, 0xAE, 0x6A, 0x8E, 0x9A,
+ 0x87, 0x70, 0x16, 0x0A, 0xF1, 0xA9, 0xEA, 0xD2,
+ 0xF3, 0x2A, 0xA3, 0xB0, 0x32, 0xD3, 0x28, 0x92,
+ 0x19, 0x02, 0x77, 0xB2, 0x26, 0x1B, 0xC0, 0x59,
+ 0x1A, 0x67, 0x4B, 0x51, 0x04, 0x83, 0x25, 0xF6,
+ 0x4E, 0xA3, 0x92, 0x55, 0xCD, 0x74, 0xF5, 0x23,
+ 0x98, 0xB7, 0xB9, 0x75, 0xA6, 0xDC, 0x4E, 0xD2,
+ 0x93, 0x39, 0x94, 0xA3, 0xCD, 0xF8, 0xD5, 0xA4,
+ 0xA9, 0x3C, 0x97, 0xED, 0x18, 0x54, 0xBB, 0x59,
+ 0x7E, 0x75, 0xE3, 0xEE, 0x5A, 0x36, 0x53, 0x3D,
+ 0x02, 0xFB, 0x47, 0xE6, 0x10, 0x1D, 0xDF, 0x21,
+ 0xDA, 0x44, 0x66, 0x05, 0x81, 0xC4, 0xE9, 0x3B,
+ 0xD5, 0x56, 0xC5, 0xAD, 0x50, 0x7A, 0xE1, 0xAA,
+ 0x91, 0x7F, 0x7C, 0x4E, 0x85, 0xCC, 0xDC, 0x4A,
+ 0xBD, 0x72, 0x1F, 0x3F, 0x21, 0x93, 0x98, 0x21,
+ 0xB8, 0xCC, 0x21, 0x00, 0x24, 0x13, 0xAB, 0x63,
+ 0xC1, 0x31, 0xCA, 0x1C, 0x67, 0x0C, 0x0E, 0x5A,
+ 0x62, 0xC3, 0x1E, 0x6D, 0x63, 0x9B, 0xDA, 0x7F,
+ 0x92, 0xC4, 0x64, 0x8B, 0xC7, 0x80, 0x04, 0x05,
+ 0x36, 0x06, 0xCE, 0xF7, 0x37, 0x3A, 0xA8, 0xDF,
+ 0x2C, 0x61, 0x46, 0x48, 0x2A, 0x5C, 0xDC, 0xBB,
+ 0x34, 0x16, 0xDA, 0x59, 0x0D, 0x5B, 0xE5, 0x08,
+ 0x05, 0xAF, 0x59, 0x86, 0x34, 0x56, 0x88, 0x4F,
+ 0xAF, 0xA0, 0x28, 0xC4, 0xF2, 0x97, 0xC6, 0xF4,
+ 0x7C, 0xFD, 0xA4, 0x20, 0xE4, 0x20, 0x2C, 0xDA,
+ 0x98, 0xAD, 0x4A, 0x96, 0x9F, 0x62, 0xBB, 0x1C,
+ 0xB6, 0x28, 0xB0, 0xD0, 0xEA, 0x3B, 0xB8, 0x3C,
+ 0x8D, 0x4E, 0x97, 0xEA, 0x9E, 0x63, 0xAA, 0xAA,
+ 0x8D, 0x22, 0x61, 0x29, 0x1A, 0x82, 0x29, 0x86,
+ 0x87, 0x62, 0xA4, 0xE4, 0xC0, 0x03, 0x93, 0x55,
+ 0x7D, 0x40, 0xE9, 0x43, 0x14, 0x9C, 0xE2, 0xDA,
+ 0x0A, 0x00, 0x37, 0xEA, 0x80, 0xB8, 0x31, 0x66,
+ 0xAE, 0xD7, 0xF8, 0xB4, 0x24, 0xC7, 0x78, 0xDB,
+ 0x23, 0x66, 0x61, 0xA5, 0x47, 0x10, 0x4A, 0xE8,
+ 0x79, 0xB6, 0xBD, 0x6A, 0xB6, 0x22, 0x6C, 0x7B,
+ 0x19, 0xF3, 0x80, 0x35, 0x9A, 0x15, 0x36, 0x89,
+ 0x2A, 0x2C, 0xED, 0x26, 0xF1, 0x96, 0xFF, 0x30,
+ 0x4D, 0x03, 0x82, 0x2C, 0x69, 0x31, 0x40, 0x40,
+ 0x0C, 0xD1, 0x40, 0x3E, 0xE0, 0xB5, 0x37, 0xA8,
+ 0x6D, 0x68, 0x68, 0x8F, 0x9E, 0xC5, 0x12, 0x05,
+ 0xF5, 0x36, 0x85, 0x5F, 0x8A, 0x52, 0x7B, 0x60,
+ 0xA2, 0xC2, 0x39, 0xAF, 0x87, 0xC8, 0xC4, 0xF2,
+ 0x53, 0x1D, 0xF0, 0x84, 0x32, 0x8A, 0x04, 0x42,
+ 0xBF, 0xCA, 0x1A, 0x08, 0x89, 0x71, 0xCA, 0xC8,
+ 0x11, 0xA6, 0xDC, 0xED, 0x31, 0xFC, 0x41, 0xFC,
+ 0x84, 0x75, 0xA5, 0x98, 0x67, 0x9E, 0xE7, 0x69,
+ 0x9F, 0x3C, 0x42, 0x94, 0x69, 0xA5, 0x1D, 0xC2,
+ 0x55, 0x9D, 0x2E, 0xC9, 0x13, 0x3C, 0x68, 0x94,
+ 0xA9, 0x7E, 0x6A, 0xFC, 0x11, 0xE3, 0x97, 0x0D,
+ 0x59, 0x8F, 0x8D, 0x05, 0x01, 0x4A, 0x74, 0xB6,
+ 0x96, 0x6D, 0x1E, 0x7F, 0x63, 0xA5, 0x54, 0x39,
+ 0x5A, 0x6C, 0xB9, 0x9B, 0xC3, 0x79, 0xCB, 0xD3,
+ 0xC9, 0xE9, 0x37, 0x9E, 0xDC, 0x3B, 0xE2, 0xD2,
+ 0xE6, 0x07, 0x77, 0xFA, 0xD5, 0xCF, 0xD6, 0x42,
+ 0x45, 0x46, 0xFE, 0x06, 0x28, 0x0E, 0x4D, 0xA6,
+ 0x72, 0xE6, 0x6F, 0x39, 0xCD, 0xB9, 0xBF, 0x43,
+ 0xF7, 0xD9, 0x99, 0x9B, 0x46, 0xF1, 0x80, 0xBA,
+ 0x07, 0x4A, 0x0A, 0xC3, 0x9F, 0xD3, 0xCB, 0xA5,
+ 0x85, 0xF7, 0xEE, 0x9F, 0x6D, 0xC0, 0x52, 0xD7,
+ 0x17, 0x27, 0x81, 0x92, 0x98, 0xA0, 0xBB, 0xC9,
+ 0xD7, 0xE4, 0xF0, 0xF8, 0x25, 0x2C, 0x38, 0x3B,
+ 0x44, 0x4E, 0x55, 0x57, 0x5F, 0x86, 0x9A, 0xA1,
+ 0xB2, 0xBB, 0xBF, 0xE4, 0xF4, 0x04, 0x0D, 0x16,
+ 0x34, 0x48, 0x5A, 0x68, 0x72, 0x88, 0x8B, 0x97,
+ 0xA5, 0xAC, 0xBC, 0xC4, 0xC9, 0xCC, 0xF3, 0x03,
+ 0x12, 0x14, 0x2C, 0x3B, 0x3E, 0x3F, 0x4D, 0x52,
+ 0x79, 0x7A, 0x91, 0x96, 0xA8, 0xB3, 0xBB, 0xCC,
+ 0xCF, 0xD7, 0xDF, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x0C, 0x1D, 0x2F, 0x43,
+};
+
+uint8_t mldsa_44_sign_dtrm[] = {
+ 0xC7, 0x9B, 0x98, 0x33, 0xEA, 0xCC, 0xCC, 0x47,
+ 0x51, 0x8F, 0xE1, 0xC0, 0xDA, 0x82, 0x9B, 0xAA,
+ 0x74, 0x0A, 0x14, 0x25, 0x73, 0x53, 0x27, 0x7C,
+ 0x4C, 0x4A, 0xD6, 0xB7, 0x7C, 0x71, 0x25, 0xD8,
+ 0xC5, 0xBA, 0xA8, 0x18, 0x59, 0xC7, 0x64, 0x2E,
+ 0x9A, 0x65, 0xE9, 0x63, 0x83, 0x3D, 0xB8, 0x2B,
+ 0x42, 0x0A, 0x6D, 0x7D, 0xCE, 0x6A, 0xA3, 0x11,
+ 0xD6, 0xB6, 0x54, 0x82, 0x36, 0x38, 0xF1, 0x84,
+ 0x01, 0x3E, 0x26, 0x66, 0xC7, 0x38, 0x6F, 0x41,
+ 0x4C, 0xC2, 0x42, 0xB5, 0xBF, 0xCE, 0x7B, 0x8B,
+ 0xD5, 0x4F, 0x9B, 0x9F, 0x0E, 0xFE, 0x52, 0xAF,
+ 0x6D, 0x39, 0xD8, 0x0A, 0x27, 0x4D, 0xDB, 0x9F,
+ 0x04, 0x25, 0x8C, 0xF5, 0x89, 0x5B, 0xEB, 0xDC,
+ 0x70, 0xBF, 0x23, 0x1D, 0x14, 0x73, 0x9E, 0xFC,
+ 0x41, 0xA0, 0xA2, 0xA5, 0xAA, 0xAB, 0x8E, 0xCD,
+ 0x5E, 0x29, 0xC5, 0x3E, 0xEC, 0x0A, 0xD9, 0x06,
+ 0xEB, 0x8C, 0x18, 0x41, 0x80, 0x91, 0x5D, 0xC9,
+ 0x28, 0xCE, 0xDB, 0x1C, 0x0E, 0x8F, 0xE3, 0xC0,
+ 0xE5, 0xB1, 0x0A, 0x38, 0x06, 0x43, 0x36, 0x0D,
+ 0xF8, 0xE6, 0x2E, 0xCB, 0xEB, 0x51, 0xC8, 0x82,
+ 0xE0, 0x3B, 0xCB, 0xCE, 0x62, 0xC8, 0xE0, 0x8A,
+ 0x2C, 0xE4, 0xC7, 0xD7, 0xC1, 0x8A, 0x64, 0x0D,
+ 0xE8, 0x5A, 0x52, 0x83, 0x58, 0x93, 0x2A, 0x25,
+ 0x92, 0x65, 0x19, 0x4F, 0x0F, 0x44, 0x1C, 0xCE,
+ 0x56, 0x91, 0xB4, 0xC9, 0x8D, 0x9E, 0x6A, 0xD5,
+ 0xCA, 0xE3, 0xA0, 0xE3, 0x46, 0x45, 0x0D, 0x59,
+ 0x14, 0x90, 0x72, 0x67, 0xC5, 0x95, 0xAC, 0x2A,
+ 0xC4, 0x70, 0xD2, 0xCA, 0x40, 0x1D, 0x62, 0x0E,
+ 0x2A, 0xA4, 0x3E, 0xAC, 0x45, 0x00, 0x96, 0xA4,
+ 0xF6, 0xAC, 0xEE, 0x01, 0x19, 0xEB, 0xCC, 0x8C,
+ 0x62, 0xE3, 0x26, 0x59, 0xB7, 0xB9, 0xBF, 0xBB,
+ 0x8D, 0x72, 0xAD, 0x1A, 0xA2, 0x8D, 0xB1, 0x1B,
+ 0xC1, 0xC1, 0xD8, 0xFF, 0x03, 0xAF, 0x74, 0xDF,
+ 0xC6, 0x3C, 0xCE, 0xBD, 0xA8, 0x12, 0xFA, 0xAC,
+ 0x83, 0x8C, 0x7E, 0x26, 0x8E, 0x0D, 0x03, 0xF3,
+ 0x02, 0x17, 0x23, 0x13, 0x25, 0x01, 0x1F, 0xB9,
+ 0x3B, 0x1C, 0x3E, 0xA7, 0xD9, 0xE4, 0xE6, 0xCA,
+ 0xD6, 0x0C, 0x91, 0xAC, 0xEE, 0xD4, 0x42, 0xA7,
+ 0xB1, 0x62, 0x70, 0xC8, 0xC9, 0x2F, 0x0A, 0xE0,
+ 0xA8, 0xC7, 0x22, 0xE5, 0x2C, 0x06, 0x1C, 0x09,
+ 0x4B, 0x45, 0x25, 0x02, 0x07, 0x8A, 0x86, 0xCC,
+ 0xF8, 0xF9, 0x36, 0x11, 0x67, 0x75, 0xEE, 0xD2,
+ 0xD4, 0x67, 0x09, 0xFD, 0x37, 0xB4, 0x3F, 0x7B,
+ 0x4A, 0x2B, 0x4E, 0x05, 0x37, 0x5E, 0xFD, 0x97,
+ 0x79, 0x95, 0x13, 0x95, 0x0D, 0x75, 0x15, 0x94,
+ 0x27, 0x36, 0xB3, 0x97, 0x03, 0xBE, 0x38, 0x14,
+ 0xCB, 0x4D, 0x16, 0x49, 0xFF, 0x36, 0xFA, 0xED,
+ 0x4D, 0x97, 0x45, 0xE4, 0xFB, 0xF9, 0x61, 0x7D,
+ 0x1F, 0xCE, 0xC3, 0xBF, 0xB3, 0xEA, 0x3C, 0xBC,
+ 0x13, 0x05, 0xBB, 0xAB, 0x44, 0x84, 0x80, 0x0F,
+ 0xB9, 0x1F, 0x6E, 0x88, 0x27, 0x4A, 0xFA, 0x17,
+ 0x83, 0x76, 0x5F, 0xC5, 0xE8, 0x85, 0xBC, 0xFF,
+ 0xD8, 0x48, 0x4A, 0x9F, 0xEB, 0xC3, 0xF8, 0x41,
+ 0x86, 0x4D, 0xC2, 0x75, 0x7B, 0x74, 0x71, 0x68,
+ 0x9C, 0xF8, 0x05, 0x8B, 0xB7, 0xFA, 0xE0, 0xA7,
+ 0xFA, 0xEC, 0x53, 0x91, 0xD0, 0xB4, 0x44, 0x7A,
+ 0x42, 0xAD, 0xD2, 0x17, 0x86, 0x8F, 0x96, 0xF0,
+ 0xF1, 0xD6, 0xC0, 0x26, 0x34, 0x83, 0x2E, 0x62,
+ 0x0D, 0x3F, 0x13, 0x5A, 0x8C, 0xBC, 0x6D, 0x8D,
+ 0x39, 0x05, 0x91, 0x10, 0x32, 0xBE, 0x17, 0x8A,
+ 0xC7, 0x23, 0xBE, 0x3F, 0xC5, 0x8A, 0xE0, 0x81,
+ 0xFC, 0x0D, 0x4C, 0x62, 0xE8, 0x67, 0xBA, 0x8B,
+ 0xBD, 0xA5, 0xCB, 0xE9, 0x34, 0x3D, 0x8D, 0x84,
+ 0x9D, 0x65, 0xBF, 0x9E, 0xC4, 0xB3, 0x16, 0x2E,
+ 0x12, 0xB9, 0x1F, 0x82, 0xA1, 0x46, 0x26, 0xD5,
+ 0xAB, 0x7D, 0x80, 0x4D, 0x9C, 0xD3, 0xEB, 0x77,
+ 0xFB, 0x67, 0xEA, 0xEE, 0xA8, 0x7B, 0x17, 0x02,
+ 0x2E, 0x6E, 0xF5, 0x11, 0x44, 0xFC, 0x31, 0x01,
+ 0xCC, 0x03, 0x94, 0x0A, 0xCF, 0x8D, 0x12, 0xCD,
+ 0x87, 0x66, 0x38, 0x62, 0x3C, 0xBB, 0x98, 0x2E,
+ 0xC5, 0x58, 0xD0, 0xF3, 0x06, 0xBD, 0x49, 0x7E,
+ 0x18, 0x7D, 0x64, 0xEF, 0x37, 0x71, 0x87, 0xD4,
+ 0xE6, 0x2C, 0xCC, 0x32, 0x59, 0x65, 0x05, 0x45,
+ 0xDE, 0xF3, 0x17, 0x62, 0x2C, 0x25, 0x62, 0x9C,
+ 0x67, 0xBD, 0x74, 0xF1, 0x54, 0x0E, 0x0B, 0x4A,
+ 0x01, 0x9F, 0x5E, 0x3B, 0x08, 0x57, 0xFA, 0x09,
+ 0x99, 0x11, 0x2C, 0x3E, 0x1D, 0x2D, 0xF7, 0x91,
+ 0x12, 0x14, 0xFF, 0x95, 0x6F, 0xCA, 0xBD, 0x25,
+ 0xEC, 0x3C, 0x98, 0xA7, 0xAB, 0x98, 0x0E, 0xFB,
+ 0x01, 0xE9, 0x5E, 0x58, 0x5A, 0x49, 0x3F, 0xB2,
+ 0xC4, 0xBC, 0xF3, 0xEC, 0xCA, 0x4C, 0x54, 0x27,
+ 0x66, 0x10, 0x60, 0x76, 0x5C, 0xFB, 0x9E, 0xAC,
+ 0xD2, 0x09, 0x75, 0xB3, 0x2B, 0x65, 0xE7, 0x93,
+ 0xAD, 0x2B, 0x08, 0x6F, 0x87, 0x73, 0x45, 0x2A,
+ 0x1B, 0xB1, 0x75, 0xE7, 0x02, 0x9B, 0xCB, 0x56,
+ 0x87, 0x6F, 0x66, 0x44, 0xED, 0xDC, 0x03, 0x92,
+ 0xD4, 0xF1, 0x07, 0x40, 0xD6, 0x86, 0xD5, 0x98,
+ 0x3D, 0x9B, 0x03, 0xD9, 0x0A, 0x58, 0x59, 0x54,
+ 0x06, 0x10, 0xFD, 0x15, 0xFB, 0xE0, 0x5B, 0x81,
+ 0x8D, 0x0C, 0xC0, 0xA6, 0x15, 0x94, 0x72, 0x5E,
+ 0xC8, 0x4A, 0x73, 0x59, 0x63, 0xF6, 0xCD, 0x8D,
+ 0xA4, 0xC2, 0xAF, 0x2E, 0xAE, 0xC1, 0x93, 0x89,
+ 0x08, 0x81, 0x10, 0xEC, 0x44, 0x81, 0x33, 0x96,
+ 0x88, 0xDC, 0x19, 0xDF, 0xD4, 0x2E, 0x5D, 0x52,
+ 0xD7, 0x6A, 0x02, 0x64, 0x11, 0x6B, 0x24, 0xCF,
+ 0x51, 0xD4, 0xC1, 0x85, 0x7D, 0xA3, 0x24, 0x6A,
+ 0x05, 0x13, 0x55, 0xC3, 0x64, 0x5A, 0xDF, 0x9A,
+ 0x23, 0x73, 0xEA, 0xD9, 0x48, 0x98, 0xBE, 0x1C,
+ 0xBD, 0x3A, 0x5D, 0x37, 0xDC, 0x18, 0xA8, 0x39,
+ 0x71, 0x67, 0xC4, 0x12, 0xF1, 0xE3, 0x53, 0x73,
+ 0xF1, 0xD8, 0xC4, 0x8B, 0xB3, 0x64, 0x03, 0xDD,
+ 0x6E, 0xFB, 0x4A, 0xEF, 0xA4, 0x37, 0xA8, 0xA5,
+ 0x2E, 0x86, 0x40, 0x73, 0x16, 0xBA, 0x3A, 0xF5,
+ 0x5E, 0x47, 0xD8, 0x13, 0x58, 0xB7, 0x73, 0xA2,
+ 0xDA, 0x00, 0x9F, 0x90, 0xE4, 0xF8, 0x0B, 0xB3,
+ 0x26, 0x0A, 0xD8, 0x46, 0x8A, 0x09, 0x55, 0xCB,
+ 0xCB, 0xA4, 0x89, 0x84, 0xF8, 0x35, 0xF0, 0xDF,
+ 0x45, 0x5B, 0x6A, 0xAF, 0xA8, 0x5F, 0xE4, 0xC2,
+ 0x03, 0x9B, 0x5B, 0x11, 0x21, 0xA7, 0xDE, 0xC5,
+ 0x71, 0x3D, 0x00, 0xF8, 0x6F, 0x13, 0x75, 0x4E,
+ 0x7B, 0x6B, 0x32, 0x95, 0xD3, 0x77, 0x82, 0xA2,
+ 0x70, 0x6A, 0x80, 0xC3, 0x0D, 0x53, 0xC4, 0x1C,
+ 0xF0, 0x1C, 0x5A, 0x54, 0xA4, 0xB2, 0x7A, 0x90,
+ 0x6E, 0x48, 0x39, 0x19, 0x4A, 0x70, 0x8E, 0xEB,
+ 0xF2, 0x94, 0x07, 0x45, 0xC0, 0x4B, 0xC1, 0x80,
+ 0xED, 0x87, 0x39, 0x6B, 0x5F, 0x7E, 0x63, 0x95,
+ 0x38, 0xFB, 0xF4, 0x6E, 0x3A, 0x6A, 0x78, 0x59,
+ 0x1E, 0x51, 0x45, 0x02, 0x14, 0xAC, 0xA1, 0xD1,
+ 0xC8, 0x44, 0x03, 0xB6, 0x3D, 0xA4, 0xB3, 0xC1,
+ 0xE0, 0x1C, 0x39, 0xA1, 0x87, 0x21, 0x7C, 0xBA,
+ 0x9A, 0x92, 0xA3, 0xAF, 0x37, 0xE8, 0x27, 0xA1,
+ 0xBE, 0x2D, 0x83, 0x0A, 0x46, 0xE7, 0x68, 0x44,
+ 0x9D, 0xB1, 0x57, 0x33, 0x9B, 0x57, 0xE3, 0x0C,
+ 0xE7, 0xDF, 0x7F, 0x15, 0xC0, 0xB5, 0x3A, 0x08,
+ 0x6C, 0xEF, 0xB3, 0xD3, 0xA2, 0x3C, 0x84, 0xC8,
+ 0x77, 0x46, 0xB8, 0xD0, 0x1E, 0x6D, 0x7A, 0x0B,
+ 0xD7, 0x25, 0x77, 0x74, 0xF6, 0xD8, 0xF4, 0x75,
+ 0x54, 0x6A, 0x91, 0x48, 0x98, 0xAC, 0x52, 0xA6,
+ 0x6C, 0xBC, 0x05, 0xA4, 0x92, 0x8F, 0x52, 0xAA,
+ 0x97, 0xCC, 0x23, 0xD8, 0x69, 0x01, 0xB0, 0xD2,
+ 0x1F, 0x22, 0xE8, 0xD1, 0xF7, 0x89, 0x8E, 0x40,
+ 0xF1, 0xD8, 0xCE, 0xB5, 0x26, 0x82, 0xBD, 0xD7,
+ 0x7C, 0x88, 0xB2, 0x6F, 0x16, 0x07, 0x67, 0x51,
+ 0x45, 0x6C, 0x90, 0x63, 0xBA, 0x58, 0xB6, 0x71,
+ 0xB8, 0x54, 0xA8, 0x4C, 0x54, 0xB3, 0xD7, 0x2B,
+ 0x9F, 0xEB, 0x1C, 0xC6, 0x9F, 0xFA, 0xD6, 0xD3,
+ 0xED, 0xA0, 0x99, 0x18, 0x99, 0x23, 0xB0, 0x1A,
+ 0xAA, 0x3A, 0x42, 0xCB, 0x4A, 0xFC, 0x54, 0x30,
+ 0x86, 0xD0, 0xFF, 0x44, 0x62, 0xAB, 0x94, 0xBD,
+ 0xA9, 0x95, 0x27, 0x22, 0xD6, 0x57, 0xE7, 0x26,
+ 0x94, 0x18, 0x0C, 0xDE, 0x10, 0x44, 0xE4, 0x1A,
+ 0x55, 0x15, 0x07, 0xDC, 0x33, 0xCF, 0x4D, 0x27,
+ 0x9D, 0xDD, 0x19, 0xF7, 0x81, 0xD5, 0x2A, 0xFA,
+ 0x52, 0xC5, 0xA3, 0xF9, 0x6A, 0x78, 0xE8, 0x95,
+ 0xE5, 0x00, 0x98, 0x46, 0xD0, 0xA5, 0x55, 0x23,
+ 0x0D, 0xED, 0xA3, 0x84, 0x87, 0xAF, 0xB5, 0xBC,
+ 0x10, 0x8B, 0x1F, 0x0A, 0x5E, 0x08, 0x5B, 0xC8,
+ 0x78, 0xBD, 0xBC, 0xB5, 0x29, 0x56, 0x3C, 0xFB,
+ 0x95, 0xC7, 0xFD, 0xDD, 0xB6, 0x2A, 0x66, 0x5A,
+ 0x2B, 0x86, 0xA2, 0x74, 0x33, 0x67, 0xA3, 0x48,
+ 0xE7, 0xFA, 0xCB, 0x08, 0x6B, 0x9D, 0x85, 0x8E,
+ 0x38, 0x14, 0xCB, 0xC2, 0xDA, 0xA7, 0x06, 0x3A,
+ 0x23, 0xE6, 0x80, 0x7D, 0x01, 0xA2, 0xA4, 0x1B,
+ 0x3C, 0xCF, 0x9C, 0xE8, 0x62, 0x4D, 0xF5, 0x81,
+ 0x8C, 0xB9, 0x11, 0x8A, 0x12, 0x07, 0x93, 0xDD,
+ 0xB9, 0x40, 0x32, 0xB4, 0x60, 0x6A, 0xC8, 0x76,
+ 0x95, 0x54, 0x0A, 0x48, 0xE2, 0xFF, 0x8A, 0x3C,
+ 0xA2, 0x17, 0x4A, 0xCF, 0xDF, 0x6A, 0xAF, 0xBE,
+ 0x7A, 0x62, 0x7A, 0x3A, 0x60, 0x0B, 0x64, 0x2D,
+ 0xF0, 0xF9, 0x9C, 0x0B, 0xB2, 0x1C, 0xFD, 0xB6,
+ 0x3F, 0x86, 0x9F, 0x4A, 0xA7, 0xA3, 0x8C, 0xC1,
+ 0xAD, 0xF3, 0xFA, 0x86, 0xEF, 0x3F, 0xD7, 0x86,
+ 0x05, 0xF0, 0x8D, 0xA6, 0xD1, 0xFE, 0xE0, 0xB4,
+ 0x12, 0xC6, 0x35, 0x88, 0xFE, 0x77, 0xDE, 0x4E,
+ 0x36, 0x4C, 0x8A, 0x81, 0x62, 0xC0, 0x30, 0x95,
+ 0xC9, 0x2E, 0xB5, 0xCD, 0x09, 0x8D, 0x14, 0xDF,
+ 0xED, 0x2E, 0x2D, 0xCE, 0x8A, 0x94, 0x7A, 0xC7,
+ 0x12, 0x51, 0x8B, 0xEF, 0x45, 0xE9, 0x7A, 0x5E,
+ 0x1E, 0x51, 0x73, 0x34, 0x51, 0x2B, 0xCB, 0x45,
+ 0x22, 0xE6, 0x6F, 0x62, 0x20, 0xBA, 0xD2, 0x4E,
+ 0xDC, 0x1F, 0xEE, 0x25, 0xFD, 0xA5, 0xD8, 0xD6,
+ 0x4A, 0x24, 0xEA, 0x5E, 0x36, 0xF5, 0x76, 0x76,
+ 0x27, 0x53, 0x99, 0xA1, 0xAF, 0x9E, 0x3C, 0x2A,
+ 0x4C, 0xB2, 0x26, 0x8C, 0xC4, 0x1D, 0x38, 0x0C,
+ 0x3F, 0xC6, 0x34, 0xA8, 0x5B, 0x96, 0x46, 0x4B,
+ 0xE4, 0x17, 0xB8, 0x91, 0xD8, 0x28, 0x43, 0x99,
+ 0xD8, 0xB5, 0x1D, 0x87, 0x66, 0x29, 0xC6, 0x8E,
+ 0x43, 0x07, 0xAE, 0xBC, 0x05, 0xBF, 0xE2, 0xED,
+ 0xDD, 0xCD, 0xDA, 0x65, 0xC9, 0x87, 0x95, 0xB6,
+ 0x4D, 0x64, 0x64, 0x23, 0x5D, 0x19, 0x9D, 0x47,
+ 0xC0, 0xED, 0x36, 0x03, 0x8C, 0x1E, 0xBA, 0xDE,
+ 0xB7, 0x5D, 0xB8, 0x2B, 0x10, 0x35, 0x6F, 0xCD,
+ 0x8E, 0xF8, 0xFA, 0xC0, 0x95, 0x34, 0x5C, 0x6A,
+ 0x49, 0xDB, 0x30, 0x4C, 0xEB, 0x61, 0x27, 0xF5,
+ 0x83, 0x5B, 0xBF, 0x05, 0x1F, 0x56, 0xF0, 0x4B,
+ 0x1C, 0x25, 0x29, 0xA2, 0xC5, 0xF2, 0x46, 0xF6,
+ 0xE9, 0x04, 0xDB, 0x7D, 0x01, 0xDE, 0xE0, 0x76,
+ 0xEC, 0xB0, 0x24, 0x61, 0x41, 0xA4, 0x53, 0x84,
+ 0x90, 0x1B, 0x96, 0x72, 0x00, 0x14, 0x26, 0xED,
+ 0x20, 0x31, 0x5A, 0x24, 0xC9, 0xB2, 0x2A, 0x75,
+ 0xD3, 0x94, 0xC1, 0xFE, 0xCE, 0xC1, 0x57, 0xDE,
+ 0x5E, 0xCD, 0xBE, 0xCB, 0x85, 0x4A, 0x6B, 0x85,
+ 0x28, 0xC3, 0x7B, 0x12, 0x7F, 0x61, 0xDD, 0x22,
+ 0x07, 0x37, 0xFF, 0x70, 0x7A, 0xA1, 0x56, 0xD4,
+ 0x5A, 0x3A, 0x59, 0xAB, 0xC3, 0x65, 0x09, 0x0E,
+ 0x3C, 0x9C, 0x52, 0x34, 0xB2, 0x5D, 0x89, 0x62,
+ 0x89, 0x3E, 0x22, 0x07, 0x56, 0xA2, 0x4D, 0x95,
+ 0xE8, 0x78, 0x15, 0x44, 0x6F, 0xA3, 0xB2, 0x2C,
+ 0x24, 0xF8, 0xA0, 0x54, 0x9E, 0x73, 0x25, 0x0C,
+ 0xB3, 0x2C, 0xFF, 0x2D, 0x9C, 0x2E, 0xE1, 0xD8,
+ 0x70, 0xF1, 0xCA, 0xA4, 0xB6, 0x6E, 0xDF, 0xE5,
+ 0xB4, 0x0F, 0xF7, 0x00, 0x8E, 0x4F, 0x6F, 0xBA,
+ 0x0D, 0x1F, 0x82, 0xF0, 0x7D, 0xC4, 0xED, 0x1E,
+ 0xB6, 0x61, 0xE2, 0x3C, 0xBF, 0x5F, 0x88, 0xE1,
+ 0x37, 0x5E, 0xE7, 0x7D, 0xE2, 0x13, 0xDF, 0x9A,
+ 0x59, 0x6D, 0x85, 0x87, 0x7D, 0xCD, 0x8A, 0x01,
+ 0xA1, 0xA7, 0x10, 0x63, 0xB5, 0xA4, 0xC4, 0xE0,
+ 0x1A, 0x45, 0x4E, 0x91, 0x07, 0x54, 0x62, 0xD6,
+ 0xC2, 0x3F, 0x96, 0x40, 0xF6, 0x8B, 0x12, 0x59,
+ 0xEF, 0x20, 0x9C, 0x35, 0x4E, 0x37, 0x1F, 0xAA,
+ 0x9C, 0x01, 0xA5, 0x59, 0x85, 0x37, 0x9E, 0x4F,
+ 0xDF, 0xFF, 0x40, 0x6E, 0xC2, 0xE6, 0xC6, 0x09,
+ 0x70, 0x1C, 0xC2, 0xCD, 0x11, 0x6B, 0x84, 0xC8,
+ 0xEE, 0x2B, 0xAE, 0xD5, 0x36, 0xFC, 0x83, 0x30,
+ 0xDD, 0x5E, 0x13, 0x6C, 0x49, 0xDB, 0x44, 0xDD,
+ 0x39, 0x58, 0xE7, 0x83, 0xD6, 0x1A, 0x5F, 0xE2,
+ 0xF7, 0x82, 0xDC, 0x1E, 0x80, 0x44, 0xDB, 0x48,
+ 0x62, 0x32, 0x75, 0x3E, 0x6A, 0x98, 0x1A, 0xB2,
+ 0x36, 0xBF, 0xDE, 0x35, 0x6B, 0x20, 0x9F, 0x83,
+ 0xB5, 0xA2, 0x1C, 0x0B, 0x99, 0x9B, 0x49, 0x43,
+ 0xEB, 0x5B, 0x67, 0x03, 0x8A, 0x24, 0x45, 0x8C,
+ 0x15, 0xEE, 0xD2, 0xB2, 0x35, 0xEB, 0xAF, 0x69,
+ 0x0E, 0x69, 0x3B, 0x4D, 0x80, 0x9B, 0x75, 0x43,
+ 0xB4, 0x40, 0x30, 0xCF, 0xD9, 0x76, 0xD4, 0x58,
+ 0xAC, 0x2B, 0x10, 0xE3, 0xF3, 0x48, 0x38, 0xAA,
+ 0x69, 0x18, 0x51, 0x5C, 0xD7, 0xB0, 0x9E, 0x5B,
+ 0x1E, 0x18, 0xD7, 0x59, 0x67, 0x80, 0x4F, 0xCB,
+ 0x40, 0x49, 0xB0, 0x27, 0xCD, 0x15, 0xF7, 0x56,
+ 0x34, 0x0F, 0x5D, 0xD9, 0xC4, 0xCB, 0xF7, 0x83,
+ 0x0D, 0x8B, 0xCF, 0x21, 0x13, 0x1A, 0x49, 0xE6,
+ 0x21, 0x49, 0x0F, 0x67, 0xFF, 0xE6, 0xE7, 0xF4,
+ 0xD7, 0x2C, 0xDA, 0xFD, 0xBD, 0x86, 0xC2, 0x40,
+ 0x6E, 0xB5, 0x05, 0xFC, 0x3E, 0x88, 0xB4, 0xB1,
+ 0xC1, 0x5C, 0x6D, 0x0F, 0x0F, 0x17, 0xE6, 0xE0,
+ 0x96, 0x4B, 0x45, 0xBD, 0x07, 0x20, 0x00, 0x3F,
+ 0xF0, 0xF2, 0xB9, 0xA1, 0x35, 0x01, 0x47, 0x81,
+ 0x1F, 0xA2, 0x0A, 0x66, 0xDB, 0xD7, 0x58, 0x51,
+ 0x88, 0x78, 0x24, 0x6F, 0x7E, 0x68, 0x46, 0xBD,
+ 0xEB, 0x5C, 0xA9, 0xDE, 0x30, 0x34, 0x2B, 0xA3,
+ 0xCF, 0x93, 0x82, 0x14, 0x3D, 0x96, 0x1E, 0xD7,
+ 0x6E, 0x9A, 0x2E, 0x72, 0xD0, 0x49, 0xDB, 0x24,
+ 0xA0, 0x6E, 0x8E, 0xBA, 0x4A, 0x36, 0xCB, 0xF6,
+ 0x9A, 0x46, 0x24, 0xEF, 0x18, 0xA8, 0xD9, 0xD5,
+ 0x93, 0xF0, 0x2B, 0xEC, 0x60, 0x0A, 0x8F, 0x1C,
+ 0xE8, 0x69, 0x56, 0x14, 0x6C, 0x8E, 0x30, 0xFB,
+ 0x8D, 0xCF, 0xE7, 0x92, 0x59, 0x2E, 0x32, 0xEB,
+ 0xAE, 0xA3, 0x99, 0x87, 0x98, 0x49, 0x03, 0xCD,
+ 0x5E, 0xBD, 0xDE, 0xB9, 0xF5, 0xD0, 0xB3, 0xB1,
+ 0x65, 0xAC, 0xC7, 0x42, 0xB4, 0xCF, 0x6A, 0x00,
+ 0x3E, 0x2E, 0xC1, 0x13, 0xDF, 0x5E, 0x2E, 0xEC,
+ 0x30, 0x00, 0xE2, 0xAB, 0xFD, 0x74, 0x06, 0x55,
+ 0x88, 0x07, 0xB0, 0xEF, 0x6D, 0x43, 0x81, 0x15,
+ 0xFC, 0xF6, 0x96, 0x91, 0x3D, 0x2F, 0xFE, 0xBD,
+ 0xB9, 0xC8, 0xB6, 0x81, 0x63, 0xE0, 0xA9, 0x20,
+ 0x93, 0xEC, 0x4B, 0x1E, 0xB9, 0xC2, 0xD0, 0x8C,
+ 0x2E, 0xCB, 0x18, 0x7D, 0x1F, 0x66, 0xA6, 0x96,
+ 0xA9, 0xB8, 0x66, 0x4F, 0xCF, 0xC9, 0xDA, 0x97,
+ 0xCA, 0x07, 0xF1, 0xC8, 0xFB, 0x56, 0x60, 0xA9,
+ 0x25, 0xDB, 0xFC, 0x2D, 0xB6, 0x0A, 0x42, 0x6A,
+ 0x7F, 0xCE, 0x70, 0x91, 0xFF, 0x3B, 0xAC, 0xAB,
+ 0xF2, 0x23, 0x4E, 0x50, 0xE9, 0xE6, 0x2F, 0xCB,
+ 0x98, 0xBA, 0x7D, 0xD2, 0x8A, 0xDE, 0x6E, 0x80,
+ 0x0C, 0xC6, 0xEB, 0xD9, 0x64, 0xD4, 0x59, 0xD5,
+ 0x75, 0x00, 0x18, 0x7F, 0xD6, 0x86, 0xC2, 0x25,
+ 0x98, 0xA9, 0x28, 0x1C, 0x40, 0x46, 0xC6, 0xA5,
+ 0xAF, 0x6E, 0x5D, 0x25, 0x3C, 0x77, 0xF9, 0x43,
+ 0xF3, 0x20, 0xFC, 0x43, 0xA0, 0x5E, 0xF1, 0x65,
+ 0x5D, 0x8B, 0x33, 0x5A, 0x94, 0xF0, 0x6F, 0xB6,
+ 0xD4, 0x4A, 0x48, 0x64, 0x31, 0x73, 0x17, 0x4E,
+ 0x88, 0x90, 0xA8, 0xE4, 0x7F, 0xCE, 0xD5, 0x7D,
+ 0xE3, 0x84, 0x3F, 0x38, 0x71, 0x4C, 0xB1, 0x18,
+ 0xAA, 0x46, 0xF2, 0xCF, 0x99, 0xF0, 0x24, 0xD7,
+ 0xFD, 0xFC, 0x2C, 0x81, 0x25, 0xC1, 0x8E, 0x0B,
+ 0x82, 0x4C, 0x14, 0x6C, 0x44, 0xB1, 0x78, 0x67,
+ 0x14, 0x47, 0xCA, 0x70, 0x0D, 0x13, 0xD1, 0xA8,
+ 0x73, 0xBC, 0x4E, 0xAA, 0x1E, 0xB7, 0x59, 0xC0,
+ 0xAC, 0xE3, 0x21, 0x2B, 0x55, 0x22, 0x6A, 0x53,
+ 0xA6, 0xF0, 0xE9, 0x56, 0x37, 0x3B, 0xD6, 0x1B,
+ 0x2E, 0x57, 0x98, 0x4D, 0x6A, 0x7E, 0xEB, 0x2E,
+ 0x9B, 0xB8, 0xE2, 0x27, 0x55, 0xE2, 0x2E, 0xFA,
+ 0x3C, 0xD2, 0x7A, 0xCC, 0xDB, 0x5C, 0x45, 0x85,
+ 0xA6, 0x92, 0x49, 0x79, 0x9D, 0x18, 0x20, 0x50,
+ 0x7B, 0xF4, 0x0F, 0x43, 0x2F, 0x7B, 0x3E, 0x90,
+ 0xEF, 0xF3, 0x29, 0x66, 0xDF, 0xD1, 0xE9, 0x44,
+ 0xC6, 0x28, 0xAA, 0x48, 0x00, 0x5F, 0x12, 0xEB,
+ 0xEC, 0x26, 0x7F, 0xB3, 0x83, 0xFB, 0x50, 0x5A,
+ 0x5F, 0x5A, 0x8E, 0x08, 0x4B, 0xF7, 0x50, 0x10,
+ 0xC8, 0x73, 0x8F, 0x9C, 0xF6, 0xCB, 0xDC, 0xEA,
+ 0x78, 0x07, 0x2C, 0x59, 0xAD, 0x66, 0x48, 0xFF,
+ 0x56, 0x7D, 0x99, 0xC9, 0xEA, 0x64, 0xE7, 0xD0,
+ 0x46, 0x88, 0x58, 0x46, 0x93, 0x37, 0x70, 0xC1,
+ 0x91, 0x40, 0x02, 0x46, 0x7F, 0x8D, 0xFE, 0x66,
+ 0x07, 0x1F, 0x3B, 0x56, 0x6A, 0x6E, 0x88, 0xB2,
+ 0xBB, 0xC0, 0xC8, 0xCD, 0xD9, 0xDC, 0x0A, 0x0C,
+ 0x1A, 0x27, 0x35, 0x48, 0x4C, 0x5B, 0x5D, 0x90,
+ 0xA0, 0xB2, 0xB4, 0xCE, 0xD6, 0xDD, 0x07, 0x13,
+ 0x7C, 0x93, 0x99, 0xA2, 0xA9, 0xAC, 0xC3, 0xC8,
+ 0xEF, 0x1D, 0x33, 0x3D, 0x3F, 0x42, 0x52, 0x57,
+ 0x6D, 0x72, 0x87, 0x8E, 0x96, 0xA8, 0xC6, 0xC8,
+ 0xD0, 0xD6, 0xE8, 0xEB, 0xF2, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x0E, 0x1E, 0x29, 0x3D,
+};
+
+uint8_t mldsa_44_message[] = {
+ 0x54, 0x68, 0x69, 0x73, 0x20, 0x69, 0x73, 0x20,
+ 0x74, 0x68, 0x65, 0x20, 0x6D, 0x65, 0x73, 0x73,
+ 0x61, 0x67, 0x65, 0x20, 0x74, 0x6F, 0x20, 0x62,
+ 0x65, 0x20, 0x73, 0x69, 0x67, 0x6E, 0x65, 0x64,
+ 0x2E, 0x0A,
+};
+
+uint8_t mldsa_44_privkey_noseed[] = {
+ 0xBA, 0x71, 0xF9, 0xF6, 0x4E, 0x11, 0xBA, 0xEB,
+ 0x58, 0xFA, 0x9C, 0x6F, 0xBB, 0x6E, 0x14, 0xE6,
+ 0x1F, 0x18, 0x64, 0x3D, 0xAB, 0x49, 0x5B, 0x47,
+ 0x53, 0x9A, 0x91, 0x66, 0xCA, 0x01, 0x98, 0x13,
+ 0xC5, 0xC1, 0x61, 0x0A, 0x40, 0x77, 0x4F, 0x0E,
+ 0xBA, 0x33, 0x34, 0xF8, 0xB5, 0xBE, 0x56, 0xE8,
+ 0x78, 0x71, 0xB3, 0xC3, 0xA7, 0x72, 0xC0, 0x72,
+ 0x0F, 0xA3, 0x76, 0x66, 0xAE, 0x17, 0x35, 0xFD,
+ 0xE6, 0xBC, 0x38, 0xA1, 0xC3, 0x5F, 0x8C, 0xF0,
+ 0x8E, 0x44, 0x09, 0x24, 0xC9, 0x03, 0x71, 0x97,
+ 0xBB, 0x87, 0xFD, 0xC4, 0x64, 0x6B, 0x86, 0xDA,
+ 0x5A, 0x05, 0x89, 0xA3, 0x26, 0xCC, 0x0C, 0x0D,
+ 0x95, 0x0F, 0xF8, 0xB5, 0xA9, 0xEA, 0x41, 0x35,
+ 0xEA, 0xB8, 0xA9, 0x3F, 0x80, 0xF0, 0x92, 0x7E,
+ 0x12, 0x40, 0x46, 0xE2, 0x5B, 0x23, 0x66, 0xAE,
+ 0xA2, 0x5A, 0x6D, 0x1D, 0x0F, 0xEF, 0x98, 0x21,
+ 0x04, 0xB8, 0x09, 0xDA, 0x12, 0x48, 0xE1, 0x40,
+ 0x4C, 0x11, 0x03, 0x85, 0x99, 0xB0, 0x4D, 0x61,
+ 0x04, 0x0A, 0xC0, 0x34, 0x28, 0x23, 0x16, 0x80,
+ 0x08, 0x12, 0x45, 0x11, 0x49, 0x0C, 0xA0, 0x96,
+ 0x0D, 0xC2, 0xA8, 0x01, 0x5A, 0xA8, 0x65, 0x0B,
+ 0x30, 0x25, 0xD4, 0x94, 0x44, 0x8B, 0x38, 0x8C,
+ 0x10, 0x24, 0x41, 0x22, 0x40, 0x69, 0x8A, 0x04,
+ 0x04, 0x0C, 0x26, 0x90, 0xE3, 0x34, 0x69, 0xE2,
+ 0x24, 0x68, 0xD2, 0x44, 0x00, 0x0B, 0x84, 0x90,
+ 0xD9, 0x42, 0x30, 0x21, 0x91, 0x8D, 0x09, 0x99,
+ 0x64, 0x20, 0x39, 0x60, 0x82, 0x36, 0x21, 0x90,
+ 0xB8, 0x4C, 0x02, 0x12, 0x6E, 0x24, 0xB9, 0x70,
+ 0x84, 0x22, 0x01, 0xE1, 0xA8, 0x64, 0x44, 0x26,
+ 0x0E, 0x01, 0x17, 0x0E, 0x82, 0x40, 0x6C, 0x9B,
+ 0x38, 0x6D, 0x0A, 0x93, 0x50, 0x52, 0x22, 0x25,
+ 0xD2, 0xB8, 0x2C, 0x1A, 0x25, 0x70, 0x90, 0x12,
+ 0x22, 0xD1, 0xB2, 0x80, 0x22, 0x46, 0x52, 0x01,
+ 0x34, 0x80, 0x40, 0x34, 0x21, 0x41, 0x06, 0x60,
+ 0x41, 0x32, 0x82, 0x09, 0x45, 0x90, 0x52, 0x32,
+ 0x29, 0x83, 0xB8, 0x44, 0x93, 0x12, 0x21, 0x0A,
+ 0xC1, 0x89, 0x60, 0xB2, 0x71, 0x9C, 0x90, 0x69,
+ 0x90, 0x82, 0x90, 0x89, 0x12, 0x49, 0x40, 0x40,
+ 0x8A, 0x5C, 0x10, 0x48, 0x01, 0x87, 0x91, 0x03,
+ 0xB0, 0x40, 0xD8, 0x44, 0x05, 0xE4, 0xB0, 0x29,
+ 0x0A, 0x27, 0x90, 0x1C, 0xC9, 0x50, 0x58, 0x02,
+ 0x8C, 0x0C, 0x49, 0x72, 0x49, 0x84, 0x6D, 0x19,
+ 0xA7, 0x85, 0xE3, 0x00, 0x30, 0xCB, 0xB2, 0x21,
+ 0xE2, 0x08, 0x10, 0x4C, 0xA6, 0x64, 0x11, 0x43,
+ 0x68, 0x52, 0x94, 0x69, 0x09, 0x39, 0x32, 0x91,
+ 0xA2, 0x6D, 0xCA, 0x14, 0x84, 0xA3, 0x92, 0x11,
+ 0xDB, 0x94, 0x2C, 0x01, 0xC6, 0x64, 0x14, 0x24,
+ 0x8C, 0x1A, 0x15, 0x31, 0x0A, 0x23, 0x0E, 0xD8,
+ 0x14, 0x20, 0xA4, 0x26, 0x8D, 0x43, 0x40, 0x90,
+ 0xC1, 0x42, 0x86, 0xA4, 0x10, 0x85, 0x0C, 0x32,
+ 0x2D, 0xD0, 0x28, 0x60, 0x1B, 0xB5, 0x0C, 0x52,
+ 0x44, 0x61, 0x21, 0x15, 0x24, 0x21, 0xC6, 0x04,
+ 0xD8, 0x34, 0x31, 0xD9, 0x12, 0x71, 0xA2, 0x34,
+ 0x52, 0x03, 0x30, 0x89, 0x54, 0x32, 0x10, 0x83,
+ 0x94, 0x08, 0xA2, 0xA4, 0x70, 0xD2, 0x44, 0x8E,
+ 0x84, 0x02, 0x8D, 0x91, 0x46, 0x62, 0x0B, 0x00,
+ 0x0E, 0xA0, 0x80, 0x70, 0x22, 0xB5, 0x0D, 0x0B,
+ 0xC9, 0x01, 0x00, 0x26, 0x4D, 0xE2, 0x44, 0x80,
+ 0x93, 0x20, 0x65, 0xC9, 0x96, 0x8D, 0xDA, 0xA4,
+ 0x4C, 0x42, 0xA8, 0x00, 0x1A, 0x27, 0x6C, 0xC3,
+ 0x94, 0x4D, 0xA0, 0x30, 0x2C, 0xDA, 0xA8, 0x04,
+ 0x23, 0xA3, 0x8C, 0x63, 0x04, 0x0E, 0x59, 0xC8,
+ 0x00, 0x20, 0xA9, 0x6C, 0xC4, 0x06, 0x50, 0x82,
+ 0x00, 0x2E, 0x5C, 0xB2, 0x44, 0x61, 0x46, 0x51,
+ 0x0C, 0x18, 0x80, 0x18, 0x38, 0x82, 0x02, 0x25,
+ 0x90, 0xA2, 0xB2, 0x60, 0x21, 0xA0, 0x40, 0x20,
+ 0x49, 0x2A, 0x52, 0x28, 0x86, 0x93, 0xA6, 0x28,
+ 0x9B, 0x94, 0x71, 0x02, 0xB9, 0x6D, 0x49, 0xA0,
+ 0x2C, 0x09, 0x25, 0x28, 0x18, 0x93, 0x09, 0x10,
+ 0x49, 0x4C, 0x13, 0xC9, 0x91, 0xA3, 0x88, 0x2D,
+ 0x50, 0x12, 0x31, 0x63, 0xB4, 0x00, 0xA4, 0x00,
+ 0x4E, 0x04, 0xB6, 0x68, 0x12, 0x14, 0x26, 0x00,
+ 0x17, 0x00, 0x9B, 0x92, 0x0C, 0x51, 0xA8, 0x65,
+ 0x11, 0x44, 0x8D, 0x44, 0x42, 0x69, 0x00, 0xA5,
+ 0x6D, 0x04, 0xA9, 0x69, 0x43, 0x94, 0x69, 0x4C,
+ 0xB8, 0x60, 0x52, 0x26, 0x2D, 0xA1, 0x32, 0x11,
+ 0xCA, 0xB0, 0x11, 0x1B, 0x81, 0x90, 0x12, 0x11,
+ 0x64, 0x42, 0x86, 0x89, 0x9C, 0x12, 0x51, 0xDC,
+ 0xC0, 0x41, 0x13, 0x47, 0x68, 0x9B, 0x20, 0x92,
+ 0xD1, 0x26, 0x91, 0x23, 0x96, 0x50, 0x13, 0xB3,
+ 0x4C, 0x12, 0x45, 0x69, 0x21, 0xA1, 0x05, 0x0C,
+ 0x46, 0x65, 0x99, 0x38, 0x30, 0x99, 0x26, 0x2E,
+ 0x19, 0x15, 0x72, 0x11, 0x39, 0x8D, 0x89, 0x10,
+ 0x92, 0x50, 0x28, 0x11, 0x48, 0xA6, 0x49, 0x52,
+ 0xB6, 0x2D, 0xC9, 0x30, 0x4D, 0xD9, 0x32, 0x91,
+ 0x91, 0xA6, 0x44, 0x14, 0x34, 0x0E, 0x13, 0x39,
+ 0x41, 0x98, 0x06, 0x0C, 0x22, 0x25, 0x6C, 0x14,
+ 0x47, 0x70, 0x21, 0xB7, 0x60, 0x23, 0x98, 0x25,
+ 0xE4, 0x24, 0x6E, 0xD1, 0xB6, 0x44, 0x51, 0x44,
+ 0x2C, 0x41, 0x42, 0x85, 0xE0, 0x44, 0x42, 0x89,
+ 0x42, 0x42, 0xC2, 0x24, 0x2C, 0xCC, 0xC6, 0x68,
+ 0x13, 0x12, 0x69, 0x98, 0x30, 0x49, 0x23, 0x28,
+ 0x6A, 0x14, 0xC8, 0x11, 0x60, 0x92, 0x30, 0x11,
+ 0x26, 0x52, 0x92, 0xB0, 0x00, 0x5C, 0x84, 0x2C,
+ 0x4A, 0x14, 0x29, 0x0A, 0x95, 0x40, 0xE3, 0xC4,
+ 0x05, 0x0C, 0xC8, 0x89, 0xA1, 0x26, 0x62, 0x44,
+ 0x10, 0x04, 0x9A, 0xC8, 0x70, 0x0C, 0x93, 0x60,
+ 0x90, 0xB6, 0x30, 0x12, 0x17, 0x71, 0x23, 0x18,
+ 0x4E, 0xD4, 0x10, 0x85, 0x84, 0x20, 0x82, 0x5C,
+ 0x14, 0x48, 0x90, 0xB2, 0x51, 0xA0, 0x18, 0x92,
+ 0x93, 0x94, 0x80, 0x1A, 0x49, 0x72, 0xC0, 0x02,
+ 0x06, 0xE1, 0xA6, 0x8C, 0xA1, 0x80, 0x4D, 0x4A,
+ 0x26, 0x08, 0x18, 0x07, 0x62, 0x42, 0x12, 0x40,
+ 0x10, 0x41, 0x64, 0x44, 0x42, 0x4E, 0x94, 0x86,
+ 0x40, 0x9A, 0x84, 0x41, 0x21, 0xC5, 0x71, 0x01,
+ 0x96, 0x29, 0x19, 0x40, 0x4E, 0x94, 0x36, 0x22,
+ 0x24, 0xC3, 0x20, 0x22, 0x88, 0x71, 0x84, 0x20,
+ 0x4C, 0x64, 0x20, 0x69, 0xDB, 0x84, 0x24, 0x24,
+ 0x25, 0x46, 0x21, 0xC4, 0x71, 0x8A, 0x14, 0x64,
+ 0x01, 0x20, 0x65, 0x4C, 0x84, 0x24, 0x44, 0x24,
+ 0x30, 0x8A, 0x22, 0x70, 0xD3, 0x86, 0x30, 0x0B,
+ 0xB4, 0x6C, 0x21, 0x23, 0x0A, 0x8B, 0xB8, 0x4D,
+ 0x5C, 0xF7, 0xD7, 0xE6, 0x89, 0x30, 0x2B, 0xED,
+ 0xB1, 0xC5, 0x86, 0x7E, 0x7D, 0x26, 0x9B, 0x1C,
+ 0xDB, 0x07, 0xF8, 0x25, 0x64, 0x10, 0x82, 0xE1,
+ 0x9A, 0x8D, 0xA2, 0xF9, 0x30, 0x77, 0xE8, 0xB1,
+ 0xFC, 0x3D, 0x4E, 0x6B, 0x2D, 0x32, 0x58, 0x33,
+ 0x6B, 0x4F, 0x9C, 0x64, 0x55, 0x15, 0x3A, 0xC0,
+ 0x40, 0xA8, 0x47, 0xFB, 0x64, 0x7F, 0xBB, 0x6B,
+ 0x55, 0x2A, 0x40, 0x00, 0x71, 0xFE, 0x17, 0x72,
+ 0x48, 0x5B, 0x7A, 0x9D, 0x1F, 0x0D, 0x14, 0x7B,
+ 0xF3, 0x38, 0x8C, 0x56, 0x54, 0x71, 0xE4, 0xE6,
+ 0x2C, 0xC3, 0xCE, 0x0D, 0x0C, 0x0F, 0xC3, 0x60,
+ 0xDF, 0x92, 0x89, 0xED, 0x99, 0x18, 0x37, 0x6B,
+ 0x8B, 0x8B, 0x93, 0x14, 0x50, 0x47, 0xF8, 0xFE,
+ 0xA2, 0x98, 0x60, 0x07, 0xC2, 0xAA, 0x89, 0x92,
+ 0x2F, 0x69, 0xEB, 0x47, 0x5B, 0x59, 0x7B, 0x2B,
+ 0xBA, 0x23, 0x7B, 0x9C, 0x84, 0x2E, 0x3F, 0xF1,
+ 0xD3, 0x25, 0xE8, 0x2A, 0x1F, 0x23, 0xE9, 0x49,
+ 0x89, 0xD0, 0x06, 0xBC, 0x7C, 0xE4, 0x94, 0x6F,
+ 0x2E, 0x8B, 0x77, 0xE1, 0x08, 0x48, 0x46, 0x3C,
+ 0x47, 0xFE, 0x7B, 0x20, 0x9E, 0x2A, 0x61, 0x7D,
+ 0xDD, 0x41, 0x79, 0x6A, 0xE6, 0x14, 0x5E, 0x70,
+ 0x9C, 0xDA, 0x94, 0x06, 0xF2, 0x26, 0x12, 0x57,
+ 0xC2, 0x13, 0xB4, 0xB3, 0x0D, 0xA3, 0x0A, 0xC2,
+ 0x5B, 0x0D, 0x06, 0xCF, 0x79, 0xA8, 0x12, 0xC5,
+ 0xFC, 0xB0, 0xEF, 0x11, 0xD9, 0xFE, 0xDF, 0xE0,
+ 0x99, 0x4A, 0xFE, 0x3B, 0x69, 0xB0, 0x6A, 0x29,
+ 0x16, 0xCF, 0x69, 0x2B, 0x9D, 0xA7, 0x60, 0x28,
+ 0xE5, 0xF3, 0xA0, 0x48, 0x79, 0xE6, 0x96, 0xD2,
+ 0x1F, 0x73, 0x5C, 0x37, 0x83, 0x15, 0x36, 0x4D,
+ 0xB0, 0xA4, 0xE0, 0xAB, 0x6B, 0x53, 0xD3, 0x1E,
+ 0xFA, 0xF3, 0x0D, 0x65, 0xE3, 0x7A, 0x1B, 0x6A,
+ 0x77, 0x04, 0x6F, 0x04, 0xC6, 0x4B, 0xA1, 0x07,
+ 0x2A, 0x97, 0x80, 0xE0, 0xC5, 0x66, 0xC9, 0x43,
+ 0x39, 0xA4, 0xD1, 0x9D, 0x00, 0x68, 0xC5, 0x7D,
+ 0x6E, 0x6F, 0x0B, 0x51, 0x2D, 0xB7, 0x13, 0x4A,
+ 0x95, 0x0E, 0xAF, 0x4F, 0x7B, 0x01, 0xA5, 0xFD,
+ 0xD0, 0x65, 0xB9, 0x1B, 0xFA, 0x29, 0xE4, 0x42,
+ 0x36, 0x79, 0xCD, 0xE7, 0x4B, 0xC6, 0xA8, 0xF1,
+ 0xC8, 0x4C, 0x4D, 0xF7, 0x83, 0x87, 0x23, 0x1D,
+ 0xC8, 0x5C, 0xE3, 0x26, 0x70, 0x44, 0x59, 0x03,
+ 0xC4, 0xBE, 0xBE, 0xE3, 0xF5, 0x0C, 0x43, 0xE5,
+ 0x04, 0x49, 0x49, 0x69, 0x11, 0xAA, 0x93, 0xE7,
+ 0xE3, 0x95, 0x78, 0x74, 0x14, 0xD3, 0x17, 0x68,
+ 0xD9, 0x91, 0x25, 0x20, 0xF8, 0x3C, 0x02, 0xFF,
+ 0x01, 0x12, 0x4D, 0xCF, 0x0E, 0x12, 0x5F, 0xAF,
+ 0xD5, 0xB9, 0xD7, 0xE7, 0xDD, 0xA4, 0xF5, 0xB5,
+ 0x0C, 0x70, 0xAE, 0xBB, 0x85, 0x99, 0xA2, 0xE4,
+ 0x47, 0x6A, 0x0D, 0xE5, 0x31, 0xB0, 0x40, 0x26,
+ 0x72, 0xDF, 0x75, 0x75, 0x14, 0x2D, 0x86, 0x01,
+ 0x60, 0x5C, 0x94, 0x01, 0x79, 0x23, 0xF6, 0x4A,
+ 0xC5, 0x77, 0xC4, 0xBE, 0xD8, 0xD8, 0xE8, 0x9A,
+ 0x74, 0xCA, 0x9F, 0x38, 0x19, 0xCB, 0xF1, 0x42,
+ 0xA7, 0x2D, 0xEB, 0xE7, 0x7C, 0x4E, 0xFB, 0x71,
+ 0x27, 0xE2, 0xD8, 0xC1, 0xB7, 0xBF, 0xB6, 0x42,
+ 0x86, 0xC0, 0xBD, 0x52, 0x23, 0x3F, 0x43, 0xC6,
+ 0x7D, 0x57, 0x17, 0xF9, 0x7A, 0xD8, 0x28, 0x54,
+ 0x87, 0x3D, 0xDC, 0x7F, 0x71, 0xD6, 0x56, 0xAA,
+ 0xA6, 0xEF, 0x70, 0x70, 0x60, 0xAF, 0x28, 0x0B,
+ 0x9F, 0x45, 0x4B, 0x4F, 0xED, 0xB4, 0x77, 0x6E,
+ 0x83, 0xB2, 0xFD, 0xBA, 0x20, 0xA4, 0x5A, 0xEF,
+ 0xEB, 0x54, 0x9A, 0x1E, 0xD0, 0x38, 0x20, 0x21,
+ 0x89, 0x3C, 0xA9, 0xA6, 0xE7, 0x4C, 0xCC, 0x30,
+ 0xA2, 0x55, 0x39, 0x37, 0xCC, 0xEF, 0x34, 0x38,
+ 0x99, 0xB5, 0x02, 0xCF, 0x46, 0xDD, 0xB8, 0xDD,
+ 0x1D, 0x95, 0xFE, 0xFB, 0x60, 0xC9, 0xB2, 0x04,
+ 0x69, 0xA1, 0x50, 0x3B, 0x2A, 0x68, 0x75, 0x87,
+ 0x83, 0x0D, 0x33, 0xCE, 0xE9, 0xA7, 0x2D, 0x79,
+ 0x8F, 0xCF, 0x4A, 0x9B, 0x45, 0x2C, 0x85, 0x49,
+ 0xF5, 0x59, 0xC5, 0xD9, 0xFC, 0x6B, 0xFE, 0x08,
+ 0x3F, 0x44, 0x6C, 0x2D, 0x90, 0x39, 0x81, 0xD9,
+ 0xF2, 0x64, 0x92, 0x48, 0x3A, 0xB4, 0x52, 0xEA,
+ 0x5B, 0xB1, 0x00, 0x8F, 0xFE, 0xAC, 0x97, 0x5D,
+ 0xA0, 0x27, 0x59, 0x59, 0x3E, 0x7E, 0x06, 0x63,
+ 0x61, 0x07, 0x3A, 0x83, 0xB2, 0x7B, 0x53, 0x1A,
+ 0x3D, 0x0D, 0xDA, 0x51, 0x7C, 0xA9, 0x90, 0xEA,
+ 0x32, 0x35, 0xD1, 0xD7, 0xB5, 0xE0, 0x9D, 0xA5,
+ 0xF0, 0x2D, 0xC1, 0x52, 0x5B, 0x1D, 0xA6, 0x85,
+ 0x96, 0x5B, 0x54, 0xFC, 0x2A, 0x3A, 0x73, 0xA1,
+ 0x79, 0x0E, 0x0E, 0xFB, 0x69, 0xE7, 0x0A, 0x78,
+ 0xFA, 0x55, 0x03, 0x44, 0xEA, 0x8C, 0x75, 0x3D,
+ 0xBF, 0x18, 0x63, 0x9B, 0xAA, 0x8C, 0xB1, 0x25,
+ 0x9A, 0xA7, 0x4F, 0x68, 0xF9, 0x2A, 0xBA, 0x80,
+ 0x07, 0xC6, 0x18, 0xCC, 0xB6, 0xF5, 0x06, 0x9F,
+ 0xF4, 0x6B, 0x97, 0x51, 0xBB, 0xFF, 0xF3, 0x7D,
+ 0xF3, 0x21, 0x36, 0x0F, 0x0F, 0x5C, 0x0E, 0x7F,
+ 0x56, 0x26, 0xDD, 0x12, 0x9A, 0xE3, 0xAE, 0x2A,
+ 0x7C, 0x56, 0xCD, 0xB6, 0x11, 0xED, 0xA4, 0xC9,
+ 0x8F, 0xEC, 0x83, 0x16, 0x3C, 0xD5, 0x11, 0x68,
+ 0x78, 0xC1, 0xA9, 0x3E, 0xBA, 0xA2, 0x6D, 0xB4,
+ 0x05, 0xEA, 0xF4, 0xA7, 0xAB, 0xA2, 0x77, 0x83,
+ 0x7D, 0xE9, 0xA5, 0x15, 0x04, 0x70, 0x76, 0x24,
+ 0xEF, 0x2E, 0x1B, 0xBB, 0xCA, 0x29, 0x24, 0x11,
+ 0x16, 0x7F, 0x2E, 0x3D, 0x39, 0x0C, 0x0E, 0x51,
+ 0xF8, 0x4A, 0x2F, 0x13, 0x83, 0x90, 0xE3, 0x3F,
+ 0x85, 0x83, 0x5D, 0x38, 0xA9, 0x4D, 0xBB, 0xE7,
+ 0x1E, 0x6C, 0x82, 0x1E, 0x86, 0xB1, 0x1F, 0xFD,
+ 0x89, 0xEF, 0xF4, 0xBF, 0xE2, 0x08, 0xD6, 0x00,
+ 0x5D, 0x28, 0xF7, 0x04, 0xBA, 0xEA, 0xD1, 0xF2,
+ 0x5D, 0xE0, 0xEB, 0x24, 0x1B, 0x18, 0xFC, 0x7F,
+ 0xA0, 0xDD, 0xD9, 0x0D, 0xC1, 0x39, 0xBE, 0x7F,
+ 0xCB, 0xEB, 0x97, 0x30, 0xFA, 0xE4, 0xB5, 0xD1,
+ 0x72, 0x70, 0xCE, 0x4C, 0x67, 0x0C, 0x42, 0x57,
+ 0x0A, 0x9C, 0xF2, 0x5B, 0xC4, 0xFA, 0xE5, 0xCD,
+ 0x31, 0xE5, 0xD5, 0x5A, 0xD0, 0x22, 0x6A, 0x94,
+ 0xBE, 0x52, 0x94, 0x8C, 0x67, 0x02, 0xA9, 0x86,
+ 0xA0, 0xAD, 0xBF, 0xCD, 0x3A, 0xC4, 0x82, 0xBB,
+ 0x12, 0xAB, 0xBB, 0x79, 0xA2, 0xF6, 0x60, 0x28,
+ 0x42, 0x15, 0x3B, 0x2F, 0x82, 0xA3, 0xB3, 0xCD,
+ 0x16, 0x88, 0xE7, 0x4D, 0x36, 0x53, 0x4B, 0xFF,
+ 0x8C, 0x48, 0xD3, 0xC4, 0x51, 0xEB, 0x2C, 0x5F,
+ 0x98, 0xFE, 0xB9, 0xE7, 0x86, 0x4D, 0x60, 0xAF,
+ 0x96, 0xE8, 0x3B, 0x21, 0x62, 0x46, 0x74, 0x82,
+ 0xF0, 0x58, 0x63, 0x9C, 0x86, 0xA7, 0x85, 0xA9,
+ 0xA1, 0xD4, 0xB6, 0x9B, 0xC3, 0x0E, 0x77, 0xA6,
+ 0x4C, 0x3B, 0xBC, 0xD7, 0xDE, 0xB4, 0xE3, 0xD3,
+ 0x0F, 0x1A, 0x67, 0x21, 0x20, 0x3D, 0x87, 0xA8,
+ 0x8A, 0xB8, 0x5E, 0x02, 0x7A, 0x97, 0x42, 0xFC,
+ 0x68, 0x8F, 0x0A, 0xDF, 0x15, 0x72, 0x8E, 0x59,
+ 0x7E, 0x91, 0x0C, 0xFE, 0x5D, 0xF3, 0x3C, 0x56,
+ 0xA1, 0x36, 0xEF, 0x39, 0xC7, 0xCA, 0x5D, 0x65,
+ 0x0C, 0x2B, 0x9F, 0x90, 0x1C, 0x9B, 0x89, 0xE1,
+ 0xE0, 0x93, 0x54, 0x93, 0x61, 0xF3, 0x03, 0xBE,
+ 0x88, 0x39, 0xD1, 0x45, 0x4C, 0xCE, 0xB5, 0xFB,
+ 0xC4, 0x43, 0x5F, 0xA0, 0xDA, 0xB5, 0x8A, 0x8F,
+ 0xC2, 0x85, 0x36, 0x0E, 0xEA, 0x49, 0x1C, 0xA0,
+ 0x77, 0x96, 0x1C, 0x4A, 0xAA, 0x3E, 0x96, 0xDE,
+ 0x99, 0x71, 0xB9, 0x4F, 0xDF, 0xA5, 0x20, 0x7C,
+ 0xCF, 0x0D, 0x9D, 0xAB, 0x2C, 0x48, 0x96, 0xF0,
+ 0x7E, 0xB6, 0x77, 0x1A, 0x38, 0x3C, 0x65, 0x12,
+ 0xF4, 0x1E, 0xA2, 0x8D, 0xEE, 0xE4, 0x07, 0xFD,
+ 0xAE, 0x3C, 0x57, 0x4F, 0x5D, 0x41, 0x6A, 0x89,
+ 0x7A, 0x27, 0xEF, 0x7C, 0xF5, 0x96, 0xF0, 0x43,
+ 0x2D, 0x62, 0x4A, 0x2C, 0x4E, 0xAC, 0xE5, 0x2F,
+ 0x3C, 0xBF, 0x2C, 0x63, 0x31, 0xB8, 0x0C, 0x9C,
+ 0x91, 0x65, 0xBF, 0x13, 0x34, 0x24, 0x69, 0x32,
+ 0x02, 0x4E, 0xC0, 0xBE, 0x44, 0xB3, 0x21, 0x36,
+ 0xB4, 0xE4, 0x34, 0x27, 0x91, 0x35, 0x85, 0x03,
+ 0x64, 0xC7, 0x57, 0xF1, 0xDC, 0xFA, 0x63, 0x85,
+ 0xE2, 0x56, 0x33, 0x12, 0xC5, 0xF5, 0x53, 0xF0,
+ 0xC8, 0x44, 0xEA, 0xBB, 0x79, 0x11, 0xCE, 0xE7,
+ 0x60, 0xCA, 0xEB, 0x3E, 0x19, 0x3B, 0xF3, 0xA9,
+ 0xC3, 0x81, 0x14, 0x87, 0x23, 0x9A, 0xD2, 0xE0,
+ 0x14, 0x78, 0xF4, 0x6E, 0x41, 0x8A, 0x5D, 0xE5,
+ 0x6B, 0x7F, 0x17, 0x55, 0xBA, 0x68, 0xF9, 0xA3,
+ 0x74, 0x61, 0x3B, 0x5D, 0xE2, 0xED, 0x26, 0xC5,
+ 0x80, 0xC7, 0x72, 0xDB, 0xDB, 0xFA, 0xB1, 0xF7,
+ 0xE3, 0xF5, 0x7D, 0x94, 0xF8, 0x4E, 0x30, 0xDE,
+ 0xB2, 0x9D, 0x70, 0xA9, 0x1D, 0xF2, 0x88, 0xFC,
+ 0x43, 0xA2, 0x76, 0xDF, 0xED, 0x58, 0xE2, 0xB0,
+ 0xDB, 0x53, 0x83, 0xE5, 0x32, 0xB6, 0xEE, 0xDF,
+ 0xB3, 0x92, 0xE4, 0x3D, 0xC3, 0xDA, 0x72, 0x01,
+ 0xA0, 0x68, 0xF5, 0x23, 0x1E, 0xE5, 0x22, 0x09,
+ 0x8D, 0x68, 0x59, 0xB2, 0xD5, 0x64, 0x63, 0xA8,
+ 0x91, 0x7B, 0x3C, 0x25, 0x61, 0x65, 0x79, 0x66,
+ 0xDB, 0xC4, 0x78, 0x56, 0xB6, 0xFF, 0xC8, 0x2B,
+ 0xCC, 0x37, 0x9F, 0xFD, 0x08, 0xB2, 0x59, 0xF3,
+ 0xD9, 0xD7, 0x87, 0x3B, 0xA8, 0xFC, 0xBE, 0x4C,
+ 0x94, 0x13, 0xB6, 0x01, 0x15, 0x91, 0x60, 0x70,
+ 0x1D, 0xF0, 0x04, 0x70, 0xB1, 0x49, 0xBD, 0xF3,
+ 0x2F, 0x4D, 0x3C, 0xFC, 0xFB, 0x9D, 0xEB, 0xC7,
+ 0x72, 0x41, 0x71, 0x7D, 0x13, 0x06, 0x7A, 0xAE,
+ 0xD2, 0x3C, 0x7A, 0x26, 0x51, 0x18, 0x51, 0x69,
+ 0xF1, 0x26, 0x70, 0x61, 0xFB, 0x6B, 0x30, 0xE4,
+ 0xFE, 0xA7, 0x3F, 0x66, 0xF4, 0xF9, 0x27, 0x56,
+ 0xAC, 0x26, 0x23, 0x41, 0x8A, 0xF8, 0xB2, 0xA3,
+ 0x98, 0x71, 0x1B, 0x7C, 0x68, 0x07, 0xB4, 0x34,
+ 0x25, 0xE1, 0xD9, 0x9B, 0xFD, 0xCD, 0x5D, 0xF5,
+ 0x31, 0x95, 0x28, 0x79, 0x06, 0xA3, 0x32, 0xF5,
+ 0x99, 0x71, 0xA0, 0xC3, 0x43, 0x97, 0x5F, 0xC3,
+ 0x20, 0xAD, 0x13, 0x7C, 0x9E, 0x34, 0xCE, 0x7C,
+ 0xE8, 0x55, 0x20, 0xB2, 0x6C, 0xA1, 0x97, 0xA1,
+ 0xFA, 0x2D, 0xF2, 0xEC, 0xD4, 0xE3, 0xFA, 0x83,
+ 0x3B, 0x3B, 0xD2, 0xC2, 0x44, 0x82, 0x80, 0x42,
+ 0x52, 0xCF, 0x1D, 0xF6, 0xAD, 0xC6, 0x39, 0x8F,
+ 0x35, 0xE9, 0x8A, 0xB1, 0x87, 0x10, 0x40, 0x76,
+ 0x80, 0xC9, 0xC1, 0xDB, 0xAC, 0x8C, 0x7E, 0xDC,
+ 0x86, 0x46, 0xB9, 0x70, 0x82, 0xE2, 0xE1, 0x21,
+ 0xFA, 0xA7, 0xFA, 0xC2, 0x1E, 0x4A, 0x33, 0x83,
+ 0x84, 0xCB, 0x92, 0x20, 0x2C, 0x61, 0xBD, 0x12,
+ 0x6C, 0x5D, 0xDD, 0x45, 0x8B, 0x32, 0x7A, 0x18,
+ 0xBD, 0x71, 0x6F, 0x14, 0x2C, 0xA5, 0xCD, 0xB3,
+ 0xE4, 0x9D, 0x7E, 0xB8, 0xD9, 0x62, 0xB5, 0xB8,
+ 0x5A, 0x88, 0xF7, 0x99, 0xB6, 0x9A, 0x6A, 0x66,
+ 0xC7, 0xBD, 0x62, 0x9F, 0x56, 0xB4, 0x3C, 0x02,
+ 0x90, 0x62, 0x9B, 0x5E, 0x27, 0x4C, 0xDE, 0xC7,
+ 0xA0, 0x72, 0x29, 0xE7, 0x93, 0x9A, 0x77, 0xD3,
+ 0x2E, 0x8E, 0xF7, 0x30, 0xFC, 0xCE, 0xAD, 0x9C,
+ 0x4E, 0x06, 0x77, 0xA8, 0x3A, 0x03, 0x30, 0xAB,
+ 0x76, 0x5D, 0x33, 0x6D, 0xD2, 0xAA, 0x15, 0x5D,
+ 0xCD, 0x2A, 0xC7, 0xF3, 0x15, 0x29, 0x77, 0x4F,
+ 0x49, 0x36, 0xB0, 0x5D, 0x0B, 0x14, 0xB4, 0x8F,
+ 0xAA, 0x1E, 0x8C, 0xD4, 0x50, 0x56, 0xE5, 0x6C,
+ 0x13, 0x9B, 0x17, 0xF8, 0x90, 0x71, 0x5A, 0xD6,
+ 0x3D, 0x6C, 0x4A, 0x9F, 0x2D, 0x97, 0x6C, 0x8B,
+ 0x63, 0x5B, 0xDF, 0xE5, 0x86, 0x02, 0x81, 0x6F,
+ 0x61, 0x2C, 0x6E, 0x4B, 0x22, 0x53, 0x67, 0xCB,
+ 0x9A, 0x7B, 0xB7, 0x9C, 0x01, 0x8F, 0x1B, 0x8C,
+ 0x53, 0x15, 0x18, 0x0A, 0xAD, 0xBE, 0x3A, 0xB7,
+ 0x5A, 0xC3, 0x56, 0x20, 0x6F, 0xE2, 0x7C, 0x12,
+ 0xDF, 0x3B, 0x56, 0x97, 0x84, 0xE3, 0xA5, 0x38,
+ 0xFB, 0x05, 0x24, 0x18, 0x26, 0x6E, 0x72, 0xDB,
+ 0x40, 0x0D, 0x6F, 0x32, 0xC4, 0x29, 0x7F, 0x34,
+ 0xF9, 0xF1, 0xAF, 0x18, 0x6C, 0x37, 0x65, 0x65,
+ 0x5F, 0x11, 0xB3, 0xE5, 0xA3, 0xC8, 0x04, 0x9B,
+ 0x7D, 0xF1, 0x40, 0x11, 0xFF, 0x21, 0x5F, 0xBF,
+ 0x17, 0xBF, 0x89, 0xEE, 0x97, 0x6C, 0xF0, 0xDB,
+ 0xAB, 0x62, 0x70, 0x10, 0x4E, 0x7E, 0x31, 0x9D,
+ 0x1F, 0x64, 0xC5, 0x9E, 0x20, 0x9E, 0x35, 0x82,
+};
+
+uint8_t mldsa_44_pubkey_noseed[] = {
+ 0xBA, 0x71, 0xF9, 0xF6, 0x4E, 0x11, 0xBA, 0xEB,
+ 0x58, 0xFA, 0x9C, 0x6F, 0xBB, 0x6E, 0x14, 0xE6,
+ 0x1F, 0x18, 0x64, 0x3D, 0xAB, 0x49, 0x5B, 0x47,
+ 0x53, 0x9A, 0x91, 0x66, 0xCA, 0x01, 0x98, 0x13,
+ 0x1C, 0x44, 0xF8, 0x26, 0xBB, 0xD5, 0x6E, 0x34,
+ 0xE5, 0x5D, 0xB5, 0xE5, 0xE2, 0xD7, 0x33, 0x48,
+ 0x5E, 0x39, 0xEA, 0x26, 0x0F, 0xC6, 0x00, 0x0C,
+ 0x5E, 0xA4, 0xBA, 0x80, 0xD3, 0x45, 0x5C, 0xDE,
+ 0x53, 0xB4, 0x6F, 0x34, 0x48, 0x2A, 0xED, 0xFD,
+ 0x54, 0x50, 0xFC, 0x2E, 0x1B, 0xA4, 0xF2, 0x5D,
+ 0x15, 0xF9, 0xC1, 0x44, 0x24, 0x2F, 0xB3, 0x9B,
+ 0xB5, 0x22, 0x87, 0x18, 0x90, 0x30, 0xC5, 0x04,
+ 0x98, 0xE1, 0x71, 0x7B, 0x7C, 0x75, 0x8B, 0x19,
+ 0x0A, 0x67, 0x48, 0xEA, 0x9A, 0xA3, 0xF7, 0xAC,
+ 0xAA, 0xF2, 0xC7, 0xCB, 0x52, 0x6E, 0xD7, 0x17,
+ 0xC9, 0xF7, 0x9A, 0xEB, 0x84, 0x21, 0x4F, 0xA5,
+ 0xCD, 0x8D, 0xED, 0x92, 0xA0, 0xC3, 0xFA, 0x15,
+ 0x58, 0x81, 0x0F, 0x12, 0xC7, 0x05, 0x0A, 0x36,
+ 0x77, 0x08, 0xD1, 0x96, 0xCD, 0x24, 0xE5, 0xAF,
+ 0x97, 0x49, 0x04, 0xAE, 0xD8, 0xE4, 0xCE, 0x88,
+ 0x72, 0xE8, 0x69, 0x6B, 0x0B, 0x7B, 0xCA, 0x50,
+ 0xE4, 0x52, 0xCD, 0x7D, 0x30, 0xEA, 0x9A, 0x4A,
+ 0xDA, 0xC0, 0x31, 0x1D, 0x67, 0x2C, 0x6B, 0xDE,
+ 0x84, 0x96, 0x24, 0x0B, 0x07, 0x43, 0x14, 0x63,
+ 0x70, 0x88, 0x95, 0xCD, 0x9B, 0xAF, 0xC3, 0x16,
+ 0x32, 0xD7, 0x39, 0x76, 0x49, 0x38, 0x8F, 0xDA,
+ 0xFC, 0xBF, 0x7D, 0x30, 0x5A, 0x3D, 0xE9, 0xA4,
+ 0x95, 0xEC, 0xA7, 0x43, 0x3A, 0x8F, 0x83, 0xBA,
+ 0x0F, 0x0B, 0x25, 0xC4, 0x13, 0xC6, 0xE3, 0x9C,
+ 0x96, 0xEB, 0x7D, 0x69, 0x1B, 0x34, 0xD3, 0x7C,
+ 0xE3, 0x7F, 0x1E, 0xEA, 0xD1, 0xCF, 0x21, 0x7E,
+ 0x25, 0xEF, 0x34, 0xEE, 0xCF, 0x3F, 0x7C, 0x60,
+ 0xF8, 0x4B, 0x8E, 0xDF, 0xDD, 0xE8, 0x40, 0x5D,
+ 0x4F, 0x83, 0x25, 0x76, 0xC6, 0x1E, 0xF9, 0x8E,
+ 0x0A, 0x2F, 0x28, 0xDA, 0x18, 0x77, 0x00, 0x95,
+ 0x39, 0x24, 0xF6, 0x86, 0xB9, 0x46, 0x14, 0x70,
+ 0x5B, 0xCF, 0x53, 0xD3, 0x3F, 0xED, 0xD4, 0x34,
+ 0x8E, 0xDD, 0xDB, 0xDF, 0x28, 0xB5, 0x06, 0x5E,
+ 0x1F, 0x20, 0x77, 0x50, 0x43, 0xE8, 0x5C, 0xF9,
+ 0x31, 0xF8, 0x29, 0x17, 0x93, 0x63, 0xA1, 0xA7,
+ 0xE7, 0x40, 0x4A, 0x83, 0x8E, 0xC0, 0x00, 0x86,
+ 0xB0, 0x97, 0x63, 0x86, 0xFE, 0x63, 0x7C, 0x98,
+ 0x24, 0x47, 0x57, 0xE3, 0xF7, 0x69, 0xDD, 0xD4,
+ 0x46, 0x74, 0x71, 0xBF, 0xAD, 0x67, 0x0F, 0x9A,
+ 0x05, 0xF8, 0x24, 0x6E, 0xE5, 0x0A, 0x7B, 0x1E,
+ 0xAF, 0x87, 0xFC, 0x40, 0x69, 0xC3, 0xAE, 0x2A,
+ 0xA2, 0x03, 0x32, 0x58, 0x11, 0x77, 0x92, 0xF0,
+ 0xBC, 0xD4, 0x9E, 0x08, 0x3F, 0xD1, 0xBC, 0x74,
+ 0x96, 0xAB, 0xFF, 0x29, 0xCC, 0x94, 0xE4, 0x86,
+ 0x8B, 0x21, 0x21, 0x4E, 0xD3, 0x16, 0x52, 0x53,
+ 0x99, 0xA6, 0x10, 0xFB, 0xDD, 0x4A, 0x80, 0xE7,
+ 0xC8, 0x07, 0x15, 0xF2, 0x95, 0x78, 0xE2, 0xA8,
+ 0x4B, 0xB4, 0x0B, 0xDD, 0xDB, 0xD9, 0xF4, 0x7A,
+ 0x11, 0xB6, 0xE7, 0xDA, 0x11, 0x8A, 0x1B, 0x65,
+ 0x8D, 0x35, 0x9E, 0x8A, 0xEF, 0x55, 0xEB, 0x46,
+ 0xB5, 0x37, 0x6B, 0x5B, 0x65, 0x59, 0x79, 0x98,
+ 0x4A, 0x92, 0x2B, 0xEE, 0xBF, 0xC5, 0x9B, 0xCD,
+ 0x60, 0x0D, 0x53, 0x09, 0xDC, 0xCD, 0x72, 0xDB,
+ 0xF0, 0x78, 0x7D, 0xB8, 0xBA, 0x75, 0x7B, 0x53,
+ 0x7C, 0x1E, 0xAF, 0xD5, 0xC0, 0xF5, 0x0E, 0xA4,
+ 0xBC, 0x95, 0x83, 0x54, 0x9E, 0x28, 0x29, 0xA4,
+ 0x2C, 0x28, 0xCA, 0xC2, 0x48, 0xC9, 0x6D, 0x78,
+ 0x12, 0x4C, 0x47, 0x15, 0x9B, 0x18, 0xAE, 0xDD,
+ 0x75, 0x4A, 0xBA, 0x17, 0xB1, 0x9D, 0x43, 0x0F,
+ 0xB7, 0x8F, 0x63, 0x3E, 0xA9, 0xD2, 0x6F, 0x54,
+ 0xA9, 0xBD, 0x50, 0xF8, 0xD8, 0xF6, 0xB7, 0x35,
+ 0x94, 0xF8, 0x28, 0x97, 0x6E, 0x7E, 0xA0, 0x9C,
+ 0x53, 0xBB, 0xB9, 0xF1, 0x1A, 0x56, 0xC9, 0x50,
+ 0x7F, 0xB8, 0x9B, 0x9A, 0x5E, 0xBC, 0x03, 0x7A,
+ 0x37, 0x26, 0x7A, 0x95, 0xF8, 0x5B, 0x8D, 0x64,
+ 0xCA, 0x97, 0x19, 0x2B, 0x10, 0xA6, 0x6F, 0x41,
+ 0x7B, 0x3F, 0x61, 0xFE, 0x9C, 0xA5, 0x71, 0x30,
+ 0xA4, 0x8F, 0xD9, 0x25, 0xEA, 0xE2, 0xAB, 0x55,
+ 0x02, 0xD5, 0x71, 0xC8, 0xA5, 0x19, 0x03, 0xC1,
+ 0xD3, 0x98, 0xF4, 0xC1, 0xF7, 0x6A, 0x7E, 0x11,
+ 0x74, 0x39, 0x76, 0xAF, 0xDB, 0xC6, 0x97, 0xF2,
+ 0x30, 0x94, 0xA3, 0xCD, 0x76, 0x1F, 0xF9, 0x68,
+ 0x5D, 0xE3, 0x2E, 0x09, 0xFB, 0x3C, 0x28, 0xAD,
+ 0xD4, 0x53, 0x49, 0x03, 0x00, 0xBC, 0x7C, 0x89,
+ 0xDC, 0x01, 0x78, 0x00, 0x96, 0x07, 0x17, 0x22,
+ 0x94, 0x57, 0x75, 0xF2, 0x64, 0xE1, 0xB0, 0x62,
+ 0x3B, 0xCF, 0x46, 0x19, 0xC7, 0x12, 0xC8, 0x38,
+ 0x76, 0x12, 0x05, 0xD8, 0x76, 0x91, 0xB7, 0x5E,
+ 0xF3, 0x60, 0x19, 0x6C, 0xBB, 0x9E, 0x9B, 0x92,
+ 0xA0, 0xD4, 0xC4, 0xED, 0x62, 0x32, 0x6E, 0x50,
+ 0x24, 0xD7, 0x75, 0x10, 0xB8, 0xEE, 0x2C, 0x74,
+ 0x26, 0xCC, 0x22, 0xEA, 0xE2, 0x09, 0xDC, 0x9F,
+ 0x13, 0xBD, 0xE6, 0xBF, 0x08, 0xF5, 0xE7, 0x18,
+ 0x1B, 0xD3, 0xB4, 0x59, 0x45, 0x0B, 0x45, 0x1A,
+ 0x51, 0x53, 0x9A, 0x71, 0x5C, 0x21, 0xD6, 0x7D,
+ 0xD3, 0x30, 0xEB, 0x59, 0x70, 0xDB, 0x00, 0xD9,
+ 0xED, 0xBF, 0xB2, 0x82, 0x2B, 0x03, 0x6F, 0xA1,
+ 0x3B, 0xAF, 0xEB, 0x86, 0xD8, 0xDC, 0x78, 0x86,
+ 0x6E, 0x3F, 0x8D, 0x43, 0xE5, 0x3D, 0x78, 0xCC,
+ 0xA5, 0x59, 0x5A, 0x6F, 0xAF, 0x88, 0x6B, 0x5D,
+ 0xC1, 0x12, 0xF1, 0xCF, 0x4A, 0xDC, 0xFA, 0x87,
+ 0x58, 0x00, 0xD9, 0x0B, 0x48, 0x88, 0x3A, 0xF9,
+ 0x73, 0x16, 0xFE, 0x15, 0x06, 0x87, 0x3F, 0xC1,
+ 0x57, 0xE5, 0x70, 0xEA, 0xCB, 0xFD, 0x22, 0x28,
+ 0x68, 0xD1, 0x42, 0x34, 0x10, 0x19, 0x66, 0xAF,
+ 0xB6, 0xBF, 0x99, 0x40, 0x82, 0x92, 0x53, 0xA9,
+ 0x53, 0xAD, 0xA8, 0x9F, 0xC7, 0x56, 0xB6, 0xA8,
+ 0x49, 0xF7, 0x0A, 0xCB, 0x98, 0x38, 0xE6, 0x9F,
+ 0xAA, 0x50, 0xBB, 0xA7, 0x5E, 0x3E, 0x89, 0xC2,
+ 0xAD, 0xB5, 0x7E, 0x86, 0xD0, 0x88, 0xAB, 0x9B,
+ 0x04, 0xA2, 0x8E, 0x67, 0x07, 0x09, 0x17, 0x22,
+ 0x43, 0xEC, 0x5E, 0x00, 0x08, 0xA5, 0xCE, 0xAF,
+ 0x3F, 0x87, 0x22, 0xF4, 0x87, 0x30, 0x25, 0x96,
+ 0xFF, 0xD7, 0x55, 0xAD, 0x1B, 0x82, 0xA4, 0x9C,
+ 0x34, 0xB3, 0x46, 0x95, 0x15, 0xB4, 0x6A, 0xA2,
+ 0x90, 0xCD, 0x86, 0xEE, 0x38, 0xEA, 0x7A, 0x9B,
+ 0xE3, 0xF1, 0x03, 0x61, 0x03, 0x35, 0xB5, 0x31,
+ 0xCC, 0xA3, 0x33, 0xDD, 0xFE, 0x32, 0xB1, 0x45,
+ 0x10, 0xF4, 0xB0, 0x7E, 0xF9, 0x5F, 0xC6, 0x68,
+ 0x4E, 0x8C, 0x45, 0x4A, 0x92, 0xC1, 0x0D, 0xBB,
+ 0x5D, 0x59, 0xC7, 0xA7, 0xC6, 0x3F, 0xB3, 0x05,
+ 0xFE, 0x88, 0x19, 0x67, 0xD9, 0x9E, 0x66, 0x9E,
+ 0xB6, 0x32, 0x84, 0x05, 0x82, 0x56, 0x0B, 0xB4,
+ 0x03, 0x43, 0x1D, 0x40, 0xF7, 0x5A, 0x49, 0x54,
+ 0x90, 0x84, 0x82, 0x27, 0x82, 0x92, 0x82, 0x1F,
+ 0x4E, 0xA9, 0x1E, 0x42, 0xE7, 0x8F, 0xA4, 0x8C,
+ 0xAE, 0xE3, 0xC8, 0x36, 0x14, 0x6D, 0xCF, 0xD7,
+ 0x38, 0xD1, 0x17, 0xE9, 0x2E, 0x9A, 0x15, 0x13,
+ 0x7D, 0x28, 0xE8, 0xE6, 0xA4, 0xB4, 0x62, 0x26,
+ 0x50, 0xCB, 0x41, 0x35, 0x04, 0xCB, 0x3A, 0x33,
+ 0x5D, 0x44, 0xBE, 0xEC, 0x57, 0x46, 0xC1, 0xC2,
+ 0x94, 0xB1, 0xE8, 0xCB, 0x99, 0xCB, 0x60, 0x8D,
+ 0x92, 0x8F, 0x8C, 0xE3, 0x56, 0x36, 0x32, 0xC5,
+ 0x21, 0xF2, 0x3D, 0x13, 0xC6, 0x1A, 0x8F, 0x61,
+ 0xC0, 0x1D, 0xF8, 0xC9, 0x6C, 0x73, 0x60, 0xDB,
+ 0x4F, 0x3C, 0x68, 0xAA, 0x5D, 0x2F, 0xDD, 0x34,
+ 0x2A, 0x62, 0xFF, 0x34, 0x59, 0xC1, 0x16, 0x38,
+ 0x94, 0x21, 0xAB, 0x43, 0xE8, 0x58, 0x4C, 0x45,
+ 0x88, 0x2B, 0x50, 0xE6, 0xE4, 0xE9, 0x6D, 0xB6,
+ 0xF0, 0xB8, 0xFD, 0xE8, 0x90, 0xD5, 0xDB, 0xFA,
+ 0xDC, 0xD8, 0x86, 0x90, 0xB4, 0x49, 0xE6, 0x42,
+ 0x40, 0xDD, 0xB2, 0x02, 0x37, 0x47, 0xF3, 0x08,
+ 0x36, 0x3E, 0x30, 0x1A, 0xA7, 0x77, 0x57, 0x16,
+ 0x9F, 0xC6, 0x15, 0x06, 0x28, 0xD5, 0x92, 0x0B,
+ 0x5A, 0xA1, 0xAB, 0x1C, 0x8C, 0xBF, 0x44, 0xCB,
+ 0x00, 0xE0, 0x25, 0xD7, 0x87, 0x9D, 0x72, 0xB4,
+ 0x79, 0xE3, 0xAF, 0x53, 0x11, 0xC7, 0x85, 0x72,
+ 0x55, 0x90, 0xDA, 0x9C, 0x89, 0xB9, 0xFC, 0x3B,
+ 0x84, 0x50, 0x76, 0x95, 0x54, 0xEB, 0x44, 0xD2,
+ 0x03, 0xEB, 0xA2, 0xBB, 0xAE, 0xF9, 0xCA, 0xD2,
+ 0x23, 0x70, 0x11, 0xC2, 0xEA, 0x44, 0xEF, 0xF0,
+ 0x0F, 0x29, 0x9A, 0x48, 0xFF, 0xE2, 0x8C, 0xA9,
+ 0x3D, 0xDF, 0x85, 0xF7, 0x66, 0x08, 0x24, 0x2E,
+ 0xF8, 0xD6, 0xCC, 0x24, 0x61, 0x0A, 0x1E, 0x20,
+ 0x78, 0xFC, 0xAC, 0x4F, 0x93, 0x85, 0xC3, 0x14,
+ 0x90, 0x5E, 0xCA, 0xA8, 0x2E, 0x55, 0x39, 0x16,
+ 0xD9, 0x4D, 0x1A, 0x7C, 0x1E, 0xC6, 0x52, 0xAA,
+ 0x08, 0x89, 0x70, 0x83, 0xDA, 0xA2, 0xEB, 0xB1,
+ 0x77, 0x5F, 0xBC, 0x47, 0x1A, 0xE2, 0x77, 0x77,
+ 0xD7, 0x90, 0x4E, 0xA9, 0xF1, 0xB9, 0x2B, 0xCA,
+ 0xC3, 0xD8, 0xA3, 0x15, 0x84, 0x26, 0x08, 0x7B,
+ 0x64, 0x5B, 0x11, 0x08, 0xF0, 0xD6, 0x5F, 0xEC,
+ 0x93, 0x78, 0x9C, 0x05, 0x37, 0x43, 0xCA, 0x14,
+ 0xFD, 0x63, 0xD0, 0x5E, 0x98, 0xB6, 0x52, 0xDF,
+ 0x2B, 0x9C, 0x2F, 0xF9, 0xCE, 0x05, 0xF1, 0x94,
+ 0x07, 0x03, 0xFF, 0xB2, 0x73, 0xF8, 0x0E, 0x0E,
+ 0x27, 0x32, 0xEC, 0xA9, 0x96, 0x0D, 0x98, 0x1B,
+ 0x4C, 0xFD, 0x3B, 0x7B, 0xB8, 0x04, 0x5B, 0x3C,
+ 0x38, 0x30, 0x54, 0x6B, 0x9D, 0xD8, 0xDB, 0x0D,
+};
+
+uint8_t mldsa_44_extmu[] = {
+ 0x1B, 0x2A, 0xA5, 0xD5, 0xE3, 0xF3, 0xC1, 0xD4,
+ 0x8D, 0x7C, 0xE6, 0x8F, 0xE9, 0xE3, 0xD1, 0xF5,
+ 0x2E, 0x4B, 0x7C, 0xD6, 0x2A, 0xF4, 0x2E, 0xE5,
+ 0x8F, 0xC3, 0xB1, 0xA4, 0xB9, 0xD1, 0x2C, 0x8B,
+ 0x9E, 0x4D, 0x7A, 0xA9, 0x7C, 0xD3, 0xE6, 0x5A,
+ 0xF4, 0x1B, 0x2A, 0xA5, 0xD5, 0xE3, 0xF3, 0xC1,
+ 0xD4, 0x8D, 0x7C, 0xE6, 0x8F, 0xE9, 0xE3, 0xD1,
+ 0xF5, 0x2E, 0x4B, 0x7C, 0xD6, 0x2A, 0xF4, 0x2E,
+};
+
struct
cperf_rsa_test_data rsa_qt_perf_data[4] = {
{
@@ -4292,6 +5910,49 @@ struct cperf_mlkem_test_data mlkem_decap_perf_data[] = {
}
};
+struct cperf_mldsa_test_data mldsa_sign_perf_data[] = {
+ {
+ .name = "mldsa_44_sign (deterministic)",
+ .type = RTE_CRYPTO_ML_DSA_44,
+ .privkey = {
+ .data = mldsa_44_privkey_noseed,
+ .length = sizeof(mldsa_44_privkey_noseed),
+ },
+ .pubkey = {
+ .data = mldsa_44_pubkey_noseed,
+ .length = sizeof(mldsa_44_pubkey_noseed),
+ },
+ .message = {
+ .data = mldsa_44_message,
+ .length = sizeof(mldsa_44_message),
+ },
+ .sign = {
+ .data = mldsa_44_sign_dtrm,
+ .length = sizeof(mldsa_44_sign_dtrm),
+ },
+ .sign_deterministic = true,
+ },
+};
+
+struct cperf_mldsa_test_data mldsa_verify_perf_data[] = {
+ {
+ .name = "mldsa_44_verify",
+ .type = RTE_CRYPTO_ML_DSA_44,
+ .pubkey = {
+ .data = mldsa_44_pubkey_noseed,
+ .length = sizeof(mldsa_44_pubkey_noseed),
+ },
+ .message = {
+ .data = mldsa_44_message,
+ .length = sizeof(mldsa_44_message),
+ },
+ .sign = {
+ .data = mldsa_44_sign_dtrm,
+ .length = sizeof(mldsa_44_sign_dtrm),
+ },
+ },
+};
+
struct cperf_test_vector*
cperf_test_vector_get_dummy(struct cperf_options *options)
{
diff --git a/app/test-crypto-perf/cperf_test_vectors.h b/app/test-crypto-perf/cperf_test_vectors.h
index e498196ae3..6b4b4f22b4 100644
--- a/app/test-crypto-perf/cperf_test_vectors.h
+++ b/app/test-crypto-perf/cperf_test_vectors.h
@@ -188,6 +188,41 @@ struct cperf_sm2_test_data {
int curve;
};
+struct cperf_mldsa_test_data {
+ const char *name;
+ enum rte_crypto_ml_dsa_type type;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } privkey;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } pubkey;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } message;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } sign;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } ctx;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } seed;
+ struct {
+ uint8_t *data;
+ uint32_t length;
+ } mu;
+ enum rte_crypto_auth_algorithm hash;
+ bool sign_deterministic;
+};
+
struct cperf_test_vector*
cperf_test_vector_get_dummy(struct cperf_options *options);
@@ -215,5 +250,7 @@ extern struct cperf_rsa_test_data rsa_qt_perf_data[4];
extern struct cperf_rsa_plaintext rsa_plaintext;
extern struct cperf_mlkem_test_data mlkem_encap_perf_data[];
extern struct cperf_mlkem_test_data mlkem_decap_perf_data[];
+extern struct cperf_mldsa_test_data mldsa_sign_perf_data[];
+extern struct cperf_mldsa_test_data mldsa_verify_perf_data[];
#endif
diff --git a/app/test-crypto-perf/main.c b/app/test-crypto-perf/main.c
index 2e228201ab..c4d844d225 100644
--- a/app/test-crypto-perf/main.c
+++ b/app/test-crypto-perf/main.c
@@ -55,6 +55,7 @@ const char *cperf_op_type_strs[] = {
[CPERF_ASYM_SM2] = "sm2",
[CPERF_TLS] = "tls-record",
[CPERF_ASYM_MLKEM512] = "mlkem_512",
+ [CPERF_ASYM_MLDSA44] = "mldsa_44",
};
const char *cperf_rsa_priv_keytype_strs[] = {
@@ -247,6 +248,7 @@ cperf_initialize_cryptodev(struct cperf_options *opts, uint8_t *enabled_cdevs)
case CPERF_ASYM_SM2:
case CPERF_ASYM_RSA:
case CPERF_ASYM_MLKEM512:
+ case CPERF_ASYM_MLDSA44:
case CPERF_ASYM_MODEX:
conf.ff_disable |= (RTE_CRYPTODEV_FF_SECURITY |
RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO);
@@ -516,6 +518,21 @@ cperf_verify_devices_capabilities(struct cperf_options *opts,
opts->sm2_data->sign_s.length = sm2_perf_data.sign_s.length;
}
}
+ if (opts->op_type == CPERF_ASYM_MLDSA44) {
+ asym_cap_idx.type = RTE_CRYPTO_ASYM_XFORM_ML_DSA;
+ asym_capability = rte_cryptodev_asym_capability_get(cdev_id, &asym_cap_idx);
+ if (asym_capability == NULL)
+ return -1;
+
+ if (opts->asym_op_type == RTE_CRYPTO_ASYM_OP_SIGN)
+ opts->mldsa_data = &mldsa_sign_perf_data[0];
+ else if (opts->asym_op_type == RTE_CRYPTO_ASYM_OP_VERIFY)
+ opts->mldsa_data = &mldsa_verify_perf_data[0];
+ else {
+ RTE_LOG(ERR, USER1, "Unsupported MLDSA operation type\n");
+ return -ENOTSUP;
+ }
+ }
if (opts->op_type == CPERF_ASYM_MLKEM512) {
asym_cap_idx.type = RTE_CRYPTO_ASYM_XFORM_ML_KEM;
diff --git a/doc/guides/tools/cryptoperf.rst b/doc/guides/tools/cryptoperf.rst
index 13f9db66f4..7a7ee5c884 100644
--- a/doc/guides/tools/cryptoperf.rst
+++ b/doc/guides/tools/cryptoperf.rst
@@ -184,6 +184,7 @@ The following are the application command-line options:
rsa
sm2
mlkem_512
+ mldsa_44
ipsec
tls-record
--
2.43.0
^ permalink raw reply related
* [PATCH v2] net/mlx5: fix counter TAILQ race between free and query callback
From: Linhu Li @ 2026-06-08 13:25 UTC (permalink / raw)
To: dev; +Cc: stable, dsosnowski, Linhu Li
In-Reply-To: <20260604101112.72177-1-lilinhu618@gmail.com>
flow_dv_counter_free() inserts counters into
pool->counters[pool->query_gen] under pool->csl. Meanwhile,
mlx5_flow_async_pool_query_handle() moves counters from
pool->counters[query_gen ^ 1] to the global free list via
TAILQ_CONCAT while holding only cmng->csl, not pool->csl.
The comment in flow_dv_counter_free() claims the lock is not needed
because the query callback and the release function operate on
different lists. That holds only if the free path always observes
the up-to-date query_gen. It can be violated:
1. A counter free thread (non-PMD, e.g. OVS offload thread) reads
pool->query_gen == 0 and is about to insert into counters[0].
2. The free thread is preempted by the OS scheduler; it is a regular
pthread, not pinned to a core.
3. The eal-intr-thread alarm fires: query_gen++ (now 1) and the async
query is sent.
4. Hardware completes the query and the callback runs TAILQ_CONCAT on
counters[0] (= query_gen ^ 1).
5. The free thread resumes and runs TAILQ_INSERT_TAIL on counters[0]
concurrently with step 4 on another core.
Because the two paths take different locks, TAILQ_INSERT_TAIL and
TAILQ_CONCAT run concurrently on the same list with no
synchronization and corrupt it: the pool-local list ends up with a
NULL head but a dangling tqh_last, and the global free list tail no
longer points to the real tail. The just-freed counter and every
counter inserted afterwards become unreachable and are leaked.
Non-PMD threads can be preempted for hundreds of microseconds under
CPU pressure, which is well within the async query round-trip time,
so the window is reachable in practice.
Fix it by taking pool->csl in the query completion callback before
operating on pool->counters[query_gen], serializing the CONCAT with
any concurrent INSERT. The lock is taken once per pool per query
completion in the eal-intr-thread context, not on the datapath, so
the cost is negligible. Lock order is pool->csl then cmng->csl,
matching all other sites.
Also handle the error path: previously the counters accumulated in
pool->counters[query_gen] were abandoned when a query failed. Move
them back to the global free list to avoid a leak on persistent
query failures.
Fixes: ac79183dc6f7 ("net/mlx5: optimize free counter lookup")
Cc: stable@dpdk.org
Signed-off-by: Linhu Li <lilinhu618@gmail.com>
---
drivers/net/mlx5/mlx5_flow.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 915ea29a5a..20aad87f5d 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -9904,6 +9904,20 @@ mlx5_flow_async_pool_query_handle(struct mlx5_dev_ctx_shared *sh,
if (unlikely(status)) {
raw_to_free = pool->raw_hw;
+ /*
+ * The query failed, so the freed counters accumulated in the
+ * old-gen list during this round would otherwise be stranded.
+ * Move them back to the global free list to avoid a leak when
+ * queries fail persistently.
+ */
+ if (!TAILQ_EMPTY(&pool->counters[query_gen])) {
+ rte_spinlock_lock(&pool->csl);
+ rte_spinlock_lock(&cmng->csl[cnt_type]);
+ TAILQ_CONCAT(&cmng->counters[cnt_type],
+ &pool->counters[query_gen], next);
+ rte_spinlock_unlock(&cmng->csl[cnt_type]);
+ rte_spinlock_unlock(&pool->csl);
+ }
} else {
raw_to_free = pool->raw;
if (pool->is_aged)
@@ -9913,11 +9927,20 @@ mlx5_flow_async_pool_query_handle(struct mlx5_dev_ctx_shared *sh,
rte_spinlock_unlock(&pool->sl);
/* Be sure the new raw counters data is updated in memory. */
rte_io_wmb();
+ /*
+ * A counter free thread may have read a stale query_gen
+ * before the generation was flipped and could still be
+ * inserting into this same old-gen list. Hold pool->csl to
+ * serialize TAILQ_CONCAT with that TAILQ_INSERT_TAIL and
+ * avoid corrupting the list.
+ */
if (!TAILQ_EMPTY(&pool->counters[query_gen])) {
+ rte_spinlock_lock(&pool->csl);
rte_spinlock_lock(&cmng->csl[cnt_type]);
TAILQ_CONCAT(&cmng->counters[cnt_type],
&pool->counters[query_gen], next);
rte_spinlock_unlock(&cmng->csl[cnt_type]);
+ rte_spinlock_unlock(&pool->csl);
}
}
LIST_INSERT_HEAD(&sh->sws_cmng.free_stat_raws, raw_to_free, next);
--
2.39.3 (Apple Git-146)
^ permalink raw reply related
* RE: [PATCH v4] ethdev: support inline calculating masked item value
From: Bing Zhao @ 2026-06-09 5:23 UTC (permalink / raw)
To: Stephen Hemminger
Cc: Slava Ovsiienko, dev@dpdk.org, Raslan Darawsheh, Ori Kam,
Dariusz Sosnowski, Suanming Mou, Matan Azrad,
NBU-Contact-Thomas Monjalon (EXTERNAL)
In-Reply-To: <20260608084558.4dd4c07c@phoenix.local>
Hi,
Let me check and update with v5.
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Monday, June 8, 2026 11:46 PM
> To: Bing Zhao <bingz@nvidia.com>
> Cc: Slava Ovsiienko <viacheslavo@nvidia.com>; dev@dpdk.org; Raslan
> Darawsheh <rasland@nvidia.com>; Ori Kam <orika@nvidia.com>; Dariusz
> Sosnowski <dsosnowski@nvidia.com>; Suanming Mou <suanmingm@nvidia.com>;
> Matan Azrad <matan@nvidia.com>; NBU-Contact-Thomas Monjalon (EXTERNAL)
> <thomas@monjalon.net>
> Subject: Re: [PATCH v4] ethdev: support inline calculating masked item
> value
>
> External email: Use caution opening links or attachments
>
>
> On Wed, 3 Jun 2026 12:28:05 +0300
> Bing Zhao <bingz@nvidia.com> wrote:
>
> > In the asynchronous API definition and some drivers, the rte_flow_item
> > spec value may not be calculated by the driver due to the reason of
> > speed of light rule insertion rate and sometimes the input parameters
> > will be copied and changed internally.
> >
> > After copying, the spec and last will be protected by the keyword
> > const and cannot be changed in the code itself. And also the driver
> > needs some extra memory to do the calculation and extra conditions to
> > understand the length of each item spec. This is not efficient.
> >
> > To solve the issue and support usage of the following fix, a new OP
> > was introduced to calculate the spec and last values after applying
> > the mask inline.
> >
> > Signed-off-by: Bing Zhao <bingz@nvidia.com>
> > ---
>
> More detailed AI review found some things that still need addressing.
>
> On Wed, 3 Jun 2026 12:28:05 +0300, Bing Zhao wrote:
> > Subject: [PATCH v4] ethdev: support inline calculating masked item
> > value
>
> Error: byte-wise masking corrupts embedded pointers in deep-copy item
> types (RAW, FLEX, GENEVE_OPT).
>
> In rte_flow_conv_pattern(), the new mask application runs over the fixed
> item struct:
>
> size_t item_mask_size = mask ? rte_flow_conv_item_mask_size(src) :
> 0;
> ...
> size_t mask_size = RTE_MIN(ret, item_mask_size);
>
> for (j = 0; j < mask_size; j++)
> c_spec[j] &= mask[j];
>
> item_mask_size is rte_flow_desc_item[type].size, the size of the fixed
> item struct. For RTE_FLOW_ITEM_TYPE_RAW, FLEX, and GENEVE_OPT, that fixed
> struct ends in an embedded pointer that rte_flow_conv_item_spec() has just
> populated to point at the deep-copied trailing data
> (rte_flow_item_raw.pattern, rte_flow_item_flex.pattern,
> rte_flow_item_geneve_opt.data). Because the masked range covers the whole
> fixed struct, the loop ANDs the bytes of that pointer with the mask's
> corresponding bytes (typically a NULL mask pointer), zeroing or garbling
> it.
>
> The converted item's pattern/data pointer is clobbered while the copied
> payload it should reference is left unreachable. A consumer that follows
> conv->pattern then dereferences NULL or a corrupt address. Plain value
> conv->items
> (eth, ipv4, ...) are unaffected; only the deep-copy item types break,
> which is exactly what the test does not exercise.
>
> Suggested fix: do not blind-mask the entire fixed struct for items that
> carry an embedded pointer / desc_fn deep copy. Either skip masking when
> rte_flow_desc_item[type].desc_fn != NULL, or mask only the leading plain-
> data region and leave the pointer field (and trailing copied bytes)
> intact.
>
> Warning: the new test validates only an ETH pattern, so the
> RAW/FLEX/GENEVE_OPT path above is untested. A RAW item case would have
> surfaced the pointer corruption.
>
> Info: the Doxygen block for RTE_FLOW_CONV_OP_PATTERN_MASKED uses @p mask,
> @p spec, @p last, but those are item fields, not parameters of the op; the
> neighboring enum entries only document the @p src / @p dst types.
^ permalink raw reply
* Re: [PATCH 2/2] net/cnxk: add FEC get set and capability ops
From: Jerin Jacob @ 2026-06-09 5:06 UTC (permalink / raw)
To: rkudurumalla
Cc: Nithin Dabilpuram, Kiran Kumar K, Sunil Kumar Kori, Satha Rao,
Harman Kalra, dev, jerinj
In-Reply-To: <20260416120031.3553798-2-rkudurumalla@marvell.com>
On Thu, Apr 16, 2026 at 5:41 PM rkudurumalla <rkudurumalla@marvell.com> wrote:
>
> From: Rakesh Kudurumalla <rkudurumalla@marvell.com>
>
> Add ethdev FEC operations for cnxk NIX driver:
> - fec_get_capability: Report supported FEC modes per speed.
> If firmware provides supported FEC info, return actual
> capabilities for current link speed. Otherwise, fall back
> to a default capability table for common speeds.
> - fec_get: Query current FEC mode from link info
> - fec_set: Configure FEC mode on the link. AUTO mode
> defaults to Reed-Solomon FEC.
>
> Signed-off-by: Rakesh Kudurumalla <rkudurumalla@marvell.com>
Update doc/guides/nics/cnxk.rst and doc/guides/nics/features/cnxk* for
this new feature,.
^ permalink raw reply
* Re: [PATCH 2/2] common/cnxk: fix thread-unsafe NIX telemetry parsing
From: Jerin Jacob @ 2026-06-09 5:02 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, stable, Nithin Dabilpuram, Kiran Kumar K, Sunil Kumar Kori,
Satha Rao, Harman Kalra, Gowrishankar Muthukrishnan, Jerin Jacob
In-Reply-To: <20260605224514.651081-3-stephen@networkplumber.org>
On Sat, Jun 6, 2026 at 4:15 AM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> cnxk_nix_tel_handle_info_x() backs the /cnxk/nix/{rq,cq,sq}/{info,ctx}
> telemetry commands and parsed its "<pcidev>,<queue_id>" parameter with
> strtok(), which keeps non-reentrant state and races when telemetry
> callbacks run on per-connection threads.
>
> Split the parameter with strchr() and parse the queue id with strtoul().
> While here, copy the full parameter (the length was capped at
> PCI_PRI_STR_SIZE + 1, truncating the id for longer device addresses) and
> reject non-numeric or out-of-range ids instead of letting strtol() alias
> them to queue 0.
>
> Fixes: af75aac78978 ("common/cnxk: support telemetry for NIX")
> Cc: stable@dpdk.org
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
@Nithin Dabilpuram Please review and Ack the series.
^ permalink raw reply
* [PATCH 1/1] mldev: fix incorrect captions for model_ops
From: Srikanth Yalavarthi @ 2026-06-09 5:00 UTC (permalink / raw)
To: Srikanth Yalavarthi, Anup Prabhu; +Cc: dev, jerinj, sshankarnara, ptakkar
Fixed incorrect captions in images depecting the
model_ops test flows.
Fixes: f6661e6d9a3a ("app/mldev: validate model operations")
Signed-off-by: Srikanth Yalavarthi <syalavarthi@marvell.com>
---
doc/guides/tools/img/mldev_model_ops_subtest_a.svg | 4 ++--
doc/guides/tools/img/mldev_model_ops_subtest_b.svg | 4 ++--
doc/guides/tools/img/mldev_model_ops_subtest_d.svg | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/doc/guides/tools/img/mldev_model_ops_subtest_a.svg b/doc/guides/tools/img/mldev_model_ops_subtest_a.svg
index ed12cc5a05f..9b38300d1d7 100644
--- a/doc/guides/tools/img/mldev_model_ops_subtest_a.svg
+++ b/doc/guides/tools/img/mldev_model_ops_subtest_a.svg
@@ -10,7 +10,7 @@
version="1.1"
id="svg5"
inkscape:version="1.2.1 (9c6d41e410, 2022-07-14)"
- sodipodi:docname="mldev_model_ops_subtest_d.svg"
+ sodipodi:docname="mldev_model_ops_subtest_a.svg"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns="http://www.w3.org/2000/svg"
@@ -407,7 +407,7 @@
id="tspan390"
style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:5.64444px;font-family:Arial;-inkscape-font-specification:'Arial Bold';fill:#000000;stroke-width:0.5"
x="-120.71075"
- y="220.77164">mldev: model_ops / subtest D</tspan></text>
+ y="220.77164">mldev: model_ops / subtest A</tspan></text>
<rect
style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.448724;stroke-linecap:round;stroke-linejoin:round;stroke-dasharray:none;stroke-opacity:1;paint-order:stroke fill markers"
id="rect2164"
diff --git a/doc/guides/tools/img/mldev_model_ops_subtest_b.svg b/doc/guides/tools/img/mldev_model_ops_subtest_b.svg
index 173a2c6c059..d692b2ee023 100644
--- a/doc/guides/tools/img/mldev_model_ops_subtest_b.svg
+++ b/doc/guides/tools/img/mldev_model_ops_subtest_b.svg
@@ -10,7 +10,7 @@
version="1.1"
id="svg5"
inkscape:version="1.2.1 (9c6d41e410, 2022-07-14)"
- sodipodi:docname="mldev_model_ops_subtest_a.svg"
+ sodipodi:docname="mldev_model_ops_subtest_b.svg"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns="http://www.w3.org/2000/svg"
@@ -410,7 +410,7 @@
id="tspan390"
style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:5.64444px;font-family:Arial;-inkscape-font-specification:'Arial Bold';fill:#000000;stroke-width:0.5"
x="-119.78458"
- y="220.32892">mldev: model_ops / subtest A</tspan></text>
+ y="220.32892">mldev: model_ops / subtest B</tspan></text>
<rect
style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.442854;stroke-linecap:round;stroke-linejoin:round;paint-order:stroke fill markers"
id="rect3967"
diff --git a/doc/guides/tools/img/mldev_model_ops_subtest_d.svg b/doc/guides/tools/img/mldev_model_ops_subtest_d.svg
index 3e2b89ad256..bf64f2b8360 100644
--- a/doc/guides/tools/img/mldev_model_ops_subtest_d.svg
+++ b/doc/guides/tools/img/mldev_model_ops_subtest_d.svg
@@ -10,7 +10,7 @@
version="1.1"
id="svg5"
inkscape:version="1.2.1 (9c6d41e410, 2022-07-14)"
- sodipodi:docname="mldev_model_ops_subtest_b.svg"
+ sodipodi:docname="mldev_model_ops_subtest_d.svg"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns="http://www.w3.org/2000/svg"
@@ -411,7 +411,7 @@
id="tspan390"
style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:5.64444px;font-family:Arial;-inkscape-font-specification:'Arial Bold';fill:#000000;stroke-width:0.5"
x="-122.00021"
- y="222.26495">mldev: model_ops / subest B</tspan></text>
+ y="222.26495">mldev: model_ops / subtest D</tspan></text>
<rect
style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.462341;stroke-linecap:round;stroke-linejoin:round;paint-order:stroke fill markers"
id="rect3475"
--
2.34.1
^ permalink raw reply related
* Re: [EXTERNAL] [PATCH dpdk] graph: replace circular buffer with priority-based bitmap scheduling
From: Jerin Jacob @ 2026-06-09 4:52 UTC (permalink / raw)
To: Robin Jarry
Cc: Jerin Jacob, dev@dpdk.org, Kiran Kumar Kokkilagadda,
Nithin Kumar Dabilpuram, Zhirun Yan, Vladimir Medvedkin,
Christophe Fontaine, David Marchand, Konstantin Ananyev,
Maxime Leroy
In-Reply-To: <DIMZ41ANY5TQ.3SXX22QGRY92R@redhat.com>
On Wed, May 20, 2026 at 3:20 AM Robin Jarry <rjarry@redhat.com> wrote:
>
> Hi Jerin,
>
> Jerin Jacob, May 19, 2026 at 12:25:
> > Some high-level comments:
> >
> > 1)What will be the performance overhead for graph walk. Try
> > app/test/test_graph_perf.c and l3fwd_graph
>
> I have made some measurements. There was a regression caused by
> systematic access to a cache line. This will be fixed in v2.
Thanks. We will review the v2.
> > 2)If priorities are same, Does it have similar performance with
> > existing code?
>
> Yes with v2. I have added a topo order as a fallback when all priorities
> are equal so that it behaves as previously.
>
> > 3)Does it have any effect on packet ordering on egress.i.e for a given
> > flow ingress order != egress order
>
> Some independent flows "may" be reordered but that depends on the
> topology, not the graph walk algorithm.
>
> For example, the following graph:
>
> +--------+
> | eth_in |
> +--------+
> / \
> v v
> +-------+ +---------+
> | ip_in | <-- | mpls_in |
> +-------+ +---------+
> |
> | ,------------.
> v v |
> +--------+ +----------+
> | ip_out | --> | ipip_out |
> +--------+ +----------+
> |
> v
> +---------+
> | eth_out |
> +---------+
>
> If you receive the following stream of packets in "eth_in":
>
> 1. Eth/IP
> 2. Eth/IP
> 3. Eth/MPLS/IP
> 4. Eth/MPLS/IP
> 5. Eth/MPLS/IP
> 6. Eth/IP (needs IPIP encap)
> 7. Eth/IP (needs IPIP encap)
> 8. Eth/IP
>
> The "eth_out" node will see:
>
> 1. Eth/IP
> 2. Eth/IP
> 3. Eth/IP
> 4. Eth/IP
> 5. Eth/IP
> * 8. Eth/IP <----------- reordered before 6 and 7
> 6. Eth/IP/IP
> 7. Eth/IP/IP
>
> With the FIFO circular buffer and with the new bitmap priority scheduler.
>
> The only difference with the bitmap system is that If you set "mpls_in"
> and "ip_out" higher priorities, the "ip_in" and "eth_out" nodes will be
> visited only once.
>
> > If there is performance regression we may need to consider new
> > RTE_GRAPH_MODEL_XXXXX to enable this feature
>
>
> --
> Robin
>
> > Times approximate.
>
^ permalink raw reply
* Re: [PATCH v2 2/3] event/cnxk: add pause to spinloops
From: Jerin Jacob @ 2026-06-09 4:50 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, Pavan Nikhilesh, Shijith Thotton
In-Reply-To: <20260608104426.6ceb4ad8@phoenix.local>
On Mon, Jun 8, 2026 at 11:14 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Mon, 8 Jun 2026 21:19:20 +0530
> Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> > On Mon, Apr 13, 2026 at 10:36 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >
> > > On SMT systems when a spinloop is done without a pause
> > > it may cause excessive latency. This problem was found
> > > by the fix_empty_spinloops coccinelle script.
> > >
> > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> >
> > rte_pause() translates to YIELD instruction. Since cnxk is an
> > integrated SoC and it is a single threaded core, it won't help on
> > anything other than adding one instruction bit more latency.
> > In general 3/3 devtool is good. Please send a it separate version so
> > that 3/3 patches can be merged through the main tree.
>
>
> It matters if your SOC has SMT where two cores are sharing
> and one core is waiting for its partner.
Yes. As I mentioned earlier, it is not cnxk is not SMT so it will not
be useful for cnxk.
^ permalink raw reply
* [PATCH 3/3] ml/cnxk: updated checks for number of MRVL layers
From: Srikanth Yalavarthi @ 2026-06-09 4:42 UTC (permalink / raw)
To: Srikanth Yalavarthi; +Cc: dev, jerinj, sshankarnara, ptakkar, aprabhu
In-Reply-To: <20260609044202.936153-1-syalavarthi@marvell.com>
Add checks to validate the current active MRVL layers
and increased the maximum number of layers supported
to 512 for LLVM only models.
Signed-off-by: Srikanth Yalavarthi <syalavarthi@marvell.com>
---
drivers/ml/cnxk/cn10k_ml_ops.c | 6 +++---
drivers/ml/cnxk/cnxk_ml_dev.h | 4 ++--
drivers/ml/cnxk/cnxk_ml_io.h | 2 +-
drivers/ml/cnxk/cnxk_ml_ops.c | 14 +++++++-------
drivers/ml/cnxk/tvmrt_ml_model.c | 17 +++++++++++++++++
5 files changed, 30 insertions(+), 13 deletions(-)
diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c
index b30af7c7a44..00c0d87a6d6 100644
--- a/drivers/ml/cnxk/cn10k_ml_ops.c
+++ b/drivers/ml/cnxk/cn10k_ml_ops.c
@@ -624,15 +624,15 @@ cn10k_ml_layer_load(void *device, uint16_t model_id, const char *layer_name, uin
return ret;
/* Get index */
- for (idx = 0; idx < cnxk_mldev->max_nb_layers; idx++) {
+ for (idx = 0; idx < cnxk_mldev->max_mrvl_layers; idx++) {
if (!cnxk_mldev->index_map[idx].active) {
layer->index = idx;
break;
}
}
- if (idx >= cnxk_mldev->max_nb_layers) {
- plt_err("No slots available for model layers, model_id = %u, layer_id = %u",
+ if (idx >= cnxk_mldev->max_mrvl_layers) {
+ plt_err("No slots available for model MRVL layers, model_id = %u, layer_id = %u",
model->model_id, layer_id);
return -1;
}
diff --git a/drivers/ml/cnxk/cnxk_ml_dev.h b/drivers/ml/cnxk/cnxk_ml_dev.h
index e93d76d1af8..181a04737d7 100644
--- a/drivers/ml/cnxk/cnxk_ml_dev.h
+++ b/drivers/ml/cnxk/cnxk_ml_dev.h
@@ -93,9 +93,9 @@ struct cnxk_ml_dev {
#endif
/* Maximum number of layers */
- uint64_t max_nb_layers;
+ uint64_t max_mrvl_layers;
- /* Index map */
+ /* Index map for MRVL layers */
struct cnxk_ml_index_map *index_map;
};
diff --git a/drivers/ml/cnxk/cnxk_ml_io.h b/drivers/ml/cnxk/cnxk_ml_io.h
index 3ca852706f5..e6efd588f75 100644
--- a/drivers/ml/cnxk/cnxk_ml_io.h
+++ b/drivers/ml/cnxk/cnxk_ml_io.h
@@ -16,7 +16,7 @@
/* Maximum number of layers per model */
#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
-#define ML_CNXK_MODEL_MAX_LAYERS 128
+#define ML_CNXK_MODEL_MAX_LAYERS 512
#else
#define ML_CNXK_MODEL_MAX_LAYERS 1
#endif
diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c
index 675cdaae2bd..dc057643068 100644
--- a/drivers/ml/cnxk/cnxk_ml_ops.c
+++ b/drivers/ml/cnxk/cnxk_ml_ops.c
@@ -636,30 +636,30 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co
/* Set device capabilities */
if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI)
- cnxk_mldev->max_nb_layers =
+ cnxk_mldev->max_mrvl_layers =
cnxk_mldev->cn10k_mldev.fw.req->cn10k_req.jd.fw_load.cap.s.max_models;
else
- cnxk_mldev->max_nb_layers = ML_CNXK_MAX_MODELS;
+ cnxk_mldev->max_mrvl_layers = 0;
cnxk_mldev->mldev->enqueue_burst = cnxk_ml_enqueue_burst;
cnxk_mldev->mldev->dequeue_burst = cnxk_ml_dequeue_burst;
cnxk_mldev->mldev->op_error_get = cnxk_ml_op_error_get;
/* Allocate and initialize index_map */
- if (cnxk_mldev->index_map == NULL) {
+ if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI && cnxk_mldev->index_map == NULL) {
cnxk_mldev->index_map =
rte_zmalloc("cnxk_ml_index_map",
- sizeof(struct cnxk_ml_index_map) * cnxk_mldev->max_nb_layers,
+ sizeof(struct cnxk_ml_index_map) * cnxk_mldev->max_mrvl_layers,
RTE_CACHE_LINE_SIZE);
if (cnxk_mldev->index_map == NULL) {
- plt_err("Failed to get memory for index_map, nb_layers %" PRIu64,
- cnxk_mldev->max_nb_layers);
+ plt_err("Failed to get memory for index_map, nb_mrvl_layers %" PRIu64,
+ cnxk_mldev->max_mrvl_layers);
ret = -ENOMEM;
goto error;
}
}
- for (i = 0; i < cnxk_mldev->max_nb_layers; i++)
+ for (i = 0; i < cnxk_mldev->max_mrvl_layers; i++)
cnxk_mldev->index_map[i].active = false;
/* Initialize xstats */
diff --git a/drivers/ml/cnxk/tvmrt_ml_model.c b/drivers/ml/cnxk/tvmrt_ml_model.c
index 9bff424f0d8..319c53a05b5 100644
--- a/drivers/ml/cnxk/tvmrt_ml_model.c
+++ b/drivers/ml/cnxk/tvmrt_ml_model.c
@@ -398,6 +398,7 @@ tvmrt_ml_json_graph_get_arrays(json_t *json_parsed, json_t **nodes, json_t **arg
int
tvmrt_ml_model_json_parse(struct cnxk_ml_model *model)
{
+ struct cnxk_ml_dev *cnxk_mldev = model->cnxk_mldev;
struct tvmrt_ml_param_names param_names;
json_error_t json_error;
json_t *json_parsed;
@@ -407,6 +408,7 @@ tvmrt_ml_model_json_parse(struct cnxk_ml_model *model)
json_t *json_node_row_ptr;
json_t *json_shape_values;
json_t *json_dtype_values;
+ uint16_t nb_active_mrvl_layers;
uint16_t nb_mrvl_layers;
uint16_t nb_llvm_layers;
DLDevice device;
@@ -511,6 +513,21 @@ tvmrt_ml_model_json_parse(struct cnxk_ml_model *model)
goto error;
}
+ nb_active_mrvl_layers = 0;
+ for (i = 0; i < cnxk_mldev->max_mrvl_layers; i++) {
+ if (cnxk_mldev->index_map[i].active)
+ nb_active_mrvl_layers++;
+ }
+
+ if (nb_active_mrvl_layers + nb_mrvl_layers > cnxk_mldev->max_mrvl_layers) {
+ ret = -ENOSPC;
+ plt_err("TVM runtime: Total MRVL layers (%u) exceeds maximum supported "
+ "MRVL layers (%" PRIu64 "), model_id = %u, error = %d",
+ nb_active_mrvl_layers + nb_mrvl_layers, cnxk_mldev->max_mrvl_layers,
+ model->model_id, ret);
+ goto error;
+ }
+
/* Set model subtype */
if ((nb_llvm_layers == 0) && (nb_mrvl_layers == 1))
model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_MRVL;
--
2.34.1
^ permalink raw reply related
* [PATCH 2/3] ml/cnxk: rename tvm runtime module components
From: Srikanth Yalavarthi @ 2026-06-09 4:42 UTC (permalink / raw)
To: Srikanth Yalavarthi; +Cc: dev, jerinj, sshankarnara, ptakkar, aprabhu
In-Reply-To: <20260609044202.936153-1-syalavarthi@marvell.com>
Rename all TVM runtime specific files and functions with
tvmrt prefix. Retain mvtvm name prefix for functions and
files related to mvtvm vdev component.
Signed-off-by: Srikanth Yalavarthi <syalavarthi@marvell.com>
---
drivers/ml/cnxk/cn10k_ml_model.c | 2 +-
drivers/ml/cnxk/cnxk_ml_dev.h | 4 +-
drivers/ml/cnxk/cnxk_ml_io.h | 6 +-
drivers/ml/cnxk/cnxk_ml_model.c | 4 +-
drivers/ml/cnxk/cnxk_ml_model.h | 12 +-
drivers/ml/cnxk/cnxk_ml_ops.c | 24 +-
drivers/ml/cnxk/cnxk_ml_ops.h | 10 +-
drivers/ml/cnxk/meson.build | 6 +-
drivers/ml/cnxk/mvtvm_ml_ops.c | 818 +-----------------
drivers/ml/cnxk/mvtvm_ml_ops.h | 66 --
drivers/ml/cnxk/mvtvm_ml_stubs.c | 97 ---
drivers/ml/cnxk/mvtvm_ml_stubs.h | 19 -
.../{mvtvm_ml_model.c => tvmrt_ml_model.c} | 269 +++---
.../{mvtvm_ml_model.h => tvmrt_ml_model.h} | 44 +-
drivers/ml/cnxk/tvmrt_ml_ops.c | 818 ++++++++++++++++++
drivers/ml/cnxk/tvmrt_ml_ops.h | 79 ++
drivers/ml/cnxk/tvmrt_ml_stubs.c | 106 +++
drivers/ml/cnxk/tvmrt_ml_stubs.h | 32 +
18 files changed, 1232 insertions(+), 1184 deletions(-)
rename drivers/ml/cnxk/{mvtvm_ml_model.c => tvmrt_ml_model.c} (72%)
rename drivers/ml/cnxk/{mvtvm_ml_model.h => tvmrt_ml_model.h} (78%)
create mode 100644 drivers/ml/cnxk/tvmrt_ml_ops.c
create mode 100644 drivers/ml/cnxk/tvmrt_ml_ops.h
create mode 100644 drivers/ml/cnxk/tvmrt_ml_stubs.c
create mode 100644 drivers/ml/cnxk/tvmrt_ml_stubs.h
diff --git a/drivers/ml/cnxk/cn10k_ml_model.c b/drivers/ml/cnxk/cn10k_ml_model.c
index 12a2dda800e..9cfecea4448 100644
--- a/drivers/ml/cnxk/cn10k_ml_model.c
+++ b/drivers/ml/cnxk/cn10k_ml_model.c
@@ -725,7 +725,7 @@ int
cn10k_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name, uint16_t *layer_id)
{
if (model->type == ML_CNXK_MODEL_TYPE_TVM)
- return mvtvm_ml_model_get_layer_id(model, layer_name, layer_id);
+ return tvmrt_ml_model_get_layer_id(model, layer_name, layer_id);
*layer_id = 0;
diff --git a/drivers/ml/cnxk/cnxk_ml_dev.h b/drivers/ml/cnxk/cnxk_ml_dev.h
index 9e373e65715..e93d76d1af8 100644
--- a/drivers/ml/cnxk/cnxk_ml_dev.h
+++ b/drivers/ml/cnxk/cnxk_ml_dev.h
@@ -9,7 +9,7 @@
#include "cn10k_ml_dev.h"
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
#include "mvtvm_ml_dev.h"
#endif
@@ -87,7 +87,7 @@ struct cnxk_ml_dev {
/* CN10K device structure */
struct cn10k_ml_dev cn10k_mldev;
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
/* MVTVM device structure */
struct mvtvm_ml_dev mvtvm_mldev;
#endif
diff --git a/drivers/ml/cnxk/cnxk_ml_io.h b/drivers/ml/cnxk/cnxk_ml_io.h
index 17f5b4619f0..3ca852706f5 100644
--- a/drivers/ml/cnxk/cnxk_ml_io.h
+++ b/drivers/ml/cnxk/cnxk_ml_io.h
@@ -5,7 +5,7 @@
#ifndef _CNXK_ML_IO_H_
#define _CNXK_ML_IO_H_
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
#include <dlpack/dlpack.h>
#endif
@@ -15,7 +15,7 @@
#define ML_CNXK_MAX_MODELS 16
/* Maximum number of layers per model */
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
#define ML_CNXK_MODEL_MAX_LAYERS 128
#else
#define ML_CNXK_MODEL_MAX_LAYERS 1
@@ -59,7 +59,7 @@ struct cnxk_ml_io {
/* Zero point */
int64_t zero_point;
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
/* Shape - int64_t */
int64_t shape_i64[ML_CNXK_MODEL_MAX_DIMS];
diff --git a/drivers/ml/cnxk/cnxk_ml_model.c b/drivers/ml/cnxk/cnxk_ml_model.c
index ed6a1ed8665..442bc9460c0 100644
--- a/drivers/ml/cnxk/cnxk_ml_model.c
+++ b/drivers/ml/cnxk/cnxk_ml_model.c
@@ -16,7 +16,7 @@ cnxk_ml_model_get_type(struct rte_ml_model_params *params)
uint32_t payload_crc32c;
uint32_t header_crc32c;
- type = mvtvm_ml_model_type_get(params);
+ type = tvmrt_ml_model_type_get(params);
if (type == ML_CNXK_MODEL_TYPE_TVM)
return ML_CNXK_MODEL_TYPE_TVM;
else if (type == ML_CNXK_MODEL_TYPE_INVALID)
@@ -89,6 +89,6 @@ cnxk_ml_model_dump(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
if (layer->type == ML_CNXK_LAYER_TYPE_MRVL)
cn10k_ml_layer_print(cnxk_mldev, layer, fp);
else
- mvtvm_ml_layer_print(cnxk_mldev, layer, fp);
+ tvmrt_ml_layer_print(cnxk_mldev, layer, fp);
}
}
diff --git a/drivers/ml/cnxk/cnxk_ml_model.h b/drivers/ml/cnxk/cnxk_ml_model.h
index 1cd5ca1906a..59aea16f970 100644
--- a/drivers/ml/cnxk/cnxk_ml_model.h
+++ b/drivers/ml/cnxk/cnxk_ml_model.h
@@ -11,10 +11,10 @@
#include "cn10k_ml_model.h"
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
-#include "mvtvm_ml_model.h"
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
+#include "tvmrt_ml_model.h"
#else
-#include "mvtvm_ml_stubs.h"
+#include "tvmrt_ml_stubs.h"
#endif
#include "cnxk_ml_io.h"
@@ -152,9 +152,9 @@ struct cnxk_ml_model {
/* Model specific data - glow */
struct cn10k_ml_model_data glow;
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
- /* Model type specific data - mvtvm */
- struct mvtvm_ml_model_data mvtvm;
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
+ /* Model type specific data - tvmrt */
+ struct tvmrt_ml_model_data tvmrt;
#endif
};
diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c
index 938982c7556..675cdaae2bd 100644
--- a/drivers/ml/cnxk/cnxk_ml_ops.c
+++ b/drivers/ml/cnxk/cnxk_ml_ops.c
@@ -250,7 +250,7 @@ cnxk_ml_xstats_model_name_update(struct cnxk_ml_dev *cnxk_mldev, uint16_t model_
if (model->type == ML_CNXK_MODEL_TYPE_GLOW)
cn10k_ml_xstat_model_name_set(cnxk_mldev, model, stat_id, i, suffix);
else
- mvtvm_ml_model_xstat_name_set(cnxk_mldev, model, stat_id, i, suffix);
+ tvmrt_ml_model_xstat_name_set(cnxk_mldev, model, stat_id, i, suffix);
stat_id++;
}
@@ -312,7 +312,7 @@ cnxk_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, uint16_t obj_idx, int32_
goto exit_xstats;
model_xstats:
- value = mvtvm_ml_model_xstat_get(cnxk_mldev, model, type);
+ value = tvmrt_ml_model_xstat_get(cnxk_mldev, model, type);
exit_xstats:
roc_clk_freq_get(&rclk_freq, &sclk_freq);
@@ -1211,7 +1211,7 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u
if (type == ML_CNXK_MODEL_TYPE_GLOW)
ret = cn10k_ml_model_load(cnxk_mldev, params, model);
else
- ret = mvtvm_ml_model_load(cnxk_mldev, params, model);
+ ret = tvmrt_ml_model_load(cnxk_mldev, params, model);
if (ret != 0)
goto error;
@@ -1225,7 +1225,7 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u
total_wb_pages = total_wb_pages + model->layer[layer_id].glow.ocm_map.wb_pages;
max_scratch_pages = PLT_MAX(max_scratch_pages,
model->layer[layer_id].glow.ocm_map.scratch_pages);
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
} else {
for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
if (model->layer[layer_id].type == ML_CNXK_LAYER_TYPE_MRVL) {
@@ -1247,7 +1247,7 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u
plt_ml_dbg("layer_id = %u: wb_pages = %u, scratch_pages = %u", layer_id,
model->layer[layer_id].glow.ocm_map.wb_pages,
model->layer[layer_id].glow.ocm_map.scratch_pages);
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
} else {
for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
if (model->layer[layer_id].type == ML_CNXK_LAYER_TYPE_MRVL) {
@@ -1263,9 +1263,9 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u
if (model->type == ML_CNXK_MODEL_TYPE_GLOW)
cn10k_ml_model_unload(cnxk_mldev, model);
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
else {
- mvtvm_ml_model_unload(cnxk_mldev, model);
+ tvmrt_ml_model_unload(cnxk_mldev, model);
return -ENOMEM;
}
#endif
@@ -1312,7 +1312,7 @@ cnxk_ml_model_unload(struct rte_ml_dev *dev, uint16_t model_id)
if (model->type == ML_CNXK_MODEL_TYPE_GLOW)
ret = cn10k_ml_model_unload(cnxk_mldev, model);
else
- ret = mvtvm_ml_model_unload(cnxk_mldev, model);
+ ret = tvmrt_ml_model_unload(cnxk_mldev, model);
if (ret != 0)
return ret;
@@ -1343,7 +1343,7 @@ cnxk_ml_model_start(struct rte_ml_dev *dev, uint16_t model_id)
if (model->type == ML_CNXK_MODEL_TYPE_GLOW)
return cn10k_ml_model_start(cnxk_mldev, model);
else
- return mvtvm_ml_model_start(cnxk_mldev, model);
+ return tvmrt_ml_model_start(cnxk_mldev, model);
return 0;
}
@@ -1368,7 +1368,7 @@ cnxk_ml_model_stop(struct rte_ml_dev *dev, uint16_t model_id)
if (model->type == ML_CNXK_MODEL_TYPE_GLOW)
return cn10k_ml_model_stop(cnxk_mldev, model);
else
- return mvtvm_ml_model_stop(cnxk_mldev, model);
+ return tvmrt_ml_model_stop(cnxk_mldev, model);
return 0;
}
@@ -1444,7 +1444,7 @@ cnxk_ml_io_quantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_buf
if (model->type == ML_CNXK_MODEL_TYPE_GLOW)
info = cn10k_ml_model_io_info_get(model, 0);
else
- info = mvtvm_ml_model_io_info_get(model, 0);
+ info = tvmrt_ml_model_io_info_get(model, 0);
if (info == NULL)
return -EINVAL;
@@ -1500,7 +1500,7 @@ cnxk_ml_io_dequantize(struct rte_ml_dev *dev, uint16_t model_id, struct rte_ml_b
if (model->type == ML_CNXK_MODEL_TYPE_GLOW)
info = cn10k_ml_model_io_info_get(model, model->nb_layers - 1);
else
- info = mvtvm_ml_model_io_info_get(model, model->nb_layers - 1);
+ info = tvmrt_ml_model_io_info_get(model, model->nb_layers - 1);
if (info == NULL)
return -EINVAL;
diff --git a/drivers/ml/cnxk/cnxk_ml_ops.h b/drivers/ml/cnxk/cnxk_ml_ops.h
index 7a79fec412e..ad10c3d45dc 100644
--- a/drivers/ml/cnxk/cnxk_ml_ops.h
+++ b/drivers/ml/cnxk/cnxk_ml_ops.h
@@ -12,10 +12,12 @@
#include "cn10k_ml_ops.h"
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
#include "mvtvm_ml_ops.h"
+#include "tvmrt_ml_ops.h"
#else
#include "mvtvm_ml_stubs.h"
+#include "tvmrt_ml_stubs.h"
#endif
/* Request structure */
@@ -25,9 +27,9 @@ struct __rte_aligned(ROC_ALIGN) cnxk_ml_req {
/* CN10K */
struct cn10k_ml_req cn10k_req;
-#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
- /* MVTVM */
- struct mvtvm_ml_req mvtvm_req;
+#ifdef RTE_MLDEV_CNXK_ENABLE_TVMRT
+ /* TVMRT */
+ struct tvmrt_ml_req tvmrt_req;
#endif
};
diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build
index 5f078bd4fd7..da3cb7a2967 100644
--- a/drivers/ml/cnxk/meson.build
+++ b/drivers/ml/cnxk/meson.build
@@ -53,12 +53,13 @@ deps += ['mldev', 'common_cnxk', 'kvargs', 'hash']
if enable_mvtvm
-dpdk_conf.set('RTE_MLDEV_CNXK_ENABLE_MVTVM', 1)
+dpdk_conf.set('RTE_MLDEV_CNXK_ENABLE_TVMRT', 1)
sources += files(
'mvtvm_ml_dev.c',
'mvtvm_ml_ops.c',
- 'mvtvm_ml_model.c',
+ 'tvmrt_ml_ops.c',
+ 'tvmrt_ml_model.c',
)
ext_deps += jansson_dep
@@ -80,6 +81,7 @@ message('drivers/ml/cnxk: Disabled TVM model support')
sources += files(
'mvtvm_ml_stubs.c',
+ 'tvmrt_ml_stubs.c',
)
endif
diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c
index bc47a4bbd75..69b800c8421 100644
--- a/drivers/ml/cnxk/mvtvm_ml_ops.c
+++ b/drivers/ml/cnxk/mvtvm_ml_ops.c
@@ -2,189 +2,19 @@
* Copyright (c) 2023 Marvell.
*/
-#include <errno.h>
-#include <linux/limits.h>
-#include <stdbool.h>
#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <sys/mman.h>
-#include <unistd.h>
-
-#include <dlpack/dlpack.h>
-#include <jansson.h>
-
-#include <rte_common.h>
-#include <rte_cycles.h>
-#include <rte_mldev.h>
-#include <rte_mldev_pmd.h>
-
-#include <mldev_utils.h>
#include "cnxk_ml_dev.h"
-#include "cnxk_ml_model.h"
-#include "cnxk_ml_ops.h"
-#include "cnxk_ml_xstats.h"
-
-/* ML model macros */
-#define MVTVM_ML_MODEL_MEMZONE_NAME "ml_mvtvm_model_mz"
-
-/* Shared memory file descriptor name */
-#define ML_MODEL_SHMFD_NAME "mvtvm_shmfd"
-
-/* Shared memory file descriptor path */
-#define ML_MODEL_SHMFD_PATH "/proc/%d/fd/%d"
-
-static int
-mvtvm_ml_tvm_func_get(struct cnxk_ml_model *model, TVMModuleHandle module, const char *name,
- TVMFunctionHandle *func)
-{
- int ret;
-
- ret = TVMModGetFunction(module, name, 0, func);
- if (ret != 0) {
- plt_err("Model load failed, model_id = %u, ret = %d, msg = %s", model->model_id,
- ret, TVMGetLastError());
- return ret;
- }
-
- if (*func == NULL) {
- ret = -ENOENT;
- plt_err("Model load failed, model_id = %u, function '%s' not found",
- model->model_id, name);
- }
-
- return ret;
-}
-
-static int
-mvtvm_ml_tvm_func_call(struct cnxk_ml_model *model, TVMFunctionHandle func, const char *name,
- TVMValue *values, int *types, int num_args, TVMValue *ret_val, int *ret_type,
- int ret_type_code)
-{
- int ret;
-
- ret = TVMFuncCall(func, values, types, num_args, ret_val, ret_type);
- if (ret != 0) {
- plt_err("Error calling TVM function '%s', model_id = %u, ret = %d, msg = %s", name,
- model->model_id, ret, TVMGetLastError());
- return ret;
- }
-
- if (*ret_type != ret_type_code) {
- ret = -EINVAL;
- plt_err("TVM function '%s' returned unexpected type, model_id = %u, expected = %d, "
- "actual = %d",
- name, model->model_id, ret_type_code, *ret_type);
- }
-
- return ret;
-}
-
-static void
-mvtvm_ml_tvm_func_free(TVMFunctionHandle *func)
-{
- if ((func != NULL) && (*func != NULL)) {
- TVMFuncFree(*func);
- *func = NULL;
- }
-}
-
-static void
-mvtvm_ml_tvm_mod_free(TVMModuleHandle *mod)
-{
- if ((mod != NULL) && (*mod != NULL)) {
- TVMModFree(*mod);
- *mod = NULL;
- }
-}
-
-__rte_hot static void
-mvtvm_ml_set_poll_addr(struct cnxk_ml_req *req)
-{
- req->status = &req->mvtvm_req.status;
-}
-
-void
-mvtvm_ml_model_xstat_name_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
- uint16_t stat_id, uint16_t entry, char *suffix)
-{
- snprintf(cnxk_mldev->xstats.entries[stat_id].map.name,
- sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", model->name,
- model_xstats[entry].name, suffix);
-}
-
-#define ML_AVG_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count) \
- do { \
- value = 0; \
- for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \
- value += model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_tot; \
- count += model->mvtvm.burst_xstats[qp_id].dequeued_count - \
- model->mvtvm.burst_xstats[qp_id].tvm_rt_reset_count; \
- } \
- if (count != 0) \
- value = value / count; \
- } while (0)
-
-#define ML_MIN_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count) \
- do { \
- value = UINT64_MAX; \
- for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \
- value = PLT_MIN(value, \
- model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_min); \
- count += model->mvtvm.burst_xstats[qp_id].dequeued_count - \
- model->mvtvm.burst_xstats[qp_id].tvm_rt_reset_count; \
- } \
- if (count == 0) \
- value = 0; \
- } while (0)
-
-#define ML_MAX_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count) \
- do { \
- value = 0; \
- for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \
- value = PLT_MAX(value, \
- model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_max); \
- count += model->mvtvm.burst_xstats[qp_id].dequeued_count - \
- model->mvtvm.burst_xstats[qp_id].tvm_rt_reset_count; \
- } \
- if (count == 0) \
- value = 0; \
- } while (0)
-
-uint64_t
-mvtvm_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
- enum cnxk_ml_xstats_type type)
-{
- uint64_t count = 0;
- uint64_t value = 0;
- uint32_t qp_id;
-
- switch (type) {
- case avg_rt_latency:
- ML_AVG_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count);
- break;
- case min_rt_latency:
- ML_MIN_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count);
- break;
- case max_rt_latency:
- ML_MAX_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count);
- break;
- default:
- value = 0;
- }
-
- return value;
-}
+#include "mvtvm_ml_ops.h"
int
mvtvm_ml_dev_info_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_dev_info *dev_info)
{
- struct mvtvm_ml_dev *mvtvm_mldev;
+ struct mvtvm_ml_dev *tvmrt_mldev;
- mvtvm_mldev = &cnxk_mldev->mvtvm_mldev;
+ tvmrt_mldev = &cnxk_mldev->mvtvm_mldev;
- dev_info->max_queue_pairs = mvtvm_mldev->max_nb_qpairs;
+ dev_info->max_queue_pairs = tvmrt_mldev->max_nb_qpairs;
dev_info->max_desc = ML_MVTVM_MAX_DESC_PER_QP;
dev_info->max_io = ML_CNXK_MODEL_MAX_INPUT_OUTPUT;
dev_info->max_segments = ML_MVTVM_MAX_SEGMENTS;
@@ -201,643 +31,3 @@ mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp)
return 0;
}
-
-int
-mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
- struct cnxk_ml_model *model)
-{
- struct mvtvm_ml_model_object object[ML_MVTVM_MODEL_OBJECT_MAX];
- struct tvmrt_glow_callback *callback;
- char str[RTE_MEMZONE_NAMESIZE];
- char path[PATH_MAX];
- const struct plt_memzone *mz;
- size_t model_object_size = 0;
- size_t model_xstats_size = 0;
- uint64_t mz_size = 0;
- TVMFunctionHandle create_fn = NULL;
- TVMFunctionHandle register_cb_fn = NULL;
- TVMModuleHandle module_so = NULL;
- TVMByteArray tvm_params;
- TVMValue ret_value = {0};
- TVMValue arg_values[4] = {0};
- TVMValue tvm_arg_values[1] = {0};
- int ret_type = kTVMNullptr;
- int arg_types[4] = {0};
- int tvm_arg_types[1] = {0};
- DLDevice device;
- int ret;
-
- RTE_SET_USED(cnxk_mldev);
- model->mvtvm.fd = -1;
-
- ret = mvtvm_ml_model_blob_parse(params, object);
- if (ret != 0)
- return ret;
-
- model_object_size = RTE_ALIGN_CEIL(object[0].size, RTE_CACHE_LINE_MIN_SIZE) +
- RTE_ALIGN_CEIL(object[1].size, RTE_CACHE_LINE_MIN_SIZE) +
- RTE_ALIGN_CEIL(object[2].size, RTE_CACHE_LINE_MIN_SIZE);
-
- model_xstats_size =
- cnxk_mldev->mldev->data->nb_queue_pairs * sizeof(struct mvtvm_ml_model_xstats);
-
- mz_size += model_object_size + model_xstats_size;
-
- /* Allocate memzone for model object */
- snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", MVTVM_ML_MODEL_MEMZONE_NAME, model->model_id);
- mz = plt_memzone_reserve_aligned(str, mz_size, 0, ML_CN10K_ALIGN_SIZE);
- if (!mz) {
- plt_err("plt_memzone_reserve failed : %s", str);
- return -ENOMEM;
- }
-
- /* Copy mod.so */
- model->mvtvm.so.buffer = mz->addr;
- model->mvtvm.so.size = object[0].size;
- rte_memcpy(model->mvtvm.so.name, object[0].name, RTE_ML_STR_MAX);
- rte_memcpy(model->mvtvm.so.buffer, object[0].buffer, object[0].size);
- rte_free(object[0].buffer);
-
- /* Copy mod.json */
- model->mvtvm.json.buffer =
- RTE_PTR_ADD(model->mvtvm.so.buffer,
- RTE_ALIGN_CEIL(model->mvtvm.so.size, RTE_CACHE_LINE_MIN_SIZE));
- model->mvtvm.json.size = object[1].size;
- rte_memcpy(model->mvtvm.json.name, object[1].name, RTE_ML_STR_MAX);
- rte_memcpy(model->mvtvm.json.buffer, object[1].buffer, object[1].size);
- rte_free(object[1].buffer);
-
- /* Copy mod.params */
- model->mvtvm.params.buffer =
- RTE_PTR_ADD(model->mvtvm.json.buffer,
- RTE_ALIGN_CEIL(model->mvtvm.json.size, RTE_CACHE_LINE_MIN_SIZE));
- model->mvtvm.params.size = object[2].size;
- rte_memcpy(model->mvtvm.params.name, object[2].name, RTE_ML_STR_MAX);
- rte_memcpy(model->mvtvm.params.buffer, object[2].buffer, object[2].size);
- rte_free(object[2].buffer);
-
- ret = mvtvm_ml_model_json_parse(model);
- if (ret != 0)
- goto error;
-
- if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_VDEV &&
- model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_LLVM) {
- plt_err("Unsupported model sub-type");
- ret = -ENOTSUP;
- goto error;
- }
-
- /* Set callback function array */
- if (model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_LLVM) {
- callback = &model->mvtvm.cb;
- callback->tvmrt_glow_layer_load = cn10k_ml_layer_load;
- callback->tvmrt_glow_layer_unload = cn10k_ml_layer_unload;
- callback->tvmrt_io_alloc = cn10k_ml_io_alloc;
- callback->tvmrt_io_free = cn10k_ml_io_free;
- callback->tvmrt_malloc = cn10k_ml_malloc;
- callback->tvmrt_free = cn10k_ml_free;
- callback->tvmrt_quantize = mvtvm_ml_io_quantize;
- callback->tvmrt_dequantize = mvtvm_ml_io_dequantize;
- callback->tvmrt_inference = cn10k_ml_inference_sync;
- } else {
- callback = NULL;
- }
-
- /* Initialize model in TVM runtime */
- if (model->mvtvm.graph_module != NULL) {
- ret = -EBUSY;
- plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
- model->model_id, ret);
- goto error;
- }
-
- snprintf(path, sizeof(path), "%s_%d_%u", ML_MODEL_SHMFD_NAME, getpid(), model->model_id);
- model->mvtvm.fd = memfd_create(path, 0);
- if (model->mvtvm.fd < 0) {
- ret = (errno == 0) ? -EIO : -errno;
- plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
- model->model_id, ret);
- goto error;
- }
-
- if (write(model->mvtvm.fd, model->mvtvm.so.buffer, model->mvtvm.so.size) !=
- (ssize_t)model->mvtvm.so.size) {
- ret = (errno == 0) ? -EIO : -errno;
- plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
- model->model_id, ret);
- goto error;
- }
-
- if (lseek(model->mvtvm.fd, 0, SEEK_SET) < 0) {
- ret = (errno == 0) ? -EIO : -errno;
- plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
- model->model_id, ret);
- goto error;
- }
-
- snprintf(path, sizeof(path), ML_MODEL_SHMFD_PATH, getpid(), model->mvtvm.fd);
- ret = TVMModLoadFromFile(path, "so", &module_so);
- if (ret != 0) {
- plt_err("TVM runtime: Model load failed, model_id = %u, ret = %d, msg = %s",
- model->model_id, ret, TVMGetLastError());
- goto error;
- }
-
- /* Set device info */
- device.device_type = kDLCPU;
- device.device_id = 0;
-
- if (callback != NULL) {
- ret = mvtvm_ml_tvm_func_get(model, module_so, "register_cb", ®ister_cb_fn);
- if (ret != 0)
- goto error;
-
- arg_values[0].v_handle = callback;
- arg_types[0] = kTVMOpaqueHandle;
- arg_values[1].v_handle = cnxk_mldev;
- arg_types[1] = kTVMOpaqueHandle;
- arg_values[2].v_int64 = model->model_id;
- arg_types[2] = kDLInt;
-
- ret = mvtvm_ml_tvm_func_call(model, register_cb_fn, "register_cb", arg_values,
- arg_types, 3, &ret_value, &ret_type, kTVMNullptr);
- if (ret != 0)
- goto error;
- }
-
- ret = TVMFuncGetGlobal("tvm.graph_executor.create", &create_fn);
- if (ret != 0) {
- plt_err("TVM runtime: Model load failed, model_id = %u, ret = %d, msg = %s",
- model->model_id, ret, TVMGetLastError());
- goto error;
- }
-
- arg_values[0].v_str = (const char *)model->mvtvm.json.buffer;
- arg_types[0] = kTVMStr;
- arg_values[1].v_handle = module_so;
- arg_types[1] = kTVMModuleHandle;
- arg_values[2].v_int64 = device.device_type;
- arg_types[2] = kDLInt;
- arg_values[3].v_int64 = device.device_id;
- arg_types[3] = kDLInt;
-
- ret = mvtvm_ml_tvm_func_call(model, create_fn, "tvm.graph_executor.create", arg_values,
- arg_types, 4, &ret_value, &ret_type, kTVMModuleHandle);
- if (ret != 0)
- goto error;
- model->mvtvm.graph_module = ret_value.v_handle;
-
- ret = mvtvm_ml_tvm_func_get(model, model->mvtvm.graph_module, "load_params",
- &model->mvtvm.load_params);
- if (ret != 0)
- goto error;
- ret = mvtvm_ml_tvm_func_get(model, model->mvtvm.graph_module, "set_input_zero_copy",
- &model->mvtvm.set_input_zero_copy);
- if (ret != 0)
- goto error;
- ret = mvtvm_ml_tvm_func_get(model, model->mvtvm.graph_module, "set_output_zero_copy",
- &model->mvtvm.set_output_zero_copy);
- if (ret != 0)
- goto error;
- ret = mvtvm_ml_tvm_func_get(model, model->mvtvm.graph_module, "run", &model->mvtvm.run);
- if (ret != 0)
- goto error;
-
- mvtvm_ml_tvm_func_free(®ister_cb_fn);
- mvtvm_ml_tvm_mod_free(&module_so);
-
- /* Load model parameters into TVM runtime */
- tvm_params.data = (const char *)model->mvtvm.params.buffer;
- tvm_params.size = model->mvtvm.params.size;
- tvm_arg_values[0].v_handle = &tvm_params;
- tvm_arg_types[0] = kTVMBytes;
-
- ret = mvtvm_ml_tvm_func_call(model, model->mvtvm.load_params, "load_params", tvm_arg_values,
- tvm_arg_types, 1, &ret_value, &ret_type, kTVMNullptr);
- if (ret != 0)
- goto error;
-
- /* Update model I/O data */
- mvtvm_ml_model_io_info_set(model);
-
- /* Set model info */
- mvtvm_ml_model_info_set(cnxk_mldev, model);
-
- /* Update model xstats name */
- cnxk_ml_xstats_model_name_update(cnxk_mldev, model->model_id);
-
- model->mvtvm.burst_xstats =
- RTE_PTR_ADD(model->mvtvm.params.buffer,
- RTE_ALIGN_CEIL(model->mvtvm.params.size, RTE_CACHE_LINE_MIN_SIZE));
-
- for (int qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) {
- model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_tot = 0;
- model->mvtvm.burst_xstats[qp_id].tvm_rt_latency = 0;
- model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_min = UINT64_MAX;
- model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_max = 0;
- model->mvtvm.burst_xstats[qp_id].tvm_rt_reset_count = 0;
- model->mvtvm.burst_xstats[qp_id].dequeued_count = 0;
- }
-
- /* Set model specific fast path functions */
- if (model->subtype == ML_CNXK_MODEL_SUBTYPE_TVM_MRVL) {
- model->enqueue_single = cn10k_ml_enqueue_single;
- model->result_update = cn10k_ml_result_update;
- model->set_error_code = cn10k_ml_set_error_code;
- model->set_poll_addr = cn10k_ml_set_poll_addr;
- model->op_error_get = cn10k_ml_op_error_get;
- } else {
- model->enqueue_single = mvtvm_ml_enqueue_single;
- model->result_update = mvtvm_ml_result_update;
- model->set_error_code = mvtvm_ml_set_error_code;
- model->set_poll_addr = mvtvm_ml_set_poll_addr;
- model->op_error_get = mvtvm_ml_op_error_get;
- }
-
- return 0;
-
-error:
- mvtvm_ml_tvm_func_free(®ister_cb_fn);
- if (model != NULL) {
- mvtvm_ml_tvm_func_free(&model->mvtvm.load_params);
- mvtvm_ml_tvm_func_free(&model->mvtvm.set_input_zero_copy);
- mvtvm_ml_tvm_func_free(&model->mvtvm.set_output_zero_copy);
- mvtvm_ml_tvm_func_free(&model->mvtvm.run);
- mvtvm_ml_tvm_mod_free(&model->mvtvm.graph_module);
- if (model->mvtvm.fd >= 0)
- close(model->mvtvm.fd);
- memset(&model->mvtvm, 0, sizeof(model->mvtvm));
- model->mvtvm.fd = -1;
- }
- mvtvm_ml_tvm_mod_free(&module_so);
- plt_memzone_free(mz);
-
- return ret;
-}
-
-int
-mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
-{
- char str[RTE_MEMZONE_NAMESIZE];
- const struct plt_memzone *mz;
-
- RTE_SET_USED(cnxk_mldev);
-
- /* Unload model from TVM runtime */
- if (model->model_id >= cnxk_mldev->mldev->data->nb_models)
- return -EINVAL;
-
- if (model->mvtvm.graph_module == NULL)
- return -EINVAL;
-
- mvtvm_ml_tvm_func_free(&model->mvtvm.load_params);
- mvtvm_ml_tvm_func_free(&model->mvtvm.set_input_zero_copy);
- mvtvm_ml_tvm_func_free(&model->mvtvm.set_output_zero_copy);
- mvtvm_ml_tvm_func_free(&model->mvtvm.run);
- mvtvm_ml_tvm_mod_free(&model->mvtvm.graph_module);
- if (model->mvtvm.fd >= 0)
- close(model->mvtvm.fd);
- memset(&model->mvtvm, 0, sizeof(model->mvtvm));
- model->mvtvm.fd = -1;
-
- snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", MVTVM_ML_MODEL_MEMZONE_NAME, model->model_id);
- mz = plt_memzone_lookup(str);
- if (mz == NULL) {
- plt_err("Memzone lookup failed for TVM model: model_id = %u, mz = %s",
- model->model_id, str);
- return -EINVAL;
- }
-
- return plt_memzone_free(mz);
-}
-
-int
-mvtvm_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
-{
- struct cnxk_ml_layer *layer;
-
- uint16_t layer_id = 0;
- int ret = 0;
-
-next_layer:
- layer = &model->layer[layer_id];
- if (layer->type == ML_CNXK_LAYER_TYPE_MRVL) {
- ret = cn10k_ml_layer_start(cnxk_mldev, model->model_id, layer->name);
- if (ret != 0) {
- plt_err("Layer start failed, model_id = %u, layer_name = %s, error = %d",
- model->model_id, layer->name, ret);
- return ret;
- }
- }
- layer_id++;
-
- if (layer_id < model->nb_layers)
- goto next_layer;
-
- return 0;
-}
-
-int
-mvtvm_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
-{
- struct cnxk_ml_layer *layer;
-
- uint16_t layer_id = 0;
- int ret = 0;
-
-next_layer:
- layer = &model->layer[layer_id];
- if (layer->type == ML_CNXK_LAYER_TYPE_MRVL) {
- ret = cn10k_ml_layer_stop(cnxk_mldev, model->model_id, layer->name);
- if (ret != 0) {
- plt_err("Layer stop failed, model_id = %u, layer_name = %s, error = %d",
- model->model_id, layer->name, ret);
- return ret;
- }
- }
- layer_id++;
-
- if (layer_id < model->nb_layers)
- goto next_layer;
-
- return 0;
-}
-
-int
-mvtvm_ml_io_quantize(void *device, uint16_t model_id, const char *layer_name,
- const DLTensor **deq_tensor, void *qbuffer)
-{
- struct cnxk_ml_io_info *info = NULL;
- struct cnxk_ml_dev *cnxk_mldev;
- struct cnxk_ml_model *model;
- uint16_t layer_id = 0;
- uint8_t *lcl_dbuffer;
- uint8_t *lcl_qbuffer;
- uint32_t i;
- int ret;
-
-#ifdef CNXK_ML_DEV_DEBUG
- if ((device == NULL) || (deq_tensor == NULL) || (qbuffer == NULL))
- return -EINVAL;
-#endif
-
- cnxk_mldev = (struct cnxk_ml_dev *)device;
-
- model = cnxk_mldev->mldev->data->models[model_id];
-#ifdef CNXK_ML_DEV_DEBUG
- if (model == NULL) {
- plt_err("Invalid model_id = %u", model_id);
- return -EINVAL;
- }
-#endif
-
- /* Get layer id */
- for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
- if (strcmp(model->layer[layer_id].name, layer_name) == 0)
- break;
- }
-
-#ifdef CNXK_ML_DEV_DEBUG
- if (layer_id == model->nb_layers) {
- plt_err("Invalid layer name: %s", layer_name);
- return -EINVAL;
- }
-
- if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) {
- plt_err("Invalid layer name / type: %s", layer_name);
- return -EINVAL;
- }
-#endif
-
- info = &model->layer[layer_id].info;
- lcl_qbuffer = (uint8_t *)qbuffer;
-
- for (i = 0; i < info->nb_inputs; i++) {
- lcl_dbuffer = PLT_PTR_ADD(deq_tensor[i]->data, deq_tensor[i]->byte_offset);
-
- ret = cnxk_ml_io_quantize_single(&info->input[i], lcl_dbuffer, lcl_qbuffer);
- if (ret < 0)
- return ret;
-
- lcl_qbuffer += info->input[i].sz_q;
- }
-
- return 0;
-}
-
-int
-mvtvm_ml_io_dequantize(void *device, uint16_t model_id, const char *layer_name, void *qbuffer,
- const DLTensor **deq_tensor)
-{
- struct cnxk_ml_io_info *info = NULL;
- struct cnxk_ml_dev *cnxk_mldev;
- struct cnxk_ml_model *model;
- uint16_t layer_id = 0;
- uint8_t *lcl_dbuffer;
- uint8_t *lcl_qbuffer;
- uint32_t i;
- int ret;
-
-#ifdef CNXK_ML_DEV_DEBUG
- if ((device == NULL) || (deq_tensor == NULL) || (qbuffer == NULL))
- return -EINVAL;
-#endif
-
- cnxk_mldev = (struct cnxk_ml_dev *)device;
-
- model = cnxk_mldev->mldev->data->models[model_id];
-#ifdef CNXK_ML_DEV_DEBUG
- if (model == NULL) {
- plt_err("Invalid model_id = %u", model_id);
- return -EINVAL;
- }
-#endif
-
- for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
- if (strcmp(model->layer[layer_id].name, layer_name) == 0)
- break;
- }
-
-#ifdef CNXK_ML_DEV_DEBUG
- if (layer_id == model->nb_layers) {
- plt_err("Invalid layer name: %s", layer_name);
- return -EINVAL;
- }
-
- if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) {
- plt_err("Invalid layer name / type: %s", layer_name);
- return -EINVAL;
- }
-#endif
-
- info = &model->layer[layer_id].info;
- lcl_qbuffer = (uint8_t *)qbuffer;
-
- for (i = 0; i < info->nb_outputs; i++) {
- lcl_dbuffer = PLT_PTR_ADD(deq_tensor[i]->data, deq_tensor[i]->byte_offset);
-
- ret = cnxk_ml_io_dequantize_single(&info->output[i], lcl_qbuffer, lcl_dbuffer);
- if (ret < 0)
- return ret;
-
- lcl_qbuffer += info->output[i].sz_q;
- }
-
- return 0;
-}
-
-static int
-mvtvm_ml_model_run(struct cnxk_ml_model *model, struct rte_ml_op *op, struct cnxk_ml_req *req)
-{
- uint8_t i;
- struct mvtvm_ml_result *run_result;
- TVMValue arg_values[2] = {0};
- int arg_types[2] = {0};
- TVMValue ret_value = {0};
- int ret_type = kTVMNullptr;
- int ret = 0;
-
- rte_memcpy(req->mvtvm_req.input_tensor, model->mvtvm.input_tensor,
- model->mvtvm.info.nb_inputs * sizeof(DLTensor));
- for (i = 0; i < model->mvtvm.info.nb_inputs; i++) {
- req->mvtvm_req.input_tensor[i].data = op->input[i]->addr;
- req->mvtvm_req.input_tensor[i].byte_offset = 0;
- }
-
- rte_memcpy(req->mvtvm_req.output_tensor, model->mvtvm.output_tensor,
- model->mvtvm.info.nb_outputs * sizeof(DLTensor));
- for (i = 0; i < model->mvtvm.info.nb_outputs; i++) {
- req->mvtvm_req.output_tensor[i].data = op->output[i]->addr;
- req->mvtvm_req.output_tensor[i].byte_offset = 0;
- }
-
- run_result = &req->mvtvm_req.result;
- run_result->stats.start_ns = rte_get_tsc_cycles();
- run_result->error_code = 0;
-
- for (i = 0; i < model->mvtvm.info.nb_inputs; i++) {
- arg_values[0].v_int64 = i;
- arg_types[0] = kDLInt;
- arg_values[1].v_handle = &req->mvtvm_req.input_tensor[i];
- arg_types[1] = kTVMDLTensorHandle;
- ret = mvtvm_ml_tvm_func_call(model, model->mvtvm.set_input_zero_copy,
- "set_input_zero_copy", arg_values, arg_types, 2,
- &ret_value, &ret_type, kTVMNullptr);
- if (ret != 0)
- goto out;
- }
-
- for (i = 0; i < model->mvtvm.info.nb_outputs; i++) {
- arg_values[0].v_int64 = i;
- arg_types[0] = kDLInt;
- arg_values[1].v_handle = &req->mvtvm_req.output_tensor[i];
- arg_types[1] = kTVMDLTensorHandle;
- ret = mvtvm_ml_tvm_func_call(model, model->mvtvm.set_output_zero_copy,
- "set_output_zero_copy", arg_values, arg_types, 2,
- &ret_value, &ret_type, kTVMNullptr);
- if (ret != 0)
- goto out;
- }
-
- ret = mvtvm_ml_tvm_func_call(model, model->mvtvm.run, "run", NULL, NULL, 0, &ret_value,
- &ret_type, kTVMNullptr);
- if (ret != 0)
- goto out;
-
-out:
- run_result->stats.end_ns = rte_get_tsc_cycles();
- req->mvtvm_req.status = 0x1;
-
- plt_write64(ML_CNXK_POLL_JOB_FINISH, req->status);
-
- if (ret != 0)
- run_result->error_code = -EIO;
-
- return 0;
-}
-
-__rte_hot void
-mvtvm_ml_set_error_code(struct cnxk_ml_req *req, uint64_t etype, uint64_t stype)
-{
- RTE_SET_USED(stype);
-
- req->mvtvm_req.result.error_code = etype;
-}
-
-__rte_hot int
-mvtvm_ml_op_error_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op,
- struct rte_ml_op_error *error)
-{
- RTE_SET_USED(cnxk_mldev);
- RTE_SET_USED(op);
- RTE_SET_USED(error);
-
- return 0;
-}
-
-__rte_hot bool
-mvtvm_ml_enqueue_single(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op, uint16_t layer_id,
- struct cnxk_ml_qp *qp, uint64_t head)
-{
- struct cnxk_ml_model *model;
- struct cnxk_ml_queue *queue;
- struct cnxk_ml_req *req;
-
- RTE_SET_USED(layer_id);
-
- queue = &qp->queue;
- req = &queue->reqs[head];
- model = cnxk_mldev->mldev->data->models[op->model_id];
-
- model->set_poll_addr(req);
- memset(&req->mvtvm_req.result, 0, sizeof(struct mvtvm_ml_result));
- req->mvtvm_req.result.error_code = 0x0;
- req->mvtvm_req.result.user_ptr = op->user_ptr;
-
- cnxk_ml_set_poll_ptr(req);
- mvtvm_ml_model_run(model, op, req);
- req->timeout = plt_tsc_cycles() + queue->wait_cycles;
- req->op = op;
-
- return true;
-}
-
-__rte_hot void
-mvtvm_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request)
-{
- struct mvtvm_ml_model_xstats *xstats;
- struct mvtvm_ml_result *result;
- struct cnxk_ml_model *model;
- struct cnxk_ml_req *req;
- uint64_t tvm_rt_latency;
- struct cnxk_ml_qp *qp;
- struct rte_ml_op *op;
-
- req = (struct cnxk_ml_req *)request;
- result = &req->mvtvm_req.result;
- op = req->op;
- qp = cnxk_mldev->mldev->data->queue_pairs[qp_id];
- op->impl_opaque = result->error_code;
-
- if (likely(result->error_code == 0)) {
- qp->stats.dequeued_count++;
- op->status = RTE_ML_OP_STATUS_SUCCESS;
-
- model = cnxk_mldev->mldev->data->models[op->model_id];
- xstats = &model->mvtvm.burst_xstats[qp_id];
-
- if (unlikely(xstats->dequeued_count == xstats->tvm_rt_reset_count)) {
- xstats->tvm_rt_latency_min = UINT64_MAX;
- xstats->tvm_rt_latency_max = 0;
- }
- tvm_rt_latency = result->stats.end_ns - result->stats.start_ns;
- xstats->tvm_rt_latency = tvm_rt_latency;
- xstats->tvm_rt_latency_tot += tvm_rt_latency;
- xstats->tvm_rt_latency_min = RTE_MIN(xstats->tvm_rt_latency_min, tvm_rt_latency);
- xstats->tvm_rt_latency_max = RTE_MAX(xstats->tvm_rt_latency_max, tvm_rt_latency);
- xstats->dequeued_count++;
- } else {
- qp->stats.dequeue_err_count++;
- op->status = RTE_ML_OP_STATUS_ERROR;
- }
-}
diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.h b/drivers/ml/cnxk/mvtvm_ml_ops.h
index 593f4585b3c..b83246517e7 100644
--- a/drivers/ml/cnxk/mvtvm_ml_ops.h
+++ b/drivers/ml/cnxk/mvtvm_ml_ops.h
@@ -5,77 +5,11 @@
#ifndef _MVTVM_ML_OPS_H_
#define _MVTVM_ML_OPS_H_
-#include <dlpack/dlpack.h>
-
#include <rte_mldev.h>
#include "cnxk_ml_xstats.h"
-#include "mvtvm_ml_model.h"
-
-struct cnxk_ml_dev;
-struct cnxk_ml_model;
-struct cnxk_ml_layer;
-struct cnxk_ml_qp;
-struct cnxk_ml_req;
-
-/* Inference stats */
-struct mvtvm_ml_stats {
- /* Start ns */
- uint64_t start_ns;
-
- /* Start ns */
- uint64_t end_ns;
-};
-
-/* Result structure */
-struct mvtvm_ml_result {
- /* Job error code */
- uint64_t error_code;
-
- /* Inference stats */
- struct mvtvm_ml_stats stats;
-
- /* User context pointer */
- void *user_ptr;
-};
-
-/* MVTVM specific request */
-struct mvtvm_ml_req {
- /* Input tensors */
- DLTensor input_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT];
-
- /* Output tensors */
- DLTensor output_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT];
-
- /* Status field for poll mode requests */
- volatile uint64_t status;
-
- /* Result */
- struct mvtvm_ml_result result;
-};
int mvtvm_ml_dev_info_get(struct cnxk_ml_dev *mldev, struct rte_ml_dev_info *dev_info);
int mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp);
-int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
- struct cnxk_ml_model *model);
-int mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
-int mvtvm_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
-int mvtvm_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
-int mvtvm_ml_io_quantize(void *device, uint16_t model_id, const char *layer_name,
- const DLTensor **deq_tensor, void *qbuffer);
-int mvtvm_ml_io_dequantize(void *device, uint16_t model_id, const char *layer_name, void *qbuffer,
- const DLTensor **deq_tensor);
-
-__rte_hot bool mvtvm_ml_enqueue_single(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op,
- uint16_t layer_id, struct cnxk_ml_qp *qp, uint64_t head);
-__rte_hot int mvtvm_ml_op_error_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op,
- struct rte_ml_op_error *error);
-__rte_hot void mvtvm_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request);
-__rte_hot void mvtvm_ml_set_error_code(struct cnxk_ml_req *req, uint64_t etype, uint64_t stype);
-
-void mvtvm_ml_model_xstat_name_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
- uint16_t stat_id, uint16_t entry, char *suffix);
-uint64_t mvtvm_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
- enum cnxk_ml_xstats_type type);
#endif /* _MVTVM_ML_OPS_H_ */
diff --git a/drivers/ml/cnxk/mvtvm_ml_stubs.c b/drivers/ml/cnxk/mvtvm_ml_stubs.c
index 7c13fac42d6..2341658657f 100644
--- a/drivers/ml/cnxk/mvtvm_ml_stubs.c
+++ b/drivers/ml/cnxk/mvtvm_ml_stubs.c
@@ -7,65 +7,6 @@
#include "mvtvm_ml_stubs.h"
#include "cnxk_ml_dev.h"
-#include "cnxk_ml_model.h"
-#include "cnxk_ml_xstats.h"
-
-enum cnxk_ml_model_type
-mvtvm_ml_model_type_get(struct rte_ml_model_params *params)
-{
- RTE_SET_USED(params);
-
- return ML_CNXK_MODEL_TYPE_UNKNOWN;
-}
-
-int
-mvtvm_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name, uint16_t *layer_id)
-{
- RTE_SET_USED(model);
- RTE_SET_USED(layer_name);
- RTE_SET_USED(layer_id);
-
- return -EINVAL;
-}
-
-struct cnxk_ml_io_info *
-mvtvm_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id)
-{
- RTE_SET_USED(model);
- RTE_SET_USED(layer_id);
-
- return NULL;
-}
-
-void
-mvtvm_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp)
-{
- RTE_SET_USED(cnxk_mldev);
- RTE_SET_USED(layer);
- RTE_SET_USED(fp);
-}
-
-void
-mvtvm_ml_model_xstat_name_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
- uint16_t stat_id, uint16_t entry, char *suffix)
-{
- RTE_SET_USED(cnxk_mldev);
- RTE_SET_USED(model);
- RTE_SET_USED(stat_id);
- RTE_SET_USED(entry);
- RTE_SET_USED(suffix);
-}
-
-uint64_t
-mvtvm_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
- enum cnxk_ml_xstats_type type)
-{
- RTE_SET_USED(cnxk_mldev);
- RTE_SET_USED(model);
- RTE_SET_USED(type);
-
- return 0;
-}
int
mvtvm_ml_dev_info_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_dev_info *dev_info)
@@ -84,41 +25,3 @@ mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp)
return -EINVAL;
}
-
-int
-mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
- struct cnxk_ml_model *model)
-{
- RTE_SET_USED(cnxk_mldev);
- RTE_SET_USED(params);
- RTE_SET_USED(model);
-
- return -EINVAL;
-}
-
-int
-mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
-{
- RTE_SET_USED(cnxk_mldev);
- RTE_SET_USED(model);
-
- return -EINVAL;
-}
-
-int
-mvtvm_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
-{
- RTE_SET_USED(cnxk_mldev);
- RTE_SET_USED(model);
-
- return -EINVAL;
-}
-
-int
-mvtvm_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
-{
- RTE_SET_USED(cnxk_mldev);
- RTE_SET_USED(model);
-
- return -EINVAL;
-}
diff --git a/drivers/ml/cnxk/mvtvm_ml_stubs.h b/drivers/ml/cnxk/mvtvm_ml_stubs.h
index 15985a75bf4..0eebdbb1ed2 100644
--- a/drivers/ml/cnxk/mvtvm_ml_stubs.h
+++ b/drivers/ml/cnxk/mvtvm_ml_stubs.h
@@ -7,28 +7,9 @@
#include <rte_mldev.h>
-#include "cnxk_ml_xstats.h"
-
struct cnxk_ml_dev;
-struct cnxk_ml_model;
-struct cnxk_ml_layer;
-enum cnxk_ml_model_type mvtvm_ml_model_type_get(struct rte_ml_model_params *params);
int mvtvm_ml_dev_info_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_dev_info *dev_info);
int mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp);
-int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
- struct cnxk_ml_model *model);
-int mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
-int mvtvm_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
-int mvtvm_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
-
-int mvtvm_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name,
- uint16_t *layer_id);
-struct cnxk_ml_io_info *mvtvm_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id);
-void mvtvm_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp);
-void mvtvm_ml_model_xstat_name_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
- uint16_t stat_id, uint16_t entry, char *suffix);
-uint64_t mvtvm_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
- enum cnxk_ml_xstats_type type);
#endif /* _MVTVM_ML_STUBS_H_ */
diff --git a/drivers/ml/cnxk/mvtvm_ml_model.c b/drivers/ml/cnxk/tvmrt_ml_model.c
similarity index 72%
rename from drivers/ml/cnxk/mvtvm_ml_model.c
rename to drivers/ml/cnxk/tvmrt_ml_model.c
index e1946607629..9bff424f0d8 100644
--- a/drivers/ml/cnxk/mvtvm_ml_model.c
+++ b/drivers/ml/cnxk/tvmrt_ml_model.c
@@ -1,5 +1,5 @@
/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright (c) 2023 Marvell.
+ * Copyright (c) 2026 Marvell.
*/
#include <errno.h>
@@ -22,13 +22,13 @@
#include "cnxk_ml_utils.h"
/* Objects list */
-char mvtvm_object_list[ML_MVTVM_MODEL_OBJECT_MAX][RTE_ML_STR_MAX] = {"mod.so", "mod.json",
- "mod.params"};
+static char tvmrt_object_list[ML_TVMRT_MODEL_OBJECT_MAX][RTE_ML_STR_MAX] = {"mod.so", "mod.json",
+ "mod.params"};
enum cnxk_ml_model_type
-mvtvm_ml_model_type_get(struct rte_ml_model_params *params)
+tvmrt_ml_model_type_get(struct rte_ml_model_params *params)
{
- bool object_found[ML_MVTVM_MODEL_OBJECT_MAX] = {false, false, false};
+ bool object_found[ML_TVMRT_MODEL_OBJECT_MAX] = {false, false, false};
enum cnxk_ml_model_type model_type;
struct archive_entry *entry;
struct archive *a;
@@ -47,18 +47,18 @@ mvtvm_ml_model_type_get(struct rte_ml_model_params *params)
/* Parse buffer for available objects */
while (archive_read_next_header(a, &entry) == ARCHIVE_OK) {
- for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) {
+ for (i = 0; i < ML_TVMRT_MODEL_OBJECT_MAX; i++) {
if (!object_found[i] &&
- (strcmp(archive_entry_pathname(entry), mvtvm_object_list[i]) == 0))
+ (strcmp(archive_entry_pathname(entry), tvmrt_object_list[i]) == 0))
object_found[i] = true;
}
archive_read_data_skip(a);
}
/* Check if all objects are available */
- for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) {
+ for (i = 0; i < ML_TVMRT_MODEL_OBJECT_MAX; i++) {
if (!object_found[i]) {
- plt_err("Object %s not found in archive!", mvtvm_object_list[i]);
+ plt_err("Object %s not found in archive!", tvmrt_object_list[i]);
model_type = ML_CNXK_MODEL_TYPE_INVALID;
goto cleanup;
}
@@ -74,15 +74,15 @@ mvtvm_ml_model_type_get(struct rte_ml_model_params *params)
}
int
-mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_model_object *object)
+tvmrt_ml_model_blob_parse(struct rte_ml_model_params *params, struct tvmrt_ml_model_object *object)
{
- bool object_found[ML_MVTVM_MODEL_OBJECT_MAX] = {false, false, false};
+ bool object_found[ML_TVMRT_MODEL_OBJECT_MAX] = {false, false, false};
struct archive_entry *entry;
struct archive *a;
uint8_t i;
int ret;
- memset(object, 0, ML_MVTVM_MODEL_OBJECT_MAX * sizeof(*object));
+ memset(object, 0, ML_TVMRT_MODEL_OBJECT_MAX * sizeof(*object));
/* Open archive */
a = archive_read_new();
@@ -97,10 +97,10 @@ mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_mo
/* Read archive */
while (archive_read_next_header(a, &entry) == ARCHIVE_OK) {
- for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) {
+ for (i = 0; i < ML_TVMRT_MODEL_OBJECT_MAX; i++) {
if (!object_found[i] &&
- (strcmp(archive_entry_pathname(entry), mvtvm_object_list[i]) == 0)) {
- memcpy(object[i].name, mvtvm_object_list[i], RTE_ML_STR_MAX);
+ (strcmp(archive_entry_pathname(entry), tvmrt_object_list[i]) == 0)) {
+ memcpy(object[i].name, tvmrt_object_list[i], RTE_ML_STR_MAX);
object[i].size = archive_entry_size(entry);
object[i].buffer = rte_malloc(NULL, object[i].size, 0);
if (object[i].buffer == NULL) {
@@ -122,9 +122,9 @@ mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_mo
}
/* Check if all objects are parsed */
- for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) {
+ for (i = 0; i < ML_TVMRT_MODEL_OBJECT_MAX; i++) {
if (!object_found[i]) {
- plt_err("Object %s not found in archive!", mvtvm_object_list[i]);
+ plt_err("Object %s not found in archive!", tvmrt_object_list[i]);
ret = -EINVAL;
goto error;
}
@@ -133,7 +133,7 @@ mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_mo
goto cleanup;
error:
- for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) {
+ for (i = 0; i < ML_TVMRT_MODEL_OBJECT_MAX; i++) {
if (object[i].buffer != NULL)
rte_free(object[i].buffer);
}
@@ -146,7 +146,7 @@ mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_mo
}
int
-mvtvm_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name, uint16_t *layer_id)
+tvmrt_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name, uint16_t *layer_id)
{
uint16_t i;
@@ -171,7 +171,7 @@ mvtvm_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name,
}
static void
-mvtvm_ml_param_names_free(struct mvtvm_ml_param_names *param_names)
+tvmrt_ml_param_names_free(struct tvmrt_ml_param_names *param_names)
{
size_t i;
@@ -184,7 +184,7 @@ mvtvm_ml_param_names_free(struct mvtvm_ml_param_names *param_names)
}
static int
-mvtvm_ml_blob_read_u64(const uint8_t *blob, size_t blob_size, size_t *offset, uint64_t *value)
+tvmrt_ml_blob_read_u64(const uint8_t *blob, size_t blob_size, size_t *offset, uint64_t *value)
{
if (*offset + sizeof(*value) > blob_size)
return -EINVAL;
@@ -196,8 +196,8 @@ mvtvm_ml_blob_read_u64(const uint8_t *blob, size_t blob_size, size_t *offset, ui
}
static int
-mvtvm_ml_param_names_parse(const uint8_t *blob, size_t blob_size,
- struct mvtvm_ml_param_names *param_names)
+tvmrt_ml_param_names_parse(const uint8_t *blob, size_t blob_size,
+ struct tvmrt_ml_param_names *param_names)
{
uint64_t magic = 0;
uint64_t reserved = 0;
@@ -208,18 +208,18 @@ mvtvm_ml_param_names_parse(const uint8_t *blob, size_t blob_size,
memset(param_names, 0, sizeof(*param_names));
- ret = mvtvm_ml_blob_read_u64(blob, blob_size, &offset, &magic);
+ ret = tvmrt_ml_blob_read_u64(blob, blob_size, &offset, &magic);
if (ret != 0)
return ret;
if (magic != TVM_NDARRAY_LIST_MAGIC)
return -EINVAL;
- ret = mvtvm_ml_blob_read_u64(blob, blob_size, &offset, &reserved);
+ ret = tvmrt_ml_blob_read_u64(blob, blob_size, &offset, &reserved);
if (ret != 0)
return ret;
- ret = mvtvm_ml_blob_read_u64(blob, blob_size, &offset, &count);
+ ret = tvmrt_ml_blob_read_u64(blob, blob_size, &offset, &count);
if (ret != 0)
return ret;
@@ -234,7 +234,7 @@ mvtvm_ml_param_names_parse(const uint8_t *blob, size_t blob_size,
for (i = 0; i < count; i++) {
uint64_t name_len = 0;
- ret = mvtvm_ml_blob_read_u64(blob, blob_size, &offset, &name_len);
+ ret = tvmrt_ml_blob_read_u64(blob, blob_size, &offset, &name_len);
if (ret != 0)
goto error;
@@ -261,12 +261,12 @@ mvtvm_ml_param_names_parse(const uint8_t *blob, size_t blob_size,
return 0;
error:
- mvtvm_ml_param_names_free(param_names);
+ tvmrt_ml_param_names_free(param_names);
return ret;
}
static int
-mvtvm_ml_dtype_parse(const char *dtype_str, DLDataType *dtype)
+tvmrt_ml_dtype_parse(const char *dtype_str, DLDataType *dtype)
{
const char *lanes_str;
char base[32] = {0};
@@ -321,7 +321,7 @@ mvtvm_ml_dtype_parse(const char *dtype_str, DLDataType *dtype)
}
static int
-mvtvm_ml_model_io_set(struct cnxk_ml_io *io, const char *name, json_t *shape, const char *dtype_str,
+tvmrt_ml_model_io_set(struct cnxk_ml_io *io, const char *name, json_t *shape, const char *dtype_str,
DLDevice device)
{
size_t i;
@@ -346,7 +346,7 @@ mvtvm_ml_model_io_set(struct cnxk_ml_io *io, const char *name, json_t *shape, co
io->shape_i64[i] = json_integer_value(dim);
}
- ret = mvtvm_ml_dtype_parse(dtype_str, &io->datatype);
+ ret = tvmrt_ml_dtype_parse(dtype_str, &io->datatype);
if (ret != 0)
return ret;
@@ -357,8 +357,9 @@ mvtvm_ml_model_io_set(struct cnxk_ml_io *io, const char *name, json_t *shape, co
return 0;
}
+
static int
-mvtvm_ml_json_graph_get_arrays(json_t *json_parsed, json_t **nodes, json_t **arg_nodes,
+tvmrt_ml_json_graph_get_arrays(json_t *json_parsed, json_t **nodes, json_t **arg_nodes,
json_t **heads, json_t **node_row_ptr, json_t **shape_values,
json_t **dtype_values)
{
@@ -395,9 +396,9 @@ mvtvm_ml_json_graph_get_arrays(json_t *json_parsed, json_t **nodes, json_t **arg
}
int
-mvtvm_ml_model_json_parse(struct cnxk_ml_model *model)
+tvmrt_ml_model_json_parse(struct cnxk_ml_model *model)
{
- struct mvtvm_ml_param_names param_names;
+ struct tvmrt_ml_param_names param_names;
json_error_t json_error;
json_t *json_parsed;
json_t *json_nodes;
@@ -414,16 +415,16 @@ mvtvm_ml_model_json_parse(struct cnxk_ml_model *model)
size_t j;
/* Parse param names from params buffer */
- ret = mvtvm_ml_param_names_parse(model->mvtvm.params.buffer, model->mvtvm.params.size,
+ ret = tvmrt_ml_param_names_parse(model->tvmrt.params.buffer, model->tvmrt.params.size,
¶m_names);
if (ret != 0)
return ret;
/* Load JSON graph (single load for both stages) */
- json_parsed = json_loadb((const char *)model->mvtvm.json.buffer, model->mvtvm.json.size, 0,
+ json_parsed = json_loadb((const char *)model->tvmrt.json.buffer, model->tvmrt.json.size, 0,
&json_error);
if (json_parsed == NULL) {
- mvtvm_ml_param_names_free(¶m_names);
+ tvmrt_ml_param_names_free(¶m_names);
plt_err("TVM runtime: Failed to parse JSON graph, model_id = %u", model->model_id);
return -EINVAL;
}
@@ -522,12 +523,12 @@ mvtvm_ml_model_json_parse(struct cnxk_ml_model *model)
device.device_type = kDLCPU;
device.device_id = 0;
- ret = mvtvm_ml_json_graph_get_arrays(json_parsed, &json_nodes, &json_arg_nodes, &json_heads,
+ ret = tvmrt_ml_json_graph_get_arrays(json_parsed, &json_nodes, &json_arg_nodes, &json_heads,
&json_node_row_ptr, &json_shape_values,
&json_dtype_values);
if (ret == 0) {
- model->mvtvm.info.nb_inputs = 0;
- model->mvtvm.info.nb_outputs = 0;
+ model->tvmrt.info.nb_inputs = 0;
+ model->tvmrt.info.nb_outputs = 0;
for (i = 0; i < json_array_size(json_arg_nodes); i++) {
json_t *arg_node_idx = json_array_get(json_arg_nodes, i);
@@ -561,18 +562,18 @@ mvtvm_ml_model_json_parse(struct cnxk_ml_model *model)
if (j < param_names.count)
continue;
- if (model->mvtvm.info.nb_inputs >= ML_CNXK_MODEL_MAX_INPUT_OUTPUT) {
+ if (model->tvmrt.info.nb_inputs >= ML_CNXK_MODEL_MAX_INPUT_OUTPUT) {
ret = -ENOTSUP;
break;
}
- ret = mvtvm_ml_model_io_set(
- &model->mvtvm.info.input[model->mvtvm.info.nb_inputs],
+ ret = tvmrt_ml_model_io_set(
+ &model->tvmrt.info.input[model->tvmrt.info.nb_inputs],
json_string_value(name), shape, json_string_value(dtype), device);
if (ret != 0)
break;
- model->mvtvm.info.nb_inputs++;
+ model->tvmrt.info.nb_inputs++;
}
for (i = 0; ret == 0 && i < json_array_size(json_heads); i++) {
@@ -641,18 +642,18 @@ mvtvm_ml_model_json_parse(struct cnxk_ml_model *model)
break;
}
- if (model->mvtvm.info.nb_outputs >= ML_CNXK_MODEL_MAX_INPUT_OUTPUT) {
+ if (model->tvmrt.info.nb_outputs >= ML_CNXK_MODEL_MAX_INPUT_OUTPUT) {
ret = -ENOTSUP;
break;
}
- ret = mvtvm_ml_model_io_set(
- &model->mvtvm.info.output[model->mvtvm.info.nb_outputs],
+ ret = tvmrt_ml_model_io_set(
+ &model->tvmrt.info.output[model->tvmrt.info.nb_outputs],
json_string_value(name), shape, json_string_value(dtype), device);
if (ret != 0)
break;
- model->mvtvm.info.nb_outputs++;
+ model->tvmrt.info.nb_outputs++;
}
}
@@ -661,17 +662,17 @@ mvtvm_ml_model_json_parse(struct cnxk_ml_model *model)
model->model_id, ret);
json_decref(json_parsed);
- mvtvm_ml_param_names_free(¶m_names);
+ tvmrt_ml_param_names_free(¶m_names);
return ret;
error:
json_decref(json_parsed);
- mvtvm_ml_param_names_free(¶m_names);
+ tvmrt_ml_param_names_free(¶m_names);
return ret;
}
static enum rte_ml_io_type
-mvtvm_ml_io_type_map(DLDataType dltype)
+tvmrt_ml_io_type_map(DLDataType dltype)
{
switch (dltype.code) {
case kDLInt:
@@ -714,7 +715,7 @@ mvtvm_ml_io_type_map(DLDataType dltype)
}
void
-mvtvm_ml_model_io_info_set(struct cnxk_ml_model *model)
+tvmrt_ml_model_io_info_set(struct cnxk_ml_model *model)
{
int32_t i;
uint32_t j;
@@ -723,97 +724,97 @@ mvtvm_ml_model_io_info_set(struct cnxk_ml_model *model)
goto tvm_mrvl_model;
/* Inputs, set for layer_id = 0 */
- model->mvtvm.info.total_input_sz_d = 0;
- model->mvtvm.info.total_input_sz_q = 0;
- for (i = 0; i < model->mvtvm.info.nb_inputs; i++) {
- model->mvtvm.info.input[i].dtype =
- mvtvm_ml_io_type_map(model->mvtvm.info.input[i].datatype);
- model->mvtvm.info.input[i].qtype =
- mvtvm_ml_io_type_map(model->mvtvm.info.input[i].model_datatype);
-
- model->mvtvm.info.input[i].nb_elements = 1;
- for (j = 0; j < model->mvtvm.info.input[i].nb_dims; j++) {
- model->mvtvm.info.input[i].shape[j] =
- PLT_U32_CAST(model->mvtvm.info.input[i].shape_i64[j]);
- model->mvtvm.info.input[i].nb_elements *=
- model->mvtvm.info.input[i].shape[j];
+ model->tvmrt.info.total_input_sz_d = 0;
+ model->tvmrt.info.total_input_sz_q = 0;
+ for (i = 0; i < model->tvmrt.info.nb_inputs; i++) {
+ model->tvmrt.info.input[i].dtype =
+ tvmrt_ml_io_type_map(model->tvmrt.info.input[i].datatype);
+ model->tvmrt.info.input[i].qtype =
+ tvmrt_ml_io_type_map(model->tvmrt.info.input[i].model_datatype);
+
+ model->tvmrt.info.input[i].nb_elements = 1;
+ for (j = 0; j < model->tvmrt.info.input[i].nb_dims; j++) {
+ model->tvmrt.info.input[i].shape[j] =
+ PLT_U32_CAST(model->tvmrt.info.input[i].shape_i64[j]);
+ model->tvmrt.info.input[i].nb_elements *=
+ model->tvmrt.info.input[i].shape[j];
}
- model->mvtvm.info.input[i].sz_d =
- model->mvtvm.info.input[i].nb_elements *
- rte_ml_io_type_size_get(model->mvtvm.info.input[i].dtype);
- model->mvtvm.info.input[i].sz_q =
- model->mvtvm.info.input[i].nb_elements *
- rte_ml_io_type_size_get(model->mvtvm.info.input[i].qtype);
+ model->tvmrt.info.input[i].sz_d =
+ model->tvmrt.info.input[i].nb_elements *
+ rte_ml_io_type_size_get(model->tvmrt.info.input[i].dtype);
+ model->tvmrt.info.input[i].sz_q =
+ model->tvmrt.info.input[i].nb_elements *
+ rte_ml_io_type_size_get(model->tvmrt.info.input[i].qtype);
- model->mvtvm.info.total_input_sz_d += model->mvtvm.info.input[i].sz_d;
- model->mvtvm.info.total_input_sz_q += model->mvtvm.info.input[i].sz_q;
+ model->tvmrt.info.total_input_sz_d += model->tvmrt.info.input[i].sz_d;
+ model->tvmrt.info.total_input_sz_q += model->tvmrt.info.input[i].sz_q;
- model->mvtvm.input_tensor[i].device = model->mvtvm.info.input[i].device;
- model->mvtvm.input_tensor[i].ndim = model->mvtvm.info.input[i].nb_dims;
- model->mvtvm.input_tensor[i].dtype = model->mvtvm.info.input[i].datatype;
- model->mvtvm.input_tensor[i].shape = model->mvtvm.info.input[i].shape_i64;
- model->mvtvm.input_tensor[i].strides = NULL;
- model->mvtvm.input_tensor[i].byte_offset = 0;
+ model->tvmrt.input_tensor[i].device = model->tvmrt.info.input[i].device;
+ model->tvmrt.input_tensor[i].ndim = model->tvmrt.info.input[i].nb_dims;
+ model->tvmrt.input_tensor[i].dtype = model->tvmrt.info.input[i].datatype;
+ model->tvmrt.input_tensor[i].shape = model->tvmrt.info.input[i].shape_i64;
+ model->tvmrt.input_tensor[i].strides = NULL;
+ model->tvmrt.input_tensor[i].byte_offset = 0;
plt_ml_dbg("model_id = %u, input[%u] - sz_d = %u sz_q = %u", model->model_id, i,
- model->mvtvm.info.input[i].sz_d, model->mvtvm.info.input[i].sz_q);
+ model->tvmrt.info.input[i].sz_d, model->tvmrt.info.input[i].sz_q);
}
/* Outputs, set for nb_layers - 1 */
- model->mvtvm.info.total_output_sz_d = 0;
- model->mvtvm.info.total_output_sz_q = 0;
- for (i = 0; i < model->mvtvm.info.nb_outputs; i++) {
- model->mvtvm.info.output[i].dtype =
- mvtvm_ml_io_type_map(model->mvtvm.info.output[i].datatype);
- model->mvtvm.info.output[i].qtype =
- mvtvm_ml_io_type_map(model->mvtvm.info.output[i].model_datatype);
-
- model->mvtvm.info.output[i].nb_elements = 1;
- for (j = 0; j < model->mvtvm.info.output[i].nb_dims; j++) {
- model->mvtvm.info.output[i].shape[j] =
- PLT_U32_CAST(model->mvtvm.info.output[i].shape_i64[j]);
- model->mvtvm.info.output[i].nb_elements *=
- model->mvtvm.info.output[i].shape[j];
+ model->tvmrt.info.total_output_sz_d = 0;
+ model->tvmrt.info.total_output_sz_q = 0;
+ for (i = 0; i < model->tvmrt.info.nb_outputs; i++) {
+ model->tvmrt.info.output[i].dtype =
+ tvmrt_ml_io_type_map(model->tvmrt.info.output[i].datatype);
+ model->tvmrt.info.output[i].qtype =
+ tvmrt_ml_io_type_map(model->tvmrt.info.output[i].model_datatype);
+
+ model->tvmrt.info.output[i].nb_elements = 1;
+ for (j = 0; j < model->tvmrt.info.output[i].nb_dims; j++) {
+ model->tvmrt.info.output[i].shape[j] =
+ PLT_U32_CAST(model->tvmrt.info.output[i].shape_i64[j]);
+ model->tvmrt.info.output[i].nb_elements *=
+ model->tvmrt.info.output[i].shape[j];
}
- model->mvtvm.info.output[i].sz_d =
- model->mvtvm.info.output[i].nb_elements *
- rte_ml_io_type_size_get(model->mvtvm.info.output[i].dtype);
- model->mvtvm.info.output[i].sz_q =
- model->mvtvm.info.output[i].nb_elements *
- rte_ml_io_type_size_get(model->mvtvm.info.output[i].qtype);
+ model->tvmrt.info.output[i].sz_d =
+ model->tvmrt.info.output[i].nb_elements *
+ rte_ml_io_type_size_get(model->tvmrt.info.output[i].dtype);
+ model->tvmrt.info.output[i].sz_q =
+ model->tvmrt.info.output[i].nb_elements *
+ rte_ml_io_type_size_get(model->tvmrt.info.output[i].qtype);
- model->mvtvm.info.total_output_sz_d += model->mvtvm.info.output[i].sz_d;
- model->mvtvm.info.total_output_sz_q += model->mvtvm.info.output[i].sz_q;
+ model->tvmrt.info.total_output_sz_d += model->tvmrt.info.output[i].sz_d;
+ model->tvmrt.info.total_output_sz_q += model->tvmrt.info.output[i].sz_q;
- model->mvtvm.output_tensor[i].device = model->mvtvm.info.output[i].device;
- model->mvtvm.output_tensor[i].ndim = model->mvtvm.info.output[i].nb_dims;
- model->mvtvm.output_tensor[i].dtype = model->mvtvm.info.output[i].datatype;
- model->mvtvm.output_tensor[i].shape = model->mvtvm.info.output[i].shape_i64;
- model->mvtvm.output_tensor[i].strides = NULL;
- model->mvtvm.output_tensor[i].byte_offset = 0;
+ model->tvmrt.output_tensor[i].device = model->tvmrt.info.output[i].device;
+ model->tvmrt.output_tensor[i].ndim = model->tvmrt.info.output[i].nb_dims;
+ model->tvmrt.output_tensor[i].dtype = model->tvmrt.info.output[i].datatype;
+ model->tvmrt.output_tensor[i].shape = model->tvmrt.info.output[i].shape_i64;
+ model->tvmrt.output_tensor[i].strides = NULL;
+ model->tvmrt.output_tensor[i].byte_offset = 0;
plt_ml_dbg("model_id = %u, output[%u] - sz_d = %u sz_q = %u", model->model_id, i,
- model->mvtvm.info.output[i].sz_d, model->mvtvm.info.output[i].sz_q);
+ model->tvmrt.info.output[i].sz_d, model->tvmrt.info.output[i].sz_q);
}
return;
tvm_mrvl_model:
- cn10k_ml_layer_io_info_set(&model->mvtvm.info, &model->layer[0].glow.metadata);
+ cn10k_ml_layer_io_info_set(&model->tvmrt.info, &model->layer[0].glow.metadata);
}
struct cnxk_ml_io_info *
-mvtvm_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id)
+tvmrt_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id)
{
RTE_SET_USED(layer_id);
- return &model->mvtvm.info;
+ return &model->tvmrt.info;
}
void
-mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
+tvmrt_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
{
struct rte_ml_model_info *info;
struct rte_ml_io_info *output;
@@ -837,43 +838,43 @@ mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mo
info->io_layout = RTE_ML_IO_LAYOUT_SPLIT;
info->min_batches = model->batch_size;
info->max_batches = model->batch_size;
- info->nb_inputs = model->mvtvm.info.nb_inputs;
+ info->nb_inputs = model->tvmrt.info.nb_inputs;
info->input_info = input;
- info->nb_outputs = model->mvtvm.info.nb_outputs;
+ info->nb_outputs = model->tvmrt.info.nb_outputs;
info->output_info = output;
info->wb_size = 0;
/* Set input info */
for (i = 0; i < info->nb_inputs; i++) {
- rte_memcpy(input[i].name, model->mvtvm.info.input[i].name, MRVL_ML_INPUT_NAME_LEN);
- input[i].nb_dims = model->mvtvm.info.input[i].nb_dims;
- input[i].shape = &model->mvtvm.info.input[i].shape[0];
- input[i].type = model->mvtvm.info.input[i].qtype;
- input[i].nb_elements = model->mvtvm.info.input[i].nb_elements;
- input[i].size = model->mvtvm.info.input[i].nb_elements *
- rte_ml_io_type_size_get(model->mvtvm.info.input[i].qtype);
- input[i].scale = model->mvtvm.info.input[i].scale;
+ rte_memcpy(input[i].name, model->tvmrt.info.input[i].name, MRVL_ML_INPUT_NAME_LEN);
+ input[i].nb_dims = model->tvmrt.info.input[i].nb_dims;
+ input[i].shape = &model->tvmrt.info.input[i].shape[0];
+ input[i].type = model->tvmrt.info.input[i].qtype;
+ input[i].nb_elements = model->tvmrt.info.input[i].nb_elements;
+ input[i].size = model->tvmrt.info.input[i].nb_elements *
+ rte_ml_io_type_size_get(model->tvmrt.info.input[i].qtype);
+ input[i].scale = model->tvmrt.info.input[i].scale;
input[i].zero_point = 0;
}
/* Set output info */
for (i = 0; i < info->nb_outputs; i++) {
- rte_memcpy(output[i].name, model->mvtvm.info.output[i].name,
+ rte_memcpy(output[i].name, model->tvmrt.info.output[i].name,
MRVL_ML_OUTPUT_NAME_LEN);
- output[i].nb_dims = model->mvtvm.info.output[i].nb_dims;
- output[i].shape = &model->mvtvm.info.output[i].shape[0];
- output[i].type = model->mvtvm.info.output[i].qtype;
- output[i].nb_elements = model->mvtvm.info.output[i].nb_elements;
- output[i].size = model->mvtvm.info.output[i].nb_elements *
- rte_ml_io_type_size_get(model->mvtvm.info.output[i].qtype);
- output[i].scale = model->mvtvm.info.output[i].scale;
+ output[i].nb_dims = model->tvmrt.info.output[i].nb_dims;
+ output[i].shape = &model->tvmrt.info.output[i].shape[0];
+ output[i].type = model->tvmrt.info.output[i].qtype;
+ output[i].nb_elements = model->tvmrt.info.output[i].nb_elements;
+ output[i].size = model->tvmrt.info.output[i].nb_elements *
+ rte_ml_io_type_size_get(model->tvmrt.info.output[i].qtype);
+ output[i].scale = model->tvmrt.info.output[i].scale;
output[i].zero_point = 0;
}
return;
tvm_mrvl_model:
- cn10k_ml_model_info_set(cnxk_mldev, model, &model->mvtvm.info,
+ cn10k_ml_model_info_set(cnxk_mldev, model, &model->tvmrt.info,
&model->layer[0].glow.metadata);
strlcpy(info->name, model->name, RTE_ML_STR_MAX);
@@ -882,7 +883,7 @@ mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mo
}
void
-mvtvm_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp)
+tvmrt_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp)
{
char str[STR_LEN];
uint8_t i;
diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/tvmrt_ml_model.h
similarity index 78%
rename from drivers/ml/cnxk/mvtvm_ml_model.h
rename to drivers/ml/cnxk/tvmrt_ml_model.h
index 1151b0a39ed..bf35dadb8a6 100644
--- a/drivers/ml/cnxk/mvtvm_ml_model.h
+++ b/drivers/ml/cnxk/tvmrt_ml_model.h
@@ -1,9 +1,9 @@
/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright (c) 2023 Marvell.
+ * Copyright (c) 2026 Marvell.
*/
-#ifndef _MVTVM_ML_MODEL_H_
-#define _MVTVM_ML_MODEL_H_
+#ifndef _TVMRT_ML_MODEL_H_
+#define _TVMRT_ML_MODEL_H_
#include <dlpack/dlpack.h>
#include <jansson.h>
@@ -30,19 +30,19 @@ struct cnxk_ml_model;
struct cnxk_ml_layer;
/* Maximum number of objects per model */
-#define ML_MVTVM_MODEL_OBJECT_MAX 3
+#define ML_TVMRT_MODEL_OBJECT_MAX 3
/* Magic number for TVM parameter blobs. */
#define TVM_NDARRAY_LIST_MAGIC 0xF7E58D4F05049CB7ULL
/* TVM parameter names structure */
-struct mvtvm_ml_param_names {
+struct tvmrt_ml_param_names {
char **name;
size_t count;
};
/* TVM object / artifact info structure */
-struct mvtvm_ml_model_object {
+struct tvmrt_ml_model_object {
/* Name */
char name[RTE_ML_STR_MAX];
@@ -85,7 +85,7 @@ struct tvmrt_glow_callback {
};
/* Model fast-path stats */
-struct mvtvm_ml_model_xstats {
+struct tvmrt_ml_model_xstats {
/* Total TVM runtime latency, sum of all inferences */
uint64_t tvm_rt_latency_tot;
@@ -105,11 +105,11 @@ struct mvtvm_ml_model_xstats {
uint64_t tvm_rt_reset_count;
};
-struct mvtvm_ml_model_data {
+struct tvmrt_ml_model_data {
/* Model objects */
- struct mvtvm_ml_model_object so;
- struct mvtvm_ml_model_object json;
- struct mvtvm_ml_model_object params;
+ struct tvmrt_ml_model_object so;
+ struct tvmrt_ml_model_object json;
+ struct tvmrt_ml_model_object params;
/* TVM runtime callbacks */
struct tvmrt_glow_callback cb;
@@ -130,7 +130,7 @@ struct mvtvm_ml_model_data {
struct cnxk_ml_io_info info;
/* Stats for burst ops */
- struct mvtvm_ml_model_xstats *burst_xstats;
+ struct tvmrt_ml_model_xstats *burst_xstats;
/* Input Tensor */
DLTensor input_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT];
@@ -139,15 +139,15 @@ struct mvtvm_ml_model_data {
DLTensor output_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT];
};
-enum cnxk_ml_model_type mvtvm_ml_model_type_get(struct rte_ml_model_params *params);
-int mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params,
- struct mvtvm_ml_model_object *object);
-int mvtvm_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name,
+enum cnxk_ml_model_type tvmrt_ml_model_type_get(struct rte_ml_model_params *params);
+int tvmrt_ml_model_blob_parse(struct rte_ml_model_params *params,
+ struct tvmrt_ml_model_object *object);
+int tvmrt_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name,
uint16_t *layer_id);
-void mvtvm_ml_model_io_info_set(struct cnxk_ml_model *model);
-struct cnxk_ml_io_info *mvtvm_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id);
-void mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
-void mvtvm_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp);
-int mvtvm_ml_model_json_parse(struct cnxk_ml_model *model);
+void tvmrt_ml_model_io_info_set(struct cnxk_ml_model *model);
+struct cnxk_ml_io_info *tvmrt_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id);
+void tvmrt_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
+void tvmrt_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp);
+int tvmrt_ml_model_json_parse(struct cnxk_ml_model *model);
-#endif /* _MVTVM_ML_MODEL_H_ */
+#endif /* _TVMRT_ML_MODEL_H_ */
diff --git a/drivers/ml/cnxk/tvmrt_ml_ops.c b/drivers/ml/cnxk/tvmrt_ml_ops.c
new file mode 100644
index 00000000000..f6b3a51c51e
--- /dev/null
+++ b/drivers/ml/cnxk/tvmrt_ml_ops.c
@@ -0,0 +1,818 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2026 Marvell.
+ */
+
+#include <errno.h>
+#include <linux/limits.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+#include <dlpack/dlpack.h>
+#include <jansson.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_mldev.h>
+#include <rte_mldev_pmd.h>
+
+#include <mldev_utils.h>
+
+#include "cnxk_ml_dev.h"
+#include "cnxk_ml_model.h"
+#include "cnxk_ml_ops.h"
+#include "cnxk_ml_xstats.h"
+
+/* ML model macros */
+#define TVMRT_ML_MODEL_MEMZONE_NAME "ml_tvmrt_model_mz"
+
+/* Shared memory file descriptor name */
+#define ML_MODEL_SHMFD_NAME "tvmrt_shmfd"
+
+/* Shared memory file descriptor path */
+#define ML_MODEL_SHMFD_PATH "/proc/%d/fd/%d"
+
+static int
+tvmrt_ml_tvm_func_get(struct cnxk_ml_model *model, TVMModuleHandle module, const char *name,
+ TVMFunctionHandle *func)
+{
+ int ret;
+
+ ret = TVMModGetFunction(module, name, 0, func);
+ if (ret != 0) {
+ plt_err("Model load failed, model_id = %u, ret = %d, msg = %s", model->model_id,
+ ret, TVMGetLastError());
+ return ret;
+ }
+
+ if (*func == NULL) {
+ ret = -ENOENT;
+ plt_err("Model load failed, model_id = %u, function '%s' not found",
+ model->model_id, name);
+ }
+
+ return ret;
+}
+
+static int
+tvmrt_ml_tvm_func_call(struct cnxk_ml_model *model, TVMFunctionHandle func, const char *name,
+ TVMValue *values, int *types, int num_args, TVMValue *ret_val, int *ret_type,
+ int ret_type_code)
+{
+ int ret;
+
+ ret = TVMFuncCall(func, values, types, num_args, ret_val, ret_type);
+ if (ret != 0) {
+ plt_err("Error calling TVM function '%s', model_id = %u, ret = %d, msg = %s", name,
+ model->model_id, ret, TVMGetLastError());
+ return ret;
+ }
+
+ if (*ret_type != ret_type_code) {
+ ret = -EINVAL;
+ plt_err("TVM function '%s' returned unexpected type, model_id = %u, expected = %d, "
+ "actual = %d",
+ name, model->model_id, ret_type_code, *ret_type);
+ }
+
+ return ret;
+}
+
+static void
+tvmrt_ml_tvm_func_free(TVMFunctionHandle *func)
+{
+ if ((func != NULL) && (*func != NULL)) {
+ TVMFuncFree(*func);
+ *func = NULL;
+ }
+}
+
+static void
+tvmrt_ml_tvm_mod_free(TVMModuleHandle *mod)
+{
+ if ((mod != NULL) && (*mod != NULL)) {
+ TVMModFree(*mod);
+ *mod = NULL;
+ }
+}
+
+__rte_hot static void
+tvmrt_ml_set_poll_addr(struct cnxk_ml_req *req)
+{
+ req->status = &req->tvmrt_req.status;
+}
+
+void
+tvmrt_ml_model_xstat_name_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
+ uint16_t stat_id, uint16_t entry, char *suffix)
+{
+ snprintf(cnxk_mldev->xstats.entries[stat_id].map.name,
+ sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", model->name,
+ model_xstats[entry].name, suffix);
+}
+
+#define ML_AVG_FOREACH_QP_TVMRT(cnxk_mldev, model, qp_id, value, count) \
+ do { \
+ value = 0; \
+ for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \
+ value += model->tvmrt.burst_xstats[qp_id].tvm_rt_latency_tot; \
+ count += model->tvmrt.burst_xstats[qp_id].dequeued_count - \
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_reset_count; \
+ } \
+ if (count != 0) \
+ value = value / count; \
+ } while (0)
+
+#define ML_MIN_FOREACH_QP_TVMRT(cnxk_mldev, model, qp_id, value, count) \
+ do { \
+ value = UINT64_MAX; \
+ for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \
+ value = PLT_MIN(value, \
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_latency_min); \
+ count += model->tvmrt.burst_xstats[qp_id].dequeued_count - \
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_reset_count; \
+ } \
+ if (count == 0) \
+ value = 0; \
+ } while (0)
+
+#define ML_MAX_FOREACH_QP_TVMRT(cnxk_mldev, model, qp_id, value, count) \
+ do { \
+ value = 0; \
+ for (qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) { \
+ value = PLT_MAX(value, \
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_latency_max); \
+ count += model->tvmrt.burst_xstats[qp_id].dequeued_count - \
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_reset_count; \
+ } \
+ if (count == 0) \
+ value = 0; \
+ } while (0)
+
+uint64_t
+tvmrt_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
+ enum cnxk_ml_xstats_type type)
+{
+ uint64_t count = 0;
+ uint64_t value = 0;
+ uint32_t qp_id;
+
+ switch (type) {
+ case avg_rt_latency:
+ ML_AVG_FOREACH_QP_TVMRT(cnxk_mldev, model, qp_id, value, count);
+ break;
+ case min_rt_latency:
+ ML_MIN_FOREACH_QP_TVMRT(cnxk_mldev, model, qp_id, value, count);
+ break;
+ case max_rt_latency:
+ ML_MAX_FOREACH_QP_TVMRT(cnxk_mldev, model, qp_id, value, count);
+ break;
+ default:
+ value = 0;
+ }
+
+ return value;
+}
+
+int
+tvmrt_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
+ struct cnxk_ml_model *model)
+{
+ struct tvmrt_ml_model_object object[ML_TVMRT_MODEL_OBJECT_MAX];
+ struct tvmrt_glow_callback *callback;
+ char str[RTE_MEMZONE_NAMESIZE];
+ char path[PATH_MAX];
+ const struct plt_memzone *mz;
+ size_t model_object_size = 0;
+ size_t model_xstats_size = 0;
+ uint64_t mz_size = 0;
+ TVMFunctionHandle create_fn = NULL;
+ TVMFunctionHandle register_cb_fn = NULL;
+ TVMModuleHandle module_so = NULL;
+ TVMByteArray tvm_params;
+ TVMValue ret_value = {0};
+ TVMValue arg_values[4] = {0};
+ TVMValue tvm_arg_values[1] = {0};
+ int ret_type = kTVMNullptr;
+ int arg_types[4] = {0};
+ int tvm_arg_types[1] = {0};
+ DLDevice device;
+ int ret;
+
+ RTE_SET_USED(cnxk_mldev);
+ model->tvmrt.fd = -1;
+
+ ret = tvmrt_ml_model_blob_parse(params, object);
+ if (ret != 0)
+ return ret;
+
+ model_object_size = RTE_ALIGN_CEIL(object[0].size, RTE_CACHE_LINE_MIN_SIZE) +
+ RTE_ALIGN_CEIL(object[1].size, RTE_CACHE_LINE_MIN_SIZE) +
+ RTE_ALIGN_CEIL(object[2].size, RTE_CACHE_LINE_MIN_SIZE);
+
+ model_xstats_size =
+ cnxk_mldev->mldev->data->nb_queue_pairs * sizeof(struct tvmrt_ml_model_xstats);
+
+ mz_size += model_object_size + model_xstats_size;
+
+ /* Allocate memzone for model object */
+ snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", TVMRT_ML_MODEL_MEMZONE_NAME, model->model_id);
+ mz = plt_memzone_reserve_aligned(str, mz_size, 0, ML_CN10K_ALIGN_SIZE);
+ if (!mz) {
+ plt_err("plt_memzone_reserve failed : %s", str);
+ return -ENOMEM;
+ }
+
+ /* Copy mod.so */
+ model->tvmrt.so.buffer = mz->addr;
+ model->tvmrt.so.size = object[0].size;
+ rte_memcpy(model->tvmrt.so.name, object[0].name, RTE_ML_STR_MAX);
+ rte_memcpy(model->tvmrt.so.buffer, object[0].buffer, object[0].size);
+ rte_free(object[0].buffer);
+
+ /* Copy mod.json */
+ model->tvmrt.json.buffer =
+ RTE_PTR_ADD(model->tvmrt.so.buffer,
+ RTE_ALIGN_CEIL(model->tvmrt.so.size, RTE_CACHE_LINE_MIN_SIZE));
+ model->tvmrt.json.size = object[1].size;
+ rte_memcpy(model->tvmrt.json.name, object[1].name, RTE_ML_STR_MAX);
+ rte_memcpy(model->tvmrt.json.buffer, object[1].buffer, object[1].size);
+ rte_free(object[1].buffer);
+
+ /* Copy mod.params */
+ model->tvmrt.params.buffer =
+ RTE_PTR_ADD(model->tvmrt.json.buffer,
+ RTE_ALIGN_CEIL(model->tvmrt.json.size, RTE_CACHE_LINE_MIN_SIZE));
+ model->tvmrt.params.size = object[2].size;
+ rte_memcpy(model->tvmrt.params.name, object[2].name, RTE_ML_STR_MAX);
+ rte_memcpy(model->tvmrt.params.buffer, object[2].buffer, object[2].size);
+ rte_free(object[2].buffer);
+
+ ret = tvmrt_ml_model_json_parse(model);
+ if (ret != 0)
+ goto error;
+
+ if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_VDEV &&
+ model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_LLVM) {
+ plt_err("Unsupported model sub-type");
+ ret = -ENOTSUP;
+ goto error;
+ }
+
+ /* Set callback function array */
+ if (model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_LLVM) {
+ callback = &model->tvmrt.cb;
+ callback->tvmrt_glow_layer_load = cn10k_ml_layer_load;
+ callback->tvmrt_glow_layer_unload = cn10k_ml_layer_unload;
+ callback->tvmrt_io_alloc = cn10k_ml_io_alloc;
+ callback->tvmrt_io_free = cn10k_ml_io_free;
+ callback->tvmrt_malloc = cn10k_ml_malloc;
+ callback->tvmrt_free = cn10k_ml_free;
+ callback->tvmrt_quantize = tvmrt_ml_io_quantize;
+ callback->tvmrt_dequantize = tvmrt_ml_io_dequantize;
+ callback->tvmrt_inference = cn10k_ml_inference_sync;
+ } else {
+ callback = NULL;
+ }
+
+ /* Initialize model in TVM runtime */
+ if (model->tvmrt.graph_module != NULL) {
+ ret = -EBUSY;
+ plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ snprintf(path, sizeof(path), "%s_%d_%u", ML_MODEL_SHMFD_NAME, getpid(), model->model_id);
+ model->tvmrt.fd = memfd_create(path, 0);
+ if (model->tvmrt.fd < 0) {
+ ret = (errno == 0) ? -EIO : -errno;
+ plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ if (write(model->tvmrt.fd, model->tvmrt.so.buffer, model->tvmrt.so.size) !=
+ (ssize_t)model->tvmrt.so.size) {
+ ret = (errno == 0) ? -EIO : -errno;
+ plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ if (lseek(model->tvmrt.fd, 0, SEEK_SET) < 0) {
+ ret = (errno == 0) ? -EIO : -errno;
+ plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ snprintf(path, sizeof(path), ML_MODEL_SHMFD_PATH, getpid(), model->tvmrt.fd);
+ ret = TVMModLoadFromFile(path, "so", &module_so);
+ if (ret != 0) {
+ plt_err("TVM runtime: Model load failed, model_id = %u, ret = %d, msg = %s",
+ model->model_id, ret, TVMGetLastError());
+ goto error;
+ }
+
+ /* Set device info */
+ device.device_type = kDLCPU;
+ device.device_id = 0;
+
+ if (callback != NULL) {
+ ret = tvmrt_ml_tvm_func_get(model, module_so, "register_cb", ®ister_cb_fn);
+ if (ret != 0)
+ goto error;
+
+ arg_values[0].v_handle = callback;
+ arg_types[0] = kTVMOpaqueHandle;
+ arg_values[1].v_handle = cnxk_mldev;
+ arg_types[1] = kTVMOpaqueHandle;
+ arg_values[2].v_int64 = model->model_id;
+ arg_types[2] = kDLInt;
+
+ ret = tvmrt_ml_tvm_func_call(model, register_cb_fn, "register_cb", arg_values,
+ arg_types, 3, &ret_value, &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto error;
+ }
+
+ ret = TVMFuncGetGlobal("tvm.graph_executor.create", &create_fn);
+ if (ret != 0) {
+ plt_err("TVM runtime: Model load failed, model_id = %u, ret = %d, msg = %s",
+ model->model_id, ret, TVMGetLastError());
+ goto error;
+ }
+
+ arg_values[0].v_str = (const char *)model->tvmrt.json.buffer;
+ arg_types[0] = kTVMStr;
+ arg_values[1].v_handle = module_so;
+ arg_types[1] = kTVMModuleHandle;
+ arg_values[2].v_int64 = device.device_type;
+ arg_types[2] = kDLInt;
+ arg_values[3].v_int64 = device.device_id;
+ arg_types[3] = kDLInt;
+
+ ret = tvmrt_ml_tvm_func_call(model, create_fn, "tvm.graph_executor.create", arg_values,
+ arg_types, 4, &ret_value, &ret_type, kTVMModuleHandle);
+ if (ret != 0)
+ goto error;
+ model->tvmrt.graph_module = ret_value.v_handle;
+
+ ret = tvmrt_ml_tvm_func_get(model, model->tvmrt.graph_module, "load_params",
+ &model->tvmrt.load_params);
+ if (ret != 0)
+ goto error;
+ ret = tvmrt_ml_tvm_func_get(model, model->tvmrt.graph_module, "set_input_zero_copy",
+ &model->tvmrt.set_input_zero_copy);
+ if (ret != 0)
+ goto error;
+ ret = tvmrt_ml_tvm_func_get(model, model->tvmrt.graph_module, "set_output_zero_copy",
+ &model->tvmrt.set_output_zero_copy);
+ if (ret != 0)
+ goto error;
+ ret = tvmrt_ml_tvm_func_get(model, model->tvmrt.graph_module, "run", &model->tvmrt.run);
+ if (ret != 0)
+ goto error;
+
+ tvmrt_ml_tvm_func_free(®ister_cb_fn);
+ tvmrt_ml_tvm_mod_free(&module_so);
+
+ /* Load model parameters into TVM runtime */
+ tvm_params.data = (const char *)model->tvmrt.params.buffer;
+ tvm_params.size = model->tvmrt.params.size;
+ tvm_arg_values[0].v_handle = &tvm_params;
+ tvm_arg_types[0] = kTVMBytes;
+
+ ret = tvmrt_ml_tvm_func_call(model, model->tvmrt.load_params, "load_params", tvm_arg_values,
+ tvm_arg_types, 1, &ret_value, &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto error;
+
+ /* Update model I/O data */
+ tvmrt_ml_model_io_info_set(model);
+
+ /* Set model info */
+ tvmrt_ml_model_info_set(cnxk_mldev, model);
+
+ /* Update model xstats name */
+ cnxk_ml_xstats_model_name_update(cnxk_mldev, model->model_id);
+
+ model->tvmrt.burst_xstats =
+ RTE_PTR_ADD(model->tvmrt.params.buffer,
+ RTE_ALIGN_CEIL(model->tvmrt.params.size, RTE_CACHE_LINE_MIN_SIZE));
+
+ for (int qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) {
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_latency_tot = 0;
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_latency = 0;
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_latency_min = UINT64_MAX;
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_latency_max = 0;
+ model->tvmrt.burst_xstats[qp_id].tvm_rt_reset_count = 0;
+ model->tvmrt.burst_xstats[qp_id].dequeued_count = 0;
+ }
+
+ /* Set model specific fast path functions */
+ if (model->subtype == ML_CNXK_MODEL_SUBTYPE_TVM_MRVL) {
+ model->enqueue_single = cn10k_ml_enqueue_single;
+ model->result_update = cn10k_ml_result_update;
+ model->set_error_code = cn10k_ml_set_error_code;
+ model->set_poll_addr = cn10k_ml_set_poll_addr;
+ model->op_error_get = cn10k_ml_op_error_get;
+ } else {
+ model->enqueue_single = tvmrt_ml_enqueue_single;
+ model->result_update = tvmrt_ml_result_update;
+ model->set_error_code = tvmrt_ml_set_error_code;
+ model->set_poll_addr = tvmrt_ml_set_poll_addr;
+ model->op_error_get = tvmrt_ml_op_error_get;
+ }
+
+ return 0;
+
+error:
+ tvmrt_ml_tvm_func_free(®ister_cb_fn);
+ if (model != NULL) {
+ tvmrt_ml_tvm_func_free(&model->tvmrt.load_params);
+ tvmrt_ml_tvm_func_free(&model->tvmrt.set_input_zero_copy);
+ tvmrt_ml_tvm_func_free(&model->tvmrt.set_output_zero_copy);
+ tvmrt_ml_tvm_func_free(&model->tvmrt.run);
+ tvmrt_ml_tvm_mod_free(&model->tvmrt.graph_module);
+ if (model->tvmrt.fd >= 0)
+ close(model->tvmrt.fd);
+ memset(&model->tvmrt, 0, sizeof(model->tvmrt));
+ model->tvmrt.fd = -1;
+ }
+ tvmrt_ml_tvm_mod_free(&module_so);
+ plt_memzone_free(mz);
+
+ return ret;
+}
+
+int
+tvmrt_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
+{
+ char str[RTE_MEMZONE_NAMESIZE];
+ const struct plt_memzone *mz;
+
+ RTE_SET_USED(cnxk_mldev);
+
+ /* Unload model from TVM runtime */
+ if (model->model_id >= cnxk_mldev->mldev->data->nb_models)
+ return -EINVAL;
+
+ if (model->tvmrt.graph_module == NULL)
+ return -EINVAL;
+
+ tvmrt_ml_tvm_func_free(&model->tvmrt.load_params);
+ tvmrt_ml_tvm_func_free(&model->tvmrt.set_input_zero_copy);
+ tvmrt_ml_tvm_func_free(&model->tvmrt.set_output_zero_copy);
+ tvmrt_ml_tvm_func_free(&model->tvmrt.run);
+ tvmrt_ml_tvm_mod_free(&model->tvmrt.graph_module);
+ if (model->tvmrt.fd >= 0)
+ close(model->tvmrt.fd);
+ memset(&model->tvmrt, 0, sizeof(model->tvmrt));
+ model->tvmrt.fd = -1;
+
+ snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", TVMRT_ML_MODEL_MEMZONE_NAME, model->model_id);
+ mz = plt_memzone_lookup(str);
+ if (mz == NULL) {
+ plt_err("Memzone lookup failed for TVM model: model_id = %u, mz = %s",
+ model->model_id, str);
+ return -EINVAL;
+ }
+
+ return plt_memzone_free(mz);
+}
+
+int
+tvmrt_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
+{
+ struct cnxk_ml_layer *layer;
+
+ uint16_t layer_id = 0;
+ int ret = 0;
+
+next_layer:
+ layer = &model->layer[layer_id];
+ if (layer->type == ML_CNXK_LAYER_TYPE_MRVL) {
+ ret = cn10k_ml_layer_start(cnxk_mldev, model->model_id, layer->name);
+ if (ret != 0) {
+ plt_err("Layer start failed, model_id = %u, layer_name = %s, error = %d",
+ model->model_id, layer->name, ret);
+ return ret;
+ }
+ }
+ layer_id++;
+
+ if (layer_id < model->nb_layers)
+ goto next_layer;
+
+ return 0;
+}
+
+int
+tvmrt_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
+{
+ struct cnxk_ml_layer *layer;
+
+ uint16_t layer_id = 0;
+ int ret = 0;
+
+next_layer:
+ layer = &model->layer[layer_id];
+ if (layer->type == ML_CNXK_LAYER_TYPE_MRVL) {
+ ret = cn10k_ml_layer_stop(cnxk_mldev, model->model_id, layer->name);
+ if (ret != 0) {
+ plt_err("Layer stop failed, model_id = %u, layer_name = %s, error = %d",
+ model->model_id, layer->name, ret);
+ return ret;
+ }
+ }
+ layer_id++;
+
+ if (layer_id < model->nb_layers)
+ goto next_layer;
+
+ return 0;
+}
+
+int
+tvmrt_ml_io_quantize(void *device, uint16_t model_id, const char *layer_name,
+ const DLTensor **deq_tensor, void *qbuffer)
+{
+ struct cnxk_ml_io_info *info = NULL;
+ struct cnxk_ml_dev *cnxk_mldev;
+ struct cnxk_ml_model *model;
+ uint16_t layer_id = 0;
+ uint8_t *lcl_dbuffer;
+ uint8_t *lcl_qbuffer;
+ uint32_t i;
+ int ret;
+
+#ifdef CNXK_ML_DEV_DEBUG
+ if ((device == NULL) || (deq_tensor == NULL) || (qbuffer == NULL))
+ return -EINVAL;
+#endif
+
+ cnxk_mldev = (struct cnxk_ml_dev *)device;
+
+ model = cnxk_mldev->mldev->data->models[model_id];
+#ifdef CNXK_ML_DEV_DEBUG
+ if (model == NULL) {
+ plt_err("Invalid model_id = %u", model_id);
+ return -EINVAL;
+ }
+#endif
+
+ /* Get layer id */
+ for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
+ if (strcmp(model->layer[layer_id].name, layer_name) == 0)
+ break;
+ }
+
+#ifdef CNXK_ML_DEV_DEBUG
+ if (layer_id == model->nb_layers) {
+ plt_err("Invalid layer name: %s", layer_name);
+ return -EINVAL;
+ }
+
+ if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) {
+ plt_err("Invalid layer name / type: %s", layer_name);
+ return -EINVAL;
+ }
+#endif
+
+ info = &model->layer[layer_id].info;
+ lcl_qbuffer = (uint8_t *)qbuffer;
+
+ for (i = 0; i < info->nb_inputs; i++) {
+ lcl_dbuffer = PLT_PTR_ADD(deq_tensor[i]->data, deq_tensor[i]->byte_offset);
+
+ ret = cnxk_ml_io_quantize_single(&info->input[i], lcl_dbuffer, lcl_qbuffer);
+ if (ret < 0)
+ return ret;
+
+ lcl_qbuffer += info->input[i].sz_q;
+ }
+
+ return 0;
+}
+
+int
+tvmrt_ml_io_dequantize(void *device, uint16_t model_id, const char *layer_name, void *qbuffer,
+ const DLTensor **deq_tensor)
+{
+ struct cnxk_ml_io_info *info = NULL;
+ struct cnxk_ml_dev *cnxk_mldev;
+ struct cnxk_ml_model *model;
+ uint16_t layer_id = 0;
+ uint8_t *lcl_dbuffer;
+ uint8_t *lcl_qbuffer;
+ uint32_t i;
+ int ret;
+
+#ifdef CNXK_ML_DEV_DEBUG
+ if ((device == NULL) || (deq_tensor == NULL) || (qbuffer == NULL))
+ return -EINVAL;
+#endif
+
+ cnxk_mldev = (struct cnxk_ml_dev *)device;
+
+ model = cnxk_mldev->mldev->data->models[model_id];
+#ifdef CNXK_ML_DEV_DEBUG
+ if (model == NULL) {
+ plt_err("Invalid model_id = %u", model_id);
+ return -EINVAL;
+ }
+#endif
+
+ for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
+ if (strcmp(model->layer[layer_id].name, layer_name) == 0)
+ break;
+ }
+
+#ifdef CNXK_ML_DEV_DEBUG
+ if (layer_id == model->nb_layers) {
+ plt_err("Invalid layer name: %s", layer_name);
+ return -EINVAL;
+ }
+
+ if (model->layer[layer_id].type != ML_CNXK_LAYER_TYPE_MRVL) {
+ plt_err("Invalid layer name / type: %s", layer_name);
+ return -EINVAL;
+ }
+#endif
+
+ info = &model->layer[layer_id].info;
+ lcl_qbuffer = (uint8_t *)qbuffer;
+
+ for (i = 0; i < info->nb_outputs; i++) {
+ lcl_dbuffer = PLT_PTR_ADD(deq_tensor[i]->data, deq_tensor[i]->byte_offset);
+
+ ret = cnxk_ml_io_dequantize_single(&info->output[i], lcl_qbuffer, lcl_dbuffer);
+ if (ret < 0)
+ return ret;
+
+ lcl_qbuffer += info->output[i].sz_q;
+ }
+
+ return 0;
+}
+
+static int
+tvmrt_ml_model_run(struct cnxk_ml_model *model, struct rte_ml_op *op, struct cnxk_ml_req *req)
+{
+ uint8_t i;
+ struct tvmrt_ml_result *run_result;
+ TVMValue arg_values[2] = {0};
+ int arg_types[2] = {0};
+ TVMValue ret_value = {0};
+ int ret_type = kTVMNullptr;
+ int ret = 0;
+
+ rte_memcpy(req->tvmrt_req.input_tensor, model->tvmrt.input_tensor,
+ model->tvmrt.info.nb_inputs * sizeof(DLTensor));
+ for (i = 0; i < model->tvmrt.info.nb_inputs; i++) {
+ req->tvmrt_req.input_tensor[i].data = op->input[i]->addr;
+ req->tvmrt_req.input_tensor[i].byte_offset = 0;
+ }
+
+ rte_memcpy(req->tvmrt_req.output_tensor, model->tvmrt.output_tensor,
+ model->tvmrt.info.nb_outputs * sizeof(DLTensor));
+ for (i = 0; i < model->tvmrt.info.nb_outputs; i++) {
+ req->tvmrt_req.output_tensor[i].data = op->output[i]->addr;
+ req->tvmrt_req.output_tensor[i].byte_offset = 0;
+ }
+
+ run_result = &req->tvmrt_req.result;
+ run_result->stats.start_ns = rte_get_tsc_cycles();
+ run_result->error_code = 0;
+
+ for (i = 0; i < model->tvmrt.info.nb_inputs; i++) {
+ arg_values[0].v_int64 = i;
+ arg_types[0] = kDLInt;
+ arg_values[1].v_handle = &req->tvmrt_req.input_tensor[i];
+ arg_types[1] = kTVMDLTensorHandle;
+ ret = tvmrt_ml_tvm_func_call(model, model->tvmrt.set_input_zero_copy,
+ "set_input_zero_copy", arg_values, arg_types, 2,
+ &ret_value, &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto out;
+ }
+
+ for (i = 0; i < model->tvmrt.info.nb_outputs; i++) {
+ arg_values[0].v_int64 = i;
+ arg_types[0] = kDLInt;
+ arg_values[1].v_handle = &req->tvmrt_req.output_tensor[i];
+ arg_types[1] = kTVMDLTensorHandle;
+ ret = tvmrt_ml_tvm_func_call(model, model->tvmrt.set_output_zero_copy,
+ "set_output_zero_copy", arg_values, arg_types, 2,
+ &ret_value, &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto out;
+ }
+
+ ret = tvmrt_ml_tvm_func_call(model, model->tvmrt.run, "run", NULL, NULL, 0, &ret_value,
+ &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto out;
+
+out:
+ run_result->stats.end_ns = rte_get_tsc_cycles();
+ req->tvmrt_req.status = 0x1;
+
+ plt_write64(ML_CNXK_POLL_JOB_FINISH, req->status);
+
+ if (ret != 0)
+ run_result->error_code = -EIO;
+
+ return 0;
+}
+
+__rte_hot void
+tvmrt_ml_set_error_code(struct cnxk_ml_req *req, uint64_t etype, uint64_t stype)
+{
+ RTE_SET_USED(stype);
+
+ req->tvmrt_req.result.error_code = etype;
+}
+
+__rte_hot int
+tvmrt_ml_op_error_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op,
+ struct rte_ml_op_error *error)
+{
+ RTE_SET_USED(cnxk_mldev);
+ RTE_SET_USED(op);
+ RTE_SET_USED(error);
+
+ return 0;
+}
+
+__rte_hot bool
+tvmrt_ml_enqueue_single(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op, uint16_t layer_id,
+ struct cnxk_ml_qp *qp, uint64_t head)
+{
+ struct cnxk_ml_model *model;
+ struct cnxk_ml_queue *queue;
+ struct cnxk_ml_req *req;
+
+ RTE_SET_USED(layer_id);
+
+ queue = &qp->queue;
+ req = &queue->reqs[head];
+ model = cnxk_mldev->mldev->data->models[op->model_id];
+
+ model->set_poll_addr(req);
+ memset(&req->tvmrt_req.result, 0, sizeof(struct tvmrt_ml_result));
+ req->tvmrt_req.result.error_code = 0x0;
+ req->tvmrt_req.result.user_ptr = op->user_ptr;
+
+ cnxk_ml_set_poll_ptr(req);
+ tvmrt_ml_model_run(model, op, req);
+ req->timeout = plt_tsc_cycles() + queue->wait_cycles;
+ req->op = op;
+
+ return true;
+}
+
+__rte_hot void
+tvmrt_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request)
+{
+ struct tvmrt_ml_model_xstats *xstats;
+ struct tvmrt_ml_result *result;
+ struct cnxk_ml_model *model;
+ struct cnxk_ml_req *req;
+ uint64_t tvm_rt_latency;
+ struct cnxk_ml_qp *qp;
+ struct rte_ml_op *op;
+
+ req = (struct cnxk_ml_req *)request;
+ result = &req->tvmrt_req.result;
+ op = req->op;
+ qp = cnxk_mldev->mldev->data->queue_pairs[qp_id];
+ op->impl_opaque = result->error_code;
+
+ if (likely(result->error_code == 0)) {
+ qp->stats.dequeued_count++;
+ op->status = RTE_ML_OP_STATUS_SUCCESS;
+
+ model = cnxk_mldev->mldev->data->models[op->model_id];
+ xstats = &model->tvmrt.burst_xstats[qp_id];
+
+ if (unlikely(xstats->dequeued_count == xstats->tvm_rt_reset_count)) {
+ xstats->tvm_rt_latency_min = UINT64_MAX;
+ xstats->tvm_rt_latency_max = 0;
+ }
+ tvm_rt_latency = result->stats.end_ns - result->stats.start_ns;
+ xstats->tvm_rt_latency = tvm_rt_latency;
+ xstats->tvm_rt_latency_tot += tvm_rt_latency;
+ xstats->tvm_rt_latency_min = RTE_MIN(xstats->tvm_rt_latency_min, tvm_rt_latency);
+ xstats->tvm_rt_latency_max = RTE_MAX(xstats->tvm_rt_latency_max, tvm_rt_latency);
+ xstats->dequeued_count++;
+ } else {
+ qp->stats.dequeue_err_count++;
+ op->status = RTE_ML_OP_STATUS_ERROR;
+ }
+}
diff --git a/drivers/ml/cnxk/tvmrt_ml_ops.h b/drivers/ml/cnxk/tvmrt_ml_ops.h
new file mode 100644
index 00000000000..f76f2ea000d
--- /dev/null
+++ b/drivers/ml/cnxk/tvmrt_ml_ops.h
@@ -0,0 +1,79 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2026 Marvell.
+ */
+
+#ifndef _TVMRT_ML_OPS_H_
+#define _TVMRT_ML_OPS_H_
+
+#include <dlpack/dlpack.h>
+
+#include <rte_mldev.h>
+
+#include "cnxk_ml_xstats.h"
+#include "tvmrt_ml_model.h"
+
+struct cnxk_ml_dev;
+struct cnxk_ml_model;
+struct cnxk_ml_layer;
+struct cnxk_ml_qp;
+struct cnxk_ml_req;
+
+/* Inference stats */
+struct tvmrt_ml_stats {
+ /* Start ns */
+ uint64_t start_ns;
+
+ /* Start ns */
+ uint64_t end_ns;
+};
+
+/* Result structure */
+struct tvmrt_ml_result {
+ /* Job error code */
+ uint64_t error_code;
+
+ /* Inference stats */
+ struct tvmrt_ml_stats stats;
+
+ /* User context pointer */
+ void *user_ptr;
+};
+
+/* TVMRT specific request */
+struct tvmrt_ml_req {
+ /* Input tensors */
+ DLTensor input_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT];
+
+ /* Output tensors */
+ DLTensor output_tensor[ML_CNXK_MODEL_MAX_INPUT_OUTPUT];
+
+ /* Status field for poll mode requests */
+ volatile uint64_t status;
+
+ /* Result */
+ struct tvmrt_ml_result result;
+};
+
+int tvmrt_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
+ struct cnxk_ml_model *model);
+int tvmrt_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
+int tvmrt_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
+int tvmrt_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
+int tvmrt_ml_io_quantize(void *device, uint16_t model_id, const char *layer_name,
+ const DLTensor **deq_tensor, void *qbuffer);
+int tvmrt_ml_io_dequantize(void *device, uint16_t model_id, const char *layer_name, void *qbuffer,
+ const DLTensor **deq_tensor);
+
+__rte_hot bool tvmrt_ml_enqueue_single(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op,
+ uint16_t layer_id, struct cnxk_ml_qp *qp, uint64_t head);
+__rte_hot int tvmrt_ml_op_error_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_op *op,
+ struct rte_ml_op_error *error);
+__rte_hot void tvmrt_ml_result_update(struct cnxk_ml_dev *cnxk_mldev, int qp_id, void *request);
+__rte_hot void tvmrt_ml_set_error_code(struct cnxk_ml_req *req, uint64_t etype, uint64_t stype);
+
+void tvmrt_ml_model_xstat_name_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
+ uint16_t stat_id, uint16_t entry, char *suffix);
+uint64_t tvmrt_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
+ enum cnxk_ml_xstats_type type);
+
+#endif /* _TVMRT_ML_OPS_H_ */
diff --git a/drivers/ml/cnxk/tvmrt_ml_stubs.c b/drivers/ml/cnxk/tvmrt_ml_stubs.c
new file mode 100644
index 00000000000..0b4e53b4327
--- /dev/null
+++ b/drivers/ml/cnxk/tvmrt_ml_stubs.c
@@ -0,0 +1,106 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2026 Marvell.
+ */
+
+#include <rte_mldev.h>
+
+#include "tvmrt_ml_stubs.h"
+
+#include "cnxk_ml_dev.h"
+#include "cnxk_ml_model.h"
+#include "cnxk_ml_xstats.h"
+
+enum cnxk_ml_model_type
+tvmrt_ml_model_type_get(struct rte_ml_model_params *params)
+{
+ RTE_SET_USED(params);
+
+ return ML_CNXK_MODEL_TYPE_UNKNOWN;
+}
+
+int
+tvmrt_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name, uint16_t *layer_id)
+{
+ RTE_SET_USED(model);
+ RTE_SET_USED(layer_name);
+ RTE_SET_USED(layer_id);
+
+ return -EINVAL;
+}
+
+struct cnxk_ml_io_info *
+tvmrt_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id)
+{
+ RTE_SET_USED(model);
+ RTE_SET_USED(layer_id);
+
+ return NULL;
+}
+
+void
+tvmrt_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp)
+{
+ RTE_SET_USED(cnxk_mldev);
+ RTE_SET_USED(layer);
+ RTE_SET_USED(fp);
+}
+
+void
+tvmrt_ml_model_xstat_name_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
+ uint16_t stat_id, uint16_t entry, char *suffix)
+{
+ RTE_SET_USED(cnxk_mldev);
+ RTE_SET_USED(model);
+ RTE_SET_USED(stat_id);
+ RTE_SET_USED(entry);
+ RTE_SET_USED(suffix);
+}
+
+uint64_t
+tvmrt_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
+ enum cnxk_ml_xstats_type type)
+{
+ RTE_SET_USED(cnxk_mldev);
+ RTE_SET_USED(model);
+ RTE_SET_USED(type);
+
+ return 0;
+}
+
+int
+tvmrt_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
+ struct cnxk_ml_model *model)
+{
+ RTE_SET_USED(cnxk_mldev);
+ RTE_SET_USED(params);
+ RTE_SET_USED(model);
+
+ return -EINVAL;
+}
+
+int
+tvmrt_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
+{
+ RTE_SET_USED(cnxk_mldev);
+ RTE_SET_USED(model);
+
+ return -EINVAL;
+}
+
+int
+tvmrt_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
+{
+ RTE_SET_USED(cnxk_mldev);
+ RTE_SET_USED(model);
+
+ return -EINVAL;
+}
+
+int
+tvmrt_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
+{
+ RTE_SET_USED(cnxk_mldev);
+ RTE_SET_USED(model);
+
+ return -EINVAL;
+}
diff --git a/drivers/ml/cnxk/tvmrt_ml_stubs.h b/drivers/ml/cnxk/tvmrt_ml_stubs.h
new file mode 100644
index 00000000000..e0260d36514
--- /dev/null
+++ b/drivers/ml/cnxk/tvmrt_ml_stubs.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2026 Marvell.
+ */
+
+#ifndef _TVMRT_ML_STUBS_H_
+#define _TVMRT_ML_STUBS_H_
+
+#include <rte_mldev.h>
+
+#include "cnxk_ml_xstats.h"
+
+struct cnxk_ml_dev;
+struct cnxk_ml_model;
+struct cnxk_ml_layer;
+
+enum cnxk_ml_model_type tvmrt_ml_model_type_get(struct rte_ml_model_params *params);
+int tvmrt_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
+ struct cnxk_ml_model *model);
+int tvmrt_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
+int tvmrt_ml_model_start(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
+int tvmrt_ml_model_stop(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
+
+int tvmrt_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name,
+ uint16_t *layer_id);
+struct cnxk_ml_io_info *tvmrt_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id);
+void tvmrt_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp);
+void tvmrt_ml_model_xstat_name_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
+ uint16_t stat_id, uint16_t entry, char *suffix);
+uint64_t tvmrt_ml_model_xstat_get(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model,
+ enum cnxk_ml_xstats_type type);
+
+#endif /* _TVMRT_ML_STUBS_H_ */
--
2.34.1
^ permalink raw reply related
* [PATCH 1/3] ml/cnxk: update driver to use TVM C runtime
From: Srikanth Yalavarthi @ 2026-06-09 4:42 UTC (permalink / raw)
To: Srikanth Yalavarthi; +Cc: dev, jerinj, sshankarnara, ptakkar, aprabhu
Updated MVTVM layer of the driver to call TVM C runtime
API functions, instead on the TVMDP wrapper library APIs.
Drop all dependencies on TVMDP wrapper library.
Signed-off-by: Srikanth Yalavarthi <syalavarthi@marvell.com>
---
doc/guides/mldevs/cnxk.rst | 35 +-
drivers/ml/cnxk/cnxk_ml_io.h | 20 +-
drivers/ml/cnxk/cnxk_ml_ops.c | 12 +-
drivers/ml/cnxk/meson.build | 14 +-
drivers/ml/cnxk/mvtvm_ml_dev.h | 3 -
drivers/ml/cnxk/mvtvm_ml_model.c | 638 ++++++++++++++++++++++++++++---
drivers/ml/cnxk/mvtvm_ml_model.h | 81 +++-
drivers/ml/cnxk/mvtvm_ml_ops.c | 440 ++++++++++++++-------
drivers/ml/cnxk/mvtvm_ml_ops.h | 5 +-
drivers/ml/cnxk/mvtvm_ml_stubs.c | 17 -
drivers/ml/cnxk/mvtvm_ml_stubs.h | 2 -
11 files changed, 985 insertions(+), 282 deletions(-)
diff --git a/doc/guides/mldevs/cnxk.rst b/doc/guides/mldevs/cnxk.rst
index fc1bcd9cdb4..17e58d81c0b 100644
--- a/doc/guides/mldevs/cnxk.rst
+++ b/doc/guides/mldevs/cnxk.rst
@@ -111,13 +111,13 @@ on CPU cores or hardware accelerators.
.. note::
- DPDK CNXK ML driver requires TVM version 0.10.0
+ DPDK CNXK ML driver requires TVM version legacy-v0.19.post branch.
.. code-block:: console
git clone https://github.com/apache/tvm.git
cd tvm
- git checkout v0.11.0 -b v0.11.0
+ git checkout legacy-v0.19.post -b legacy-v0.19.post
git submodule update --init
cmake -S ./ -B build \
-DCMAKE_INSTALL_PREFIX=<install_prefix> \
@@ -135,37 +135,6 @@ When cross-compiling, more options must be provided to CMake:
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER \
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY
-TVMDP
-~~~~~
-
- Marvell's `TVM Dataplane Library <https://github.com/MarvellEmbeddedProcessors/tvmdp>`_
- works as an interface between TVM runtime and DPDK drivers.
- TVMDP library provides a simplified C interface
- for TVM's runtime based on C++.
-
-.. note::
-
- TVMDP library is dependent on TVM, dlpack, jansson and dmlc-core libraries.
-
-.. code-block:: console
-
- git clone https://github.com/MarvellEmbeddedProcessors/tvmdp.git
- cd tvmdp
- git checkout main
- cmake -S ./ -B build \
- -DCMAKE_INSTALL_PREFIX=<install_prefix> \
- -DBUILD_SHARED_LIBS=ON
- make -C build
- make -C build install
-
-When cross-compiling, more options must be provided to CMake:
-
-.. code-block:: console
-
- -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \
- -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \
- -DCMAKE_FIND_ROOT_PATH=<install_prefix>
-
libarchive
~~~~~~~~~~
diff --git a/drivers/ml/cnxk/cnxk_ml_io.h b/drivers/ml/cnxk/cnxk_ml_io.h
index 83329c237aa..17f5b4619f0 100644
--- a/drivers/ml/cnxk/cnxk_ml_io.h
+++ b/drivers/ml/cnxk/cnxk_ml_io.h
@@ -6,7 +6,7 @@
#define _CNXK_ML_IO_H_
#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
-#include <tvmdp.h>
+#include <dlpack/dlpack.h>
#endif
#include <rte_mldev.h>
@@ -16,7 +16,7 @@
/* Maximum number of layers per model */
#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
-#define ML_CNXK_MODEL_MAX_LAYERS TVMDP_MODEL_LAYERS_MAX
+#define ML_CNXK_MODEL_MAX_LAYERS 128
#else
#define ML_CNXK_MODEL_MAX_LAYERS 1
#endif
@@ -41,7 +41,7 @@ struct cnxk_ml_io {
/* Number of dimensions in shape */
uint32_t nb_dims;
- /* Shape of input */
+ /* Shape */
uint32_t shape[ML_CNXK_MODEL_MAX_DIMS];
/* Number of elements */
@@ -58,6 +58,20 @@ struct cnxk_ml_io {
/* Zero point */
int64_t zero_point;
+
+#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
+ /* Shape - int64_t */
+ int64_t shape_i64[ML_CNXK_MODEL_MAX_DIMS];
+
+ /* Data type */
+ DLDataType datatype;
+
+ /* Model data type */
+ DLDataType model_datatype;
+
+ /* Device */
+ DLDevice device;
+#endif
};
/* Model / Layer IO structure */
diff --git a/drivers/ml/cnxk/cnxk_ml_ops.c b/drivers/ml/cnxk/cnxk_ml_ops.c
index 9958945670a..938982c7556 100644
--- a/drivers/ml/cnxk/cnxk_ml_ops.c
+++ b/drivers/ml/cnxk/cnxk_ml_ops.c
@@ -634,10 +634,6 @@ cnxk_ml_dev_configure(struct rte_ml_dev *dev, const struct rte_ml_dev_config *co
}
}
- ret = mvtvm_ml_dev_configure(cnxk_mldev, conf);
- if (ret != 0)
- goto error;
-
/* Set device capabilities */
if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI)
cnxk_mldev->max_nb_layers =
@@ -705,9 +701,6 @@ cnxk_ml_dev_close(struct rte_ml_dev *dev)
/* Un-initialize xstats */
cnxk_ml_xstats_uninit(cnxk_mldev);
- if (mvtvm_ml_dev_close(cnxk_mldev) != 0)
- plt_err("Failed to close MVTVM ML Device");
-
if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_PCI) {
if (cn10k_ml_dev_close(cnxk_mldev) != 0)
plt_err("Failed to close CN10K ML Device");
@@ -1234,7 +1227,7 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u
model->layer[layer_id].glow.ocm_map.scratch_pages);
#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
} else {
- for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) {
+ for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
if (model->layer[layer_id].type == ML_CNXK_LAYER_TYPE_MRVL) {
total_wb_pages = total_wb_pages +
model->layer[layer_id].glow.ocm_map.wb_pages;
@@ -1256,8 +1249,7 @@ cnxk_ml_model_load(struct rte_ml_dev *dev, struct rte_ml_model_params *params, u
model->layer[layer_id].glow.ocm_map.scratch_pages);
#ifdef RTE_MLDEV_CNXK_ENABLE_MVTVM
} else {
- for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers;
- layer_id++) {
+ for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
if (model->layer[layer_id].type == ML_CNXK_LAYER_TYPE_MRVL) {
plt_ml_dbg(
"layer_id = %u: wb_pages = %u, scratch_pages = %u",
diff --git a/drivers/ml/cnxk/meson.build b/drivers/ml/cnxk/meson.build
index ef7e21d7476..5f078bd4fd7 100644
--- a/drivers/ml/cnxk/meson.build
+++ b/drivers/ml/cnxk/meson.build
@@ -19,30 +19,24 @@ if not jansson_dep.found()
enable_mvtvm = false
endif
-dlpack_dep = dependency('dlpack', method: 'cmake', required: false, cmake_args: 'CONFIG')
+dlpack_dep = dependency('dlpack', method: 'cmake', required: false)
if not dlpack_dep.found()
message('drivers/ml/cnxk: dlpack not found')
enable_mvtvm = false
endif
-dmlc_dep = dependency('dmlc', method: 'cmake', required: false, cmake_args: 'CONFIG')
+dmlc_dep = dependency('dmlc', method: 'cmake', required: false)
if not dmlc_dep.found()
message('drivers/ml/cnxk: dmlc not found')
enable_mvtvm = false
endif
-tvm_dep = dependency('tvm', method: 'cmake', required: false, cmake_args: 'CONFIG', modules : ['tvm::tvm_runtime'])
+tvm_dep = dependency('tvm', method: 'cmake', required: false, modules : ['tvm::tvm_runtime'])
if not tvm_dep.found()
message('drivers/ml/cnxk: tvm_runtime not found')
enable_mvtvm = false
endif
-tvmdp_dep = dependency('tvmdp', method: 'pkg-config', required: false)
-if not tvmdp_dep.found()
- message('drivers/ml/cnxk: tvmdp not found')
- enable_mvtvm = false
-endif
-
sources = files(
'cn10k_ml_dev.c',
'cn10k_ml_ops.c',
@@ -68,10 +62,10 @@ sources += files(
)
ext_deps += jansson_dep
+ext_deps += libarchive
ext_deps += dlpack_dep
ext_deps += dmlc_dep
ext_deps += tvm_dep
-ext_deps += tvmdp_dep
stdcpp_dep = cc.find_library('stdc++', required: true)
if not cc.links(min_c_code, dependencies: stdcpp_dep)
error('broken dependency, "libstdc++"')
diff --git a/drivers/ml/cnxk/mvtvm_ml_dev.h b/drivers/ml/cnxk/mvtvm_ml_dev.h
index 6922c193372..05e30f094cd 100644
--- a/drivers/ml/cnxk/mvtvm_ml_dev.h
+++ b/drivers/ml/cnxk/mvtvm_ml_dev.h
@@ -19,9 +19,6 @@ extern struct rte_ml_dev_ops cnxk_ml_ops;
/* Maximum number of descriptors per queue-pair */
#define ML_MVTVM_MAX_DESC_PER_QP 1024
-/* Maximum number of inputs / outputs per model */
-#define ML_MVTVM_MAX_INPUT_OUTPUT 32
-
/* Maximum number of segments for IO data */
#define ML_MVTVM_MAX_SEGMENTS 1
diff --git a/drivers/ml/cnxk/mvtvm_ml_model.c b/drivers/ml/cnxk/mvtvm_ml_model.c
index 4c7aa906da0..e1946607629 100644
--- a/drivers/ml/cnxk/mvtvm_ml_model.c
+++ b/drivers/ml/cnxk/mvtvm_ml_model.c
@@ -2,6 +2,12 @@
* Copyright (c) 2023 Marvell.
*/
+#include <errno.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
#include <archive.h>
#include <archive_entry.h>
@@ -23,6 +29,7 @@ enum cnxk_ml_model_type
mvtvm_ml_model_type_get(struct rte_ml_model_params *params)
{
bool object_found[ML_MVTVM_MODEL_OBJECT_MAX] = {false, false, false};
+ enum cnxk_ml_model_type model_type;
struct archive_entry *entry;
struct archive *a;
uint8_t i;
@@ -33,9 +40,10 @@ mvtvm_ml_model_type_get(struct rte_ml_model_params *params)
archive_read_support_filter_all(a);
archive_read_support_format_all(a);
+ model_type = ML_CNXK_MODEL_TYPE_UNKNOWN;
ret = archive_read_open_memory(a, params->addr, params->size);
if (ret != ARCHIVE_OK)
- return ML_CNXK_MODEL_TYPE_UNKNOWN;
+ goto cleanup;
/* Parse buffer for available objects */
while (archive_read_next_header(a, &entry) == ARCHIVE_OK) {
@@ -51,11 +59,18 @@ mvtvm_ml_model_type_get(struct rte_ml_model_params *params)
for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) {
if (!object_found[i]) {
plt_err("Object %s not found in archive!", mvtvm_object_list[i]);
- return ML_CNXK_MODEL_TYPE_INVALID;
+ model_type = ML_CNXK_MODEL_TYPE_INVALID;
+ goto cleanup;
}
}
- return ML_CNXK_MODEL_TYPE_TVM;
+ model_type = ML_CNXK_MODEL_TYPE_TVM;
+
+cleanup:
+ archive_read_close(a);
+ archive_read_free(a);
+
+ return model_type;
}
int
@@ -67,14 +82,18 @@ mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_mo
uint8_t i;
int ret;
+ memset(object, 0, ML_MVTVM_MODEL_OBJECT_MAX * sizeof(*object));
+
/* Open archive */
a = archive_read_new();
archive_read_support_filter_all(a);
archive_read_support_format_all(a);
ret = archive_read_open_memory(a, params->addr, params->size);
- if (ret != ARCHIVE_OK)
- return archive_errno(a);
+ if (ret != ARCHIVE_OK) {
+ ret = archive_errno(a);
+ goto cleanup;
+ }
/* Read archive */
while (archive_read_next_header(a, &entry) == ARCHIVE_OK) {
@@ -84,11 +103,16 @@ mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_mo
memcpy(object[i].name, mvtvm_object_list[i], RTE_ML_STR_MAX);
object[i].size = archive_entry_size(entry);
object[i].buffer = rte_malloc(NULL, object[i].size, 0);
+ if (object[i].buffer == NULL) {
+ ret = -ENOMEM;
+ goto error;
+ }
if (archive_read_data(a, object[i].buffer, object[i].size) !=
object[i].size) {
plt_err("Failed to read object from model archive: %s",
object[i].name);
+ ret = -EINVAL;
goto error;
}
object_found[i] = true;
@@ -101,17 +125,24 @@ mvtvm_ml_model_blob_parse(struct rte_ml_model_params *params, struct mvtvm_ml_mo
for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) {
if (!object_found[i]) {
plt_err("Object %s not found in archive!", mvtvm_object_list[i]);
+ ret = -EINVAL;
goto error;
}
}
- return 0;
+ ret = 0;
+ goto cleanup;
error:
for (i = 0; i < ML_MVTVM_MODEL_OBJECT_MAX; i++) {
- rte_free(object[i].buffer);
+ if (object[i].buffer != NULL)
+ rte_free(object[i].buffer);
}
- return -EINVAL;
+cleanup:
+ archive_read_close(a);
+ archive_read_free(a);
+
+ return ret;
}
int
@@ -119,12 +150,12 @@ mvtvm_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name,
{
uint16_t i;
- for (i = 0; i < model->mvtvm.metadata.model.nb_layers; i++) {
+ for (i = 0; i < model->nb_layers; i++) {
if (strcmp(model->layer[i].name, layer_name) == 0)
break;
}
- if (i == model->mvtvm.metadata.model.nb_layers) {
+ if (i == model->nb_layers) {
plt_err("Invalid layer name: %s", layer_name);
return -EINVAL;
}
@@ -139,6 +170,506 @@ mvtvm_ml_model_get_layer_id(struct cnxk_ml_model *model, const char *layer_name,
return 0;
}
+static void
+mvtvm_ml_param_names_free(struct mvtvm_ml_param_names *param_names)
+{
+ size_t i;
+
+ for (i = 0; i < param_names->count; i++)
+ free(param_names->name[i]);
+
+ free(param_names->name);
+ param_names->name = NULL;
+ param_names->count = 0;
+}
+
+static int
+mvtvm_ml_blob_read_u64(const uint8_t *blob, size_t blob_size, size_t *offset, uint64_t *value)
+{
+ if (*offset + sizeof(*value) > blob_size)
+ return -EINVAL;
+
+ memcpy(value, blob + *offset, sizeof(*value));
+ *offset += sizeof(*value);
+
+ return 0;
+}
+
+static int
+mvtvm_ml_param_names_parse(const uint8_t *blob, size_t blob_size,
+ struct mvtvm_ml_param_names *param_names)
+{
+ uint64_t magic = 0;
+ uint64_t reserved = 0;
+ uint64_t count = 0;
+ size_t offset = 0;
+ int ret;
+ size_t i;
+
+ memset(param_names, 0, sizeof(*param_names));
+
+ ret = mvtvm_ml_blob_read_u64(blob, blob_size, &offset, &magic);
+ if (ret != 0)
+ return ret;
+
+ if (magic != TVM_NDARRAY_LIST_MAGIC)
+ return -EINVAL;
+
+ ret = mvtvm_ml_blob_read_u64(blob, blob_size, &offset, &reserved);
+ if (ret != 0)
+ return ret;
+
+ ret = mvtvm_ml_blob_read_u64(blob, blob_size, &offset, &count);
+ if (ret != 0)
+ return ret;
+
+ if (count > SIZE_MAX / sizeof(*param_names->name))
+ return -EINVAL;
+
+ param_names->name = calloc(count, sizeof(*param_names->name));
+ if (count != 0 && param_names->name == NULL)
+ return -ENOMEM;
+
+ param_names->count = count;
+ for (i = 0; i < count; i++) {
+ uint64_t name_len = 0;
+
+ ret = mvtvm_ml_blob_read_u64(blob, blob_size, &offset, &name_len);
+ if (ret != 0)
+ goto error;
+
+ if (name_len == SIZE_MAX) {
+ ret = -EINVAL;
+ goto error;
+ }
+
+ if (offset + name_len > blob_size) {
+ ret = -EINVAL;
+ goto error;
+ }
+
+ param_names->name[i] = calloc(name_len + 1, 1);
+ if (param_names->name[i] == NULL) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ memcpy(param_names->name[i], blob + offset, name_len);
+ offset += name_len;
+ }
+
+ return 0;
+
+error:
+ mvtvm_ml_param_names_free(param_names);
+ return ret;
+}
+
+static int
+mvtvm_ml_dtype_parse(const char *dtype_str, DLDataType *dtype)
+{
+ const char *lanes_str;
+ char base[32] = {0};
+ long bits;
+ long lanes = 1;
+
+ if (dtype_str == NULL || dtype == NULL)
+ return -EINVAL;
+
+ lanes_str = strchr(dtype_str, 'x');
+ if (lanes_str != NULL) {
+ size_t base_len = lanes_str - dtype_str;
+
+ if (base_len >= sizeof(base))
+ return -EINVAL;
+ memcpy(base, dtype_str, base_len);
+ lanes = strtol(lanes_str + 1, NULL, 10);
+ if (lanes <= 0 || lanes > UINT16_MAX)
+ return -EINVAL;
+ } else {
+ snprintf(base, sizeof(base), "%s", dtype_str);
+ }
+
+ memset(dtype, 0, sizeof(*dtype));
+ dtype->lanes = (uint16_t)lanes;
+
+ if (strncmp(base, "int", 3) == 0) {
+ bits = strtol(base + 3, NULL, 10);
+ dtype->code = kDLInt;
+ } else if (strncmp(base, "uint", 4) == 0) {
+ bits = strtol(base + 4, NULL, 10);
+ dtype->code = kDLUInt;
+ } else if (strncmp(base, "float", 5) == 0) {
+ bits = strtol(base + 5, NULL, 10);
+ dtype->code = kDLFloat;
+ } else if (strncmp(base, "bfloat", 6) == 0) {
+ bits = strtol(base + 6, NULL, 10);
+ dtype->code = kDLBfloat;
+ } else if (strcmp(base, "bool") == 0) {
+ bits = 1;
+ dtype->code = kDLUInt;
+ } else {
+ return -EINVAL;
+ }
+
+ if (bits <= 0 || bits > UINT8_MAX)
+ return -EINVAL;
+
+ dtype->bits = (uint8_t)bits;
+
+ return 0;
+}
+
+static int
+mvtvm_ml_model_io_set(struct cnxk_ml_io *io, const char *name, json_t *shape, const char *dtype_str,
+ DLDevice device)
+{
+ size_t i;
+ int ret;
+
+ memset(io, 0, sizeof(*io));
+ if (name != NULL)
+ snprintf(io->name, sizeof(io->name), "%s", name);
+
+ if (!json_is_array(shape))
+ return -EINVAL;
+
+ io->nb_dims = json_array_size(shape);
+ if (io->nb_dims > ML_CNXK_MODEL_MAX_DIMS)
+ return -ENOTSUP;
+
+ for (i = 0; i < (size_t)io->nb_dims; i++) {
+ json_t *dim = json_array_get(shape, i);
+
+ if (!json_is_integer(dim))
+ return -EINVAL;
+ io->shape_i64[i] = json_integer_value(dim);
+ }
+
+ ret = mvtvm_ml_dtype_parse(dtype_str, &io->datatype);
+ if (ret != 0)
+ return ret;
+
+ io->model_datatype = io->datatype;
+ io->scale = 1.0f;
+ io->device = device;
+
+ return 0;
+}
+
+static int
+mvtvm_ml_json_graph_get_arrays(json_t *json_parsed, json_t **nodes, json_t **arg_nodes,
+ json_t **heads, json_t **node_row_ptr, json_t **shape_values,
+ json_t **dtype_values)
+{
+ json_t *attrs;
+ json_t *shape_attr;
+ json_t *dtype_attr;
+
+ *nodes = json_object_get(json_parsed, "nodes");
+ *arg_nodes = json_object_get(json_parsed, "arg_nodes");
+ *heads = json_object_get(json_parsed, "heads");
+ *node_row_ptr = json_object_get(json_parsed, "node_row_ptr");
+ attrs = json_object_get(json_parsed, "attrs");
+
+ if (!json_is_array(*nodes) || !json_is_array(*arg_nodes) || !json_is_array(*heads) ||
+ !json_is_array(*node_row_ptr) || !json_is_object(attrs))
+ return -EINVAL;
+
+ if (json_array_size(*node_row_ptr) != json_array_size(*nodes) + 1)
+ return -EINVAL;
+
+ shape_attr = json_object_get(attrs, "shape");
+ dtype_attr = json_object_get(attrs, "dltype");
+ if (!json_is_array(shape_attr) || json_array_size(shape_attr) < 2 ||
+ !json_is_array(dtype_attr) || json_array_size(dtype_attr) < 2)
+ return -EINVAL;
+
+ *shape_values = json_array_get(shape_attr, 1);
+ *dtype_values = json_array_get(dtype_attr, 1);
+
+ if (!json_is_array(*shape_values) || !json_is_array(*dtype_values))
+ return -EINVAL;
+
+ return 0;
+}
+
+int
+mvtvm_ml_model_json_parse(struct cnxk_ml_model *model)
+{
+ struct mvtvm_ml_param_names param_names;
+ json_error_t json_error;
+ json_t *json_parsed;
+ json_t *json_nodes;
+ json_t *json_arg_nodes;
+ json_t *json_heads;
+ json_t *json_node_row_ptr;
+ json_t *json_shape_values;
+ json_t *json_dtype_values;
+ uint16_t nb_mrvl_layers;
+ uint16_t nb_llvm_layers;
+ DLDevice device;
+ int ret;
+ size_t i;
+ size_t j;
+
+ /* Parse param names from params buffer */
+ ret = mvtvm_ml_param_names_parse(model->mvtvm.params.buffer, model->mvtvm.params.size,
+ ¶m_names);
+ if (ret != 0)
+ return ret;
+
+ /* Load JSON graph (single load for both stages) */
+ json_parsed = json_loadb((const char *)model->mvtvm.json.buffer, model->mvtvm.json.size, 0,
+ &json_error);
+ if (json_parsed == NULL) {
+ mvtvm_ml_param_names_free(¶m_names);
+ plt_err("TVM runtime: Failed to parse JSON graph, model_id = %u", model->model_id);
+ return -EINVAL;
+ }
+
+ /* Parse nodes to extract layer info */
+ json_nodes = json_object_get(json_parsed, "nodes");
+ if (!json_is_array(json_nodes)) {
+ ret = -EINVAL;
+ plt_err("TVM runtime: Failed to parse JSON nodes, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ model->nb_layers = 0;
+ nb_mrvl_layers = 0;
+ nb_llvm_layers = 0;
+ for (i = 0; i < json_array_size(json_nodes); i++) {
+ json_t *node = json_array_get(json_nodes, i);
+ json_t *op = json_object_get(node, "op");
+ json_t *name;
+ json_t *attrs;
+ json_t *compiler;
+
+ if (!json_is_string(op) || strcmp(json_string_value(op), "tvm_op") != 0)
+ continue;
+
+ if (model->nb_layers >= ML_CNXK_MODEL_MAX_LAYERS) {
+ ret = -ENOTSUP;
+ plt_err("TVM runtime: Number of layers exceeds maximum (%u), model_id = %u, error = %d",
+ ML_CNXK_MODEL_MAX_LAYERS, model->model_id, ret);
+ goto error;
+ }
+
+ name = json_object_get(node, "name");
+ if (!json_is_string(name)) {
+ ret = -EINVAL;
+ plt_err("TVM runtime: Failed to parse JSON node name, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ if (json_string_value(name) != NULL)
+ snprintf(model->layer[model->nb_layers].name,
+ sizeof(model->layer[model->nb_layers].name), "%s",
+ json_string_value(name));
+ else
+ snprintf(model->layer[model->nb_layers].name,
+ sizeof(model->layer[model->nb_layers].name), "tvm_layer_%u",
+ model->nb_layers);
+
+ attrs = json_object_get(node, "attrs");
+ compiler = json_is_object(attrs) ? json_object_get(attrs, "Compiler") : NULL;
+ if (json_is_string(compiler)) {
+ if (strcmp(json_string_value(compiler), "mrvl") == 0 ||
+ strcmp(json_string_value(compiler), "MRVL") == 0) {
+ model->layer[model->nb_layers].type = ML_CNXK_LAYER_TYPE_MRVL;
+ nb_mrvl_layers++;
+ } else {
+ model->layer[model->nb_layers].type = ML_CNXK_LAYER_TYPE_UNKNOWN;
+ }
+ } else {
+ model->layer[model->nb_layers].type = ML_CNXK_LAYER_TYPE_LLVM;
+ nb_llvm_layers++;
+ }
+ model->nb_layers++;
+ }
+
+ /* Set model fields */
+ snprintf(model->name, sizeof(model->name), "tvm_model_%u", model->model_id);
+ model->batch_size = 1;
+
+ /* Validate layer counts */
+ if ((nb_llvm_layers == 0) && (nb_mrvl_layers == 0)) {
+ plt_err("Invalid model, nb_llvm_layers = %u, nb_mrvl_layers = %u", nb_llvm_layers,
+ nb_mrvl_layers);
+ ret = -EINVAL;
+ goto error;
+ }
+
+ if (nb_llvm_layers + nb_mrvl_layers != model->nb_layers) {
+ plt_err("Invalid model, nb_llvm_layers = %u, nb_mrvl_layers = %u, nb_layers = %u",
+ nb_llvm_layers, nb_mrvl_layers, model->nb_layers);
+ ret = -EINVAL;
+ goto error;
+ }
+
+ /* Set model subtype */
+ if ((nb_llvm_layers == 0) && (nb_mrvl_layers == 1))
+ model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_MRVL;
+ else if ((nb_llvm_layers > 0) && (nb_mrvl_layers == 0))
+ model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_LLVM;
+ else
+ model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_HYBRID;
+
+ /* Parse I/O info from the same JSON graph */
+ device.device_type = kDLCPU;
+ device.device_id = 0;
+
+ ret = mvtvm_ml_json_graph_get_arrays(json_parsed, &json_nodes, &json_arg_nodes, &json_heads,
+ &json_node_row_ptr, &json_shape_values,
+ &json_dtype_values);
+ if (ret == 0) {
+ model->mvtvm.info.nb_inputs = 0;
+ model->mvtvm.info.nb_outputs = 0;
+
+ for (i = 0; i < json_array_size(json_arg_nodes); i++) {
+ json_t *arg_node_idx = json_array_get(json_arg_nodes, i);
+ json_t *node;
+ json_t *shape;
+ json_t *dtype;
+ json_t *name;
+ json_int_t node_id;
+
+ if (!json_is_integer(arg_node_idx)) {
+ ret = -EINVAL;
+ break;
+ }
+
+ node_id = json_integer_value(arg_node_idx);
+ node = json_array_get(json_nodes, node_id);
+ shape = json_array_get(json_shape_values, node_id);
+ dtype = json_array_get(json_dtype_values, node_id);
+ name = json_object_get(node, "name");
+ if (!json_is_object(node) || !json_is_array(shape) ||
+ !json_is_string(dtype) || !json_is_string(name)) {
+ ret = -EINVAL;
+ break;
+ }
+
+ for (j = 0; j < param_names.count; j++) {
+ if (strcmp(param_names.name[j], json_string_value(name)) == 0)
+ break;
+ }
+
+ if (j < param_names.count)
+ continue;
+
+ if (model->mvtvm.info.nb_inputs >= ML_CNXK_MODEL_MAX_INPUT_OUTPUT) {
+ ret = -ENOTSUP;
+ break;
+ }
+
+ ret = mvtvm_ml_model_io_set(
+ &model->mvtvm.info.input[model->mvtvm.info.nb_inputs],
+ json_string_value(name), shape, json_string_value(dtype), device);
+ if (ret != 0)
+ break;
+
+ model->mvtvm.info.nb_inputs++;
+ }
+
+ for (i = 0; ret == 0 && i < json_array_size(json_heads); i++) {
+ json_t *head = json_array_get(json_heads, i);
+ json_t *node;
+ json_t *shape;
+ json_t *dtype;
+ json_t *name;
+ json_t *node_id_json;
+ json_t *output_idx_json;
+ json_t *entry_base_json;
+ json_t *entry_limit_json;
+ json_int_t node_id;
+ json_int_t output_idx;
+ json_int_t entry_base;
+ json_int_t entry_limit;
+ size_t entry_id;
+
+ if (!json_is_array(head) || json_array_size(head) < 2) {
+ ret = -EINVAL;
+ break;
+ }
+
+ node_id_json = json_array_get(head, 0);
+ output_idx_json = json_array_get(head, 1);
+ if (!json_is_integer(node_id_json) || !json_is_integer(output_idx_json)) {
+ ret = -EINVAL;
+ break;
+ }
+
+ node_id = json_integer_value(node_id_json);
+ output_idx = json_integer_value(output_idx_json);
+ if (node_id < 0 || output_idx < 0) {
+ ret = -EINVAL;
+ break;
+ }
+
+ node = json_array_get(json_nodes, (size_t)node_id);
+ entry_base_json = json_array_get(json_node_row_ptr, (size_t)node_id);
+ entry_limit_json = json_array_get(json_node_row_ptr, (size_t)node_id + 1);
+ if (!json_is_object(node) || !json_is_integer(entry_base_json) ||
+ !json_is_integer(entry_limit_json)) {
+ ret = -EINVAL;
+ break;
+ }
+
+ entry_base = json_integer_value(entry_base_json);
+ entry_limit = json_integer_value(entry_limit_json);
+ if (entry_base < 0 || entry_limit < entry_base) {
+ ret = -EINVAL;
+ break;
+ }
+
+ if (output_idx >= (entry_limit - entry_base)) {
+ ret = -EINVAL;
+ break;
+ }
+
+ entry_id = (size_t)entry_base + (size_t)output_idx;
+ shape = json_array_get(json_shape_values, entry_id);
+ dtype = json_array_get(json_dtype_values, entry_id);
+ name = json_object_get(node, "name");
+ if (!json_is_array(shape) || !json_is_string(dtype) ||
+ !json_is_string(name)) {
+ ret = -EINVAL;
+ break;
+ }
+
+ if (model->mvtvm.info.nb_outputs >= ML_CNXK_MODEL_MAX_INPUT_OUTPUT) {
+ ret = -ENOTSUP;
+ break;
+ }
+
+ ret = mvtvm_ml_model_io_set(
+ &model->mvtvm.info.output[model->mvtvm.info.nb_outputs],
+ json_string_value(name), shape, json_string_value(dtype), device);
+ if (ret != 0)
+ break;
+
+ model->mvtvm.info.nb_outputs++;
+ }
+ }
+
+ if (ret != 0)
+ plt_err("TVM runtime: Failed to get metadata, model_id = %u, error = %d",
+ model->model_id, ret);
+
+ json_decref(json_parsed);
+ mvtvm_ml_param_names_free(¶m_names);
+ return ret;
+
+error:
+ json_decref(json_parsed);
+ mvtvm_ml_param_names_free(¶m_names);
+ return ret;
+}
+
static enum rte_ml_io_type
mvtvm_ml_io_type_map(DLDataType dltype)
{
@@ -185,32 +716,27 @@ mvtvm_ml_io_type_map(DLDataType dltype)
void
mvtvm_ml_model_io_info_set(struct cnxk_ml_model *model)
{
- struct tvmdp_model_metadata *metadata;
int32_t i;
- int32_t j;
+ uint32_t j;
if (model->subtype == ML_CNXK_MODEL_SUBTYPE_TVM_MRVL)
goto tvm_mrvl_model;
- metadata = &model->mvtvm.metadata;
-
/* Inputs, set for layer_id = 0 */
- model->mvtvm.info.nb_inputs = metadata->model.num_input;
model->mvtvm.info.total_input_sz_d = 0;
model->mvtvm.info.total_input_sz_q = 0;
- for (i = 0; i < metadata->model.num_input; i++) {
- rte_strscpy(model->mvtvm.info.input[i].name, metadata->input[i].name,
- TVMDP_NAME_STRLEN);
+ for (i = 0; i < model->mvtvm.info.nb_inputs; i++) {
model->mvtvm.info.input[i].dtype =
- mvtvm_ml_io_type_map(metadata->input[i].datatype);
+ mvtvm_ml_io_type_map(model->mvtvm.info.input[i].datatype);
model->mvtvm.info.input[i].qtype =
- mvtvm_ml_io_type_map(metadata->input[i].model_datatype);
- model->mvtvm.info.input[i].nb_dims = metadata->input[i].ndim;
+ mvtvm_ml_io_type_map(model->mvtvm.info.input[i].model_datatype);
model->mvtvm.info.input[i].nb_elements = 1;
- for (j = 0; j < metadata->input[i].ndim; j++) {
- model->mvtvm.info.input[i].shape[j] = metadata->input[i].shape[j];
- model->mvtvm.info.input[i].nb_elements *= metadata->input[i].shape[j];
+ for (j = 0; j < model->mvtvm.info.input[i].nb_dims; j++) {
+ model->mvtvm.info.input[i].shape[j] =
+ PLT_U32_CAST(model->mvtvm.info.input[i].shape_i64[j]);
+ model->mvtvm.info.input[i].nb_elements *=
+ model->mvtvm.info.input[i].shape[j];
}
model->mvtvm.info.input[i].sz_d =
@@ -219,15 +745,14 @@ mvtvm_ml_model_io_info_set(struct cnxk_ml_model *model)
model->mvtvm.info.input[i].sz_q =
model->mvtvm.info.input[i].nb_elements *
rte_ml_io_type_size_get(model->mvtvm.info.input[i].qtype);
- model->mvtvm.info.input[i].scale = metadata->input[i].scale;
model->mvtvm.info.total_input_sz_d += model->mvtvm.info.input[i].sz_d;
model->mvtvm.info.total_input_sz_q += model->mvtvm.info.input[i].sz_q;
- model->mvtvm.input_tensor[i].device = metadata->input[i].device;
- model->mvtvm.input_tensor[i].ndim = metadata->input[i].ndim;
- model->mvtvm.input_tensor[i].dtype = metadata->input[i].datatype;
- model->mvtvm.input_tensor[i].shape = metadata->input[i].shape;
+ model->mvtvm.input_tensor[i].device = model->mvtvm.info.input[i].device;
+ model->mvtvm.input_tensor[i].ndim = model->mvtvm.info.input[i].nb_dims;
+ model->mvtvm.input_tensor[i].dtype = model->mvtvm.info.input[i].datatype;
+ model->mvtvm.input_tensor[i].shape = model->mvtvm.info.input[i].shape_i64;
model->mvtvm.input_tensor[i].strides = NULL;
model->mvtvm.input_tensor[i].byte_offset = 0;
@@ -236,22 +761,20 @@ mvtvm_ml_model_io_info_set(struct cnxk_ml_model *model)
}
/* Outputs, set for nb_layers - 1 */
- model->mvtvm.info.nb_outputs = metadata->model.num_output;
model->mvtvm.info.total_output_sz_d = 0;
model->mvtvm.info.total_output_sz_q = 0;
- for (i = 0; i < metadata->model.num_output; i++) {
- rte_strscpy(model->mvtvm.info.output[i].name, metadata->output[i].name,
- TVMDP_NAME_STRLEN);
+ for (i = 0; i < model->mvtvm.info.nb_outputs; i++) {
model->mvtvm.info.output[i].dtype =
- mvtvm_ml_io_type_map(metadata->output[i].datatype);
+ mvtvm_ml_io_type_map(model->mvtvm.info.output[i].datatype);
model->mvtvm.info.output[i].qtype =
- mvtvm_ml_io_type_map(metadata->output[i].model_datatype);
- model->mvtvm.info.output[i].nb_dims = metadata->output[i].ndim;
+ mvtvm_ml_io_type_map(model->mvtvm.info.output[i].model_datatype);
model->mvtvm.info.output[i].nb_elements = 1;
- for (j = 0; j < metadata->output[i].ndim; j++) {
- model->mvtvm.info.output[i].shape[j] = metadata->output[i].shape[j];
- model->mvtvm.info.output[i].nb_elements *= metadata->output[i].shape[j];
+ for (j = 0; j < model->mvtvm.info.output[i].nb_dims; j++) {
+ model->mvtvm.info.output[i].shape[j] =
+ PLT_U32_CAST(model->mvtvm.info.output[i].shape_i64[j]);
+ model->mvtvm.info.output[i].nb_elements *=
+ model->mvtvm.info.output[i].shape[j];
}
model->mvtvm.info.output[i].sz_d =
@@ -260,15 +783,14 @@ mvtvm_ml_model_io_info_set(struct cnxk_ml_model *model)
model->mvtvm.info.output[i].sz_q =
model->mvtvm.info.output[i].nb_elements *
rte_ml_io_type_size_get(model->mvtvm.info.output[i].qtype);
- model->mvtvm.info.output[i].scale = metadata->output[i].scale;
model->mvtvm.info.total_output_sz_d += model->mvtvm.info.output[i].sz_d;
model->mvtvm.info.total_output_sz_q += model->mvtvm.info.output[i].sz_q;
- model->mvtvm.output_tensor[i].device = metadata->output[i].device;
- model->mvtvm.output_tensor[i].ndim = metadata->output[i].ndim;
- model->mvtvm.output_tensor[i].dtype = metadata->output[i].datatype;
- model->mvtvm.output_tensor[i].shape = metadata->output[i].shape;
+ model->mvtvm.output_tensor[i].device = model->mvtvm.info.output[i].device;
+ model->mvtvm.output_tensor[i].ndim = model->mvtvm.info.output[i].nb_dims;
+ model->mvtvm.output_tensor[i].dtype = model->mvtvm.info.output[i].datatype;
+ model->mvtvm.output_tensor[i].shape = model->mvtvm.info.output[i].shape_i64;
model->mvtvm.output_tensor[i].strides = NULL;
model->mvtvm.output_tensor[i].byte_offset = 0;
@@ -293,7 +815,6 @@ mvtvm_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id)
void
mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model)
{
- struct tvmdp_model_metadata *metadata;
struct rte_ml_model_info *info;
struct rte_ml_io_info *output;
struct rte_ml_io_info *input;
@@ -309,26 +830,23 @@ mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mo
if (model->subtype == ML_CNXK_MODEL_SUBTYPE_TVM_MRVL)
goto tvm_mrvl_model;
- metadata = &model->mvtvm.metadata;
- rte_memcpy(info->name, metadata->model.name, TVMDP_NAME_STRLEN);
- snprintf(info->version, RTE_ML_STR_MAX, "%u.%u.%u.%u", metadata->model.version[0],
- metadata->model.version[1], metadata->model.version[2],
- metadata->model.version[3]);
+ rte_memcpy(info->name, model->name, RTE_ML_STR_MAX);
+ snprintf(info->version, RTE_ML_STR_MAX, "%u.%u.%u.%u", 0, 0, 0, 0);
info->model_id = model->model_id;
info->device_id = cnxk_mldev->mldev->data->dev_id;
info->io_layout = RTE_ML_IO_LAYOUT_SPLIT;
info->min_batches = model->batch_size;
info->max_batches = model->batch_size;
- info->nb_inputs = metadata->model.num_input;
+ info->nb_inputs = model->mvtvm.info.nb_inputs;
info->input_info = input;
- info->nb_outputs = metadata->model.num_output;
+ info->nb_outputs = model->mvtvm.info.nb_outputs;
info->output_info = output;
info->wb_size = 0;
/* Set input info */
for (i = 0; i < info->nb_inputs; i++) {
- rte_memcpy(input[i].name, metadata->input[i].name, MRVL_ML_INPUT_NAME_LEN);
- input[i].nb_dims = metadata->input[i].ndim;
+ rte_memcpy(input[i].name, model->mvtvm.info.input[i].name, MRVL_ML_INPUT_NAME_LEN);
+ input[i].nb_dims = model->mvtvm.info.input[i].nb_dims;
input[i].shape = &model->mvtvm.info.input[i].shape[0];
input[i].type = model->mvtvm.info.input[i].qtype;
input[i].nb_elements = model->mvtvm.info.input[i].nb_elements;
@@ -340,15 +858,16 @@ mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mo
/* Set output info */
for (i = 0; i < info->nb_outputs; i++) {
- rte_memcpy(output[i].name, metadata->output[i].name, MRVL_ML_OUTPUT_NAME_LEN);
- output[i].nb_dims = metadata->output[i].ndim;
+ rte_memcpy(output[i].name, model->mvtvm.info.output[i].name,
+ MRVL_ML_OUTPUT_NAME_LEN);
+ output[i].nb_dims = model->mvtvm.info.output[i].nb_dims;
output[i].shape = &model->mvtvm.info.output[i].shape[0];
output[i].type = model->mvtvm.info.output[i].qtype;
output[i].nb_elements = model->mvtvm.info.output[i].nb_elements;
output[i].size = model->mvtvm.info.output[i].nb_elements *
rte_ml_io_type_size_get(model->mvtvm.info.output[i].qtype);
- input[i].scale = model->mvtvm.info.output[i].scale;
- input[i].zero_point = 0;
+ output[i].scale = model->mvtvm.info.output[i].scale;
+ output[i].zero_point = 0;
}
return;
@@ -357,8 +876,7 @@ mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mo
cn10k_ml_model_info_set(cnxk_mldev, model, &model->mvtvm.info,
&model->layer[0].glow.metadata);
- metadata = &model->mvtvm.metadata;
- strlcpy(info->name, metadata->model.name, TVMDP_NAME_STRLEN);
+ strlcpy(info->name, model->name, RTE_ML_STR_MAX);
info->io_layout = RTE_ML_IO_LAYOUT_PACKED;
}
diff --git a/drivers/ml/cnxk/mvtvm_ml_model.h b/drivers/ml/cnxk/mvtvm_ml_model.h
index 7ffce380945..1151b0a39ed 100644
--- a/drivers/ml/cnxk/mvtvm_ml_model.h
+++ b/drivers/ml/cnxk/mvtvm_ml_model.h
@@ -5,7 +5,21 @@
#ifndef _MVTVM_ML_MODEL_H_
#define _MVTVM_ML_MODEL_H_
-#include <tvmdp.h>
+#include <dlpack/dlpack.h>
+#include <jansson.h>
+#include <rte_common.h>
+
+#if defined(RTE_CC_GCC) || defined(RTE_CC_CLANG)
+__rte_diagnostic_push
+RTE_PRAGMA(GCC diagnostic ignored "-Wstrict-prototypes")
+#include <tvm/runtime/c_runtime_api.h>
+__rte_diagnostic_pop
+#else
+#include <tvm/runtime/c_runtime_api.h>
+#endif
+
+#include <stddef.h>
+#include <stdint.h>
#include <rte_mldev.h>
@@ -18,19 +32,56 @@ struct cnxk_ml_layer;
/* Maximum number of objects per model */
#define ML_MVTVM_MODEL_OBJECT_MAX 3
-/* Objects list */
-extern char mvtvm_object_list[ML_MVTVM_MODEL_OBJECT_MAX][RTE_ML_STR_MAX];
+/* Magic number for TVM parameter blobs. */
+#define TVM_NDARRAY_LIST_MAGIC 0xF7E58D4F05049CB7ULL
-/* Model object structure */
+/* TVM parameter names structure */
+struct mvtvm_ml_param_names {
+ char **name;
+ size_t count;
+};
+
+/* TVM object / artifact info structure */
struct mvtvm_ml_model_object {
/* Name */
char name[RTE_ML_STR_MAX];
- /* Temporary buffer */
+ /* Buffer address */
uint8_t *buffer;
/* Buffer size */
int64_t size;
+
+ /* Offset */
+ uint32_t offset;
+};
+
+/* Glow model callback functions */
+typedef int (*tvmrt_glow_layer_load_cb)(void *device, uint16_t model_id, const char *layer_name,
+ uint8_t *buffer, size_t size, uint16_t *index);
+typedef int (*tvmrt_glow_layer_unload_cb)(void *device, uint16_t model_id, const char *layer_name);
+typedef int (*tvmrt_io_alloc_cb)(void *device, uint16_t model_id, const char *layer_name,
+ uint64_t **input_qbuffer, uint64_t **output_qbuffer);
+typedef int (*tvmrt_io_free_cb)(void *device, uint16_t model_id, const char *layer_name);
+typedef int (*tvmrt_malloc_cb)(const char *name, size_t size, uint32_t align, void **addr);
+typedef int (*tvmrt_free_cb)(const char *name);
+typedef int (*tvmrt_quantize_cb)(void *device, uint16_t model_id, const char *layer_name,
+ const DLTensor **deq_tensor, void *qbuffer);
+typedef int (*tvmrt_dequantize_cb)(void *device, uint16_t model_id, const char *layer_name,
+ void *qbuffer, const DLTensor **deq_tensor);
+typedef int (*tvmrt_inference_cb)(void *device, uint16_t index, void *input, void *output,
+ uint16_t nb_batches);
+
+struct tvmrt_glow_callback {
+ tvmrt_glow_layer_load_cb tvmrt_glow_layer_load;
+ tvmrt_glow_layer_unload_cb tvmrt_glow_layer_unload;
+ tvmrt_io_alloc_cb tvmrt_io_alloc;
+ tvmrt_io_free_cb tvmrt_io_free;
+ tvmrt_malloc_cb tvmrt_malloc;
+ tvmrt_free_cb tvmrt_free;
+ tvmrt_quantize_cb tvmrt_quantize;
+ tvmrt_dequantize_cb tvmrt_dequantize;
+ tvmrt_inference_cb tvmrt_inference;
};
/* Model fast-path stats */
@@ -55,15 +106,26 @@ struct mvtvm_ml_model_xstats {
};
struct mvtvm_ml_model_data {
- /* Model metadata */
- struct tvmdp_model_metadata metadata;
-
/* Model objects */
- struct tvmdp_model_object object;
+ struct mvtvm_ml_model_object so;
+ struct mvtvm_ml_model_object json;
+ struct mvtvm_ml_model_object params;
/* TVM runtime callbacks */
struct tvmrt_glow_callback cb;
+ /* TVM Graph Module */
+ TVMModuleHandle graph_module;
+
+ /* Shared object memfd used to load the TVM module */
+ int fd;
+
+ /* TVM Function Handles */
+ TVMFunctionHandle load_params;
+ TVMFunctionHandle set_input_zero_copy;
+ TVMFunctionHandle set_output_zero_copy;
+ TVMFunctionHandle run;
+
/* Model I/O info */
struct cnxk_ml_io_info info;
@@ -86,5 +148,6 @@ void mvtvm_ml_model_io_info_set(struct cnxk_ml_model *model);
struct cnxk_ml_io_info *mvtvm_ml_model_io_info_get(struct cnxk_ml_model *model, uint16_t layer_id);
void mvtvm_ml_model_info_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *model);
void mvtvm_ml_layer_print(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_layer *layer, FILE *fp);
+int mvtvm_ml_model_json_parse(struct cnxk_ml_model *model);
#endif /* _MVTVM_ML_MODEL_H_ */
diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.c b/drivers/ml/cnxk/mvtvm_ml_ops.c
index 80b22460db9..bc47a4bbd75 100644
--- a/drivers/ml/cnxk/mvtvm_ml_ops.c
+++ b/drivers/ml/cnxk/mvtvm_ml_ops.c
@@ -2,7 +2,17 @@
* Copyright (c) 2023 Marvell.
*/
+#include <errno.h>
+#include <linux/limits.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
#include <dlpack/dlpack.h>
+#include <jansson.h>
#include <rte_common.h>
#include <rte_cycles.h>
@@ -19,6 +29,76 @@
/* ML model macros */
#define MVTVM_ML_MODEL_MEMZONE_NAME "ml_mvtvm_model_mz"
+/* Shared memory file descriptor name */
+#define ML_MODEL_SHMFD_NAME "mvtvm_shmfd"
+
+/* Shared memory file descriptor path */
+#define ML_MODEL_SHMFD_PATH "/proc/%d/fd/%d"
+
+static int
+mvtvm_ml_tvm_func_get(struct cnxk_ml_model *model, TVMModuleHandle module, const char *name,
+ TVMFunctionHandle *func)
+{
+ int ret;
+
+ ret = TVMModGetFunction(module, name, 0, func);
+ if (ret != 0) {
+ plt_err("Model load failed, model_id = %u, ret = %d, msg = %s", model->model_id,
+ ret, TVMGetLastError());
+ return ret;
+ }
+
+ if (*func == NULL) {
+ ret = -ENOENT;
+ plt_err("Model load failed, model_id = %u, function '%s' not found",
+ model->model_id, name);
+ }
+
+ return ret;
+}
+
+static int
+mvtvm_ml_tvm_func_call(struct cnxk_ml_model *model, TVMFunctionHandle func, const char *name,
+ TVMValue *values, int *types, int num_args, TVMValue *ret_val, int *ret_type,
+ int ret_type_code)
+{
+ int ret;
+
+ ret = TVMFuncCall(func, values, types, num_args, ret_val, ret_type);
+ if (ret != 0) {
+ plt_err("Error calling TVM function '%s', model_id = %u, ret = %d, msg = %s", name,
+ model->model_id, ret, TVMGetLastError());
+ return ret;
+ }
+
+ if (*ret_type != ret_type_code) {
+ ret = -EINVAL;
+ plt_err("TVM function '%s' returned unexpected type, model_id = %u, expected = %d, "
+ "actual = %d",
+ name, model->model_id, ret_type_code, *ret_type);
+ }
+
+ return ret;
+}
+
+static void
+mvtvm_ml_tvm_func_free(TVMFunctionHandle *func)
+{
+ if ((func != NULL) && (*func != NULL)) {
+ TVMFuncFree(*func);
+ *func = NULL;
+ }
+}
+
+static void
+mvtvm_ml_tvm_mod_free(TVMModuleHandle *mod)
+{
+ if ((mod != NULL) && (*mod != NULL)) {
+ TVMModFree(*mod);
+ *mod = NULL;
+ }
+}
+
__rte_hot static void
mvtvm_ml_set_poll_addr(struct cnxk_ml_req *req)
{
@@ -30,8 +110,8 @@ mvtvm_ml_model_xstat_name_set(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_mod
uint16_t stat_id, uint16_t entry, char *suffix)
{
snprintf(cnxk_mldev->xstats.entries[stat_id].map.name,
- sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s",
- model->mvtvm.metadata.model.name, model_xstats[entry].name, suffix);
+ sizeof(cnxk_mldev->xstats.entries[stat_id].map.name), "%s-%s-%s", model->name,
+ model_xstats[entry].name, suffix);
}
#define ML_AVG_FOREACH_QP_MVTVM(cnxk_mldev, model, qp_id, value, count) \
@@ -106,43 +186,13 @@ mvtvm_ml_dev_info_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_dev_info *de
dev_info->max_queue_pairs = mvtvm_mldev->max_nb_qpairs;
dev_info->max_desc = ML_MVTVM_MAX_DESC_PER_QP;
- dev_info->max_io = ML_MVTVM_MAX_INPUT_OUTPUT;
+ dev_info->max_io = ML_CNXK_MODEL_MAX_INPUT_OUTPUT;
dev_info->max_segments = ML_MVTVM_MAX_SEGMENTS;
dev_info->align_size = RTE_CACHE_LINE_SIZE;
return 0;
}
-int
-mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf)
-{
- int ret;
-
- RTE_SET_USED(conf);
-
- /* Configure TVMDP library */
- ret = tvmdp_configure(cnxk_mldev->mldev->data->nb_models, rte_get_tsc_cycles);
- if (ret != 0)
- plt_err("TVMDP configuration failed, error = %d", ret);
-
- return ret;
-}
-
-int
-mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev)
-{
- int ret;
-
- RTE_SET_USED(cnxk_mldev);
-
- /* Close TVMDP library configuration */
- ret = tvmdp_close();
- if (ret != 0)
- plt_err("TVMDP close failed, error = %d", ret);
-
- return ret;
-}
-
int
mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp)
{
@@ -159,16 +209,26 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *
struct mvtvm_ml_model_object object[ML_MVTVM_MODEL_OBJECT_MAX];
struct tvmrt_glow_callback *callback;
char str[RTE_MEMZONE_NAMESIZE];
+ char path[PATH_MAX];
const struct plt_memzone *mz;
size_t model_object_size = 0;
size_t model_xstats_size = 0;
- uint16_t nb_mrvl_layers;
- uint16_t nb_llvm_layers;
- uint8_t layer_id = 0;
uint64_t mz_size = 0;
+ TVMFunctionHandle create_fn = NULL;
+ TVMFunctionHandle register_cb_fn = NULL;
+ TVMModuleHandle module_so = NULL;
+ TVMByteArray tvm_params;
+ TVMValue ret_value = {0};
+ TVMValue arg_values[4] = {0};
+ TVMValue tvm_arg_values[1] = {0};
+ int ret_type = kTVMNullptr;
+ int arg_types[4] = {0};
+ int tvm_arg_types[1] = {0};
+ DLDevice device;
int ret;
RTE_SET_USED(cnxk_mldev);
+ model->mvtvm.fd = -1;
ret = mvtvm_ml_model_blob_parse(params, object);
if (ret != 0)
@@ -192,80 +252,39 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *
}
/* Copy mod.so */
- model->mvtvm.object.so.addr = mz->addr;
- model->mvtvm.object.so.size = object[0].size;
- rte_memcpy(model->mvtvm.object.so.name, object[0].name, TVMDP_NAME_STRLEN);
- rte_memcpy(model->mvtvm.object.so.addr, object[0].buffer, object[0].size);
+ model->mvtvm.so.buffer = mz->addr;
+ model->mvtvm.so.size = object[0].size;
+ rte_memcpy(model->mvtvm.so.name, object[0].name, RTE_ML_STR_MAX);
+ rte_memcpy(model->mvtvm.so.buffer, object[0].buffer, object[0].size);
rte_free(object[0].buffer);
/* Copy mod.json */
- model->mvtvm.object.json.addr =
- RTE_PTR_ADD(model->mvtvm.object.so.addr,
- RTE_ALIGN_CEIL(model->mvtvm.object.so.size, RTE_CACHE_LINE_MIN_SIZE));
- model->mvtvm.object.json.size = object[1].size;
- rte_memcpy(model->mvtvm.object.json.name, object[1].name, TVMDP_NAME_STRLEN);
- rte_memcpy(model->mvtvm.object.json.addr, object[1].buffer, object[1].size);
+ model->mvtvm.json.buffer =
+ RTE_PTR_ADD(model->mvtvm.so.buffer,
+ RTE_ALIGN_CEIL(model->mvtvm.so.size, RTE_CACHE_LINE_MIN_SIZE));
+ model->mvtvm.json.size = object[1].size;
+ rte_memcpy(model->mvtvm.json.name, object[1].name, RTE_ML_STR_MAX);
+ rte_memcpy(model->mvtvm.json.buffer, object[1].buffer, object[1].size);
rte_free(object[1].buffer);
/* Copy mod.params */
- model->mvtvm.object.params.addr =
- RTE_PTR_ADD(model->mvtvm.object.json.addr,
- RTE_ALIGN_CEIL(model->mvtvm.object.json.size, RTE_CACHE_LINE_MIN_SIZE));
- model->mvtvm.object.params.size = object[2].size;
- rte_memcpy(model->mvtvm.object.params.name, object[2].name, TVMDP_NAME_STRLEN);
- rte_memcpy(model->mvtvm.object.params.addr, object[2].buffer, object[2].size);
+ model->mvtvm.params.buffer =
+ RTE_PTR_ADD(model->mvtvm.json.buffer,
+ RTE_ALIGN_CEIL(model->mvtvm.json.size, RTE_CACHE_LINE_MIN_SIZE));
+ model->mvtvm.params.size = object[2].size;
+ rte_memcpy(model->mvtvm.params.name, object[2].name, RTE_ML_STR_MAX);
+ rte_memcpy(model->mvtvm.params.buffer, object[2].buffer, object[2].size);
rte_free(object[2].buffer);
- /* Get metadata - stage 1 */
- ret = tvmdp_model_metadata_get_stage1(model->mvtvm.object.json.addr,
- model->mvtvm.object.json.size,
- &model->mvtvm.metadata);
- if (ret != 0) {
- plt_err("TVMDP: Failed to parse metadata - stage 1, model_id = %u, error = %d",
- model->model_id, ret);
- goto error;
- }
-
- /* Set model fields */
- plt_strlcpy(model->name, model->mvtvm.metadata.model.name, TVMDP_NAME_STRLEN);
- model->batch_size = 1;
- model->nb_layers = model->mvtvm.metadata.model.nb_layers;
-
- /* Update layer info */
- nb_mrvl_layers = 0;
- nb_llvm_layers = 0;
- for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) {
- rte_strscpy(model->layer[layer_id].name,
- model->mvtvm.metadata.model.layer[layer_id].name, TVMDP_NAME_STRLEN);
- if (strcmp(model->mvtvm.metadata.model.layer[layer_id].type, "mrvl") == 0 ||
- strcmp(model->mvtvm.metadata.model.layer[layer_id].type, "MRVL") == 0) {
- model->layer[layer_id].type = ML_CNXK_LAYER_TYPE_MRVL;
- nb_mrvl_layers++;
- } else if (strcmp(model->mvtvm.metadata.model.layer[layer_id].type, "llvm") == 0 ||
- strcmp(model->mvtvm.metadata.model.layer[layer_id].type, "LLVM") == 0) {
- model->layer[layer_id].type = ML_CNXK_LAYER_TYPE_LLVM;
- nb_llvm_layers++;
- }
- }
-
- if ((nb_llvm_layers == 0) && (nb_mrvl_layers == 0)) {
- plt_err("Invalid model, nb_llvm_layers = %u, nb_mrvl_layers = %u", nb_llvm_layers,
- nb_mrvl_layers);
+ ret = mvtvm_ml_model_json_parse(model);
+ if (ret != 0)
goto error;
- }
-
- /* Set model subtype */
- if ((nb_llvm_layers == 0) && (nb_mrvl_layers == 1))
- model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_MRVL;
- else if ((nb_llvm_layers > 0) && (nb_mrvl_layers == 0))
- model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_LLVM;
- else
- model->subtype = ML_CNXK_MODEL_SUBTYPE_TVM_HYBRID;
if (cnxk_mldev->type == CNXK_ML_DEV_TYPE_VDEV &&
model->subtype != ML_CNXK_MODEL_SUBTYPE_TVM_LLVM) {
plt_err("Unsupported model sub-type");
- return -ENOTSUP;
+ ret = -ENOTSUP;
+ goto error;
}
/* Set callback function array */
@@ -284,23 +303,120 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *
callback = NULL;
}
- /* Initialize model in TVMDP */
- ret = tvmdp_model_load(cnxk_mldev, model->model_id, (void *)(&model->mvtvm.object),
- callback);
+ /* Initialize model in TVM runtime */
+ if (model->mvtvm.graph_module != NULL) {
+ ret = -EBUSY;
+ plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ snprintf(path, sizeof(path), "%s_%d_%u", ML_MODEL_SHMFD_NAME, getpid(), model->model_id);
+ model->mvtvm.fd = memfd_create(path, 0);
+ if (model->mvtvm.fd < 0) {
+ ret = (errno == 0) ? -EIO : -errno;
+ plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ if (write(model->mvtvm.fd, model->mvtvm.so.buffer, model->mvtvm.so.size) !=
+ (ssize_t)model->mvtvm.so.size) {
+ ret = (errno == 0) ? -EIO : -errno;
+ plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ if (lseek(model->mvtvm.fd, 0, SEEK_SET) < 0) {
+ ret = (errno == 0) ? -EIO : -errno;
+ plt_err("TVM runtime: Model load failed, model_id = %u, error = %d",
+ model->model_id, ret);
+ goto error;
+ }
+
+ snprintf(path, sizeof(path), ML_MODEL_SHMFD_PATH, getpid(), model->mvtvm.fd);
+ ret = TVMModLoadFromFile(path, "so", &module_so);
if (ret != 0) {
- plt_err("TVMDP: Model load failed, model_id = %u, error = %d", model->model_id,
- ret);
+ plt_err("TVM runtime: Model load failed, model_id = %u, ret = %d, msg = %s",
+ model->model_id, ret, TVMGetLastError());
goto error;
}
- /* Get model metadata - stage 2 */
- ret = tvmdp_model_metadata_get_stage2(model->model_id, &model->mvtvm.metadata);
+ /* Set device info */
+ device.device_type = kDLCPU;
+ device.device_id = 0;
+
+ if (callback != NULL) {
+ ret = mvtvm_ml_tvm_func_get(model, module_so, "register_cb", ®ister_cb_fn);
+ if (ret != 0)
+ goto error;
+
+ arg_values[0].v_handle = callback;
+ arg_types[0] = kTVMOpaqueHandle;
+ arg_values[1].v_handle = cnxk_mldev;
+ arg_types[1] = kTVMOpaqueHandle;
+ arg_values[2].v_int64 = model->model_id;
+ arg_types[2] = kDLInt;
+
+ ret = mvtvm_ml_tvm_func_call(model, register_cb_fn, "register_cb", arg_values,
+ arg_types, 3, &ret_value, &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto error;
+ }
+
+ ret = TVMFuncGetGlobal("tvm.graph_executor.create", &create_fn);
if (ret != 0) {
- plt_err("TVMDP: Failed to get metadata, model_id = %u, error = %d",
- model->model_id, ret);
+ plt_err("TVM runtime: Model load failed, model_id = %u, ret = %d, msg = %s",
+ model->model_id, ret, TVMGetLastError());
goto error;
}
+ arg_values[0].v_str = (const char *)model->mvtvm.json.buffer;
+ arg_types[0] = kTVMStr;
+ arg_values[1].v_handle = module_so;
+ arg_types[1] = kTVMModuleHandle;
+ arg_values[2].v_int64 = device.device_type;
+ arg_types[2] = kDLInt;
+ arg_values[3].v_int64 = device.device_id;
+ arg_types[3] = kDLInt;
+
+ ret = mvtvm_ml_tvm_func_call(model, create_fn, "tvm.graph_executor.create", arg_values,
+ arg_types, 4, &ret_value, &ret_type, kTVMModuleHandle);
+ if (ret != 0)
+ goto error;
+ model->mvtvm.graph_module = ret_value.v_handle;
+
+ ret = mvtvm_ml_tvm_func_get(model, model->mvtvm.graph_module, "load_params",
+ &model->mvtvm.load_params);
+ if (ret != 0)
+ goto error;
+ ret = mvtvm_ml_tvm_func_get(model, model->mvtvm.graph_module, "set_input_zero_copy",
+ &model->mvtvm.set_input_zero_copy);
+ if (ret != 0)
+ goto error;
+ ret = mvtvm_ml_tvm_func_get(model, model->mvtvm.graph_module, "set_output_zero_copy",
+ &model->mvtvm.set_output_zero_copy);
+ if (ret != 0)
+ goto error;
+ ret = mvtvm_ml_tvm_func_get(model, model->mvtvm.graph_module, "run", &model->mvtvm.run);
+ if (ret != 0)
+ goto error;
+
+ mvtvm_ml_tvm_func_free(®ister_cb_fn);
+ mvtvm_ml_tvm_mod_free(&module_so);
+
+ /* Load model parameters into TVM runtime */
+ tvm_params.data = (const char *)model->mvtvm.params.buffer;
+ tvm_params.size = model->mvtvm.params.size;
+ tvm_arg_values[0].v_handle = &tvm_params;
+ tvm_arg_types[0] = kTVMBytes;
+
+ ret = mvtvm_ml_tvm_func_call(model, model->mvtvm.load_params, "load_params", tvm_arg_values,
+ tvm_arg_types, 1, &ret_value, &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto error;
+
/* Update model I/O data */
mvtvm_ml_model_io_info_set(model);
@@ -310,9 +426,9 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *
/* Update model xstats name */
cnxk_ml_xstats_model_name_update(cnxk_mldev, model->model_id);
- model->mvtvm.burst_xstats = RTE_PTR_ADD(
- model->mvtvm.object.params.addr,
- RTE_ALIGN_CEIL(model->mvtvm.object.params.size, RTE_CACHE_LINE_MIN_SIZE));
+ model->mvtvm.burst_xstats =
+ RTE_PTR_ADD(model->mvtvm.params.buffer,
+ RTE_ALIGN_CEIL(model->mvtvm.params.size, RTE_CACHE_LINE_MIN_SIZE));
for (int qp_id = 0; qp_id < cnxk_mldev->mldev->data->nb_queue_pairs; qp_id++) {
model->mvtvm.burst_xstats[qp_id].tvm_rt_latency_tot = 0;
@@ -341,7 +457,20 @@ mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *
return 0;
error:
- rte_memzone_free(mz);
+ mvtvm_ml_tvm_func_free(®ister_cb_fn);
+ if (model != NULL) {
+ mvtvm_ml_tvm_func_free(&model->mvtvm.load_params);
+ mvtvm_ml_tvm_func_free(&model->mvtvm.set_input_zero_copy);
+ mvtvm_ml_tvm_func_free(&model->mvtvm.set_output_zero_copy);
+ mvtvm_ml_tvm_func_free(&model->mvtvm.run);
+ mvtvm_ml_tvm_mod_free(&model->mvtvm.graph_module);
+ if (model->mvtvm.fd >= 0)
+ close(model->mvtvm.fd);
+ memset(&model->mvtvm, 0, sizeof(model->mvtvm));
+ model->mvtvm.fd = -1;
+ }
+ mvtvm_ml_tvm_mod_free(&module_so);
+ plt_memzone_free(mz);
return ret;
}
@@ -351,20 +480,28 @@ mvtvm_ml_model_unload(struct cnxk_ml_dev *cnxk_mldev, struct cnxk_ml_model *mode
{
char str[RTE_MEMZONE_NAMESIZE];
const struct plt_memzone *mz;
- int ret;
RTE_SET_USED(cnxk_mldev);
- /* Initialize model in TVMDP */
- ret = tvmdp_model_unload(model->model_id);
- if (ret != 0) {
- plt_err("TVMDP: Model unload failed, model_id = %u, error = %d", model->model_id,
- ret);
- return ret;
- }
+ /* Unload model from TVM runtime */
+ if (model->model_id >= cnxk_mldev->mldev->data->nb_models)
+ return -EINVAL;
+
+ if (model->mvtvm.graph_module == NULL)
+ return -EINVAL;
+
+ mvtvm_ml_tvm_func_free(&model->mvtvm.load_params);
+ mvtvm_ml_tvm_func_free(&model->mvtvm.set_input_zero_copy);
+ mvtvm_ml_tvm_func_free(&model->mvtvm.set_output_zero_copy);
+ mvtvm_ml_tvm_func_free(&model->mvtvm.run);
+ mvtvm_ml_tvm_mod_free(&model->mvtvm.graph_module);
+ if (model->mvtvm.fd >= 0)
+ close(model->mvtvm.fd);
+ memset(&model->mvtvm, 0, sizeof(model->mvtvm));
+ model->mvtvm.fd = -1;
snprintf(str, RTE_MEMZONE_NAMESIZE, "%s_%u", MVTVM_ML_MODEL_MEMZONE_NAME, model->model_id);
- mz = rte_memzone_lookup(str);
+ mz = plt_memzone_lookup(str);
if (mz == NULL) {
plt_err("Memzone lookup failed for TVM model: model_id = %u, mz = %s",
model->model_id, str);
@@ -455,13 +592,13 @@ mvtvm_ml_io_quantize(void *device, uint16_t model_id, const char *layer_name,
#endif
/* Get layer id */
- for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) {
+ for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
if (strcmp(model->layer[layer_id].name, layer_name) == 0)
break;
}
#ifdef CNXK_ML_DEV_DEBUG
- if (layer_id == model->mvtvm.metadata.model.nb_layers) {
+ if (layer_id == model->nb_layers) {
plt_err("Invalid layer name: %s", layer_name);
return -EINVAL;
}
@@ -516,13 +653,13 @@ mvtvm_ml_io_dequantize(void *device, uint16_t model_id, const char *layer_name,
}
#endif
- for (layer_id = 0; layer_id < model->mvtvm.metadata.model.nb_layers; layer_id++) {
+ for (layer_id = 0; layer_id < model->nb_layers; layer_id++) {
if (strcmp(model->layer[layer_id].name, layer_name) == 0)
break;
}
#ifdef CNXK_ML_DEV_DEBUG
- if (layer_id == model->mvtvm.metadata.model.nb_layers) {
+ if (layer_id == model->nb_layers) {
plt_err("Invalid layer name: %s", layer_name);
return -EINVAL;
}
@@ -553,28 +690,69 @@ static int
mvtvm_ml_model_run(struct cnxk_ml_model *model, struct rte_ml_op *op, struct cnxk_ml_req *req)
{
uint8_t i;
+ struct mvtvm_ml_result *run_result;
+ TVMValue arg_values[2] = {0};
+ int arg_types[2] = {0};
+ TVMValue ret_value = {0};
+ int ret_type = kTVMNullptr;
+ int ret = 0;
rte_memcpy(req->mvtvm_req.input_tensor, model->mvtvm.input_tensor,
- model->mvtvm.metadata.model.num_input * sizeof(DLTensor));
- for (i = 0; i < model->mvtvm.metadata.model.num_input; i++) {
+ model->mvtvm.info.nb_inputs * sizeof(DLTensor));
+ for (i = 0; i < model->mvtvm.info.nb_inputs; i++) {
req->mvtvm_req.input_tensor[i].data = op->input[i]->addr;
req->mvtvm_req.input_tensor[i].byte_offset = 0;
}
rte_memcpy(req->mvtvm_req.output_tensor, model->mvtvm.output_tensor,
- model->mvtvm.metadata.model.num_output * sizeof(DLTensor));
- for (i = 0; i < model->mvtvm.metadata.model.num_output; i++) {
+ model->mvtvm.info.nb_outputs * sizeof(DLTensor));
+ for (i = 0; i < model->mvtvm.info.nb_outputs; i++) {
req->mvtvm_req.output_tensor[i].data = op->output[i]->addr;
req->mvtvm_req.output_tensor[i].byte_offset = 0;
}
- tvmdp_model_run(model->model_id, model->mvtvm.metadata.model.num_input,
- req->mvtvm_req.input_tensor, model->mvtvm.metadata.model.num_output,
- req->mvtvm_req.output_tensor, &req->mvtvm_req.result,
- &req->mvtvm_req.status);
+ run_result = &req->mvtvm_req.result;
+ run_result->stats.start_ns = rte_get_tsc_cycles();
+ run_result->error_code = 0;
+
+ for (i = 0; i < model->mvtvm.info.nb_inputs; i++) {
+ arg_values[0].v_int64 = i;
+ arg_types[0] = kDLInt;
+ arg_values[1].v_handle = &req->mvtvm_req.input_tensor[i];
+ arg_types[1] = kTVMDLTensorHandle;
+ ret = mvtvm_ml_tvm_func_call(model, model->mvtvm.set_input_zero_copy,
+ "set_input_zero_copy", arg_values, arg_types, 2,
+ &ret_value, &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto out;
+ }
+
+ for (i = 0; i < model->mvtvm.info.nb_outputs; i++) {
+ arg_values[0].v_int64 = i;
+ arg_types[0] = kDLInt;
+ arg_values[1].v_handle = &req->mvtvm_req.output_tensor[i];
+ arg_types[1] = kTVMDLTensorHandle;
+ ret = mvtvm_ml_tvm_func_call(model, model->mvtvm.set_output_zero_copy,
+ "set_output_zero_copy", arg_values, arg_types, 2,
+ &ret_value, &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto out;
+ }
+
+ ret = mvtvm_ml_tvm_func_call(model, model->mvtvm.run, "run", NULL, NULL, 0, &ret_value,
+ &ret_type, kTVMNullptr);
+ if (ret != 0)
+ goto out;
+
+out:
+ run_result->stats.end_ns = rte_get_tsc_cycles();
+ req->mvtvm_req.status = 0x1;
plt_write64(ML_CNXK_POLL_JOB_FINISH, req->status);
+ if (ret != 0)
+ run_result->error_code = -EIO;
+
return 0;
}
diff --git a/drivers/ml/cnxk/mvtvm_ml_ops.h b/drivers/ml/cnxk/mvtvm_ml_ops.h
index d8f2f361fb1..593f4585b3c 100644
--- a/drivers/ml/cnxk/mvtvm_ml_ops.h
+++ b/drivers/ml/cnxk/mvtvm_ml_ops.h
@@ -7,11 +7,10 @@
#include <dlpack/dlpack.h>
-#include <tvmdp.h>
-
#include <rte_mldev.h>
#include "cnxk_ml_xstats.h"
+#include "mvtvm_ml_model.h"
struct cnxk_ml_dev;
struct cnxk_ml_model;
@@ -56,8 +55,6 @@ struct mvtvm_ml_req {
};
int mvtvm_ml_dev_info_get(struct cnxk_ml_dev *mldev, struct rte_ml_dev_info *dev_info);
-int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf);
-int mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev);
int mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp);
int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
struct cnxk_ml_model *model);
diff --git a/drivers/ml/cnxk/mvtvm_ml_stubs.c b/drivers/ml/cnxk/mvtvm_ml_stubs.c
index 126a954c916..7c13fac42d6 100644
--- a/drivers/ml/cnxk/mvtvm_ml_stubs.c
+++ b/drivers/ml/cnxk/mvtvm_ml_stubs.c
@@ -76,23 +76,6 @@ mvtvm_ml_dev_info_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_dev_info *de
return -ENOTSUP;
}
-int
-mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf)
-{
- RTE_SET_USED(cnxk_mldev);
- RTE_SET_USED(conf);
-
- return 0;
-}
-
-int
-mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev)
-{
- RTE_SET_USED(cnxk_mldev);
-
- return 0;
-}
-
int
mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp)
{
diff --git a/drivers/ml/cnxk/mvtvm_ml_stubs.h b/drivers/ml/cnxk/mvtvm_ml_stubs.h
index 4220a963f2d..15985a75bf4 100644
--- a/drivers/ml/cnxk/mvtvm_ml_stubs.h
+++ b/drivers/ml/cnxk/mvtvm_ml_stubs.h
@@ -15,8 +15,6 @@ struct cnxk_ml_layer;
enum cnxk_ml_model_type mvtvm_ml_model_type_get(struct rte_ml_model_params *params);
int mvtvm_ml_dev_info_get(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_dev_info *dev_info);
-int mvtvm_ml_dev_configure(struct cnxk_ml_dev *cnxk_mldev, const struct rte_ml_dev_config *conf);
-int mvtvm_ml_dev_close(struct cnxk_ml_dev *cnxk_mldev);
int mvtvm_ml_dev_dump(struct cnxk_ml_dev *cnxk_mldev, FILE *fp);
int mvtvm_ml_model_load(struct cnxk_ml_dev *cnxk_mldev, struct rte_ml_model_params *params,
struct cnxk_ml_model *model);
--
2.34.1
^ permalink raw reply related
* [PATCH 1/1] buildtools: remove absolute paths from pc file
From: Srikanth Yalavarthi @ 2026-06-09 4:32 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev, Srikanth Yalavarthi
When linking with non-versioned libraries, absolute paths
of the libraries are added to libdpdk.pc. This patch replaces
the absolute path with correct linker flags, -l<libname>.
https://github.com/mesonbuild/meson/issues/7766
Signed-off-by: Srikanth Yalavarthi <syalavarthi@marvell.com>
---
buildtools/pkg-config/set-static-linker-flags.py | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/buildtools/pkg-config/set-static-linker-flags.py b/buildtools/pkg-config/set-static-linker-flags.py
index 2745db34c29..bd217b07fc8 100644
--- a/buildtools/pkg-config/set-static-linker-flags.py
+++ b/buildtools/pkg-config/set-static-linker-flags.py
@@ -6,9 +6,16 @@
# Should be called from meson build itself
import os
import sys
+import re
def fix_ldflag(f):
+ if f.startswith('/'):
+ libfile = os.path.basename(f)
+ m = re.match(r'^lib(.+?)\.(a|so(?:\..*)?)$', libfile)
+ if m:
+ return '-l' + m.group(1)
+
if not f.startswith('-lrte_'):
return f
return '-l:lib' + f[2:] + '.a'
--
2.34.1
^ permalink raw reply related
* [PATCH 1/1] mldev: fix typos in code documentation
From: Srikanth Yalavarthi @ 2026-06-09 4:05 UTC (permalink / raw)
To: Srikanth Yalavarthi, Jerin Jacob; +Cc: dev, sshankarnara, ptakkar, aprabhu
Fix typos in MLDEV spec and code documentation.
Fixes: d82cac584f84 ("mldev: introduce machine learning device API")
Signed-off-by: Srikanth Yalavarthi <syalavarthi@marvell.com>
---
lib/mldev/rte_mldev.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/lib/mldev/rte_mldev.h b/lib/mldev/rte_mldev.h
index 3f51083cb7d..594c46cf72c 100644
--- a/lib/mldev/rte_mldev.h
+++ b/lib/mldev/rte_mldev.h
@@ -497,7 +497,7 @@ struct __rte_cache_aligned rte_ml_op {
*
* The rte_ml_enqueue_burst() function returns the number of inferences it
* actually enqueued for processing. A return value equal to *nb_ops* means that
- * all packets have been enqueued.
+ * all inferences have been enqueued.
*
* @param dev_id
* The identifier of the device.
@@ -540,7 +540,7 @@ rte_ml_enqueue_burst(int16_t dev_id, uint16_t qp_id, struct rte_ml_op **ops, uin
* @param dev_id
* The identifier of the device.
* @param qp_id
- * The index of the queue pair from which to retrieve processed packets.
+ * The index of the queue pair from which to retrieve processed inferences.
* The value must be in the range [0, nb_queue_pairs - 1] previously supplied to
* rte_ml_dev_configure().
* @param ops
@@ -840,7 +840,7 @@ rte_ml_model_unload(int16_t dev_id, uint16_t model_id);
* Identifier of the model to be started.
*
* @return
- * - 0: Success, Model loaded.
+ * - 0: Success, Model started.
* - < 0: Failure, Error code of the model start driver function.
*/
__rte_experimental
@@ -859,7 +859,7 @@ rte_ml_model_start(int16_t dev_id, uint16_t model_id);
* Identifier of the model to be stopped.
*
* @return
- * - 0: Success, Model unloaded.
+ * - 0: Success, Model stopped.
* - < 0: Failure, Error code of the model stop driver function.
*/
__rte_experimental
--
2.34.1
^ permalink raw reply related
* [PATCH v2 1/1] ml/cnxk: fix overwriting layer name during load
From: Srikanth Yalavarthi @ 2026-06-09 3:52 UTC (permalink / raw)
To: Srikanth Yalavarthi; +Cc: dev, jerinj
In-Reply-To: <20260331085445.1105590-1-syalavarthi@marvell.com>
Layer name is initialized during metadata fetch and parsing
stage and should not be overwritten during model load. Fix
overwriting the layer name during layer load.
Fixes: cc254869ae19 ("ml/cnxk: update model load/unload functions")
Signed-off-by: Srikanth Yalavarthi <syalavarthi@marvell.com>
---
v2:
* Updated commit message with fixline
v1:
* Initial patch
drivers/ml/cnxk/cn10k_ml_ops.c | 3 ---
1 file changed, 3 deletions(-)
diff --git a/drivers/ml/cnxk/cn10k_ml_ops.c b/drivers/ml/cnxk/cn10k_ml_ops.c
index b30af7c7a44..cb9e6561442 100644
--- a/drivers/ml/cnxk/cn10k_ml_ops.c
+++ b/drivers/ml/cnxk/cn10k_ml_ops.c
@@ -671,9 +671,6 @@ cn10k_ml_layer_load(void *device, uint16_t model_id, const char *layer_name, uin
rte_memcpy(&layer->glow.metadata, buffer, sizeof(struct cn10k_ml_model_metadata));
cn10k_ml_model_metadata_update(&layer->glow.metadata);
- /* Set layer name */
- rte_memcpy(layer->name, layer->glow.metadata.model.name, MRVL_ML_MODEL_NAME_LEN);
-
/* Enable support for batch_size of 256 */
if (layer->glow.metadata.model.batch_size == 0)
layer->batch_size = 256;
--
2.34.1
^ permalink raw reply related
* [PATCH v14 19/20] drivers: add testpmd commands for private features
From: liujie5 @ 2026-06-09 1:39 UTC (permalink / raw)
To: stephen; +Cc: dev, Jie Liu
In-Reply-To: <20260609013951.3359199-1-liujie5@linkdatatechnology.com>
From: Jie Liu <liujie5@linkdatatechnology.com>
Introduce private testpmd commands and implementation files to enable
debugging and testing of sxe2-specific hardware features (such as
packet scheduling reset, UDP tunnel configuration, and IPsec ingress/
egress offloads) directly within the testpmd application.
The parameters are parsed using the standard 'rte_kvargs' library during
the PCI/vdev probing phase. Documentation for these parameters is also
updated.
During memory hotplug events, the SXE2 driver needs to track memory
segment layout changes to maintain internal DMA mappings. However,
existing memseg walk functions (rte_memseg_walk) acquire memory locks
and cannot be called from within memory event callbacks, leading to
potential deadlocks.
This commit introduces sxe2_memseg_walk_cb() as a helper that walks
memory segments using the thread-unsafe variant
rte_memseg_walk_thread_unsafe(), which is safe to call from
memory-related callbacks [citation:1][citation:3][citation:5].
The implementation follows the standard rte_memseg_walk_t prototype,
processing each memseg to update driver-specific data structures.
Signed-off-by: Jie Liu <liujie5@linkdatatechnology.com>
---
drivers/common/sxe2/sxe2_common.c | 110 +++
drivers/common/sxe2/sxe2_common.h | 2 +
drivers/common/sxe2/sxe2_ioctl_chnl.c | 2 +-
drivers/net/sxe2/meson.build | 5 +-
drivers/net/sxe2/sxe2_cmd_chnl.c | 21 +
drivers/net/sxe2/sxe2_cmd_chnl.h | 3 +
drivers/net/sxe2/sxe2_drv_cmd.h | 17 +
drivers/net/sxe2/sxe2_dump.c | 15 +
drivers/net/sxe2/sxe2_ethdev.c | 287 +++++++-
drivers/net/sxe2/sxe2_ethdev.h | 8 +
drivers/net/sxe2/sxe2_irq.c | 34 +-
drivers/net/sxe2/sxe2_rx.c | 12 +
drivers/net/sxe2/sxe2_testpmd.c | 733 +++++++++++++++++++
drivers/net/sxe2/sxe2_testpmd_lib.c | 969 ++++++++++++++++++++++++++
drivers/net/sxe2/sxe2_testpmd_lib.h | 142 ++++
drivers/net/sxe2/sxe2_tm.c | 18 +
drivers/net/sxe2/sxe2_tm.h | 2 +
17 files changed, 2374 insertions(+), 6 deletions(-)
create mode 100644 drivers/net/sxe2/sxe2_testpmd.c
create mode 100644 drivers/net/sxe2/sxe2_testpmd_lib.c
create mode 100644 drivers/net/sxe2/sxe2_testpmd_lib.h
diff --git a/drivers/common/sxe2/sxe2_common.c b/drivers/common/sxe2/sxe2_common.c
index c000a55cd0..5c5db85f29 100644
--- a/drivers/common/sxe2/sxe2_common.c
+++ b/drivers/common/sxe2/sxe2_common.c
@@ -196,6 +196,102 @@ static int32_t sxe2_parse_representor(const char *key, const char *value, void *
PMD_LOG_INFO(COM, "representor arg %s: \"%s\".", key, value);
+l_end:
+ return ret;
+}
+static int32_t sxe2_dma_mem_map(struct sxe2_common_device *cdev,
+ const void *addr, size_t len, bool do_map)
+{
+ struct rte_memseg_list *msl;
+ struct rte_memseg *ms;
+ size_t cur_len = 0;
+ int32_t ret = 0;
+
+ msl = rte_mem_virt2memseg_list(addr);
+ if (msl == NULL) {
+ ret = -EINVAL;
+ PMD_LOG_ERR(COM, "Invalid virt addr=%p.", addr);
+ goto l_end;
+ }
+
+ if ((uintptr_t)addr != RTE_ALIGN((uintptr_t)addr, msl->page_sz) ||
+ (len != RTE_ALIGN(len, msl->page_sz))) {
+ ret = -EINVAL;
+ PMD_LOG_ERR(COM, "Addr=%p and len=%zu not align page size=%" PRIu64 ".",
+ addr, len, msl->page_sz);
+ goto l_end;
+ }
+
+ /* memsegs are contiguous in memory */
+ ms = rte_mem_virt2memseg(addr, msl);
+ while (cur_len < len) {
+ /* some memory segments may have invalid IOVA */
+ if (ms->iova == RTE_BAD_IOVA) {
+ PMD_LOG_WARN(COM, "Memory segment at %p has bad IOVA, skipping.",
+ ms->addr);
+ goto next;
+ }
+ if (do_map)
+ sxe2_drv_dev_dma_map(cdev, ms->addr_64,
+ ms->iova, ms->len);
+ else
+ sxe2_drv_dev_dma_unmap(cdev, ms->iova);
+
+next:
+ cur_len += ms->len;
+ ++ms;
+ }
+
+l_end:
+ return ret;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(sxe2_common_mem_event_cb)
+void
+sxe2_common_mem_event_cb(enum rte_mem_event type,
+ const void *addr, size_t size, void *arg __rte_unused)
+{
+ struct sxe2_common_device *cdev = NULL;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ goto l_end;
+
+ pthread_mutex_lock(&sxe2_common_devices_list_lock);
+ switch (type) {
+ case RTE_MEM_EVENT_FREE:
+ TAILQ_FOREACH(cdev, &sxe2_common_devices_list, next)
+ (void)sxe2_dma_mem_map(cdev, addr, size, 0);
+ break;
+ case RTE_MEM_EVENT_ALLOC:
+ TAILQ_FOREACH(cdev, &sxe2_common_devices_list, next)
+ (void)sxe2_dma_mem_map(cdev, addr, size, 1);
+ break;
+ default:
+ break;
+ }
+ pthread_mutex_unlock(&sxe2_common_devices_list_lock);
+l_end:
+ return;
+}
+
+static int32_t sxe2_memseg_walk_cb(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct sxe2_common_device *cdev = arg;
+ int32_t ret = 0;
+
+ if (msl->external && !msl->heap)
+ goto l_end;
+
+ if (ms->iova == RTE_BAD_IOVA)
+ goto l_end;
+
+ ret = sxe2_drv_dev_dma_map(cdev, ms->addr_64, ms->iova, ms->len);
+ if (ret != 0) {
+ PMD_LOG_ERR(COM, "Fail to memseg dma map.");
+ goto l_end;
+ }
+
l_end:
return ret;
}
@@ -220,6 +316,18 @@ static int32_t sxe2_common_device_setup(struct sxe2_common_device *cdev)
goto l_close_dev;
}
+ rte_mcfg_mem_read_lock();
+ ret = rte_memseg_walk_thread_unsafe(sxe2_memseg_walk_cb, cdev);
+ if (ret) {
+ PMD_LOG_ERR(COM, "Fail to walk memseg, ret=%d", ret);
+ rte_mcfg_mem_read_unlock();
+ goto l_close_dev;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ (void)rte_mem_event_callback_register("SXE2_MEM_EVENT_CB",
+ sxe2_common_mem_event_cb, NULL);
+
goto l_end;
l_close_dev:
@@ -251,6 +359,7 @@ static struct sxe2_common_device *sxe2_common_device_alloc(
}
cdev->dev = rte_dev;
cdev->class_type = class_type;
+ cdev->config.cmd_fd = SXE2_CMD_FD_INVALID;
cdev->config.kernel_reset = false;
pthread_mutex_init(&cdev->config.lock, NULL);
@@ -631,6 +740,7 @@ static int32_t sxe2_common_pci_id_table_update(const struct rte_pci_id *id_table
updated_table = calloc(num_ids, sizeof(*updated_table));
if (!updated_table) {
+ ret = -ENOMEM;
PMD_LOG_ERR(COM, "Failed to allocate memory for PCI ID table");
goto l_end;
}
diff --git a/drivers/common/sxe2/sxe2_common.h b/drivers/common/sxe2/sxe2_common.h
index b02b6317da..efc8d3585a 100644
--- a/drivers/common/sxe2/sxe2_common.h
+++ b/drivers/common/sxe2/sxe2_common.h
@@ -14,6 +14,8 @@
#define SXE2_COMMON_PCI_DRIVER_NAME "sxe2_pci"
+#define SXE2_CMD_FD_INVALID (-1)
+
#define SXE2_CDEV_TO_CMD_FD(cdev) \
((cdev)->config.cmd_fd)
diff --git a/drivers/common/sxe2/sxe2_ioctl_chnl.c b/drivers/common/sxe2/sxe2_ioctl_chnl.c
index 173d8d57ae..a233a78136 100644
--- a/drivers/common/sxe2/sxe2_ioctl_chnl.c
+++ b/drivers/common/sxe2/sxe2_ioctl_chnl.c
@@ -110,7 +110,7 @@ sxe2_drv_dev_close(struct sxe2_common_device *cdev)
if (fd >= 0)
close(fd);
PMD_LOG_INFO(COM, "closed device fd=%d", fd);
- SXE2_CDEV_TO_CMD_FD(cdev) = -1;
+ SXE2_CDEV_TO_CMD_FD(cdev) = SXE2_CMD_FD_INVALID;
}
RTE_EXPORT_INTERNAL_SYMBOL(sxe2_drv_dev_handshake)
diff --git a/drivers/net/sxe2/meson.build b/drivers/net/sxe2/meson.build
index 4fb2333926..04369402b7 100644
--- a/drivers/net/sxe2/meson.build
+++ b/drivers/net/sxe2/meson.build
@@ -9,9 +9,10 @@ endif
cflags += ['-g']
-deps += ['common_sxe2', 'hash','cryptodev','security']
+deps += ['common_sxe2', 'hash', 'cryptodev', 'security', 'cmdline']
includes += include_directories('../../common/sxe2')
+testpmd_sources = files('sxe2_testpmd.c')
if arch_subdir == 'x86'
sources += files('sxe2_txrx_vec_sse.c')
@@ -79,5 +80,5 @@ sources += files(
'sxe2_flow_parse_engine.c',
'sxe2_dump.c',
'sxe2_txrx_check_mbuf.c',
-
+ 'sxe2_testpmd_lib.c',
)
diff --git a/drivers/net/sxe2/sxe2_cmd_chnl.c b/drivers/net/sxe2/sxe2_cmd_chnl.c
index 43e8c59487..b09989fe50 100644
--- a/drivers/net/sxe2/sxe2_cmd_chnl.c
+++ b/drivers/net/sxe2/sxe2_cmd_chnl.c
@@ -99,6 +99,27 @@ int32_t sxe2_drv_dev_info_get(struct sxe2_adapter *adapter,
return ret;
}
+int32_t sxe2_drv_fc_state_get(struct sxe2_adapter *adapter,
+ struct sxe2_drv_vsi_fc_get_resp *dev_fc_state_resp)
+{
+ int32_t ret = 0;
+ struct sxe2_common_device *cdev = adapter->cdev;
+ struct sxe2_drv_cmd_params param = {0};
+ struct sxe2_drv_vsi_fc_get_req req = {0};
+
+ req.vsi_id = adapter->vsi_ctxt.main_vsi->vsi_id;
+ sxe2_drv_cmd_params_fill(adapter, ¶m, SXE2_DRV_CMD_VSI_FC_GET,
+ &req, sizeof(req),
+ dev_fc_state_resp,
+ sizeof(*dev_fc_state_resp));
+ ret = sxe2_drv_cmd_exec(cdev, ¶m);
+ if (ret) {
+ PMD_DEV_LOG_ERR(adapter, DRV, "get fc state failed, ret=%d", ret);
+ ret = -EIO;
+ }
+ return ret;
+}
+
int32_t sxe2_drv_dev_fw_info_get(struct sxe2_adapter *adapter,
struct sxe2_drv_dev_fw_info_resp *dev_fw_info_resp)
{
diff --git a/drivers/net/sxe2/sxe2_cmd_chnl.h b/drivers/net/sxe2/sxe2_cmd_chnl.h
index 988d4b458b..d63caad526 100644
--- a/drivers/net/sxe2/sxe2_cmd_chnl.h
+++ b/drivers/net/sxe2/sxe2_cmd_chnl.h
@@ -99,6 +99,9 @@ int32_t sxe2_drv_vsi_stats_reset(struct sxe2_adapter *adapter);
int32_t sxe2_drv_queue_info_get_update(struct sxe2_adapter *adapter,
struct eth_queue_stats *qstats);
+int32_t sxe2_drv_fc_state_get(struct sxe2_adapter *adapter,
+ struct sxe2_drv_vsi_fc_get_resp *dev_fc_state_resp);
+
int32_t sxe2_drv_rxq_mapping_set(struct rte_eth_dev *eth_dev, uint16_t queue_id, uint8_t pool_idx);
int32_t sxe2_drv_txq_mapping_set(struct rte_eth_dev *eth_dev, uint16_t queue_id, uint8_t pool_idx);
diff --git a/drivers/net/sxe2/sxe2_drv_cmd.h b/drivers/net/sxe2/sxe2_drv_cmd.h
index 09b2f7d125..59a8aa6f13 100644
--- a/drivers/net/sxe2/sxe2_drv_cmd.h
+++ b/drivers/net/sxe2/sxe2_drv_cmd.h
@@ -651,6 +651,23 @@ struct __rte_aligned(4) __rte_packed_begin sxe2_drv_sfp_resp {
uint8_t data[];
} __rte_packed_end;
+enum sxe2_fc_type {
+ SXE2_FC_T_DIS = 0,
+ SXE2_FC_T_LFC,
+ SXE2_FC_T_PFC,
+ SXE2_FC_T_UNKNOWN = 255,
+};
+
+struct __rte_aligned(4) __rte_packed_begin sxe2_drv_vsi_fc_get_req {
+ uint16_t vsi_id;
+ uint8_t rsv[2];
+} __rte_packed_end;
+
+struct __rte_aligned(4) __rte_packed_begin sxe2_drv_vsi_fc_get_resp {
+ uint8_t fc_enable;
+ uint8_t rsv[3];
+} __rte_packed_end;
+
enum sxe2_drv_cmd_module {
SXE2_DRV_CMD_MODULE_HANDSHAKE = 0,
SXE2_DRV_CMD_MODULE_DEV = 1,
diff --git a/drivers/net/sxe2/sxe2_dump.c b/drivers/net/sxe2/sxe2_dump.c
index 1753eccf99..fd0a99d6fd 100644
--- a/drivers/net/sxe2/sxe2_dump.c
+++ b/drivers/net/sxe2/sxe2_dump.c
@@ -188,6 +188,20 @@ static void sxe2_dump_filter_info(FILE *file, struct rte_eth_dev *dev)
return;
}
+static void sxe2_dump_fc_state(FILE *file, struct rte_eth_dev *dev)
+{
+ struct sxe2_adapter *adapter = SXE2_DEV_PRIVATE_TO_ADAPTER(dev);
+
+ if (!(adapter->cap_flags & SXE2_DEV_CAPS_OFFLOAD_FC_STATE))
+ goto l_end;
+
+ fprintf(file, " -- fc state:\n"
+ "\t -- curr_state: %u\n",
+ adapter->fc_state_ctx.curr_state);
+l_end:
+ return;
+}
+
static const char *sxe2_vsi_id_str(uint16_t vsi_id, char *buf, size_t len)
{
if (vsi_id == SXE2_INVALID_VSI_ID)
@@ -274,6 +288,7 @@ int32_t sxe2_eth_dev_priv_dump(struct rte_eth_dev *dev, FILE *file)
sxe2_dump_dev_args_info(str, dev);
sxe2_dump_filter_info(str, dev);
sxe2_dump_switchdev_info(str, dev);
+ sxe2_dump_fc_state(str, dev);
(void)fflush(str);
diff --git a/drivers/net/sxe2/sxe2_ethdev.c b/drivers/net/sxe2/sxe2_ethdev.c
index 3c86424c15..b768a780b4 100644
--- a/drivers/net/sxe2/sxe2_ethdev.c
+++ b/drivers/net/sxe2/sxe2_ethdev.c
@@ -68,7 +68,14 @@ static const struct rte_pci_id pci_id_sxe2_tbl[] = {
{ RTE_PCI_DEVICE(SXE2_PCI_VENDOR_ID_206F, SXE2_PCI_DEVICE_ID_VF_1)},
{ .vendor_id = 0, },
};
-
+#define SXE2_TXSCH_NODE_ADJ_LVL_MAX 3
+#define SXE2_DEVARG_FLOW_DULP_PATTERN_MODE "flow-duplicate-pattern"
+#define SXE2_DEVARG_FUNC_FLOW_DIRCT "function-flow-direct"
+#define SXE2_DEVARG_FNAV_STAT_TYPE "fnav-stat-type"
+#define SXE2_DEVARG_SW_STATS "drv-sw-stats"
+#define SXE2_DEVARG_HIGH_PERFORMANCE_MODE "high-performance-mode"
+#define SXE2_DEVARG_SCHED_LAYER_MODE "sched-layer-mode"
+#define SXE2_DEVARG_RX_LOW_LATENCY "rx-low-latency"
static struct sxe2_pci_map_addr_info sxe2_net_map_addr_info_pf[SXE2_PCI_MAP_RES_MAX_COUNT] = {
[SXE2_PCI_MAP_RES_INVALID] = {.addr_base = 0,
.bar_idx = 0,
@@ -980,6 +987,149 @@ static inline void sxe2_init_ptype_tbl(struct rte_eth_dev *dev)
sxe2_init_ptype_list(ptype);
}
+static int32_t sxe2_parse_fnav_stat_type(const char *key, const char *value, void *args)
+{
+ int32_t ret = -EINVAL;
+ uint8_t *num = (uint8_t *)args;
+ uint8_t fnav_stat_type = 0;
+ char *endptr = NULL;
+
+ if (value == NULL || args == NULL) {
+ ret = 0;
+ goto l_end;
+ }
+ errno = 0;
+ fnav_stat_type = (uint8_t)strtoul(value, &endptr, 10);
+ if (errno != 0 || *endptr != '\0') {
+ PMD_LOG_WARN(INIT, "%s: \"%s\" is not a valid int value.",
+ key, value);
+ goto l_end;
+ }
+ if (fnav_stat_type > SXE2_FNAV_STAT_ENA_ALL ||
+ fnav_stat_type == SXE2_FNAV_STAT_ENA_NONE) {
+ PMD_LOG_ERR(INIT, "%s: \"%s\" out of range [1-3].",
+ key, value);
+ goto l_end;
+ }
+ *num = fnav_stat_type;
+ ret = 0;
+l_end:
+ return ret;
+}
+static int32_t sxe2_parse_sched_layer_mode(const char *key, const char *value, void *args)
+{
+ int32_t ret = -EINVAL;
+ uint8_t *num = (uint8_t *)args;
+ uint8_t sched_layer_mode;
+ char *endptr = NULL;
+
+ if (value == NULL || args == NULL) {
+ ret = 0;
+ goto l_end;
+ }
+ errno = 0;
+ sched_layer_mode = (uint8_t)strtoul(value, &endptr, 10);
+ if (errno != 0 || *endptr != '\0') {
+ PMD_LOG_WARN(INIT, "%s: \"%s\" is not a valid int value.",
+ key, value);
+ goto l_end;
+ }
+ if (sched_layer_mode > SXE2_TXSCH_NODE_ADJ_LVL_MAX) {
+ PMD_LOG_ERR(INIT, "%s: \"%s\" > 3.",
+ key, value);
+ goto l_end;
+ }
+ *num = sched_layer_mode;
+ ret = 0;
+l_end:
+ return ret;
+}
+static int32_t sxe2_parse_high_performance_mode(const char *key, const char *value, void *args)
+{
+ int32_t ret = -EINVAL;
+ uint8_t *num = (uint8_t *)args;
+ uint8_t high_performance_mode;
+ char *endptr = NULL;
+
+ if (value == NULL || args == NULL) {
+ ret = 0;
+ goto l_end;
+ }
+ errno = 0;
+ high_performance_mode = (uint8_t)strtoul(value, &endptr, 10);
+ if (errno != 0 || *endptr != '\0') {
+ PMD_LOG_WARN(INIT, "%s: \"%s\" is not a valid int value.",
+ key, value);
+ goto l_end;
+ }
+ if (high_performance_mode != 1) {
+ PMD_LOG_ERR(INIT, "%s: \"%s\" != 1.",
+ key, value);
+ goto l_end;
+ }
+ *num = high_performance_mode;
+ ret = 0;
+l_end:
+ return ret;
+}
+static int32_t sxe2_parse_u8(const char *key, const char *value, void *args)
+{
+ uint8_t *num = (uint8_t *)args;
+ char *end;
+ unsigned long val;
+ int32_t ret = -EINVAL;
+
+ if (value == NULL || args == NULL) {
+ ret = 0;
+ goto l_end;
+ }
+ errno = 0;
+ val = strtoul(value, &end, 10);
+ if (errno != 0 || end == value || *end != '\0') {
+ PMD_LOG_ERR(INIT, "Invalid 8-bit integer value for key %s: %s", key, value);
+ return -EINVAL;
+ }
+
+ if (val > UINT8_MAX) {
+ PMD_LOG_ERR(INIT, "%s: \"%s\" out of range [0-255].",
+ key, value);
+ return -ERANGE;
+ }
+
+ *num = val;
+ ret = 0;
+l_end:
+ return ret;
+}
+static int32_t sxe2_parse_bool(const char *key, const char *value, void *args)
+{
+ int32_t ret = -EINVAL;
+ uint8_t *num = (uint8_t *)args;
+ uint8_t bool_val = 0;
+ char *endptr = NULL;
+
+ if (value == NULL || args == NULL) {
+ ret = 0;
+ goto l_end;
+ }
+ errno = 0;
+ bool_val = (uint8_t)strtoul(value, &endptr, 10);
+ if (errno != 0 || *endptr != '\0') {
+ PMD_LOG_WARN(INIT, "%s: \"%s\" is not a valid int value.",
+ key, value);
+ goto l_end;
+ }
+ if (bool_val != 0 && bool_val != 1) {
+ PMD_LOG_ERR(INIT, "%s: \"%s\" out of range [0|1].",
+ key, value);
+ goto l_end;
+ }
+ *num = bool_val;
+ ret = 0;
+l_end:
+ return ret;
+}
+
struct sxe2_pci_map_bar_info *sxe2_dev_get_bar_info(struct sxe2_adapter *adapter,
enum sxe2_pci_map_resource res_type)
{
@@ -1047,6 +1197,69 @@ void *sxe2_pci_map_addr_get(struct sxe2_adapter *adapter,
return addr;
}
+static int32_t sxe2_args_parse(struct rte_eth_dev *dev, struct sxe2_dev_kvargs_info *kvargs)
+{
+ struct sxe2_adapter *adapter = SXE2_DEV_PRIVATE_TO_ADAPTER(dev);
+ int32_t ret = 0;
+ PMD_INIT_FUNC_TRACE();
+
+ if (kvargs == NULL)
+ goto l_end;
+ ret = sxe2_kvargs_process(kvargs, SXE2_DEVARG_FNAV_STAT_TYPE,
+ &sxe2_parse_fnav_stat_type,
+ &adapter->devargs.fnav_stat_type);
+ if (ret) {
+ PMD_DEV_LOG_ERR(adapter, INIT, "Failed to parse fnav stat type, ret:%d", ret);
+ goto l_end;
+ }
+ ret = sxe2_kvargs_process(kvargs, SXE2_DEVARG_SW_STATS,
+ &sxe2_parse_bool,
+ &adapter->devargs.sw_stats_en);
+ if (ret) {
+ PMD_DEV_LOG_ERR(adapter, INIT, "Failed to parse sw stats enable, ret:%d", ret);
+ goto l_end;
+ }
+ ret = sxe2_kvargs_process(kvargs, SXE2_DEVARG_HIGH_PERFORMANCE_MODE,
+ &sxe2_parse_high_performance_mode,
+ &adapter->devargs.high_performance_mode);
+ if (ret) {
+ PMD_DEV_LOG_ERR(adapter, INIT, "Failed to parse high performance, ret:%d", ret);
+ goto l_end;
+ }
+ ret = sxe2_kvargs_process(kvargs, SXE2_DEVARG_SCHED_LAYER_MODE,
+ &sxe2_parse_sched_layer_mode,
+ &adapter->devargs.sched_layer_mode);
+ if (ret) {
+ PMD_DEV_LOG_ERR(adapter, INIT, "Failed to parse sched layer mode, ret:%d", ret);
+ goto l_end;
+ }
+ ret = sxe2_kvargs_process(kvargs, SXE2_DEVARG_FLOW_DULP_PATTERN_MODE,
+ &sxe2_parse_u8,
+ &adapter->devargs.flow_dup_pattern_mode);
+ if (ret) {
+ PMD_DEV_LOG_ERR(adapter, INIT, "Failed to parse switch dulpliate flow pattern mode,"
+ "ret:%d", ret);
+ goto l_end;
+ }
+ ret = sxe2_kvargs_process(kvargs, SXE2_DEVARG_FUNC_FLOW_DIRCT,
+ &sxe2_parse_bool,
+ &adapter->devargs.func_flow_direct_en);
+ if (ret) {
+ PMD_DEV_LOG_ERR(adapter, INIT, "Failed to parse function flow rule enable,"
+ "ret:%d", ret);
+ goto l_end;
+ }
+ ret = sxe2_kvargs_process(kvargs, SXE2_DEVARG_RX_LOW_LATENCY,
+ &sxe2_parse_bool,
+ &adapter->devargs.rx_low_latency);
+ if (ret) {
+ PMD_DEV_LOG_ERR(adapter, INIT, "Failed to parse rx low latency, ret:%d", ret);
+ goto l_end;
+ }
+l_end:
+ return ret;
+}
+
static int32_t sxe2_eth_init(struct rte_eth_dev *dev)
{
int32_t ret = 0;
@@ -1599,6 +1812,37 @@ void sxe2_dev_pci_map_uinit(struct rte_eth_dev *dev)
adapter->dev_info.dev_data = NULL;
}
+static int32_t sxe2_fc_state_init(struct rte_eth_dev *dev)
+{
+ struct sxe2_adapter *adapter =
+ SXE2_DEV_PRIVATE_TO_ADAPTER(dev);
+ struct sxe2_drv_vsi_fc_get_resp fc_resp = {0};
+ int32_t ret;
+
+ if (!(adapter->cap_flags & SXE2_DEV_CAPS_OFFLOAD_FC_STATE)) {
+ adapter->fc_state_ctx.cfg_state = 0;
+ adapter->fc_state_ctx.curr_state = 0;
+ ret = 0;
+ goto l_end;
+ }
+ ret = sxe2_drv_fc_state_get(adapter, &fc_resp);
+ if (ret) {
+ PMD_LOG_ERR(INIT, "Failed to get fc state, ret=[%d]", ret);
+ goto l_end;
+ }
+ adapter->fc_state_ctx.cfg_state = fc_resp.fc_enable;
+ adapter->fc_state_ctx.curr_state = 0;
+l_end:
+ return ret;
+}
+static void sxe2_fc_state_uinit(struct rte_eth_dev *dev)
+{
+ struct sxe2_adapter *adapter =
+ SXE2_DEV_PRIVATE_TO_ADAPTER(dev);
+ adapter->fc_state_ctx.cfg_state = 0;
+ adapter->fc_state_ctx.curr_state = 0;
+}
+
uint32_t sxe2_sched_mode_get(struct sxe2_adapter *adapter)
{
uint32_t ret_mode = SXE2_SCHED_MODE_INVALID;
@@ -1661,6 +1905,32 @@ static int32_t sxe2_sched_uinit(struct rte_eth_dev *dev)
return ret;
}
+int32_t sxe2_sched_reset(struct rte_eth_dev *dev)
+{
+ int32_t ret = 0;
+
+ if (dev->data->dev_started) {
+ PMD_LOG_ERR(DRV, "Device failed to Stop.");
+ ret = -EPERM;
+ goto l_end;
+ }
+
+ ret = sxe2_tm_conf_reset(dev);
+ if (ret)
+ goto l_end;
+
+ ret = sxe2_sched_uinit(dev);
+ if (ret)
+ goto l_end;
+
+ ret = sxe2_sched_init(dev);
+ if (ret)
+ goto l_end;
+
+l_end:
+ return ret;
+}
+
static int32_t sxe2_dev_init(struct rte_eth_dev *dev,
struct sxe2_dev_kvargs_info *kvargs __rte_unused)
{
@@ -1683,6 +1953,12 @@ static int32_t sxe2_dev_init(struct rte_eth_dev *dev,
sxe2_init_ptype_tbl(dev);
+ ret = sxe2_args_parse(dev, kvargs);
+ if (ret) {
+ PMD_LOG_ERR(INIT, "Failed to parse devargs, ret=%d", ret);
+ goto l_end;
+ }
+
ret = sxe2_hw_init(dev);
if (ret) {
PMD_LOG_ERR(INIT, "Failed to initialize hw, ret=[%d]", ret);
@@ -1749,6 +2025,12 @@ static int32_t sxe2_dev_init(struct rte_eth_dev *dev,
goto init_flow_err;
}
+ ret = sxe2_fc_state_init(dev);
+ if (ret) {
+ PMD_LOG_ERR(INIT, "Failed to init fc state, ret=%d", ret);
+ goto init_fc_state_err;
+ }
+
ret = sxe2_sched_init(dev);
if (ret) {
PMD_LOG_ERR(INIT, "Failed to init sched, ret=%d", ret);
@@ -1772,6 +2054,8 @@ static int32_t sxe2_dev_init(struct rte_eth_dev *dev,
init_xstats_err:
(void)sxe2_sched_uinit(dev);
init_sched_err:
+ sxe2_fc_state_uinit(dev);
+init_fc_state_err:
(void)sxe2_flow_uninit(dev);
init_flow_err:
init_rss_err:
@@ -1817,6 +2101,7 @@ static int32_t sxe2_dev_close(struct rte_eth_dev *dev)
sxe2_eth_uinit(dev);
sxe2_dev_pci_map_uinit(dev);
sxe2_free_repr_info(dev);
+ sxe2_fc_state_uinit(dev);
l_end:
return 0;
diff --git a/drivers/net/sxe2/sxe2_ethdev.h b/drivers/net/sxe2/sxe2_ethdev.h
index 510f5139c3..14ad26b439 100644
--- a/drivers/net/sxe2/sxe2_ethdev.h
+++ b/drivers/net/sxe2/sxe2_ethdev.h
@@ -311,6 +311,11 @@ struct sxe2_filter_context {
bool cur_l2_config;
};
+struct sxe2_fc_state_ctxt {
+ uint8_t curr_state;
+ uint8_t cfg_state;
+};
+
struct sxe2_adapter {
struct sxe2_common_device *cdev;
struct sxe2_dev_info dev_info;
@@ -332,6 +337,7 @@ struct sxe2_adapter {
struct sxe2_security_ctx security_ctx;
struct sxe2_repr_context repr_ctxt;
struct sxe2_switchdev_info switchdev_info;
+ struct sxe2_fc_state_ctxt fc_state_ctx;
bool rule_started;
bool flow_isolated;
bool flow_isolate_cfg;
@@ -359,6 +365,8 @@ bool sxe2_ethdev_check(struct rte_eth_dev *dev);
uint32_t sxe2_sched_mode_get(struct sxe2_adapter *adapter);
+int32_t sxe2_sched_reset(struct rte_eth_dev *dev);
+
struct sxe2_pci_map_bar_info *sxe2_dev_get_bar_info(struct sxe2_adapter *adapter,
enum sxe2_pci_map_resource res_type);
diff --git a/drivers/net/sxe2/sxe2_irq.c b/drivers/net/sxe2/sxe2_irq.c
index c26098ef3a..3306504761 100644
--- a/drivers/net/sxe2/sxe2_irq.c
+++ b/drivers/net/sxe2/sxe2_irq.c
@@ -10,6 +10,7 @@
#include <rte_alarm.h>
#include <fcntl.h>
#include <rte_stdatomic.h>
+#include <rte_common.h>
#include "sxe2_ethdev.h"
#include "sxe2_irq.h"
@@ -47,6 +48,31 @@ static struct sxe2_event_handler event_handler = {
static RTE_ATOMIC(uint32_t)event_thread_run;
+static int32_t sxe2_fc_state_callback(struct rte_eth_dev *dev)
+{
+ struct sxe2_adapter *adapter = SXE2_DEV_PRIVATE_TO_ADAPTER(dev);
+ struct sxe2_drv_vsi_fc_get_resp fc_resp = {0};
+ int32_t ret;
+
+ if (!(adapter->cap_flags & SXE2_DEV_CAPS_OFFLOAD_FC_STATE)) {
+ ret = 0;
+ goto l_end;
+ }
+ ret = sxe2_drv_fc_state_get(adapter, &fc_resp);
+ if (ret) {
+ PMD_LOG_ERR(INIT, "Failed to get fc state, ret=[%d]", ret);
+ goto l_end;
+ }
+ adapter->fc_state_ctx.cfg_state = fc_resp.fc_enable;
+ if (dev->data->dev_started) {
+ PMD_LOG_NOTICE(DRV, "Interrupt event: FC status changed."
+ "cfg_state:%u curr_state:%u",
+ adapter->fc_state_ctx.cfg_state,
+ adapter->fc_state_ctx.curr_state);
+ }
+l_end:
+ return ret;
+}
static void sxe2_event_irq_common_handler(struct sxe2_adapter *adapter, uint64_t oicr)
{
@@ -68,6 +94,10 @@ static void sxe2_event_irq_common_handler(struct sxe2_adapter *adapter, uint64_t
PMD_DEV_LOG_INFO(adapter, DRV, "event notify legacy");
(void)sxe2_switchdev_notify_callback(adapter, false);
}
+ if (oicr & RTE_BIT32(SXE2_COM_FC_ST_CHANGE)) {
+ PMD_DEV_LOG_INFO(adapter, DRV, "fc event notify legacy");
+ (void)sxe2_fc_state_callback(dev);
+ }
}
static uint32_t sxe2_event_intr_handle(void *param __rte_unused)
@@ -436,7 +466,7 @@ int32_t sxe2_intr_init(struct rte_eth_dev *dev)
{
struct sxe2_adapter *adapter =
SXE2_DEV_PRIVATE_TO_ADAPTER(dev);
- struct rte_pci_device *pci_dev = SXE2_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = container_of(dev->device, struct rte_pci_device, device);
struct rte_intr_handle *reset_handle = NULL;
int32_t ofd = -1;
int32_t rfd = -1;
@@ -518,7 +548,7 @@ void sxe2_intr_uninit(struct rte_eth_dev *dev)
{
struct sxe2_adapter *adapter =
SXE2_DEV_PRIVATE_TO_ADAPTER(dev);
- struct rte_pci_device *pci_dev = SXE2_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = container_of(dev->device, struct rte_pci_device, device);
sxe2_reset_intr_unregister(dev);
sxe2_intr_handler_destroy(adapter->irq_ctxt.reset_handle,
diff --git a/drivers/net/sxe2/sxe2_rx.c b/drivers/net/sxe2/sxe2_rx.c
index 79e65cfbf1..b5dd9950f0 100644
--- a/drivers/net/sxe2/sxe2_rx.c
+++ b/drivers/net/sxe2/sxe2_rx.c
@@ -467,12 +467,24 @@ int32_t __rte_cold sxe2_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queu
int32_t __rte_cold sxe2_rxqs_all_start(struct rte_eth_dev *dev)
{
struct rte_eth_dev_data *data = dev->data;
+ struct sxe2_adapter *adapter = SXE2_DEV_PRIVATE_TO_ADAPTER(dev);
+ struct sxe2_drv_vsi_fc_get_resp fc_resp = {0};
struct sxe2_rx_queue *rxq;
uint16_t nb_rxq;
uint16_t nb_started_rxq;
int32_t ret;
PMD_INIT_FUNC_TRACE();
+ if (adapter->cap_flags & SXE2_DEV_CAPS_OFFLOAD_FC_STATE) {
+ ret = sxe2_drv_fc_state_get(adapter, &fc_resp);
+ if (ret) {
+ PMD_LOG_ERR(RX, "Failed to get fc state, ret=[%d]", ret);
+ goto l_end;
+ }
+ adapter->fc_state_ctx.cfg_state = fc_resp.fc_enable;
+ adapter->fc_state_ctx.curr_state = adapter->fc_state_ctx.cfg_state;
+ }
+
for (nb_rxq = 0; nb_rxq < data->nb_rx_queues; nb_rxq++) {
rxq = dev->data->rx_queues[nb_rxq];
if (!rxq || rxq->rx_deferred_start)
diff --git a/drivers/net/sxe2/sxe2_testpmd.c b/drivers/net/sxe2/sxe2_testpmd.c
new file mode 100644
index 0000000000..5792058212
--- /dev/null
+++ b/drivers/net/sxe2/sxe2_testpmd.c
@@ -0,0 +1,733 @@
+
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C), 2025, Wuxi Stars Micro System Technologies Co., Ltd.
+ */
+
+#ifndef SXE2_TEST
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <stdlib.h>
+#include <testpmd.h>
+
+#include "sxe2_common_log.h"
+#include "sxe2_testpmd_lib.h"
+
+#define SXE2_SWITCH_BUFF_SIZE (4 * 1024 * 1024)
+
+struct cmd_stats_info_show_result {
+ cmdline_fixed_string_t sxe2;
+ cmdline_fixed_string_t show;
+ cmdline_fixed_string_t stats;
+ portid_t port_id;
+};
+cmdline_parse_token_string_t cmd_stats_info_sxe2 =
+ TOKEN_STRING_INITIALIZER(struct cmd_stats_info_show_result, sxe2, "sxe2");
+cmdline_parse_token_string_t cmd_stats_info_show =
+ TOKEN_STRING_INITIALIZER(struct cmd_stats_info_show_result, show, "show");
+cmdline_parse_token_string_t cmd_stats_info_stats =
+ TOKEN_STRING_INITIALIZER(struct cmd_stats_info_show_result, stats, "stats");
+cmdline_parse_token_num_t cmd_stats_info_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_stats_info_show_result, port_id, RTE_UINT16);
+
+struct cmd_flow_rule_result {
+ cmdline_fixed_string_t sxe2;
+ cmdline_fixed_string_t flow;
+ cmdline_fixed_string_t rule;
+ cmdline_fixed_string_t dump;
+ portid_t port_id;
+};
+cmdline_parse_token_string_t cmd_flow_rule_sxe2 =
+ TOKEN_STRING_INITIALIZER(struct cmd_flow_rule_result, sxe2, "sxe2");
+cmdline_parse_token_string_t cmd_flow_rule_flow =
+ TOKEN_STRING_INITIALIZER(struct cmd_flow_rule_result, flow, "flow");
+cmdline_parse_token_string_t cmd_flow_rule_rule =
+ TOKEN_STRING_INITIALIZER(struct cmd_flow_rule_result, rule, "rule");
+cmdline_parse_token_string_t cmd_flow_rule_dmp =
+ TOKEN_STRING_INITIALIZER(struct cmd_flow_rule_result, dump, "dump");
+cmdline_parse_token_num_t cmd_flow_rule_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_flow_rule_result, port_id, RTE_UINT16);
+
+struct cmd_udp_tunnel {
+ cmdline_fixed_string_t sxe2;
+ cmdline_fixed_string_t tunnel_type;
+ cmdline_fixed_string_t action;
+ cmdline_fixed_string_t udp_tunnel_port;
+ uint16_t udp_port;
+ portid_t port_id;
+};
+
+cmdline_parse_token_string_t cmd_udp_tunnel_sxe2 =
+ TOKEN_STRING_INITIALIZER(struct cmd_udp_tunnel, sxe2, "sxe2");
+cmdline_parse_token_string_t cmd_udp_tunnel_action =
+ TOKEN_STRING_INITIALIZER(struct cmd_udp_tunnel, action, "add#rm#show");
+cmdline_parse_token_string_t cmd_udp_tunnel_udp_tunnel_port =
+ TOKEN_STRING_INITIALIZER(struct cmd_udp_tunnel, udp_tunnel_port, "udp_tunnel_port");
+cmdline_parse_token_string_t cmd_udp_tunnel_tunnel_type =
+ TOKEN_STRING_INITIALIZER(struct cmd_udp_tunnel,
+ tunnel_type, "vxlan#vxlan-gpe#geneve#gtp-c#gtp-u#pfcp#ecpri#mpls#nvgre#l2tp#teredo");
+cmdline_parse_token_num_t cmd_udp_tunnel_udp_port =
+ TOKEN_NUM_INITIALIZER(struct cmd_udp_tunnel, udp_port, RTE_UINT16);
+cmdline_parse_token_num_t cmd_udp_tunnel_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_udp_tunnel, port_id, RTE_UINT16);
+
+struct cmd_sched_result {
+ cmdline_fixed_string_t sxe2;
+ cmdline_fixed_string_t sched;
+ cmdline_fixed_string_t reset;
+ portid_t port_id;
+};
+
+cmdline_parse_token_string_t cmd_sched_sxe2 =
+ TOKEN_STRING_INITIALIZER(struct cmd_sched_result, sxe2, "sxe2");
+cmdline_parse_token_string_t cmd_sched_sched =
+ TOKEN_STRING_INITIALIZER(struct cmd_sched_result, sched, "sched");
+cmdline_parse_token_string_t cmd_sched_reset =
+ TOKEN_STRING_INITIALIZER(struct cmd_sched_result, reset, "reset");
+cmdline_parse_token_num_t cmd_sched_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_sched_result, port_id, RTE_UINT16);
+
+struct cmd_ipsec_result {
+ cmdline_fixed_string_t sxe2;
+ cmdline_fixed_string_t engin;
+ cmdline_fixed_string_t dir;
+ cmdline_fixed_string_t op;
+ portid_t port_id;
+ uint16_t session_id;
+ cmdline_fixed_string_t encrypt_algo;
+ cmdline_fixed_string_t encrypt_key;
+ cmdline_fixed_string_t auth_algo;
+ cmdline_fixed_string_t auth_key;
+ cmdline_fixed_string_t dst_ip;
+ uint16_t sport;
+ uint16_t dport;
+ uint32_t spi;
+};
+cmdline_parse_token_string_t cmd_ipsec_mgt_sxe2 =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_result, sxe2, "sxe2");
+cmdline_parse_token_string_t cmd_ipsec_mgt_module =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_result, engin, "ipsec");
+cmdline_parse_token_string_t cmd_ipsec_mgt_dir =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_result, dir, "egress#ingress");
+cmdline_parse_token_string_t cmd_ipsec_mgt_op =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_result, op, "add#rm#show");
+cmdline_parse_token_num_t cmd_ipsec_mgt_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_ipsec_result, port_id, RTE_UINT16);
+cmdline_parse_token_num_t cmd_ipsec_mgt_session_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_ipsec_result, session_id, RTE_UINT16);
+cmdline_parse_token_string_t cmd_ipsec_mgt_encrypt_algo =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_result, encrypt_algo, "aes-cbc#sm4-cbc#null");
+cmdline_parse_token_string_t cmd_ipsec_mgt_encrypt_key =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_result, encrypt_key, NULL);
+cmdline_parse_token_string_t cmd_ipsec_mgt_auth_algo =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_result, auth_algo, "sha-hmac#sm3-hmac#null");
+cmdline_parse_token_string_t cmd_ipsec_mgt_auth_key =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_result, auth_key, NULL);
+cmdline_parse_token_string_t cmd_ipsec_mgt_dst_ip =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_result, dst_ip, NULL);
+cmdline_parse_token_num_t cmd_ipsec_mgt_sport =
+ TOKEN_NUM_INITIALIZER(struct cmd_ipsec_result, sport, RTE_UINT16);
+cmdline_parse_token_num_t cmd_ipsec_mgt_dport =
+ TOKEN_NUM_INITIALIZER(struct cmd_ipsec_result, dport, RTE_UINT16);
+cmdline_parse_token_num_t cmd_ipsec_mgt_spi =
+ TOKEN_NUM_INITIALIZER(struct cmd_ipsec_result, spi, RTE_UINT32);
+
+struct cmd_ipsec_set_result {
+ cmdline_fixed_string_t sxe2;
+ cmdline_fixed_string_t engin;
+ cmdline_fixed_string_t op;
+ cmdline_fixed_string_t type;
+ portid_t port_id;
+ uint16_t conf_value;
+};
+cmdline_parse_token_string_t cmd_ipsec_set_sxe2 =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_set_result, sxe2, "sxe2");
+cmdline_parse_token_string_t cmd_ipsec_set_module =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_set_result, engin, "ipsec");
+cmdline_parse_token_string_t cmd_ipsec_set_op =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_set_result, op, "set#get");
+cmdline_parse_token_string_t cmd_ipsec_set_type =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_set_result, type, "session-id#esp-hdr-offset");
+cmdline_parse_token_num_t cmd_ipsec_set_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_ipsec_set_result, port_id, RTE_UINT16);
+cmdline_parse_token_num_t cmd_ipsec_set_value =
+ TOKEN_NUM_INITIALIZER(struct cmd_ipsec_set_result, conf_value, RTE_UINT16);
+
+struct cmd_ipsec_flush_result {
+ cmdline_fixed_string_t sxe2;
+ cmdline_fixed_string_t engin;
+ cmdline_fixed_string_t op;
+ portid_t port_id;
+};
+cmdline_parse_token_string_t cmd_ipsec_flush_sxe2 =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_flush_result, sxe2, "sxe2");
+cmdline_parse_token_string_t cmd_ipsec_flush_module =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_flush_result, engin, "ipsec");
+cmdline_parse_token_string_t cmd_ipsec_flush_op =
+ TOKEN_STRING_INITIALIZER(struct cmd_ipsec_flush_result, op, "flush");
+cmdline_parse_token_num_t cmd_ipsec_flush_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_ipsec_flush_result, port_id, RTE_UINT16);
+
+struct cmd_inject_irq {
+ cmdline_fixed_string_t sxe2;
+ cmdline_fixed_string_t inject;
+ cmdline_fixed_string_t irq;
+ portid_t port_id;
+ cmdline_fixed_string_t type;
+};
+cmdline_parse_token_string_t cmd_inject_irq_sxe2 =
+ TOKEN_STRING_INITIALIZER(struct cmd_inject_irq, sxe2, "sxe2");
+cmdline_parse_token_string_t cmd_inject_irq_inject =
+ TOKEN_STRING_INITIALIZER(struct cmd_inject_irq, inject, "inject");
+cmdline_parse_token_string_t cmd_inject_irq_irq =
+ TOKEN_STRING_INITIALIZER(struct cmd_inject_irq, irq, "irq");
+cmdline_parse_token_num_t cmd_inject_irq_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_inject_irq, port_id, RTE_UINT16);
+cmdline_parse_token_string_t cmd_inject_irq_type =
+ TOKEN_STRING_INITIALIZER(struct cmd_inject_irq, type, "reset#lsc");
+
+static void cmd_dump_flow_rule_parsed(void *parsed_result,
+ struct cmdline *cl,
+ __rte_unused void *data)
+{
+ struct cmd_flow_rule_result *res = parsed_result;
+ int ret = -1;
+
+ ret = sxe2_flow_rule_dump(res->port_id, cl);
+ switch (ret) {
+ case 0:
+ break;
+ case -EINVAL:
+ cmdline_printf(cl, "Invalid parameters.\n");
+ break;
+ case -ENODEV:
+ cmdline_printf(cl, "Device doesn't support\n");
+ break;
+ default:
+ cmdline_printf(cl,
+ "Failed to switch rule dump,"
+ " error: (%s)\n",
+ strerror(-ret));
+ }
+}
+
+static void cmd_udp_tunnel_set_parsed(void *parsed_result,
+ struct cmdline *cl,
+ __rte_unused void *data)
+{
+ struct cmd_udp_tunnel *res = parsed_result;
+ int32_t ret = -1;
+ uint8_t action;
+ const char *action_str[SXE2_TESTPMD_CMD_UDP_TUNNEL_MAX] = {
+ [SXE2_TESTPMD_CMD_UDP_TUNNEL_ADD] = "add",
+ [SXE2_TESTPMD_CMD_UDP_TUNNEL_DEL] = "rm",
+ [SXE2_TESTPMD_CMD_UDP_TUNNEL_GET] = "show"};
+
+ for (action = 0; action < SXE2_TESTPMD_CMD_UDP_TUNNEL_MAX; action++)
+ if (!strcmp(res->action, action_str[action]))
+ break;
+
+ if (action >= SXE2_TESTPMD_CMD_UDP_TUNNEL_MAX) {
+ cmdline_printf(cl, "Invalid action!\n");
+ return;
+ }
+
+ ret = sxe2_udp_tunnel_operations(res->port_id, cl, action,
+ res->udp_port,
+ res->tunnel_type);
+ if (ret)
+ cmdline_printf(cl, "%s udp tunnel port failed, ret = %d\n",
+ action_str[action], ret);
+}
+
+static void cmd_dump_stats_info_parsed(void *parsed_result,
+ struct cmdline *cl,
+ __rte_unused void *data)
+{
+ struct cmd_stats_info_show_result *res = parsed_result;
+ int ret = -1;
+
+ ret = sxe2_stats_info_show(res->port_id);
+ switch (ret) {
+ case 0:
+ break;
+ case -EINVAL:
+ cmdline_printf(cl, "Invalid parameters.\n");
+ break;
+ case -ENODEV:
+ cmdline_printf(cl, "Device doesn't support\n");
+ break;
+ default:
+ cmdline_printf(cl,
+ "Failed to show stats info,"
+ " error: (%s)\n", strerror(-ret));
+ }
+}
+
+static uint8_t cmd_ipsec_op_get(char *op)
+{
+ uint8_t i;
+ const char *op_type[SXE2_TESTPMD_CMD_IPSEC_OP_MAX] = {
+ [SXE2_TESTPMD_CMD_IPSEC_OP_ADD] = "add",
+ [SXE2_TESTPMD_CMD_IPSEC_OP_RM] = "rm",
+ [SXE2_TESTPMD_CMD_IPSEC_OP_SHOW] = "show",
+ };
+
+ for (i = 0; i < SXE2_TESTPMD_CMD_IPSEC_OP_MAX; i++) {
+ if (!strcmp(op, op_type[i]))
+ break;
+ }
+
+ return i;
+}
+
+static uint8_t cmd_ipsec_dir_get(char *dir)
+{
+ uint8_t i;
+ const char *dir_type[SXE2_TESTPMD_CMD_IPSEC_DIR_MAX] = {
+ [SXE2_TESTPMD_CMD_IPSEC_DIR_EGRESS] = "egress",
+ [SXE2_TESTPMD_CMD_IPSEC_DIR_INGRESS] = "ingress"
+ };
+
+ for (i = 0; i < SXE2_TESTPMD_CMD_IPSEC_DIR_MAX; i++) {
+ if (!strcmp(dir, dir_type[i]))
+ break;
+ }
+
+ return i;
+}
+
+static int sxe2_hex_to_val(char c)
+{
+ int val = 0;
+
+ if (c >= '0' && c <= '9')
+ val = c - '0';
+ if (c >= 'A' && c <= 'F')
+ val = 10 + c - 'A';
+ if (c >= 'a' && c <= 'f')
+ val = 10 + c - 'a';
+ return val;
+}
+
+static void sxe2_hex_to_bytes(uint8_t *enc_key, char *hex_str, uint8_t len)
+{
+ uint8_t i;
+ int high = 0;
+ int low = 0;
+
+ for (i = 0; i < len; i++) {
+ high = sxe2_hex_to_val(hex_str[2 * i]);
+ low = sxe2_hex_to_val(hex_str[2 * i + 1]);
+ enc_key[i] = (high << 4) | low;
+ }
+}
+
+static int32_t cmd_ipsec_add_param_fill(struct sxe2_ipsec_conf_param *param,
+ struct cmdline *cl,
+ struct cmd_ipsec_result *res)
+{
+ uint8_t i;
+ uint8_t j;
+ int32_t ret = -1;
+ const char *encrypt_algo[SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_MAX] = {
+ [SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_AES_CBC] = "aes-cbc",
+ [SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_SM4_CBC] = "sm4-cbc",
+ [SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_NULL] = "null"
+ };
+
+ const char *auth_algo[SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_MAX] = {
+ [SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SHA_HMAC] = "sha-hmac",
+ [SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SM3_HMAC] = "sm3-hmac",
+ [SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL] = "null"
+ };
+
+ for (i = 0; i < SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_MAX; i++)
+ if (!strcmp(res->encrypt_algo, encrypt_algo[i]))
+ break;
+
+ if (i >= SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_MAX) {
+ cmdline_printf(cl, "Invalid ipsec encrypt algo: %s!\n", res->encrypt_algo);
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ for (j = 0; j < SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_MAX; j++) {
+ if (!strcmp(res->auth_algo, auth_algo[j]))
+ break;
+ }
+
+
+ if (j >= SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_MAX) {
+ cmdline_printf(cl, "Invalid ipsec auth algo: %s!\n", res->auth_algo);
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ param->encrypt_algo = i;
+ param->auth_algo = j;
+ if (param->encrypt_algo == SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_SM4_CBC)
+ param->enc_len = 16;
+ else
+ param->enc_len = 32;
+
+ sxe2_hex_to_bytes(param->enc_key, res->encrypt_key, param->enc_len);
+ if (param->auth_algo != SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL) {
+ param->auth_len = 32;
+ sxe2_hex_to_bytes(param->auth_key, res->auth_key, param->auth_len);
+ }
+
+ ret = 0;
+
+l_end:
+ return ret;
+}
+
+static int32_t cmd_ipsec_egress_op_parsed(struct sxe2_ipsec_conf_param *param,
+ struct cmdline *cl,
+ struct cmd_ipsec_result *res)
+{
+ int32_t ret = -1;
+
+ switch (param->op) {
+ case SXE2_TESTPMD_CMD_IPSEC_OP_ADD:
+ ret = cmd_ipsec_add_param_fill(param, cl, res);
+ if (ret)
+ goto l_end;
+ ret = sxe2_ipsec_egress_create(param, cl);
+ break;
+ case SXE2_TESTPMD_CMD_IPSEC_OP_RM:
+ param->session_id = res->session_id;
+ ret = sxe2_ipsec_egress_destroy(param, cl);
+ break;
+ case SXE2_TESTPMD_CMD_IPSEC_OP_SHOW:
+ ret = sxe2_ipsec_egress_show(param, cl);
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+l_end:
+ return ret;
+}
+
+static int32_t cmd_ipsec_ip_addr_parsed(struct sxe2_ipsec_conf_param *param,
+ struct cmdline *cl,
+ struct cmd_ipsec_result *res)
+{
+ int32_t ret = -1;
+ struct in_addr addr4;
+ struct in6_addr addr6;
+
+ if (inet_pton(AF_INET, res->dst_ip, &addr4) == 1) {
+ param->ip_addr.type = RTE_SECURITY_IPSEC_TUNNEL_IPV4;
+ param->ip_addr.dst_ipv4 = addr4.s_addr;
+ ret = 0;
+ } else if (inet_pton(AF_INET6, res->dst_ip, &addr6) == 1) {
+ param->ip_addr.type = RTE_SECURITY_IPSEC_TUNNEL_IPV6;
+ memcpy(¶m->ip_addr.dst_ipv6, &addr6, sizeof(param->ip_addr.dst_ipv6));
+ ret = 0;
+ } else {
+ cmdline_printf(cl, "Invalid ip address: %s!\n", res->dst_ip);
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+l_end:
+ return ret;
+}
+
+static int32_t cmd_ipsec_ingress_op_parsed(struct sxe2_ipsec_conf_param *param,
+ struct cmdline *cl,
+ struct cmd_ipsec_result *res)
+{
+ int32_t ret = -1;
+
+ switch (param->op) {
+ case SXE2_TESTPMD_CMD_IPSEC_OP_ADD:
+ ret = cmd_ipsec_add_param_fill(param, cl, res);
+ if (ret)
+ goto l_end;
+ param->sport = htons(res->sport);
+ param->dport = htons(res->dport);
+ param->spi = htonl(res->spi);
+ ret = cmd_ipsec_ip_addr_parsed(param, cl, res);
+ if (ret)
+ goto l_end;
+ ret = sxe2_ipsec_ingress_create(param, cl);
+ break;
+ case SXE2_TESTPMD_CMD_IPSEC_OP_RM:
+ param->session_id = res->session_id;
+ ret = sxe2_ipsec_ingress_destroy(param, cl);
+ break;
+ case SXE2_TESTPMD_CMD_IPSEC_OP_SHOW:
+ ret = sxe2_ipsec_ingress_show(param, cl);
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+l_end:
+ return ret;
+}
+
+static int32_t cmd_ipsec_dir_parsed(struct sxe2_ipsec_conf_param *param,
+ struct cmdline *cl,
+ struct cmd_ipsec_result *res)
+{
+ int32_t ret = -1;
+
+ switch (param->dir) {
+ case SXE2_TESTPMD_CMD_IPSEC_DIR_EGRESS:
+ ret = cmd_ipsec_egress_op_parsed(param, cl, res);
+ break;
+ case SXE2_TESTPMD_CMD_IPSEC_DIR_INGRESS:
+ ret = cmd_ipsec_ingress_op_parsed(param, cl, res);
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void cmd_ipsec_mgt_parsed(void *parsed_result,
+ struct cmdline *cl,
+ __rte_unused void *data)
+{
+ struct cmd_ipsec_result *res = parsed_result;
+ struct sxe2_ipsec_conf_param param;
+ int32_t ret = -1;
+ uint8_t dir = 0;
+ uint8_t op = 0;
+
+ dir = cmd_ipsec_dir_get(res->dir);
+ if (dir >= SXE2_TESTPMD_CMD_IPSEC_DIR_MAX) {
+ cmdline_printf(cl, "Invalid ipsec direction: %s!\n", res->dir);
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ op = cmd_ipsec_op_get(res->op);
+ if (op >= SXE2_TESTPMD_CMD_IPSEC_OP_MAX) {
+ cmdline_printf(cl, "Invalid ipsec operation: %s!\n", res->op);
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ memset(¶m, 0, sizeof(struct sxe2_ipsec_conf_param));
+ param.dir = dir;
+ param.op = op;
+ param.port_id = res->port_id;
+ ret = cmd_ipsec_dir_parsed(¶m, cl, res);
+
+ if (ret)
+ cmdline_printf(cl, "Command execute failed, ret = %d\n", ret);
+
+l_end:
+ return;
+}
+
+static void cmd_ipsec_set_parsed(void *parsed_result,
+ struct cmdline *cl,
+ __rte_unused void *data)
+{
+ struct cmd_ipsec_set_result *res = parsed_result;
+ int32_t ret = -1;
+
+ if (!strcmp(res->op, "set"))
+ ret = sxe2_ipsec_conf_set(res->port_id, cl, res->type, res->conf_value);
+ else if (!strcmp(res->op, "get"))
+ ret = sxe2_ipsec_conf_get(res->port_id, cl, res->type);
+ else
+ cmdline_printf(cl, "Invalid op: %s\n", res->op);
+
+ if (ret)
+ cmdline_printf(cl, "Command execute failed, ret = %d\n", ret);
+}
+
+static void cmd_ipsec_flush_parsed(void *parsed_result,
+ struct cmdline *cl,
+ __rte_unused void *data)
+{
+ struct cmd_ipsec_flush_result *res = parsed_result;
+ int32_t ret = -1;
+
+ ret = sxe2_ipsec_flush(res->port_id, cl);
+
+ if (ret)
+ cmdline_printf(cl, "Command execute failed, ret = %d\n", ret);
+}
+
+cmdline_parse_inst_t cmd_flow_rule_dump = {
+ .f = cmd_dump_flow_rule_parsed,
+ .data = NULL,
+ .help_str = "sxe2 flow rule dump <port_id>",
+ .tokens = {
+ (void *)&cmd_flow_rule_sxe2,
+ (void *)&cmd_flow_rule_flow,
+ (void *)&cmd_flow_rule_rule,
+ (void *)&cmd_flow_rule_dmp,
+ (void *)&cmd_flow_rule_port_id,
+ NULL,
+ },
+};
+
+cmdline_parse_inst_t cmd_udp_tunnel_set = {
+ .f = cmd_udp_tunnel_set_parsed,
+ .data = NULL,
+ .help_str = "sxe2 <port_id> udp_tunnel_port add|rm|show "
+ "vxlan|vxlan-gpe|geneve|gtp-c|gtp-u|pfcp|ecpri|mpls|nvgre|l2tp|teredo <udp_port>",
+ .tokens = {
+ (void *)&cmd_udp_tunnel_sxe2,
+ (void *)&cmd_udp_tunnel_port_id,
+ (void *)&cmd_udp_tunnel_udp_tunnel_port,
+ (void *)&cmd_udp_tunnel_action,
+ (void *)&cmd_udp_tunnel_tunnel_type,
+ (void *)&cmd_udp_tunnel_udp_port,
+ NULL,
+ },
+};
+
+cmdline_parse_inst_t cmd_stats_mgt = {
+ .f = cmd_dump_stats_info_parsed,
+ .data = NULL,
+ .help_str = "sxe2 show stats <port_id>",
+ .tokens = {
+ (void *)&cmd_stats_info_sxe2,
+ (void *)&cmd_stats_info_show,
+ (void *)&cmd_stats_info_stats,
+ (void *)&cmd_stats_info_port_id,
+ NULL,
+ },
+};
+
+static void cmd_sched_reset_cfg(void *parsed_result,
+ struct cmdline *cl,
+ __rte_unused void *data)
+{
+ struct cmd_sched_result *res = parsed_result;
+ int32_t ret = -1;
+
+ ret = sxe2_testpmd_sched_reset(res->port_id);
+ switch (ret) {
+ case 0:
+ break;
+ case -EINVAL:
+ cmdline_printf(cl, "invalid sched ops\n");
+ break;
+ case -ENOTSUP:
+ cmdline_printf(cl, "function not implemented\n");
+ break;
+ default:
+ cmdline_printf(cl, "programming error: (%s)\n",
+ strerror(-ret));
+ }
+}
+
+cmdline_parse_inst_t cmd_sched_reset_cmd = {
+ .f = cmd_sched_reset_cfg,
+ .data = NULL,
+ .help_str = "sxe2 sched reset <port_id>",
+ .tokens = {
+ (void *)&cmd_sched_sxe2,
+ (void *)&cmd_sched_sched,
+ (void *)&cmd_sched_reset,
+ (void *)&cmd_sched_port_id,
+ NULL,
+ },
+};
+
+cmdline_parse_inst_t cmd_ipsec_mgt = {
+ .f = cmd_ipsec_mgt_parsed,
+ .data = NULL,
+ .help_str = "sxe2 ipsec egress|ingress add|rm|show "
+ "<port_id> <session_id> aes-cbc|sm4-cbc|null <encrypt_key> sha-hmac|sm3-hmac|null "
+ "<auth_key> <dst_ip> <sport> <dport> <spi>",
+ .tokens = {
+ (void *)&cmd_ipsec_mgt_sxe2,
+ (void *)&cmd_ipsec_mgt_module,
+ (void *)&cmd_ipsec_mgt_dir,
+ (void *)&cmd_ipsec_mgt_op,
+ (void *)&cmd_ipsec_mgt_port_id,
+ (void *)&cmd_ipsec_mgt_session_id,
+ (void *)&cmd_ipsec_mgt_encrypt_algo,
+ (void *)&cmd_ipsec_mgt_encrypt_key,
+ (void *)&cmd_ipsec_mgt_auth_algo,
+ (void *)&cmd_ipsec_mgt_auth_key,
+ (void *)&cmd_ipsec_mgt_dst_ip,
+ (void *)&cmd_ipsec_mgt_sport,
+ (void *)&cmd_ipsec_mgt_dport,
+ (void *)&cmd_ipsec_mgt_spi,
+ NULL,
+ },
+};
+
+cmdline_parse_inst_t cmd_ipsec_set = {
+ .f = cmd_ipsec_set_parsed,
+ .data = NULL,
+ .help_str = "sxe2 ipsec set|get esp-hdr-offset|session-id <port_id> <value>",
+ .tokens = {
+ (void *)&cmd_ipsec_set_sxe2,
+ (void *)&cmd_ipsec_set_module,
+ (void *)&cmd_ipsec_set_op,
+ (void *)&cmd_ipsec_set_type,
+ (void *)&cmd_ipsec_set_port_id,
+ (void *)&cmd_ipsec_set_value,
+ NULL,
+ },
+};
+
+cmdline_parse_inst_t cmd_ipsec_flush = {
+ .f = cmd_ipsec_flush_parsed,
+ .data = NULL,
+ .help_str = "sxe2 ipsec flush <port_id>.\n",
+ .tokens = {
+ (void *)&cmd_ipsec_flush_sxe2,
+ (void *)&cmd_ipsec_flush_module,
+ (void *)&cmd_ipsec_flush_op,
+ (void *)&cmd_ipsec_flush_port_id,
+ NULL,
+ },
+};
+
+static struct testpmd_driver_commands sxe2_cmds = {
+ .commands = {
+ {
+ &cmd_udp_tunnel_set,
+ "sxe2 udp tunnel port set.\n"
+ "Add or remove a customed udp port for specific tunnel protocol\n\n",
+ },
+ {
+ &cmd_sched_reset_cmd,
+ "sxe2 sched reset <port_id>.\n"
+ "Reset sched node on the port\n\n",
+ },
+ {
+ &cmd_stats_mgt,
+ "sxe2 show stats.\n"
+ "Dump a runtime sxe2 dev stats on a port\n\n",
+ },
+ {
+ &cmd_ipsec_mgt,
+ "sxe2 ipsec <dir> <op> <port_id> <session_id> <encrypt_algo> <encrypt_key>"
+ "<encrypt_len> <auth_algo> <auth_key> <auth_len> <dst_ip> <sport> <dport> <spi>.\n"
+ "Create/query/remove ipsec security session\n\n",
+ },
+ {
+ &cmd_ipsec_set,
+ "sxe2 ipsec set <port_id> <session_id> <esp_hdr_offset>.\n"
+ "Set enabled tx session id or esp offset.\n\n",
+ },
+ {
+ &cmd_ipsec_flush,
+ "sxe2 ipsec flush <port_id>.\n"
+ "Flush ipsec all configurations\n\n",
+ },
+ { NULL, NULL},
+ },
+};
+TESTPMD_ADD_DRIVER_COMMANDS(sxe2_cmds)
+#endif
diff --git a/drivers/net/sxe2/sxe2_testpmd_lib.c b/drivers/net/sxe2/sxe2_testpmd_lib.c
new file mode 100644
index 0000000000..ab2530ffe6
--- /dev/null
+++ b/drivers/net/sxe2/sxe2_testpmd_lib.c
@@ -0,0 +1,969 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C), 2025, Wuxi Stars Micro System Technologies Co., Ltd.
+ */
+
+#include <rte_bus.h>
+#include <eal_export.h>
+
+#include "sxe2_common_log.h"
+#include "sxe2_ethdev.h"
+#include "sxe2_stats.h"
+#include "sxe2_testpmd_lib.h"
+
+struct rte_mempool *g_sess_pool;
+
+bool g_sxe2_ipsec_mgt_init;
+struct sxe2_ipsec_session_mgt g_tx_session[SXE2_IPSEC_PORT_MAX][SXE2_IPSEC_SESSION_MAX];
+struct sxe2_ipsec_session_mgt g_rx_session[SXE2_IPSEC_PORT_MAX][SXE2_IPSEC_SESSION_MAX];
+uint16_t g_tx_sess_id[SXE2_IPSEC_PORT_MAX] = {0};
+uint16_t g_esp_header_offset[SXE2_IPSEC_PORT_MAX] = {0};
+
+static bool sxe2_is_supported(struct rte_eth_dev *dev)
+{
+ return sxe2_ethdev_check(dev);
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_testpmd_sched_reset, 26.07)
+int32_t
+sxe2_testpmd_sched_reset(uint16_t port_id)
+{
+ struct rte_eth_dev *dev = NULL;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+ dev = &rte_eth_devices[port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ return -ENODEV;
+ }
+
+ return sxe2_sched_reset(dev);
+}
+
+extern const char *sxe2_flow_type_name[SXE2_FLOW_TYPE_MAX];
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_flow_rule_dump, 26.07)
+int32_t
+sxe2_flow_rule_dump(uint16_t port_id, struct cmdline *cl)
+{
+ struct rte_eth_dev *dev = NULL;
+ struct sxe2_adapter *adapter = NULL;
+ int32_t ret = -1;
+ struct rte_flow_list_t *flow_list = NULL;
+ struct rte_flow *flow = NULL;
+ uint32_t index = 0;
+ struct sxe2_flow *hw_flow = NULL;
+ uint8_t i = 0;
+
+ const char *sxe2_flow_engine_name[SXE2_FLOW_ENGINE_MAX] = {
+ [SXE2_FLOW_ENGINE_ACL] = "acl",
+ [SXE2_FLOW_ENGINE_RSS] = "rss",
+ [SXE2_FLOW_ENGINE_SWITCH] = "switch",
+ [SXE2_FLOW_ENGINE_FNAV] = "fnav",
+ };
+ const char *sxe2_flow_action_name[SXE2_FLOW_ACTION_MAX] = {
+ [SXE2_FLOW_ACTION_DROP] = "drop",
+ [SXE2_FLOW_ACTION_TC_REDIRECT] = "tc_redirect",
+ [SXE2_FLOW_ACTION_TO_VSI] = "to_vsi",
+ [SXE2_FLOW_ACTION_TO_VSI_LIST] = "to_vsi_list",
+ [SXE2_FLOW_ACTION_PASSTHRU] = "passthru",
+ [SXE2_FLOW_ACTION_QUEUE] = "queue",
+ [SXE2_FLOW_ACTION_Q_REGION] = "q_region",
+ [SXE2_FLOW_ACTION_MARK] = "mark",
+ [SXE2_FLOW_ACTION_COUNT] = "count",
+ [SXE2_FLOW_ACTION_RSS] = "rss",
+ };
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+ dev = &rte_eth_devices[port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev");
+ ret = -ENODEV;
+ goto l_end;
+ }
+ adapter = SXE2_DEV_PRIVATE_TO_ADAPTER(dev);
+ flow_list = &adapter->flow_ctxt.rte_flow_list;
+ cmdline_printf(cl, "Dump sxe2 flow rule:\n");
+ TAILQ_FOREACH(flow, flow_list, next) {
+ cmdline_printf(cl, "rule index: %d\n", index++);
+ TAILQ_FOREACH(hw_flow, &flow->sxe2_flow_list, next) {
+ cmdline_printf(cl, "\thw flow id: %d\n", hw_flow->flow_id);
+ cmdline_printf(cl, "\t\ttype: %s\n",
+ sxe2_flow_type_name[hw_flow->meta.flow_type]);
+ cmdline_printf(cl, "\t\tprio: %d\n", hw_flow->meta.flow_prio);
+ cmdline_printf(cl, "\t\tsrc vsi: %d,rule vsi: %d\n",
+ hw_flow->meta.flow_src_vsi, hw_flow->meta.flow_rule_vsi);
+ cmdline_printf(cl, "\t\tengine type: %s\n",
+ sxe2_flow_engine_name[hw_flow->engine_type]);
+ cmdline_printf(cl, "\t\taction:");
+ for (i = 0; i < SXE2_FLOW_ACTION_MAX; i++) {
+ if (sxe2_test_bit(i, hw_flow->action.act_types))
+ cmdline_printf(cl, "%s ", sxe2_flow_action_name[i]);
+ }
+ cmdline_printf(cl, "\n");
+ }
+ }
+ cmdline_printf(cl, "Dump sxe2 flow rule end.\n");
+ ret = 0;
+l_end:
+ return ret;
+}
+
+static const char *tunnel_type_list[SXE2_UDP_TUNNEL_MAX] = {
+ [SXE2_UDP_TUNNEL_PROTOCOL_VXLAN] = "vxlan",
+ [SXE2_UDP_TUNNEL_PROTOCOL_VXLAN_GPE] = "vxlan-gpe",
+ [SXE2_UDP_TUNNEL_PROTOCOL_GENEVE] = "geneve",
+ [SXE2_UDP_TUNNEL_PROTOCOL_GTP_C] = "gtp-c",
+ [SXE2_UDP_TUNNEL_PROTOCOL_GTP_U] = "gtp-u",
+ [SXE2_UDP_TUNNEL_PROTOCOL_PFCP] = "pfcp",
+ [SXE2_UDP_TUNNEL_PROTOCOL_ECPRI] = "ecpri",
+ [SXE2_UDP_TUNNEL_PROTOCOL_MPLS] = "mpls",
+ [SXE2_UDP_TUNNEL_PROTOCOL_NVGRE] = "nvgre",
+ [SXE2_UDP_TUNNEL_PROTOCOL_L2TP] = "l2tp",
+ [SXE2_UDP_TUNNEL_PROTOCOL_TEREDO] = "teredo"
+};
+
+static enum sxe2_udp_tunnel_protocol sxe2_udp_tunnel_type_str2proto(const char *tunnel_type)
+{
+ enum sxe2_udp_tunnel_protocol proto;
+
+ for (proto = 0; proto < SXE2_UDP_TUNNEL_MAX; proto++) {
+ if (tunnel_type_list[proto] != NULL &&
+ strcmp(tunnel_type_list[proto], tunnel_type) == 0) {
+ break;
+ }
+ }
+
+ return proto;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_udp_tunnel_operations, 26.07)
+int32_t
+sxe2_udp_tunnel_operations(uint16_t port_id, struct cmdline *cl, uint8_t action,
+ uint16_t udp_port, const char *tunnel_type)
+{
+ enum sxe2_udp_tunnel_protocol proto = sxe2_udp_tunnel_type_str2proto(tunnel_type);
+ struct rte_eth_dev *dev = NULL;
+ struct sxe2_adapter *adapter = NULL;
+ struct sxe2_udp_tunnel_cfg tunnel_config = { 0 };
+ int32_t ret = -1;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+ dev = &rte_eth_devices[port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ ret = -ENODEV;
+ goto l_end;
+ }
+
+ if (proto >= SXE2_UDP_TUNNEL_MAX) {
+ cmdline_printf(cl, "Invalid tunnel type!\n");
+ goto l_end;
+ }
+ adapter = dev->data->dev_private;
+ switch (action) {
+ case SXE2_TESTPMD_CMD_UDP_TUNNEL_ADD:
+ ret = sxe2_udp_tunnel_port_add_common(adapter, proto, udp_port);
+ break;
+ case SXE2_TESTPMD_CMD_UDP_TUNNEL_DEL:
+ ret = sxe2_udp_tunnel_port_del_common(adapter, proto, udp_port);
+ break;
+ case SXE2_TESTPMD_CMD_UDP_TUNNEL_GET:
+ tunnel_config.protocol = proto;
+ ret = sxe2_udp_tunnel_port_get_common(adapter, &tunnel_config);
+ if (!ret) {
+ cmdline_printf(cl, "Dump firmware udp tunnel config: [proto:%s, port:%d,"
+ "enable:%d, src/dst:%d/%d, used:%d]\n",
+ tunnel_type_list[proto], tunnel_config.fw_port,
+ tunnel_config.fw_status, tunnel_config.fw_src_en,
+ tunnel_config.fw_dst_en, tunnel_config.fw_used);
+ }
+ break;
+ default:
+ break;
+ }
+
+l_end:
+ return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_stats_info_show, 26.07)
+int32_t
+sxe2_stats_info_show(uint16_t port_id)
+{
+ struct rte_eth_dev *dev = NULL;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+ dev = &rte_eth_devices[port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ return -ENODEV;
+ }
+
+ return 0;
+}
+
+static int32_t sxe2_ipsec_init_mempools(void *sec_ctx)
+{
+ uint16_t nb_sess = 8192;
+ uint32_t sess_sz;
+ char s[64];
+ int32_t ret = -1;
+
+ sess_sz = rte_security_session_get_size(sec_ctx);
+ if (g_sess_pool == NULL) {
+ snprintf(s, sizeof(s), "sess_pool");
+ g_sess_pool = rte_mempool_create(s, nb_sess, sess_sz,
+ MEMPOOL_CACHE_SIZE, 0,
+ NULL, NULL, NULL, NULL,
+ SOCKET_ID_ANY, 0);
+ if (g_sess_pool == NULL) {
+ ret = -ENOMEM;
+ PMD_LOG_ERR(DRV, "Failed to malloc session pool memory.");
+ goto l_end;
+ }
+ }
+ ret = 0;
+
+l_end:
+ return ret;
+}
+
+static void sxe2_ipsec_init_session_mgt(void)
+{
+ uint16_t i;
+ uint8_t port_id;
+
+ if (g_sxe2_ipsec_mgt_init)
+ return;
+
+ for (port_id = 0; port_id < SXE2_IPSEC_PORT_MAX; port_id++) {
+ for (i = 0; i < SXE2_IPSEC_SESSION_MAX; i++) {
+ g_tx_session[port_id][i].session = NULL;
+ g_tx_session[port_id][i].encrypt_algo = SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_NULL;
+ g_tx_session[port_id][i].auth_algo = SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL;
+ g_tx_session[port_id][i].session_id = i;
+ g_tx_session[port_id][i].status = 0;
+ }
+ }
+
+ for (port_id = 0; port_id < SXE2_IPSEC_PORT_MAX; port_id++) {
+ for (i = 0; i < SXE2_IPSEC_SESSION_MAX; i++) {
+ g_rx_session[port_id][i].session = NULL;
+ g_rx_session[port_id][i].encrypt_algo = SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_NULL;
+ g_rx_session[port_id][i].auth_algo = SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL;
+ g_rx_session[port_id][i].session_id = i;
+ g_rx_session[port_id][i].status = 0;
+ }
+ }
+
+ g_sxe2_ipsec_mgt_init = true;
+}
+
+static uint16_t sxe2_ipsec_session_mgt_alloc(enum sxe2_testpmd_ipsec_dir dir, uint16_t port_id)
+{
+ uint16_t i;
+ uint16_t index = 0XFFFF;
+ struct sxe2_ipsec_session_mgt *mgt = NULL;
+
+ if (dir == SXE2_TESTPMD_CMD_IPSEC_DIR_EGRESS)
+ mgt = g_tx_session[port_id];
+ else
+ mgt = g_rx_session[port_id];
+
+ for (i = 0; i < SXE2_IPSEC_SESSION_MAX; i++) {
+ if (mgt[i].status == 0) {
+ index = i;
+ mgt[i].status = 1;
+ break;
+ }
+ }
+
+ return index;
+}
+
+static void sxe2_ipsec_session_mgt_free(enum sxe2_testpmd_ipsec_dir dir,
+ uint16_t index, uint16_t port_id)
+{
+ struct sxe2_ipsec_session_mgt *mgt = NULL;
+
+ if (dir == SXE2_TESTPMD_CMD_IPSEC_DIR_EGRESS)
+ mgt = g_tx_session[port_id];
+ else
+ mgt = g_rx_session[port_id];
+
+ mgt[index].session = NULL;
+ mgt[index].status = 0;
+}
+
+static int32_t sxe2_ipsec_egress_construct(struct cmdline *cl,
+ struct rte_crypto_sym_xform **xform,
+ struct sxe2_ipsec_conf_param *param)
+{
+ struct rte_crypto_sym_xform *cur_xform = NULL;
+ struct rte_crypto_sym_xform *next_xform = NULL;
+ int32_t ret = -1;
+
+ cur_xform = rte_zmalloc("current xform",
+ sizeof(struct rte_crypto_sym_xform), 0);
+ if (cur_xform == NULL) {
+ ret = -ENOMEM;
+ cmdline_printf(cl, "Failed to malloc memory!\n");
+ goto l_end;
+ }
+ cur_xform->type = RTE_CRYPTO_SYM_XFORM_CIPHER;
+ cur_xform->cipher.op = RTE_CRYPTO_CIPHER_OP_ENCRYPT;
+ if (param->encrypt_algo == SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_AES_CBC)
+ cur_xform->cipher.algo = SXE2_RTE_CRYPTO_CIPHER_AES_CBC;
+ else
+ cur_xform->cipher.algo = SXE2_RTE_RTE_CRYPTO_CIPHER_SM4_CBC;
+ cur_xform->cipher.key.length = param->enc_len;
+ cur_xform->cipher.key.data = param->enc_key;
+
+ if (param->auth_algo == SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL) {
+ ret = 0;
+ goto l_end;
+ }
+
+ next_xform = rte_zmalloc("next xform",
+ sizeof(struct rte_crypto_sym_xform), 0);
+ if (next_xform == NULL) {
+ rte_free(cur_xform);
+ ret = -ENOMEM;
+ cmdline_printf(cl, "Failed to malloc memory!\n");
+ goto l_end;
+ }
+ next_xform->type = RTE_CRYPTO_SYM_XFORM_AUTH;
+ next_xform->auth.op = RTE_CRYPTO_AUTH_OP_GENERATE;
+ if (param->auth_algo == SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SHA_HMAC)
+ next_xform->auth.algo = SXE2_RTE_CRYPTO_AUTH_SHA256_HMAC;
+ else
+ next_xform->auth.algo = SXE2_RTE_CRYPTO_AUTH_SM3_HMAC;
+ next_xform->auth.key.length = param->auth_len;
+ next_xform->auth.key.data = param->auth_key;
+ cur_xform->next = next_xform;
+ ret = 0;
+
+l_end:
+ *xform = cur_xform;
+ return ret;
+}
+
+static int32_t sxe2_ipsec_ingress_construct(struct cmdline *cl,
+ struct rte_crypto_sym_xform **xform,
+ struct sxe2_ipsec_conf_param *param)
+{
+ struct rte_crypto_sym_xform *cur_xform = NULL;
+ struct rte_crypto_sym_xform *next_xform = NULL;
+ int32_t ret = -1;
+
+ cur_xform = rte_zmalloc("current xform",
+ sizeof(struct rte_crypto_sym_xform), 0);
+ if (cur_xform == NULL) {
+ ret = -ENOMEM;
+ cmdline_printf(cl, "Failed to malloc memory!\n");
+ goto l_end;
+ }
+
+ if (param->auth_algo == SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL) {
+ cur_xform->type = RTE_CRYPTO_SYM_XFORM_CIPHER;
+ cur_xform->cipher.op = RTE_CRYPTO_CIPHER_OP_DECRYPT;
+ if (param->encrypt_algo == SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_AES_CBC)
+ cur_xform->cipher.algo = SXE2_RTE_CRYPTO_CIPHER_AES_CBC;
+ else
+ cur_xform->cipher.algo = SXE2_RTE_RTE_CRYPTO_CIPHER_SM4_CBC;
+ cur_xform->cipher.key.length = param->enc_len;
+ cur_xform->cipher.key.data = param->enc_key;
+ ret = 0;
+ goto l_end;
+ }
+
+ cur_xform->type = RTE_CRYPTO_SYM_XFORM_AUTH;
+ cur_xform->auth.op = RTE_CRYPTO_AUTH_OP_VERIFY;
+ if (param->auth_algo == SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SHA_HMAC)
+ cur_xform->auth.algo = SXE2_RTE_CRYPTO_AUTH_SHA256_HMAC;
+ else
+ cur_xform->auth.algo = SXE2_RTE_CRYPTO_AUTH_SM3_HMAC;
+
+ cur_xform->auth.key.length = param->auth_len;
+ cur_xform->auth.key.data = param->auth_key;
+
+ next_xform = rte_zmalloc("next xform",
+ sizeof(struct rte_crypto_sym_xform), 0);
+ if (next_xform == NULL) {
+ rte_free(cur_xform);
+ ret = -ENOMEM;
+ cmdline_printf(cl, "Failed to malloc memory!\n");
+ goto l_end;
+ }
+
+ next_xform->type = RTE_CRYPTO_SYM_XFORM_CIPHER;
+ next_xform->cipher.op = RTE_CRYPTO_CIPHER_OP_DECRYPT;
+ if (param->encrypt_algo == SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_AES_CBC)
+ next_xform->cipher.algo = SXE2_RTE_CRYPTO_CIPHER_AES_CBC;
+ else
+ next_xform->cipher.algo = SXE2_RTE_RTE_CRYPTO_CIPHER_SM4_CBC;
+ next_xform->cipher.key.length = param->enc_len;
+ next_xform->cipher.key.data = param->enc_key;
+ cur_xform->next = next_xform;
+ ret = 0;
+
+l_end:
+ *xform = cur_xform;
+ return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_ingress_create, 26.07)
+int32_t
+sxe2_ipsec_ingress_create(struct sxe2_ipsec_conf_param *param, struct cmdline *cl)
+{
+ struct rte_eth_dev *dev = NULL;
+ struct rte_security_session_conf conf;
+ struct rte_crypto_sym_xform *encrypt_xform = NULL;
+ void *session = NULL;
+ struct rte_security_ctx *p_ctx = NULL;
+ int32_t ret = -1;
+ uint16_t index;
+ uint8_t i;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(param->port_id, -ENODEV);
+
+ dev = &rte_eth_devices[param->port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ ret = -ENODEV;
+ goto l_end;
+ }
+
+ if (dev->data->dev_started != 0) {
+ cmdline_printf(cl, "port %d must be stopped.\n", dev->data->port_id);
+ ret = 0;
+ goto l_end;
+ }
+
+ p_ctx = rte_eth_dev_get_sec_ctx(param->port_id);
+
+ if (g_sess_pool == NULL) {
+ ret = sxe2_ipsec_init_mempools(p_ctx);
+ if (ret)
+ goto l_end;
+ }
+
+ sxe2_ipsec_init_session_mgt();
+
+ memset(&conf, 0, sizeof(conf));
+ conf.protocol = RTE_SECURITY_PROTOCOL_IPSEC;
+ conf.action_type = RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO;
+ conf.ipsec.mode = RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+ conf.ipsec.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+ conf.ipsec.direction = RTE_SECURITY_IPSEC_SA_DIR_INGRESS;
+ conf.ipsec.spi = param->spi;
+ conf.ipsec.udp.sport = param->sport;
+ conf.ipsec.udp.dport = param->dport;
+ conf.ipsec.tunnel.type = param->ip_addr.type;
+ if (param->sport || param->dport)
+ conf.ipsec.options.udp_encap = true;
+ if (param->ip_addr.type == RTE_SECURITY_IPSEC_TUNNEL_IPV4)
+ conf.ipsec.tunnel.ipv4.dst_ip.s_addr = param->ip_addr.dst_ipv4;
+ else
+ memcpy(&conf.ipsec.tunnel.ipv6.dst_addr,
+ ¶m->ip_addr.dst_ipv6,
+ sizeof(param->ip_addr.dst_ipv6));
+
+ ret = sxe2_ipsec_ingress_construct(cl, &encrypt_xform, param);
+ if (ret)
+ goto l_end;
+ conf.crypto_xform = encrypt_xform;
+
+ session = rte_security_session_create(p_ctx, &conf, g_sess_pool);
+ if (session == NULL) {
+ ret = -1;
+ goto l_free;
+ }
+
+ index = sxe2_ipsec_session_mgt_alloc(param->dir, param->port_id);
+ if (index == 0XFFFF) {
+ ret = -1;
+ goto l_free;
+ }
+
+ g_rx_session[param->port_id][index].session = session;
+ g_rx_session[param->port_id][index].encrypt_algo = param->encrypt_algo;
+ g_rx_session[param->port_id][index].auth_algo = param->auth_algo;
+ for (i = 0; i < 32; i++) {
+ g_rx_session[param->port_id][index].enc_key[i] = param->enc_key[i];
+ g_rx_session[param->port_id][index].auth_key[i] = param->auth_key[i];
+ }
+ g_rx_session[param->port_id][index].sport = ntohs(param->sport);
+ g_rx_session[param->port_id][index].dport = ntohs(param->dport);
+ g_rx_session[param->port_id][index].spi = ntohl(param->spi);
+ memcpy(&g_rx_session[param->port_id][index].ip_addr,
+ ¶m->ip_addr,
+ sizeof(struct sxe2_ipsec_ip_param));
+
+ ret = 0;
+
+l_free:
+ if (encrypt_xform->next)
+ rte_free(encrypt_xform->next);
+ if (encrypt_xform)
+ rte_free(encrypt_xform);
+
+l_end:
+ return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_ingress_destroy, 26.07)
+int32_t
+sxe2_ipsec_ingress_destroy(struct sxe2_ipsec_conf_param *param, struct cmdline *cl)
+{
+ struct rte_eth_dev *dev = NULL;
+ struct rte_security_ctx *p_ctx = NULL;
+ struct rte_security_session *session = NULL;
+ int32_t ret = -1;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(param->port_id, -ENODEV);
+
+ dev = &rte_eth_devices[param->port_id];
+ if (!sxe2_is_supported(dev)) {
+ cmdline_printf(cl, "Invalid dev.\n");
+ ret = -ENODEV;
+ goto l_end;
+ }
+
+ if (dev->data->dev_started != 0) {
+ cmdline_printf(cl, "port %d must be stopped.\n", dev->data->port_id);
+ ret = 0;
+ goto l_end;
+ }
+
+ if (param->session_id >= SXE2_IPSEC_SESSION_MAX) {
+ PMD_LOG_ERR(DRV, "Invalid session id.");
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ if (!g_rx_session[param->port_id][param->session_id].status) {
+ PMD_LOG_ERR(DRV, "Invalid session status.");
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ if (g_rx_session[param->port_id][param->session_id].session == NULL) {
+ PMD_LOG_ERR(DRV, "Invalid session data.");
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ p_ctx = rte_eth_dev_get_sec_ctx(param->port_id);
+
+ session = g_rx_session[param->port_id][param->session_id].session;
+ ret = rte_security_session_destroy(p_ctx, session);
+ if (ret)
+ goto l_end;
+ sxe2_ipsec_session_mgt_free(param->dir, param->session_id, param->port_id);
+
+ ret = 0;
+l_end:
+ return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_ingress_show, 26.07)
+int32_t
+sxe2_ipsec_ingress_show(struct sxe2_ipsec_conf_param *param, struct cmdline *cl)
+{
+ struct rte_eth_dev *dev = NULL;
+ int32_t ret = -1;
+ uint16_t i;
+ uint8_t j;
+ char encrypt_key[65];
+ char auth_key[65];
+ const char *encrypt_algo[SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_MAX] = {
+ [SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_AES_CBC] = "aes-cbc",
+ [SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_SM4_CBC] = "sm4-cbc",
+ [SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_NULL] = "null"
+ };
+
+ const char *auth_algo[SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_MAX] = {
+ [SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SHA_HMAC] = "sha-hmac",
+ [SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SM3_HMAC] = "sm3-hmac",
+ [SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL] = "null"
+ };
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(param->port_id, -ENODEV);
+
+ dev = &rte_eth_devices[param->port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ ret = -ENODEV;
+ goto l_end;
+ }
+
+ for (i = 0; i < SXE2_IPSEC_SESSION_MAX; i++) {
+ if (g_rx_session[param->port_id][i].status &&
+ g_rx_session[param->port_id][i].session) {
+ memset(encrypt_key, '\0', sizeof(encrypt_key));
+ memset(auth_key, '\0', sizeof(auth_key));
+ for (j = 0; j < 32; j++) {
+ sprintf(encrypt_key + 2 * j, "%02x",
+ g_rx_session[param->port_id][i].enc_key[j]);
+ }
+
+ if (g_rx_session[param->port_id][i].auth_algo !=
+ SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL) {
+ for (j = 0; j < 32; j++) {
+ sprintf(auth_key + 2 * j, "%02x",
+ g_rx_session[param->port_id][i].auth_key[j]);
+ }
+ }
+
+ cmdline_printf(cl, "session_id:%u, direction:rx ,"
+ "encrypt_algo:%s, encrypt_key:0x%s,"
+ "auth_algo:%s, auth_key:0x%s, sport:%u, dport:%u, spi:%u\n",
+ i,
+ encrypt_algo[g_rx_session[param->port_id][i].encrypt_algo],
+ encrypt_key,
+ auth_algo[g_rx_session[param->port_id][i].auth_algo],
+ auth_key,
+ g_rx_session[param->port_id][i].sport,
+ g_rx_session[param->port_id][i].dport,
+ g_rx_session[param->port_id][i].spi);
+ }
+ }
+
+ ret = 0;
+
+l_end:
+ return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_egress_create, 26.07)
+int32_t
+sxe2_ipsec_egress_create(struct sxe2_ipsec_conf_param *param, struct cmdline *cl)
+{
+ struct rte_eth_dev *dev = NULL;
+ struct rte_security_session_conf conf;
+ struct rte_crypto_sym_xform *encrypt_xform = NULL;
+ void *session = NULL;
+ struct rte_security_ctx *p_ctx = NULL;
+ int32_t ret = -1;
+ uint16_t index;
+ uint8_t i;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(param->port_id, -ENODEV);
+
+ dev = &rte_eth_devices[param->port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ ret = -ENODEV;
+ goto l_end;
+ }
+
+ if (dev->data->dev_started != 0) {
+ cmdline_printf(cl, "port %d must be stopped.\n", dev->data->port_id);
+ ret = 0;
+ goto l_end;
+ }
+
+ p_ctx = rte_eth_dev_get_sec_ctx(param->port_id);
+
+ if (g_sess_pool == NULL) {
+ ret = sxe2_ipsec_init_mempools(p_ctx);
+ if (ret)
+ goto l_end;
+ }
+
+ sxe2_ipsec_init_session_mgt();
+
+ memset(&conf, 0, sizeof(conf));
+ conf.protocol = RTE_SECURITY_PROTOCOL_IPSEC;
+ conf.action_type = RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO;
+ conf.ipsec.mode = RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+ conf.ipsec.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+ conf.ipsec.direction = RTE_SECURITY_IPSEC_SA_DIR_EGRESS;
+
+ ret = sxe2_ipsec_egress_construct(cl, &encrypt_xform, param);
+ if (ret)
+ goto l_end;
+ conf.crypto_xform = encrypt_xform;
+
+ session = rte_security_session_create(p_ctx, &conf, g_sess_pool);
+ if (session == NULL) {
+ ret = -1;
+ goto l_free;
+ }
+
+ index = sxe2_ipsec_session_mgt_alloc(param->dir, param->port_id);
+ if (index == 0XFFFF) {
+ ret = -1;
+ goto l_free;
+ }
+
+ g_tx_session[param->port_id][index].session = session;
+ g_tx_session[param->port_id][index].encrypt_algo = param->encrypt_algo;
+ g_tx_session[param->port_id][index].auth_algo = param->auth_algo;
+ for (i = 0; i < 32; i++) {
+ g_tx_session[param->port_id][index].enc_key[i] = param->enc_key[i];
+ g_tx_session[param->port_id][index].auth_key[i] = param->auth_key[i];
+ }
+ ret = 0;
+
+l_free:
+ if (encrypt_xform->next)
+ rte_free(encrypt_xform->next);
+ if (encrypt_xform)
+ rte_free(encrypt_xform);
+
+l_end:
+ return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_egress_destroy, 26.07)
+int32_t
+sxe2_ipsec_egress_destroy(struct sxe2_ipsec_conf_param *param, struct cmdline *cl)
+{
+ struct rte_eth_dev *dev = NULL;
+ struct rte_security_ctx *p_ctx = NULL;
+ struct rte_security_session *session = NULL;
+ int32_t ret = -1;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(param->port_id, -ENODEV);
+
+ dev = &rte_eth_devices[param->port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ ret = -ENODEV;
+ goto l_end;
+ }
+
+ if (dev->data->dev_started != 0) {
+ cmdline_printf(cl, "port %d must be stopped.\n", dev->data->port_id);
+ ret = 0;
+ goto l_end;
+ }
+
+ if (param->session_id >= SXE2_IPSEC_SESSION_MAX) {
+ PMD_LOG_ERR(DRV, "Invalid session id.");
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ if (!g_tx_session[param->port_id][param->session_id].status) {
+ PMD_LOG_ERR(DRV, "Invalid session status.");
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ if (g_tx_session[param->port_id][param->session_id].session == NULL) {
+ PMD_LOG_ERR(DRV, "Invalid session data.");
+ ret = -EINVAL;
+ goto l_end;
+ }
+
+ p_ctx = rte_eth_dev_get_sec_ctx(param->port_id);
+
+ session = g_tx_session[param->port_id][param->session_id].session;
+ ret = rte_security_session_destroy(p_ctx, session);
+ if (ret)
+ goto l_end;
+ sxe2_ipsec_session_mgt_free(param->dir, param->session_id, param->port_id);
+
+ ret = 0;
+
+l_end:
+ return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_egress_show, 26.07)
+int32_t
+sxe2_ipsec_egress_show(struct sxe2_ipsec_conf_param *param, struct cmdline *cl)
+{
+ struct rte_eth_dev *dev = NULL;
+ int32_t ret = -1;
+ uint16_t i;
+ uint8_t j;
+ char encrypt_key[65];
+ char auth_key[65];
+ const char *encrypt_algo[SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_MAX] = {
+ [SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_AES_CBC] = "aes-cbc",
+ [SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_SM4_CBC] = "sm4-cbc",
+ [SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_NULL] = "null"
+ };
+
+ const char *auth_algo[SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_MAX] = {
+ [SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SHA_HMAC] = "sha-hmac",
+ [SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SM3_HMAC] = "sm3-hmac",
+ [SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL] = "null"
+ };
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(param->port_id, -ENODEV);
+
+ dev = &rte_eth_devices[param->port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ ret = -ENODEV;
+ goto l_end;
+ }
+
+ for (i = 0; i < SXE2_IPSEC_SESSION_MAX; i++) {
+ if (g_tx_session[param->port_id][i].status &&
+ g_tx_session[param->port_id][i].session) {
+ memset(encrypt_key, '\0', sizeof(encrypt_key));
+ memset(auth_key, '\0', sizeof(auth_key));
+ for (j = 0; j < 32; j++)
+ sprintf(encrypt_key + 2 * j, "%02x",
+ g_tx_session[param->port_id][i].enc_key[j]);
+ if (g_tx_session[param->port_id][i].auth_algo !=
+ SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL)
+ for (j = 0; j < 32; j++)
+ sprintf(auth_key + 2 * j, "%02x",
+ g_tx_session[param->port_id][i].auth_key[j]);
+
+ cmdline_printf(cl, "id:%u, tx , encrypt_algo:%s,"
+ "encrypt_key:0x%s, auth_algo:%s, auth_key:0x%s.\n",
+ i,
+ encrypt_algo[g_tx_session[param->port_id][i].encrypt_algo],
+ encrypt_key,
+ auth_algo[g_tx_session[param->port_id][i].auth_algo],
+ auth_key);
+ }
+ }
+
+ ret = 0;
+
+l_end:
+ return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_conf_get, 26.07)
+int32_t
+sxe2_ipsec_conf_get(uint16_t port_id, struct cmdline *cl, char type[])
+{
+ struct rte_eth_dev *dev = NULL;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+ dev = &rte_eth_devices[port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ return -ENODEV;
+ }
+ if (!strcmp(type, "session-id"))
+ cmdline_printf(cl, "session-id: %u\n",
+ g_tx_sess_id[port_id]);
+ else if (!strcmp(type, "esp-hdr-offset"))
+ cmdline_printf(cl, "esp-hdr-offset: %u\n",
+ g_esp_header_offset[port_id]);
+ else
+ cmdline_printf(cl, "Invalid type: %s\n", type);
+
+ return 0;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_conf_set, 26.07)
+int32_t
+sxe2_ipsec_conf_set(uint16_t port_id, struct cmdline *cl, char type[], uint16_t value)
+{
+ struct rte_eth_dev *dev = NULL;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+ dev = &rte_eth_devices[port_id];
+ if (!sxe2_is_supported(dev)) {
+ PMD_LOG_ERR(DRV, "Invalid dev.");
+ return -ENODEV;
+ }
+ if (!strcmp(type, "session-id")) {
+ if (value >= 4096 || !g_tx_session[port_id][value].status) {
+ cmdline_printf(cl, "Invalid session-id: %u,"
+ "0 <= value <= 4095 or the session is inactive.\n", value);
+ return -EINVAL;
+ }
+ g_tx_sess_id[port_id] = value;
+ cmdline_printf(cl, "session-id: %u\n", g_tx_sess_id[port_id]);
+ } else if (!strcmp(type, "esp-hdr-offset")) {
+ if (value < 34 || value > 512) {
+ cmdline_printf(cl, "Invalid esp-hdr-offset: %u,"
+ "34 <= value <= 512.\n", value);
+ return -EINVAL;
+ }
+ g_esp_header_offset[port_id] = value;
+ cmdline_printf(cl, "esp-hdr-offset: %u\n",
+ g_esp_header_offset[port_id]);
+ } else {
+ cmdline_printf(cl, "Invalid type: %s\n", type);
+ }
+
+ return 0;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_stats_show, 26.07)
+int32_t
+sxe2_ipsec_stats_show(uint16_t port_id)
+{
+ (void)port_id;
+ return 0;
+}
+
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(sxe2_ipsec_flush, 26.07)
+int32_t
+sxe2_ipsec_flush(uint16_t port_id, struct cmdline *cl)
+{
+ struct rte_eth_dev *dev = NULL;
+ struct rte_security_ctx *p_ctx = NULL;
+ struct rte_security_session *session = NULL;
+ int32_t ret = -1;
+ uint16_t i;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+ dev = &rte_eth_devices[port_id];
+ if (!sxe2_is_supported(dev)) {
+ cmdline_printf(cl, "Invalid dev.\n");
+ ret = -ENODEV;
+ goto l_end;
+ }
+
+ if (dev->data->dev_started != 0) {
+ cmdline_printf(cl, "port %d must be stopped.\n", dev->data->port_id);
+ ret = 0;
+ goto l_end;
+ }
+
+ p_ctx = rte_eth_dev_get_sec_ctx(port_id);
+
+ g_esp_header_offset[port_id] = 0;
+ g_tx_sess_id[port_id] = 0;
+
+ for (i = 0; i < SXE2_IPSEC_SESSION_MAX; i++) {
+ session = g_tx_session[port_id][i].session;
+ if (g_tx_session[port_id][i].status && session) {
+ ret = rte_security_session_destroy(p_ctx, session);
+ if (ret)
+ cmdline_printf(cl, "failed to destroy tx session: %d.\n", i);
+ else
+ sxe2_ipsec_session_mgt_free(SXE2_TESTPMD_CMD_IPSEC_DIR_EGRESS,
+ i, port_id);
+ }
+ }
+
+ for (i = 0; i < SXE2_IPSEC_SESSION_MAX; i++) {
+ session = g_rx_session[port_id][i].session;
+ if (g_rx_session[port_id][i].status && session) {
+ ret = rte_security_session_destroy(p_ctx, session);
+ if (ret)
+ cmdline_printf(cl, "failed to destroy rx session: %d.\n", i);
+ else
+ sxe2_ipsec_session_mgt_free(SXE2_TESTPMD_CMD_IPSEC_DIR_INGRESS,
+ i, port_id);
+ }
+ }
+
+ g_sxe2_ipsec_mgt_init = false;
+ ret = 0;
+
+l_end:
+ return ret;
+}
diff --git a/drivers/net/sxe2/sxe2_testpmd_lib.h b/drivers/net/sxe2/sxe2_testpmd_lib.h
new file mode 100644
index 0000000000..3d2659ef00
--- /dev/null
+++ b/drivers/net/sxe2/sxe2_testpmd_lib.h
@@ -0,0 +1,142 @@
+
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C), 2025, Wuxi Stars Micro System Technologies Co., Ltd.
+ */
+
+#ifndef __SXE2_TESTPMD_LIB_H__
+#define __SXE2_TESTPMD_LIB_H__
+#include <cmdline.h>
+#include "sxe2_ipsec.h"
+
+#define SXE2_IPSEC_SESSION_MAX (4096)
+#define SXE2_IPSEC_PORT_MAX RTE_MAX_ETHPORTS
+#define MEMPOOL_CACHE_SIZE (512 / 2)
+
+enum {
+ SXE2_TESTPMD_CMD_UDP_TUNNEL_ADD = 0,
+ SXE2_TESTPMD_CMD_UDP_TUNNEL_DEL = 1,
+ SXE2_TESTPMD_CMD_UDP_TUNNEL_GET = 2,
+ SXE2_TESTPMD_CMD_UDP_TUNNEL_MAX,
+};
+
+enum sxe2_testpmd_ipsec_op {
+ SXE2_TESTPMD_CMD_IPSEC_OP_ADD = 0,
+ SXE2_TESTPMD_CMD_IPSEC_OP_RM = 1,
+ SXE2_TESTPMD_CMD_IPSEC_OP_SHOW = 2,
+ SXE2_TESTPMD_CMD_IPSEC_OP_MAX,
+};
+
+enum sxe2_testpmd_ipsec_dir {
+ SXE2_TESTPMD_CMD_IPSEC_DIR_EGRESS = 0,
+ SXE2_TESTPMD_CMD_IPSEC_DIR_INGRESS = 1,
+ SXE2_TESTPMD_CMD_IPSEC_DIR_MAX,
+};
+
+enum sxe2_testpmd_ipsec_encrypt_algo {
+ SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_AES_CBC = 0,
+ SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_SM4_CBC = 1,
+ SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_NULL = 2,
+ SXE2_TESTPMD_CMD_IPSEC_EN_ALGO_MAX,
+};
+
+enum sxe2_testpmd_ipsec_auth_algo {
+ SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SHA_HMAC = 0,
+ SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_SM3_HMAC = 1,
+ SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_NULL = 2,
+ SXE2_TESTPMD_CMD_IPSEC_AUTH_ALGO_MAX,
+};
+
+struct sxe2_ipsec_conf_param {
+ enum sxe2_testpmd_ipsec_dir dir;
+ enum sxe2_testpmd_ipsec_op op;
+ enum sxe2_testpmd_ipsec_encrypt_algo encrypt_algo;
+ enum sxe2_testpmd_ipsec_auth_algo auth_algo;
+ struct sxe2_ipsec_ip_param ip_addr;
+ uint32_t spi;
+ uint16_t port_id;
+ uint16_t session_id;
+ uint16_t sport;
+ uint16_t dport;
+ uint8_t enc_key[32];
+ uint8_t enc_len;
+ uint8_t auth_key[32];
+ uint8_t auth_len;
+};
+
+struct sxe2_ipsec_session_mgt {
+ void *session;
+ enum sxe2_testpmd_ipsec_encrypt_algo encrypt_algo;
+ enum sxe2_testpmd_ipsec_auth_algo auth_algo;
+ struct sxe2_ipsec_ip_param ip_addr;
+ uint32_t spi;
+ uint16_t session_id;
+ uint16_t sport;
+ uint16_t dport;
+ uint8_t enc_key[32];
+ uint8_t auth_key[32];
+ uint8_t status;
+};
+
+__rte_experimental
+int32_t
+sxe2_testpmd_sched_reset(uint16_t port_id);
+
+__rte_experimental
+int32_t
+sxe2_flow_rule_dump(uint16_t port_id, struct cmdline *cl);
+
+__rte_experimental
+int32_t
+sxe2_udp_tunnel_operations(uint16_t port_id, struct cmdline *cl, uint8_t action,
+ uint16_t udp_port, const char *tunnel_type);
+
+__rte_experimental
+int32_t
+sxe2_stats_info_show(uint16_t port_id);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_ingress_create(struct sxe2_ipsec_conf_param *param, struct cmdline *cl);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_ingress_destroy(struct sxe2_ipsec_conf_param *param, struct cmdline *cl);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_ingress_show(struct sxe2_ipsec_conf_param *param, struct cmdline *cl);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_egress_create(struct sxe2_ipsec_conf_param *param, struct cmdline *cl);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_egress_destroy(struct sxe2_ipsec_conf_param *param, struct cmdline *cl);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_egress_show(struct sxe2_ipsec_conf_param *param, struct cmdline *cl);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_conf_get(uint16_t port_id, struct cmdline *cl, char type[]);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_conf_set(uint16_t port_id, struct cmdline *cl, char type[], uint16_t value);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_stats_show(uint16_t port_id);
+
+__rte_experimental
+int32_t
+sxe2_ipsec_flush(uint16_t port_id, struct cmdline *cl);
+
+extern struct sxe2_ipsec_session_mgt g_tx_session[SXE2_IPSEC_PORT_MAX][SXE2_IPSEC_SESSION_MAX];
+extern uint16_t g_tx_sess_id[SXE2_IPSEC_PORT_MAX];
+extern uint16_t g_esp_header_offset[SXE2_IPSEC_PORT_MAX];
+extern struct rte_mempool *g_sess_pool;
+
+#endif /* __SXE2_TESTPMD_LIB_H__ */
diff --git a/drivers/net/sxe2/sxe2_tm.c b/drivers/net/sxe2/sxe2_tm.c
index 4c4f793cd5..5de9b5d3b7 100644
--- a/drivers/net/sxe2/sxe2_tm.c
+++ b/drivers/net/sxe2/sxe2_tm.c
@@ -982,6 +982,24 @@ int32_t sxe2_tm_init(struct rte_eth_dev *dev)
return ret;
}
+int32_t sxe2_tm_conf_reset(struct rte_eth_dev *dev)
+{
+ int32_t ret;
+
+ ret = sxe2_tm_uninit(dev);
+ if (ret)
+ goto l_end;
+
+ ret = sxe2_tm_init(dev);
+ if (ret)
+ goto l_end;
+
+ PMD_LOG_DEBUG(DRV, "Tm config reset succeed.");
+
+l_end:
+ return ret;
+}
+
static int32_t sxe2_tm_chk_all_leaf(struct rte_eth_dev *dev)
{
int32_t ret = 0;
diff --git a/drivers/net/sxe2/sxe2_tm.h b/drivers/net/sxe2/sxe2_tm.h
index c4f8da6a8e..b0bfc2091d 100644
--- a/drivers/net/sxe2/sxe2_tm.h
+++ b/drivers/net/sxe2/sxe2_tm.h
@@ -73,4 +73,6 @@ int32_t sxe2_tm_init(struct rte_eth_dev *dev);
int32_t sxe2_tm_uninit(struct rte_eth_dev *dev);
+int32_t sxe2_tm_conf_reset(struct rte_eth_dev *dev);
+
#endif /* __SXE2_TM_H__ */
--
2.52.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox