* Re: [PATCH 1/2] bus/uacce: support driver forward compatibility
From: David Marchand @ 2026-05-26 7:31 UTC (permalink / raw)
To: fengchengwen; +Cc: thomas, dev, qianweili, liuyonglong
In-Reply-To: <c0f24426-19b2-42bf-9432-a1c58ac19c66@huawei.com>
Hello,
On Fri, 22 May 2026 at 03:05, fengchengwen <fengchengwen@huawei.com> wrote:
> >> {
> >> + bool forward_compat = !!(dr->drv_flags & RTE_UACCE_DRV_FORWARD_COMPATIBILITY_DEV);
> >> + uint32_t api_ver = uacce_calc_api_ver(dev->api, NULL);
> >
> > This conversion from a string to integer could be placed in the scanning step.
> > Why place it here?
>
> I think it's OK to place in the scanning step.
> Should I submit a commit to fix this?
Let me handle it.
I'll post a new revision of the series with a fix soon.
> > The dev->api_ver has no in-tree user.
> > This field was (silently?) dropped by my best AI friend in the bus
> > refactoring series I posted.
> > https://patchwork.dpdk.org/project/dpdk/patch/20260506155201.2709810-12-david.marchand@redhat.com/
> >
> > Please advise if I can drop this field (it is just the integer value
> > extracted from dev->api afaiu), or if it should be moved to the
> > scanning step.
>
> Please keep this field.
> This field mainly used for driver to know which api-version the device support, so driver could do
> bug-fix for specific device by judging this field.
I am not a fan of leaving unused stuff, for a potential future case
where it could (maybe) be needed.
This is your bus and drivers, and the change is simple enough, so ok,
no strong objection from me.
Thanks for the reply (and sorry for not replying sooner, I was out for
some days).
--
David Marchand
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Mattias Rönnblom @ 2026-05-26 7:01 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260525151824.6a32e829@phoenix.local>
On 5/26/26 00:18, Stephen Hemminger wrote:
> On Mon, 25 May 2026 21:39:20 +0200
> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>
>> On 5/25/26 16:30, Stephen Hemminger wrote:
>>> On Mon, 25 May 2026 12:36:39 +0200
>>> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>>>
>>>> This RFC introduces fastmem, a general-purpose small-object allocator
>>>> for DPDK. It is intended to replace per-type mempools with a single
>>>> allocator that handles arbitrary sizes, grows on demand, and matches
>>>> mempool-level performance on the hot path.
>>>
>>> Makes sense, what a simple wrapper inline to allow full replacement
>>> testing/performance A/B comparison?
>>
>> Do you mean a mempool or a heap wrapper? Or both?
>>
>> I haven't looked into what options there are with mempools. A mempool
>> driver should be possible, but then I guess one might attempt a
>> whole-sale mempool-compatible API as well.
>
> My thinking is a yet another allocator in DPDK is just another source
> of confusion and bugs. BUT if it can consolidate and fully replace
> one or more existing allocators then it would be great improvement.
>
> Mempools are fast, but fixed and space inefficient.
> Rte_malloc is slow, but flexible.
>
> Also, need to make whatever is added play well with static
> and dynamic checkers.
I'm not sure it's possible to replace mempools with a slab allocator
like fastmem. They have different semantics, and I suspect that there
are times when you prefer a mempool.
# Object Identity & Content Preservation
A mempool always returns one of the same pre-populated objects, with its
contents untouched since last use. This enables pre-initialized fields,
hardware-registered buffers, and constructors that run only once.
# Safe Use-After-Free
Returned objects remain valid, typed memory even after release. Stale
references do not segfault or observe unrelated data, enabling RCU-style
deferred reclamation.
# Bounded, Failure-Free Operation
A mempool operates with a fixed number of objects and performs no
runtime memory allocation. This guarantees deterministic latency,
natural backpressure, and eliminates `ENOMEM` failures after initialization.
# Known IOVA at Initialization Time
All object addresses, both virtual and physical, are fixed and
enumerable from creation time. This enables pre-programming DMA
descriptors and IOMMU registration.
# Memory Accounting
A mempool provides an exact, attributable memory footprint per pool,
without sharing backing memory across unrelated users.
# Dense, Enumerable Object Set
Objects share a common base address and fixed stride, enabling efficient
iteration and pointer compression.
Considering many apps use DPDK for I/O and other hardware abstraction
only, and carry all other OS/kernel/platform type infrastructure
themselves, replacing the mempools with something else will likely cause
a lot of friction.
A fastmem-backed mempool backend (with limitations)? Sure.
Replacing rte_malloc seems easier, but I haven't looked into that in
detail yet.
^ permalink raw reply
* RE: [PATCH] net/mlx5: redirect LACP traffic for legacy E-Switch
From: Bing Zhao @ 2026-05-26 5:57 UTC (permalink / raw)
To: Dariusz Sosnowski, Slava Ovsiienko, Ori Kam, Suanming Mou,
Matan Azrad
Cc: dev@dpdk.org, stable@dpdk.org
In-Reply-To: <20260515123700.354341-1-dsosnowski@nvidia.com>
Hi,
> -----Original Message-----
> From: Dariusz Sosnowski <dsosnowski@nvidia.com>
> Sent: Friday, May 15, 2026 8:37 PM
> To: Slava Ovsiienko <viacheslavo@nvidia.com>; Bing Zhao
> <bingz@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Cc: dev@dpdk.org; stable@dpdk.org
> Subject: [PATCH] net/mlx5: redirect LACP traffic for legacy E-Switch
>
> Offending patch fixed the LACP miss rule logic for NICs where switchdev is
> enabled. In this case, LACP miss rules should be inserted if and only if
> started port is a main port on the embedded switch.
> Side effect of that change was that LACP miss rules are not inserted when
> switchdev is disabled and legacy SR-IOV switch mode is used.
>
> This patch addresses that:
>
> - Fix the LACP rule insertion condition.
> - Move HWS table for LACP rule creation out of FDB rules,
> so they can be created separately.
>
> Fixes: 87e4384d2662 ("net/mlx5: fix condition of LACP miss flow")
> Cc: stable@dpdk.org
>
> Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
> ---
> drivers/net/mlx5/mlx5.h | 1 +
> drivers/net/mlx5/mlx5_flow.h | 40 ++++++++-
> drivers/net/mlx5/mlx5_flow_hw.c | 140 ++++++++++++++++++++++++--------
> drivers/net/mlx5/mlx5_trigger.c | 7 +-
> 4 files changed, 145 insertions(+), 43 deletions(-)
>
> diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index
> 49a0c03544..ab5c76bfc4 100644
> --- a/drivers/net/mlx5/mlx5.h
> +++ b/drivers/net/mlx5/mlx5.h
> @@ -2040,6 +2040,7 @@ struct mlx5_priv {
> rte_spinlock_t hw_ctrl_lock;
> LIST_HEAD(hw_ctrl_flow, mlx5_ctrl_flow_entry) hw_ctrl_flows;
> LIST_HEAD(hw_ext_ctrl_flow, mlx5_ctrl_flow_entry) hw_ext_ctrl_flows;
> + struct mlx5_flow_hw_lacp_miss *hw_lacp_miss; /* HWS LACP miss flow
> +tables */
> struct mlx5_flow_hw_ctrl_fdb *hw_ctrl_fdb; /* FDB control flow
> context */
> struct mlx5_flow_hw_ctrl_nic *hw_ctrl_nic; /* NIC control flow
> context */
> struct rte_flow_pattern_template *hw_tx_repr_tagging_pt; diff --git
> a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index
> c9e72a33d6..3f5ba55bf9 100644
> --- a/drivers/net/mlx5/mlx5_flow.h
> +++ b/drivers/net/mlx5/mlx5_flow.h
> @@ -3031,6 +3031,13 @@ struct mlx5_flow_hw_ctrl_rx {
>
> [MLX5_FLOW_HW_CTRL_RX_EXPANDED_RSS_MAX];
> };
>
> +/* Contains all templates and table required for redirecting LACP
> +traffic with HWS. */ struct mlx5_flow_hw_lacp_miss {
> + struct rte_flow_pattern_template *lacp_rx_items_tmpl;
> + struct rte_flow_actions_template *lacp_rx_actions_tmpl;
> + struct rte_flow_template_table *hw_lacp_rx_tbl; };
> +
> /* Contains all templates required for control flow rules in FDB with
> HWS. */ struct mlx5_flow_hw_ctrl_fdb {
> struct rte_flow_pattern_template *esw_mgr_items_tmpl; @@ -3042,9
> +3049,6 @@ struct mlx5_flow_hw_ctrl_fdb {
> struct rte_flow_pattern_template *port_items_tmpl;
> struct rte_flow_actions_template *jump_one_actions_tmpl;
> struct rte_flow_template_table *hw_esw_zero_tbl;
> - struct rte_flow_pattern_template *lacp_rx_items_tmpl;
> - struct rte_flow_actions_template *lacp_rx_actions_tmpl;
> - struct rte_flow_template_table *hw_lacp_rx_tbl;
> };
>
> struct mlx5_flow_hw_ctrl_nic {
> @@ -3735,6 +3739,36 @@ mlx5_indirect_list_handles_release(struct
> rte_eth_dev *dev);
>
> bool mlx5_flow_is_steering_disabled(void);
>
> +/**
> + * Returns true if Rx control rule for LACP traffic is needed.
> + *
> + * mlx5 PMD needs to create a rule matching LACP traffic and forwarding
> it back to kernel if:
> + *
> + * - Underlying device is a bond interface.
> + * - User did not request to handle LACP traffic in user space.
> + *
> + * Creation of this rule is also controlled by the E-Switch mode:
> + *
> + * - It must be created in legacy mode.
> + * - It must be created only on proxy port in switchdev mode.
> + *
> + * @param[in] priv
> + * Pointer to Ethernet device structure.
> + *
> + * @return
> + * True if LACP rules must be created.
> + * False otherwise.
> + */
> +static inline bool
> +mlx5_flow_lacp_miss_needed(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv = dev->data->dev_private;
> +
> + return !priv->sh->config.lacp_by_user &&
> + priv->pf_bond >= 0 &&
> + (!priv->sh->esw_mode || (priv->sh->esw_mode && priv->master)); }
> +
> #ifdef HAVE_MLX5_HWS_SUPPORT
>
> #define MLX5_REPR_STC_MEMORY_LOG 11
> diff --git a/drivers/net/mlx5/mlx5_flow_hw.c
> b/drivers/net/mlx5/mlx5_flow_hw.c index b6bb9f12a6..c133230cb7 100644
> --- a/drivers/net/mlx5/mlx5_flow_hw.c
> +++ b/drivers/net/mlx5/mlx5_flow_hw.c
> @@ -10820,15 +10820,6 @@ flow_hw_cleanup_ctrl_fdb_tables(struct
> rte_eth_dev *dev)
> if (!priv->hw_ctrl_fdb)
> return;
> hw_ctrl_fdb = priv->hw_ctrl_fdb;
> - /* Clean up templates used for LACP default miss table. */
> - if (hw_ctrl_fdb->hw_lacp_rx_tbl)
> - claim_zero(flow_hw_table_destroy(dev, hw_ctrl_fdb-
> >hw_lacp_rx_tbl, NULL));
> - if (hw_ctrl_fdb->lacp_rx_actions_tmpl)
> - claim_zero(flow_hw_actions_template_destroy(dev, hw_ctrl_fdb-
> >lacp_rx_actions_tmpl,
> - NULL));
> - if (hw_ctrl_fdb->lacp_rx_items_tmpl)
> - claim_zero(flow_hw_pattern_template_destroy(dev, hw_ctrl_fdb-
> >lacp_rx_items_tmpl,
> - NULL));
> /* Clean up templates used for default FDB jump rule. */
> if (hw_ctrl_fdb->hw_esw_zero_tbl)
> claim_zero(flow_hw_table_destroy(dev, hw_ctrl_fdb-
> >hw_esw_zero_tbl, NULL)); @@ -10898,6 +10889,99 @@
> flow_hw_create_lacp_rx_table(struct rte_eth_dev *dev,
> return flow_hw_table_create(dev, &cfg, &it, 1, &at, 1, error); }
>
> +/*
> + * Clean up templates and table used for redirecting LACP traffic to
> kernel.
> + *
> + * @param dev
> + * Pointer to Ethernet device.
> + */
> +static void
> +flow_hw_cleanup_lacp_miss_tables(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv = dev->data->dev_private;
> + struct mlx5_flow_hw_lacp_miss *hw_lacp_miss;
> +
> + if (priv->hw_lacp_miss == NULL)
> + return;
> +
> + hw_lacp_miss = priv->hw_lacp_miss;
> +
> + if (hw_lacp_miss->hw_lacp_rx_tbl)
> + claim_zero(flow_hw_table_destroy(dev, hw_lacp_miss-
> >hw_lacp_rx_tbl, NULL));
> + if (hw_lacp_miss->lacp_rx_actions_tmpl)
> + claim_zero(flow_hw_actions_template_destroy(dev,
> + hw_lacp_miss-
> >lacp_rx_actions_tmpl,
> + NULL));
> + if (hw_lacp_miss->lacp_rx_items_tmpl)
> + claim_zero(flow_hw_pattern_template_destroy(dev,
> + hw_lacp_miss-
> >lacp_rx_items_tmpl,
> + NULL));
> +
> + mlx5_free(hw_lacp_miss);
> + priv->hw_lacp_miss = NULL;
> +}
> +
> +/*
> + * Create templates and table for redirecting LACP traffic to kernel.
> + *
> + * LACP traffic redirection is needed whenever LACP bond is managed by
> the kernel.
> + * Required rule has a following structure:
> + *
> + * - ingress rule on root table
> + * - match EtherType 0x8809
> + * - action DEFAULT_MISS
> + *
> + * @param dev
> + * Pointer to Ethernet device.
> + *
> + * @return
> + * 0 on success. Negative errno otherwise.
> + */
> +static int
> +flow_hw_create_lacp_miss_tables(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv = dev->data->dev_private;
> + struct mlx5_flow_hw_lacp_miss *hw_lacp_miss;
> +
> + if (mlx5_flow_is_steering_disabled())
> + return 0;
> +
> + hw_lacp_miss = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*hw_lacp_miss), 0,
> SOCKET_ID_ANY);
> + if (!hw_lacp_miss) {
> + DRV_LOG(ERR, "port %u Failed to allocate memory for LACP miss
> tables",
> + dev->data->port_id);
> + return -ENOMEM;
> + }
> + priv->hw_lacp_miss = hw_lacp_miss;
> +
> + hw_lacp_miss->lacp_rx_items_tmpl =
> flow_hw_create_lacp_rx_pattern_template(dev, NULL);
> + if (!hw_lacp_miss->lacp_rx_items_tmpl) {
> + DRV_LOG(ERR, "port %u Failed to create pattern template for
> LACP Rx traffic",
> + dev->data->port_id);
> + goto error;
> + }
> + hw_lacp_miss->lacp_rx_actions_tmpl =
> flow_hw_create_lacp_rx_actions_template(dev, NULL);
> + if (!hw_lacp_miss->lacp_rx_actions_tmpl) {
> + DRV_LOG(ERR, "port %u Failed to create actions template for
> LACP Rx traffic",
> + dev->data->port_id);
> + goto error;
> + }
> + hw_lacp_miss->hw_lacp_rx_tbl =
> + flow_hw_create_lacp_rx_table(dev, hw_lacp_miss-
> >lacp_rx_items_tmpl,
> + hw_lacp_miss->lacp_rx_actions_tmpl,
> NULL);
> + if (!hw_lacp_miss->hw_lacp_rx_tbl) {
> + DRV_LOG(ERR, "port %u Failed to create template table for LACP
> Rx traffic",
> + dev->data->port_id);
> + goto error;
> + }
> +
> + return 0;
> +
> +error:
> + flow_hw_cleanup_lacp_miss_tables(dev);
> + return -EINVAL;
> +}
> +
> /**
> * Creates a set of flow tables used to create control flows used
> * when E-Switch is engaged.
> @@ -11000,31 +11084,6 @@ flow_hw_create_fdb_ctrl_tables(struct rte_eth_dev
> *dev, struct rte_flow_error *e
> goto err;
> }
> }
> - /* Create LACP default miss table. */
> - if (!priv->sh->config.lacp_by_user && priv->pf_bond >= 0 && priv-
> >master) {
> - hw_ctrl_fdb->lacp_rx_items_tmpl =
> - flow_hw_create_lacp_rx_pattern_template(dev,
> error);
> - if (!hw_ctrl_fdb->lacp_rx_items_tmpl) {
> - DRV_LOG(ERR, "port %u failed to create pattern template"
> - " for LACP Rx traffic", dev->data->port_id);
> - goto err;
> - }
> - hw_ctrl_fdb->lacp_rx_actions_tmpl =
> - flow_hw_create_lacp_rx_actions_template(dev,
> error);
> - if (!hw_ctrl_fdb->lacp_rx_actions_tmpl) {
> - DRV_LOG(ERR, "port %u failed to create actions template"
> - " for LACP Rx traffic", dev->data->port_id);
> - goto err;
> - }
> - hw_ctrl_fdb->hw_lacp_rx_tbl = flow_hw_create_lacp_rx_table
> - (dev, hw_ctrl_fdb->lacp_rx_items_tmpl,
> - hw_ctrl_fdb->lacp_rx_actions_tmpl, error);
> - if (!hw_ctrl_fdb->hw_lacp_rx_tbl) {
> - DRV_LOG(ERR, "port %u failed to create template table
> for"
> - " for LACP Rx traffic", dev->data->port_id);
> - goto err;
> - }
> - }
> return 0;
>
> err:
> @@ -11754,6 +11813,7 @@ __mlx5_flow_hw_resource_release(struct rte_eth_dev
> *dev, bool ctx_close)
>
> mlx5_flow_hw_rxq_flag_set(dev, false);
> flow_hw_flush_all_ctrl_flows(dev);
> + flow_hw_cleanup_lacp_miss_tables(dev);
> flow_hw_cleanup_ctrl_fdb_tables(dev);
> flow_hw_cleanup_ctrl_nic_tables(dev);
> flow_hw_cleanup_tx_repr_tagging(dev);
> @@ -12160,6 +12220,14 @@ __flow_hw_configure(struct rte_eth_dev *dev,
> if (ret)
> goto err;
> }
> + if (mlx5_flow_lacp_miss_needed(dev)) {
> + ret = flow_hw_create_lacp_miss_tables(dev);
> + if (ret) {
> + rte_flow_error_set(error, -ret,
> RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
> + "Unable to create LACP miss flow
> tables");
> + goto err;
> + }
> + }
> if (is_proxy) {
> ret = flow_hw_create_vport_actions(priv);
> if (ret) {
> @@ -16234,10 +16302,10 @@ mlx5_flow_hw_lacp_rx_flow(struct rte_eth_dev
> *dev)
> .type = MLX5_CTRL_FLOW_TYPE_LACP_RX,
> };
>
> - if (!priv->dr_ctx || !priv->hw_ctrl_fdb || !priv->hw_ctrl_fdb-
> >hw_lacp_rx_tbl)
> + if (!priv->dr_ctx || !priv->hw_lacp_miss ||
> +!priv->hw_lacp_miss->hw_lacp_rx_tbl)
> return 0;
> return flow_hw_create_ctrl_flow(dev, dev,
> - priv->hw_ctrl_fdb->hw_lacp_rx_tbl,
> + priv->hw_lacp_miss->hw_lacp_rx_tbl,
> eth_lacp, 0, miss_action, 0, &flow_info,
> false); }
>
> diff --git a/drivers/net/mlx5/mlx5_trigger.c
> b/drivers/net/mlx5/mlx5_trigger.c index a070aaecfd..32cd18717d 100644
> --- a/drivers/net/mlx5/mlx5_trigger.c
> +++ b/drivers/net/mlx5/mlx5_trigger.c
> @@ -1672,9 +1672,8 @@ mlx5_traffic_enable_hws(struct rte_eth_dev *dev)
> } else {
> DRV_LOG(INFO, "port %u FDB default rule is disabled", dev-
> >data->port_id);
> }
> - if (!priv->sh->config.lacp_by_user && priv->pf_bond >= 0 && priv-
> >master)
> - if (mlx5_flow_hw_lacp_rx_flow(dev))
> - goto error;
> + if (mlx5_flow_lacp_miss_needed(dev) &&
> mlx5_flow_hw_lacp_rx_flow(dev) != 0)
> + goto error;
> if (priv->isolated)
> return 0;
> ret = mlx5_flow_hw_create_ctrl_rx_tables(dev);
> @@ -1796,7 +1795,7 @@ mlx5_traffic_enable(struct rte_eth_dev *dev)
> DRV_LOG(INFO, "port %u FDB default rule is disabled",
> dev->data->port_id);
> }
> - if (!priv->sh->config.lacp_by_user && priv->pf_bond >= 0 && priv-
> >master) {
> + if (mlx5_flow_lacp_miss_needed(dev)) {
> ret = mlx5_flow_lacp_miss(dev);
> if (ret)
> DRV_LOG(INFO, "port %u LACP rule cannot be created - "
> --
> 2.47.3
Acked-by: Bing Zhao <bingz@nvidia.com>
^ permalink raw reply
* [v3 1/1] net/hinic3: Fix VXLAN TSO issue
From: Feifei Wang @ 2026-05-26 3:31 UTC (permalink / raw)
To: dev; +Cc: Feifei Wang
In-Reply-To: <20260526033147.1045-1-wff_light@vip.163.com>
From: Feifei Wang <wangfeifei40@huawei.com>
VXLAN TSO lacks a flag bit, causing the processing function
to determine that the hardware does not support it, leading
to improper handling.
Signed-off-by: Feifei Wang <wangfeifei40@huawei.com>
---
drivers/net/hinic3/hinic3_ethdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/hinic3_ethdev.c b/drivers/net/hinic3/hinic3_ethdev.c
index f4eb788..5071b64 100644
--- a/drivers/net/hinic3/hinic3_ethdev.c
+++ b/drivers/net/hinic3/hinic3_ethdev.c
@@ -652,6 +652,8 @@ hinic3_dev_configure(struct rte_eth_dev *dev)
static void
hinic3_dev_tnl_tso_support(struct rte_eth_dev_info *info, struct hinic3_nic_dev *nic_dev)
{
+ if (HINIC3_SUPPORT_VXLAN_OFFLOAD(nic_dev))
+ info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_VXLAN_TNL_TSO;
if (HINIC3_SUPPORT_GENEVE_OFFLOAD(nic_dev))
info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_GENEVE_TNL_TSO;
if (HINIC3_SUPPORT_IPXIP_OFFLOAD(nic_dev))
@@ -698,7 +700,6 @@ hinic3_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)
RTE_ETH_TX_OFFLOAD_SCTP_CKSUM |
RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM |
RTE_ETH_TX_OFFLOAD_TCP_TSO | RTE_ETH_TX_OFFLOAD_MULTI_SEGS;
- if (nic_dev->feature_cap & NIC_F_HTN_CMDQ)
hinic3_dev_tnl_tso_support(info, nic_dev);
info->hash_key_size = HINIC3_RSS_KEY_SIZE;
--
2.47.0.windows.2
^ permalink raw reply related
* [v3 0/1] net/hinic3: Fix VXLAN TSO issue
From: Feifei Wang @ 2026-05-26 3:31 UTC (permalink / raw)
To: dev
In-Reply-To: <20260520065817.931-2-wff_light@vip.163.com>
v1: The RTE_ETH_TX_OFFLOAD_VXLAN_TNL_TSO flag is added to support the VXLAN TSO function
v2: Modify the commit information issue and supplement the commit information
v3: Revise review comments. First, deterine whether the hardware supports it, then add the flag bit.
Feifei Wang (1):
net/hinic3: Fix VXLAN TSO issue
drivers/net/hinic3/hinic3_ethdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
2.47.0.windows.2
^ permalink raw reply
* Re: [PATCH] eal/riscv: implement prefetch using __builtin_prefetch
From: sunyuechi @ 2026-05-26 3:30 UTC (permalink / raw)
To: shiwei dang; +Cc: dev, stanislaw.kardach, stephen, david.marchand
In-Reply-To: <CAF-WEyHKBEwYW_E6UvjKwEL7LQO4pfFS7qtv-3gjz1B7c5cNYQ@mail.gmail.com>
On 3/29/26 12:38 AM, shiwei dang wrote:
> Hi ,
>
> Thanks for the review. I'll address both points (real name in
> Signed-off-by and test results) together in v2 once I have access to
> physical RISC-V hardware.
>
> Best regards, Dang Shiwei
>
>
Hi,
Gentle ping — any update on v2?
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Stephen Hemminger @ 2026-05-25 22:18 UTC (permalink / raw)
To: Mattias Rönnblom
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <af3a4738-df30-43c4-b3d9-5ec1912a307a@lysator.liu.se>
On Mon, 25 May 2026 21:39:20 +0200
Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> On 5/25/26 16:30, Stephen Hemminger wrote:
> > On Mon, 25 May 2026 12:36:39 +0200
> > Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> >
> >> This RFC introduces fastmem, a general-purpose small-object allocator
> >> for DPDK. It is intended to replace per-type mempools with a single
> >> allocator that handles arbitrary sizes, grows on demand, and matches
> >> mempool-level performance on the hot path.
> >
> > Makes sense, what a simple wrapper inline to allow full replacement
> > testing/performance A/B comparison?
>
> Do you mean a mempool or a heap wrapper? Or both?
>
> I haven't looked into what options there are with mempools. A mempool
> driver should be possible, but then I guess one might attempt a
> whole-sale mempool-compatible API as well.
My thinking is a yet another allocator in DPDK is just another source
of confusion and bugs. BUT if it can consolidate and fully replace
one or more existing allocators then it would be great improvement.
Mempools are fast, but fixed and space inefficient.
Rte_malloc is slow, but flexible.
Also, need to make whatever is added play well with static
and dynamic checkers.
^ permalink raw reply
* Re: [PATCH] dma/ae4dma: add AMD AE4DMA DMA PMD
From: Ningoji, Raghavendra @ 2026-05-25 19:57 UTC (permalink / raw)
To: david.marchand@redhat.com
Cc: Modali, Bhagyada, Sebastian, Selwin, dev@dpdk.org,
fengchengwen@huawei.com, Ningoji, Raghavendra, rjarry@redhat.com,
thomas@monjalon.net
[-- Attachment #1: Type: text/plain, Size: 656 bytes --]
Hi David,
Thanks for the review.
- Please fix the below warnings raised by checkpatches.sh, and run this script before submitting a new revision
- Please also checks the copyright years.
- Globally in those changes, rte_iova_t should probably be used instead of phys_addr_t.
>>All the review comments have been addressed in v2 series . Now all the patches are clean and clears patch checks.
- The patch is big, splitting it into logical patches introducing one feature at a time would help.
>>As you suggested to refactor the v1 patch into logical multiple patches, the v2 patch has been split into 3 patches.
Thanks,
Raghavendra
[-- Attachment #2: Type: text/html, Size: 3102 bytes --]
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Mattias Rönnblom @ 2026-05-25 19:43 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260525113648.5bf540ca@phoenix.local>
On 5/25/26 20:36, Stephen Hemminger wrote:
> On Mon, 25 May 2026 12:36:39 +0200
> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>
>> This RFC introduces fastmem, a general-purpose small-object allocator
>> for DPDK. It is intended to replace per-type mempools with a single
>> allocator that handles arbitrary sizes, grows on demand, and matches
>> mempool-level performance on the hot path.
>>
>> Motivation
>> ----------
>>
>> DPDK applications commonly maintain many mempools — one per object
>> type (connections, sessions, timers, work items). Each must be sized
>> up front, wastes memory when over-provisioned, and cannot serve
>> objects of a different size. Fastmem eliminates this by accepting
>> arbitrary sizes at runtime, backed by a slab allocator that
>> repurposes memory across size classes as demand shifts.
>>
>> Design
>> ------
>>
>> Three-layer architecture:
>>
>> 1. Backing memory: 128 MiB IOVA-contiguous memzones from EAL,
>> reserved lazily (or pre-reserved for deterministic latency).
>>
>> 2. Slabs: 2 MiB, 2 MiB-aligned regions carved from memzones.
>> The alignment enables O(1) slab lookup from any object pointer
>> via bitmask — no radix tree or index structure. Slabs move
>> freely between 18 power-of-2 size classes (8 B to 1 MiB).
>>
>> 3. Per-lcore caches: bounded LIFO stacks (no locks on the hot
>> path). Cache misses trigger bulk transfers to/from the shared
>> bin under a spinlock.
>>
>> Key properties:
>>
>> - Zero per-object metadata in the production build.
>> - NUMA-aware, with per-socket bins and free-slab pools.
>> - DMA-usable memory with O(1) virt-to-IOVA translation.
>> - Bulk alloc/free with all-or-nothing semantics.
>> - Backing memory never returned during lifetime (slabs recycled).
>> - Non-EAL threads supported (bypass cache, take bin lock).
>>
>> API surface
>> -----------
>>
>> rte_fastmem_init / deinit
>> rte_fastmem_reserve
>> rte_fastmem_set_limit / get_limit
>> rte_fastmem_alloc / alloc_socket
>> rte_fastmem_alloc_bulk / alloc_bulk_socket
>> rte_fastmem_free / free_bulk
>> rte_fastmem_virt2iova
>> rte_fastmem_cache_flush
>> rte_fastmem_max_size / classes
>> rte_fastmem_stats / stats_class / stats_lcore / stats_lcore_class
>> rte_fastmem_stats_reset
>>
>> All APIs are marked __rte_experimental.
>>
>> Performance
>> -----------
>>
>> The single-object hot path is roughly 2-3x the cost of mempool
>> and an order of magnitude faster than rte_malloc. Under
>> multi-lcore contention, fastmem scales similarly to mempool,
>> while rte_malloc collapses.
>>
>> Limitations
>> -----------
>>
>> - Maximum allocation: 1 MiB. Larger requests should use rte_malloc.
>> - Power-of-2 classes only; worst-case internal fragmentation ~50%.
>> - Backing memory not reclaimable short of deinit.
>>
>> Future work
>> -----------
>>
>> - Lcore-affine allocations (false-sharing-free by construction).
>> - Mempool ops driver for transparent drop-in use.
>> - Pre-resolved allocator handle binding size class and socket,
>> eliminating per-call class lookup and enabling an inline
>> cache-hit fast path.
>> - Debug mode (cookies, double-free detection, poison-on-free).
>> - Telemetry integration.
>> - EAL integration, allowing EAL-internal subsystems to use
>> fastmem for their small-object allocations.
>>
>> Mattias Rönnblom (3):
>> doc: add fastmem programming guide
>> lib: add fastmem library
>> app/test: add fastmem test suite
>>
>> app/test/meson.build | 3 +
>> app/test/test_fastmem.c | 1682 +++++++++++++++++++++++++
>> app/test/test_fastmem_perf.c | 997 +++++++++++++++
>> app/test/test_fastmem_profile.c | 157 +++
>> doc/api/doxy-api-index.md | 1 +
>> doc/api/doxy-api.conf.in | 1 +
>> doc/guides/prog_guide/fastmem_lib.rst | 301 +++++
>> doc/guides/prog_guide/index.rst | 1 +
>> lib/fastmem/meson.build | 6 +
>> lib/fastmem/rte_fastmem.c | 1486 ++++++++++++++++++++++
>> lib/fastmem/rte_fastmem.h | 644 ++++++++++
>> lib/meson.build | 1 +
>> 12 files changed, 5280 insertions(+)
>> create mode 100644 app/test/test_fastmem.c
>> create mode 100644 app/test/test_fastmem_perf.c
>> create mode 100644 app/test/test_fastmem_profile.c
>> create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
>> create mode 100644 lib/fastmem/meson.build
>> create mode 100644 lib/fastmem/rte_fastmem.c
>> create mode 100644 lib/fastmem/rte_fastmem.h
>>
>
> Largish patchset so did AI review with full claude model.
>
> Series review: [RFC 0/3] add fastmem allocator
> Reviewed against the v1 RFC posted 2026-05-25.
>
>
> [RFC 1/3] doc: add fastmem programming guide
>
> Info: doc/guides/prog_guide/fastmem_lib.rst -- "\ No newline at end of file"
> The new RST file does not end with a newline.
>
>
> [RFC 2/3] lib: add fastmem library
>
> Error: lib/fastmem/rte_fastmem.c -- use-after-free during rte_fastmem_deinit()
> when caches were allocated cross-socket.
>
> cache_create() places the cache struct on the *calling thread's* socket,
> not on the socket the cache serves:
>
> unsigned int own_socket = rte_socket_id();
> ...
> alloc_socket = &fastmem->sockets[own_socket];
> cache = bin_alloc_one(&alloc_socket->bins[cache_class]);
> ...
> *slot = cache; /* slot is in socket K's caches[][] */
>
> So an lcore on socket S that calls rte_fastmem_alloc_socket(..., K) with
> S != K creates a cache whose memory lives in socket S's memzone but is
> reachable through socket K's caches[lcore][class].
>
> rte_fastmem_deinit() then walks sockets in index order:
>
> for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
> release_socket(&fastmem->sockets[i]);
>
> and release_socket() does, in this order:
>
> socket_release_caches(socket); /* (1) */
> for (c...) bin_release(&socket->bins[c], socket); /* (2) */
> for (i...) rte_memzone_free(socket->memzones[i]); /* (3) */
>
> When i = S, step (3) frees socket S's memzones. When i = K (K > S),
> socket_release_caches(K) runs:
>
> cache_slab = slab_of(cache); /* in socket S's freed mz */
> bin_free_one(cache_slab->bin, cache); /* reads cache_slab->bin */
>
> cache_slab points into a freed memzone, so cache_slab->bin and the
> subsequent push (slab->free_head = obj; slab->free_count++; in
> bin_push_locked()) read and write released memory. slab_release() may
> then re-attach the slab to socket S's free_head, which was zeroed and
> whose backing is gone.
>
> This is triggered by any application that allocates from a non-local
> socket via SOCKET_ID_ANY fallback or explicit socket_id, which the
> programming guide describes as a normal mode of operation. The
> existing test_alloc_socket and test_alloc_socket_numa_placement use
> rte_socket_id_by_idx(0) (the local socket) so the bug is not
> exercised by the test suite.
>
> Either order the teardown in three phases (all caches across all
> sockets first, then all bins, then all memzones), or allocate the
> cache struct from the socket it serves rather than the calling
> thread's socket.
>
> Warning: lib/fastmem/rte_fastmem.c -- non-atomic access to shared 64-bit
> statistics counters.
>
> cache->alloc_cache_hits, alloc_cache_misses, alloc_nomem,
> free_cache_hits, free_cache_misses, and the bin counters
> slab_acquires, slab_releases, slabs_partial, slabs_full are
> incremented as plain C reads/writes by the owning lcore and read
> from another thread via rte_fastmem_stats(), rte_fastmem_stats_class(),
> rte_fastmem_stats_lcore(), and rte_fastmem_stats_lcore_class(). On
> architectures where uint64_t is not naturally atomic (and per the C
> standard generally) this is a data race; even on x86-64 it is
> undefined behavior under -fsanitize=thread.
>
> Use rte_atomic_fetch_add_explicit() with rte_memory_order_relaxed on
> the producer side and rte_atomic_load_explicit() with relaxed
> ordering on the reader side. Per AGENTS.md / the DPDK convention,
> relaxed ordering is appropriate for these counters.
>
> Warning: lib/fastmem/rte_fastmem.c -- pointer publish in cache_create()
> without release ordering.
>
> *slot = cache;
> return cache;
>
> The struct fields (count, capacity, target, the stats counters) are
> written before this store but with no fence or release barrier. A
> concurrent stats reader doing socket->caches[l][c] followed by
> cache->* could observe the pointer but not all initialized fields.
> Even ignoring the stats reader, rte_fastmem_cache_flush() invoked
> from a different lcore on the same cache (not currently possible by
> API contract, but the field is technically reachable) would race.
> Pair with rte_atomic_store_explicit(..., rte_memory_order_release)
> and a matching acquire load on the reader path.
>
> Warning: lib/fastmem/rte_fastmem.c -- spurious ENOMEM window during slab
> release.
>
> bin_push_locked() removes a fully-drained slab from bin->partial
> before bin_free_one() drops the bin lock; slab_release() then puts
> it on socket->free_head under the socket lock. Between the unlock
> and slab_release(), another lcore allocating in any class on the
> same socket can see free_head == NULL, hit the memory_limit (or
> FASTMEM_MAX_MEMZONES_PER_SOCKET) check in grow_socket(), and return
> ENOMEM even though the slab is about to become available. Not a
> correctness issue but visible to applications that pin tightly to
> their limit.
>
> Info: lib/fastmem/rte_fastmem.c local_socket_id() final fallback:
>
> return (unsigned int)rte_socket_id_by_idx(0);
>
> rte_socket_id_by_idx() returns int and is documented to return -1 on
> error. If there are zero configured sockets the cast yields UINT_MAX
> and fastmem->sockets[UINT_MAX] is out of bounds. Realistically there
> is always at least one socket, but a defensive check (return 0, or
> fail allocation explicitly) would avoid the corner case.
>
> Info: lib/fastmem/rte_fastmem.c cache_pop() refills to cache->target
> (half capacity) rather than to capacity. Subsequent single-object
> allocs only get target-1 hits before the next bin trip. Likely
> intentional for fairness with bulk callers, but worth a comment.
>
> Info: lib/meson.build inserts 'fastmem' between 'dispatcher' and
> 'gpudev'. The natural alphabetical position is between 'efd' and
> 'fib'; fastmem has no dependency on dispatcher.
>
>
> [RFC 3/3] app/test: add fastmem test suite
>
> Warning: app/test/test_fastmem.c -- REGISTER_FAST_TEST uses NOHUGE_OK
> but the functional tests need real memzone-backed memory.
>
> REGISTER_FAST_TEST(fastmem_autotest, NOHUGE_OK, ASAN_OK,
> test_fastmem);
>
> test_fastmem runs both the lifecycle suite (no allocations) and the
> functional suite, which requests 128 MiB IOVA-contiguous memzones.
> In --no-huge mode IOVA-contiguous reservation of that size is not
> reliable, so NOHUGE_SKIP is more honest. If you want the lifecycle
> tests to remain no-huge-friendly, register them as a separate
> test command.
>
> Warning: app/test/test_fastmem.c -- the suite never exercises
> cross-socket cache allocation.
>
> test_alloc_socket and test_alloc_socket_numa_placement both use
> rte_socket_id_by_idx(0) (the local socket). Add a test that runs on
> a worker lcore whose rte_socket_id() differs from the target
> socket_id passed to rte_fastmem_alloc_socket(), then calls
> rte_fastmem_deinit(). This would have caught the deinit UAF above.
>
> Info: app/test/test_fastmem.c -- several test functions declare an
> uninitialized `int rc;` that is never read or written (e.g.
> test_alloc_too_big, test_alloc_invalid_align, test_alloc_free_small,
> test_alloc_alignment, test_alloc_socket, test_alloc_block_repurposing
> and others). Drop the declarations.
>
> Info: app/test/test_fastmem.c trailing blank-line clusters (two blank
> lines before "return TEST_SUCCESS;" in test_reserve_multiple_memzones,
> test_reserve_cumulative, test_reserve_invalid_socket,
> test_reserve_any_socket, test_alloc_too_big, ...). Drop the extra
> blank line.
Thanks. I've addressed the above issues and the fixes will be available
as an RFC v2, except:
#2 - Non-atomic stats counters
Diagnostic counters read cross-thread. On all DPDK-supported
architectures, aligned uint64_t stores are atomic in practice;
a torn read (e.g., on 32-bit x86) at worst yields a slightly
stale counter value. Not worth the ceremony.
#3 - Pointer publish without release ordering
On weakly-ordered architectures a stats reader could briefly see
uninitialized counter values for a newly-created cache. Acceptable
for diagnostic data.
#4 - Spurious ENOMEM window during slab release
Narrow timing window, not a correctness bug. Closing it would
require holding the bin lock across slab_release(), reintroducing
the contention the design avoids.
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Mattias Rönnblom @ 2026-05-25 19:39 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260525073029.235b4d40@phoenix.local>
On 5/25/26 16:30, Stephen Hemminger wrote:
> On Mon, 25 May 2026 12:36:39 +0200
> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>
>> This RFC introduces fastmem, a general-purpose small-object allocator
>> for DPDK. It is intended to replace per-type mempools with a single
>> allocator that handles arbitrary sizes, grows on demand, and matches
>> mempool-level performance on the hot path.
>
> Makes sense, what a simple wrapper inline to allow full replacement
> testing/performance A/B comparison?
Do you mean a mempool or a heap wrapper? Or both?
I haven't looked into what options there are with mempools. A mempool
driver should be possible, but then I guess one might attempt a
whole-sale mempool-compatible API as well.
The role(s) fastmem could serve are:
a) An lcore/fast path small-object allocator when you don't know the
object size and/or count beforehand (i.e., what the cover letter says).
b) Do what mempools do and a.
c) Do what the rte_malloc heap does, but lcore/fast path-friendly. In
other words, option a but with larger objects too.
e) Something that's both b and c.
I haven't really formed an opinion yet, other than that option a seems
like a natural first step.
Fastmem is significantly slower than mempools for the moment. Claude
will tell you to inline, but that doesn't help (at least not in the
micro benchmarks). Then it will tell you to go remove the statistics,
which also doesn't help. (Latency is data dependency-driven, so stats
load/store/compute runs on resources that otherwise would have been idle.)
What does help however is pre-compute socket and bin-related info and
put into a handle, which the application may optionally use to quickly
retrieve objects of-a-certain-size/from-a-certain-socket. Still slower
than mempool though.
> === Scenario 1: Single-object hot path — cycles per (alloc + free) ===
> allocator 8 B 64 B 256 B 1024 B 4096 B
> fastmem 16.9 16.7 17.7 17.6 17.9
> fastmem_h 9.5 9.4 9.5 9.5 9.4
> mempool 6.9 6.9 6.9 7.0 6.6
> rte_malloc 93.7 93.8 94.8 100.1 130.0
> libc 118.8 119.2 20.4 20.4 111.0
>
> === Scenario 2: Batch alloc, FIFO free — cycles per alloc ===
> allocator 8 B 64 B 256 B 1024 B 4096 B
> fastmem 10.1 10.2 10.8 12.7 14.7
> fastmem_h 6.8 6.7 7.4 9.3 11.4
> mempool 4.2 4.1 4.1 4.1 4.1
> rte_malloc 58.6 58.5 62.1 67.5 68.5
> libc 104.8 104.6 73.7 203.9 1254.0
Intel(R) Xeon(R) Gold 6421N / Ubuntu 24.04 / clang
Best regards,
Mattias
^ permalink raw reply
* [PATCH v2 3/3] dma/ae4dma: add data path operations
From: Raghavendra Ningoji @ 2026-05-25 18:42 UTC (permalink / raw)
To: dev
Cc: Thomas Monjalon, Bhagyada Modali, Robin Jarry, Selwin.Sebastian,
david.marchand, Raghavendra Ningoji
In-Reply-To: <20260525184244.1758825-1-raghavendra.ningoji@amd.com>
Implement the dmadev fast path for the AMD AE4DMA PMD.
This commit adds:
- copy enqueue (rte_dma_copy): write an AE4DMA descriptor for a
memory-to-memory transfer; on RTE_DMA_OP_FLAG_SUBMIT the doorbell
is rung immediately.
- submit (rte_dma_submit): advance the per-queue write_idx
register to expose pending descriptors to the hardware.
- completion (rte_dma_completed / rte_dma_completed_status):
completion is detected via the hardware's per-queue read_idx
register, which the engine advances as it processes descriptors.
The descriptor status / err_code bytes are read only to classify
each drained slot as success or failure, and HW error codes are
translated to the dmadev RTE_DMA_STATUS_* enumeration.
- burst capacity (rte_dma_burst_capacity): report the number of
free descriptor slots, taking into account the one slot reserved
to distinguish full from empty on the power-of-two ring.
The fast path entry points are wired through fp_obj in
ae4dma_dmadev_create(). The fill capability is not advertised;
fp_obj->fill is left zero-initialised.
Signed-off-by: Raghavendra Ningoji <raghavendra.ningoji@amd.com>
---
doc/guides/dmadevs/ae4dma.rst | 22 +++
drivers/dma/ae4dma/ae4dma_dmadev.c | 288 +++++++++++++++++++++++++++++
2 files changed, 310 insertions(+)
diff --git a/doc/guides/dmadevs/ae4dma.rst b/doc/guides/dmadevs/ae4dma.rst
index a85c1d92ca..37a2096ccf 100644
--- a/doc/guides/dmadevs/ae4dma.rst
+++ b/doc/guides/dmadevs/ae4dma.rst
@@ -51,3 +51,25 @@ On probe the PMD performs the following steps for each PCI function:
IOVA-contiguous memory, programs the queue base address and ring
depth into the per-queue registers, and enables the queue.
* Interrupts are masked; completion is polled by the application.
+
+Usage
+-----
+
+Once a dmadev has been started, copies are submitted with
+``rte_dma_copy()`` and completions are reaped with ``rte_dma_completed()``
+or ``rte_dma_completed_status()``. See the
+:ref:`Enqueue / Dequeue API <dmadev_enqueue_dequeue>` section of the
+dmadev library documentation for details.
+
+Limitations
+-----------
+
+* Only memory-to-memory copies are supported. Fill, scatter-gather and
+ any other operation types are not advertised in
+ ``rte_dma_info::dev_capa``.
+* The maximum number of descriptors per virtual channel is fixed by
+ hardware at 32. The PMD rounds the requested ring size up to a
+ power of two and clamps it to 32.
+* Only a single virtual channel per dmadev is supported; use the 16
+ per-PCI-function dmadevs to obtain channel-level parallelism.
+* Interrupt-driven completion is not supported.
diff --git a/drivers/dma/ae4dma/ae4dma_dmadev.c b/drivers/dma/ae4dma/ae4dma_dmadev.c
index dfda723c13..0f223fc40c 100644
--- a/drivers/dma/ae4dma/ae4dma_dmadev.c
+++ b/drivers/dma/ae4dma/ae4dma_dmadev.c
@@ -167,6 +167,73 @@ ae4dma_dev_close(struct rte_dma_dev *dev)
cmd_q->qbase_phys_addr = 0;
return 0;
}
+
+/* trigger h/w to process enqued desc:doorbell - by next_write */
+static inline void
+__submit(struct ae4dma_dmadev *ae4dma)
+{
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t write_idx = cmd_q->next_write;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->write_idx, write_idx);
+ if (nb != 0)
+ cmd_q->stats.submitted += (uint16_t)((cmd_q->next_write - cmd_q->last_write +
+ nb) % nb);
+ cmd_q->last_write = cmd_q->next_write;
+}
+
+static int
+ae4dma_submit(void *dev_private, uint16_t vchan __rte_unused)
+{
+ struct ae4dma_dmadev *ae4dma = dev_private;
+
+ __submit(ae4dma);
+ return 0;
+}
+
+/* Write descriptor for enqueue (copy only). */
+static inline int
+__write_desc_copy(void *dev_private, rte_iova_t src, rte_iova_t dst,
+ uint32_t len, uint64_t flags)
+{
+ struct ae4dma_dmadev *ae4dma = dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ struct ae4dma_desc *dma_desc;
+ uint16_t ret;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+ uint16_t write = cmd_q->next_write;
+
+ if (nb == 0)
+ return -EINVAL;
+
+ /* Reserve one slot to distinguish full from empty (power-of-two ring). */
+ if ((uint32_t)cmd_q->ring_buff_count >= (uint32_t)(nb - 1))
+ return -ENOSPC;
+
+ dma_desc = &cmd_q->qbase_desc[write];
+ memset(dma_desc, 0, sizeof(*dma_desc));
+ dma_desc->length = len;
+ dma_desc->src_hi = upper_32_bits(src);
+ dma_desc->src_lo = lower_32_bits(src);
+ dma_desc->dst_hi = upper_32_bits(dst);
+ dma_desc->dst_lo = lower_32_bits(dst);
+ cmd_q->ring_buff_count++;
+ cmd_q->next_write = (uint16_t)((write + 1) % nb);
+ ret = write;
+ if (flags & RTE_DMA_OP_FLAG_SUBMIT)
+ __submit(ae4dma);
+ return ret;
+}
+
+/* Enqueue a copy operation onto the ae4dma device. */
+static int
+ae4dma_enqueue_copy(void *dev_private, uint16_t vchan __rte_unused,
+ rte_iova_t src, rte_iova_t dst, uint32_t length, uint64_t flags)
+{
+ return __write_desc_copy(dev_private, src, dst, length, flags);
+}
+
/* Dump DMA device info. */
static int
ae4dma_dev_dump(const struct rte_dma_dev *dev, FILE *f)
@@ -197,6 +264,220 @@ ae4dma_dev_dump(const struct rte_dma_dev *dev, FILE *f)
cmd_q->stats.errors);
return 0;
}
+
+/* Translates AE4DMA ChanERRs to DMA error codes. */
+static inline enum rte_dma_status_code
+__translate_status_ae4dma_to_dma(enum ae4dma_dma_err status)
+{
+ AE4DMA_PMD_DEBUG("ae4dma desc status = %d", status);
+
+ switch (status) {
+ case AE4DMA_DMA_ERR_NO_ERR:
+ return RTE_DMA_STATUS_SUCCESSFUL;
+ case AE4DMA_DMA_ERR_INV_LEN:
+ return RTE_DMA_STATUS_INVALID_LENGTH;
+ case AE4DMA_DMA_ERR_INV_SRC:
+ return RTE_DMA_STATUS_INVALID_SRC_ADDR;
+ case AE4DMA_DMA_ERR_INV_DST:
+ return RTE_DMA_STATUS_INVALID_DST_ADDR;
+ case AE4DMA_DMA_ERR_INV_ALIGN:
+ /* Name matches DPDK public enum spelling. */
+ return RTE_DMA_STATUS_DATA_POISION;
+ case AE4DMA_DMA_ERR_INV_HEADER:
+ case AE4DMA_DMA_ERR_INV_STATUS:
+ return RTE_DMA_STATUS_ERROR_UNKNOWN;
+ default:
+ return RTE_DMA_STATUS_ERROR_UNKNOWN;
+ }
+}
+
+/*
+ * Scan HW queue for completed descriptors (non-blocking).
+ *
+ * The AE4DMA engine signals completion by advancing the per-queue
+ * `read_idx` register; it does not (reliably) write a status value
+ * back into the descriptor. We therefore use the HW `read_idx`
+ * register as the source of truth and only inspect the descriptor's
+ * `dw1.err_code` byte to classify each completion as success or
+ * failure.
+ *
+ * @param cmd_q
+ * The AE4DMA command queue.
+ * @param max_ops
+ * Maximum descriptors to process this call.
+ * @param[out] failed_count
+ * Number of completed descriptors that did not report success.
+ * @return
+ * Number of descriptors completed (success + failure), <= max_ops.
+ */
+static inline uint16_t
+ae4dma_scan_hwq(struct ae4dma_cmd_queue *cmd_q, uint16_t max_ops,
+ uint16_t *failed_count)
+{
+ volatile struct ae4dma_desc *hw_desc;
+ uint16_t events_count = 0, fails = 0;
+ uint16_t tail;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+ uint16_t mask;
+ uint16_t hw_read_idx;
+ uint16_t in_flight;
+ uint16_t scan_cap;
+
+ if (nb == 0 || cmd_q->ring_buff_count == 0) {
+ *failed_count = 0;
+ return 0;
+ }
+ mask = nb - 1;
+
+ hw_read_idx = (uint16_t)(AE4DMA_READ_REG(&cmd_q->hwq_regs->read_idx) & mask);
+ tail = cmd_q->next_read;
+
+ /*
+ * Descriptors completed since our last visit live in the
+ * half-open ring range [tail, hw_read_idx). If HW hasn't
+ * moved we have nothing to do.
+ */
+ in_flight = (uint16_t)((hw_read_idx - tail) & mask);
+ if (in_flight == 0) {
+ *failed_count = 0;
+ return 0;
+ }
+
+ scan_cap = max_ops;
+ if (scan_cap > AE4DMA_DESCRIPTORS_PER_CMDQ)
+ scan_cap = AE4DMA_DESCRIPTORS_PER_CMDQ;
+ if (scan_cap > in_flight)
+ scan_cap = in_flight;
+ if (scan_cap > cmd_q->ring_buff_count)
+ scan_cap = (uint16_t)cmd_q->ring_buff_count;
+
+ while (events_count < scan_cap) {
+ uint8_t hw_status;
+ uint8_t hw_err;
+
+ hw_desc = &cmd_q->qbase_desc[tail];
+ hw_status = hw_desc->dw1.status;
+ hw_err = hw_desc->dw1.err_code;
+
+ /*
+ * read_idx advancing is the definitive completion
+ * signal. The per-descriptor status byte is informational
+ * and may not yet be written when we observe it:
+ *
+ * AE4DMA_DMA_DESC_ERROR (4)
+ * Hard failure - err_code names the precise cause.
+ * AE4DMA_DMA_DESC_COMPLETED (3) or 0
+ * Success.
+ * AE4DMA_DMA_DESC_VALIDATED (1) / _PROCESSED (2)
+ * Benign race: HW had not finished updating the
+ * status byte at the instant we read it. Since
+ * read_idx has moved past this slot, treat it as
+ * success unless err_code says otherwise.
+ *
+ * A non-zero err_code is treated as a failure regardless
+ * of the observed status value.
+ */
+ if (hw_status == AE4DMA_DMA_DESC_ERROR ||
+ hw_err != AE4DMA_DMA_ERR_NO_ERR) {
+ fails++;
+ AE4DMA_PMD_WARN("Desc failed: status=%u err=%u",
+ hw_status, hw_err);
+ }
+ cmd_q->status[events_count] = (enum ae4dma_dma_err)hw_err;
+ cmd_q->ring_buff_count--;
+ events_count++;
+ tail = (tail + 1) & mask;
+ }
+
+ cmd_q->stats.completed += events_count;
+ cmd_q->stats.errors += fails;
+ cmd_q->next_read = tail;
+ *failed_count = fails;
+ return events_count;
+}
+
+/* Returns successful operations count and sets error flag if any errors. */
+static uint16_t
+ae4dma_completed(void *dev_private, uint16_t vchan __rte_unused,
+ const uint16_t max_ops, uint16_t *last_idx, bool *has_error)
+{
+ struct ae4dma_dmadev *ae4dma = dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t cpl_count, sl_count;
+ uint16_t err_count = 0;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+
+ *has_error = false;
+
+ cpl_count = ae4dma_scan_hwq(cmd_q, max_ops, &err_count);
+
+ if (cpl_count > max_ops)
+ cpl_count = max_ops;
+
+ if (cpl_count > 0 && last_idx != NULL)
+ *last_idx = (uint16_t)((cmd_q->next_read - 1 + nb) % nb);
+
+ sl_count = cpl_count - err_count;
+ if (err_count)
+ *has_error = true;
+
+ return sl_count;
+}
+
+static uint16_t
+ae4dma_completed_status(void *dev_private, uint16_t vchan __rte_unused,
+ uint16_t max_ops, uint16_t *last_idx,
+ enum rte_dma_status_code *status)
+{
+ struct ae4dma_dmadev *ae4dma = dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t cpl_count;
+ uint16_t i;
+ uint16_t err_count = 0;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+
+ cpl_count = ae4dma_scan_hwq(cmd_q, max_ops, &err_count);
+
+ if (cpl_count > max_ops)
+ cpl_count = max_ops;
+
+ if (cpl_count > 0 && last_idx != NULL)
+ *last_idx = (uint16_t)((cmd_q->next_read - 1 + nb) % nb);
+
+ if (likely(err_count == 0)) {
+ for (i = 0; i < cpl_count; i++)
+ status[i] = RTE_DMA_STATUS_SUCCESSFUL;
+ } else {
+ for (i = 0; i < cpl_count; i++)
+ status[i] = __translate_status_ae4dma_to_dma(cmd_q->status[i]);
+ }
+
+ return cpl_count;
+}
+
+/* Get the remaining capacity of the ring. */
+static uint16_t
+ae4dma_burst_capacity(const void *dev_private, uint16_t vchan __rte_unused)
+{
+ const struct ae4dma_dmadev *ae4dma = dev_private;
+ const struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+ uint16_t mask;
+ uint16_t read_idx = cmd_q->next_read;
+ uint16_t write_idx = cmd_q->next_write;
+ uint16_t used;
+
+ if (nb < 2 || !rte_is_power_of_2(nb))
+ return 0;
+
+ mask = nb - 1;
+ used = (uint16_t)((write_idx - read_idx) & mask);
+ /* One slot reserved (same rule as enqueue). */
+ if (used >= nb - 1)
+ return 0;
+ return (uint16_t)(nb - 1 - used);
+}
+
/* Retrieve the generic stats of a DMA device. */
static int
ae4dma_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused,
@@ -357,6 +638,13 @@ ae4dma_dmadev_create(const char *name, struct rte_pci_device *dev, uint8_t qn)
dmadev->fp_obj->dev_private = dmadev->data->dev_private;
dmadev->dev_ops = &ae4dma_dmadev_ops;
+ dmadev->fp_obj->burst_capacity = ae4dma_burst_capacity;
+ dmadev->fp_obj->completed = ae4dma_completed;
+ dmadev->fp_obj->completed_status = ae4dma_completed_status;
+ dmadev->fp_obj->copy = ae4dma_enqueue_copy;
+ dmadev->fp_obj->submit = ae4dma_submit;
+ /* fill capability not advertised: leave fp_obj->fill as zero-initialised. */
+
ae4dma = dmadev->data->dev_private;
ae4dma->dmadev = dmadev;
ae4dma->pci = dev;
--
2.34.1
^ permalink raw reply related
* [PATCH v2 2/3] dma/ae4dma: add control path operations
From: Raghavendra Ningoji @ 2026-05-25 18:42 UTC (permalink / raw)
To: dev
Cc: Thomas Monjalon, Bhagyada Modali, Robin Jarry, Selwin.Sebastian,
david.marchand, Raghavendra Ningoji
In-Reply-To: <20260525184244.1758825-1-raghavendra.ningoji@amd.com>
Implement the dmadev control path for the AMD AE4DMA PMD.
This commit adds:
- dev_configure / vchan_setup: accept a single virtual channel per
dmadev and clamp the requested ring size to the hardware maximum
of 32 descriptors (rounded up to a power of two).
- dev_start / dev_stop / dev_close: program the per-queue control
register to enable/disable the hardware queue and release the
descriptor ring memzone on close.
- dev_info_get: advertise RTE_DMA_CAPA_MEM_TO_MEM and the fixed
ring depth.
- dev_dump: print the queue identifiers, ring layout and software
completion counters.
- stats_get / stats_reset: expose submitted / completed / errors
counters maintained by the driver.
- vchan_status: report IDLE / ACTIVE based on hardware read_idx vs
write_idx, and HALTED_ERROR when the queue is not enabled.
The dmadev framework is wired through dev_ops in ae4dma_dmadev_create().
Signed-off-by: Raghavendra Ningoji <raghavendra.ningoji@amd.com>
---
drivers/dma/ae4dma/ae4dma_dmadev.c | 223 +++++++++++++++++++++++++++++
1 file changed, 223 insertions(+)
diff --git a/drivers/dma/ae4dma/ae4dma_dmadev.c b/drivers/dma/ae4dma/ae4dma_dmadev.c
index 76de2cde45..dfda723c13 100644
--- a/drivers/dma/ae4dma/ae4dma_dmadev.c
+++ b/drivers/dma/ae4dma/ae4dma_dmadev.c
@@ -53,6 +53,215 @@ ae4dma_queue_dma_zone_reserve(const char *queue_name,
socket_id, RTE_MEMZONE_IOVA_CONTIG, queue_size);
}
+/* Configure a device. */
+static int
+ae4dma_dev_configure(struct rte_dma_dev *dev __rte_unused,
+ const struct rte_dma_conf *dev_conf,
+ uint32_t conf_sz)
+{
+ if (sizeof(struct rte_dma_conf) != conf_sz)
+ return -EINVAL;
+
+ if (dev_conf->nb_vchans != 1)
+ return -EINVAL;
+
+ return 0;
+}
+
+/* Setup a virtual channel for AE4DMA, only 1 vchan is supported per dmadev. */
+static int
+ae4dma_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused,
+ const struct rte_dma_vchan_conf *qconf, uint32_t qconf_sz)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t max_desc = qconf->nb_desc;
+
+ if (sizeof(struct rte_dma_vchan_conf) != qconf_sz)
+ return -EINVAL;
+
+ if (max_desc < 2)
+ return -EINVAL;
+
+ if (!rte_is_power_of_2(max_desc))
+ max_desc = rte_align32pow2(max_desc);
+
+ if (max_desc > AE4DMA_DESCRIPTORS_PER_CMDQ) {
+ AE4DMA_PMD_DEBUG("DMA dev %u nb_desc clamped to %u",
+ dev->data->dev_id, AE4DMA_DESCRIPTORS_PER_CMDQ);
+ max_desc = AE4DMA_DESCRIPTORS_PER_CMDQ;
+ }
+
+ cmd_q->qcfg = *qconf;
+ cmd_q->qcfg.nb_desc = max_desc;
+
+ /* Ensure all counters are reset, if reconfiguring/restarting device. */
+ memset(&cmd_q->stats, 0, sizeof(cmd_q->stats));
+ return 0;
+}
+
+/* Start a configured device. */
+static int
+ae4dma_dev_start(struct rte_dma_dev *dev)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+
+ if (nb == 0)
+ return -EBUSY;
+
+ /* Program ring depth expected by hardware. */
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->max_idx, nb);
+ return 0;
+}
+
+/* Stop a configured device. */
+static int
+ae4dma_dev_stop(struct rte_dma_dev *dev)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+
+ if (cmd_q->hwq_regs != NULL)
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->control_reg.control_raw,
+ AE4DMA_CMD_QUEUE_DISABLE);
+ return 0;
+}
+
+/* Get device information of a device. */
+static int
+ae4dma_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info,
+ uint32_t size)
+{
+ if (size < sizeof(*info))
+ return -EINVAL;
+ info->dev_name = dev->device->name;
+ info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM;
+ info->max_vchans = 1;
+ info->min_desc = 2;
+ info->max_desc = AE4DMA_DESCRIPTORS_PER_CMDQ;
+ info->nb_vchans = 1;
+ return 0;
+}
+
+/* Close a configured device. */
+static int
+ae4dma_dev_close(struct rte_dma_dev *dev)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+
+ if (cmd_q->hwq_regs != NULL)
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->control_reg.control_raw,
+ AE4DMA_CMD_QUEUE_DISABLE);
+
+ if (cmd_q->memz_name[0] != '\0') {
+ const struct rte_memzone *mz = rte_memzone_lookup(cmd_q->memz_name);
+
+ if (mz != NULL)
+ rte_memzone_free(mz);
+ }
+ cmd_q->qbase_desc = NULL;
+ cmd_q->qbase_addr = NULL;
+ cmd_q->qbase_phys_addr = 0;
+ return 0;
+}
+/* Dump DMA device info. */
+static int
+ae4dma_dev_dump(const struct rte_dma_dev *dev, FILE *f)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q;
+ void *ae4dma_mmio_base_addr = (uint8_t *)ae4dma->io_regs;
+
+ cmd_q = &ae4dma->cmd_q;
+ fprintf(f, "cmd_q->id = %" PRIx64 "\n", cmd_q->id);
+ fprintf(f, "cmd_q->qidx = %" PRIx64 "\n", cmd_q->qidx);
+ fprintf(f, "cmd_q->qsize = %" PRIx64 "\n", cmd_q->qsize);
+ fprintf(f, "mmio_base_addr = %p\n", ae4dma_mmio_base_addr);
+ fprintf(f, "queues per ae4dma engine = %d\n", AE4DMA_READ_REG_OFFSET(
+ ae4dma_mmio_base_addr, AE4DMA_COMMON_CONFIG_OFFSET));
+ fprintf(f, "== Private Data ==\n");
+ fprintf(f, " Config: { ring_size: %u }\n", cmd_q->qcfg.nb_desc);
+ fprintf(f, " Ring virt: %p\tphys: %#" PRIx64 "\n",
+ (void *)cmd_q->qbase_desc,
+ (uint64_t)cmd_q->qbase_phys_addr);
+ fprintf(f, " Next write: %u\n", cmd_q->next_write);
+ fprintf(f, " Next read: %u\n", cmd_q->next_read);
+ fprintf(f, " current queue depth: %u\n", cmd_q->ring_buff_count);
+ fprintf(f, " }\n");
+ fprintf(f, " Key Stats { submitted: %" PRIu64 ", comp: %" PRIu64 ", failed: %" PRIu64 " }\n",
+ cmd_q->stats.submitted,
+ cmd_q->stats.completed,
+ cmd_q->stats.errors);
+ return 0;
+}
+/* Retrieve the generic stats of a DMA device. */
+static int
+ae4dma_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused,
+ struct rte_dma_stats *rte_stats, uint32_t size)
+{
+ const struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ const struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ const struct rte_dma_stats *stats = &cmd_q->stats;
+
+ if (size < sizeof(*rte_stats))
+ return -EINVAL;
+ if (rte_stats == NULL)
+ return -EINVAL;
+
+ *rte_stats = *stats;
+ return 0;
+}
+
+/* Reset the generic stat counters for the DMA device. */
+static int
+ae4dma_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+
+ memset(&cmd_q->stats, 0, sizeof(cmd_q->stats));
+ return 0;
+}
+
+/*
+ * Report channel state to the dmadev framework.
+ *
+ * RTE_DMA_VCHAN_HALTED_ERROR - HW queue is disabled (never started, or
+ * stopped via dev_stop()).
+ * RTE_DMA_VCHAN_IDLE - HW has caught up: read_idx == write_idx,
+ * no descriptors in flight.
+ * RTE_DMA_VCHAN_ACTIVE - HW still has descriptors to process.
+ */
+static int
+ae4dma_vchan_status(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused,
+ enum rte_dma_vchan_status *status)
+{
+ const struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ const struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint32_t ctrl, hw_read, hw_write;
+
+ if (cmd_q->hwq_regs == NULL) {
+ *status = RTE_DMA_VCHAN_HALTED_ERROR;
+ return 0;
+ }
+
+ ctrl = AE4DMA_READ_REG(&cmd_q->hwq_regs->control_reg.control_raw);
+ if ((ctrl & AE4DMA_CMD_QUEUE_ENABLE) == 0) {
+ *status = RTE_DMA_VCHAN_HALTED_ERROR;
+ return 0;
+ }
+
+ hw_read = AE4DMA_READ_REG(&cmd_q->hwq_regs->read_idx);
+ hw_write = AE4DMA_READ_REG(&cmd_q->hwq_regs->write_idx);
+
+ *status = (hw_read == hw_write) ? RTE_DMA_VCHAN_IDLE
+ : RTE_DMA_VCHAN_ACTIVE;
+ return 0;
+}
+
static int
ae4dma_add_queue(struct ae4dma_dmadev *dev, uint8_t qn, const char *pci_name)
{
@@ -114,6 +323,19 @@ ae4dma_channel_dev_name(char *out, size_t outlen, const char *pci_name,
static int
ae4dma_dmadev_create(const char *name, struct rte_pci_device *dev, uint8_t qn)
{
+ static const struct rte_dma_dev_ops ae4dma_dmadev_ops = {
+ .dev_close = ae4dma_dev_close,
+ .dev_configure = ae4dma_dev_configure,
+ .dev_dump = ae4dma_dev_dump,
+ .dev_info_get = ae4dma_dev_info_get,
+ .dev_start = ae4dma_dev_start,
+ .dev_stop = ae4dma_dev_stop,
+ .stats_get = ae4dma_stats_get,
+ .stats_reset = ae4dma_stats_reset,
+ .vchan_status = ae4dma_vchan_status,
+ .vchan_setup = ae4dma_vchan_setup,
+ };
+
struct rte_dma_dev *dmadev = NULL;
struct ae4dma_dmadev *ae4dma = NULL;
char hwq_dev_name[RTE_DEV_NAME_MAX_LEN];
@@ -133,6 +355,7 @@ ae4dma_dmadev_create(const char *name, struct rte_pci_device *dev, uint8_t qn)
}
dmadev->device = &dev->device;
dmadev->fp_obj->dev_private = dmadev->data->dev_private;
+ dmadev->dev_ops = &ae4dma_dmadev_ops;
ae4dma = dmadev->data->dev_private;
ae4dma->dmadev = dmadev;
--
2.34.1
^ permalink raw reply related
* [PATCH v2 1/3] dma/ae4dma: introduce AMD AE4DMA DMA PMD
From: Raghavendra Ningoji @ 2026-05-25 18:42 UTC (permalink / raw)
To: dev
Cc: Thomas Monjalon, Bhagyada Modali, Robin Jarry, Selwin.Sebastian,
david.marchand, Raghavendra Ningoji
In-Reply-To: <20260525184244.1758825-1-raghavendra.ningoji@amd.com>
Add the skeleton of a new dmadev poll-mode driver for the AMD AE4DMA
hardware DMA engine, providing only PCI probe/remove and per-queue
hardware initialisation. An AE4DMA engine exposes 16 hardware command
queues, each with a 32-entry descriptor ring; the PMD maps each
hardware channel to its own dmadev with a single virtual channel,
so a PCI function appears as 16 dmadevs named "<pci-bdf>-ch0" ..
"<pci-bdf>-ch15".
This patch only registers the PCI driver, allocates the dmadev
objects, reserves the per-queue descriptor rings and programs the
hardware queue base addresses. Control and data path operations are
added in subsequent patches.
Signed-off-by: Raghavendra Ningoji <raghavendra.ningoji@amd.com>
---
.mailmap | 1 +
MAINTAINERS | 5 +
doc/guides/dmadevs/ae4dma.rst | 53 ++++++
doc/guides/dmadevs/index.rst | 1 +
doc/guides/rel_notes/release_26_07.rst | 7 +
drivers/dma/ae4dma/ae4dma_dmadev.c | 227 +++++++++++++++++++++++++
drivers/dma/ae4dma/ae4dma_hw_defs.h | 160 +++++++++++++++++
drivers/dma/ae4dma/ae4dma_internal.h | 118 +++++++++++++
drivers/dma/ae4dma/meson.build | 7 +
drivers/dma/meson.build | 1 +
usertools/dpdk-devbind.py | 5 +-
11 files changed, 584 insertions(+), 1 deletion(-)
create mode 100644 doc/guides/dmadevs/ae4dma.rst
create mode 100644 drivers/dma/ae4dma/ae4dma_dmadev.c
create mode 100644 drivers/dma/ae4dma/ae4dma_hw_defs.h
create mode 100644 drivers/dma/ae4dma/ae4dma_internal.h
create mode 100644 drivers/dma/ae4dma/meson.build
diff --git a/.mailmap b/.mailmap
index 89ba6ffccc..60180818f9 100644
--- a/.mailmap
+++ b/.mailmap
@@ -203,6 +203,7 @@ Benoît Ganne <bganne@cisco.com>
Bernard Iremonger <bernard.iremonger@intel.com>
Bert van Leeuwen <bert.vanleeuwen@netronome.com>
Bhagyada Modali <bhagyada.modali@amd.com>
+Raghavendra Ningoji <raghavendra.ningoji@amd.com>
Bharat Mota <bharat.mota@broadcom.com> <bmota@vmware.com>
Bhuvan Mital <bhuvan.mital@amd.com>
Bibo Mao <maobibo@loongson.cn>
diff --git a/MAINTAINERS b/MAINTAINERS
index 9143d028bc..2e27af49f4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1361,6 +1361,11 @@ F: doc/guides/compressdevs/features/zsda.ini
DMAdev Drivers
--------------
+AMD AE4DMA
+M: Bhagyada Modali <bhagyada.modali@amd.com>
+F: drivers/dma/ae4dma/
+F: doc/guides/dmadevs/ae4dma.rst
+
Intel IDXD - EXPERIMENTAL
M: Bruce Richardson <bruce.richardson@intel.com>
M: Kevin Laatz <kevin.laatz@intel.com>
diff --git a/doc/guides/dmadevs/ae4dma.rst b/doc/guides/dmadevs/ae4dma.rst
new file mode 100644
index 0000000000..a85c1d92ca
--- /dev/null
+++ b/doc/guides/dmadevs/ae4dma.rst
@@ -0,0 +1,53 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2025 Advanced Micro Devices, Inc.
+
+.. include:: <isonum.txt>
+
+AMD AE4DMA DMA Device Driver
+============================
+
+The ``ae4dma`` dmadev driver is a poll-mode driver (PMD) for the
+AMD AE4DMA hardware DMA engine. The engine exposes 16 independent
+hardware command queues, each with a ring of 32 descriptors. The PMD
+maps each hardware command queue to a separate DPDK dmadev with a
+single virtual channel, so a single PCI function appears as 16 dmadevs
+named ``<pci-bdf>-ch0`` through ``<pci-bdf>-ch15``.
+
+The driver supports memory-to-memory copy operations only.
+
+Hardware Requirements
+---------------------
+
+The ``dpdk-devbind.py`` script can be used to list AE4DMA devices on
+the system::
+
+ dpdk-devbind.py --status-dev dma
+
+AE4DMA devices appear with vendor ID ``0x1022`` and device ID
+``0x149b``.
+
+Compilation
+-----------
+
+The driver is built as part of the standard DPDK build on x86 platforms
+using ``meson`` and ``ninja``; no extra configuration is required.
+
+Device Setup
+------------
+
+The AE4DMA device must be bound to a DPDK-compatible kernel module such
+as ``vfio-pci`` before it can be used::
+
+ dpdk-devbind.py -b vfio-pci <pci-bdf>
+
+Initialization
+~~~~~~~~~~~~~~
+
+On probe the PMD performs the following steps for each PCI function:
+
+* Reads BAR0 and programs the common configuration register with the
+ number of hardware queues to enable (16).
+* For each hardware queue it allocates a 32-entry descriptor ring in
+ IOVA-contiguous memory, programs the queue base address and ring
+ depth into the per-queue registers, and enables the queue.
+* Interrupts are masked; completion is polled by the application.
diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst
index 56beb1733f..97399590f6 100644
--- a/doc/guides/dmadevs/index.rst
+++ b/doc/guides/dmadevs/index.rst
@@ -11,6 +11,7 @@ an application through DMA API.
:maxdepth: 1
:numbered:
+ ae4dma
cnxk
dpaa
dpaa2
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index f012d47a4b..9a78a7ef62 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -63,6 +63,13 @@ New Features
``rte_eal_init`` and the application is responsible for probing each device,
* ``--auto-probing`` enables the initial bus probing, which is the current default behavior.
+* **Added AMD AE4DMA DMA PMD.**
+
+ Added a new ``dma/ae4dma`` driver for the AMD AE4DMA hardware DMA engine.
+ Each PCI function exposes 16 hardware command queues; the PMD registers one
+ dmadev per channel with a single virtual channel and supports
+ memory-to-memory copy operations.
+
Removed Items
-------------
diff --git a/drivers/dma/ae4dma/ae4dma_dmadev.c b/drivers/dma/ae4dma/ae4dma_dmadev.c
new file mode 100644
index 0000000000..76de2cde45
--- /dev/null
+++ b/drivers/dma/ae4dma/ae4dma_dmadev.c
@@ -0,0 +1,227 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#include <errno.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+#include <rte_bus_pci.h>
+#include <bus_pci_driver.h>
+#include <rte_dmadev_pmd.h>
+#include <rte_malloc.h>
+
+#include "ae4dma_internal.h"
+
+/*
+ * One dmadev per AE4DMA hardware channel; each dmadev has exactly one
+ * virtual channel. The HW's per-queue register block must be densely
+ * packed right after the engine-common config register at BAR0+0; the
+ * build-time check below catches an accidental layout change.
+ */
+static_assert(sizeof(struct ae4dma_hwq_regs) == 32,
+ "ae4dma_hwq_regs stride changed; per-queue offset math will break");
+
+RTE_LOG_REGISTER_DEFAULT(ae4dma_pmd_logtype, INFO);
+
+#define AE4DMA_PMD_NAME dmadev_ae4dma
+
+static const struct rte_memzone *
+ae4dma_queue_dma_zone_reserve(const char *queue_name,
+ uint32_t queue_size, int socket_id)
+{
+ const struct rte_memzone *mz;
+
+ mz = rte_memzone_lookup(queue_name);
+ if (mz != NULL) {
+ if (((size_t)queue_size <= mz->len) &&
+ ((socket_id == SOCKET_ID_ANY) ||
+ (socket_id == mz->socket_id))) {
+ AE4DMA_PMD_INFO("reuse memzone already "
+ "allocated for %s", queue_name);
+ return mz;
+ }
+ AE4DMA_PMD_ERR("Incompatible memzone already "
+ "allocated %s, size %u, socket %d. "
+ "Requested size %u, socket %u",
+ queue_name, (uint32_t)mz->len,
+ mz->socket_id, queue_size, socket_id);
+ return NULL;
+ }
+ return rte_memzone_reserve_aligned(queue_name, queue_size,
+ socket_id, RTE_MEMZONE_IOVA_CONTIG, queue_size);
+}
+
+static int
+ae4dma_add_queue(struct ae4dma_dmadev *dev, uint8_t qn, const char *pci_name)
+{
+ uint32_t dma_addr_lo, dma_addr_hi;
+ struct ae4dma_cmd_queue *cmd_q;
+ const struct rte_memzone *q_mz;
+
+ dev->io_regs = dev->pci->mem_resource[AE4DMA_PCIE_BAR].addr;
+
+ cmd_q = &dev->cmd_q;
+ cmd_q->id = qn;
+ cmd_q->qidx = 0;
+ cmd_q->qsize = AE4DMA_QUEUE_SIZE(AE4DMA_QUEUE_DESC_SIZE);
+ cmd_q->hwq_regs = (volatile struct ae4dma_hwq_regs *)dev->io_regs + (qn + 1);
+
+ /*
+ * Memzone name must be globally unique. Embed PCI BDF so multiple
+ * PCI functions probed concurrently don't collide.
+ */
+ snprintf(cmd_q->memz_name, sizeof(cmd_q->memz_name),
+ "ae4dma_%s_q%u", pci_name, (unsigned int)qn);
+
+ q_mz = ae4dma_queue_dma_zone_reserve(cmd_q->memz_name,
+ cmd_q->qsize, rte_socket_id());
+ if (q_mz == NULL) {
+ AE4DMA_PMD_ERR("memzone reserve failed for %s", cmd_q->memz_name);
+ return -ENOMEM;
+ }
+
+ cmd_q->qbase_addr = (void *)q_mz->addr;
+ cmd_q->qbase_desc = (struct ae4dma_desc *)q_mz->addr;
+ cmd_q->qbase_phys_addr = q_mz->iova;
+
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->max_idx, AE4DMA_DESCRIPTORS_PER_CMDQ);
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->control_reg.control_raw,
+ AE4DMA_CMD_QUEUE_ENABLE);
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->intr_status_reg.intr_status_raw,
+ AE4DMA_DISABLE_INTR);
+ cmd_q->next_write = (uint16_t)AE4DMA_READ_REG(&cmd_q->hwq_regs->write_idx);
+ cmd_q->next_read = (uint16_t)AE4DMA_READ_REG(&cmd_q->hwq_regs->read_idx);
+ cmd_q->ring_buff_count = 0;
+
+ dma_addr_lo = low32_value(cmd_q->qbase_phys_addr);
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->qbase_lo, dma_addr_lo);
+ dma_addr_hi = high32_value(cmd_q->qbase_phys_addr);
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->qbase_hi, dma_addr_hi);
+
+ return 0;
+}
+
+static void
+ae4dma_channel_dev_name(char *out, size_t outlen, const char *pci_name,
+ unsigned int ch)
+{
+ snprintf(out, outlen, "%s-ch%u", pci_name, ch);
+}
+
+/* Create a dmadev(dpdk DMA device) */
+static int
+ae4dma_dmadev_create(const char *name, struct rte_pci_device *dev, uint8_t qn)
+{
+ struct rte_dma_dev *dmadev = NULL;
+ struct ae4dma_dmadev *ae4dma = NULL;
+ char hwq_dev_name[RTE_DEV_NAME_MAX_LEN];
+
+ if (!name) {
+ AE4DMA_PMD_ERR("Invalid name of the device!");
+ return -EINVAL;
+ }
+ memset(hwq_dev_name, 0, sizeof(hwq_dev_name));
+ ae4dma_channel_dev_name(hwq_dev_name, sizeof(hwq_dev_name), name, qn);
+
+ dmadev = rte_dma_pmd_allocate(hwq_dev_name, dev->device.numa_node,
+ sizeof(struct ae4dma_dmadev));
+ if (dmadev == NULL) {
+ AE4DMA_PMD_ERR("Unable to allocate dma device");
+ return -ENOMEM;
+ }
+ dmadev->device = &dev->device;
+ dmadev->fp_obj->dev_private = dmadev->data->dev_private;
+
+ ae4dma = dmadev->data->dev_private;
+ ae4dma->dmadev = dmadev;
+ ae4dma->pci = dev;
+
+ if (ae4dma_add_queue(ae4dma, qn, name) != 0)
+ goto init_error;
+ return 0;
+
+init_error:
+ AE4DMA_PMD_ERR("driver %s(): failed", __func__);
+ rte_dma_pmd_release(hwq_dev_name);
+ return -ENOMEM;
+}
+
+/* Probe DMA device. */
+static int
+ae4dma_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev)
+{
+ char name[32];
+ char chname[RTE_DEV_NAME_MAX_LEN];
+ void *mmio_base;
+ uint32_t q_per_eng;
+ int ret = 0;
+ uint8_t i;
+
+ rte_pci_device_name(&dev->addr, name, sizeof(name));
+ AE4DMA_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node);
+ dev->device.driver = &drv->driver;
+
+ mmio_base = dev->mem_resource[AE4DMA_PCIE_BAR].addr;
+ if (mmio_base == NULL) {
+ AE4DMA_PMD_ERR("%s: BAR%d not mapped", name, AE4DMA_PCIE_BAR);
+ return -ENODEV;
+ }
+
+ /* Program the per-engine HW queue count once. */
+ AE4DMA_WRITE_REG_OFFSET(mmio_base, AE4DMA_COMMON_CONFIG_OFFSET,
+ AE4DMA_MAX_HW_QUEUES);
+ q_per_eng = AE4DMA_READ_REG_OFFSET(mmio_base, AE4DMA_COMMON_CONFIG_OFFSET);
+ AE4DMA_PMD_INFO("%s: AE4DMA queues per engine = %u", name, q_per_eng);
+
+ for (i = 0; i < AE4DMA_MAX_HW_QUEUES; i++) {
+ ret = ae4dma_dmadev_create(name, dev, i);
+ if (ret != 0) {
+ AE4DMA_PMD_ERR("%s create dmadev %u failed!", name, i);
+ while (i > 0) {
+ i--;
+ ae4dma_channel_dev_name(chname, sizeof(chname), name, i);
+ rte_dma_pmd_release(chname);
+ }
+ break;
+ }
+ }
+ return ret;
+}
+
+/* Remove DMA device. */
+static int
+ae4dma_dmadev_remove(struct rte_pci_device *dev)
+{
+ char name[32];
+ char chname[RTE_DEV_NAME_MAX_LEN];
+ unsigned int i;
+
+ rte_pci_device_name(&dev->addr, name, sizeof(name));
+
+ AE4DMA_PMD_INFO("Closing %s on NUMA node %d",
+ name, dev->device.numa_node);
+
+ for (i = 0; i < AE4DMA_MAX_HW_QUEUES; i++) {
+ ae4dma_channel_dev_name(chname, sizeof(chname), name, i);
+ rte_dma_pmd_release(chname);
+ }
+ return 0;
+}
+
+static const struct rte_pci_id pci_id_ae4dma_map[] = {
+ { RTE_PCI_DEVICE(AMD_VENDOR_ID, AE4DMA_DEVICE_ID) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver ae4dma_pmd_drv = {
+ .id_table = pci_id_ae4dma_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = ae4dma_dmadev_probe,
+ .remove = ae4dma_dmadev_remove,
+};
+
+RTE_PMD_REGISTER_PCI(AE4DMA_PMD_NAME, ae4dma_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(AE4DMA_PMD_NAME, pci_id_ae4dma_map);
+RTE_PMD_REGISTER_KMOD_DEP(AE4DMA_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci");
diff --git a/drivers/dma/ae4dma/ae4dma_hw_defs.h b/drivers/dma/ae4dma/ae4dma_hw_defs.h
new file mode 100644
index 0000000000..62b6a1b30b
--- /dev/null
+++ b/drivers/dma/ae4dma/ae4dma_hw_defs.h
@@ -0,0 +1,160 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#ifndef __AE4DMA_HW_DEFS_H__
+#define __AE4DMA_HW_DEFS_H__
+
+#include <rte_bus_pci.h>
+#include <rte_byteorder.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_memzone.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define AE4DMA_BIT(nr) (1UL << (nr))
+
+/* ae4dma device details */
+#define AMD_VENDOR_ID 0x1022
+#define AE4DMA_DEVICE_ID 0x149b
+#define AE4DMA_PCIE_BAR 0
+
+/*
+ * An AE4DMA engine has 16 DMA queues. Each queue supports 32 descriptors.
+ */
+#define AE4DMA_MAX_HW_QUEUES 16
+#define AE4DMA_QUEUE_START_INDEX 0
+#define AE4DMA_CMD_QUEUE_ENABLE 0x1
+#define AE4DMA_CMD_QUEUE_DISABLE 0x0
+
+/* Common to all queues */
+#define AE4DMA_COMMON_CONFIG_OFFSET 0x00
+
+#define AE4DMA_DISABLE_INTR 0x01
+
+/* Descriptor status */
+enum ae4dma_dma_status {
+ AE4DMA_DMA_DESC_SUBMITTED = 0,
+ AE4DMA_DMA_DESC_VALIDATED = 1,
+ AE4DMA_DMA_DESC_PROCESSED = 2,
+ AE4DMA_DMA_DESC_COMPLETED = 3,
+ AE4DMA_DMA_DESC_ERROR = 4,
+};
+
+/* Descriptor error-code */
+enum ae4dma_dma_err {
+ AE4DMA_DMA_ERR_NO_ERR = 0,
+ AE4DMA_DMA_ERR_INV_HEADER = 1,
+ AE4DMA_DMA_ERR_INV_STATUS = 2,
+ AE4DMA_DMA_ERR_INV_LEN = 3,
+ AE4DMA_DMA_ERR_INV_SRC = 4,
+ AE4DMA_DMA_ERR_INV_DST = 5,
+ AE4DMA_DMA_ERR_INV_ALIGN = 6,
+ AE4DMA_DMA_ERR_UNKNOWN = 7,
+};
+
+/* HW Queue status */
+enum ae4dma_hwqueue_status {
+ AE4DMA_HWQUEUE_EMPTY = 0,
+ AE4DMA_HWQUEUE_FULL = 1,
+ AE4DMA_HWQUEUE_NOT_EMPTY = 4
+};
+/*
+ * descriptor for AE4DMA commands
+ * 8 32-bit words:
+ * word 0: source memory type; destination memory type ; control bits
+ * word 1: desc_id; error code; status
+ * word 2: length
+ * word 3: reserved
+ * word 4: upper 32 bits of source pointer
+ * word 5: low 32 bits of source pointer
+ * word 6: upper 32 bits of destination pointer
+ * word 7: low 32 bits of destination pointer
+ */
+
+/* AE4DMA Descriptor - DWORD0 - Controls bits: Reserved for future use */
+#define AE4DMA_DWORD0_STOP_ON_COMPLETION AE4DMA_BIT(0)
+#define AE4DMA_DWORD0_INTERRUPT_ON_COMPLETION AE4DMA_BIT(1)
+#define AE4DMA_DWORD0_START_OF_MESSAGE AE4DMA_BIT(3)
+#define AE4DMA_DWORD0_END_OF_MESSAGE AE4DMA_BIT(4)
+#define AE4DMA_DWORD0_DESTINATION_MEMORY_TYPE RTE_GENMASK64(5, 4)
+#define AE4DMA_DWORD0_SOURCE_MEMEORY_TYPE RTE_GENMASK64(7, 6)
+
+#define AE4DMA_DWORD0_DESTINATION_MEMORY_TYPE_MEMORY (0x0)
+#define AE4DMA_DWORD0_DESTINATION_MEMORY_TYPE_IOMEMORY (1<<4)
+#define AE4DMA_DWORD0_SOURCE_MEMEORY_TYPE_MEMORY (0x0)
+#define AE4DMA_DWORD0_SOURCE_MEMEORY_TYPE_IOMEMORY (1<<6)
+
+struct ae4dma_desc_dword0 {
+ uint8_t byte0;
+ uint8_t byte1;
+ uint16_t timestamp;
+};
+
+struct ae4dma_desc_dword1 {
+ uint8_t status;
+ uint8_t err_code;
+ uint16_t desc_id;
+};
+
+struct ae4dma_desc {
+ struct ae4dma_desc_dword0 dw0;
+ struct ae4dma_desc_dword1 dw1;
+ uint32_t length;
+ uint32_t reserved;
+ uint32_t src_lo;
+ uint32_t src_hi;
+ uint32_t dst_lo;
+ uint32_t dst_hi;
+};
+
+/*
+ * Registers for each queue :4 bytes length
+ * Effective address : offset + reg
+ */
+struct ae4dma_hwq_regs {
+ union {
+ uint32_t control_raw;
+ struct {
+ uint32_t queue_enable: 1;
+ uint32_t reserved_internal: 31;
+ } control;
+ } control_reg;
+
+ union {
+ uint32_t status_raw;
+ struct {
+ uint32_t reserved0: 1;
+ /* 0–empty, 1–full, 2–stopped, 3–error , 4–Not Empty */
+ uint32_t queue_status: 2;
+ uint32_t reserved1: 21;
+ uint32_t interrupt_type: 4;
+ uint32_t reserved2: 4;
+ } status;
+ } status_reg;
+
+ uint32_t max_idx;
+ uint32_t read_idx;
+ uint32_t write_idx;
+
+ union {
+ uint32_t intr_status_raw;
+ struct {
+ uint32_t intr_status: 1;
+ uint32_t reserved: 31;
+ } intr_status;
+ } intr_status_reg;
+
+ uint32_t qbase_lo;
+ uint32_t qbase_hi;
+
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* AE4DMA_HW_DEFS_H */
diff --git a/drivers/dma/ae4dma/ae4dma_internal.h b/drivers/dma/ae4dma/ae4dma_internal.h
new file mode 100644
index 0000000000..9892d6697f
--- /dev/null
+++ b/drivers/dma/ae4dma/ae4dma_internal.h
@@ -0,0 +1,118 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#ifndef _AE4DMA_INTERNAL_H_
+#define _AE4DMA_INTERNAL_H_
+
+#include <stdint.h>
+
+#include "ae4dma_hw_defs.h"
+
+/**
+ * upper_32_bits - return bits 32-63 of a number
+ * @n: the number we're accessing
+ */
+#define upper_32_bits(n) ((uint32_t)(((n) >> 16) >> 16))
+
+/**
+ * lower_32_bits - return bits 0-31 of a number
+ * @n: the number we're accessing
+ */
+#define lower_32_bits(n) ((uint32_t)((n) & 0xffffffff))
+
+/** Hardware ring depth (slots per queue); must be power of two. */
+#define AE4DMA_DESCRIPTORS_PER_CMDQ 32
+#define AE4DMA_QUEUE_DESC_SIZE sizeof(struct ae4dma_desc)
+#define AE4DMA_QUEUE_SIZE(n) (AE4DMA_DESCRIPTORS_PER_CMDQ * (n))
+
+
+/** AE4DMA registers Write/Read */
+static inline void ae4dma_pci_reg_write(void *base, int offset,
+ uint32_t value)
+{
+ volatile void *reg_addr = ((uint8_t *)base + offset);
+
+ rte_write32((rte_cpu_to_le_32(value)), reg_addr);
+}
+
+static inline uint32_t ae4dma_pci_reg_read(void *base, int offset)
+{
+ volatile void *reg_addr = ((uint8_t *)base + offset);
+
+ return rte_le_to_cpu_32(rte_read32(reg_addr));
+}
+
+#define AE4DMA_READ_REG_OFFSET(hw_addr, reg_offset) \
+ ae4dma_pci_reg_read(hw_addr, reg_offset)
+
+#define AE4DMA_WRITE_REG_OFFSET(hw_addr, reg_offset, value) \
+ ae4dma_pci_reg_write(hw_addr, reg_offset, value)
+
+
+#define AE4DMA_READ_REG(hw_addr) \
+ ae4dma_pci_reg_read((void *)(uintptr_t)(hw_addr), 0)
+
+#define AE4DMA_WRITE_REG(hw_addr, value) \
+ ae4dma_pci_reg_write((void *)(uintptr_t)(hw_addr), 0, value)
+
+static inline uint32_t
+low32_value(unsigned long addr)
+{
+ return ((uint64_t)addr) & 0xffffffffUL;
+}
+
+static inline uint32_t
+high32_value(unsigned long addr)
+{
+ return (uint32_t)(((uint64_t)addr) >> 32);
+}
+
+/**
+ * A structure describing a AE4DMA command queue.
+ */
+struct __rte_cache_aligned ae4dma_cmd_queue {
+ char memz_name[RTE_MEMZONE_NAMESIZE];
+ volatile struct ae4dma_hwq_regs *hwq_regs;
+
+ struct rte_dma_vchan_conf qcfg;
+ struct rte_dma_stats stats;
+ /* Queue address */
+ struct ae4dma_desc *qbase_desc;
+ void *qbase_addr;
+ rte_iova_t qbase_phys_addr;
+ enum ae4dma_dma_err status[AE4DMA_DESCRIPTORS_PER_CMDQ];
+ /* Queue identifier */
+ uint64_t id; /**< queue id */
+ uint64_t qidx; /**< queue index */
+ uint64_t qsize; /**< queue size */
+ uint32_t ring_buff_count;
+ unsigned short next_read;
+ unsigned short next_write;
+ unsigned short last_write; /* Used to compute submitted count. */
+};
+
+/*
+ * One dmadev per AE4DMA hardware channel: probe creates AE4DMA_MAX_HW_QUEUES
+ * dmadevs per PCI function, each owning a single HW command queue.
+ */
+struct ae4dma_dmadev {
+ struct rte_dma_dev *dmadev;
+ void *io_regs;
+ struct ae4dma_cmd_queue cmd_q; /**< single HW queue owned by this dmadev */
+ struct rte_pci_device *pci; /**< owning PCI device (not owned) */
+};
+
+
+extern int ae4dma_pmd_logtype;
+#define RTE_LOGTYPE_AE4DMA_PMD ae4dma_pmd_logtype
+
+#define AE4DMA_PMD_LOG(level, ...) \
+ RTE_LOG_LINE_PREFIX(level, AE4DMA_PMD, "%s(): ", __func__, __VA_ARGS__)
+
+#define AE4DMA_PMD_DEBUG(...) AE4DMA_PMD_LOG(DEBUG, __VA_ARGS__)
+#define AE4DMA_PMD_INFO(...) AE4DMA_PMD_LOG(INFO, __VA_ARGS__)
+#define AE4DMA_PMD_ERR(...) AE4DMA_PMD_LOG(ERR, __VA_ARGS__)
+#define AE4DMA_PMD_WARN(...) AE4DMA_PMD_LOG(WARNING, __VA_ARGS__)
+
+#endif /* _AE4DMA_INTERNAL_H_ */
diff --git a/drivers/dma/ae4dma/meson.build b/drivers/dma/ae4dma/meson.build
new file mode 100644
index 0000000000..e48ab0d561
--- /dev/null
+++ b/drivers/dma/ae4dma/meson.build
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2024 Advanced Micro Devices, Inc. All rights reserved.
+
+build = dpdk_conf.has('RTE_ARCH_X86')
+reason = 'only supported on x86'
+sources = files('ae4dma_dmadev.c')
+deps += ['bus_pci', 'dmadev']
diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build
index e0d94db967..c230ac5a06 100644
--- a/drivers/dma/meson.build
+++ b/drivers/dma/meson.build
@@ -2,6 +2,7 @@
# Copyright 2021 HiSilicon Limited
drivers = [
+ 'ae4dma',
'cnxk',
'dpaa',
'dpaa2',
diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index 93f2383dff..7d09f155de 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -86,6 +86,9 @@
cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
'SVendor': None, 'SDevice': None}
+amd_ae4dma = {'Class': '08', 'Vendor': '1022', 'Device': '149b',
+ 'SVendor': None, 'SDevice': None}
+
virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042',
'SVendor': None, 'SDevice': None}
@@ -95,7 +98,7 @@
network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
baseband_devices = [acceleration_class]
crypto_devices = [encryption_class, intel_processor_class]
-dma_devices = [cnxk_dma, hisilicon_dma,
+dma_devices = [amd_ae4dma, cnxk_dma, hisilicon_dma,
intel_idxd_gnrd, intel_idxd_dmr, intel_idxd_spr,
intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx,
odm_dma]
--
2.34.1
^ permalink raw reply related
* [PATCH v2 0/3] dma/ae4dma: add AMD AE4DMA DMA PMD
From: Raghavendra Ningoji @ 2026-05-25 18:42 UTC (permalink / raw)
To: dev
Cc: Thomas Monjalon, Bhagyada Modali, Robin Jarry, Selwin.Sebastian,
david.marchand, Raghavendra Ningoji
In-Reply-To: <20260518181856.1228373-1-raghavendra.ningoji@amd.com>
This series adds a new dmadev poll-mode driver for the AMD AE4DMA
hardware DMA engine. An AE4DMA engine exposes 16 hardware command
queues, each with a 32-entry descriptor ring; the PMD maps each
hardware channel to its own dmadev with a single virtual channel,
so a PCI function appears as 16 dmadevs named "<pci-bdf>-ch0" ..
"<pci-bdf>-ch15".
Driver characteristics:
- Memory-to-memory copy operations only (RTE_DMA_CAPA_MEM_TO_MEM).
- Completion is detected via the hardware's per-queue read_idx
register, which the engine advances as it processes descriptors.
The descriptor status / err_code bytes are read only to classify
each drained slot as success or failure.
- vchan_status reports IDLE/ACTIVE based on HW read_idx vs write_idx
and HALTED_ERROR when the queue is not enabled.
- depends on bus_pci and dmadev.
The v1 was submitted as a single patch. Per review feedback the
driver is now introduced in three logical patches, following the
pattern of the recent hisi_acc dmadev driver:
1/3 - introduce driver (probe, remove, per-queue HW init)
2/3 - add control path operations (dev_ops)
3/3 - add data path operations (copy, submit, completion)
---
Changes in v2:
- Split the monolithic v1 patch into three logical patches
(introduce / control path / data path), mirroring the
structure used by drivers/dma/hisi_acc.
- Fix checkpatches.sh warnings in drivers/dma/ae4dma/ae4dma_internal.h:
* Use RTE_LOG_LINE_PREFIX (with RTE_LOGTYPE_AE4DMA_PMD) instead
of the deprecated rte_log() call form.
* Replace the GCC variadic argument-pack extension ("args...")
with C99 __VA_ARGS__ in the AE4DMA_PMD_{LOG,DEBUG,INFO,ERR,
WARN} macros.
- Move __rte_cache_aligned to the "struct" keyword position on
struct ae4dma_cmd_queue, as required by checkpatches.sh.
v1:https://patches.dpdk.org/project/dpdk/patch/20260518181856.1228373-1-raghavendra.ningoji@amd.com/
Raghavendra Ningoji (3):
dma/ae4dma: introduce AMD AE4DMA DMA PMD
dma/ae4dma: add control path operations
dma/ae4dma: add data path operations
.mailmap | 1 +
MAINTAINERS | 5 +
doc/guides/dmadevs/ae4dma.rst | 75 +++
doc/guides/dmadevs/index.rst | 1 +
doc/guides/rel_notes/release_26_07.rst | 7 +
drivers/dma/ae4dma/ae4dma_dmadev.c | 738 +++++++++++++++++++++++++
drivers/dma/ae4dma/ae4dma_hw_defs.h | 160 ++++++
drivers/dma/ae4dma/ae4dma_internal.h | 118 ++++
drivers/dma/ae4dma/meson.build | 7 +
drivers/dma/meson.build | 1 +
usertools/dpdk-devbind.py | 5 +-
11 files changed, 1117 insertions(+), 1 deletion(-)
create mode 100644 doc/guides/dmadevs/ae4dma.rst
create mode 100644 drivers/dma/ae4dma/ae4dma_dmadev.c
create mode 100644 drivers/dma/ae4dma/ae4dma_hw_defs.h
create mode 100644 drivers/dma/ae4dma/ae4dma_internal.h
create mode 100644 drivers/dma/ae4dma/meson.build
base-commit: f724d1c0d1c1636b9c171c34db3f17c3defaa2f3
--
2.34.1
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Stephen Hemminger @ 2026-05-25 18:36 UTC (permalink / raw)
To: Mattias Rönnblom
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260525103642.55255-1-hofors@lysator.liu.se>
On Mon, 25 May 2026 12:36:39 +0200
Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> This RFC introduces fastmem, a general-purpose small-object allocator
> for DPDK. It is intended to replace per-type mempools with a single
> allocator that handles arbitrary sizes, grows on demand, and matches
> mempool-level performance on the hot path.
>
> Motivation
> ----------
>
> DPDK applications commonly maintain many mempools — one per object
> type (connections, sessions, timers, work items). Each must be sized
> up front, wastes memory when over-provisioned, and cannot serve
> objects of a different size. Fastmem eliminates this by accepting
> arbitrary sizes at runtime, backed by a slab allocator that
> repurposes memory across size classes as demand shifts.
>
> Design
> ------
>
> Three-layer architecture:
>
> 1. Backing memory: 128 MiB IOVA-contiguous memzones from EAL,
> reserved lazily (or pre-reserved for deterministic latency).
>
> 2. Slabs: 2 MiB, 2 MiB-aligned regions carved from memzones.
> The alignment enables O(1) slab lookup from any object pointer
> via bitmask — no radix tree or index structure. Slabs move
> freely between 18 power-of-2 size classes (8 B to 1 MiB).
>
> 3. Per-lcore caches: bounded LIFO stacks (no locks on the hot
> path). Cache misses trigger bulk transfers to/from the shared
> bin under a spinlock.
>
> Key properties:
>
> - Zero per-object metadata in the production build.
> - NUMA-aware, with per-socket bins and free-slab pools.
> - DMA-usable memory with O(1) virt-to-IOVA translation.
> - Bulk alloc/free with all-or-nothing semantics.
> - Backing memory never returned during lifetime (slabs recycled).
> - Non-EAL threads supported (bypass cache, take bin lock).
>
> API surface
> -----------
>
> rte_fastmem_init / deinit
> rte_fastmem_reserve
> rte_fastmem_set_limit / get_limit
> rte_fastmem_alloc / alloc_socket
> rte_fastmem_alloc_bulk / alloc_bulk_socket
> rte_fastmem_free / free_bulk
> rte_fastmem_virt2iova
> rte_fastmem_cache_flush
> rte_fastmem_max_size / classes
> rte_fastmem_stats / stats_class / stats_lcore / stats_lcore_class
> rte_fastmem_stats_reset
>
> All APIs are marked __rte_experimental.
>
> Performance
> -----------
>
> The single-object hot path is roughly 2-3x the cost of mempool
> and an order of magnitude faster than rte_malloc. Under
> multi-lcore contention, fastmem scales similarly to mempool,
> while rte_malloc collapses.
>
> Limitations
> -----------
>
> - Maximum allocation: 1 MiB. Larger requests should use rte_malloc.
> - Power-of-2 classes only; worst-case internal fragmentation ~50%.
> - Backing memory not reclaimable short of deinit.
>
> Future work
> -----------
>
> - Lcore-affine allocations (false-sharing-free by construction).
> - Mempool ops driver for transparent drop-in use.
> - Pre-resolved allocator handle binding size class and socket,
> eliminating per-call class lookup and enabling an inline
> cache-hit fast path.
> - Debug mode (cookies, double-free detection, poison-on-free).
> - Telemetry integration.
> - EAL integration, allowing EAL-internal subsystems to use
> fastmem for their small-object allocations.
>
> Mattias Rönnblom (3):
> doc: add fastmem programming guide
> lib: add fastmem library
> app/test: add fastmem test suite
>
> app/test/meson.build | 3 +
> app/test/test_fastmem.c | 1682 +++++++++++++++++++++++++
> app/test/test_fastmem_perf.c | 997 +++++++++++++++
> app/test/test_fastmem_profile.c | 157 +++
> doc/api/doxy-api-index.md | 1 +
> doc/api/doxy-api.conf.in | 1 +
> doc/guides/prog_guide/fastmem_lib.rst | 301 +++++
> doc/guides/prog_guide/index.rst | 1 +
> lib/fastmem/meson.build | 6 +
> lib/fastmem/rte_fastmem.c | 1486 ++++++++++++++++++++++
> lib/fastmem/rte_fastmem.h | 644 ++++++++++
> lib/meson.build | 1 +
> 12 files changed, 5280 insertions(+)
> create mode 100644 app/test/test_fastmem.c
> create mode 100644 app/test/test_fastmem_perf.c
> create mode 100644 app/test/test_fastmem_profile.c
> create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
> create mode 100644 lib/fastmem/meson.build
> create mode 100644 lib/fastmem/rte_fastmem.c
> create mode 100644 lib/fastmem/rte_fastmem.h
>
Largish patchset so did AI review with full claude model.
Series review: [RFC 0/3] add fastmem allocator
Reviewed against the v1 RFC posted 2026-05-25.
[RFC 1/3] doc: add fastmem programming guide
Info: doc/guides/prog_guide/fastmem_lib.rst -- "\ No newline at end of file"
The new RST file does not end with a newline.
[RFC 2/3] lib: add fastmem library
Error: lib/fastmem/rte_fastmem.c -- use-after-free during rte_fastmem_deinit()
when caches were allocated cross-socket.
cache_create() places the cache struct on the *calling thread's* socket,
not on the socket the cache serves:
unsigned int own_socket = rte_socket_id();
...
alloc_socket = &fastmem->sockets[own_socket];
cache = bin_alloc_one(&alloc_socket->bins[cache_class]);
...
*slot = cache; /* slot is in socket K's caches[][] */
So an lcore on socket S that calls rte_fastmem_alloc_socket(..., K) with
S != K creates a cache whose memory lives in socket S's memzone but is
reachable through socket K's caches[lcore][class].
rte_fastmem_deinit() then walks sockets in index order:
for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
release_socket(&fastmem->sockets[i]);
and release_socket() does, in this order:
socket_release_caches(socket); /* (1) */
for (c...) bin_release(&socket->bins[c], socket); /* (2) */
for (i...) rte_memzone_free(socket->memzones[i]); /* (3) */
When i = S, step (3) frees socket S's memzones. When i = K (K > S),
socket_release_caches(K) runs:
cache_slab = slab_of(cache); /* in socket S's freed mz */
bin_free_one(cache_slab->bin, cache); /* reads cache_slab->bin */
cache_slab points into a freed memzone, so cache_slab->bin and the
subsequent push (slab->free_head = obj; slab->free_count++; in
bin_push_locked()) read and write released memory. slab_release() may
then re-attach the slab to socket S's free_head, which was zeroed and
whose backing is gone.
This is triggered by any application that allocates from a non-local
socket via SOCKET_ID_ANY fallback or explicit socket_id, which the
programming guide describes as a normal mode of operation. The
existing test_alloc_socket and test_alloc_socket_numa_placement use
rte_socket_id_by_idx(0) (the local socket) so the bug is not
exercised by the test suite.
Either order the teardown in three phases (all caches across all
sockets first, then all bins, then all memzones), or allocate the
cache struct from the socket it serves rather than the calling
thread's socket.
Warning: lib/fastmem/rte_fastmem.c -- non-atomic access to shared 64-bit
statistics counters.
cache->alloc_cache_hits, alloc_cache_misses, alloc_nomem,
free_cache_hits, free_cache_misses, and the bin counters
slab_acquires, slab_releases, slabs_partial, slabs_full are
incremented as plain C reads/writes by the owning lcore and read
from another thread via rte_fastmem_stats(), rte_fastmem_stats_class(),
rte_fastmem_stats_lcore(), and rte_fastmem_stats_lcore_class(). On
architectures where uint64_t is not naturally atomic (and per the C
standard generally) this is a data race; even on x86-64 it is
undefined behavior under -fsanitize=thread.
Use rte_atomic_fetch_add_explicit() with rte_memory_order_relaxed on
the producer side and rte_atomic_load_explicit() with relaxed
ordering on the reader side. Per AGENTS.md / the DPDK convention,
relaxed ordering is appropriate for these counters.
Warning: lib/fastmem/rte_fastmem.c -- pointer publish in cache_create()
without release ordering.
*slot = cache;
return cache;
The struct fields (count, capacity, target, the stats counters) are
written before this store but with no fence or release barrier. A
concurrent stats reader doing socket->caches[l][c] followed by
cache->* could observe the pointer but not all initialized fields.
Even ignoring the stats reader, rte_fastmem_cache_flush() invoked
from a different lcore on the same cache (not currently possible by
API contract, but the field is technically reachable) would race.
Pair with rte_atomic_store_explicit(..., rte_memory_order_release)
and a matching acquire load on the reader path.
Warning: lib/fastmem/rte_fastmem.c -- spurious ENOMEM window during slab
release.
bin_push_locked() removes a fully-drained slab from bin->partial
before bin_free_one() drops the bin lock; slab_release() then puts
it on socket->free_head under the socket lock. Between the unlock
and slab_release(), another lcore allocating in any class on the
same socket can see free_head == NULL, hit the memory_limit (or
FASTMEM_MAX_MEMZONES_PER_SOCKET) check in grow_socket(), and return
ENOMEM even though the slab is about to become available. Not a
correctness issue but visible to applications that pin tightly to
their limit.
Info: lib/fastmem/rte_fastmem.c local_socket_id() final fallback:
return (unsigned int)rte_socket_id_by_idx(0);
rte_socket_id_by_idx() returns int and is documented to return -1 on
error. If there are zero configured sockets the cast yields UINT_MAX
and fastmem->sockets[UINT_MAX] is out of bounds. Realistically there
is always at least one socket, but a defensive check (return 0, or
fail allocation explicitly) would avoid the corner case.
Info: lib/fastmem/rte_fastmem.c cache_pop() refills to cache->target
(half capacity) rather than to capacity. Subsequent single-object
allocs only get target-1 hits before the next bin trip. Likely
intentional for fairness with bulk callers, but worth a comment.
Info: lib/meson.build inserts 'fastmem' between 'dispatcher' and
'gpudev'. The natural alphabetical position is between 'efd' and
'fib'; fastmem has no dependency on dispatcher.
[RFC 3/3] app/test: add fastmem test suite
Warning: app/test/test_fastmem.c -- REGISTER_FAST_TEST uses NOHUGE_OK
but the functional tests need real memzone-backed memory.
REGISTER_FAST_TEST(fastmem_autotest, NOHUGE_OK, ASAN_OK,
test_fastmem);
test_fastmem runs both the lifecycle suite (no allocations) and the
functional suite, which requests 128 MiB IOVA-contiguous memzones.
In --no-huge mode IOVA-contiguous reservation of that size is not
reliable, so NOHUGE_SKIP is more honest. If you want the lifecycle
tests to remain no-huge-friendly, register them as a separate
test command.
Warning: app/test/test_fastmem.c -- the suite never exercises
cross-socket cache allocation.
test_alloc_socket and test_alloc_socket_numa_placement both use
rte_socket_id_by_idx(0) (the local socket). Add a test that runs on
a worker lcore whose rte_socket_id() differs from the target
socket_id passed to rte_fastmem_alloc_socket(), then calls
rte_fastmem_deinit(). This would have caught the deinit UAF above.
Info: app/test/test_fastmem.c -- several test functions declare an
uninitialized `int rc;` that is never read or written (e.g.
test_alloc_too_big, test_alloc_invalid_align, test_alloc_free_small,
test_alloc_alignment, test_alloc_socket, test_alloc_block_repurposing
and others). Drop the declarations.
Info: app/test/test_fastmem.c trailing blank-line clusters (two blank
lines before "return TEST_SUCCESS;" in test_reserve_multiple_memzones,
test_reserve_cumulative, test_reserve_invalid_socket,
test_reserve_any_socket, test_alloc_too_big, ...). Drop the extra
blank line.
^ permalink raw reply
* Re: [PATCH 2/2] eal/freebsd: implement virtual to IOVA translation fn
From: Lewis Donzis @ 2026-05-25 18:23 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev
In-Reply-To: <20260525171340.1701509-3-bruce.richardson@intel.com>
Tested with igc driver, which previously failed and now works properly.
----- On May 25, 2026, at 12:13 PM, Bruce Richardson bruce.richardson@intel.com wrote:
> The function rte_mem_virt2iova() function was always returning BAD_IOVA
> on FreeBSD as it was not implemented. Unfortunately, this function was
> used by some drivers - expecting it to work as they used addresses from
> rte_malloc calls. Implement this call on FreeBSD to fix things and
> enable more drivers on BSD. To verify the function works as expected
> add a basic unit test to the memory_autotest for it.
>
> At the same time, we can trivially use this to implement the
> rte_mem_virt2phy function for the case of using contigmem where IOVAs
> are physical addresses.
>
> Reported-by: Lewis Donzis <lew@perftech.com>
Tested-by: Lewis Donzis <lew@perftech.com>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>
> NOTE: This is one of those patches that could be either a bugfix or a
> feature. For now, I'm treating it as adding new feature support, though
> in doing so it enables previously broken drivers.
> ---
> app/test/test_memory.c | 67 ++++++++++++++++++++++++++++++++++++
> lib/eal/freebsd/eal_memory.c | 22 +++++++++---
> 2 files changed, 84 insertions(+), 5 deletions(-)
>
> diff --git a/app/test/test_memory.c b/app/test/test_memory.c
> index 628714c0b8..260d5ed8f8 100644
> --- a/app/test/test_memory.c
> +++ b/app/test/test_memory.c
> @@ -8,6 +8,7 @@
> #include <rte_eal.h>
> #include <rte_errno.h>
> #include <rte_memory.h>
> +#include <rte_malloc.h>
> #include <rte_common.h>
> #include <rte_memzone.h>
>
> @@ -75,6 +76,68 @@ check_seg_fds(const struct rte_memseg_list *msl, const struct
> rte_memseg *ms,
> return 0;
> }
>
> +static int
> +check_malloc_virt2iova(void)
> +{
> + const size_t alloc_sz = 128;
> + const size_t off = 32;
> + struct rte_memseg *ms;
> + char *p;
> + rte_iova_t iova, iova_off;
> +
> + p = rte_malloc("memory_autotest", alloc_sz, RTE_CACHE_LINE_SIZE);
> + if (p == NULL) {
> + printf("rte_malloc failed\n");
> + return -1;
> + }
> +
> + iova = rte_mem_virt2iova(p);
> + if (iova == RTE_BAD_IOVA) {
> + printf("rte_mem_virt2iova failed for rte_malloc pointer\n");
> + rte_free(p);
> + return -1;
> + }
> +
> + ms = rte_mem_virt2memseg(p, NULL);
> + if (ms == NULL) {
> + printf("failed to resolve memseg for rte_malloc pointer\n");
> + rte_free(p);
> + return -1;
> + }
> +
> + if (rte_eal_iova_mode() == RTE_IOVA_PA) {
> + if (ms->iova == RTE_BAD_IOVA || iova < ms->iova ||
> + iova >= ms->iova + ms->len) {
> + printf("translated iova is outside memseg range\n");
> + rte_free(p);
> + return -1;
> + }
> +
> + phys_addr_t physaddr = rte_mem_virt2phy(p);
> + if (physaddr == RTE_BAD_PHYS_ADDR || physaddr != iova) {
> + printf("rte_mem_virt2phy failed for rte_malloc pointer\n");
> + rte_free(p);
> + return -1;
> + }
> + } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
> + if (iova != (uintptr_t)p) {
> + printf("rte_mem_virt2iova did not return VA in VA mode\n");
> + rte_free(p);
> + return -1;
> + }
> + }
> +
> + iova_off = rte_mem_virt2iova(p + off);
> + if (iova_off == RTE_BAD_IOVA || iova_off != iova + off) {
> + printf("translated iova for interior pointer is invalid\n");
> + rte_free(p);
> + return -1;
> + }
> +
> + rte_free(p);
> + return 0;
> +}
> +
> static int
> test_memory(void)
> {
> @@ -107,6 +170,10 @@ test_memory(void)
> return -1;
> }
>
> + ret = check_malloc_virt2iova();
> + if (ret < 0)
> + return ret;
> +
> return 0;
> }
>
> diff --git a/lib/eal/freebsd/eal_memory.c b/lib/eal/freebsd/eal_memory.c
> index e56d149a22..6952b38630 100644
> --- a/lib/eal/freebsd/eal_memory.c
> +++ b/lib/eal/freebsd/eal_memory.c
> @@ -40,16 +40,28 @@ RTE_EXPORT_SYMBOL(rte_mem_virt2phy)
> phys_addr_t
> rte_mem_virt2phy(const void *virtaddr)
> {
> - /* XXX not implemented. This function is only used by
> - * rte_mempool_virt2iova() when hugepages are disabled. */
> - (void)virtaddr;
> - return RTE_BAD_IOVA;
> + /* not implemented for FreeBSD when not using contigmem memory */
> + if (virtaddr == NULL || rte_eal_iova_mode() != RTE_IOVA_PA)
> + return RTE_BAD_IOVA;
> + /* when IOVA == PA, return the IOVA */
> + return rte_mem_virt2iova(virtaddr);
> }
> +
> RTE_EXPORT_SYMBOL(rte_mem_virt2iova)
> rte_iova_t
> rte_mem_virt2iova(const void *virtaddr)
> {
> - return rte_mem_virt2phy(virtaddr);
> + if (virtaddr == NULL)
> + return RTE_BAD_IOVA;
> +
> + if (rte_eal_iova_mode() == RTE_IOVA_VA)
> + return (uintptr_t)virtaddr;
> +
> + const struct rte_memseg *ms = rte_mem_virt2memseg(virtaddr, NULL);
> + if (ms != NULL && ms->iova != RTE_BAD_IOVA)
> + return ms->iova + RTE_PTR_DIFF(virtaddr, ms->addr);
> +
> + return RTE_BAD_IOVA;
> }
>
> int
> --
> 2.53.0
^ permalink raw reply
* [PATCH 2/2] eal/freebsd: implement virtual to IOVA translation fn
From: Bruce Richardson @ 2026-05-25 17:13 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Lewis Donzis
In-Reply-To: <20260525171340.1701509-1-bruce.richardson@intel.com>
The function rte_mem_virt2iova() function was always returning BAD_IOVA
on FreeBSD as it was not implemented. Unfortunately, this function was
used by some drivers - expecting it to work as they used addresses from
rte_malloc calls. Implement this call on FreeBSD to fix things and
enable more drivers on BSD. To verify the function works as expected
add a basic unit test to the memory_autotest for it.
At the same time, we can trivially use this to implement the
rte_mem_virt2phy function for the case of using contigmem where IOVAs
are physical addresses.
Reported-by: Lewis Donzis <lew@perftech.com>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
NOTE: This is one of those patches that could be either a bugfix or a
feature. For now, I'm treating it as adding new feature support, though
in doing so it enables previously broken drivers.
---
app/test/test_memory.c | 67 ++++++++++++++++++++++++++++++++++++
lib/eal/freebsd/eal_memory.c | 22 +++++++++---
2 files changed, 84 insertions(+), 5 deletions(-)
diff --git a/app/test/test_memory.c b/app/test/test_memory.c
index 628714c0b8..260d5ed8f8 100644
--- a/app/test/test_memory.c
+++ b/app/test/test_memory.c
@@ -8,6 +8,7 @@
#include <rte_eal.h>
#include <rte_errno.h>
#include <rte_memory.h>
+#include <rte_malloc.h>
#include <rte_common.h>
#include <rte_memzone.h>
@@ -75,6 +76,68 @@ check_seg_fds(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
return 0;
}
+static int
+check_malloc_virt2iova(void)
+{
+ const size_t alloc_sz = 128;
+ const size_t off = 32;
+ struct rte_memseg *ms;
+ char *p;
+ rte_iova_t iova, iova_off;
+
+ p = rte_malloc("memory_autotest", alloc_sz, RTE_CACHE_LINE_SIZE);
+ if (p == NULL) {
+ printf("rte_malloc failed\n");
+ return -1;
+ }
+
+ iova = rte_mem_virt2iova(p);
+ if (iova == RTE_BAD_IOVA) {
+ printf("rte_mem_virt2iova failed for rte_malloc pointer\n");
+ rte_free(p);
+ return -1;
+ }
+
+ ms = rte_mem_virt2memseg(p, NULL);
+ if (ms == NULL) {
+ printf("failed to resolve memseg for rte_malloc pointer\n");
+ rte_free(p);
+ return -1;
+ }
+
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ if (ms->iova == RTE_BAD_IOVA || iova < ms->iova ||
+ iova >= ms->iova + ms->len) {
+ printf("translated iova is outside memseg range\n");
+ rte_free(p);
+ return -1;
+ }
+
+ phys_addr_t physaddr = rte_mem_virt2phy(p);
+ if (physaddr == RTE_BAD_PHYS_ADDR || physaddr != iova) {
+ printf("rte_mem_virt2phy failed for rte_malloc pointer\n");
+ rte_free(p);
+ return -1;
+ }
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ if (iova != (uintptr_t)p) {
+ printf("rte_mem_virt2iova did not return VA in VA mode\n");
+ rte_free(p);
+ return -1;
+ }
+ }
+
+ iova_off = rte_mem_virt2iova(p + off);
+ if (iova_off == RTE_BAD_IOVA || iova_off != iova + off) {
+ printf("translated iova for interior pointer is invalid\n");
+ rte_free(p);
+ return -1;
+ }
+
+ rte_free(p);
+ return 0;
+}
+
static int
test_memory(void)
{
@@ -107,6 +170,10 @@ test_memory(void)
return -1;
}
+ ret = check_malloc_virt2iova();
+ if (ret < 0)
+ return ret;
+
return 0;
}
diff --git a/lib/eal/freebsd/eal_memory.c b/lib/eal/freebsd/eal_memory.c
index e56d149a22..6952b38630 100644
--- a/lib/eal/freebsd/eal_memory.c
+++ b/lib/eal/freebsd/eal_memory.c
@@ -40,16 +40,28 @@ RTE_EXPORT_SYMBOL(rte_mem_virt2phy)
phys_addr_t
rte_mem_virt2phy(const void *virtaddr)
{
- /* XXX not implemented. This function is only used by
- * rte_mempool_virt2iova() when hugepages are disabled. */
- (void)virtaddr;
- return RTE_BAD_IOVA;
+ /* not implemented for FreeBSD when not using contigmem memory */
+ if (virtaddr == NULL || rte_eal_iova_mode() != RTE_IOVA_PA)
+ return RTE_BAD_IOVA;
+ /* when IOVA == PA, return the IOVA */
+ return rte_mem_virt2iova(virtaddr);
}
+
RTE_EXPORT_SYMBOL(rte_mem_virt2iova)
rte_iova_t
rte_mem_virt2iova(const void *virtaddr)
{
- return rte_mem_virt2phy(virtaddr);
+ if (virtaddr == NULL)
+ return RTE_BAD_IOVA;
+
+ if (rte_eal_iova_mode() == RTE_IOVA_VA)
+ return (uintptr_t)virtaddr;
+
+ const struct rte_memseg *ms = rte_mem_virt2memseg(virtaddr, NULL);
+ if (ms != NULL && ms->iova != RTE_BAD_IOVA)
+ return ms->iova + RTE_PTR_DIFF(virtaddr, ms->addr);
+
+ return RTE_BAD_IOVA;
}
int
--
2.53.0
^ permalink raw reply related
* [PATCH 1/2] eal/freebsd: fix memseg addresses in EAL VA mode
From: Bruce Richardson @ 2026-05-25 17:13 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, stable
In-Reply-To: <20260525171340.1701509-1-bruce.richardson@intel.com>
In BSD, when mapping the contigmem segments, the iova is unconditionally
set to the physical address, irrespective of the actual IOVA mode.
Change this to use virtual addresses when iova mode is IOVA_AS_VA.
Fixes: 764bf26873b9 ("add FreeBSD support")
Cc: stable@dpdk.org
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
lib/eal/freebsd/eal_memory.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/lib/eal/freebsd/eal_memory.c b/lib/eal/freebsd/eal_memory.c
index cd608db9f9..e56d149a22 100644
--- a/lib/eal/freebsd/eal_memory.c
+++ b/lib/eal/freebsd/eal_memory.c
@@ -214,7 +214,10 @@ rte_eal_hugepage_init(void)
}
seg->addr = addr;
- seg->iova = physaddr;
+ if (rte_eal_iova_mode() == RTE_IOVA_VA)
+ seg->iova = (uintptr_t)addr;
+ else
+ seg->iova = physaddr;
seg->hugepage_sz = page_sz;
seg->len = page_sz;
seg->nchannel = mcfg->nchannel;
--
2.53.0
^ permalink raw reply related
* [PATCH 0/2] implement missing APIs on FreeBSD
From: Bruce Richardson @ 2026-05-25 17:13 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson
The APIs for converting a virtual address to an IOVA or a physical
address were implemented only as stubs on FreeBSD, but unfortunately
they are used by drivers such as "igc", part of e1000, causing the
drivers to fail on BSD. A workaround would be to change the use of these
functions with the rte_malloc equivalents (e.g. rte_malloc_virt2iova) -
which do work on FreeBSD - but a better solution is just to replace the
stubs with proper implementations in FreeBSD-specific EAL flags.
Bruce Richardson (2):
eal/freebsd: fix memseg addresses in IOVA_AS_VA mode
eal/freebsd: implement virtual to IOVA translation fn
app/test/test_memory.c | 67 ++++++++++++++++++++++++++++++++++++
lib/eal/freebsd/eal_memory.c | 27 +++++++++++----
2 files changed, 88 insertions(+), 6 deletions(-)
--
2.53.0
^ permalink raw reply
* RE: [PATCH v3 03/27] ring: use compare-and-swap wrapper
From: Morten Brørup @ 2026-05-25 15:47 UTC (permalink / raw)
To: Stephen Hemminger, Konstantin Ananyev; +Cc: dev
In-Reply-To: <20260525083509.00cf76a1@phoenix.local>
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Monday, 25 May 2026 17.35
>
> On Mon, 25 May 2026 07:41:13 +0000
> Konstantin Ananyev <konstantin.ananyev@huawei.com> wrote:
>
> > Hi Stephen,
> >
> > > The rte_atomic32_cmpset is deprecated. Initial attempts at
> > > changing this with direct conversion to
> > > rte_atomic_compare_exchange_weak_explicit()
> > > regressed MP/MC contended performance on x86 by 10-30%,
> > > because the C11 builtin's failure-writeback semantic forces
> > > GCC to emit extra instructions on the CAS critical path.
> > >
> > > Add an internal __rte_ring_compare_and_swap() wrapper that calls
> > > __sync_bool_compare_and_swap() directly, which keeps the original
> > > instruction sequence. Add equivalent function for MSVC.
> >
> > In fact, in rte_ring we do have 2 implementations of the same core
> functions:
> > lib/ring/rte_ring_c11_pvt.h - uses C11 atomics
> > lib/ring/rte_ring_generic_pvt.h - uses legacy instructions (smp_mb,
> extra),
> > If we going remove these legacy instructions anyway (or
> reimplementing them using C11 atomics),
> > then there is probably no point to keep rte_ring_generic_pvt.h.
> > Konstantin
>
> Have been deep diving into why C11 atomics give 20-30% performance
> drop versus atomic32 version. So far it comes down to GCC optimizer
> not doing as well with C11 versus assembly. The C11 form with the
> excessive use of always_inline consumes more registers.
Just an idea:
Perhaps adding "const" and/or "restrict" to relevant parameters will give the optimizer the information it needs?
^ permalink raw reply
* Re: [PATCH v3 03/27] ring: use compare-and-swap wrapper
From: Stephen Hemminger @ 2026-05-25 15:35 UTC (permalink / raw)
To: Konstantin Ananyev; +Cc: dev@dpdk.org
In-Reply-To: <b827bdd2f5994f24b9806e5cbae82cd3@huawei.com>
On Mon, 25 May 2026 07:41:13 +0000
Konstantin Ananyev <konstantin.ananyev@huawei.com> wrote:
> Hi Stephen,
>
> > The rte_atomic32_cmpset is deprecated. Initial attempts at
> > changing this with direct conversion to
> > rte_atomic_compare_exchange_weak_explicit()
> > regressed MP/MC contended performance on x86 by 10-30%,
> > because the C11 builtin's failure-writeback semantic forces
> > GCC to emit extra instructions on the CAS critical path.
> >
> > Add an internal __rte_ring_compare_and_swap() wrapper that calls
> > __sync_bool_compare_and_swap() directly, which keeps the original
> > instruction sequence. Add equivalent function for MSVC.
>
> In fact, in rte_ring we do have 2 implementations of the same core functions:
> lib/ring/rte_ring_c11_pvt.h - uses C11 atomics
> lib/ring/rte_ring_generic_pvt.h - uses legacy instructions (smp_mb, extra),
> If we going remove these legacy instructions anyway (or reimplementing them using C11 atomics),
> then there is probably no point to keep rte_ring_generic_pvt.h.
> Konstantin
Have been deep diving into why C11 atomics give 20-30% performance
drop versus atomic32 version. So far it comes down to GCC optimizer
not doing as well with C11 versus assembly. The C11 form with the
excessive use of always_inline consumes more registers.
^ permalink raw reply
* Re: [PATCH v3 03/27] ring: use compare-and-swap wrapper
From: Stephen Hemminger @ 2026-05-25 14:31 UTC (permalink / raw)
To: Konstantin Ananyev; +Cc: dev@dpdk.org
In-Reply-To: <b827bdd2f5994f24b9806e5cbae82cd3@huawei.com>
On Mon, 25 May 2026 07:41:13 +0000
Konstantin Ananyev <konstantin.ananyev@huawei.com> wrote:
> In fact, in rte_ring we do have 2 implementations of the same core functions:
> lib/ring/rte_ring_c11_pvt.h - uses C11 atomics
> lib/ring/rte_ring_generic_pvt.h - uses legacy instructions (smp_mb, extra),
> If we going remove these legacy instructions anyway (or reimplementing them using C11 atomics),
> then there is probably no point to keep rte_ring_generic_pvt.h.
> Konstantin
Good point will try heading that way.
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Stephen Hemminger @ 2026-05-25 14:30 UTC (permalink / raw)
To: Mattias Rönnblom
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260525103642.55255-1-hofors@lysator.liu.se>
On Mon, 25 May 2026 12:36:39 +0200
Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> This RFC introduces fastmem, a general-purpose small-object allocator
> for DPDK. It is intended to replace per-type mempools with a single
> allocator that handles arbitrary sizes, grows on demand, and matches
> mempool-level performance on the hot path.
Makes sense, what a simple wrapper inline to allow full replacement
testing/performance A/B comparison?
^ permalink raw reply
* [PATCH v5 27/27] net/ice: use common action checks for ACL
From: Anatoly Burakov @ 2026-05-25 14:06 UTC (permalink / raw)
To: dev, Bruce Richardson
In-Reply-To: <cover.1779717943.git.anatoly.burakov@intel.com>
From: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
Use the common flow action checking parsing infrastructure for checking
flow actions for ACL filter.
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
drivers/net/intel/ice/ice_acl_filter.c | 143 +++++++++++++++----------
1 file changed, 84 insertions(+), 59 deletions(-)
diff --git a/drivers/net/intel/ice/ice_acl_filter.c b/drivers/net/intel/ice/ice_acl_filter.c
index 0b50f06c46..30ff1254c9 100644
--- a/drivers/net/intel/ice/ice_acl_filter.c
+++ b/drivers/net/intel/ice/ice_acl_filter.c
@@ -645,60 +645,6 @@ ice_acl_filter_free(struct rte_flow *flow)
flow->rule = NULL;
}
-static int
-ice_acl_parse_action(__rte_unused struct ice_adapter *ad,
- const struct rte_flow_action actions[],
- struct rte_flow_error *error,
- struct ice_acl_conf *filter)
-{
- struct ice_pf *pf = &ad->pf;
- const struct rte_flow_action_queue *act_q;
- uint32_t dest_num = 0;
-
- for (; actions->type != RTE_FLOW_ACTION_TYPE_END; actions++) {
- switch (actions->type) {
- case RTE_FLOW_ACTION_TYPE_VOID:
- break;
- case RTE_FLOW_ACTION_TYPE_DROP:
- dest_num++;
-
- filter->input.dest_ctl =
- ICE_FLTR_PRGM_DESC_DEST_DROP_PKT;
- break;
- case RTE_FLOW_ACTION_TYPE_QUEUE:
- dest_num++;
-
- act_q = actions->conf;
- filter->input.q_index = act_q->index;
- if (filter->input.q_index >=
- pf->dev_data->nb_rx_queues) {
- rte_flow_error_set(error, EINVAL,
- RTE_FLOW_ERROR_TYPE_ACTION,
- actions,
- "Invalid queue for FDIR.");
- return -rte_errno;
- }
- filter->input.dest_ctl =
- ICE_FLTR_PRGM_DESC_DEST_DIRECT_PKT_QINDEX;
- break;
- default:
- rte_flow_error_set(error, EINVAL,
- RTE_FLOW_ERROR_TYPE_ACTION, actions,
- "Invalid action.");
- return -rte_errno;
- }
- }
-
- if (dest_num == 0 || dest_num >= 2) {
- rte_flow_error_set(error, EINVAL,
- RTE_FLOW_ERROR_TYPE_ACTION, actions,
- "Unsupported action combination");
- return -rte_errno;
- }
-
- return 0;
-}
-
static int
ice_acl_parse_pattern(__rte_unused struct ice_adapter *ad,
const struct rte_flow_item pattern[],
@@ -966,6 +912,69 @@ ice_acl_parse_pattern(__rte_unused struct ice_adapter *ad,
return 0;
}
+static int
+ice_acl_parse_action(const struct ci_flow_actions *actions,
+ struct ice_acl_conf *filter,
+ struct rte_flow_error *error)
+{
+ const struct rte_flow_action *act = actions->actions[0];
+
+ switch (act->type) {
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ filter->input.dest_ctl =
+ ICE_FLTR_PRGM_DESC_DEST_DROP_PKT;
+ break;
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ {
+ const struct rte_flow_action_queue *act_q = act->conf;
+
+ filter->input.q_index = act_q->index;
+ filter->input.dest_ctl =
+ ICE_FLTR_PRGM_DESC_DEST_DIRECT_PKT_QINDEX;
+ break;
+ }
+ default:
+ return rte_flow_error_set(error, EINVAL,
+ RTE_FLOW_ERROR_TYPE_ACTION, act,
+ "Invalid action.");
+ }
+
+ return 0;
+}
+
+static int
+ice_acl_parse_action_check(const struct ci_flow_actions *actions,
+ const struct ci_flow_actions_check_param *param,
+ struct rte_flow_error *error)
+{
+ struct ice_adapter *ad = param->driver_ctx;
+ struct ice_pf *pf = &ad->pf;
+ const struct rte_flow_action *act = actions->actions[0];
+
+ switch (act->type) {
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ break;
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ {
+ const struct rte_flow_action_queue *act_q = act->conf;
+
+ if (act_q->index >= pf->dev_data->nb_rx_queues) {
+ return rte_flow_error_set(error, EINVAL,
+ RTE_FLOW_ERROR_TYPE_ACTION, act,
+ "Invalid queue for ACL.");
+ }
+ break;
+ }
+ default:
+ /* shouldn't happen */
+ return rte_flow_error_set(error, EINVAL,
+ RTE_FLOW_ERROR_TYPE_ACTION, act,
+ "Invalid action.");
+ }
+
+ return 0;
+}
+
static int
ice_acl_parse(struct ice_adapter *ad,
struct ice_pattern_match_item *array,
@@ -976,17 +985,33 @@ ice_acl_parse(struct ice_adapter *ad,
void **meta,
struct rte_flow_error *error)
{
+ struct ci_flow_actions parsed_actions = {0};
+ struct ci_flow_actions_check_param param = {
+ .allowed_types = (enum rte_flow_action_type[]){
+ RTE_FLOW_ACTION_TYPE_DROP,
+ RTE_FLOW_ACTION_TYPE_QUEUE,
+ RTE_FLOW_ACTION_TYPE_END
+ },
+ .max_actions = 1,
+ .check = ice_acl_parse_action_check,
+ .driver_ctx = ad,
+ };
struct ice_pf *pf = &ad->pf;
struct ice_acl_conf *filter = &pf->acl.conf;
struct ice_pattern_match_item *item = NULL;
uint64_t input_set;
int ret;
- ret = ci_flow_check_attr(attr, NULL, error);
- if (ret)
- return ret;
-
memset(filter, 0, sizeof(*filter));
+
+ ret = ci_flow_check_attr(attr, NULL, error);
+ if (ret)
+ return ret;
+
+ ret = ci_flow_check_actions(actions, ¶m, &parsed_actions, error);
+ if (ret)
+ return ret;
+
item = ice_search_pattern_match_item(ad, pattern, array, array_len,
error);
if (!item)
@@ -1005,7 +1030,7 @@ ice_acl_parse(struct ice_adapter *ad,
goto error;
}
- ret = ice_acl_parse_action(ad, actions, error, filter);
+ ret = ice_acl_parse_action(&parsed_actions, filter, error);
if (ret)
goto error;
--
2.47.3
^ permalink raw reply related
* [PATCH v5 26/27] net/ice: use common action checks for switch
From: Anatoly Burakov @ 2026-05-25 14:06 UTC (permalink / raw)
To: dev, Bruce Richardson
In-Reply-To: <cover.1779717943.git.anatoly.burakov@intel.com>
From: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
Use the common flow action checking parsing infrastructure for checking
flow actions for switch filter.
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
drivers/net/intel/ice/ice_switch_filter.c | 371 +++++++++++-----------
1 file changed, 185 insertions(+), 186 deletions(-)
diff --git a/drivers/net/intel/ice/ice_switch_filter.c b/drivers/net/intel/ice/ice_switch_filter.c
index 67943b591c..5d9e0062af 100644
--- a/drivers/net/intel/ice/ice_switch_filter.c
+++ b/drivers/net/intel/ice/ice_switch_filter.c
@@ -35,6 +35,8 @@
#define ICE_IPV4_PROTO_NVGRE 0x002F
#define ICE_SW_PRI_BASE 6
+#define ICE_SW_MAX_QUEUES 128
+
#define ICE_SW_INSET_ETHER ( \
ICE_INSET_DMAC | ICE_INSET_SMAC | ICE_INSET_ETHERTYPE)
#define ICE_SW_INSET_MAC_VLAN ( \
@@ -1527,85 +1529,38 @@ ice_switch_parse_pattern(const struct rte_flow_item pattern[],
}
static int
-ice_switch_parse_dcf_action(struct ice_dcf_adapter *ad,
- const struct rte_flow_action *actions,
+ice_switch_parse_dcf_action(const struct rte_flow_action *action,
uint32_t priority,
struct rte_flow_error *error,
struct ice_adv_rule_info *rule_info)
{
const struct rte_flow_action_ethdev *act_ethdev;
- const struct rte_flow_action *action;
const struct rte_eth_dev *repr_dev;
enum rte_flow_action_type action_type;
- uint16_t rule_port_id, backer_port_id;
- for (action = actions; action->type !=
- RTE_FLOW_ACTION_TYPE_END; action++) {
- action_type = action->type;
- switch (action_type) {
- case RTE_FLOW_ACTION_TYPE_PORT_REPRESENTOR:
- rule_info->sw_act.fltr_act = ICE_FWD_TO_VSI;
- act_ethdev = action->conf;
-
- if (!rte_eth_dev_is_valid_port(act_ethdev->port_id))
- goto invalid_port_id;
-
- /* For traffic to original DCF port */
- rule_port_id = ad->parent.pf.dev_data->port_id;
-
- if (rule_port_id != act_ethdev->port_id)
- goto invalid_port_id;
-
- rule_info->sw_act.vsi_handle = 0;
-
- break;
-
-invalid_port_id:
- rte_flow_error_set(error,
- EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
- actions,
- "Invalid port_id");
- return -rte_errno;
-
- case RTE_FLOW_ACTION_TYPE_REPRESENTED_PORT:
- rule_info->sw_act.fltr_act = ICE_FWD_TO_VSI;
- act_ethdev = action->conf;
-
- if (!rte_eth_dev_is_valid_port(act_ethdev->port_id))
- goto invalid;
-
- repr_dev = &rte_eth_devices[act_ethdev->port_id];
-
- if (!repr_dev->data)
- goto invalid;
-
- rule_port_id = ad->parent.pf.dev_data->port_id;
- backer_port_id = repr_dev->data->backer_port_id;
-
- if (backer_port_id != rule_port_id)
- goto invalid;
-
- rule_info->sw_act.vsi_handle = repr_dev->data->representor_id;
- break;
-
-invalid:
- rte_flow_error_set(error,
- EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
- actions,
- "Invalid ethdev_port_id");
- return -rte_errno;
-
- case RTE_FLOW_ACTION_TYPE_DROP:
- rule_info->sw_act.fltr_act = ICE_DROP_PACKET;
- break;
-
- default:
- rte_flow_error_set(error,
- EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
- actions,
- "Invalid action type");
- return -rte_errno;
- }
+ action_type = action->type;
+ switch (action_type) {
+ case RTE_FLOW_ACTION_TYPE_PORT_REPRESENTOR:
+ rule_info->sw_act.fltr_act = ICE_FWD_TO_VSI;
+ rule_info->sw_act.vsi_handle = 0;
+ break;
+
+ case RTE_FLOW_ACTION_TYPE_REPRESENTED_PORT:
+ rule_info->sw_act.fltr_act = ICE_FWD_TO_VSI;
+ act_ethdev = action->conf;
+ repr_dev = &rte_eth_devices[act_ethdev->port_id];
+ rule_info->sw_act.vsi_handle = repr_dev->data->representor_id;
+ break;
+
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ rule_info->sw_act.fltr_act = ICE_DROP_PACKET;
+ break;
+
+ default:
+ /* Should never reach */
+ rte_flow_error_set(error, EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
+ action, "Invalid action type");
+ return -rte_errno;
}
rule_info->sw_act.src = rule_info->sw_act.vsi_handle;
@@ -1621,73 +1576,38 @@ ice_switch_parse_dcf_action(struct ice_dcf_adapter *ad,
static int
ice_switch_parse_action(struct ice_pf *pf,
- const struct rte_flow_action *actions,
+ const struct rte_flow_action *action,
uint32_t priority,
struct rte_flow_error *error,
struct ice_adv_rule_info *rule_info)
{
struct ice_vsi *vsi = pf->main_vsi;
- struct rte_eth_dev_data *dev_data = pf->adapter->pf.dev_data;
const struct rte_flow_action_queue *act_q;
const struct rte_flow_action_rss *act_qgrop;
- uint16_t base_queue, i;
- const struct rte_flow_action *action;
+ uint16_t base_queue;
enum rte_flow_action_type action_type;
- uint16_t valid_qgrop_number[MAX_QGRP_NUM_TYPE] = {
- 2, 4, 8, 16, 32, 64, 128};
base_queue = pf->base_queue + vsi->base_queue;
- for (action = actions; action->type !=
- RTE_FLOW_ACTION_TYPE_END; action++) {
- action_type = action->type;
- switch (action_type) {
- case RTE_FLOW_ACTION_TYPE_RSS:
- act_qgrop = action->conf;
- if (act_qgrop->queue_num <= 1)
- goto error;
- rule_info->sw_act.fltr_act =
- ICE_FWD_TO_QGRP;
- rule_info->sw_act.fwd_id.q_id =
- base_queue + act_qgrop->queue[0];
- for (i = 0; i < MAX_QGRP_NUM_TYPE; i++) {
- if (act_qgrop->queue_num ==
- valid_qgrop_number[i])
- break;
- }
- if (i == MAX_QGRP_NUM_TYPE)
- goto error;
- if ((act_qgrop->queue[0] +
- act_qgrop->queue_num) >
- dev_data->nb_rx_queues)
- goto error1;
- for (i = 0; i < act_qgrop->queue_num - 1; i++)
- if (act_qgrop->queue[i + 1] !=
- act_qgrop->queue[i] + 1)
- goto error2;
- rule_info->sw_act.qgrp_size =
- act_qgrop->queue_num;
- break;
- case RTE_FLOW_ACTION_TYPE_QUEUE:
- act_q = action->conf;
- if (act_q->index >= dev_data->nb_rx_queues)
- goto error;
- rule_info->sw_act.fltr_act =
- ICE_FWD_TO_Q;
- rule_info->sw_act.fwd_id.q_id =
- base_queue + act_q->index;
- break;
-
- case RTE_FLOW_ACTION_TYPE_DROP:
- rule_info->sw_act.fltr_act =
- ICE_DROP_PACKET;
- break;
-
- case RTE_FLOW_ACTION_TYPE_VOID:
- break;
-
- default:
- goto error;
- }
+ action_type = action->type;
+ switch (action_type) {
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ act_qgrop = action->conf;
+ rule_info->sw_act.fltr_act = ICE_FWD_TO_QGRP;
+ rule_info->sw_act.fwd_id.q_id = base_queue + act_qgrop->queue[0];
+ rule_info->sw_act.qgrp_size = act_qgrop->queue_num;
+ break;
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ act_q = action->conf;
+ rule_info->sw_act.fltr_act = ICE_FWD_TO_Q;
+ rule_info->sw_act.fwd_id.q_id = base_queue + act_q->index;
+ break;
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ rule_info->sw_act.fltr_act = ICE_DROP_PACKET;
+ break;
+ default:
+ rte_flow_error_set(error, EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
+ action, "Invalid action type or queue number");
+ return -rte_errno;
}
rule_info->sw_act.vsi_handle = vsi->idx;
@@ -1699,65 +1619,120 @@ ice_switch_parse_action(struct ice_pf *pf,
rule_info->priority = ICE_SW_PRI_BASE - priority;
return 0;
-
-error:
- rte_flow_error_set(error,
- EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
- actions,
- "Invalid action type or queue number");
- return -rte_errno;
-
-error1:
- rte_flow_error_set(error,
- EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
- actions,
- "Invalid queue region indexes");
- return -rte_errno;
-
-error2:
- rte_flow_error_set(error,
- EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
- actions,
- "Discontinuous queue region");
- return -rte_errno;
}
static int
-ice_switch_check_action(const struct rte_flow_action *actions,
- struct rte_flow_error *error)
+ice_switch_dcf_action_check(const struct ci_flow_actions *actions,
+ const struct ci_flow_actions_check_param *param,
+ struct rte_flow_error *error)
{
+ struct ice_dcf_adapter *ad = param->driver_ctx;
const struct rte_flow_action *action;
enum rte_flow_action_type action_type;
- uint16_t actions_num = 0;
-
- for (action = actions; action->type !=
- RTE_FLOW_ACTION_TYPE_END; action++) {
- action_type = action->type;
- switch (action_type) {
- case RTE_FLOW_ACTION_TYPE_RSS:
- case RTE_FLOW_ACTION_TYPE_QUEUE:
- case RTE_FLOW_ACTION_TYPE_DROP:
- case RTE_FLOW_ACTION_TYPE_REPRESENTED_PORT:
- case RTE_FLOW_ACTION_TYPE_PORT_REPRESENTOR:
- actions_num++;
- break;
- case RTE_FLOW_ACTION_TYPE_VOID:
- continue;
- default:
- rte_flow_error_set(error,
- EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
- actions,
- "Invalid action type");
- return -rte_errno;
+ const struct rte_flow_action_ethdev *act_ethdev;
+ const struct rte_eth_dev *repr_dev;
+
+ action = actions->actions[0];
+ action_type = action->type;
+
+ switch (action_type) {
+ case RTE_FLOW_ACTION_TYPE_PORT_REPRESENTOR:
+ case RTE_FLOW_ACTION_TYPE_REPRESENTED_PORT:
+ {
+ uint16_t expected_port_id, backer_port_id;
+ act_ethdev = action->conf;
+
+ if (!rte_eth_dev_is_valid_port(act_ethdev->port_id))
+ goto invalid_port_id;
+
+ expected_port_id = ad->parent.pf.dev_data->port_id;
+
+ if (action_type == RTE_FLOW_ACTION_TYPE_PORT_REPRESENTOR) {
+ if (expected_port_id != act_ethdev->port_id)
+ goto invalid_port_id;
+ } else {
+ repr_dev = &rte_eth_devices[act_ethdev->port_id];
+
+ if (!repr_dev->data)
+ goto invalid_port_id;
+
+ backer_port_id = repr_dev->data->backer_port_id;
+
+ if (backer_port_id != expected_port_id)
+ goto invalid_port_id;
}
+
+ break;
+invalid_port_id:
+ return rte_flow_error_set(error, EINVAL,
+ RTE_FLOW_ERROR_TYPE_ACTION, action,
+ "Invalid port ID");
+ }
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ break;
+ default:
+ /* Should never reach */
+ return rte_flow_error_set(error, EINVAL,
+ RTE_FLOW_ERROR_TYPE_ACTION, action,
+ "Invalid action type");
}
- if (actions_num != 1) {
- rte_flow_error_set(error,
- EINVAL, RTE_FLOW_ERROR_TYPE_ACTION,
- actions,
- "Invalid action number");
- return -rte_errno;
+ return 0;
+}
+
+static int
+ice_switch_action_check(const struct ci_flow_actions *actions,
+ const struct ci_flow_actions_check_param *param,
+ struct rte_flow_error *error)
+{
+ struct ice_adapter *ad = param->driver_ctx;
+ struct ice_pf *pf = &ad->pf;
+ struct rte_eth_dev_data *dev_data = pf->dev_data;
+ const struct rte_flow_action *action = actions->actions[0];
+
+ switch (action->type) {
+ case RTE_FLOW_ACTION_TYPE_RSS:
+ {
+ const struct rte_flow_action_rss *act_qgrop;
+ act_qgrop = action->conf;
+
+ /* Check bounds on number of queues */
+ if (act_qgrop->queue_num < 2 || act_qgrop->queue_num > ICE_SW_MAX_QUEUES)
+ goto err_rss;
+
+ /* must be power of 2 */
+ if (!rte_is_power_of_2(act_qgrop->queue_num))
+ goto err_rss;
+
+ /* queues are monotonous and contiguous so check last queue */
+ if ((act_qgrop->queue[0] + act_qgrop->queue_num) > dev_data->nb_rx_queues)
+ goto err_rss;
+
+ break;
+err_rss:
+ return rte_flow_error_set(error, EINVAL,
+ RTE_FLOW_ERROR_TYPE_ACTION, action,
+ "Invalid queue region");
+ }
+ case RTE_FLOW_ACTION_TYPE_QUEUE:
+ {
+ const struct rte_flow_action_queue *act_q;
+ act_q = action->conf;
+ if (act_q->index >= dev_data->nb_rx_queues) {
+ return rte_flow_error_set(error, EINVAL,
+ RTE_FLOW_ERROR_TYPE_ACTION, action,
+ "Invalid queue");
+ }
+
+ break;
+ }
+ case RTE_FLOW_ACTION_TYPE_DROP:
+ break;
+ default:
+ /* Should never reach */
+ return rte_flow_error_set(error, EINVAL,
+ RTE_FLOW_ERROR_TYPE_ACTION, action,
+ "Invalid action type");
}
return 0;
@@ -1788,11 +1763,39 @@ ice_switch_parse_pattern_action(struct ice_adapter *ad,
struct ci_flow_attr_check_param attr_param = {
.allow_priority = true,
};
+ struct ci_flow_actions parsed_actions = {0};
+ struct ci_flow_actions_check_param dcf_param = {
+ .allowed_types = (enum rte_flow_action_type[]){
+ RTE_FLOW_ACTION_TYPE_REPRESENTED_PORT,
+ RTE_FLOW_ACTION_TYPE_PORT_REPRESENTOR,
+ RTE_FLOW_ACTION_TYPE_DROP,
+ RTE_FLOW_ACTION_TYPE_END
+ },
+ .max_actions = 1,
+ .check = ice_switch_dcf_action_check,
+ };
+ struct ci_flow_actions_check_param param = {
+ .allowed_types = (enum rte_flow_action_type[]){
+ RTE_FLOW_ACTION_TYPE_RSS,
+ RTE_FLOW_ACTION_TYPE_QUEUE,
+ RTE_FLOW_ACTION_TYPE_DROP,
+ RTE_FLOW_ACTION_TYPE_END
+ },
+ .max_actions = 1,
+ .check = ice_switch_action_check,
+ .driver_ctx = ad,
+ .rss_queues_contig = true,
+ };
ret = ci_flow_check_attr(attr, &attr_param, error);
if (ret)
return ret;
+ ret = ci_flow_check_actions(actions, (ad->hw.dcf_enabled) ? &dcf_param : ¶m,
+ &parsed_actions, error);
+ if (ret)
+ goto error;
+
/* Allow only two priority values - 0 or 1 */
if (attr->priority > 1) {
rte_flow_error_set(error, EINVAL,
@@ -1870,16 +1873,12 @@ ice_switch_parse_pattern_action(struct ice_adapter *ad,
memset(&rule_info, 0, sizeof(rule_info));
rule_info.tun_type = tun_type;
- ret = ice_switch_check_action(actions, error);
- if (ret)
- goto error;
-
if (ad->hw.dcf_enabled)
- ret = ice_switch_parse_dcf_action((void *)ad, actions, attr->priority,
- error, &rule_info);
+ ret = ice_switch_parse_dcf_action(parsed_actions.actions[0],
+ attr->priority, error, &rule_info);
else
- ret = ice_switch_parse_action(pf, actions, attr->priority, error,
- &rule_info);
+ ret = ice_switch_parse_action(pf, parsed_actions.actions[0],
+ attr->priority, error, &rule_info);
if (ret)
goto error;
--
2.47.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox