* [PATCH] net/zxdh : update MAINTAINERS entry
From: Wenqiang Chen @ 2026-05-29 5:55 UTC (permalink / raw)
To: dev; +Cc: ran.ming, 陈文强10327674
[-- Attachment #1.1.1: Type: text/plain, Size: 644 bytes --]
From: 陈文强10327674 <chen.wenqiang2@zte.com.cn>
Change the GDTC rawdev maintainer from Yong Zhang
to Wenqiang Chen.
Signed-off-by: 陈文强10327674 <chen.wenqiang2@zte.com.cn>
---
MAINTAINERS | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index b390446ed3..48c72e1d67 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1612,7 +1612,7 @@ F: drivers/raw/dpaa2_cmdif/
F: doc/guides/rawdevs/dpaa2_cmdif.rst
ZTE GDTC
-M: Yong Zhang <zhang.yong25@zte.com.cn>
+M: Wenqiang Chen <chen.wenqiang2@zte.com.cn>
F: drivers/raw/gdtc/
F: doc/guides/rawdevs/gdtc.rst
--
2.27.0
[-- Attachment #1.1.2: Type: text/html , Size: 1045 bytes --]
^ permalink raw reply related
* [PATCH] raw/gdtc : update MAINTAINERS entry
From: Wenqiang Chen @ 2026-05-29 6:20 UTC (permalink / raw)
To: dev; +Cc: ran.ming, 陈文强10327674
[-- Attachment #1.1.1: Type: text/plain, Size: 644 bytes --]
From: 陈文强10327674 <chen.wenqiang2@zte.com.cn>
Change the GDTC rawdev maintainer from Yong Zhang
to Wenqiang Chen.
Signed-off-by: 陈文强10327674 <chen.wenqiang2@zte.com.cn>
---
MAINTAINERS | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index b390446ed3..48c72e1d67 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1612,7 +1612,7 @@ F: drivers/raw/dpaa2_cmdif/
F: doc/guides/rawdevs/dpaa2_cmdif.rst
ZTE GDTC
-M: Yong Zhang <zhang.yong25@zte.com.cn>
+M: Wenqiang Chen <chen.wenqiang2@zte.com.cn>
F: drivers/raw/gdtc/
F: doc/guides/rawdevs/gdtc.rst
--
2.27.0
[-- Attachment #1.1.2: Type: text/html , Size: 1045 bytes --]
^ permalink raw reply related
* Re: [PATCH v4 15/25] bus: support multiple probe
From: fengchengwen @ 2026-05-29 7:27 UTC (permalink / raw)
To: David Marchand, dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Rosen Xu, Chenbo Xia, Tomasz Duszynski, Long Li, Wei Hu
In-Reply-To: <20260527075654.3780732-16-david.marchand@redhat.com>
Currently, the bus probe is still the specific impl which don't test allow_multi_probe flag.
And local_dev_probe only cover the plug device case.
On 5/27/2026 3:56 PM, David Marchand wrote:
> Add infrastructure to declare support for multiple probe attempts
> on the same device. This prepares for the introduction of
> generic probe_device operation.
>
> The PCI bus enables this feature to support drivers with
> RTE_PCI_DRV_PROBE_AGAIN flag.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
...
^ permalink raw reply
* RE: [RFC 8/8] net/txgbe: privatize legacy flow director types
From: Jiawen Wu @ 2026-05-29 7:29 UTC (permalink / raw)
To: 'Stephen Hemminger', dev; +Cc: 'Zaiyu Wang'
In-Reply-To: <20260528213816.562910-9-stephen@networkplumber.org>
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Friday, May 29, 2026 5:37 AM
> To: dev@dpdk.org
> Cc: Stephen Hemminger <stephen@networkplumber.org>; Jiawen Wu <jiawenwu@trustnetic.com>; Zaiyu Wang
> <zaiyuwang@trustnetic.com>
> Subject: [RFC 8/8] net/txgbe: privatize legacy flow director types
>
> The txgbe driver inherited its flow director representation from
> ixgbe (it derives from the same hardware lineage) and uses the
> same legacy ethdev types (struct rte_eth_fdir_conf, enum
> rte_fdir_mode, and related) as its internal IR for translating
> rte_flow rules into hardware programming state. These types are
> scheduled for removal from ethdev in DPDK 26.11.
>
> Clone the relevant types into txgbe_ethdev.h with a txgbe-private
> prefix so the driver no longer depends on the legacy ABI. The
> embedded rte_eth_ipv4_flow / rte_eth_ipv6_flow types are cloned
> too since they live in the same removal candidate set.
>
> Unlike ixgbe, txgbe base/ does not pre-declare an
> enum txgbe_fdir_pballoc_type, so the cloned one is defined here.
> txgbe uses the integer value of pballoc directly in the table-
> size calculation
>
> (1024 << (fdir_conf->pballoc + 1)) - 2
>
> so the enum values match the original ethdev numbering
> (64K=0, 128K=1, 256K=2). pballoc is set explicitly from devargs
> in txgbe_handle_devarg(), so no extra initialization is required.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> drivers/net/txgbe/txgbe_ethdev.c | 6 +-
> drivers/net/txgbe/txgbe_ethdev.h | 101 +++++++++++++++++++++-
> drivers/net/txgbe/txgbe_fdir.c | 78 ++++++++---------
> drivers/net/txgbe/txgbe_flow.c | 24 ++---
> drivers/net/txgbe/txgbe_rxtx_vec_common.h | 4 +-
> 5 files changed, 155 insertions(+), 58 deletions(-)
LGTM, thanks.
> diff --git a/drivers/net/txgbe/txgbe_ethdev.c b/drivers/net/txgbe/txgbe_ethdev.c
> index 5d360f8305..258ec4d97c 100644
> --- a/drivers/net/txgbe/txgbe_ethdev.c
> +++ b/drivers/net/txgbe/txgbe_ethdev.c
> @@ -524,7 +524,7 @@ txgbe_handle_devarg(__rte_unused const char *key, const char *value,
> static void
> txgbe_parse_devargs(struct rte_eth_dev *dev)
> {
> - struct rte_eth_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
> + struct txgbe_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
> struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
> struct rte_devargs *devargs = pci_dev->device.devargs;
> struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
> @@ -934,7 +934,7 @@ static int txgbe_l2_tn_filter_uninit(struct rte_eth_dev *eth_dev)
>
> int txgbe_fdir_filter_init(struct rte_eth_dev *eth_dev)
> {
> - struct rte_eth_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(eth_dev);
> + struct txgbe_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(eth_dev);
> struct txgbe_hw_fdir_info *fdir_info = TXGBE_DEV_FDIR(eth_dev);
> char fdir_hash_name[RTE_HASH_NAMESIZE];
> u16 max_fdir_num = (1024 << (fdir_conf->pballoc + 1)) - 2;
> @@ -1834,7 +1834,7 @@ txgbe_dev_start(struct rte_eth_dev *dev)
> txgbe_configure_port(dev);
> txgbe_configure_dcb(dev);
>
> - if (TXGBE_DEV_FDIR_CONF(dev)->mode != RTE_FDIR_MODE_NONE) {
> + if (TXGBE_DEV_FDIR_CONF(dev)->mode != TXGBE_FDIR_MODE_NONE) {
> err = txgbe_fdir_configure(dev);
> if (err)
> goto error;
> diff --git a/drivers/net/txgbe/txgbe_ethdev.h b/drivers/net/txgbe/txgbe_ethdev.h
> index 189fbac541..1f4ce4f3b8 100644
> --- a/drivers/net/txgbe/txgbe_ethdev.h
> +++ b/drivers/net/txgbe/txgbe_ethdev.h
> @@ -83,6 +83,103 @@
> /*
> * Information about the fdir mode.
> */
> +/*
> + * Privatized flow director configuration types.
> + *
> + * Cloned from <rte_eth_ctrl.h> and <ethdev_driver.h> so that txgbe no
> + * longer depends on the legacy ethdev flow director ABI, which is
> + * scheduled for removal in DPDK 26.11. Semantics unchanged; only the
> + * names are txgbe-private.
> + *
> + * NB: txgbe uses the integer value of pballoc directly in
> + * (1024 << (fdir_conf->pballoc + 1)) - 2
> + * to size the HW filter table. The enum values therefore match the
> + * original ethdev numbering (64K=0, 128K=1, 256K=2).
> + */
> +#define TXGBE_FDIR_MAX_FLEXLEN 16 /**< Max length of flexbytes. */
> +
> +struct txgbe_ipv4_flow {
> + uint32_t src_ip;
> + uint32_t dst_ip;
> + uint8_t tos;
> + uint8_t ttl;
> + uint8_t proto;
> +};
> +
> +struct txgbe_ipv6_flow {
> + uint32_t src_ip[4];
> + uint32_t dst_ip[4];
> + uint8_t tc;
> + uint8_t proto;
> + uint8_t hop_limits;
> +};
> +
> +struct txgbe_fdir_masks {
> + uint16_t vlan_tci_mask;
> + struct txgbe_ipv4_flow ipv4_mask;
> + struct txgbe_ipv6_flow ipv6_mask;
> + uint16_t src_port_mask;
> + uint16_t dst_port_mask;
> + uint8_t mac_addr_byte_mask;
> + uint32_t tunnel_id_mask;
> + uint8_t tunnel_type_mask;
> +};
> +
> +enum txgbe_payload_type {
> + TXGBE_PAYLOAD_UNKNOWN = 0,
> + TXGBE_RAW_PAYLOAD,
> + TXGBE_L2_PAYLOAD,
> + TXGBE_L3_PAYLOAD,
> + TXGBE_L4_PAYLOAD,
> + TXGBE_PAYLOAD_MAX = 8,
> +};
> +
> +struct txgbe_flex_payload_cfg {
> + enum txgbe_payload_type type;
> + uint16_t src_offset[TXGBE_FDIR_MAX_FLEXLEN];
> +};
> +
> +struct txgbe_fdir_flex_mask {
> + uint16_t flow_type;
> + uint8_t mask[TXGBE_FDIR_MAX_FLEXLEN];
> +};
> +
> +struct txgbe_fdir_flex_conf {
> + uint16_t nb_payloads;
> + uint16_t nb_flexmasks;
> + struct txgbe_flex_payload_cfg flex_set[TXGBE_PAYLOAD_MAX];
> + struct txgbe_fdir_flex_mask flex_mask[RTE_ETH_FLOW_MAX];
> +};
> +
> +enum txgbe_fdir_mode {
> + TXGBE_FDIR_MODE_NONE = 0,
> + TXGBE_FDIR_MODE_SIGNATURE,
> + TXGBE_FDIR_MODE_PERFECT,
> + TXGBE_FDIR_MODE_PERFECT_MAC_VLAN,
> + TXGBE_FDIR_MODE_PERFECT_TUNNEL,
> +};
> +
> +enum txgbe_fdir_pballoc_type {
> + TXGBE_FDIR_PBALLOC_64K = 0,
> + TXGBE_FDIR_PBALLOC_128K,
> + TXGBE_FDIR_PBALLOC_256K,
> +};
> +
> +enum txgbe_fdir_status_mode {
> + TXGBE_FDIR_NO_REPORT_STATUS = 0,
> + TXGBE_FDIR_REPORT_STATUS,
> + TXGBE_FDIR_REPORT_STATUS_ALWAYS,
> +};
> +
> +struct txgbe_fdir_conf {
> + enum txgbe_fdir_mode mode;
> + enum txgbe_fdir_pballoc_type pballoc;
> + enum txgbe_fdir_status_mode status;
> + uint8_t drop_queue;
> + struct txgbe_fdir_masks mask;
> + struct txgbe_fdir_flex_conf flex_conf;
> +};
> +
> struct txgbe_hw_fdir_mask {
> uint16_t vlan_tci_mask;
> uint32_t src_ipv4_mask;
> @@ -111,7 +208,7 @@ struct txgbe_fdir_rule {
> struct txgbe_atr_input input; /* key of fdir filter */
> bool b_spec; /* If TRUE, input, fdirflags, queue have meaning. */
> bool b_mask; /* If TRUE, mask has meaning. */
> - enum rte_fdir_mode mode; /* IP, MAC VLAN, Tunnel */
> + enum txgbe_fdir_mode mode; /* IP, MAC VLAN, Tunnel */
> uint32_t fdirflags; /* drop or forward */
> uint32_t soft_id; /* an unique value for this rule */
> uint8_t queue; /* assigned rx queue */
> @@ -352,7 +449,7 @@ struct txgbe_tm_conf {
> struct txgbe_adapter {
> struct txgbe_hw hw;
> struct txgbe_hw_stats stats;
> - struct rte_eth_fdir_conf fdir_conf;
> + struct txgbe_fdir_conf fdir_conf;
> struct txgbe_hw_fdir_info fdir;
> struct txgbe_interrupt intr;
> struct txgbe_stat_mappings stat_mappings;
> diff --git a/drivers/net/txgbe/txgbe_fdir.c b/drivers/net/txgbe/txgbe_fdir.c
> index 67f586ffc7..ded01341a4 100644
> --- a/drivers/net/txgbe/txgbe_fdir.c
> +++ b/drivers/net/txgbe/txgbe_fdir.c
> @@ -102,22 +102,22 @@ txgbe_fdir_enable(struct txgbe_hw *hw, uint32_t fdirctrl)
> * flexbytes matching field, and drop queue (only for perfect matching mode).
> */
> static inline int
> -configure_fdir_flags(const struct rte_eth_fdir_conf *conf,
> +configure_fdir_flags(const struct txgbe_fdir_conf *conf,
> uint32_t *fdirctrl, uint32_t *flex)
> {
> *fdirctrl = 0;
> *flex = 0;
>
> switch (conf->pballoc) {
> - case RTE_ETH_FDIR_PBALLOC_64K:
> + case TXGBE_FDIR_PBALLOC_64K:
> /* 8k - 1 signature filters */
> *fdirctrl |= TXGBE_FDIRCTL_BUF_64K;
> break;
> - case RTE_ETH_FDIR_PBALLOC_128K:
> + case TXGBE_FDIR_PBALLOC_128K:
> /* 16k - 1 signature filters */
> *fdirctrl |= TXGBE_FDIRCTL_BUF_128K;
> break;
> - case RTE_ETH_FDIR_PBALLOC_256K:
> + case TXGBE_FDIR_PBALLOC_256K:
> /* 32k - 1 signature filters */
> *fdirctrl |= TXGBE_FDIRCTL_BUF_256K;
> break;
> @@ -129,14 +129,14 @@ configure_fdir_flags(const struct rte_eth_fdir_conf *conf,
>
> /* status flags: write hash & swindex in the rx descriptor */
> switch (conf->status) {
> - case RTE_FDIR_NO_REPORT_STATUS:
> + case TXGBE_FDIR_NO_REPORT_STATUS:
> /* do nothing, default mode */
> break;
> - case RTE_FDIR_REPORT_STATUS:
> + case TXGBE_FDIR_REPORT_STATUS:
> /* report status when the packet matches a fdir rule */
> *fdirctrl |= TXGBE_FDIRCTL_REPORT_MATCH;
> break;
> - case RTE_FDIR_REPORT_STATUS_ALWAYS:
> + case TXGBE_FDIR_REPORT_STATUS_ALWAYS:
> /* always report status */
> *fdirctrl |= TXGBE_FDIRCTL_REPORT_ALWAYS;
> break;
> @@ -150,9 +150,9 @@ configure_fdir_flags(const struct rte_eth_fdir_conf *conf,
> *flex |= TXGBE_FDIRFLEXCFG_OFST(TXGBE_DEFAULT_FLEXBYTES_OFFSET / 2);
>
> switch (conf->mode) {
> - case RTE_FDIR_MODE_SIGNATURE:
> + case TXGBE_FDIR_MODE_SIGNATURE:
> break;
> - case RTE_FDIR_MODE_PERFECT:
> + case TXGBE_FDIR_MODE_PERFECT:
> *fdirctrl |= TXGBE_FDIRCTL_PERFECT;
> *fdirctrl |= TXGBE_FDIRCTL_DROPQP(conf->drop_queue);
> break;
> @@ -179,7 +179,7 @@ txgbe_fdir_set_input_mask(struct rte_eth_dev *dev)
> {
> struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
> struct txgbe_hw_fdir_info *info = TXGBE_DEV_FDIR(dev);
> - enum rte_fdir_mode mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
> + enum txgbe_fdir_mode mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
> /*
> * mask VM pool and DIPv6 since there are currently not supported
> * mask FLEX byte, it will be set in flex_conf
> @@ -190,8 +190,8 @@ txgbe_fdir_set_input_mask(struct rte_eth_dev *dev)
>
> PMD_INIT_FUNC_TRACE();
>
> - if (mode != RTE_FDIR_MODE_SIGNATURE &&
> - mode != RTE_FDIR_MODE_PERFECT) {
> + if (mode != TXGBE_FDIR_MODE_SIGNATURE &&
> + mode != TXGBE_FDIR_MODE_PERFECT) {
> PMD_DRV_LOG(ERR, "Not supported fdir mode - %d!", mode);
> return -ENOTSUP;
> }
> @@ -232,14 +232,14 @@ txgbe_fdir_set_input_mask(struct rte_eth_dev *dev)
> static int
> txgbe_fdir_store_input_mask(struct rte_eth_dev *dev)
> {
> - struct rte_eth_fdir_masks *input_mask = &TXGBE_DEV_FDIR_CONF(dev)->mask;
> - enum rte_fdir_mode mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
> + struct txgbe_fdir_masks *input_mask = &TXGBE_DEV_FDIR_CONF(dev)->mask;
> + enum txgbe_fdir_mode mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
> struct txgbe_hw_fdir_info *info = TXGBE_DEV_FDIR(dev);
> uint16_t dst_ipv6m = 0;
> uint16_t src_ipv6m = 0;
>
> - if (mode != RTE_FDIR_MODE_SIGNATURE &&
> - mode != RTE_FDIR_MODE_PERFECT) {
> + if (mode != TXGBE_FDIR_MODE_SIGNATURE &&
> + mode != TXGBE_FDIR_MODE_PERFECT) {
> PMD_DRV_LOG(ERR, "Not supported fdir mode - %d!", mode);
> return -ENOTSUP;
> }
> @@ -304,12 +304,12 @@ txgbe_fdir_set_flexbytes_offset(struct rte_eth_dev *dev,
> static int
> txgbe_set_fdir_flex_conf(struct rte_eth_dev *dev, uint32_t flex)
> {
> - const struct rte_eth_fdir_flex_conf *conf =
> + const struct txgbe_fdir_flex_conf *conf =
> &TXGBE_DEV_FDIR_CONF(dev)->flex_conf;
> struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
> struct txgbe_hw_fdir_info *info = TXGBE_DEV_FDIR(dev);
> - const struct rte_eth_flex_payload_cfg *flex_cfg;
> - const struct rte_eth_fdir_flex_mask *flex_mask;
> + const struct txgbe_flex_payload_cfg *flex_cfg;
> + const struct txgbe_fdir_flex_mask *flex_mask;
> uint16_t flexbytes = 0;
> uint16_t i;
>
> @@ -322,7 +322,7 @@ txgbe_set_fdir_flex_conf(struct rte_eth_dev *dev, uint32_t flex)
>
> for (i = 0; i < conf->nb_payloads; i++) {
> flex_cfg = &conf->flex_set[i];
> - if (flex_cfg->type != RTE_ETH_RAW_PAYLOAD) {
> + if (flex_cfg->type != TXGBE_RAW_PAYLOAD) {
> PMD_DRV_LOG(ERR, "unsupported payload type.");
> return -EINVAL;
> }
> @@ -375,13 +375,13 @@ txgbe_fdir_configure(struct rte_eth_dev *dev)
> int err;
> uint32_t fdirctrl, flex, pbsize;
> int i;
> - enum rte_fdir_mode mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
> + enum txgbe_fdir_mode mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
>
> PMD_INIT_FUNC_TRACE();
>
> /* supports mac-vlan and tunnel mode */
> - if (mode != RTE_FDIR_MODE_SIGNATURE &&
> - mode != RTE_FDIR_MODE_PERFECT)
> + if (mode != TXGBE_FDIR_MODE_SIGNATURE &&
> + mode != TXGBE_FDIR_MODE_PERFECT)
> return -ENOSYS;
>
> err = configure_fdir_flags(TXGBE_DEV_FDIR_CONF(dev), &fdirctrl, &flex);
> @@ -531,15 +531,15 @@ txgbe_atr_compute_hash(struct txgbe_atr_input *atr_input,
>
> static uint32_t
> atr_compute_perfect_hash(struct txgbe_atr_input *input,
> - enum rte_eth_fdir_pballoc_type pballoc)
> + enum txgbe_fdir_pballoc_type pballoc)
> {
> uint32_t bucket_hash;
>
> bucket_hash = txgbe_atr_compute_hash(input,
> TXGBE_ATR_BUCKET_HASH_KEY);
> - if (pballoc == RTE_ETH_FDIR_PBALLOC_256K)
> + if (pballoc == TXGBE_FDIR_PBALLOC_256K)
> bucket_hash &= PERFECT_BUCKET_256KB_HASH_MASK;
> - else if (pballoc == RTE_ETH_FDIR_PBALLOC_128K)
> + else if (pballoc == TXGBE_FDIR_PBALLOC_128K)
> bucket_hash &= PERFECT_BUCKET_128KB_HASH_MASK;
> else
> bucket_hash &= PERFECT_BUCKET_64KB_HASH_MASK;
> @@ -574,15 +574,15 @@ txgbe_fdir_check_cmd_complete(struct txgbe_hw *hw, uint32_t *fdircmd)
> */
> static uint32_t
> atr_compute_signature_hash(struct txgbe_atr_input *input,
> - enum rte_eth_fdir_pballoc_type pballoc)
> + enum txgbe_fdir_pballoc_type pballoc)
> {
> uint32_t bucket_hash, sig_hash;
>
> bucket_hash = txgbe_atr_compute_hash(input,
> TXGBE_ATR_BUCKET_HASH_KEY);
> - if (pballoc == RTE_ETH_FDIR_PBALLOC_256K)
> + if (pballoc == TXGBE_FDIR_PBALLOC_256K)
> bucket_hash &= SIG_BUCKET_256KB_HASH_MASK;
> - else if (pballoc == RTE_ETH_FDIR_PBALLOC_128K)
> + else if (pballoc == TXGBE_FDIR_PBALLOC_128K)
> bucket_hash &= SIG_BUCKET_128KB_HASH_MASK;
> else
> bucket_hash &= SIG_BUCKET_64KB_HASH_MASK;
> @@ -603,7 +603,7 @@ static int
> fdir_write_perfect_filter(struct txgbe_hw *hw,
> struct txgbe_atr_input *input, uint8_t queue,
> uint32_t fdircmd, uint32_t fdirhash,
> - enum rte_fdir_mode mode)
> + enum txgbe_fdir_mode mode)
> {
> uint32_t fdirport, fdirflex;
> int err = 0;
> @@ -830,14 +830,14 @@ txgbe_fdir_filter_program(struct rte_eth_dev *dev,
> bool is_perfect = FALSE;
> int err;
> struct txgbe_hw_fdir_info *info = TXGBE_DEV_FDIR(dev);
> - enum rte_fdir_mode fdir_mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
> + enum txgbe_fdir_mode fdir_mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
> struct txgbe_fdir_filter *node;
>
> - if (fdir_mode == RTE_FDIR_MODE_NONE ||
> + if (fdir_mode == TXGBE_FDIR_MODE_NONE ||
> fdir_mode != rule->mode)
> return -ENOTSUP;
>
> - if (fdir_mode >= RTE_FDIR_MODE_PERFECT)
> + if (fdir_mode >= TXGBE_FDIR_MODE_PERFECT)
> is_perfect = TRUE;
>
> txgbe_fdir_mask_input(&info->mask, &rule->input);
> @@ -1021,7 +1021,7 @@ txgbevf_fdir_filter_program(struct rte_eth_dev *dev,
> uint32_t fdirhash;
> int ret;
>
> - if (rule->mode != RTE_FDIR_MODE_PERFECT ||
> + if (rule->mode != TXGBE_FDIR_MODE_PERFECT ||
> rule->fdirflags == TXGBE_FDIRPICMD_DROP)
> return -ENOTSUP;
>
> @@ -1109,10 +1109,10 @@ txgbe_fdir_filter_restore(struct rte_eth_dev *dev)
> struct txgbe_hw_fdir_info *fdir_info = TXGBE_DEV_FDIR(dev);
> struct txgbe_fdir_filter *node;
> bool is_perfect = FALSE;
> - enum rte_fdir_mode fdir_mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
> + enum txgbe_fdir_mode fdir_mode = TXGBE_DEV_FDIR_CONF(dev)->mode;
>
> - if (fdir_mode >= RTE_FDIR_MODE_PERFECT &&
> - fdir_mode <= RTE_FDIR_MODE_PERFECT_TUNNEL)
> + if (fdir_mode >= TXGBE_FDIR_MODE_PERFECT &&
> + fdir_mode <= TXGBE_FDIR_MODE_PERFECT_TUNNEL)
> is_perfect = TRUE;
>
> if (is_perfect) {
> @@ -1139,7 +1139,7 @@ txgbe_fdir_filter_restore(struct rte_eth_dev *dev)
> int
> txgbe_clear_all_fdir_filter(struct rte_eth_dev *dev)
> {
> - struct rte_eth_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
> + struct txgbe_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
> struct txgbe_hw_fdir_info *fdir_info = TXGBE_DEV_FDIR(dev);
> struct txgbe_fdir_filter *fdir_filter;
> struct txgbe_fdir_filter *filter_flag;
> @@ -1150,7 +1150,7 @@ txgbe_clear_all_fdir_filter(struct rte_eth_dev *dev)
> memset(fdir_info->hash_map, 0,
> sizeof(struct txgbe_fdir_filter *) *
> ((1024 << (fdir_conf->pballoc + 1)) - 2));
> - fdir_conf->mode = RTE_FDIR_MODE_NONE;
> + fdir_conf->mode = TXGBE_FDIR_MODE_NONE;
> filter_flag = TAILQ_FIRST(&fdir_info->fdir_list);
> while ((fdir_filter = TAILQ_FIRST(&fdir_info->fdir_list))) {
> TAILQ_REMOVE(&fdir_info->fdir_list,
> diff --git a/drivers/net/txgbe/txgbe_flow.c b/drivers/net/txgbe/txgbe_flow.c
> index a97588e57a..8470bf78af 100644
> --- a/drivers/net/txgbe/txgbe_flow.c
> +++ b/drivers/net/txgbe/txgbe_flow.c
> @@ -1412,7 +1412,7 @@ txgbe_parse_fdir_act_attr(const struct rte_flow_attr *attr,
> rule->queue = act_q->index;
> } else { /* drop */
> /* signature mode does not support drop action. */
> - if (rule->mode == RTE_FDIR_MODE_SIGNATURE) {
> + if (rule->mode == TXGBE_FDIR_MODE_SIGNATURE) {
> memset(rule, 0, sizeof(struct txgbe_fdir_rule));
> rte_flow_error_set(error, EINVAL,
> RTE_FLOW_ERROR_TYPE_ACTION,
> @@ -1646,9 +1646,9 @@ txgbe_parse_fdir_filter_normal(struct rte_eth_dev *dev __rte_unused,
> }
>
> if (signature_match(pattern))
> - rule->mode = RTE_FDIR_MODE_SIGNATURE;
> + rule->mode = TXGBE_FDIR_MODE_SIGNATURE;
> else
> - rule->mode = RTE_FDIR_MODE_PERFECT;
> + rule->mode = TXGBE_FDIR_MODE_PERFECT;
>
> /*Not supported last point for range*/
> if (item->last) {
> @@ -1678,7 +1678,7 @@ txgbe_parse_fdir_filter_normal(struct rte_eth_dev *dev __rte_unused,
>
> /* Ether type should be masked. */
> if (eth_mask->hdr.ether_type ||
> - rule->mode == RTE_FDIR_MODE_SIGNATURE) {
> + rule->mode == TXGBE_FDIR_MODE_SIGNATURE) {
> memset(rule, 0, sizeof(struct txgbe_fdir_rule));
> rte_flow_error_set(error, EINVAL,
> RTE_FLOW_ERROR_TYPE_ITEM,
> @@ -1687,7 +1687,7 @@ txgbe_parse_fdir_filter_normal(struct rte_eth_dev *dev __rte_unused,
> }
>
> /* If ethernet has meaning, it means MAC VLAN mode. */
> - rule->mode = RTE_FDIR_MODE_PERFECT_MAC_VLAN;
> + rule->mode = TXGBE_FDIR_MODE_PERFECT_MAC_VLAN;
>
> /**
> * src MAC address must be masked,
> @@ -1718,7 +1718,7 @@ txgbe_parse_fdir_filter_normal(struct rte_eth_dev *dev __rte_unused,
> * IPv6 is not supported.
> */
> item = next_no_fuzzy_pattern(pattern, item);
> - if (rule->mode == RTE_FDIR_MODE_PERFECT_MAC_VLAN) {
> + if (rule->mode == TXGBE_FDIR_MODE_PERFECT_MAC_VLAN) {
> if (item->type != RTE_FLOW_ITEM_TYPE_VLAN) {
> memset(rule, 0, sizeof(struct txgbe_fdir_rule));
> rte_flow_error_set(error, EINVAL,
> @@ -2282,7 +2282,7 @@ txgbe_parse_fdir_filter_tunnel(const struct rte_flow_attr *attr,
> return -rte_errno;
> }
>
> - rule->mode = RTE_FDIR_MODE_PERFECT;
> + rule->mode = TXGBE_FDIR_MODE_PERFECT;
> ptid = TXGBE_PTID_PKT_TUN;
>
> /* Skip MAC. */
> @@ -2949,7 +2949,7 @@ txgbe_parse_fdir_filter(struct rte_eth_dev *dev,
> struct rte_flow_error *error)
> {
> int ret;
> - struct rte_eth_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
> + struct txgbe_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
>
> ret = txgbe_parse_fdir_filter_normal(dev, attr, pattern,
> actions, rule, error);
> @@ -2965,11 +2965,11 @@ txgbe_parse_fdir_filter(struct rte_eth_dev *dev,
> if (!txgbe_is_pf(TXGBE_DEV_HW(dev)))
> return ret;
>
> - if (fdir_conf->mode == RTE_FDIR_MODE_NONE) {
> + if (fdir_conf->mode == TXGBE_FDIR_MODE_NONE) {
> fdir_conf->mode = rule->mode;
> ret = txgbe_fdir_configure(dev);
> if (ret) {
> - fdir_conf->mode = RTE_FDIR_MODE_NONE;
> + fdir_conf->mode = TXGBE_FDIR_MODE_NONE;
> return ret;
> }
> } else if (fdir_conf->mode != rule->mode) {
> @@ -3562,7 +3562,7 @@ txgbe_flow_destroy(struct rte_eth_dev *dev,
> struct txgbe_fdir_rule_ele *fdir_rule_ptr;
> struct txgbe_flow_mem *txgbe_flow_mem_ptr;
> struct txgbe_hw_fdir_info *fdir_info = TXGBE_DEV_FDIR(dev);
> - struct rte_eth_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
> + struct txgbe_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
> struct txgbe_rss_conf_ele *rss_filter_ptr;
>
> switch (filter_type) {
> @@ -3625,7 +3625,7 @@ txgbe_flow_destroy(struct rte_eth_dev *dev,
> fdir_info->mask_added = false;
> fdir_info->flex_relative = false;
> fdir_info->flex_bytes_offset = 0;
> - fdir_conf->mode = RTE_FDIR_MODE_NONE;
> + fdir_conf->mode = TXGBE_FDIR_MODE_NONE;
> }
> }
> break;
> diff --git a/drivers/net/txgbe/txgbe_rxtx_vec_common.h b/drivers/net/txgbe/txgbe_rxtx_vec_common.h
> index 00847d087b..c6144cad70 100644
> --- a/drivers/net/txgbe/txgbe_rxtx_vec_common.h
> +++ b/drivers/net/txgbe/txgbe_rxtx_vec_common.h
> @@ -299,10 +299,10 @@ static inline int
> txgbe_rx_vec_dev_conf_condition_check_default(struct rte_eth_dev *dev)
> {
> #ifndef RTE_LIBRTE_IEEE1588
> - struct rte_eth_fdir_conf *fconf = TXGBE_DEV_FDIR_CONF(dev);
> + struct txgbe_fdir_conf *fconf = TXGBE_DEV_FDIR_CONF(dev);
>
> /* no fdir support */
> - if (fconf->mode != RTE_FDIR_MODE_NONE)
> + if (fconf->mode != TXGBE_FDIR_MODE_NONE)
> return -1;
>
> return 0;
> --
> 2.53.0
>
>
^ permalink raw reply
* [PATCH v2] ethdev: promote flow metadata APIs to stable
From: Dariusz Sosnowski @ 2026-05-29 7:28 UTC (permalink / raw)
To: Ori Kam, Thomas Monjalon, Andrew Rybchenko; +Cc: dev
In-Reply-To: <20260527103246.633327-1-dsosnowski@nvidia.com>
Following experimental symbols related to flow metadata
were added in v19.11:
- rte_flow_dynf_metadata_register
- rte_flow_dynf_metadata_offs
- rte_flow_dynf_metadata_mask
Type of rte_flow_dynf_metadata_offs was changed from int to int32_t
in v20.05 release.
There were no changes to these symbols since then.
This patch promotes these symbols and removes __rte_experimental
from the following inline functions:
- rte_flow_dynf_metadata_avail
- rte_flow_dynf_metadata_get
- rte_flow_dynf_metadata_set
All these symbols and functions will be used by netdev-doca
backend in Open vSwitch [1].
Stabilizing these symbols is required by current OVS policy
to remove the need for ALLOW_EXPERIMENTAL_API [2].
[1]: https://patchwork.ozlabs.org/project/openvswitch/list/?series=504726&state=%2A&archive=both
[2]: https://mail.openvswitch.org/pipermail/ovs-dev/2026-May/432066.html
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
v2:
- Updated 26.07 release notes.
doc/guides/rel_notes/release_26_07.rst | 11 +++++++++++
lib/ethdev/rte_flow.c | 6 +++---
lib/ethdev/rte_flow.h | 4 ----
3 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 8b4f8401e2..6eba91a5e9 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -116,6 +116,17 @@ API Changes
Also, make sure to start the actual text at the margin.
=======================================================
+* **ethdev: promoted flow metadata APIs from experimental to stable.**
+
+ The following ethdev APIs and symbols are no longer marked experimental:
+
+ - ``rte_flow_dynf_metadata_register``
+ - ``rte_flow_dynf_metadata_offs``
+ - ``rte_flow_dynf_metadata_mask``
+ - ``rte_flow_dynf_metadata_avail``
+ - ``rte_flow_dynf_metadata_get``
+ - ``rte_flow_dynf_metadata_set``
+
ABI Changes
-----------
diff --git a/lib/ethdev/rte_flow.c b/lib/ethdev/rte_flow.c
index 7a51b667cf..ec0fe08355 100644
--- a/lib/ethdev/rte_flow.c
+++ b/lib/ethdev/rte_flow.c
@@ -23,11 +23,11 @@
#define FLOW_LOG RTE_ETHDEV_LOG_LINE
/* Mbuf dynamic field name for metadata. */
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_flow_dynf_metadata_offs, 19.11)
+RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_offs)
int32_t rte_flow_dynf_metadata_offs = -1;
/* Mbuf dynamic field flag bit number for metadata. */
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_flow_dynf_metadata_mask, 19.11)
+RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_mask)
uint64_t rte_flow_dynf_metadata_mask;
/**
@@ -281,7 +281,7 @@ static const struct rte_flow_desc_data rte_flow_desc_action[] = {
MK_FLOW_ACTION(JUMP_TO_TABLE_INDEX, sizeof(struct rte_flow_action_jump_to_table_index)),
};
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_flow_dynf_metadata_register, 19.11)
+RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_register)
int
rte_flow_dynf_metadata_register(void)
{
diff --git a/lib/ethdev/rte_flow.h b/lib/ethdev/rte_flow.h
index ba3bcc89a3..0c6d42c0d8 100644
--- a/lib/ethdev/rte_flow.h
+++ b/lib/ethdev/rte_flow.h
@@ -4253,14 +4253,12 @@ extern uint64_t rte_flow_dynf_metadata_mask;
#define RTE_MBUF_DYNFLAG_RX_METADATA (rte_flow_dynf_metadata_mask)
#define RTE_MBUF_DYNFLAG_TX_METADATA (rte_flow_dynf_metadata_mask)
-__rte_experimental
static inline uint32_t
rte_flow_dynf_metadata_get(struct rte_mbuf *m)
{
return *RTE_FLOW_DYNF_METADATA(m);
}
-__rte_experimental
static inline void
rte_flow_dynf_metadata_set(struct rte_mbuf *m, uint32_t v)
{
@@ -4590,7 +4588,6 @@ rte_flow_dev_dump(uint16_t port_id, struct rte_flow *flow,
* @return
* True if registered, false otherwise.
*/
-__rte_experimental
static inline int
rte_flow_dynf_metadata_avail(void)
{
@@ -4607,7 +4604,6 @@ rte_flow_dynf_metadata_avail(void)
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-__rte_experimental
int
rte_flow_dynf_metadata_register(void);
--
2.47.3
^ permalink raw reply related
* [PATCH v2] net/mlx5: promote private API to stable
From: Dariusz Sosnowski @ 2026-05-29 7:32 UTC (permalink / raw)
To: Viacheslav Ovsiienko, Bing Zhao, Ori Kam, Suanming Mou,
Matan Azrad; +Cc: dev
In-Reply-To: <20260527103357.633429-1-dsosnowski@nvidia.com>
Following experimental functions are exposed by mlx5 PMD
since 25.11 release:
- rte_pmd_mlx5_driver_event_cb_register
- rte_pmd_mlx5_driver_event_cb_unregister
- rte_pmd_mlx5_enable_steering
- rte_pmd_mlx5_disable_steering
First two are used to register callbacks for driver events
(when Rx/Tx queues are created or destroyed).
Other two are used to enable/disable flow steering in mlx5 PMD.
No changes were made and no changes are planned to these symbols.
These are currently used by NVIDIA DOCA SDK since version 3.3,
which started depending on upstream DPDK releases [1].
Purpose of their use is to:
- expose HW identifiers of Rx/Tx mlx5 queues managed by DPDK and
- allow flow steering to happen outside of DPDK.
Also, some of these symbols will be used by netdev-doca backend
in Open vSwitch [2].
Whenever a DOCA netdev would be added/removed in Open vSwitch,
it will have to disable/enable steering for mlx5 driver.
Stabilizing these symbols is required by current OVS policy
to remove the need for ALLOW_EXPERIMENTAL_API [3].
This patch promotes aforementioned symbols to stable.
[1]: https://docs.nvidia.com/doca/sdk/customer-affecting-changes/index.html
[2]: https://patchwork.ozlabs.org/project/openvswitch/list/?series=504726&state=%2A&archive=both
[3]: https://mail.openvswitch.org/pipermail/ovs-dev/2026-May/432066.html
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
v2:
- Updated 26.07 release notes.
doc/guides/rel_notes/release_26_07.rst | 9 +++++++++
drivers/net/mlx5/mlx5_driver_event.c | 4 ++--
drivers/net/mlx5/mlx5_flow.c | 4 ++--
drivers/net/mlx5/rte_pmd_mlx5.h | 4 ----
4 files changed, 13 insertions(+), 8 deletions(-)
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 6eba91a5e9..2af24d40a0 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -127,6 +127,15 @@ API Changes
- ``rte_flow_dynf_metadata_get``
- ``rte_flow_dynf_metadata_set``
+* **mlx5: promoted driver event and steering management APIs from experimental to stable.**
+
+ The following mlx5 APIs are no longer marked experimental:
+
+ - ``rte_pmd_mlx5_driver_event_cb_register``
+ - ``rte_pmd_mlx5_driver_event_cb_unregister``
+ - ``rte_pmd_mlx5_enable_steering``
+ - ``rte_pmd_mlx5_disable_steering``
+
ABI Changes
-----------
diff --git a/drivers/net/mlx5/mlx5_driver_event.c b/drivers/net/mlx5/mlx5_driver_event.c
index 1dc8029ee5..89e49331c8 100644
--- a/drivers/net/mlx5/mlx5_driver_event.c
+++ b/drivers/net/mlx5/mlx5_driver_event.c
@@ -236,7 +236,7 @@ notify_existing_devices(rte_pmd_mlx5_driver_event_callback_t cb, void *opaque)
notify_existing_queues(port_id, cb, opaque);
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pmd_mlx5_driver_event_cb_register, 25.11)
+RTE_EXPORT_SYMBOL(rte_pmd_mlx5_driver_event_cb_register)
int
rte_pmd_mlx5_driver_event_cb_register(rte_pmd_mlx5_driver_event_callback_t cb, void *opaque)
{
@@ -264,7 +264,7 @@ rte_pmd_mlx5_driver_event_cb_register(rte_pmd_mlx5_driver_event_callback_t cb, v
return 0;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pmd_mlx5_driver_event_cb_unregister, 25.11)
+RTE_EXPORT_SYMBOL(rte_pmd_mlx5_driver_event_cb_unregister)
int
rte_pmd_mlx5_driver_event_cb_unregister(rte_pmd_mlx5_driver_event_callback_t cb)
{
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 915ea29a5a..a95dd9dc94 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -12506,7 +12506,7 @@ flow_disable_steering_run_on_related(struct rte_eth_dev *dev,
}
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pmd_mlx5_disable_steering, 25.11)
+RTE_EXPORT_SYMBOL(rte_pmd_mlx5_disable_steering)
void
rte_pmd_mlx5_disable_steering(void)
{
@@ -12532,7 +12532,7 @@ rte_pmd_mlx5_disable_steering(void)
mlx5_steering_disabled = true;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pmd_mlx5_enable_steering, 25.11)
+RTE_EXPORT_SYMBOL(rte_pmd_mlx5_enable_steering)
int
rte_pmd_mlx5_enable_steering(void)
{
diff --git a/drivers/net/mlx5/rte_pmd_mlx5.h b/drivers/net/mlx5/rte_pmd_mlx5.h
index 55a26cb067..f482875434 100644
--- a/drivers/net/mlx5/rte_pmd_mlx5.h
+++ b/drivers/net/mlx5/rte_pmd_mlx5.h
@@ -620,7 +620,6 @@ int rte_pmd_mlx5_external_sq_disable(uint16_t port_id, uint32_t sq_num);
* - (-EEXIST) if @p cb was already registered.
* - (-ENOMEM) if failed to allocate memory for callback entry.
*/
-__rte_experimental
int
rte_pmd_mlx5_driver_event_cb_register(rte_pmd_mlx5_driver_event_callback_t cb, void *opaque);
@@ -641,7 +640,6 @@ rte_pmd_mlx5_driver_event_cb_register(rte_pmd_mlx5_driver_event_callback_t cb, v
* - 0 if callback was successfully unregistered or if no such callback was registered.
* - (-EINVAL) if @p cb is NULL.
*/
-__rte_experimental
int
rte_pmd_mlx5_driver_event_cb_unregister(rte_pmd_mlx5_driver_event_callback_t cb);
@@ -679,7 +677,6 @@ rte_pmd_mlx5_driver_event_cb_unregister(rte_pmd_mlx5_driver_event_callback_t cb)
* with other functions from ethdev API used to configure any of the mlx5 ports.
* It is the responsibility of the application to enforce this.
*/
-__rte_experimental
void
rte_pmd_mlx5_disable_steering(void);
@@ -697,7 +694,6 @@ rte_pmd_mlx5_disable_steering(void);
* - 0 - Flow steering was successfully enabled or it flow steering was never disabled.
* - (-EBUSY) - There are mlx5 ports probed and re-enabling steering cannot be done safely.
*/
-__rte_experimental
int
rte_pmd_mlx5_enable_steering(void);
--
2.47.3
^ permalink raw reply related
* Re: [PATCH v4 00/25] Consolidate bus driver infrastructure
From: fengchengwen @ 2026-05-29 7:47 UTC (permalink / raw)
To: David Marchand, dev; +Cc: thomas, stephen, bruce.richardson
In-Reply-To: <20260527075654.3780732-1-david.marchand@redhat.com>
Hi David,
First, this is a great refactoring, thank you very much.
I have tested this patchset on the UACCE bus and it passes. Two issues were
found during testing and review:
1. Patch 25/25 fails to apply to the current main branch.
2. Patch 15/25 (bus: support multiple probing) has issues with directly
enabling multiple probing. I suggest merging this patch into Patch 17/25.
The UACCE bus part has passed testing. Please add my Acked-by tag to the
related commits.
Thanks.
On 5/27/2026 3:56 PM, David Marchand wrote:
> This is a continuation of the work I started on the bus infrastructure,
> but this time, a lot of the changes were done by a AI "friend".
> It is still an unfinished topic as the current series focuses on probing
> only. The detaching/cleanup aspect is postponed to another release/time.
>
> My AI "friend" really *sucked* at git and at separating unrelated changes,
> so it required quite a lot of massage/polishing afterwards.
> But it seems good enough now for upstream submission.
>
> I would like to see this series merged in 26.07, so that we have enough
> time to stabilize it before the next LTS.
> And seeing how it affects drivers, it is probably better to merge it
> the sooner possible (so Thomas does not have to solve too many conflicts
> when pulling next-* subtrees after, especially wrt the last patch).
>
>
> This series refactors the DPDK bus infrastructure to consolidate common
> operations and reduce code duplication across all bus drivers.
> Currently, each bus implements its own specific device/driver lists,
> probe logic, and lookup functions.
> This series moves these common patterns into the EAL bus layer,
> providing generic helpers that all buses can use.
>
> The refactoring removes approximately 1,400 lines of duplicated code across
> the codebase while maintaining full functional equivalence.
>
> Key changes:
> - Factorize device and driver lists into struct rte_bus
> - Implement generic probe, device/driver lookup, and iteration helpers in EAL
> - Introduce conversion macros (RTE_BUS_DEVICE, RTE_BUS_DRIVER, RTE_CLASS_TO_BUS_DEVICE)
> to safely convert between generic and bus-specific types
> - Remove bus-specific device/driver types from most driver code
> - Move probe logic from individual buses to rte_bus_generic_probe()
> - Separate NXP-specific metadata from generic bus structures
>
> Benefits:
> - Significant code reduction (~1,400 lines removed)
> - Consistent behavior across all bus types
> - Simplified bus driver implementation
> - Easier maintenance and future enhancements
>
> The series is structured as a progressive refactoring:
> - Remove redundant checks and helpers (patches 1-5)
> - Add conversion macros and factorize lists (patches 6-8)
> - Consolidate device/driver lookup and iteration (patches 9-11)
> - Refactor probe logic (patches 12-15)
> - Remove bus-specific types from drivers (patches 16-23)
>
> Note on ABI:
> This series breaks the ABI for drivers (changes to rte_pci_device,
> rte_pci_driver, and similar structures for other buses). However, the DPDK
> ABI policy does not provide guarantees for driver-level interfaces.
>
>
^ permalink raw reply
* Re: [PATCH] examples: Fix vm_power_manager scratch area to /run/dpdk/powermanager
From: Bruce Richardson @ 2026-05-29 8:01 UTC (permalink / raw)
To: Sudheendra Sampath; +Cc: dev, Anatoly Burakov, Sivaprasad Tummala
In-Reply-To: <20260528190449.66254-1-giveback4fun@gmail.com>
On Thu, May 28, 2026 at 07:04:48PM +0000, Sudheendra Sampath wrote:
> This patch for bug 1832 will do the following:
> 1. If /run/dpdk is not present, it will create it first with and
> then create powermanager directory underneath it.
> 2. If /run/dpdk is present, it will verify it is actually a directory
> before creating subdirectory, powermanager.
>
I would suggest using $XDG_RUNTIME_DIR for the directory path, rather than
hardcoding it by default. If XDG_RUNTIME_DIR is not set, then maybe
consider using /run/dpdk. However, rather than /run/dpdk, I'd suggest using
the normal runtime dir path on most distros as the default:
/run/user/<uid>.
/Bruce
^ permalink raw reply
* Re: [PATCH v5] net/mlx5: prepend implicit items in sync flow creation path
From: Dariusz Sosnowski @ 2026-05-29 8:44 UTC (permalink / raw)
To: Maxime Peim; +Cc: dev, viacheslavo, bingz, orika, suanmingm, matan
In-Reply-To: <20260527103531.1266488-1-maxime.peim@gmail.com>
On Wed, May 27, 2026 at 12:35:31PM +0200, Maxime Peim wrote:
> In eSwitch mode, the async (template) flow creation path automatically
> prepends implicit match items to scope flow rules to the correct
> representor port:
> - Ingress: REPRESENTED_PORT item matching dev->data->port_id
> - Egress: REG_C_0 TAG item matching the port's tx tag value
>
> The sync path (flow_hw_list_create) was missing this logic, causing all
> flow rules created via the non-template API to match traffic from all
> ports rather than being scoped to the specific representor.
>
> Add the same implicit item prepending to flow_hw_list_create, right
> after pattern validation and before any branching (sample/RSS/single/
> prefix), mirroring the behavior of flow_hw_pattern_template_create
> and flow_hw_get_rule_items. The ingress case prepends
> REPRESENTED_PORT with the current port_id; the egress case prepends
> MLX5_RTE_FLOW_ITEM_TYPE_TAG with REG_C_0 value/mask (skipped when
> user provides an explicit SQ item).
>
> Also fix a pre-existing bug where 'return split' on metadata split
> failure returned a negative int cast to uintptr_t, which callers
> would treat as a valid flow handle instead of an error.
>
> Fixes: e38776c36c8a ("net/mlx5: introduce HWS for non-template flow API")
> Fixes: 821a6a5cc495 ("net/mlx5: add metadata split for compatibility")
> Signed-off-by: Maxime Peim <maxime.peim@gmail.com>
Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
^ permalink raw reply
* Re: [PATCH v6] mempool: improve cache behaviour and performance
From: fengchengwen @ 2026-05-29 8:53 UTC (permalink / raw)
To: Morten Brørup, dev, Andrew Rybchenko, Bruce Richardson,
Jingjing Wu, Praveen Shetty, Hemant Agrawal, Sachin Saxena
In-Reply-To: <20260526140000.175092-1-mb@smartsharesystems.com>
On 5/26/2026 10:00 PM, Morten Brørup wrote:
> This patch refactors the mempool cache to eliminate some unexpected
> behaviour and reduce the mempool cache miss rate.
>
> 1.
> The actual cache size was 1.5 times the cache size specified at run-time
> mempool creation.
> This was obviously not expected by application developers.
>
> 2.
> In get operations, the check for when to use the cache as bounce buffer
> did not respect the run-time configured cache size,
> but compared to the build time maximum possible cache size
> (RTE_MEMPOOL_CACHE_MAX_SIZE, default 512).
> E.g. with a configured cache size of 32 objects, getting 256 objects
> would first fetch 32 + 256 = 288 objects into the cache,
> and then move the 256 objects from the cache to the destination memory,
> instead of fetching the 256 objects directly to the destination memory.
> This had a performance cost.
> However, this is unlikely to occur in real applications, so it is not
> important in itself.
>
> 3.
> When putting objects into a mempool, and the mempool cache did not have
> free space for so many objects,
> the cache was flushed completely, and the new objects were then put into
> the cache.
> I.e. the cache drain level was zero.
> This (complete cache flush) meant that a subsequent get operation (with
> the same number of objects) completely emptied the cache,
> so another subsequent get operation required replenishing the cache.
>
> Similarly,
> When getting objects from a mempool, and the mempool cache did not hold so
> many objects,
> the cache was replenished to cache->size + remaining objects,
> and then (the remaining part of) the requested objects were fetched via
> the cache,
> which left the cache filled (to cache->size) at completion.
> I.e. the cache refill level was cache->size (plus some, depending on
> request size).
>
> (1) was improved by generally comparing to cache->size instead of
> cache->flushthresh, when considering the capacity of the cache.
> The cache->flushthresh field is kept for API/ABI compatibility purposes,
> and initialized to cache->size instead of cache->size * 1.5.
>
> (2) was improved by generally comparing to cache->size / 2 instead of
> RTE_MEMPOOL_CACHE_MAX_SIZE, when checking the bounce buffer limit.
>
> (3) was improved by flushing and replenishing the cache by half its size,
> so a flush/refill can be followed randomly by get or put requests.
> This also reduced the number of objects in each flush/refill operation.
>
> As a consequence of these changes, the size of the array holding the
> objects in the cache (cache->objs[]) no longer needs to be
> 2 * RTE_MEMPOOL_CACHE_MAX_SIZE, and can be reduced to
> RTE_MEMPOOL_CACHE_MAX_SIZE at an API/ABI breaking release.
>
> Performance data:
> With a real WAN Optimization application, where the number of allocated
> packets varies (as they are held in e.g. shaper queues), the mempool
> cache miss rate dropped from ca. 1/20 objects to ca. 1/48 objects.
> This was deployed in production at an ISP, and using an effective cache
> size of 384 objects.
Does the application run as a RTC (run-to-complete) mode?
How about pipeline model which NIC recv packets and enqueue ring, another
work thread dequeue packets, process packets and then free packets mbuf?
^ permalink raw reply
* [v1] crypto/cnxk: add ML crypto support
From: Gowrishankar Muthukrishnan @ 2026-05-29 9:13 UTC (permalink / raw)
To: dev, Akhil Goyal, Nithin Dabilpuram, Kiran Kumar K,
Sunil Kumar Kori, Satha Rao, Harman Kalra, Ankur Dwivedi,
Anoob Joseph, Tejasree Kondoj
Cc: Gowrishankar Muthukrishnan
Add ML-KEM and ML-DSA support.
Signed-off-by: Gowrishankar Muthukrishnan <gmuthukrishn@marvell.com>
---
doc/guides/cryptodevs/features/cn20k.ini | 3 +
doc/guides/cryptodevs/features/default.ini | 1 +
doc/guides/rel_notes/release_26_07.rst | 3 +
drivers/common/cnxk/hw/cpt.h | 1 +
drivers/common/cnxk/meson.build | 1 +
drivers/common/cnxk/roc_cpt.c | 4 +
drivers/common/cnxk/roc_platform.h | 5 +
.../common/cnxk/roc_platform_base_symbols.c | 2 +
drivers/common/cnxk/roc_re.h | 31 ++
drivers/common/cnxk/roc_re_ml_tables.c | 248 +++++++++++
drivers/common/cnxk/roc_re_ml_tables.h | 19 +
drivers/crypto/cnxk/cnxk_ae.h | 399 +++++++++++++++++-
drivers/crypto/cnxk/cnxk_cryptodev.c | 11 +
drivers/crypto/cnxk/cnxk_cryptodev.h | 4 +-
.../crypto/cnxk/cnxk_cryptodev_capabilities.c | 62 ++-
drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 45 +-
16 files changed, 817 insertions(+), 22 deletions(-)
create mode 100644 drivers/common/cnxk/roc_re.h
create mode 100644 drivers/common/cnxk/roc_re_ml_tables.c
create mode 100644 drivers/common/cnxk/roc_re_ml_tables.h
diff --git a/doc/guides/cryptodevs/features/cn20k.ini b/doc/guides/cryptodevs/features/cn20k.ini
index d4c52082c6..8180128744 100644
--- a/doc/guides/cryptodevs/features/cn20k.ini
+++ b/doc/guides/cryptodevs/features/cn20k.ini
@@ -21,6 +21,7 @@ Asymmetric sessionless = Y
Sym raw data path API = Y
Inner checksum = Y
Rx inject = Y
+ML-DSA sign prehash = Y
;
; Supported crypto algorithms of 'cn20k' crypto driver.
@@ -110,6 +111,8 @@ ECDSA = Y
ECPM = Y
SM2 = Y
EdDSA = Y
+ML-DSA = Y
+ML-KEM = Y
;
; Supported Operating systems of the 'cn20k' crypto driver.
diff --git a/doc/guides/cryptodevs/features/default.ini b/doc/guides/cryptodevs/features/default.ini
index d8026c3750..be4ee777fb 100644
--- a/doc/guides/cryptodevs/features/default.ini
+++ b/doc/guides/cryptodevs/features/default.ini
@@ -35,6 +35,7 @@ Cipher multiple data units =
Cipher wrapped key =
Inner checksum =
Rx inject =
+ML-DSA sign prehash =
;
; Supported crypto algorithms of a default crypto driver.
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 8b4f8401e2..67c1842afe 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -87,6 +87,9 @@ New Features
Added AGENTS.md file for AI review
and supporting scripts to review patches and documentation.
+* **Updated Marvell cnxk crypto driver.**
+
+ * Added support for ML-KEM and ML-DSA on CN20K platform.
Removed Items
-------------
diff --git a/drivers/common/cnxk/hw/cpt.h b/drivers/common/cnxk/hw/cpt.h
index eb795f61ac..12aeb4d054 100644
--- a/drivers/common/cnxk/hw/cpt.h
+++ b/drivers/common/cnxk/hw/cpt.h
@@ -58,6 +58,7 @@ enum cpt_eng_type {
CPT_ENG_TYPE_AE = 1,
CPT_ENG_TYPE_SE = 2,
CPT_ENG_TYPE_IE = 3,
+ CPT_ENG_TYPE_RE = 4,
CPT_MAX_ENG_TYPES,
};
diff --git a/drivers/common/cnxk/meson.build b/drivers/common/cnxk/meson.build
index 9db77a9702..3303ad9354 100644
--- a/drivers/common/cnxk/meson.build
+++ b/drivers/common/cnxk/meson.build
@@ -65,6 +65,7 @@ sources = files(
'roc_npc_utils.c',
'roc_platform.c',
'roc_platform_base_symbols.c',
+ 'roc_re_ml_tables.c',
'roc_se.c',
'roc_sso.c',
'roc_sso_debug.c',
diff --git a/drivers/common/cnxk/roc_cpt.c b/drivers/common/cnxk/roc_cpt.c
index 21d5e7f2ba..686c9eae7e 100644
--- a/drivers/common/cnxk/roc_cpt.c
+++ b/drivers/common/cnxk/roc_cpt.c
@@ -632,6 +632,9 @@ roc_cpt_dev_configure(struct roc_cpt *roc_cpt, int nb_lf, bool rxc_ena, uint16_t
eng_grpmsk = (1 << roc_cpt->eng_grp[CPT_ENG_TYPE_AE]) |
(1 << roc_cpt->eng_grp[CPT_ENG_TYPE_SE]);
+ if (roc_model_is_cn20k())
+ eng_grpmsk |= (1 << roc_cpt->eng_grp[CPT_ENG_TYPE_RE]);
+
if (roc_errata_cpt_has_ctx_fetch_issue()) {
ctx_ilen_valid = true;
/* Inbound SA size is max context size */
@@ -1097,6 +1100,7 @@ roc_cpt_eng_grp_add(struct roc_cpt *roc_cpt, enum cpt_eng_type eng_type)
case CPT_ENG_TYPE_AE:
case CPT_ENG_TYPE_SE:
case CPT_ENG_TYPE_IE:
+ case CPT_ENG_TYPE_RE:
break;
default:
ret = -EINVAL;
diff --git a/drivers/common/cnxk/roc_platform.h b/drivers/common/cnxk/roc_platform.h
index e22a50d47a..19f1d6005b 100644
--- a/drivers/common/cnxk/roc_platform.h
+++ b/drivers/common/cnxk/roc_platform.h
@@ -41,6 +41,7 @@
".arch_extension lse\n"
#endif
+#define PLT_ATOMIC RTE_ATOMIC
#define PLT_ASSERT RTE_ASSERT
#define PLT_VERIFY RTE_VERIFY
#define PLT_MEMZONE_NAMESIZE RTE_MEMZONE_NAMESIZE
@@ -217,6 +218,10 @@ plt_thread_is_valid(plt_thread_t thr)
#define plt_memory_order_release rte_memory_order_release
#define plt_memory_order_acquire rte_memory_order_acquire
#define plt_memory_order_relaxed rte_memory_order_relaxed
+#define plt_memory_order_seq_cst rte_memory_order_seq_cst
+
+#define plt_atomic_fetch_add_explicit rte_atomic_fetch_add_explicit
+#define plt_atomic_fetch_sub_explicit rte_atomic_fetch_sub_explicit
#define plt_bit_relaxed_get32 rte_bit_relaxed_get32
#define plt_bit_relaxed_set32 rte_bit_relaxed_set32
diff --git a/drivers/common/cnxk/roc_platform_base_symbols.c b/drivers/common/cnxk/roc_platform_base_symbols.c
index cf080b1bdc..75775fbe09 100644
--- a/drivers/common/cnxk/roc_platform_base_symbols.c
+++ b/drivers/common/cnxk/roc_platform_base_symbols.c
@@ -497,6 +497,8 @@ RTE_EXPORT_INTERNAL_SYMBOL(roc_npc_aged_flow_ctx_get)
RTE_EXPORT_INTERNAL_SYMBOL(roc_npc_defrag_mcam_banks)
RTE_EXPORT_INTERNAL_SYMBOL(roc_npc_get_key_type)
RTE_EXPORT_INTERNAL_SYMBOL(roc_npc_flow_mcam_dump)
+RTE_EXPORT_INTERNAL_SYMBOL(roc_re_ml_zeta_get)
+RTE_EXPORT_INTERNAL_SYMBOL(roc_re_ml_zeta_put)
RTE_EXPORT_INTERNAL_SYMBOL(roc_ree_queues_attach)
RTE_EXPORT_INTERNAL_SYMBOL(roc_ree_queues_detach)
RTE_EXPORT_INTERNAL_SYMBOL(roc_ree_msix_offsets_get)
diff --git a/drivers/common/cnxk/roc_re.h b/drivers/common/cnxk/roc_re.h
new file mode 100644
index 0000000000..a8cbda41c0
--- /dev/null
+++ b/drivers/common/cnxk/roc_re.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2026 Marvell.
+ */
+
+#ifndef __ROC_RE_H__
+#define __ROC_RE_H__
+
+/* RE ML opcodes */
+#define ROC_RE_MAJOR_OP_MLKEM 0x1A
+#define ROC_RE_MAJOR_OP_MLDSA 0x1B
+#define ROC_RE_MINOR_OP_MLKEM_KEYGEN 0x00
+#define ROC_RE_MINOR_OP_MLKEM_ENCAP 0x01
+#define ROC_RE_MINOR_OP_MLKEM_DECAP 0x02
+#define ROC_RE_MINOR_OP_MLDSA_KEYGEN 0x00
+#define ROC_RE_MINOR_OP_MLDSA_SIGN 0x01
+#define ROC_RE_MINOR_OP_MLDSA_VERIFY 0x02
+
+/* ML-KEM param2 fields */
+#define ROC_RE_ML_KEM_PARAM2_INMSG_BIT 4
+#define ROC_RE_ML_KEM_PARAM2_INSEED_BIT 5
+
+/* ML-DSA param2 fields */
+#define ROC_RE_ML_DSA_PARAM2_SIGN_BIT 4
+#define ROC_RE_ML_DSA_PARAM2_SEED_BIT 5
+#define ROC_RE_ML_DSA_PARAM2_CTXN_BIT 8
+
+/* ML-DSA minor op fields */
+#define ROC_RE_ML_DSA_MINOR_SIGN_TYPE_BIT 2
+#define ROC_RE_ML_DSA_MINOR_MSG_TYPE_BIT 4
+
+#endif /* __ROC_RE_H__ */
diff --git a/drivers/common/cnxk/roc_re_ml_tables.c b/drivers/common/cnxk/roc_re_ml_tables.c
new file mode 100644
index 0000000000..933e35e3c9
--- /dev/null
+++ b/drivers/common/cnxk/roc_re_ml_tables.c
@@ -0,0 +1,248 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2026 Marvell.
+ */
+
+#include "roc_re_ml_tables.h"
+
+#define RE_ML_TBL_NAME "re_ml_tbl"
+#define RE_MLKEM_ZETA_LEN 256
+#define RE_MLDSA_ZETA_LEN 1024
+
+/* ML table address and length */
+struct re_ml_entry {
+ const uint8_t *data;
+ int len;
+};
+
+struct re_ml_tbl {
+ PLT_ATOMIC(uint64_t) refcount;
+ uint8_t ml_tbl[];
+};
+
+const uint8_t re_mlkem_zeta_tbl[RE_MLKEM_ZETA_LEN] = {
+ 0x01, 0x00, 0xc1, 0x06, 0x14, 0x0a, 0xd9, 0x0c,
+ 0x52, 0x0a, 0x76, 0x02, 0x69, 0x07, 0x50, 0x03,
+ 0x26, 0x04, 0x7f, 0x07, 0xc1, 0x00, 0x1d, 0x03,
+ 0xe2, 0x0a, 0xbc, 0x0c, 0x39, 0x02, 0xd2, 0x06,
+ 0x28, 0x01, 0x8f, 0x09, 0x3b, 0x05, 0xc4, 0x05,
+ 0xe6, 0x0b, 0x38, 0x00, 0xc0, 0x08, 0x35, 0x05,
+ 0x92, 0x05, 0x2e, 0x08, 0x17, 0x02, 0x42, 0x0b,
+ 0x59, 0x09, 0x3f, 0x0b, 0xb6, 0x07, 0x35, 0x03,
+ 0x21, 0x01, 0x4b, 0x01, 0xb5, 0x0c, 0xdc, 0x06,
+ 0xad, 0x04, 0x00, 0x09, 0xe5, 0x08, 0x07, 0x08,
+ 0x8a, 0x02, 0xb9, 0x07, 0xd1, 0x09, 0x78, 0x02,
+ 0x31, 0x0b, 0x21, 0x00, 0x28, 0x05, 0x7b, 0x07,
+ 0x0f, 0x09, 0x9b, 0x05, 0x27, 0x03, 0xc4, 0x01,
+ 0x9e, 0x05, 0x34, 0x0b, 0xfe, 0x05, 0x62, 0x09,
+ 0x57, 0x0a, 0x39, 0x0a, 0xc9, 0x05, 0x88, 0x02,
+ 0xaa, 0x09, 0x26, 0x0c, 0xcb, 0x04, 0x8e, 0x03,
+ 0x11, 0x00, 0xc9, 0x0a, 0x47, 0x02, 0x59, 0x0a,
+ 0x65, 0x06, 0xd3, 0x02, 0xf0, 0x08, 0x4c, 0x04,
+ 0x81, 0x05, 0x66, 0x0a, 0xd1, 0x0c, 0xe9, 0x00,
+ 0xf4, 0x02, 0x6c, 0x08, 0xc7, 0x0b, 0xea, 0x0b,
+ 0xa7, 0x06, 0x73, 0x06, 0xe5, 0x0a, 0xfd, 0x06,
+ 0x37, 0x07, 0xb8, 0x03, 0xb5, 0x05, 0x7f, 0x0a,
+ 0xab, 0x03, 0x04, 0x09, 0x85, 0x09, 0x54, 0x09,
+ 0xdd, 0x02, 0x21, 0x09, 0x0c, 0x01, 0x81, 0x02,
+ 0x30, 0x06, 0xfa, 0x08, 0xf5, 0x07, 0x94, 0x0c,
+ 0x77, 0x01, 0xf5, 0x09, 0x2a, 0x08, 0x6d, 0x06,
+ 0x27, 0x04, 0x3f, 0x01, 0xd5, 0x0a, 0xf5, 0x02,
+ 0x33, 0x08, 0x31, 0x02, 0xa2, 0x09, 0x22, 0x0a,
+ 0xf4, 0x0a, 0x44, 0x04, 0x93, 0x01, 0x02, 0x04,
+ 0x77, 0x04, 0x66, 0x08, 0xd7, 0x0a, 0x76, 0x03,
+ 0xba, 0x06, 0xbc, 0x04, 0x52, 0x07, 0x05, 0x04,
+ 0x3e, 0x08, 0x77, 0x0b, 0x75, 0x03, 0x6a, 0x08,
+};
+
+const uint8_t re_mldsa_zeta_tbl[RE_MLDSA_ZETA_LEN] = {
+ 0x01, 0x00, 0x00, 0x00, 0x02, 0x5e, 0x49, 0x00,
+ 0x67, 0x75, 0x39, 0x00, 0x69, 0x65, 0x39, 0x00,
+ 0x2b, 0x06, 0x4f, 0x00, 0x73, 0xdf, 0x53, 0x00,
+ 0x33, 0xe0, 0x4f, 0x00, 0x6b, 0x06, 0x4f, 0x00,
+ 0xae, 0xb1, 0x76, 0x00, 0xd5, 0x0d, 0x36, 0x00,
+ 0xb0, 0xed, 0x28, 0x00, 0xe4, 0x7f, 0x20, 0x00,
+ 0x83, 0x72, 0x39, 0x00, 0x4a, 0x89, 0x70, 0x00,
+ 0x92, 0x81, 0x08, 0x00, 0xc8, 0x3d, 0x6d, 0x00,
+ 0x94, 0x72, 0x4c, 0x00, 0xb4, 0xe0, 0x41, 0x00,
+ 0xd2, 0xa3, 0x28, 0x00, 0x8a, 0x52, 0x66, 0x00,
+ 0xa7, 0x18, 0x4a, 0x00, 0x34, 0x40, 0x79, 0x00,
+ 0xee, 0x52, 0x0a, 0x00, 0x81, 0x7d, 0x6b, 0x00,
+ 0x1d, 0x9f, 0x4e, 0x00, 0x77, 0x28, 0x1a, 0x00,
+ 0xdf, 0x71, 0x25, 0x00, 0xee, 0x49, 0x16, 0x00,
+ 0xbd, 0x11, 0x76, 0x00, 0xb7, 0x2b, 0x49, 0x00,
+ 0x97, 0xf6, 0x2a, 0x00, 0xd5, 0xd8, 0x22, 0x00,
+ 0x2a, 0xf7, 0x36, 0x00, 0x1e, 0x91, 0x30, 0x00,
+ 0x3f, 0xd1, 0x29, 0x00, 0x73, 0x26, 0x49, 0x00,
+ 0x5f, 0x68, 0x50, 0x00, 0xa2, 0x10, 0x20, 0x00,
+ 0xf7, 0x87, 0x38, 0x00, 0xc3, 0xb2, 0x11, 0x00,
+ 0xa4, 0x03, 0x06, 0x00, 0xed, 0x2b, 0x0e, 0x00,
+ 0x2c, 0xb7, 0x10, 0x00, 0x35, 0x5f, 0x4a, 0x00,
+ 0x15, 0x9d, 0x1f, 0x00, 0xd4, 0x8c, 0x42, 0x00,
+ 0xf4, 0x77, 0x31, 0x00, 0x12, 0xe6, 0x20, 0x00,
+ 0x1d, 0x1c, 0x34, 0x00, 0x73, 0xd8, 0x1a, 0x00,
+ 0x81, 0x66, 0x73, 0x00, 0x3f, 0x55, 0x49, 0x00,
+ 0xf6, 0x52, 0x39, 0x00, 0x4a, 0x56, 0x62, 0x00,
+ 0x05, 0xad, 0x65, 0x00, 0x1c, 0x9a, 0x43, 0x00,
+ 0x5f, 0xaa, 0x53, 0x00, 0x22, 0xb6, 0x30, 0x00,
+ 0x38, 0x7f, 0x08, 0x00, 0x6d, 0x0e, 0x3b, 0x00,
+ 0xda, 0x83, 0x2c, 0x00, 0x6e, 0x49, 0x1c, 0x00,
+ 0x2b, 0x0e, 0x33, 0x00, 0x70, 0x5b, 0x1c, 0x00,
+ 0xf1, 0xe3, 0x2e, 0x00, 0xb9, 0x7e, 0x13, 0x00,
+ 0x30, 0xa9, 0x57, 0x00, 0xef, 0xc6, 0x3a, 0x00,
+ 0x4c, 0xd5, 0x3f, 0x00, 0xea, 0xb2, 0x4e, 0x00,
+ 0xe1, 0x3e, 0x50, 0x00, 0x75, 0xb1, 0x7b, 0x00,
+ 0xb4, 0x48, 0x26, 0x00, 0x56, 0xf2, 0x1e, 0x00,
+ 0xa2, 0x90, 0x1d, 0x00, 0xd4, 0xa6, 0x45, 0x00,
+ 0x9b, 0xe5, 0x2a, 0x00, 0x9c, 0x58, 0x52, 0x00,
+ 0xf5, 0xf1, 0x6e, 0x00, 0x88, 0x72, 0x3f, 0x00,
+ 0x02, 0x51, 0x17, 0x00, 0x59, 0x5d, 0x07, 0x00,
+ 0xba, 0x87, 0x11, 0x00, 0xa9, 0xac, 0x52, 0x00,
+ 0x9e, 0x3e, 0x77, 0x00, 0xd8, 0x96, 0x02, 0x00,
+ 0xec, 0x92, 0x25, 0x00, 0x12, 0xff, 0x4c, 0x00,
+ 0xe8, 0x4c, 0x40, 0x00, 0x82, 0xa5, 0x4a, 0x00,
+ 0xe6, 0x54, 0x1e, 0x00, 0xc1, 0x16, 0x4f, 0x00,
+ 0x79, 0x7e, 0x1a, 0x00, 0x8f, 0x97, 0x03, 0x00,
+ 0x17, 0x48, 0x4e, 0x00, 0x59, 0xb8, 0x31, 0x00,
+ 0xcc, 0x84, 0x58, 0x00, 0x27, 0x48, 0x1b, 0x00,
+ 0xd0, 0x63, 0x5b, 0x00, 0x7a, 0x78, 0x5d, 0x00,
+ 0x5e, 0x22, 0x35, 0x00, 0x7e, 0x0c, 0x40, 0x00,
+ 0xd1, 0x09, 0x6c, 0x00, 0x32, 0xd5, 0x5b, 0x00,
+ 0xd3, 0xc4, 0x6b, 0x00, 0xcb, 0x8e, 0x25, 0x00,
+ 0x4c, 0x53, 0x2e, 0x00, 0x6c, 0x7a, 0x09, 0x00,
+ 0x20, 0x88, 0x3b, 0x00, 0x5c, 0x28, 0x6d, 0x00,
+ 0xf8, 0xa4, 0x2c, 0x00, 0xaa, 0x7c, 0x33, 0x00,
+ 0xa0, 0xb2, 0x14, 0x00, 0x36, 0x85, 0x55, 0x00,
+ 0x86, 0xf1, 0x28, 0x00, 0x5d, 0x79, 0x55, 0x00,
+ 0x70, 0xf6, 0x4a, 0x00, 0x86, 0x4a, 0x23, 0x00,
+ 0x26, 0xe8, 0x75, 0x00, 0x66, 0xde, 0x78, 0x00,
+ 0x8c, 0x52, 0x05, 0x00, 0x59, 0xdf, 0x7a, 0x00,
+ 0x17, 0x6e, 0x0f, 0x00, 0xda, 0xf3, 0x5b, 0x00,
+ 0x7e, 0x9b, 0x45, 0x00, 0x34, 0x8b, 0x62, 0x00,
+ 0xcb, 0xbe, 0x5d, 0x00, 0x7b, 0x9e, 0x1a, 0x00,
+ 0xd9, 0x06, 0x00, 0x00, 0xc5, 0x57, 0x62, 0x00,
+ 0x3c, 0x4b, 0x57, 0x00, 0xef, 0xa8, 0x69, 0x00,
+ 0x38, 0x98, 0x28, 0x00, 0xfe, 0xb5, 0x64, 0x00,
+ 0xf5, 0xf8, 0x7e, 0x00, 0x78, 0x4e, 0x2a, 0x00,
+ 0x23, 0x0a, 0x12, 0x00, 0xa8, 0x54, 0x01, 0x00,
+ 0xff, 0xb7, 0x09, 0x00, 0x87, 0x5e, 0x43, 0x00,
+ 0xf8, 0x7f, 0x43, 0x00, 0xb4, 0xd5, 0x5c, 0x00,
+ 0x4e, 0xc0, 0x4d, 0x00, 0xaf, 0x28, 0x47, 0x00,
+ 0x5d, 0x73, 0x7f, 0x00, 0x0d, 0x8d, 0x0c, 0x00,
+ 0xd5, 0x66, 0x0f, 0x00, 0x80, 0x6d, 0x5a, 0x00,
+ 0x98, 0xab, 0x61, 0x00, 0x96, 0x5d, 0x18, 0x00,
+ 0x31, 0x7f, 0x43, 0x00, 0x98, 0x82, 0x46, 0x00,
+ 0x60, 0x29, 0x66, 0x00, 0x79, 0xd5, 0x4b, 0x00,
+ 0x06, 0xde, 0x28, 0x00, 0x8d, 0x5d, 0x46, 0x00,
+ 0xe3, 0xb0, 0x49, 0x00, 0x34, 0xb4, 0x09, 0x00,
+ 0xb3, 0x0d, 0x7c, 0x00, 0xb0, 0x68, 0x5a, 0x00,
+ 0xa9, 0x9b, 0x40, 0x00, 0xd5, 0xd3, 0x64, 0x00,
+ 0x2a, 0x76, 0x21, 0x00, 0x91, 0x85, 0x65, 0x00,
+ 0x39, 0x6e, 0x24, 0x00, 0x9b, 0xc3, 0x48, 0x00,
+ 0x59, 0xc7, 0x7b, 0x00, 0x59, 0x58, 0x4f, 0x00,
+ 0xb2, 0x2d, 0x39, 0x00, 0x23, 0x09, 0x23, 0x00,
+ 0x67, 0xeb, 0x12, 0x00, 0xf2, 0x4d, 0x45, 0x00,
+ 0x1c, 0xc3, 0x30, 0x00, 0x24, 0x54, 0x28, 0x00,
+ 0x2e, 0x23, 0x13, 0x00, 0x80, 0xaf, 0x7f, 0x00,
+ 0xcb, 0xbf, 0x2d, 0x00, 0x0b, 0x2a, 0x02, 0x00,
+ 0x2c, 0x83, 0x7e, 0x00, 0x7a, 0x58, 0x26, 0x00,
+ 0x75, 0x33, 0x6b, 0x00, 0x76, 0x5b, 0x09, 0x00,
+ 0xcc, 0xe1, 0x6b, 0x00, 0x1e, 0x06, 0x5e, 0x00,
+ 0x0d, 0xe0, 0x78, 0x00, 0x37, 0x8c, 0x62, 0x00,
+ 0x04, 0xa6, 0x3d, 0x00, 0x3c, 0xe5, 0x4a, 0x00,
+ 0x68, 0x1d, 0x1f, 0x00, 0xbb, 0x30, 0x63, 0x00,
+ 0xb8, 0x61, 0x73, 0x00, 0x6c, 0xa0, 0x5e, 0x00,
+ 0xc7, 0x1a, 0x67, 0x00, 0xc6, 0x1f, 0x20, 0x00,
+ 0xff, 0xa4, 0x5b, 0x00, 0x72, 0xd7, 0x60, 0x00,
+ 0x01, 0xf2, 0x08, 0x00, 0x24, 0xe0, 0x6d, 0x00,
+ 0x6d, 0x0e, 0x08, 0x00, 0x8e, 0x03, 0x56, 0x00,
+ 0x88, 0x56, 0x69, 0x00, 0x3e, 0x6d, 0x1e, 0x00,
+ 0xbd, 0x03, 0x26, 0x00, 0xfa, 0x9d, 0x6a, 0x00,
+ 0x17, 0xc0, 0x07, 0x00, 0xd4, 0xbf, 0x6d, 0x00,
+ 0xbd, 0xd0, 0x74, 0x00, 0xe3, 0xe1, 0x63, 0x00,
+ 0x73, 0x95, 0x51, 0x00, 0x0d, 0xb6, 0x7a, 0x00,
+ 0xba, 0x67, 0x28, 0x00, 0xd4, 0xec, 0x2d, 0x00,
+ 0x8c, 0x01, 0x58, 0x00, 0xf5, 0x4c, 0x3f, 0x00,
+ 0x09, 0x70, 0x0b, 0x00, 0x23, 0x7e, 0x42, 0x00,
+ 0x37, 0xbd, 0x3c, 0x00, 0x33, 0x33, 0x27, 0x00,
+ 0x57, 0x39, 0x67, 0x00, 0x5d, 0x4b, 0x1a, 0x00,
+ 0x26, 0x69, 0x19, 0x00, 0x06, 0xf2, 0x1e, 0x00,
+ 0x4e, 0xc1, 0x11, 0x00, 0xc8, 0x76, 0x4c, 0x00,
+ 0x2f, 0xf4, 0x3c, 0x00, 0x9a, 0xb1, 0x7f, 0x00,
+ 0x6c, 0xf6, 0x6a, 0x00, 0x69, 0x16, 0x2e, 0x00,
+ 0xd6, 0x52, 0x33, 0x00, 0x60, 0x47, 0x03, 0x00,
+ 0x60, 0x52, 0x08, 0x00, 0x78, 0x1e, 0x74, 0x00,
+ 0x16, 0x63, 0x2f, 0x00, 0x11, 0x0a, 0x6f, 0x00,
+ 0xf1, 0xc0, 0x07, 0x00, 0x0b, 0x6d, 0x77, 0x00,
+ 0xf0, 0x1f, 0x0d, 0x00, 0x24, 0x58, 0x34, 0x00,
+ 0xd4, 0x23, 0x02, 0x00, 0x59, 0xc5, 0x68, 0x00,
+ 0x85, 0x88, 0x5e, 0x00, 0x32, 0xaa, 0x2f, 0x00,
+ 0x65, 0xfc, 0x23, 0x00, 0x42, 0x69, 0x5e, 0x00,
+ 0xed, 0xe0, 0x51, 0x00, 0xb3, 0xad, 0x65, 0x00,
+ 0xe6, 0xa5, 0x2c, 0x00, 0xfe, 0xe1, 0x79, 0x00,
+ 0x64, 0x40, 0x7b, 0x00, 0xdd, 0xe1, 0x35, 0x00,
+ 0xac, 0x3a, 0x43, 0x00, 0xde, 0x4a, 0x46, 0x00,
+ 0x14, 0xfe, 0x1c, 0x00, 0xce, 0xf1, 0x73, 0x00,
+ 0x0e, 0x17, 0x10, 0x00, 0xd7, 0xb6, 0x74, 0x00,
+};
+
+const struct re_ml_entry re_ml_zeta_tbl[2] = {
+ {
+ .data = re_mlkem_zeta_tbl,
+ .len = sizeof(re_mlkem_zeta_tbl)
+ },
+ {
+ .data = re_mldsa_zeta_tbl,
+ .len = sizeof(re_mldsa_zeta_tbl)
+ }
+};
+
+int
+roc_re_ml_zeta_get(uint64_t *tbl)
+{
+ int len = (RE_MLKEM_ZETA_LEN + RE_MLDSA_ZETA_LEN);
+ const char name[] = RE_ML_TBL_NAME;
+ const struct plt_memzone *mz;
+ struct re_ml_tbl *ml;
+ uint8_t *data;
+
+ if (tbl == NULL)
+ return -EINVAL;
+
+ mz = plt_memzone_lookup(name);
+ if (mz == NULL) {
+ /* Create memzone first time */
+ mz = plt_memzone_reserve_cache_align(name, sizeof(struct re_ml_tbl) + len);
+ if (mz == NULL)
+ return -ENOMEM;
+ }
+
+ ml = (struct re_ml_tbl *)mz->addr;
+ if (plt_atomic_fetch_add_explicit(&ml->refcount, 1, plt_memory_order_seq_cst) != 0)
+ return 0;
+
+ data = PLT_PTR_ADD(mz->addr, sizeof(uint64_t));
+ memcpy(data, re_ml_zeta_tbl[0].data, re_ml_zeta_tbl[0].len);
+ tbl[0] = plt_cpu_to_be_64((uintptr_t)data);
+
+ data = PLT_PTR_ADD(data, re_ml_zeta_tbl[0].len);
+ memcpy(data, re_ml_zeta_tbl[1].data, re_ml_zeta_tbl[1].len);
+ tbl[1] = plt_cpu_to_be_64((uintptr_t)data);
+
+ return 0;
+}
+
+void
+roc_re_ml_zeta_put(void)
+{
+ const char name[] = RE_ML_TBL_NAME;
+ const struct plt_memzone *mz;
+ struct re_ml_tbl *ml;
+
+ mz = plt_memzone_lookup(name);
+ if (mz == NULL)
+ return;
+
+ ml = (struct re_ml_tbl *)mz->addr;
+ if (plt_atomic_fetch_sub_explicit(&ml->refcount, 1, plt_memory_order_seq_cst) == 1)
+ plt_memzone_free(mz);
+}
diff --git a/drivers/common/cnxk/roc_re_ml_tables.h b/drivers/common/cnxk/roc_re_ml_tables.h
new file mode 100644
index 0000000000..0a425f711d
--- /dev/null
+++ b/drivers/common/cnxk/roc_re_ml_tables.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2026 Marvell.
+ */
+
+#ifndef _ROC_RE_ML_TABLES_H_
+#define _ROC_RE_ML_TABLES_H_
+
+#include "roc_platform.h"
+
+enum roc_re_ml_zeta_idx {
+ ROC_RE_ML_ZETA_IDX_KEM = 0,
+ ROC_RE_ML_ZETA_IDX_DSA,
+ ROC_RE_ML_ZETA_IDX_MAX
+};
+
+int __roc_api roc_re_ml_zeta_get(uint64_t *tbl);
+void __roc_api roc_re_ml_zeta_put(void);
+
+#endif /* _ROC_RE_ML_TABLES_H_ */
diff --git a/drivers/crypto/cnxk/cnxk_ae.h b/drivers/crypto/cnxk/cnxk_ae.h
index 21a0c8068a..691f9bfce5 100644
--- a/drivers/crypto/cnxk/cnxk_ae.h
+++ b/drivers/crypto/cnxk/cnxk_ae.h
@@ -10,6 +10,7 @@
#include <rte_malloc.h>
#include "roc_ae.h"
+#include "roc_re.h"
#include "cnxk_cryptodev_ops.h"
@@ -24,8 +25,11 @@ struct cnxk_ae_sess {
struct rte_crypto_rsa_xform rsa_ctx;
struct rte_crypto_modex_xform mod_ctx;
struct roc_ae_ec_ctx ec_ctx;
+ struct rte_crypto_ml_kem_xform ml_kem_ctx;
+ struct rte_crypto_ml_dsa_xform ml_dsa_ctx;
};
uint64_t *cnxk_fpm_iova;
+ uint64_t *cnxk_ml_iova;
struct roc_ae_ec_group **ec_grp;
uint64_t cpt_inst_w4;
uint64_t cpt_inst_w7;
@@ -52,6 +56,15 @@ struct cnxk_ae_sess {
} hw_ctx __plt_aligned(ROC_ALIGN);
};
+static const uint8_t mldsa_hash_algo[] = {
+ [RTE_CRYPTO_AUTH_SHA3_224] = 0xA,
+ [RTE_CRYPTO_AUTH_SHA3_256] = 0xB,
+ [RTE_CRYPTO_AUTH_SHA3_384] = 0xC,
+ [RTE_CRYPTO_AUTH_SHA3_512] = 0xD,
+ [RTE_CRYPTO_AUTH_SHAKE_128] = 0xE,
+ [RTE_CRYPTO_AUTH_SHAKE_256] = 0xF,
+};
+
static __rte_always_inline void
cnxk_ae_modex_param_normalize(uint8_t **data, size_t *len, size_t max)
{
@@ -259,6 +272,32 @@ cnxk_ae_fill_ec_params(struct cnxk_ae_sess *sess, struct rte_crypto_asym_xform *
return 0;
}
+static __rte_always_inline int
+cnxk_ae_fill_ml_kem_params(struct cnxk_ae_sess *sess,
+ struct rte_crypto_asym_xform *xform)
+{
+ struct rte_crypto_ml_kem_xform *ml_kem = &sess->ml_kem_ctx;
+ if (xform->mlkem.type == RTE_CRYPTO_ML_KEM_NONE)
+ return -EINVAL;
+
+ ml_kem->type = xform->mlkem.type;
+ return 0;
+}
+
+static __rte_always_inline int
+cnxk_ae_fill_ml_dsa_params(struct cnxk_ae_sess *sess,
+ struct rte_crypto_asym_xform *xform)
+{
+ struct rte_crypto_ml_dsa_xform *ml_dsa = &sess->ml_dsa_ctx;
+ if (xform->mldsa.type == RTE_CRYPTO_ML_DSA_NONE)
+ return -EINVAL;
+
+ ml_dsa->type = xform->mldsa.type;
+ ml_dsa->sign_deterministic = xform->mldsa.sign_deterministic;
+ ml_dsa->sign_prehash = xform->mldsa.sign_prehash;
+ return 0;
+}
+
static __rte_always_inline int
cnxk_ae_fill_session_parameters(struct cnxk_ae_sess *sess,
struct rte_crypto_asym_xform *xform)
@@ -284,6 +323,12 @@ cnxk_ae_fill_session_parameters(struct cnxk_ae_sess *sess,
case RTE_CRYPTO_ASYM_XFORM_EDDSA:
ret = cnxk_ae_fill_ec_params(sess, xform);
break;
+ case RTE_CRYPTO_ASYM_XFORM_ML_KEM:
+ ret = cnxk_ae_fill_ml_kem_params(sess, xform);
+ break;
+ case RTE_CRYPTO_ASYM_XFORM_ML_DSA:
+ ret = cnxk_ae_fill_ml_dsa_params(sess, xform);
+ break;
default:
return -ENOTSUP;
}
@@ -563,6 +608,280 @@ cnxk_ae_enqueue_rsa_op(struct rte_crypto_op *op, struct roc_ae_buf_ptr *meta_buf
return 0;
}
+static __rte_always_inline int __rte_hot
+cnxk_ae_enqueue_ml_kem_op(struct rte_crypto_op *op, struct roc_ae_buf_ptr *meta_buf,
+ struct cnxk_ae_sess *sess, struct cpt_inst_s *inst)
+{
+ size_t metabuf_len = cnxk_cpt_asym_get_mlen(), reqbuf_len;
+ struct rte_crypto_ml_kem_op *mlkem = &op->asym->mlkem;
+ union cpt_inst_w4 w4;
+ uint32_t dlen = 0;
+ uint16_t param2;
+ uint8_t *dptr;
+
+ /* Input buffer */
+ dptr = meta_buf->vaddr;
+ inst->dptr = (uintptr_t)dptr;
+
+ switch (mlkem->op) {
+ case RTE_CRYPTO_ML_KEM_OP_KEYGEN:
+ reqbuf_len = mlkem->keygen.d.length + mlkem->keygen.z.length;
+ if (reqbuf_len > (metabuf_len - dlen)) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -ENOMEM;
+ }
+
+ memcpy(dptr, mlkem->keygen.d.data, mlkem->keygen.d.length);
+ dptr += mlkem->keygen.d.length;
+ memcpy(dptr, mlkem->keygen.z.data, mlkem->keygen.z.length);
+ dptr += mlkem->keygen.z.length;
+
+ dlen = mlkem->keygen.d.length + mlkem->keygen.z.length;
+ w4.s.opcode_major = ROC_RE_MAJOR_OP_MLKEM;
+ w4.s.opcode_minor = ROC_RE_MINOR_OP_MLKEM_KEYGEN;
+ param2 = sess->ml_kem_ctx.type;
+ param2 |= (!!dlen << ROC_RE_ML_KEM_PARAM2_INSEED_BIT);
+ break;
+ case RTE_CRYPTO_ML_KEM_OP_ENCAP:
+ reqbuf_len = mlkem->encap.message.length + mlkem->encap.ek.length;
+ if (reqbuf_len > (metabuf_len - dlen)) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -ENOMEM;
+ }
+
+ memcpy(dptr, mlkem->encap.message.data, mlkem->encap.message.length);
+ dptr += mlkem->encap.message.length;
+ memcpy(dptr, mlkem->encap.ek.data, mlkem->encap.ek.length);
+ dptr += mlkem->encap.ek.length;
+
+ dlen = mlkem->encap.message.length + mlkem->encap.ek.length;
+ w4.s.opcode_major = ROC_RE_MAJOR_OP_MLKEM;
+ w4.s.opcode_minor = ROC_RE_MINOR_OP_MLKEM_ENCAP;
+ param2 = sess->ml_kem_ctx.type;
+ param2 |= (!!mlkem->encap.message.length << ROC_RE_ML_KEM_PARAM2_INMSG_BIT);
+ break;
+ case RTE_CRYPTO_ML_KEM_OP_DECAP:
+ reqbuf_len = mlkem->decap.dk.length + mlkem->decap.cipher.length;
+ if (reqbuf_len > (metabuf_len - dlen)) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -ENOMEM;
+ }
+
+ memcpy(dptr, mlkem->decap.dk.data, mlkem->decap.dk.length);
+ dptr += mlkem->decap.dk.length;
+ memcpy(dptr, mlkem->decap.cipher.data, mlkem->decap.cipher.length);
+ dptr += mlkem->decap.cipher.length;
+
+ dlen = mlkem->decap.cipher.length + mlkem->decap.dk.length;
+ w4.s.opcode_major = ROC_RE_MAJOR_OP_MLKEM;
+ w4.s.opcode_minor = ROC_RE_MINOR_OP_MLKEM_DECAP;
+ param2 = sess->ml_kem_ctx.type;
+ break;
+ default:
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -EINVAL;
+ }
+
+ w4.s.param1 = 0;
+ w4.s.param2 = param2;
+ w4.s.dlen = dlen;
+
+ inst->w4.u64 = w4.u64;
+
+ /* Reuse entire space of meta buffer as output is large in PQC */
+ inst->rptr = (uintptr_t)meta_buf->vaddr;
+
+ return 0;
+}
+
+static __rte_always_inline int __rte_hot
+cnxk_ae_enqueue_ml_dsa_op(struct rte_crypto_op *op, struct roc_ae_buf_ptr *meta_buf,
+ struct cnxk_ae_sess *sess, struct cpt_inst_s *inst)
+{
+ size_t metabuf_len = cnxk_cpt_asym_get_mlen(), reqbuf_len;
+ struct rte_crypto_ml_dsa_op *mldsa = &op->asym->mldsa;
+ enum rte_crypto_auth_algorithm hash;
+ bool sign_deterministic;
+ union cpt_inst_w4 w4;
+ uint16_t param1 = 0;
+ uint32_t dlen = 0;
+ uint16_t param2;
+ uint8_t *dptr;
+ uint8_t minor;
+
+ /* Input buffer */
+ dptr = meta_buf->vaddr;
+ inst->dptr = (uintptr_t)dptr;
+
+ switch (mldsa->op) {
+ case RTE_CRYPTO_ML_DSA_OP_KEYGEN:
+ reqbuf_len = mldsa->keygen.seed.length;
+ if (reqbuf_len > (metabuf_len - dlen)) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -ENOMEM;
+ }
+
+ param2 = sess->ml_dsa_ctx.type;
+
+ memcpy(dptr, mldsa->keygen.seed.data, mldsa->keygen.seed.length);
+ dptr += mldsa->keygen.seed.length;
+ param2 |= (!!mldsa->keygen.seed.length << ROC_RE_ML_DSA_PARAM2_SEED_BIT);
+
+ dlen += mldsa->keygen.seed.length;
+ w4.s.opcode_major = ROC_RE_MAJOR_OP_MLDSA;
+ w4.s.opcode_minor = ROC_RE_MINOR_OP_MLDSA_KEYGEN;
+ break;
+ case RTE_CRYPTO_ML_DSA_OP_SIGN:
+ reqbuf_len = mldsa->siggen.message.length + mldsa->siggen.privkey.length +
+ mldsa->siggen.ctx.length + mldsa->siggen.mu.length +
+ mldsa->siggen.seed.length;
+
+ if (reqbuf_len > (metabuf_len - dlen)) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -ENOMEM;
+ }
+
+ sign_deterministic = sess->ml_dsa_ctx.sign_deterministic;
+ hash = op->asym->mldsa.siggen.hash;
+ minor = ROC_RE_MINOR_OP_MLDSA_SIGN;
+
+ param1 = mldsa->siggen.message.length;
+ param2 = sess->ml_dsa_ctx.type;
+ if (hash == 0) {
+ minor |= (0 << ROC_RE_ML_DSA_MINOR_MSG_TYPE_BIT);
+ } else if (mldsa->siggen.mu.length != 0) {
+ minor |= (3 << ROC_RE_ML_DSA_MINOR_MSG_TYPE_BIT);
+ } else {
+ if (!sess->ml_dsa_ctx.sign_prehash ||
+ hash >= RTE_DIM(mldsa_hash_algo) || mldsa_hash_algo[hash] == 0) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -EINVAL;
+ }
+
+ minor |= (1 << ROC_RE_ML_DSA_MINOR_MSG_TYPE_BIT);
+ param2 |= (mldsa_hash_algo[hash] << ROC_RE_ML_DSA_PARAM2_SIGN_BIT);
+ }
+
+ minor |= ((sign_deterministic ? 0 : 2) << ROC_RE_ML_DSA_MINOR_SIGN_TYPE_BIT);
+
+ if (!sign_deterministic) {
+ if (!mldsa->siggen.seed.length) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -EINVAL;
+ }
+
+ memcpy(dptr, mldsa->siggen.seed.data, mldsa->siggen.seed.length);
+ dptr += mldsa->siggen.seed.length;
+ dlen += mldsa->siggen.seed.length;
+ }
+
+ memcpy(dptr, mldsa->siggen.privkey.data, mldsa->siggen.privkey.length);
+ dptr += mldsa->siggen.privkey.length;
+ dlen += mldsa->siggen.privkey.length;
+
+ memcpy(dptr, mldsa->siggen.ctx.data, mldsa->siggen.ctx.length);
+ dptr += mldsa->siggen.ctx.length;
+ dlen += mldsa->siggen.ctx.length;
+ if (mldsa->siggen.ctx.length > (UINT16_MAX >> ROC_RE_ML_DSA_PARAM2_CTXN_BIT)) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -EINVAL;
+ }
+ param2 |= ((uint16_t)mldsa->siggen.ctx.length
+ << ROC_RE_ML_DSA_PARAM2_CTXN_BIT);
+
+ if (mldsa->siggen.mu.length != 0) {
+ memcpy(dptr, mldsa->siggen.mu.data, mldsa->siggen.mu.length);
+ dptr += mldsa->siggen.mu.length;
+ dlen += mldsa->siggen.mu.length;
+ param1 = mldsa->siggen.mu.length;
+ } else if (mldsa->siggen.message.length != 0) {
+ memcpy(dptr, mldsa->siggen.message.data, mldsa->siggen.message.length);
+ dptr += mldsa->siggen.message.length;
+ dlen += mldsa->siggen.message.length;
+ }
+
+ w4.s.opcode_major = ROC_RE_MAJOR_OP_MLDSA;
+ w4.s.opcode_minor = minor;
+ break;
+ case RTE_CRYPTO_ML_DSA_OP_VERIFY:
+ reqbuf_len = mldsa->sigver.message.length + mldsa->sigver.pubkey.length +
+ mldsa->sigver.ctx.length + mldsa->sigver.mu.length +
+ mldsa->sigver.sign.length;
+
+ if (reqbuf_len > (metabuf_len - dlen)) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -ENOMEM;
+ }
+
+ hash = op->asym->mldsa.sigver.hash;
+ minor = ROC_RE_MINOR_OP_MLDSA_VERIFY;
+
+ param1 = mldsa->sigver.message.length;
+ param2 = sess->ml_dsa_ctx.type;
+ if (hash == 0) {
+ minor |= (0 << ROC_RE_ML_DSA_MINOR_MSG_TYPE_BIT);
+ } else if (mldsa->sigver.mu.length != 0) {
+ minor |= (3 << ROC_RE_ML_DSA_MINOR_MSG_TYPE_BIT);
+ } else {
+ if (!sess->ml_dsa_ctx.sign_prehash ||
+ hash >= RTE_DIM(mldsa_hash_algo) || mldsa_hash_algo[hash] == 0) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -EINVAL;
+ }
+
+ minor |= (1 << ROC_RE_ML_DSA_MINOR_MSG_TYPE_BIT);
+ param2 |= (mldsa_hash_algo[hash] << ROC_RE_ML_DSA_PARAM2_SIGN_BIT);
+ }
+
+ memcpy(dptr, mldsa->sigver.pubkey.data, mldsa->sigver.pubkey.length);
+ dptr += mldsa->sigver.pubkey.length;
+ dlen += mldsa->sigver.pubkey.length;
+
+ memcpy(dptr, mldsa->sigver.ctx.data, mldsa->sigver.ctx.length);
+ dptr += mldsa->sigver.ctx.length;
+ dlen += mldsa->sigver.ctx.length;
+ if (mldsa->sigver.ctx.length > (UINT16_MAX >> ROC_RE_ML_DSA_PARAM2_CTXN_BIT)) {
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -EINVAL;
+ }
+ param2 |= ((uint16_t)mldsa->sigver.ctx.length
+ << ROC_RE_ML_DSA_PARAM2_CTXN_BIT);
+
+ if (mldsa->sigver.mu.length != 0) {
+ memcpy(dptr, mldsa->sigver.mu.data, mldsa->sigver.mu.length);
+ dptr += mldsa->sigver.mu.length;
+ dlen += mldsa->sigver.mu.length;
+ param1 = mldsa->sigver.mu.length;
+ } else if (mldsa->sigver.message.length != 0) {
+ memcpy(dptr, mldsa->sigver.message.data, mldsa->sigver.message.length);
+ dptr += mldsa->sigver.message.length;
+ dlen += mldsa->sigver.message.length;
+ }
+
+ memcpy(dptr, mldsa->sigver.sign.data, mldsa->sigver.sign.length);
+ dptr += mldsa->sigver.sign.length;
+ dlen += mldsa->sigver.sign.length;
+
+ w4.s.opcode_major = ROC_RE_MAJOR_OP_MLDSA;
+ w4.s.opcode_minor = minor;
+ break;
+ default:
+ op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+ return -EINVAL;
+ }
+
+ w4.s.param1 = param1;
+ w4.s.param2 = param2;
+ w4.s.dlen = dlen;
+
+ inst->w4.u64 = w4.u64;
+
+ /* Reuse entire space of meta buffer as output is large in PQC */
+ inst->rptr = (uintptr_t)meta_buf->vaddr;
+
+ return 0;
+}
+
static __rte_always_inline void
cnxk_ae_ecdsa_sign_prep(struct rte_crypto_ecdsa_op_param *ecdsa,
struct roc_ae_buf_ptr *meta_buf,
@@ -1712,6 +2031,55 @@ cnxk_ae_dequeue_ecdh_op(struct rte_crypto_ecdh_op_param *ecdh, uint8_t *rptr,
}
}
+static __rte_always_inline void
+cnxk_ae_dequeue_mlkem_op(struct rte_crypto_ml_kem_op *mlkem, uint8_t *rptr,
+ enum rte_crypto_ml_kem_type type)
+{
+ switch (mlkem->op) {
+ case RTE_CRYPTO_ML_KEM_OP_KEYGEN:
+ mlkem->keygen.dk.length = rte_crypto_ml_kem_privkey_size[type];
+ memcpy(mlkem->keygen.dk.data, rptr, mlkem->keygen.dk.length);
+ mlkem->keygen.ek.length = rte_crypto_ml_kem_pubkey_size[type];
+ memcpy(mlkem->keygen.ek.data, rptr + 384 * (type + 1), mlkem->keygen.ek.length);
+ break;
+ case RTE_CRYPTO_ML_KEM_OP_ENCAP:
+ mlkem->encap.sk.length = 32;
+ memcpy(mlkem->encap.sk.data, rptr, mlkem->encap.sk.length);
+ mlkem->encap.cipher.length = rte_crypto_ml_kem_cipher_size[type];
+ memcpy(mlkem->encap.cipher.data, rptr + 32, mlkem->encap.cipher.length);
+ break;
+ case RTE_CRYPTO_ML_KEM_OP_DECAP:
+ mlkem->decap.sk.length = 32;
+ memcpy(mlkem->decap.sk.data, rptr, mlkem->decap.sk.length);
+ break;
+ default:
+ break;
+ }
+}
+
+static __rte_always_inline void
+cnxk_ae_dequeue_mldsa_op(struct rte_crypto_ml_dsa_op *mldsa, uint8_t *rptr,
+ enum rte_crypto_ml_dsa_type type)
+{
+ switch (mldsa->op) {
+ case RTE_CRYPTO_ML_DSA_OP_KEYGEN:
+ mldsa->keygen.pubkey.length = rte_crypto_ml_dsa_pubkey_size[type];
+ memcpy(mldsa->keygen.pubkey.data, rptr, mldsa->keygen.pubkey.length);
+ mldsa->keygen.privkey.length = rte_crypto_ml_dsa_privkey_size[type];
+ memcpy(mldsa->keygen.privkey.data, rptr + mldsa->keygen.pubkey.length,
+ mldsa->keygen.privkey.length);
+ break;
+ case RTE_CRYPTO_ML_DSA_OP_SIGN:
+ mldsa->siggen.sign.length = rte_crypto_ml_dsa_sign_size[type];
+ memcpy(mldsa->siggen.sign.data, rptr, mldsa->siggen.sign.length);
+ break;
+ case RTE_CRYPTO_ML_DSA_OP_VERIFY:
+ break;
+ default:
+ break;
+ }
+}
+
static __rte_always_inline void *
cnxk_ae_alloc_meta(struct roc_ae_buf_ptr *buf,
struct rte_mempool *cpt_meta_pool,
@@ -1752,56 +2120,46 @@ cnxk_ae_enqueue(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op,
switch (sess->xfrm_type) {
case RTE_CRYPTO_ASYM_XFORM_MODEX:
ret = cnxk_ae_modex_prep(op, &meta_buf, &sess->mod_ctx, inst);
- if (unlikely(ret))
- goto req_fail;
break;
case RTE_CRYPTO_ASYM_XFORM_RSA:
ret = cnxk_ae_enqueue_rsa_op(op, &meta_buf, sess, inst);
- if (unlikely(ret))
- goto req_fail;
break;
case RTE_CRYPTO_ASYM_XFORM_ECDSA:
ret = cnxk_ae_enqueue_ecdsa_op(op, &meta_buf, sess,
sess->cnxk_fpm_iova,
sess->ec_grp, inst);
- if (unlikely(ret))
- goto req_fail;
break;
case RTE_CRYPTO_ASYM_XFORM_EDDSA:
ret = cnxk_ae_enqueue_eddsa_op(op, &meta_buf, sess,
sess->cnxk_fpm_iova,
sess->ec_grp, inst);
- if (unlikely(ret))
- goto req_fail;
break;
case RTE_CRYPTO_ASYM_XFORM_SM2:
ret = cnxk_ae_enqueue_sm2_op(op, &meta_buf, sess,
sess->cnxk_fpm_iova,
sess->ec_grp, inst);
- if (unlikely(ret))
- goto req_fail;
break;
case RTE_CRYPTO_ASYM_XFORM_ECPM:
ret = cnxk_ae_ecpm_prep(&asym_op->ecpm.scalar, &asym_op->ecpm.p, &meta_buf,
sess->ec_grp[sess->ec_ctx.curveid],
sess->ec_ctx.curveid, inst);
- if (unlikely(ret))
- goto req_fail;
break;
case RTE_CRYPTO_ASYM_XFORM_ECFPM:
ret = cnxk_ae_ecfpm_prep(&asym_op->ecpm.scalar, &meta_buf,
sess->cnxk_fpm_iova,
sess->ec_grp[sess->ec_ctx.curveid],
sess->ec_ctx.curveid, inst);
- if (unlikely(ret))
- goto req_fail;
break;
case RTE_CRYPTO_ASYM_XFORM_ECDH:
ret = cnxk_ae_enqueue_ecdh_op(op, &meta_buf, sess,
sess->cnxk_fpm_iova,
sess->ec_grp, inst);
- if (unlikely(ret))
- goto req_fail;
+ break;
+ case RTE_CRYPTO_ASYM_XFORM_ML_KEM:
+ ret = cnxk_ae_enqueue_ml_kem_op(op, &meta_buf, sess, inst);
+ break;
+ case RTE_CRYPTO_ASYM_XFORM_ML_DSA:
+ ret = cnxk_ae_enqueue_ml_dsa_op(op, &meta_buf, sess, inst);
break;
default:
op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
@@ -1809,6 +2167,9 @@ cnxk_ae_enqueue(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op,
goto req_fail;
}
+ if (unlikely(ret))
+ goto req_fail;
+
mop = mdata;
mop[0] = inst->rptr;
return 0;
@@ -1852,6 +2213,12 @@ cnxk_ae_post_process(struct rte_crypto_op *cop, struct cnxk_ae_sess *sess,
cnxk_ae_dequeue_ecdh_op(&op->ecdh, rptr, &sess->ec_ctx,
sess->ec_grp, op->flags);
break;
+ case RTE_CRYPTO_ASYM_XFORM_ML_KEM:
+ cnxk_ae_dequeue_mlkem_op(&op->mlkem, rptr, sess->ml_kem_ctx.type);
+ break;
+ case RTE_CRYPTO_ASYM_XFORM_ML_DSA:
+ cnxk_ae_dequeue_mldsa_op(&op->mldsa, rptr, sess->ml_dsa_ctx.type);
+ break;
default:
cop->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
break;
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev.c b/drivers/crypto/cnxk/cnxk_cryptodev.c
index 5828a502e4..de27c4a580 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev.c
@@ -28,6 +28,9 @@ cnxk_cpt_default_ff_get(void)
if (roc_model_is_cn10ka_b0() || roc_model_is_cn10kb() || roc_model_is_cn20k())
ff |= RTE_CRYPTODEV_FF_SECURITY_RX_INJECT;
+ if (roc_model_is_cn20k())
+ ff |= RTE_CRYPTODEV_FF_MLDSA_SIGN_PREHASH;
+
return ff;
}
@@ -56,6 +59,14 @@ cnxk_cpt_eng_grp_add(struct roc_cpt *roc_cpt)
return -ENOTSUP;
}
+ if (roc_model_is_cn20k()) {
+ ret = roc_cpt_eng_grp_add(roc_cpt, CPT_ENG_TYPE_RE);
+ if (ret < 0) {
+ plt_err("Could not add CPT RE engines");
+ return ret;
+ }
+ }
+
return 0;
}
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev.h b/drivers/crypto/cnxk/cnxk_cryptodev.h
index f88162ad3c..bdc5752905 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev.h
+++ b/drivers/crypto/cnxk/cnxk_cryptodev.h
@@ -10,8 +10,9 @@
#include "roc_ae.h"
#include "roc_cpt.h"
+#include "roc_re_ml_tables.h"
-#define CNXK_CPT_MAX_CAPS 60
+#define CNXK_CPT_MAX_CAPS 62
#define CNXK_SEC_IPSEC_CRYPTO_MAX_CAPS 16
#define CNXK_SEC_TLS_1_3_CRYPTO_MAX_CAPS 3
#define CNXK_SEC_TLS_1_2_CRYPTO_MAX_CAPS 7
@@ -33,6 +34,7 @@ struct cnxk_cpt_vf {
sec_dtls_1_2_crypto_caps[CNXK_SEC_TLS_1_2_CRYPTO_MAX_CAPS];
struct rte_security_capability sec_caps[CNXK_SEC_MAX_CAPS];
uint64_t cnxk_fpm_iova[ROC_AE_EC_ID_PMAX];
+ uint64_t cnxk_ml_iova[ROC_RE_ML_ZETA_IDX_MAX];
struct roc_ae_ec_group *ec_grp[ROC_AE_EC_ID_PMAX];
uint16_t max_qps_limit;
uint16_t rx_inject_qp;
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
index be6d383717..736d588bde 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_capabilities.c
@@ -129,6 +129,63 @@ static const struct rte_cryptodev_capabilities caps_mul[] = {
},
};
+static const struct rte_cryptodev_capabilities caps_pqc[] = {
+ {
+ /* ML-KEM */
+ .op = RTE_CRYPTO_OP_TYPE_ASYMMETRIC,
+ .asym = {
+ .xform_capa = {
+ .xform_type = RTE_CRYPTO_ASYM_XFORM_ML_KEM,
+ .op_types =
+ ((1 << RTE_CRYPTO_ML_KEM_OP_KEYGEN) |
+ (1 << RTE_CRYPTO_ML_KEM_OP_ENCAP) |
+ (1 << RTE_CRYPTO_ML_KEM_OP_DECAP)),
+ .mlkem_capa = {
+ [RTE_CRYPTO_ML_KEM_OP_KEYGEN] =
+ (1 << RTE_CRYPTO_ML_KEM_512) |
+ (1 << RTE_CRYPTO_ML_KEM_768) |
+ (1 << RTE_CRYPTO_ML_KEM_1024),
+ [RTE_CRYPTO_ML_KEM_OP_ENCAP] =
+ (1 << RTE_CRYPTO_ML_KEM_512) |
+ (1 << RTE_CRYPTO_ML_KEM_768) |
+ (1 << RTE_CRYPTO_ML_KEM_1024),
+ [RTE_CRYPTO_ML_KEM_OP_DECAP] =
+ (1 << RTE_CRYPTO_ML_KEM_512) |
+ (1 << RTE_CRYPTO_ML_KEM_768) |
+ (1 << RTE_CRYPTO_ML_KEM_1024)
+ }
+ }
+ }
+ },
+ {
+ /* ML-DSA */
+ .op = RTE_CRYPTO_OP_TYPE_ASYMMETRIC,
+ .asym = {
+ .xform_capa = {
+ .xform_type = RTE_CRYPTO_ASYM_XFORM_ML_DSA,
+ .op_types =
+ ((1 << RTE_CRYPTO_ML_DSA_OP_SIGN) |
+ (1 << RTE_CRYPTO_ML_DSA_OP_KEYGEN) |
+ (1 << RTE_CRYPTO_ML_DSA_OP_VERIFY)),
+ .mldsa_capa = {
+ [RTE_CRYPTO_ML_DSA_OP_KEYGEN] =
+ (1 << RTE_CRYPTO_ML_DSA_44) |
+ (1 << RTE_CRYPTO_ML_DSA_65) |
+ (1 << RTE_CRYPTO_ML_DSA_87),
+ [RTE_CRYPTO_ML_DSA_OP_SIGN] =
+ (1 << RTE_CRYPTO_ML_DSA_44) |
+ (1 << RTE_CRYPTO_ML_DSA_65) |
+ (1 << RTE_CRYPTO_ML_DSA_87),
+ [RTE_CRYPTO_ML_DSA_OP_VERIFY] =
+ (1 << RTE_CRYPTO_ML_DSA_44) |
+ (1 << RTE_CRYPTO_ML_DSA_65) |
+ (1 << RTE_CRYPTO_ML_DSA_87)
+ }
+ }
+ }
+ },
+};
+
static const struct rte_cryptodev_capabilities caps_sha1_sha2[] = {
{ /* SHA1 */
.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
@@ -2079,10 +2136,13 @@ crypto_caps_populate(struct rte_cryptodev_capabilities cnxk_caps[],
if (roc_model_is_cn10k() || roc_model_is_cn20k())
cn10k_20k_crypto_caps_add(cnxk_caps, hw_caps, &cur_pos);
- if (roc_model_is_cn20k())
+ if (roc_model_is_cn20k()) {
CPT_CAPS_ADD(cnxk_caps, &cur_pos, hw_caps, zuc256_snow5g);
+ cpt_caps_add(cnxk_caps, &cur_pos, caps_pqc, RTE_DIM(caps_pqc));
+ }
cpt_caps_add(cnxk_caps, &cur_pos, caps_null, RTE_DIM(caps_null));
+
cpt_caps_add(cnxk_caps, &cur_pos, caps_end, RTE_DIM(caps_end));
if (roc_model_is_cn10k() || roc_model_is_cn20k())
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_ops.c b/drivers/crypto/cnxk/cnxk_cryptodev_ops.c
index 2f9eb322dc..0f44a393db 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_ops.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_ops.c
@@ -23,6 +23,7 @@
#else
#include "roc_io_generic.h"
#endif
+#include "roc_re_ml_tables.h"
#include "cnxk_ae.h"
#include "cnxk_cryptodev.h"
@@ -41,6 +42,14 @@
#define CNXK_CPT_MAX_ASYM_OP_NUM_PARAMS 5
#define CNXK_CPT_MAX_ASYM_OP_MOD_LEN 1024
+
+/*
+ * PQC requests currently use a shared metabuf region for concatenated input
+ * and output. ML-DSA-87 SIGN requires at least 9523 bytes for private key
+ * input plus signature output, along with additional space for message and
+ * context parameters, so set it for the possible max.
+ */
+#define CNXK_CPT_MAX_ASYM_OP_PQC_LEN 16384
#define CNXK_CPT_META_BUF_MAX_CACHE_SIZE 128
static_assert((uint16_t)RTE_PMD_CNXK_AE_EC_ID_P192 == (uint16_t)ROC_AE_EC_ID_P192,
@@ -107,7 +116,10 @@ cnxk_cpt_asym_get_mlen(void)
len = sizeof(uint64_t);
/* Get meta len for asymmetric operations */
- len += CNXK_CPT_MAX_ASYM_OP_NUM_PARAMS * CNXK_CPT_MAX_ASYM_OP_MOD_LEN;
+ if (roc_model_is_cn20k())
+ len += CNXK_CPT_MAX_ASYM_OP_PQC_LEN;
+ else
+ len += CNXK_CPT_MAX_ASYM_OP_NUM_PARAMS * CNXK_CPT_MAX_ASYM_OP_MOD_LEN;
return len;
}
@@ -121,6 +133,8 @@ cnxk_cpt_dev_clear(struct rte_cryptodev *dev)
if (dev->feature_flags & RTE_CRYPTODEV_FF_ASYMMETRIC_CRYPTO) {
roc_ae_fpm_put();
roc_ae_ec_grp_put();
+ if (roc_model_is_cn20k())
+ roc_re_ml_zeta_put();
}
ret = roc_cpt_int_misc_cb_unregister(cnxk_cpt_int_misc_cb, NULL);
@@ -182,8 +196,15 @@ cnxk_cpt_dev_config(struct rte_cryptodev *dev, struct rte_cryptodev_config *conf
ret = roc_ae_ec_grp_get(vf->ec_grp);
if (ret) {
plt_err("Could not get EC grp table");
- roc_ae_fpm_put();
- return ret;
+ goto fpm_put;
+ }
+
+ if (roc_model_is_cn20k()) {
+ ret = roc_re_ml_zeta_get(vf->cnxk_ml_iova);
+ if (ret) {
+ plt_err("Could not initialize RE ML lookup table");
+ goto ec_grp_put;
+ }
}
}
roc_cpt->opaque = dev;
@@ -191,6 +212,12 @@ cnxk_cpt_dev_config(struct rte_cryptodev *dev, struct rte_cryptodev_config *conf
roc_cpt_int_misc_cb_register(cnxk_cpt_int_misc_cb, NULL);
return 0;
+
+ec_grp_put:
+ roc_ae_ec_grp_put();
+fpm_put:
+ roc_ae_fpm_put();
+ return ret;
}
int
@@ -992,7 +1019,16 @@ cnxk_ae_session_cfg(struct rte_cryptodev *dev, struct rte_crypto_asym_xform *xfo
priv->lf = roc_cpt->lf[0];
w7.u64 = 0;
- w7.s.egrp = roc_cpt->eng_grp[CPT_ENG_TYPE_AE];
+ if (xform->xform_type == RTE_CRYPTO_ASYM_XFORM_ML_KEM) {
+ w7.s.egrp = roc_cpt->eng_grp[CPT_ENG_TYPE_RE];
+ w7.s.cptr = rte_cpu_to_be_64(vf->cnxk_ml_iova[ROC_RE_ML_ZETA_IDX_KEM]);
+ } else if (xform->xform_type == RTE_CRYPTO_ASYM_XFORM_ML_DSA) {
+ w7.s.egrp = roc_cpt->eng_grp[CPT_ENG_TYPE_RE];
+ w7.s.cptr = rte_cpu_to_be_64(vf->cnxk_ml_iova[ROC_RE_ML_ZETA_IDX_DSA]);
+ } else {
+ w7.s.egrp = roc_cpt->eng_grp[CPT_ENG_TYPE_AE];
+ w7.s.cptr = 0;
+ }
if (roc_errata_cpt_hang_on_mixed_ctx_val()) {
hwc = &priv->hw_ctx;
@@ -1007,6 +1043,7 @@ cnxk_ae_session_cfg(struct rte_cryptodev *dev, struct rte_crypto_asym_xform *xfo
priv->cpt_inst_w7 = w7.u64;
priv->cnxk_fpm_iova = vf->cnxk_fpm_iova;
+ priv->cnxk_ml_iova = vf->cnxk_ml_iova;
priv->ec_grp = vf->ec_grp;
return 0;
--
2.37.1
^ permalink raw reply related
* Re: [PATCH 0/2] ip_frag: TAILQ and hash fixes
From: Thomas Monjalon @ 2026-05-29 10:21 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
In-Reply-To: <20260408161947.285185-1-stephen@networkplumber.org>
08/04/2026 18:16, Stephen Hemminger:
> The ip_frag library had a couple of issues found during review.
> The use of TAILQ made assumptions about TAILQ_REMOVE,
> and the hash function usage could be abused by attacker.
>
> Stephen Hemminger (2):
> ip_frag: fix unsafe TAILQ usage
> ip_frag: randomize hash seed
Applied, thanks.
^ permalink raw reply
* Re: [PATCH] net/sxe2: fix 32-bit SSE build
From: Thomas Monjalon @ 2026-05-29 10:24 UTC (permalink / raw)
To: Jie Liu; +Cc: dev, David Marchand
In-Reply-To: <CAJFAV8xY0uM=RbcR64ZMc-eOTVkzZc0PFTCBCaQkNNE7uuvZeg@mail.gmail.com>
28/05/2026 11:47, David Marchand:
> On Thu, 28 May 2026 at 10:47, Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > Seen in OBS on i586 Debian:
> >
> > from ../drivers/net/sxe2/sxe2_txrx_vec_sse.c:5:
> > In function ‘_mm_loadu_si128’,
> > inlined from ‘rte_memcpy’
> > inlined from ‘sxe2_rx_pkts_refactor’
> > at ../drivers/net/sxe2/sxe2_txrx_vec_common.h:233:2:
> > /usr/lib/gcc/i686-linux-gnu/12/include/emmintrin.h:703:10: error:
> > array subscript 8 is outside array bounds of ‘struct rte_mbuf *[32]’
> >
> > The important options to reproduce are "-m32 -O2 -march=corei7".
> >
> > In 32-bit build the pointer array done_pkts[32] is smaller:
> > 32 * 4 = 128 bytes
> > so an SSE access would be outside the bound.
> >
> > The libc memcpy does not trigger such warning
> > and is a good choice to copy an array of pointers.
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
>
> I reproduced and checked Debian 13 32 bits build with OBS.
> Tested-by: David Marchand <david.marchand@redhat.com>
Fixes: ac60f302cbef ("net/sxe2: add vectorized Rx and Tx")
Applied
^ permalink raw reply
* RE: [PATCH v6] mempool: improve cache behaviour and performance
From: Morten Brørup @ 2026-05-29 11:43 UTC (permalink / raw)
To: fengchengwen, dev, Andrew Rybchenko, Bruce Richardson,
Jingjing Wu, Praveen Shetty, Hemant Agrawal, Sachin Saxena
In-Reply-To: <9b627380-0fc4-40b8-b927-c443a7bbba1b@huawei.com>
> From: fengchengwen [mailto:fengchengwen@huawei.com]
> Sent: Friday, 29 May 2026 10.54
>
> On 5/26/2026 10:00 PM, Morten Brørup wrote:
> > This patch refactors the mempool cache to eliminate some unexpected
> > behaviour and reduce the mempool cache miss rate.
> >
> > 1.
> > The actual cache size was 1.5 times the cache size specified at run-
> time
> > mempool creation.
> > This was obviously not expected by application developers.
> >
> > 2.
> > In get operations, the check for when to use the cache as bounce
> buffer
> > did not respect the run-time configured cache size,
> > but compared to the build time maximum possible cache size
> > (RTE_MEMPOOL_CACHE_MAX_SIZE, default 512).
> > E.g. with a configured cache size of 32 objects, getting 256 objects
> > would first fetch 32 + 256 = 288 objects into the cache,
> > and then move the 256 objects from the cache to the destination
> memory,
> > instead of fetching the 256 objects directly to the destination
> memory.
> > This had a performance cost.
> > However, this is unlikely to occur in real applications, so it is not
> > important in itself.
> >
> > 3.
> > When putting objects into a mempool, and the mempool cache did not
> have
> > free space for so many objects,
> > the cache was flushed completely, and the new objects were then put
> into
> > the cache.
> > I.e. the cache drain level was zero.
> > This (complete cache flush) meant that a subsequent get operation
> (with
> > the same number of objects) completely emptied the cache,
> > so another subsequent get operation required replenishing the cache.
> >
> > Similarly,
> > When getting objects from a mempool, and the mempool cache did not
> hold so
> > many objects,
> > the cache was replenished to cache->size + remaining objects,
> > and then (the remaining part of) the requested objects were fetched
> via
> > the cache,
> > which left the cache filled (to cache->size) at completion.
> > I.e. the cache refill level was cache->size (plus some, depending on
> > request size).
> >
> > (1) was improved by generally comparing to cache->size instead of
> > cache->flushthresh, when considering the capacity of the cache.
> > The cache->flushthresh field is kept for API/ABI compatibility
> purposes,
> > and initialized to cache->size instead of cache->size * 1.5.
> >
> > (2) was improved by generally comparing to cache->size / 2 instead of
> > RTE_MEMPOOL_CACHE_MAX_SIZE, when checking the bounce buffer limit.
> >
> > (3) was improved by flushing and replenishing the cache by half its
> size,
> > so a flush/refill can be followed randomly by get or put requests.
> > This also reduced the number of objects in each flush/refill
> operation.
> >
> > As a consequence of these changes, the size of the array holding the
> > objects in the cache (cache->objs[]) no longer needs to be
> > 2 * RTE_MEMPOOL_CACHE_MAX_SIZE, and can be reduced to
> > RTE_MEMPOOL_CACHE_MAX_SIZE at an API/ABI breaking release.
> >
> > Performance data:
> > With a real WAN Optimization application, where the number of
> allocated
> > packets varies (as they are held in e.g. shaper queues), the mempool
> > cache miss rate dropped from ca. 1/20 objects to ca. 1/48 objects.
> > This was deployed in production at an ISP, and using an effective
> cache
> > size of 384 objects.
>
> Does the application run as a RTC (run-to-complete) mode?
Yes, the application runs as RTC mode.
> How about pipeline model which NIC recv packets and enqueue ring,
> another
> work thread dequeue packets, process packets and then free packets
> mbuf?
>
If one thread only receives packets (mempool get) and another thread only transmits/frees (mempool put), their cache miss rate roughly doubles.
But the number of objects copied to/from the backend per cache miss roughly halves to exactly size/2.
And the backend copy operations become CPU cache aligned (assuming all transactions with the backend go via the mempool cache).
The release notes mention that such pipelined applications should double their configured mempool cache size.
^ permalink raw reply
* Re: [PATCH v4 00/25] Consolidate bus driver infrastructure
From: David Marchand @ 2026-05-29 11:53 UTC (permalink / raw)
To: fengchengwen; +Cc: dev, thomas, stephen, bruce.richardson
In-Reply-To: <8fe9d77c-9ade-4cbe-ac99-582f95531cb4@huawei.com>
Hello Chengwen,
On Fri, 29 May 2026 at 09:47, fengchengwen <fengchengwen@huawei.com> wrote:
>
> Hi David,
>
> First, this is a great refactoring, thank you very much.
>
> I have tested this patchset on the UACCE bus and it passes. Two issues were
> found during testing and review:
>
> 1. Patch 25/25 fails to apply to the current main branch.
Yes, I am aware: next-net-intel got pulled in main after my rebase for v4.
> 2. Patch 15/25 (bus: support multiple probing) has issues with directly
> enabling multiple probing. I suggest merging this patch into Patch 17/25.
I am not sure I understand the issue, but I will have a look asap and post a v5.
> The UACCE bus part has passed testing. Please add my Acked-by tag to the
> related commits.
>
Thanks a lot for your reviews and testing!
--
David Marchand
^ permalink raw reply
* [PATCH v4 0/1] net/mana: add device reset support
From: Wei Hu @ 2026-05-29 12:48 UTC (permalink / raw)
To: dev, stephen; +Cc: longli, weh
From: Wei Hu <weh@microsoft.com>
Add support for handling hardware service reset events in the
MANA driver. When the MANA kernel driver receives a hardware
service event, it initiates a device reset and notifies userspace
via IBV_EVENT_DEVICE_FATAL. The MANA PMD handles this by
performing an automatic teardown and recovery sequence.
The driver uses ethdev recovery events (ERR_RECOVERING,
RECOVERY_SUCCESS, RECOVERY_FAILED) to notify upper layers of
the reset lifecycle, and a PCI device removal event callback
to distinguish hot-remove from service reset.
Changes since v3:
- Converted reset_ops_lock from rte_spinlock_t to pthread_mutex_t
with PTHREAD_PROCESS_SHARED, since the lock is held across
blocking IB verbs calls and IPC with 5s timeout
- Removed rte_dev_event_callback_unregister retry loop to avoid
deadlock: the callback itself blocks on reset_ops_lock, so
retrying on -EAGAIN while holding the lock is a deadlock
- Introduced mana_join_reset_thread() helper using CAS on
reset_thread_active to prevent double-join undefined behavior
- Added reset thread join in mana_dev_uninit to prevent thread
leak on device removal
- Fixed ibv handle leak: priv->ib_ctx is now only set to NULL
after ibv_close_device succeeds
- Fixed misleading "All secondary threads are quiescent" log in
mana_mp_reset_enter — changed to "Secondary doorbell pages
unmapped" since actual quiescence is enforced by the primary's
RCU QSBR check before IPC is sent
- Changed event list in mana.rst to RST definition list style
- Squashed documentation into the feature patch per convention
Changes since v2:
- Fixed dev_state_qsv memory leak on device removal
- Fixed reset thread TCB/stack leak: reset_thread_active is now
only cleared by the joiner, not the thread itself
- Fixed second reset crash: removed reset thread join logic from
mana_dev_close (inner function) to avoid corrupting dev_state
when called from mana_reset_enter
- Made reset_thread_active RTE_ATOMIC(bool) with explicit ordering
- Added retry loop for rte_dev_event_callback_unregister on -EAGAIN
- Initialized condvar/mutex with PTHREAD_PROCESS_SHARED since priv
is in hugepage shared memory
- Added re-check of dev_state after lock acquisition in
mana_intr_handler to prevent racing with pci_remove_event_cb
- Replaced (void *)0 with NULL in mp.c
- Added lock ownership comment block at mana_reset_enter
- Documented rte_dev_event_monitor_start() requirement
- Added mana.rst documentation and release note
Changes since v1:
- Removed net/netvsc patch from this series
- Simplified reset exit: mana_reset_exit calls
mana_reset_exit_delay directly instead of spawning a thread
- Added __rte_no_thread_safety_analysis annotations for clang
- Switched to rte_thread_create_internal_control
- Fixed declaration-after-statement style issues
- Removed unnecessary blank lines and stale comments
Wei Hu (1):
net/mana: add device reset support
doc/guides/nics/mana.rst | 38 +
doc/guides/rel_notes/release_26_07.rst | 8 +
drivers/net/mana/mana.c | 1005 ++++++++++++++++++++++--
drivers/net/mana/mana.h | 33 +-
drivers/net/mana/meson.build | 2 +-
drivers/net/mana/mp.c | 89 ++-
drivers/net/mana/mr.c | 6 +-
drivers/net/mana/rx.c | 24 +-
drivers/net/mana/tx.c | 40 +-
9 files changed, 1138 insertions(+), 107 deletions(-)
--
2.34.1
^ permalink raw reply
* [PATCH v4 1/1] net/mana: add device reset support
From: Wei Hu @ 2026-05-29 12:49 UTC (permalink / raw)
To: dev, stephen; +Cc: longli, weh
In-Reply-To: <20260529124900.146698-1-weh@linux.microsoft.com>
From: Wei Hu <weh@microsoft.com>
Add support for handling hardware reset events in the MANA driver.
When the MANA kernel driver receives a hardware service event, it
initiates a device reset and notifies userspace via
IBV_EVENT_DEVICE_FATAL. The DPDK driver handles this by performing
an automatic teardown and recovery sequence.
The reset flow has two phases. In the enter phase, running on the
EAL interrupt thread, the driver transitions the device state,
waits for data path threads to reach a quiescent state using RCU,
stops queues, tears down IB resources, and frees per-queue MR
caches. A control thread is then spawned to handle the exit phase:
it waits for the hardware to recover, unregisters the interrupt
handler, re-probes the PCI device, reinitializes MR caches, and
restarts queues.
A per-device mutex serializes the reset path with ethdev
operations. The mutex uses PTHREAD_PROCESS_SHARED for multi-process
support and is held across blocking IB verbs calls. Operations that
cannot wait (configure, queue setup) return -EBUSY during reset,
while dev_stop and dev_close join the reset thread before acquiring
the lock to ensure proper sequencing. A CAS-based helper prevents
double-join of the reset thread.
Multi-process support is included: secondary processes unmap and
remap doorbell pages via IPC during the reset enter and exit
phases. Data path functions in both primary and secondary
processes check the device state atomically and return early when
the device is not active. RCU quiescent state tracking uses
per-queue thread IDs in shared hugepage memory, covering both
primary and secondary process data path threads.
The driver uses ethdev recovery events to notify upper layers
(e.g. netvsc) of the reset lifecycle: RTE_ETH_EVENT_ERR_RECOVERING
on entry, RTE_ETH_EVENT_RECOVERY_SUCCESS or
RTE_ETH_EVENT_RECOVERY_FAILED on completion. A PCI device removal
event callback distinguishes hot-remove from service reset.
Documentation for the device reset feature is added in the MANA
NIC guide and the 26.07 release notes.
Signed-off-by: Wei Hu <weh@microsoft.com>
---
doc/guides/nics/mana.rst | 38 +
doc/guides/rel_notes/release_26_07.rst | 8 +
drivers/net/mana/mana.c | 1005 ++++++++++++++++++++++--
drivers/net/mana/mana.h | 33 +-
drivers/net/mana/meson.build | 2 +-
drivers/net/mana/mp.c | 89 ++-
drivers/net/mana/mr.c | 6 +-
drivers/net/mana/rx.c | 24 +-
drivers/net/mana/tx.c | 40 +-
9 files changed, 1138 insertions(+), 107 deletions(-)
diff --git a/doc/guides/nics/mana.rst b/doc/guides/nics/mana.rst
index 0fcab6e2f6..2b6bed4928 100644
--- a/doc/guides/nics/mana.rst
+++ b/doc/guides/nics/mana.rst
@@ -71,3 +71,41 @@ The user can specify below argument in devargs.
The default value is not set,
meaning all the NICs will be probed and loaded.
User can specify multiple mac=xx:xx:xx:xx:xx:xx arguments for up to 8 NICs.
+
+Device Reset Support
+--------------------
+
+The MANA PMD supports automatic recovery from hardware service reset events.
+When the MANA kernel driver receives a hardware service event,
+it initiates a device reset and notifies userspace
+via ``IBV_EVENT_DEVICE_FATAL``.
+
+The driver handles this transparently through a two-phase reset flow:
+
+* **Enter phase**: The driver stops the data path,
+ waits for all threads to reach a quiescent state using RCU,
+ tears down IB resources and queues,
+ and unmaps secondary process doorbell pages.
+
+* **Exit phase**: After a delay for hardware recovery,
+ a control thread re-probes the PCI device,
+ reinstalls the interrupt handler,
+ reinitializes resources, and restarts queues.
+
+The driver emits the following ethdev recovery events
+to notify upper layers (e.g. netvsc) of the reset lifecycle:
+
+``RTE_ETH_EVENT_ERR_RECOVERING``
+ Reset has started.
+
+``RTE_ETH_EVENT_RECOVERY_SUCCESS``
+ Device has recovered successfully.
+
+``RTE_ETH_EVENT_RECOVERY_FAILED``
+ Recovery failed.
+
+To distinguish a PCI hot-remove from a service reset,
+the driver registers for PCI device removal events.
+This requires the application to call ``rte_dev_event_monitor_start()``
+for removal events to be delivered
+(e.g. testpmd ``--hot-plug-handling`` option).
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 92c90673bc..114bc09c5d 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -77,6 +77,14 @@ New Features
Added network driver for the Linkdata Network Adapters.
+* **Added device reset support to the MANA PMD.**
+
+ Added automatic recovery from hardware service reset events
+ in the MANA poll mode driver. The driver uses ethdev recovery events
+ (``RTE_ETH_EVENT_ERR_RECOVERING``, ``RTE_ETH_EVENT_RECOVERY_SUCCESS``,
+ ``RTE_ETH_EVENT_RECOVERY_FAILED``) to notify upper layers of the
+ reset lifecycle.
+
Removed Items
-------------
diff --git a/drivers/net/mana/mana.c b/drivers/net/mana/mana.c
index 67396cda1f..cc549cb4eb 100644
--- a/drivers/net/mana/mana.c
+++ b/drivers/net/mana/mana.c
@@ -13,7 +13,10 @@
#include <ethdev_pci.h>
#include <rte_kvargs.h>
#include <rte_eal_paging.h>
+#include <rte_alarm.h>
#include <rte_pci.h>
+#include <rte_rcu_qsbr.h>
+#include <rte_lock_annotations.h>
#include <infiniband/verbs.h>
#include <infiniband/manadv.h>
@@ -103,6 +106,23 @@ mana_dev_configure(struct rte_eth_dev *dev)
RTE_ETH_RX_OFFLOAD_VLAN_STRIP);
priv->num_queues = dev->data->nb_rx_queues;
+ DRV_LOG(DEBUG, "priv %p, port %u, dev port %u, num_queues: %u",
+ priv, priv->port_id, priv->dev_port, priv->num_queues);
+
+ /*
+ * Register data path thread IDs (rx and tx) with the RCU
+ * quiescent state variable for device state synchronization.
+ */
+ for (int i = 0; i < 2 * priv->num_queues; i++) {
+ if (rte_rcu_qsbr_thread_register(priv->dev_state_qsv, i) != 0) {
+ DRV_LOG(ERR, "Failed to register rcu qsv thread %d of total %d",
+ i, 2 * priv->num_queues - 1);
+ return -EINVAL;
+ }
+ DRV_LOG(DEBUG,
+ "Register thread 0x%x for priv %p, port %u",
+ i, priv, priv->port_id);
+ }
manadv_set_context_attr(priv->ib_ctx, MANADV_CTX_ATTR_BUF_ALLOCATORS,
(void *)((uintptr_t)&(struct manadv_ctx_allocators){
@@ -214,8 +234,8 @@ mana_dev_start(struct rte_eth_dev *dev)
DRV_LOG(INFO, "TX/RX queues have started");
- /* Enable datapath for secondary processes */
- mana_mp_req_on_rxtx(dev, MANA_MP_REQ_START_RXTX);
+ /* Intentionally ignore errors — secondary may not be running */
+ (void)mana_mp_req_on_rxtx(dev, MANA_MP_REQ_START_RXTX);
ret = rxq_intr_enable(priv);
if (ret) {
@@ -242,26 +262,33 @@ mana_dev_stop(struct rte_eth_dev *dev)
{
int ret;
struct mana_priv *priv = dev->data->dev_private;
-
- rxq_intr_disable(priv);
+ enum mana_device_state state;
+
+ state = rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire);
+ if (state == MANA_DEV_ACTIVE ||
+ state == MANA_DEV_RESET_FAILED) {
+ rxq_intr_disable(priv);
+ DRV_LOG(DEBUG, "rxq_intr_disable called");
+ }
dev->tx_pkt_burst = mana_tx_burst_removed;
dev->rx_pkt_burst = mana_rx_burst_removed;
- /* Stop datapath on secondary processes */
- mana_mp_req_on_rxtx(dev, MANA_MP_REQ_STOP_RXTX);
+ /* Intentionally ignore errors — secondary may not be running */
+ (void)mana_mp_req_on_rxtx(dev, MANA_MP_REQ_STOP_RXTX);
rte_wmb();
ret = mana_stop_tx_queues(dev);
if (ret) {
- DRV_LOG(ERR, "failed to stop tx queues");
+ DRV_LOG(ERR, "failed to stop tx queues, ret %d", ret);
return ret;
}
ret = mana_stop_rx_queues(dev);
if (ret) {
- DRV_LOG(ERR, "failed to stop tx queues");
+ DRV_LOG(ERR, "failed to stop rx queues, ret %d", ret);
return ret;
}
@@ -275,36 +302,70 @@ mana_dev_close(struct rte_eth_dev *dev)
{
struct mana_priv *priv = dev->data->dev_private;
int ret;
+ enum mana_device_state state;
+ DRV_LOG(DEBUG, "Free MR for priv %p", priv);
mana_remove_all_mr(priv);
- ret = mana_intr_uninstall(priv);
- if (ret)
- return ret;
+ state = rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire);
+ if (state == MANA_DEV_ACTIVE ||
+ state == MANA_DEV_RESET_FAILED) {
+ ret = mana_intr_uninstall(priv);
+ if (ret)
+ return ret;
+ }
if (priv->ib_parent_pd) {
- int err = ibv_dealloc_pd(priv->ib_parent_pd);
- if (err)
- DRV_LOG(ERR, "Failed to deallocate parent PD: %d", err);
+ ret = ibv_dealloc_pd(priv->ib_parent_pd);
+ if (ret)
+ DRV_LOG(ERR,
+ "Failed to deallocate parent PD: %d", ret);
priv->ib_parent_pd = NULL;
}
if (priv->ib_pd) {
- int err = ibv_dealloc_pd(priv->ib_pd);
- if (err)
- DRV_LOG(ERR, "Failed to deallocate PD: %d", err);
+ ret = ibv_dealloc_pd(priv->ib_pd);
+ if (ret)
+ DRV_LOG(ERR, "Failed to deallocate PD: %d", ret);
priv->ib_pd = NULL;
}
- ret = ibv_close_device(priv->ib_ctx);
- if (ret) {
- ret = errno;
- return ret;
+ state = rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire);
+ if (state == MANA_DEV_ACTIVE ||
+ state == MANA_DEV_RESET_FAILED) {
+ if (priv->ib_ctx) {
+ ret = ibv_close_device(priv->ib_ctx);
+ if (ret) {
+ ret = errno;
+ return ret;
+ }
+ priv->ib_ctx = NULL;
+ }
}
return 0;
}
+/*
+ * Called from mana_pci_remove to free resources allocated
+ * during probe that are not freed by dev_close.
+ */
+static void
+mana_dev_free_resources(struct rte_eth_dev *dev)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+
+ if (priv->dev_state_qsv) {
+ rte_free(priv->dev_state_qsv);
+ priv->dev_state_qsv = NULL;
+ }
+ pthread_mutex_destroy(&priv->reset_ops_lock);
+ pthread_mutex_destroy(&priv->reset_cond_mutex);
+ pthread_cond_destroy(&priv->reset_cond);
+}
+
static int
mana_dev_info_get(struct rte_eth_dev *dev,
struct rte_eth_dev_info *dev_info)
@@ -391,6 +452,27 @@ mana_dev_info_get(struct rte_eth_dev *dev,
return 0;
}
+static int
+mana_dev_info_get_lock(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ if (!pthread_mutex_trylock(&priv->reset_ops_lock)) {
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return -EBUSY;
+ }
+ ret = mana_dev_info_get(dev, dev_info);
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ } else {
+ ret = -EBUSY;
+ }
+ return ret;
+}
+
static void
mana_dev_tx_queue_info(struct rte_eth_dev *dev, uint16_t queue_id,
struct rte_eth_txq_info *qinfo)
@@ -552,6 +634,29 @@ mana_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,
return ret;
}
+static int
+mana_dev_tx_queue_setup_lock(struct rte_eth_dev *dev, uint16_t queue_idx,
+ uint16_t nb_desc, unsigned int socket_id,
+ const struct rte_eth_txconf *tx_conf)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ if (!pthread_mutex_trylock(&priv->reset_ops_lock)) {
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return -EBUSY;
+ }
+ ret = mana_dev_tx_queue_setup(dev, queue_idx,
+ nb_desc, socket_id, tx_conf);
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ } else {
+ ret = -EBUSY;
+ }
+ return ret;
+}
+
static void
mana_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
{
@@ -629,6 +734,30 @@ mana_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,
return ret;
}
+static int
+mana_dev_rx_queue_setup_lock(struct rte_eth_dev *dev, uint16_t queue_idx,
+ uint16_t nb_desc, unsigned int socket_id,
+ const struct rte_eth_rxconf *rx_conf __rte_unused,
+ struct rte_mempool *mp)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ if (!pthread_mutex_trylock(&priv->reset_ops_lock)) {
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return -EBUSY;
+ }
+ ret = mana_dev_rx_queue_setup(dev, queue_idx, nb_desc,
+ socket_id, rx_conf, mp);
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ } else {
+ ret = -EBUSY;
+ }
+ return ret;
+}
+
static void
mana_dev_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
{
@@ -820,33 +949,215 @@ mana_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
return mana_ifreq(priv, SIOCSIFMTU, &request);
}
+#define MANA_OPS_1_LOCK(_func) \
+static int \
+_func##_lock(struct rte_eth_dev *dev) \
+{ \
+ struct mana_priv *priv = dev->data->dev_private; \
+ int ret; \
+ if (!pthread_mutex_trylock(&priv->reset_ops_lock)) { \
+ if (rte_atomic_load_explicit(&priv->dev_state, \
+ rte_memory_order_acquire) != \
+ MANA_DEV_ACTIVE) { \
+ pthread_mutex_unlock(&priv->reset_ops_lock); \
+ return -EBUSY; \
+ } \
+ ret = _func(dev); \
+ pthread_mutex_unlock(&priv->reset_ops_lock); \
+ } else { \
+ ret = -EBUSY; \
+ } \
+ return ret; \
+}
+
+MANA_OPS_1_LOCK(mana_dev_configure)
+
+MANA_OPS_1_LOCK(mana_dev_start)
+
+#undef MANA_OPS_1_LOCK
+
+/*
+ * Join the reset thread if it is active. Uses CAS on
+ * reset_thread_active to ensure only one caller joins.
+ */
+static void
+mana_join_reset_thread(struct mana_priv *priv)
+{
+ bool expected = true;
+
+ if (rte_atomic_compare_exchange_strong_explicit(
+ &priv->reset_thread_active, &expected, false,
+ rte_memory_order_acq_rel,
+ rte_memory_order_acquire)) {
+ pthread_mutex_lock(&priv->reset_cond_mutex);
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_ACTIVE, rte_memory_order_release);
+ pthread_cond_signal(&priv->reset_cond);
+ pthread_mutex_unlock(&priv->reset_cond_mutex);
+ rte_thread_join(priv->reset_thread, NULL);
+ }
+}
+
+/*
+ * Custom lock wrappers for dev_stop and dev_close.
+ * These join any active reset thread and use a blocking lock (not
+ * trylock) so they wait for any in-progress reset processing to
+ * finish, rather than returning -EBUSY. When the device is not in
+ * MANA_DEV_ACTIVE state, they transition state to MANA_DEV_ACTIVE.
+ */
+static int
+mana_dev_stop_lock(struct rte_eth_dev *dev)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ mana_join_reset_thread(priv);
+
+ pthread_mutex_lock(&priv->reset_ops_lock);
+
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_ACTIVE, rte_memory_order_release);
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return 0;
+ }
+
+ ret = mana_dev_stop(dev);
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return ret;
+}
+
+static int
+mana_dev_close_lock(struct rte_eth_dev *dev)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ mana_join_reset_thread(priv);
+
+ pthread_mutex_lock(&priv->reset_ops_lock);
+
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_ACTIVE, rte_memory_order_release);
+ }
+
+ ret = mana_dev_close(dev);
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return ret;
+}
+
+#define MANA_OPS_2_LOCK(_func) \
+static int \
+_func##_lock(struct rte_eth_dev *dev, \
+ struct rte_eth_rss_conf *rss_conf) \
+{ \
+ struct mana_priv *priv = dev->data->dev_private; \
+ int ret; \
+ if (!pthread_mutex_trylock(&priv->reset_ops_lock)) { \
+ if (rte_atomic_load_explicit(&priv->dev_state, \
+ rte_memory_order_acquire) != \
+ MANA_DEV_ACTIVE) { \
+ pthread_mutex_unlock(&priv->reset_ops_lock); \
+ return -EBUSY; \
+ } \
+ ret = _func(dev, rss_conf); \
+ pthread_mutex_unlock(&priv->reset_ops_lock); \
+ } else { \
+ ret = -EBUSY; \
+ } \
+ return ret; \
+}
+
+MANA_OPS_2_LOCK(mana_rss_hash_update)
+
+MANA_OPS_2_LOCK(mana_rss_hash_conf_get)
+#undef MANA_OPS_2_LOCK
+
+#define MANA_OPS_3_LOCK(_func, _arg) \
+static void \
+_func##_lock(struct rte_eth_dev *dev, uint16_t _arg) \
+{ \
+ struct mana_priv *priv = dev->data->dev_private; \
+ if (!pthread_mutex_trylock(&priv->reset_ops_lock)) { \
+ if (rte_atomic_load_explicit(&priv->dev_state, \
+ rte_memory_order_acquire) != \
+ MANA_DEV_ACTIVE) { \
+ pthread_mutex_unlock(&priv->reset_ops_lock); \
+ DRV_LOG(ERR, "Device reset in progress, " \
+ "%s not called", #_func); \
+ return; \
+ } \
+ _func(dev, _arg); \
+ pthread_mutex_unlock(&priv->reset_ops_lock); \
+ } else { \
+ DRV_LOG(ERR, "Device reset in progress, " \
+ "%s not called", #_func); \
+ } \
+}
+
+MANA_OPS_3_LOCK(mana_dev_tx_queue_release, qid)
+
+MANA_OPS_3_LOCK(mana_dev_rx_queue_release, qid)
+#undef MANA_OPS_3_LOCK
+
+#define MANA_OPS_4_LOCK(_func, _arg) \
+static int \
+_func##_lock(struct rte_eth_dev *dev, uint16_t _arg) \
+{ \
+ struct mana_priv *priv = dev->data->dev_private; \
+ int ret; \
+ if (!pthread_mutex_trylock(&priv->reset_ops_lock)) { \
+ if (rte_atomic_load_explicit(&priv->dev_state, \
+ rte_memory_order_acquire) != \
+ MANA_DEV_ACTIVE) { \
+ pthread_mutex_unlock(&priv->reset_ops_lock); \
+ return -EBUSY; \
+ } \
+ ret = _func(dev, _arg); \
+ pthread_mutex_unlock(&priv->reset_ops_lock); \
+ } else { \
+ ret = -EBUSY; \
+ } \
+ return ret; \
+}
+
+MANA_OPS_4_LOCK(mana_rx_intr_enable, rx_queue_id)
+
+MANA_OPS_4_LOCK(mana_rx_intr_disable, rx_queue_id)
+
+MANA_OPS_4_LOCK(mana_mtu_set, mtu)
+#undef MANA_OPS_4_LOCK
+
static const struct eth_dev_ops mana_dev_ops = {
- .dev_configure = mana_dev_configure,
- .dev_start = mana_dev_start,
- .dev_stop = mana_dev_stop,
- .dev_close = mana_dev_close,
- .dev_infos_get = mana_dev_info_get,
+ .dev_configure = mana_dev_configure_lock,
+ .dev_start = mana_dev_start_lock,
+ .dev_stop = mana_dev_stop_lock,
+ .dev_close = mana_dev_close_lock,
+ .dev_infos_get = mana_dev_info_get_lock,
.txq_info_get = mana_dev_tx_queue_info,
.rxq_info_get = mana_dev_rx_queue_info,
.dev_supported_ptypes_get = mana_supported_ptypes,
- .rss_hash_update = mana_rss_hash_update,
- .rss_hash_conf_get = mana_rss_hash_conf_get,
- .tx_queue_setup = mana_dev_tx_queue_setup,
- .tx_queue_release = mana_dev_tx_queue_release,
- .rx_queue_setup = mana_dev_rx_queue_setup,
- .rx_queue_release = mana_dev_rx_queue_release,
- .rx_queue_intr_enable = mana_rx_intr_enable,
- .rx_queue_intr_disable = mana_rx_intr_disable,
+ .rss_hash_update = mana_rss_hash_update_lock,
+ .rss_hash_conf_get = mana_rss_hash_conf_get_lock,
+ .tx_queue_setup = mana_dev_tx_queue_setup_lock,
+ .tx_queue_release = mana_dev_tx_queue_release_lock,
+ .rx_queue_setup = mana_dev_rx_queue_setup_lock,
+ .rx_queue_release = mana_dev_rx_queue_release_lock,
+ .rx_queue_intr_enable = mana_rx_intr_enable_lock,
+ .rx_queue_intr_disable = mana_rx_intr_disable_lock,
.link_update = mana_dev_link_update,
.stats_get = mana_dev_stats_get,
.stats_reset = mana_dev_stats_reset,
- .mtu_set = mana_mtu_set,
+ .mtu_set = mana_mtu_set_lock,
};
static const struct eth_dev_ops mana_dev_secondary_ops = {
.stats_get = mana_dev_stats_get,
.stats_reset = mana_dev_stats_reset,
- .dev_infos_get = mana_dev_info_get,
+ .dev_infos_get = mana_dev_info_get_lock,
};
uint16_t
@@ -1031,28 +1342,441 @@ mana_ibv_device_to_pci_addr(const struct ibv_device *device,
return 0;
}
+static int mana_pci_probe(struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev);
+static void mana_intr_handler(void *arg);
+static void mana_reset_exit(struct mana_priv *priv);
+
+/* Delay before initiating reset exit after reset enter completes */
+#define MANA_RESET_TIMER_US (15 * 1000000ULL) /* 15 seconds */
+
/*
- * Interrupt handler from IB layer to notify this device is being removed.
+ * Callback for PCI device removal events from EAL.
+ * If the device is in reset (RESET_EXIT state), this means the PCI
+ * device was hot-removed rather than a service reset. Wake the reset
+ * thread via condvar and notify netvsc via RTE_ETH_EVENT_INTR_RMV.
+ */
+static void
+mana_pci_remove_event_cb(const char *device_name,
+ enum rte_dev_event_type event, void *cb_arg)
+ __rte_no_thread_safety_analysis
+{
+ struct mana_priv *priv = cb_arg;
+ struct rte_eth_dev *dev;
+
+ if (event != RTE_DEV_EVENT_REMOVE)
+ return;
+
+ DRV_LOG(INFO, "PCI device %s removed", device_name);
+
+ /* Wake the reset thread immediately */
+ pthread_mutex_lock(&priv->reset_cond_mutex);
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_RESET_FAILED, rte_memory_order_release);
+ pthread_cond_signal(&priv->reset_cond);
+ pthread_mutex_unlock(&priv->reset_cond_mutex);
+
+ pthread_mutex_lock(&priv->reset_ops_lock);
+
+ dev = &rte_eth_devices[priv->port_id];
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_INTR_RMV for port %u",
+ priv->port_id);
+ rte_eth_dev_callback_process(dev,
+ RTE_ETH_EVENT_INTR_RMV, NULL);
+
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+}
+
+/*
+ * Reset thread: sleeps for the reset timer period, then performs
+ * the reset exit sequence. Runs on a control thread so it can call
+ * rte_intr_callback_unregister (which fails from alarm/intr thread).
+ */
+static uint32_t
+mana_reset_thread(void *arg)
+ __rte_no_thread_safety_analysis
+{
+ struct mana_priv *priv = (struct mana_priv *)arg;
+ struct timespec ts;
+
+ DRV_LOG(INFO, "Reset thread started, waiting %us",
+ (unsigned int)(MANA_RESET_TIMER_US / 1000000));
+
+ /* Wait on condvar with timeout — can be woken early by PCI remove */
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_sec += MANA_RESET_TIMER_US / 1000000;
+
+ pthread_mutex_lock(&priv->reset_cond_mutex);
+ pthread_cond_timedwait(&priv->reset_cond, &priv->reset_cond_mutex, &ts);
+ pthread_mutex_unlock(&priv->reset_cond_mutex);
+
+ pthread_mutex_lock(&priv->reset_ops_lock);
+
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_RESET_EXIT) {
+ DRV_LOG(INFO, "Reset thread: dev_state=%d, skipping",
+ (int)rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire));
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return 0;
+ }
+
+ DRV_LOG(INFO, "Reset thread: initiating reset exit");
+ mana_reset_exit(priv);
+ /* Lock is released by mana_reset_exit_delay at the end of
+ * the reset exit processing.
+ *
+ * reset_thread_active is NOT cleared here — the joiner
+ * (dev_stop_lock/dev_close_lock) is responsible for joining
+ * and clearing the flag to avoid leaking the thread.
+ */
+ return 0;
+}
+
+static void
+mana_reset_enter(struct mana_priv *priv)
+ __rte_no_thread_safety_analysis
+{
+ int ret;
+ uint64_t ticket;
+ struct rte_eth_dev *dev = &rte_eth_devices[priv->port_id];
+
+ /*
+ * Lock ownership for reset_ops_lock through the reset path:
+ *
+ * mana_intr_handler — acquires the lock
+ * mana_reset_enter — called with lock held, releases it
+ * after spawning the reset thread
+ * mana_reset_thread — re-acquires the lock after the
+ * recovery delay
+ * mana_reset_exit — called with lock held, passes it
+ * to mana_reset_exit_delay
+ * mana_reset_exit_delay — called with lock held, releases it
+ * on completion
+ */
+
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_ENTER,
+ rte_memory_order_release);
+
+ DRV_LOG(DEBUG, "Entering into device reset state");
+ DRV_LOG(DEBUG, "Resetting dev = %p, priv = %p", dev, priv);
+
+ ticket = rte_rcu_qsbr_start(priv->dev_state_qsv);
+
+ while (rte_rcu_qsbr_check(priv->dev_state_qsv, ticket, false) == 0)
+ rte_pause();
+
+ DRV_LOG(DEBUG, "All threads are quiescent");
+
+ /* Stop data path on primary and secondary before unmapping doorbell */
+ ret = mana_dev_stop(dev);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to stop mana dev ret %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto reset_failed;
+ }
+
+ /* Unmap secondary doorbell pages after data path is stopped */
+ ret = mana_mp_req_on_rxtx(dev, MANA_MP_REQ_RESET_ENTER);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to reset secondary processes ret = %d",
+ ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto reset_failed;
+ }
+
+ ret = mana_dev_close(dev);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to close mana dev ret %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto reset_failed;
+ }
+
+ for (int i = 0; i < priv->num_queues; i++) {
+ struct mana_rxq *rxq = dev->data->rx_queues[i];
+ struct mana_txq *txq = dev->data->tx_queues[i];
+
+ DRV_LOG(DEBUG, "Free MR for priv = %p, rxq %u, txq %u",
+ priv, rxq->rxq_idx, txq->txq_idx);
+ mana_mr_btree_free(&rxq->mr_btree);
+ mana_mr_btree_free(&txq->mr_btree);
+ }
+
+ DRV_LOG(DEBUG, "Reset processing exited successfully");
+
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_EXIT,
+ rte_memory_order_release);
+
+ /* Join previous reset thread if it completed but was not joined.
+ * Use CAS to avoid double-join if another path joined first.
+ * Don't use mana_join_reset_thread() here — we are already in
+ * RESET_ENTER state and must not change dev_state to ACTIVE.
+ */
+ {
+ bool expected = true;
+
+ if (rte_atomic_compare_exchange_strong_explicit(
+ &priv->reset_thread_active, &expected, false,
+ rte_memory_order_acq_rel,
+ rte_memory_order_acquire))
+ rte_thread_join(priv->reset_thread, NULL);
+ }
+
+ ret = rte_thread_create_internal_control(&priv->reset_thread,
+ "mana-reset",
+ mana_reset_thread, priv);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to create reset thread ret %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return;
+ }
+ rte_atomic_store_explicit(&priv->reset_thread_active,
+ true, rte_memory_order_release);
+
+ DRV_LOG(DEBUG, "Reset thread started");
+
+ /* Release the lock so the application can call dev_stop/dev_close */
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return;
+
+reset_failed:
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+}
+
+static uint32_t
+mana_reset_exit_delay(void *arg)
+ __rte_no_thread_safety_analysis
+{
+ struct mana_priv *priv = (struct mana_priv *)arg;
+ uint32_t ret = 0;
+ int i;
+ struct rte_eth_dev *dev;
+ struct rte_pci_device *pci_dev;
+
+ DRV_LOG(DEBUG, "Delayed mana device reset complete processing");
+
+ /* If the app called dev_stop/dev_close during the timer window,
+ * state is no longer RESET_EXIT. Nothing to do.
+ */
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_RESET_EXIT) {
+ DRV_LOG(DEBUG, "State is not RESET_EXIT, skipping");
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return ret;
+ }
+
+ dev = &rte_eth_devices[priv->port_id];
+ pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+
+ DRV_LOG(DEBUG, "Resetting dev = %p, priv = %p", dev, priv);
+
+ ret = ibv_close_device(priv->ib_ctx);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to close ibv device %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto out;
+ }
+ priv->ib_ctx = NULL;
+
+ ret = mana_pci_probe(NULL, pci_dev);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to probe mana pci dev ret %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto out;
+ }
+
+ /*
+ * Init the local MR caches.
+ */
+ for (i = 0; i < priv->num_queues; i++) {
+ struct mana_rxq *rxq = dev->data->rx_queues[i];
+ struct mana_txq *txq = dev->data->tx_queues[i];
+
+ ret = mana_mr_btree_init(&rxq->mr_btree,
+ MANA_MR_BTREE_PER_QUEUE_N,
+ rxq->socket);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to init RXQ %d MR btree "
+ "on socket %u, ret %d", i, rxq->socket, ret);
+ goto mr_init_failed_rxq;
+ }
+
+ ret = mana_mr_btree_init(&txq->mr_btree,
+ MANA_MR_BTREE_PER_QUEUE_N,
+ txq->socket);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to init TXQ %d MR btree "
+ "on socket %u, ret %d", i, txq->socket, ret);
+ goto mr_init_failed_txq;
+ }
+ }
+ DRV_LOG(DEBUG, "priv %p, num_queues %u", priv, priv->num_queues);
+
+ /* Start secondaries */
+ ret = mana_mp_req_on_rxtx(dev, MANA_MP_REQ_RESET_EXIT);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to start secondary processes ret = %d",
+ ret);
+ goto mr_init_failed_all;
+ }
+
+ ret = mana_dev_start(dev);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to start mana dev ret %d", ret);
+ goto mr_init_failed_all;
+ }
+
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_ACTIVE,
+ rte_memory_order_release);
+
+ DRV_LOG(DEBUG, "Exiting the reset complete processing");
+
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_RECOVERY_SUCCESS for port %u",
+ priv->port_id);
+ rte_eth_dev_callback_process(dev,
+ RTE_ETH_EVENT_RECOVERY_SUCCESS, NULL);
+
+out:
+ if (ret) {
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_RECOVERY_FAILED for port %u",
+ priv->port_id);
+ rte_eth_dev_callback_process(dev,
+ RTE_ETH_EVENT_RECOVERY_FAILED, NULL);
+ }
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return ret;
+
+mr_init_failed_all:
+ i = priv->num_queues;
+ goto mr_init_failed_rxq;
+
+mr_init_failed_txq:
+ /* RXQ btree at index i was initialized, free it */
+ mana_mr_btree_free(&((struct mana_rxq *)
+ dev->data->rx_queues[i])->mr_btree);
+
+mr_init_failed_rxq:
+ /* Free all fully initialized btrees for indices < i */
+ for (int j = 0; j < i; j++) {
+ struct mana_rxq *rxq = dev->data->rx_queues[j];
+ struct mana_txq *txq = dev->data->tx_queues[j];
+
+ mana_mr_btree_free(&rxq->mr_btree);
+ mana_mr_btree_free(&txq->mr_btree);
+ }
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_RECOVERY_FAILED (MR init) for port %u",
+ priv->port_id);
+ rte_eth_dev_callback_process(dev,
+ RTE_ETH_EVENT_RECOVERY_FAILED, NULL);
+
+ pthread_mutex_unlock(&priv->reset_ops_lock);
+ return ret;
+}
+
+static void
+mana_reset_exit(struct mana_priv *priv)
+ __rte_no_thread_safety_analysis
+{
+ int ret;
+
+ if (!priv) {
+ DRV_LOG(ERR, "Private structure invalid");
+ return;
+ }
+ DRV_LOG(DEBUG, "Entering into device reset complete processing");
+
+ rxq_intr_disable(priv);
+
+ /* Unregister the interrupt handler. Since mana_reset_exit is always
+ * called from mana_reset_thread (a non-interrupt thread), the
+ * interrupt source is inactive and rte_intr_callback_unregister
+ * succeeds directly.
+ */
+ if (priv->intr_handle) {
+ ret = rte_intr_callback_unregister(priv->intr_handle,
+ mana_intr_handler, priv);
+ if (ret < 0)
+ DRV_LOG(ERR, "Failed to unregister intr callback ret %d",
+ ret);
+ else
+ DRV_LOG(DEBUG, "%d intr callback(s) removed", ret);
+
+ rte_intr_instance_free(priv->intr_handle);
+ priv->intr_handle = NULL;
+ }
+
+ /* Proceed directly to reset exit delay (re-probe and restart).
+ * No need for a separate thread - we are already on
+ * mana_reset_thread which is a non-interrupt control thread.
+ */
+ mana_reset_exit_delay(priv);
+}
+
+/*
+ * Interrupt handler from IB layer to notify this device is
+ * being removed or reset.
*/
static void
mana_intr_handler(void *arg)
+ __rte_no_thread_safety_analysis
{
struct mana_priv *priv = arg;
struct ibv_context *ctx = priv->ib_ctx;
- struct ibv_async_event event;
+ struct ibv_async_event event = { 0 };
+ struct rte_eth_dev *dev;
/* Read and ack all messages from IB device */
while (true) {
if (ibv_get_async_event(ctx, &event))
break;
- if (event.event_type == IBV_EVENT_DEVICE_FATAL) {
- struct rte_eth_dev *dev;
-
- dev = &rte_eth_devices[priv->port_id];
- if (dev->data->dev_conf.intr_conf.rmv)
+ switch (event.event_type) {
+ case IBV_EVENT_DEVICE_FATAL:
+ DRV_LOG(INFO, "IBV_EVENT_DEVICE_FATAL received, dev_state=%d",
+ (int)rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire));
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) == MANA_DEV_ACTIVE) {
+ pthread_mutex_lock(&priv->reset_ops_lock);
+
+ /* Re-check after lock to avoid racing with
+ * mana_pci_remove_event_cb which may have
+ * set RESET_FAILED while we waited.
+ */
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) !=
+ MANA_DEV_ACTIVE) {
+ rte_spinlock_unlock(
+ &priv->reset_ops_lock);
+ break;
+ }
+ mana_reset_enter(priv);
+
+ dev = &rte_eth_devices[priv->port_id];
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_ERR_RECOVERING for port %u",
+ priv->port_id);
rte_eth_dev_callback_process(dev,
- RTE_ETH_EVENT_INTR_RMV, NULL);
+ RTE_ETH_EVENT_ERR_RECOVERING, NULL);
+ } else {
+ DRV_LOG(ERR, "Already in reset handling, dev_state=%d",
+ (int)rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire));
+ }
+ break;
+
+ default:
+ break;
}
ibv_ack_async_event(&event);
@@ -1063,6 +1787,23 @@ static int
mana_intr_uninstall(struct mana_priv *priv)
{
int ret;
+ struct rte_eth_dev *dev;
+
+ if (!priv->intr_handle)
+ return 0;
+
+ /* Unregister PCI device removal event callback.
+ * Do not retry on -EAGAIN to avoid deadlock: the callback
+ * may be blocked waiting for reset_ops_lock which we hold.
+ */
+ dev = &rte_eth_devices[priv->port_id];
+ if (dev->device) {
+ ret = rte_dev_event_callback_unregister(dev->device->name,
+ mana_pci_remove_event_cb, priv);
+ if (ret < 0 && ret != -ENOENT)
+ DRV_LOG(WARNING, "Failed to unregister PCI remove cb ret %d",
+ ret);
+ }
ret = rte_intr_callback_unregister(priv->intr_handle,
mana_intr_handler, priv);
@@ -1072,6 +1813,7 @@ mana_intr_uninstall(struct mana_priv *priv)
}
rte_intr_instance_free(priv->intr_handle);
+ priv->intr_handle = NULL;
return 0;
}
@@ -1127,6 +1869,16 @@ mana_intr_install(struct rte_eth_dev *eth_dev, struct mana_priv *priv)
goto free_intr;
}
+ /* Register for PCI device removal events to distinguish
+ * PCI hot-remove from service reset. This requires the
+ * application to call rte_dev_event_monitor_start() for
+ * events to be delivered (e.g. testpmd --hot-plug-handling).
+ */
+ ret = rte_dev_event_callback_register(eth_dev->device->name,
+ mana_pci_remove_event_cb, priv);
+ if (ret)
+ DRV_LOG(WARNING, "Failed to register PCI remove event callback");
+
eth_dev->intr_handle = priv->intr_handle;
return 0;
@@ -1156,7 +1908,7 @@ mana_proc_priv_init(struct rte_eth_dev *dev)
/*
* Map the doorbell page for the secondary process through IB device handle.
*/
-static int
+int
mana_map_doorbell_secondary(struct rte_eth_dev *eth_dev, int fd)
{
struct mana_process_priv *priv = eth_dev->process_private;
@@ -1294,17 +2046,30 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
char name[RTE_ETH_NAME_MAX_LEN];
int ret;
struct ibv_context *ctx = NULL;
+ size_t sz;
+ bool is_reset = false;
+ pthread_mutexattr_t mattr;
+ pthread_condattr_t cattr;
rte_ether_format_addr(address, sizeof(address), addr);
- DRV_LOG(INFO, "device located port %u address %s", port, address);
- priv = rte_zmalloc_socket(NULL, sizeof(*priv), RTE_CACHE_LINE_SIZE,
- SOCKET_ID_ANY);
- if (!priv)
- return -ENOMEM;
+ DRV_LOG(DEBUG, "device located port %u address %s", port, address);
snprintf(name, sizeof(name), "%s_port%d", pci_dev->device.name, port);
+ eth_dev = rte_eth_dev_allocated(name);
+ if (eth_dev) {
+ is_reset = true;
+ priv = eth_dev->data->dev_private;
+ DRV_LOG(DEBUG, "Device reset for eth_dev %p priv %p",
+ eth_dev, priv);
+ } else {
+ priv = rte_zmalloc_socket(NULL, sizeof(*priv), RTE_CACHE_LINE_SIZE,
+ SOCKET_ID_ANY);
+ if (!priv)
+ return -ENOMEM;
+ }
+
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
int fd;
@@ -1317,6 +2082,7 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
eth_dev->device = &pci_dev->device;
eth_dev->dev_ops = &mana_dev_secondary_ops;
+
ret = mana_proc_priv_init(eth_dev);
if (ret)
goto failed;
@@ -1336,7 +2102,7 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
goto failed;
}
- /* fd is no not used after mapping doorbell */
+ /* fd is not used after mapping doorbell */
close(fd);
eth_dev->tx_pkt_burst = mana_tx_burst;
@@ -1355,22 +2121,6 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
goto failed;
}
- eth_dev = rte_eth_dev_allocate(name);
- if (!eth_dev) {
- ret = -ENOMEM;
- goto failed;
- }
-
- eth_dev->data->mac_addrs =
- rte_calloc("mana_mac", 1,
- sizeof(struct rte_ether_addr), 0);
- if (!eth_dev->data->mac_addrs) {
- ret = -ENOMEM;
- goto failed;
- }
-
- rte_ether_addr_copy(addr, eth_dev->data->mac_addrs);
-
priv->ib_pd = ibv_alloc_pd(ctx);
if (!priv->ib_pd) {
DRV_LOG(ERR, "ibv_alloc_pd failed port %d", port);
@@ -1390,10 +2140,6 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
}
priv->ib_ctx = ctx;
- priv->port_id = eth_dev->data->port_id;
- priv->dev_port = port;
- eth_dev->data->dev_private = priv;
- priv->dev_data = eth_dev->data;
priv->max_rx_queues = dev_attr->orig_attr.max_qp;
priv->max_tx_queues = dev_attr->orig_attr.max_qp;
@@ -1415,23 +2161,92 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
name, priv->max_rx_queues, priv->max_rx_desc,
priv->max_send_sge, priv->max_mr_size);
+ if (!is_reset) {
+ eth_dev = rte_eth_dev_allocate(name);
+ if (!eth_dev) {
+ ret = -ENOMEM;
+ goto failed;
+ }
+
+ eth_dev->data->mac_addrs =
+ rte_calloc("mana_mac", 1,
+ sizeof(struct rte_ether_addr), 0);
+ if (!eth_dev->data->mac_addrs) {
+ ret = -ENOMEM;
+ goto failed;
+ }
+
+ rte_ether_addr_copy(addr, eth_dev->data->mac_addrs);
+ } else {
+ /*
+ * Reset path.
+ */
+ rte_ether_format_addr(address, RTE_ETHER_ADDR_FMT_SIZE,
+ eth_dev->data->mac_addrs);
+ DRV_LOG(DEBUG, "Found existing eth_dev %p with mac addr %s",
+ eth_dev, address);
+ DRV_LOG(DEBUG, "ib_ctx = %p", priv->ib_ctx);
+ goto out;
+ }
+
+ priv->port_id = eth_dev->data->port_id;
+ priv->dev_port = port;
+ eth_dev->data->dev_private = priv;
+ priv->dev_data = eth_dev->data;
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_ACTIVE,
+ rte_memory_order_release);
+
rte_eth_copy_pci_info(eth_dev, pci_dev);
- /* Create async interrupt handler */
- ret = mana_intr_install(eth_dev, priv);
- if (ret) {
- DRV_LOG(ERR, "Failed to install intr handler");
+ /*
+ * Now we've got maximum queues. Init the qsv to be the
+ * double of maximum queues for both rx and tx queues.
+ */
+ sz = rte_rcu_qsbr_get_memsize(2 * priv->max_rx_queues);
+ priv->dev_state_qsv = rte_zmalloc_socket("mana_rcu", sz,
+ RTE_CACHE_LINE_SIZE,
+ SOCKET_ID_ANY);
+ if (!priv->dev_state_qsv) {
+ DRV_LOG(ERR, "No memory for dev_state_qsv");
+ ret = -ENOMEM;
+ goto failed;
+ }
+ ret = rte_rcu_qsbr_init(priv->dev_state_qsv, 2 * priv->max_rx_queues);
+ if (ret < 0) {
+ DRV_LOG(ERR, "Init dev_state_qsv failed ret %d", ret);
goto failed;
}
- eth_dev->device = &pci_dev->device;
+ pthread_mutexattr_init(&mattr);
+ pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED);
+ pthread_mutex_init(&priv->reset_ops_lock, &mattr);
+ pthread_mutex_init(&priv->reset_cond_mutex, &mattr);
+ pthread_mutexattr_destroy(&mattr);
- DRV_LOG(INFO, "device %s at port %u", name, eth_dev->data->port_id);
+ pthread_condattr_init(&cattr);
+ pthread_condattr_setpshared(&cattr, PTHREAD_PROCESS_SHARED);
+ pthread_cond_init(&priv->reset_cond, &cattr);
+ pthread_condattr_destroy(&cattr);
+
+ eth_dev->device = &pci_dev->device;
eth_dev->rx_pkt_burst = mana_rx_burst_removed;
eth_dev->tx_pkt_burst = mana_tx_burst_removed;
eth_dev->dev_ops = &mana_dev_ops;
+out:
+ /* Create async interrupt handler */
+ ret = mana_intr_install(eth_dev, priv);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to install intr handler, ret %d", ret);
+ goto failed;
+ } else {
+ DRV_LOG(INFO, "mana_intr_install succeeded");
+ }
+
+ DRV_LOG(INFO, "device %s priv %p dev port %d at port %u",
+ name, priv, priv->dev_port, eth_dev->data->port_id);
+
rte_eth_dev_probing_finish(eth_dev);
return 0;
@@ -1439,20 +2254,32 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
failed:
/* Free the resource for the port failed */
if (priv) {
- if (priv->ib_parent_pd)
+ if (!is_reset && priv->dev_state_qsv)
+ rte_free(priv->dev_state_qsv);
+
+ if (priv->ib_parent_pd) {
ibv_dealloc_pd(priv->ib_parent_pd);
+ priv->ib_parent_pd = NULL;
+ }
- if (priv->ib_pd)
+ if (priv->ib_pd) {
ibv_dealloc_pd(priv->ib_pd);
+ priv->ib_pd = NULL;
+ }
}
- if (eth_dev)
- rte_eth_dev_release_port(eth_dev);
+ if (!is_reset) {
+ if (eth_dev)
+ rte_eth_dev_release_port(eth_dev);
- rte_free(priv);
+ rte_free(priv);
+ }
- if (ctx)
+ if (ctx) {
ibv_close_device(ctx);
+ if (is_reset && priv)
+ priv->ib_ctx = NULL;
+ }
return ret;
}
@@ -1617,7 +2444,17 @@ mana_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
static int
mana_dev_uninit(struct rte_eth_dev *dev)
{
- return mana_dev_close(dev);
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ /* Join reset thread before teardown to ensure it has exited
+ * before we destroy the condvar/mutex in free_resources.
+ */
+ mana_join_reset_thread(priv);
+
+ ret = mana_dev_close(dev);
+ mana_dev_free_resources(dev);
+ return ret;
}
/*
diff --git a/drivers/net/mana/mana.h b/drivers/net/mana/mana.h
index 79cc47b6ab..115bc722f4 100644
--- a/drivers/net/mana/mana.h
+++ b/drivers/net/mana/mana.h
@@ -5,6 +5,8 @@
#ifndef __MANA_H__
#define __MANA_H__
+#include <pthread.h>
+
#define PCI_VENDOR_ID_MICROSOFT 0x1414
#define PCI_DEVICE_ID_MICROSOFT_MANA_PF 0x00b9
#define PCI_DEVICE_ID_MICROSOFT_MANA 0x00ba
@@ -337,6 +339,20 @@ struct mana_process_priv {
void *db_page;
};
+enum mana_device_state {
+ /* Normal running */
+ MANA_DEV_ACTIVE = 0,
+ /* In reset enter processing */
+ MANA_DEV_RESET_ENTER = 1,
+ /*
+ * Reset enter processing completed.
+ * Waiting for reset exit or in reset exit processing.
+ */
+ MANA_DEV_RESET_EXIT = 2,
+ /* Reset failed */
+ MANA_DEV_RESET_FAILED = 3,
+};
+
struct mana_priv {
struct rte_eth_dev_data *dev_data;
struct mana_process_priv *process_priv;
@@ -368,6 +384,16 @@ struct mana_priv {
uint64_t max_mr_size;
struct mana_mr_btree mr_btree;
rte_spinlock_t mr_btree_lock;
+ RTE_ATOMIC(enum mana_device_state) dev_state;
+ struct rte_rcu_qsbr *dev_state_qsv;
+ /* mutex for synchronizing mana reset and some mana_dev_ops callbacks */
+ pthread_mutex_t reset_ops_lock;
+ /* Reset thread ID, valid when reset_thread_active is true */
+ rte_thread_t reset_thread;
+ RTE_ATOMIC(bool) reset_thread_active;
+ /* Condvar to wake reset thread early on PCI remove */
+ pthread_mutex_t reset_cond_mutex;
+ pthread_cond_t reset_cond;
};
struct mana_txq_desc {
@@ -427,6 +453,7 @@ struct mana_txq {
struct mana_mr_btree mr_btree;
struct mana_stats stats;
unsigned int socket;
+ unsigned int txq_idx;
};
struct mana_rxq {
@@ -462,6 +489,7 @@ struct mana_rxq {
struct mana_mr_btree mr_btree;
unsigned int socket;
+ unsigned int rxq_idx;
};
extern int mana_logtype_driver;
@@ -543,6 +571,8 @@ enum mana_mp_req_type {
MANA_MP_REQ_CREATE_MR,
MANA_MP_REQ_START_RXTX,
MANA_MP_REQ_STOP_RXTX,
+ MANA_MP_REQ_RESET_ENTER,
+ MANA_MP_REQ_RESET_EXIT,
};
/* Pameters for IPC. */
@@ -563,8 +593,9 @@ void mana_mp_uninit_primary(void);
void mana_mp_uninit_secondary(void);
int mana_mp_req_verbs_cmd_fd(struct rte_eth_dev *dev);
int mana_mp_req_mr_create(struct mana_priv *priv, uintptr_t addr, uint32_t len);
+int mana_map_doorbell_secondary(struct rte_eth_dev *eth_dev, int fd);
-void mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type);
+int mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type);
void *mana_alloc_verbs_buf(size_t size, void *data);
void mana_free_verbs_buf(void *ptr, void *data __rte_unused);
diff --git a/drivers/net/mana/meson.build b/drivers/net/mana/meson.build
index 19d4b3695e..5b01d9f57e 100644
--- a/drivers/net/mana/meson.build
+++ b/drivers/net/mana/meson.build
@@ -7,7 +7,7 @@ if not is_linux or not (dpdk_conf.has('RTE_ARCH_X86') or dpdk_conf.has('RTE_ARCH
subdir_done()
endif
-deps += ['pci', 'bus_pci', 'net', 'eal', 'kvargs']
+deps += ['pci', 'bus_pci', 'net', 'eal', 'kvargs', 'rcu']
sources += files(
'gdma.c',
diff --git a/drivers/net/mana/mp.c b/drivers/net/mana/mp.c
index 72417fc0c7..1161ebd71c 100644
--- a/drivers/net/mana/mp.c
+++ b/drivers/net/mana/mp.c
@@ -2,10 +2,13 @@
* Copyright 2022 Microsoft Corporation
*/
+#include <sys/mman.h>
#include <rte_malloc.h>
#include <ethdev_driver.h>
#include <rte_log.h>
+#include <rte_eal_paging.h>
#include <stdlib.h>
+#include <unistd.h>
#include <infiniband/verbs.h>
@@ -119,6 +122,23 @@ mana_mp_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
return ret;
}
+static int
+mana_mp_reset_enter(struct rte_eth_dev *dev)
+{
+ struct mana_process_priv *proc_priv = dev->process_private;
+
+ void *addr = proc_priv->db_page;
+
+ /* Reset the db_page to NULL */
+ proc_priv->db_page = NULL;
+
+ if (addr)
+ (void)munmap(addr, rte_mem_page_size());
+
+ DRV_LOG(DEBUG, "Secondary doorbell pages unmapped");
+ return 0;
+}
+
static int
mana_mp_secondary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
{
@@ -171,6 +191,49 @@ mana_mp_secondary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
ret = rte_mp_reply(&mp_res, peer);
break;
+ case MANA_MP_REQ_RESET_ENTER:
+ DRV_LOG(INFO, "Port %u reset enter", dev->data->port_id);
+ res->result = mana_mp_reset_enter(dev);
+
+ ret = rte_mp_reply(&mp_res, peer);
+ break;
+
+ case MANA_MP_REQ_RESET_EXIT:
+ DRV_LOG(INFO, "Port %u reset exit", dev->data->port_id);
+ {
+ struct mana_process_priv *proc_priv =
+ dev->process_private;
+
+ if (proc_priv->db_page != NULL) {
+ DRV_LOG(DEBUG,
+ "Secondary doorbell already "
+ "mapped to %p",
+ proc_priv->db_page);
+ res->result = 0;
+ } else if (mp_msg->num_fds < 1) {
+ DRV_LOG(ERR,
+ "No FD in RESET_EXIT message");
+ res->result = -EINVAL;
+ } else {
+ int fd = mp_msg->fds[0];
+
+ ret = mana_map_doorbell_secondary(dev,
+ fd);
+ if (ret) {
+ DRV_LOG(ERR,
+ "Failed secondary "
+ "doorbell map %d",
+ fd);
+ res->result = -ENODEV;
+ } else {
+ res->result = 0;
+ }
+ close(fd);
+ }
+ }
+ ret = rte_mp_reply(&mp_res, peer);
+ break;
+
default:
DRV_LOG(ERR, "Port %u unknown secondary MP type %u",
param->port_id, param->type);
@@ -254,7 +317,7 @@ mana_mp_req_verbs_cmd_fd(struct rte_eth_dev *dev)
}
ret = mp_res->fds[0];
- DRV_LOG(ERR, "port %u command FD from primary is %d",
+ DRV_LOG(DEBUG, "port %u command FD from primary is %d",
dev->data->port_id, ret);
exit:
free(mp_rep.msgs);
@@ -298,27 +361,36 @@ mana_mp_req_mr_create(struct mana_priv *priv, uintptr_t addr, uint32_t len)
return ret;
}
-void
+int
mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type)
{
struct rte_mp_msg mp_req = { 0 };
struct rte_mp_msg *mp_res;
- struct rte_mp_reply mp_rep;
+ struct rte_mp_reply mp_rep = { 0 };
struct mana_mp_param *res;
struct timespec ts = {.tv_sec = MANA_MP_REQ_TIMEOUT_SEC, .tv_nsec = 0};
- int i, ret;
+ int i, ret = 0;
- if (type != MANA_MP_REQ_START_RXTX && type != MANA_MP_REQ_STOP_RXTX) {
+ if (type != MANA_MP_REQ_START_RXTX && type != MANA_MP_REQ_STOP_RXTX &&
+ type != MANA_MP_REQ_RESET_ENTER && type != MANA_MP_REQ_RESET_EXIT) {
DRV_LOG(ERR, "port %u unknown request (req_type %d)",
dev->data->port_id, type);
- return;
+ return -EINVAL;
}
if (rte_atomic_load_explicit(&mana_shared_data->secondary_cnt, rte_memory_order_relaxed) == 0)
- return;
+ return 0;
mp_init_msg(&mp_req, type, dev->data->port_id);
+ /* Include IB cmd FD for secondary doorbell remap */
+ if (type == MANA_MP_REQ_RESET_EXIT) {
+ struct mana_priv *priv = dev->data->dev_private;
+
+ mp_req.num_fds = 1;
+ mp_req.fds[0] = priv->ib_ctx->cmd_fd;
+ }
+
ret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);
if (ret) {
if (rte_errno != ENOTSUP)
@@ -329,6 +401,7 @@ mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type)
if (mp_rep.nb_sent != mp_rep.nb_received) {
DRV_LOG(ERR, "port %u not all secondaries responded (%d)",
dev->data->port_id, type);
+ ret = -ETIMEDOUT;
goto exit;
}
for (i = 0; i < mp_rep.nb_received; i++) {
@@ -337,9 +410,11 @@ mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type)
if (res->result) {
DRV_LOG(ERR, "port %u request failed on secondary %d",
dev->data->port_id, i);
+ ret = res->result;
goto exit;
}
}
exit:
free(mp_rep.msgs);
+ return ret;
}
diff --git a/drivers/net/mana/mr.c b/drivers/net/mana/mr.c
index c4045141bc..8914f4cf04 100644
--- a/drivers/net/mana/mr.c
+++ b/drivers/net/mana/mr.c
@@ -314,8 +314,10 @@ mana_mr_btree_init(struct mana_mr_btree *bt, int n, int socket)
void
mana_mr_btree_free(struct mana_mr_btree *bt)
{
- rte_free(bt->table);
- memset(bt, 0, sizeof(*bt));
+ if (bt && bt->table) {
+ rte_free(bt->table);
+ memset(bt, 0, sizeof(*bt));
+ }
}
int
diff --git a/drivers/net/mana/rx.c b/drivers/net/mana/rx.c
index 1b8ba1f3a9..ae05d8dd2f 100644
--- a/drivers/net/mana/rx.c
+++ b/drivers/net/mana/rx.c
@@ -2,6 +2,7 @@
* Copyright 2022 Microsoft Corporation
*/
#include <ethdev_driver.h>
+#include <rte_rcu_qsbr.h>
#include <infiniband/verbs.h>
#include <infiniband/manadv.h>
@@ -36,6 +37,11 @@ mana_rq_ring_doorbell(struct mana_rxq *rxq)
db_page = process_priv->db_page;
}
+ if (!db_page) {
+ DP_LOG(ERR, "db_page is NULL, cannot ring RX doorbell");
+ return -EINVAL;
+ }
+
/* Hardware Spec specifies that software client should set 0 for
* wqe_cnt for Receive Queues.
*/
@@ -172,7 +178,7 @@ mana_stop_rx_queues(struct rte_eth_dev *dev)
for (i = 0; i < priv->num_queues; i++)
if (dev->data->rx_queue_state[i] == RTE_ETH_QUEUE_STATE_STOPPED)
- return -EINVAL;
+ return 0;
if (priv->rwq_qp) {
ret = ibv_destroy_qp(priv->rwq_qp);
@@ -256,6 +262,9 @@ mana_start_rx_queues(struct rte_eth_dev *dev)
struct mana_rxq *rxq = dev->data->rx_queues[i];
struct ibv_wq_init_attr wq_attr = {};
+ rxq->rxq_idx = i;
+ DRV_LOG(DEBUG, "assigning rxq_idx to %d", i);
+
manadv_set_context_attr(priv->ib_ctx,
MANADV_CTX_ATTR_BUF_ALLOCATORS,
(void *)((uintptr_t)&(struct manadv_ctx_allocators){
@@ -451,6 +460,17 @@ mana_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
uint32_t pkt_len;
uint32_t i;
int polled = 0;
+ struct rte_rcu_qsbr *dstate_qsv = priv->dev_state_qsv;
+ unsigned int tid = rxq->rxq_idx;
+
+ rte_rcu_qsbr_thread_online(dstate_qsv, tid);
+
+ if (unlikely(rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE)) {
+ /* Device reset occurred. */
+ rte_rcu_qsbr_thread_offline(dstate_qsv, tid);
+ return 0;
+ }
repoll:
/* Polling on new completions if we have no backlog */
@@ -592,6 +612,8 @@ mana_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
wqe_consumed, ret);
}
+ rte_rcu_qsbr_thread_offline(dstate_qsv, tid);
+
return pkt_received;
}
diff --git a/drivers/net/mana/tx.c b/drivers/net/mana/tx.c
index 57dbbc3651..3b07a8b9a6 100644
--- a/drivers/net/mana/tx.c
+++ b/drivers/net/mana/tx.c
@@ -3,6 +3,7 @@
*/
#include <ethdev_driver.h>
+#include <rte_rcu_qsbr.h>
#include <infiniband/verbs.h>
#include <infiniband/manadv.h>
@@ -17,7 +18,7 @@ mana_stop_tx_queues(struct rte_eth_dev *dev)
for (i = 0; i < priv->num_queues; i++)
if (dev->data->tx_queue_state[i] == RTE_ETH_QUEUE_STATE_STOPPED)
- return -EINVAL;
+ return 0;
for (i = 0; i < priv->num_queues; i++) {
struct mana_txq *txq = dev->data->tx_queues[i];
@@ -83,6 +84,9 @@ mana_start_tx_queues(struct rte_eth_dev *dev)
txq = dev->data->tx_queues[i];
+ txq->txq_idx = i;
+ DRV_LOG(DEBUG, "assigning txq_idx to %d", txq->txq_idx);
+
manadv_set_context_attr(priv->ib_ctx,
MANADV_CTX_ATTR_BUF_ALLOCATORS,
(void *)((uintptr_t)&(struct manadv_ctx_allocators){
@@ -190,10 +194,30 @@ mana_tx_burst(void *dpdk_txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
void *db_page;
uint16_t pkt_sent = 0;
uint32_t num_comp, i;
+ unsigned int tid = priv->num_queues + txq->txq_idx;
+ struct rte_rcu_qsbr *dstate_qsv = priv->dev_state_qsv;
#ifdef RTE_ARCH_32
uint32_t wqe_count = 0;
#endif
+ db_page = priv->db_page;
+ if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+ struct rte_eth_dev *dev =
+ &rte_eth_devices[priv->dev_data->port_id];
+ struct mana_process_priv *process_priv = dev->process_private;
+
+ db_page = process_priv->db_page;
+ }
+
+ rte_rcu_qsbr_thread_online(dstate_qsv, tid);
+
+ if (unlikely(rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE || !db_page)) {
+ /* Device reset event occurred. */
+ rte_rcu_qsbr_thread_offline(dstate_qsv, tid);
+ return 0;
+ }
+
/* Process send completions from GDMA */
num_comp = gdma_poll_completion_queue(&txq->gdma_cq,
txq->gdma_comp_buf, txq->num_desc);
@@ -216,7 +240,8 @@ mana_tx_burst(void *dpdk_txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
}
if (!desc->pkt) {
- DP_LOG(ERR, "mana_txq_desc has a NULL pkt");
+ DP_LOG(ERR, "mana_txq_desc has a NULL pkt, priv %p, "
+ "txq = %d", priv, txq->txq_idx);
} else {
txq->stats.bytes += desc->pkt->pkt_len;
rte_pktmbuf_free(desc->pkt);
@@ -474,15 +499,6 @@ mana_tx_burst(void *dpdk_txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
}
/* Ring hardware door bell */
- db_page = priv->db_page;
- if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
- struct rte_eth_dev *dev =
- &rte_eth_devices[priv->dev_data->port_id];
- struct mana_process_priv *process_priv = dev->process_private;
-
- db_page = process_priv->db_page;
- }
-
if (pkt_sent) {
#ifdef RTE_ARCH_32
ret = mana_ring_short_doorbell(db_page, GDMA_QUEUE_SEND,
@@ -501,5 +517,7 @@ mana_tx_burst(void *dpdk_txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
DP_LOG(ERR, "mana_ring_doorbell failed ret %d", ret);
}
+ rte_rcu_qsbr_thread_offline(dstate_qsv, tid);
+
return pkt_sent;
}
--
2.34.1
^ permalink raw reply related
* [PATCH v4 00/10] selective Rx
From: Thomas Monjalon @ 2026-05-29 13:33 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger
In-Reply-To: <20260202160903.254621-1-getelson@nvidia.com>
This is a new feature in ethdev with tests and mlx5 implementation.
Selective Rx allows to receive partial data,
saving some hardware bandwidth.
Note 1: mlx5 support patch is not correctly indented
to make review easier. An indent patch follows to be squashed.
Note 2: DTS patch is an attempt to test the feature on day 1,
it is not mandatory if it is blocking the merge.
v2: rework after Gregory
v3: fix bugs found with AI by Stephen
v4: fix packet type in DTS test
Gregory Etelson (4):
ethdev: introduce selective Rx
app/testpmd: support selective Rx
common/mlx5: add null MR functions
net/mlx5: support selective Rx
Thomas Monjalon (6):
app/testpmd: print Rx split capabilities
net/mlx5: fix Rx split segment counter type
net/mlx5: reindent previous changes
common/mlx5: remove callbacks for MR registration
dts: fix topology capability comparison
dts: add selective Rx tests
app/test-pmd/config.c | 17 ++
app/test-pmd/testpmd.c | 71 ++++-
devtools/libabigail.abignore | 7 +
doc/guides/nics/features.rst | 14 +
doc/guides/nics/features/default.ini | 1 +
doc/guides/nics/features/mlx5.ini | 1 +
doc/guides/nics/mlx5.rst | 86 ++++--
doc/guides/rel_notes/release_26_07.rst | 11 +
doc/guides/testpmd_app_ug/run_app.rst | 20 ++
drivers/common/mlx5/linux/mlx5_common_verbs.c | 53 ++--
drivers/common/mlx5/mlx5_common.c | 6 +-
drivers/common/mlx5/mlx5_common_mr.c | 37 ++-
drivers/common/mlx5/mlx5_common_mr.h | 29 +-
drivers/common/mlx5/windows/mlx5_common_os.c | 31 ++-
drivers/compress/mlx5/mlx5_compress.c | 4 +-
drivers/crypto/mlx5/mlx5_crypto.h | 2 -
drivers/crypto/mlx5/mlx5_crypto_gcm.c | 6 +-
drivers/net/mlx5/mlx5.c | 7 +
drivers/net/mlx5/mlx5.h | 4 +-
drivers/net/mlx5/mlx5_ethdev.c | 25 ++
drivers/net/mlx5/mlx5_flow_aso.c | 21 +-
drivers/net/mlx5/mlx5_flow_hw.c | 11 +-
drivers/net/mlx5/mlx5_flow_quota.c | 6 +-
drivers/net/mlx5/mlx5_hws_cnt.c | 19 +-
drivers/net/mlx5/mlx5_rx.c | 102 ++++---
drivers/net/mlx5/mlx5_rx.h | 5 +-
drivers/net/mlx5/mlx5_rxq.c | 75 +++--
drivers/net/mlx5/mlx5_trigger.c | 64 ++++-
dts/api/capabilities.py | 2 +
dts/api/testpmd/__init__.py | 17 ++
dts/api/testpmd/types.py | 6 +
dts/framework/testbed_model/capability.py | 10 +-
dts/tests/TestSuite_rx_split.py | 262 ++++++++++++++++++
lib/ethdev/rte_ethdev.c | 24 +-
lib/ethdev/rte_ethdev.h | 14 +-
35 files changed, 831 insertions(+), 239 deletions(-)
create mode 100644 dts/tests/TestSuite_rx_split.py
--
2.54.0
^ permalink raw reply
* [PATCH v4 01/10] app/testpmd: print Rx split capabilities
From: Thomas Monjalon @ 2026-05-29 13:33 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger, Aman Singh
In-Reply-To: <20260529133522.2646044-1-thomas@monjalon.net>
The capabilities from rte_eth_rxseg_capa are added
to the command "show port info".
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
app/test-pmd/config.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index c950793aaf..55d1c6d696 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -790,6 +790,12 @@ rss_offload_types_display(uint64_t offload_types, uint16_t char_num_per_line)
printf("\n");
}
+static void
+print_bool_capa(const char *label, int value)
+{
+ printf("%s: %s\n", label, value ? "supported" : "not supported");
+}
+
void
port_infos_display(portid_t port_id)
{
@@ -911,6 +917,16 @@ port_infos_display(portid_t port_id)
dev_info.max_rx_pktlen);
printf("Maximum configurable size of LRO aggregated packet: %u\n",
dev_info.max_lro_pkt_size);
+
+ printf("Rx split:\n");
+ printf("\tMax segments: %hu\n", dev_info.rx_seg_capa.max_nseg);
+ if (dev_info.rx_seg_capa.max_nseg > 0) {
+ print_bool_capa("\tMulti-pool", dev_info.rx_seg_capa.multi_pools);
+ print_bool_capa("\tBuffer offset", dev_info.rx_seg_capa.offset_allowed);
+ printf("\tOffset alignment: %u\n",
+ RTE_BIT32(dev_info.rx_seg_capa.offset_align_log2));
+ }
+
if (dev_info.max_vfs)
printf("Maximum number of VFs: %u\n", dev_info.max_vfs);
if (dev_info.max_vmdq_pools)
--
2.54.0
^ permalink raw reply related
* [PATCH v4 02/10] ethdev: introduce selective Rx
From: Thomas Monjalon @ 2026-05-29 13:33 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger, Gregory Etelson, Aman Singh, Andrew Rybchenko
In-Reply-To: <20260529133522.2646044-1-thomas@monjalon.net>
From: Gregory Etelson <getelson@nvidia.com>
Receiving an entire packet is not always needed.
The Rx performance can be improved by receiving only partial data
and safely discard the rest of the packet data,
because it reduces the PCI bandwidth and the memory consumption.
Selective Rx allows an application to receive
only pre-configured packet segments and discard the rest.
For example:
- Deliver the first N bytes only.
- Deliver the last N bytes only.
- Deliver N1 bytes from offset Off1 and N2 bytes from offset Off2.
Selective Rx is implemented on top of the Rx buffer split API:
- rte_eth_rxseg_split uses the null mempool for segments
that should be discarded.
- the PMD does not create mbuf segments if no data read.
For example: Deliver Ethernet header only
Rx queue segments configuration:
struct rte_eth_rxseg_split split[2] = {
{
.mp = <some mempool>,
.length = sizeof(struct rte_ether_hdr)
},
{
.mp = NULL, /* discard data */
.length = 0 /* default to buffer size */
}
};
Received mbuf:
pkt_len = sizeof(struct rte_ether_hdr);
data_len = sizeof(struct rte_ether_hdr);
next = NULL; /* The next segment did not deliver data */
After selective Rx, the mbuf packet length reflects only the data
that was actually received,
and can be less than the original wire packet length.
A PMD activates the selective Rx capability by setting
the rte_eth_rxseg_capa.selective_rx bit.
This new capability bit is inserted in a bitmap hole
of the struct rte_eth_rxseg_capa,
but it needs to be ignored in the ABI check as libabigail sees a change.
Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
app/test-pmd/config.c | 1 +
devtools/libabigail.abignore | 7 +++++++
doc/guides/nics/features.rst | 14 ++++++++++++++
doc/guides/nics/features/default.ini | 1 +
doc/guides/rel_notes/release_26_07.rst | 7 +++++++
lib/ethdev/rte_ethdev.c | 24 ++++++++++++++++--------
lib/ethdev/rte_ethdev.h | 14 +++++++++++++-
7 files changed, 59 insertions(+), 9 deletions(-)
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 55d1c6d696..9d457ca88e 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -925,6 +925,7 @@ port_infos_display(portid_t port_id)
print_bool_capa("\tBuffer offset", dev_info.rx_seg_capa.offset_allowed);
printf("\tOffset alignment: %u\n",
RTE_BIT32(dev_info.rx_seg_capa.offset_align_log2));
+ print_bool_capa("\tSelective Rx", dev_info.rx_seg_capa.selective_rx);
}
if (dev_info.max_vfs)
diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore
index 21b8cd6113..2a0efd718e 100644
--- a/devtools/libabigail.abignore
+++ b/devtools/libabigail.abignore
@@ -33,3 +33,10 @@
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Temporary exceptions till next major ABI version ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+; Ignore new bit selective_rx in rte_eth_rxseg_capa bitmap hole
+[suppress_type]
+ name = rte_eth_rxseg_capa
+ type_kind = struct
+ has_size_change = no
+ has_data_member_inserted_at = 6
diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index a075c057ec..26357036ca 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -199,6 +199,20 @@ Scatters the packets being received on specified boundaries to segmented mbufs.
* **[related] API**: ``rte_eth_rx_queue_setup()``, ``rte_eth_buffer_split_get_supported_hdr_ptypes()``.
+.. _nic_features_selective_rx:
+
+Selective Rx
+------------
+
+Discards some segments of buffer split on Rx.
+
+* **[uses] rte_eth_rxconf,rte_eth_rxmode**: ``offloads:RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT``.
+* **[uses] rte_eth_rxconf**: ``rx_seg.mp = NULL`` to discard segments.
+* **[provides] rte_eth_dev_info**: ``rx_offload_capa:RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT``.
+* **[provides] rte_eth_dev_info**: ``rx_seg_capa.selective_rx``.
+* **[related] API**: ``rte_eth_rx_queue_setup()``.
+
+
.. _nic_features_lro:
LRO
diff --git a/doc/guides/nics/features/default.ini b/doc/guides/nics/features/default.ini
index e50514d750..8303a530c1 100644
--- a/doc/guides/nics/features/default.ini
+++ b/doc/guides/nics/features/default.ini
@@ -25,6 +25,7 @@ Burst mode info =
Power mgmt address monitor =
MTU update =
Buffer split on Rx =
+Selective Rx =
Scattered Rx =
LRO =
TSO =
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 8b4f8401e2..d3b77028d6 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -63,6 +63,13 @@ New Features
``rte_eal_init`` and the application is responsible for probing each device,
* ``--auto-probing`` enables the initial bus probing, which is the current default behavior.
+* **Added selective Rx in ethdev API.**
+
+ Some parts of packets may be discarded in Rx
+ by configuring a split of packets received in a queue,
+ and assigning no mempool to some configuration segments.
+ This is a driver capability advertised in the ``selective_rx`` bit.
+
* **Added LinkData sxe2 ethernet driver.**
Added network driver for the LinkData network adapters.
diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
index d0273e3f7b..67d609820c 100644
--- a/lib/ethdev/rte_ethdev.c
+++ b/lib/ethdev/rte_ethdev.c
@@ -2129,7 +2129,7 @@ rte_eth_rx_queue_check_split(uint16_t port_id,
const struct rte_eth_dev_info *dev_info)
{
const struct rte_eth_rxseg_capa *seg_capa = &dev_info->rx_seg_capa;
- struct rte_mempool *mp_first;
+ struct rte_mempool *mp_first = NULL;
uint32_t offset_mask;
uint16_t seg_idx;
int ret = 0;
@@ -2148,7 +2148,6 @@ rte_eth_rx_queue_check_split(uint16_t port_id,
* Check the sizes and offsets against buffer sizes
* for each segment specified in extended configuration.
*/
- mp_first = rx_seg[0].mp;
offset_mask = RTE_BIT32(seg_capa->offset_align_log2) - 1;
ptypes = NULL;
@@ -2160,13 +2159,17 @@ rte_eth_rx_queue_check_split(uint16_t port_id,
uint32_t offset = rx_seg[seg_idx].offset;
uint32_t proto_hdr = rx_seg[seg_idx].proto_hdr;
- if (mpl == NULL) {
- RTE_ETHDEV_LOG_LINE(ERR, "null mempool pointer");
- ret = -EINVAL;
- goto out;
+ if (mpl == NULL) { /* discarded segment */
+ if (seg_capa->selective_rx == 0) { /* not supported */
+ RTE_ETHDEV_LOG_LINE(ERR, "null mempool pointer");
+ ret = -EINVAL;
+ goto out;
+ }
+ continue; /* next checks are not relevant if no mempool */
}
- if (seg_idx != 0 && mp_first != mpl &&
- seg_capa->multi_pools == 0) {
+ if (mp_first == NULL)
+ mp_first = mpl;
+ if (mp_first != mpl && seg_capa->multi_pools == 0) {
RTE_ETHDEV_LOG_LINE(ERR, "Receiving to multiple pools is not supported");
ret = -ENOTSUP;
goto out;
@@ -2233,6 +2236,11 @@ rte_eth_rx_queue_check_split(uint16_t port_id,
if (ret != 0)
goto out;
}
+ if (mp_first == NULL) {
+ RTE_ETHDEV_LOG_LINE(ERR, "At least one Rx segment must have a mempool");
+ ret = -EINVAL;
+ goto out;
+ }
out:
free(ptypes);
return ret;
diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
index 3f4d2438e4..f048b5821d 100644
--- a/lib/ethdev/rte_ethdev.h
+++ b/lib/ethdev/rte_ethdev.h
@@ -1073,6 +1073,7 @@ struct rte_eth_txmode {
* - The first network buffer will be allocated from the memory pool,
* specified in the first array element, the second buffer, from the
* pool in the second element, and so on.
+ * If the pool is NULL, the segment will be discarded, i.e. not received.
*
* - The proto_hdrs in the elements define the split position of
* received packets.
@@ -1121,7 +1122,15 @@ struct rte_eth_txmode {
* The rest will be put into the last valid pool.
*/
struct rte_eth_rxseg_split {
- struct rte_mempool *mp; /**< Memory pool to allocate segment from. */
+ /**
+ * Memory pool to allocate segment from.
+ *
+ * NULL means discarded segment.
+ * Length of discarded segment is not reflected in mbuf packet length
+ * and not accounted in ibytes statistics.
+ * @see rte_eth_rxseg_capa::selective_rx
+ */
+ struct rte_mempool *mp;
uint16_t length; /**< Segment data length, configures split point. */
uint16_t offset; /**< Data offset from beginning of mbuf data buffer. */
/**
@@ -1752,12 +1761,15 @@ struct rte_eth_switch_info {
* @b EXPERIMENTAL: this structure may change without prior notice.
*
* Ethernet device Rx buffer segmentation capabilities.
+ *
+ * @see rte_eth_rxseg_split
*/
struct rte_eth_rxseg_capa {
__extension__
uint32_t multi_pools:1; /**< Supports receiving to multiple pools.*/
uint32_t offset_allowed:1; /**< Supports buffer offsets. */
uint32_t offset_align_log2:4; /**< Required offset alignment. */
+ uint32_t selective_rx:1; /**< Supports discarding segment. */
uint16_t max_nseg; /**< Maximum amount of segments to split. */
uint16_t reserved; /**< Reserved field. */
};
--
2.54.0
^ permalink raw reply related
* [PATCH v4 03/10] app/testpmd: support selective Rx
From: Thomas Monjalon @ 2026-05-29 13:33 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger, Gregory Etelson, Aman Singh
In-Reply-To: <20260529133522.2646044-1-thomas@monjalon.net>
From: Gregory Etelson <getelson@nvidia.com>
Add support for selective Rx using existing rxoffs and rxpkts
command line parameters.
When both rxoffs and rxpkts are specified
on PMDs supporting selective Rx, testpmd automatically:
1. Inserts segments with NULL mempool
for gaps between configured segments to discard unwanted data.
2. Adds a trailing segment with NULL mempool
to cover any remaining data up to the max packet length.
Example usage to receive only Ethernet header and a segment at offset 128:
--rxoffs=0,128 --rxpkts=14,64
This creates segments:
- [0-13]: 14 bytes with mempool (received)
- [14-127]: 114 bytes with NULL mempool (discarded)
- [128-191]: 64 bytes with mempool (received)
- [192-max]: remaining bytes with NULL mempool (discarded)
Note: RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT is required for this feature
and is checked at ethdev API level.
This check is removed from testpmd to allow negative testing of the API.
Signed-off-by: Gregory Etelson <getelson@nvidia.com>
---
app/test-pmd/testpmd.c | 71 ++++++++++++++++++++++-----
doc/guides/testpmd_app_ug/run_app.rst | 20 ++++++++
2 files changed, 80 insertions(+), 11 deletions(-)
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index a9b35f530a..d14341d3ff 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -2731,6 +2731,16 @@ port_is_started(portid_t port_id)
return 1;
}
+static struct rte_eth_rxseg_split *
+next_rx_seg(union rte_eth_rxseg *segs, uint16_t *idx)
+{
+ if (*idx >= MAX_SEGS_BUFFER_SPLIT) {
+ fprintf(stderr, "Too many segments (max %u)\n", MAX_SEGS_BUFFER_SPLIT);
+ return NULL;
+ }
+ return &segs[(*idx)++].split;
+}
+
/* Configure the Rx with optional split. */
int
rx_queue_setup(uint16_t port_id, uint16_t rx_queue_id,
@@ -2744,31 +2754,70 @@ rx_queue_setup(uint16_t port_id, uint16_t rx_queue_id,
uint32_t prev_hdrs = 0;
int ret;
- if ((rx_pkt_nb_segs > 1) &&
- (rx_conf->offloads & RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT)) {
+ if (rx_pkt_nb_segs > 1 || rx_pkt_nb_offs > 0) {
+ struct rte_eth_dev_info dev_info;
+ uint16_t seg_idx = 0;
+ uint16_t next_offset = 0;
+
+ ret = rte_eth_dev_info_get(port_id, &dev_info);
+ if (ret != 0)
+ return ret;
+
/* multi-segment configuration */
for (i = 0; i < rx_pkt_nb_segs; i++) {
- struct rte_eth_rxseg_split *rx_seg = &rx_useg[i].split;
- /*
- * Use last valid pool for the segments with number
- * exceeding the pool index.
- */
+ struct rte_eth_rxseg_split *rx_seg;
+ uint16_t seg_offset;
+
+ if (i < rx_pkt_nb_offs)
+ seg_offset = rx_pkt_seg_offsets[i];
+ else
+ seg_offset = rx_pkt_nb_offs > 0 ? next_offset : 0;
+
+ /* Insert selective Rx discard segment if there's a gap */
+ if (seg_offset > next_offset) {
+ rx_seg = next_rx_seg(rx_useg, &seg_idx);
+ if (rx_seg == NULL)
+ return -EINVAL;
+ rx_seg->offset = next_offset;
+ rx_seg->length = seg_offset - next_offset;
+ rx_seg->mp = NULL;
+ next_offset = seg_offset;
+ }
+
+ rx_seg = next_rx_seg(rx_useg, &seg_idx);
+ if (rx_seg == NULL)
+ return -EINVAL;
mp_n = (i >= mbuf_data_size_n) ? mbuf_data_size_n - 1 : i;
mpx = mbuf_pool_find(socket_id, mp_n);
- /* Handle zero as mbuf data buffer size. */
- rx_seg->offset = i < rx_pkt_nb_offs ?
- rx_pkt_seg_offsets[i] : 0;
+ rx_seg->offset = seg_offset;
rx_seg->mp = mpx ? mpx : mp;
if (rx_pkt_hdr_protos[i] != 0 && rx_pkt_seg_lengths[i] == 0) {
rx_seg->proto_hdr = rx_pkt_hdr_protos[i] & ~prev_hdrs;
prev_hdrs |= rx_seg->proto_hdr;
+ } else if (rx_pkt_nb_offs > 0 && rx_pkt_seg_lengths[i] == 0) {
+ /* Insert fake discard segment if explicitly requested */
+ rx_seg->mp = NULL;
+ rx_seg->length = 0;
} else {
rx_seg->length = rx_pkt_seg_lengths[i] ?
rx_pkt_seg_lengths[i] :
mbuf_data_size[mp_n];
}
+
+ next_offset = seg_offset + rx_seg->length;
}
- rx_conf->rx_nseg = rx_pkt_nb_segs;
+
+ /* Add trailing selective Rx discard segment up to max packet length */
+ if (rx_pkt_nb_offs > 0 && next_offset < dev_info.max_rx_pktlen) {
+ struct rte_eth_rxseg_split *rx_seg = next_rx_seg(rx_useg, &seg_idx);
+ if (rx_seg == NULL)
+ return -EINVAL;
+ rx_seg->offset = next_offset;
+ rx_seg->length = dev_info.max_rx_pktlen - next_offset;
+ rx_seg->mp = NULL;
+ }
+
+ rx_conf->rx_nseg = seg_idx;
rx_conf->rx_seg = rx_useg;
rx_conf->rx_mempools = NULL;
rx_conf->rx_nmempool = 0;
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index 1a4a4b6c12..b59991ed89 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -364,6 +364,11 @@ The command line options are:
feature is engaged. Affects only the queues configured
with split offloads (currently BUFFER_SPLIT is supported only).
+ When used with ``--rxpkts`` on PMDs supporting selective Rx,
+ enables receiving only specific packet segments and discarding the rest.
+ Gaps between configured segments and any trailing data up to the max packet length
+ are automatically filled with NULL mempool segments (data is discarded).
+
* ``--rxpkts=X[,Y]``
Set the length of segments to scatter packets on receiving if split
@@ -373,6 +378,21 @@ The command line options are:
command line parameter and the mbufs to receive will be allocated
sequentially from these extra memory pools.
+ Note: ``--rxoffs`` is required to enable selective Rx in testpmd.
+ To receive only the first N bytes, use ``--rxoffs=0 --rxpkts=N``.
+
+ To receive only the Ethernet header (14 bytes at offset 0) and
+ a 64-byte segment starting at offset 128, while discarding the rest::
+
+ --rxoffs=0,128 --rxpkts=14,64
+
+ This configuration will:
+
+ * Receive 14 bytes at offset 0 (Ethernet header)
+ * Discard bytes 14-127 (inserted NULL mempool segment)
+ * Receive 64 bytes at offset 128
+ * Discard remaining bytes (inserted NULL mempool segment)
+
* ``--txpkts=X[,Y]``
Set TX segment sizes or total packet length. Valid for ``tx-only``
--
2.54.0
^ permalink raw reply related
* [PATCH v4 04/10] common/mlx5: add null MR functions
From: Thomas Monjalon @ 2026-05-29 13:33 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, Gregory Etelson, Dariusz Sosnowski,
Viacheslav Ovsiienko, Bing Zhao, Ori Kam, Suanming Mou,
Matan Azrad
In-Reply-To: <20260529133522.2646044-1-thomas@monjalon.net>
From: Gregory Etelson <getelson@nvidia.com>
Add functions to allocate and free a null Memory Region (MR)
using ibverbs on Linux.
There is no implementation for DevX on Windows.
Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
drivers/common/mlx5/linux/mlx5_common_verbs.c | 35 +++++++++++++++++++
drivers/common/mlx5/mlx5_common_mr.h | 9 +++++
drivers/common/mlx5/windows/mlx5_common_os.c | 16 +++++++++
3 files changed, 60 insertions(+)
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 2322d9d033..6d44e1f566 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -161,3 +161,38 @@ mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
*reg_mr_cb = mlx5_common_verbs_reg_mr;
*dereg_mr_cb = mlx5_common_verbs_dereg_mr;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_alloc_null_mr)
+struct mlx5_pmd_mr *
+mlx5_os_alloc_null_mr(struct rte_device *dev, void *pd)
+{
+ struct ibv_mr *ibv_mr;
+ struct mlx5_pmd_mr *null_mr;
+
+ null_mr = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*null_mr), 0, dev->numa_node);
+ if (!null_mr)
+ return NULL;
+ ibv_mr = mlx5_glue->alloc_null_mr(pd);
+ if (!ibv_mr) {
+ mlx5_free(null_mr);
+ return NULL;
+ }
+ *null_mr = (struct mlx5_pmd_mr) {
+ .lkey = rte_cpu_to_be_32(ibv_mr->lkey),
+ .addr = ibv_mr->addr,
+ .len = ibv_mr->length,
+ .obj = (void *)ibv_mr,
+ };
+ return null_mr;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_free_null_mr)
+void
+mlx5_os_free_null_mr(struct mlx5_pmd_mr *null_mr)
+{
+ if (!null_mr)
+ return;
+ if (null_mr->obj)
+ claim_zero(mlx5_glue->dereg_mr(null_mr->obj));
+ mlx5_free(null_mr);
+}
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index cf7c685e9b..00f3d832c3 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -21,6 +21,8 @@
#include "mlx5_common_mp.h"
#include "mlx5_common_defs.h"
+struct rte_device;
+
/* mlx5 PMD MR struct. */
struct mlx5_pmd_mr {
uint32_t lkey;
@@ -258,6 +260,13 @@ __rte_internal
void
mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
+__rte_internal
+struct mlx5_pmd_mr *
+mlx5_os_alloc_null_mr(struct rte_device *dev, void *pd);
+__rte_internal
+void
+mlx5_os_free_null_mr(struct mlx5_pmd_mr *null_mr);
+
__rte_internal
int
mlx5_mr_mempool_register(struct mlx5_common_device *cdev,
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index 16fcc5f9fc..692517a9bf 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -454,6 +454,22 @@ mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
*dereg_mr_cb = mlx5_os_dereg_mr;
}
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_alloc_null_mr)
+struct mlx5_pmd_mr *
+mlx5_os_alloc_null_mr(struct rte_device *dev, void *pd)
+{
+ RTE_SET_USED(dev);
+ RTE_SET_USED(pd);
+ return NULL;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_free_null_mr)
+void
+mlx5_os_free_null_mr(struct mlx5_pmd_mr *null_mr)
+{
+ RTE_SET_USED(null_mr);
+}
+
/*
* In Windows, no need to wrap the MR, no known issue for it in kernel.
* Use the regular function to create direct MR.
--
2.54.0
^ permalink raw reply related
* [PATCH v4 05/10] net/mlx5: fix Rx split segment counter type
From: Thomas Monjalon @ 2026-05-29 13:33 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, stable, Dariusz Sosnowski,
Viacheslav Ovsiienko, Bing Zhao, Ori Kam, Suanming Mou,
Matan Azrad
In-Reply-To: <20260529133522.2646044-1-thomas@monjalon.net>
In the API, rx_nseg and max_nseg are uint16_t.
In mlx5, MLX5_MAX_RXQ_NSEG is 32.
So there is no reason to have rxseg_n as uint32_t.
Reduce the fields to uint16_t and move them to avoid struct holes.
Fixes: 9f209b59c8b0 ("net/mlx5: support Rx buffer split description")
Fixes: 572c9d4bda08 ("net/mlx5: fix shared Rx queue segment configuration match")
Cc: stable@dpdk.org
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
drivers/net/mlx5/mlx5_rx.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index dffab3955b..01b563d981 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -164,9 +164,9 @@ struct __rte_cache_aligned mlx5_rxq_data {
uint64_t flow_meta_mask;
int32_t flow_meta_offset;
uint32_t flow_meta_port_mask;
- uint32_t rxseg_n; /* Number of split segment descriptions. */
struct mlx5_eth_rxseg rxseg[MLX5_MAX_RXQ_NSEG];
/* Buffer split segment descriptions - sizes, offsets, pools. */
+ uint16_t rxseg_n; /* Number of split segment descriptions. */
uint16_t rq_win_cnt; /* Number of packets in the sliding window data. */
uint16_t rq_win_idx_mask; /* Sliding window index wrapping mask. */
uint16_t rq_win_idx; /* Index of the first element in sliding window. */
@@ -191,9 +191,9 @@ struct mlx5_rxq_ctrl {
unsigned int irq:1; /* Whether IRQ is enabled. */
uint32_t flow_tunnels_n[MLX5_FLOW_TUNNEL]; /* Tunnels counters. */
uint32_t wqn; /* WQ number. */
- uint32_t rxseg_n; /* Number of split segment descriptions. */
struct rte_eth_rxseg_split rxseg[MLX5_MAX_RXQ_NSEG];
/* Saved original buffer split segment configuration. */
+ uint16_t rxseg_n; /* Number of split segment descriptions. */
uint16_t dump_file_n; /* Number of dump files. */
};
--
2.54.0
^ permalink raw reply related
* [PATCH v4 06/10] net/mlx5: support selective Rx
From: Thomas Monjalon @ 2026-05-29 13:34 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, Gregory Etelson, Dariusz Sosnowski,
Viacheslav Ovsiienko, Bing Zhao, Ori Kam, Suanming Mou,
Matan Azrad
In-Reply-To: <20260529133522.2646044-1-thomas@monjalon.net>
From: Gregory Etelson <getelson@nvidia.com>
Selective Rx may save some PCI bandwidth.
Implement selective Rx in the (quite slow) scalar SPRQ Rx path
mlx5_rx_burst() where the performance impact
of the added condition branches is acceptable.
Other Rx functions do not support this feature.
When using selective Rx, mlx5_rx_burst will be selected.
A null Memory Region (MR) is always allocated
at shared device context initialization.
The selective Rx capability is not advertised
if this special MR allocation fails.
For each Rx segment configured with a NULL mempool,
a "null mbuf" is created.
It is a fake mbuf allocated outside any mempool,
used as a placeholder in the Rx ring.
The null MR lkey is used in the WQE for these segments
so the NIC writes received data to a discard buffer.
The mbuf data room size is resolved from the first segment having a pool.
For null segments, the buffer length is from the last seen pool,
so that the WQE stride size remains consistent.
In mlx5_rx_burst, discarded segments are not chained
into the packet mbuf list, NB_SEGS is decremented accordingly,
and no replacement buffer is allocated.
A separate data_seg_len accumulator tracks the total length
of delivered segments only.
The packet length is adjusted to reflect only the data
actually delivered to the application.
Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
doc/guides/nics/features/mlx5.ini | 1 +
doc/guides/nics/mlx5.rst | 86 +++++++++++++++++++-------
doc/guides/rel_notes/release_26_07.rst | 4 ++
drivers/net/mlx5/mlx5.c | 7 +++
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_ethdev.c | 25 ++++++++
drivers/net/mlx5/mlx5_rx.c | 26 +++++++-
drivers/net/mlx5/mlx5_rx.h | 1 +
drivers/net/mlx5/mlx5_rxq.c | 45 ++++++++++++--
drivers/net/mlx5/mlx5_trigger.c | 52 +++++++++++++---
10 files changed, 211 insertions(+), 37 deletions(-)
diff --git a/doc/guides/nics/features/mlx5.ini b/doc/guides/nics/features/mlx5.ini
index 3b3eda28b8..ae8c83057b 100644
--- a/doc/guides/nics/features/mlx5.ini
+++ b/doc/guides/nics/features/mlx5.ini
@@ -16,6 +16,7 @@ Burst mode info = Y
Power mgmt address monitor = Y
MTU update = Y
Buffer split on Rx = Y
+Selective Rx = Y
Scattered Rx = Y
LRO = Y
TSO = Y
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 137fbaf889..b4455f3e7d 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -84,6 +84,9 @@ The Rx / Tx data path use different techniques to offer the best performance.
with :ref:`multi-packet Rx queues (MPRQ) <mlx5_mprq_params>`.
This feature is disabled by default.
+- Some PCI bandwidth is saved by receiving partial packets
+ with :ref:`selective Rx <mlx5_selective_rx>`.
+
More details about Rx implementations and their configurations are provided
in the chapter about :ref:`mlx5_rx_functions`.
@@ -879,6 +882,8 @@ MLX5 supports various methods to report statistics:
Basic port statistics can be queried using ``rte_eth_stats_get()``.
The received and sent statistics are through SW only
and counts the number of packets received or sent successfully by the PMD.
+In the case of :ref:`selective Rx <mlx5_selective_rx>`,
+the ``ibytes`` counter matches segments delivered, not the skipped ones.
The ``imissed`` counter is the amount of packets that could not be delivered
to SW because a queue was full.
Packets not received due to congestion in the bus or on the NIC
@@ -992,25 +997,26 @@ These configurations may also have an impact on the behavior:
.. table:: Rx burst functions
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- || Function Name || Parameters to Enable || Scatter|| Error Recovery || CQE || Large|| Shared |
- | | | | || comp|| MTU | RxQ |
- +===================+========================+=========+=================+======+=======+=========+
- | rx_burst | rx_vec_en=0 | Yes | Yes | Yes | Yes | No |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- | rx_burst_vec | rx_vec_en=1 (default) | No | if CQE comp off | Yes | No | No |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- | rx_burst_mprq || mprq_en=1 | No | Yes | Yes | Yes | No |
- | || RxQs >= rxqs_min_mprq | | | | | |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- | rx_burst_mprq_vec || rx_vec_en=1 (default) | No | if CQE comp off | Yes | Yes | No |
- | || mprq_en=1 | | | | | |
- | || RxQs >= rxqs_min_mprq | | | | | |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- | rx_burst | at least one Rx queue | Yes | Yes | Yes | Yes | Yes |
- | (out of order) | on the device | | | | | |
- | | is shared | | | | | |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ || Function|| Parameters to Enable || Scatter|| Selec-|| Error || CQE || Large|| Shared|
+ || Name | | || tive || Recovery|| comp|| MTU || RxQ |
+ +==========+=======================+=========+========+==========+======+=======+========+
+ | rx_burst | rx_vec_en=0 | Yes | Yes | Yes | Yes | Yes | No |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ | _vec | rx_vec_en=1 (default) | No | No || if CQE | Yes | No | No |
+ | | | | || comp off| | | |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ | _mprq || mprq_en=1 | No | No | Yes | Yes | Yes | No |
+ | || RxQs >= rxqs_min_mprq| | | | | | |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ | _mprq_vec|| rx_vec_en=1 (default)| No | No || if CQE | Yes | Yes | No |
+ | || mprq_en=1 | | || comp off| | | |
+ | || RxQs >= rxqs_min_mprq| | | | | | |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ || _out_of || at least one Rx queue| Yes | No | Yes | Yes | Yes | Yes |
+ || _order || on the device | | | | | | |
+ | || is shared | | | | | | |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
Rx/Tx Tuning
@@ -1105,13 +1111,14 @@ Rx interrupt X
:ref:`Rx threshold <mlx5_rx_threshold>` X X
:ref:`Rx drop delay <mlx5_drop>` X X
:ref:`Rx timestamp <mlx5_rx_timstp>` X X
+:ref:`buffer split <mlx5_buf_split>` X X
+:ref:`selective Rx <mlx5_selective_rx>` X
+:ref:`multi-segment <mlx5_multiseg>` X X
:ref:`Tx scheduling <mlx5_tx_sched>` X
:ref:`Tx rate limit <mlx5_rate_limit>` X
:ref:`Tx inline <mlx5_tx_inline>` X X
:ref:`Tx fast free <mlx5_tx_fast_free>` X X
:ref:`Tx affinity <mlx5_aggregated>` X
-:ref:`buffer split <mlx5_buf_split>` X X
-:ref:`multi-segment <mlx5_multiseg>` X X
promiscuous X X
multicast promiscuous X X
multiple MAC addresses X
@@ -2248,13 +2255,50 @@ OFED 5.1-2
DPDK 20.11
========= ==========
+Runtime configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+The offload flag ``RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT`` is required.
+
+When calling ``rte_eth_rx_queue_setup()``,
+the input ``rte_eth_rxconf::rx_seg`` defines the configuration of the segments,
+mainly offset and length.
+
Limitations
^^^^^^^^^^^
+#. Splitting per protocol header is not supported.
+
#. Buffer split offload is supported with regular Rx burst routine only,
no MPRQ feature or vectorized code can be engaged.
+.. _mlx5_selective_rx:
+
+Selective Rx
+~~~~~~~~~~~~
+
+Some PCI bandwidth can be saved
+by :ref:`skipping some parts of Rx data <nic_features_selective_rx>`.
+It is enabled when using :ref:`buffer split <mlx5_buf_split>`
+and configuring no mempool in some segments to discard.
+
+Runtime configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+The offload flag ``RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT`` is required.
+
+When calling ``rte_eth_rx_queue_setup()``,
+the segment to discard (``rte_eth_rxconf::rx_seg::split``)
+is marked by the absence of mempool (``mp = NULL``).
+
+Limitations
+^^^^^^^^^^^
+
+#. Selective Rx is supported with regular Rx burst routine only,
+ no MPRQ feature or vectorized code can be engaged.
+
+
.. _mlx5_multiseg:
Multi-Segment Scatter/Gather
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index d3b77028d6..ca2da106e8 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -79,6 +79,10 @@ New Features
* Added support for transmitting LLDP packets based on mbuf packet type.
* Implemented AVX2 context descriptor transmit paths.
+* **Updated NVIDIA mlx5 ethernet driver.**
+
+ * Added support for selective Rx in scalar SPRQ Rx path.
+
* **Updated PCAP ethernet driver.**
* Added support for VLAN insertion and stripping.
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index f190654756..61c26d1206 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1975,6 +1975,9 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
/* Init counter pool list header and lock. */
LIST_INIT(&sh->hws_cpool_list);
rte_spinlock_init(&sh->cpool_lock);
+ sh->null_mr = mlx5_os_alloc_null_mr(sh->cdev->dev, sh->cdev->pd);
+ if (!sh->null_mr)
+ DRV_LOG(DEBUG, "Fail to initialize NULL MR, selective Rx is disabled.");
exit:
pthread_mutex_unlock(&mlx5_dev_ctx_list_mutex);
return sh;
@@ -2139,6 +2142,10 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
MLX5_ASSERT(sh->geneve_tlv_option_resource == NULL);
pthread_mutex_destroy(&sh->txpp.mutex);
mlx5_lwm_unset(sh);
+ if (sh->null_mr) {
+ mlx5_os_free_null_mr(sh->null_mr);
+ sh->null_mr = NULL;
+ }
mlx5_physical_device_destroy(sh->phdev);
mlx5_free(sh);
return;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 92a00cfaa8..bd6ef35b53 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1674,6 +1674,7 @@ struct mlx5_dev_ctx_shared {
rte_spinlock_t cpool_lock;
LIST_HEAD(hws_cpool_list, mlx5_hws_cnt_pool) hws_cpool_list; /* Count pool list. */
struct mlx5_dev_registers registers;
+ struct mlx5_pmd_mr *null_mr;
struct mlx5_dev_shared_port port[]; /* per device port data array. */
};
diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c
index a29cdeeb50..7b7536fa1e 100644
--- a/drivers/net/mlx5/mlx5_ethdev.c
+++ b/drivers/net/mlx5/mlx5_ethdev.c
@@ -381,6 +381,7 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)
info->rx_seg_capa.multi_pools = !priv->config.mprq.enabled;
info->rx_seg_capa.offset_allowed = !priv->config.mprq.enabled;
info->rx_seg_capa.offset_align_log2 = 0;
+ info->rx_seg_capa.selective_rx = !!priv->sh->null_mr;
info->rx_offload_capa = (mlx5_get_rx_port_offloads() |
info->rx_queue_offload_capa);
info->tx_offload_capa = mlx5_get_tx_port_offloads(dev);
@@ -708,6 +709,25 @@ mlx5_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
return -rte_errno;
}
+static bool
+mlx5_selective_rx_enabled(struct rte_eth_dev *dev)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ for (uint32_t q = 0; q < priv->rxqs_n; ++q) {
+ struct mlx5_rxq_ctrl *rxq_ctrl = mlx5_rxq_ctrl_get(dev, q);
+
+ if (rxq_ctrl == NULL || rxq_ctrl->is_hairpin)
+ continue;
+ for (uint16_t s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
+ if (rxq_ctrl->rxq.rxseg[s].mp == NULL)
+ return true;
+ }
+ }
+
+ return false;
+}
+
/**
* Configure the RX function to use.
*
@@ -723,6 +743,11 @@ mlx5_select_rx_function(struct rte_eth_dev *dev)
eth_rx_burst_t rx_pkt_burst = mlx5_rx_burst;
MLX5_ASSERT(dev != NULL);
+ if (mlx5_selective_rx_enabled(dev)) {
+ DRV_LOG(DEBUG, "port %u forced to scalar SPRQ Rx (selective Rx configured)",
+ dev->data->port_id);
+ return rx_pkt_burst;
+ }
if (mlx5_shared_rq_enabled(dev)) {
rx_pkt_burst = mlx5_rx_burst_out_of_order;
DRV_LOG(DEBUG, "port %u forced to use SPRQ"
diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c
index 185bfd4fff..586bf6c935 100644
--- a/drivers/net/mlx5/mlx5_rx.c
+++ b/drivers/net/mlx5/mlx5_rx.c
@@ -486,7 +486,7 @@ mlx5_rxq_initialize(struct mlx5_rxq_data *rxq)
rxq->wqes)[i];
addr = rte_pktmbuf_mtod(buf, uintptr_t);
byte_count = DATA_LEN(buf);
- lkey = mlx5_rx_mb2mr(rxq, buf);
+ lkey = buf->pool ? mlx5_rx_mb2mr(rxq, buf) : rxq->sh->null_mr->lkey;
}
/* scat->addr must be able to store a pointer. */
MLX5_ASSERT(sizeof(scat->addr) >= sizeof(uintptr_t));
@@ -1044,11 +1044,13 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
const unsigned int sges_n = rxq->sges_n;
struct rte_mbuf *pkt = NULL;
struct rte_mbuf *seg = NULL;
+ struct rte_mbuf *tail = NULL;
volatile struct mlx5_cqe *cqe =
&(*rxq->cqes)[rxq->cq_ci & cqe_mask];
unsigned int i = 0;
unsigned int rq_ci = rxq->rq_ci << sges_n;
int len = 0; /* keep its value across iterations. */
+ uint32_t data_seg_len = 0;
while (pkts_n) {
uint16_t skip_cnt;
@@ -1058,12 +1060,17 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
struct rte_mbuf *rep = (*rxq->elts)[idx];
volatile struct mlx5_mini_cqe8 *mcqe = NULL;
- if (pkt)
- NEXT(seg) = rep;
+ if (pkt) {
+ if (rep->pool)
+ NEXT(tail) = rep;
+ else
+ --NB_SEGS(pkt);
+ }
seg = rep;
rte_prefetch0(seg);
rte_prefetch0(cqe);
rte_prefetch0(wqe);
+ if (seg->pool) {
/* Allocate the buf from the same pool. */
rep = rte_mbuf_raw_alloc(seg->pool);
if (unlikely(rep == NULL)) {
@@ -1088,12 +1095,14 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
rq_ci <<= sges_n;
break;
}
+ }
if (!pkt) {
cqe = &(*rxq->cqes)[rxq->cq_ci & cqe_mask];
len = mlx5_rx_poll_len(rxq, cqe, cqe_n, cqe_mask,
&mcqe, &skip_cnt, false, NULL);
if (unlikely(len & MLX5_ERROR_CQE_MASK)) {
/* We drop packets with non-critical errors */
+ if (seg->pool)
rte_mbuf_raw_free(rep);
if (len == MLX5_CRITICAL_ERROR_CQE_RET) {
rq_ci = rxq->rq_ci << sges_n;
@@ -1107,6 +1116,7 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
continue;
}
if (len == 0) {
+ if (seg->pool)
rte_mbuf_raw_free(rep);
break;
}
@@ -1127,6 +1137,8 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
pkt->tso_segsz = len / cqe->lro_num_seg;
}
}
+ if (seg->pool) {
+ tail = seg;
DATA_LEN(rep) = DATA_LEN(seg);
PKT_LEN(rep) = PKT_LEN(seg);
SET_DATA_OFF(rep, DATA_OFF(seg));
@@ -1141,17 +1153,25 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
/* If there's only one MR, no need to replace LKey in WQE. */
if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1))
wqe->lkey = mlx5_rx_mb2mr(rxq, rep);
+ }
if (len > DATA_LEN(seg)) {
+ if (seg->pool)
+ data_seg_len += DATA_LEN(seg);
len -= DATA_LEN(seg);
++NB_SEGS(pkt);
++rq_ci;
continue;
}
+ if (seg->pool) {
DATA_LEN(seg) = len;
+ data_seg_len += len;
+ }
+ PKT_LEN(pkt) = RTE_MIN(PKT_LEN(pkt), data_seg_len);
#ifdef MLX5_PMD_SOFT_COUNTERS
/* Increment bytes counter. */
rxq->stats.ibytes += PKT_LEN(pkt);
#endif
+ data_seg_len = 0;
/* Return packet. */
*(pkts++) = pkt;
pkt = NULL;
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 01b563d981..cd48ee37ef 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -96,6 +96,7 @@ struct mlx5_eth_rxseg {
uint16_t length; /**< Segment data length, configures split point. */
uint16_t offset; /**< Data offset from beginning of mbuf data buffer. */
uint32_t reserved; /**< Reserved field. */
+ struct rte_mbuf *null_mbuf; /**< For selective Rx. */
};
/* RX queue descriptor. */
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 48d982a8c2..3fae189fa4 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -151,6 +151,7 @@ rxq_alloc_elts_sprq(struct mlx5_rxq_ctrl *rxq_ctrl)
struct mlx5_eth_rxseg *seg = &rxq_ctrl->rxq.rxseg[i % sges_n];
struct rte_mbuf *buf;
+ if (seg->mp) {
buf = rte_pktmbuf_alloc(seg->mp);
if (buf == NULL) {
if (rxq_ctrl->share_group == 0)
@@ -167,6 +168,9 @@ rxq_alloc_elts_sprq(struct mlx5_rxq_ctrl *rxq_ctrl)
/* Only vectored Rx routines rely on headroom size. */
MLX5_ASSERT(!has_vec_support ||
DATA_OFF(buf) >= RTE_PKTMBUF_HEADROOM);
+ } else {
+ buf = seg->null_mbuf;
+ }
/* Buffer is supposed to be empty. */
MLX5_ASSERT(rte_pktmbuf_data_len(buf) == 0);
MLX5_ASSERT(rte_pktmbuf_pkt_len(buf) == 0);
@@ -324,10 +328,14 @@ rxq_free_elts_sprq(struct mlx5_rxq_ctrl *rxq_ctrl)
rxq->rq_pi = elts_ci;
}
for (i = 0; i != q_n; ++i) {
- if ((*rxq->elts)[i] != NULL)
+ if ((*rxq->elts)[i] != NULL && (*rxq->elts)[i]->pool != NULL)
rte_pktmbuf_free_seg((*rxq->elts)[i]);
(*rxq->elts)[i] = NULL;
}
+ for (i = 0; i < rxq->rxseg_n; i++) {
+ mlx5_free(rxq->rxseg[i].null_mbuf);
+ rxq->rxseg[i].null_mbuf = NULL;
+ }
}
/**
@@ -1815,7 +1823,9 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
int ret;
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_rxq_ctrl *tmpl;
- unsigned int mb_len = rte_pktmbuf_data_room_size(rx_seg[0].mp);
+ struct rte_mempool *first_mp = NULL;
+ struct rte_mempool *last_mp = NULL;
+ unsigned int mb_len;
struct mlx5_port_config *config = &priv->config;
uint64_t offloads = conf->offloads |
dev->data->dev_conf.rxmode.offloads;
@@ -1827,7 +1837,7 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
unsigned int non_scatter_min_mbuf_size = max_rx_pktlen +
RTE_PKTMBUF_HEADROOM;
unsigned int max_lro_size = 0;
- unsigned int first_mb_free_size = mb_len - RTE_PKTMBUF_HEADROOM;
+ unsigned int first_mb_free_size;
uint32_t mprq_log_actual_stride_num = 0;
uint32_t mprq_log_actual_stride_size = 0;
bool rx_seg_en = n_seg != 1 || rx_seg[0].offset || rx_seg[0].length;
@@ -1845,6 +1855,21 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
const struct rte_eth_rxseg_split *qs_seg = rx_seg;
unsigned int tail_len;
+ /* Find first segment with a mempool. */
+ for (uint16_t seg = 0; seg < n_seg; seg++) {
+ if (rx_seg[seg].mp != NULL) {
+ first_mp = rx_seg[seg].mp;
+ break;
+ }
+ }
+ if (first_mp == NULL) {
+ DRV_LOG(ERR, "port %u Rx queue %u has no mempool", dev->data->port_id, idx);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ mb_len = rte_pktmbuf_data_room_size(first_mp);
+ first_mb_free_size = mb_len - RTE_PKTMBUF_HEADROOM;
+
if (mprq_en) {
/* Trim the number of descs needed. */
desc >>= mprq_log_actual_stride_num;
@@ -1884,13 +1909,20 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
do {
struct mlx5_eth_rxseg *hw_seg =
&tmpl->rxq.rxseg[tmpl->rxq.rxseg_n];
- uint32_t buf_len, offset, seg_len;
+ uint32_t buf_len = 0, offset, seg_len;
/*
* For the buffers beyond descriptions offset is zero,
* the first buffer contains head room.
*/
- buf_len = rte_pktmbuf_data_room_size(qs_seg->mp);
+ if (qs_seg->mp != NULL) {
+ last_mp = qs_seg->mp;
+ buf_len = rte_pktmbuf_data_room_size(qs_seg->mp);
+ } else if (last_mp != NULL) {
+ buf_len = rte_pktmbuf_data_room_size(last_mp);
+ } else {
+ buf_len = mb_len;
+ }
offset = (tmpl->rxq.rxseg_n >= n_seg ? 0 : qs_seg->offset) +
(tmpl->rxq.rxseg_n ? 0 : RTE_PKTMBUF_HEADROOM);
/*
@@ -2077,7 +2109,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
/* Save port ID. */
tmpl->rxq.port_id = dev->data->port_id;
tmpl->sh = priv->sh;
- tmpl->rxq.mp = rx_seg[0].mp;
+ tmpl->rxq.sh = priv->sh;
+ tmpl->rxq.mp = first_mp;
tmpl->rxq.elts_n = log2above(desc);
tmpl->rxq.rq_repl_thresh = MLX5_VPMD_RXQ_RPLNSH_THRESH(desc_n);
tmpl->rxq.elts = (struct rte_mbuf *(*)[])(tmpl + 1);
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index a070aaecfd..5b04d9a234 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -116,6 +116,27 @@ mlx5_txq_start(struct rte_eth_dev *dev)
return -rte_errno;
}
+static struct rte_mbuf *
+mlx5_alloc_null_mbuf(uint32_t data_len)
+{
+ size_t alloc_size = sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM +
+ rte_align32pow2(data_len);
+ struct rte_mbuf *m;
+
+ m = mlx5_malloc(MLX5_MEM_ZERO, alloc_size, 0, SOCKET_ID_ANY);
+ if (m == NULL)
+ return NULL;
+ m->buf_addr = RTE_PTR_ADD(m, sizeof(*m));
+ m->buf_len = alloc_size - sizeof(*m);
+ rte_mbuf_iova_set(m, rte_mem_virt2iova(m->buf_addr));
+ m->data_off = RTE_PKTMBUF_HEADROOM;
+ m->refcnt = 1;
+ m->nb_segs = 1;
+ m->port = RTE_MBUF_PORT_INVALID;
+ m->pool = NULL;
+ return m;
+}
+
/**
* Register Rx queue mempools and fill the Rx queue cache.
* This function tolerates repeated mempool registration.
@@ -130,7 +151,8 @@ static int
mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
{
struct rte_mempool *mp;
- uint32_t s;
+ struct mlx5_eth_rxseg *seg;
+ uint16_t s;
int ret = 0;
mlx5_mr_flush_local_cache(&rxq_ctrl->rxq.mr_ctrl);
@@ -139,21 +161,37 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
return mlx5_mr_mempool_populate_cache(&rxq_ctrl->rxq.mr_ctrl,
rxq_ctrl->rxq.mprq_mp);
for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
- bool is_extmem;
-
- mp = rxq_ctrl->rxq.rxseg[s].mp;
- is_extmem = (rte_pktmbuf_priv_flags(mp) &
+ seg = &rxq_ctrl->rxq.rxseg[s];
+ mp = seg->mp;
+ if (mp) { /* Regular segment */
+ bool is_extmem = (rte_pktmbuf_priv_flags(mp) &
RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) != 0;
ret = mlx5_mr_mempool_register(rxq_ctrl->sh->cdev, mp,
is_extmem);
if (ret < 0 && rte_errno != EEXIST)
- return ret;
+ goto error;
ret = mlx5_mr_mempool_populate_cache(&rxq_ctrl->rxq.mr_ctrl,
mp);
if (ret < 0)
- return ret;
+ goto error;
+ } else { /* NULL segment used in selective Rx */
+ seg->null_mbuf = mlx5_alloc_null_mbuf(seg->length);
+ if (seg->null_mbuf == NULL) {
+ rte_errno = ENOMEM;
+ ret = -rte_errno;
+ goto error;
+ }
+ }
}
return 0;
+
+error:
+ while (s-- > 0) {
+ seg = &rxq_ctrl->rxq.rxseg[s];
+ mlx5_free(seg->null_mbuf);
+ seg->null_mbuf = NULL;
+ }
+ return ret;
}
/**
--
2.54.0
^ permalink raw reply related
* [PATCH v4 07/10] net/mlx5: reindent previous changes
From: Thomas Monjalon @ 2026-05-29 13:34 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, Dariusz Sosnowski, Viacheslav Ovsiienko,
Bing Zhao, Ori Kam, Suanming Mou, Matan Azrad
In-Reply-To: <20260529133522.2646044-1-thomas@monjalon.net>
Fix indent which was left untouched to help reviews.
This must be squashed before merging.
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
drivers/net/mlx5/mlx5_rx.c | 82 ++++++++++++++++-----------------
drivers/net/mlx5/mlx5_rxq.c | 32 ++++++-------
drivers/net/mlx5/mlx5_trigger.c | 18 ++++----
3 files changed, 65 insertions(+), 67 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c
index 586bf6c935..ad40511422 100644
--- a/drivers/net/mlx5/mlx5_rx.c
+++ b/drivers/net/mlx5/mlx5_rx.c
@@ -1071,30 +1071,30 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
rte_prefetch0(cqe);
rte_prefetch0(wqe);
if (seg->pool) {
- /* Allocate the buf from the same pool. */
- rep = rte_mbuf_raw_alloc(seg->pool);
- if (unlikely(rep == NULL)) {
- ++rxq->stats.rx_nombuf;
- if (!pkt) {
- /*
- * no buffers before we even started,
- * bail out silently.
- */
+ /* Allocate the buf from the same pool. */
+ rep = rte_mbuf_raw_alloc(seg->pool);
+ if (unlikely(rep == NULL)) {
+ ++rxq->stats.rx_nombuf;
+ if (!pkt) {
+ /*
+ * no buffers before we even started,
+ * bail out silently.
+ */
+ break;
+ }
+ while (pkt != seg) {
+ MLX5_ASSERT(pkt != (*rxq->elts)[idx]);
+ rep = NEXT(pkt);
+ NEXT(pkt) = NULL;
+ NB_SEGS(pkt) = 1;
+ rte_mbuf_raw_free(pkt);
+ pkt = rep;
+ }
+ rq_ci >>= sges_n;
+ ++rq_ci;
+ rq_ci <<= sges_n;
break;
}
- while (pkt != seg) {
- MLX5_ASSERT(pkt != (*rxq->elts)[idx]);
- rep = NEXT(pkt);
- NEXT(pkt) = NULL;
- NB_SEGS(pkt) = 1;
- rte_mbuf_raw_free(pkt);
- pkt = rep;
- }
- rq_ci >>= sges_n;
- ++rq_ci;
- rq_ci <<= sges_n;
- break;
- }
}
if (!pkt) {
cqe = &(*rxq->cqes)[rxq->cq_ci & cqe_mask];
@@ -1103,7 +1103,7 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
if (unlikely(len & MLX5_ERROR_CQE_MASK)) {
/* We drop packets with non-critical errors */
if (seg->pool)
- rte_mbuf_raw_free(rep);
+ rte_mbuf_raw_free(rep);
if (len == MLX5_CRITICAL_ERROR_CQE_RET) {
rq_ci = rxq->rq_ci << sges_n;
break;
@@ -1117,7 +1117,7 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
}
if (len == 0) {
if (seg->pool)
- rte_mbuf_raw_free(rep);
+ rte_mbuf_raw_free(rep);
break;
}
pkt = seg;
@@ -1138,21 +1138,21 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
}
}
if (seg->pool) {
- tail = seg;
- DATA_LEN(rep) = DATA_LEN(seg);
- PKT_LEN(rep) = PKT_LEN(seg);
- SET_DATA_OFF(rep, DATA_OFF(seg));
- PORT(rep) = PORT(seg);
- (*rxq->elts)[idx] = rep;
- /*
- * Fill NIC descriptor with the new buffer. The lkey and size
- * of the buffers are already known, only the buffer address
- * changes.
- */
- wqe->addr = rte_cpu_to_be_64(rte_pktmbuf_mtod(rep, uintptr_t));
- /* If there's only one MR, no need to replace LKey in WQE. */
- if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1))
- wqe->lkey = mlx5_rx_mb2mr(rxq, rep);
+ tail = seg;
+ DATA_LEN(rep) = DATA_LEN(seg);
+ PKT_LEN(rep) = PKT_LEN(seg);
+ SET_DATA_OFF(rep, DATA_OFF(seg));
+ PORT(rep) = PORT(seg);
+ (*rxq->elts)[idx] = rep;
+ /*
+ * Fill NIC descriptor with the new buffer. The lkey and size
+ * of the buffers are already known, only the buffer address
+ * changes.
+ */
+ wqe->addr = rte_cpu_to_be_64(rte_pktmbuf_mtod(rep, uintptr_t));
+ /* If there's only one MR, no need to replace LKey in WQE. */
+ if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1))
+ wqe->lkey = mlx5_rx_mb2mr(rxq, rep);
}
if (len > DATA_LEN(seg)) {
if (seg->pool)
@@ -1163,8 +1163,8 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
continue;
}
if (seg->pool) {
- DATA_LEN(seg) = len;
- data_seg_len += len;
+ DATA_LEN(seg) = len;
+ data_seg_len += len;
}
PKT_LEN(pkt) = RTE_MIN(PKT_LEN(pkt), data_seg_len);
#ifdef MLX5_PMD_SOFT_COUNTERS
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 3fae189fa4..6ca29f7543 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -152,22 +152,22 @@ rxq_alloc_elts_sprq(struct mlx5_rxq_ctrl *rxq_ctrl)
struct rte_mbuf *buf;
if (seg->mp) {
- buf = rte_pktmbuf_alloc(seg->mp);
- if (buf == NULL) {
- if (rxq_ctrl->share_group == 0)
- DRV_LOG(ERR, "port %u queue %u empty mbuf pool",
- RXQ_PORT_ID(rxq_ctrl),
- rxq_ctrl->rxq.idx);
- else
- DRV_LOG(ERR, "share group %u queue %u empty mbuf pool",
- rxq_ctrl->share_group,
- rxq_ctrl->share_qid);
- rte_errno = ENOMEM;
- goto error;
- }
- /* Only vectored Rx routines rely on headroom size. */
- MLX5_ASSERT(!has_vec_support ||
- DATA_OFF(buf) >= RTE_PKTMBUF_HEADROOM);
+ buf = rte_pktmbuf_alloc(seg->mp);
+ if (buf == NULL) {
+ if (rxq_ctrl->share_group == 0)
+ DRV_LOG(ERR, "port %u queue %u empty mbuf pool",
+ RXQ_PORT_ID(rxq_ctrl),
+ rxq_ctrl->rxq.idx);
+ else
+ DRV_LOG(ERR, "share group %u queue %u empty mbuf pool",
+ rxq_ctrl->share_group,
+ rxq_ctrl->share_qid);
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ /* Only vectored Rx routines rely on headroom size. */
+ MLX5_ASSERT(!has_vec_support ||
+ DATA_OFF(buf) >= RTE_PKTMBUF_HEADROOM);
} else {
buf = seg->null_mbuf;
}
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index 5b04d9a234..ac966c51b4 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -164,16 +164,14 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
seg = &rxq_ctrl->rxq.rxseg[s];
mp = seg->mp;
if (mp) { /* Regular segment */
- bool is_extmem = (rte_pktmbuf_priv_flags(mp) &
- RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) != 0;
- ret = mlx5_mr_mempool_register(rxq_ctrl->sh->cdev, mp,
- is_extmem);
- if (ret < 0 && rte_errno != EEXIST)
- goto error;
- ret = mlx5_mr_mempool_populate_cache(&rxq_ctrl->rxq.mr_ctrl,
- mp);
- if (ret < 0)
- goto error;
+ bool is_extmem = (rte_pktmbuf_priv_flags(mp) &
+ RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) != 0;
+ ret = mlx5_mr_mempool_register(rxq_ctrl->sh->cdev, mp, is_extmem);
+ if (ret < 0 && rte_errno != EEXIST)
+ goto error;
+ ret = mlx5_mr_mempool_populate_cache(&rxq_ctrl->rxq.mr_ctrl, mp);
+ if (ret < 0)
+ goto error;
} else { /* NULL segment used in selective Rx */
seg->null_mbuf = mlx5_alloc_null_mbuf(seg->length);
if (seg->null_mbuf == NULL) {
--
2.54.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox