* [PATCH] dma/cnxk: fix crash on secondary process cleanup
From: pbhagavatula @ 2026-06-05 8:16 UTC (permalink / raw)
To: jerinj, Vamsi Attunuru, Anatoly Burakov, Radha Mohan Chintakuntla
Cc: dev, Pavan Nikhilesh, stable
From: Pavan Nikhilesh <pbhagavatula@marvell.com>
cnxk_dmadev_probe() ran in secondary processes too, overwriting the
shared rdpi->pci_dev with a process-local pointer and marking the
device ready. With buses now cleaned up on shutdown, the primary's
roc_dpi_dev_fini() dereferences that stale pointer and crashes.
Skip HW init in secondary processes: attach to the shared device data
and return, leaving rdpi and the device state untouched.
Fixes: 53f6d7328bf4 ("dma/cnxk: create and initialize device on PCI probing")
Cc: stable@dpdk.org
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
drivers/dma/cnxk/cnxk_dmadev.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/dma/cnxk/cnxk_dmadev.c b/drivers/dma/cnxk/cnxk_dmadev.c
index 0dcebc1b0b3b..6ae7fdca3b1b 100644
--- a/drivers/dma/cnxk/cnxk_dmadev.c
+++ b/drivers/dma/cnxk/cnxk_dmadev.c
@@ -617,6 +617,9 @@ cnxk_dmadev_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_de
dmadev->fp_obj->dev_private = dpivf;
dmadev->dev_ops = &cnxk_dmadev_ops;
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
dpivf->is_cn10k = roc_model_is_cn10k();
dpivf->mcs_lock = NULL;
rdpi = &dpivf->rdpi;
--
2.50.1 (Apple Git-155)
^ permalink raw reply related
* [PATCH v2 5/5] net/bnxt: fix for segmentation fault that would occur on exit
From: Mohammad Shuab Siddique @ 2026-06-05 0:50 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Keegan Freyhof,
Mohammad Shuab Siddique
In-Reply-To: <20260605005016.2290160-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Keegan Freyhof <keegan.freyhof@broadcom.com>
When exiting dpdk while having still having members in a bonded
port, the program would experience a segmentation fault due to the
bonding driver relying on the ethdev driver to free the rx and tx
queues of the bond members. The member ports would then try and
close using the bnxt driver, which would then try and access memory
freed by the ethdev driver causing the seen issue. The bnxt stuct's
rx and tx queues pointers would still point to the freed memory,
while the ethdev pointer would reflect the status changes to the rx
and tx queues array and be nulled.
- Changed net/bnxt/bnxt_stats.c and net/bnxt/bnxt_rxr.c to check
that the rx queues had not already been freed by the ethdev driver
and changed net/bnxt/bnxt_stats.c and net/bnxt/bnxt_txr.c to check
that the tx queues had not already been freed by the ethdev driver.
Fixes: 898248fc4287 ("net/bnxt: support statistics query when port is stopped")
Cc: stable@dpdk.org
Signed-off-by: Keegan Freyhof <keegan.freyhof@broadcom.com>
Signed-off-by: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
---
drivers/net/bnxt/bnxt_rxr.c | 2 +-
drivers/net/bnxt/bnxt_stats.c | 17 +++++++++++------
drivers/net/bnxt/bnxt_txr.c | 3 +++
3 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/drivers/net/bnxt/bnxt_rxr.c b/drivers/net/bnxt/bnxt_rxr.c
index 293b5c03b6..ab2175d21a 100644
--- a/drivers/net/bnxt/bnxt_rxr.c
+++ b/drivers/net/bnxt/bnxt_rxr.c
@@ -1491,7 +1491,7 @@ void bnxt_free_rx_rings(struct bnxt *bp)
int i;
struct bnxt_rx_queue *rxq;
- if (!bp->rx_queues)
+ if (!bp->rx_queues || !bp->eth_dev->data->rx_queues)
return;
for (i = 0; i < (int)bp->rx_nr_rings; i++) {
diff --git a/drivers/net/bnxt/bnxt_stats.c b/drivers/net/bnxt/bnxt_stats.c
index 7b96cf0df9..49367588e4 100644
--- a/drivers/net/bnxt/bnxt_stats.c
+++ b/drivers/net/bnxt/bnxt_stats.c
@@ -547,15 +547,20 @@ void bnxt_free_stats(struct bnxt *bp)
{
int i;
- for (i = 0; i < (int)bp->tx_cp_nr_rings; i++) {
- struct bnxt_tx_queue *txq = bp->tx_queues[i];
+ if (bp->tx_queues && bp->eth_dev->data->tx_queues) {
+ for (i = 0; i < (int)bp->tx_cp_nr_rings; i++) {
+ struct bnxt_tx_queue *txq = bp->tx_queues[i];
- bnxt_free_txq_stats(txq);
+ bnxt_free_txq_stats(txq);
+ }
}
- for (i = 0; i < (int)bp->rx_cp_nr_rings; i++) {
- struct bnxt_rx_queue *rxq = bp->rx_queues[i];
- bnxt_free_rxq_stats(rxq);
+ if (bp->rx_queues && bp->eth_dev->data->rx_queues) {
+ for (i = 0; i < (int)bp->rx_cp_nr_rings; i++) {
+ struct bnxt_rx_queue *rxq = bp->rx_queues[i];
+
+ bnxt_free_rxq_stats(rxq);
+ }
}
}
diff --git a/drivers/net/bnxt/bnxt_txr.c b/drivers/net/bnxt/bnxt_txr.c
index 83d18e220b..d37a38735c 100644
--- a/drivers/net/bnxt/bnxt_txr.c
+++ b/drivers/net/bnxt/bnxt_txr.c
@@ -24,6 +24,9 @@ void bnxt_free_tx_rings(struct bnxt *bp)
{
int i;
+ if (!bp->tx_queues || !bp->eth_dev->data->tx_queues)
+ return;
+
for (i = 0; i < (int)bp->tx_nr_rings; i++) {
struct bnxt_tx_queue *txq = bp->tx_queues[i];
--
2.47.3
^ permalink raw reply related
* [PATCH v2 4/5] net/bnxt: prevent a potential segfault
From: Mohammad Shuab Siddique @ 2026-06-05 0:50 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Ajit Khaparde,
Mohammad Shuab Siddique
In-Reply-To: <20260605005016.2290160-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Ajit Khaparde <ajit.khaparde@broadcom.com>
Add checks to prevent a segfault while accessing members of
structures which may not have been initialized yet.
Fixes: 1b7ceba3e375 ("net/bnxt: support Rx queue count")
Cc: stable@dpdk.org
Signed-off-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Signed-off-by: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
---
drivers/net/bnxt/bnxt_ethdev.c | 3 +++
drivers/net/bnxt/bnxt_hwrm.c | 3 +++
drivers/net/bnxt/bnxt_ring.c | 11 ++++++++++-
3 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index 72a31dfbb5..c45afdb20a 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -1435,6 +1435,9 @@ static int bnxt_scattered_rx(struct rte_eth_dev *eth_dev)
for (i = 0; i < eth_dev->data->nb_rx_queues; i++) {
struct bnxt_rx_queue *rxq = eth_dev->data->rx_queues[i];
+ if (rxq == NULL)
+ continue;
+
buf_size = (uint16_t)(rte_pktmbuf_data_room_size(rxq->mb_pool) -
RTE_PKTMBUF_HEADROOM);
if (eth_dev->data->mtu + overhead > buf_size)
diff --git a/drivers/net/bnxt/bnxt_hwrm.c b/drivers/net/bnxt/bnxt_hwrm.c
index e4ae27d3f4..02a5d00738 100644
--- a/drivers/net/bnxt/bnxt_hwrm.c
+++ b/drivers/net/bnxt/bnxt_hwrm.c
@@ -3328,6 +3328,9 @@ bnxt_free_all_hwrm_stat_ctxs(struct bnxt *bp)
for (i = 0; i < bp->rx_cp_nr_rings; i++) {
+ if (bp->rx_queues[i] == NULL)
+ continue;
+
cpr = bp->rx_queues[i]->cp_ring;
if (BNXT_HAS_RING_GRPS(bp))
bp->grp_info[i].fw_stats_ctx = -1;
diff --git a/drivers/net/bnxt/bnxt_ring.c b/drivers/net/bnxt/bnxt_ring.c
index 579b73d2ce..064520aa62 100644
--- a/drivers/net/bnxt/bnxt_ring.c
+++ b/drivers/net/bnxt/bnxt_ring.c
@@ -723,6 +723,9 @@ static void bnxt_init_all_rings(struct bnxt *bp)
for (i = 0; i < bp->rx_cp_nr_rings; i++) {
rxq = bp->rx_queues[i];
+
+ if (rxq == NULL)
+ continue;
/* Rx-compl */
cp_ring = rxq->cp_ring->cp_ring_struct;
cp_ring->fw_ring_id = INVALID_HW_RING_ID;
@@ -765,9 +768,15 @@ int bnxt_alloc_hwrm_rings(struct bnxt *bp)
for (i = 0; i < bp->rx_cp_nr_rings; i++) {
unsigned int soc_id = bp->eth_dev->device->numa_node;
struct bnxt_rx_queue *rxq = bp->rx_queues[i];
- struct bnxt_rx_ring_info *rxr = rxq->rx_ring;
+ struct bnxt_rx_ring_info *rxr;
struct bnxt_ring *ring;
+
+ if (rxq == NULL)
+ return -EINVAL;
+
+ rxr = rxq->rx_ring;
+
if (bnxt_need_agg_ring(bp->eth_dev)) {
ring = rxr->ag_ring_struct;
if (ring == NULL) {
--
2.47.3
^ permalink raw reply related
* [PATCH v2 3/5] net/bnxt: fix IOVA range check for external mbuf head node
From: Mohammad Shuab Siddique @ 2026-06-05 0:50 UTC (permalink / raw)
To: dev
Cc: kishore.padmanabha, stable, Mohammad Shuab Siddique,
Damodharam Ammepalli
In-Reply-To: <20260605005016.2290160-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
For extmem, the mbuf head node may be allocated from a pool
different from the data buffer pool. In that case the IOVA
address check using rte_mempool_virt2iova() will fail. To fix
this, skip the IOVA range check for external mbufs.
Also add unlikely() in checking invalid mbuf size.
Fixes: d01de33f98e2 ("net/bnxt: skip IOVA range check for external mbuf")
Cc: stable@dpdk.org
Signed-off-by: Damodharam Ammepalli <damodharam.ammepalli@broadcom.com>
Signed-off-by: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
---
drivers/net/bnxt/bnxt_txr.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/net/bnxt/bnxt_txr.c b/drivers/net/bnxt/bnxt_txr.c
index 51b886c1a6..83d18e220b 100644
--- a/drivers/net/bnxt/bnxt_txr.c
+++ b/drivers/net/bnxt/bnxt_txr.c
@@ -220,8 +220,10 @@ static int bnxt_invalid_mbuf(struct rte_mbuf *mbuf)
if (unlikely(rte_mbuf_check(mbuf, 1, &reason)))
return -EINVAL;
- if (unlikely(!(mbuf->ol_flags & RTE_MBUF_F_EXTERNAL) &&
- (mbuf->buf_iova < mbuf_size ||
+ if (unlikely(mbuf->ol_flags & RTE_MBUF_F_EXTERNAL))
+ return 0;
+
+ if (unlikely((mbuf->buf_iova < mbuf_size ||
(mbuf->buf_iova != rte_mempool_virt2iova(mbuf) + mbuf_size))))
return -EINVAL;
--
2.47.3
^ permalink raw reply related
* [PATCH v2 2/5] net/bnxt: use buf address for external mbuf
From: Mohammad Shuab Siddique @ 2026-06-05 0:50 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Ajit Khaparde,
Mohammad Shuab Siddique
In-Reply-To: <20260605005016.2290160-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Ajit Khaparde <ajit.khaparde@broadcom.com>
Use buf_addr for mbufs from external pool instead of using iova
addresses. Uses rte_pktmbuf_mtod_offset() to compute the buffer
address for both the first segment and subsequent segments when
the mbuf is marked as RTE_MBUF_F_EXTERNAL.
Fixes: 42b883535804 ("net/bnxt: use new API to get IOVA address")
Cc: stable@dpdk.org
Signed-off-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Signed-off-by: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
---
drivers/net/bnxt/bnxt_txr.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/drivers/net/bnxt/bnxt_txr.c b/drivers/net/bnxt/bnxt_txr.c
index fb5be224d9..51b886c1a6 100644
--- a/drivers/net/bnxt/bnxt_txr.c
+++ b/drivers/net/bnxt/bnxt_txr.c
@@ -325,7 +325,10 @@ static int bnxt_start_xmit(struct rte_mbuf *tx_pkt,
txbd->flags_type |= TX_BD_LONG_FLAGS_LHINT_GTE2K;
else
txbd->flags_type |= lhint_arr[tx_pkt->pkt_len >> 9];
- txbd->address = rte_cpu_to_le_64(rte_mbuf_data_iova(tx_pkt));
+ if (tx_pkt->ol_flags & RTE_MBUF_F_EXTERNAL)
+ txbd->address = rte_cpu_to_le_64(rte_pktmbuf_mtod_offset(tx_pkt, uint64_t, 0));
+ else
+ txbd->address = rte_cpu_to_le_64(rte_mbuf_data_iova(tx_pkt));
*last_txbd = txbd;
if (long_bd) {
@@ -482,7 +485,11 @@ static int bnxt_start_xmit(struct rte_mbuf *tx_pkt,
*tx_buf = m_seg;
txbd = &txr->tx_desc_ring[prod];
- txbd->address = rte_cpu_to_le_64(rte_mbuf_data_iova(m_seg));
+ if (m_seg->ol_flags & RTE_MBUF_F_EXTERNAL)
+ txbd->address = rte_cpu_to_le_64(
+ rte_pktmbuf_mtod_offset(m_seg, uint64_t, 0));
+ else
+ txbd->address = rte_cpu_to_le_64(rte_mbuf_data_iova(m_seg));
txbd->flags_type = TX_BD_SHORT_TYPE_TX_BD_SHORT;
txbd->len = m_seg->data_len;
--
2.47.3
^ permalink raw reply related
* [PATCH v2 1/5] net/bnxt: fix NQ/CQ processing for interrupt handling
From: Mohammad Shuab Siddique @ 2026-06-05 0:50 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Keegan Freyhof,
Mohammad Shuab Siddique
In-Reply-To: <20260605005016.2290160-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Keegan Freyhof <keegan.freyhof@broadcom.com>
Restructure CQ and NQ arming in NQ processing to be more readable.
Fix incorrect completion validation being used for NQEs and RX
completions in NQ processing.
Fixed the issue of excess interrupts by properly tracking the valid
bit for NQs and using the correct RX completion validation code.
Fixes: 683e5cf79249 ("net/bnxt: use common NQ ring")
Cc: stable@dpdk.org
Signed-off-by: Keegan Freyhof <keegan.freyhof@broadcom.com>
Signed-off-by: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
---
drivers/net/bnxt/bnxt.h | 2 +
drivers/net/bnxt/bnxt_cpr.c | 100 +++++++++++++++++++++++++++++++++++++
drivers/net/bnxt/bnxt_cpr.h | 34 ++++++++++++-
drivers/net/bnxt/bnxt_rxq.c | 47 +++++++++++++++++-
4 files changed, 180 insertions(+), 3 deletions(-)
diff --git a/drivers/net/bnxt/bnxt.h b/drivers/net/bnxt/bnxt.h
index 03df28e64a..f21753e40c 100644
--- a/drivers/net/bnxt/bnxt.h
+++ b/drivers/net/bnxt/bnxt.h
@@ -433,8 +433,10 @@ struct bnxt_coal {
#define DBR_TYPE_SQ (0x0ULL << 60)
#define DBR_TYPE_SRQ (0x2ULL << 60)
#define DBR_TYPE_CQ (0x4ULL << 60)
+#define DBR_TYPE_CQ_ARMALL (0x6ULL << 60)
#define DBR_TYPE_NQ (0xaULL << 60)
#define DBR_TYPE_NQ_ARM (0xbULL << 60)
+#define DBR_TYPE_NQ_MASK (0xeULL << 60)
#define DB_PF_OFFSET 0x10000
#define DB_VF_OFFSET 0x4000
diff --git a/drivers/net/bnxt/bnxt_cpr.c b/drivers/net/bnxt/bnxt_cpr.c
index c6606e19a8..60d1ec0b4a 100644
--- a/drivers/net/bnxt/bnxt_cpr.c
+++ b/drivers/net/bnxt/bnxt_cpr.c
@@ -13,6 +13,106 @@
#include "hsi_struct_def_dpdk.h"
#include "tfc_vf2pf_msg.h"
+void bnxt_process_async_msg(struct bnxt *bp, struct tx_cmpl *cmpl)
+{
+ uint16_t type = cmpl->flags_type & TX_CMPL_TYPE_MASK;
+
+ switch (type) {
+ case HWRM_CMPL_TYPE_HWRM_DONE:
+ break;
+ case HWRM_ASYNC_EVENT_CMPL_TYPE_HWRM_ASYNC_EVENT:
+ bnxt_handle_async_event(bp, (struct cmpl_base *)cmpl);
+ break;
+ default:
+ PMD_DRV_LOG_LINE(ERR, "Port: %d Unhandled async message %x",
+ bp->eth_dev->data->port_id, type);
+ break;
+ }
+}
+
+void bnxt_process_nq(struct bnxt *bp, struct bnxt_cp_ring_info *nqr,
+ struct bnxt_cp_ring_info *rx_cpr)
+{
+ struct nqe_cn *nqcmps = (struct nqe_cn *)nqr->cp_desc_ring;
+ uint32_t ring_mask = nqr->cp_ring_struct->ring_mask;
+ uint32_t raw_cons = nqr->cp_raw_cons;
+ uint16_t nq_type, nqe_cnt = 0;
+ bool v_bit = nqr->valid;
+ uint32_t cons = RING_CMPL(ring_mask, raw_cons);
+
+ while (1) {
+ if (!CMPL_VALID(&nqcmps[cons], v_bit)) {
+ if (nqe_cnt) {
+ nqr->cp_raw_cons = raw_cons;
+ nqr->valid = v_bit;
+ }
+ return;
+ }
+
+ nq_type = NQ_CN_TYPE_MASK & nqcmps[cons].type;
+
+ if (CMP_TYPE((struct cmpl_base *)&nqcmps[cons]) != NQ_CN_TYPE_CQ_NOTIFICATION)
+ bnxt_process_async_msg(bp, (struct tx_cmpl *)&nqcmps[cons]);
+ else
+ rx_cpr->toggle = NQE_CN_TOGGLE((uint64_t)nqcmps[cons].type);
+
+ NEXT_CMPL(nqr, cons, v_bit, 1);
+ raw_cons++;
+ if (nq_type)
+ nqe_cnt++;
+ }
+}
+
+/* Arms/disarms the given NQ for Thor/Thor2, with P5+ represening the ASIC family */
+void bnxt_arm_nq_p5p(struct bnxt_cp_ring_info *nqr, bool enable_irq)
+{
+ uint32_t raw_cons = nqr->cp_raw_cons;
+ uint64_t db_msg = 0;
+ uint32_t toggle = 0;
+
+ if (enable_irq == 1)
+ toggle = nqr->toggle;
+
+ db_msg = nqr->cp_db.db_key64 | (raw_cons & nqr->cp_db.db_ring_mask)
+ | DB_EPOCH(&nqr->cp_db, raw_cons) | DB_TOGGLE(toggle);
+
+ if (enable_irq)
+ db_msg |= DBR_TYPE_NQ_ARM;
+ else
+ db_msg |= DBR_TYPE_NQ_MASK;
+
+ rte_compiler_barrier();
+ rte_write64(db_msg, nqr->cp_db.doorbell);
+ rte_compiler_barrier();
+}
+
+/* Arms/disarms the given CQ for Thor/Thor2, with P5+ represening the ASIC family */
+void bnxt_arm_rx_cq_p5p(struct bnxt_cp_ring_info *cpr, bool enable_irq)
+{
+ uint32_t raw_cons = cpr->cp_raw_cons;
+ uint64_t db_msg = 0;
+ uint32_t toggle = 0;
+
+ if (raw_cons == UINT32_MAX)
+ raw_cons = 0;
+
+ if (enable_irq == 1)
+ toggle = cpr->toggle;
+
+ db_msg = cpr->cp_db.db_key64 | (raw_cons & cpr->cp_db.db_ring_mask)
+ | DB_EPOCH(&cpr->cp_db, raw_cons) | DB_TOGGLE(toggle);
+
+ if (enable_irq)
+ db_msg |= DBR_TYPE_CQ_ARMALL;
+ else
+ db_msg |= DBR_TYPE_CQ;
+
+ rte_compiler_barrier();
+ rte_write64(db_msg, cpr->cp_db.doorbell);
+ rte_compiler_barrier();
+}
+
+
void bnxt_wait_for_device_shutdown(struct bnxt *bp)
{
uint32_t val, timeout;
diff --git a/drivers/net/bnxt/bnxt_cpr.h b/drivers/net/bnxt/bnxt_cpr.h
index 858ee15c47..e097f1034f 100644
--- a/drivers/net/bnxt/bnxt_cpr.h
+++ b/drivers/net/bnxt/bnxt_cpr.h
@@ -30,6 +30,14 @@ struct bnxt_db_info;
#define RING_CMPL(ring_mask, idx) ((idx) & (ring_mask))
#define NEXT_CMP(idx) RING_CMP(ADV_RAW_CMP(idx, 1))
+#define DBR_TOGGLE_SFT 25
+#define DB_TOGGLE(tgl) ((tgl) << DBR_TOGGLE_SFT)
+
+#define NQ_CN_TOGGLE_MASK 0xc0UL
+#define NQ_CN_TOGGLE_SFT 6
+#define NQE_CN_TOGGLE(type) (((type) & NQ_CN_TOGGLE_MASK) >> \
+ NQ_CN_TOGGLE_SFT)
+
#define DB_CP_REARM_FLAGS (DB_KEY_CP | DB_IDX_VALID)
#define DB_CP_FLAGS (DB_KEY_CP | DB_IDX_VALID | DB_IRQ_DIS)
@@ -73,6 +81,21 @@ struct bnxt_db_info {
#define DB_RING_IDX(db, idx) (((idx) & (db)->db_ring_mask) | \
DB_EPOCH(db, idx))
+struct nqe_cn {
+ rte_le16_t type;
+ #define NQ_CN_TYPE_MASK 0x3fUL
+ #define NQ_CN_TYPE_SFT 0
+ #define NQ_CN_TYPE_CQ_NOTIFICATION 0x30UL
+ #define NQ_CN_TYPE_LAST NQ_CN_TYPE_CQ_NOTIFICATION
+ #define NQ_CN_TOGGLE_MASK 0xc0UL
+ #define NQ_CN_TOGGLE_SFT 6
+ rte_le16_t reserved16;
+ rte_le32_t cq_handle_low;
+ rte_le32_t v;
+ #define NQ_CN_V 0x1UL
+ rte_le32_t cq_handle_high;
+};
+
struct bnxt_ring;
struct bnxt_cp_ring_info {
uint32_t cp_raw_cons;
@@ -89,7 +112,9 @@ struct bnxt_cp_ring_info {
struct bnxt_ring *cp_ring_struct;
bool valid;
uint32_t epoch;
- uint8_t dpi;
+ uint32_t toggle;
+ uint8_t dpi; /* Doorbell page index for multi-doorbell support */
+
};
#define RX_CMP_L2_ERRORS \
@@ -101,6 +126,11 @@ void bnxt_handle_fwd_req(struct bnxt *bp, struct cmpl_base *cmp);
int bnxt_event_hwrm_resp_handler(struct bnxt *bp, struct cmpl_base *cmp);
void bnxt_dev_reset_and_resume(void *arg);
void bnxt_wait_for_device_shutdown(struct bnxt *bp);
+void bnxt_arm_nq_p5p(struct bnxt_cp_ring_info *nqr, bool enable_irq);
+void bnxt_arm_rx_cq_p5p(struct bnxt_cp_ring_info *cpr, bool enable_irq);
+void bnxt_process_async_msg(struct bnxt *bp, struct tx_cmpl *cmpl);
+void bnxt_process_nq(struct bnxt *bp, struct bnxt_cp_ring_info *nqr,
+ struct bnxt_cp_ring_info *rx_cpr);
#define EVENT_DATA1_REASON_CODE_FW_EXCEPTION_FATAL \
HWRM_ASYNC_EVENT_CMPL_RESET_NOTIFY_EVENT_DATA1_REASON_CODE_FW_EXCEPTION_FATAL
@@ -152,4 +182,6 @@ bnxt_cpr_cmp_valid(const void *cmpl, uint32_t raw_cons, uint32_t ring_size)
}
return false;
}
+
#endif
+
diff --git a/drivers/net/bnxt/bnxt_rxq.c b/drivers/net/bnxt/bnxt_rxq.c
index 91b3555df6..b93f5043de 100644
--- a/drivers/net/bnxt/bnxt_rxq.c
+++ b/drivers/net/bnxt/bnxt_rxq.c
@@ -461,7 +461,14 @@ bnxt_rx_queue_intr_enable_op(struct rte_eth_dev *eth_dev, uint16_t queue_id)
return -EINVAL;
cpr = rxq->cp_ring;
- B_CP_DB_REARM(cpr, cpr->cp_raw_cons);
+ if (BNXT_CHIP_P5_P7(bp)) {
+ struct bnxt_cp_ring_info *nqr = bp->rxtx_nq_ring;
+
+ bnxt_arm_nq_p5p(nqr, 1);
+ bnxt_arm_rx_cq_p5p(rxq->cp_ring, 1);
+ } else {
+ B_CP_DB_ARM(cpr);
+ }
}
return rc;
}
@@ -484,7 +491,43 @@ bnxt_rx_queue_intr_disable_op(struct rte_eth_dev *eth_dev, uint16_t queue_id)
return -EINVAL;
cpr = rxq->cp_ring;
- B_CP_DB_DISARM(cpr);
+ if (BNXT_CHIP_P5_P7(bp)) {
+ struct bnxt_cp_ring_info *nqr = bp->rxtx_nq_ring;
+ struct bnxt_cp_ring_info *rx_cpr;
+
+ bnxt_arm_nq_p5p(nqr, 0);
+ /* Loops through all RXQs and finds the one that
+ * received a packet if any
+ */
+ uint32_t cons = RING_CMPL(nqr->cp_ring_struct->ring_mask,
+ nqr->cp_raw_cons);
+ struct cmpl_base *cmpl = &((nqr->cp_desc_ring)[cons]);
+
+ if (CMPL_VALID(cmpl, nqr->valid)) {
+ unsigned int rxid = 0;
+
+ rx_cpr = rxq->cp_ring;
+ for (; rxid < bp->rx_nr_rings; rxid++) {
+ rxq = bp->rx_queues[rxid];
+ rx_cpr = rxq->cp_ring;
+ cons = RING_CMPL(rx_cpr->cp_ring_struct->ring_mask,
+ rx_cpr->cp_raw_cons);
+ cmpl = &((rx_cpr->cp_desc_ring)[cons]);
+ /* Because the valid bit in the rx
+ * completion queue is not updated the
+ * cmp valid using raw cons is used
+ */
+ if (bnxt_cpr_cmp_valid(cmpl,
+ rx_cpr->cp_raw_cons,
+ rx_cpr->cp_ring_struct->ring_size)) {
+ break;
+ }
+ }
+ bnxt_process_nq(bp, nqr, rx_cpr);
+ }
+ } else {
+ B_CP_DB_DISARM(cpr);
+ }
}
return rc;
}
--
2.47.3
^ permalink raw reply related
* [PATCH v2 0/5] net/bnxt: interrupt handling, external mbuf and stability fixes
From: Mohammad Shuab Siddique @ 2026-06-05 0:50 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Mohammad Shuab Siddique
In-Reply-To: <20260603211341.2112344-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
This series addresses interrupt handling, external memory, and crash bugs:
- Fix incorrect completion validation for NQEs and RX completions causing
excess interrupts
- Use buf_addr instead of IOVA for mbufs from external memory pools
- Skip IOVA range check for external mbuf head nodes to avoid false failures
- Add null checks to prevent segfaults when accessing uninitialized structures
- Fix segfault on exit when bonded ports are present, by checking whether
ethdev has already freed the RX/TX queue arrays
All patches carry Fixes: tags and Cc: stable@dpdk.org.
Note: this series depends on series "net/bnxt: ULP stats timer and PTP".
Changes in v2:
- Patch 1/5: replace printf() with PMD_DRV_LOG_LINE() (DPDK logging standard)
- Patch 2/5: replace custom bnxt_mbuf_buf_addr() with rte_pktmbuf_mtod_offset()
Ajit Khaparde (2):
net/bnxt: use buf address for external mbuf
net/bnxt: prevent a potential segfault
Keegan Freyhof (2):
net/bnxt: fix NQ/CQ processing for interrupt handling
net/bnxt: fix for segmentation fault that would occur on exit
Mohammad Shuab Siddique (1):
net/bnxt: fix IOVA range check for external mbuf head node
drivers/net/bnxt/bnxt.h | 2 +
drivers/net/bnxt/bnxt_cpr.c | 100 ++++++++++++++++++++++++++++++++++
drivers/net/bnxt/bnxt_cpr.h | 34 +++++++++++-
drivers/net/bnxt/bnxt_ethdev.c | 3 ++
drivers/net/bnxt/bnxt_hwrm.c | 3 ++
drivers/net/bnxt/bnxt_ring.c | 11 +++-
drivers/net/bnxt/bnxt_rxq.c | 47 +++++++++++++++-
drivers/net/bnxt/bnxt_rxr.c | 2 +-
drivers/net/bnxt/bnxt_stats.c | 17 +++---
drivers/net/bnxt/bnxt_txr.c | 19 +++++--
10 files changed, 223 insertions(+), 15 deletions(-)
--
2.47.3
^ permalink raw reply
* [PATCH v15 4/5] vhost: add mem region add/remove handlers
From: pravin.bathija @ 2026-06-04 23:57 UTC (permalink / raw)
To: dev, fengchengwen, stephen, maxime.coquelin; +Cc: pravin.bathija, thomas
In-Reply-To: <20260604235723.1046607-1-pravin.bathija@dell.com>
From: Pravin M Bathija <pravin.bathija@dell.com>
Add support for VHOST_USER_ADD_MEM_REG, VHOST_USER_REM_MEM_REG and
VHOST_USER_GET_MAX_MEM_SLOTS. Refactor memory initialization into
common helper and add supporting functions for dynamic memory management.
Signed-off-by: Pravin M Bathija <pravin.bathija@dell.com>
---
lib/vhost/vhost_user.c | 253 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 253 insertions(+)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 94fca8b589..b13c23ccf8 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -71,6 +71,9 @@ VHOST_MESSAGE_HANDLER(VHOST_USER_SET_FEATURES, vhost_user_set_features, false, t
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_OWNER, vhost_user_set_owner, false, true) \
VHOST_MESSAGE_HANDLER(VHOST_USER_RESET_OWNER, vhost_user_reset_owner, false, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_MEM_TABLE, vhost_user_set_mem_table, true, true) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_MAX_MEM_SLOTS, vhost_user_get_max_mem_slots, false, false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_ADD_MEM_REG, vhost_user_add_mem_reg, true, true) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_REM_MEM_REG, vhost_user_rem_mem_reg, false, true) \
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_LOG_BASE, vhost_user_set_log_base, true, true) \
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_LOG_FD, vhost_user_set_log_fd, true, true) \
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_VRING_NUM, vhost_user_set_vring_num, false, true) \
@@ -1167,6 +1170,24 @@ add_guest_pages(struct virtio_net *dev, struct rte_vhost_mem_region *reg,
return 0;
}
+static void
+remove_guest_pages(struct virtio_net *dev, struct rte_vhost_mem_region *reg)
+{
+ uint64_t reg_start = reg->host_user_addr;
+ uint64_t reg_end = reg_start + reg->size;
+ uint32_t i, j = 0;
+
+ for (i = 0; i < dev->nr_guest_pages; i++) {
+ if (dev->guest_pages[i].host_user_addr >= reg_start &&
+ dev->guest_pages[i].host_user_addr < reg_end)
+ continue;
+ if (j != i)
+ dev->guest_pages[j] = dev->guest_pages[i];
+ j++;
+ }
+ dev->nr_guest_pages = j;
+}
+
#ifdef RTE_LIBRTE_VHOST_DEBUG
/* TODO: enable it only in debug mode? */
static void
@@ -1591,6 +1612,238 @@ vhost_user_set_mem_table(struct virtio_net **pdev,
return RTE_VHOST_MSG_RESULT_ERR;
}
+
+static int
+vhost_user_get_max_mem_slots(struct virtio_net **pdev __rte_unused,
+ struct vhu_msg_context *ctx,
+ int main_fd __rte_unused)
+{
+ uint32_t max_mem_slots = VHOST_MEMORY_MAX_NREGIONS;
+
+ ctx->msg.payload.u64 = max_mem_slots;
+ ctx->msg.size = sizeof(ctx->msg.payload.u64);
+ ctx->fd_num = 0;
+
+ return RTE_VHOST_MSG_RESULT_REPLY;
+}
+
+/*
+ * Invalidate and re-translate all vring addresses after the memory table
+ * has been modified (add/remove region).
+ *
+ * translate_ring_addresses() may call numa_realloc(), which can reallocate
+ * the device structure. The updated pointer is written back through *pdev
+ * so callers must refresh their local "dev" afterwards: dev = *pdev.
+ */
+static void
+vhost_user_invalidate_vrings(struct virtio_net **pdev)
+{
+ struct virtio_net *dev = *pdev;
+ uint32_t i;
+
+ for (i = 0; i < dev->nr_vring; i++) {
+ struct vhost_virtqueue *vq = dev->virtqueue[i];
+
+ if (!vq)
+ continue;
+
+ if (vq->desc || vq->avail || vq->used) {
+ vq_assert_lock(dev, vq);
+
+ vring_invalidate(dev, vq);
+
+ translate_ring_addresses(&dev, &vq);
+ }
+ }
+
+ *pdev = dev;
+}
+
+/*
+ * Macro wrapper that performs the compile-time lock assertion with the
+ * correct message ID at the call site, then calls the implementation.
+ */
+#define dev_invalidate_vrings(pdev, id) do { \
+ static_assert(id ## _LOCK_ALL_QPS, \
+ #id " handler is not declared as locking all queue pairs"); \
+ vhost_user_invalidate_vrings(pdev); \
+} while (0)
+
+static int
+vhost_user_add_mem_reg(struct virtio_net **pdev,
+ struct vhu_msg_context *ctx,
+ int main_fd __rte_unused)
+{
+ struct VhostUserMemoryRegion *region = &ctx->msg.payload.memreg.region;
+ struct virtio_net *dev = *pdev;
+ uint32_t i;
+
+ /* convert first region add to normal memory table set */
+ if (dev->mem == NULL) {
+ if (vhost_user_initialize_memory(pdev) < 0)
+ goto close_msg_fds;
+ }
+
+ /* make sure new region will fit */
+ if (dev->mem->nregions >= VHOST_MEMORY_MAX_NREGIONS) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "too many memory regions already (%u)",
+ dev->mem->nregions);
+ goto close_msg_fds;
+ }
+
+ /* make sure supplied memory fd present */
+ if (ctx->fd_num != 1) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "fd count makes no sense (%u)", ctx->fd_num);
+ goto close_msg_fds;
+ }
+
+ /* Make sure no overlap in guest virtual address space */
+ for (i = 0; i < dev->mem->nregions; i++) {
+ struct rte_vhost_mem_region *cur = &dev->mem->regions[i];
+ uint64_t cur_start = cur->guest_user_addr;
+ uint64_t cur_end = cur_start + cur->size - 1;
+ uint64_t new_start = region->userspace_addr;
+ uint64_t new_end = new_start + region->memory_size - 1;
+
+ if (new_end >= cur_start && new_start <= cur_end) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "requested memory region overlaps with another region");
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "\tRequested region address:0x%" PRIx64,
+ region->userspace_addr);
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "\tRequested region size:0x%" PRIx64,
+ region->memory_size);
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "\tOverlapping region address:0x%" PRIx64,
+ cur->guest_user_addr);
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "\tOverlapping region size:0x%" PRIx64,
+ cur->size);
+ goto close_msg_fds;
+ }
+ }
+
+ /* New region goes at the end of the contiguous array */
+ struct rte_vhost_mem_region *reg = &dev->mem->regions[dev->mem->nregions];
+
+ reg->guest_phys_addr = region->guest_phys_addr;
+ reg->guest_user_addr = region->userspace_addr;
+ reg->size = region->memory_size;
+ reg->fd = ctx->fds[0];
+ ctx->fds[0] = -1;
+
+ if (vhost_user_mmap_region(dev, reg, region->mmap_offset) < 0) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "failed to mmap region");
+ if (reg->mmap_addr) {
+ /* mmap succeeded but a later step (e.g. add_guest_pages)
+ * failed; undo the mapping and any guest-page entries.
+ */
+ remove_guest_pages(dev, reg);
+ free_mem_region(reg);
+ } else {
+ close(reg->fd);
+ reg->fd = -1;
+ }
+ goto close_msg_fds;
+ }
+
+ dev->mem->nregions++;
+
+ if (dev->async_copy && rte_vfio_is_enabled("vfio")) {
+ if (async_dma_map_region(dev, reg, true) < 0)
+ goto free_new_region_no_dma;
+ }
+
+ if (dev->postcopy_listening) {
+ /*
+ * Cannot use vhost_user_postcopy_register() here because it
+ * reads ctx->msg.payload.memory (SET_MEM_TABLE layout), but
+ * ADD_MEM_REG uses the memreg payload. Register the
+ * single new region directly instead.
+ */
+ if (vhost_user_postcopy_region_register(dev, reg) < 0)
+ goto free_new_region;
+ }
+
+ dev_invalidate_vrings(pdev, VHOST_USER_ADD_MEM_REG);
+ dev = *pdev;
+ dump_guest_pages(dev);
+
+ /* Reply with the back-end's mapping address per vhost-user spec */
+ ctx->msg.payload.memreg.region.userspace_addr = reg->host_user_addr;
+ ctx->msg.size = sizeof(ctx->msg.payload.memreg);
+ ctx->fd_num = 0;
+
+ return RTE_VHOST_MSG_RESULT_REPLY;
+
+free_new_region:
+ if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ async_dma_map_region(dev, reg, false);
+free_new_region_no_dma:
+ remove_guest_pages(dev, reg);
+ free_mem_region(reg);
+ dev->mem->nregions--;
+close_msg_fds:
+ close_msg_fds(ctx);
+ return RTE_VHOST_MSG_RESULT_ERR;
+}
+
+static int
+vhost_user_rem_mem_reg(struct virtio_net **pdev,
+ struct vhu_msg_context *ctx,
+ int main_fd __rte_unused)
+{
+ struct VhostUserMemoryRegion *region = &ctx->msg.payload.memreg.region;
+ struct virtio_net *dev = *pdev;
+ uint32_t i;
+
+ if (dev->mem == NULL || dev->mem->nregions == 0) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "no memory regions to remove");
+ return RTE_VHOST_MSG_RESULT_ERR;
+ }
+
+ for (i = 0; i < dev->mem->nregions; i++) {
+ struct rte_vhost_mem_region *current_region = &dev->mem->regions[i];
+
+ /*
+ * According to the vhost-user specification:
+ * The memory region to be removed is identified by its GPA,
+ * user address and size. The mmap offset is ignored.
+ */
+ if (region->userspace_addr == current_region->guest_user_addr
+ && region->guest_phys_addr == current_region->guest_phys_addr
+ && region->memory_size == current_region->size) {
+ if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ async_dma_map_region(dev, current_region, false);
+ if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
+ vhost_user_iotlb_cache_remove(dev,
+ current_region->guest_phys_addr,
+ current_region->size);
+ remove_guest_pages(dev, current_region);
+ free_mem_region(current_region);
+
+ /* Compact the regions array to keep it contiguous */
+ if (i < dev->mem->nregions - 1) {
+ memmove(&dev->mem->regions[i],
+ &dev->mem->regions[i + 1],
+ (dev->mem->nregions - 1 - i) *
+ sizeof(struct rte_vhost_mem_region));
+ memset(&dev->mem->regions[dev->mem->nregions - 1],
+ 0, sizeof(struct rte_vhost_mem_region));
+ }
+
+ dev->mem->nregions--;
+ dev_invalidate_vrings(pdev, VHOST_USER_REM_MEM_REG);
+ dev = *pdev;
+ return RTE_VHOST_MSG_RESULT_OK;
+ }
+ }
+
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "failed to find region");
+ return RTE_VHOST_MSG_RESULT_ERR;
+}
+
static bool
vq_is_ready(struct virtio_net *dev, struct vhost_virtqueue *vq)
{
--
2.43.0
^ permalink raw reply related
* [PATCH v15 5/5] vhost: enable configure memory slots
From: pravin.bathija @ 2026-06-04 23:57 UTC (permalink / raw)
To: dev, fengchengwen, stephen, maxime.coquelin
Cc: pravin.bathija, thomas, Stephen Hemminger
In-Reply-To: <20260604235723.1046607-1-pravin.bathija@dell.com>
From: Pravin M Bathija <pravin.bathija@dell.com>
This patch enables configure memory slots in the header define
VHOST_USER_PROTOCOL_FEATURES.
Signed-off-by: Pravin M Bathija <pravin.bathija@dell.com>
Reviewed-by: Stephen Hemminger <stephen@networkplumber.com>
Reviewed-by: Fengchengwen <fengchengwen@huawei.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/vhost_user.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/vhost/vhost_user.h b/lib/vhost/vhost_user.h
index 6435816534..732aa4dc02 100644
--- a/lib/vhost/vhost_user.h
+++ b/lib/vhost/vhost_user.h
@@ -32,6 +32,7 @@
(1ULL << VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD) | \
(1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER) | \
(1ULL << VHOST_USER_PROTOCOL_F_PAGEFAULT) | \
+ (1ULL << VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS) | \
(1ULL << VHOST_USER_PROTOCOL_F_STATUS))
typedef enum VhostUserRequest {
--
2.43.0
^ permalink raw reply related
* [PATCH v15 2/5] vhost: header defines for add/rem mem region
From: pravin.bathija @ 2026-06-04 23:57 UTC (permalink / raw)
To: dev, fengchengwen, stephen, maxime.coquelin
Cc: pravin.bathija, thomas, Stephen Hemminger
In-Reply-To: <20260604235723.1046607-1-pravin.bathija@dell.com>
From: Pravin M Bathija <pravin.bathija@dell.com>
The changes in this file cover the enum message requests for
supporting add/remove memory regions. The front-end vhost-user
client sends messages like get max memory slots, add memory region
and remove memory region which are covered in these changes which
are on the vhost-user back-end. The changes also include data structure
definition of memory region to be added/removed. The data structure
VhostUserMsg has been changed to include the memory region.
Signed-off-by: Pravin M Bathija <pravin.bathija@dell.com>
Reviewed-by: Fengchengwen <fengchengwen@huawei.com>
Reviewed-by: Stephen Hemminger <stephen@networkplumber.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/vhost_user.h | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/lib/vhost/vhost_user.h b/lib/vhost/vhost_user.h
index ef486545ba..6435816534 100644
--- a/lib/vhost/vhost_user.h
+++ b/lib/vhost/vhost_user.h
@@ -67,6 +67,9 @@ typedef enum VhostUserRequest {
VHOST_USER_POSTCOPY_END = 30,
VHOST_USER_GET_INFLIGHT_FD = 31,
VHOST_USER_SET_INFLIGHT_FD = 32,
+ VHOST_USER_GET_MAX_MEM_SLOTS = 36,
+ VHOST_USER_ADD_MEM_REG = 37,
+ VHOST_USER_REM_MEM_REG = 38,
VHOST_USER_SET_STATUS = 39,
VHOST_USER_GET_STATUS = 40,
} VhostUserRequest;
@@ -91,6 +94,11 @@ typedef struct VhostUserMemory {
VhostUserMemoryRegion regions[VHOST_MEMORY_MAX_NREGIONS];
} VhostUserMemory;
+typedef struct VhostUserMemRegMsg {
+ uint64_t padding;
+ VhostUserMemoryRegion region;
+} VhostUserMemRegMsg;
+
typedef struct VhostUserLog {
uint64_t mmap_size;
uint64_t mmap_offset;
@@ -186,6 +194,7 @@ typedef struct __rte_packed_begin VhostUserMsg {
struct vhost_vring_state state;
struct vhost_vring_addr addr;
VhostUserMemory memory;
+ VhostUserMemRegMsg memreg;
VhostUserLog log;
struct vhost_iotlb_msg iotlb;
VhostUserCryptoSessionParam crypto_session;
--
2.43.0
^ permalink raw reply related
* [PATCH v15 3/5] vhost: refactor memory helper functions
From: pravin.bathija @ 2026-06-04 23:57 UTC (permalink / raw)
To: dev, fengchengwen, stephen, maxime.coquelin
Cc: pravin.bathija, thomas, Stephen Hemminger
In-Reply-To: <20260604235723.1046607-1-pravin.bathija@dell.com>
From: Pravin M Bathija <pravin.bathija@dell.com>
- Extract reusable helper routines for vhost-user backend memory
operations.
- Split DMA map/unmap into per-region logic.
- Decouple and rework memory region free routines.
- Iterate over VHOST_MEMORY_MAX_NREGIONS uniformly
across related functions to simplify code reuse
Signed-off-by: Pravin M Bathija <pravin.bathija@dell.com>
Reviewed-by: Stephen Hemminger <stephen@networkplumber.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/vhost_user.c | 172 ++++++++++++++++++++++++++---------------
1 file changed, 110 insertions(+), 62 deletions(-)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 4bfb13fb98..94fca8b589 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -171,20 +171,27 @@ get_blk_size(int fd)
return ret == -1 ? (uint64_t)-1 : (uint64_t)stat.st_blksize;
}
-static void
-async_dma_map(struct virtio_net *dev, bool do_map)
+static int
+async_dma_map_region(struct virtio_net *dev, struct rte_vhost_mem_region *reg, bool do_map)
{
- int ret = 0;
uint32_t i;
- struct guest_page *page;
+ int ret;
+ uint64_t reg_start = reg->host_user_addr;
+ uint64_t reg_end = reg_start + reg->size;
- if (do_map) {
- for (i = 0; i < dev->nr_guest_pages; i++) {
- page = &dev->guest_pages[i];
+ for (i = 0; i < dev->nr_guest_pages; i++) {
+ struct guest_page *page = &dev->guest_pages[i];
+
+ /* Only process pages belonging to this region */
+ if (page->host_user_addr < reg_start ||
+ page->host_user_addr >= reg_end)
+ continue;
+
+ if (do_map) {
ret = rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
- page->host_user_addr,
- page->host_iova,
- page->size);
+ page->host_user_addr,
+ page->host_iova,
+ page->size);
if (ret) {
/*
* DMA device may bind with kernel driver, in this case,
@@ -199,33 +206,57 @@ async_dma_map(struct virtio_net *dev, bool do_map)
* normal case in async path. This is a workaround.
*/
if (rte_errno == ENODEV)
- return;
+ return 0;
/* DMA mapping errors won't stop VHOST_USER_SET_MEM_TABLE. */
VHOST_CONFIG_LOG(dev->ifname, ERR, "DMA engine map failed");
+ return -1;
}
- }
-
- } else {
- for (i = 0; i < dev->nr_guest_pages; i++) {
- page = &dev->guest_pages[i];
+ } else {
ret = rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
- page->host_user_addr,
- page->host_iova,
- page->size);
+ page->host_user_addr,
+ page->host_iova,
+ page->size);
if (ret) {
/* like DMA map, ignore the kernel driver case when unmap. */
if (rte_errno == EINVAL)
- return;
+ return 0;
VHOST_CONFIG_LOG(dev->ifname, ERR, "DMA engine unmap failed");
+ return -1;
}
}
}
+
+ return 0;
+}
+
+static void
+async_dma_map(struct virtio_net *dev, bool do_map)
+{
+ uint32_t i;
+ struct rte_vhost_mem_region *reg;
+
+ for (i = 0; i < VHOST_MEMORY_MAX_NREGIONS; i++) {
+ reg = &dev->mem->regions[i];
+ if (reg->host_user_addr == 0)
+ continue;
+ async_dma_map_region(dev, reg, do_map);
+ }
}
static void
-free_mem_region(struct virtio_net *dev)
+free_mem_region(struct rte_vhost_mem_region *reg)
+{
+ if (reg != NULL && reg->mmap_addr) {
+ munmap(reg->mmap_addr, reg->mmap_size);
+ close(reg->fd);
+ memset(reg, 0, sizeof(struct rte_vhost_mem_region));
+ }
+}
+
+static void
+free_all_mem_regions(struct virtio_net *dev)
{
uint32_t i;
struct rte_vhost_mem_region *reg;
@@ -236,12 +267,10 @@ free_mem_region(struct virtio_net *dev)
if (dev->async_copy && rte_vfio_is_enabled("vfio"))
async_dma_map(dev, false);
- for (i = 0; i < dev->mem->nregions; i++) {
+ for (i = 0; i < VHOST_MEMORY_MAX_NREGIONS; i++) {
reg = &dev->mem->regions[i];
- if (reg->host_user_addr) {
- munmap(reg->mmap_addr, reg->mmap_size);
- close(reg->fd);
- }
+ if (reg->mmap_addr)
+ free_mem_region(reg);
}
}
@@ -255,7 +284,7 @@ vhost_backend_cleanup(struct virtio_net *dev)
vdpa_dev->ops->dev_cleanup(dev->vid);
if (dev->mem) {
- free_mem_region(dev);
+ free_all_mem_regions(dev);
rte_free(dev->mem);
dev->mem = NULL;
}
@@ -704,7 +733,7 @@ numa_realloc(struct virtio_net **pdev, struct vhost_virtqueue **pvq)
vhost_devices[dev->vid] = dev;
mem_size = sizeof(struct rte_vhost_memory) +
- sizeof(struct rte_vhost_mem_region) * dev->mem->nregions;
+ sizeof(struct rte_vhost_mem_region) * VHOST_MEMORY_MAX_NREGIONS;
mem = rte_realloc_socket(dev->mem, mem_size, 0, node);
if (!mem) {
VHOST_CONFIG_LOG(dev->ifname, ERR,
@@ -808,8 +837,10 @@ hua_to_alignment(struct rte_vhost_memory *mem, void *ptr)
uint32_t i;
uintptr_t hua = (uintptr_t)ptr;
- for (i = 0; i < mem->nregions; i++) {
+ for (i = 0; i < VHOST_MEMORY_MAX_NREGIONS; i++) {
r = &mem->regions[i];
+ if (r->host_user_addr == 0)
+ continue;
if (hua >= r->host_user_addr &&
hua < r->host_user_addr + r->size) {
return get_blk_size(r->fd);
@@ -1382,6 +1413,52 @@ vhost_user_mmap_region(struct virtio_net *dev,
return 0;
}
+static int
+vhost_user_initialize_memory(struct virtio_net **pdev)
+{
+ struct virtio_net *dev = *pdev;
+ int numa_node = SOCKET_ID_ANY;
+
+ if (dev->mem != NULL) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "memory already initialized, free it first");
+ return -1;
+ }
+
+ /*
+ * If VQ 0 has already been allocated, try to allocate on the same
+ * NUMA node. It can be reallocated later in numa_realloc().
+ */
+ if (dev->nr_vring > 0)
+ numa_node = dev->virtqueue[0]->numa_node;
+
+ dev->nr_guest_pages = 0;
+ if (dev->guest_pages == NULL) {
+ dev->max_guest_pages = 8;
+ dev->guest_pages = rte_zmalloc_socket(NULL,
+ dev->max_guest_pages *
+ sizeof(struct guest_page),
+ RTE_CACHE_LINE_SIZE,
+ numa_node);
+ if (dev->guest_pages == NULL) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "failed to allocate memory for dev->guest_pages");
+ return -1;
+ }
+ }
+
+ dev->mem = rte_zmalloc_socket("vhost-mem-table", sizeof(struct rte_vhost_memory) +
+ sizeof(struct rte_vhost_mem_region) * VHOST_MEMORY_MAX_NREGIONS, 0, numa_node);
+ if (dev->mem == NULL) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "failed to allocate memory for dev->mem");
+ rte_free(dev->guest_pages);
+ dev->guest_pages = NULL;
+ return -1;
+ }
+
+ return 0;
+}
+
static int
vhost_user_set_mem_table(struct virtio_net **pdev,
struct vhu_msg_context *ctx,
@@ -1390,7 +1467,6 @@ vhost_user_set_mem_table(struct virtio_net **pdev,
struct virtio_net *dev = *pdev;
struct VhostUserMemory *memory = &ctx->msg.payload.memory;
struct rte_vhost_mem_region *reg;
- int numa_node = SOCKET_ID_ANY;
uint64_t mmap_offset;
uint32_t i;
bool async_notify = false;
@@ -1435,39 +1511,13 @@ vhost_user_set_mem_table(struct virtio_net **pdev,
if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
vhost_user_iotlb_flush_all(dev);
- free_mem_region(dev);
+ free_all_mem_regions(dev);
rte_free(dev->mem);
dev->mem = NULL;
}
- /*
- * If VQ 0 has already been allocated, try to allocate on the same
- * NUMA node. It can be reallocated later in numa_realloc().
- */
- if (dev->nr_vring > 0)
- numa_node = dev->virtqueue[0]->numa_node;
-
- dev->nr_guest_pages = 0;
- if (dev->guest_pages == NULL) {
- dev->max_guest_pages = 8;
- dev->guest_pages = rte_zmalloc_socket(NULL,
- dev->max_guest_pages *
- sizeof(struct guest_page),
- RTE_CACHE_LINE_SIZE,
- numa_node);
- if (dev->guest_pages == NULL) {
- VHOST_CONFIG_LOG(dev->ifname, ERR,
- "failed to allocate memory for dev->guest_pages");
- goto close_msg_fds;
- }
- }
-
- dev->mem = rte_zmalloc_socket("vhost-mem-table", sizeof(struct rte_vhost_memory) +
- sizeof(struct rte_vhost_mem_region) * memory->nregions, 0, numa_node);
- if (dev->mem == NULL) {
- VHOST_CONFIG_LOG(dev->ifname, ERR, "failed to allocate memory for dev->mem");
- goto free_guest_pages;
- }
+ if (vhost_user_initialize_memory(pdev) < 0)
+ goto close_msg_fds;
for (i = 0; i < memory->nregions; i++) {
reg = &dev->mem->regions[i];
@@ -1531,11 +1581,9 @@ vhost_user_set_mem_table(struct virtio_net **pdev,
return RTE_VHOST_MSG_RESULT_OK;
free_mem_table:
- free_mem_region(dev);
+ free_all_mem_regions(dev);
rte_free(dev->mem);
dev->mem = NULL;
-
-free_guest_pages:
rte_free(dev->guest_pages);
dev->guest_pages = NULL;
close_msg_fds:
--
2.43.0
^ permalink raw reply related
* [PATCH v15 1/5] vhost: add user to mailmap and define to vhost hdr
From: pravin.bathija @ 2026-06-04 23:57 UTC (permalink / raw)
To: dev, fengchengwen, stephen, maxime.coquelin
Cc: pravin.bathija, thomas, Stephen Hemminger
In-Reply-To: <20260604235723.1046607-1-pravin.bathija@dell.com>
From: Pravin M Bathija <pravin.bathija@dell.com>
- add user to mailmap file.
- define a bit-field called VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS
that depicts if the feature/capability to add/remove memory regions
is supported. This is a part of the overall support for add/remove
memory region feature in this patchset.
Signed-off-by: Pravin M Bathija <pravin.bathija@dell.com>
Acked-by: Fengchengwen <fengchengwen@huawei.com>
Reviewed-by: Stephen Hemminger <stephen@networkplumber.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
.mailmap | 1 +
lib/vhost/rte_vhost.h | 4 ++++
2 files changed, 5 insertions(+)
diff --git a/.mailmap b/.mailmap
index 0e0d83e1c6..cc44e27036 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1295,6 +1295,7 @@ Prateek Agarwal <prateekag@cse.iitb.ac.in>
Prathisna Padmasanan <prathisna.padmasanan@intel.com>
Praveen Kaligineedi <pkaligineedi@google.com>
Praveen Shetty <praveen.shetty@intel.com>
+Pravin M Bathija <pravin.bathija@dell.com>
Pravin Pathak <pravin.pathak.dev@gmail.com> <pravin.pathak@intel.com>
Prince Takkar <ptakkar@marvell.com>
Priyalee Kushwaha <priyalee.kushwaha@intel.com>
diff --git a/lib/vhost/rte_vhost.h b/lib/vhost/rte_vhost.h
index 2f7c4c0080..a7f9700538 100644
--- a/lib/vhost/rte_vhost.h
+++ b/lib/vhost/rte_vhost.h
@@ -109,6 +109,10 @@ extern "C" {
#define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12
#endif
+#ifndef VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS
+#define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
+#endif
+
#ifndef VHOST_USER_PROTOCOL_F_STATUS
#define VHOST_USER_PROTOCOL_F_STATUS 16
#endif
--
2.43.0
^ permalink raw reply related
* [PATCH v15 0/5] Support add/remove memory region and get-max-slots
From: pravin.bathija @ 2026-06-04 23:57 UTC (permalink / raw)
To: dev, fengchengwen, stephen, maxime.coquelin; +Cc: pravin.bathija, thomas
From: Pravin M Bathija <pravin.bathija@dell.com>
This is version v15 of the patchset and it incorporates the
recommendations made by Maxime Coquelin.
Patch 4/5
- Changed VHOST_USER_REM_MEM_REG handler declaration from
accepts_fd=true to accepts_fd=false, as the remove request does not
expect FDs in ancillary data.
- Removed all close_msg_fds(ctx) calls from vhost_user_rem_mem_reg(), no
longer needed since the handler is declared as not accepting FDs.
- Removed validate_msg_fds(dev, ctx, 0) check from
vhost_user_rem_mem_reg(), as FD validation is now handled generically
by the framework.
- Added targeted IOTLB cache invalidation in vhost_user_rem_mem_reg()
using vhost_user_iotlb_cache_remove() for the removed region's GPA
range, instead of the nuclear iotlb_flush_all() used by set_mem_table.
This implementation has been extensively tested by doing Read/Write I/O
from multiple instances of fio + libblkio (front-end) talking to
spdk/dpdk (back-end) based drives. Tested with qemu front-end talking to
dpdk testpmd (back-end) performing add/removal of memory regions. Also
tested post-copy live migration after doing add_memory_region.
Version Log:
Version v15 (Current version): Incorporate code review suggestions from
Maxime Coquelin as described above.
Version v14: Incorporate code review suggestions from Stephen Hemminger
and Fengcheng Wen.
Changes from Fengcheng Wen review:
Patch 3/5
- Moved free_all_mem_regions() call sites in vhost_user_set_mem_table()
from patch 4/5 to patch 3/5 so each commit compiles independently
Patch 4/5
- Renamed _dev_invalidate_vrings() to vhost_user_invalidate_vrings() to
follow vhost naming convention
- Added comment explaining *pdev propagation through
translate_ring_addresses / numa_realloc()
- Reordered local variables in vhost_user_add_mem_reg() and
vhost_user_rem_mem_reg() by descending line length
- Shortened overlap check variable names (current_region_guest_start/end
--> cur_start/end, proposed_region_guest_start/end -> new_start/end)
- Fixed DMA error path in vhost_user_add_mem_reg(): added
free_new_region_no_dma label so async_dma_map_region(false) is not
called when the map itself failed.
Changes from Stephen Hemminger review:
Patch 4/5
- vhost_user_add_mem_reg() now constructs a reply with the back-end's
host mapping address in userspace_addr and returns
RTE_VHOST_MSG_RESULT_REPLY per the vhost-user spec
- Added validate_msg_fds(dev, ctx, 0) in vhost_user_rem_mem_reg() to
reject malformed messages with unexpected file descriptors
- Dropped unnecessary (uint64_t) cast in vhost_user_get_max_mem_slots()
Version v13: Incorporate code review suggestions from Fengcheng Wen
Patch 2/5
Renamed VhostUserSingleMemReg to VhostUserMemRegMsg and memory_single
to memreg
Patches 3/5 and 4/5
Relocated function remove_guest_pages from patch 3/5 to 4/5
Version v12: Incorporate code review suggestions from Maxime Coquelin
and ai-code-review.
Patch 3/5
Refactored async_dma_map() to delegate to async_dma_map_region(),
eliminating code duplication between the two functions.
Restored original comments in async_dma_map_region() explaining why
ENODEV and EINVAL errors are ignored (these were stripped in v10)
Reverted unnecessary changes to vhost_user_postcopy_register() --
removed the host_user_addr == 0 checks and reg_msg_index indirection
that were added in v10, since this function is only called from
vhost_user_set_mem_table() where regions are always contiguous.
Version v11: Incorporate code review suggestions from Stephen Hemminger.
Patch 4/5
Fix incomplete cleanup in vhost_user_add_mem_reg() when
vhost_user_mmap_region() fails after the mmap succeeds (e.g.
add_guest_pages() realloc failure) realloc failure). The error path now
calls remove_guest_pages() and free_mem_region() to undo the mapping
and stale guest-page entries, preventing a leaked mmap and slot reuse
corruption. The plain close(fd) path is kept for pre-mmap failures.
Version v10: Incorporate code review suggestions from Stephen Hemminger.
Patch 4/5
Moved dev_invalidate_vrings after free_mem_region, array compaction, and
nregions decrement. This ensures translate_ring_addresses only sees
surviving memory regions, preventing vring pointers from resolving into
a region that is about to be unmapped.
Version v9: Incorporate code review suggestions from Stephen Hemminger.
Patch 3/5
Restored max_guest_pages initial value to hardcoded 8 instead of
VHOST_MEMORY_MAX_NREGIONS, matching upstream semantics.
Patch 4/5
Added close(reg->fd) and reg->fd = -1 before goto close_msg_fds in the
mmap failure path to fix fd leak after fd was moved from ctx->fds[0].
Converted dev_invalidate_vrings from a plain function to a macro +
implementation function pair, accepting message ID as a parameter so
the static_assert reports the correct handler at each call site.
Updated dev_invalidate_vrings call in add_mem_reg to pass
VHOST_USER_ADD_MEM_REG as message ID.
Updated dev_invalidate_vrings call in rem_mem_reg to pass
VHOST_USER_REM_MEM_REG as message ID.
Version v8: Incorporate code review suggestions from Stephen Hemminger.
rewrite async_dma_map_region function to iterate guest pages by host
address range matching
change function dev_invalidate_vrings to accept a double pointer to
propagate pointer updates
new function remove_guest_pages was added
add_mem_reg error path was narrowed to only clean up the single failed
region instead of destroting all existing regions
Version v7: Incorporate code review suggestions from Maxime Coquelin.
Add debug messages to vhost_postcopy_register function.
Version v6: Added the enablement of this feature as a final patch in
this patch-set and other code optimizations as suggested by Maxime
Coquelin.
Version v5: removed the patch that increased the number of memory regions
from 8 to 128. This will be submitted as a separate feature at a later
point after incorporating additional optimizations. Also includes code
optimizations as suggested by Feng Cheng Wen.
Version v4: code optimizations as suggested by Feng Cheng Wen.
Version v3: code optimizations as suggested by Maxime Coquelin
and Thomas Monjalon.
Version v2: code optimizations as suggested by Maxime Coquelin.
Version v1: Initial patch set.
Pravin M Bathija (5):
vhost: add user to mailmap and define to vhost hdr
vhost: header defines for add/rem mem region
vhost: refactor memory helper functions
vhost: add mem region add/remove handlers
vhost: enable configure memory slots
.mailmap | 1 +
lib/vhost/rte_vhost.h | 4 +
lib/vhost/vhost_user.c | 425 +++++++++++++++++++++++++++++++++++------
lib/vhost/vhost_user.h | 10 +
4 files changed, 378 insertions(+), 62 deletions(-)
--
2.43.0
^ permalink raw reply
* RE: [PATCH v14 4/5] vhost: add mem region add/remove handlers
From: Bathija, Pravin @ 2026-06-04 23:56 UTC (permalink / raw)
To: Maxime Coquelin
Cc: dev@dpdk.org, fengchengwen@huawei.com, stephen@networkplumber.org,
thomas@monjalon.net, Stephen Hemminger
In-Reply-To: <CAO55csyTk2TwYjiLmA3J5Tt+sJHKpNPE+0S0rYEbTyH71+-3Qg@mail.gmail.com>
Hi Maxime,
The responses are inline. Since your message was received as html, the formatting was lost after converting to plain text. I have tagged my responses as <Pravin>. The forthcoming patch v15 has the suggested changes. There was also a mix-up on my part with regards to "reviewed-by" and "Acked-by" which I have also corrected. Thank you for pointing that out.
Regards,
Pravin
Internal Use - Confidential
From: Maxime Coquelin <maxime.coquelin@redhat.com>
Sent: Thursday, June 4, 2026 3:27 AM
To: Bathija, Pravin <Pravin.Bathija@dell.com>
Cc: dev@dpdk.org; fengchengwen@huawei.com; stephen@networkplumber.org; thomas@monjalon.net; Stephen Hemminger <stephen@networkplumber.com>
Subject: Re: [PATCH v14 4/5] vhost: add mem region add/remove handlers
[EXTERNAL EMAIL]
On Wed, May 20, 2026 at 4:20 AM <mailto:pravin.bathija@dell.com> wrote:
From: Pravin M Bathija <mailto:pravin.bathija@dell.com>
Add support for VHOST_USER_ADD_MEM_REG, VHOST_USER_REM_MEM_REG and
VHOST_USER_GET_MAX_MEM_SLOTS. Refactor memory initialization into
common helper and add supporting functions for dynamic memory management.
Signed-off-by: Pravin M Bathija <mailto:pravin.bathija@dell.com>
Acked-by: Fengchengwen <mailto:fengchengwen@huawei.com>
Reviewed-by: Stephen Hemminger <mailto:stephen@networkplumber.com>
Acked-by: Maxime Coquelin <mailto:maxime.coquelin@redhat.com>
---
lib/vhost/vhost_user.c | 255 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 255 insertions(+)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 94fca8b589..522ba1db82 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -71,6 +71,9 @@ VHOST_MESSAGE_HANDLER(VHOST_USER_SET_FEATURES, vhost_user_set_features, false, t
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_OWNER, vhost_user_set_owner, false, true) \
VHOST_MESSAGE_HANDLER(VHOST_USER_RESET_OWNER, vhost_user_reset_owner, false, false) \
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_MEM_TABLE, vhost_user_set_mem_table, true, true) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_GET_MAX_MEM_SLOTS, vhost_user_get_max_mem_slots, false, false) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_ADD_MEM_REG, vhost_user_add_mem_reg, true, true) \
+VHOST_MESSAGE_HANDLER(VHOST_USER_REM_MEM_REG, vhost_user_rem_mem_reg, true, true) \
The removal request does not expect FDs in ancillary data.
It should be:
VHOST_MESSAGE_HANDLER(VHOST_USER_REM_MEM_REG, vhost_user_rem_mem_reg, false, true)
<Pravin> Fixed, changed handler declaration from (true, true) to (false, true).
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_LOG_BASE, vhost_user_set_log_base, true, true) \
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_LOG_FD, vhost_user_set_log_fd, true, true) \
VHOST_MESSAGE_HANDLER(VHOST_USER_SET_VRING_NUM, vhost_user_set_vring_num, false, true) \
@@ -1167,6 +1170,24 @@ add_guest_pages(struct virtio_net *dev, struct rte_vhost_mem_region *reg,
return 0;
}
+static void
+remove_guest_pages(struct virtio_net *dev, struct rte_vhost_mem_region *reg)
+{
+ uint64_t reg_start = reg->host_user_addr;
+ uint64_t reg_end = reg_start + reg->size;
+ uint32_t i, j = 0;
+
+ for (i = 0; i < dev->nr_guest_pages; i++) {
+ if (dev->guest_pages[i].host_user_addr >= reg_start &&
+ dev->guest_pages[i].host_user_addr < reg_end)
+ continue;
+ if (j != i)
+ dev->guest_pages[j] = dev->guest_pages[i];
+ j++;
+ }
+ dev->nr_guest_pages = j;
+}
+
#ifdef RTE_LIBRTE_VHOST_DEBUG
/* TODO: enable it only in debug mode? */
static void
@@ -1591,6 +1612,240 @@ vhost_user_set_mem_table(struct virtio_net **pdev,
return RTE_VHOST_MSG_RESULT_ERR;
}
+
+static int
+vhost_user_get_max_mem_slots(struct virtio_net **pdev __rte_unused,
+ struct vhu_msg_context *ctx,
+ int main_fd __rte_unused)
+{
+ uint32_t max_mem_slots = VHOST_MEMORY_MAX_NREGIONS;
+
+ ctx->msg.payload.u64 = max_mem_slots;
+ ctx->msg.size = sizeof(ctx->msg.payload.u64);
+ ctx->fd_num = 0;
+
+ return RTE_VHOST_MSG_RESULT_REPLY;
+}
+
+/*
+ * Invalidate and re-translate all vring addresses after the memory table
+ * has been modified (add/remove region).
+ *
+ * translate_ring_addresses() may call numa_realloc(), which can reallocate
+ * the device structure. The updated pointer is written back through *pdev
+ * so callers must refresh their local "dev" afterwards: dev = *pdev.
+ */
+static void
+vhost_user_invalidate_vrings(struct virtio_net **pdev)
+{
+ struct virtio_net *dev = *pdev;
+ uint32_t i;
+
+ for (i = 0; i < dev->nr_vring; i++) {
+ struct vhost_virtqueue *vq = dev->virtqueue[i];
+
+ if (!vq)
+ continue;
+
+ if (vq->desc || vq->avail || vq->used) {
+ vq_assert_lock(dev, vq);
+
+ vring_invalidate(dev, vq);
+
+ translate_ring_addresses(&dev, &vq);
+ }
+ }
+
+ *pdev = dev;
+}
+
+/*
+ * Macro wrapper that performs the compile-time lock assertion with the
+ * correct message ID at the call site, then calls the implementation.
+ */
+#define dev_invalidate_vrings(pdev, id) do { \
+ static_assert(id ## _LOCK_ALL_QPS, \
+ #id " handler is not declared as locking all queue pairs"); \
+ vhost_user_invalidate_vrings(pdev); \
+} while (0)
+
+static int
+vhost_user_add_mem_reg(struct virtio_net **pdev,
+ struct vhu_msg_context *ctx,
+ int main_fd __rte_unused)
+{
+ struct VhostUserMemoryRegion *region = &ctx->msg.payload.memreg.region;
+ struct virtio_net *dev = *pdev;
+ uint32_t i;
+
+ /* convert first region add to normal memory table set */
+ if (dev->mem == NULL) {
+ if (vhost_user_initialize_memory(pdev) < 0)
+ goto close_msg_fds;
+ }
+
+ /* make sure new region will fit */
+ if (dev->mem->nregions >= VHOST_MEMORY_MAX_NREGIONS) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "too many memory regions already (%u)",
+ dev->mem->nregions);
+ goto close_msg_fds;
+ }
+
+ /* make sure supplied memory fd present */
+ if (ctx->fd_num != 1) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "fd count makes no sense (%u)", ctx->fd_num);
+ goto close_msg_fds;
+ }
+
+ /* Make sure no overlap in guest virtual address space */
+ for (i = 0; i < dev->mem->nregions; i++) {
+ struct rte_vhost_mem_region *cur = &dev->mem->regions[i];
+ uint64_t cur_start = cur->guest_user_addr;
+ uint64_t cur_end = cur_start + cur->size - 1;
+ uint64_t new_start = region->userspace_addr;
+ uint64_t new_end = new_start + region->memory_size - 1;
+
+ if (new_end >= cur_start && new_start <= cur_end) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "requested memory region overlaps with another region");
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "\tRequested region address:0x%" PRIx64,
+ region->userspace_addr);
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "\tRequested region size:0x%" PRIx64,
+ region->memory_size);
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "\tOverlapping region address:0x%" PRIx64,
+ cur->guest_user_addr);
+ VHOST_CONFIG_LOG(dev->ifname, ERR,
+ "\tOverlapping region size:0x%" PRIx64,
+ cur->size);
+ goto close_msg_fds;
+ }
+ }
+
+ /* New region goes at the end of the contiguous array */
+ struct rte_vhost_mem_region *reg = &dev->mem->regions[dev->mem->nregions];
+
+ reg->guest_phys_addr = region->guest_phys_addr;
+ reg->guest_user_addr = region->userspace_addr;
+ reg->size = region->memory_size;
+ reg->fd = ctx->fds[0];
+ ctx->fds[0] = -1;
+
+ if (vhost_user_mmap_region(dev, reg, region->mmap_offset) < 0) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "failed to mmap region");
+ if (reg->mmap_addr) {
+ /* mmap succeeded but a later step (e.g. add_guest_pages)
+ * failed; undo the mapping and any guest-page entries.
+ */
+ remove_guest_pages(dev, reg);
+ free_mem_region(reg);
+ } else {
+ close(reg->fd);
+ reg->fd = -1;
+ }
+ goto close_msg_fds;
+ }
+
+ dev->mem->nregions++;
+
+ if (dev->async_copy && rte_vfio_is_enabled("vfio")) {
+ if (async_dma_map_region(dev, reg, true) < 0)
+ goto free_new_region_no_dma;
+ }
+
+ if (dev->postcopy_listening) {
+ /*
+ * Cannot use vhost_user_postcopy_register() here because it
+ * reads ctx->msg.payload.memory (SET_MEM_TABLE layout), but
+ * ADD_MEM_REG uses the memreg payload. Register the
+ * single new region directly instead.
+ */
+ if (vhost_user_postcopy_region_register(dev, reg) < 0)
+ goto free_new_region;
+ }
+
+ dev_invalidate_vrings(pdev, VHOST_USER_ADD_MEM_REG);
+ dev = *pdev;
+ dump_guest_pages(dev);
+
+ /* Reply with the back-end's mapping address per vhost-user spec */
+ ctx->msg.payload.memreg.region.userspace_addr = reg->host_user_addr;
+ ctx->msg.size = sizeof(ctx->msg.payload.memreg);
+ ctx->fd_num = 0;
+
+ return RTE_VHOST_MSG_RESULT_REPLY;
+
+free_new_region:
+ if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ async_dma_map_region(dev, reg, false);
+free_new_region_no_dma:
+ remove_guest_pages(dev, reg);
+ free_mem_region(reg);
+ dev->mem->nregions--;
+close_msg_fds:
+ close_msg_fds(ctx);
+ return RTE_VHOST_MSG_RESULT_ERR;
+}
+
+static int
+vhost_user_rem_mem_reg(struct virtio_net **pdev,
+ struct vhu_msg_context *ctx,
+ int main_fd __rte_unused)
+{
+ struct VhostUserMemoryRegion *region = &ctx->msg.payload.memreg.region;
+ struct virtio_net *dev = *pdev;
+ uint32_t i;
+
+ if (dev->mem == NULL || dev->mem->nregions == 0) {
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "no memory regions to remove");
+ close_msg_fds(ctx);
Not needed if properly declared.
<Pravin> Agreed, removed.
+ return RTE_VHOST_MSG_RESULT_ERR;
+ }
+
+ if (validate_msg_fds(dev, ctx, 0) != 0)
+ return RTE_VHOST_MSG_RESULT_ERR;
With proper declaration, we can remove this check, as it is done in a generic way.
<Pravin> Agreed, removed -- the framework now rejects unexpected FDs before the handler is called.
+
+ for (i = 0; i < dev->mem->nregions; i++) {
+ struct rte_vhost_mem_region *current_region = &dev->mem->regions[i];
+
+ /*
+ * According to the vhost-user specification:
+ * The memory region to be removed is identified by its GPA,
+ * user address and size. The mmap offset is ignored.
+ */
+ if (region->userspace_addr == current_region->guest_user_addr
+ && region->guest_phys_addr == current_region->guest_phys_addr
+ && region->memory_size == current_region->size) {
+ if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ async_dma_map_region(dev, current_region, false);
+ remove_guest_pages(dev, current_region);
You are missing the step to clear the IOTLB cache entries matching with this removed region.
In vhost_user_set_mem_table(), a vhost_user_iotlb_flush_all() call is made,
but that would be kind of a nuclear option for memory hotplug.
I suggest removing only the entries matching the removed area, something like this:
if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
vhost_user_iotlb_cache_remove(dev, current_region->guest_phys_addr, current_region->size);
<Pravin> Fixed, added targeted vhost_user_iotlb_cache_remove(dev, current_region->guest_phys_addr, current_region->size)
guarded by VIRTIO_F_IOMMU_PLATFORM, instead of the nuclear iotlb_flush_all().
+ free_mem_region(current_region);
+
+ /* Compact the regions array to keep it contiguous */
+ if (i < dev->mem->nregions - 1) {
+ memmove(&dev->mem->regions[i],
+ &dev->mem->regions[i + 1],
+ (dev->mem->nregions - 1 - i) *
+ sizeof(struct rte_vhost_mem_region));
+ memset(&dev->mem->regions[dev->mem->nregions - 1],
+ 0, sizeof(struct rte_vhost_mem_region));
+ }
+
+ dev->mem->nregions--;
+ dev_invalidate_vrings(pdev, VHOST_USER_REM_MEM_REG);
+ dev = *pdev;
+ close_msg_fds(ctx);
And no need to close FDs, are we are now sure none were provided.
<Pravin> Agreed, removed.
+ return RTE_VHOST_MSG_RESULT_OK;
+ }
+ }
+
+ VHOST_CONFIG_LOG(dev->ifname, ERR, "failed to find region");
+ close_msg_fds(ctx);
Same, no more needed.
<Pravin> Agreed, removed.
+ return RTE_VHOST_MSG_RESULT_ERR;
+}
+
static bool
vq_is_ready(struct virtio_net *dev, struct vhost_virtqueue *vq)
{
--
2.43.0
^ permalink raw reply related
* [PATCH v2 4/4] net/bnxt: fix RSS hash mode configuration for VFs
From: Mohammad Shuab Siddique @ 2026-06-04 22:56 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Mohammad Shuab Siddique
In-Reply-To: <20260604225622.2285191-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
Fixed VFs attempting global RSS configuration which is not
permitted by firmware. VFs (including trusted VFs) must use
per-VNIC RSS configuration with actual vnic_id and rss_ctx_idx
values.
Fixes: 8b9adaf0da6b ("net/bnxt: support RSS on ESP and AH headers")
Cc: stable@dpdk.org
Signed-off-by: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
---
drivers/net/bnxt/bnxt_hwrm.c | 18 ++++++++++++++++--
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/drivers/net/bnxt/bnxt_hwrm.c b/drivers/net/bnxt/bnxt_hwrm.c
index 0c82935de9..afc948ac29 100644
--- a/drivers/net/bnxt/bnxt_hwrm.c
+++ b/drivers/net/bnxt/bnxt_hwrm.c
@@ -2970,8 +2970,22 @@ bnxt_hwrm_vnic_rss_cfg_hash_mode_p5(struct bnxt *bp, struct bnxt_vnic_info *vnic
req.hash_mode_flags = BNXT_HASH_MODE_INNERMOST;
else
req.hash_mode_flags = vnic->hash_mode;
- req.vnic_id = rte_cpu_to_le_16(BNXT_DFLT_VNIC_ID_INVALID);
- req.rss_ctx_idx = rte_cpu_to_le_16(BNXT_RSS_CTX_IDX_INVALID);
+
+ /* VFs must use actual vnic_id for per-VNIC configuration.
+ * PFs can use INVALID vnic_id for global configuration.
+ * This is because VFs don't have permission to configure
+ * global hash mode, even if they're trusted.
+ */
+ if (BNXT_VF(bp)) {
+ req.vnic_id = rte_cpu_to_le_16(vnic->fw_vnic_id);
+ req.rss_ctx_idx = rte_cpu_to_le_16(vnic->fw_grp_ids[0]);
+ PMD_DRV_LOG_LINE(DEBUG, "VF using per-VNIC RSS config (vnic_id=%u)",
+ vnic->fw_vnic_id);
+ } else {
+ req.vnic_id = rte_cpu_to_le_16(BNXT_DFLT_VNIC_ID_INVALID);
+ req.rss_ctx_idx = rte_cpu_to_le_16(BNXT_RSS_CTX_IDX_INVALID);
+ PMD_DRV_LOG_LINE(DEBUG, "PF using global RSS config");
+ }
PMD_DRV_LOG_LINE(DEBUG, "RSS CFG: Hash level %d", req.hash_mode_flags);
rc = bnxt_hwrm_send_message(bp, &req, sizeof(req),
--
2.47.3
^ permalink raw reply related
* [PATCH v2 3/4] net/bnxt: remove implicit integer sign-extension
From: Mohammad Shuab Siddique @ 2026-06-04 22:56 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Zoe Cheimets, Mohammad Shuab Siddique
In-Reply-To: <20260604225622.2285191-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Zoe Cheimets <zoe.cheimets@broadcom.com>
In bnxt_ring.c, the result on line 389 was auto-sign extended by
the compiler because the arithmetic result is an int, but the
dpi_offset is uint64_t. Fix by casting the result to uint64_t
before the multiplication forces extension. To ensure that a
negative integer is not being cast to uint64_t, add a check in
the if-statement.
Fixes: 7a1f9c782b50 ("net/bnxt: add multi-doorbell support")
Cc: stable@dpdk.org
Signed-off-by: Zoe Cheimets <zoe.cheimets@broadcom.com>
Signed-off-by: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
---
drivers/net/bnxt/bnxt_ring.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/drivers/net/bnxt/bnxt_ring.c b/drivers/net/bnxt/bnxt_ring.c
index ccca779b97..579b73d2ce 100644
--- a/drivers/net/bnxt/bnxt_ring.c
+++ b/drivers/net/bnxt/bnxt_ring.c
@@ -385,9 +385,10 @@ void bnxt_set_db(struct bnxt *bp,
db->doorbell = (char *)bp->doorbell_base + db_offset;
if (bp->fw_cap & BNXT_FW_CAP_MULTI_DB &&
- dpi != BNXT_PRIVILEGED_DPI) {
- dpi_offset = (dpi - bp->nq_dpi_start) *
- bp->db_page_size;
+ dpi != BNXT_PRIVILEGED_DPI &&
+ dpi >= bp->nq_dpi_start) {
+ dpi_offset = (uint64_t)(dpi - bp->nq_dpi_start) *
+ bp->db_page_size;
db->doorbell = (char *)db->doorbell + dpi_offset;
}
db->db_key64 |= (uint64_t)fid << DBR_XID_SFT;
--
2.47.3
^ permalink raw reply related
* [PATCH v2 2/4] net/bnxt: fix QP resource count in backing store config
From: Mohammad Shuab Siddique @ 2026-06-04 22:56 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Ajit Khaparde,
Mohammad Shuab Siddique
In-Reply-To: <20260604225622.2285191-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Ajit Khaparde <ajit.khaparde@broadcom.com>
The driver is not passing the QP1 count while
configuring backing store for QP type.
This can result in a smaller set of entries allocated by the
driver with the firmware.
The number of entries is provided by the firmware as a
part of backing store qcaps v2 HWRM command.
Fixes: 5b5d398434f9 ("net/bnxt: add support for backing store v2")
Cc: stable@dpdk.org
Signed-off-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Signed-off-by: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
---
drivers/net/bnxt/bnxt_ethdev.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index b677f9491d..7d70d0f3ec 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -5404,11 +5404,11 @@ int bnxt_alloc_ctx_pg_tbls(struct bnxt *bp)
if (ctxm->type == HWRM_FUNC_BACKING_STORE_CFG_V2_INPUT_TYPE_CQ)
entries = ctxm->cq_l2_entries;
else if (ctxm->type == HWRM_FUNC_BACKING_STORE_CFG_V2_INPUT_TYPE_QP)
- entries = ctxm->qp_l2_entries;
+ entries = ctxm->qp_l2_entries + ctxm->qp_qp1_entries;
else if (ctxm->type == HWRM_FUNC_BACKING_STORE_CFG_V2_INPUT_TYPE_MRAV)
entries = ctxm->mrav_av_entries;
else if (ctxm->type == HWRM_FUNC_BACKING_STORE_CFG_V2_INPUT_TYPE_TIM)
- entries = ctx2->qp_l2_entries;
+ entries = ctx2->qp_l2_entries + ctx2->qp_qp1_entries;
entries = clamp_t(uint32_t, entries, ctxm->min_entries,
ctxm->max_entries);
ctx_pg[i].entries = entries;
--
2.47.3
^ permalink raw reply related
* [PATCH v2 1/4] net/bnxt: modify check for short Tx BDs
From: Mohammad Shuab Siddique @ 2026-06-04 22:56 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Ajit Khaparde,
Mohammad Shuab Siddique
In-Reply-To: <20260604225622.2285191-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Ajit Khaparde <ajit.khaparde@broadcom.com>
There is no need to use the long BDs for transmits
where only checksum offload is needed.
Modify the check for long BD and use long BDs only in cases
where TSO and other offloads are requested.
Fixes: 527b10089cc5 ("net/bnxt: optimize Tx completion handling")
Cc: stable@dpdk.org
Signed-off-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Signed-off-by: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
---
drivers/net/bnxt/bnxt_txr.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/net/bnxt/bnxt_txr.c b/drivers/net/bnxt/bnxt_txr.c
index 27758898b0..7ef5b15ae8 100644
--- a/drivers/net/bnxt/bnxt_txr.c
+++ b/drivers/net/bnxt/bnxt_txr.c
@@ -111,8 +111,7 @@ int bnxt_init_tx_ring_struct(struct bnxt_tx_queue *txq, unsigned int socket_id)
static bool
bnxt_xmit_need_long_bd(struct rte_mbuf *tx_pkt, struct bnxt_tx_queue *txq)
{
- if (tx_pkt->ol_flags & (RTE_MBUF_F_TX_TCP_SEG | RTE_MBUF_F_TX_TCP_CKSUM |
- RTE_MBUF_F_TX_UDP_CKSUM | RTE_MBUF_F_TX_IP_CKSUM |
+ if (tx_pkt->ol_flags & (RTE_MBUF_F_TX_TCP_SEG |
RTE_MBUF_F_TX_VLAN | RTE_MBUF_F_TX_OUTER_IP_CKSUM |
RTE_MBUF_F_TX_TUNNEL_GRE | RTE_MBUF_F_TX_TUNNEL_VXLAN |
RTE_MBUF_F_TX_TUNNEL_GENEVE | RTE_MBUF_F_TX_IEEE1588_TMST |
--
2.47.3
^ permalink raw reply related
* [PATCH v2 0/4] net/bnxt: miscellaneous bug fixes
From: Mohammad Shuab Siddique @ 2026-06-04 22:56 UTC (permalink / raw)
To: dev; +Cc: kishore.padmanabha, stable, Mohammad Shuab Siddique
In-Reply-To: <20260603175137.1990204-1-Mohammad-Shuab.Siddique@broadcom.com>
From: Mohammad Shuab Siddique <mohammad-shuab.siddique@broadcom.com>
This series collects four independent bug fixes for the bnxt PMD:
- Eliminate unnecessary long TX BDs when only checksum offload is needed
- Pass QP1 resource count correctly when configuring backing store
- Fix implicit integer sign-extension in the doorbell calculation
- Prevent VFs from attempting global RSS configuration
All patches carry Fixes: tags and Cc: stable@dpdk.org.
Changes in v2:
- Patch 4/4: add missing Fixes: tag for RSS hash mode fix
Ajit Khaparde (2):
net/bnxt: modify check for short Tx BDs
net/bnxt: fix QP resource count in backing store config
Mohammad Shuab Siddique (1):
net/bnxt: fix RSS hash mode configuration for VFs
Zoe Cheimets (1):
net/bnxt: remove implicit integer sign-extension
drivers/net/bnxt/bnxt_ethdev.c | 4 ++--
drivers/net/bnxt/bnxt_hwrm.c | 18 ++++++++++++++++--
drivers/net/bnxt/bnxt_ring.c | 7 ++++---
drivers/net/bnxt/bnxt_txr.c | 3 +--
4 files changed, 23 insertions(+), 9 deletions(-)
--
2.47.3
^ permalink raw reply
* [PATCH v8 9/9] dts: add selective Rx tests
From: Thomas Monjalon @ 2026-06-04 19:31 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger, Luca Vizzarro, Patrick Robb
In-Reply-To: <20260604193324.1996141-1-thomas@monjalon.net>
Add TestSuite_rx_split with 7 test cases:
- 3 positive: headers only, payload only, two non-contiguous segments
- 4 negative: missing offload flag, out-of-range, overlap, all-discard
Add selective Rx capability detection via testpmd "show port info".
The test suite could be completed later for the basic buffer split
configuration based on offsets or protocols.
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
dts/api/capabilities.py | 2 +
dts/api/testpmd/__init__.py | 17 ++
dts/api/testpmd/types.py | 6 +
dts/framework/testbed_model/capability.py | 2 +
dts/tests/TestSuite_rx_split.py | 265 ++++++++++++++++++++++
5 files changed, 292 insertions(+)
create mode 100644 dts/tests/TestSuite_rx_split.py
diff --git a/dts/api/capabilities.py b/dts/api/capabilities.py
index 09bc538523..b0c1d81d36 100644
--- a/dts/api/capabilities.py
+++ b/dts/api/capabilities.py
@@ -136,6 +136,8 @@ class NicCapability(IntEnum):
#: Device supports all VLAN capabilities.
PORT_RX_OFFLOAD_VLAN = auto()
QUEUE_RX_OFFLOAD_VLAN = auto()
+ #: Device supports selective Rx.
+ SELECTIVE_RX = auto()
#: Device supports Rx queue setup after device started.
RUNTIME_RX_QUEUE_SETUP = auto()
#: Device supports Tx queue setup after device started.
diff --git a/dts/api/testpmd/__init__.py b/dts/api/testpmd/__init__.py
index e9187440bb..6973a64573 100644
--- a/dts/api/testpmd/__init__.py
+++ b/dts/api/testpmd/__init__.py
@@ -1409,6 +1409,23 @@ def get_capabilities_show_port_info(
self.ports[0].device_capabilities,
)
+ def get_capabilities_selective_rx(
+ self,
+ supported_capabilities: MutableSet["NicCapability"],
+ unsupported_capabilities: MutableSet["NicCapability"],
+ ) -> None:
+ """Get selective Rx capability from show port info.
+
+ Args:
+ supported_capabilities: Supported capabilities will be added to this set.
+ unsupported_capabilities: Unsupported capabilities will be added to this set.
+ """
+ port_info = self.show_port_info(self.ports[0].id)
+ if port_info.selective_rx:
+ supported_capabilities.add(NicCapability.SELECTIVE_RX)
+ else:
+ unsupported_capabilities.add(NicCapability.SELECTIVE_RX)
+
def get_capabilities_mcast_filtering(
self,
supported_capabilities: MutableSet["NicCapability"],
diff --git a/dts/api/testpmd/types.py b/dts/api/testpmd/types.py
index 0d322aece2..6f1eaf47cc 100644
--- a/dts/api/testpmd/types.py
+++ b/dts/api/testpmd/types.py
@@ -614,6 +614,12 @@ def _validate(info: str) -> str | None:
metadata=VLANOffloadFlag.make_parser(),
)
+ #: Selective Rx support
+ selective_rx: bool = field(
+ default=False,
+ metadata=TextParser.find(r"Selective Rx: supported"),
+ )
+
#: Maximum size of RX buffer
max_rx_bufsize: int | None = field(
default=None, metadata=TextParser.find_int(r"Maximum size of RX buffer: (\d+)")
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
index 96e1cd449f..b10799ea4b 100644
--- a/dts/framework/testbed_model/capability.py
+++ b/dts/framework/testbed_model/capability.py
@@ -324,6 +324,8 @@ def mapping(cap: NicCapability) -> TestPmdNicCapability:
| NicCapability.FLOW_SHARED_OBJECT_KEEP
):
return (TestPmd.get_capabilities_show_port_info, None)
+ case NicCapability.SELECTIVE_RX:
+ return (TestPmd.get_capabilities_selective_rx, None)
case NicCapability.MCAST_FILTERING:
return (TestPmd.get_capabilities_mcast_filtering, None)
case NicCapability.FLOW_CTRL:
diff --git a/dts/tests/TestSuite_rx_split.py b/dts/tests/TestSuite_rx_split.py
new file mode 100644
index 0000000000..5e7dc76463
--- /dev/null
+++ b/dts/tests/TestSuite_rx_split.py
@@ -0,0 +1,265 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2026 NVIDIA Corporation & Affiliates
+
+"""Rx split test suite.
+
+Test configuring a packet split on Rx,
+and discarding some segments (selective Rx) at NIC level.
+"""
+
+from typing import Any
+
+from scapy.layers.inet import IP
+from scapy.layers.l2 import Ether
+from scapy.packet import Packet, Raw
+
+from api.capabilities import (
+ NicCapability,
+ requires_nic_capability,
+)
+from api.packet import send_packet_and_capture
+from api.test import fail, verify
+from api.testpmd import TestPmd
+from api.testpmd.config import SimpleForwardingModes
+from api.testpmd.types import RxOffloadCapability, TxOffloadCapability
+from framework.exception import InteractiveCommandExecutionError
+from framework.test_suite import TestSuite, func_test
+
+PAYLOAD = bytes(range(256))
+ETHER_HDR_LEN = len(Ether())
+IP_HDR_LEN = len(IP())
+ETHER_IP_HDR_LEN = ETHER_HDR_LEN + IP_HDR_LEN
+
+
+@requires_nic_capability(NicCapability.PORT_RX_OFFLOAD_BUFFER_SPLIT)
+@requires_nic_capability(NicCapability.SELECTIVE_RX)
+class TestRxSplit(TestSuite):
+ """Rx split test suite.
+
+ Configure testpmd with various Rx segment offset/length combinations
+ and verify that only the requested portions of the packet are received
+ and forwarded.
+ """
+
+ def _create_testpmd(self, **kwargs: Any) -> TestPmd:
+ """Create a TestPmd instance with defaults overridden by kwargs."""
+ defaults: dict[str, Any] = {
+ "forward_mode": SimpleForwardingModes.mac,
+ "rx_offloads": RxOffloadCapability.BUFFER_SPLIT,
+ "enable_scatter": True,
+ }
+ return TestPmd(**{**defaults, **kwargs})
+
+ def _build_packet(self) -> Packet:
+ """Build a test packet with an incrementing byte pattern payload."""
+ return Ether() / IP() / Raw(load=PAYLOAD)
+
+ def _send_and_verify(
+ self,
+ testpmd: TestPmd,
+ packet: Packet,
+ expected_bytes: bytes,
+ ) -> None:
+ """Clear stats, send a packet, and verify received content and stats.
+
+ Args:
+ testpmd: The running testpmd instance.
+ packet: The packet to send.
+ expected_bytes: Expected raw bytes of the received packet.
+ """
+ expected_len = len(expected_bytes)
+ testpmd.clear_port_stats_all(verify=False)
+
+ received = send_packet_and_capture(packet)
+ verify(
+ len(received) > 0,
+ "Did not receive any packets.",
+ )
+
+ recv_bytes = bytes(received[0])
+ verify(
+ len(recv_bytes) == expected_len,
+ f"Expected packet length {expected_len}, got {len(recv_bytes)}.",
+ )
+ verify(
+ recv_bytes == expected_bytes,
+ "Received packet content does not match expected bytes.",
+ )
+
+ all_stats, _ = testpmd.show_port_stats_all()
+ total_rx_packets = sum(s.rx_packets for s in all_stats)
+ total_rx_bytes = sum(s.rx_bytes for s in all_stats)
+ verify(
+ total_rx_packets == 1,
+ f"Expected 1 Rx packet, got {total_rx_packets}.",
+ )
+ verify(
+ total_rx_bytes == expected_len,
+ f"Expected {expected_len} Rx bytes, got {total_rx_bytes}.",
+ )
+
+ @func_test
+ def selective_rx_headers(self) -> None:
+ """Keep only the Ethernet + IP headers, discard the payload.
+
+ Steps:
+ Start testpmd with rxpkts, mbuf-size and buffer split enabled.
+ Send an Ether/IP/payload packet.
+
+ Verify:
+ Received packet has Ether + IP headers only.
+ Port stats show expected rx_packets and rx_bytes.
+ """
+ with self._create_testpmd(
+ rx_segments_length=[ETHER_IP_HDR_LEN, 0],
+ mbuf_size=[256, 0],
+ ) as testpmd:
+ testpmd.start()
+ packet = self._build_packet()
+ expected = bytes(packet)[:ETHER_IP_HDR_LEN]
+ self._send_and_verify(testpmd, packet, expected)
+
+ @func_test
+ def selective_rx_payload_only(self) -> None:
+ """Skip the Ethernet + IP headers, keep only the payload.
+
+ Steps:
+ Start testpmd with rxpkts, mbuf-size and buffer split enabled.
+ Send an Ether/IP/payload packet.
+
+ Verify:
+ Received packet is matching the original payload.
+ Port stats show expected rx_packets and rx_bytes.
+ """
+ with self._create_testpmd(
+ rx_segments_length=[ETHER_IP_HDR_LEN, len(PAYLOAD)],
+ mbuf_size=[0, 512],
+ ) as testpmd:
+ testpmd.start()
+ self._send_and_verify(testpmd, self._build_packet(), PAYLOAD)
+
+ @func_test
+ def selective_rx_two_segments(self) -> None:
+ """Keep the IP header and the middle of the payload, skip the rest.
+
+ Steps:
+ Start testpmd with rxpkts, mbuf-size, buffer split
+ and multi-segment Tx enabled.
+ Send an Ether/IP/payload packet.
+
+ Verify:
+ Received packet is matching the IP header and middle of payload.
+ Port stats show expected rx_packets and rx_bytes.
+ """
+ payload_offset = 100
+ payload_length = 100
+ with self._create_testpmd(
+ tx_offloads=TxOffloadCapability.MULTI_SEGS,
+ rx_segments_length=[
+ ETHER_HDR_LEN, IP_HDR_LEN,
+ payload_offset, payload_length, 0,
+ ],
+ mbuf_size=[0, 256, 0, 256, 0],
+ ) as testpmd:
+ testpmd.start()
+ packet = self._build_packet()
+ raw = bytes(packet)
+ payload_start = ETHER_IP_HDR_LEN + payload_offset
+ expected = (
+ raw[ETHER_HDR_LEN:ETHER_IP_HDR_LEN]
+ + raw[payload_start : payload_start + payload_length]
+ )
+ self._send_and_verify(testpmd, packet, expected)
+
+ @func_test
+ def selective_rx_no_offload(self) -> None:
+ """Configure selective Rx with buffer split disabled.
+
+ Steps:
+ Start testpmd with rxpkts, mbuf-size including a discard segment,
+ buffer split disabled, and device start disabled.
+ Attempt to start ports.
+
+ Verify:
+ Queue configuration fails.
+ """
+ with self._create_testpmd(
+ rx_offloads=None,
+ rx_segments_length=[ETHER_IP_HDR_LEN, 0],
+ mbuf_size=[256, 0],
+ disable_device_start=True,
+ ) as testpmd:
+ try:
+ testpmd.start_all_ports()
+ fail("Expected configuration to fail with buffer split disabled.")
+ except InteractiveCommandExecutionError:
+ pass
+
+ @func_test
+ def selective_rx_out_of_range(self) -> None:
+ """Configure selective Rx with total lengths exceeding max_rx_pktlen.
+
+ Steps:
+ Start testpmd with rxpkts too big, buffer split enabled,
+ and device start disabled.
+ Attempt to start ports.
+
+ Verify:
+ Queue configuration fails.
+ """
+ with self._create_testpmd(
+ rx_segments_length=[ETHER_IP_HDR_LEN, 20000],
+ mbuf_size=[256, 0],
+ disable_device_start=True,
+ ) as testpmd:
+ try:
+ testpmd.start_all_ports()
+ fail("Expected configuration to fail with out-of-range length.")
+ except InteractiveCommandExecutionError:
+ pass
+
+ @func_test
+ def selective_rx_segment_exceeds_mbuf(self) -> None:
+ """Configure selective Rx with segment length exceeding mbuf capacity.
+
+ Steps:
+ Start testpmd with rxpkts larger than mbuf-size,
+ buffer split enabled, and device start disabled.
+ Attempt to start ports.
+
+ Verify:
+ Queue configuration fails.
+ """
+ with self._create_testpmd(
+ rx_segments_length=[4096, 0],
+ mbuf_size=[128, 0],
+ disable_device_start=True,
+ ) as testpmd:
+ try:
+ testpmd.start_all_ports()
+ fail("Expected configuration to fail with segment > mbuf size.")
+ except InteractiveCommandExecutionError:
+ pass
+
+ @func_test
+ def selective_rx_all_discard(self) -> None:
+ """Configure selective Rx with only discard segment.
+
+ Steps:
+ Start testpmd with only discard segment,
+ buffer split enabled, and device start disabled.
+ Attempt to start ports.
+
+ Verify:
+ Queue configuration fails.
+ """
+ with self._create_testpmd(
+ rx_segments_length=[0],
+ mbuf_size=[0],
+ disable_device_start=True,
+ ) as testpmd:
+ try:
+ testpmd.start_all_ports()
+ fail("Expected configuration to fail with only discard segment.")
+ except InteractiveCommandExecutionError:
+ pass
--
2.54.0
^ permalink raw reply related
* [PATCH v8 8/9] dts: fix topology capability comparison
From: Thomas Monjalon @ 2026-06-04 19:31 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, stable, Luca Vizzarro, Patrick Robb,
Juraj Linkeš, Dean Marx, Jeremy Spewock
In-Reply-To: <20260604193324.1996141-1-thomas@monjalon.net>
TopologyCapability.__gt__() was delegating to __lt__(),
which caused infinite recursion when "other" is not a TopologyCapability:
other.__lt__(self) returns NotImplemented,
Python retries with self.__gt__(other),
and the cycle repeats.
dts/framework/testbed_model/capability.py", line 579, in __gt__
return other < self
^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded
Similarly, __le__() was delegating to "not __gt__()",
which returns True for non-comparable types instead of False.
Fix both by checking is_comparable_with() first
and comparing topology_type directly, consistent with __lt__().
Fixes: 039256daa8bf ("dts: add topology capability")
Cc: stable@dpdk.org
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
dts/framework/testbed_model/capability.py | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py
index 960370fc72..96e1cd449f 100644
--- a/dts/framework/testbed_model/capability.py
+++ b/dts/framework/testbed_model/capability.py
@@ -574,7 +574,9 @@ def __gt__(self, other: Any) -> bool:
Returns:
:data:`True` if the instance's topology type is more complex than the compared object's.
"""
- return other < self
+ if not self.is_comparable_with(other):
+ return False
+ return self.topology_type > other.topology_type
def __le__(self, other: Any) -> bool:
"""Compare the :attr:`~TopologyCapability.topology_type`s.
@@ -586,7 +588,9 @@ def __le__(self, other: Any) -> bool:
:data:`True` if the instance's topology type is less complex or equal than
the compared object's.
"""
- return not self > other
+ if not self.is_comparable_with(other):
+ return False
+ return self.topology_type <= other.topology_type
def __hash__(self):
"""Each instance is identified by :attr:`topology_type`."""
--
2.54.0
^ permalink raw reply related
* [PATCH v8 7/9] common/mlx5: remove callbacks for MR registration
From: Thomas Monjalon @ 2026-06-04 19:30 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, Dariusz Sosnowski, Viacheslav Ovsiienko,
Bing Zhao, Ori Kam, Suanming Mou, Matan Azrad, Fan Zhang,
Ashish Gupta
In-Reply-To: <20260604193324.1996141-1-thomas@monjalon.net>
The functions register/unregister for a Memory Region (MR)
were not called directly.
There are only 2 implementations for Linux and Windows,
no need of handling this difference with function pointers.
The callback pointers are replaced with direct calls
and link time decision based on the Operating System.
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
drivers/common/mlx5/linux/mlx5_common_verbs.c | 26 +++----------
drivers/common/mlx5/mlx5_common.c | 6 +--
drivers/common/mlx5/mlx5_common_mr.c | 37 ++++++++-----------
drivers/common/mlx5/mlx5_common_mr.h | 26 +++----------
drivers/common/mlx5/windows/mlx5_common_os.c | 23 ++----------
drivers/compress/mlx5/mlx5_compress.c | 4 +-
drivers/crypto/mlx5/mlx5_crypto.h | 2 -
drivers/crypto/mlx5/mlx5_crypto_gcm.c | 6 +--
drivers/net/mlx5/mlx5.h | 3 +-
drivers/net/mlx5/mlx5_flow_aso.c | 21 +++++------
drivers/net/mlx5/mlx5_flow_hw.c | 11 ++----
drivers/net/mlx5/mlx5_flow_quota.c | 6 +--
drivers/net/mlx5/mlx5_hws_cnt.c | 19 ++++------
13 files changed, 61 insertions(+), 129 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 6d44e1f566..5e23c5844d 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -106,10 +106,10 @@ mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx)
* @return
* 0 on successful registration, -1 otherwise
*/
-RTE_EXPORT_INTERNAL_SYMBOL(mlx5_common_verbs_reg_mr)
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_reg_mr)
int
-mlx5_common_verbs_reg_mr(void *pd, void *addr, size_t length,
- struct mlx5_pmd_mr *pmd_mr)
+mlx5_os_reg_mr(void *pd, void *addr, size_t length,
+ struct mlx5_pmd_mr *pmd_mr)
{
struct ibv_mr *ibv_mr;
@@ -136,9 +136,9 @@ mlx5_common_verbs_reg_mr(void *pd, void *addr, size_t length,
* pmd_mr struct set with lkey, address, length and pointer to mr object
*
*/
-RTE_EXPORT_INTERNAL_SYMBOL(mlx5_common_verbs_dereg_mr)
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_dereg_mr)
void
-mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
+mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
{
if (pmd_mr && pmd_mr->obj != NULL) {
claim_zero(mlx5_glue->dereg_mr(pmd_mr->obj));
@@ -146,22 +146,6 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
}
}
-/**
- * Set the reg_mr and dereg_mr callbacks.
- *
- * @param[out] reg_mr_cb
- * Pointer to reg_mr func
- * @param[out] dereg_mr_cb
- * Pointer to dereg_mr func
- */
-RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_set_reg_mr_cb)
-void
-mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
-{
- *reg_mr_cb = mlx5_common_verbs_reg_mr;
- *dereg_mr_cb = mlx5_common_verbs_dereg_mr;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_alloc_null_mr)
struct mlx5_pmd_mr *
mlx5_os_alloc_null_mr(struct rte_device *dev, void *pd)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index f71dbe4637..87de6d0ff0 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -1135,7 +1135,7 @@ mlx5_common_dev_dma_map(struct rte_device *rte_dev, void *addr,
return -1;
}
mr = mlx5_create_mr_ext(dev->pd, (uintptr_t)addr, len,
- SOCKET_ID_ANY, dev->mr_scache.reg_mr_cb);
+ SOCKET_ID_ANY);
if (!mr) {
DRV_LOG(WARNING, "Device %s unable to DMA map", rte_dev->name);
rte_errno = EINVAL;
@@ -1165,7 +1165,7 @@ mlx5_common_dev_dma_map(struct rte_device *rte_dev, void *addr,
ret = mlx5_mr_expand_cache(&dev->mr_scache, size,
rte_dev->numa_node);
if (ret < 0) {
- mlx5_mr_free(mr, dev->mr_scache.dereg_mr_cb);
+ mlx5_mr_free(mr);
rte_errno = ret;
return -1;
}
@@ -1221,7 +1221,7 @@ mlx5_common_dev_dma_unmap(struct rte_device *rte_dev, void *addr,
}
LIST_REMOVE(mr, mr);
DRV_LOG(DEBUG, "MR(%p) is removed from list.", (void *)mr);
- mlx5_mr_free(mr, dev->mr_scache.dereg_mr_cb);
+ mlx5_mr_free(mr);
mlx5_mr_rebuild_cache(&dev->mr_scache);
/*
* No explicit wmb is needed after updating dev_gen due to
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 64ffc7f4ea..aa2d5e88a4 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -492,12 +492,12 @@ mlx5_mr_lookup_cache(struct mlx5_mr_share_cache *share_cache,
* Pointer to MR to free.
*/
void
-mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb)
+mlx5_mr_free(struct mlx5_mr *mr)
{
if (mr == NULL)
return;
DRV_LOG(DEBUG, "freeing MR(%p):", (void *)mr);
- dereg_mr_cb(&mr->pmd_mr);
+ mlx5_os_dereg_mr(&mr->pmd_mr);
rte_bitmap_free(mr->ms_bmp);
mlx5_free(mr);
}
@@ -545,7 +545,7 @@ mlx5_mr_garbage_collect(struct mlx5_mr_share_cache *share_cache)
struct mlx5_mr *mr = mr_next;
mr_next = LIST_NEXT(mr, mr);
- mlx5_mr_free(mr, share_cache->dereg_mr_cb);
+ mlx5_mr_free(mr);
}
}
@@ -821,7 +821,7 @@ mlx5_mr_create_primary(void *pd,
data.start = RTE_ALIGN_FLOOR(addr, msl->page_sz);
data.end = data.start + msl->page_sz;
rte_mcfg_mem_read_unlock();
- mlx5_mr_free(mr, share_cache->dereg_mr_cb);
+ mlx5_mr_free(mr);
goto alloc_resources;
}
MLX5_ASSERT(data.msl == data_re.msl);
@@ -845,7 +845,7 @@ mlx5_mr_create_primary(void *pd,
* Must be unlocked before calling rte_free() because
* mlx5_mr_mem_event_free_cb() can be called inside.
*/
- mlx5_mr_free(mr, share_cache->dereg_mr_cb);
+ mlx5_mr_free(mr);
return entry->lkey;
}
/*
@@ -912,7 +912,7 @@ mlx5_mr_create_primary(void *pd,
* mlx5_alloc_buf_extern() which eventually calls rte_malloc_socket()
* through mlx5_alloc_verbs_buf().
*/
- share_cache->reg_mr_cb(pd, (void *)data.start, len, &mr->pmd_mr);
+ mlx5_os_reg_mr(pd, (void *)data.start, len, &mr->pmd_mr);
if (mr->pmd_mr.obj == NULL) {
DRV_LOG(DEBUG, "Fail to create an MR for address (%p)",
(void *)addr);
@@ -948,7 +948,7 @@ mlx5_mr_create_primary(void *pd,
* calling rte_free() because mlx5_mr_mem_event_free_cb() can be called
* inside.
*/
- mlx5_mr_free(mr, share_cache->dereg_mr_cb);
+ mlx5_mr_free(mr);
return UINT32_MAX;
}
@@ -1139,9 +1139,6 @@ mlx5_mr_release_cache(struct mlx5_mr_share_cache *share_cache)
int
mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket)
{
- /* Set the reg_mr and dereg_mr callback functions */
- mlx5_os_set_reg_mr_cb(&share_cache->reg_mr_cb,
- &share_cache->dereg_mr_cb);
rte_rwlock_init(&share_cache->rwlock);
rte_rwlock_init(&share_cache->mprwlock);
/* Initialize B-tree and allocate memory for global MR cache table. */
@@ -1189,8 +1186,7 @@ mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl)
* Pointer to MR structure on success, NULL otherwise.
*/
struct mlx5_mr *
-mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id,
- mlx5_reg_mr_t reg_mr_cb)
+mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id)
{
struct mlx5_mr *mr = NULL;
@@ -1199,7 +1195,7 @@ mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id,
RTE_CACHE_LINE_SIZE, socket_id);
if (mr == NULL)
return NULL;
- reg_mr_cb(pd, (void *)addr, len, &mr->pmd_mr);
+ mlx5_os_reg_mr(pd, (void *)addr, len, &mr->pmd_mr);
if (mr->pmd_mr.obj == NULL) {
DRV_LOG(WARNING,
"Fail to create MR for address (%p)",
@@ -1624,14 +1620,13 @@ mlx5_mempool_reg_create(struct rte_mempool *mp, unsigned int mrs_n,
* Whether @p mpr owns its MRs exclusively, i.e. they are not shared.
*/
static void
-mlx5_mempool_reg_destroy(struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mempool_reg *mpr, bool standalone)
+mlx5_mempool_reg_destroy(struct mlx5_mempool_reg *mpr, bool standalone)
{
if (standalone) {
unsigned int i;
for (i = 0; i < mpr->mrs_n; i++)
- share_cache->dereg_mr_cb(&mpr->mrs[i].pmd_mr);
+ mlx5_os_dereg_mr(&mpr->mrs[i].pmd_mr);
mlx5_free(mpr->mrs);
}
mlx5_free(mpr);
@@ -1748,7 +1743,7 @@ mlx5_mr_mempool_register_primary(struct mlx5_mr_share_cache *share_cache,
const struct mlx5_range *range = &ranges[i];
size_t len = range->end - range->start;
- if (share_cache->reg_mr_cb(pd, (void *)range->start, len,
+ if (mlx5_os_reg_mr(pd, (void *)range->start, len,
&mr->pmd_mr) < 0) {
DRV_LOG(ERR,
"Failed to create an MR in PD %p for address range "
@@ -1763,7 +1758,7 @@ mlx5_mr_mempool_register_primary(struct mlx5_mr_share_cache *share_cache,
mp->name);
}
if (i != ranges_n) {
- mlx5_mempool_reg_destroy(share_cache, new_mpr, true);
+ mlx5_mempool_reg_destroy(new_mpr, true);
rte_errno = EINVAL;
goto exit;
}
@@ -1785,13 +1780,13 @@ mlx5_mr_mempool_register_primary(struct mlx5_mr_share_cache *share_cache,
if (mpr != NULL) {
DRV_LOG(DEBUG, "Mempool %s is already registered for PD %p",
mp->name, pd);
- mlx5_mempool_reg_destroy(share_cache, new_mpr, true);
+ mlx5_mempool_reg_destroy(new_mpr, true);
rte_errno = EEXIST;
goto exit;
} else if (old_mpr != NULL) {
DRV_LOG(DEBUG, "Mempool %s registration for PD %p updated for external memory",
mp->name, pd);
- mlx5_mempool_reg_destroy(share_cache, old_mpr, standalone);
+ mlx5_mempool_reg_destroy(old_mpr, standalone);
}
exit:
free(ranges);
@@ -1860,7 +1855,7 @@ mlx5_mr_mempool_unregister_primary(struct mlx5_mr_share_cache *share_cache,
rte_errno = ENOENT;
return -1;
}
- mlx5_mempool_reg_destroy(share_cache, mpr, standalone);
+ mlx5_mempool_reg_destroy(mpr, standalone);
return 0;
}
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 00f3d832c3..5fb931a1b5 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -32,13 +32,6 @@ struct mlx5_pmd_mr {
struct mlx5_devx_obj *mkey; /* devx mkey object. */
};
-/**
- * mr operations typedef
- */
-typedef int (*mlx5_reg_mr_t)(void *pd, void *addr, size_t length,
- struct mlx5_pmd_mr *pmd_mr);
-typedef void (*mlx5_dereg_mr_t)(struct mlx5_pmd_mr *pmd_mr);
-
/* Memory Region object. */
struct mlx5_mr {
LIST_ENTRY(mlx5_mr) mr; /**< Pointer to the prev/next entry. */
@@ -88,8 +81,6 @@ struct __rte_packed_begin mlx5_mr_share_cache {
struct mlx5_mr_list mr_list; /* Registered MR list. */
struct mlx5_mr_list mr_free_list; /* Freed MR list. */
struct mlx5_mempool_reg_list mempool_reg_list; /* Mempool database. */
- mlx5_reg_mr_t reg_mr_cb; /* Callback to reg_mr func */
- mlx5_dereg_mr_t dereg_mr_cb; /* Callback to dereg_mr func */
} __rte_packed_end;
/* Multi-Packet RQ buffer header. */
@@ -233,9 +224,8 @@ struct mlx5_mr *
mlx5_mr_lookup_list(struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr);
struct mlx5_mr *
-mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id,
- mlx5_reg_mr_t reg_mr_cb);
-void mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
+mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id);
+void mlx5_mr_free(struct mlx5_mr *mr);
__rte_internal
uint32_t
mlx5_mr_create(struct mlx5_common_device *cdev,
@@ -246,19 +236,13 @@ __rte_internal
uint32_t
mlx5_mr_addr2mr_bh(struct mlx5_mr_ctrl *mr_ctrl, uintptr_t addr);
-/* mlx5_common_verbs.c */
-
__rte_internal
int
-mlx5_common_verbs_reg_mr(void *pd, void *addr, size_t length,
- struct mlx5_pmd_mr *pmd_mr);
+mlx5_os_reg_mr(void *pd, void *addr, size_t length,
+ struct mlx5_pmd_mr *pmd_mr);
__rte_internal
void
-mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
-
-__rte_internal
-void
-mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
+mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
__rte_internal
struct mlx5_pmd_mr *
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index 692517a9bf..bf1b654da3 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -377,7 +377,8 @@ mlx5_os_umem_dereg(void *pumem)
* @return
* 0 on successful registration, -1 otherwise
*/
-static int
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_reg_mr)
+int
mlx5_os_reg_mr(void *pd,
void *addr, size_t length, struct mlx5_pmd_mr *pmd_mr)
{
@@ -425,7 +426,8 @@ mlx5_os_reg_mr(void *pd,
* @param[in] pmd_mr
* Pointer to PMD mr object
*/
-static void
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_dereg_mr)
+void
mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
{
if (!pmd_mr)
@@ -437,23 +439,6 @@ mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
memset(pmd_mr, 0, sizeof(*pmd_mr));
}
-/**
- * Set the reg_mr and dereg_mr callbacks.
- *
- * @param[out] reg_mr_cb
- * Pointer to reg_mr func
- * @param[out] dereg_mr_cb
- * Pointer to dereg_mr func
- *
- */
-RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_set_reg_mr_cb)
-void
-mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
-{
- *reg_mr_cb = mlx5_os_reg_mr;
- *dereg_mr_cb = mlx5_os_dereg_mr;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_alloc_null_mr)
struct mlx5_pmd_mr *
mlx5_os_alloc_null_mr(struct rte_device *dev, void *pd)
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index e5325c6150..1361dab630 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -117,7 +117,7 @@ mlx5_compress_qp_release(struct rte_compressdev *dev, uint16_t qp_id)
if (qp->opaque_mr.obj != NULL) {
void *opaq = qp->opaque_mr.addr;
- mlx5_common_verbs_dereg_mr(&qp->opaque_mr);
+ mlx5_os_dereg_mr(&qp->opaque_mr);
rte_free(opaq);
}
mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh);
@@ -199,7 +199,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
qp->priv = priv;
qp->ops = (struct rte_comp_op **)RTE_ALIGN((uintptr_t)(qp + 1),
RTE_CACHE_LINE_SIZE);
- if (mlx5_common_verbs_reg_mr(priv->cdev->pd, opaq_buf, qp->entries_n *
+ if (mlx5_os_reg_mr(priv->cdev->pd, opaq_buf, qp->entries_n *
sizeof(union mlx5_gga_compress_opaque),
&qp->opaque_mr) != 0) {
rte_free(opaq_buf);
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index f9f127e9e6..93a2bb2c78 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -40,8 +40,6 @@ struct mlx5_crypto_priv {
TAILQ_ENTRY(mlx5_crypto_priv) next;
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
struct rte_cryptodev *crypto_dev;
- mlx5_reg_mr_t reg_mr_cb; /* Callback to reg_mr func */
- mlx5_dereg_mr_t dereg_mr_cb; /* Callback to dereg_mr func */
struct mlx5_uar uar; /* User Access Region. */
uint32_t max_segs_num; /* Maximum supported data segs. */
uint32_t max_klm_num; /* Maximum supported klm. */
diff --git a/drivers/crypto/mlx5/mlx5_crypto_gcm.c b/drivers/crypto/mlx5/mlx5_crypto_gcm.c
index 89f32c7722..1a2600655a 100644
--- a/drivers/crypto/mlx5/mlx5_crypto_gcm.c
+++ b/drivers/crypto/mlx5/mlx5_crypto_gcm.c
@@ -219,7 +219,6 @@ mlx5_crypto_gcm_mkey_klm_update(struct mlx5_crypto_priv *priv,
static int
mlx5_crypto_gcm_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
{
- struct mlx5_crypto_priv *priv = dev->data->dev_private;
struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
if (qp->umr_qp_obj.qp != NULL)
@@ -231,7 +230,7 @@ mlx5_crypto_gcm_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
if (qp->mr.obj != NULL) {
void *opaq = qp->mr.addr;
- priv->dereg_mr_cb(&qp->mr);
+ mlx5_os_dereg_mr(&qp->mr);
rte_free(opaq);
}
mlx5_crypto_indirect_mkeys_release(qp, qp->entries_n);
@@ -363,7 +362,7 @@ mlx5_crypto_gcm_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
rte_errno = ENOMEM;
goto err;
}
- if (priv->reg_mr_cb(priv->cdev->pd, mr_buf, mr_size, &qp->mr) != 0) {
+ if (mlx5_os_reg_mr(priv->cdev->pd, mr_buf, mr_size, &qp->mr) != 0) {
rte_free(mr_buf);
DRV_LOG(ERR, "Failed to register opaque MR.");
rte_errno = ENOMEM;
@@ -1186,7 +1185,6 @@ mlx5_crypto_gcm_init(struct mlx5_crypto_priv *priv)
/* Override AES-GCM specified ops. */
dev_ops->sym_session_configure = mlx5_crypto_sym_gcm_session_configure;
- mlx5_os_set_reg_mr_cb(&priv->reg_mr_cb, &priv->dereg_mr_cb);
dev_ops->queue_pair_setup = mlx5_crypto_gcm_qp_setup;
dev_ops->queue_pair_release = mlx5_crypto_gcm_qp_release;
if (mlx5_crypto_is_ipsec_opt(priv)) {
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index bd6ef35b53..a4d5392e8f 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -2706,8 +2706,7 @@ int mlx5_aso_cnt_query(struct mlx5_dev_ctx_shared *sh,
int mlx5_aso_ct_queue_init(struct mlx5_dev_ctx_shared *sh,
struct mlx5_aso_ct_pools_mng *ct_mng,
uint32_t nb_queues);
-int mlx5_aso_ct_queue_uninit(struct mlx5_dev_ctx_shared *sh,
- struct mlx5_aso_ct_pools_mng *ct_mng);
+int mlx5_aso_ct_queue_uninit(struct mlx5_aso_ct_pools_mng *ct_mng);
int
mlx5_aso_sq_create(struct mlx5_common_device *cdev, struct mlx5_aso_sq *sq,
void *uar, uint16_t log_desc_n);
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 5e2a81ef9c..cd84ab1966 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -19,17 +19,15 @@
/**
* Free MR resources.
*
- * @param[in] cdev
- * Pointer to the mlx5 common device.
* @param[in] mr
* MR to free.
*/
static void
-mlx5_aso_dereg_mr(struct mlx5_common_device *cdev, struct mlx5_pmd_mr *mr)
+mlx5_aso_dereg_mr(struct mlx5_pmd_mr *mr)
{
void *addr = mr->addr;
- cdev->mr_scache.dereg_mr_cb(mr);
+ mlx5_os_dereg_mr(mr);
mlx5_free(addr);
memset(mr, 0, sizeof(*mr));
}
@@ -59,7 +57,7 @@ mlx5_aso_reg_mr(struct mlx5_common_device *cdev, size_t length,
DRV_LOG(ERR, "Failed to create ASO bits mem for MR.");
return -1;
}
- ret = cdev->mr_scache.reg_mr_cb(cdev->pd, mr->addr, length, mr);
+ ret = mlx5_os_reg_mr(cdev->pd, mr->addr, length, mr);
if (ret) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
mlx5_free(mr->addr);
@@ -362,7 +360,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev, &sh->aso_age_mng->aso_sq,
sh->tx_uar.obj,
MLX5_ASO_QUEUE_LOG_DESC)) {
- mlx5_aso_dereg_mr(cdev, &sh->aso_age_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(&sh->aso_age_mng->aso_sq.mr);
return -1;
}
mlx5_aso_age_init_sq(&sh->aso_age_mng->aso_sq);
@@ -399,14 +397,14 @@ mlx5_aso_queue_uninit(struct mlx5_dev_ctx_shared *sh,
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
- mlx5_aso_dereg_mr(sh->cdev, &sh->aso_age_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(&sh->aso_age_mng->aso_sq.mr);
sq = &sh->aso_age_mng->aso_sq;
break;
case ASO_OPC_MOD_POLICER:
mlx5_aso_mtr_queue_uninit(sh, NULL, &sh->mtrmng->pools_mng);
break;
case ASO_OPC_MOD_CONNECTION_TRACKING:
- mlx5_aso_ct_queue_uninit(sh, sh->ct_mng);
+ mlx5_aso_ct_queue_uninit(sh->ct_mng);
break;
default:
DRV_LOG(ERR, "Unknown ASO operation mode");
@@ -1147,15 +1145,14 @@ __mlx5_aso_ct_get_pool(struct mlx5_dev_ctx_shared *sh,
}
int
-mlx5_aso_ct_queue_uninit(struct mlx5_dev_ctx_shared *sh,
- struct mlx5_aso_ct_pools_mng *ct_mng)
+mlx5_aso_ct_queue_uninit(struct mlx5_aso_ct_pools_mng *ct_mng)
{
uint32_t i;
/* 64B per object for query. */
for (i = 0; i < ct_mng->nb_sq; i++) {
if (ct_mng->aso_sqs[i].mr.addr)
- mlx5_aso_dereg_mr(sh->cdev, &ct_mng->aso_sqs[i].mr);
+ mlx5_aso_dereg_mr(&ct_mng->aso_sqs[i].mr);
mlx5_aso_destroy_sq(&ct_mng->aso_sqs[i]);
}
return 0;
@@ -1197,7 +1194,7 @@ mlx5_aso_ct_queue_init(struct mlx5_dev_ctx_shared *sh,
error:
do {
if (ct_mng->aso_sqs[i].mr.addr)
- mlx5_aso_dereg_mr(sh->cdev, &ct_mng->aso_sqs[i].mr);
+ mlx5_aso_dereg_mr(&ct_mng->aso_sqs[i].mr);
mlx5_aso_destroy_sq(&ct_mng->aso_sqs[i]);
} while (i--);
ct_mng->nb_sq = 0;
diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c
index b6bb9f12a6..7cc601d681 100644
--- a/drivers/net/mlx5/mlx5_flow_hw.c
+++ b/drivers/net/mlx5/mlx5_flow_hw.c
@@ -11086,12 +11086,9 @@ flow_hw_create_nic_ctrl_tables(struct rte_eth_dev *dev, struct rte_flow_error *e
}
static void
-flow_hw_ct_mng_destroy(struct rte_eth_dev *dev,
- struct mlx5_aso_ct_pools_mng *ct_mng)
+flow_hw_ct_mng_destroy(struct mlx5_aso_ct_pools_mng *ct_mng)
{
- struct mlx5_priv *priv = dev->data->dev_private;
-
- mlx5_aso_ct_queue_uninit(priv->sh, ct_mng);
+ mlx5_aso_ct_queue_uninit(ct_mng);
mlx5_free(ct_mng);
}
@@ -11230,7 +11227,7 @@ mlx5_flow_ct_init(struct rte_eth_dev *dev,
priv->hws_ctpool = NULL;
}
if (priv->ct_mng) {
- flow_hw_ct_mng_destroy(dev, priv->ct_mng);
+ flow_hw_ct_mng_destroy(priv->ct_mng);
priv->ct_mng = NULL;
}
return ret;
@@ -11804,7 +11801,7 @@ __mlx5_flow_hw_resource_release(struct rte_eth_dev *dev, bool ctx_close)
priv->hws_ctpool = NULL;
}
if (priv->ct_mng) {
- flow_hw_ct_mng_destroy(dev, priv->ct_mng);
+ flow_hw_ct_mng_destroy(priv->ct_mng);
priv->ct_mng = NULL;
}
mlx5_flow_quota_destroy(dev);
diff --git a/drivers/net/mlx5/mlx5_flow_quota.c b/drivers/net/mlx5/mlx5_flow_quota.c
index d94167d0b0..b661bd376e 100644
--- a/drivers/net/mlx5/mlx5_flow_quota.c
+++ b/drivers/net/mlx5/mlx5_flow_quota.c
@@ -412,12 +412,11 @@ mlx5_quota_alloc_sq(struct mlx5_priv *priv)
static void
mlx5_quota_destroy_read_buf(struct mlx5_priv *priv)
{
- struct mlx5_dev_ctx_shared *sh = priv->sh;
struct mlx5_quota_ctx *qctx = &priv->quota_ctx;
if (qctx->mr.lkey) {
void *addr = qctx->mr.addr;
- sh->cdev->mr_scache.dereg_mr_cb(&qctx->mr);
+ mlx5_os_dereg_mr(&qctx->mr);
mlx5_free(addr);
}
if (qctx->read_buf)
@@ -446,8 +445,7 @@ mlx5_quota_alloc_read_buf(struct mlx5_priv *priv)
DRV_LOG(DEBUG, "QUOTA: failed to allocate MTR ASO READ buffer [1]");
return -ENOMEM;
}
- ret = sh->cdev->mr_scache.reg_mr_cb(sh->cdev->pd, buf,
- rd_buf_size, &qctx->mr);
+ ret = mlx5_os_reg_mr(sh->cdev->pd, buf, rd_buf_size, &qctx->mr);
if (ret) {
DRV_LOG(DEBUG, "QUOTA: failed to register MTR ASO READ MR");
return -errno;
diff --git a/drivers/net/mlx5/mlx5_hws_cnt.c b/drivers/net/mlx5/mlx5_hws_cnt.c
index 1b6acb7a3b..d0c4ead71b 100644
--- a/drivers/net/mlx5/mlx5_hws_cnt.c
+++ b/drivers/net/mlx5/mlx5_hws_cnt.c
@@ -259,12 +259,11 @@ mlx5_hws_aging_check(struct mlx5_priv *priv, struct mlx5_hws_cnt_pool *cpool)
}
static void
-mlx5_hws_cnt_raw_data_free(struct mlx5_dev_ctx_shared *sh,
- struct mlx5_hws_cnt_raw_data_mng *mng)
+mlx5_hws_cnt_raw_data_free(struct mlx5_hws_cnt_raw_data_mng *mng)
{
if (mng == NULL)
return;
- sh->cdev->mr_scache.dereg_mr_cb(&mng->mr);
+ mlx5_os_dereg_mr(&mng->mr);
mlx5_free(mng->raw);
mlx5_free(mng);
}
@@ -296,8 +295,7 @@ mlx5_hws_cnt_raw_data_alloc(struct mlx5_dev_ctx_shared *sh, uint32_t n,
NULL, "failed to allocate raw counters memory");
goto error;
}
- ret = sh->cdev->mr_scache.reg_mr_cb(sh->cdev->pd, mng->raw, sz,
- &mng->mr);
+ ret = mlx5_os_reg_mr(sh->cdev->pd, mng->raw, sz, &mng->mr);
if (ret) {
rte_flow_error_set(error, errno,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
@@ -306,7 +304,7 @@ mlx5_hws_cnt_raw_data_alloc(struct mlx5_dev_ctx_shared *sh, uint32_t n,
}
return mng;
error:
- mlx5_hws_cnt_raw_data_free(sh, mng);
+ mlx5_hws_cnt_raw_data_free(mng);
return NULL;
}
@@ -639,8 +637,7 @@ mlx5_hws_cnt_pool_dcs_alloc(struct mlx5_dev_ctx_shared *sh,
}
static void
-mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
- struct mlx5_hws_cnt_pool *cpool)
+mlx5_hws_cnt_pool_dcs_free(struct mlx5_hws_cnt_pool *cpool)
{
uint32_t idx;
@@ -649,7 +646,7 @@ mlx5_hws_cnt_pool_dcs_free(struct mlx5_dev_ctx_shared *sh,
for (idx = 0; idx < MLX5_HWS_CNT_DCS_NUM; idx++)
mlx5_devx_cmd_destroy(cpool->dcs_mng.dcs[idx].obj);
if (cpool->raw_mng) {
- mlx5_hws_cnt_raw_data_free(sh, cpool->raw_mng);
+ mlx5_hws_cnt_raw_data_free(cpool->raw_mng);
cpool->raw_mng = NULL;
}
}
@@ -842,8 +839,8 @@ mlx5_hws_cnt_pool_destroy(struct mlx5_dev_ctx_shared *sh,
}
mlx5_hws_cnt_pool_action_destroy(cpool);
if (cpool->cfg.host_cpool == NULL) {
- mlx5_hws_cnt_pool_dcs_free(sh, cpool);
- mlx5_hws_cnt_raw_data_free(sh, cpool->raw_mng);
+ mlx5_hws_cnt_pool_dcs_free(cpool);
+ mlx5_hws_cnt_raw_data_free(cpool->raw_mng);
}
mlx5_free((void *)cpool->cfg.name);
mlx5_hws_cnt_pool_deinit(cpool);
--
2.54.0
^ permalink raw reply related
* [PATCH v8 6/9] net/mlx5: support selective Rx
From: Thomas Monjalon @ 2026-06-04 19:30 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, Gregory Etelson, Dariusz Sosnowski,
Viacheslav Ovsiienko, Bing Zhao, Ori Kam, Suanming Mou,
Matan Azrad
In-Reply-To: <20260604193324.1996141-1-thomas@monjalon.net>
From: Gregory Etelson <getelson@nvidia.com>
Selective Rx may save some PCI bandwidth.
Implement selective Rx in the (quite slow) scalar SPRQ Rx path
mlx5_rx_burst() where the performance impact
of the added condition branches is acceptable.
Other Rx functions do not support this feature.
When using selective Rx, mlx5_rx_burst will be selected.
A null Memory Region (MR) is always allocated
at shared device context initialization.
The selective Rx capability is not advertised
if this special MR allocation fails.
For each Rx segment configured with a NULL mempool,
a "null mbuf" is created.
It is a fake mbuf allocated outside any mempool,
used as a placeholder in the Rx ring.
The null MR lkey is used in the WQE for these segments
so the NIC writes received data to a discard buffer.
The mbuf data room size is resolved from the first segment having a pool.
For null segments, the buffer length is from the last seen pool,
so that the WQE stride size remains consistent.
In mlx5_rx_burst, discarded segments are not chained
into the packet mbuf list, NB_SEGS is decremented accordingly,
and no replacement buffer is allocated.
A separate data_seg_len accumulator tracks the total length
of delivered segments only.
The packet length is adjusted to reflect only the data
actually delivered to the application.
Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
doc/guides/nics/features/mlx5.ini | 1 +
doc/guides/nics/mlx5.rst | 86 +++++++++---
doc/guides/rel_notes/release_26_07.rst | 4 +
drivers/net/mlx5/mlx5.c | 7 +
drivers/net/mlx5/mlx5.h | 1 +
drivers/net/mlx5/mlx5_ethdev.c | 25 ++++
drivers/net/mlx5/mlx5_rx.c | 187 +++++++++++++++----------
drivers/net/mlx5/mlx5_rx.h | 1 +
drivers/net/mlx5/mlx5_rxq.c | 95 +++++++++----
drivers/net/mlx5/mlx5_trigger.c | 64 +++++++--
10 files changed, 330 insertions(+), 141 deletions(-)
diff --git a/doc/guides/nics/features/mlx5.ini b/doc/guides/nics/features/mlx5.ini
index 3b3eda28b8..ae8c83057b 100644
--- a/doc/guides/nics/features/mlx5.ini
+++ b/doc/guides/nics/features/mlx5.ini
@@ -16,6 +16,7 @@ Burst mode info = Y
Power mgmt address monitor = Y
MTU update = Y
Buffer split on Rx = Y
+Selective Rx = Y
Scattered Rx = Y
LRO = Y
TSO = Y
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 00bfb31370..afbf040e66 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -84,6 +84,9 @@ The Rx / Tx data path use different techniques to offer the best performance.
with :ref:`multi-packet Rx queues (MPRQ) <mlx5_mprq_params>`.
This feature is disabled by default.
+- Some PCI bandwidth is saved by receiving partial packets
+ with :ref:`selective Rx <mlx5_selective_rx>`.
+
More details about Rx implementations and their configurations are provided
in the chapter about :ref:`mlx5_rx_functions`.
@@ -879,6 +882,8 @@ MLX5 supports various methods to report statistics:
Basic port statistics can be queried using ``rte_eth_stats_get()``.
The received and sent statistics are through SW only
and counts the number of packets received or sent successfully by the PMD.
+In the case of :ref:`selective Rx <mlx5_selective_rx>`,
+the ``ibytes`` counter matches segments delivered, not the skipped ones.
The ``imissed`` counter is the amount of packets that could not be delivered
to SW because a queue was full.
Packets not received due to congestion in the bus or on the NIC
@@ -992,25 +997,26 @@ These configurations may also have an impact on the behavior:
.. table:: Rx burst functions
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- || Function Name || Parameters to Enable || Scatter|| Error Recovery || CQE || Large|| Shared |
- | | | | || comp|| MTU | RxQ |
- +===================+========================+=========+=================+======+=======+=========+
- | rx_burst | rx_vec_en=0 | Yes | Yes | Yes | Yes | No |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- | rx_burst_vec | rx_vec_en=1 (default) | No | if CQE comp off | Yes | No | No |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- | rx_burst_mprq || mprq_en=1 | No | Yes | Yes | Yes | No |
- | || RxQs >= rxqs_min_mprq | | | | | |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- | rx_burst_mprq_vec || rx_vec_en=1 (default) | No | if CQE comp off | Yes | Yes | No |
- | || mprq_en=1 | | | | | |
- | || RxQs >= rxqs_min_mprq | | | | | |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
- | rx_burst | at least one Rx queue | Yes | Yes | Yes | Yes | Yes |
- | (out of order) | on the device | | | | | |
- | | is shared | | | | | |
- +-------------------+------------------------+---------+-----------------+------+-------+---------+
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ || Function|| Parameters to Enable || Scatter|| Selec-|| Error || CQE || Large|| Shared|
+ || Name | | || tive || Recovery|| comp|| MTU || RxQ |
+ +==========+=======================+=========+========+==========+======+=======+========+
+ | rx_burst | rx_vec_en=0 | Yes | Yes | Yes | Yes | Yes | No |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ | _vec | rx_vec_en=1 (default) | No | No || if CQE | Yes | No | No |
+ | | | | || comp off| | | |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ | _mprq || mprq_en=1 | No | No | Yes | Yes | Yes | No |
+ | || RxQs >= rxqs_min_mprq| | | | | | |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ | _mprq_vec|| rx_vec_en=1 (default)| No | No || if CQE | Yes | Yes | No |
+ | || mprq_en=1 | | || comp off| | | |
+ | || RxQs >= rxqs_min_mprq| | | | | | |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
+ || _out_of || at least one Rx queue| Yes | No | Yes | Yes | Yes | Yes |
+ || _order || on the device | | | | | | |
+ | || is shared | | | | | | |
+ +----------+-----------------------+---------+--------+----------+------+-------+--------+
Rx/Tx Tuning
@@ -1105,13 +1111,14 @@ Rx interrupt X
:ref:`Rx threshold <mlx5_rx_threshold>` X X
:ref:`Rx drop delay <mlx5_drop>` X X
:ref:`Rx timestamp <mlx5_rx_timstp>` X X
+:ref:`buffer split <mlx5_buf_split>` X X
+:ref:`selective Rx <mlx5_selective_rx>` X
+:ref:`multi-segment <mlx5_multiseg>` X X
:ref:`Tx scheduling <mlx5_tx_sched>` X
:ref:`Tx rate limit <mlx5_rate_limit>` X
:ref:`Tx inline <mlx5_tx_inline>` X X
:ref:`Tx fast free <mlx5_tx_fast_free>` X X
:ref:`Tx affinity <mlx5_aggregated>` X
-:ref:`buffer split <mlx5_buf_split>` X X
-:ref:`multi-segment <mlx5_multiseg>` X X
promiscuous X X
multicast promiscuous X X
multiple MAC addresses X
@@ -2248,13 +2255,50 @@ OFED 5.1-2
DPDK 20.11
========= ==========
+Runtime configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+The offload flag ``RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT`` is required.
+
+When calling ``rte_eth_rx_queue_setup()``,
+the input ``rte_eth_rxconf::rx_seg`` defines the configuration of the segments,
+mainly offset and length.
+
Limitations
^^^^^^^^^^^
+#. Splitting per protocol header is not supported.
+
#. Buffer split offload is supported with regular Rx burst routine only,
no MPRQ feature or vectorized code can be engaged.
+.. _mlx5_selective_rx:
+
+Selective Rx
+~~~~~~~~~~~~
+
+Some PCI bandwidth can be saved
+by :ref:`skipping some parts of Rx data <nic_features_selective_rx>`.
+It is enabled when using :ref:`buffer split <mlx5_buf_split>`
+and configuring no mempool in some segments to discard.
+
+Runtime configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+The offload flag ``RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT`` is required.
+
+When calling ``rte_eth_rx_queue_setup()``,
+the segment to discard (``rte_eth_rxconf::rx_seg::split``)
+is marked by the absence of mempool (``mp = NULL``).
+
+Limitations
+^^^^^^^^^^^
+
+#. Selective Rx is supported with regular Rx burst routine only,
+ no MPRQ feature or vectorized code can be engaged.
+
+
.. _mlx5_multiseg:
Multi-Segment Scatter/Gather
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 46a8fe2cc1..0ac9816a85 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -103,6 +103,10 @@ New Features
* Added support for transmitting LLDP packets based on mbuf packet type.
* Implemented AVX2 context descriptor transmit paths.
+* **Updated NVIDIA mlx5 ethernet driver.**
+
+ * Added support for selective Rx in scalar SPRQ Rx path.
+
* **Updated PCAP ethernet driver.**
* Added support for VLAN insertion and stripping.
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index f190654756..61c26d1206 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1975,6 +1975,9 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
/* Init counter pool list header and lock. */
LIST_INIT(&sh->hws_cpool_list);
rte_spinlock_init(&sh->cpool_lock);
+ sh->null_mr = mlx5_os_alloc_null_mr(sh->cdev->dev, sh->cdev->pd);
+ if (!sh->null_mr)
+ DRV_LOG(DEBUG, "Fail to initialize NULL MR, selective Rx is disabled.");
exit:
pthread_mutex_unlock(&mlx5_dev_ctx_list_mutex);
return sh;
@@ -2139,6 +2142,10 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
MLX5_ASSERT(sh->geneve_tlv_option_resource == NULL);
pthread_mutex_destroy(&sh->txpp.mutex);
mlx5_lwm_unset(sh);
+ if (sh->null_mr) {
+ mlx5_os_free_null_mr(sh->null_mr);
+ sh->null_mr = NULL;
+ }
mlx5_physical_device_destroy(sh->phdev);
mlx5_free(sh);
return;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 92a00cfaa8..bd6ef35b53 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1674,6 +1674,7 @@ struct mlx5_dev_ctx_shared {
rte_spinlock_t cpool_lock;
LIST_HEAD(hws_cpool_list, mlx5_hws_cnt_pool) hws_cpool_list; /* Count pool list. */
struct mlx5_dev_registers registers;
+ struct mlx5_pmd_mr *null_mr;
struct mlx5_dev_shared_port port[]; /* per device port data array. */
};
diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c
index a29cdeeb50..7b7536fa1e 100644
--- a/drivers/net/mlx5/mlx5_ethdev.c
+++ b/drivers/net/mlx5/mlx5_ethdev.c
@@ -381,6 +381,7 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)
info->rx_seg_capa.multi_pools = !priv->config.mprq.enabled;
info->rx_seg_capa.offset_allowed = !priv->config.mprq.enabled;
info->rx_seg_capa.offset_align_log2 = 0;
+ info->rx_seg_capa.selective_rx = !!priv->sh->null_mr;
info->rx_offload_capa = (mlx5_get_rx_port_offloads() |
info->rx_queue_offload_capa);
info->tx_offload_capa = mlx5_get_tx_port_offloads(dev);
@@ -708,6 +709,25 @@ mlx5_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
return -rte_errno;
}
+static bool
+mlx5_selective_rx_enabled(struct rte_eth_dev *dev)
+{
+ struct mlx5_priv *priv = dev->data->dev_private;
+
+ for (uint32_t q = 0; q < priv->rxqs_n; ++q) {
+ struct mlx5_rxq_ctrl *rxq_ctrl = mlx5_rxq_ctrl_get(dev, q);
+
+ if (rxq_ctrl == NULL || rxq_ctrl->is_hairpin)
+ continue;
+ for (uint16_t s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
+ if (rxq_ctrl->rxq.rxseg[s].mp == NULL)
+ return true;
+ }
+ }
+
+ return false;
+}
+
/**
* Configure the RX function to use.
*
@@ -723,6 +743,11 @@ mlx5_select_rx_function(struct rte_eth_dev *dev)
eth_rx_burst_t rx_pkt_burst = mlx5_rx_burst;
MLX5_ASSERT(dev != NULL);
+ if (mlx5_selective_rx_enabled(dev)) {
+ DRV_LOG(DEBUG, "port %u forced to scalar SPRQ Rx (selective Rx configured)",
+ dev->data->port_id);
+ return rx_pkt_burst;
+ }
if (mlx5_shared_rq_enabled(dev)) {
rx_pkt_burst = mlx5_rx_burst_out_of_order;
DRV_LOG(DEBUG, "port %u forced to use SPRQ"
diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c
index 185bfd4fff..9812bc7929 100644
--- a/drivers/net/mlx5/mlx5_rx.c
+++ b/drivers/net/mlx5/mlx5_rx.c
@@ -486,7 +486,7 @@ mlx5_rxq_initialize(struct mlx5_rxq_data *rxq)
rxq->wqes)[i];
addr = rte_pktmbuf_mtod(buf, uintptr_t);
byte_count = DATA_LEN(buf);
- lkey = mlx5_rx_mb2mr(rxq, buf);
+ lkey = buf->pool ? mlx5_rx_mb2mr(rxq, buf) : rxq->sh->null_mr->lkey;
}
/* scat->addr must be able to store a pointer. */
MLX5_ASSERT(sizeof(scat->addr) >= sizeof(uintptr_t));
@@ -1044,11 +1044,14 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
const unsigned int sges_n = rxq->sges_n;
struct rte_mbuf *pkt = NULL;
struct rte_mbuf *seg = NULL;
+ struct rte_mbuf *tail = NULL;
volatile struct mlx5_cqe *cqe =
&(*rxq->cqes)[rxq->cq_ci & cqe_mask];
+ volatile struct mlx5_mini_cqe8 *mcqe = NULL;
unsigned int i = 0;
unsigned int rq_ci = rxq->rq_ci << sges_n;
int len = 0; /* keep its value across iterations. */
+ uint32_t data_seg_len = 0;
while (pkts_n) {
uint16_t skip_cnt;
@@ -1056,105 +1059,137 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
volatile struct mlx5_wqe_data_seg *wqe =
&((volatile struct mlx5_wqe_data_seg *)rxq->wqes)[idx];
struct rte_mbuf *rep = (*rxq->elts)[idx];
- volatile struct mlx5_mini_cqe8 *mcqe = NULL;
- if (pkt)
- NEXT(seg) = rep;
+ if (pkt) {
+ if (rep->pool)
+ NEXT(tail) = rep;
+ else
+ --NB_SEGS(pkt);
+ }
seg = rep;
rte_prefetch0(seg);
rte_prefetch0(cqe);
rte_prefetch0(wqe);
- /* Allocate the buf from the same pool. */
- rep = rte_mbuf_raw_alloc(seg->pool);
- if (unlikely(rep == NULL)) {
- ++rxq->stats.rx_nombuf;
- if (!pkt) {
- /*
- * no buffers before we even started,
- * bail out silently.
- */
- break;
- }
- while (pkt != seg) {
- MLX5_ASSERT(pkt != (*rxq->elts)[idx]);
- rep = NEXT(pkt);
- NEXT(pkt) = NULL;
- NB_SEGS(pkt) = 1;
- rte_mbuf_raw_free(pkt);
- pkt = rep;
- }
- rq_ci >>= sges_n;
- ++rq_ci;
- rq_ci <<= sges_n;
- break;
- }
- if (!pkt) {
- cqe = &(*rxq->cqes)[rxq->cq_ci & cqe_mask];
- len = mlx5_rx_poll_len(rxq, cqe, cqe_n, cqe_mask,
- &mcqe, &skip_cnt, false, NULL);
- if (unlikely(len & MLX5_ERROR_CQE_MASK)) {
- /* We drop packets with non-critical errors */
- rte_mbuf_raw_free(rep);
- if (len == MLX5_CRITICAL_ERROR_CQE_RET) {
- rq_ci = rxq->rq_ci << sges_n;
+ if (seg->pool) {
+ /* Allocate the buf from the same pool. */
+ rep = rte_mbuf_raw_alloc(seg->pool);
+ if (unlikely(rep == NULL)) {
+ ++rxq->stats.rx_nombuf;
+ if (!pkt) {
+ /*
+ * no buffers before we even started,
+ * bail out silently.
+ */
break;
}
- /* Skip specified amount of error CQEs packets */
+ while (pkt != seg) {
+ MLX5_ASSERT(pkt != (*rxq->elts)[idx]);
+ rep = NEXT(pkt);
+ NEXT(pkt) = NULL;
+ NB_SEGS(pkt) = 1;
+ rte_mbuf_raw_free(pkt);
+ pkt = rep;
+ }
rq_ci >>= sges_n;
- rq_ci += skip_cnt;
+ ++rq_ci;
rq_ci <<= sges_n;
- MLX5_ASSERT(!pkt);
- continue;
- }
- if (len == 0) {
- rte_mbuf_raw_free(rep);
break;
}
- pkt = seg;
- MLX5_ASSERT(len >= (int)(rxq->crc_present << 2));
- pkt->ol_flags &= RTE_MBUF_F_EXTERNAL;
- if (rxq->cqe_comp_layout && mcqe)
- cqe = &rxq->title_cqe;
- rxq_cq_to_mbuf(rxq, pkt, cqe, mcqe);
- if (rxq->crc_present)
- len -= RTE_ETHER_CRC_LEN;
- PKT_LEN(pkt) = len;
- if (cqe->lro_num_seg > 1) {
- mlx5_lro_update_hdr
- (rte_pktmbuf_mtod(pkt, uint8_t *), cqe,
- mcqe, rxq, len);
- pkt->ol_flags |= RTE_MBUF_F_RX_LRO;
- pkt->tso_segsz = len / cqe->lro_num_seg;
+ }
+ if (!pkt) { /* new packet */
+ if (len == 0) { /* no CQE polled yet */
+ mcqe = NULL;
+ cqe = &(*rxq->cqes)[rxq->cq_ci & cqe_mask];
+ len = mlx5_rx_poll_len(rxq, cqe, cqe_n, cqe_mask,
+ &mcqe, &skip_cnt, false, NULL);
+ if (unlikely(len & MLX5_ERROR_CQE_MASK)) {
+ /* We drop packets with non-critical errors */
+ if (seg->pool)
+ rte_mbuf_raw_free(rep);
+ if (len == MLX5_CRITICAL_ERROR_CQE_RET) {
+ rq_ci = rxq->rq_ci << sges_n;
+ break;
+ }
+ /* Skip specified amount of error CQEs packets */
+ rq_ci >>= sges_n;
+ rq_ci += skip_cnt;
+ rq_ci <<= sges_n;
+ MLX5_ASSERT(!pkt);
+ len = 0;
+ continue;
+ }
+ if (len == 0) {
+ if (seg->pool)
+ rte_mbuf_raw_free(rep);
+ break;
+ }
+ MLX5_ASSERT(len >= (int)(rxq->crc_present << 2));
+ if (rxq->crc_present)
+ len -= RTE_ETHER_CRC_LEN;
+ }
+ if (seg->pool) { /* first real segment */
+ pkt = seg;
+ pkt->ol_flags &= RTE_MBUF_F_EXTERNAL;
+ if (rxq->cqe_comp_layout && mcqe)
+ cqe = &rxq->title_cqe;
+ rxq_cq_to_mbuf(rxq, pkt, cqe, mcqe);
+ PKT_LEN(pkt) = len;
+ if (cqe->lro_num_seg > 1) {
+ mlx5_lro_update_hdr
+ (rte_pktmbuf_mtod(pkt, uint8_t *), cqe,
+ mcqe, rxq, len);
+ pkt->ol_flags |= RTE_MBUF_F_RX_LRO;
+ pkt->tso_segsz = len / cqe->lro_num_seg;
+ }
}
}
- DATA_LEN(rep) = DATA_LEN(seg);
- PKT_LEN(rep) = PKT_LEN(seg);
- SET_DATA_OFF(rep, DATA_OFF(seg));
- PORT(rep) = PORT(seg);
- (*rxq->elts)[idx] = rep;
- /*
- * Fill NIC descriptor with the new buffer. The lkey and size
- * of the buffers are already known, only the buffer address
- * changes.
- */
- wqe->addr = rte_cpu_to_be_64(rte_pktmbuf_mtod(rep, uintptr_t));
- /* If there's only one MR, no need to replace LKey in WQE. */
- if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1))
- wqe->lkey = mlx5_rx_mb2mr(rxq, rep);
- if (len > DATA_LEN(seg)) {
+ if (seg->pool) { /* real segment: replenish WQE */
+ tail = seg;
+ DATA_LEN(rep) = DATA_LEN(seg);
+ PKT_LEN(rep) = PKT_LEN(seg);
+ SET_DATA_OFF(rep, DATA_OFF(seg));
+ PORT(rep) = PORT(seg);
+ (*rxq->elts)[idx] = rep;
+ /*
+ * Fill NIC descriptor with the new buffer. The lkey and size
+ * of the buffers are already known, only the buffer address
+ * changes.
+ */
+ wqe->addr = rte_cpu_to_be_64(rte_pktmbuf_mtod(rep, uintptr_t));
+ /* If there's only one MR, no need to replace LKey in WQE. */
+ if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1))
+ wqe->lkey = mlx5_rx_mb2mr(rxq, rep);
+ }
+ if (len > DATA_LEN(seg)) { /* more data: move to next segment */
+ if (seg->pool)
+ data_seg_len += DATA_LEN(seg);
len -= DATA_LEN(seg);
- ++NB_SEGS(pkt);
+ if (pkt)
+ ++NB_SEGS(pkt);
++rq_ci;
continue;
}
- DATA_LEN(seg) = len;
+ if (seg->pool) { /* last segment */
+ DATA_LEN(seg) = len;
+ data_seg_len += len;
+ }
+ if (unlikely(!pkt)) { /* no real segment found, skip packet */
+ len = 0;
+ rq_ci >>= sges_n;
+ ++rq_ci;
+ rq_ci <<= sges_n;
+ continue;
+ }
+ PKT_LEN(pkt) = RTE_MIN(PKT_LEN(pkt), data_seg_len);
#ifdef MLX5_PMD_SOFT_COUNTERS
/* Increment bytes counter. */
rxq->stats.ibytes += PKT_LEN(pkt);
#endif
+ data_seg_len = 0;
/* Return packet. */
*(pkts++) = pkt;
pkt = NULL;
+ len = 0;
--pkts_n;
++i;
/* Align consumer index to the next stride. */
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 01b563d981..cd48ee37ef 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -96,6 +96,7 @@ struct mlx5_eth_rxseg {
uint16_t length; /**< Segment data length, configures split point. */
uint16_t offset; /**< Data offset from beginning of mbuf data buffer. */
uint32_t reserved; /**< Reserved field. */
+ struct rte_mbuf *null_mbuf; /**< For selective Rx. */
};
/* RX queue descriptor. */
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 48d982a8c2..25dba7f4d9 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -151,26 +151,30 @@ rxq_alloc_elts_sprq(struct mlx5_rxq_ctrl *rxq_ctrl)
struct mlx5_eth_rxseg *seg = &rxq_ctrl->rxq.rxseg[i % sges_n];
struct rte_mbuf *buf;
- buf = rte_pktmbuf_alloc(seg->mp);
- if (buf == NULL) {
- if (rxq_ctrl->share_group == 0)
- DRV_LOG(ERR, "port %u queue %u empty mbuf pool",
- RXQ_PORT_ID(rxq_ctrl),
- rxq_ctrl->rxq.idx);
- else
- DRV_LOG(ERR, "share group %u queue %u empty mbuf pool",
- rxq_ctrl->share_group,
- rxq_ctrl->share_qid);
- rte_errno = ENOMEM;
- goto error;
+ if (seg->mp) {
+ buf = rte_pktmbuf_alloc(seg->mp);
+ if (buf == NULL) {
+ if (rxq_ctrl->share_group == 0)
+ DRV_LOG(ERR, "port %u queue %u empty mbuf pool",
+ RXQ_PORT_ID(rxq_ctrl),
+ rxq_ctrl->rxq.idx);
+ else
+ DRV_LOG(ERR, "share group %u queue %u empty mbuf pool",
+ rxq_ctrl->share_group,
+ rxq_ctrl->share_qid);
+ rte_errno = ENOMEM;
+ goto error;
+ }
+ /* Only vectored Rx routines rely on headroom size. */
+ MLX5_ASSERT(!has_vec_support ||
+ DATA_OFF(buf) >= RTE_PKTMBUF_HEADROOM);
+ /* Buffer is supposed to be empty. */
+ MLX5_ASSERT(rte_pktmbuf_data_len(buf) == 0);
+ MLX5_ASSERT(rte_pktmbuf_pkt_len(buf) == 0);
+ MLX5_ASSERT(!buf->next);
+ } else {
+ buf = seg->null_mbuf;
}
- /* Only vectored Rx routines rely on headroom size. */
- MLX5_ASSERT(!has_vec_support ||
- DATA_OFF(buf) >= RTE_PKTMBUF_HEADROOM);
- /* Buffer is supposed to be empty. */
- MLX5_ASSERT(rte_pktmbuf_data_len(buf) == 0);
- MLX5_ASSERT(rte_pktmbuf_pkt_len(buf) == 0);
- MLX5_ASSERT(!buf->next);
SET_DATA_OFF(buf, seg->offset);
PORT(buf) = rxq_ctrl->rxq.port_id;
DATA_LEN(buf) = seg->length;
@@ -324,10 +328,14 @@ rxq_free_elts_sprq(struct mlx5_rxq_ctrl *rxq_ctrl)
rxq->rq_pi = elts_ci;
}
for (i = 0; i != q_n; ++i) {
- if ((*rxq->elts)[i] != NULL)
+ if ((*rxq->elts)[i] != NULL && (*rxq->elts)[i]->pool != NULL)
rte_pktmbuf_free_seg((*rxq->elts)[i]);
(*rxq->elts)[i] = NULL;
}
+ for (i = 0; i < rxq->rxseg_n; i++) {
+ mlx5_free(rxq->rxseg[i].null_mbuf);
+ rxq->rxseg[i].null_mbuf = NULL;
+ }
}
/**
@@ -1815,7 +1823,9 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
int ret;
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_rxq_ctrl *tmpl;
- unsigned int mb_len = rte_pktmbuf_data_room_size(rx_seg[0].mp);
+ struct rte_mempool *first_mp = NULL;
+ struct rte_mempool *last_mp = NULL;
+ unsigned int mb_len;
struct mlx5_port_config *config = &priv->config;
uint64_t offloads = conf->offloads |
dev->data->dev_conf.rxmode.offloads;
@@ -1827,7 +1837,7 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
unsigned int non_scatter_min_mbuf_size = max_rx_pktlen +
RTE_PKTMBUF_HEADROOM;
unsigned int max_lro_size = 0;
- unsigned int first_mb_free_size = mb_len - RTE_PKTMBUF_HEADROOM;
+ unsigned int first_mb_free_size;
uint32_t mprq_log_actual_stride_num = 0;
uint32_t mprq_log_actual_stride_size = 0;
bool rx_seg_en = n_seg != 1 || rx_seg[0].offset || rx_seg[0].length;
@@ -1845,6 +1855,21 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
const struct rte_eth_rxseg_split *qs_seg = rx_seg;
unsigned int tail_len;
+ /* Find first segment with a mempool. */
+ for (uint16_t seg = 0; seg < n_seg; seg++) {
+ if (rx_seg[seg].mp != NULL) {
+ first_mp = rx_seg[seg].mp;
+ break;
+ }
+ }
+ if (first_mp == NULL) {
+ DRV_LOG(ERR, "port %u Rx queue %u has no mempool", dev->data->port_id, idx);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ mb_len = rte_pktmbuf_data_room_size(first_mp);
+ first_mb_free_size = mb_len - RTE_PKTMBUF_HEADROOM;
+
if (mprq_en) {
/* Trim the number of descs needed. */
desc >>= mprq_log_actual_stride_num;
@@ -1884,35 +1909,44 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
do {
struct mlx5_eth_rxseg *hw_seg =
&tmpl->rxq.rxseg[tmpl->rxq.rxseg_n];
- uint32_t buf_len, offset, seg_len;
+ uint32_t buf_len = 0, offset, seg_len;
/*
* For the buffers beyond descriptions offset is zero,
* the first buffer contains head room.
*/
- buf_len = rte_pktmbuf_data_room_size(qs_seg->mp);
+ if (qs_seg->mp != NULL) {
+ last_mp = qs_seg->mp;
+ buf_len = rte_pktmbuf_data_room_size(qs_seg->mp);
+ } else if (last_mp != NULL) {
+ buf_len = rte_pktmbuf_data_room_size(last_mp);
+ } else {
+ buf_len = mb_len;
+ }
offset = (tmpl->rxq.rxseg_n >= n_seg ? 0 : qs_seg->offset) +
(tmpl->rxq.rxseg_n ? 0 : RTE_PKTMBUF_HEADROOM);
/*
* For the buffers beyond descriptions the length is
* pool buffer length, zero lengths are replaced with
- * pool buffer length either.
+ * pool buffer length for real segments,
+ * or remaining packet length for discard segments.
*/
seg_len = tmpl->rxq.rxseg_n >= n_seg ? buf_len :
qs_seg->length ?
qs_seg->length :
- (buf_len - offset);
+ qs_seg->mp != NULL ?
+ (buf_len - offset) : tail_len;
/* Check is done in long int, now overflows. */
- if (buf_len < seg_len + offset) {
+ if (qs_seg->mp != NULL && buf_len < seg_len + offset) {
DRV_LOG(ERR, "port %u Rx queue %u: Split offset/length "
"%u/%u can't be satisfied",
dev->data->port_id, idx,
- qs_seg->length, qs_seg->offset);
+ qs_seg->offset, qs_seg->length);
rte_errno = EINVAL;
goto error;
}
if (seg_len > tail_len)
- seg_len = buf_len - offset;
+ seg_len = qs_seg->mp != NULL ? buf_len - offset : tail_len;
if (++tmpl->rxq.rxseg_n > MLX5_MAX_RXQ_NSEG) {
DRV_LOG(ERR,
"port %u too many SGEs (%u) needed to handle"
@@ -2077,7 +2111,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
/* Save port ID. */
tmpl->rxq.port_id = dev->data->port_id;
tmpl->sh = priv->sh;
- tmpl->rxq.mp = rx_seg[0].mp;
+ tmpl->rxq.sh = priv->sh;
+ tmpl->rxq.mp = first_mp;
tmpl->rxq.elts_n = log2above(desc);
tmpl->rxq.rq_repl_thresh = MLX5_VPMD_RXQ_RPLNSH_THRESH(desc_n);
tmpl->rxq.elts = (struct rte_mbuf *(*)[])(tmpl + 1);
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index a070aaecfd..ac966c51b4 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -116,6 +116,27 @@ mlx5_txq_start(struct rte_eth_dev *dev)
return -rte_errno;
}
+static struct rte_mbuf *
+mlx5_alloc_null_mbuf(uint32_t data_len)
+{
+ size_t alloc_size = sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM +
+ rte_align32pow2(data_len);
+ struct rte_mbuf *m;
+
+ m = mlx5_malloc(MLX5_MEM_ZERO, alloc_size, 0, SOCKET_ID_ANY);
+ if (m == NULL)
+ return NULL;
+ m->buf_addr = RTE_PTR_ADD(m, sizeof(*m));
+ m->buf_len = alloc_size - sizeof(*m);
+ rte_mbuf_iova_set(m, rte_mem_virt2iova(m->buf_addr));
+ m->data_off = RTE_PKTMBUF_HEADROOM;
+ m->refcnt = 1;
+ m->nb_segs = 1;
+ m->port = RTE_MBUF_PORT_INVALID;
+ m->pool = NULL;
+ return m;
+}
+
/**
* Register Rx queue mempools and fill the Rx queue cache.
* This function tolerates repeated mempool registration.
@@ -130,7 +151,8 @@ static int
mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
{
struct rte_mempool *mp;
- uint32_t s;
+ struct mlx5_eth_rxseg *seg;
+ uint16_t s;
int ret = 0;
mlx5_mr_flush_local_cache(&rxq_ctrl->rxq.mr_ctrl);
@@ -139,21 +161,35 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
return mlx5_mr_mempool_populate_cache(&rxq_ctrl->rxq.mr_ctrl,
rxq_ctrl->rxq.mprq_mp);
for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
- bool is_extmem;
-
- mp = rxq_ctrl->rxq.rxseg[s].mp;
- is_extmem = (rte_pktmbuf_priv_flags(mp) &
- RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) != 0;
- ret = mlx5_mr_mempool_register(rxq_ctrl->sh->cdev, mp,
- is_extmem);
- if (ret < 0 && rte_errno != EEXIST)
- return ret;
- ret = mlx5_mr_mempool_populate_cache(&rxq_ctrl->rxq.mr_ctrl,
- mp);
- if (ret < 0)
- return ret;
+ seg = &rxq_ctrl->rxq.rxseg[s];
+ mp = seg->mp;
+ if (mp) { /* Regular segment */
+ bool is_extmem = (rte_pktmbuf_priv_flags(mp) &
+ RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) != 0;
+ ret = mlx5_mr_mempool_register(rxq_ctrl->sh->cdev, mp, is_extmem);
+ if (ret < 0 && rte_errno != EEXIST)
+ goto error;
+ ret = mlx5_mr_mempool_populate_cache(&rxq_ctrl->rxq.mr_ctrl, mp);
+ if (ret < 0)
+ goto error;
+ } else { /* NULL segment used in selective Rx */
+ seg->null_mbuf = mlx5_alloc_null_mbuf(seg->length);
+ if (seg->null_mbuf == NULL) {
+ rte_errno = ENOMEM;
+ ret = -rte_errno;
+ goto error;
+ }
+ }
}
return 0;
+
+error:
+ while (s-- > 0) {
+ seg = &rxq_ctrl->rxq.rxseg[s];
+ mlx5_free(seg->null_mbuf);
+ seg->null_mbuf = NULL;
+ }
+ return ret;
}
/**
--
2.54.0
^ permalink raw reply related
* [PATCH v8 5/9] net/mlx5: fix Rx split segment counter type
From: Thomas Monjalon @ 2026-06-04 19:30 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, stable, Dariusz Sosnowski,
Viacheslav Ovsiienko, Bing Zhao, Ori Kam, Suanming Mou,
Matan Azrad
In-Reply-To: <20260604193324.1996141-1-thomas@monjalon.net>
In the API, rx_nseg and max_nseg are uint16_t.
In mlx5, MLX5_MAX_RXQ_NSEG is 32.
So there is no reason to have rxseg_n as uint32_t.
Reduce the fields to uint16_t and move them to avoid struct holes.
Fixes: 9f209b59c8b0 ("net/mlx5: support Rx buffer split description")
Fixes: 572c9d4bda08 ("net/mlx5: fix shared Rx queue segment configuration match")
Cc: stable@dpdk.org
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
drivers/net/mlx5/mlx5_rx.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index dffab3955b..01b563d981 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -164,9 +164,9 @@ struct __rte_cache_aligned mlx5_rxq_data {
uint64_t flow_meta_mask;
int32_t flow_meta_offset;
uint32_t flow_meta_port_mask;
- uint32_t rxseg_n; /* Number of split segment descriptions. */
struct mlx5_eth_rxseg rxseg[MLX5_MAX_RXQ_NSEG];
/* Buffer split segment descriptions - sizes, offsets, pools. */
+ uint16_t rxseg_n; /* Number of split segment descriptions. */
uint16_t rq_win_cnt; /* Number of packets in the sliding window data. */
uint16_t rq_win_idx_mask; /* Sliding window index wrapping mask. */
uint16_t rq_win_idx; /* Index of the first element in sliding window. */
@@ -191,9 +191,9 @@ struct mlx5_rxq_ctrl {
unsigned int irq:1; /* Whether IRQ is enabled. */
uint32_t flow_tunnels_n[MLX5_FLOW_TUNNEL]; /* Tunnels counters. */
uint32_t wqn; /* WQ number. */
- uint32_t rxseg_n; /* Number of split segment descriptions. */
struct rte_eth_rxseg_split rxseg[MLX5_MAX_RXQ_NSEG];
/* Saved original buffer split segment configuration. */
+ uint16_t rxseg_n; /* Number of split segment descriptions. */
uint16_t dump_file_n; /* Number of dump files. */
};
--
2.54.0
^ permalink raw reply related
* [PATCH v8 4/9] common/mlx5: add null MR functions
From: Thomas Monjalon @ 2026-06-04 19:30 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, Gregory Etelson, Dariusz Sosnowski,
Viacheslav Ovsiienko, Bing Zhao, Ori Kam, Suanming Mou,
Matan Azrad
In-Reply-To: <20260604193324.1996141-1-thomas@monjalon.net>
From: Gregory Etelson <getelson@nvidia.com>
Add functions to allocate and free a null Memory Region (MR)
using ibverbs on Linux.
There is no implementation for DevX on Windows.
Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
drivers/common/mlx5/linux/mlx5_common_verbs.c | 35 +++++++++++++++++++
drivers/common/mlx5/mlx5_common_mr.h | 9 +++++
drivers/common/mlx5/windows/mlx5_common_os.c | 16 +++++++++
3 files changed, 60 insertions(+)
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 2322d9d033..6d44e1f566 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -161,3 +161,38 @@ mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
*reg_mr_cb = mlx5_common_verbs_reg_mr;
*dereg_mr_cb = mlx5_common_verbs_dereg_mr;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_alloc_null_mr)
+struct mlx5_pmd_mr *
+mlx5_os_alloc_null_mr(struct rte_device *dev, void *pd)
+{
+ struct ibv_mr *ibv_mr;
+ struct mlx5_pmd_mr *null_mr;
+
+ null_mr = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*null_mr), 0, dev->numa_node);
+ if (!null_mr)
+ return NULL;
+ ibv_mr = mlx5_glue->alloc_null_mr(pd);
+ if (!ibv_mr) {
+ mlx5_free(null_mr);
+ return NULL;
+ }
+ *null_mr = (struct mlx5_pmd_mr) {
+ .lkey = rte_cpu_to_be_32(ibv_mr->lkey),
+ .addr = ibv_mr->addr,
+ .len = ibv_mr->length,
+ .obj = (void *)ibv_mr,
+ };
+ return null_mr;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_free_null_mr)
+void
+mlx5_os_free_null_mr(struct mlx5_pmd_mr *null_mr)
+{
+ if (!null_mr)
+ return;
+ if (null_mr->obj)
+ claim_zero(mlx5_glue->dereg_mr(null_mr->obj));
+ mlx5_free(null_mr);
+}
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index cf7c685e9b..00f3d832c3 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -21,6 +21,8 @@
#include "mlx5_common_mp.h"
#include "mlx5_common_defs.h"
+struct rte_device;
+
/* mlx5 PMD MR struct. */
struct mlx5_pmd_mr {
uint32_t lkey;
@@ -258,6 +260,13 @@ __rte_internal
void
mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
+__rte_internal
+struct mlx5_pmd_mr *
+mlx5_os_alloc_null_mr(struct rte_device *dev, void *pd);
+__rte_internal
+void
+mlx5_os_free_null_mr(struct mlx5_pmd_mr *null_mr);
+
__rte_internal
int
mlx5_mr_mempool_register(struct mlx5_common_device *cdev,
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index 16fcc5f9fc..692517a9bf 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -454,6 +454,22 @@ mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
*dereg_mr_cb = mlx5_os_dereg_mr;
}
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_alloc_null_mr)
+struct mlx5_pmd_mr *
+mlx5_os_alloc_null_mr(struct rte_device *dev, void *pd)
+{
+ RTE_SET_USED(dev);
+ RTE_SET_USED(pd);
+ return NULL;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(mlx5_os_free_null_mr)
+void
+mlx5_os_free_null_mr(struct mlx5_pmd_mr *null_mr)
+{
+ RTE_SET_USED(null_mr);
+}
+
/*
* In Windows, no need to wrap the MR, no known issue for it in kernel.
* Use the regular function to create direct MR.
--
2.54.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox