* [PATCH v3 09/20] net/txgbe: fix link flow control config for Sapphire
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang, stable, Jiawen Wu, Ferruh Yigit
In-Reply-To: <20260509112823.7728-1-zaiyuwang@trustnetic.com>
SP chips have a hardware bug preventing XON flow control support,
so the driver disables it.
Fixes: 69ce8c8a4ce3 ("net/txgbe: support flow control")
Cc: stable@dpdk.org
Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
---
drivers/net/txgbe/base/txgbe_hw.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/net/txgbe/base/txgbe_hw.c b/drivers/net/txgbe/base/txgbe_hw.c
index 0d3310e15c..b1a1483dab 100644
--- a/drivers/net/txgbe/base/txgbe_hw.c
+++ b/drivers/net/txgbe/base/txgbe_hw.c
@@ -1037,8 +1037,10 @@ s32 txgbe_fc_enable(struct txgbe_hw *hw)
for (i = 0; i < TXGBE_DCB_TC_MAX; i++) {
if ((hw->fc.current_mode & txgbe_fc_tx_pause) &&
hw->fc.high_water[i]) {
- fcrtl = TXGBE_FCWTRLO_TH(hw->fc.low_water[i]) |
- TXGBE_FCWTRLO_XON;
+ fcrtl = TXGBE_FCWTRLO_TH(hw->fc.low_water[i]);
+ /* SP doesn't support xon */
+ if (hw->mac.type == txgbe_mac_aml || hw->mac.type == txgbe_mac_aml40)
+ fcrtl |= TXGBE_FCWTRLO_XON;
fcrth = TXGBE_FCWTRHI_TH(hw->fc.high_water[i]) |
TXGBE_FCWTRHI_XOFF;
} else {
--
2.21.0.windows.1
^ permalink raw reply related
* [PATCH v3 08/20] net/txgbe: fix link flow control registers for Amber-Lite
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang, stable, Jiawen Wu
In-Reply-To: <20260509112823.7728-1-zaiyuwang@trustnetic.com>
The flow control counter registers on AML NICs differ from those on SP
NICs. Update the register offsets accordingly to ensure the counters
work correctly.
Fixes: fb6eb170dfa2 ("net/txgbe: add basic link configuration for Amber-Lite")
Cc: stable@dpdk.org
Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
---
drivers/net/txgbe/base/txgbe_hw.c | 7 ++++++
drivers/net/txgbe/base/txgbe_regs.h | 2 ++
drivers/net/txgbe/base/txgbe_type.h | 4 ++++
drivers/net/txgbe/txgbe_ethdev.c | 34 +++++++++++++++++++----------
4 files changed, 36 insertions(+), 11 deletions(-)
diff --git a/drivers/net/txgbe/base/txgbe_hw.c b/drivers/net/txgbe/base/txgbe_hw.c
index 0f3db3a1ad..0d3310e15c 100644
--- a/drivers/net/txgbe/base/txgbe_hw.c
+++ b/drivers/net/txgbe/base/txgbe_hw.c
@@ -394,6 +394,13 @@ s32 txgbe_clear_hw_cntrs(struct txgbe_hw *hw)
rd32(hw, TXGBE_PBTXLNKXON);
rd32(hw, TXGBE_PBTXLNKXOFF);
+ if (hw->mac.type == txgbe_mac_aml || hw->mac.type == txgbe_mac_aml40) {
+ wr32(hw, TXGBE_PBRXLNKXON_AML, 0);
+ wr32(hw, TXGBE_PBRXLNKXOFF_AML, 0);
+ hw->last_stats.rx_xon_packets = 0;
+ hw->last_stats.rx_xoff_packets = 0;
+ }
+
/* DMA Stats */
rd32(hw, TXGBE_DMARXPKT);
rd32(hw, TXGBE_DMATXPKT);
diff --git a/drivers/net/txgbe/base/txgbe_regs.h b/drivers/net/txgbe/base/txgbe_regs.h
index 060757323a..de382601c9 100644
--- a/drivers/net/txgbe/base/txgbe_regs.h
+++ b/drivers/net/txgbe/base/txgbe_regs.h
@@ -1085,6 +1085,8 @@ enum txgbe_5tuple_protocol {
#define TXGBE_PBRXDROP 0x019068
#define TXGBE_PBRXLNKXOFF 0x011988
#define TXGBE_PBRXLNKXON 0x011E0C
+#define TXGBE_PBRXLNKXOFF_AML 0x011F80
+#define TXGBE_PBRXLNKXON_AML 0x011F84
#define TXGBE_PBRXUPXON(up) (0x011E30 + (up) * 4)
#define TXGBE_PBRXUPXOFF(up) (0x011E10 + (up) * 4)
diff --git a/drivers/net/txgbe/base/txgbe_type.h b/drivers/net/txgbe/base/txgbe_type.h
index ede780321f..505f598fb7 100644
--- a/drivers/net/txgbe/base/txgbe_type.h
+++ b/drivers/net/txgbe/base/txgbe_type.h
@@ -876,6 +876,10 @@ struct txgbe_hw {
u64 tx_qp_bytes;
u64 rx_qp_mc_packets;
} qp_last[TXGBE_MAX_QP];
+ struct {
+ u64 rx_xon_packets;
+ u64 rx_xoff_packets;
+ } last_stats;
rte_spinlock_t phy_lock;
/*amlite: new SW-FW mbox */
diff --git a/drivers/net/txgbe/txgbe_ethdev.c b/drivers/net/txgbe/txgbe_ethdev.c
index 414107d7a7..3ae233f70a 100644
--- a/drivers/net/txgbe/txgbe_ethdev.c
+++ b/drivers/net/txgbe/txgbe_ethdev.c
@@ -2264,16 +2264,18 @@ txgbe_dev_reset(struct rte_eth_dev *dev)
return ret;
}
+#define TXGBE_UPDATE_COUNTER_32BIT_GENERIC(reg, last, count, reset) \
+ do { \
+ uint32_t current = rd32(hw, reg); \
+ if ((current) < (last)) \
+ current += 0x100000000ULL; \
+ if (reset) \
+ (last) = current; \
+ (count) = (uint32_t)((current) - (last)); \
+ } while (0)
+
#define UPDATE_QP_COUNTER_32bit(reg, last_counter, counter) \
- { \
- uint32_t current_counter = rd32(hw, reg); \
- if (current_counter < last_counter) \
- current_counter += 0x100000000LL; \
- if (!hw->offset_loaded) \
- last_counter = current_counter; \
- counter = current_counter - last_counter; \
- counter &= 0xFFFFFFFFLL; \
- }
+ TXGBE_UPDATE_COUNTER_32BIT_GENERIC(reg, last_counter, counter, !hw->offset_loaded)
#define UPDATE_QP_COUNTER_36bit(reg_lsb, reg_msb, last_counter, counter) \
{ \
@@ -2331,8 +2333,18 @@ txgbe_read_stats_registers(struct txgbe_hw *hw,
hw_stats->up[i].rx_up_dropped +=
rd32(hw, TXGBE_PBRXMISS(i));
}
- hw_stats->rx_xon_packets += rd32(hw, TXGBE_PBRXLNKXON);
- hw_stats->rx_xoff_packets += rd32(hw, TXGBE_PBRXLNKXOFF);
+
+ if (hw->mac.type == txgbe_mac_aml || hw->mac.type == txgbe_mac_aml40) {
+ TXGBE_UPDATE_COUNTER_32BIT_GENERIC(TXGBE_PBRXLNKXON_AML,
+ hw->last_stats.rx_xon_packets,
+ hw_stats->rx_xon_packets, !hw->offset_loaded);
+ TXGBE_UPDATE_COUNTER_32BIT_GENERIC(TXGBE_PBRXLNKXOFF_AML,
+ hw->last_stats.rx_xoff_packets,
+ hw_stats->rx_xoff_packets, !hw->offset_loaded);
+ } else {
+ hw_stats->rx_xon_packets += rd32(hw, TXGBE_PBRXLNKXON);
+ hw_stats->rx_xoff_packets += rd32(hw, TXGBE_PBRXLNKXOFF);
+ }
hw_stats->tx_xon_packets += rd32(hw, TXGBE_PBTXLNKXON);
hw_stats->tx_xoff_packets += rd32(hw, TXGBE_PBTXLNKXOFF);
--
2.21.0.windows.1
^ permalink raw reply related
* [PATCH v3 07/20] net/txgbe: fix Tx desc free logic
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang, stable, Jiawen Wu
In-Reply-To: <20260509112823.7728-1-zaiyuwang@trustnetic.com>
On some server environments, this driver caused TDM non-fatal errors
or PCIe request errors during Tx operation
In Amber-Lite NIC's Tx head write-back mode, the hardware periodically
writes back a head index pointing to the next descriptor it is adout
to process in Tx ring. All descriptors before the head are considered
processed by hardware and can be safely freed by the driver.
The root cause is that the driver can safely free a batch of descriptors
only when the hardware's write-back head pointer has advanced beyond all
descriptors in that batch, meaning they have all been processed by the
hardware. If the driver frees a descriptor before the hardware has
finished processing it, invalid memory access may occur, leading to the
observed bug.
To fix the issue, correct the boundary check in all three Tx cleanup
functions, each of which was missing the proper condition to prevent
freeing unprocessed descriptors.
Fixes: 8ada71d0bb7f ("net/txgbe: add Tx head write-back mode for Amber-Lite")
Cc: stable@dpdk.org
Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
---
drivers/net/txgbe/txgbe_rxtx.c | 9 ++++++++-
drivers/net/txgbe/txgbe_rxtx_vec_common.h | 7 +++++--
2 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/drivers/net/txgbe/txgbe_rxtx.c b/drivers/net/txgbe/txgbe_rxtx.c
index e2cd9b8841..72a4965693 100644
--- a/drivers/net/txgbe/txgbe_rxtx.c
+++ b/drivers/net/txgbe/txgbe_rxtx.c
@@ -100,7 +100,9 @@ txgbe_tx_free_bufs(struct txgbe_tx_queue *txq)
volatile uint16_t head = (uint16_t)*txq->headwb_mem;
- if (txq->tx_next_dd > head && head > tx_last_dd)
+ if (txq->tx_next_dd == head)
+ return 0;
+ else if (txq->tx_next_dd > head && head > tx_last_dd)
return 0;
else if (tx_last_dd > txq->tx_next_dd &&
(head > tx_last_dd || head < txq->tx_next_dd))
@@ -652,6 +654,11 @@ txgbe_xmit_cleanup(struct txgbe_tx_queue *txq)
/* we have caught up to head, no work left to do */
if (desc_to_clean_to == head)
return -(1);
+ else if (desc_to_clean_to > head && head > last_desc_cleaned)
+ return -(1);
+ else if (last_desc_cleaned > desc_to_clean_to &&
+ (head > last_desc_cleaned || head < desc_to_clean_to))
+ return -(1);
} else {
if (!(status & rte_cpu_to_le_32(TXGBE_TXD_DD))) {
PMD_TX_FREE_LOG(DEBUG,
diff --git a/drivers/net/txgbe/txgbe_rxtx_vec_common.h b/drivers/net/txgbe/txgbe_rxtx_vec_common.h
index 00847d087b..edf3586b77 100644
--- a/drivers/net/txgbe/txgbe_rxtx_vec_common.h
+++ b/drivers/net/txgbe/txgbe_rxtx_vec_common.h
@@ -94,8 +94,11 @@ txgbe_tx_free_bufs(struct txgbe_tx_queue *txq)
txq->tx_next_dd - txq->tx_free_thresh;
if (tx_last_dd >= txq->nb_tx_desc)
tx_last_dd -= txq->nb_tx_desc;
- volatile uint16_t head = (uint16_t)*txq->headwb_mem;
- if (txq->tx_next_dd > head && head > tx_last_dd)
+
+ volatile uint16_t head = (uint16_t)*txq->headwb_mem;
+ if (txq->tx_next_dd == head)
+ return 0;
+ else if (txq->tx_next_dd > head && head > tx_last_dd)
return 0;
else if (tx_last_dd > txq->tx_next_dd &&
(head > tx_last_dd || head < txq->tx_next_dd))
--
2.21.0.windows.1
^ permalink raw reply related
* [PATCH v3 05/20] net/txgbe: fix inaccuracy in TX rate limiting
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang, stable, Jiawen Wu
In-Reply-To: <20260509112823.7728-1-zaiyuwang@trustnetic.com>
Amber-lite NIC's TX rate limiting has large deviations for small
packets. To fix this issue, there are some changes:
1. Set TDM_RL_ADJ (0x1820c) to 21B (includes 7B Ethernet preamble,
1B SFD, 1B EFD, and 12B IPG).
2) Remove the rate offset in the driver (e.g., 105 / 100, a rough
compensation value from Linux kernel driver tests).
After these changes, accuracy deviation for 64B packets is within
~5%, while large packets show lower deviation.
Fixes: a309ab43acf3 ("net/txgbe: support Tx queue rate limiting for Amber-Lite")
Cc: stable@dpdk.org
Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
---
drivers/net/txgbe/base/txgbe_regs.h | 1 +
drivers/net/txgbe/txgbe_ethdev.c | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/txgbe/base/txgbe_regs.h b/drivers/net/txgbe/base/txgbe_regs.h
index 95c585a025..060757323a 100644
--- a/drivers/net/txgbe/base/txgbe_regs.h
+++ b/drivers/net/txgbe/base/txgbe_regs.h
@@ -1670,6 +1670,7 @@ enum txgbe_5tuple_protocol {
#define TXGBE_TDM_FACTOR_INT_SHIFT 16
#define TXGBE_TDM_FACTOR_FRA_SHIFT 2
+#define TXGBE_TDM_RL_ADJ 0x1820C
#define TXGBE_TDM_RL_VM_IDX 0x018218
#define TXGBE_TDM_RL_VM_CFG 0x01821C
#define TXGBE_TDM_RL_CFG 0x018400
diff --git a/drivers/net/txgbe/txgbe_ethdev.c b/drivers/net/txgbe/txgbe_ethdev.c
index 779874aac9..414107d7a7 100644
--- a/drivers/net/txgbe/txgbe_ethdev.c
+++ b/drivers/net/txgbe/txgbe_ethdev.c
@@ -4314,7 +4314,6 @@ txgbe_set_queue_rate_limit(struct rte_eth_dev *dev,
u16 frac;
link_speed = dev->data->dev_link.link_speed;
- tx_rate = tx_rate * 105 / 100;
/* Calculate the rate factor values to set */
factor_int = link_speed / tx_rate;
frac = (link_speed % tx_rate) * 10000 / tx_rate;
@@ -4324,6 +4323,7 @@ txgbe_set_queue_rate_limit(struct rte_eth_dev *dev,
factor_fra = 0;
}
+ wr32(hw, TXGBE_TDM_RL_ADJ, 21);
wr32(hw, TXGBE_TDM_RL_QUEUE_IDX, queue_idx);
wr32m(hw, TXGBE_TDM_RL_QUEUE_CFG,
TXGBE_TDM_FACTOR_INT_MASK, factor_int << TXGBE_TDM_FACTOR_INT_SHIFT);
--
2.21.0.windows.1
^ permalink raw reply related
* [PATCH v3 06/20] net/txgbe: fix link status check condition
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang, stable, Jiawen Wu
In-Reply-To: <20260509112823.7728-1-zaiyuwang@trustnetic.com>
The original code incorrectly used 'if (link_up)' instead of
'if (*link_up)', causing the condition to always evaluate to true
because the pointer itself is non-NULL. This led to incorrect speed
assignment.
Fixes: fb6eb170dfa2 ("net/txgbe: add basic link configuration for Amber-Lite")
Cc: stable@dpdk.org
Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
---
drivers/net/txgbe/base/txgbe_aml.c | 2 +-
drivers/net/txgbe/base/txgbe_aml40.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/txgbe/base/txgbe_aml.c b/drivers/net/txgbe/base/txgbe_aml.c
index b376eca5b5..de9a1b1c93 100644
--- a/drivers/net/txgbe/base/txgbe_aml.c
+++ b/drivers/net/txgbe/base/txgbe_aml.c
@@ -67,7 +67,7 @@ s32 txgbe_check_mac_link_aml(struct txgbe_hw *hw, u32 *speed,
*link_up = false;
}
- if (link_up) {
+ if (*link_up) {
switch (links_reg & TXGBE_CFG_PORT_ST_AML_LINK_MASK) {
case TXGBE_CFG_PORT_ST_AML_LINK_25G:
*speed = TXGBE_LINK_SPEED_25GB_FULL;
diff --git a/drivers/net/txgbe/base/txgbe_aml40.c b/drivers/net/txgbe/base/txgbe_aml40.c
index 733bbac13a..eefd7119fd 100644
--- a/drivers/net/txgbe/base/txgbe_aml40.c
+++ b/drivers/net/txgbe/base/txgbe_aml40.c
@@ -68,7 +68,7 @@ s32 txgbe_check_mac_link_aml40(struct txgbe_hw *hw, u32 *speed,
*link_up = false;
}
- if (link_up) {
+ if (*link_up) {
if ((links_reg & TXGBE_CFG_PORT_ST_AML_LINK_40G) ==
TXGBE_CFG_PORT_ST_AML_LINK_40G)
*speed = TXGBE_LINK_SPEED_40GB_FULL;
--
2.21.0.windows.1
^ permalink raw reply related
* [PATCH v3 04/20] net/ngbe: fix VF promiscuous and allmulticast
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang, stable, Jiawen Wu
In-Reply-To: <20260509112823.7728-1-zaiyuwang@trustnetic.com>
The configuration of allmulti and promiscuous modes conflicts
together. For instance, if we enable promiscuous mode, then enable and
disable allmulti, then the promiscuous mode is wrongly disabled.
Fix this behavior by:
- doing nothing when we set/unset allmulti if promiscuous mode is on
- restorting the proper mode (none or allmulti) when we disable
promiscuous mode
Fixes: 7744e90805b5 ("net/ngbe: add promiscuous and allmulticast ops for VF device")
Cc: stable@dpdk.org
Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
---
drivers/net/ngbe/ngbe_ethdev_vf.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ngbe/ngbe_ethdev_vf.c b/drivers/net/ngbe/ngbe_ethdev_vf.c
index 6406df40d0..81511fed8a 100644
--- a/drivers/net/ngbe/ngbe_ethdev_vf.c
+++ b/drivers/net/ngbe/ngbe_ethdev_vf.c
@@ -1196,9 +1196,13 @@ static int
ngbevf_dev_promiscuous_disable(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
+ int mode = NGBEVF_XCAST_MODE_NONE;
int ret;
- switch (hw->mac.update_xcast_mode(hw, NGBEVF_XCAST_MODE_NONE)) {
+ if (dev->data->all_multicast)
+ mode = NGBEVF_XCAST_MODE_ALLMULTI;
+
+ switch (hw->mac.update_xcast_mode(hw, mode)) {
case 0:
ret = 0;
break;
@@ -1219,7 +1223,7 @@ ngbevf_dev_allmulticast_enable(struct rte_eth_dev *dev)
struct ngbe_hw *hw = ngbe_dev_hw(dev);
int ret;
- if (dev->data->promiscuous == 1)
+ if (dev->data->promiscuous)
return 0;
switch (hw->mac.update_xcast_mode(hw, NGBEVF_XCAST_MODE_ALLMULTI)) {
@@ -1243,6 +1247,9 @@ ngbevf_dev_allmulticast_disable(struct rte_eth_dev *dev)
struct ngbe_hw *hw = ngbe_dev_hw(dev);
int ret;
+ if (dev->data->promiscuous)
+ return 0;
+
switch (hw->mac.update_xcast_mode(hw, NGBEVF_XCAST_MODE_MULTI)) {
case 0:
ret = 0;
--
2.21.0.windows.1
^ permalink raw reply related
* [PATCH v3 03/20] net/ngbe: add missing CDR config for YT PHY
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang, stable, Jiawen Wu
In-Reply-To: <20260509112823.7728-1-zaiyuwang@trustnetic.com>
According to the PHY vendor, when YT8531S operates in UTP-to-Fiber or
RGMII-to-Fiber mode with auto-negotiation disabled (Force mode),
additional CDR (Clock Data Recovery) configuration is required to
improve link connectivity. Without this config, link may be unstable
or fail to establish.
Fixes: f1268369403d ("net/ngbe: support autoneg on/off for external PHY SFI mode")
Cc: stable@dpdk.org
Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
---
drivers/net/ngbe/base/ngbe_phy_yt.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/net/ngbe/base/ngbe_phy_yt.c b/drivers/net/ngbe/base/ngbe_phy_yt.c
index d110fbc8b2..ab0778d246 100644
--- a/drivers/net/ngbe/base/ngbe_phy_yt.c
+++ b/drivers/net/ngbe/base/ngbe_phy_yt.c
@@ -264,6 +264,9 @@ s32 ngbe_setup_phy_link_yt(struct ngbe_hw *hw, u32 speed,
value = YT_BCR_RESET | YT_BCR_ANE | YT_BCR_RESTART_AN |
YT_BCR_DUPLEX | YT_BCR_SPEED_SELECT1;
} else {
+ /* force mode need to config cdr */
+ ngbe_write_phy_reg_sds_ext_yt(hw, 0x3, 0, 0x1434);
+ ngbe_write_phy_reg_sds_ext_yt(hw, 0xe, 0, 0x163);
value = YT_BCR_RESET | YT_BCR_DUPLEX;
if (speed & NGBE_LINK_SPEED_1GB_FULL)
value |= YT_BCR_SPEED_SELECT1;
--
2.21.0.windows.1
^ permalink raw reply related
* [PATCH v3 02/20] net/ngbe: remove duplicate xstats counters
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang, stable, Jiawen Wu
In-Reply-To: <20260509112823.7728-1-zaiyuwang@trustnetic.com>
Remove four redundant counters (tx_xon_packets, rx_xon_packets,
tx_xoff_packets and rx_xoff_packets) from xstats, as they were duplicates
of tx_flow_control_xon_packets and others. Both sets were reading the same
registers but being output twice under different names. After removing
these entries, the flow control counters in DPDK now align with those in
our Linux kernel driver.
Fixes: 8b433d04adc9 ("net/ngbe: support device xstats")
Cc: stable@dpdk.org
Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
---
drivers/net/ngbe/ngbe_ethdev.c | 5 -----
1 file changed, 5 deletions(-)
diff --git a/drivers/net/ngbe/ngbe_ethdev.c b/drivers/net/ngbe/ngbe_ethdev.c
index 8b9d6371fb..6df53f3266 100644
--- a/drivers/net/ngbe/ngbe_ethdev.c
+++ b/drivers/net/ngbe/ngbe_ethdev.c
@@ -227,11 +227,6 @@ static const struct rte_ngbe_xstats_name_off rte_ngbe_stats_strings[] = {
HW_XSTAT(tx_size_1024_to_max_packets),
/* Flow Control */
- HW_XSTAT(tx_xon_packets),
- HW_XSTAT(rx_xon_packets),
- HW_XSTAT(tx_xoff_packets),
- HW_XSTAT(rx_xoff_packets),
-
HW_XSTAT_NAME(tx_xon_packets, "tx_flow_control_xon_packets"),
HW_XSTAT_NAME(rx_xon_packets, "rx_flow_control_xon_packets"),
HW_XSTAT_NAME(tx_xoff_packets, "tx_flow_control_xoff_packets"),
--
2.21.0.windows.1
^ permalink raw reply related
* [PATCH v3 01/20] net/txgbe: remove duplicate xstats counters
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang, stable, Jiawen Wu, Ferruh Yigit
In-Reply-To: <20260509112823.7728-1-zaiyuwang@trustnetic.com>
Remove four redundant counters (tx_xon_packets, rx_xon_packets,
tx_xoff_packets and rx_xoff_packets) from xstats, as they were duplicates
of tx_flow_control_xon_packets and others. Both sets were reading the same
registers but being output twice under different names. After removing
these entries, the flow control counters in DPDK now align with those in
our Linux kernel driver.
Fixes: 91fe49c87d76 ("net/txgbe: support device xstats")
Cc: stable@dpdk.org
Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
---
drivers/net/txgbe/txgbe_ethdev.c | 5 -----
1 file changed, 5 deletions(-)
diff --git a/drivers/net/txgbe/txgbe_ethdev.c b/drivers/net/txgbe/txgbe_ethdev.c
index 5d360f8305..779874aac9 100644
--- a/drivers/net/txgbe/txgbe_ethdev.c
+++ b/drivers/net/txgbe/txgbe_ethdev.c
@@ -261,11 +261,6 @@ static const struct rte_txgbe_xstats_name_off rte_txgbe_stats_strings[] = {
HW_XSTAT(tx_size_1024_to_max_packets),
/* Flow Control */
- HW_XSTAT(tx_xon_packets),
- HW_XSTAT(rx_xon_packets),
- HW_XSTAT(tx_xoff_packets),
- HW_XSTAT(rx_xoff_packets),
-
HW_XSTAT_NAME(tx_xon_packets, "tx_flow_control_xon_packets"),
HW_XSTAT_NAME(rx_xon_packets, "rx_flow_control_xon_packets"),
HW_XSTAT_NAME(tx_xoff_packets, "tx_flow_control_xoff_packets"),
--
2.21.0.windows.1
^ permalink raw reply related
* [PATCH v3 00/20] Wangxun Fixes
From: Zaiyu Wang @ 2026-05-09 11:28 UTC (permalink / raw)
To: dev; +Cc: Zaiyu Wang
In-Reply-To: <20260423034024.14404-1-zaiyuwang@trustnetic.com>
This series fixes several issues found on Wangxun Emerald, Sapphire and
Amber-lite NICs, with a focus on link-related problems.
---
v3:
- Addressed Stephen's comments
---
v2:
- Fixed compilation error and code style issues
---
Zaiyu Wang (20):
net/txgbe: remove duplicate xstats counters
net/ngbe: remove duplicate xstats counters
net/ngbe: add missing CDR config for YT PHY
net/ngbe: fix VF promiscuous and allmulticast
net/txgbe: fix inaccuracy in TX rate limiting
net/txgbe: fix link status check condition
net/txgbe: fix Tx desc free logic
net/txgbe: fix link flow control registers for Amber-Lite
net/txgbe: fix link flow control config for Sapphire
net/txgbe: fix a mass of unknown interrupts
net/txgbe: fix traffic class priority configuration
net/txgbe: fix link stability for 25G NIC
net/txgbe: fix link stability for 40G NIC
net/txgbe: fix link stability for Amber-Lite backplane mode
net/txgbe: fix FEC mode configuration on 25G NIC
net/txgbe: fix SFP module identification
net/txgbe: fix get module info operation
net/txgbe: fix get eeprom operation
net/txgbe: fix to reset Tx write-back pointer
net/txgbe: fix to enable Tx desc check
drivers/net/ngbe/base/ngbe_phy_yt.c | 3 +
drivers/net/ngbe/ngbe_ethdev.c | 5 -
drivers/net/ngbe/ngbe_ethdev_vf.c | 11 +-
drivers/net/txgbe/base/meson.build | 2 +
drivers/net/txgbe/base/txgbe.h | 2 +
drivers/net/txgbe/base/txgbe_aml.c | 187 +-
drivers/net/txgbe/base/txgbe_aml.h | 6 +-
drivers/net/txgbe/base/txgbe_aml40.c | 113 +-
drivers/net/txgbe/base/txgbe_aml40.h | 6 +-
drivers/net/txgbe/base/txgbe_dcb_hw.c | 2 +-
drivers/net/txgbe/base/txgbe_e56.c | 3774 +++++++++++++++++++++
drivers/net/txgbe/base/txgbe_e56.h | 1744 ++++++++++
drivers/net/txgbe/base/txgbe_e56_bp.c | 2593 ++++++++++++++
drivers/net/txgbe/base/txgbe_e56_bp.h | 278 ++
drivers/net/txgbe/base/txgbe_hw.c | 54 +-
drivers/net/txgbe/base/txgbe_hw.h | 4 +-
drivers/net/txgbe/base/txgbe_osdep.h | 4 +
drivers/net/txgbe/base/txgbe_phy.c | 360 +-
drivers/net/txgbe/base/txgbe_phy.h | 45 +-
drivers/net/txgbe/base/txgbe_regs.h | 11 +-
drivers/net/txgbe/base/txgbe_type.h | 43 +-
drivers/net/txgbe/txgbe_ethdev.c | 414 ++-
drivers/net/txgbe/txgbe_ethdev.h | 6 +-
drivers/net/txgbe/txgbe_rxtx.c | 102 +-
drivers/net/txgbe/txgbe_rxtx.h | 1 +
drivers/net/txgbe/txgbe_rxtx_vec_common.h | 14 +-
26 files changed, 9350 insertions(+), 434 deletions(-)
create mode 100644 drivers/net/txgbe/base/txgbe_e56.c
create mode 100644 drivers/net/txgbe/base/txgbe_e56.h
create mode 100644 drivers/net/txgbe/base/txgbe_e56_bp.c
create mode 100644 drivers/net/txgbe/base/txgbe_e56_bp.h
--
2.21.0.windows.1
^ permalink raw reply
* RE: [PATCH v2 11/20] net/txgbe: fix traffic class priority configuration
From: Zaiyu Wang @ 2026-05-09 11:06 UTC (permalink / raw)
To: 'Stephen Hemminger'; +Cc: dev, stable
In-Reply-To: <20260429081136.3defdeba@phoenix.local>
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday, April 29, 2026 11:12 PM
> To: Zaiyu Wang <zaiyuwang@trustnetic.com>; Zaiyu Wang
> <zaiyuwang@trustnetic.com>
> Cc: dev@dpdk.org; stable@dpdk.org; Jiawen Wu <jiawenwu@trustnetic.com>;
> Ferruh Yigit <ferruh.yigit@amd.com>; dev@dpdk.org; stable@dpdk.org; Jiawen
> Wu <jiawenwu@trustnetic.com>; Ferruh Yigit <ferruh.yigit@amd.com>
> Subject: Re: [PATCH v2 11/20] net/txgbe: fix traffic class priority configuration
>
> On Wed, 29 Apr 2026 18:25:05 +0800
> Zaiyu Wang <zaiyuwang@trustnetic.com> wrote:
>
> > After applying the following testpmd command, 802.1Q packets with
> > specific priorities were not properly directed to the corresponding traffic classes:
> > port config 0 dcb vt off 4 pfc off
> >
> > The old driver had two issues:
> > 1. The hardware uses a 4-bit mapping register per traffic class for
> > priority-to-TC mapping, but the driver incorrectly configured it
> > as 3 bits.
> > 2. The DCB TX configuration mistakenly wrote to the RX register.
> >
> > Fix both issues, ensuring that tc-prio mapping works as expected.
> >
> > Fixes: 8bdc7882f376 ("net/txgbe: support DCB")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Zaiyu Wang <zaiyuwang@trustnetic.com>
> > ---
>
> AI review spotted issues:
>
> Error: The fix is incomplete. The same priority-to-TC layout is also written in
> drivers/net/txgbe/txgbe_rxtx.c around line 3380 in txgbe_vmdq_dcb_configure():
> for (i = 0; i < RTE_ETH_DCB_NUM_USER_PRIORITIES; i++)
> /*
> * mapping is done with 3 bits per priority,
> * so shift by i*3 each time
> */
> queue_mapping |= ((cfg->dcb_tc[i] & 0x07) << (i * 3));
>
> wr32(hw, TXGBE_RPUP2TC, queue_mapping);
> This writes the same TXGBE_RPUP2TC register (0x019008) the patch is fixing,
> using a hardcoded i*3 shift and a comment claiming 3 bits per priority. After this
> patch, txgbe_dcb_config_rx_arbiter_raptor() encodes priorities at 4-bit positions
> (via the updated TXGBE_RPUP2TC_UP_SHIFT=4), but txgbe_vmdq_dcb_configure()
> still uses 3-bit positions. The VMDQ+DCB path will produce a wrong tc-prio
> mapping. txgbe_vmdq_dcb_configure() needs the same fix (use
> TXGBE_RPUP2TC_UP_SHIFT and update the comment).
> Warning: TXGBE_DCBUP2TC_DEC and TXGBE_DCBUP2TC_MAP are now
> inconsistent. After this patch:
> #define TXGBE_DCBUP2TC_MAP(tc, v) LS(v, 4 * (tc), 0x7)
> #define TXGBE_DCBUP2TC_DEC(tc, r) RS(r, 3 * (tc), 0x7)
> DEC will not decode what MAP encoded for any tc > 0. TXGBE_DCBUP2TC_DEC
> isn't currently used in tree, so this is not a runtime bug, but it's a latent trap. Either
> update DEC to also use 4*(tc), or remove DEC since it has no callers.
>
Hi Stephen,
Thank you for the thorough AI‑assisted review. I really appreciate it.
I have addressed all the issues you raised, and an updated patch set will be sent shortly.
^ permalink raw reply
* RE: [PATCH 13/20] mbuf: allow NULL array in rte_pktmbuf_free_bulk
From: Morten Brørup @ 2026-05-09 8:47 UTC (permalink / raw)
To: Stephen Hemminger, dev
In-Reply-To: <20260508203607.1003036-14-stephen@networkplumber.org>
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Friday, 8 May 2026 22.34
>
> This allows callers to avoid NULL checks and just call
> rte_pktmbuf_free_bulk, similar to rte_pktmbuf_free.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
I disagree with this patch.
The parameter is an array of (pointers to) mbufs.
We already accept that the array can contain NULL pointers (no mbuf present).
This is extremely forgiving, considering that other fast path functions don't allow NULL pointers in arrays;
e.g. rte_eth_tx_burst(), rte_mempool_put_bulk().
But since it's a "free()" class of function, I don't object to it.
However, this patch changes the parameter type from "array" to "array or NULL (no array present)".
And I don't think we should change the parameter type; it should remain "array" only.
If there are any scenarios where a non-present array (NULL) is passed to the function, the count should be zero too.
And when the count is zero, the function does not dereference the array, so explicitly checking for NULL is superfluous.
We have a convention of not checking parameter validity in fast path functions.
And I consider it invalid parameters passing NULL with a non-zero count.
You might argue that this is a "free()" class of function, which warrants checking for NULL; but since it already accepts NULL with zero count, it is already covered.
We could change the function declaration for clarity:
void rte_pktmbuf_free_bulk(
unsigned int count;
struct rte_mbuf *mbufs[count], unsigned int count);
Or add a debug assertion at the start of the function:
RTE_ASSERT(mbufs != NULL || count == 0);
> ---
> devtools/cocci/nullfree.cocci | 5 ++++-
> doc/guides/rel_notes/release_26_07.rst | 5 +++++
> lib/mbuf/rte_mbuf.c | 3 +++
> lib/mbuf/rte_mbuf.h | 1 +
> 4 files changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/devtools/cocci/nullfree.cocci
> b/devtools/cocci/nullfree.cocci
> index e7417b69ff..78e30f730a 100644
> --- a/devtools/cocci/nullfree.cocci
> +++ b/devtools/cocci/nullfree.cocci
> @@ -4,7 +4,7 @@
> // free(NULL) as a no-op.
> //
> @@
> -expression E;
> +expression E, N;
> @@
> (
> - if (E != NULL) cmdline_free(E);
> @@ -79,6 +79,9 @@ expression E;
> - if (E != NULL) rte_pktmbuf_free(E);
> + rte_pktmbuf_free(E);
> |
> +- if (E != NULL) rte_pktmbuf_free_bulk(E, N);
> ++ rte_pktmbuf_free_bulk(E, N);
> +|
> - if (E != NULL) rte_rib_free(E);
> + rte_rib_free(E);
> |
> diff --git a/doc/guides/rel_notes/release_26_07.rst
> b/doc/guides/rel_notes/release_26_07.rst
> index f012d47a4b..6f2f7f849f 100644
> --- a/doc/guides/rel_notes/release_26_07.rst
> +++ b/doc/guides/rel_notes/release_26_07.rst
> @@ -63,6 +63,11 @@ New Features
> ``rte_eal_init`` and the application is responsible for probing
> each device,
> * ``--auto-probing`` enables the initial bus probing, which is the
> current default behavior.
>
> +* **mbuf: rte_pktmbuf_free_bulk now accepts NULL array pointer.**
> +
> + Calling ``rte_pktmbuf_free_bulk`` with a NULL ``mbufs`` argument is
> a no-op,
> + matching the behavior of ``rte_pktmbuf_free()`` and ``rte_free()``.
> +
>
> Removed Items
> -------------
> diff --git a/lib/mbuf/rte_mbuf.c b/lib/mbuf/rte_mbuf.c
> index c2476e7704..8ec3bb53bb 100644
> --- a/lib/mbuf/rte_mbuf.c
> +++ b/lib/mbuf/rte_mbuf.c
> @@ -562,6 +562,9 @@ void rte_pktmbuf_free_bulk(struct rte_mbuf **mbufs,
> unsigned int count)
> struct rte_mbuf *m, *m_next,
> *pending[RTE_PKTMBUF_FREE_PENDING_SZ];
> unsigned int idx, nb_pending = 0;
>
> + if (mbufs == NULL)
> + return;
> +
> rte_mbuf_history_mark_bulk(mbufs, count,
> RTE_MBUF_HISTORY_OP_LIB_FREE);
>
> for (idx = 0; idx < count; idx++) {
> diff --git a/lib/mbuf/rte_mbuf.h b/lib/mbuf/rte_mbuf.h
> index e7c3bbadd4..e9f8932e80 100644
> --- a/lib/mbuf/rte_mbuf.h
> +++ b/lib/mbuf/rte_mbuf.h
> @@ -1534,6 +1534,7 @@ static inline void rte_pktmbuf_free(struct
> rte_mbuf *m)
> * @param mbufs
> * Array of pointers to packet mbufs.
> * The array may contain NULL pointers.
> + * The array pointer is NULL function does nothing.
> * @param count
> * Array size.
> */
> --
> 2.53.0
^ permalink raw reply
* [PATCH 3/3] power: use common decimal macro definition
From: Huisong Li @ 2026-05-09 8:45 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
In-Reply-To: <20260509084503.2917038-1-lihuisong@huawei.com>
Power QoS uses the common decimal macro definition
to replace magic number.
Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
lib/power/rte_power_qos.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/power/rte_power_qos.c b/lib/power/rte_power_qos.c
index be230d1c50..f991230532 100644
--- a/lib/power/rte_power_qos.c
+++ b/lib/power/rte_power_qos.c
@@ -115,7 +115,7 @@ rte_power_qos_get_cpu_resume_latency(uint16_t lcore_id)
if (strcmp(buf, "n/a") == 0)
latency = RTE_POWER_QOS_STRICT_LATENCY_VALUE;
else {
- latency = strtoul(buf, NULL, 10);
+ latency = strtoul(buf, NULL, POWER_CONVERT_TO_DECIMAL);
latency = latency == 0 ? RTE_POWER_QOS_RESUME_LATENCY_NO_CONSTRAINT : latency;
}
--
2.33.0
^ permalink raw reply related
* [PATCH 2/3] power: unify decimal format macro for strtoul
From: Huisong Li @ 2026-05-09 8:45 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
In-Reply-To: <20260509084503.2917038-1-lihuisong@huawei.com>
Unify and move POWER_CONVERT_TO_DECIMAL definition to common header
file to facilitate the use of drivers and library.
Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
drivers/power/acpi/acpi_cpufreq.c | 1 -
drivers/power/amd_pstate/amd_pstate_cpufreq.c | 2 --
drivers/power/cppc/cppc_cpufreq.c | 2 --
lib/power/power_common.c | 1 -
lib/power/power_common.h | 2 ++
5 files changed, 2 insertions(+), 6 deletions(-)
diff --git a/drivers/power/acpi/acpi_cpufreq.c b/drivers/power/acpi/acpi_cpufreq.c
index 21991af36a..875c66336d 100644
--- a/drivers/power/acpi/acpi_cpufreq.c
+++ b/drivers/power/acpi/acpi_cpufreq.c
@@ -14,7 +14,6 @@
#include "power_common.h"
#define STR_SIZE 1024
-#define POWER_CONVERT_TO_DECIMAL 10
#define POWER_GOVERNOR_USERSPACE "userspace"
#define POWER_SYSFILE_AVAIL_FREQ \
diff --git a/drivers/power/amd_pstate/amd_pstate_cpufreq.c b/drivers/power/amd_pstate/amd_pstate_cpufreq.c
index 318f8c00e1..bc67981d71 100644
--- a/drivers/power/amd_pstate/amd_pstate_cpufreq.c
+++ b/drivers/power/amd_pstate/amd_pstate_cpufreq.c
@@ -16,8 +16,6 @@
#define FREQ_ROUNDING_DELTA 500
#define ROUND_FREQ_TO_N_1000 1000
-#define POWER_CONVERT_TO_DECIMAL 10
-
#define POWER_GOVERNOR_USERSPACE "userspace"
#define POWER_SYSFILE_SETSPEED \
"/sys/devices/system/cpu/cpu%u/cpufreq/scaling_setspeed"
diff --git a/drivers/power/cppc/cppc_cpufreq.c b/drivers/power/cppc/cppc_cpufreq.c
index e3ba9bb60a..9ae25bad27 100644
--- a/drivers/power/cppc/cppc_cpufreq.c
+++ b/drivers/power/cppc/cppc_cpufreq.c
@@ -20,8 +20,6 @@
*/
#define UNIT_DIFF 10000
-#define POWER_CONVERT_TO_DECIMAL 10
-
#define POWER_GOVERNOR_USERSPACE "userspace"
#define POWER_SYSFILE_SETSPEED \
"/sys/devices/system/cpu/cpu%u/cpufreq/scaling_setspeed"
diff --git a/lib/power/power_common.c b/lib/power/power_common.c
index 2da034e9d0..da22a4d160 100644
--- a/lib/power/power_common.c
+++ b/lib/power/power_common.c
@@ -21,7 +21,6 @@ RTE_LOG_REGISTER_DEFAULT(rte_power_logtype, INFO);
"/sys/devices/system/cpu/cpu%u/cpufreq/scaling_driver"
#define POWER_SYSFILE_GOVERNOR \
"/sys/devices/system/cpu/cpu%u/cpufreq/scaling_governor"
-#define POWER_CONVERT_TO_DECIMAL 10
RTE_EXPORT_INTERNAL_SYMBOL(cpufreq_check_scaling_driver)
int
diff --git a/lib/power/power_common.h b/lib/power/power_common.h
index 3f56b1103d..e2d5b68a17 100644
--- a/lib/power/power_common.h
+++ b/lib/power/power_common.h
@@ -23,6 +23,8 @@ extern int rte_power_logtype;
#define POWER_DEBUG_LOG(...)
#endif
+#define POWER_CONVERT_TO_DECIMAL 10
+
/* check if scaling driver matches one we want */
__rte_internal
int cpufreq_check_scaling_driver(const char *driver);
--
2.33.0
^ permalink raw reply related
* [PATCH 1/3] power: move power state structure to power cpufreq header
From: Huisong Li @ 2026-05-09 8:45 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
In-Reply-To: <20260509084503.2917038-1-lihuisong@huawei.com>
The "enum power_state" structure is defined by each cpufreq driver.
So move it to power_cpufreq.h to facilitate maintenance.
Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
drivers/power/acpi/acpi_cpufreq.c | 7 -------
drivers/power/amd_pstate/amd_pstate_cpufreq.c | 7 -------
drivers/power/cppc/cppc_cpufreq.c | 7 -------
drivers/power/intel_pstate/intel_pstate_cpufreq.c | 8 --------
lib/power/power_cpufreq.h | 10 ++++++++++
5 files changed, 10 insertions(+), 29 deletions(-)
diff --git a/drivers/power/acpi/acpi_cpufreq.c b/drivers/power/acpi/acpi_cpufreq.c
index 81a5e3f6ea..21991af36a 100644
--- a/drivers/power/acpi/acpi_cpufreq.c
+++ b/drivers/power/acpi/acpi_cpufreq.c
@@ -31,13 +31,6 @@
#define IA32_PERF_CTL 0x199
#define CORE_TURBO_DISABLE_BIT ((uint64_t)1<<32)
-enum power_state {
- POWER_IDLE = 0,
- POWER_ONGOING,
- POWER_USED,
- POWER_UNKNOWN
-};
-
/**
* Power info per lcore.
*/
diff --git a/drivers/power/amd_pstate/amd_pstate_cpufreq.c b/drivers/power/amd_pstate/amd_pstate_cpufreq.c
index 95495bff7d..318f8c00e1 100644
--- a/drivers/power/amd_pstate/amd_pstate_cpufreq.c
+++ b/drivers/power/amd_pstate/amd_pstate_cpufreq.c
@@ -35,13 +35,6 @@
#define POWER_AMD_PSTATE_DRIVER "amd-pstate"
#define BUS_FREQ 1000 /* khz */
-enum power_state {
- POWER_IDLE = 0,
- POWER_ONGOING,
- POWER_USED,
- POWER_UNKNOWN
-};
-
/**
* Power info per lcore.
*/
diff --git a/drivers/power/cppc/cppc_cpufreq.c b/drivers/power/cppc/cppc_cpufreq.c
index 3cd4165c83..e3ba9bb60a 100644
--- a/drivers/power/cppc/cppc_cpufreq.c
+++ b/drivers/power/cppc/cppc_cpufreq.c
@@ -39,13 +39,6 @@
#define POWER_CPPC_DRIVER "cppc_cpufreq"
#define BUS_FREQ 100000
-enum power_state {
- POWER_IDLE = 0,
- POWER_ONGOING,
- POWER_USED,
- POWER_UNKNOWN
-};
-
/**
* Power info per lcore.
*/
diff --git a/drivers/power/intel_pstate/intel_pstate_cpufreq.c b/drivers/power/intel_pstate/intel_pstate_cpufreq.c
index 8e27570e3c..22a1b4465a 100644
--- a/drivers/power/intel_pstate/intel_pstate_cpufreq.c
+++ b/drivers/power/intel_pstate/intel_pstate_cpufreq.c
@@ -41,14 +41,6 @@
"/sys/devices/system/cpu/intel_pstate/turbo_pct"
#define POWER_PSTATE_DRIVER "intel_pstate"
-
-enum power_state {
- POWER_IDLE = 0,
- POWER_ONGOING,
- POWER_USED,
- POWER_UNKNOWN
-};
-
struct __rte_cache_aligned pstate_power_info {
unsigned int lcore_id; /**< Logical core id */
uint32_t freqs[RTE_MAX_LCORE_FREQS]; /**< Frequency array */
diff --git a/lib/power/power_cpufreq.h b/lib/power/power_cpufreq.h
index fb0b7feb82..41bfaed7bb 100644
--- a/lib/power/power_cpufreq.h
+++ b/lib/power/power_cpufreq.h
@@ -18,6 +18,16 @@
#define RTE_POWER_DRIVER_NAMESZ 24
+/**
+ * Power state of cpufreq driver
+ */
+enum power_state {
+ POWER_IDLE = 0,
+ POWER_ONGOING,
+ POWER_USED,
+ POWER_UNKNOWN
+};
+
/**
* Initialize power management for a specific lcore. If rte_power_set_env() has
* not been called then an auto-detect of the environment will start and
--
2.33.0
^ permalink raw reply related
* [PATCH 0/3] power: some cleancode for cpufreq library
From: Huisong Li @ 2026-05-09 8:45 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
Move some common definition to common header file.
Huisong Li (3):
power: move power state structure to power cpufreq header
power: unify decimal format macro for strtoul
power: use common decimal macro definition
drivers/power/acpi/acpi_cpufreq.c | 8 --------
drivers/power/amd_pstate/amd_pstate_cpufreq.c | 9 ---------
drivers/power/cppc/cppc_cpufreq.c | 9 ---------
drivers/power/intel_pstate/intel_pstate_cpufreq.c | 8 --------
lib/power/power_common.c | 1 -
lib/power/power_common.h | 2 ++
lib/power/power_cpufreq.h | 10 ++++++++++
lib/power/rte_power_qos.c | 2 +-
8 files changed, 13 insertions(+), 36 deletions(-)
--
2.33.0
^ permalink raw reply
* Re: [PATCH v2] net/hns3: fix L4 checksum incorrect for tunnel packets
From: fengchengwen @ 2026-05-09 7:18 UTC (permalink / raw)
To: Xingui Yang, dev
Cc: stephen, david.marchand, lihuisong, liuyonglong, kangfenglong
In-Reply-To: <20260509064753.3292074-1-yangxingui@huawei.com>
Reviewed-by: Chengwen Feng <fengchengwen@huawei.com>
On 5/9/2026 2:47 PM, Xingui Yang wrote:
> In HIP09 simple BD mode, the driver incorrectly calculates L4_START
> position for tunnel packets. The current implementation only uses
> l2_len + l3_len without considering outer header lengths (outer_l2_len
> and outer_l3_len), causing hardware to calculate checksum from wrong
> offset.
>
> For a typical NvGRE tunnel packet with structure:
> Outer Eth(14) + Outer IPv6(40) + NvGRE(8) + Inner Eth(14) +
> Inner IPv6(40) + TCP(20)
>
> Expected L4_START: 116 bytes (from packet start to inner TCP header)
> Actual calculation: 62 bytes (missing outer headers)
>
> This results in incorrect TCP/UDP checksum for all tunnel packets when
> using simple BD mode, including NvGRE, VxLAN, GRE, GENEVE tunnels.
>
> Fix by adding outer_l2_len and outer_l3_len to L4_START calculation
> when RTE_MBUF_F_TX_TUNNEL_MASK flag is set.
>
> Fixes: 6393fc0b823c ("net/hns3: simplify hardware checksum offloading")
> Cc: stable@dpdk.org
>
> Signed-off-by: Xingui Yang <yangxingui@huawei.com>
^ permalink raw reply
* [PATCH v3 3/3] net/zxdh: optimize Tx xmit pkts performance
From: Junlong Wang @ 2026-05-09 6:29 UTC (permalink / raw)
To: stephen; +Cc: dev, Junlong Wang
In-Reply-To: <20260509062930.3254766-1-wang.junlong1@zte.com.cn>
[-- Attachment #1.1.1: Type: text/plain, Size: 18737 bytes --]
Add simple Tx xmit functions (zxdh_xmit_pkts_simple)
for single-segment packet xmit.
Signed-off-by: Junlong Wang <wang.junlong1@zte.com.cn>
---
drivers/net/zxdh/zxdh_ethdev.c | 11 +-
drivers/net/zxdh/zxdh_rxtx.c | 341 +++++++++++++++++++++++++--------
drivers/net/zxdh/zxdh_rxtx.h | 11 +-
3 files changed, 271 insertions(+), 92 deletions(-)
diff --git a/drivers/net/zxdh/zxdh_ethdev.c b/drivers/net/zxdh/zxdh_ethdev.c
index 0ab137189b..54d43b54d9 100644
--- a/drivers/net/zxdh/zxdh_ethdev.c
+++ b/drivers/net/zxdh/zxdh_ethdev.c
@@ -490,7 +490,7 @@ zxdh_dev_free_mbufs(struct rte_eth_dev *dev)
if (!vq)
continue;
while ((buf = zxdh_queue_detach_unused(vq)) != NULL)
- rte_pktmbuf_free(buf);
+ rte_pktmbuf_free_seg(buf);
PMD_DRV_LOG(DEBUG, "freeing %s[%d] used and unused buf",
"rxq", i * 2);
}
@@ -499,7 +499,7 @@ zxdh_dev_free_mbufs(struct rte_eth_dev *dev)
if (!vq)
continue;
while ((buf = zxdh_queue_detach_unused(vq)) != NULL)
- rte_pktmbuf_free(buf);
+ rte_pktmbuf_free_seg(buf);
PMD_DRV_LOG(DEBUG, "freeing %s[%d] used and unused buf",
"txq", i * 2 + 1);
}
@@ -1291,10 +1291,15 @@ static int zxdh_scattered_rx(struct rte_eth_dev *eth_dev)
static int32_t
zxdh_set_rxtx_funcs(struct rte_eth_dev *eth_dev)
{
+ uint64_t tx_offloads = eth_dev->data->dev_conf.txmode.offloads;
+
eth_dev->tx_pkt_prepare = zxdh_xmit_pkts_prepare;
eth_dev->data->scattered_rx = zxdh_scattered_rx(eth_dev);
- eth_dev->tx_pkt_burst = &zxdh_xmit_pkts_packed;
+ if (!(tx_offloads & RTE_ETH_TX_OFFLOAD_MULTI_SEGS))
+ eth_dev->tx_pkt_burst = &zxdh_xmit_pkts_simple;
+ else
+ eth_dev->tx_pkt_burst = &zxdh_xmit_pkts_packed;
if (eth_dev->data->scattered_rx)
eth_dev->rx_pkt_burst = &zxdh_recv_pkts_packed;
diff --git a/drivers/net/zxdh/zxdh_rxtx.c b/drivers/net/zxdh/zxdh_rxtx.c
index 4723d4b1d2..e8f1cd65b0 100644
--- a/drivers/net/zxdh/zxdh_rxtx.c
+++ b/drivers/net/zxdh/zxdh_rxtx.c
@@ -114,6 +114,22 @@
RTE_MBUF_F_TX_SEC_OFFLOAD | \
RTE_MBUF_F_TX_UDP_SEG)
+#if RTE_CACHE_LINE_SIZE == 128
+#define NEXT_CACHELINE_OFF_16B 8
+#define NEXT_CACHELINE_OFF_8B 16
+#elif RTE_CACHE_LINE_SIZE == 64
+#define NEXT_CACHELINE_OFF_16B 4
+#define NEXT_CACHELINE_OFF_8B 8
+#else
+#define NEXT_CACHELINE_OFF_16B (RTE_CACHE_LINE_SIZE / 16)
+#define NEXT_CACHELINE_OFF_8B (RTE_CACHE_LINE_SIZE / 8)
+#endif
+#define N_PER_LOOP NEXT_CACHELINE_OFF_8B
+#define N_PER_LOOP_MASK (N_PER_LOOP - 1)
+
+#define rxq_get_vq(q) ((q)->vq)
+#define txq_get_vq(q) ((q)->vq)
+
uint32_t zxdh_outer_l2_type[16] = {
0,
RTE_PTYPE_L2_ETHER,
@@ -201,43 +217,6 @@ uint32_t zxdh_inner_l4_type[16] = {
0,
};
-static void
-zxdh_xmit_cleanup_inorder_packed(struct zxdh_virtqueue *vq, int32_t num)
-{
- uint16_t used_idx = 0;
- uint16_t id = 0;
- uint16_t curr_id = 0;
- uint16_t free_cnt = 0;
- uint16_t size = vq->vq_nentries;
- struct zxdh_vring_packed_desc *desc = vq->vq_packed.ring.desc;
- struct zxdh_vq_desc_extra *dxp = NULL;
-
- used_idx = vq->vq_used_cons_idx;
- /* desc_is_used has a load-acquire or rte_io_rmb inside
- * and wait for used desc in virtqueue.
- */
- while (num > 0 && desc_is_used(&desc[used_idx], vq)) {
- id = desc[used_idx].id;
- do {
- curr_id = used_idx;
- dxp = &vq->vq_descx[used_idx];
- used_idx += dxp->ndescs;
- free_cnt += dxp->ndescs;
- num -= dxp->ndescs;
- if (used_idx >= size) {
- used_idx -= size;
- vq->used_wrap_counter ^= 1;
- }
- if (dxp->cookie != NULL) {
- rte_pktmbuf_free(dxp->cookie);
- dxp->cookie = NULL;
- }
- } while (curr_id != id);
- }
- vq->vq_used_cons_idx = used_idx;
- vq->vq_free_cnt += free_cnt;
-}
-
static inline uint16_t
zxdh_get_mtu(struct zxdh_virtqueue *vq)
{
@@ -334,7 +313,7 @@ zxdh_xmit_fill_net_hdr(struct zxdh_virtqueue *vq, struct rte_mbuf *cookie,
}
static inline void
-zxdh_enqueue_xmit_packed_fast(struct zxdh_virtnet_tx *txvq,
+zxdh_xmit_enqueue_push(struct zxdh_virtnet_tx *txvq,
struct rte_mbuf *cookie)
{
struct zxdh_virtqueue *vq = txvq->vq;
@@ -345,7 +324,6 @@ zxdh_enqueue_xmit_packed_fast(struct zxdh_virtnet_tx *txvq,
uint8_t hdr_len = vq->hw->dl_net_hdr_len;
struct zxdh_vring_packed_desc *dp = &vq->vq_packed.ring.desc[id];
- dxp->ndescs = 1;
dxp->cookie = cookie;
hdr = rte_pktmbuf_mtod_offset(cookie, struct zxdh_net_hdr_dl *, -hdr_len);
zxdh_xmit_fill_net_hdr(vq, cookie, hdr);
@@ -362,52 +340,57 @@ zxdh_enqueue_xmit_packed_fast(struct zxdh_virtnet_tx *txvq,
}
static inline void
-zxdh_enqueue_xmit_packed(struct zxdh_virtnet_tx *txvq,
+zxdh_xmit_enqueue_append(struct zxdh_virtnet_tx *txvq,
struct rte_mbuf *cookie,
uint16_t needed)
{
struct zxdh_tx_region *txr = txvq->zxdh_net_hdr_mz->addr;
struct zxdh_virtqueue *vq = txvq->vq;
- uint16_t id = vq->vq_avail_idx;
- struct zxdh_vq_desc_extra *dxp = &vq->vq_descx[id];
+ struct zxdh_vq_desc_extra *dep = &vq->vq_descx[0];
uint16_t head_idx = vq->vq_avail_idx;
uint16_t idx = head_idx;
struct zxdh_vring_packed_desc *start_dp = vq->vq_packed.ring.desc;
struct zxdh_vring_packed_desc *head_dp = &vq->vq_packed.ring.desc[idx];
struct zxdh_net_hdr_dl *hdr = NULL;
-
- uint16_t head_flags = cookie->next ? ZXDH_VRING_DESC_F_NEXT : 0;
+ uint16_t id = vq->vq_avail_idx;
+ struct zxdh_vq_desc_extra *dxp = &vq->vq_descx[id];
uint8_t hdr_len = vq->hw->dl_net_hdr_len;
+ uint16_t head_flags = 0;
- dxp->ndescs = needed;
- dxp->cookie = cookie;
- head_flags |= vq->cached_flags;
+ /*
+ * IMPORTANT: For multi-seg packets, we set the head descriptor's cookie to NULL
+ * and store each segment's mbuf in its corresponding vq_descx[idx].cookie.
+ * This is required for the per-descriptor mbuf free in zxdh_xmit_fast_flush()
+ * which uses rte_pktmbuf_free_seg() to free individual segments.
+ * Any code path that attempts to read vq_descx[head_id].cookie will see NULL
+ * and must handle this case appropriately.
+ */
+ dxp->cookie = NULL;
+ /* setup first tx ring slot to point to header stored in reserved region. */
start_dp[idx].addr = txvq->zxdh_net_hdr_mem + RTE_PTR_DIFF(&txr[idx].tx_hdr, txr);
start_dp[idx].len = hdr_len;
- head_flags |= ZXDH_VRING_DESC_F_NEXT;
+ start_dp[idx].id = idx;
+ head_flags |= vq->cached_flags | ZXDH_VRING_DESC_F_NEXT;
hdr = (void *)&txr[idx].tx_hdr;
- rte_prefetch1(hdr);
+ zxdh_xmit_fill_net_hdr(vq, cookie, hdr);
+
idx++;
if (idx >= vq->vq_nentries) {
idx -= vq->vq_nentries;
vq->cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
}
- zxdh_xmit_fill_net_hdr(vq, cookie, hdr);
-
do {
start_dp[idx].addr = rte_pktmbuf_iova(cookie);
start_dp[idx].len = cookie->data_len;
- start_dp[idx].id = id;
- if (likely(idx != head_idx)) {
- uint16_t flags = cookie->next ? ZXDH_VRING_DESC_F_NEXT : 0;
-
- flags |= vq->cached_flags;
- start_dp[idx].flags = flags;
- }
+ start_dp[idx].id = idx;
+ dep[idx].cookie = cookie;
+ uint16_t flags = cookie->next ? ZXDH_VRING_DESC_F_NEXT : 0;
+ flags |= vq->cached_flags;
+ start_dp[idx].flags = flags;
idx++;
if (idx >= vq->vq_nentries) {
idx -= vq->vq_nentries;
@@ -417,7 +400,6 @@ zxdh_enqueue_xmit_packed(struct zxdh_virtnet_tx *txvq,
vq->vq_free_cnt = (uint16_t)(vq->vq_free_cnt - needed);
vq->vq_avail_idx = idx;
-
zxdh_queue_store_flags_packed(head_dp, head_flags);
}
@@ -456,7 +438,7 @@ zxdh_update_packet_stats(struct zxdh_virtnet_stats *stats, struct rte_mbuf *mbuf
}
static void
-zxdh_xmit_flush(struct zxdh_virtqueue *vq)
+zxdh_xmit_fast_flush(struct zxdh_virtqueue *vq)
{
uint16_t id = 0;
uint16_t curr_id = 0;
@@ -472,20 +454,22 @@ zxdh_xmit_flush(struct zxdh_virtqueue *vq)
* for a used descriptor in the virtqueue.
*/
while (desc_is_used(&desc[used_idx], vq)) {
+ rte_prefetch0(&desc[used_idx + NEXT_CACHELINE_OFF_16B]);
id = desc[used_idx].id;
do {
+ desc[used_idx].id = used_idx;
curr_id = used_idx;
dxp = &vq->vq_descx[used_idx];
- used_idx += dxp->ndescs;
- free_cnt += dxp->ndescs;
- if (used_idx >= size) {
- used_idx -= size;
- vq->used_wrap_counter ^= 1;
- }
if (dxp->cookie != NULL) {
- rte_pktmbuf_free(dxp->cookie);
+ rte_pktmbuf_free_seg(dxp->cookie);
dxp->cookie = NULL;
}
+ used_idx += 1;
+ free_cnt += 1;
+ if (unlikely(used_idx == size)) {
+ used_idx = 0;
+ vq->used_wrap_counter ^= 1;
+ }
} while (curr_id != id);
}
vq->vq_used_cons_idx = used_idx;
@@ -499,13 +483,12 @@ zxdh_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkt
struct zxdh_virtqueue *vq = txvq->vq;
uint16_t nb_tx = 0;
- zxdh_xmit_flush(vq);
+ zxdh_xmit_fast_flush(vq);
for (nb_tx = 0; nb_tx < nb_pkts; nb_tx++) {
struct rte_mbuf *txm = tx_pkts[nb_tx];
int32_t can_push = 0;
int32_t slots = 0;
- int32_t need = 0;
rte_prefetch0(txm);
/* optimize ring usage */
@@ -522,26 +505,15 @@ zxdh_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkt
* default => number of segments + 1
**/
slots = txm->nb_segs + !can_push;
- need = slots - vq->vq_free_cnt;
/* Positive value indicates it need free vring descriptors */
- if (unlikely(need > 0)) {
- zxdh_xmit_cleanup_inorder_packed(vq, need);
- need = slots - vq->vq_free_cnt;
- if (unlikely(need > 0)) {
- PMD_TX_LOG(ERR,
- " No enough %d free tx descriptors to transmit."
- "freecnt %d",
- need,
- vq->vq_free_cnt);
- break;
- }
- }
+ if (unlikely(slots > vq->vq_free_cnt))
+ break;
/* Enqueue Packet buffers */
if (can_push)
- zxdh_enqueue_xmit_packed_fast(txvq, txm);
+ zxdh_xmit_enqueue_push(txvq, txm);
else
- zxdh_enqueue_xmit_packed(txvq, txm, slots);
+ zxdh_xmit_enqueue_append(txvq, txm, slots);
zxdh_update_packet_stats(&txvq->stats, txm);
}
txvq->stats.packets += nb_tx;
@@ -1083,3 +1055,204 @@ uint16_t zxdh_recv_single_pkts(void *rx_queue, struct rte_mbuf **rcv_pkts, uint1
}
return nb_rx;
}
+
+static inline void pkt_padding(struct rte_mbuf *cookie, struct zxdh_hw *hw)
+{
+ uint16_t mtu_or_mss = 0;
+ uint16_t pkt_flag_lw16 = ZXDH_NO_IPID_UPDATE;
+ uint16_t l3_offset;
+ uint8_t pcode = ZXDH_PCODE_NO_IP_PKT_TYPE;
+ uint8_t l3_ptype = ZXDH_PI_L3TYPE_NOIP;
+ struct zxdh_pi_hdr *pi_hdr;
+ struct zxdh_pd_hdr_dl *pd_hdr;
+ struct zxdh_net_hdr_dl *net_hdr_dl = hw->net_hdr_dl;
+ uint8_t hdr_len = hw->dl_net_hdr_len;
+ uint16_t ol_flag = 0;
+ struct zxdh_net_hdr_dl *hdr;
+
+ hdr = (struct zxdh_net_hdr_dl *)rte_pktmbuf_prepend(cookie, hdr_len);
+ rte_memcpy(hdr, net_hdr_dl, hdr_len);
+
+ if (hw->has_tx_offload) {
+ pi_hdr = &hdr->pipd_hdr_dl.pi_hdr;
+ pd_hdr = &hdr->pipd_hdr_dl.pd_hdr;
+
+ pcode = ZXDH_PCODE_IP_PKT_TYPE;
+ if (cookie->ol_flags & RTE_MBUF_F_TX_IPV6)
+ l3_ptype = ZXDH_PI_L3TYPE_IPV6;
+ else if (cookie->ol_flags & RTE_MBUF_F_TX_IPV4)
+ l3_ptype = ZXDH_PI_L3TYPE_IP;
+ else
+ pcode = ZXDH_PCODE_NO_IP_PKT_TYPE;
+
+ if (cookie->ol_flags & RTE_MBUF_F_TX_TCP_SEG) {
+ mtu_or_mss = (cookie->tso_segsz >= ZXDH_MIN_MSS) ?
+ cookie->tso_segsz : ZXDH_MIN_MSS;
+ pi_hdr->pkt_flag_hi8 |= ZXDH_TX_TCPUDP_CKSUM_CAL;
+ pkt_flag_lw16 |= ZXDH_NO_IP_FRAGMENT | ZXDH_TX_IP_CKSUM_CAL;
+ pcode = ZXDH_PCODE_TCP_PKT_TYPE;
+ } else if (cookie->ol_flags & RTE_MBUF_F_TX_UDP_SEG) {
+ mtu_or_mss = hw->eth_dev->data->mtu;
+ mtu_or_mss = (mtu_or_mss >= ZXDH_MIN_MSS) ? mtu_or_mss : ZXDH_MIN_MSS;
+ pkt_flag_lw16 |= ZXDH_TX_IP_CKSUM_CAL;
+ pi_hdr->pkt_flag_hi8 |= ZXDH_NO_TCP_FRAGMENT | ZXDH_TX_TCPUDP_CKSUM_CAL;
+ pcode = ZXDH_PCODE_UDP_PKT_TYPE;
+ } else {
+ pkt_flag_lw16 |= ZXDH_NO_IP_FRAGMENT;
+ pi_hdr->pkt_flag_hi8 |= ZXDH_NO_TCP_FRAGMENT;
+ }
+
+ if (cookie->ol_flags & RTE_MBUF_F_TX_IP_CKSUM)
+ pkt_flag_lw16 |= ZXDH_TX_IP_CKSUM_CAL;
+
+ if ((cookie->ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) == RTE_MBUF_F_TX_UDP_CKSUM) {
+ pcode = ZXDH_PCODE_UDP_PKT_TYPE;
+ pi_hdr->pkt_flag_hi8 |= ZXDH_TX_TCPUDP_CKSUM_CAL;
+ } else if ((cookie->ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) ==
+ RTE_MBUF_F_TX_TCP_CKSUM) {
+ pcode = ZXDH_PCODE_TCP_PKT_TYPE;
+ pi_hdr->pkt_flag_hi8 |= ZXDH_TX_TCPUDP_CKSUM_CAL;
+ }
+ pkt_flag_lw16 |= (mtu_or_mss >> ZXDH_MTU_MSS_UNIT_SHIFTBIT) & ZXDH_MTU_MSS_MASK;
+ pi_hdr->pkt_flag_lw16 = rte_be_to_cpu_16(pkt_flag_lw16);
+ pi_hdr->pkt_type = l3_ptype | ZXDH_PKT_FORM_CPU | pcode;
+
+ l3_offset = hdr_len + cookie->l2_len;
+ l3_offset += (cookie->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) ?
+ cookie->outer_l2_len + cookie->outer_l3_len : 0;
+ pi_hdr->l3_offset = rte_be_to_cpu_16(l3_offset);
+ pi_hdr->l4_offset = rte_be_to_cpu_16(l3_offset + cookie->l3_len);
+ if (cookie->ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM)
+ ol_flag |= ZXDH_PD_OFFLOAD_OUTER_IPCSUM;
+ } else {
+ pd_hdr = &hdr->pd_hdr;
+ }
+
+ pd_hdr->dst_vfid = rte_be_to_cpu_16(cookie->port);
+
+ if (cookie->ol_flags & (RTE_MBUF_F_TX_VLAN | RTE_MBUF_F_TX_QINQ)) {
+ ol_flag |= ZXDH_PD_OFFLOAD_CVLAN_INSERT;
+ pd_hdr->cvlan_insert = rte_be_to_cpu_16(cookie->vlan_tci);
+ if (cookie->ol_flags & RTE_MBUF_F_TX_QINQ) {
+ ol_flag |= ZXDH_PD_OFFLOAD_SVLAN_INSERT;
+ pd_hdr->svlan_insert = rte_be_to_cpu_16(cookie->vlan_tci_outer);
+ }
+ }
+
+ pd_hdr->ol_flag = rte_be_to_cpu_16(ol_flag);
+}
+
+/*
+ * Populate N_PER_LOOP descriptors with data from N_PER_LOOP single-segment mbufs.
+ * Note: The simple transmit path (zxdh_xmit_pkts_simple) is selected only when
+ * RTE_ETH_TX_OFFLOAD_MULTI_SEGS is disabled, so all packets handled here are
+ * guaranteed to be single-segment.
+ */
+static inline void
+tx_bunch(struct zxdh_virtqueue *vq, volatile struct zxdh_vring_packed_desc *txdp,
+ struct rte_mbuf **pkts, uint16_t start_id)
+{
+ uint16_t flags = vq->cached_flags;
+ int i;
+ for (i = 0; i < N_PER_LOOP; ++i, ++txdp, ++pkts) {
+ /* write data to descriptor */
+ txdp->addr = rte_mbuf_data_iova(*pkts);
+ txdp->len = (*pkts)->data_len;
+ txdp->id = start_id + i;
+ txdp->flags = flags;
+ }
+}
+
+/* Populate 1 descriptor with data from 1 single-segment mbuf */
+static inline void
+tx1(struct zxdh_virtqueue *vq, volatile struct zxdh_vring_packed_desc *txdp,
+ struct rte_mbuf *pkts, uint16_t id)
+{
+ uint16_t flags = vq->cached_flags;
+ txdp->addr = rte_mbuf_data_iova(pkts);
+ txdp->len = pkts->data_len;
+ txdp->id = id;
+ txdp->flags = flags;
+}
+
+static void submit_to_backend_simple(struct zxdh_virtqueue *vq,
+ struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
+{
+ struct zxdh_hw *hw = vq->hw;
+ struct rte_mbuf *m = NULL;
+ uint16_t id = vq->vq_avail_idx;
+ struct zxdh_vring_packed_desc *txdp = &vq->vq_packed.ring.desc[id];
+ struct zxdh_vq_desc_extra *dxp = &vq->vq_descx[id];
+ int mainpart, leftover;
+ int i, j;
+
+ /*
+ * Process most of the packets in chunks of N pkts. Any
+ * leftover packets will get processed one at a time.
+ */
+ mainpart = (nb_pkts & ~N_PER_LOOP_MASK);
+ leftover = (nb_pkts & N_PER_LOOP_MASK);
+
+ for (i = 0; i < mainpart; i += N_PER_LOOP) {
+ rte_prefetch0(dxp + i);
+ rte_prefetch0(tx_pkts + i);
+ for (j = 0; j < N_PER_LOOP; ++j) {
+ m = *(tx_pkts + i + j);
+ pkt_padding(m, hw);
+ (dxp + i + j)->cookie = (void *)m;
+ }
+ /* write data to descriptor */
+ tx_bunch(vq, txdp + i, tx_pkts + i, id + i);
+ }
+
+ if (leftover > 0) {
+ rte_prefetch0(dxp + mainpart);
+ rte_prefetch0(tx_pkts + mainpart);
+
+ for (i = 0; i < leftover; ++i) {
+ m = *(tx_pkts + mainpart + i);
+ pkt_padding(m, hw);
+ (dxp + mainpart + i)->cookie = m;
+ tx1(vq, txdp + mainpart + i, *(tx_pkts + mainpart + i), id + mainpart + i);
+ }
+ }
+}
+
+uint16_t zxdh_xmit_pkts_simple(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
+{
+ struct zxdh_virtnet_tx *txvq = tx_queue;
+ struct zxdh_virtqueue *vq = txq_get_vq(txvq);
+ uint16_t nb_tx = 0, nb_tx_left;
+
+ zxdh_xmit_fast_flush(vq);
+
+ nb_pkts = (uint16_t)RTE_MIN(nb_pkts, vq->vq_free_cnt);
+ if (unlikely(nb_pkts == 0)) {
+ txvq->stats.idle++;
+ return 0;
+ }
+
+ nb_tx_left = nb_pkts;
+ if ((vq->vq_avail_idx + nb_pkts) >= vq->vq_nentries) {
+ nb_tx = vq->vq_nentries - vq->vq_avail_idx;
+ nb_tx_left = nb_pkts - nb_tx;
+ submit_to_backend_simple(vq, tx_pkts, nb_tx);
+ vq->vq_avail_idx = 0;
+ vq->cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
+
+ vq->vq_free_cnt -= nb_tx;
+ tx_pkts += nb_tx;
+ }
+ if (nb_tx_left) {
+ submit_to_backend_simple(vq, tx_pkts, nb_tx_left);
+ vq->vq_avail_idx += nb_tx_left;
+ vq->vq_free_cnt -= nb_tx_left;
+ }
+
+ zxdh_queue_notify(vq);
+ txvq->stats.packets += nb_pkts;
+ for (nb_tx = 0; nb_tx < nb_pkts; nb_tx++)
+ zxdh_update_packet_stats(&txvq->stats, tx_pkts[nb_tx]);
+
+ return nb_pkts;
+}
diff --git a/drivers/net/zxdh/zxdh_rxtx.h b/drivers/net/zxdh/zxdh_rxtx.h
index dba9567414..783fb456de 100644
--- a/drivers/net/zxdh/zxdh_rxtx.h
+++ b/drivers/net/zxdh/zxdh_rxtx.h
@@ -56,18 +56,19 @@ struct __rte_cache_aligned zxdh_virtnet_rx {
struct __rte_cache_aligned zxdh_virtnet_tx {
struct zxdh_virtqueue *vq;
-
- rte_iova_t zxdh_net_hdr_mem; /* hdr for each xmit packet */
- uint16_t queue_id; /* DPDK queue index. */
- uint16_t port_id; /* Device port identifier. */
+ const struct rte_memzone *zxdh_net_hdr_mz; /* memzone to populate hdr. */
+ rte_iova_t zxdh_net_hdr_mem; /* hdr for each xmit packet */
struct zxdh_virtnet_stats stats;
const struct rte_memzone *mz; /* mem zone to populate TX ring. */
- const struct rte_memzone *zxdh_net_hdr_mz; /* memzone to populate hdr. */
+ uint64_t offloads;
+ uint16_t queue_id; /* DPDK queue index. */
+ uint16_t port_id; /* Device port identifier. */
};
uint16_t zxdh_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
uint16_t zxdh_xmit_pkts_prepare(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
uint16_t zxdh_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
uint16_t zxdh_recv_single_pkts(void *rx_queue, struct rte_mbuf **rcv_pkts, uint16_t nb_pkts);
+uint16_t zxdh_xmit_pkts_simple(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
#endif /* ZXDH_RXTX_H */
--
2.27.0
[-- Attachment #1.1.2: Type: text/html , Size: 47065 bytes --]
^ permalink raw reply related
* [PATCH v3 2/3] net/zxdh: optimize Rx recv pkts performance
From: Junlong Wang @ 2026-05-09 6:29 UTC (permalink / raw)
To: stephen; +Cc: dev, Junlong Wang
In-Reply-To: <20260509062930.3254766-1-wang.junlong1@zte.com.cn>
[-- Attachment #1.1.1: Type: text/plain, Size: 16207 bytes --]
- Add simple RX recv functions (zxdh_recv_single_pkts)
for single-segment packet recv.
- And optimize Rx recv pkts packed ops.
- Remove unnecessary ZXDH_NET_F_MRG_RXBUF negotiation check and
some unnecessary statistical counters form the xstats name tables.
Signed-off-by: Junlong Wang <wang.junlong1@zte.com.cn>
---
drivers/net/zxdh/zxdh_ethdev.c | 39 +++++--
drivers/net/zxdh/zxdh_ethdev_ops.c | 23 ++--
drivers/net/zxdh/zxdh_ethdev_ops.h | 4 +
drivers/net/zxdh/zxdh_rxtx.c | 173 +++++++++++++++++++++++------
drivers/net/zxdh/zxdh_rxtx.h | 16 +--
5 files changed, 192 insertions(+), 63 deletions(-)
diff --git a/drivers/net/zxdh/zxdh_ethdev.c b/drivers/net/zxdh/zxdh_ethdev.c
index 08119e28c7..0ab137189b 100644
--- a/drivers/net/zxdh/zxdh_ethdev.c
+++ b/drivers/net/zxdh/zxdh_ethdev.c
@@ -1263,18 +1263,43 @@ zxdh_dev_close(struct rte_eth_dev *dev)
return ret;
}
-static int32_t
-zxdh_set_rxtx_funcs(struct rte_eth_dev *eth_dev)
+/*
+ * Determine whether the current configuration requires support for scattered
+ * receive; return 1 if scattered receive is required and 0 if not.
+ */
+static int zxdh_scattered_rx(struct rte_eth_dev *eth_dev)
{
- struct zxdh_hw *hw = eth_dev->data->dev_private;
+ uint16_t buf_size;
- if (!zxdh_pci_with_feature(hw, ZXDH_NET_F_MRG_RXBUF)) {
- PMD_DRV_LOG(ERR, "port %u not support rx mergeable", eth_dev->data->port_id);
- return -1;
+ if (eth_dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_TCP_LRO) {
+ eth_dev->data->lro = 1;
+ return 1;
}
+
+ if (eth_dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_SCATTER)
+ return 1;
+
+ PMD_DRV_LOG(DEBUG, "port %u min_rx_buf_size %u",
+ eth_dev->data->port_id, eth_dev->data->min_rx_buf_size);
+ buf_size = eth_dev->data->min_rx_buf_size - RTE_PKTMBUF_HEADROOM;
+ if (eth_dev->data->mtu + ZXDH_ETH_OVERHEAD > buf_size)
+ return 1;
+
+ return 0;
+}
+
+static int32_t
+zxdh_set_rxtx_funcs(struct rte_eth_dev *eth_dev)
+{
eth_dev->tx_pkt_prepare = zxdh_xmit_pkts_prepare;
+ eth_dev->data->scattered_rx = zxdh_scattered_rx(eth_dev);
+
eth_dev->tx_pkt_burst = &zxdh_xmit_pkts_packed;
- eth_dev->rx_pkt_burst = &zxdh_recv_pkts_packed;
+
+ if (eth_dev->data->scattered_rx)
+ eth_dev->rx_pkt_burst = &zxdh_recv_pkts_packed;
+ else
+ eth_dev->rx_pkt_burst = &zxdh_recv_single_pkts;
return 0;
}
diff --git a/drivers/net/zxdh/zxdh_ethdev_ops.c b/drivers/net/zxdh/zxdh_ethdev_ops.c
index 50247116d9..9a8e05e941 100644
--- a/drivers/net/zxdh/zxdh_ethdev_ops.c
+++ b/drivers/net/zxdh/zxdh_ethdev_ops.c
@@ -95,10 +95,6 @@ static const struct rte_zxdh_xstats_name_off zxdh_rxq_stat_strings[] = {
{"good_bytes", offsetof(struct zxdh_virtnet_rx, stats.bytes)},
{"errors", offsetof(struct zxdh_virtnet_rx, stats.errors)},
{"idle", offsetof(struct zxdh_virtnet_rx, stats.idle)},
- {"full", offsetof(struct zxdh_virtnet_rx, stats.full)},
- {"norefill", offsetof(struct zxdh_virtnet_rx, stats.norefill)},
- {"multicast_packets", offsetof(struct zxdh_virtnet_rx, stats.multicast)},
- {"broadcast_packets", offsetof(struct zxdh_virtnet_rx, stats.broadcast)},
{"truncated_err", offsetof(struct zxdh_virtnet_rx, stats.truncated_err)},
{"offload_cfg_err", offsetof(struct zxdh_virtnet_rx, stats.offload_cfg_err)},
{"invalid_hdr_len_err", offsetof(struct zxdh_virtnet_rx, stats.invalid_hdr_len_err)},
@@ -117,14 +113,12 @@ static const struct rte_zxdh_xstats_name_off zxdh_txq_stat_strings[] = {
{"good_packets", offsetof(struct zxdh_virtnet_tx, stats.packets)},
{"good_bytes", offsetof(struct zxdh_virtnet_tx, stats.bytes)},
{"errors", offsetof(struct zxdh_virtnet_tx, stats.errors)},
- {"idle", offsetof(struct zxdh_virtnet_tx, stats.idle)},
- {"norefill", offsetof(struct zxdh_virtnet_tx, stats.norefill)},
- {"multicast_packets", offsetof(struct zxdh_virtnet_tx, stats.multicast)},
- {"broadcast_packets", offsetof(struct zxdh_virtnet_tx, stats.broadcast)},
+ {"idle", offsetof(struct zxdh_virtnet_tx, stats.idle)},
{"truncated_err", offsetof(struct zxdh_virtnet_tx, stats.truncated_err)},
{"offload_cfg_err", offsetof(struct zxdh_virtnet_tx, stats.offload_cfg_err)},
{"invalid_hdr_len_err", offsetof(struct zxdh_virtnet_tx, stats.invalid_hdr_len_err)},
{"no_segs_err", offsetof(struct zxdh_virtnet_tx, stats.no_segs_err)},
+ {"no_free_tx_desc_err", offsetof(struct zxdh_virtnet_tx, stats.no_free_tx_desc_err)},
{"undersize_packets", offsetof(struct zxdh_virtnet_tx, stats.size_bins[0])},
{"size_64_packets", offsetof(struct zxdh_virtnet_tx, stats.size_bins[1])},
{"size_65_127_packets", offsetof(struct zxdh_virtnet_tx, stats.size_bins[2])},
@@ -2026,6 +2020,19 @@ int zxdh_dev_mtu_set(struct rte_eth_dev *dev, uint16_t new_mtu)
uint16_t vfid = zxdh_vport_to_vfid(hw->vport);
int ret;
+ /* If device is started, refuse mtu that requires the support of
+ * scattered packets when this feature has not been enabled before.
+ */
+ if (dev->data->dev_started) {
+ uint32_t buf_size = dev->data->min_rx_buf_size - RTE_PKTMBUF_HEADROOM;
+ uint8_t need_scatter = (uint32_t)ZXDH_MTU_TO_PKTLEN(new_mtu) > buf_size;
+
+ if (need_scatter != dev->data->scattered_rx) {
+ PMD_DRV_LOG(ERR, "Stop port first.");
+ return -EINVAL;
+ }
+ }
+
if (hw->is_pf) {
ret = zxdh_get_panel_attr(dev, &panel);
if (ret != 0) {
diff --git a/drivers/net/zxdh/zxdh_ethdev_ops.h b/drivers/net/zxdh/zxdh_ethdev_ops.h
index 6dfe4be473..c49d79c232 100644
--- a/drivers/net/zxdh/zxdh_ethdev_ops.h
+++ b/drivers/net/zxdh/zxdh_ethdev_ops.h
@@ -40,6 +40,10 @@
#define ZXDH_SPM_SPEED_4X_100G RTE_BIT32(10)
#define ZXDH_SPM_SPEED_4X_200G RTE_BIT32(11)
+#define ZXDH_VLAN_TAG_LEN 4
+#define ZXDH_ETH_OVERHEAD (RTE_ETHER_HDR_LEN + RTE_ETHER_CRC_LEN + ZXDH_VLAN_TAG_LEN * 2)
+#define ZXDH_MTU_TO_PKTLEN(mtu) ((mtu) + ZXDH_ETH_OVERHEAD)
+
struct zxdh_np_stats_data {
uint64_t n_pkts_dropped;
uint64_t n_bytes_dropped;
diff --git a/drivers/net/zxdh/zxdh_rxtx.c b/drivers/net/zxdh/zxdh_rxtx.c
index 93506a4b49..4723d4b1d2 100644
--- a/drivers/net/zxdh/zxdh_rxtx.c
+++ b/drivers/net/zxdh/zxdh_rxtx.c
@@ -613,10 +613,12 @@ zxdh_dequeue_burst_rx_packed(struct zxdh_virtqueue *vq,
uint16_t i, used_idx;
uint16_t id;
+ used_idx = vq->vq_used_cons_idx;
+ rte_prefetch0(&desc[used_idx]);
+
for (i = 0; i < num; i++) {
used_idx = vq->vq_used_cons_idx;
- /**
- * desc_is_used has a load-acquire or rte_io_rmb inside
+ /* desc_is_used has a load-acquire or rte_io_rmb inside
* and wait for used desc in virtqueue.
*/
if (!desc_is_used(&desc[used_idx], vq))
@@ -823,17 +825,52 @@ zxdh_rx_update_mbuf(struct zxdh_hw *hw, struct rte_mbuf *m, struct zxdh_net_hdr_
}
}
-static void zxdh_discard_rxbuf(struct zxdh_virtqueue *vq, struct rte_mbuf *m)
+static void refill_desc_unwrap(struct zxdh_virtqueue *vq,
+ struct rte_mbuf **cookie, uint16_t nb_pkts)
{
- int32_t error = 0;
- /*
- * Requeue the discarded mbuf. This should always be
- * successful since it was just dequeued.
- */
- error = zxdh_enqueue_recv_refill_packed(vq, &m, 1);
- if (unlikely(error)) {
- PMD_RX_LOG(ERR, "cannot enqueue discarded mbuf");
- rte_pktmbuf_free(m);
+ struct zxdh_vring_packed_desc *start_dp = vq->vq_packed.ring.desc;
+ struct zxdh_vq_desc_extra *dxp;
+ uint16_t flags = vq->cached_flags;
+ int32_t i;
+ uint16_t idx;
+
+ idx = vq->vq_avail_idx;
+ for (i = 0; i < nb_pkts; i++) {
+ dxp = &vq->vq_descx[idx];
+ dxp->cookie = (void *)cookie[i];
+ start_dp[idx].addr = rte_mbuf_iova_get(cookie[i]) + RTE_PKTMBUF_HEADROOM;
+ start_dp[idx].len = cookie[i]->buf_len - RTE_PKTMBUF_HEADROOM;
+ zxdh_queue_store_flags_packed(&start_dp[idx], flags);
+ idx++;
+ }
+ vq->vq_avail_idx += nb_pkts;
+ vq->vq_free_cnt = vq->vq_free_cnt - nb_pkts;
+}
+
+static void refill_que_descs(struct zxdh_virtqueue *vq, struct rte_eth_dev *dev)
+{
+ /* free_cnt may include mrg descs */
+ struct rte_mbuf *new_pkts[ZXDH_MBUF_BURST_SZ];
+ uint16_t free_cnt = RTE_MIN(ZXDH_MBUF_BURST_SZ, vq->vq_free_cnt);
+ struct zxdh_virtnet_rx *rxvq = &vq->rxq;
+ uint16_t unwrap_cnt, left_cnt;
+
+ if (!rte_pktmbuf_alloc_bulk(rxvq->mpool, new_pkts, free_cnt)) {
+ left_cnt = free_cnt;
+ unwrap_cnt = 0;
+ if ((vq->vq_avail_idx + free_cnt) >= vq->vq_nentries) {
+ unwrap_cnt = vq->vq_nentries - vq->vq_avail_idx;
+ left_cnt = free_cnt - unwrap_cnt;
+ refill_desc_unwrap(vq, new_pkts, unwrap_cnt);
+ vq->vq_avail_idx = 0;
+ vq->cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
+ }
+ if (left_cnt)
+ refill_desc_unwrap(vq, new_pkts + unwrap_cnt, left_cnt);
+
+ rte_io_wmb();
+ } else {
+ dev->data->rx_mbuf_alloc_failed += free_cnt;
}
}
@@ -852,7 +889,6 @@ zxdh_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
uint16_t len = 0;
uint32_t seg_num = 0;
uint32_t seg_res = 0;
- uint32_t error = 0;
uint16_t hdr_size = 0;
uint16_t nb_rx = 0;
uint16_t i;
@@ -873,7 +909,8 @@ zxdh_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
rx_pkts[nb_rx] = rxm;
prev = rxm;
len = lens[i];
- header = rte_pktmbuf_mtod(rxm, struct zxdh_net_hdr_ul *);
+ header = (struct zxdh_net_hdr_ul *)((char *)
+ rxm->buf_addr + RTE_PKTMBUF_HEADROOM);
seg_num = header->type_hdr.num_buffers;
@@ -886,7 +923,7 @@ zxdh_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
rxvq->stats.invalid_hdr_len_err++;
continue;
}
- rxm->data_off += hdr_size;
+ rxm->data_off = RTE_PKTMBUF_HEADROOM + hdr_size;
rxm->nb_segs = seg_num;
rxm->ol_flags = 0;
rcvd_pkt_len = len - hdr_size;
@@ -902,18 +939,19 @@ zxdh_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
len = lens[i];
rxm = rcv_pkts[i];
rxm->data_len = len;
+ rxm->data_off = RTE_PKTMBUF_HEADROOM;
rcvd_pkt_len += len;
prev->next = rxm;
prev = rxm;
rxm->next = NULL;
- seg_res -= 1;
+ seg_res--;
}
if (!seg_res) {
if (rcvd_pkt_len != rx_pkts[nb_rx]->pkt_len) {
PMD_RX_LOG(ERR, "dropped rcvd_pkt_len %d pktlen %d",
rcvd_pkt_len, rx_pkts[nb_rx]->pkt_len);
- zxdh_discard_rxbuf(vq, rx_pkts[nb_rx]);
+ rte_pktmbuf_free(rx_pkts[nb_rx]);
rxvq->stats.errors++;
rxvq->stats.truncated_err++;
continue;
@@ -942,14 +980,14 @@ zxdh_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
prev->next = rxm;
prev = rxm;
rxm->next = NULL;
- extra_idx += 1;
+ extra_idx++;
}
seg_res -= rcv_cnt;
if (!seg_res) {
if (unlikely(rcvd_pkt_len != rx_pkts[nb_rx]->pkt_len)) {
PMD_RX_LOG(ERR, "dropped rcvd_pkt_len %d pktlen %d",
rcvd_pkt_len, rx_pkts[nb_rx]->pkt_len);
- zxdh_discard_rxbuf(vq, rx_pkts[nb_rx]);
+ rte_pktmbuf_free(rx_pkts[nb_rx]);
rxvq->stats.errors++;
rxvq->stats.truncated_err++;
continue;
@@ -961,26 +999,87 @@ zxdh_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
rxvq->stats.packets += nb_rx;
refill:
- /* Allocate new mbuf for the used descriptor */
- if (likely(!zxdh_queue_full(vq))) {
- struct rte_mbuf *new_pkts[ZXDH_MBUF_BURST_SZ];
- /* free_cnt may include mrg descs */
- uint16_t free_cnt = RTE_MIN(vq->vq_free_cnt, ZXDH_MBUF_BURST_SZ);
-
- if (!rte_pktmbuf_alloc_bulk(rxvq->mpool, new_pkts, free_cnt)) {
- error = zxdh_enqueue_recv_refill_packed(vq, new_pkts, free_cnt);
- if (unlikely(error)) {
- for (i = 0; i < free_cnt; i++)
- rte_pktmbuf_free(new_pkts[i]);
- }
+ if (vq->vq_free_cnt > 0) {
+ struct rte_eth_dev *dev = hw->eth_dev;
+ refill_que_descs(vq, dev);
+ zxdh_queue_notify(vq);
+ }
- if (unlikely(zxdh_queue_kick_prepare_packed(vq)))
- zxdh_queue_notify(vq);
- } else {
- struct rte_eth_dev *dev = hw->eth_dev;
+ return nb_rx;
+}
- dev->data->rx_mbuf_alloc_failed += free_cnt;
- }
+static inline int zxdh_init_mbuf(struct rte_mbuf *rxm, uint16_t len,
+ struct zxdh_hw *hw, struct zxdh_virtnet_rx *rxvq)
+{
+ uint16_t hdr_size = 0;
+ struct zxdh_net_hdr_ul *header;
+
+ header = rte_pktmbuf_mtod(rxm, struct zxdh_net_hdr_ul *);
+ rxm->ol_flags = 0;
+ rxm->vlan_tci = 0;
+ rxm->vlan_tci_outer = 0;
+
+ hdr_size = header->type_hdr.pd_len << 1;
+ if (unlikely(header->type_hdr.num_buffers != 1)) {
+ PMD_RX_LOG(DEBUG, "hdr_size:%u nb_segs %d is invalid",
+ hdr_size, header->type_hdr.num_buffers);
+ rte_pktmbuf_free(rxm);
+ rxvq->stats.invalid_hdr_len_err++;
+ return -1;
+ }
+ zxdh_rx_update_mbuf(hw, rxm, header);
+
+ rxm->nb_segs = 1;
+ rxm->data_off = RTE_PKTMBUF_HEADROOM + hdr_size;
+ rxm->data_len = len - hdr_size;
+ rxm->port = hw->port_id;
+
+ if (rxm->data_len != rxm->pkt_len) {
+ PMD_RX_LOG(ERR, "dropped rcvd_pkt_len %d pktlen %d bufaddr %p.",
+ rxm->data_len, rxm->pkt_len, rxm->buf_addr);
+ rte_pktmbuf_free(rxm);
+ rxvq->stats.truncated_err++;
+ rxvq->stats.errors++;
+ return -1;
+ }
+ return 0;
+}
+
+uint16_t zxdh_recv_single_pkts(void *rx_queue, struct rte_mbuf **rcv_pkts, uint16_t nb_pkts)
+{
+ struct zxdh_virtnet_rx *rxvq = rx_queue;
+ struct zxdh_virtqueue *vq = rxvq->vq;
+ struct zxdh_hw *hw = vq->hw;
+ uint32_t lens[ZXDH_MBUF_BURST_SZ];
+ uint16_t nb_rx = 0;
+ uint16_t num;
+ uint16_t i;
+
+ num = nb_pkts;
+ if (unlikely(num > ZXDH_MBUF_BURST_SZ))
+ num = ZXDH_MBUF_BURST_SZ;
+ num = zxdh_dequeue_burst_rx_packed(vq, rcv_pkts, lens, num);
+ if (num == 0) {
+ rxvq->stats.idle++;
+ goto refill;
+ }
+
+ for (i = 0; i < num; i++) {
+ struct rte_mbuf *rxm = rcv_pkts[i];
+ uint16_t len = lens[i];
+
+ if (unlikely(zxdh_init_mbuf(rxm, len, hw, &vq->rxq) < 0))
+ continue;
+ zxdh_update_packet_stats(&rxvq->stats, rxm);
+ nb_rx++;
+ }
+ rxvq->stats.packets += nb_rx;
+
+refill:
+ if (vq->vq_free_cnt > 0) {
+ struct rte_eth_dev *dev = hw->eth_dev;
+ refill_que_descs(vq, dev);
+ zxdh_queue_notify(vq);
}
return nb_rx;
}
diff --git a/drivers/net/zxdh/zxdh_rxtx.h b/drivers/net/zxdh/zxdh_rxtx.h
index 424048607e..dba9567414 100644
--- a/drivers/net/zxdh/zxdh_rxtx.h
+++ b/drivers/net/zxdh/zxdh_rxtx.h
@@ -36,29 +36,22 @@ struct zxdh_virtnet_stats {
uint64_t bytes;
uint64_t errors;
uint64_t idle;
- uint64_t full;
- uint64_t norefill;
- uint64_t multicast;
- uint64_t broadcast;
uint64_t truncated_err;
uint64_t offload_cfg_err;
uint64_t invalid_hdr_len_err;
uint64_t no_segs_err;
+ uint64_t no_free_tx_desc_err;
uint64_t size_bins[8];
};
struct __rte_cache_aligned zxdh_virtnet_rx {
struct zxdh_virtqueue *vq;
-
- uint64_t mbuf_initializer; /* value to init mbufs. */
struct rte_mempool *mpool; /* mempool for mbuf allocation */
- uint16_t queue_id; /* DPDK queue index. */
- uint16_t port_id; /* Device port identifier. */
struct zxdh_virtnet_stats stats;
const struct rte_memzone *mz; /* mem zone to populate RX ring. */
-
- /* dummy mbuf, for wraparound when processing RX ring. */
- struct rte_mbuf fake_mbuf;
+ uint64_t offloads;
+ uint16_t queue_id; /* DPDK queue index. */
+ uint16_t port_id; /* Device port identifier. */
};
struct __rte_cache_aligned zxdh_virtnet_tx {
@@ -75,5 +68,6 @@ struct __rte_cache_aligned zxdh_virtnet_tx {
uint16_t zxdh_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
uint16_t zxdh_xmit_pkts_prepare(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
uint16_t zxdh_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+uint16_t zxdh_recv_single_pkts(void *rx_queue, struct rte_mbuf **rcv_pkts, uint16_t nb_pkts);
#endif /* ZXDH_RXTX_H */
--
2.27.0
[-- Attachment #1.1.2: Type: text/html , Size: 39003 bytes --]
^ permalink raw reply related
* [PATCH v3 0/3] net/zxdh: optimize Rx/Tx path performance
From: Junlong Wang @ 2026-05-09 6:29 UTC (permalink / raw)
To: stephen; +Cc: dev, Junlong Wang
In-Reply-To: <20260423011820.2426203-1-wang.junlong1@zte.com.cn>
[-- Attachment #1.1.1: Type: text/plain, Size: 2932 bytes --]
v3:
- remove unnecessary NULL check in zxdh_init_queue.
- Split Ring: Bit[31] is unused and reserved, zxdh_queue_notify(): removing the
zxdh_pci_with_feature(hw, ZXDH_F_RING_PACKED) check;
- remove unnecessary double-free in in zxdh_recv_single_pkts();
- used rte_pktmbuf_mtod();
- remove rxq_get_vq(q) macro, use q->vq and apply it consistently;
- Refactoring scatter and mtu check logic in zxdh_dev_mtu_set();
- set txdp->id = avail_idx + i in tx_bunch/tx1.
- add comment documenting zxdh_xmit_enqueue_append() now sets dxp->cookie = NULL for
the head slot and stores cookies per descriptor via dep[idx].cookie.
- add one-line comment noting tx_bunch() is the simple path handles single-segment.
- remove unnecessary Extra initialization and the uint32_t cast.
v2:
- zxdh_rxtx.c, pkt_padding(): modifyed the return value of pkt_padding();
- zxdh_rxtx.c, zxdh_recv_single_pkts(): modifyed When zxdh_init_mbuf() fails
the loop does "continue" and free mbufs;
- zxdh_rxtx.c, refill_desc_unwrap(): Add rte_io_wmb() before writing flags
in the refill_que_descs();
- zxdh_queue.h, zxdh_queue_enable_intr(): Remove unnecessary function of zxdh_queue_enable_intr;
- zxdh_ethdev.c, zxdh_init_queue(): changed the hdr_mz NULL check logic;
- zxdh_rxtx.c, zxdh_xmit_pkts_simple()、zxdh_recv_single_pkts(): add stats.bytes count;
- zxdh_rxtx.c, zxdh_init_mbuf():remove rte_pktmbuf_dump(stdout, rxm, 40);
- zxdh_ethdev.c, zxdh_dev_free_mbufs(): using rte_pktmbuf_free() to free mbufs;
- Splitting into separate patches, structure reorganization and sw_ring removal、
RX recv optimize、Tx xmit optimize、Tx;
v1:
This patch optimizes the ZXDH PMD's receive and transmit path for better
performance through several improvements:
- Add simple TX/RX burst functions (zxdh_xmit_pkts_simple and
zxdh_recv_single_pkts) for single-segment packet scenarios.
- Remove RX software ring (sw_ring) to reduce memory allocation and
copy.
- Optimize descriptor management with prefetching and simplified
cleanup.
- Reorganize structure fields for better cache locality.
These changes reduce CPU cycles and memory bandwidth consumption,
resulting in improved packet processing throughput.
Junlong Wang (3):
net/zxdh: optimize queue structure to improve performance
net/zxdh: optimize Rx recv pkts performance
net/zxdh: optimize Tx xmit pkts performance
drivers/net/zxdh/zxdh_ethdev.c | 81 ++---
drivers/net/zxdh/zxdh_ethdev_ops.c | 23 +-
drivers/net/zxdh/zxdh_ethdev_ops.h | 4 +
drivers/net/zxdh/zxdh_pci.c | 2 +-
drivers/net/zxdh/zxdh_queue.c | 11 +-
drivers/net/zxdh/zxdh_queue.h | 120 ++++---
drivers/net/zxdh/zxdh_rxtx.c | 524 ++++++++++++++++++++++-------
drivers/net/zxdh/zxdh_rxtx.h | 27 +-
8 files changed, 533 insertions(+), 259 deletions(-)
--
2.27.0
[-- Attachment #1.1.2: Type: text/html , Size: 5226 bytes --]
^ permalink raw reply
* [PATCH v3 1/3] net/zxdh: optimize queue structure to improve performance
From: Junlong Wang @ 2026-05-09 6:29 UTC (permalink / raw)
To: stephen; +Cc: dev, Junlong Wang
In-Reply-To: <20260509062930.3254766-1-wang.junlong1@zte.com.cn>
[-- Attachment #1.1.1: Type: text/plain, Size: 16714 bytes --]
Reorganize structure fields for better cache locality.
Remove RX software ring (sw_ring) to reduce memory allocation and
copy.
Signed-off-by: Junlong Wang <wang.junlong1@zte.com.cn>
---
drivers/net/zxdh/zxdh_ethdev.c | 33 +--------
drivers/net/zxdh/zxdh_pci.c | 2 +-
drivers/net/zxdh/zxdh_queue.c | 11 ++-
drivers/net/zxdh/zxdh_queue.h | 120 ++++++++++++++++-----------------
drivers/net/zxdh/zxdh_rxtx.c | 22 +++---
5 files changed, 77 insertions(+), 111 deletions(-)
diff --git a/drivers/net/zxdh/zxdh_ethdev.c b/drivers/net/zxdh/zxdh_ethdev.c
index aeb01f4652..08119e28c7 100644
--- a/drivers/net/zxdh/zxdh_ethdev.c
+++ b/drivers/net/zxdh/zxdh_ethdev.c
@@ -644,7 +644,6 @@ zxdh_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_logic_qidx)
struct zxdh_virtnet_tx *txvq = NULL;
struct zxdh_virtqueue *vq = NULL;
size_t sz_hdr_mz = 0;
- void *sw_ring = NULL;
int32_t queue_type = zxdh_get_queue_type(vtpci_logic_qidx);
int32_t numa_node = dev->device->numa_node;
uint16_t vtpci_phy_qidx = 0;
@@ -692,11 +691,10 @@ zxdh_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_logic_qidx)
vq->vq_queue_index = vtpci_phy_qidx;
vq->vq_nentries = vq_size;
- vq->vq_packed.used_wrap_counter = 1;
- vq->vq_packed.cached_flags = ZXDH_VRING_PACKED_DESC_F_AVAIL;
- vq->vq_packed.event_flags_shadow = 0;
+ vq->used_wrap_counter = 1;
+ vq->cached_flags = ZXDH_VRING_PACKED_DESC_F_AVAIL;
if (queue_type == ZXDH_VTNET_RQ)
- vq->vq_packed.cached_flags |= ZXDH_VRING_DESC_F_WRITE;
+ vq->cached_flags |= ZXDH_VRING_DESC_F_WRITE;
/*
* Reserve a memzone for vring elements
@@ -741,16 +739,6 @@ zxdh_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_logic_qidx)
}
if (queue_type == ZXDH_VTNET_RQ) {
- size_t sz_sw = (ZXDH_MBUF_BURST_SZ + vq_size) * sizeof(vq->sw_ring[0]);
-
- sw_ring = rte_zmalloc_socket("sw_ring", sz_sw, RTE_CACHE_LINE_SIZE, numa_node);
- if (!sw_ring) {
- PMD_DRV_LOG(ERR, "can not allocate RX soft ring");
- ret = -ENOMEM;
- goto fail_q_alloc;
- }
-
- vq->sw_ring = sw_ring;
rxvq = &vq->rxq;
rxvq->vq = vq;
rxvq->port_id = dev->data->port_id;
@@ -764,23 +752,9 @@ zxdh_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_logic_qidx)
txvq->zxdh_net_hdr_mem = hdr_mz->iova;
}
- vq->offset = offsetof(struct rte_mbuf, buf_iova);
if (queue_type == ZXDH_VTNET_TQ) {
struct zxdh_tx_region *txr = hdr_mz->addr;
- uint32_t i;
-
memset(txr, 0, vq_size * sizeof(*txr));
- for (i = 0; i < vq_size; i++) {
- /* first indirect descriptor is always the tx header */
- struct zxdh_vring_packed_desc *start_dp = txr[i].tx_packed_indir;
-
- zxdh_vring_desc_init_indirect_packed(start_dp,
- RTE_DIM(txr[i].tx_packed_indir));
- start_dp->addr = txvq->zxdh_net_hdr_mem + i * sizeof(*txr) +
- offsetof(struct zxdh_tx_region, tx_hdr);
- /* length will be updated to actual pi hdr size when xmit pkt */
- start_dp->len = 0;
- }
}
if (ZXDH_VTPCI_OPS(hw)->setup_queue(hw, vq) < 0) {
PMD_DRV_LOG(ERR, "setup_queue failed");
@@ -788,7 +762,6 @@ zxdh_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_logic_qidx)
}
return 0;
fail_q_alloc:
- rte_free(sw_ring);
rte_memzone_free(hdr_mz);
rte_memzone_free(mz);
rte_free(vq);
diff --git a/drivers/net/zxdh/zxdh_pci.c b/drivers/net/zxdh/zxdh_pci.c
index 4ba31905fc..0bc27ed111 100644
--- a/drivers/net/zxdh/zxdh_pci.c
+++ b/drivers/net/zxdh/zxdh_pci.c
@@ -231,7 +231,7 @@ zxdh_notify_queue(struct zxdh_hw *hw, struct zxdh_virtqueue *vq)
notify_data = ((uint32_t)vq->vq_avail_idx << 16) | vq->vq_queue_index;
if (zxdh_pci_with_feature(hw, ZXDH_F_RING_PACKED) &&
- (vq->vq_packed.cached_flags & ZXDH_VRING_PACKED_DESC_F_AVAIL))
+ (vq->cached_flags & ZXDH_VRING_PACKED_DESC_F_AVAIL))
notify_data |= RTE_BIT32(31);
PMD_DRV_LOG(DEBUG, "queue:%d notify_data 0x%x notify_addr 0x%p",
diff --git a/drivers/net/zxdh/zxdh_queue.c b/drivers/net/zxdh/zxdh_queue.c
index 7162593b16..4668cb5d13 100644
--- a/drivers/net/zxdh/zxdh_queue.c
+++ b/drivers/net/zxdh/zxdh_queue.c
@@ -407,7 +407,7 @@ int32_t zxdh_enqueue_recv_refill_packed(struct zxdh_virtqueue *vq,
{
struct zxdh_vring_packed_desc *start_dp = vq->vq_packed.ring.desc;
struct zxdh_vq_desc_extra *dxp;
- uint16_t flags = vq->vq_packed.cached_flags;
+ uint16_t flags = vq->cached_flags;
int32_t i;
uint16_t idx;
@@ -415,7 +415,6 @@ int32_t zxdh_enqueue_recv_refill_packed(struct zxdh_virtqueue *vq,
idx = vq->vq_avail_idx;
dxp = &vq->vq_descx[idx];
dxp->cookie = (void *)cookie[i];
- dxp->ndescs = 1;
/* rx pkt fill in data_off */
start_dp[idx].addr = rte_mbuf_iova_get(cookie[i]) + RTE_PKTMBUF_HEADROOM;
start_dp[idx].len = cookie[i]->buf_len - RTE_PKTMBUF_HEADROOM;
@@ -423,8 +422,8 @@ int32_t zxdh_enqueue_recv_refill_packed(struct zxdh_virtqueue *vq,
zxdh_queue_store_flags_packed(&start_dp[idx], flags);
if (++vq->vq_avail_idx >= vq->vq_nentries) {
vq->vq_avail_idx -= vq->vq_nentries;
- vq->vq_packed.cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
- flags = vq->vq_packed.cached_flags;
+ vq->cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
+ flags = vq->cached_flags;
}
}
vq->vq_free_cnt = (uint16_t)(vq->vq_free_cnt - num);
@@ -467,7 +466,7 @@ void zxdh_queue_rxvq_flush(struct zxdh_virtqueue *vq)
int32_t cnt = 0;
i = vq->vq_used_cons_idx;
- while (zxdh_desc_used(&descs[i], vq) && cnt++ < vq->vq_nentries) {
+ while (desc_is_used(&descs[i], vq) && cnt++ < vq->vq_nentries) {
dxp = &vq->vq_descx[descs[i].id];
if (dxp->cookie != NULL) {
rte_pktmbuf_free(dxp->cookie);
@@ -477,7 +476,7 @@ void zxdh_queue_rxvq_flush(struct zxdh_virtqueue *vq)
vq->vq_used_cons_idx++;
if (vq->vq_used_cons_idx >= vq->vq_nentries) {
vq->vq_used_cons_idx -= vq->vq_nentries;
- vq->vq_packed.used_wrap_counter ^= 1;
+ vq->used_wrap_counter ^= 1;
}
i = vq->vq_used_cons_idx;
}
diff --git a/drivers/net/zxdh/zxdh_queue.h b/drivers/net/zxdh/zxdh_queue.h
index 1a0c8a0d90..b079272162 100644
--- a/drivers/net/zxdh/zxdh_queue.h
+++ b/drivers/net/zxdh/zxdh_queue.h
@@ -9,6 +9,7 @@
#include <rte_common.h>
#include <rte_atomic.h>
+#include <rte_io.h>
#include "zxdh_ethdev.h"
#include "zxdh_rxtx.h"
@@ -117,7 +118,6 @@ struct zxdh_vring_packed_desc_event {
};
struct zxdh_vring_packed {
- uint32_t num;
struct zxdh_vring_packed_desc *desc;
struct zxdh_vring_packed_desc_event *driver;
struct zxdh_vring_packed_desc_event *device;
@@ -129,50 +129,59 @@ struct zxdh_vq_desc_extra {
uint16_t next;
};
+struct zxdh_vring {
+ uint32_t num;
+ struct zxdh_vring_desc *desc;
+ struct zxdh_vring_avail *avail;
+ struct zxdh_vring_used *used;
+};
+
struct zxdh_virtqueue {
+ union {
+ struct {
+ struct zxdh_vring ring; /**< vring keeping desc, used and avail */
+ } vq_split;
+ struct __rte_packed_begin {
+ struct zxdh_vring_packed ring;
+ } __rte_packed_end vq_packed;
+ };
struct zxdh_hw *hw; /* < zxdh_hw structure pointer. */
- struct {
- /* vring keeping descs and events */
- struct zxdh_vring_packed ring;
- uint8_t used_wrap_counter;
- uint8_t rsv;
- uint16_t cached_flags; /* < cached flags for descs */
- uint16_t event_flags_shadow;
- uint16_t rsv1;
- } vq_packed;
-
- uint16_t vq_used_cons_idx; /* < last consumed descriptor */
- uint16_t vq_nentries; /* < vring desc numbers */
- uint16_t vq_free_cnt; /* < num of desc available */
- uint16_t vq_avail_idx; /* < sync until needed */
- uint16_t vq_free_thresh; /* < free threshold */
- uint16_t rsv2;
-
- void *vq_ring_virt_mem; /* < linear address of vring */
- uint32_t vq_ring_size;
+ uint16_t vq_used_cons_idx; /**< last consumed descriptor */
+ uint16_t vq_avail_idx; /**< sync until needed */
+ uint16_t vq_nentries; /**< vring desc numbers */
+ uint16_t vq_free_cnt; /**< num of desc available */
+
+ uint16_t cached_flags; /**< cached flags for descs */
+ uint8_t used_wrap_counter;
+ uint8_t rsv;
+ uint16_t vq_free_thresh; /**< free threshold */
+ uint16_t next_qidx;
+
+ void *notify_addr;
union {
struct zxdh_virtnet_rx rxq;
struct zxdh_virtnet_tx txq;
};
- /*
- * physical address of vring, or virtual address
- */
- rte_iova_t vq_ring_mem;
+ uint16_t vq_queue_index; /* PACKED: phy_idx, SPLIT: logic_idx */
+ uint16_t event_flags_shadow;
+ uint32_t vq_ring_size;
- /*
+ /**
* Head of the free chain in the descriptor table. If
* there are no free descriptors, this will be set to
* VQ_RING_DESC_CHAIN_END.
- */
+ **/
uint16_t vq_desc_head_idx;
uint16_t vq_desc_tail_idx;
- uint16_t vq_queue_index; /* < PCI queue index */
- uint16_t offset; /* < relative offset to obtain addr in mbuf */
- uint16_t *notify_addr;
- struct rte_mbuf **sw_ring; /* < RX software ring. */
+ uint32_t rsv_8B;
+
+ void *vq_ring_virt_mem; /**< linear address of vring*/
+ /* physical address of vring, or virtual address for virtio_user. */
+ rte_iova_t vq_ring_mem;
+
struct zxdh_vq_desc_extra vq_descx[];
};
@@ -296,10 +305,9 @@ static inline void
zxdh_vring_init_packed(struct zxdh_vring_packed *vr, uint8_t *p,
unsigned long align, uint32_t num)
{
- vr->num = num;
vr->desc = (struct zxdh_vring_packed_desc *)p;
vr->driver = (struct zxdh_vring_packed_desc_event *)(p +
- vr->num * sizeof(struct zxdh_vring_packed_desc));
+ num * sizeof(struct zxdh_vring_packed_desc));
vr->device = (struct zxdh_vring_packed_desc_event *)RTE_ALIGN_CEIL(((uintptr_t)vr->driver +
sizeof(struct zxdh_vring_packed_desc_event)), align);
}
@@ -331,30 +339,21 @@ zxdh_vring_desc_init_indirect_packed(struct zxdh_vring_packed_desc *dp, int32_t
static inline void
zxdh_queue_disable_intr(struct zxdh_virtqueue *vq)
{
- if (vq->vq_packed.event_flags_shadow != ZXDH_RING_EVENT_FLAGS_DISABLE) {
- vq->vq_packed.event_flags_shadow = ZXDH_RING_EVENT_FLAGS_DISABLE;
- vq->vq_packed.ring.driver->desc_event_flags = vq->vq_packed.event_flags_shadow;
+ if (vq->event_flags_shadow != ZXDH_RING_EVENT_FLAGS_DISABLE) {
+ vq->event_flags_shadow = ZXDH_RING_EVENT_FLAGS_DISABLE;
+ vq->vq_packed.ring.driver->desc_event_flags = vq->event_flags_shadow;
}
}
static inline void
zxdh_queue_enable_intr(struct zxdh_virtqueue *vq)
{
- if (vq->vq_packed.event_flags_shadow == ZXDH_RING_EVENT_FLAGS_DISABLE) {
- vq->vq_packed.event_flags_shadow = ZXDH_RING_EVENT_FLAGS_DISABLE;
- vq->vq_packed.ring.driver->desc_event_flags = vq->vq_packed.event_flags_shadow;
+ if (vq->event_flags_shadow != ZXDH_RING_EVENT_FLAGS_ENABLE) {
+ vq->event_flags_shadow = ZXDH_RING_EVENT_FLAGS_ENABLE;
+ vq->vq_packed.ring.driver->desc_event_flags = vq->event_flags_shadow;
}
}
-static inline void
-zxdh_mb(uint8_t weak_barriers)
-{
- if (weak_barriers)
- rte_atomic_thread_fence(rte_memory_order_seq_cst);
- else
- rte_mb();
-}
-
static inline
int32_t desc_is_used(struct zxdh_vring_packed_desc *desc, struct zxdh_virtqueue *vq)
{
@@ -365,7 +364,7 @@ int32_t desc_is_used(struct zxdh_vring_packed_desc *desc, struct zxdh_virtqueue
rte_io_rmb();
used = !!(flags & ZXDH_VRING_PACKED_DESC_F_USED);
avail = !!(flags & ZXDH_VRING_PACKED_DESC_F_AVAIL);
- return avail == used && used == vq->vq_packed.used_wrap_counter;
+ return avail == used && used == vq->used_wrap_counter;
}
static inline int32_t
@@ -381,22 +380,17 @@ zxdh_queue_store_flags_packed(struct zxdh_vring_packed_desc *dp, uint16_t flags)
dp->flags = flags;
}
-static inline int32_t
-zxdh_desc_used(struct zxdh_vring_packed_desc *desc, struct zxdh_virtqueue *vq)
-{
- uint16_t flags;
- uint16_t used, avail;
-
- flags = desc->flags;
- rte_io_rmb();
- used = !!(flags & ZXDH_VRING_PACKED_DESC_F_USED);
- avail = !!(flags & ZXDH_VRING_PACKED_DESC_F_AVAIL);
- return avail == used && used == vq->vq_packed.used_wrap_counter;
-}
-
static inline void zxdh_queue_notify(struct zxdh_virtqueue *vq)
{
- ZXDH_VTPCI_OPS(vq->hw)->notify_queue(vq->hw, vq);
+ /* Bit[0:15]: vq queue index
+ * Bit[16:30]: avail index
+ * Bit[31]: avail wrap counter
+ */
+ uint32_t notify_data = ((uint32_t)(!!(vq->cached_flags &
+ ZXDH_VRING_PACKED_DESC_F_AVAIL)) << 31) |
+ ((uint32_t)vq->vq_avail_idx << 16) |
+ vq->vq_queue_index;
+ rte_write32(notify_data, vq->notify_addr);
}
static inline int32_t
@@ -404,7 +398,7 @@ zxdh_queue_kick_prepare_packed(struct zxdh_virtqueue *vq)
{
uint16_t flags = 0;
- zxdh_mb(1);
+ rte_mb();
flags = vq->vq_packed.ring.device->desc_event_flags;
return (flags != ZXDH_RING_EVENT_FLAGS_DISABLE);
diff --git a/drivers/net/zxdh/zxdh_rxtx.c b/drivers/net/zxdh/zxdh_rxtx.c
index db86922aea..93506a4b49 100644
--- a/drivers/net/zxdh/zxdh_rxtx.c
+++ b/drivers/net/zxdh/zxdh_rxtx.c
@@ -216,7 +216,7 @@ zxdh_xmit_cleanup_inorder_packed(struct zxdh_virtqueue *vq, int32_t num)
/* desc_is_used has a load-acquire or rte_io_rmb inside
* and wait for used desc in virtqueue.
*/
- while (num > 0 && zxdh_desc_used(&desc[used_idx], vq)) {
+ while (num > 0 && desc_is_used(&desc[used_idx], vq)) {
id = desc[used_idx].id;
do {
curr_id = used_idx;
@@ -226,7 +226,7 @@ zxdh_xmit_cleanup_inorder_packed(struct zxdh_virtqueue *vq, int32_t num)
num -= dxp->ndescs;
if (used_idx >= size) {
used_idx -= size;
- vq->vq_packed.used_wrap_counter ^= 1;
+ vq->used_wrap_counter ^= 1;
}
if (dxp->cookie != NULL) {
rte_pktmbuf_free(dxp->cookie);
@@ -340,7 +340,7 @@ zxdh_enqueue_xmit_packed_fast(struct zxdh_virtnet_tx *txvq,
struct zxdh_virtqueue *vq = txvq->vq;
uint16_t id = vq->vq_avail_idx;
struct zxdh_vq_desc_extra *dxp = &vq->vq_descx[id];
- uint16_t flags = vq->vq_packed.cached_flags;
+ uint16_t flags = vq->cached_flags;
struct zxdh_net_hdr_dl *hdr = NULL;
uint8_t hdr_len = vq->hw->dl_net_hdr_len;
struct zxdh_vring_packed_desc *dp = &vq->vq_packed.ring.desc[id];
@@ -355,7 +355,7 @@ zxdh_enqueue_xmit_packed_fast(struct zxdh_virtnet_tx *txvq,
dp->id = id;
if (++vq->vq_avail_idx >= vq->vq_nentries) {
vq->vq_avail_idx -= vq->vq_nentries;
- vq->vq_packed.cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
+ vq->cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
}
vq->vq_free_cnt--;
zxdh_queue_store_flags_packed(dp, flags);
@@ -381,7 +381,7 @@ zxdh_enqueue_xmit_packed(struct zxdh_virtnet_tx *txvq,
dxp->ndescs = needed;
dxp->cookie = cookie;
- head_flags |= vq->vq_packed.cached_flags;
+ head_flags |= vq->cached_flags;
start_dp[idx].addr = txvq->zxdh_net_hdr_mem + RTE_PTR_DIFF(&txr[idx].tx_hdr, txr);
start_dp[idx].len = hdr_len;
@@ -392,7 +392,7 @@ zxdh_enqueue_xmit_packed(struct zxdh_virtnet_tx *txvq,
idx++;
if (idx >= vq->vq_nentries) {
idx -= vq->vq_nentries;
- vq->vq_packed.cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
+ vq->cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
}
zxdh_xmit_fill_net_hdr(vq, cookie, hdr);
@@ -404,14 +404,14 @@ zxdh_enqueue_xmit_packed(struct zxdh_virtnet_tx *txvq,
if (likely(idx != head_idx)) {
uint16_t flags = cookie->next ? ZXDH_VRING_DESC_F_NEXT : 0;
- flags |= vq->vq_packed.cached_flags;
+ flags |= vq->cached_flags;
start_dp[idx].flags = flags;
}
idx++;
if (idx >= vq->vq_nentries) {
idx -= vq->vq_nentries;
- vq->vq_packed.cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
+ vq->cached_flags ^= ZXDH_VRING_PACKED_DESC_F_AVAIL_USED;
}
} while ((cookie = cookie->next) != NULL);
@@ -480,7 +480,7 @@ zxdh_xmit_flush(struct zxdh_virtqueue *vq)
free_cnt += dxp->ndescs;
if (used_idx >= size) {
used_idx -= size;
- vq->vq_packed.used_wrap_counter ^= 1;
+ vq->used_wrap_counter ^= 1;
}
if (dxp->cookie != NULL) {
rte_pktmbuf_free(dxp->cookie);
@@ -619,7 +619,7 @@ zxdh_dequeue_burst_rx_packed(struct zxdh_virtqueue *vq,
* desc_is_used has a load-acquire or rte_io_rmb inside
* and wait for used desc in virtqueue.
*/
- if (!zxdh_desc_used(&desc[used_idx], vq))
+ if (!desc_is_used(&desc[used_idx], vq))
return i;
len[i] = desc[used_idx].len;
id = desc[used_idx].id;
@@ -637,7 +637,7 @@ zxdh_dequeue_burst_rx_packed(struct zxdh_virtqueue *vq,
vq->vq_used_cons_idx++;
if (vq->vq_used_cons_idx >= vq->vq_nentries) {
vq->vq_used_cons_idx -= vq->vq_nentries;
- vq->vq_packed.used_wrap_counter ^= 1;
+ vq->used_wrap_counter ^= 1;
}
}
return i;
--
2.27.0
[-- Attachment #1.1.2: Type: text/html , Size: 38617 bytes --]
^ permalink raw reply related
* [PATCH v2] net/hns3: fix L4 checksum incorrect for tunnel packets
From: Xingui Yang @ 2026-05-09 6:47 UTC (permalink / raw)
To: dev
Cc: stephen, david.marchand, fengchengwen, lihuisong, liuyonglong,
kangfenglong
In HIP09 simple BD mode, the driver incorrectly calculates L4_START
position for tunnel packets. The current implementation only uses
l2_len + l3_len without considering outer header lengths (outer_l2_len
and outer_l3_len), causing hardware to calculate checksum from wrong
offset.
For a typical NvGRE tunnel packet with structure:
Outer Eth(14) + Outer IPv6(40) + NvGRE(8) + Inner Eth(14) +
Inner IPv6(40) + TCP(20)
Expected L4_START: 116 bytes (from packet start to inner TCP header)
Actual calculation: 62 bytes (missing outer headers)
This results in incorrect TCP/UDP checksum for all tunnel packets when
using simple BD mode, including NvGRE, VxLAN, GRE, GENEVE tunnels.
Fix by adding outer_l2_len and outer_l3_len to L4_START calculation
when RTE_MBUF_F_TX_TUNNEL_MASK flag is set.
Fixes: 6393fc0b823c ("net/hns3: simplify hardware checksum offloading")
Cc: stable@dpdk.org
Signed-off-by: Xingui Yang <yangxingui@huawei.com>
---
drivers/net/hns3/hns3_rxtx.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hns3/hns3_rxtx.c b/drivers/net/hns3/hns3_rxtx.c
index 573604b0cd..4d48cbdc11 100644
--- a/drivers/net/hns3/hns3_rxtx.c
+++ b/drivers/net/hns3/hns3_rxtx.c
@@ -3982,6 +3982,7 @@ hns3_handle_simple_bd(struct hns3_tx_queue *txq, struct hns3_desc *desc,
{
#define HNS3_TCP_CSUM_OFFSET 16
#define HNS3_UDP_CSUM_OFFSET 6
+ uint32_t l4_start;
/*
* In HIP09, NIC HW support Tx simple BD mode that the HW will
@@ -4003,9 +4004,13 @@ hns3_handle_simple_bd(struct hns3_tx_queue *txq, struct hns3_desc *desc,
((m->ol_flags & RTE_MBUF_F_TX_L4_MASK) == RTE_MBUF_F_TX_TCP_CKSUM ||
(m->ol_flags & RTE_MBUF_F_TX_L4_MASK) == RTE_MBUF_F_TX_UDP_CKSUM)) {
/* set checksum start and offset, defined in 2 Bytes */
+ l4_start = m->l2_len + m->l3_len;
+ if (m->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK)
+ l4_start += m->outer_l2_len + m->outer_l3_len;
+
hns3_set_field(desc->tx.type_cs_vlan_tso_len,
HNS3_TXD_L4_START_M, HNS3_TXD_L4_START_S,
- (m->l2_len + m->l3_len) >> HNS3_SIMPLE_BD_UNIT);
+ l4_start >> HNS3_SIMPLE_BD_UNIT);
hns3_set_field(desc->tx.ol_type_vlan_len_msec,
HNS3_TXD_L4_CKS_OFFSET_M, HNS3_TXD_L4_CKS_OFFSET_S,
(m->ol_flags & RTE_MBUF_F_TX_L4_MASK) ==
--
2.33.0
^ permalink raw reply related
* [PATCH] net/hns3: fix L4 checksum incorrect for tunnel packets
From: Xingui Yang @ 2026-05-09 6:36 UTC (permalink / raw)
To: dev
Cc: stephen, david.marchand, fengchengwen, lihuisong, liuyonglong,
kangfenglong
In HIP09 simple BD mode, the driver incorrectly calculates L4_START
position for tunnel packets. The current implementation only uses
l2_len + l3_len without considering outer header lengths (outer_l2_len
and outer_l3_len), causing hardware to calculate checksum from wrong
offset.
For a typical NvGRE tunnel packet with structure:
Outer Eth(14) + Outer IPv6(40) + NvGRE(8) + Inner Eth(14) +
Inner IPv6(40) + TCP(20)
Expected L4_START: 116 bytes (from packet start to inner TCP header)
Actual calculation: 62 bytes (missing outer headers)
This results in incorrect TCP/UDP checksum for all tunnel packets when
using simple BD mode, including NvGRE, VxLAN, GRE, GENEVE tunnels.
Fix by adding outer_l2_len and outer_l3_len to L4_START calculation
when RTE_MBUF_F_TX_TUNNEL_MASK flag is set.
Fixes: 6393fc0b823 ("net/hns3: simplify hardware checksum offloading")
Cc: stable@dpdk.org
Signed-off-by: Xingui Yang <yangxingui@huawei.com>
---
drivers/net/hns3/hns3_rxtx.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hns3/hns3_rxtx.c b/drivers/net/hns3/hns3_rxtx.c
index 573604b0cd..4d48cbdc11 100644
--- a/drivers/net/hns3/hns3_rxtx.c
+++ b/drivers/net/hns3/hns3_rxtx.c
@@ -3982,6 +3982,7 @@ hns3_handle_simple_bd(struct hns3_tx_queue *txq, struct hns3_desc *desc,
{
#define HNS3_TCP_CSUM_OFFSET 16
#define HNS3_UDP_CSUM_OFFSET 6
+ uint32_t l4_start;
/*
* In HIP09, NIC HW support Tx simple BD mode that the HW will
@@ -4003,9 +4004,13 @@ hns3_handle_simple_bd(struct hns3_tx_queue *txq, struct hns3_desc *desc,
((m->ol_flags & RTE_MBUF_F_TX_L4_MASK) == RTE_MBUF_F_TX_TCP_CKSUM ||
(m->ol_flags & RTE_MBUF_F_TX_L4_MASK) == RTE_MBUF_F_TX_UDP_CKSUM)) {
/* set checksum start and offset, defined in 2 Bytes */
+ l4_start = m->l2_len + m->l3_len;
+ if (m->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK)
+ l4_start += m->outer_l2_len + m->outer_l3_len;
+
hns3_set_field(desc->tx.type_cs_vlan_tso_len,
HNS3_TXD_L4_START_M, HNS3_TXD_L4_START_S,
- (m->l2_len + m->l3_len) >> HNS3_SIMPLE_BD_UNIT);
+ l4_start >> HNS3_SIMPLE_BD_UNIT);
hns3_set_field(desc->tx.ol_type_vlan_len_msec,
HNS3_TXD_L4_CKS_OFFSET_M, HNS3_TXD_L4_CKS_OFFSET_S,
(m->ol_flags & RTE_MBUF_F_TX_L4_MASK) ==
--
2.33.0
^ permalink raw reply related
* [PATCH v2 1/1] net/mana: add device reset support
From: Wei Hu @ 2026-05-09 6:01 UTC (permalink / raw)
To: dev, stephen; +Cc: longli, weh
In-Reply-To: <20260509060113.61686-1-weh@linux.microsoft.com>
From: Wei Hu <weh@microsoft.com>
Add support for handling hardware reset events in the MANA driver.
When the MANA kernel driver receives a hardware service event, it
initiates a device reset and notifies userspace via
IBV_EVENT_DEVICE_FATAL. The DPDK driver handles this by performing
an automatic teardown and recovery sequence.
The reset flow has two phases. In the enter phase, running on the
EAL interrupt thread, the driver transitions the device state,
waits for data path threads to reach a quiescent state using RCU,
stops queues, tears down IB resources, and frees per-queue MR
caches. A control thread is then spawned to handle the exit phase:
it waits for the hardware to recover, unregisters the interrupt
handler, re-probes the PCI device, reinitializes MR caches, and
restarts queues.
A per-device spinlock serializes the reset path with ethdev
operations. Operations that cannot wait (configure, queue setup)
return -EBUSY during reset, while dev_stop and dev_close join the
reset thread and use a blocking lock to ensure proper sequencing.
Multi-process support is included: secondary processes unmap and
remap doorbell pages via IPC during the reset enter and exit
phases. Data path functions in both primary and secondary
processes check the device state atomically and return early when
the device is not active.
The driver uses ethdev recovery events to notify upper layers
(e.g. netvsc) of the reset lifecycle: RTE_ETH_EVENT_ERR_RECOVERING
on entry, RTE_ETH_EVENT_RECOVERY_SUCCESS or
RTE_ETH_EVENT_RECOVERY_FAILED on completion. A PCI device removal
event callback distinguishes hot-remove from service reset.
Signed-off-by: Wei Hu <weh@microsoft.com>
---
drivers/net/mana/mana.c | 928 +++++++++++++++++++++++++++++++----
drivers/net/mana/mana.h | 33 +-
drivers/net/mana/meson.build | 2 +-
drivers/net/mana/mp.c | 89 +++-
drivers/net/mana/mr.c | 6 +-
drivers/net/mana/rx.c | 24 +-
drivers/net/mana/tx.c | 40 +-
7 files changed, 1016 insertions(+), 106 deletions(-)
diff --git a/drivers/net/mana/mana.c b/drivers/net/mana/mana.c
index 67396cda1f..87ab6f15b1 100644
--- a/drivers/net/mana/mana.c
+++ b/drivers/net/mana/mana.c
@@ -13,7 +13,10 @@
#include <ethdev_pci.h>
#include <rte_kvargs.h>
#include <rte_eal_paging.h>
+#include <rte_alarm.h>
#include <rte_pci.h>
+#include <rte_rcu_qsbr.h>
+#include <rte_lock_annotations.h>
#include <infiniband/verbs.h>
#include <infiniband/manadv.h>
@@ -103,6 +106,23 @@ mana_dev_configure(struct rte_eth_dev *dev)
RTE_ETH_RX_OFFLOAD_VLAN_STRIP);
priv->num_queues = dev->data->nb_rx_queues;
+ DRV_LOG(DEBUG, "priv %p, port %u, dev port %u, num_queues: %u",
+ priv, priv->port_id, priv->dev_port, priv->num_queues);
+
+ /*
+ * Register data path thread IDs (rx and tx) with the RCU
+ * quiescent state variable for device state synchronization.
+ */
+ for (int i = 0; i < 2 * priv->num_queues; i++) {
+ if (rte_rcu_qsbr_thread_register(priv->dev_state_qsv, i) != 0) {
+ DRV_LOG(ERR, "Failed to register rcu qsv thread %d of total %d",
+ i, 2 * priv->num_queues - 1);
+ return -EINVAL;
+ }
+ DRV_LOG(DEBUG,
+ "Register thread 0x%x for priv %p, port %u",
+ i, priv, priv->port_id);
+ }
manadv_set_context_attr(priv->ib_ctx, MANADV_CTX_ATTR_BUF_ALLOCATORS,
(void *)((uintptr_t)&(struct manadv_ctx_allocators){
@@ -214,8 +234,8 @@ mana_dev_start(struct rte_eth_dev *dev)
DRV_LOG(INFO, "TX/RX queues have started");
- /* Enable datapath for secondary processes */
- mana_mp_req_on_rxtx(dev, MANA_MP_REQ_START_RXTX);
+ /* Intentionally ignore errors — secondary may not be running */
+ (void)mana_mp_req_on_rxtx(dev, MANA_MP_REQ_START_RXTX);
ret = rxq_intr_enable(priv);
if (ret) {
@@ -242,26 +262,33 @@ mana_dev_stop(struct rte_eth_dev *dev)
{
int ret;
struct mana_priv *priv = dev->data->dev_private;
-
- rxq_intr_disable(priv);
+ enum mana_device_state state;
+
+ state = rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire);
+ if (state == MANA_DEV_ACTIVE ||
+ state == MANA_DEV_RESET_FAILED) {
+ rxq_intr_disable(priv);
+ DRV_LOG(DEBUG, "rxq_intr_disable called");
+ }
dev->tx_pkt_burst = mana_tx_burst_removed;
dev->rx_pkt_burst = mana_rx_burst_removed;
- /* Stop datapath on secondary processes */
- mana_mp_req_on_rxtx(dev, MANA_MP_REQ_STOP_RXTX);
+ /* Intentionally ignore errors — secondary may not be running */
+ (void)mana_mp_req_on_rxtx(dev, MANA_MP_REQ_STOP_RXTX);
rte_wmb();
ret = mana_stop_tx_queues(dev);
if (ret) {
- DRV_LOG(ERR, "failed to stop tx queues");
+ DRV_LOG(ERR, "failed to stop tx queues, ret %d", ret);
return ret;
}
ret = mana_stop_rx_queues(dev);
if (ret) {
- DRV_LOG(ERR, "failed to stop tx queues");
+ DRV_LOG(ERR, "failed to stop rx queues, ret %d", ret);
return ret;
}
@@ -275,31 +302,58 @@ mana_dev_close(struct rte_eth_dev *dev)
{
struct mana_priv *priv = dev->data->dev_private;
int ret;
+ enum mana_device_state state;
+
+ /* Wait for reset thread to finish before freeing resources */
+ if (priv->reset_thread_active) {
+ pthread_mutex_lock(&priv->reset_cond_mutex);
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_ACTIVE, rte_memory_order_release);
+ pthread_cond_signal(&priv->reset_cond);
+ pthread_mutex_unlock(&priv->reset_cond_mutex);
+ rte_thread_join(priv->reset_thread, NULL);
+ priv->reset_thread_active = false;
+ }
+ DRV_LOG(DEBUG, "Free MR for priv %p", priv);
mana_remove_all_mr(priv);
- ret = mana_intr_uninstall(priv);
- if (ret)
- return ret;
+ state = rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire);
+ if (state == MANA_DEV_ACTIVE ||
+ state == MANA_DEV_RESET_FAILED) {
+ ret = mana_intr_uninstall(priv);
+ if (ret)
+ return ret;
+ }
if (priv->ib_parent_pd) {
- int err = ibv_dealloc_pd(priv->ib_parent_pd);
- if (err)
- DRV_LOG(ERR, "Failed to deallocate parent PD: %d", err);
+ ret = ibv_dealloc_pd(priv->ib_parent_pd);
+ if (ret)
+ DRV_LOG(ERR,
+ "Failed to deallocate parent PD: %d", ret);
priv->ib_parent_pd = NULL;
}
if (priv->ib_pd) {
- int err = ibv_dealloc_pd(priv->ib_pd);
- if (err)
- DRV_LOG(ERR, "Failed to deallocate PD: %d", err);
+ ret = ibv_dealloc_pd(priv->ib_pd);
+ if (ret)
+ DRV_LOG(ERR, "Failed to deallocate PD: %d", ret);
priv->ib_pd = NULL;
}
- ret = ibv_close_device(priv->ib_ctx);
- if (ret) {
- ret = errno;
- return ret;
+ state = rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire);
+ if (state == MANA_DEV_ACTIVE ||
+ state == MANA_DEV_RESET_FAILED) {
+ if (priv->ib_ctx) {
+ ret = ibv_close_device(priv->ib_ctx);
+ if (ret) {
+ ret = errno;
+ return ret;
+ }
+ priv->ib_ctx = NULL;
+ }
}
return 0;
@@ -391,6 +445,27 @@ mana_dev_info_get(struct rte_eth_dev *dev,
return 0;
}
+static int
+mana_dev_info_get_lock(struct rte_eth_dev *dev,
+ struct rte_eth_dev_info *dev_info)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ if (rte_spinlock_trylock(&priv->reset_ops_lock)) {
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return -EBUSY;
+ }
+ ret = mana_dev_info_get(dev, dev_info);
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ } else {
+ ret = -EBUSY;
+ }
+ return ret;
+}
+
static void
mana_dev_tx_queue_info(struct rte_eth_dev *dev, uint16_t queue_id,
struct rte_eth_txq_info *qinfo)
@@ -552,6 +627,29 @@ mana_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,
return ret;
}
+static int
+mana_dev_tx_queue_setup_lock(struct rte_eth_dev *dev, uint16_t queue_idx,
+ uint16_t nb_desc, unsigned int socket_id,
+ const struct rte_eth_txconf *tx_conf)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ if (rte_spinlock_trylock(&priv->reset_ops_lock)) {
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return -EBUSY;
+ }
+ ret = mana_dev_tx_queue_setup(dev, queue_idx,
+ nb_desc, socket_id, tx_conf);
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ } else {
+ ret = -EBUSY;
+ }
+ return ret;
+}
+
static void
mana_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
{
@@ -629,6 +727,30 @@ mana_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,
return ret;
}
+static int
+mana_dev_rx_queue_setup_lock(struct rte_eth_dev *dev, uint16_t queue_idx,
+ uint16_t nb_desc, unsigned int socket_id,
+ const struct rte_eth_rxconf *rx_conf __rte_unused,
+ struct rte_mempool *mp)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ if (rte_spinlock_trylock(&priv->reset_ops_lock)) {
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return -EBUSY;
+ }
+ ret = mana_dev_rx_queue_setup(dev, queue_idx, nb_desc,
+ socket_id, rx_conf, mp);
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ } else {
+ ret = -EBUSY;
+ }
+ return ret;
+}
+
static void
mana_dev_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
{
@@ -820,33 +942,217 @@ mana_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
return mana_ifreq(priv, SIOCSIFMTU, &request);
}
+#define MANA_OPS_1_LOCK(_func) \
+static int \
+_func##_lock(struct rte_eth_dev *dev) \
+{ \
+ struct mana_priv *priv = dev->data->dev_private; \
+ int ret; \
+ if (rte_spinlock_trylock(&priv->reset_ops_lock)) { \
+ if (rte_atomic_load_explicit(&priv->dev_state, \
+ rte_memory_order_acquire) != \
+ MANA_DEV_ACTIVE) { \
+ rte_spinlock_unlock(&priv->reset_ops_lock); \
+ return -EBUSY; \
+ } \
+ ret = _func(dev); \
+ rte_spinlock_unlock(&priv->reset_ops_lock); \
+ } else { \
+ ret = -EBUSY; \
+ } \
+ return ret; \
+}
+
+MANA_OPS_1_LOCK(mana_dev_configure)
+
+MANA_OPS_1_LOCK(mana_dev_start)
+
+#undef MANA_OPS_1_LOCK
+
+/*
+ * Custom lock wrappers for dev_stop and dev_close.
+ * These join any active reset thread and use a blocking lock (not
+ * trylock) so they wait for any in-progress reset processing to
+ * finish, rather than returning -EBUSY. When the device is not in
+ * MANA_DEV_ACTIVE state, they transition state to MANA_DEV_ACTIVE.
+ */
+static int
+mana_dev_stop_lock(struct rte_eth_dev *dev)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ /* Signal reset thread to stop by setting state, then wait for it.
+ * Must be done before acquiring the lock to avoid deadlock
+ * (reset thread also acquires the lock).
+ */
+ if (priv->reset_thread_active) {
+ pthread_mutex_lock(&priv->reset_cond_mutex);
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_ACTIVE, rte_memory_order_release);
+ pthread_cond_signal(&priv->reset_cond);
+ pthread_mutex_unlock(&priv->reset_cond_mutex);
+ rte_thread_join(priv->reset_thread, NULL);
+ priv->reset_thread_active = false;
+ }
+
+ rte_spinlock_lock(&priv->reset_ops_lock);
+
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_ACTIVE, rte_memory_order_release);
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return 0;
+ }
+
+ ret = mana_dev_stop(dev);
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return ret;
+}
+
+static int
+mana_dev_close_lock(struct rte_eth_dev *dev)
+{
+ struct mana_priv *priv = dev->data->dev_private;
+ int ret;
+
+ /* Signal reset thread to stop by setting state, then wait for it.
+ * Must be done before acquiring the lock to avoid deadlock
+ * (reset thread also acquires the lock).
+ */
+ if (priv->reset_thread_active) {
+ pthread_mutex_lock(&priv->reset_cond_mutex);
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_ACTIVE, rte_memory_order_release);
+ pthread_cond_signal(&priv->reset_cond);
+ pthread_mutex_unlock(&priv->reset_cond_mutex);
+ rte_thread_join(priv->reset_thread, NULL);
+ priv->reset_thread_active = false;
+ }
+
+ rte_spinlock_lock(&priv->reset_ops_lock);
+
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_ACTIVE, rte_memory_order_release);
+ }
+
+ ret = mana_dev_close(dev);
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return ret;
+}
+
+#define MANA_OPS_2_LOCK(_func) \
+static int \
+_func##_lock(struct rte_eth_dev *dev, \
+ struct rte_eth_rss_conf *rss_conf) \
+{ \
+ struct mana_priv *priv = dev->data->dev_private; \
+ int ret; \
+ if (rte_spinlock_trylock(&priv->reset_ops_lock)) { \
+ if (rte_atomic_load_explicit(&priv->dev_state, \
+ rte_memory_order_acquire) != \
+ MANA_DEV_ACTIVE) { \
+ rte_spinlock_unlock(&priv->reset_ops_lock); \
+ return -EBUSY; \
+ } \
+ ret = _func(dev, rss_conf); \
+ rte_spinlock_unlock(&priv->reset_ops_lock); \
+ } else { \
+ ret = -EBUSY; \
+ } \
+ return ret; \
+}
+
+MANA_OPS_2_LOCK(mana_rss_hash_update)
+
+MANA_OPS_2_LOCK(mana_rss_hash_conf_get)
+#undef MANA_OPS_2_LOCK
+
+#define MANA_OPS_3_LOCK(_func, _arg) \
+static void \
+_func##_lock(struct rte_eth_dev *dev, uint16_t _arg) \
+{ \
+ struct mana_priv *priv = dev->data->dev_private; \
+ if (rte_spinlock_trylock(&priv->reset_ops_lock)) { \
+ if (rte_atomic_load_explicit(&priv->dev_state, \
+ rte_memory_order_acquire) != \
+ MANA_DEV_ACTIVE) { \
+ rte_spinlock_unlock(&priv->reset_ops_lock); \
+ DRV_LOG(ERR, "Device reset in progress, " \
+ "%s not called", #_func); \
+ return; \
+ } \
+ _func(dev, _arg); \
+ rte_spinlock_unlock(&priv->reset_ops_lock); \
+ } else { \
+ DRV_LOG(ERR, "Device reset in progress, " \
+ "%s not called", #_func); \
+ } \
+}
+
+MANA_OPS_3_LOCK(mana_dev_tx_queue_release, qid)
+
+MANA_OPS_3_LOCK(mana_dev_rx_queue_release, qid)
+#undef MANA_OPS_3_LOCK
+
+#define MANA_OPS_4_LOCK(_func, _arg) \
+static int \
+_func##_lock(struct rte_eth_dev *dev, uint16_t _arg) \
+{ \
+ struct mana_priv *priv = dev->data->dev_private; \
+ int ret; \
+ if (rte_spinlock_trylock(&priv->reset_ops_lock)) { \
+ if (rte_atomic_load_explicit(&priv->dev_state, \
+ rte_memory_order_acquire) != \
+ MANA_DEV_ACTIVE) { \
+ rte_spinlock_unlock(&priv->reset_ops_lock); \
+ return -EBUSY; \
+ } \
+ ret = _func(dev, _arg); \
+ rte_spinlock_unlock(&priv->reset_ops_lock); \
+ } else { \
+ ret = -EBUSY; \
+ } \
+ return ret; \
+}
+
+MANA_OPS_4_LOCK(mana_rx_intr_enable, rx_queue_id)
+
+MANA_OPS_4_LOCK(mana_rx_intr_disable, rx_queue_id)
+
+MANA_OPS_4_LOCK(mana_mtu_set, mtu)
+#undef MANA_OPS_4_LOCK
+
static const struct eth_dev_ops mana_dev_ops = {
- .dev_configure = mana_dev_configure,
- .dev_start = mana_dev_start,
- .dev_stop = mana_dev_stop,
- .dev_close = mana_dev_close,
- .dev_infos_get = mana_dev_info_get,
+ .dev_configure = mana_dev_configure_lock,
+ .dev_start = mana_dev_start_lock,
+ .dev_stop = mana_dev_stop_lock,
+ .dev_close = mana_dev_close_lock,
+ .dev_infos_get = mana_dev_info_get_lock,
.txq_info_get = mana_dev_tx_queue_info,
.rxq_info_get = mana_dev_rx_queue_info,
.dev_supported_ptypes_get = mana_supported_ptypes,
- .rss_hash_update = mana_rss_hash_update,
- .rss_hash_conf_get = mana_rss_hash_conf_get,
- .tx_queue_setup = mana_dev_tx_queue_setup,
- .tx_queue_release = mana_dev_tx_queue_release,
- .rx_queue_setup = mana_dev_rx_queue_setup,
- .rx_queue_release = mana_dev_rx_queue_release,
- .rx_queue_intr_enable = mana_rx_intr_enable,
- .rx_queue_intr_disable = mana_rx_intr_disable,
+ .rss_hash_update = mana_rss_hash_update_lock,
+ .rss_hash_conf_get = mana_rss_hash_conf_get_lock,
+ .tx_queue_setup = mana_dev_tx_queue_setup_lock,
+ .tx_queue_release = mana_dev_tx_queue_release_lock,
+ .rx_queue_setup = mana_dev_rx_queue_setup_lock,
+ .rx_queue_release = mana_dev_rx_queue_release_lock,
+ .rx_queue_intr_enable = mana_rx_intr_enable_lock,
+ .rx_queue_intr_disable = mana_rx_intr_disable_lock,
.link_update = mana_dev_link_update,
.stats_get = mana_dev_stats_get,
.stats_reset = mana_dev_stats_reset,
- .mtu_set = mana_mtu_set,
+ .mtu_set = mana_mtu_set_lock,
};
static const struct eth_dev_ops mana_dev_secondary_ops = {
.stats_get = mana_dev_stats_get,
.stats_reset = mana_dev_stats_reset,
- .dev_infos_get = mana_dev_info_get,
+ .dev_infos_get = mana_dev_info_get_lock,
};
uint16_t
@@ -1031,28 +1337,398 @@ mana_ibv_device_to_pci_addr(const struct ibv_device *device,
return 0;
}
+static int mana_pci_probe(struct rte_pci_driver *pci_drv,
+ struct rte_pci_device *pci_dev);
+static void mana_intr_handler(void *arg);
+static void mana_reset_exit(struct mana_priv *priv);
+
+/* Delay before initiating reset exit after reset enter completes */
+#define MANA_RESET_TIMER_US (15 * 1000000ULL) /* 15 seconds */
+
/*
- * Interrupt handler from IB layer to notify this device is being removed.
+ * Callback for PCI device removal events from EAL.
+ * If the device is in reset (RESET_EXIT state), this means the PCI
+ * device was hot-removed rather than a service reset. Wake the reset
+ * thread via condvar and notify netvsc via RTE_ETH_EVENT_INTR_RMV.
+ */
+static void
+mana_pci_remove_event_cb(const char *device_name,
+ enum rte_dev_event_type event, void *cb_arg)
+ __rte_no_thread_safety_analysis
+{
+ struct mana_priv *priv = cb_arg;
+ struct rte_eth_dev *dev;
+
+ if (event != RTE_DEV_EVENT_REMOVE)
+ return;
+
+ DRV_LOG(INFO, "PCI device %s removed", device_name);
+
+ /* Wake the reset thread immediately */
+ pthread_mutex_lock(&priv->reset_cond_mutex);
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_RESET_FAILED, rte_memory_order_release);
+ pthread_cond_signal(&priv->reset_cond);
+ pthread_mutex_unlock(&priv->reset_cond_mutex);
+
+ rte_spinlock_lock(&priv->reset_ops_lock);
+
+ dev = &rte_eth_devices[priv->port_id];
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_INTR_RMV for port %u",
+ priv->port_id);
+ rte_eth_dev_callback_process(dev,
+ RTE_ETH_EVENT_INTR_RMV, NULL);
+
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+}
+
+/*
+ * Reset thread: sleeps for the reset timer period, then performs
+ * the reset exit sequence. Runs on a control thread so it can call
+ * rte_intr_callback_unregister (which fails from alarm/intr thread).
+ */
+static uint32_t
+mana_reset_thread(void *arg)
+ __rte_no_thread_safety_analysis
+{
+ struct mana_priv *priv = (struct mana_priv *)arg;
+ struct timespec ts;
+
+ DRV_LOG(INFO, "Reset thread started, waiting %us",
+ (unsigned int)(MANA_RESET_TIMER_US / 1000000));
+
+ /* Wait on condvar with timeout — can be woken early by PCI remove */
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_sec += MANA_RESET_TIMER_US / 1000000;
+
+ pthread_mutex_lock(&priv->reset_cond_mutex);
+ pthread_cond_timedwait(&priv->reset_cond, &priv->reset_cond_mutex, &ts);
+ pthread_mutex_unlock(&priv->reset_cond_mutex);
+
+ rte_spinlock_lock(&priv->reset_ops_lock);
+
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_RESET_EXIT) {
+ DRV_LOG(INFO, "Reset thread: dev_state=%d, skipping",
+ (int)rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire));
+ priv->reset_thread_active = false;
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return 0;
+ }
+
+ DRV_LOG(INFO, "Reset thread: initiating reset exit");
+ mana_reset_exit(priv);
+ /* Lock is released by mana_reset_exit_delay at the end of
+ * the reset exit processing. Thread flag is cleared there too.
+ */
+ return 0;
+}
+
+static void
+mana_reset_enter(struct mana_priv *priv)
+ __rte_no_thread_safety_analysis
+{
+ int ret;
+ uint64_t ticket;
+ struct rte_eth_dev *dev = &rte_eth_devices[priv->port_id];
+
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_ENTER,
+ rte_memory_order_release);
+
+ DRV_LOG(DEBUG, "Entering into device reset state");
+ DRV_LOG(DEBUG, "Resetting dev = %p, priv = %p", dev, priv);
+
+ ticket = rte_rcu_qsbr_start(priv->dev_state_qsv);
+
+ while (rte_rcu_qsbr_check(priv->dev_state_qsv, ticket, false) == 0)
+ rte_pause();
+
+ DRV_LOG(DEBUG, "All threads are quiescent");
+
+ /* Stop data path on primary and secondary before unmapping doorbell */
+ ret = mana_dev_stop(dev);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to stop mana dev ret %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto reset_failed;
+ }
+
+ /* Unmap secondary doorbell pages after data path is stopped */
+ ret = mana_mp_req_on_rxtx(dev, MANA_MP_REQ_RESET_ENTER);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to reset secondary processes ret = %d",
+ ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto reset_failed;
+ }
+
+ ret = mana_dev_close(dev);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to close mana dev ret %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto reset_failed;
+ }
+
+ for (int i = 0; i < priv->num_queues; i++) {
+ struct mana_rxq *rxq = dev->data->rx_queues[i];
+ struct mana_txq *txq = dev->data->tx_queues[i];
+
+ DRV_LOG(DEBUG, "Free MR for priv = %p, rxq %u, txq %u",
+ priv, rxq->rxq_idx, txq->txq_idx);
+ mana_mr_btree_free(&rxq->mr_btree);
+ mana_mr_btree_free(&txq->mr_btree);
+ }
+
+ DRV_LOG(DEBUG, "Reset processing exited successfully");
+
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_EXIT,
+ rte_memory_order_release);
+
+ ret = rte_thread_create_internal_control(&priv->reset_thread,
+ "mana-reset",
+ mana_reset_thread, priv);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to create reset thread ret %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state,
+ MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return;
+ }
+ priv->reset_thread_active = true;
+
+ DRV_LOG(DEBUG, "Reset thread started");
+
+ /* Release the lock so the application can call dev_stop/dev_close */
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return;
+
+reset_failed:
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+}
+
+static uint32_t
+mana_reset_exit_delay(void *arg)
+ __rte_no_thread_safety_analysis
+{
+ struct mana_priv *priv = (struct mana_priv *)arg;
+ uint32_t ret = 0;
+ int i;
+ struct rte_eth_dev *dev;
+ struct rte_pci_device *pci_dev;
+
+ DRV_LOG(DEBUG, "Delayed mana device reset complete processing");
+
+ /* If the app called dev_stop/dev_close during the timer window,
+ * state is no longer RESET_EXIT. Nothing to do.
+ */
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_RESET_EXIT) {
+ DRV_LOG(DEBUG, "State is not RESET_EXIT, skipping");
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return ret;
+ }
+
+ dev = &rte_eth_devices[priv->port_id];
+ pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+
+ DRV_LOG(DEBUG, "Resetting dev = %p, priv = %p", dev, priv);
+
+ ret = ibv_close_device(priv->ib_ctx);
+ priv->ib_ctx = NULL;
+ if (ret) {
+ DRV_LOG(ERR, "Failed to close ibv device %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto out;
+ }
+
+ ret = mana_pci_probe(NULL, pci_dev);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to probe mana pci dev ret %d", ret);
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+ goto out;
+ }
+
+ /*
+ * Init the local MR caches.
+ */
+ for (i = 0; i < priv->num_queues; i++) {
+ struct mana_rxq *rxq = dev->data->rx_queues[i];
+ struct mana_txq *txq = dev->data->tx_queues[i];
+
+ ret = mana_mr_btree_init(&rxq->mr_btree,
+ MANA_MR_BTREE_PER_QUEUE_N,
+ rxq->socket);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to init RXQ %d MR btree "
+ "on socket %u, ret %d", i, rxq->socket, ret);
+ goto mr_init_failed_rxq;
+ }
+
+ ret = mana_mr_btree_init(&txq->mr_btree,
+ MANA_MR_BTREE_PER_QUEUE_N,
+ txq->socket);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to init TXQ %d MR btree "
+ "on socket %u, ret %d", i, txq->socket, ret);
+ goto mr_init_failed_txq;
+ }
+ }
+ DRV_LOG(DEBUG, "priv %p, num_queues %u", priv, priv->num_queues);
+
+ /* Start secondaries */
+ ret = mana_mp_req_on_rxtx(dev, MANA_MP_REQ_RESET_EXIT);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to start secondary processes ret = %d",
+ ret);
+ goto mr_init_failed_all;
+ }
+
+ ret = mana_dev_start(dev);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to start mana dev ret %d", ret);
+ goto mr_init_failed_all;
+ }
+
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_ACTIVE,
+ rte_memory_order_release);
+
+ DRV_LOG(DEBUG, "Exiting the reset complete processing");
+
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_RECOVERY_SUCCESS for port %u",
+ priv->port_id);
+ rte_eth_dev_callback_process(dev,
+ RTE_ETH_EVENT_RECOVERY_SUCCESS, NULL);
+
+out:
+ if (ret) {
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_RECOVERY_FAILED for port %u",
+ priv->port_id);
+ rte_eth_dev_callback_process(dev,
+ RTE_ETH_EVENT_RECOVERY_FAILED, NULL);
+ }
+ priv->reset_thread_active = false;
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return ret;
+
+mr_init_failed_all:
+ i = priv->num_queues;
+ goto mr_init_failed_rxq;
+
+mr_init_failed_txq:
+ /* RXQ btree at index i was initialized, free it */
+ mana_mr_btree_free(&((struct mana_rxq *)
+ dev->data->rx_queues[i])->mr_btree);
+
+mr_init_failed_rxq:
+ /* Free all fully initialized btrees for indices < i */
+ for (int j = 0; j < i; j++) {
+ struct mana_rxq *rxq = dev->data->rx_queues[j];
+ struct mana_txq *txq = dev->data->tx_queues[j];
+
+ mana_mr_btree_free(&rxq->mr_btree);
+ mana_mr_btree_free(&txq->mr_btree);
+ }
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_RESET_FAILED,
+ rte_memory_order_release);
+
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_RECOVERY_FAILED (MR init) for port %u",
+ priv->port_id);
+ rte_eth_dev_callback_process(dev,
+ RTE_ETH_EVENT_RECOVERY_FAILED, NULL);
+
+ priv->reset_thread_active = false;
+ rte_spinlock_unlock(&priv->reset_ops_lock);
+ return ret;
+}
+
+static void
+mana_reset_exit(struct mana_priv *priv)
+ __rte_no_thread_safety_analysis
+{
+ int ret;
+
+ if (!priv) {
+ DRV_LOG(ERR, "Private structure invalid");
+ return;
+ }
+ DRV_LOG(DEBUG, "Entering into device reset complete processing");
+
+ rxq_intr_disable(priv);
+
+ /* Unregister the interrupt handler. Since mana_reset_exit is always
+ * called from mana_reset_thread (a non-interrupt thread), the
+ * interrupt source is inactive and rte_intr_callback_unregister
+ * succeeds directly.
+ */
+ if (priv->intr_handle) {
+ ret = rte_intr_callback_unregister(priv->intr_handle,
+ mana_intr_handler, priv);
+ if (ret < 0)
+ DRV_LOG(ERR, "Failed to unregister intr callback ret %d",
+ ret);
+ else
+ DRV_LOG(DEBUG, "%d intr callback(s) removed", ret);
+
+ rte_intr_instance_free(priv->intr_handle);
+ priv->intr_handle = NULL;
+ }
+
+ /* Proceed directly to reset exit delay (re-probe and restart).
+ * No need for a separate thread - we are already on
+ * mana_reset_thread which is a non-interrupt control thread.
+ */
+ mana_reset_exit_delay(priv);
+}
+
+/*
+ * Interrupt handler from IB layer to notify this device is
+ * being removed or reset.
*/
static void
mana_intr_handler(void *arg)
+ __rte_no_thread_safety_analysis
{
struct mana_priv *priv = arg;
struct ibv_context *ctx = priv->ib_ctx;
- struct ibv_async_event event;
+ struct ibv_async_event event = { 0 };
+ struct rte_eth_dev *dev;
/* Read and ack all messages from IB device */
while (true) {
if (ibv_get_async_event(ctx, &event))
break;
- if (event.event_type == IBV_EVENT_DEVICE_FATAL) {
- struct rte_eth_dev *dev;
-
- dev = &rte_eth_devices[priv->port_id];
- if (dev->data->dev_conf.intr_conf.rmv)
+ switch (event.event_type) {
+ case IBV_EVENT_DEVICE_FATAL:
+ DRV_LOG(INFO, "IBV_EVENT_DEVICE_FATAL received, dev_state=%d",
+ (int)rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire));
+ if (rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) == MANA_DEV_ACTIVE) {
+ rte_spinlock_lock(&priv->reset_ops_lock);
+ mana_reset_enter(priv);
+
+ dev = &rte_eth_devices[priv->port_id];
+ DRV_LOG(INFO, "Sending RTE_ETH_EVENT_ERR_RECOVERING for port %u",
+ priv->port_id);
rte_eth_dev_callback_process(dev,
- RTE_ETH_EVENT_INTR_RMV, NULL);
+ RTE_ETH_EVENT_ERR_RECOVERING, NULL);
+ } else {
+ DRV_LOG(ERR, "Already in reset handling, dev_state=%d",
+ (int)rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire));
+ }
+ break;
+
+ default:
+ break;
}
ibv_ack_async_event(&event);
@@ -1063,6 +1739,17 @@ static int
mana_intr_uninstall(struct mana_priv *priv)
{
int ret;
+ struct rte_eth_dev *dev;
+
+ if (!priv->intr_handle)
+ return 0;
+
+ /* Unregister PCI device removal event callback */
+ dev = &rte_eth_devices[priv->port_id];
+ if (dev->device)
+ rte_dev_event_callback_unregister(dev->device->name,
+ mana_pci_remove_event_cb,
+ priv);
ret = rte_intr_callback_unregister(priv->intr_handle,
mana_intr_handler, priv);
@@ -1072,6 +1759,7 @@ mana_intr_uninstall(struct mana_priv *priv)
}
rte_intr_instance_free(priv->intr_handle);
+ priv->intr_handle = NULL;
return 0;
}
@@ -1127,6 +1815,14 @@ mana_intr_install(struct rte_eth_dev *eth_dev, struct mana_priv *priv)
goto free_intr;
}
+ /* Register for PCI device removal events to distinguish
+ * PCI hot-remove from service reset.
+ */
+ ret = rte_dev_event_callback_register(eth_dev->device->name,
+ mana_pci_remove_event_cb, priv);
+ if (ret)
+ DRV_LOG(WARNING, "Failed to register PCI remove event callback");
+
eth_dev->intr_handle = priv->intr_handle;
return 0;
@@ -1156,7 +1852,7 @@ mana_proc_priv_init(struct rte_eth_dev *dev)
/*
* Map the doorbell page for the secondary process through IB device handle.
*/
-static int
+int
mana_map_doorbell_secondary(struct rte_eth_dev *eth_dev, int fd)
{
struct mana_process_priv *priv = eth_dev->process_private;
@@ -1294,17 +1990,28 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
char name[RTE_ETH_NAME_MAX_LEN];
int ret;
struct ibv_context *ctx = NULL;
+ size_t sz;
+ bool is_reset = false;
rte_ether_format_addr(address, sizeof(address), addr);
- DRV_LOG(INFO, "device located port %u address %s", port, address);
- priv = rte_zmalloc_socket(NULL, sizeof(*priv), RTE_CACHE_LINE_SIZE,
- SOCKET_ID_ANY);
- if (!priv)
- return -ENOMEM;
+ DRV_LOG(DEBUG, "device located port %u address %s", port, address);
snprintf(name, sizeof(name), "%s_port%d", pci_dev->device.name, port);
+ eth_dev = rte_eth_dev_allocated(name);
+ if (eth_dev) {
+ is_reset = true;
+ priv = eth_dev->data->dev_private;
+ DRV_LOG(DEBUG, "Device reset for eth_dev %p priv %p",
+ eth_dev, priv);
+ } else {
+ priv = rte_zmalloc_socket(NULL, sizeof(*priv), RTE_CACHE_LINE_SIZE,
+ SOCKET_ID_ANY);
+ if (!priv)
+ return -ENOMEM;
+ }
+
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
int fd;
@@ -1317,6 +2024,7 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
eth_dev->device = &pci_dev->device;
eth_dev->dev_ops = &mana_dev_secondary_ops;
+
ret = mana_proc_priv_init(eth_dev);
if (ret)
goto failed;
@@ -1336,7 +2044,7 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
goto failed;
}
- /* fd is no not used after mapping doorbell */
+ /* fd is not used after mapping doorbell */
close(fd);
eth_dev->tx_pkt_burst = mana_tx_burst;
@@ -1355,22 +2063,6 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
goto failed;
}
- eth_dev = rte_eth_dev_allocate(name);
- if (!eth_dev) {
- ret = -ENOMEM;
- goto failed;
- }
-
- eth_dev->data->mac_addrs =
- rte_calloc("mana_mac", 1,
- sizeof(struct rte_ether_addr), 0);
- if (!eth_dev->data->mac_addrs) {
- ret = -ENOMEM;
- goto failed;
- }
-
- rte_ether_addr_copy(addr, eth_dev->data->mac_addrs);
-
priv->ib_pd = ibv_alloc_pd(ctx);
if (!priv->ib_pd) {
DRV_LOG(ERR, "ibv_alloc_pd failed port %d", port);
@@ -1390,10 +2082,6 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
}
priv->ib_ctx = ctx;
- priv->port_id = eth_dev->data->port_id;
- priv->dev_port = port;
- eth_dev->data->dev_private = priv;
- priv->dev_data = eth_dev->data;
priv->max_rx_queues = dev_attr->orig_attr.max_qp;
priv->max_tx_queues = dev_attr->orig_attr.max_qp;
@@ -1415,23 +2103,85 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
name, priv->max_rx_queues, priv->max_rx_desc,
priv->max_send_sge, priv->max_mr_size);
+ if (!is_reset) {
+ eth_dev = rte_eth_dev_allocate(name);
+ if (!eth_dev) {
+ ret = -ENOMEM;
+ goto failed;
+ }
+
+ eth_dev->data->mac_addrs =
+ rte_calloc("mana_mac", 1,
+ sizeof(struct rte_ether_addr), 0);
+ if (!eth_dev->data->mac_addrs) {
+ ret = -ENOMEM;
+ goto failed;
+ }
+
+ rte_ether_addr_copy(addr, eth_dev->data->mac_addrs);
+ } else {
+ /*
+ * Reset path.
+ */
+ rte_ether_format_addr(address, RTE_ETHER_ADDR_FMT_SIZE,
+ eth_dev->data->mac_addrs);
+ DRV_LOG(DEBUG, "Found existing eth_dev %p with mac addr %s",
+ eth_dev, address);
+ DRV_LOG(DEBUG, "ib_ctx = %p", priv->ib_ctx);
+ goto out;
+ }
+
+ priv->port_id = eth_dev->data->port_id;
+ priv->dev_port = port;
+ eth_dev->data->dev_private = priv;
+ priv->dev_data = eth_dev->data;
+ rte_atomic_store_explicit(&priv->dev_state, MANA_DEV_ACTIVE,
+ rte_memory_order_release);
+
rte_eth_copy_pci_info(eth_dev, pci_dev);
- /* Create async interrupt handler */
- ret = mana_intr_install(eth_dev, priv);
- if (ret) {
- DRV_LOG(ERR, "Failed to install intr handler");
+ /*
+ * Now we've got maximum queues. Init the qsv to be the
+ * double of maximum queues for both rx and tx queues.
+ */
+ sz = rte_rcu_qsbr_get_memsize(2 * priv->max_rx_queues);
+ priv->dev_state_qsv = rte_zmalloc_socket("mana_rcu", sz,
+ RTE_CACHE_LINE_SIZE,
+ SOCKET_ID_ANY);
+ if (!priv->dev_state_qsv) {
+ DRV_LOG(ERR, "No memory for dev_state_qsv");
+ ret = -ENOMEM;
+ goto failed;
+ }
+ ret = rte_rcu_qsbr_init(priv->dev_state_qsv, 2 * priv->max_rx_queues);
+ if (ret < 0) {
+ DRV_LOG(ERR, "Init dev_state_qsv failed ret %d", ret);
goto failed;
}
- eth_dev->device = &pci_dev->device;
+ rte_spinlock_init(&priv->reset_ops_lock);
+ pthread_mutex_init(&priv->reset_cond_mutex, NULL);
+ pthread_cond_init(&priv->reset_cond, NULL);
- DRV_LOG(INFO, "device %s at port %u", name, eth_dev->data->port_id);
+ eth_dev->device = &pci_dev->device;
eth_dev->rx_pkt_burst = mana_rx_burst_removed;
eth_dev->tx_pkt_burst = mana_tx_burst_removed;
eth_dev->dev_ops = &mana_dev_ops;
+out:
+ /* Create async interrupt handler */
+ ret = mana_intr_install(eth_dev, priv);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to install intr handler, ret %d", ret);
+ goto failed;
+ } else {
+ DRV_LOG(INFO, "mana_intr_install succeeded");
+ }
+
+ DRV_LOG(INFO, "device %s priv %p dev port %d at port %u",
+ name, priv, priv->dev_port, eth_dev->data->port_id);
+
rte_eth_dev_probing_finish(eth_dev);
return 0;
@@ -1439,20 +2189,32 @@ mana_probe_port(struct ibv_device *ibdev, struct ibv_device_attr_ex *dev_attr,
failed:
/* Free the resource for the port failed */
if (priv) {
- if (priv->ib_parent_pd)
+ if (!is_reset && priv->dev_state_qsv)
+ rte_free(priv->dev_state_qsv);
+
+ if (priv->ib_parent_pd) {
ibv_dealloc_pd(priv->ib_parent_pd);
+ priv->ib_parent_pd = NULL;
+ }
- if (priv->ib_pd)
+ if (priv->ib_pd) {
ibv_dealloc_pd(priv->ib_pd);
+ priv->ib_pd = NULL;
+ }
}
- if (eth_dev)
- rte_eth_dev_release_port(eth_dev);
+ if (!is_reset) {
+ if (eth_dev)
+ rte_eth_dev_release_port(eth_dev);
- rte_free(priv);
+ rte_free(priv);
+ }
- if (ctx)
+ if (ctx) {
ibv_close_device(ctx);
+ if (is_reset && priv)
+ priv->ib_ctx = NULL;
+ }
return ret;
}
diff --git a/drivers/net/mana/mana.h b/drivers/net/mana/mana.h
index 79cc47b6ab..7f8f1ff638 100644
--- a/drivers/net/mana/mana.h
+++ b/drivers/net/mana/mana.h
@@ -5,6 +5,8 @@
#ifndef __MANA_H__
#define __MANA_H__
+#include <pthread.h>
+
#define PCI_VENDOR_ID_MICROSOFT 0x1414
#define PCI_DEVICE_ID_MICROSOFT_MANA_PF 0x00b9
#define PCI_DEVICE_ID_MICROSOFT_MANA 0x00ba
@@ -337,6 +339,20 @@ struct mana_process_priv {
void *db_page;
};
+enum mana_device_state {
+ /* Normal running */
+ MANA_DEV_ACTIVE = 0,
+ /* In reset enter processing */
+ MANA_DEV_RESET_ENTER = 1,
+ /*
+ * Reset enter processing completed.
+ * Waiting for reset exit or in reset exit processing.
+ */
+ MANA_DEV_RESET_EXIT = 2,
+ /* Reset failed */
+ MANA_DEV_RESET_FAILED = 3,
+};
+
struct mana_priv {
struct rte_eth_dev_data *dev_data;
struct mana_process_priv *process_priv;
@@ -368,6 +384,16 @@ struct mana_priv {
uint64_t max_mr_size;
struct mana_mr_btree mr_btree;
rte_spinlock_t mr_btree_lock;
+ RTE_ATOMIC(enum mana_device_state) dev_state;
+ struct rte_rcu_qsbr *dev_state_qsv;
+ /* lock for synchronizing mana reset and some mana_dev_ops callbacks */
+ rte_spinlock_t reset_ops_lock;
+ /* Reset thread ID, valid when reset_thread_active is true */
+ rte_thread_t reset_thread;
+ bool reset_thread_active;
+ /* Condvar to wake reset thread early on PCI remove */
+ pthread_mutex_t reset_cond_mutex;
+ pthread_cond_t reset_cond;
};
struct mana_txq_desc {
@@ -427,6 +453,7 @@ struct mana_txq {
struct mana_mr_btree mr_btree;
struct mana_stats stats;
unsigned int socket;
+ unsigned int txq_idx;
};
struct mana_rxq {
@@ -462,6 +489,7 @@ struct mana_rxq {
struct mana_mr_btree mr_btree;
unsigned int socket;
+ unsigned int rxq_idx;
};
extern int mana_logtype_driver;
@@ -543,6 +571,8 @@ enum mana_mp_req_type {
MANA_MP_REQ_CREATE_MR,
MANA_MP_REQ_START_RXTX,
MANA_MP_REQ_STOP_RXTX,
+ MANA_MP_REQ_RESET_ENTER,
+ MANA_MP_REQ_RESET_EXIT,
};
/* Pameters for IPC. */
@@ -563,8 +593,9 @@ void mana_mp_uninit_primary(void);
void mana_mp_uninit_secondary(void);
int mana_mp_req_verbs_cmd_fd(struct rte_eth_dev *dev);
int mana_mp_req_mr_create(struct mana_priv *priv, uintptr_t addr, uint32_t len);
+int mana_map_doorbell_secondary(struct rte_eth_dev *eth_dev, int fd);
-void mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type);
+int mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type);
void *mana_alloc_verbs_buf(size_t size, void *data);
void mana_free_verbs_buf(void *ptr, void *data __rte_unused);
diff --git a/drivers/net/mana/meson.build b/drivers/net/mana/meson.build
index 19d4b3695e..5b01d9f57e 100644
--- a/drivers/net/mana/meson.build
+++ b/drivers/net/mana/meson.build
@@ -7,7 +7,7 @@ if not is_linux or not (dpdk_conf.has('RTE_ARCH_X86') or dpdk_conf.has('RTE_ARCH
subdir_done()
endif
-deps += ['pci', 'bus_pci', 'net', 'eal', 'kvargs']
+deps += ['pci', 'bus_pci', 'net', 'eal', 'kvargs', 'rcu']
sources += files(
'gdma.c',
diff --git a/drivers/net/mana/mp.c b/drivers/net/mana/mp.c
index 72417fc0c7..c06641f7d9 100644
--- a/drivers/net/mana/mp.c
+++ b/drivers/net/mana/mp.c
@@ -2,10 +2,13 @@
* Copyright 2022 Microsoft Corporation
*/
+#include <sys/mman.h>
#include <rte_malloc.h>
#include <ethdev_driver.h>
#include <rte_log.h>
+#include <rte_eal_paging.h>
#include <stdlib.h>
+#include <unistd.h>
#include <infiniband/verbs.h>
@@ -119,6 +122,23 @@ mana_mp_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
return ret;
}
+static int
+mana_mp_reset_enter(struct rte_eth_dev *dev)
+{
+ struct mana_process_priv *proc_priv = dev->process_private;
+
+ void *addr = proc_priv->db_page;
+
+ /* Reset the db_page to NULL */
+ proc_priv->db_page = (void *)0;
+
+ if (addr)
+ (void)munmap(addr, rte_mem_page_size());
+
+ DRV_LOG(DEBUG, "All secondary threads are quiescent");
+ return 0;
+}
+
static int
mana_mp_secondary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
{
@@ -171,6 +191,49 @@ mana_mp_secondary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
ret = rte_mp_reply(&mp_res, peer);
break;
+ case MANA_MP_REQ_RESET_ENTER:
+ DRV_LOG(INFO, "Port %u reset enter", dev->data->port_id);
+ res->result = mana_mp_reset_enter(dev);
+
+ ret = rte_mp_reply(&mp_res, peer);
+ break;
+
+ case MANA_MP_REQ_RESET_EXIT:
+ DRV_LOG(INFO, "Port %u reset exit", dev->data->port_id);
+ {
+ struct mana_process_priv *proc_priv =
+ dev->process_private;
+
+ if (proc_priv->db_page != 0) {
+ DRV_LOG(DEBUG,
+ "Secondary doorbell already "
+ "mapped to %p",
+ proc_priv->db_page);
+ res->result = 0;
+ } else if (mp_msg->num_fds < 1) {
+ DRV_LOG(ERR,
+ "No FD in RESET_EXIT message");
+ res->result = -EINVAL;
+ } else {
+ int fd = mp_msg->fds[0];
+
+ ret = mana_map_doorbell_secondary(dev,
+ fd);
+ if (ret) {
+ DRV_LOG(ERR,
+ "Failed secondary "
+ "doorbell map %d",
+ fd);
+ res->result = -ENODEV;
+ } else {
+ res->result = 0;
+ }
+ close(fd);
+ }
+ }
+ ret = rte_mp_reply(&mp_res, peer);
+ break;
+
default:
DRV_LOG(ERR, "Port %u unknown secondary MP type %u",
param->port_id, param->type);
@@ -254,7 +317,7 @@ mana_mp_req_verbs_cmd_fd(struct rte_eth_dev *dev)
}
ret = mp_res->fds[0];
- DRV_LOG(ERR, "port %u command FD from primary is %d",
+ DRV_LOG(DEBUG, "port %u command FD from primary is %d",
dev->data->port_id, ret);
exit:
free(mp_rep.msgs);
@@ -298,27 +361,36 @@ mana_mp_req_mr_create(struct mana_priv *priv, uintptr_t addr, uint32_t len)
return ret;
}
-void
+int
mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type)
{
struct rte_mp_msg mp_req = { 0 };
struct rte_mp_msg *mp_res;
- struct rte_mp_reply mp_rep;
+ struct rte_mp_reply mp_rep = { 0 };
struct mana_mp_param *res;
struct timespec ts = {.tv_sec = MANA_MP_REQ_TIMEOUT_SEC, .tv_nsec = 0};
- int i, ret;
+ int i, ret = 0;
- if (type != MANA_MP_REQ_START_RXTX && type != MANA_MP_REQ_STOP_RXTX) {
+ if (type != MANA_MP_REQ_START_RXTX && type != MANA_MP_REQ_STOP_RXTX &&
+ type != MANA_MP_REQ_RESET_ENTER && type != MANA_MP_REQ_RESET_EXIT) {
DRV_LOG(ERR, "port %u unknown request (req_type %d)",
dev->data->port_id, type);
- return;
+ return -EINVAL;
}
if (rte_atomic_load_explicit(&mana_shared_data->secondary_cnt, rte_memory_order_relaxed) == 0)
- return;
+ return 0;
mp_init_msg(&mp_req, type, dev->data->port_id);
+ /* Include IB cmd FD for secondary doorbell remap */
+ if (type == MANA_MP_REQ_RESET_EXIT) {
+ struct mana_priv *priv = dev->data->dev_private;
+
+ mp_req.num_fds = 1;
+ mp_req.fds[0] = priv->ib_ctx->cmd_fd;
+ }
+
ret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);
if (ret) {
if (rte_errno != ENOTSUP)
@@ -329,6 +401,7 @@ mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type)
if (mp_rep.nb_sent != mp_rep.nb_received) {
DRV_LOG(ERR, "port %u not all secondaries responded (%d)",
dev->data->port_id, type);
+ ret = -ETIMEDOUT;
goto exit;
}
for (i = 0; i < mp_rep.nb_received; i++) {
@@ -337,9 +410,11 @@ mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type)
if (res->result) {
DRV_LOG(ERR, "port %u request failed on secondary %d",
dev->data->port_id, i);
+ ret = res->result;
goto exit;
}
}
exit:
free(mp_rep.msgs);
+ return ret;
}
diff --git a/drivers/net/mana/mr.c b/drivers/net/mana/mr.c
index c4045141bc..8914f4cf04 100644
--- a/drivers/net/mana/mr.c
+++ b/drivers/net/mana/mr.c
@@ -314,8 +314,10 @@ mana_mr_btree_init(struct mana_mr_btree *bt, int n, int socket)
void
mana_mr_btree_free(struct mana_mr_btree *bt)
{
- rte_free(bt->table);
- memset(bt, 0, sizeof(*bt));
+ if (bt && bt->table) {
+ rte_free(bt->table);
+ memset(bt, 0, sizeof(*bt));
+ }
}
int
diff --git a/drivers/net/mana/rx.c b/drivers/net/mana/rx.c
index 1b8ba1f3a9..ae05d8dd2f 100644
--- a/drivers/net/mana/rx.c
+++ b/drivers/net/mana/rx.c
@@ -2,6 +2,7 @@
* Copyright 2022 Microsoft Corporation
*/
#include <ethdev_driver.h>
+#include <rte_rcu_qsbr.h>
#include <infiniband/verbs.h>
#include <infiniband/manadv.h>
@@ -36,6 +37,11 @@ mana_rq_ring_doorbell(struct mana_rxq *rxq)
db_page = process_priv->db_page;
}
+ if (!db_page) {
+ DP_LOG(ERR, "db_page is NULL, cannot ring RX doorbell");
+ return -EINVAL;
+ }
+
/* Hardware Spec specifies that software client should set 0 for
* wqe_cnt for Receive Queues.
*/
@@ -172,7 +178,7 @@ mana_stop_rx_queues(struct rte_eth_dev *dev)
for (i = 0; i < priv->num_queues; i++)
if (dev->data->rx_queue_state[i] == RTE_ETH_QUEUE_STATE_STOPPED)
- return -EINVAL;
+ return 0;
if (priv->rwq_qp) {
ret = ibv_destroy_qp(priv->rwq_qp);
@@ -256,6 +262,9 @@ mana_start_rx_queues(struct rte_eth_dev *dev)
struct mana_rxq *rxq = dev->data->rx_queues[i];
struct ibv_wq_init_attr wq_attr = {};
+ rxq->rxq_idx = i;
+ DRV_LOG(DEBUG, "assigning rxq_idx to %d", i);
+
manadv_set_context_attr(priv->ib_ctx,
MANADV_CTX_ATTR_BUF_ALLOCATORS,
(void *)((uintptr_t)&(struct manadv_ctx_allocators){
@@ -451,6 +460,17 @@ mana_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
uint32_t pkt_len;
uint32_t i;
int polled = 0;
+ struct rte_rcu_qsbr *dstate_qsv = priv->dev_state_qsv;
+ unsigned int tid = rxq->rxq_idx;
+
+ rte_rcu_qsbr_thread_online(dstate_qsv, tid);
+
+ if (unlikely(rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE)) {
+ /* Device reset occurred. */
+ rte_rcu_qsbr_thread_offline(dstate_qsv, tid);
+ return 0;
+ }
repoll:
/* Polling on new completions if we have no backlog */
@@ -592,6 +612,8 @@ mana_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)
wqe_consumed, ret);
}
+ rte_rcu_qsbr_thread_offline(dstate_qsv, tid);
+
return pkt_received;
}
diff --git a/drivers/net/mana/tx.c b/drivers/net/mana/tx.c
index 57dbbc3651..3b07a8b9a6 100644
--- a/drivers/net/mana/tx.c
+++ b/drivers/net/mana/tx.c
@@ -3,6 +3,7 @@
*/
#include <ethdev_driver.h>
+#include <rte_rcu_qsbr.h>
#include <infiniband/verbs.h>
#include <infiniband/manadv.h>
@@ -17,7 +18,7 @@ mana_stop_tx_queues(struct rte_eth_dev *dev)
for (i = 0; i < priv->num_queues; i++)
if (dev->data->tx_queue_state[i] == RTE_ETH_QUEUE_STATE_STOPPED)
- return -EINVAL;
+ return 0;
for (i = 0; i < priv->num_queues; i++) {
struct mana_txq *txq = dev->data->tx_queues[i];
@@ -83,6 +84,9 @@ mana_start_tx_queues(struct rte_eth_dev *dev)
txq = dev->data->tx_queues[i];
+ txq->txq_idx = i;
+ DRV_LOG(DEBUG, "assigning txq_idx to %d", txq->txq_idx);
+
manadv_set_context_attr(priv->ib_ctx,
MANADV_CTX_ATTR_BUF_ALLOCATORS,
(void *)((uintptr_t)&(struct manadv_ctx_allocators){
@@ -190,10 +194,30 @@ mana_tx_burst(void *dpdk_txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
void *db_page;
uint16_t pkt_sent = 0;
uint32_t num_comp, i;
+ unsigned int tid = priv->num_queues + txq->txq_idx;
+ struct rte_rcu_qsbr *dstate_qsv = priv->dev_state_qsv;
#ifdef RTE_ARCH_32
uint32_t wqe_count = 0;
#endif
+ db_page = priv->db_page;
+ if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+ struct rte_eth_dev *dev =
+ &rte_eth_devices[priv->dev_data->port_id];
+ struct mana_process_priv *process_priv = dev->process_private;
+
+ db_page = process_priv->db_page;
+ }
+
+ rte_rcu_qsbr_thread_online(dstate_qsv, tid);
+
+ if (unlikely(rte_atomic_load_explicit(&priv->dev_state,
+ rte_memory_order_acquire) != MANA_DEV_ACTIVE || !db_page)) {
+ /* Device reset event occurred. */
+ rte_rcu_qsbr_thread_offline(dstate_qsv, tid);
+ return 0;
+ }
+
/* Process send completions from GDMA */
num_comp = gdma_poll_completion_queue(&txq->gdma_cq,
txq->gdma_comp_buf, txq->num_desc);
@@ -216,7 +240,8 @@ mana_tx_burst(void *dpdk_txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
}
if (!desc->pkt) {
- DP_LOG(ERR, "mana_txq_desc has a NULL pkt");
+ DP_LOG(ERR, "mana_txq_desc has a NULL pkt, priv %p, "
+ "txq = %d", priv, txq->txq_idx);
} else {
txq->stats.bytes += desc->pkt->pkt_len;
rte_pktmbuf_free(desc->pkt);
@@ -474,15 +499,6 @@ mana_tx_burst(void *dpdk_txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
}
/* Ring hardware door bell */
- db_page = priv->db_page;
- if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
- struct rte_eth_dev *dev =
- &rte_eth_devices[priv->dev_data->port_id];
- struct mana_process_priv *process_priv = dev->process_private;
-
- db_page = process_priv->db_page;
- }
-
if (pkt_sent) {
#ifdef RTE_ARCH_32
ret = mana_ring_short_doorbell(db_page, GDMA_QUEUE_SEND,
@@ -501,5 +517,7 @@ mana_tx_burst(void *dpdk_txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
DP_LOG(ERR, "mana_ring_doorbell failed ret %d", ret);
}
+ rte_rcu_qsbr_thread_offline(dstate_qsv, tid);
+
return pkt_sent;
}
--
2.34.1
^ permalink raw reply related
* [PATCH v2 0/1] net/mana: add device reset support
From: Wei Hu @ 2026-05-09 6:01 UTC (permalink / raw)
To: dev, stephen; +Cc: longli, weh
From: Wei Hu <weh@microsoft.com>
Add support for handling hardware service reset events in the
MANA driver. When the MANA kernel driver receives a hardware
service event, it initiates a device reset and notifies userspace
via IBV_EVENT_DEVICE_FATAL. The MANA PMD handles this by
performing an automatic teardown and recovery sequence.
The driver uses ethdev recovery events (ERR_RECOVERING,
RECOVERY_SUCCESS, RECOVERY_FAILED) to notify upper layers of
the reset lifecycle, and a PCI device removal event callback
to distinguish hot-remove from service reset.
Changes since v1:
- Removed net/netvsc patch from this series (will be submitted
separately)
- Simplified reset exit: mana_reset_exit now calls
mana_reset_exit_delay directly instead of spawning a separate
thread, since it already runs on mana_reset_thread
- Added __rte_no_thread_safety_analysis annotations to fix clang
-Wthread-safety build errors
- Switched from rte_thread_create_control to
rte_thread_create_internal_control per DPDK preference
- Fixed declaration-after-statement style issues
- Fixed stale comments and removed unnecessary blank lines
- Added comments explaining intentionally ignored return values
Wei Hu (1):
net/mana: add device reset support
drivers/net/mana/mana.c | 928 +++++++++++++++++++++++++++++++----
drivers/net/mana/mana.h | 33 +-
drivers/net/mana/meson.build | 2 +-
drivers/net/mana/mp.c | 89 +++-
drivers/net/mana/mr.c | 6 +-
drivers/net/mana/rx.c | 24 +-
drivers/net/mana/tx.c | 40 +-
7 files changed, 1016 insertions(+), 106 deletions(-)
--
2.34.1
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox