* [PATCH 00/13] Consolidate ice and iavf vector Tx paths
@ 2026-09-03 17:01 Bruce Richardson
2026-09-03 17:01 ` [PATCH 01/13] net/iavf: remove unnecessary alignment calls Bruce Richardson
` (13 more replies)
0 siblings, 14 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson
This patchset consolidates the ice and iavf vector Tx paths, moving the
code to net/intel/common. The iavf code is used as the basis for the
move, because it has a superset of the features. When the common code is
adopted by the ice driver, it then gains features such as QinQ support
and tunneling support.
Depends-on: series-39162 ("VLAN and QinQ fixes for iavf")
Bruce Richardson (13):
net/iavf: remove unnecessary alignment calls
net/intel: make Tx context flag common
net/iavf: use separate params for VLAN and QinQ position
net/iavf: deduplicate tunnel field fill functions
net/intel: define common macros for tunneling bit-shifts
net/intel: move iavf descriptor writing functions to common
net/intel: use function callback for lldp
net/ice: use common descriptor creation functions
net/intel: move vector Tx paths to common
net/ice: use common AVX Tx functions
net/intel: add common vector Tx fns with context handling
net/intel: improve Tx path selection logic
net/ice: enable context desc offloads for vector Tx
doc/guides/rel_notes/release_26_11.rst | 6 +
drivers/net/intel/common/tx.h | 77 +-
drivers/net/intel/common/tx_vec_x86.h | 783 ++++++++++++++++++
drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +-
drivers/net/intel/i40e/i40e_rxtx.c | 2 +-
drivers/net/intel/iavf/iavf_rxtx.c | 76 +-
drivers/net/intel/iavf/iavf_rxtx.h | 34 -
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 496 +----------
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 495 +----------
drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 76 +-
drivers/net/intel/ice/ice_ethdev.h | 3 +
drivers/net/intel/ice/ice_rxtx.c | 34 +-
drivers/net/intel/ice/ice_rxtx.h | 11 +
drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 159 +---
drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 154 +---
drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 --
drivers/net/intel/idpf/idpf_rxtx.c | 3 +-
17 files changed, 1025 insertions(+), 1446 deletions(-)
create mode 100644 drivers/net/intel/common/tx_vec_x86.h
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 01/13] net/iavf: remove unnecessary alignment calls
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
@ 2026-09-03 17:01 ` Bruce Richardson
2026-09-03 17:01 ` [PATCH 02/13] net/intel: make Tx context flag common Bruce Richardson
` (12 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Vladimir Medvedkin
When we have a context descriptor on Tx of a packet from the vector
paths, that means that we always have 32-bytes being written per
descriptor/per packet, so the stores are always 32-bit aligned. This
means we can use aligned stores for each single descriptor store, and
that we never need to do an initial descriptor write for alignment when
doing a burst of descriptors.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 9 ++-------
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 9 ++-------
2 files changed, 4 insertions(+), 14 deletions(-)
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
index 7217f32cef..9b62ef53d3 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
@@ -1933,7 +1933,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
__m256i ctx_data_desc = _mm256_set_epi64x(high_data_qw, pkt->buf_iova + pkt->data_off,
high_ctx_qw, low_ctx_qw);
- _mm256_storeu_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc);
+ /* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
+ _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc);
}
static __rte_always_inline void
@@ -1944,12 +1945,6 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
uint64_t hi_data_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA |
((uint64_t)flags << IAVF_TXD_QW1_CMD_SHIFT));
- /* if unaligned on 32-bit boundary, do one to align */
- if (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {
- ctx_vtx1(txdp, *pkt, flags, offload, vlan_flag, ptype_lldp_enabled);
- nb_pkts--; txdp++; pkt++;
- }
-
for (; nb_pkts > 1; txdp += 4, pkt += 2, nb_pkts -= 2) {
uint64_t hi_ctx_qw1 = IAVF_TX_DESC_DTYPE_CONTEXT;
uint64_t hi_ctx_qw0 = IAVF_TX_DESC_DTYPE_CONTEXT;
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
index bf0245e8f4..4609c2245a 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
@@ -2078,7 +2078,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
__m256i ctx_data_desc = _mm256_set_epi64x(high_data_qw, pkt->buf_iova + pkt->data_off,
high_ctx_qw, low_ctx_qw);
- _mm256_storeu_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc);
+ /* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
+ _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc);
}
static __rte_always_inline void
@@ -2088,12 +2089,6 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
{
uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
- /* if unaligned on 32-bit boundary, do one to align */
- if (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {
- ctx_vtx1(txdp, *pkt, flags, offload, vlan_flag, lldp_enabled);
- nb_pkts--; txdp++; pkt++;
- }
-
for (; nb_pkts > 1; txdp += 4, pkt += 2, nb_pkts -= 2) {
uint64_t hi_ctx_qw1 = IAVF_TX_DESC_DTYPE_CONTEXT;
uint64_t hi_ctx_qw0 = IAVF_TX_DESC_DTYPE_CONTEXT;
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 02/13] net/intel: make Tx context flag common
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
2026-09-03 17:01 ` [PATCH 01/13] net/iavf: remove unnecessary alignment calls Bruce Richardson
@ 2026-09-03 17:01 ` Bruce Richardson
2026-09-03 17:01 ` [PATCH 03/13] net/iavf: use separate params for VLAN and QinQ position Bruce Richardson
` (11 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson
The flag for enabling context descriptors in the vector paths should not
be limited to just the iavf driver, as other drivers may use context
paths in future also. Therefore move the flag out of the iavf-specific
part of the Tx queue structure, to put with other standard queue flags.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/common/tx.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/intel/common/tx.h b/drivers/net/intel/common/tx.h
index 2d4a8b5d3c..fb5268c972 100644
--- a/drivers/net/intel/common/tx.h
+++ b/drivers/net/intel/common/tx.h
@@ -173,6 +173,7 @@ struct ci_tx_queue {
bool tx_deferred_start; /* don't start this queue in dev start */
bool q_set; /* indicate if tx queue has been configured */
bool use_vec_entry; /* use sw_ring_vec (true for vector and simple paths) */
+ bool use_ctx; /* with ctx info, each pkt needs two desc in vec paths */
union { /* the VSI this queue belongs to */
struct i40e_vsi *i40e_vsi;
struct iavf_vsi *iavf_vsi;
@@ -194,7 +195,6 @@ struct ci_tx_queue {
#define IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 BIT(1)
uint8_t vlan_flag;
uint8_t tc;
- bool use_ctx; /* with ctx info, each pkt needs two descriptors */
bool lldp_enabled;
};
struct { /* ixgbe specific values */
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 03/13] net/iavf: use separate params for VLAN and QinQ position
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
2026-09-03 17:01 ` [PATCH 01/13] net/iavf: remove unnecessary alignment calls Bruce Richardson
2026-09-03 17:01 ` [PATCH 02/13] net/intel: make Tx context flag common Bruce Richardson
@ 2026-09-03 17:01 ` Bruce Richardson
2026-09-03 17:01 ` [PATCH 04/13] net/iavf: deduplicate tunnel field fill functions Bruce Richardson
` (10 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Vladimir Medvedkin
Like is the case with the scalar path, use separate offload parameters
for the vector path to distinguish between the position of the single
VLAN tag and the QinQ outer tag. With iavf these are currently always
the same, but other drivers - with whom we may want to share code - have
different positions for these.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 64 +++++++++++--------
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 62 ++++++++++--------
drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 13 ++--
3 files changed, 79 insertions(+), 60 deletions(-)
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
index 9b62ef53d3..2aaa713702 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
@@ -1619,13 +1619,14 @@ iavf_recv_scattered_pkts_vec_avx2_flex_rxd_offload(void *rx_queue,
static __rte_always_inline void
iavf_vtx1(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf *pkt, uint64_t flags, bool offload, uint8_t vlan_flag)
+ struct rte_mbuf *pkt, uint64_t flags, bool offload,
+ enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
{
uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA |
((uint64_t)flags << CI_TXD_QW1_CMD_S) |
((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
if (offload)
- iavf_txd_enable_offload(pkt, &high_qw, vlan_flag);
+ iavf_txd_enable_offload(pkt, &high_qw, single_vlan_pos, qinq_outer_pos);
__m128i descriptor = _mm_set_epi64x(high_qw,
pkt->buf_iova + pkt->data_off);
@@ -1634,13 +1635,14 @@ iavf_vtx1(volatile struct ci_tx_desc *txdp,
static __rte_always_inline void
iavf_vtx(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload, uint8_t vlan_flag)
+ struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload,
+ enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
{
const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
/* if unaligned on 32-bit boundary, do one to align */
if (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {
- iavf_vtx1(txdp, *pkt, flags, offload, vlan_flag);
+ iavf_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
nb_pkts--; txdp++; pkt++;
}
@@ -1649,19 +1651,19 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,
uint64_t hi_qw3 = hi_qw_tmpl |
((uint64_t)pkt[3]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
if (offload)
- iavf_txd_enable_offload(pkt[3], &hi_qw3, vlan_flag);
+ iavf_txd_enable_offload(pkt[3], &hi_qw3, single_vlan_pos, qinq_outer_pos);
uint64_t hi_qw2 = hi_qw_tmpl |
((uint64_t)pkt[2]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
if (offload)
- iavf_txd_enable_offload(pkt[2], &hi_qw2, vlan_flag);
+ iavf_txd_enable_offload(pkt[2], &hi_qw2, single_vlan_pos, qinq_outer_pos);
uint64_t hi_qw1 = hi_qw_tmpl |
((uint64_t)pkt[1]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
if (offload)
- iavf_txd_enable_offload(pkt[1], &hi_qw1, vlan_flag);
+ iavf_txd_enable_offload(pkt[1], &hi_qw1, single_vlan_pos, qinq_outer_pos);
uint64_t hi_qw0 = hi_qw_tmpl |
((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
if (offload)
- iavf_txd_enable_offload(pkt[0], &hi_qw0, vlan_flag);
+ iavf_txd_enable_offload(pkt[0], &hi_qw0, single_vlan_pos, qinq_outer_pos);
__m256i desc2_3 =
_mm256_set_epi64x
@@ -1681,7 +1683,7 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,
/* do any last ones */
while (nb_pkts) {
- iavf_vtx1(txdp, *pkt, flags, offload, vlan_flag);
+ iavf_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
txdp++; pkt++; nb_pkts--;
}
}
@@ -1697,6 +1699,9 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
/* bit2 is reserved and must be set to 1 according to Spec */
uint64_t flags = CI_TX_DESC_CMD_EOP | CI_TX_DESC_CMD_ICRC;
uint64_t rs = CI_TX_DESC_CMD_RS | flags;
+ /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
+ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
+ CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
if (txq->nb_tx_free < txq->tx_free_thresh)
ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, false);
@@ -1716,11 +1721,11 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
if (nb_commit >= n) {
ci_tx_backlog_entry_vec(txep, tx_pkts, n);
- iavf_vtx(txdp, tx_pkts, n - 1, flags, offload, txq->vlan_flag);
+ iavf_vtx(txdp, tx_pkts, n - 1, flags, offload, vlan_pos, vlan_pos);
tx_pkts += (n - 1);
txdp += (n - 1);
- iavf_vtx1(txdp, *tx_pkts++, rs, offload, txq->vlan_flag);
+ iavf_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos);
nb_commit = (uint16_t)(nb_commit - n);
@@ -1734,7 +1739,7 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit);
- iavf_vtx(txdp, tx_pkts, nb_commit, flags, offload, txq->vlan_flag);
+ iavf_vtx(txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
@@ -1903,7 +1908,8 @@ iavf_fill_ctx_desc_tunneling_field(volatile uint64_t *qw0,
static __rte_always_inline void
ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
- uint64_t flags, bool offload, uint8_t vlan_flag, bool ptype_lldp_enabled)
+ uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
+ enum ci_l2tag_pos qinq_outer_pos, bool ptype_lldp_enabled)
{
uint64_t high_ctx_qw = IAVF_TX_DESC_DTYPE_CONTEXT;
uint64_t low_ctx_qw = 0;
@@ -1911,13 +1917,13 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
if (offload) {
iavf_fill_ctx_desc_tunneling_avx2(&low_ctx_qw, pkt);
if (pkt->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ?
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt->vlan_tci_outer :
(uint64_t)pkt->vlan_tci;
high_ctx_qw |= IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
low_ctx_qw |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
} else if ((pkt->ol_flags & RTE_MBUF_F_TX_VLAN) &&
- vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2) {
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
high_ctx_qw |= IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
low_ctx_qw |= (uint64_t)pkt->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
}
@@ -1928,7 +1934,7 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
((uint64_t)flags << IAVF_TXD_QW1_CMD_SHIFT) |
((uint64_t)pkt->data_len << IAVF_TXD_QW1_TX_BUF_SZ_SHIFT));
if (offload)
- iavf_txd_enable_offload(pkt, &high_data_qw, vlan_flag);
+ iavf_txd_enable_offload(pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);
__m256i ctx_data_desc = _mm256_set_epi64x(high_data_qw, pkt->buf_iova + pkt->data_off,
high_ctx_qw, low_ctx_qw);
@@ -1940,7 +1946,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
static __rte_always_inline void
ctx_vtx(volatile struct ci_tx_desc *txdp,
struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
- bool offload, uint8_t vlan_flag, bool ptype_lldp_enabled)
+ bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
+ bool ptype_lldp_enabled)
{
uint64_t hi_data_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA |
((uint64_t)flags << IAVF_TXD_QW1_CMD_SHIFT));
@@ -1964,14 +1971,14 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */
iavf_fill_ctx_desc_tunneling_field(&low_ctx_qw1, pkt[1]);
if (pkt[1]->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ?
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt[1]->vlan_tci_outer :
(uint64_t)pkt[1]->vlan_tci;
hi_ctx_qw1 |= IAVF_TX_CTX_DESC_IL2TAG2 <<
IAVF_TXD_CTX_QW1_CMD_SHIFT;
low_ctx_qw1 |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
} else if (pkt[1]->ol_flags & RTE_MBUF_F_TX_VLAN &&
- vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2) {
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
hi_ctx_qw1 |=
IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
low_ctx_qw1 |=
@@ -1985,14 +1992,14 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
iavf_fill_ctx_desc_tunneling_field(&low_ctx_qw0, pkt[0]);
if (pkt[0]->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ?
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt[0]->vlan_tci_outer :
(uint64_t)pkt[0]->vlan_tci;
hi_ctx_qw0 |= IAVF_TX_CTX_DESC_IL2TAG2 <<
IAVF_TXD_CTX_QW1_CMD_SHIFT;
low_ctx_qw0 |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
} else if (pkt[0]->ol_flags & RTE_MBUF_F_TX_VLAN &&
- vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2) {
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
hi_ctx_qw0 |=
IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
low_ctx_qw0 |=
@@ -2003,8 +2010,8 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
hi_ctx_qw0 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << IAVF_TXD_CTX_QW1_CMD_SHIFT;
if (offload) {
- iavf_txd_enable_offload(pkt[1], &hi_data_qw1, vlan_flag);
- iavf_txd_enable_offload(pkt[0], &hi_data_qw0, vlan_flag);
+ iavf_txd_enable_offload(pkt[1], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
+ iavf_txd_enable_offload(pkt[0], &hi_data_qw0, single_vlan_pos, qinq_outer_pos);
}
__m256i desc2_3 =
@@ -2020,7 +2027,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
}
if (nb_pkts)
- ctx_vtx1(txdp, *pkt, flags, offload, vlan_flag, ptype_lldp_enabled);
+ ctx_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled);
}
static __rte_always_inline uint16_t
@@ -2035,6 +2042,9 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
uint64_t flags = IAVF_TX_DESC_CMD_EOP | IAVF_TX_DESC_CMD_ICRC;
uint64_t rs = IAVF_TX_DESC_CMD_RS | flags;
bool lldp_enabled = txq->lldp_enabled;
+ /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
+ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
+ CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
if (txq->nb_tx_free < txq->tx_free_thresh)
ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, true);
@@ -2057,10 +2067,10 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = n >> 1;
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
- ctx_vtx(txdp, tx_pkts, nb_mbuf - 1, flags, offload, txq->vlan_flag, lldp_enabled);
+ ctx_vtx(txdp, tx_pkts, nb_mbuf - 1, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
tx_pkts += (nb_mbuf - 1);
txdp += (n - 2);
- ctx_vtx1(txdp, *tx_pkts++, rs, offload, txq->vlan_flag, lldp_enabled);
+ ctx_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled);
nb_commit = (uint16_t)(nb_commit - n);
@@ -2074,7 +2084,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = nb_commit >> 1;
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
- ctx_vtx(txdp, tx_pkts, nb_mbuf, flags, offload, txq->vlan_flag, lldp_enabled);
+ ctx_vtx(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
index 4609c2245a..8e44406511 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
@@ -1830,13 +1830,13 @@ tx_backlog_entry_avx512(struct ci_tx_entry_vec *txep,
static __rte_always_inline void
iavf_vtx1(volatile struct ci_tx_desc *txdp,
struct rte_mbuf *pkt, uint64_t flags,
- bool offload, uint8_t vlan_flag)
+ bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
{
uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA |
((uint64_t)flags << CI_TXD_QW1_CMD_S) |
((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
if (offload)
- iavf_txd_enable_offload(pkt, &high_qw, vlan_flag);
+ iavf_txd_enable_offload(pkt, &high_qw, single_vlan_pos, qinq_outer_pos);
__m128i descriptor = _mm_set_epi64x(high_qw,
pkt->buf_iova + pkt->data_off);
@@ -1848,13 +1848,13 @@ iavf_vtx1(volatile struct ci_tx_desc *txdp,
static __rte_always_inline void
iavf_vtx(volatile struct ci_tx_desc *txdp,
struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
- bool offload, uint8_t vlan_flag)
+ bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
{
const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
/* if unaligned on 32-bit boundary, do one to align */
if (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {
- iavf_vtx1(txdp, *pkt, flags, offload, vlan_flag);
+ iavf_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
nb_pkts--; txdp++; pkt++;
}
@@ -1869,10 +1869,10 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,
uint64_t hi_qw0 = hi_qw_tmpl |
((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
if (offload) {
- iavf_txd_enable_offload(pkt[3], &hi_qw3, vlan_flag);
- iavf_txd_enable_offload(pkt[2], &hi_qw2, vlan_flag);
- iavf_txd_enable_offload(pkt[1], &hi_qw1, vlan_flag);
- iavf_txd_enable_offload(pkt[0], &hi_qw0, vlan_flag);
+ iavf_txd_enable_offload(pkt[3], &hi_qw3, single_vlan_pos, qinq_outer_pos);
+ iavf_txd_enable_offload(pkt[2], &hi_qw2, single_vlan_pos, qinq_outer_pos);
+ iavf_txd_enable_offload(pkt[1], &hi_qw1, single_vlan_pos, qinq_outer_pos);
+ iavf_txd_enable_offload(pkt[0], &hi_qw0, single_vlan_pos, qinq_outer_pos);
}
__m512i desc0_3 =
@@ -1890,7 +1890,7 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,
/* do any last ones */
while (nb_pkts) {
- iavf_vtx1(txdp, *pkt, flags, offload, vlan_flag);
+ iavf_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
txdp++; pkt++; nb_pkts--;
}
}
@@ -2047,7 +2047,8 @@ iavf_fill_ctx_desc_tunnelling_field(volatile uint64_t *qw0,
static __rte_always_inline void
ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
- uint64_t flags, bool offload, uint8_t vlan_flag, bool lldp_enabled)
+ uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
+ enum ci_l2tag_pos qinq_outer_pos, bool lldp_enabled)
{
uint64_t high_ctx_qw = IAVF_TX_DESC_DTYPE_CONTEXT;
uint64_t low_ctx_qw = 0;
@@ -2055,13 +2056,13 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
if (offload) {
iavf_fill_ctx_desc_tunneling_avx512(&low_ctx_qw, pkt);
if (pkt->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ?
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt->vlan_tci_outer :
(uint64_t)pkt->vlan_tci;
high_ctx_qw |= IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
low_ctx_qw |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
} else if ((pkt->ol_flags & RTE_MBUF_F_TX_VLAN) &&
- vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2) {
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
high_ctx_qw |= IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
low_ctx_qw |= (uint64_t)pkt->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
}
@@ -2073,7 +2074,7 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
((uint64_t)flags << CI_TXD_QW1_CMD_S) |
((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
if (offload)
- iavf_txd_enable_offload(pkt, &high_data_qw, vlan_flag);
+ iavf_txd_enable_offload(pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);
__m256i ctx_data_desc = _mm256_set_epi64x(high_data_qw, pkt->buf_iova + pkt->data_off,
high_ctx_qw, low_ctx_qw);
@@ -2085,7 +2086,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
static __rte_always_inline void
ctx_vtx(volatile struct ci_tx_desc *txdp,
struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
- bool offload, uint8_t vlan_flag, bool lldp_enabled)
+ bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
+ bool lldp_enabled)
{
uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
@@ -2106,13 +2108,13 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */
iavf_fill_ctx_desc_tunnelling_field(&low_ctx_qw1, pkt[1]);
if (pkt[1]->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ?
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt[1]->vlan_tci_outer :
(uint64_t)pkt[1]->vlan_tci;
hi_ctx_qw1 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
low_ctx_qw1 |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
} else if (pkt[1]->ol_flags & RTE_MBUF_F_TX_VLAN &&
- vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2) {
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
hi_ctx_qw1 |= IAVF_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
low_ctx_qw1 |=
(uint64_t)pkt[1]->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
@@ -2126,13 +2128,13 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
iavf_fill_ctx_desc_tunnelling_field(&low_ctx_qw0, pkt[0]);
if (pkt[0]->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ?
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt[0]->vlan_tci_outer :
(uint64_t)pkt[0]->vlan_tci;
hi_ctx_qw0 |= IAVF_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
low_ctx_qw0 |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
} else if (pkt[0]->ol_flags & RTE_MBUF_F_TX_VLAN &&
- vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2) {
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
hi_ctx_qw0 |= IAVF_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
low_ctx_qw0 |=
(uint64_t)pkt[0]->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
@@ -2142,8 +2144,8 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
hi_ctx_qw0 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
if (offload) {
- iavf_txd_enable_offload(pkt[1], &hi_data_qw1, vlan_flag);
- iavf_txd_enable_offload(pkt[0], &hi_data_qw0, vlan_flag);
+ iavf_txd_enable_offload(pkt[1], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
+ iavf_txd_enable_offload(pkt[0], &hi_data_qw0, single_vlan_pos, qinq_outer_pos);
}
__m512i desc0_3 =
@@ -2156,7 +2158,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
}
if (nb_pkts)
- ctx_vtx1(txdp, *pkt, flags, offload, vlan_flag, lldp_enabled);
+ ctx_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos, lldp_enabled);
}
static __rte_always_inline uint16_t
@@ -2170,6 +2172,9 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
/* bit2 is reserved and must be set to 1 according to Spec */
uint64_t flags = CI_TX_DESC_CMD_EOP | CI_TX_DESC_CMD_ICRC;
uint64_t rs = CI_TX_DESC_CMD_RS | flags;
+ /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
+ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
+ CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
if (txq->nb_tx_free < txq->tx_free_thresh)
ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, false);
@@ -2190,11 +2195,11 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
if (nb_commit >= n) {
tx_backlog_entry_avx512(txep, tx_pkts, n);
- iavf_vtx(txdp, tx_pkts, n - 1, flags, offload, txq->vlan_flag);
+ iavf_vtx(txdp, tx_pkts, n - 1, flags, offload, vlan_pos, vlan_pos);
tx_pkts += (n - 1);
txdp += (n - 1);
- iavf_vtx1(txdp, *tx_pkts++, rs, offload, txq->vlan_flag);
+ iavf_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos);
nb_commit = (uint16_t)(nb_commit - n);
@@ -2209,7 +2214,7 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
tx_backlog_entry_avx512(txep, tx_pkts, nb_commit);
- iavf_vtx(txdp, tx_pkts, nb_commit, flags, offload, txq->vlan_flag);
+ iavf_vtx(txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
@@ -2238,6 +2243,9 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
uint64_t flags = CI_TX_DESC_CMD_EOP | CI_TX_DESC_CMD_ICRC;
uint64_t rs = CI_TX_DESC_CMD_RS | flags;
bool lldp_enabled = txq->lldp_enabled;
+ /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
+ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
+ CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
if (txq->nb_tx_free < txq->tx_free_thresh)
ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, true);
@@ -2260,10 +2268,10 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = n >> 1;
tx_backlog_entry_avx512(txep, tx_pkts, nb_mbuf);
- ctx_vtx(txdp, tx_pkts, nb_mbuf - 1, flags, offload, txq->vlan_flag, lldp_enabled);
+ ctx_vtx(txdp, tx_pkts, nb_mbuf - 1, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
tx_pkts += (nb_mbuf - 1);
txdp += (n - 2);
- ctx_vtx1(txdp, *tx_pkts++, rs, offload, txq->vlan_flag, lldp_enabled);
+ ctx_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled);
nb_commit = (uint16_t)(nb_commit - n);
@@ -2277,7 +2285,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = nb_commit >> 1;
tx_backlog_entry_avx512(txep, tx_pkts, nb_mbuf);
- ctx_vtx(txdp, tx_pkts, nb_mbuf, flags, offload, txq->vlan_flag, lldp_enabled);
+ ctx_vtx(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h
index 1e9ce924ee..74446fcf4a 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h
@@ -121,14 +121,13 @@ iavf_tx_vec_dev_check_default(struct rte_eth_dev *dev)
static __rte_always_inline void
iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt,
- uint64_t *txd_hi, uint8_t vlan_flag)
+ uint64_t *txd_hi, enum ci_l2tag_pos single_vlan_pos,
+ enum ci_l2tag_pos qinq_outer_pos)
{
uint64_t ol_flags = tx_pkt->ol_flags;
uint32_t td_cmd = 0;
uint32_t td_offset = 0;
- RTE_SET_USED(vlan_flag);
-
/* Set MACLEN */
if (ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK)
td_offset |= (tx_pkt->outer_l2_len >> 1)
@@ -179,12 +178,14 @@ iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt,
if (ol_flags & RTE_MBUF_F_TX_QINQ) {
td_cmd |= IAVF_TX_DESC_CMD_IL2TAG1;
- /* vlan_flag specifies outer tag location for QinQ. */
- if (vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1)
+ /* L2Tag1 always carries a tag for QinQ: the outer tag if that's
+ * where it is placed, otherwise the inner.
+ */
+ if (qinq_outer_pos == CI_TAG_IN_DATA_DESC)
*txd_hi |= ((uint64_t)tx_pkt->vlan_tci_outer << CI_TXD_QW1_L2TAG1_S);
else
*txd_hi |= ((uint64_t)tx_pkt->vlan_tci << CI_TXD_QW1_L2TAG1_S);
- } else if (ol_flags & RTE_MBUF_F_TX_VLAN && vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) {
+ } else if (ol_flags & RTE_MBUF_F_TX_VLAN && single_vlan_pos == CI_TAG_IN_DATA_DESC) {
td_cmd |= CI_TX_DESC_CMD_IL2TAG1;
*txd_hi |= ((uint64_t)tx_pkt->vlan_tci << CI_TXD_QW1_L2TAG1_S);
}
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 04/13] net/iavf: deduplicate tunnel field fill functions
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (2 preceding siblings ...)
2026-09-03 17:01 ` [PATCH 03/13] net/iavf: use separate params for VLAN and QinQ position Bruce Richardson
@ 2026-09-03 17:01 ` Bruce Richardson
2026-09-04 7:01 ` David Marchand
2026-09-03 17:01 ` [PATCH 05/13] net/intel: define common macros for tunneling bit-shifts Bruce Richardson
` (9 subsequent siblings)
13 siblings, 1 reply; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Vladimir Medvedkin
Each of the avx2 and avx512 Tx functions used two separate subfunctions
to insert the tunneling fields into the context descriptor. Except for
some minor differences (a flags guard on one) the two were identical, so
remove the one missing the additional guard.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 77 +------------------
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 77 +------------------
2 files changed, 4 insertions(+), 150 deletions(-)
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
index 2aaa713702..10e96a8510 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
@@ -1833,79 +1833,6 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
}
}
-static inline void
-iavf_fill_ctx_desc_tunneling_field(volatile uint64_t *qw0,
- const struct rte_mbuf *m)
-{
- uint64_t eip_typ = IAVF_TX_CTX_DESC_EIPT_NONE;
- uint64_t eip_len = 0;
- uint64_t eip_noinc = 0;
- /* Default - IP_ID is increment in each segment of LSO */
-
- switch (m->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 |
- RTE_MBUF_F_TX_OUTER_IPV6 |
- RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {
- case RTE_MBUF_F_TX_OUTER_IPV4:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD;
- eip_len = m->outer_l3_len >> 2;
- break;
- case RTE_MBUF_F_TX_OUTER_IPV4 | RTE_MBUF_F_TX_OUTER_IP_CKSUM:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD;
- eip_len = m->outer_l3_len >> 2;
- break;
- case RTE_MBUF_F_TX_OUTER_IPV6:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV6;
- eip_len = m->outer_l3_len >> 2;
- break;
- }
-
- /* L4TUNT: L4 Tunneling Type */
- switch (m->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
- case RTE_MBUF_F_TX_TUNNEL_IPIP:
- /* for non UDP / GRE tunneling, set to 00b */
- break;
- case RTE_MBUF_F_TX_TUNNEL_VXLAN:
- case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE:
- case RTE_MBUF_F_TX_TUNNEL_GTP:
- case RTE_MBUF_F_TX_TUNNEL_GENEVE:
- eip_typ |= IAVF_TXD_CTX_UDP_TUNNELING;
- break;
- case RTE_MBUF_F_TX_TUNNEL_GRE:
- eip_typ |= IAVF_TXD_CTX_GRE_TUNNELING;
- break;
- default:
- PMD_TX_LOG(ERR, "Tunnel type not supported");
- return;
- }
-
- /* L4TUNLEN: L4 Tunneling Length, in Words
- *
- * We depend on app to set rte_mbuf.l2_len correctly.
- * For IP in GRE it should be set to the length of the GRE
- * header;
- * For MAC in GRE or MAC in UDP it should be set to the length
- * of the GRE or UDP headers plus the inner MAC up to including
- * its last Ethertype.
- * If MPLS labels exists, it should include them as well.
- */
- eip_typ |= (m->l2_len >> 1) << IAVF_TXD_CTX_QW0_NATLEN_SHIFT;
-
- /**
- * Calculate the tunneling UDP checksum.
- * Shall be set only if L4TUNT = 01b and EIPT is not zero
- */
- if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV6 |
- IAVF_TX_CTX_EXT_IP_IPV4 |
- IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM)) &&
- (eip_typ & IAVF_TXD_CTX_UDP_TUNNELING) &&
- (m->ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM))
- eip_typ |= IAVF_TXD_CTX_QW0_L4T_CS_MASK;
-
- *qw0 = eip_typ << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT |
- eip_len << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT |
- eip_noinc << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT;
-}
-
static __rte_always_inline void
ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
@@ -1969,7 +1896,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
if (offload) {
/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */
- iavf_fill_ctx_desc_tunneling_field(&low_ctx_qw1, pkt[1]);
+ iavf_fill_ctx_desc_tunneling_avx2(&low_ctx_qw1, pkt[1]);
if (pkt[1]->ol_flags & RTE_MBUF_F_TX_QINQ) {
uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt[1]->vlan_tci_outer :
@@ -1990,7 +1917,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
if (offload) {
/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
- iavf_fill_ctx_desc_tunneling_field(&low_ctx_qw0, pkt[0]);
+ iavf_fill_ctx_desc_tunneling_avx2(&low_ctx_qw0, pkt[0]);
if (pkt[0]->ol_flags & RTE_MBUF_F_TX_QINQ) {
uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt[0]->vlan_tci_outer :
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
index 8e44406511..58ed89a72a 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
@@ -1972,79 +1972,6 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
}
}
-static inline void
-iavf_fill_ctx_desc_tunnelling_field(volatile uint64_t *qw0,
- const struct rte_mbuf *m)
-{
- uint64_t eip_typ = IAVF_TX_CTX_DESC_EIPT_NONE;
- uint64_t eip_len = 0;
- uint64_t eip_noinc = 0;
- /* Default - IP_ID is increment in each segment of LSO */
-
- switch (m->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 |
- RTE_MBUF_F_TX_OUTER_IPV6 |
- RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {
- case RTE_MBUF_F_TX_OUTER_IPV4:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD;
- eip_len = m->outer_l3_len >> 2;
- break;
- case RTE_MBUF_F_TX_OUTER_IPV4 | RTE_MBUF_F_TX_OUTER_IP_CKSUM:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD;
- eip_len = m->outer_l3_len >> 2;
- break;
- case RTE_MBUF_F_TX_OUTER_IPV6:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV6;
- eip_len = m->outer_l3_len >> 2;
- break;
- }
-
- /* L4TUNT: L4 Tunneling Type */
- switch (m->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
- case RTE_MBUF_F_TX_TUNNEL_IPIP:
- /* for non UDP / GRE tunneling, set to 00b */
- break;
- case RTE_MBUF_F_TX_TUNNEL_VXLAN:
- case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE:
- case RTE_MBUF_F_TX_TUNNEL_GTP:
- case RTE_MBUF_F_TX_TUNNEL_GENEVE:
- eip_typ |= IAVF_TXD_CTX_UDP_TUNNELING;
- break;
- case RTE_MBUF_F_TX_TUNNEL_GRE:
- eip_typ |= IAVF_TXD_CTX_GRE_TUNNELING;
- break;
- default:
- PMD_TX_LOG(ERR, "Tunnel type not supported");
- return;
- }
-
- /* L4TUNLEN: L4 Tunneling Length, in Words
- *
- * We depend on app to set rte_mbuf.l2_len correctly.
- * For IP in GRE it should be set to the length of the GRE
- * header;
- * For MAC in GRE or MAC in UDP it should be set to the length
- * of the GRE or UDP headers plus the inner MAC up to including
- * its last Ethertype.
- * If MPLS labels exists, it should include them as well.
- */
- eip_typ |= (m->l2_len >> 1) << IAVF_TXD_CTX_QW0_NATLEN_SHIFT;
-
- /**
- * Calculate the tunneling UDP checksum.
- * Shall be set only if L4TUNT = 01b and EIPT is not zero
- */
- if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV6 |
- IAVF_TX_CTX_EXT_IP_IPV4 |
- IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM)) &&
- (eip_typ & IAVF_TXD_CTX_UDP_TUNNELING) &&
- (m->ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM))
- eip_typ |= IAVF_TXD_CTX_QW0_L4T_CS_MASK;
-
- *qw0 = eip_typ << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT |
- eip_len << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT |
- eip_noinc << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT;
-}
-
static __rte_always_inline void
ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
@@ -2106,7 +2033,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
if (offload) {
/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */
- iavf_fill_ctx_desc_tunnelling_field(&low_ctx_qw1, pkt[1]);
+ iavf_fill_ctx_desc_tunneling_avx512(&low_ctx_qw1, pkt[1]);
if (pkt[1]->ol_flags & RTE_MBUF_F_TX_QINQ) {
uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt[1]->vlan_tci_outer :
@@ -2126,7 +2053,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,
if (offload) {
/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
- iavf_fill_ctx_desc_tunnelling_field(&low_ctx_qw0, pkt[0]);
+ iavf_fill_ctx_desc_tunneling_avx512(&low_ctx_qw0, pkt[0]);
if (pkt[0]->ol_flags & RTE_MBUF_F_TX_QINQ) {
uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
(uint64_t)pkt[0]->vlan_tci_outer :
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 05/13] net/intel: define common macros for tunneling bit-shifts
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (3 preceding siblings ...)
2026-09-03 17:01 ` [PATCH 04/13] net/iavf: deduplicate tunnel field fill functions Bruce Richardson
@ 2026-09-03 17:01 ` Bruce Richardson
2026-09-03 17:01 ` [PATCH 06/13] net/intel: move iavf descriptor writing functions to common Bruce Richardson
` (8 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Vladimir Medvedkin
Define common macros for use in iavf (for now) and other drivers
(future) defining the shifts needed for tunnel packet Tx
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/common/tx.h | 24 +++++++++++++
drivers/net/intel/iavf/iavf_rxtx.c | 30 ++++++++--------
drivers/net/intel/iavf/iavf_rxtx.h | 34 -------------------
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 30 ++++++++--------
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 30 ++++++++--------
5 files changed, 69 insertions(+), 79 deletions(-)
diff --git a/drivers/net/intel/common/tx.h b/drivers/net/intel/common/tx.h
index fb5268c972..c8fadee712 100644
--- a/drivers/net/intel/common/tx.h
+++ b/drivers/net/intel/common/tx.h
@@ -45,6 +45,30 @@
#define CI_TX_CTX_DESC_TSYN 0x02
#define CI_TX_CTX_DESC_IL2TAG2 0x04
+/* Common TX Context Descriptor QW0 Field Shifts */
+#define CI_TXD_CTX_QW0_L2TAG2_S 32
+#define CI_TXD_CTX_QW0_EIPT_S 0 /* 2 BITS - external IP type */
+#define CI_TXD_CTX_QW0_EIPLEN_S 2 /* 7 BITS - external IP header length */
+#define CI_TXD_CTX_QW0_NATT_S 9 /* 2 BITS - L4 tunnel type */
+#define CI_TXD_CTX_QW0_EIP_NOINC_S 11 /* 1 BIT - external IP ID not incremented */
+#define CI_TXD_CTX_QW0_NATLEN_S 12 /* 7 BITS - L4 tunnel header length */
+#define CI_TXD_CTX_QW0_L4T_CS_S 23 /* 1 BIT - tunnel UDP checksum enable */
+
+/**
+ * Enum for the external (outer) IP type field of a Tx context descriptor's
+ * tunnel parameters (QW0 bits 0-1).
+ */
+enum ci_tx_ctx_eipt {
+ CI_TX_CTX_EIPT_NONE,
+ CI_TX_CTX_EIPT_IPV6,
+ CI_TX_CTX_EIPT_IPV4_NO_CSUM,
+ CI_TX_CTX_EIPT_IPV4,
+};
+
+#define CI_TXD_CTX_UDP_TUNNELING (0x1ULL << CI_TXD_CTX_QW0_NATT_S)
+#define CI_TXD_CTX_GRE_TUNNELING (0x2ULL << CI_TXD_CTX_QW0_NATT_S)
+#define CI_TXD_CTX_QW0_L4T_CS_M (0x1ULL << CI_TXD_CTX_QW0_L4T_CS_S)
+
/**
* Enum to specify where a VLAN tag is to be placed for packet Tx.
*/
diff --git a/drivers/net/intel/iavf/iavf_rxtx.c b/drivers/net/intel/iavf/iavf_rxtx.c
index 80c9912ccc..849fc33dac 100644
--- a/drivers/net/intel/iavf/iavf_rxtx.c
+++ b/drivers/net/intel/iavf/iavf_rxtx.c
@@ -2345,7 +2345,7 @@ iavf_calc_context_desc(const struct rte_mbuf *mb, uint8_t vlan_flag, bool lldp_e
static inline void
iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const struct rte_mbuf *m)
{
- uint64_t eip_typ = IAVF_TX_CTX_DESC_EIPT_NONE;
+ uint64_t eip_typ = CI_TX_CTX_EIPT_NONE;
uint64_t eip_len = 0;
uint64_t eip_noinc = 0;
/* Default - IP_ID is increment in each segment of LSO */
@@ -2354,15 +2354,15 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const stru
RTE_MBUF_F_TX_OUTER_IPV6 |
RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {
case RTE_MBUF_F_TX_OUTER_IPV4:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD;
+ eip_typ = CI_TX_CTX_EIPT_IPV4_NO_CSUM;
eip_len = m->outer_l3_len >> 2;
break;
case RTE_MBUF_F_TX_OUTER_IPV4 | RTE_MBUF_F_TX_OUTER_IP_CKSUM:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD;
+ eip_typ = CI_TX_CTX_EIPT_IPV4;
eip_len = m->outer_l3_len >> 2;
break;
case RTE_MBUF_F_TX_OUTER_IPV6:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV6;
+ eip_typ = CI_TX_CTX_EIPT_IPV6;
eip_len = m->outer_l3_len >> 2;
break;
}
@@ -2377,10 +2377,10 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const stru
case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE:
case RTE_MBUF_F_TX_TUNNEL_GTP:
case RTE_MBUF_F_TX_TUNNEL_GENEVE:
- eip_typ |= IAVF_TXD_CTX_UDP_TUNNELING;
+ eip_typ |= CI_TXD_CTX_UDP_TUNNELING;
break;
case RTE_MBUF_F_TX_TUNNEL_GRE:
- eip_typ |= IAVF_TXD_CTX_GRE_TUNNELING;
+ eip_typ |= CI_TXD_CTX_GRE_TUNNELING;
break;
default:
PMD_TX_LOG(ERR, "Tunnel type not supported");
@@ -2397,23 +2397,23 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const stru
* its last Ethertype.
* If MPLS labels exists, it should include them as well.
*/
- eip_typ |= (m->l2_len >> 1) << IAVF_TXD_CTX_QW0_NATLEN_SHIFT;
+ eip_typ |= (m->l2_len >> 1) << CI_TXD_CTX_QW0_NATLEN_S;
/**
* Calculate the tunneling UDP checksum.
* Shall be set only if L4TUNT = 01b and EIPT is not zero
*/
- if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV6 |
- IAVF_TX_CTX_EXT_IP_IPV4 |
- IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM)) &&
- (eip_typ & IAVF_TXD_CTX_UDP_TUNNELING) &&
+ if ((eip_typ & (CI_TX_CTX_EIPT_IPV6 |
+ CI_TX_CTX_EIPT_IPV4 |
+ CI_TX_CTX_EIPT_IPV4_NO_CSUM)) &&
+ (eip_typ & CI_TXD_CTX_UDP_TUNNELING) &&
(ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM))
- eip_typ |= IAVF_TXD_CTX_QW0_L4T_CS_MASK;
+ eip_typ |= CI_TXD_CTX_QW0_L4T_CS_M;
}
- *qw0 = eip_typ << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT |
- eip_len << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT |
- eip_noinc << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT;
+ *qw0 = eip_typ << CI_TXD_CTX_QW0_EIPT_S |
+ eip_len << CI_TXD_CTX_QW0_EIPLEN_S |
+ eip_noinc << CI_TXD_CTX_QW0_EIP_NOINC_S;
}
static inline uint16_t
diff --git a/drivers/net/intel/iavf/iavf_rxtx.h b/drivers/net/intel/iavf/iavf_rxtx.h
index 3ff49b94b4..19931f5f7c 100644
--- a/drivers/net/intel/iavf/iavf_rxtx.h
+++ b/drivers/net/intel/iavf/iavf_rxtx.h
@@ -444,40 +444,6 @@ enum iavf_rx_flex_desc_ipsec_crypto_status {
#define IAVF_TXD_CTX_QW1_SEG_PARAMS_MSS_MASK \
(0x3FFFUL << IAVF_TXD_CTX_QW1_SEG_PARAMS_MSS_SHIFT)
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT (0)
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_MASK (0x3UL)
-
-enum iavf_tx_ctx_desc_tunnel_external_ip_type {
- IAVF_TX_CTX_DESC_EIPT_NONE,
- IAVF_TX_CTX_DESC_EIPT_IPV6,
- IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD,
- IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD
-};
-
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT (2)
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_MASK (0x7FUL)
-
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNT_SHIFT (9)
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNT_MASK (0x3UL)
-
-enum iavf_tx_ctx_desc_tunnel_l4_tunnel_type {
- IAVF_TX_CTX_DESC_L4_TUN_TYP_NO_UDP_GRE,
- IAVF_TX_CTX_DESC_L4_TUN_TYP_UDP,
- IAVF_TX_CTX_DESC_L4_TUN_TYP_GRE
-};
-
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT (11)
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_MASK (0x1UL)
-
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNLEN_SHIFT (12)
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNLEN_MASK (0x7FUL)
-
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_DECTTL_SHIFT (19)
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_DECTTL_MASK (0xFUL)
-
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4T_CS_SHIFT (23)
-#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4T_CS_MASK (0x1UL)
-
#define IAVF_TXD_CTX_QW0_L2TAG2_PARAM (32)
#define IAVF_TXD_CTX_QW0_L2TAG2_MASK (0xFFFFUL)
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
index 10e96a8510..f95aabe577 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
@@ -1760,7 +1760,7 @@ static inline void
iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
{
if (pkt->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
- uint64_t eip_typ = IAVF_TX_CTX_DESC_EIPT_NONE;
+ uint64_t eip_typ = CI_TX_CTX_EIPT_NONE;
uint64_t eip_len = 0;
uint64_t eip_noinc = 0;
/* Default - IP_ID is increment in each segment of LSO */
@@ -1769,15 +1769,15 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
RTE_MBUF_F_TX_OUTER_IPV6 |
RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {
case RTE_MBUF_F_TX_OUTER_IPV4:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD;
+ eip_typ = CI_TX_CTX_EIPT_IPV4_NO_CSUM;
eip_len = pkt->outer_l3_len >> 2;
break;
case RTE_MBUF_F_TX_OUTER_IPV4 | RTE_MBUF_F_TX_OUTER_IP_CKSUM:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD;
+ eip_typ = CI_TX_CTX_EIPT_IPV4;
eip_len = pkt->outer_l3_len >> 2;
break;
case RTE_MBUF_F_TX_OUTER_IPV6:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV6;
+ eip_typ = CI_TX_CTX_EIPT_IPV6;
eip_len = pkt->outer_l3_len >> 2;
break;
}
@@ -1791,10 +1791,10 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE:
case RTE_MBUF_F_TX_TUNNEL_GTP:
case RTE_MBUF_F_TX_TUNNEL_GENEVE:
- eip_typ |= IAVF_TXD_CTX_UDP_TUNNELING;
+ eip_typ |= CI_TXD_CTX_UDP_TUNNELING;
break;
case RTE_MBUF_F_TX_TUNNEL_GRE:
- eip_typ |= IAVF_TXD_CTX_GRE_TUNNELING;
+ eip_typ |= CI_TXD_CTX_GRE_TUNNELING;
break;
default:
PMD_TX_LOG(ERR, "Tunnel type not supported");
@@ -1811,22 +1811,22 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
* its last Ethertype.
* If MPLS labels exists, it should include them as well.
*/
- eip_typ |= (pkt->l2_len >> 1) << IAVF_TXD_CTX_QW0_NATLEN_SHIFT;
+ eip_typ |= (pkt->l2_len >> 1) << CI_TXD_CTX_QW0_NATLEN_S;
/**
* Calculate the tunneling UDP checksum.
* Shall be set only if L4TUNT = 01b and EIPT is not zero
*/
- if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV4 |
- IAVF_TX_CTX_EXT_IP_IPV6 |
- IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM)) &&
- (eip_typ & IAVF_TXD_CTX_UDP_TUNNELING) &&
+ if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 |
+ CI_TX_CTX_EIPT_IPV6 |
+ CI_TX_CTX_EIPT_IPV4_NO_CSUM)) &&
+ (eip_typ & CI_TXD_CTX_UDP_TUNNELING) &&
(pkt->ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM))
- eip_typ |= IAVF_TXD_CTX_QW0_L4T_CS_MASK;
+ eip_typ |= CI_TXD_CTX_QW0_L4T_CS_M;
- *low_ctx_qw = eip_typ << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT |
- eip_len << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT |
- eip_noinc << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT;
+ *low_ctx_qw = eip_typ << CI_TXD_CTX_QW0_EIPT_S |
+ eip_len << CI_TXD_CTX_QW0_EIPLEN_S |
+ eip_noinc << CI_TXD_CTX_QW0_EIP_NOINC_S;
} else {
*low_ctx_qw = 0;
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
index 58ed89a72a..74d91fff31 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
@@ -1899,7 +1899,7 @@ static __rte_always_inline void
iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
{
if (pkt->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
- uint64_t eip_typ = IAVF_TX_CTX_DESC_EIPT_NONE;
+ uint64_t eip_typ = CI_TX_CTX_EIPT_NONE;
uint64_t eip_len = 0;
uint64_t eip_noinc = 0;
/* Default - IP_ID is increment in each segment of LSO */
@@ -1908,15 +1908,15 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
RTE_MBUF_F_TX_OUTER_IPV6 |
RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {
case RTE_MBUF_F_TX_OUTER_IPV4:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD;
+ eip_typ = CI_TX_CTX_EIPT_IPV4_NO_CSUM;
eip_len = pkt->outer_l3_len >> 2;
break;
case RTE_MBUF_F_TX_OUTER_IPV4 | RTE_MBUF_F_TX_OUTER_IP_CKSUM:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD;
+ eip_typ = CI_TX_CTX_EIPT_IPV4;
eip_len = pkt->outer_l3_len >> 2;
break;
case RTE_MBUF_F_TX_OUTER_IPV6:
- eip_typ = IAVF_TX_CTX_DESC_EIPT_IPV6;
+ eip_typ = CI_TX_CTX_EIPT_IPV6;
eip_len = pkt->outer_l3_len >> 2;
break;
}
@@ -1930,10 +1930,10 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE:
case RTE_MBUF_F_TX_TUNNEL_GTP:
case RTE_MBUF_F_TX_TUNNEL_GENEVE:
- eip_typ |= IAVF_TXD_CTX_UDP_TUNNELING;
+ eip_typ |= CI_TXD_CTX_UDP_TUNNELING;
break;
case RTE_MBUF_F_TX_TUNNEL_GRE:
- eip_typ |= IAVF_TXD_CTX_GRE_TUNNELING;
+ eip_typ |= CI_TXD_CTX_GRE_TUNNELING;
break;
default:
PMD_TX_LOG(ERR, "Tunnel type not supported");
@@ -1950,22 +1950,22 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
* its last Ethertype.
* If MPLS labels exists, it should include them as well.
*/
- eip_typ |= (pkt->l2_len >> 1) << IAVF_TXD_CTX_QW0_NATLEN_SHIFT;
+ eip_typ |= (pkt->l2_len >> 1) << CI_TXD_CTX_QW0_NATLEN_S;
/**
* Calculate the tunneling UDP checksum.
* Shall be set only if L4TUNT = 01b and EIPT is not zero
*/
- if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV4 |
- IAVF_TX_CTX_EXT_IP_IPV6 |
- IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM)) &&
- (eip_typ & IAVF_TXD_CTX_UDP_TUNNELING) &&
+ if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 |
+ CI_TX_CTX_EIPT_IPV6 |
+ CI_TX_CTX_EIPT_IPV4_NO_CSUM)) &&
+ (eip_typ & CI_TXD_CTX_UDP_TUNNELING) &&
(pkt->ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM))
- eip_typ |= IAVF_TXD_CTX_QW0_L4T_CS_MASK;
+ eip_typ |= CI_TXD_CTX_QW0_L4T_CS_M;
- *low_ctx_qw = eip_typ << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT |
- eip_len << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT |
- eip_noinc << IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT;
+ *low_ctx_qw = eip_typ << CI_TXD_CTX_QW0_EIPT_S |
+ eip_len << CI_TXD_CTX_QW0_EIPLEN_S |
+ eip_noinc << CI_TXD_CTX_QW0_EIP_NOINC_S;
} else {
*low_ctx_qw = 0;
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 06/13] net/intel: move iavf descriptor writing functions to common
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (4 preceding siblings ...)
2026-09-03 17:01 ` [PATCH 05/13] net/intel: define common macros for tunneling bit-shifts Bruce Richardson
@ 2026-09-03 17:01 ` Bruce Richardson
2026-09-03 17:01 ` [PATCH 07/13] net/intel: use function callback for lldp Bruce Richardson
` (7 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Vladimir Medvedkin
In order to promote reuse of code, move the Tx vector functions for
writing descriptors from the iavf driver to the common folder in a new
tx_vec_x86.h file. This will allow reuse in later commits by other
drivers such as ice.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/common/tx_vec_x86.h | 513 ++++++++++++++++++
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 285 +---------
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 272 +---------
drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 77 +--
4 files changed, 532 insertions(+), 615 deletions(-)
create mode 100644 drivers/net/intel/common/tx_vec_x86.h
diff --git a/drivers/net/intel/common/tx_vec_x86.h b/drivers/net/intel/common/tx_vec_x86.h
new file mode 100644
index 0000000000..b65bc9fc78
--- /dev/null
+++ b/drivers/net/intel/common/tx_vec_x86.h
@@ -0,0 +1,513 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#ifndef _COMMON_INTEL_TX_VEC_X86_H_
+#define _COMMON_INTEL_TX_VEC_X86_H_
+
+#include <stdint.h>
+
+#include <rte_mbuf.h>
+
+#include "tx.h"
+
+static __rte_always_inline void
+ci_fill_ctx_desc_tunneling(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
+{
+ if (pkt->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
+ uint64_t eip_typ = CI_TX_CTX_EIPT_NONE;
+ uint64_t eip_len = 0;
+ uint64_t eip_noinc = 0;
+ /* Default - IP_ID is increment in each segment of LSO */
+
+ switch (pkt->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 |
+ RTE_MBUF_F_TX_OUTER_IPV6 |
+ RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {
+ case RTE_MBUF_F_TX_OUTER_IPV4:
+ eip_typ = CI_TX_CTX_EIPT_IPV4_NO_CSUM;
+ eip_len = pkt->outer_l3_len >> 2;
+ break;
+ case RTE_MBUF_F_TX_OUTER_IPV4 | RTE_MBUF_F_TX_OUTER_IP_CKSUM:
+ eip_typ = CI_TX_CTX_EIPT_IPV4;
+ eip_len = pkt->outer_l3_len >> 2;
+ break;
+ case RTE_MBUF_F_TX_OUTER_IPV6:
+ eip_typ = CI_TX_CTX_EIPT_IPV6;
+ eip_len = pkt->outer_l3_len >> 2;
+ break;
+ }
+
+ /* L4TUNT: L4 Tunneling Type */
+ switch (pkt->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
+ case RTE_MBUF_F_TX_TUNNEL_IPIP:
+ /* for non UDP / GRE tunneling, set to 00b */
+ break;
+ case RTE_MBUF_F_TX_TUNNEL_VXLAN:
+ case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE:
+ case RTE_MBUF_F_TX_TUNNEL_GTP:
+ case RTE_MBUF_F_TX_TUNNEL_GENEVE:
+ eip_typ |= CI_TXD_CTX_UDP_TUNNELING;
+ break;
+ case RTE_MBUF_F_TX_TUNNEL_GRE:
+ eip_typ |= CI_TXD_CTX_GRE_TUNNELING;
+ break;
+ default:
+ PMD_TX_LOG(ERR, "Tunnel type not supported");
+ return;
+ }
+
+ /* L4TUNLEN: L4 Tunneling Length, in Words
+ *
+ * We depend on app to set rte_mbuf.l2_len correctly.
+ * For IP in GRE it should be set to the length of the GRE
+ * header;
+ * For MAC in GRE or MAC in UDP it should be set to the length
+ * of the GRE or UDP headers plus the inner MAC up to including
+ * its last Ethertype.
+ * If MPLS labels exists, it should include them as well.
+ */
+ eip_typ |= (pkt->l2_len >> 1) << CI_TXD_CTX_QW0_NATLEN_S;
+
+ /**
+ * Calculate the tunneling UDP checksum.
+ * Shall be set only if L4TUNT = 01b and EIPT is not zero
+ */
+ if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 |
+ CI_TX_CTX_EIPT_IPV6 |
+ CI_TX_CTX_EIPT_IPV4_NO_CSUM)) &&
+ (eip_typ & CI_TXD_CTX_UDP_TUNNELING) &&
+ (pkt->ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM))
+ eip_typ |= CI_TXD_CTX_QW0_L4T_CS_M;
+
+ *low_ctx_qw = eip_typ << CI_TXD_CTX_QW0_EIPT_S |
+ eip_len << CI_TXD_CTX_QW0_EIPLEN_S |
+ eip_noinc << CI_TXD_CTX_QW0_EIP_NOINC_S;
+
+ } else {
+ *low_ctx_qw = 0;
+ }
+}
+
+static __rte_always_inline void
+ci_tx_vec_offload(struct rte_mbuf *tx_pkt, uint64_t *txd_hi,
+ enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
+{
+ uint64_t ol_flags = tx_pkt->ol_flags;
+ uint32_t td_cmd = 0;
+ uint32_t td_offset = 0;
+
+ /* Set MACLEN */
+ if (ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK)
+ td_offset |= (tx_pkt->outer_l2_len >> 1) << CI_TX_DESC_LEN_MACLEN_S;
+ else
+ td_offset |= (tx_pkt->l2_len >> 1) << CI_TX_DESC_LEN_MACLEN_S;
+
+ /* Enable L3 checksum offloads */
+ if (ol_flags & RTE_MBUF_F_TX_IP_CKSUM) {
+ if (ol_flags & RTE_MBUF_F_TX_IPV4) {
+ td_cmd |= CI_TX_DESC_CMD_IIPT_IPV4_CSUM;
+ td_offset |= (tx_pkt->l3_len >> 2) << CI_TX_DESC_LEN_IPLEN_S;
+ }
+ } else if (ol_flags & RTE_MBUF_F_TX_IPV4) {
+ td_cmd |= CI_TX_DESC_CMD_IIPT_IPV4;
+ td_offset |= (tx_pkt->l3_len >> 2) << CI_TX_DESC_LEN_IPLEN_S;
+ } else if (ol_flags & RTE_MBUF_F_TX_IPV6) {
+ td_cmd |= CI_TX_DESC_CMD_IIPT_IPV6;
+ td_offset |= (tx_pkt->l3_len >> 2) << CI_TX_DESC_LEN_IPLEN_S;
+ }
+
+ /* Enable L4 checksum offloads */
+ switch (ol_flags & RTE_MBUF_F_TX_L4_MASK) {
+ case RTE_MBUF_F_TX_TCP_CKSUM:
+ td_cmd |= CI_TX_DESC_CMD_L4T_EOFT_TCP;
+ td_offset |= (sizeof(struct rte_tcp_hdr) >> 2) << CI_TX_DESC_LEN_L4_LEN_S;
+ break;
+ case RTE_MBUF_F_TX_SCTP_CKSUM:
+ td_cmd |= CI_TX_DESC_CMD_L4T_EOFT_SCTP;
+ td_offset |= (sizeof(struct rte_sctp_hdr) >> 2) << CI_TX_DESC_LEN_L4_LEN_S;
+ break;
+ case RTE_MBUF_F_TX_UDP_CKSUM:
+ td_cmd |= CI_TX_DESC_CMD_L4T_EOFT_UDP;
+ td_offset |= (sizeof(struct rte_udp_hdr) >> 2) << CI_TX_DESC_LEN_L4_LEN_S;
+ break;
+ default:
+ break;
+ }
+
+ *txd_hi |= ((uint64_t)td_offset) << CI_TXD_QW1_OFFSET_S;
+
+ if (ol_flags & RTE_MBUF_F_TX_QINQ) {
+ td_cmd |= CI_TX_DESC_CMD_IL2TAG1;
+ /* L2Tag1 always carries a tag for QinQ: the outer tag if that's
+ * where it is placed, otherwise the inner.
+ */
+ if (qinq_outer_pos == CI_TAG_IN_DATA_DESC)
+ *txd_hi |= ((uint64_t)tx_pkt->vlan_tci_outer << CI_TXD_QW1_L2TAG1_S);
+ else
+ *txd_hi |= ((uint64_t)tx_pkt->vlan_tci << CI_TXD_QW1_L2TAG1_S);
+ } else if (ol_flags & RTE_MBUF_F_TX_VLAN && single_vlan_pos == CI_TAG_IN_DATA_DESC) {
+ td_cmd |= CI_TX_DESC_CMD_IL2TAG1;
+ *txd_hi |= ((uint64_t)tx_pkt->vlan_tci << CI_TXD_QW1_L2TAG1_S);
+ }
+
+ *txd_hi |= ((uint64_t)td_cmd) << CI_TXD_QW1_CMD_S;
+}
+
+static __rte_always_inline void
+ci_vtx1(volatile struct ci_tx_desc *txdp,
+ struct rte_mbuf *pkt, uint64_t flags, bool offload,
+ enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
+{
+ uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA |
+ ((uint64_t)flags << CI_TXD_QW1_CMD_S) |
+ ((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
+ if (offload)
+ ci_tx_vec_offload(pkt, &high_qw, single_vlan_pos, qinq_outer_pos);
+
+ __m128i descriptor = _mm_set_epi64x(high_qw, pkt->buf_iova + pkt->data_off);
+ _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor);
+}
+
+#ifdef __AVX2__
+
+static __rte_always_inline void
+ci_vtx_avx2(volatile struct ci_tx_desc *txdp,
+ struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload,
+ enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
+{
+ const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
+
+ /* if unaligned on 32-bit boundary, do one to align */
+ if (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {
+ ci_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
+ nb_pkts--; txdp++; pkt++;
+ }
+
+ /* do two at a time while possible, in bursts */
+ for (; nb_pkts > 3; txdp += 4, pkt += 4, nb_pkts -= 4) {
+ uint64_t hi_qw3 = hi_qw_tmpl |
+ ((uint64_t)pkt[3]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+ if (offload)
+ ci_tx_vec_offload(pkt[3], &hi_qw3, single_vlan_pos, qinq_outer_pos);
+ uint64_t hi_qw2 = hi_qw_tmpl |
+ ((uint64_t)pkt[2]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+ if (offload)
+ ci_tx_vec_offload(pkt[2], &hi_qw2, single_vlan_pos, qinq_outer_pos);
+ uint64_t hi_qw1 = hi_qw_tmpl |
+ ((uint64_t)pkt[1]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+ if (offload)
+ ci_tx_vec_offload(pkt[1], &hi_qw1, single_vlan_pos, qinq_outer_pos);
+ uint64_t hi_qw0 = hi_qw_tmpl |
+ ((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+ if (offload)
+ ci_tx_vec_offload(pkt[0], &hi_qw0, single_vlan_pos, qinq_outer_pos);
+
+ __m256i desc2_3 =
+ _mm256_set_epi64x
+ (hi_qw3,
+ pkt[3]->buf_iova + pkt[3]->data_off,
+ hi_qw2,
+ pkt[2]->buf_iova + pkt[2]->data_off);
+ __m256i desc0_1 =
+ _mm256_set_epi64x
+ (hi_qw1,
+ pkt[1]->buf_iova + pkt[1]->data_off,
+ hi_qw0,
+ pkt[0]->buf_iova + pkt[0]->data_off);
+ _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3);
+ _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1);
+ }
+
+ /* do any last ones */
+ while (nb_pkts) {
+ ci_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
+ txdp++; pkt++; nb_pkts--;
+ }
+}
+
+static __rte_always_inline void
+ci_vtx1_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
+ uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
+ enum ci_l2tag_pos qinq_outer_pos, bool ptype_lldp_enabled)
+{
+ uint64_t high_ctx_qw = CI_TX_DESC_DTYPE_CTX;
+ uint64_t low_ctx_qw = 0;
+
+ if (offload) {
+ ci_fill_ctx_desc_tunneling(&low_ctx_qw, pkt);
+ if (pkt->ol_flags & RTE_MBUF_F_TX_QINQ) {
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
+ (uint64_t)pkt->vlan_tci_outer :
+ (uint64_t)pkt->vlan_tci;
+ high_ctx_qw |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw |= qinq_tag << CI_TXD_CTX_QW0_L2TAG2_S;
+ } else if ((pkt->ol_flags & RTE_MBUF_F_TX_VLAN) &&
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
+ high_ctx_qw |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw |= (uint64_t)pkt->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
+ }
+ }
+ if (IAVF_CHECK_TX_LLDP(pkt, ptype_lldp_enabled))
+ high_ctx_qw |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA |
+ ((uint64_t)flags << CI_TXD_QW1_CMD_S) |
+ ((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
+ if (offload)
+ ci_tx_vec_offload(pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);
+
+ __m256i ctx_data_desc = _mm256_set_epi64x(high_data_qw, pkt->buf_iova + pkt->data_off,
+ high_ctx_qw, low_ctx_qw);
+
+ /* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
+ _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc);
+}
+
+static __rte_always_inline void
+ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp,
+ struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
+ bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
+ bool ptype_lldp_enabled)
+{
+ uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
+
+ for (; nb_pkts > 1; txdp += 4, pkt += 2, nb_pkts -= 2) {
+ uint64_t hi_ctx_qw1 = CI_TX_DESC_DTYPE_CTX;
+ uint64_t hi_ctx_qw0 = CI_TX_DESC_DTYPE_CTX;
+ uint64_t low_ctx_qw1 = 0;
+ uint64_t low_ctx_qw0 = 0;
+ uint64_t hi_data_qw1 = 0;
+ uint64_t hi_data_qw0 = 0;
+
+ hi_data_qw1 = hi_data_qw_tmpl |
+ ((uint64_t)pkt[1]->data_len <<
+ CI_TXD_QW1_TX_BUF_SZ_S);
+ hi_data_qw0 = hi_data_qw_tmpl |
+ ((uint64_t)pkt[0]->data_len <<
+ CI_TXD_QW1_TX_BUF_SZ_S);
+
+ if (offload) {
+ /* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */
+ ci_fill_ctx_desc_tunneling(&low_ctx_qw1, pkt[1]);
+ if (pkt[1]->ol_flags & RTE_MBUF_F_TX_QINQ) {
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
+ (uint64_t)pkt[1]->vlan_tci_outer :
+ (uint64_t)pkt[1]->vlan_tci;
+ hi_ctx_qw1 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw1 |= qinq_tag << CI_TXD_CTX_QW0_L2TAG2_S;
+ } else if (pkt[1]->ol_flags & RTE_MBUF_F_TX_VLAN &&
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
+ hi_ctx_qw1 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw1 |= (uint64_t)pkt[1]->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
+ }
+ }
+ if (IAVF_CHECK_TX_LLDP(pkt[1], ptype_lldp_enabled))
+ hi_ctx_qw1 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+
+ if (offload) {
+ /* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
+ ci_fill_ctx_desc_tunneling(&low_ctx_qw0, pkt[0]);
+ if (pkt[0]->ol_flags & RTE_MBUF_F_TX_QINQ) {
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
+ (uint64_t)pkt[0]->vlan_tci_outer :
+ (uint64_t)pkt[0]->vlan_tci;
+ hi_ctx_qw0 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw0 |= qinq_tag << CI_TXD_CTX_QW0_L2TAG2_S;
+ } else if (pkt[0]->ol_flags & RTE_MBUF_F_TX_VLAN &&
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
+ hi_ctx_qw0 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw0 |= (uint64_t)pkt[0]->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
+ }
+ }
+ if (IAVF_CHECK_TX_LLDP(pkt[0], ptype_lldp_enabled))
+ hi_ctx_qw0 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+
+ if (offload) {
+ ci_tx_vec_offload(pkt[1], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
+ ci_tx_vec_offload(pkt[0], &hi_data_qw0, single_vlan_pos, qinq_outer_pos);
+ }
+
+ __m256i desc2_3 = _mm256_set_epi64x
+ (hi_data_qw1, pkt[1]->buf_iova + pkt[1]->data_off,
+ hi_ctx_qw1, low_ctx_qw1);
+ __m256i desc0_1 = _mm256_set_epi64x
+ (hi_data_qw0, pkt[0]->buf_iova + pkt[0]->data_off,
+ hi_ctx_qw0, low_ctx_qw0);
+ _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3);
+ _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1);
+ }
+
+ if (nb_pkts)
+ ci_vtx1_ctx_avx2(txdp, *pkt, flags, offload,
+ single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled);
+}
+
+#endif /* __AVX2__ */
+
+#ifdef __AVX512VL__
+
+static __rte_always_inline void
+ci_vtx_avx512(volatile struct ci_tx_desc *txdp,
+ struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
+ bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
+{
+ const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
+
+ /* if unaligned on 32-bit boundary, do one to align */
+ if (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {
+ ci_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
+ nb_pkts--; txdp++; pkt++;
+ }
+
+ /* do 4 at a time while possible, in bursts */
+ for (; nb_pkts > 3; txdp += 4, pkt += 4, nb_pkts -= 4) {
+ uint64_t hi_qw3 = hi_qw_tmpl |
+ ((uint64_t)pkt[3]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+ uint64_t hi_qw2 = hi_qw_tmpl |
+ ((uint64_t)pkt[2]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+ uint64_t hi_qw1 = hi_qw_tmpl |
+ ((uint64_t)pkt[1]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+ uint64_t hi_qw0 = hi_qw_tmpl |
+ ((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+ if (offload) {
+ ci_tx_vec_offload(pkt[3], &hi_qw3, single_vlan_pos, qinq_outer_pos);
+ ci_tx_vec_offload(pkt[2], &hi_qw2, single_vlan_pos, qinq_outer_pos);
+ ci_tx_vec_offload(pkt[1], &hi_qw1, single_vlan_pos, qinq_outer_pos);
+ ci_tx_vec_offload(pkt[0], &hi_qw0, single_vlan_pos, qinq_outer_pos);
+ }
+
+ __m512i desc0_3 =
+ _mm512_set_epi64
+ (hi_qw3,
+ pkt[3]->buf_iova + pkt[3]->data_off,
+ hi_qw2,
+ pkt[2]->buf_iova + pkt[2]->data_off,
+ hi_qw1,
+ pkt[1]->buf_iova + pkt[1]->data_off,
+ hi_qw0,
+ pkt[0]->buf_iova + pkt[0]->data_off);
+ _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3);
+ }
+
+ /* do any last ones */
+ while (nb_pkts) {
+ ci_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
+ txdp++; pkt++; nb_pkts--;
+ }
+}
+
+static __rte_always_inline void
+ci_vtx1_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
+ uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
+ enum ci_l2tag_pos qinq_outer_pos, bool lldp_enabled)
+{
+ uint64_t high_ctx_qw = CI_TX_DESC_DTYPE_CTX;
+ uint64_t low_ctx_qw = 0;
+
+ if (offload) {
+ ci_fill_ctx_desc_tunneling(&low_ctx_qw, pkt);
+ if (pkt->ol_flags & RTE_MBUF_F_TX_QINQ) {
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
+ (uint64_t)pkt->vlan_tci_outer :
+ (uint64_t)pkt->vlan_tci;
+ high_ctx_qw |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw |= qinq_tag << CI_TXD_CTX_QW0_L2TAG2_S;
+ } else if ((pkt->ol_flags & RTE_MBUF_F_TX_VLAN) &&
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
+ high_ctx_qw |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw |= (uint64_t)pkt->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
+ }
+ }
+ if (IAVF_CHECK_TX_LLDP(pkt, lldp_enabled))
+ high_ctx_qw |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA |
+ ((uint64_t)flags << CI_TXD_QW1_CMD_S) |
+ ((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
+ if (offload)
+ ci_tx_vec_offload(pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);
+
+ __m256i ctx_data_desc = _mm256_set_epi64x
+ (high_data_qw, pkt->buf_iova + pkt->data_off,
+ high_ctx_qw, low_ctx_qw);
+
+ /* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
+ _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc);
+}
+
+static __rte_always_inline void
+ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp,
+ struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
+ bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
+ bool lldp_enabled)
+{
+ uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
+
+ for (; nb_pkts > 1; txdp += 4, pkt += 2, nb_pkts -= 2) {
+ uint64_t hi_ctx_qw1 = CI_TX_DESC_DTYPE_CTX;
+ uint64_t hi_ctx_qw0 = CI_TX_DESC_DTYPE_CTX;
+ uint64_t low_ctx_qw1 = 0;
+ uint64_t low_ctx_qw0 = 0;
+ uint64_t hi_data_qw1 = 0;
+ uint64_t hi_data_qw0 = 0;
+
+ hi_data_qw1 = hi_data_qw_tmpl |
+ ((uint64_t)pkt[1]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+ hi_data_qw0 = hi_data_qw_tmpl |
+ ((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
+
+ if (offload) {
+ /* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */
+ ci_fill_ctx_desc_tunneling(&low_ctx_qw1, pkt[1]);
+ if (pkt[1]->ol_flags & RTE_MBUF_F_TX_QINQ) {
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
+ (uint64_t)pkt[1]->vlan_tci_outer :
+ (uint64_t)pkt[1]->vlan_tci;
+ hi_ctx_qw1 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw1 |= qinq_tag << CI_TXD_CTX_QW0_L2TAG2_S;
+ } else if (pkt[1]->ol_flags & RTE_MBUF_F_TX_VLAN &&
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
+ hi_ctx_qw1 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw1 |= (uint64_t)pkt[1]->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
+ }
+ }
+ if (IAVF_CHECK_TX_LLDP(pkt[1], lldp_enabled))
+ hi_ctx_qw1 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+
+ if (offload) {
+ /* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
+ ci_fill_ctx_desc_tunneling(&low_ctx_qw0, pkt[0]);
+ if (pkt[0]->ol_flags & RTE_MBUF_F_TX_QINQ) {
+ uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
+ (uint64_t)pkt[0]->vlan_tci_outer :
+ (uint64_t)pkt[0]->vlan_tci;
+ hi_ctx_qw0 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw0 |= qinq_tag << CI_TXD_CTX_QW0_L2TAG2_S;
+ } else if (pkt[0]->ol_flags & RTE_MBUF_F_TX_VLAN &&
+ single_vlan_pos == CI_TAG_IN_CTX_DESC) {
+ hi_ctx_qw0 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
+ low_ctx_qw0 |= (uint64_t)pkt[0]->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
+ }
+ }
+ if (IAVF_CHECK_TX_LLDP(pkt[0], lldp_enabled))
+ hi_ctx_qw0 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+
+ if (offload) {
+ ci_tx_vec_offload(pkt[1], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
+ ci_tx_vec_offload(pkt[0], &hi_data_qw0, single_vlan_pos, qinq_outer_pos);
+ }
+
+ __m512i desc0_3 = _mm512_set_epi64
+ (hi_data_qw1, pkt[1]->buf_iova + pkt[1]->data_off,
+ hi_ctx_qw1, low_ctx_qw1,
+ hi_data_qw0, pkt[0]->buf_iova + pkt[0]->data_off,
+ hi_ctx_qw0, low_ctx_qw0);
+ _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3);
+ }
+
+ if (nb_pkts)
+ ci_vtx1_ctx_avx512(txdp, *pkt, flags, offload,
+ single_vlan_pos, qinq_outer_pos, lldp_enabled);
+}
+
+#endif /* __AVX512VL__ */
+
+#endif /* _COMMON_INTEL_TX_VEC_X86_H_ */
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
index f95aabe577..1d0492edd9 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
@@ -1617,77 +1617,6 @@ iavf_recv_scattered_pkts_vec_avx2_flex_rxd_offload(void *rx_queue,
}
-static __rte_always_inline void
-iavf_vtx1(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf *pkt, uint64_t flags, bool offload,
- enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
-{
- uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA |
- ((uint64_t)flags << CI_TXD_QW1_CMD_S) |
- ((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
- if (offload)
- iavf_txd_enable_offload(pkt, &high_qw, single_vlan_pos, qinq_outer_pos);
-
- __m128i descriptor = _mm_set_epi64x(high_qw,
- pkt->buf_iova + pkt->data_off);
- _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor);
-}
-
-static __rte_always_inline void
-iavf_vtx(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload,
- enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
-{
- const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
-
- /* if unaligned on 32-bit boundary, do one to align */
- if (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {
- iavf_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
- nb_pkts--; txdp++; pkt++;
- }
-
- /* do two at a time while possible, in bursts */
- for (; nb_pkts > 3; txdp += 4, pkt += 4, nb_pkts -= 4) {
- uint64_t hi_qw3 = hi_qw_tmpl |
- ((uint64_t)pkt[3]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (offload)
- iavf_txd_enable_offload(pkt[3], &hi_qw3, single_vlan_pos, qinq_outer_pos);
- uint64_t hi_qw2 = hi_qw_tmpl |
- ((uint64_t)pkt[2]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (offload)
- iavf_txd_enable_offload(pkt[2], &hi_qw2, single_vlan_pos, qinq_outer_pos);
- uint64_t hi_qw1 = hi_qw_tmpl |
- ((uint64_t)pkt[1]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (offload)
- iavf_txd_enable_offload(pkt[1], &hi_qw1, single_vlan_pos, qinq_outer_pos);
- uint64_t hi_qw0 = hi_qw_tmpl |
- ((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (offload)
- iavf_txd_enable_offload(pkt[0], &hi_qw0, single_vlan_pos, qinq_outer_pos);
-
- __m256i desc2_3 =
- _mm256_set_epi64x
- (hi_qw3,
- pkt[3]->buf_iova + pkt[3]->data_off,
- hi_qw2,
- pkt[2]->buf_iova + pkt[2]->data_off);
- __m256i desc0_1 =
- _mm256_set_epi64x
- (hi_qw1,
- pkt[1]->buf_iova + pkt[1]->data_off,
- hi_qw0,
- pkt[0]->buf_iova + pkt[0]->data_off);
- _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3);
- _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1);
- }
-
- /* do any last ones */
- while (nb_pkts) {
- iavf_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
- txdp++; pkt++; nb_pkts--;
- }
-}
-
static __rte_always_inline uint16_t
iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts, bool offload)
@@ -1721,11 +1650,11 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
if (nb_commit >= n) {
ci_tx_backlog_entry_vec(txep, tx_pkts, n);
- iavf_vtx(txdp, tx_pkts, n - 1, flags, offload, vlan_pos, vlan_pos);
+ ci_vtx_avx2(txdp, tx_pkts, n - 1, flags, offload, vlan_pos, vlan_pos);
tx_pkts += (n - 1);
txdp += (n - 1);
- iavf_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos);
+ ci_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos);
nb_commit = (uint16_t)(nb_commit - n);
@@ -1739,7 +1668,7 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit);
- iavf_vtx(txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos);
+ ci_vtx_avx2(txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
@@ -1756,207 +1685,6 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
return nb_pkts;
}
-static inline void
-iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
-{
- if (pkt->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
- uint64_t eip_typ = CI_TX_CTX_EIPT_NONE;
- uint64_t eip_len = 0;
- uint64_t eip_noinc = 0;
- /* Default - IP_ID is increment in each segment of LSO */
-
- switch (pkt->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 |
- RTE_MBUF_F_TX_OUTER_IPV6 |
- RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {
- case RTE_MBUF_F_TX_OUTER_IPV4:
- eip_typ = CI_TX_CTX_EIPT_IPV4_NO_CSUM;
- eip_len = pkt->outer_l3_len >> 2;
- break;
- case RTE_MBUF_F_TX_OUTER_IPV4 | RTE_MBUF_F_TX_OUTER_IP_CKSUM:
- eip_typ = CI_TX_CTX_EIPT_IPV4;
- eip_len = pkt->outer_l3_len >> 2;
- break;
- case RTE_MBUF_F_TX_OUTER_IPV6:
- eip_typ = CI_TX_CTX_EIPT_IPV6;
- eip_len = pkt->outer_l3_len >> 2;
- break;
- }
-
- /* L4TUNT: L4 Tunneling Type */
- switch (pkt->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
- case RTE_MBUF_F_TX_TUNNEL_IPIP:
- /* for non UDP / GRE tunneling, set to 00b */
- break;
- case RTE_MBUF_F_TX_TUNNEL_VXLAN:
- case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE:
- case RTE_MBUF_F_TX_TUNNEL_GTP:
- case RTE_MBUF_F_TX_TUNNEL_GENEVE:
- eip_typ |= CI_TXD_CTX_UDP_TUNNELING;
- break;
- case RTE_MBUF_F_TX_TUNNEL_GRE:
- eip_typ |= CI_TXD_CTX_GRE_TUNNELING;
- break;
- default:
- PMD_TX_LOG(ERR, "Tunnel type not supported");
- return;
- }
-
- /* L4TUNLEN: L4 Tunneling Length, in Words
- *
- * We depend on app to set rte_mbuf.l2_len correctly.
- * For IP in GRE it should be set to the length of the GRE
- * header;
- * For MAC in GRE or MAC in UDP it should be set to the length
- * of the GRE or UDP headers plus the inner MAC up to including
- * its last Ethertype.
- * If MPLS labels exists, it should include them as well.
- */
- eip_typ |= (pkt->l2_len >> 1) << CI_TXD_CTX_QW0_NATLEN_S;
-
- /**
- * Calculate the tunneling UDP checksum.
- * Shall be set only if L4TUNT = 01b and EIPT is not zero
- */
- if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 |
- CI_TX_CTX_EIPT_IPV6 |
- CI_TX_CTX_EIPT_IPV4_NO_CSUM)) &&
- (eip_typ & CI_TXD_CTX_UDP_TUNNELING) &&
- (pkt->ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM))
- eip_typ |= CI_TXD_CTX_QW0_L4T_CS_M;
-
- *low_ctx_qw = eip_typ << CI_TXD_CTX_QW0_EIPT_S |
- eip_len << CI_TXD_CTX_QW0_EIPLEN_S |
- eip_noinc << CI_TXD_CTX_QW0_EIP_NOINC_S;
-
- } else {
- *low_ctx_qw = 0;
- }
-}
-
-static __rte_always_inline void
-ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
- uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
- enum ci_l2tag_pos qinq_outer_pos, bool ptype_lldp_enabled)
-{
- uint64_t high_ctx_qw = IAVF_TX_DESC_DTYPE_CONTEXT;
- uint64_t low_ctx_qw = 0;
-
- if (offload) {
- iavf_fill_ctx_desc_tunneling_avx2(&low_ctx_qw, pkt);
- if (pkt->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
- (uint64_t)pkt->vlan_tci_outer :
- (uint64_t)pkt->vlan_tci;
- high_ctx_qw |= IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
- low_ctx_qw |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- } else if ((pkt->ol_flags & RTE_MBUF_F_TX_VLAN) &&
- single_vlan_pos == CI_TAG_IN_CTX_DESC) {
- high_ctx_qw |= IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
- low_ctx_qw |= (uint64_t)pkt->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- }
- }
- if (IAVF_CHECK_TX_LLDP(pkt, ptype_lldp_enabled))
- high_ctx_qw |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << IAVF_TXD_CTX_QW1_CMD_SHIFT;
- uint64_t high_data_qw = (IAVF_TX_DESC_DTYPE_DATA |
- ((uint64_t)flags << IAVF_TXD_QW1_CMD_SHIFT) |
- ((uint64_t)pkt->data_len << IAVF_TXD_QW1_TX_BUF_SZ_SHIFT));
- if (offload)
- iavf_txd_enable_offload(pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);
-
- __m256i ctx_data_desc = _mm256_set_epi64x(high_data_qw, pkt->buf_iova + pkt->data_off,
- high_ctx_qw, low_ctx_qw);
-
- /* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
- _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc);
-}
-
-static __rte_always_inline void
-ctx_vtx(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
- bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
- bool ptype_lldp_enabled)
-{
- uint64_t hi_data_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA |
- ((uint64_t)flags << IAVF_TXD_QW1_CMD_SHIFT));
-
- for (; nb_pkts > 1; txdp += 4, pkt += 2, nb_pkts -= 2) {
- uint64_t hi_ctx_qw1 = IAVF_TX_DESC_DTYPE_CONTEXT;
- uint64_t hi_ctx_qw0 = IAVF_TX_DESC_DTYPE_CONTEXT;
- uint64_t low_ctx_qw1 = 0;
- uint64_t low_ctx_qw0 = 0;
- uint64_t hi_data_qw1 = 0;
- uint64_t hi_data_qw0 = 0;
-
- hi_data_qw1 = hi_data_qw_tmpl |
- ((uint64_t)pkt[1]->data_len <<
- IAVF_TXD_QW1_TX_BUF_SZ_SHIFT);
- hi_data_qw0 = hi_data_qw_tmpl |
- ((uint64_t)pkt[0]->data_len <<
- IAVF_TXD_QW1_TX_BUF_SZ_SHIFT);
-
- if (offload) {
- /* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */
- iavf_fill_ctx_desc_tunneling_avx2(&low_ctx_qw1, pkt[1]);
- if (pkt[1]->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
- (uint64_t)pkt[1]->vlan_tci_outer :
- (uint64_t)pkt[1]->vlan_tci;
- hi_ctx_qw1 |= IAVF_TX_CTX_DESC_IL2TAG2 <<
- IAVF_TXD_CTX_QW1_CMD_SHIFT;
- low_ctx_qw1 |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- } else if (pkt[1]->ol_flags & RTE_MBUF_F_TX_VLAN &&
- single_vlan_pos == CI_TAG_IN_CTX_DESC) {
- hi_ctx_qw1 |=
- IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
- low_ctx_qw1 |=
- (uint64_t)pkt[1]->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- }
- }
- if (IAVF_CHECK_TX_LLDP(pkt[1], ptype_lldp_enabled))
- hi_ctx_qw1 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << IAVF_TXD_CTX_QW1_CMD_SHIFT;
-
- if (offload) {
- /* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
- iavf_fill_ctx_desc_tunneling_avx2(&low_ctx_qw0, pkt[0]);
- if (pkt[0]->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
- (uint64_t)pkt[0]->vlan_tci_outer :
- (uint64_t)pkt[0]->vlan_tci;
- hi_ctx_qw0 |= IAVF_TX_CTX_DESC_IL2TAG2 <<
- IAVF_TXD_CTX_QW1_CMD_SHIFT;
- low_ctx_qw0 |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- } else if (pkt[0]->ol_flags & RTE_MBUF_F_TX_VLAN &&
- single_vlan_pos == CI_TAG_IN_CTX_DESC) {
- hi_ctx_qw0 |=
- IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
- low_ctx_qw0 |=
- (uint64_t)pkt[0]->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- }
- }
- if (IAVF_CHECK_TX_LLDP(pkt[0], ptype_lldp_enabled))
- hi_ctx_qw0 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << IAVF_TXD_CTX_QW1_CMD_SHIFT;
-
- if (offload) {
- iavf_txd_enable_offload(pkt[1], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
- iavf_txd_enable_offload(pkt[0], &hi_data_qw0, single_vlan_pos, qinq_outer_pos);
- }
-
- __m256i desc2_3 =
- _mm256_set_epi64x
- (hi_data_qw1, pkt[1]->buf_iova + pkt[1]->data_off,
- hi_ctx_qw1, low_ctx_qw1);
- __m256i desc0_1 =
- _mm256_set_epi64x
- (hi_data_qw0, pkt[0]->buf_iova + pkt[0]->data_off,
- hi_ctx_qw0, low_ctx_qw0);
- _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3);
- _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1);
- }
-
- if (nb_pkts)
- ctx_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled);
-}
-
static __rte_always_inline uint16_t
iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts, bool offload)
@@ -1994,10 +1722,11 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = n >> 1;
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
- ctx_vtx(txdp, tx_pkts, nb_mbuf - 1, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx_ctx_avx2(txdp, tx_pkts, nb_mbuf - 1, flags, offload,
+ vlan_pos, vlan_pos, lldp_enabled);
tx_pkts += (nb_mbuf - 1);
txdp += (n - 2);
- ctx_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx1_ctx_avx2(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled);
nb_commit = (uint16_t)(nb_commit - n);
@@ -2011,7 +1740,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = nb_commit >> 1;
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
- ctx_vtx(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx_ctx_avx2(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
index 74d91fff31..0c738e4882 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
@@ -1827,266 +1827,8 @@ tx_backlog_entry_avx512(struct ci_tx_entry_vec *txep,
txep[i].mbuf = tx_pkts[i];
}
-static __rte_always_inline void
-iavf_vtx1(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf *pkt, uint64_t flags,
- bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
-{
- uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA |
- ((uint64_t)flags << CI_TXD_QW1_CMD_S) |
- ((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
- if (offload)
- iavf_txd_enable_offload(pkt, &high_qw, single_vlan_pos, qinq_outer_pos);
-
- __m128i descriptor = _mm_set_epi64x(high_qw,
- pkt->buf_iova + pkt->data_off);
- _mm_storeu_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor);
-}
-
#define IAVF_TX_LEN_MASK 0xAA
#define IAVF_TX_OFF_MASK 0x55
-static __rte_always_inline void
-iavf_vtx(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
- bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
-{
- const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
-
- /* if unaligned on 32-bit boundary, do one to align */
- if (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {
- iavf_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
- nb_pkts--; txdp++; pkt++;
- }
-
- /* do 4 at a time while possible, in bursts */
- for (; nb_pkts > 3; txdp += 4, pkt += 4, nb_pkts -= 4) {
- uint64_t hi_qw3 = hi_qw_tmpl |
- ((uint64_t)pkt[3]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- uint64_t hi_qw2 = hi_qw_tmpl |
- ((uint64_t)pkt[2]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- uint64_t hi_qw1 = hi_qw_tmpl |
- ((uint64_t)pkt[1]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- uint64_t hi_qw0 = hi_qw_tmpl |
- ((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (offload) {
- iavf_txd_enable_offload(pkt[3], &hi_qw3, single_vlan_pos, qinq_outer_pos);
- iavf_txd_enable_offload(pkt[2], &hi_qw2, single_vlan_pos, qinq_outer_pos);
- iavf_txd_enable_offload(pkt[1], &hi_qw1, single_vlan_pos, qinq_outer_pos);
- iavf_txd_enable_offload(pkt[0], &hi_qw0, single_vlan_pos, qinq_outer_pos);
- }
-
- __m512i desc0_3 =
- _mm512_set_epi64
- (hi_qw3,
- pkt[3]->buf_iova + pkt[3]->data_off,
- hi_qw2,
- pkt[2]->buf_iova + pkt[2]->data_off,
- hi_qw1,
- pkt[1]->buf_iova + pkt[1]->data_off,
- hi_qw0,
- pkt[0]->buf_iova + pkt[0]->data_off);
- _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3);
- }
-
- /* do any last ones */
- while (nb_pkts) {
- iavf_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos);
- txdp++; pkt++; nb_pkts--;
- }
-}
-
-static __rte_always_inline void
-iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
-{
- if (pkt->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
- uint64_t eip_typ = CI_TX_CTX_EIPT_NONE;
- uint64_t eip_len = 0;
- uint64_t eip_noinc = 0;
- /* Default - IP_ID is increment in each segment of LSO */
-
- switch (pkt->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 |
- RTE_MBUF_F_TX_OUTER_IPV6 |
- RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {
- case RTE_MBUF_F_TX_OUTER_IPV4:
- eip_typ = CI_TX_CTX_EIPT_IPV4_NO_CSUM;
- eip_len = pkt->outer_l3_len >> 2;
- break;
- case RTE_MBUF_F_TX_OUTER_IPV4 | RTE_MBUF_F_TX_OUTER_IP_CKSUM:
- eip_typ = CI_TX_CTX_EIPT_IPV4;
- eip_len = pkt->outer_l3_len >> 2;
- break;
- case RTE_MBUF_F_TX_OUTER_IPV6:
- eip_typ = CI_TX_CTX_EIPT_IPV6;
- eip_len = pkt->outer_l3_len >> 2;
- break;
- }
-
- /* L4TUNT: L4 Tunneling Type */
- switch (pkt->ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK) {
- case RTE_MBUF_F_TX_TUNNEL_IPIP:
- /* for non UDP / GRE tunneling, set to 00b */
- break;
- case RTE_MBUF_F_TX_TUNNEL_VXLAN:
- case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE:
- case RTE_MBUF_F_TX_TUNNEL_GTP:
- case RTE_MBUF_F_TX_TUNNEL_GENEVE:
- eip_typ |= CI_TXD_CTX_UDP_TUNNELING;
- break;
- case RTE_MBUF_F_TX_TUNNEL_GRE:
- eip_typ |= CI_TXD_CTX_GRE_TUNNELING;
- break;
- default:
- PMD_TX_LOG(ERR, "Tunnel type not supported");
- return;
- }
-
- /* L4TUNLEN: L4 Tunneling Length, in Words
- *
- * We depend on app to set rte_mbuf.l2_len correctly.
- * For IP in GRE it should be set to the length of the GRE
- * header;
- * For MAC in GRE or MAC in UDP it should be set to the length
- * of the GRE or UDP headers plus the inner MAC up to including
- * its last Ethertype.
- * If MPLS labels exists, it should include them as well.
- */
- eip_typ |= (pkt->l2_len >> 1) << CI_TXD_CTX_QW0_NATLEN_S;
-
- /**
- * Calculate the tunneling UDP checksum.
- * Shall be set only if L4TUNT = 01b and EIPT is not zero
- */
- if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 |
- CI_TX_CTX_EIPT_IPV6 |
- CI_TX_CTX_EIPT_IPV4_NO_CSUM)) &&
- (eip_typ & CI_TXD_CTX_UDP_TUNNELING) &&
- (pkt->ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM))
- eip_typ |= CI_TXD_CTX_QW0_L4T_CS_M;
-
- *low_ctx_qw = eip_typ << CI_TXD_CTX_QW0_EIPT_S |
- eip_len << CI_TXD_CTX_QW0_EIPLEN_S |
- eip_noinc << CI_TXD_CTX_QW0_EIP_NOINC_S;
-
- } else {
- *low_ctx_qw = 0;
- }
-}
-
-static __rte_always_inline void
-ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
- uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
- enum ci_l2tag_pos qinq_outer_pos, bool lldp_enabled)
-{
- uint64_t high_ctx_qw = IAVF_TX_DESC_DTYPE_CONTEXT;
- uint64_t low_ctx_qw = 0;
-
- if (offload) {
- iavf_fill_ctx_desc_tunneling_avx512(&low_ctx_qw, pkt);
- if (pkt->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
- (uint64_t)pkt->vlan_tci_outer :
- (uint64_t)pkt->vlan_tci;
- high_ctx_qw |= IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
- low_ctx_qw |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- } else if ((pkt->ol_flags & RTE_MBUF_F_TX_VLAN) &&
- single_vlan_pos == CI_TAG_IN_CTX_DESC) {
- high_ctx_qw |= IAVF_TX_CTX_DESC_IL2TAG2 << IAVF_TXD_CTX_QW1_CMD_SHIFT;
- low_ctx_qw |= (uint64_t)pkt->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- }
- }
- if (IAVF_CHECK_TX_LLDP(pkt, lldp_enabled))
- high_ctx_qw |= IAVF_TX_CTX_DESC_SWTCH_UPLINK
- << IAVF_TXD_CTX_QW1_CMD_SHIFT;
- uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA |
- ((uint64_t)flags << CI_TXD_QW1_CMD_S) |
- ((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
- if (offload)
- iavf_txd_enable_offload(pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);
-
- __m256i ctx_data_desc = _mm256_set_epi64x(high_data_qw, pkt->buf_iova + pkt->data_off,
- high_ctx_qw, low_ctx_qw);
-
- /* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
- _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc);
-}
-
-static __rte_always_inline void
-ctx_vtx(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
- bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
- bool lldp_enabled)
-{
- uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
-
- for (; nb_pkts > 1; txdp += 4, pkt += 2, nb_pkts -= 2) {
- uint64_t hi_ctx_qw1 = IAVF_TX_DESC_DTYPE_CONTEXT;
- uint64_t hi_ctx_qw0 = IAVF_TX_DESC_DTYPE_CONTEXT;
- uint64_t low_ctx_qw1 = 0;
- uint64_t low_ctx_qw0 = 0;
- uint64_t hi_data_qw1 = 0;
- uint64_t hi_data_qw0 = 0;
-
- hi_data_qw1 = hi_data_qw_tmpl |
- ((uint64_t)pkt[1]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- hi_data_qw0 = hi_data_qw_tmpl |
- ((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
-
- if (offload) {
- /* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */
- iavf_fill_ctx_desc_tunneling_avx512(&low_ctx_qw1, pkt[1]);
- if (pkt[1]->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
- (uint64_t)pkt[1]->vlan_tci_outer :
- (uint64_t)pkt[1]->vlan_tci;
- hi_ctx_qw1 |= CI_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
- low_ctx_qw1 |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- } else if (pkt[1]->ol_flags & RTE_MBUF_F_TX_VLAN &&
- single_vlan_pos == CI_TAG_IN_CTX_DESC) {
- hi_ctx_qw1 |= IAVF_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
- low_ctx_qw1 |=
- (uint64_t)pkt[1]->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- }
- }
- if (IAVF_CHECK_TX_LLDP(pkt[1], lldp_enabled))
- hi_ctx_qw1 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK
- << CI_TXD_QW1_CMD_S;
-
- if (offload) {
- /* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
- iavf_fill_ctx_desc_tunneling_avx512(&low_ctx_qw0, pkt[0]);
- if (pkt[0]->ol_flags & RTE_MBUF_F_TX_QINQ) {
- uint64_t qinq_tag = qinq_outer_pos == CI_TAG_IN_CTX_DESC ?
- (uint64_t)pkt[0]->vlan_tci_outer :
- (uint64_t)pkt[0]->vlan_tci;
- hi_ctx_qw0 |= IAVF_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
- low_ctx_qw0 |= qinq_tag << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- } else if (pkt[0]->ol_flags & RTE_MBUF_F_TX_VLAN &&
- single_vlan_pos == CI_TAG_IN_CTX_DESC) {
- hi_ctx_qw0 |= IAVF_TX_CTX_DESC_IL2TAG2 << CI_TXD_QW1_CMD_S;
- low_ctx_qw0 |=
- (uint64_t)pkt[0]->vlan_tci << IAVF_TXD_CTX_QW0_L2TAG2_PARAM;
- }
- }
- if (IAVF_CHECK_TX_LLDP(pkt[0], lldp_enabled))
- hi_ctx_qw0 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
-
- if (offload) {
- iavf_txd_enable_offload(pkt[1], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
- iavf_txd_enable_offload(pkt[0], &hi_data_qw0, single_vlan_pos, qinq_outer_pos);
- }
-
- __m512i desc0_3 =
- _mm512_set_epi64
- (hi_data_qw1, pkt[1]->buf_iova + pkt[1]->data_off,
- hi_ctx_qw1, low_ctx_qw1,
- hi_data_qw0, pkt[0]->buf_iova + pkt[0]->data_off,
- hi_ctx_qw0, low_ctx_qw0);
- _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3);
- }
-
- if (nb_pkts)
- ctx_vtx1(txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos, lldp_enabled);
-}
static __rte_always_inline uint16_t
iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
@@ -2122,11 +1864,11 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
if (nb_commit >= n) {
tx_backlog_entry_avx512(txep, tx_pkts, n);
- iavf_vtx(txdp, tx_pkts, n - 1, flags, offload, vlan_pos, vlan_pos);
+ ci_vtx_avx512(txdp, tx_pkts, n - 1, flags, offload, vlan_pos, vlan_pos);
tx_pkts += (n - 1);
txdp += (n - 1);
- iavf_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos);
+ ci_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos);
nb_commit = (uint16_t)(nb_commit - n);
@@ -2141,7 +1883,7 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
tx_backlog_entry_avx512(txep, tx_pkts, nb_commit);
- iavf_vtx(txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos);
+ ci_vtx_avx512(txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
@@ -2195,10 +1937,12 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = n >> 1;
tx_backlog_entry_avx512(txep, tx_pkts, nb_mbuf);
- ctx_vtx(txdp, tx_pkts, nb_mbuf - 1, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx_ctx_avx512(txdp, tx_pkts, nb_mbuf - 1, flags, offload,
+ vlan_pos, vlan_pos, lldp_enabled);
tx_pkts += (nb_mbuf - 1);
txdp += (n - 2);
- ctx_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx1_ctx_avx512(txdp, *tx_pkts++, rs, offload,
+ vlan_pos, vlan_pos, lldp_enabled);
nb_commit = (uint16_t)(nb_commit - n);
@@ -2212,7 +1956,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = nb_commit >> 1;
tx_backlog_entry_avx512(txep, tx_pkts, nb_mbuf);
- ctx_vtx(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx_ctx_avx512(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h
index 74446fcf4a..61478036ec 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h
@@ -11,6 +11,10 @@
#include "iavf.h"
#include "iavf_rxtx.h"
+#ifdef RTE_ARCH_X86
+#include "../common/tx_vec_x86.h"
+#endif
+
static inline int
iavf_tx_desc_done(struct ci_tx_queue *txq, uint16_t idx)
{
@@ -119,77 +123,4 @@ iavf_tx_vec_dev_check_default(struct rte_eth_dev *dev)
return ret;
}
-static __rte_always_inline void
-iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt,
- uint64_t *txd_hi, enum ci_l2tag_pos single_vlan_pos,
- enum ci_l2tag_pos qinq_outer_pos)
-{
- uint64_t ol_flags = tx_pkt->ol_flags;
- uint32_t td_cmd = 0;
- uint32_t td_offset = 0;
-
- /* Set MACLEN */
- if (ol_flags & RTE_MBUF_F_TX_TUNNEL_MASK)
- td_offset |= (tx_pkt->outer_l2_len >> 1)
- << CI_TX_DESC_LEN_MACLEN_S;
- else
- td_offset |= (tx_pkt->l2_len >> 1)
- << CI_TX_DESC_LEN_MACLEN_S;
-
- /* Enable L3 checksum offloads */
- if (ol_flags & RTE_MBUF_F_TX_IP_CKSUM) {
- if (ol_flags & RTE_MBUF_F_TX_IPV4) {
- td_cmd |= CI_TX_DESC_CMD_IIPT_IPV4_CSUM;
- td_offset |= (tx_pkt->l3_len >> 2) <<
- CI_TX_DESC_LEN_IPLEN_S;
- }
- } else if (ol_flags & RTE_MBUF_F_TX_IPV4) {
- td_cmd |= CI_TX_DESC_CMD_IIPT_IPV4;
- td_offset |= (tx_pkt->l3_len >> 2) <<
- CI_TX_DESC_LEN_IPLEN_S;
- } else if (ol_flags & RTE_MBUF_F_TX_IPV6) {
- td_cmd |= CI_TX_DESC_CMD_IIPT_IPV6;
- td_offset |= (tx_pkt->l3_len >> 2) <<
- CI_TX_DESC_LEN_IPLEN_S;
- }
-
- /* Enable L4 checksum offloads */
- switch (ol_flags & RTE_MBUF_F_TX_L4_MASK) {
- case RTE_MBUF_F_TX_TCP_CKSUM:
- td_cmd |= IAVF_TX_DESC_CMD_L4T_EOFT_TCP;
- td_offset |= (sizeof(struct rte_tcp_hdr) >> 2) <<
- IAVF_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
- break;
- case RTE_MBUF_F_TX_SCTP_CKSUM:
- td_cmd |= IAVF_TX_DESC_CMD_L4T_EOFT_SCTP;
- td_offset |= (sizeof(struct rte_sctp_hdr) >> 2) <<
- IAVF_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
- break;
- case RTE_MBUF_F_TX_UDP_CKSUM:
- td_cmd |= IAVF_TX_DESC_CMD_L4T_EOFT_UDP;
- td_offset |= (sizeof(struct rte_udp_hdr) >> 2) <<
- IAVF_TX_DESC_LENGTH_L4_FC_LEN_SHIFT;
- break;
- default:
- break;
- }
-
- *txd_hi |= ((uint64_t)td_offset) << CI_TXD_QW1_OFFSET_S;
-
- if (ol_flags & RTE_MBUF_F_TX_QINQ) {
- td_cmd |= IAVF_TX_DESC_CMD_IL2TAG1;
- /* L2Tag1 always carries a tag for QinQ: the outer tag if that's
- * where it is placed, otherwise the inner.
- */
- if (qinq_outer_pos == CI_TAG_IN_DATA_DESC)
- *txd_hi |= ((uint64_t)tx_pkt->vlan_tci_outer << CI_TXD_QW1_L2TAG1_S);
- else
- *txd_hi |= ((uint64_t)tx_pkt->vlan_tci << CI_TXD_QW1_L2TAG1_S);
- } else if (ol_flags & RTE_MBUF_F_TX_VLAN && single_vlan_pos == CI_TAG_IN_DATA_DESC) {
- td_cmd |= CI_TX_DESC_CMD_IL2TAG1;
- *txd_hi |= ((uint64_t)tx_pkt->vlan_tci << CI_TXD_QW1_L2TAG1_S);
- }
-
- *txd_hi |= ((uint64_t)td_cmd) << CI_TXD_QW1_CMD_S;
-}
#endif
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 07/13] net/intel: use function callback for lldp
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (5 preceding siblings ...)
2026-09-03 17:01 ` [PATCH 06/13] net/intel: move iavf descriptor writing functions to common Bruce Richardson
@ 2026-09-03 17:01 ` Bruce Richardson
2026-09-03 17:01 ` [PATCH 08/13] net/ice: use common descriptor creation functions Bruce Richardson
` (6 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Vladimir Medvedkin
Limit LLDP support for iavf drivers for now, by using a callback to
handle its processing. Drivers not supporting this pass a NULL pointer
and the compiler will optimize away the branches. For iavf, the compiler
can inline the subfunction directly as it's known at compile-time.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/common/tx_vec_x86.h | 42 +++++++++++--------
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 16 +++++--
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 17 +++++---
3 files changed, 48 insertions(+), 27 deletions(-)
diff --git a/drivers/net/intel/common/tx_vec_x86.h b/drivers/net/intel/common/tx_vec_x86.h
index b65bc9fc78..f769765cba 100644
--- a/drivers/net/intel/common/tx_vec_x86.h
+++ b/drivers/net/intel/common/tx_vec_x86.h
@@ -11,6 +11,12 @@
#include "tx.h"
+/* Optional per-driver LLDP switch-uplink check for ctx descriptors.
+ * Takes the ctx-desc high qword and returns it with the LLDP bit applied
+ * if appropriate. NULL disables the check.
+ */
+typedef uint64_t (*ci_tx_ctx_lldp_fn)(struct rte_mbuf *pkt, uint64_t high_ctx_qw);
+
static __rte_always_inline void
ci_fill_ctx_desc_tunneling(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)
{
@@ -228,7 +234,7 @@ ci_vtx_avx2(volatile struct ci_tx_desc *txdp,
static __rte_always_inline void
ci_vtx1_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
- enum ci_l2tag_pos qinq_outer_pos, bool ptype_lldp_enabled)
+ enum ci_l2tag_pos qinq_outer_pos, ci_tx_ctx_lldp_fn lldp_check)
{
uint64_t high_ctx_qw = CI_TX_DESC_DTYPE_CTX;
uint64_t low_ctx_qw = 0;
@@ -247,8 +253,8 @@ ci_vtx1_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
low_ctx_qw |= (uint64_t)pkt->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
}
}
- if (IAVF_CHECK_TX_LLDP(pkt, ptype_lldp_enabled))
- high_ctx_qw |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ if (lldp_check != NULL)
+ high_ctx_qw = lldp_check(pkt, high_ctx_qw);
uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA |
((uint64_t)flags << CI_TXD_QW1_CMD_S) |
((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
@@ -266,7 +272,7 @@ static __rte_always_inline void
ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp,
struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
- bool ptype_lldp_enabled)
+ ci_tx_ctx_lldp_fn lldp_check)
{
uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
@@ -300,8 +306,8 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp,
low_ctx_qw1 |= (uint64_t)pkt[1]->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
}
}
- if (IAVF_CHECK_TX_LLDP(pkt[1], ptype_lldp_enabled))
- hi_ctx_qw1 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ if (lldp_check != NULL)
+ hi_ctx_qw1 = lldp_check(pkt[1], hi_ctx_qw1);
if (offload) {
/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
@@ -318,8 +324,8 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp,
low_ctx_qw0 |= (uint64_t)pkt[0]->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
}
}
- if (IAVF_CHECK_TX_LLDP(pkt[0], ptype_lldp_enabled))
- hi_ctx_qw0 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ if (lldp_check != NULL)
+ hi_ctx_qw0 = lldp_check(pkt[0], hi_ctx_qw0);
if (offload) {
ci_tx_vec_offload(pkt[1], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
@@ -338,7 +344,7 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp,
if (nb_pkts)
ci_vtx1_ctx_avx2(txdp, *pkt, flags, offload,
- single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled);
+ single_vlan_pos, qinq_outer_pos, lldp_check);
}
#endif /* __AVX2__ */
@@ -398,7 +404,7 @@ ci_vtx_avx512(volatile struct ci_tx_desc *txdp,
static __rte_always_inline void
ci_vtx1_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
- enum ci_l2tag_pos qinq_outer_pos, bool lldp_enabled)
+ enum ci_l2tag_pos qinq_outer_pos, ci_tx_ctx_lldp_fn lldp_check)
{
uint64_t high_ctx_qw = CI_TX_DESC_DTYPE_CTX;
uint64_t low_ctx_qw = 0;
@@ -417,8 +423,8 @@ ci_vtx1_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,
low_ctx_qw |= (uint64_t)pkt->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
}
}
- if (IAVF_CHECK_TX_LLDP(pkt, lldp_enabled))
- high_ctx_qw |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ if (lldp_check != NULL)
+ high_ctx_qw = lldp_check(pkt, high_ctx_qw);
uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA |
((uint64_t)flags << CI_TXD_QW1_CMD_S) |
((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
@@ -437,7 +443,7 @@ static __rte_always_inline void
ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp,
struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags,
bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
- bool lldp_enabled)
+ ci_tx_ctx_lldp_fn lldp_check)
{
uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
@@ -469,8 +475,8 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp,
low_ctx_qw1 |= (uint64_t)pkt[1]->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
}
}
- if (IAVF_CHECK_TX_LLDP(pkt[1], lldp_enabled))
- hi_ctx_qw1 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ if (lldp_check != NULL)
+ hi_ctx_qw1 = lldp_check(pkt[1], hi_ctx_qw1);
if (offload) {
/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
@@ -487,8 +493,8 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp,
low_ctx_qw0 |= (uint64_t)pkt[0]->vlan_tci << CI_TXD_CTX_QW0_L2TAG2_S;
}
}
- if (IAVF_CHECK_TX_LLDP(pkt[0], lldp_enabled))
- hi_ctx_qw0 |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ if (lldp_check != NULL)
+ hi_ctx_qw0 = lldp_check(pkt[0], hi_ctx_qw0);
if (offload) {
ci_tx_vec_offload(pkt[1], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
@@ -505,7 +511,7 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp,
if (nb_pkts)
ci_vtx1_ctx_avx512(txdp, *pkt, flags, offload,
- single_vlan_pos, qinq_outer_pos, lldp_enabled);
+ single_vlan_pos, qinq_outer_pos, lldp_check);
}
#endif /* __AVX512VL__ */
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
index 1d0492edd9..813611015d 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
@@ -1685,6 +1685,14 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
return nb_pkts;
}
+static __rte_always_inline uint64_t
+iavf_tx_ctx_lldp_check(struct rte_mbuf *pkt, uint64_t high_ctx_qw)
+{
+ if (IAVF_CHECK_TX_LLDP(pkt, true))
+ high_ctx_qw |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ return high_ctx_qw;
+}
+
static __rte_always_inline uint16_t
iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts, bool offload)
@@ -1696,7 +1704,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
/* bit2 is reserved and must be set to 1 according to Spec */
uint64_t flags = IAVF_TX_DESC_CMD_EOP | IAVF_TX_DESC_CMD_ICRC;
uint64_t rs = IAVF_TX_DESC_CMD_RS | flags;
- bool lldp_enabled = txq->lldp_enabled;
+ ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check : NULL;
/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
@@ -1723,10 +1731,10 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
ci_vtx_ctx_avx2(txdp, tx_pkts, nb_mbuf - 1, flags, offload,
- vlan_pos, vlan_pos, lldp_enabled);
+ vlan_pos, vlan_pos, lldp_check);
tx_pkts += (nb_mbuf - 1);
txdp += (n - 2);
- ci_vtx1_ctx_avx2(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx1_ctx_avx2(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check);
nb_commit = (uint16_t)(nb_commit - n);
@@ -1740,7 +1748,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = nb_commit >> 1;
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
- ci_vtx_ctx_avx2(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx_ctx_avx2(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
index 0c738e4882..fe00416660 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
@@ -1900,6 +1900,14 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
return nb_pkts;
}
+static __rte_always_inline uint64_t
+iavf_tx_ctx_lldp_check(struct rte_mbuf *pkt, uint64_t high_ctx_qw)
+{
+ if (IAVF_CHECK_TX_LLDP(pkt, true))
+ high_ctx_qw |= IAVF_TX_CTX_DESC_SWTCH_UPLINK << CI_TXD_QW1_CMD_S;
+ return high_ctx_qw;
+}
+
static __rte_always_inline uint16_t
iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts, bool offload)
@@ -1911,7 +1919,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
/* bit2 is reserved and must be set to 1 according to Spec */
uint64_t flags = CI_TX_DESC_CMD_EOP | CI_TX_DESC_CMD_ICRC;
uint64_t rs = CI_TX_DESC_CMD_RS | flags;
- bool lldp_enabled = txq->lldp_enabled;
+ ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check : NULL;
/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
@@ -1938,11 +1946,10 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
tx_backlog_entry_avx512(txep, tx_pkts, nb_mbuf);
ci_vtx_ctx_avx512(txdp, tx_pkts, nb_mbuf - 1, flags, offload,
- vlan_pos, vlan_pos, lldp_enabled);
+ vlan_pos, vlan_pos, lldp_check);
tx_pkts += (nb_mbuf - 1);
txdp += (n - 2);
- ci_vtx1_ctx_avx512(txdp, *tx_pkts++, rs, offload,
- vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx1_ctx_avx512(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check);
nb_commit = (uint16_t)(nb_commit - n);
@@ -1956,7 +1963,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
nb_mbuf = nb_commit >> 1;
tx_backlog_entry_avx512(txep, tx_pkts, nb_mbuf);
- ci_vtx_ctx_avx512(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled);
+ ci_vtx_ctx_avx512(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 08/13] net/ice: use common descriptor creation functions
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (6 preceding siblings ...)
2026-09-03 17:01 ` [PATCH 07/13] net/intel: use function callback for lldp Bruce Richardson
@ 2026-09-03 17:01 ` Bruce Richardson
2026-09-03 17:01 ` [PATCH 09/13] net/intel: move vector Tx paths to common Bruce Richardson
` (5 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Anatoly Burakov
Update ice driver to use the descriptor creation functions from common
rather than having it maintain its own copies of them.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 73 ++-------------------
drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 64 ++----------------
drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 -----------------
3 files changed, 12 insertions(+), 184 deletions(-)
diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c
index b72f69a47b..68401560ce 100644
--- a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c
+++ b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c
@@ -5,6 +5,7 @@
#include "ice_rxtx_vec_common.h"
#include "../common/rx_vec_x86.h"
+#include "../common/tx_vec_x86.h"
#include <rte_vect.h>
@@ -773,70 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue,
true);
}
-static __rte_always_inline void
-ice_vtx1(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf *pkt, uint64_t flags, bool offload)
-{
- uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA |
- ((uint64_t)flags << CI_TXD_QW1_CMD_S) |
- ((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
- if (offload)
- ice_txd_enable_offload(pkt, &high_qw);
-
- __m128i descriptor = _mm_set_epi64x(high_qw, rte_pktmbuf_iova(pkt));
- _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor);
-}
-
-static __rte_always_inline void
-ice_vtx(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload)
-{
- const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
-
- /* if unaligned on 32-bit boundary, do one to align */
- if (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {
- ice_vtx1(txdp, *pkt, flags, offload);
- nb_pkts--; txdp++; pkt++;
- }
-
- /* do four at a time while possible, in bursts */
- for (; nb_pkts > 3; txdp += 4, pkt += 4, nb_pkts -= 4) {
- uint64_t hi_qw3 = hi_qw_tmpl |
- ((uint64_t)pkt[3]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (offload)
- ice_txd_enable_offload(pkt[3], &hi_qw3);
- uint64_t hi_qw2 = hi_qw_tmpl |
- ((uint64_t)pkt[2]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (offload)
- ice_txd_enable_offload(pkt[2], &hi_qw2);
- uint64_t hi_qw1 = hi_qw_tmpl |
- ((uint64_t)pkt[1]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (offload)
- ice_txd_enable_offload(pkt[1], &hi_qw1);
- uint64_t hi_qw0 = hi_qw_tmpl |
- ((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (offload)
- ice_txd_enable_offload(pkt[0], &hi_qw0);
-
- __m256i desc2_3 =
- _mm256_set_epi64x
- (hi_qw3, rte_pktmbuf_iova(pkt[3]),
- hi_qw2, rte_pktmbuf_iova(pkt[2]));
- __m256i desc0_1 =
- _mm256_set_epi64x
- (hi_qw1, rte_pktmbuf_iova(pkt[1]),
- hi_qw0, rte_pktmbuf_iova(pkt[0]));
- _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3);
- _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1);
- }
-
- /* do any last ones */
- while (nb_pkts) {
- ice_vtx1(txdp, *pkt, flags, offload);
- txdp++; pkt++; nb_pkts--;
- }
-}
-
static __rte_always_inline uint16_t
ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts, bool offload)
@@ -868,11 +805,12 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
if (nb_commit >= n) {
ci_tx_backlog_entry_vec(txep, tx_pkts, n);
- ice_vtx(txdp, tx_pkts, n - 1, flags, offload);
+ ci_vtx_avx2(txdp, tx_pkts, n - 1, flags, offload,
+ CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
tx_pkts += (n - 1);
txdp += (n - 1);
- ice_vtx1(txdp, *tx_pkts++, rs, offload);
+ ci_vtx1(txdp, *tx_pkts++, rs, offload, CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
nb_commit = (uint16_t)(nb_commit - n);
@@ -886,7 +824,8 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit);
- ice_vtx(txdp, tx_pkts, nb_commit, flags, offload);
+ ci_vtx_avx2(txdp, tx_pkts, nb_commit, flags, offload,
+ CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c
index 309ab9fca7..b4695c398e 100644
--- a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c
+++ b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c
@@ -5,6 +5,7 @@
#include "ice_rxtx_vec_common.h"
#include "../common/rx_vec_x86.h"
+#include "../common/tx_vec_x86.h"
#include <rte_vect.h>
@@ -846,61 +847,6 @@ ice_recv_scattered_pkts_vec_avx512_offload(void *rx_queue,
rx_pkts + retval, nb_pkts);
}
-static __rte_always_inline void
-ice_vtx1(volatile struct ci_tx_desc *txdp,
- struct rte_mbuf *pkt, uint64_t flags, bool do_offload)
-{
- uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA |
- ((uint64_t)flags << CI_TXD_QW1_CMD_S) |
- ((uint64_t)pkt->data_len << CI_TXD_QW1_TX_BUF_SZ_S));
-
- if (do_offload)
- ice_txd_enable_offload(pkt, &high_qw);
-
- __m128i descriptor = _mm_set_epi64x(high_qw, rte_pktmbuf_iova(pkt));
- _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor);
-}
-
-static __rte_always_inline void
-ice_vtx(volatile struct ci_tx_desc *txdp, struct rte_mbuf **pkt,
- uint16_t nb_pkts, uint64_t flags, bool do_offload)
-{
- const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
-
- for (; nb_pkts > 3; txdp += 4, pkt += 4, nb_pkts -= 4) {
- uint64_t hi_qw3 = hi_qw_tmpl |
- ((uint64_t)pkt[3]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (do_offload)
- ice_txd_enable_offload(pkt[3], &hi_qw3);
- uint64_t hi_qw2 = hi_qw_tmpl |
- ((uint64_t)pkt[2]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (do_offload)
- ice_txd_enable_offload(pkt[2], &hi_qw2);
- uint64_t hi_qw1 = hi_qw_tmpl |
- ((uint64_t)pkt[1]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (do_offload)
- ice_txd_enable_offload(pkt[1], &hi_qw1);
- uint64_t hi_qw0 = hi_qw_tmpl |
- ((uint64_t)pkt[0]->data_len << CI_TXD_QW1_TX_BUF_SZ_S);
- if (do_offload)
- ice_txd_enable_offload(pkt[0], &hi_qw0);
-
- __m512i desc0_3 =
- _mm512_set_epi64
- (hi_qw3, rte_pktmbuf_iova(pkt[3]),
- hi_qw2, rte_pktmbuf_iova(pkt[2]),
- hi_qw1, rte_pktmbuf_iova(pkt[1]),
- hi_qw0, rte_pktmbuf_iova(pkt[0]));
- _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3);
- }
-
- /* do any last ones */
- while (nb_pkts) {
- ice_vtx1(txdp, *pkt, flags, do_offload);
- txdp++; pkt++; nb_pkts--;
- }
-}
-
static __rte_always_inline uint16_t
ice_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts, bool do_offload)
@@ -933,11 +879,12 @@ ice_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
if (nb_commit >= n) {
ci_tx_backlog_entry_vec(txep, tx_pkts, n);
- ice_vtx(txdp, tx_pkts, n - 1, flags, do_offload);
+ ci_vtx_avx512(txdp, tx_pkts, n - 1, flags, do_offload,
+ CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
tx_pkts += (n - 1);
txdp += (n - 1);
- ice_vtx1(txdp, *tx_pkts++, rs, do_offload);
+ ci_vtx1(txdp, *tx_pkts++, rs, do_offload, CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
nb_commit = (uint16_t)(nb_commit - n);
@@ -951,7 +898,8 @@ ice_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit);
- ice_vtx(txdp, tx_pkts, nb_commit, flags, do_offload);
+ ci_vtx_avx512(txdp, tx_pkts, nb_commit, flags, do_offload,
+ CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
tx_id = (uint16_t)(tx_id + nb_commit);
if (tx_id > txq->tx_next_rs) {
diff --git a/drivers/net/intel/ice/ice_rxtx_vec_common.h b/drivers/net/intel/ice/ice_rxtx_vec_common.h
index 1d83a087cc..53d1eb42a3 100644
--- a/drivers/net/intel/ice/ice_rxtx_vec_common.h
+++ b/drivers/net/intel/ice/ice_rxtx_vec_common.h
@@ -113,63 +113,4 @@ ice_tx_vec_dev_check_default(struct rte_eth_dev *dev)
return ret;
}
-static inline void
-ice_txd_enable_offload(struct rte_mbuf *tx_pkt,
- uint64_t *txd_hi)
-{
- uint64_t ol_flags = tx_pkt->ol_flags;
- uint32_t td_cmd = 0;
- uint32_t td_offset = 0;
-
- /* Tx Checksum Offload */
- /* SET MACLEN */
- td_offset |= (tx_pkt->l2_len >> 1) <<
- CI_TX_DESC_LEN_MACLEN_S;
-
- /* Enable L3 checksum offload */
- if (ol_flags & RTE_MBUF_F_TX_IP_CKSUM) {
- td_cmd |= CI_TX_DESC_CMD_IIPT_IPV4_CSUM;
- td_offset |= (tx_pkt->l3_len >> 2) <<
- CI_TX_DESC_LEN_IPLEN_S;
- } else if (ol_flags & RTE_MBUF_F_TX_IPV4) {
- td_cmd |= CI_TX_DESC_CMD_IIPT_IPV4;
- td_offset |= (tx_pkt->l3_len >> 2) <<
- CI_TX_DESC_LEN_IPLEN_S;
- } else if (ol_flags & RTE_MBUF_F_TX_IPV6) {
- td_cmd |= CI_TX_DESC_CMD_IIPT_IPV6;
- td_offset |= (tx_pkt->l3_len >> 2) <<
- CI_TX_DESC_LEN_IPLEN_S;
- }
-
- /* Enable L4 checksum offloads */
- switch (ol_flags & RTE_MBUF_F_TX_L4_MASK) {
- case RTE_MBUF_F_TX_TCP_CKSUM:
- td_cmd |= CI_TX_DESC_CMD_L4T_EOFT_TCP;
- td_offset |= (sizeof(struct rte_tcp_hdr) >> 2) <<
- CI_TX_DESC_LEN_L4_LEN_S;
- break;
- case RTE_MBUF_F_TX_SCTP_CKSUM:
- td_cmd |= CI_TX_DESC_CMD_L4T_EOFT_SCTP;
- td_offset |= (sizeof(struct rte_sctp_hdr) >> 2) <<
- CI_TX_DESC_LEN_L4_LEN_S;
- break;
- case RTE_MBUF_F_TX_UDP_CKSUM:
- td_cmd |= CI_TX_DESC_CMD_L4T_EOFT_UDP;
- td_offset |= (sizeof(struct rte_udp_hdr) >> 2) <<
- CI_TX_DESC_LEN_L4_LEN_S;
- break;
- default:
- break;
- }
-
- *txd_hi |= ((uint64_t)td_offset) << CI_TXD_QW1_OFFSET_S;
-
- /* Tx VLAN/QINQ insertion Offload */
- if (ol_flags & RTE_MBUF_F_TX_VLAN) {
- td_cmd |= CI_TX_DESC_CMD_IL2TAG1;
- *txd_hi |= ((uint64_t)tx_pkt->vlan_tci << CI_TXD_QW1_L2TAG1_S);
- }
-
- *txd_hi |= ((uint64_t)td_cmd) << CI_TXD_QW1_CMD_S;
-}
#endif
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 09/13] net/intel: move vector Tx paths to common
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (7 preceding siblings ...)
2026-09-03 17:01 ` [PATCH 08/13] net/ice: use common descriptor creation functions Bruce Richardson
@ 2026-09-03 17:01 ` Bruce Richardson
2026-09-03 17:03 ` [PATCH 10/13] net/ice: use common AVX Tx functions Bruce Richardson
` (4 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:01 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Vladimir Medvedkin
The main loops for vector Tx, both avx2 and avx512 are almost identical
across a couple of drivers, so move the iavf copies to common for later
reuse by other drivers.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/common/tx.h | 10 ++
drivers/net/intel/common/tx_vec_x86.h | 128 ++++++++++++++++++
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 75 +---------
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 77 +----------
4 files changed, 148 insertions(+), 142 deletions(-)
diff --git a/drivers/net/intel/common/tx.h b/drivers/net/intel/common/tx.h
index c8fadee712..630df8cb19 100644
--- a/drivers/net/intel/common/tx.h
+++ b/drivers/net/intel/common/tx.h
@@ -9,6 +9,8 @@
#include <rte_mbuf.h>
#include <rte_ethdev.h>
#include <rte_vect.h>
+#include <rte_io.h>
+#include <rte_byteorder.h>
/* Common TX Descriptor QW1 Field Definitions */
#define CI_TXD_QW1_DTYPE_S 0
@@ -273,6 +275,14 @@ ci_tx_backlog_entry_vec(struct ci_tx_entry_vec *txep, struct rte_mbuf **tx_pkts,
txep[i].mbuf = tx_pkts[i];
}
+
+/* Write the Tx tail register, byte-swapped for hardware regardless of host endianness. */
+static __rte_always_inline void
+ci_tx_qtx_tail_write(struct ci_tx_queue *txq, uint16_t tx_id)
+{
+ rte_write32_wc(rte_cpu_to_le_32((uint32_t)tx_id), txq->qtx_tail);
+}
+
#define IETH_VPMD_TX_MAX_FREE_BUF 64
typedef int (*ci_desc_done_fn)(struct ci_tx_queue *txq, uint16_t idx);
diff --git a/drivers/net/intel/common/tx_vec_x86.h b/drivers/net/intel/common/tx_vec_x86.h
index f769765cba..87386df8cd 100644
--- a/drivers/net/intel/common/tx_vec_x86.h
+++ b/drivers/net/intel/common/tx_vec_x86.h
@@ -347,6 +347,69 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp,
single_vlan_pos, qinq_outer_pos, lldp_check);
}
+static __rte_always_inline uint16_t
+ci_xmit_fixed_burst_vec_avx2(struct ci_tx_queue *txq, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts, bool offload,
+ enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
+{
+ volatile struct ci_tx_desc *txdp;
+ struct ci_tx_entry_vec *txep;
+ uint16_t n, nb_commit, tx_id;
+ uint64_t flags = CI_TX_DESC_CMD_DEFAULT;
+ uint64_t rs = CI_TX_DESC_CMD_RS | flags;
+
+ if (txq->nb_tx_free < txq->tx_free_thresh)
+ ci_tx_free_bufs_vec(txq, ci_tx_desc_done_simple, false);
+
+ nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts);
+ if (unlikely(nb_pkts == 0))
+ return 0;
+ nb_commit = nb_pkts;
+
+ tx_id = txq->tx_tail;
+ txdp = &txq->ci_tx_ring[tx_id];
+ txep = &txq->sw_ring_vec[tx_id];
+
+ txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_pkts);
+
+ n = (uint16_t)(txq->nb_tx_desc - tx_id);
+ if (nb_commit >= n) {
+ ci_tx_backlog_entry_vec(txep, tx_pkts, n);
+
+ ci_vtx_avx2(txdp, tx_pkts, n - 1, flags, offload, single_vlan_pos, qinq_outer_pos);
+ tx_pkts += (n - 1);
+ txdp += (n - 1);
+
+ ci_vtx1(txdp, *tx_pkts++, rs, offload, single_vlan_pos, qinq_outer_pos);
+
+ nb_commit = (uint16_t)(nb_commit - n);
+
+ tx_id = 0;
+ txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
+
+ /* avoid reach the end of ring */
+ txdp = &txq->ci_tx_ring[tx_id];
+ txep = &txq->sw_ring_vec[tx_id];
+ }
+
+ ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit);
+
+ ci_vtx_avx2(txdp, tx_pkts, nb_commit, flags, offload, single_vlan_pos, qinq_outer_pos);
+
+ tx_id = (uint16_t)(tx_id + nb_commit);
+ if (tx_id > txq->tx_next_rs) {
+ txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
+ rte_cpu_to_le_64(((uint64_t)CI_TX_DESC_CMD_RS) << CI_TXD_QW1_CMD_S);
+ txq->tx_next_rs = (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
+ }
+
+ txq->tx_tail = tx_id;
+
+ ci_tx_qtx_tail_write(txq, tx_id);
+
+ return nb_pkts;
+}
+
#endif /* __AVX2__ */
#ifdef __AVX512VL__
@@ -514,6 +577,71 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp,
single_vlan_pos, qinq_outer_pos, lldp_check);
}
+static __rte_always_inline uint16_t
+ci_xmit_fixed_burst_vec_avx512(struct ci_tx_queue *txq, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts, bool offload,
+ enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos)
+{
+ volatile struct ci_tx_desc *txdp;
+ struct ci_tx_entry_vec *txep;
+ uint16_t n, nb_commit, tx_id;
+ /* bit2 is reserved and must be set to 1 according to Spec */
+ uint64_t flags = CI_TX_DESC_CMD_DEFAULT;
+ uint64_t rs = CI_TX_DESC_CMD_RS | flags;
+
+ if (txq->nb_tx_free < txq->tx_free_thresh)
+ ci_tx_free_bufs_vec(txq, ci_tx_desc_done_simple, false);
+
+ nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts);
+ if (unlikely(nb_pkts == 0))
+ return 0;
+ nb_commit = nb_pkts;
+
+ tx_id = txq->tx_tail;
+ txdp = &txq->ci_tx_ring[tx_id];
+ txep = &txq->sw_ring_vec[tx_id];
+
+ txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_pkts);
+
+ n = (uint16_t)(txq->nb_tx_desc - tx_id);
+ if (nb_commit >= n) {
+ ci_tx_backlog_entry_vec(txep, tx_pkts, n);
+
+ ci_vtx_avx512(txdp, tx_pkts, n - 1, flags, offload,
+ single_vlan_pos, qinq_outer_pos);
+ tx_pkts += (n - 1);
+ txdp += (n - 1);
+
+ ci_vtx1(txdp, *tx_pkts++, rs, offload, single_vlan_pos, qinq_outer_pos);
+
+ nb_commit = (uint16_t)(nb_commit - n);
+
+ tx_id = 0;
+ txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
+
+ /* avoid reach the end of ring */
+ txdp = &txq->ci_tx_ring[tx_id];
+ txep = &txq->sw_ring_vec[tx_id];
+ }
+
+ ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit);
+
+ ci_vtx_avx512(txdp, tx_pkts, nb_commit, flags, offload, single_vlan_pos, qinq_outer_pos);
+
+ tx_id = (uint16_t)(tx_id + nb_commit);
+ if (tx_id > txq->tx_next_rs) {
+ txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
+ rte_cpu_to_le_64(((uint64_t)CI_TX_DESC_CMD_RS) << CI_TXD_QW1_CMD_S);
+ txq->tx_next_rs = (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
+ }
+
+ txq->tx_tail = tx_id;
+
+ ci_tx_qtx_tail_write(txq, tx_id);
+
+ return nb_pkts;
+}
+
#endif /* __AVX512VL__ */
#endif /* _COMMON_INTEL_TX_VEC_X86_H_ */
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
index 813611015d..1289581038 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
@@ -1617,74 +1617,6 @@ iavf_recv_scattered_pkts_vec_avx2_flex_rxd_offload(void *rx_queue,
}
-static __rte_always_inline uint16_t
-iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
- uint16_t nb_pkts, bool offload)
-{
- struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
- volatile struct ci_tx_desc *txdp;
- struct ci_tx_entry_vec *txep;
- uint16_t n, nb_commit, tx_id;
- /* bit2 is reserved and must be set to 1 according to Spec */
- uint64_t flags = CI_TX_DESC_CMD_EOP | CI_TX_DESC_CMD_ICRC;
- uint64_t rs = CI_TX_DESC_CMD_RS | flags;
- /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
- enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
- CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
-
- if (txq->nb_tx_free < txq->tx_free_thresh)
- ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, false);
-
- nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts);
- if (unlikely(nb_pkts == 0))
- return 0;
- nb_commit = nb_pkts;
-
- tx_id = txq->tx_tail;
- txdp = &txq->ci_tx_ring[tx_id];
- txep = &txq->sw_ring_vec[tx_id];
-
- txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_pkts);
-
- n = (uint16_t)(txq->nb_tx_desc - tx_id);
- if (nb_commit >= n) {
- ci_tx_backlog_entry_vec(txep, tx_pkts, n);
-
- ci_vtx_avx2(txdp, tx_pkts, n - 1, flags, offload, vlan_pos, vlan_pos);
- tx_pkts += (n - 1);
- txdp += (n - 1);
-
- ci_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos);
-
- nb_commit = (uint16_t)(nb_commit - n);
-
- tx_id = 0;
- txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
-
- /* avoid reach the end of ring */
- txdp = &txq->ci_tx_ring[tx_id];
- txep = &txq->sw_ring_vec[tx_id];
- }
-
- ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit);
-
- ci_vtx_avx2(txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos);
-
- tx_id = (uint16_t)(tx_id + nb_commit);
- if (tx_id > txq->tx_next_rs) {
- txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
- rte_cpu_to_le_64(((uint64_t)CI_TX_DESC_CMD_RS) << CI_TXD_QW1_CMD_S);
- txq->tx_next_rs =
- (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
- }
-
- txq->tx_tail = tx_id;
-
- IAVF_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail);
-
- return nb_pkts;
-}
-
static __rte_always_inline uint64_t
iavf_tx_ctx_lldp_check(struct rte_mbuf *pkt, uint64_t high_ctx_qw)
{
@@ -1809,14 +1741,17 @@ iavf_xmit_pkts_vec_avx2_common(void *tx_queue, struct rte_mbuf **tx_pkts,
{
uint16_t nb_tx = 0;
struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
+ /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
+ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
+ CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
while (nb_pkts) {
uint16_t ret, num;
/* cross rs_thresh boundary is not allowed */
num = (uint16_t)RTE_MIN(nb_pkts, txq->tx_rs_thresh);
- ret = iavf_xmit_fixed_burst_vec_avx2(tx_queue, &tx_pkts[nb_tx],
- num, offload);
+ ret = ci_xmit_fixed_burst_vec_avx2(txq, &tx_pkts[nb_tx], num,
+ offload, vlan_pos, vlan_pos);
nb_tx += ret;
nb_pkts -= ret;
if (ret < num)
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
index fe00416660..9f773f226b 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
@@ -1830,76 +1830,6 @@ tx_backlog_entry_avx512(struct ci_tx_entry_vec *txep,
#define IAVF_TX_LEN_MASK 0xAA
#define IAVF_TX_OFF_MASK 0x55
-static __rte_always_inline uint16_t
-iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
- uint16_t nb_pkts, bool offload)
-{
- struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
- volatile struct ci_tx_desc *txdp;
- struct ci_tx_entry_vec *txep;
- uint16_t n, nb_commit, tx_id;
- /* bit2 is reserved and must be set to 1 according to Spec */
- uint64_t flags = CI_TX_DESC_CMD_EOP | CI_TX_DESC_CMD_ICRC;
- uint64_t rs = CI_TX_DESC_CMD_RS | flags;
- /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
- enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
- CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
-
- if (txq->nb_tx_free < txq->tx_free_thresh)
- ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, false);
-
- nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts);
- if (unlikely(nb_pkts == 0))
- return 0;
- nb_commit = nb_pkts;
-
- tx_id = txq->tx_tail;
- txdp = &txq->ci_tx_ring[tx_id];
- txep = (void *)txq->sw_ring;
- txep += tx_id;
-
- txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_pkts);
-
- n = (uint16_t)(txq->nb_tx_desc - tx_id);
- if (nb_commit >= n) {
- tx_backlog_entry_avx512(txep, tx_pkts, n);
-
- ci_vtx_avx512(txdp, tx_pkts, n - 1, flags, offload, vlan_pos, vlan_pos);
- tx_pkts += (n - 1);
- txdp += (n - 1);
-
- ci_vtx1(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos);
-
- nb_commit = (uint16_t)(nb_commit - n);
-
- tx_id = 0;
- txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
-
- /* avoid reach the end of ring */
- txdp = &txq->ci_tx_ring[tx_id];
- txep = (void *)txq->sw_ring;
- txep += tx_id;
- }
-
- tx_backlog_entry_avx512(txep, tx_pkts, nb_commit);
-
- ci_vtx_avx512(txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos);
-
- tx_id = (uint16_t)(tx_id + nb_commit);
- if (tx_id > txq->tx_next_rs) {
- txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
- rte_cpu_to_le_64(((uint64_t)CI_TX_DESC_CMD_RS) << CI_TXD_QW1_CMD_S);
- txq->tx_next_rs =
- (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
- }
-
- txq->tx_tail = tx_id;
-
- IAVF_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail);
-
- return nb_pkts;
-}
-
static __rte_always_inline uint64_t
iavf_tx_ctx_lldp_check(struct rte_mbuf *pkt, uint64_t high_ctx_qw)
{
@@ -1985,14 +1915,17 @@ iavf_xmit_pkts_vec_avx512_cmn(void *tx_queue, struct rte_mbuf **tx_pkts,
{
uint16_t nb_tx = 0;
struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
+ /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
+ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
+ CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
while (nb_pkts) {
uint16_t ret, num;
/* cross rs_thresh boundary is not allowed */
num = (uint16_t)RTE_MIN(nb_pkts, txq->tx_rs_thresh);
- ret = iavf_xmit_fixed_burst_vec_avx512(tx_queue, &tx_pkts[nb_tx],
- num, offload);
+ ret = ci_xmit_fixed_burst_vec_avx512(txq, &tx_pkts[nb_tx], num,
+ offload, vlan_pos, vlan_pos);
nb_tx += ret;
nb_pkts -= ret;
if (ret < num)
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 10/13] net/ice: use common AVX Tx functions
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (8 preceding siblings ...)
2026-09-03 17:01 ` [PATCH 09/13] net/intel: move vector Tx paths to common Bruce Richardson
@ 2026-09-03 17:03 ` Bruce Richardson
2026-09-03 17:04 ` [PATCH 11/13] net/intel: add common vector Tx fns with context handling Bruce Richardson
` (3 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:03 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Anatoly Burakov
Use the newly created common Tx functions in the vector path.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 72 +-------------
drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 102 +++-----------------
2 files changed, 14 insertions(+), 160 deletions(-)
diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c
index 68401560ce..00b5d51aab 100644
--- a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c
+++ b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c
@@ -774,74 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue,
true);
}
-static __rte_always_inline uint16_t
-ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
- uint16_t nb_pkts, bool offload)
-{
- struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
- volatile struct ci_tx_desc *txdp;
- struct ci_tx_entry_vec *txep;
- uint16_t n, nb_commit, tx_id;
- uint64_t flags = CI_TX_DESC_CMD_DEFAULT;
- uint64_t rs = CI_TX_DESC_CMD_RS | CI_TX_DESC_CMD_DEFAULT;
-
- /* cross rx_thresh boundary is not allowed */
- nb_pkts = RTE_MIN(nb_pkts, txq->tx_rs_thresh);
-
- if (txq->nb_tx_free < txq->tx_free_thresh)
- ci_tx_free_bufs_vec(txq, ice_tx_desc_done, false);
-
- nb_commit = nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts);
- if (unlikely(nb_pkts == 0))
- return 0;
-
- tx_id = txq->tx_tail;
- txdp = &txq->ci_tx_ring[tx_id];
- txep = &txq->sw_ring_vec[tx_id];
-
- txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_pkts);
-
- n = (uint16_t)(txq->nb_tx_desc - tx_id);
- if (nb_commit >= n) {
- ci_tx_backlog_entry_vec(txep, tx_pkts, n);
-
- ci_vtx_avx2(txdp, tx_pkts, n - 1, flags, offload,
- CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
- tx_pkts += (n - 1);
- txdp += (n - 1);
-
- ci_vtx1(txdp, *tx_pkts++, rs, offload, CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
-
- nb_commit = (uint16_t)(nb_commit - n);
-
- tx_id = 0;
- txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
-
- /* avoid reach the end of ring */
- txdp = &txq->ci_tx_ring[tx_id];
- txep = &txq->sw_ring_vec[tx_id];
- }
-
- ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit);
-
- ci_vtx_avx2(txdp, tx_pkts, nb_commit, flags, offload,
- CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
-
- tx_id = (uint16_t)(tx_id + nb_commit);
- if (tx_id > txq->tx_next_rs) {
- txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
- rte_cpu_to_le_64(((uint64_t)CI_TX_DESC_CMD_RS) << CI_TXD_QW1_CMD_S);
- txq->tx_next_rs =
- (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
- }
-
- txq->tx_tail = tx_id;
-
- ICE_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail);
-
- return nb_pkts;
-}
-
static __rte_always_inline uint16_t
ice_xmit_pkts_vec_avx2_common(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts, bool offload)
@@ -853,8 +785,8 @@ ice_xmit_pkts_vec_avx2_common(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t ret, num;
num = (uint16_t)RTE_MIN(nb_pkts, txq->tx_rs_thresh);
- ret = ice_xmit_fixed_burst_vec_avx2(tx_queue, &tx_pkts[nb_tx],
- num, offload);
+ ret = ci_xmit_fixed_burst_vec_avx2(tx_queue, &tx_pkts[nb_tx], num,
+ offload, CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
nb_tx += ret;
nb_pkts -= ret;
if (ret < num)
diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c
index b4695c398e..d8ede5cb42 100644
--- a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c
+++ b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c
@@ -848,77 +848,8 @@ ice_recv_scattered_pkts_vec_avx512_offload(void *rx_queue,
}
static __rte_always_inline uint16_t
-ice_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
- uint16_t nb_pkts, bool do_offload)
-{
- struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
- volatile struct ci_tx_desc *txdp;
- struct ci_tx_entry_vec *txep;
- uint16_t n, nb_commit, tx_id;
- uint64_t flags = CI_TX_DESC_CMD_DEFAULT;
- uint64_t rs = CI_TX_DESC_CMD_RS | CI_TX_DESC_CMD_DEFAULT;
-
- /* cross rx_thresh boundary is not allowed */
- nb_pkts = RTE_MIN(nb_pkts, txq->tx_rs_thresh);
-
- if (txq->nb_tx_free < txq->tx_free_thresh)
- ci_tx_free_bufs_vec(txq, ice_tx_desc_done, false);
-
- nb_commit = nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts);
- if (unlikely(nb_pkts == 0))
- return 0;
-
- tx_id = txq->tx_tail;
- txdp = &txq->ci_tx_ring[tx_id];
- txep = (void *)txq->sw_ring;
- txep += tx_id;
-
- txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_pkts);
-
- n = (uint16_t)(txq->nb_tx_desc - tx_id);
- if (nb_commit >= n) {
- ci_tx_backlog_entry_vec(txep, tx_pkts, n);
-
- ci_vtx_avx512(txdp, tx_pkts, n - 1, flags, do_offload,
- CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
- tx_pkts += (n - 1);
- txdp += (n - 1);
-
- ci_vtx1(txdp, *tx_pkts++, rs, do_offload, CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
-
- nb_commit = (uint16_t)(nb_commit - n);
-
- tx_id = 0;
- txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
-
- /* avoid reach the end of ring */
- txdp = txq->ci_tx_ring;
- txep = (void *)txq->sw_ring;
- }
-
- ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit);
-
- ci_vtx_avx512(txdp, tx_pkts, nb_commit, flags, do_offload,
- CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
-
- tx_id = (uint16_t)(tx_id + nb_commit);
- if (tx_id > txq->tx_next_rs) {
- txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
- rte_cpu_to_le_64(((uint64_t)CI_TX_DESC_CMD_RS) << CI_TXD_QW1_CMD_S);
- txq->tx_next_rs =
- (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
- }
-
- txq->tx_tail = tx_id;
-
- ICE_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail);
-
- return nb_pkts;
-}
-
-uint16_t
-ice_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
- uint16_t nb_pkts)
+ice_xmit_pkts_vec_avx512_common(void *tx_queue, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts, bool offload)
{
uint16_t nb_tx = 0;
struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
@@ -927,8 +858,8 @@ ice_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t ret, num;
num = (uint16_t)RTE_MIN(nb_pkts, txq->tx_rs_thresh);
- ret = ice_xmit_fixed_burst_vec_avx512(tx_queue,
- &tx_pkts[nb_tx], num, false);
+ ret = ci_xmit_fixed_burst_vec_avx512(tx_queue, &tx_pkts[nb_tx], num,
+ offload, CI_TAG_IN_DATA_DESC, CI_TAG_IN_DATA_DESC);
nb_tx += ret;
nb_pkts -= ret;
if (ret < num)
@@ -938,25 +869,16 @@ ice_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
return nb_tx;
}
+uint16_t
+ice_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts)
+{
+ return ice_xmit_pkts_vec_avx512_common(tx_queue, tx_pkts, nb_pkts, false);
+}
+
uint16_t
ice_xmit_pkts_vec_avx512_offload(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts)
{
- uint16_t nb_tx = 0;
- struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
-
- while (nb_pkts) {
- uint16_t ret, num;
-
- num = (uint16_t)RTE_MIN(nb_pkts, txq->tx_rs_thresh);
- ret = ice_xmit_fixed_burst_vec_avx512(tx_queue,
- &tx_pkts[nb_tx], num, true);
-
- nb_tx += ret;
- nb_pkts -= ret;
- if (ret < num)
- break;
- }
-
- return nb_tx;
+ return ice_xmit_pkts_vec_avx512_common(tx_queue, tx_pkts, nb_pkts, true);
}
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 11/13] net/intel: add common vector Tx fns with context handling
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (9 preceding siblings ...)
2026-09-03 17:03 ` [PATCH 10/13] net/ice: use common AVX Tx functions Bruce Richardson
@ 2026-09-03 17:04 ` Bruce Richardson
2026-09-03 17:04 ` [PATCH 12/13] net/intel: improve Tx path selection logic Bruce Richardson
` (2 subsequent siblings)
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:04 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Vladimir Medvedkin
The iavf vector driver has support for more Tx offloads because it has
Tx functions which handle transmit context descriptors. Move those
context-handling functions to the common folder for reuse by other
drivers.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/common/tx_vec_x86.h | 136 ++++++++++++++++++
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 78 +---------
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 89 +-----------
3 files changed, 147 insertions(+), 156 deletions(-)
diff --git a/drivers/net/intel/common/tx_vec_x86.h b/drivers/net/intel/common/tx_vec_x86.h
index 87386df8cd..f361137d23 100644
--- a/drivers/net/intel/common/tx_vec_x86.h
+++ b/drivers/net/intel/common/tx_vec_x86.h
@@ -410,6 +410,74 @@ ci_xmit_fixed_burst_vec_avx2(struct ci_tx_queue *txq, struct rte_mbuf **tx_pkts,
return nb_pkts;
}
+static __rte_always_inline uint16_t
+ci_xmit_fixed_burst_vec_ctx_avx2(struct ci_tx_queue *txq, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts, bool offload,
+ enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
+ ci_tx_ctx_lldp_fn lldp_check)
+{
+ volatile struct ci_tx_desc *txdp;
+ struct ci_tx_entry_vec *txep;
+ uint16_t n, nb_commit, nb_mbuf, tx_id;
+ uint64_t flags = CI_TX_DESC_CMD_DEFAULT;
+ uint64_t rs = CI_TX_DESC_CMD_RS | flags;
+
+ if (txq->nb_tx_free < txq->tx_free_thresh)
+ ci_tx_free_bufs_vec(txq, ci_tx_desc_done_simple, true);
+
+ nb_commit = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts << 1);
+ nb_commit &= 0xFFFE;
+ if (unlikely(nb_commit == 0))
+ return 0;
+
+ nb_pkts = nb_commit >> 1;
+ tx_id = txq->tx_tail;
+ txdp = &txq->ci_tx_ring[tx_id];
+ txep = (void *)txq->sw_ring;
+ txep += (tx_id >> 1);
+
+ txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_commit);
+ n = (uint16_t)(txq->nb_tx_desc - tx_id);
+
+ if (n != 0 && nb_commit >= n) {
+ nb_mbuf = n >> 1;
+ ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
+
+ ci_vtx_ctx_avx2(txdp, tx_pkts, nb_mbuf - 1, flags, offload,
+ single_vlan_pos, qinq_outer_pos, lldp_check);
+ tx_pkts += (nb_mbuf - 1);
+ txdp += (n - 2);
+ ci_vtx1_ctx_avx2(txdp, *tx_pkts++, rs, offload,
+ single_vlan_pos, qinq_outer_pos, lldp_check);
+
+ nb_commit = (uint16_t)(nb_commit - n);
+
+ txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
+ tx_id = 0;
+ /* avoid reach the end of ring */
+ txdp = txq->ci_tx_ring;
+ txep = (void *)txq->sw_ring;
+ }
+
+ nb_mbuf = nb_commit >> 1;
+ ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
+
+ ci_vtx_ctx_avx2(txdp, tx_pkts, nb_mbuf, flags, offload,
+ single_vlan_pos, qinq_outer_pos, lldp_check);
+ tx_id = (uint16_t)(tx_id + nb_commit);
+
+ if (tx_id > txq->tx_next_rs) {
+ txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
+ rte_cpu_to_le_64(((uint64_t)CI_TX_DESC_CMD_RS) << CI_TXD_QW1_CMD_S);
+ txq->tx_next_rs = (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
+ }
+
+ txq->tx_tail = tx_id;
+
+ ci_tx_qtx_tail_write(txq, tx_id);
+ return nb_pkts;
+}
+
#endif /* __AVX2__ */
#ifdef __AVX512VL__
@@ -642,6 +710,74 @@ ci_xmit_fixed_burst_vec_avx512(struct ci_tx_queue *txq, struct rte_mbuf **tx_pkt
return nb_pkts;
}
+static __rte_always_inline uint16_t
+ci_xmit_fixed_burst_vec_ctx_avx512(struct ci_tx_queue *txq, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts, bool offload,
+ enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos,
+ ci_tx_ctx_lldp_fn lldp_check)
+{
+ volatile struct ci_tx_desc *txdp;
+ struct ci_tx_entry_vec *txep;
+ uint16_t n, nb_commit, nb_mbuf, tx_id;
+ uint64_t flags = CI_TX_DESC_CMD_DEFAULT;
+ uint64_t rs = CI_TX_DESC_CMD_RS | flags;
+
+ if (txq->nb_tx_free < txq->tx_free_thresh)
+ ci_tx_free_bufs_vec(txq, ci_tx_desc_done_simple, true);
+
+ nb_commit = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts << 1);
+ nb_commit &= 0xFFFE;
+ if (unlikely(nb_commit == 0))
+ return 0;
+
+ nb_pkts = nb_commit >> 1;
+ tx_id = txq->tx_tail;
+ txdp = &txq->ci_tx_ring[tx_id];
+ txep = (void *)txq->sw_ring;
+ txep += (tx_id >> 1);
+
+ txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_commit);
+ n = (uint16_t)(txq->nb_tx_desc - tx_id);
+
+ if (n != 0 && nb_commit >= n) {
+ nb_mbuf = n >> 1;
+ ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
+
+ ci_vtx_ctx_avx512(txdp, tx_pkts, nb_mbuf - 1, flags, offload,
+ single_vlan_pos, qinq_outer_pos, lldp_check);
+ tx_pkts += (nb_mbuf - 1);
+ txdp += (n - 2);
+ ci_vtx1_ctx_avx512(txdp, *tx_pkts++, rs, offload,
+ single_vlan_pos, qinq_outer_pos, lldp_check);
+
+ nb_commit = (uint16_t)(nb_commit - n);
+
+ txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
+ tx_id = 0;
+ /* avoid reach the end of ring */
+ txdp = txq->ci_tx_ring;
+ txep = (void *)txq->sw_ring;
+ }
+
+ nb_mbuf = nb_commit >> 1;
+ ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
+
+ ci_vtx_ctx_avx512(txdp, tx_pkts, nb_mbuf, flags, offload,
+ single_vlan_pos, qinq_outer_pos, lldp_check);
+ tx_id = (uint16_t)(tx_id + nb_commit);
+
+ if (tx_id > txq->tx_next_rs) {
+ txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
+ rte_cpu_to_le_64(((uint64_t)CI_TX_DESC_CMD_RS) << CI_TXD_QW1_CMD_S);
+ txq->tx_next_rs = (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
+ }
+
+ txq->tx_tail = tx_id;
+
+ ci_tx_qtx_tail_write(txq, tx_id);
+ return nb_pkts;
+}
+
#endif /* __AVX512VL__ */
#endif /* _COMMON_INTEL_TX_VEC_X86_H_ */
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
index 1289581038..833d553343 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c
@@ -1626,92 +1626,24 @@ iavf_tx_ctx_lldp_check(struct rte_mbuf *pkt, uint64_t high_ctx_qw)
}
static __rte_always_inline uint16_t
-iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
- uint16_t nb_pkts, bool offload)
+iavf_xmit_pkts_vec_avx2_ctx_cmn(void *tx_queue, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts, bool offload)
{
+ uint16_t nb_tx = 0;
struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
- volatile struct ci_tx_desc *txdp;
- struct ci_tx_entry_vec *txep;
- uint16_t n, nb_commit, nb_mbuf, tx_id;
- /* bit2 is reserved and must be set to 1 according to Spec */
- uint64_t flags = IAVF_TX_DESC_CMD_EOP | IAVF_TX_DESC_CMD_ICRC;
- uint64_t rs = IAVF_TX_DESC_CMD_RS | flags;
ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check : NULL;
/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
- if (txq->nb_tx_free < txq->tx_free_thresh)
- ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, true);
-
- nb_commit = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts << 1);
- nb_commit &= 0xFFFE;
- if (unlikely(nb_commit == 0))
- return 0;
-
- nb_pkts = nb_commit >> 1;
- tx_id = txq->tx_tail;
- txdp = &txq->ci_tx_ring[tx_id];
- txep = (void *)txq->sw_ring;
- txep += (tx_id >> 1);
-
- txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_commit);
- n = (uint16_t)(txq->nb_tx_desc - tx_id);
-
- if (n != 0 && nb_commit >= n) {
- nb_mbuf = n >> 1;
- ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
-
- ci_vtx_ctx_avx2(txdp, tx_pkts, nb_mbuf - 1, flags, offload,
- vlan_pos, vlan_pos, lldp_check);
- tx_pkts += (nb_mbuf - 1);
- txdp += (n - 2);
- ci_vtx1_ctx_avx2(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check);
-
- nb_commit = (uint16_t)(nb_commit - n);
-
- txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
- tx_id = 0;
- /* avoid reach the end of ring */
- txdp = txq->ci_tx_ring;
- txep = (void *)txq->sw_ring;
- }
-
- nb_mbuf = nb_commit >> 1;
- ci_tx_backlog_entry_vec(txep, tx_pkts, nb_mbuf);
-
- ci_vtx_ctx_avx2(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check);
- tx_id = (uint16_t)(tx_id + nb_commit);
-
- if (tx_id > txq->tx_next_rs) {
- txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
- rte_cpu_to_le_64(((uint64_t)IAVF_TX_DESC_CMD_RS) <<
- IAVF_TXD_QW1_CMD_SHIFT);
- txq->tx_next_rs =
- (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
- }
-
- txq->tx_tail = tx_id;
-
- IAVF_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail);
- return nb_pkts;
-}
-
-static __rte_always_inline uint16_t
-iavf_xmit_pkts_vec_avx2_ctx_cmn(void *tx_queue, struct rte_mbuf **tx_pkts,
- uint16_t nb_pkts, bool offload)
-{
- uint16_t nb_tx = 0;
- struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
-
while (nb_pkts) {
uint16_t ret, num;
/* cross rs_thresh boundary is not allowed */
num = (uint16_t)RTE_MIN(nb_pkts << 1, txq->tx_rs_thresh);
num = num >> 1;
- ret = iavf_xmit_fixed_burst_vec_avx2_ctx(tx_queue, &tx_pkts[nb_tx],
- num, offload);
+ ret = ci_xmit_fixed_burst_vec_ctx_avx2(txq, &tx_pkts[nb_tx], num,
+ offload, vlan_pos, vlan_pos, lldp_check);
nb_tx += ret;
nb_pkts -= ret;
if (ret < num)
diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
index 9f773f226b..c5aa860f81 100644
--- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
+++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c
@@ -1817,16 +1817,6 @@ iavf_recv_scattered_pkts_vec_avx512_flex_rxd_offload(void *rx_queue,
true);
}
-static __rte_always_inline void
-tx_backlog_entry_avx512(struct ci_tx_entry_vec *txep,
- struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
-{
- int i;
-
- for (i = 0; i < (int)nb_pkts; ++i)
- txep[i].mbuf = tx_pkts[i];
-}
-
#define IAVF_TX_LEN_MASK 0xAA
#define IAVF_TX_OFF_MASK 0x55
@@ -1838,77 +1828,6 @@ iavf_tx_ctx_lldp_check(struct rte_mbuf *pkt, uint64_t high_ctx_qw)
return high_ctx_qw;
}
-static __rte_always_inline uint16_t
-iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,
- uint16_t nb_pkts, bool offload)
-{
- struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
- volatile struct ci_tx_desc *txdp;
- struct ci_tx_entry_vec *txep;
- uint16_t n, nb_commit, nb_mbuf, tx_id;
- /* bit2 is reserved and must be set to 1 according to Spec */
- uint64_t flags = CI_TX_DESC_CMD_EOP | CI_TX_DESC_CMD_ICRC;
- uint64_t rs = CI_TX_DESC_CMD_RS | flags;
- ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check : NULL;
- /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
- enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
- CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
-
- if (txq->nb_tx_free < txq->tx_free_thresh)
- ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, true);
-
- nb_commit = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts << 1);
- nb_commit &= 0xFFFE;
- if (unlikely(nb_commit == 0))
- return 0;
-
- nb_pkts = nb_commit >> 1;
- tx_id = txq->tx_tail;
- txdp = &txq->ci_tx_ring[tx_id];
- txep = (void *)txq->sw_ring;
- txep += (tx_id >> 1);
-
- txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_commit);
- n = (uint16_t)(txq->nb_tx_desc - tx_id);
-
- if (n != 0 && nb_commit >= n) {
- nb_mbuf = n >> 1;
- tx_backlog_entry_avx512(txep, tx_pkts, nb_mbuf);
-
- ci_vtx_ctx_avx512(txdp, tx_pkts, nb_mbuf - 1, flags, offload,
- vlan_pos, vlan_pos, lldp_check);
- tx_pkts += (nb_mbuf - 1);
- txdp += (n - 2);
- ci_vtx1_ctx_avx512(txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check);
-
- nb_commit = (uint16_t)(nb_commit - n);
-
- txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1);
- tx_id = 0;
- /* avoid reach the end of ring */
- txdp = txq->ci_tx_ring;
- txep = (void *)txq->sw_ring;
- }
-
- nb_mbuf = nb_commit >> 1;
- tx_backlog_entry_avx512(txep, tx_pkts, nb_mbuf);
-
- ci_vtx_ctx_avx512(txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check);
- tx_id = (uint16_t)(tx_id + nb_commit);
-
- if (tx_id > txq->tx_next_rs) {
- txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |=
- rte_cpu_to_le_64(((uint64_t)CI_TX_DESC_CMD_RS) << CI_TXD_QW1_CMD_S);
- txq->tx_next_rs =
- (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh);
- }
-
- txq->tx_tail = tx_id;
-
- IAVF_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail);
- return nb_pkts;
-}
-
static __rte_always_inline uint16_t
iavf_xmit_pkts_vec_avx512_cmn(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts, bool offload)
@@ -1955,6 +1874,10 @@ iavf_xmit_pkts_vec_avx512_ctx_cmn(void *tx_queue, struct rte_mbuf **tx_pkts,
{
uint16_t nb_tx = 0;
struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
+ ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check : NULL;
+ /* vlan_flag gives both the single-VLAN and the QinQ outer tag position */
+ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag & IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) ?
+ CI_TAG_IN_DATA_DESC : CI_TAG_IN_CTX_DESC;
while (nb_pkts) {
uint16_t ret, num;
@@ -1962,8 +1885,8 @@ iavf_xmit_pkts_vec_avx512_ctx_cmn(void *tx_queue, struct rte_mbuf **tx_pkts,
/* cross rs_thresh boundary is not allowed */
num = (uint16_t)RTE_MIN(nb_pkts << 1, txq->tx_rs_thresh);
num = num >> 1;
- ret = iavf_xmit_fixed_burst_vec_avx512_ctx(tx_queue, &tx_pkts[nb_tx],
- num, offload);
+ ret = ci_xmit_fixed_burst_vec_ctx_avx512(txq, &tx_pkts[nb_tx], num,
+ offload, vlan_pos, vlan_pos, lldp_check);
nb_tx += ret;
nb_pkts -= ret;
if (ret < num)
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 12/13] net/intel: improve Tx path selection logic
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (10 preceding siblings ...)
2026-09-03 17:04 ` [PATCH 11/13] net/intel: add common vector Tx fns with context handling Bruce Richardson
@ 2026-09-03 17:04 ` Bruce Richardson
2026-09-03 17:04 ` [PATCH 13/13] net/ice: enable context desc offloads for vector Tx Bruce Richardson
2026-09-04 7:10 ` [PATCH 00/13] Consolidate ice and iavf vector Tx paths David Marchand
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:04 UTC (permalink / raw)
To: dev
Cc: Bruce Richardson, Praveen Shetty, Vladimir Medvedkin,
Anatoly Burakov, Jingjing Wu
Rework the relevant priority of context descriptor vs SIMD width when
selecting a Tx path so that presence of context descriptors is only
considered when all else is equal.
Normally, path select takes place without having to worry about context
descriptors, since offloads requiring context descriptors are specified
using the offloads bitmap for each Tx path. However, in some cases, the
need to use a context descriptor is not encodable via this offload path,
e.g. VLAN offload for iavf can sometimes use the data descriptor and
sometimes the context depending on what PF specifies at runtime. To
account for those cases, we separate out the use of contexts from the
path features, but instead note it as part of the path information, and
add an additional parameter to the Tx path selection for drivers to
mandate a context descriptor be present. Otherwise, ctx is largely
ignored in path selection, except as a last-resort tie-breaker between
two paths.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/net/intel/common/tx.h | 41 +++++++++++++++++---------
drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +-
drivers/net/intel/i40e/i40e_rxtx.c | 2 +-
drivers/net/intel/iavf/iavf_rxtx.c | 46 ++++++++++++++++--------------
drivers/net/intel/ice/ice_rxtx.c | 3 +-
drivers/net/intel/idpf/idpf_rxtx.c | 3 +-
6 files changed, 58 insertions(+), 40 deletions(-)
diff --git a/drivers/net/intel/common/tx.h b/drivers/net/intel/common/tx.h
index 630df8cb19..0f36989b32 100644
--- a/drivers/net/intel/common/tx.h
+++ b/drivers/net/intel/common/tx.h
@@ -249,7 +249,6 @@ struct ci_tx_path_features {
uint32_t tx_offloads;
enum rte_vect_max_simd simd_width;
bool simple_tx;
- bool ctx_desc;
bool disabled;
bool single_queue;
};
@@ -259,6 +258,7 @@ struct ci_tx_path_info {
const char *info;
struct ci_tx_path_features features;
eth_tx_prep_t pkt_prep;
+ bool supports_ctx;
};
static __rte_always_inline void
@@ -411,6 +411,11 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)
* Number of available paths in the infos array
* @param default_path
* Index of the default path to use if no suitable path is found
+ * @param force_ctx
+ * If true, only paths that support context descriptors may be selected.
+ * Use this for offloads that require a context descriptor but cannot be
+ * discovered purely from req_features->tx_offloads (e.g. a driver-specific
+ * devarg, or a runtime/hardware-negotiated tag placement).
*
* @return
* The packet burst function index that best matches the requested features,
@@ -420,10 +425,12 @@ static inline int
ci_tx_path_select(const struct ci_tx_path_features *req_features,
const struct ci_tx_path_info *infos,
size_t num_paths,
- int default_path)
+ int default_path,
+ bool force_ctx)
{
int idx = default_path;
const struct ci_tx_path_features *chosen_path_features = NULL;
+ bool chosen_supports_ctx = false;
for (unsigned int i = 0; i < num_paths; i++) {
const struct ci_tx_path_features *path_features = &infos[i].features;
@@ -440,8 +447,8 @@ ci_tx_path_select(const struct ci_tx_path_features *req_features,
if (path_features->simple_tx && !req_features->simple_tx)
continue;
- /* If a context descriptor is requested, ensure the path supports it. */
- if (!path_features->ctx_desc && req_features->ctx_desc)
+ /* If a context descriptor is required, ensure the path supports it. */
+ if (!infos[i].supports_ctx && force_ctx)
continue;
/* If requested, ensure the path supports single queue TX. */
@@ -462,22 +469,28 @@ ci_tx_path_select(const struct ci_tx_path_features *req_features,
/* Do not select paths with lower SIMD width than the chosen path. */
if (path_features->simd_width < chosen_path_features->simd_width)
continue;
- /* Do not select paths with more offloads enabled than the chosen path if
- * the SIMD widths are the same.
+ /* The following tie-breaks only matter when SIMD widths are tied;
+ * a strictly wider path is always preferred regardless of offload
+ * count or ctx-descriptor use.
*/
- if (path_features->simd_width == chosen_path_features->simd_width &&
- rte_popcount32(path_features->tx_offloads) >
- rte_popcount32(chosen_path_features->tx_offloads))
- continue;
-
- /* Don't use a context descriptor unless necessary */
- if (path_features->ctx_desc && !chosen_path_features->ctx_desc)
- continue;
+ if (path_features->simd_width == chosen_path_features->simd_width) {
+ /* Do not select paths with more offloads enabled than the
+ * chosen path.
+ */
+ if (rte_popcount32(path_features->tx_offloads) >
+ rte_popcount32(chosen_path_features->tx_offloads))
+ continue;
+
+ /* Don't use a context descriptor unless necessary */
+ if (infos[i].supports_ctx && !chosen_supports_ctx)
+ continue;
+ }
}
/* Finally, select the path since it has met all the requirements. */
idx = i;
chosen_path_features = &infos[idx].features;
+ chosen_supports_ctx = infos[idx].supports_ctx;
}
return idx;
diff --git a/drivers/net/intel/cpfl/cpfl_rxtx.c b/drivers/net/intel/cpfl/cpfl_rxtx.c
index 2f1641ea76..aba3a5e916 100644
--- a/drivers/net/intel/cpfl/cpfl_rxtx.c
+++ b/drivers/net/intel/cpfl/cpfl_rxtx.c
@@ -1542,7 +1542,8 @@ cpfl_set_tx_function(struct rte_eth_dev *dev)
ad->tx_func_type = ci_tx_path_select(&req_features,
&idpf_tx_path_infos[0],
IDPF_TX_MAX,
- IDPF_TX_DEFAULT);
+ IDPF_TX_DEFAULT,
+ false);
/* Set use_vec_entry for single queue mode - only IDPF_TX_SINGLEQ uses regular entries */
if (vport->txq_model == VIRTCHNL2_QUEUE_MODEL_SINGLE) {
diff --git a/drivers/net/intel/i40e/i40e_rxtx.c b/drivers/net/intel/i40e/i40e_rxtx.c
index e2fffdb70a..f289d26790 100644
--- a/drivers/net/intel/i40e/i40e_rxtx.c
+++ b/drivers/net/intel/i40e/i40e_rxtx.c
@@ -3120,7 +3120,7 @@ i40e_set_tx_function(struct rte_eth_dev *dev)
}
ad->tx_func_type = ci_tx_path_select(&req_features, &i40e_tx_path_infos[0],
- RTE_DIM(i40e_tx_path_infos), I40E_TX_DEFAULT);
+ RTE_DIM(i40e_tx_path_infos), I40E_TX_DEFAULT, false);
out:
dev->tx_pkt_burst = mbuf_check ? i40e_xmit_pkts_check :
diff --git a/drivers/net/intel/iavf/iavf_rxtx.c b/drivers/net/intel/iavf/iavf_rxtx.c
index 849fc33dac..3be8b00bbd 100644
--- a/drivers/net/intel/iavf/iavf_rxtx.c
+++ b/drivers/net/intel/iavf/iavf_rxtx.c
@@ -3611,9 +3611,9 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = {
.pkt_burst = iavf_xmit_pkts,
.info = "Scalar",
.features = {
- .tx_offloads = IAVF_TX_SCALAR_OFFLOADS,
- .ctx_desc = true
- }
+ .tx_offloads = IAVF_TX_SCALAR_OFFLOADS
+ },
+ .supports_ctx = true
},
#ifdef RTE_ARCH_X86
[IAVF_TX_AVX2] = {
@@ -3637,18 +3637,18 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = {
.info = "Vector AVX2 Ctx",
.features = {
.tx_offloads = IAVF_TX_VECTOR_OFFLOADS,
- .simd_width = RTE_VECT_SIMD_256,
- .ctx_desc = true
- }
+ .simd_width = RTE_VECT_SIMD_256
+ },
+ .supports_ctx = true
},
[IAVF_TX_AVX2_CTX_OFFLOAD] = {
.pkt_burst = iavf_xmit_pkts_vec_avx2_ctx_offload,
.info = "Vector AVX2 Ctx Offload",
.features = {
.tx_offloads = IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADS,
- .simd_width = RTE_VECT_SIMD_256,
- .ctx_desc = true
- }
+ .simd_width = RTE_VECT_SIMD_256
+ },
+ .supports_ctx = true
},
#ifdef CC_AVX512_SUPPORT
[IAVF_TX_AVX512] = {
@@ -3672,18 +3672,18 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = {
.info = "Vector AVX512 Ctx",
.features = {
.tx_offloads = IAVF_TX_VECTOR_OFFLOADS,
- .simd_width = RTE_VECT_SIMD_512,
- .ctx_desc = true
- }
+ .simd_width = RTE_VECT_SIMD_512
+ },
+ .supports_ctx = true
},
[IAVF_TX_AVX512_CTX_OFFLOAD] = {
.pkt_burst = iavf_xmit_pkts_vec_avx512_ctx_offload,
.info = "Vector AVX512 Ctx Offload",
.features = {
.tx_offloads = IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADS,
- .simd_width = RTE_VECT_SIMD_512,
- .ctx_desc = true
- }
+ .simd_width = RTE_VECT_SIMD_512
+ },
+ .supports_ctx = true
},
#endif
#elif defined(RTE_ARCH_ARM64)
@@ -3918,12 +3918,13 @@ iavf_set_tx_function(struct rte_eth_dev *dev)
#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM64)
struct ci_tx_queue *txq;
int i;
- const struct ci_tx_path_features *selected_features;
+ const struct ci_tx_path_info *selected_info;
#endif
struct ci_tx_path_features req_features = {
.tx_offloads = dev->data->dev_conf.txmode.offloads,
.simd_width = RTE_VECT_SIMD_DISABLED,
};
+ bool force_ctx = false;
/* If the device has started the function has already been selected. */
if (dev->data->dev_started)
@@ -3934,7 +3935,7 @@ iavf_set_tx_function(struct rte_eth_dev *dev)
req_features.simd_width = iavf_get_max_simd_bitwidth();
if (adapter->devargs.enable_lldp)
- req_features.ctx_desc = true;
+ force_ctx = true;
for (i = 0; i < dev->data->nb_tx_queues; i++) {
txq = dev->data->tx_queues[i];
@@ -3942,24 +3943,25 @@ iavf_set_tx_function(struct rte_eth_dev *dev)
continue;
if (txq->offloads & RTE_ETH_TX_OFFLOAD_VLAN_INSERT &&
txq->vlan_flag == IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2)
- req_features.ctx_desc = true;
+ force_ctx = true;
}
#endif
adapter->tx_func_type = ci_tx_path_select(&req_features,
&iavf_tx_path_infos[0],
RTE_DIM(iavf_tx_path_infos),
- IAVF_TX_DEFAULT);
+ IAVF_TX_DEFAULT,
+ force_ctx);
out:
#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM64)
- selected_features = &iavf_tx_path_infos[adapter->tx_func_type].features;
+ selected_info = &iavf_tx_path_infos[adapter->tx_func_type];
for (i = 0; i < dev->data->nb_tx_queues; i++) {
txq = dev->data->tx_queues[i];
if (!txq)
continue;
- txq->use_ctx = selected_features->ctx_desc;
- txq->use_vec_entry = selected_features->simd_width >= RTE_VECT_SIMD_128;
+ txq->use_ctx = selected_info->supports_ctx;
+ txq->use_vec_entry = selected_info->features.simd_width >= RTE_VECT_SIMD_128;
}
#endif
diff --git a/drivers/net/intel/ice/ice_rxtx.c b/drivers/net/intel/ice/ice_rxtx.c
index 3569ffcf82..313cbd25d5 100644
--- a/drivers/net/intel/ice/ice_rxtx.c
+++ b/drivers/net/intel/ice/ice_rxtx.c
@@ -3777,7 +3777,8 @@ ice_set_tx_function(struct rte_eth_dev *dev)
ad->tx_func_type = ci_tx_path_select(&req_features,
&ice_tx_path_infos[0],
RTE_DIM(ice_tx_path_infos),
- ICE_TX_DEFAULT);
+ ICE_TX_DEFAULT,
+ false);
out:
#if defined(RTE_ARCH_X86)
diff --git a/drivers/net/intel/idpf/idpf_rxtx.c b/drivers/net/intel/idpf/idpf_rxtx.c
index bc3f9a0798..077a92a8a9 100644
--- a/drivers/net/intel/idpf/idpf_rxtx.c
+++ b/drivers/net/intel/idpf/idpf_rxtx.c
@@ -887,7 +887,8 @@ idpf_set_tx_function(struct rte_eth_dev *dev)
ad->tx_func_type = ci_tx_path_select(&req_features,
&idpf_tx_path_infos[0],
IDPF_TX_MAX,
- IDPF_TX_DEFAULT);
+ IDPF_TX_DEFAULT,
+ false);
/* Set use_vec_entry for single queue mode - only IDPF_TX_SINGLEQ uses regular entries */
if (vport->txq_model == VIRTCHNL2_QUEUE_MODEL_SINGLE) {
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 13/13] net/ice: enable context desc offloads for vector Tx
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (11 preceding siblings ...)
2026-09-03 17:04 ` [PATCH 12/13] net/intel: improve Tx path selection logic Bruce Richardson
@ 2026-09-03 17:04 ` Bruce Richardson
2026-09-04 7:10 ` [PATCH 00/13] Consolidate ice and iavf vector Tx paths David Marchand
13 siblings, 0 replies; 16+ messages in thread
From: Bruce Richardson @ 2026-09-03 17:04 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Anatoly Burakov
Use the context descriptor Tx vector paths from intel/common in ice
driver to enable more offloads on the vector path for ice driver.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
doc/guides/rel_notes/release_26_11.rst | 6 ++++
drivers/net/intel/ice/ice_ethdev.h | 3 ++
drivers/net/intel/ice/ice_rxtx.c | 31 ++++++++++++++++++---
drivers/net/intel/ice/ice_rxtx.h | 11 ++++++++
drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++++++++++++
drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 24 ++++++++++++++++
6 files changed, 95 insertions(+), 4 deletions(-)
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index 907f9013ff..47eccd9401 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -64,6 +64,12 @@ New Features
* Renamed the ``enable_ptype_lldp`` devarg to ``enable_lldp``.
The old name is no longer accepted.
+* **Updated Intel ice driver.**
+
+ * Added Tx context descriptor support to the AVX2 and AVX512 vector Tx
+ paths, enabling QinQ tag insertion and outer IPv4/UDP checksum
+ offloads on those paths.
+
Removed Items
-------------
diff --git a/drivers/net/intel/ice/ice_ethdev.h b/drivers/net/intel/ice/ice_ethdev.h
index 7ee3ea8a70..d357181a2a 100644
--- a/drivers/net/intel/ice/ice_ethdev.h
+++ b/drivers/net/intel/ice/ice_ethdev.h
@@ -213,8 +213,10 @@ enum ice_tx_func_type {
ICE_TX_SIMPLE,
ICE_TX_AVX2,
ICE_TX_AVX2_OFFLOAD,
+ ICE_TX_AVX2_CTX_OFFLOAD,
ICE_TX_AVX512,
ICE_TX_AVX512_OFFLOAD,
+ ICE_TX_AVX512_CTX_OFFLOAD,
ICE_TX_NEON,
};
@@ -673,6 +675,7 @@ struct ice_adapter {
bool rx_bulk_alloc_allowed;
bool tx_vec_allowed;
bool tx_simple_allowed;
+ bool use_ctx;
enum ice_rx_func_type rx_func_type;
enum ice_tx_func_type tx_func_type;
/* ptype mapping table */
diff --git a/drivers/net/intel/ice/ice_rxtx.c b/drivers/net/intel/ice/ice_rxtx.c
index 313cbd25d5..c6a464b2c1 100644
--- a/drivers/net/intel/ice/ice_rxtx.c
+++ b/drivers/net/intel/ice/ice_rxtx.c
@@ -883,6 +883,7 @@ ice_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id)
/* record what kind of descriptor cleanup we need on teardown */
txq->use_vec_entry = ad->tx_vec_allowed || ad->tx_simple_allowed;
+ txq->use_ctx = ad->use_ctx;
if (txq->tsq != NULL && txq->tsq->ts_flag > 0) {
struct ice_aqc_set_txtime_qgrp *ts_elem;
@@ -1193,7 +1194,7 @@ ice_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id)
return -EINVAL;
}
- ci_txq_release_all_mbufs(txq, false);
+ ci_txq_release_all_mbufs(txq, txq->use_ctx);
ice_reset_tx_queue(txq);
dev->data->tx_queue_state[tx_queue_id] = RTE_ETH_QUEUE_STATE_STOPPED;
@@ -1256,7 +1257,7 @@ ice_fdir_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id)
return -EINVAL;
}
- ci_txq_release_all_mbufs(txq, false);
+ ci_txq_release_all_mbufs(txq, txq->use_ctx);
txq->qtx_tail = NULL;
return 0;
@@ -1744,7 +1745,7 @@ ice_tx_queue_release(void *txq)
return;
}
- ci_txq_release_all_mbufs(q, false);
+ ci_txq_release_all_mbufs(q, q->use_ctx);
rte_free(q->sw_ring);
rte_free(q->rs_last_id);
if (q->tsq) {
@@ -3526,7 +3527,8 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = {
.features = {
.tx_offloads = ICE_TX_SCALAR_OFFLOADS
},
- .pkt_prep = ice_prep_pkts
+ .pkt_prep = ice_prep_pkts,
+ .supports_ctx = true
},
[ICE_TX_SIMPLE] = {
.pkt_burst = ice_xmit_pkts_simple,
@@ -3556,6 +3558,16 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = {
},
.pkt_prep = ice_prep_pkts
},
+ [ICE_TX_AVX2_CTX_OFFLOAD] = {
+ .pkt_burst = ice_xmit_pkts_vec_avx2_ctx_offload,
+ .info = "Offload Vector AVX2 Ctx",
+ .features = {
+ .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS,
+ .simd_width = RTE_VECT_SIMD_256
+ },
+ .pkt_prep = ice_prep_pkts,
+ .supports_ctx = true
+ },
#ifdef CC_AVX512_SUPPORT
[ICE_TX_AVX512] = {
.pkt_burst = ice_xmit_pkts_vec_avx512,
@@ -3575,6 +3587,16 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = {
},
.pkt_prep = ice_prep_pkts
},
+ [ICE_TX_AVX512_CTX_OFFLOAD] = {
+ .pkt_burst = ice_xmit_pkts_vec_avx512_ctx_offload,
+ .info = "Offload Vector AVX512 Ctx",
+ .features = {
+ .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS,
+ .simd_width = RTE_VECT_SIMD_512
+ },
+ .pkt_prep = ice_prep_pkts,
+ .supports_ctx = true
+ },
#endif
#elif defined(RTE_ARCH_ARM64)
[ICE_TX_NEON] = {
@@ -3788,6 +3810,7 @@ ice_set_tx_function(struct rte_eth_dev *dev)
ad->tx_vec_allowed =
(ice_tx_path_infos[ad->tx_func_type].features.simd_width >= RTE_VECT_SIMD_128);
#endif
+ ad->use_ctx = ice_tx_path_infos[ad->tx_func_type].supports_ctx;
dev->tx_pkt_burst = mbuf_check ? ice_xmit_pkts_check :
ice_tx_path_infos[ad->tx_func_type].pkt_burst;
diff --git a/drivers/net/intel/ice/ice_rxtx.h b/drivers/net/intel/ice/ice_rxtx.h
index 999b6b30d6..bee8a464f5 100644
--- a/drivers/net/intel/ice/ice_rxtx.h
+++ b/drivers/net/intel/ice/ice_rxtx.h
@@ -135,6 +135,12 @@
RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \
RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \
RTE_ETH_TX_OFFLOAD_SCTP_CKSUM)
+/* vector ctx offload path: QinQ outer tag + outer/tunnel checksum via ctx desc */
+#define ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \
+ ICE_TX_VECTOR_OFFLOAD_OFFLOADS | \
+ RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \
+ RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \
+ RTE_ETH_TX_OFFLOAD_QINQ_INSERT)
/* Max header size can be 2K - 64 bytes */
#define ICE_RX_HDR_BUF_SIZE (2048 - 64)
@@ -284,6 +290,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts);
uint16_t ice_xmit_pkts_vec_avx2_offload(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts);
+uint16_t ice_xmit_pkts_vec_avx2_ctx_offload(void *tx_queue, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts);
uint16_t ice_recv_pkts_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts,
uint16_t nb_pkts);
uint16_t ice_recv_pkts_vec_avx512_offload(void *rx_queue,
@@ -300,6 +308,9 @@ uint16_t ice_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,
uint16_t ice_xmit_pkts_vec_avx512_offload(void *tx_queue,
struct rte_mbuf **tx_pkts,
uint16_t nb_pkts);
+uint16_t ice_xmit_pkts_vec_avx512_ctx_offload(void *tx_queue,
+ struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts);
int ice_fdir_programming(struct ice_pf *pf, struct ice_fltr_desc *fdir_desc);
int ice_tx_done_cleanup(void *txq, uint32_t free_cnt);
int ice_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc);
diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c
index 00b5d51aab..056b2dc49b 100644
--- a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c
+++ b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c
@@ -810,6 +810,30 @@ ice_xmit_pkts_vec_avx2_offload(void *tx_queue, struct rte_mbuf **tx_pkts,
return ice_xmit_pkts_vec_avx2_common(tx_queue, tx_pkts, nb_pkts, true);
}
+uint16_t
+ice_xmit_pkts_vec_avx2_ctx_offload(void *tx_queue, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts)
+{
+ uint16_t nb_tx = 0;
+ struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
+
+ while (nb_pkts) {
+ uint16_t ret, num;
+
+ /* cross rs_thresh boundary is not allowed */
+ num = (uint16_t)RTE_MIN(nb_pkts << 1, txq->tx_rs_thresh);
+ num = num >> 1;
+ ret = ci_xmit_fixed_burst_vec_ctx_avx2(tx_queue, &tx_pkts[nb_tx], num,
+ true, CI_TAG_IN_DATA_DESC, CI_TAG_IN_CTX_DESC, NULL);
+ nb_tx += ret;
+ nb_pkts -= ret;
+ if (ret < num)
+ break;
+ }
+
+ return nb_tx;
+}
+
int __rte_cold
ice_rxq_vec_setup(struct ci_rx_queue *rxq)
{
diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c
index d8ede5cb42..0548c293bb 100644
--- a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c
+++ b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c
@@ -882,3 +882,27 @@ ice_xmit_pkts_vec_avx512_offload(void *tx_queue, struct rte_mbuf **tx_pkts,
{
return ice_xmit_pkts_vec_avx512_common(tx_queue, tx_pkts, nb_pkts, true);
}
+
+uint16_t
+ice_xmit_pkts_vec_avx512_ctx_offload(void *tx_queue, struct rte_mbuf **tx_pkts,
+ uint16_t nb_pkts)
+{
+ uint16_t nb_tx = 0;
+ struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue;
+
+ while (nb_pkts) {
+ uint16_t ret, num;
+
+ /* cross rs_thresh boundary is not allowed */
+ num = (uint16_t)RTE_MIN(nb_pkts << 1, txq->tx_rs_thresh);
+ num = num >> 1;
+ ret = ci_xmit_fixed_burst_vec_ctx_avx512(tx_queue, &tx_pkts[nb_tx], num,
+ true, CI_TAG_IN_DATA_DESC, CI_TAG_IN_CTX_DESC, NULL);
+ nb_tx += ret;
+ nb_pkts -= ret;
+ if (ret < num)
+ break;
+ }
+
+ return nb_tx;
+}
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH 04/13] net/iavf: deduplicate tunnel field fill functions
2026-09-03 17:01 ` [PATCH 04/13] net/iavf: deduplicate tunnel field fill functions Bruce Richardson
@ 2026-09-04 7:01 ` David Marchand
0 siblings, 0 replies; 16+ messages in thread
From: David Marchand @ 2026-09-04 7:01 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev, Vladimir Medvedkin
On Thu, 3 Sept 2026 at 19:02, Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> Each of the avx2 and avx512 Tx functions used two separate subfunctions
> to insert the tunneling fields into the context descriptor. Except for
> some minor differences (a flags guard on one) the two were identical, so
> remove the one missing the additional guard.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Thanks for the cleanup.
Reviewed-by: David Marchand <david.marchand@redhat.com>
--
David Marchand
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 00/13] Consolidate ice and iavf vector Tx paths
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
` (12 preceding siblings ...)
2026-09-03 17:04 ` [PATCH 13/13] net/ice: enable context desc offloads for vector Tx Bruce Richardson
@ 2026-09-04 7:10 ` David Marchand
13 siblings, 0 replies; 16+ messages in thread
From: David Marchand @ 2026-09-04 7:10 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev
On Thu, 3 Sept 2026 at 19:02, Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> This patchset consolidates the ice and iavf vector Tx paths, moving the
> code to net/intel/common. The iavf code is used as the basis for the
> move, because it has a superset of the features. When the common code is
> adopted by the ice driver, it then gains features such as QinQ support
> and tunneling support.
>
> Depends-on: series-39162 ("VLAN and QinQ fixes for iavf")
>
> Bruce Richardson (13):
> net/iavf: remove unnecessary alignment calls
> net/intel: make Tx context flag common
> net/iavf: use separate params for VLAN and QinQ position
> net/iavf: deduplicate tunnel field fill functions
> net/intel: define common macros for tunneling bit-shifts
> net/intel: move iavf descriptor writing functions to common
> net/intel: use function callback for lldp
> net/ice: use common descriptor creation functions
> net/intel: move vector Tx paths to common
> net/ice: use common AVX Tx functions
> net/intel: add common vector Tx fns with context handling
> net/intel: improve Tx path selection logic
> net/ice: enable context desc offloads for vector Tx
>
> doc/guides/rel_notes/release_26_11.rst | 6 +
> drivers/net/intel/common/tx.h | 77 +-
> drivers/net/intel/common/tx_vec_x86.h | 783 ++++++++++++++++++
> drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +-
> drivers/net/intel/i40e/i40e_rxtx.c | 2 +-
> drivers/net/intel/iavf/iavf_rxtx.c | 76 +-
> drivers/net/intel/iavf/iavf_rxtx.h | 34 -
> drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 496 +----------
> drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 495 +----------
> drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 76 +-
> drivers/net/intel/ice/ice_ethdev.h | 3 +
> drivers/net/intel/ice/ice_rxtx.c | 34 +-
> drivers/net/intel/ice/ice_rxtx.h | 11 +
> drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 159 +---
> drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 154 +---
> drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 --
> drivers/net/intel/idpf/idpf_rxtx.c | 3 +-
> 17 files changed, 1025 insertions(+), 1446 deletions(-)
> create mode 100644 drivers/net/intel/common/tx_vec_x86.h
I do not have the time to review in detail but this series looks good to me.
My bench setup is unavailable and I am busy on other topics, so I
won't be able to test it soon.
But in any case, thank you Bruce, nice cleanup again.
--
David Marchand
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2026-09-04 7:10 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-09-03 17:01 [PATCH 00/13] Consolidate ice and iavf vector Tx paths Bruce Richardson
2026-09-03 17:01 ` [PATCH 01/13] net/iavf: remove unnecessary alignment calls Bruce Richardson
2026-09-03 17:01 ` [PATCH 02/13] net/intel: make Tx context flag common Bruce Richardson
2026-09-03 17:01 ` [PATCH 03/13] net/iavf: use separate params for VLAN and QinQ position Bruce Richardson
2026-09-03 17:01 ` [PATCH 04/13] net/iavf: deduplicate tunnel field fill functions Bruce Richardson
2026-09-04 7:01 ` David Marchand
2026-09-03 17:01 ` [PATCH 05/13] net/intel: define common macros for tunneling bit-shifts Bruce Richardson
2026-09-03 17:01 ` [PATCH 06/13] net/intel: move iavf descriptor writing functions to common Bruce Richardson
2026-09-03 17:01 ` [PATCH 07/13] net/intel: use function callback for lldp Bruce Richardson
2026-09-03 17:01 ` [PATCH 08/13] net/ice: use common descriptor creation functions Bruce Richardson
2026-09-03 17:01 ` [PATCH 09/13] net/intel: move vector Tx paths to common Bruce Richardson
2026-09-03 17:03 ` [PATCH 10/13] net/ice: use common AVX Tx functions Bruce Richardson
2026-09-03 17:04 ` [PATCH 11/13] net/intel: add common vector Tx fns with context handling Bruce Richardson
2026-09-03 17:04 ` [PATCH 12/13] net/intel: improve Tx path selection logic Bruce Richardson
2026-09-03 17:04 ` [PATCH 13/13] net/ice: enable context desc offloads for vector Tx Bruce Richardson
2026-09-04 7:10 ` [PATCH 00/13] Consolidate ice and iavf vector Tx paths David Marchand
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox