* [PATCH iwl-next v3 0/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support
@ 2026-11-19 14:09 Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 1/3] idpf: always reserve one empty descriptor on XDPSQs Alexander Lobakin
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Alexander Lobakin @ 2026-11-19 14:09 UTC (permalink / raw)
To: intel-wired-lan; +Cc: Alexander Lobakin, Tony Nguyen, Przemek Kitszel
From the first days of XDP implementation in idpf, it relied and
worked solely on top of the queue-based scheduling Tx mode, which
basically means simple FIFO. However, turned out not every firmware
supports this mode and XDP doesn't work there at all.
First, fix two already existing issues found by Sashiko, then
introduce the flow-based fallback for all XDP paths which is always
supported by any FW. The actual implementation is selected during
XDP program setup basing on the caps that the FW advertises, so
that firmwares that already work correctly won't get performance
regressions the fallback brings.
Alexander Lobakin (3):
idpf: always reserve one empty descriptor on XDPSQs
idpf: enforce strict ordering on BE when reading XDPCQEs
idpf: add flow-based XDP fallback for FWs without Tx FIFO support
drivers/net/ethernet/intel/idpf/idpf_txrx.h | 18 ++-
drivers/net/ethernet/intel/idpf/xdp.h | 60 ++++++++
include/net/libeth/xdp.h | 13 ++
drivers/net/ethernet/intel/idpf/idpf_txrx.c | 19 ++-
drivers/net/ethernet/intel/idpf/xdp.c | 152 ++++++++++++++++++--
drivers/net/ethernet/intel/idpf/xsk.c | 2 +-
6 files changed, 242 insertions(+), 22 deletions(-)
---
From v2[0]:
* new fix (0001): fix already existing issue with not reserving
a required free Tx descriptor in case of XDP Tx, xmit, and XSk Tx
(Sashiko);
* new fix (0002): enforce strict ordering when parsing XDPCQEs on
Big Endian (Intel's Sashiko);
* rewrite the bitmap set-reset implementation from scratch to fix
fix S/G: HW generates a completion per packet, not per descriptor
(Sashiko);
* fix excessive allocations, possible pointer overwrites, and corner
case static key imbalance when setting up XDPSQs (Intel's Sashiko).
From v1[1]:
* rework static key management: move to idpf_xdpsqs_{get,put}() to
avoid refcount imbalance issues as .ndo_bpf() is not always called
in pairs (hardware reset etc.) (Sashiko, Intel's Sashiko);
* don't zero the whole pending window but only the frames sent since
the last batch to avoid missed OOO completions (Intel's Sashiko);
* micro-optimize idpf_xdpsq_set_rs{,_fb}().
Sorry I've had health issues for the last year. I hope I'll be
restoring my usual "usefulness" (reviews, submissions) on
the LKML bit by bit.
[0] https://lore.kernel.org/intel-wired-lan/20260818155101.2416665-1-aleksander.lobakin@intel.com
[1] https://lore.kernel.org/intel-wired-lan/20260708151327.1091570-1-aleksander.lobakin@intel.com
--
2.56.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH iwl-next v3 1/3] idpf: always reserve one empty descriptor on XDPSQs
2026-11-19 14:09 [PATCH iwl-next v3 0/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support Alexander Lobakin
@ 2026-11-19 14:09 ` Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 2/3] idpf: enforce strict ordering on BE when reading XDPCQEs Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 3/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support Alexander Lobakin
2 siblings, 0 replies; 5+ messages in thread
From: Alexander Lobakin @ 2026-11-19 14:09 UTC (permalink / raw)
To: intel-wired-lan; +Cc: Alexander Lobakin, Tony Nguyen, Przemek Kitszel, stable
idpf-powered HW requires at least one Tx descriptor to be always
free, so that head and tail are never equal, otherwise the HW will
deadlock. This is already taken into account in the XSk xmit code,
however, not in regular XDP Tx, xmit, and XSk Tx -- these are much
slower and are in fact almost unable to trigger that.
Either way, be better safe than sorry and always reserve one empty
Tx descriptor by passing `free - 1` (this can't underflow since
`free` is always at least 1 due to this reservation).
Found by Sashiko.
Fixes: cba102cd7190 ("idpf: add support for XDP on Rx")
Fixes: 9705d6552f58 ("idpf: implement Rx path for AF_XDP")
Cc: stable@vger.kernel.org # 6.18+
Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
---
drivers/net/ethernet/intel/idpf/xdp.c | 2 +-
drivers/net/ethernet/intel/idpf/xsk.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/intel/idpf/xdp.c b/drivers/net/ethernet/intel/idpf/xdp.c
index cbccd4546768..1d8ed7ac3e17 100644
--- a/drivers/net/ethernet/intel/idpf/xdp.c
+++ b/drivers/net/ethernet/intel/idpf/xdp.c
@@ -351,7 +351,7 @@ static u32 idpf_xdp_tx_prep(void *_xdpsq, struct libeth_xdpsq *sq)
.xdp_tx = &xdpsq->xdp_tx,
};
- return free;
+ return free - 1;
}
LIBETH_XDP_DEFINE_START();
diff --git a/drivers/net/ethernet/intel/idpf/xsk.c b/drivers/net/ethernet/intel/idpf/xsk.c
index 5a0a1d8a7a86..ef079d2cfaa5 100644
--- a/drivers/net/ethernet/intel/idpf/xsk.c
+++ b/drivers/net/ethernet/intel/idpf/xsk.c
@@ -294,7 +294,7 @@ static u32 idpf_xsk_tx_prep(void *_xdpsq, struct libeth_xdpsq *sq)
.xdp_tx = &xdpsq->xdp_tx,
};
- return free;
+ return free - 1;
}
static u32 idpf_xsk_xmit_prep(void *_xdpsq, struct libeth_xdpsq *sq)
--
2.56.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH iwl-next v3 2/3] idpf: enforce strict ordering on BE when reading XDPCQEs
2026-11-19 14:09 [PATCH iwl-next v3 0/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 1/3] idpf: always reserve one empty descriptor on XDPSQs Alexander Lobakin
@ 2026-11-19 14:09 ` Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 3/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support Alexander Lobakin
2 siblings, 0 replies; 5+ messages in thread
From: Alexander Lobakin @ 2026-11-19 14:09 UTC (permalink / raw)
To: intel-wired-lan; +Cc: Alexander Lobakin, Tony Nguyen, Przemek Kitszel, stable
On Little Endian platforms, everything we need from the completion
descriptor is read at once, while on Big Endian we read it
field-by-field. Depending on the compiler and optimization settings,
reads can be combined or stay distinct. The latter leaves us with no
barrier and with possible stale completion index or tag.
Change the logic a bit and read only the "readiness" field first on
Big Endian, then perform strictly-ordered read of the cookie after
making sure it is valid.
The Little Endian code remains intact. Found by Sashiko.
Fixes: cba102cd7190 ("idpf: add support for XDP on Rx")
Cc: stable@vger.kernel.org # 6.18+
Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
---
drivers/net/ethernet/intel/idpf/xdp.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/intel/idpf/xdp.c b/drivers/net/ethernet/intel/idpf/xdp.c
index 1d8ed7ac3e17..48320ea004df 100644
--- a/drivers/net/ethernet/intel/idpf/xdp.c
+++ b/drivers/net/ethernet/intel/idpf/xdp.c
@@ -236,8 +236,7 @@ static int idpf_xdp_parse_cqe(const struct idpf_splitq_4b_tx_compl_desc *desc,
#ifdef __LIBETH_WORD_ACCESS
val = *(const u32 *)desc;
#else
- val = ((u32)le16_to_cpu(desc->q_head_compl_tag.q_head) << 16) |
- le16_to_cpu(desc->qid_comptype_gen);
+ val = le16_to_cpu(desc->qid_comptype_gen);
#endif
if (!!(val & IDPF_TXD_COMPLQ_GEN_M) != gen)
return -ENODATA;
@@ -247,7 +246,13 @@ static int idpf_xdp_parse_cqe(const struct idpf_splitq_4b_tx_compl_desc *desc,
IDPF_TXD_COMPLT_RS)))
return -EINVAL;
+#ifdef __LIBETH_WORD_ACCESS
return upper_16_bits(val);
+#else
+ dma_rmb();
+
+ return le16_to_cpu(desc->q_head_compl_tag.q_head);
+#endif
}
u32 idpf_xdpsq_poll(struct idpf_tx_queue *xdpsq, u32 budget)
--
2.56.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH iwl-next v3 3/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support
2026-11-19 14:09 [PATCH iwl-next v3 0/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 1/3] idpf: always reserve one empty descriptor on XDPSQs Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 2/3] idpf: enforce strict ordering on BE when reading XDPCQEs Alexander Lobakin
@ 2026-11-19 14:09 ` Alexander Lobakin
2026-11-19 15:02 ` sashiko-bot
2 siblings, 1 reply; 5+ messages in thread
From: Alexander Lobakin @ 2026-11-19 14:09 UTC (permalink / raw)
To: intel-wired-lan
Cc: Alexander Lobakin, Tony Nguyen, Przemek Kitszel,
Aleksandr Loktionov, YiFei Zhu, Patryk Holda
From the first days of XDP implementation in idpf, it relied and
worked solely on top of the queue-based scheduling Tx mode, which
basically means simple FIFO. However, turned out not every firmware
supports this mode and XDP doesn't work there at all.
Since the flow-based scheduling Tx mode is mandatory and supported
by every FW, introduce a simple fallback guarded by a static key
to not hurt the more performant mode. The FB mode generates a
completion for each packet and never guarantees that there won't
be any out-of-order completions. Serialize that using a bitmap of
completed descriptors and report contiguous blocks of free bits to
match XDP and XSk expectations and avoid further code complication.
In case of multi-buffer frames, only the bit corresponding to the
first descriptor is toggled as there will be only one completion
for the entire packet carrying the index of this first descriptor.
The usage of a bitmap on hotpath might sound scary, but this
fallback is able to reach around 70% of the QB mode's performance,
which is comparable to what ice gives us. The main bottlenecks are
unlikely()s and one completion per each packet, while in the QB
mode we have one completion per batch (which might contain 64 or
even 128 frames), plus the size of the completion descriptor is
8 bytes in this mode (4 bytes in the QB mode), which means a lot
of additional PCI traffic.
bloat-o-meter shows .text increase in about 2 Kb without adding new
functions or uninlining any of the existing ones. I played a bunch
with inlining and uninlining certain pieces or the whole fallback,
but the compiler collapses and optimizes libeth templates so hardly
so that each additional external call only makes things worse.
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Tested-by: YiFei Zhu <zhuyifei@google.com>
Tested-by: Patryk Holda <patryk.holda@intel.com>
Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
---
drivers/net/ethernet/intel/idpf/idpf_txrx.h | 18 ++-
drivers/net/ethernet/intel/idpf/xdp.h | 60 +++++++++
include/net/libeth/xdp.h | 13 ++
drivers/net/ethernet/intel/idpf/idpf_txrx.c | 19 ++-
drivers/net/ethernet/intel/idpf/xdp.c | 141 +++++++++++++++++++-
5 files changed, 233 insertions(+), 18 deletions(-)
diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethernet/intel/idpf/idpf_txrx.h
index 4e67837cf35f..283ed9bdf933 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h
+++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h
@@ -628,12 +628,14 @@ libeth_cacheline_set_assert(struct idpf_rx_queue,
* @clean_budget: singleq only, queue cleaning budget
* @cleaned_pkts: Number of packets cleaned for the above said case
* @refillq: Pointer to refill queue
+ * @cached_tstamp_caps: Tx timestamp capabilities negotiated with the CP
+ * @tstamp_task: Work that handles Tx timestamp read
* @pending: number of pending descriptors to send in QB
* @xdp_tx: number of pending &xdp_buff or &xdp_frame buffers
* @timer: timer for XDP Tx queue cleanup
* @xdp_lock: lock for XDP Tx queues sharing
- * @cached_tstamp_caps: Tx timestamp capabilities negotiated with the CP
- * @tstamp_task: Work that handles Tx timestamp read
+ * @pending_mask: mask of buffers waiting for completion in the FB XDP mode
+ * @curr_tag: current completion tag written to Tx descs in the FB XDP mode
* @stats_sync: See struct u64_stats_sync
* @q_stats: See union idpf_tx_queue_stats
* @q_id: Queue id
@@ -695,6 +697,9 @@ struct idpf_tx_queue {
u16 cleaned_pkts;
struct idpf_sw_queue *refillq;
+
+ struct idpf_ptp_vport_tx_tstamp_caps *cached_tstamp_caps;
+ struct work_struct *tstamp_task;
};
struct {
u32 pending;
@@ -702,12 +707,12 @@ struct idpf_tx_queue {
struct libeth_xdpsq_timer *timer;
struct libeth_xdpsq_lock xdp_lock;
+
+ unsigned long *pending_mask;
+ u32 curr_tag;
};
};
- struct idpf_ptp_vport_tx_tstamp_caps *cached_tstamp_caps;
- struct work_struct *tstamp_task;
-
struct u64_stats_sync stats_sync;
struct idpf_tx_queue_stats q_stats;
__cacheline_group_end_aligned(read_write);
@@ -724,8 +729,7 @@ struct idpf_tx_queue {
__cacheline_group_end_aligned(cold);
};
libeth_cacheline_set_assert(struct idpf_tx_queue, 64,
- 104 +
- offsetof(struct idpf_tx_queue, cached_tstamp_caps) -
+ 96 + offsetof(struct idpf_tx_queue, tstamp_task) -
offsetofend(struct idpf_tx_queue, timer) +
offsetof(struct idpf_tx_queue, q_stats) -
offsetofend(struct idpf_tx_queue, tstamp_task),
diff --git a/drivers/net/ethernet/intel/idpf/xdp.h b/drivers/net/ethernet/intel/idpf/xdp.h
index 63e56f7d43e0..1a657feb6d5b 100644
--- a/drivers/net/ethernet/intel/idpf/xdp.h
+++ b/drivers/net/ethernet/intel/idpf/xdp.h
@@ -8,6 +8,10 @@
#include "idpf_txrx.h"
+struct idpf_q_vec_rsrc;
+
+DECLARE_STATIC_KEY_FALSE(idpf_xdp_fb);
+
int idpf_xdp_rxq_info_init(struct idpf_rx_queue *rxq);
int idpf_xdp_rxq_info_init_all(const struct idpf_q_vec_rsrc *rsrc);
void idpf_xdp_rxq_info_deinit(struct idpf_rx_queue *rxq, u32 model);
@@ -21,6 +25,50 @@ void idpf_xdpsqs_put(const struct idpf_vport *vport);
u32 idpf_xdpsq_poll(struct idpf_tx_queue *xdpsq, u32 budget);
bool idpf_xdp_tx_flush_bulk(struct libeth_xdp_tx_bulk *bq, u32 flags);
+static inline void idpf_xdp_tx_xmit_fb(struct libeth_xdp_tx_desc desc, u32 i,
+ const struct libeth_xdpsq *sq, u64 priv)
+{
+ struct idpf_tx_queue *xdpsq = libeth_xdpsq_to_sq(sq,
+ struct idpf_tx_queue,
+ next_to_use);
+ struct idpf_flex_tx_sched_desc *tx_desc = sq->descs;
+ bool last = !!(desc.flags & LIBETH_XDP_TX_LAST);
+ u32 tag = xdpsq->curr_tag;
+ u32 cmd;
+
+ if (tag == U32_MAX) {
+ __set_bit(i, xdpsq->pending_mask);
+ xdpsq->curr_tag = i;
+ tag = i;
+ }
+
+ if (last)
+ xdpsq->curr_tag = U32_MAX;
+
+ cmd = FIELD_PREP(IDPF_TXD_FLEX_FLOW_DTYPE_M,
+ IDPF_TX_DESC_DTYPE_FLEX_FLOW_SCHE);
+ if (last)
+ cmd |= IDPF_TXD_FLEX_FLOW_CMD_EOP;
+ if (priv && (desc.flags & LIBETH_XDP_TX_CSUM))
+ cmd |= IDPF_TXD_FLEX_FLOW_CMD_CS_EN;
+
+ desc.len &= IDPF_TXD_FLEX_FLOW_BUFSIZE_M;
+
+ tx_desc = &tx_desc[i];
+ tx_desc->buf_addr = cpu_to_le64(desc.addr);
+
+#ifdef __LIBETH_WORD_ACCESS
+ *(u64 *)&tx_desc->qw1 = ((u64)desc.len << 48) | ((u64)tag << 32) | cmd;
+#else
+ tx_desc->qw1.rxr_bufsize = cpu_to_le16(desc.len);
+ tx_desc->qw1.compl_tag = cpu_to_le16(tag);
+ tx_desc->qw1.ts[0] = 0;
+ tx_desc->qw1.ts[1] = 0;
+ tx_desc->qw1.ts[2] = 0;
+ tx_desc->qw1.cmd_dtype = cmd;
+#endif
+}
+
/**
* idpf_xdp_tx_xmit - produce a single HW Tx descriptor out of XDP desc
* @desc: XDP descriptor to pull the DMA address and length from
@@ -34,6 +82,14 @@ static inline void idpf_xdp_tx_xmit(struct libeth_xdp_tx_desc desc, u32 i,
struct idpf_flex_tx_desc *tx_desc = sq->descs;
u32 cmd;
+ if (static_branch_unlikely(&idpf_xdp_fb) &&
+ idpf_queue_has(FLOW_SCH_EN,
+ libeth_xdpsq_to_sq(sq, struct idpf_tx_queue,
+ next_to_use))) {
+ idpf_xdp_tx_xmit_fb(desc, i, sq, priv);
+ return;
+ }
+
cmd = FIELD_PREP(IDPF_FLEX_TXD_QW1_DTYPE_M,
IDPF_TX_DESC_DTYPE_FLEX_L2TAG1_L2TAG2);
if (desc.flags & LIBETH_XDP_TX_LAST)
@@ -57,6 +113,10 @@ static inline void idpf_xdpsq_set_rs(const struct idpf_tx_queue *xdpsq)
{
u32 ntu, cmd;
+ if (static_branch_unlikely(&idpf_xdp_fb) &&
+ idpf_queue_has(FLOW_SCH_EN, xdpsq))
+ return;
+
ntu = xdpsq->next_to_use;
if (unlikely(!ntu))
ntu = xdpsq->desc_count;
diff --git a/include/net/libeth/xdp.h b/include/net/libeth/xdp.h
index 898723ab62e8..ed61d83bda2d 100644
--- a/include/net/libeth/xdp.h
+++ b/include/net/libeth/xdp.h
@@ -430,6 +430,19 @@ struct libeth_xdpsq {
struct libeth_xdpsq_lock *lock;
};
+/**
+ * libeth_xdpsq_to_sq - get SQ pointer from an XDPSQ pointer
+ * @xdpsq: &libeth_xdpsq corresponding to the queue
+ * @type: typeof() of the driver Tx queue structure
+ * @member: name of the NTU field inside @type
+ *
+ * Some of the sending callbacks take only &libeth_xdpsq pointer and no pointer
+ * to the actual driver Tx queue structure. Use this helper to quickly jump to
+ * the latter when needed.
+ */
+#define libeth_xdpsq_to_sq(xdpsq, type, member) \
+ container_of_const((xdpsq)->ntu, type, member)
+
/**
* struct libeth_xdp_tx_desc - abstraction for an XDP Tx descriptor
* @addr: DMA address of the frame
diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethernet/intel/idpf/idpf_txrx.c
index e32e368209ee..169af2812fb3 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c
@@ -179,10 +179,11 @@ static int idpf_tx_buf_alloc_all(struct idpf_tx_queue *tx_q)
/* Allocate book keeping buffers only. Buffers to be supplied to HW
* are allocated by kernel network stack and received as part of skb
*/
- if (idpf_queue_has(FLOW_SCH_EN, tx_q))
+ if (idpf_queue_has(FLOW_SCH_EN, tx_q) && !idpf_queue_has(XDP, tx_q))
tx_q->buf_pool_size = U16_MAX;
else
tx_q->buf_pool_size = tx_q->desc_count;
+
tx_q->tx_buf = kzalloc_objs(*tx_q->tx_buf, tx_q->buf_pool_size);
if (!tx_q->tx_buf)
return -ENOMEM;
@@ -227,7 +228,7 @@ static int idpf_tx_desc_alloc(const struct idpf_vport *vport,
idpf_xsk_setup_queue(vport, tx_q, VIRTCHNL2_QUEUE_TYPE_TX);
- if (!idpf_queue_has(FLOW_SCH_EN, tx_q))
+ if (!idpf_queue_has(FLOW_SCH_EN, tx_q) || idpf_queue_has(XDP, tx_q))
return 0;
refillq = tx_q->refillq;
@@ -1057,6 +1058,13 @@ static void idpf_clean_queue_set(const struct idpf_queue_set *qs)
if (idpf_queue_has(XDP, q->txq)) {
q->txq->pending = 0;
q->txq->xdp_tx = 0;
+
+ if (static_branch_unlikely(&idpf_xdp_fb) &&
+ idpf_queue_has(FLOW_SCH_EN, q->txq)) {
+ bitmap_zero(q->txq->pending_mask,
+ q->txq->desc_count);
+ q->txq->curr_tag = U32_MAX;
+ }
} else {
q->txq->txq_grp->num_completions_pending = 0;
}
@@ -1319,7 +1327,8 @@ static void idpf_txq_group_rel(struct idpf_q_vec_rsrc *rsrc)
if (!txq_grp->txqs[j])
continue;
- if (idpf_queue_has(FLOW_SCH_EN, txq_grp->txqs[j])) {
+ if (idpf_queue_has(FLOW_SCH_EN, txq_grp->txqs[j]) &&
+ !idpf_queue_has(XDP, txq_grp->txqs[j])) {
kfree(txq_grp->txqs[j]->refillq);
txq_grp->txqs[j]->refillq = NULL;
}
@@ -1442,6 +1451,8 @@ static int idpf_vport_init_fast_path_txqs(struct idpf_vport *vport,
{
struct idpf_ptp_vport_tx_tstamp_caps *caps = vport->tx_tstamp_caps;
struct work_struct *tstamp_task = &vport->tstamp_task;
+ /* __IDPF_Q_XDP is never set yet at this point */
+ u32 end = min_not_zero(rsrc->xdp_txq_offset, U32_MAX);
int k = 0;
vport->txqs = kzalloc_objs(*vport->txqs, rsrc->num_txq);
@@ -1456,7 +1467,7 @@ static int idpf_vport_init_fast_path_txqs(struct idpf_vport *vport,
vport->txqs[k] = tx_grp->txqs[j];
vport->txqs[k]->idx = k;
- if (!caps)
+ if (!caps || k >= end)
continue;
vport->txqs[k]->cached_tstamp_caps = caps;
diff --git a/drivers/net/ethernet/intel/idpf/xdp.c b/drivers/net/ethernet/intel/idpf/xdp.c
index 48320ea004df..0b4ead7fbcaf 100644
--- a/drivers/net/ethernet/intel/idpf/xdp.c
+++ b/drivers/net/ethernet/intel/idpf/xdp.c
@@ -7,6 +7,8 @@
#include "xdp.h"
#include "xsk.h"
+DEFINE_STATIC_KEY_FALSE(idpf_xdp_fb);
+
static int idpf_rxq_for_each(const struct idpf_q_vec_rsrc *rsrc,
int (*fn)(struct idpf_rx_queue *rxq, void *arg),
void *arg)
@@ -151,11 +153,13 @@ static void idpf_xdp_tx_timer(struct work_struct *work);
int idpf_xdpsqs_get(const struct idpf_vport *vport)
{
- struct libeth_xdpsq_timer **timers __free(kvfree) = NULL;
- struct net_device *dev;
+ struct libeth_xdpsq_timer **timers;
+ const struct net_device *dev;
+ unsigned long **masks;
+ bool fb, warn;
u32 sqs;
- if (!idpf_xdp_enabled(vport))
+ if (!idpf_xdp_enabled(vport) || !vport->num_xdp_txq)
return 0;
timers = kvzalloc_objs(*timers, vport->num_xdp_txq);
@@ -169,13 +173,42 @@ int idpf_xdpsqs_get(const struct idpf_vport *vport)
for (int j = i - 1; j >= 0; j--)
kfree(timers[j]);
- return -ENOMEM;
+ goto err_timers;
}
}
dev = vport->netdev;
sqs = vport->dflt_qv_rsrc.xdp_txq_offset;
+ fb = !idpf_is_cap_ena(vport->adapter, IDPF_OTHER_CAPS,
+ VIRTCHNL2_CAP_SPLITQ_QSCHED);
+ if (!fb)
+ goto setup;
+
+ masks = kvzalloc_objs(*masks, vport->num_xdp_txq);
+ if (!masks)
+ goto err_masks;
+
+ for (u32 i = 0; i < vport->num_xdp_txq; i++) {
+ masks[i] = bitmap_zalloc_node(vport->txqs[sqs + i]->desc_count,
+ GFP_KERNEL, cpu_to_mem(i));
+ if (!masks[i]) {
+ for (int j = i - 1; j >= 0; j--)
+ bitmap_free(masks[j]);
+
+ kvfree(masks);
+ goto err_masks;
+ }
+ }
+
+ warn = !static_key_enabled(&idpf_xdp_fb);
+ static_branch_inc(&idpf_xdp_fb);
+
+ if (warn && net_ratelimit())
+ netdev_warn(dev,
+ "The FW doesn't support Tx in FIFO mode, XDP Tx performance might be suboptimal\n");
+
+setup:
for (u32 i = sqs; i < vport->num_txq; i++) {
struct idpf_tx_queue *xdpsq = vport->txqs[i];
@@ -183,8 +216,8 @@ int idpf_xdpsqs_get(const struct idpf_vport *vport)
kfree(xdpsq->refillq);
xdpsq->refillq = NULL;
- idpf_queue_clear(FLOW_SCH_EN, xdpsq);
- idpf_queue_clear(FLOW_SCH_EN, xdpsq->complq);
+ idpf_queue_assign(FLOW_SCH_EN, xdpsq, fb);
+ idpf_queue_assign(FLOW_SCH_EN, xdpsq->complq, fb);
idpf_queue_set(NOIRQ, xdpsq);
idpf_queue_set(XDP, xdpsq);
idpf_queue_set(XDP, xdpsq->complq);
@@ -197,17 +230,36 @@ int idpf_xdpsqs_get(const struct idpf_vport *vport)
xdpsq->pending = 0;
xdpsq->xdp_tx = 0;
xdpsq->thresh = libeth_xdp_queue_threshold(xdpsq->desc_count);
+
+ if (static_branch_unlikely(&idpf_xdp_fb) && fb) {
+ xdpsq->pending_mask = masks[i - sqs];
+ xdpsq->curr_tag = U32_MAX;
+ }
}
+ kvfree(timers);
+ if (fb)
+ kvfree(masks);
+
return 0;
+
+err_masks:
+ for (u32 i = 0; i < vport->num_xdp_txq; i++)
+ kfree(timers[i]);
+
+err_timers:
+ kvfree(timers);
+
+ return -ENOMEM;
}
void idpf_xdpsqs_put(const struct idpf_vport *vport)
{
struct net_device *dev;
+ bool fb = false;
u32 sqs;
- if (!idpf_xdp_enabled(vport))
+ if (!idpf_xdp_enabled(vport) || !vport->num_xdp_txq)
return;
dev = vport->netdev;
@@ -222,10 +274,22 @@ void idpf_xdpsqs_put(const struct idpf_vport *vport)
libeth_xdpsq_deinit_timer(xdpsq->timer);
libeth_xdpsq_put(&xdpsq->xdp_lock, dev);
+ if (static_branch_unlikely(&idpf_xdp_fb) &&
+ idpf_queue_has(FLOW_SCH_EN, xdpsq)) {
+ bitmap_free(xdpsq->pending_mask);
+ xdpsq->pending_mask = NULL;
+ xdpsq->curr_tag = 0;
+
+ fb = true;
+ }
+
kfree(xdpsq->timer);
xdpsq->refillq = NULL;
idpf_queue_clear(NOIRQ, xdpsq);
}
+
+ if (fb)
+ static_branch_dec(&idpf_xdp_fb);
}
static int idpf_xdp_parse_cqe(const struct idpf_splitq_4b_tx_compl_desc *desc,
@@ -255,6 +319,65 @@ static int idpf_xdp_parse_cqe(const struct idpf_splitq_4b_tx_compl_desc *desc,
#endif
}
+static u32 idpf_xdpsq_poll_fb(struct idpf_tx_queue *xdpsq, u32 budget)
+{
+ struct idpf_compl_queue *cq = xdpsq->complq;
+ unsigned long *mask = xdpsq->pending_mask;
+ u32 done_frames, tx_cnt, new_ntc;
+ u32 ntc = cq->next_to_clean;
+ u32 cnt = cq->desc_count;
+ bool gen;
+
+ gen = idpf_queue_has(GEN_CHK, cq);
+
+ for (done_frames = 0; done_frames < budget; ) {
+ int ret;
+
+ ret = idpf_xdp_parse_cqe(&cq->comp[ntc].common, gen);
+ if (ret >= 0) {
+ __clear_bit(ret, mask);
+ done_frames++;
+
+ goto next;
+ }
+
+ switch (ret) {
+ case -ENODATA:
+ goto out;
+ case -EINVAL:
+ break;
+ }
+
+next:
+ if (unlikely(++ntc == cnt)) {
+ ntc = 0;
+ gen = !gen;
+ idpf_queue_change(GEN_CHK, cq);
+ }
+ }
+
+out:
+ cq->next_to_clean = ntc;
+
+ if (unlikely(!done_frames))
+ return 0;
+
+ tx_cnt = xdpsq->desc_count;
+
+ /* Don't go past next_to_use */
+ __set_bit(xdpsq->next_to_use, mask);
+
+ new_ntc = find_next_bit(mask, tx_cnt, xdpsq->next_to_clean);
+ done_frames = new_ntc - xdpsq->next_to_clean;
+
+ if (new_ntc == tx_cnt)
+ done_frames += find_first_bit(mask, tx_cnt);
+
+ __clear_bit(xdpsq->next_to_use, mask);
+
+ return done_frames;
+}
+
u32 idpf_xdpsq_poll(struct idpf_tx_queue *xdpsq, u32 budget)
{
struct idpf_compl_queue *cq = xdpsq->complq;
@@ -265,6 +388,10 @@ u32 idpf_xdpsq_poll(struct idpf_tx_queue *xdpsq, u32 budget)
u32 done_frames;
bool gen;
+ if (static_branch_unlikely(&idpf_xdp_fb) &&
+ idpf_queue_has(FLOW_SCH_EN, xdpsq))
+ return idpf_xdpsq_poll_fb(xdpsq, budget);
+
gen = idpf_queue_has(GEN_CHK, cq);
for (done_frames = 0; done_frames < budget; ) {
--
2.56.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH iwl-next v3 3/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support
2026-11-19 14:09 ` [PATCH iwl-next v3 3/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support Alexander Lobakin
@ 2026-11-19 15:02 ` sashiko-bot
0 siblings, 0 replies; 5+ messages in thread
From: sashiko-bot @ 2026-11-19 15:02 UTC (permalink / raw)
To: Alexander Lobakin, intel-wired-lan
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] idpf: unchecked cpu_to_mem() for i >= nr_cpu_ids in
idpf_xdpsqs_get()
- [Medium] idpf: premature CQE poll exit on out-of-order CQEs in
idpf_xdpsq_poll_fb()
--
commit 6119bbbc742528036710b0619a594398135c70b2
Author: Alexander Lobakin <aleksander.lobakin@intel.com>
idpf: add flow-based XDP fallback for FWs without Tx FIFO support
This commit introduces a flow-based Tx scheduling fallback for XDP and
AF_XDP on firmware that lacks queue-based (FIFO) Tx scheduling support,
using a per-queue bitmap to serialize out-of-order per-packet completion
tags into contiguous descriptor ranges.
> diff --git a/drivers/net/ethernet/intel/idpf/xdp.c b/drivers/net/ethernet/intel/idpf/xdp.c
> index 48320ea004dfb..0b4ead7fbcaf6 100644
> --- a/drivers/net/ethernet/intel/idpf/xdp.c
> +++ b/drivers/net/ethernet/intel/idpf/xdp.c
[ ... ]
> @@ -169,13 +173,42 @@ int idpf_xdpsqs_get(const struct idpf_vport *vport)
> for (int j = i - 1; j >= 0; j--)
> kfree(timers[j]);
>
> - return -ENOMEM;
> + goto err_timers;
> }
> }
>
> dev = vport->netdev;
> sqs = vport->dflt_qv_rsrc.xdp_txq_offset;
>
> + fb = !idpf_is_cap_ena(vport->adapter, IDPF_OTHER_CAPS,
> + VIRTCHNL2_CAP_SPLITQ_QSCHED);
> + if (!fb)
> + goto setup;
> +
> + masks = kvzalloc_objs(*masks, vport->num_xdp_txq);
> + if (!masks)
> + goto err_masks;
> +
> + for (u32 i = 0; i < vport->num_xdp_txq; i++) {
> + masks[i] = bitmap_zalloc_node(vport->txqs[sqs + i]->desc_count,
> + GFP_KERNEL, cpu_to_mem(i));
[Severity: High]
Can cpu_to_mem(i) index past nr_cpu_ids when vport->num_xdp_txq exceeds
the number of CPUs?
idpf_vport_calc_total_qs() sizes num_xdp_txq via libeth_xdpsq_num(),
which returns min(max(nr_cpu_ids, rxq), max - txq). When an
administrator configures more Rx queues than nr_cpu_ids via ethtool -L,
i reaches >= nr_cpu_ids (as it also does in the pre-existing timers[i]
loop above), indexing an uninitialized or out-of-bounds
__per_cpu_offset[i] entry on NUMA kernels. idpf_q_vector_to_mem() guards
this with:
cpu < nr_cpu_ids ? cpu_to_mem(cpu) : NUMA_NO_NODE
[ ... ]
> @@ -255,6 +319,65 @@ static int idpf_xdp_parse_cqe(const struct idpf_splitq_4b_tx_compl_desc *desc,
> #endif
> }
>
> +static u32 idpf_xdpsq_poll_fb(struct idpf_tx_queue *xdpsq, u32 budget)
> +{
> + struct idpf_compl_queue *cq = xdpsq->complq;
> + unsigned long *mask = xdpsq->pending_mask;
> + u32 done_frames, tx_cnt, new_ntc;
> + u32 ntc = cq->next_to_clean;
> + u32 cnt = cq->desc_count;
> + bool gen;
> +
> + gen = idpf_queue_has(GEN_CHK, cq);
> +
> + for (done_frames = 0; done_frames < budget; ) {
> + int ret;
> +
> + ret = idpf_xdp_parse_cqe(&cq->comp[ntc].common, gen);
> + if (ret >= 0) {
> + __clear_bit(ret, mask);
> + done_frames++;
> +
> + goto next;
> + }
> +
> + switch (ret) {
> + case -ENODATA:
> + goto out;
> + case -EINVAL:
> + break;
> + }
> +
> +next:
> + if (unlikely(++ntc == cnt)) {
> + ntc = 0;
> + gen = !gen;
> + idpf_queue_change(GEN_CHK, cq);
> + }
> + }
> +
> +out:
> + cq->next_to_clean = ntc;
> +
> + if (unlikely(!done_frames))
> + return 0;
> +
> + tx_cnt = xdpsq->desc_count;
> +
> + /* Don't go past next_to_use */
> + __set_bit(xdpsq->next_to_use, mask);
> +
> + new_ntc = find_next_bit(mask, tx_cnt, xdpsq->next_to_clean);
> + done_frames = new_ntc - xdpsq->next_to_clean;
> +
> + if (new_ntc == tx_cnt)
> + done_frames += find_first_bit(mask, tx_cnt);
> +
> + __clear_bit(xdpsq->next_to_use, mask);
> +
> + return done_frames;
> +}
[Severity: Medium]
Can idpf_xdpsq_poll_fb() return 0 after consuming out-of-order CQEs if
the descriptor at xdpsq->next_to_clean has not completed yet?
The loop stops after reading budget raw CQEs, but the return value is
overwritten with the contiguous run of cleared bits starting at
xdpsq->next_to_clean. If budget younger packets complete ahead of
xdpsq->next_to_clean, idpf_xdp_tx_prep() (which passes budget =
xdpsq->thresh) gets 0 reclaimed descriptors and drops frames when the
ring is full even if the CQE for xdpsq->next_to_clean is already waiting
next in cq. In addition, if the delayed cleanup timer runs
libeth_xdpsq_run_timer() and consumes out-of-order CQEs while
xdpsq->next_to_clean is still in flight in hardware, poll() returns 0 and
libeth_xdpsq_run_timer() stops re-arming the timer with pending frames
still on the queue.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119140938.1363722-1-aleksander.lobakin@intel.com?part=3
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-19 15:02 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 14:09 [PATCH iwl-next v3 0/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 1/3] idpf: always reserve one empty descriptor on XDPSQs Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 2/3] idpf: enforce strict ordering on BE when reading XDPCQEs Alexander Lobakin
2026-11-19 14:09 ` [PATCH iwl-next v3 3/3] idpf: add flow-based XDP fallback for FWs without Tx FIFO support Alexander Lobakin
2026-11-19 15:02 ` sashiko-bot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox