From: Pravin Pathak <pravinpathakdev@gmail.com>
To: dev@dpdk.org
Cc: jerinj@marvell.com, bruce.richardson@intel.com,
thomas@monjalon.net, david.marchand@redhat.com,
nipun.gupta@amd.com, chenbox@nvidia.com,
tirthendu.sarkar@intel.com, mattias.ronnblom@ericsson.com,
stephen@networkplumber.org,
Pravin Pathak <pravin.pathak.dev@gmail.com>
Subject: [PATCH v2 8/8] event/slb: address ARM64 build and all code reviews
Date: Tue, 17 Nov 2026 12:57:08 -0500 [thread overview]
Message-ID: <20261117175709.2346400-9-pravinpathakdev@gmail.com> (raw)
In-Reply-To: <20261117175709.2346400-1-pravinpathakdev@gmail.com>
From: Pravin Pathak <pravin.pathak.dev@gmail.com>
- Add missing standard C headers (<stdio.h>, <stdlib.h>, <string.h>)
to fix implicit declaration of abs() on ARM64 builds.
- Defer credit return for released ordered events until
slb_reorder_drain() reclaims the reorder buffer slot so the ordered
queue ring cannot wrap onto an undrained slot.
- Re-check buf[drain_idx].state after unlocking queue->reorder_lock in
slb_reorder_drain() to prevent a lost wakeup when another port marks
drain_idx READY/RELEASED and fails trylock just before unlock.
- Use release memory ordering when resetting reorder buffer slots to
SLB_REORDER_EMPTY and assert queue ring enqueue burst counts.
- Set default new_event_threshold to 3/4 of SLB_IN_RING_SIZE and check
new_event_threshold inside slb_port_acquire_credits() only when
borrowing from the central pool after checking the port's local
credit cache.
- Return 0 with rte_errno set to EINVAL/ENOSPC on release underflow or
credit exhaustion instead of aborting with RTE_VERIFY().
- Widen hold_count to uint32_t, clamp hold buffer copy lengths, and
define SLB_HOLD_BUFFER_REFILL_HEADROOM.
- Widen timer frequency multiplication for migration cooldown cycles
and avoid unsigned subtraction underflow in cooldown checks.
- Clean up queue resources via slb_queue_release() on setup failure
and unlink all port references (both single-link and q_entries) on
queue release.
- Guard against NULL q->q_ring in slb_dump() when dumping unconfigured
queues.
- Make internal functions (slb_dump, slb_reorder_drain,
slb_port_return_credits) static and update SLB documentation and
26.11 release notes.
Signed-off-by: Pravin Pathak <pravin.pathak.dev@gmail.com>
---
doc/guides/eventdevs/slb.rst | 14 +-
doc/guides/rel_notes/release_26_11.rst | 7 +
drivers/event/slb/slb_evdev.c | 129 +++++++++------
drivers/event/slb/slb_evdev.h | 19 +--
drivers/event/slb/slb_event.c | 219 +++++++++++++++++--------
drivers/event/slb/slb_xstats.c | 1 +
6 files changed, 248 insertions(+), 141 deletions(-)
diff --git a/doc/guides/eventdevs/slb.rst b/doc/guides/eventdevs/slb.rst
index b109a1e2ca..73c0491045 100644
--- a/doc/guides/eventdevs/slb.rst
+++ b/doc/guides/eventdevs/slb.rst
@@ -56,6 +56,14 @@ Limitations
-----------
* **Maximum Ports and Queues**: Supports up to 64 ports and 64 queues.
-* **Priorities and linking**: Supports two levels of service priority to link up to
- eight queues per port at each priority. SLB currently does not support event priority.
-* **Queues Types**: Does not support all-type queues.
+* **Priorities and linking**: Priority is configured per port-to-queue link via
+ ``rte_event_port_link()`` (supports two service priority levels, configured at compile time
+ via ``SLB_MAX_PRIORITIES``). SLB does not support per-queue priority
+ (``RTE_EVENT_DEV_CAP_QUEUE_QOS``), per-event priority (``RTE_EVENT_DEV_CAP_EVENT_QOS``),
+ or runtime port linking/unlinking after device start (``RTE_EVENT_DEV_CAP_RUNTIME_PORT_LINK``).
+* **Queues Types**: Does not support all-types queues (``RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES``).
+* **In-Flight Event Limit**: Each port maintains a local cache of up to ``SLB_PORT_MAX_CREDITS``
+ credits. To avoid credit-cache starvation when ``nb_events_limit`` is configured smaller than
+ ``nb_event_ports * SLB_PORT_MAX_CREDITS``, SLB raises the internal credit pool size to
+ ``max(nb_events_limit, nb_event_ports * SLB_PORT_MAX_CREDITS)`` (capped at ``SLB_MAX_EVENTS``),
+ so the device may hold more in-flight events than the configured ``nb_events_limit``.
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index 8ca8379cc6..1548ef0f15 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -312,6 +312,13 @@ New Features
with AMD-specific hardware access, a primary/secondary topology
and a packed scratchpad handshake.
+* **Added Software Load Balancer (SLB) eventdev PMD.**
+
+ Introduced a new poll-mode eventdev driver (``event_slb``) supporting
+ parallel, atomic, ordered, and single-link scheduling with dynamic
+ atomic flow migration and strict two-level port-queue service priorities.
+ Runtime port linking and unlinking after device start is not supported.
+
* **Supported auto-detection of uncore power driver.**
The uncore power library got automatic probing of multiple uncore drivers
diff --git a/drivers/event/slb/slb_evdev.c b/drivers/event/slb/slb_evdev.c
index 346b4b38ed..f47e08b3f9 100644
--- a/drivers/event/slb/slb_evdev.c
+++ b/drivers/event/slb/slb_evdev.c
@@ -3,6 +3,9 @@
*/
#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
#include <inttypes.h>
#include <rte_bitops.h>
#include <rte_cycles.h>
@@ -104,7 +107,7 @@ slb_port_def_conf(struct rte_eventdev *dev,
uint8_t port_id __rte_unused,
struct rte_event_port_conf *port_conf)
{
- int32_t threshold = SLB_IN_RING_SIZE / 4;
+ int32_t threshold = (3 * SLB_IN_RING_SIZE) / 4;
if (dev->data->dev_conf.nb_events_limit > 0 &&
threshold > dev->data->dev_conf.nb_events_limit)
@@ -122,6 +125,33 @@ slb_port_release(void *p __rte_unused)
{
}
+static void
+slb_port_unlink_queue_from_prio(struct slb_port *port, int prio, int slot)
+{
+ /* Shift the queue entries to overwriting the unlinked queue slot */
+ for (int k = slot; k < port->num_q[prio] - 1; k++)
+ port->q_entries[prio][k] = port->q_entries[prio][k + 1];
+
+ /* Decrement mapped queue count and Clear the last slot after shifting */
+ port->num_q[prio]--;
+ memset(&port->q_entries[prio][port->num_q[prio]], 0,
+ sizeof(struct slb_port_q_entry));
+ /* Adjust the next queue index if it points beyond the new count */
+ if (port->next_q[prio] >= port->num_q[prio])
+ port->next_q[prio] = 0;
+}
+
+/* Find the slot of a queue in a specific priority level, returning -1 if not found */
+static int
+slb_port_find_queue_slot(struct slb_port *port, int prio, uint8_t qid)
+{
+ for (int s = 0; s < port->num_q[prio]; s++) {
+ if (port->q_entries[prio][s].queue_id == qid)
+ return s;
+ }
+ return -1;
+}
+
/* Called to free queue resources on device close or reconfiguration */
static void
slb_queue_release(struct rte_eventdev *dev, uint8_t queue_id)
@@ -129,6 +159,26 @@ slb_queue_release(struct rte_eventdev *dev, uint8_t queue_id)
struct slb_evdev *slb = slb_pmd_priv(dev);
struct slb_queue *queue = &slb->queues[queue_id];
+ /* Unlink any port references before freeing queue rings */
+ for (int i = 0; i < slb->num_ports; i++) {
+ struct slb_port *port = &slb->ports[i];
+
+ if (port->single_linked_q == queue) {
+ port->single_linked_q = NULL;
+ port->single_linked_ring = NULL;
+ } else {
+ for (int pr = 0; pr < SLB_MAX_PRIORITIES; pr++) {
+ int slot = slb_port_find_queue_slot(
+ port, pr, queue_id);
+ if (slot >= 0)
+ slb_port_unlink_queue_from_prio(
+ port, pr, slot);
+ }
+ }
+ }
+ queue->serving_ports_mask = 0;
+ queue->num_serving_ports = 0;
+
if (queue->atomic_state) {
rte_free(queue->atomic_state);
queue->atomic_state = NULL;
@@ -160,6 +210,7 @@ slb_queue_setup(struct rte_eventdev *dev, uint8_t queue_id,
struct slb_queue *queue = &slb->queues[queue_id];
struct rte_event_ring *ring;
char ring_name[RTE_RING_NAMESIZE];
+ unsigned int ring_flags = RING_F_EXACT_SZ;
uint32_t q_size = (slb->max_inflight > 0) ?
(uint32_t)slb->max_inflight : SLB_IN_RING_SIZE;
@@ -172,25 +223,23 @@ slb_queue_setup(struct rte_eventdev *dev, uint8_t queue_id,
if (conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_SINGLE_LINK) {
/* MP -> SC ring for single-link queues */
queue->is_single_link = true;
- ring = rte_event_ring_create(ring_name, q_size,
- dev->data->socket_id, RING_F_SC_DEQ | RING_F_EXACT_SZ);
+ ring_flags |= RING_F_SC_DEQ;
} else if (conf->schedule_type == RTE_SCHED_TYPE_ATOMIC) {
/* MP -> SC ring for atomic queues main ring */
queue->is_single_link = false;
- ring = rte_event_ring_create(ring_name, q_size,
- dev->data->socket_id, RING_F_SC_DEQ | RING_F_EXACT_SZ);
+ ring_flags |= RING_F_SC_DEQ;
} else if (conf->schedule_type == RTE_SCHED_TYPE_PARALLEL ||
conf->schedule_type == RTE_SCHED_TYPE_ORDERED) {
/* MP -> MC ring for parallel or ordered queues */
queue->is_single_link = false;
- ring = rte_event_ring_create(ring_name, q_size,
- dev->data->socket_id, RING_F_EXACT_SZ);
} else {
SLB_LOG_DP_LINE(ERR, "Unsupported schedule_type %u",
conf->schedule_type);
return -ENOTSUP;
}
+ ring = rte_event_ring_create(ring_name, q_size, dev->data->socket_id,
+ ring_flags);
if (ring == NULL)
return -ENOMEM;
@@ -209,7 +258,7 @@ slb_queue_setup(struct rte_eventdev *dev, uint8_t queue_id,
sizeof(struct slb_atomic_state),
RTE_CACHE_LINE_SIZE, dev->data->socket_id);
if (queue->atomic_state == NULL) {
- rte_event_ring_free(ring);
+ slb_queue_release(dev, queue_id);
return -ENOMEM;
}
for (int i = 0; i < SLB_ATOMIC_FLOWS; i++)
@@ -236,7 +285,7 @@ slb_queue_setup(struct rte_eventdev *dev, uint8_t queue_id,
sizeof(struct slb_reorder_entry) * ring->r.size,
RTE_CACHE_LINE_SIZE, dev->data->socket_id);
if (queue->reorder_buffer == NULL) {
- rte_event_ring_free(ring);
+ slb_queue_release(dev, queue_id);
return -ENOMEM;
}
queue->reorder_drain_idx = 0;
@@ -261,33 +310,6 @@ slb_queue_def_conf(struct rte_eventdev *dev __rte_unused,
};
}
-static void
-slb_port_unlink_queue_from_prio(struct slb_port *port, int prio, int slot)
-{
- /* Shift the queue entries to overwriting the unlinked queue slot */
- for (int k = slot; k < port->num_q[prio] - 1; k++)
- port->q_entries[prio][k] = port->q_entries[prio][k + 1];
-
- /* Decrement mapped queue count and Clear the last slot after shifting */
- port->num_q[prio]--;
- memset(&port->q_entries[prio][port->num_q[prio]], 0,
- sizeof(struct slb_port_q_entry));
- /* Adjust the next queue index if it points beyond the new count */
- if (port->next_q[prio] >= port->num_q[prio])
- port->next_q[prio] = 0;
-}
-
-/* Find the slot of a queue in a specific priority level, returning -1 if not found */
-static int
-slb_port_find_queue_slot(struct slb_port *port, int prio, uint8_t qid)
-{
- for (int s = 0; s < port->num_q[prio]; s++) {
- if (port->q_entries[prio][s].queue_id == qid)
- return s;
- }
- return -1;
-}
-
/* Link a queue to a port at a specific priority level */
static int
slb_port_link_one(struct slb_port *port, struct slb_queue *queue,
@@ -303,15 +325,14 @@ slb_port_link_one(struct slb_port *port, struct slb_queue *queue,
rte_errno = EINVAL;
return -EINVAL;
}
- /* Only one queue can be linked to a single-linked port */
- if (port->single_linked_q != NULL) {
- if (port->single_linked_q == queue)
- return 0;
- rte_errno = EINVAL;
- return -EINVAL;
- }
- /* A single-link queue can only be linked to one port */
- if (queue->serving_ports_mask != 0) {
+ /* Re-linking the same queue to the same port is a no-op */
+ if (port->single_linked_q == queue)
+ return 0;
+ /* Only one queue can be linked to a single-linked port, and a
+ * single-link queue can only be linked to one port.
+ */
+ if (port->single_linked_q != NULL ||
+ queue->serving_ports_mask != 0) {
rte_errno = EINVAL;
return -EINVAL;
}
@@ -476,7 +497,7 @@ slb_info_get(struct rte_eventdev *dev __rte_unused,
.driver_name = SLB_PMD_NAME,
.max_event_queues = SLB_MAX_QUEUES,
.max_event_queue_flows = SLB_MAX_FLOWS,
- .max_event_queue_priority_levels = SLB_MAX_PRIORITIES,
+ .max_event_queue_priority_levels = 1,
.max_event_priority_levels = 1,
.max_event_ports = SLB_MAX_PORTS,
.max_event_port_dequeue_depth = SLB_MAX_PORT_DEQUEUE_DEPTH,
@@ -534,14 +555,14 @@ slb_configure(const struct rte_eventdev *dev)
slb->num_queues = conf->nb_event_queues;
/* Compute minimum credits required to avoid deadlock */
- min_max_in_flight = conf->nb_event_ports * SLB_PORT_MAX_CREDITS;
+ min_max_in_flight = (int32_t)conf->nb_event_ports * SLB_PORT_MAX_CREDITS;
if (min_max_in_flight > SLB_MAX_EVENTS)
min_max_in_flight = SLB_MAX_EVENTS;
/* Set credit pool size to max of nb_events_limit and minimum required to avoid deadlock */
slb->max_inflight = RTE_MAX(conf->nb_events_limit, min_max_in_flight);
slb->migration_cooldown_cycles =
- (rte_get_timer_hz() * SLB_MIGRATION_COOLDOWN_MS) / 1000;
+ ((uint64_t)rte_get_timer_hz() * SLB_MIGRATION_COOLDOWN_MS) / 1000;
return 0;
}
@@ -613,7 +634,7 @@ slb_stop(struct rte_eventdev *dev)
struct slb_atomic_state *astate = q->atomic_state;
/* Call user callback for each held event */
- for (uint16_t h = 0; h < astate->hold_count; h++) {
+ for (uint32_t h = 0; h < astate->hold_count; h++) {
if (flush != NULL)
flush(dev_id, astate->hold_buffer[h],
flush_arg);
@@ -655,7 +676,7 @@ slb_stop(struct rte_eventdev *dev)
rte_atomic_store_explicit(
&q->reorder_buffer[s].state,
SLB_REORDER_EMPTY,
- rte_memory_order_relaxed);
+ rte_memory_order_release);
}
q->reorder_drain_idx = 0;
}
@@ -696,7 +717,7 @@ slb_close(struct rte_eventdev *dev)
}
/* Dump the event device current state and statistics */
-void
+static void
slb_dump(struct rte_eventdev *dev, FILE *f)
{
struct slb_evdev *slb = slb_pmd_priv(dev);
@@ -778,6 +799,12 @@ slb_dump(struct rte_eventdev *dev, FILE *f)
fprintf(f, "\n ALL QUEUES STATUS:\n");
for (i = 0; i < slb->num_queues; i++) {
struct slb_queue *q = &slb->queues[i];
+
+ if (q->q_ring == NULL) {
+ fprintf(f, " Queue %2u: (not configured)\n", i);
+ continue;
+ }
+
uint32_t count = rte_event_ring_count(q->q_ring);
uint32_t ring_size = rte_event_ring_get_capacity(q->q_ring);
diff --git a/drivers/event/slb/slb_evdev.h b/drivers/event/slb/slb_evdev.h
index c01290e539..457376d1d6 100644
--- a/drivers/event/slb/slb_evdev.h
+++ b/drivers/event/slb/slb_evdev.h
@@ -1,4 +1,5 @@
/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Ericsson AB
* Copyright(c) 2026 Pravin Pathak <pravin.pathak.dev@gmail.com>
*/
@@ -37,13 +38,7 @@
/* Maximum number of queues that can be linked to a port per priority */
#define SLB_PORT_MAX_Q_MAP_PER_PRIO (SLB_MAX_QUEUES)
/* Number of priority levels supported by SLB. Currently fixed to 2 */
-#ifndef SLB_MAX_PRIORITIES
#define SLB_MAX_PRIORITIES 2
-#endif
-
-#if (SLB_MAX_PRIORITIES != 2)
-#error "SLB_MAX_PRIORITIES must be set to 2 for now"
-#endif
/* Total max mapping slots (per-prio * priorities) */
#define SLB_PORT_MAX_Q_MAP (SLB_PORT_MAX_Q_MAP_PER_PRIO * SLB_MAX_PRIORITIES)
@@ -87,8 +82,9 @@ struct slb_reorder_entry {
#define SLB_ATOMIC_FLOWS 2048
#define SLB_ATOMIC_FLOWS_MASK (SLB_ATOMIC_FLOWS - 1)
#define SLB_HOLD_BUFFER_SIZE (SLB_MAX_PORT_DEQUEUE_DEPTH * 64)
+#define SLB_HOLD_BUFFER_REFILL_HEADROOM 3
#define SLB_MIGRATION_LOAD_THRESHOLD 16
-#define SLB_MIGRATION_COOLDOWN_MS 500
+#define SLB_MIGRATION_COOLDOWN_MS 500ULL
/* Per-flow entry for atomic scheduling affinity */
struct slb_flow {
@@ -120,7 +116,7 @@ struct slb_atomic_state {
struct slb_flow_map flow_map;
uint32_t next_port_rr;
rte_spinlock_t refill_lock;
- uint16_t hold_count;
+ uint32_t hold_count;
struct slb_migration_state migration;
uint64_t migrations_count;
uint64_t flow_last_migrated[SLB_ATOMIC_FLOWS];
@@ -223,13 +219,6 @@ uint16_t slb_event_enqueue_forward_burst(void *port,
uint16_t slb_event_dequeue_burst(void *port, struct rte_event *events,
uint16_t num, uint64_t wait);
-void slb_port_return_credits(struct slb_evdev *slb, struct slb_port *port,
- int32_t credits);
-
-void slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue);
-
-void slb_dump(struct rte_eventdev *dev, FILE *f);
-
int slb_xstats_get_names(const struct rte_eventdev *dev,
enum rte_event_dev_xstats_mode mode,
uint8_t queue_port_id,
diff --git a/drivers/event/slb/slb_event.c b/drivers/event/slb/slb_event.c
index 3d0436e4a0..f53e44b61b 100644
--- a/drivers/event/slb/slb_event.c
+++ b/drivers/event/slb/slb_event.c
@@ -1,11 +1,15 @@
/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Ericsson AB
* Copyright(c) 2026 Pravin Pathak <pravin.pathak.dev@gmail.com>
*/
#include <stdbool.h>
+#include <stdlib.h>
+#include <string.h>
#include <limits.h>
#include <rte_bitops.h>
#include <rte_cycles.h>
+#include <rte_errno.h>
#include <rte_ring_elem.h>
#include <rte_stdatomic.h>
@@ -31,6 +35,9 @@ slb_port_acquire_credits(struct slb_evdev *slb, struct slb_port *port,
total_inflight = rte_atomic_load_explicit(&slb->credits_on_loan,
rte_memory_order_relaxed);
+ if (unlikely(total_inflight > port->new_event_threshold))
+ return false;
+
available = slb->max_inflight - total_inflight;
required_credits = RTE_MAX(abs(remaining_credits), SLB_PORT_MIN_CREDITS);
@@ -61,7 +68,7 @@ slb_port_acquire_credits(struct slb_evdev *slb, struct slb_port *port,
/* Port returning credits (Terminated events). Keep local cache.
* If local cache exceeds the threshold, return excess credits to the pool
*/
-void
+static void
slb_port_return_credits(struct slb_evdev *slb, struct slb_port *port,
int32_t credits)
{
@@ -103,16 +110,25 @@ slb_port_queue_enqueue_burst_stats(struct slb_port *source_port,
#define REORDER_BURST_SZ 64
-/* Drain contiguous ready events from ordered queue's reorder buffer */
-void
-slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
+/* Drain contiguous ready events from ordered queue's reorder buffer.
+ * Thread-safe: uses queue->reorder_lock internally.
+ */
+static void
+slb_reorder_drain(struct slb_evdev *slb, struct slb_port *port,
+ struct slb_queue *queue)
{
if (queue == NULL || queue->reorder_buffer == NULL)
return;
+ int32_t released_credits = 0;
+
+retry:
/* Only one thread can drain the reorder buffer at a time */
- if (!rte_spinlock_trylock(&queue->reorder_lock))
+ if (!rte_spinlock_trylock(&queue->reorder_lock)) {
+ if (released_credits > 0)
+ slb_port_return_credits(slb, port, released_credits);
return;
+ }
uint32_t drain_idx = queue->reorder_drain_idx;
uint32_t mask = queue->q_ring->r.mask;
@@ -120,7 +136,7 @@ slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
uint8_t state;
/* Drain events from the current drain index till an empty slot.
- * Released event slots need no processing except marking them as empty.
+ * Released event slots return their held credit once drained.
*/
while ((state = rte_atomic_load_explicit(&buf[drain_idx].state,
rte_memory_order_acquire)) !=
@@ -128,8 +144,9 @@ slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
if (state == SLB_REORDER_RELEASED) {
rte_atomic_store_explicit(&buf[drain_idx].state,
SLB_REORDER_EMPTY,
- rte_memory_order_relaxed);
+ rte_memory_order_release);
drain_idx = (drain_idx + 1) & mask;
+ released_credits++;
continue;
}
@@ -142,7 +159,7 @@ slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
burst[burst_size++] = buf[drain_idx].event;
rte_atomic_store_explicit(&buf[drain_idx].state,
SLB_REORDER_EMPTY,
- rte_memory_order_relaxed);
+ rte_memory_order_release);
drain_idx = (drain_idx + 1) & mask;
} while (burst_size < REORDER_BURST_SZ &&
rte_atomic_load_explicit(&buf[drain_idx].state,
@@ -151,12 +168,25 @@ slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
buf[drain_idx].event.queue_id == target_qid);
/* Target queue is sized to max_inflight and can never be full */
- rte_event_ring_enqueue_burst(slb->queues[target_qid].q_ring,
- burst, burst_size, NULL);
+ uint16_t enq = rte_event_ring_enqueue_burst(
+ slb->queues[target_qid].q_ring, burst, burst_size, NULL);
+ RTE_ASSERT(enq == burst_size);
}
queue->reorder_drain_idx = drain_idx;
rte_spinlock_unlock(&queue->reorder_lock);
+
+ /* Re-check after unlock in case another port marked drain_idx
+ * READY/RELEASED and failed trylock before we unlocked.
+ */
+ rte_atomic_thread_fence(rte_memory_order_seq_cst);
+ if (unlikely(rte_atomic_load_explicit(&buf[drain_idx].state,
+ rte_memory_order_acquire) !=
+ SLB_REORDER_EMPTY))
+ goto retry;
+
+ if (released_credits > 0)
+ slb_port_return_credits(slb, port, released_credits);
}
/* Hash 20-bit flow ID into atomic flow slot index (SLB_ATOMIC_FLOWS) */
@@ -212,7 +242,7 @@ slb_find_lightest_flow_on_port(struct slb_queue *queue, uint8_t port_id,
rte_memory_order_relaxed) == (int32_t)port_id) {
/* Skip flows that have been migrated recently */
if (astate->flow_last_migrated[f] != 0 &&
- (now - astate->flow_last_migrated[f]) < cooldown_cycles)
+ now < astate->flow_last_migrated[f] + cooldown_cycles)
continue;
if (counts[f] < min_events) {
@@ -256,7 +286,7 @@ slb_check_and_trigger_migration(struct slb_evdev *slb, struct slb_queue *queue,
/* Port is still in cooldown period */
if (slb->ports[full_port_id].last_migrated_time != 0 &&
- (now - slb->ports[full_port_id].last_migrated_time) < cooldown_cycles)
+ now < slb->ports[full_port_id].last_migrated_time + cooldown_cycles)
return;
/* Find flow to migrate*/
@@ -291,7 +321,7 @@ slb_find_migration_target_port(struct slb_evdev *slb, struct slb_queue *queue,
continue;
if (cp->last_migrated_time == 0 ||
- (now - cp->last_migrated_time) >= cooldown_cycles) {
+ now >= cp->last_migrated_time + cooldown_cycles) {
if ((int)astate->port_stats[cand_port].flow_count < min_flows) {
min_flows = (int)astate->port_stats[cand_port].flow_count;
target_port = cand_port;
@@ -315,7 +345,7 @@ slb_complete_flow_migration(struct slb_evdev *slb, struct slb_atomic_state *asta
target_port, rte_memory_order_release);
/* Switch all events in hold buffer to the target port */
- for (uint16_t h = 0; h < astate->hold_count; h++) {
+ for (uint32_t h = 0; h < astate->hold_count; h++) {
if (astate->hold_ports[h] == mig_port &&
slb_flow_hash(astate->hold_buffer[h].flow_id) == f_idx) {
astate->hold_ports[h] = (uint8_t)target_port;
@@ -429,38 +459,38 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
/* 1. Process Hold Buffer events */
if (astate->hold_count > 0) {
- uint16_t retained = 0;
- uint16_t count = astate->hold_count;
+ uint32_t retained = 0;
+ uint32_t count = astate->hold_count;
+ uint32_t h_idx = 0;
- i = 0;
- while (i < count) {
- uint8_t dest_port_id = astate->hold_ports[i];
- uint16_t burst_len = 1;
+ while (h_idx < count) {
+ uint8_t dest_port_id = astate->hold_ports[h_idx];
+ uint32_t burst_len = 1;
/* Collect consecutive events for the same destination port */
- while ((i + burst_len) < count &&
- astate->hold_ports[i + burst_len] == dest_port_id)
+ while ((h_idx + burst_len) < count &&
+ astate->hold_ports[h_idx + burst_len] == dest_port_id)
burst_len++;
/* If destination port is full, retain the event in hold buffer*/
if (full_ports_mask & (1ULL << dest_port_id)) {
- if (retained != i) {
+ if (retained != h_idx) {
memmove(&astate->hold_buffer[retained],
- &astate->hold_buffer[i],
+ &astate->hold_buffer[h_idx],
burst_len * sizeof(struct rte_event));
memmove(&astate->hold_ports[retained],
- &astate->hold_ports[i],
+ &astate->hold_ports[h_idx],
burst_len);
}
retained += burst_len;
- i += burst_len;
+ h_idx += burst_len;
continue;
}
/* Enqueue events to the destination port */
uint16_t enqueued = rte_event_ring_enqueue_burst(
queue->per_port_rings[dest_port_id],
- &astate->hold_buffer[i], burst_len, NULL);
+ &astate->hold_buffer[h_idx], burst_len, NULL);
astate->port_stats[dest_port_id].q_enqueued += enqueued;
/* If enqueue was successful then decrement the load counter */
@@ -470,8 +500,8 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
/* If enqueue was not successful, retain the event in hold buffer */
if (enqueued < burst_len) {
- uint16_t rem = burst_len - enqueued;
- uint16_t start_idx = i + enqueued;
+ uint32_t rem = burst_len - enqueued;
+ uint32_t start_idx = h_idx + enqueued;
full_ports_mask |= (1ULL << dest_port_id);
if (retained != start_idx) {
@@ -489,7 +519,7 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
slb, queue, astate, dest_port_id);
}
- i += burst_len;
+ h_idx += burst_len;
}
astate->hold_count = retained;
@@ -497,7 +527,7 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
* if hold buffer does not have enough free space.
*/
if (SLB_HOLD_BUFFER_SIZE - astate->hold_count <
- 3 * RTE_DIM(rcv_events)) {
+ SLB_HOLD_BUFFER_REFILL_HEADROOM * RTE_DIM(rcv_events)) {
rte_spinlock_unlock(&astate->refill_lock);
return;
}
@@ -589,11 +619,17 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
/* If destination port is full, retain events in hold buffer */
if (full_ports_mask & (1ULL << p)) {
- rte_memcpy(&astate->hold_buffer[astate->hold_count],
- &sorted_events[p_start],
- p_count * sizeof(struct rte_event));
- memset(&astate->hold_ports[astate->hold_count], p, p_count);
- astate->hold_count += p_count;
+ uint32_t to_copy = RTE_MIN(
+ (uint32_t)p_count,
+ SLB_HOLD_BUFFER_SIZE - astate->hold_count);
+ if (likely(to_copy > 0)) {
+ rte_memcpy(&astate->hold_buffer[astate->hold_count],
+ &sorted_events[p_start],
+ to_copy * sizeof(struct rte_event));
+ memset(&astate->hold_ports[astate->hold_count],
+ p, to_copy);
+ astate->hold_count += to_copy;
+ }
continue;
}
@@ -607,14 +643,20 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
if (unlikely(astate->port_stats[p].load_counter > 0))
astate->port_stats[p].load_counter--;
} else {
- uint16_t rem = p_count - enqueued;
+ uint32_t rem = p_count - enqueued;
+ uint32_t to_copy = RTE_MIN(
+ rem,
+ SLB_HOLD_BUFFER_SIZE - astate->hold_count);
/* Mark the destination port as full and retain remaining events */
full_ports_mask |= (1ULL << p);
- rte_memcpy(&astate->hold_buffer[astate->hold_count],
- &sorted_events[p_start + enqueued],
- rem * sizeof(struct rte_event));
- memset(&astate->hold_ports[astate->hold_count], p, rem);
- astate->hold_count += rem;
+ if (likely(to_copy > 0)) {
+ rte_memcpy(&astate->hold_buffer[astate->hold_count],
+ &sorted_events[p_start + enqueued],
+ to_copy * sizeof(struct rte_event));
+ memset(&astate->hold_ports[astate->hold_count],
+ p, to_copy);
+ astate->hold_count += to_copy;
+ }
/* If not a single event was enqueued, check for migration */
if (enqueued == 0)
slb_check_and_trigger_migration(
@@ -626,6 +668,16 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
rte_spinlock_unlock(&astate->refill_lock);
}
+/* Enqueue a burst of events from source_port.
+ *
+ * Design invariant: Every queue ring (q_ring) is created with RING_F_EXACT_SZ
+ * and capacity equal to slb->max_inflight. Because NEW events must acquire
+ * credits from slb->max_inflight upfront and FORWARD events are verified
+ * against source_port->pending_releases, the total number of in-flight events
+ * across the entire device can never exceed slb->max_inflight. Consequently,
+ * q_ring can never fill and rte_event_ring_enqueue_burst() onto q_ring is
+ * guaranteed to enqueue the full burst.
+ */
static inline uint16_t
slb_event_enqueue_burst_generic(struct slb_port *source_port,
const struct rte_event events[],
@@ -656,25 +708,22 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
}
}
- /* Check credit threshold if there are NEW events */
- if (num_new > 0) {
- if (unlikely(rte_atomic_load_explicit(&slb->credits_on_loan,
- rte_memory_order_relaxed) >
- source_port->new_event_threshold))
- return 0;
+ /* Check for release underflow before acquiring credits */
+ cur_pending = source_port->pending_releases;
+ if (unlikely(num_forward + num_release > cur_pending)) {
+ rte_errno = EINVAL;
+ return 0;
+ }
+ /* Acquire credits if there are NEW events */
+ if (num_new > 0) {
if (unlikely(!slb_port_acquire_credits(slb, source_port,
- num_new)))
+ num_new))) {
+ rte_errno = ENOSPC;
return 0;
+ }
}
- /* Return credits for RELEASE events */
- if (num_release > 0)
- slb_port_return_credits(slb, source_port, num_release);
-
- /* Check for release underflow and balance pending releases */
- cur_pending = source_port->pending_releases;
- RTE_VERIFY(num_forward + num_release <= cur_pending);
source_port->pending_releases = cur_pending - (num_forward + num_release);
slb_port_enqueue_stats(source_port, num_new, num_forward, num_release);
@@ -695,8 +744,9 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
/* Enqueue the entire burst if going to the same queue */
if (likely(burst_size == events_len)) {
- rte_event_ring_enqueue_burst(slb->queues[q0].q_ring,
- events, events_len, NULL);
+ uint16_t enq = rte_event_ring_enqueue_burst(
+ slb->queues[q0].q_ring, events, events_len, NULL);
+ RTE_ASSERT(enq == events_len);
slb_port_queue_enqueue_burst_stats(source_port, q0,
events_len);
} else { /* Sort events by destination queue */
@@ -748,11 +798,13 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
uint8_t q = rte_ctz64(qmask);
uint16_t count = queue_counts[q];
uint16_t offset = queue_offsets[q];
+ uint16_t enq;
qmask &= (qmask - 1);
- rte_event_ring_enqueue_burst(slb->queues[q].q_ring,
- &sorted_events[offset],
- count, NULL);
+ enq = rte_event_ring_enqueue_burst(
+ slb->queues[q].q_ring,
+ &sorted_events[offset], count, NULL);
+ RTE_ASSERT(enq == count);
slb_port_queue_enqueue_burst_stats(source_port, q,
count);
}
@@ -765,6 +817,7 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
/* General path for bursts with any mix of events */
uint64_t ordered_drain_mask = 0;
+ uint16_t ordered_releases = 0;
uint16_t freed_count = 0;
uint16_t head = source_port->history_head;
uint16_t n_enq = 0;
@@ -792,11 +845,14 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
uint16_t slot = hist->seq_slot;
if (ev->op == RTE_EVENT_OP_RELEASE) {
- /* Release the reorder buffer entry */
+ /* Release the reorder buffer entry; credit is
+ * returned when slb_reorder_drain reclaims slot
+ */
rte_atomic_store_explicit(
&orig_q->reorder_buffer[slot].state,
SLB_REORDER_RELEASED,
rte_memory_order_release);
+ ordered_releases++;
} else {
/* Put event in reorder buffer */
orig_q->reorder_buffer[slot].event = *ev;
@@ -874,9 +930,11 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
n_enq == events_len)) {
/* Entire batch is going to a single queue */
uint8_t q = rte_ctz64(batch_queues_mask);
+ uint16_t enq;
- rte_event_ring_enqueue_burst(slb->queues[q].q_ring,
- events, events_len, NULL);
+ enq = rte_event_ring_enqueue_burst(
+ slb->queues[q].q_ring, events, events_len, NULL);
+ RTE_ASSERT(enq == events_len);
slb_port_queue_enqueue_burst_stats(source_port, q,
events_len);
} else {
@@ -909,17 +967,26 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
uint8_t q = rte_ctz64(qmask);
uint16_t count = queue_counts[q];
uint16_t offset = queue_offsets[q];
+ uint16_t enq;
qmask &= (qmask - 1);
- rte_event_ring_enqueue_burst(slb->queues[q].q_ring,
- &sorted_events[offset],
- count, NULL);
+ enq = rte_event_ring_enqueue_burst(
+ slb->queues[q].q_ring,
+ &sorted_events[offset], count, NULL);
+ RTE_ASSERT(enq == count);
slb_port_queue_enqueue_burst_stats(source_port, q,
count);
}
}
}
+ /* Return credits for non-ordered RELEASE events immediately; ordered
+ * RELEASE credits are returned when slb_reorder_drain reclaims the slot.
+ */
+ if (num_release > ordered_releases)
+ slb_port_return_credits(slb, source_port,
+ num_release - ordered_releases);
+
/* Update the history list head once after all frees*/
if (freed_count > 0) {
source_port->history_head = head;
@@ -931,7 +998,7 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
uint8_t qid = rte_ctz64(ordered_drain_mask);
ordered_drain_mask &= ~(1ULL << qid);
- slb_reorder_drain(slb, &slb->queues[qid]);
+ slb_reorder_drain(slb, source_port, &slb->queues[qid]);
}
SLB_LOG_DP_PORT_LINE(DEBUG, source_port->id,
@@ -1134,14 +1201,16 @@ slb_port_process_implicit_release(struct slb_port *port, struct slb_evdev *slb,
uint16_t pending)
{
uint16_t head;
+ uint16_t ordered_releases = 0;
uint64_t ordered_drain_mask = 0;
- slb_port_return_credits(slb, port, pending);
port->pending_releases = 0;
if (port->is_single_linked ||
- port->history_avail >= SLB_PORT_HISTORY_MAX_TOKENS)
+ port->history_avail >= SLB_PORT_HISTORY_MAX_TOKENS) {
+ slb_port_return_credits(slb, port, pending);
return;
+ }
/* Process all history entries */
head = port->history_head;
@@ -1151,7 +1220,9 @@ slb_port_process_implicit_release(struct slb_port *port, struct slb_evdev *slb,
if (!hist->active)
continue;
- /* Ordered releases will free the reorder buffer slot*/
+ /* Ordered releases will free the reorder buffer slot; credit is
+ * returned when slb_reorder_drain reclaims the slot.
+ */
if (hist->sched_type == RTE_SCHED_TYPE_ORDERED) {
struct slb_queue *oq = &slb->queues[hist->orig_queue_id];
@@ -1161,6 +1232,7 @@ slb_port_process_implicit_release(struct slb_port *port, struct slb_evdev *slb,
SLB_REORDER_RELEASED,
rte_memory_order_release);
ordered_drain_mask |= (1ULL << hist->orig_queue_id);
+ ordered_releases++;
}
} else if (hist->sched_type == RTE_SCHED_TYPE_ATOMIC) {
/* Atomic releases are counted as returned events */
@@ -1180,12 +1252,15 @@ slb_port_process_implicit_release(struct slb_port *port, struct slb_evdev *slb,
}
port->history_head = head;
+ if (pending > ordered_releases)
+ slb_port_return_credits(slb, port, pending - ordered_releases);
+
/* If there are ordered releases, call the reorder drain function */
while (ordered_drain_mask != 0) {
uint8_t qid = rte_ctz64(ordered_drain_mask);
ordered_drain_mask &= ~(1ULL << qid);
- slb_reorder_drain(slb, &slb->queues[qid]);
+ slb_reorder_drain(slb, port, &slb->queues[qid]);
}
}
diff --git a/drivers/event/slb/slb_xstats.c b/drivers/event/slb/slb_xstats.c
index fdb7dd6cc5..ff898a3ef2 100644
--- a/drivers/event/slb/slb_xstats.c
+++ b/drivers/event/slb/slb_xstats.c
@@ -1,4 +1,5 @@
/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018-2019 Ericsson AB
* Copyright(c) 2026 Pravin Pathak <pravin.pathak.dev@gmail.com>
*/
--
2.53.0
next prev parent reply other threads:[~2026-11-17 17:58 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-07 1:30 [PATCH v1 0/8] event/slb: introducing software load balancer eventdev PMD Pravin Pathak
2026-11-07 1:30 ` [PATCH v1 1/8] event/slb: introduce software load balancer PMD Pravin Pathak
2026-11-07 1:30 ` [PATCH v1 2/8] event/slb: add support for ordered queues Pravin Pathak
2026-11-07 1:30 ` [PATCH v1 3/8] event/slb: add support for atomic queues Pravin Pathak
2026-11-07 1:30 ` [PATCH v1 4/8] event/slb: add dynamic flow migration " Pravin Pathak
2026-11-07 1:30 ` [PATCH v1 5/8] event/slb: add extended statistics and diagnostics Pravin Pathak
2026-11-07 1:30 ` [PATCH v1 6/8] event/slb: add self-test and documentation Pravin Pathak
2026-11-07 1:30 ` [PATCH v1 7/8] event/slb: update resets and credit defaults Pravin Pathak
2026-11-07 1:30 ` [PATCH v1 8/8] event/slb: improve port linking, config checks and atomic refill Pravin Pathak
2026-11-17 17:57 ` [PATCH v2 0/8] event/slb: introducing software load balancer eventdev PMD Pravin Pathak
2026-11-17 17:57 ` [PATCH v2 1/8] event/slb: introduce software load balancer PMD Pravin Pathak
2026-11-17 17:57 ` [PATCH v2 2/8] event/slb: add support for ordered queues Pravin Pathak
2026-11-17 17:57 ` [PATCH v2 3/8] event/slb: add support for atomic queues Pravin Pathak
2026-11-17 17:57 ` [PATCH v2 4/8] event/slb: add dynamic flow migration " Pravin Pathak
2026-11-17 17:57 ` [PATCH v2 5/8] event/slb: add extended statistics and diagnostics Pravin Pathak
2026-11-17 17:57 ` [PATCH v2 6/8] event/slb: add self-test and documentation Pravin Pathak
2026-11-17 17:57 ` [PATCH v2 7/8] event/slb: fine tuning and fixes Pravin Pathak
2026-11-17 17:57 ` Pravin Pathak [this message]
2026-11-17 18:46 ` [PATCH v2 0/8] event/slb: introducing software load balancer eventdev PMD Stephen Hemminger
2026-11-07 7:29 ` [PATCH v1 " Jerin Jacob
2026-11-08 17:10 ` Stephen Hemminger
2026-11-09 2:30 ` Pravin Pathak
2026-11-09 12:55 ` Jerin Jacob
2026-11-11 14:22 ` Sarkar, Tirthendu
2026-11-17 17:09 ` Pravin Pathak
2026-11-17 17:25 ` Jerin Jacob
2026-11-17 17:39 ` Bruce Richardson
2026-11-17 17:55 ` Pravin Pathak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117175709.2346400-9-pravinpathakdev@gmail.com \
--to=pravinpathakdev@gmail.com \
--cc=bruce.richardson@intel.com \
--cc=chenbox@nvidia.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=jerinj@marvell.com \
--cc=mattias.ronnblom@ericsson.com \
--cc=nipun.gupta@amd.com \
--cc=pravin.pathak.dev@gmail.com \
--cc=stephen@networkplumber.org \
--cc=thomas@monjalon.net \
--cc=tirthendu.sarkar@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox