All of lore.kernel.org
 help / color / mirror / Atom feed
From: Pravin Pathak <pravinpathakdev@gmail.com>
To: dev@dpdk.org
Cc: jerinj@marvell.com, bruce.richardson@intel.com,
	thomas@monjalon.net, david.marchand@redhat.com,
	nipun.gupta@amd.com, chenbox@nvidia.com,
	tirthendu.sarkar@intel.com, mattias.ronnblom@ericsson.com,
	stephen@networkplumber.org,
	Pravin Pathak <pravin.pathak.dev@gmail.com>
Subject: [PATCH v2 8/8] event/slb: address ARM64 build and all code reviews
Date: Tue, 17 Nov 2026 12:57:08 -0500	[thread overview]
Message-ID: <20261117175709.2346400-9-pravinpathakdev@gmail.com> (raw)
In-Reply-To: <20261117175709.2346400-1-pravinpathakdev@gmail.com>

From: Pravin Pathak <pravin.pathak.dev@gmail.com>

- Add missing standard C headers (<stdio.h>, <stdlib.h>, <string.h>)
  to fix implicit declaration of abs() on ARM64 builds.
- Defer credit return for released ordered events until
  slb_reorder_drain() reclaims the reorder buffer slot so the ordered
  queue ring cannot wrap onto an undrained slot.
- Re-check buf[drain_idx].state after unlocking queue->reorder_lock in
  slb_reorder_drain() to prevent a lost wakeup when another port marks
  drain_idx READY/RELEASED and fails trylock just before unlock.
- Use release memory ordering when resetting reorder buffer slots to
  SLB_REORDER_EMPTY and assert queue ring enqueue burst counts.
- Set default new_event_threshold to 3/4 of SLB_IN_RING_SIZE and check
  new_event_threshold inside slb_port_acquire_credits() only when
  borrowing from the central pool after checking the port's local
  credit cache.
- Return 0 with rte_errno set to EINVAL/ENOSPC on release underflow or
  credit exhaustion instead of aborting with RTE_VERIFY().
- Widen hold_count to uint32_t, clamp hold buffer copy lengths, and
  define SLB_HOLD_BUFFER_REFILL_HEADROOM.
- Widen timer frequency multiplication for migration cooldown cycles
  and avoid unsigned subtraction underflow in cooldown checks.
- Clean up queue resources via slb_queue_release() on setup failure
  and unlink all port references (both single-link and q_entries) on
  queue release.
- Guard against NULL q->q_ring in slb_dump() when dumping unconfigured
  queues.
- Make internal functions (slb_dump, slb_reorder_drain,
  slb_port_return_credits) static and update SLB documentation and
  26.11 release notes.

Signed-off-by: Pravin Pathak <pravin.pathak.dev@gmail.com>
---
 doc/guides/eventdevs/slb.rst           |  14 +-
 doc/guides/rel_notes/release_26_11.rst |   7 +
 drivers/event/slb/slb_evdev.c          | 129 +++++++++------
 drivers/event/slb/slb_evdev.h          |  19 +--
 drivers/event/slb/slb_event.c          | 219 +++++++++++++++++--------
 drivers/event/slb/slb_xstats.c         |   1 +
 6 files changed, 248 insertions(+), 141 deletions(-)

diff --git a/doc/guides/eventdevs/slb.rst b/doc/guides/eventdevs/slb.rst
index b109a1e2ca..73c0491045 100644
--- a/doc/guides/eventdevs/slb.rst
+++ b/doc/guides/eventdevs/slb.rst
@@ -56,6 +56,14 @@ Limitations
 -----------
 
 * **Maximum Ports and Queues**: Supports up to 64 ports and 64 queues.
-* **Priorities and linking**: Supports two levels of service priority to link up to
-  eight queues per port at each priority. SLB currently does not support event priority.
-* **Queues Types**: Does not support all-type queues.
+* **Priorities and linking**: Priority is configured per port-to-queue link via
+  ``rte_event_port_link()`` (supports two service priority levels, configured at compile time
+  via ``SLB_MAX_PRIORITIES``). SLB does not support per-queue priority
+  (``RTE_EVENT_DEV_CAP_QUEUE_QOS``), per-event priority (``RTE_EVENT_DEV_CAP_EVENT_QOS``),
+  or runtime port linking/unlinking after device start (``RTE_EVENT_DEV_CAP_RUNTIME_PORT_LINK``).
+* **Queues Types**: Does not support all-types queues (``RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES``).
+* **In-Flight Event Limit**: Each port maintains a local cache of up to ``SLB_PORT_MAX_CREDITS``
+  credits. To avoid credit-cache starvation when ``nb_events_limit`` is configured smaller than
+  ``nb_event_ports * SLB_PORT_MAX_CREDITS``, SLB raises the internal credit pool size to
+  ``max(nb_events_limit, nb_event_ports * SLB_PORT_MAX_CREDITS)`` (capped at ``SLB_MAX_EVENTS``),
+  so the device may hold more in-flight events than the configured ``nb_events_limit``.
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index 8ca8379cc6..1548ef0f15 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -312,6 +312,13 @@ New Features
   with AMD-specific hardware access, a primary/secondary topology
   and a packed scratchpad handshake.
 
+* **Added Software Load Balancer (SLB) eventdev PMD.**
+
+  Introduced a new poll-mode eventdev driver (``event_slb``) supporting
+  parallel, atomic, ordered, and single-link scheduling with dynamic
+  atomic flow migration and strict two-level port-queue service priorities.
+  Runtime port linking and unlinking after device start is not supported.
+
 * **Supported auto-detection of uncore power driver.**
 
   The uncore power library got automatic probing of multiple uncore drivers
diff --git a/drivers/event/slb/slb_evdev.c b/drivers/event/slb/slb_evdev.c
index 346b4b38ed..f47e08b3f9 100644
--- a/drivers/event/slb/slb_evdev.c
+++ b/drivers/event/slb/slb_evdev.c
@@ -3,6 +3,9 @@
  */
 
 #include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
 #include <inttypes.h>
 #include <rte_bitops.h>
 #include <rte_cycles.h>
@@ -104,7 +107,7 @@ slb_port_def_conf(struct rte_eventdev *dev,
 		  uint8_t port_id __rte_unused,
 		  struct rte_event_port_conf *port_conf)
 {
-	int32_t threshold = SLB_IN_RING_SIZE / 4;
+	int32_t threshold = (3 * SLB_IN_RING_SIZE) / 4;
 
 	if (dev->data->dev_conf.nb_events_limit > 0 &&
 	    threshold > dev->data->dev_conf.nb_events_limit)
@@ -122,6 +125,33 @@ slb_port_release(void *p __rte_unused)
 {
 }
 
+static void
+slb_port_unlink_queue_from_prio(struct slb_port *port, int prio, int slot)
+{
+	/* Shift the queue entries to overwriting the unlinked queue slot */
+	for (int k = slot; k < port->num_q[prio] - 1; k++)
+		port->q_entries[prio][k] = port->q_entries[prio][k + 1];
+
+	/* Decrement mapped queue count and Clear the last slot after shifting */
+	port->num_q[prio]--;
+	memset(&port->q_entries[prio][port->num_q[prio]], 0,
+	       sizeof(struct slb_port_q_entry));
+	/* Adjust the next queue index if it points beyond the new count */
+	if (port->next_q[prio] >= port->num_q[prio])
+		port->next_q[prio] = 0;
+}
+
+/* Find the slot of a queue in a specific priority level, returning -1 if not found */
+static int
+slb_port_find_queue_slot(struct slb_port *port, int prio, uint8_t qid)
+{
+	for (int s = 0; s < port->num_q[prio]; s++) {
+		if (port->q_entries[prio][s].queue_id == qid)
+			return s;
+	}
+	return -1;
+}
+
 /* Called to free queue resources on device close or reconfiguration */
 static void
 slb_queue_release(struct rte_eventdev *dev, uint8_t queue_id)
@@ -129,6 +159,26 @@ slb_queue_release(struct rte_eventdev *dev, uint8_t queue_id)
 	struct slb_evdev *slb = slb_pmd_priv(dev);
 	struct slb_queue *queue = &slb->queues[queue_id];
 
+	/* Unlink any port references before freeing queue rings */
+	for (int i = 0; i < slb->num_ports; i++) {
+		struct slb_port *port = &slb->ports[i];
+
+		if (port->single_linked_q == queue) {
+			port->single_linked_q = NULL;
+			port->single_linked_ring = NULL;
+		} else {
+			for (int pr = 0; pr < SLB_MAX_PRIORITIES; pr++) {
+				int slot = slb_port_find_queue_slot(
+					port, pr, queue_id);
+				if (slot >= 0)
+					slb_port_unlink_queue_from_prio(
+						port, pr, slot);
+			}
+		}
+	}
+	queue->serving_ports_mask = 0;
+	queue->num_serving_ports = 0;
+
 	if (queue->atomic_state) {
 		rte_free(queue->atomic_state);
 		queue->atomic_state = NULL;
@@ -160,6 +210,7 @@ slb_queue_setup(struct rte_eventdev *dev, uint8_t queue_id,
 	struct slb_queue *queue = &slb->queues[queue_id];
 	struct rte_event_ring *ring;
 	char ring_name[RTE_RING_NAMESIZE];
+	unsigned int ring_flags = RING_F_EXACT_SZ;
 	uint32_t q_size = (slb->max_inflight > 0) ?
 			  (uint32_t)slb->max_inflight : SLB_IN_RING_SIZE;
 
@@ -172,25 +223,23 @@ slb_queue_setup(struct rte_eventdev *dev, uint8_t queue_id,
 	if (conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_SINGLE_LINK) {
 		/* MP -> SC ring for single-link queues */
 		queue->is_single_link = true;
-		ring = rte_event_ring_create(ring_name, q_size,
-			dev->data->socket_id, RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		ring_flags |= RING_F_SC_DEQ;
 	} else if (conf->schedule_type == RTE_SCHED_TYPE_ATOMIC) {
 		/* MP -> SC ring for atomic queues main ring */
 		queue->is_single_link = false;
-		ring = rte_event_ring_create(ring_name, q_size,
-			dev->data->socket_id, RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		ring_flags |= RING_F_SC_DEQ;
 	} else if (conf->schedule_type == RTE_SCHED_TYPE_PARALLEL ||
 		   conf->schedule_type == RTE_SCHED_TYPE_ORDERED) {
 		/* MP -> MC ring for parallel or ordered queues */
 		queue->is_single_link = false;
-		ring = rte_event_ring_create(ring_name, q_size,
-			dev->data->socket_id, RING_F_EXACT_SZ);
 	} else {
 		SLB_LOG_DP_LINE(ERR, "Unsupported schedule_type %u",
 				conf->schedule_type);
 		return -ENOTSUP;
 	}
 
+	ring = rte_event_ring_create(ring_name, q_size, dev->data->socket_id,
+				     ring_flags);
 	if (ring == NULL)
 		return -ENOMEM;
 
@@ -209,7 +258,7 @@ slb_queue_setup(struct rte_eventdev *dev, uint8_t queue_id,
 			sizeof(struct slb_atomic_state),
 			RTE_CACHE_LINE_SIZE, dev->data->socket_id);
 		if (queue->atomic_state == NULL) {
-			rte_event_ring_free(ring);
+			slb_queue_release(dev, queue_id);
 			return -ENOMEM;
 		}
 		for (int i = 0; i < SLB_ATOMIC_FLOWS; i++)
@@ -236,7 +285,7 @@ slb_queue_setup(struct rte_eventdev *dev, uint8_t queue_id,
 			sizeof(struct slb_reorder_entry) * ring->r.size,
 			RTE_CACHE_LINE_SIZE, dev->data->socket_id);
 		if (queue->reorder_buffer == NULL) {
-			rte_event_ring_free(ring);
+			slb_queue_release(dev, queue_id);
 			return -ENOMEM;
 		}
 		queue->reorder_drain_idx = 0;
@@ -261,33 +310,6 @@ slb_queue_def_conf(struct rte_eventdev *dev __rte_unused,
 	};
 }
 
-static void
-slb_port_unlink_queue_from_prio(struct slb_port *port, int prio, int slot)
-{
-	/* Shift the queue entries to overwriting the unlinked queue slot */
-	for (int k = slot; k < port->num_q[prio] - 1; k++)
-		port->q_entries[prio][k] = port->q_entries[prio][k + 1];
-
-	/* Decrement mapped queue count and Clear the last slot after shifting */
-	port->num_q[prio]--;
-	memset(&port->q_entries[prio][port->num_q[prio]], 0,
-	       sizeof(struct slb_port_q_entry));
-	/* Adjust the next queue index if it points beyond the new count */
-	if (port->next_q[prio] >= port->num_q[prio])
-		port->next_q[prio] = 0;
-}
-
-/* Find the slot of a queue in a specific priority level, returning -1 if not found */
-static int
-slb_port_find_queue_slot(struct slb_port *port, int prio, uint8_t qid)
-{
-	for (int s = 0; s < port->num_q[prio]; s++) {
-		if (port->q_entries[prio][s].queue_id == qid)
-			return s;
-	}
-	return -1;
-}
-
 /* Link a queue to a port at a specific priority level */
 static int
 slb_port_link_one(struct slb_port *port, struct slb_queue *queue,
@@ -303,15 +325,14 @@ slb_port_link_one(struct slb_port *port, struct slb_queue *queue,
 			rte_errno = EINVAL;
 			return -EINVAL;
 		}
-		/* Only one queue can be linked to a single-linked port */
-		if (port->single_linked_q != NULL) {
-			if (port->single_linked_q == queue)
-				return 0;
-			rte_errno = EINVAL;
-			return -EINVAL;
-		}
-		/* A single-link queue can only be linked to one port */
-		if (queue->serving_ports_mask != 0) {
+		/* Re-linking the same queue to the same port is a no-op */
+		if (port->single_linked_q == queue)
+			return 0;
+		/* Only one queue can be linked to a single-linked port, and a
+		 * single-link queue can only be linked to one port.
+		 */
+		if (port->single_linked_q != NULL ||
+		    queue->serving_ports_mask != 0) {
 			rte_errno = EINVAL;
 			return -EINVAL;
 		}
@@ -476,7 +497,7 @@ slb_info_get(struct rte_eventdev *dev __rte_unused,
 		.driver_name = SLB_PMD_NAME,
 		.max_event_queues = SLB_MAX_QUEUES,
 		.max_event_queue_flows = SLB_MAX_FLOWS,
-		.max_event_queue_priority_levels = SLB_MAX_PRIORITIES,
+		.max_event_queue_priority_levels = 1,
 		.max_event_priority_levels = 1,
 		.max_event_ports = SLB_MAX_PORTS,
 		.max_event_port_dequeue_depth = SLB_MAX_PORT_DEQUEUE_DEPTH,
@@ -534,14 +555,14 @@ slb_configure(const struct rte_eventdev *dev)
 	slb->num_queues = conf->nb_event_queues;
 
 	/* Compute minimum credits required to avoid deadlock */
-	min_max_in_flight = conf->nb_event_ports * SLB_PORT_MAX_CREDITS;
+	min_max_in_flight = (int32_t)conf->nb_event_ports * SLB_PORT_MAX_CREDITS;
 	if (min_max_in_flight > SLB_MAX_EVENTS)
 		min_max_in_flight = SLB_MAX_EVENTS;
 
 	/* Set credit pool size to max of nb_events_limit and minimum required to avoid deadlock */
 	slb->max_inflight = RTE_MAX(conf->nb_events_limit, min_max_in_flight);
 	slb->migration_cooldown_cycles =
-		(rte_get_timer_hz() * SLB_MIGRATION_COOLDOWN_MS) / 1000;
+		((uint64_t)rte_get_timer_hz() * SLB_MIGRATION_COOLDOWN_MS) / 1000;
 	return 0;
 }
 
@@ -613,7 +634,7 @@ slb_stop(struct rte_eventdev *dev)
 			struct slb_atomic_state *astate = q->atomic_state;
 
 			/* Call user callback for each held event */
-			for (uint16_t h = 0; h < astate->hold_count; h++) {
+			for (uint32_t h = 0; h < astate->hold_count; h++) {
 				if (flush != NULL)
 					flush(dev_id, astate->hold_buffer[h],
 					      flush_arg);
@@ -655,7 +676,7 @@ slb_stop(struct rte_eventdev *dev)
 				rte_atomic_store_explicit(
 					&q->reorder_buffer[s].state,
 					SLB_REORDER_EMPTY,
-					rte_memory_order_relaxed);
+					rte_memory_order_release);
 			}
 			q->reorder_drain_idx = 0;
 		}
@@ -696,7 +717,7 @@ slb_close(struct rte_eventdev *dev)
 }
 
 /* Dump the event device current state and statistics */
-void
+static void
 slb_dump(struct rte_eventdev *dev, FILE *f)
 {
 	struct slb_evdev *slb = slb_pmd_priv(dev);
@@ -778,6 +799,12 @@ slb_dump(struct rte_eventdev *dev, FILE *f)
 	fprintf(f, "\n  ALL QUEUES STATUS:\n");
 	for (i = 0; i < slb->num_queues; i++) {
 		struct slb_queue *q = &slb->queues[i];
+
+		if (q->q_ring == NULL) {
+			fprintf(f, "    Queue %2u: (not configured)\n", i);
+			continue;
+		}
+
 		uint32_t count = rte_event_ring_count(q->q_ring);
 		uint32_t ring_size = rte_event_ring_get_capacity(q->q_ring);
 
diff --git a/drivers/event/slb/slb_evdev.h b/drivers/event/slb/slb_evdev.h
index c01290e539..457376d1d6 100644
--- a/drivers/event/slb/slb_evdev.h
+++ b/drivers/event/slb/slb_evdev.h
@@ -1,4 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Ericsson AB
  * Copyright(c) 2026 Pravin Pathak <pravin.pathak.dev@gmail.com>
  */
 
@@ -37,13 +38,7 @@
 /* Maximum number of queues that can be linked to a port per priority */
 #define SLB_PORT_MAX_Q_MAP_PER_PRIO (SLB_MAX_QUEUES)
 /* Number of priority levels supported by SLB. Currently fixed to 2 */
-#ifndef SLB_MAX_PRIORITIES
 #define SLB_MAX_PRIORITIES 2
-#endif
-
-#if (SLB_MAX_PRIORITIES != 2)
-#error "SLB_MAX_PRIORITIES must be set to 2 for now"
-#endif
 
 /* Total max mapping slots (per-prio * priorities) */
 #define SLB_PORT_MAX_Q_MAP (SLB_PORT_MAX_Q_MAP_PER_PRIO * SLB_MAX_PRIORITIES)
@@ -87,8 +82,9 @@ struct slb_reorder_entry {
 #define SLB_ATOMIC_FLOWS 2048
 #define SLB_ATOMIC_FLOWS_MASK (SLB_ATOMIC_FLOWS - 1)
 #define SLB_HOLD_BUFFER_SIZE (SLB_MAX_PORT_DEQUEUE_DEPTH * 64)
+#define SLB_HOLD_BUFFER_REFILL_HEADROOM 3
 #define SLB_MIGRATION_LOAD_THRESHOLD 16
-#define SLB_MIGRATION_COOLDOWN_MS 500
+#define SLB_MIGRATION_COOLDOWN_MS 500ULL
 
 /* Per-flow entry for atomic scheduling affinity */
 struct slb_flow {
@@ -120,7 +116,7 @@ struct slb_atomic_state {
 	struct slb_flow_map flow_map;
 	uint32_t next_port_rr;
 	rte_spinlock_t refill_lock;
-	uint16_t hold_count;
+	uint32_t hold_count;
 	struct slb_migration_state migration;
 	uint64_t migrations_count;
 	uint64_t flow_last_migrated[SLB_ATOMIC_FLOWS];
@@ -223,13 +219,6 @@ uint16_t slb_event_enqueue_forward_burst(void *port,
 uint16_t slb_event_dequeue_burst(void *port, struct rte_event *events,
 				 uint16_t num, uint64_t wait);
 
-void slb_port_return_credits(struct slb_evdev *slb, struct slb_port *port,
-			     int32_t credits);
-
-void slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue);
-
-void slb_dump(struct rte_eventdev *dev, FILE *f);
-
 int slb_xstats_get_names(const struct rte_eventdev *dev,
 			 enum rte_event_dev_xstats_mode mode,
 			 uint8_t queue_port_id,
diff --git a/drivers/event/slb/slb_event.c b/drivers/event/slb/slb_event.c
index 3d0436e4a0..f53e44b61b 100644
--- a/drivers/event/slb/slb_event.c
+++ b/drivers/event/slb/slb_event.c
@@ -1,11 +1,15 @@
 /* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Ericsson AB
  * Copyright(c) 2026 Pravin Pathak <pravin.pathak.dev@gmail.com>
  */
 
 #include <stdbool.h>
+#include <stdlib.h>
+#include <string.h>
 #include <limits.h>
 #include <rte_bitops.h>
 #include <rte_cycles.h>
+#include <rte_errno.h>
 #include <rte_ring_elem.h>
 #include <rte_stdatomic.h>
 
@@ -31,6 +35,9 @@ slb_port_acquire_credits(struct slb_evdev *slb, struct slb_port *port,
 
 	total_inflight = rte_atomic_load_explicit(&slb->credits_on_loan,
 						  rte_memory_order_relaxed);
+	if (unlikely(total_inflight > port->new_event_threshold))
+		return false;
+
 	available = slb->max_inflight - total_inflight;
 	required_credits = RTE_MAX(abs(remaining_credits), SLB_PORT_MIN_CREDITS);
 
@@ -61,7 +68,7 @@ slb_port_acquire_credits(struct slb_evdev *slb, struct slb_port *port,
 /* Port returning credits (Terminated events). Keep local cache.
  * If local cache exceeds the threshold, return excess credits to the pool
  */
-void
+static void
 slb_port_return_credits(struct slb_evdev *slb, struct slb_port *port,
 			int32_t credits)
 {
@@ -103,16 +110,25 @@ slb_port_queue_enqueue_burst_stats(struct slb_port *source_port,
 
 #define REORDER_BURST_SZ 64
 
-/* Drain contiguous ready events from ordered queue's reorder buffer */
-void
-slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
+/* Drain contiguous ready events from ordered queue's reorder buffer.
+ * Thread-safe: uses queue->reorder_lock internally.
+ */
+static void
+slb_reorder_drain(struct slb_evdev *slb, struct slb_port *port,
+		  struct slb_queue *queue)
 {
 	if (queue == NULL || queue->reorder_buffer == NULL)
 		return;
 
+	int32_t released_credits = 0;
+
+retry:
 	/* Only one thread can drain the reorder buffer at a time */
-	if (!rte_spinlock_trylock(&queue->reorder_lock))
+	if (!rte_spinlock_trylock(&queue->reorder_lock)) {
+		if (released_credits > 0)
+			slb_port_return_credits(slb, port, released_credits);
 		return;
+	}
 
 	uint32_t drain_idx = queue->reorder_drain_idx;
 	uint32_t mask = queue->q_ring->r.mask;
@@ -120,7 +136,7 @@ slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
 	uint8_t state;
 
 	/* Drain events from the current drain index till an empty slot.
-	 * Released event slots need no processing except marking them as empty.
+	 * Released event slots return their held credit once drained.
 	 */
 	while ((state = rte_atomic_load_explicit(&buf[drain_idx].state,
 						 rte_memory_order_acquire)) !=
@@ -128,8 +144,9 @@ slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
 		if (state == SLB_REORDER_RELEASED) {
 			rte_atomic_store_explicit(&buf[drain_idx].state,
 						  SLB_REORDER_EMPTY,
-						  rte_memory_order_relaxed);
+						  rte_memory_order_release);
 			drain_idx = (drain_idx + 1) & mask;
+			released_credits++;
 			continue;
 		}
 
@@ -142,7 +159,7 @@ slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
 			burst[burst_size++] = buf[drain_idx].event;
 			rte_atomic_store_explicit(&buf[drain_idx].state,
 						  SLB_REORDER_EMPTY,
-						  rte_memory_order_relaxed);
+						  rte_memory_order_release);
 			drain_idx = (drain_idx + 1) & mask;
 		} while (burst_size < REORDER_BURST_SZ &&
 			 rte_atomic_load_explicit(&buf[drain_idx].state,
@@ -151,12 +168,25 @@ slb_reorder_drain(struct slb_evdev *slb, struct slb_queue *queue)
 			 buf[drain_idx].event.queue_id == target_qid);
 
 		/* Target queue is sized to max_inflight and can never be full */
-		rte_event_ring_enqueue_burst(slb->queues[target_qid].q_ring,
-					     burst, burst_size, NULL);
+		uint16_t enq = rte_event_ring_enqueue_burst(
+			slb->queues[target_qid].q_ring, burst, burst_size, NULL);
+		RTE_ASSERT(enq == burst_size);
 	}
 
 	queue->reorder_drain_idx = drain_idx;
 	rte_spinlock_unlock(&queue->reorder_lock);
+
+	/* Re-check after unlock in case another port marked drain_idx
+	 * READY/RELEASED and failed trylock before we unlocked.
+	 */
+	rte_atomic_thread_fence(rte_memory_order_seq_cst);
+	if (unlikely(rte_atomic_load_explicit(&buf[drain_idx].state,
+					      rte_memory_order_acquire) !=
+		     SLB_REORDER_EMPTY))
+		goto retry;
+
+	if (released_credits > 0)
+		slb_port_return_credits(slb, port, released_credits);
 }
 
 /* Hash 20-bit flow ID into atomic flow slot index (SLB_ATOMIC_FLOWS) */
@@ -212,7 +242,7 @@ slb_find_lightest_flow_on_port(struct slb_queue *queue, uint8_t port_id,
 					     rte_memory_order_relaxed) == (int32_t)port_id) {
 			/* Skip flows that have been migrated recently */
 			if (astate->flow_last_migrated[f] != 0 &&
-			    (now - astate->flow_last_migrated[f]) < cooldown_cycles)
+			    now < astate->flow_last_migrated[f] + cooldown_cycles)
 				continue;
 
 			if (counts[f] < min_events) {
@@ -256,7 +286,7 @@ slb_check_and_trigger_migration(struct slb_evdev *slb, struct slb_queue *queue,
 
 	/* Port is still in cooldown period */
 	if (slb->ports[full_port_id].last_migrated_time != 0 &&
-	    (now - slb->ports[full_port_id].last_migrated_time) < cooldown_cycles)
+	    now < slb->ports[full_port_id].last_migrated_time + cooldown_cycles)
 		return;
 
 	/* Find flow to migrate*/
@@ -291,7 +321,7 @@ slb_find_migration_target_port(struct slb_evdev *slb, struct slb_queue *queue,
 			continue;
 
 		if (cp->last_migrated_time == 0 ||
-		    (now - cp->last_migrated_time) >= cooldown_cycles) {
+		    now >= cp->last_migrated_time + cooldown_cycles) {
 			if ((int)astate->port_stats[cand_port].flow_count < min_flows) {
 				min_flows = (int)astate->port_stats[cand_port].flow_count;
 				target_port = cand_port;
@@ -315,7 +345,7 @@ slb_complete_flow_migration(struct slb_evdev *slb, struct slb_atomic_state *asta
 				  target_port, rte_memory_order_release);
 
 	/* Switch all events in hold buffer to the target port */
-	for (uint16_t h = 0; h < astate->hold_count; h++) {
+	for (uint32_t h = 0; h < astate->hold_count; h++) {
 		if (astate->hold_ports[h] == mig_port &&
 		    slb_flow_hash(astate->hold_buffer[h].flow_id) == f_idx) {
 			astate->hold_ports[h] = (uint8_t)target_port;
@@ -429,38 +459,38 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
 
 	/* 1. Process Hold Buffer events */
 	if (astate->hold_count > 0) {
-		uint16_t retained = 0;
-		uint16_t count = astate->hold_count;
+		uint32_t retained = 0;
+		uint32_t count = astate->hold_count;
+		uint32_t h_idx = 0;
 
-		i = 0;
-		while (i < count) {
-			uint8_t dest_port_id = astate->hold_ports[i];
-			uint16_t burst_len = 1;
+		while (h_idx < count) {
+			uint8_t dest_port_id = astate->hold_ports[h_idx];
+			uint32_t burst_len = 1;
 
 			/* Collect consecutive events for the same destination port */
-			while ((i + burst_len) < count &&
-			       astate->hold_ports[i + burst_len] == dest_port_id)
+			while ((h_idx + burst_len) < count &&
+			       astate->hold_ports[h_idx + burst_len] == dest_port_id)
 				burst_len++;
 
 			/* If destination port is full, retain the event in hold buffer*/
 			if (full_ports_mask & (1ULL << dest_port_id)) {
-				if (retained != i) {
+				if (retained != h_idx) {
 					memmove(&astate->hold_buffer[retained],
-						&astate->hold_buffer[i],
+						&astate->hold_buffer[h_idx],
 						burst_len * sizeof(struct rte_event));
 					memmove(&astate->hold_ports[retained],
-						&astate->hold_ports[i],
+						&astate->hold_ports[h_idx],
 						burst_len);
 				}
 				retained += burst_len;
-				i += burst_len;
+				h_idx += burst_len;
 				continue;
 			}
 
 			/* Enqueue events to the destination port */
 			uint16_t enqueued = rte_event_ring_enqueue_burst(
 				queue->per_port_rings[dest_port_id],
-				&astate->hold_buffer[i], burst_len, NULL);
+				&astate->hold_buffer[h_idx], burst_len, NULL);
 			astate->port_stats[dest_port_id].q_enqueued += enqueued;
 
 			/* If enqueue was successful then decrement the load counter */
@@ -470,8 +500,8 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
 
 			/* If enqueue was not successful, retain the event in hold buffer */
 			if (enqueued < burst_len) {
-				uint16_t rem = burst_len - enqueued;
-				uint16_t start_idx = i + enqueued;
+				uint32_t rem = burst_len - enqueued;
+				uint32_t start_idx = h_idx + enqueued;
 
 				full_ports_mask |= (1ULL << dest_port_id);
 				if (retained != start_idx) {
@@ -489,7 +519,7 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
 						slb, queue, astate, dest_port_id);
 			}
 
-			i += burst_len;
+			h_idx += burst_len;
 		}
 
 		astate->hold_count = retained;
@@ -497,7 +527,7 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
 		 * if hold buffer does not have enough free space.
 		 */
 		if (SLB_HOLD_BUFFER_SIZE - astate->hold_count <
-		    3 * RTE_DIM(rcv_events)) {
+		    SLB_HOLD_BUFFER_REFILL_HEADROOM * RTE_DIM(rcv_events)) {
 			rte_spinlock_unlock(&astate->refill_lock);
 			return;
 		}
@@ -589,11 +619,17 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
 
 			/* If destination port is full, retain events in hold buffer */
 			if (full_ports_mask & (1ULL << p)) {
-				rte_memcpy(&astate->hold_buffer[astate->hold_count],
-					   &sorted_events[p_start],
-					   p_count * sizeof(struct rte_event));
-				memset(&astate->hold_ports[astate->hold_count], p, p_count);
-				astate->hold_count += p_count;
+				uint32_t to_copy = RTE_MIN(
+					(uint32_t)p_count,
+					SLB_HOLD_BUFFER_SIZE - astate->hold_count);
+				if (likely(to_copy > 0)) {
+					rte_memcpy(&astate->hold_buffer[astate->hold_count],
+						   &sorted_events[p_start],
+						   to_copy * sizeof(struct rte_event));
+					memset(&astate->hold_ports[astate->hold_count],
+					       p, to_copy);
+					astate->hold_count += to_copy;
+				}
 				continue;
 			}
 
@@ -607,14 +643,20 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
 				if (unlikely(astate->port_stats[p].load_counter > 0))
 					astate->port_stats[p].load_counter--;
 			} else {
-				uint16_t rem = p_count - enqueued;
+				uint32_t rem = p_count - enqueued;
+				uint32_t to_copy = RTE_MIN(
+					rem,
+					SLB_HOLD_BUFFER_SIZE - astate->hold_count);
 				/* Mark the destination port as full and retain remaining events */
 				full_ports_mask |= (1ULL << p);
-				rte_memcpy(&astate->hold_buffer[astate->hold_count],
-					   &sorted_events[p_start + enqueued],
-					   rem * sizeof(struct rte_event));
-				memset(&astate->hold_ports[astate->hold_count], p, rem);
-				astate->hold_count += rem;
+				if (likely(to_copy > 0)) {
+					rte_memcpy(&astate->hold_buffer[astate->hold_count],
+						   &sorted_events[p_start + enqueued],
+						   to_copy * sizeof(struct rte_event));
+					memset(&astate->hold_ports[astate->hold_count],
+					       p, to_copy);
+					astate->hold_count += to_copy;
+				}
 				/* If not a single event was enqueued, check for migration */
 				if (enqueued == 0)
 					slb_check_and_trigger_migration(
@@ -626,6 +668,16 @@ slb_atomic_refill(struct slb_port *port, struct slb_queue *queue)
 	rte_spinlock_unlock(&astate->refill_lock);
 }
 
+/* Enqueue a burst of events from source_port.
+ *
+ * Design invariant: Every queue ring (q_ring) is created with RING_F_EXACT_SZ
+ * and capacity equal to slb->max_inflight. Because NEW events must acquire
+ * credits from slb->max_inflight upfront and FORWARD events are verified
+ * against source_port->pending_releases, the total number of in-flight events
+ * across the entire device can never exceed slb->max_inflight. Consequently,
+ * q_ring can never fill and rte_event_ring_enqueue_burst() onto q_ring is
+ * guaranteed to enqueue the full burst.
+ */
 static inline uint16_t
 slb_event_enqueue_burst_generic(struct slb_port *source_port,
 				const struct rte_event events[],
@@ -656,25 +708,22 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
 		}
 	}
 
-	/* Check credit threshold if there are NEW events */
-	if (num_new > 0) {
-		if (unlikely(rte_atomic_load_explicit(&slb->credits_on_loan,
-						      rte_memory_order_relaxed) >
-			     source_port->new_event_threshold))
-			return 0;
+	/* Check for release underflow before acquiring credits */
+	cur_pending = source_port->pending_releases;
+	if (unlikely(num_forward + num_release > cur_pending)) {
+		rte_errno = EINVAL;
+		return 0;
+	}
 
+	/* Acquire credits if there are NEW events */
+	if (num_new > 0) {
 		if (unlikely(!slb_port_acquire_credits(slb, source_port,
-						       num_new)))
+						       num_new))) {
+			rte_errno = ENOSPC;
 			return 0;
+		}
 	}
 
-	/* Return credits for RELEASE events */
-	if (num_release > 0)
-		slb_port_return_credits(slb, source_port, num_release);
-
-	/* Check for release underflow and balance pending releases */
-	cur_pending = source_port->pending_releases;
-	RTE_VERIFY(num_forward + num_release <= cur_pending);
 	source_port->pending_releases = cur_pending - (num_forward + num_release);
 
 	slb_port_enqueue_stats(source_port, num_new, num_forward, num_release);
@@ -695,8 +744,9 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
 
 		/* Enqueue the entire burst if going to the same queue */
 		if (likely(burst_size == events_len)) {
-			rte_event_ring_enqueue_burst(slb->queues[q0].q_ring,
-						     events, events_len, NULL);
+			uint16_t enq = rte_event_ring_enqueue_burst(
+				slb->queues[q0].q_ring, events, events_len, NULL);
+			RTE_ASSERT(enq == events_len);
 			slb_port_queue_enqueue_burst_stats(source_port, q0,
 							   events_len);
 		} else { /* Sort events by destination queue */
@@ -748,11 +798,13 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
 				uint8_t q = rte_ctz64(qmask);
 				uint16_t count = queue_counts[q];
 				uint16_t offset = queue_offsets[q];
+				uint16_t enq;
 
 				qmask &= (qmask - 1);
-				rte_event_ring_enqueue_burst(slb->queues[q].q_ring,
-							     &sorted_events[offset],
-							     count, NULL);
+				enq = rte_event_ring_enqueue_burst(
+					slb->queues[q].q_ring,
+					&sorted_events[offset], count, NULL);
+				RTE_ASSERT(enq == count);
 				slb_port_queue_enqueue_burst_stats(source_port, q,
 								   count);
 			}
@@ -765,6 +817,7 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
 
 	/* General path for bursts with any mix of events */
 	uint64_t ordered_drain_mask = 0;
+	uint16_t ordered_releases = 0;
 	uint16_t freed_count = 0;
 	uint16_t head = source_port->history_head;
 	uint16_t n_enq = 0;
@@ -792,11 +845,14 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
 				uint16_t slot = hist->seq_slot;
 
 				if (ev->op == RTE_EVENT_OP_RELEASE) {
-					/* Release the reorder buffer entry */
+					/* Release the reorder buffer entry; credit is
+					 * returned when slb_reorder_drain reclaims slot
+					 */
 					rte_atomic_store_explicit(
 						&orig_q->reorder_buffer[slot].state,
 						SLB_REORDER_RELEASED,
 						rte_memory_order_release);
+					ordered_releases++;
 				} else {
 					/* Put event in reorder buffer */
 					orig_q->reorder_buffer[slot].event = *ev;
@@ -874,9 +930,11 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
 			   n_enq == events_len)) {
 			/* Entire batch is going to a single queue */
 			uint8_t q = rte_ctz64(batch_queues_mask);
+			uint16_t enq;
 
-			rte_event_ring_enqueue_burst(slb->queues[q].q_ring,
-						     events, events_len, NULL);
+			enq = rte_event_ring_enqueue_burst(
+				slb->queues[q].q_ring, events, events_len, NULL);
+			RTE_ASSERT(enq == events_len);
 			slb_port_queue_enqueue_burst_stats(source_port, q,
 							   events_len);
 		} else {
@@ -909,17 +967,26 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
 				uint8_t q = rte_ctz64(qmask);
 				uint16_t count = queue_counts[q];
 				uint16_t offset = queue_offsets[q];
+				uint16_t enq;
 
 				qmask &= (qmask - 1);
-				rte_event_ring_enqueue_burst(slb->queues[q].q_ring,
-							     &sorted_events[offset],
-							     count, NULL);
+				enq = rte_event_ring_enqueue_burst(
+					slb->queues[q].q_ring,
+					&sorted_events[offset], count, NULL);
+				RTE_ASSERT(enq == count);
 				slb_port_queue_enqueue_burst_stats(source_port, q,
 								   count);
 			}
 		}
 	}
 
+	/* Return credits for non-ordered RELEASE events immediately; ordered
+	 * RELEASE credits are returned when slb_reorder_drain reclaims the slot.
+	 */
+	if (num_release > ordered_releases)
+		slb_port_return_credits(slb, source_port,
+					num_release - ordered_releases);
+
 	/* Update the history list head once after all frees*/
 	if (freed_count > 0) {
 		source_port->history_head = head;
@@ -931,7 +998,7 @@ slb_event_enqueue_burst_generic(struct slb_port *source_port,
 		uint8_t qid = rte_ctz64(ordered_drain_mask);
 
 		ordered_drain_mask &= ~(1ULL << qid);
-		slb_reorder_drain(slb, &slb->queues[qid]);
+		slb_reorder_drain(slb, source_port, &slb->queues[qid]);
 	}
 
 	SLB_LOG_DP_PORT_LINE(DEBUG, source_port->id,
@@ -1134,14 +1201,16 @@ slb_port_process_implicit_release(struct slb_port *port, struct slb_evdev *slb,
 				  uint16_t pending)
 {
 	uint16_t head;
+	uint16_t ordered_releases = 0;
 	uint64_t ordered_drain_mask = 0;
 
-	slb_port_return_credits(slb, port, pending);
 	port->pending_releases = 0;
 
 	if (port->is_single_linked ||
-	    port->history_avail >= SLB_PORT_HISTORY_MAX_TOKENS)
+	    port->history_avail >= SLB_PORT_HISTORY_MAX_TOKENS) {
+		slb_port_return_credits(slb, port, pending);
 		return;
+	}
 
 	/* Process all history entries */
 	head = port->history_head;
@@ -1151,7 +1220,9 @@ slb_port_process_implicit_release(struct slb_port *port, struct slb_evdev *slb,
 		if (!hist->active)
 			continue;
 
-		/* Ordered releases will free the reorder buffer slot*/
+		/* Ordered releases will free the reorder buffer slot; credit is
+		 * returned when slb_reorder_drain reclaims the slot.
+		 */
 		if (hist->sched_type == RTE_SCHED_TYPE_ORDERED) {
 			struct slb_queue *oq = &slb->queues[hist->orig_queue_id];
 
@@ -1161,6 +1232,7 @@ slb_port_process_implicit_release(struct slb_port *port, struct slb_evdev *slb,
 					SLB_REORDER_RELEASED,
 					rte_memory_order_release);
 				ordered_drain_mask |= (1ULL << hist->orig_queue_id);
+				ordered_releases++;
 			}
 		} else if (hist->sched_type == RTE_SCHED_TYPE_ATOMIC) {
 			/* Atomic releases are counted as returned events */
@@ -1180,12 +1252,15 @@ slb_port_process_implicit_release(struct slb_port *port, struct slb_evdev *slb,
 	}
 	port->history_head = head;
 
+	if (pending > ordered_releases)
+		slb_port_return_credits(slb, port, pending - ordered_releases);
+
 	/* If there are ordered releases, call the reorder drain function */
 	while (ordered_drain_mask != 0) {
 		uint8_t qid = rte_ctz64(ordered_drain_mask);
 
 		ordered_drain_mask &= ~(1ULL << qid);
-		slb_reorder_drain(slb, &slb->queues[qid]);
+		slb_reorder_drain(slb, port, &slb->queues[qid]);
 	}
 }
 
diff --git a/drivers/event/slb/slb_xstats.c b/drivers/event/slb/slb_xstats.c
index fdb7dd6cc5..ff898a3ef2 100644
--- a/drivers/event/slb/slb_xstats.c
+++ b/drivers/event/slb/slb_xstats.c
@@ -1,4 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018-2019 Ericsson AB
  * Copyright(c) 2026 Pravin Pathak <pravin.pathak.dev@gmail.com>
  */
 
-- 
2.53.0


  parent reply	other threads:[~2026-11-17 17:58 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-07  1:30 [PATCH v1 0/8] event/slb: introducing software load balancer eventdev PMD Pravin Pathak
2026-11-07  1:30 ` [PATCH v1 1/8] event/slb: introduce software load balancer PMD Pravin Pathak
2026-11-07  1:30 ` [PATCH v1 2/8] event/slb: add support for ordered queues Pravin Pathak
2026-11-07  1:30 ` [PATCH v1 3/8] event/slb: add support for atomic queues Pravin Pathak
2026-11-07  1:30 ` [PATCH v1 4/8] event/slb: add dynamic flow migration " Pravin Pathak
2026-11-07  1:30 ` [PATCH v1 5/8] event/slb: add extended statistics and diagnostics Pravin Pathak
2026-11-07  1:30 ` [PATCH v1 6/8] event/slb: add self-test and documentation Pravin Pathak
2026-11-07  1:30 ` [PATCH v1 7/8] event/slb: update resets and credit defaults Pravin Pathak
2026-11-07  1:30 ` [PATCH v1 8/8] event/slb: improve port linking, config checks and atomic refill Pravin Pathak
2026-11-17 17:57   ` [PATCH v2 0/8] event/slb: introducing software load balancer eventdev PMD Pravin Pathak
2026-11-17 17:57     ` [PATCH v2 1/8] event/slb: introduce software load balancer PMD Pravin Pathak
2026-11-17 17:57     ` [PATCH v2 2/8] event/slb: add support for ordered queues Pravin Pathak
2026-11-17 17:57     ` [PATCH v2 3/8] event/slb: add support for atomic queues Pravin Pathak
2026-11-17 17:57     ` [PATCH v2 4/8] event/slb: add dynamic flow migration " Pravin Pathak
2026-11-17 17:57     ` [PATCH v2 5/8] event/slb: add extended statistics and diagnostics Pravin Pathak
2026-11-17 17:57     ` [PATCH v2 6/8] event/slb: add self-test and documentation Pravin Pathak
2026-11-17 17:57     ` [PATCH v2 7/8] event/slb: fine tuning and fixes Pravin Pathak
2026-11-17 17:57     ` Pravin Pathak [this message]
2026-11-17 18:46     ` [PATCH v2 0/8] event/slb: introducing software load balancer eventdev PMD Stephen Hemminger
2026-11-07  7:29 ` [PATCH v1 " Jerin Jacob
2026-11-08 17:10   ` Stephen Hemminger
2026-11-09  2:30   ` Pravin Pathak
2026-11-09 12:55     ` Jerin Jacob
2026-11-11 14:22       ` Sarkar, Tirthendu
2026-11-17 17:09         ` Pravin Pathak
2026-11-17 17:25           ` Jerin Jacob
2026-11-17 17:39             ` Bruce Richardson
2026-11-17 17:55               ` Pravin Pathak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261117175709.2346400-9-pravinpathakdev@gmail.com \
    --to=pravinpathakdev@gmail.com \
    --cc=bruce.richardson@intel.com \
    --cc=chenbox@nvidia.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=jerinj@marvell.com \
    --cc=mattias.ronnblom@ericsson.com \
    --cc=nipun.gupta@amd.com \
    --cc=pravin.pathak.dev@gmail.com \
    --cc=stephen@networkplumber.org \
    --cc=thomas@monjalon.net \
    --cc=tirthendu.sarkar@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.