[PATCH 0/9] bpf: clean up and document ethdev callback synchronization

[PATCH 0/9] bpf: clean up and document ethdev callback synchronization

[Marat Khalili]

The DPDK BPF library provides mechanisms to attach BPF programs to
ethdev Rx/Tx callbacks (rte_bpf_eth_rx_elf_load and
rte_bpf_eth_tx_elf_load). Safe unloading of these callbacks relies on
a synchronization protocol between the control plane and datapath
threads, built upon the assumption that each port/queue callback is
serviced by at most one datapath thread.

This patch series improves the documentation, safety, and implementation
of the callback synchronization mechanism:

1. Documentation:
   - Add an ASCII sequence diagram and comprehensive documentation in
     bpf_pkt.c explaining the synchronization protocol between the
     control plane and datapath during callback unload.
   - Document the single-thread-per-queue datapath requirement in
     public API headers.

2. Safety and Validation:
   - Enforce the single-threaded datapath assumption by rejecting Tx
     callback installation when the device advertises
     RTE_ETH_TX_OFFLOAD_MT_LOCKFREE.
   - Add a new flag (RTE_BPF_ETH_F_SKIP_MT_CHECK) allowing applications
     to override this check when application architecture guarantees
     thread exclusivity.
   - Add automated unit test test_bpf_elf_tx_mt_lockfree.

3. Synchronization Cleanups:
   - Remove a redundant control plane memory barrier (rte_smp_mb) on
     callback unload.
   - Simplify datapath inuse tracking (cbi->use) from an even/odd
     counter to a simple boolean flag (0 and 1), simplifying
     bpf_eth_cbi_wait().
   - Switch cbi->use stores to explicit rte_atomic_store_explicit() with
     relaxed memory order, avoiding implicit seq_cst operations in C11.
   - Make cbi->cb atomic with relaxed atomic load/store operations,
     eliminating C11 data-race undefined behavior while compiling to
     identical machine code, and making it self-evident that cb is an
     integral part of the synchronization mechanism on par with use.

Note that a full rework of the synchronization protocol (e.g. replacing
standalone memory barriers entirely with acquire/release semantics
across variables) was intentionally avoided to keep each commit simple,
localized, and non-controversial. More substantial architectural
changes, such as that or extending the library to support concurrent
multi-threaded dataplane execution, can be cleanly built on top of this
work in the future.

Marat Khalili (9):
  bpf: document packet callback unload sequence
  bpf: document ethdev callback threading limitations
  bpf: reject Tx callback installation with MT_LOCKFREE
  bpf: add callback flag to skip multi-thread check
  bpf: remove redundant memory barrier on unload
  bpf: simplify callback inuse tracking to boolean flag
  bpf: reformat packet callback unload diagram
  bpf: use relaxed stores for callback inuse tracking
  bpf: make callback handle atomic with relaxed access

 app/test/test_bpf.c      | 123 ++++++++++++++++++++++++++++
 lib/bpf/bpf_pkt.c        | 172 +++++++++++++++++++++++++++++++--------
 lib/bpf/rte_bpf_ethdev.h |  11 ++-
 3 files changed, 271 insertions(+), 35 deletions(-)

-- 
2.43.0

[PATCH 1/9] bpf: document packet callback unload sequence

[Marat Khalili]

Add a detailed comment with the sequence diagram to bpf_pkt.c describing
the current de-facto synchronization protocol on BPF program unload
between the control plane and the data plane.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 64 insertions(+), 1 deletion(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index f072fdaaed..4223ef624e 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -21,9 +21,72 @@
 #include "bpf_impl.h"
 
 /*
- * information about installed BPF rx/tx callback
+ * CP-DP synchronization mechanism
+ *
+ * After the callback is installed to a port/queue, removing it safely is a non-trivial task.
+ * The `rte_bpf` object can only be destroyed while the callback is not being executed,
+ * while the number of synchronization calls in the packet callback must be minimized.
+ *
+ * APIs `rte_eth_remove_rx_callback` and `rte_eth_remove_tx_callback` remove the callback from
+ * the callback list and upon return guarantee that it will not be called again,
+ * but not that any existing invocation of the callback has completed.
+ *
+ * Current synchronization mechanism relies on the following per-port/queue entities:
+ * 1. List of installed port/queue callbacks ("cbs" on the diagram);
+ * 2. Member `cb` of struct `bpf_eth_cbi`;
+ * 3. Member `use` of struct `bpf_eth_cbi`;
+ *
+ * BPF program is only executed if the first two are set, and increments `use` at the start and
+ * end of the callback execution so that in the safe state its value is even.
+ * This relies on only one thread calling the per-port/queue callback at a time.
+ *
+ * Unload processing starts with clearing the first two entities,
+ * then waits for `use` to become even before destroying the BPF program.
+ * The whole operation is protected by a control-plane-only spinlock (not shown on the diagram).
+ *
+ * Typical sequence of synchronization actions between CP and DP on unload is shown below:
+ * ```
+ *   +----+                +-----+         +-----+        +----+         +-----+          +----+
+ *   | CP |                | cbs |         | use |        | cb |         | bpf |          | DP |
+ *   +----+                +-----+         +-----+        +----+         +-----+          +----+
+ *     |                      |               |              |              |                |
+ *     |                      |<---------------------- fetch callback -----------------------|
+ *     |                      |               |              |              |                |
+ *     |                      |               |<------------- increment to odd --------------|
+ *     |                      |               |              |              |                |
+ *     |-- remove callback -->|               |              |              |       rte_smp_mb()
+ *     |                      |               |              |              |                |
+ *     |                      |               |              |<------ fetch non-NULL --------|
+ *     |                      |               |              |              |                |
+ *     |------------------- store NULL --------------------->|              |                |
+ *     |                      |               |              |              |                |
+ *   rte_smp_mb()             |               |              |              |                |
+ *     |                      |               |              |              |                |
+ *   rte_smp_mb()             |               |              |              |                |
+ *     |                      |               |              |              |                |
+ *     +-if use is odd        |               |              |              | if cb not NULL-+
+ *     | |                    |               |              |              |              | |
+ *     | +------------ fetch odd ------------>|              |              |<-- execute --+ |
+ *     | |                    |               |              |              |  BPF program | |
+ *     | +-while use is same  |               |              |              |              +-+
+ *     | | |                  |               |              |              |                |
+ *     | | +---------- fetch odd ------------>|              |              |      rte_smp_rmb()
+ *     | | |                  |               |              |              |                |
+ *     +-+-+                  |               |<------------- increment to even -------------|
+ *     |                      |               |              |              |                |
+ *     |------------- fetch even ------------>|              |              |                |
+ *     |                      |               |              |              |                |
+ *     |----------------- destroy BPF program and cleanup ----------------->|                |
+ *     |                      |               |              |              |                |
+ *   +----+                +-----+         +-----+        +----+         +-----+          +----+
+ *   | CP |                | cbs |         | use |        | cb |         | bpf |          | DP |
+ *   +----+                +-----+         +-----+        +----+         +-----+          +----+
+ * ```
  */
 
+/*
+ * information about installed BPF rx/tx callback
+ */
 struct __rte_cache_aligned bpf_eth_cbi {
 	/* used by both data & control path */
 	RTE_ATOMIC(uint32_t) use;    /*usage counter */
-- 
2.43.0

[PATCH 2/9] bpf: document ethdev callback threading limitations

[Marat Khalili]

Document the data plane limitation in the file docstring of
rte_bpf_ethdev.h: packet callback execution expects only one
thread per port/queue pair at a time, making concurrent bursts
(such as with RTE_ETH_TX_OFFLOAD_MT_LOCKFREE, shared RX queues,
or multi-threaded software PMDs) unsupported.

Rephrase the existing control plane multi-threading limitation
to explicitly distinguish it from the data plane requirement.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/rte_bpf_ethdev.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/lib/bpf/rte_bpf_ethdev.h b/lib/bpf/rte_bpf_ethdev.h
index 686fda8d94..cf3b4048f0 100644
--- a/lib/bpf/rte_bpf_ethdev.h
+++ b/lib/bpf/rte_bpf_ethdev.h
@@ -10,8 +10,12 @@
  *
  * API to install BPF filter as RX/TX callbacks for eth devices.
  * Note that right now:
- * - it is not MT safe, i.e. it is not allowed to do load/unload for the
- *   same port/queue from different threads in parallel.
+ * - in the data plane, it expects only one thread per port/queue pair at a time.
+ *   Calling RX/TX burst functions concurrently on the same queue (e.g. when using
+ *   RTE_ETH_TX_OFFLOAD_MT_LOCKFREE, shared RX queues, or multi-threaded
+ *   software PMDs) is not supported.
+ * - in the control plane, it is not MT safe, i.e. it is not allowed to do
+ *   load/unload for the same port/queue from different threads in parallel.
  * - though it allows load/unload at runtime
  *   (while RX/TX is ongoing on given port/queue).
  * - allows only one BPF program per port/queue,
-- 
2.43.0

[PATCH 3/9] bpf: reject Tx callback installation with MT_LOCKFREE

[Marat Khalili]

The BPF ethdev packet callback synchronization mechanism expects only
one dataplane thread per port/queue pair at a time. When
RTE_ETH_TX_OFFLOAD_MT_LOCKFREE is enabled, multiple threads can invoke
rte_eth_tx_burst() concurrently on the same queue without locks. This
breaks the callback parity counter and can lead to use-after-free or
deadlocks during BPF program unload.

Reject BPF Tx callback installation with -ENOTSUP when
RTE_ETH_TX_OFFLOAD_MT_LOCKFREE is enabled on the port or queue.
Add a unit test in test_bpf verifying this rejection.

Bugzilla ID: 2051
Fixes: a93ff62a8938 ("bpf: introduce basic Rx/Tx filters")
Cc: stable@dpdk.org

Reported-by: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 app/test/test_bpf.c | 103 ++++++++++++++++++++++++++++++++++++++++++++
 lib/bpf/bpf_pkt.c   |  46 ++++++++++++++++++++
 2 files changed, 149 insertions(+)

diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c
index 3205afaa63..02523984b5 100644
--- a/app/test/test_bpf.c
+++ b/app/test/test_bpf.c
@@ -4617,6 +4617,107 @@ test_bpf_elf_rx_load(load_elf_image_t load_elf_image)
 	return ret == 0 ? TEST_SUCCESS : TEST_FAILED;
 }
 
+static int
+null_vdev_mt_setup(const char *name, uint16_t *port, struct rte_mempool *pool)
+{
+	struct rte_eth_conf conf = {
+		.txmode = {
+			.offloads = RTE_ETH_TX_OFFLOAD_MT_LOCKFREE,
+		},
+	};
+	struct rte_eth_txconf txconf = { };
+	struct rte_eth_rxconf rxconf = { };
+	int ret;
+
+	ret = rte_vdev_init(name, NULL);
+	TEST_ASSERT(ret == 0, "rte_vdev_init(%s) failed: %d", name, ret);
+
+	ret = rte_eth_dev_get_port_by_name(name, port);
+	TEST_ASSERT(ret == 0, "failed to get port id for %s: %d", name, ret);
+
+	ret = rte_eth_dev_configure(*port, 1, 1, &conf);
+	TEST_ASSERT(ret == 0, "failed to configure port %u: %d", *port, ret);
+
+	ret = rte_eth_tx_queue_setup(*port, 0, BPF_TEST_BURST, SOCKET_ID_ANY, &txconf);
+	TEST_ASSERT(ret == 0, "failed to setup tx queue port %u: %d", *port, ret);
+
+	ret = rte_eth_rx_queue_setup(*port, 0, BPF_TEST_BURST, SOCKET_ID_ANY,
+				     &rxconf, pool);
+	TEST_ASSERT(ret == 0, "failed to setup rx queue port %u: %d", *port, ret);
+
+	ret = rte_eth_dev_start(*port);
+	TEST_ASSERT(ret == 0, "failed to start port %u: %d", *port, ret);
+
+	return 0;
+}
+
+/* Test that installing a Tx filter is rejected when MT_LOCKFREE is enabled */
+static int
+test_bpf_elf_tx_mt_lockfree(load_elf_image_t load_elf_image)
+{
+	static const struct rte_bpf_arg prog_arg = {
+		.type = RTE_BPF_ARG_PTR,
+		.size = sizeof(struct dummy_net),
+	};
+	static const char null_dev[] = "net_null_bpf_mt";
+	struct rte_mempool *mb_pool = NULL;
+	struct rte_bpf *bpf = NULL;
+	uint16_t port = UINT16_MAX;
+	int ret;
+
+	mb_pool = rte_pktmbuf_pool_create("bpf_mt_test_pool", BPF_TEST_POOLSIZE,
+					  0, 0, RTE_MBUF_DEFAULT_BUF_SIZE,
+					  SOCKET_ID_ANY);
+	TEST_ASSERT_NOT_NULL(mb_pool, "failed to create mempool");
+
+	ret = null_vdev_mt_setup(null_dev, &port, mb_pool);
+	if (ret != 0)
+		goto fail;
+
+	bpf = load_elf_image(app_test_bpf_filter_o, app_test_bpf_filter_o_len, "filter",
+			     NULL, 0, &prog_arg);
+	if (bpf == NULL) {
+		printf("%s@%d: failed to load BPF filter image\n",
+		       __func__, __LINE__);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+
+	/* Tx installation must fail with -ENOTSUP due to MT_LOCKFREE */
+	ret = rte_bpf_eth_tx_install(port, 0, bpf, 0);
+	if (ret != -ENOTSUP) {
+		printf("%s@%d: expected rte_bpf_eth_tx_install to return -ENOTSUP, got %d\n",
+		       __func__, __LINE__, ret);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+
+	/* Rx installation on the same port must still succeed */
+	ret = rte_bpf_eth_rx_install(port, 0, bpf, 0);
+	if (ret != 0) {
+		printf("%s@%d: expected rte_bpf_eth_rx_install to succeed, got %d\n",
+		       __func__, __LINE__, ret);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+	rte_bpf_eth_rx_unload(port, 0);
+	/* bpf is destroyed by rte_bpf_eth_rx_unload */
+	bpf = NULL;
+
+	ret = TEST_SUCCESS;
+
+fail:
+	if (bpf != NULL)
+		rte_bpf_destroy(bpf);
+
+	if (port != UINT16_MAX) {
+		rte_eth_dev_stop(port);
+		rte_vdev_uninit(null_dev);
+	}
+
+	rte_mempool_free(mb_pool);
+	return ret;
+}
 
 static int
 test_bpf_elf(void)
@@ -4633,6 +4734,8 @@ test_bpf_elf(void)
 			ret = test_bpf_elf_load(elf_image_loaders[li]);
 		if (ret == TEST_SUCCESS)
 			ret = test_bpf_elf_tx_load(elf_image_loaders[li]);
+		if (ret == TEST_SUCCESS)
+			ret = test_bpf_elf_tx_mt_lockfree(elf_image_loaders[li]);
 		if (ret == TEST_SUCCESS)
 			ret = test_bpf_elf_rx_load(elf_image_loaders[li]);
 	}
diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index 4223ef624e..eab85f7417 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -552,6 +552,48 @@ rte_bpf_eth_tx_unload(uint16_t port, uint16_t queue)
 	rte_spinlock_unlock(&cbh->lock);
 }
 
+/*
+ * Check if the supplied port/queue pair is suitable for BPF callback execution.
+ */
+static int
+bpf_eth_check_mt(const struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue)
+{
+	struct rte_eth_txq_info qinfo;
+	struct rte_eth_conf conf;
+	int ret;
+
+	if (cbh->type == BPF_ETH_TX) {
+		ret = rte_eth_dev_conf_get(port, &conf);
+		if (ret != 0) {
+			RTE_BPF_LOG_LINE(ERR,
+				"%s(%u, %u): failed to get device configuration: %d;",
+				__func__, port, queue, ret);
+			return ret;
+		}
+
+		if ((conf.txmode.offloads & RTE_ETH_TX_OFFLOAD_MT_LOCKFREE) != 0) {
+			RTE_BPF_LOG_LINE(ERR,
+				"%s(%u, %u): MT_LOCKFREE Tx offload is enabled on port;",
+				__func__, port, queue);
+			return -ENOTSUP;
+		}
+
+		/*
+		 * Ignore errors from tx_queue_info_get: the PMD might not implement
+		 * txq_info_get (e.g. net/null), or the queue might not be set up yet.
+		 */
+		if (rte_eth_tx_queue_info_get(port, queue, &qinfo) == 0 &&
+				(qinfo.conf.offloads & RTE_ETH_TX_OFFLOAD_MT_LOCKFREE) != 0) {
+			RTE_BPF_LOG_LINE(ERR,
+				"%s(%u, %u): MT_LOCKFREE Tx offload is enabled on queue;",
+				__func__, port, queue);
+			return -ENOTSUP;
+		}
+	}
+
+	return 0;
+}
+
 static int
 bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 	struct rte_bpf *bpf, uint32_t flags)
@@ -572,6 +614,10 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 	if (bpf->prm.nb_prog_arg != 1)
 		return -EINVAL;
 
+	rc = bpf_eth_check_mt(cbh, port, queue);
+	if (rc != 0)
+		return rc;
+
 	if (cbh->type == BPF_ETH_RX)
 		frx = select_rx_callback(bpf->prm.prog_arg[0].type, flags);
 	else
-- 
2.43.0

[PATCH 4/9] bpf: add callback flag to skip multi-thread check

[Marat Khalili]

Add RTE_BPF_ETH_F_SKIP_MT_CHECK flag allowing callers to bypass the
suitability check when installing BPF packet callbacks.

This allows overriding rejections in cases where the port or queue is
configured with multi-thread capabilities (such as MT_LOCKFREE),
but the application guarantees that only one dataplane thread
accesses the queue at a time.

Update the unit test to verify that callback installation succeeds
when this flag is specified.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 app/test/test_bpf.c      | 20 ++++++++++++++++++++
 lib/bpf/bpf_pkt.c        |  8 +++++---
 lib/bpf/rte_bpf_ethdev.h |  5 ++++-
 3 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c
index 02523984b5..0ab8aa8759 100644
--- a/app/test/test_bpf.c
+++ b/app/test/test_bpf.c
@@ -4692,7 +4692,27 @@ test_bpf_elf_tx_mt_lockfree(load_elf_image_t load_elf_image)
 		goto fail;
 	}
 
+	/* Tx installation must succeed when check is skipped with flag */
+	ret = rte_bpf_eth_tx_install(port, 0, bpf, RTE_BPF_ETH_F_SKIP_MT_CHECK);
+	if (ret != 0) {
+		printf("%s@%d: expected rte_bpf_eth_tx_install with SKIP_MT_CHECK to succeed, got %d\n",
+		       __func__, __LINE__, ret);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+	rte_bpf_eth_tx_unload(port, 0);
+	bpf = NULL;
+
 	/* Rx installation on the same port must still succeed */
+	bpf = load_elf_image(app_test_bpf_filter_o, app_test_bpf_filter_o_len, "filter",
+			     NULL, 0, &prog_arg);
+	if (bpf == NULL) {
+		printf("%s@%d: failed to load BPF filter image\n",
+		       __func__, __LINE__);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+
 	ret = rte_bpf_eth_rx_install(port, 0, bpf, 0);
 	if (ret != 0) {
 		printf("%s@%d: expected rte_bpf_eth_rx_install to succeed, got %d\n",
diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index eab85f7417..7ee16cfa69 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -614,9 +614,11 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 	if (bpf->prm.nb_prog_arg != 1)
 		return -EINVAL;
 
-	rc = bpf_eth_check_mt(cbh, port, queue);
-	if (rc != 0)
-		return rc;
+	if ((flags & RTE_BPF_ETH_F_SKIP_MT_CHECK) == 0) {
+		rc = bpf_eth_check_mt(cbh, port, queue);
+		if (rc != 0)
+			return rc;
+	}
 
 	if (cbh->type == BPF_ETH_RX)
 		frx = select_rx_callback(bpf->prm.prog_arg[0].type, flags);
diff --git a/lib/bpf/rte_bpf_ethdev.h b/lib/bpf/rte_bpf_ethdev.h
index cf3b4048f0..7a9f87e6cb 100644
--- a/lib/bpf/rte_bpf_ethdev.h
+++ b/lib/bpf/rte_bpf_ethdev.h
@@ -13,7 +13,8 @@
  * - in the data plane, it expects only one thread per port/queue pair at a time.
  *   Calling RX/TX burst functions concurrently on the same queue (e.g. when using
  *   RTE_ETH_TX_OFFLOAD_MT_LOCKFREE, shared RX queues, or multi-threaded
- *   software PMDs) is not supported.
+ *   software PMDs) is not supported and will be rejected at installation time
+ *   unless overridden by RTE_BPF_ETH_F_SKIP_MT_CHECK.
  * - in the control plane, it is not MT safe, i.e. it is not allowed to do
  *   load/unload for the same port/queue from different threads in parallel.
  * - though it allows load/unload at runtime
@@ -37,6 +38,8 @@ extern "C" {
 enum {
 	RTE_BPF_ETH_F_NONE = 0,
 	RTE_BPF_ETH_F_JIT  = 0x1, /**< use compiled into native ISA code */
+	RTE_BPF_ETH_F_SKIP_MT_CHECK = 0x2,
+	/**< skip check for concurrent data plane thread usage */
 };
 
 /**
-- 
2.43.0

[PATCH 5/9] bpf: remove redundant memory barrier on unload

[Marat Khalili]

In bpf_eth_cbi_unload(), rte_smp_mb() was called immediately before
bpf_eth_cbi_wait(), which itself starts with rte_smp_mb().

Remove the redundant rte_smp_mb() call from bpf_eth_cbi_unload() and
update the unload sequence diagram accordingly.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index 7ee16cfa69..68671848e9 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -62,8 +62,6 @@
  *     |                      |               |              |              |                |
  *   rte_smp_mb()             |               |              |              |                |
  *     |                      |               |              |              |                |
- *   rte_smp_mb()             |               |              |              |                |
- *     |                      |               |              |              |                |
  *     +-if use is odd        |               |              |              | if cb not NULL-+
  *     | |                    |               |              |              |              | |
  *     | +------------ fetch odd ------------>|              |              |<-- execute --+ |
@@ -502,7 +500,6 @@ bpf_eth_cbi_unload(struct bpf_eth_cbi *bc)
 {
 	/* mark this cbi as empty */
 	bc->cb = NULL;
-	rte_smp_mb();
 
 	/* make sure datapath doesn't use bpf anymore, then destroy bpf */
 	bpf_eth_cbi_wait(bc);
-- 
2.43.0

[PATCH 6/9] bpf: simplify callback inuse tracking to boolean flag

[Marat Khalili]

The even/odd usage counter semantics in struct bpf_eth_cbi was
originally used to detect when in-flight packet callbacks complete.
However, with at most one thread running the callback per port/queue
pair and no risk of the ABA problem, a binary flag (1 for in use,
0 for idle) suffices.

Replace the counter increment in the datapath with stores of 1 and 0,
and simplify bpf_eth_cbi_wait() to wait directly until the flag is 0.
Update the documentation and synchronization sequence diagram
accordingly.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 46 ++++++++++++++++++++--------------------------
 1 file changed, 20 insertions(+), 26 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index 68671848e9..7ae64ed7d0 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -36,12 +36,12 @@
  * 2. Member `cb` of struct `bpf_eth_cbi`;
  * 3. Member `use` of struct `bpf_eth_cbi`;
  *
- * BPF program is only executed if the first two are set, and increments `use` at the start and
- * end of the callback execution so that in the safe state its value is even.
+ * BPF program is only executed if the first two are set, and sets `use` to 1 at the start and
+ * to 0 at the end of the callback execution.
  * This relies on only one thread calling the per-port/queue callback at a time.
  *
  * Unload processing starts with clearing the first two entities,
- * then waits for `use` to become even before destroying the BPF program.
+ * then waits for `use` to become 0 before destroying the BPF program.
  * The whole operation is protected by a control-plane-only spinlock (not shown on the diagram).
  *
  * Typical sequence of synchronization actions between CP and DP on unload is shown below:
@@ -52,7 +52,7 @@
  *     |                      |               |              |              |                |
  *     |                      |<---------------------- fetch callback -----------------------|
  *     |                      |               |              |              |                |
- *     |                      |               |<------------- increment to odd --------------|
+ *     |                      |               |<------------- store 1 -----------------------|
  *     |                      |               |              |              |                |
  *     |-- remove callback -->|               |              |              |       rte_smp_mb()
  *     |                      |               |              |              |                |
@@ -62,17 +62,17 @@
  *     |                      |               |              |              |                |
  *   rte_smp_mb()             |               |              |              |                |
  *     |                      |               |              |              |                |
- *     +-if use is odd        |               |              |              | if cb not NULL-+
+ *     +-while use is 1       |               |              |              | if cb not NULL-+
  *     | |                    |               |              |              |              | |
- *     | +------------ fetch odd ------------>|              |              |<-- execute --+ |
+ *     | +------------ fetch 1 -------------->|              |              |<-- execute --+ |
  *     | |                    |               |              |              |  BPF program | |
- *     | +-while use is same  |               |              |              |              +-+
- *     | | |                  |               |              |              |                |
- *     | | +---------- fetch odd ------------>|              |              |      rte_smp_rmb()
- *     | | |                  |               |              |              |                |
- *     +-+-+                  |               |<------------- increment to even -------------|
+ *     | |                    |               |              |              |              +-+
+ *     | |                    |               |              |              |                |
+ *     | |                    |               |              |              |      rte_smp_rmb()
+ *     | |                    |               |              |              |                |
+ *     +-+                    |               |<------------- store 0 -----------------------|
  *     |                      |               |              |              |                |
- *     |------------- fetch even ------------>|              |              |                |
+ *     |------------- fetch 0 --------------->|              |              |                |
  *     |                      |               |              |              |                |
  *     |----------------- destroy BPF program and cleanup ----------------->|                |
  *     |                      |               |              |              |                |
@@ -87,7 +87,7 @@
  */
 struct __rte_cache_aligned bpf_eth_cbi {
 	/* used by both data & control path */
-	RTE_ATOMIC(uint32_t) use;    /*usage counter */
+	RTE_ATOMIC(uint32_t) use;    /* usage flag: 1 when in use, 0 otherwise */
 	const struct rte_eth_rxtx_callback *cb;  /* callback handle */
 	struct rte_bpf *bpf;
 	struct rte_bpf_jit jit;
@@ -98,8 +98,8 @@ struct __rte_cache_aligned bpf_eth_cbi {
 };
 
 /*
- * Odd number means that callback is used by datapath.
- * Even number means that callback is not used by datapath.
+ * 1 means that callback is used by datapath.
+ * 0 means that callback is not used by datapath.
  */
 #define BPF_ETH_CBI_INUSE  1
 
@@ -141,20 +141,20 @@ static struct bpf_eth_cbh tx_cbh = {
 static __rte_always_inline void
 bpf_eth_cbi_inuse(struct bpf_eth_cbi *cbi)
 {
-	cbi->use++;
+	cbi->use = BPF_ETH_CBI_INUSE;
 	/* make sure no store/load reordering could happen */
 	rte_smp_mb();
 }
 
 /*
- * Marks given callback list as not used by datapath.
+ * Marks given callback as not used by datapath.
  */
 static __rte_always_inline void
 bpf_eth_cbi_unuse(struct bpf_eth_cbi *cbi)
 {
 	/* make sure all previous loads are completed */
 	rte_smp_rmb();
-	cbi->use++;
+	cbi->use = 0;
 }
 
 /*
@@ -163,18 +163,12 @@ bpf_eth_cbi_unuse(struct bpf_eth_cbi *cbi)
 static void
 bpf_eth_cbi_wait(const struct bpf_eth_cbi *cbi)
 {
-	uint32_t puse;
-
 	/* make sure all previous loads and stores are completed */
 	rte_smp_mb();
 
-	puse = cbi->use;
-
 	/* in use, busy wait till current RX/TX iteration is finished */
-	if ((puse & BPF_ETH_CBI_INUSE) != 0) {
-		RTE_WAIT_UNTIL_MASKED((__rte_atomic uint32_t *)(uintptr_t)&cbi->use,
-			UINT32_MAX, !=, puse, rte_memory_order_relaxed);
-	}
+	RTE_WAIT_UNTIL_MASKED((__rte_atomic uint32_t *)(uintptr_t)&cbi->use,
+		UINT32_MAX, ==, 0, rte_memory_order_relaxed);
 }
 
 static void
-- 
2.43.0

[PATCH 7/9] bpf: reformat packet callback unload diagram

[Marat Khalili]

Reformat the CP-DP synchronization sequence diagram in bpf_pkt.c by
removing unnecessary vertical lines and re-centering arrow labels.

This comment-only commit is intentionally kept separate to make the
diff of the preceding semantic change more readable.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 22 +++++++++-------------
 1 file changed, 9 insertions(+), 13 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index 7ae64ed7d0..ea7e240494 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -52,27 +52,23 @@
  *     |                      |               |              |              |                |
  *     |                      |<---------------------- fetch callback -----------------------|
  *     |                      |               |              |              |                |
- *     |                      |               |<------------- store 1 -----------------------|
+ *     |                      |               |<------------------ store 1 ------------------|
  *     |                      |               |              |              |                |
  *     |-- remove callback -->|               |              |              |       rte_smp_mb()
  *     |                      |               |              |              |                |
  *     |                      |               |              |<------ fetch non-NULL --------|
  *     |                      |               |              |              |                |
- *     |------------------- store NULL --------------------->|              |                |
- *     |                      |               |              |              |                |
- *   rte_smp_mb()             |               |              |              |                |
- *     |                      |               |              |              |                |
- *     +-while use is 1       |               |              |              | if cb not NULL-+
- *     | |                    |               |              |              |              | |
- *     | +------------ fetch 1 -------------->|              |              |<-- execute --+ |
- *     | |                    |               |              |              |  BPF program | |
- *     | |                    |               |              |              |              +-+
+ *     |------------------- store NULL --------------------->|              | if cb not NULL-+
+ *     |                      |               |              |              |              | |
+ *   rte_smp_mb()             |               |              |              |<-- execute --+ |
+ *     |                      |               |              |              |  BPF program | |
+ *     +-while use is 1       |               |              |              |              +-+
  *     | |                    |               |              |              |                |
- *     | |                    |               |              |              |      rte_smp_rmb()
+ *     | +------------ fetch 1 -------------->|              |              |      rte_smp_rmb()
  *     | |                    |               |              |              |                |
- *     +-+                    |               |<------------- store 0 -----------------------|
+ *     +-+                    |               |<------------------ store 0 ------------------|
  *     |                      |               |              |              |                |
- *     |------------- fetch 0 --------------->|              |              |                |
+ *     |-------------- fetch 0 -------------->|              |              |                |
  *     |                      |               |              |              |                |
  *     |----------------- destroy BPF program and cleanup ----------------->|                |
  *     |                      |               |              |              |                |
-- 
2.43.0

[PATCH 8/9] bpf: use relaxed stores for callback inuse tracking

[Marat Khalili]

Direct assignment to an atomic variable defaults to sequentially
consistent memory order in C11. Since cross-thread memory ordering
for packet callback execution and unload is already explicitly
enforced by memory barriers, sequentially consistent stores are
unnecessary and may emit redundant barrier instructions on weakly
ordered architectures.

Update bpf_eth_cbi_inuse() and bpf_eth_cbi_unuse() to use
rte_atomic_store_explicit() with rte_memory_order_relaxed, matching
the relaxed loads already used in bpf_eth_cbi_wait().

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index ea7e240494..9dee001f0f 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -137,7 +137,7 @@ static struct bpf_eth_cbh tx_cbh = {
 static __rte_always_inline void
 bpf_eth_cbi_inuse(struct bpf_eth_cbi *cbi)
 {
-	cbi->use = BPF_ETH_CBI_INUSE;
+	rte_atomic_store_explicit(&cbi->use, BPF_ETH_CBI_INUSE, rte_memory_order_relaxed);
 	/* make sure no store/load reordering could happen */
 	rte_smp_mb();
 }
@@ -150,7 +150,7 @@ bpf_eth_cbi_unuse(struct bpf_eth_cbi *cbi)
 {
 	/* make sure all previous loads are completed */
 	rte_smp_rmb();
-	cbi->use = 0;
+	rte_atomic_store_explicit(&cbi->use, 0, rte_memory_order_relaxed);
 }
 
 /*
-- 
2.43.0

[PATCH 9/9] bpf: make callback handle atomic with relaxed access

[Marat Khalili]

The callback handle member cb of struct bpf_eth_cbi is concurrently
cleared by the control plane on unload and read by the datapath to
decide whether to execute the BPF filter program. Without atomic typing,
this concurrent access constitutes a data race in the C11 memory model,
even though hardware pointer accesses are single-copy atomic and untorn.

Change cb to RTE_ATOMIC and use rte_atomic_load_explicit() and
rte_atomic_store_explicit() with rte_memory_order_relaxed for all its
accesses. On modern architectures this produces the exact same machine
code while eliminating undefined behavior.

In addition, marking cb as atomic improves code self-documentation by
making it immediately evident that cb is an active part of the cross-
thread synchronization mechanism on par with use.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index 9dee001f0f..fabdb52d7e 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -84,7 +84,7 @@
 struct __rte_cache_aligned bpf_eth_cbi {
 	/* used by both data & control path */
 	RTE_ATOMIC(uint32_t) use;    /* usage flag: 1 when in use, 0 otherwise */
-	const struct rte_eth_rxtx_callback *cb;  /* callback handle */
+	RTE_ATOMIC(const struct rte_eth_rxtx_callback *) cb;  /* callback handle */
 	struct rte_bpf *bpf;
 	struct rte_bpf_jit jit;
 	/* used by control path only */
@@ -318,7 +318,7 @@ bpf_rx_callback_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	cbi = user_param;
 
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 1) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -335,7 +335,7 @@ bpf_rx_callback_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_jit(&cbi->jit, pkt, nb_pkts, 1) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -351,7 +351,7 @@ bpf_tx_callback_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 0) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -367,7 +367,7 @@ bpf_tx_callback_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_jit(&cbi->jit, pkt, nb_pkts, 0) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -388,7 +388,7 @@ bpf_rx_callback_mb_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_mb_vm(cbi->bpf, pkt, nb_pkts, 1) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -405,7 +405,7 @@ bpf_rx_callback_mb_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_mb_jit(&cbi->jit, pkt, nb_pkts, 1) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -421,7 +421,7 @@ bpf_tx_callback_mb_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_mb_vm(cbi->bpf, pkt, nb_pkts, 0) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -437,7 +437,7 @@ bpf_tx_callback_mb_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_mb_jit(&cbi->jit, pkt, nb_pkts, 0) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -489,7 +489,7 @@ static void
 bpf_eth_cbi_unload(struct bpf_eth_cbi *bc)
 {
 	/* mark this cbi as empty */
-	bc->cb = NULL;
+	rte_atomic_store_explicit(&bc->cb, NULL, rte_memory_order_relaxed);
 
 	/* make sure datapath doesn't use bpf anymore, then destroy bpf */
 	bpf_eth_cbi_wait(bc);
@@ -500,16 +500,21 @@ bpf_eth_cbi_unload(struct bpf_eth_cbi *bc)
 static void
 bpf_eth_unload(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue)
 {
+	const struct rte_eth_rxtx_callback *cb;
 	struct bpf_eth_cbi *bc;
 
 	bc = bpf_eth_cbh_find(cbh, port, queue);
-	if (bc == NULL || bc->cb == NULL)
+	if (bc == NULL)
+		return;
+
+	cb = rte_atomic_load_explicit(&bc->cb, rte_memory_order_relaxed);
+	if (cb == NULL)
 		return;
 
 	if (cbh->type == BPF_ETH_RX)
-		rte_eth_remove_rx_callback(port, queue, bc->cb);
+		rte_eth_remove_rx_callback(port, queue, cb);
 	else
-		rte_eth_remove_tx_callback(port, queue, bc->cb);
+		rte_eth_remove_tx_callback(port, queue, cb);
 
 	bpf_eth_cbi_unload(bc);
 }
@@ -586,6 +591,7 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 	struct rte_bpf *bpf, uint32_t flags)
 {
 	int32_t rc;
+	const struct rte_eth_rxtx_callback *cb;
 	struct bpf_eth_cbi *bc;
 	rte_rx_callback_fn frx;
 	rte_tx_callback_fn ftx;
@@ -632,18 +638,20 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 		return -ENOMEM;
 
 	/* remove old one, if any */
-	if (bc->cb != NULL)
+	if (rte_atomic_load_explicit(&bc->cb, rte_memory_order_relaxed) != NULL)
 		bpf_eth_unload(cbh, port, queue);
 
 	bc->bpf = bpf;
 	bc->jit = jit;
 
 	if (cbh->type == BPF_ETH_RX)
-		bc->cb = rte_eth_add_rx_callback(port, queue, frx, bc);
+		cb = rte_eth_add_rx_callback(port, queue, frx, bc);
 	else
-		bc->cb = rte_eth_add_tx_callback(port, queue, ftx, bc);
+		cb = rte_eth_add_tx_callback(port, queue, ftx, bc);
+
+	rte_atomic_store_explicit(&bc->cb, cb, rte_memory_order_relaxed);
 
-	if (bc->cb == NULL) {
+	if (cb == NULL) {
 		rc = -rte_errno;
 		bpf_eth_cbi_cleanup(bc);
 	} else
-- 
2.43.0

Re: [PATCH 4/9] bpf: add callback flag to skip multi-thread check

[Stephen Hemminger]

On Sun, 11 Oct 2026 18:21:52 +0000
Marat Khalili <qm2k21@gmail.com> wrote:

> Add RTE_BPF_ETH_F_SKIP_MT_CHECK flag allowing callers to bypass the
> suitability check when installing BPF packet callbacks.
> 
> This allows overriding rejections in cases where the port or queue is
> configured with multi-thread capabilities (such as MT_LOCKFREE),
> but the application guarantees that only one dataplane thread
> accesses the queue at a time.
> 
> Update the unit test to verify that callback installation succeeds
> when this flag is specified.
> 
> Signed-off-by: Marat Khalili <qm2k21@gmail.com>
> ---

Rather than introduce another flag, I would rather the application
do the right thing. It can just not set the offload flag.

Re: [PATCH 5/9] bpf: remove redundant memory barrier on unload

[Stephen Hemminger]

On Sun, 11 Oct 2026 18:21:53 +0000
Marat Khalili <qm2k21@gmail.com> wrote:

> In bpf_eth_cbi_unload(), rte_smp_mb() was called immediately before
> bpf_eth_cbi_wait(), which itself starts with rte_smp_mb().
> 
> Remove the redundant rte_smp_mb() call from bpf_eth_cbi_unload() and
> update the unload sequence diagram accordingly.
> 
> Signed-off-by: Marat Khalili <qm2k21@gmail.com>
> ---
>  lib/bpf/bpf_pkt.c | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
> index 7ee16cfa69..68671848e9 100644
> --- a/lib/bpf/bpf_pkt.c
> +++ b/lib/bpf/bpf_pkt.c
> @@ -62,8 +62,6 @@
>   *     |                      |               |              |              |                |
>   *   rte_smp_mb()             |               |              |              |                |
>   *     |                      |               |              |              |                |
> - *   rte_smp_mb()             |               |              |              |                |
> - *     |                      |               |              |              |                |
>   *     +-if use is odd        |               |              |              | if cb not NULL-+
>   *     | |                    |               |              |              |              | |
>   *     | +------------ fetch odd ------------>|              |              |<-- execute --+ |
> @@ -502,7 +500,6 @@ bpf_eth_cbi_unload(struct bpf_eth_cbi *bc)
>  {
>  	/* mark this cbi as empty */
>  	bc->cb = NULL;
> -	rte_smp_mb();
>  
>  	/* make sure datapath doesn't use bpf anymore, then destroy bpf */
>  	bpf_eth_cbi_wait(bc);

I think this already covered by:

https://patchwork.dpdk.org/project/dpdk/patch/20261008233649.1260843-2-stephen@networkplumber.org/

Re: [PATCH 6/9] bpf: simplify callback inuse tracking to boolean flag

[Stephen Hemminger]

On Sun, 11 Oct 2026 18:21:54 +0000
Marat Khalili <qm2k21@gmail.com> wrote:

> --- a/lib/bpf/bpf_pkt.c
> +++ b/lib/bpf/bpf_pkt.c
> @@ -36,12 +36,12 @@
>   * 2. Member `cb` of struct `bpf_eth_cbi`;
>   * 3. Member `use` of struct `bpf_eth_cbi`;
>   *
> - * BPF program is only executed if the first two are set, and increments `use` at the start and
> - * end of the callback execution so that in the safe state its value is even.
> + * BPF program is only executed if the first two are set, and sets `use` to 1 at the start and
> + * to 0 at the end of the callback execution.
>   * This relies on only one thread calling the per-port/queue callback at a time.
>   *
>   * Unload processing starts with clearing the first two entities,
> - * then waits for `use` to become even before destroying the BPF program.
> + * then waits for `use` to become 0 before destroying the BPF program.
>   * The whole operation is protected by a control-plane-only spinlock (not shown on the diagram).
>   *
>   * Typical sequence of synchronization actions between CP and DP on unload is shown below:
> @@ -52,7 +52,7 @@
>   *     |                      |               |              |              |                |
>   *     |                      |<---------------------- fetch callback -----------------------|
>   *     |                      |               |              |              |                |
> - *     |                      |               |<------------- increment to odd --------------|
> + *     |                      |               |<------------- store 1 -----------------------|
>   *     |                      |               |              |              |                |
>   *     |-- remove callback -->|               |              |              |       rte_smp_mb()
>   *     |                      |               |              |              |                |
> @@ -62,17 +62,17 @@
>   *     |                      |               |              |              |                |
>   *   rte_smp_mb()             |               |              |              |                |
>   *     |                      |               |              |              |                |
> - *     +-if use is odd        |               |              |              | if cb not NULL-+
> + *     +-while use is 1       |               |              |              | if cb not NULL-+
>   *     | |                    |               |              |              |              | |
> - *     | +------------ fetch odd ------------>|              |              |<-- execute --+ |
> + *     | +------------ fetch 1 -------------->|              |              |<-- execute --+ |
>   *     | |                    |               |              |              |  BPF program | |
> - *     | +-while use is same  |               |              |              |              +-+
> - *     | | |                  |               |              |              |                |
> - *     | | +---------- fetch odd ------------>|              |              |      rte_smp_rmb()
> - *     | | |                  |               |              |              |                |
> - *     +-+-+                  |               |<------------- increment to even -------------|
> + *     | |                    |               |              |              |              +-+
> + *     | |                    |               |              |              |                |
> + *     | |                    |               |              |              |      rte_smp_rmb()
> + *     | |                    |               |              |              |                |
> + *     +-+                    |               |<------------- store 0 -----------------------|
>   *     |                      |               |              |              |                |
> - *     |------------- fetch even ------------>|              |              |                |
> + *     |------------- fetch 0 --------------->|              |              |                |
>   *     |                      |               |              |              |                |
>   *     |----------------- destroy BPF program and cleanup ----------------->|                |
>   *     |                      |               |              |              |                |
> @@ -87,7 +87,7 @@
>   */
>  struct __rte_cache_aligned bpf_eth_cbi {
>  	/* used by both data & control path */
> -	RTE_ATOMIC(uint32_t) use;    /*usage counter */
> +	RTE_ATOMIC(uint32_t) use;    /* usage flag: 1 when in use, 0 otherwise */

If it is only on/off now make it a bool.

Re: [PATCH 8/9] bpf: use relaxed stores for callback inuse tracking

[Stephen Hemminger]

On Sun, 11 Oct 2026 18:21:56 +0000
Marat Khalili <qm2k21@gmail.com> wrote:

> Direct assignment to an atomic variable defaults to sequentially
> consistent memory order in C11. Since cross-thread memory ordering
> for packet callback execution and unload is already explicitly
> enforced by memory barriers, sequentially consistent stores are
> unnecessary and may emit redundant barrier instructions on weakly
> ordered architectures.
> 
> Update bpf_eth_cbi_inuse() and bpf_eth_cbi_unuse() to use
> rte_atomic_store_explicit() with rte_memory_order_relaxed, matching
> the relaxed loads already used in bpf_eth_cbi_wait().
> 
> Signed-off-by: Marat Khalili <qm2k21@gmail.com>
> ---

Already covered by smb_mb conversion patch.

If you want, can just make that into one patch which does
the conversion to bool and gets rid of smp_mb.

Re: [PATCH 9/9] bpf: make callback handle atomic with relaxed access

[Stephen Hemminger]

On Sun, 11 Oct 2026 18:21:57 +0000
Marat Khalili <qm2k21@gmail.com> wrote:

> The callback handle member cb of struct bpf_eth_cbi is concurrently
> cleared by the control plane on unload and read by the datapath to
> decide whether to execute the BPF filter program. Without atomic typing,
> this concurrent access constitutes a data race in the C11 memory model,
> even though hardware pointer accesses are single-copy atomic and untorn.
> 
> Change cb to RTE_ATOMIC and use rte_atomic_load_explicit() and
> rte_atomic_store_explicit() with rte_memory_order_relaxed for all its
> accesses. On modern architectures this produces the exact same machine
> code while eliminating undefined behavior.
> 
> In addition, marking cb as atomic improves code self-documentation by
> making it immediately evident that cb is an active part of the cross-
> thread synchronization mechanism on par with use.
> 
> Signed-off-by: Marat Khalili <qm2k21@gmail.com>
> ---

Since inuse and cb are inter-related would be easier to squash the mb
conversion and the bool and the atomic cb into one patch.

Re: [PATCH 4/9] bpf: add callback flag to skip multi-thread check

[Marat Khalili]

> Rather than introduce another flag, I would rather the application
> do the right thing. It can just not set the offload flag.

I can imagine a complex app where this new flag would make user's life 
easier when switching to a new version.

We can always skip this patch if there is no consensus, nothing depends 
on it.

Re: [PATCH 5/9] bpf: remove redundant memory barrier on unload

[Marat Khalili]

> I think this already covered by:
>
> https://patchwork.dpdk.org/project/dpdk/patch/20261008233649.1260843-2-stephen@networkplumber.org/
Sorry about that, I will credit you in v2. Since current patch also 
involves documentation update, I cannot just drop it, and I do not see 
your change in trunk.

Re: [PATCH 6/9] bpf: simplify callback inuse tracking to boolean flag

[Marat Khalili]

> If it is only on/off now make it a bool.

Changing it to bool may have performance impact on some architectures. 
Also, we'll have to make it int again when adding support for 
multi-threaded data planes.

You seem to be using this API more than me so I will leave the decision 
to you (and other reviewers), but please consider the points above first.

Re: [PATCH 8/9] bpf: use relaxed stores for callback inuse tracking

[Marat Khalili]

> Already covered by smb_mb conversion patch.
Will credit you in v2, but these patches are not exactly 
interchangeable. In fact, I was hoping to make your work of removing 
dependence on deprecated symbols easier here, since there there seem to 
be multiple diverging opinions about the scope and the best way forward.
> If you want, can just make that into one patch which does
> the conversion to bool and gets rid of smp_mb.
Not sure how they are related, especially since this one seems to be 
trivial and safe, while the bool one is potentially tricky.

[RFC] bpf: fix callback unload with concurrent Tx burst

[Stephen Hemminger]

This is AI remix of the smp_mb barrier removal patch, the patch series from Marat.

The Rx/Tx callback use counter assumes only one thread at a time
does Rx/Tx burst on a queue. That is not true for Tx with
RTE_ETH_TX_OFFLOAD_MT_LOCKFREE. With two threads in the callback
the counter can be even while the program is still running, so
unload can destroy a BPF program that is in use. Without
enable_stdatomic the increment is not atomic and an update can be
lost, so unload can also wait forever.

Replace the odd/even counter with an atomic count of the threads
inside the callback. The datapath increments it on entry and
decrements it with release on exit. Unload clears cb and waits for
zero with acquire.

Make cb atomic. The datapath increment and cb load, and the unload
cb store and count load, are seq_cst; this replaces rte_smp_mb()
and rte_smp_rmb(). Install publishes cb with release so the
datapath sees bpf and jit.

On x86 the datapath is now two locked instructions per burst
instead of one.

Add a test that loads and unloads a Tx filter while worker lcores
transmit on the same MT_LOCKFREE queue of a null device.

Bugzilla ID: 2051
Fixes: a93ff62a8938 ("bpf: introduce basic Rx/Tx filters")
Cc: stable@dpdk.org

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 app/test/test_bpf.c      | 148 +++++++++++++++++++++++++++++++-
 lib/bpf/bpf_pkt.c        | 177 +++++++++++++++++----------------------
 lib/bpf/rte_bpf_ethdev.h |   2 +
 3 files changed, 225 insertions(+), 102 deletions(-)

diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c
index 3205afaa63..f21e906c6f 100644
--- a/app/test/test_bpf.c
+++ b/app/test/test_bpf.c
@@ -4329,6 +4329,10 @@ test_bpf_elf_load(load_elf_image_t load_elf_image)
 #include <rte_ethdev.h>
 #include <rte_bpf_ethdev.h>
 #include <rte_bus_vdev.h>
+#include <rte_cycles.h>
+#include <rte_launch.h>
+#include <rte_lcore.h>
+#include <rte_pause.h>
 
 #include "test_bpf_filter.h"
 
@@ -4336,7 +4340,8 @@ test_bpf_elf_load(load_elf_image_t load_elf_image)
 #define BPF_TEST_POOLSIZE	256 /* at least 2x burst */
 #define BPF_TEST_PKT_LEN	64 /* Ether + IP + TCP */
 
-static int null_vdev_setup(const char *name, uint16_t *port, struct rte_mempool *pool)
+static int null_vdev_setup(const char *name, uint16_t *port, struct rte_mempool *pool,
+			   uint64_t tx_offloads)
 {
 	int ret;
 
@@ -4347,7 +4352,9 @@ static int null_vdev_setup(const char *name, uint16_t *port, struct rte_mempool
 	ret = rte_eth_dev_get_port_by_name(name, port);
 	TEST_ASSERT(ret == 0, "failed to get port id for %s: %d", name, ret);
 
-	struct rte_eth_conf conf = { };
+	struct rte_eth_conf conf = {
+		.txmode.offloads = tx_offloads,
+	};
 	ret = rte_eth_dev_configure(*port, 1, 1, &conf);
 	TEST_ASSERT(ret == 0, "failed to configure port %u: %d", *port, ret);
 
@@ -4488,7 +4495,7 @@ test_bpf_elf_tx_load(load_elf_image_t load_elf_image)
 					  SOCKET_ID_ANY);
 	TEST_ASSERT(mb_pool != NULL, "failed to create mempool");
 
-	ret = null_vdev_setup(null_dev, &port, mb_pool);
+	ret = null_vdev_setup(null_dev, &port, mb_pool, 0);
 	if (ret != 0)
 		goto fail;
 
@@ -4577,7 +4584,7 @@ test_bpf_elf_rx_load(load_elf_image_t load_elf_image)
 					  SOCKET_ID_ANY);
 	TEST_ASSERT(pool != NULL, "failed to create mempool");
 
-	ret = null_vdev_setup(null_dev, &port, pool);
+	ret = null_vdev_setup(null_dev, &port, pool, 0);
 	if (ret != 0)
 		goto fail;
 
@@ -4618,6 +4625,136 @@ test_bpf_elf_rx_load(load_elf_image_t load_elf_image)
 }
 
 
+#define BPF_MT_BURST		32
+#define BPF_MT_POOLSIZE		4096
+#define BPF_MT_ITERATIONS	1000
+
+static RTE_ATOMIC(bool) bpf_mt_stop;
+
+struct bpf_mt_arg {
+	uint16_t port;
+	struct rte_mempool *pool;
+};
+
+/* Transmit on the same queue as the other workers until told to stop. */
+static int
+bpf_mt_tx_worker(void *arg)
+{
+	const struct bpf_mt_arg *mt = arg;
+	struct rte_mbuf *pkts[BPF_MT_BURST];
+
+	while (!rte_atomic_load_explicit(&bpf_mt_stop, rte_memory_order_relaxed)) {
+		if (rte_pktmbuf_alloc_bulk(mt->pool, pkts, BPF_MT_BURST) != 0) {
+			rte_pause();
+			continue;
+		}
+		setup_mbufs(pkts, BPF_MT_BURST);
+
+		uint16_t sent = rte_eth_tx_burst(mt->port, 0, pkts, BPF_MT_BURST);
+		rte_pktmbuf_free_bulk(pkts + sent, BPF_MT_BURST - sent);
+	}
+	return 0;
+}
+
+/*
+ * Load and unload a Tx filter while several lcores do tx burst
+ * on the same MT_LOCKFREE queue.
+ */
+static int
+test_bpf_elf_tx_mt_lockfree(load_elf_image_t load_elf_image)
+{
+	static const struct rte_bpf_arg prog_arg = {
+		.type = RTE_BPF_ARG_PTR,
+		.size = sizeof(struct dummy_net),
+	};
+	static const char null_dev[] = "net_null_bpf_mt";
+	struct bpf_mt_arg mt = { .port = UINT16_MAX };
+	struct rte_eth_dev_info dev_info;
+	unsigned int lcore, workers = 0;
+	int ret;
+
+	RTE_LCORE_FOREACH_WORKER(lcore)
+		workers++;
+	if (workers < 2) {
+		printf("%s: need at least 2 worker lcores, skipping\n", __func__);
+		return TEST_SKIPPED;
+	}
+
+	mt.pool = rte_pktmbuf_pool_create("bpf_mt_test_pool", BPF_MT_POOLSIZE,
+					  0, 0, RTE_MBUF_DEFAULT_BUF_SIZE,
+					  SOCKET_ID_ANY);
+	TEST_ASSERT_NOT_NULL(mt.pool, "failed to create mempool");
+
+	ret = null_vdev_setup(null_dev, &mt.port, mt.pool, RTE_ETH_TX_OFFLOAD_MT_LOCKFREE);
+	if (ret != 0)
+		goto fail;
+
+	ret = rte_eth_dev_info_get(mt.port, &dev_info);
+	if (ret != 0)
+		goto fail;
+	if (!(dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_MT_LOCKFREE)) {
+		ret = TEST_SKIPPED;
+		goto fail;
+	}
+
+	rte_atomic_store_explicit(&bpf_mt_stop, false, rte_memory_order_relaxed);
+	RTE_LCORE_FOREACH_WORKER(lcore)
+		rte_eal_remote_launch(bpf_mt_tx_worker, &mt, lcore);
+
+	for (unsigned int i = 0; i < BPF_MT_ITERATIONS; i++) {
+		uint32_t flags = (i & 1) ? RTE_BPF_ETH_F_JIT : 0;
+		struct rte_bpf *bpf;
+		struct rte_bpf_jit jit;
+
+		bpf = load_elf_image(app_test_bpf_filter_o, app_test_bpf_filter_o_len,
+				     "filter", NULL, 0, &prog_arg);
+		if (bpf == NULL) {
+			printf("%s@%d: failed to load BPF filter\n", __func__, __LINE__);
+			ret = TEST_FAILED;
+			break;
+		}
+
+		rte_bpf_get_jit(bpf, &jit);
+		if (jit.func == NULL)
+			flags = 0;
+
+		ret = rte_bpf_eth_tx_install(mt.port, 0, bpf, flags);
+		if (ret != 0) {
+			printf("%s@%d: install failed: %d\n", __func__, __LINE__, ret);
+			rte_bpf_destroy(bpf);
+			ret = TEST_FAILED;
+			break;
+		}
+
+		rte_delay_us(10);
+		rte_bpf_eth_tx_unload(mt.port, 0);
+	}
+
+	rte_atomic_store_explicit(&bpf_mt_stop, true, rte_memory_order_relaxed);
+	rte_eal_mp_wait_lcore();
+
+	if (ret == 0) {
+		unsigned int avail = rte_mempool_avail_count(mt.pool);
+
+		if (avail != BPF_MT_POOLSIZE) {
+			printf("%s: mempool available %u != %u\n",
+			       __func__, avail, BPF_MT_POOLSIZE);
+			ret = TEST_FAILED;
+		}
+	}
+
+fail:
+	if (mt.port != UINT16_MAX) {
+		rte_eth_dev_stop(mt.port);
+		rte_vdev_uninit(null_dev);
+	}
+	rte_mempool_free(mt.pool);
+
+	if (ret == TEST_SKIPPED)
+		return TEST_SKIPPED;
+	return ret == 0 ? TEST_SUCCESS : TEST_FAILED;
+}
+
 static int
 test_bpf_elf(void)
 {
@@ -4637,6 +4774,9 @@ test_bpf_elf(void)
 			ret = test_bpf_elf_rx_load(elf_image_loaders[li]);
 	}
 
+	if (ret == TEST_SUCCESS)
+		ret = test_bpf_elf_tx_mt_lockfree(load_elf_image_direct);
+
 	return ret;
 }
 
diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index f072fdaaed..e9fe4aeb5a 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -26,8 +26,8 @@
 
 struct __rte_cache_aligned bpf_eth_cbi {
 	/* used by both data & control path */
-	RTE_ATOMIC(uint32_t) use;    /*usage counter */
-	const struct rte_eth_rxtx_callback *cb;  /* callback handle */
+	RTE_ATOMIC(uint32_t) use;    /* number of datapath threads in callback */
+	RTE_ATOMIC(const struct rte_eth_rxtx_callback *) cb;  /* callback handle */
 	struct rte_bpf *bpf;
 	struct rte_bpf_jit jit;
 	/* used by control path only */
@@ -36,12 +36,6 @@ struct __rte_cache_aligned bpf_eth_cbi {
 	uint16_t queue;
 };
 
-/*
- * Odd number means that callback is used by datapath.
- * Even number means that callback is not used by datapath.
- */
-#define BPF_ETH_CBI_INUSE  1
-
 /*
  * List to manage RX/TX installed callbacks.
  */
@@ -75,45 +69,50 @@ static struct bpf_eth_cbh tx_cbh = {
 };
 
 /*
- * Marks given callback as used by datapath.
+ * Removing an rx/tx callback is done in two steps (similar to RCU).
+ * The callback is removed from the ethdev queue so later bursts do not
+ * call it, but a burst already in progress may still be running it.
+ *
+ * The use counter is the number of threads inside the callback.
+ * There can be more than one on a Tx queue with MT_LOCKFREE.
+ *
+ * Datapath increments use then loads cb; unload stores cb then loads use.
+ * These are seq_cst, so either the datapath sees cb == NULL and skips
+ * the program, or unload sees use != 0 and waits for zero. The release
+ * decrement pairs with the acquire wait, so the program is destroyed
+ * only after its last use.
+ */
+
+/*
+ * Enter the callback. Returns true if the BPF program may be run.
  */
-static __rte_always_inline void
-bpf_eth_cbi_inuse(struct bpf_eth_cbi *cbi)
+static inline bool
+bpf_eth_cbi_enter(struct bpf_eth_cbi *cbi)
 {
-	cbi->use++;
-	/* make sure no store/load reordering could happen */
-	rte_smp_mb();
+	rte_atomic_fetch_add_explicit(&cbi->use, 1, rte_memory_order_seq_cst);
+	return rte_atomic_load_explicit(&cbi->cb, rte_memory_order_seq_cst) != NULL;
 }
 
 /*
- * Marks given callback list as not used by datapath.
+ * Leave the callback.
  */
-static __rte_always_inline void
-bpf_eth_cbi_unuse(struct bpf_eth_cbi *cbi)
+static inline void
+bpf_eth_cbi_exit(struct bpf_eth_cbi *cbi)
 {
-	/* make sure all previous loads are completed */
-	rte_smp_rmb();
-	cbi->use++;
+	rte_atomic_fetch_sub_explicit(&cbi->use, 1, rte_memory_order_release);
 }
 
 /*
  * Waits till datapath finished using given callback.
+ * Must be called after cb is cleared.
  */
 static void
 bpf_eth_cbi_wait(const struct bpf_eth_cbi *cbi)
 {
-	uint32_t puse;
+	__rte_atomic uint32_t *use = (__rte_atomic uint32_t *)(uintptr_t)&cbi->use;
 
-	/* make sure all previous loads and stores are completed */
-	rte_smp_mb();
-
-	puse = cbi->use;
-
-	/* in use, busy wait till current RX/TX iteration is finished */
-	if ((puse & BPF_ETH_CBI_INUSE) != 0) {
-		RTE_WAIT_UNTIL_MASKED((__rte_atomic uint32_t *)(uintptr_t)&cbi->use,
-			UINT32_MAX, !=, puse, rte_memory_order_relaxed);
-	}
+	if (rte_atomic_load_explicit(use, rte_memory_order_seq_cst) != 0)
+		RTE_WAIT_UNTIL_MASKED(use, UINT32_MAX, ==, 0, rte_memory_order_acquire);
 }
 
 static void
@@ -262,16 +261,12 @@ bpf_rx_callback_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	__rte_unused uint16_t max_pkts, void *user_param)
 {
 	struct bpf_eth_cbi *cbi;
-	uint16_t rc;
 
 	cbi = user_param;
-
-	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
-		pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 1) :
-		nb_pkts;
-	bpf_eth_cbi_unuse(cbi);
-	return rc;
+	if (bpf_eth_cbi_enter(cbi))
+		nb_pkts = pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 1);
+	bpf_eth_cbi_exit(cbi);
+	return nb_pkts;
 }
 
 static uint16_t
@@ -280,15 +275,12 @@ bpf_rx_callback_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	__rte_unused uint16_t max_pkts, void *user_param)
 {
 	struct bpf_eth_cbi *cbi;
-	uint16_t rc;
 
 	cbi = user_param;
-	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
-		pkt_filter_jit(&cbi->jit, pkt, nb_pkts, 1) :
-		nb_pkts;
-	bpf_eth_cbi_unuse(cbi);
-	return rc;
+	if (bpf_eth_cbi_enter(cbi))
+		nb_pkts = pkt_filter_jit(&cbi->jit, pkt, nb_pkts, 1);
+	bpf_eth_cbi_exit(cbi);
+	return nb_pkts;
 }
 
 static uint16_t
@@ -296,15 +288,12 @@ bpf_tx_callback_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	struct rte_mbuf *pkt[], uint16_t nb_pkts, void *user_param)
 {
 	struct bpf_eth_cbi *cbi;
-	uint16_t rc;
 
 	cbi = user_param;
-	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
-		pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 0) :
-		nb_pkts;
-	bpf_eth_cbi_unuse(cbi);
-	return rc;
+	if (bpf_eth_cbi_enter(cbi))
+		nb_pkts = pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 0);
+	bpf_eth_cbi_exit(cbi);
+	return nb_pkts;
 }
 
 static uint16_t
@@ -312,15 +301,12 @@ bpf_tx_callback_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	struct rte_mbuf *pkt[], uint16_t nb_pkts, void *user_param)
 {
 	struct bpf_eth_cbi *cbi;
-	uint16_t rc;
 
 	cbi = user_param;
-	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
-		pkt_filter_jit(&cbi->jit, pkt, nb_pkts, 0) :
-		nb_pkts;
-	bpf_eth_cbi_unuse(cbi);
-	return rc;
+	if (bpf_eth_cbi_enter(cbi))
+		nb_pkts = pkt_filter_jit(&cbi->jit, pkt, nb_pkts, 0);
+	bpf_eth_cbi_exit(cbi);
+	return nb_pkts;
 }
 
 /*
@@ -333,15 +319,12 @@ bpf_rx_callback_mb_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	__rte_unused uint16_t max_pkts, void *user_param)
 {
 	struct bpf_eth_cbi *cbi;
-	uint16_t rc;
 
 	cbi = user_param;
-	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
-		pkt_filter_mb_vm(cbi->bpf, pkt, nb_pkts, 1) :
-		nb_pkts;
-	bpf_eth_cbi_unuse(cbi);
-	return rc;
+	if (bpf_eth_cbi_enter(cbi))
+		nb_pkts = pkt_filter_mb_vm(cbi->bpf, pkt, nb_pkts, 1);
+	bpf_eth_cbi_exit(cbi);
+	return nb_pkts;
 }
 
 static uint16_t
@@ -350,15 +333,12 @@ bpf_rx_callback_mb_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	__rte_unused uint16_t max_pkts, void *user_param)
 {
 	struct bpf_eth_cbi *cbi;
-	uint16_t rc;
 
 	cbi = user_param;
-	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
-		pkt_filter_mb_jit(&cbi->jit, pkt, nb_pkts, 1) :
-		nb_pkts;
-	bpf_eth_cbi_unuse(cbi);
-	return rc;
+	if (bpf_eth_cbi_enter(cbi))
+		nb_pkts = pkt_filter_mb_jit(&cbi->jit, pkt, nb_pkts, 1);
+	bpf_eth_cbi_exit(cbi);
+	return nb_pkts;
 }
 
 static uint16_t
@@ -366,15 +346,12 @@ bpf_tx_callback_mb_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	struct rte_mbuf *pkt[], uint16_t nb_pkts, void *user_param)
 {
 	struct bpf_eth_cbi *cbi;
-	uint16_t rc;
 
 	cbi = user_param;
-	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
-		pkt_filter_mb_vm(cbi->bpf, pkt, nb_pkts, 0) :
-		nb_pkts;
-	bpf_eth_cbi_unuse(cbi);
-	return rc;
+	if (bpf_eth_cbi_enter(cbi))
+		nb_pkts = pkt_filter_mb_vm(cbi->bpf, pkt, nb_pkts, 0);
+	bpf_eth_cbi_exit(cbi);
+	return nb_pkts;
 }
 
 static uint16_t
@@ -382,15 +359,12 @@ bpf_tx_callback_mb_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	struct rte_mbuf *pkt[], uint16_t nb_pkts, void *user_param)
 {
 	struct bpf_eth_cbi *cbi;
-	uint16_t rc;
 
 	cbi = user_param;
-	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
-		pkt_filter_mb_jit(&cbi->jit, pkt, nb_pkts, 0) :
-		nb_pkts;
-	bpf_eth_cbi_unuse(cbi);
-	return rc;
+	if (bpf_eth_cbi_enter(cbi))
+		nb_pkts = pkt_filter_mb_jit(&cbi->jit, pkt, nb_pkts, 0);
+	bpf_eth_cbi_exit(cbi);
+	return nb_pkts;
 }
 
 static rte_rx_callback_fn
@@ -438,8 +412,7 @@ static void
 bpf_eth_cbi_unload(struct bpf_eth_cbi *bc)
 {
 	/* mark this cbi as empty */
-	bc->cb = NULL;
-	rte_smp_mb();
+	rte_atomic_store_explicit(&bc->cb, NULL, rte_memory_order_seq_cst);
 
 	/* make sure datapath doesn't use bpf anymore, then destroy bpf */
 	bpf_eth_cbi_wait(bc);
@@ -450,16 +423,21 @@ bpf_eth_cbi_unload(struct bpf_eth_cbi *bc)
 static void
 bpf_eth_unload(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue)
 {
+	const struct rte_eth_rxtx_callback *cb;
 	struct bpf_eth_cbi *bc;
 
 	bc = bpf_eth_cbh_find(cbh, port, queue);
-	if (bc == NULL || bc->cb == NULL)
+	if (bc == NULL)
+		return;
+
+	cb = rte_atomic_load_explicit(&bc->cb, rte_memory_order_relaxed);
+	if (cb == NULL)
 		return;
 
 	if (cbh->type == BPF_ETH_RX)
-		rte_eth_remove_rx_callback(port, queue, bc->cb);
+		rte_eth_remove_rx_callback(port, queue, cb);
 	else
-		rte_eth_remove_tx_callback(port, queue, bc->cb);
+		rte_eth_remove_tx_callback(port, queue, cb);
 
 	bpf_eth_cbi_unload(bc);
 }
@@ -494,6 +472,7 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 	struct rte_bpf *bpf, uint32_t flags)
 {
 	int32_t rc;
+	const struct rte_eth_rxtx_callback *cb;
 	struct bpf_eth_cbi *bc;
 	rte_rx_callback_fn frx;
 	rte_tx_callback_fn ftx;
@@ -534,18 +513,20 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 		return -ENOMEM;
 
 	/* remove old one, if any */
-	if (bc->cb != NULL)
-		bpf_eth_unload(cbh, port, queue);
+	bpf_eth_unload(cbh, port, queue);
 
 	bc->bpf = bpf;
 	bc->jit = jit;
 
 	if (cbh->type == BPF_ETH_RX)
-		bc->cb = rte_eth_add_rx_callback(port, queue, frx, bc);
+		cb = rte_eth_add_rx_callback(port, queue, frx, bc);
 	else
-		bc->cb = rte_eth_add_tx_callback(port, queue, ftx, bc);
+		cb = rte_eth_add_tx_callback(port, queue, ftx, bc);
+
+	/* publish bpf and jit to the datapath */
+	rte_atomic_store_explicit(&bc->cb, cb, rte_memory_order_release);
 
-	if (bc->cb == NULL) {
+	if (cb == NULL) {
 		rc = -rte_errno;
 		bpf_eth_cbi_cleanup(bc);
 	} else
diff --git a/lib/bpf/rte_bpf_ethdev.h b/lib/bpf/rte_bpf_ethdev.h
index 686fda8d94..9f14a624d4 100644
--- a/lib/bpf/rte_bpf_ethdev.h
+++ b/lib/bpf/rte_bpf_ethdev.h
@@ -14,6 +14,8 @@
  *   same port/queue from different threads in parallel.
  * - though it allows load/unload at runtime
  *   (while RX/TX is ongoing on given port/queue).
+ * - TX burst on the same port/queue may be called from more than
+ *   one thread when RTE_ETH_TX_OFFLOAD_MT_LOCKFREE is enabled.
  * - allows only one BPF program per port/queue,
  * i.e. new load will replace previously loaded for that port/queue BPF program.
  * Filter behaviour - if BPF program returns zero value for a given packet,
-- 
2.53.0

Re: [PATCH 9/9] bpf: make callback handle atomic with relaxed access

[Marat Khalili]

> Since inuse and cb are inter-related would be easier to squash the mb
> conversion and the bool and the atomic cb into one patch.

Apologies if number of patches seems excessive, it really is a matter of 
style and taste. Technically all of these patches are all solving 
different problems and can be applied (or not applied) independently. If 
the maintainer decides to squash them so be it, fine for me.

Thanks for the reviews btw.

[PATCH v2 0/9] bpf: clean up and document ethdev callback synchronization

[Marat Khalili]

The DPDK BPF library provides mechanisms to attach BPF programs to
ethdev Rx/Tx callbacks (rte_bpf_eth_rx_elf_load and
rte_bpf_eth_tx_elf_load). Safe unloading of these callbacks relies on
a synchronization protocol between the control plane and datapath
threads, built upon the assumption that each port/queue callback is
serviced by at most one datapath thread.

This patch series improves the documentation, safety, and implementation
of the callback synchronization mechanism:

1. Documentation:
   - Add an ASCII sequence diagram and comprehensive documentation in
     bpf_pkt.c explaining the synchronization protocol between the
     control plane and datapath during callback unload.
   - Document the single-thread-per-queue datapath requirement in
     public API headers.

2. Safety and Validation:
   - Enforce the single-threaded datapath assumption by rejecting Tx
     callback installation when the device advertises
     RTE_ETH_TX_OFFLOAD_MT_LOCKFREE.
   - Add a new flag (RTE_BPF_ETH_F_SKIP_MT_CHECK) allowing applications
     to override this check when application architecture guarantees
     thread exclusivity.
   - Add automated unit test test_bpf_elf_tx_mt_lockfree.

3. Synchronization Cleanups:
   - Remove a redundant control plane memory barrier (rte_smp_mb) on
     callback unload.
   - Simplify datapath inuse tracking (cbi->use) from an even/odd
     counter to a boolean flag (true and false), simplifying
     bpf_eth_cbi_wait().
   - Switch cbi->use stores and control plane wait loop loads to
     explicit rte_atomic_store_explicit() and rte_atomic_load_explicit()
     with relaxed memory order, avoiding implicit seq_cst operations
     in C11.
   - Make cbi->cb atomic with relaxed atomic load/store operations,
     eliminating C11 data-race undefined behavior while compiling to
     identical machine code, and making it self-evident that cb is an
     integral part of the synchronization mechanism on par with use.

Patches 4 and 7 were originally proposed by Stephen Hemminger as part
of a larger series replacing deprecated SMP barriers with C11 fences,
and are carried here with his authorship.

Note that a full rework of the synchronization protocol (e.g. replacing
standalone memory barriers entirely with acquire/release semantics
across variables) was intentionally avoided to keep each commit simple,
localized, and non-controversial. More substantial architectural
changes, such as that or extending the library to support concurrent
multi-threaded dataplane execution, can be cleanly built on top of this
work in the future.

v2:
- Reordered the series so that the patch adding
  RTE_BPF_ETH_F_SKIP_MT_CHECK is last, allowing it to be easily dropped
  if no agreement is reached.
- Added release note in release_26_11.rst documenting the MT_LOCKFREE
  Tx callback rejection and the RTE_BPF_ETH_F_SKIP_MT_CHECK override
  flag.
- Credited Stephen Hemminger for earlier authorship of patches 4 and 7
  ("remove redundant memory barrier on unload" and "use relaxed accesses
  for callback inuse tracking").
- Changed callback usage tracking variable `use` to type `bool` with
  `true`/`false` values instead of uint32_t with 1/0.
- Replaced RTE_WAIT_UNTIL_MASKED with an explicit pause loop in
  bpf_eth_cbi_wait(), eliminating macro and type casts.

Marat Khalili (9):
  bpf: document packet callback unload sequence
  bpf: document ethdev callback threading limitations
  bpf: reject Tx callback installation with MT_LOCKFREE
  bpf: remove redundant memory barrier on unload
  bpf: simplify callback inuse tracking to boolean flag
  bpf: reformat packet callback unload diagram
  bpf: use relaxed accesses for callback inuse tracking
  bpf: make callback handle atomic with relaxed access
  bpf: add callback flag to skip multi-thread check

 app/test/test_bpf.c                    | 123 +++++++++++++++++
 doc/guides/rel_notes/release_26_11.rst |   5 +
 lib/bpf/bpf_pkt.c                      | 175 +++++++++++++++++++------
 lib/bpf/rte_bpf_ethdev.h               |  11 +-
 4 files changed, 275 insertions(+), 39 deletions(-)

-- 
2.43.0

[PATCH v2 1/9] bpf: document packet callback unload sequence

[Marat Khalili]

Add a detailed comment with the sequence diagram to bpf_pkt.c describing
the current de-facto synchronization protocol on BPF program unload
between the control plane and the data plane.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 64 insertions(+), 1 deletion(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index f072fdaaed..4223ef624e 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -21,9 +21,72 @@
 #include "bpf_impl.h"
 
 /*
- * information about installed BPF rx/tx callback
+ * CP-DP synchronization mechanism
+ *
+ * After the callback is installed to a port/queue, removing it safely is a non-trivial task.
+ * The `rte_bpf` object can only be destroyed while the callback is not being executed,
+ * while the number of synchronization calls in the packet callback must be minimized.
+ *
+ * APIs `rte_eth_remove_rx_callback` and `rte_eth_remove_tx_callback` remove the callback from
+ * the callback list and upon return guarantee that it will not be called again,
+ * but not that any existing invocation of the callback has completed.
+ *
+ * Current synchronization mechanism relies on the following per-port/queue entities:
+ * 1. List of installed port/queue callbacks ("cbs" on the diagram);
+ * 2. Member `cb` of struct `bpf_eth_cbi`;
+ * 3. Member `use` of struct `bpf_eth_cbi`;
+ *
+ * BPF program is only executed if the first two are set, and increments `use` at the start and
+ * end of the callback execution so that in the safe state its value is even.
+ * This relies on only one thread calling the per-port/queue callback at a time.
+ *
+ * Unload processing starts with clearing the first two entities,
+ * then waits for `use` to become even before destroying the BPF program.
+ * The whole operation is protected by a control-plane-only spinlock (not shown on the diagram).
+ *
+ * Typical sequence of synchronization actions between CP and DP on unload is shown below:
+ * ```
+ *   +----+                +-----+         +-----+        +----+         +-----+          +----+
+ *   | CP |                | cbs |         | use |        | cb |         | bpf |          | DP |
+ *   +----+                +-----+         +-----+        +----+         +-----+          +----+
+ *     |                      |               |              |              |                |
+ *     |                      |<---------------------- fetch callback -----------------------|
+ *     |                      |               |              |              |                |
+ *     |                      |               |<------------- increment to odd --------------|
+ *     |                      |               |              |              |                |
+ *     |-- remove callback -->|               |              |              |       rte_smp_mb()
+ *     |                      |               |              |              |                |
+ *     |                      |               |              |<------ fetch non-NULL --------|
+ *     |                      |               |              |              |                |
+ *     |------------------- store NULL --------------------->|              |                |
+ *     |                      |               |              |              |                |
+ *   rte_smp_mb()             |               |              |              |                |
+ *     |                      |               |              |              |                |
+ *   rte_smp_mb()             |               |              |              |                |
+ *     |                      |               |              |              |                |
+ *     +-if use is odd        |               |              |              | if cb not NULL-+
+ *     | |                    |               |              |              |              | |
+ *     | +------------ fetch odd ------------>|              |              |<-- execute --+ |
+ *     | |                    |               |              |              |  BPF program | |
+ *     | +-while use is same  |               |              |              |              +-+
+ *     | | |                  |               |              |              |                |
+ *     | | +---------- fetch odd ------------>|              |              |      rte_smp_rmb()
+ *     | | |                  |               |              |              |                |
+ *     +-+-+                  |               |<------------- increment to even -------------|
+ *     |                      |               |              |              |                |
+ *     |------------- fetch even ------------>|              |              |                |
+ *     |                      |               |              |              |                |
+ *     |----------------- destroy BPF program and cleanup ----------------->|                |
+ *     |                      |               |              |              |                |
+ *   +----+                +-----+         +-----+        +----+         +-----+          +----+
+ *   | CP |                | cbs |         | use |        | cb |         | bpf |          | DP |
+ *   +----+                +-----+         +-----+        +----+         +-----+          +----+
+ * ```
  */
 
+/*
+ * information about installed BPF rx/tx callback
+ */
 struct __rte_cache_aligned bpf_eth_cbi {
 	/* used by both data & control path */
 	RTE_ATOMIC(uint32_t) use;    /*usage counter */
-- 
2.43.0

[PATCH v2 2/9] bpf: document ethdev callback threading limitations

[Marat Khalili]

Document the data plane limitation in the file docstring of
rte_bpf_ethdev.h: packet callback execution expects only one
thread per port/queue pair at a time, making concurrent bursts
(such as with RTE_ETH_TX_OFFLOAD_MT_LOCKFREE, shared RX queues,
or multi-threaded software PMDs) unsupported.

Rephrase the existing control plane multi-threading limitation
to explicitly distinguish it from the data plane requirement.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/rte_bpf_ethdev.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/lib/bpf/rte_bpf_ethdev.h b/lib/bpf/rte_bpf_ethdev.h
index 686fda8d94..cf3b4048f0 100644
--- a/lib/bpf/rte_bpf_ethdev.h
+++ b/lib/bpf/rte_bpf_ethdev.h
@@ -10,8 +10,12 @@
  *
  * API to install BPF filter as RX/TX callbacks for eth devices.
  * Note that right now:
- * - it is not MT safe, i.e. it is not allowed to do load/unload for the
- *   same port/queue from different threads in parallel.
+ * - in the data plane, it expects only one thread per port/queue pair at a time.
+ *   Calling RX/TX burst functions concurrently on the same queue (e.g. when using
+ *   RTE_ETH_TX_OFFLOAD_MT_LOCKFREE, shared RX queues, or multi-threaded
+ *   software PMDs) is not supported.
+ * - in the control plane, it is not MT safe, i.e. it is not allowed to do
+ *   load/unload for the same port/queue from different threads in parallel.
  * - though it allows load/unload at runtime
  *   (while RX/TX is ongoing on given port/queue).
  * - allows only one BPF program per port/queue,
-- 
2.43.0

[PATCH v2 3/9] bpf: reject Tx callback installation with MT_LOCKFREE

[Marat Khalili]

The BPF ethdev packet callback synchronization mechanism expects only
one dataplane thread per port/queue pair at a time. When
RTE_ETH_TX_OFFLOAD_MT_LOCKFREE is enabled, multiple threads can invoke
rte_eth_tx_burst() concurrently on the same queue without locks. This
breaks the callback parity counter and can lead to use-after-free or
deadlocks during BPF program unload.

Reject BPF Tx callback installation with -ENOTSUP when
RTE_ETH_TX_OFFLOAD_MT_LOCKFREE is enabled on the port or queue.
Add a unit test in test_bpf verifying this rejection.

Bugzilla ID: 2051
Fixes: a93ff62a8938 ("bpf: introduce basic Rx/Tx filters")
Cc: stable@dpdk.org

Reported-by: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 app/test/test_bpf.c | 103 ++++++++++++++++++++++++++++++++++++++++++++
 lib/bpf/bpf_pkt.c   |  46 ++++++++++++++++++++
 2 files changed, 149 insertions(+)

diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c
index 3205afaa63..02523984b5 100644
--- a/app/test/test_bpf.c
+++ b/app/test/test_bpf.c
@@ -4617,6 +4617,107 @@ test_bpf_elf_rx_load(load_elf_image_t load_elf_image)
 	return ret == 0 ? TEST_SUCCESS : TEST_FAILED;
 }
 
+static int
+null_vdev_mt_setup(const char *name, uint16_t *port, struct rte_mempool *pool)
+{
+	struct rte_eth_conf conf = {
+		.txmode = {
+			.offloads = RTE_ETH_TX_OFFLOAD_MT_LOCKFREE,
+		},
+	};
+	struct rte_eth_txconf txconf = { };
+	struct rte_eth_rxconf rxconf = { };
+	int ret;
+
+	ret = rte_vdev_init(name, NULL);
+	TEST_ASSERT(ret == 0, "rte_vdev_init(%s) failed: %d", name, ret);
+
+	ret = rte_eth_dev_get_port_by_name(name, port);
+	TEST_ASSERT(ret == 0, "failed to get port id for %s: %d", name, ret);
+
+	ret = rte_eth_dev_configure(*port, 1, 1, &conf);
+	TEST_ASSERT(ret == 0, "failed to configure port %u: %d", *port, ret);
+
+	ret = rte_eth_tx_queue_setup(*port, 0, BPF_TEST_BURST, SOCKET_ID_ANY, &txconf);
+	TEST_ASSERT(ret == 0, "failed to setup tx queue port %u: %d", *port, ret);
+
+	ret = rte_eth_rx_queue_setup(*port, 0, BPF_TEST_BURST, SOCKET_ID_ANY,
+				     &rxconf, pool);
+	TEST_ASSERT(ret == 0, "failed to setup rx queue port %u: %d", *port, ret);
+
+	ret = rte_eth_dev_start(*port);
+	TEST_ASSERT(ret == 0, "failed to start port %u: %d", *port, ret);
+
+	return 0;
+}
+
+/* Test that installing a Tx filter is rejected when MT_LOCKFREE is enabled */
+static int
+test_bpf_elf_tx_mt_lockfree(load_elf_image_t load_elf_image)
+{
+	static const struct rte_bpf_arg prog_arg = {
+		.type = RTE_BPF_ARG_PTR,
+		.size = sizeof(struct dummy_net),
+	};
+	static const char null_dev[] = "net_null_bpf_mt";
+	struct rte_mempool *mb_pool = NULL;
+	struct rte_bpf *bpf = NULL;
+	uint16_t port = UINT16_MAX;
+	int ret;
+
+	mb_pool = rte_pktmbuf_pool_create("bpf_mt_test_pool", BPF_TEST_POOLSIZE,
+					  0, 0, RTE_MBUF_DEFAULT_BUF_SIZE,
+					  SOCKET_ID_ANY);
+	TEST_ASSERT_NOT_NULL(mb_pool, "failed to create mempool");
+
+	ret = null_vdev_mt_setup(null_dev, &port, mb_pool);
+	if (ret != 0)
+		goto fail;
+
+	bpf = load_elf_image(app_test_bpf_filter_o, app_test_bpf_filter_o_len, "filter",
+			     NULL, 0, &prog_arg);
+	if (bpf == NULL) {
+		printf("%s@%d: failed to load BPF filter image\n",
+		       __func__, __LINE__);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+
+	/* Tx installation must fail with -ENOTSUP due to MT_LOCKFREE */
+	ret = rte_bpf_eth_tx_install(port, 0, bpf, 0);
+	if (ret != -ENOTSUP) {
+		printf("%s@%d: expected rte_bpf_eth_tx_install to return -ENOTSUP, got %d\n",
+		       __func__, __LINE__, ret);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+
+	/* Rx installation on the same port must still succeed */
+	ret = rte_bpf_eth_rx_install(port, 0, bpf, 0);
+	if (ret != 0) {
+		printf("%s@%d: expected rte_bpf_eth_rx_install to succeed, got %d\n",
+		       __func__, __LINE__, ret);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+	rte_bpf_eth_rx_unload(port, 0);
+	/* bpf is destroyed by rte_bpf_eth_rx_unload */
+	bpf = NULL;
+
+	ret = TEST_SUCCESS;
+
+fail:
+	if (bpf != NULL)
+		rte_bpf_destroy(bpf);
+
+	if (port != UINT16_MAX) {
+		rte_eth_dev_stop(port);
+		rte_vdev_uninit(null_dev);
+	}
+
+	rte_mempool_free(mb_pool);
+	return ret;
+}
 
 static int
 test_bpf_elf(void)
@@ -4633,6 +4734,8 @@ test_bpf_elf(void)
 			ret = test_bpf_elf_load(elf_image_loaders[li]);
 		if (ret == TEST_SUCCESS)
 			ret = test_bpf_elf_tx_load(elf_image_loaders[li]);
+		if (ret == TEST_SUCCESS)
+			ret = test_bpf_elf_tx_mt_lockfree(elf_image_loaders[li]);
 		if (ret == TEST_SUCCESS)
 			ret = test_bpf_elf_rx_load(elf_image_loaders[li]);
 	}
diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index 4223ef624e..eab85f7417 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -552,6 +552,48 @@ rte_bpf_eth_tx_unload(uint16_t port, uint16_t queue)
 	rte_spinlock_unlock(&cbh->lock);
 }
 
+/*
+ * Check if the supplied port/queue pair is suitable for BPF callback execution.
+ */
+static int
+bpf_eth_check_mt(const struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue)
+{
+	struct rte_eth_txq_info qinfo;
+	struct rte_eth_conf conf;
+	int ret;
+
+	if (cbh->type == BPF_ETH_TX) {
+		ret = rte_eth_dev_conf_get(port, &conf);
+		if (ret != 0) {
+			RTE_BPF_LOG_LINE(ERR,
+				"%s(%u, %u): failed to get device configuration: %d;",
+				__func__, port, queue, ret);
+			return ret;
+		}
+
+		if ((conf.txmode.offloads & RTE_ETH_TX_OFFLOAD_MT_LOCKFREE) != 0) {
+			RTE_BPF_LOG_LINE(ERR,
+				"%s(%u, %u): MT_LOCKFREE Tx offload is enabled on port;",
+				__func__, port, queue);
+			return -ENOTSUP;
+		}
+
+		/*
+		 * Ignore errors from tx_queue_info_get: the PMD might not implement
+		 * txq_info_get (e.g. net/null), or the queue might not be set up yet.
+		 */
+		if (rte_eth_tx_queue_info_get(port, queue, &qinfo) == 0 &&
+				(qinfo.conf.offloads & RTE_ETH_TX_OFFLOAD_MT_LOCKFREE) != 0) {
+			RTE_BPF_LOG_LINE(ERR,
+				"%s(%u, %u): MT_LOCKFREE Tx offload is enabled on queue;",
+				__func__, port, queue);
+			return -ENOTSUP;
+		}
+	}
+
+	return 0;
+}
+
 static int
 bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 	struct rte_bpf *bpf, uint32_t flags)
@@ -572,6 +614,10 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 	if (bpf->prm.nb_prog_arg != 1)
 		return -EINVAL;
 
+	rc = bpf_eth_check_mt(cbh, port, queue);
+	if (rc != 0)
+		return rc;
+
 	if (cbh->type == BPF_ETH_RX)
 		frx = select_rx_callback(bpf->prm.prog_arg[0].type, flags);
 	else
-- 
2.43.0

[PATCH v2 4/9] bpf: remove redundant memory barrier on unload

[Marat Khalili]

From: Stephen Hemminger <stephen@networkplumber.org>

In bpf_eth_cbi_unload(), rte_smp_mb() was called immediately before
bpf_eth_cbi_wait(), which itself starts with rte_smp_mb().

Remove the redundant rte_smp_mb() call from bpf_eth_cbi_unload() and
update the unload sequence diagram accordingly.

Originally proposed by Stephen Hemminger as part of a series replacing
deprecated SMP barriers with C11 fences.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index eab85f7417..b1e25e4e7c 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -62,8 +62,6 @@
  *     |                      |               |              |              |                |
  *   rte_smp_mb()             |               |              |              |                |
  *     |                      |               |              |              |                |
- *   rte_smp_mb()             |               |              |              |                |
- *     |                      |               |              |              |                |
  *     +-if use is odd        |               |              |              | if cb not NULL-+
  *     | |                    |               |              |              |              | |
  *     | +------------ fetch odd ------------>|              |              |<-- execute --+ |
@@ -502,7 +500,6 @@ bpf_eth_cbi_unload(struct bpf_eth_cbi *bc)
 {
 	/* mark this cbi as empty */
 	bc->cb = NULL;
-	rte_smp_mb();
 
 	/* make sure datapath doesn't use bpf anymore, then destroy bpf */
 	bpf_eth_cbi_wait(bc);
-- 
2.43.0

[PATCH v2 5/9] bpf: simplify callback inuse tracking to boolean flag

[Marat Khalili]

The even/odd usage counter semantics in struct bpf_eth_cbi was
originally used to detect when in-flight packet callbacks complete.
However, with at most one thread running the callback per port/queue
pair and no risk of the ABA problem, a boolean flag (true for in use,
false for idle) suffices.

Change member use to bool and replace the counter increment in the
datapath with stores of true and false. Simplify bpf_eth_cbi_wait()
to a direct pause loop waiting until the flag is false, removing the
now unnecessary RTE_WAIT_UNTIL_MASKED macro and casts. Update the
documentation and synchronization sequence diagram accordingly.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 49 ++++++++++++++++++-----------------------------
 1 file changed, 19 insertions(+), 30 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index b1e25e4e7c..429feb76ad 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <errno.h>
 #include <stdint.h>
+#include <stdbool.h>
 
 #include <sys/queue.h>
 
@@ -36,12 +37,12 @@
  * 2. Member `cb` of struct `bpf_eth_cbi`;
  * 3. Member `use` of struct `bpf_eth_cbi`;
  *
- * BPF program is only executed if the first two are set, and increments `use` at the start and
- * end of the callback execution so that in the safe state its value is even.
+ * BPF program is only executed if the first two are set, and sets `use` to true at the start and
+ * to false at the end of the callback execution.
  * This relies on only one thread calling the per-port/queue callback at a time.
  *
  * Unload processing starts with clearing the first two entities,
- * then waits for `use` to become even before destroying the BPF program.
+ * then waits for `use` to become false before destroying the BPF program.
  * The whole operation is protected by a control-plane-only spinlock (not shown on the diagram).
  *
  * Typical sequence of synchronization actions between CP and DP on unload is shown below:
@@ -52,7 +53,7 @@
  *     |                      |               |              |              |                |
  *     |                      |<---------------------- fetch callback -----------------------|
  *     |                      |               |              |              |                |
- *     |                      |               |<------------- increment to odd --------------|
+ *     |                      |               |<------------ store true ---------------------|
  *     |                      |               |              |              |                |
  *     |-- remove callback -->|               |              |              |       rte_smp_mb()
  *     |                      |               |              |              |                |
@@ -62,17 +63,17 @@
  *     |                      |               |              |              |                |
  *   rte_smp_mb()             |               |              |              |                |
  *     |                      |               |              |              |                |
- *     +-if use is odd        |               |              |              | if cb not NULL-+
+ *     +-while use is true    |               |              |              | if cb not NULL-+
  *     | |                    |               |              |              |              | |
- *     | +------------ fetch odd ------------>|              |              |<-- execute --+ |
+ *     | +----------- fetch true ------------>|              |              |<-- execute --+ |
  *     | |                    |               |              |              |  BPF program | |
- *     | +-while use is same  |               |              |              |              +-+
- *     | | |                  |               |              |              |                |
- *     | | +---------- fetch odd ------------>|              |              |      rte_smp_rmb()
- *     | | |                  |               |              |              |                |
- *     +-+-+                  |               |<------------- increment to even -------------|
+ *     | |                    |               |              |              |              +-+
+ *     | |                    |               |              |              |                |
+ *     | |                    |               |              |              |      rte_smp_rmb()
+ *     | |                    |               |              |              |                |
+ *     +-+                    |               |<------------ store false --------------------|
  *     |                      |               |              |              |                |
- *     |------------- fetch even ------------>|              |              |                |
+ *     |------------ fetch false ------------>|              |              |                |
  *     |                      |               |              |              |                |
  *     |----------------- destroy BPF program and cleanup ----------------->|                |
  *     |                      |               |              |              |                |
@@ -87,7 +88,7 @@
  */
 struct __rte_cache_aligned bpf_eth_cbi {
 	/* used by both data & control path */
-	RTE_ATOMIC(uint32_t) use;    /*usage counter */
+	RTE_ATOMIC(bool) use;    /* usage flag: true when in use, false otherwise */
 	const struct rte_eth_rxtx_callback *cb;  /* callback handle */
 	struct rte_bpf *bpf;
 	struct rte_bpf_jit jit;
@@ -97,12 +98,6 @@ struct __rte_cache_aligned bpf_eth_cbi {
 	uint16_t queue;
 };
 
-/*
- * Odd number means that callback is used by datapath.
- * Even number means that callback is not used by datapath.
- */
-#define BPF_ETH_CBI_INUSE  1
-
 /*
  * List to manage RX/TX installed callbacks.
  */
@@ -141,20 +136,20 @@ static struct bpf_eth_cbh tx_cbh = {
 static __rte_always_inline void
 bpf_eth_cbi_inuse(struct bpf_eth_cbi *cbi)
 {
-	cbi->use++;
+	cbi->use = true;
 	/* make sure no store/load reordering could happen */
 	rte_smp_mb();
 }
 
 /*
- * Marks given callback list as not used by datapath.
+ * Marks given callback as not used by datapath.
  */
 static __rte_always_inline void
 bpf_eth_cbi_unuse(struct bpf_eth_cbi *cbi)
 {
 	/* make sure all previous loads are completed */
 	rte_smp_rmb();
-	cbi->use++;
+	cbi->use = false;
 }
 
 /*
@@ -163,18 +158,12 @@ bpf_eth_cbi_unuse(struct bpf_eth_cbi *cbi)
 static void
 bpf_eth_cbi_wait(const struct bpf_eth_cbi *cbi)
 {
-	uint32_t puse;
-
 	/* make sure all previous loads and stores are completed */
 	rte_smp_mb();
 
-	puse = cbi->use;
-
 	/* in use, busy wait till current RX/TX iteration is finished */
-	if ((puse & BPF_ETH_CBI_INUSE) != 0) {
-		RTE_WAIT_UNTIL_MASKED((__rte_atomic uint32_t *)(uintptr_t)&cbi->use,
-			UINT32_MAX, !=, puse, rte_memory_order_relaxed);
-	}
+	while (cbi->use)
+		rte_pause();
 }
 
 static void
-- 
2.43.0

[PATCH v2 6/9] bpf: reformat packet callback unload diagram

[Marat Khalili]

Reformat the CP-DP synchronization sequence diagram in bpf_pkt.c by
removing unnecessary vertical lines and re-centering arrow labels.

This comment-only commit is intentionally kept separate to make the
diff of the preceding semantic change more readable.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index 429feb76ad..55f5947460 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -53,25 +53,21 @@
  *     |                      |               |              |              |                |
  *     |                      |<---------------------- fetch callback -----------------------|
  *     |                      |               |              |              |                |
- *     |                      |               |<------------ store true ---------------------|
+ *     |                      |               |<---------------- store true -----------------|
  *     |                      |               |              |              |                |
  *     |-- remove callback -->|               |              |              |       rte_smp_mb()
  *     |                      |               |              |              |                |
  *     |                      |               |              |<------ fetch non-NULL --------|
  *     |                      |               |              |              |                |
- *     |------------------- store NULL --------------------->|              |                |
- *     |                      |               |              |              |                |
- *   rte_smp_mb()             |               |              |              |                |
- *     |                      |               |              |              |                |
- *     +-while use is true    |               |              |              | if cb not NULL-+
- *     | |                    |               |              |              |              | |
- *     | +----------- fetch true ------------>|              |              |<-- execute --+ |
- *     | |                    |               |              |              |  BPF program | |
- *     | |                    |               |              |              |              +-+
+ *     |------------------- store NULL --------------------->|              | if cb not NULL-+
+ *     |                      |               |              |              |              | |
+ *   rte_smp_mb()             |               |              |              |<-- execute --+ |
+ *     |                      |               |              |              |  BPF program | |
+ *     +-while use is true    |               |              |              |              +-+
  *     | |                    |               |              |              |                |
- *     | |                    |               |              |              |      rte_smp_rmb()
+ *     | +---------- fetch true ------------->|              |              |      rte_smp_rmb()
  *     | |                    |               |              |              |                |
- *     +-+                    |               |<------------ store false --------------------|
+ *     +-+                    |               |<---------------- store false ----------------|
  *     |                      |               |              |              |                |
  *     |------------ fetch false ------------>|              |              |                |
  *     |                      |               |              |              |                |
-- 
2.43.0

[PATCH v2 7/9] bpf: use relaxed accesses for callback inuse tracking

[Marat Khalili]

From: Stephen Hemminger <stephen@networkplumber.org>

Direct assignment to or reading from an atomic variable defaults to
sequentially consistent memory order in C11. Since cross-thread memory
ordering for packet callback execution and unload is already explicitly
enforced by memory barriers, sequentially consistent accesses are
unnecessary and may emit redundant barrier or locked instructions on
weakly ordered architectures.

Update bpf_eth_cbi_inuse() and bpf_eth_cbi_unuse() to use
rte_atomic_store_explicit() with rte_memory_order_relaxed, and update
bpf_eth_cbi_wait() to use rte_atomic_load_explicit() with
rte_memory_order_relaxed.

Originally proposed by Stephen Hemminger as part of a series replacing
deprecated SMP barriers with C11 fences.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index 55f5947460..c5fa85bf84 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -132,7 +132,7 @@ static struct bpf_eth_cbh tx_cbh = {
 static __rte_always_inline void
 bpf_eth_cbi_inuse(struct bpf_eth_cbi *cbi)
 {
-	cbi->use = true;
+	rte_atomic_store_explicit(&cbi->use, true, rte_memory_order_relaxed);
 	/* make sure no store/load reordering could happen */
 	rte_smp_mb();
 }
@@ -145,7 +145,7 @@ bpf_eth_cbi_unuse(struct bpf_eth_cbi *cbi)
 {
 	/* make sure all previous loads are completed */
 	rte_smp_rmb();
-	cbi->use = false;
+	rte_atomic_store_explicit(&cbi->use, false, rte_memory_order_relaxed);
 }
 
 /*
@@ -158,7 +158,7 @@ bpf_eth_cbi_wait(const struct bpf_eth_cbi *cbi)
 	rte_smp_mb();
 
 	/* in use, busy wait till current RX/TX iteration is finished */
-	while (cbi->use)
+	while (rte_atomic_load_explicit(&cbi->use, rte_memory_order_relaxed))
 		rte_pause();
 }
 
-- 
2.43.0

[PATCH v2 8/9] bpf: make callback handle atomic with relaxed access

[Marat Khalili]

The callback handle member cb of struct bpf_eth_cbi is concurrently
cleared by the control plane on unload and read by the datapath to
decide whether to execute the BPF filter program. Without atomic typing,
this concurrent access constitutes a data race in the C11 memory model,
even though hardware pointer accesses are single-copy atomic and untorn.

Change cb to RTE_ATOMIC and use rte_atomic_load_explicit() and
rte_atomic_store_explicit() with rte_memory_order_relaxed for all its
accesses. On modern architectures this produces the exact same machine
code while eliminating undefined behavior.

In addition, marking cb as atomic improves code self-documentation by
making it immediately evident that cb is an active part of the cross-
thread synchronization mechanism on par with use.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 lib/bpf/bpf_pkt.c | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)

diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index c5fa85bf84..0b59ce22f1 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -85,7 +85,7 @@
 struct __rte_cache_aligned bpf_eth_cbi {
 	/* used by both data & control path */
 	RTE_ATOMIC(bool) use;    /* usage flag: true when in use, false otherwise */
-	const struct rte_eth_rxtx_callback *cb;  /* callback handle */
+	RTE_ATOMIC(const struct rte_eth_rxtx_callback *) cb;  /* callback handle */
 	struct rte_bpf *bpf;
 	struct rte_bpf_jit jit;
 	/* used by control path only */
@@ -313,7 +313,7 @@ bpf_rx_callback_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 	cbi = user_param;
 
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 1) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -330,7 +330,7 @@ bpf_rx_callback_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_jit(&cbi->jit, pkt, nb_pkts, 1) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -346,7 +346,7 @@ bpf_tx_callback_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 0) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -362,7 +362,7 @@ bpf_tx_callback_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_jit(&cbi->jit, pkt, nb_pkts, 0) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -383,7 +383,7 @@ bpf_rx_callback_mb_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_mb_vm(cbi->bpf, pkt, nb_pkts, 1) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -400,7 +400,7 @@ bpf_rx_callback_mb_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_mb_jit(&cbi->jit, pkt, nb_pkts, 1) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -416,7 +416,7 @@ bpf_tx_callback_mb_vm(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_mb_vm(cbi->bpf, pkt, nb_pkts, 0) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -432,7 +432,7 @@ bpf_tx_callback_mb_jit(__rte_unused uint16_t port, __rte_unused uint16_t queue,
 
 	cbi = user_param;
 	bpf_eth_cbi_inuse(cbi);
-	rc = (cbi->cb != NULL) ?
+	rc = (rte_atomic_load_explicit(&cbi->cb, rte_memory_order_relaxed) != NULL) ?
 		pkt_filter_mb_jit(&cbi->jit, pkt, nb_pkts, 0) :
 		nb_pkts;
 	bpf_eth_cbi_unuse(cbi);
@@ -484,7 +484,7 @@ static void
 bpf_eth_cbi_unload(struct bpf_eth_cbi *bc)
 {
 	/* mark this cbi as empty */
-	bc->cb = NULL;
+	rte_atomic_store_explicit(&bc->cb, NULL, rte_memory_order_relaxed);
 
 	/* make sure datapath doesn't use bpf anymore, then destroy bpf */
 	bpf_eth_cbi_wait(bc);
@@ -495,16 +495,21 @@ bpf_eth_cbi_unload(struct bpf_eth_cbi *bc)
 static void
 bpf_eth_unload(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue)
 {
+	const struct rte_eth_rxtx_callback *cb;
 	struct bpf_eth_cbi *bc;
 
 	bc = bpf_eth_cbh_find(cbh, port, queue);
-	if (bc == NULL || bc->cb == NULL)
+	if (bc == NULL)
+		return;
+
+	cb = rte_atomic_load_explicit(&bc->cb, rte_memory_order_relaxed);
+	if (cb == NULL)
 		return;
 
 	if (cbh->type == BPF_ETH_RX)
-		rte_eth_remove_rx_callback(port, queue, bc->cb);
+		rte_eth_remove_rx_callback(port, queue, cb);
 	else
-		rte_eth_remove_tx_callback(port, queue, bc->cb);
+		rte_eth_remove_tx_callback(port, queue, cb);
 
 	bpf_eth_cbi_unload(bc);
 }
@@ -581,6 +586,7 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 	struct rte_bpf *bpf, uint32_t flags)
 {
 	int32_t rc;
+	const struct rte_eth_rxtx_callback *cb;
 	struct bpf_eth_cbi *bc;
 	rte_rx_callback_fn frx;
 	rte_tx_callback_fn ftx;
@@ -625,18 +631,20 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 		return -ENOMEM;
 
 	/* remove old one, if any */
-	if (bc->cb != NULL)
+	if (rte_atomic_load_explicit(&bc->cb, rte_memory_order_relaxed) != NULL)
 		bpf_eth_unload(cbh, port, queue);
 
 	bc->bpf = bpf;
 	bc->jit = jit;
 
 	if (cbh->type == BPF_ETH_RX)
-		bc->cb = rte_eth_add_rx_callback(port, queue, frx, bc);
+		cb = rte_eth_add_rx_callback(port, queue, frx, bc);
 	else
-		bc->cb = rte_eth_add_tx_callback(port, queue, ftx, bc);
+		cb = rte_eth_add_tx_callback(port, queue, ftx, bc);
+
+	rte_atomic_store_explicit(&bc->cb, cb, rte_memory_order_relaxed);
 
-	if (bc->cb == NULL) {
+	if (cb == NULL) {
 		rc = -rte_errno;
 		bpf_eth_cbi_cleanup(bc);
 	} else
-- 
2.43.0

[PATCH v2 9/9] bpf: add callback flag to skip multi-thread check

[Marat Khalili]

Add RTE_BPF_ETH_F_SKIP_MT_CHECK flag allowing callers to bypass the
suitability check when installing BPF packet callbacks.

This allows overriding rejections in cases where the port or queue is
configured with multi-thread capabilities (such as MT_LOCKFREE),
but the application guarantees that only one dataplane thread
accesses the queue at a time.

Update the unit test to verify that callback installation succeeds
when this flag is specified, and add a release note documenting the
restriction and the override flag.

Signed-off-by: Marat Khalili <qm2k21@gmail.com>
---
 app/test/test_bpf.c                    | 20 ++++++++++++++++++++
 doc/guides/rel_notes/release_26_11.rst |  5 +++++
 lib/bpf/bpf_pkt.c                      |  8 +++++---
 lib/bpf/rte_bpf_ethdev.h               |  5 ++++-
 4 files changed, 34 insertions(+), 4 deletions(-)

diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c
index 02523984b5..0ab8aa8759 100644
--- a/app/test/test_bpf.c
+++ b/app/test/test_bpf.c
@@ -4692,7 +4692,27 @@ test_bpf_elf_tx_mt_lockfree(load_elf_image_t load_elf_image)
 		goto fail;
 	}
 
+	/* Tx installation must succeed when check is skipped with flag */
+	ret = rte_bpf_eth_tx_install(port, 0, bpf, RTE_BPF_ETH_F_SKIP_MT_CHECK);
+	if (ret != 0) {
+		printf("%s@%d: expected rte_bpf_eth_tx_install with SKIP_MT_CHECK to succeed, got %d\n",
+		       __func__, __LINE__, ret);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+	rte_bpf_eth_tx_unload(port, 0);
+	bpf = NULL;
+
 	/* Rx installation on the same port must still succeed */
+	bpf = load_elf_image(app_test_bpf_filter_o, app_test_bpf_filter_o_len, "filter",
+			     NULL, 0, &prog_arg);
+	if (bpf == NULL) {
+		printf("%s@%d: failed to load BPF filter image\n",
+		       __func__, __LINE__);
+		ret = TEST_FAILED;
+		goto fail;
+	}
+
 	ret = rte_bpf_eth_rx_install(port, 0, bpf, 0);
 	if (ret != 0) {
 		printf("%s@%d: expected rte_bpf_eth_rx_install to succeed, got %d\n",
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index e282fa5581..a0a0161e39 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -366,6 +366,11 @@ API Changes
     ``rte_event_port_profile_links_get`` and
     ``rte_event_port_profile_unlink``
 
+* bpf: BPF packet callbacks can no longer be installed on
+  ``RTE_ETH_TX_OFFLOAD_MT_LOCKFREE`` ports and queues to prevent race
+  conditions on unload, but this safety feature can be overridden with
+  the new ``RTE_BPF_ETH_F_SKIP_MT_CHECK`` flag.
+
 * reorder: Promoted the following API from experimental to stable:
   ``rte_reorder_seqn``, ``rte_reorder_drain_up_to_seqn``,
   ``rte_reorder_min_seqn_set`` and ``rte_reorder_memory_footprint_get``.
diff --git a/lib/bpf/bpf_pkt.c b/lib/bpf/bpf_pkt.c
index 0b59ce22f1..7ccc8daa88 100644
--- a/lib/bpf/bpf_pkt.c
+++ b/lib/bpf/bpf_pkt.c
@@ -602,9 +602,11 @@ bpf_eth_elf_install(struct bpf_eth_cbh *cbh, uint16_t port, uint16_t queue,
 	if (bpf->prm.nb_prog_arg != 1)
 		return -EINVAL;
 
-	rc = bpf_eth_check_mt(cbh, port, queue);
-	if (rc != 0)
-		return rc;
+	if ((flags & RTE_BPF_ETH_F_SKIP_MT_CHECK) == 0) {
+		rc = bpf_eth_check_mt(cbh, port, queue);
+		if (rc != 0)
+			return rc;
+	}
 
 	if (cbh->type == BPF_ETH_RX)
 		frx = select_rx_callback(bpf->prm.prog_arg[0].type, flags);
diff --git a/lib/bpf/rte_bpf_ethdev.h b/lib/bpf/rte_bpf_ethdev.h
index cf3b4048f0..7a9f87e6cb 100644
--- a/lib/bpf/rte_bpf_ethdev.h
+++ b/lib/bpf/rte_bpf_ethdev.h
@@ -13,7 +13,8 @@
  * - in the data plane, it expects only one thread per port/queue pair at a time.
  *   Calling RX/TX burst functions concurrently on the same queue (e.g. when using
  *   RTE_ETH_TX_OFFLOAD_MT_LOCKFREE, shared RX queues, or multi-threaded
- *   software PMDs) is not supported.
+ *   software PMDs) is not supported and will be rejected at installation time
+ *   unless overridden by RTE_BPF_ETH_F_SKIP_MT_CHECK.
  * - in the control plane, it is not MT safe, i.e. it is not allowed to do
  *   load/unload for the same port/queue from different threads in parallel.
  * - though it allows load/unload at runtime
@@ -37,6 +38,8 @@ extern "C" {
 enum {
 	RTE_BPF_ETH_F_NONE = 0,
 	RTE_BPF_ETH_F_JIT  = 0x1, /**< use compiled into native ISA code */
+	RTE_BPF_ETH_F_SKIP_MT_CHECK = 0x2,
+	/**< skip check for concurrent data plane thread usage */
 };
 
 /**
-- 
2.43.0

Re: [PATCH v2 2/9] bpf: document ethdev callback threading limitations

[Stephen Hemminger]

On Mon, 12 Oct 2026 21:09:09 +0000
Marat Khalili <qm2k21@gmail.com> wrote:

> Document the data plane limitation in the file docstring of
> rte_bpf_ethdev.h: packet callback execution expects only one
> thread per port/queue pair at a time, making concurrent bursts
> (such as with RTE_ETH_TX_OFFLOAD_MT_LOCKFREE, shared RX queues,
> or multi-threaded software PMDs) unsupported.
> 
> Rephrase the existing control plane multi-threading limitation
> to explicitly distinguish it from the data plane requirement.
> 
> Signed-off-by: Marat Khalili <qm2k21@gmail.com>
> ---

Worth mentioning that packet callback doesn't work as expected
across primary / secondary. It only has action in the process
that registered. 

Re: [PATCH v2 9/9] bpf: add callback flag to skip multi-thread check

[Stephen Hemminger]

On Mon, 12 Oct 2026 21:09:16 +0000
Marat Khalili <qm2k21@gmail.com> wrote:

> Add RTE_BPF_ETH_F_SKIP_MT_CHECK flag allowing callers to bypass the
> suitability check when installing BPF packet callbacks.
> 
> This allows overriding rejections in cases where the port or queue is
> configured with multi-thread capabilities (such as MT_LOCKFREE),
> but the application guarantees that only one dataplane thread
> accesses the queue at a time.
> 
> Update the unit test to verify that callback installation succeeds
> when this flag is specified, and add a release note documenting the
> restriction and the override flag.
> 
> Signed-off-by: Marat Khalili <qm2k21@gmail.com>
> ---

No, this just adds more ways for things to go wrong and get abused.
Application should just not turn on lockfree.