* [PATCH 1/2] eal/freebsd: fix memseg addresses in EAL VA mode
From: Bruce Richardson @ 2026-05-25 17:13 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, stable
In-Reply-To: <20260525171340.1701509-1-bruce.richardson@intel.com>
In BSD, when mapping the contigmem segments, the iova is unconditionally
set to the physical address, irrespective of the actual IOVA mode.
Change this to use virtual addresses when iova mode is IOVA_AS_VA.
Fixes: 764bf26873b9 ("add FreeBSD support")
Cc: stable@dpdk.org
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
lib/eal/freebsd/eal_memory.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/lib/eal/freebsd/eal_memory.c b/lib/eal/freebsd/eal_memory.c
index cd608db9f9..e56d149a22 100644
--- a/lib/eal/freebsd/eal_memory.c
+++ b/lib/eal/freebsd/eal_memory.c
@@ -214,7 +214,10 @@ rte_eal_hugepage_init(void)
}
seg->addr = addr;
- seg->iova = physaddr;
+ if (rte_eal_iova_mode() == RTE_IOVA_VA)
+ seg->iova = (uintptr_t)addr;
+ else
+ seg->iova = physaddr;
seg->hugepage_sz = page_sz;
seg->len = page_sz;
seg->nchannel = mcfg->nchannel;
--
2.53.0
^ permalink raw reply related
* [PATCH 2/2] eal/freebsd: implement virtual to IOVA translation fn
From: Bruce Richardson @ 2026-05-25 17:13 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Lewis Donzis
In-Reply-To: <20260525171340.1701509-1-bruce.richardson@intel.com>
The function rte_mem_virt2iova() function was always returning BAD_IOVA
on FreeBSD as it was not implemented. Unfortunately, this function was
used by some drivers - expecting it to work as they used addresses from
rte_malloc calls. Implement this call on FreeBSD to fix things and
enable more drivers on BSD. To verify the function works as expected
add a basic unit test to the memory_autotest for it.
At the same time, we can trivially use this to implement the
rte_mem_virt2phy function for the case of using contigmem where IOVAs
are physical addresses.
Reported-by: Lewis Donzis <lew@perftech.com>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
NOTE: This is one of those patches that could be either a bugfix or a
feature. For now, I'm treating it as adding new feature support, though
in doing so it enables previously broken drivers.
---
app/test/test_memory.c | 67 ++++++++++++++++++++++++++++++++++++
lib/eal/freebsd/eal_memory.c | 22 +++++++++---
2 files changed, 84 insertions(+), 5 deletions(-)
diff --git a/app/test/test_memory.c b/app/test/test_memory.c
index 628714c0b8..260d5ed8f8 100644
--- a/app/test/test_memory.c
+++ b/app/test/test_memory.c
@@ -8,6 +8,7 @@
#include <rte_eal.h>
#include <rte_errno.h>
#include <rte_memory.h>
+#include <rte_malloc.h>
#include <rte_common.h>
#include <rte_memzone.h>
@@ -75,6 +76,68 @@ check_seg_fds(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
return 0;
}
+static int
+check_malloc_virt2iova(void)
+{
+ const size_t alloc_sz = 128;
+ const size_t off = 32;
+ struct rte_memseg *ms;
+ char *p;
+ rte_iova_t iova, iova_off;
+
+ p = rte_malloc("memory_autotest", alloc_sz, RTE_CACHE_LINE_SIZE);
+ if (p == NULL) {
+ printf("rte_malloc failed\n");
+ return -1;
+ }
+
+ iova = rte_mem_virt2iova(p);
+ if (iova == RTE_BAD_IOVA) {
+ printf("rte_mem_virt2iova failed for rte_malloc pointer\n");
+ rte_free(p);
+ return -1;
+ }
+
+ ms = rte_mem_virt2memseg(p, NULL);
+ if (ms == NULL) {
+ printf("failed to resolve memseg for rte_malloc pointer\n");
+ rte_free(p);
+ return -1;
+ }
+
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ if (ms->iova == RTE_BAD_IOVA || iova < ms->iova ||
+ iova >= ms->iova + ms->len) {
+ printf("translated iova is outside memseg range\n");
+ rte_free(p);
+ return -1;
+ }
+
+ phys_addr_t physaddr = rte_mem_virt2phy(p);
+ if (physaddr == RTE_BAD_PHYS_ADDR || physaddr != iova) {
+ printf("rte_mem_virt2phy failed for rte_malloc pointer\n");
+ rte_free(p);
+ return -1;
+ }
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ if (iova != (uintptr_t)p) {
+ printf("rte_mem_virt2iova did not return VA in VA mode\n");
+ rte_free(p);
+ return -1;
+ }
+ }
+
+ iova_off = rte_mem_virt2iova(p + off);
+ if (iova_off == RTE_BAD_IOVA || iova_off != iova + off) {
+ printf("translated iova for interior pointer is invalid\n");
+ rte_free(p);
+ return -1;
+ }
+
+ rte_free(p);
+ return 0;
+}
+
static int
test_memory(void)
{
@@ -107,6 +170,10 @@ test_memory(void)
return -1;
}
+ ret = check_malloc_virt2iova();
+ if (ret < 0)
+ return ret;
+
return 0;
}
diff --git a/lib/eal/freebsd/eal_memory.c b/lib/eal/freebsd/eal_memory.c
index e56d149a22..6952b38630 100644
--- a/lib/eal/freebsd/eal_memory.c
+++ b/lib/eal/freebsd/eal_memory.c
@@ -40,16 +40,28 @@ RTE_EXPORT_SYMBOL(rte_mem_virt2phy)
phys_addr_t
rte_mem_virt2phy(const void *virtaddr)
{
- /* XXX not implemented. This function is only used by
- * rte_mempool_virt2iova() when hugepages are disabled. */
- (void)virtaddr;
- return RTE_BAD_IOVA;
+ /* not implemented for FreeBSD when not using contigmem memory */
+ if (virtaddr == NULL || rte_eal_iova_mode() != RTE_IOVA_PA)
+ return RTE_BAD_IOVA;
+ /* when IOVA == PA, return the IOVA */
+ return rte_mem_virt2iova(virtaddr);
}
+
RTE_EXPORT_SYMBOL(rte_mem_virt2iova)
rte_iova_t
rte_mem_virt2iova(const void *virtaddr)
{
- return rte_mem_virt2phy(virtaddr);
+ if (virtaddr == NULL)
+ return RTE_BAD_IOVA;
+
+ if (rte_eal_iova_mode() == RTE_IOVA_VA)
+ return (uintptr_t)virtaddr;
+
+ const struct rte_memseg *ms = rte_mem_virt2memseg(virtaddr, NULL);
+ if (ms != NULL && ms->iova != RTE_BAD_IOVA)
+ return ms->iova + RTE_PTR_DIFF(virtaddr, ms->addr);
+
+ return RTE_BAD_IOVA;
}
int
--
2.53.0
^ permalink raw reply related
* Re: [PATCH 2/2] eal/freebsd: implement virtual to IOVA translation fn
From: Lewis Donzis @ 2026-05-25 18:23 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev
In-Reply-To: <20260525171340.1701509-3-bruce.richardson@intel.com>
Tested with igc driver, which previously failed and now works properly.
----- On May 25, 2026, at 12:13 PM, Bruce Richardson bruce.richardson@intel.com wrote:
> The function rte_mem_virt2iova() function was always returning BAD_IOVA
> on FreeBSD as it was not implemented. Unfortunately, this function was
> used by some drivers - expecting it to work as they used addresses from
> rte_malloc calls. Implement this call on FreeBSD to fix things and
> enable more drivers on BSD. To verify the function works as expected
> add a basic unit test to the memory_autotest for it.
>
> At the same time, we can trivially use this to implement the
> rte_mem_virt2phy function for the case of using contigmem where IOVAs
> are physical addresses.
>
> Reported-by: Lewis Donzis <lew@perftech.com>
Tested-by: Lewis Donzis <lew@perftech.com>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>
> NOTE: This is one of those patches that could be either a bugfix or a
> feature. For now, I'm treating it as adding new feature support, though
> in doing so it enables previously broken drivers.
> ---
> app/test/test_memory.c | 67 ++++++++++++++++++++++++++++++++++++
> lib/eal/freebsd/eal_memory.c | 22 +++++++++---
> 2 files changed, 84 insertions(+), 5 deletions(-)
>
> diff --git a/app/test/test_memory.c b/app/test/test_memory.c
> index 628714c0b8..260d5ed8f8 100644
> --- a/app/test/test_memory.c
> +++ b/app/test/test_memory.c
> @@ -8,6 +8,7 @@
> #include <rte_eal.h>
> #include <rte_errno.h>
> #include <rte_memory.h>
> +#include <rte_malloc.h>
> #include <rte_common.h>
> #include <rte_memzone.h>
>
> @@ -75,6 +76,68 @@ check_seg_fds(const struct rte_memseg_list *msl, const struct
> rte_memseg *ms,
> return 0;
> }
>
> +static int
> +check_malloc_virt2iova(void)
> +{
> + const size_t alloc_sz = 128;
> + const size_t off = 32;
> + struct rte_memseg *ms;
> + char *p;
> + rte_iova_t iova, iova_off;
> +
> + p = rte_malloc("memory_autotest", alloc_sz, RTE_CACHE_LINE_SIZE);
> + if (p == NULL) {
> + printf("rte_malloc failed\n");
> + return -1;
> + }
> +
> + iova = rte_mem_virt2iova(p);
> + if (iova == RTE_BAD_IOVA) {
> + printf("rte_mem_virt2iova failed for rte_malloc pointer\n");
> + rte_free(p);
> + return -1;
> + }
> +
> + ms = rte_mem_virt2memseg(p, NULL);
> + if (ms == NULL) {
> + printf("failed to resolve memseg for rte_malloc pointer\n");
> + rte_free(p);
> + return -1;
> + }
> +
> + if (rte_eal_iova_mode() == RTE_IOVA_PA) {
> + if (ms->iova == RTE_BAD_IOVA || iova < ms->iova ||
> + iova >= ms->iova + ms->len) {
> + printf("translated iova is outside memseg range\n");
> + rte_free(p);
> + return -1;
> + }
> +
> + phys_addr_t physaddr = rte_mem_virt2phy(p);
> + if (physaddr == RTE_BAD_PHYS_ADDR || physaddr != iova) {
> + printf("rte_mem_virt2phy failed for rte_malloc pointer\n");
> + rte_free(p);
> + return -1;
> + }
> + } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
> + if (iova != (uintptr_t)p) {
> + printf("rte_mem_virt2iova did not return VA in VA mode\n");
> + rte_free(p);
> + return -1;
> + }
> + }
> +
> + iova_off = rte_mem_virt2iova(p + off);
> + if (iova_off == RTE_BAD_IOVA || iova_off != iova + off) {
> + printf("translated iova for interior pointer is invalid\n");
> + rte_free(p);
> + return -1;
> + }
> +
> + rte_free(p);
> + return 0;
> +}
> +
> static int
> test_memory(void)
> {
> @@ -107,6 +170,10 @@ test_memory(void)
> return -1;
> }
>
> + ret = check_malloc_virt2iova();
> + if (ret < 0)
> + return ret;
> +
> return 0;
> }
>
> diff --git a/lib/eal/freebsd/eal_memory.c b/lib/eal/freebsd/eal_memory.c
> index e56d149a22..6952b38630 100644
> --- a/lib/eal/freebsd/eal_memory.c
> +++ b/lib/eal/freebsd/eal_memory.c
> @@ -40,16 +40,28 @@ RTE_EXPORT_SYMBOL(rte_mem_virt2phy)
> phys_addr_t
> rte_mem_virt2phy(const void *virtaddr)
> {
> - /* XXX not implemented. This function is only used by
> - * rte_mempool_virt2iova() when hugepages are disabled. */
> - (void)virtaddr;
> - return RTE_BAD_IOVA;
> + /* not implemented for FreeBSD when not using contigmem memory */
> + if (virtaddr == NULL || rte_eal_iova_mode() != RTE_IOVA_PA)
> + return RTE_BAD_IOVA;
> + /* when IOVA == PA, return the IOVA */
> + return rte_mem_virt2iova(virtaddr);
> }
> +
> RTE_EXPORT_SYMBOL(rte_mem_virt2iova)
> rte_iova_t
> rte_mem_virt2iova(const void *virtaddr)
> {
> - return rte_mem_virt2phy(virtaddr);
> + if (virtaddr == NULL)
> + return RTE_BAD_IOVA;
> +
> + if (rte_eal_iova_mode() == RTE_IOVA_VA)
> + return (uintptr_t)virtaddr;
> +
> + const struct rte_memseg *ms = rte_mem_virt2memseg(virtaddr, NULL);
> + if (ms != NULL && ms->iova != RTE_BAD_IOVA)
> + return ms->iova + RTE_PTR_DIFF(virtaddr, ms->addr);
> +
> + return RTE_BAD_IOVA;
> }
>
> int
> --
> 2.53.0
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Stephen Hemminger @ 2026-05-25 18:36 UTC (permalink / raw)
To: Mattias Rönnblom
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260525103642.55255-1-hofors@lysator.liu.se>
On Mon, 25 May 2026 12:36:39 +0200
Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> This RFC introduces fastmem, a general-purpose small-object allocator
> for DPDK. It is intended to replace per-type mempools with a single
> allocator that handles arbitrary sizes, grows on demand, and matches
> mempool-level performance on the hot path.
>
> Motivation
> ----------
>
> DPDK applications commonly maintain many mempools — one per object
> type (connections, sessions, timers, work items). Each must be sized
> up front, wastes memory when over-provisioned, and cannot serve
> objects of a different size. Fastmem eliminates this by accepting
> arbitrary sizes at runtime, backed by a slab allocator that
> repurposes memory across size classes as demand shifts.
>
> Design
> ------
>
> Three-layer architecture:
>
> 1. Backing memory: 128 MiB IOVA-contiguous memzones from EAL,
> reserved lazily (or pre-reserved for deterministic latency).
>
> 2. Slabs: 2 MiB, 2 MiB-aligned regions carved from memzones.
> The alignment enables O(1) slab lookup from any object pointer
> via bitmask — no radix tree or index structure. Slabs move
> freely between 18 power-of-2 size classes (8 B to 1 MiB).
>
> 3. Per-lcore caches: bounded LIFO stacks (no locks on the hot
> path). Cache misses trigger bulk transfers to/from the shared
> bin under a spinlock.
>
> Key properties:
>
> - Zero per-object metadata in the production build.
> - NUMA-aware, with per-socket bins and free-slab pools.
> - DMA-usable memory with O(1) virt-to-IOVA translation.
> - Bulk alloc/free with all-or-nothing semantics.
> - Backing memory never returned during lifetime (slabs recycled).
> - Non-EAL threads supported (bypass cache, take bin lock).
>
> API surface
> -----------
>
> rte_fastmem_init / deinit
> rte_fastmem_reserve
> rte_fastmem_set_limit / get_limit
> rte_fastmem_alloc / alloc_socket
> rte_fastmem_alloc_bulk / alloc_bulk_socket
> rte_fastmem_free / free_bulk
> rte_fastmem_virt2iova
> rte_fastmem_cache_flush
> rte_fastmem_max_size / classes
> rte_fastmem_stats / stats_class / stats_lcore / stats_lcore_class
> rte_fastmem_stats_reset
>
> All APIs are marked __rte_experimental.
>
> Performance
> -----------
>
> The single-object hot path is roughly 2-3x the cost of mempool
> and an order of magnitude faster than rte_malloc. Under
> multi-lcore contention, fastmem scales similarly to mempool,
> while rte_malloc collapses.
>
> Limitations
> -----------
>
> - Maximum allocation: 1 MiB. Larger requests should use rte_malloc.
> - Power-of-2 classes only; worst-case internal fragmentation ~50%.
> - Backing memory not reclaimable short of deinit.
>
> Future work
> -----------
>
> - Lcore-affine allocations (false-sharing-free by construction).
> - Mempool ops driver for transparent drop-in use.
> - Pre-resolved allocator handle binding size class and socket,
> eliminating per-call class lookup and enabling an inline
> cache-hit fast path.
> - Debug mode (cookies, double-free detection, poison-on-free).
> - Telemetry integration.
> - EAL integration, allowing EAL-internal subsystems to use
> fastmem for their small-object allocations.
>
> Mattias Rönnblom (3):
> doc: add fastmem programming guide
> lib: add fastmem library
> app/test: add fastmem test suite
>
> app/test/meson.build | 3 +
> app/test/test_fastmem.c | 1682 +++++++++++++++++++++++++
> app/test/test_fastmem_perf.c | 997 +++++++++++++++
> app/test/test_fastmem_profile.c | 157 +++
> doc/api/doxy-api-index.md | 1 +
> doc/api/doxy-api.conf.in | 1 +
> doc/guides/prog_guide/fastmem_lib.rst | 301 +++++
> doc/guides/prog_guide/index.rst | 1 +
> lib/fastmem/meson.build | 6 +
> lib/fastmem/rte_fastmem.c | 1486 ++++++++++++++++++++++
> lib/fastmem/rte_fastmem.h | 644 ++++++++++
> lib/meson.build | 1 +
> 12 files changed, 5280 insertions(+)
> create mode 100644 app/test/test_fastmem.c
> create mode 100644 app/test/test_fastmem_perf.c
> create mode 100644 app/test/test_fastmem_profile.c
> create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
> create mode 100644 lib/fastmem/meson.build
> create mode 100644 lib/fastmem/rte_fastmem.c
> create mode 100644 lib/fastmem/rte_fastmem.h
>
Largish patchset so did AI review with full claude model.
Series review: [RFC 0/3] add fastmem allocator
Reviewed against the v1 RFC posted 2026-05-25.
[RFC 1/3] doc: add fastmem programming guide
Info: doc/guides/prog_guide/fastmem_lib.rst -- "\ No newline at end of file"
The new RST file does not end with a newline.
[RFC 2/3] lib: add fastmem library
Error: lib/fastmem/rte_fastmem.c -- use-after-free during rte_fastmem_deinit()
when caches were allocated cross-socket.
cache_create() places the cache struct on the *calling thread's* socket,
not on the socket the cache serves:
unsigned int own_socket = rte_socket_id();
...
alloc_socket = &fastmem->sockets[own_socket];
cache = bin_alloc_one(&alloc_socket->bins[cache_class]);
...
*slot = cache; /* slot is in socket K's caches[][] */
So an lcore on socket S that calls rte_fastmem_alloc_socket(..., K) with
S != K creates a cache whose memory lives in socket S's memzone but is
reachable through socket K's caches[lcore][class].
rte_fastmem_deinit() then walks sockets in index order:
for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
release_socket(&fastmem->sockets[i]);
and release_socket() does, in this order:
socket_release_caches(socket); /* (1) */
for (c...) bin_release(&socket->bins[c], socket); /* (2) */
for (i...) rte_memzone_free(socket->memzones[i]); /* (3) */
When i = S, step (3) frees socket S's memzones. When i = K (K > S),
socket_release_caches(K) runs:
cache_slab = slab_of(cache); /* in socket S's freed mz */
bin_free_one(cache_slab->bin, cache); /* reads cache_slab->bin */
cache_slab points into a freed memzone, so cache_slab->bin and the
subsequent push (slab->free_head = obj; slab->free_count++; in
bin_push_locked()) read and write released memory. slab_release() may
then re-attach the slab to socket S's free_head, which was zeroed and
whose backing is gone.
This is triggered by any application that allocates from a non-local
socket via SOCKET_ID_ANY fallback or explicit socket_id, which the
programming guide describes as a normal mode of operation. The
existing test_alloc_socket and test_alloc_socket_numa_placement use
rte_socket_id_by_idx(0) (the local socket) so the bug is not
exercised by the test suite.
Either order the teardown in three phases (all caches across all
sockets first, then all bins, then all memzones), or allocate the
cache struct from the socket it serves rather than the calling
thread's socket.
Warning: lib/fastmem/rte_fastmem.c -- non-atomic access to shared 64-bit
statistics counters.
cache->alloc_cache_hits, alloc_cache_misses, alloc_nomem,
free_cache_hits, free_cache_misses, and the bin counters
slab_acquires, slab_releases, slabs_partial, slabs_full are
incremented as plain C reads/writes by the owning lcore and read
from another thread via rte_fastmem_stats(), rte_fastmem_stats_class(),
rte_fastmem_stats_lcore(), and rte_fastmem_stats_lcore_class(). On
architectures where uint64_t is not naturally atomic (and per the C
standard generally) this is a data race; even on x86-64 it is
undefined behavior under -fsanitize=thread.
Use rte_atomic_fetch_add_explicit() with rte_memory_order_relaxed on
the producer side and rte_atomic_load_explicit() with relaxed
ordering on the reader side. Per AGENTS.md / the DPDK convention,
relaxed ordering is appropriate for these counters.
Warning: lib/fastmem/rte_fastmem.c -- pointer publish in cache_create()
without release ordering.
*slot = cache;
return cache;
The struct fields (count, capacity, target, the stats counters) are
written before this store but with no fence or release barrier. A
concurrent stats reader doing socket->caches[l][c] followed by
cache->* could observe the pointer but not all initialized fields.
Even ignoring the stats reader, rte_fastmem_cache_flush() invoked
from a different lcore on the same cache (not currently possible by
API contract, but the field is technically reachable) would race.
Pair with rte_atomic_store_explicit(..., rte_memory_order_release)
and a matching acquire load on the reader path.
Warning: lib/fastmem/rte_fastmem.c -- spurious ENOMEM window during slab
release.
bin_push_locked() removes a fully-drained slab from bin->partial
before bin_free_one() drops the bin lock; slab_release() then puts
it on socket->free_head under the socket lock. Between the unlock
and slab_release(), another lcore allocating in any class on the
same socket can see free_head == NULL, hit the memory_limit (or
FASTMEM_MAX_MEMZONES_PER_SOCKET) check in grow_socket(), and return
ENOMEM even though the slab is about to become available. Not a
correctness issue but visible to applications that pin tightly to
their limit.
Info: lib/fastmem/rte_fastmem.c local_socket_id() final fallback:
return (unsigned int)rte_socket_id_by_idx(0);
rte_socket_id_by_idx() returns int and is documented to return -1 on
error. If there are zero configured sockets the cast yields UINT_MAX
and fastmem->sockets[UINT_MAX] is out of bounds. Realistically there
is always at least one socket, but a defensive check (return 0, or
fail allocation explicitly) would avoid the corner case.
Info: lib/fastmem/rte_fastmem.c cache_pop() refills to cache->target
(half capacity) rather than to capacity. Subsequent single-object
allocs only get target-1 hits before the next bin trip. Likely
intentional for fairness with bulk callers, but worth a comment.
Info: lib/meson.build inserts 'fastmem' between 'dispatcher' and
'gpudev'. The natural alphabetical position is between 'efd' and
'fib'; fastmem has no dependency on dispatcher.
[RFC 3/3] app/test: add fastmem test suite
Warning: app/test/test_fastmem.c -- REGISTER_FAST_TEST uses NOHUGE_OK
but the functional tests need real memzone-backed memory.
REGISTER_FAST_TEST(fastmem_autotest, NOHUGE_OK, ASAN_OK,
test_fastmem);
test_fastmem runs both the lifecycle suite (no allocations) and the
functional suite, which requests 128 MiB IOVA-contiguous memzones.
In --no-huge mode IOVA-contiguous reservation of that size is not
reliable, so NOHUGE_SKIP is more honest. If you want the lifecycle
tests to remain no-huge-friendly, register them as a separate
test command.
Warning: app/test/test_fastmem.c -- the suite never exercises
cross-socket cache allocation.
test_alloc_socket and test_alloc_socket_numa_placement both use
rte_socket_id_by_idx(0) (the local socket). Add a test that runs on
a worker lcore whose rte_socket_id() differs from the target
socket_id passed to rte_fastmem_alloc_socket(), then calls
rte_fastmem_deinit(). This would have caught the deinit UAF above.
Info: app/test/test_fastmem.c -- several test functions declare an
uninitialized `int rc;` that is never read or written (e.g.
test_alloc_too_big, test_alloc_invalid_align, test_alloc_free_small,
test_alloc_alignment, test_alloc_socket, test_alloc_block_repurposing
and others). Drop the declarations.
Info: app/test/test_fastmem.c trailing blank-line clusters (two blank
lines before "return TEST_SUCCESS;" in test_reserve_multiple_memzones,
test_reserve_cumulative, test_reserve_invalid_socket,
test_reserve_any_socket, test_alloc_too_big, ...). Drop the extra
blank line.
^ permalink raw reply
* [PATCH v2 0/3] dma/ae4dma: add AMD AE4DMA DMA PMD
From: Raghavendra Ningoji @ 2026-05-25 18:42 UTC (permalink / raw)
To: dev
Cc: Thomas Monjalon, Bhagyada Modali, Robin Jarry, Selwin.Sebastian,
david.marchand, Raghavendra Ningoji
In-Reply-To: <20260518181856.1228373-1-raghavendra.ningoji@amd.com>
This series adds a new dmadev poll-mode driver for the AMD AE4DMA
hardware DMA engine. An AE4DMA engine exposes 16 hardware command
queues, each with a 32-entry descriptor ring; the PMD maps each
hardware channel to its own dmadev with a single virtual channel,
so a PCI function appears as 16 dmadevs named "<pci-bdf>-ch0" ..
"<pci-bdf>-ch15".
Driver characteristics:
- Memory-to-memory copy operations only (RTE_DMA_CAPA_MEM_TO_MEM).
- Completion is detected via the hardware's per-queue read_idx
register, which the engine advances as it processes descriptors.
The descriptor status / err_code bytes are read only to classify
each drained slot as success or failure.
- vchan_status reports IDLE/ACTIVE based on HW read_idx vs write_idx
and HALTED_ERROR when the queue is not enabled.
- depends on bus_pci and dmadev.
The v1 was submitted as a single patch. Per review feedback the
driver is now introduced in three logical patches, following the
pattern of the recent hisi_acc dmadev driver:
1/3 - introduce driver (probe, remove, per-queue HW init)
2/3 - add control path operations (dev_ops)
3/3 - add data path operations (copy, submit, completion)
---
Changes in v2:
- Split the monolithic v1 patch into three logical patches
(introduce / control path / data path), mirroring the
structure used by drivers/dma/hisi_acc.
- Fix checkpatches.sh warnings in drivers/dma/ae4dma/ae4dma_internal.h:
* Use RTE_LOG_LINE_PREFIX (with RTE_LOGTYPE_AE4DMA_PMD) instead
of the deprecated rte_log() call form.
* Replace the GCC variadic argument-pack extension ("args...")
with C99 __VA_ARGS__ in the AE4DMA_PMD_{LOG,DEBUG,INFO,ERR,
WARN} macros.
- Move __rte_cache_aligned to the "struct" keyword position on
struct ae4dma_cmd_queue, as required by checkpatches.sh.
v1:https://patches.dpdk.org/project/dpdk/patch/20260518181856.1228373-1-raghavendra.ningoji@amd.com/
Raghavendra Ningoji (3):
dma/ae4dma: introduce AMD AE4DMA DMA PMD
dma/ae4dma: add control path operations
dma/ae4dma: add data path operations
.mailmap | 1 +
MAINTAINERS | 5 +
doc/guides/dmadevs/ae4dma.rst | 75 +++
doc/guides/dmadevs/index.rst | 1 +
doc/guides/rel_notes/release_26_07.rst | 7 +
drivers/dma/ae4dma/ae4dma_dmadev.c | 738 +++++++++++++++++++++++++
drivers/dma/ae4dma/ae4dma_hw_defs.h | 160 ++++++
drivers/dma/ae4dma/ae4dma_internal.h | 118 ++++
drivers/dma/ae4dma/meson.build | 7 +
drivers/dma/meson.build | 1 +
usertools/dpdk-devbind.py | 5 +-
11 files changed, 1117 insertions(+), 1 deletion(-)
create mode 100644 doc/guides/dmadevs/ae4dma.rst
create mode 100644 drivers/dma/ae4dma/ae4dma_dmadev.c
create mode 100644 drivers/dma/ae4dma/ae4dma_hw_defs.h
create mode 100644 drivers/dma/ae4dma/ae4dma_internal.h
create mode 100644 drivers/dma/ae4dma/meson.build
base-commit: f724d1c0d1c1636b9c171c34db3f17c3defaa2f3
--
2.34.1
^ permalink raw reply
* [PATCH v2 1/3] dma/ae4dma: introduce AMD AE4DMA DMA PMD
From: Raghavendra Ningoji @ 2026-05-25 18:42 UTC (permalink / raw)
To: dev
Cc: Thomas Monjalon, Bhagyada Modali, Robin Jarry, Selwin.Sebastian,
david.marchand, Raghavendra Ningoji
In-Reply-To: <20260525184244.1758825-1-raghavendra.ningoji@amd.com>
Add the skeleton of a new dmadev poll-mode driver for the AMD AE4DMA
hardware DMA engine, providing only PCI probe/remove and per-queue
hardware initialisation. An AE4DMA engine exposes 16 hardware command
queues, each with a 32-entry descriptor ring; the PMD maps each
hardware channel to its own dmadev with a single virtual channel,
so a PCI function appears as 16 dmadevs named "<pci-bdf>-ch0" ..
"<pci-bdf>-ch15".
This patch only registers the PCI driver, allocates the dmadev
objects, reserves the per-queue descriptor rings and programs the
hardware queue base addresses. Control and data path operations are
added in subsequent patches.
Signed-off-by: Raghavendra Ningoji <raghavendra.ningoji@amd.com>
---
.mailmap | 1 +
MAINTAINERS | 5 +
doc/guides/dmadevs/ae4dma.rst | 53 ++++++
doc/guides/dmadevs/index.rst | 1 +
doc/guides/rel_notes/release_26_07.rst | 7 +
drivers/dma/ae4dma/ae4dma_dmadev.c | 227 +++++++++++++++++++++++++
drivers/dma/ae4dma/ae4dma_hw_defs.h | 160 +++++++++++++++++
drivers/dma/ae4dma/ae4dma_internal.h | 118 +++++++++++++
drivers/dma/ae4dma/meson.build | 7 +
drivers/dma/meson.build | 1 +
usertools/dpdk-devbind.py | 5 +-
11 files changed, 584 insertions(+), 1 deletion(-)
create mode 100644 doc/guides/dmadevs/ae4dma.rst
create mode 100644 drivers/dma/ae4dma/ae4dma_dmadev.c
create mode 100644 drivers/dma/ae4dma/ae4dma_hw_defs.h
create mode 100644 drivers/dma/ae4dma/ae4dma_internal.h
create mode 100644 drivers/dma/ae4dma/meson.build
diff --git a/.mailmap b/.mailmap
index 89ba6ffccc..60180818f9 100644
--- a/.mailmap
+++ b/.mailmap
@@ -203,6 +203,7 @@ Benoît Ganne <bganne@cisco.com>
Bernard Iremonger <bernard.iremonger@intel.com>
Bert van Leeuwen <bert.vanleeuwen@netronome.com>
Bhagyada Modali <bhagyada.modali@amd.com>
+Raghavendra Ningoji <raghavendra.ningoji@amd.com>
Bharat Mota <bharat.mota@broadcom.com> <bmota@vmware.com>
Bhuvan Mital <bhuvan.mital@amd.com>
Bibo Mao <maobibo@loongson.cn>
diff --git a/MAINTAINERS b/MAINTAINERS
index 9143d028bc..2e27af49f4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1361,6 +1361,11 @@ F: doc/guides/compressdevs/features/zsda.ini
DMAdev Drivers
--------------
+AMD AE4DMA
+M: Bhagyada Modali <bhagyada.modali@amd.com>
+F: drivers/dma/ae4dma/
+F: doc/guides/dmadevs/ae4dma.rst
+
Intel IDXD - EXPERIMENTAL
M: Bruce Richardson <bruce.richardson@intel.com>
M: Kevin Laatz <kevin.laatz@intel.com>
diff --git a/doc/guides/dmadevs/ae4dma.rst b/doc/guides/dmadevs/ae4dma.rst
new file mode 100644
index 0000000000..a85c1d92ca
--- /dev/null
+++ b/doc/guides/dmadevs/ae4dma.rst
@@ -0,0 +1,53 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2025 Advanced Micro Devices, Inc.
+
+.. include:: <isonum.txt>
+
+AMD AE4DMA DMA Device Driver
+============================
+
+The ``ae4dma`` dmadev driver is a poll-mode driver (PMD) for the
+AMD AE4DMA hardware DMA engine. The engine exposes 16 independent
+hardware command queues, each with a ring of 32 descriptors. The PMD
+maps each hardware command queue to a separate DPDK dmadev with a
+single virtual channel, so a single PCI function appears as 16 dmadevs
+named ``<pci-bdf>-ch0`` through ``<pci-bdf>-ch15``.
+
+The driver supports memory-to-memory copy operations only.
+
+Hardware Requirements
+---------------------
+
+The ``dpdk-devbind.py`` script can be used to list AE4DMA devices on
+the system::
+
+ dpdk-devbind.py --status-dev dma
+
+AE4DMA devices appear with vendor ID ``0x1022`` and device ID
+``0x149b``.
+
+Compilation
+-----------
+
+The driver is built as part of the standard DPDK build on x86 platforms
+using ``meson`` and ``ninja``; no extra configuration is required.
+
+Device Setup
+------------
+
+The AE4DMA device must be bound to a DPDK-compatible kernel module such
+as ``vfio-pci`` before it can be used::
+
+ dpdk-devbind.py -b vfio-pci <pci-bdf>
+
+Initialization
+~~~~~~~~~~~~~~
+
+On probe the PMD performs the following steps for each PCI function:
+
+* Reads BAR0 and programs the common configuration register with the
+ number of hardware queues to enable (16).
+* For each hardware queue it allocates a 32-entry descriptor ring in
+ IOVA-contiguous memory, programs the queue base address and ring
+ depth into the per-queue registers, and enables the queue.
+* Interrupts are masked; completion is polled by the application.
diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst
index 56beb1733f..97399590f6 100644
--- a/doc/guides/dmadevs/index.rst
+++ b/doc/guides/dmadevs/index.rst
@@ -11,6 +11,7 @@ an application through DMA API.
:maxdepth: 1
:numbered:
+ ae4dma
cnxk
dpaa
dpaa2
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index f012d47a4b..9a78a7ef62 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -63,6 +63,13 @@ New Features
``rte_eal_init`` and the application is responsible for probing each device,
* ``--auto-probing`` enables the initial bus probing, which is the current default behavior.
+* **Added AMD AE4DMA DMA PMD.**
+
+ Added a new ``dma/ae4dma`` driver for the AMD AE4DMA hardware DMA engine.
+ Each PCI function exposes 16 hardware command queues; the PMD registers one
+ dmadev per channel with a single virtual channel and supports
+ memory-to-memory copy operations.
+
Removed Items
-------------
diff --git a/drivers/dma/ae4dma/ae4dma_dmadev.c b/drivers/dma/ae4dma/ae4dma_dmadev.c
new file mode 100644
index 0000000000..76de2cde45
--- /dev/null
+++ b/drivers/dma/ae4dma/ae4dma_dmadev.c
@@ -0,0 +1,227 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#include <errno.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+#include <rte_bus_pci.h>
+#include <bus_pci_driver.h>
+#include <rte_dmadev_pmd.h>
+#include <rte_malloc.h>
+
+#include "ae4dma_internal.h"
+
+/*
+ * One dmadev per AE4DMA hardware channel; each dmadev has exactly one
+ * virtual channel. The HW's per-queue register block must be densely
+ * packed right after the engine-common config register at BAR0+0; the
+ * build-time check below catches an accidental layout change.
+ */
+static_assert(sizeof(struct ae4dma_hwq_regs) == 32,
+ "ae4dma_hwq_regs stride changed; per-queue offset math will break");
+
+RTE_LOG_REGISTER_DEFAULT(ae4dma_pmd_logtype, INFO);
+
+#define AE4DMA_PMD_NAME dmadev_ae4dma
+
+static const struct rte_memzone *
+ae4dma_queue_dma_zone_reserve(const char *queue_name,
+ uint32_t queue_size, int socket_id)
+{
+ const struct rte_memzone *mz;
+
+ mz = rte_memzone_lookup(queue_name);
+ if (mz != NULL) {
+ if (((size_t)queue_size <= mz->len) &&
+ ((socket_id == SOCKET_ID_ANY) ||
+ (socket_id == mz->socket_id))) {
+ AE4DMA_PMD_INFO("reuse memzone already "
+ "allocated for %s", queue_name);
+ return mz;
+ }
+ AE4DMA_PMD_ERR("Incompatible memzone already "
+ "allocated %s, size %u, socket %d. "
+ "Requested size %u, socket %u",
+ queue_name, (uint32_t)mz->len,
+ mz->socket_id, queue_size, socket_id);
+ return NULL;
+ }
+ return rte_memzone_reserve_aligned(queue_name, queue_size,
+ socket_id, RTE_MEMZONE_IOVA_CONTIG, queue_size);
+}
+
+static int
+ae4dma_add_queue(struct ae4dma_dmadev *dev, uint8_t qn, const char *pci_name)
+{
+ uint32_t dma_addr_lo, dma_addr_hi;
+ struct ae4dma_cmd_queue *cmd_q;
+ const struct rte_memzone *q_mz;
+
+ dev->io_regs = dev->pci->mem_resource[AE4DMA_PCIE_BAR].addr;
+
+ cmd_q = &dev->cmd_q;
+ cmd_q->id = qn;
+ cmd_q->qidx = 0;
+ cmd_q->qsize = AE4DMA_QUEUE_SIZE(AE4DMA_QUEUE_DESC_SIZE);
+ cmd_q->hwq_regs = (volatile struct ae4dma_hwq_regs *)dev->io_regs + (qn + 1);
+
+ /*
+ * Memzone name must be globally unique. Embed PCI BDF so multiple
+ * PCI functions probed concurrently don't collide.
+ */
+ snprintf(cmd_q->memz_name, sizeof(cmd_q->memz_name),
+ "ae4dma_%s_q%u", pci_name, (unsigned int)qn);
+
+ q_mz = ae4dma_queue_dma_zone_reserve(cmd_q->memz_name,
+ cmd_q->qsize, rte_socket_id());
+ if (q_mz == NULL) {
+ AE4DMA_PMD_ERR("memzone reserve failed for %s", cmd_q->memz_name);
+ return -ENOMEM;
+ }
+
+ cmd_q->qbase_addr = (void *)q_mz->addr;
+ cmd_q->qbase_desc = (struct ae4dma_desc *)q_mz->addr;
+ cmd_q->qbase_phys_addr = q_mz->iova;
+
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->max_idx, AE4DMA_DESCRIPTORS_PER_CMDQ);
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->control_reg.control_raw,
+ AE4DMA_CMD_QUEUE_ENABLE);
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->intr_status_reg.intr_status_raw,
+ AE4DMA_DISABLE_INTR);
+ cmd_q->next_write = (uint16_t)AE4DMA_READ_REG(&cmd_q->hwq_regs->write_idx);
+ cmd_q->next_read = (uint16_t)AE4DMA_READ_REG(&cmd_q->hwq_regs->read_idx);
+ cmd_q->ring_buff_count = 0;
+
+ dma_addr_lo = low32_value(cmd_q->qbase_phys_addr);
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->qbase_lo, dma_addr_lo);
+ dma_addr_hi = high32_value(cmd_q->qbase_phys_addr);
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->qbase_hi, dma_addr_hi);
+
+ return 0;
+}
+
+static void
+ae4dma_channel_dev_name(char *out, size_t outlen, const char *pci_name,
+ unsigned int ch)
+{
+ snprintf(out, outlen, "%s-ch%u", pci_name, ch);
+}
+
+/* Create a dmadev(dpdk DMA device) */
+static int
+ae4dma_dmadev_create(const char *name, struct rte_pci_device *dev, uint8_t qn)
+{
+ struct rte_dma_dev *dmadev = NULL;
+ struct ae4dma_dmadev *ae4dma = NULL;
+ char hwq_dev_name[RTE_DEV_NAME_MAX_LEN];
+
+ if (!name) {
+ AE4DMA_PMD_ERR("Invalid name of the device!");
+ return -EINVAL;
+ }
+ memset(hwq_dev_name, 0, sizeof(hwq_dev_name));
+ ae4dma_channel_dev_name(hwq_dev_name, sizeof(hwq_dev_name), name, qn);
+
+ dmadev = rte_dma_pmd_allocate(hwq_dev_name, dev->device.numa_node,
+ sizeof(struct ae4dma_dmadev));
+ if (dmadev == NULL) {
+ AE4DMA_PMD_ERR("Unable to allocate dma device");
+ return -ENOMEM;
+ }
+ dmadev->device = &dev->device;
+ dmadev->fp_obj->dev_private = dmadev->data->dev_private;
+
+ ae4dma = dmadev->data->dev_private;
+ ae4dma->dmadev = dmadev;
+ ae4dma->pci = dev;
+
+ if (ae4dma_add_queue(ae4dma, qn, name) != 0)
+ goto init_error;
+ return 0;
+
+init_error:
+ AE4DMA_PMD_ERR("driver %s(): failed", __func__);
+ rte_dma_pmd_release(hwq_dev_name);
+ return -ENOMEM;
+}
+
+/* Probe DMA device. */
+static int
+ae4dma_dmadev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev)
+{
+ char name[32];
+ char chname[RTE_DEV_NAME_MAX_LEN];
+ void *mmio_base;
+ uint32_t q_per_eng;
+ int ret = 0;
+ uint8_t i;
+
+ rte_pci_device_name(&dev->addr, name, sizeof(name));
+ AE4DMA_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node);
+ dev->device.driver = &drv->driver;
+
+ mmio_base = dev->mem_resource[AE4DMA_PCIE_BAR].addr;
+ if (mmio_base == NULL) {
+ AE4DMA_PMD_ERR("%s: BAR%d not mapped", name, AE4DMA_PCIE_BAR);
+ return -ENODEV;
+ }
+
+ /* Program the per-engine HW queue count once. */
+ AE4DMA_WRITE_REG_OFFSET(mmio_base, AE4DMA_COMMON_CONFIG_OFFSET,
+ AE4DMA_MAX_HW_QUEUES);
+ q_per_eng = AE4DMA_READ_REG_OFFSET(mmio_base, AE4DMA_COMMON_CONFIG_OFFSET);
+ AE4DMA_PMD_INFO("%s: AE4DMA queues per engine = %u", name, q_per_eng);
+
+ for (i = 0; i < AE4DMA_MAX_HW_QUEUES; i++) {
+ ret = ae4dma_dmadev_create(name, dev, i);
+ if (ret != 0) {
+ AE4DMA_PMD_ERR("%s create dmadev %u failed!", name, i);
+ while (i > 0) {
+ i--;
+ ae4dma_channel_dev_name(chname, sizeof(chname), name, i);
+ rte_dma_pmd_release(chname);
+ }
+ break;
+ }
+ }
+ return ret;
+}
+
+/* Remove DMA device. */
+static int
+ae4dma_dmadev_remove(struct rte_pci_device *dev)
+{
+ char name[32];
+ char chname[RTE_DEV_NAME_MAX_LEN];
+ unsigned int i;
+
+ rte_pci_device_name(&dev->addr, name, sizeof(name));
+
+ AE4DMA_PMD_INFO("Closing %s on NUMA node %d",
+ name, dev->device.numa_node);
+
+ for (i = 0; i < AE4DMA_MAX_HW_QUEUES; i++) {
+ ae4dma_channel_dev_name(chname, sizeof(chname), name, i);
+ rte_dma_pmd_release(chname);
+ }
+ return 0;
+}
+
+static const struct rte_pci_id pci_id_ae4dma_map[] = {
+ { RTE_PCI_DEVICE(AMD_VENDOR_ID, AE4DMA_DEVICE_ID) },
+ { .vendor_id = 0, /* sentinel */ },
+};
+
+static struct rte_pci_driver ae4dma_pmd_drv = {
+ .id_table = pci_id_ae4dma_map,
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+ .probe = ae4dma_dmadev_probe,
+ .remove = ae4dma_dmadev_remove,
+};
+
+RTE_PMD_REGISTER_PCI(AE4DMA_PMD_NAME, ae4dma_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(AE4DMA_PMD_NAME, pci_id_ae4dma_map);
+RTE_PMD_REGISTER_KMOD_DEP(AE4DMA_PMD_NAME, "* igb_uio | uio_pci_generic | vfio-pci");
diff --git a/drivers/dma/ae4dma/ae4dma_hw_defs.h b/drivers/dma/ae4dma/ae4dma_hw_defs.h
new file mode 100644
index 0000000000..62b6a1b30b
--- /dev/null
+++ b/drivers/dma/ae4dma/ae4dma_hw_defs.h
@@ -0,0 +1,160 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#ifndef __AE4DMA_HW_DEFS_H__
+#define __AE4DMA_HW_DEFS_H__
+
+#include <rte_bus_pci.h>
+#include <rte_byteorder.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_memzone.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define AE4DMA_BIT(nr) (1UL << (nr))
+
+/* ae4dma device details */
+#define AMD_VENDOR_ID 0x1022
+#define AE4DMA_DEVICE_ID 0x149b
+#define AE4DMA_PCIE_BAR 0
+
+/*
+ * An AE4DMA engine has 16 DMA queues. Each queue supports 32 descriptors.
+ */
+#define AE4DMA_MAX_HW_QUEUES 16
+#define AE4DMA_QUEUE_START_INDEX 0
+#define AE4DMA_CMD_QUEUE_ENABLE 0x1
+#define AE4DMA_CMD_QUEUE_DISABLE 0x0
+
+/* Common to all queues */
+#define AE4DMA_COMMON_CONFIG_OFFSET 0x00
+
+#define AE4DMA_DISABLE_INTR 0x01
+
+/* Descriptor status */
+enum ae4dma_dma_status {
+ AE4DMA_DMA_DESC_SUBMITTED = 0,
+ AE4DMA_DMA_DESC_VALIDATED = 1,
+ AE4DMA_DMA_DESC_PROCESSED = 2,
+ AE4DMA_DMA_DESC_COMPLETED = 3,
+ AE4DMA_DMA_DESC_ERROR = 4,
+};
+
+/* Descriptor error-code */
+enum ae4dma_dma_err {
+ AE4DMA_DMA_ERR_NO_ERR = 0,
+ AE4DMA_DMA_ERR_INV_HEADER = 1,
+ AE4DMA_DMA_ERR_INV_STATUS = 2,
+ AE4DMA_DMA_ERR_INV_LEN = 3,
+ AE4DMA_DMA_ERR_INV_SRC = 4,
+ AE4DMA_DMA_ERR_INV_DST = 5,
+ AE4DMA_DMA_ERR_INV_ALIGN = 6,
+ AE4DMA_DMA_ERR_UNKNOWN = 7,
+};
+
+/* HW Queue status */
+enum ae4dma_hwqueue_status {
+ AE4DMA_HWQUEUE_EMPTY = 0,
+ AE4DMA_HWQUEUE_FULL = 1,
+ AE4DMA_HWQUEUE_NOT_EMPTY = 4
+};
+/*
+ * descriptor for AE4DMA commands
+ * 8 32-bit words:
+ * word 0: source memory type; destination memory type ; control bits
+ * word 1: desc_id; error code; status
+ * word 2: length
+ * word 3: reserved
+ * word 4: upper 32 bits of source pointer
+ * word 5: low 32 bits of source pointer
+ * word 6: upper 32 bits of destination pointer
+ * word 7: low 32 bits of destination pointer
+ */
+
+/* AE4DMA Descriptor - DWORD0 - Controls bits: Reserved for future use */
+#define AE4DMA_DWORD0_STOP_ON_COMPLETION AE4DMA_BIT(0)
+#define AE4DMA_DWORD0_INTERRUPT_ON_COMPLETION AE4DMA_BIT(1)
+#define AE4DMA_DWORD0_START_OF_MESSAGE AE4DMA_BIT(3)
+#define AE4DMA_DWORD0_END_OF_MESSAGE AE4DMA_BIT(4)
+#define AE4DMA_DWORD0_DESTINATION_MEMORY_TYPE RTE_GENMASK64(5, 4)
+#define AE4DMA_DWORD0_SOURCE_MEMEORY_TYPE RTE_GENMASK64(7, 6)
+
+#define AE4DMA_DWORD0_DESTINATION_MEMORY_TYPE_MEMORY (0x0)
+#define AE4DMA_DWORD0_DESTINATION_MEMORY_TYPE_IOMEMORY (1<<4)
+#define AE4DMA_DWORD0_SOURCE_MEMEORY_TYPE_MEMORY (0x0)
+#define AE4DMA_DWORD0_SOURCE_MEMEORY_TYPE_IOMEMORY (1<<6)
+
+struct ae4dma_desc_dword0 {
+ uint8_t byte0;
+ uint8_t byte1;
+ uint16_t timestamp;
+};
+
+struct ae4dma_desc_dword1 {
+ uint8_t status;
+ uint8_t err_code;
+ uint16_t desc_id;
+};
+
+struct ae4dma_desc {
+ struct ae4dma_desc_dword0 dw0;
+ struct ae4dma_desc_dword1 dw1;
+ uint32_t length;
+ uint32_t reserved;
+ uint32_t src_lo;
+ uint32_t src_hi;
+ uint32_t dst_lo;
+ uint32_t dst_hi;
+};
+
+/*
+ * Registers for each queue :4 bytes length
+ * Effective address : offset + reg
+ */
+struct ae4dma_hwq_regs {
+ union {
+ uint32_t control_raw;
+ struct {
+ uint32_t queue_enable: 1;
+ uint32_t reserved_internal: 31;
+ } control;
+ } control_reg;
+
+ union {
+ uint32_t status_raw;
+ struct {
+ uint32_t reserved0: 1;
+ /* 0–empty, 1–full, 2–stopped, 3–error , 4–Not Empty */
+ uint32_t queue_status: 2;
+ uint32_t reserved1: 21;
+ uint32_t interrupt_type: 4;
+ uint32_t reserved2: 4;
+ } status;
+ } status_reg;
+
+ uint32_t max_idx;
+ uint32_t read_idx;
+ uint32_t write_idx;
+
+ union {
+ uint32_t intr_status_raw;
+ struct {
+ uint32_t intr_status: 1;
+ uint32_t reserved: 31;
+ } intr_status;
+ } intr_status_reg;
+
+ uint32_t qbase_lo;
+ uint32_t qbase_hi;
+
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* AE4DMA_HW_DEFS_H */
diff --git a/drivers/dma/ae4dma/ae4dma_internal.h b/drivers/dma/ae4dma/ae4dma_internal.h
new file mode 100644
index 0000000000..9892d6697f
--- /dev/null
+++ b/drivers/dma/ae4dma/ae4dma_internal.h
@@ -0,0 +1,118 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
+ */
+
+#ifndef _AE4DMA_INTERNAL_H_
+#define _AE4DMA_INTERNAL_H_
+
+#include <stdint.h>
+
+#include "ae4dma_hw_defs.h"
+
+/**
+ * upper_32_bits - return bits 32-63 of a number
+ * @n: the number we're accessing
+ */
+#define upper_32_bits(n) ((uint32_t)(((n) >> 16) >> 16))
+
+/**
+ * lower_32_bits - return bits 0-31 of a number
+ * @n: the number we're accessing
+ */
+#define lower_32_bits(n) ((uint32_t)((n) & 0xffffffff))
+
+/** Hardware ring depth (slots per queue); must be power of two. */
+#define AE4DMA_DESCRIPTORS_PER_CMDQ 32
+#define AE4DMA_QUEUE_DESC_SIZE sizeof(struct ae4dma_desc)
+#define AE4DMA_QUEUE_SIZE(n) (AE4DMA_DESCRIPTORS_PER_CMDQ * (n))
+
+
+/** AE4DMA registers Write/Read */
+static inline void ae4dma_pci_reg_write(void *base, int offset,
+ uint32_t value)
+{
+ volatile void *reg_addr = ((uint8_t *)base + offset);
+
+ rte_write32((rte_cpu_to_le_32(value)), reg_addr);
+}
+
+static inline uint32_t ae4dma_pci_reg_read(void *base, int offset)
+{
+ volatile void *reg_addr = ((uint8_t *)base + offset);
+
+ return rte_le_to_cpu_32(rte_read32(reg_addr));
+}
+
+#define AE4DMA_READ_REG_OFFSET(hw_addr, reg_offset) \
+ ae4dma_pci_reg_read(hw_addr, reg_offset)
+
+#define AE4DMA_WRITE_REG_OFFSET(hw_addr, reg_offset, value) \
+ ae4dma_pci_reg_write(hw_addr, reg_offset, value)
+
+
+#define AE4DMA_READ_REG(hw_addr) \
+ ae4dma_pci_reg_read((void *)(uintptr_t)(hw_addr), 0)
+
+#define AE4DMA_WRITE_REG(hw_addr, value) \
+ ae4dma_pci_reg_write((void *)(uintptr_t)(hw_addr), 0, value)
+
+static inline uint32_t
+low32_value(unsigned long addr)
+{
+ return ((uint64_t)addr) & 0xffffffffUL;
+}
+
+static inline uint32_t
+high32_value(unsigned long addr)
+{
+ return (uint32_t)(((uint64_t)addr) >> 32);
+}
+
+/**
+ * A structure describing a AE4DMA command queue.
+ */
+struct __rte_cache_aligned ae4dma_cmd_queue {
+ char memz_name[RTE_MEMZONE_NAMESIZE];
+ volatile struct ae4dma_hwq_regs *hwq_regs;
+
+ struct rte_dma_vchan_conf qcfg;
+ struct rte_dma_stats stats;
+ /* Queue address */
+ struct ae4dma_desc *qbase_desc;
+ void *qbase_addr;
+ rte_iova_t qbase_phys_addr;
+ enum ae4dma_dma_err status[AE4DMA_DESCRIPTORS_PER_CMDQ];
+ /* Queue identifier */
+ uint64_t id; /**< queue id */
+ uint64_t qidx; /**< queue index */
+ uint64_t qsize; /**< queue size */
+ uint32_t ring_buff_count;
+ unsigned short next_read;
+ unsigned short next_write;
+ unsigned short last_write; /* Used to compute submitted count. */
+};
+
+/*
+ * One dmadev per AE4DMA hardware channel: probe creates AE4DMA_MAX_HW_QUEUES
+ * dmadevs per PCI function, each owning a single HW command queue.
+ */
+struct ae4dma_dmadev {
+ struct rte_dma_dev *dmadev;
+ void *io_regs;
+ struct ae4dma_cmd_queue cmd_q; /**< single HW queue owned by this dmadev */
+ struct rte_pci_device *pci; /**< owning PCI device (not owned) */
+};
+
+
+extern int ae4dma_pmd_logtype;
+#define RTE_LOGTYPE_AE4DMA_PMD ae4dma_pmd_logtype
+
+#define AE4DMA_PMD_LOG(level, ...) \
+ RTE_LOG_LINE_PREFIX(level, AE4DMA_PMD, "%s(): ", __func__, __VA_ARGS__)
+
+#define AE4DMA_PMD_DEBUG(...) AE4DMA_PMD_LOG(DEBUG, __VA_ARGS__)
+#define AE4DMA_PMD_INFO(...) AE4DMA_PMD_LOG(INFO, __VA_ARGS__)
+#define AE4DMA_PMD_ERR(...) AE4DMA_PMD_LOG(ERR, __VA_ARGS__)
+#define AE4DMA_PMD_WARN(...) AE4DMA_PMD_LOG(WARNING, __VA_ARGS__)
+
+#endif /* _AE4DMA_INTERNAL_H_ */
diff --git a/drivers/dma/ae4dma/meson.build b/drivers/dma/ae4dma/meson.build
new file mode 100644
index 0000000000..e48ab0d561
--- /dev/null
+++ b/drivers/dma/ae4dma/meson.build
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2024 Advanced Micro Devices, Inc. All rights reserved.
+
+build = dpdk_conf.has('RTE_ARCH_X86')
+reason = 'only supported on x86'
+sources = files('ae4dma_dmadev.c')
+deps += ['bus_pci', 'dmadev']
diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build
index e0d94db967..c230ac5a06 100644
--- a/drivers/dma/meson.build
+++ b/drivers/dma/meson.build
@@ -2,6 +2,7 @@
# Copyright 2021 HiSilicon Limited
drivers = [
+ 'ae4dma',
'cnxk',
'dpaa',
'dpaa2',
diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index 93f2383dff..7d09f155de 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -86,6 +86,9 @@
cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4',
'SVendor': None, 'SDevice': None}
+amd_ae4dma = {'Class': '08', 'Vendor': '1022', 'Device': '149b',
+ 'SVendor': None, 'SDevice': None}
+
virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042',
'SVendor': None, 'SDevice': None}
@@ -95,7 +98,7 @@
network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
baseband_devices = [acceleration_class]
crypto_devices = [encryption_class, intel_processor_class]
-dma_devices = [cnxk_dma, hisilicon_dma,
+dma_devices = [amd_ae4dma, cnxk_dma, hisilicon_dma,
intel_idxd_gnrd, intel_idxd_dmr, intel_idxd_spr,
intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx,
odm_dma]
--
2.34.1
^ permalink raw reply related
* [PATCH v2 2/3] dma/ae4dma: add control path operations
From: Raghavendra Ningoji @ 2026-05-25 18:42 UTC (permalink / raw)
To: dev
Cc: Thomas Monjalon, Bhagyada Modali, Robin Jarry, Selwin.Sebastian,
david.marchand, Raghavendra Ningoji
In-Reply-To: <20260525184244.1758825-1-raghavendra.ningoji@amd.com>
Implement the dmadev control path for the AMD AE4DMA PMD.
This commit adds:
- dev_configure / vchan_setup: accept a single virtual channel per
dmadev and clamp the requested ring size to the hardware maximum
of 32 descriptors (rounded up to a power of two).
- dev_start / dev_stop / dev_close: program the per-queue control
register to enable/disable the hardware queue and release the
descriptor ring memzone on close.
- dev_info_get: advertise RTE_DMA_CAPA_MEM_TO_MEM and the fixed
ring depth.
- dev_dump: print the queue identifiers, ring layout and software
completion counters.
- stats_get / stats_reset: expose submitted / completed / errors
counters maintained by the driver.
- vchan_status: report IDLE / ACTIVE based on hardware read_idx vs
write_idx, and HALTED_ERROR when the queue is not enabled.
The dmadev framework is wired through dev_ops in ae4dma_dmadev_create().
Signed-off-by: Raghavendra Ningoji <raghavendra.ningoji@amd.com>
---
drivers/dma/ae4dma/ae4dma_dmadev.c | 223 +++++++++++++++++++++++++++++
1 file changed, 223 insertions(+)
diff --git a/drivers/dma/ae4dma/ae4dma_dmadev.c b/drivers/dma/ae4dma/ae4dma_dmadev.c
index 76de2cde45..dfda723c13 100644
--- a/drivers/dma/ae4dma/ae4dma_dmadev.c
+++ b/drivers/dma/ae4dma/ae4dma_dmadev.c
@@ -53,6 +53,215 @@ ae4dma_queue_dma_zone_reserve(const char *queue_name,
socket_id, RTE_MEMZONE_IOVA_CONTIG, queue_size);
}
+/* Configure a device. */
+static int
+ae4dma_dev_configure(struct rte_dma_dev *dev __rte_unused,
+ const struct rte_dma_conf *dev_conf,
+ uint32_t conf_sz)
+{
+ if (sizeof(struct rte_dma_conf) != conf_sz)
+ return -EINVAL;
+
+ if (dev_conf->nb_vchans != 1)
+ return -EINVAL;
+
+ return 0;
+}
+
+/* Setup a virtual channel for AE4DMA, only 1 vchan is supported per dmadev. */
+static int
+ae4dma_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan __rte_unused,
+ const struct rte_dma_vchan_conf *qconf, uint32_t qconf_sz)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t max_desc = qconf->nb_desc;
+
+ if (sizeof(struct rte_dma_vchan_conf) != qconf_sz)
+ return -EINVAL;
+
+ if (max_desc < 2)
+ return -EINVAL;
+
+ if (!rte_is_power_of_2(max_desc))
+ max_desc = rte_align32pow2(max_desc);
+
+ if (max_desc > AE4DMA_DESCRIPTORS_PER_CMDQ) {
+ AE4DMA_PMD_DEBUG("DMA dev %u nb_desc clamped to %u",
+ dev->data->dev_id, AE4DMA_DESCRIPTORS_PER_CMDQ);
+ max_desc = AE4DMA_DESCRIPTORS_PER_CMDQ;
+ }
+
+ cmd_q->qcfg = *qconf;
+ cmd_q->qcfg.nb_desc = max_desc;
+
+ /* Ensure all counters are reset, if reconfiguring/restarting device. */
+ memset(&cmd_q->stats, 0, sizeof(cmd_q->stats));
+ return 0;
+}
+
+/* Start a configured device. */
+static int
+ae4dma_dev_start(struct rte_dma_dev *dev)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+
+ if (nb == 0)
+ return -EBUSY;
+
+ /* Program ring depth expected by hardware. */
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->max_idx, nb);
+ return 0;
+}
+
+/* Stop a configured device. */
+static int
+ae4dma_dev_stop(struct rte_dma_dev *dev)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+
+ if (cmd_q->hwq_regs != NULL)
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->control_reg.control_raw,
+ AE4DMA_CMD_QUEUE_DISABLE);
+ return 0;
+}
+
+/* Get device information of a device. */
+static int
+ae4dma_dev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *info,
+ uint32_t size)
+{
+ if (size < sizeof(*info))
+ return -EINVAL;
+ info->dev_name = dev->device->name;
+ info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM;
+ info->max_vchans = 1;
+ info->min_desc = 2;
+ info->max_desc = AE4DMA_DESCRIPTORS_PER_CMDQ;
+ info->nb_vchans = 1;
+ return 0;
+}
+
+/* Close a configured device. */
+static int
+ae4dma_dev_close(struct rte_dma_dev *dev)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+
+ if (cmd_q->hwq_regs != NULL)
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->control_reg.control_raw,
+ AE4DMA_CMD_QUEUE_DISABLE);
+
+ if (cmd_q->memz_name[0] != '\0') {
+ const struct rte_memzone *mz = rte_memzone_lookup(cmd_q->memz_name);
+
+ if (mz != NULL)
+ rte_memzone_free(mz);
+ }
+ cmd_q->qbase_desc = NULL;
+ cmd_q->qbase_addr = NULL;
+ cmd_q->qbase_phys_addr = 0;
+ return 0;
+}
+/* Dump DMA device info. */
+static int
+ae4dma_dev_dump(const struct rte_dma_dev *dev, FILE *f)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q;
+ void *ae4dma_mmio_base_addr = (uint8_t *)ae4dma->io_regs;
+
+ cmd_q = &ae4dma->cmd_q;
+ fprintf(f, "cmd_q->id = %" PRIx64 "\n", cmd_q->id);
+ fprintf(f, "cmd_q->qidx = %" PRIx64 "\n", cmd_q->qidx);
+ fprintf(f, "cmd_q->qsize = %" PRIx64 "\n", cmd_q->qsize);
+ fprintf(f, "mmio_base_addr = %p\n", ae4dma_mmio_base_addr);
+ fprintf(f, "queues per ae4dma engine = %d\n", AE4DMA_READ_REG_OFFSET(
+ ae4dma_mmio_base_addr, AE4DMA_COMMON_CONFIG_OFFSET));
+ fprintf(f, "== Private Data ==\n");
+ fprintf(f, " Config: { ring_size: %u }\n", cmd_q->qcfg.nb_desc);
+ fprintf(f, " Ring virt: %p\tphys: %#" PRIx64 "\n",
+ (void *)cmd_q->qbase_desc,
+ (uint64_t)cmd_q->qbase_phys_addr);
+ fprintf(f, " Next write: %u\n", cmd_q->next_write);
+ fprintf(f, " Next read: %u\n", cmd_q->next_read);
+ fprintf(f, " current queue depth: %u\n", cmd_q->ring_buff_count);
+ fprintf(f, " }\n");
+ fprintf(f, " Key Stats { submitted: %" PRIu64 ", comp: %" PRIu64 ", failed: %" PRIu64 " }\n",
+ cmd_q->stats.submitted,
+ cmd_q->stats.completed,
+ cmd_q->stats.errors);
+ return 0;
+}
+/* Retrieve the generic stats of a DMA device. */
+static int
+ae4dma_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused,
+ struct rte_dma_stats *rte_stats, uint32_t size)
+{
+ const struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ const struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ const struct rte_dma_stats *stats = &cmd_q->stats;
+
+ if (size < sizeof(*rte_stats))
+ return -EINVAL;
+ if (rte_stats == NULL)
+ return -EINVAL;
+
+ *rte_stats = *stats;
+ return 0;
+}
+
+/* Reset the generic stat counters for the DMA device. */
+static int
+ae4dma_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused)
+{
+ struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+
+ memset(&cmd_q->stats, 0, sizeof(cmd_q->stats));
+ return 0;
+}
+
+/*
+ * Report channel state to the dmadev framework.
+ *
+ * RTE_DMA_VCHAN_HALTED_ERROR - HW queue is disabled (never started, or
+ * stopped via dev_stop()).
+ * RTE_DMA_VCHAN_IDLE - HW has caught up: read_idx == write_idx,
+ * no descriptors in flight.
+ * RTE_DMA_VCHAN_ACTIVE - HW still has descriptors to process.
+ */
+static int
+ae4dma_vchan_status(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused,
+ enum rte_dma_vchan_status *status)
+{
+ const struct ae4dma_dmadev *ae4dma = dev->fp_obj->dev_private;
+ const struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint32_t ctrl, hw_read, hw_write;
+
+ if (cmd_q->hwq_regs == NULL) {
+ *status = RTE_DMA_VCHAN_HALTED_ERROR;
+ return 0;
+ }
+
+ ctrl = AE4DMA_READ_REG(&cmd_q->hwq_regs->control_reg.control_raw);
+ if ((ctrl & AE4DMA_CMD_QUEUE_ENABLE) == 0) {
+ *status = RTE_DMA_VCHAN_HALTED_ERROR;
+ return 0;
+ }
+
+ hw_read = AE4DMA_READ_REG(&cmd_q->hwq_regs->read_idx);
+ hw_write = AE4DMA_READ_REG(&cmd_q->hwq_regs->write_idx);
+
+ *status = (hw_read == hw_write) ? RTE_DMA_VCHAN_IDLE
+ : RTE_DMA_VCHAN_ACTIVE;
+ return 0;
+}
+
static int
ae4dma_add_queue(struct ae4dma_dmadev *dev, uint8_t qn, const char *pci_name)
{
@@ -114,6 +323,19 @@ ae4dma_channel_dev_name(char *out, size_t outlen, const char *pci_name,
static int
ae4dma_dmadev_create(const char *name, struct rte_pci_device *dev, uint8_t qn)
{
+ static const struct rte_dma_dev_ops ae4dma_dmadev_ops = {
+ .dev_close = ae4dma_dev_close,
+ .dev_configure = ae4dma_dev_configure,
+ .dev_dump = ae4dma_dev_dump,
+ .dev_info_get = ae4dma_dev_info_get,
+ .dev_start = ae4dma_dev_start,
+ .dev_stop = ae4dma_dev_stop,
+ .stats_get = ae4dma_stats_get,
+ .stats_reset = ae4dma_stats_reset,
+ .vchan_status = ae4dma_vchan_status,
+ .vchan_setup = ae4dma_vchan_setup,
+ };
+
struct rte_dma_dev *dmadev = NULL;
struct ae4dma_dmadev *ae4dma = NULL;
char hwq_dev_name[RTE_DEV_NAME_MAX_LEN];
@@ -133,6 +355,7 @@ ae4dma_dmadev_create(const char *name, struct rte_pci_device *dev, uint8_t qn)
}
dmadev->device = &dev->device;
dmadev->fp_obj->dev_private = dmadev->data->dev_private;
+ dmadev->dev_ops = &ae4dma_dmadev_ops;
ae4dma = dmadev->data->dev_private;
ae4dma->dmadev = dmadev;
--
2.34.1
^ permalink raw reply related
* [PATCH v2 3/3] dma/ae4dma: add data path operations
From: Raghavendra Ningoji @ 2026-05-25 18:42 UTC (permalink / raw)
To: dev
Cc: Thomas Monjalon, Bhagyada Modali, Robin Jarry, Selwin.Sebastian,
david.marchand, Raghavendra Ningoji
In-Reply-To: <20260525184244.1758825-1-raghavendra.ningoji@amd.com>
Implement the dmadev fast path for the AMD AE4DMA PMD.
This commit adds:
- copy enqueue (rte_dma_copy): write an AE4DMA descriptor for a
memory-to-memory transfer; on RTE_DMA_OP_FLAG_SUBMIT the doorbell
is rung immediately.
- submit (rte_dma_submit): advance the per-queue write_idx
register to expose pending descriptors to the hardware.
- completion (rte_dma_completed / rte_dma_completed_status):
completion is detected via the hardware's per-queue read_idx
register, which the engine advances as it processes descriptors.
The descriptor status / err_code bytes are read only to classify
each drained slot as success or failure, and HW error codes are
translated to the dmadev RTE_DMA_STATUS_* enumeration.
- burst capacity (rte_dma_burst_capacity): report the number of
free descriptor slots, taking into account the one slot reserved
to distinguish full from empty on the power-of-two ring.
The fast path entry points are wired through fp_obj in
ae4dma_dmadev_create(). The fill capability is not advertised;
fp_obj->fill is left zero-initialised.
Signed-off-by: Raghavendra Ningoji <raghavendra.ningoji@amd.com>
---
doc/guides/dmadevs/ae4dma.rst | 22 +++
drivers/dma/ae4dma/ae4dma_dmadev.c | 288 +++++++++++++++++++++++++++++
2 files changed, 310 insertions(+)
diff --git a/doc/guides/dmadevs/ae4dma.rst b/doc/guides/dmadevs/ae4dma.rst
index a85c1d92ca..37a2096ccf 100644
--- a/doc/guides/dmadevs/ae4dma.rst
+++ b/doc/guides/dmadevs/ae4dma.rst
@@ -51,3 +51,25 @@ On probe the PMD performs the following steps for each PCI function:
IOVA-contiguous memory, programs the queue base address and ring
depth into the per-queue registers, and enables the queue.
* Interrupts are masked; completion is polled by the application.
+
+Usage
+-----
+
+Once a dmadev has been started, copies are submitted with
+``rte_dma_copy()`` and completions are reaped with ``rte_dma_completed()``
+or ``rte_dma_completed_status()``. See the
+:ref:`Enqueue / Dequeue API <dmadev_enqueue_dequeue>` section of the
+dmadev library documentation for details.
+
+Limitations
+-----------
+
+* Only memory-to-memory copies are supported. Fill, scatter-gather and
+ any other operation types are not advertised in
+ ``rte_dma_info::dev_capa``.
+* The maximum number of descriptors per virtual channel is fixed by
+ hardware at 32. The PMD rounds the requested ring size up to a
+ power of two and clamps it to 32.
+* Only a single virtual channel per dmadev is supported; use the 16
+ per-PCI-function dmadevs to obtain channel-level parallelism.
+* Interrupt-driven completion is not supported.
diff --git a/drivers/dma/ae4dma/ae4dma_dmadev.c b/drivers/dma/ae4dma/ae4dma_dmadev.c
index dfda723c13..0f223fc40c 100644
--- a/drivers/dma/ae4dma/ae4dma_dmadev.c
+++ b/drivers/dma/ae4dma/ae4dma_dmadev.c
@@ -167,6 +167,73 @@ ae4dma_dev_close(struct rte_dma_dev *dev)
cmd_q->qbase_phys_addr = 0;
return 0;
}
+
+/* trigger h/w to process enqued desc:doorbell - by next_write */
+static inline void
+__submit(struct ae4dma_dmadev *ae4dma)
+{
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t write_idx = cmd_q->next_write;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+
+ AE4DMA_WRITE_REG(&cmd_q->hwq_regs->write_idx, write_idx);
+ if (nb != 0)
+ cmd_q->stats.submitted += (uint16_t)((cmd_q->next_write - cmd_q->last_write +
+ nb) % nb);
+ cmd_q->last_write = cmd_q->next_write;
+}
+
+static int
+ae4dma_submit(void *dev_private, uint16_t vchan __rte_unused)
+{
+ struct ae4dma_dmadev *ae4dma = dev_private;
+
+ __submit(ae4dma);
+ return 0;
+}
+
+/* Write descriptor for enqueue (copy only). */
+static inline int
+__write_desc_copy(void *dev_private, rte_iova_t src, rte_iova_t dst,
+ uint32_t len, uint64_t flags)
+{
+ struct ae4dma_dmadev *ae4dma = dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ struct ae4dma_desc *dma_desc;
+ uint16_t ret;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+ uint16_t write = cmd_q->next_write;
+
+ if (nb == 0)
+ return -EINVAL;
+
+ /* Reserve one slot to distinguish full from empty (power-of-two ring). */
+ if ((uint32_t)cmd_q->ring_buff_count >= (uint32_t)(nb - 1))
+ return -ENOSPC;
+
+ dma_desc = &cmd_q->qbase_desc[write];
+ memset(dma_desc, 0, sizeof(*dma_desc));
+ dma_desc->length = len;
+ dma_desc->src_hi = upper_32_bits(src);
+ dma_desc->src_lo = lower_32_bits(src);
+ dma_desc->dst_hi = upper_32_bits(dst);
+ dma_desc->dst_lo = lower_32_bits(dst);
+ cmd_q->ring_buff_count++;
+ cmd_q->next_write = (uint16_t)((write + 1) % nb);
+ ret = write;
+ if (flags & RTE_DMA_OP_FLAG_SUBMIT)
+ __submit(ae4dma);
+ return ret;
+}
+
+/* Enqueue a copy operation onto the ae4dma device. */
+static int
+ae4dma_enqueue_copy(void *dev_private, uint16_t vchan __rte_unused,
+ rte_iova_t src, rte_iova_t dst, uint32_t length, uint64_t flags)
+{
+ return __write_desc_copy(dev_private, src, dst, length, flags);
+}
+
/* Dump DMA device info. */
static int
ae4dma_dev_dump(const struct rte_dma_dev *dev, FILE *f)
@@ -197,6 +264,220 @@ ae4dma_dev_dump(const struct rte_dma_dev *dev, FILE *f)
cmd_q->stats.errors);
return 0;
}
+
+/* Translates AE4DMA ChanERRs to DMA error codes. */
+static inline enum rte_dma_status_code
+__translate_status_ae4dma_to_dma(enum ae4dma_dma_err status)
+{
+ AE4DMA_PMD_DEBUG("ae4dma desc status = %d", status);
+
+ switch (status) {
+ case AE4DMA_DMA_ERR_NO_ERR:
+ return RTE_DMA_STATUS_SUCCESSFUL;
+ case AE4DMA_DMA_ERR_INV_LEN:
+ return RTE_DMA_STATUS_INVALID_LENGTH;
+ case AE4DMA_DMA_ERR_INV_SRC:
+ return RTE_DMA_STATUS_INVALID_SRC_ADDR;
+ case AE4DMA_DMA_ERR_INV_DST:
+ return RTE_DMA_STATUS_INVALID_DST_ADDR;
+ case AE4DMA_DMA_ERR_INV_ALIGN:
+ /* Name matches DPDK public enum spelling. */
+ return RTE_DMA_STATUS_DATA_POISION;
+ case AE4DMA_DMA_ERR_INV_HEADER:
+ case AE4DMA_DMA_ERR_INV_STATUS:
+ return RTE_DMA_STATUS_ERROR_UNKNOWN;
+ default:
+ return RTE_DMA_STATUS_ERROR_UNKNOWN;
+ }
+}
+
+/*
+ * Scan HW queue for completed descriptors (non-blocking).
+ *
+ * The AE4DMA engine signals completion by advancing the per-queue
+ * `read_idx` register; it does not (reliably) write a status value
+ * back into the descriptor. We therefore use the HW `read_idx`
+ * register as the source of truth and only inspect the descriptor's
+ * `dw1.err_code` byte to classify each completion as success or
+ * failure.
+ *
+ * @param cmd_q
+ * The AE4DMA command queue.
+ * @param max_ops
+ * Maximum descriptors to process this call.
+ * @param[out] failed_count
+ * Number of completed descriptors that did not report success.
+ * @return
+ * Number of descriptors completed (success + failure), <= max_ops.
+ */
+static inline uint16_t
+ae4dma_scan_hwq(struct ae4dma_cmd_queue *cmd_q, uint16_t max_ops,
+ uint16_t *failed_count)
+{
+ volatile struct ae4dma_desc *hw_desc;
+ uint16_t events_count = 0, fails = 0;
+ uint16_t tail;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+ uint16_t mask;
+ uint16_t hw_read_idx;
+ uint16_t in_flight;
+ uint16_t scan_cap;
+
+ if (nb == 0 || cmd_q->ring_buff_count == 0) {
+ *failed_count = 0;
+ return 0;
+ }
+ mask = nb - 1;
+
+ hw_read_idx = (uint16_t)(AE4DMA_READ_REG(&cmd_q->hwq_regs->read_idx) & mask);
+ tail = cmd_q->next_read;
+
+ /*
+ * Descriptors completed since our last visit live in the
+ * half-open ring range [tail, hw_read_idx). If HW hasn't
+ * moved we have nothing to do.
+ */
+ in_flight = (uint16_t)((hw_read_idx - tail) & mask);
+ if (in_flight == 0) {
+ *failed_count = 0;
+ return 0;
+ }
+
+ scan_cap = max_ops;
+ if (scan_cap > AE4DMA_DESCRIPTORS_PER_CMDQ)
+ scan_cap = AE4DMA_DESCRIPTORS_PER_CMDQ;
+ if (scan_cap > in_flight)
+ scan_cap = in_flight;
+ if (scan_cap > cmd_q->ring_buff_count)
+ scan_cap = (uint16_t)cmd_q->ring_buff_count;
+
+ while (events_count < scan_cap) {
+ uint8_t hw_status;
+ uint8_t hw_err;
+
+ hw_desc = &cmd_q->qbase_desc[tail];
+ hw_status = hw_desc->dw1.status;
+ hw_err = hw_desc->dw1.err_code;
+
+ /*
+ * read_idx advancing is the definitive completion
+ * signal. The per-descriptor status byte is informational
+ * and may not yet be written when we observe it:
+ *
+ * AE4DMA_DMA_DESC_ERROR (4)
+ * Hard failure - err_code names the precise cause.
+ * AE4DMA_DMA_DESC_COMPLETED (3) or 0
+ * Success.
+ * AE4DMA_DMA_DESC_VALIDATED (1) / _PROCESSED (2)
+ * Benign race: HW had not finished updating the
+ * status byte at the instant we read it. Since
+ * read_idx has moved past this slot, treat it as
+ * success unless err_code says otherwise.
+ *
+ * A non-zero err_code is treated as a failure regardless
+ * of the observed status value.
+ */
+ if (hw_status == AE4DMA_DMA_DESC_ERROR ||
+ hw_err != AE4DMA_DMA_ERR_NO_ERR) {
+ fails++;
+ AE4DMA_PMD_WARN("Desc failed: status=%u err=%u",
+ hw_status, hw_err);
+ }
+ cmd_q->status[events_count] = (enum ae4dma_dma_err)hw_err;
+ cmd_q->ring_buff_count--;
+ events_count++;
+ tail = (tail + 1) & mask;
+ }
+
+ cmd_q->stats.completed += events_count;
+ cmd_q->stats.errors += fails;
+ cmd_q->next_read = tail;
+ *failed_count = fails;
+ return events_count;
+}
+
+/* Returns successful operations count and sets error flag if any errors. */
+static uint16_t
+ae4dma_completed(void *dev_private, uint16_t vchan __rte_unused,
+ const uint16_t max_ops, uint16_t *last_idx, bool *has_error)
+{
+ struct ae4dma_dmadev *ae4dma = dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t cpl_count, sl_count;
+ uint16_t err_count = 0;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+
+ *has_error = false;
+
+ cpl_count = ae4dma_scan_hwq(cmd_q, max_ops, &err_count);
+
+ if (cpl_count > max_ops)
+ cpl_count = max_ops;
+
+ if (cpl_count > 0 && last_idx != NULL)
+ *last_idx = (uint16_t)((cmd_q->next_read - 1 + nb) % nb);
+
+ sl_count = cpl_count - err_count;
+ if (err_count)
+ *has_error = true;
+
+ return sl_count;
+}
+
+static uint16_t
+ae4dma_completed_status(void *dev_private, uint16_t vchan __rte_unused,
+ uint16_t max_ops, uint16_t *last_idx,
+ enum rte_dma_status_code *status)
+{
+ struct ae4dma_dmadev *ae4dma = dev_private;
+ struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t cpl_count;
+ uint16_t i;
+ uint16_t err_count = 0;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+
+ cpl_count = ae4dma_scan_hwq(cmd_q, max_ops, &err_count);
+
+ if (cpl_count > max_ops)
+ cpl_count = max_ops;
+
+ if (cpl_count > 0 && last_idx != NULL)
+ *last_idx = (uint16_t)((cmd_q->next_read - 1 + nb) % nb);
+
+ if (likely(err_count == 0)) {
+ for (i = 0; i < cpl_count; i++)
+ status[i] = RTE_DMA_STATUS_SUCCESSFUL;
+ } else {
+ for (i = 0; i < cpl_count; i++)
+ status[i] = __translate_status_ae4dma_to_dma(cmd_q->status[i]);
+ }
+
+ return cpl_count;
+}
+
+/* Get the remaining capacity of the ring. */
+static uint16_t
+ae4dma_burst_capacity(const void *dev_private, uint16_t vchan __rte_unused)
+{
+ const struct ae4dma_dmadev *ae4dma = dev_private;
+ const struct ae4dma_cmd_queue *cmd_q = &ae4dma->cmd_q;
+ uint16_t nb = cmd_q->qcfg.nb_desc;
+ uint16_t mask;
+ uint16_t read_idx = cmd_q->next_read;
+ uint16_t write_idx = cmd_q->next_write;
+ uint16_t used;
+
+ if (nb < 2 || !rte_is_power_of_2(nb))
+ return 0;
+
+ mask = nb - 1;
+ used = (uint16_t)((write_idx - read_idx) & mask);
+ /* One slot reserved (same rule as enqueue). */
+ if (used >= nb - 1)
+ return 0;
+ return (uint16_t)(nb - 1 - used);
+}
+
/* Retrieve the generic stats of a DMA device. */
static int
ae4dma_stats_get(const struct rte_dma_dev *dev, uint16_t vchan __rte_unused,
@@ -357,6 +638,13 @@ ae4dma_dmadev_create(const char *name, struct rte_pci_device *dev, uint8_t qn)
dmadev->fp_obj->dev_private = dmadev->data->dev_private;
dmadev->dev_ops = &ae4dma_dmadev_ops;
+ dmadev->fp_obj->burst_capacity = ae4dma_burst_capacity;
+ dmadev->fp_obj->completed = ae4dma_completed;
+ dmadev->fp_obj->completed_status = ae4dma_completed_status;
+ dmadev->fp_obj->copy = ae4dma_enqueue_copy;
+ dmadev->fp_obj->submit = ae4dma_submit;
+ /* fill capability not advertised: leave fp_obj->fill as zero-initialised. */
+
ae4dma = dmadev->data->dev_private;
ae4dma->dmadev = dmadev;
ae4dma->pci = dev;
--
2.34.1
^ permalink raw reply related
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Mattias Rönnblom @ 2026-05-25 19:39 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260525073029.235b4d40@phoenix.local>
On 5/25/26 16:30, Stephen Hemminger wrote:
> On Mon, 25 May 2026 12:36:39 +0200
> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>
>> This RFC introduces fastmem, a general-purpose small-object allocator
>> for DPDK. It is intended to replace per-type mempools with a single
>> allocator that handles arbitrary sizes, grows on demand, and matches
>> mempool-level performance on the hot path.
>
> Makes sense, what a simple wrapper inline to allow full replacement
> testing/performance A/B comparison?
Do you mean a mempool or a heap wrapper? Or both?
I haven't looked into what options there are with mempools. A mempool
driver should be possible, but then I guess one might attempt a
whole-sale mempool-compatible API as well.
The role(s) fastmem could serve are:
a) An lcore/fast path small-object allocator when you don't know the
object size and/or count beforehand (i.e., what the cover letter says).
b) Do what mempools do and a.
c) Do what the rte_malloc heap does, but lcore/fast path-friendly. In
other words, option a but with larger objects too.
e) Something that's both b and c.
I haven't really formed an opinion yet, other than that option a seems
like a natural first step.
Fastmem is significantly slower than mempools for the moment. Claude
will tell you to inline, but that doesn't help (at least not in the
micro benchmarks). Then it will tell you to go remove the statistics,
which also doesn't help. (Latency is data dependency-driven, so stats
load/store/compute runs on resources that otherwise would have been idle.)
What does help however is pre-compute socket and bin-related info and
put into a handle, which the application may optionally use to quickly
retrieve objects of-a-certain-size/from-a-certain-socket. Still slower
than mempool though.
> === Scenario 1: Single-object hot path — cycles per (alloc + free) ===
> allocator 8 B 64 B 256 B 1024 B 4096 B
> fastmem 16.9 16.7 17.7 17.6 17.9
> fastmem_h 9.5 9.4 9.5 9.5 9.4
> mempool 6.9 6.9 6.9 7.0 6.6
> rte_malloc 93.7 93.8 94.8 100.1 130.0
> libc 118.8 119.2 20.4 20.4 111.0
>
> === Scenario 2: Batch alloc, FIFO free — cycles per alloc ===
> allocator 8 B 64 B 256 B 1024 B 4096 B
> fastmem 10.1 10.2 10.8 12.7 14.7
> fastmem_h 6.8 6.7 7.4 9.3 11.4
> mempool 4.2 4.1 4.1 4.1 4.1
> rte_malloc 58.6 58.5 62.1 67.5 68.5
> libc 104.8 104.6 73.7 203.9 1254.0
Intel(R) Xeon(R) Gold 6421N / Ubuntu 24.04 / clang
Best regards,
Mattias
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Mattias Rönnblom @ 2026-05-25 19:43 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260525113648.5bf540ca@phoenix.local>
On 5/25/26 20:36, Stephen Hemminger wrote:
> On Mon, 25 May 2026 12:36:39 +0200
> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>
>> This RFC introduces fastmem, a general-purpose small-object allocator
>> for DPDK. It is intended to replace per-type mempools with a single
>> allocator that handles arbitrary sizes, grows on demand, and matches
>> mempool-level performance on the hot path.
>>
>> Motivation
>> ----------
>>
>> DPDK applications commonly maintain many mempools — one per object
>> type (connections, sessions, timers, work items). Each must be sized
>> up front, wastes memory when over-provisioned, and cannot serve
>> objects of a different size. Fastmem eliminates this by accepting
>> arbitrary sizes at runtime, backed by a slab allocator that
>> repurposes memory across size classes as demand shifts.
>>
>> Design
>> ------
>>
>> Three-layer architecture:
>>
>> 1. Backing memory: 128 MiB IOVA-contiguous memzones from EAL,
>> reserved lazily (or pre-reserved for deterministic latency).
>>
>> 2. Slabs: 2 MiB, 2 MiB-aligned regions carved from memzones.
>> The alignment enables O(1) slab lookup from any object pointer
>> via bitmask — no radix tree or index structure. Slabs move
>> freely between 18 power-of-2 size classes (8 B to 1 MiB).
>>
>> 3. Per-lcore caches: bounded LIFO stacks (no locks on the hot
>> path). Cache misses trigger bulk transfers to/from the shared
>> bin under a spinlock.
>>
>> Key properties:
>>
>> - Zero per-object metadata in the production build.
>> - NUMA-aware, with per-socket bins and free-slab pools.
>> - DMA-usable memory with O(1) virt-to-IOVA translation.
>> - Bulk alloc/free with all-or-nothing semantics.
>> - Backing memory never returned during lifetime (slabs recycled).
>> - Non-EAL threads supported (bypass cache, take bin lock).
>>
>> API surface
>> -----------
>>
>> rte_fastmem_init / deinit
>> rte_fastmem_reserve
>> rte_fastmem_set_limit / get_limit
>> rte_fastmem_alloc / alloc_socket
>> rte_fastmem_alloc_bulk / alloc_bulk_socket
>> rte_fastmem_free / free_bulk
>> rte_fastmem_virt2iova
>> rte_fastmem_cache_flush
>> rte_fastmem_max_size / classes
>> rte_fastmem_stats / stats_class / stats_lcore / stats_lcore_class
>> rte_fastmem_stats_reset
>>
>> All APIs are marked __rte_experimental.
>>
>> Performance
>> -----------
>>
>> The single-object hot path is roughly 2-3x the cost of mempool
>> and an order of magnitude faster than rte_malloc. Under
>> multi-lcore contention, fastmem scales similarly to mempool,
>> while rte_malloc collapses.
>>
>> Limitations
>> -----------
>>
>> - Maximum allocation: 1 MiB. Larger requests should use rte_malloc.
>> - Power-of-2 classes only; worst-case internal fragmentation ~50%.
>> - Backing memory not reclaimable short of deinit.
>>
>> Future work
>> -----------
>>
>> - Lcore-affine allocations (false-sharing-free by construction).
>> - Mempool ops driver for transparent drop-in use.
>> - Pre-resolved allocator handle binding size class and socket,
>> eliminating per-call class lookup and enabling an inline
>> cache-hit fast path.
>> - Debug mode (cookies, double-free detection, poison-on-free).
>> - Telemetry integration.
>> - EAL integration, allowing EAL-internal subsystems to use
>> fastmem for their small-object allocations.
>>
>> Mattias Rönnblom (3):
>> doc: add fastmem programming guide
>> lib: add fastmem library
>> app/test: add fastmem test suite
>>
>> app/test/meson.build | 3 +
>> app/test/test_fastmem.c | 1682 +++++++++++++++++++++++++
>> app/test/test_fastmem_perf.c | 997 +++++++++++++++
>> app/test/test_fastmem_profile.c | 157 +++
>> doc/api/doxy-api-index.md | 1 +
>> doc/api/doxy-api.conf.in | 1 +
>> doc/guides/prog_guide/fastmem_lib.rst | 301 +++++
>> doc/guides/prog_guide/index.rst | 1 +
>> lib/fastmem/meson.build | 6 +
>> lib/fastmem/rte_fastmem.c | 1486 ++++++++++++++++++++++
>> lib/fastmem/rte_fastmem.h | 644 ++++++++++
>> lib/meson.build | 1 +
>> 12 files changed, 5280 insertions(+)
>> create mode 100644 app/test/test_fastmem.c
>> create mode 100644 app/test/test_fastmem_perf.c
>> create mode 100644 app/test/test_fastmem_profile.c
>> create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
>> create mode 100644 lib/fastmem/meson.build
>> create mode 100644 lib/fastmem/rte_fastmem.c
>> create mode 100644 lib/fastmem/rte_fastmem.h
>>
>
> Largish patchset so did AI review with full claude model.
>
> Series review: [RFC 0/3] add fastmem allocator
> Reviewed against the v1 RFC posted 2026-05-25.
>
>
> [RFC 1/3] doc: add fastmem programming guide
>
> Info: doc/guides/prog_guide/fastmem_lib.rst -- "\ No newline at end of file"
> The new RST file does not end with a newline.
>
>
> [RFC 2/3] lib: add fastmem library
>
> Error: lib/fastmem/rte_fastmem.c -- use-after-free during rte_fastmem_deinit()
> when caches were allocated cross-socket.
>
> cache_create() places the cache struct on the *calling thread's* socket,
> not on the socket the cache serves:
>
> unsigned int own_socket = rte_socket_id();
> ...
> alloc_socket = &fastmem->sockets[own_socket];
> cache = bin_alloc_one(&alloc_socket->bins[cache_class]);
> ...
> *slot = cache; /* slot is in socket K's caches[][] */
>
> So an lcore on socket S that calls rte_fastmem_alloc_socket(..., K) with
> S != K creates a cache whose memory lives in socket S's memzone but is
> reachable through socket K's caches[lcore][class].
>
> rte_fastmem_deinit() then walks sockets in index order:
>
> for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
> release_socket(&fastmem->sockets[i]);
>
> and release_socket() does, in this order:
>
> socket_release_caches(socket); /* (1) */
> for (c...) bin_release(&socket->bins[c], socket); /* (2) */
> for (i...) rte_memzone_free(socket->memzones[i]); /* (3) */
>
> When i = S, step (3) frees socket S's memzones. When i = K (K > S),
> socket_release_caches(K) runs:
>
> cache_slab = slab_of(cache); /* in socket S's freed mz */
> bin_free_one(cache_slab->bin, cache); /* reads cache_slab->bin */
>
> cache_slab points into a freed memzone, so cache_slab->bin and the
> subsequent push (slab->free_head = obj; slab->free_count++; in
> bin_push_locked()) read and write released memory. slab_release() may
> then re-attach the slab to socket S's free_head, which was zeroed and
> whose backing is gone.
>
> This is triggered by any application that allocates from a non-local
> socket via SOCKET_ID_ANY fallback or explicit socket_id, which the
> programming guide describes as a normal mode of operation. The
> existing test_alloc_socket and test_alloc_socket_numa_placement use
> rte_socket_id_by_idx(0) (the local socket) so the bug is not
> exercised by the test suite.
>
> Either order the teardown in three phases (all caches across all
> sockets first, then all bins, then all memzones), or allocate the
> cache struct from the socket it serves rather than the calling
> thread's socket.
>
> Warning: lib/fastmem/rte_fastmem.c -- non-atomic access to shared 64-bit
> statistics counters.
>
> cache->alloc_cache_hits, alloc_cache_misses, alloc_nomem,
> free_cache_hits, free_cache_misses, and the bin counters
> slab_acquires, slab_releases, slabs_partial, slabs_full are
> incremented as plain C reads/writes by the owning lcore and read
> from another thread via rte_fastmem_stats(), rte_fastmem_stats_class(),
> rte_fastmem_stats_lcore(), and rte_fastmem_stats_lcore_class(). On
> architectures where uint64_t is not naturally atomic (and per the C
> standard generally) this is a data race; even on x86-64 it is
> undefined behavior under -fsanitize=thread.
>
> Use rte_atomic_fetch_add_explicit() with rte_memory_order_relaxed on
> the producer side and rte_atomic_load_explicit() with relaxed
> ordering on the reader side. Per AGENTS.md / the DPDK convention,
> relaxed ordering is appropriate for these counters.
>
> Warning: lib/fastmem/rte_fastmem.c -- pointer publish in cache_create()
> without release ordering.
>
> *slot = cache;
> return cache;
>
> The struct fields (count, capacity, target, the stats counters) are
> written before this store but with no fence or release barrier. A
> concurrent stats reader doing socket->caches[l][c] followed by
> cache->* could observe the pointer but not all initialized fields.
> Even ignoring the stats reader, rte_fastmem_cache_flush() invoked
> from a different lcore on the same cache (not currently possible by
> API contract, but the field is technically reachable) would race.
> Pair with rte_atomic_store_explicit(..., rte_memory_order_release)
> and a matching acquire load on the reader path.
>
> Warning: lib/fastmem/rte_fastmem.c -- spurious ENOMEM window during slab
> release.
>
> bin_push_locked() removes a fully-drained slab from bin->partial
> before bin_free_one() drops the bin lock; slab_release() then puts
> it on socket->free_head under the socket lock. Between the unlock
> and slab_release(), another lcore allocating in any class on the
> same socket can see free_head == NULL, hit the memory_limit (or
> FASTMEM_MAX_MEMZONES_PER_SOCKET) check in grow_socket(), and return
> ENOMEM even though the slab is about to become available. Not a
> correctness issue but visible to applications that pin tightly to
> their limit.
>
> Info: lib/fastmem/rte_fastmem.c local_socket_id() final fallback:
>
> return (unsigned int)rte_socket_id_by_idx(0);
>
> rte_socket_id_by_idx() returns int and is documented to return -1 on
> error. If there are zero configured sockets the cast yields UINT_MAX
> and fastmem->sockets[UINT_MAX] is out of bounds. Realistically there
> is always at least one socket, but a defensive check (return 0, or
> fail allocation explicitly) would avoid the corner case.
>
> Info: lib/fastmem/rte_fastmem.c cache_pop() refills to cache->target
> (half capacity) rather than to capacity. Subsequent single-object
> allocs only get target-1 hits before the next bin trip. Likely
> intentional for fairness with bulk callers, but worth a comment.
>
> Info: lib/meson.build inserts 'fastmem' between 'dispatcher' and
> 'gpudev'. The natural alphabetical position is between 'efd' and
> 'fib'; fastmem has no dependency on dispatcher.
>
>
> [RFC 3/3] app/test: add fastmem test suite
>
> Warning: app/test/test_fastmem.c -- REGISTER_FAST_TEST uses NOHUGE_OK
> but the functional tests need real memzone-backed memory.
>
> REGISTER_FAST_TEST(fastmem_autotest, NOHUGE_OK, ASAN_OK,
> test_fastmem);
>
> test_fastmem runs both the lifecycle suite (no allocations) and the
> functional suite, which requests 128 MiB IOVA-contiguous memzones.
> In --no-huge mode IOVA-contiguous reservation of that size is not
> reliable, so NOHUGE_SKIP is more honest. If you want the lifecycle
> tests to remain no-huge-friendly, register them as a separate
> test command.
>
> Warning: app/test/test_fastmem.c -- the suite never exercises
> cross-socket cache allocation.
>
> test_alloc_socket and test_alloc_socket_numa_placement both use
> rte_socket_id_by_idx(0) (the local socket). Add a test that runs on
> a worker lcore whose rte_socket_id() differs from the target
> socket_id passed to rte_fastmem_alloc_socket(), then calls
> rte_fastmem_deinit(). This would have caught the deinit UAF above.
>
> Info: app/test/test_fastmem.c -- several test functions declare an
> uninitialized `int rc;` that is never read or written (e.g.
> test_alloc_too_big, test_alloc_invalid_align, test_alloc_free_small,
> test_alloc_alignment, test_alloc_socket, test_alloc_block_repurposing
> and others). Drop the declarations.
>
> Info: app/test/test_fastmem.c trailing blank-line clusters (two blank
> lines before "return TEST_SUCCESS;" in test_reserve_multiple_memzones,
> test_reserve_cumulative, test_reserve_invalid_socket,
> test_reserve_any_socket, test_alloc_too_big, ...). Drop the extra
> blank line.
Thanks. I've addressed the above issues and the fixes will be available
as an RFC v2, except:
#2 - Non-atomic stats counters
Diagnostic counters read cross-thread. On all DPDK-supported
architectures, aligned uint64_t stores are atomic in practice;
a torn read (e.g., on 32-bit x86) at worst yields a slightly
stale counter value. Not worth the ceremony.
#3 - Pointer publish without release ordering
On weakly-ordered architectures a stats reader could briefly see
uninitialized counter values for a newly-created cache. Acceptable
for diagnostic data.
#4 - Spurious ENOMEM window during slab release
Narrow timing window, not a correctness bug. Closing it would
require holding the bin lock across slab_release(), reintroducing
the contention the design avoids.
^ permalink raw reply
* Re: [PATCH] dma/ae4dma: add AMD AE4DMA DMA PMD
From: Ningoji, Raghavendra @ 2026-05-25 19:57 UTC (permalink / raw)
To: david.marchand@redhat.com
Cc: Modali, Bhagyada, Sebastian, Selwin, dev@dpdk.org,
fengchengwen@huawei.com, Ningoji, Raghavendra, rjarry@redhat.com,
thomas@monjalon.net
[-- Attachment #1: Type: text/plain, Size: 656 bytes --]
Hi David,
Thanks for the review.
- Please fix the below warnings raised by checkpatches.sh, and run this script before submitting a new revision
- Please also checks the copyright years.
- Globally in those changes, rte_iova_t should probably be used instead of phys_addr_t.
>>All the review comments have been addressed in v2 series . Now all the patches are clean and clears patch checks.
- The patch is big, splitting it into logical patches introducing one feature at a time would help.
>>As you suggested to refactor the v1 patch into logical multiple patches, the v2 patch has been split into 3 patches.
Thanks,
Raghavendra
[-- Attachment #2: Type: text/html, Size: 3102 bytes --]
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Stephen Hemminger @ 2026-05-25 22:18 UTC (permalink / raw)
To: Mattias Rönnblom
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <af3a4738-df30-43c4-b3d9-5ec1912a307a@lysator.liu.se>
On Mon, 25 May 2026 21:39:20 +0200
Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> On 5/25/26 16:30, Stephen Hemminger wrote:
> > On Mon, 25 May 2026 12:36:39 +0200
> > Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> >
> >> This RFC introduces fastmem, a general-purpose small-object allocator
> >> for DPDK. It is intended to replace per-type mempools with a single
> >> allocator that handles arbitrary sizes, grows on demand, and matches
> >> mempool-level performance on the hot path.
> >
> > Makes sense, what a simple wrapper inline to allow full replacement
> > testing/performance A/B comparison?
>
> Do you mean a mempool or a heap wrapper? Or both?
>
> I haven't looked into what options there are with mempools. A mempool
> driver should be possible, but then I guess one might attempt a
> whole-sale mempool-compatible API as well.
My thinking is a yet another allocator in DPDK is just another source
of confusion and bugs. BUT if it can consolidate and fully replace
one or more existing allocators then it would be great improvement.
Mempools are fast, but fixed and space inefficient.
Rte_malloc is slow, but flexible.
Also, need to make whatever is added play well with static
and dynamic checkers.
^ permalink raw reply
* Re: [PATCH] eal/riscv: implement prefetch using __builtin_prefetch
From: sunyuechi @ 2026-05-26 3:30 UTC (permalink / raw)
To: shiwei dang; +Cc: dev, stanislaw.kardach, stephen, david.marchand
In-Reply-To: <CAF-WEyHKBEwYW_E6UvjKwEL7LQO4pfFS7qtv-3gjz1B7c5cNYQ@mail.gmail.com>
On 3/29/26 12:38 AM, shiwei dang wrote:
> Hi ,
>
> Thanks for the review. I'll address both points (real name in
> Signed-off-by and test results) together in v2 once I have access to
> physical RISC-V hardware.
>
> Best regards, Dang Shiwei
>
>
Hi,
Gentle ping — any update on v2?
^ permalink raw reply
* [v3 0/1] net/hinic3: Fix VXLAN TSO issue
From: Feifei Wang @ 2026-05-26 3:31 UTC (permalink / raw)
To: dev
In-Reply-To: <20260520065817.931-2-wff_light@vip.163.com>
v1: The RTE_ETH_TX_OFFLOAD_VXLAN_TNL_TSO flag is added to support the VXLAN TSO function
v2: Modify the commit information issue and supplement the commit information
v3: Revise review comments. First, deterine whether the hardware supports it, then add the flag bit.
Feifei Wang (1):
net/hinic3: Fix VXLAN TSO issue
drivers/net/hinic3/hinic3_ethdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
2.47.0.windows.2
^ permalink raw reply
* [v3 1/1] net/hinic3: Fix VXLAN TSO issue
From: Feifei Wang @ 2026-05-26 3:31 UTC (permalink / raw)
To: dev; +Cc: Feifei Wang
In-Reply-To: <20260526033147.1045-1-wff_light@vip.163.com>
From: Feifei Wang <wangfeifei40@huawei.com>
VXLAN TSO lacks a flag bit, causing the processing function
to determine that the hardware does not support it, leading
to improper handling.
Signed-off-by: Feifei Wang <wangfeifei40@huawei.com>
---
drivers/net/hinic3/hinic3_ethdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/hinic3_ethdev.c b/drivers/net/hinic3/hinic3_ethdev.c
index f4eb788..5071b64 100644
--- a/drivers/net/hinic3/hinic3_ethdev.c
+++ b/drivers/net/hinic3/hinic3_ethdev.c
@@ -652,6 +652,8 @@ hinic3_dev_configure(struct rte_eth_dev *dev)
static void
hinic3_dev_tnl_tso_support(struct rte_eth_dev_info *info, struct hinic3_nic_dev *nic_dev)
{
+ if (HINIC3_SUPPORT_VXLAN_OFFLOAD(nic_dev))
+ info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_VXLAN_TNL_TSO;
if (HINIC3_SUPPORT_GENEVE_OFFLOAD(nic_dev))
info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_GENEVE_TNL_TSO;
if (HINIC3_SUPPORT_IPXIP_OFFLOAD(nic_dev))
@@ -698,7 +700,6 @@ hinic3_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)
RTE_ETH_TX_OFFLOAD_SCTP_CKSUM |
RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM |
RTE_ETH_TX_OFFLOAD_TCP_TSO | RTE_ETH_TX_OFFLOAD_MULTI_SEGS;
- if (nic_dev->feature_cap & NIC_F_HTN_CMDQ)
hinic3_dev_tnl_tso_support(info, nic_dev);
info->hash_key_size = HINIC3_RSS_KEY_SIZE;
--
2.47.0.windows.2
^ permalink raw reply related
* RE: [PATCH] net/mlx5: redirect LACP traffic for legacy E-Switch
From: Bing Zhao @ 2026-05-26 5:57 UTC (permalink / raw)
To: Dariusz Sosnowski, Slava Ovsiienko, Ori Kam, Suanming Mou,
Matan Azrad
Cc: dev@dpdk.org, stable@dpdk.org
In-Reply-To: <20260515123700.354341-1-dsosnowski@nvidia.com>
Hi,
> -----Original Message-----
> From: Dariusz Sosnowski <dsosnowski@nvidia.com>
> Sent: Friday, May 15, 2026 8:37 PM
> To: Slava Ovsiienko <viacheslavo@nvidia.com>; Bing Zhao
> <bingz@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
> <suanmingm@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Cc: dev@dpdk.org; stable@dpdk.org
> Subject: [PATCH] net/mlx5: redirect LACP traffic for legacy E-Switch
>
> Offending patch fixed the LACP miss rule logic for NICs where switchdev is
> enabled. In this case, LACP miss rules should be inserted if and only if
> started port is a main port on the embedded switch.
> Side effect of that change was that LACP miss rules are not inserted when
> switchdev is disabled and legacy SR-IOV switch mode is used.
>
> This patch addresses that:
>
> - Fix the LACP rule insertion condition.
> - Move HWS table for LACP rule creation out of FDB rules,
> so they can be created separately.
>
> Fixes: 87e4384d2662 ("net/mlx5: fix condition of LACP miss flow")
> Cc: stable@dpdk.org
>
> Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
> ---
> drivers/net/mlx5/mlx5.h | 1 +
> drivers/net/mlx5/mlx5_flow.h | 40 ++++++++-
> drivers/net/mlx5/mlx5_flow_hw.c | 140 ++++++++++++++++++++++++--------
> drivers/net/mlx5/mlx5_trigger.c | 7 +-
> 4 files changed, 145 insertions(+), 43 deletions(-)
>
> diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index
> 49a0c03544..ab5c76bfc4 100644
> --- a/drivers/net/mlx5/mlx5.h
> +++ b/drivers/net/mlx5/mlx5.h
> @@ -2040,6 +2040,7 @@ struct mlx5_priv {
> rte_spinlock_t hw_ctrl_lock;
> LIST_HEAD(hw_ctrl_flow, mlx5_ctrl_flow_entry) hw_ctrl_flows;
> LIST_HEAD(hw_ext_ctrl_flow, mlx5_ctrl_flow_entry) hw_ext_ctrl_flows;
> + struct mlx5_flow_hw_lacp_miss *hw_lacp_miss; /* HWS LACP miss flow
> +tables */
> struct mlx5_flow_hw_ctrl_fdb *hw_ctrl_fdb; /* FDB control flow
> context */
> struct mlx5_flow_hw_ctrl_nic *hw_ctrl_nic; /* NIC control flow
> context */
> struct rte_flow_pattern_template *hw_tx_repr_tagging_pt; diff --git
> a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index
> c9e72a33d6..3f5ba55bf9 100644
> --- a/drivers/net/mlx5/mlx5_flow.h
> +++ b/drivers/net/mlx5/mlx5_flow.h
> @@ -3031,6 +3031,13 @@ struct mlx5_flow_hw_ctrl_rx {
>
> [MLX5_FLOW_HW_CTRL_RX_EXPANDED_RSS_MAX];
> };
>
> +/* Contains all templates and table required for redirecting LACP
> +traffic with HWS. */ struct mlx5_flow_hw_lacp_miss {
> + struct rte_flow_pattern_template *lacp_rx_items_tmpl;
> + struct rte_flow_actions_template *lacp_rx_actions_tmpl;
> + struct rte_flow_template_table *hw_lacp_rx_tbl; };
> +
> /* Contains all templates required for control flow rules in FDB with
> HWS. */ struct mlx5_flow_hw_ctrl_fdb {
> struct rte_flow_pattern_template *esw_mgr_items_tmpl; @@ -3042,9
> +3049,6 @@ struct mlx5_flow_hw_ctrl_fdb {
> struct rte_flow_pattern_template *port_items_tmpl;
> struct rte_flow_actions_template *jump_one_actions_tmpl;
> struct rte_flow_template_table *hw_esw_zero_tbl;
> - struct rte_flow_pattern_template *lacp_rx_items_tmpl;
> - struct rte_flow_actions_template *lacp_rx_actions_tmpl;
> - struct rte_flow_template_table *hw_lacp_rx_tbl;
> };
>
> struct mlx5_flow_hw_ctrl_nic {
> @@ -3735,6 +3739,36 @@ mlx5_indirect_list_handles_release(struct
> rte_eth_dev *dev);
>
> bool mlx5_flow_is_steering_disabled(void);
>
> +/**
> + * Returns true if Rx control rule for LACP traffic is needed.
> + *
> + * mlx5 PMD needs to create a rule matching LACP traffic and forwarding
> it back to kernel if:
> + *
> + * - Underlying device is a bond interface.
> + * - User did not request to handle LACP traffic in user space.
> + *
> + * Creation of this rule is also controlled by the E-Switch mode:
> + *
> + * - It must be created in legacy mode.
> + * - It must be created only on proxy port in switchdev mode.
> + *
> + * @param[in] priv
> + * Pointer to Ethernet device structure.
> + *
> + * @return
> + * True if LACP rules must be created.
> + * False otherwise.
> + */
> +static inline bool
> +mlx5_flow_lacp_miss_needed(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv = dev->data->dev_private;
> +
> + return !priv->sh->config.lacp_by_user &&
> + priv->pf_bond >= 0 &&
> + (!priv->sh->esw_mode || (priv->sh->esw_mode && priv->master)); }
> +
> #ifdef HAVE_MLX5_HWS_SUPPORT
>
> #define MLX5_REPR_STC_MEMORY_LOG 11
> diff --git a/drivers/net/mlx5/mlx5_flow_hw.c
> b/drivers/net/mlx5/mlx5_flow_hw.c index b6bb9f12a6..c133230cb7 100644
> --- a/drivers/net/mlx5/mlx5_flow_hw.c
> +++ b/drivers/net/mlx5/mlx5_flow_hw.c
> @@ -10820,15 +10820,6 @@ flow_hw_cleanup_ctrl_fdb_tables(struct
> rte_eth_dev *dev)
> if (!priv->hw_ctrl_fdb)
> return;
> hw_ctrl_fdb = priv->hw_ctrl_fdb;
> - /* Clean up templates used for LACP default miss table. */
> - if (hw_ctrl_fdb->hw_lacp_rx_tbl)
> - claim_zero(flow_hw_table_destroy(dev, hw_ctrl_fdb-
> >hw_lacp_rx_tbl, NULL));
> - if (hw_ctrl_fdb->lacp_rx_actions_tmpl)
> - claim_zero(flow_hw_actions_template_destroy(dev, hw_ctrl_fdb-
> >lacp_rx_actions_tmpl,
> - NULL));
> - if (hw_ctrl_fdb->lacp_rx_items_tmpl)
> - claim_zero(flow_hw_pattern_template_destroy(dev, hw_ctrl_fdb-
> >lacp_rx_items_tmpl,
> - NULL));
> /* Clean up templates used for default FDB jump rule. */
> if (hw_ctrl_fdb->hw_esw_zero_tbl)
> claim_zero(flow_hw_table_destroy(dev, hw_ctrl_fdb-
> >hw_esw_zero_tbl, NULL)); @@ -10898,6 +10889,99 @@
> flow_hw_create_lacp_rx_table(struct rte_eth_dev *dev,
> return flow_hw_table_create(dev, &cfg, &it, 1, &at, 1, error); }
>
> +/*
> + * Clean up templates and table used for redirecting LACP traffic to
> kernel.
> + *
> + * @param dev
> + * Pointer to Ethernet device.
> + */
> +static void
> +flow_hw_cleanup_lacp_miss_tables(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv = dev->data->dev_private;
> + struct mlx5_flow_hw_lacp_miss *hw_lacp_miss;
> +
> + if (priv->hw_lacp_miss == NULL)
> + return;
> +
> + hw_lacp_miss = priv->hw_lacp_miss;
> +
> + if (hw_lacp_miss->hw_lacp_rx_tbl)
> + claim_zero(flow_hw_table_destroy(dev, hw_lacp_miss-
> >hw_lacp_rx_tbl, NULL));
> + if (hw_lacp_miss->lacp_rx_actions_tmpl)
> + claim_zero(flow_hw_actions_template_destroy(dev,
> + hw_lacp_miss-
> >lacp_rx_actions_tmpl,
> + NULL));
> + if (hw_lacp_miss->lacp_rx_items_tmpl)
> + claim_zero(flow_hw_pattern_template_destroy(dev,
> + hw_lacp_miss-
> >lacp_rx_items_tmpl,
> + NULL));
> +
> + mlx5_free(hw_lacp_miss);
> + priv->hw_lacp_miss = NULL;
> +}
> +
> +/*
> + * Create templates and table for redirecting LACP traffic to kernel.
> + *
> + * LACP traffic redirection is needed whenever LACP bond is managed by
> the kernel.
> + * Required rule has a following structure:
> + *
> + * - ingress rule on root table
> + * - match EtherType 0x8809
> + * - action DEFAULT_MISS
> + *
> + * @param dev
> + * Pointer to Ethernet device.
> + *
> + * @return
> + * 0 on success. Negative errno otherwise.
> + */
> +static int
> +flow_hw_create_lacp_miss_tables(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv = dev->data->dev_private;
> + struct mlx5_flow_hw_lacp_miss *hw_lacp_miss;
> +
> + if (mlx5_flow_is_steering_disabled())
> + return 0;
> +
> + hw_lacp_miss = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*hw_lacp_miss), 0,
> SOCKET_ID_ANY);
> + if (!hw_lacp_miss) {
> + DRV_LOG(ERR, "port %u Failed to allocate memory for LACP miss
> tables",
> + dev->data->port_id);
> + return -ENOMEM;
> + }
> + priv->hw_lacp_miss = hw_lacp_miss;
> +
> + hw_lacp_miss->lacp_rx_items_tmpl =
> flow_hw_create_lacp_rx_pattern_template(dev, NULL);
> + if (!hw_lacp_miss->lacp_rx_items_tmpl) {
> + DRV_LOG(ERR, "port %u Failed to create pattern template for
> LACP Rx traffic",
> + dev->data->port_id);
> + goto error;
> + }
> + hw_lacp_miss->lacp_rx_actions_tmpl =
> flow_hw_create_lacp_rx_actions_template(dev, NULL);
> + if (!hw_lacp_miss->lacp_rx_actions_tmpl) {
> + DRV_LOG(ERR, "port %u Failed to create actions template for
> LACP Rx traffic",
> + dev->data->port_id);
> + goto error;
> + }
> + hw_lacp_miss->hw_lacp_rx_tbl =
> + flow_hw_create_lacp_rx_table(dev, hw_lacp_miss-
> >lacp_rx_items_tmpl,
> + hw_lacp_miss->lacp_rx_actions_tmpl,
> NULL);
> + if (!hw_lacp_miss->hw_lacp_rx_tbl) {
> + DRV_LOG(ERR, "port %u Failed to create template table for LACP
> Rx traffic",
> + dev->data->port_id);
> + goto error;
> + }
> +
> + return 0;
> +
> +error:
> + flow_hw_cleanup_lacp_miss_tables(dev);
> + return -EINVAL;
> +}
> +
> /**
> * Creates a set of flow tables used to create control flows used
> * when E-Switch is engaged.
> @@ -11000,31 +11084,6 @@ flow_hw_create_fdb_ctrl_tables(struct rte_eth_dev
> *dev, struct rte_flow_error *e
> goto err;
> }
> }
> - /* Create LACP default miss table. */
> - if (!priv->sh->config.lacp_by_user && priv->pf_bond >= 0 && priv-
> >master) {
> - hw_ctrl_fdb->lacp_rx_items_tmpl =
> - flow_hw_create_lacp_rx_pattern_template(dev,
> error);
> - if (!hw_ctrl_fdb->lacp_rx_items_tmpl) {
> - DRV_LOG(ERR, "port %u failed to create pattern template"
> - " for LACP Rx traffic", dev->data->port_id);
> - goto err;
> - }
> - hw_ctrl_fdb->lacp_rx_actions_tmpl =
> - flow_hw_create_lacp_rx_actions_template(dev,
> error);
> - if (!hw_ctrl_fdb->lacp_rx_actions_tmpl) {
> - DRV_LOG(ERR, "port %u failed to create actions template"
> - " for LACP Rx traffic", dev->data->port_id);
> - goto err;
> - }
> - hw_ctrl_fdb->hw_lacp_rx_tbl = flow_hw_create_lacp_rx_table
> - (dev, hw_ctrl_fdb->lacp_rx_items_tmpl,
> - hw_ctrl_fdb->lacp_rx_actions_tmpl, error);
> - if (!hw_ctrl_fdb->hw_lacp_rx_tbl) {
> - DRV_LOG(ERR, "port %u failed to create template table
> for"
> - " for LACP Rx traffic", dev->data->port_id);
> - goto err;
> - }
> - }
> return 0;
>
> err:
> @@ -11754,6 +11813,7 @@ __mlx5_flow_hw_resource_release(struct rte_eth_dev
> *dev, bool ctx_close)
>
> mlx5_flow_hw_rxq_flag_set(dev, false);
> flow_hw_flush_all_ctrl_flows(dev);
> + flow_hw_cleanup_lacp_miss_tables(dev);
> flow_hw_cleanup_ctrl_fdb_tables(dev);
> flow_hw_cleanup_ctrl_nic_tables(dev);
> flow_hw_cleanup_tx_repr_tagging(dev);
> @@ -12160,6 +12220,14 @@ __flow_hw_configure(struct rte_eth_dev *dev,
> if (ret)
> goto err;
> }
> + if (mlx5_flow_lacp_miss_needed(dev)) {
> + ret = flow_hw_create_lacp_miss_tables(dev);
> + if (ret) {
> + rte_flow_error_set(error, -ret,
> RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL,
> + "Unable to create LACP miss flow
> tables");
> + goto err;
> + }
> + }
> if (is_proxy) {
> ret = flow_hw_create_vport_actions(priv);
> if (ret) {
> @@ -16234,10 +16302,10 @@ mlx5_flow_hw_lacp_rx_flow(struct rte_eth_dev
> *dev)
> .type = MLX5_CTRL_FLOW_TYPE_LACP_RX,
> };
>
> - if (!priv->dr_ctx || !priv->hw_ctrl_fdb || !priv->hw_ctrl_fdb-
> >hw_lacp_rx_tbl)
> + if (!priv->dr_ctx || !priv->hw_lacp_miss ||
> +!priv->hw_lacp_miss->hw_lacp_rx_tbl)
> return 0;
> return flow_hw_create_ctrl_flow(dev, dev,
> - priv->hw_ctrl_fdb->hw_lacp_rx_tbl,
> + priv->hw_lacp_miss->hw_lacp_rx_tbl,
> eth_lacp, 0, miss_action, 0, &flow_info,
> false); }
>
> diff --git a/drivers/net/mlx5/mlx5_trigger.c
> b/drivers/net/mlx5/mlx5_trigger.c index a070aaecfd..32cd18717d 100644
> --- a/drivers/net/mlx5/mlx5_trigger.c
> +++ b/drivers/net/mlx5/mlx5_trigger.c
> @@ -1672,9 +1672,8 @@ mlx5_traffic_enable_hws(struct rte_eth_dev *dev)
> } else {
> DRV_LOG(INFO, "port %u FDB default rule is disabled", dev-
> >data->port_id);
> }
> - if (!priv->sh->config.lacp_by_user && priv->pf_bond >= 0 && priv-
> >master)
> - if (mlx5_flow_hw_lacp_rx_flow(dev))
> - goto error;
> + if (mlx5_flow_lacp_miss_needed(dev) &&
> mlx5_flow_hw_lacp_rx_flow(dev) != 0)
> + goto error;
> if (priv->isolated)
> return 0;
> ret = mlx5_flow_hw_create_ctrl_rx_tables(dev);
> @@ -1796,7 +1795,7 @@ mlx5_traffic_enable(struct rte_eth_dev *dev)
> DRV_LOG(INFO, "port %u FDB default rule is disabled",
> dev->data->port_id);
> }
> - if (!priv->sh->config.lacp_by_user && priv->pf_bond >= 0 && priv-
> >master) {
> + if (mlx5_flow_lacp_miss_needed(dev)) {
> ret = mlx5_flow_lacp_miss(dev);
> if (ret)
> DRV_LOG(INFO, "port %u LACP rule cannot be created - "
> --
> 2.47.3
Acked-by: Bing Zhao <bingz@nvidia.com>
^ permalink raw reply
* Re: [RFC 0/3] lib/fastmem: fast small-object allocator
From: Mattias Rönnblom @ 2026-05-26 7:01 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260525151824.6a32e829@phoenix.local>
On 5/26/26 00:18, Stephen Hemminger wrote:
> On Mon, 25 May 2026 21:39:20 +0200
> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>
>> On 5/25/26 16:30, Stephen Hemminger wrote:
>>> On Mon, 25 May 2026 12:36:39 +0200
>>> Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>>>
>>>> This RFC introduces fastmem, a general-purpose small-object allocator
>>>> for DPDK. It is intended to replace per-type mempools with a single
>>>> allocator that handles arbitrary sizes, grows on demand, and matches
>>>> mempool-level performance on the hot path.
>>>
>>> Makes sense, what a simple wrapper inline to allow full replacement
>>> testing/performance A/B comparison?
>>
>> Do you mean a mempool or a heap wrapper? Or both?
>>
>> I haven't looked into what options there are with mempools. A mempool
>> driver should be possible, but then I guess one might attempt a
>> whole-sale mempool-compatible API as well.
>
> My thinking is a yet another allocator in DPDK is just another source
> of confusion and bugs. BUT if it can consolidate and fully replace
> one or more existing allocators then it would be great improvement.
>
> Mempools are fast, but fixed and space inefficient.
> Rte_malloc is slow, but flexible.
>
> Also, need to make whatever is added play well with static
> and dynamic checkers.
I'm not sure it's possible to replace mempools with a slab allocator
like fastmem. They have different semantics, and I suspect that there
are times when you prefer a mempool.
# Object Identity & Content Preservation
A mempool always returns one of the same pre-populated objects, with its
contents untouched since last use. This enables pre-initialized fields,
hardware-registered buffers, and constructors that run only once.
# Safe Use-After-Free
Returned objects remain valid, typed memory even after release. Stale
references do not segfault or observe unrelated data, enabling RCU-style
deferred reclamation.
# Bounded, Failure-Free Operation
A mempool operates with a fixed number of objects and performs no
runtime memory allocation. This guarantees deterministic latency,
natural backpressure, and eliminates `ENOMEM` failures after initialization.
# Known IOVA at Initialization Time
All object addresses, both virtual and physical, are fixed and
enumerable from creation time. This enables pre-programming DMA
descriptors and IOMMU registration.
# Memory Accounting
A mempool provides an exact, attributable memory footprint per pool,
without sharing backing memory across unrelated users.
# Dense, Enumerable Object Set
Objects share a common base address and fixed stride, enabling efficient
iteration and pointer compression.
Considering many apps use DPDK for I/O and other hardware abstraction
only, and carry all other OS/kernel/platform type infrastructure
themselves, replacing the mempools with something else will likely cause
a lot of friction.
A fastmem-backed mempool backend (with limitations)? Sure.
Replacing rte_malloc seems easier, but I haven't looked into that in
detail yet.
^ permalink raw reply
* Re: [PATCH 1/2] bus/uacce: support driver forward compatibility
From: David Marchand @ 2026-05-26 7:31 UTC (permalink / raw)
To: fengchengwen; +Cc: thomas, dev, qianweili, liuyonglong
In-Reply-To: <c0f24426-19b2-42bf-9432-a1c58ac19c66@huawei.com>
Hello,
On Fri, 22 May 2026 at 03:05, fengchengwen <fengchengwen@huawei.com> wrote:
> >> {
> >> + bool forward_compat = !!(dr->drv_flags & RTE_UACCE_DRV_FORWARD_COMPATIBILITY_DEV);
> >> + uint32_t api_ver = uacce_calc_api_ver(dev->api, NULL);
> >
> > This conversion from a string to integer could be placed in the scanning step.
> > Why place it here?
>
> I think it's OK to place in the scanning step.
> Should I submit a commit to fix this?
Let me handle it.
I'll post a new revision of the series with a fix soon.
> > The dev->api_ver has no in-tree user.
> > This field was (silently?) dropped by my best AI friend in the bus
> > refactoring series I posted.
> > https://patchwork.dpdk.org/project/dpdk/patch/20260506155201.2709810-12-david.marchand@redhat.com/
> >
> > Please advise if I can drop this field (it is just the integer value
> > extracted from dev->api afaiu), or if it should be moved to the
> > scanning step.
>
> Please keep this field.
> This field mainly used for driver to know which api-version the device support, so driver could do
> bug-fix for specific device by judging this field.
I am not a fan of leaving unused stuff, for a potential future case
where it could (maybe) be needed.
This is your bus and drivers, and the change is simple enough, so ok,
no strong objection from me.
Thanks for the reply (and sorry for not replying sooner, I was out for
some days).
--
David Marchand
^ permalink raw reply
* [PATCH v2 1/5] examples/l3fwd-power: fix uncore deinit for non-legacy
From: Huisong Li @ 2026-05-26 8:11 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
In-Reply-To: <20260526081138.1434947-1-lihuisong@huawei.com>
Uncore resources were not being deinitialized in non-legacy modes (such
as pmd-mgmt), causing the uncore frequency not to return to its original
value after the application exited.
The root cause is that uncore initialization can be performed for all
modes, whereas the deinitialization logic is incorrectly restricted
to legacy mode only. So do the deinitialization of uncore on all app
modes.
Fixes: 10db2a5b8724 ("examples/l3fwd-power: add options for uncore frequency")
Cc: stable@dpdk.org
Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
examples/l3fwd-power/main.c | 66 ++++++++++++++++++++-----------------
1 file changed, 35 insertions(+), 31 deletions(-)
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 02ec17d799..1122aeb930 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -2271,28 +2271,31 @@ init_power_library(void)
unsigned int lcore_id;
int ret = 0;
- RTE_LCORE_FOREACH(lcore_id) {
- /* init power management library */
- ret = rte_power_init(lcore_id);
- if (ret) {
- RTE_LOG(ERR, L3FWD_POWER,
- "Library initialization failed on core %u\n",
- lcore_id);
- return ret;
- }
- /* we're not supporting the VM channel mode */
- env = rte_power_get_env();
- if (env != PM_ENV_ACPI_CPUFREQ &&
- env != PM_ENV_PSTATE_CPUFREQ &&
- env != PM_ENV_AMD_PSTATE_CPUFREQ &&
- env != PM_ENV_CPPC_CPUFREQ) {
- RTE_LOG(ERR, L3FWD_POWER,
- "Only ACPI and PSTATE mode are supported\n");
- return -1;
+ /* only legacy mode relies on the initialization of cpufreq library */
+ if (app_mode == APP_MODE_LEGACY) {
+ RTE_LCORE_FOREACH(lcore_id) {
+ /* init power management library */
+ ret = rte_power_init(lcore_id);
+ if (ret) {
+ RTE_LOG(ERR, L3FWD_POWER,
+ "Library initialization failed on core %u\n",
+ lcore_id);
+ return ret;
+ }
+ /* we're not supporting the VM channel mode */
+ env = rte_power_get_env();
+ if (env != PM_ENV_ACPI_CPUFREQ &&
+ env != PM_ENV_PSTATE_CPUFREQ &&
+ env != PM_ENV_AMD_PSTATE_CPUFREQ &&
+ env != PM_ENV_CPPC_CPUFREQ) {
+ RTE_LOG(ERR, L3FWD_POWER,
+ "Only ACPI and PSTATE mode are supported\n");
+ return -1;
+ }
}
}
- if (cpu_resume_latency != -1) {
+ if (app_mode == APP_MODE_LEGACY && cpu_resume_latency != -1) {
RTE_LCORE_FOREACH(lcore_id) {
/* Back old CPU resume latency. */
ret = rte_power_qos_get_cpu_resume_latency(lcore_id);
@@ -2329,14 +2332,16 @@ deinit_power_library(void)
unsigned int lcore_id, max_pkg, max_die, die, pkg;
int ret = 0;
- RTE_LCORE_FOREACH(lcore_id) {
- /* deinit power management library */
- ret = rte_power_exit(lcore_id);
- if (ret) {
- RTE_LOG(ERR, L3FWD_POWER,
- "Library deinitialization failed on core %u\n",
- lcore_id);
- return ret;
+ if (app_mode == APP_MODE_LEGACY) {
+ RTE_LCORE_FOREACH(lcore_id) {
+ /* deinit power management library */
+ ret = rte_power_exit(lcore_id);
+ if (ret) {
+ RTE_LOG(ERR, L3FWD_POWER,
+ "Library deinitialization failed on core %u\n",
+ lcore_id);
+ return ret;
+ }
}
}
@@ -2360,7 +2365,7 @@ deinit_power_library(void)
}
}
- if (cpu_resume_latency != -1) {
+ if (app_mode == APP_MODE_LEGACY && cpu_resume_latency != -1) {
RTE_LCORE_FOREACH(lcore_id) {
/* Restore the original value. */
rte_power_qos_set_cpu_resume_latency(lcore_id,
@@ -2602,8 +2607,7 @@ main(int argc, char **argv)
RTE_LOG(INFO, L3FWD_POWER, "Selected operation mode: %s\n",
mode_to_str(app_mode));
- /* only legacy mode relies on power library */
- if ((app_mode == APP_MODE_LEGACY) && init_power_library())
+ if (init_power_library())
rte_exit(EXIT_FAILURE, "init_power_library failed\n");
if (update_lcore_params() < 0)
@@ -2975,7 +2979,7 @@ main(int argc, char **argv)
rte_eth_dev_close(portid);
}
- if ((app_mode == APP_MODE_LEGACY) && deinit_power_library())
+ if (deinit_power_library())
rte_exit(EXIT_FAILURE, "deinit_power_library failed\n");
if (rte_eal_cleanup() < 0)
--
2.33.0
^ permalink raw reply related
* [PATCH v2 0/5] uncore power improvements and auto-detection
From: Huisong Li @ 2026-05-26 8:11 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
This series improves the uncore power management in l3fwd-power
and adds automatic detection of uncore drivers in the power library.
- Fix uncore deinitialization for non-legacy modes
- Enable power QoS for all modes (not just legacy)
- Fix uncore help text and log messages
- Relocate uncore initialization from arg parsing to init_power_library()
- Support automatic probing of multiple uncore drivers in AUTO_DETECT
---
v2:
- Remove the patch which add global uncore init and deinit interface.
- Remove the last patch in l3fwd-power about these new interface.
Will send them after this series.
- v1 link:
https://inbox.dpdk.org/dev/20260512023513.460169-1-lihuisong@huawei.com/
---
Huisong Li (5):
examples/l3fwd-power: fix uncore deinit for non-legacy
examples/l3fwd-power: enable power QoS for all modes
examples/l3fwd-power: fix uncore help and log info
examples/l3fwd-power: relocate uncore initialization
power: support automatic detection of uncore driver
doc/guides/rel_notes/release_26_07.rst | 6 +
.../sample_app_ug/l3_forward_power_man.rst | 2 +-
examples/l3fwd-power/main.c | 256 +++++++++---------
lib/power/rte_power_uncore.c | 46 +++-
4 files changed, 178 insertions(+), 132 deletions(-)
--
2.33.0
^ permalink raw reply
* [PATCH v2 2/5] examples/l3fwd-power: enable power QoS for all modes
From: Huisong Li @ 2026-05-26 8:11 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
In-Reply-To: <20260526081138.1434947-1-lihuisong@huawei.com>
Power QoS feature doesn't depend on app mode, here open it for all.
Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
examples/l3fwd-power/main.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 1122aeb930..ba2be5bf32 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -2295,7 +2295,7 @@ init_power_library(void)
}
}
- if (app_mode == APP_MODE_LEGACY && cpu_resume_latency != -1) {
+ if (cpu_resume_latency != -1) {
RTE_LCORE_FOREACH(lcore_id) {
/* Back old CPU resume latency. */
ret = rte_power_qos_get_cpu_resume_latency(lcore_id);
@@ -2365,7 +2365,7 @@ deinit_power_library(void)
}
}
- if (app_mode == APP_MODE_LEGACY && cpu_resume_latency != -1) {
+ if (cpu_resume_latency != -1) {
RTE_LCORE_FOREACH(lcore_id) {
/* Restore the original value. */
rte_power_qos_set_cpu_resume_latency(lcore_id,
--
2.33.0
^ permalink raw reply related
* [PATCH v2 3/5] examples/l3fwd-power: fix uncore help and log info
From: Huisong Li @ 2026-05-26 8:11 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
In-Reply-To: <20260526081138.1434947-1-lihuisong@huawei.com>
The '-i' parameter is to set any uncore frequency rather than
min/max value. And the error logs are not clear enough, fix it
by the way.
Fixes: 10db2a5b8724 ("examples/l3fwd-power: add options for uncore frequency")
Cc: stable@dpdk.org
Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
doc/guides/sample_app_ug/l3_forward_power_man.rst | 2 +-
examples/l3fwd-power/main.c | 8 ++++----
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/doc/guides/sample_app_ug/l3_forward_power_man.rst b/doc/guides/sample_app_ug/l3_forward_power_man.rst
index 3271bc2154..dc882c5d77 100644
--- a/doc/guides/sample_app_ug/l3_forward_power_man.rst
+++ b/doc/guides/sample_app_ug/l3_forward_power_man.rst
@@ -106,7 +106,7 @@ where,
* -U: optional, sets uncore min/max frequency to maximum value.
-* -i (frequency index): optional, sets uncore frequency to frequency index value, by setting min and max values to be the same.
+* -i (frequency index): set target frequency for uncore by specified frequency index.
* --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores.
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index ba2be5bf32..45b6697c85 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -1498,7 +1498,7 @@ print_usage(const char *prgname)
" -P: enable promiscuous mode\n"
" -u: set min/max frequency for uncore to minimum value\n"
" -U: set min/max frequency for uncore to maximum value\n"
- " -i (frequency index): set min/max frequency for uncore to specified frequency index\n"
+ " -i (frequency index): set target frequency for uncore by specified frequency index\n"
" --config (port,queue,lcore): rx queues configuration\n"
" --eth-link-speed: force link speed\n"
" --cpu-resume-latency LATENCY: set CPU resume latency to control C-state selection,"
@@ -1582,7 +1582,7 @@ parse_uncore_options(enum uncore_choice choice, const char *argument)
ret = rte_power_uncore_freq_min(pkg, die);
if (ret == -1) {
RTE_LOG(INFO, L3FWD_POWER,
- "Unable to set the uncore min/max to minimum uncore frequency value for pkg %02u die %02u\n"
+ "Unable to set the uncore frequency to minimum value for pkg %02u die %02u\n"
, pkg, die);
return ret;
}
@@ -1590,7 +1590,7 @@ parse_uncore_options(enum uncore_choice choice, const char *argument)
ret = rte_power_uncore_freq_max(pkg, die);
if (ret == -1) {
RTE_LOG(INFO, L3FWD_POWER,
- "Unable to set uncore min/max to maximum uncore frequency value for pkg %02u die %02u\n"
+ "Unable to set uncore frequency to maximum value for pkg %02u die %02u\n"
, pkg, die);
return ret;
}
@@ -1611,7 +1611,7 @@ parse_uncore_options(enum uncore_choice choice, const char *argument)
ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
if (ret == -1) {
RTE_LOG(INFO, L3FWD_POWER,
- "Unable to set min/max uncore index value for pkg %02u die %02u\n",
+ "Unable to set specified frequency index for pkg %02u die %02u\n",
pkg, die);
return ret;
}
--
2.33.0
^ permalink raw reply related
* [PATCH v2 4/5] examples/l3fwd-power: relocate uncore initialization
From: Huisong Li @ 2026-05-26 8:11 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
In-Reply-To: <20260526081138.1434947-1-lihuisong@huawei.com>
Currently, the deinitialization of uncore is in deinit_power_library().
But its initialization is located in the parameter parsing function,
which is not good to maintain.
So move this logic to init_power_library().
Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
examples/l3fwd-power/main.c | 192 ++++++++++++++++++------------------
1 file changed, 97 insertions(+), 95 deletions(-)
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 45b6697c85..a22634a04e 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -149,9 +149,6 @@ static struct rte_timer telemetry_timer;
/* stats index returned by metrics lib */
int telstats_index;
-/* flag to check if uncore option enabled */
-int enabled_uncore = -1;
-
struct telstats_name {
char name[RTE_ETH_XSTATS_NAME_SIZE];
};
@@ -170,12 +167,6 @@ enum busy_rate {
FULL = 100
};
-enum uncore_choice {
- UNCORE_MIN = 0,
- UNCORE_MAX = 1,
- UNCORE_IDX = 2
-};
-
/* reference poll count to measure core busyness */
#define DEFAULT_COUNT 10000
/*
@@ -212,6 +203,19 @@ enum freq_scale_hint_t
FREQ_HIGHEST = 2
};
+enum uncore_choice {
+ UNCORE_MIN = 0,
+ UNCORE_MAX = 1,
+ UNCORE_IDX = 2
+};
+
+/* flag to check if uncore option enabled */
+int enabled_uncore = -1;
+struct uncore_cfg {
+ enum uncore_choice uncore_choice;
+ uint32_t freq_idx;
+} g_uncore_cfg;
+
struct __rte_cache_aligned lcore_rx_queue {
uint16_t port_id;
uint16_t queue_id;
@@ -1552,80 +1556,6 @@ parse_uint(const char *opt, uint32_t max, uint32_t *res)
return 0;
}
-static int
-parse_uncore_options(enum uncore_choice choice, const char *argument)
-{
- unsigned int die, pkg, max_pkg, max_die;
- int ret = 0;
- ret = rte_power_set_uncore_env(RTE_UNCORE_PM_ENV_AUTO_DETECT);
- if (ret < 0) {
- RTE_LOG(INFO, L3FWD_POWER, "Failed to set uncore env\n");
- return ret;
- }
-
- max_pkg = rte_power_uncore_get_num_pkgs();
- if (max_pkg == 0)
- return -1;
-
- for (pkg = 0; pkg < max_pkg; pkg++) {
- max_die = rte_power_uncore_get_num_dies(pkg);
- if (max_die == 0)
- return -1;
- for (die = 0; die < max_die; die++) {
- ret = rte_power_uncore_init(pkg, die);
- if (ret == -1) {
- RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
- , pkg, die);
- return ret;
- }
- if (choice == UNCORE_MIN) {
- ret = rte_power_uncore_freq_min(pkg, die);
- if (ret == -1) {
- RTE_LOG(INFO, L3FWD_POWER,
- "Unable to set the uncore frequency to minimum value for pkg %02u die %02u\n"
- , pkg, die);
- return ret;
- }
- } else if (choice == UNCORE_MAX) {
- ret = rte_power_uncore_freq_max(pkg, die);
- if (ret == -1) {
- RTE_LOG(INFO, L3FWD_POWER,
- "Unable to set uncore frequency to maximum value for pkg %02u die %02u\n"
- , pkg, die);
- return ret;
- }
- } else if (choice == UNCORE_IDX) {
- char *ptr = NULL;
- int frequency_index = strtol(argument, &ptr, 10);
- if (argument == ptr) {
- RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
- return -1;
- }
- int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
- if (frequency_index > freq_array_len - 1) {
- RTE_LOG(INFO, L3FWD_POWER,
- "Frequency index given out of range, please choose a value from 0 to %d.\n",
- freq_array_len);
- return -1;
- }
- ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
- if (ret == -1) {
- RTE_LOG(INFO, L3FWD_POWER,
- "Unable to set specified frequency index for pkg %02u die %02u\n",
- pkg, die);
- return ret;
- }
- } else {
- RTE_LOG(INFO, L3FWD_POWER, "Uncore choice provided invalid\n");
- return -1;
- }
- }
- }
-
- RTE_LOG(INFO, L3FWD_POWER, "Successfully set max/min/index uncore frequency.\n");
- return ret;
-}
-
static int
parse_portmask(const char *portmask)
{
@@ -1793,25 +1723,21 @@ parse_args(int argc, char **argv)
promiscuous_on = 1;
break;
case 'u':
- enabled_uncore = parse_uncore_options(UNCORE_MIN, NULL);
- if (enabled_uncore < 0) {
- print_usage(prgname);
- return -1;
- }
+ enabled_uncore = 0;
+ g_uncore_cfg.uncore_choice = UNCORE_MIN;
break;
case 'U':
- enabled_uncore = parse_uncore_options(UNCORE_MAX, NULL);
- if (enabled_uncore < 0) {
- print_usage(prgname);
- return -1;
- }
+ enabled_uncore = 0;
+ g_uncore_cfg.uncore_choice = UNCORE_MAX;
break;
case 'i':
- enabled_uncore = parse_uncore_options(UNCORE_IDX, optarg);
- if (enabled_uncore < 0) {
+ enabled_uncore = 0;
+ if (parse_uint(optarg, UINT32_MAX, &g_uncore_cfg.freq_idx) != 0) {
+ RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
print_usage(prgname);
return -1;
}
+ g_uncore_cfg.uncore_choice = UNCORE_IDX;
break;
/* long options */
case 0:
@@ -2264,6 +2190,78 @@ static int check_ptype(uint16_t portid)
}
+static int
+power_uncore_init(void)
+{
+ unsigned int die, pkg, max_pkg, max_die;
+ int ret;
+
+ if (enabled_uncore == -1)
+ return 0;
+
+ ret = rte_power_set_uncore_env(RTE_UNCORE_PM_ENV_AUTO_DETECT);
+ if (ret < 0) {
+ RTE_LOG(INFO, L3FWD_POWER, "Failed to set uncore env\n");
+ return ret;
+ }
+
+ max_pkg = rte_power_uncore_get_num_pkgs();
+ if (max_pkg == 0)
+ return -1;
+
+ for (pkg = 0; pkg < max_pkg; pkg++) {
+ max_die = rte_power_uncore_get_num_dies(pkg);
+ if (max_die == 0)
+ return -1;
+ for (die = 0; die < max_die; die++) {
+ ret = rte_power_uncore_init(pkg, die);
+ if (ret == -1) {
+ RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
+ , pkg, die);
+ return ret;
+ }
+ if (g_uncore_cfg.uncore_choice == UNCORE_MIN) {
+ ret = rte_power_uncore_freq_min(pkg, die);
+ if (ret == -1) {
+ RTE_LOG(INFO, L3FWD_POWER,
+ "Unable to set the uncore frequency to minimum value for pkg %02u die %02u\n"
+ , pkg, die);
+ return ret;
+ }
+ } else if (g_uncore_cfg.uncore_choice == UNCORE_MAX) {
+ ret = rte_power_uncore_freq_max(pkg, die);
+ if (ret == -1) {
+ RTE_LOG(INFO, L3FWD_POWER,
+ "Unable to set uncore frequency to maximum value for pkg %02u die %02u\n"
+ , pkg, die);
+ return ret;
+ }
+ } else if (g_uncore_cfg.uncore_choice == UNCORE_IDX) {
+ int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
+ if (freq_array_len <= 0) {
+ RTE_LOG(INFO, L3FWD_POWER, "Get uncore frequency number failed.\n");
+ return -1;
+ }
+ if (g_uncore_cfg.freq_idx > (uint32_t)(freq_array_len - 1)) {
+ RTE_LOG(INFO, L3FWD_POWER,
+ "Frequency index given out of range, please choose a value from 0 to %d.\n",
+ freq_array_len);
+ return -1;
+ }
+ ret = rte_power_set_uncore_freq(pkg, die, g_uncore_cfg.freq_idx);
+ if (ret == -1) {
+ RTE_LOG(INFO, L3FWD_POWER,
+ "Unable to set specified frequency index for pkg %02u die %02u\n",
+ pkg, die);
+ return ret;
+ }
+ }
+ }
+ }
+
+ return 0;
+}
+
static int
init_power_library(void)
{
@@ -2295,6 +2293,10 @@ init_power_library(void)
}
}
+ ret = power_uncore_init();
+ if (ret != 0)
+ return ret;
+
if (cpu_resume_latency != -1) {
RTE_LCORE_FOREACH(lcore_id) {
/* Back old CPU resume latency. */
--
2.33.0
^ permalink raw reply related
* [PATCH v2 5/5] power: support automatic detection of uncore driver
From: Huisong Li @ 2026-05-26 8:11 UTC (permalink / raw)
To: anatoly.burakov, sivaprasad.tummala
Cc: dev, thomas, stephen, fengchengwen, yangxingui, zhanjie9,
lihuisong
In-Reply-To: <20260526081138.1434947-1-lihuisong@huawei.com>
Currently, uncore driver library just supports intel_uncore driver
as default driver when user use AUTO_DETECT, which is not good to
application. So it is necessary to support probing for multi-uncore
drivers.
Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
doc/guides/rel_notes/release_26_07.rst | 6 ++++
lib/power/rte_power_uncore.c | 46 ++++++++++++++++++++++----
2 files changed, 46 insertions(+), 6 deletions(-)
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index f012d47a4b..14436f9d7f 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -63,6 +63,12 @@ New Features
``rte_eal_init`` and the application is responsible for probing each device,
* ``--auto-probing`` enables the initial bus probing, which is the current default behavior.
+* **Supported auto-detection of uncore power driver.**
+
+ The uncore power library now supports automatic probing of multiple
+ uncore drivers when using ``RTE_UNCORE_PM_ENV_AUTO_DETECT``,
+ instead of defaulting only to the Intel uncore driver.
+
Removed Items
-------------
diff --git a/lib/power/rte_power_uncore.c b/lib/power/rte_power_uncore.c
index 25bdb113c5..ff11f0105c 100644
--- a/lib/power/rte_power_uncore.c
+++ b/lib/power/rte_power_uncore.c
@@ -2,6 +2,7 @@
* Copyright(c) 2010-2014 Intel Corporation
* Copyright(c) 2023 AMD Corporation
*/
+#include <errno.h>
#include <eal_export.h>
#include <rte_spinlock.h>
@@ -46,6 +47,39 @@ rte_power_register_uncore_ops(struct rte_power_uncore_ops *driver_ops)
return 0;
}
+static uint32_t power_uncore_driver_name2env(char *name)
+{
+ for (uint32_t i = 0; i < RTE_DIM(uncore_env_str); i++) {
+ if (!strcmp(name, uncore_env_str[i]))
+ return i;
+ }
+
+ return UINT32_MAX;
+}
+
+static int power_uncore_probe_driver(void)
+{
+ struct rte_power_uncore_ops *ops;
+ int ret;
+
+ global_uncore_ops = NULL;
+ /* Use package-0 and die-0 to probe uncore driver. */
+ RTE_TAILQ_FOREACH(ops, &uncore_ops_list, next) {
+ ret = ops->init(0, 0);
+ if (ret == 0) {
+ uint32_t env = power_uncore_driver_name2env(ops->name);
+ if (env == UINT32_MAX)
+ continue;
+ global_uncore_env = env;
+ global_uncore_ops = ops;
+ ops->exit(0, 0);
+ break;
+ }
+ }
+
+ return global_uncore_ops ? 0 : -ENODEV;
+}
+
RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_power_set_uncore_env, 23.11)
int
rte_power_set_uncore_env(enum rte_uncore_power_mgmt_env env)
@@ -60,12 +94,12 @@ rte_power_set_uncore_env(enum rte_uncore_power_mgmt_env env)
goto out;
}
- if (env == RTE_UNCORE_PM_ENV_AUTO_DETECT)
- /* Currently only intel_uncore is supported.
- * This will be extended with auto-detection support
- * for multiple uncore implementations.
- */
- env = RTE_UNCORE_PM_ENV_INTEL_UNCORE;
+ if (env == RTE_UNCORE_PM_ENV_AUTO_DETECT) {
+ ret = power_uncore_probe_driver();
+ if (ret != 0)
+ POWER_LOG(ERR, "Probe uncore driver failed, ret = %d", ret);
+ goto out;
+ }
if (env <= RTE_DIM(uncore_env_str)) {
RTE_TAILQ_FOREACH(ops, &uncore_ops_list, next)
--
2.33.0
^ permalink raw reply related
* RE: [PATCH v5] mempool: improve cache behaviour and performance
From: Morten Brørup @ 2026-05-26 8:41 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, Andrew Rybchenko, Jingjing Wu, Praveen Shetty,
Hemant Agrawal, Sachin Saxena
In-Reply-To: <ahCAPT1LEn_Rc7Pk@bricha3-mobl1.ger.corp.intel.com>
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 22 May 2026 18.12
>
> On Sun, Apr 19, 2026 at 09:55:26AM +0000, Morten Brørup wrote:
> > This patch refactors the mempool cache to eliminate some unexpected
> > behaviour and reduce the mempool cache miss rate.
> >
>
> Agree in principle with most of these changes. As we dicussed at the
> DPDK
> summit conference, only issue I really have is with the threshold
> limits
> here - allocating and freeing only half the cache at a time seems
> overly
> conservative. Thinking about use-cases:
>
> 1 for apps where alloc + free (generally Rx+Tx) is on the same core(s),
> then we should run (almost) entirely out of cache.
I strongly disagree about any goal to run the cache low.
The primary goal is to minimize the cache miss (refill and replenish) rate.
> 2 for apps where we have alloc and free on different cores, then we
> have
> some caches always being filled and others always being emptied
Agree.
>
> For case #1, we only need worry about the thresholds for the odd case
> where we have a burst that causes us to overflow our cache (and we
> can't increase our cache size to cope and avoid that).
> Otherwise the thresholds don't matter.
It seems like you assume the application only does something like this:
Rx -> Rewrite -> Tx
In that case, the per-lcore cache only needs capacity for one burst, yes.
With my patch, the cache can be rightsized by requesting a cache size of 2 * burst size.
(Then the fill level will be either size/2 or empty, i.e. one burst or zero.
This also happens to meet your suggested goal about low fill level, which I disagree with.)
However, I don't think that is a realistic use case.
Many apps do something like this:
|-> Rewrite ->|
Rx ->| |-> Tx
|-> Hold |
Release ->|
They often hold back packets before they are transmitted.
For a simple router, when the destination IP address is not in the neighbor table, packets to that IP address are queued until ARP/ND has been resolved, and then they are dequeued and transmitted.
Or apps performing shaping or pacing, where packets are held back in queues, and dequeued at the appropriate time.
For such apps, the waves are much bigger (than the simple Rx->Rewrite->Tx use case).
With a random enqueue/dequeue pattern, replenishing/draining the cache to size/2 minimizes the probability of reaching one of its edges (empty or full), triggering a "cache miss" (refill/replenish).
> However, for case #2, the thresholds are constantly involved as
> we
> always are going to backing store. In this case, we really want to have
> the
> allocs *always* fill the cache completely, and the frees completely
> empty
> the cache.
Agree.
>
> Because of this, while we want to avoid cases where we fill the cache
> completely only to have a further free causing it to be flushed,
> because of
> case #2 we cannot be overly conservative in how much we free/empty.
> Ideally, we want to fill to full less a single burst, and empty leaving
> only a single burst in the cache. Unfortunately, we don't know what
> those
> burst limits are, so we have to try and guess the best behaviour from
> everything else.
I agree about not wanting to be overly conservative.
But in the use cases I have described for #1, I don't think a target fill level of size/2 is overly conservative.
I also acknowledge that this patch doubles the mempool cache miss rate for #2.
E.g. with a cache size of 512 and burst size of 64, the per-burst miss rate will be 64 / (1/2 * 512) = 1/4, compared to 64 / 512 = 1/8 with a full replenish/drain algorithm.
In theory, we could make it build time configurable to optimize mempools for #2.
But mempools are also used for other objects than mbufs, so that would have unwanted side effects for non-mbuf mempools.
If we went for an algorithm targeting replenish/drain at 25 % from the edges, the per-burst miss rate for #2 would be: 64 / (3/4 * 512) = 1/6.
How about addressing #2 in the release notes:
We describe that the cache refill/drain algorithm has been changed to only refill/drain to 50 % of the cache size, so pipelined applications performing Rx (mempool get) and Tx (mempool put) on separate cores should configure their mbuf pools with double the cache size of what they previously were to achieve similar performance.
>
> All that said, commits with specific suggestions inline.
>
> /Bruce
>
> <snip>
>
> > diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
> > index 2e54fc4466..432c43ab15 100644
> > --- a/lib/mempool/rte_mempool.h
> > +++ b/lib/mempool/rte_mempool.h
> > @@ -89,7 +89,7 @@ struct __rte_cache_aligned rte_mempool_debug_stats
> {
> > */
> > struct __rte_cache_aligned rte_mempool_cache {
> > uint32_t size; /**< Size of the cache */
> > - uint32_t flushthresh; /**< Threshold before we flush excess
> elements */
> > + uint32_t flushthresh; /**< Obsolete; for API/ABI compatibility
> purposes only */
> > uint32_t len; /**< Current cache count */
> > #ifdef RTE_LIBRTE_MEMPOOL_STATS
> > uint32_t unused;
> > @@ -107,8 +107,10 @@ struct __rte_cache_aligned rte_mempool_cache {
> > /**
> > * Cache objects
> > *
> > - * Cache is allocated to this size to allow it to overflow in
> certain
> > - * cases to avoid needless emptying of cache.
> > + * Note:
> > + * Cache is allocated at double size for API/ABI compatibility
> purposes only.
> > + * When reducing its size at an API/ABI breaking release,
> > + * remember to add a cache guard after it.
> > */
> > alignas(RTE_CACHE_LINE_SIZE) void
> *objs[RTE_MEMPOOL_CACHE_MAX_SIZE * 2];
> > };
> > @@ -1046,12 +1048,17 @@ rte_mempool_free(struct rte_mempool *mp);
> > * @param cache_size
> > * If cache_size is non-zero, the rte_mempool library will try to
> > * limit the accesses to the common lockless pool, by maintaining
> a
> > - * per-lcore object cache. This argument must be lower or equal to
> > - * RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.
> > + * per-lcore object cache. This argument must be an even number,
> > + * lower or equal to RTE_MEMPOOL_CACHE_MAX_SIZE and n.
> > * The access to the per-lcore table is of course
> > * faster than the multi-producer/consumer pool. The cache can be
> > * disabled if the cache_size argument is set to 0; it can be
> useful to
> > * avoid losing objects in cache.
> > + * Note:
> > + * Mempool put/get requests of more than cache_size / 2 objects
> may be
> > + * partially or fully served directly by the multi-
> producer/consumer
> > + * pool, to avoid the overhead of copying the objects twice
> (instead of
> > + * once) when using the cache as a bounce buffer.
> > * @param private_data_size
> > * The size of the private data appended after the mempool
> > * structure. This is useful for storing some private data after
> the
> > @@ -1390,24 +1397,32 @@ rte_mempool_do_generic_put(struct rte_mempool
> *mp, void * const *obj_table,
> > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
> > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
> >
> > - __rte_assume(cache->flushthresh <= RTE_MEMPOOL_CACHE_MAX_SIZE *
> 2);
> > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > - __rte_assume(cache->len <= cache->flushthresh);
> > - if (likely(cache->len + n <= cache->flushthresh)) {
> > + __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > + __rte_assume(cache->len <= cache->size);
> > + if (likely(cache->len + n <= cache->size)) {
> > /* Sufficient room in the cache for the objects. */
> > cache_objs = &cache->objs[cache->len];
> > cache->len += n;
> > - } else if (n <= cache->flushthresh) {
> > + } else if (n <= cache->size / 2) {
> > /*
> > - * The cache is big enough for the objects, but - as
> detected by
> > - * the comparison above - has insufficient room for them.
> > - * Flush the cache to make room for the objects.
> > + * The number of objects is within the cache bounce buffer
> limit,
> > + * but - as detected by the comparison above - the cache
> has
> > + * insufficient room for them.
> > + * Flush the cache to the backend to make room for the
> objects;
> > + * flush (size / 2) objects from the bottom of the cache,
> where
> > + * objects are less hot, and move down the remaining
> objects, which
> > + * are more hot, from the upper half of the cache.
> > */
> > - cache_objs = &cache->objs[0];
> > - rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
> > - cache->len = n;
> > + __rte_assume(cache->len > cache->size / 2);
> > + rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0], cache-
> >size / 2);
> > + rte_memcpy(&cache->objs[0], &cache->objs[cache->size / 2],
> > + sizeof(void *) * (cache->len - cache->size /
> 2));
> > + cache_objs = &cache->objs[cache->len - cache->size / 2];
> > + cache->len = cache->len - cache->size / 2 + n;
>
> The flushing of only half the cache I'm not so certain about. I agree
> that
> we want to not flush to empty, but I also think that we want to do more
> than a half-flush, especially since we do an enqueue to the cache
> immediately afterwards. Consider the case where we have a cache size of
> 128, and we do an enqueue of 32, with the cache currently full. In that
> case we only flush 64, reducing the cache to 64, but then immediately
> bringing it back up to 96.
I thought in depth about whether the flush/replenish sizes should consider the request size or not. (E.g. if I should replenish size/2 or size/2+request.)
I decided for not considering the request size, for two reasons:
a) It roughly doesn't matter, especially when considering a sequence of random get/put requests.
b) The size of the backend transactions become fixed, which has performance benefits: With my patch, they are always size/2, so if the cache size is 2^N, the backend transactions are 2^N and CPU cache aligned.
> For cases where we have pipelines with all
> alloc
> on one core and all free on another, this half-flush would be
> inefficient.
>
> Instead, I would look to have a lower target threshold post-flush, and
> I
> would suggest 25% - taking into account the newly freed buffers.
It's not good for #1.
I agree that it is better for #2. But I don't think #2 is the likely use case.
After our discussion at the summit, I did start working a patch targeting fill levels at 25% from the cache edges, but I don't think it's better; so I'd rather stick with a target fill level of 50%.
> For example:
>
> /* if n > our target of 1/4 full, flush everything,
> * else flush so that we end up with 1/4 full after n added.
> */
> flush_count = n > cache->size/4 ? cache->len :
> (cache->len + n) - cache->size/4;
>
>
> > } else {
> > - /* The request itself is too big for the cache. */
> > + /* The request itself is too big. */
> > goto driver_enqueue_stats_incremented;
>
> I think original comment is better. The request itself is not too big
> for
> the whole mempool, just for the cache.
Ack.
>
> > }
> >
> > @@ -1524,7 +1539,7 @@ rte_mempool_do_generic_get(struct rte_mempool
> *mp, void **obj_table,
> > /* The cache is a stack, so copy will be in reverse order. */
> > cache_objs = &cache->objs[cache->len];
> >
> > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > if (likely(n <= cache->len)) {
> > /* The entire request can be satisfied from the cache. */
> > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
> > @@ -1548,13 +1563,13 @@ rte_mempool_do_generic_get(struct rte_mempool
> *mp, void **obj_table,
> > for (index = 0; index < len; index++)
> > *obj_table++ = *--cache_objs;
> >
> > - /* Dequeue below would overflow mem allocated for cache? */
> > - if (unlikely(remaining > RTE_MEMPOOL_CACHE_MAX_SIZE))
> > + /* Dequeue below would exceed the cache bounce buffer limit? */
> > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> > + if (unlikely(remaining > cache->size / 2))
> > goto driver_dequeue;
> >
> > - /* Fill the cache from the backend; fetch size + remaining
> objects. */
> > - ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs,
> > - cache->size + remaining);
> > + /* Fill the cache from the backend; fetch (size / 2) objects. */
> > + ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache->size /
> 2);
>
> Again, the cache->size / 2 doesn't seem right here. We at most half-
> fill
> the cache and then take some objects from that, meaning that have just
> done
> a re-fill of cache but end the function with it less than half full.
> Since
> we take from this value, I'd suggest just filling the cache completely.
The issues at the edges of the cache are symmetrical.
If we replenish the cache to full, and the next transaction is a put, the cache needs to be drained.
That's why I replenish to size/2.
>
> > if (unlikely(ret < 0)) {
> > /*
> > * We are buffer constrained, and not able to fetch all
> that.
> > @@ -1568,10 +1583,11 @@ rte_mempool_do_generic_get(struct rte_mempool
> *mp, void **obj_table,
> > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
> > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
> >
> > - __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > - __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > - cache_objs = &cache->objs[cache->size + remaining];
> > - cache->len = cache->size;
> > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> > + __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> > + __rte_assume(remaining <= cache->size / 2);
> > + cache_objs = &cache->objs[cache->size / 2];
> > + cache->len = cache->size / 2 - remaining;
> > for (index = 0; index < remaining; index++)
> > *obj_table++ = *--cache_objs;
> >
> > --
> > 2.43.0
> >
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox