* Re: [PATCH v2 3/6] eal/memory: allocate all VA space in one go
From: Bruce Richardson @ 2026-05-26 15:25 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev
In-Reply-To: <23100128428096f0179a8c64904449a95311bca6.1773417833.git.anatoly.burakov@intel.com>
On Fri, Mar 13, 2026 at 04:06:34PM +0000, Anatoly Burakov wrote:
> Instead of allocating VA space per memseg list in dynmem mode, allocate it
> all in one go, and then assign memseg lists portions of that space. In a
> similar way, for dynmem initialization in secondary processes, also attach
> all VA space in one go. Legacy/32-bit paths are untouched.
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
As with previous patch, stylistically there are a few places where lines
are split unncessarily.
^ permalink raw reply
* Re: [PATCH v2 2/6] eal/memory: remove per-list segment and memory limits
From: Bruce Richardson @ 2026-05-26 14:57 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev
In-Reply-To: <c5fa90b0700411d63652a5fae547996699da4b0b.1773417833.git.anatoly.burakov@intel.com>
On Fri, Mar 13, 2026 at 04:06:33PM +0000, Anatoly Burakov wrote:
> Initially, the dynamic memory mode has used multiple segment lists for
> backing of different memory types, with the motivation being that it should
> be easier for secondary processes to map many smaller segments than fewer
> but larger ones, but in practice this does not seem to make any difference
> for 64-bit platforms, as there's usually plenty of address space.
>
I think it's worth clarifying here that each list corresponds to a VA
address space block. The term segment is also a little confusing here,
since the EAL memseg used in segment lists is now a page IIRC, but the
"smaller segments" for secondaries are blocks of VA space, right?
> To reduce the amount of complexity in how memory segment lists are handled,
> collapse the multi-list logic to always use single segment list.
>
> That does not mean that all memory types will always get one segment - in
> some cases (e.g. 32-bit) we may not be able to allocate enough contiguous
> VA spaces to fit entire memory type into one list, in which case the number
> of memseg lists for that type will be more than one. It is more about
> lifting the upper limit on how many segment lists can a type have. If we
> end up blowing up our number of segment lists so much that we exceed a
> very generous default maximum memseg lists number then the user has bigger
> problems to address.
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Some comments inline below. Nothing major, just some style suggestions.
> ---
> config/rte_config.h | 2 -
> .../prog_guide/env_abstraction_layer.rst | 4 -
> lib/eal/common/eal_common_dynmem.c | 110 +++++-------------
> lib/eal/common/eal_common_memory.c | 6 +-
> lib/eal/common/eal_filesystem.h | 13 +++
> lib/eal/common/eal_private.h | 6 +-
> lib/eal/freebsd/eal_memory.c | 75 ++++--------
> lib/eal/linux/eal_memalloc.c | 4 +-
> lib/eal/linux/eal_memory.c | 88 ++++++--------
> 9 files changed, 107 insertions(+), 201 deletions(-)
>
> diff --git a/config/rte_config.h b/config/rte_config.h
> index a2609fa403..0447cdf2ad 100644
> --- a/config/rte_config.h
> +++ b/config/rte_config.h
> @@ -43,8 +43,6 @@
> #define RTE_MAX_HEAPS 32
> #define RTE_MAX_LCORE_VAR 131072
> #define RTE_MAX_MEMSEG_LISTS 128
> -#define RTE_MAX_MEMSEG_PER_LIST 8192
> -#define RTE_MAX_MEM_MB_PER_LIST 32768
> #define RTE_MAX_MEMSEG_PER_TYPE 32768
> #define RTE_MAX_MEM_MB_PER_TYPE 65536
> #define RTE_MAX_TAILQ 32
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index d716895c1d..04368a3950 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -204,10 +204,6 @@ of virtual memory being preallocated at startup by editing the following config
> variables:
>
> * ``RTE_MAX_MEMSEG_LISTS`` controls how many segment lists can DPDK have
> -* ``RTE_MAX_MEM_MB_PER_LIST`` controls how much megabytes of memory each
> - segment list can address
> -* ``RTE_MAX_MEMSEG_PER_LIST`` controls how many segments each segment list
> - can have
> * ``RTE_MAX_MEMSEG_PER_TYPE`` controls how many segments each memory type
> can have (where "type" is defined as "page size + NUMA node" combination)
> * ``RTE_MAX_MEM_MB_PER_TYPE`` controls how much megabytes of memory each
> diff --git a/lib/eal/common/eal_common_dynmem.c b/lib/eal/common/eal_common_dynmem.c
> index 8f51d6dd4a..ef0270cc30 100644
> --- a/lib/eal/common/eal_common_dynmem.c
> +++ b/lib/eal/common/eal_common_dynmem.c
> @@ -24,11 +24,10 @@ eal_dynmem_memseg_lists_init(void)
> struct memtype {
> uint64_t page_sz;
> int socket_id;
> - } *memtypes = NULL;
> + } memtypes[RTE_MAX_MEMSEG_LISTS] = {0};
> int i, hpi_idx, msl_idx, ret = -1; /* fail unless told to succeed */
> struct rte_memseg_list *msl;
> uint64_t max_mem, max_mem_per_type;
> - unsigned int max_seglists_per_type;
> unsigned int n_memtypes, cur_type;
> struct internal_config *internal_conf =
> eal_get_internal_configuration();
> @@ -45,8 +44,7 @@ eal_dynmem_memseg_lists_init(void)
> *
> * deciding amount of memory going towards each memory type is a
> * balancing act between maximum segments per type, maximum memory per
> - * type, and number of detected NUMA nodes. the goal is to make sure
> - * each memory type gets at least one memseg list.
> + * type, and number of detected NUMA nodes.
> *
> * the total amount of memory is limited by RTE_MAX_MEM_MB value.
> *
> @@ -57,26 +55,18 @@ eal_dynmem_memseg_lists_init(void)
> * smaller page sizes, it can take hundreds of thousands of segments to
> * reach the above specified per-type memory limits.
> *
> - * additionally, each type may have multiple memseg lists associated
> - * with it, each limited by either RTE_MAX_MEM_MB_PER_LIST for bigger
> - * page sizes, or RTE_MAX_MEMSEG_PER_LIST segments for smaller ones.
> - *
> - * the number of memseg lists per type is decided based on the above
> - * limits, and also taking number of detected NUMA nodes, to make sure
> - * that we don't run out of memseg lists before we populate all NUMA
> - * nodes with memory.
> - *
> - * we do this in three stages. first, we collect the number of types.
> - * then, we figure out memory constraints and populate the list of
> - * would-be memseg lists. then, we go ahead and allocate the memseg
> - * lists.
> + * each memory type is allotted a single memseg list. the size of that
> + * list is calculated here to respect the per-type memory and segment
> + * limits that apply.
> */
>
> - /* create space for mem types */
> + /* maximum number of memtypes we're ever going to get */
> n_memtypes = internal_conf->num_hugepage_sizes * rte_socket_count();
> - memtypes = calloc(n_memtypes, sizeof(*memtypes));
> - if (memtypes == NULL) {
> - EAL_LOG(ERR, "Cannot allocate space for memory types");
> +
> + /* can we fit all memtypes into the memseg lists? */
> + if (n_memtypes > RTE_MAX_MEMSEG_LISTS) {
> + EAL_LOG(ERR, "Too many memory types detected: %u. Please increase "
> + "RTE_MAX_MEMSEG_LISTS in configuration.", n_memtypes);
> return -1;
Don't split log messages across lines. Go over the 100 char limit if
necessary to avoid splits.
> }
>
> @@ -113,91 +103,49 @@ eal_dynmem_memseg_lists_init(void)
> max_mem = (uint64_t)RTE_MAX_MEM_MB << 20;
> max_mem_per_type = RTE_MIN((uint64_t)RTE_MAX_MEM_MB_PER_TYPE << 20,
> max_mem / n_memtypes);
> - /*
> - * limit maximum number of segment lists per type to ensure there's
> - * space for memseg lists for all NUMA nodes with all page sizes
> - */
> - max_seglists_per_type = RTE_MAX_MEMSEG_LISTS / n_memtypes;
> -
> - if (max_seglists_per_type == 0) {
> - EAL_LOG(ERR, "Cannot accommodate all memory types, please increase RTE_MAX_MEMSEG_LISTS");
> - goto out;
> - }
>
> /* go through all mem types and create segment lists */
> msl_idx = 0;
> for (cur_type = 0; cur_type < n_memtypes; cur_type++) {
> - unsigned int cur_seglist, n_seglists, n_segs;
> - unsigned int max_segs_per_type, max_segs_per_list;
> + unsigned int n_segs;
> struct memtype *type = &memtypes[cur_type];
> - uint64_t max_mem_per_list, pagesz;
> + uint64_t pagesz;
> int socket_id;
>
> pagesz = type->page_sz;
> socket_id = type->socket_id;
>
> /*
> - * we need to create segment lists for this type. we must take
> + * we need to create a segment list for this type. we must take
> * into account the following things:
> *
> - * 1. total amount of memory we can use for this memory type
> - * 2. total amount of memory per memseg list allowed
> + * 1. total amount of memory to use for this memory type
> + * 2. total amount of memory allowed per type
> * 3. number of segments needed to fit the amount of memory
> * 4. number of segments allowed per type
> - * 5. number of segments allowed per memseg list
> - * 6. number of memseg lists we are allowed to take up
> */
> + n_segs = max_mem_per_type / pagesz;
> + n_segs = RTE_MIN(n_segs, (unsigned int)RTE_MAX_MEMSEG_PER_TYPE);
>
> - /* calculate how much segments we will need in total */
> - max_segs_per_type = max_mem_per_type / pagesz;
> - /* limit number of segments to maximum allowed per type */
> - max_segs_per_type = RTE_MIN(max_segs_per_type,
> - (unsigned int)RTE_MAX_MEMSEG_PER_TYPE);
> - /* limit number of segments to maximum allowed per list */
> - max_segs_per_list = RTE_MIN(max_segs_per_type,
> - (unsigned int)RTE_MAX_MEMSEG_PER_LIST);
> + EAL_LOG(DEBUG, "Creating segment list: "
> + "n_segs:%u socket_id:%i hugepage_sz:%" PRIu64,
> + n_segs, socket_id, pagesz);
>
Again, better not to split this message.
> - /* calculate how much memory we can have per segment list */
> - max_mem_per_list = RTE_MIN(max_segs_per_list * pagesz,
> - (uint64_t)RTE_MAX_MEM_MB_PER_LIST << 20);
> + msl = &mcfg->memsegs[msl_idx];
>
> - /* calculate how many segments each segment list will have */
> - n_segs = RTE_MIN(max_segs_per_list, max_mem_per_list / pagesz);
> + if (eal_memseg_list_init(msl, pagesz, n_segs, socket_id,
> + msl_idx, true))
This can actually fit on one line.
> + goto out;
>
> - /* calculate how many segment lists we can have */
> - n_seglists = RTE_MIN(max_segs_per_type / n_segs,
> - max_mem_per_type / max_mem_per_list);
> -
> - /* limit number of segment lists according to our maximum */
> - n_seglists = RTE_MIN(n_seglists, max_seglists_per_type);
> -
> - EAL_LOG(DEBUG, "Creating %i segment lists: "
> - "n_segs:%i socket_id:%i hugepage_sz:%" PRIu64,
> - n_seglists, n_segs, socket_id, pagesz);
> -
> - /* create all segment lists */
> - for (cur_seglist = 0; cur_seglist < n_seglists; cur_seglist++) {
> - if (msl_idx >= RTE_MAX_MEMSEG_LISTS) {
> - EAL_LOG(ERR,
> - "No more space in memseg lists, please increase RTE_MAX_MEMSEG_LISTS");
> - goto out;
> - }
> - msl = &mcfg->memsegs[msl_idx++];
> -
> - if (eal_memseg_list_init(msl, pagesz, n_segs,
> - socket_id, cur_seglist, true))
> - goto out;
> -
> - if (eal_memseg_list_alloc(msl, 0)) {
> - EAL_LOG(ERR, "Cannot allocate VA space for memseg list");
> - goto out;
> - }
> + if (eal_memseg_list_alloc(msl, 0)) {
> + EAL_LOG(ERR, "Cannot allocate VA space for memseg list");
> + goto out;
> }
> + msl_idx++;
> }
> /* we're successful */
> ret = 0;
> out:
> - free(memtypes);
> return ret;
> }
>
> diff --git a/lib/eal/common/eal_common_memory.c b/lib/eal/common/eal_common_memory.c
> index dccf9406c5..b9388021ff 100644
> --- a/lib/eal/common/eal_common_memory.c
> +++ b/lib/eal/common/eal_common_memory.c
> @@ -228,12 +228,12 @@ eal_memseg_list_init_named(struct rte_memseg_list *msl, const char *name,
>
> int
> eal_memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
> - int n_segs, int socket_id, int type_msl_idx, bool heap)
> + int n_segs, int socket_id, int msl_idx, bool heap)
> {
> char name[RTE_FBARRAY_NAME_LEN];
>
> - snprintf(name, sizeof(name), MEMSEG_LIST_FMT, page_sz >> 10, socket_id,
> - type_msl_idx);
> + snprintf(name, sizeof(name), MEMSEG_LIST_FMT,
> + page_sz >> 10, socket_id, msl_idx);
>
Fits on single line.
> return eal_memseg_list_init_named(
> msl, name, page_sz, n_segs, socket_id, heap);
> diff --git a/lib/eal/common/eal_filesystem.h b/lib/eal/common/eal_filesystem.h
> index 6b99d22160..2d22b52e76 100644
> --- a/lib/eal/common/eal_filesystem.h
> +++ b/lib/eal/common/eal_filesystem.h
> @@ -114,6 +114,19 @@ eal_get_hugefile_path(char *buffer, size_t buflen, const char *hugedir, int f_id
> return buffer;
> }
>
> +#define HUGEFILE_FMT_LIST_SEG "%s/%smap_%u_%u"
> +static inline __rte_warn_unused_result const char *
> +eal_get_hugefile_list_seg_path(char *buffer, size_t buflen,
> + const char *hugedir, unsigned int list_idx, unsigned int seg_idx)
> +{
> + if (snprintf(buffer, buflen, HUGEFILE_FMT_LIST_SEG,
> + hugedir, eal_get_hugefile_prefix(), list_idx, seg_idx)
> + >= (int)buflen)
Can fit in two lines not three, e.g. move up the ">= (int)buflen".
> + return NULL;
> + else
> + return buffer;
While I know it's a copy of the previous function in the file, the "else"
is really unnecessary.
> +}
> +
> /** define the default filename prefix for the %s values above */
> #define HUGEFILE_PREFIX_DEFAULT "rte"
>
> diff --git a/lib/eal/common/eal_private.h b/lib/eal/common/eal_private.h
> index e032dd10c9..70f7b46699 100644
> --- a/lib/eal/common/eal_private.h
> +++ b/lib/eal/common/eal_private.h
> @@ -299,14 +299,14 @@ eal_memseg_list_init_named(struct rte_memseg_list *msl, const char *name,
> * Initialize memory segment list and create its backing storage
> * with a name corresponding to MSL parameters.
> *
> - * @param type_msl_idx
> - * Index of the MSL among other MSLs of the same socket and page size.
> + * @param msl_idx
> + * Index of the MSL in memsegs array.
> *
> * @see eal_memseg_list_init_named for remaining parameters description.
> */
> int
> eal_memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
> - int n_segs, int socket_id, int type_msl_idx, bool heap);
> + int n_segs, int socket_id, int msl_idx, bool heap);
>
> /**
> * Reserve VA space for a memory segment list
> diff --git a/lib/eal/freebsd/eal_memory.c b/lib/eal/freebsd/eal_memory.c
> index cd608db9f9..3eb5d193ec 100644
> --- a/lib/eal/freebsd/eal_memory.c
> +++ b/lib/eal/freebsd/eal_memory.c
> @@ -190,8 +190,8 @@ rte_eal_hugepage_init(void)
> break;
> }
> if (msl_idx == RTE_MAX_MEMSEG_LISTS) {
> - EAL_LOG(ERR, "Could not find space for memseg. Please increase RTE_MAX_MEMSEG_PER_LIST "
> - "RTE_MAX_MEMSEG_PER_TYPE and/or RTE_MAX_MEM_MB_PER_TYPE in configuration.");
> + EAL_LOG(ERR,
> + "Could not find suitable space for memseg in existing memseg lists");
> return -1;
> }
> arr = &msl->memseg_arr;
> @@ -320,23 +320,6 @@ rte_eal_using_phys_addrs(void)
> return 0;
> }
>
> -static uint64_t
> -get_mem_amount(uint64_t page_sz, uint64_t max_mem)
> -{
> - uint64_t area_sz, max_pages;
> -
> - /* limit to RTE_MAX_MEMSEG_PER_LIST pages or RTE_MAX_MEM_MB_PER_LIST */
> - max_pages = RTE_MAX_MEMSEG_PER_LIST;
> - max_mem = RTE_MIN((uint64_t)RTE_MAX_MEM_MB_PER_LIST << 20, max_mem);
> -
> - area_sz = RTE_MIN(page_sz * max_pages, max_mem);
> -
> - /* make sure the list isn't smaller than the page size */
> - area_sz = RTE_MAX(area_sz, page_sz);
> -
> - return RTE_ALIGN(area_sz, page_sz);
> -}
> -
> static int
> memseg_list_alloc(struct rte_memseg_list *msl)
> {
> @@ -380,9 +363,10 @@ memseg_primary_init(void)
> hpi_idx++) {
> uint64_t max_type_mem, total_type_mem = 0;
> uint64_t avail_mem;
> - int type_msl_idx, max_segs, avail_segs, total_segs = 0;
> + unsigned int avail_segs;
> struct hugepage_info *hpi;
> uint64_t hugepage_sz;
> + unsigned int n_segs;
>
> hpi = &internal_conf->hugepage_info[hpi_idx];
> hugepage_sz = hpi->hugepage_sz;
> @@ -396,7 +380,6 @@ memseg_primary_init(void)
> /* first, calculate theoretical limits according to config */
> max_type_mem = RTE_MIN(max_mem - total_mem,
> (uint64_t)RTE_MAX_MEM_MB_PER_TYPE << 20);
> - max_segs = RTE_MAX_MEMSEG_PER_TYPE;
>
> /* now, limit all of that to whatever will actually be
> * available to us, because without dynamic allocation support,
> @@ -412,42 +395,30 @@ memseg_primary_init(void)
> avail_mem = avail_segs * hugepage_sz;
>
> max_type_mem = RTE_MIN(avail_mem, max_type_mem);
> - max_segs = RTE_MIN(avail_segs, max_segs);
> -
> - type_msl_idx = 0;
> - while (total_type_mem < max_type_mem &&
> - total_segs < max_segs) {
> - uint64_t cur_max_mem, cur_mem;
> - unsigned int n_segs;
> -
> - if (msl_idx >= RTE_MAX_MEMSEG_LISTS) {
> - EAL_LOG(ERR,
> - "No more space in memseg lists, please increase RTE_MAX_MEMSEG_LISTS");
> - return -1;
> - }
> -
> - msl = &mcfg->memsegs[msl_idx++];
> -
> - cur_max_mem = max_type_mem - total_type_mem;
> -
> - cur_mem = get_mem_amount(hugepage_sz,
> - cur_max_mem);
> - n_segs = cur_mem / hugepage_sz;
> + n_segs = max_type_mem / hugepage_sz;
> + if (n_segs == 0)
> + continue;
> +
> + if (msl_idx >= RTE_MAX_MEMSEG_LISTS) {
> + EAL_LOG(ERR,
> + "No more space in memseg lists, please increase RTE_MAX_MEMSEG_LISTS");
Not sure it's worth splitting this across two lines. You only save 3 or 4
chars. :-)
> + return -1;
> + }
>
> - if (eal_memseg_list_init(msl, hugepage_sz, n_segs,
> - 0, type_msl_idx, false))
> - return -1;
> + msl = &mcfg->memsegs[msl_idx];
>
> - total_segs += msl->memseg_arr.len;
> - total_type_mem = total_segs * hugepage_sz;
> - type_msl_idx++;
> + if (eal_memseg_list_init(msl, hugepage_sz, n_segs,
> + 0, msl_idx, false))
Fits on one line.
> + return -1;
>
> - if (memseg_list_alloc(msl)) {
> - EAL_LOG(ERR, "Cannot allocate VA space for memseg list");
> - return -1;
> - }
> + total_type_mem = n_segs * hugepage_sz;
> + if (memseg_list_alloc(msl)) {
> + EAL_LOG(ERR, "Cannot allocate VA space for memseg list");
> + return -1;
> }
> +
> total_mem += total_type_mem;
> + msl_idx++;
> }
> return 0;
> }
> diff --git a/lib/eal/linux/eal_memalloc.c b/lib/eal/linux/eal_memalloc.c
> index a39bc31c7b..2227b1c52b 100644
> --- a/lib/eal/linux/eal_memalloc.c
> +++ b/lib/eal/linux/eal_memalloc.c
> @@ -282,8 +282,8 @@ get_seg_fd(char *path, int buflen, struct hugepage_info *hi,
> huge_path = eal_get_hugefile_path(path, buflen, hi->hugedir, list_idx);
> } else {
> out_fd = &fd_list[list_idx].fds[seg_idx];
> - huge_path = eal_get_hugefile_path(path, buflen, hi->hugedir,
> - list_idx * RTE_MAX_MEMSEG_PER_LIST + seg_idx);
> + huge_path = eal_get_hugefile_list_seg_path(path, buflen,
> + hi->hugedir, list_idx, seg_idx);
> }
> if (huge_path == NULL) {
> EAL_LOG(DEBUG, "%s(): hugefile path truncated: '%s'",
> diff --git a/lib/eal/linux/eal_memory.c b/lib/eal/linux/eal_memory.c
> index bf783e3c76..691d8eb3cc 100644
> --- a/lib/eal/linux/eal_memory.c
> +++ b/lib/eal/linux/eal_memory.c
> @@ -740,8 +740,8 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end)
> break;
> }
> if (msl_idx == RTE_MAX_MEMSEG_LISTS) {
> - EAL_LOG(ERR, "Could not find space for memseg. Please increase RTE_MAX_MEMSEG_PER_LIST "
> - "RTE_MAX_MEMSEG_PER_TYPE and/or RTE_MAX_MEM_MB_PER_TYPE in configuration.");
> + EAL_LOG(ERR,
> + "Could not find suitable space for memseg in existing memseg lists");
> return -1;
> }
>
> @@ -822,23 +822,6 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end)
> return seg_len;
> }
>
> -static uint64_t
> -get_mem_amount(uint64_t page_sz, uint64_t max_mem)
> -{
> - uint64_t area_sz, max_pages;
> -
> - /* limit to RTE_MAX_MEMSEG_PER_LIST pages or RTE_MAX_MEM_MB_PER_LIST */
> - max_pages = RTE_MAX_MEMSEG_PER_LIST;
> - max_mem = RTE_MIN((uint64_t)RTE_MAX_MEM_MB_PER_LIST << 20, max_mem);
> -
> - area_sz = RTE_MIN(page_sz * max_pages, max_mem);
> -
> - /* make sure the list isn't smaller than the page size */
> - area_sz = RTE_MAX(area_sz, page_sz);
> -
> - return RTE_ALIGN(area_sz, page_sz);
> -}
> -
> static int
> memseg_list_free(struct rte_memseg_list *msl)
> {
> @@ -1831,7 +1814,6 @@ memseg_primary_init_32(void)
> uint64_t max_pagesz_mem, cur_pagesz_mem = 0;
> uint64_t hugepage_sz;
> struct hugepage_info *hpi;
> - int type_msl_idx, max_segs, total_segs = 0;
>
> hpi = &internal_conf->hugepage_info[hpi_idx];
> hugepage_sz = hpi->hugepage_sz;
> @@ -1840,62 +1822,60 @@ memseg_primary_init_32(void)
> if (hpi->num_pages[socket_id] == 0)
> continue;
>
> - max_segs = RTE_MAX_MEMSEG_PER_TYPE;
> max_pagesz_mem = max_socket_mem - cur_socket_mem;
>
> /* make it multiple of page size */
> max_pagesz_mem = RTE_ALIGN_FLOOR(max_pagesz_mem,
> hugepage_sz);
>
> + if (max_pagesz_mem == 0)
> + continue;
> +
> EAL_LOG(DEBUG, "Attempting to preallocate "
> "%" PRIu64 "M on socket %i",
> max_pagesz_mem >> 20, socket_id);
>
> - type_msl_idx = 0;
> - while (cur_pagesz_mem < max_pagesz_mem &&
> - total_segs < max_segs) {
> - uint64_t cur_mem;
> + while (cur_pagesz_mem < max_pagesz_mem) {
> + uint64_t rem_mem;
> unsigned int n_segs;
>
> - if (msl_idx >= RTE_MAX_MEMSEG_LISTS) {
> - EAL_LOG(ERR,
> - "No more space in memseg lists, please increase RTE_MAX_MEMSEG_LISTS");
> - return -1;
> - }
> + rem_mem = max_pagesz_mem - cur_pagesz_mem;
> + n_segs = rem_mem / hugepage_sz;
>
> - msl = &mcfg->memsegs[msl_idx];
> + while (n_segs > 0) {
> + if (msl_idx >= RTE_MAX_MEMSEG_LISTS) {
> + EAL_LOG(ERR,
> + "No more space in memseg lists, please increase RTE_MAX_MEMSEG_LISTS");
> + return -1;
> + }
>
> - cur_mem = get_mem_amount(hugepage_sz,
> - max_pagesz_mem);
> - n_segs = cur_mem / hugepage_sz;
> + msl = &mcfg->memsegs[msl_idx];
>
> - if (eal_memseg_list_init(msl, hugepage_sz,
> - n_segs, socket_id, type_msl_idx,
> - true)) {
> - /* failing to allocate a memseg list is
> - * a serious error.
> - */
> - EAL_LOG(ERR, "Cannot allocate memseg list");
> - return -1;
> - }
> + if (eal_memseg_list_init(msl, hugepage_sz,
> + n_segs, socket_id, msl_idx, true) < 0) {
> + /* failing to allocate a memseg list is a serious error. */
> + EAL_LOG(ERR, "Cannot allocate memseg list");
> + return -1;
> + }
> +
> + if (eal_memseg_list_alloc(msl, 0) == 0)
> + break;
>
> - if (eal_memseg_list_alloc(msl, 0)) {
> - /* if we couldn't allocate VA space, we
> - * can try with smaller page sizes.
> - */
> - EAL_LOG(ERR, "Cannot allocate VA space for memseg list, retrying with different page size");
> - /* deallocate memseg list */
> if (memseg_list_free(msl))
> return -1;
> - break;
> +
> + EAL_LOG(DEBUG,
> + "Cannot allocate VA space for memseg list, retrying with smaller chunk");
> + n_segs /= 2;
> }
>
> - total_segs += msl->memseg_arr.len;
> - cur_pagesz_mem = total_segs * hugepage_sz;
> - type_msl_idx++;
> + if (n_segs == 0)
> + break;
> +
> + cur_pagesz_mem += (uint64_t)n_segs * hugepage_sz;
> + cur_socket_mem += (uint64_t)n_segs * hugepage_sz;
> msl_idx++;
> }
> - cur_socket_mem += cur_pagesz_mem;
> }
> if (cur_socket_mem == 0) {
> EAL_LOG(ERR, "Cannot allocate VA space on socket %u",
> --
> 2.47.3
>
^ permalink raw reply
* Re: [PATCH v19 00/11]net/sxe2: fix logic errors and address feedback
From: Thomas Monjalon @ 2026-05-26 14:13 UTC (permalink / raw)
To: Jie Liu; +Cc: dev, stephen, David Marchand
In-Reply-To: <9XBXAO5XSSSHFRFv8zy4yQ@monjalon.net>
21/05/2026 17:16, Thomas Monjalon:
> Hello,
>
> 20/05/2026 04:17, liujie5@linkdatatechnology.com:
> > common/sxe2: add sxe2 basic structures
>
> Are you planning to add a crypto or compress driver?
> This is usually the reason to have a common library.
> If you don't intend to share some code between different driver,
> then you should not have a common library.
We are curious about your plans for sxe2.
Please, could you reply?
Will you add another driver class to sxe2?
^ permalink raw reply
* [PATCH v6] mempool: improve cache behaviour and performance
From: Morten Brørup @ 2026-05-26 14:00 UTC (permalink / raw)
To: dev, Andrew Rybchenko, Bruce Richardson, Jingjing Wu,
Praveen Shetty, Hemant Agrawal, Sachin Saxena
Cc: Morten Brørup
In-Reply-To: <20260408141315.904381-1-mb@smartsharesystems.com>
This patch refactors the mempool cache to eliminate some unexpected
behaviour and reduce the mempool cache miss rate.
1.
The actual cache size was 1.5 times the cache size specified at run-time
mempool creation.
This was obviously not expected by application developers.
2.
In get operations, the check for when to use the cache as bounce buffer
did not respect the run-time configured cache size,
but compared to the build time maximum possible cache size
(RTE_MEMPOOL_CACHE_MAX_SIZE, default 512).
E.g. with a configured cache size of 32 objects, getting 256 objects
would first fetch 32 + 256 = 288 objects into the cache,
and then move the 256 objects from the cache to the destination memory,
instead of fetching the 256 objects directly to the destination memory.
This had a performance cost.
However, this is unlikely to occur in real applications, so it is not
important in itself.
3.
When putting objects into a mempool, and the mempool cache did not have
free space for so many objects,
the cache was flushed completely, and the new objects were then put into
the cache.
I.e. the cache drain level was zero.
This (complete cache flush) meant that a subsequent get operation (with
the same number of objects) completely emptied the cache,
so another subsequent get operation required replenishing the cache.
Similarly,
When getting objects from a mempool, and the mempool cache did not hold so
many objects,
the cache was replenished to cache->size + remaining objects,
and then (the remaining part of) the requested objects were fetched via
the cache,
which left the cache filled (to cache->size) at completion.
I.e. the cache refill level was cache->size (plus some, depending on
request size).
(1) was improved by generally comparing to cache->size instead of
cache->flushthresh, when considering the capacity of the cache.
The cache->flushthresh field is kept for API/ABI compatibility purposes,
and initialized to cache->size instead of cache->size * 1.5.
(2) was improved by generally comparing to cache->size / 2 instead of
RTE_MEMPOOL_CACHE_MAX_SIZE, when checking the bounce buffer limit.
(3) was improved by flushing and replenishing the cache by half its size,
so a flush/refill can be followed randomly by get or put requests.
This also reduced the number of objects in each flush/refill operation.
As a consequence of these changes, the size of the array holding the
objects in the cache (cache->objs[]) no longer needs to be
2 * RTE_MEMPOOL_CACHE_MAX_SIZE, and can be reduced to
RTE_MEMPOOL_CACHE_MAX_SIZE at an API/ABI breaking release.
Performance data:
With a real WAN Optimization application, where the number of allocated
packets varies (as they are held in e.g. shaper queues), the mempool
cache miss rate dropped from ca. 1/20 objects to ca. 1/48 objects.
This was deployed in production at an ISP, and using an effective cache
size of 384 objects.
Bugzilla ID: 1027
Fixes: ea5dd2744b90 ("mempool: cache optimisations")
Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
---
Depends-on: patch-163181 ("net/intel: do not bypass mbuf lib for mbuf fast-free")
---
v6:
* Moved driver changes out as separate patches, for easier review. (Bruce)
Tests using the Intel idpf PMD in AVX512 mode may fail with this patch.
* Reverted a small code comment change. The original was better. (Bruce)
* Reverted rte_mempool_create() description requiring the cache_size to be
an even number. There is no such requirement.
v5:
* Flush the cache from the bottom, where objects are colder, and move down
the remaining objects, which are hotter.
* In the Intel idpf PMD, move up the hot objects in the cache and refill
with cold objects at the bottom.
v4:
* Added Bugzilla ID.
* Added Fixes tag. For reference only.
* Moved fast-free related update of Intel common driver out as a separate
patch, and depend on that patch.
* Omitted unrelated changes to the Intel idpf AVX512 driver, specifically
fixing an indentation and adding mbuf instrumentation.
* Omitted unrelated changes to the mempool library, specifically adding
__rte_restrict and changing a couple of comments to proper sentences.
* Please checkpatches by swapping operators in a couple of comparisons.
v3:
* Fixed my copy-paste bug in idpf_splitq_rearm().
v2:
* Fixed issue found by abidiff:
Reverted cache objects array size reduction. Added a note instead.
* Added missing mbuf instrumentation to the Intel idpf AVX512 driver.
* Updated idpf_splitq_rearm() like idpf_singleq_rearm().
* Added a few more __rte_assume(). (Inspired by AI review)
* Updated NXP dpaa and dpaa2 mempool drivers to not set mempool cache
flush threshold.
* Added release notes.
* Added deprecation notes.
---
doc/guides/rel_notes/deprecation.rst | 7 +++
doc/guides/rel_notes/release_26_07.rst | 11 +++++
lib/mempool/rte_mempool.c | 14 +-----
lib/mempool/rte_mempool.h | 66 ++++++++++++++++----------
4 files changed, 61 insertions(+), 37 deletions(-)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 35c9b4e06c..40760fffbb 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -154,3 +154,10 @@ Deprecation Notices
* bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
Those API functions are used internally by DPDK core and netvsc PMD.
+
+* mempool: The ``flushthresh`` field in ``struct rte_mempool_cache``
+ is obsolete, and will be removed in DPDK 26.11.
+
+* mempool: The object array in ``struct rte_mempool_cache`` is oversize by
+ factor two, and will be reduced to ``RTE_MEMPOOL_CACHE_MAX_SIZE`` in
+ DPDK 26.11.
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 6f43d9b61c..3f793f504a 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -63,6 +63,17 @@ New Features
``rte_eal_init`` and the application is responsible for probing each device,
* ``--auto-probing`` enables the initial bus probing, which is the current default behavior.
+* **Changed effective size of mempool cache.**
+
+ * The effective size of a mempool cache was changed to match the specified size at mempool creation; the effective size was previously 50 % larger than requested.
+ * The ``flushthresh`` field of the ``struct rte_mempool_cache`` became obsolete, but was kept for API/ABI compatibility purposes.
+ * The effective size of the ``objs`` array in the ``struct rte_mempool_cache`` was reduced to ``RTE_MEMPOOL_CACHE_MAX_SIZE``, but its size was kept for API/ABI compatibility purposes.
+
+* **Improved mempool cache flush/refill algorithm.**
+
+ The mempool cache flush/refill algorithm was improved, to reduce the mempool cache miss rate for most application types.
+ Applications where each lcore only puts or gets to a mempool, e.g. pipelined applications where ethdev Rx and Tx run on separate lcores, should adapt to the new algorithm by doubling their configured mempool cache size, to avoid doubling their mempool cache miss rate.
+
* **Updated PCAP ethernet driver.**
* Added support for VLAN insertion and stripping.
diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c
index 3042d94c14..805b52cc58 100644
--- a/lib/mempool/rte_mempool.c
+++ b/lib/mempool/rte_mempool.c
@@ -52,11 +52,6 @@ static void
mempool_event_callback_invoke(enum rte_mempool_event event,
struct rte_mempool *mp);
-/* Note: avoid using floating point since that compiler
- * may not think that is constant.
- */
-#define CALC_CACHE_FLUSHTHRESH(c) (((c) * 3) / 2)
-
#if defined(RTE_ARCH_X86)
/*
* return the greatest common divisor between a and b (fast algorithm)
@@ -757,13 +752,8 @@ rte_mempool_free(struct rte_mempool *mp)
static void
mempool_cache_init(struct rte_mempool_cache *cache, uint32_t size)
{
- /* Check that cache have enough space for flush threshold */
- RTE_BUILD_BUG_ON(CALC_CACHE_FLUSHTHRESH(RTE_MEMPOOL_CACHE_MAX_SIZE) >
- RTE_SIZEOF_FIELD(struct rte_mempool_cache, objs) /
- RTE_SIZEOF_FIELD(struct rte_mempool_cache, objs[0]));
-
cache->size = size;
- cache->flushthresh = CALC_CACHE_FLUSHTHRESH(size);
+ cache->flushthresh = size; /* Obsolete; for API/ABI compatibility purposes only */
cache->len = 0;
}
@@ -850,7 +840,7 @@ rte_mempool_create_empty(const char *name, unsigned n, unsigned elt_size,
/* asked cache too big */
if (cache_size > RTE_MEMPOOL_CACHE_MAX_SIZE ||
- CALC_CACHE_FLUSHTHRESH(cache_size) > n) {
+ cache_size > n) {
rte_errno = EINVAL;
return NULL;
}
diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
index 2e54fc4466..cd0f229b59 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -89,7 +89,7 @@ struct __rte_cache_aligned rte_mempool_debug_stats {
*/
struct __rte_cache_aligned rte_mempool_cache {
uint32_t size; /**< Size of the cache */
- uint32_t flushthresh; /**< Threshold before we flush excess elements */
+ uint32_t flushthresh; /**< Obsolete; for API/ABI compatibility purposes only */
uint32_t len; /**< Current cache count */
#ifdef RTE_LIBRTE_MEMPOOL_STATS
uint32_t unused;
@@ -107,8 +107,10 @@ struct __rte_cache_aligned rte_mempool_cache {
/**
* Cache objects
*
- * Cache is allocated to this size to allow it to overflow in certain
- * cases to avoid needless emptying of cache.
+ * Note:
+ * Cache is allocated at double size for API/ABI compatibility purposes only.
+ * When reducing its size at an API/ABI breaking release,
+ * remember to add a cache guard after it.
*/
alignas(RTE_CACHE_LINE_SIZE) void *objs[RTE_MEMPOOL_CACHE_MAX_SIZE * 2];
};
@@ -1047,11 +1049,16 @@ rte_mempool_free(struct rte_mempool *mp);
* If cache_size is non-zero, the rte_mempool library will try to
* limit the accesses to the common lockless pool, by maintaining a
* per-lcore object cache. This argument must be lower or equal to
- * RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.
+ * RTE_MEMPOOL_CACHE_MAX_SIZE and n.
* The access to the per-lcore table is of course
* faster than the multi-producer/consumer pool. The cache can be
* disabled if the cache_size argument is set to 0; it can be useful to
* avoid losing objects in cache.
+ * Note:
+ * Mempool put/get requests of more than cache_size / 2 objects may be
+ * partially or fully served directly by the multi-producer/consumer
+ * pool, to avoid the overhead of copying the objects twice (instead of
+ * once) when using the cache as a bounce buffer.
* @param private_data_size
* The size of the private data appended after the mempool
* structure. This is useful for storing some private data after the
@@ -1390,22 +1397,30 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
- __rte_assume(cache->flushthresh <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
- __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
- __rte_assume(cache->len <= cache->flushthresh);
- if (likely(cache->len + n <= cache->flushthresh)) {
+ __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
+ __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
+ __rte_assume(cache->len <= cache->size);
+ if (likely(cache->len + n <= cache->size)) {
/* Sufficient room in the cache for the objects. */
cache_objs = &cache->objs[cache->len];
cache->len += n;
- } else if (n <= cache->flushthresh) {
+ } else if (n <= cache->size / 2) {
/*
- * The cache is big enough for the objects, but - as detected by
- * the comparison above - has insufficient room for them.
- * Flush the cache to make room for the objects.
+ * The number of objects is within the cache bounce buffer limit,
+ * but - as detected by the comparison above - the cache has
+ * insufficient room for them.
+ * Flush the cache to the backend to make room for the objects;
+ * flush (size / 2) objects from the bottom of the cache, where
+ * objects are less hot, and move down the remaining objects, which
+ * are more hot, from the upper half of the cache.
*/
- cache_objs = &cache->objs[0];
- rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
- cache->len = n;
+ __rte_assume(cache->len > cache->size / 2);
+ rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0], cache->size / 2);
+ rte_memcpy(&cache->objs[0], &cache->objs[cache->size / 2],
+ sizeof(void *) * (cache->len - cache->size / 2));
+ cache_objs = &cache->objs[cache->len - cache->size / 2];
+ cache->len = cache->len - cache->size / 2 + n;
} else {
/* The request itself is too big for the cache. */
goto driver_enqueue_stats_incremented;
@@ -1524,7 +1539,7 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
/* The cache is a stack, so copy will be in reverse order. */
cache_objs = &cache->objs[cache->len];
- __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
+ __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
if (likely(n <= cache->len)) {
/* The entire request can be satisfied from the cache. */
RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
@@ -1548,13 +1563,13 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
for (index = 0; index < len; index++)
*obj_table++ = *--cache_objs;
- /* Dequeue below would overflow mem allocated for cache? */
- if (unlikely(remaining > RTE_MEMPOOL_CACHE_MAX_SIZE))
+ /* Dequeue below would exceed the cache bounce buffer limit? */
+ __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ if (unlikely(remaining > cache->size / 2))
goto driver_dequeue;
- /* Fill the cache from the backend; fetch size + remaining objects. */
- ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs,
- cache->size + remaining);
+ /* Fill the cache from the backend; fetch (size / 2) objects. */
+ ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache->size / 2);
if (unlikely(ret < 0)) {
/*
* We are buffer constrained, and not able to fetch all that.
@@ -1568,10 +1583,11 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
- __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
- __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE);
- cache_objs = &cache->objs[cache->size + remaining];
- cache->len = cache->size;
+ __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ __rte_assume(remaining <= cache->size / 2);
+ cache_objs = &cache->objs[cache->size / 2];
+ cache->len = cache->size / 2 - remaining;
for (index = 0; index < remaining; index++)
*obj_table++ = *--cache_objs;
--
2.43.0
^ permalink raw reply related
* Re: [PATCH v3 00/25] Consolidate bus driver infrastructure
From: David Marchand @ 2026-05-26 13:59 UTC (permalink / raw)
To: dev; +Cc: thomas, stephen, bruce.richardson
In-Reply-To: <CAJFAV8wAt1hyHrL0N1AxQ1kBjKoQNtuo89m_H7LHZJmtGea_gQ@mail.gmail.com>
On Tue, 26 May 2026 at 11:03, David Marchand <david.marchand@redhat.com> wrote:
>
> On Tue, 26 May 2026 at 10:42, David Marchand <david.marchand@redhat.com> wrote:
> >
> > This is a continuation of the work I started on the bus infrastructure,
> > but this time, a lot of the changes were done by a AI "friend".
> > It is still an unfinished topic as the current series focuses on probing
> > only. The detaching/cleanup aspect is postponed to another release/time.
> >
> > My AI "friend" really *sucked* at git and at separating unrelated changes,
> > so it required quite a lot of massage/polishing afterwards.
> > But it seems good enough now for upstream submission.
> >
> > I would like to see this series merged in 26.07, so that we have enough
> > time to stabilize it before the next LTS.
> > And seeing how it affects drivers, it is probably better to merge it
> > the sooner possible (so Thomas does not have to solve too many conflicts
> > when pulling next-* subtrees after, especially wrt the last patch).
> >
> >
> > This series refactors the DPDK bus infrastructure to consolidate common
> > operations and reduce code duplication across all bus drivers.
> > Currently, each bus implements its own specific device/driver lists,
> > probe logic, and lookup functions.
> > This series moves these common patterns into the EAL bus layer,
> > providing generic helpers that all buses can use.
> >
> > The refactoring removes approximately 1,400 lines of duplicated code across
> > the codebase while maintaining full functional equivalence.
> >
> > Key changes:
> > - Factorize device and driver lists into struct rte_bus
> > - Implement generic probe, device/driver lookup, and iteration helpers in EAL
> > - Introduce conversion macros (RTE_BUS_DEVICE, RTE_BUS_DRIVER, RTE_CLASS_TO_BUS_DEVICE)
> > to safely convert between generic and bus-specific types
> > - Remove bus-specific device/driver types from most driver code
> > - Move probe logic from individual buses to rte_bus_generic_probe()
> > - Separate NXP-specific metadata from generic bus structures
> >
> > Benefits:
> > - Significant code reduction (~1,400 lines removed)
> > - Consistent behavior across all bus types
> > - Simplified bus driver implementation
> > - Easier maintenance and future enhancements
> >
> > The series is structured as a progressive refactoring:
> > - Remove redundant checks and helpers (patches 1-5)
> > - Add conversion macros and factorize lists (patches 6-8)
> > - Consolidate device/driver lookup and iteration (patches 9-11)
> > - Refactor probe logic (patches 12-15)
> > - Remove bus-specific types from drivers (patches 16-23)
> >
> > Note on ABI:
> > This series breaks the ABI for drivers (changes to rte_pci_device,
> > rte_pci_driver, and similar structures for other buses). However, the DPDK
> > ABI policy does not provide guarantees for driver-level interfaces.
> >
> >
> > --
> > David Marchand
> >
> > Changes since v2:
> > - fixed dma/idxd probing as reported by Bruce,
> > - moved api_ver setting (from match to scan) in bus/uacce,
> > - fixed transient bug in the vdev code (in the middle of the series).
> > tl;dr the high level difference for bus/vdev between v2 and v3 == 0,
> > - fixed doxygen,
> > - enhanced API description,
>
> There was some hiccup from Red Hat SMTP when sending the series..
> hopefully my resending of the second half of the patches is good
> enough.
> At least, patchwork looks happy.
The sxe drivers got merged and pulled in main, so I'll prepare a v4
and send it probably tomorrow.
Comments still welcome.
--
David Marchand
^ permalink raw reply
* Re: [PATCH v1 00/23] add net/sxe2 support for flow control
From: Stephen Hemminger @ 2026-05-26 13:29 UTC (permalink / raw)
To: liujie5; +Cc: dev
In-Reply-To: <20260524093259.397506-1-liujie5@linkdatatechnology.com>
On Sun, 24 May 2026 17:32:36 +0800
liujie5@linkdatatechnology.com wrote:
> From: Jie Liu <liujie5@linkdatatechnology.com>
>
> v1:
> -- add support for flow control
> -- add support for flow control status interrupt notification
> -- add support for ipsec inline protocol offload
>
> Jie Liu (23):
> net/sxe2: support AVX512 vectorized path for Rx and Tx
> net/sxe2: add AVX2 vector data path for Rx and Tx
> drivers: add supported packet types get callback
> net/sxe2: support L2 filtering and MAC config
> drivers: support RSS feature
> net/sxe2: support TM hierarchy and shaping
> net/sxe2: support IPsec inline protocol offload
> net/sxe2: support statistics and multi-process
> drivers: interrupt handling
> net/sxe2: add NEON vec Rx/Tx burst functions
> net/sxe2: add support for VF representors
> net/sxe2: add support for custom UDP tunnel ports
> net/sxe2: support firmware version reading
> net/sxe2: implement get monitor address
> common/sxe2: add shared SFP module definitions
> net/sxe2: support SFP module info and EEPROM access
> net/sxe2: implement private dump info
> net/sxe2: add mbuf validation in Tx debug mode
> net/sxe2: add testpmd commands for private features
> net/sxe2: add private devargs parsing
> net/sxe2: support flow control status interrupt notification
> net/sxe2: update sxe2 feature matrix docs
> common/sxe2: add memseg walk callback
>
> doc/guides/nics/features/sxe2.ini | 66 +
> drivers/common/sxe2/sxe2_common.c | 156 ++
> drivers/common/sxe2/sxe2_common.h | 4 +
> drivers/common/sxe2/sxe2_flow_public.h | 633 +++++++
> drivers/common/sxe2/sxe2_ioctl_chnl.c | 179 +-
> drivers/common/sxe2/sxe2_ioctl_chnl_func.h | 18 +
> drivers/common/sxe2/sxe2_msg.h | 117 ++
> drivers/common/sxe2/sxe2_ptype.h | 1793 ++++++++++++++++++
> drivers/net/sxe2/meson.build | 56 +-
> drivers/net/sxe2/sxe2_cmd_chnl.c | 1588 +++++++++++++++-
> drivers/net/sxe2/sxe2_cmd_chnl.h | 139 ++
> drivers/net/sxe2/sxe2_drv_cmd.h | 439 ++++-
> drivers/net/sxe2/sxe2_dump.c | 304 +++
> drivers/net/sxe2/sxe2_dump.h | 12 +
> drivers/net/sxe2/sxe2_ethdev.c | 1526 ++++++++++++++-
> drivers/net/sxe2/sxe2_ethdev.h | 113 +-
> drivers/net/sxe2/sxe2_ethdev_repr.c | 610 ++++++
> drivers/net/sxe2/sxe2_ethdev_repr.h | 32 +
> drivers/net/sxe2/sxe2_filter.c | 897 +++++++++
> drivers/net/sxe2/sxe2_filter.h | 100 +
> drivers/net/sxe2/sxe2_flow.c | 1391 ++++++++++++++
> drivers/net/sxe2/sxe2_flow.h | 30 +
> drivers/net/sxe2/sxe2_flow_define.h | 144 ++
> drivers/net/sxe2/sxe2_flow_parse_action.c | 1182 ++++++++++++
> drivers/net/sxe2/sxe2_flow_parse_action.h | 23 +
> drivers/net/sxe2/sxe2_flow_parse_engine.c | 106 ++
> drivers/net/sxe2/sxe2_flow_parse_engine.h | 13 +
> drivers/net/sxe2/sxe2_flow_parse_pattern.c | 1935 ++++++++++++++++++++
> drivers/net/sxe2/sxe2_flow_parse_pattern.h | 46 +
> drivers/net/sxe2/sxe2_ipsec.c | 1565 ++++++++++++++++
> drivers/net/sxe2/sxe2_ipsec.h | 254 +++
> drivers/net/sxe2/sxe2_irq.c | 1024 +++++++++++
> drivers/net/sxe2/sxe2_irq.h | 25 +
> drivers/net/sxe2/sxe2_mac.c | 535 ++++++
> drivers/net/sxe2/sxe2_mac.h | 84 +
> drivers/net/sxe2/sxe2_mp.c | 413 +++++
> drivers/net/sxe2/sxe2_mp.h | 73 +
> drivers/net/sxe2/sxe2_queue.c | 17 +-
> drivers/net/sxe2/sxe2_rss.c | 584 ++++++
> drivers/net/sxe2/sxe2_rss.h | 81 +
> drivers/net/sxe2/sxe2_rx.c | 38 +
> drivers/net/sxe2/sxe2_rx.h | 2 +
> drivers/net/sxe2/sxe2_security.c | 335 ++++
> drivers/net/sxe2/sxe2_security.h | 77 +
> drivers/net/sxe2/sxe2_stats.c | 591 ++++++
> drivers/net/sxe2/sxe2_stats.h | 39 +
> drivers/net/sxe2/sxe2_switchdev.c | 332 ++++
> drivers/net/sxe2/sxe2_switchdev.h | 33 +
> drivers/net/sxe2/sxe2_testpmd.c | 733 ++++++++
> drivers/net/sxe2/sxe2_testpmd_lib.c | 969 ++++++++++
> drivers/net/sxe2/sxe2_testpmd_lib.h | 142 ++
> drivers/net/sxe2/sxe2_tm.c | 1169 ++++++++++++
> drivers/net/sxe2/sxe2_tm.h | 78 +
> drivers/net/sxe2/sxe2_tx.c | 7 +
> drivers/net/sxe2/sxe2_txrx.c | 174 +-
> drivers/net/sxe2/sxe2_txrx.h | 4 +
> drivers/net/sxe2/sxe2_txrx_check_mbuf.c | 595 ++++++
> drivers/net/sxe2/sxe2_txrx_check_mbuf.h | 38 +
> drivers/net/sxe2/sxe2_txrx_poll.c | 243 ++-
> drivers/net/sxe2/sxe2_txrx_vec.c | 46 +-
> drivers/net/sxe2/sxe2_txrx_vec.h | 38 +-
> drivers/net/sxe2/sxe2_txrx_vec_avx2.c | 777 ++++++++
> drivers/net/sxe2/sxe2_txrx_vec_avx512.c | 897 +++++++++
> drivers/net/sxe2/sxe2_txrx_vec_common.h | 1 -
> drivers/net/sxe2/sxe2_txrx_vec_neon.c | 707 +++++++
> drivers/net/sxe2/sxe2_vsi.c | 146 ++
> drivers/net/sxe2/sxe2_vsi.h | 12 +-
> drivers/net/sxe2/sxe2vf_regs.h | 82 +
> 68 files changed, 26531 insertions(+), 81 deletions(-)
> create mode 100644 drivers/common/sxe2/sxe2_flow_public.h
> create mode 100644 drivers/common/sxe2/sxe2_msg.h
> create mode 100644 drivers/common/sxe2/sxe2_ptype.h
> create mode 100644 drivers/net/sxe2/sxe2_dump.c
> create mode 100644 drivers/net/sxe2/sxe2_dump.h
> create mode 100644 drivers/net/sxe2/sxe2_ethdev_repr.c
> create mode 100644 drivers/net/sxe2/sxe2_ethdev_repr.h
> create mode 100644 drivers/net/sxe2/sxe2_filter.c
> create mode 100644 drivers/net/sxe2/sxe2_filter.h
> create mode 100644 drivers/net/sxe2/sxe2_flow.c
> create mode 100644 drivers/net/sxe2/sxe2_flow.h
> create mode 100644 drivers/net/sxe2/sxe2_flow_define.h
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_action.c
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_action.h
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_engine.c
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_engine.h
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_pattern.c
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_pattern.h
> create mode 100644 drivers/net/sxe2/sxe2_ipsec.c
> create mode 100644 drivers/net/sxe2/sxe2_ipsec.h
> create mode 100644 drivers/net/sxe2/sxe2_irq.c
> create mode 100644 drivers/net/sxe2/sxe2_mac.c
> create mode 100644 drivers/net/sxe2/sxe2_mac.h
> create mode 100644 drivers/net/sxe2/sxe2_mp.c
> create mode 100644 drivers/net/sxe2/sxe2_mp.h
> create mode 100644 drivers/net/sxe2/sxe2_rss.c
> create mode 100644 drivers/net/sxe2/sxe2_rss.h
> create mode 100644 drivers/net/sxe2/sxe2_security.c
> create mode 100644 drivers/net/sxe2/sxe2_security.h
> create mode 100644 drivers/net/sxe2/sxe2_stats.c
> create mode 100644 drivers/net/sxe2/sxe2_stats.h
> create mode 100644 drivers/net/sxe2/sxe2_switchdev.c
> create mode 100644 drivers/net/sxe2/sxe2_switchdev.h
> create mode 100644 drivers/net/sxe2/sxe2_testpmd.c
> create mode 100644 drivers/net/sxe2/sxe2_testpmd_lib.c
> create mode 100644 drivers/net/sxe2/sxe2_testpmd_lib.h
> create mode 100644 drivers/net/sxe2/sxe2_tm.c
> create mode 100644 drivers/net/sxe2/sxe2_tm.h
> create mode 100644 drivers/net/sxe2/sxe2_txrx_check_mbuf.c
> create mode 100644 drivers/net/sxe2/sxe2_txrx_check_mbuf.h
> create mode 100644 drivers/net/sxe2/sxe2_txrx_vec_avx2.c
> create mode 100644 drivers/net/sxe2/sxe2_txrx_vec_avx512.c
> create mode 100644 drivers/net/sxe2/sxe2_txrx_vec_neon.c
> create mode 100644 drivers/net/sxe2/sxe2vf_regs.h
>
Lots of AI review feedback issues (see below).
One other thing which I have recently started looking at is use of __rte_always_inline.
Since much of DPDK has just used __rte_always_inline without justification,
the resulting code is usually slower! Therefore in any new submission,
any use of always inline requires actual benchmark data that it helps.
There are a few case like when there it can help, but very few.
Review of "[PATCH v1 00/23] net/sxe2: PMD updates" series
20 of 23 patches are in the bundle (01-20); 21-23 are missing.
Note: the series targets main, not an LTS branch.
Patch 01/23 (net/sxe2: support AVX512 vectorized path for Rx and Tx)
====================================================================
Error: sxe2_tx_mode_func_set() now contains two back-to-back
'if (tx_mode_flags & SXE2_TX_MODE_VEC_SET_MASK)' blocks. The first
sets dev->tx_pkt_burst to sxe2_tx_pkts_vec_sse/sse_simple; the
second immediately overwrites those assignments (and sets
tx_pkt_prepare = NULL before the inner if/else decides what to
set). The first block is dead. Either it was meant to be deleted
when the AVX512-aware block was added, or the new code should
have replaced rather than duplicated the old one.
Error: After this patch the first if-block above is no longer wrapped
in #ifdef RTE_ARCH_X86 (the original '-#ifdef RTE_ARCH_X86' line
was removed). On non-x86 builds the references to
sxe2_tx_pkts_vec_sse / sxe2_tx_pkts_vec_sse_simple are
undefined and the file will not compile.
Error: In sxe2_tx_queue_mbufs_release_vec(), the new non-AVX512 'else'
branch has 'buffer = txq->buffer_ring;' written twice in a row:
+ buffer = txq->buffer_ring;
+ buffer = txq->buffer_ring;
This is a dead store and indicates careless editing.
Warning: In the same function, after the new
#ifdef CC_AVX512_SUPPORT / else / #endif block, the original
trailing 'for (; i < txq->next_use; ++i) { ... }' loop is still
there. By the time control reaches it, the inner branches have
already advanced i to txq->next_use, so the loop never runs.
This is dead code; remove it.
Warning: In sxe2_tx_bufs_free_vec_avx512(), the 'normal' path
(sxe2_txrx_vec_avx512.c around the rs_thresh loop) is
misindented and has mismatched braces visually:
} else {
rte_mempool_put_bulk(mbuf_free_arr[0]->pool, ...);
mbuf_free_arr[0] = mbuf; /* wrong indent */
free_num = 1;
}
}
The code is syntactically correct but unreadable. Reindent.
Info: 'tx_mode_flags |= SXE2_TX_MODE_VEC_SSE;' (double space) and the
Yoda '(0 == (tx_mode_flags & ...))' style are inconsistent with
the rest of the function which uses 'x == 0'.
Patch 02/23 (net/sxe2: add AVX2 vector data path for Rx and Tx)
===============================================================
Warning: The commit message body ends with
Signed-off-by: ...
net/sxe2: support AVX512 vectorized path for Rx and Tx
The trailing line looks like a stray subject from patch 01
pasted in. Remove it.
Warning: The new AVX2 meson block adds '-mavx2' to c_args without
gating on cc.has_argument('-mavx2') or __AVX2__. The AVX512
block above it checks both compiler and CPU support. Apply
the same pattern to AVX2 for consistency.
Note: this patch does not delete the duplicate-block / non-x86 bugs
introduced in patch 01; it extends them with an AVX2 branch in
the second block while the first SSE-only block is still
executing first and being overwritten.
Patch 03/23 (drivers: add supported packet types get callback)
==============================================================
Error: drivers/net/sxe2/sxe2_ethdev.c new
sxe2_buffer_split_supported_hdr_ptypes_get() declares its
'size_t *no_of_elements' parameter as __rte_unused and never
writes to it. The ethdev caller
(rte_eth_buffer_split_get_supported_hdr_ptypes in
lib/ethdev/rte_ethdev.c) reads no_of_elements after the
callback returns and uses it as a loop bound. The caller's
no_of_elements is an uninitialized stack variable, so this
will iterate a garbage number of times and walk past the
ptypes[] array. Set *no_of_elements = RTE_DIM(ptypes); before
returning.
Error: drivers/common/sxe2/sxe2_ptype.h opens with
#ifndef _SXE2_PTYPE_H_
#define _SXE2_PTYPE_H_
and ends with
#endif /* _RTE_PTYPE_TUNNEL_GRENAT_H_ */
The closing comment is from a different header. Compilation
isn't affected but it makes the file look generated/copied.
Fix the comment.
Warning: The file is a 1793-line static inline function
(sxe2_init_ptype_list) plus a smaller static inline in a
header installed under drivers/common/sxe2/. Every TU that
includes it gets its own copy of the array and code. Move the
body to a .c file and expose only a prototype.
Warning: sxe2_mtu_set() in sxe2_mac.c declares 'uint16_t mtu
__rte_unused' and does nothing with the value. It only checks
dev_started and returns 0. It does not configure hardware,
does not validate against scatter Rx state, and does not
re-select Rx/Tx burst functions. ethdev will write
dev->data->mtu = mtu on return, so the driver silently accepts
any value the caller passes that passes ethdev's range check.
If this is a stub, mark it with a TODO; if not, finish it.
Info: sxe2_mac.h begins with a blank line before the SPDX comment.
Patch 04/23 (net/sxe2: support L2 filtering and MAC config)
===========================================================
No findings.
Patch 05/23 (drivers: support RSS feature)
==========================================
Info: '(uint8_t)rte_rand()' in sxe2_rss_hash_key_init throws away
56 bits of entropy per byte. Use rte_rand() once per 8 key bytes
or memcpy a uint64_t. Not a correctness bug.
Patch 06/23 (net/sxe2: support TM hierarchy and shaping)
========================================================
Error: Nine occurrences in sxe2_tm.c return positive ENOTSUP instead
of negative -ENOTSUP:
ret = ENOTSUP;
goto l_end;
Surrounding code uses -EINVAL and -ENOMEM (negative), so this
is inconsistent and wrong. A positive return is treated as
success by callers that check 'if (ret)'/'if (ret < 0)'.
Change all nine to '-ENOTSUP'.
Patch 07/23 (net/sxe2: support IPsec inline protocol offload)
=============================================================
Info: sxe2_ipsec_id_alloc() does a linear bitmap scan for every SA
allocation. For max_tx_sa/max_rx_sa in the thousands this is
slow but functionally correct. Consider rte_bitmap_scan or a
free-list.
Info: '0XFFFF' uppercase X is unusual; rest of the tree uses '0xFFFF'.
Patch 08/23 (net/sxe2: support statistics and multi-process)
============================================================
Info: 'send_reply = false;' assigns false (bool) to an int32_t. Either
change the type of send_reply to bool or use 0/1.
Patch 09/23 (drivers: interrupt handling)
=========================================
Warning: sxe2_event_intr_post() uses
rte_malloc(NULL, sizeof(struct sxe2_event_element), 0) for a
per-event control structure. This consumes hugepage memory
for an object that does not need DMA, is not shared with
secondary processes, and is freed on the same thread.
Use plain malloc().
Warning: The 'write(handler->fd[1], ¬ify_byte, 1)' return is
masked with RTE_SET_USED(nw). On EAGAIN/EINTR/short write the
element is queued but never delivered, and the consumer side
is never woken. Either loop on EINTR/EAGAIN or check the
return and unqueue/free the element on failure.
Patch 10/23 (net/sxe2: add NEON vec Rx/Tx burst functions)
==========================================================
Not reviewed in detail this round.
Patch 11/23 (net/sxe2: add support for VF representors)
=======================================================
Error: sxe2_flow_check_rss_action_attr() declares
'int32_t ret = ENOTSUP;' (positive). It is then overwritten in
every reachable path (-EINVAL in the default case, 0 at the
end), so the initial value is a dead store - but the bigger
problem is the function returns 0 after calling
rte_flow_error_set() for rss->level > 2, rss->key_len != 0, or
rss->queue_num != 0. The errors are reported via the error
object but ret is not set to -EINVAL/-ENOTSUP, so the caller
sees success. Each of those if-branches should set ret to a
negative errno and goto l_end (or return immediately).
Additionally, rte_flow_error_set() is called with positive
ENOTSUP as the error code. rte_flow API expects negative errno
(-rte_errno style). Use -ENOTSUP.
(Same dead-store pattern around 'ret = ENOTSUP;' exists at
least once more in this patch - audit all occurrences.)
Patches 12-20
=============
Not reviewed in detail this round.
General observations across the series
======================================
- Several patches mix positive and negative errno returns; this is a
recurring class of bug. Search the whole series for
'ret = E[A-Z]' (no minus) and audit each hit.
- Multiple files use 'rte_malloc' for per-call control structures
that do not need hugepage memory. Use malloc() unless the buffer
is for DMA, shared between primary/secondary, or NUMA-pinned.
- A few places use 'rte_zmalloc' (no _socket variant) for what looks
like queue-related state. Where the buffer is touched by the data
path or by secondary processes, use rte_zmalloc_socket().
- The Signed-off-by chain in some patches has stray lines after the
s-o-b (see patch 02). Re-run 'git format-patch' or hand-edit.
^ permalink raw reply
* Re: [RFC v2 2/3] lib: add fastmem library
From: Stephen Hemminger @ 2026-05-26 13:23 UTC (permalink / raw)
To: Mattias Rönnblom
Cc: dev, Morten Brørup, Konstantin Ananyev,
Mattias Rönnblom, Yogaraj Baskaravel
In-Reply-To: <20260526085743.64396-3-hofors@lysator.liu.se>
On Tue, 26 May 2026 10:57:42 +0200
Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> +__rte_experimental
> +void *
> +rte_fastmem_alloc(size_t size, size_t align, unsigned int flags)
> + __rte_alloc_size(1) __rte_alloc_align(2);
Should also add attribute __rte_malloc which tells compiler
that pointer returned cannot alias other memory
And add __rte_dealloc(rte_fastmem_free, 1)
which tells compiler that the returned pointer should only go
back to fastmem (not free, rte_free, etc).
^ permalink raw reply
* Re: [v4 1/1] net/hinic3: Fix VXLAN TSO issue
From: Stephen Hemminger @ 2026-05-26 13:15 UTC (permalink / raw)
To: Feifei Wang; +Cc: dev, Feifei Wang
In-Reply-To: <20260526121820.1093-2-wff_light@vip.163.com>
On Tue, 26 May 2026 20:18:18 +0800
Feifei Wang <wff_light@vip.163.com> wrote:
> From: Feifei Wang <wangfeifei40@huawei.com>
>
> VXLAN TSO lacks a flag bit, causing the processing function
> to determine that the hardware does not support it, leading
> to improper handling.
>
> Signed-off-by: Feifei Wang <wangfeifei40@huawei.com>
If this is a bug fix, it needs Fixes: tag and should
have Cc: stable@dpdk.org
> ---
> drivers/net/hinic3/hinic3_ethdev.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/hinic3/hinic3_ethdev.c b/drivers/net/hinic3/hinic3_ethdev.c
> index f4eb788..4776bc1 100644
> --- a/drivers/net/hinic3/hinic3_ethdev.c
> +++ b/drivers/net/hinic3/hinic3_ethdev.c
> @@ -652,8 +652,12 @@ hinic3_dev_configure(struct rte_eth_dev *dev)
> static void
> hinic3_dev_tnl_tso_support(struct rte_eth_dev_info *info, struct hinic3_nic_dev *nic_dev)
> {
> + if (HINIC3_SUPPORT_VXLAN_OFFLOAD(nic_dev))
> + info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_VXLAN_TNL_TSO;
> +
> if (HINIC3_SUPPORT_GENEVE_OFFLOAD(nic_dev))
> info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_GENEVE_TNL_TSO;
> +
> if (HINIC3_SUPPORT_IPXIP_OFFLOAD(nic_dev))
> info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_IPIP_TNL_TSO;
> }
> @@ -698,7 +702,6 @@ hinic3_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)
> RTE_ETH_TX_OFFLOAD_SCTP_CKSUM |
> RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM |
> RTE_ETH_TX_OFFLOAD_TCP_TSO | RTE_ETH_TX_OFFLOAD_MULTI_SEGS;
> - if (nic_dev->feature_cap & NIC_F_HTN_CMDQ)
> hinic3_dev_tnl_tso_support(info, nic_dev);
If you remove the if here, then you need to fix indentation on
the line below.
^ permalink raw reply
* [v4 1/1] net/hinic3: Fix VXLAN TSO issue
From: Feifei Wang @ 2026-05-26 12:18 UTC (permalink / raw)
To: dev; +Cc: Feifei Wang
In-Reply-To: <20260526121820.1093-1-wff_light@vip.163.com>
From: Feifei Wang <wangfeifei40@huawei.com>
VXLAN TSO lacks a flag bit, causing the processing function
to determine that the hardware does not support it, leading
to improper handling.
Signed-off-by: Feifei Wang <wangfeifei40@huawei.com>
---
drivers/net/hinic3/hinic3_ethdev.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/hinic3_ethdev.c b/drivers/net/hinic3/hinic3_ethdev.c
index f4eb788..4776bc1 100644
--- a/drivers/net/hinic3/hinic3_ethdev.c
+++ b/drivers/net/hinic3/hinic3_ethdev.c
@@ -652,8 +652,12 @@ hinic3_dev_configure(struct rte_eth_dev *dev)
static void
hinic3_dev_tnl_tso_support(struct rte_eth_dev_info *info, struct hinic3_nic_dev *nic_dev)
{
+ if (HINIC3_SUPPORT_VXLAN_OFFLOAD(nic_dev))
+ info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_VXLAN_TNL_TSO;
+
if (HINIC3_SUPPORT_GENEVE_OFFLOAD(nic_dev))
info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_GENEVE_TNL_TSO;
+
if (HINIC3_SUPPORT_IPXIP_OFFLOAD(nic_dev))
info->tx_offload_capa |= RTE_ETH_TX_OFFLOAD_IPIP_TNL_TSO;
}
@@ -698,7 +702,6 @@ hinic3_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)
RTE_ETH_TX_OFFLOAD_SCTP_CKSUM |
RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM |
RTE_ETH_TX_OFFLOAD_TCP_TSO | RTE_ETH_TX_OFFLOAD_MULTI_SEGS;
- if (nic_dev->feature_cap & NIC_F_HTN_CMDQ)
hinic3_dev_tnl_tso_support(info, nic_dev);
info->hash_key_size = HINIC3_RSS_KEY_SIZE;
--
2.47.0.windows.2
^ permalink raw reply related
* [v4 0/1] net/hinic3: Fix VXLAN TSO issue
From: Feifei Wang @ 2026-05-26 12:18 UTC (permalink / raw)
To: dev
In-Reply-To: <20260520065817.931-2-wff_light@vip.163.com>
v1: The RTE_ETH_TX_OFFLOAD_VXLAN_TNL_TSO flag is added to support the VXLAN TSO function
v2: Modify the commit information issue and supplement the commit information
v3: Revise review comments. First, deterine whether the hardware supports it, then add the flag bit
v4: Fix the compilation error caused by leading spaces
Feifei Wang (1):
net/hinic3: Fix VXLAN TSO issue
drivers/net/hinic3/hinic3_ethdev.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
--
2.47.0.windows.2
^ permalink raw reply
* RE: [PATCH v5] mempool: improve cache behaviour and performance
From: Morten Brørup @ 2026-05-26 10:37 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, Andrew Rybchenko, Jingjing Wu, Praveen Shetty,
Hemant Agrawal, Sachin Saxena
In-Reply-To: <ahVqYpVXMn59Tl2d@bricha3-mobl1.ger.corp.intel.com>
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Tuesday, 26 May 2026 11.40
>
> On Tue, May 26, 2026 at 10:41:44AM +0200, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Friday, 22 May 2026 18.12
> > >
> > > On Sun, Apr 19, 2026 at 09:55:26AM +0000, Morten Brørup wrote:
> > > > This patch refactors the mempool cache to eliminate some
> unexpected
> > > > behaviour and reduce the mempool cache miss rate.
> > > >
> > >
> > > Agree in principle with most of these changes. As we dicussed at
> the
> > > DPDK
> > > summit conference, only issue I really have is with the threshold
> > > limits
> > > here - allocating and freeing only half the cache at a time seems
> > > overly
> > > conservative. Thinking about use-cases:
> > >
> > > 1 for apps where alloc + free (generally Rx+Tx) is on the same
> core(s),
> > > then we should run (almost) entirely out of cache.
> >
> > I strongly disagree about any goal to run the cache low.
> > The primary goal is to minimize the cache miss (refill and replenish)
> rate.
> >
> > > 2 for apps where we have alloc and free on different cores, then we
> > > have
> > > some caches always being filled and others always being emptied
> >
> > Agree.
> >
> > >
> > > For case #1, we only need worry about the thresholds for the odd
> case
> > > where we have a burst that causes us to overflow our cache (and we
> > > can't increase our cache size to cope and avoid that).
> > > Otherwise the thresholds don't matter.
> >
> > It seems like you assume the application only does something like
> this:
> > Rx -> Rewrite -> Tx
> >
> > In that case, the per-lcore cache only needs capacity for one burst,
> yes.
> > With my patch, the cache can be rightsized by requesting a cache size
> of 2 * burst size.
> > (Then the fill level will be either size/2 or empty, i.e. one burst
> or zero.
> > This also happens to meet your suggested goal about low fill level,
> which I disagree with.)
> >
> > However, I don't think that is a realistic use case.
> > Many apps do something like this:
> >
> > |-> Rewrite ->|
> > Rx ->| |-> Tx
> > |-> Hold |
> > Release ->|
> >
>
> I agree, that's why our cache size needs to be bigger than our burst
> size.
>
> > They often hold back packets before they are transmitted.
> > For a simple router, when the destination IP address is not in the
> neighbor table, packets to that IP address are queued until ARP/ND has
> been resolved, and then they are dequeued and transmitted.
> > Or apps performing shaping or pacing, where packets are held back in
> queues, and dequeued at the appropriate time.
> > For such apps, the waves are much bigger (than the simple Rx-
> >Rewrite->Tx use case).
> >
> > With a random enqueue/dequeue pattern, replenishing/draining the
> cache to size/2 minimizes the probability of reaching one of its edges
> (empty or full), triggering a "cache miss" (refill/replenish).
> >
> > > However, for case #2, the thresholds are constantly involved as
> > > we
> > > always are going to backing store. In this case, we really want to
> have
> > > the
> > > allocs *always* fill the cache completely, and the frees completely
> > > empty
> > > the cache.
> >
> > Agree.
> >
> > >
> > > Because of this, while we want to avoid cases where we fill the
> cache
> > > completely only to have a further free causing it to be flushed,
> > > because of
> > > case #2 we cannot be overly conservative in how much we free/empty.
> > > Ideally, we want to fill to full less a single burst, and empty
> leaving
> > > only a single burst in the cache. Unfortunately, we don't know what
> > > those
> > > burst limits are, so we have to try and guess the best behaviour
> from
> > > everything else.
> >
> > I agree about not wanting to be overly conservative.
> > But in the use cases I have described for #1, I don't think a target
> fill level of size/2 is overly conservative.
> >
> > I also acknowledge that this patch doubles the mempool cache miss
> rate for #2.
> > E.g. with a cache size of 512 and burst size of 64, the per-burst
> miss rate will be 64 / (1/2 * 512) = 1/4, compared to 64 / 512 = 1/8
> with a full replenish/drain algorithm.
> >
> > In theory, we could make it build time configurable to optimize
> mempools for #2.
> > But mempools are also used for other objects than mbufs, so that
> would have unwanted side effects for non-mbuf mempools.
> >
> > If we went for an algorithm targeting replenish/drain at 25 % from
> the edges, the per-burst miss rate for #2 would be: 64 / (3/4 * 512) =
> 1/6.
> >
> > How about addressing #2 in the release notes:
> > We describe that the cache refill/drain algorithm has been changed to
> only refill/drain to 50 % of the cache size, so pipelined applications
> performing Rx (mempool get) and Tx (mempool put) on separate cores
> should configure their mbuf pools with double the cache size of what
> they previously were to achieve similar performance.
> >
> > >
> > > All that said, commits with specific suggestions inline.
> > >
> > > /Bruce
> > >
> > > <snip>
> > >
> > > > diff --git a/lib/mempool/rte_mempool.h
> b/lib/mempool/rte_mempool.h
> > > > index 2e54fc4466..432c43ab15 100644
> > > > --- a/lib/mempool/rte_mempool.h
> > > > +++ b/lib/mempool/rte_mempool.h
> > > > @@ -89,7 +89,7 @@ struct __rte_cache_aligned
> rte_mempool_debug_stats
> > > {
> > > > */
> > > > struct __rte_cache_aligned rte_mempool_cache {
> > > > uint32_t size; /**< Size of the cache */
> > > > - uint32_t flushthresh; /**< Threshold before we flush excess
> > > elements */
> > > > + uint32_t flushthresh; /**< Obsolete; for API/ABI
> compatibility
> > > purposes only */
> > > > uint32_t len; /**< Current cache count */
> > > > #ifdef RTE_LIBRTE_MEMPOOL_STATS
> > > > uint32_t unused;
> > > > @@ -107,8 +107,10 @@ struct __rte_cache_aligned rte_mempool_cache
> {
> > > > /**
> > > > * Cache objects
> > > > *
> > > > - * Cache is allocated to this size to allow it to overflow
> in
> > > certain
> > > > - * cases to avoid needless emptying of cache.
> > > > + * Note:
> > > > + * Cache is allocated at double size for API/ABI
> compatibility
> > > purposes only.
> > > > + * When reducing its size at an API/ABI breaking release,
> > > > + * remember to add a cache guard after it.
> > > > */
> > > > alignas(RTE_CACHE_LINE_SIZE) void
> > > *objs[RTE_MEMPOOL_CACHE_MAX_SIZE * 2];
> > > > };
> > > > @@ -1046,12 +1048,17 @@ rte_mempool_free(struct rte_mempool *mp);
> > > > * @param cache_size
> > > > * If cache_size is non-zero, the rte_mempool library will try
> to
> > > > * limit the accesses to the common lockless pool, by
> maintaining
> > > a
> > > > - * per-lcore object cache. This argument must be lower or
> equal to
> > > > - * RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.
> > > > + * per-lcore object cache. This argument must be an even
> number,
> > > > + * lower or equal to RTE_MEMPOOL_CACHE_MAX_SIZE and n.
> > > > * The access to the per-lcore table is of course
> > > > * faster than the multi-producer/consumer pool. The cache can
> be
> > > > * disabled if the cache_size argument is set to 0; it can be
> > > useful to
> > > > * avoid losing objects in cache.
> > > > + * Note:
> > > > + * Mempool put/get requests of more than cache_size / 2
> objects
> > > may be
> > > > + * partially or fully served directly by the multi-
> > > producer/consumer
> > > > + * pool, to avoid the overhead of copying the objects twice
> > > (instead of
> > > > + * once) when using the cache as a bounce buffer.
> > > > * @param private_data_size
> > > > * The size of the private data appended after the mempool
> > > > * structure. This is useful for storing some private data
> after
> > > the
> > > > @@ -1390,24 +1397,32 @@ rte_mempool_do_generic_put(struct
> rte_mempool
> > > *mp, void * const *obj_table,
> > > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
> > > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
> > > >
> > > > - __rte_assume(cache->flushthresh <=
> RTE_MEMPOOL_CACHE_MAX_SIZE *
> > > 2);
> > > > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > > > - __rte_assume(cache->len <= cache->flushthresh);
> > > > - if (likely(cache->len + n <= cache->flushthresh)) {
> > > > + __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE
> / 2);
> > > > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > > + __rte_assume(cache->len <= cache->size);
> > > > + if (likely(cache->len + n <= cache->size)) {
> > > > /* Sufficient room in the cache for the objects. */
> > > > cache_objs = &cache->objs[cache->len];
> > > > cache->len += n;
> > > > - } else if (n <= cache->flushthresh) {
> > > > + } else if (n <= cache->size / 2) {
> > > > /*
> > > > - * The cache is big enough for the objects, but - as
> > > detected by
> > > > - * the comparison above - has insufficient room for
> them.
> > > > - * Flush the cache to make room for the objects.
> > > > + * The number of objects is within the cache bounce
> buffer
> > > limit,
> > > > + * but - as detected by the comparison above - the
> cache
> > > has
> > > > + * insufficient room for them.
> > > > + * Flush the cache to the backend to make room for
> the
> > > objects;
> > > > + * flush (size / 2) objects from the bottom of the
> cache,
> > > where
> > > > + * objects are less hot, and move down the remaining
> > > objects, which
> > > > + * are more hot, from the upper half of the cache.
> > > > */
> > > > - cache_objs = &cache->objs[0];
> > > > - rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache-
> >len);
> > > > - cache->len = n;
> > > > + __rte_assume(cache->len > cache->size / 2);
> > > > + rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0],
> cache-
> > > >size / 2);
> > > > + rte_memcpy(&cache->objs[0], &cache->objs[cache->size
> / 2],
> > > > + sizeof(void *) * (cache->len - cache-
> >size /
> > > 2));
> > > > + cache_objs = &cache->objs[cache->len - cache->size /
> 2];
> > > > + cache->len = cache->len - cache->size / 2 + n;
> > >
> > > The flushing of only half the cache I'm not so certain about. I
> agree
> > > that
> > > we want to not flush to empty, but I also think that we want to do
> more
> > > than a half-flush, especially since we do an enqueue to the cache
> > > immediately afterwards. Consider the case where we have a cache
> size of
> > > 128, and we do an enqueue of 32, with the cache currently full. In
> that
> > > case we only flush 64, reducing the cache to 64, but then
> immediately
> > > bringing it back up to 96.
> >
> > I thought in depth about whether the flush/replenish sizes should
> consider the request size or not. (E.g. if I should replenish size/2 or
> size/2+request.)
> > I decided for not considering the request size, for two reasons:
> > a) It roughly doesn't matter, especially when considering a sequence
> of random get/put requests.
> > b) The size of the backend transactions become fixed, which has
> performance benefits: With my patch, they are always size/2, so if the
> cache size is 2^N, the backend transactions are 2^N and CPU cache
> aligned.
> >
> > > For cases where we have pipelines with all
> > > alloc
> > > on one core and all free on another, this half-flush would be
> > > inefficient.
> > >
> > > Instead, I would look to have a lower target threshold post-flush,
> and
> > > I
> > > would suggest 25% - taking into account the newly freed buffers.
> >
> > It's not good for #1.
> > I agree that it is better for #2. But I don't think #2 is the likely
> use case.
> >
> > After our discussion at the summit, I did start working a patch
> targeting fill levels at 25% from the cache edges, but I don't think
> it's better; so I'd rather stick with a target fill level of 50%.
> >
> > > For example:
> > >
> > > /* if n > our target of 1/4 full, flush everything,
> > > * else flush so that we end up with 1/4 full after n added.
> > > */
> > > flush_count = n > cache->size/4 ? cache->len :
> > > (cache->len + n) - cache->size/4;
> > >
> > >
> > > > } else {
> > > > - /* The request itself is too big for the cache. */
> > > > + /* The request itself is too big. */
> > > > goto driver_enqueue_stats_incremented;
> > >
> > > I think original comment is better. The request itself is not too
> big
> > > for
> > > the whole mempool, just for the cache.
> >
> > Ack.
> >
> > >
> > > > }
> > > >
> > > > @@ -1524,7 +1539,7 @@ rte_mempool_do_generic_get(struct
> rte_mempool
> > > *mp, void **obj_table,
> > > > /* The cache is a stack, so copy will be in reverse order.
> */
> > > > cache_objs = &cache->objs[cache->len];
> > > >
> > > > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > > > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > > if (likely(n <= cache->len)) {
> > > > /* The entire request can be satisfied from the
> cache. */
> > > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk,
> 1);
> > > > @@ -1548,13 +1563,13 @@ rte_mempool_do_generic_get(struct
> rte_mempool
> > > *mp, void **obj_table,
> > > > for (index = 0; index < len; index++)
> > > > *obj_table++ = *--cache_objs;
> > > >
> > > > - /* Dequeue below would overflow mem allocated for cache? */
> > > > - if (unlikely(remaining > RTE_MEMPOOL_CACHE_MAX_SIZE))
> > > > + /* Dequeue below would exceed the cache bounce buffer
> limit? */
> > > > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE
> / 2);
> > > > + if (unlikely(remaining > cache->size / 2))
> > > > goto driver_dequeue;
> > > >
> > > > - /* Fill the cache from the backend; fetch size + remaining
> > > objects. */
> > > > - ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs,
> > > > - cache->size + remaining);
> > > > + /* Fill the cache from the backend; fetch (size / 2)
> objects. */
> > > > + ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache-
> >size /
> > > 2);
> > >
> > > Again, the cache->size / 2 doesn't seem right here. We at most
> half-
> > > fill
> > > the cache and then take some objects from that, meaning that have
> just
> > > done
> > > a re-fill of cache but end the function with it less than half
> full.
> > > Since
> > > we take from this value, I'd suggest just filling the cache
> completely.
> >
> > The issues at the edges of the cache are symmetrical.
> > If we replenish the cache to full, and the next transaction is a put,
> the cache needs to be drained.
> > That's why I replenish to size/2.
> >
>
> Not necessarily. After you fill the cache here (to 50%) you then take
> elements from it. If we filled to 100%, then we'd only hit a flush if
> we
> freed back more elements than we just allocated. Now, that's reasonably
> likely which is why I'm ok to not filling to 100%. However, doing a
> refill
> as part of the op, and then leaving the cache less than half full after
> the
> op finishes is just wasteful IMHO!
I'm targeting half full, and disregard if it is pre-op or post-op.
This makes the backend transactions cache aligned in both addressing and size, which opens an opportunity to performance optimize the mempool drivers for this.
>
> One other point. The obvious worst case scenario we want to avoid is
> constant fill/empty/fill/empty sequences. However, so long as we have
> an
> asymmetry in how we do fills and empty thresholds, a repeating sequence
> should never occur, and even pairs of fill/empty should be very rare.
> Consider the case, for example, where we fill to 100% but only drain to
> 25%. For this, we can ignore scenario #2, since we should have pretty
> good
> cache usage. For scenario #1, of allocs/frees on a single core,
> randomly
> interleaved, our worst case is:
> 1) alloc with cache empty, in which case we fill to 100%, then take the
> alloc
> 2) have a free of a burst greater than that which we just allocated,
> causing an immediate flush again.
>
> That's pretty poor behaviour, but the thing is that after the flush we
> now
> have mempool at 25% + freed burst - so expected between 25-50% of cache
> utilization. That's the sweet spot we want to target - after each
> operation, the mempool cache should ideally be at 50%. Whether the next
> op
> is an alloc or a free, our cache can handle it, and likely a couple of
> each
> in sequence. Therefore, our possible fill/empty combinations are likely
> to
> be rare occurances.
>
> [In all this, I am making the assumption that burst size is well less
> than
> cache size. Also, similar logic would be applicable for the inverse
> scenario, e.g. flush to empty (and fill burst) and fill to 75%]
I'm not so sure about this assumption.
With a cache size of 512 and a bursts of 64, the cache only holds 8 bursts.
50% is 4 bursts, and 25% is only 2 bursts.
Using a replenish/drain level in the middle requires 5 bursts in either direction to pass the edge (and trigger replenish/flush).
Using a replenish/drain level 25% from the edge requires only 3 bursts in the wrong direction to pass the edge (and trigger replenish/flush). Much higher probability with random get/put.
>
> Now, all said, I tend to agree that we want to leave space for a decent
> size burst after a fill. That is why I think that filling to 75% is
> reasonable. After an alloc that triggers a fill, I don't want the cache
> less than 50% full, but not completely full so there is room for a free
> without a flush, and similarly for a free that triggers a flush, the
> cache
> should not be empty, but also should not be more than half full.
>
> One suggestion - we could always add a simple tunable that specifies
> the
> margin, or reserved entries for alloc and free. We can then guide in
> the
> docs that the value should be e.g. "zero for apps where alloc and free
> take
> place on different cores. 20%-50% of cache is recommended where alloc
> and
> free take place on the same core"
Yes, a simple tunable is a really good idea.
At this point, I think we should optimize for use case #1, and go for the 50% fill level.
Then we can add a tunable to optimize for use case #2 later. I will try to come up with a draft for such a follow-up patch within the next few days.
The 50% fill level in this patch is not as bad for use case #2 (roughly doubling the burst miss rate from 1/8 to 1/4), compared to how bad the original algorithm is for use case #1 (very high miss probability - only two ops in the wrong direction - after drain/replenish).
-Morten
^ permalink raw reply
* Re: [PATCH v3 04/25] dma/idxd: clear device at scan
From: Bruce Richardson @ 2026-05-26 9:50 UTC (permalink / raw)
To: David Marchand; +Cc: dev, thomas, stephen, Kevin Laatz
In-Reply-To: <CAJFAV8zC=LUR8r4s7q5o9W2KaAa3PjKD8+qoAUsV70Ddqvgp1A@mail.gmail.com>
On Tue, May 26, 2026 at 11:48:46AM +0200, David Marchand wrote:
> On Tue, 26 May 2026 at 11:42, Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Tue, May 26, 2026 at 10:41:44AM +0200, David Marchand wrote:
> > > Currently, the device object is only manipulated by the dma/idxd bus
> > > callbacks and EAL is not looking too much into this object.
> > >
> > > However, in the next refactoring, EAL will expect a clean object, like
> > > when checking that the device has been already probed
> > > (iow dev->driver != NULL).
> > >
> > > Request a 0'd object when allocating.
> > >
> > > Reported-by: Bruce Richardson <bruce.richardson@intel.com>
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> > > drivers/dma/idxd/idxd_bus.c | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> >
> > Out of interest, did you (or Claude :-) ) run a scan over the other bus
> > drivers to see if any others are similarly not properly zeroed on init?
>
> I did check, the "legacy" way.
>
> All other are calling calloc(), or rte_zmalloc_*, or manually clearing.
> I hesitated at changing the malloc+memset pattern to calloc, but I
> left it for later.
> The series is big enough.
>
Great, thanks for confirming. I also agree with your choice to not bother
changing the patterns for this set.
/Bruce
^ permalink raw reply
* Re: [PATCH v3 04/25] dma/idxd: clear device at scan
From: David Marchand @ 2026-05-26 9:48 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev, thomas, stephen, Kevin Laatz
In-Reply-To: <ahVq9dJrOK04Ujwd@bricha3-mobl1.ger.corp.intel.com>
On Tue, 26 May 2026 at 11:42, Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Tue, May 26, 2026 at 10:41:44AM +0200, David Marchand wrote:
> > Currently, the device object is only manipulated by the dma/idxd bus
> > callbacks and EAL is not looking too much into this object.
> >
> > However, in the next refactoring, EAL will expect a clean object, like
> > when checking that the device has been already probed
> > (iow dev->driver != NULL).
> >
> > Request a 0'd object when allocating.
> >
> > Reported-by: Bruce Richardson <bruce.richardson@intel.com>
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> > drivers/dma/idxd/idxd_bus.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
>
> Out of interest, did you (or Claude :-) ) run a scan over the other bus
> drivers to see if any others are similarly not properly zeroed on init?
I did check, the "legacy" way.
All other are calling calloc(), or rte_zmalloc_*, or manually clearing.
I hesitated at changing the malloc+memset pattern to calloc, but I
left it for later.
The series is big enough.
--
David Marchand
^ permalink raw reply
* Re: [PATCH v3 04/25] dma/idxd: clear device at scan
From: Bruce Richardson @ 2026-05-26 9:42 UTC (permalink / raw)
To: David Marchand; +Cc: dev, thomas, stephen, Kevin Laatz
In-Reply-To: <20260526084212.3145685-5-david.marchand@redhat.com>
On Tue, May 26, 2026 at 10:41:44AM +0200, David Marchand wrote:
> Currently, the device object is only manipulated by the dma/idxd bus
> callbacks and EAL is not looking too much into this object.
>
> However, in the next refactoring, EAL will expect a clean object, like
> when checking that the device has been already probed
> (iow dev->driver != NULL).
>
> Request a 0'd object when allocating.
>
> Reported-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> drivers/dma/idxd/idxd_bus.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
Out of interest, did you (or Claude :-) ) run a scan over the other bus
drivers to see if any others are similarly not properly zeroed on init?
/Bruce
> diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c
> index 291cd6c707..f267c20a59 100644
> --- a/drivers/dma/idxd/idxd_bus.c
> +++ b/drivers/dma/idxd/idxd_bus.c
> @@ -322,7 +322,7 @@ dsa_scan(void)
> }
> IDXD_PMD_DEBUG("%s(): found %s/%s", __func__, path, wq->d_name);
>
> - dev = malloc(sizeof(*dev));
> + dev = calloc(1, sizeof(*dev));
> if (dev == NULL) {
> closedir(dev_dir);
> return -ENOMEM;
> --
> 2.53.0
>
^ permalink raw reply
* Re: [PATCH v3 04/25] dma/idxd: clear device at scan
From: Bruce Richardson @ 2026-05-26 9:41 UTC (permalink / raw)
To: David Marchand; +Cc: dev, thomas, stephen, Kevin Laatz
In-Reply-To: <20260526084212.3145685-5-david.marchand@redhat.com>
On Tue, May 26, 2026 at 10:41:44AM +0200, David Marchand wrote:
> Currently, the device object is only manipulated by the dma/idxd bus
> callbacks and EAL is not looking too much into this object.
>
> However, in the next refactoring, EAL will expect a clean object, like
> when checking that the device has been already probed
> (iow dev->driver != NULL).
>
> Request a 0'd object when allocating.
>
> Reported-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> drivers/dma/idxd/idxd_bus.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c
> index 291cd6c707..f267c20a59 100644
> --- a/drivers/dma/idxd/idxd_bus.c
> +++ b/drivers/dma/idxd/idxd_bus.c
> @@ -322,7 +322,7 @@ dsa_scan(void)
> }
> IDXD_PMD_DEBUG("%s(): found %s/%s", __func__, path, wq->d_name);
>
> - dev = malloc(sizeof(*dev));
> + dev = calloc(1, sizeof(*dev));
> if (dev == NULL) {
> closedir(dev_dir);
> return -ENOMEM;
> --
> 2.53.0
>
^ permalink raw reply
* Re: [PATCH v5] mempool: improve cache behaviour and performance
From: Bruce Richardson @ 2026-05-26 9:39 UTC (permalink / raw)
To: Morten Brørup
Cc: dev, Andrew Rybchenko, Jingjing Wu, Praveen Shetty,
Hemant Agrawal, Sachin Saxena
In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35F6589A@smartserver.smartshare.dk>
On Tue, May 26, 2026 at 10:41:44AM +0200, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Friday, 22 May 2026 18.12
> >
> > On Sun, Apr 19, 2026 at 09:55:26AM +0000, Morten Brørup wrote:
> > > This patch refactors the mempool cache to eliminate some unexpected
> > > behaviour and reduce the mempool cache miss rate.
> > >
> >
> > Agree in principle with most of these changes. As we dicussed at the
> > DPDK
> > summit conference, only issue I really have is with the threshold
> > limits
> > here - allocating and freeing only half the cache at a time seems
> > overly
> > conservative. Thinking about use-cases:
> >
> > 1 for apps where alloc + free (generally Rx+Tx) is on the same core(s),
> > then we should run (almost) entirely out of cache.
>
> I strongly disagree about any goal to run the cache low.
> The primary goal is to minimize the cache miss (refill and replenish) rate.
>
> > 2 for apps where we have alloc and free on different cores, then we
> > have
> > some caches always being filled and others always being emptied
>
> Agree.
>
> >
> > For case #1, we only need worry about the thresholds for the odd case
> > where we have a burst that causes us to overflow our cache (and we
> > can't increase our cache size to cope and avoid that).
> > Otherwise the thresholds don't matter.
>
> It seems like you assume the application only does something like this:
> Rx -> Rewrite -> Tx
>
> In that case, the per-lcore cache only needs capacity for one burst, yes.
> With my patch, the cache can be rightsized by requesting a cache size of 2 * burst size.
> (Then the fill level will be either size/2 or empty, i.e. one burst or zero.
> This also happens to meet your suggested goal about low fill level, which I disagree with.)
>
> However, I don't think that is a realistic use case.
> Many apps do something like this:
>
> |-> Rewrite ->|
> Rx ->| |-> Tx
> |-> Hold |
> Release ->|
>
I agree, that's why our cache size needs to be bigger than our burst size.
> They often hold back packets before they are transmitted.
> For a simple router, when the destination IP address is not in the neighbor table, packets to that IP address are queued until ARP/ND has been resolved, and then they are dequeued and transmitted.
> Or apps performing shaping or pacing, where packets are held back in queues, and dequeued at the appropriate time.
> For such apps, the waves are much bigger (than the simple Rx->Rewrite->Tx use case).
>
> With a random enqueue/dequeue pattern, replenishing/draining the cache to size/2 minimizes the probability of reaching one of its edges (empty or full), triggering a "cache miss" (refill/replenish).
>
> > However, for case #2, the thresholds are constantly involved as
> > we
> > always are going to backing store. In this case, we really want to have
> > the
> > allocs *always* fill the cache completely, and the frees completely
> > empty
> > the cache.
>
> Agree.
>
> >
> > Because of this, while we want to avoid cases where we fill the cache
> > completely only to have a further free causing it to be flushed,
> > because of
> > case #2 we cannot be overly conservative in how much we free/empty.
> > Ideally, we want to fill to full less a single burst, and empty leaving
> > only a single burst in the cache. Unfortunately, we don't know what
> > those
> > burst limits are, so we have to try and guess the best behaviour from
> > everything else.
>
> I agree about not wanting to be overly conservative.
> But in the use cases I have described for #1, I don't think a target fill level of size/2 is overly conservative.
>
> I also acknowledge that this patch doubles the mempool cache miss rate for #2.
> E.g. with a cache size of 512 and burst size of 64, the per-burst miss rate will be 64 / (1/2 * 512) = 1/4, compared to 64 / 512 = 1/8 with a full replenish/drain algorithm.
>
> In theory, we could make it build time configurable to optimize mempools for #2.
> But mempools are also used for other objects than mbufs, so that would have unwanted side effects for non-mbuf mempools.
>
> If we went for an algorithm targeting replenish/drain at 25 % from the edges, the per-burst miss rate for #2 would be: 64 / (3/4 * 512) = 1/6.
>
> How about addressing #2 in the release notes:
> We describe that the cache refill/drain algorithm has been changed to only refill/drain to 50 % of the cache size, so pipelined applications performing Rx (mempool get) and Tx (mempool put) on separate cores should configure their mbuf pools with double the cache size of what they previously were to achieve similar performance.
>
> >
> > All that said, commits with specific suggestions inline.
> >
> > /Bruce
> >
> > <snip>
> >
> > > diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
> > > index 2e54fc4466..432c43ab15 100644
> > > --- a/lib/mempool/rte_mempool.h
> > > +++ b/lib/mempool/rte_mempool.h
> > > @@ -89,7 +89,7 @@ struct __rte_cache_aligned rte_mempool_debug_stats
> > {
> > > */
> > > struct __rte_cache_aligned rte_mempool_cache {
> > > uint32_t size; /**< Size of the cache */
> > > - uint32_t flushthresh; /**< Threshold before we flush excess
> > elements */
> > > + uint32_t flushthresh; /**< Obsolete; for API/ABI compatibility
> > purposes only */
> > > uint32_t len; /**< Current cache count */
> > > #ifdef RTE_LIBRTE_MEMPOOL_STATS
> > > uint32_t unused;
> > > @@ -107,8 +107,10 @@ struct __rte_cache_aligned rte_mempool_cache {
> > > /**
> > > * Cache objects
> > > *
> > > - * Cache is allocated to this size to allow it to overflow in
> > certain
> > > - * cases to avoid needless emptying of cache.
> > > + * Note:
> > > + * Cache is allocated at double size for API/ABI compatibility
> > purposes only.
> > > + * When reducing its size at an API/ABI breaking release,
> > > + * remember to add a cache guard after it.
> > > */
> > > alignas(RTE_CACHE_LINE_SIZE) void
> > *objs[RTE_MEMPOOL_CACHE_MAX_SIZE * 2];
> > > };
> > > @@ -1046,12 +1048,17 @@ rte_mempool_free(struct rte_mempool *mp);
> > > * @param cache_size
> > > * If cache_size is non-zero, the rte_mempool library will try to
> > > * limit the accesses to the common lockless pool, by maintaining
> > a
> > > - * per-lcore object cache. This argument must be lower or equal to
> > > - * RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.
> > > + * per-lcore object cache. This argument must be an even number,
> > > + * lower or equal to RTE_MEMPOOL_CACHE_MAX_SIZE and n.
> > > * The access to the per-lcore table is of course
> > > * faster than the multi-producer/consumer pool. The cache can be
> > > * disabled if the cache_size argument is set to 0; it can be
> > useful to
> > > * avoid losing objects in cache.
> > > + * Note:
> > > + * Mempool put/get requests of more than cache_size / 2 objects
> > may be
> > > + * partially or fully served directly by the multi-
> > producer/consumer
> > > + * pool, to avoid the overhead of copying the objects twice
> > (instead of
> > > + * once) when using the cache as a bounce buffer.
> > > * @param private_data_size
> > > * The size of the private data appended after the mempool
> > > * structure. This is useful for storing some private data after
> > the
> > > @@ -1390,24 +1397,32 @@ rte_mempool_do_generic_put(struct rte_mempool
> > *mp, void * const *obj_table,
> > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
> > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
> > >
> > > - __rte_assume(cache->flushthresh <= RTE_MEMPOOL_CACHE_MAX_SIZE *
> > 2);
> > > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > > - __rte_assume(cache->len <= cache->flushthresh);
> > > - if (likely(cache->len + n <= cache->flushthresh)) {
> > > + __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> > > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > + __rte_assume(cache->len <= cache->size);
> > > + if (likely(cache->len + n <= cache->size)) {
> > > /* Sufficient room in the cache for the objects. */
> > > cache_objs = &cache->objs[cache->len];
> > > cache->len += n;
> > > - } else if (n <= cache->flushthresh) {
> > > + } else if (n <= cache->size / 2) {
> > > /*
> > > - * The cache is big enough for the objects, but - as
> > detected by
> > > - * the comparison above - has insufficient room for them.
> > > - * Flush the cache to make room for the objects.
> > > + * The number of objects is within the cache bounce buffer
> > limit,
> > > + * but - as detected by the comparison above - the cache
> > has
> > > + * insufficient room for them.
> > > + * Flush the cache to the backend to make room for the
> > objects;
> > > + * flush (size / 2) objects from the bottom of the cache,
> > where
> > > + * objects are less hot, and move down the remaining
> > objects, which
> > > + * are more hot, from the upper half of the cache.
> > > */
> > > - cache_objs = &cache->objs[0];
> > > - rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
> > > - cache->len = n;
> > > + __rte_assume(cache->len > cache->size / 2);
> > > + rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0], cache-
> > >size / 2);
> > > + rte_memcpy(&cache->objs[0], &cache->objs[cache->size / 2],
> > > + sizeof(void *) * (cache->len - cache->size /
> > 2));
> > > + cache_objs = &cache->objs[cache->len - cache->size / 2];
> > > + cache->len = cache->len - cache->size / 2 + n;
> >
> > The flushing of only half the cache I'm not so certain about. I agree
> > that
> > we want to not flush to empty, but I also think that we want to do more
> > than a half-flush, especially since we do an enqueue to the cache
> > immediately afterwards. Consider the case where we have a cache size of
> > 128, and we do an enqueue of 32, with the cache currently full. In that
> > case we only flush 64, reducing the cache to 64, but then immediately
> > bringing it back up to 96.
>
> I thought in depth about whether the flush/replenish sizes should consider the request size or not. (E.g. if I should replenish size/2 or size/2+request.)
> I decided for not considering the request size, for two reasons:
> a) It roughly doesn't matter, especially when considering a sequence of random get/put requests.
> b) The size of the backend transactions become fixed, which has performance benefits: With my patch, they are always size/2, so if the cache size is 2^N, the backend transactions are 2^N and CPU cache aligned.
>
> > For cases where we have pipelines with all
> > alloc
> > on one core and all free on another, this half-flush would be
> > inefficient.
> >
> > Instead, I would look to have a lower target threshold post-flush, and
> > I
> > would suggest 25% - taking into account the newly freed buffers.
>
> It's not good for #1.
> I agree that it is better for #2. But I don't think #2 is the likely use case.
>
> After our discussion at the summit, I did start working a patch targeting fill levels at 25% from the cache edges, but I don't think it's better; so I'd rather stick with a target fill level of 50%.
>
> > For example:
> >
> > /* if n > our target of 1/4 full, flush everything,
> > * else flush so that we end up with 1/4 full after n added.
> > */
> > flush_count = n > cache->size/4 ? cache->len :
> > (cache->len + n) - cache->size/4;
> >
> >
> > > } else {
> > > - /* The request itself is too big for the cache. */
> > > + /* The request itself is too big. */
> > > goto driver_enqueue_stats_incremented;
> >
> > I think original comment is better. The request itself is not too big
> > for
> > the whole mempool, just for the cache.
>
> Ack.
>
> >
> > > }
> > >
> > > @@ -1524,7 +1539,7 @@ rte_mempool_do_generic_get(struct rte_mempool
> > *mp, void **obj_table,
> > > /* The cache is a stack, so copy will be in reverse order. */
> > > cache_objs = &cache->objs[cache->len];
> > >
> > > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > if (likely(n <= cache->len)) {
> > > /* The entire request can be satisfied from the cache. */
> > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
> > > @@ -1548,13 +1563,13 @@ rte_mempool_do_generic_get(struct rte_mempool
> > *mp, void **obj_table,
> > > for (index = 0; index < len; index++)
> > > *obj_table++ = *--cache_objs;
> > >
> > > - /* Dequeue below would overflow mem allocated for cache? */
> > > - if (unlikely(remaining > RTE_MEMPOOL_CACHE_MAX_SIZE))
> > > + /* Dequeue below would exceed the cache bounce buffer limit? */
> > > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> > > + if (unlikely(remaining > cache->size / 2))
> > > goto driver_dequeue;
> > >
> > > - /* Fill the cache from the backend; fetch size + remaining
> > objects. */
> > > - ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs,
> > > - cache->size + remaining);
> > > + /* Fill the cache from the backend; fetch (size / 2) objects. */
> > > + ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache->size /
> > 2);
> >
> > Again, the cache->size / 2 doesn't seem right here. We at most half-
> > fill
> > the cache and then take some objects from that, meaning that have just
> > done
> > a re-fill of cache but end the function with it less than half full.
> > Since
> > we take from this value, I'd suggest just filling the cache completely.
>
> The issues at the edges of the cache are symmetrical.
> If we replenish the cache to full, and the next transaction is a put, the cache needs to be drained.
> That's why I replenish to size/2.
>
Not necessarily. After you fill the cache here (to 50%) you then take
elements from it. If we filled to 100%, then we'd only hit a flush if we
freed back more elements than we just allocated. Now, that's reasonably
likely which is why I'm ok to not filling to 100%. However, doing a refill
as part of the op, and then leaving the cache less than half full after the
op finishes is just wasteful IMHO!
One other point. The obvious worst case scenario we want to avoid is
constant fill/empty/fill/empty sequences. However, so long as we have an
asymmetry in how we do fills and empty thresholds, a repeating sequence
should never occur, and even pairs of fill/empty should be very rare.
Consider the case, for example, where we fill to 100% but only drain to
25%. For this, we can ignore scenario #2, since we should have pretty good
cache usage. For scenario #1, of allocs/frees on a single core, randomly
interleaved, our worst case is:
1) alloc with cache empty, in which case we fill to 100%, then take the
alloc
2) have a free of a burst greater than that which we just allocated,
causing an immediate flush again.
That's pretty poor behaviour, but the thing is that after the flush we now
have mempool at 25% + freed burst - so expected between 25-50% of cache
utilization. That's the sweet spot we want to target - after each
operation, the mempool cache should ideally be at 50%. Whether the next op
is an alloc or a free, our cache can handle it, and likely a couple of each
in sequence. Therefore, our possible fill/empty combinations are likely to
be rare occurances.
[In all this, I am making the assumption that burst size is well less than
cache size. Also, similar logic would be applicable for the inverse
scenario, e.g. flush to empty (and fill burst) and fill to 75%]
Now, all said, I tend to agree that we want to leave space for a decent
size burst after a fill. That is why I think that filling to 75% is
reasonable. After an alloc that triggers a fill, I don't want the cache
less than 50% full, but not completely full so there is room for a free
without a flush, and similarly for a free that triggers a flush, the cache
should not be empty, but also should not be more than half full.
One suggestion - we could always add a simple tunable that specifies the
margin, or reserved entries for alloc and free. We can then guide in the
docs that the value should be e.g. "zero for apps where alloc and free take
place on different cores. 20%-50% of cache is recommended where alloc and
free take place on the same core"
/Bruce
^ permalink raw reply
* Re: [PATCH v3 00/25] Consolidate bus driver infrastructure
From: David Marchand @ 2026-05-26 9:03 UTC (permalink / raw)
To: dev; +Cc: thomas, stephen, bruce.richardson
In-Reply-To: <20260526084212.3145685-1-david.marchand@redhat.com>
On Tue, 26 May 2026 at 10:42, David Marchand <david.marchand@redhat.com> wrote:
>
> This is a continuation of the work I started on the bus infrastructure,
> but this time, a lot of the changes were done by a AI "friend".
> It is still an unfinished topic as the current series focuses on probing
> only. The detaching/cleanup aspect is postponed to another release/time.
>
> My AI "friend" really *sucked* at git and at separating unrelated changes,
> so it required quite a lot of massage/polishing afterwards.
> But it seems good enough now for upstream submission.
>
> I would like to see this series merged in 26.07, so that we have enough
> time to stabilize it before the next LTS.
> And seeing how it affects drivers, it is probably better to merge it
> the sooner possible (so Thomas does not have to solve too many conflicts
> when pulling next-* subtrees after, especially wrt the last patch).
>
>
> This series refactors the DPDK bus infrastructure to consolidate common
> operations and reduce code duplication across all bus drivers.
> Currently, each bus implements its own specific device/driver lists,
> probe logic, and lookup functions.
> This series moves these common patterns into the EAL bus layer,
> providing generic helpers that all buses can use.
>
> The refactoring removes approximately 1,400 lines of duplicated code across
> the codebase while maintaining full functional equivalence.
>
> Key changes:
> - Factorize device and driver lists into struct rte_bus
> - Implement generic probe, device/driver lookup, and iteration helpers in EAL
> - Introduce conversion macros (RTE_BUS_DEVICE, RTE_BUS_DRIVER, RTE_CLASS_TO_BUS_DEVICE)
> to safely convert between generic and bus-specific types
> - Remove bus-specific device/driver types from most driver code
> - Move probe logic from individual buses to rte_bus_generic_probe()
> - Separate NXP-specific metadata from generic bus structures
>
> Benefits:
> - Significant code reduction (~1,400 lines removed)
> - Consistent behavior across all bus types
> - Simplified bus driver implementation
> - Easier maintenance and future enhancements
>
> The series is structured as a progressive refactoring:
> - Remove redundant checks and helpers (patches 1-5)
> - Add conversion macros and factorize lists (patches 6-8)
> - Consolidate device/driver lookup and iteration (patches 9-11)
> - Refactor probe logic (patches 12-15)
> - Remove bus-specific types from drivers (patches 16-23)
>
> Note on ABI:
> This series breaks the ABI for drivers (changes to rte_pci_device,
> rte_pci_driver, and similar structures for other buses). However, the DPDK
> ABI policy does not provide guarantees for driver-level interfaces.
>
>
> --
> David Marchand
>
> Changes since v2:
> - fixed dma/idxd probing as reported by Bruce,
> - moved api_ver setting (from match to scan) in bus/uacce,
> - fixed transient bug in the vdev code (in the middle of the series).
> tl;dr the high level difference for bus/vdev between v2 and v3 == 0,
> - fixed doxygen,
> - enhanced API description,
There was some hiccup from Red Hat SMTP when sending the series..
hopefully my resending of the second half of the patches is good
enough.
At least, patchwork looks happy.
--
David Marchand
^ permalink raw reply
* [RFC v2 3/3] app/test: add fastmem test suite
From: Mattias Rönnblom @ 2026-05-26 8:57 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Mattias Rönnblom
In-Reply-To: <20260526085743.64396-1-hofors@lysator.liu.se>
Add functional, performance, and profiling test suites for the
fastmem library.
--
Signed-off-by: Mattias Rönnblom <hofors@lysator.liu.se>
---
app/test/meson.build | 3 +
app/test/test_fastmem.c | 1673 +++++++++++++++++++++++++++++++
app/test/test_fastmem_perf.c | 1040 +++++++++++++++++++
app/test/test_fastmem_profile.c | 157 +++
4 files changed, 2873 insertions(+)
create mode 100644 app/test/test_fastmem.c
create mode 100644 app/test/test_fastmem_perf.c
create mode 100644 app/test/test_fastmem_profile.c
diff --git a/app/test/meson.build b/app/test/meson.build
index 7d458f9c07..d11c63be6f 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -82,6 +82,9 @@ source_file_deps = {
'test_event_vector_adapter.c': ['eventdev', 'bus_vdev'],
'test_eventdev.c': ['eventdev', 'bus_vdev'],
'test_external_mem.c': [],
+ 'test_fastmem.c': ['fastmem'],
+ 'test_fastmem_perf.c': ['fastmem', 'mempool'],
+ 'test_fastmem_profile.c': ['fastmem'],
'test_fbarray.c': [],
'test_fib.c': ['net', 'fib'],
'test_fib6.c': ['rib', 'fib'],
diff --git a/app/test/test_fastmem.c b/app/test/test_fastmem.c
new file mode 100644
index 0000000000..6981de28be
--- /dev/null
+++ b/app/test/test_fastmem.c
@@ -0,0 +1,1673 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#include <errno.h>
+#include <inttypes.h>
+#include <stdalign.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_lcore.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_thread.h>
+
+#include <rte_fastmem.h>
+
+#include "test.h"
+
+#define FASTMEM_MEMZONE_SIZE (128U << 20)
+
+/*
+ * Count memzones whose names begin with the fastmem prefix.
+ * Used to verify that rte_fastmem_reserve() really did reserve
+ * backing memzones.
+ */
+static int fastmem_memzone_count;
+
+static void
+count_fastmem_memzones_walk(const struct rte_memzone *mz, void *arg)
+{
+ RTE_SET_USED(arg);
+
+ if (strncmp(mz->name, "fastmem_", strlen("fastmem_")) == 0)
+ fastmem_memzone_count++;
+}
+
+static unsigned int
+count_fastmem_memzones(void)
+{
+ fastmem_memzone_count = 0;
+ rte_memzone_walk(count_fastmem_memzones_walk, NULL);
+ return fastmem_memzone_count;
+}
+
+static int
+test_init_deinit(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_init() failed: %d", rc);
+
+ rte_fastmem_deinit();
+
+ /* A subsequent init/deinit cycle must succeed. */
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, 0, "second rte_fastmem_init() failed: %d", rc);
+
+ rte_fastmem_deinit();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_init_is_not_idempotent(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_init() failed: %d", rc);
+
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, -EBUSY,
+ "expected -EBUSY on re-init, got %d", rc);
+
+ rte_fastmem_deinit();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_deinit_without_init(void)
+{
+ /* Must be a no-op, not a crash. */
+ rte_fastmem_deinit();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_max_size(void)
+{
+ size_t max;
+
+ max = rte_fastmem_max_size();
+ TEST_ASSERT(max >= (1U << 20),
+ "max_size=%zu below required 1 MiB minimum", max);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_small(void)
+{
+ int socket_id;
+ unsigned int before, after;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ before = count_fastmem_memzones();
+
+ /*
+ * A small reserve request (1 byte) must result in exactly
+ * one memzone reservation: the internal rounding is to
+ * memzone granularity.
+ */
+ rc = rte_fastmem_reserve(1, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_reserve() failed: %d", rc);
+
+ after = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after - before, 1,
+ "expected 1 new memzone, got %u", after - before);
+
+ rte_fastmem_deinit();
+
+ /* After deinit the memzones must be released. */
+ TEST_ASSERT_EQUAL(count_fastmem_memzones(), 0,
+ "%u fastmem memzones leaked after deinit",
+ count_fastmem_memzones());
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_multiple_memzones(void)
+{
+ int socket_id;
+ unsigned int before, after;
+ size_t reserve_size;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ before = count_fastmem_memzones();
+
+ /*
+ * Request just over one memzone's worth; this must force
+ * a second memzone to be reserved.
+ */
+ reserve_size = FASTMEM_MEMZONE_SIZE + 1;
+ rc = rte_fastmem_reserve(reserve_size, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_reserve(%zu) failed: %d",
+ reserve_size, rc);
+
+ after = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after - before, 2,
+ "expected 2 new memzones for %zu-byte reserve, got %u",
+ reserve_size, after - before);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_cumulative(void)
+{
+ int socket_id;
+ unsigned int after_first, after_second;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ rc = rte_fastmem_reserve(FASTMEM_MEMZONE_SIZE, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "first reserve failed: %d", rc);
+
+ after_first = count_fastmem_memzones();
+
+ /*
+ * A second call requesting the same amount that's already
+ * reserved must not trigger any new memzone reservation.
+ */
+ rc = rte_fastmem_reserve(FASTMEM_MEMZONE_SIZE, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "second reserve failed: %d", rc);
+
+ after_second = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after_first, after_second,
+ "reserve of already-reserved amount added memzones (%u -> %u)",
+ after_first, after_second);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_invalid_socket(void)
+{
+ int rc;
+
+ rc = rte_fastmem_reserve(1, RTE_MAX_NUMA_NODES);
+ TEST_ASSERT_EQUAL(rc, -EINVAL,
+ "expected -EINVAL for out-of-range socket, got %d", rc);
+
+ rc = rte_fastmem_reserve(1, -2);
+ TEST_ASSERT_EQUAL(rc, -EINVAL,
+ "expected -EINVAL for negative socket, got %d", rc);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_without_init(void)
+{
+ int rc;
+
+ rc = rte_fastmem_reserve(1, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0,
+ "expected failure without init, got %d", rc);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_any_socket(void)
+{
+ unsigned int before, after;
+ int rc;
+
+ before = count_fastmem_memzones();
+
+ /*
+ * SOCKET_ID_ANY should succeed on any system with at least
+ * one configured socket. The allocator picks the caller's
+ * socket first and falls back to other sockets if needed.
+ */
+ rc = rte_fastmem_reserve(1, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0,
+ "rte_fastmem_reserve(SOCKET_ID_ANY) failed: %d", rc);
+
+ after = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after - before, 1,
+ "expected 1 new memzone, got %u", after - before);
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Stage 2 tests: allocation and free.
+ */
+
+static int
+test_alloc_too_big(void)
+{
+ void *p;
+ rte_errno = 0;
+ p = rte_fastmem_alloc(rte_fastmem_max_size() + 1, 0, 0);
+ TEST_ASSERT_NULL(p, "alloc above max_size returned non-NULL");
+ TEST_ASSERT_EQUAL(rte_errno, E2BIG,
+ "expected rte_errno=E2BIG, got %d", rte_errno);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_invalid_align(void)
+{
+ void *p;
+ rte_errno = 0;
+ p = rte_fastmem_alloc(16, 3, 0); /* 3 is not a power of 2 */
+ TEST_ASSERT_NULL(p, "alloc with align=3 returned non-NULL");
+ TEST_ASSERT_EQUAL(rte_errno, EINVAL,
+ "expected rte_errno=EINVAL, got %d", rte_errno);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_free_small(void)
+{
+ void *p;
+ p = rte_fastmem_alloc(8, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc(8) failed: rte_errno=%d", rte_errno);
+
+ /* Writing into the object must not crash. */
+ memset(p, 0xa5, 8);
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_free_various_sizes(void)
+{
+ static const size_t sizes[] = {
+ 1, 8, 16, 17, 63, 64, 128, 1024, 4096,
+ 64 * 1024, 256 * 1024, 1024 * 1024,
+ };
+ void *ptrs[RTE_DIM(sizes)];
+ unsigned int i;
+ for (i = 0; i < RTE_DIM(sizes); i++) {
+ ptrs[i] = rte_fastmem_alloc(sizes[i], 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "alloc(%zu) failed: rte_errno=%d",
+ sizes[i], rte_errno);
+ memset(ptrs[i], 0x5a, sizes[i]);
+ }
+
+ for (i = 0; i < RTE_DIM(sizes); i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_alignment(void)
+{
+ static const size_t aligns[] = {
+ 8, 16, 64, 256, 4096, 65536,
+ };
+ unsigned int i;
+ for (i = 0; i < RTE_DIM(aligns); i++) {
+ void *p = rte_fastmem_alloc(1, aligns[i], 0);
+
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc(1, align=%zu) failed: rte_errno=%d",
+ aligns[i], rte_errno);
+ TEST_ASSERT((uintptr_t)p % aligns[i] == 0,
+ "pointer %p not aligned on %zu",
+ p, aligns[i]);
+ rte_fastmem_free(p);
+ }
+
+ /* Default (align=0) gives at least RTE_CACHE_LINE_SIZE. */
+ {
+ void *p = rte_fastmem_alloc(1, 0, 0);
+
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc(1, align=0) failed: rte_errno=%d", rte_errno);
+ TEST_ASSERT((uintptr_t)p % RTE_CACHE_LINE_SIZE == 0,
+ "default-align pointer %p not cache-line aligned",
+ p);
+ rte_fastmem_free(p);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_zero_flag(void)
+{
+ uint8_t *p;
+ unsigned int i;
+ bool all_zero = true;
+
+ /*
+ * Dirty a slab first by allocating without F_ZERO, writing
+ * a non-zero pattern, and freeing. A subsequent F_ZERO
+ * allocation on the same slab must return zeroed memory.
+ */
+ p = rte_fastmem_alloc(128, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "priming alloc failed");
+ memset(p, 0xff, 128);
+ rte_fastmem_free(p);
+
+ p = rte_fastmem_alloc(128, 0, RTE_FASTMEM_F_ZERO);
+ TEST_ASSERT_NOT_NULL(p, "F_ZERO alloc failed");
+ for (i = 0; i < 128; i++) {
+ if (p[i] != 0) {
+ all_zero = false;
+ break;
+ }
+ }
+ TEST_ASSERT(all_zero, "F_ZERO returned non-zero byte at offset %u", i);
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_reuse(void)
+{
+ void *first, *second;
+ first = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(first, "first alloc failed");
+ rte_fastmem_free(first);
+
+ second = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(second, "second alloc failed");
+
+ /*
+ * The slab's free list is LIFO, so the most recently freed
+ * object is at the head of the list. A subsequent alloc in
+ * the same class returns it.
+ */
+ TEST_ASSERT_EQUAL(first, second,
+ "free + alloc did not reuse: first=%p second=%p",
+ first, second);
+
+ rte_fastmem_free(second);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_many_in_class(void)
+{
+ /*
+ * Allocate more objects in one class than fit in a single
+ * slab, forcing the bin to pull a second block. This
+ * exercises the partial->full transition and the cross-slab
+ * allocation path.
+ */
+ enum { CLASS_SIZE = 8, COUNT = 300000 };
+ void **ptrs;
+ unsigned int i;
+
+ ptrs = calloc(COUNT, sizeof(*ptrs));
+ TEST_ASSERT_NOT_NULL(ptrs, "calloc for test ptrs failed");
+
+ for (i = 0; i < COUNT; i++) {
+ ptrs[i] = rte_fastmem_alloc(CLASS_SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "alloc[%u] failed: rte_errno=%d",
+ i, rte_errno);
+ }
+
+ for (i = 0; i < COUNT; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ free(ptrs);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_socket(void)
+{
+ void *p;
+ int socket_id;
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ p = rte_fastmem_alloc_socket(64, 0, 0, socket_id);
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc_socket(%d) failed: rte_errno=%d",
+ socket_id, rte_errno);
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_block_repurposing(void)
+{
+ void *small, *large;
+
+ /*
+ * Allocate and free a small object, forcing a block to be
+ * assigned to the small class and then returned to the
+ * free-block pool. A subsequent allocation in a different
+ * class must be able to reuse that block.
+ */
+ small = rte_fastmem_alloc(8, 0, 0);
+ TEST_ASSERT_NOT_NULL(small, "small alloc failed");
+ rte_fastmem_free(small);
+
+ large = rte_fastmem_alloc(256 * 1024, 0, 0);
+ TEST_ASSERT_NOT_NULL(large, "large alloc failed");
+ rte_fastmem_free(large);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_block_repurposing_no_growth(void)
+{
+ struct rte_fastmem_stats stats;
+ void *small, *large;
+ uint64_t after_small;
+ int rc;
+
+ /*
+ * Stronger version of test_alloc_block_repurposing: assert
+ * that the cross-class allocation does not grow the
+ * backing memory (bytes_backing stays flat). Because the
+ * free-block pool is shared across size classes — not
+ * partitioned per class — the block freed from the small
+ * class must serve the large allocation without triggering
+ * a new memzone reservation.
+ */
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_stats() failed: %d", rc);
+ TEST_ASSERT_EQUAL(stats.bytes_backing, (uint64_t)0,
+ "unexpected pre-alloc bytes_backing: %" PRIu64,
+ stats.bytes_backing);
+
+ small = rte_fastmem_alloc(8, 0, 0);
+ TEST_ASSERT_NOT_NULL(small, "small alloc failed");
+
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_stats() failed: %d", rc);
+ TEST_ASSERT(stats.bytes_backing > 0,
+ "bytes_backing did not grow on first alloc");
+ after_small = stats.bytes_backing;
+
+ rte_fastmem_free(small);
+ rte_fastmem_cache_flush();
+
+ large = rte_fastmem_alloc(256 * 1024, 0, 0);
+ TEST_ASSERT_NOT_NULL(large,
+ "large alloc failed: rte_errno=%d", rte_errno);
+
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_stats() failed: %d", rc);
+ TEST_ASSERT_EQUAL(stats.bytes_backing, after_small,
+ "cross-class alloc grew backing memory from %" PRIu64
+ " to %" PRIu64,
+ after_small, stats.bytes_backing);
+
+ rte_fastmem_free(large);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_free_null(void)
+{
+ /* Must be a no-op, not a crash. */
+ rte_fastmem_free(NULL);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_content_integrity(void)
+{
+ /*
+ * Allocate a batch of objects, fill each with a distinct
+ * byte pattern, then verify none of the patterns overlap.
+ * This catches header overwrites (slab header corrupted by
+ * object access) and slot-overlap bugs (two pointers pointing
+ * at overlapping slots).
+ */
+ enum { N = 256, SIZE = 128 };
+ uint8_t *ptrs[N];
+ unsigned int i, j;
+ for (i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ memset(ptrs[i], (int)i, SIZE);
+ }
+
+ for (i = 0; i < N; i++)
+ for (j = 0; j < SIZE; j++)
+ TEST_ASSERT_EQUAL(ptrs[i][j], (uint8_t)i,
+ "corruption at ptrs[%u][%u]: got 0x%x, want 0x%x",
+ i, j, ptrs[i][j], (uint8_t)i);
+
+ for (i = 0; i < N; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_align_too_big(void)
+{
+ void *p;
+ /*
+ * A small size with an alignment larger than the maximum
+ * size class cannot be served. The class selected must be
+ * large enough for the alignment, but no such class exists.
+ */
+ rte_errno = 0;
+ p = rte_fastmem_alloc(1, rte_fastmem_max_size() * 2, 0);
+ TEST_ASSERT_NULL(p,
+ "alloc with align>max_size returned non-NULL");
+ TEST_ASSERT_EQUAL(rte_errno, E2BIG,
+ "expected rte_errno=E2BIG, got %d", rte_errno);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_align_one(void)
+{
+ void *p;
+ /* align=1 is a valid power of 2 and must be accepted. */
+ p = rte_fastmem_alloc(8, 1, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc(8, 1) failed: rte_errno=%d",
+ rte_errno);
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_socket_numa_placement(void)
+{
+ void *p;
+ int socket_id;
+ struct rte_memseg *ms;
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ p = rte_fastmem_alloc_socket(64, 0, 0, socket_id);
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc_socket(%d) failed: rte_errno=%d",
+ socket_id, rte_errno);
+
+ /*
+ * Walk the memory to find the memseg for this pointer and
+ * verify its socket. Skip the check if lookup fails (e.g.,
+ * --no-huge mode may not populate memsegs for fastmem's
+ * allocations in a way that rte_mem_virt2memseg can find).
+ */
+ ms = rte_mem_virt2memseg(p, NULL);
+ if (ms != NULL) {
+ TEST_ASSERT_EQUAL(ms->socket_id, socket_id,
+ "alloc on socket %d landed on socket %d",
+ socket_id, ms->socket_id);
+ }
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Allocate from a socket different from the calling lcore's socket,
+ * triggering a cross-socket cache allocation. Then deinit to exercise
+ * the teardown path where a cache's backing memory lives on a
+ * different socket than the one it serves.
+ */
+static int
+test_alloc_cross_socket_deinit(void)
+{
+ int local_sid, remote_sid;
+ unsigned int i, n_sockets;
+ void *p;
+
+ local_sid = (int)rte_socket_id();
+ if (local_sid < 0 || (unsigned int)local_sid >= RTE_MAX_NUMA_NODES)
+ local_sid = rte_socket_id_by_idx(0);
+
+ n_sockets = rte_socket_count();
+ if (n_sockets < 2)
+ return TEST_SKIPPED;
+
+ /* Find a socket different from the local one. */
+ remote_sid = -1;
+ for (i = 0; i < n_sockets; i++) {
+ int sid = rte_socket_id_by_idx(i);
+ if (sid >= 0 && sid != local_sid) {
+ remote_sid = sid;
+ break;
+ }
+ }
+ if (remote_sid < 0)
+ return TEST_SKIPPED;
+
+ p = rte_fastmem_alloc_socket(64, 0, 0, remote_sid);
+ TEST_ASSERT_NOT_NULL(p,
+ "cross-socket alloc(socket %d) failed: rte_errno=%d",
+ remote_sid, rte_errno);
+
+ rte_fastmem_free(p);
+
+ /* Teardown and re-init to exercise the deinit path with
+ * cross-socket caches.
+ */
+ rte_fastmem_deinit();
+
+ TEST_ASSERT_EQUAL(rte_fastmem_init(), 0,
+ "re-init after cross-socket deinit failed");
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Stage 3 tests: per-lcore caches.
+ */
+
+static int
+test_cache_flush(void)
+{
+ void *p;
+ /*
+ * Alloc and free one object, leaving it in the cache. Then
+ * flush and verify that a subsequent alloc may or may not
+ * return the same pointer (not asserting same/different —
+ * just checking that flush does not crash and a follow-up
+ * alloc still works).
+ */
+ p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "first alloc failed");
+ rte_fastmem_free(p);
+
+ rte_fastmem_cache_flush();
+
+ /* Flush again — must be idempotent. */
+ rte_fastmem_cache_flush();
+
+ p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "post-flush alloc failed");
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_cache_flush_without_init(void)
+{
+ /* Must be a no-op, not a crash. */
+ rte_fastmem_cache_flush();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_cache_exceeds_capacity(void)
+{
+ /*
+ * Free more objects at a single size class than the cache
+ * capacity (64 for classes <= 4 KiB). This forces the
+ * cache-drain slow path and verifies no corruption.
+ */
+ enum { COUNT = 200, SIZE = 64 };
+ void *ptrs[COUNT];
+ unsigned int i;
+
+ for (i = 0; i < COUNT; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "alloc[%u] failed: rte_errno=%d", i, rte_errno);
+ }
+
+ for (i = 0; i < COUNT; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ /* Re-alloc the same count should still work. */
+ for (i = 0; i < COUNT; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "re-alloc[%u] failed: rte_errno=%d", i, rte_errno);
+ }
+
+ for (i = 0; i < COUNT; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+struct non_eal_args {
+ int ok;
+ char pad[64];
+};
+
+static uint32_t
+non_eal_thread_main(void *arg)
+{
+ struct non_eal_args *args = arg;
+ uint8_t *p;
+
+ p = rte_fastmem_alloc(128, 0, 0);
+ if (p == NULL)
+ return 1;
+
+ memset(p, 0x7e, 128);
+
+ rte_fastmem_free(p);
+
+ args->ok = 1;
+ return 0;
+}
+
+static int
+test_non_eal_thread(void)
+{
+ rte_thread_t thread_id;
+ struct non_eal_args args = { 0 };
+ int rc;
+
+ rc = rte_thread_create(&thread_id, NULL, non_eal_thread_main, &args);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_thread_create() failed: %d", rc);
+
+ rc = rte_thread_join(thread_id, NULL);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_thread_join() failed: %d", rc);
+
+ TEST_ASSERT_EQUAL(args.ok, 1,
+ "non-EAL thread did not complete alloc/free successfully");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_cache_flush_returns_memory(void)
+{
+ /*
+ * When an entire slab's worth of objects is freed, the
+ * slab's block is returned to the free-block pool and can
+ * be reassigned to another size class. Verify the cache
+ * does not permanently hold objects that prevent this.
+ *
+ * Allocate enough objects in one class to force multiple
+ * slabs, free them all, then flush the cache. After the
+ * flush, all cached objects are drained to their bins and
+ * empty slabs are returned to the block pool.
+ */
+ enum { N = 200, SIZE = 64 };
+ void *ptrs[N];
+ unsigned int i;
+
+ for (i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ }
+ for (i = 0; i < N; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ rte_fastmem_cache_flush();
+
+ /*
+ * An allocation in a completely different class should
+ * succeed now, having access to any blocks freed by the
+ * flush.
+ */
+ {
+ void *other = rte_fastmem_alloc(65536, 0, 0);
+
+ TEST_ASSERT_NOT_NULL(other,
+ "post-flush cross-class alloc failed");
+ rte_fastmem_free(other);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_basic(void)
+{
+ enum { N = 32 };
+ void *ptrs[N];
+ int rc;
+
+ rc = rte_fastmem_alloc_bulk(ptrs, N, 64, 0, 0);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk failed: %d", rc);
+
+ /* Verify all pointers are non-NULL and distinct. */
+ for (unsigned int i = 0; i < N; i++) {
+ TEST_ASSERT_NOT_NULL(ptrs[i], "ptrs[%u] is NULL", i);
+ for (unsigned int j = 0; j < i; j++)
+ TEST_ASSERT(ptrs[i] != ptrs[j],
+ "ptrs[%u] == ptrs[%u]", i, j);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_zero_flag(void)
+{
+ enum { N = 8, SIZE = 128 };
+ void *ptrs[N];
+ int rc;
+
+ rc = rte_fastmem_alloc_bulk(ptrs, N, SIZE, 0, RTE_FASTMEM_F_ZERO);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk failed: %d", rc);
+
+ for (unsigned int i = 0; i < N; i++) {
+ uint8_t *p = ptrs[i];
+
+ for (unsigned int b = 0; b < SIZE; b++)
+ TEST_ASSERT_EQUAL(p[b], 0,
+ "ptrs[%u][%u] != 0", i, b);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_exceeds_cache(void)
+{
+ /* Allocate more than cache capacity (64) in one bulk call. */
+ enum { N = 128 };
+ void *ptrs[N];
+ int rc;
+
+ rc = rte_fastmem_alloc_bulk(ptrs, N, 64, 0, 0);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk(%u) failed: %d", N, rc);
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_socket(void)
+{
+ enum { N = 16 };
+ void *ptrs[N];
+ int socket_id;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no sockets");
+
+ rc = rte_fastmem_alloc_bulk_socket(ptrs, N, 64, 0, 0, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk_socket failed: %d", rc);
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ /* SOCKET_ID_ANY */
+ rc = rte_fastmem_alloc_bulk_socket(ptrs, N, 64, 0, 0, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk_socket(ANY) failed: %d", rc);
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_free_bulk(void)
+{
+ enum { N = 64 };
+ void *ptrs[N];
+ /* Allocate individually, free in bulk. */
+ for (unsigned int i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ /* Verify memory is reusable. */
+ for (unsigned int i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "re-alloc[%u] failed", i);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_classes(void)
+{
+ size_t sizes[32];
+ unsigned int n;
+
+ n = rte_fastmem_classes(NULL);
+ TEST_ASSERT_EQUAL(n, 18u, "expected 18 classes, got %u", n);
+
+ n = rte_fastmem_classes(sizes);
+ TEST_ASSERT_EQUAL(n, 18u, "expected 18 classes, got %u", n);
+ TEST_ASSERT_EQUAL(sizes[0], (size_t)8, "class 0 != 8");
+ TEST_ASSERT_EQUAL(sizes[n - 1], (size_t)(1 << 20),
+ "last class != 1 MiB");
+
+ for (unsigned int i = 0; i < n; i++) {
+ TEST_ASSERT(sizes[i] != 0 && (sizes[i] & (sizes[i] - 1)) == 0,
+ "class %u size %zu not power of 2", i, sizes[i]);
+ if (i > 0)
+ TEST_ASSERT(sizes[i] > sizes[i - 1],
+ "classes not ascending at %u", i);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_class(void)
+{
+ enum { N = 10 };
+ struct rte_fastmem_class_stats cs;
+ void *ptrs[N];
+ int rc;
+
+ for (unsigned int i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ }
+
+ rc = rte_fastmem_stats_class(64, &cs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_class failed: %d", rc);
+ TEST_ASSERT_EQUAL(cs.class_size, (size_t)64, "wrong class_size");
+ TEST_ASSERT(cs.alloc_cache_hits + cs.alloc_cache_misses == N,
+ "alloc count != N: hits=%" PRIu64 " misses=%" PRIu64,
+ cs.alloc_cache_hits, cs.alloc_cache_misses);
+ TEST_ASSERT_EQUAL(cs.in_use, (uint64_t)N, "in_use != N");
+
+ for (unsigned int i = 0; i < N; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ rc = rte_fastmem_stats_class(64, &cs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_class after free failed: %d", rc);
+ TEST_ASSERT_EQUAL(cs.in_use, (uint64_t)0, "in_use != 0 after free");
+
+ /* Invalid class size. */
+ rc = rte_fastmem_stats_class(13, &cs);
+ TEST_ASSERT_EQUAL(rc, -EINVAL, "expected -EINVAL for bad size");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_lcore(void)
+{
+ struct rte_fastmem_lcore_stats ls;
+ void *ptr;
+ int rc;
+
+ ptr = rte_fastmem_alloc(128, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ rc = rte_fastmem_stats_lcore(rte_lcore_id(), &ls);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_lcore failed: %d", rc);
+ TEST_ASSERT(ls.alloc_cache_hits + ls.alloc_cache_misses > 0,
+ "no alloc activity on this lcore");
+
+ rte_fastmem_free(ptr);
+
+ rc = rte_fastmem_stats_lcore(rte_lcore_id(), &ls);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_lcore after free failed: %d", rc);
+ TEST_ASSERT(ls.free_cache_hits + ls.free_cache_misses > 0,
+ "no free activity on this lcore");
+
+ /* Invalid lcore. */
+ rc = rte_fastmem_stats_lcore(RTE_MAX_LCORE, &ls);
+ TEST_ASSERT_EQUAL(rc, -EINVAL, "expected -EINVAL for bad lcore");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_lcore_class(void)
+{
+ struct rte_fastmem_lcore_class_stats lcs;
+ void *ptr;
+ int rc;
+
+ ptr = rte_fastmem_alloc(256, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ rc = rte_fastmem_stats_lcore_class(rte_lcore_id(), 256, &lcs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_lcore_class failed: %d", rc);
+ TEST_ASSERT_EQUAL(lcs.class_size, (size_t)256, "wrong class_size");
+ TEST_ASSERT(lcs.alloc_cache_hits + lcs.alloc_cache_misses > 0,
+ "no alloc activity");
+
+ rte_fastmem_free(ptr);
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_reset(void)
+{
+ struct rte_fastmem_stats gs;
+ void *ptr;
+ int rc;
+
+ ptr = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+ rte_fastmem_free(ptr);
+
+ rte_fastmem_stats_reset();
+
+ rc = rte_fastmem_stats(&gs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats failed: %d", rc);
+ TEST_ASSERT_EQUAL(gs.alloc_total, (uint64_t)0,
+ "alloc_total not zero after reset");
+ TEST_ASSERT_EQUAL(gs.free_total, (uint64_t)0,
+ "free_total not zero after reset");
+
+ return TEST_SUCCESS;
+}
+
+
+#define MIXED_LONG_LIVED_COUNT 25
+#define MIXED_SHORT_LIVED_ITERS 1000
+#define MIXED_MIN_LCORES 3
+
+static const size_t mixed_long_sizes[] = { 64, 256, 4096 };
+static const size_t mixed_short_sizes[] = { 8, 16, 32, 64, 128, 256, 512, 1024 };
+
+struct mixed_worker_args {
+ uint32_t seed;
+ int result;
+};
+
+static uint32_t
+xorshift32(uint32_t *state)
+{
+ uint32_t x = *state;
+
+ x ^= x << 13;
+ x ^= x >> 17;
+ x ^= x << 5;
+ *state = x;
+ return x;
+}
+
+static int
+mixed_worker(void *arg)
+{
+ struct mixed_worker_args *args = arg;
+ uint32_t seed = args->seed;
+ void *long_lived[MIXED_LONG_LIVED_COUNT];
+ size_t long_sizes[MIXED_LONG_LIVED_COUNT];
+ unsigned int i;
+
+ /* Allocate long-lived objects of mixed sizes. */
+ for (i = 0; i < MIXED_LONG_LIVED_COUNT; i++) {
+ long_sizes[i] = mixed_long_sizes[i % RTE_DIM(mixed_long_sizes)];
+ long_lived[i] = rte_fastmem_alloc(long_sizes[i], 0, 0);
+ if (long_lived[i] == NULL) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ memset(long_lived[i], (int)(i + 1), long_sizes[i]);
+ }
+
+ /* Rapidly cycle short-lived objects. */
+ for (i = 0; i < MIXED_SHORT_LIVED_ITERS; i++) {
+ size_t sz = mixed_short_sizes[xorshift32(&seed) %
+ RTE_DIM(mixed_short_sizes)];
+ uint8_t pattern = (uint8_t)(i & 0xff);
+ uint8_t *p;
+
+ p = rte_fastmem_alloc(sz, 0, 0);
+ if (p == NULL) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ memset(p, pattern, sz);
+
+ /* Verify before freeing. */
+ for (size_t j = 0; j < sz; j++) {
+ if (p[j] != pattern) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ }
+ rte_fastmem_free(p);
+ }
+
+ /* Verify long-lived objects are still intact. */
+ for (i = 0; i < MIXED_LONG_LIVED_COUNT; i++) {
+ uint8_t *bytes = long_lived[i];
+ uint8_t expected = (uint8_t)(i + 1);
+
+ for (size_t j = 0; j < long_sizes[i]; j++) {
+ if (bytes[j] != expected) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ }
+ rte_fastmem_free(long_lived[i]);
+ }
+
+ args->result = TEST_SUCCESS;
+ return 0;
+}
+
+static int
+test_mixed_lifetimes_multi_lcore(void)
+{
+ struct mixed_worker_args args[RTE_MAX_LCORE];
+ unsigned int lcore_id;
+ unsigned int count = 0;
+ struct rte_fastmem_stats stats;
+ int rc;
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id)
+ count++;
+
+ if (count < MIXED_MIN_LCORES) {
+ printf("Not enough worker lcores (%u < %u), skipping\n",
+ count, MIXED_MIN_LCORES);
+ return TEST_SKIPPED;
+ }
+
+ /* Launch workers with distinct seeds. */
+ uint32_t seed = 0xdeadbeef;
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ args[lcore_id].seed = seed;
+ args[lcore_id].result = TEST_FAILED;
+ seed += 0x12345678;
+ rte_eal_remote_launch(mixed_worker, &args[lcore_id], lcore_id);
+ }
+
+ rte_eal_mp_wait_lcore();
+
+ /* Check all workers succeeded. */
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ TEST_ASSERT_EQUAL(args[lcore_id].result, TEST_SUCCESS,
+ "worker on lcore %u failed", lcore_id);
+ }
+
+ /* Verify no memory leak. */
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "stats failed: %d", rc);
+ TEST_ASSERT_EQUAL(stats.bytes_in_use, (uint64_t)0,
+ "bytes_in_use not zero after test: %" PRIu64,
+ stats.bytes_in_use);
+
+ return TEST_SUCCESS;
+}
+
+
+/*
+ * Memory limit tests.
+ *
+ * FASTMEM_MEMZONE_SIZE is 128 MiB. We use a limit of 128 MiB
+ * (one memzone) for most tests, and large objects (256 KiB) to
+ * exhaust slabs quickly.
+ */
+
+#define LIMIT_ONE_MZ ((size_t)128 << 20)
+#define LIMIT_OBJ_SIZE ((size_t)256 * 1024)
+
+static int
+test_memory_limit_basic(void)
+{
+ int rc;
+
+ rc = rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+ TEST_ASSERT_EQUAL(rc, 0, "set_memory_limit failed: %d", rc);
+
+ const size_t got = rte_fastmem_get_limit(0);
+ TEST_ASSERT_EQUAL(got, LIMIT_ONE_MZ,
+ "get_memory_limit mismatch: %zu", got);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "first reserve failed: %d", rc);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ + 1, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0, "second reserve should have failed");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_alloc_exhaustion(void)
+{
+ const unsigned int max_ptrs = 1024;
+ void *ptrs[max_ptrs];
+ unsigned int count = 0;
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (count = 0; count < max_ptrs; count++) {
+ ptrs[count] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (ptrs[count] == NULL)
+ break;
+ }
+
+ TEST_ASSERT(count > 0, "should have allocated at least one");
+ TEST_ASSERT(count < max_ptrs, "should have hit the limit");
+ TEST_ASSERT_EQUAL(rte_errno, ENOMEM, "expected ENOMEM, got %d", rte_errno);
+
+ rte_fastmem_free(ptrs[count - 1]);
+ void *p = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc after free should succeed");
+ rte_fastmem_free(p);
+
+ for (unsigned int i = 0; i < count - 1; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_zero_blocks_growth(void)
+{
+ int rc;
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 0);
+
+ rc = rte_fastmem_reserve(1, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0, "reserve with limit=0 should fail");
+
+ void *p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NULL(p, "alloc with limit=0 should fail");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_below_current(void)
+{
+ int rc;
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "reserve failed: %d", rc);
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 1);
+
+ void *p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc from existing backing should work");
+ rte_fastmem_free(p);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ * 2, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0, "growth beyond limit should fail");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_socket_id_any(void)
+{
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 42);
+
+ for (unsigned int i = 0; i < rte_socket_count(); i++) {
+ const int sid = rte_socket_id_by_idx(i);
+ const size_t lim = rte_fastmem_get_limit(sid);
+
+ TEST_ASSERT_EQUAL(lim, (size_t)42,
+ "socket %d limit mismatch: %zu", sid, lim);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_unlimited(void)
+{
+ int rc;
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 0);
+ rte_fastmem_set_limit(SOCKET_ID_ANY, SIZE_MAX);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "reserve after reset failed: %d", rc);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_alloc_integrity_under_oom(void)
+{
+ const unsigned int n = 128;
+ const size_t obj_size = 1024;
+ uint8_t *ptrs[n];
+ const unsigned int extra_max = 1024;
+ void *extra[extra_max];
+ unsigned int n_extra = 0;
+ unsigned int i;
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (i = 0; i < n; i++) {
+ ptrs[i] = rte_fastmem_alloc(obj_size, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ memset(ptrs[i], (int)(i & 0xff), obj_size);
+ }
+
+ /* Exhaust remaining backing with large objects. */
+ for (n_extra = 0; n_extra < extra_max; n_extra++) {
+ extra[n_extra] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (extra[n_extra] == NULL)
+ break;
+ }
+
+ /* Verify original objects are intact. */
+ for (i = 0; i < n; i++) {
+ const uint8_t expected = (uint8_t)(i & 0xff);
+ for (unsigned int j = 0; j < obj_size; j++)
+ TEST_ASSERT_EQUAL(ptrs[i][j], expected,
+ "corruption at [%u][%u]", i, j);
+ }
+
+ for (i = 0; i < n; i++)
+ rte_fastmem_free(ptrs[i]);
+ for (i = 0; i < n_extra; i++)
+ rte_fastmem_free(extra[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_bulk_alloc_oom(void)
+{
+ const unsigned int bulk_n = 64;
+ const unsigned int drain_max = 512;
+ void *ptrs[bulk_n];
+ void *drain[drain_max];
+ unsigned int drained = 0;
+ int rc;
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (drained = 0; drained < drain_max; drained++) {
+ drain[drained] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (drain[drained] == NULL)
+ break;
+ }
+
+ /* Free a few — enough for some but not bulk_n objects. */
+ const unsigned int freed = RTE_MIN(drained, 4u);
+ for (unsigned int i = 0; i < freed; i++)
+ rte_fastmem_free(drain[--drained]);
+
+ rc = rte_fastmem_alloc_bulk(ptrs, bulk_n, LIMIT_OBJ_SIZE, 0, 0);
+ TEST_ASSERT(rc < 0, "bulk alloc should fail");
+
+ for (unsigned int i = 0; i < drained; i++)
+ rte_fastmem_free(drain[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_recovery_after_free(void)
+{
+ const unsigned int max_ptrs = 512;
+ void *ptrs[max_ptrs];
+ unsigned int count = 0;
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (count = 0; count < max_ptrs; count++) {
+ ptrs[count] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (ptrs[count] == NULL)
+ break;
+ }
+ TEST_ASSERT(count > 0 && count < max_ptrs,
+ "expected partial fill, got %u", count);
+
+ const unsigned int half = count / 2;
+ for (unsigned int i = 0; i < half; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ for (unsigned int i = 0; i < half; i++) {
+ ptrs[i] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "recovery alloc[%u] failed", i);
+ }
+
+ for (unsigned int i = 0; i < count; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+struct limit_worker_args {
+ unsigned int alloc_count;
+ int result;
+};
+
+static int
+limit_worker(void *arg)
+{
+ struct limit_worker_args *args = arg;
+ const unsigned int max_ptrs = 128;
+ void *ptrs[max_ptrs];
+ unsigned int i;
+
+ args->alloc_count = 0;
+
+ for (i = 0; i < max_ptrs; i++) {
+ ptrs[i] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (ptrs[i] == NULL)
+ break;
+ memset(ptrs[i], 0xab, LIMIT_OBJ_SIZE);
+ args->alloc_count++;
+ }
+
+ for (unsigned int j = 0; j < args->alloc_count; j++) {
+ uint8_t *bytes = ptrs[j];
+ for (size_t k = 0; k < LIMIT_OBJ_SIZE; k++) {
+ if (bytes[k] != 0xab) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ }
+ rte_fastmem_free(ptrs[j]);
+ }
+
+ args->result = TEST_SUCCESS;
+ return 0;
+}
+
+static int
+test_memory_limit_multi_lcore_oom(void)
+{
+ struct limit_worker_args args[RTE_MAX_LCORE];
+ unsigned int lcore_id;
+ unsigned int worker_count = 0;
+ RTE_LCORE_FOREACH_WORKER(lcore_id)
+ worker_count++;
+
+ if (worker_count < 2) {
+ printf("Not enough workers (%u < 2), skipping\n", worker_count);
+ return TEST_SKIPPED;
+ }
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ args[lcore_id].result = TEST_FAILED;
+ rte_eal_remote_launch(limit_worker, &args[lcore_id], lcore_id);
+ }
+
+ rte_eal_mp_wait_lcore();
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ TEST_ASSERT_EQUAL(args[lcore_id].result, TEST_SUCCESS,
+ "worker on lcore %u failed", lcore_id);
+ }
+
+ struct rte_fastmem_stats stats;
+ rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(stats.bytes_in_use, (uint64_t)0,
+ "bytes_in_use not zero: %" PRIu64, stats.bytes_in_use);
+
+ return TEST_SUCCESS;
+}
+
+static int
+fastmem_setup(void)
+{
+ return rte_fastmem_init();
+}
+
+static void
+fastmem_teardown(void)
+{
+ rte_fastmem_deinit();
+}
+
+static struct unit_test_suite fastmem_lifecycle_testsuite = {
+ .suite_name = "fastmem lifecycle tests",
+ .setup = NULL,
+ .teardown = NULL,
+ .unit_test_cases = {
+ TEST_CASE(test_init_deinit),
+ TEST_CASE(test_init_is_not_idempotent),
+ TEST_CASE(test_deinit_without_init),
+ TEST_CASE(test_max_size),
+ TEST_CASE(test_reserve_without_init),
+ TEST_CASE(test_cache_flush_without_init),
+ TEST_CASE(test_classes),
+ TEST_CASES_END()
+ }
+};
+
+static struct unit_test_suite fastmem_functional_testsuite = {
+ .suite_name = "fastmem functional tests",
+ .setup = NULL,
+ .teardown = NULL,
+ .unit_test_cases = {
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_small),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_multiple_memzones),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_cumulative),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_invalid_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_any_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_too_big),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_invalid_align),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_free_small),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_free_various_sizes),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_alignment),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_zero_flag),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_reuse),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_many_in_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_block_repurposing),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_block_repurposing_no_growth),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_free_null),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_content_integrity),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_align_too_big),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_align_one),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_socket_numa_placement),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_cross_socket_deinit),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_cache_flush),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_cache_exceeds_capacity),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_non_eal_thread),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_cache_flush_returns_memory),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_basic),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_zero_flag),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_exceeds_cache),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_free_bulk),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_lcore),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_lcore_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_reset),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_mixed_lifetimes_multi_lcore),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_basic),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_alloc_exhaustion),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_zero_blocks_growth),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_below_current),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_socket_id_any),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_unlimited),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_alloc_integrity_under_oom),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_bulk_alloc_oom),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_recovery_after_free),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_multi_lcore_oom),
+ TEST_CASES_END()
+ }
+};
+
+static int
+test_fastmem(void)
+{
+ int rc;
+
+ rc = unit_test_suite_runner(&fastmem_lifecycle_testsuite);
+ if (rc != 0)
+ return rc;
+
+ return unit_test_suite_runner(&fastmem_functional_testsuite);
+}
+
+REGISTER_FAST_TEST(fastmem_autotest, NOHUGE_SKIP, ASAN_OK, test_fastmem);
diff --git a/app/test/test_fastmem_perf.c b/app/test/test_fastmem_perf.c
new file mode 100644
index 0000000000..73c0a4c6ce
--- /dev/null
+++ b/app/test/test_fastmem_perf.c
@@ -0,0 +1,1040 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_launch.h>
+#include <rte_lcore.h>
+#include <rte_malloc.h>
+#include <rte_mempool.h>
+#include <rte_stdatomic.h>
+
+#include <rte_fastmem.h>
+
+#include "test.h"
+
+#define TEST_LOG(...) printf(__VA_ARGS__)
+
+static const size_t SIZES[] = { 8, 64, 256, 1024, 4096 };
+#define N_SIZES RTE_DIM(SIZES)
+
+/* Number of ops for warmup and measurement. */
+#define WARMUP_OPS 20000u
+#define MEASURE_OPS 2000000u
+
+/* Buffer for scenarios that allocate N then free N. */
+#define BATCH_N 256
+
+/*
+ * Allocator vtable: a thin adapter exposing alloc / free /
+ * per-allocator setup/teardown. Each scenario calls these
+ * indirectly so the same timing loop serves all allocators.
+ */
+struct allocator {
+ const char *name;
+ int (*setup)(size_t size, unsigned int n_max);
+ void (*teardown)(void);
+ void *(*alloc)(void);
+ void (*free_obj)(void *ptr);
+ int (*alloc_bulk)(void **ptrs, unsigned int n);
+ void (*free_bulk)(void **ptrs, unsigned int n);
+};
+
+/* Fastmem adapter -------------------------------------------------- */
+
+static size_t fastmem_size;
+
+static int
+fastmem_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ fastmem_size = size;
+ return 0;
+}
+
+static void
+fastmem_teardown(void)
+{
+ rte_fastmem_cache_flush();
+}
+
+static void * __rte_noinline
+fastmem_alloc(void)
+{
+ return rte_fastmem_alloc(fastmem_size, 0, 0);
+}
+
+static void __rte_noinline
+fastmem_free(void *ptr)
+{
+ rte_fastmem_free(ptr);
+}
+
+/* Mempool adapter -------------------------------------------------- */
+
+static struct rte_mempool *mempool_pool;
+
+static int
+mempool_setup(size_t size, unsigned int n_max)
+{
+ char name[RTE_MEMPOOL_NAMESIZE];
+ unsigned int cache_size;
+
+ /*
+ * Pool size must accommodate the full batch burst plus
+ * per-lcore cache capacity. Use mempool's default cache
+ * size so we're measuring its standard hot path.
+ */
+ cache_size = RTE_MEMPOOL_CACHE_MAX_SIZE;
+
+ snprintf(name, sizeof(name), "fmperf_mp_%zu", size);
+ mempool_pool = rte_mempool_create(name, n_max + cache_size * 2,
+ size, cache_size, 0, NULL, NULL, NULL, NULL,
+ SOCKET_ID_ANY, 0);
+ if (mempool_pool == NULL) {
+ TEST_LOG("mempool_create(%zu) failed\n", size);
+ return -1;
+ }
+
+ return 0;
+}
+
+static void
+mempool_teardown(void)
+{
+ rte_mempool_free(mempool_pool);
+ mempool_pool = NULL;
+}
+
+static void * __rte_noinline
+mempool_alloc_one(void)
+{
+ void *obj = NULL;
+
+ if (rte_mempool_get(mempool_pool, &obj) < 0)
+ return NULL;
+ return obj;
+}
+
+static void __rte_noinline
+mempool_free_one(void *ptr)
+{
+ rte_mempool_put(mempool_pool, ptr);
+}
+
+/* rte_malloc adapter ----------------------------------------------- */
+
+static size_t malloc_size;
+
+static int
+malloc_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ malloc_size = size;
+ return 0;
+}
+
+static void
+malloc_teardown(void)
+{
+}
+
+static void * __rte_noinline
+malloc_alloc(void)
+{
+ return rte_malloc(NULL, malloc_size, 0);
+}
+
+static void __rte_noinline
+malloc_free(void *ptr)
+{
+ rte_free(ptr);
+}
+
+/* libc (glibc) malloc adapter -------------------------------------- */
+
+static size_t libc_size;
+
+static int
+libc_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ /*
+ * Round up to cache-line alignment to match the other
+ * allocators' default alignment guarantees and keep the
+ * comparison honest. aligned_alloc() requires size to be
+ * a multiple of the alignment.
+ */
+ libc_size = RTE_ALIGN_CEIL(size, RTE_CACHE_LINE_SIZE);
+ return 0;
+}
+
+static void
+libc_teardown(void)
+{
+}
+
+static void * __rte_noinline
+libc_alloc(void)
+{
+ return aligned_alloc(RTE_CACHE_LINE_SIZE, libc_size);
+}
+
+static void __rte_noinline
+libc_free(void *ptr)
+{
+ free(ptr);
+}
+
+/* Bulk adapters ---------------------------------------------------- */
+
+static int __rte_noinline
+fastmem_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return rte_fastmem_alloc_bulk(ptrs, n, fastmem_size, 0, 0);
+}
+
+static void __rte_noinline
+fastmem_free_bulk(void **ptrs, unsigned int n)
+{
+ rte_fastmem_free_bulk(ptrs, n);
+}
+
+/* Fastmem handle adapter ------------------------------------------- */
+
+static rte_fastmem_handle_t fastmem_handle;
+
+static int
+fastmem_h_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ return rte_fastmem_hlookup(size, 0, rte_socket_id(), &fastmem_handle);
+}
+
+static void
+fastmem_h_teardown(void)
+{
+ rte_fastmem_cache_flush();
+}
+
+static void * __rte_noinline
+fastmem_h_alloc(void)
+{
+ return rte_fastmem_halloc(fastmem_handle, 0);
+}
+
+static void __rte_noinline
+fastmem_h_free(void *ptr)
+{
+ rte_fastmem_hfree(fastmem_handle, ptr);
+}
+
+static int __rte_noinline
+fastmem_h_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return rte_fastmem_halloc_bulk(fastmem_handle, ptrs, n, 0);
+}
+
+static void __rte_noinline
+fastmem_h_free_bulk(void **ptrs, unsigned int n)
+{
+ rte_fastmem_hfree_bulk(fastmem_handle, ptrs, n);
+}
+
+/* Mempool adapter -------------------------------------------------- */
+
+static int __rte_noinline
+mempool_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return rte_mempool_get_bulk(mempool_pool, ptrs, n);
+}
+
+static void __rte_noinline
+mempool_free_bulk(void **ptrs, unsigned int n)
+{
+ rte_mempool_put_bulk(mempool_pool, ptrs, n);
+}
+
+static int __rte_noinline
+generic_alloc_bulk(void **ptrs, unsigned int n, void *(*alloc_fn)(void))
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++) {
+ ptrs[i] = alloc_fn();
+ if (ptrs[i] == NULL)
+ return -1;
+ }
+ return 0;
+}
+
+static int __rte_noinline
+malloc_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return generic_alloc_bulk(ptrs, n, malloc_alloc);
+}
+
+static void __rte_noinline
+malloc_free_bulk(void **ptrs, unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ malloc_free(ptrs[i]);
+}
+
+static int __rte_noinline
+libc_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return generic_alloc_bulk(ptrs, n, libc_alloc);
+}
+
+static void __rte_noinline
+libc_free_bulk(void **ptrs, unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ libc_free(ptrs[i]);
+}
+
+/* Adapter table ---------------------------------------------------- */
+
+static const struct allocator allocators[] = {
+ { "fastmem", fastmem_setup, fastmem_teardown, fastmem_alloc, fastmem_free, fastmem_alloc_bulk, fastmem_free_bulk },
+ { "fastmem_h", fastmem_h_setup, fastmem_h_teardown, fastmem_h_alloc, fastmem_h_free, fastmem_h_alloc_bulk, fastmem_h_free_bulk },
+ { "mempool", mempool_setup, mempool_teardown, mempool_alloc_one, mempool_free_one, mempool_alloc_bulk, mempool_free_bulk },
+ { "rte_malloc", malloc_setup, malloc_teardown, malloc_alloc, malloc_free, malloc_alloc_bulk, malloc_free_bulk },
+ { "libc", libc_setup, libc_teardown, libc_alloc, libc_free, libc_alloc_bulk, libc_free_bulk },
+};
+#define N_ALLOCATORS RTE_DIM(allocators)
+
+/*
+ * Scenario 1: tight alloc+free loop. A single object is cycled
+ * repeatedly. The LIFO path keeps the same pointer hot, giving
+ * a best-case measurement.
+ */
+static double
+run_tight(const struct allocator *alloc, size_t size)
+{
+ void *p;
+ uint64_t tsc;
+ unsigned int i;
+
+ if (alloc->setup(size, 1) < 0)
+ return -1.0;
+
+ /* Warmup. */
+ for (i = 0; i < WARMUP_OPS; i++) {
+ p = alloc->alloc();
+ if (p == NULL)
+ goto err;
+ alloc->free_obj(p);
+ }
+
+ tsc = rte_rdtsc_precise();
+ for (i = 0; i < MEASURE_OPS; i++) {
+ p = alloc->alloc();
+ if (p == NULL)
+ goto err;
+ alloc->free_obj(p);
+ }
+ tsc = rte_rdtsc_precise() - tsc;
+
+ alloc->teardown();
+
+ return (double)tsc / MEASURE_OPS;
+err:
+ alloc->teardown();
+ return -1.0;
+}
+
+/*
+ * Scenario 2: allocate N, free N (FIFO free order). Exercises
+ * cache refill and drain paths when N exceeds cache capacity.
+ */
+static void
+run_batch(const struct allocator *alloc, size_t size,
+ double *cycles_alloc, double *cycles_free)
+{
+ void *ptrs[BATCH_N];
+ uint64_t tsc_alloc = 0, tsc_free = 0;
+ unsigned int iter, i;
+ unsigned int iters;
+
+ *cycles_alloc = -1.0;
+ *cycles_free = -1.0;
+
+ if (alloc->setup(size, BATCH_N) < 0)
+ return;
+
+ /* Pick iteration count so total ops ~= MEASURE_OPS. */
+ iters = MEASURE_OPS / BATCH_N;
+
+ /* Warmup. */
+ for (iter = 0; iter < WARMUP_OPS / BATCH_N; iter++) {
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ for (i = 0; i < BATCH_N; i++)
+ alloc->free_obj(ptrs[i]);
+ }
+
+ for (iter = 0; iter < iters; iter++) {
+ uint64_t t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ tsc_alloc += rte_rdtsc_precise() - t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < BATCH_N; i++)
+ alloc->free_obj(ptrs[i]);
+ tsc_free += rte_rdtsc_precise() - t0;
+ }
+
+ alloc->teardown();
+
+ *cycles_alloc = (double)tsc_alloc / (iters * BATCH_N);
+ *cycles_free = (double)tsc_free / (iters * BATCH_N);
+ return;
+err:
+ alloc->teardown();
+}
+
+/*
+ * Scenario 3: allocate N, free N in reverse order.
+ */
+static void
+run_batch_reverse(const struct allocator *alloc, size_t size,
+ double *cycles_alloc, double *cycles_free)
+{
+ void *ptrs[BATCH_N];
+ uint64_t tsc_alloc = 0, tsc_free = 0;
+ unsigned int iter, i;
+ unsigned int iters;
+
+ *cycles_alloc = -1.0;
+ *cycles_free = -1.0;
+
+ if (alloc->setup(size, BATCH_N) < 0)
+ return;
+
+ iters = MEASURE_OPS / BATCH_N;
+
+ for (iter = 0; iter < WARMUP_OPS / BATCH_N; iter++) {
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ for (i = BATCH_N; i > 0; i--)
+ alloc->free_obj(ptrs[i - 1]);
+ }
+
+ for (iter = 0; iter < iters; iter++) {
+ uint64_t t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ tsc_alloc += rte_rdtsc_precise() - t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = BATCH_N; i > 0; i--)
+ alloc->free_obj(ptrs[i - 1]);
+ tsc_free += rte_rdtsc_precise() - t0;
+ }
+
+ alloc->teardown();
+
+ *cycles_alloc = (double)tsc_alloc / (iters * BATCH_N);
+ *cycles_free = (double)tsc_free / (iters * BATCH_N);
+ return;
+err:
+ alloc->teardown();
+}
+
+/*
+ * Scenario 4: multi-lcore alloc/work/free with a dummy-work
+ * baseline. Each worker runs a tight alloc → touch → free loop
+ * on its own lcore. A second run with the same dummy work but
+ * no allocator traffic establishes a baseline; the per-op
+ * allocator cost is reported as (alloc_run - baseline_run).
+ *
+ * Fixed size class and a fixed amount of dummy work per op —
+ * this scenario sweeps lcore count rather than size.
+ */
+#define MULTI_SIZE 256u
+#define MULTI_WORK_BYTES 64u
+#define MULTI_WORK_PASSES 8u /* RMW passes over the work region. */
+#define MULTI_OPS 200000u
+#define MULTI_WARMUP 2000u
+#define MAX_MULTI_LCORES 32u
+
+/*
+ * Per-worker volatile sink. Each worker writes to its own
+ * slot, preventing dead-code elimination of touch_buffer() and
+ * avoiding cross-lcore cache-line sharing on the hot path.
+ * Padded to cache-line stride to prevent false sharing between
+ * neighboring workers' slots.
+ */
+struct worker_sink {
+ volatile uint64_t value;
+} __rte_cache_aligned;
+
+static struct worker_sink worker_sinks[RTE_MAX_LCORE];
+
+/*
+ * Out-of-line dummy workload: run MULTI_WORK_PASSES
+ * read-modify-write passes over the first 'bytes' of the
+ * buffer. Each pass reads what the previous pass wrote, so the
+ * compiler cannot unroll or parallelize across passes — the
+ * work scales linearly with MULTI_WORK_PASSES. Returns an
+ * accumulator so the caller can feed it into a volatile sink;
+ * without that, the compiler could elide the whole function.
+ *
+ * __rte_noinline so it looks identical to the compiler in both
+ * the baseline (pre-allocated scratch buffer) and alloc-path
+ * runs, making the cycle-delta subtraction valid.
+ *
+ * The purpose of this being tunably expensive is to keep
+ * worker-per-iteration cost high relative to the allocator's
+ * critical section, so that even serialized allocators like
+ * rte_malloc spend most of their time outside the lock and the
+ * measured per-op allocator cost reflects its own work rather
+ * than its contention queue.
+ */
+static uint64_t __rte_noinline
+touch_buffer(void *buf, size_t bytes)
+{
+ uint64_t *p = buf;
+ size_t n = bytes / sizeof(uint64_t);
+ uint64_t acc = 0;
+ unsigned int pass;
+ size_t i;
+
+ /* Prime the buffer with a known pattern. */
+ for (i = 0; i < n; i++)
+ p[i] = i * 0x9E3779B97F4A7C15ULL;
+
+ /*
+ * Dependent RMW passes: each pass reads p[i] written by
+ * the previous pass, mixes the pass index in, and writes
+ * back. The XOR into acc keeps the chain live.
+ */
+ for (pass = 0; pass < MULTI_WORK_PASSES; pass++) {
+ for (i = 0; i < n; i++) {
+ uint64_t v = p[i];
+
+ v = v * 0xC2B2AE3D27D4EB4FULL + pass;
+ v ^= v >> 33;
+ p[i] = v;
+ acc ^= v;
+ }
+ }
+
+ return acc;
+}
+
+struct worker_args {
+ const struct allocator *alloc;
+ void *scratch; /* baseline only; NULL => alloc path */
+ unsigned int iters;
+ unsigned int warmup;
+ unsigned int bulk_n; /* 0 = single-object, >0 = bulk */
+ RTE_ATOMIC(bool) start_flag; /* barrier at worker entry */
+ uint64_t cycles; /* out */
+ unsigned int ops; /* out */
+ int err; /* out */
+};
+
+static int
+worker_run(void *arg)
+{
+ struct worker_args *wa = arg;
+ unsigned int lcore = rte_lcore_id();
+ uint64_t acc = 0;
+ uint64_t t0;
+ unsigned int i;
+
+ wa->err = 0;
+ wa->ops = 0;
+ wa->cycles = 0;
+
+ /* Wait for start flag (spin-barrier set by main). */
+ while (!rte_atomic_load_explicit(&wa->start_flag,
+ rte_memory_order_acquire))
+ rte_pause();
+
+ /* Warmup. */
+ for (i = 0; i < wa->warmup; i++) {
+ void *p;
+
+ if (wa->scratch != NULL)
+ p = wa->scratch;
+ else {
+ p = wa->alloc->alloc();
+ if (p == NULL) {
+ wa->err = -1;
+ return -1;
+ }
+ }
+ acc ^= touch_buffer(p, MULTI_WORK_BYTES);
+ if (wa->scratch == NULL)
+ wa->alloc->free_obj(p);
+ }
+
+ /* Measured loop. */
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < wa->iters; i++) {
+ void *p;
+
+ if (wa->scratch != NULL)
+ p = wa->scratch;
+ else {
+ p = wa->alloc->alloc();
+ if (p == NULL) {
+ wa->err = -1;
+ break;
+ }
+ }
+ acc ^= touch_buffer(p, MULTI_WORK_BYTES);
+ if (wa->scratch == NULL)
+ wa->alloc->free_obj(p);
+ }
+ wa->cycles = rte_rdtsc_precise() - t0;
+ wa->ops = i;
+
+ /* Publish accumulator to defeat dead-code elimination. */
+ worker_sinks[lcore].value ^= acc;
+
+ return 0;
+}
+
+static int
+worker_run_bulk(void *arg)
+{
+ struct worker_args *wa = arg;
+ unsigned int lcore = rte_lcore_id();
+ void *ptrs[BATCH_N];
+ uint64_t acc = 0;
+ uint64_t t0;
+ unsigned int i, j;
+ unsigned int bulk_n = wa->bulk_n;
+
+ wa->err = 0;
+ wa->ops = 0;
+ wa->cycles = 0;
+
+ while (!rte_atomic_load_explicit(&wa->start_flag,
+ rte_memory_order_acquire))
+ rte_pause();
+
+ /* Warmup. */
+ for (i = 0; i < wa->warmup; i++) {
+ if (wa->alloc->alloc_bulk(ptrs, bulk_n) < 0) {
+ wa->err = -1;
+ return -1;
+ }
+ for (j = 0; j < bulk_n; j++)
+ acc ^= touch_buffer(ptrs[j], MULTI_WORK_BYTES);
+ wa->alloc->free_bulk(ptrs, bulk_n);
+ }
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < wa->iters; i++) {
+ if (wa->alloc->alloc_bulk(ptrs, bulk_n) < 0) {
+ wa->err = -1;
+ break;
+ }
+ for (j = 0; j < bulk_n; j++)
+ acc ^= touch_buffer(ptrs[j], MULTI_WORK_BYTES);
+ wa->alloc->free_bulk(ptrs, bulk_n);
+ }
+ wa->cycles = rte_rdtsc_precise() - t0;
+ wa->ops = i * bulk_n;
+
+ worker_sinks[lcore].value ^= acc;
+
+ return 0;
+}
+
+/*
+ * Launch workers on the first 'n_workers' worker lcores, run
+ * either the baseline (scratch != NULL) or the alloc path
+ * (scratch == NULL), and return the mean per-op cycle cost
+ * averaged across participating workers.
+ *
+ * On any worker error, returns -1.0.
+ */
+static double
+run_multi_workers(const struct allocator *alloc, unsigned int n_workers,
+ void *const *scratches, unsigned int bulk_n)
+{
+ struct worker_args wargs[RTE_MAX_LCORE];
+ unsigned int worker_lcores[MAX_MULTI_LCORES];
+ unsigned int n = 0;
+ unsigned int lcore_id;
+ unsigned int i;
+ lcore_function_t *fn = bulk_n > 0 ? worker_run_bulk : worker_run;
+
+ /* Collect the first n_workers worker lcores. */
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ if (n >= n_workers)
+ break;
+ worker_lcores[n++] = lcore_id;
+ }
+ if (n < n_workers)
+ return -1.0;
+
+ /* Prepare per-worker args. */
+ for (i = 0; i < n_workers; i++) {
+ struct worker_args *wa = &wargs[worker_lcores[i]];
+
+ wa->alloc = alloc;
+ wa->scratch = scratches != NULL ? scratches[i] : NULL;
+ wa->iters = MULTI_OPS;
+ wa->warmup = MULTI_WARMUP;
+ wa->bulk_n = bulk_n;
+ rte_atomic_store_explicit(&wa->start_flag, false,
+ rte_memory_order_relaxed);
+ }
+
+ /* Launch workers. They spin on start_flag until released. */
+ for (i = 0; i < n_workers; i++)
+ rte_eal_remote_launch(fn, &wargs[worker_lcores[i]],
+ worker_lcores[i]);
+
+ /* Release all workers roughly simultaneously. */
+ for (i = 0; i < n_workers; i++)
+ rte_atomic_store_explicit(
+ &wargs[worker_lcores[i]].start_flag, true,
+ rte_memory_order_release);
+
+ /* Wait for completion. */
+ for (i = 0; i < n_workers; i++)
+ rte_eal_wait_lcore(worker_lcores[i]);
+
+ /* Aggregate: mean cycles per op across workers. */
+ {
+ double sum_cycles_per_op = 0.0;
+ unsigned int n_ok = 0;
+
+ for (i = 0; i < n_workers; i++) {
+ struct worker_args *wa = &wargs[worker_lcores[i]];
+
+ if (wa->err != 0 || wa->ops == 0)
+ return -1.0;
+ sum_cycles_per_op +=
+ (double)wa->cycles / (double)wa->ops;
+ n_ok++;
+ }
+ return sum_cycles_per_op / n_ok;
+ }
+}
+
+/*
+ * One sub-run of Scenario 4: given an allocator and a worker
+ * count, return (baseline, alloc_path) mean cycles per op.
+ */
+static void
+run_multi_lcore(const struct allocator *alloc, unsigned int n_workers,
+ unsigned int bulk_n, double *baseline, double *alloc_path)
+{
+ void *scratches[MAX_MULTI_LCORES] = {0};
+ unsigned int n_alloced = 0;
+ unsigned int i;
+
+ *baseline = -1.0;
+ *alloc_path = -1.0;
+
+ if (alloc->setup(MULTI_SIZE, n_workers * 64) < 0)
+ return;
+
+ /* Baseline: pre-allocate one scratch per worker. */
+ for (i = 0; i < n_workers; i++) {
+ scratches[i] = alloc->alloc();
+ if (scratches[i] == NULL)
+ goto err;
+ n_alloced++;
+ }
+
+ *baseline = run_multi_workers(alloc, n_workers, scratches, 0);
+
+ for (i = 0; i < n_alloced; i++)
+ alloc->free_obj(scratches[i]);
+ n_alloced = 0;
+
+ /* Alloc path: workers alloc+free each iter. */
+ *alloc_path = run_multi_workers(alloc, n_workers, NULL, bulk_n);
+
+ alloc->teardown();
+ return;
+err:
+ for (i = 0; i < n_alloced; i++)
+ alloc->free_obj(scratches[i]);
+ alloc->teardown();
+}
+
+/* Reporting -------------------------------------------------------- */
+
+static void
+print_header(const char *title)
+{
+ size_t i;
+
+ TEST_LOG("\n=== %s ===\n", title);
+ TEST_LOG("%-12s", "allocator");
+ for (i = 0; i < N_SIZES; i++)
+ TEST_LOG(" %10zu B", SIZES[i]);
+ TEST_LOG("\n");
+}
+
+static void
+print_row(const char *name, const double *values)
+{
+ size_t i;
+
+ TEST_LOG("%-12s", name);
+ for (i = 0; i < N_SIZES; i++) {
+ if (values[i] < 0)
+ TEST_LOG(" %12s", "--");
+ else
+ TEST_LOG(" %12.1f", values[i]);
+ }
+ TEST_LOG("\n");
+}
+
+static void
+print_multi_header(const char *title, const unsigned int *lcore_counts,
+ unsigned int n_counts)
+{
+ unsigned int i;
+
+ TEST_LOG("\n=== %s ===\n", title);
+ TEST_LOG("%-12s", "allocator");
+ for (i = 0; i < n_counts; i++)
+ TEST_LOG(" %8u lcore%c", lcore_counts[i],
+ lcore_counts[i] == 1 ? ' ' : 's');
+ TEST_LOG("\n");
+}
+
+static void
+print_multi_row(const char *name, const double *values, unsigned int n_counts)
+{
+ unsigned int i;
+
+ TEST_LOG("%-12s", name);
+ for (i = 0; i < n_counts; i++) {
+ if (values[i] < 0)
+ TEST_LOG(" %14s", "--");
+ else
+ TEST_LOG(" %14.1f", values[i]);
+ }
+ TEST_LOG("\n");
+}
+
+/* Driver ----------------------------------------------------------- */
+
+static int
+test_fastmem_perf(void)
+{
+ size_t i;
+ size_t a;
+ int rc;
+
+ rc = rte_fastmem_init();
+ if (rc < 0) {
+ TEST_LOG("rte_fastmem_init() failed: %d\n", rc);
+ return -1;
+ }
+
+ rc = rte_fastmem_reserve(128 * 1024 * 1024, SOCKET_ID_ANY);
+ if (rc < 0) {
+ TEST_LOG("rte_fastmem_reserve() failed: %d\n", rc);
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ TEST_LOG("\nfastmem performance — single-lcore, fixed-size\n");
+ TEST_LOG("All numbers are TSC cycles.\n");
+
+ /* Scenario 1: tight alloc+free. */
+ print_header("Scenario 1: Single-object hot path — cycles per (alloc + free)");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ vals[i] = run_tight(&allocators[a], SIZES[i]);
+ print_row(allocators[a].name, vals);
+ }
+
+ /* Scenario 2: batched, FIFO free. */
+ print_header("Scenario 2: Batch alloc, FIFO free — cycles per alloc");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_alloc);
+ }
+ print_header("Scenario 2: Batch alloc, FIFO free — cycles per free");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_free);
+ }
+
+ /* Scenario 3: batched, reverse free. */
+ print_header("Scenario 3: Batch alloc, LIFO free — cycles per alloc");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch_reverse(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_alloc);
+ }
+ print_header("Scenario 3: Batch alloc, LIFO free — cycles per free");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch_reverse(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_free);
+ }
+
+ /* Scenario 4: multi-lcore alloc/work/free with baseline. */
+ {
+ unsigned int max_workers = rte_lcore_count() - 1;
+ unsigned int lcore_counts[8];
+ unsigned int n_counts = 0;
+ unsigned int w;
+ double base_vals[N_ALLOCATORS][8];
+ double alloc_vals[N_ALLOCATORS][8];
+ double delta_vals[N_ALLOCATORS][8];
+
+ if (max_workers > MAX_MULTI_LCORES)
+ max_workers = MAX_MULTI_LCORES;
+
+ /* Sweep lcore counts: 1, 2, 4, 8, ... up to max_workers. */
+ for (w = 1; w <= max_workers && n_counts < RTE_DIM(lcore_counts); w *= 2)
+ lcore_counts[n_counts++] = w;
+ /* Ensure max_workers is the final column if not power of two. */
+ if (n_counts > 0 && lcore_counts[n_counts - 1] != max_workers &&
+ n_counts < RTE_DIM(lcore_counts) && max_workers >= 1)
+ lcore_counts[n_counts++] = max_workers;
+
+ if (n_counts == 0) {
+ TEST_LOG("\nScenario 4 (Multi-lcore contention) skipped: no worker lcores available.\n");
+ } else {
+ TEST_LOG("\nScenario 4 parameters: size=%u B\n",
+ MULTI_SIZE);
+
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ unsigned int c;
+
+ for (c = 0; c < n_counts; c++)
+ run_multi_lcore(&allocators[a], lcore_counts[c],
+ 0, &base_vals[a][c],
+ &alloc_vals[a][c]);
+ for (c = 0; c < n_counts; c++) {
+ if (base_vals[a][c] < 0 || alloc_vals[a][c] < 0)
+ delta_vals[a][c] = -1.0;
+ else
+ delta_vals[a][c] = alloc_vals[a][c] -
+ base_vals[a][c];
+ }
+ }
+
+ TEST_LOG("Baseline (domain logic only): %.1f cycles/op\n",
+ base_vals[0][0]);
+
+ print_multi_header("Scenario 4: Multi-lcore contention — allocator overhead (cycles/op)",
+ lcore_counts, n_counts);
+ for (a = 0; a < N_ALLOCATORS; a++)
+ print_multi_row(allocators[a].name,
+ delta_vals[a], n_counts);
+ }
+ }
+
+ /* Scenario 5: multi-lcore bulk alloc/work/free. */
+ {
+ unsigned int max_workers = rte_lcore_count() - 1;
+ unsigned int lcore_counts[8];
+ unsigned int n_counts = 0;
+ unsigned int w;
+ double base_vals[N_ALLOCATORS][8];
+ double alloc_vals[N_ALLOCATORS][8];
+ double delta_vals[N_ALLOCATORS][8];
+ unsigned int bulk_n = 8;
+
+ if (max_workers > MAX_MULTI_LCORES)
+ max_workers = MAX_MULTI_LCORES;
+
+ for (w = 1; w <= max_workers && n_counts < RTE_DIM(lcore_counts); w *= 2)
+ lcore_counts[n_counts++] = w;
+ if (n_counts > 0 && lcore_counts[n_counts - 1] != max_workers &&
+ n_counts < RTE_DIM(lcore_counts) && max_workers >= 1)
+ lcore_counts[n_counts++] = max_workers;
+
+ if (n_counts == 0) {
+ TEST_LOG("\nScenario 5 (Multi-lcore bulk contention) skipped: no worker lcores available.\n");
+ } else {
+ TEST_LOG("\nScenario 5 parameters: size=%u B, "
+ "bulk=%u\n",
+ MULTI_SIZE, bulk_n);
+
+ for (size_t a = 0; a < N_ALLOCATORS; a++) {
+ unsigned int c;
+
+ for (c = 0; c < n_counts; c++)
+ run_multi_lcore(&allocators[a],
+ lcore_counts[c], bulk_n,
+ &base_vals[a][c],
+ &alloc_vals[a][c]);
+ for (c = 0; c < n_counts; c++) {
+ if (base_vals[a][c] < 0 || alloc_vals[a][c] < 0)
+ delta_vals[a][c] = -1.0;
+ else
+ delta_vals[a][c] = alloc_vals[a][c] -
+ base_vals[a][c];
+ }
+ }
+
+ TEST_LOG("Baseline (domain logic only): %.1f cycles/op\n",
+ base_vals[0][0]);
+
+ print_multi_header("Scenario 5: Multi-lcore bulk contention — allocator overhead (cycles/op)",
+ lcore_counts, n_counts);
+ for (size_t a = 0; a < N_ALLOCATORS; a++)
+ print_multi_row(allocators[a].name,
+ delta_vals[a], n_counts);
+ }
+ }
+
+ TEST_LOG("\n");
+ rte_fastmem_deinit();
+ return 0;
+}
+
+REGISTER_PERF_TEST(fastmem_perf_autotest, test_fastmem_perf);
diff --git a/app/test/test_fastmem_profile.c b/app/test/test_fastmem_profile.c
new file mode 100644
index 0000000000..9a5dc94018
--- /dev/null
+++ b/app/test/test_fastmem_profile.c
@@ -0,0 +1,157 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+/*
+ * A minimal fastmem workload intended for use with perf record /
+ * perf report. Runs a tight alloc/free loop for a fixed duration
+ * so that sampling profilers can attribute cycles to individual
+ * functions and instructions within the fastmem hot path.
+ *
+ * Usage:
+ * perf record -g -- dpdk-test --no-huge --no-pci -m 8192 \
+ * -l 0 <<< fastmem_profile_autotest
+ * perf report
+ */
+
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_lcore.h>
+#include <rte_memory.h>
+
+#include <rte_fastmem.h>
+
+#include "test.h"
+
+/* Duration of each sub-test in TSC cycles (~3 seconds at 3 GHz). */
+#define PROFILE_DURATION_CYCLES (3ULL * rte_get_tsc_hz())
+
+/* Allocation size for the profiling workload. */
+#define PROFILE_SIZE 256u
+
+/*
+ * Sub-test 1: tight alloc+free, exercises only the per-lcore
+ * cache (no bin interaction after warmup).
+ */
+static int
+profile_cache_hit(void)
+{
+ uint64_t deadline;
+ uint64_t ops = 0;
+
+ deadline = rte_rdtsc() + PROFILE_DURATION_CYCLES;
+
+ while (rte_rdtsc() < deadline) {
+ void *p = rte_fastmem_alloc(PROFILE_SIZE, 0, 0);
+
+ if (p == NULL)
+ return -1;
+ rte_fastmem_free(p);
+ ops++;
+ }
+
+ printf(" cache_hit: %" PRIu64 " ops\n", ops);
+ return 0;
+}
+
+/*
+ * Sub-test 2: alloc N then free N, where N exceeds the cache
+ * capacity. This forces repeated cache refills and drains,
+ * exercising the bin lock and slab free-list traversal.
+ */
+#define PROFILE_BATCH 256u
+
+static int
+profile_cache_miss(void)
+{
+ void *ptrs[PROFILE_BATCH];
+ uint64_t deadline;
+ uint64_t ops = 0;
+ unsigned int i;
+
+ deadline = rte_rdtsc() + PROFILE_DURATION_CYCLES;
+
+ while (rte_rdtsc() < deadline) {
+ for (i = 0; i < PROFILE_BATCH; i++) {
+ ptrs[i] = rte_fastmem_alloc(PROFILE_SIZE, 0, 0);
+ if (ptrs[i] == NULL)
+ return -1;
+ }
+ for (i = 0; i < PROFILE_BATCH; i++)
+ rte_fastmem_free(ptrs[i]);
+ ops += PROFILE_BATCH;
+ }
+
+ printf(" cache_miss: %" PRIu64 " ops\n", ops);
+ return 0;
+}
+
+static int
+test_fastmem_profile_cache_hit(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ if (rc < 0) {
+ printf("rte_fastmem_init() failed: %d\n", rc);
+ return -1;
+ }
+
+ rc = rte_fastmem_reserve(128 * 1024 * 1024, SOCKET_ID_ANY);
+ if (rc < 0) {
+ printf("rte_fastmem_reserve() failed: %d\n", rc);
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ printf("fastmem profile: cache-hit workload (size=%u, ~%u s)\n",
+ PROFILE_SIZE, 3);
+
+ if (profile_cache_hit() < 0) {
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ rte_fastmem_deinit();
+ return 0;
+}
+
+static int
+test_fastmem_profile_cache_miss(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ if (rc < 0) {
+ printf("rte_fastmem_init() failed: %d\n", rc);
+ return -1;
+ }
+
+ rc = rte_fastmem_reserve(128 * 1024 * 1024, SOCKET_ID_ANY);
+ if (rc < 0) {
+ printf("rte_fastmem_reserve() failed: %d\n", rc);
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ printf("fastmem profile: cache-miss workload (size=%u, ~%u s)\n",
+ PROFILE_SIZE, 3);
+
+ if (profile_cache_miss() < 0) {
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ rte_fastmem_deinit();
+ return 0;
+}
+
+REGISTER_PERF_TEST(fastmem_profile_cache_hit_autotest,
+ test_fastmem_profile_cache_hit);
+REGISTER_PERF_TEST(fastmem_profile_cache_miss_autotest,
+ test_fastmem_profile_cache_miss);
--
2.43.0
^ permalink raw reply related
* [RFC v2 2/3] lib: add fastmem library
From: Mattias Rönnblom @ 2026-05-26 8:57 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Mattias Rönnblom
In-Reply-To: <20260526085743.64396-1-hofors@lysator.liu.se>
Introduce fastmem, a fast general-purpose small-object allocator
for DPDK applications. It allows an application to replace its
many per-type mempools with a single allocator that handles
arbitrary sizes, grows on demand, and offers mempool-level
performance on the hot path.
Applications that manage many object types (connections, sessions,
work items, timers) currently maintain a separate mempool for each,
requiring upfront sizing and wasting memory on over-provisioned
pools. Fastmem removes both constraints.
Key properties:
* Huge-page-backed, NUMA-aware, DMA-usable.
* Per-lcore caches for lock-free alloc/free on EAL threads.
* Bulk alloc and free APIs.
* Power-of-two size classes from 8 B to 1 MiB.
* Backing memory grows lazily; rte_fastmem_reserve() allows
upfront reservation to avoid latency spikes.
* Always-on per-lcore and per-class statistics.
Bounded to small objects; requests above rte_fastmem_max_size()
are rejected. Replacing rte_malloc is currently not a goal.
--
RFC v2:
* Fix use-after-free in rte_fastmem_deinit() when caches were
allocated cross-socket. Restructured teardown into three phases.
* Add defensive bounds check to local_socket_id() final fallback.
* Add secondary process support. Shared state is discovered lazily
on first allocation; secondaries operate without per-lcore caches.
* Add handle-based allocation API (rte_fastmem_hlookup,
rte_fastmem_halloc, rte_fastmem_halloc_bulk).
* Add test_alloc_cross_socket_deinit exercising cross-socket
teardown path.
* Fix clang -Wthread-safety-analysis warnings.
* Move fastmem to alphabetical position in lib/meson.build.
* Remove trailing double blank lines in test_fastmem.c.
* Split programming guide into separate commit.
Signed-off-by: Mattias Rönnblom <hofors@lysator.liu.se>
---
doc/api/doxy-api-index.md | 1 +
doc/api/doxy-api.conf.in | 1 +
lib/fastmem/meson.build | 6 +
lib/fastmem/rte_fastmem.c | 1694 +++++++++++++++++++++++++++++++++++++
lib/fastmem/rte_fastmem.h | 774 +++++++++++++++++
lib/meson.build | 1 +
6 files changed, 2477 insertions(+)
create mode 100644 lib/fastmem/meson.build
create mode 100644 lib/fastmem/rte_fastmem.c
create mode 100644 lib/fastmem/rte_fastmem.h
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 9296042119..7ebf1201ce 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -70,6 +70,7 @@ The public API headers are grouped by topics:
[memzone](@ref rte_memzone.h),
[mempool](@ref rte_mempool.h),
[malloc](@ref rte_malloc.h),
+ [fastmem](@ref rte_fastmem.h),
[memcpy](@ref rte_memcpy.h)
- **timers**:
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index bedd944681..4355e9fb2d 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -43,6 +43,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \
@TOPDIR@/lib/efd \
@TOPDIR@/lib/ethdev \
@TOPDIR@/lib/eventdev \
+ @TOPDIR@/lib/fastmem \
@TOPDIR@/lib/fib \
@TOPDIR@/lib/gpudev \
@TOPDIR@/lib/graph \
diff --git a/lib/fastmem/meson.build b/lib/fastmem/meson.build
new file mode 100644
index 0000000000..6c7834608f
--- /dev/null
+++ b/lib/fastmem/meson.build
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2026 Ericsson AB
+
+sources = files('rte_fastmem.c')
+headers = files('rte_fastmem.h')
+deps += ['eal']
diff --git a/lib/fastmem/rte_fastmem.c b/lib/fastmem/rte_fastmem.c
new file mode 100644
index 0000000000..84d97ac36f
--- /dev/null
+++ b/lib/fastmem/rte_fastmem.c
@@ -0,0 +1,1694 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/queue.h>
+
+#include <eal_export.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_eal.h>
+#include <rte_errno.h>
+#include <rte_lcore.h>
+#include <rte_log.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_spinlock.h>
+
+#include <rte_fastmem.h>
+
+RTE_LOG_REGISTER_DEFAULT(fastmem_logtype, NOTICE);
+
+#define RTE_LOGTYPE_FASTMEM fastmem_logtype
+
+#define FASTMEM_LOG(level, ...) \
+ RTE_LOG_LINE(level, FASTMEM, "" __VA_ARGS__)
+
+#define FASTMEM_MEMZONE_SIZE_LOG2 27 /* 128 MiB */
+#define FASTMEM_MEMZONE_SIZE ((size_t)1 << FASTMEM_MEMZONE_SIZE_LOG2)
+
+#define FASTMEM_SLAB_SIZE_LOG2 21 /* 2 MiB */
+#define FASTMEM_SLAB_SIZE ((size_t)1 << FASTMEM_SLAB_SIZE_LOG2)
+#define FASTMEM_SLAB_MASK (FASTMEM_SLAB_SIZE - 1)
+
+#define FASTMEM_SLABS_PER_MEMZONE (FASTMEM_MEMZONE_SIZE / FASTMEM_SLAB_SIZE)
+
+#define FASTMEM_MAX_MEMZONES_PER_SOCKET 64
+
+#define FASTMEM_MIN_CLASS_LOG2 3 /* 8 B */
+#define FASTMEM_MAX_CLASS_LOG2 20 /* 1 MiB */
+#define FASTMEM_N_CLASSES (FASTMEM_MAX_CLASS_LOG2 - FASTMEM_MIN_CLASS_LOG2 + 1)
+
+#define FASTMEM_MIN_SIZE ((size_t)1 << FASTMEM_MIN_CLASS_LOG2)
+#define FASTMEM_MAX_ALLOC_SIZE ((size_t)1 << FASTMEM_MAX_CLASS_LOG2)
+
+#define FASTMEM_SLAB_HEADER_SIZE RTE_CACHE_LINE_SIZE
+
+#define FASTMEM_CACHE_BASE_CAPACITY 64
+#define FASTMEM_CACHE_FLOOR_CAPACITY 4
+#define FASTMEM_CACHE_BASE_CLASS_LOG2 12 /* 4 KiB */
+
+struct fastmem_bin;
+
+/*
+ * Slab header at offset 0 of each 2 MiB slab. Either free (linked
+ * via next_free) or assigned to a bin (linked via list).
+ */
+struct fastmem_slab {
+ struct fastmem_bin *bin;
+ void *free_head;
+ uint32_t free_count;
+ uint32_t n_slots;
+ struct fastmem_slab *next_free;
+ TAILQ_ENTRY(fastmem_slab) list;
+ rte_iova_t iova_base;
+} __rte_aligned(FASTMEM_SLAB_HEADER_SIZE);
+
+TAILQ_HEAD(fastmem_slab_list, fastmem_slab);
+
+struct fastmem_bin {
+ rte_spinlock_t lock;
+ uint32_t slot_size;
+ uint32_t slots_per_slab;
+ uint32_t class_idx;
+ struct fastmem_slab_list partial;
+ struct fastmem_slab_list full;
+ int socket_id;
+ uint64_t slab_acquires;
+ uint64_t slab_releases;
+ uint32_t slabs_partial;
+ uint32_t slabs_full;
+};
+
+/* Per-(lcore, class, socket) bounded LIFO of free object pointers. */
+struct fastmem_cache {
+ uint32_t count;
+ uint32_t capacity;
+ uint32_t target;
+ uint64_t alloc_cache_hits;
+ uint64_t alloc_cache_misses;
+ uint64_t alloc_nomem;
+ uint64_t free_cache_hits;
+ uint64_t free_cache_misses;
+ void *objs[];
+} __rte_cache_aligned;
+
+struct fastmem_socket_state {
+ rte_spinlock_t lock;
+ struct fastmem_slab *free_head;
+ size_t reserved_bytes;
+ size_t memory_limit;
+ unsigned int n_memzones;
+ unsigned int memzone_seq;
+ const struct rte_memzone *memzones[FASTMEM_MAX_MEMZONES_PER_SOCKET];
+ struct fastmem_bin bins[FASTMEM_N_CLASSES];
+ struct fastmem_cache *caches[RTE_MAX_LCORE][FASTMEM_N_CLASSES];
+};
+
+struct fastmem {
+ struct fastmem_socket_state sockets[RTE_MAX_NUMA_NODES];
+};
+
+static struct fastmem *fastmem;
+static const struct rte_memzone *fastmem_mz;
+static bool fastmem_is_primary; /* cached; avoids function call on hot path */
+
+static struct fastmem *
+fastmem_get(void)
+{
+ const struct rte_memzone *mz;
+
+ if (likely(fastmem != NULL))
+ return fastmem;
+
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ rte_errno = ENODEV;
+ return NULL;
+ }
+
+ mz = rte_memzone_lookup("fastmem_state");
+ if (mz == NULL) {
+ rte_errno = ENODEV;
+ return NULL;
+ }
+
+ fastmem_mz = mz;
+ fastmem = mz->addr;
+ return fastmem;
+}
+
+static inline unsigned int
+size_to_class(size_t size, size_t align)
+{
+ size_t effective;
+ unsigned int log2;
+
+ effective = size < FASTMEM_MIN_SIZE ? FASTMEM_MIN_SIZE : size;
+ if (align > effective)
+ effective = align;
+
+ log2 = 64u - rte_clz64(effective - 1);
+
+ if (log2 < FASTMEM_MIN_CLASS_LOG2)
+ log2 = FASTMEM_MIN_CLASS_LOG2;
+ if (log2 > FASTMEM_MAX_CLASS_LOG2)
+ return FASTMEM_N_CLASSES;
+
+ return log2 - FASTMEM_MIN_CLASS_LOG2;
+}
+
+static inline size_t
+class_size(unsigned int class_idx)
+{
+ return (size_t)1 << (class_idx + FASTMEM_MIN_CLASS_LOG2);
+}
+
+static_assert(sizeof(struct fastmem_slab) == FASTMEM_SLAB_HEADER_SIZE,
+ "fastmem slab header must fit in exactly one cache line");
+static_assert(sizeof(struct fastmem_slab) <= FASTMEM_SLAB_SIZE,
+ "slab header larger than a slab makes no sense");
+
+static __rte_always_inline struct fastmem_slab *
+slab_of(void *obj)
+{
+ return (struct fastmem_slab *)
+ ((uintptr_t)obj & ~(uintptr_t)FASTMEM_SLAB_MASK);
+}
+
+static inline size_t
+slab_slot0_offset(size_t class_size)
+{
+ return class_size < FASTMEM_SLAB_HEADER_SIZE ?
+ FASTMEM_SLAB_HEADER_SIZE : class_size;
+}
+
+static inline uint32_t
+slab_slot_count(size_t class_size)
+{
+ size_t offset = slab_slot0_offset(class_size);
+
+ return (uint32_t)((FASTMEM_SLAB_SIZE - offset) / class_size);
+}
+
+/* Must be called with bin->lock held. */
+static void
+slab_init(struct fastmem_bin *bin, struct fastmem_slab *slab)
+{
+ size_t slot_size = bin->slot_size;
+ size_t offset = slab_slot0_offset(slot_size);
+ uint32_t n = bin->slots_per_slab;
+ void *prev = NULL;
+ uint32_t i;
+
+ slab->bin = bin;
+ slab->n_slots = n;
+ slab->free_count = n;
+
+ /* Build in reverse so pops yield sequential addresses. */
+ for (i = 0; i < n; i++) {
+ void *slot = RTE_PTR_ADD(slab, offset + i * slot_size);
+ *(void **)slot = prev;
+ prev = slot;
+ }
+ slab->free_head = prev;
+}
+
+static int
+grow_socket(struct fastmem_socket_state *socket, int socket_id)
+{
+ char name[RTE_MEMZONE_NAMESIZE];
+ const struct rte_memzone *mz;
+ unsigned int i;
+
+ if (socket->reserved_bytes + FASTMEM_MEMZONE_SIZE > socket->memory_limit) {
+ FASTMEM_LOG(ERR,
+ "reserve would exceed memory_limit (%zu) on socket %d",
+ socket->memory_limit, socket_id);
+ return -ENOMEM;
+ }
+
+ if (socket->n_memzones == FASTMEM_MAX_MEMZONES_PER_SOCKET) {
+ FASTMEM_LOG(ERR,
+ "reached per-socket memzone cap (%u) on socket %d",
+ FASTMEM_MAX_MEMZONES_PER_SOCKET, socket_id);
+ return -ENOMEM;
+ }
+
+ snprintf(name, sizeof(name), "fastmem_%d_%u", socket_id,
+ socket->memzone_seq++);
+
+ mz = rte_memzone_reserve_aligned(name, FASTMEM_MEMZONE_SIZE,
+ socket_id, RTE_MEMZONE_IOVA_CONTIG,
+ FASTMEM_SLAB_SIZE);
+ if (mz == NULL) {
+ FASTMEM_LOG(ERR,
+ "failed to reserve %zu-byte memzone '%s' on socket %d: %s",
+ (size_t)FASTMEM_MEMZONE_SIZE, name, socket_id,
+ rte_strerror(rte_errno));
+ return -ENOMEM;
+ }
+
+ socket->memzones[socket->n_memzones++] = mz;
+ socket->reserved_bytes += FASTMEM_MEMZONE_SIZE;
+
+ for (i = 0; i < FASTMEM_SLABS_PER_MEMZONE; i++) {
+ struct fastmem_slab *slab = RTE_PTR_ADD(mz->addr,
+ i * FASTMEM_SLAB_SIZE);
+
+ slab->iova_base = mz->iova + i * FASTMEM_SLAB_SIZE;
+ slab->next_free = socket->free_head;
+ socket->free_head = slab;
+ }
+
+ FASTMEM_LOG(DEBUG,
+ "reserved memzone '%s' (%zu bytes) on socket %d; %zu slabs added",
+ name, (size_t)FASTMEM_MEMZONE_SIZE, socket_id,
+ (size_t)FASTMEM_SLABS_PER_MEMZONE);
+
+ return 0;
+}
+
+static struct fastmem_slab *
+slab_acquire(struct fastmem_socket_state *socket, int socket_id)
+{
+ struct fastmem_slab *slab;
+
+ rte_spinlock_lock(&socket->lock);
+
+ if (socket->free_head == NULL) {
+ int rc = grow_socket(socket, socket_id);
+
+ if (rc < 0) {
+ rte_spinlock_unlock(&socket->lock);
+ return NULL;
+ }
+ }
+
+ slab = socket->free_head;
+ socket->free_head = slab->next_free;
+ slab->next_free = NULL;
+
+ rte_spinlock_unlock(&socket->lock);
+
+ return slab;
+}
+
+static void
+slab_release(struct fastmem_socket_state *socket,
+ struct fastmem_slab *slab)
+{
+ rte_spinlock_lock(&socket->lock);
+
+ slab->next_free = socket->free_head;
+ socket->free_head = slab;
+
+ rte_spinlock_unlock(&socket->lock);
+}
+
+static void
+bin_init(struct fastmem_bin *bin, unsigned int class_idx, int socket_id)
+{
+ size_t slot_size = class_size(class_idx);
+
+ rte_spinlock_init(&bin->lock);
+ bin->slot_size = (uint32_t)slot_size;
+ bin->slots_per_slab = slab_slot_count(slot_size);
+ bin->class_idx = class_idx;
+ TAILQ_INIT(&bin->partial);
+ TAILQ_INIT(&bin->full);
+ bin->socket_id = socket_id;
+ bin->slab_acquires = 0;
+ bin->slab_releases = 0;
+ bin->slabs_partial = 0;
+ bin->slabs_full = 0;
+}
+
+static void
+bin_release(struct fastmem_bin *bin, struct fastmem_socket_state *socket)
+{
+ struct fastmem_slab *slab;
+
+ while ((slab = TAILQ_FIRST(&bin->partial)) != NULL) {
+ TAILQ_REMOVE(&bin->partial, slab, list);
+ slab_release(socket, slab);
+ }
+ while ((slab = TAILQ_FIRST(&bin->full)) != NULL) {
+ TAILQ_REMOVE(&bin->full, slab, list);
+ slab_release(socket, slab);
+ }
+}
+
+static unsigned int
+bin_pop_locked(struct fastmem_bin *bin, void **objs, unsigned int n)
+{
+ unsigned int got = 0;
+
+ while (got < n) {
+ struct fastmem_slab *slab = TAILQ_FIRST(&bin->partial);
+ void *obj;
+
+ if (slab == NULL)
+ break;
+
+ obj = slab->free_head;
+ slab->free_head = *(void **)obj;
+ slab->free_count--;
+ objs[got++] = obj;
+
+ if (slab->free_count == 0) {
+ TAILQ_REMOVE(&bin->partial, slab, list);
+ TAILQ_INSERT_HEAD(&bin->full, slab, list);
+ bin->slabs_partial--;
+ bin->slabs_full++;
+ }
+ }
+
+ return got;
+}
+
+/*
+ * Fully-drained slabs are accumulated in @p to_release for the
+ * caller to return after dropping the lock.
+ */
+static unsigned int
+bin_push_locked(struct fastmem_bin *bin, void **objs, unsigned int n,
+ struct fastmem_slab **to_release)
+{
+ unsigned int n_release = 0;
+ unsigned int i;
+
+ for (i = 0; i < n; i++) {
+ void *obj = objs[i];
+ struct fastmem_slab *slab = (struct fastmem_slab *)
+ ((uintptr_t)obj & ~(uintptr_t)FASTMEM_SLAB_MASK);
+ bool was_full = slab->free_count == 0;
+
+ *(void **)obj = slab->free_head;
+ slab->free_head = obj;
+ slab->free_count++;
+
+ if (was_full) {
+ TAILQ_REMOVE(&bin->full, slab, list);
+ TAILQ_INSERT_HEAD(&bin->partial, slab, list);
+ bin->slabs_full--;
+ bin->slabs_partial++;
+ }
+
+ if (slab->free_count == slab->n_slots) {
+ TAILQ_REMOVE(&bin->partial, slab, list);
+ bin->slabs_partial--;
+ bin->slab_releases++;
+ to_release[n_release++] = slab;
+ }
+ }
+
+ return n_release;
+}
+
+static void *
+bin_alloc_one(struct fastmem_bin *bin)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[bin->socket_id];
+ void *obj;
+
+ rte_spinlock_lock(&bin->lock);
+
+ while (bin_pop_locked(bin, &obj, 1) == 0) {
+ struct fastmem_slab *slab;
+
+ if (TAILQ_FIRST(&bin->partial) != NULL)
+ continue;
+
+ rte_spinlock_unlock(&bin->lock);
+
+ slab = slab_acquire(socket, bin->socket_id);
+ if (slab == NULL) {
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ rte_spinlock_lock(&bin->lock);
+
+ if (unlikely(TAILQ_FIRST(&bin->partial) != NULL)) {
+ /* Release surplus slab without holding bin->lock. */
+ rte_spinlock_unlock(&bin->lock);
+ slab_release(socket, slab);
+ rte_spinlock_lock(&bin->lock);
+ } else {
+ slab_init(bin, slab);
+ TAILQ_INSERT_HEAD(&bin->partial, slab, list);
+ bin->slabs_partial++;
+ bin->slab_acquires++;
+ }
+ }
+
+ rte_spinlock_unlock(&bin->lock);
+
+ return obj;
+}
+
+static unsigned int
+bin_alloc_bulk(struct fastmem_bin *bin, void **objs, unsigned int n)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[bin->socket_id];
+ unsigned int got = 0;
+
+ rte_spinlock_lock(&bin->lock);
+
+ while (got < n) {
+ struct fastmem_slab *slab;
+
+ got += bin_pop_locked(bin, objs + got, n - got);
+ if (got == n)
+ break;
+
+ if (TAILQ_FIRST(&bin->partial) != NULL)
+ continue;
+
+ rte_spinlock_unlock(&bin->lock);
+
+ slab = slab_acquire(socket, bin->socket_id);
+ if (slab == NULL) {
+ rte_spinlock_lock(&bin->lock);
+ break;
+ }
+
+ rte_spinlock_lock(&bin->lock);
+
+ if (unlikely(TAILQ_FIRST(&bin->partial) != NULL)) {
+ /* Release surplus slab without holding bin->lock. */
+ rte_spinlock_unlock(&bin->lock);
+ slab_release(socket, slab);
+ rte_spinlock_lock(&bin->lock);
+ } else {
+ slab_init(bin, slab);
+ TAILQ_INSERT_HEAD(&bin->partial, slab, list);
+ bin->slabs_partial++;
+ bin->slab_acquires++;
+ }
+ }
+
+ rte_spinlock_unlock(&bin->lock);
+
+ return got;
+}
+
+static void
+bin_free_one(struct fastmem_bin *bin, void *obj)
+{
+ unsigned int n_release;
+ struct fastmem_slab *slab_to_release = NULL;
+ struct fastmem_socket_state *socket;
+
+ rte_spinlock_lock(&bin->lock);
+ n_release = bin_push_locked(bin, &obj, 1, &slab_to_release);
+ rte_spinlock_unlock(&bin->lock);
+
+ if (n_release > 0) {
+ socket = &fastmem->sockets[bin->socket_id];
+ slab_release(socket, slab_to_release);
+ }
+}
+
+static void
+bin_free_bulk(struct fastmem_bin *bin, void **objs, unsigned int n)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[bin->socket_id];
+ struct fastmem_slab *to_release[FASTMEM_CACHE_BASE_CAPACITY];
+ unsigned int n_release;
+ unsigned int i;
+
+ RTE_VERIFY(n <= RTE_DIM(to_release));
+
+ rte_spinlock_lock(&bin->lock);
+ n_release = bin_push_locked(bin, objs, n, to_release);
+ rte_spinlock_unlock(&bin->lock);
+
+ for (i = 0; i < n_release; i++)
+ slab_release(socket, to_release[i]);
+}
+
+static inline unsigned int
+cache_capacity(unsigned int class_idx)
+{
+ unsigned int class_log2 = class_idx + FASTMEM_MIN_CLASS_LOG2;
+ unsigned int shift;
+ unsigned int cap;
+
+ if (class_log2 <= FASTMEM_CACHE_BASE_CLASS_LOG2)
+ return FASTMEM_CACHE_BASE_CAPACITY;
+
+ shift = class_log2 - FASTMEM_CACHE_BASE_CLASS_LOG2;
+ cap = FASTMEM_CACHE_BASE_CAPACITY >> shift;
+
+ return cap < FASTMEM_CACHE_FLOOR_CAPACITY ?
+ FASTMEM_CACHE_FLOOR_CAPACITY : cap;
+}
+
+static inline struct fastmem_cache **
+cache_slot(struct fastmem_socket_state *socket, unsigned int class_idx,
+ unsigned int lcore_id)
+{
+ if (lcore_id >= RTE_MAX_LCORE)
+ return NULL;
+ return &socket->caches[lcore_id][class_idx];
+}
+
+static struct fastmem_cache *
+cache_create(struct fastmem_socket_state *socket,
+ unsigned int class_idx, unsigned int lcore_id)
+{
+ struct fastmem_cache **slot = cache_slot(socket, class_idx, lcore_id);
+ struct fastmem_cache *cache;
+ unsigned int capacity;
+ size_t cache_size;
+ unsigned int cache_class;
+ unsigned int own_socket;
+ struct fastmem_socket_state *alloc_socket;
+
+ if (slot == NULL)
+ return NULL;
+
+ cache = *slot;
+ if (cache != NULL)
+ return cache;
+
+ capacity = cache_capacity(class_idx);
+ cache_size = sizeof(*cache) + capacity * sizeof(void *);
+
+ /*
+ * Allocate the cache struct from fastmem on the calling
+ * lcore's socket (NUMA-local to the writer). Bypasses the
+ * cache layer to avoid recursion.
+ */
+ cache_class = size_to_class(cache_size, RTE_CACHE_LINE_SIZE);
+ own_socket = rte_socket_id();
+
+ if (cache_class >= FASTMEM_N_CLASSES) {
+ FASTMEM_LOG(ERR,
+ "cache size %zu exceeds max size class",
+ cache_size);
+ return NULL;
+ }
+
+ if (own_socket >= RTE_MAX_NUMA_NODES)
+ own_socket = (unsigned int)socket->bins[0].socket_id;
+
+ alloc_socket = &fastmem->sockets[own_socket];
+
+ cache = bin_alloc_one(&alloc_socket->bins[cache_class]);
+ if (cache == NULL) {
+ FASTMEM_LOG(ERR,
+ "failed to allocate cache for class %u on socket %u",
+ class_idx, own_socket);
+ return NULL;
+ }
+
+ cache->count = 0;
+ cache->capacity = capacity;
+ cache->target = capacity / 2;
+ cache->alloc_cache_hits = 0;
+ cache->alloc_cache_misses = 0;
+ cache->alloc_nomem = 0;
+ cache->free_cache_hits = 0;
+ cache->free_cache_misses = 0;
+
+ *slot = cache;
+
+ return cache;
+}
+
+static __rte_always_inline struct fastmem_cache *
+cache_get(struct fastmem_socket_state *socket, unsigned int class_idx,
+ unsigned int lcore_id)
+{
+ struct fastmem_cache **slot;
+ struct fastmem_cache *cache;
+
+ if (unlikely(!fastmem_is_primary))
+ return NULL;
+
+ slot = cache_slot(socket, class_idx, lcore_id);
+
+ if (slot == NULL)
+ return NULL;
+
+ cache = *slot;
+ if (cache != NULL)
+ return cache;
+
+ return cache_create(socket, class_idx, lcore_id);
+}
+
+static __rte_always_inline void *
+cache_pop(struct fastmem_cache *cache, struct fastmem_bin *bin)
+{
+ if (cache->count > 0) {
+ cache->alloc_cache_hits++;
+ return cache->objs[--cache->count];
+ }
+
+ cache->count = bin_alloc_bulk(bin, cache->objs, cache->target);
+ if (cache->count == 0)
+ return NULL;
+
+ cache->alloc_cache_misses++;
+ return cache->objs[--cache->count];
+}
+
+static __rte_always_inline void
+cache_push(struct fastmem_cache *cache, struct fastmem_bin *bin, void *obj)
+{
+ unsigned int drain;
+
+ if (cache->count < cache->capacity) {
+ cache->free_cache_hits++;
+ cache->objs[cache->count++] = obj;
+ return;
+ }
+
+ cache->free_cache_misses++;
+
+ /*
+ * Drain the oldest (bottom) half to the bin, keeping the
+ * newest (top) half for temporal reuse.
+ */
+ drain = cache->count - cache->target;
+ bin_free_bulk(bin, cache->objs, drain);
+ memmove(cache->objs, cache->objs + drain,
+ cache->target * sizeof(cache->objs[0]));
+ cache->count = cache->target;
+
+ cache->objs[cache->count++] = obj;
+}
+
+static void
+socket_release_caches(struct fastmem_socket_state *socket)
+{
+ unsigned int lcore;
+ unsigned int c;
+
+ for (lcore = 0; lcore < RTE_MAX_LCORE; lcore++) {
+ for (c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_cache *cache = socket->caches[lcore][c];
+ struct fastmem_slab *cache_slab;
+
+ if (cache == NULL)
+ continue;
+
+ if (cache->count > 0) {
+ bin_free_bulk(&socket->bins[c],
+ cache->objs, cache->count);
+ cache->count = 0;
+ }
+
+ cache_slab = slab_of(cache);
+ bin_free_one(cache_slab->bin, cache);
+
+ socket->caches[lcore][c] = NULL;
+ }
+ }
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_init, 24.11)
+rte_fastmem_init(void)
+{
+ unsigned int s, c;
+
+ if (fastmem != NULL)
+ return -EBUSY;
+
+ fastmem_mz = rte_memzone_reserve_aligned("fastmem_state",
+ sizeof(*fastmem), SOCKET_ID_ANY, 0,
+ RTE_CACHE_LINE_SIZE);
+ if (fastmem_mz == NULL)
+ return -ENOMEM;
+
+ fastmem = fastmem_mz->addr;
+ fastmem_is_primary = true;
+ memset(fastmem, 0, sizeof(*fastmem));
+
+ for (s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ rte_spinlock_init(&socket->lock);
+ socket->memory_limit = SIZE_MAX;
+
+ for (c = 0; c < FASTMEM_N_CLASSES; c++)
+ bin_init(&socket->bins[c], c, (int)s);
+ }
+
+ return 0;
+}
+
+static void
+release_socket_caches(struct fastmem_socket_state *socket)
+{
+ socket_release_caches(socket);
+}
+
+static void
+release_socket_bins(struct fastmem_socket_state *socket)
+{
+ unsigned int c;
+
+ for (c = 0; c < FASTMEM_N_CLASSES; c++)
+ bin_release(&socket->bins[c], socket);
+}
+
+static void
+release_socket_memzones(struct fastmem_socket_state *socket)
+{
+ unsigned int i;
+
+ for (i = 0; i < socket->n_memzones; i++)
+ rte_memzone_free(socket->memzones[i]);
+
+ socket->free_head = NULL;
+ socket->reserved_bytes = 0;
+ socket->n_memzones = 0;
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_deinit, 24.11)
+rte_fastmem_deinit(void)
+{
+ unsigned int i;
+
+ if (fastmem == NULL)
+ return;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ fastmem = NULL;
+ fastmem_mz = NULL;
+ return;
+ }
+
+ for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ release_socket_caches(&fastmem->sockets[i]);
+
+ for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ release_socket_bins(&fastmem->sockets[i]);
+
+ for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ release_socket_memzones(&fastmem->sockets[i]);
+
+ rte_memzone_free(fastmem_mz);
+ fastmem_mz = NULL;
+ fastmem = NULL;
+}
+
+/* Same resolution order as rte_malloc's malloc_get_numa_socket(). */
+static __rte_always_inline unsigned int
+local_socket_id(void)
+{
+ int sid = (int)rte_socket_id();
+
+ if (likely(sid >= 0 && sid < RTE_MAX_NUMA_NODES))
+ return sid;
+
+ sid = (int)rte_lcore_to_socket_id(rte_get_main_lcore());
+ if (likely(sid >= 0 && sid < RTE_MAX_NUMA_NODES))
+ return sid;
+
+ sid = rte_socket_id_by_idx(0);
+ if (likely(sid >= 0 && sid < RTE_MAX_NUMA_NODES))
+ return sid;
+
+ return 0;
+}
+
+static int
+reserve_on_socket(int sid, size_t size)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[sid];
+ int rc = 0;
+
+ rte_spinlock_lock(&socket->lock);
+
+ while (socket->reserved_bytes < size) {
+ rc = grow_socket(socket, sid);
+ if (rc < 0)
+ break;
+ }
+
+ rte_spinlock_unlock(&socket->lock);
+
+ return rc;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_reserve, 24.11)
+rte_fastmem_reserve(size_t size, int socket_id)
+{
+ unsigned int i;
+ int rc;
+
+ if (fastmem == NULL)
+ return -EINVAL;
+
+ if (socket_id != SOCKET_ID_ANY) {
+ if (socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return -EINVAL;
+ return reserve_on_socket(socket_id, size);
+ }
+
+ rc = reserve_on_socket(local_socket_id(), size);
+ if (rc == 0)
+ return 0;
+
+ for (i = 0; i < rte_socket_count(); i++) {
+ int sid = rte_socket_id_by_idx(i);
+
+ if (sid < 0 || (unsigned int)sid == local_socket_id())
+ continue;
+
+ rc = reserve_on_socket(sid, size);
+ if (rc == 0)
+ return 0;
+ }
+
+ return rc;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_set_limit, 24.11)
+rte_fastmem_set_limit(int socket_id, size_t max_bytes)
+{
+ if (fastmem == NULL)
+ return -EINVAL;
+
+ if (socket_id == SOCKET_ID_ANY) {
+ for (unsigned int i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ fastmem->sockets[i].memory_limit = max_bytes;
+ return 0;
+ }
+
+ if (socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return -EINVAL;
+
+ fastmem->sockets[socket_id].memory_limit = max_bytes;
+ return 0;
+}
+
+size_t
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_get_limit, 24.11)
+rte_fastmem_get_limit(int socket_id)
+{
+ if (fastmem == NULL || socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return 0;
+
+ return fastmem->sockets[socket_id].memory_limit;
+}
+
+size_t
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_max_size, 24.11)
+rte_fastmem_max_size(void)
+{
+ return FASTMEM_MAX_ALLOC_SIZE;
+}
+
+static __rte_always_inline void *
+alloc_from_socket(struct fastmem_socket_state *socket,
+ unsigned int class_idx, unsigned int lcore_id)
+{
+ struct fastmem_cache *cache;
+
+ cache = cache_get(socket, class_idx, lcore_id);
+ if (likely(cache != NULL))
+ return cache_pop(cache, &socket->bins[class_idx]);
+ return bin_alloc_one(&socket->bins[class_idx]);
+}
+
+static __rte_always_inline void
+do_free(void *ptr)
+{
+ struct fastmem_slab *slab;
+ struct fastmem_bin *bin;
+ struct fastmem_socket_state *socket;
+ unsigned int lcore_id;
+ struct fastmem_cache *cache;
+
+ slab = slab_of(ptr);
+ bin = slab->bin;
+ socket = &fastmem->sockets[bin->socket_id];
+
+ lcore_id = rte_lcore_id();
+ cache = cache_get(socket, bin->class_idx, lcore_id);
+ if (likely(cache != NULL))
+ cache_push(cache, bin, ptr);
+ else
+ bin_free_one(bin, ptr);
+}
+
+static __rte_always_inline int
+do_alloc_bulk(void **ptrs, unsigned int n, size_t size, size_t align,
+ unsigned int flags, unsigned int lcore_id,
+ int socket_id, bool fallback)
+{
+ unsigned int class_idx;
+ struct fastmem_socket_state *socket;
+ struct fastmem_cache *cache;
+ unsigned int got = 0;
+
+ if (unlikely(fastmem_get() == NULL))
+ return -rte_errno;
+
+ if (align == 0)
+ align = RTE_CACHE_LINE_SIZE;
+ else if (unlikely((align & (align - 1)) != 0)) {
+ rte_errno = EINVAL;
+ return -EINVAL;
+ }
+
+ class_idx = size_to_class(size, align);
+ if (unlikely(class_idx >= FASTMEM_N_CLASSES)) {
+ rte_errno = E2BIG;
+ return -E2BIG;
+ }
+
+ socket = &fastmem->sockets[socket_id];
+ cache = cache_get(socket, class_idx, lcore_id);
+
+ if (likely(cache != NULL)) {
+ /* Drain from cache. */
+ unsigned int avail = RTE_MIN(cache->count, n);
+
+ cache->count -= avail;
+ memcpy(ptrs, &cache->objs[cache->count],
+ avail * sizeof(void *));
+ got = avail;
+ cache->alloc_cache_hits += avail;
+
+ if (got < n) {
+ unsigned int need = n - got;
+ unsigned int want = RTE_MAX(need, cache->target);
+ unsigned int filled;
+
+ if (want <= cache->capacity) {
+ /* Refill into cache, give caller their share. */
+ filled = bin_alloc_bulk(
+ &socket->bins[class_idx],
+ cache->objs, want);
+ if (filled > 0) {
+ cache->alloc_cache_misses += RTE_MIN(filled, need);
+ }
+ if (filled >= need) {
+ memcpy(ptrs + got,
+ cache->objs + filled - need,
+ need * sizeof(void *));
+ cache->count = filled - need;
+ got = n;
+ } else {
+ memcpy(ptrs + got, cache->objs,
+ filled * sizeof(void *));
+ got += filled;
+ cache->count = 0;
+ }
+ } else {
+ /* n exceeds cache capacity; pull directly. */
+ unsigned int direct = bin_alloc_bulk(
+ &socket->bins[class_idx],
+ ptrs + got, need);
+ if (direct > 0)
+ cache->alloc_cache_misses += direct;
+ got += direct;
+ }
+ }
+ } else {
+ got = bin_alloc_bulk(&socket->bins[class_idx], ptrs, n);
+ }
+
+ if (unlikely(got < n) && fallback) {
+ unsigned int i;
+
+ for (i = 0; i < rte_socket_count() && got < n; i++) {
+ int sid = rte_socket_id_by_idx(i);
+
+ if (sid < 0 || sid == socket_id)
+ continue;
+
+ socket = &fastmem->sockets[sid];
+ cache = cache_get(socket, class_idx, lcore_id);
+ if (likely(cache != NULL)) {
+ unsigned int avail =
+ RTE_MIN(cache->count, n - got);
+ cache->count -= avail;
+ memcpy(ptrs + got,
+ &cache->objs[cache->count],
+ avail * sizeof(void *));
+ cache->alloc_cache_hits += avail;
+ got += avail;
+ }
+ if (got < n) {
+ unsigned int direct = bin_alloc_bulk(
+ &socket->bins[class_idx],
+ ptrs + got, n - got);
+ if (direct > 0 && cache != NULL)
+ cache->alloc_cache_misses += direct;
+ got += direct;
+ }
+ }
+ }
+
+ if (unlikely(got < n)) {
+ /* All-or-nothing: return what we got. */
+ struct fastmem_cache **slot;
+ unsigned int i;
+
+ for (i = 0; i < got; i++)
+ do_free(ptrs[i]);
+
+ slot = cache_slot(
+ &fastmem->sockets[socket_id], class_idx,
+ lcore_id);
+ if (slot != NULL && *slot != NULL)
+ (*slot)->alloc_nomem++;
+ rte_errno = ENOMEM;
+ return -ENOMEM;
+ }
+
+ if (flags & RTE_FASTMEM_F_ZERO) {
+ size_t cs = class_size(class_idx);
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ memset(ptrs[i], 0, cs);
+ }
+
+ return 0;
+}
+
+static __rte_always_inline void *
+do_alloc(size_t size, size_t align, unsigned int flags,
+ unsigned int lcore_id, int socket_id, bool fallback)
+{
+ unsigned int class_idx;
+ struct fastmem_cache **slot;
+ void *obj;
+
+ if (unlikely(fastmem_get() == NULL))
+ return NULL;
+
+ if (align == 0)
+ align = RTE_CACHE_LINE_SIZE;
+ else if (unlikely((align & (align - 1)) != 0)) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ class_idx = size_to_class(size, align);
+ if (unlikely(class_idx >= FASTMEM_N_CLASSES)) {
+ rte_errno = E2BIG;
+ return NULL;
+ }
+
+ obj = alloc_from_socket(&fastmem->sockets[socket_id],
+ class_idx, lcore_id);
+
+ if (likely(obj != NULL))
+ goto out;
+
+ if (fallback) {
+ unsigned int i;
+
+ for (i = 0; i < rte_socket_count(); i++) {
+ int sid = rte_socket_id_by_idx(i);
+
+ if (sid < 0 || sid == socket_id)
+ continue;
+
+ obj = alloc_from_socket(&fastmem->sockets[sid],
+ class_idx, lcore_id);
+ if (obj != NULL)
+ goto out;
+ }
+ }
+
+ slot = cache_slot(
+ &fastmem->sockets[socket_id], class_idx, lcore_id);
+ if (slot != NULL && *slot != NULL)
+ (*slot)->alloc_nomem++;
+ rte_errno = ENOMEM;
+ return NULL;
+
+out:
+ if (flags & RTE_FASTMEM_F_ZERO)
+ memset(obj, 0, class_size(class_idx));
+
+ return obj;
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc, 24.11)
+rte_fastmem_alloc(size_t size, size_t align, unsigned int flags)
+{
+ return do_alloc(size, align, flags, rte_lcore_id(),
+ local_socket_id(), false);
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc_socket, 24.11)
+rte_fastmem_alloc_socket(size_t size, size_t align, unsigned int flags,
+ int socket_id)
+{
+ if (socket_id == SOCKET_ID_ANY)
+ return do_alloc(size, align, flags, rte_lcore_id(),
+ local_socket_id(), true);
+
+ if (unlikely(socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ return do_alloc(size, align, flags, rte_lcore_id(), socket_id, false);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_free, 24.11)
+rte_fastmem_free(void *ptr)
+{
+ if (unlikely(ptr == NULL))
+ return;
+
+ do_free(ptr);
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc_bulk, 24.11)
+rte_fastmem_alloc_bulk(void **ptrs, unsigned int n, size_t size, size_t align,
+ unsigned int flags)
+{
+ return do_alloc_bulk(ptrs, n, size, align, flags,
+ rte_lcore_id(), local_socket_id(), false);
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc_bulk_socket, 24.11)
+rte_fastmem_alloc_bulk_socket(void **ptrs, unsigned int n, size_t size,
+ size_t align, unsigned int flags, int socket_id)
+{
+ if (socket_id == SOCKET_ID_ANY)
+ return do_alloc_bulk(ptrs, n, size, align, flags,
+ rte_lcore_id(), local_socket_id(), true);
+
+ if (unlikely(socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)) {
+ rte_errno = EINVAL;
+ return -EINVAL;
+ }
+
+ return do_alloc_bulk(ptrs, n, size, align, flags,
+ rte_lcore_id(), socket_id, false);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_free_bulk, 24.11)
+rte_fastmem_free_bulk(void **ptrs, unsigned int n)
+{
+ unsigned int lcore_id;
+ struct fastmem_slab *slab;
+ struct fastmem_bin *bin;
+ struct fastmem_socket_state *socket;
+ struct fastmem_cache *cache;
+ unsigned int space;
+ unsigned int i;
+
+ if (unlikely(n == 0))
+ return;
+
+ lcore_id = rte_lcore_id();
+
+ /* Fast path: check if first object gives us the bin. */
+ slab = slab_of(ptrs[0]);
+ bin = slab->bin;
+ socket = &fastmem->sockets[bin->socket_id];
+ cache = cache_get(socket, bin->class_idx, lcore_id);
+
+ if (unlikely(cache == NULL)) {
+ for (i = 0; i < n; i++)
+ do_free(ptrs[i]);
+ return;
+ }
+
+ /*
+ * Try to push all objects into the cache in one memcpy.
+ * If any object belongs to a different bin, fall back to
+ * per-object free for the remainder.
+ */
+ space = cache->capacity - cache->count;
+ if (likely(n <= space)) {
+ /* Verify all same bin (common case). */
+ for (i = 1; i < n; i++) {
+ if (slab_of(ptrs[i])->bin != bin)
+ goto slow;
+ }
+ cache->free_cache_hits += n;
+ memcpy(&cache->objs[cache->count], ptrs,
+ n * sizeof(void *));
+ cache->count += n;
+ return;
+ }
+
+ /* Would overflow cache — drain first, then push. */
+ if (n <= cache->capacity) {
+ unsigned int drain;
+
+ for (i = 1; i < n; i++) {
+ if (slab_of(ptrs[i])->bin != bin)
+ goto slow;
+ }
+
+ cache->free_cache_misses += n;
+ drain = cache->count - cache->target + n;
+ if (drain > cache->count)
+ drain = cache->count;
+ if (drain > 0) {
+ bin_free_bulk(bin, cache->objs, drain);
+ cache->count -= drain;
+ memmove(cache->objs, cache->objs + drain,
+ cache->count * sizeof(cache->objs[0]));
+ }
+ memcpy(&cache->objs[cache->count], ptrs,
+ n * sizeof(void *));
+ cache->count += n;
+ return;
+ }
+
+slow:
+ for (i = 0; i < n; i++)
+ do_free(ptrs[i]);
+}
+
+#define fastmem_handle_class_BITS 8
+
+static inline rte_fastmem_handle_t
+fastmem_handle_pack(unsigned int class_idx, int socket_id)
+{
+ return (uint32_t)class_idx |
+ ((uint32_t)socket_id << fastmem_handle_class_BITS);
+}
+
+static inline unsigned int
+fastmem_handle_class(rte_fastmem_handle_t h)
+{
+ return h & ((1U << fastmem_handle_class_BITS) - 1);
+}
+
+static inline int
+fastmem_handle_socket(rte_fastmem_handle_t h)
+{
+ return (int)(h >> fastmem_handle_class_BITS);
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_hlookup, 24.11)
+rte_fastmem_hlookup(size_t size, size_t align, int socket_id,
+ rte_fastmem_handle_t *handle)
+{
+ unsigned int class_idx;
+ struct fastmem_socket_state *socket;
+
+ if (handle == NULL)
+ return -EINVAL;
+
+ if (align == 0)
+ align = RTE_CACHE_LINE_SIZE;
+ else if ((align & (align - 1)) != 0)
+ return -EINVAL;
+
+ if (socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return -EINVAL;
+
+ class_idx = size_to_class(size, align);
+ if (class_idx >= FASTMEM_N_CLASSES)
+ return -E2BIG;
+
+ /* Pre-create the cache for the calling lcore. */
+ socket = &fastmem->sockets[socket_id];
+ cache_create(socket, class_idx, rte_lcore_id());
+
+ *handle = fastmem_handle_pack(class_idx, socket_id);
+ return 0;
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_halloc, 24.11)
+rte_fastmem_halloc(rte_fastmem_handle_t handle, unsigned int flags)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+ unsigned int lcore_id = rte_lcore_id();
+ struct fastmem_socket_state *socket = &fastmem->sockets[socket_id];
+ struct fastmem_bin *bin = &socket->bins[class_idx];
+ struct fastmem_cache *cache;
+ void *obj;
+
+ RTE_ASSERT(fastmem != NULL);
+ RTE_ASSERT(lcore_id < RTE_MAX_LCORE);
+
+ cache = socket->caches[lcore_id][class_idx];
+ RTE_ASSERT(cache != NULL);
+
+ obj = cache_pop(cache, bin);
+ if (unlikely(obj == NULL)) {
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ if (flags & RTE_FASTMEM_F_ZERO)
+ memset(obj, 0, class_size(class_idx));
+
+ return obj;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_halloc_bulk, 24.11)
+rte_fastmem_halloc_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n, unsigned int flags)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+
+ return do_alloc_bulk(ptrs, n, class_size(class_idx),
+ RTE_CACHE_LINE_SIZE, flags, rte_lcore_id(),
+ socket_id, false);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_hfree, 24.11)
+rte_fastmem_hfree(rte_fastmem_handle_t handle, void *ptr)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+ struct fastmem_socket_state *socket = &fastmem->sockets[socket_id];
+ struct fastmem_bin *bin = &socket->bins[class_idx];
+ unsigned int lcore_id = rte_lcore_id();
+ struct fastmem_cache *cache;
+
+ if (unlikely(ptr == NULL))
+ return;
+
+ RTE_ASSERT(lcore_id < RTE_MAX_LCORE);
+
+ cache = socket->caches[lcore_id][class_idx];
+ RTE_ASSERT(cache != NULL);
+
+ cache_push(cache, bin, ptr);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_hfree_bulk, 24.11)
+rte_fastmem_hfree_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+ struct fastmem_socket_state *socket = &fastmem->sockets[socket_id];
+ struct fastmem_bin *bin = &socket->bins[class_idx];
+ unsigned int lcore_id;
+ struct fastmem_cache *cache;
+ unsigned int i;
+
+ if (unlikely(n == 0))
+ return;
+
+ lcore_id = rte_lcore_id();
+ cache = cache_get(socket, class_idx, lcore_id);
+
+ if (likely(cache != NULL)) {
+ for (i = 0; i < n; i++)
+ cache_push(cache, bin, ptrs[i]);
+ } else {
+ for (i = 0; i < n; i++)
+ bin_free_one(bin, ptrs[i]);
+ }
+}
+
+rte_iova_t
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_virt2iova, 24.11)
+rte_fastmem_virt2iova(const void *ptr)
+{
+ struct fastmem_slab *slab;
+
+ RTE_ASSERT(fastmem != NULL);
+
+ slab = slab_of((void *)(uintptr_t)ptr);
+
+ return slab->iova_base + ((uintptr_t)ptr - (uintptr_t)slab);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_cache_flush, 24.11)
+rte_fastmem_cache_flush(void)
+{
+ unsigned int lcore_id;
+ unsigned int s, c;
+
+ if (fastmem == NULL)
+ return;
+
+ lcore_id = rte_lcore_id();
+ if (lcore_id >= RTE_MAX_LCORE)
+ return;
+
+ for (s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ for (c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_cache *cache =
+ socket->caches[lcore_id][c];
+ struct fastmem_slab *cache_slab;
+
+ if (cache == NULL)
+ continue;
+
+ if (cache->count > 0) {
+ bin_free_bulk(&socket->bins[c],
+ cache->objs, cache->count);
+ cache->count = 0;
+ }
+
+ cache_slab = slab_of(cache);
+ bin_free_one(cache_slab->bin, cache);
+
+ socket->caches[lcore_id][c] = NULL;
+ }
+ }
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats, 24.11)
+rte_fastmem_stats(struct rte_fastmem_stats *stats)
+{
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_stats){0};
+ stats->n_classes = FASTMEM_N_CLASSES;
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ stats->bytes_backing += socket->reserved_bytes;
+
+ for (unsigned int c = 0; c < FASTMEM_N_CLASSES; c++) {
+ uint64_t class_allocs = 0, class_frees = 0;
+
+ for (unsigned int l = 0; l < RTE_MAX_LCORE; l++) {
+ struct fastmem_cache *cache =
+ socket->caches[l][c];
+ if (cache == NULL)
+ continue;
+ class_allocs += cache->alloc_cache_hits +
+ cache->alloc_cache_misses;
+ class_frees += cache->free_cache_hits +
+ cache->free_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ }
+ stats->alloc_total += class_allocs;
+ stats->free_total += class_frees;
+ if (class_allocs > class_frees)
+ stats->bytes_in_use += class_size(c) *
+ (class_allocs - class_frees);
+ }
+ }
+
+ return 0;
+}
+
+static inline unsigned int
+exact_class_idx(size_t sz)
+{
+ unsigned int log2;
+
+ if (sz < FASTMEM_MIN_SIZE || sz > FASTMEM_MAX_ALLOC_SIZE)
+ return FASTMEM_N_CLASSES;
+ if ((sz & (sz - 1)) != 0)
+ return FASTMEM_N_CLASSES;
+
+ log2 = (unsigned int)rte_ctz64(sz);
+ if (log2 < FASTMEM_MIN_CLASS_LOG2 || log2 > FASTMEM_MAX_CLASS_LOG2)
+ return FASTMEM_N_CLASSES;
+
+ return log2 - FASTMEM_MIN_CLASS_LOG2;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_class, 24.11)
+rte_fastmem_stats_class(size_t class_size_arg,
+ struct rte_fastmem_class_stats *stats)
+{
+ unsigned int c;
+ uint64_t allocs, frees;
+
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+
+ c = exact_class_idx(class_size_arg);
+ if (c >= FASTMEM_N_CLASSES)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_class_stats){0};
+ stats->class_size = class_size(c);
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+ struct fastmem_bin *bin = &socket->bins[c];
+
+ for (unsigned int l = 0; l < RTE_MAX_LCORE; l++) {
+ struct fastmem_cache *cache = socket->caches[l][c];
+ if (cache == NULL)
+ continue;
+ stats->alloc_cache_hits += cache->alloc_cache_hits;
+ stats->alloc_cache_misses += cache->alloc_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ stats->free_cache_hits += cache->free_cache_hits;
+ stats->free_cache_misses += cache->free_cache_misses;
+ }
+
+ stats->slab_acquires += bin->slab_acquires;
+ stats->slab_releases += bin->slab_releases;
+ stats->slabs_partial += bin->slabs_partial;
+ stats->slabs_full += bin->slabs_full;
+ }
+
+ allocs = stats->alloc_cache_hits + stats->alloc_cache_misses;
+ frees = stats->free_cache_hits + stats->free_cache_misses;
+ if (allocs > frees)
+ stats->in_use = allocs - frees;
+
+ return 0;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_lcore, 24.11)
+rte_fastmem_stats_lcore(unsigned int lcore_id,
+ struct rte_fastmem_lcore_stats *stats)
+{
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+ if (lcore_id >= RTE_MAX_LCORE)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_lcore_stats){0};
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ for (unsigned int c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_cache *cache =
+ socket->caches[lcore_id][c];
+ if (cache == NULL)
+ continue;
+ stats->alloc_cache_hits += cache->alloc_cache_hits;
+ stats->alloc_cache_misses += cache->alloc_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ stats->free_cache_hits += cache->free_cache_hits;
+ stats->free_cache_misses += cache->free_cache_misses;
+ }
+ }
+
+ return 0;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_lcore_class, 24.11)
+rte_fastmem_stats_lcore_class(unsigned int lcore_id, size_t class_size_arg,
+ struct rte_fastmem_lcore_class_stats *stats)
+{
+ unsigned int c;
+
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+ if (lcore_id >= RTE_MAX_LCORE)
+ return -EINVAL;
+
+ c = exact_class_idx(class_size_arg);
+ if (c >= FASTMEM_N_CLASSES)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_lcore_class_stats){0};
+ stats->class_size = class_size(c);
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_cache *cache =
+ fastmem->sockets[s].caches[lcore_id][c];
+ if (cache == NULL)
+ continue;
+ stats->alloc_cache_hits += cache->alloc_cache_hits;
+ stats->alloc_cache_misses += cache->alloc_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ stats->free_cache_hits += cache->free_cache_hits;
+ stats->free_cache_misses += cache->free_cache_misses;
+ }
+
+ return 0;
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_reset, 24.11)
+rte_fastmem_stats_reset(void)
+{
+ if (fastmem == NULL)
+ return;
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ for (unsigned int c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_bin *bin = &socket->bins[c];
+
+ bin->slab_acquires = 0;
+ bin->slab_releases = 0;
+
+ for (unsigned int l = 0; l < RTE_MAX_LCORE; l++) {
+ struct fastmem_cache *cache =
+ socket->caches[l][c];
+ if (cache == NULL)
+ continue;
+ cache->alloc_cache_hits = 0;
+ cache->alloc_cache_misses = 0;
+ cache->alloc_nomem = 0;
+ cache->free_cache_hits = 0;
+ cache->free_cache_misses = 0;
+ }
+ }
+ }
+}
+
+unsigned int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_classes, 24.11)
+rte_fastmem_classes(size_t *sizes)
+{
+ if (sizes != NULL)
+ for (unsigned int i = 0; i < FASTMEM_N_CLASSES; i++)
+ sizes[i] = class_size(i);
+ return FASTMEM_N_CLASSES;
+}
diff --git a/lib/fastmem/rte_fastmem.h b/lib/fastmem/rte_fastmem.h
new file mode 100644
index 0000000000..4da893e7f3
--- /dev/null
+++ b/lib/fastmem/rte_fastmem.h
@@ -0,0 +1,774 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#ifndef _RTE_FASTMEM_H_
+#define _RTE_FASTMEM_H_
+
+/**
+ * @file
+ *
+ * RTE Fastmem
+ *
+ * @warning
+ * @b EXPERIMENTAL:
+ * All functions in this file may be changed or removed without prior notice.
+ *
+ * The fastmem library is a fast, general-purpose small-object
+ * allocator for DPDK applications. It is intended to allow an
+ * application to replace its many per-type mempools — each sized
+ * for a single object type (a connection, a session, a work item,
+ * a timer, etc.) — with a single allocator that handles arbitrary
+ * object sizes, grows on demand, and offers mempool-level
+ * performance for the common allocation and free paths.
+ *
+ * Like mempool, fastmem is backed by huge pages, is NUMA-aware,
+ * supports bulk operations, and uses per-lcore caches to reduce
+ * shared-state contention. Unlike mempool, it does not require the
+ * caller to declare object sizes or counts up front.
+ *
+ * There is a single, global fastmem instance per process. The
+ * instance is brought up with rte_fastmem_init() and torn down with
+ * rte_fastmem_deinit(). Allocations are made with
+ * rte_fastmem_alloc() and freed with rte_fastmem_free().
+ *
+ * The allocator is bounded to small-object allocations. Requests
+ * larger than rte_fastmem_max_size() are rejected; callers with
+ * such needs should use rte_malloc() directly.
+ *
+ * Backing memory is reserved from DPDK memzones. Once reserved,
+ * backing memory is not returned to the system during the
+ * allocator's lifetime. Callers that need predictable latency may
+ * pre-reserve backing memory up front using rte_fastmem_reserve(),
+ * avoiding memzone-reservation overhead during steady-state
+ * operation.
+ *
+ * Alignment argument, @c align:
+ * If non-zero, @c align specifies an exact minimum alignment and
+ * must be a power of 2. If zero, the default alignment is
+ * @c RTE_CACHE_LINE_SIZE, so that objects obtained from distinct
+ * calls cannot false-share a cache line.
+ *
+ * Threads and per-lcore caches:
+ * Allocate and free calls from EAL threads are served through a
+ * per-lcore cache, which makes the common path lock-free.
+ * Unregistered non-EAL threads do not use a cache; their
+ * allocate and free calls go directly to shared state, take an
+ * internal lock, and cost more per call.
+ *
+ * Non-preemptible caller:
+ * Callers should not be preemptible while inside a fastmem call.
+ * Fastmem uses internal spinlocks; if a caller is preempted
+ * while holding one, any other thread that subsequently needs
+ * the same lock stalls until the preempted caller resumes.
+ */
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <rte_bitops.h>
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Flag for rte_fastmem_alloc() and its variants: initialize the
+ * returned memory to zero before returning it to the caller.
+ */
+#define RTE_FASTMEM_F_ZERO RTE_BIT32(0)
+
+/**
+ * Initialize the fastmem allocator.
+ *
+ * Sets up the library's internal state. Must be called before any
+ * allocation call. Typically called once per process, after
+ * rte_eal_init() and before the application's worker threads begin
+ * making allocations.
+ *
+ * Initialization does not pre-reserve any backing memory; memzones
+ * are reserved lazily as allocations require. An application that
+ * wants to avoid memzone-reservation latency on the allocation
+ * path should follow rte_fastmem_init() with one or more calls to
+ * rte_fastmem_reserve().
+ *
+ * This function is not thread-safe and must not be called
+ * concurrently with any other fastmem function.
+ *
+ * @return
+ * - 0: Success.
+ * - -EBUSY: The allocator is already initialized.
+ * - -ENOMEM: Unable to allocate internal state.
+ */
+__rte_experimental
+int
+rte_fastmem_init(void);
+
+/**
+ * Tear down the fastmem allocator.
+ *
+ * Releases the library's internal state and frees all backing
+ * memzones. After this call, no fastmem allocations or frees may
+ * be made until rte_fastmem_init() is called again.
+ *
+ * The caller is responsible for ensuring that no fastmem-allocated
+ * objects remain in use. Outstanding allocations at deinit time
+ * result in undefined behavior.
+ *
+ * This function is not thread-safe and must not be called
+ * concurrently with any other fastmem function.
+ */
+__rte_experimental
+void
+rte_fastmem_deinit(void);
+
+/**
+ * Pre-reserve backing memory.
+ *
+ * Ensures that at least @p size bytes of memzone-backed memory are
+ * available to the allocator on @p socket_id, reserving additional
+ * memzones from EAL as needed to reach that total. Subsequent
+ * allocations served from the pre-reserved memory do not incur
+ * memzone-reservation cost.
+ *
+ * The reservation is cumulative: repeated calls to
+ * rte_fastmem_reserve() with the same @p socket_id grow the
+ * reservation monotonically. Reserved memory is never returned to
+ * the system during the allocator's lifetime.
+ *
+ * A typical use is to call rte_fastmem_reserve() once at
+ * application startup, with a size chosen to cover the expected
+ * steady-state working set. Allocations and frees during
+ * steady-state operation then avoid memzone reservations entirely.
+ *
+ * @param size
+ * The minimum amount of backing memory, in bytes, to make
+ * available on @p socket_id. The allocator may reserve more than
+ * the requested amount due to internal rounding (e.g., to memzone
+ * or block granularity).
+ *
+ * @param socket_id
+ * The NUMA socket on which to reserve memory, or SOCKET_ID_ANY
+ * to leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the reservation.
+ *
+ * @return
+ * - 0: Success.
+ * - -ENOMEM: Insufficient huge-page memory to satisfy the request.
+ * - -EINVAL: Invalid @p socket_id.
+ */
+__rte_experimental
+int
+rte_fastmem_reserve(size_t size, int socket_id);
+
+/**
+ * Set the maximum backing memory that may be reserved on a socket.
+ *
+ * Once the limit is reached, allocations that would require new
+ * backing memory on the constrained socket fail with ENOMEM.
+ * Already-reserved memory is not released.
+ *
+ * Setting a limit below the current reserved amount is allowed and
+ * prevents further growth.
+ *
+ * @param socket_id
+ * The NUMA socket to constrain, or SOCKET_ID_ANY to apply the
+ * limit to all sockets.
+ * @param max_bytes
+ * Maximum backing memory in bytes, or SIZE_MAX for unlimited (the default).
+ * @return
+ * - 0: Success.
+ * - -EINVAL: Fastmem not initialized, or invalid @p socket_id.
+ */
+__rte_experimental
+int
+rte_fastmem_set_limit(int socket_id, size_t max_bytes);
+
+/**
+ * Get the maximum backing memory limit for a socket.
+ *
+ * @param socket_id
+ * The NUMA socket to query.
+ * @return
+ * The limit in bytes, or SIZE_MAX if unlimited.
+ */
+__rte_experimental
+size_t
+rte_fastmem_get_limit(int socket_id);
+
+/**
+ * Retrieve the largest allocation size the allocator supports.
+ *
+ * Requests larger than this size are rejected by the allocation
+ * functions. The returned value is a property of the allocator
+ * implementation and does not change across the lifetime of the
+ * process.
+ *
+ * @return
+ * The largest supported allocation size, in bytes.
+ */
+__rte_experimental
+size_t
+rte_fastmem_max_size(void);
+
+/**
+ * Allocate an object from the fastmem allocator.
+ *
+ * Allocates at least @p size bytes, aligned to at least @p align
+ * bytes. The returned memory is backed by huge pages and is
+ * DMA-usable; its IOVA can be obtained via rte_fastmem_virt2iova().
+ *
+ * On NUMA systems, the memory is allocated on the socket of the
+ * calling lcore. Use rte_fastmem_alloc_socket() to target a
+ * specific socket.
+ *
+ * The allocated memory must be freed with rte_fastmem_free(). An
+ * allocation may be freed from any lcore, not only the lcore that
+ * made the allocation.
+ *
+ * This function is MT-safe.
+ *
+ * @param size
+ * Requested allocation size, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, the returned pointer will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, the returned pointer will
+ * be aligned on a multiple of @p align, which must be a power of
+ * 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags. Use
+ * RTE_FASTMEM_F_ZERO to obtain zero-initialized memory.
+ *
+ * @return
+ * - A pointer to the allocated object on success.
+ * - NULL on failure, with @c rte_errno set:
+ * - E2BIG: @p size exceeds rte_fastmem_max_size().
+ * - EINVAL: Invalid @p align (not a power of two).
+ * - ENOMEM: Allocation could not be served from existing
+ * backing memory and no additional memzone could be reserved.
+ */
+__rte_experimental
+void *
+rte_fastmem_alloc(size_t size, size_t align, unsigned int flags)
+ __rte_alloc_size(1) __rte_alloc_align(2);
+
+/**
+ * Allocate an object on a specific NUMA socket.
+ *
+ * Like rte_fastmem_alloc(), but targets the specified NUMA socket
+ * rather than the socket of the calling lcore. Use this variant
+ * when the lifetime or access pattern of the allocation is not
+ * tied to the calling lcore's socket.
+ *
+ * This function is MT-safe.
+ *
+ * @param size
+ * Requested allocation size, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, the returned pointer will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, the returned pointer will
+ * be aligned on a multiple of @p align, which must be a power of
+ * 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags.
+ *
+ * @param socket_id
+ * The NUMA socket on which to allocate, or SOCKET_ID_ANY to
+ * leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the request.
+ *
+ * @return
+ * - A pointer to the allocated object on success.
+ * - NULL on failure, with @c rte_errno set (see rte_fastmem_alloc()).
+ */
+__rte_experimental
+void *
+rte_fastmem_alloc_socket(size_t size, size_t align, unsigned int flags,
+ int socket_id)
+ __rte_alloc_size(1) __rte_alloc_align(2);
+
+/**
+ * Free an object previously allocated by the fastmem allocator.
+ *
+ * @p ptr must have been returned by a prior call to any fastmem
+ * allocation function, or be NULL. If @p ptr is NULL, no operation
+ * is performed.
+ *
+ * Free may be called from any lcore, regardless of which lcore
+ * made the original allocation.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptr
+ * Pointer to an object previously allocated by fastmem, or NULL.
+ */
+__rte_experimental
+void
+rte_fastmem_free(void *ptr);
+
+/**
+ * Allocate multiple objects in bulk.
+ *
+ * Allocates @p n objects, each of size at least @p size and aligned
+ * to at least @p align bytes, and stores the resulting pointers
+ * into @p ptrs. All @p n objects have the same size and alignment.
+ *
+ * On NUMA systems, the memory is allocated on the socket of the
+ * calling lcore. Use rte_fastmem_alloc_bulk_socket() to target a
+ * specific socket.
+ *
+ * The bulk path amortizes per-object overhead and is typically
+ * faster than @p n individual calls to rte_fastmem_alloc().
+ *
+ * On failure no objects are allocated and @p ptrs is left
+ * untouched.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptrs
+ * An array of at least @p n pointers into which the newly
+ * allocated object pointers are written.
+ *
+ * @param n
+ * The number of objects to allocate.
+ *
+ * @param size
+ * Requested size of each object, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, returned pointers will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, returned pointers will be
+ * aligned on a multiple of @p align, which must be a power of 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags.
+ *
+ * @return
+ * - 0: All @p n objects were allocated and stored in @p ptrs.
+ * - -E2BIG: @p size exceeds rte_fastmem_max_size().
+ * - -EINVAL: Invalid @p align.
+ * - -ENOMEM: Not enough objects could be allocated to fill the
+ * request.
+ */
+__rte_experimental
+int
+rte_fastmem_alloc_bulk(void **ptrs, unsigned int n, size_t size, size_t align,
+ unsigned int flags);
+
+/**
+ * Allocate multiple objects in bulk on a specific NUMA socket.
+ *
+ * Like rte_fastmem_alloc_bulk(), but targets the specified NUMA
+ * socket rather than the socket of the calling lcore.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptrs
+ * An array of at least @p n pointers into which the newly
+ * allocated object pointers are written.
+ *
+ * @param n
+ * The number of objects to allocate.
+ *
+ * @param size
+ * Requested size of each object, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, returned pointers will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, returned pointers will be
+ * aligned on a multiple of @p align, which must be a power of 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags.
+ *
+ * @param socket_id
+ * The NUMA socket on which to allocate, or SOCKET_ID_ANY to
+ * leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the request.
+ *
+ * @return
+ * - 0: All @p n objects were allocated and stored in @p ptrs.
+ * - Negative errno on failure (see rte_fastmem_alloc_bulk()).
+ */
+__rte_experimental
+int
+rte_fastmem_alloc_bulk_socket(void **ptrs, unsigned int n, size_t size,
+ size_t align, unsigned int flags, int socket_id);
+
+/**
+ * Free multiple objects in bulk.
+ *
+ * Frees the @p n objects pointed to by @p ptrs. Each pointer in
+ * the array must have been returned by a prior fastmem allocation
+ * call and must not have been freed. The objects need not have
+ * the same size, alignment, or socket.
+ *
+ * The bulk path amortizes per-object overhead and is typically
+ * faster than @p n individual calls to rte_fastmem_free().
+ *
+ * This function is MT-safe.
+ *
+ * @param ptrs
+ * An array of @p n pointers to fastmem-allocated objects.
+ *
+ * @param n
+ * The number of objects to free.
+ */
+__rte_experimental
+void
+rte_fastmem_free_bulk(void **ptrs, unsigned int n);
+
+/**
+ * Opaque handle encoding a (size class, NUMA socket) pair.
+ *
+ * Obtained via rte_fastmem_hlookup(). Passing a handle to
+ * rte_fastmem_halloc() avoids the per-call size-class
+ * lookup and socket resolution, improving allocation throughput
+ * for fixed-size objects.
+ */
+typedef uint32_t rte_fastmem_handle_t;
+
+/**
+ * Look up a handle for a given object size and NUMA socket.
+ *
+ * The returned handle encodes the size class and socket, and can
+ * be passed to rte_fastmem_halloc() to allocate objects
+ * without repeating the class lookup.
+ *
+ * @param size
+ * Object size in bytes. Must not exceed rte_fastmem_max_size().
+ *
+ * @param align
+ * Alignment requirement (power of two), or 0 for the default
+ * (RTE_CACHE_LINE_SIZE).
+ *
+ * @param socket_id
+ * NUMA socket to allocate from.
+ *
+ * @param[out] handle
+ * On success, set to the resolved handle.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: Invalid alignment or socket_id.
+ * - -E2BIG: @p size exceeds rte_fastmem_max_size().
+ */
+__rte_experimental
+int
+rte_fastmem_hlookup(size_t size, size_t align, int socket_id,
+ rte_fastmem_handle_t *handle);
+
+/**
+ * Allocate an object using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_alloc() but skips the size-class
+ * lookup and socket resolution, using the pre-resolved handle
+ * instead.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param flags
+ * Allocation flags (e.g., RTE_FASTMEM_F_ZERO).
+ *
+ * @return
+ * A pointer to the allocated object, or NULL on failure
+ * (rte_errno is set).
+ */
+__rte_experimental
+void *
+rte_fastmem_halloc(rte_fastmem_handle_t handle, unsigned int flags);
+
+/**
+ * Bulk-allocate objects using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_alloc_bulk() but uses a pre-resolved
+ * handle. All-or-nothing semantics apply.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param[out] ptrs
+ * Array to receive @p n allocated pointers.
+ *
+ * @param n
+ * Number of objects to allocate.
+ *
+ * @param flags
+ * Allocation flags (e.g., RTE_FASTMEM_F_ZERO).
+ *
+ * @return
+ * - 0: All @p n objects allocated successfully.
+ * - -ENOMEM: Allocation failed; no objects were allocated.
+ */
+__rte_experimental
+int
+rte_fastmem_halloc_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n, unsigned int flags);
+
+/**
+ * Free an object using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_free() but skips the slab-header
+ * lookup by using the class and socket encoded in the handle.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param ptr
+ * A pointer previously returned by a fastmem allocation function.
+ * Must belong to the same size class and socket as @p handle.
+ * NULL is permitted (no-op).
+ */
+__rte_experimental
+void
+rte_fastmem_hfree(rte_fastmem_handle_t handle, void *ptr);
+
+/**
+ * Bulk-free objects using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_free_bulk() but skips per-object
+ * slab-header lookups.
+ *
+ * All objects must belong to the same size class and socket as
+ * @p handle.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param ptrs
+ * An array of @p n pointers to fastmem-allocated objects.
+ *
+ * @param n
+ * The number of objects to free.
+ */
+__rte_experimental
+void
+rte_fastmem_hfree_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n);
+
+/**
+ * Obtain the IOVA for a fastmem-allocated pointer.
+ *
+ * Translates a virtual address returned by a fastmem allocation
+ * function into the corresponding IOVA, suitable for use in device
+ * DMA descriptors.
+ *
+ * The returned IOVA is valid for the lifetime of the allocation.
+ *
+ * @p ptr must have been returned by a prior fastmem allocation
+ * function. Passing any other pointer results in undefined
+ * behavior.
+ *
+ * @param ptr
+ * A pointer previously returned by a fastmem allocation
+ * function.
+ *
+ * @return
+ * The IOVA corresponding to @p ptr.
+ */
+__rte_experimental
+rte_iova_t
+rte_fastmem_virt2iova(const void *ptr);
+
+/**
+ * Flush the calling lcore's per-lcore caches.
+ *
+ * Drains every cached object from the calling lcore's
+ * per-(size class, NUMA socket) caches back to their shared
+ * bins, and releases the cache state itself. A subsequent
+ * allocation or free on this lcore lazily recreates any caches
+ * it needs.
+ *
+ * This is useful in applications that have finished a bursty
+ * phase and want to release memory that would otherwise sit idle
+ * in caches. It is also useful in tests that want to observe
+ * bin-level state without per-lcore caching hiding activity.
+ *
+ * The call has no effect when invoked from a non-EAL thread.
+ *
+ * This function is not thread-safe with respect to concurrent
+ * allocations or frees on the calling lcore; call it only when
+ * the calling lcore is not making other fastmem calls.
+ */
+__rte_experimental
+void
+rte_fastmem_cache_flush(void);
+
+/**
+ * Global summary statistics.
+ */
+struct rte_fastmem_stats {
+ uint64_t bytes_backing; /**< Bytes of backing memory (memzones) reserved from EAL. */
+ uint64_t bytes_in_use; /**< Approximate bytes in live objects. */
+ uint64_t alloc_total; /**< Total successful alloc operations (hits + misses). */
+ uint64_t free_total; /**< Total free operations (hits + misses). */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ unsigned int n_classes; /**< Number of size classes. */
+};
+
+/**
+ * Per-size-class statistics (aggregated across all lcores).
+ *
+ * Allocation and free counters count individual objects, not
+ * operations. A bulk allocation of 32 objects that hits the cache
+ * increments alloc_cache_hits by 32.
+ */
+struct rte_fastmem_class_stats {
+ size_t class_size; /**< Usable size of this class (bytes). */
+ uint64_t in_use; /**< Objects currently live (allocs - frees). */
+ uint64_t alloc_cache_hits; /**< Allocs served from a per-lcore cache. */
+ uint64_t alloc_cache_misses; /**< Allocs that triggered a bin refill. */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ uint64_t free_cache_hits; /**< Frees absorbed by a per-lcore cache. */
+ uint64_t free_cache_misses; /**< Frees that triggered a bin drain. */
+ uint64_t slab_acquires; /**< Slabs pulled from the free pool. */
+ uint64_t slab_releases; /**< Slabs returned to the free pool. */
+ uint32_t slabs_partial; /**< Current partial slab count. */
+ uint32_t slabs_full; /**< Current full slab count. */
+};
+
+/**
+ * Per-lcore statistics (aggregated across all classes).
+ */
+struct rte_fastmem_lcore_stats {
+ uint64_t alloc_cache_hits; /**< Allocs served from this lcore's caches. */
+ uint64_t alloc_cache_misses; /**< Allocs that missed this lcore's caches. */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ uint64_t free_cache_hits; /**< Frees absorbed by this lcore's caches. */
+ uint64_t free_cache_misses; /**< Frees that bypassed this lcore's caches. */
+};
+
+/**
+ * Per-lcore, per-class statistics (no aggregation).
+ */
+struct rte_fastmem_lcore_class_stats {
+ size_t class_size; /**< Usable size of this class (bytes). */
+ uint64_t alloc_cache_hits; /**< Allocs served from cache. */
+ uint64_t alloc_cache_misses; /**< Allocs that triggered a bin refill. */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ uint64_t free_cache_hits; /**< Frees absorbed by cache. */
+ uint64_t free_cache_misses; /**< Frees that triggered a bin drain. */
+};
+
+/**
+ * Get the number of size classes and optionally their sizes.
+ *
+ * @param[out] sizes
+ * If non-NULL, filled with the size (in bytes) of each class.
+ * The caller must provide space for at least the returned number
+ * of entries.
+ *
+ * @return
+ * The number of size classes.
+ */
+__rte_experimental
+unsigned int
+rte_fastmem_classes(size_t *sizes);
+
+/**
+ * Retrieve global summary statistics.
+ *
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL or fastmem is not initialized.
+ */
+__rte_experimental
+int
+rte_fastmem_stats(struct rte_fastmem_stats *stats);
+
+/**
+ * Retrieve statistics for a single size class.
+ *
+ * @param class_size
+ * Exact size of the class to query (must match one of the values
+ * returned by rte_fastmem_classes()).
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL, fastmem is not initialized, or
+ * @p class_size does not match any size class.
+ */
+__rte_experimental
+int
+rte_fastmem_stats_class(size_t class_size,
+ struct rte_fastmem_class_stats *stats);
+
+/**
+ * Retrieve per-lcore statistics (aggregated across all classes).
+ *
+ * @param lcore_id
+ * The lcore to query.
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL, fastmem is not initialized, or
+ * @p lcore_id is invalid.
+ */
+__rte_experimental
+int
+rte_fastmem_stats_lcore(unsigned int lcore_id,
+ struct rte_fastmem_lcore_stats *stats);
+
+/**
+ * Retrieve per-lcore, per-class statistics.
+ *
+ * @param lcore_id
+ * The lcore to query.
+ * @param class_size
+ * Exact size of the class to query.
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL, fastmem is not initialized,
+ * @p lcore_id is invalid, or @p class_size does not match any
+ * size class.
+ */
+__rte_experimental
+int
+rte_fastmem_stats_lcore_class(unsigned int lcore_id, size_t class_size,
+ struct rte_fastmem_lcore_class_stats *stats);
+
+/**
+ * Reset all statistics counters to zero.
+ *
+ * Zeroes per-lcore cache counters and per-bin counters. Does not
+ * affect the allocator's operational state.
+ */
+__rte_experimental
+void
+rte_fastmem_stats_reset(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_FASTMEM_H_ */
diff --git a/lib/meson.build b/lib/meson.build
index 8f5cfd28a5..10906d4d53 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -38,6 +38,7 @@ libraries = [
'distributor',
'dmadev', # eventdev depends on this
'efd',
+ 'fastmem',
'eventdev',
'dispatcher', # dispatcher depends on eventdev
'gpudev',
--
2.43.0
^ permalink raw reply related
* [RFC v2 1/3] doc: add fastmem programming guide
From: Mattias Rönnblom @ 2026-05-26 8:57 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Mattias Rönnblom
In-Reply-To: <20260526085743.64396-1-hofors@lysator.liu.se>
Add a programming guide for the fastmem library covering usage,
API overview, design, and implementation details.
Signed-off-by: Mattias Rönnblom <hofors@lysator.liu.se>
---
doc/guides/prog_guide/fastmem_lib.rst | 314 ++++++++++++++++++++++++++
doc/guides/prog_guide/index.rst | 1 +
2 files changed, 315 insertions(+)
create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
diff --git a/doc/guides/prog_guide/fastmem_lib.rst b/doc/guides/prog_guide/fastmem_lib.rst
new file mode 100644
index 0000000000..564d34b78f
--- /dev/null
+++ b/doc/guides/prog_guide/fastmem_lib.rst
@@ -0,0 +1,314 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2026 Ericsson AB
+
+Fastmem Library
+===============
+
+The fastmem library is a fast, general-purpose small-object
+allocator for DPDK applications. It lets an application replace
+its many per-type mempools — each sized for a single object type
+— with a single allocator that handles arbitrary object sizes,
+grows on demand, and offers mempool-level performance for the
+common allocation and free paths.
+
+Like mempool, fastmem is backed by huge pages, is NUMA-aware,
+supports bulk operations, and uses per-lcore caches to reduce
+shared-state contention. Unlike mempool, it does not require the
+caller to declare object sizes or counts up front.
+
+
+When to use fastmem
+-------------------
+
+Use fastmem when:
+
+* Small objects (up to 1 MiB) are allocated and freed on the
+ data path with low, predictable latency requirements.
+
+* Many object types of varying sizes exist and maintaining a
+ separate mempool for each is impractical.
+
+* DMA-usable memory with efficient virtual-to-IOVA translation
+ is needed.
+
+Do not use fastmem for allocations larger than 1 MiB. Use
+``rte_malloc()`` instead.
+
+
+Initialization and teardown
+----------------------------
+
+.. code-block:: c
+
+ /* At startup, after rte_eal_init(). */
+ rte_fastmem_init();
+
+ /* Optional: pre-reserve backing memory to avoid latency
+ * spikes from on-demand memzone reservation. */
+ rte_fastmem_reserve(64 * 1024 * 1024, SOCKET_ID_ANY);
+
+ /* ... application runs ... */
+
+ /* At shutdown, after all allocations have been freed. */
+ rte_fastmem_deinit();
+
+Neither ``rte_fastmem_init()`` nor ``rte_fastmem_deinit()`` is
+thread-safe; call them from the main lcore during startup and
+shutdown.
+
+
+Allocation and free
+-------------------
+
+.. code-block:: c
+
+ void *obj = rte_fastmem_alloc(128, 0, 0);
+ /* Use obj... */
+ rte_fastmem_free(obj);
+
+``rte_fastmem_alloc()`` allocates on the calling lcore's NUMA
+socket. Use ``rte_fastmem_alloc_socket()`` to target a specific
+socket or to enable cross-socket fallback with ``SOCKET_ID_ANY``.
+
+Alignment
+~~~~~~~~~
+
+When ``align`` is 0, the returned pointer is aligned to at least
+``RTE_CACHE_LINE_SIZE``. A non-zero ``align`` must be a power of
+two. Specifying an alignment smaller than ``RTE_CACHE_LINE_SIZE``
+is permitted but the returned object may then share a cache line
+with an adjacent allocation, risking false sharing.
+
+Zeroing
+~~~~~~~
+
+Pass ``RTE_FASTMEM_F_ZERO`` to receive zero-initialized memory:
+
+.. code-block:: c
+
+ void *obj = rte_fastmem_alloc(256, 0, RTE_FASTMEM_F_ZERO);
+
+
+Bulk allocation and free
+-------------------------
+
+.. code-block:: c
+
+ void *ptrs[32];
+
+ if (rte_fastmem_alloc_bulk(ptrs, 32, 64, 0, 0) < 0)
+ /* handle error */;
+
+ /* Use objects... */
+
+ rte_fastmem_free_bulk(ptrs, 32);
+
+Bulk allocation has all-or-nothing semantics: either all
+requested objects are returned, or none are (and ``rte_errno``
+is set to ``ENOMEM``).
+
+Bulk free is most efficient when all objects belong to the same
+size class; in that case the objects are pushed into the
+per-lcore cache in a single operation.
+
+
+IOVA translation
+----------------
+
+Memory returned by fastmem is DMA-usable. To obtain the IOVA
+for use in device descriptors:
+
+.. code-block:: c
+
+ rte_iova_t iova = rte_fastmem_virt2iova(obj);
+
+The translation is O(1). The returned IOVA is valid for the
+lifetime of the allocation.
+
+
+NUMA awareness
+--------------
+
+``rte_fastmem_alloc()`` allocates on the calling lcore's socket.
+``rte_fastmem_alloc_socket()`` accepts an explicit socket ID or
+``SOCKET_ID_ANY``:
+
+* Explicit socket: allocate only from that socket; fail with
+ ``ENOMEM`` if exhausted.
+
+* ``SOCKET_ID_ANY``: try the caller's local socket first, then
+ fall back to other sockets.
+
+
+Per-lcore caches
+----------------
+
+Each EAL thread has a private cache per size class. The common
+allocation and free paths operate entirely within this cache,
+avoiding locks. Cache misses (empty on alloc, full on free)
+trigger a bulk transfer to/from the shared bin under a lock.
+
+Non-EAL threads bypass the cache and take the bin lock on every
+operation.
+
+``rte_fastmem_cache_flush()`` drains the calling lcore's caches
+back to the shared bins. This is useful after bursty phases to
+release idle cached memory.
+
+
+Threading
+---------
+
+All allocation and free functions are thread-safe and may be
+called from any thread. An allocation made on one thread may be
+freed on any other.
+
+Fastmem uses internal spinlocks. A thread preempted while
+holding one delays other threads contending for the same lock
+(correctness is not affected, only latency).
+
+
+Pre-reserving memory
+--------------------
+
+By default, fastmem reserves backing memory lazily on first
+allocation. ``rte_fastmem_reserve(size, socket_id)`` forces
+reservation up front, ensuring subsequent allocations do not
+incur memzone-reservation latency:
+
+.. code-block:: c
+
+ /* Reserve 128 MiB on socket 0. */
+ rte_fastmem_reserve(128 * 1024 * 1024, 0);
+
+Once reserved, backing memory is never returned to the system
+during the allocator's lifetime.
+
+Memory limits
+~~~~~~~~~~~~~
+
+``rte_fastmem_set_limit(socket_id, max_bytes)`` caps how much
+backing memory may be reserved on a given socket. Once the limit is
+reached, allocations that would require new backing memory fail with
+``ENOMEM``. The default is ``SIZE_MAX`` (unlimited).
+``rte_fastmem_get_limit()`` returns the current limit for a socket.
+
+.. code-block:: c
+
+ /* Allow at most 256 MiB on socket 0. */
+ rte_fastmem_set_limit(0, 256 * 1024 * 1024);
+
+ /* Block all growth on socket 1. */
+ rte_fastmem_set_limit(1, 0);
+
+Pass ``SOCKET_ID_ANY`` to apply the same limit to all sockets.
+
+
+Size classes
+------------
+
+Fastmem uses power-of-two size classes from 8 bytes to 1 MiB
+(18 classes). A request for N bytes is served from the smallest
+class >= N. The maximum supported size is queryable via
+``rte_fastmem_max_size()``.
+
+With power-of-two classes, worst-case internal fragmentation is
+just under 50% (e.g., a 33-byte request occupies a 64-byte
+slot). Assuming a uniform distribution of request sizes, the
+average waste is 25%. In practice, DPDK workloads tend to
+cluster at or near powers of two, so typical waste is lower.
+
+Requests exceeding the maximum are rejected with ``E2BIG``.
+
+
+Implementation
+--------------
+
+Fastmem organizes memory in three layers: backing memzones, slabs,
+and per-lcore caches.
+
+Backing memory and slabs
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Backing memory is obtained from EAL as 128 MiB IOVA-contiguous
+memzones, each aligned to 2 MiB. A memzone is partitioned into
+64 fixed-size, 2 MiB **slabs**. Slabs are the unit of memory
+that moves between size classes: a free slab can be assigned to
+any bin on demand, and an empty slab (all objects freed) returns
+to the free-slab pool for reuse by another size class.
+
+The 2 MiB slab alignment is the key structural property. Given
+any object pointer, the allocator recovers the owning slab by
+masking off the low 21 bits — no radix tree, hash table, or
+memzone lookup is needed. This makes the free path fast: a
+single pointer-mask load reaches the slab header, which
+identifies the size class and bin.
+
+Each slab reserves 64 bytes at offset 0 for its header. The
+remaining space is divided into fixed-size slots equal to the
+size class. Allocated objects carry no per-object metadata; the
+full slot is available to the caller.
+
+Three-level allocation hierarchy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. **Per-lcore cache** — a bounded LIFO stack of free object
+ pointers, one per (lcore, size class, socket). Allocation
+ pops; free pushes. No lock is needed because only the owning
+ lcore accesses its cache.
+
+2. **Bin** — one per (size class, socket). Owns the partial and
+ full slab lists. A spinlock serializes bulk transfers between
+ the bin and per-lcore caches. Most traffic is absorbed by the
+ caches, so bin-lock contention is low.
+
+3. **Free-slab pool** — one per socket. A spinlock protects slab
+ acquisition and release. These events are rare relative to
+ object-level operations (a single small-object slab serves
+ thousands of allocations).
+
+On a cache miss (empty on alloc, full on free), the cache
+exchanges objects with the bin in bulk, targeting half-full to
+maximize headroom in both directions.
+
+Cache sizing
+~~~~~~~~~~~~
+
+Cache capacity varies by size class to bound per-lcore memory
+footprint:
+
+* Classes 8 B through 4 KiB: capacity 64.
+* Larger classes: capacity halves per class (32, 16, 8, 4),
+ flooring at 4.
+
+Even the largest classes remain cached. The capacity curve
+ensures that small, frequent allocations get the highest cache
+hit rate, while large allocations still avoid the bin lock on
+most operations.
+
+
+Statistics
+----------
+
+Fastmem maintains always-on, per-lcore counters that track
+allocation and free activity. Statistics are queryable at four
+levels of granularity: global summary, per size class, per lcore,
+and per lcore per class.
+
+``rte_fastmem_classes()`` returns the number of size classes and
+optionally fills an array with their sizes.
+
+See ``rte_fastmem.h`` for the full statistics API.
+
+
+Secondary Processes
+-------------------
+
+Fastmem works transparently in DPDK secondary processes. The shared
+state is discovered automatically on first allocation.
+
+Secondary processes do not use per-lcore caches; every allocation and
+free acquires the bin spinlock directly. This is acceptable for
+control-plane secondaries with low allocation rates. The primary
+process should pre-reserve sufficient backing memory with
+``rte_fastmem_reserve()`` since secondaries cannot grow the pool.
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index e6f24945b0..c85196c85e 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -28,6 +28,7 @@ Memory Management
mempool_lib
mbuf_lib
multi_proc_support
+ fastmem_lib
CPU Management
--
2.43.0
^ permalink raw reply related
* [RFC v2 0/3] lib/fastmem: fast small-object allocator
From: Mattias Rönnblom @ 2026-05-26 8:57 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Mattias Rönnblom
In-Reply-To: <20260525103642.55255-4-hofors@lysator.liu.se>
This RFC introduces fastmem, a general-purpose small-object allocator
for DPDK. It is intended to replace per-type mempools with a single
allocator that handles arbitrary sizes, grows on demand, and matches
mempool-level performance on the hot path.
Motivation
----------
DPDK applications commonly maintain many mempools — one per object
type (connections, sessions, timers, work items). Each must be sized
up front, wastes memory when over-provisioned, and cannot serve
objects of a different size. Fastmem eliminates this by accepting
arbitrary sizes at runtime, backed by a slab allocator that
repurposes memory across size classes as demand shifts.
Design
------
Three-layer architecture:
1. Backing memory: 128 MiB IOVA-contiguous memzones from EAL,
reserved lazily (or pre-reserved for deterministic latency).
2. Slabs: 2 MiB, 2 MiB-aligned regions carved from memzones.
The alignment enables O(1) slab lookup from any object pointer
via bitmask — no radix tree or index structure. Slabs move
freely between 18 power-of-2 size classes (8 B to 1 MiB).
3. Per-lcore caches: bounded LIFO stacks (no locks on the hot
path). Cache misses trigger bulk transfers to/from the shared
bin under a spinlock.
Key properties:
- Zero per-object metadata in the production build.
- NUMA-aware, with per-socket bins and free-slab pools.
- DMA-usable memory with O(1) virt-to-IOVA translation.
- Bulk alloc/free with all-or-nothing semantics.
- Backing memory never returned during lifetime (slabs recycled).
- Non-EAL threads supported (bypass cache, take bin lock).
- Secondary process support (lazy attach, no per-lcore caches).
API surface
-----------
rte_fastmem_init / deinit
rte_fastmem_reserve
rte_fastmem_set_limit / get_limit
rte_fastmem_alloc / alloc_socket
rte_fastmem_alloc_bulk / alloc_bulk_socket
rte_fastmem_free / free_bulk
rte_fastmem_hlookup / halloc / halloc_bulk / hfree / hfree_bulk
rte_fastmem_virt2iova
rte_fastmem_cache_flush
rte_fastmem_max_size / classes
rte_fastmem_stats / stats_class / stats_lcore / stats_lcore_class
rte_fastmem_stats_reset
All APIs are marked __rte_experimental.
Performance
-----------
The single-object hot path is roughly 2–3× the cost of mempool
and an order of magnitude faster than rte_malloc. Under
multi-lcore contention, fastmem scales similarly to mempool,
while rte_malloc collapses.
Limitations
-----------
- Maximum allocation: 1 MiB. Larger requests should use rte_malloc.
- Power-of-2 classes only; worst-case internal fragmentation ~50%.
- Backing memory not reclaimable short of deinit.
Future work
-----------
- Lcore-affine allocations (false-sharing-free by construction).
- Mempool ops driver for transparent drop-in use.
- Debug mode (cookies, double-free detection, poison-on-free).
- Telemetry integration.
- EAL integration, allowing EAL-internal subsystems to use
fastmem for their small-object allocations.
Mattias Rönnblom (3):
doc: add fastmem programming guide
lib: add fastmem library
app/test: add fastmem test suite
app/test/meson.build | 3 +
app/test/test_fastmem.c | 1673 ++++++++++++++++++++++++
app/test/test_fastmem_perf.c | 1040 +++++++++++++++
app/test/test_fastmem_profile.c | 157 +++
doc/api/doxy-api-index.md | 1 +
doc/api/doxy-api.conf.in | 1 +
doc/guides/prog_guide/fastmem_lib.rst | 314 +++++
doc/guides/prog_guide/index.rst | 1 +
lib/fastmem/meson.build | 6 +
lib/fastmem/rte_fastmem.c | 1694 +++++++++++++++++++++++++
lib/fastmem/rte_fastmem.h | 774 +++++++++++
lib/meson.build | 1 +
12 files changed, 5665 insertions(+)
create mode 100644 app/test/test_fastmem.c
create mode 100644 app/test/test_fastmem_perf.c
create mode 100644 app/test/test_fastmem_profile.c
create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
create mode 100644 lib/fastmem/meson.build
create mode 100644 lib/fastmem/rte_fastmem.c
create mode 100644 lib/fastmem/rte_fastmem.h
--
2.43.0
^ permalink raw reply
* RE: [PATCH v5] mempool: improve cache behaviour and performance
From: Morten Brørup @ 2026-05-26 8:57 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, Andrew Rybchenko, Jingjing Wu, Praveen Shetty,
Hemant Agrawal, Sachin Saxena
In-Reply-To: <ahCAgitAAK3e5Kf3@bricha3-mobl1.ger.corp.intel.com>
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 22 May 2026 18.13
>
> On Sun, Apr 19, 2026 at 09:55:26AM +0000, Morten Brørup wrote:
> > This patch refactors the mempool cache to eliminate some unexpected
> > behaviour and reduce the mempool cache miss rate.
> >
> > 1.
> > The actual cache size was 1.5 times the cache size specified at run-
> time
> > mempool creation.
> > This was obviously not expected by application developers.
> >
> > 2.
> > In get operations, the check for when to use the cache as bounce
> buffer
> > did not respect the run-time configured cache size,
> > but compared to the build time maximum possible cache size
> > (RTE_MEMPOOL_CACHE_MAX_SIZE, default 512).
> > E.g. with a configured cache size of 32 objects, getting 256 objects
> > would first fetch 32 + 256 = 288 objects into the cache,
> > and then move the 256 objects from the cache to the destination
> memory,
> > instead of fetching the 256 objects directly to the destination
> memory.
> > This had a performance cost.
> > However, this is unlikely to occur in real applications, so it is not
> > important in itself.
> >
> > 3.
> > When putting objects into a mempool, and the mempool cache did not
> have
> > free space for so many objects,
> > the cache was flushed completely, and the new objects were then put
> into
> > the cache.
> > I.e. the cache drain level was zero.
> > This (complete cache flush) meant that a subsequent get operation
> (with
> > the same number of objects) completely emptied the cache,
> > so another subsequent get operation required replenishing the cache.
> >
> > Similarly,
> > When getting objects from a mempool, and the mempool cache did not
> hold so
> > many objects,
> > the cache was replenished to cache->size + remaining objects,
> > and then (the remaining part of) the requested objects were fetched
> via
> > the cache,
> > which left the cache filled (to cache->size) at completion.
> > I.e. the cache refill level was cache->size (plus some, depending on
> > request size).
> >
> > (1) was improved by generally comparing to cache->size instead of
> > cache->flushthresh, when considering the capacity of the cache.
> > The cache->flushthresh field is kept for API/ABI compatibility
> purposes,
> > and initialized to cache->size instead of cache->size * 1.5.
> >
> > (2) was improved by generally comparing to cache->size / 2 instead of
> > RTE_MEMPOOL_CACHE_MAX_SIZE, when checking the bounce buffer limit.
> >
> > (3) was improved by flushing and replenishing the cache by half its
> size,
> > so a flush/refill can be followed randomly by get or put requests.
> > This also reduced the number of objects in each flush/refill
> operation.
> >
> > As a consequence of these changes, the size of the array holding the
> > objects in the cache (cache->objs[]) no longer needs to be
> > 2 * RTE_MEMPOOL_CACHE_MAX_SIZE, and can be reduced to
> > RTE_MEMPOOL_CACHE_MAX_SIZE at an API/ABI breaking release.
> >
> > Performance data:
> > With a real WAN Optimization application, where the number of
> allocated
> > packets varies (as they are held in e.g. shaper queues), the mempool
> > cache miss rate dropped from ca. 1/20 objects to ca. 1/48 objects.
> > This was deployed in production at an ISP, and using an effective
> cache
> > size of 384 objects.
> >
> > As a consequence of the improved mempool cache algorithm, some
> drivers
> > were updated accordingly:
> > - The Intel idpf PMD was updated regarding how much to backfill the
> > mempool cache in the AVX512 code.
> > - The NXP dpaa and dpaa2 mempool drivers were updated to not set the
> > mempool cache flush threshold; doing this no longer has any effect,
> and
> > thus became superfluous.
> >
> > Bugzilla ID: 1027
> > Fixes: ea5dd2744b90 ("mempool: cache optimisations")
> > Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
> > ---
> > Depends-on: patch-163181 ("net/intel: do not bypass mbuf lib for mbuf
> fast-free")
> > ---
> > v5:
> > * Flush the cache from the bottom, where objects are colder, and move
> down
> > the remaining objects, which are hotter.
> > * In the Intel idpf PMD, move up the hot objects in the cache and
> refill
> > with cold objects at the bottom.
> > v4:
> > * Added Bugzilla ID.
> > * Added Fixes tag. For reference only.
> > * Moved fast-free related update of Intel common driver out as a
> separate
> > patch, and depend on that patch.
> > * Omitted unrelated changes to the Intel idpf AVX512 driver,
> specifically
> > fixing an indentation and adding mbuf instrumentation.
> > * Omitted unrelated changes to the mempool library, specifically
> adding
> > __rte_restrict and changing a couple of comments to proper
> sentences.
> > * Please checkpatches by swapping operators in a couple of
> comparisons.
> > v3:
> > * Fixed my copy-paste bug in idpf_splitq_rearm().
> > v2:
> > * Fixed issue found by abidiff:
> > Reverted cache objects array size reduction. Added a note instead.
> > * Added missing mbuf instrumentation to the Intel idpf AVX512 driver.
> > * Updated idpf_splitq_rearm() like idpf_singleq_rearm().
> > * Added a few more __rte_assume(). (Inspired by AI review)
> > * Updated NXP dpaa and dpaa2 mempool drivers to not set mempool cache
> > flush threshold.
> > * Added release notes.
> > * Added deprecation notes.
> > ---
> > doc/guides/rel_notes/deprecation.rst | 7 ++
> > doc/guides/rel_notes/release_26_07.rst | 10 +++
> > drivers/mempool/dpaa/dpaa_mempool.c | 14 ----
> > drivers/mempool/dpaa2/dpaa2_hw_mempool.c | 14 ----
> > .../net/intel/idpf/idpf_common_rxtx_avx512.c | 52 +++++++++++---
> > lib/mempool/rte_mempool.c | 14 +---
> > lib/mempool/rte_mempool.h | 70 ++++++++++++-----
> --
> > 7 files changed, 104 insertions(+), 77 deletions(-)
> >
> Can the idpf and dpaa changes be made in separate patches, so we can
> review
> the mempool changes along in a single patch? Even if the commits can't
> work
> logically together, perhaps they can be separated for review, and then
> squashed on apply?
Sure.
The idpf changes are required due to the driver bypassing the mempool API.
If such direct access to the mempool cache is required, it would be better to add a mempool cache zero-copy API.
The dpaa changes are simple clean-ups.
They set a variable which this mempool change makes obsolete, so no need to set it.
^ permalink raw reply
* [PATCH v3 25/25] bus: add class device conversion macro
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Nicolas Chautru, Parav Pandit,
Xueming Li, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Ashish Gupta,
Fan Zhang, Ankur Dwivedi, Anoob Joseph, Tejasree Kondoj,
Gagandeep Singh, Hemant Agrawal, Pavan Nikhilesh, Shijith Thotton,
Tirthendu Sarkar, Jerin Jacob, Shepard Siegel, Ed Czeck,
John Miller, Igor Russkikh, Steven Webster, Matt Peters,
Selwin Sebastian, Julien Aube, Kishore Padmanabha, Ajit Khaparde,
Nithin Dabilpuram, Kiran Kumar K, Sunil Kumar Kori, Satha Rao,
Harman Kalra, Potnuri Bharat Teja, Sachin Saxena, Shai Brandes,
Evgeny Schemeilin, Ron Beider, Amit Bernstein, Wajeeh Atrash,
Vanshika Shukla, John Daley, Hyong Youb Kim, Jeroen de Borst,
Joshua Washington, Xiaoyun Wang, Feifei Wang, Xingui Yang,
Chengwen Feng, Praveen Shetty, Vladimir Medvedkin,
Anatoly Burakov, Jingjing Wu, Rosen Xu, Dimon Zhao, Leon Yu,
Sam Chen, Long Li, Wei Hu, Chaoyong He, Jiawen Wu, Zaiyu Wang,
Vamsi Attunuru, Devendra Singh Rawat, Alok Prasad, Howard Wang,
Chunhao Lin, Xing Wang, Javen Xu, Wenbo Cao, Andrew Rybchenko,
Maciej Czekaj, Maxime Coquelin, Jochen Behrens, Renyong Wan,
Na Na, Rong Qian, Xiaoxiong Zhang, Dongwei Xu, Junlong Wang,
Ming Ran
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
Add a new helper macro RTE_CLASS_TO_BUS_DEVICE that provides a unified
way to convert from any device class (ethdev, cryptodev, eventdev, etc.)
to a bus-specific device type. This macro works with any device class
that has a 'device' field pointing to struct rte_device.
Remove the bus-specific ethdev convenience macros (RTE_ETH_DEV_TO_PCI,
RTE_ETH_DEV_TO_AUXILIARY, RTE_ETH_DEV_TO_VDEV) and replace all uses
with the generic RTE_CLASS_TO_BUS_DEVICE macro.
Convert all drivers to use RTE_CLASS_TO_BUS_DEVICE instead of
the pattern RTE_BUS_DEVICE(dev->device).
Simplify code that was using an intermediate struct rte_device pointer
by applying RTE_CLASS_TO_BUS_DEVICE directly on the device class pointer.
This reduces code duplication and provides a consistent interface that
can be used for all device classes.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/baseband/acc/rte_acc100_pmd.c | 4 +-
drivers/baseband/acc/rte_vrb_pmd.c | 2 +-
.../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 4 +-
drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2 +-
drivers/bus/auxiliary/bus_auxiliary_driver.h | 3 --
drivers/bus/cdx/bus_cdx_driver.h | 2 -
drivers/bus/pci/bus_pci_driver.h | 3 --
drivers/bus/vdev/bus_vdev_driver.h | 3 --
drivers/compress/octeontx/otx_zip.c | 2 +-
drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 2 +-
drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c | 3 +-
drivers/crypto/octeontx/otx_cryptodev_ops.c | 4 +-
drivers/event/cnxk/cnxk_eventdev.c | 2 +-
drivers/event/dlb2/pf/dlb2_pf.c | 2 +-
drivers/event/skeleton/skeleton_eventdev.c | 2 +-
drivers/net/ark/ark_ethdev.c | 2 +-
drivers/net/atlantic/atl_ethdev.c | 12 +++---
drivers/net/avp/avp_ethdev.c | 22 +++++-----
drivers/net/axgbe/axgbe_ethdev.c | 4 +-
drivers/net/bnx2x/bnx2x_ethdev.c | 2 +-
drivers/net/bnxt/bnxt_ethdev.c | 12 +++---
drivers/net/bnxt/tf_ulp/bnxt_ulp.c | 4 +-
drivers/net/cnxk/cnxk_ethdev.c | 2 +-
drivers/net/cnxk/cnxk_ethdev_ops.c | 2 +-
drivers/net/cxgbe/cxgbe_ethdev.c | 4 +-
drivers/net/cxgbe/cxgbevf_ethdev.c | 4 +-
drivers/net/dpaa/dpaa_ethdev.c | 16 +++-----
drivers/net/dpaa2/dpaa2_ethdev.c | 8 ++--
drivers/net/dpaa2/dpaa2_recycle.c | 6 +--
drivers/net/ena/ena_ethdev.c | 10 ++---
drivers/net/enetc/enetc4_ethdev.c | 4 +-
drivers/net/enetc/enetc4_vf.c | 4 +-
drivers/net/enetc/enetc_ethdev.c | 2 +-
drivers/net/enic/enic_ethdev.c | 4 +-
drivers/net/enic/enic_fm_flow.c | 6 +--
drivers/net/enic/enic_vf_representor.c | 2 +-
drivers/net/gve/gve_ethdev.c | 2 +-
drivers/net/hinic/hinic_pmd_ethdev.c | 8 ++--
drivers/net/hinic3/base/hinic3_hwdev.c | 7 ++--
drivers/net/hinic3/hinic3_ethdev.c | 16 ++++----
drivers/net/hns3/hns3_cmd.c | 2 +-
drivers/net/hns3/hns3_common.c | 8 ++--
drivers/net/hns3/hns3_ethdev.c | 6 +--
drivers/net/hns3/hns3_ethdev_vf.c | 6 +--
drivers/net/hns3/hns3_rxtx.c | 4 +-
drivers/net/intel/cpfl/cpfl_ethdev.c | 4 +-
drivers/net/intel/cpfl/cpfl_ethdev.h | 2 +-
drivers/net/intel/e1000/em_ethdev.c | 12 +++---
drivers/net/intel/e1000/em_rxtx.c | 2 +-
drivers/net/intel/e1000/igb_ethdev.c | 30 +++++++-------
drivers/net/intel/e1000/igb_pf.c | 2 +-
drivers/net/intel/e1000/igc_ethdev.c | 22 +++++-----
drivers/net/intel/fm10k/fm10k_ethdev.c | 16 ++++----
drivers/net/intel/i40e/i40e_ethdev.c | 28 ++++++-------
drivers/net/intel/i40e/i40e_ethdev.h | 2 +-
drivers/net/intel/iavf/iavf_ethdev.c | 8 ++--
drivers/net/intel/ice/ice_dcf.c | 6 +--
drivers/net/intel/ice/ice_ethdev.c | 6 +--
drivers/net/intel/ice/ice_ethdev.h | 2 +-
drivers/net/intel/idpf/idpf_ethdev.h | 2 +-
drivers/net/intel/ipn3ke/ipn3ke_ethdev.h | 3 --
drivers/net/intel/ipn3ke/ipn3ke_representor.c | 6 +--
drivers/net/intel/ixgbe/ixgbe_ethdev.c | 40 +++++++++----------
drivers/net/intel/ixgbe/ixgbe_flow.c | 4 +-
drivers/net/intel/ixgbe/ixgbe_pf.c | 2 +-
drivers/net/intel/ixgbe/ixgbe_tm.c | 2 +-
.../net/intel/ixgbe/ixgbe_vf_representor.c | 2 +-
drivers/net/intel/ixgbe/rte_pmd_ixgbe.c | 20 +++++-----
drivers/net/nbl/nbl_core.c | 2 +-
drivers/net/nbl/nbl_dev/nbl_dev.c | 6 +--
drivers/net/netvsc/hn_ethdev.c | 3 +-
drivers/net/nfp/nfp_ethdev.c | 8 ++--
drivers/net/nfp/nfp_ethdev_vf.c | 6 +--
drivers/net/nfp/nfp_net_common.c | 8 ++--
drivers/net/ngbe/ngbe_ethdev.c | 20 +++++-----
drivers/net/ngbe/ngbe_ethdev_vf.c | 16 ++++----
drivers/net/ngbe/ngbe_pf.c | 2 +-
drivers/net/octeon_ep/otx_ep_ethdev.c | 2 +-
drivers/net/octeon_ep/otx_ep_mbox.c | 6 +--
drivers/net/qede/qede_ethdev.c | 6 +--
drivers/net/r8169/r8169_ethdev.c | 6 +--
drivers/net/rnp/rnp_ethdev.c | 6 +--
drivers/net/sfc/sfc.c | 4 +-
drivers/net/sfc/sfc_ethdev.c | 2 +-
drivers/net/sfc/sfc_intr.c | 10 ++---
drivers/net/sfc/sfc_rx.c | 3 +-
drivers/net/sfc/sfc_sriov.c | 2 +-
drivers/net/sfc/sfc_tx.c | 3 +-
drivers/net/thunderx/nicvf_ethdev.c | 4 +-
drivers/net/txgbe/txgbe_ethdev.c | 26 ++++++------
drivers/net/txgbe/txgbe_ethdev_vf.c | 16 ++++----
drivers/net/txgbe/txgbe_flow.c | 4 +-
drivers/net/txgbe/txgbe_pf.c | 2 +-
drivers/net/txgbe/txgbe_tm.c | 2 +-
drivers/net/virtio/virtio_pci_ethdev.c | 11 ++---
drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 +-
drivers/net/xsc/xsc_ethdev.c | 2 +-
drivers/net/zxdh/zxdh_ethdev.c | 8 ++--
drivers/raw/ifpga/afu_pmd_n3000.c | 4 +-
lib/eal/include/bus_driver.h | 18 +++++++++
100 files changed, 336 insertions(+), 340 deletions(-)
diff --git a/drivers/baseband/acc/rte_acc100_pmd.c b/drivers/baseband/acc/rte_acc100_pmd.c
index 061f595a98..cbcacc7aa3 100644
--- a/drivers/baseband/acc/rte_acc100_pmd.c
+++ b/drivers/baseband/acc/rte_acc100_pmd.c
@@ -3993,7 +3993,7 @@ acc100_dequeue_ldpc_dec(struct rte_bbdev_queue_data *q_data,
static void
acc100_bbdev_init(struct rte_bbdev *dev, struct rte_pci_driver *drv)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev->dev_ops = &acc100_bbdev_ops;
dev->enqueue_enc_ops = acc100_enqueue_enc;
@@ -4646,7 +4646,7 @@ rte_acc_configure(const char *dev_name, struct rte_acc_conf *conf)
dev_name);
return -ENODEV;
}
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(bbdev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(bbdev, *pci_dev);
rte_bbdev_log(INFO, "Configure dev id %x", pci_dev->id.device_id);
if (pci_dev->id.device_id == ACC100_PF_DEVICE_ID)
return acc100_configure(dev_name, conf);
diff --git a/drivers/baseband/acc/rte_vrb_pmd.c b/drivers/baseband/acc/rte_vrb_pmd.c
index fe23c01b5c..1f85e33462 100644
--- a/drivers/baseband/acc/rte_vrb_pmd.c
+++ b/drivers/baseband/acc/rte_vrb_pmd.c
@@ -4353,7 +4353,7 @@ vrb2_dequeue_mldts(struct rte_bbdev_queue_data *q_data,
static void
vrb_bbdev_init(struct rte_bbdev *dev, struct rte_pci_driver *drv)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct acc_device *d = dev->data->dev_private;
dev->dev_ops = &vrb_bbdev_ops;
diff --git a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
index cb805a1732..45bd171ca7 100644
--- a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
+++ b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
@@ -2873,7 +2873,7 @@ fpga_5gnr_dequeue_ldpc_dec(struct rte_bbdev_queue_data *q_data,
static void
fpga_5gnr_fec_init(struct rte_bbdev *dev, struct rte_pci_driver *drv)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev->dev_ops = &fpga_5gnr_ops;
dev->enqueue_ldpc_enc_ops = fpga_5gnr_enqueue_ldpc_enc;
@@ -3376,7 +3376,7 @@ int rte_fpga_5gnr_fec_configure(const char *dev_name, const struct rte_fpga_5gnr
dev_name);
return -ENODEV;
}
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(bbdev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(bbdev, *pci_dev);
rte_bbdev_log(INFO, "Configure dev id %x", pci_dev->id.device_id);
if (pci_dev->id.device_id == VC_5GNR_PF_DEVICE_ID)
return vc_5gnr_configure(dev_name, conf);
diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
index d27164c6f4..04ac445820 100644
--- a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
+++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
@@ -2316,7 +2316,7 @@ fpga_dequeue_dec(struct rte_bbdev_queue_data *q_data,
static void
fpga_lte_fec_init(struct rte_bbdev *dev, struct rte_pci_driver *drv)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev->dev_ops = &fpga_ops;
dev->enqueue_enc_ops = fpga_enqueue_enc;
diff --git a/drivers/bus/auxiliary/bus_auxiliary_driver.h b/drivers/bus/auxiliary/bus_auxiliary_driver.h
index cab5f86d03..65e1814ec0 100644
--- a/drivers/bus/auxiliary/bus_auxiliary_driver.h
+++ b/drivers/bus/auxiliary/bus_auxiliary_driver.h
@@ -128,9 +128,6 @@ struct rte_auxiliary_driver {
uint32_t drv_flags; /**< Flags RTE_AUXILIARY_DRV_*. */
};
-#define RTE_ETH_DEV_TO_AUXILIARY(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_auxiliary_device)
-
/** Device driver needs IOVA as VA and cannot work with IOVA as PA */
#define RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA 0x002
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index d443178404..01684466ed 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -60,8 +60,6 @@ struct rte_cdx_device {
struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
-#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_BUS_DEVICE((eth_dev)->device, struct rte_cdx_device)
-
#ifdef __cplusplus
/** C++ macro used to help building up tables of device IDs. */
#define RTE_CDX_DEVICE(vend, dev) \
diff --git a/drivers/bus/pci/bus_pci_driver.h b/drivers/bus/pci/bus_pci_driver.h
index cb7039f8d6..c04ebddf59 100644
--- a/drivers/bus/pci/bus_pci_driver.h
+++ b/drivers/bus/pci/bus_pci_driver.h
@@ -47,9 +47,6 @@ struct rte_pci_device {
/**< Handler of VFIO request interrupt */
};
-#define RTE_ETH_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
-
#ifdef __cplusplus
/** C++ macro used to help building up tables of device IDs */
#define RTE_PCI_DEVICE(vend, dev) \
diff --git a/drivers/bus/vdev/bus_vdev_driver.h b/drivers/bus/vdev/bus_vdev_driver.h
index 8d114e4b3b..ecfc5384fc 100644
--- a/drivers/bus/vdev/bus_vdev_driver.h
+++ b/drivers/bus/vdev/bus_vdev_driver.h
@@ -19,9 +19,6 @@ struct rte_vdev_device {
struct rte_device device; /**< Inherit core device */
};
-#define RTE_ETH_DEV_TO_VDEV(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_vdev_device)
-
static inline const char *
rte_vdev_device_name(const struct rte_vdev_device *dev)
{
diff --git a/drivers/compress/octeontx/otx_zip.c b/drivers/compress/octeontx/otx_zip.c
index 8673561a81..7cf3283680 100644
--- a/drivers/compress/octeontx/otx_zip.c
+++ b/drivers/compress/octeontx/otx_zip.c
@@ -142,7 +142,7 @@ zipvf_push_command(struct zipvf_qp *qp, union zip_inst_s *cmd)
int
zipvf_create(struct rte_compressdev *compressdev)
{
- struct rte_pci_device *pdev = RTE_BUS_DEVICE(compressdev->device, *pdev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(compressdev, *pdev);
struct zip_vf *zipvf = NULL;
char *dev_name = compressdev->data->name;
void *vbar0;
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_ops.c b/drivers/crypto/cnxk/cnxk_cryptodev_ops.c
index f437350539..d3cf1ddd57 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_ops.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_ops.c
@@ -481,7 +481,7 @@ cnxk_cpt_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
if (dev->data->queue_pairs[qp_id] != NULL)
cnxk_cpt_queue_pair_release(dev, qp_id);
- pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (pci_dev->mem_resource[2].addr == NULL) {
plt_err("Invalid PCI mem address");
diff --git a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
index d7b53723e7..3d980d096f 100644
--- a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
+++ b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
@@ -4383,7 +4383,6 @@ static int
dpaa2_sec_dev_init(struct rte_cryptodev *cryptodev)
{
struct dpaa2_sec_dev_private *internals;
- struct rte_device *dev = cryptodev->device;
struct rte_dpaa2_device *dpaa2_dev;
struct rte_security_ctx *security_instance;
struct fsl_mc_io *dpseci;
@@ -4392,7 +4391,7 @@ dpaa2_sec_dev_init(struct rte_cryptodev *cryptodev)
int retcode, hw_id;
PMD_INIT_FUNC_TRACE();
- dpaa2_dev = RTE_BUS_DEVICE(dev, *dpaa2_dev);
+ dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(cryptodev, *dpaa2_dev);
hw_id = dpaa2_dev->object_id;
cryptodev->driver_id = cryptodev_driver_id;
diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.c b/drivers/crypto/octeontx/otx_cryptodev_ops.c
index a499c8d0bc..d6d1b2cea9 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_ops.c
+++ b/drivers/crypto/octeontx/otx_cryptodev_ops.c
@@ -156,7 +156,7 @@ otx_cpt_que_pair_setup(struct rte_cryptodev *dev,
DEFAULT_CMD_QLEN);
}
- pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (pci_dev->mem_resource[0].addr == NULL) {
CPT_LOG_ERR("PCI mem address null");
@@ -1001,7 +1001,7 @@ static struct rte_cryptodev_ops cptvf_ops = {
int
otx_cpt_dev_create(struct rte_cryptodev *c_dev)
{
- struct rte_pci_device *pdev = RTE_BUS_DEVICE(c_dev->device, *pdev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(c_dev, *pdev);
struct cpt_vf *cptvf = NULL;
void *reg_base;
char dev_name[32];
diff --git a/drivers/event/cnxk/cnxk_eventdev.c b/drivers/event/cnxk/cnxk_eventdev.c
index 6f000ff49e..272ba235a4 100644
--- a/drivers/event/cnxk/cnxk_eventdev.c
+++ b/drivers/event/cnxk/cnxk_eventdev.c
@@ -654,7 +654,7 @@ cnxk_sso_init(struct rte_eventdev *event_dev)
return -ENOMEM;
}
- pci_dev = RTE_BUS_DEVICE(event_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(event_dev, *pci_dev);
dev->sso.pci_dev = pci_dev;
*(uint64_t *)mz->addr = (uint64_t)dev;
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 82075bbf0b..c78783e59d 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -784,7 +784,7 @@ dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
dlb2_pf_iface_fn_ptrs_init();
- pci_dev = RTE_BUS_DEVICE(eventdev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eventdev, *pci_dev);
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
diff --git a/drivers/event/skeleton/skeleton_eventdev.c b/drivers/event/skeleton/skeleton_eventdev.c
index 4292644fde..c0e06e4aa0 100644
--- a/drivers/event/skeleton/skeleton_eventdev.c
+++ b/drivers/event/skeleton/skeleton_eventdev.c
@@ -332,7 +332,7 @@ skeleton_eventdev_init(struct rte_eventdev *eventdev)
if (rte_eal_process_type() != RTE_PROC_PRIMARY)
return 0;
- pci_dev = RTE_BUS_DEVICE(eventdev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eventdev, *pci_dev);
skel->reg_base = (uintptr_t)pci_dev->mem_resource[0].addr;
if (!skel->reg_base) {
diff --git a/drivers/net/ark/ark_ethdev.c b/drivers/net/ark/ark_ethdev.c
index 8b25ed948f..d6e34021ce 100644
--- a/drivers/net/ark/ark_ethdev.c
+++ b/drivers/net/ark/ark_ethdev.c
@@ -315,7 +315,7 @@ ark_dev_init(struct rte_eth_dev *dev)
if (ret)
return ret;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
rte_eth_copy_pci_info(dev, pci_dev);
dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
diff --git a/drivers/net/atlantic/atl_ethdev.c b/drivers/net/atlantic/atl_ethdev.c
index 2925dc2478..d55d6d50bb 100644
--- a/drivers/net/atlantic/atl_ethdev.c
+++ b/drivers/net/atlantic/atl_ethdev.c
@@ -359,7 +359,7 @@ static int
eth_atl_dev_init(struct rte_eth_dev *eth_dev)
{
struct atl_adapter *adapter = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct aq_hw_s *hw = ATL_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
int err = 0;
@@ -478,7 +478,7 @@ static int
atl_dev_start(struct rte_eth_dev *dev)
{
struct aq_hw_s *hw = ATL_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
int status;
@@ -606,7 +606,7 @@ atl_dev_stop(struct rte_eth_dev *dev)
struct rte_eth_link link;
struct aq_hw_s *hw =
ATL_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
PMD_INIT_FUNC_TRACE();
@@ -687,7 +687,7 @@ atl_dev_set_link_down(struct rte_eth_dev *dev)
static int
atl_dev_close(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct aq_hw_s *hw;
int ret;
@@ -1093,7 +1093,7 @@ atl_fw_version_get(struct rte_eth_dev *dev, char *fw_version, size_t fw_size)
static int
atl_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev_info->max_rx_queues = AQ_HW_MAX_RX_QUEUES;
dev_info->max_tx_queues = AQ_HW_MAX_TX_QUEUES;
@@ -1344,7 +1344,7 @@ atl_dev_link_status_print(struct rte_eth_dev *dev)
#ifdef DEBUG
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
PMD_DRV_LOG(DEBUG, "PCI Address: " PCI_PRI_FMT,
pci_dev->addr.domain,
diff --git a/drivers/net/avp/avp_ethdev.c b/drivers/net/avp/avp_ethdev.c
index 3bc5171336..8af6c45381 100644
--- a/drivers/net/avp/avp_ethdev.c
+++ b/drivers/net/avp/avp_ethdev.c
@@ -361,7 +361,7 @@ static void *
avp_dev_translate_address(struct rte_eth_dev *eth_dev,
rte_iova_t host_phys_addr)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_mem_resource *resource;
struct rte_avp_memmap_info *info;
struct rte_avp_memmap *map;
@@ -414,7 +414,7 @@ avp_dev_version_check(uint32_t version)
static int
avp_dev_check_regions(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_avp_memmap_info *memmap;
struct rte_avp_device_info *info;
struct rte_mem_resource *resource;
@@ -550,7 +550,7 @@ _avp_set_rx_queue_mappings(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
static void
_avp_set_queue_counts(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct avp_dev *avp = AVP_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
struct rte_avp_device_info *host_info;
void *addr;
@@ -610,7 +610,7 @@ avp_dev_attach(struct rte_eth_dev *eth_dev)
* re-run the device create utility which will parse the new host info
* and setup the AVP device queue pointers.
*/
- ret = avp_dev_create(RTE_ETH_DEV_TO_PCI(eth_dev), eth_dev);
+ ret = avp_dev_create(RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device), eth_dev);
if (ret < 0) {
PMD_DRV_LOG_LINE(ERR, "Failed to re-create AVP device, ret=%d",
ret);
@@ -664,7 +664,7 @@ static void
avp_dev_interrupt_handler(void *data)
{
struct rte_eth_dev *eth_dev = data;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR].addr;
uint32_t status, value;
int ret;
@@ -723,7 +723,7 @@ avp_dev_interrupt_handler(void *data)
static int
avp_dev_enable_interrupts(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR].addr;
int ret;
@@ -748,7 +748,7 @@ avp_dev_enable_interrupts(struct rte_eth_dev *eth_dev)
static int
avp_dev_disable_interrupts(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR].addr;
int ret;
@@ -773,7 +773,7 @@ avp_dev_disable_interrupts(struct rte_eth_dev *eth_dev)
static int
avp_dev_setup_interrupts(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret;
/* register a callback handler with UIO for interrupt notifications */
@@ -793,7 +793,7 @@ avp_dev_setup_interrupts(struct rte_eth_dev *eth_dev)
static int
avp_dev_migration_pending(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR].addr;
uint32_t value;
@@ -954,7 +954,7 @@ eth_avp_dev_init(struct rte_eth_dev *eth_dev)
struct rte_pci_device *pci_dev;
int ret;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
eth_dev->dev_ops = &avp_eth_dev_ops;
eth_dev->rx_pkt_burst = &avp_recv_pkts;
eth_dev->tx_pkt_burst = &avp_xmit_pkts;
@@ -1977,7 +1977,7 @@ avp_dev_tx_queue_release_all(struct rte_eth_dev *eth_dev)
static int
avp_dev_configure(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct avp_dev *avp = AVP_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
struct rte_avp_device_info *host_info;
struct rte_avp_device_config config;
diff --git a/drivers/net/axgbe/axgbe_ethdev.c b/drivers/net/axgbe/axgbe_ethdev.c
index c14d04a11d..aceec49c0c 100644
--- a/drivers/net/axgbe/axgbe_ethdev.c
+++ b/drivers/net/axgbe/axgbe_ethdev.c
@@ -2230,7 +2230,7 @@ eth_axgbe_dev_init(struct rte_eth_dev *eth_dev)
rte_bit_relaxed_set32(AXGBE_STOPPED, &pdata->dev_state);
pdata->eth_dev = eth_dev;
- pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
pdata->pci_dev = pci_dev;
pdata->xgmac_regs =
@@ -2453,7 +2453,7 @@ axgbe_dev_close(struct rte_eth_dev *eth_dev)
return 0;
pdata = eth_dev->data->dev_private;
- pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
axgbe_dev_clear_queues(eth_dev);
/* disable uio intr before callback unregister */
diff --git a/drivers/net/bnx2x/bnx2x_ethdev.c b/drivers/net/bnx2x/bnx2x_ethdev.c
index 7b96e1acee..4f1f97a999 100644
--- a/drivers/net/bnx2x/bnx2x_ethdev.c
+++ b/drivers/net/bnx2x/bnx2x_ethdev.c
@@ -639,7 +639,7 @@ bnx2x_common_dev_init(struct rte_eth_dev *eth_dev, int is_vf)
/* Extract key data structures */
sc = eth_dev->data->dev_private;
- pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
pci_addr = pci_dev->addr;
snprintf(sc->devinfo.name, NAME_SIZE, PCI_SHORT_PRI_FMT ":dpdk-port-%u",
diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index 071093aabc..5506037cc2 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -862,7 +862,7 @@ static int bnxt_alloc_prev_ring_stats(struct bnxt *bp)
static int bnxt_start_nic(struct bnxt *bp)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(bp->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(bp->eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
uint32_t queue_id, base = BNXT_MISC_VEC_ID;
@@ -1167,7 +1167,7 @@ uint64_t bnxt_eth_rss_support(struct bnxt *bp)
static int bnxt_dev_info_get_op(struct rte_eth_dev *eth_dev,
struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pdev = RTE_BUS_DEVICE(eth_dev->device, *pdev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
struct bnxt *bp = eth_dev->data->dev_private;
uint16_t max_vnics, i, j, vpool, vrxq;
unsigned int max_rx_rings;
@@ -1719,7 +1719,7 @@ static int bnxt_ptp_start(struct bnxt *bp)
static int bnxt_dev_stop(struct rte_eth_dev *eth_dev)
{
struct bnxt *bp = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rte_eth_link link;
uint16_t i;
@@ -5143,7 +5143,7 @@ bool bnxt_stratus_device(struct bnxt *bp)
static int bnxt_map_pci_bars(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct bnxt *bp = eth_dev->data->dev_private;
/* enable device (incl. PCI PM wakeup), and bus-mastering */
@@ -6600,7 +6600,7 @@ bnxt_parse_dev_args(struct bnxt *bp, struct rte_devargs *devargs)
*/
static int bnxt_drv_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct bnxt *bp = eth_dev->data->dev_private;
int rc = 0;
@@ -6684,7 +6684,7 @@ static int bnxt_drv_init(struct rte_eth_dev *eth_dev)
static int
bnxt_dev_init(struct rte_eth_dev *eth_dev, void *params __rte_unused)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
static int version_printed;
struct bnxt *bp;
int rc;
diff --git a/drivers/net/bnxt/tf_ulp/bnxt_ulp.c b/drivers/net/bnxt/tf_ulp/bnxt_ulp.c
index e1e2c0e878..8ff0e20364 100644
--- a/drivers/net/bnxt/tf_ulp/bnxt_ulp.c
+++ b/drivers/net/bnxt/tf_ulp/bnxt_ulp.c
@@ -192,7 +192,7 @@ ulp_session_init(struct bnxt *bp,
if (!bp)
return NULL;
- pci_dev = RTE_BUS_DEVICE(bp->eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(bp->eth_dev, *pci_dev);
pci_addr = &pci_dev->addr;
pthread_mutex_lock(&bnxt_ulp_global_mutex);
@@ -556,7 +556,7 @@ bnxt_ulp_port_deinit(struct bnxt *bp)
bp->eth_dev->data->port_id);
/* Get the session details */
- pci_dev = RTE_BUS_DEVICE(bp->eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(bp->eth_dev, *pci_dev);
pci_addr = &pci_dev->addr;
pthread_mutex_lock(&bnxt_ulp_global_mutex);
session = ulp_get_session(bp, pci_addr);
diff --git a/drivers/net/cnxk/cnxk_ethdev.c b/drivers/net/cnxk/cnxk_ethdev.c
index 06d1c9b362..7ae16186c6 100644
--- a/drivers/net/cnxk/cnxk_ethdev.c
+++ b/drivers/net/cnxk/cnxk_ethdev.c
@@ -2177,7 +2177,7 @@ cnxk_eth_dev_init(struct rte_eth_dev *eth_dev)
if (rte_eal_process_type() != RTE_PROC_PRIMARY)
return 0;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
rte_eth_copy_pci_info(eth_dev, pci_dev);
/* Parse devargs string */
diff --git a/drivers/net/cnxk/cnxk_ethdev_ops.c b/drivers/net/cnxk/cnxk_ethdev_ops.c
index 49e77e49a6..460ffa32b6 100644
--- a/drivers/net/cnxk/cnxk_ethdev_ops.c
+++ b/drivers/net/cnxk/cnxk_ethdev_ops.c
@@ -7,7 +7,7 @@
int
cnxk_nix_info_get(struct rte_eth_dev *eth_dev, struct rte_eth_dev_info *devinfo)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev);
int max_rx_pktlen;
diff --git a/drivers/net/cxgbe/cxgbe_ethdev.c b/drivers/net/cxgbe/cxgbe_ethdev.c
index 0c337a6cc8..82e67eeff1 100644
--- a/drivers/net/cxgbe/cxgbe_ethdev.c
+++ b/drivers/net/cxgbe/cxgbe_ethdev.c
@@ -1704,7 +1704,7 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)
eth_dev->dev_ops = &cxgbe_eth_dev_ops;
eth_dev->rx_pkt_burst = &cxgbe_recv_pkts;
eth_dev->tx_pkt_burst = &cxgbe_xmit_pkts;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* for secondary processes, we attach to ethdevs allocated by primary
* and do minimal initialization.
@@ -1767,7 +1767,7 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)
static int eth_cxgbe_dev_uninit(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
uint16_t port_id;
int err = 0;
diff --git a/drivers/net/cxgbe/cxgbevf_ethdev.c b/drivers/net/cxgbe/cxgbevf_ethdev.c
index d8eba8afef..750dc7da4d 100644
--- a/drivers/net/cxgbe/cxgbevf_ethdev.c
+++ b/drivers/net/cxgbe/cxgbevf_ethdev.c
@@ -113,7 +113,7 @@ static int eth_cxgbevf_dev_init(struct rte_eth_dev *eth_dev)
eth_dev->dev_ops = &cxgbevf_eth_dev_ops;
eth_dev->rx_pkt_burst = &cxgbe_recv_pkts;
eth_dev->tx_pkt_burst = &cxgbe_xmit_pkts;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* for secondary processes, we attach to ethdevs allocated by primary
* and do minimal initialization.
@@ -177,7 +177,7 @@ static int eth_cxgbevf_dev_init(struct rte_eth_dev *eth_dev)
static int eth_cxgbevf_dev_uninit(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
uint16_t port_id;
int err = 0;
diff --git a/drivers/net/dpaa/dpaa_ethdev.c b/drivers/net/dpaa/dpaa_ethdev.c
index d4b4793f16..9f976d179b 100644
--- a/drivers/net/dpaa/dpaa_ethdev.c
+++ b/drivers/net/dpaa/dpaa_ethdev.c
@@ -217,7 +217,6 @@ dpaa_eth_dev_configure(struct rte_eth_dev *dev)
uint64_t rx_offloads = eth_conf->rxmode.offloads;
uint64_t tx_offloads = eth_conf->txmode.offloads;
struct dpaa_if *dpaa_intf = dev->data->dev_private;
- struct rte_device *rdev = dev->device;
struct rte_eth_link *link = &dev->data->dev_link;
struct rte_dpaa_device *dpaa_dev;
struct fman_if *fif = dev->process_private;
@@ -230,7 +229,7 @@ dpaa_eth_dev_configure(struct rte_eth_dev *dev)
PMD_INIT_FUNC_TRACE();
- dpaa_dev = RTE_BUS_DEVICE(rdev, *dpaa_dev);
+ dpaa_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa_dev);
intr_handle = dpaa_dev->intr_handle;
__fif = container_of(fif, struct __fman_if, __if);
@@ -426,13 +425,12 @@ dpaa_supported_ptypes_get(struct rte_eth_dev *dev, size_t *no_of_elements)
static void dpaa_interrupt_handler(void *param)
{
struct rte_eth_dev *dev = param;
- struct rte_device *rdev = dev->device;
struct rte_dpaa_device *dpaa_dev;
struct rte_intr_handle *intr_handle;
uint64_t buf;
int bytes_read;
- dpaa_dev = RTE_BUS_DEVICE(rdev, *dpaa_dev);
+ dpaa_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa_dev);
intr_handle = dpaa_dev->intr_handle;
if (rte_intr_fd_get(intr_handle) < 0)
@@ -502,7 +500,6 @@ static int dpaa_eth_dev_close(struct rte_eth_dev *dev)
{
struct fman_if *fif = dev->process_private;
struct __fman_if *__fif;
- struct rte_device *rdev = dev->device;
struct rte_dpaa_device *dpaa_dev;
struct rte_intr_handle *intr_handle;
struct rte_eth_link *link = &dev->data->dev_link;
@@ -530,7 +527,7 @@ static int dpaa_eth_dev_close(struct rte_eth_dev *dev)
}
}
- dpaa_dev = RTE_BUS_DEVICE(rdev, *dpaa_dev);
+ dpaa_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa_dev);
intr_handle = dpaa_dev->intr_handle;
__fif = container_of(fif, struct __fman_if, __if);
@@ -1267,9 +1264,8 @@ int dpaa_eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,
/* Set up the device interrupt handler */
if (dev->intr_handle == NULL) {
struct rte_dpaa_device *dpaa_dev;
- struct rte_device *rdev = dev->device;
- dpaa_dev = RTE_BUS_DEVICE(rdev, *dpaa_dev);
+ dpaa_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa_dev);
dev->intr_handle = dpaa_dev->intr_handle;
if (rte_intr_vec_list_alloc(dev->intr_handle,
NULL, dpaa_push_queue_max_num())) {
@@ -2119,7 +2115,7 @@ dpaa_dev_init_secondary(struct rte_eth_dev *eth_dev)
PMD_INIT_FUNC_TRACE();
- dpaa_device = RTE_BUS_DEVICE(eth_dev->device, *dpaa_device);
+ dpaa_device = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa_device);
dev_id = dpaa_device->id.dev_id;
cfg = dpaa_get_eth_port_cfg(dev_id);
fman_intf = cfg->fman_if;
@@ -2236,7 +2232,7 @@ dpaa_dev_init(struct rte_eth_dev *eth_dev)
PMD_INIT_FUNC_TRACE();
- dpaa_device = RTE_BUS_DEVICE(eth_dev->device, *dpaa_device);
+ dpaa_device = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa_device);
dev_id = dpaa_device->id.dev_id;
dpaa_intf = eth_dev->data->dev_private;
cfg = dpaa_get_eth_port_cfg(dev_id);
diff --git a/drivers/net/dpaa2/dpaa2_ethdev.c b/drivers/net/dpaa2/dpaa2_ethdev.c
index dc9ea700ac..803a8321e0 100644
--- a/drivers/net/dpaa2/dpaa2_ethdev.c
+++ b/drivers/net/dpaa2/dpaa2_ethdev.c
@@ -1339,7 +1339,6 @@ dpaa2_eth_setup_irqs(struct rte_eth_dev *dev, int enable)
static int
dpaa2_dev_start(struct rte_eth_dev *dev)
{
- struct rte_device *rdev = dev->device;
struct rte_dpaa2_device *dpaa2_dev;
struct rte_eth_dev_data *data = dev->data;
struct dpaa2_dev_priv *priv = data->dev_private;
@@ -1351,7 +1350,7 @@ dpaa2_dev_start(struct rte_eth_dev *dev)
int ret, i;
struct rte_intr_handle *intr_handle;
- dpaa2_dev = RTE_BUS_DEVICE(rdev, *dpaa2_dev);
+ dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa2_dev);
intr_handle = dpaa2_dev->intr_handle;
PMD_INIT_FUNC_TRACE();
@@ -1458,12 +1457,11 @@ dpaa2_dev_stop(struct rte_eth_dev *dev)
struct fsl_mc_io *dpni = dev->process_private;
int ret;
struct rte_eth_link link;
- struct rte_device *rdev = dev->device;
struct rte_intr_handle *intr_handle;
struct rte_dpaa2_device *dpaa2_dev;
uint16_t i;
- dpaa2_dev = RTE_BUS_DEVICE(rdev, *dpaa2_dev);
+ dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa2_dev);
intr_handle = dpaa2_dev->intr_handle;
PMD_INIT_FUNC_TRACE();
@@ -2918,7 +2916,7 @@ dpaa2_dev_init(struct rte_eth_dev *eth_dev)
return 0;
}
- dpaa2_dev = RTE_BUS_DEVICE(dev, *dpaa2_dev);
+ dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa2_dev);
hw_id = dpaa2_dev->object_id;
ret = dpni_open(dpni_dev, CMD_PRI_LOW, hw_id, &priv->token);
diff --git a/drivers/net/dpaa2/dpaa2_recycle.c b/drivers/net/dpaa2/dpaa2_recycle.c
index 14416c41d0..f78d12362e 100644
--- a/drivers/net/dpaa2/dpaa2_recycle.c
+++ b/drivers/net/dpaa2/dpaa2_recycle.c
@@ -607,9 +607,8 @@ lx_serdes_eth_lpbk(uint16_t mac_id, int en)
int
dpaa2_dev_recycle_config(struct rte_eth_dev *eth_dev)
{
- struct rte_device *dev = eth_dev->device;
struct dpaa2_dev_priv *priv = eth_dev->data->dev_private;
- struct rte_dpaa2_device *dpaa2_dev = RTE_BUS_DEVICE(dev, *dpaa2_dev);
+ struct rte_dpaa2_device *dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa2_dev);
struct fsl_mc_io *dpni_dev = eth_dev->process_private;
struct dpni_port_cfg port_cfg;
int ret;
@@ -674,9 +673,8 @@ dpaa2_dev_recycle_config(struct rte_eth_dev *eth_dev)
int
dpaa2_dev_recycle_deconfig(struct rte_eth_dev *eth_dev)
{
- struct rte_device *dev = eth_dev->device;
struct dpaa2_dev_priv *priv = eth_dev->data->dev_private;
- struct rte_dpaa2_device *dpaa2_dev = RTE_BUS_DEVICE(dev, *dpaa2_dev);
+ struct rte_dpaa2_device *dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa2_dev);
struct fsl_mc_io *dpni_dev = eth_dev->process_private;
struct dpni_port_cfg port_cfg;
int ret = 0;
diff --git a/drivers/net/ena/ena_ethdev.c b/drivers/net/ena/ena_ethdev.c
index ea4afbc75d..ad2ac6dbbf 100644
--- a/drivers/net/ena/ena_ethdev.c
+++ b/drivers/net/ena/ena_ethdev.c
@@ -924,7 +924,7 @@ static inline void ena_indirect_table_release(struct ena_adapter *adapter)
static int ena_close(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ena_adapter *adapter = dev->data->dev_private;
struct ena_com_dev *ena_dev = &adapter->ena_dev;
@@ -1457,7 +1457,7 @@ static int ena_stop(struct rte_eth_dev *dev)
{
struct ena_adapter *adapter = dev->data->dev_private;
struct ena_com_dev *ena_dev = &adapter->ena_dev;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint16_t i;
int rc;
@@ -1503,7 +1503,7 @@ static int ena_create_io_queue(struct rte_eth_dev *dev, struct ena_ring *ring)
{
struct ena_adapter *adapter = ring->adapter;
struct ena_com_dev *ena_dev = &adapter->ena_dev;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ena_com_create_io_ctx ctx =
/* policy set to _HOST just to satisfy icc compiler */
@@ -2422,7 +2422,7 @@ static int eth_ena_dev_init(struct rte_eth_dev *eth_dev)
adapter->edev_data = eth_dev->data;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
PMD_INIT_LOG_LINE(INFO, "Initializing " PCI_PRI_FMT,
pci_dev->addr.domain,
@@ -3978,7 +3978,7 @@ static int ena_parse_devargs(struct ena_adapter *adapter, struct rte_devargs *de
static int ena_setup_rx_intr(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int rc;
uint16_t vectors_nb, i;
diff --git a/drivers/net/enetc/enetc4_ethdev.c b/drivers/net/enetc/enetc4_ethdev.c
index df9f007473..78eba70a08 100644
--- a/drivers/net/enetc/enetc4_ethdev.c
+++ b/drivers/net/enetc/enetc4_ethdev.c
@@ -956,7 +956,7 @@ enetc4_dev_hw_init(struct rte_eth_dev *eth_dev)
{
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
eth_dev->rx_pkt_burst = &enetc_recv_pkts_nc;
eth_dev->tx_pkt_burst = &enetc_xmit_pkts_nc;
@@ -986,7 +986,7 @@ enetc4_dev_init(struct rte_eth_dev *eth_dev)
{
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int error = 0;
uint32_t si_cap;
struct enetc_hw *enetc_hw = &hw->hw;
diff --git a/drivers/net/enetc/enetc4_vf.c b/drivers/net/enetc/enetc4_vf.c
index 3f257234a0..bec7128e41 100644
--- a/drivers/net/enetc/enetc4_vf.c
+++ b/drivers/net/enetc/enetc4_vf.c
@@ -1249,7 +1249,7 @@ enetc4_vf_dev_init(struct rte_eth_dev *eth_dev)
{
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int error = 0;
uint32_t si_cap;
struct enetc_hw *enetc_hw = &hw->hw;
@@ -1297,7 +1297,7 @@ enetc4_vf_dev_intr(struct rte_eth_dev *eth_dev, bool enable)
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
struct enetc_hw *enetc_hw = &hw->hw;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret = 0;
diff --git a/drivers/net/enetc/enetc_ethdev.c b/drivers/net/enetc/enetc_ethdev.c
index b2bbace16c..f41f3c1803 100644
--- a/drivers/net/enetc/enetc_ethdev.c
+++ b/drivers/net/enetc/enetc_ethdev.c
@@ -886,7 +886,7 @@ static int
enetc_dev_init(struct rte_eth_dev *eth_dev)
{
int error = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
diff --git a/drivers/net/enic/enic_ethdev.c b/drivers/net/enic/enic_ethdev.c
index a853a5047a..2e5cd186f9 100644
--- a/drivers/net/enic/enic_ethdev.c
+++ b/drivers/net/enic/enic_ethdev.c
@@ -454,7 +454,7 @@ static uint32_t speed_capa_from_pci_id(struct rte_eth_dev *eth_dev)
struct rte_pci_device *pdev;
uint16_t id;
- pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
id = pdev->id.subsystem_device_id;
for (m = vic_speed_capa_map; m->sub_devid != 0; m++) {
if (m->sub_devid == id)
@@ -1292,7 +1292,7 @@ static int eth_enic_dev_init(struct rte_eth_dev *eth_dev,
enic->rte_dev = eth_dev;
enic->dev_data = eth_dev->data;
- pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
rte_eth_copy_pci_info(eth_dev, pdev);
enic->pdev = pdev;
addr = &pdev->addr;
diff --git a/drivers/net/enic/enic_fm_flow.c b/drivers/net/enic/enic_fm_flow.c
index c2c3e55206..4b0a513977 100644
--- a/drivers/net/enic/enic_fm_flow.c
+++ b/drivers/net/enic/enic_fm_flow.c
@@ -3229,7 +3229,7 @@ enic_fm_init(struct enic *enic)
if (rte_eth_dev_is_repr(enic->rte_dev))
addr = &VF_ENIC_TO_VF_REP(enic)->bdf;
else
- addr = &RTE_ETH_DEV_TO_PCI(enic->rte_dev)->addr;
+ addr = &RTE_CLASS_TO_BUS_DEVICE(enic->rte_dev, struct rte_pci_device)->addr;
rc = enic_fm_find_vnic(enic, addr, &enic->fm_vnic_handle);
if (rc) {
ENICPMD_LOG(ERR, "cannot find vnic handle for %x:%x:%x",
@@ -3361,7 +3361,7 @@ enic_fm_allocate_switch_domain(struct enic *pf)
if (rte_eth_dev_is_repr(pf->rte_dev))
return -EINVAL;
cur = pf;
- cur_a = &RTE_ETH_DEV_TO_PCI(cur->rte_dev)->addr;
+ cur_a = &RTE_CLASS_TO_BUS_DEVICE(cur->rte_dev, struct rte_pci_device)->addr;
/* Go through ports and find another PF that is on the same adapter */
RTE_ETH_FOREACH_DEV(pid) {
dev = &rte_eth_devices[pid];
@@ -3373,7 +3373,7 @@ enic_fm_allocate_switch_domain(struct enic *pf)
continue;
/* dev is another PF. Is it on the same adapter? */
prev = pmd_priv(dev);
- prev_a = &RTE_ETH_DEV_TO_PCI(dev)->addr;
+ prev_a = &RTE_CLASS_TO_BUS_DEVICE(dev, struct rte_pci_device)->addr;
if (!enic_fm_find_vnic(cur, prev_a, &vnic_h)) {
ENICPMD_LOG(DEBUG, "Port %u (PF BDF %x:%x:%x) and port %u (PF BDF %x:%x:%x domain %u) are on the same VIC",
cur->rte_dev->data->port_id,
diff --git a/drivers/net/enic/enic_vf_representor.c b/drivers/net/enic/enic_vf_representor.c
index 05b2efedcb..fc836100b4 100644
--- a/drivers/net/enic/enic_vf_representor.c
+++ b/drivers/net/enic/enic_vf_representor.c
@@ -655,7 +655,7 @@ int enic_vf_representor_init(struct rte_eth_dev *eth_dev, void *init_params)
}
/* Check for non-existent VFs */
- pdev = RTE_ETH_DEV_TO_PCI(pf->rte_dev);
+ pdev = RTE_CLASS_TO_BUS_DEVICE(pf->rte_dev, *pdev);
if (vf->vf_id >= pdev->max_vfs) {
ENICPMD_LOG(ERR, "VF ID is invalid. vf_id %u max_vfs %u",
vf->vf_id, pdev->max_vfs);
diff --git a/drivers/net/gve/gve_ethdev.c b/drivers/net/gve/gve_ethdev.c
index 73f4935b1f..227e1cc70e 100644
--- a/drivers/net/gve/gve_ethdev.c
+++ b/drivers/net/gve/gve_ethdev.c
@@ -1410,7 +1410,7 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
return 0;
}
- pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
if (!reg_bar) {
diff --git a/drivers/net/hinic/hinic_pmd_ethdev.c b/drivers/net/hinic/hinic_pmd_ethdev.c
index 75534c1ce2..91a4348fb6 100644
--- a/drivers/net/hinic/hinic_pmd_ethdev.c
+++ b/drivers/net/hinic/hinic_pmd_ethdev.c
@@ -1234,7 +1234,7 @@ static int hinic_dev_stop(struct rte_eth_dev *dev)
static void hinic_disable_interrupt(struct rte_eth_dev *dev)
{
struct hinic_nic_dev *nic_dev = HINIC_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
int ret, retries = 0;
rte_bit_relaxed_clear32(HINIC_DEV_INTR_EN, &nic_dev->dev_status);
@@ -2745,7 +2745,7 @@ static int hinic_nic_dev_create(struct rte_eth_dev *eth_dev)
eth_dev->data->name);
return -ENOMEM;
}
- nic_dev->hwdev->pcidev_hdl = RTE_ETH_DEV_TO_PCI(eth_dev);
+ nic_dev->hwdev->pcidev_hdl = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *nic_dev->hwdev->pcidev_hdl);
/* init osdep*/
rc = hinic_osdep_init(nic_dev->hwdev);
@@ -3086,7 +3086,7 @@ static int hinic_func_init(struct rte_eth_dev *eth_dev)
u32 mac_size;
int rc;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* EAL is SECONDARY and eth_dev is already created */
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
@@ -3218,7 +3218,7 @@ static int hinic_dev_init(struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
PMD_DRV_LOG(INFO, "Initializing pf hinic-" PCI_PRI_FMT " in %s process",
pci_dev->addr.domain, pci_dev->addr.bus,
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index 5d12cf7b5f..d09a8f7e7d 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -74,10 +74,11 @@ struct mgmt_event_handle {
};
bool
-hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *rte_dev)
+hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
- return ((RTE_ETH_DEV_TO_PCI(rte_dev)->kdrv == RTE_PCI_KDRV_VFIO) &&
- (rte_vfio_noiommu_is_enabled() != 1));
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
+
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && rte_vfio_noiommu_is_enabled() != 1;
}
int
diff --git a/drivers/net/hinic3/hinic3_ethdev.c b/drivers/net/hinic3/hinic3_ethdev.c
index f4eb788686..361e52f7b9 100644
--- a/drivers/net/hinic3/hinic3_ethdev.c
+++ b/drivers/net/hinic3/hinic3_ethdev.c
@@ -1474,7 +1474,7 @@ hinic3_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t sq_id)
int
hinic3_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = PCI_DEV_TO_INTR_HANDLE(pci_dev);
struct hinic3_nic_dev *nic_dev = HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
uint16_t msix_intr;
@@ -1493,7 +1493,7 @@ hinic3_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
int
hinic3_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = PCI_DEV_TO_INTR_HANDLE(pci_dev);
struct hinic3_nic_dev *nic_dev = HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
uint16_t msix_intr;
@@ -1695,7 +1695,7 @@ static void
hinic3_disable_interrupt(struct rte_eth_dev *dev)
{
struct hinic3_nic_dev *nic_dev = HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!hinic3_get_bit(HINIC3_DEV_INIT, &nic_dev->dev_status))
return;
@@ -1710,7 +1710,7 @@ static void
hinic3_enable_interrupt(struct rte_eth_dev *dev)
{
struct hinic3_nic_dev *nic_dev = HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!hinic3_get_bit(HINIC3_DEV_INIT, &nic_dev->dev_status))
return;
@@ -2080,7 +2080,7 @@ hinic3_dev_release(struct rte_eth_dev *eth_dev)
{
struct hinic3_nic_dev *nic_dev =
HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(eth_dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int qid;
/* Release io resource. */
@@ -3394,7 +3394,7 @@ hinic3_func_init(struct rte_eth_dev *eth_dev)
struct rte_pci_device *pci_dev = NULL;
int err;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* EAL is secondary and eth_dev is already created. */
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
@@ -3460,7 +3460,7 @@ hinic3_func_init(struct rte_eth_dev *eth_dev)
err = -ENOMEM;
goto alloc_hwdev_mem_fail;
}
- nic_dev->hwdev->pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ nic_dev->hwdev->pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *nic_dev->hwdev->pci_dev);
nic_dev->hwdev->dev_handle = nic_dev;
nic_dev->hwdev->eth_dev = eth_dev;
nic_dev->hwdev->port_id = eth_dev->data->port_id;
@@ -3616,7 +3616,7 @@ hinic3_dev_init(struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
PMD_DRV_LOG(INFO, "Initializing %.4x:%.2x:%.2x.%x in %s process",
pci_dev->addr.domain, pci_dev->addr.bus,
diff --git a/drivers/net/hns3/hns3_cmd.c b/drivers/net/hns3/hns3_cmd.c
index ad4ef9e189..34e12e7359 100644
--- a/drivers/net/hns3/hns3_cmd.c
+++ b/drivers/net/hns3/hns3_cmd.c
@@ -551,7 +551,7 @@ hns3_set_dcb_capability(struct hns3_hw *hw)
return;
eth_dev = &rte_eth_devices[hw->data->port_id];
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
device_id = pci_dev->id.device_id;
if (device_id == HNS3_DEV_ID_25GE_RDMA ||
diff --git a/drivers/net/hns3/hns3_common.c b/drivers/net/hns3/hns3_common.c
index 28d7e94ffb..29b51856d9 100644
--- a/drivers/net/hns3/hns3_common.c
+++ b/drivers/net/hns3/hns3_common.c
@@ -812,7 +812,7 @@ hns3_init_ring_with_vector(struct hns3_hw *hw)
int
hns3_map_rx_interrupt(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct hns3_hw *hw = HNS3_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint16_t base = RTE_INTR_VEC_ZERO_OFFSET;
@@ -878,7 +878,7 @@ hns3_map_rx_interrupt(struct rte_eth_dev *dev)
void
hns3_unmap_rx_interrupt(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct hns3_adapter *hns = dev->data->dev_private;
struct hns3_hw *hw = &hns->hw;
@@ -912,7 +912,7 @@ int
hns3_restore_rx_interrupt(struct hns3_hw *hw)
{
struct rte_eth_dev *dev = &rte_eth_devices[hw->data->port_id];
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint16_t q_id;
int ret;
@@ -943,7 +943,7 @@ hns3_get_pci_revision_id(struct hns3_hw *hw, uint8_t *revision_id)
int ret;
eth_dev = &rte_eth_devices[hw->data->port_id];
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
ret = rte_pci_read_config(pci_dev, &revision, 1, RTE_PCI_REVISION_ID);
if (ret != 1) {
hns3_err(hw, "failed to read pci revision id, ret = %d", ret);
diff --git a/drivers/net/hns3/hns3_ethdev.c b/drivers/net/hns3/hns3_ethdev.c
index a66fc5d81a..dbe26df77d 100644
--- a/drivers/net/hns3/hns3_ethdev.c
+++ b/drivers/net/hns3/hns3_ethdev.c
@@ -4526,8 +4526,7 @@ hns3_get_port_supported_speed(struct rte_eth_dev *eth_dev)
static int
hns3_init_pf(struct rte_eth_dev *eth_dev)
{
- struct rte_device *dev = eth_dev->device;
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_adapter *hns = eth_dev->data->dev_private;
struct hns3_hw *hw = &hns->hw;
int ret;
@@ -4656,8 +4655,7 @@ static void
hns3_uninit_pf(struct rte_eth_dev *eth_dev)
{
struct hns3_adapter *hns = eth_dev->data->dev_private;
- struct rte_device *dev = eth_dev->device;
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_hw *hw = &hns->hw;
PMD_INIT_FUNC_TRACE();
diff --git a/drivers/net/hns3/hns3_ethdev_vf.c b/drivers/net/hns3/hns3_ethdev_vf.c
index 59fb790240..84e733a0f5 100644
--- a/drivers/net/hns3/hns3_ethdev_vf.c
+++ b/drivers/net/hns3/hns3_ethdev_vf.c
@@ -1622,7 +1622,7 @@ hns3vf_clear_vport_list(struct hns3_hw *hw)
static int
hns3vf_init_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_adapter *hns = eth_dev->data->dev_private;
struct hns3_hw *hw = &hns->hw;
int ret;
@@ -1739,7 +1739,7 @@ hns3vf_notify_uninit(struct hns3_hw *hw)
static void
hns3vf_uninit_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_adapter *hns = eth_dev->data->dev_private;
struct hns3_hw *hw = &hns->hw;
@@ -2377,7 +2377,7 @@ static int
hns3vf_reinit_dev(struct hns3_adapter *hns)
{
struct rte_eth_dev *eth_dev = &rte_eth_devices[hns->hw.data->port_id];
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_hw *hw = &hns->hw;
int ret;
diff --git a/drivers/net/hns3/hns3_rxtx.c b/drivers/net/hns3/hns3_rxtx.c
index 3528fda8a5..3b299c2f21 100644
--- a/drivers/net/hns3/hns3_rxtx.c
+++ b/drivers/net/hns3/hns3_rxtx.c
@@ -1093,7 +1093,7 @@ hns3_dev_all_rx_queue_intr_enable(struct hns3_hw *hw, bool en)
int
hns3_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct hns3_hw *hw = HNS3_DEV_PRIVATE_TO_HW(dev->data->dev_private);
@@ -3066,7 +3066,7 @@ hns3_tx_push_get_queue_tail_reg(struct rte_eth_dev *dev, uint16_t queue_id)
#define HNS3_TX_PUSH_QUICK_DOORBELL_OFFSET 64
#define HNS3_TX_PUSH_PCI_BAR_INDEX 4
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint8_t bar_id = HNS3_TX_PUSH_PCI_BAR_INDEX;
/*
diff --git a/drivers/net/intel/cpfl/cpfl_ethdev.c b/drivers/net/intel/cpfl/cpfl_ethdev.c
index 03599e6432..562b2dd3c9 100644
--- a/drivers/net/intel/cpfl/cpfl_ethdev.c
+++ b/drivers/net/intel/cpfl/cpfl_ethdev.c
@@ -2764,7 +2764,7 @@ cpfl_dev_vport_init(struct rte_eth_dev *dev, void *init_params)
uint8_t p2p_q_vc_out_info[IDPF_DFLT_MBX_BUF_SIZE] = {0};
struct cpfl_vport_id vi;
struct cpchnl2_vport_id v_id;
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
int ret = 0;
dev->dev_ops = &cpfl_eth_dev_ops;
@@ -2836,7 +2836,7 @@ cpfl_dev_vport_init(struct rte_eth_dev *dev, void *init_params)
}
/* get the vport info */
if (adapter->base.hw.device_id == IXD_DEV_ID_VCPF) {
- pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
vi.func_type = VCPF_CPCHNL2_FTYPE_LAN_VF;
vi.pf_id = CPFL_HOST0_CPF_ID;
vi.vf_id = pci_dev->addr.function;
diff --git a/drivers/net/intel/cpfl/cpfl_ethdev.h b/drivers/net/intel/cpfl/cpfl_ethdev.h
index 56f8f39829..4cc73f216b 100644
--- a/drivers/net/intel/cpfl/cpfl_ethdev.h
+++ b/drivers/net/intel/cpfl/cpfl_ethdev.h
@@ -298,7 +298,7 @@ int vcpf_add_queues(struct cpfl_adapter_ext *adapter);
int vcpf_del_queues(struct cpfl_adapter_ext *adapter);
#define CPFL_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
+ RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
#define CPFL_ADAPTER_TO_EXT(p) \
container_of((p), struct cpfl_adapter_ext, base)
#define CPFL_DEV_TO_VPORT(dev) \
diff --git a/drivers/net/intel/e1000/em_ethdev.c b/drivers/net/intel/e1000/em_ethdev.c
index 9e15e882b9..62ab57268f 100644
--- a/drivers/net/intel/e1000/em_ethdev.c
+++ b/drivers/net/intel/e1000/em_ethdev.c
@@ -273,7 +273,7 @@ eth_em_dev_is_ich8(struct e1000_hw *hw)
static int
eth_em_dev_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(eth_dev->data->dev_private);
@@ -563,7 +563,7 @@ eth_em_start(struct rte_eth_dev *dev)
E1000_DEV_PRIVATE(dev->data->dev_private);
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret, mask;
uint32_t intr_vector = 0;
@@ -762,7 +762,7 @@ eth_em_stop(struct rte_eth_dev *dev)
{
struct rte_eth_link link;
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/*
@@ -816,7 +816,7 @@ eth_em_close(struct rte_eth_dev *dev)
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
@@ -1062,7 +1062,7 @@ static int
eth_em_rx_queue_intr_enable(struct rte_eth_dev *dev, __rte_unused uint16_t queue_id)
{
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/* device interrupts are only subscribed to in primary processes */
@@ -1647,7 +1647,7 @@ static int
eth_em_interrupt_action(struct rte_eth_dev *dev,
struct rte_intr_handle *intr_handle)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_interrupt *intr =
diff --git a/drivers/net/intel/e1000/em_rxtx.c b/drivers/net/intel/e1000/em_rxtx.c
index 54971fe285..e0dcc6a82a 100644
--- a/drivers/net/intel/e1000/em_rxtx.c
+++ b/drivers/net/intel/e1000/em_rxtx.c
@@ -2093,7 +2093,7 @@ em_flush_desc_rings(struct rte_eth_dev *dev)
{
uint32_t fextnvm11, tdlen;
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t pci_cfg_status = 0;
int ret;
diff --git a/drivers/net/intel/e1000/igb_ethdev.c b/drivers/net/intel/e1000/igb_ethdev.c
index ef1599ac38..a4370fe32b 100644
--- a/drivers/net/intel/e1000/igb_ethdev.c
+++ b/drivers/net/intel/e1000/igb_ethdev.c
@@ -529,7 +529,7 @@ igb_intr_enable(struct rte_eth_dev *dev)
E1000_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (rte_intr_allow_others(intr_handle) &&
@@ -546,7 +546,7 @@ igb_intr_disable(struct rte_eth_dev *dev)
{
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (rte_intr_allow_others(intr_handle) &&
@@ -783,7 +783,7 @@ static int
eth_igb_dev_init(struct rte_eth_dev *eth_dev)
{
int error = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
struct e1000_vfta * shadow_vfta =
@@ -1004,7 +1004,7 @@ eth_igbvf_dev_init(struct rte_eth_dev *eth_dev)
return 0;
}
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
rte_eth_copy_pci_info(eth_dev, pci_dev);
hw->device_id = pci_dev->id.device_id;
@@ -1300,7 +1300,7 @@ eth_igb_start(struct rte_eth_dev *dev)
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret, mask;
uint32_t tqavctrl;
@@ -1537,7 +1537,7 @@ static int
eth_igb_stop(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_adapter *adapter =
@@ -1646,7 +1646,7 @@ eth_igb_close(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct rte_eth_link link;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_filter_info *filter_info =
E1000_DEV_PRIVATE_TO_FILTER_INFO(dev->data->dev_private);
@@ -2931,7 +2931,7 @@ static int eth_igb_rxq_interrupt_setup(struct rte_eth_dev *dev)
int ret;
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int misc_shift = rte_intr_allow_others(intr_handle) ? 1 : 0;
struct rte_eth_dev_info dev_info;
@@ -3002,7 +3002,7 @@ eth_igb_interrupt_action(struct rte_eth_dev *dev,
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_interrupt *intr =
E1000_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
int ret;
@@ -3496,7 +3496,7 @@ igbvf_dev_start(struct rte_eth_dev *dev)
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
uint32_t intr_vector = 0;
@@ -3560,7 +3560,7 @@ igbvf_dev_start(struct rte_eth_dev *dev)
static int
igbvf_dev_stop(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(dev->data->dev_private);
@@ -3608,7 +3608,7 @@ igbvf_dev_close(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct rte_ether_addr addr;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
int ret;
PMD_INIT_FUNC_TRACE();
@@ -5410,7 +5410,7 @@ eth_igb_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = E1000_MISC_VEC_ID;
@@ -5434,7 +5434,7 @@ eth_igb_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = E1000_MISC_VEC_ID;
@@ -5516,7 +5516,7 @@ eth_igb_configure_msix_intr(struct rte_eth_dev *dev)
uint32_t vec = E1000_MISC_VEC_ID;
uint32_t base = E1000_MISC_VEC_ID;
uint32_t misc_shift = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/* won't configure msix register if no mapping is done
diff --git a/drivers/net/intel/e1000/igb_pf.c b/drivers/net/intel/e1000/igb_pf.c
index c7588ea57e..50df3daeb7 100644
--- a/drivers/net/intel/e1000/igb_pf.c
+++ b/drivers/net/intel/e1000/igb_pf.c
@@ -29,7 +29,7 @@
static inline uint16_t
dev_num_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
return pci_dev->max_vfs;
}
diff --git a/drivers/net/intel/e1000/igc_ethdev.c b/drivers/net/intel/e1000/igc_ethdev.c
index 727ea36c2b..de35da2c36 100644
--- a/drivers/net/intel/e1000/igc_ethdev.c
+++ b/drivers/net/intel/e1000/igc_ethdev.c
@@ -440,7 +440,7 @@ static void
igc_intr_other_disable(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (rte_intr_allow_others(intr_handle) &&
@@ -460,7 +460,7 @@ igc_intr_other_enable(struct rte_eth_dev *dev)
{
struct igc_interrupt *intr = IGC_DEV_PRIVATE_INTR(dev);
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (rte_intr_allow_others(intr_handle) &&
@@ -576,7 +576,7 @@ static void
eth_igc_interrupt_action(struct rte_eth_dev *dev)
{
struct igc_interrupt *intr = IGC_DEV_PRIVATE_INTR(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
int ret;
@@ -679,7 +679,7 @@ eth_igc_stop(struct rte_eth_dev *dev)
{
struct igc_adapter *adapter = IGC_DEV_PRIVATE(dev);
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rte_eth_link link;
@@ -799,7 +799,7 @@ static void
igc_configure_msix_intr(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_mask;
@@ -882,7 +882,7 @@ igc_rxq_interrupt_setup(struct rte_eth_dev *dev)
{
uint32_t mask;
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int misc_shift = rte_intr_allow_others(intr_handle) ? 1 : 0;
int nb_efd;
@@ -990,7 +990,7 @@ eth_igc_start(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
struct igc_adapter *adapter = IGC_DEV_PRIVATE(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t nsec, sec, baset_l, baset_h, tqavctrl;
struct timespec system_time;
@@ -1307,7 +1307,7 @@ igc_dev_free_queues(struct rte_eth_dev *dev)
static int
eth_igc_close(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
struct igc_adapter *adapter = IGC_DEV_PRIVATE(dev);
@@ -1359,7 +1359,7 @@ igc_identify_hardware(struct rte_eth_dev *dev, struct rte_pci_device *pci_dev)
static int
eth_igc_dev_init(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct igc_adapter *igc = IGC_DEV_PRIVATE(dev);
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
int i, error = 0;
@@ -2257,7 +2257,7 @@ static int
eth_igc_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = IGC_MISC_VEC_ID;
@@ -2280,7 +2280,7 @@ static int
eth_igc_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = IGC_MISC_VEC_ID;
diff --git a/drivers/net/intel/fm10k/fm10k_ethdev.c b/drivers/net/intel/fm10k/fm10k_ethdev.c
index 97f61afec2..ca438d2d02 100644
--- a/drivers/net/intel/fm10k/fm10k_ethdev.c
+++ b/drivers/net/intel/fm10k/fm10k_ethdev.c
@@ -693,7 +693,7 @@ fm10k_dev_rx_init(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct fm10k_macvlan_filter_info *macvlan;
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
int i, ret;
struct fm10k_rx_queue *rxq;
@@ -1161,7 +1161,7 @@ static int
fm10k_dev_stop(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
int i;
@@ -1371,7 +1371,7 @@ fm10k_dev_infos_get(struct rte_eth_dev *dev,
struct rte_eth_dev_info *dev_info)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
PMD_INIT_FUNC_TRACE();
@@ -2364,7 +2364,7 @@ static int
fm10k_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
/* Enable ITR */
if (hw->mac.type == fm10k_mac_pf)
@@ -2381,7 +2381,7 @@ static int
fm10k_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
/* Disable ITR */
if (hw->mac.type == fm10k_mac_pf)
@@ -2397,7 +2397,7 @@ static int
fm10k_dev_rxq_interrupt_setup(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
uint32_t intr_vector, vec;
uint16_t queue_id;
@@ -2794,7 +2794,7 @@ static int
fm10k_dev_close(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
int ret;
@@ -3060,7 +3060,7 @@ static int
eth_fm10k_dev_init(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
int diag, i, ret;
struct fm10k_macvlan_filter_info *macvlan;
diff --git a/drivers/net/intel/i40e/i40e_ethdev.c b/drivers/net/intel/i40e/i40e_ethdev.c
index 100a751225..559df03a69 100644
--- a/drivers/net/intel/i40e/i40e_ethdev.c
+++ b/drivers/net/intel/i40e/i40e_ethdev.c
@@ -981,7 +981,7 @@ is_floating_veb_supported(struct rte_devargs *devargs)
static void
config_floating_veb(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
@@ -1549,7 +1549,7 @@ eth_i40e_dev_init(struct rte_eth_dev *dev, void *init_params __rte_unused)
return 0;
}
i40e_set_default_ptype_table(dev);
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
intr_handle = pci_dev->intr_handle;
rte_eth_copy_pci_info(dev, pci_dev);
@@ -2041,7 +2041,7 @@ void
i40e_vsi_queues_unbind_intr(struct i40e_vsi *vsi)
{
struct rte_eth_dev *dev = I40E_VSI_TO_ETH_DEV(vsi);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_VSI_TO_HW(vsi);
uint16_t msix_vect = vsi->msix_intr;
@@ -2157,7 +2157,7 @@ int
i40e_vsi_queues_bind_intr(struct i40e_vsi *vsi, uint16_t itr_idx)
{
struct rte_eth_dev *dev = I40E_VSI_TO_ETH_DEV(vsi);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_VSI_TO_HW(vsi);
uint16_t msix_vect = vsi->msix_intr;
@@ -2236,7 +2236,7 @@ void
i40e_vsi_enable_queues_intr(struct i40e_vsi *vsi)
{
struct rte_eth_dev *dev = I40E_VSI_TO_ETH_DEV(vsi);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_VSI_TO_HW(vsi);
struct i40e_pf *pf = I40E_VSI_TO_PF(vsi);
@@ -2263,7 +2263,7 @@ void
i40e_vsi_disable_queues_intr(struct i40e_vsi *vsi)
{
struct rte_eth_dev *dev = I40E_VSI_TO_ETH_DEV(vsi);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_VSI_TO_HW(vsi);
struct i40e_pf *pf = I40E_VSI_TO_PF(vsi);
@@ -2431,7 +2431,7 @@ i40e_dev_start(struct rte_eth_dev *dev)
struct i40e_adapter *ad =
I40E_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
int ret, i;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
struct i40e_vsi *vsi;
@@ -2612,7 +2612,7 @@ i40e_dev_stop(struct rte_eth_dev *dev)
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct i40e_vsi *main_vsi = pf->main_vsi;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int i;
@@ -2674,7 +2674,7 @@ i40e_dev_close(struct rte_eth_dev *dev)
{
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_filter_control_settings settings;
struct rte_flow *p_flow;
@@ -3831,7 +3831,7 @@ i40e_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct i40e_vsi *vsi = pf->main_vsi;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev_info->max_rx_queues = vsi->nb_qps;
dev_info->max_tx_queues = vsi->nb_qps;
@@ -4884,7 +4884,7 @@ i40e_pf_parameter_init(struct rte_eth_dev *dev)
{
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_PF_TO_HW(pf);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t qp_count = 0, vsi_count = 0;
if (pci_dev->max_vfs && !hw->func_caps.sr_iov_1_1) {
@@ -10033,7 +10033,7 @@ i40e_dev_flow_ops_get(struct rte_eth_dev *dev,
static void
i40e_enable_extended_tag(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint32_t buf = 0;
int ret;
@@ -11219,7 +11219,7 @@ i40e_dev_get_dcb_info(struct rte_eth_dev *dev,
static int
i40e_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint16_t msix_intr;
@@ -11247,7 +11247,7 @@ i40e_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
static int
i40e_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint16_t msix_intr;
diff --git a/drivers/net/intel/i40e/i40e_ethdev.h b/drivers/net/intel/i40e/i40e_ethdev.h
index dcbdf65047..c39a5a8802 100644
--- a/drivers/net/intel/i40e/i40e_ethdev.h
+++ b/drivers/net/intel/i40e/i40e_ethdev.h
@@ -1467,7 +1467,7 @@ int i40e_vf_representor_init(struct rte_eth_dev *ethdev, void *init_params);
int i40e_vf_representor_uninit(struct rte_eth_dev *ethdev);
#define I40E_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
+ RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
/* I40E_DEV_PRIVATE_TO */
#define I40E_DEV_PRIVATE_TO_PF(adapter) \
diff --git a/drivers/net/intel/iavf/iavf_ethdev.c b/drivers/net/intel/iavf/iavf_ethdev.c
index 1eca20bc9a..4ad9e594bb 100644
--- a/drivers/net/intel/iavf/iavf_ethdev.c
+++ b/drivers/net/intel/iavf/iavf_ethdev.c
@@ -1947,7 +1947,7 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct iavf_adapter *adapter =
IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(adapter);
uint16_t msix_intr;
@@ -1983,7 +1983,7 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
static int
iavf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint16_t msix_intr;
@@ -2767,7 +2767,7 @@ iavf_dev_init(struct rte_eth_dev *eth_dev)
IAVF_DEV_PRIVATE_TO_ADAPTER(eth_dev->data->dev_private);
struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(adapter);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret = 0;
PMD_INIT_FUNC_TRACE();
@@ -2926,7 +2926,7 @@ static int
iavf_dev_close(struct rte_eth_dev *dev)
{
struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct iavf_adapter *adapter =
IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
diff --git a/drivers/net/intel/ice/ice_dcf.c b/drivers/net/intel/ice/ice_dcf.c
index 51716a4d5b..0953fd6668 100644
--- a/drivers/net/intel/ice/ice_dcf.c
+++ b/drivers/net/intel/ice/ice_dcf.c
@@ -658,7 +658,7 @@ ice_dcf_send_aq_cmd(void *dcf_hw, struct ice_aq_desc *desc,
int
ice_dcf_handle_vsi_update_event(struct ice_dcf_hw *hw)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(hw->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(hw->eth_dev, *pci_dev);
int i = 0;
int err = -1;
@@ -738,7 +738,7 @@ dcf_get_vlan_offload_caps_v2(struct ice_dcf_hw *hw)
int
ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret, size;
hw->resetting = false;
@@ -873,7 +873,7 @@ ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
void
ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (hw->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_QOS)
diff --git a/drivers/net/intel/ice/ice_ethdev.c b/drivers/net/intel/ice/ice_ethdev.c
index 715d1522f9..f0a9a8e536 100644
--- a/drivers/net/intel/ice/ice_ethdev.c
+++ b/drivers/net/intel/ice/ice_ethdev.c
@@ -2623,7 +2623,7 @@ ice_dev_init(struct rte_eth_dev *dev)
}
ice_set_default_ptype_table(dev);
- pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
intr_handle = pci_dev->intr_handle;
pf->adapter = ICE_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
@@ -2956,7 +2956,7 @@ ice_dev_close(struct rte_eth_dev *dev)
{
struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct ice_hw *hw = ICE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ice_adapter *ad =
ICE_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
@@ -4520,7 +4520,7 @@ ice_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct ice_hw *hw = ICE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ice_vsi *vsi = pf->main_vsi;
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
bool is_safe_mode = pf->adapter->is_safe_mode;
u64 phy_type_low;
u64 phy_type_high;
diff --git a/drivers/net/intel/ice/ice_ethdev.h b/drivers/net/intel/ice/ice_ethdev.h
index ea73f8bcb3..92baa62cc4 100644
--- a/drivers/net/intel/ice/ice_ethdev.h
+++ b/drivers/net/intel/ice/ice_ethdev.h
@@ -708,7 +708,7 @@ struct ice_vsi_vlan_pvid_info {
};
#define ICE_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
+ RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
/* ICE_DEV_PRIVATE_TO */
#define ICE_DEV_PRIVATE_TO_PF(adapter) \
diff --git a/drivers/net/intel/idpf/idpf_ethdev.h b/drivers/net/intel/idpf/idpf_ethdev.h
index 5105eea1c5..99496c59da 100644
--- a/drivers/net/intel/idpf/idpf_ethdev.h
+++ b/drivers/net/intel/idpf/idpf_ethdev.h
@@ -85,7 +85,7 @@ struct idpf_adapter_ext {
TAILQ_HEAD(idpf_adapter_list, idpf_adapter_ext);
#define IDPF_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
+ RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
#define IDPF_ADAPTER_TO_EXT(p) \
container_of((p), struct idpf_adapter_ext, base)
diff --git a/drivers/net/intel/ipn3ke/ipn3ke_ethdev.h b/drivers/net/intel/ipn3ke/ipn3ke_ethdev.h
index 6d531120b8..505b8f367a 100644
--- a/drivers/net/intel/ipn3ke/ipn3ke_ethdev.h
+++ b/drivers/net/intel/ipn3ke/ipn3ke_ethdev.h
@@ -310,9 +310,6 @@ struct ipn3ke_hw {
uint8_t *hw_addr;
};
-#define RTE_ETH_DEV_TO_AFU(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_afu_device)
-
/**
* PCIe MMIO Access
*/
diff --git a/drivers/net/intel/ipn3ke/ipn3ke_representor.c b/drivers/net/intel/ipn3ke/ipn3ke_representor.c
index cd34d08055..af1af31f1d 100644
--- a/drivers/net/intel/ipn3ke/ipn3ke_representor.c
+++ b/drivers/net/intel/ipn3ke/ipn3ke_representor.c
@@ -2070,7 +2070,7 @@ ipn3ke_rpst_stats_reset(struct rte_eth_dev *ethdev)
return -EINVAL;
}
- afu_dev = RTE_ETH_DEV_TO_AFU(ethdev);
+ afu_dev = RTE_CLASS_TO_BUS_DEVICE(ethdev, *afu_dev);
if (!afu_dev) {
IPN3KE_AFU_PMD_ERR("afu device to reset is NULL!");
return -EINVAL;
@@ -2138,7 +2138,7 @@ ipn3ke_rpst_stats_get
return -EINVAL;
}
- afu_dev = RTE_ETH_DEV_TO_AFU(ethdev);
+ afu_dev = RTE_CLASS_TO_BUS_DEVICE(ethdev, *afu_dev);
if (!afu_dev) {
IPN3KE_AFU_PMD_ERR("afu device to get statistics is NULL!");
return -EINVAL;
@@ -2228,7 +2228,7 @@ ipn3ke_rpst_xstats_get
return -EINVAL;
}
- afu_dev = RTE_ETH_DEV_TO_AFU(ethdev);
+ afu_dev = RTE_CLASS_TO_BUS_DEVICE(ethdev, *afu_dev);
if (!afu_dev) {
IPN3KE_AFU_PMD_ERR("afu device to get statistics is NULL!");
return -EINVAL;
diff --git a/drivers/net/intel/ixgbe/ixgbe_ethdev.c b/drivers/net/intel/ixgbe/ixgbe_ethdev.c
index 57d929cf2c..39db5368b7 100644
--- a/drivers/net/intel/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/intel/ixgbe/ixgbe_ethdev.c
@@ -1085,7 +1085,7 @@ static int
eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
{
struct ixgbe_adapter *ad = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
@@ -1598,7 +1598,7 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
int diag;
uint32_t tc, tcs;
struct ixgbe_adapter *ad = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
@@ -2264,7 +2264,7 @@ ixgbe_vmdq_vlan_hw_filter_enable(struct rte_eth_dev *dev)
static int
ixgbe_check_vf_rss_rxq_num(struct rte_eth_dev *dev, uint16_t nb_rx_q)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
switch (nb_rx_q) {
case 1:
@@ -2506,7 +2506,7 @@ ixgbe_set_vf_rate_limit(struct rte_eth_dev *dev, uint16_t vf,
struct rte_pci_device *pci_dev;
int ret;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
ret = rte_eth_link_get_nowait(dev->data->port_id, &link);
if (ret < 0)
return ret;
@@ -2614,7 +2614,7 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ixgbe_vf_info *vfinfo =
*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
int err;
@@ -2921,7 +2921,7 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ixgbe_vf_info *vfinfo =
*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int vf;
struct ixgbe_tm_conf *tm_conf =
@@ -3082,7 +3082,7 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
{
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int retries = 0;
int ret;
@@ -3970,7 +3970,7 @@ ixgbe_fw_version_get(struct rte_eth_dev *dev, char *fw_version, size_t fw_size)
static int
ixgbe_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct rte_eth_conf *dev_conf = &dev->data->dev_conf;
@@ -4100,7 +4100,7 @@ static int
ixgbevf_dev_info_get(struct rte_eth_dev *dev,
struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -4668,7 +4668,7 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
static void
ixgbe_dev_link_status_print(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
rte_eth_linkstatus_get(dev, &link);
@@ -4780,7 +4780,7 @@ static void
ixgbe_dev_interrupt_delayed_handler(void *param)
{
struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_interrupt *intr =
IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
@@ -5344,7 +5344,7 @@ ixgbe_remove_rar(struct rte_eth_dev *dev, uint32_t index)
static int
ixgbe_set_default_mac_addr(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
/*
* This function calls into the base driver, which in turn will use
@@ -5508,7 +5508,7 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint32_t intr_vector = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int err, mask = 0;
@@ -5621,7 +5621,7 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ixgbe_adapter *adapter = dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/*
@@ -5669,7 +5669,7 @@ static int
ixgbevf_dev_close(struct rte_eth_dev *dev)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
@@ -5975,7 +5975,7 @@ ixgbe_convert_vm_rx_mask_to_val(uint16_t rx_mask, uint32_t orig_val)
static int
ixgbevf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_interrupt *intr =
IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
@@ -6005,7 +6005,7 @@ ixgbevf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = IXGBE_MISC_VEC_ID;
@@ -6025,7 +6025,7 @@ ixgbevf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
static int
ixgbe_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t mask;
struct ixgbe_hw *hw =
@@ -6162,7 +6162,7 @@ ixgbe_set_ivar_map(struct ixgbe_hw *hw, int8_t direction,
static void
ixgbevf_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
@@ -6213,7 +6213,7 @@ ixgbevf_configure_msix(struct rte_eth_dev *dev)
static void
ixgbe_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
diff --git a/drivers/net/intel/ixgbe/ixgbe_flow.c b/drivers/net/intel/ixgbe/ixgbe_flow.c
index 01cd4f9bde..659007980a 100644
--- a/drivers/net/intel/ixgbe/ixgbe_flow.c
+++ b/drivers/net/intel/ixgbe/ixgbe_flow.c
@@ -1272,7 +1272,7 @@ cons_parse_l2_tn_filter(struct rte_eth_dev *dev,
const struct rte_flow_item_e_tag *e_tag_mask;
const struct rte_flow_action *act;
const struct rte_flow_action_vf *act_vf;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!pattern) {
rte_flow_error_set(error, EINVAL,
@@ -1430,7 +1430,7 @@ ixgbe_parse_l2_tn_filter(struct rte_eth_dev *dev,
{
int ret = 0;
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t vf_num;
ret = cons_parse_l2_tn_filter(dev, attr, pattern,
diff --git a/drivers/net/intel/ixgbe/ixgbe_pf.c b/drivers/net/intel/ixgbe/ixgbe_pf.c
index d9a775f99a..2c014a0b96 100644
--- a/drivers/net/intel/ixgbe/ixgbe_pf.c
+++ b/drivers/net/intel/ixgbe/ixgbe_pf.c
@@ -32,7 +32,7 @@
static inline uint16_t
dev_num_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
return pci_dev->max_vfs;
}
diff --git a/drivers/net/intel/ixgbe/ixgbe_tm.c b/drivers/net/intel/ixgbe/ixgbe_tm.c
index 27a821285d..50c7fa228a 100644
--- a/drivers/net/intel/ixgbe/ixgbe_tm.c
+++ b/drivers/net/intel/ixgbe/ixgbe_tm.c
@@ -365,7 +365,7 @@ ixgbe_queue_base_nb_get(struct rte_eth_dev *dev, uint16_t tc_node_no,
uint16_t *base, uint16_t *nb)
{
uint8_t nb_tcs = ixgbe_tc_nb_get(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t vf_num = pci_dev->max_vfs;
*base = 0;
diff --git a/drivers/net/intel/ixgbe/ixgbe_vf_representor.c b/drivers/net/intel/ixgbe/ixgbe_vf_representor.c
index 901d80e406..bad699dd70 100644
--- a/drivers/net/intel/ixgbe/ixgbe_vf_representor.c
+++ b/drivers/net/intel/ixgbe/ixgbe_vf_representor.c
@@ -190,7 +190,7 @@ ixgbe_vf_representor_init(struct rte_eth_dev *ethdev, void *init_params)
representor->pf_ethdev =
((struct ixgbe_vf_representor *)init_params)->pf_ethdev;
- pci_dev = RTE_ETH_DEV_TO_PCI(representor->pf_ethdev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(representor->pf_ethdev, *pci_dev);
if (representor->vf_id >= pci_dev->max_vfs)
return -ENODEV;
diff --git a/drivers/net/intel/ixgbe/rte_pmd_ixgbe.c b/drivers/net/intel/ixgbe/rte_pmd_ixgbe.c
index c2300a8955..35d364fb19 100644
--- a/drivers/net/intel/ixgbe/rte_pmd_ixgbe.c
+++ b/drivers/net/intel/ixgbe/rte_pmd_ixgbe.c
@@ -25,7 +25,7 @@ rte_pmd_ixgbe_set_vf_mac_addr(uint16_t port, uint16_t vf,
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -60,7 +60,7 @@ rte_pmd_ixgbe_ping_vf(uint16_t port, uint16_t vf)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -92,7 +92,7 @@ rte_pmd_ixgbe_set_vf_vlan_anti_spoof(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -123,7 +123,7 @@ rte_pmd_ixgbe_set_vf_mac_anti_spoof(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -153,7 +153,7 @@ rte_pmd_ixgbe_set_vf_vlan_insert(uint16_t port, uint16_t vf, uint16_t vlan_id)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -252,7 +252,7 @@ rte_pmd_ixgbe_set_vf_split_drop_en(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -289,7 +289,7 @@ rte_pmd_ixgbe_set_vf_vlan_stripq(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
if (!is_ixgbe_supported(dev))
@@ -338,7 +338,7 @@ rte_pmd_ixgbe_set_vf_rxmode(uint16_t port, uint16_t vf,
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -386,7 +386,7 @@ rte_pmd_ixgbe_set_vf_rx(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -438,7 +438,7 @@ rte_pmd_ixgbe_set_vf_tx(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
diff --git a/drivers/net/nbl/nbl_core.c b/drivers/net/nbl/nbl_core.c
index df8c0c76ed..6a823e9bfb 100644
--- a/drivers/net/nbl/nbl_core.c
+++ b/drivers/net/nbl/nbl_core.c
@@ -32,7 +32,7 @@ static void nbl_init_func_caps(const struct rte_pci_device *pci_dev, struct nbl_
int nbl_core_init(struct nbl_adapter *adapter, struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
const struct nbl_product_core_ops *product_base_ops = NULL;
struct nbl_common_info *common = NBL_ADAPTER_TO_COMMON(adapter);
int ret = 0;
diff --git a/drivers/net/nbl/nbl_dev/nbl_dev.c b/drivers/net/nbl/nbl_dev/nbl_dev.c
index 2b0413fb7c..35485ea691 100644
--- a/drivers/net/nbl/nbl_dev/nbl_dev.c
+++ b/drivers/net/nbl/nbl_dev/nbl_dev.c
@@ -868,7 +868,7 @@ static int nbl_dev_common_start(struct nbl_dev_mgt *dev_mgt)
struct nbl_dev_net_mgt *net_dev = NBL_DEV_MGT_TO_NET_DEV(dev_mgt);
struct nbl_common_info *common = NBL_DEV_MGT_TO_COMMON(dev_mgt);
struct nbl_board_port_info *board_info;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(net_dev->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(net_dev->eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
u8 *mac;
int ret;
@@ -991,7 +991,7 @@ static void nbl_dev_leonis_stop(void *p)
const struct nbl_common_info *common = NBL_DEV_MGT_TO_COMMON(dev_mgt);
const struct nbl_dispatch_ops *disp_ops = NBL_DEV_MGT_TO_DISP_OPS(dev_mgt);
const struct nbl_channel_ops *chan_ops = NBL_DEV_MGT_TO_CHAN_OPS(dev_mgt);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(net_dev->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(net_dev->eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
u8 *mac;
@@ -1105,7 +1105,7 @@ static int nbl_dev_setup_net_dev(struct nbl_dev_mgt *dev_mgt,
struct nbl_register_net_param register_param = { 0 };
struct nbl_register_net_result register_result = { 0 };
struct nbl_dev_ring_mgt *ring_mgt;
- const struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ const struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret = 0;
net_dev = rte_zmalloc("nbl_dev_net", sizeof(struct nbl_dev_net_mgt), 0);
diff --git a/drivers/net/netvsc/hn_ethdev.c b/drivers/net/netvsc/hn_ethdev.c
index fa936cfde7..cb7035e15f 100644
--- a/drivers/net/netvsc/hn_ethdev.c
+++ b/drivers/net/netvsc/hn_ethdev.c
@@ -1525,7 +1525,6 @@ static int
eth_hn_dev_init(struct rte_eth_dev *eth_dev)
{
struct hn_data *hv = eth_dev->data->dev_private;
- struct rte_device *device = eth_dev->device;
struct rte_vmbus_device *vmbus;
uint32_t mtu;
unsigned int rxr_cnt;
@@ -1536,7 +1535,7 @@ eth_hn_dev_init(struct rte_eth_dev *eth_dev)
rte_spinlock_init(&hv->hotadd_lock);
LIST_INIT(&hv->hotadd_list);
- vmbus = RTE_BUS_DEVICE(device, *vmbus);
+ vmbus = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *vmbus);
eth_dev->dev_ops = &hn_eth_dev_ops;
eth_dev->rx_queue_count = hn_dev_rx_queue_count;
eth_dev->rx_descriptor_status = hn_dev_rx_queue_status;
diff --git a/drivers/net/nfp/nfp_ethdev.c b/drivers/net/nfp/nfp_ethdev.c
index cbd1deffb4..d2da18013c 100644
--- a/drivers/net/nfp/nfp_ethdev.c
+++ b/drivers/net/nfp/nfp_ethdev.c
@@ -363,7 +363,7 @@ nfp_net_start(struct rte_eth_dev *dev)
struct rte_eth_txmode *txmode;
struct nfp_net_hw_priv *hw_priv;
struct nfp_app_fw_nic *app_fw_nic;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
net_hw = dev->data->dev_private;
@@ -770,7 +770,7 @@ nfp_net_close(struct rte_eth_dev *dev)
hw = dev->data->dev_private;
pf_dev = hw_priv->pf_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
app_fw_nic = NFP_PRIV_TO_APP_FW_NIC(pf_dev->app_fw_priv);
/*
@@ -1022,7 +1022,7 @@ nfp_net_init(struct rte_eth_dev *eth_dev,
struct nfp_net_hw_priv *hw_priv;
struct nfp_app_fw_nic *app_fw_nic;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
net_hw = eth_dev->data->dev_private;
hw_init = para;
@@ -2879,7 +2879,7 @@ nfp_pci_uninit(struct rte_eth_dev *eth_dev)
uint16_t port_id;
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* Free up all physical ports under PF */
RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device)
diff --git a/drivers/net/nfp/nfp_ethdev_vf.c b/drivers/net/nfp/nfp_ethdev_vf.c
index 23fa5b82ad..a86cc36592 100644
--- a/drivers/net/nfp/nfp_ethdev_vf.c
+++ b/drivers/net/nfp/nfp_ethdev_vf.c
@@ -30,7 +30,7 @@ nfp_netvf_start(struct rte_eth_dev *dev)
struct nfp_net_hw *net_hw;
struct rte_eth_conf *dev_conf;
struct rte_eth_rxmode *rxmode;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/* Disabling queues just in case... */
@@ -169,7 +169,7 @@ nfp_netvf_close(struct rte_eth_dev *dev)
return 0;
net_hw = dev->data->dev_private;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
hw_priv = dev->process_private;
rte_free(net_hw->eth_xstats_base);
@@ -266,7 +266,7 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev)
const struct nfp_dev_info *dev_info;
port = eth_dev->data->port_id;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
dev_info = nfp_dev_info_get(pci_dev->id.device_id);
if (dev_info == NULL) {
diff --git a/drivers/net/nfp/nfp_net_common.c b/drivers/net/nfp/nfp_net_common.c
index d35eee970a..2d36311cfe 100644
--- a/drivers/net/nfp/nfp_net_common.c
+++ b/drivers/net/nfp/nfp_net_common.c
@@ -1568,7 +1568,7 @@ nfp_rx_queue_intr_enable(struct rte_eth_dev *dev,
struct nfp_net_hw *hw;
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (rte_intr_type_get(pci_dev->intr_handle) != RTE_INTR_HANDLE_UIO)
base = 1;
@@ -1589,7 +1589,7 @@ nfp_rx_queue_intr_disable(struct rte_eth_dev *dev,
struct nfp_net_hw *hw;
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (rte_intr_type_get(pci_dev->intr_handle) != RTE_INTR_HANDLE_UIO)
base = 1;
@@ -1606,7 +1606,7 @@ static void
nfp_net_dev_link_status_print(struct rte_eth_dev *dev)
{
struct rte_eth_link link;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
rte_eth_linkstatus_get(dev, &link);
if (link.link_status != 0)
@@ -1635,7 +1635,7 @@ nfp_net_irq_unmask(struct rte_eth_dev *dev)
struct rte_pci_device *pci_dev;
hw = nfp_net_get_hw(dev);
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
/* Make sure all updates are written before un-masking */
rte_wmb();
diff --git a/drivers/net/ngbe/ngbe_ethdev.c b/drivers/net/ngbe/ngbe_ethdev.c
index 8b9d6371fb..f7b4a8b159 100644
--- a/drivers/net/ngbe/ngbe_ethdev.c
+++ b/drivers/net/ngbe/ngbe_ethdev.c
@@ -321,7 +321,7 @@ ngbe_swfw_lock_reset(struct ngbe_hw *hw)
static int
eth_ngbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct ngbe_hw *hw = ngbe_dev_hw(eth_dev);
struct ngbe_vfta *shadow_vfta = NGBE_DEV_VFTA(eth_dev);
struct ngbe_hwstrip *hwstrip = NGBE_DEV_HWSTRIP(eth_dev);
@@ -958,7 +958,7 @@ ngbe_dev_start(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
struct ngbe_hw_stats *hw_stats = NGBE_DEV_STATS(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
int err;
@@ -1160,7 +1160,7 @@ ngbe_dev_stop(struct rte_eth_dev *dev)
struct ngbe_adapter *adapter = ngbe_dev_adapter(dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
struct ngbe_vf_info *vfinfo = *NGBE_DEV_VFDATA(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int vf;
@@ -1256,7 +1256,7 @@ static int
ngbe_dev_close(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int retries = 0;
int ret;
@@ -1843,7 +1843,7 @@ ngbe_fw_version_get(struct rte_eth_dev *dev, char *fw_version, size_t fw_size)
static int
ngbe_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -2258,7 +2258,7 @@ ngbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
static void
ngbe_dev_link_status_print(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
rte_eth_linkstatus_get(dev, &link);
@@ -2472,7 +2472,7 @@ static s32
ngbe_fc_hpbthresh_set(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
u32 max_frame_size, tc, dv_id, rx_pb;
s32 kb, marker;
@@ -2653,7 +2653,7 @@ ngbe_remove_rar(struct rte_eth_dev *dev, uint32_t index)
static int
ngbe_set_default_mac_addr(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
ngbe_remove_rar(dev, 0);
ngbe_add_rar(dev, addr, 0, pci_dev->max_vfs);
@@ -2797,7 +2797,7 @@ ngbe_uc_all_hash_table_set(struct rte_eth_dev *dev, uint8_t on)
static int
ngbe_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_hw *hw = ngbe_dev_hw(dev);
uint32_t mask;
@@ -2867,7 +2867,7 @@ ngbe_set_ivar_map(struct ngbe_hw *hw, int8_t direction,
static void
ngbe_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_hw *hw = ngbe_dev_hw(dev);
uint32_t queue_id, base = NGBE_MISC_VEC_ID;
diff --git a/drivers/net/ngbe/ngbe_ethdev_vf.c b/drivers/net/ngbe/ngbe_ethdev_vf.c
index 6406df40d0..ea3a988df6 100644
--- a/drivers/net/ngbe/ngbe_ethdev_vf.c
+++ b/drivers/net/ngbe/ngbe_ethdev_vf.c
@@ -152,7 +152,7 @@ eth_ngbevf_dev_init(struct rte_eth_dev *eth_dev)
{
int err;
uint32_t tc, tcs;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_hw *hw = ngbe_dev_hw(eth_dev);
struct ngbe_vfta *shadow_vfta = NGBE_DEV_VFTA(eth_dev);
@@ -465,7 +465,7 @@ static int
ngbevf_dev_info_get(struct rte_eth_dev *dev,
struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -588,7 +588,7 @@ ngbevf_dev_start(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
uint32_t intr_vector = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int err, mask = 0;
@@ -688,7 +688,7 @@ ngbevf_dev_stop(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
struct ngbe_adapter *adapter = ngbe_dev_adapter(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (hw->adapter_stopped)
@@ -725,7 +725,7 @@ static int
ngbevf_dev_close(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
@@ -898,7 +898,7 @@ ngbevf_vlan_offload_set(struct rte_eth_dev *dev, int mask)
static int
ngbevf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_interrupt *intr = ngbe_dev_intr(dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
@@ -920,7 +920,7 @@ ngbevf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct ngbe_interrupt *intr = ngbe_dev_intr(dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = NGBE_MISC_VEC_ID;
@@ -960,7 +960,7 @@ ngbevf_set_ivar_map(struct ngbe_hw *hw, int8_t direction,
static void
ngbevf_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_hw *hw = ngbe_dev_hw(dev);
uint32_t q_idx;
diff --git a/drivers/net/ngbe/ngbe_pf.c b/drivers/net/ngbe/ngbe_pf.c
index bb62e2fbb7..db2384e28c 100644
--- a/drivers/net/ngbe/ngbe_pf.c
+++ b/drivers/net/ngbe/ngbe_pf.c
@@ -18,7 +18,7 @@
static inline uint16_t
dev_num_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* EM only support 7 VFs. */
return pci_dev->max_vfs;
diff --git a/drivers/net/octeon_ep/otx_ep_ethdev.c b/drivers/net/octeon_ep/otx_ep_ethdev.c
index 99be30523a..876d2f9d7d 100644
--- a/drivers/net/octeon_ep/otx_ep_ethdev.c
+++ b/drivers/net/octeon_ep/otx_ep_ethdev.c
@@ -777,7 +777,7 @@ static int otx_ep_eth_dev_query_set_vf_mac(struct rte_eth_dev *eth_dev,
static int
otx_ep_eth_dev_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
struct otx_ep_device *otx_epvf = OTX_EP_DEV(eth_dev);
struct rte_ether_addr vf_mac_addr;
int ret = 0;
diff --git a/drivers/net/octeon_ep/otx_ep_mbox.c b/drivers/net/octeon_ep/otx_ep_mbox.c
index 3e94c66677..5e6be29a96 100644
--- a/drivers/net/octeon_ep/otx_ep_mbox.c
+++ b/drivers/net/octeon_ep/otx_ep_mbox.c
@@ -346,7 +346,7 @@ otx_ep_mbox_intr_handler(void *param)
{
struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)param;
struct otx_ep_device *otx_ep = (struct otx_ep_device *)eth_dev->data->dev_private;
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
union otx_ep_mbox_word mbox_cmd;
if (otx2_read64(otx_ep->hw_addr + CNXK_EP_R_MBOX_PF_VF_INT(0)) & CNXK_EP_MBOX_INTR) {
@@ -369,7 +369,7 @@ int
otx_ep_mbox_init(struct rte_eth_dev *eth_dev)
{
struct otx_ep_device *otx_ep = (struct otx_ep_device *)eth_dev->data->dev_private;
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
uint64_t reg_val;
int rc;
@@ -402,7 +402,7 @@ void
otx_ep_mbox_uninit(struct rte_eth_dev *eth_dev)
{
struct otx_ep_device *otx_ep = (struct otx_ep_device *)eth_dev->data->dev_private;
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
otx2_write64(0, otx_ep->hw_addr + CNXK_EP_R_MBOX_PF_VF_INT(0));
diff --git a/drivers/net/qede/qede_ethdev.c b/drivers/net/qede/qede_ethdev.c
index c676c6fa75..4efc2dd349 100644
--- a/drivers/net/qede/qede_ethdev.c
+++ b/drivers/net/qede/qede_ethdev.c
@@ -1231,7 +1231,7 @@ static int qede_args_check(const char *key, const char *val, void *opaque)
static int qede_args(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_kvargs *kvlist;
struct rte_devargs *devargs;
int ret;
@@ -1540,7 +1540,7 @@ static void qede_poll_sp_sb_cb(void *param)
static int qede_dev_close(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct qede_dev *qdev = QEDE_INIT_QDEV(eth_dev);
struct ecore_dev *edev = QEDE_INIT_EDEV(qdev);
int ret = 0;
@@ -2529,7 +2529,7 @@ static int qede_common_dev_init(struct rte_eth_dev *eth_dev, bool is_vf)
adapter = eth_dev->data->dev_private;
adapter->ethdev = eth_dev;
edev = &adapter->edev;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
pci_addr = pci_dev->addr;
PMD_INIT_FUNC_TRACE(edev);
diff --git a/drivers/net/r8169/r8169_ethdev.c b/drivers/net/r8169/r8169_ethdev.c
index b2b1882aa5..2ac0189b61 100644
--- a/drivers/net/r8169/r8169_ethdev.c
+++ b/drivers/net/r8169/r8169_ethdev.c
@@ -301,7 +301,7 @@ rtl_dev_start(struct rte_eth_dev *dev)
{
struct rtl_adapter *adapter = RTL_DEV_PRIVATE(dev);
struct rtl_hw *hw = &adapter->hw;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int err;
@@ -684,7 +684,7 @@ rtl_dev_interrupt_handler(void *param)
static int
rtl_dev_close(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rtl_adapter *adapter = RTL_DEV_PRIVATE(dev);
struct rtl_hw *hw = &adapter->hw;
@@ -908,7 +908,7 @@ rtl_rss_hash_conf_get(struct rte_eth_dev *dev, struct rte_eth_rss_conf *rss_conf
static int
rtl_dev_init(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rtl_adapter *adapter = RTL_DEV_PRIVATE(dev);
struct rtl_hw *hw = &adapter->hw;
diff --git a/drivers/net/rnp/rnp_ethdev.c b/drivers/net/rnp/rnp_ethdev.c
index 3420842823..e48ad0e317 100644
--- a/drivers/net/rnp/rnp_ethdev.c
+++ b/drivers/net/rnp/rnp_ethdev.c
@@ -728,7 +728,7 @@ static int rnp_dev_close(struct rte_eth_dev *eth_dev)
if (adapter->intr_registered && adapter->eth_dev == eth_dev)
rnp_change_manage_port(adapter);
if (adapter->closed_ports == adapter->inited_ports) {
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
if (adapter->intr_registered) {
/* disable uio irq before callback unregister */
rte_intr_disable(pci_dev->intr_handle);
@@ -1667,7 +1667,7 @@ rnp_rx_reset_pool_setup(struct rnp_eth_adapter *adapter)
static int
rnp_eth_dev_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rnp_eth_port *port = RNP_DEV_TO_PORT(eth_dev);
char name[RTE_ETH_NAME_MAX_LEN] = " ";
@@ -1798,7 +1798,7 @@ rnp_eth_dev_init(struct rte_eth_dev *eth_dev)
static int
rnp_eth_dev_uninit(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
uint16_t port_id;
int err = 0;
diff --git a/drivers/net/sfc/sfc.c b/drivers/net/sfc/sfc.c
index 69747e49ae..39cd8d519a 100644
--- a/drivers/net/sfc/sfc.c
+++ b/drivers/net/sfc/sfc.c
@@ -781,7 +781,7 @@ static int
sfc_mem_bar_init(struct sfc_adapter *sa, const efx_bar_region_t *mem_ebrp)
{
struct rte_eth_dev *eth_dev = sa->eth_dev;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
efsys_bar_t *ebp = &sa->mem_bar;
struct rte_mem_resource *res =
&pci_dev->mem_resource[mem_ebrp->ebr_index];
@@ -1283,7 +1283,7 @@ sfc_probe(struct sfc_adapter *sa)
{
efx_bar_region_t mem_ebrp;
struct rte_eth_dev *eth_dev = sa->eth_dev;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
efx_nic_t *enp;
int rc;
diff --git a/drivers/net/sfc/sfc_ethdev.c b/drivers/net/sfc/sfc_ethdev.c
index 6be98c49d0..6be91789cf 100644
--- a/drivers/net/sfc/sfc_ethdev.c
+++ b/drivers/net/sfc/sfc_ethdev.c
@@ -3309,7 +3309,7 @@ static int
sfc_eth_dev_init(struct rte_eth_dev *dev, void *init_params)
{
struct sfc_adapter_shared *sas = sfc_adapter_shared_by_eth_dev(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct sfc_ethdev_init_data *init_data = init_params;
uint32_t logtype_main;
struct sfc_adapter *sa;
diff --git a/drivers/net/sfc/sfc_intr.c b/drivers/net/sfc/sfc_intr.c
index ddddefad7b..6a09da9f67 100644
--- a/drivers/net/sfc/sfc_intr.c
+++ b/drivers/net/sfc/sfc_intr.c
@@ -56,7 +56,7 @@ sfc_intr_line_handler(void *cb_arg)
boolean_t fatal;
uint32_t qmask;
unsigned int lsc_seq = sa->port.lsc_seq;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
sfc_log_init(sa, "entry");
@@ -102,7 +102,7 @@ sfc_intr_message_handler(void *cb_arg)
efx_nic_t *enp = sa->nic;
boolean_t fatal;
unsigned int lsc_seq = sa->port.lsc_seq;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
sfc_log_init(sa, "entry");
@@ -158,7 +158,7 @@ sfc_intr_start(struct sfc_adapter *sa)
if (rc != 0)
goto fail_intr_init;
- pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
intr_handle = pci_dev->intr_handle;
if (intr->handler != NULL) {
@@ -240,7 +240,7 @@ void
sfc_intr_stop(struct sfc_adapter *sa)
{
struct sfc_intr *intr = &sa->intr;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
sfc_log_init(sa, "entry");
@@ -318,7 +318,7 @@ int
sfc_intr_attach(struct sfc_adapter *sa)
{
struct sfc_intr *intr = &sa->intr;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
sfc_log_init(sa, "entry");
diff --git a/drivers/net/sfc/sfc_rx.c b/drivers/net/sfc/sfc_rx.c
index a193229265..305c680944 100644
--- a/drivers/net/sfc/sfc_rx.c
+++ b/drivers/net/sfc/sfc_rx.c
@@ -1277,8 +1277,9 @@ sfc_rx_qinit(struct sfc_adapter *sa, sfc_sw_index_t sw_index,
info.nic_dma_info = &sas->nic_dma_info;
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
rc = sa->priv.dp_rx->qcreate(sa->eth_dev->data->port_id, sw_index,
- &RTE_ETH_DEV_TO_PCI(sa->eth_dev)->addr,
+ &pci_dev->addr,
socket_id, &info, &rxq_info->dp);
if (rc != 0)
goto fail_dp_rx_qcreate;
diff --git a/drivers/net/sfc/sfc_sriov.c b/drivers/net/sfc/sfc_sriov.c
index 009b884d8d..f41d1b1719 100644
--- a/drivers/net/sfc/sfc_sriov.c
+++ b/drivers/net/sfc/sfc_sriov.c
@@ -44,7 +44,7 @@ sriov_mac_addr_assigned(const efx_vport_config_t *vport_config,
int
sfc_sriov_attach(struct sfc_adapter *sa)
{
- const struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ const struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
struct sfc_sriov *sriov = &sa->sriov;
efx_vport_config_t *vport_config;
unsigned int i;
diff --git a/drivers/net/sfc/sfc_tx.c b/drivers/net/sfc/sfc_tx.c
index ebc0a8235b..fac56cb27c 100644
--- a/drivers/net/sfc/sfc_tx.c
+++ b/drivers/net/sfc/sfc_tx.c
@@ -230,8 +230,9 @@ sfc_tx_qinit(struct sfc_adapter *sa, sfc_sw_index_t sw_index,
info.max_pdu = encp->enc_mac_pdu_max;
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
rc = sa->priv.dp_tx->qcreate(sa->eth_dev->data->port_id, sw_index,
- &RTE_ETH_DEV_TO_PCI(sa->eth_dev)->addr,
+ &pci_dev->addr,
socket_id, &info, &txq_info->dp);
if (rc != 0)
goto fail_dp_tx_qinit;
diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 76ed76a045..6e34da7c3c 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -1471,7 +1471,7 @@ static int
nicvf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
struct nicvf *nic = nicvf_pmd_priv(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
PMD_INIT_FUNC_TRACE();
@@ -2234,7 +2234,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
}
}
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
rte_eth_copy_pci_info(eth_dev, pci_dev);
eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
diff --git a/drivers/net/txgbe/txgbe_ethdev.c b/drivers/net/txgbe/txgbe_ethdev.c
index 5d360f8305..0f484dfe91 100644
--- a/drivers/net/txgbe/txgbe_ethdev.c
+++ b/drivers/net/txgbe/txgbe_ethdev.c
@@ -525,7 +525,7 @@ static void
txgbe_parse_devargs(struct rte_eth_dev *dev)
{
struct rte_eth_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_devargs *devargs = pci_dev->device.devargs;
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
struct rte_kvargs *kvlist;
@@ -601,7 +601,7 @@ static int
eth_txgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
{
struct txgbe_adapter *ad = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(eth_dev);
struct txgbe_vfta *shadow_vfta = TXGBE_DEV_VFTA(eth_dev);
struct txgbe_hwstrip *hwstrip = TXGBE_DEV_HWSTRIP(eth_dev);
@@ -1397,7 +1397,7 @@ txgbe_vmdq_vlan_hw_filter_enable(struct rte_eth_dev *dev)
static int
txgbe_check_vf_rss_rxq_num(struct rte_eth_dev *dev, uint16_t nb_rx_q)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
switch (nb_rx_q) {
case 1:
@@ -1664,7 +1664,7 @@ txgbe_set_vf_rate_limit(struct rte_eth_dev *dev, uint16_t vf,
struct rte_pci_device *pci_dev;
int ret;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
ret = rte_eth_link_get_nowait(dev->data->port_id, &link);
if (ret < 0)
return ret;
@@ -1736,7 +1736,7 @@ txgbe_dev_start(struct rte_eth_dev *dev)
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
struct txgbe_hw_stats *hw_stats = TXGBE_DEV_STATS(dev);
struct txgbe_vf_info *vfinfo = *TXGBE_DEV_VFDATA(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
int err;
@@ -2034,7 +2034,7 @@ txgbe_dev_stop(struct rte_eth_dev *dev)
struct txgbe_adapter *adapter = TXGBE_DEV_ADAPTER(dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
struct txgbe_vf_info *vfinfo = *TXGBE_DEV_VFDATA(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int vf;
struct txgbe_tm_conf *tm_conf = TXGBE_DEV_TM_CONF(dev);
@@ -2163,7 +2163,7 @@ static int
txgbe_dev_close(struct rte_eth_dev *dev)
{
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int retries = 0;
int ret;
@@ -2822,7 +2822,7 @@ txgbe_fw_version_get(struct rte_eth_dev *dev, char *fw_version, size_t fw_size)
static int
txgbe_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -3500,7 +3500,7 @@ txgbe_dev_interrupt_get_status(struct rte_eth_dev *dev,
static void
txgbe_dev_link_status_print(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
rte_eth_linkstatus_get(dev, &link);
@@ -3631,7 +3631,7 @@ static void
txgbe_dev_interrupt_delayed_handler(void *param)
{
struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_interrupt *intr = TXGBE_DEV_INTR(dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
@@ -3978,7 +3978,7 @@ txgbe_remove_rar(struct rte_eth_dev *dev, uint32_t index)
static int
txgbe_set_default_mac_addr(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
txgbe_remove_rar(dev, 0);
txgbe_add_rar(dev, addr, 0, pci_dev->max_vfs);
@@ -4149,7 +4149,7 @@ txgbe_convert_vm_rx_mask_to_val(uint16_t rx_mask, uint32_t orig_val)
static int
txgbe_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t mask;
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
@@ -4231,7 +4231,7 @@ txgbe_set_ivar_map(struct txgbe_hw *hw, int8_t direction,
static void
txgbe_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
uint32_t queue_id, base = TXGBE_MISC_VEC_ID;
diff --git a/drivers/net/txgbe/txgbe_ethdev_vf.c b/drivers/net/txgbe/txgbe_ethdev_vf.c
index 39a5fff65c..7a50c7a855 100644
--- a/drivers/net/txgbe/txgbe_ethdev_vf.c
+++ b/drivers/net/txgbe/txgbe_ethdev_vf.c
@@ -232,7 +232,7 @@ eth_txgbevf_dev_init(struct rte_eth_dev *eth_dev)
int err;
uint32_t tc, tcs;
struct txgbe_adapter *ad = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_hw *hw = TXGBE_DEV_HW(eth_dev);
struct txgbe_vfta *shadow_vfta = TXGBE_DEV_VFTA(eth_dev);
@@ -561,7 +561,7 @@ static int
txgbevf_dev_info_get(struct rte_eth_dev *dev,
struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -696,7 +696,7 @@ txgbevf_dev_start(struct rte_eth_dev *dev)
{
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
uint32_t intr_vector = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int err, mask = 0;
@@ -801,7 +801,7 @@ txgbevf_dev_stop(struct rte_eth_dev *dev)
{
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
struct txgbe_adapter *adapter = TXGBE_DEV_ADAPTER(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (hw->adapter_stopped)
@@ -841,7 +841,7 @@ static int
txgbevf_dev_close(struct rte_eth_dev *dev)
{
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
@@ -1023,7 +1023,7 @@ txgbevf_vlan_offload_set(struct rte_eth_dev *dev, int mask)
static int
txgbevf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_interrupt *intr = TXGBE_DEV_INTR(dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
@@ -1045,7 +1045,7 @@ txgbevf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct txgbe_interrupt *intr = TXGBE_DEV_INTR(dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = TXGBE_MISC_VEC_ID;
@@ -1085,7 +1085,7 @@ txgbevf_set_ivar_map(struct txgbe_hw *hw, int8_t direction,
static void
txgbevf_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
uint32_t q_idx;
diff --git a/drivers/net/txgbe/txgbe_flow.c b/drivers/net/txgbe/txgbe_flow.c
index a97588e57a..1bb0d3978c 100644
--- a/drivers/net/txgbe/txgbe_flow.c
+++ b/drivers/net/txgbe/txgbe_flow.c
@@ -1171,7 +1171,7 @@ cons_parse_l2_tn_filter(struct rte_eth_dev *dev,
const struct rte_flow_item_e_tag *e_tag_mask;
const struct rte_flow_action *act;
const struct rte_flow_action_vf *act_vf;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!pattern) {
rte_flow_error_set(error, EINVAL,
@@ -1328,7 +1328,7 @@ txgbe_parse_l2_tn_filter(struct rte_eth_dev *dev,
struct rte_flow_error *error)
{
int ret = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t vf_num;
if (!txgbe_is_pf(TXGBE_DEV_HW(dev))) {
diff --git a/drivers/net/txgbe/txgbe_pf.c b/drivers/net/txgbe/txgbe_pf.c
index 700632bd88..91f73521fe 100644
--- a/drivers/net/txgbe/txgbe_pf.c
+++ b/drivers/net/txgbe/txgbe_pf.c
@@ -33,7 +33,7 @@
static inline uint16_t
dev_num_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
return pci_dev->max_vfs;
}
diff --git a/drivers/net/txgbe/txgbe_tm.c b/drivers/net/txgbe/txgbe_tm.c
index b62bcf54aa..29c7c4adfb 100644
--- a/drivers/net/txgbe/txgbe_tm.c
+++ b/drivers/net/txgbe/txgbe_tm.c
@@ -354,7 +354,7 @@ txgbe_queue_base_nb_get(struct rte_eth_dev *dev, uint16_t tc_node_no,
uint16_t *base, uint16_t *nb)
{
uint8_t nb_tcs = txgbe_tc_nb_get(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t vf_num = pci_dev->max_vfs;
*base = 0;
diff --git a/drivers/net/virtio/virtio_pci_ethdev.c b/drivers/net/virtio/virtio_pci_ethdev.c
index fcda002297..d4f4bb0920 100644
--- a/drivers/net/virtio/virtio_pci_ethdev.c
+++ b/drivers/net/virtio/virtio_pci_ethdev.c
@@ -73,13 +73,13 @@ eth_virtio_pci_init(struct rte_eth_dev *eth_dev)
{
struct virtio_pci_dev *dev = eth_dev->data->dev_private;
struct virtio_hw *hw = &dev->hw;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret;
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
hw->port_id = eth_dev->data->port_id;
VTPCI_DEV(hw) = pci_dev;
- ret = vtpci_init(RTE_ETH_DEV_TO_PCI(eth_dev), dev);
+ ret = vtpci_init(pci_dev, dev);
if (ret) {
PMD_INIT_LOG(ERR, "Failed to init PCI device");
return -1;
@@ -91,7 +91,7 @@ eth_virtio_pci_init(struct rte_eth_dev *eth_dev)
else
VIRTIO_OPS(hw) = &virtio_legacy_ops;
- ret = virtio_remap_pci(RTE_ETH_DEV_TO_PCI(eth_dev), dev);
+ ret = virtio_remap_pci(pci_dev, dev);
if (ret < 0) {
PMD_INIT_LOG(ERR, "Failed to remap PCI device");
return -1;
@@ -111,7 +111,7 @@ eth_virtio_pci_init(struct rte_eth_dev *eth_dev)
return 0;
err_unmap:
- rte_pci_unmap_device(RTE_ETH_DEV_TO_PCI(eth_dev));
+ rte_pci_unmap_device(pci_dev);
if (!dev->modern)
vtpci_legacy_ioport_unmap(hw);
@@ -127,11 +127,12 @@ eth_virtio_pci_uninit(struct rte_eth_dev *eth_dev)
PMD_INIT_FUNC_TRACE();
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
dev = eth_dev->data->dev_private;
hw = &dev->hw;
if (dev->modern)
- rte_pci_unmap_device(RTE_ETH_DEV_TO_PCI(eth_dev));
+ rte_pci_unmap_device(pci_dev);
else
vtpci_legacy_ioport_unmap(hw);
return 0;
diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c b/drivers/net/vmxnet3/vmxnet3_ethdev.c
index da9af08207..b7cf217724 100644
--- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
+++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
@@ -308,7 +308,7 @@ eth_vmxnet3_setup_capabilities(struct vmxnet3_hw *hw,
struct rte_eth_dev *eth_dev)
{
uint32_t dcr, ptcr, value;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
VMXNET3_CMD_GET_MAX_CAPABILITIES);
@@ -381,7 +381,7 @@ eth_vmxnet3_dev_init(struct rte_eth_dev *eth_dev)
eth_dev->tx_pkt_burst = &vmxnet3_xmit_pkts;
eth_dev->tx_pkt_prepare = vmxnet3_prep_pkts;
eth_dev->rx_queue_count = vmxnet3_dev_rx_queue_count;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* extra mbuf field is required to guess MSS */
vmxnet3_segs_dynfield_offset =
diff --git a/drivers/net/xsc/xsc_ethdev.c b/drivers/net/xsc/xsc_ethdev.c
index 07fc52ac7b..39a67ff8cd 100644
--- a/drivers/net/xsc/xsc_ethdev.c
+++ b/drivers/net/xsc/xsc_ethdev.c
@@ -1048,7 +1048,7 @@ xsc_ethdev_init(struct rte_eth_dev *eth_dev)
PMD_INIT_FUNC_TRACE();
priv->eth_dev = eth_dev;
- priv->pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ priv->pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *priv->pci_dev);
ret = xsc_dev_init(priv->pci_dev, &priv->xdev);
if (ret) {
diff --git a/drivers/net/zxdh/zxdh_ethdev.c b/drivers/net/zxdh/zxdh_ethdev.c
index aeb01f4652..80ff19b3ea 100644
--- a/drivers/net/zxdh/zxdh_ethdev.c
+++ b/drivers/net/zxdh/zxdh_ethdev.c
@@ -111,7 +111,7 @@ zxdh_intr_unmask(struct rte_eth_dev *dev)
if (rte_intr_ack(dev->intr_handle) < 0)
return -1;
- hw->use_msix = zxdh_pci_msix_detect(RTE_ETH_DEV_TO_PCI(dev));
+ hw->use_msix = zxdh_pci_msix_detect(RTE_CLASS_TO_BUS_DEVICE(dev, struct rte_pci_device));
return 0;
}
@@ -1586,7 +1586,7 @@ static int32_t
zxdh_init_device(struct rte_eth_dev *eth_dev)
{
struct zxdh_hw *hw = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret = 0;
ret = zxdh_read_pci_caps(pci_dev, hw);
@@ -1820,7 +1820,7 @@ zxdh_get_dev_shared_data_idx(uint32_t dev_serial_id)
static int zxdh_init_dev_share_data(struct rte_eth_dev *eth_dev)
{
struct zxdh_hw *hw = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
uint32_t serial_id = (pci_dev->addr.domain << 16) |
(pci_dev->addr.bus << 8) | pci_dev->addr.devid;
uint16_t slot_id = 0;
@@ -2201,7 +2201,7 @@ is_inic_pf(uint16_t device_id)
static int
zxdh_eth_dev_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct zxdh_hw *hw = eth_dev->data->dev_private;
int ret = 0;
diff --git a/drivers/raw/ifpga/afu_pmd_n3000.c b/drivers/raw/ifpga/afu_pmd_n3000.c
index f092ee2dec..d5520a0d71 100644
--- a/drivers/raw/ifpga/afu_pmd_n3000.c
+++ b/drivers/raw/ifpga/afu_pmd_n3000.c
@@ -1467,11 +1467,11 @@ static struct rte_pci_device *n3000_afu_get_pci_dev(struct afu_rawdev *dev)
if (!dev || !dev->rawdev || !dev->rawdev->device)
return NULL;
- afudev = RTE_BUS_DEVICE(dev->rawdev->device, *afudev);
+ afudev = RTE_CLASS_TO_BUS_DEVICE(dev->rawdev, *afudev);
if (!afudev->rawdev || !afudev->rawdev->device)
return NULL;
- return RTE_BUS_DEVICE(afudev->rawdev->device, struct rte_pci_device);
+ return RTE_CLASS_TO_BUS_DEVICE(afudev->rawdev, struct rte_pci_device);
}
static int dma_afu_set_irqs(struct afu_rawdev *dev, uint32_t vec_start,
diff --git a/lib/eal/include/bus_driver.h b/lib/eal/include/bus_driver.h
index 3d04efbd3f..0a7e23d98d 100644
--- a/lib/eal/include/bus_driver.h
+++ b/lib/eal/include/bus_driver.h
@@ -461,6 +461,24 @@ void rte_bus_unregister(struct rte_bus *bus);
#define RTE_BUS_DRIVER(drv, bus_drv_type) \
container_of(drv, typeof(bus_drv_type), driver)
+/**
+ * Helper macro to convert a device class pointer to a bus-specific device type.
+ * Works with any device class (ethdev, cryptodev, eventdev, bbdev, etc.) that has
+ * a 'device' field pointing to struct rte_device.
+ *
+ * Example: RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev) or
+ * RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
+ *
+ * @param class_dev
+ * Pointer to device class structure (e.g., struct rte_eth_dev *)
+ * @param bus_dev_type
+ * Bus device type for type inference (e.g., *pci_dev or struct rte_pci_device)
+ * @return
+ * Pointer to the bus-specific device structure
+ */
+#define RTE_CLASS_TO_BUS_DEVICE(class_dev, bus_dev_type) \
+ RTE_BUS_DEVICE((class_dev)->device, bus_dev_type)
+
/**
* Helper macro to iterate over all devices on a bus.
*
--
2.53.0
^ permalink raw reply related
* [PATCH v3 24/25] eventdev: rename dev field to device
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Pavan Nikhilesh,
Shijith Thotton, Tirthendu Sarkar, Jerin Jacob
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
Rename the rte_eventdev structure field from 'dev' to 'device' to align
with the naming convention used by all other device classes in DPDK
(ethdev, cryptodev, bbdev, compressdev, rawdev, regexdev, dmadev, gpudev,
and mldev).
This change provides consistency across all device classes: each device
class structure now contains a 'struct rte_device *device' field
pointing to the backing device.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/event/cnxk/cn10k_eventdev.c | 8 ++++----
drivers/event/cnxk/cn20k_eventdev.c | 8 ++++----
drivers/event/cnxk/cn9k_eventdev.c | 6 +++---
drivers/event/cnxk/cnxk_eventdev.c | 2 +-
drivers/event/dlb2/pf/dlb2_pf.c | 2 +-
drivers/event/skeleton/skeleton_eventdev.c | 2 +-
lib/eventdev/eventdev_pmd.h | 2 +-
lib/eventdev/eventdev_pmd_pci.h | 4 ++--
lib/eventdev/eventdev_pmd_vdev.h | 2 +-
lib/eventdev/rte_eventdev.c | 14 +++++++-------
10 files changed, 25 insertions(+), 25 deletions(-)
diff --git a/drivers/event/cnxk/cn10k_eventdev.c b/drivers/event/cnxk/cn10k_eventdev.c
index 2e4b8aab92..8289fc44d6 100644
--- a/drivers/event/cnxk/cn10k_eventdev.c
+++ b/drivers/event/cnxk/cn10k_eventdev.c
@@ -921,7 +921,7 @@ static int
cn10k_crypto_adapter_caps_get(const struct rte_eventdev *event_dev,
const struct rte_cryptodev *cdev, uint32_t *caps)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn10k", ENOTSUP);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn10k", ENOTSUP);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn10k", ENOTSUP);
*caps = RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_OP_FWD |
@@ -939,7 +939,7 @@ cn10k_crypto_adapter_qp_add(const struct rte_eventdev *event_dev,
{
int ret;
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn10k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn10k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn10k", EINVAL);
cn10k_sso_fp_fns_set((struct rte_eventdev *)(uintptr_t)event_dev);
@@ -954,7 +954,7 @@ static int
cn10k_crypto_adapter_qp_del(const struct rte_eventdev *event_dev, const struct rte_cryptodev *cdev,
int32_t queue_pair_id)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn10k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn10k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn10k", EINVAL);
return cnxk_crypto_adapter_qp_del(cdev, queue_pair_id);
@@ -973,7 +973,7 @@ cn10k_crypto_adapter_vec_limits(const struct rte_eventdev *event_dev,
const struct rte_cryptodev *cdev,
struct rte_event_crypto_adapter_vector_limits *limits)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn10k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn10k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn10k", EINVAL);
limits->log2_sz = false;
diff --git a/drivers/event/cnxk/cn20k_eventdev.c b/drivers/event/cnxk/cn20k_eventdev.c
index ff3aaac16a..9d34168c32 100644
--- a/drivers/event/cnxk/cn20k_eventdev.c
+++ b/drivers/event/cnxk/cn20k_eventdev.c
@@ -1125,7 +1125,7 @@ static int
cn20k_crypto_adapter_caps_get(const struct rte_eventdev *event_dev,
const struct rte_cryptodev *cdev, uint32_t *caps)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn20k", ENOTSUP);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn20k", ENOTSUP);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn20k", ENOTSUP);
*caps = RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_OP_FWD |
@@ -1142,7 +1142,7 @@ cn20k_crypto_adapter_qp_add(const struct rte_eventdev *event_dev, const struct r
{
int ret;
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn20k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn20k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn20k", EINVAL);
cn20k_sso_fp_fns_set((struct rte_eventdev *)(uintptr_t)event_dev);
@@ -1157,7 +1157,7 @@ static int
cn20k_crypto_adapter_qp_del(const struct rte_eventdev *event_dev, const struct rte_cryptodev *cdev,
int32_t queue_pair_id)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn20k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn20k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn20k", EINVAL);
return cnxk_crypto_adapter_qp_del(cdev, queue_pair_id);
@@ -1175,7 +1175,7 @@ cn20k_crypto_adapter_vec_limits(const struct rte_eventdev *event_dev,
const struct rte_cryptodev *cdev,
struct rte_event_crypto_adapter_vector_limits *limits)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn20k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn20k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn20k", EINVAL);
limits->log2_sz = false;
diff --git a/drivers/event/cnxk/cn9k_eventdev.c b/drivers/event/cnxk/cn9k_eventdev.c
index 5f24366770..313dcbb384 100644
--- a/drivers/event/cnxk/cn9k_eventdev.c
+++ b/drivers/event/cnxk/cn9k_eventdev.c
@@ -1038,7 +1038,7 @@ static int
cn9k_crypto_adapter_caps_get(const struct rte_eventdev *event_dev, const struct rte_cryptodev *cdev,
uint32_t *caps)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn9k", ENOTSUP);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn9k", ENOTSUP);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn9k", ENOTSUP);
*caps = RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_OP_FWD |
@@ -1055,7 +1055,7 @@ cn9k_crypto_adapter_qp_add(const struct rte_eventdev *event_dev,
{
int ret;
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn9k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn9k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn9k", EINVAL);
cn9k_sso_fp_fns_set((struct rte_eventdev *)(uintptr_t)event_dev);
@@ -1070,7 +1070,7 @@ static int
cn9k_crypto_adapter_qp_del(const struct rte_eventdev *event_dev, const struct rte_cryptodev *cdev,
int32_t queue_pair_id)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn9k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn9k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn9k", EINVAL);
return cnxk_crypto_adapter_qp_del(cdev, queue_pair_id);
diff --git a/drivers/event/cnxk/cnxk_eventdev.c b/drivers/event/cnxk/cnxk_eventdev.c
index 8eff2ba8e0..6f000ff49e 100644
--- a/drivers/event/cnxk/cnxk_eventdev.c
+++ b/drivers/event/cnxk/cnxk_eventdev.c
@@ -654,7 +654,7 @@ cnxk_sso_init(struct rte_eventdev *event_dev)
return -ENOMEM;
}
- pci_dev = RTE_BUS_DEVICE(event_dev->dev, *pci_dev);
+ pci_dev = RTE_BUS_DEVICE(event_dev->device, *pci_dev);
dev->sso.pci_dev = pci_dev;
*(uint64_t *)mz->addr = (uint64_t)dev;
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a498ba8c41..82075bbf0b 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -784,7 +784,7 @@ dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
dlb2_pf_iface_fn_ptrs_init();
- pci_dev = RTE_BUS_DEVICE(eventdev->dev, *pci_dev);
+ pci_dev = RTE_BUS_DEVICE(eventdev->device, *pci_dev);
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
diff --git a/drivers/event/skeleton/skeleton_eventdev.c b/drivers/event/skeleton/skeleton_eventdev.c
index e07744d2f1..4292644fde 100644
--- a/drivers/event/skeleton/skeleton_eventdev.c
+++ b/drivers/event/skeleton/skeleton_eventdev.c
@@ -332,7 +332,7 @@ skeleton_eventdev_init(struct rte_eventdev *eventdev)
if (rte_eal_process_type() != RTE_PROC_PRIMARY)
return 0;
- pci_dev = RTE_BUS_DEVICE(eventdev->dev, *pci_dev);
+ pci_dev = RTE_BUS_DEVICE(eventdev->device, *pci_dev);
skel->reg_base = (uintptr_t)pci_dev->mem_resource[0].addr;
if (!skel->reg_base) {
diff --git a/lib/eventdev/eventdev_pmd.h b/lib/eventdev/eventdev_pmd.h
index d13cc433a7..9309dce5e1 100644
--- a/lib/eventdev/eventdev_pmd.h
+++ b/lib/eventdev/eventdev_pmd.h
@@ -156,7 +156,7 @@ struct __rte_cache_aligned rte_eventdev {
/**< Pointer to device data */
struct eventdev_ops *dev_ops;
/**< Functions exported by PMD */
- struct rte_device *dev;
+ struct rte_device *device;
/**< Device info. supplied by probing */
uint8_t attached : 1;
diff --git a/lib/eventdev/eventdev_pmd_pci.h b/lib/eventdev/eventdev_pmd_pci.h
index 5cb5916a84..ebc7d12b1d 100644
--- a/lib/eventdev/eventdev_pmd_pci.h
+++ b/lib/eventdev/eventdev_pmd_pci.h
@@ -68,7 +68,7 @@ rte_event_pmd_pci_probe_named(struct rte_pci_driver *pci_drv,
"device data");
}
- eventdev->dev = &pci_dev->device;
+ eventdev->device = &pci_dev->device;
/* Invoke PMD device initialization function */
retval = devinit(eventdev);
@@ -150,7 +150,7 @@ rte_event_pmd_pci_remove(struct rte_pci_device *pci_dev,
/* Free event device */
rte_event_pmd_release(eventdev);
- eventdev->dev = NULL;
+ eventdev->device = NULL;
return 0;
}
diff --git a/lib/eventdev/eventdev_pmd_vdev.h b/lib/eventdev/eventdev_pmd_vdev.h
index 4eaefa0b0b..ae1c950bed 100644
--- a/lib/eventdev/eventdev_pmd_vdev.h
+++ b/lib/eventdev/eventdev_pmd_vdev.h
@@ -67,7 +67,7 @@ rte_event_pmd_vdev_init(const char *name, size_t dev_private_size,
rte_panic("Cannot allocate memzone for private device"
" data");
}
- eventdev->dev = &vdev->device;
+ eventdev->device = &vdev->device;
return eventdev;
}
diff --git a/lib/eventdev/rte_eventdev.c b/lib/eventdev/rte_eventdev.c
index b921142d7b..572cd5bd7d 100644
--- a/lib/eventdev/rte_eventdev.c
+++ b/lib/eventdev/rte_eventdev.c
@@ -68,8 +68,8 @@ rte_event_dev_get_dev_id(const char *name)
for (i = 0; i < eventdev_globals.nb_devs; i++) {
cmp = (strncmp(rte_event_devices[i].data->name, name,
RTE_EVENTDEV_NAME_MAX_LEN) == 0) ||
- (rte_event_devices[i].dev ? (strncmp(
- rte_event_devices[i].dev->driver->name, name,
+ (rte_event_devices[i].device ? (strncmp(
+ rte_event_devices[i].device->driver->name, name,
RTE_EVENTDEV_NAME_MAX_LEN) == 0) : 0);
if (cmp && (rte_event_devices[i].attached ==
RTE_EVENTDEV_ATTACHED)) {
@@ -114,9 +114,9 @@ rte_event_dev_info_get(uint8_t dev_id, struct rte_event_dev_info *dev_info)
dev_info->dequeue_timeout_ns = dev->data->dev_conf.dequeue_timeout_ns;
- dev_info->dev = dev->dev;
- if (dev->dev != NULL && dev->dev->driver != NULL)
- dev_info->driver_name = dev->dev->driver->name;
+ dev_info->dev = dev->device;
+ if (dev->device != NULL && dev->device->driver != NULL)
+ dev_info->driver_name = dev->device->driver->name;
rte_eventdev_trace_info_get(dev_id, dev_info, dev_info->dev);
@@ -1812,8 +1812,8 @@ handle_dev_info(const char *cmd __rte_unused,
rte_tel_data_start_dict(d);
rte_tel_data_add_dict_int(d, "dev_id", dev_id);
- rte_tel_data_add_dict_string(d, "dev_name", dev->dev->name);
- rte_tel_data_add_dict_string(d, "dev_driver", dev->dev->driver->name);
+ rte_tel_data_add_dict_string(d, "dev_name", dev->device->name);
+ rte_tel_data_add_dict_string(d, "dev_driver", dev->device->driver->name);
rte_tel_data_add_dict_string(d, "state",
dev->data->dev_started ? "started" : "stopped");
rte_tel_data_add_dict_int(d, "socket_id", dev->data->socket_id);
--
2.53.0
^ permalink raw reply related
* [PATCH v3 23/25] drivers/bus: remove specific bus types
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Chenbo Xia, Tomasz Duszynski, Chengwen Feng, Long Li, Wei Hu
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
All the buses can have a simple rte_bus object.
Mark as static whenever possible.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/auxiliary/auxiliary_common.c | 44 ++++++++-------
drivers/bus/auxiliary/linux/auxiliary.c | 10 ++--
drivers/bus/auxiliary/private.h | 9 +---
drivers/bus/cdx/bus_cdx_driver.h | 1 -
drivers/bus/cdx/cdx.c | 42 +++++++--------
drivers/bus/cdx/private.h | 7 ---
drivers/bus/dpaa/dpaa_bus.c | 58 +++++++++-----------
drivers/bus/fslmc/fslmc_bus.c | 62 +++++++++++-----------
drivers/bus/fslmc/fslmc_vfio.c | 30 +++++------
drivers/bus/fslmc/portal/dpaa2_hw_dprc.c | 2 +-
drivers/bus/fslmc/private.h | 9 +---
drivers/bus/pci/bsd/pci.c | 16 +++---
drivers/bus/pci/linux/pci.c | 13 +++--
drivers/bus/pci/pci_common.c | 60 ++++++++++-----------
drivers/bus/pci/pci_params.c | 2 +-
drivers/bus/pci/private.h | 9 +---
drivers/bus/pci/windows/pci.c | 13 +++--
drivers/bus/platform/bus_platform_driver.h | 1 -
drivers/bus/platform/platform.c | 52 +++++++++---------
drivers/bus/platform/private.h | 9 ----
drivers/bus/uacce/uacce.c | 49 +++++++----------
drivers/bus/vmbus/linux/vmbus_bus.c | 10 ++--
drivers/bus/vmbus/private.h | 9 +---
drivers/bus/vmbus/vmbus_common.c | 34 ++++++------
24 files changed, 236 insertions(+), 315 deletions(-)
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index ff05716539..048aacf254 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -56,7 +56,7 @@ auxiliary_scan(void)
void
auxiliary_on_scan(struct rte_auxiliary_device *aux_dev)
{
- aux_dev->device.devargs = rte_bus_find_devargs(&auxiliary_bus.bus, aux_dev->name);
+ aux_dev->device.devargs = rte_bus_find_devargs(&auxiliary_bus, aux_dev->name);
}
static bool
@@ -156,7 +156,7 @@ auxiliary_parse(const char *name, void *addr)
if (strlen(name) == 0)
return 0;
- RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus) {
if (drv->match(name))
break;
}
@@ -170,7 +170,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_auxiliary_register)
void
rte_auxiliary_register(struct rte_auxiliary_driver *driver)
{
- rte_bus_add_driver(&auxiliary_bus.bus, &driver->driver);
+ rte_bus_add_driver(&auxiliary_bus, &driver->driver);
}
/* Unregister a driver */
@@ -178,7 +178,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_auxiliary_unregister)
void
rte_auxiliary_unregister(struct rte_auxiliary_driver *driver)
{
- rte_bus_remove_driver(&auxiliary_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&auxiliary_bus, &driver->driver);
}
static int
@@ -189,7 +189,7 @@ auxiliary_unplug(struct rte_device *dev)
ret = rte_auxiliary_driver_remove_dev(adev);
if (ret == 0) {
- rte_bus_remove_device(&auxiliary_bus.bus, &adev->device);
+ rte_bus_remove_device(&auxiliary_bus, &adev->device);
rte_devargs_remove(dev->devargs);
rte_intr_instance_free(adev->intr_handle);
free(adev);
@@ -203,7 +203,7 @@ auxiliary_cleanup(void)
struct rte_auxiliary_device *dev;
int error = 0;
- RTE_BUS_FOREACH_DEV(dev, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &auxiliary_bus) {
int ret;
if (!rte_dev_is_probed(&dev->device))
@@ -250,7 +250,7 @@ auxiliary_get_iommu_class(void)
{
const struct rte_auxiliary_driver *drv;
- RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus) {
if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0)
return RTE_IOVA_VA;
}
@@ -258,22 +258,20 @@ auxiliary_get_iommu_class(void)
return RTE_IOVA_DC;
}
-struct rte_auxiliary_bus auxiliary_bus = {
- .bus = {
- .scan = auxiliary_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = auxiliary_cleanup,
- .find_device = rte_bus_generic_find_device,
- .match = auxiliary_bus_match,
- .probe_device = auxiliary_probe_device,
- .unplug = auxiliary_unplug,
- .parse = auxiliary_parse,
- .dma_map = auxiliary_dma_map,
- .dma_unmap = auxiliary_dma_unmap,
- .get_iommu_class = auxiliary_get_iommu_class,
- .dev_iterate = rte_bus_generic_dev_iterate,
- },
+struct rte_bus auxiliary_bus = {
+ .scan = auxiliary_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = auxiliary_cleanup,
+ .find_device = rte_bus_generic_find_device,
+ .match = auxiliary_bus_match,
+ .probe_device = auxiliary_probe_device,
+ .unplug = auxiliary_unplug,
+ .parse = auxiliary_parse,
+ .dma_map = auxiliary_dma_map,
+ .dma_unmap = auxiliary_dma_unmap,
+ .get_iommu_class = auxiliary_get_iommu_class,
+ .dev_iterate = rte_bus_generic_dev_iterate,
};
-RTE_REGISTER_BUS(auxiliary, auxiliary_bus.bus);
+RTE_REGISTER_BUS(auxiliary, auxiliary_bus);
RTE_LOG_REGISTER_DEFAULT(auxiliary_bus_logtype, NOTICE);
diff --git a/drivers/bus/auxiliary/linux/auxiliary.c b/drivers/bus/auxiliary/linux/auxiliary.c
index b40de65bdd..3a2dca2865 100644
--- a/drivers/bus/auxiliary/linux/auxiliary.c
+++ b/drivers/bus/auxiliary/linux/auxiliary.c
@@ -48,12 +48,12 @@ auxiliary_scan_one(const char *dirname, const char *name)
auxiliary_on_scan(dev);
/* Device is valid, add in list (sorted) */
- RTE_BUS_FOREACH_DEV(dev2, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &auxiliary_bus) {
ret = strcmp(dev->name, dev2->name);
if (ret > 0)
continue;
if (ret < 0) {
- rte_bus_insert_device(&auxiliary_bus.bus, &dev2->device, &dev->device);
+ rte_bus_insert_device(&auxiliary_bus, &dev2->device, &dev->device);
} else { /* already registered */
if (rte_dev_is_probed(&dev2->device) &&
dev2->device.devargs != dev->device.devargs) {
@@ -65,7 +65,7 @@ auxiliary_scan_one(const char *dirname, const char *name)
}
return 0;
}
- rte_bus_add_device(&auxiliary_bus.bus, &dev->device);
+ rte_bus_add_device(&auxiliary_bus, &dev->device);
return 0;
}
@@ -109,14 +109,14 @@ auxiliary_scan(void)
if (e->d_name[0] == '.')
continue;
- if (rte_bus_device_is_ignored(&auxiliary_bus.bus, e->d_name))
+ if (rte_bus_device_is_ignored(&auxiliary_bus, e->d_name))
continue;
snprintf(dirname, sizeof(dirname), "%s/%s",
AUXILIARY_SYSFS_PATH, e->d_name);
/* Ignore if no driver can handle. */
- RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus) {
if (drv->match(e->d_name))
break;
}
diff --git a/drivers/bus/auxiliary/private.h b/drivers/bus/auxiliary/private.h
index 116154eb56..282ad94618 100644
--- a/drivers/bus/auxiliary/private.h
+++ b/drivers/bus/auxiliary/private.h
@@ -19,14 +19,7 @@ extern int auxiliary_bus_logtype;
#define AUXILIARY_LOG(level, ...) \
RTE_LOG_LINE(level, AUXILIARY_BUS, __VA_ARGS__)
-/*
- * Structure describing the auxiliary bus
- */
-struct rte_auxiliary_bus {
- struct rte_bus bus; /* Inherit the generic class */
-};
-
-extern struct rte_auxiliary_bus auxiliary_bus;
+extern struct rte_bus auxiliary_bus;
/*
* Test whether the auxiliary device exist.
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index aa44dee5a6..d443178404 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -25,7 +25,6 @@ extern "C" {
/* Forward declarations */
struct rte_cdx_device;
struct rte_cdx_driver;
-struct rte_cdx_bus;
#define RTE_CDX_BUS_DEVICES_PATH "/sys/bus/cdx/devices"
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index cb4f755b8e..2443161e1a 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -84,7 +84,7 @@
#define CDX_DEV_PREFIX "cdx-"
-struct rte_cdx_bus rte_cdx_bus;
+static struct rte_bus rte_cdx_bus;
static int
cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
@@ -194,7 +194,7 @@ cdx_scan_one(const char *dirname, const char *dev_name)
}
dev->id.device_id = (uint16_t)tmp;
- rte_bus_add_device(&rte_cdx_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_cdx_bus, &dev->device);
return 0;
@@ -226,7 +226,7 @@ cdx_scan(void)
if (e->d_name[0] == '.')
continue;
- if (rte_bus_device_is_ignored(&rte_cdx_bus.bus, e->d_name))
+ if (rte_bus_device_is_ignored(&rte_cdx_bus, e->d_name))
continue;
snprintf(dirname, sizeof(dirname), "%s/%s",
@@ -363,7 +363,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_cdx_register)
void
rte_cdx_register(struct rte_cdx_driver *driver)
{
- rte_bus_add_driver(&rte_cdx_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_cdx_bus, &driver->driver);
}
/* unregister a driver */
@@ -371,7 +371,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_cdx_unregister)
void
rte_cdx_unregister(struct rte_cdx_driver *driver)
{
- rte_bus_remove_driver(&rte_cdx_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_cdx_bus, &driver->driver);
}
/*
@@ -412,7 +412,7 @@ cdx_unplug(struct rte_device *dev)
ret = cdx_detach_dev(cdx_dev);
if (ret == 0) {
- rte_bus_remove_device(&rte_cdx_bus.bus, &cdx_dev->device);
+ rte_bus_remove_device(&rte_cdx_bus, &cdx_dev->device);
rte_devargs_remove(dev->devargs);
free(cdx_dev);
}
@@ -440,27 +440,25 @@ cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
static enum rte_iova_mode
cdx_get_iommu_class(void)
{
- if (TAILQ_EMPTY(&rte_cdx_bus.bus.device_list))
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
return RTE_IOVA_DC;
return RTE_IOVA_VA;
}
-struct rte_cdx_bus rte_cdx_bus = {
- .bus = {
- .scan = cdx_scan,
- .probe = rte_bus_generic_probe,
- .find_device = rte_bus_generic_find_device,
- .match = cdx_bus_match,
- .probe_device = cdx_probe_device,
- .unplug = cdx_unplug,
- .parse = cdx_parse,
- .dma_map = cdx_dma_map,
- .dma_unmap = cdx_dma_unmap,
- .get_iommu_class = cdx_get_iommu_class,
- .dev_iterate = rte_bus_generic_dev_iterate,
- },
+static struct rte_bus rte_cdx_bus = {
+ .scan = cdx_scan,
+ .probe = rte_bus_generic_probe,
+ .find_device = rte_bus_generic_find_device,
+ .match = cdx_bus_match,
+ .probe_device = cdx_probe_device,
+ .unplug = cdx_unplug,
+ .parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = rte_bus_generic_dev_iterate,
};
-RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_REGISTER_BUS(cdx, rte_cdx_bus);
RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
index f69673aaab..289301bade 100644
--- a/drivers/bus/cdx/private.h
+++ b/drivers/bus/cdx/private.h
@@ -7,13 +7,6 @@
#include "bus_cdx_driver.h"
-/**
- * Structure describing the CDX bus.
- */
-struct rte_cdx_bus {
- struct rte_bus bus; /**< Inherit the generic class */
-};
-
/**
* Map a particular resource from a file.
*
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index 08c1607647..ee467b94d5 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -57,10 +57,6 @@
#define DPAA_MAX_PUSH_MODE_QUEUE 8
#define DPAA_DEFAULT_PUSH_MODE_QUEUE 4
-struct rte_dpaa_bus {
- struct rte_bus bus;
-};
-
struct rte_dpaa_bus_private {
int device_count;
int detected;
@@ -69,7 +65,7 @@ struct rte_dpaa_bus_private {
RTE_ATOMIC(uint16_t) push_rxq_num;
};
-static struct rte_dpaa_bus rte_dpaa_bus;
+static struct rte_bus rte_dpaa_bus;
static struct rte_dpaa_bus_private dpaa_bus;
static struct netcfg_info *dpaa_netcfg;
@@ -168,17 +164,17 @@ dpaa_add_to_device_list(struct rte_dpaa_device *newdev)
int comp, inserted = 0;
struct rte_dpaa_device *dev = NULL;
- RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus) {
comp = compare_dpaa_devices(newdev, dev);
if (comp < 0) {
- rte_bus_insert_device(&rte_dpaa_bus.bus, &dev->device, &newdev->device);
+ rte_bus_insert_device(&rte_dpaa_bus, &dev->device, &newdev->device);
inserted = 1;
break;
}
}
if (!inserted)
- rte_bus_add_device(&rte_dpaa_bus.bus, &newdev->device);
+ rte_bus_add_device(&rte_dpaa_bus, &newdev->device);
}
/*
@@ -253,7 +249,7 @@ dpaa_create_device_list(void)
(fman_intf->fman->idx + 1), fman_intf->mac_idx);
}
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus.bus, dev->name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus, dev->name);
if (dev->device.devargs != NULL)
DPAA_BUS_INFO("**Devargs matched %s", dev->name);
@@ -303,7 +299,7 @@ dpaa_create_device_list(void)
sprintf(dev->name, "dpaa_sec-%d", i+1);
DPAA_BUS_LOG(INFO, "%s cryptodev added", dev->name);
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus.bus, dev->name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus, dev->name);
if (dev->device.devargs != NULL)
DPAA_BUS_INFO("**Devargs matched %s", dev->name);
@@ -329,7 +325,7 @@ dpaa_create_device_list(void)
sprintf(dev->name, "dpaa_qdma-%d", i+1);
DPAA_BUS_LOG(INFO, "%s qdma device added", dev->name);
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus.bus, dev->name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus, dev->name);
if (dev->device.devargs != NULL)
DPAA_BUS_INFO("**Devargs matched %s", dev->name);
@@ -349,8 +345,8 @@ dpaa_clean_device_list(void)
{
struct rte_dpaa_device *dev = NULL;
- RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
- rte_bus_remove_device(&rte_dpaa_bus.bus, &dev->device);
+ RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus) {
+ rte_bus_remove_device(&rte_dpaa_bus, &dev->device);
rte_intr_instance_free(dev->intr_handle);
free(dev);
dev = NULL;
@@ -583,7 +579,7 @@ rte_dpaa_driver_register(struct rte_dpaa_driver *driver)
BUS_INIT_FUNC_TRACE();
- rte_bus_add_driver(&rte_dpaa_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_dpaa_bus, &driver->driver);
}
/* un-register a dpaa bus based dpaa driver */
@@ -593,7 +589,7 @@ rte_dpaa_driver_unregister(struct rte_dpaa_driver *driver)
{
BUS_INIT_FUNC_TRACE();
- rte_bus_remove_driver(&rte_dpaa_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_dpaa_bus, &driver->driver);
}
static bool
@@ -760,7 +756,7 @@ rte_dpaa_bus_scan(void)
process_once = 1;
/* If no device present on DPAA bus nothing needs to be done */
- if (TAILQ_EMPTY(&rte_dpaa_bus.bus.device_list))
+ if (TAILQ_EMPTY(&rte_dpaa_bus.device_list))
return 0;
/* Register DPAA mempool ops only if any DPAA device has
@@ -768,7 +764,7 @@ rte_dpaa_bus_scan(void)
*/
rte_mbuf_set_platform_mempool_ops(DPAA_MEMPOOL_OPS_NAME);
- RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus) {
if (dev->device_type == FSL_DPAA_ETH) {
ret = rte_dpaa_setup_intr(dev->intr_handle);
if (ret)
@@ -816,7 +812,7 @@ dpaa_bus_cleanup(void)
struct rte_dpaa_device *dev;
BUS_INIT_FUNC_TRACE();
- RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus) {
const struct rte_dpaa_driver *drv;
int ret = 0;
@@ -859,19 +855,17 @@ RTE_FINI_PRIO(dpaa_cleanup, 102)
DPAA_BUS_DEBUG("Worker thread clean up done");
}
-static struct rte_dpaa_bus rte_dpaa_bus = {
- .bus = {
- .scan = rte_dpaa_bus_scan,
- .probe = rte_bus_generic_probe,
- .parse = rte_dpaa_bus_parse,
- .dev_compare = dpaa_bus_dev_compare,
- .find_device = rte_bus_generic_find_device,
- .get_iommu_class = rte_dpaa_get_iommu_class,
- .match = dpaa_bus_match,
- .probe_device = dpaa_bus_probe_device,
- .dev_iterate = rte_bus_generic_dev_iterate,
- .cleanup = dpaa_bus_cleanup,
- },
+static struct rte_bus rte_dpaa_bus = {
+ .scan = rte_dpaa_bus_scan,
+ .probe = rte_bus_generic_probe,
+ .parse = rte_dpaa_bus_parse,
+ .dev_compare = dpaa_bus_dev_compare,
+ .find_device = rte_bus_generic_find_device,
+ .get_iommu_class = rte_dpaa_get_iommu_class,
+ .match = dpaa_bus_match,
+ .probe_device = dpaa_bus_probe_device,
+ .dev_iterate = rte_bus_generic_dev_iterate,
+ .cleanup = dpaa_bus_cleanup,
};
static struct rte_dpaa_bus_private dpaa_bus = {
@@ -879,5 +873,5 @@ static struct rte_dpaa_bus_private dpaa_bus = {
.device_count = 0,
};
-RTE_REGISTER_BUS(dpaa_bus, rte_dpaa_bus.bus);
+RTE_REGISTER_BUS(dpaa_bus, rte_dpaa_bus);
RTE_LOG_REGISTER_DEFAULT(dpaa_logtype_bus, NOTICE);
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 821fe63955..88511d4dc7 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -27,7 +27,7 @@
#define VFIO_IOMMU_GROUP_PATH "/sys/kernel/iommu_groups"
-struct rte_fslmc_bus rte_fslmc_bus;
+struct rte_bus rte_fslmc_bus;
static int fslmc_bus_device_count[DPAA2_DEVTYPE_MAX];
#define DPAA2_SEQN_DYNFIELD_NAME "dpaa2_seqn_dynfield"
@@ -48,8 +48,8 @@ cleanup_fslmc_device_list(void)
{
struct rte_dpaa2_device *dev;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
rte_intr_instance_free(dev->intr_handle);
free(dev);
dev = NULL;
@@ -85,17 +85,17 @@ insert_in_device_list(struct rte_dpaa2_device *newdev)
int comp, inserted = 0;
struct rte_dpaa2_device *dev = NULL;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
comp = compare_dpaa2_devname(newdev, dev);
if (comp < 0) {
- rte_bus_insert_device(&rte_fslmc_bus.bus, &dev->device, &newdev->device);
+ rte_bus_insert_device(&rte_fslmc_bus, &dev->device, &newdev->device);
inserted = 1;
break;
}
}
if (!inserted)
- rte_bus_add_device(&rte_fslmc_bus.bus, &newdev->device);
+ rte_bus_add_device(&rte_fslmc_bus, &newdev->device);
}
static void
@@ -106,7 +106,7 @@ dump_device_list(void)
/* Only if the log level has been set to Debugging, print list */
if (rte_log_can_log(dpaa2_logtype_bus, RTE_LOG_DEBUG)) {
DPAA2_BUS_LOG(DEBUG, "List of devices scanned on bus:");
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
DPAA2_BUS_LOG(DEBUG, "\t\t%s", dev->device.name);
}
}
@@ -198,7 +198,7 @@ scan_one_fslmc_device(char *dev_name)
ret = -ENOMEM;
goto cleanup;
}
- dev->device.devargs = rte_bus_find_devargs(&rte_fslmc_bus.bus, dev_name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_fslmc_bus, dev_name);
/* Update the device found into the device_count table */
fslmc_bus_device_count[dev->dev_type]++;
@@ -247,7 +247,7 @@ rte_fslmc_parse(const char *name, void *addr)
*/
if (sep_exists) {
/* If either of "fslmc" or "name" are starting part */
- if (!strncmp(name, rte_fslmc_bus.bus.name, strlen(rte_fslmc_bus.bus.name)) ||
+ if (!strncmp(name, rte_fslmc_bus.name, strlen(rte_fslmc_bus.name)) ||
(!strncmp(name, "name", strlen("name")))) {
goto jump_out;
} else {
@@ -317,8 +317,8 @@ rte_fslmc_scan(void)
struct rte_dpaa2_device *dev;
DPAA2_BUS_DEBUG("Fslmc bus already scanned. Not rescanning");
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
- dev->device.devargs = rte_bus_find_devargs(&rte_fslmc_bus.bus,
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
+ dev->device.devargs = rte_bus_find_devargs(&rte_fslmc_bus,
dev->device.name);
}
return 0;
@@ -369,7 +369,7 @@ rte_fslmc_scan(void)
dump_device_list();
/* Bus initialization - only if devices were found */
- if (!TAILQ_EMPTY(&rte_fslmc_bus.bus.device_list)) {
+ if (!TAILQ_EMPTY(&rte_fslmc_bus.device_list)) {
static const struct rte_mbuf_dynfield dpaa2_seqn_dynfield_desc = {
.name = DPAA2_SEQN_DYNFIELD_NAME,
.size = sizeof(dpaa2_seqn_t),
@@ -455,7 +455,7 @@ rte_fslmc_driver_register(struct rte_dpaa2_driver *driver)
RTE_VERIFY(driver);
RTE_VERIFY(driver->probe != NULL);
- rte_bus_add_driver(&rte_fslmc_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_fslmc_bus, &driver->driver);
}
/*un-register a fslmc bus based dpaa2 driver */
@@ -463,7 +463,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_fslmc_driver_unregister)
void
rte_fslmc_driver_unregister(struct rte_dpaa2_driver *driver)
{
- rte_bus_remove_driver(&rte_fslmc_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_fslmc_bus, &driver->driver);
}
/*
@@ -475,8 +475,8 @@ fslmc_all_device_support_iova(void)
struct rte_dpaa2_device *dev;
struct rte_dpaa2_driver *drv;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
- RTE_BUS_FOREACH_DRV(drv, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
+ RTE_BUS_FOREACH_DRV(drv, &rte_fslmc_bus) {
if (!fslmc_bus_match(&drv->driver, &dev->device))
continue;
/* if the driver is not supporting IOVA */
@@ -496,7 +496,7 @@ rte_dpaa2_get_iommu_class(void)
if (rte_eal_iova_mode() == RTE_IOVA_PA)
return RTE_IOVA_PA;
- if (TAILQ_EMPTY(&rte_fslmc_bus.bus.device_list))
+ if (TAILQ_EMPTY(&rte_fslmc_bus.device_list))
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
@@ -546,21 +546,19 @@ fslmc_bus_unplug(struct rte_device *rte_dev)
return -ENODEV;
}
-struct rte_fslmc_bus rte_fslmc_bus = {
- .bus = {
- .scan = rte_fslmc_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = rte_fslmc_close,
- .parse = rte_fslmc_parse,
- .dev_compare = fslmc_dev_compare,
- .find_device = rte_bus_generic_find_device,
- .get_iommu_class = rte_dpaa2_get_iommu_class,
- .match = fslmc_bus_match,
- .probe_device = fslmc_bus_probe_device,
- .unplug = fslmc_bus_unplug,
- .dev_iterate = rte_bus_generic_dev_iterate,
- },
+struct rte_bus rte_fslmc_bus = {
+ .scan = rte_fslmc_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = rte_fslmc_close,
+ .parse = rte_fslmc_parse,
+ .dev_compare = fslmc_dev_compare,
+ .find_device = rte_bus_generic_find_device,
+ .get_iommu_class = rte_dpaa2_get_iommu_class,
+ .match = fslmc_bus_match,
+ .probe_device = fslmc_bus_probe_device,
+ .unplug = fslmc_bus_unplug,
+ .dev_iterate = rte_bus_generic_dev_iterate,
};
-RTE_REGISTER_BUS(fslmc, rte_fslmc_bus.bus);
+RTE_REGISTER_BUS(fslmc, rte_fslmc_bus);
RTE_LOG_REGISTER_DEFAULT(dpaa2_logtype_bus, NOTICE);
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 5784adaf90..412b70e5ae 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -1554,12 +1554,12 @@ fslmc_vfio_close_group(void)
return -EIO;
}
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->device.devargs &&
dev->device.devargs->policy == RTE_DEV_BLOCKED) {
DPAA2_BUS_LOG(DEBUG, "%s Blacklisted, skipping",
dev->device.name);
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
continue;
}
switch (dev->dev_type) {
@@ -1599,7 +1599,7 @@ fslmc_vfio_process_group(void)
bool is_dpmcp_in_blocklist = false, is_dpio_in_blocklist = false;
int dpmcp_count = 0, dpio_count = 0, current_device;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->dev_type == DPAA2_MPORTAL) {
dpmcp_count++;
if (dev->device.devargs &&
@@ -1616,14 +1616,14 @@ fslmc_vfio_process_group(void)
/* Search the MCP as that should be initialized first. */
current_device = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->dev_type == DPAA2_MPORTAL) {
current_device++;
if (dev->device.devargs &&
dev->device.devargs->policy == RTE_DEV_BLOCKED) {
DPAA2_BUS_LOG(DEBUG, "%s Blocked, skipping",
dev->device.name);
- rte_bus_remove_device(&rte_fslmc_bus.bus,
+ rte_bus_remove_device(&rte_fslmc_bus,
&dev->device);
continue;
}
@@ -1632,7 +1632,7 @@ fslmc_vfio_process_group(void)
!is_dpmcp_in_blocklist) {
if (dpmcp_count == 1 ||
current_device != dpmcp_count) {
- rte_bus_remove_device(&rte_fslmc_bus.bus,
+ rte_bus_remove_device(&rte_fslmc_bus,
&dev->device);
continue;
}
@@ -1647,7 +1647,7 @@ fslmc_vfio_process_group(void)
found_mportal = 1;
}
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
free(dev);
dev = NULL;
/* Ideally there is only a single dpmcp, but in case
@@ -1666,26 +1666,26 @@ fslmc_vfio_process_group(void)
* other devices.
*/
current_device = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->dev_type == DPAA2_DPRC) {
ret = fslmc_process_iodevices(dev);
if (ret) {
DPAA2_BUS_ERR("Unable to process dprc");
return ret;
}
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
}
}
current_device = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->dev_type == DPAA2_IO)
current_device++;
if (dev->device.devargs &&
dev->device.devargs->policy == RTE_DEV_BLOCKED) {
DPAA2_BUS_LOG(DEBUG, "%s Blocked, skipping",
dev->device.name);
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
continue;
}
if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
@@ -1693,7 +1693,7 @@ fslmc_vfio_process_group(void)
dev->dev_type != DPAA2_CRYPTO &&
dev->dev_type != DPAA2_QDMA &&
dev->dev_type != DPAA2_IO) {
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
continue;
}
switch (dev->dev_type) {
@@ -1735,13 +1735,13 @@ fslmc_vfio_process_group(void)
if (!is_dpio_in_blocklist && dpio_count > 1) {
if (rte_eal_process_type() == RTE_PROC_SECONDARY
&& current_device != dpio_count) {
- rte_bus_remove_device(&rte_fslmc_bus.bus,
+ rte_bus_remove_device(&rte_fslmc_bus,
&dev->device);
break;
}
if (rte_eal_process_type() == RTE_PROC_PRIMARY
&& current_device == dpio_count) {
- rte_bus_remove_device(&rte_fslmc_bus.bus,
+ rte_bus_remove_device(&rte_fslmc_bus,
&dev->device);
break;
}
@@ -1760,7 +1760,7 @@ fslmc_vfio_process_group(void)
/* Unknown - ignore */
DPAA2_BUS_DEBUG("Found unknown device (%s)",
dev->device.name);
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
free(dev);
dev = NULL;
}
diff --git a/drivers/bus/fslmc/portal/dpaa2_hw_dprc.c b/drivers/bus/fslmc/portal/dpaa2_hw_dprc.c
index a66e55a456..868ed646af 100644
--- a/drivers/bus/fslmc/portal/dpaa2_hw_dprc.c
+++ b/drivers/bus/fslmc/portal/dpaa2_hw_dprc.c
@@ -49,7 +49,7 @@ rte_dpaa2_create_dprc_device(int vdev_fd __rte_unused,
return ret;
}
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
/** DPRC is always created before it's children are created.*/
dev->container = dprc_node;
if (dev->dev_type == DPAA2_ETH) {
diff --git a/drivers/bus/fslmc/private.h b/drivers/bus/fslmc/private.h
index a0dda4f9d6..20a454c3fc 100644
--- a/drivers/bus/fslmc/private.h
+++ b/drivers/bus/fslmc/private.h
@@ -9,13 +9,6 @@
#include <bus_fslmc_driver.h>
-/*
- * FSLMC bus
- */
-struct rte_fslmc_bus {
- struct rte_bus bus; /**< Generic Bus object */
-};
-
-extern struct rte_fslmc_bus rte_fslmc_bus;
+extern struct rte_bus rte_fslmc_bus;
#endif /* BUS_FSLMC_PRIVATE_H */
diff --git a/drivers/bus/pci/bsd/pci.c b/drivers/bus/pci/bsd/pci.c
index 78d14ab3ae..c6df31d486 100644
--- a/drivers/bus/pci/bsd/pci.c
+++ b/drivers/bus/pci/bsd/pci.c
@@ -297,20 +297,18 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)
}
/* device is valid, add in list (sorted) */
- if (TAILQ_EMPTY(&rte_pci_bus.bus.device_list)) {
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
- }
- else {
+ if (TAILQ_EMPTY(&rte_pci_bus.device_list)) {
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
+ } else {
struct rte_pci_device *dev2 = NULL;
int ret;
- RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus) {
ret = rte_pci_addr_cmp(&dev->addr, &dev2->addr);
if (ret > 0)
continue;
else if (ret < 0) {
- rte_bus_insert_device(&rte_pci_bus.bus, &dev2->device,
- &dev->device);
+ rte_bus_insert_device(&rte_pci_bus, &dev2->device, &dev->device);
} else { /* already registered */
dev2->kdrv = dev->kdrv;
dev2->max_vfs = dev->max_vfs;
@@ -322,7 +320,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)
}
return 0;
}
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
}
return 0;
@@ -378,7 +376,7 @@ rte_pci_scan(void)
pci_addr.function = matches[i].pc_sel.pc_func;
rte_pci_device_name(&pci_addr, name, sizeof(name));
- if (rte_bus_device_is_ignored(&rte_pci_bus.bus, name))
+ if (rte_bus_device_is_ignored(&rte_pci_bus, name))
continue;
if (pci_scan_one(fd, &matches[i]) < 0)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index cf8a60313b..9aae0a5d14 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -321,19 +321,18 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)
return 0;
}
/* device is valid, add in list (sorted) */
- if (TAILQ_EMPTY(&rte_pci_bus.bus.device_list)) {
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ if (TAILQ_EMPTY(&rte_pci_bus.device_list)) {
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
} else {
struct rte_pci_device *dev2;
- RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus) {
ret = rte_pci_addr_cmp(&dev->addr, &dev2->addr);
if (ret > 0)
continue;
if (ret < 0) {
- rte_bus_insert_device(&rte_pci_bus.bus, &dev2->device,
- &dev->device);
+ rte_bus_insert_device(&rte_pci_bus, &dev2->device, &dev->device);
} else { /* already registered */
if (!rte_dev_is_probed(&dev2->device)) {
dev2->kdrv = dev->kdrv;
@@ -377,7 +376,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)
return 0;
}
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
}
return 0;
@@ -458,7 +457,7 @@ rte_pci_scan(void)
if (parse_pci_addr_format(e->d_name, sizeof(e->d_name), &addr) != 0)
continue;
- if (rte_bus_device_is_ignored(&rte_pci_bus.bus, e->d_name))
+ if (rte_bus_device_is_ignored(&rte_pci_bus, e->d_name))
continue;
snprintf(dirname, sizeof(dirname), "%s/%s",
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 2bdb94a924..fd18b8772b 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -87,7 +87,7 @@ pci_common_set(struct rte_pci_device *dev)
dev->name, sizeof(dev->name));
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&rte_pci_bus.bus, dev->name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_pci_bus, dev->name);
if (dev->bus_info != NULL ||
asprintf(&dev->bus_info, "vendor_id=%"PRIx16", device_id=%"PRIx16,
@@ -329,7 +329,7 @@ pci_cleanup(void)
struct rte_pci_device *dev;
int error = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus) {
const struct rte_pci_driver *drv;
int ret = 0;
@@ -353,7 +353,7 @@ pci_cleanup(void)
rte_intr_instance_free(dev->vfio_req_intr_handle);
dev->vfio_req_intr_handle = NULL;
- rte_bus_remove_device(&rte_pci_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_pci_bus, &dev->device);
pci_free(RTE_PCI_DEVICE_INTERNAL(dev));
}
@@ -387,7 +387,7 @@ rte_pci_dump(FILE *f)
{
struct rte_pci_device *dev = NULL;
- RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus) {
pci_dump_one_device(f, dev);
}
}
@@ -422,7 +422,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_pci_register)
void
rte_pci_register(struct rte_pci_driver *driver)
{
- rte_bus_add_driver(&rte_pci_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_pci_bus, &driver->driver);
}
/* unregister a driver */
@@ -430,7 +430,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_pci_unregister)
void
rte_pci_unregister(struct rte_pci_driver *driver)
{
- rte_bus_remove_driver(&rte_pci_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_pci_bus, &driver->driver);
}
/*
@@ -447,7 +447,7 @@ pci_find_device_by_addr(const void *failure_addr)
check_point = (uint64_t)(uintptr_t)failure_addr;
- RTE_BUS_FOREACH_DEV(pdev, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(pdev, &rte_pci_bus) {
for (i = 0; i != RTE_DIM(pdev->mem_resource); i++) {
start = (uint64_t)(uintptr_t)pdev->mem_resource[i].addr;
len = pdev->mem_resource[i].len;
@@ -525,7 +525,7 @@ pci_unplug(struct rte_device *dev)
ret = rte_pci_detach_dev(pdev);
if (ret == 0) {
- rte_bus_remove_device(&rte_pci_bus.bus, &pdev->device);
+ rte_bus_remove_device(&rte_pci_bus, &pdev->device);
rte_devargs_remove(dev->devargs);
pci_free(RTE_PCI_DEVICE_INTERNAL(pdev));
}
@@ -582,7 +582,7 @@ rte_pci_get_iommu_class(void)
bool devices_want_pa = false;
int iommu_no_va = -1;
- RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus) {
/*
* We can check this only once, because the IOMMU hardware is
* the same for all of them.
@@ -594,7 +594,7 @@ rte_pci_get_iommu_class(void)
if (dev->kdrv == RTE_PCI_KDRV_UNKNOWN ||
dev->kdrv == RTE_PCI_KDRV_NONE)
continue;
- RTE_BUS_FOREACH_DRV(drv, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DRV(drv, &rte_pci_bus) {
enum rte_iova_mode dev_iova_mode;
if (!pci_bus_match(&drv->driver, &dev->device))
@@ -772,27 +772,25 @@ rte_pci_pasid_set_state(const struct rte_pci_device *dev,
offset + RTE_PCI_PASID_CTRL) != sizeof(pasid) ? -1 : 0;
}
-struct rte_pci_bus rte_pci_bus = {
- .bus = {
- .allow_multi_probe = true,
- .scan = rte_pci_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = pci_cleanup,
- .find_device = rte_bus_generic_find_device,
- .match = pci_bus_match,
- .probe_device = pci_probe_device,
- .unplug = pci_unplug,
- .parse = pci_parse,
- .dev_compare = pci_dev_compare,
- .devargs_parse = rte_pci_devargs_parse,
- .dma_map = pci_dma_map,
- .dma_unmap = pci_dma_unmap,
- .get_iommu_class = rte_pci_get_iommu_class,
- .dev_iterate = rte_pci_dev_iterate,
- .hot_unplug_handler = pci_hot_unplug_handler,
- .sigbus_handler = pci_sigbus_handler,
- },
+struct rte_bus rte_pci_bus = {
+ .allow_multi_probe = true,
+ .scan = rte_pci_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = pci_cleanup,
+ .find_device = rte_bus_generic_find_device,
+ .match = pci_bus_match,
+ .probe_device = pci_probe_device,
+ .unplug = pci_unplug,
+ .parse = pci_parse,
+ .dev_compare = pci_dev_compare,
+ .devargs_parse = rte_pci_devargs_parse,
+ .dma_map = pci_dma_map,
+ .dma_unmap = pci_dma_unmap,
+ .get_iommu_class = rte_pci_get_iommu_class,
+ .dev_iterate = rte_pci_dev_iterate,
+ .hot_unplug_handler = pci_hot_unplug_handler,
+ .sigbus_handler = pci_sigbus_handler,
};
-RTE_REGISTER_BUS(pci, rte_pci_bus.bus);
+RTE_REGISTER_BUS(pci, rte_pci_bus);
RTE_LOG_REGISTER_DEFAULT(pci_bus_logtype, NOTICE);
diff --git a/drivers/bus/pci/pci_params.c b/drivers/bus/pci/pci_params.c
index e308c85ed2..6cbd98b4c8 100644
--- a/drivers/bus/pci/pci_params.c
+++ b/drivers/bus/pci/pci_params.c
@@ -75,7 +75,7 @@ rte_pci_dev_iterate(const struct rte_bus *bus __rte_unused,
return NULL;
}
}
- dev = rte_bus_generic_find_device(&rte_pci_bus.bus, start, pci_dev_match, kvargs);
+ dev = rte_bus_generic_find_device(&rte_pci_bus, start, pci_dev_match, kvargs);
rte_kvargs_free(kvargs);
return dev;
}
diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
index c54ea7b9d8..8103c32881 100644
--- a/drivers/bus/pci/private.h
+++ b/drivers/bus/pci/private.h
@@ -29,14 +29,7 @@ extern int pci_bus_logtype;
#define RTE_PCI_DEVICE_INTERNAL_CONST(ptr) \
container_of(ptr, const struct rte_pci_device_internal, device)
-/**
- * Structure describing the PCI bus
- */
-struct rte_pci_bus {
- struct rte_bus bus; /**< Inherit the generic class */
-};
-
-extern struct rte_pci_bus rte_pci_bus;
+extern struct rte_bus rte_pci_bus;
struct rte_pci_driver;
struct rte_pci_device;
diff --git a/drivers/bus/pci/windows/pci.c b/drivers/bus/pci/windows/pci.c
index 3b3f97da27..7b51301d1e 100644
--- a/drivers/bus/pci/windows/pci.c
+++ b/drivers/bus/pci/windows/pci.c
@@ -382,7 +382,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data)
goto end;
rte_pci_device_name(&addr, name, sizeof(name));
- if (rte_bus_device_is_ignored(&rte_pci_bus.bus, name)) {
+ if (rte_bus_device_is_ignored(&rte_pci_bus, name)) {
/*
* We won't add this device, but we want to continue
* looking for supported devices
@@ -430,18 +430,17 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data)
}
/* device is valid, add in list (sorted) */
- if (TAILQ_EMPTY(&rte_pci_bus.bus.device_list)) {
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ if (TAILQ_EMPTY(&rte_pci_bus.device_list)) {
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
} else {
struct rte_pci_device *dev2 = NULL;
- RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus) {
ret = rte_pci_addr_cmp(&dev->addr, &dev2->addr);
if (ret > 0) {
continue;
} else if (ret < 0) {
- rte_bus_insert_device(&rte_pci_bus.bus, &dev2->device,
- &dev->device);
+ rte_bus_insert_device(&rte_pci_bus, &dev2->device, &dev->device);
} else { /* already registered */
dev2->kdrv = dev->kdrv;
dev2->max_vfs = dev->max_vfs;
@@ -451,7 +450,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data)
}
return 0;
}
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
}
return 0;
diff --git a/drivers/bus/platform/bus_platform_driver.h b/drivers/bus/platform/bus_platform_driver.h
index 3ac405a201..e4dcbacf5e 100644
--- a/drivers/bus/platform/bus_platform_driver.h
+++ b/drivers/bus/platform/bus_platform_driver.h
@@ -24,7 +24,6 @@ extern "C" {
#endif
/* Forward declarations */
-struct rte_platform_bus;
struct rte_platform_device;
struct rte_platform_driver;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 4492ed62ec..170a2e03d0 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -29,18 +29,20 @@
#define PLATFORM_BUS_DEVICES_PATH "/sys/bus/platform/devices"
+static struct rte_bus platform_bus;
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_platform_register)
void
rte_platform_register(struct rte_platform_driver *pdrv)
{
- rte_bus_add_driver(&platform_bus.bus, &pdrv->driver);
+ rte_bus_add_driver(&platform_bus, &pdrv->driver);
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_platform_unregister)
void
rte_platform_unregister(struct rte_platform_driver *pdrv)
{
- rte_bus_remove_driver(&platform_bus.bus, &pdrv->driver);
+ rte_bus_remove_driver(&platform_bus, &pdrv->driver);
}
static int
@@ -56,11 +58,11 @@ dev_add(const char *dev_name)
rte_strscpy(pdev->name, dev_name, sizeof(pdev->name));
pdev->device.name = pdev->name;
- pdev->device.devargs = rte_bus_find_devargs(&platform_bus.bus, dev_name);
+ pdev->device.devargs = rte_bus_find_devargs(&platform_bus, dev_name);
snprintf(path, sizeof(path), PLATFORM_BUS_DEVICES_PATH "/%s/numa_node", dev_name);
pdev->device.numa_node = eal_parse_sysfs_value(path, &val) ? rte_socket_id() : val;
- RTE_BUS_FOREACH_DEV(tmp, &platform_bus.bus) {
+ RTE_BUS_FOREACH_DEV(tmp, &platform_bus) {
if (!strcmp(tmp->name, pdev->name)) {
PLATFORM_LOG_LINE(INFO, "device %s already added", pdev->name);
@@ -72,7 +74,7 @@ dev_add(const char *dev_name)
}
}
- rte_bus_add_device(&platform_bus.bus, &pdev->device);
+ rte_bus_add_device(&platform_bus, &pdev->device);
PLATFORM_LOG_LINE(INFO, "adding device %s to the list", dev_name);
@@ -135,7 +137,7 @@ platform_bus_scan(void)
if (dev_name[0] == '.')
continue;
- if (rte_bus_device_is_ignored(&platform_bus.bus, dev_name))
+ if (rte_bus_device_is_ignored(&platform_bus, dev_name))
continue;
if (!dev_is_bound_vfio_platform(dev_name))
@@ -440,7 +442,7 @@ platform_bus_parse(const char *name, void *addr)
rte_strscpy(pdev.name, name, sizeof(pdev.name));
- RTE_BUS_FOREACH_DRV(pdrv, &platform_bus.bus) {
+ RTE_BUS_FOREACH_DRV(pdrv, &platform_bus) {
if (platform_bus_match(&pdrv->driver, &pdev.device))
break;
}
@@ -482,7 +484,7 @@ platform_bus_get_iommu_class(void)
const struct rte_platform_driver *pdrv;
struct rte_platform_device *pdev;
- RTE_BUS_FOREACH_DEV(pdev, &platform_bus.bus) {
+ RTE_BUS_FOREACH_DEV(pdev, &platform_bus) {
if (!rte_dev_is_probed(&pdev->device))
continue;
pdrv = RTE_BUS_DRIVER(pdev->device.driver, *pdrv);
@@ -498,8 +500,8 @@ platform_bus_cleanup(void)
{
struct rte_platform_device *pdev;
- RTE_BUS_FOREACH_DEV(pdev, &platform_bus.bus) {
- rte_bus_remove_device(&platform_bus.bus, &pdev->device);
+ RTE_BUS_FOREACH_DEV(pdev, &platform_bus) {
+ rte_bus_remove_device(&platform_bus, &pdev->device);
if (!rte_dev_is_probed(&pdev->device))
continue;
platform_bus_unplug(&pdev->device);
@@ -508,22 +510,20 @@ platform_bus_cleanup(void)
return 0;
}
-struct rte_platform_bus platform_bus = {
- .bus = {
- .scan = platform_bus_scan,
- .probe = rte_bus_generic_probe,
- .find_device = rte_bus_generic_find_device,
- .match = platform_bus_match,
- .probe_device = platform_bus_probe_device,
- .unplug = platform_bus_unplug,
- .parse = platform_bus_parse,
- .dma_map = platform_bus_dma_map,
- .dma_unmap = platform_bus_dma_unmap,
- .get_iommu_class = platform_bus_get_iommu_class,
- .dev_iterate = rte_bus_generic_dev_iterate,
- .cleanup = platform_bus_cleanup,
- },
+static struct rte_bus platform_bus = {
+ .scan = platform_bus_scan,
+ .probe = rte_bus_generic_probe,
+ .find_device = rte_bus_generic_find_device,
+ .match = platform_bus_match,
+ .probe_device = platform_bus_probe_device,
+ .unplug = platform_bus_unplug,
+ .parse = platform_bus_parse,
+ .dma_map = platform_bus_dma_map,
+ .dma_unmap = platform_bus_dma_unmap,
+ .get_iommu_class = platform_bus_get_iommu_class,
+ .dev_iterate = rte_bus_generic_dev_iterate,
+ .cleanup = platform_bus_cleanup,
};
-RTE_REGISTER_BUS(platform, platform_bus.bus);
+RTE_REGISTER_BUS(platform, platform_bus);
RTE_LOG_REGISTER_DEFAULT(platform_bus_logtype, NOTICE);
diff --git a/drivers/bus/platform/private.h b/drivers/bus/platform/private.h
index bf5d75df03..18d42d43d8 100644
--- a/drivers/bus/platform/private.h
+++ b/drivers/bus/platform/private.h
@@ -14,15 +14,6 @@
#include "bus_platform_driver.h"
-extern struct rte_platform_bus platform_bus;
-
-/*
- * Structure describing platform bus.
- */
-struct rte_platform_bus {
- struct rte_bus bus; /* Core bus */
-};
-
extern int platform_bus_logtype;
#define RTE_LOGTYPE_PLATFORM_BUS platform_bus_logtype
#define PLATFORM_LOG_LINE(level, ...) \
diff --git a/drivers/bus/uacce/uacce.c b/drivers/bus/uacce/uacce.c
index db49e887ad..eb58701a61 100644
--- a/drivers/bus/uacce/uacce.c
+++ b/drivers/bus/uacce/uacce.c
@@ -34,15 +34,8 @@
/* Support -a uacce:device-name when start DPDK application. */
#define UACCE_DEV_PREFIX "uacce:"
-/*
- * Structure describing the UACCE bus.
- */
-struct rte_uacce_bus {
- struct rte_bus bus; /* Inherit the generic class. */
-};
-
/* Forward declaration of UACCE bus. */
-static struct rte_uacce_bus uacce_bus;
+static struct rte_bus uacce_bus;
extern int uacce_bus_logtype;
@@ -229,7 +222,7 @@ uacce_scan_one(const char *dev_name)
return -ENOMEM;
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&uacce_bus.bus, dev_name);
+ dev->device.devargs = rte_bus_find_devargs(&uacce_bus, dev_name);
snprintf(dev->name, sizeof(dev->name), "%s", dev_name);
snprintf(dev->dev_root, sizeof(dev->dev_root), "%s/%s",
UACCE_BUS_CLASS_PATH, dev_name);
@@ -253,7 +246,7 @@ uacce_scan_one(const char *dev_name)
if (ret != 0)
goto err;
- rte_bus_add_device(&uacce_bus.bus, &dev->device);
+ rte_bus_add_device(&uacce_bus, &dev->device);
return 0;
err:
@@ -283,7 +276,7 @@ uacce_scan(void)
continue;
}
- if (rte_bus_device_is_ignored(&uacce_bus.bus, e->d_name))
+ if (rte_bus_device_is_ignored(&uacce_bus, e->d_name))
continue;
if (uacce_scan_one(e->d_name) < 0)
@@ -379,7 +372,7 @@ uacce_cleanup(void)
struct rte_uacce_device *dev;
int error = 0;
- RTE_BUS_FOREACH_DEV(dev, &uacce_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &uacce_bus) {
const struct rte_uacce_driver *dr;
int ret = 0;
@@ -397,7 +390,7 @@ uacce_cleanup(void)
dev->device.driver = NULL;
free:
- rte_bus_remove_device(&uacce_bus.bus, &dev->device);
+ rte_bus_remove_device(&uacce_bus, &dev->device);
free(dev);
}
@@ -431,7 +424,7 @@ uacce_unplug(struct rte_device *dev)
ret = uacce_detach_dev(uacce_dev);
if (ret == 0) {
- rte_bus_remove_device(&uacce_bus.bus, &uacce_dev->device);
+ rte_bus_remove_device(&uacce_bus, &uacce_dev->device);
rte_devargs_remove(dev->devargs);
free(uacce_dev);
}
@@ -550,29 +543,27 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_uacce_register)
void
rte_uacce_register(struct rte_uacce_driver *driver)
{
- rte_bus_add_driver(&uacce_bus.bus, &driver->driver);
+ rte_bus_add_driver(&uacce_bus, &driver->driver);
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_uacce_unregister)
void
rte_uacce_unregister(struct rte_uacce_driver *driver)
{
- rte_bus_remove_driver(&uacce_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&uacce_bus, &driver->driver);
}
-static struct rte_uacce_bus uacce_bus = {
- .bus = {
- .scan = uacce_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = uacce_cleanup,
- .match = uacce_bus_match,
- .probe_device = uacce_probe_device,
- .unplug = uacce_unplug,
- .find_device = rte_bus_generic_find_device,
- .parse = uacce_parse,
- .dev_iterate = rte_bus_generic_dev_iterate,
- },
+static struct rte_bus uacce_bus = {
+ .scan = uacce_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = uacce_cleanup,
+ .match = uacce_bus_match,
+ .probe_device = uacce_probe_device,
+ .unplug = uacce_unplug,
+ .find_device = rte_bus_generic_find_device,
+ .parse = uacce_parse,
+ .dev_iterate = rte_bus_generic_dev_iterate,
};
-RTE_REGISTER_BUS(uacce, uacce_bus.bus);
+RTE_REGISTER_BUS(uacce, uacce_bus);
RTE_LOG_REGISTER_DEFAULT(uacce_bus_logtype, NOTICE);
diff --git a/drivers/bus/vmbus/linux/vmbus_bus.c b/drivers/bus/vmbus/linux/vmbus_bus.c
index 6268a14d40..0af10f6a69 100644
--- a/drivers/bus/vmbus/linux/vmbus_bus.c
+++ b/drivers/bus/vmbus/linux/vmbus_bus.c
@@ -39,8 +39,6 @@ static const rte_uuid_t vmbus_nic_uuid = {
0xf2, 0xd2, 0xf9, 0x65, 0xed, 0xe
};
-extern struct rte_vmbus_bus rte_vmbus_bus;
-
/* Read sysfs file to get UUID */
static int
parse_sysfs_uuid(const char *filename, rte_uuid_t uu)
@@ -332,7 +330,7 @@ vmbus_scan_one(const char *name)
dev->monitor_id = UINT8_MAX;
}
- dev->device.devargs = rte_bus_find_devargs(&rte_vmbus_bus.bus, dev_name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_vmbus_bus, dev_name);
dev->device.numa_node = SOCKET_ID_ANY;
if (vmbus_use_numa(dev)) {
@@ -356,7 +354,7 @@ vmbus_scan_one(const char *name)
/* device is valid, add in list (sorted) */
VMBUS_LOG(DEBUG, "Adding vmbus device %s", name);
- RTE_BUS_FOREACH_DEV(dev2, &rte_vmbus_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &rte_vmbus_bus) {
int ret;
ret = rte_uuid_compare(dev->device_id, dev2->device_id);
@@ -364,7 +362,7 @@ vmbus_scan_one(const char *name)
continue;
if (ret < 0) {
- rte_bus_insert_device(&rte_vmbus_bus.bus, &dev2->device, &dev->device);
+ rte_bus_insert_device(&rte_vmbus_bus, &dev2->device, &dev->device);
} else { /* already registered */
VMBUS_LOG(NOTICE,
"%s already registered", name);
@@ -374,7 +372,7 @@ vmbus_scan_one(const char *name)
return 0;
}
- rte_bus_add_device(&rte_vmbus_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_vmbus_bus, &dev->device);
return 0;
error:
VMBUS_LOG(DEBUG, "failed");
diff --git a/drivers/bus/vmbus/private.h b/drivers/bus/vmbus/private.h
index 6abb97c607..6efac86b77 100644
--- a/drivers/bus/vmbus/private.h
+++ b/drivers/bus/vmbus/private.h
@@ -15,14 +15,7 @@
#include <rte_eal_paging.h>
#include <rte_vmbus_reg.h>
-/**
- * Structure describing the VM bus
- */
-struct rte_vmbus_bus {
- struct rte_bus bus; /**< Inherit the generic class */
-};
-
-extern struct rte_vmbus_bus rte_vmbus_bus;
+extern struct rte_bus rte_vmbus_bus;
extern int vmbus_logtype_bus;
#define RTE_LOGTYPE_VMBUS_BUS vmbus_logtype_bus
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 2b1730afc2..01573927ce 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -23,8 +23,6 @@
#include "private.h"
-extern struct rte_vmbus_bus rte_vmbus_bus;
-
/* map a particular resource from a file */
void *
vmbus_map_resource(void *requested_addr, int fd, off_t offset, size_t size,
@@ -128,7 +126,7 @@ RTE_EXPORT_SYMBOL(rte_vmbus_probe)
int
rte_vmbus_probe(void)
{
- return rte_bus_generic_probe(&rte_vmbus_bus.bus);
+ return rte_bus_generic_probe(&rte_vmbus_bus);
}
static int
@@ -137,7 +135,7 @@ rte_vmbus_cleanup(void)
struct rte_vmbus_device *dev;
int error = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_vmbus_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_vmbus_bus) {
const struct rte_vmbus_driver *drv;
int ret;
@@ -154,7 +152,7 @@ rte_vmbus_cleanup(void)
rte_vmbus_unmap_device(dev);
dev->device.driver = NULL;
- rte_bus_remove_device(&rte_vmbus_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_vmbus_bus, &dev->device);
free(dev);
}
@@ -194,7 +192,7 @@ rte_vmbus_register(struct rte_vmbus_driver *driver)
VMBUS_LOG(DEBUG,
"Registered driver %s", driver->driver.name);
- rte_bus_add_driver(&rte_vmbus_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_vmbus_bus, &driver->driver);
}
/* unregister vmbus driver */
@@ -202,22 +200,20 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vmbus_unregister)
void
rte_vmbus_unregister(struct rte_vmbus_driver *driver)
{
- rte_bus_remove_driver(&rte_vmbus_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_vmbus_bus, &driver->driver);
}
/* VMBUS doesn't support hotplug */
-struct rte_vmbus_bus rte_vmbus_bus = {
- .bus = {
- .scan = rte_vmbus_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = rte_vmbus_cleanup,
- .find_device = rte_bus_generic_find_device,
- .match = vmbus_bus_match,
- .probe_device = vmbus_probe_device,
- .parse = vmbus_parse,
- .dev_compare = vmbus_dev_compare,
- },
+struct rte_bus rte_vmbus_bus = {
+ .scan = rte_vmbus_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = rte_vmbus_cleanup,
+ .find_device = rte_bus_generic_find_device,
+ .match = vmbus_bus_match,
+ .probe_device = vmbus_probe_device,
+ .parse = vmbus_parse,
+ .dev_compare = vmbus_dev_compare,
};
-RTE_REGISTER_BUS(vmbus, rte_vmbus_bus.bus);
+RTE_REGISTER_BUS(vmbus, rte_vmbus_bus);
RTE_LOG_REGISTER_DEFAULT(vmbus_logtype_bus, NOTICE);
--
2.53.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox