* RE: [PATCH v5] mempool: improve cache behaviour and performance
From: Morten Brørup @ 2026-05-26 10:37 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, Andrew Rybchenko, Jingjing Wu, Praveen Shetty,
Hemant Agrawal, Sachin Saxena
In-Reply-To: <ahVqYpVXMn59Tl2d@bricha3-mobl1.ger.corp.intel.com>
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Tuesday, 26 May 2026 11.40
>
> On Tue, May 26, 2026 at 10:41:44AM +0200, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Friday, 22 May 2026 18.12
> > >
> > > On Sun, Apr 19, 2026 at 09:55:26AM +0000, Morten Brørup wrote:
> > > > This patch refactors the mempool cache to eliminate some
> unexpected
> > > > behaviour and reduce the mempool cache miss rate.
> > > >
> > >
> > > Agree in principle with most of these changes. As we dicussed at
> the
> > > DPDK
> > > summit conference, only issue I really have is with the threshold
> > > limits
> > > here - allocating and freeing only half the cache at a time seems
> > > overly
> > > conservative. Thinking about use-cases:
> > >
> > > 1 for apps where alloc + free (generally Rx+Tx) is on the same
> core(s),
> > > then we should run (almost) entirely out of cache.
> >
> > I strongly disagree about any goal to run the cache low.
> > The primary goal is to minimize the cache miss (refill and replenish)
> rate.
> >
> > > 2 for apps where we have alloc and free on different cores, then we
> > > have
> > > some caches always being filled and others always being emptied
> >
> > Agree.
> >
> > >
> > > For case #1, we only need worry about the thresholds for the odd
> case
> > > where we have a burst that causes us to overflow our cache (and we
> > > can't increase our cache size to cope and avoid that).
> > > Otherwise the thresholds don't matter.
> >
> > It seems like you assume the application only does something like
> this:
> > Rx -> Rewrite -> Tx
> >
> > In that case, the per-lcore cache only needs capacity for one burst,
> yes.
> > With my patch, the cache can be rightsized by requesting a cache size
> of 2 * burst size.
> > (Then the fill level will be either size/2 or empty, i.e. one burst
> or zero.
> > This also happens to meet your suggested goal about low fill level,
> which I disagree with.)
> >
> > However, I don't think that is a realistic use case.
> > Many apps do something like this:
> >
> > |-> Rewrite ->|
> > Rx ->| |-> Tx
> > |-> Hold |
> > Release ->|
> >
>
> I agree, that's why our cache size needs to be bigger than our burst
> size.
>
> > They often hold back packets before they are transmitted.
> > For a simple router, when the destination IP address is not in the
> neighbor table, packets to that IP address are queued until ARP/ND has
> been resolved, and then they are dequeued and transmitted.
> > Or apps performing shaping or pacing, where packets are held back in
> queues, and dequeued at the appropriate time.
> > For such apps, the waves are much bigger (than the simple Rx-
> >Rewrite->Tx use case).
> >
> > With a random enqueue/dequeue pattern, replenishing/draining the
> cache to size/2 minimizes the probability of reaching one of its edges
> (empty or full), triggering a "cache miss" (refill/replenish).
> >
> > > However, for case #2, the thresholds are constantly involved as
> > > we
> > > always are going to backing store. In this case, we really want to
> have
> > > the
> > > allocs *always* fill the cache completely, and the frees completely
> > > empty
> > > the cache.
> >
> > Agree.
> >
> > >
> > > Because of this, while we want to avoid cases where we fill the
> cache
> > > completely only to have a further free causing it to be flushed,
> > > because of
> > > case #2 we cannot be overly conservative in how much we free/empty.
> > > Ideally, we want to fill to full less a single burst, and empty
> leaving
> > > only a single burst in the cache. Unfortunately, we don't know what
> > > those
> > > burst limits are, so we have to try and guess the best behaviour
> from
> > > everything else.
> >
> > I agree about not wanting to be overly conservative.
> > But in the use cases I have described for #1, I don't think a target
> fill level of size/2 is overly conservative.
> >
> > I also acknowledge that this patch doubles the mempool cache miss
> rate for #2.
> > E.g. with a cache size of 512 and burst size of 64, the per-burst
> miss rate will be 64 / (1/2 * 512) = 1/4, compared to 64 / 512 = 1/8
> with a full replenish/drain algorithm.
> >
> > In theory, we could make it build time configurable to optimize
> mempools for #2.
> > But mempools are also used for other objects than mbufs, so that
> would have unwanted side effects for non-mbuf mempools.
> >
> > If we went for an algorithm targeting replenish/drain at 25 % from
> the edges, the per-burst miss rate for #2 would be: 64 / (3/4 * 512) =
> 1/6.
> >
> > How about addressing #2 in the release notes:
> > We describe that the cache refill/drain algorithm has been changed to
> only refill/drain to 50 % of the cache size, so pipelined applications
> performing Rx (mempool get) and Tx (mempool put) on separate cores
> should configure their mbuf pools with double the cache size of what
> they previously were to achieve similar performance.
> >
> > >
> > > All that said, commits with specific suggestions inline.
> > >
> > > /Bruce
> > >
> > > <snip>
> > >
> > > > diff --git a/lib/mempool/rte_mempool.h
> b/lib/mempool/rte_mempool.h
> > > > index 2e54fc4466..432c43ab15 100644
> > > > --- a/lib/mempool/rte_mempool.h
> > > > +++ b/lib/mempool/rte_mempool.h
> > > > @@ -89,7 +89,7 @@ struct __rte_cache_aligned
> rte_mempool_debug_stats
> > > {
> > > > */
> > > > struct __rte_cache_aligned rte_mempool_cache {
> > > > uint32_t size; /**< Size of the cache */
> > > > - uint32_t flushthresh; /**< Threshold before we flush excess
> > > elements */
> > > > + uint32_t flushthresh; /**< Obsolete; for API/ABI
> compatibility
> > > purposes only */
> > > > uint32_t len; /**< Current cache count */
> > > > #ifdef RTE_LIBRTE_MEMPOOL_STATS
> > > > uint32_t unused;
> > > > @@ -107,8 +107,10 @@ struct __rte_cache_aligned rte_mempool_cache
> {
> > > > /**
> > > > * Cache objects
> > > > *
> > > > - * Cache is allocated to this size to allow it to overflow
> in
> > > certain
> > > > - * cases to avoid needless emptying of cache.
> > > > + * Note:
> > > > + * Cache is allocated at double size for API/ABI
> compatibility
> > > purposes only.
> > > > + * When reducing its size at an API/ABI breaking release,
> > > > + * remember to add a cache guard after it.
> > > > */
> > > > alignas(RTE_CACHE_LINE_SIZE) void
> > > *objs[RTE_MEMPOOL_CACHE_MAX_SIZE * 2];
> > > > };
> > > > @@ -1046,12 +1048,17 @@ rte_mempool_free(struct rte_mempool *mp);
> > > > * @param cache_size
> > > > * If cache_size is non-zero, the rte_mempool library will try
> to
> > > > * limit the accesses to the common lockless pool, by
> maintaining
> > > a
> > > > - * per-lcore object cache. This argument must be lower or
> equal to
> > > > - * RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.
> > > > + * per-lcore object cache. This argument must be an even
> number,
> > > > + * lower or equal to RTE_MEMPOOL_CACHE_MAX_SIZE and n.
> > > > * The access to the per-lcore table is of course
> > > > * faster than the multi-producer/consumer pool. The cache can
> be
> > > > * disabled if the cache_size argument is set to 0; it can be
> > > useful to
> > > > * avoid losing objects in cache.
> > > > + * Note:
> > > > + * Mempool put/get requests of more than cache_size / 2
> objects
> > > may be
> > > > + * partially or fully served directly by the multi-
> > > producer/consumer
> > > > + * pool, to avoid the overhead of copying the objects twice
> > > (instead of
> > > > + * once) when using the cache as a bounce buffer.
> > > > * @param private_data_size
> > > > * The size of the private data appended after the mempool
> > > > * structure. This is useful for storing some private data
> after
> > > the
> > > > @@ -1390,24 +1397,32 @@ rte_mempool_do_generic_put(struct
> rte_mempool
> > > *mp, void * const *obj_table,
> > > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
> > > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
> > > >
> > > > - __rte_assume(cache->flushthresh <=
> RTE_MEMPOOL_CACHE_MAX_SIZE *
> > > 2);
> > > > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > > > - __rte_assume(cache->len <= cache->flushthresh);
> > > > - if (likely(cache->len + n <= cache->flushthresh)) {
> > > > + __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE
> / 2);
> > > > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > > + __rte_assume(cache->len <= cache->size);
> > > > + if (likely(cache->len + n <= cache->size)) {
> > > > /* Sufficient room in the cache for the objects. */
> > > > cache_objs = &cache->objs[cache->len];
> > > > cache->len += n;
> > > > - } else if (n <= cache->flushthresh) {
> > > > + } else if (n <= cache->size / 2) {
> > > > /*
> > > > - * The cache is big enough for the objects, but - as
> > > detected by
> > > > - * the comparison above - has insufficient room for
> them.
> > > > - * Flush the cache to make room for the objects.
> > > > + * The number of objects is within the cache bounce
> buffer
> > > limit,
> > > > + * but - as detected by the comparison above - the
> cache
> > > has
> > > > + * insufficient room for them.
> > > > + * Flush the cache to the backend to make room for
> the
> > > objects;
> > > > + * flush (size / 2) objects from the bottom of the
> cache,
> > > where
> > > > + * objects are less hot, and move down the remaining
> > > objects, which
> > > > + * are more hot, from the upper half of the cache.
> > > > */
> > > > - cache_objs = &cache->objs[0];
> > > > - rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache-
> >len);
> > > > - cache->len = n;
> > > > + __rte_assume(cache->len > cache->size / 2);
> > > > + rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0],
> cache-
> > > >size / 2);
> > > > + rte_memcpy(&cache->objs[0], &cache->objs[cache->size
> / 2],
> > > > + sizeof(void *) * (cache->len - cache-
> >size /
> > > 2));
> > > > + cache_objs = &cache->objs[cache->len - cache->size /
> 2];
> > > > + cache->len = cache->len - cache->size / 2 + n;
> > >
> > > The flushing of only half the cache I'm not so certain about. I
> agree
> > > that
> > > we want to not flush to empty, but I also think that we want to do
> more
> > > than a half-flush, especially since we do an enqueue to the cache
> > > immediately afterwards. Consider the case where we have a cache
> size of
> > > 128, and we do an enqueue of 32, with the cache currently full. In
> that
> > > case we only flush 64, reducing the cache to 64, but then
> immediately
> > > bringing it back up to 96.
> >
> > I thought in depth about whether the flush/replenish sizes should
> consider the request size or not. (E.g. if I should replenish size/2 or
> size/2+request.)
> > I decided for not considering the request size, for two reasons:
> > a) It roughly doesn't matter, especially when considering a sequence
> of random get/put requests.
> > b) The size of the backend transactions become fixed, which has
> performance benefits: With my patch, they are always size/2, so if the
> cache size is 2^N, the backend transactions are 2^N and CPU cache
> aligned.
> >
> > > For cases where we have pipelines with all
> > > alloc
> > > on one core and all free on another, this half-flush would be
> > > inefficient.
> > >
> > > Instead, I would look to have a lower target threshold post-flush,
> and
> > > I
> > > would suggest 25% - taking into account the newly freed buffers.
> >
> > It's not good for #1.
> > I agree that it is better for #2. But I don't think #2 is the likely
> use case.
> >
> > After our discussion at the summit, I did start working a patch
> targeting fill levels at 25% from the cache edges, but I don't think
> it's better; so I'd rather stick with a target fill level of 50%.
> >
> > > For example:
> > >
> > > /* if n > our target of 1/4 full, flush everything,
> > > * else flush so that we end up with 1/4 full after n added.
> > > */
> > > flush_count = n > cache->size/4 ? cache->len :
> > > (cache->len + n) - cache->size/4;
> > >
> > >
> > > > } else {
> > > > - /* The request itself is too big for the cache. */
> > > > + /* The request itself is too big. */
> > > > goto driver_enqueue_stats_incremented;
> > >
> > > I think original comment is better. The request itself is not too
> big
> > > for
> > > the whole mempool, just for the cache.
> >
> > Ack.
> >
> > >
> > > > }
> > > >
> > > > @@ -1524,7 +1539,7 @@ rte_mempool_do_generic_get(struct
> rte_mempool
> > > *mp, void **obj_table,
> > > > /* The cache is a stack, so copy will be in reverse order.
> */
> > > > cache_objs = &cache->objs[cache->len];
> > > >
> > > > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > > > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > > if (likely(n <= cache->len)) {
> > > > /* The entire request can be satisfied from the
> cache. */
> > > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk,
> 1);
> > > > @@ -1548,13 +1563,13 @@ rte_mempool_do_generic_get(struct
> rte_mempool
> > > *mp, void **obj_table,
> > > > for (index = 0; index < len; index++)
> > > > *obj_table++ = *--cache_objs;
> > > >
> > > > - /* Dequeue below would overflow mem allocated for cache? */
> > > > - if (unlikely(remaining > RTE_MEMPOOL_CACHE_MAX_SIZE))
> > > > + /* Dequeue below would exceed the cache bounce buffer
> limit? */
> > > > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE
> / 2);
> > > > + if (unlikely(remaining > cache->size / 2))
> > > > goto driver_dequeue;
> > > >
> > > > - /* Fill the cache from the backend; fetch size + remaining
> > > objects. */
> > > > - ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs,
> > > > - cache->size + remaining);
> > > > + /* Fill the cache from the backend; fetch (size / 2)
> objects. */
> > > > + ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache-
> >size /
> > > 2);
> > >
> > > Again, the cache->size / 2 doesn't seem right here. We at most
> half-
> > > fill
> > > the cache and then take some objects from that, meaning that have
> just
> > > done
> > > a re-fill of cache but end the function with it less than half
> full.
> > > Since
> > > we take from this value, I'd suggest just filling the cache
> completely.
> >
> > The issues at the edges of the cache are symmetrical.
> > If we replenish the cache to full, and the next transaction is a put,
> the cache needs to be drained.
> > That's why I replenish to size/2.
> >
>
> Not necessarily. After you fill the cache here (to 50%) you then take
> elements from it. If we filled to 100%, then we'd only hit a flush if
> we
> freed back more elements than we just allocated. Now, that's reasonably
> likely which is why I'm ok to not filling to 100%. However, doing a
> refill
> as part of the op, and then leaving the cache less than half full after
> the
> op finishes is just wasteful IMHO!
I'm targeting half full, and disregard if it is pre-op or post-op.
This makes the backend transactions cache aligned in both addressing and size, which opens an opportunity to performance optimize the mempool drivers for this.
>
> One other point. The obvious worst case scenario we want to avoid is
> constant fill/empty/fill/empty sequences. However, so long as we have
> an
> asymmetry in how we do fills and empty thresholds, a repeating sequence
> should never occur, and even pairs of fill/empty should be very rare.
> Consider the case, for example, where we fill to 100% but only drain to
> 25%. For this, we can ignore scenario #2, since we should have pretty
> good
> cache usage. For scenario #1, of allocs/frees on a single core,
> randomly
> interleaved, our worst case is:
> 1) alloc with cache empty, in which case we fill to 100%, then take the
> alloc
> 2) have a free of a burst greater than that which we just allocated,
> causing an immediate flush again.
>
> That's pretty poor behaviour, but the thing is that after the flush we
> now
> have mempool at 25% + freed burst - so expected between 25-50% of cache
> utilization. That's the sweet spot we want to target - after each
> operation, the mempool cache should ideally be at 50%. Whether the next
> op
> is an alloc or a free, our cache can handle it, and likely a couple of
> each
> in sequence. Therefore, our possible fill/empty combinations are likely
> to
> be rare occurances.
>
> [In all this, I am making the assumption that burst size is well less
> than
> cache size. Also, similar logic would be applicable for the inverse
> scenario, e.g. flush to empty (and fill burst) and fill to 75%]
I'm not so sure about this assumption.
With a cache size of 512 and a bursts of 64, the cache only holds 8 bursts.
50% is 4 bursts, and 25% is only 2 bursts.
Using a replenish/drain level in the middle requires 5 bursts in either direction to pass the edge (and trigger replenish/flush).
Using a replenish/drain level 25% from the edge requires only 3 bursts in the wrong direction to pass the edge (and trigger replenish/flush). Much higher probability with random get/put.
>
> Now, all said, I tend to agree that we want to leave space for a decent
> size burst after a fill. That is why I think that filling to 75% is
> reasonable. After an alloc that triggers a fill, I don't want the cache
> less than 50% full, but not completely full so there is room for a free
> without a flush, and similarly for a free that triggers a flush, the
> cache
> should not be empty, but also should not be more than half full.
>
> One suggestion - we could always add a simple tunable that specifies
> the
> margin, or reserved entries for alloc and free. We can then guide in
> the
> docs that the value should be e.g. "zero for apps where alloc and free
> take
> place on different cores. 20%-50% of cache is recommended where alloc
> and
> free take place on the same core"
Yes, a simple tunable is a really good idea.
At this point, I think we should optimize for use case #1, and go for the 50% fill level.
Then we can add a tunable to optimize for use case #2 later. I will try to come up with a draft for such a follow-up patch within the next few days.
The 50% fill level in this patch is not as bad for use case #2 (roughly doubling the burst miss rate from 1/8 to 1/4), compared to how bad the original algorithm is for use case #1 (very high miss probability - only two ops in the wrong direction - after drain/replenish).
-Morten
^ permalink raw reply
* Re: [PATCH v3 04/25] dma/idxd: clear device at scan
From: Bruce Richardson @ 2026-05-26 9:50 UTC (permalink / raw)
To: David Marchand; +Cc: dev, thomas, stephen, Kevin Laatz
In-Reply-To: <CAJFAV8zC=LUR8r4s7q5o9W2KaAa3PjKD8+qoAUsV70Ddqvgp1A@mail.gmail.com>
On Tue, May 26, 2026 at 11:48:46AM +0200, David Marchand wrote:
> On Tue, 26 May 2026 at 11:42, Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Tue, May 26, 2026 at 10:41:44AM +0200, David Marchand wrote:
> > > Currently, the device object is only manipulated by the dma/idxd bus
> > > callbacks and EAL is not looking too much into this object.
> > >
> > > However, in the next refactoring, EAL will expect a clean object, like
> > > when checking that the device has been already probed
> > > (iow dev->driver != NULL).
> > >
> > > Request a 0'd object when allocating.
> > >
> > > Reported-by: Bruce Richardson <bruce.richardson@intel.com>
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> > > drivers/dma/idxd/idxd_bus.c | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> >
> > Out of interest, did you (or Claude :-) ) run a scan over the other bus
> > drivers to see if any others are similarly not properly zeroed on init?
>
> I did check, the "legacy" way.
>
> All other are calling calloc(), or rte_zmalloc_*, or manually clearing.
> I hesitated at changing the malloc+memset pattern to calloc, but I
> left it for later.
> The series is big enough.
>
Great, thanks for confirming. I also agree with your choice to not bother
changing the patterns for this set.
/Bruce
^ permalink raw reply
* Re: [PATCH v3 04/25] dma/idxd: clear device at scan
From: David Marchand @ 2026-05-26 9:48 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev, thomas, stephen, Kevin Laatz
In-Reply-To: <ahVq9dJrOK04Ujwd@bricha3-mobl1.ger.corp.intel.com>
On Tue, 26 May 2026 at 11:42, Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Tue, May 26, 2026 at 10:41:44AM +0200, David Marchand wrote:
> > Currently, the device object is only manipulated by the dma/idxd bus
> > callbacks and EAL is not looking too much into this object.
> >
> > However, in the next refactoring, EAL will expect a clean object, like
> > when checking that the device has been already probed
> > (iow dev->driver != NULL).
> >
> > Request a 0'd object when allocating.
> >
> > Reported-by: Bruce Richardson <bruce.richardson@intel.com>
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> > drivers/dma/idxd/idxd_bus.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
>
> Out of interest, did you (or Claude :-) ) run a scan over the other bus
> drivers to see if any others are similarly not properly zeroed on init?
I did check, the "legacy" way.
All other are calling calloc(), or rte_zmalloc_*, or manually clearing.
I hesitated at changing the malloc+memset pattern to calloc, but I
left it for later.
The series is big enough.
--
David Marchand
^ permalink raw reply
* Re: [PATCH v3 04/25] dma/idxd: clear device at scan
From: Bruce Richardson @ 2026-05-26 9:42 UTC (permalink / raw)
To: David Marchand; +Cc: dev, thomas, stephen, Kevin Laatz
In-Reply-To: <20260526084212.3145685-5-david.marchand@redhat.com>
On Tue, May 26, 2026 at 10:41:44AM +0200, David Marchand wrote:
> Currently, the device object is only manipulated by the dma/idxd bus
> callbacks and EAL is not looking too much into this object.
>
> However, in the next refactoring, EAL will expect a clean object, like
> when checking that the device has been already probed
> (iow dev->driver != NULL).
>
> Request a 0'd object when allocating.
>
> Reported-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> drivers/dma/idxd/idxd_bus.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
Out of interest, did you (or Claude :-) ) run a scan over the other bus
drivers to see if any others are similarly not properly zeroed on init?
/Bruce
> diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c
> index 291cd6c707..f267c20a59 100644
> --- a/drivers/dma/idxd/idxd_bus.c
> +++ b/drivers/dma/idxd/idxd_bus.c
> @@ -322,7 +322,7 @@ dsa_scan(void)
> }
> IDXD_PMD_DEBUG("%s(): found %s/%s", __func__, path, wq->d_name);
>
> - dev = malloc(sizeof(*dev));
> + dev = calloc(1, sizeof(*dev));
> if (dev == NULL) {
> closedir(dev_dir);
> return -ENOMEM;
> --
> 2.53.0
>
^ permalink raw reply
* Re: [PATCH v3 04/25] dma/idxd: clear device at scan
From: Bruce Richardson @ 2026-05-26 9:41 UTC (permalink / raw)
To: David Marchand; +Cc: dev, thomas, stephen, Kevin Laatz
In-Reply-To: <20260526084212.3145685-5-david.marchand@redhat.com>
On Tue, May 26, 2026 at 10:41:44AM +0200, David Marchand wrote:
> Currently, the device object is only manipulated by the dma/idxd bus
> callbacks and EAL is not looking too much into this object.
>
> However, in the next refactoring, EAL will expect a clean object, like
> when checking that the device has been already probed
> (iow dev->driver != NULL).
>
> Request a 0'd object when allocating.
>
> Reported-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> drivers/dma/idxd/idxd_bus.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c
> index 291cd6c707..f267c20a59 100644
> --- a/drivers/dma/idxd/idxd_bus.c
> +++ b/drivers/dma/idxd/idxd_bus.c
> @@ -322,7 +322,7 @@ dsa_scan(void)
> }
> IDXD_PMD_DEBUG("%s(): found %s/%s", __func__, path, wq->d_name);
>
> - dev = malloc(sizeof(*dev));
> + dev = calloc(1, sizeof(*dev));
> if (dev == NULL) {
> closedir(dev_dir);
> return -ENOMEM;
> --
> 2.53.0
>
^ permalink raw reply
* Re: [PATCH v5] mempool: improve cache behaviour and performance
From: Bruce Richardson @ 2026-05-26 9:39 UTC (permalink / raw)
To: Morten Brørup
Cc: dev, Andrew Rybchenko, Jingjing Wu, Praveen Shetty,
Hemant Agrawal, Sachin Saxena
In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35F6589A@smartserver.smartshare.dk>
On Tue, May 26, 2026 at 10:41:44AM +0200, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Friday, 22 May 2026 18.12
> >
> > On Sun, Apr 19, 2026 at 09:55:26AM +0000, Morten Brørup wrote:
> > > This patch refactors the mempool cache to eliminate some unexpected
> > > behaviour and reduce the mempool cache miss rate.
> > >
> >
> > Agree in principle with most of these changes. As we dicussed at the
> > DPDK
> > summit conference, only issue I really have is with the threshold
> > limits
> > here - allocating and freeing only half the cache at a time seems
> > overly
> > conservative. Thinking about use-cases:
> >
> > 1 for apps where alloc + free (generally Rx+Tx) is on the same core(s),
> > then we should run (almost) entirely out of cache.
>
> I strongly disagree about any goal to run the cache low.
> The primary goal is to minimize the cache miss (refill and replenish) rate.
>
> > 2 for apps where we have alloc and free on different cores, then we
> > have
> > some caches always being filled and others always being emptied
>
> Agree.
>
> >
> > For case #1, we only need worry about the thresholds for the odd case
> > where we have a burst that causes us to overflow our cache (and we
> > can't increase our cache size to cope and avoid that).
> > Otherwise the thresholds don't matter.
>
> It seems like you assume the application only does something like this:
> Rx -> Rewrite -> Tx
>
> In that case, the per-lcore cache only needs capacity for one burst, yes.
> With my patch, the cache can be rightsized by requesting a cache size of 2 * burst size.
> (Then the fill level will be either size/2 or empty, i.e. one burst or zero.
> This also happens to meet your suggested goal about low fill level, which I disagree with.)
>
> However, I don't think that is a realistic use case.
> Many apps do something like this:
>
> |-> Rewrite ->|
> Rx ->| |-> Tx
> |-> Hold |
> Release ->|
>
I agree, that's why our cache size needs to be bigger than our burst size.
> They often hold back packets before they are transmitted.
> For a simple router, when the destination IP address is not in the neighbor table, packets to that IP address are queued until ARP/ND has been resolved, and then they are dequeued and transmitted.
> Or apps performing shaping or pacing, where packets are held back in queues, and dequeued at the appropriate time.
> For such apps, the waves are much bigger (than the simple Rx->Rewrite->Tx use case).
>
> With a random enqueue/dequeue pattern, replenishing/draining the cache to size/2 minimizes the probability of reaching one of its edges (empty or full), triggering a "cache miss" (refill/replenish).
>
> > However, for case #2, the thresholds are constantly involved as
> > we
> > always are going to backing store. In this case, we really want to have
> > the
> > allocs *always* fill the cache completely, and the frees completely
> > empty
> > the cache.
>
> Agree.
>
> >
> > Because of this, while we want to avoid cases where we fill the cache
> > completely only to have a further free causing it to be flushed,
> > because of
> > case #2 we cannot be overly conservative in how much we free/empty.
> > Ideally, we want to fill to full less a single burst, and empty leaving
> > only a single burst in the cache. Unfortunately, we don't know what
> > those
> > burst limits are, so we have to try and guess the best behaviour from
> > everything else.
>
> I agree about not wanting to be overly conservative.
> But in the use cases I have described for #1, I don't think a target fill level of size/2 is overly conservative.
>
> I also acknowledge that this patch doubles the mempool cache miss rate for #2.
> E.g. with a cache size of 512 and burst size of 64, the per-burst miss rate will be 64 / (1/2 * 512) = 1/4, compared to 64 / 512 = 1/8 with a full replenish/drain algorithm.
>
> In theory, we could make it build time configurable to optimize mempools for #2.
> But mempools are also used for other objects than mbufs, so that would have unwanted side effects for non-mbuf mempools.
>
> If we went for an algorithm targeting replenish/drain at 25 % from the edges, the per-burst miss rate for #2 would be: 64 / (3/4 * 512) = 1/6.
>
> How about addressing #2 in the release notes:
> We describe that the cache refill/drain algorithm has been changed to only refill/drain to 50 % of the cache size, so pipelined applications performing Rx (mempool get) and Tx (mempool put) on separate cores should configure their mbuf pools with double the cache size of what they previously were to achieve similar performance.
>
> >
> > All that said, commits with specific suggestions inline.
> >
> > /Bruce
> >
> > <snip>
> >
> > > diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
> > > index 2e54fc4466..432c43ab15 100644
> > > --- a/lib/mempool/rte_mempool.h
> > > +++ b/lib/mempool/rte_mempool.h
> > > @@ -89,7 +89,7 @@ struct __rte_cache_aligned rte_mempool_debug_stats
> > {
> > > */
> > > struct __rte_cache_aligned rte_mempool_cache {
> > > uint32_t size; /**< Size of the cache */
> > > - uint32_t flushthresh; /**< Threshold before we flush excess
> > elements */
> > > + uint32_t flushthresh; /**< Obsolete; for API/ABI compatibility
> > purposes only */
> > > uint32_t len; /**< Current cache count */
> > > #ifdef RTE_LIBRTE_MEMPOOL_STATS
> > > uint32_t unused;
> > > @@ -107,8 +107,10 @@ struct __rte_cache_aligned rte_mempool_cache {
> > > /**
> > > * Cache objects
> > > *
> > > - * Cache is allocated to this size to allow it to overflow in
> > certain
> > > - * cases to avoid needless emptying of cache.
> > > + * Note:
> > > + * Cache is allocated at double size for API/ABI compatibility
> > purposes only.
> > > + * When reducing its size at an API/ABI breaking release,
> > > + * remember to add a cache guard after it.
> > > */
> > > alignas(RTE_CACHE_LINE_SIZE) void
> > *objs[RTE_MEMPOOL_CACHE_MAX_SIZE * 2];
> > > };
> > > @@ -1046,12 +1048,17 @@ rte_mempool_free(struct rte_mempool *mp);
> > > * @param cache_size
> > > * If cache_size is non-zero, the rte_mempool library will try to
> > > * limit the accesses to the common lockless pool, by maintaining
> > a
> > > - * per-lcore object cache. This argument must be lower or equal to
> > > - * RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.
> > > + * per-lcore object cache. This argument must be an even number,
> > > + * lower or equal to RTE_MEMPOOL_CACHE_MAX_SIZE and n.
> > > * The access to the per-lcore table is of course
> > > * faster than the multi-producer/consumer pool. The cache can be
> > > * disabled if the cache_size argument is set to 0; it can be
> > useful to
> > > * avoid losing objects in cache.
> > > + * Note:
> > > + * Mempool put/get requests of more than cache_size / 2 objects
> > may be
> > > + * partially or fully served directly by the multi-
> > producer/consumer
> > > + * pool, to avoid the overhead of copying the objects twice
> > (instead of
> > > + * once) when using the cache as a bounce buffer.
> > > * @param private_data_size
> > > * The size of the private data appended after the mempool
> > > * structure. This is useful for storing some private data after
> > the
> > > @@ -1390,24 +1397,32 @@ rte_mempool_do_generic_put(struct rte_mempool
> > *mp, void * const *obj_table,
> > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
> > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
> > >
> > > - __rte_assume(cache->flushthresh <= RTE_MEMPOOL_CACHE_MAX_SIZE *
> > 2);
> > > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > > - __rte_assume(cache->len <= cache->flushthresh);
> > > - if (likely(cache->len + n <= cache->flushthresh)) {
> > > + __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> > > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > + __rte_assume(cache->len <= cache->size);
> > > + if (likely(cache->len + n <= cache->size)) {
> > > /* Sufficient room in the cache for the objects. */
> > > cache_objs = &cache->objs[cache->len];
> > > cache->len += n;
> > > - } else if (n <= cache->flushthresh) {
> > > + } else if (n <= cache->size / 2) {
> > > /*
> > > - * The cache is big enough for the objects, but - as
> > detected by
> > > - * the comparison above - has insufficient room for them.
> > > - * Flush the cache to make room for the objects.
> > > + * The number of objects is within the cache bounce buffer
> > limit,
> > > + * but - as detected by the comparison above - the cache
> > has
> > > + * insufficient room for them.
> > > + * Flush the cache to the backend to make room for the
> > objects;
> > > + * flush (size / 2) objects from the bottom of the cache,
> > where
> > > + * objects are less hot, and move down the remaining
> > objects, which
> > > + * are more hot, from the upper half of the cache.
> > > */
> > > - cache_objs = &cache->objs[0];
> > > - rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
> > > - cache->len = n;
> > > + __rte_assume(cache->len > cache->size / 2);
> > > + rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0], cache-
> > >size / 2);
> > > + rte_memcpy(&cache->objs[0], &cache->objs[cache->size / 2],
> > > + sizeof(void *) * (cache->len - cache->size /
> > 2));
> > > + cache_objs = &cache->objs[cache->len - cache->size / 2];
> > > + cache->len = cache->len - cache->size / 2 + n;
> >
> > The flushing of only half the cache I'm not so certain about. I agree
> > that
> > we want to not flush to empty, but I also think that we want to do more
> > than a half-flush, especially since we do an enqueue to the cache
> > immediately afterwards. Consider the case where we have a cache size of
> > 128, and we do an enqueue of 32, with the cache currently full. In that
> > case we only flush 64, reducing the cache to 64, but then immediately
> > bringing it back up to 96.
>
> I thought in depth about whether the flush/replenish sizes should consider the request size or not. (E.g. if I should replenish size/2 or size/2+request.)
> I decided for not considering the request size, for two reasons:
> a) It roughly doesn't matter, especially when considering a sequence of random get/put requests.
> b) The size of the backend transactions become fixed, which has performance benefits: With my patch, they are always size/2, so if the cache size is 2^N, the backend transactions are 2^N and CPU cache aligned.
>
> > For cases where we have pipelines with all
> > alloc
> > on one core and all free on another, this half-flush would be
> > inefficient.
> >
> > Instead, I would look to have a lower target threshold post-flush, and
> > I
> > would suggest 25% - taking into account the newly freed buffers.
>
> It's not good for #1.
> I agree that it is better for #2. But I don't think #2 is the likely use case.
>
> After our discussion at the summit, I did start working a patch targeting fill levels at 25% from the cache edges, but I don't think it's better; so I'd rather stick with a target fill level of 50%.
>
> > For example:
> >
> > /* if n > our target of 1/4 full, flush everything,
> > * else flush so that we end up with 1/4 full after n added.
> > */
> > flush_count = n > cache->size/4 ? cache->len :
> > (cache->len + n) - cache->size/4;
> >
> >
> > > } else {
> > > - /* The request itself is too big for the cache. */
> > > + /* The request itself is too big. */
> > > goto driver_enqueue_stats_incremented;
> >
> > I think original comment is better. The request itself is not too big
> > for
> > the whole mempool, just for the cache.
>
> Ack.
>
> >
> > > }
> > >
> > > @@ -1524,7 +1539,7 @@ rte_mempool_do_generic_get(struct rte_mempool
> > *mp, void **obj_table,
> > > /* The cache is a stack, so copy will be in reverse order. */
> > > cache_objs = &cache->objs[cache->len];
> > >
> > > - __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
> > > + __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> > > if (likely(n <= cache->len)) {
> > > /* The entire request can be satisfied from the cache. */
> > > RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
> > > @@ -1548,13 +1563,13 @@ rte_mempool_do_generic_get(struct rte_mempool
> > *mp, void **obj_table,
> > > for (index = 0; index < len; index++)
> > > *obj_table++ = *--cache_objs;
> > >
> > > - /* Dequeue below would overflow mem allocated for cache? */
> > > - if (unlikely(remaining > RTE_MEMPOOL_CACHE_MAX_SIZE))
> > > + /* Dequeue below would exceed the cache bounce buffer limit? */
> > > + __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> > > + if (unlikely(remaining > cache->size / 2))
> > > goto driver_dequeue;
> > >
> > > - /* Fill the cache from the backend; fetch size + remaining
> > objects. */
> > > - ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs,
> > > - cache->size + remaining);
> > > + /* Fill the cache from the backend; fetch (size / 2) objects. */
> > > + ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache->size /
> > 2);
> >
> > Again, the cache->size / 2 doesn't seem right here. We at most half-
> > fill
> > the cache and then take some objects from that, meaning that have just
> > done
> > a re-fill of cache but end the function with it less than half full.
> > Since
> > we take from this value, I'd suggest just filling the cache completely.
>
> The issues at the edges of the cache are symmetrical.
> If we replenish the cache to full, and the next transaction is a put, the cache needs to be drained.
> That's why I replenish to size/2.
>
Not necessarily. After you fill the cache here (to 50%) you then take
elements from it. If we filled to 100%, then we'd only hit a flush if we
freed back more elements than we just allocated. Now, that's reasonably
likely which is why I'm ok to not filling to 100%. However, doing a refill
as part of the op, and then leaving the cache less than half full after the
op finishes is just wasteful IMHO!
One other point. The obvious worst case scenario we want to avoid is
constant fill/empty/fill/empty sequences. However, so long as we have an
asymmetry in how we do fills and empty thresholds, a repeating sequence
should never occur, and even pairs of fill/empty should be very rare.
Consider the case, for example, where we fill to 100% but only drain to
25%. For this, we can ignore scenario #2, since we should have pretty good
cache usage. For scenario #1, of allocs/frees on a single core, randomly
interleaved, our worst case is:
1) alloc with cache empty, in which case we fill to 100%, then take the
alloc
2) have a free of a burst greater than that which we just allocated,
causing an immediate flush again.
That's pretty poor behaviour, but the thing is that after the flush we now
have mempool at 25% + freed burst - so expected between 25-50% of cache
utilization. That's the sweet spot we want to target - after each
operation, the mempool cache should ideally be at 50%. Whether the next op
is an alloc or a free, our cache can handle it, and likely a couple of each
in sequence. Therefore, our possible fill/empty combinations are likely to
be rare occurances.
[In all this, I am making the assumption that burst size is well less than
cache size. Also, similar logic would be applicable for the inverse
scenario, e.g. flush to empty (and fill burst) and fill to 75%]
Now, all said, I tend to agree that we want to leave space for a decent
size burst after a fill. That is why I think that filling to 75% is
reasonable. After an alloc that triggers a fill, I don't want the cache
less than 50% full, but not completely full so there is room for a free
without a flush, and similarly for a free that triggers a flush, the cache
should not be empty, but also should not be more than half full.
One suggestion - we could always add a simple tunable that specifies the
margin, or reserved entries for alloc and free. We can then guide in the
docs that the value should be e.g. "zero for apps where alloc and free take
place on different cores. 20%-50% of cache is recommended where alloc and
free take place on the same core"
/Bruce
^ permalink raw reply
* Re: [PATCH v3 00/25] Consolidate bus driver infrastructure
From: David Marchand @ 2026-05-26 9:03 UTC (permalink / raw)
To: dev; +Cc: thomas, stephen, bruce.richardson
In-Reply-To: <20260526084212.3145685-1-david.marchand@redhat.com>
On Tue, 26 May 2026 at 10:42, David Marchand <david.marchand@redhat.com> wrote:
>
> This is a continuation of the work I started on the bus infrastructure,
> but this time, a lot of the changes were done by a AI "friend".
> It is still an unfinished topic as the current series focuses on probing
> only. The detaching/cleanup aspect is postponed to another release/time.
>
> My AI "friend" really *sucked* at git and at separating unrelated changes,
> so it required quite a lot of massage/polishing afterwards.
> But it seems good enough now for upstream submission.
>
> I would like to see this series merged in 26.07, so that we have enough
> time to stabilize it before the next LTS.
> And seeing how it affects drivers, it is probably better to merge it
> the sooner possible (so Thomas does not have to solve too many conflicts
> when pulling next-* subtrees after, especially wrt the last patch).
>
>
> This series refactors the DPDK bus infrastructure to consolidate common
> operations and reduce code duplication across all bus drivers.
> Currently, each bus implements its own specific device/driver lists,
> probe logic, and lookup functions.
> This series moves these common patterns into the EAL bus layer,
> providing generic helpers that all buses can use.
>
> The refactoring removes approximately 1,400 lines of duplicated code across
> the codebase while maintaining full functional equivalence.
>
> Key changes:
> - Factorize device and driver lists into struct rte_bus
> - Implement generic probe, device/driver lookup, and iteration helpers in EAL
> - Introduce conversion macros (RTE_BUS_DEVICE, RTE_BUS_DRIVER, RTE_CLASS_TO_BUS_DEVICE)
> to safely convert between generic and bus-specific types
> - Remove bus-specific device/driver types from most driver code
> - Move probe logic from individual buses to rte_bus_generic_probe()
> - Separate NXP-specific metadata from generic bus structures
>
> Benefits:
> - Significant code reduction (~1,400 lines removed)
> - Consistent behavior across all bus types
> - Simplified bus driver implementation
> - Easier maintenance and future enhancements
>
> The series is structured as a progressive refactoring:
> - Remove redundant checks and helpers (patches 1-5)
> - Add conversion macros and factorize lists (patches 6-8)
> - Consolidate device/driver lookup and iteration (patches 9-11)
> - Refactor probe logic (patches 12-15)
> - Remove bus-specific types from drivers (patches 16-23)
>
> Note on ABI:
> This series breaks the ABI for drivers (changes to rte_pci_device,
> rte_pci_driver, and similar structures for other buses). However, the DPDK
> ABI policy does not provide guarantees for driver-level interfaces.
>
>
> --
> David Marchand
>
> Changes since v2:
> - fixed dma/idxd probing as reported by Bruce,
> - moved api_ver setting (from match to scan) in bus/uacce,
> - fixed transient bug in the vdev code (in the middle of the series).
> tl;dr the high level difference for bus/vdev between v2 and v3 == 0,
> - fixed doxygen,
> - enhanced API description,
There was some hiccup from Red Hat SMTP when sending the series..
hopefully my resending of the second half of the patches is good
enough.
At least, patchwork looks happy.
--
David Marchand
^ permalink raw reply
* [RFC v2 3/3] app/test: add fastmem test suite
From: Mattias Rönnblom @ 2026-05-26 8:57 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Mattias Rönnblom
In-Reply-To: <20260526085743.64396-1-hofors@lysator.liu.se>
Add functional, performance, and profiling test suites for the
fastmem library.
--
Signed-off-by: Mattias Rönnblom <hofors@lysator.liu.se>
---
app/test/meson.build | 3 +
app/test/test_fastmem.c | 1673 +++++++++++++++++++++++++++++++
app/test/test_fastmem_perf.c | 1040 +++++++++++++++++++
app/test/test_fastmem_profile.c | 157 +++
4 files changed, 2873 insertions(+)
create mode 100644 app/test/test_fastmem.c
create mode 100644 app/test/test_fastmem_perf.c
create mode 100644 app/test/test_fastmem_profile.c
diff --git a/app/test/meson.build b/app/test/meson.build
index 7d458f9c07..d11c63be6f 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -82,6 +82,9 @@ source_file_deps = {
'test_event_vector_adapter.c': ['eventdev', 'bus_vdev'],
'test_eventdev.c': ['eventdev', 'bus_vdev'],
'test_external_mem.c': [],
+ 'test_fastmem.c': ['fastmem'],
+ 'test_fastmem_perf.c': ['fastmem', 'mempool'],
+ 'test_fastmem_profile.c': ['fastmem'],
'test_fbarray.c': [],
'test_fib.c': ['net', 'fib'],
'test_fib6.c': ['rib', 'fib'],
diff --git a/app/test/test_fastmem.c b/app/test/test_fastmem.c
new file mode 100644
index 0000000000..6981de28be
--- /dev/null
+++ b/app/test/test_fastmem.c
@@ -0,0 +1,1673 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#include <errno.h>
+#include <inttypes.h>
+#include <stdalign.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_lcore.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_thread.h>
+
+#include <rte_fastmem.h>
+
+#include "test.h"
+
+#define FASTMEM_MEMZONE_SIZE (128U << 20)
+
+/*
+ * Count memzones whose names begin with the fastmem prefix.
+ * Used to verify that rte_fastmem_reserve() really did reserve
+ * backing memzones.
+ */
+static int fastmem_memzone_count;
+
+static void
+count_fastmem_memzones_walk(const struct rte_memzone *mz, void *arg)
+{
+ RTE_SET_USED(arg);
+
+ if (strncmp(mz->name, "fastmem_", strlen("fastmem_")) == 0)
+ fastmem_memzone_count++;
+}
+
+static unsigned int
+count_fastmem_memzones(void)
+{
+ fastmem_memzone_count = 0;
+ rte_memzone_walk(count_fastmem_memzones_walk, NULL);
+ return fastmem_memzone_count;
+}
+
+static int
+test_init_deinit(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_init() failed: %d", rc);
+
+ rte_fastmem_deinit();
+
+ /* A subsequent init/deinit cycle must succeed. */
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, 0, "second rte_fastmem_init() failed: %d", rc);
+
+ rte_fastmem_deinit();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_init_is_not_idempotent(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_init() failed: %d", rc);
+
+ rc = rte_fastmem_init();
+ TEST_ASSERT_EQUAL(rc, -EBUSY,
+ "expected -EBUSY on re-init, got %d", rc);
+
+ rte_fastmem_deinit();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_deinit_without_init(void)
+{
+ /* Must be a no-op, not a crash. */
+ rte_fastmem_deinit();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_max_size(void)
+{
+ size_t max;
+
+ max = rte_fastmem_max_size();
+ TEST_ASSERT(max >= (1U << 20),
+ "max_size=%zu below required 1 MiB minimum", max);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_small(void)
+{
+ int socket_id;
+ unsigned int before, after;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ before = count_fastmem_memzones();
+
+ /*
+ * A small reserve request (1 byte) must result in exactly
+ * one memzone reservation: the internal rounding is to
+ * memzone granularity.
+ */
+ rc = rte_fastmem_reserve(1, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_reserve() failed: %d", rc);
+
+ after = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after - before, 1,
+ "expected 1 new memzone, got %u", after - before);
+
+ rte_fastmem_deinit();
+
+ /* After deinit the memzones must be released. */
+ TEST_ASSERT_EQUAL(count_fastmem_memzones(), 0,
+ "%u fastmem memzones leaked after deinit",
+ count_fastmem_memzones());
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_multiple_memzones(void)
+{
+ int socket_id;
+ unsigned int before, after;
+ size_t reserve_size;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ before = count_fastmem_memzones();
+
+ /*
+ * Request just over one memzone's worth; this must force
+ * a second memzone to be reserved.
+ */
+ reserve_size = FASTMEM_MEMZONE_SIZE + 1;
+ rc = rte_fastmem_reserve(reserve_size, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_reserve(%zu) failed: %d",
+ reserve_size, rc);
+
+ after = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after - before, 2,
+ "expected 2 new memzones for %zu-byte reserve, got %u",
+ reserve_size, after - before);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_cumulative(void)
+{
+ int socket_id;
+ unsigned int after_first, after_second;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ rc = rte_fastmem_reserve(FASTMEM_MEMZONE_SIZE, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "first reserve failed: %d", rc);
+
+ after_first = count_fastmem_memzones();
+
+ /*
+ * A second call requesting the same amount that's already
+ * reserved must not trigger any new memzone reservation.
+ */
+ rc = rte_fastmem_reserve(FASTMEM_MEMZONE_SIZE, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "second reserve failed: %d", rc);
+
+ after_second = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after_first, after_second,
+ "reserve of already-reserved amount added memzones (%u -> %u)",
+ after_first, after_second);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_invalid_socket(void)
+{
+ int rc;
+
+ rc = rte_fastmem_reserve(1, RTE_MAX_NUMA_NODES);
+ TEST_ASSERT_EQUAL(rc, -EINVAL,
+ "expected -EINVAL for out-of-range socket, got %d", rc);
+
+ rc = rte_fastmem_reserve(1, -2);
+ TEST_ASSERT_EQUAL(rc, -EINVAL,
+ "expected -EINVAL for negative socket, got %d", rc);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_without_init(void)
+{
+ int rc;
+
+ rc = rte_fastmem_reserve(1, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0,
+ "expected failure without init, got %d", rc);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_reserve_any_socket(void)
+{
+ unsigned int before, after;
+ int rc;
+
+ before = count_fastmem_memzones();
+
+ /*
+ * SOCKET_ID_ANY should succeed on any system with at least
+ * one configured socket. The allocator picks the caller's
+ * socket first and falls back to other sockets if needed.
+ */
+ rc = rte_fastmem_reserve(1, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0,
+ "rte_fastmem_reserve(SOCKET_ID_ANY) failed: %d", rc);
+
+ after = count_fastmem_memzones();
+ TEST_ASSERT_EQUAL(after - before, 1,
+ "expected 1 new memzone, got %u", after - before);
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Stage 2 tests: allocation and free.
+ */
+
+static int
+test_alloc_too_big(void)
+{
+ void *p;
+ rte_errno = 0;
+ p = rte_fastmem_alloc(rte_fastmem_max_size() + 1, 0, 0);
+ TEST_ASSERT_NULL(p, "alloc above max_size returned non-NULL");
+ TEST_ASSERT_EQUAL(rte_errno, E2BIG,
+ "expected rte_errno=E2BIG, got %d", rte_errno);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_invalid_align(void)
+{
+ void *p;
+ rte_errno = 0;
+ p = rte_fastmem_alloc(16, 3, 0); /* 3 is not a power of 2 */
+ TEST_ASSERT_NULL(p, "alloc with align=3 returned non-NULL");
+ TEST_ASSERT_EQUAL(rte_errno, EINVAL,
+ "expected rte_errno=EINVAL, got %d", rte_errno);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_free_small(void)
+{
+ void *p;
+ p = rte_fastmem_alloc(8, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc(8) failed: rte_errno=%d", rte_errno);
+
+ /* Writing into the object must not crash. */
+ memset(p, 0xa5, 8);
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_free_various_sizes(void)
+{
+ static const size_t sizes[] = {
+ 1, 8, 16, 17, 63, 64, 128, 1024, 4096,
+ 64 * 1024, 256 * 1024, 1024 * 1024,
+ };
+ void *ptrs[RTE_DIM(sizes)];
+ unsigned int i;
+ for (i = 0; i < RTE_DIM(sizes); i++) {
+ ptrs[i] = rte_fastmem_alloc(sizes[i], 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "alloc(%zu) failed: rte_errno=%d",
+ sizes[i], rte_errno);
+ memset(ptrs[i], 0x5a, sizes[i]);
+ }
+
+ for (i = 0; i < RTE_DIM(sizes); i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_alignment(void)
+{
+ static const size_t aligns[] = {
+ 8, 16, 64, 256, 4096, 65536,
+ };
+ unsigned int i;
+ for (i = 0; i < RTE_DIM(aligns); i++) {
+ void *p = rte_fastmem_alloc(1, aligns[i], 0);
+
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc(1, align=%zu) failed: rte_errno=%d",
+ aligns[i], rte_errno);
+ TEST_ASSERT((uintptr_t)p % aligns[i] == 0,
+ "pointer %p not aligned on %zu",
+ p, aligns[i]);
+ rte_fastmem_free(p);
+ }
+
+ /* Default (align=0) gives at least RTE_CACHE_LINE_SIZE. */
+ {
+ void *p = rte_fastmem_alloc(1, 0, 0);
+
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc(1, align=0) failed: rte_errno=%d", rte_errno);
+ TEST_ASSERT((uintptr_t)p % RTE_CACHE_LINE_SIZE == 0,
+ "default-align pointer %p not cache-line aligned",
+ p);
+ rte_fastmem_free(p);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_zero_flag(void)
+{
+ uint8_t *p;
+ unsigned int i;
+ bool all_zero = true;
+
+ /*
+ * Dirty a slab first by allocating without F_ZERO, writing
+ * a non-zero pattern, and freeing. A subsequent F_ZERO
+ * allocation on the same slab must return zeroed memory.
+ */
+ p = rte_fastmem_alloc(128, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "priming alloc failed");
+ memset(p, 0xff, 128);
+ rte_fastmem_free(p);
+
+ p = rte_fastmem_alloc(128, 0, RTE_FASTMEM_F_ZERO);
+ TEST_ASSERT_NOT_NULL(p, "F_ZERO alloc failed");
+ for (i = 0; i < 128; i++) {
+ if (p[i] != 0) {
+ all_zero = false;
+ break;
+ }
+ }
+ TEST_ASSERT(all_zero, "F_ZERO returned non-zero byte at offset %u", i);
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_reuse(void)
+{
+ void *first, *second;
+ first = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(first, "first alloc failed");
+ rte_fastmem_free(first);
+
+ second = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(second, "second alloc failed");
+
+ /*
+ * The slab's free list is LIFO, so the most recently freed
+ * object is at the head of the list. A subsequent alloc in
+ * the same class returns it.
+ */
+ TEST_ASSERT_EQUAL(first, second,
+ "free + alloc did not reuse: first=%p second=%p",
+ first, second);
+
+ rte_fastmem_free(second);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_many_in_class(void)
+{
+ /*
+ * Allocate more objects in one class than fit in a single
+ * slab, forcing the bin to pull a second block. This
+ * exercises the partial->full transition and the cross-slab
+ * allocation path.
+ */
+ enum { CLASS_SIZE = 8, COUNT = 300000 };
+ void **ptrs;
+ unsigned int i;
+
+ ptrs = calloc(COUNT, sizeof(*ptrs));
+ TEST_ASSERT_NOT_NULL(ptrs, "calloc for test ptrs failed");
+
+ for (i = 0; i < COUNT; i++) {
+ ptrs[i] = rte_fastmem_alloc(CLASS_SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "alloc[%u] failed: rte_errno=%d",
+ i, rte_errno);
+ }
+
+ for (i = 0; i < COUNT; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ free(ptrs);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_socket(void)
+{
+ void *p;
+ int socket_id;
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ p = rte_fastmem_alloc_socket(64, 0, 0, socket_id);
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc_socket(%d) failed: rte_errno=%d",
+ socket_id, rte_errno);
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_block_repurposing(void)
+{
+ void *small, *large;
+
+ /*
+ * Allocate and free a small object, forcing a block to be
+ * assigned to the small class and then returned to the
+ * free-block pool. A subsequent allocation in a different
+ * class must be able to reuse that block.
+ */
+ small = rte_fastmem_alloc(8, 0, 0);
+ TEST_ASSERT_NOT_NULL(small, "small alloc failed");
+ rte_fastmem_free(small);
+
+ large = rte_fastmem_alloc(256 * 1024, 0, 0);
+ TEST_ASSERT_NOT_NULL(large, "large alloc failed");
+ rte_fastmem_free(large);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_block_repurposing_no_growth(void)
+{
+ struct rte_fastmem_stats stats;
+ void *small, *large;
+ uint64_t after_small;
+ int rc;
+
+ /*
+ * Stronger version of test_alloc_block_repurposing: assert
+ * that the cross-class allocation does not grow the
+ * backing memory (bytes_backing stays flat). Because the
+ * free-block pool is shared across size classes — not
+ * partitioned per class — the block freed from the small
+ * class must serve the large allocation without triggering
+ * a new memzone reservation.
+ */
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_stats() failed: %d", rc);
+ TEST_ASSERT_EQUAL(stats.bytes_backing, (uint64_t)0,
+ "unexpected pre-alloc bytes_backing: %" PRIu64,
+ stats.bytes_backing);
+
+ small = rte_fastmem_alloc(8, 0, 0);
+ TEST_ASSERT_NOT_NULL(small, "small alloc failed");
+
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_stats() failed: %d", rc);
+ TEST_ASSERT(stats.bytes_backing > 0,
+ "bytes_backing did not grow on first alloc");
+ after_small = stats.bytes_backing;
+
+ rte_fastmem_free(small);
+ rte_fastmem_cache_flush();
+
+ large = rte_fastmem_alloc(256 * 1024, 0, 0);
+ TEST_ASSERT_NOT_NULL(large,
+ "large alloc failed: rte_errno=%d", rte_errno);
+
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_fastmem_stats() failed: %d", rc);
+ TEST_ASSERT_EQUAL(stats.bytes_backing, after_small,
+ "cross-class alloc grew backing memory from %" PRIu64
+ " to %" PRIu64,
+ after_small, stats.bytes_backing);
+
+ rte_fastmem_free(large);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_free_null(void)
+{
+ /* Must be a no-op, not a crash. */
+ rte_fastmem_free(NULL);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_content_integrity(void)
+{
+ /*
+ * Allocate a batch of objects, fill each with a distinct
+ * byte pattern, then verify none of the patterns overlap.
+ * This catches header overwrites (slab header corrupted by
+ * object access) and slot-overlap bugs (two pointers pointing
+ * at overlapping slots).
+ */
+ enum { N = 256, SIZE = 128 };
+ uint8_t *ptrs[N];
+ unsigned int i, j;
+ for (i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ memset(ptrs[i], (int)i, SIZE);
+ }
+
+ for (i = 0; i < N; i++)
+ for (j = 0; j < SIZE; j++)
+ TEST_ASSERT_EQUAL(ptrs[i][j], (uint8_t)i,
+ "corruption at ptrs[%u][%u]: got 0x%x, want 0x%x",
+ i, j, ptrs[i][j], (uint8_t)i);
+
+ for (i = 0; i < N; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_align_too_big(void)
+{
+ void *p;
+ /*
+ * A small size with an alignment larger than the maximum
+ * size class cannot be served. The class selected must be
+ * large enough for the alignment, but no such class exists.
+ */
+ rte_errno = 0;
+ p = rte_fastmem_alloc(1, rte_fastmem_max_size() * 2, 0);
+ TEST_ASSERT_NULL(p,
+ "alloc with align>max_size returned non-NULL");
+ TEST_ASSERT_EQUAL(rte_errno, E2BIG,
+ "expected rte_errno=E2BIG, got %d", rte_errno);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_align_one(void)
+{
+ void *p;
+ /* align=1 is a valid power of 2 and must be accepted. */
+ p = rte_fastmem_alloc(8, 1, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc(8, 1) failed: rte_errno=%d",
+ rte_errno);
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_socket_numa_placement(void)
+{
+ void *p;
+ int socket_id;
+ struct rte_memseg *ms;
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no available sockets");
+
+ p = rte_fastmem_alloc_socket(64, 0, 0, socket_id);
+ TEST_ASSERT_NOT_NULL(p,
+ "alloc_socket(%d) failed: rte_errno=%d",
+ socket_id, rte_errno);
+
+ /*
+ * Walk the memory to find the memseg for this pointer and
+ * verify its socket. Skip the check if lookup fails (e.g.,
+ * --no-huge mode may not populate memsegs for fastmem's
+ * allocations in a way that rte_mem_virt2memseg can find).
+ */
+ ms = rte_mem_virt2memseg(p, NULL);
+ if (ms != NULL) {
+ TEST_ASSERT_EQUAL(ms->socket_id, socket_id,
+ "alloc on socket %d landed on socket %d",
+ socket_id, ms->socket_id);
+ }
+
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Allocate from a socket different from the calling lcore's socket,
+ * triggering a cross-socket cache allocation. Then deinit to exercise
+ * the teardown path where a cache's backing memory lives on a
+ * different socket than the one it serves.
+ */
+static int
+test_alloc_cross_socket_deinit(void)
+{
+ int local_sid, remote_sid;
+ unsigned int i, n_sockets;
+ void *p;
+
+ local_sid = (int)rte_socket_id();
+ if (local_sid < 0 || (unsigned int)local_sid >= RTE_MAX_NUMA_NODES)
+ local_sid = rte_socket_id_by_idx(0);
+
+ n_sockets = rte_socket_count();
+ if (n_sockets < 2)
+ return TEST_SKIPPED;
+
+ /* Find a socket different from the local one. */
+ remote_sid = -1;
+ for (i = 0; i < n_sockets; i++) {
+ int sid = rte_socket_id_by_idx(i);
+ if (sid >= 0 && sid != local_sid) {
+ remote_sid = sid;
+ break;
+ }
+ }
+ if (remote_sid < 0)
+ return TEST_SKIPPED;
+
+ p = rte_fastmem_alloc_socket(64, 0, 0, remote_sid);
+ TEST_ASSERT_NOT_NULL(p,
+ "cross-socket alloc(socket %d) failed: rte_errno=%d",
+ remote_sid, rte_errno);
+
+ rte_fastmem_free(p);
+
+ /* Teardown and re-init to exercise the deinit path with
+ * cross-socket caches.
+ */
+ rte_fastmem_deinit();
+
+ TEST_ASSERT_EQUAL(rte_fastmem_init(), 0,
+ "re-init after cross-socket deinit failed");
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Stage 3 tests: per-lcore caches.
+ */
+
+static int
+test_cache_flush(void)
+{
+ void *p;
+ /*
+ * Alloc and free one object, leaving it in the cache. Then
+ * flush and verify that a subsequent alloc may or may not
+ * return the same pointer (not asserting same/different —
+ * just checking that flush does not crash and a follow-up
+ * alloc still works).
+ */
+ p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "first alloc failed");
+ rte_fastmem_free(p);
+
+ rte_fastmem_cache_flush();
+
+ /* Flush again — must be idempotent. */
+ rte_fastmem_cache_flush();
+
+ p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "post-flush alloc failed");
+ rte_fastmem_free(p);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_cache_flush_without_init(void)
+{
+ /* Must be a no-op, not a crash. */
+ rte_fastmem_cache_flush();
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_cache_exceeds_capacity(void)
+{
+ /*
+ * Free more objects at a single size class than the cache
+ * capacity (64 for classes <= 4 KiB). This forces the
+ * cache-drain slow path and verifies no corruption.
+ */
+ enum { COUNT = 200, SIZE = 64 };
+ void *ptrs[COUNT];
+ unsigned int i;
+
+ for (i = 0; i < COUNT; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "alloc[%u] failed: rte_errno=%d", i, rte_errno);
+ }
+
+ for (i = 0; i < COUNT; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ /* Re-alloc the same count should still work. */
+ for (i = 0; i < COUNT; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i],
+ "re-alloc[%u] failed: rte_errno=%d", i, rte_errno);
+ }
+
+ for (i = 0; i < COUNT; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+struct non_eal_args {
+ int ok;
+ char pad[64];
+};
+
+static uint32_t
+non_eal_thread_main(void *arg)
+{
+ struct non_eal_args *args = arg;
+ uint8_t *p;
+
+ p = rte_fastmem_alloc(128, 0, 0);
+ if (p == NULL)
+ return 1;
+
+ memset(p, 0x7e, 128);
+
+ rte_fastmem_free(p);
+
+ args->ok = 1;
+ return 0;
+}
+
+static int
+test_non_eal_thread(void)
+{
+ rte_thread_t thread_id;
+ struct non_eal_args args = { 0 };
+ int rc;
+
+ rc = rte_thread_create(&thread_id, NULL, non_eal_thread_main, &args);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_thread_create() failed: %d", rc);
+
+ rc = rte_thread_join(thread_id, NULL);
+ TEST_ASSERT_EQUAL(rc, 0, "rte_thread_join() failed: %d", rc);
+
+ TEST_ASSERT_EQUAL(args.ok, 1,
+ "non-EAL thread did not complete alloc/free successfully");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_cache_flush_returns_memory(void)
+{
+ /*
+ * When an entire slab's worth of objects is freed, the
+ * slab's block is returned to the free-block pool and can
+ * be reassigned to another size class. Verify the cache
+ * does not permanently hold objects that prevent this.
+ *
+ * Allocate enough objects in one class to force multiple
+ * slabs, free them all, then flush the cache. After the
+ * flush, all cached objects are drained to their bins and
+ * empty slabs are returned to the block pool.
+ */
+ enum { N = 200, SIZE = 64 };
+ void *ptrs[N];
+ unsigned int i;
+
+ for (i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ }
+ for (i = 0; i < N; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ rte_fastmem_cache_flush();
+
+ /*
+ * An allocation in a completely different class should
+ * succeed now, having access to any blocks freed by the
+ * flush.
+ */
+ {
+ void *other = rte_fastmem_alloc(65536, 0, 0);
+
+ TEST_ASSERT_NOT_NULL(other,
+ "post-flush cross-class alloc failed");
+ rte_fastmem_free(other);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_basic(void)
+{
+ enum { N = 32 };
+ void *ptrs[N];
+ int rc;
+
+ rc = rte_fastmem_alloc_bulk(ptrs, N, 64, 0, 0);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk failed: %d", rc);
+
+ /* Verify all pointers are non-NULL and distinct. */
+ for (unsigned int i = 0; i < N; i++) {
+ TEST_ASSERT_NOT_NULL(ptrs[i], "ptrs[%u] is NULL", i);
+ for (unsigned int j = 0; j < i; j++)
+ TEST_ASSERT(ptrs[i] != ptrs[j],
+ "ptrs[%u] == ptrs[%u]", i, j);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_zero_flag(void)
+{
+ enum { N = 8, SIZE = 128 };
+ void *ptrs[N];
+ int rc;
+
+ rc = rte_fastmem_alloc_bulk(ptrs, N, SIZE, 0, RTE_FASTMEM_F_ZERO);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk failed: %d", rc);
+
+ for (unsigned int i = 0; i < N; i++) {
+ uint8_t *p = ptrs[i];
+
+ for (unsigned int b = 0; b < SIZE; b++)
+ TEST_ASSERT_EQUAL(p[b], 0,
+ "ptrs[%u][%u] != 0", i, b);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_exceeds_cache(void)
+{
+ /* Allocate more than cache capacity (64) in one bulk call. */
+ enum { N = 128 };
+ void *ptrs[N];
+ int rc;
+
+ rc = rte_fastmem_alloc_bulk(ptrs, N, 64, 0, 0);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk(%u) failed: %d", N, rc);
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_alloc_bulk_socket(void)
+{
+ enum { N = 16 };
+ void *ptrs[N];
+ int socket_id;
+ int rc;
+
+ socket_id = rte_socket_id_by_idx(0);
+ TEST_ASSERT(socket_id >= 0, "no sockets");
+
+ rc = rte_fastmem_alloc_bulk_socket(ptrs, N, 64, 0, 0, socket_id);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk_socket failed: %d", rc);
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ /* SOCKET_ID_ANY */
+ rc = rte_fastmem_alloc_bulk_socket(ptrs, N, 64, 0, 0, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "alloc_bulk_socket(ANY) failed: %d", rc);
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_free_bulk(void)
+{
+ enum { N = 64 };
+ void *ptrs[N];
+ /* Allocate individually, free in bulk. */
+ for (unsigned int i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ /* Verify memory is reusable. */
+ for (unsigned int i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "re-alloc[%u] failed", i);
+ }
+
+ rte_fastmem_free_bulk(ptrs, N);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_classes(void)
+{
+ size_t sizes[32];
+ unsigned int n;
+
+ n = rte_fastmem_classes(NULL);
+ TEST_ASSERT_EQUAL(n, 18u, "expected 18 classes, got %u", n);
+
+ n = rte_fastmem_classes(sizes);
+ TEST_ASSERT_EQUAL(n, 18u, "expected 18 classes, got %u", n);
+ TEST_ASSERT_EQUAL(sizes[0], (size_t)8, "class 0 != 8");
+ TEST_ASSERT_EQUAL(sizes[n - 1], (size_t)(1 << 20),
+ "last class != 1 MiB");
+
+ for (unsigned int i = 0; i < n; i++) {
+ TEST_ASSERT(sizes[i] != 0 && (sizes[i] & (sizes[i] - 1)) == 0,
+ "class %u size %zu not power of 2", i, sizes[i]);
+ if (i > 0)
+ TEST_ASSERT(sizes[i] > sizes[i - 1],
+ "classes not ascending at %u", i);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_class(void)
+{
+ enum { N = 10 };
+ struct rte_fastmem_class_stats cs;
+ void *ptrs[N];
+ int rc;
+
+ for (unsigned int i = 0; i < N; i++) {
+ ptrs[i] = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ }
+
+ rc = rte_fastmem_stats_class(64, &cs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_class failed: %d", rc);
+ TEST_ASSERT_EQUAL(cs.class_size, (size_t)64, "wrong class_size");
+ TEST_ASSERT(cs.alloc_cache_hits + cs.alloc_cache_misses == N,
+ "alloc count != N: hits=%" PRIu64 " misses=%" PRIu64,
+ cs.alloc_cache_hits, cs.alloc_cache_misses);
+ TEST_ASSERT_EQUAL(cs.in_use, (uint64_t)N, "in_use != N");
+
+ for (unsigned int i = 0; i < N; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ rc = rte_fastmem_stats_class(64, &cs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_class after free failed: %d", rc);
+ TEST_ASSERT_EQUAL(cs.in_use, (uint64_t)0, "in_use != 0 after free");
+
+ /* Invalid class size. */
+ rc = rte_fastmem_stats_class(13, &cs);
+ TEST_ASSERT_EQUAL(rc, -EINVAL, "expected -EINVAL for bad size");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_lcore(void)
+{
+ struct rte_fastmem_lcore_stats ls;
+ void *ptr;
+ int rc;
+
+ ptr = rte_fastmem_alloc(128, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ rc = rte_fastmem_stats_lcore(rte_lcore_id(), &ls);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_lcore failed: %d", rc);
+ TEST_ASSERT(ls.alloc_cache_hits + ls.alloc_cache_misses > 0,
+ "no alloc activity on this lcore");
+
+ rte_fastmem_free(ptr);
+
+ rc = rte_fastmem_stats_lcore(rte_lcore_id(), &ls);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_lcore after free failed: %d", rc);
+ TEST_ASSERT(ls.free_cache_hits + ls.free_cache_misses > 0,
+ "no free activity on this lcore");
+
+ /* Invalid lcore. */
+ rc = rte_fastmem_stats_lcore(RTE_MAX_LCORE, &ls);
+ TEST_ASSERT_EQUAL(rc, -EINVAL, "expected -EINVAL for bad lcore");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_lcore_class(void)
+{
+ struct rte_fastmem_lcore_class_stats lcs;
+ void *ptr;
+ int rc;
+
+ ptr = rte_fastmem_alloc(256, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+
+ rc = rte_fastmem_stats_lcore_class(rte_lcore_id(), 256, &lcs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats_lcore_class failed: %d", rc);
+ TEST_ASSERT_EQUAL(lcs.class_size, (size_t)256, "wrong class_size");
+ TEST_ASSERT(lcs.alloc_cache_hits + lcs.alloc_cache_misses > 0,
+ "no alloc activity");
+
+ rte_fastmem_free(ptr);
+ return TEST_SUCCESS;
+}
+
+static int
+test_stats_reset(void)
+{
+ struct rte_fastmem_stats gs;
+ void *ptr;
+ int rc;
+
+ ptr = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptr, "alloc failed");
+ rte_fastmem_free(ptr);
+
+ rte_fastmem_stats_reset();
+
+ rc = rte_fastmem_stats(&gs);
+ TEST_ASSERT_EQUAL(rc, 0, "stats failed: %d", rc);
+ TEST_ASSERT_EQUAL(gs.alloc_total, (uint64_t)0,
+ "alloc_total not zero after reset");
+ TEST_ASSERT_EQUAL(gs.free_total, (uint64_t)0,
+ "free_total not zero after reset");
+
+ return TEST_SUCCESS;
+}
+
+
+#define MIXED_LONG_LIVED_COUNT 25
+#define MIXED_SHORT_LIVED_ITERS 1000
+#define MIXED_MIN_LCORES 3
+
+static const size_t mixed_long_sizes[] = { 64, 256, 4096 };
+static const size_t mixed_short_sizes[] = { 8, 16, 32, 64, 128, 256, 512, 1024 };
+
+struct mixed_worker_args {
+ uint32_t seed;
+ int result;
+};
+
+static uint32_t
+xorshift32(uint32_t *state)
+{
+ uint32_t x = *state;
+
+ x ^= x << 13;
+ x ^= x >> 17;
+ x ^= x << 5;
+ *state = x;
+ return x;
+}
+
+static int
+mixed_worker(void *arg)
+{
+ struct mixed_worker_args *args = arg;
+ uint32_t seed = args->seed;
+ void *long_lived[MIXED_LONG_LIVED_COUNT];
+ size_t long_sizes[MIXED_LONG_LIVED_COUNT];
+ unsigned int i;
+
+ /* Allocate long-lived objects of mixed sizes. */
+ for (i = 0; i < MIXED_LONG_LIVED_COUNT; i++) {
+ long_sizes[i] = mixed_long_sizes[i % RTE_DIM(mixed_long_sizes)];
+ long_lived[i] = rte_fastmem_alloc(long_sizes[i], 0, 0);
+ if (long_lived[i] == NULL) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ memset(long_lived[i], (int)(i + 1), long_sizes[i]);
+ }
+
+ /* Rapidly cycle short-lived objects. */
+ for (i = 0; i < MIXED_SHORT_LIVED_ITERS; i++) {
+ size_t sz = mixed_short_sizes[xorshift32(&seed) %
+ RTE_DIM(mixed_short_sizes)];
+ uint8_t pattern = (uint8_t)(i & 0xff);
+ uint8_t *p;
+
+ p = rte_fastmem_alloc(sz, 0, 0);
+ if (p == NULL) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ memset(p, pattern, sz);
+
+ /* Verify before freeing. */
+ for (size_t j = 0; j < sz; j++) {
+ if (p[j] != pattern) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ }
+ rte_fastmem_free(p);
+ }
+
+ /* Verify long-lived objects are still intact. */
+ for (i = 0; i < MIXED_LONG_LIVED_COUNT; i++) {
+ uint8_t *bytes = long_lived[i];
+ uint8_t expected = (uint8_t)(i + 1);
+
+ for (size_t j = 0; j < long_sizes[i]; j++) {
+ if (bytes[j] != expected) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ }
+ rte_fastmem_free(long_lived[i]);
+ }
+
+ args->result = TEST_SUCCESS;
+ return 0;
+}
+
+static int
+test_mixed_lifetimes_multi_lcore(void)
+{
+ struct mixed_worker_args args[RTE_MAX_LCORE];
+ unsigned int lcore_id;
+ unsigned int count = 0;
+ struct rte_fastmem_stats stats;
+ int rc;
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id)
+ count++;
+
+ if (count < MIXED_MIN_LCORES) {
+ printf("Not enough worker lcores (%u < %u), skipping\n",
+ count, MIXED_MIN_LCORES);
+ return TEST_SKIPPED;
+ }
+
+ /* Launch workers with distinct seeds. */
+ uint32_t seed = 0xdeadbeef;
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ args[lcore_id].seed = seed;
+ args[lcore_id].result = TEST_FAILED;
+ seed += 0x12345678;
+ rte_eal_remote_launch(mixed_worker, &args[lcore_id], lcore_id);
+ }
+
+ rte_eal_mp_wait_lcore();
+
+ /* Check all workers succeeded. */
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ TEST_ASSERT_EQUAL(args[lcore_id].result, TEST_SUCCESS,
+ "worker on lcore %u failed", lcore_id);
+ }
+
+ /* Verify no memory leak. */
+ rc = rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(rc, 0, "stats failed: %d", rc);
+ TEST_ASSERT_EQUAL(stats.bytes_in_use, (uint64_t)0,
+ "bytes_in_use not zero after test: %" PRIu64,
+ stats.bytes_in_use);
+
+ return TEST_SUCCESS;
+}
+
+
+/*
+ * Memory limit tests.
+ *
+ * FASTMEM_MEMZONE_SIZE is 128 MiB. We use a limit of 128 MiB
+ * (one memzone) for most tests, and large objects (256 KiB) to
+ * exhaust slabs quickly.
+ */
+
+#define LIMIT_ONE_MZ ((size_t)128 << 20)
+#define LIMIT_OBJ_SIZE ((size_t)256 * 1024)
+
+static int
+test_memory_limit_basic(void)
+{
+ int rc;
+
+ rc = rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+ TEST_ASSERT_EQUAL(rc, 0, "set_memory_limit failed: %d", rc);
+
+ const size_t got = rte_fastmem_get_limit(0);
+ TEST_ASSERT_EQUAL(got, LIMIT_ONE_MZ,
+ "get_memory_limit mismatch: %zu", got);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "first reserve failed: %d", rc);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ + 1, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0, "second reserve should have failed");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_alloc_exhaustion(void)
+{
+ const unsigned int max_ptrs = 1024;
+ void *ptrs[max_ptrs];
+ unsigned int count = 0;
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (count = 0; count < max_ptrs; count++) {
+ ptrs[count] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (ptrs[count] == NULL)
+ break;
+ }
+
+ TEST_ASSERT(count > 0, "should have allocated at least one");
+ TEST_ASSERT(count < max_ptrs, "should have hit the limit");
+ TEST_ASSERT_EQUAL(rte_errno, ENOMEM, "expected ENOMEM, got %d", rte_errno);
+
+ rte_fastmem_free(ptrs[count - 1]);
+ void *p = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc after free should succeed");
+ rte_fastmem_free(p);
+
+ for (unsigned int i = 0; i < count - 1; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_zero_blocks_growth(void)
+{
+ int rc;
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 0);
+
+ rc = rte_fastmem_reserve(1, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0, "reserve with limit=0 should fail");
+
+ void *p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NULL(p, "alloc with limit=0 should fail");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_below_current(void)
+{
+ int rc;
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "reserve failed: %d", rc);
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 1);
+
+ void *p = rte_fastmem_alloc(64, 0, 0);
+ TEST_ASSERT_NOT_NULL(p, "alloc from existing backing should work");
+ rte_fastmem_free(p);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ * 2, SOCKET_ID_ANY);
+ TEST_ASSERT(rc < 0, "growth beyond limit should fail");
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_socket_id_any(void)
+{
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 42);
+
+ for (unsigned int i = 0; i < rte_socket_count(); i++) {
+ const int sid = rte_socket_id_by_idx(i);
+ const size_t lim = rte_fastmem_get_limit(sid);
+
+ TEST_ASSERT_EQUAL(lim, (size_t)42,
+ "socket %d limit mismatch: %zu", sid, lim);
+ }
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_unlimited(void)
+{
+ int rc;
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, 0);
+ rte_fastmem_set_limit(SOCKET_ID_ANY, SIZE_MAX);
+
+ rc = rte_fastmem_reserve(LIMIT_ONE_MZ, SOCKET_ID_ANY);
+ TEST_ASSERT_EQUAL(rc, 0, "reserve after reset failed: %d", rc);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_alloc_integrity_under_oom(void)
+{
+ const unsigned int n = 128;
+ const size_t obj_size = 1024;
+ uint8_t *ptrs[n];
+ const unsigned int extra_max = 1024;
+ void *extra[extra_max];
+ unsigned int n_extra = 0;
+ unsigned int i;
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (i = 0; i < n; i++) {
+ ptrs[i] = rte_fastmem_alloc(obj_size, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "alloc[%u] failed", i);
+ memset(ptrs[i], (int)(i & 0xff), obj_size);
+ }
+
+ /* Exhaust remaining backing with large objects. */
+ for (n_extra = 0; n_extra < extra_max; n_extra++) {
+ extra[n_extra] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (extra[n_extra] == NULL)
+ break;
+ }
+
+ /* Verify original objects are intact. */
+ for (i = 0; i < n; i++) {
+ const uint8_t expected = (uint8_t)(i & 0xff);
+ for (unsigned int j = 0; j < obj_size; j++)
+ TEST_ASSERT_EQUAL(ptrs[i][j], expected,
+ "corruption at [%u][%u]", i, j);
+ }
+
+ for (i = 0; i < n; i++)
+ rte_fastmem_free(ptrs[i]);
+ for (i = 0; i < n_extra; i++)
+ rte_fastmem_free(extra[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_bulk_alloc_oom(void)
+{
+ const unsigned int bulk_n = 64;
+ const unsigned int drain_max = 512;
+ void *ptrs[bulk_n];
+ void *drain[drain_max];
+ unsigned int drained = 0;
+ int rc;
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (drained = 0; drained < drain_max; drained++) {
+ drain[drained] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (drain[drained] == NULL)
+ break;
+ }
+
+ /* Free a few — enough for some but not bulk_n objects. */
+ const unsigned int freed = RTE_MIN(drained, 4u);
+ for (unsigned int i = 0; i < freed; i++)
+ rte_fastmem_free(drain[--drained]);
+
+ rc = rte_fastmem_alloc_bulk(ptrs, bulk_n, LIMIT_OBJ_SIZE, 0, 0);
+ TEST_ASSERT(rc < 0, "bulk alloc should fail");
+
+ for (unsigned int i = 0; i < drained; i++)
+ rte_fastmem_free(drain[i]);
+
+ return TEST_SUCCESS;
+}
+
+static int
+test_memory_limit_recovery_after_free(void)
+{
+ const unsigned int max_ptrs = 512;
+ void *ptrs[max_ptrs];
+ unsigned int count = 0;
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ for (count = 0; count < max_ptrs; count++) {
+ ptrs[count] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (ptrs[count] == NULL)
+ break;
+ }
+ TEST_ASSERT(count > 0 && count < max_ptrs,
+ "expected partial fill, got %u", count);
+
+ const unsigned int half = count / 2;
+ for (unsigned int i = 0; i < half; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ for (unsigned int i = 0; i < half; i++) {
+ ptrs[i] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ TEST_ASSERT_NOT_NULL(ptrs[i], "recovery alloc[%u] failed", i);
+ }
+
+ for (unsigned int i = 0; i < count; i++)
+ rte_fastmem_free(ptrs[i]);
+
+ return TEST_SUCCESS;
+}
+
+struct limit_worker_args {
+ unsigned int alloc_count;
+ int result;
+};
+
+static int
+limit_worker(void *arg)
+{
+ struct limit_worker_args *args = arg;
+ const unsigned int max_ptrs = 128;
+ void *ptrs[max_ptrs];
+ unsigned int i;
+
+ args->alloc_count = 0;
+
+ for (i = 0; i < max_ptrs; i++) {
+ ptrs[i] = rte_fastmem_alloc(LIMIT_OBJ_SIZE, 0, 0);
+ if (ptrs[i] == NULL)
+ break;
+ memset(ptrs[i], 0xab, LIMIT_OBJ_SIZE);
+ args->alloc_count++;
+ }
+
+ for (unsigned int j = 0; j < args->alloc_count; j++) {
+ uint8_t *bytes = ptrs[j];
+ for (size_t k = 0; k < LIMIT_OBJ_SIZE; k++) {
+ if (bytes[k] != 0xab) {
+ args->result = TEST_FAILED;
+ return -1;
+ }
+ }
+ rte_fastmem_free(ptrs[j]);
+ }
+
+ args->result = TEST_SUCCESS;
+ return 0;
+}
+
+static int
+test_memory_limit_multi_lcore_oom(void)
+{
+ struct limit_worker_args args[RTE_MAX_LCORE];
+ unsigned int lcore_id;
+ unsigned int worker_count = 0;
+ RTE_LCORE_FOREACH_WORKER(lcore_id)
+ worker_count++;
+
+ if (worker_count < 2) {
+ printf("Not enough workers (%u < 2), skipping\n", worker_count);
+ return TEST_SKIPPED;
+ }
+
+ rte_fastmem_set_limit(SOCKET_ID_ANY, LIMIT_ONE_MZ);
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ args[lcore_id].result = TEST_FAILED;
+ rte_eal_remote_launch(limit_worker, &args[lcore_id], lcore_id);
+ }
+
+ rte_eal_mp_wait_lcore();
+
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ TEST_ASSERT_EQUAL(args[lcore_id].result, TEST_SUCCESS,
+ "worker on lcore %u failed", lcore_id);
+ }
+
+ struct rte_fastmem_stats stats;
+ rte_fastmem_stats(&stats);
+ TEST_ASSERT_EQUAL(stats.bytes_in_use, (uint64_t)0,
+ "bytes_in_use not zero: %" PRIu64, stats.bytes_in_use);
+
+ return TEST_SUCCESS;
+}
+
+static int
+fastmem_setup(void)
+{
+ return rte_fastmem_init();
+}
+
+static void
+fastmem_teardown(void)
+{
+ rte_fastmem_deinit();
+}
+
+static struct unit_test_suite fastmem_lifecycle_testsuite = {
+ .suite_name = "fastmem lifecycle tests",
+ .setup = NULL,
+ .teardown = NULL,
+ .unit_test_cases = {
+ TEST_CASE(test_init_deinit),
+ TEST_CASE(test_init_is_not_idempotent),
+ TEST_CASE(test_deinit_without_init),
+ TEST_CASE(test_max_size),
+ TEST_CASE(test_reserve_without_init),
+ TEST_CASE(test_cache_flush_without_init),
+ TEST_CASE(test_classes),
+ TEST_CASES_END()
+ }
+};
+
+static struct unit_test_suite fastmem_functional_testsuite = {
+ .suite_name = "fastmem functional tests",
+ .setup = NULL,
+ .teardown = NULL,
+ .unit_test_cases = {
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_small),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_multiple_memzones),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_cumulative),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_invalid_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_reserve_any_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_too_big),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_invalid_align),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_free_small),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_free_various_sizes),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_alignment),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_zero_flag),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_reuse),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_many_in_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_block_repurposing),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_block_repurposing_no_growth),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_free_null),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_content_integrity),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_align_too_big),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_align_one),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_socket_numa_placement),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_cross_socket_deinit),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_cache_flush),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_cache_exceeds_capacity),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_non_eal_thread),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_cache_flush_returns_memory),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_basic),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_zero_flag),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_exceeds_cache),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_alloc_bulk_socket),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_free_bulk),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_lcore),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_lcore_class),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_stats_reset),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_mixed_lifetimes_multi_lcore),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_basic),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_alloc_exhaustion),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_zero_blocks_growth),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_below_current),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_socket_id_any),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_unlimited),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_alloc_integrity_under_oom),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_bulk_alloc_oom),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_recovery_after_free),
+ TEST_CASE_ST(fastmem_setup, fastmem_teardown,
+ test_memory_limit_multi_lcore_oom),
+ TEST_CASES_END()
+ }
+};
+
+static int
+test_fastmem(void)
+{
+ int rc;
+
+ rc = unit_test_suite_runner(&fastmem_lifecycle_testsuite);
+ if (rc != 0)
+ return rc;
+
+ return unit_test_suite_runner(&fastmem_functional_testsuite);
+}
+
+REGISTER_FAST_TEST(fastmem_autotest, NOHUGE_SKIP, ASAN_OK, test_fastmem);
diff --git a/app/test/test_fastmem_perf.c b/app/test/test_fastmem_perf.c
new file mode 100644
index 0000000000..73c0a4c6ce
--- /dev/null
+++ b/app/test/test_fastmem_perf.c
@@ -0,0 +1,1040 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_launch.h>
+#include <rte_lcore.h>
+#include <rte_malloc.h>
+#include <rte_mempool.h>
+#include <rte_stdatomic.h>
+
+#include <rte_fastmem.h>
+
+#include "test.h"
+
+#define TEST_LOG(...) printf(__VA_ARGS__)
+
+static const size_t SIZES[] = { 8, 64, 256, 1024, 4096 };
+#define N_SIZES RTE_DIM(SIZES)
+
+/* Number of ops for warmup and measurement. */
+#define WARMUP_OPS 20000u
+#define MEASURE_OPS 2000000u
+
+/* Buffer for scenarios that allocate N then free N. */
+#define BATCH_N 256
+
+/*
+ * Allocator vtable: a thin adapter exposing alloc / free /
+ * per-allocator setup/teardown. Each scenario calls these
+ * indirectly so the same timing loop serves all allocators.
+ */
+struct allocator {
+ const char *name;
+ int (*setup)(size_t size, unsigned int n_max);
+ void (*teardown)(void);
+ void *(*alloc)(void);
+ void (*free_obj)(void *ptr);
+ int (*alloc_bulk)(void **ptrs, unsigned int n);
+ void (*free_bulk)(void **ptrs, unsigned int n);
+};
+
+/* Fastmem adapter -------------------------------------------------- */
+
+static size_t fastmem_size;
+
+static int
+fastmem_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ fastmem_size = size;
+ return 0;
+}
+
+static void
+fastmem_teardown(void)
+{
+ rte_fastmem_cache_flush();
+}
+
+static void * __rte_noinline
+fastmem_alloc(void)
+{
+ return rte_fastmem_alloc(fastmem_size, 0, 0);
+}
+
+static void __rte_noinline
+fastmem_free(void *ptr)
+{
+ rte_fastmem_free(ptr);
+}
+
+/* Mempool adapter -------------------------------------------------- */
+
+static struct rte_mempool *mempool_pool;
+
+static int
+mempool_setup(size_t size, unsigned int n_max)
+{
+ char name[RTE_MEMPOOL_NAMESIZE];
+ unsigned int cache_size;
+
+ /*
+ * Pool size must accommodate the full batch burst plus
+ * per-lcore cache capacity. Use mempool's default cache
+ * size so we're measuring its standard hot path.
+ */
+ cache_size = RTE_MEMPOOL_CACHE_MAX_SIZE;
+
+ snprintf(name, sizeof(name), "fmperf_mp_%zu", size);
+ mempool_pool = rte_mempool_create(name, n_max + cache_size * 2,
+ size, cache_size, 0, NULL, NULL, NULL, NULL,
+ SOCKET_ID_ANY, 0);
+ if (mempool_pool == NULL) {
+ TEST_LOG("mempool_create(%zu) failed\n", size);
+ return -1;
+ }
+
+ return 0;
+}
+
+static void
+mempool_teardown(void)
+{
+ rte_mempool_free(mempool_pool);
+ mempool_pool = NULL;
+}
+
+static void * __rte_noinline
+mempool_alloc_one(void)
+{
+ void *obj = NULL;
+
+ if (rte_mempool_get(mempool_pool, &obj) < 0)
+ return NULL;
+ return obj;
+}
+
+static void __rte_noinline
+mempool_free_one(void *ptr)
+{
+ rte_mempool_put(mempool_pool, ptr);
+}
+
+/* rte_malloc adapter ----------------------------------------------- */
+
+static size_t malloc_size;
+
+static int
+malloc_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ malloc_size = size;
+ return 0;
+}
+
+static void
+malloc_teardown(void)
+{
+}
+
+static void * __rte_noinline
+malloc_alloc(void)
+{
+ return rte_malloc(NULL, malloc_size, 0);
+}
+
+static void __rte_noinline
+malloc_free(void *ptr)
+{
+ rte_free(ptr);
+}
+
+/* libc (glibc) malloc adapter -------------------------------------- */
+
+static size_t libc_size;
+
+static int
+libc_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ /*
+ * Round up to cache-line alignment to match the other
+ * allocators' default alignment guarantees and keep the
+ * comparison honest. aligned_alloc() requires size to be
+ * a multiple of the alignment.
+ */
+ libc_size = RTE_ALIGN_CEIL(size, RTE_CACHE_LINE_SIZE);
+ return 0;
+}
+
+static void
+libc_teardown(void)
+{
+}
+
+static void * __rte_noinline
+libc_alloc(void)
+{
+ return aligned_alloc(RTE_CACHE_LINE_SIZE, libc_size);
+}
+
+static void __rte_noinline
+libc_free(void *ptr)
+{
+ free(ptr);
+}
+
+/* Bulk adapters ---------------------------------------------------- */
+
+static int __rte_noinline
+fastmem_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return rte_fastmem_alloc_bulk(ptrs, n, fastmem_size, 0, 0);
+}
+
+static void __rte_noinline
+fastmem_free_bulk(void **ptrs, unsigned int n)
+{
+ rte_fastmem_free_bulk(ptrs, n);
+}
+
+/* Fastmem handle adapter ------------------------------------------- */
+
+static rte_fastmem_handle_t fastmem_handle;
+
+static int
+fastmem_h_setup(size_t size, unsigned int n_max __rte_unused)
+{
+ return rte_fastmem_hlookup(size, 0, rte_socket_id(), &fastmem_handle);
+}
+
+static void
+fastmem_h_teardown(void)
+{
+ rte_fastmem_cache_flush();
+}
+
+static void * __rte_noinline
+fastmem_h_alloc(void)
+{
+ return rte_fastmem_halloc(fastmem_handle, 0);
+}
+
+static void __rte_noinline
+fastmem_h_free(void *ptr)
+{
+ rte_fastmem_hfree(fastmem_handle, ptr);
+}
+
+static int __rte_noinline
+fastmem_h_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return rte_fastmem_halloc_bulk(fastmem_handle, ptrs, n, 0);
+}
+
+static void __rte_noinline
+fastmem_h_free_bulk(void **ptrs, unsigned int n)
+{
+ rte_fastmem_hfree_bulk(fastmem_handle, ptrs, n);
+}
+
+/* Mempool adapter -------------------------------------------------- */
+
+static int __rte_noinline
+mempool_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return rte_mempool_get_bulk(mempool_pool, ptrs, n);
+}
+
+static void __rte_noinline
+mempool_free_bulk(void **ptrs, unsigned int n)
+{
+ rte_mempool_put_bulk(mempool_pool, ptrs, n);
+}
+
+static int __rte_noinline
+generic_alloc_bulk(void **ptrs, unsigned int n, void *(*alloc_fn)(void))
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++) {
+ ptrs[i] = alloc_fn();
+ if (ptrs[i] == NULL)
+ return -1;
+ }
+ return 0;
+}
+
+static int __rte_noinline
+malloc_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return generic_alloc_bulk(ptrs, n, malloc_alloc);
+}
+
+static void __rte_noinline
+malloc_free_bulk(void **ptrs, unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ malloc_free(ptrs[i]);
+}
+
+static int __rte_noinline
+libc_alloc_bulk(void **ptrs, unsigned int n)
+{
+ return generic_alloc_bulk(ptrs, n, libc_alloc);
+}
+
+static void __rte_noinline
+libc_free_bulk(void **ptrs, unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ libc_free(ptrs[i]);
+}
+
+/* Adapter table ---------------------------------------------------- */
+
+static const struct allocator allocators[] = {
+ { "fastmem", fastmem_setup, fastmem_teardown, fastmem_alloc, fastmem_free, fastmem_alloc_bulk, fastmem_free_bulk },
+ { "fastmem_h", fastmem_h_setup, fastmem_h_teardown, fastmem_h_alloc, fastmem_h_free, fastmem_h_alloc_bulk, fastmem_h_free_bulk },
+ { "mempool", mempool_setup, mempool_teardown, mempool_alloc_one, mempool_free_one, mempool_alloc_bulk, mempool_free_bulk },
+ { "rte_malloc", malloc_setup, malloc_teardown, malloc_alloc, malloc_free, malloc_alloc_bulk, malloc_free_bulk },
+ { "libc", libc_setup, libc_teardown, libc_alloc, libc_free, libc_alloc_bulk, libc_free_bulk },
+};
+#define N_ALLOCATORS RTE_DIM(allocators)
+
+/*
+ * Scenario 1: tight alloc+free loop. A single object is cycled
+ * repeatedly. The LIFO path keeps the same pointer hot, giving
+ * a best-case measurement.
+ */
+static double
+run_tight(const struct allocator *alloc, size_t size)
+{
+ void *p;
+ uint64_t tsc;
+ unsigned int i;
+
+ if (alloc->setup(size, 1) < 0)
+ return -1.0;
+
+ /* Warmup. */
+ for (i = 0; i < WARMUP_OPS; i++) {
+ p = alloc->alloc();
+ if (p == NULL)
+ goto err;
+ alloc->free_obj(p);
+ }
+
+ tsc = rte_rdtsc_precise();
+ for (i = 0; i < MEASURE_OPS; i++) {
+ p = alloc->alloc();
+ if (p == NULL)
+ goto err;
+ alloc->free_obj(p);
+ }
+ tsc = rte_rdtsc_precise() - tsc;
+
+ alloc->teardown();
+
+ return (double)tsc / MEASURE_OPS;
+err:
+ alloc->teardown();
+ return -1.0;
+}
+
+/*
+ * Scenario 2: allocate N, free N (FIFO free order). Exercises
+ * cache refill and drain paths when N exceeds cache capacity.
+ */
+static void
+run_batch(const struct allocator *alloc, size_t size,
+ double *cycles_alloc, double *cycles_free)
+{
+ void *ptrs[BATCH_N];
+ uint64_t tsc_alloc = 0, tsc_free = 0;
+ unsigned int iter, i;
+ unsigned int iters;
+
+ *cycles_alloc = -1.0;
+ *cycles_free = -1.0;
+
+ if (alloc->setup(size, BATCH_N) < 0)
+ return;
+
+ /* Pick iteration count so total ops ~= MEASURE_OPS. */
+ iters = MEASURE_OPS / BATCH_N;
+
+ /* Warmup. */
+ for (iter = 0; iter < WARMUP_OPS / BATCH_N; iter++) {
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ for (i = 0; i < BATCH_N; i++)
+ alloc->free_obj(ptrs[i]);
+ }
+
+ for (iter = 0; iter < iters; iter++) {
+ uint64_t t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ tsc_alloc += rte_rdtsc_precise() - t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < BATCH_N; i++)
+ alloc->free_obj(ptrs[i]);
+ tsc_free += rte_rdtsc_precise() - t0;
+ }
+
+ alloc->teardown();
+
+ *cycles_alloc = (double)tsc_alloc / (iters * BATCH_N);
+ *cycles_free = (double)tsc_free / (iters * BATCH_N);
+ return;
+err:
+ alloc->teardown();
+}
+
+/*
+ * Scenario 3: allocate N, free N in reverse order.
+ */
+static void
+run_batch_reverse(const struct allocator *alloc, size_t size,
+ double *cycles_alloc, double *cycles_free)
+{
+ void *ptrs[BATCH_N];
+ uint64_t tsc_alloc = 0, tsc_free = 0;
+ unsigned int iter, i;
+ unsigned int iters;
+
+ *cycles_alloc = -1.0;
+ *cycles_free = -1.0;
+
+ if (alloc->setup(size, BATCH_N) < 0)
+ return;
+
+ iters = MEASURE_OPS / BATCH_N;
+
+ for (iter = 0; iter < WARMUP_OPS / BATCH_N; iter++) {
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ for (i = BATCH_N; i > 0; i--)
+ alloc->free_obj(ptrs[i - 1]);
+ }
+
+ for (iter = 0; iter < iters; iter++) {
+ uint64_t t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < BATCH_N; i++) {
+ ptrs[i] = alloc->alloc();
+ if (ptrs[i] == NULL)
+ goto err;
+ }
+ tsc_alloc += rte_rdtsc_precise() - t0;
+
+ t0 = rte_rdtsc_precise();
+ for (i = BATCH_N; i > 0; i--)
+ alloc->free_obj(ptrs[i - 1]);
+ tsc_free += rte_rdtsc_precise() - t0;
+ }
+
+ alloc->teardown();
+
+ *cycles_alloc = (double)tsc_alloc / (iters * BATCH_N);
+ *cycles_free = (double)tsc_free / (iters * BATCH_N);
+ return;
+err:
+ alloc->teardown();
+}
+
+/*
+ * Scenario 4: multi-lcore alloc/work/free with a dummy-work
+ * baseline. Each worker runs a tight alloc → touch → free loop
+ * on its own lcore. A second run with the same dummy work but
+ * no allocator traffic establishes a baseline; the per-op
+ * allocator cost is reported as (alloc_run - baseline_run).
+ *
+ * Fixed size class and a fixed amount of dummy work per op —
+ * this scenario sweeps lcore count rather than size.
+ */
+#define MULTI_SIZE 256u
+#define MULTI_WORK_BYTES 64u
+#define MULTI_WORK_PASSES 8u /* RMW passes over the work region. */
+#define MULTI_OPS 200000u
+#define MULTI_WARMUP 2000u
+#define MAX_MULTI_LCORES 32u
+
+/*
+ * Per-worker volatile sink. Each worker writes to its own
+ * slot, preventing dead-code elimination of touch_buffer() and
+ * avoiding cross-lcore cache-line sharing on the hot path.
+ * Padded to cache-line stride to prevent false sharing between
+ * neighboring workers' slots.
+ */
+struct worker_sink {
+ volatile uint64_t value;
+} __rte_cache_aligned;
+
+static struct worker_sink worker_sinks[RTE_MAX_LCORE];
+
+/*
+ * Out-of-line dummy workload: run MULTI_WORK_PASSES
+ * read-modify-write passes over the first 'bytes' of the
+ * buffer. Each pass reads what the previous pass wrote, so the
+ * compiler cannot unroll or parallelize across passes — the
+ * work scales linearly with MULTI_WORK_PASSES. Returns an
+ * accumulator so the caller can feed it into a volatile sink;
+ * without that, the compiler could elide the whole function.
+ *
+ * __rte_noinline so it looks identical to the compiler in both
+ * the baseline (pre-allocated scratch buffer) and alloc-path
+ * runs, making the cycle-delta subtraction valid.
+ *
+ * The purpose of this being tunably expensive is to keep
+ * worker-per-iteration cost high relative to the allocator's
+ * critical section, so that even serialized allocators like
+ * rte_malloc spend most of their time outside the lock and the
+ * measured per-op allocator cost reflects its own work rather
+ * than its contention queue.
+ */
+static uint64_t __rte_noinline
+touch_buffer(void *buf, size_t bytes)
+{
+ uint64_t *p = buf;
+ size_t n = bytes / sizeof(uint64_t);
+ uint64_t acc = 0;
+ unsigned int pass;
+ size_t i;
+
+ /* Prime the buffer with a known pattern. */
+ for (i = 0; i < n; i++)
+ p[i] = i * 0x9E3779B97F4A7C15ULL;
+
+ /*
+ * Dependent RMW passes: each pass reads p[i] written by
+ * the previous pass, mixes the pass index in, and writes
+ * back. The XOR into acc keeps the chain live.
+ */
+ for (pass = 0; pass < MULTI_WORK_PASSES; pass++) {
+ for (i = 0; i < n; i++) {
+ uint64_t v = p[i];
+
+ v = v * 0xC2B2AE3D27D4EB4FULL + pass;
+ v ^= v >> 33;
+ p[i] = v;
+ acc ^= v;
+ }
+ }
+
+ return acc;
+}
+
+struct worker_args {
+ const struct allocator *alloc;
+ void *scratch; /* baseline only; NULL => alloc path */
+ unsigned int iters;
+ unsigned int warmup;
+ unsigned int bulk_n; /* 0 = single-object, >0 = bulk */
+ RTE_ATOMIC(bool) start_flag; /* barrier at worker entry */
+ uint64_t cycles; /* out */
+ unsigned int ops; /* out */
+ int err; /* out */
+};
+
+static int
+worker_run(void *arg)
+{
+ struct worker_args *wa = arg;
+ unsigned int lcore = rte_lcore_id();
+ uint64_t acc = 0;
+ uint64_t t0;
+ unsigned int i;
+
+ wa->err = 0;
+ wa->ops = 0;
+ wa->cycles = 0;
+
+ /* Wait for start flag (spin-barrier set by main). */
+ while (!rte_atomic_load_explicit(&wa->start_flag,
+ rte_memory_order_acquire))
+ rte_pause();
+
+ /* Warmup. */
+ for (i = 0; i < wa->warmup; i++) {
+ void *p;
+
+ if (wa->scratch != NULL)
+ p = wa->scratch;
+ else {
+ p = wa->alloc->alloc();
+ if (p == NULL) {
+ wa->err = -1;
+ return -1;
+ }
+ }
+ acc ^= touch_buffer(p, MULTI_WORK_BYTES);
+ if (wa->scratch == NULL)
+ wa->alloc->free_obj(p);
+ }
+
+ /* Measured loop. */
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < wa->iters; i++) {
+ void *p;
+
+ if (wa->scratch != NULL)
+ p = wa->scratch;
+ else {
+ p = wa->alloc->alloc();
+ if (p == NULL) {
+ wa->err = -1;
+ break;
+ }
+ }
+ acc ^= touch_buffer(p, MULTI_WORK_BYTES);
+ if (wa->scratch == NULL)
+ wa->alloc->free_obj(p);
+ }
+ wa->cycles = rte_rdtsc_precise() - t0;
+ wa->ops = i;
+
+ /* Publish accumulator to defeat dead-code elimination. */
+ worker_sinks[lcore].value ^= acc;
+
+ return 0;
+}
+
+static int
+worker_run_bulk(void *arg)
+{
+ struct worker_args *wa = arg;
+ unsigned int lcore = rte_lcore_id();
+ void *ptrs[BATCH_N];
+ uint64_t acc = 0;
+ uint64_t t0;
+ unsigned int i, j;
+ unsigned int bulk_n = wa->bulk_n;
+
+ wa->err = 0;
+ wa->ops = 0;
+ wa->cycles = 0;
+
+ while (!rte_atomic_load_explicit(&wa->start_flag,
+ rte_memory_order_acquire))
+ rte_pause();
+
+ /* Warmup. */
+ for (i = 0; i < wa->warmup; i++) {
+ if (wa->alloc->alloc_bulk(ptrs, bulk_n) < 0) {
+ wa->err = -1;
+ return -1;
+ }
+ for (j = 0; j < bulk_n; j++)
+ acc ^= touch_buffer(ptrs[j], MULTI_WORK_BYTES);
+ wa->alloc->free_bulk(ptrs, bulk_n);
+ }
+
+ t0 = rte_rdtsc_precise();
+ for (i = 0; i < wa->iters; i++) {
+ if (wa->alloc->alloc_bulk(ptrs, bulk_n) < 0) {
+ wa->err = -1;
+ break;
+ }
+ for (j = 0; j < bulk_n; j++)
+ acc ^= touch_buffer(ptrs[j], MULTI_WORK_BYTES);
+ wa->alloc->free_bulk(ptrs, bulk_n);
+ }
+ wa->cycles = rte_rdtsc_precise() - t0;
+ wa->ops = i * bulk_n;
+
+ worker_sinks[lcore].value ^= acc;
+
+ return 0;
+}
+
+/*
+ * Launch workers on the first 'n_workers' worker lcores, run
+ * either the baseline (scratch != NULL) or the alloc path
+ * (scratch == NULL), and return the mean per-op cycle cost
+ * averaged across participating workers.
+ *
+ * On any worker error, returns -1.0.
+ */
+static double
+run_multi_workers(const struct allocator *alloc, unsigned int n_workers,
+ void *const *scratches, unsigned int bulk_n)
+{
+ struct worker_args wargs[RTE_MAX_LCORE];
+ unsigned int worker_lcores[MAX_MULTI_LCORES];
+ unsigned int n = 0;
+ unsigned int lcore_id;
+ unsigned int i;
+ lcore_function_t *fn = bulk_n > 0 ? worker_run_bulk : worker_run;
+
+ /* Collect the first n_workers worker lcores. */
+ RTE_LCORE_FOREACH_WORKER(lcore_id) {
+ if (n >= n_workers)
+ break;
+ worker_lcores[n++] = lcore_id;
+ }
+ if (n < n_workers)
+ return -1.0;
+
+ /* Prepare per-worker args. */
+ for (i = 0; i < n_workers; i++) {
+ struct worker_args *wa = &wargs[worker_lcores[i]];
+
+ wa->alloc = alloc;
+ wa->scratch = scratches != NULL ? scratches[i] : NULL;
+ wa->iters = MULTI_OPS;
+ wa->warmup = MULTI_WARMUP;
+ wa->bulk_n = bulk_n;
+ rte_atomic_store_explicit(&wa->start_flag, false,
+ rte_memory_order_relaxed);
+ }
+
+ /* Launch workers. They spin on start_flag until released. */
+ for (i = 0; i < n_workers; i++)
+ rte_eal_remote_launch(fn, &wargs[worker_lcores[i]],
+ worker_lcores[i]);
+
+ /* Release all workers roughly simultaneously. */
+ for (i = 0; i < n_workers; i++)
+ rte_atomic_store_explicit(
+ &wargs[worker_lcores[i]].start_flag, true,
+ rte_memory_order_release);
+
+ /* Wait for completion. */
+ for (i = 0; i < n_workers; i++)
+ rte_eal_wait_lcore(worker_lcores[i]);
+
+ /* Aggregate: mean cycles per op across workers. */
+ {
+ double sum_cycles_per_op = 0.0;
+ unsigned int n_ok = 0;
+
+ for (i = 0; i < n_workers; i++) {
+ struct worker_args *wa = &wargs[worker_lcores[i]];
+
+ if (wa->err != 0 || wa->ops == 0)
+ return -1.0;
+ sum_cycles_per_op +=
+ (double)wa->cycles / (double)wa->ops;
+ n_ok++;
+ }
+ return sum_cycles_per_op / n_ok;
+ }
+}
+
+/*
+ * One sub-run of Scenario 4: given an allocator and a worker
+ * count, return (baseline, alloc_path) mean cycles per op.
+ */
+static void
+run_multi_lcore(const struct allocator *alloc, unsigned int n_workers,
+ unsigned int bulk_n, double *baseline, double *alloc_path)
+{
+ void *scratches[MAX_MULTI_LCORES] = {0};
+ unsigned int n_alloced = 0;
+ unsigned int i;
+
+ *baseline = -1.0;
+ *alloc_path = -1.0;
+
+ if (alloc->setup(MULTI_SIZE, n_workers * 64) < 0)
+ return;
+
+ /* Baseline: pre-allocate one scratch per worker. */
+ for (i = 0; i < n_workers; i++) {
+ scratches[i] = alloc->alloc();
+ if (scratches[i] == NULL)
+ goto err;
+ n_alloced++;
+ }
+
+ *baseline = run_multi_workers(alloc, n_workers, scratches, 0);
+
+ for (i = 0; i < n_alloced; i++)
+ alloc->free_obj(scratches[i]);
+ n_alloced = 0;
+
+ /* Alloc path: workers alloc+free each iter. */
+ *alloc_path = run_multi_workers(alloc, n_workers, NULL, bulk_n);
+
+ alloc->teardown();
+ return;
+err:
+ for (i = 0; i < n_alloced; i++)
+ alloc->free_obj(scratches[i]);
+ alloc->teardown();
+}
+
+/* Reporting -------------------------------------------------------- */
+
+static void
+print_header(const char *title)
+{
+ size_t i;
+
+ TEST_LOG("\n=== %s ===\n", title);
+ TEST_LOG("%-12s", "allocator");
+ for (i = 0; i < N_SIZES; i++)
+ TEST_LOG(" %10zu B", SIZES[i]);
+ TEST_LOG("\n");
+}
+
+static void
+print_row(const char *name, const double *values)
+{
+ size_t i;
+
+ TEST_LOG("%-12s", name);
+ for (i = 0; i < N_SIZES; i++) {
+ if (values[i] < 0)
+ TEST_LOG(" %12s", "--");
+ else
+ TEST_LOG(" %12.1f", values[i]);
+ }
+ TEST_LOG("\n");
+}
+
+static void
+print_multi_header(const char *title, const unsigned int *lcore_counts,
+ unsigned int n_counts)
+{
+ unsigned int i;
+
+ TEST_LOG("\n=== %s ===\n", title);
+ TEST_LOG("%-12s", "allocator");
+ for (i = 0; i < n_counts; i++)
+ TEST_LOG(" %8u lcore%c", lcore_counts[i],
+ lcore_counts[i] == 1 ? ' ' : 's');
+ TEST_LOG("\n");
+}
+
+static void
+print_multi_row(const char *name, const double *values, unsigned int n_counts)
+{
+ unsigned int i;
+
+ TEST_LOG("%-12s", name);
+ for (i = 0; i < n_counts; i++) {
+ if (values[i] < 0)
+ TEST_LOG(" %14s", "--");
+ else
+ TEST_LOG(" %14.1f", values[i]);
+ }
+ TEST_LOG("\n");
+}
+
+/* Driver ----------------------------------------------------------- */
+
+static int
+test_fastmem_perf(void)
+{
+ size_t i;
+ size_t a;
+ int rc;
+
+ rc = rte_fastmem_init();
+ if (rc < 0) {
+ TEST_LOG("rte_fastmem_init() failed: %d\n", rc);
+ return -1;
+ }
+
+ rc = rte_fastmem_reserve(128 * 1024 * 1024, SOCKET_ID_ANY);
+ if (rc < 0) {
+ TEST_LOG("rte_fastmem_reserve() failed: %d\n", rc);
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ TEST_LOG("\nfastmem performance — single-lcore, fixed-size\n");
+ TEST_LOG("All numbers are TSC cycles.\n");
+
+ /* Scenario 1: tight alloc+free. */
+ print_header("Scenario 1: Single-object hot path — cycles per (alloc + free)");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ vals[i] = run_tight(&allocators[a], SIZES[i]);
+ print_row(allocators[a].name, vals);
+ }
+
+ /* Scenario 2: batched, FIFO free. */
+ print_header("Scenario 2: Batch alloc, FIFO free — cycles per alloc");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_alloc);
+ }
+ print_header("Scenario 2: Batch alloc, FIFO free — cycles per free");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_free);
+ }
+
+ /* Scenario 3: batched, reverse free. */
+ print_header("Scenario 3: Batch alloc, LIFO free — cycles per alloc");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch_reverse(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_alloc);
+ }
+ print_header("Scenario 3: Batch alloc, LIFO free — cycles per free");
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ double vals_alloc[N_SIZES], vals_free[N_SIZES];
+
+ for (i = 0; i < N_SIZES; i++)
+ run_batch_reverse(&allocators[a], SIZES[i],
+ &vals_alloc[i], &vals_free[i]);
+ print_row(allocators[a].name, vals_free);
+ }
+
+ /* Scenario 4: multi-lcore alloc/work/free with baseline. */
+ {
+ unsigned int max_workers = rte_lcore_count() - 1;
+ unsigned int lcore_counts[8];
+ unsigned int n_counts = 0;
+ unsigned int w;
+ double base_vals[N_ALLOCATORS][8];
+ double alloc_vals[N_ALLOCATORS][8];
+ double delta_vals[N_ALLOCATORS][8];
+
+ if (max_workers > MAX_MULTI_LCORES)
+ max_workers = MAX_MULTI_LCORES;
+
+ /* Sweep lcore counts: 1, 2, 4, 8, ... up to max_workers. */
+ for (w = 1; w <= max_workers && n_counts < RTE_DIM(lcore_counts); w *= 2)
+ lcore_counts[n_counts++] = w;
+ /* Ensure max_workers is the final column if not power of two. */
+ if (n_counts > 0 && lcore_counts[n_counts - 1] != max_workers &&
+ n_counts < RTE_DIM(lcore_counts) && max_workers >= 1)
+ lcore_counts[n_counts++] = max_workers;
+
+ if (n_counts == 0) {
+ TEST_LOG("\nScenario 4 (Multi-lcore contention) skipped: no worker lcores available.\n");
+ } else {
+ TEST_LOG("\nScenario 4 parameters: size=%u B\n",
+ MULTI_SIZE);
+
+ for (a = 0; a < N_ALLOCATORS; a++) {
+ unsigned int c;
+
+ for (c = 0; c < n_counts; c++)
+ run_multi_lcore(&allocators[a], lcore_counts[c],
+ 0, &base_vals[a][c],
+ &alloc_vals[a][c]);
+ for (c = 0; c < n_counts; c++) {
+ if (base_vals[a][c] < 0 || alloc_vals[a][c] < 0)
+ delta_vals[a][c] = -1.0;
+ else
+ delta_vals[a][c] = alloc_vals[a][c] -
+ base_vals[a][c];
+ }
+ }
+
+ TEST_LOG("Baseline (domain logic only): %.1f cycles/op\n",
+ base_vals[0][0]);
+
+ print_multi_header("Scenario 4: Multi-lcore contention — allocator overhead (cycles/op)",
+ lcore_counts, n_counts);
+ for (a = 0; a < N_ALLOCATORS; a++)
+ print_multi_row(allocators[a].name,
+ delta_vals[a], n_counts);
+ }
+ }
+
+ /* Scenario 5: multi-lcore bulk alloc/work/free. */
+ {
+ unsigned int max_workers = rte_lcore_count() - 1;
+ unsigned int lcore_counts[8];
+ unsigned int n_counts = 0;
+ unsigned int w;
+ double base_vals[N_ALLOCATORS][8];
+ double alloc_vals[N_ALLOCATORS][8];
+ double delta_vals[N_ALLOCATORS][8];
+ unsigned int bulk_n = 8;
+
+ if (max_workers > MAX_MULTI_LCORES)
+ max_workers = MAX_MULTI_LCORES;
+
+ for (w = 1; w <= max_workers && n_counts < RTE_DIM(lcore_counts); w *= 2)
+ lcore_counts[n_counts++] = w;
+ if (n_counts > 0 && lcore_counts[n_counts - 1] != max_workers &&
+ n_counts < RTE_DIM(lcore_counts) && max_workers >= 1)
+ lcore_counts[n_counts++] = max_workers;
+
+ if (n_counts == 0) {
+ TEST_LOG("\nScenario 5 (Multi-lcore bulk contention) skipped: no worker lcores available.\n");
+ } else {
+ TEST_LOG("\nScenario 5 parameters: size=%u B, "
+ "bulk=%u\n",
+ MULTI_SIZE, bulk_n);
+
+ for (size_t a = 0; a < N_ALLOCATORS; a++) {
+ unsigned int c;
+
+ for (c = 0; c < n_counts; c++)
+ run_multi_lcore(&allocators[a],
+ lcore_counts[c], bulk_n,
+ &base_vals[a][c],
+ &alloc_vals[a][c]);
+ for (c = 0; c < n_counts; c++) {
+ if (base_vals[a][c] < 0 || alloc_vals[a][c] < 0)
+ delta_vals[a][c] = -1.0;
+ else
+ delta_vals[a][c] = alloc_vals[a][c] -
+ base_vals[a][c];
+ }
+ }
+
+ TEST_LOG("Baseline (domain logic only): %.1f cycles/op\n",
+ base_vals[0][0]);
+
+ print_multi_header("Scenario 5: Multi-lcore bulk contention — allocator overhead (cycles/op)",
+ lcore_counts, n_counts);
+ for (size_t a = 0; a < N_ALLOCATORS; a++)
+ print_multi_row(allocators[a].name,
+ delta_vals[a], n_counts);
+ }
+ }
+
+ TEST_LOG("\n");
+ rte_fastmem_deinit();
+ return 0;
+}
+
+REGISTER_PERF_TEST(fastmem_perf_autotest, test_fastmem_perf);
diff --git a/app/test/test_fastmem_profile.c b/app/test/test_fastmem_profile.c
new file mode 100644
index 0000000000..9a5dc94018
--- /dev/null
+++ b/app/test/test_fastmem_profile.c
@@ -0,0 +1,157 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+/*
+ * A minimal fastmem workload intended for use with perf record /
+ * perf report. Runs a tight alloc/free loop for a fixed duration
+ * so that sampling profilers can attribute cycles to individual
+ * functions and instructions within the fastmem hot path.
+ *
+ * Usage:
+ * perf record -g -- dpdk-test --no-huge --no-pci -m 8192 \
+ * -l 0 <<< fastmem_profile_autotest
+ * perf report
+ */
+
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_lcore.h>
+#include <rte_memory.h>
+
+#include <rte_fastmem.h>
+
+#include "test.h"
+
+/* Duration of each sub-test in TSC cycles (~3 seconds at 3 GHz). */
+#define PROFILE_DURATION_CYCLES (3ULL * rte_get_tsc_hz())
+
+/* Allocation size for the profiling workload. */
+#define PROFILE_SIZE 256u
+
+/*
+ * Sub-test 1: tight alloc+free, exercises only the per-lcore
+ * cache (no bin interaction after warmup).
+ */
+static int
+profile_cache_hit(void)
+{
+ uint64_t deadline;
+ uint64_t ops = 0;
+
+ deadline = rte_rdtsc() + PROFILE_DURATION_CYCLES;
+
+ while (rte_rdtsc() < deadline) {
+ void *p = rte_fastmem_alloc(PROFILE_SIZE, 0, 0);
+
+ if (p == NULL)
+ return -1;
+ rte_fastmem_free(p);
+ ops++;
+ }
+
+ printf(" cache_hit: %" PRIu64 " ops\n", ops);
+ return 0;
+}
+
+/*
+ * Sub-test 2: alloc N then free N, where N exceeds the cache
+ * capacity. This forces repeated cache refills and drains,
+ * exercising the bin lock and slab free-list traversal.
+ */
+#define PROFILE_BATCH 256u
+
+static int
+profile_cache_miss(void)
+{
+ void *ptrs[PROFILE_BATCH];
+ uint64_t deadline;
+ uint64_t ops = 0;
+ unsigned int i;
+
+ deadline = rte_rdtsc() + PROFILE_DURATION_CYCLES;
+
+ while (rte_rdtsc() < deadline) {
+ for (i = 0; i < PROFILE_BATCH; i++) {
+ ptrs[i] = rte_fastmem_alloc(PROFILE_SIZE, 0, 0);
+ if (ptrs[i] == NULL)
+ return -1;
+ }
+ for (i = 0; i < PROFILE_BATCH; i++)
+ rte_fastmem_free(ptrs[i]);
+ ops += PROFILE_BATCH;
+ }
+
+ printf(" cache_miss: %" PRIu64 " ops\n", ops);
+ return 0;
+}
+
+static int
+test_fastmem_profile_cache_hit(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ if (rc < 0) {
+ printf("rte_fastmem_init() failed: %d\n", rc);
+ return -1;
+ }
+
+ rc = rte_fastmem_reserve(128 * 1024 * 1024, SOCKET_ID_ANY);
+ if (rc < 0) {
+ printf("rte_fastmem_reserve() failed: %d\n", rc);
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ printf("fastmem profile: cache-hit workload (size=%u, ~%u s)\n",
+ PROFILE_SIZE, 3);
+
+ if (profile_cache_hit() < 0) {
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ rte_fastmem_deinit();
+ return 0;
+}
+
+static int
+test_fastmem_profile_cache_miss(void)
+{
+ int rc;
+
+ rc = rte_fastmem_init();
+ if (rc < 0) {
+ printf("rte_fastmem_init() failed: %d\n", rc);
+ return -1;
+ }
+
+ rc = rte_fastmem_reserve(128 * 1024 * 1024, SOCKET_ID_ANY);
+ if (rc < 0) {
+ printf("rte_fastmem_reserve() failed: %d\n", rc);
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ printf("fastmem profile: cache-miss workload (size=%u, ~%u s)\n",
+ PROFILE_SIZE, 3);
+
+ if (profile_cache_miss() < 0) {
+ rte_fastmem_deinit();
+ return -1;
+ }
+
+ rte_fastmem_deinit();
+ return 0;
+}
+
+REGISTER_PERF_TEST(fastmem_profile_cache_hit_autotest,
+ test_fastmem_profile_cache_hit);
+REGISTER_PERF_TEST(fastmem_profile_cache_miss_autotest,
+ test_fastmem_profile_cache_miss);
--
2.43.0
^ permalink raw reply related
* [RFC v2 2/3] lib: add fastmem library
From: Mattias Rönnblom @ 2026-05-26 8:57 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Mattias Rönnblom
In-Reply-To: <20260526085743.64396-1-hofors@lysator.liu.se>
Introduce fastmem, a fast general-purpose small-object allocator
for DPDK applications. It allows an application to replace its
many per-type mempools with a single allocator that handles
arbitrary sizes, grows on demand, and offers mempool-level
performance on the hot path.
Applications that manage many object types (connections, sessions,
work items, timers) currently maintain a separate mempool for each,
requiring upfront sizing and wasting memory on over-provisioned
pools. Fastmem removes both constraints.
Key properties:
* Huge-page-backed, NUMA-aware, DMA-usable.
* Per-lcore caches for lock-free alloc/free on EAL threads.
* Bulk alloc and free APIs.
* Power-of-two size classes from 8 B to 1 MiB.
* Backing memory grows lazily; rte_fastmem_reserve() allows
upfront reservation to avoid latency spikes.
* Always-on per-lcore and per-class statistics.
Bounded to small objects; requests above rte_fastmem_max_size()
are rejected. Replacing rte_malloc is currently not a goal.
--
RFC v2:
* Fix use-after-free in rte_fastmem_deinit() when caches were
allocated cross-socket. Restructured teardown into three phases.
* Add defensive bounds check to local_socket_id() final fallback.
* Add secondary process support. Shared state is discovered lazily
on first allocation; secondaries operate without per-lcore caches.
* Add handle-based allocation API (rte_fastmem_hlookup,
rte_fastmem_halloc, rte_fastmem_halloc_bulk).
* Add test_alloc_cross_socket_deinit exercising cross-socket
teardown path.
* Fix clang -Wthread-safety-analysis warnings.
* Move fastmem to alphabetical position in lib/meson.build.
* Remove trailing double blank lines in test_fastmem.c.
* Split programming guide into separate commit.
Signed-off-by: Mattias Rönnblom <hofors@lysator.liu.se>
---
doc/api/doxy-api-index.md | 1 +
doc/api/doxy-api.conf.in | 1 +
lib/fastmem/meson.build | 6 +
lib/fastmem/rte_fastmem.c | 1694 +++++++++++++++++++++++++++++++++++++
lib/fastmem/rte_fastmem.h | 774 +++++++++++++++++
lib/meson.build | 1 +
6 files changed, 2477 insertions(+)
create mode 100644 lib/fastmem/meson.build
create mode 100644 lib/fastmem/rte_fastmem.c
create mode 100644 lib/fastmem/rte_fastmem.h
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 9296042119..7ebf1201ce 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -70,6 +70,7 @@ The public API headers are grouped by topics:
[memzone](@ref rte_memzone.h),
[mempool](@ref rte_mempool.h),
[malloc](@ref rte_malloc.h),
+ [fastmem](@ref rte_fastmem.h),
[memcpy](@ref rte_memcpy.h)
- **timers**:
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index bedd944681..4355e9fb2d 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -43,6 +43,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \
@TOPDIR@/lib/efd \
@TOPDIR@/lib/ethdev \
@TOPDIR@/lib/eventdev \
+ @TOPDIR@/lib/fastmem \
@TOPDIR@/lib/fib \
@TOPDIR@/lib/gpudev \
@TOPDIR@/lib/graph \
diff --git a/lib/fastmem/meson.build b/lib/fastmem/meson.build
new file mode 100644
index 0000000000..6c7834608f
--- /dev/null
+++ b/lib/fastmem/meson.build
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2026 Ericsson AB
+
+sources = files('rte_fastmem.c')
+headers = files('rte_fastmem.h')
+deps += ['eal']
diff --git a/lib/fastmem/rte_fastmem.c b/lib/fastmem/rte_fastmem.c
new file mode 100644
index 0000000000..84d97ac36f
--- /dev/null
+++ b/lib/fastmem/rte_fastmem.c
@@ -0,0 +1,1694 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#include <errno.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/queue.h>
+
+#include <eal_export.h>
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_eal.h>
+#include <rte_errno.h>
+#include <rte_lcore.h>
+#include <rte_log.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_spinlock.h>
+
+#include <rte_fastmem.h>
+
+RTE_LOG_REGISTER_DEFAULT(fastmem_logtype, NOTICE);
+
+#define RTE_LOGTYPE_FASTMEM fastmem_logtype
+
+#define FASTMEM_LOG(level, ...) \
+ RTE_LOG_LINE(level, FASTMEM, "" __VA_ARGS__)
+
+#define FASTMEM_MEMZONE_SIZE_LOG2 27 /* 128 MiB */
+#define FASTMEM_MEMZONE_SIZE ((size_t)1 << FASTMEM_MEMZONE_SIZE_LOG2)
+
+#define FASTMEM_SLAB_SIZE_LOG2 21 /* 2 MiB */
+#define FASTMEM_SLAB_SIZE ((size_t)1 << FASTMEM_SLAB_SIZE_LOG2)
+#define FASTMEM_SLAB_MASK (FASTMEM_SLAB_SIZE - 1)
+
+#define FASTMEM_SLABS_PER_MEMZONE (FASTMEM_MEMZONE_SIZE / FASTMEM_SLAB_SIZE)
+
+#define FASTMEM_MAX_MEMZONES_PER_SOCKET 64
+
+#define FASTMEM_MIN_CLASS_LOG2 3 /* 8 B */
+#define FASTMEM_MAX_CLASS_LOG2 20 /* 1 MiB */
+#define FASTMEM_N_CLASSES (FASTMEM_MAX_CLASS_LOG2 - FASTMEM_MIN_CLASS_LOG2 + 1)
+
+#define FASTMEM_MIN_SIZE ((size_t)1 << FASTMEM_MIN_CLASS_LOG2)
+#define FASTMEM_MAX_ALLOC_SIZE ((size_t)1 << FASTMEM_MAX_CLASS_LOG2)
+
+#define FASTMEM_SLAB_HEADER_SIZE RTE_CACHE_LINE_SIZE
+
+#define FASTMEM_CACHE_BASE_CAPACITY 64
+#define FASTMEM_CACHE_FLOOR_CAPACITY 4
+#define FASTMEM_CACHE_BASE_CLASS_LOG2 12 /* 4 KiB */
+
+struct fastmem_bin;
+
+/*
+ * Slab header at offset 0 of each 2 MiB slab. Either free (linked
+ * via next_free) or assigned to a bin (linked via list).
+ */
+struct fastmem_slab {
+ struct fastmem_bin *bin;
+ void *free_head;
+ uint32_t free_count;
+ uint32_t n_slots;
+ struct fastmem_slab *next_free;
+ TAILQ_ENTRY(fastmem_slab) list;
+ rte_iova_t iova_base;
+} __rte_aligned(FASTMEM_SLAB_HEADER_SIZE);
+
+TAILQ_HEAD(fastmem_slab_list, fastmem_slab);
+
+struct fastmem_bin {
+ rte_spinlock_t lock;
+ uint32_t slot_size;
+ uint32_t slots_per_slab;
+ uint32_t class_idx;
+ struct fastmem_slab_list partial;
+ struct fastmem_slab_list full;
+ int socket_id;
+ uint64_t slab_acquires;
+ uint64_t slab_releases;
+ uint32_t slabs_partial;
+ uint32_t slabs_full;
+};
+
+/* Per-(lcore, class, socket) bounded LIFO of free object pointers. */
+struct fastmem_cache {
+ uint32_t count;
+ uint32_t capacity;
+ uint32_t target;
+ uint64_t alloc_cache_hits;
+ uint64_t alloc_cache_misses;
+ uint64_t alloc_nomem;
+ uint64_t free_cache_hits;
+ uint64_t free_cache_misses;
+ void *objs[];
+} __rte_cache_aligned;
+
+struct fastmem_socket_state {
+ rte_spinlock_t lock;
+ struct fastmem_slab *free_head;
+ size_t reserved_bytes;
+ size_t memory_limit;
+ unsigned int n_memzones;
+ unsigned int memzone_seq;
+ const struct rte_memzone *memzones[FASTMEM_MAX_MEMZONES_PER_SOCKET];
+ struct fastmem_bin bins[FASTMEM_N_CLASSES];
+ struct fastmem_cache *caches[RTE_MAX_LCORE][FASTMEM_N_CLASSES];
+};
+
+struct fastmem {
+ struct fastmem_socket_state sockets[RTE_MAX_NUMA_NODES];
+};
+
+static struct fastmem *fastmem;
+static const struct rte_memzone *fastmem_mz;
+static bool fastmem_is_primary; /* cached; avoids function call on hot path */
+
+static struct fastmem *
+fastmem_get(void)
+{
+ const struct rte_memzone *mz;
+
+ if (likely(fastmem != NULL))
+ return fastmem;
+
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ rte_errno = ENODEV;
+ return NULL;
+ }
+
+ mz = rte_memzone_lookup("fastmem_state");
+ if (mz == NULL) {
+ rte_errno = ENODEV;
+ return NULL;
+ }
+
+ fastmem_mz = mz;
+ fastmem = mz->addr;
+ return fastmem;
+}
+
+static inline unsigned int
+size_to_class(size_t size, size_t align)
+{
+ size_t effective;
+ unsigned int log2;
+
+ effective = size < FASTMEM_MIN_SIZE ? FASTMEM_MIN_SIZE : size;
+ if (align > effective)
+ effective = align;
+
+ log2 = 64u - rte_clz64(effective - 1);
+
+ if (log2 < FASTMEM_MIN_CLASS_LOG2)
+ log2 = FASTMEM_MIN_CLASS_LOG2;
+ if (log2 > FASTMEM_MAX_CLASS_LOG2)
+ return FASTMEM_N_CLASSES;
+
+ return log2 - FASTMEM_MIN_CLASS_LOG2;
+}
+
+static inline size_t
+class_size(unsigned int class_idx)
+{
+ return (size_t)1 << (class_idx + FASTMEM_MIN_CLASS_LOG2);
+}
+
+static_assert(sizeof(struct fastmem_slab) == FASTMEM_SLAB_HEADER_SIZE,
+ "fastmem slab header must fit in exactly one cache line");
+static_assert(sizeof(struct fastmem_slab) <= FASTMEM_SLAB_SIZE,
+ "slab header larger than a slab makes no sense");
+
+static __rte_always_inline struct fastmem_slab *
+slab_of(void *obj)
+{
+ return (struct fastmem_slab *)
+ ((uintptr_t)obj & ~(uintptr_t)FASTMEM_SLAB_MASK);
+}
+
+static inline size_t
+slab_slot0_offset(size_t class_size)
+{
+ return class_size < FASTMEM_SLAB_HEADER_SIZE ?
+ FASTMEM_SLAB_HEADER_SIZE : class_size;
+}
+
+static inline uint32_t
+slab_slot_count(size_t class_size)
+{
+ size_t offset = slab_slot0_offset(class_size);
+
+ return (uint32_t)((FASTMEM_SLAB_SIZE - offset) / class_size);
+}
+
+/* Must be called with bin->lock held. */
+static void
+slab_init(struct fastmem_bin *bin, struct fastmem_slab *slab)
+{
+ size_t slot_size = bin->slot_size;
+ size_t offset = slab_slot0_offset(slot_size);
+ uint32_t n = bin->slots_per_slab;
+ void *prev = NULL;
+ uint32_t i;
+
+ slab->bin = bin;
+ slab->n_slots = n;
+ slab->free_count = n;
+
+ /* Build in reverse so pops yield sequential addresses. */
+ for (i = 0; i < n; i++) {
+ void *slot = RTE_PTR_ADD(slab, offset + i * slot_size);
+ *(void **)slot = prev;
+ prev = slot;
+ }
+ slab->free_head = prev;
+}
+
+static int
+grow_socket(struct fastmem_socket_state *socket, int socket_id)
+{
+ char name[RTE_MEMZONE_NAMESIZE];
+ const struct rte_memzone *mz;
+ unsigned int i;
+
+ if (socket->reserved_bytes + FASTMEM_MEMZONE_SIZE > socket->memory_limit) {
+ FASTMEM_LOG(ERR,
+ "reserve would exceed memory_limit (%zu) on socket %d",
+ socket->memory_limit, socket_id);
+ return -ENOMEM;
+ }
+
+ if (socket->n_memzones == FASTMEM_MAX_MEMZONES_PER_SOCKET) {
+ FASTMEM_LOG(ERR,
+ "reached per-socket memzone cap (%u) on socket %d",
+ FASTMEM_MAX_MEMZONES_PER_SOCKET, socket_id);
+ return -ENOMEM;
+ }
+
+ snprintf(name, sizeof(name), "fastmem_%d_%u", socket_id,
+ socket->memzone_seq++);
+
+ mz = rte_memzone_reserve_aligned(name, FASTMEM_MEMZONE_SIZE,
+ socket_id, RTE_MEMZONE_IOVA_CONTIG,
+ FASTMEM_SLAB_SIZE);
+ if (mz == NULL) {
+ FASTMEM_LOG(ERR,
+ "failed to reserve %zu-byte memzone '%s' on socket %d: %s",
+ (size_t)FASTMEM_MEMZONE_SIZE, name, socket_id,
+ rte_strerror(rte_errno));
+ return -ENOMEM;
+ }
+
+ socket->memzones[socket->n_memzones++] = mz;
+ socket->reserved_bytes += FASTMEM_MEMZONE_SIZE;
+
+ for (i = 0; i < FASTMEM_SLABS_PER_MEMZONE; i++) {
+ struct fastmem_slab *slab = RTE_PTR_ADD(mz->addr,
+ i * FASTMEM_SLAB_SIZE);
+
+ slab->iova_base = mz->iova + i * FASTMEM_SLAB_SIZE;
+ slab->next_free = socket->free_head;
+ socket->free_head = slab;
+ }
+
+ FASTMEM_LOG(DEBUG,
+ "reserved memzone '%s' (%zu bytes) on socket %d; %zu slabs added",
+ name, (size_t)FASTMEM_MEMZONE_SIZE, socket_id,
+ (size_t)FASTMEM_SLABS_PER_MEMZONE);
+
+ return 0;
+}
+
+static struct fastmem_slab *
+slab_acquire(struct fastmem_socket_state *socket, int socket_id)
+{
+ struct fastmem_slab *slab;
+
+ rte_spinlock_lock(&socket->lock);
+
+ if (socket->free_head == NULL) {
+ int rc = grow_socket(socket, socket_id);
+
+ if (rc < 0) {
+ rte_spinlock_unlock(&socket->lock);
+ return NULL;
+ }
+ }
+
+ slab = socket->free_head;
+ socket->free_head = slab->next_free;
+ slab->next_free = NULL;
+
+ rte_spinlock_unlock(&socket->lock);
+
+ return slab;
+}
+
+static void
+slab_release(struct fastmem_socket_state *socket,
+ struct fastmem_slab *slab)
+{
+ rte_spinlock_lock(&socket->lock);
+
+ slab->next_free = socket->free_head;
+ socket->free_head = slab;
+
+ rte_spinlock_unlock(&socket->lock);
+}
+
+static void
+bin_init(struct fastmem_bin *bin, unsigned int class_idx, int socket_id)
+{
+ size_t slot_size = class_size(class_idx);
+
+ rte_spinlock_init(&bin->lock);
+ bin->slot_size = (uint32_t)slot_size;
+ bin->slots_per_slab = slab_slot_count(slot_size);
+ bin->class_idx = class_idx;
+ TAILQ_INIT(&bin->partial);
+ TAILQ_INIT(&bin->full);
+ bin->socket_id = socket_id;
+ bin->slab_acquires = 0;
+ bin->slab_releases = 0;
+ bin->slabs_partial = 0;
+ bin->slabs_full = 0;
+}
+
+static void
+bin_release(struct fastmem_bin *bin, struct fastmem_socket_state *socket)
+{
+ struct fastmem_slab *slab;
+
+ while ((slab = TAILQ_FIRST(&bin->partial)) != NULL) {
+ TAILQ_REMOVE(&bin->partial, slab, list);
+ slab_release(socket, slab);
+ }
+ while ((slab = TAILQ_FIRST(&bin->full)) != NULL) {
+ TAILQ_REMOVE(&bin->full, slab, list);
+ slab_release(socket, slab);
+ }
+}
+
+static unsigned int
+bin_pop_locked(struct fastmem_bin *bin, void **objs, unsigned int n)
+{
+ unsigned int got = 0;
+
+ while (got < n) {
+ struct fastmem_slab *slab = TAILQ_FIRST(&bin->partial);
+ void *obj;
+
+ if (slab == NULL)
+ break;
+
+ obj = slab->free_head;
+ slab->free_head = *(void **)obj;
+ slab->free_count--;
+ objs[got++] = obj;
+
+ if (slab->free_count == 0) {
+ TAILQ_REMOVE(&bin->partial, slab, list);
+ TAILQ_INSERT_HEAD(&bin->full, slab, list);
+ bin->slabs_partial--;
+ bin->slabs_full++;
+ }
+ }
+
+ return got;
+}
+
+/*
+ * Fully-drained slabs are accumulated in @p to_release for the
+ * caller to return after dropping the lock.
+ */
+static unsigned int
+bin_push_locked(struct fastmem_bin *bin, void **objs, unsigned int n,
+ struct fastmem_slab **to_release)
+{
+ unsigned int n_release = 0;
+ unsigned int i;
+
+ for (i = 0; i < n; i++) {
+ void *obj = objs[i];
+ struct fastmem_slab *slab = (struct fastmem_slab *)
+ ((uintptr_t)obj & ~(uintptr_t)FASTMEM_SLAB_MASK);
+ bool was_full = slab->free_count == 0;
+
+ *(void **)obj = slab->free_head;
+ slab->free_head = obj;
+ slab->free_count++;
+
+ if (was_full) {
+ TAILQ_REMOVE(&bin->full, slab, list);
+ TAILQ_INSERT_HEAD(&bin->partial, slab, list);
+ bin->slabs_full--;
+ bin->slabs_partial++;
+ }
+
+ if (slab->free_count == slab->n_slots) {
+ TAILQ_REMOVE(&bin->partial, slab, list);
+ bin->slabs_partial--;
+ bin->slab_releases++;
+ to_release[n_release++] = slab;
+ }
+ }
+
+ return n_release;
+}
+
+static void *
+bin_alloc_one(struct fastmem_bin *bin)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[bin->socket_id];
+ void *obj;
+
+ rte_spinlock_lock(&bin->lock);
+
+ while (bin_pop_locked(bin, &obj, 1) == 0) {
+ struct fastmem_slab *slab;
+
+ if (TAILQ_FIRST(&bin->partial) != NULL)
+ continue;
+
+ rte_spinlock_unlock(&bin->lock);
+
+ slab = slab_acquire(socket, bin->socket_id);
+ if (slab == NULL) {
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ rte_spinlock_lock(&bin->lock);
+
+ if (unlikely(TAILQ_FIRST(&bin->partial) != NULL)) {
+ /* Release surplus slab without holding bin->lock. */
+ rte_spinlock_unlock(&bin->lock);
+ slab_release(socket, slab);
+ rte_spinlock_lock(&bin->lock);
+ } else {
+ slab_init(bin, slab);
+ TAILQ_INSERT_HEAD(&bin->partial, slab, list);
+ bin->slabs_partial++;
+ bin->slab_acquires++;
+ }
+ }
+
+ rte_spinlock_unlock(&bin->lock);
+
+ return obj;
+}
+
+static unsigned int
+bin_alloc_bulk(struct fastmem_bin *bin, void **objs, unsigned int n)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[bin->socket_id];
+ unsigned int got = 0;
+
+ rte_spinlock_lock(&bin->lock);
+
+ while (got < n) {
+ struct fastmem_slab *slab;
+
+ got += bin_pop_locked(bin, objs + got, n - got);
+ if (got == n)
+ break;
+
+ if (TAILQ_FIRST(&bin->partial) != NULL)
+ continue;
+
+ rte_spinlock_unlock(&bin->lock);
+
+ slab = slab_acquire(socket, bin->socket_id);
+ if (slab == NULL) {
+ rte_spinlock_lock(&bin->lock);
+ break;
+ }
+
+ rte_spinlock_lock(&bin->lock);
+
+ if (unlikely(TAILQ_FIRST(&bin->partial) != NULL)) {
+ /* Release surplus slab without holding bin->lock. */
+ rte_spinlock_unlock(&bin->lock);
+ slab_release(socket, slab);
+ rte_spinlock_lock(&bin->lock);
+ } else {
+ slab_init(bin, slab);
+ TAILQ_INSERT_HEAD(&bin->partial, slab, list);
+ bin->slabs_partial++;
+ bin->slab_acquires++;
+ }
+ }
+
+ rte_spinlock_unlock(&bin->lock);
+
+ return got;
+}
+
+static void
+bin_free_one(struct fastmem_bin *bin, void *obj)
+{
+ unsigned int n_release;
+ struct fastmem_slab *slab_to_release = NULL;
+ struct fastmem_socket_state *socket;
+
+ rte_spinlock_lock(&bin->lock);
+ n_release = bin_push_locked(bin, &obj, 1, &slab_to_release);
+ rte_spinlock_unlock(&bin->lock);
+
+ if (n_release > 0) {
+ socket = &fastmem->sockets[bin->socket_id];
+ slab_release(socket, slab_to_release);
+ }
+}
+
+static void
+bin_free_bulk(struct fastmem_bin *bin, void **objs, unsigned int n)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[bin->socket_id];
+ struct fastmem_slab *to_release[FASTMEM_CACHE_BASE_CAPACITY];
+ unsigned int n_release;
+ unsigned int i;
+
+ RTE_VERIFY(n <= RTE_DIM(to_release));
+
+ rte_spinlock_lock(&bin->lock);
+ n_release = bin_push_locked(bin, objs, n, to_release);
+ rte_spinlock_unlock(&bin->lock);
+
+ for (i = 0; i < n_release; i++)
+ slab_release(socket, to_release[i]);
+}
+
+static inline unsigned int
+cache_capacity(unsigned int class_idx)
+{
+ unsigned int class_log2 = class_idx + FASTMEM_MIN_CLASS_LOG2;
+ unsigned int shift;
+ unsigned int cap;
+
+ if (class_log2 <= FASTMEM_CACHE_BASE_CLASS_LOG2)
+ return FASTMEM_CACHE_BASE_CAPACITY;
+
+ shift = class_log2 - FASTMEM_CACHE_BASE_CLASS_LOG2;
+ cap = FASTMEM_CACHE_BASE_CAPACITY >> shift;
+
+ return cap < FASTMEM_CACHE_FLOOR_CAPACITY ?
+ FASTMEM_CACHE_FLOOR_CAPACITY : cap;
+}
+
+static inline struct fastmem_cache **
+cache_slot(struct fastmem_socket_state *socket, unsigned int class_idx,
+ unsigned int lcore_id)
+{
+ if (lcore_id >= RTE_MAX_LCORE)
+ return NULL;
+ return &socket->caches[lcore_id][class_idx];
+}
+
+static struct fastmem_cache *
+cache_create(struct fastmem_socket_state *socket,
+ unsigned int class_idx, unsigned int lcore_id)
+{
+ struct fastmem_cache **slot = cache_slot(socket, class_idx, lcore_id);
+ struct fastmem_cache *cache;
+ unsigned int capacity;
+ size_t cache_size;
+ unsigned int cache_class;
+ unsigned int own_socket;
+ struct fastmem_socket_state *alloc_socket;
+
+ if (slot == NULL)
+ return NULL;
+
+ cache = *slot;
+ if (cache != NULL)
+ return cache;
+
+ capacity = cache_capacity(class_idx);
+ cache_size = sizeof(*cache) + capacity * sizeof(void *);
+
+ /*
+ * Allocate the cache struct from fastmem on the calling
+ * lcore's socket (NUMA-local to the writer). Bypasses the
+ * cache layer to avoid recursion.
+ */
+ cache_class = size_to_class(cache_size, RTE_CACHE_LINE_SIZE);
+ own_socket = rte_socket_id();
+
+ if (cache_class >= FASTMEM_N_CLASSES) {
+ FASTMEM_LOG(ERR,
+ "cache size %zu exceeds max size class",
+ cache_size);
+ return NULL;
+ }
+
+ if (own_socket >= RTE_MAX_NUMA_NODES)
+ own_socket = (unsigned int)socket->bins[0].socket_id;
+
+ alloc_socket = &fastmem->sockets[own_socket];
+
+ cache = bin_alloc_one(&alloc_socket->bins[cache_class]);
+ if (cache == NULL) {
+ FASTMEM_LOG(ERR,
+ "failed to allocate cache for class %u on socket %u",
+ class_idx, own_socket);
+ return NULL;
+ }
+
+ cache->count = 0;
+ cache->capacity = capacity;
+ cache->target = capacity / 2;
+ cache->alloc_cache_hits = 0;
+ cache->alloc_cache_misses = 0;
+ cache->alloc_nomem = 0;
+ cache->free_cache_hits = 0;
+ cache->free_cache_misses = 0;
+
+ *slot = cache;
+
+ return cache;
+}
+
+static __rte_always_inline struct fastmem_cache *
+cache_get(struct fastmem_socket_state *socket, unsigned int class_idx,
+ unsigned int lcore_id)
+{
+ struct fastmem_cache **slot;
+ struct fastmem_cache *cache;
+
+ if (unlikely(!fastmem_is_primary))
+ return NULL;
+
+ slot = cache_slot(socket, class_idx, lcore_id);
+
+ if (slot == NULL)
+ return NULL;
+
+ cache = *slot;
+ if (cache != NULL)
+ return cache;
+
+ return cache_create(socket, class_idx, lcore_id);
+}
+
+static __rte_always_inline void *
+cache_pop(struct fastmem_cache *cache, struct fastmem_bin *bin)
+{
+ if (cache->count > 0) {
+ cache->alloc_cache_hits++;
+ return cache->objs[--cache->count];
+ }
+
+ cache->count = bin_alloc_bulk(bin, cache->objs, cache->target);
+ if (cache->count == 0)
+ return NULL;
+
+ cache->alloc_cache_misses++;
+ return cache->objs[--cache->count];
+}
+
+static __rte_always_inline void
+cache_push(struct fastmem_cache *cache, struct fastmem_bin *bin, void *obj)
+{
+ unsigned int drain;
+
+ if (cache->count < cache->capacity) {
+ cache->free_cache_hits++;
+ cache->objs[cache->count++] = obj;
+ return;
+ }
+
+ cache->free_cache_misses++;
+
+ /*
+ * Drain the oldest (bottom) half to the bin, keeping the
+ * newest (top) half for temporal reuse.
+ */
+ drain = cache->count - cache->target;
+ bin_free_bulk(bin, cache->objs, drain);
+ memmove(cache->objs, cache->objs + drain,
+ cache->target * sizeof(cache->objs[0]));
+ cache->count = cache->target;
+
+ cache->objs[cache->count++] = obj;
+}
+
+static void
+socket_release_caches(struct fastmem_socket_state *socket)
+{
+ unsigned int lcore;
+ unsigned int c;
+
+ for (lcore = 0; lcore < RTE_MAX_LCORE; lcore++) {
+ for (c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_cache *cache = socket->caches[lcore][c];
+ struct fastmem_slab *cache_slab;
+
+ if (cache == NULL)
+ continue;
+
+ if (cache->count > 0) {
+ bin_free_bulk(&socket->bins[c],
+ cache->objs, cache->count);
+ cache->count = 0;
+ }
+
+ cache_slab = slab_of(cache);
+ bin_free_one(cache_slab->bin, cache);
+
+ socket->caches[lcore][c] = NULL;
+ }
+ }
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_init, 24.11)
+rte_fastmem_init(void)
+{
+ unsigned int s, c;
+
+ if (fastmem != NULL)
+ return -EBUSY;
+
+ fastmem_mz = rte_memzone_reserve_aligned("fastmem_state",
+ sizeof(*fastmem), SOCKET_ID_ANY, 0,
+ RTE_CACHE_LINE_SIZE);
+ if (fastmem_mz == NULL)
+ return -ENOMEM;
+
+ fastmem = fastmem_mz->addr;
+ fastmem_is_primary = true;
+ memset(fastmem, 0, sizeof(*fastmem));
+
+ for (s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ rte_spinlock_init(&socket->lock);
+ socket->memory_limit = SIZE_MAX;
+
+ for (c = 0; c < FASTMEM_N_CLASSES; c++)
+ bin_init(&socket->bins[c], c, (int)s);
+ }
+
+ return 0;
+}
+
+static void
+release_socket_caches(struct fastmem_socket_state *socket)
+{
+ socket_release_caches(socket);
+}
+
+static void
+release_socket_bins(struct fastmem_socket_state *socket)
+{
+ unsigned int c;
+
+ for (c = 0; c < FASTMEM_N_CLASSES; c++)
+ bin_release(&socket->bins[c], socket);
+}
+
+static void
+release_socket_memzones(struct fastmem_socket_state *socket)
+{
+ unsigned int i;
+
+ for (i = 0; i < socket->n_memzones; i++)
+ rte_memzone_free(socket->memzones[i]);
+
+ socket->free_head = NULL;
+ socket->reserved_bytes = 0;
+ socket->n_memzones = 0;
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_deinit, 24.11)
+rte_fastmem_deinit(void)
+{
+ unsigned int i;
+
+ if (fastmem == NULL)
+ return;
+
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ fastmem = NULL;
+ fastmem_mz = NULL;
+ return;
+ }
+
+ for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ release_socket_caches(&fastmem->sockets[i]);
+
+ for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ release_socket_bins(&fastmem->sockets[i]);
+
+ for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ release_socket_memzones(&fastmem->sockets[i]);
+
+ rte_memzone_free(fastmem_mz);
+ fastmem_mz = NULL;
+ fastmem = NULL;
+}
+
+/* Same resolution order as rte_malloc's malloc_get_numa_socket(). */
+static __rte_always_inline unsigned int
+local_socket_id(void)
+{
+ int sid = (int)rte_socket_id();
+
+ if (likely(sid >= 0 && sid < RTE_MAX_NUMA_NODES))
+ return sid;
+
+ sid = (int)rte_lcore_to_socket_id(rte_get_main_lcore());
+ if (likely(sid >= 0 && sid < RTE_MAX_NUMA_NODES))
+ return sid;
+
+ sid = rte_socket_id_by_idx(0);
+ if (likely(sid >= 0 && sid < RTE_MAX_NUMA_NODES))
+ return sid;
+
+ return 0;
+}
+
+static int
+reserve_on_socket(int sid, size_t size)
+{
+ struct fastmem_socket_state *socket = &fastmem->sockets[sid];
+ int rc = 0;
+
+ rte_spinlock_lock(&socket->lock);
+
+ while (socket->reserved_bytes < size) {
+ rc = grow_socket(socket, sid);
+ if (rc < 0)
+ break;
+ }
+
+ rte_spinlock_unlock(&socket->lock);
+
+ return rc;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_reserve, 24.11)
+rte_fastmem_reserve(size_t size, int socket_id)
+{
+ unsigned int i;
+ int rc;
+
+ if (fastmem == NULL)
+ return -EINVAL;
+
+ if (socket_id != SOCKET_ID_ANY) {
+ if (socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return -EINVAL;
+ return reserve_on_socket(socket_id, size);
+ }
+
+ rc = reserve_on_socket(local_socket_id(), size);
+ if (rc == 0)
+ return 0;
+
+ for (i = 0; i < rte_socket_count(); i++) {
+ int sid = rte_socket_id_by_idx(i);
+
+ if (sid < 0 || (unsigned int)sid == local_socket_id())
+ continue;
+
+ rc = reserve_on_socket(sid, size);
+ if (rc == 0)
+ return 0;
+ }
+
+ return rc;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_set_limit, 24.11)
+rte_fastmem_set_limit(int socket_id, size_t max_bytes)
+{
+ if (fastmem == NULL)
+ return -EINVAL;
+
+ if (socket_id == SOCKET_ID_ANY) {
+ for (unsigned int i = 0; i < RTE_MAX_NUMA_NODES; i++)
+ fastmem->sockets[i].memory_limit = max_bytes;
+ return 0;
+ }
+
+ if (socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return -EINVAL;
+
+ fastmem->sockets[socket_id].memory_limit = max_bytes;
+ return 0;
+}
+
+size_t
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_get_limit, 24.11)
+rte_fastmem_get_limit(int socket_id)
+{
+ if (fastmem == NULL || socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return 0;
+
+ return fastmem->sockets[socket_id].memory_limit;
+}
+
+size_t
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_max_size, 24.11)
+rte_fastmem_max_size(void)
+{
+ return FASTMEM_MAX_ALLOC_SIZE;
+}
+
+static __rte_always_inline void *
+alloc_from_socket(struct fastmem_socket_state *socket,
+ unsigned int class_idx, unsigned int lcore_id)
+{
+ struct fastmem_cache *cache;
+
+ cache = cache_get(socket, class_idx, lcore_id);
+ if (likely(cache != NULL))
+ return cache_pop(cache, &socket->bins[class_idx]);
+ return bin_alloc_one(&socket->bins[class_idx]);
+}
+
+static __rte_always_inline void
+do_free(void *ptr)
+{
+ struct fastmem_slab *slab;
+ struct fastmem_bin *bin;
+ struct fastmem_socket_state *socket;
+ unsigned int lcore_id;
+ struct fastmem_cache *cache;
+
+ slab = slab_of(ptr);
+ bin = slab->bin;
+ socket = &fastmem->sockets[bin->socket_id];
+
+ lcore_id = rte_lcore_id();
+ cache = cache_get(socket, bin->class_idx, lcore_id);
+ if (likely(cache != NULL))
+ cache_push(cache, bin, ptr);
+ else
+ bin_free_one(bin, ptr);
+}
+
+static __rte_always_inline int
+do_alloc_bulk(void **ptrs, unsigned int n, size_t size, size_t align,
+ unsigned int flags, unsigned int lcore_id,
+ int socket_id, bool fallback)
+{
+ unsigned int class_idx;
+ struct fastmem_socket_state *socket;
+ struct fastmem_cache *cache;
+ unsigned int got = 0;
+
+ if (unlikely(fastmem_get() == NULL))
+ return -rte_errno;
+
+ if (align == 0)
+ align = RTE_CACHE_LINE_SIZE;
+ else if (unlikely((align & (align - 1)) != 0)) {
+ rte_errno = EINVAL;
+ return -EINVAL;
+ }
+
+ class_idx = size_to_class(size, align);
+ if (unlikely(class_idx >= FASTMEM_N_CLASSES)) {
+ rte_errno = E2BIG;
+ return -E2BIG;
+ }
+
+ socket = &fastmem->sockets[socket_id];
+ cache = cache_get(socket, class_idx, lcore_id);
+
+ if (likely(cache != NULL)) {
+ /* Drain from cache. */
+ unsigned int avail = RTE_MIN(cache->count, n);
+
+ cache->count -= avail;
+ memcpy(ptrs, &cache->objs[cache->count],
+ avail * sizeof(void *));
+ got = avail;
+ cache->alloc_cache_hits += avail;
+
+ if (got < n) {
+ unsigned int need = n - got;
+ unsigned int want = RTE_MAX(need, cache->target);
+ unsigned int filled;
+
+ if (want <= cache->capacity) {
+ /* Refill into cache, give caller their share. */
+ filled = bin_alloc_bulk(
+ &socket->bins[class_idx],
+ cache->objs, want);
+ if (filled > 0) {
+ cache->alloc_cache_misses += RTE_MIN(filled, need);
+ }
+ if (filled >= need) {
+ memcpy(ptrs + got,
+ cache->objs + filled - need,
+ need * sizeof(void *));
+ cache->count = filled - need;
+ got = n;
+ } else {
+ memcpy(ptrs + got, cache->objs,
+ filled * sizeof(void *));
+ got += filled;
+ cache->count = 0;
+ }
+ } else {
+ /* n exceeds cache capacity; pull directly. */
+ unsigned int direct = bin_alloc_bulk(
+ &socket->bins[class_idx],
+ ptrs + got, need);
+ if (direct > 0)
+ cache->alloc_cache_misses += direct;
+ got += direct;
+ }
+ }
+ } else {
+ got = bin_alloc_bulk(&socket->bins[class_idx], ptrs, n);
+ }
+
+ if (unlikely(got < n) && fallback) {
+ unsigned int i;
+
+ for (i = 0; i < rte_socket_count() && got < n; i++) {
+ int sid = rte_socket_id_by_idx(i);
+
+ if (sid < 0 || sid == socket_id)
+ continue;
+
+ socket = &fastmem->sockets[sid];
+ cache = cache_get(socket, class_idx, lcore_id);
+ if (likely(cache != NULL)) {
+ unsigned int avail =
+ RTE_MIN(cache->count, n - got);
+ cache->count -= avail;
+ memcpy(ptrs + got,
+ &cache->objs[cache->count],
+ avail * sizeof(void *));
+ cache->alloc_cache_hits += avail;
+ got += avail;
+ }
+ if (got < n) {
+ unsigned int direct = bin_alloc_bulk(
+ &socket->bins[class_idx],
+ ptrs + got, n - got);
+ if (direct > 0 && cache != NULL)
+ cache->alloc_cache_misses += direct;
+ got += direct;
+ }
+ }
+ }
+
+ if (unlikely(got < n)) {
+ /* All-or-nothing: return what we got. */
+ struct fastmem_cache **slot;
+ unsigned int i;
+
+ for (i = 0; i < got; i++)
+ do_free(ptrs[i]);
+
+ slot = cache_slot(
+ &fastmem->sockets[socket_id], class_idx,
+ lcore_id);
+ if (slot != NULL && *slot != NULL)
+ (*slot)->alloc_nomem++;
+ rte_errno = ENOMEM;
+ return -ENOMEM;
+ }
+
+ if (flags & RTE_FASTMEM_F_ZERO) {
+ size_t cs = class_size(class_idx);
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ memset(ptrs[i], 0, cs);
+ }
+
+ return 0;
+}
+
+static __rte_always_inline void *
+do_alloc(size_t size, size_t align, unsigned int flags,
+ unsigned int lcore_id, int socket_id, bool fallback)
+{
+ unsigned int class_idx;
+ struct fastmem_cache **slot;
+ void *obj;
+
+ if (unlikely(fastmem_get() == NULL))
+ return NULL;
+
+ if (align == 0)
+ align = RTE_CACHE_LINE_SIZE;
+ else if (unlikely((align & (align - 1)) != 0)) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ class_idx = size_to_class(size, align);
+ if (unlikely(class_idx >= FASTMEM_N_CLASSES)) {
+ rte_errno = E2BIG;
+ return NULL;
+ }
+
+ obj = alloc_from_socket(&fastmem->sockets[socket_id],
+ class_idx, lcore_id);
+
+ if (likely(obj != NULL))
+ goto out;
+
+ if (fallback) {
+ unsigned int i;
+
+ for (i = 0; i < rte_socket_count(); i++) {
+ int sid = rte_socket_id_by_idx(i);
+
+ if (sid < 0 || sid == socket_id)
+ continue;
+
+ obj = alloc_from_socket(&fastmem->sockets[sid],
+ class_idx, lcore_id);
+ if (obj != NULL)
+ goto out;
+ }
+ }
+
+ slot = cache_slot(
+ &fastmem->sockets[socket_id], class_idx, lcore_id);
+ if (slot != NULL && *slot != NULL)
+ (*slot)->alloc_nomem++;
+ rte_errno = ENOMEM;
+ return NULL;
+
+out:
+ if (flags & RTE_FASTMEM_F_ZERO)
+ memset(obj, 0, class_size(class_idx));
+
+ return obj;
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc, 24.11)
+rte_fastmem_alloc(size_t size, size_t align, unsigned int flags)
+{
+ return do_alloc(size, align, flags, rte_lcore_id(),
+ local_socket_id(), false);
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc_socket, 24.11)
+rte_fastmem_alloc_socket(size_t size, size_t align, unsigned int flags,
+ int socket_id)
+{
+ if (socket_id == SOCKET_ID_ANY)
+ return do_alloc(size, align, flags, rte_lcore_id(),
+ local_socket_id(), true);
+
+ if (unlikely(socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ return do_alloc(size, align, flags, rte_lcore_id(), socket_id, false);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_free, 24.11)
+rte_fastmem_free(void *ptr)
+{
+ if (unlikely(ptr == NULL))
+ return;
+
+ do_free(ptr);
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc_bulk, 24.11)
+rte_fastmem_alloc_bulk(void **ptrs, unsigned int n, size_t size, size_t align,
+ unsigned int flags)
+{
+ return do_alloc_bulk(ptrs, n, size, align, flags,
+ rte_lcore_id(), local_socket_id(), false);
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_alloc_bulk_socket, 24.11)
+rte_fastmem_alloc_bulk_socket(void **ptrs, unsigned int n, size_t size,
+ size_t align, unsigned int flags, int socket_id)
+{
+ if (socket_id == SOCKET_ID_ANY)
+ return do_alloc_bulk(ptrs, n, size, align, flags,
+ rte_lcore_id(), local_socket_id(), true);
+
+ if (unlikely(socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)) {
+ rte_errno = EINVAL;
+ return -EINVAL;
+ }
+
+ return do_alloc_bulk(ptrs, n, size, align, flags,
+ rte_lcore_id(), socket_id, false);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_free_bulk, 24.11)
+rte_fastmem_free_bulk(void **ptrs, unsigned int n)
+{
+ unsigned int lcore_id;
+ struct fastmem_slab *slab;
+ struct fastmem_bin *bin;
+ struct fastmem_socket_state *socket;
+ struct fastmem_cache *cache;
+ unsigned int space;
+ unsigned int i;
+
+ if (unlikely(n == 0))
+ return;
+
+ lcore_id = rte_lcore_id();
+
+ /* Fast path: check if first object gives us the bin. */
+ slab = slab_of(ptrs[0]);
+ bin = slab->bin;
+ socket = &fastmem->sockets[bin->socket_id];
+ cache = cache_get(socket, bin->class_idx, lcore_id);
+
+ if (unlikely(cache == NULL)) {
+ for (i = 0; i < n; i++)
+ do_free(ptrs[i]);
+ return;
+ }
+
+ /*
+ * Try to push all objects into the cache in one memcpy.
+ * If any object belongs to a different bin, fall back to
+ * per-object free for the remainder.
+ */
+ space = cache->capacity - cache->count;
+ if (likely(n <= space)) {
+ /* Verify all same bin (common case). */
+ for (i = 1; i < n; i++) {
+ if (slab_of(ptrs[i])->bin != bin)
+ goto slow;
+ }
+ cache->free_cache_hits += n;
+ memcpy(&cache->objs[cache->count], ptrs,
+ n * sizeof(void *));
+ cache->count += n;
+ return;
+ }
+
+ /* Would overflow cache — drain first, then push. */
+ if (n <= cache->capacity) {
+ unsigned int drain;
+
+ for (i = 1; i < n; i++) {
+ if (slab_of(ptrs[i])->bin != bin)
+ goto slow;
+ }
+
+ cache->free_cache_misses += n;
+ drain = cache->count - cache->target + n;
+ if (drain > cache->count)
+ drain = cache->count;
+ if (drain > 0) {
+ bin_free_bulk(bin, cache->objs, drain);
+ cache->count -= drain;
+ memmove(cache->objs, cache->objs + drain,
+ cache->count * sizeof(cache->objs[0]));
+ }
+ memcpy(&cache->objs[cache->count], ptrs,
+ n * sizeof(void *));
+ cache->count += n;
+ return;
+ }
+
+slow:
+ for (i = 0; i < n; i++)
+ do_free(ptrs[i]);
+}
+
+#define fastmem_handle_class_BITS 8
+
+static inline rte_fastmem_handle_t
+fastmem_handle_pack(unsigned int class_idx, int socket_id)
+{
+ return (uint32_t)class_idx |
+ ((uint32_t)socket_id << fastmem_handle_class_BITS);
+}
+
+static inline unsigned int
+fastmem_handle_class(rte_fastmem_handle_t h)
+{
+ return h & ((1U << fastmem_handle_class_BITS) - 1);
+}
+
+static inline int
+fastmem_handle_socket(rte_fastmem_handle_t h)
+{
+ return (int)(h >> fastmem_handle_class_BITS);
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_hlookup, 24.11)
+rte_fastmem_hlookup(size_t size, size_t align, int socket_id,
+ rte_fastmem_handle_t *handle)
+{
+ unsigned int class_idx;
+ struct fastmem_socket_state *socket;
+
+ if (handle == NULL)
+ return -EINVAL;
+
+ if (align == 0)
+ align = RTE_CACHE_LINE_SIZE;
+ else if ((align & (align - 1)) != 0)
+ return -EINVAL;
+
+ if (socket_id < 0 || socket_id >= RTE_MAX_NUMA_NODES)
+ return -EINVAL;
+
+ class_idx = size_to_class(size, align);
+ if (class_idx >= FASTMEM_N_CLASSES)
+ return -E2BIG;
+
+ /* Pre-create the cache for the calling lcore. */
+ socket = &fastmem->sockets[socket_id];
+ cache_create(socket, class_idx, rte_lcore_id());
+
+ *handle = fastmem_handle_pack(class_idx, socket_id);
+ return 0;
+}
+
+void *
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_halloc, 24.11)
+rte_fastmem_halloc(rte_fastmem_handle_t handle, unsigned int flags)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+ unsigned int lcore_id = rte_lcore_id();
+ struct fastmem_socket_state *socket = &fastmem->sockets[socket_id];
+ struct fastmem_bin *bin = &socket->bins[class_idx];
+ struct fastmem_cache *cache;
+ void *obj;
+
+ RTE_ASSERT(fastmem != NULL);
+ RTE_ASSERT(lcore_id < RTE_MAX_LCORE);
+
+ cache = socket->caches[lcore_id][class_idx];
+ RTE_ASSERT(cache != NULL);
+
+ obj = cache_pop(cache, bin);
+ if (unlikely(obj == NULL)) {
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ if (flags & RTE_FASTMEM_F_ZERO)
+ memset(obj, 0, class_size(class_idx));
+
+ return obj;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_halloc_bulk, 24.11)
+rte_fastmem_halloc_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n, unsigned int flags)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+
+ return do_alloc_bulk(ptrs, n, class_size(class_idx),
+ RTE_CACHE_LINE_SIZE, flags, rte_lcore_id(),
+ socket_id, false);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_hfree, 24.11)
+rte_fastmem_hfree(rte_fastmem_handle_t handle, void *ptr)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+ struct fastmem_socket_state *socket = &fastmem->sockets[socket_id];
+ struct fastmem_bin *bin = &socket->bins[class_idx];
+ unsigned int lcore_id = rte_lcore_id();
+ struct fastmem_cache *cache;
+
+ if (unlikely(ptr == NULL))
+ return;
+
+ RTE_ASSERT(lcore_id < RTE_MAX_LCORE);
+
+ cache = socket->caches[lcore_id][class_idx];
+ RTE_ASSERT(cache != NULL);
+
+ cache_push(cache, bin, ptr);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_hfree_bulk, 24.11)
+rte_fastmem_hfree_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n)
+{
+ unsigned int class_idx = fastmem_handle_class(handle);
+ int socket_id = fastmem_handle_socket(handle);
+ struct fastmem_socket_state *socket = &fastmem->sockets[socket_id];
+ struct fastmem_bin *bin = &socket->bins[class_idx];
+ unsigned int lcore_id;
+ struct fastmem_cache *cache;
+ unsigned int i;
+
+ if (unlikely(n == 0))
+ return;
+
+ lcore_id = rte_lcore_id();
+ cache = cache_get(socket, class_idx, lcore_id);
+
+ if (likely(cache != NULL)) {
+ for (i = 0; i < n; i++)
+ cache_push(cache, bin, ptrs[i]);
+ } else {
+ for (i = 0; i < n; i++)
+ bin_free_one(bin, ptrs[i]);
+ }
+}
+
+rte_iova_t
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_virt2iova, 24.11)
+rte_fastmem_virt2iova(const void *ptr)
+{
+ struct fastmem_slab *slab;
+
+ RTE_ASSERT(fastmem != NULL);
+
+ slab = slab_of((void *)(uintptr_t)ptr);
+
+ return slab->iova_base + ((uintptr_t)ptr - (uintptr_t)slab);
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_cache_flush, 24.11)
+rte_fastmem_cache_flush(void)
+{
+ unsigned int lcore_id;
+ unsigned int s, c;
+
+ if (fastmem == NULL)
+ return;
+
+ lcore_id = rte_lcore_id();
+ if (lcore_id >= RTE_MAX_LCORE)
+ return;
+
+ for (s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ for (c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_cache *cache =
+ socket->caches[lcore_id][c];
+ struct fastmem_slab *cache_slab;
+
+ if (cache == NULL)
+ continue;
+
+ if (cache->count > 0) {
+ bin_free_bulk(&socket->bins[c],
+ cache->objs, cache->count);
+ cache->count = 0;
+ }
+
+ cache_slab = slab_of(cache);
+ bin_free_one(cache_slab->bin, cache);
+
+ socket->caches[lcore_id][c] = NULL;
+ }
+ }
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats, 24.11)
+rte_fastmem_stats(struct rte_fastmem_stats *stats)
+{
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_stats){0};
+ stats->n_classes = FASTMEM_N_CLASSES;
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ stats->bytes_backing += socket->reserved_bytes;
+
+ for (unsigned int c = 0; c < FASTMEM_N_CLASSES; c++) {
+ uint64_t class_allocs = 0, class_frees = 0;
+
+ for (unsigned int l = 0; l < RTE_MAX_LCORE; l++) {
+ struct fastmem_cache *cache =
+ socket->caches[l][c];
+ if (cache == NULL)
+ continue;
+ class_allocs += cache->alloc_cache_hits +
+ cache->alloc_cache_misses;
+ class_frees += cache->free_cache_hits +
+ cache->free_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ }
+ stats->alloc_total += class_allocs;
+ stats->free_total += class_frees;
+ if (class_allocs > class_frees)
+ stats->bytes_in_use += class_size(c) *
+ (class_allocs - class_frees);
+ }
+ }
+
+ return 0;
+}
+
+static inline unsigned int
+exact_class_idx(size_t sz)
+{
+ unsigned int log2;
+
+ if (sz < FASTMEM_MIN_SIZE || sz > FASTMEM_MAX_ALLOC_SIZE)
+ return FASTMEM_N_CLASSES;
+ if ((sz & (sz - 1)) != 0)
+ return FASTMEM_N_CLASSES;
+
+ log2 = (unsigned int)rte_ctz64(sz);
+ if (log2 < FASTMEM_MIN_CLASS_LOG2 || log2 > FASTMEM_MAX_CLASS_LOG2)
+ return FASTMEM_N_CLASSES;
+
+ return log2 - FASTMEM_MIN_CLASS_LOG2;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_class, 24.11)
+rte_fastmem_stats_class(size_t class_size_arg,
+ struct rte_fastmem_class_stats *stats)
+{
+ unsigned int c;
+ uint64_t allocs, frees;
+
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+
+ c = exact_class_idx(class_size_arg);
+ if (c >= FASTMEM_N_CLASSES)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_class_stats){0};
+ stats->class_size = class_size(c);
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+ struct fastmem_bin *bin = &socket->bins[c];
+
+ for (unsigned int l = 0; l < RTE_MAX_LCORE; l++) {
+ struct fastmem_cache *cache = socket->caches[l][c];
+ if (cache == NULL)
+ continue;
+ stats->alloc_cache_hits += cache->alloc_cache_hits;
+ stats->alloc_cache_misses += cache->alloc_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ stats->free_cache_hits += cache->free_cache_hits;
+ stats->free_cache_misses += cache->free_cache_misses;
+ }
+
+ stats->slab_acquires += bin->slab_acquires;
+ stats->slab_releases += bin->slab_releases;
+ stats->slabs_partial += bin->slabs_partial;
+ stats->slabs_full += bin->slabs_full;
+ }
+
+ allocs = stats->alloc_cache_hits + stats->alloc_cache_misses;
+ frees = stats->free_cache_hits + stats->free_cache_misses;
+ if (allocs > frees)
+ stats->in_use = allocs - frees;
+
+ return 0;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_lcore, 24.11)
+rte_fastmem_stats_lcore(unsigned int lcore_id,
+ struct rte_fastmem_lcore_stats *stats)
+{
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+ if (lcore_id >= RTE_MAX_LCORE)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_lcore_stats){0};
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ for (unsigned int c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_cache *cache =
+ socket->caches[lcore_id][c];
+ if (cache == NULL)
+ continue;
+ stats->alloc_cache_hits += cache->alloc_cache_hits;
+ stats->alloc_cache_misses += cache->alloc_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ stats->free_cache_hits += cache->free_cache_hits;
+ stats->free_cache_misses += cache->free_cache_misses;
+ }
+ }
+
+ return 0;
+}
+
+int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_lcore_class, 24.11)
+rte_fastmem_stats_lcore_class(unsigned int lcore_id, size_t class_size_arg,
+ struct rte_fastmem_lcore_class_stats *stats)
+{
+ unsigned int c;
+
+ if (stats == NULL || fastmem == NULL)
+ return -EINVAL;
+ if (lcore_id >= RTE_MAX_LCORE)
+ return -EINVAL;
+
+ c = exact_class_idx(class_size_arg);
+ if (c >= FASTMEM_N_CLASSES)
+ return -EINVAL;
+
+ *stats = (struct rte_fastmem_lcore_class_stats){0};
+ stats->class_size = class_size(c);
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_cache *cache =
+ fastmem->sockets[s].caches[lcore_id][c];
+ if (cache == NULL)
+ continue;
+ stats->alloc_cache_hits += cache->alloc_cache_hits;
+ stats->alloc_cache_misses += cache->alloc_cache_misses;
+ stats->alloc_nomem += cache->alloc_nomem;
+ stats->free_cache_hits += cache->free_cache_hits;
+ stats->free_cache_misses += cache->free_cache_misses;
+ }
+
+ return 0;
+}
+
+void
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_stats_reset, 24.11)
+rte_fastmem_stats_reset(void)
+{
+ if (fastmem == NULL)
+ return;
+
+ for (unsigned int s = 0; s < RTE_MAX_NUMA_NODES; s++) {
+ struct fastmem_socket_state *socket = &fastmem->sockets[s];
+
+ for (unsigned int c = 0; c < FASTMEM_N_CLASSES; c++) {
+ struct fastmem_bin *bin = &socket->bins[c];
+
+ bin->slab_acquires = 0;
+ bin->slab_releases = 0;
+
+ for (unsigned int l = 0; l < RTE_MAX_LCORE; l++) {
+ struct fastmem_cache *cache =
+ socket->caches[l][c];
+ if (cache == NULL)
+ continue;
+ cache->alloc_cache_hits = 0;
+ cache->alloc_cache_misses = 0;
+ cache->alloc_nomem = 0;
+ cache->free_cache_hits = 0;
+ cache->free_cache_misses = 0;
+ }
+ }
+ }
+}
+
+unsigned int
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fastmem_classes, 24.11)
+rte_fastmem_classes(size_t *sizes)
+{
+ if (sizes != NULL)
+ for (unsigned int i = 0; i < FASTMEM_N_CLASSES; i++)
+ sizes[i] = class_size(i);
+ return FASTMEM_N_CLASSES;
+}
diff --git a/lib/fastmem/rte_fastmem.h b/lib/fastmem/rte_fastmem.h
new file mode 100644
index 0000000000..4da893e7f3
--- /dev/null
+++ b/lib/fastmem/rte_fastmem.h
@@ -0,0 +1,774 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Ericsson AB
+ */
+
+#ifndef _RTE_FASTMEM_H_
+#define _RTE_FASTMEM_H_
+
+/**
+ * @file
+ *
+ * RTE Fastmem
+ *
+ * @warning
+ * @b EXPERIMENTAL:
+ * All functions in this file may be changed or removed without prior notice.
+ *
+ * The fastmem library is a fast, general-purpose small-object
+ * allocator for DPDK applications. It is intended to allow an
+ * application to replace its many per-type mempools — each sized
+ * for a single object type (a connection, a session, a work item,
+ * a timer, etc.) — with a single allocator that handles arbitrary
+ * object sizes, grows on demand, and offers mempool-level
+ * performance for the common allocation and free paths.
+ *
+ * Like mempool, fastmem is backed by huge pages, is NUMA-aware,
+ * supports bulk operations, and uses per-lcore caches to reduce
+ * shared-state contention. Unlike mempool, it does not require the
+ * caller to declare object sizes or counts up front.
+ *
+ * There is a single, global fastmem instance per process. The
+ * instance is brought up with rte_fastmem_init() and torn down with
+ * rte_fastmem_deinit(). Allocations are made with
+ * rte_fastmem_alloc() and freed with rte_fastmem_free().
+ *
+ * The allocator is bounded to small-object allocations. Requests
+ * larger than rte_fastmem_max_size() are rejected; callers with
+ * such needs should use rte_malloc() directly.
+ *
+ * Backing memory is reserved from DPDK memzones. Once reserved,
+ * backing memory is not returned to the system during the
+ * allocator's lifetime. Callers that need predictable latency may
+ * pre-reserve backing memory up front using rte_fastmem_reserve(),
+ * avoiding memzone-reservation overhead during steady-state
+ * operation.
+ *
+ * Alignment argument, @c align:
+ * If non-zero, @c align specifies an exact minimum alignment and
+ * must be a power of 2. If zero, the default alignment is
+ * @c RTE_CACHE_LINE_SIZE, so that objects obtained from distinct
+ * calls cannot false-share a cache line.
+ *
+ * Threads and per-lcore caches:
+ * Allocate and free calls from EAL threads are served through a
+ * per-lcore cache, which makes the common path lock-free.
+ * Unregistered non-EAL threads do not use a cache; their
+ * allocate and free calls go directly to shared state, take an
+ * internal lock, and cost more per call.
+ *
+ * Non-preemptible caller:
+ * Callers should not be preemptible while inside a fastmem call.
+ * Fastmem uses internal spinlocks; if a caller is preempted
+ * while holding one, any other thread that subsequently needs
+ * the same lock stalls until the preempted caller resumes.
+ */
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <rte_bitops.h>
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Flag for rte_fastmem_alloc() and its variants: initialize the
+ * returned memory to zero before returning it to the caller.
+ */
+#define RTE_FASTMEM_F_ZERO RTE_BIT32(0)
+
+/**
+ * Initialize the fastmem allocator.
+ *
+ * Sets up the library's internal state. Must be called before any
+ * allocation call. Typically called once per process, after
+ * rte_eal_init() and before the application's worker threads begin
+ * making allocations.
+ *
+ * Initialization does not pre-reserve any backing memory; memzones
+ * are reserved lazily as allocations require. An application that
+ * wants to avoid memzone-reservation latency on the allocation
+ * path should follow rte_fastmem_init() with one or more calls to
+ * rte_fastmem_reserve().
+ *
+ * This function is not thread-safe and must not be called
+ * concurrently with any other fastmem function.
+ *
+ * @return
+ * - 0: Success.
+ * - -EBUSY: The allocator is already initialized.
+ * - -ENOMEM: Unable to allocate internal state.
+ */
+__rte_experimental
+int
+rte_fastmem_init(void);
+
+/**
+ * Tear down the fastmem allocator.
+ *
+ * Releases the library's internal state and frees all backing
+ * memzones. After this call, no fastmem allocations or frees may
+ * be made until rte_fastmem_init() is called again.
+ *
+ * The caller is responsible for ensuring that no fastmem-allocated
+ * objects remain in use. Outstanding allocations at deinit time
+ * result in undefined behavior.
+ *
+ * This function is not thread-safe and must not be called
+ * concurrently with any other fastmem function.
+ */
+__rte_experimental
+void
+rte_fastmem_deinit(void);
+
+/**
+ * Pre-reserve backing memory.
+ *
+ * Ensures that at least @p size bytes of memzone-backed memory are
+ * available to the allocator on @p socket_id, reserving additional
+ * memzones from EAL as needed to reach that total. Subsequent
+ * allocations served from the pre-reserved memory do not incur
+ * memzone-reservation cost.
+ *
+ * The reservation is cumulative: repeated calls to
+ * rte_fastmem_reserve() with the same @p socket_id grow the
+ * reservation monotonically. Reserved memory is never returned to
+ * the system during the allocator's lifetime.
+ *
+ * A typical use is to call rte_fastmem_reserve() once at
+ * application startup, with a size chosen to cover the expected
+ * steady-state working set. Allocations and frees during
+ * steady-state operation then avoid memzone reservations entirely.
+ *
+ * @param size
+ * The minimum amount of backing memory, in bytes, to make
+ * available on @p socket_id. The allocator may reserve more than
+ * the requested amount due to internal rounding (e.g., to memzone
+ * or block granularity).
+ *
+ * @param socket_id
+ * The NUMA socket on which to reserve memory, or SOCKET_ID_ANY
+ * to leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the reservation.
+ *
+ * @return
+ * - 0: Success.
+ * - -ENOMEM: Insufficient huge-page memory to satisfy the request.
+ * - -EINVAL: Invalid @p socket_id.
+ */
+__rte_experimental
+int
+rte_fastmem_reserve(size_t size, int socket_id);
+
+/**
+ * Set the maximum backing memory that may be reserved on a socket.
+ *
+ * Once the limit is reached, allocations that would require new
+ * backing memory on the constrained socket fail with ENOMEM.
+ * Already-reserved memory is not released.
+ *
+ * Setting a limit below the current reserved amount is allowed and
+ * prevents further growth.
+ *
+ * @param socket_id
+ * The NUMA socket to constrain, or SOCKET_ID_ANY to apply the
+ * limit to all sockets.
+ * @param max_bytes
+ * Maximum backing memory in bytes, or SIZE_MAX for unlimited (the default).
+ * @return
+ * - 0: Success.
+ * - -EINVAL: Fastmem not initialized, or invalid @p socket_id.
+ */
+__rte_experimental
+int
+rte_fastmem_set_limit(int socket_id, size_t max_bytes);
+
+/**
+ * Get the maximum backing memory limit for a socket.
+ *
+ * @param socket_id
+ * The NUMA socket to query.
+ * @return
+ * The limit in bytes, or SIZE_MAX if unlimited.
+ */
+__rte_experimental
+size_t
+rte_fastmem_get_limit(int socket_id);
+
+/**
+ * Retrieve the largest allocation size the allocator supports.
+ *
+ * Requests larger than this size are rejected by the allocation
+ * functions. The returned value is a property of the allocator
+ * implementation and does not change across the lifetime of the
+ * process.
+ *
+ * @return
+ * The largest supported allocation size, in bytes.
+ */
+__rte_experimental
+size_t
+rte_fastmem_max_size(void);
+
+/**
+ * Allocate an object from the fastmem allocator.
+ *
+ * Allocates at least @p size bytes, aligned to at least @p align
+ * bytes. The returned memory is backed by huge pages and is
+ * DMA-usable; its IOVA can be obtained via rte_fastmem_virt2iova().
+ *
+ * On NUMA systems, the memory is allocated on the socket of the
+ * calling lcore. Use rte_fastmem_alloc_socket() to target a
+ * specific socket.
+ *
+ * The allocated memory must be freed with rte_fastmem_free(). An
+ * allocation may be freed from any lcore, not only the lcore that
+ * made the allocation.
+ *
+ * This function is MT-safe.
+ *
+ * @param size
+ * Requested allocation size, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, the returned pointer will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, the returned pointer will
+ * be aligned on a multiple of @p align, which must be a power of
+ * 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags. Use
+ * RTE_FASTMEM_F_ZERO to obtain zero-initialized memory.
+ *
+ * @return
+ * - A pointer to the allocated object on success.
+ * - NULL on failure, with @c rte_errno set:
+ * - E2BIG: @p size exceeds rte_fastmem_max_size().
+ * - EINVAL: Invalid @p align (not a power of two).
+ * - ENOMEM: Allocation could not be served from existing
+ * backing memory and no additional memzone could be reserved.
+ */
+__rte_experimental
+void *
+rte_fastmem_alloc(size_t size, size_t align, unsigned int flags)
+ __rte_alloc_size(1) __rte_alloc_align(2);
+
+/**
+ * Allocate an object on a specific NUMA socket.
+ *
+ * Like rte_fastmem_alloc(), but targets the specified NUMA socket
+ * rather than the socket of the calling lcore. Use this variant
+ * when the lifetime or access pattern of the allocation is not
+ * tied to the calling lcore's socket.
+ *
+ * This function is MT-safe.
+ *
+ * @param size
+ * Requested allocation size, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, the returned pointer will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, the returned pointer will
+ * be aligned on a multiple of @p align, which must be a power of
+ * 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags.
+ *
+ * @param socket_id
+ * The NUMA socket on which to allocate, or SOCKET_ID_ANY to
+ * leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the request.
+ *
+ * @return
+ * - A pointer to the allocated object on success.
+ * - NULL on failure, with @c rte_errno set (see rte_fastmem_alloc()).
+ */
+__rte_experimental
+void *
+rte_fastmem_alloc_socket(size_t size, size_t align, unsigned int flags,
+ int socket_id)
+ __rte_alloc_size(1) __rte_alloc_align(2);
+
+/**
+ * Free an object previously allocated by the fastmem allocator.
+ *
+ * @p ptr must have been returned by a prior call to any fastmem
+ * allocation function, or be NULL. If @p ptr is NULL, no operation
+ * is performed.
+ *
+ * Free may be called from any lcore, regardless of which lcore
+ * made the original allocation.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptr
+ * Pointer to an object previously allocated by fastmem, or NULL.
+ */
+__rte_experimental
+void
+rte_fastmem_free(void *ptr);
+
+/**
+ * Allocate multiple objects in bulk.
+ *
+ * Allocates @p n objects, each of size at least @p size and aligned
+ * to at least @p align bytes, and stores the resulting pointers
+ * into @p ptrs. All @p n objects have the same size and alignment.
+ *
+ * On NUMA systems, the memory is allocated on the socket of the
+ * calling lcore. Use rte_fastmem_alloc_bulk_socket() to target a
+ * specific socket.
+ *
+ * The bulk path amortizes per-object overhead and is typically
+ * faster than @p n individual calls to rte_fastmem_alloc().
+ *
+ * On failure no objects are allocated and @p ptrs is left
+ * untouched.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptrs
+ * An array of at least @p n pointers into which the newly
+ * allocated object pointers are written.
+ *
+ * @param n
+ * The number of objects to allocate.
+ *
+ * @param size
+ * Requested size of each object, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, returned pointers will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, returned pointers will be
+ * aligned on a multiple of @p align, which must be a power of 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags.
+ *
+ * @return
+ * - 0: All @p n objects were allocated and stored in @p ptrs.
+ * - -E2BIG: @p size exceeds rte_fastmem_max_size().
+ * - -EINVAL: Invalid @p align.
+ * - -ENOMEM: Not enough objects could be allocated to fill the
+ * request.
+ */
+__rte_experimental
+int
+rte_fastmem_alloc_bulk(void **ptrs, unsigned int n, size_t size, size_t align,
+ unsigned int flags);
+
+/**
+ * Allocate multiple objects in bulk on a specific NUMA socket.
+ *
+ * Like rte_fastmem_alloc_bulk(), but targets the specified NUMA
+ * socket rather than the socket of the calling lcore.
+ *
+ * This function is MT-safe.
+ *
+ * @param ptrs
+ * An array of at least @p n pointers into which the newly
+ * allocated object pointers are written.
+ *
+ * @param n
+ * The number of objects to allocate.
+ *
+ * @param size
+ * Requested size of each object, in bytes. Must not exceed
+ * rte_fastmem_max_size().
+ *
+ * @param align
+ * If 0, returned pointers will be aligned to at least
+ * @c RTE_CACHE_LINE_SIZE. Otherwise, returned pointers will be
+ * aligned on a multiple of @p align, which must be a power of 2.
+ *
+ * @param flags
+ * A bitwise OR of zero or more RTE_FASTMEM_F_* flags.
+ *
+ * @param socket_id
+ * The NUMA socket on which to allocate, or SOCKET_ID_ANY to
+ * leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the request.
+ *
+ * @return
+ * - 0: All @p n objects were allocated and stored in @p ptrs.
+ * - Negative errno on failure (see rte_fastmem_alloc_bulk()).
+ */
+__rte_experimental
+int
+rte_fastmem_alloc_bulk_socket(void **ptrs, unsigned int n, size_t size,
+ size_t align, unsigned int flags, int socket_id);
+
+/**
+ * Free multiple objects in bulk.
+ *
+ * Frees the @p n objects pointed to by @p ptrs. Each pointer in
+ * the array must have been returned by a prior fastmem allocation
+ * call and must not have been freed. The objects need not have
+ * the same size, alignment, or socket.
+ *
+ * The bulk path amortizes per-object overhead and is typically
+ * faster than @p n individual calls to rte_fastmem_free().
+ *
+ * This function is MT-safe.
+ *
+ * @param ptrs
+ * An array of @p n pointers to fastmem-allocated objects.
+ *
+ * @param n
+ * The number of objects to free.
+ */
+__rte_experimental
+void
+rte_fastmem_free_bulk(void **ptrs, unsigned int n);
+
+/**
+ * Opaque handle encoding a (size class, NUMA socket) pair.
+ *
+ * Obtained via rte_fastmem_hlookup(). Passing a handle to
+ * rte_fastmem_halloc() avoids the per-call size-class
+ * lookup and socket resolution, improving allocation throughput
+ * for fixed-size objects.
+ */
+typedef uint32_t rte_fastmem_handle_t;
+
+/**
+ * Look up a handle for a given object size and NUMA socket.
+ *
+ * The returned handle encodes the size class and socket, and can
+ * be passed to rte_fastmem_halloc() to allocate objects
+ * without repeating the class lookup.
+ *
+ * @param size
+ * Object size in bytes. Must not exceed rte_fastmem_max_size().
+ *
+ * @param align
+ * Alignment requirement (power of two), or 0 for the default
+ * (RTE_CACHE_LINE_SIZE).
+ *
+ * @param socket_id
+ * NUMA socket to allocate from.
+ *
+ * @param[out] handle
+ * On success, set to the resolved handle.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: Invalid alignment or socket_id.
+ * - -E2BIG: @p size exceeds rte_fastmem_max_size().
+ */
+__rte_experimental
+int
+rte_fastmem_hlookup(size_t size, size_t align, int socket_id,
+ rte_fastmem_handle_t *handle);
+
+/**
+ * Allocate an object using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_alloc() but skips the size-class
+ * lookup and socket resolution, using the pre-resolved handle
+ * instead.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param flags
+ * Allocation flags (e.g., RTE_FASTMEM_F_ZERO).
+ *
+ * @return
+ * A pointer to the allocated object, or NULL on failure
+ * (rte_errno is set).
+ */
+__rte_experimental
+void *
+rte_fastmem_halloc(rte_fastmem_handle_t handle, unsigned int flags);
+
+/**
+ * Bulk-allocate objects using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_alloc_bulk() but uses a pre-resolved
+ * handle. All-or-nothing semantics apply.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param[out] ptrs
+ * Array to receive @p n allocated pointers.
+ *
+ * @param n
+ * Number of objects to allocate.
+ *
+ * @param flags
+ * Allocation flags (e.g., RTE_FASTMEM_F_ZERO).
+ *
+ * @return
+ * - 0: All @p n objects allocated successfully.
+ * - -ENOMEM: Allocation failed; no objects were allocated.
+ */
+__rte_experimental
+int
+rte_fastmem_halloc_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n, unsigned int flags);
+
+/**
+ * Free an object using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_free() but skips the slab-header
+ * lookup by using the class and socket encoded in the handle.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param ptr
+ * A pointer previously returned by a fastmem allocation function.
+ * Must belong to the same size class and socket as @p handle.
+ * NULL is permitted (no-op).
+ */
+__rte_experimental
+void
+rte_fastmem_hfree(rte_fastmem_handle_t handle, void *ptr);
+
+/**
+ * Bulk-free objects using a pre-resolved handle.
+ *
+ * Equivalent to rte_fastmem_free_bulk() but skips per-object
+ * slab-header lookups.
+ *
+ * All objects must belong to the same size class and socket as
+ * @p handle.
+ *
+ * @param handle
+ * A handle previously obtained from rte_fastmem_hlookup().
+ *
+ * @param ptrs
+ * An array of @p n pointers to fastmem-allocated objects.
+ *
+ * @param n
+ * The number of objects to free.
+ */
+__rte_experimental
+void
+rte_fastmem_hfree_bulk(rte_fastmem_handle_t handle,
+ void **ptrs, unsigned int n);
+
+/**
+ * Obtain the IOVA for a fastmem-allocated pointer.
+ *
+ * Translates a virtual address returned by a fastmem allocation
+ * function into the corresponding IOVA, suitable for use in device
+ * DMA descriptors.
+ *
+ * The returned IOVA is valid for the lifetime of the allocation.
+ *
+ * @p ptr must have been returned by a prior fastmem allocation
+ * function. Passing any other pointer results in undefined
+ * behavior.
+ *
+ * @param ptr
+ * A pointer previously returned by a fastmem allocation
+ * function.
+ *
+ * @return
+ * The IOVA corresponding to @p ptr.
+ */
+__rte_experimental
+rte_iova_t
+rte_fastmem_virt2iova(const void *ptr);
+
+/**
+ * Flush the calling lcore's per-lcore caches.
+ *
+ * Drains every cached object from the calling lcore's
+ * per-(size class, NUMA socket) caches back to their shared
+ * bins, and releases the cache state itself. A subsequent
+ * allocation or free on this lcore lazily recreates any caches
+ * it needs.
+ *
+ * This is useful in applications that have finished a bursty
+ * phase and want to release memory that would otherwise sit idle
+ * in caches. It is also useful in tests that want to observe
+ * bin-level state without per-lcore caching hiding activity.
+ *
+ * The call has no effect when invoked from a non-EAL thread.
+ *
+ * This function is not thread-safe with respect to concurrent
+ * allocations or frees on the calling lcore; call it only when
+ * the calling lcore is not making other fastmem calls.
+ */
+__rte_experimental
+void
+rte_fastmem_cache_flush(void);
+
+/**
+ * Global summary statistics.
+ */
+struct rte_fastmem_stats {
+ uint64_t bytes_backing; /**< Bytes of backing memory (memzones) reserved from EAL. */
+ uint64_t bytes_in_use; /**< Approximate bytes in live objects. */
+ uint64_t alloc_total; /**< Total successful alloc operations (hits + misses). */
+ uint64_t free_total; /**< Total free operations (hits + misses). */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ unsigned int n_classes; /**< Number of size classes. */
+};
+
+/**
+ * Per-size-class statistics (aggregated across all lcores).
+ *
+ * Allocation and free counters count individual objects, not
+ * operations. A bulk allocation of 32 objects that hits the cache
+ * increments alloc_cache_hits by 32.
+ */
+struct rte_fastmem_class_stats {
+ size_t class_size; /**< Usable size of this class (bytes). */
+ uint64_t in_use; /**< Objects currently live (allocs - frees). */
+ uint64_t alloc_cache_hits; /**< Allocs served from a per-lcore cache. */
+ uint64_t alloc_cache_misses; /**< Allocs that triggered a bin refill. */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ uint64_t free_cache_hits; /**< Frees absorbed by a per-lcore cache. */
+ uint64_t free_cache_misses; /**< Frees that triggered a bin drain. */
+ uint64_t slab_acquires; /**< Slabs pulled from the free pool. */
+ uint64_t slab_releases; /**< Slabs returned to the free pool. */
+ uint32_t slabs_partial; /**< Current partial slab count. */
+ uint32_t slabs_full; /**< Current full slab count. */
+};
+
+/**
+ * Per-lcore statistics (aggregated across all classes).
+ */
+struct rte_fastmem_lcore_stats {
+ uint64_t alloc_cache_hits; /**< Allocs served from this lcore's caches. */
+ uint64_t alloc_cache_misses; /**< Allocs that missed this lcore's caches. */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ uint64_t free_cache_hits; /**< Frees absorbed by this lcore's caches. */
+ uint64_t free_cache_misses; /**< Frees that bypassed this lcore's caches. */
+};
+
+/**
+ * Per-lcore, per-class statistics (no aggregation).
+ */
+struct rte_fastmem_lcore_class_stats {
+ size_t class_size; /**< Usable size of this class (bytes). */
+ uint64_t alloc_cache_hits; /**< Allocs served from cache. */
+ uint64_t alloc_cache_misses; /**< Allocs that triggered a bin refill. */
+ uint64_t alloc_nomem; /**< Alloc attempts that failed with ENOMEM. */
+ uint64_t free_cache_hits; /**< Frees absorbed by cache. */
+ uint64_t free_cache_misses; /**< Frees that triggered a bin drain. */
+};
+
+/**
+ * Get the number of size classes and optionally their sizes.
+ *
+ * @param[out] sizes
+ * If non-NULL, filled with the size (in bytes) of each class.
+ * The caller must provide space for at least the returned number
+ * of entries.
+ *
+ * @return
+ * The number of size classes.
+ */
+__rte_experimental
+unsigned int
+rte_fastmem_classes(size_t *sizes);
+
+/**
+ * Retrieve global summary statistics.
+ *
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL or fastmem is not initialized.
+ */
+__rte_experimental
+int
+rte_fastmem_stats(struct rte_fastmem_stats *stats);
+
+/**
+ * Retrieve statistics for a single size class.
+ *
+ * @param class_size
+ * Exact size of the class to query (must match one of the values
+ * returned by rte_fastmem_classes()).
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL, fastmem is not initialized, or
+ * @p class_size does not match any size class.
+ */
+__rte_experimental
+int
+rte_fastmem_stats_class(size_t class_size,
+ struct rte_fastmem_class_stats *stats);
+
+/**
+ * Retrieve per-lcore statistics (aggregated across all classes).
+ *
+ * @param lcore_id
+ * The lcore to query.
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL, fastmem is not initialized, or
+ * @p lcore_id is invalid.
+ */
+__rte_experimental
+int
+rte_fastmem_stats_lcore(unsigned int lcore_id,
+ struct rte_fastmem_lcore_stats *stats);
+
+/**
+ * Retrieve per-lcore, per-class statistics.
+ *
+ * @param lcore_id
+ * The lcore to query.
+ * @param class_size
+ * Exact size of the class to query.
+ * @param[out] stats
+ * Structure to fill.
+ *
+ * @return
+ * - 0: Success.
+ * - -EINVAL: @p stats is NULL, fastmem is not initialized,
+ * @p lcore_id is invalid, or @p class_size does not match any
+ * size class.
+ */
+__rte_experimental
+int
+rte_fastmem_stats_lcore_class(unsigned int lcore_id, size_t class_size,
+ struct rte_fastmem_lcore_class_stats *stats);
+
+/**
+ * Reset all statistics counters to zero.
+ *
+ * Zeroes per-lcore cache counters and per-bin counters. Does not
+ * affect the allocator's operational state.
+ */
+__rte_experimental
+void
+rte_fastmem_stats_reset(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_FASTMEM_H_ */
diff --git a/lib/meson.build b/lib/meson.build
index 8f5cfd28a5..10906d4d53 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -38,6 +38,7 @@ libraries = [
'distributor',
'dmadev', # eventdev depends on this
'efd',
+ 'fastmem',
'eventdev',
'dispatcher', # dispatcher depends on eventdev
'gpudev',
--
2.43.0
^ permalink raw reply related
* [RFC v2 1/3] doc: add fastmem programming guide
From: Mattias Rönnblom @ 2026-05-26 8:57 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Mattias Rönnblom
In-Reply-To: <20260526085743.64396-1-hofors@lysator.liu.se>
Add a programming guide for the fastmem library covering usage,
API overview, design, and implementation details.
Signed-off-by: Mattias Rönnblom <hofors@lysator.liu.se>
---
doc/guides/prog_guide/fastmem_lib.rst | 314 ++++++++++++++++++++++++++
doc/guides/prog_guide/index.rst | 1 +
2 files changed, 315 insertions(+)
create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
diff --git a/doc/guides/prog_guide/fastmem_lib.rst b/doc/guides/prog_guide/fastmem_lib.rst
new file mode 100644
index 0000000000..564d34b78f
--- /dev/null
+++ b/doc/guides/prog_guide/fastmem_lib.rst
@@ -0,0 +1,314 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2026 Ericsson AB
+
+Fastmem Library
+===============
+
+The fastmem library is a fast, general-purpose small-object
+allocator for DPDK applications. It lets an application replace
+its many per-type mempools — each sized for a single object type
+— with a single allocator that handles arbitrary object sizes,
+grows on demand, and offers mempool-level performance for the
+common allocation and free paths.
+
+Like mempool, fastmem is backed by huge pages, is NUMA-aware,
+supports bulk operations, and uses per-lcore caches to reduce
+shared-state contention. Unlike mempool, it does not require the
+caller to declare object sizes or counts up front.
+
+
+When to use fastmem
+-------------------
+
+Use fastmem when:
+
+* Small objects (up to 1 MiB) are allocated and freed on the
+ data path with low, predictable latency requirements.
+
+* Many object types of varying sizes exist and maintaining a
+ separate mempool for each is impractical.
+
+* DMA-usable memory with efficient virtual-to-IOVA translation
+ is needed.
+
+Do not use fastmem for allocations larger than 1 MiB. Use
+``rte_malloc()`` instead.
+
+
+Initialization and teardown
+----------------------------
+
+.. code-block:: c
+
+ /* At startup, after rte_eal_init(). */
+ rte_fastmem_init();
+
+ /* Optional: pre-reserve backing memory to avoid latency
+ * spikes from on-demand memzone reservation. */
+ rte_fastmem_reserve(64 * 1024 * 1024, SOCKET_ID_ANY);
+
+ /* ... application runs ... */
+
+ /* At shutdown, after all allocations have been freed. */
+ rte_fastmem_deinit();
+
+Neither ``rte_fastmem_init()`` nor ``rte_fastmem_deinit()`` is
+thread-safe; call them from the main lcore during startup and
+shutdown.
+
+
+Allocation and free
+-------------------
+
+.. code-block:: c
+
+ void *obj = rte_fastmem_alloc(128, 0, 0);
+ /* Use obj... */
+ rte_fastmem_free(obj);
+
+``rte_fastmem_alloc()`` allocates on the calling lcore's NUMA
+socket. Use ``rte_fastmem_alloc_socket()`` to target a specific
+socket or to enable cross-socket fallback with ``SOCKET_ID_ANY``.
+
+Alignment
+~~~~~~~~~
+
+When ``align`` is 0, the returned pointer is aligned to at least
+``RTE_CACHE_LINE_SIZE``. A non-zero ``align`` must be a power of
+two. Specifying an alignment smaller than ``RTE_CACHE_LINE_SIZE``
+is permitted but the returned object may then share a cache line
+with an adjacent allocation, risking false sharing.
+
+Zeroing
+~~~~~~~
+
+Pass ``RTE_FASTMEM_F_ZERO`` to receive zero-initialized memory:
+
+.. code-block:: c
+
+ void *obj = rte_fastmem_alloc(256, 0, RTE_FASTMEM_F_ZERO);
+
+
+Bulk allocation and free
+-------------------------
+
+.. code-block:: c
+
+ void *ptrs[32];
+
+ if (rte_fastmem_alloc_bulk(ptrs, 32, 64, 0, 0) < 0)
+ /* handle error */;
+
+ /* Use objects... */
+
+ rte_fastmem_free_bulk(ptrs, 32);
+
+Bulk allocation has all-or-nothing semantics: either all
+requested objects are returned, or none are (and ``rte_errno``
+is set to ``ENOMEM``).
+
+Bulk free is most efficient when all objects belong to the same
+size class; in that case the objects are pushed into the
+per-lcore cache in a single operation.
+
+
+IOVA translation
+----------------
+
+Memory returned by fastmem is DMA-usable. To obtain the IOVA
+for use in device descriptors:
+
+.. code-block:: c
+
+ rte_iova_t iova = rte_fastmem_virt2iova(obj);
+
+The translation is O(1). The returned IOVA is valid for the
+lifetime of the allocation.
+
+
+NUMA awareness
+--------------
+
+``rte_fastmem_alloc()`` allocates on the calling lcore's socket.
+``rte_fastmem_alloc_socket()`` accepts an explicit socket ID or
+``SOCKET_ID_ANY``:
+
+* Explicit socket: allocate only from that socket; fail with
+ ``ENOMEM`` if exhausted.
+
+* ``SOCKET_ID_ANY``: try the caller's local socket first, then
+ fall back to other sockets.
+
+
+Per-lcore caches
+----------------
+
+Each EAL thread has a private cache per size class. The common
+allocation and free paths operate entirely within this cache,
+avoiding locks. Cache misses (empty on alloc, full on free)
+trigger a bulk transfer to/from the shared bin under a lock.
+
+Non-EAL threads bypass the cache and take the bin lock on every
+operation.
+
+``rte_fastmem_cache_flush()`` drains the calling lcore's caches
+back to the shared bins. This is useful after bursty phases to
+release idle cached memory.
+
+
+Threading
+---------
+
+All allocation and free functions are thread-safe and may be
+called from any thread. An allocation made on one thread may be
+freed on any other.
+
+Fastmem uses internal spinlocks. A thread preempted while
+holding one delays other threads contending for the same lock
+(correctness is not affected, only latency).
+
+
+Pre-reserving memory
+--------------------
+
+By default, fastmem reserves backing memory lazily on first
+allocation. ``rte_fastmem_reserve(size, socket_id)`` forces
+reservation up front, ensuring subsequent allocations do not
+incur memzone-reservation latency:
+
+.. code-block:: c
+
+ /* Reserve 128 MiB on socket 0. */
+ rte_fastmem_reserve(128 * 1024 * 1024, 0);
+
+Once reserved, backing memory is never returned to the system
+during the allocator's lifetime.
+
+Memory limits
+~~~~~~~~~~~~~
+
+``rte_fastmem_set_limit(socket_id, max_bytes)`` caps how much
+backing memory may be reserved on a given socket. Once the limit is
+reached, allocations that would require new backing memory fail with
+``ENOMEM``. The default is ``SIZE_MAX`` (unlimited).
+``rte_fastmem_get_limit()`` returns the current limit for a socket.
+
+.. code-block:: c
+
+ /* Allow at most 256 MiB on socket 0. */
+ rte_fastmem_set_limit(0, 256 * 1024 * 1024);
+
+ /* Block all growth on socket 1. */
+ rte_fastmem_set_limit(1, 0);
+
+Pass ``SOCKET_ID_ANY`` to apply the same limit to all sockets.
+
+
+Size classes
+------------
+
+Fastmem uses power-of-two size classes from 8 bytes to 1 MiB
+(18 classes). A request for N bytes is served from the smallest
+class >= N. The maximum supported size is queryable via
+``rte_fastmem_max_size()``.
+
+With power-of-two classes, worst-case internal fragmentation is
+just under 50% (e.g., a 33-byte request occupies a 64-byte
+slot). Assuming a uniform distribution of request sizes, the
+average waste is 25%. In practice, DPDK workloads tend to
+cluster at or near powers of two, so typical waste is lower.
+
+Requests exceeding the maximum are rejected with ``E2BIG``.
+
+
+Implementation
+--------------
+
+Fastmem organizes memory in three layers: backing memzones, slabs,
+and per-lcore caches.
+
+Backing memory and slabs
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Backing memory is obtained from EAL as 128 MiB IOVA-contiguous
+memzones, each aligned to 2 MiB. A memzone is partitioned into
+64 fixed-size, 2 MiB **slabs**. Slabs are the unit of memory
+that moves between size classes: a free slab can be assigned to
+any bin on demand, and an empty slab (all objects freed) returns
+to the free-slab pool for reuse by another size class.
+
+The 2 MiB slab alignment is the key structural property. Given
+any object pointer, the allocator recovers the owning slab by
+masking off the low 21 bits — no radix tree, hash table, or
+memzone lookup is needed. This makes the free path fast: a
+single pointer-mask load reaches the slab header, which
+identifies the size class and bin.
+
+Each slab reserves 64 bytes at offset 0 for its header. The
+remaining space is divided into fixed-size slots equal to the
+size class. Allocated objects carry no per-object metadata; the
+full slot is available to the caller.
+
+Three-level allocation hierarchy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. **Per-lcore cache** — a bounded LIFO stack of free object
+ pointers, one per (lcore, size class, socket). Allocation
+ pops; free pushes. No lock is needed because only the owning
+ lcore accesses its cache.
+
+2. **Bin** — one per (size class, socket). Owns the partial and
+ full slab lists. A spinlock serializes bulk transfers between
+ the bin and per-lcore caches. Most traffic is absorbed by the
+ caches, so bin-lock contention is low.
+
+3. **Free-slab pool** — one per socket. A spinlock protects slab
+ acquisition and release. These events are rare relative to
+ object-level operations (a single small-object slab serves
+ thousands of allocations).
+
+On a cache miss (empty on alloc, full on free), the cache
+exchanges objects with the bin in bulk, targeting half-full to
+maximize headroom in both directions.
+
+Cache sizing
+~~~~~~~~~~~~
+
+Cache capacity varies by size class to bound per-lcore memory
+footprint:
+
+* Classes 8 B through 4 KiB: capacity 64.
+* Larger classes: capacity halves per class (32, 16, 8, 4),
+ flooring at 4.
+
+Even the largest classes remain cached. The capacity curve
+ensures that small, frequent allocations get the highest cache
+hit rate, while large allocations still avoid the bin lock on
+most operations.
+
+
+Statistics
+----------
+
+Fastmem maintains always-on, per-lcore counters that track
+allocation and free activity. Statistics are queryable at four
+levels of granularity: global summary, per size class, per lcore,
+and per lcore per class.
+
+``rte_fastmem_classes()`` returns the number of size classes and
+optionally fills an array with their sizes.
+
+See ``rte_fastmem.h`` for the full statistics API.
+
+
+Secondary Processes
+-------------------
+
+Fastmem works transparently in DPDK secondary processes. The shared
+state is discovered automatically on first allocation.
+
+Secondary processes do not use per-lcore caches; every allocation and
+free acquires the bin spinlock directly. This is acceptable for
+control-plane secondaries with low allocation rates. The primary
+process should pre-reserve sufficient backing memory with
+``rte_fastmem_reserve()`` since secondaries cannot grow the pool.
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index e6f24945b0..c85196c85e 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -28,6 +28,7 @@ Memory Management
mempool_lib
mbuf_lib
multi_proc_support
+ fastmem_lib
CPU Management
--
2.43.0
^ permalink raw reply related
* [RFC v2 0/3] lib/fastmem: fast small-object allocator
From: Mattias Rönnblom @ 2026-05-26 8:57 UTC (permalink / raw)
To: dev
Cc: Morten Brørup, Konstantin Ananyev, Mattias Rönnblom,
Yogaraj Baskaravel, Stephen Hemminger, Mattias Rönnblom
In-Reply-To: <20260525103642.55255-4-hofors@lysator.liu.se>
This RFC introduces fastmem, a general-purpose small-object allocator
for DPDK. It is intended to replace per-type mempools with a single
allocator that handles arbitrary sizes, grows on demand, and matches
mempool-level performance on the hot path.
Motivation
----------
DPDK applications commonly maintain many mempools — one per object
type (connections, sessions, timers, work items). Each must be sized
up front, wastes memory when over-provisioned, and cannot serve
objects of a different size. Fastmem eliminates this by accepting
arbitrary sizes at runtime, backed by a slab allocator that
repurposes memory across size classes as demand shifts.
Design
------
Three-layer architecture:
1. Backing memory: 128 MiB IOVA-contiguous memzones from EAL,
reserved lazily (or pre-reserved for deterministic latency).
2. Slabs: 2 MiB, 2 MiB-aligned regions carved from memzones.
The alignment enables O(1) slab lookup from any object pointer
via bitmask — no radix tree or index structure. Slabs move
freely between 18 power-of-2 size classes (8 B to 1 MiB).
3. Per-lcore caches: bounded LIFO stacks (no locks on the hot
path). Cache misses trigger bulk transfers to/from the shared
bin under a spinlock.
Key properties:
- Zero per-object metadata in the production build.
- NUMA-aware, with per-socket bins and free-slab pools.
- DMA-usable memory with O(1) virt-to-IOVA translation.
- Bulk alloc/free with all-or-nothing semantics.
- Backing memory never returned during lifetime (slabs recycled).
- Non-EAL threads supported (bypass cache, take bin lock).
- Secondary process support (lazy attach, no per-lcore caches).
API surface
-----------
rte_fastmem_init / deinit
rte_fastmem_reserve
rte_fastmem_set_limit / get_limit
rte_fastmem_alloc / alloc_socket
rte_fastmem_alloc_bulk / alloc_bulk_socket
rte_fastmem_free / free_bulk
rte_fastmem_hlookup / halloc / halloc_bulk / hfree / hfree_bulk
rte_fastmem_virt2iova
rte_fastmem_cache_flush
rte_fastmem_max_size / classes
rte_fastmem_stats / stats_class / stats_lcore / stats_lcore_class
rte_fastmem_stats_reset
All APIs are marked __rte_experimental.
Performance
-----------
The single-object hot path is roughly 2–3× the cost of mempool
and an order of magnitude faster than rte_malloc. Under
multi-lcore contention, fastmem scales similarly to mempool,
while rte_malloc collapses.
Limitations
-----------
- Maximum allocation: 1 MiB. Larger requests should use rte_malloc.
- Power-of-2 classes only; worst-case internal fragmentation ~50%.
- Backing memory not reclaimable short of deinit.
Future work
-----------
- Lcore-affine allocations (false-sharing-free by construction).
- Mempool ops driver for transparent drop-in use.
- Debug mode (cookies, double-free detection, poison-on-free).
- Telemetry integration.
- EAL integration, allowing EAL-internal subsystems to use
fastmem for their small-object allocations.
Mattias Rönnblom (3):
doc: add fastmem programming guide
lib: add fastmem library
app/test: add fastmem test suite
app/test/meson.build | 3 +
app/test/test_fastmem.c | 1673 ++++++++++++++++++++++++
app/test/test_fastmem_perf.c | 1040 +++++++++++++++
app/test/test_fastmem_profile.c | 157 +++
doc/api/doxy-api-index.md | 1 +
doc/api/doxy-api.conf.in | 1 +
doc/guides/prog_guide/fastmem_lib.rst | 314 +++++
doc/guides/prog_guide/index.rst | 1 +
lib/fastmem/meson.build | 6 +
lib/fastmem/rte_fastmem.c | 1694 +++++++++++++++++++++++++
lib/fastmem/rte_fastmem.h | 774 +++++++++++
lib/meson.build | 1 +
12 files changed, 5665 insertions(+)
create mode 100644 app/test/test_fastmem.c
create mode 100644 app/test/test_fastmem_perf.c
create mode 100644 app/test/test_fastmem_profile.c
create mode 100644 doc/guides/prog_guide/fastmem_lib.rst
create mode 100644 lib/fastmem/meson.build
create mode 100644 lib/fastmem/rte_fastmem.c
create mode 100644 lib/fastmem/rte_fastmem.h
--
2.43.0
^ permalink raw reply
* RE: [PATCH v5] mempool: improve cache behaviour and performance
From: Morten Brørup @ 2026-05-26 8:57 UTC (permalink / raw)
To: Bruce Richardson
Cc: dev, Andrew Rybchenko, Jingjing Wu, Praveen Shetty,
Hemant Agrawal, Sachin Saxena
In-Reply-To: <ahCAgitAAK3e5Kf3@bricha3-mobl1.ger.corp.intel.com>
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 22 May 2026 18.13
>
> On Sun, Apr 19, 2026 at 09:55:26AM +0000, Morten Brørup wrote:
> > This patch refactors the mempool cache to eliminate some unexpected
> > behaviour and reduce the mempool cache miss rate.
> >
> > 1.
> > The actual cache size was 1.5 times the cache size specified at run-
> time
> > mempool creation.
> > This was obviously not expected by application developers.
> >
> > 2.
> > In get operations, the check for when to use the cache as bounce
> buffer
> > did not respect the run-time configured cache size,
> > but compared to the build time maximum possible cache size
> > (RTE_MEMPOOL_CACHE_MAX_SIZE, default 512).
> > E.g. with a configured cache size of 32 objects, getting 256 objects
> > would first fetch 32 + 256 = 288 objects into the cache,
> > and then move the 256 objects from the cache to the destination
> memory,
> > instead of fetching the 256 objects directly to the destination
> memory.
> > This had a performance cost.
> > However, this is unlikely to occur in real applications, so it is not
> > important in itself.
> >
> > 3.
> > When putting objects into a mempool, and the mempool cache did not
> have
> > free space for so many objects,
> > the cache was flushed completely, and the new objects were then put
> into
> > the cache.
> > I.e. the cache drain level was zero.
> > This (complete cache flush) meant that a subsequent get operation
> (with
> > the same number of objects) completely emptied the cache,
> > so another subsequent get operation required replenishing the cache.
> >
> > Similarly,
> > When getting objects from a mempool, and the mempool cache did not
> hold so
> > many objects,
> > the cache was replenished to cache->size + remaining objects,
> > and then (the remaining part of) the requested objects were fetched
> via
> > the cache,
> > which left the cache filled (to cache->size) at completion.
> > I.e. the cache refill level was cache->size (plus some, depending on
> > request size).
> >
> > (1) was improved by generally comparing to cache->size instead of
> > cache->flushthresh, when considering the capacity of the cache.
> > The cache->flushthresh field is kept for API/ABI compatibility
> purposes,
> > and initialized to cache->size instead of cache->size * 1.5.
> >
> > (2) was improved by generally comparing to cache->size / 2 instead of
> > RTE_MEMPOOL_CACHE_MAX_SIZE, when checking the bounce buffer limit.
> >
> > (3) was improved by flushing and replenishing the cache by half its
> size,
> > so a flush/refill can be followed randomly by get or put requests.
> > This also reduced the number of objects in each flush/refill
> operation.
> >
> > As a consequence of these changes, the size of the array holding the
> > objects in the cache (cache->objs[]) no longer needs to be
> > 2 * RTE_MEMPOOL_CACHE_MAX_SIZE, and can be reduced to
> > RTE_MEMPOOL_CACHE_MAX_SIZE at an API/ABI breaking release.
> >
> > Performance data:
> > With a real WAN Optimization application, where the number of
> allocated
> > packets varies (as they are held in e.g. shaper queues), the mempool
> > cache miss rate dropped from ca. 1/20 objects to ca. 1/48 objects.
> > This was deployed in production at an ISP, and using an effective
> cache
> > size of 384 objects.
> >
> > As a consequence of the improved mempool cache algorithm, some
> drivers
> > were updated accordingly:
> > - The Intel idpf PMD was updated regarding how much to backfill the
> > mempool cache in the AVX512 code.
> > - The NXP dpaa and dpaa2 mempool drivers were updated to not set the
> > mempool cache flush threshold; doing this no longer has any effect,
> and
> > thus became superfluous.
> >
> > Bugzilla ID: 1027
> > Fixes: ea5dd2744b90 ("mempool: cache optimisations")
> > Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
> > ---
> > Depends-on: patch-163181 ("net/intel: do not bypass mbuf lib for mbuf
> fast-free")
> > ---
> > v5:
> > * Flush the cache from the bottom, where objects are colder, and move
> down
> > the remaining objects, which are hotter.
> > * In the Intel idpf PMD, move up the hot objects in the cache and
> refill
> > with cold objects at the bottom.
> > v4:
> > * Added Bugzilla ID.
> > * Added Fixes tag. For reference only.
> > * Moved fast-free related update of Intel common driver out as a
> separate
> > patch, and depend on that patch.
> > * Omitted unrelated changes to the Intel idpf AVX512 driver,
> specifically
> > fixing an indentation and adding mbuf instrumentation.
> > * Omitted unrelated changes to the mempool library, specifically
> adding
> > __rte_restrict and changing a couple of comments to proper
> sentences.
> > * Please checkpatches by swapping operators in a couple of
> comparisons.
> > v3:
> > * Fixed my copy-paste bug in idpf_splitq_rearm().
> > v2:
> > * Fixed issue found by abidiff:
> > Reverted cache objects array size reduction. Added a note instead.
> > * Added missing mbuf instrumentation to the Intel idpf AVX512 driver.
> > * Updated idpf_splitq_rearm() like idpf_singleq_rearm().
> > * Added a few more __rte_assume(). (Inspired by AI review)
> > * Updated NXP dpaa and dpaa2 mempool drivers to not set mempool cache
> > flush threshold.
> > * Added release notes.
> > * Added deprecation notes.
> > ---
> > doc/guides/rel_notes/deprecation.rst | 7 ++
> > doc/guides/rel_notes/release_26_07.rst | 10 +++
> > drivers/mempool/dpaa/dpaa_mempool.c | 14 ----
> > drivers/mempool/dpaa2/dpaa2_hw_mempool.c | 14 ----
> > .../net/intel/idpf/idpf_common_rxtx_avx512.c | 52 +++++++++++---
> > lib/mempool/rte_mempool.c | 14 +---
> > lib/mempool/rte_mempool.h | 70 ++++++++++++-----
> --
> > 7 files changed, 104 insertions(+), 77 deletions(-)
> >
> Can the idpf and dpaa changes be made in separate patches, so we can
> review
> the mempool changes along in a single patch? Even if the commits can't
> work
> logically together, perhaps they can be separated for review, and then
> squashed on apply?
Sure.
The idpf changes are required due to the driver bypassing the mempool API.
If such direct access to the mempool cache is required, it would be better to add a mempool cache zero-copy API.
The dpaa changes are simple clean-ups.
They set a variable which this mempool change makes obsolete, so no need to set it.
^ permalink raw reply
* [PATCH v3 25/25] bus: add class device conversion macro
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Nicolas Chautru, Parav Pandit,
Xueming Li, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Ashish Gupta,
Fan Zhang, Ankur Dwivedi, Anoob Joseph, Tejasree Kondoj,
Gagandeep Singh, Hemant Agrawal, Pavan Nikhilesh, Shijith Thotton,
Tirthendu Sarkar, Jerin Jacob, Shepard Siegel, Ed Czeck,
John Miller, Igor Russkikh, Steven Webster, Matt Peters,
Selwin Sebastian, Julien Aube, Kishore Padmanabha, Ajit Khaparde,
Nithin Dabilpuram, Kiran Kumar K, Sunil Kumar Kori, Satha Rao,
Harman Kalra, Potnuri Bharat Teja, Sachin Saxena, Shai Brandes,
Evgeny Schemeilin, Ron Beider, Amit Bernstein, Wajeeh Atrash,
Vanshika Shukla, John Daley, Hyong Youb Kim, Jeroen de Borst,
Joshua Washington, Xiaoyun Wang, Feifei Wang, Xingui Yang,
Chengwen Feng, Praveen Shetty, Vladimir Medvedkin,
Anatoly Burakov, Jingjing Wu, Rosen Xu, Dimon Zhao, Leon Yu,
Sam Chen, Long Li, Wei Hu, Chaoyong He, Jiawen Wu, Zaiyu Wang,
Vamsi Attunuru, Devendra Singh Rawat, Alok Prasad, Howard Wang,
Chunhao Lin, Xing Wang, Javen Xu, Wenbo Cao, Andrew Rybchenko,
Maciej Czekaj, Maxime Coquelin, Jochen Behrens, Renyong Wan,
Na Na, Rong Qian, Xiaoxiong Zhang, Dongwei Xu, Junlong Wang,
Ming Ran
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
Add a new helper macro RTE_CLASS_TO_BUS_DEVICE that provides a unified
way to convert from any device class (ethdev, cryptodev, eventdev, etc.)
to a bus-specific device type. This macro works with any device class
that has a 'device' field pointing to struct rte_device.
Remove the bus-specific ethdev convenience macros (RTE_ETH_DEV_TO_PCI,
RTE_ETH_DEV_TO_AUXILIARY, RTE_ETH_DEV_TO_VDEV) and replace all uses
with the generic RTE_CLASS_TO_BUS_DEVICE macro.
Convert all drivers to use RTE_CLASS_TO_BUS_DEVICE instead of
the pattern RTE_BUS_DEVICE(dev->device).
Simplify code that was using an intermediate struct rte_device pointer
by applying RTE_CLASS_TO_BUS_DEVICE directly on the device class pointer.
This reduces code duplication and provides a consistent interface that
can be used for all device classes.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/baseband/acc/rte_acc100_pmd.c | 4 +-
drivers/baseband/acc/rte_vrb_pmd.c | 2 +-
.../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 4 +-
drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 2 +-
drivers/bus/auxiliary/bus_auxiliary_driver.h | 3 --
drivers/bus/cdx/bus_cdx_driver.h | 2 -
drivers/bus/pci/bus_pci_driver.h | 3 --
drivers/bus/vdev/bus_vdev_driver.h | 3 --
drivers/compress/octeontx/otx_zip.c | 2 +-
drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 2 +-
drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c | 3 +-
drivers/crypto/octeontx/otx_cryptodev_ops.c | 4 +-
drivers/event/cnxk/cnxk_eventdev.c | 2 +-
drivers/event/dlb2/pf/dlb2_pf.c | 2 +-
drivers/event/skeleton/skeleton_eventdev.c | 2 +-
drivers/net/ark/ark_ethdev.c | 2 +-
drivers/net/atlantic/atl_ethdev.c | 12 +++---
drivers/net/avp/avp_ethdev.c | 22 +++++-----
drivers/net/axgbe/axgbe_ethdev.c | 4 +-
drivers/net/bnx2x/bnx2x_ethdev.c | 2 +-
drivers/net/bnxt/bnxt_ethdev.c | 12 +++---
drivers/net/bnxt/tf_ulp/bnxt_ulp.c | 4 +-
drivers/net/cnxk/cnxk_ethdev.c | 2 +-
drivers/net/cnxk/cnxk_ethdev_ops.c | 2 +-
drivers/net/cxgbe/cxgbe_ethdev.c | 4 +-
drivers/net/cxgbe/cxgbevf_ethdev.c | 4 +-
drivers/net/dpaa/dpaa_ethdev.c | 16 +++-----
drivers/net/dpaa2/dpaa2_ethdev.c | 8 ++--
drivers/net/dpaa2/dpaa2_recycle.c | 6 +--
drivers/net/ena/ena_ethdev.c | 10 ++---
drivers/net/enetc/enetc4_ethdev.c | 4 +-
drivers/net/enetc/enetc4_vf.c | 4 +-
drivers/net/enetc/enetc_ethdev.c | 2 +-
drivers/net/enic/enic_ethdev.c | 4 +-
drivers/net/enic/enic_fm_flow.c | 6 +--
drivers/net/enic/enic_vf_representor.c | 2 +-
drivers/net/gve/gve_ethdev.c | 2 +-
drivers/net/hinic/hinic_pmd_ethdev.c | 8 ++--
drivers/net/hinic3/base/hinic3_hwdev.c | 7 ++--
drivers/net/hinic3/hinic3_ethdev.c | 16 ++++----
drivers/net/hns3/hns3_cmd.c | 2 +-
drivers/net/hns3/hns3_common.c | 8 ++--
drivers/net/hns3/hns3_ethdev.c | 6 +--
drivers/net/hns3/hns3_ethdev_vf.c | 6 +--
drivers/net/hns3/hns3_rxtx.c | 4 +-
drivers/net/intel/cpfl/cpfl_ethdev.c | 4 +-
drivers/net/intel/cpfl/cpfl_ethdev.h | 2 +-
drivers/net/intel/e1000/em_ethdev.c | 12 +++---
drivers/net/intel/e1000/em_rxtx.c | 2 +-
drivers/net/intel/e1000/igb_ethdev.c | 30 +++++++-------
drivers/net/intel/e1000/igb_pf.c | 2 +-
drivers/net/intel/e1000/igc_ethdev.c | 22 +++++-----
drivers/net/intel/fm10k/fm10k_ethdev.c | 16 ++++----
drivers/net/intel/i40e/i40e_ethdev.c | 28 ++++++-------
drivers/net/intel/i40e/i40e_ethdev.h | 2 +-
drivers/net/intel/iavf/iavf_ethdev.c | 8 ++--
drivers/net/intel/ice/ice_dcf.c | 6 +--
drivers/net/intel/ice/ice_ethdev.c | 6 +--
drivers/net/intel/ice/ice_ethdev.h | 2 +-
drivers/net/intel/idpf/idpf_ethdev.h | 2 +-
drivers/net/intel/ipn3ke/ipn3ke_ethdev.h | 3 --
drivers/net/intel/ipn3ke/ipn3ke_representor.c | 6 +--
drivers/net/intel/ixgbe/ixgbe_ethdev.c | 40 +++++++++----------
drivers/net/intel/ixgbe/ixgbe_flow.c | 4 +-
drivers/net/intel/ixgbe/ixgbe_pf.c | 2 +-
drivers/net/intel/ixgbe/ixgbe_tm.c | 2 +-
.../net/intel/ixgbe/ixgbe_vf_representor.c | 2 +-
drivers/net/intel/ixgbe/rte_pmd_ixgbe.c | 20 +++++-----
drivers/net/nbl/nbl_core.c | 2 +-
drivers/net/nbl/nbl_dev/nbl_dev.c | 6 +--
drivers/net/netvsc/hn_ethdev.c | 3 +-
drivers/net/nfp/nfp_ethdev.c | 8 ++--
drivers/net/nfp/nfp_ethdev_vf.c | 6 +--
drivers/net/nfp/nfp_net_common.c | 8 ++--
drivers/net/ngbe/ngbe_ethdev.c | 20 +++++-----
drivers/net/ngbe/ngbe_ethdev_vf.c | 16 ++++----
drivers/net/ngbe/ngbe_pf.c | 2 +-
drivers/net/octeon_ep/otx_ep_ethdev.c | 2 +-
drivers/net/octeon_ep/otx_ep_mbox.c | 6 +--
drivers/net/qede/qede_ethdev.c | 6 +--
drivers/net/r8169/r8169_ethdev.c | 6 +--
drivers/net/rnp/rnp_ethdev.c | 6 +--
drivers/net/sfc/sfc.c | 4 +-
drivers/net/sfc/sfc_ethdev.c | 2 +-
drivers/net/sfc/sfc_intr.c | 10 ++---
drivers/net/sfc/sfc_rx.c | 3 +-
drivers/net/sfc/sfc_sriov.c | 2 +-
drivers/net/sfc/sfc_tx.c | 3 +-
drivers/net/thunderx/nicvf_ethdev.c | 4 +-
drivers/net/txgbe/txgbe_ethdev.c | 26 ++++++------
drivers/net/txgbe/txgbe_ethdev_vf.c | 16 ++++----
drivers/net/txgbe/txgbe_flow.c | 4 +-
drivers/net/txgbe/txgbe_pf.c | 2 +-
drivers/net/txgbe/txgbe_tm.c | 2 +-
drivers/net/virtio/virtio_pci_ethdev.c | 11 ++---
drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 +-
drivers/net/xsc/xsc_ethdev.c | 2 +-
drivers/net/zxdh/zxdh_ethdev.c | 8 ++--
drivers/raw/ifpga/afu_pmd_n3000.c | 4 +-
lib/eal/include/bus_driver.h | 18 +++++++++
100 files changed, 336 insertions(+), 340 deletions(-)
diff --git a/drivers/baseband/acc/rte_acc100_pmd.c b/drivers/baseband/acc/rte_acc100_pmd.c
index 061f595a98..cbcacc7aa3 100644
--- a/drivers/baseband/acc/rte_acc100_pmd.c
+++ b/drivers/baseband/acc/rte_acc100_pmd.c
@@ -3993,7 +3993,7 @@ acc100_dequeue_ldpc_dec(struct rte_bbdev_queue_data *q_data,
static void
acc100_bbdev_init(struct rte_bbdev *dev, struct rte_pci_driver *drv)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev->dev_ops = &acc100_bbdev_ops;
dev->enqueue_enc_ops = acc100_enqueue_enc;
@@ -4646,7 +4646,7 @@ rte_acc_configure(const char *dev_name, struct rte_acc_conf *conf)
dev_name);
return -ENODEV;
}
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(bbdev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(bbdev, *pci_dev);
rte_bbdev_log(INFO, "Configure dev id %x", pci_dev->id.device_id);
if (pci_dev->id.device_id == ACC100_PF_DEVICE_ID)
return acc100_configure(dev_name, conf);
diff --git a/drivers/baseband/acc/rte_vrb_pmd.c b/drivers/baseband/acc/rte_vrb_pmd.c
index fe23c01b5c..1f85e33462 100644
--- a/drivers/baseband/acc/rte_vrb_pmd.c
+++ b/drivers/baseband/acc/rte_vrb_pmd.c
@@ -4353,7 +4353,7 @@ vrb2_dequeue_mldts(struct rte_bbdev_queue_data *q_data,
static void
vrb_bbdev_init(struct rte_bbdev *dev, struct rte_pci_driver *drv)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct acc_device *d = dev->data->dev_private;
dev->dev_ops = &vrb_bbdev_ops;
diff --git a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
index cb805a1732..45bd171ca7 100644
--- a/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
+++ b/drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c
@@ -2873,7 +2873,7 @@ fpga_5gnr_dequeue_ldpc_dec(struct rte_bbdev_queue_data *q_data,
static void
fpga_5gnr_fec_init(struct rte_bbdev *dev, struct rte_pci_driver *drv)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev->dev_ops = &fpga_5gnr_ops;
dev->enqueue_ldpc_enc_ops = fpga_5gnr_enqueue_ldpc_enc;
@@ -3376,7 +3376,7 @@ int rte_fpga_5gnr_fec_configure(const char *dev_name, const struct rte_fpga_5gnr
dev_name);
return -ENODEV;
}
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(bbdev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(bbdev, *pci_dev);
rte_bbdev_log(INFO, "Configure dev id %x", pci_dev->id.device_id);
if (pci_dev->id.device_id == VC_5GNR_PF_DEVICE_ID)
return vc_5gnr_configure(dev_name, conf);
diff --git a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
index d27164c6f4..04ac445820 100644
--- a/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
+++ b/drivers/baseband/fpga_lte_fec/fpga_lte_fec.c
@@ -2316,7 +2316,7 @@ fpga_dequeue_dec(struct rte_bbdev_queue_data *q_data,
static void
fpga_lte_fec_init(struct rte_bbdev *dev, struct rte_pci_driver *drv)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev->dev_ops = &fpga_ops;
dev->enqueue_enc_ops = fpga_enqueue_enc;
diff --git a/drivers/bus/auxiliary/bus_auxiliary_driver.h b/drivers/bus/auxiliary/bus_auxiliary_driver.h
index cab5f86d03..65e1814ec0 100644
--- a/drivers/bus/auxiliary/bus_auxiliary_driver.h
+++ b/drivers/bus/auxiliary/bus_auxiliary_driver.h
@@ -128,9 +128,6 @@ struct rte_auxiliary_driver {
uint32_t drv_flags; /**< Flags RTE_AUXILIARY_DRV_*. */
};
-#define RTE_ETH_DEV_TO_AUXILIARY(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_auxiliary_device)
-
/** Device driver needs IOVA as VA and cannot work with IOVA as PA */
#define RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA 0x002
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index d443178404..01684466ed 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -60,8 +60,6 @@ struct rte_cdx_device {
struct rte_intr_handle *intr_handle; /**< Interrupt handle */
};
-#define RTE_ETH_DEV_TO_CDX_DEV(eth_dev) RTE_BUS_DEVICE((eth_dev)->device, struct rte_cdx_device)
-
#ifdef __cplusplus
/** C++ macro used to help building up tables of device IDs. */
#define RTE_CDX_DEVICE(vend, dev) \
diff --git a/drivers/bus/pci/bus_pci_driver.h b/drivers/bus/pci/bus_pci_driver.h
index cb7039f8d6..c04ebddf59 100644
--- a/drivers/bus/pci/bus_pci_driver.h
+++ b/drivers/bus/pci/bus_pci_driver.h
@@ -47,9 +47,6 @@ struct rte_pci_device {
/**< Handler of VFIO request interrupt */
};
-#define RTE_ETH_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
-
#ifdef __cplusplus
/** C++ macro used to help building up tables of device IDs */
#define RTE_PCI_DEVICE(vend, dev) \
diff --git a/drivers/bus/vdev/bus_vdev_driver.h b/drivers/bus/vdev/bus_vdev_driver.h
index 8d114e4b3b..ecfc5384fc 100644
--- a/drivers/bus/vdev/bus_vdev_driver.h
+++ b/drivers/bus/vdev/bus_vdev_driver.h
@@ -19,9 +19,6 @@ struct rte_vdev_device {
struct rte_device device; /**< Inherit core device */
};
-#define RTE_ETH_DEV_TO_VDEV(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_vdev_device)
-
static inline const char *
rte_vdev_device_name(const struct rte_vdev_device *dev)
{
diff --git a/drivers/compress/octeontx/otx_zip.c b/drivers/compress/octeontx/otx_zip.c
index 8673561a81..7cf3283680 100644
--- a/drivers/compress/octeontx/otx_zip.c
+++ b/drivers/compress/octeontx/otx_zip.c
@@ -142,7 +142,7 @@ zipvf_push_command(struct zipvf_qp *qp, union zip_inst_s *cmd)
int
zipvf_create(struct rte_compressdev *compressdev)
{
- struct rte_pci_device *pdev = RTE_BUS_DEVICE(compressdev->device, *pdev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(compressdev, *pdev);
struct zip_vf *zipvf = NULL;
char *dev_name = compressdev->data->name;
void *vbar0;
diff --git a/drivers/crypto/cnxk/cnxk_cryptodev_ops.c b/drivers/crypto/cnxk/cnxk_cryptodev_ops.c
index f437350539..d3cf1ddd57 100644
--- a/drivers/crypto/cnxk/cnxk_cryptodev_ops.c
+++ b/drivers/crypto/cnxk/cnxk_cryptodev_ops.c
@@ -481,7 +481,7 @@ cnxk_cpt_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
if (dev->data->queue_pairs[qp_id] != NULL)
cnxk_cpt_queue_pair_release(dev, qp_id);
- pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (pci_dev->mem_resource[2].addr == NULL) {
plt_err("Invalid PCI mem address");
diff --git a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
index d7b53723e7..3d980d096f 100644
--- a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
+++ b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c
@@ -4383,7 +4383,6 @@ static int
dpaa2_sec_dev_init(struct rte_cryptodev *cryptodev)
{
struct dpaa2_sec_dev_private *internals;
- struct rte_device *dev = cryptodev->device;
struct rte_dpaa2_device *dpaa2_dev;
struct rte_security_ctx *security_instance;
struct fsl_mc_io *dpseci;
@@ -4392,7 +4391,7 @@ dpaa2_sec_dev_init(struct rte_cryptodev *cryptodev)
int retcode, hw_id;
PMD_INIT_FUNC_TRACE();
- dpaa2_dev = RTE_BUS_DEVICE(dev, *dpaa2_dev);
+ dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(cryptodev, *dpaa2_dev);
hw_id = dpaa2_dev->object_id;
cryptodev->driver_id = cryptodev_driver_id;
diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.c b/drivers/crypto/octeontx/otx_cryptodev_ops.c
index a499c8d0bc..d6d1b2cea9 100644
--- a/drivers/crypto/octeontx/otx_cryptodev_ops.c
+++ b/drivers/crypto/octeontx/otx_cryptodev_ops.c
@@ -156,7 +156,7 @@ otx_cpt_que_pair_setup(struct rte_cryptodev *dev,
DEFAULT_CMD_QLEN);
}
- pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (pci_dev->mem_resource[0].addr == NULL) {
CPT_LOG_ERR("PCI mem address null");
@@ -1001,7 +1001,7 @@ static struct rte_cryptodev_ops cptvf_ops = {
int
otx_cpt_dev_create(struct rte_cryptodev *c_dev)
{
- struct rte_pci_device *pdev = RTE_BUS_DEVICE(c_dev->device, *pdev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(c_dev, *pdev);
struct cpt_vf *cptvf = NULL;
void *reg_base;
char dev_name[32];
diff --git a/drivers/event/cnxk/cnxk_eventdev.c b/drivers/event/cnxk/cnxk_eventdev.c
index 6f000ff49e..272ba235a4 100644
--- a/drivers/event/cnxk/cnxk_eventdev.c
+++ b/drivers/event/cnxk/cnxk_eventdev.c
@@ -654,7 +654,7 @@ cnxk_sso_init(struct rte_eventdev *event_dev)
return -ENOMEM;
}
- pci_dev = RTE_BUS_DEVICE(event_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(event_dev, *pci_dev);
dev->sso.pci_dev = pci_dev;
*(uint64_t *)mz->addr = (uint64_t)dev;
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 82075bbf0b..c78783e59d 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -784,7 +784,7 @@ dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
dlb2_pf_iface_fn_ptrs_init();
- pci_dev = RTE_BUS_DEVICE(eventdev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eventdev, *pci_dev);
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
diff --git a/drivers/event/skeleton/skeleton_eventdev.c b/drivers/event/skeleton/skeleton_eventdev.c
index 4292644fde..c0e06e4aa0 100644
--- a/drivers/event/skeleton/skeleton_eventdev.c
+++ b/drivers/event/skeleton/skeleton_eventdev.c
@@ -332,7 +332,7 @@ skeleton_eventdev_init(struct rte_eventdev *eventdev)
if (rte_eal_process_type() != RTE_PROC_PRIMARY)
return 0;
- pci_dev = RTE_BUS_DEVICE(eventdev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eventdev, *pci_dev);
skel->reg_base = (uintptr_t)pci_dev->mem_resource[0].addr;
if (!skel->reg_base) {
diff --git a/drivers/net/ark/ark_ethdev.c b/drivers/net/ark/ark_ethdev.c
index 8b25ed948f..d6e34021ce 100644
--- a/drivers/net/ark/ark_ethdev.c
+++ b/drivers/net/ark/ark_ethdev.c
@@ -315,7 +315,7 @@ ark_dev_init(struct rte_eth_dev *dev)
if (ret)
return ret;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
rte_eth_copy_pci_info(dev, pci_dev);
dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
diff --git a/drivers/net/atlantic/atl_ethdev.c b/drivers/net/atlantic/atl_ethdev.c
index 2925dc2478..d55d6d50bb 100644
--- a/drivers/net/atlantic/atl_ethdev.c
+++ b/drivers/net/atlantic/atl_ethdev.c
@@ -359,7 +359,7 @@ static int
eth_atl_dev_init(struct rte_eth_dev *eth_dev)
{
struct atl_adapter *adapter = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct aq_hw_s *hw = ATL_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
int err = 0;
@@ -478,7 +478,7 @@ static int
atl_dev_start(struct rte_eth_dev *dev)
{
struct aq_hw_s *hw = ATL_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
int status;
@@ -606,7 +606,7 @@ atl_dev_stop(struct rte_eth_dev *dev)
struct rte_eth_link link;
struct aq_hw_s *hw =
ATL_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
PMD_INIT_FUNC_TRACE();
@@ -687,7 +687,7 @@ atl_dev_set_link_down(struct rte_eth_dev *dev)
static int
atl_dev_close(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct aq_hw_s *hw;
int ret;
@@ -1093,7 +1093,7 @@ atl_fw_version_get(struct rte_eth_dev *dev, char *fw_version, size_t fw_size)
static int
atl_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev_info->max_rx_queues = AQ_HW_MAX_RX_QUEUES;
dev_info->max_tx_queues = AQ_HW_MAX_TX_QUEUES;
@@ -1344,7 +1344,7 @@ atl_dev_link_status_print(struct rte_eth_dev *dev)
#ifdef DEBUG
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
PMD_DRV_LOG(DEBUG, "PCI Address: " PCI_PRI_FMT,
pci_dev->addr.domain,
diff --git a/drivers/net/avp/avp_ethdev.c b/drivers/net/avp/avp_ethdev.c
index 3bc5171336..8af6c45381 100644
--- a/drivers/net/avp/avp_ethdev.c
+++ b/drivers/net/avp/avp_ethdev.c
@@ -361,7 +361,7 @@ static void *
avp_dev_translate_address(struct rte_eth_dev *eth_dev,
rte_iova_t host_phys_addr)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_mem_resource *resource;
struct rte_avp_memmap_info *info;
struct rte_avp_memmap *map;
@@ -414,7 +414,7 @@ avp_dev_version_check(uint32_t version)
static int
avp_dev_check_regions(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_avp_memmap_info *memmap;
struct rte_avp_device_info *info;
struct rte_mem_resource *resource;
@@ -550,7 +550,7 @@ _avp_set_rx_queue_mappings(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
static void
_avp_set_queue_counts(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct avp_dev *avp = AVP_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
struct rte_avp_device_info *host_info;
void *addr;
@@ -610,7 +610,7 @@ avp_dev_attach(struct rte_eth_dev *eth_dev)
* re-run the device create utility which will parse the new host info
* and setup the AVP device queue pointers.
*/
- ret = avp_dev_create(RTE_ETH_DEV_TO_PCI(eth_dev), eth_dev);
+ ret = avp_dev_create(RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device), eth_dev);
if (ret < 0) {
PMD_DRV_LOG_LINE(ERR, "Failed to re-create AVP device, ret=%d",
ret);
@@ -664,7 +664,7 @@ static void
avp_dev_interrupt_handler(void *data)
{
struct rte_eth_dev *eth_dev = data;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR].addr;
uint32_t status, value;
int ret;
@@ -723,7 +723,7 @@ avp_dev_interrupt_handler(void *data)
static int
avp_dev_enable_interrupts(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR].addr;
int ret;
@@ -748,7 +748,7 @@ avp_dev_enable_interrupts(struct rte_eth_dev *eth_dev)
static int
avp_dev_disable_interrupts(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR].addr;
int ret;
@@ -773,7 +773,7 @@ avp_dev_disable_interrupts(struct rte_eth_dev *eth_dev)
static int
avp_dev_setup_interrupts(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret;
/* register a callback handler with UIO for interrupt notifications */
@@ -793,7 +793,7 @@ avp_dev_setup_interrupts(struct rte_eth_dev *eth_dev)
static int
avp_dev_migration_pending(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR].addr;
uint32_t value;
@@ -954,7 +954,7 @@ eth_avp_dev_init(struct rte_eth_dev *eth_dev)
struct rte_pci_device *pci_dev;
int ret;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
eth_dev->dev_ops = &avp_eth_dev_ops;
eth_dev->rx_pkt_burst = &avp_recv_pkts;
eth_dev->tx_pkt_burst = &avp_xmit_pkts;
@@ -1977,7 +1977,7 @@ avp_dev_tx_queue_release_all(struct rte_eth_dev *eth_dev)
static int
avp_dev_configure(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct avp_dev *avp = AVP_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
struct rte_avp_device_info *host_info;
struct rte_avp_device_config config;
diff --git a/drivers/net/axgbe/axgbe_ethdev.c b/drivers/net/axgbe/axgbe_ethdev.c
index c14d04a11d..aceec49c0c 100644
--- a/drivers/net/axgbe/axgbe_ethdev.c
+++ b/drivers/net/axgbe/axgbe_ethdev.c
@@ -2230,7 +2230,7 @@ eth_axgbe_dev_init(struct rte_eth_dev *eth_dev)
rte_bit_relaxed_set32(AXGBE_STOPPED, &pdata->dev_state);
pdata->eth_dev = eth_dev;
- pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
pdata->pci_dev = pci_dev;
pdata->xgmac_regs =
@@ -2453,7 +2453,7 @@ axgbe_dev_close(struct rte_eth_dev *eth_dev)
return 0;
pdata = eth_dev->data->dev_private;
- pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
axgbe_dev_clear_queues(eth_dev);
/* disable uio intr before callback unregister */
diff --git a/drivers/net/bnx2x/bnx2x_ethdev.c b/drivers/net/bnx2x/bnx2x_ethdev.c
index 7b96e1acee..4f1f97a999 100644
--- a/drivers/net/bnx2x/bnx2x_ethdev.c
+++ b/drivers/net/bnx2x/bnx2x_ethdev.c
@@ -639,7 +639,7 @@ bnx2x_common_dev_init(struct rte_eth_dev *eth_dev, int is_vf)
/* Extract key data structures */
sc = eth_dev->data->dev_private;
- pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
pci_addr = pci_dev->addr;
snprintf(sc->devinfo.name, NAME_SIZE, PCI_SHORT_PRI_FMT ":dpdk-port-%u",
diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index 071093aabc..5506037cc2 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -862,7 +862,7 @@ static int bnxt_alloc_prev_ring_stats(struct bnxt *bp)
static int bnxt_start_nic(struct bnxt *bp)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(bp->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(bp->eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
uint32_t queue_id, base = BNXT_MISC_VEC_ID;
@@ -1167,7 +1167,7 @@ uint64_t bnxt_eth_rss_support(struct bnxt *bp)
static int bnxt_dev_info_get_op(struct rte_eth_dev *eth_dev,
struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pdev = RTE_BUS_DEVICE(eth_dev->device, *pdev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
struct bnxt *bp = eth_dev->data->dev_private;
uint16_t max_vnics, i, j, vpool, vrxq;
unsigned int max_rx_rings;
@@ -1719,7 +1719,7 @@ static int bnxt_ptp_start(struct bnxt *bp)
static int bnxt_dev_stop(struct rte_eth_dev *eth_dev)
{
struct bnxt *bp = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rte_eth_link link;
uint16_t i;
@@ -5143,7 +5143,7 @@ bool bnxt_stratus_device(struct bnxt *bp)
static int bnxt_map_pci_bars(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct bnxt *bp = eth_dev->data->dev_private;
/* enable device (incl. PCI PM wakeup), and bus-mastering */
@@ -6600,7 +6600,7 @@ bnxt_parse_dev_args(struct bnxt *bp, struct rte_devargs *devargs)
*/
static int bnxt_drv_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct bnxt *bp = eth_dev->data->dev_private;
int rc = 0;
@@ -6684,7 +6684,7 @@ static int bnxt_drv_init(struct rte_eth_dev *eth_dev)
static int
bnxt_dev_init(struct rte_eth_dev *eth_dev, void *params __rte_unused)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
static int version_printed;
struct bnxt *bp;
int rc;
diff --git a/drivers/net/bnxt/tf_ulp/bnxt_ulp.c b/drivers/net/bnxt/tf_ulp/bnxt_ulp.c
index e1e2c0e878..8ff0e20364 100644
--- a/drivers/net/bnxt/tf_ulp/bnxt_ulp.c
+++ b/drivers/net/bnxt/tf_ulp/bnxt_ulp.c
@@ -192,7 +192,7 @@ ulp_session_init(struct bnxt *bp,
if (!bp)
return NULL;
- pci_dev = RTE_BUS_DEVICE(bp->eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(bp->eth_dev, *pci_dev);
pci_addr = &pci_dev->addr;
pthread_mutex_lock(&bnxt_ulp_global_mutex);
@@ -556,7 +556,7 @@ bnxt_ulp_port_deinit(struct bnxt *bp)
bp->eth_dev->data->port_id);
/* Get the session details */
- pci_dev = RTE_BUS_DEVICE(bp->eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(bp->eth_dev, *pci_dev);
pci_addr = &pci_dev->addr;
pthread_mutex_lock(&bnxt_ulp_global_mutex);
session = ulp_get_session(bp, pci_addr);
diff --git a/drivers/net/cnxk/cnxk_ethdev.c b/drivers/net/cnxk/cnxk_ethdev.c
index 06d1c9b362..7ae16186c6 100644
--- a/drivers/net/cnxk/cnxk_ethdev.c
+++ b/drivers/net/cnxk/cnxk_ethdev.c
@@ -2177,7 +2177,7 @@ cnxk_eth_dev_init(struct rte_eth_dev *eth_dev)
if (rte_eal_process_type() != RTE_PROC_PRIMARY)
return 0;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
rte_eth_copy_pci_info(eth_dev, pci_dev);
/* Parse devargs string */
diff --git a/drivers/net/cnxk/cnxk_ethdev_ops.c b/drivers/net/cnxk/cnxk_ethdev_ops.c
index 49e77e49a6..460ffa32b6 100644
--- a/drivers/net/cnxk/cnxk_ethdev_ops.c
+++ b/drivers/net/cnxk/cnxk_ethdev_ops.c
@@ -7,7 +7,7 @@
int
cnxk_nix_info_get(struct rte_eth_dev *eth_dev, struct rte_eth_dev_info *devinfo)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev);
int max_rx_pktlen;
diff --git a/drivers/net/cxgbe/cxgbe_ethdev.c b/drivers/net/cxgbe/cxgbe_ethdev.c
index 0c337a6cc8..82e67eeff1 100644
--- a/drivers/net/cxgbe/cxgbe_ethdev.c
+++ b/drivers/net/cxgbe/cxgbe_ethdev.c
@@ -1704,7 +1704,7 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)
eth_dev->dev_ops = &cxgbe_eth_dev_ops;
eth_dev->rx_pkt_burst = &cxgbe_recv_pkts;
eth_dev->tx_pkt_burst = &cxgbe_xmit_pkts;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* for secondary processes, we attach to ethdevs allocated by primary
* and do minimal initialization.
@@ -1767,7 +1767,7 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)
static int eth_cxgbe_dev_uninit(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
uint16_t port_id;
int err = 0;
diff --git a/drivers/net/cxgbe/cxgbevf_ethdev.c b/drivers/net/cxgbe/cxgbevf_ethdev.c
index d8eba8afef..750dc7da4d 100644
--- a/drivers/net/cxgbe/cxgbevf_ethdev.c
+++ b/drivers/net/cxgbe/cxgbevf_ethdev.c
@@ -113,7 +113,7 @@ static int eth_cxgbevf_dev_init(struct rte_eth_dev *eth_dev)
eth_dev->dev_ops = &cxgbevf_eth_dev_ops;
eth_dev->rx_pkt_burst = &cxgbe_recv_pkts;
eth_dev->tx_pkt_burst = &cxgbe_xmit_pkts;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* for secondary processes, we attach to ethdevs allocated by primary
* and do minimal initialization.
@@ -177,7 +177,7 @@ static int eth_cxgbevf_dev_init(struct rte_eth_dev *eth_dev)
static int eth_cxgbevf_dev_uninit(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
uint16_t port_id;
int err = 0;
diff --git a/drivers/net/dpaa/dpaa_ethdev.c b/drivers/net/dpaa/dpaa_ethdev.c
index d4b4793f16..9f976d179b 100644
--- a/drivers/net/dpaa/dpaa_ethdev.c
+++ b/drivers/net/dpaa/dpaa_ethdev.c
@@ -217,7 +217,6 @@ dpaa_eth_dev_configure(struct rte_eth_dev *dev)
uint64_t rx_offloads = eth_conf->rxmode.offloads;
uint64_t tx_offloads = eth_conf->txmode.offloads;
struct dpaa_if *dpaa_intf = dev->data->dev_private;
- struct rte_device *rdev = dev->device;
struct rte_eth_link *link = &dev->data->dev_link;
struct rte_dpaa_device *dpaa_dev;
struct fman_if *fif = dev->process_private;
@@ -230,7 +229,7 @@ dpaa_eth_dev_configure(struct rte_eth_dev *dev)
PMD_INIT_FUNC_TRACE();
- dpaa_dev = RTE_BUS_DEVICE(rdev, *dpaa_dev);
+ dpaa_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa_dev);
intr_handle = dpaa_dev->intr_handle;
__fif = container_of(fif, struct __fman_if, __if);
@@ -426,13 +425,12 @@ dpaa_supported_ptypes_get(struct rte_eth_dev *dev, size_t *no_of_elements)
static void dpaa_interrupt_handler(void *param)
{
struct rte_eth_dev *dev = param;
- struct rte_device *rdev = dev->device;
struct rte_dpaa_device *dpaa_dev;
struct rte_intr_handle *intr_handle;
uint64_t buf;
int bytes_read;
- dpaa_dev = RTE_BUS_DEVICE(rdev, *dpaa_dev);
+ dpaa_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa_dev);
intr_handle = dpaa_dev->intr_handle;
if (rte_intr_fd_get(intr_handle) < 0)
@@ -502,7 +500,6 @@ static int dpaa_eth_dev_close(struct rte_eth_dev *dev)
{
struct fman_if *fif = dev->process_private;
struct __fman_if *__fif;
- struct rte_device *rdev = dev->device;
struct rte_dpaa_device *dpaa_dev;
struct rte_intr_handle *intr_handle;
struct rte_eth_link *link = &dev->data->dev_link;
@@ -530,7 +527,7 @@ static int dpaa_eth_dev_close(struct rte_eth_dev *dev)
}
}
- dpaa_dev = RTE_BUS_DEVICE(rdev, *dpaa_dev);
+ dpaa_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa_dev);
intr_handle = dpaa_dev->intr_handle;
__fif = container_of(fif, struct __fman_if, __if);
@@ -1267,9 +1264,8 @@ int dpaa_eth_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,
/* Set up the device interrupt handler */
if (dev->intr_handle == NULL) {
struct rte_dpaa_device *dpaa_dev;
- struct rte_device *rdev = dev->device;
- dpaa_dev = RTE_BUS_DEVICE(rdev, *dpaa_dev);
+ dpaa_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa_dev);
dev->intr_handle = dpaa_dev->intr_handle;
if (rte_intr_vec_list_alloc(dev->intr_handle,
NULL, dpaa_push_queue_max_num())) {
@@ -2119,7 +2115,7 @@ dpaa_dev_init_secondary(struct rte_eth_dev *eth_dev)
PMD_INIT_FUNC_TRACE();
- dpaa_device = RTE_BUS_DEVICE(eth_dev->device, *dpaa_device);
+ dpaa_device = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa_device);
dev_id = dpaa_device->id.dev_id;
cfg = dpaa_get_eth_port_cfg(dev_id);
fman_intf = cfg->fman_if;
@@ -2236,7 +2232,7 @@ dpaa_dev_init(struct rte_eth_dev *eth_dev)
PMD_INIT_FUNC_TRACE();
- dpaa_device = RTE_BUS_DEVICE(eth_dev->device, *dpaa_device);
+ dpaa_device = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa_device);
dev_id = dpaa_device->id.dev_id;
dpaa_intf = eth_dev->data->dev_private;
cfg = dpaa_get_eth_port_cfg(dev_id);
diff --git a/drivers/net/dpaa2/dpaa2_ethdev.c b/drivers/net/dpaa2/dpaa2_ethdev.c
index dc9ea700ac..803a8321e0 100644
--- a/drivers/net/dpaa2/dpaa2_ethdev.c
+++ b/drivers/net/dpaa2/dpaa2_ethdev.c
@@ -1339,7 +1339,6 @@ dpaa2_eth_setup_irqs(struct rte_eth_dev *dev, int enable)
static int
dpaa2_dev_start(struct rte_eth_dev *dev)
{
- struct rte_device *rdev = dev->device;
struct rte_dpaa2_device *dpaa2_dev;
struct rte_eth_dev_data *data = dev->data;
struct dpaa2_dev_priv *priv = data->dev_private;
@@ -1351,7 +1350,7 @@ dpaa2_dev_start(struct rte_eth_dev *dev)
int ret, i;
struct rte_intr_handle *intr_handle;
- dpaa2_dev = RTE_BUS_DEVICE(rdev, *dpaa2_dev);
+ dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa2_dev);
intr_handle = dpaa2_dev->intr_handle;
PMD_INIT_FUNC_TRACE();
@@ -1458,12 +1457,11 @@ dpaa2_dev_stop(struct rte_eth_dev *dev)
struct fsl_mc_io *dpni = dev->process_private;
int ret;
struct rte_eth_link link;
- struct rte_device *rdev = dev->device;
struct rte_intr_handle *intr_handle;
struct rte_dpaa2_device *dpaa2_dev;
uint16_t i;
- dpaa2_dev = RTE_BUS_DEVICE(rdev, *dpaa2_dev);
+ dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *dpaa2_dev);
intr_handle = dpaa2_dev->intr_handle;
PMD_INIT_FUNC_TRACE();
@@ -2918,7 +2916,7 @@ dpaa2_dev_init(struct rte_eth_dev *eth_dev)
return 0;
}
- dpaa2_dev = RTE_BUS_DEVICE(dev, *dpaa2_dev);
+ dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa2_dev);
hw_id = dpaa2_dev->object_id;
ret = dpni_open(dpni_dev, CMD_PRI_LOW, hw_id, &priv->token);
diff --git a/drivers/net/dpaa2/dpaa2_recycle.c b/drivers/net/dpaa2/dpaa2_recycle.c
index 14416c41d0..f78d12362e 100644
--- a/drivers/net/dpaa2/dpaa2_recycle.c
+++ b/drivers/net/dpaa2/dpaa2_recycle.c
@@ -607,9 +607,8 @@ lx_serdes_eth_lpbk(uint16_t mac_id, int en)
int
dpaa2_dev_recycle_config(struct rte_eth_dev *eth_dev)
{
- struct rte_device *dev = eth_dev->device;
struct dpaa2_dev_priv *priv = eth_dev->data->dev_private;
- struct rte_dpaa2_device *dpaa2_dev = RTE_BUS_DEVICE(dev, *dpaa2_dev);
+ struct rte_dpaa2_device *dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa2_dev);
struct fsl_mc_io *dpni_dev = eth_dev->process_private;
struct dpni_port_cfg port_cfg;
int ret;
@@ -674,9 +673,8 @@ dpaa2_dev_recycle_config(struct rte_eth_dev *eth_dev)
int
dpaa2_dev_recycle_deconfig(struct rte_eth_dev *eth_dev)
{
- struct rte_device *dev = eth_dev->device;
struct dpaa2_dev_priv *priv = eth_dev->data->dev_private;
- struct rte_dpaa2_device *dpaa2_dev = RTE_BUS_DEVICE(dev, *dpaa2_dev);
+ struct rte_dpaa2_device *dpaa2_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *dpaa2_dev);
struct fsl_mc_io *dpni_dev = eth_dev->process_private;
struct dpni_port_cfg port_cfg;
int ret = 0;
diff --git a/drivers/net/ena/ena_ethdev.c b/drivers/net/ena/ena_ethdev.c
index ea4afbc75d..ad2ac6dbbf 100644
--- a/drivers/net/ena/ena_ethdev.c
+++ b/drivers/net/ena/ena_ethdev.c
@@ -924,7 +924,7 @@ static inline void ena_indirect_table_release(struct ena_adapter *adapter)
static int ena_close(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ena_adapter *adapter = dev->data->dev_private;
struct ena_com_dev *ena_dev = &adapter->ena_dev;
@@ -1457,7 +1457,7 @@ static int ena_stop(struct rte_eth_dev *dev)
{
struct ena_adapter *adapter = dev->data->dev_private;
struct ena_com_dev *ena_dev = &adapter->ena_dev;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint16_t i;
int rc;
@@ -1503,7 +1503,7 @@ static int ena_create_io_queue(struct rte_eth_dev *dev, struct ena_ring *ring)
{
struct ena_adapter *adapter = ring->adapter;
struct ena_com_dev *ena_dev = &adapter->ena_dev;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ena_com_create_io_ctx ctx =
/* policy set to _HOST just to satisfy icc compiler */
@@ -2422,7 +2422,7 @@ static int eth_ena_dev_init(struct rte_eth_dev *eth_dev)
adapter->edev_data = eth_dev->data;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
PMD_INIT_LOG_LINE(INFO, "Initializing " PCI_PRI_FMT,
pci_dev->addr.domain,
@@ -3978,7 +3978,7 @@ static int ena_parse_devargs(struct ena_adapter *adapter, struct rte_devargs *de
static int ena_setup_rx_intr(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int rc;
uint16_t vectors_nb, i;
diff --git a/drivers/net/enetc/enetc4_ethdev.c b/drivers/net/enetc/enetc4_ethdev.c
index df9f007473..78eba70a08 100644
--- a/drivers/net/enetc/enetc4_ethdev.c
+++ b/drivers/net/enetc/enetc4_ethdev.c
@@ -956,7 +956,7 @@ enetc4_dev_hw_init(struct rte_eth_dev *eth_dev)
{
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
eth_dev->rx_pkt_burst = &enetc_recv_pkts_nc;
eth_dev->tx_pkt_burst = &enetc_xmit_pkts_nc;
@@ -986,7 +986,7 @@ enetc4_dev_init(struct rte_eth_dev *eth_dev)
{
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int error = 0;
uint32_t si_cap;
struct enetc_hw *enetc_hw = &hw->hw;
diff --git a/drivers/net/enetc/enetc4_vf.c b/drivers/net/enetc/enetc4_vf.c
index 3f257234a0..bec7128e41 100644
--- a/drivers/net/enetc/enetc4_vf.c
+++ b/drivers/net/enetc/enetc4_vf.c
@@ -1249,7 +1249,7 @@ enetc4_vf_dev_init(struct rte_eth_dev *eth_dev)
{
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int error = 0;
uint32_t si_cap;
struct enetc_hw *enetc_hw = &hw->hw;
@@ -1297,7 +1297,7 @@ enetc4_vf_dev_intr(struct rte_eth_dev *eth_dev, bool enable)
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
struct enetc_hw *enetc_hw = &hw->hw;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret = 0;
diff --git a/drivers/net/enetc/enetc_ethdev.c b/drivers/net/enetc/enetc_ethdev.c
index b2bbace16c..f41f3c1803 100644
--- a/drivers/net/enetc/enetc_ethdev.c
+++ b/drivers/net/enetc/enetc_ethdev.c
@@ -886,7 +886,7 @@ static int
enetc_dev_init(struct rte_eth_dev *eth_dev)
{
int error = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct enetc_eth_hw *hw =
ENETC_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
diff --git a/drivers/net/enic/enic_ethdev.c b/drivers/net/enic/enic_ethdev.c
index a853a5047a..2e5cd186f9 100644
--- a/drivers/net/enic/enic_ethdev.c
+++ b/drivers/net/enic/enic_ethdev.c
@@ -454,7 +454,7 @@ static uint32_t speed_capa_from_pci_id(struct rte_eth_dev *eth_dev)
struct rte_pci_device *pdev;
uint16_t id;
- pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
id = pdev->id.subsystem_device_id;
for (m = vic_speed_capa_map; m->sub_devid != 0; m++) {
if (m->sub_devid == id)
@@ -1292,7 +1292,7 @@ static int eth_enic_dev_init(struct rte_eth_dev *eth_dev,
enic->rte_dev = eth_dev;
enic->dev_data = eth_dev->data;
- pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
rte_eth_copy_pci_info(eth_dev, pdev);
enic->pdev = pdev;
addr = &pdev->addr;
diff --git a/drivers/net/enic/enic_fm_flow.c b/drivers/net/enic/enic_fm_flow.c
index c2c3e55206..4b0a513977 100644
--- a/drivers/net/enic/enic_fm_flow.c
+++ b/drivers/net/enic/enic_fm_flow.c
@@ -3229,7 +3229,7 @@ enic_fm_init(struct enic *enic)
if (rte_eth_dev_is_repr(enic->rte_dev))
addr = &VF_ENIC_TO_VF_REP(enic)->bdf;
else
- addr = &RTE_ETH_DEV_TO_PCI(enic->rte_dev)->addr;
+ addr = &RTE_CLASS_TO_BUS_DEVICE(enic->rte_dev, struct rte_pci_device)->addr;
rc = enic_fm_find_vnic(enic, addr, &enic->fm_vnic_handle);
if (rc) {
ENICPMD_LOG(ERR, "cannot find vnic handle for %x:%x:%x",
@@ -3361,7 +3361,7 @@ enic_fm_allocate_switch_domain(struct enic *pf)
if (rte_eth_dev_is_repr(pf->rte_dev))
return -EINVAL;
cur = pf;
- cur_a = &RTE_ETH_DEV_TO_PCI(cur->rte_dev)->addr;
+ cur_a = &RTE_CLASS_TO_BUS_DEVICE(cur->rte_dev, struct rte_pci_device)->addr;
/* Go through ports and find another PF that is on the same adapter */
RTE_ETH_FOREACH_DEV(pid) {
dev = &rte_eth_devices[pid];
@@ -3373,7 +3373,7 @@ enic_fm_allocate_switch_domain(struct enic *pf)
continue;
/* dev is another PF. Is it on the same adapter? */
prev = pmd_priv(dev);
- prev_a = &RTE_ETH_DEV_TO_PCI(dev)->addr;
+ prev_a = &RTE_CLASS_TO_BUS_DEVICE(dev, struct rte_pci_device)->addr;
if (!enic_fm_find_vnic(cur, prev_a, &vnic_h)) {
ENICPMD_LOG(DEBUG, "Port %u (PF BDF %x:%x:%x) and port %u (PF BDF %x:%x:%x domain %u) are on the same VIC",
cur->rte_dev->data->port_id,
diff --git a/drivers/net/enic/enic_vf_representor.c b/drivers/net/enic/enic_vf_representor.c
index 05b2efedcb..fc836100b4 100644
--- a/drivers/net/enic/enic_vf_representor.c
+++ b/drivers/net/enic/enic_vf_representor.c
@@ -655,7 +655,7 @@ int enic_vf_representor_init(struct rte_eth_dev *eth_dev, void *init_params)
}
/* Check for non-existent VFs */
- pdev = RTE_ETH_DEV_TO_PCI(pf->rte_dev);
+ pdev = RTE_CLASS_TO_BUS_DEVICE(pf->rte_dev, *pdev);
if (vf->vf_id >= pdev->max_vfs) {
ENICPMD_LOG(ERR, "VF ID is invalid. vf_id %u max_vfs %u",
vf->vf_id, pdev->max_vfs);
diff --git a/drivers/net/gve/gve_ethdev.c b/drivers/net/gve/gve_ethdev.c
index 73f4935b1f..227e1cc70e 100644
--- a/drivers/net/gve/gve_ethdev.c
+++ b/drivers/net/gve/gve_ethdev.c
@@ -1410,7 +1410,7 @@ gve_dev_init(struct rte_eth_dev *eth_dev)
return 0;
}
- pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
reg_bar = pci_dev->mem_resource[GVE_REG_BAR].addr;
if (!reg_bar) {
diff --git a/drivers/net/hinic/hinic_pmd_ethdev.c b/drivers/net/hinic/hinic_pmd_ethdev.c
index 75534c1ce2..91a4348fb6 100644
--- a/drivers/net/hinic/hinic_pmd_ethdev.c
+++ b/drivers/net/hinic/hinic_pmd_ethdev.c
@@ -1234,7 +1234,7 @@ static int hinic_dev_stop(struct rte_eth_dev *dev)
static void hinic_disable_interrupt(struct rte_eth_dev *dev)
{
struct hinic_nic_dev *nic_dev = HINIC_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
int ret, retries = 0;
rte_bit_relaxed_clear32(HINIC_DEV_INTR_EN, &nic_dev->dev_status);
@@ -2745,7 +2745,7 @@ static int hinic_nic_dev_create(struct rte_eth_dev *eth_dev)
eth_dev->data->name);
return -ENOMEM;
}
- nic_dev->hwdev->pcidev_hdl = RTE_ETH_DEV_TO_PCI(eth_dev);
+ nic_dev->hwdev->pcidev_hdl = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *nic_dev->hwdev->pcidev_hdl);
/* init osdep*/
rc = hinic_osdep_init(nic_dev->hwdev);
@@ -3086,7 +3086,7 @@ static int hinic_func_init(struct rte_eth_dev *eth_dev)
u32 mac_size;
int rc;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* EAL is SECONDARY and eth_dev is already created */
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
@@ -3218,7 +3218,7 @@ static int hinic_dev_init(struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
PMD_DRV_LOG(INFO, "Initializing pf hinic-" PCI_PRI_FMT " in %s process",
pci_dev->addr.domain, pci_dev->addr.bus,
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index 5d12cf7b5f..d09a8f7e7d 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -74,10 +74,11 @@ struct mgmt_event_handle {
};
bool
-hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *rte_dev)
+hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
- return ((RTE_ETH_DEV_TO_PCI(rte_dev)->kdrv == RTE_PCI_KDRV_VFIO) &&
- (rte_vfio_noiommu_is_enabled() != 1));
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
+
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && rte_vfio_noiommu_is_enabled() != 1;
}
int
diff --git a/drivers/net/hinic3/hinic3_ethdev.c b/drivers/net/hinic3/hinic3_ethdev.c
index f4eb788686..361e52f7b9 100644
--- a/drivers/net/hinic3/hinic3_ethdev.c
+++ b/drivers/net/hinic3/hinic3_ethdev.c
@@ -1474,7 +1474,7 @@ hinic3_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t sq_id)
int
hinic3_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = PCI_DEV_TO_INTR_HANDLE(pci_dev);
struct hinic3_nic_dev *nic_dev = HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
uint16_t msix_intr;
@@ -1493,7 +1493,7 @@ hinic3_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
int
hinic3_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = PCI_DEV_TO_INTR_HANDLE(pci_dev);
struct hinic3_nic_dev *nic_dev = HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
uint16_t msix_intr;
@@ -1695,7 +1695,7 @@ static void
hinic3_disable_interrupt(struct rte_eth_dev *dev)
{
struct hinic3_nic_dev *nic_dev = HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!hinic3_get_bit(HINIC3_DEV_INIT, &nic_dev->dev_status))
return;
@@ -1710,7 +1710,7 @@ static void
hinic3_enable_interrupt(struct rte_eth_dev *dev)
{
struct hinic3_nic_dev *nic_dev = HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!hinic3_get_bit(HINIC3_DEV_INIT, &nic_dev->dev_status))
return;
@@ -2080,7 +2080,7 @@ hinic3_dev_release(struct rte_eth_dev *eth_dev)
{
struct hinic3_nic_dev *nic_dev =
HINIC3_ETH_DEV_TO_PRIVATE_NIC_DEV(eth_dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int qid;
/* Release io resource. */
@@ -3394,7 +3394,7 @@ hinic3_func_init(struct rte_eth_dev *eth_dev)
struct rte_pci_device *pci_dev = NULL;
int err;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* EAL is secondary and eth_dev is already created. */
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
@@ -3460,7 +3460,7 @@ hinic3_func_init(struct rte_eth_dev *eth_dev)
err = -ENOMEM;
goto alloc_hwdev_mem_fail;
}
- nic_dev->hwdev->pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ nic_dev->hwdev->pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *nic_dev->hwdev->pci_dev);
nic_dev->hwdev->dev_handle = nic_dev;
nic_dev->hwdev->eth_dev = eth_dev;
nic_dev->hwdev->port_id = eth_dev->data->port_id;
@@ -3616,7 +3616,7 @@ hinic3_dev_init(struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
PMD_DRV_LOG(INFO, "Initializing %.4x:%.2x:%.2x.%x in %s process",
pci_dev->addr.domain, pci_dev->addr.bus,
diff --git a/drivers/net/hns3/hns3_cmd.c b/drivers/net/hns3/hns3_cmd.c
index ad4ef9e189..34e12e7359 100644
--- a/drivers/net/hns3/hns3_cmd.c
+++ b/drivers/net/hns3/hns3_cmd.c
@@ -551,7 +551,7 @@ hns3_set_dcb_capability(struct hns3_hw *hw)
return;
eth_dev = &rte_eth_devices[hw->data->port_id];
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
device_id = pci_dev->id.device_id;
if (device_id == HNS3_DEV_ID_25GE_RDMA ||
diff --git a/drivers/net/hns3/hns3_common.c b/drivers/net/hns3/hns3_common.c
index 28d7e94ffb..29b51856d9 100644
--- a/drivers/net/hns3/hns3_common.c
+++ b/drivers/net/hns3/hns3_common.c
@@ -812,7 +812,7 @@ hns3_init_ring_with_vector(struct hns3_hw *hw)
int
hns3_map_rx_interrupt(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct hns3_hw *hw = HNS3_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint16_t base = RTE_INTR_VEC_ZERO_OFFSET;
@@ -878,7 +878,7 @@ hns3_map_rx_interrupt(struct rte_eth_dev *dev)
void
hns3_unmap_rx_interrupt(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct hns3_adapter *hns = dev->data->dev_private;
struct hns3_hw *hw = &hns->hw;
@@ -912,7 +912,7 @@ int
hns3_restore_rx_interrupt(struct hns3_hw *hw)
{
struct rte_eth_dev *dev = &rte_eth_devices[hw->data->port_id];
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint16_t q_id;
int ret;
@@ -943,7 +943,7 @@ hns3_get_pci_revision_id(struct hns3_hw *hw, uint8_t *revision_id)
int ret;
eth_dev = &rte_eth_devices[hw->data->port_id];
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
ret = rte_pci_read_config(pci_dev, &revision, 1, RTE_PCI_REVISION_ID);
if (ret != 1) {
hns3_err(hw, "failed to read pci revision id, ret = %d", ret);
diff --git a/drivers/net/hns3/hns3_ethdev.c b/drivers/net/hns3/hns3_ethdev.c
index a66fc5d81a..dbe26df77d 100644
--- a/drivers/net/hns3/hns3_ethdev.c
+++ b/drivers/net/hns3/hns3_ethdev.c
@@ -4526,8 +4526,7 @@ hns3_get_port_supported_speed(struct rte_eth_dev *eth_dev)
static int
hns3_init_pf(struct rte_eth_dev *eth_dev)
{
- struct rte_device *dev = eth_dev->device;
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_adapter *hns = eth_dev->data->dev_private;
struct hns3_hw *hw = &hns->hw;
int ret;
@@ -4656,8 +4655,7 @@ static void
hns3_uninit_pf(struct rte_eth_dev *eth_dev)
{
struct hns3_adapter *hns = eth_dev->data->dev_private;
- struct rte_device *dev = eth_dev->device;
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_hw *hw = &hns->hw;
PMD_INIT_FUNC_TRACE();
diff --git a/drivers/net/hns3/hns3_ethdev_vf.c b/drivers/net/hns3/hns3_ethdev_vf.c
index 59fb790240..84e733a0f5 100644
--- a/drivers/net/hns3/hns3_ethdev_vf.c
+++ b/drivers/net/hns3/hns3_ethdev_vf.c
@@ -1622,7 +1622,7 @@ hns3vf_clear_vport_list(struct hns3_hw *hw)
static int
hns3vf_init_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_adapter *hns = eth_dev->data->dev_private;
struct hns3_hw *hw = &hns->hw;
int ret;
@@ -1739,7 +1739,7 @@ hns3vf_notify_uninit(struct hns3_hw *hw)
static void
hns3vf_uninit_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_adapter *hns = eth_dev->data->dev_private;
struct hns3_hw *hw = &hns->hw;
@@ -2377,7 +2377,7 @@ static int
hns3vf_reinit_dev(struct hns3_adapter *hns)
{
struct rte_eth_dev *eth_dev = &rte_eth_devices[hns->hw.data->port_id];
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct hns3_hw *hw = &hns->hw;
int ret;
diff --git a/drivers/net/hns3/hns3_rxtx.c b/drivers/net/hns3/hns3_rxtx.c
index 3528fda8a5..3b299c2f21 100644
--- a/drivers/net/hns3/hns3_rxtx.c
+++ b/drivers/net/hns3/hns3_rxtx.c
@@ -1093,7 +1093,7 @@ hns3_dev_all_rx_queue_intr_enable(struct hns3_hw *hw, bool en)
int
hns3_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct hns3_hw *hw = HNS3_DEV_PRIVATE_TO_HW(dev->data->dev_private);
@@ -3066,7 +3066,7 @@ hns3_tx_push_get_queue_tail_reg(struct rte_eth_dev *dev, uint16_t queue_id)
#define HNS3_TX_PUSH_QUICK_DOORBELL_OFFSET 64
#define HNS3_TX_PUSH_PCI_BAR_INDEX 4
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint8_t bar_id = HNS3_TX_PUSH_PCI_BAR_INDEX;
/*
diff --git a/drivers/net/intel/cpfl/cpfl_ethdev.c b/drivers/net/intel/cpfl/cpfl_ethdev.c
index 03599e6432..562b2dd3c9 100644
--- a/drivers/net/intel/cpfl/cpfl_ethdev.c
+++ b/drivers/net/intel/cpfl/cpfl_ethdev.c
@@ -2764,7 +2764,7 @@ cpfl_dev_vport_init(struct rte_eth_dev *dev, void *init_params)
uint8_t p2p_q_vc_out_info[IDPF_DFLT_MBX_BUF_SIZE] = {0};
struct cpfl_vport_id vi;
struct cpchnl2_vport_id v_id;
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
int ret = 0;
dev->dev_ops = &cpfl_eth_dev_ops;
@@ -2836,7 +2836,7 @@ cpfl_dev_vport_init(struct rte_eth_dev *dev, void *init_params)
}
/* get the vport info */
if (adapter->base.hw.device_id == IXD_DEV_ID_VCPF) {
- pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
vi.func_type = VCPF_CPCHNL2_FTYPE_LAN_VF;
vi.pf_id = CPFL_HOST0_CPF_ID;
vi.vf_id = pci_dev->addr.function;
diff --git a/drivers/net/intel/cpfl/cpfl_ethdev.h b/drivers/net/intel/cpfl/cpfl_ethdev.h
index 56f8f39829..4cc73f216b 100644
--- a/drivers/net/intel/cpfl/cpfl_ethdev.h
+++ b/drivers/net/intel/cpfl/cpfl_ethdev.h
@@ -298,7 +298,7 @@ int vcpf_add_queues(struct cpfl_adapter_ext *adapter);
int vcpf_del_queues(struct cpfl_adapter_ext *adapter);
#define CPFL_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
+ RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
#define CPFL_ADAPTER_TO_EXT(p) \
container_of((p), struct cpfl_adapter_ext, base)
#define CPFL_DEV_TO_VPORT(dev) \
diff --git a/drivers/net/intel/e1000/em_ethdev.c b/drivers/net/intel/e1000/em_ethdev.c
index 9e15e882b9..62ab57268f 100644
--- a/drivers/net/intel/e1000/em_ethdev.c
+++ b/drivers/net/intel/e1000/em_ethdev.c
@@ -273,7 +273,7 @@ eth_em_dev_is_ich8(struct e1000_hw *hw)
static int
eth_em_dev_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(eth_dev->data->dev_private);
@@ -563,7 +563,7 @@ eth_em_start(struct rte_eth_dev *dev)
E1000_DEV_PRIVATE(dev->data->dev_private);
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret, mask;
uint32_t intr_vector = 0;
@@ -762,7 +762,7 @@ eth_em_stop(struct rte_eth_dev *dev)
{
struct rte_eth_link link;
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/*
@@ -816,7 +816,7 @@ eth_em_close(struct rte_eth_dev *dev)
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
@@ -1062,7 +1062,7 @@ static int
eth_em_rx_queue_intr_enable(struct rte_eth_dev *dev, __rte_unused uint16_t queue_id)
{
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/* device interrupts are only subscribed to in primary processes */
@@ -1647,7 +1647,7 @@ static int
eth_em_interrupt_action(struct rte_eth_dev *dev,
struct rte_intr_handle *intr_handle)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_interrupt *intr =
diff --git a/drivers/net/intel/e1000/em_rxtx.c b/drivers/net/intel/e1000/em_rxtx.c
index 54971fe285..e0dcc6a82a 100644
--- a/drivers/net/intel/e1000/em_rxtx.c
+++ b/drivers/net/intel/e1000/em_rxtx.c
@@ -2093,7 +2093,7 @@ em_flush_desc_rings(struct rte_eth_dev *dev)
{
uint32_t fextnvm11, tdlen;
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t pci_cfg_status = 0;
int ret;
diff --git a/drivers/net/intel/e1000/igb_ethdev.c b/drivers/net/intel/e1000/igb_ethdev.c
index ef1599ac38..a4370fe32b 100644
--- a/drivers/net/intel/e1000/igb_ethdev.c
+++ b/drivers/net/intel/e1000/igb_ethdev.c
@@ -529,7 +529,7 @@ igb_intr_enable(struct rte_eth_dev *dev)
E1000_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (rte_intr_allow_others(intr_handle) &&
@@ -546,7 +546,7 @@ igb_intr_disable(struct rte_eth_dev *dev)
{
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (rte_intr_allow_others(intr_handle) &&
@@ -783,7 +783,7 @@ static int
eth_igb_dev_init(struct rte_eth_dev *eth_dev)
{
int error = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
struct e1000_vfta * shadow_vfta =
@@ -1004,7 +1004,7 @@ eth_igbvf_dev_init(struct rte_eth_dev *eth_dev)
return 0;
}
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
rte_eth_copy_pci_info(eth_dev, pci_dev);
hw->device_id = pci_dev->id.device_id;
@@ -1300,7 +1300,7 @@ eth_igb_start(struct rte_eth_dev *dev)
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret, mask;
uint32_t tqavctrl;
@@ -1537,7 +1537,7 @@ static int
eth_igb_stop(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_adapter *adapter =
@@ -1646,7 +1646,7 @@ eth_igb_close(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct rte_eth_link link;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_filter_info *filter_info =
E1000_DEV_PRIVATE_TO_FILTER_INFO(dev->data->dev_private);
@@ -2931,7 +2931,7 @@ static int eth_igb_rxq_interrupt_setup(struct rte_eth_dev *dev)
int ret;
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int misc_shift = rte_intr_allow_others(intr_handle) ? 1 : 0;
struct rte_eth_dev_info dev_info;
@@ -3002,7 +3002,7 @@ eth_igb_interrupt_action(struct rte_eth_dev *dev,
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_interrupt *intr =
E1000_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
int ret;
@@ -3496,7 +3496,7 @@ igbvf_dev_start(struct rte_eth_dev *dev)
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
uint32_t intr_vector = 0;
@@ -3560,7 +3560,7 @@ igbvf_dev_start(struct rte_eth_dev *dev)
static int
igbvf_dev_stop(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_adapter *adapter =
E1000_DEV_PRIVATE(dev->data->dev_private);
@@ -3608,7 +3608,7 @@ igbvf_dev_close(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct rte_ether_addr addr;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
int ret;
PMD_INIT_FUNC_TRACE();
@@ -5410,7 +5410,7 @@ eth_igb_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = E1000_MISC_VEC_ID;
@@ -5434,7 +5434,7 @@ eth_igb_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct e1000_hw *hw =
E1000_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = E1000_MISC_VEC_ID;
@@ -5516,7 +5516,7 @@ eth_igb_configure_msix_intr(struct rte_eth_dev *dev)
uint32_t vec = E1000_MISC_VEC_ID;
uint32_t base = E1000_MISC_VEC_ID;
uint32_t misc_shift = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/* won't configure msix register if no mapping is done
diff --git a/drivers/net/intel/e1000/igb_pf.c b/drivers/net/intel/e1000/igb_pf.c
index c7588ea57e..50df3daeb7 100644
--- a/drivers/net/intel/e1000/igb_pf.c
+++ b/drivers/net/intel/e1000/igb_pf.c
@@ -29,7 +29,7 @@
static inline uint16_t
dev_num_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
return pci_dev->max_vfs;
}
diff --git a/drivers/net/intel/e1000/igc_ethdev.c b/drivers/net/intel/e1000/igc_ethdev.c
index 727ea36c2b..de35da2c36 100644
--- a/drivers/net/intel/e1000/igc_ethdev.c
+++ b/drivers/net/intel/e1000/igc_ethdev.c
@@ -440,7 +440,7 @@ static void
igc_intr_other_disable(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (rte_intr_allow_others(intr_handle) &&
@@ -460,7 +460,7 @@ igc_intr_other_enable(struct rte_eth_dev *dev)
{
struct igc_interrupt *intr = IGC_DEV_PRIVATE_INTR(dev);
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (rte_intr_allow_others(intr_handle) &&
@@ -576,7 +576,7 @@ static void
eth_igc_interrupt_action(struct rte_eth_dev *dev)
{
struct igc_interrupt *intr = IGC_DEV_PRIVATE_INTR(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
int ret;
@@ -679,7 +679,7 @@ eth_igc_stop(struct rte_eth_dev *dev)
{
struct igc_adapter *adapter = IGC_DEV_PRIVATE(dev);
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rte_eth_link link;
@@ -799,7 +799,7 @@ static void
igc_configure_msix_intr(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_mask;
@@ -882,7 +882,7 @@ igc_rxq_interrupt_setup(struct rte_eth_dev *dev)
{
uint32_t mask;
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int misc_shift = rte_intr_allow_others(intr_handle) ? 1 : 0;
int nb_efd;
@@ -990,7 +990,7 @@ eth_igc_start(struct rte_eth_dev *dev)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
struct igc_adapter *adapter = IGC_DEV_PRIVATE(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t nsec, sec, baset_l, baset_h, tqavctrl;
struct timespec system_time;
@@ -1307,7 +1307,7 @@ igc_dev_free_queues(struct rte_eth_dev *dev)
static int
eth_igc_close(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
struct igc_adapter *adapter = IGC_DEV_PRIVATE(dev);
@@ -1359,7 +1359,7 @@ igc_identify_hardware(struct rte_eth_dev *dev, struct rte_pci_device *pci_dev)
static int
eth_igc_dev_init(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct igc_adapter *igc = IGC_DEV_PRIVATE(dev);
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
int i, error = 0;
@@ -2257,7 +2257,7 @@ static int
eth_igc_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = IGC_MISC_VEC_ID;
@@ -2280,7 +2280,7 @@ static int
eth_igc_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct e1000_hw *hw = IGC_DEV_PRIVATE_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = IGC_MISC_VEC_ID;
diff --git a/drivers/net/intel/fm10k/fm10k_ethdev.c b/drivers/net/intel/fm10k/fm10k_ethdev.c
index 97f61afec2..ca438d2d02 100644
--- a/drivers/net/intel/fm10k/fm10k_ethdev.c
+++ b/drivers/net/intel/fm10k/fm10k_ethdev.c
@@ -693,7 +693,7 @@ fm10k_dev_rx_init(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct fm10k_macvlan_filter_info *macvlan;
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
int i, ret;
struct fm10k_rx_queue *rxq;
@@ -1161,7 +1161,7 @@ static int
fm10k_dev_stop(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
int i;
@@ -1371,7 +1371,7 @@ fm10k_dev_infos_get(struct rte_eth_dev *dev,
struct rte_eth_dev_info *dev_info)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
PMD_INIT_FUNC_TRACE();
@@ -2364,7 +2364,7 @@ static int
fm10k_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
/* Enable ITR */
if (hw->mac.type == fm10k_mac_pf)
@@ -2381,7 +2381,7 @@ static int
fm10k_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
/* Disable ITR */
if (hw->mac.type == fm10k_mac_pf)
@@ -2397,7 +2397,7 @@ static int
fm10k_dev_rxq_interrupt_setup(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
uint32_t intr_vector, vec;
uint16_t queue_id;
@@ -2794,7 +2794,7 @@ static int
fm10k_dev_close(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
int ret;
@@ -3060,7 +3060,7 @@ static int
eth_fm10k_dev_init(struct rte_eth_dev *dev)
{
struct fm10k_hw *hw = FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(dev, *pdev);
struct rte_intr_handle *intr_handle = pdev->intr_handle;
int diag, i, ret;
struct fm10k_macvlan_filter_info *macvlan;
diff --git a/drivers/net/intel/i40e/i40e_ethdev.c b/drivers/net/intel/i40e/i40e_ethdev.c
index 100a751225..559df03a69 100644
--- a/drivers/net/intel/i40e/i40e_ethdev.c
+++ b/drivers/net/intel/i40e/i40e_ethdev.c
@@ -981,7 +981,7 @@ is_floating_veb_supported(struct rte_devargs *devargs)
static void
config_floating_veb(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
@@ -1549,7 +1549,7 @@ eth_i40e_dev_init(struct rte_eth_dev *dev, void *init_params __rte_unused)
return 0;
}
i40e_set_default_ptype_table(dev);
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
intr_handle = pci_dev->intr_handle;
rte_eth_copy_pci_info(dev, pci_dev);
@@ -2041,7 +2041,7 @@ void
i40e_vsi_queues_unbind_intr(struct i40e_vsi *vsi)
{
struct rte_eth_dev *dev = I40E_VSI_TO_ETH_DEV(vsi);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_VSI_TO_HW(vsi);
uint16_t msix_vect = vsi->msix_intr;
@@ -2157,7 +2157,7 @@ int
i40e_vsi_queues_bind_intr(struct i40e_vsi *vsi, uint16_t itr_idx)
{
struct rte_eth_dev *dev = I40E_VSI_TO_ETH_DEV(vsi);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_VSI_TO_HW(vsi);
uint16_t msix_vect = vsi->msix_intr;
@@ -2236,7 +2236,7 @@ void
i40e_vsi_enable_queues_intr(struct i40e_vsi *vsi)
{
struct rte_eth_dev *dev = I40E_VSI_TO_ETH_DEV(vsi);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_VSI_TO_HW(vsi);
struct i40e_pf *pf = I40E_VSI_TO_PF(vsi);
@@ -2263,7 +2263,7 @@ void
i40e_vsi_disable_queues_intr(struct i40e_vsi *vsi)
{
struct rte_eth_dev *dev = I40E_VSI_TO_ETH_DEV(vsi);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_VSI_TO_HW(vsi);
struct i40e_pf *pf = I40E_VSI_TO_PF(vsi);
@@ -2431,7 +2431,7 @@ i40e_dev_start(struct rte_eth_dev *dev)
struct i40e_adapter *ad =
I40E_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
int ret, i;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
struct i40e_vsi *vsi;
@@ -2612,7 +2612,7 @@ i40e_dev_stop(struct rte_eth_dev *dev)
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct i40e_vsi *main_vsi = pf->main_vsi;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int i;
@@ -2674,7 +2674,7 @@ i40e_dev_close(struct rte_eth_dev *dev)
{
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_filter_control_settings settings;
struct rte_flow *p_flow;
@@ -3831,7 +3831,7 @@ i40e_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct i40e_vsi *vsi = pf->main_vsi;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
dev_info->max_rx_queues = vsi->nb_qps;
dev_info->max_tx_queues = vsi->nb_qps;
@@ -4884,7 +4884,7 @@ i40e_pf_parameter_init(struct rte_eth_dev *dev)
{
struct i40e_pf *pf = I40E_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct i40e_hw *hw = I40E_PF_TO_HW(pf);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t qp_count = 0, vsi_count = 0;
if (pci_dev->max_vfs && !hw->func_caps.sr_iov_1_1) {
@@ -10033,7 +10033,7 @@ i40e_dev_flow_ops_get(struct rte_eth_dev *dev,
static void
i40e_enable_extended_tag(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint32_t buf = 0;
int ret;
@@ -11219,7 +11219,7 @@ i40e_dev_get_dcb_info(struct rte_eth_dev *dev,
static int
i40e_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint16_t msix_intr;
@@ -11247,7 +11247,7 @@ i40e_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
static int
i40e_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint16_t msix_intr;
diff --git a/drivers/net/intel/i40e/i40e_ethdev.h b/drivers/net/intel/i40e/i40e_ethdev.h
index dcbdf65047..c39a5a8802 100644
--- a/drivers/net/intel/i40e/i40e_ethdev.h
+++ b/drivers/net/intel/i40e/i40e_ethdev.h
@@ -1467,7 +1467,7 @@ int i40e_vf_representor_init(struct rte_eth_dev *ethdev, void *init_params);
int i40e_vf_representor_uninit(struct rte_eth_dev *ethdev);
#define I40E_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
+ RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
/* I40E_DEV_PRIVATE_TO */
#define I40E_DEV_PRIVATE_TO_PF(adapter) \
diff --git a/drivers/net/intel/iavf/iavf_ethdev.c b/drivers/net/intel/iavf/iavf_ethdev.c
index 1eca20bc9a..4ad9e594bb 100644
--- a/drivers/net/intel/iavf/iavf_ethdev.c
+++ b/drivers/net/intel/iavf/iavf_ethdev.c
@@ -1947,7 +1947,7 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct iavf_adapter *adapter =
IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(adapter);
uint16_t msix_intr;
@@ -1983,7 +1983,7 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
static int
iavf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint16_t msix_intr;
@@ -2767,7 +2767,7 @@ iavf_dev_init(struct rte_eth_dev *eth_dev)
IAVF_DEV_PRIVATE_TO_ADAPTER(eth_dev->data->dev_private);
struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(adapter);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret = 0;
PMD_INIT_FUNC_TRACE();
@@ -2926,7 +2926,7 @@ static int
iavf_dev_close(struct rte_eth_dev *dev)
{
struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct iavf_adapter *adapter =
IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
diff --git a/drivers/net/intel/ice/ice_dcf.c b/drivers/net/intel/ice/ice_dcf.c
index 51716a4d5b..0953fd6668 100644
--- a/drivers/net/intel/ice/ice_dcf.c
+++ b/drivers/net/intel/ice/ice_dcf.c
@@ -658,7 +658,7 @@ ice_dcf_send_aq_cmd(void *dcf_hw, struct ice_aq_desc *desc,
int
ice_dcf_handle_vsi_update_event(struct ice_dcf_hw *hw)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(hw->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(hw->eth_dev, *pci_dev);
int i = 0;
int err = -1;
@@ -738,7 +738,7 @@ dcf_get_vlan_offload_caps_v2(struct ice_dcf_hw *hw)
int
ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret, size;
hw->resetting = false;
@@ -873,7 +873,7 @@ ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
void
ice_dcf_uninit_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (hw->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_QOS)
diff --git a/drivers/net/intel/ice/ice_ethdev.c b/drivers/net/intel/ice/ice_ethdev.c
index 715d1522f9..f0a9a8e536 100644
--- a/drivers/net/intel/ice/ice_ethdev.c
+++ b/drivers/net/intel/ice/ice_ethdev.c
@@ -2623,7 +2623,7 @@ ice_dev_init(struct rte_eth_dev *dev)
}
ice_set_default_ptype_table(dev);
- pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
intr_handle = pci_dev->intr_handle;
pf->adapter = ICE_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
@@ -2956,7 +2956,7 @@ ice_dev_close(struct rte_eth_dev *dev)
{
struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct ice_hw *hw = ICE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ice_adapter *ad =
ICE_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
@@ -4520,7 +4520,7 @@ ice_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private);
struct ice_hw *hw = ICE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ice_vsi *vsi = pf->main_vsi;
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
bool is_safe_mode = pf->adapter->is_safe_mode;
u64 phy_type_low;
u64 phy_type_high;
diff --git a/drivers/net/intel/ice/ice_ethdev.h b/drivers/net/intel/ice/ice_ethdev.h
index ea73f8bcb3..92baa62cc4 100644
--- a/drivers/net/intel/ice/ice_ethdev.h
+++ b/drivers/net/intel/ice/ice_ethdev.h
@@ -708,7 +708,7 @@ struct ice_vsi_vlan_pvid_info {
};
#define ICE_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
+ RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
/* ICE_DEV_PRIVATE_TO */
#define ICE_DEV_PRIVATE_TO_PF(adapter) \
diff --git a/drivers/net/intel/idpf/idpf_ethdev.h b/drivers/net/intel/idpf/idpf_ethdev.h
index 5105eea1c5..99496c59da 100644
--- a/drivers/net/intel/idpf/idpf_ethdev.h
+++ b/drivers/net/intel/idpf/idpf_ethdev.h
@@ -85,7 +85,7 @@ struct idpf_adapter_ext {
TAILQ_HEAD(idpf_adapter_list, idpf_adapter_ext);
#define IDPF_DEV_TO_PCI(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_pci_device)
+ RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
#define IDPF_ADAPTER_TO_EXT(p) \
container_of((p), struct idpf_adapter_ext, base)
diff --git a/drivers/net/intel/ipn3ke/ipn3ke_ethdev.h b/drivers/net/intel/ipn3ke/ipn3ke_ethdev.h
index 6d531120b8..505b8f367a 100644
--- a/drivers/net/intel/ipn3ke/ipn3ke_ethdev.h
+++ b/drivers/net/intel/ipn3ke/ipn3ke_ethdev.h
@@ -310,9 +310,6 @@ struct ipn3ke_hw {
uint8_t *hw_addr;
};
-#define RTE_ETH_DEV_TO_AFU(eth_dev) \
- RTE_BUS_DEVICE((eth_dev)->device, struct rte_afu_device)
-
/**
* PCIe MMIO Access
*/
diff --git a/drivers/net/intel/ipn3ke/ipn3ke_representor.c b/drivers/net/intel/ipn3ke/ipn3ke_representor.c
index cd34d08055..af1af31f1d 100644
--- a/drivers/net/intel/ipn3ke/ipn3ke_representor.c
+++ b/drivers/net/intel/ipn3ke/ipn3ke_representor.c
@@ -2070,7 +2070,7 @@ ipn3ke_rpst_stats_reset(struct rte_eth_dev *ethdev)
return -EINVAL;
}
- afu_dev = RTE_ETH_DEV_TO_AFU(ethdev);
+ afu_dev = RTE_CLASS_TO_BUS_DEVICE(ethdev, *afu_dev);
if (!afu_dev) {
IPN3KE_AFU_PMD_ERR("afu device to reset is NULL!");
return -EINVAL;
@@ -2138,7 +2138,7 @@ ipn3ke_rpst_stats_get
return -EINVAL;
}
- afu_dev = RTE_ETH_DEV_TO_AFU(ethdev);
+ afu_dev = RTE_CLASS_TO_BUS_DEVICE(ethdev, *afu_dev);
if (!afu_dev) {
IPN3KE_AFU_PMD_ERR("afu device to get statistics is NULL!");
return -EINVAL;
@@ -2228,7 +2228,7 @@ ipn3ke_rpst_xstats_get
return -EINVAL;
}
- afu_dev = RTE_ETH_DEV_TO_AFU(ethdev);
+ afu_dev = RTE_CLASS_TO_BUS_DEVICE(ethdev, *afu_dev);
if (!afu_dev) {
IPN3KE_AFU_PMD_ERR("afu device to get statistics is NULL!");
return -EINVAL;
diff --git a/drivers/net/intel/ixgbe/ixgbe_ethdev.c b/drivers/net/intel/ixgbe/ixgbe_ethdev.c
index 57d929cf2c..39db5368b7 100644
--- a/drivers/net/intel/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/intel/ixgbe/ixgbe_ethdev.c
@@ -1085,7 +1085,7 @@ static int
eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
{
struct ixgbe_adapter *ad = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
@@ -1598,7 +1598,7 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
int diag;
uint32_t tc, tcs;
struct ixgbe_adapter *ad = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
@@ -2264,7 +2264,7 @@ ixgbe_vmdq_vlan_hw_filter_enable(struct rte_eth_dev *dev)
static int
ixgbe_check_vf_rss_rxq_num(struct rte_eth_dev *dev, uint16_t nb_rx_q)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
switch (nb_rx_q) {
case 1:
@@ -2506,7 +2506,7 @@ ixgbe_set_vf_rate_limit(struct rte_eth_dev *dev, uint16_t vf,
struct rte_pci_device *pci_dev;
int ret;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
ret = rte_eth_link_get_nowait(dev->data->port_id, &link);
if (ret < 0)
return ret;
@@ -2614,7 +2614,7 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ixgbe_vf_info *vfinfo =
*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
int err;
@@ -2921,7 +2921,7 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ixgbe_vf_info *vfinfo =
*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int vf;
struct ixgbe_tm_conf *tm_conf =
@@ -3082,7 +3082,7 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
{
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int retries = 0;
int ret;
@@ -3970,7 +3970,7 @@ ixgbe_fw_version_get(struct rte_eth_dev *dev, char *fw_version, size_t fw_size)
static int
ixgbe_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct rte_eth_conf *dev_conf = &dev->data->dev_conf;
@@ -4100,7 +4100,7 @@ static int
ixgbevf_dev_info_get(struct rte_eth_dev *dev,
struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -4668,7 +4668,7 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
static void
ixgbe_dev_link_status_print(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
rte_eth_linkstatus_get(dev, &link);
@@ -4780,7 +4780,7 @@ static void
ixgbe_dev_interrupt_delayed_handler(void *param)
{
struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_interrupt *intr =
IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
@@ -5344,7 +5344,7 @@ ixgbe_remove_rar(struct rte_eth_dev *dev, uint32_t index)
static int
ixgbe_set_default_mac_addr(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
/*
* This function calls into the base driver, which in turn will use
@@ -5508,7 +5508,7 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
uint32_t intr_vector = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int err, mask = 0;
@@ -5621,7 +5621,7 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ixgbe_adapter *adapter = dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/*
@@ -5669,7 +5669,7 @@ static int
ixgbevf_dev_close(struct rte_eth_dev *dev)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
@@ -5975,7 +5975,7 @@ ixgbe_convert_vm_rx_mask_to_val(uint16_t rx_mask, uint32_t orig_val)
static int
ixgbevf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_interrupt *intr =
IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
@@ -6005,7 +6005,7 @@ ixgbevf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = IXGBE_MISC_VEC_ID;
@@ -6025,7 +6025,7 @@ ixgbevf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
static int
ixgbe_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t mask;
struct ixgbe_hw *hw =
@@ -6162,7 +6162,7 @@ ixgbe_set_ivar_map(struct ixgbe_hw *hw, int8_t direction,
static void
ixgbevf_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
@@ -6213,7 +6213,7 @@ ixgbevf_configure_msix(struct rte_eth_dev *dev)
static void
ixgbe_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
diff --git a/drivers/net/intel/ixgbe/ixgbe_flow.c b/drivers/net/intel/ixgbe/ixgbe_flow.c
index 01cd4f9bde..659007980a 100644
--- a/drivers/net/intel/ixgbe/ixgbe_flow.c
+++ b/drivers/net/intel/ixgbe/ixgbe_flow.c
@@ -1272,7 +1272,7 @@ cons_parse_l2_tn_filter(struct rte_eth_dev *dev,
const struct rte_flow_item_e_tag *e_tag_mask;
const struct rte_flow_action *act;
const struct rte_flow_action_vf *act_vf;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!pattern) {
rte_flow_error_set(error, EINVAL,
@@ -1430,7 +1430,7 @@ ixgbe_parse_l2_tn_filter(struct rte_eth_dev *dev,
{
int ret = 0;
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t vf_num;
ret = cons_parse_l2_tn_filter(dev, attr, pattern,
diff --git a/drivers/net/intel/ixgbe/ixgbe_pf.c b/drivers/net/intel/ixgbe/ixgbe_pf.c
index d9a775f99a..2c014a0b96 100644
--- a/drivers/net/intel/ixgbe/ixgbe_pf.c
+++ b/drivers/net/intel/ixgbe/ixgbe_pf.c
@@ -32,7 +32,7 @@
static inline uint16_t
dev_num_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
return pci_dev->max_vfs;
}
diff --git a/drivers/net/intel/ixgbe/ixgbe_tm.c b/drivers/net/intel/ixgbe/ixgbe_tm.c
index 27a821285d..50c7fa228a 100644
--- a/drivers/net/intel/ixgbe/ixgbe_tm.c
+++ b/drivers/net/intel/ixgbe/ixgbe_tm.c
@@ -365,7 +365,7 @@ ixgbe_queue_base_nb_get(struct rte_eth_dev *dev, uint16_t tc_node_no,
uint16_t *base, uint16_t *nb)
{
uint8_t nb_tcs = ixgbe_tc_nb_get(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t vf_num = pci_dev->max_vfs;
*base = 0;
diff --git a/drivers/net/intel/ixgbe/ixgbe_vf_representor.c b/drivers/net/intel/ixgbe/ixgbe_vf_representor.c
index 901d80e406..bad699dd70 100644
--- a/drivers/net/intel/ixgbe/ixgbe_vf_representor.c
+++ b/drivers/net/intel/ixgbe/ixgbe_vf_representor.c
@@ -190,7 +190,7 @@ ixgbe_vf_representor_init(struct rte_eth_dev *ethdev, void *init_params)
representor->pf_ethdev =
((struct ixgbe_vf_representor *)init_params)->pf_ethdev;
- pci_dev = RTE_ETH_DEV_TO_PCI(representor->pf_ethdev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(representor->pf_ethdev, *pci_dev);
if (representor->vf_id >= pci_dev->max_vfs)
return -ENODEV;
diff --git a/drivers/net/intel/ixgbe/rte_pmd_ixgbe.c b/drivers/net/intel/ixgbe/rte_pmd_ixgbe.c
index c2300a8955..35d364fb19 100644
--- a/drivers/net/intel/ixgbe/rte_pmd_ixgbe.c
+++ b/drivers/net/intel/ixgbe/rte_pmd_ixgbe.c
@@ -25,7 +25,7 @@ rte_pmd_ixgbe_set_vf_mac_addr(uint16_t port, uint16_t vf,
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -60,7 +60,7 @@ rte_pmd_ixgbe_ping_vf(uint16_t port, uint16_t vf)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -92,7 +92,7 @@ rte_pmd_ixgbe_set_vf_vlan_anti_spoof(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -123,7 +123,7 @@ rte_pmd_ixgbe_set_vf_mac_anti_spoof(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -153,7 +153,7 @@ rte_pmd_ixgbe_set_vf_vlan_insert(uint16_t port, uint16_t vf, uint16_t vlan_id)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -252,7 +252,7 @@ rte_pmd_ixgbe_set_vf_split_drop_en(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -289,7 +289,7 @@ rte_pmd_ixgbe_set_vf_vlan_stripq(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
if (!is_ixgbe_supported(dev))
@@ -338,7 +338,7 @@ rte_pmd_ixgbe_set_vf_rxmode(uint16_t port, uint16_t vf,
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -386,7 +386,7 @@ rte_pmd_ixgbe_set_vf_rx(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
@@ -438,7 +438,7 @@ rte_pmd_ixgbe_set_vf_tx(uint16_t port, uint16_t vf, uint8_t on)
RTE_ETH_VALID_PORTID_OR_ERR_RET(port, -ENODEV);
dev = &rte_eth_devices[port];
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!is_ixgbe_supported(dev))
return -ENOTSUP;
diff --git a/drivers/net/nbl/nbl_core.c b/drivers/net/nbl/nbl_core.c
index df8c0c76ed..6a823e9bfb 100644
--- a/drivers/net/nbl/nbl_core.c
+++ b/drivers/net/nbl/nbl_core.c
@@ -32,7 +32,7 @@ static void nbl_init_func_caps(const struct rte_pci_device *pci_dev, struct nbl_
int nbl_core_init(struct nbl_adapter *adapter, struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
const struct nbl_product_core_ops *product_base_ops = NULL;
struct nbl_common_info *common = NBL_ADAPTER_TO_COMMON(adapter);
int ret = 0;
diff --git a/drivers/net/nbl/nbl_dev/nbl_dev.c b/drivers/net/nbl/nbl_dev/nbl_dev.c
index 2b0413fb7c..35485ea691 100644
--- a/drivers/net/nbl/nbl_dev/nbl_dev.c
+++ b/drivers/net/nbl/nbl_dev/nbl_dev.c
@@ -868,7 +868,7 @@ static int nbl_dev_common_start(struct nbl_dev_mgt *dev_mgt)
struct nbl_dev_net_mgt *net_dev = NBL_DEV_MGT_TO_NET_DEV(dev_mgt);
struct nbl_common_info *common = NBL_DEV_MGT_TO_COMMON(dev_mgt);
struct nbl_board_port_info *board_info;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(net_dev->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(net_dev->eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
u8 *mac;
int ret;
@@ -991,7 +991,7 @@ static void nbl_dev_leonis_stop(void *p)
const struct nbl_common_info *common = NBL_DEV_MGT_TO_COMMON(dev_mgt);
const struct nbl_dispatch_ops *disp_ops = NBL_DEV_MGT_TO_DISP_OPS(dev_mgt);
const struct nbl_channel_ops *chan_ops = NBL_DEV_MGT_TO_CHAN_OPS(dev_mgt);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(net_dev->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(net_dev->eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
u8 *mac;
@@ -1105,7 +1105,7 @@ static int nbl_dev_setup_net_dev(struct nbl_dev_mgt *dev_mgt,
struct nbl_register_net_param register_param = { 0 };
struct nbl_register_net_result register_result = { 0 };
struct nbl_dev_ring_mgt *ring_mgt;
- const struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ const struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret = 0;
net_dev = rte_zmalloc("nbl_dev_net", sizeof(struct nbl_dev_net_mgt), 0);
diff --git a/drivers/net/netvsc/hn_ethdev.c b/drivers/net/netvsc/hn_ethdev.c
index fa936cfde7..cb7035e15f 100644
--- a/drivers/net/netvsc/hn_ethdev.c
+++ b/drivers/net/netvsc/hn_ethdev.c
@@ -1525,7 +1525,6 @@ static int
eth_hn_dev_init(struct rte_eth_dev *eth_dev)
{
struct hn_data *hv = eth_dev->data->dev_private;
- struct rte_device *device = eth_dev->device;
struct rte_vmbus_device *vmbus;
uint32_t mtu;
unsigned int rxr_cnt;
@@ -1536,7 +1535,7 @@ eth_hn_dev_init(struct rte_eth_dev *eth_dev)
rte_spinlock_init(&hv->hotadd_lock);
LIST_INIT(&hv->hotadd_list);
- vmbus = RTE_BUS_DEVICE(device, *vmbus);
+ vmbus = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *vmbus);
eth_dev->dev_ops = &hn_eth_dev_ops;
eth_dev->rx_queue_count = hn_dev_rx_queue_count;
eth_dev->rx_descriptor_status = hn_dev_rx_queue_status;
diff --git a/drivers/net/nfp/nfp_ethdev.c b/drivers/net/nfp/nfp_ethdev.c
index cbd1deffb4..d2da18013c 100644
--- a/drivers/net/nfp/nfp_ethdev.c
+++ b/drivers/net/nfp/nfp_ethdev.c
@@ -363,7 +363,7 @@ nfp_net_start(struct rte_eth_dev *dev)
struct rte_eth_txmode *txmode;
struct nfp_net_hw_priv *hw_priv;
struct nfp_app_fw_nic *app_fw_nic;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
net_hw = dev->data->dev_private;
@@ -770,7 +770,7 @@ nfp_net_close(struct rte_eth_dev *dev)
hw = dev->data->dev_private;
pf_dev = hw_priv->pf_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
app_fw_nic = NFP_PRIV_TO_APP_FW_NIC(pf_dev->app_fw_priv);
/*
@@ -1022,7 +1022,7 @@ nfp_net_init(struct rte_eth_dev *eth_dev,
struct nfp_net_hw_priv *hw_priv;
struct nfp_app_fw_nic *app_fw_nic;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
net_hw = eth_dev->data->dev_private;
hw_init = para;
@@ -2879,7 +2879,7 @@ nfp_pci_uninit(struct rte_eth_dev *eth_dev)
uint16_t port_id;
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* Free up all physical ports under PF */
RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device)
diff --git a/drivers/net/nfp/nfp_ethdev_vf.c b/drivers/net/nfp/nfp_ethdev_vf.c
index 23fa5b82ad..a86cc36592 100644
--- a/drivers/net/nfp/nfp_ethdev_vf.c
+++ b/drivers/net/nfp/nfp_ethdev_vf.c
@@ -30,7 +30,7 @@ nfp_netvf_start(struct rte_eth_dev *dev)
struct nfp_net_hw *net_hw;
struct rte_eth_conf *dev_conf;
struct rte_eth_rxmode *rxmode;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
/* Disabling queues just in case... */
@@ -169,7 +169,7 @@ nfp_netvf_close(struct rte_eth_dev *dev)
return 0;
net_hw = dev->data->dev_private;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
hw_priv = dev->process_private;
rte_free(net_hw->eth_xstats_base);
@@ -266,7 +266,7 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev)
const struct nfp_dev_info *dev_info;
port = eth_dev->data->port_id;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
dev_info = nfp_dev_info_get(pci_dev->id.device_id);
if (dev_info == NULL) {
diff --git a/drivers/net/nfp/nfp_net_common.c b/drivers/net/nfp/nfp_net_common.c
index d35eee970a..2d36311cfe 100644
--- a/drivers/net/nfp/nfp_net_common.c
+++ b/drivers/net/nfp/nfp_net_common.c
@@ -1568,7 +1568,7 @@ nfp_rx_queue_intr_enable(struct rte_eth_dev *dev,
struct nfp_net_hw *hw;
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (rte_intr_type_get(pci_dev->intr_handle) != RTE_INTR_HANDLE_UIO)
base = 1;
@@ -1589,7 +1589,7 @@ nfp_rx_queue_intr_disable(struct rte_eth_dev *dev,
struct nfp_net_hw *hw;
struct rte_pci_device *pci_dev;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (rte_intr_type_get(pci_dev->intr_handle) != RTE_INTR_HANDLE_UIO)
base = 1;
@@ -1606,7 +1606,7 @@ static void
nfp_net_dev_link_status_print(struct rte_eth_dev *dev)
{
struct rte_eth_link link;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
rte_eth_linkstatus_get(dev, &link);
if (link.link_status != 0)
@@ -1635,7 +1635,7 @@ nfp_net_irq_unmask(struct rte_eth_dev *dev)
struct rte_pci_device *pci_dev;
hw = nfp_net_get_hw(dev);
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
/* Make sure all updates are written before un-masking */
rte_wmb();
diff --git a/drivers/net/ngbe/ngbe_ethdev.c b/drivers/net/ngbe/ngbe_ethdev.c
index 8b9d6371fb..f7b4a8b159 100644
--- a/drivers/net/ngbe/ngbe_ethdev.c
+++ b/drivers/net/ngbe/ngbe_ethdev.c
@@ -321,7 +321,7 @@ ngbe_swfw_lock_reset(struct ngbe_hw *hw)
static int
eth_ngbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct ngbe_hw *hw = ngbe_dev_hw(eth_dev);
struct ngbe_vfta *shadow_vfta = NGBE_DEV_VFTA(eth_dev);
struct ngbe_hwstrip *hwstrip = NGBE_DEV_HWSTRIP(eth_dev);
@@ -958,7 +958,7 @@ ngbe_dev_start(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
struct ngbe_hw_stats *hw_stats = NGBE_DEV_STATS(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
int err;
@@ -1160,7 +1160,7 @@ ngbe_dev_stop(struct rte_eth_dev *dev)
struct ngbe_adapter *adapter = ngbe_dev_adapter(dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
struct ngbe_vf_info *vfinfo = *NGBE_DEV_VFDATA(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int vf;
@@ -1256,7 +1256,7 @@ static int
ngbe_dev_close(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int retries = 0;
int ret;
@@ -1843,7 +1843,7 @@ ngbe_fw_version_get(struct rte_eth_dev *dev, char *fw_version, size_t fw_size)
static int
ngbe_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -2258,7 +2258,7 @@ ngbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
static void
ngbe_dev_link_status_print(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
rte_eth_linkstatus_get(dev, &link);
@@ -2472,7 +2472,7 @@ static s32
ngbe_fc_hpbthresh_set(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
u32 max_frame_size, tc, dv_id, rx_pb;
s32 kb, marker;
@@ -2653,7 +2653,7 @@ ngbe_remove_rar(struct rte_eth_dev *dev, uint32_t index)
static int
ngbe_set_default_mac_addr(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
ngbe_remove_rar(dev, 0);
ngbe_add_rar(dev, addr, 0, pci_dev->max_vfs);
@@ -2797,7 +2797,7 @@ ngbe_uc_all_hash_table_set(struct rte_eth_dev *dev, uint8_t on)
static int
ngbe_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_hw *hw = ngbe_dev_hw(dev);
uint32_t mask;
@@ -2867,7 +2867,7 @@ ngbe_set_ivar_map(struct ngbe_hw *hw, int8_t direction,
static void
ngbe_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_hw *hw = ngbe_dev_hw(dev);
uint32_t queue_id, base = NGBE_MISC_VEC_ID;
diff --git a/drivers/net/ngbe/ngbe_ethdev_vf.c b/drivers/net/ngbe/ngbe_ethdev_vf.c
index 6406df40d0..ea3a988df6 100644
--- a/drivers/net/ngbe/ngbe_ethdev_vf.c
+++ b/drivers/net/ngbe/ngbe_ethdev_vf.c
@@ -152,7 +152,7 @@ eth_ngbevf_dev_init(struct rte_eth_dev *eth_dev)
{
int err;
uint32_t tc, tcs;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_hw *hw = ngbe_dev_hw(eth_dev);
struct ngbe_vfta *shadow_vfta = NGBE_DEV_VFTA(eth_dev);
@@ -465,7 +465,7 @@ static int
ngbevf_dev_info_get(struct rte_eth_dev *dev,
struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -588,7 +588,7 @@ ngbevf_dev_start(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
uint32_t intr_vector = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int err, mask = 0;
@@ -688,7 +688,7 @@ ngbevf_dev_stop(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
struct ngbe_adapter *adapter = ngbe_dev_adapter(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (hw->adapter_stopped)
@@ -725,7 +725,7 @@ static int
ngbevf_dev_close(struct rte_eth_dev *dev)
{
struct ngbe_hw *hw = ngbe_dev_hw(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
@@ -898,7 +898,7 @@ ngbevf_vlan_offload_set(struct rte_eth_dev *dev, int mask)
static int
ngbevf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_interrupt *intr = ngbe_dev_intr(dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
@@ -920,7 +920,7 @@ ngbevf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct ngbe_interrupt *intr = ngbe_dev_intr(dev);
struct ngbe_hw *hw = ngbe_dev_hw(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = NGBE_MISC_VEC_ID;
@@ -960,7 +960,7 @@ ngbevf_set_ivar_map(struct ngbe_hw *hw, int8_t direction,
static void
ngbevf_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ngbe_hw *hw = ngbe_dev_hw(dev);
uint32_t q_idx;
diff --git a/drivers/net/ngbe/ngbe_pf.c b/drivers/net/ngbe/ngbe_pf.c
index bb62e2fbb7..db2384e28c 100644
--- a/drivers/net/ngbe/ngbe_pf.c
+++ b/drivers/net/ngbe/ngbe_pf.c
@@ -18,7 +18,7 @@
static inline uint16_t
dev_num_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* EM only support 7 VFs. */
return pci_dev->max_vfs;
diff --git a/drivers/net/octeon_ep/otx_ep_ethdev.c b/drivers/net/octeon_ep/otx_ep_ethdev.c
index 99be30523a..876d2f9d7d 100644
--- a/drivers/net/octeon_ep/otx_ep_ethdev.c
+++ b/drivers/net/octeon_ep/otx_ep_ethdev.c
@@ -777,7 +777,7 @@ static int otx_ep_eth_dev_query_set_vf_mac(struct rte_eth_dev *eth_dev,
static int
otx_ep_eth_dev_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
struct otx_ep_device *otx_epvf = OTX_EP_DEV(eth_dev);
struct rte_ether_addr vf_mac_addr;
int ret = 0;
diff --git a/drivers/net/octeon_ep/otx_ep_mbox.c b/drivers/net/octeon_ep/otx_ep_mbox.c
index 3e94c66677..5e6be29a96 100644
--- a/drivers/net/octeon_ep/otx_ep_mbox.c
+++ b/drivers/net/octeon_ep/otx_ep_mbox.c
@@ -346,7 +346,7 @@ otx_ep_mbox_intr_handler(void *param)
{
struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)param;
struct otx_ep_device *otx_ep = (struct otx_ep_device *)eth_dev->data->dev_private;
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
union otx_ep_mbox_word mbox_cmd;
if (otx2_read64(otx_ep->hw_addr + CNXK_EP_R_MBOX_PF_VF_INT(0)) & CNXK_EP_MBOX_INTR) {
@@ -369,7 +369,7 @@ int
otx_ep_mbox_init(struct rte_eth_dev *eth_dev)
{
struct otx_ep_device *otx_ep = (struct otx_ep_device *)eth_dev->data->dev_private;
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
uint64_t reg_val;
int rc;
@@ -402,7 +402,7 @@ void
otx_ep_mbox_uninit(struct rte_eth_dev *eth_dev)
{
struct otx_ep_device *otx_ep = (struct otx_ep_device *)eth_dev->data->dev_private;
- struct rte_pci_device *pdev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pdev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pdev);
otx2_write64(0, otx_ep->hw_addr + CNXK_EP_R_MBOX_PF_VF_INT(0));
diff --git a/drivers/net/qede/qede_ethdev.c b/drivers/net/qede/qede_ethdev.c
index c676c6fa75..4efc2dd349 100644
--- a/drivers/net/qede/qede_ethdev.c
+++ b/drivers/net/qede/qede_ethdev.c
@@ -1231,7 +1231,7 @@ static int qede_args_check(const char *key, const char *val, void *opaque)
static int qede_args(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_kvargs *kvlist;
struct rte_devargs *devargs;
int ret;
@@ -1540,7 +1540,7 @@ static void qede_poll_sp_sb_cb(void *param)
static int qede_dev_close(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct qede_dev *qdev = QEDE_INIT_QDEV(eth_dev);
struct ecore_dev *edev = QEDE_INIT_EDEV(qdev);
int ret = 0;
@@ -2529,7 +2529,7 @@ static int qede_common_dev_init(struct rte_eth_dev *eth_dev, bool is_vf)
adapter = eth_dev->data->dev_private;
adapter->ethdev = eth_dev;
edev = &adapter->edev;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
pci_addr = pci_dev->addr;
PMD_INIT_FUNC_TRACE(edev);
diff --git a/drivers/net/r8169/r8169_ethdev.c b/drivers/net/r8169/r8169_ethdev.c
index b2b1882aa5..2ac0189b61 100644
--- a/drivers/net/r8169/r8169_ethdev.c
+++ b/drivers/net/r8169/r8169_ethdev.c
@@ -301,7 +301,7 @@ rtl_dev_start(struct rte_eth_dev *dev)
{
struct rtl_adapter *adapter = RTL_DEV_PRIVATE(dev);
struct rtl_hw *hw = &adapter->hw;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int err;
@@ -684,7 +684,7 @@ rtl_dev_interrupt_handler(void *param)
static int
rtl_dev_close(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rtl_adapter *adapter = RTL_DEV_PRIVATE(dev);
struct rtl_hw *hw = &adapter->hw;
@@ -908,7 +908,7 @@ rtl_rss_hash_conf_get(struct rte_eth_dev *dev, struct rte_eth_rss_conf *rss_conf
static int
rtl_dev_init(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rtl_adapter *adapter = RTL_DEV_PRIVATE(dev);
struct rtl_hw *hw = &adapter->hw;
diff --git a/drivers/net/rnp/rnp_ethdev.c b/drivers/net/rnp/rnp_ethdev.c
index 3420842823..e48ad0e317 100644
--- a/drivers/net/rnp/rnp_ethdev.c
+++ b/drivers/net/rnp/rnp_ethdev.c
@@ -728,7 +728,7 @@ static int rnp_dev_close(struct rte_eth_dev *eth_dev)
if (adapter->intr_registered && adapter->eth_dev == eth_dev)
rnp_change_manage_port(adapter);
if (adapter->closed_ports == adapter->inited_ports) {
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
if (adapter->intr_registered) {
/* disable uio irq before callback unregister */
rte_intr_disable(pci_dev->intr_handle);
@@ -1667,7 +1667,7 @@ rnp_rx_reset_pool_setup(struct rnp_eth_adapter *adapter)
static int
rnp_eth_dev_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct rnp_eth_port *port = RNP_DEV_TO_PORT(eth_dev);
char name[RTE_ETH_NAME_MAX_LEN] = " ";
@@ -1798,7 +1798,7 @@ rnp_eth_dev_init(struct rte_eth_dev *eth_dev)
static int
rnp_eth_dev_uninit(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(eth_dev->device, *pci_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
uint16_t port_id;
int err = 0;
diff --git a/drivers/net/sfc/sfc.c b/drivers/net/sfc/sfc.c
index 69747e49ae..39cd8d519a 100644
--- a/drivers/net/sfc/sfc.c
+++ b/drivers/net/sfc/sfc.c
@@ -781,7 +781,7 @@ static int
sfc_mem_bar_init(struct sfc_adapter *sa, const efx_bar_region_t *mem_ebrp)
{
struct rte_eth_dev *eth_dev = sa->eth_dev;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
efsys_bar_t *ebp = &sa->mem_bar;
struct rte_mem_resource *res =
&pci_dev->mem_resource[mem_ebrp->ebr_index];
@@ -1283,7 +1283,7 @@ sfc_probe(struct sfc_adapter *sa)
{
efx_bar_region_t mem_ebrp;
struct rte_eth_dev *eth_dev = sa->eth_dev;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
efx_nic_t *enp;
int rc;
diff --git a/drivers/net/sfc/sfc_ethdev.c b/drivers/net/sfc/sfc_ethdev.c
index 6be98c49d0..6be91789cf 100644
--- a/drivers/net/sfc/sfc_ethdev.c
+++ b/drivers/net/sfc/sfc_ethdev.c
@@ -3309,7 +3309,7 @@ static int
sfc_eth_dev_init(struct rte_eth_dev *dev, void *init_params)
{
struct sfc_adapter_shared *sas = sfc_adapter_shared_by_eth_dev(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct sfc_ethdev_init_data *init_data = init_params;
uint32_t logtype_main;
struct sfc_adapter *sa;
diff --git a/drivers/net/sfc/sfc_intr.c b/drivers/net/sfc/sfc_intr.c
index ddddefad7b..6a09da9f67 100644
--- a/drivers/net/sfc/sfc_intr.c
+++ b/drivers/net/sfc/sfc_intr.c
@@ -56,7 +56,7 @@ sfc_intr_line_handler(void *cb_arg)
boolean_t fatal;
uint32_t qmask;
unsigned int lsc_seq = sa->port.lsc_seq;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
sfc_log_init(sa, "entry");
@@ -102,7 +102,7 @@ sfc_intr_message_handler(void *cb_arg)
efx_nic_t *enp = sa->nic;
boolean_t fatal;
unsigned int lsc_seq = sa->port.lsc_seq;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
sfc_log_init(sa, "entry");
@@ -158,7 +158,7 @@ sfc_intr_start(struct sfc_adapter *sa)
if (rc != 0)
goto fail_intr_init;
- pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
intr_handle = pci_dev->intr_handle;
if (intr->handler != NULL) {
@@ -240,7 +240,7 @@ void
sfc_intr_stop(struct sfc_adapter *sa)
{
struct sfc_intr *intr = &sa->intr;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
sfc_log_init(sa, "entry");
@@ -318,7 +318,7 @@ int
sfc_intr_attach(struct sfc_adapter *sa)
{
struct sfc_intr *intr = &sa->intr;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
sfc_log_init(sa, "entry");
diff --git a/drivers/net/sfc/sfc_rx.c b/drivers/net/sfc/sfc_rx.c
index a193229265..305c680944 100644
--- a/drivers/net/sfc/sfc_rx.c
+++ b/drivers/net/sfc/sfc_rx.c
@@ -1277,8 +1277,9 @@ sfc_rx_qinit(struct sfc_adapter *sa, sfc_sw_index_t sw_index,
info.nic_dma_info = &sas->nic_dma_info;
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
rc = sa->priv.dp_rx->qcreate(sa->eth_dev->data->port_id, sw_index,
- &RTE_ETH_DEV_TO_PCI(sa->eth_dev)->addr,
+ &pci_dev->addr,
socket_id, &info, &rxq_info->dp);
if (rc != 0)
goto fail_dp_rx_qcreate;
diff --git a/drivers/net/sfc/sfc_sriov.c b/drivers/net/sfc/sfc_sriov.c
index 009b884d8d..f41d1b1719 100644
--- a/drivers/net/sfc/sfc_sriov.c
+++ b/drivers/net/sfc/sfc_sriov.c
@@ -44,7 +44,7 @@ sriov_mac_addr_assigned(const efx_vport_config_t *vport_config,
int
sfc_sriov_attach(struct sfc_adapter *sa)
{
- const struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev);
+ const struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
struct sfc_sriov *sriov = &sa->sriov;
efx_vport_config_t *vport_config;
unsigned int i;
diff --git a/drivers/net/sfc/sfc_tx.c b/drivers/net/sfc/sfc_tx.c
index ebc0a8235b..fac56cb27c 100644
--- a/drivers/net/sfc/sfc_tx.c
+++ b/drivers/net/sfc/sfc_tx.c
@@ -230,8 +230,9 @@ sfc_tx_qinit(struct sfc_adapter *sa, sfc_sw_index_t sw_index,
info.max_pdu = encp->enc_mac_pdu_max;
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(sa->eth_dev, *pci_dev);
rc = sa->priv.dp_tx->qcreate(sa->eth_dev->data->port_id, sw_index,
- &RTE_ETH_DEV_TO_PCI(sa->eth_dev)->addr,
+ &pci_dev->addr,
socket_id, &info, &txq_info->dp);
if (rc != 0)
goto fail_dp_tx_qinit;
diff --git a/drivers/net/thunderx/nicvf_ethdev.c b/drivers/net/thunderx/nicvf_ethdev.c
index 76ed76a045..6e34da7c3c 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -1471,7 +1471,7 @@ static int
nicvf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
struct nicvf *nic = nicvf_pmd_priv(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
PMD_INIT_FUNC_TRACE();
@@ -2234,7 +2234,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
}
}
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
rte_eth_copy_pci_info(eth_dev, pci_dev);
eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
diff --git a/drivers/net/txgbe/txgbe_ethdev.c b/drivers/net/txgbe/txgbe_ethdev.c
index 5d360f8305..0f484dfe91 100644
--- a/drivers/net/txgbe/txgbe_ethdev.c
+++ b/drivers/net/txgbe/txgbe_ethdev.c
@@ -525,7 +525,7 @@ static void
txgbe_parse_devargs(struct rte_eth_dev *dev)
{
struct rte_eth_fdir_conf *fdir_conf = TXGBE_DEV_FDIR_CONF(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_devargs *devargs = pci_dev->device.devargs;
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
struct rte_kvargs *kvlist;
@@ -601,7 +601,7 @@ static int
eth_txgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
{
struct txgbe_adapter *ad = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(eth_dev);
struct txgbe_vfta *shadow_vfta = TXGBE_DEV_VFTA(eth_dev);
struct txgbe_hwstrip *hwstrip = TXGBE_DEV_HWSTRIP(eth_dev);
@@ -1397,7 +1397,7 @@ txgbe_vmdq_vlan_hw_filter_enable(struct rte_eth_dev *dev)
static int
txgbe_check_vf_rss_rxq_num(struct rte_eth_dev *dev, uint16_t nb_rx_q)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
switch (nb_rx_q) {
case 1:
@@ -1664,7 +1664,7 @@ txgbe_set_vf_rate_limit(struct rte_eth_dev *dev, uint16_t vf,
struct rte_pci_device *pci_dev;
int ret;
- pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
ret = rte_eth_link_get_nowait(dev->data->port_id, &link);
if (ret < 0)
return ret;
@@ -1736,7 +1736,7 @@ txgbe_dev_start(struct rte_eth_dev *dev)
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
struct txgbe_hw_stats *hw_stats = TXGBE_DEV_STATS(dev);
struct txgbe_vf_info *vfinfo = *TXGBE_DEV_VFDATA(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t intr_vector = 0;
int err;
@@ -2034,7 +2034,7 @@ txgbe_dev_stop(struct rte_eth_dev *dev)
struct txgbe_adapter *adapter = TXGBE_DEV_ADAPTER(dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
struct txgbe_vf_info *vfinfo = *TXGBE_DEV_VFDATA(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int vf;
struct txgbe_tm_conf *tm_conf = TXGBE_DEV_TM_CONF(dev);
@@ -2163,7 +2163,7 @@ static int
txgbe_dev_close(struct rte_eth_dev *dev)
{
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int retries = 0;
int ret;
@@ -2822,7 +2822,7 @@ txgbe_fw_version_get(struct rte_eth_dev *dev, char *fw_version, size_t fw_size)
static int
txgbe_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -3500,7 +3500,7 @@ txgbe_dev_interrupt_get_status(struct rte_eth_dev *dev,
static void
txgbe_dev_link_status_print(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_eth_link link;
rte_eth_linkstatus_get(dev, &link);
@@ -3631,7 +3631,7 @@ static void
txgbe_dev_interrupt_delayed_handler(void *param)
{
struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_interrupt *intr = TXGBE_DEV_INTR(dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
@@ -3978,7 +3978,7 @@ txgbe_remove_rar(struct rte_eth_dev *dev, uint32_t index)
static int
txgbe_set_default_mac_addr(struct rte_eth_dev *dev, struct rte_ether_addr *addr)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
txgbe_remove_rar(dev, 0);
txgbe_add_rar(dev, addr, 0, pci_dev->max_vfs);
@@ -4149,7 +4149,7 @@ txgbe_convert_vm_rx_mask_to_val(uint16_t rx_mask, uint32_t orig_val)
static int
txgbe_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t mask;
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
@@ -4231,7 +4231,7 @@ txgbe_set_ivar_map(struct txgbe_hw *hw, int8_t direction,
static void
txgbe_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
uint32_t queue_id, base = TXGBE_MISC_VEC_ID;
diff --git a/drivers/net/txgbe/txgbe_ethdev_vf.c b/drivers/net/txgbe/txgbe_ethdev_vf.c
index 39a5fff65c..7a50c7a855 100644
--- a/drivers/net/txgbe/txgbe_ethdev_vf.c
+++ b/drivers/net/txgbe/txgbe_ethdev_vf.c
@@ -232,7 +232,7 @@ eth_txgbevf_dev_init(struct rte_eth_dev *eth_dev)
int err;
uint32_t tc, tcs;
struct txgbe_adapter *ad = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_hw *hw = TXGBE_DEV_HW(eth_dev);
struct txgbe_vfta *shadow_vfta = TXGBE_DEV_VFTA(eth_dev);
@@ -561,7 +561,7 @@ static int
txgbevf_dev_info_get(struct rte_eth_dev *dev,
struct rte_eth_dev_info *dev_info)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
dev_info->max_rx_queues = (uint16_t)hw->mac.max_rx_queues;
@@ -696,7 +696,7 @@ txgbevf_dev_start(struct rte_eth_dev *dev)
{
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
uint32_t intr_vector = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int err, mask = 0;
@@ -801,7 +801,7 @@ txgbevf_dev_stop(struct rte_eth_dev *dev)
{
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
struct txgbe_adapter *adapter = TXGBE_DEV_ADAPTER(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
if (hw->adapter_stopped)
@@ -841,7 +841,7 @@ static int
txgbevf_dev_close(struct rte_eth_dev *dev)
{
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
int ret;
@@ -1023,7 +1023,7 @@ txgbevf_vlan_offload_set(struct rte_eth_dev *dev, int mask)
static int
txgbevf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_interrupt *intr = TXGBE_DEV_INTR(dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
@@ -1045,7 +1045,7 @@ txgbevf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
{
struct txgbe_interrupt *intr = TXGBE_DEV_INTR(dev);
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
uint32_t vec = TXGBE_MISC_VEC_ID;
@@ -1085,7 +1085,7 @@ txgbevf_set_ivar_map(struct txgbe_hw *hw, int8_t direction,
static void
txgbevf_configure_msix(struct rte_eth_dev *dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct txgbe_hw *hw = TXGBE_DEV_HW(dev);
uint32_t q_idx;
diff --git a/drivers/net/txgbe/txgbe_flow.c b/drivers/net/txgbe/txgbe_flow.c
index a97588e57a..1bb0d3978c 100644
--- a/drivers/net/txgbe/txgbe_flow.c
+++ b/drivers/net/txgbe/txgbe_flow.c
@@ -1171,7 +1171,7 @@ cons_parse_l2_tn_filter(struct rte_eth_dev *dev,
const struct rte_flow_item_e_tag *e_tag_mask;
const struct rte_flow_action *act;
const struct rte_flow_action_vf *act_vf;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
if (!pattern) {
rte_flow_error_set(error, EINVAL,
@@ -1328,7 +1328,7 @@ txgbe_parse_l2_tn_filter(struct rte_eth_dev *dev,
struct rte_flow_error *error)
{
int ret = 0;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t vf_num;
if (!txgbe_is_pf(TXGBE_DEV_HW(dev))) {
diff --git a/drivers/net/txgbe/txgbe_pf.c b/drivers/net/txgbe/txgbe_pf.c
index 700632bd88..91f73521fe 100644
--- a/drivers/net/txgbe/txgbe_pf.c
+++ b/drivers/net/txgbe/txgbe_pf.c
@@ -33,7 +33,7 @@
static inline uint16_t
dev_num_vf(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
return pci_dev->max_vfs;
}
diff --git a/drivers/net/txgbe/txgbe_tm.c b/drivers/net/txgbe/txgbe_tm.c
index b62bcf54aa..29c7c4adfb 100644
--- a/drivers/net/txgbe/txgbe_tm.c
+++ b/drivers/net/txgbe/txgbe_tm.c
@@ -354,7 +354,7 @@ txgbe_queue_base_nb_get(struct rte_eth_dev *dev, uint16_t tc_node_no,
uint16_t *base, uint16_t *nb)
{
uint8_t nb_tcs = txgbe_tc_nb_get(dev);
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(dev, *pci_dev);
uint16_t vf_num = pci_dev->max_vfs;
*base = 0;
diff --git a/drivers/net/virtio/virtio_pci_ethdev.c b/drivers/net/virtio/virtio_pci_ethdev.c
index fcda002297..d4f4bb0920 100644
--- a/drivers/net/virtio/virtio_pci_ethdev.c
+++ b/drivers/net/virtio/virtio_pci_ethdev.c
@@ -73,13 +73,13 @@ eth_virtio_pci_init(struct rte_eth_dev *eth_dev)
{
struct virtio_pci_dev *dev = eth_dev->data->dev_private;
struct virtio_hw *hw = &dev->hw;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret;
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
hw->port_id = eth_dev->data->port_id;
VTPCI_DEV(hw) = pci_dev;
- ret = vtpci_init(RTE_ETH_DEV_TO_PCI(eth_dev), dev);
+ ret = vtpci_init(pci_dev, dev);
if (ret) {
PMD_INIT_LOG(ERR, "Failed to init PCI device");
return -1;
@@ -91,7 +91,7 @@ eth_virtio_pci_init(struct rte_eth_dev *eth_dev)
else
VIRTIO_OPS(hw) = &virtio_legacy_ops;
- ret = virtio_remap_pci(RTE_ETH_DEV_TO_PCI(eth_dev), dev);
+ ret = virtio_remap_pci(pci_dev, dev);
if (ret < 0) {
PMD_INIT_LOG(ERR, "Failed to remap PCI device");
return -1;
@@ -111,7 +111,7 @@ eth_virtio_pci_init(struct rte_eth_dev *eth_dev)
return 0;
err_unmap:
- rte_pci_unmap_device(RTE_ETH_DEV_TO_PCI(eth_dev));
+ rte_pci_unmap_device(pci_dev);
if (!dev->modern)
vtpci_legacy_ioport_unmap(hw);
@@ -127,11 +127,12 @@ eth_virtio_pci_uninit(struct rte_eth_dev *eth_dev)
PMD_INIT_FUNC_TRACE();
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
dev = eth_dev->data->dev_private;
hw = &dev->hw;
if (dev->modern)
- rte_pci_unmap_device(RTE_ETH_DEV_TO_PCI(eth_dev));
+ rte_pci_unmap_device(pci_dev);
else
vtpci_legacy_ioport_unmap(hw);
return 0;
diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c b/drivers/net/vmxnet3/vmxnet3_ethdev.c
index da9af08207..b7cf217724 100644
--- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
+++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
@@ -308,7 +308,7 @@ eth_vmxnet3_setup_capabilities(struct vmxnet3_hw *hw,
struct rte_eth_dev *eth_dev)
{
uint32_t dcr, ptcr, value;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
VMXNET3_CMD_GET_MAX_CAPABILITIES);
@@ -381,7 +381,7 @@ eth_vmxnet3_dev_init(struct rte_eth_dev *eth_dev)
eth_dev->tx_pkt_burst = &vmxnet3_xmit_pkts;
eth_dev->tx_pkt_prepare = vmxnet3_prep_pkts;
eth_dev->rx_queue_count = vmxnet3_dev_rx_queue_count;
- pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
/* extra mbuf field is required to guess MSS */
vmxnet3_segs_dynfield_offset =
diff --git a/drivers/net/xsc/xsc_ethdev.c b/drivers/net/xsc/xsc_ethdev.c
index 07fc52ac7b..39a67ff8cd 100644
--- a/drivers/net/xsc/xsc_ethdev.c
+++ b/drivers/net/xsc/xsc_ethdev.c
@@ -1048,7 +1048,7 @@ xsc_ethdev_init(struct rte_eth_dev *eth_dev)
PMD_INIT_FUNC_TRACE();
priv->eth_dev = eth_dev;
- priv->pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ priv->pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *priv->pci_dev);
ret = xsc_dev_init(priv->pci_dev, &priv->xdev);
if (ret) {
diff --git a/drivers/net/zxdh/zxdh_ethdev.c b/drivers/net/zxdh/zxdh_ethdev.c
index aeb01f4652..80ff19b3ea 100644
--- a/drivers/net/zxdh/zxdh_ethdev.c
+++ b/drivers/net/zxdh/zxdh_ethdev.c
@@ -111,7 +111,7 @@ zxdh_intr_unmask(struct rte_eth_dev *dev)
if (rte_intr_ack(dev->intr_handle) < 0)
return -1;
- hw->use_msix = zxdh_pci_msix_detect(RTE_ETH_DEV_TO_PCI(dev));
+ hw->use_msix = zxdh_pci_msix_detect(RTE_CLASS_TO_BUS_DEVICE(dev, struct rte_pci_device));
return 0;
}
@@ -1586,7 +1586,7 @@ static int32_t
zxdh_init_device(struct rte_eth_dev *eth_dev)
{
struct zxdh_hw *hw = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
int ret = 0;
ret = zxdh_read_pci_caps(pci_dev, hw);
@@ -1820,7 +1820,7 @@ zxdh_get_dev_shared_data_idx(uint32_t dev_serial_id)
static int zxdh_init_dev_share_data(struct rte_eth_dev *eth_dev)
{
struct zxdh_hw *hw = eth_dev->data->dev_private;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
uint32_t serial_id = (pci_dev->addr.domain << 16) |
(pci_dev->addr.bus << 8) | pci_dev->addr.devid;
uint16_t slot_id = 0;
@@ -2201,7 +2201,7 @@ is_inic_pf(uint16_t device_id)
static int
zxdh_eth_dev_init(struct rte_eth_dev *eth_dev)
{
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+ struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
struct zxdh_hw *hw = eth_dev->data->dev_private;
int ret = 0;
diff --git a/drivers/raw/ifpga/afu_pmd_n3000.c b/drivers/raw/ifpga/afu_pmd_n3000.c
index f092ee2dec..d5520a0d71 100644
--- a/drivers/raw/ifpga/afu_pmd_n3000.c
+++ b/drivers/raw/ifpga/afu_pmd_n3000.c
@@ -1467,11 +1467,11 @@ static struct rte_pci_device *n3000_afu_get_pci_dev(struct afu_rawdev *dev)
if (!dev || !dev->rawdev || !dev->rawdev->device)
return NULL;
- afudev = RTE_BUS_DEVICE(dev->rawdev->device, *afudev);
+ afudev = RTE_CLASS_TO_BUS_DEVICE(dev->rawdev, *afudev);
if (!afudev->rawdev || !afudev->rawdev->device)
return NULL;
- return RTE_BUS_DEVICE(afudev->rawdev->device, struct rte_pci_device);
+ return RTE_CLASS_TO_BUS_DEVICE(afudev->rawdev, struct rte_pci_device);
}
static int dma_afu_set_irqs(struct afu_rawdev *dev, uint32_t vec_start,
diff --git a/lib/eal/include/bus_driver.h b/lib/eal/include/bus_driver.h
index 3d04efbd3f..0a7e23d98d 100644
--- a/lib/eal/include/bus_driver.h
+++ b/lib/eal/include/bus_driver.h
@@ -461,6 +461,24 @@ void rte_bus_unregister(struct rte_bus *bus);
#define RTE_BUS_DRIVER(drv, bus_drv_type) \
container_of(drv, typeof(bus_drv_type), driver)
+/**
+ * Helper macro to convert a device class pointer to a bus-specific device type.
+ * Works with any device class (ethdev, cryptodev, eventdev, bbdev, etc.) that has
+ * a 'device' field pointing to struct rte_device.
+ *
+ * Example: RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev) or
+ * RTE_CLASS_TO_BUS_DEVICE(eth_dev, struct rte_pci_device)
+ *
+ * @param class_dev
+ * Pointer to device class structure (e.g., struct rte_eth_dev *)
+ * @param bus_dev_type
+ * Bus device type for type inference (e.g., *pci_dev or struct rte_pci_device)
+ * @return
+ * Pointer to the bus-specific device structure
+ */
+#define RTE_CLASS_TO_BUS_DEVICE(class_dev, bus_dev_type) \
+ RTE_BUS_DEVICE((class_dev)->device, bus_dev_type)
+
/**
* Helper macro to iterate over all devices on a bus.
*
--
2.53.0
^ permalink raw reply related
* [PATCH v3 24/25] eventdev: rename dev field to device
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Pavan Nikhilesh,
Shijith Thotton, Tirthendu Sarkar, Jerin Jacob
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
Rename the rte_eventdev structure field from 'dev' to 'device' to align
with the naming convention used by all other device classes in DPDK
(ethdev, cryptodev, bbdev, compressdev, rawdev, regexdev, dmadev, gpudev,
and mldev).
This change provides consistency across all device classes: each device
class structure now contains a 'struct rte_device *device' field
pointing to the backing device.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/event/cnxk/cn10k_eventdev.c | 8 ++++----
drivers/event/cnxk/cn20k_eventdev.c | 8 ++++----
drivers/event/cnxk/cn9k_eventdev.c | 6 +++---
drivers/event/cnxk/cnxk_eventdev.c | 2 +-
drivers/event/dlb2/pf/dlb2_pf.c | 2 +-
drivers/event/skeleton/skeleton_eventdev.c | 2 +-
lib/eventdev/eventdev_pmd.h | 2 +-
lib/eventdev/eventdev_pmd_pci.h | 4 ++--
lib/eventdev/eventdev_pmd_vdev.h | 2 +-
lib/eventdev/rte_eventdev.c | 14 +++++++-------
10 files changed, 25 insertions(+), 25 deletions(-)
diff --git a/drivers/event/cnxk/cn10k_eventdev.c b/drivers/event/cnxk/cn10k_eventdev.c
index 2e4b8aab92..8289fc44d6 100644
--- a/drivers/event/cnxk/cn10k_eventdev.c
+++ b/drivers/event/cnxk/cn10k_eventdev.c
@@ -921,7 +921,7 @@ static int
cn10k_crypto_adapter_caps_get(const struct rte_eventdev *event_dev,
const struct rte_cryptodev *cdev, uint32_t *caps)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn10k", ENOTSUP);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn10k", ENOTSUP);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn10k", ENOTSUP);
*caps = RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_OP_FWD |
@@ -939,7 +939,7 @@ cn10k_crypto_adapter_qp_add(const struct rte_eventdev *event_dev,
{
int ret;
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn10k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn10k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn10k", EINVAL);
cn10k_sso_fp_fns_set((struct rte_eventdev *)(uintptr_t)event_dev);
@@ -954,7 +954,7 @@ static int
cn10k_crypto_adapter_qp_del(const struct rte_eventdev *event_dev, const struct rte_cryptodev *cdev,
int32_t queue_pair_id)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn10k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn10k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn10k", EINVAL);
return cnxk_crypto_adapter_qp_del(cdev, queue_pair_id);
@@ -973,7 +973,7 @@ cn10k_crypto_adapter_vec_limits(const struct rte_eventdev *event_dev,
const struct rte_cryptodev *cdev,
struct rte_event_crypto_adapter_vector_limits *limits)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn10k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn10k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn10k", EINVAL);
limits->log2_sz = false;
diff --git a/drivers/event/cnxk/cn20k_eventdev.c b/drivers/event/cnxk/cn20k_eventdev.c
index ff3aaac16a..9d34168c32 100644
--- a/drivers/event/cnxk/cn20k_eventdev.c
+++ b/drivers/event/cnxk/cn20k_eventdev.c
@@ -1125,7 +1125,7 @@ static int
cn20k_crypto_adapter_caps_get(const struct rte_eventdev *event_dev,
const struct rte_cryptodev *cdev, uint32_t *caps)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn20k", ENOTSUP);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn20k", ENOTSUP);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn20k", ENOTSUP);
*caps = RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_OP_FWD |
@@ -1142,7 +1142,7 @@ cn20k_crypto_adapter_qp_add(const struct rte_eventdev *event_dev, const struct r
{
int ret;
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn20k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn20k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn20k", EINVAL);
cn20k_sso_fp_fns_set((struct rte_eventdev *)(uintptr_t)event_dev);
@@ -1157,7 +1157,7 @@ static int
cn20k_crypto_adapter_qp_del(const struct rte_eventdev *event_dev, const struct rte_cryptodev *cdev,
int32_t queue_pair_id)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn20k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn20k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn20k", EINVAL);
return cnxk_crypto_adapter_qp_del(cdev, queue_pair_id);
@@ -1175,7 +1175,7 @@ cn20k_crypto_adapter_vec_limits(const struct rte_eventdev *event_dev,
const struct rte_cryptodev *cdev,
struct rte_event_crypto_adapter_vector_limits *limits)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn20k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn20k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn20k", EINVAL);
limits->log2_sz = false;
diff --git a/drivers/event/cnxk/cn9k_eventdev.c b/drivers/event/cnxk/cn9k_eventdev.c
index 5f24366770..313dcbb384 100644
--- a/drivers/event/cnxk/cn9k_eventdev.c
+++ b/drivers/event/cnxk/cn9k_eventdev.c
@@ -1038,7 +1038,7 @@ static int
cn9k_crypto_adapter_caps_get(const struct rte_eventdev *event_dev, const struct rte_cryptodev *cdev,
uint32_t *caps)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn9k", ENOTSUP);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn9k", ENOTSUP);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn9k", ENOTSUP);
*caps = RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_OP_FWD |
@@ -1055,7 +1055,7 @@ cn9k_crypto_adapter_qp_add(const struct rte_eventdev *event_dev,
{
int ret;
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn9k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn9k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn9k", EINVAL);
cn9k_sso_fp_fns_set((struct rte_eventdev *)(uintptr_t)event_dev);
@@ -1070,7 +1070,7 @@ static int
cn9k_crypto_adapter_qp_del(const struct rte_eventdev *event_dev, const struct rte_cryptodev *cdev,
int32_t queue_pair_id)
{
- CNXK_VALID_DEV_OR_ERR_RET(event_dev->dev, "event_cn9k", EINVAL);
+ CNXK_VALID_DEV_OR_ERR_RET(event_dev->device, "event_cn9k", EINVAL);
CNXK_VALID_DEV_OR_ERR_RET(cdev->device, "crypto_cn9k", EINVAL);
return cnxk_crypto_adapter_qp_del(cdev, queue_pair_id);
diff --git a/drivers/event/cnxk/cnxk_eventdev.c b/drivers/event/cnxk/cnxk_eventdev.c
index 8eff2ba8e0..6f000ff49e 100644
--- a/drivers/event/cnxk/cnxk_eventdev.c
+++ b/drivers/event/cnxk/cnxk_eventdev.c
@@ -654,7 +654,7 @@ cnxk_sso_init(struct rte_eventdev *event_dev)
return -ENOMEM;
}
- pci_dev = RTE_BUS_DEVICE(event_dev->dev, *pci_dev);
+ pci_dev = RTE_BUS_DEVICE(event_dev->device, *pci_dev);
dev->sso.pci_dev = pci_dev;
*(uint64_t *)mz->addr = (uint64_t)dev;
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a498ba8c41..82075bbf0b 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -784,7 +784,7 @@ dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
dlb2_pf_iface_fn_ptrs_init();
- pci_dev = RTE_BUS_DEVICE(eventdev->dev, *pci_dev);
+ pci_dev = RTE_BUS_DEVICE(eventdev->device, *pci_dev);
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
diff --git a/drivers/event/skeleton/skeleton_eventdev.c b/drivers/event/skeleton/skeleton_eventdev.c
index e07744d2f1..4292644fde 100644
--- a/drivers/event/skeleton/skeleton_eventdev.c
+++ b/drivers/event/skeleton/skeleton_eventdev.c
@@ -332,7 +332,7 @@ skeleton_eventdev_init(struct rte_eventdev *eventdev)
if (rte_eal_process_type() != RTE_PROC_PRIMARY)
return 0;
- pci_dev = RTE_BUS_DEVICE(eventdev->dev, *pci_dev);
+ pci_dev = RTE_BUS_DEVICE(eventdev->device, *pci_dev);
skel->reg_base = (uintptr_t)pci_dev->mem_resource[0].addr;
if (!skel->reg_base) {
diff --git a/lib/eventdev/eventdev_pmd.h b/lib/eventdev/eventdev_pmd.h
index d13cc433a7..9309dce5e1 100644
--- a/lib/eventdev/eventdev_pmd.h
+++ b/lib/eventdev/eventdev_pmd.h
@@ -156,7 +156,7 @@ struct __rte_cache_aligned rte_eventdev {
/**< Pointer to device data */
struct eventdev_ops *dev_ops;
/**< Functions exported by PMD */
- struct rte_device *dev;
+ struct rte_device *device;
/**< Device info. supplied by probing */
uint8_t attached : 1;
diff --git a/lib/eventdev/eventdev_pmd_pci.h b/lib/eventdev/eventdev_pmd_pci.h
index 5cb5916a84..ebc7d12b1d 100644
--- a/lib/eventdev/eventdev_pmd_pci.h
+++ b/lib/eventdev/eventdev_pmd_pci.h
@@ -68,7 +68,7 @@ rte_event_pmd_pci_probe_named(struct rte_pci_driver *pci_drv,
"device data");
}
- eventdev->dev = &pci_dev->device;
+ eventdev->device = &pci_dev->device;
/* Invoke PMD device initialization function */
retval = devinit(eventdev);
@@ -150,7 +150,7 @@ rte_event_pmd_pci_remove(struct rte_pci_device *pci_dev,
/* Free event device */
rte_event_pmd_release(eventdev);
- eventdev->dev = NULL;
+ eventdev->device = NULL;
return 0;
}
diff --git a/lib/eventdev/eventdev_pmd_vdev.h b/lib/eventdev/eventdev_pmd_vdev.h
index 4eaefa0b0b..ae1c950bed 100644
--- a/lib/eventdev/eventdev_pmd_vdev.h
+++ b/lib/eventdev/eventdev_pmd_vdev.h
@@ -67,7 +67,7 @@ rte_event_pmd_vdev_init(const char *name, size_t dev_private_size,
rte_panic("Cannot allocate memzone for private device"
" data");
}
- eventdev->dev = &vdev->device;
+ eventdev->device = &vdev->device;
return eventdev;
}
diff --git a/lib/eventdev/rte_eventdev.c b/lib/eventdev/rte_eventdev.c
index b921142d7b..572cd5bd7d 100644
--- a/lib/eventdev/rte_eventdev.c
+++ b/lib/eventdev/rte_eventdev.c
@@ -68,8 +68,8 @@ rte_event_dev_get_dev_id(const char *name)
for (i = 0; i < eventdev_globals.nb_devs; i++) {
cmp = (strncmp(rte_event_devices[i].data->name, name,
RTE_EVENTDEV_NAME_MAX_LEN) == 0) ||
- (rte_event_devices[i].dev ? (strncmp(
- rte_event_devices[i].dev->driver->name, name,
+ (rte_event_devices[i].device ? (strncmp(
+ rte_event_devices[i].device->driver->name, name,
RTE_EVENTDEV_NAME_MAX_LEN) == 0) : 0);
if (cmp && (rte_event_devices[i].attached ==
RTE_EVENTDEV_ATTACHED)) {
@@ -114,9 +114,9 @@ rte_event_dev_info_get(uint8_t dev_id, struct rte_event_dev_info *dev_info)
dev_info->dequeue_timeout_ns = dev->data->dev_conf.dequeue_timeout_ns;
- dev_info->dev = dev->dev;
- if (dev->dev != NULL && dev->dev->driver != NULL)
- dev_info->driver_name = dev->dev->driver->name;
+ dev_info->dev = dev->device;
+ if (dev->device != NULL && dev->device->driver != NULL)
+ dev_info->driver_name = dev->device->driver->name;
rte_eventdev_trace_info_get(dev_id, dev_info, dev_info->dev);
@@ -1812,8 +1812,8 @@ handle_dev_info(const char *cmd __rte_unused,
rte_tel_data_start_dict(d);
rte_tel_data_add_dict_int(d, "dev_id", dev_id);
- rte_tel_data_add_dict_string(d, "dev_name", dev->dev->name);
- rte_tel_data_add_dict_string(d, "dev_driver", dev->dev->driver->name);
+ rte_tel_data_add_dict_string(d, "dev_name", dev->device->name);
+ rte_tel_data_add_dict_string(d, "dev_driver", dev->device->driver->name);
rte_tel_data_add_dict_string(d, "state",
dev->data->dev_started ? "started" : "stopped");
rte_tel_data_add_dict_int(d, "socket_id", dev->data->socket_id);
--
2.53.0
^ permalink raw reply related
* [PATCH v3 23/25] drivers/bus: remove specific bus types
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Chenbo Xia, Tomasz Duszynski, Chengwen Feng, Long Li, Wei Hu
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
All the buses can have a simple rte_bus object.
Mark as static whenever possible.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/auxiliary/auxiliary_common.c | 44 ++++++++-------
drivers/bus/auxiliary/linux/auxiliary.c | 10 ++--
drivers/bus/auxiliary/private.h | 9 +---
drivers/bus/cdx/bus_cdx_driver.h | 1 -
drivers/bus/cdx/cdx.c | 42 +++++++--------
drivers/bus/cdx/private.h | 7 ---
drivers/bus/dpaa/dpaa_bus.c | 58 +++++++++-----------
drivers/bus/fslmc/fslmc_bus.c | 62 +++++++++++-----------
drivers/bus/fslmc/fslmc_vfio.c | 30 +++++------
drivers/bus/fslmc/portal/dpaa2_hw_dprc.c | 2 +-
drivers/bus/fslmc/private.h | 9 +---
drivers/bus/pci/bsd/pci.c | 16 +++---
drivers/bus/pci/linux/pci.c | 13 +++--
drivers/bus/pci/pci_common.c | 60 ++++++++++-----------
drivers/bus/pci/pci_params.c | 2 +-
drivers/bus/pci/private.h | 9 +---
drivers/bus/pci/windows/pci.c | 13 +++--
drivers/bus/platform/bus_platform_driver.h | 1 -
drivers/bus/platform/platform.c | 52 +++++++++---------
drivers/bus/platform/private.h | 9 ----
drivers/bus/uacce/uacce.c | 49 +++++++----------
drivers/bus/vmbus/linux/vmbus_bus.c | 10 ++--
drivers/bus/vmbus/private.h | 9 +---
drivers/bus/vmbus/vmbus_common.c | 34 ++++++------
24 files changed, 236 insertions(+), 315 deletions(-)
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index ff05716539..048aacf254 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -56,7 +56,7 @@ auxiliary_scan(void)
void
auxiliary_on_scan(struct rte_auxiliary_device *aux_dev)
{
- aux_dev->device.devargs = rte_bus_find_devargs(&auxiliary_bus.bus, aux_dev->name);
+ aux_dev->device.devargs = rte_bus_find_devargs(&auxiliary_bus, aux_dev->name);
}
static bool
@@ -156,7 +156,7 @@ auxiliary_parse(const char *name, void *addr)
if (strlen(name) == 0)
return 0;
- RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus) {
if (drv->match(name))
break;
}
@@ -170,7 +170,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_auxiliary_register)
void
rte_auxiliary_register(struct rte_auxiliary_driver *driver)
{
- rte_bus_add_driver(&auxiliary_bus.bus, &driver->driver);
+ rte_bus_add_driver(&auxiliary_bus, &driver->driver);
}
/* Unregister a driver */
@@ -178,7 +178,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_auxiliary_unregister)
void
rte_auxiliary_unregister(struct rte_auxiliary_driver *driver)
{
- rte_bus_remove_driver(&auxiliary_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&auxiliary_bus, &driver->driver);
}
static int
@@ -189,7 +189,7 @@ auxiliary_unplug(struct rte_device *dev)
ret = rte_auxiliary_driver_remove_dev(adev);
if (ret == 0) {
- rte_bus_remove_device(&auxiliary_bus.bus, &adev->device);
+ rte_bus_remove_device(&auxiliary_bus, &adev->device);
rte_devargs_remove(dev->devargs);
rte_intr_instance_free(adev->intr_handle);
free(adev);
@@ -203,7 +203,7 @@ auxiliary_cleanup(void)
struct rte_auxiliary_device *dev;
int error = 0;
- RTE_BUS_FOREACH_DEV(dev, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &auxiliary_bus) {
int ret;
if (!rte_dev_is_probed(&dev->device))
@@ -250,7 +250,7 @@ auxiliary_get_iommu_class(void)
{
const struct rte_auxiliary_driver *drv;
- RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus) {
if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0)
return RTE_IOVA_VA;
}
@@ -258,22 +258,20 @@ auxiliary_get_iommu_class(void)
return RTE_IOVA_DC;
}
-struct rte_auxiliary_bus auxiliary_bus = {
- .bus = {
- .scan = auxiliary_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = auxiliary_cleanup,
- .find_device = rte_bus_generic_find_device,
- .match = auxiliary_bus_match,
- .probe_device = auxiliary_probe_device,
- .unplug = auxiliary_unplug,
- .parse = auxiliary_parse,
- .dma_map = auxiliary_dma_map,
- .dma_unmap = auxiliary_dma_unmap,
- .get_iommu_class = auxiliary_get_iommu_class,
- .dev_iterate = rte_bus_generic_dev_iterate,
- },
+struct rte_bus auxiliary_bus = {
+ .scan = auxiliary_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = auxiliary_cleanup,
+ .find_device = rte_bus_generic_find_device,
+ .match = auxiliary_bus_match,
+ .probe_device = auxiliary_probe_device,
+ .unplug = auxiliary_unplug,
+ .parse = auxiliary_parse,
+ .dma_map = auxiliary_dma_map,
+ .dma_unmap = auxiliary_dma_unmap,
+ .get_iommu_class = auxiliary_get_iommu_class,
+ .dev_iterate = rte_bus_generic_dev_iterate,
};
-RTE_REGISTER_BUS(auxiliary, auxiliary_bus.bus);
+RTE_REGISTER_BUS(auxiliary, auxiliary_bus);
RTE_LOG_REGISTER_DEFAULT(auxiliary_bus_logtype, NOTICE);
diff --git a/drivers/bus/auxiliary/linux/auxiliary.c b/drivers/bus/auxiliary/linux/auxiliary.c
index b40de65bdd..3a2dca2865 100644
--- a/drivers/bus/auxiliary/linux/auxiliary.c
+++ b/drivers/bus/auxiliary/linux/auxiliary.c
@@ -48,12 +48,12 @@ auxiliary_scan_one(const char *dirname, const char *name)
auxiliary_on_scan(dev);
/* Device is valid, add in list (sorted) */
- RTE_BUS_FOREACH_DEV(dev2, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &auxiliary_bus) {
ret = strcmp(dev->name, dev2->name);
if (ret > 0)
continue;
if (ret < 0) {
- rte_bus_insert_device(&auxiliary_bus.bus, &dev2->device, &dev->device);
+ rte_bus_insert_device(&auxiliary_bus, &dev2->device, &dev->device);
} else { /* already registered */
if (rte_dev_is_probed(&dev2->device) &&
dev2->device.devargs != dev->device.devargs) {
@@ -65,7 +65,7 @@ auxiliary_scan_one(const char *dirname, const char *name)
}
return 0;
}
- rte_bus_add_device(&auxiliary_bus.bus, &dev->device);
+ rte_bus_add_device(&auxiliary_bus, &dev->device);
return 0;
}
@@ -109,14 +109,14 @@ auxiliary_scan(void)
if (e->d_name[0] == '.')
continue;
- if (rte_bus_device_is_ignored(&auxiliary_bus.bus, e->d_name))
+ if (rte_bus_device_is_ignored(&auxiliary_bus, e->d_name))
continue;
snprintf(dirname, sizeof(dirname), "%s/%s",
AUXILIARY_SYSFS_PATH, e->d_name);
/* Ignore if no driver can handle. */
- RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus.bus) {
+ RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus) {
if (drv->match(e->d_name))
break;
}
diff --git a/drivers/bus/auxiliary/private.h b/drivers/bus/auxiliary/private.h
index 116154eb56..282ad94618 100644
--- a/drivers/bus/auxiliary/private.h
+++ b/drivers/bus/auxiliary/private.h
@@ -19,14 +19,7 @@ extern int auxiliary_bus_logtype;
#define AUXILIARY_LOG(level, ...) \
RTE_LOG_LINE(level, AUXILIARY_BUS, __VA_ARGS__)
-/*
- * Structure describing the auxiliary bus
- */
-struct rte_auxiliary_bus {
- struct rte_bus bus; /* Inherit the generic class */
-};
-
-extern struct rte_auxiliary_bus auxiliary_bus;
+extern struct rte_bus auxiliary_bus;
/*
* Test whether the auxiliary device exist.
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index aa44dee5a6..d443178404 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -25,7 +25,6 @@ extern "C" {
/* Forward declarations */
struct rte_cdx_device;
struct rte_cdx_driver;
-struct rte_cdx_bus;
#define RTE_CDX_BUS_DEVICES_PATH "/sys/bus/cdx/devices"
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index cb4f755b8e..2443161e1a 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -84,7 +84,7 @@
#define CDX_DEV_PREFIX "cdx-"
-struct rte_cdx_bus rte_cdx_bus;
+static struct rte_bus rte_cdx_bus;
static int
cdx_get_kernel_driver_by_path(const char *filename, char *driver_name,
@@ -194,7 +194,7 @@ cdx_scan_one(const char *dirname, const char *dev_name)
}
dev->id.device_id = (uint16_t)tmp;
- rte_bus_add_device(&rte_cdx_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_cdx_bus, &dev->device);
return 0;
@@ -226,7 +226,7 @@ cdx_scan(void)
if (e->d_name[0] == '.')
continue;
- if (rte_bus_device_is_ignored(&rte_cdx_bus.bus, e->d_name))
+ if (rte_bus_device_is_ignored(&rte_cdx_bus, e->d_name))
continue;
snprintf(dirname, sizeof(dirname), "%s/%s",
@@ -363,7 +363,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_cdx_register)
void
rte_cdx_register(struct rte_cdx_driver *driver)
{
- rte_bus_add_driver(&rte_cdx_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_cdx_bus, &driver->driver);
}
/* unregister a driver */
@@ -371,7 +371,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_cdx_unregister)
void
rte_cdx_unregister(struct rte_cdx_driver *driver)
{
- rte_bus_remove_driver(&rte_cdx_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_cdx_bus, &driver->driver);
}
/*
@@ -412,7 +412,7 @@ cdx_unplug(struct rte_device *dev)
ret = cdx_detach_dev(cdx_dev);
if (ret == 0) {
- rte_bus_remove_device(&rte_cdx_bus.bus, &cdx_dev->device);
+ rte_bus_remove_device(&rte_cdx_bus, &cdx_dev->device);
rte_devargs_remove(dev->devargs);
free(cdx_dev);
}
@@ -440,27 +440,25 @@ cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
static enum rte_iova_mode
cdx_get_iommu_class(void)
{
- if (TAILQ_EMPTY(&rte_cdx_bus.bus.device_list))
+ if (TAILQ_EMPTY(&rte_cdx_bus.device_list))
return RTE_IOVA_DC;
return RTE_IOVA_VA;
}
-struct rte_cdx_bus rte_cdx_bus = {
- .bus = {
- .scan = cdx_scan,
- .probe = rte_bus_generic_probe,
- .find_device = rte_bus_generic_find_device,
- .match = cdx_bus_match,
- .probe_device = cdx_probe_device,
- .unplug = cdx_unplug,
- .parse = cdx_parse,
- .dma_map = cdx_dma_map,
- .dma_unmap = cdx_dma_unmap,
- .get_iommu_class = cdx_get_iommu_class,
- .dev_iterate = rte_bus_generic_dev_iterate,
- },
+static struct rte_bus rte_cdx_bus = {
+ .scan = cdx_scan,
+ .probe = rte_bus_generic_probe,
+ .find_device = rte_bus_generic_find_device,
+ .match = cdx_bus_match,
+ .probe_device = cdx_probe_device,
+ .unplug = cdx_unplug,
+ .parse = cdx_parse,
+ .dma_map = cdx_dma_map,
+ .dma_unmap = cdx_dma_unmap,
+ .get_iommu_class = cdx_get_iommu_class,
+ .dev_iterate = rte_bus_generic_dev_iterate,
};
-RTE_REGISTER_BUS(cdx, rte_cdx_bus.bus);
+RTE_REGISTER_BUS(cdx, rte_cdx_bus);
RTE_LOG_REGISTER_DEFAULT(cdx_logtype_bus, NOTICE);
diff --git a/drivers/bus/cdx/private.h b/drivers/bus/cdx/private.h
index f69673aaab..289301bade 100644
--- a/drivers/bus/cdx/private.h
+++ b/drivers/bus/cdx/private.h
@@ -7,13 +7,6 @@
#include "bus_cdx_driver.h"
-/**
- * Structure describing the CDX bus.
- */
-struct rte_cdx_bus {
- struct rte_bus bus; /**< Inherit the generic class */
-};
-
/**
* Map a particular resource from a file.
*
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index 08c1607647..ee467b94d5 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -57,10 +57,6 @@
#define DPAA_MAX_PUSH_MODE_QUEUE 8
#define DPAA_DEFAULT_PUSH_MODE_QUEUE 4
-struct rte_dpaa_bus {
- struct rte_bus bus;
-};
-
struct rte_dpaa_bus_private {
int device_count;
int detected;
@@ -69,7 +65,7 @@ struct rte_dpaa_bus_private {
RTE_ATOMIC(uint16_t) push_rxq_num;
};
-static struct rte_dpaa_bus rte_dpaa_bus;
+static struct rte_bus rte_dpaa_bus;
static struct rte_dpaa_bus_private dpaa_bus;
static struct netcfg_info *dpaa_netcfg;
@@ -168,17 +164,17 @@ dpaa_add_to_device_list(struct rte_dpaa_device *newdev)
int comp, inserted = 0;
struct rte_dpaa_device *dev = NULL;
- RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus) {
comp = compare_dpaa_devices(newdev, dev);
if (comp < 0) {
- rte_bus_insert_device(&rte_dpaa_bus.bus, &dev->device, &newdev->device);
+ rte_bus_insert_device(&rte_dpaa_bus, &dev->device, &newdev->device);
inserted = 1;
break;
}
}
if (!inserted)
- rte_bus_add_device(&rte_dpaa_bus.bus, &newdev->device);
+ rte_bus_add_device(&rte_dpaa_bus, &newdev->device);
}
/*
@@ -253,7 +249,7 @@ dpaa_create_device_list(void)
(fman_intf->fman->idx + 1), fman_intf->mac_idx);
}
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus.bus, dev->name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus, dev->name);
if (dev->device.devargs != NULL)
DPAA_BUS_INFO("**Devargs matched %s", dev->name);
@@ -303,7 +299,7 @@ dpaa_create_device_list(void)
sprintf(dev->name, "dpaa_sec-%d", i+1);
DPAA_BUS_LOG(INFO, "%s cryptodev added", dev->name);
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus.bus, dev->name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus, dev->name);
if (dev->device.devargs != NULL)
DPAA_BUS_INFO("**Devargs matched %s", dev->name);
@@ -329,7 +325,7 @@ dpaa_create_device_list(void)
sprintf(dev->name, "dpaa_qdma-%d", i+1);
DPAA_BUS_LOG(INFO, "%s qdma device added", dev->name);
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus.bus, dev->name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_dpaa_bus, dev->name);
if (dev->device.devargs != NULL)
DPAA_BUS_INFO("**Devargs matched %s", dev->name);
@@ -349,8 +345,8 @@ dpaa_clean_device_list(void)
{
struct rte_dpaa_device *dev = NULL;
- RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
- rte_bus_remove_device(&rte_dpaa_bus.bus, &dev->device);
+ RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus) {
+ rte_bus_remove_device(&rte_dpaa_bus, &dev->device);
rte_intr_instance_free(dev->intr_handle);
free(dev);
dev = NULL;
@@ -583,7 +579,7 @@ rte_dpaa_driver_register(struct rte_dpaa_driver *driver)
BUS_INIT_FUNC_TRACE();
- rte_bus_add_driver(&rte_dpaa_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_dpaa_bus, &driver->driver);
}
/* un-register a dpaa bus based dpaa driver */
@@ -593,7 +589,7 @@ rte_dpaa_driver_unregister(struct rte_dpaa_driver *driver)
{
BUS_INIT_FUNC_TRACE();
- rte_bus_remove_driver(&rte_dpaa_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_dpaa_bus, &driver->driver);
}
static bool
@@ -760,7 +756,7 @@ rte_dpaa_bus_scan(void)
process_once = 1;
/* If no device present on DPAA bus nothing needs to be done */
- if (TAILQ_EMPTY(&rte_dpaa_bus.bus.device_list))
+ if (TAILQ_EMPTY(&rte_dpaa_bus.device_list))
return 0;
/* Register DPAA mempool ops only if any DPAA device has
@@ -768,7 +764,7 @@ rte_dpaa_bus_scan(void)
*/
rte_mbuf_set_platform_mempool_ops(DPAA_MEMPOOL_OPS_NAME);
- RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus) {
if (dev->device_type == FSL_DPAA_ETH) {
ret = rte_dpaa_setup_intr(dev->intr_handle);
if (ret)
@@ -816,7 +812,7 @@ dpaa_bus_cleanup(void)
struct rte_dpaa_device *dev;
BUS_INIT_FUNC_TRACE();
- RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus) {
const struct rte_dpaa_driver *drv;
int ret = 0;
@@ -859,19 +855,17 @@ RTE_FINI_PRIO(dpaa_cleanup, 102)
DPAA_BUS_DEBUG("Worker thread clean up done");
}
-static struct rte_dpaa_bus rte_dpaa_bus = {
- .bus = {
- .scan = rte_dpaa_bus_scan,
- .probe = rte_bus_generic_probe,
- .parse = rte_dpaa_bus_parse,
- .dev_compare = dpaa_bus_dev_compare,
- .find_device = rte_bus_generic_find_device,
- .get_iommu_class = rte_dpaa_get_iommu_class,
- .match = dpaa_bus_match,
- .probe_device = dpaa_bus_probe_device,
- .dev_iterate = rte_bus_generic_dev_iterate,
- .cleanup = dpaa_bus_cleanup,
- },
+static struct rte_bus rte_dpaa_bus = {
+ .scan = rte_dpaa_bus_scan,
+ .probe = rte_bus_generic_probe,
+ .parse = rte_dpaa_bus_parse,
+ .dev_compare = dpaa_bus_dev_compare,
+ .find_device = rte_bus_generic_find_device,
+ .get_iommu_class = rte_dpaa_get_iommu_class,
+ .match = dpaa_bus_match,
+ .probe_device = dpaa_bus_probe_device,
+ .dev_iterate = rte_bus_generic_dev_iterate,
+ .cleanup = dpaa_bus_cleanup,
};
static struct rte_dpaa_bus_private dpaa_bus = {
@@ -879,5 +873,5 @@ static struct rte_dpaa_bus_private dpaa_bus = {
.device_count = 0,
};
-RTE_REGISTER_BUS(dpaa_bus, rte_dpaa_bus.bus);
+RTE_REGISTER_BUS(dpaa_bus, rte_dpaa_bus);
RTE_LOG_REGISTER_DEFAULT(dpaa_logtype_bus, NOTICE);
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 821fe63955..88511d4dc7 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -27,7 +27,7 @@
#define VFIO_IOMMU_GROUP_PATH "/sys/kernel/iommu_groups"
-struct rte_fslmc_bus rte_fslmc_bus;
+struct rte_bus rte_fslmc_bus;
static int fslmc_bus_device_count[DPAA2_DEVTYPE_MAX];
#define DPAA2_SEQN_DYNFIELD_NAME "dpaa2_seqn_dynfield"
@@ -48,8 +48,8 @@ cleanup_fslmc_device_list(void)
{
struct rte_dpaa2_device *dev;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
rte_intr_instance_free(dev->intr_handle);
free(dev);
dev = NULL;
@@ -85,17 +85,17 @@ insert_in_device_list(struct rte_dpaa2_device *newdev)
int comp, inserted = 0;
struct rte_dpaa2_device *dev = NULL;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
comp = compare_dpaa2_devname(newdev, dev);
if (comp < 0) {
- rte_bus_insert_device(&rte_fslmc_bus.bus, &dev->device, &newdev->device);
+ rte_bus_insert_device(&rte_fslmc_bus, &dev->device, &newdev->device);
inserted = 1;
break;
}
}
if (!inserted)
- rte_bus_add_device(&rte_fslmc_bus.bus, &newdev->device);
+ rte_bus_add_device(&rte_fslmc_bus, &newdev->device);
}
static void
@@ -106,7 +106,7 @@ dump_device_list(void)
/* Only if the log level has been set to Debugging, print list */
if (rte_log_can_log(dpaa2_logtype_bus, RTE_LOG_DEBUG)) {
DPAA2_BUS_LOG(DEBUG, "List of devices scanned on bus:");
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
DPAA2_BUS_LOG(DEBUG, "\t\t%s", dev->device.name);
}
}
@@ -198,7 +198,7 @@ scan_one_fslmc_device(char *dev_name)
ret = -ENOMEM;
goto cleanup;
}
- dev->device.devargs = rte_bus_find_devargs(&rte_fslmc_bus.bus, dev_name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_fslmc_bus, dev_name);
/* Update the device found into the device_count table */
fslmc_bus_device_count[dev->dev_type]++;
@@ -247,7 +247,7 @@ rte_fslmc_parse(const char *name, void *addr)
*/
if (sep_exists) {
/* If either of "fslmc" or "name" are starting part */
- if (!strncmp(name, rte_fslmc_bus.bus.name, strlen(rte_fslmc_bus.bus.name)) ||
+ if (!strncmp(name, rte_fslmc_bus.name, strlen(rte_fslmc_bus.name)) ||
(!strncmp(name, "name", strlen("name")))) {
goto jump_out;
} else {
@@ -317,8 +317,8 @@ rte_fslmc_scan(void)
struct rte_dpaa2_device *dev;
DPAA2_BUS_DEBUG("Fslmc bus already scanned. Not rescanning");
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
- dev->device.devargs = rte_bus_find_devargs(&rte_fslmc_bus.bus,
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
+ dev->device.devargs = rte_bus_find_devargs(&rte_fslmc_bus,
dev->device.name);
}
return 0;
@@ -369,7 +369,7 @@ rte_fslmc_scan(void)
dump_device_list();
/* Bus initialization - only if devices were found */
- if (!TAILQ_EMPTY(&rte_fslmc_bus.bus.device_list)) {
+ if (!TAILQ_EMPTY(&rte_fslmc_bus.device_list)) {
static const struct rte_mbuf_dynfield dpaa2_seqn_dynfield_desc = {
.name = DPAA2_SEQN_DYNFIELD_NAME,
.size = sizeof(dpaa2_seqn_t),
@@ -455,7 +455,7 @@ rte_fslmc_driver_register(struct rte_dpaa2_driver *driver)
RTE_VERIFY(driver);
RTE_VERIFY(driver->probe != NULL);
- rte_bus_add_driver(&rte_fslmc_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_fslmc_bus, &driver->driver);
}
/*un-register a fslmc bus based dpaa2 driver */
@@ -463,7 +463,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_fslmc_driver_unregister)
void
rte_fslmc_driver_unregister(struct rte_dpaa2_driver *driver)
{
- rte_bus_remove_driver(&rte_fslmc_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_fslmc_bus, &driver->driver);
}
/*
@@ -475,8 +475,8 @@ fslmc_all_device_support_iova(void)
struct rte_dpaa2_device *dev;
struct rte_dpaa2_driver *drv;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
- RTE_BUS_FOREACH_DRV(drv, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
+ RTE_BUS_FOREACH_DRV(drv, &rte_fslmc_bus) {
if (!fslmc_bus_match(&drv->driver, &dev->device))
continue;
/* if the driver is not supporting IOVA */
@@ -496,7 +496,7 @@ rte_dpaa2_get_iommu_class(void)
if (rte_eal_iova_mode() == RTE_IOVA_PA)
return RTE_IOVA_PA;
- if (TAILQ_EMPTY(&rte_fslmc_bus.bus.device_list))
+ if (TAILQ_EMPTY(&rte_fslmc_bus.device_list))
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
@@ -546,21 +546,19 @@ fslmc_bus_unplug(struct rte_device *rte_dev)
return -ENODEV;
}
-struct rte_fslmc_bus rte_fslmc_bus = {
- .bus = {
- .scan = rte_fslmc_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = rte_fslmc_close,
- .parse = rte_fslmc_parse,
- .dev_compare = fslmc_dev_compare,
- .find_device = rte_bus_generic_find_device,
- .get_iommu_class = rte_dpaa2_get_iommu_class,
- .match = fslmc_bus_match,
- .probe_device = fslmc_bus_probe_device,
- .unplug = fslmc_bus_unplug,
- .dev_iterate = rte_bus_generic_dev_iterate,
- },
+struct rte_bus rte_fslmc_bus = {
+ .scan = rte_fslmc_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = rte_fslmc_close,
+ .parse = rte_fslmc_parse,
+ .dev_compare = fslmc_dev_compare,
+ .find_device = rte_bus_generic_find_device,
+ .get_iommu_class = rte_dpaa2_get_iommu_class,
+ .match = fslmc_bus_match,
+ .probe_device = fslmc_bus_probe_device,
+ .unplug = fslmc_bus_unplug,
+ .dev_iterate = rte_bus_generic_dev_iterate,
};
-RTE_REGISTER_BUS(fslmc, rte_fslmc_bus.bus);
+RTE_REGISTER_BUS(fslmc, rte_fslmc_bus);
RTE_LOG_REGISTER_DEFAULT(dpaa2_logtype_bus, NOTICE);
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 5784adaf90..412b70e5ae 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -1554,12 +1554,12 @@ fslmc_vfio_close_group(void)
return -EIO;
}
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->device.devargs &&
dev->device.devargs->policy == RTE_DEV_BLOCKED) {
DPAA2_BUS_LOG(DEBUG, "%s Blacklisted, skipping",
dev->device.name);
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
continue;
}
switch (dev->dev_type) {
@@ -1599,7 +1599,7 @@ fslmc_vfio_process_group(void)
bool is_dpmcp_in_blocklist = false, is_dpio_in_blocklist = false;
int dpmcp_count = 0, dpio_count = 0, current_device;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->dev_type == DPAA2_MPORTAL) {
dpmcp_count++;
if (dev->device.devargs &&
@@ -1616,14 +1616,14 @@ fslmc_vfio_process_group(void)
/* Search the MCP as that should be initialized first. */
current_device = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->dev_type == DPAA2_MPORTAL) {
current_device++;
if (dev->device.devargs &&
dev->device.devargs->policy == RTE_DEV_BLOCKED) {
DPAA2_BUS_LOG(DEBUG, "%s Blocked, skipping",
dev->device.name);
- rte_bus_remove_device(&rte_fslmc_bus.bus,
+ rte_bus_remove_device(&rte_fslmc_bus,
&dev->device);
continue;
}
@@ -1632,7 +1632,7 @@ fslmc_vfio_process_group(void)
!is_dpmcp_in_blocklist) {
if (dpmcp_count == 1 ||
current_device != dpmcp_count) {
- rte_bus_remove_device(&rte_fslmc_bus.bus,
+ rte_bus_remove_device(&rte_fslmc_bus,
&dev->device);
continue;
}
@@ -1647,7 +1647,7 @@ fslmc_vfio_process_group(void)
found_mportal = 1;
}
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
free(dev);
dev = NULL;
/* Ideally there is only a single dpmcp, but in case
@@ -1666,26 +1666,26 @@ fslmc_vfio_process_group(void)
* other devices.
*/
current_device = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->dev_type == DPAA2_DPRC) {
ret = fslmc_process_iodevices(dev);
if (ret) {
DPAA2_BUS_ERR("Unable to process dprc");
return ret;
}
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
}
}
current_device = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
if (dev->dev_type == DPAA2_IO)
current_device++;
if (dev->device.devargs &&
dev->device.devargs->policy == RTE_DEV_BLOCKED) {
DPAA2_BUS_LOG(DEBUG, "%s Blocked, skipping",
dev->device.name);
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
continue;
}
if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
@@ -1693,7 +1693,7 @@ fslmc_vfio_process_group(void)
dev->dev_type != DPAA2_CRYPTO &&
dev->dev_type != DPAA2_QDMA &&
dev->dev_type != DPAA2_IO) {
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
continue;
}
switch (dev->dev_type) {
@@ -1735,13 +1735,13 @@ fslmc_vfio_process_group(void)
if (!is_dpio_in_blocklist && dpio_count > 1) {
if (rte_eal_process_type() == RTE_PROC_SECONDARY
&& current_device != dpio_count) {
- rte_bus_remove_device(&rte_fslmc_bus.bus,
+ rte_bus_remove_device(&rte_fslmc_bus,
&dev->device);
break;
}
if (rte_eal_process_type() == RTE_PROC_PRIMARY
&& current_device == dpio_count) {
- rte_bus_remove_device(&rte_fslmc_bus.bus,
+ rte_bus_remove_device(&rte_fslmc_bus,
&dev->device);
break;
}
@@ -1760,7 +1760,7 @@ fslmc_vfio_process_group(void)
/* Unknown - ignore */
DPAA2_BUS_DEBUG("Found unknown device (%s)",
dev->device.name);
- rte_bus_remove_device(&rte_fslmc_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_fslmc_bus, &dev->device);
free(dev);
dev = NULL;
}
diff --git a/drivers/bus/fslmc/portal/dpaa2_hw_dprc.c b/drivers/bus/fslmc/portal/dpaa2_hw_dprc.c
index a66e55a456..868ed646af 100644
--- a/drivers/bus/fslmc/portal/dpaa2_hw_dprc.c
+++ b/drivers/bus/fslmc/portal/dpaa2_hw_dprc.c
@@ -49,7 +49,7 @@ rte_dpaa2_create_dprc_device(int vdev_fd __rte_unused,
return ret;
}
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus) {
/** DPRC is always created before it's children are created.*/
dev->container = dprc_node;
if (dev->dev_type == DPAA2_ETH) {
diff --git a/drivers/bus/fslmc/private.h b/drivers/bus/fslmc/private.h
index a0dda4f9d6..20a454c3fc 100644
--- a/drivers/bus/fslmc/private.h
+++ b/drivers/bus/fslmc/private.h
@@ -9,13 +9,6 @@
#include <bus_fslmc_driver.h>
-/*
- * FSLMC bus
- */
-struct rte_fslmc_bus {
- struct rte_bus bus; /**< Generic Bus object */
-};
-
-extern struct rte_fslmc_bus rte_fslmc_bus;
+extern struct rte_bus rte_fslmc_bus;
#endif /* BUS_FSLMC_PRIVATE_H */
diff --git a/drivers/bus/pci/bsd/pci.c b/drivers/bus/pci/bsd/pci.c
index 78d14ab3ae..c6df31d486 100644
--- a/drivers/bus/pci/bsd/pci.c
+++ b/drivers/bus/pci/bsd/pci.c
@@ -297,20 +297,18 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)
}
/* device is valid, add in list (sorted) */
- if (TAILQ_EMPTY(&rte_pci_bus.bus.device_list)) {
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
- }
- else {
+ if (TAILQ_EMPTY(&rte_pci_bus.device_list)) {
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
+ } else {
struct rte_pci_device *dev2 = NULL;
int ret;
- RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus) {
ret = rte_pci_addr_cmp(&dev->addr, &dev2->addr);
if (ret > 0)
continue;
else if (ret < 0) {
- rte_bus_insert_device(&rte_pci_bus.bus, &dev2->device,
- &dev->device);
+ rte_bus_insert_device(&rte_pci_bus, &dev2->device, &dev->device);
} else { /* already registered */
dev2->kdrv = dev->kdrv;
dev2->max_vfs = dev->max_vfs;
@@ -322,7 +320,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)
}
return 0;
}
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
}
return 0;
@@ -378,7 +376,7 @@ rte_pci_scan(void)
pci_addr.function = matches[i].pc_sel.pc_func;
rte_pci_device_name(&pci_addr, name, sizeof(name));
- if (rte_bus_device_is_ignored(&rte_pci_bus.bus, name))
+ if (rte_bus_device_is_ignored(&rte_pci_bus, name))
continue;
if (pci_scan_one(fd, &matches[i]) < 0)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index cf8a60313b..9aae0a5d14 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -321,19 +321,18 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)
return 0;
}
/* device is valid, add in list (sorted) */
- if (TAILQ_EMPTY(&rte_pci_bus.bus.device_list)) {
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ if (TAILQ_EMPTY(&rte_pci_bus.device_list)) {
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
} else {
struct rte_pci_device *dev2;
- RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus) {
ret = rte_pci_addr_cmp(&dev->addr, &dev2->addr);
if (ret > 0)
continue;
if (ret < 0) {
- rte_bus_insert_device(&rte_pci_bus.bus, &dev2->device,
- &dev->device);
+ rte_bus_insert_device(&rte_pci_bus, &dev2->device, &dev->device);
} else { /* already registered */
if (!rte_dev_is_probed(&dev2->device)) {
dev2->kdrv = dev->kdrv;
@@ -377,7 +376,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)
return 0;
}
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
}
return 0;
@@ -458,7 +457,7 @@ rte_pci_scan(void)
if (parse_pci_addr_format(e->d_name, sizeof(e->d_name), &addr) != 0)
continue;
- if (rte_bus_device_is_ignored(&rte_pci_bus.bus, e->d_name))
+ if (rte_bus_device_is_ignored(&rte_pci_bus, e->d_name))
continue;
snprintf(dirname, sizeof(dirname), "%s/%s",
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 2bdb94a924..fd18b8772b 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -87,7 +87,7 @@ pci_common_set(struct rte_pci_device *dev)
dev->name, sizeof(dev->name));
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&rte_pci_bus.bus, dev->name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_pci_bus, dev->name);
if (dev->bus_info != NULL ||
asprintf(&dev->bus_info, "vendor_id=%"PRIx16", device_id=%"PRIx16,
@@ -329,7 +329,7 @@ pci_cleanup(void)
struct rte_pci_device *dev;
int error = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus) {
const struct rte_pci_driver *drv;
int ret = 0;
@@ -353,7 +353,7 @@ pci_cleanup(void)
rte_intr_instance_free(dev->vfio_req_intr_handle);
dev->vfio_req_intr_handle = NULL;
- rte_bus_remove_device(&rte_pci_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_pci_bus, &dev->device);
pci_free(RTE_PCI_DEVICE_INTERNAL(dev));
}
@@ -387,7 +387,7 @@ rte_pci_dump(FILE *f)
{
struct rte_pci_device *dev = NULL;
- RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus) {
pci_dump_one_device(f, dev);
}
}
@@ -422,7 +422,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_pci_register)
void
rte_pci_register(struct rte_pci_driver *driver)
{
- rte_bus_add_driver(&rte_pci_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_pci_bus, &driver->driver);
}
/* unregister a driver */
@@ -430,7 +430,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_pci_unregister)
void
rte_pci_unregister(struct rte_pci_driver *driver)
{
- rte_bus_remove_driver(&rte_pci_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_pci_bus, &driver->driver);
}
/*
@@ -447,7 +447,7 @@ pci_find_device_by_addr(const void *failure_addr)
check_point = (uint64_t)(uintptr_t)failure_addr;
- RTE_BUS_FOREACH_DEV(pdev, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(pdev, &rte_pci_bus) {
for (i = 0; i != RTE_DIM(pdev->mem_resource); i++) {
start = (uint64_t)(uintptr_t)pdev->mem_resource[i].addr;
len = pdev->mem_resource[i].len;
@@ -525,7 +525,7 @@ pci_unplug(struct rte_device *dev)
ret = rte_pci_detach_dev(pdev);
if (ret == 0) {
- rte_bus_remove_device(&rte_pci_bus.bus, &pdev->device);
+ rte_bus_remove_device(&rte_pci_bus, &pdev->device);
rte_devargs_remove(dev->devargs);
pci_free(RTE_PCI_DEVICE_INTERNAL(pdev));
}
@@ -582,7 +582,7 @@ rte_pci_get_iommu_class(void)
bool devices_want_pa = false;
int iommu_no_va = -1;
- RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus) {
/*
* We can check this only once, because the IOMMU hardware is
* the same for all of them.
@@ -594,7 +594,7 @@ rte_pci_get_iommu_class(void)
if (dev->kdrv == RTE_PCI_KDRV_UNKNOWN ||
dev->kdrv == RTE_PCI_KDRV_NONE)
continue;
- RTE_BUS_FOREACH_DRV(drv, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DRV(drv, &rte_pci_bus) {
enum rte_iova_mode dev_iova_mode;
if (!pci_bus_match(&drv->driver, &dev->device))
@@ -772,27 +772,25 @@ rte_pci_pasid_set_state(const struct rte_pci_device *dev,
offset + RTE_PCI_PASID_CTRL) != sizeof(pasid) ? -1 : 0;
}
-struct rte_pci_bus rte_pci_bus = {
- .bus = {
- .allow_multi_probe = true,
- .scan = rte_pci_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = pci_cleanup,
- .find_device = rte_bus_generic_find_device,
- .match = pci_bus_match,
- .probe_device = pci_probe_device,
- .unplug = pci_unplug,
- .parse = pci_parse,
- .dev_compare = pci_dev_compare,
- .devargs_parse = rte_pci_devargs_parse,
- .dma_map = pci_dma_map,
- .dma_unmap = pci_dma_unmap,
- .get_iommu_class = rte_pci_get_iommu_class,
- .dev_iterate = rte_pci_dev_iterate,
- .hot_unplug_handler = pci_hot_unplug_handler,
- .sigbus_handler = pci_sigbus_handler,
- },
+struct rte_bus rte_pci_bus = {
+ .allow_multi_probe = true,
+ .scan = rte_pci_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = pci_cleanup,
+ .find_device = rte_bus_generic_find_device,
+ .match = pci_bus_match,
+ .probe_device = pci_probe_device,
+ .unplug = pci_unplug,
+ .parse = pci_parse,
+ .dev_compare = pci_dev_compare,
+ .devargs_parse = rte_pci_devargs_parse,
+ .dma_map = pci_dma_map,
+ .dma_unmap = pci_dma_unmap,
+ .get_iommu_class = rte_pci_get_iommu_class,
+ .dev_iterate = rte_pci_dev_iterate,
+ .hot_unplug_handler = pci_hot_unplug_handler,
+ .sigbus_handler = pci_sigbus_handler,
};
-RTE_REGISTER_BUS(pci, rte_pci_bus.bus);
+RTE_REGISTER_BUS(pci, rte_pci_bus);
RTE_LOG_REGISTER_DEFAULT(pci_bus_logtype, NOTICE);
diff --git a/drivers/bus/pci/pci_params.c b/drivers/bus/pci/pci_params.c
index e308c85ed2..6cbd98b4c8 100644
--- a/drivers/bus/pci/pci_params.c
+++ b/drivers/bus/pci/pci_params.c
@@ -75,7 +75,7 @@ rte_pci_dev_iterate(const struct rte_bus *bus __rte_unused,
return NULL;
}
}
- dev = rte_bus_generic_find_device(&rte_pci_bus.bus, start, pci_dev_match, kvargs);
+ dev = rte_bus_generic_find_device(&rte_pci_bus, start, pci_dev_match, kvargs);
rte_kvargs_free(kvargs);
return dev;
}
diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
index c54ea7b9d8..8103c32881 100644
--- a/drivers/bus/pci/private.h
+++ b/drivers/bus/pci/private.h
@@ -29,14 +29,7 @@ extern int pci_bus_logtype;
#define RTE_PCI_DEVICE_INTERNAL_CONST(ptr) \
container_of(ptr, const struct rte_pci_device_internal, device)
-/**
- * Structure describing the PCI bus
- */
-struct rte_pci_bus {
- struct rte_bus bus; /**< Inherit the generic class */
-};
-
-extern struct rte_pci_bus rte_pci_bus;
+extern struct rte_bus rte_pci_bus;
struct rte_pci_driver;
struct rte_pci_device;
diff --git a/drivers/bus/pci/windows/pci.c b/drivers/bus/pci/windows/pci.c
index 3b3f97da27..7b51301d1e 100644
--- a/drivers/bus/pci/windows/pci.c
+++ b/drivers/bus/pci/windows/pci.c
@@ -382,7 +382,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data)
goto end;
rte_pci_device_name(&addr, name, sizeof(name));
- if (rte_bus_device_is_ignored(&rte_pci_bus.bus, name)) {
+ if (rte_bus_device_is_ignored(&rte_pci_bus, name)) {
/*
* We won't add this device, but we want to continue
* looking for supported devices
@@ -430,18 +430,17 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data)
}
/* device is valid, add in list (sorted) */
- if (TAILQ_EMPTY(&rte_pci_bus.bus.device_list)) {
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ if (TAILQ_EMPTY(&rte_pci_bus.device_list)) {
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
} else {
struct rte_pci_device *dev2 = NULL;
- RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &rte_pci_bus) {
ret = rte_pci_addr_cmp(&dev->addr, &dev2->addr);
if (ret > 0) {
continue;
} else if (ret < 0) {
- rte_bus_insert_device(&rte_pci_bus.bus, &dev2->device,
- &dev->device);
+ rte_bus_insert_device(&rte_pci_bus, &dev2->device, &dev->device);
} else { /* already registered */
dev2->kdrv = dev->kdrv;
dev2->max_vfs = dev->max_vfs;
@@ -451,7 +450,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data)
}
return 0;
}
- rte_bus_add_device(&rte_pci_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_pci_bus, &dev->device);
}
return 0;
diff --git a/drivers/bus/platform/bus_platform_driver.h b/drivers/bus/platform/bus_platform_driver.h
index 3ac405a201..e4dcbacf5e 100644
--- a/drivers/bus/platform/bus_platform_driver.h
+++ b/drivers/bus/platform/bus_platform_driver.h
@@ -24,7 +24,6 @@ extern "C" {
#endif
/* Forward declarations */
-struct rte_platform_bus;
struct rte_platform_device;
struct rte_platform_driver;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 4492ed62ec..170a2e03d0 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -29,18 +29,20 @@
#define PLATFORM_BUS_DEVICES_PATH "/sys/bus/platform/devices"
+static struct rte_bus platform_bus;
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_platform_register)
void
rte_platform_register(struct rte_platform_driver *pdrv)
{
- rte_bus_add_driver(&platform_bus.bus, &pdrv->driver);
+ rte_bus_add_driver(&platform_bus, &pdrv->driver);
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_platform_unregister)
void
rte_platform_unregister(struct rte_platform_driver *pdrv)
{
- rte_bus_remove_driver(&platform_bus.bus, &pdrv->driver);
+ rte_bus_remove_driver(&platform_bus, &pdrv->driver);
}
static int
@@ -56,11 +58,11 @@ dev_add(const char *dev_name)
rte_strscpy(pdev->name, dev_name, sizeof(pdev->name));
pdev->device.name = pdev->name;
- pdev->device.devargs = rte_bus_find_devargs(&platform_bus.bus, dev_name);
+ pdev->device.devargs = rte_bus_find_devargs(&platform_bus, dev_name);
snprintf(path, sizeof(path), PLATFORM_BUS_DEVICES_PATH "/%s/numa_node", dev_name);
pdev->device.numa_node = eal_parse_sysfs_value(path, &val) ? rte_socket_id() : val;
- RTE_BUS_FOREACH_DEV(tmp, &platform_bus.bus) {
+ RTE_BUS_FOREACH_DEV(tmp, &platform_bus) {
if (!strcmp(tmp->name, pdev->name)) {
PLATFORM_LOG_LINE(INFO, "device %s already added", pdev->name);
@@ -72,7 +74,7 @@ dev_add(const char *dev_name)
}
}
- rte_bus_add_device(&platform_bus.bus, &pdev->device);
+ rte_bus_add_device(&platform_bus, &pdev->device);
PLATFORM_LOG_LINE(INFO, "adding device %s to the list", dev_name);
@@ -135,7 +137,7 @@ platform_bus_scan(void)
if (dev_name[0] == '.')
continue;
- if (rte_bus_device_is_ignored(&platform_bus.bus, dev_name))
+ if (rte_bus_device_is_ignored(&platform_bus, dev_name))
continue;
if (!dev_is_bound_vfio_platform(dev_name))
@@ -440,7 +442,7 @@ platform_bus_parse(const char *name, void *addr)
rte_strscpy(pdev.name, name, sizeof(pdev.name));
- RTE_BUS_FOREACH_DRV(pdrv, &platform_bus.bus) {
+ RTE_BUS_FOREACH_DRV(pdrv, &platform_bus) {
if (platform_bus_match(&pdrv->driver, &pdev.device))
break;
}
@@ -482,7 +484,7 @@ platform_bus_get_iommu_class(void)
const struct rte_platform_driver *pdrv;
struct rte_platform_device *pdev;
- RTE_BUS_FOREACH_DEV(pdev, &platform_bus.bus) {
+ RTE_BUS_FOREACH_DEV(pdev, &platform_bus) {
if (!rte_dev_is_probed(&pdev->device))
continue;
pdrv = RTE_BUS_DRIVER(pdev->device.driver, *pdrv);
@@ -498,8 +500,8 @@ platform_bus_cleanup(void)
{
struct rte_platform_device *pdev;
- RTE_BUS_FOREACH_DEV(pdev, &platform_bus.bus) {
- rte_bus_remove_device(&platform_bus.bus, &pdev->device);
+ RTE_BUS_FOREACH_DEV(pdev, &platform_bus) {
+ rte_bus_remove_device(&platform_bus, &pdev->device);
if (!rte_dev_is_probed(&pdev->device))
continue;
platform_bus_unplug(&pdev->device);
@@ -508,22 +510,20 @@ platform_bus_cleanup(void)
return 0;
}
-struct rte_platform_bus platform_bus = {
- .bus = {
- .scan = platform_bus_scan,
- .probe = rte_bus_generic_probe,
- .find_device = rte_bus_generic_find_device,
- .match = platform_bus_match,
- .probe_device = platform_bus_probe_device,
- .unplug = platform_bus_unplug,
- .parse = platform_bus_parse,
- .dma_map = platform_bus_dma_map,
- .dma_unmap = platform_bus_dma_unmap,
- .get_iommu_class = platform_bus_get_iommu_class,
- .dev_iterate = rte_bus_generic_dev_iterate,
- .cleanup = platform_bus_cleanup,
- },
+static struct rte_bus platform_bus = {
+ .scan = platform_bus_scan,
+ .probe = rte_bus_generic_probe,
+ .find_device = rte_bus_generic_find_device,
+ .match = platform_bus_match,
+ .probe_device = platform_bus_probe_device,
+ .unplug = platform_bus_unplug,
+ .parse = platform_bus_parse,
+ .dma_map = platform_bus_dma_map,
+ .dma_unmap = platform_bus_dma_unmap,
+ .get_iommu_class = platform_bus_get_iommu_class,
+ .dev_iterate = rte_bus_generic_dev_iterate,
+ .cleanup = platform_bus_cleanup,
};
-RTE_REGISTER_BUS(platform, platform_bus.bus);
+RTE_REGISTER_BUS(platform, platform_bus);
RTE_LOG_REGISTER_DEFAULT(platform_bus_logtype, NOTICE);
diff --git a/drivers/bus/platform/private.h b/drivers/bus/platform/private.h
index bf5d75df03..18d42d43d8 100644
--- a/drivers/bus/platform/private.h
+++ b/drivers/bus/platform/private.h
@@ -14,15 +14,6 @@
#include "bus_platform_driver.h"
-extern struct rte_platform_bus platform_bus;
-
-/*
- * Structure describing platform bus.
- */
-struct rte_platform_bus {
- struct rte_bus bus; /* Core bus */
-};
-
extern int platform_bus_logtype;
#define RTE_LOGTYPE_PLATFORM_BUS platform_bus_logtype
#define PLATFORM_LOG_LINE(level, ...) \
diff --git a/drivers/bus/uacce/uacce.c b/drivers/bus/uacce/uacce.c
index db49e887ad..eb58701a61 100644
--- a/drivers/bus/uacce/uacce.c
+++ b/drivers/bus/uacce/uacce.c
@@ -34,15 +34,8 @@
/* Support -a uacce:device-name when start DPDK application. */
#define UACCE_DEV_PREFIX "uacce:"
-/*
- * Structure describing the UACCE bus.
- */
-struct rte_uacce_bus {
- struct rte_bus bus; /* Inherit the generic class. */
-};
-
/* Forward declaration of UACCE bus. */
-static struct rte_uacce_bus uacce_bus;
+static struct rte_bus uacce_bus;
extern int uacce_bus_logtype;
@@ -229,7 +222,7 @@ uacce_scan_one(const char *dev_name)
return -ENOMEM;
dev->device.name = dev->name;
- dev->device.devargs = rte_bus_find_devargs(&uacce_bus.bus, dev_name);
+ dev->device.devargs = rte_bus_find_devargs(&uacce_bus, dev_name);
snprintf(dev->name, sizeof(dev->name), "%s", dev_name);
snprintf(dev->dev_root, sizeof(dev->dev_root), "%s/%s",
UACCE_BUS_CLASS_PATH, dev_name);
@@ -253,7 +246,7 @@ uacce_scan_one(const char *dev_name)
if (ret != 0)
goto err;
- rte_bus_add_device(&uacce_bus.bus, &dev->device);
+ rte_bus_add_device(&uacce_bus, &dev->device);
return 0;
err:
@@ -283,7 +276,7 @@ uacce_scan(void)
continue;
}
- if (rte_bus_device_is_ignored(&uacce_bus.bus, e->d_name))
+ if (rte_bus_device_is_ignored(&uacce_bus, e->d_name))
continue;
if (uacce_scan_one(e->d_name) < 0)
@@ -379,7 +372,7 @@ uacce_cleanup(void)
struct rte_uacce_device *dev;
int error = 0;
- RTE_BUS_FOREACH_DEV(dev, &uacce_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &uacce_bus) {
const struct rte_uacce_driver *dr;
int ret = 0;
@@ -397,7 +390,7 @@ uacce_cleanup(void)
dev->device.driver = NULL;
free:
- rte_bus_remove_device(&uacce_bus.bus, &dev->device);
+ rte_bus_remove_device(&uacce_bus, &dev->device);
free(dev);
}
@@ -431,7 +424,7 @@ uacce_unplug(struct rte_device *dev)
ret = uacce_detach_dev(uacce_dev);
if (ret == 0) {
- rte_bus_remove_device(&uacce_bus.bus, &uacce_dev->device);
+ rte_bus_remove_device(&uacce_bus, &uacce_dev->device);
rte_devargs_remove(dev->devargs);
free(uacce_dev);
}
@@ -550,29 +543,27 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_uacce_register)
void
rte_uacce_register(struct rte_uacce_driver *driver)
{
- rte_bus_add_driver(&uacce_bus.bus, &driver->driver);
+ rte_bus_add_driver(&uacce_bus, &driver->driver);
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_uacce_unregister)
void
rte_uacce_unregister(struct rte_uacce_driver *driver)
{
- rte_bus_remove_driver(&uacce_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&uacce_bus, &driver->driver);
}
-static struct rte_uacce_bus uacce_bus = {
- .bus = {
- .scan = uacce_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = uacce_cleanup,
- .match = uacce_bus_match,
- .probe_device = uacce_probe_device,
- .unplug = uacce_unplug,
- .find_device = rte_bus_generic_find_device,
- .parse = uacce_parse,
- .dev_iterate = rte_bus_generic_dev_iterate,
- },
+static struct rte_bus uacce_bus = {
+ .scan = uacce_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = uacce_cleanup,
+ .match = uacce_bus_match,
+ .probe_device = uacce_probe_device,
+ .unplug = uacce_unplug,
+ .find_device = rte_bus_generic_find_device,
+ .parse = uacce_parse,
+ .dev_iterate = rte_bus_generic_dev_iterate,
};
-RTE_REGISTER_BUS(uacce, uacce_bus.bus);
+RTE_REGISTER_BUS(uacce, uacce_bus);
RTE_LOG_REGISTER_DEFAULT(uacce_bus_logtype, NOTICE);
diff --git a/drivers/bus/vmbus/linux/vmbus_bus.c b/drivers/bus/vmbus/linux/vmbus_bus.c
index 6268a14d40..0af10f6a69 100644
--- a/drivers/bus/vmbus/linux/vmbus_bus.c
+++ b/drivers/bus/vmbus/linux/vmbus_bus.c
@@ -39,8 +39,6 @@ static const rte_uuid_t vmbus_nic_uuid = {
0xf2, 0xd2, 0xf9, 0x65, 0xed, 0xe
};
-extern struct rte_vmbus_bus rte_vmbus_bus;
-
/* Read sysfs file to get UUID */
static int
parse_sysfs_uuid(const char *filename, rte_uuid_t uu)
@@ -332,7 +330,7 @@ vmbus_scan_one(const char *name)
dev->monitor_id = UINT8_MAX;
}
- dev->device.devargs = rte_bus_find_devargs(&rte_vmbus_bus.bus, dev_name);
+ dev->device.devargs = rte_bus_find_devargs(&rte_vmbus_bus, dev_name);
dev->device.numa_node = SOCKET_ID_ANY;
if (vmbus_use_numa(dev)) {
@@ -356,7 +354,7 @@ vmbus_scan_one(const char *name)
/* device is valid, add in list (sorted) */
VMBUS_LOG(DEBUG, "Adding vmbus device %s", name);
- RTE_BUS_FOREACH_DEV(dev2, &rte_vmbus_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev2, &rte_vmbus_bus) {
int ret;
ret = rte_uuid_compare(dev->device_id, dev2->device_id);
@@ -364,7 +362,7 @@ vmbus_scan_one(const char *name)
continue;
if (ret < 0) {
- rte_bus_insert_device(&rte_vmbus_bus.bus, &dev2->device, &dev->device);
+ rte_bus_insert_device(&rte_vmbus_bus, &dev2->device, &dev->device);
} else { /* already registered */
VMBUS_LOG(NOTICE,
"%s already registered", name);
@@ -374,7 +372,7 @@ vmbus_scan_one(const char *name)
return 0;
}
- rte_bus_add_device(&rte_vmbus_bus.bus, &dev->device);
+ rte_bus_add_device(&rte_vmbus_bus, &dev->device);
return 0;
error:
VMBUS_LOG(DEBUG, "failed");
diff --git a/drivers/bus/vmbus/private.h b/drivers/bus/vmbus/private.h
index 6abb97c607..6efac86b77 100644
--- a/drivers/bus/vmbus/private.h
+++ b/drivers/bus/vmbus/private.h
@@ -15,14 +15,7 @@
#include <rte_eal_paging.h>
#include <rte_vmbus_reg.h>
-/**
- * Structure describing the VM bus
- */
-struct rte_vmbus_bus {
- struct rte_bus bus; /**< Inherit the generic class */
-};
-
-extern struct rte_vmbus_bus rte_vmbus_bus;
+extern struct rte_bus rte_vmbus_bus;
extern int vmbus_logtype_bus;
#define RTE_LOGTYPE_VMBUS_BUS vmbus_logtype_bus
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 2b1730afc2..01573927ce 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -23,8 +23,6 @@
#include "private.h"
-extern struct rte_vmbus_bus rte_vmbus_bus;
-
/* map a particular resource from a file */
void *
vmbus_map_resource(void *requested_addr, int fd, off_t offset, size_t size,
@@ -128,7 +126,7 @@ RTE_EXPORT_SYMBOL(rte_vmbus_probe)
int
rte_vmbus_probe(void)
{
- return rte_bus_generic_probe(&rte_vmbus_bus.bus);
+ return rte_bus_generic_probe(&rte_vmbus_bus);
}
static int
@@ -137,7 +135,7 @@ rte_vmbus_cleanup(void)
struct rte_vmbus_device *dev;
int error = 0;
- RTE_BUS_FOREACH_DEV(dev, &rte_vmbus_bus.bus) {
+ RTE_BUS_FOREACH_DEV(dev, &rte_vmbus_bus) {
const struct rte_vmbus_driver *drv;
int ret;
@@ -154,7 +152,7 @@ rte_vmbus_cleanup(void)
rte_vmbus_unmap_device(dev);
dev->device.driver = NULL;
- rte_bus_remove_device(&rte_vmbus_bus.bus, &dev->device);
+ rte_bus_remove_device(&rte_vmbus_bus, &dev->device);
free(dev);
}
@@ -194,7 +192,7 @@ rte_vmbus_register(struct rte_vmbus_driver *driver)
VMBUS_LOG(DEBUG,
"Registered driver %s", driver->driver.name);
- rte_bus_add_driver(&rte_vmbus_bus.bus, &driver->driver);
+ rte_bus_add_driver(&rte_vmbus_bus, &driver->driver);
}
/* unregister vmbus driver */
@@ -202,22 +200,20 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vmbus_unregister)
void
rte_vmbus_unregister(struct rte_vmbus_driver *driver)
{
- rte_bus_remove_driver(&rte_vmbus_bus.bus, &driver->driver);
+ rte_bus_remove_driver(&rte_vmbus_bus, &driver->driver);
}
/* VMBUS doesn't support hotplug */
-struct rte_vmbus_bus rte_vmbus_bus = {
- .bus = {
- .scan = rte_vmbus_scan,
- .probe = rte_bus_generic_probe,
- .cleanup = rte_vmbus_cleanup,
- .find_device = rte_bus_generic_find_device,
- .match = vmbus_bus_match,
- .probe_device = vmbus_probe_device,
- .parse = vmbus_parse,
- .dev_compare = vmbus_dev_compare,
- },
+struct rte_bus rte_vmbus_bus = {
+ .scan = rte_vmbus_scan,
+ .probe = rte_bus_generic_probe,
+ .cleanup = rte_vmbus_cleanup,
+ .find_device = rte_bus_generic_find_device,
+ .match = vmbus_bus_match,
+ .probe_device = vmbus_probe_device,
+ .parse = vmbus_parse,
+ .dev_compare = vmbus_dev_compare,
};
-RTE_REGISTER_BUS(vmbus, rte_vmbus_bus.bus);
+RTE_REGISTER_BUS(vmbus, rte_vmbus_bus);
RTE_LOG_REGISTER_DEFAULT(vmbus_logtype_bus, NOTICE);
--
2.53.0
^ permalink raw reply related
* [PATCH v3 22/25] drivers/bus: separate specific bus metadata for NXP drivers
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev; +Cc: thomas, stephen, bruce.richardson, Hemant Agrawal, Sachin Saxena
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
As a preparation for the next commit, split the specific bus types and
move specific fields to a private structure/singleton variable.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/dpaa/dpaa_bus.c | 55 ++++++++++++++++++++---------------
drivers/bus/fslmc/fslmc_bus.c | 6 ++--
drivers/bus/fslmc/private.h | 2 --
3 files changed, 34 insertions(+), 29 deletions(-)
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index ade093cd13..08c1607647 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -59,6 +59,9 @@
struct rte_dpaa_bus {
struct rte_bus bus;
+};
+
+struct rte_dpaa_bus_private {
int device_count;
int detected;
uint32_t svr_ver;
@@ -67,6 +70,7 @@ struct rte_dpaa_bus {
};
static struct rte_dpaa_bus rte_dpaa_bus;
+static struct rte_dpaa_bus_private dpaa_bus;
static struct netcfg_info *dpaa_netcfg;
/* define a variable to hold the portal_key, once created.*/
@@ -87,7 +91,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(dpaa_get_eth_port_cfg)
RTE_EXPORT_INTERNAL_SYMBOL(dpaa_soc_ver)
uint32_t dpaa_soc_ver(void)
{
- return rte_dpaa_bus.svr_ver;
+ return dpaa_bus.svr_ver;
}
struct fm_eth_port_cfg *
@@ -103,11 +107,11 @@ dpaa_push_queue_num_update(void)
int ret = false;
uint16_t current, new_val;
- current = rte_atomic_load_explicit(&rte_dpaa_bus.push_rxq_num,
+ current = rte_atomic_load_explicit(&dpaa_bus.push_rxq_num,
rte_memory_order_acquire);
- if (current < rte_dpaa_bus.max_push_rxq_num) {
+ if (current < dpaa_bus.max_push_rxq_num) {
new_val = current + 1;
- if (rte_atomic_compare_exchange_strong_explicit(&rte_dpaa_bus.push_rxq_num,
+ if (rte_atomic_compare_exchange_strong_explicit(&dpaa_bus.push_rxq_num,
¤t, new_val,
rte_memory_order_release,
rte_memory_order_acquire))
@@ -121,7 +125,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(dpaa_push_queue_max_num)
uint16_t
dpaa_push_queue_max_num(void)
{
- return rte_dpaa_bus.max_push_rxq_num;
+ return dpaa_bus.max_push_rxq_num;
}
static int
@@ -204,7 +208,7 @@ dpaa_create_device_list(void)
struct fm_eth_port_cfg *cfg;
struct fman_if *fman_intf;
- rte_dpaa_bus.device_count = 0;
+ dpaa_bus.device_count = 0;
/* Creating Ethernet Devices */
for (i = 0; dpaa_netcfg && (i < dpaa_netcfg->num_ethports); i++) {
@@ -256,7 +260,7 @@ dpaa_create_device_list(void)
dpaa_add_to_device_list(dev);
}
- rte_dpaa_bus.device_count += i;
+ dpaa_bus.device_count += i;
/* Unlike case of ETH, RTE_LIBRTE_DPAA_MAX_CRYPTODEV SEC devices are
* constantly created only if "sec" property is found in the device
@@ -289,7 +293,7 @@ dpaa_create_device_list(void)
}
dev->device_type = FSL_DPAA_CRYPTO;
- dev->id.dev_id = rte_dpaa_bus.device_count + i;
+ dev->id.dev_id = dpaa_bus.device_count + i;
/* Even though RTE_CRYPTODEV_NAME_MAX_LEN is valid length of
* crypto PMD, using RTE_ETH_NAME_MAX_LEN as that is the size
@@ -306,7 +310,7 @@ dpaa_create_device_list(void)
dpaa_add_to_device_list(dev);
}
- rte_dpaa_bus.device_count += i;
+ dpaa_bus.device_count += i;
qdma_dpaa:
/* Creating QDMA Device */
@@ -319,7 +323,7 @@ dpaa_create_device_list(void)
}
dev->device_type = FSL_DPAA_QDMA;
- dev->id.dev_id = rte_dpaa_bus.device_count + i;
+ dev->id.dev_id = dpaa_bus.device_count + i;
memset(dev->name, 0, RTE_ETH_NAME_MAX_LEN);
sprintf(dev->name, "dpaa_qdma-%d", i+1);
@@ -331,7 +335,7 @@ dpaa_create_device_list(void)
dpaa_add_to_device_list(dev);
}
- rte_dpaa_bus.device_count += i;
+ dpaa_bus.device_count += i;
return 0;
@@ -688,10 +692,10 @@ rte_dpaa_bus_scan(void)
return 0;
}
- if (rte_dpaa_bus.detected)
+ if (dpaa_bus.detected)
return 0;
- rte_dpaa_bus.detected = 1;
+ dpaa_bus.detected = 1;
/* create the key, supplying a function that'll be invoked
* when a portal affined thread will be deleted.
@@ -707,34 +711,34 @@ rte_dpaa_bus_scan(void)
svr_file = fopen(DPAA_SOC_ID_FILE, "r");
if (svr_file) {
if (fscanf(svr_file, "svr:%x", &svr_ver) > 0)
- rte_dpaa_bus.svr_ver = svr_ver & DPAA_SVR_MASK;
+ dpaa_bus.svr_ver = svr_ver & DPAA_SVR_MASK;
else
- rte_dpaa_bus.svr_ver = 0;
+ dpaa_bus.svr_ver = 0;
fclose(svr_file);
} else {
- rte_dpaa_bus.svr_ver = 0;
+ dpaa_bus.svr_ver = 0;
}
- if (rte_dpaa_bus.svr_ver == SVR_LS1046A_FAMILY) {
+ if (dpaa_bus.svr_ver == SVR_LS1046A_FAMILY) {
DPAA_BUS_LOG(INFO, "This is LS1046A family SoC.");
- } else if (rte_dpaa_bus.svr_ver == SVR_LS1043A_FAMILY) {
+ } else if (dpaa_bus.svr_ver == SVR_LS1043A_FAMILY) {
DPAA_BUS_LOG(INFO, "This is LS1043A family SoC.");
} else {
DPAA_BUS_LOG(WARNING,
"This is Unknown(%08x) DPAA1 family SoC.",
- rte_dpaa_bus.svr_ver);
+ dpaa_bus.svr_ver);
}
/* Disabling the default push mode for LS1043A */
- if (rte_dpaa_bus.svr_ver == SVR_LS1043A_FAMILY) {
- rte_dpaa_bus.max_push_rxq_num = 0;
+ if (dpaa_bus.svr_ver == SVR_LS1043A_FAMILY) {
+ dpaa_bus.max_push_rxq_num = 0;
return 0;
}
penv = getenv("DPAA_PUSH_QUEUES_NUMBER");
if (penv)
- rte_dpaa_bus.max_push_rxq_num = atoi(penv);
- if (rte_dpaa_bus.max_push_rxq_num > DPAA_MAX_PUSH_MODE_QUEUE)
- rte_dpaa_bus.max_push_rxq_num = DPAA_MAX_PUSH_MODE_QUEUE;
+ dpaa_bus.max_push_rxq_num = atoi(penv);
+ if (dpaa_bus.max_push_rxq_num > DPAA_MAX_PUSH_MODE_QUEUE)
+ dpaa_bus.max_push_rxq_num = DPAA_MAX_PUSH_MODE_QUEUE;
/* Device list creation is only done once */
if (!process_once) {
@@ -868,6 +872,9 @@ static struct rte_dpaa_bus rte_dpaa_bus = {
.dev_iterate = rte_bus_generic_dev_iterate,
.cleanup = dpaa_bus_cleanup,
},
+};
+
+static struct rte_dpaa_bus_private dpaa_bus = {
.max_push_rxq_num = DPAA_DEFAULT_PUSH_MODE_QUEUE,
.device_count = 0,
};
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index d855d6b36b..821fe63955 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -28,6 +28,7 @@
#define VFIO_IOMMU_GROUP_PATH "/sys/kernel/iommu_groups"
struct rte_fslmc_bus rte_fslmc_bus;
+static int fslmc_bus_device_count[DPAA2_DEVTYPE_MAX];
#define DPAA2_SEQN_DYNFIELD_NAME "dpaa2_seqn_dynfield"
RTE_EXPORT_INTERNAL_SYMBOL(dpaa2_seqn_dynfield_offset)
@@ -39,7 +40,7 @@ rte_fslmc_get_device_count(enum rte_dpaa2_dev_type device_type)
{
if (device_type >= DPAA2_DEVTYPE_MAX)
return 0;
- return rte_fslmc_bus.device_count[device_type];
+ return fslmc_bus_device_count[device_type];
}
static void
@@ -200,7 +201,7 @@ scan_one_fslmc_device(char *dev_name)
dev->device.devargs = rte_bus_find_devargs(&rte_fslmc_bus.bus, dev_name);
/* Update the device found into the device_count table */
- rte_fslmc_bus.device_count[dev->dev_type]++;
+ fslmc_bus_device_count[dev->dev_type]++;
/* Add device in the fslmc device list */
insert_in_device_list(dev);
@@ -559,7 +560,6 @@ struct rte_fslmc_bus rte_fslmc_bus = {
.unplug = fslmc_bus_unplug,
.dev_iterate = rte_bus_generic_dev_iterate,
},
- .device_count = {0},
};
RTE_REGISTER_BUS(fslmc, rte_fslmc_bus.bus);
diff --git a/drivers/bus/fslmc/private.h b/drivers/bus/fslmc/private.h
index 2fe592f24d..a0dda4f9d6 100644
--- a/drivers/bus/fslmc/private.h
+++ b/drivers/bus/fslmc/private.h
@@ -14,8 +14,6 @@
*/
struct rte_fslmc_bus {
struct rte_bus bus; /**< Generic Bus object */
- int device_count[DPAA2_DEVTYPE_MAX];
- /**< Count of all devices scanned */
};
extern struct rte_fslmc_bus rte_fslmc_bus;
--
2.53.0
^ permalink raw reply related
* [PATCH v3 21/25] dma/idxd: remove specific bus type
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev; +Cc: thomas, stephen, bruce.richardson, Kevin Laatz
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
There is nothing that requires a specific bus type.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/dma/idxd/idxd_bus.c | 38 ++++++++++++++-----------------------
1 file changed, 14 insertions(+), 24 deletions(-)
diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c
index 1a9c07580e..4810d52f2a 100644
--- a/drivers/dma/idxd/idxd_bus.c
+++ b/drivers/dma/idxd/idxd_bus.c
@@ -41,34 +41,24 @@ struct rte_dsa_device {
};
/* forward prototypes */
-struct dsa_bus;
static int dsa_scan(void);
static bool dsa_match(const struct rte_driver *drv, const struct rte_device *dev);
static int dsa_probe_device(struct rte_driver *drv, struct rte_device *dev);
static enum rte_iova_mode dsa_get_iommu_class(void);
static int dsa_addr_parse(const char *name, void *addr);
-/**
- * Structure describing the DSA bus
- */
-struct dsa_bus {
- struct rte_bus bus; /**< Inherit the generic class */
- struct rte_driver driver; /**< Driver struct for devices to point to */
+struct rte_bus dsa_bus = {
+ .scan = dsa_scan,
+ .probe = rte_bus_generic_probe,
+ .match = dsa_match,
+ .probe_device = dsa_probe_device,
+ .find_device = rte_bus_generic_find_device,
+ .get_iommu_class = dsa_get_iommu_class,
+ .parse = dsa_addr_parse,
};
-struct dsa_bus dsa_bus = {
- .bus = {
- .scan = dsa_scan,
- .probe = rte_bus_generic_probe,
- .match = dsa_match,
- .probe_device = dsa_probe_device,
- .find_device = rte_bus_generic_find_device,
- .get_iommu_class = dsa_get_iommu_class,
- .parse = dsa_addr_parse,
- },
- .driver = {
- .name = "dmadev_idxd",
- },
+struct rte_driver dsa_driver = {
+ .name = "dmadev_idxd",
};
static inline const char *
@@ -267,7 +257,7 @@ static bool dsa_match(const struct rte_driver *drv, const struct rte_device *dev
{
const struct rte_dsa_device *dsa_dev = RTE_BUS_DEVICE(dev, *dsa_dev);
- if (drv == &dsa_bus.driver) {
+ if (drv == &dsa_driver) {
char type[64], name[64];
if (read_wq_string(dsa_dev, "type", type, sizeof(type)) >= 0 &&
@@ -319,7 +309,7 @@ dsa_scan(void)
continue;
}
strlcpy(dev->wq_name, wq->d_name, sizeof(dev->wq_name));
- rte_bus_add_device(&dsa_bus.bus, &dev->device);
+ rte_bus_add_device(&dsa_bus, &dev->device);
devcount++;
read_device_int(dev, "numa_node", &numa_node);
@@ -357,8 +347,8 @@ dsa_addr_parse(const char *name, void *addr)
return 0;
}
-RTE_REGISTER_BUS(dsa, dsa_bus.bus);
+RTE_REGISTER_BUS(dsa, dsa_bus);
RTE_INIT(dsa_bus_init)
{
- rte_bus_add_driver(&dsa_bus.bus, &dsa_bus.driver);
+ rte_bus_add_driver(&dsa_bus, &dsa_driver);
}
--
2.53.0
^ permalink raw reply related
* [PATCH v3 20/25] drivers/bus: remove bus-specific driver references
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Rosen Xu, Chenbo Xia, Tomasz Duszynski, Chengwen Feng, Long Li,
Wei Hu
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
With the driver reference being set in the EAL before calling
bus->probe_device, buses now rely on the generic device.driver
field instead of maintaining their own bus-specific driver pointers.
This patch removes the use of bus-specific driver fields in favor
of the generic device.driver field, accessed through the RTE_BUS_DRIVER
macro in remove/cleanup paths.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/auxiliary/auxiliary_common.c | 4 ----
drivers/bus/auxiliary/bus_auxiliary_driver.h | 1 -
drivers/bus/cdx/bus_cdx_driver.h | 1 -
drivers/bus/cdx/cdx.c | 2 --
drivers/bus/dpaa/bus_dpaa_driver.h | 1 -
drivers/bus/dpaa/dpaa_bus.c | 3 ---
drivers/bus/fslmc/bus_fslmc_driver.h | 1 -
drivers/bus/fslmc/fslmc_bus.c | 2 --
drivers/bus/ifpga/bus_ifpga_driver.h | 1 -
drivers/bus/ifpga/ifpga_bus.c | 12 +-----------
drivers/bus/pci/bus_pci_driver.h | 1 -
drivers/bus/pci/pci_common.c | 5 -----
drivers/bus/platform/bus_platform_driver.h | 1 -
drivers/bus/platform/platform.c | 5 +----
drivers/bus/uacce/bus_uacce_driver.h | 1 -
drivers/bus/uacce/uacce.c | 3 ---
drivers/bus/vmbus/bus_vmbus_driver.h | 1 -
drivers/bus/vmbus/vmbus_common.c | 8 +-------
18 files changed, 3 insertions(+), 50 deletions(-)
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index 2000ffa369..ff05716539 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -111,13 +111,10 @@ auxiliary_probe_device(struct rte_driver *drv, struct rte_device *dev)
return -ENOMEM;
}
- aux_dev->driver = aux_drv;
-
AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
aux_drv->driver.name, aux_dev->name, aux_dev->device.numa_node);
ret = aux_drv->probe(aux_drv, aux_dev);
if (ret != 0) {
- aux_dev->driver = NULL;
rte_intr_instance_free(aux_dev->intr_handle);
aux_dev->intr_handle = NULL;
}
@@ -144,7 +141,6 @@ rte_auxiliary_driver_remove_dev(struct rte_auxiliary_device *dev)
}
/* clear driver structure */
- dev->driver = NULL;
dev->device.driver = NULL;
return 0;
diff --git a/drivers/bus/auxiliary/bus_auxiliary_driver.h b/drivers/bus/auxiliary/bus_auxiliary_driver.h
index 165145b15e..cab5f86d03 100644
--- a/drivers/bus/auxiliary/bus_auxiliary_driver.h
+++ b/drivers/bus/auxiliary/bus_auxiliary_driver.h
@@ -113,7 +113,6 @@ struct rte_auxiliary_device {
struct rte_device device; /**< Inherit core device */
char name[RTE_DEV_NAME_MAX_LEN + 1]; /**< ASCII device name */
struct rte_intr_handle *intr_handle; /**< Interrupt handle */
- struct rte_auxiliary_driver *driver; /**< Device driver */
};
/**
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index aa62ab5730..aa44dee5a6 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -54,7 +54,6 @@ struct rte_cdx_id {
*/
struct rte_cdx_device {
struct rte_device device; /**< Inherit core device */
- struct rte_cdx_driver *driver; /**< CDX driver used in probing */
char name[RTE_DEV_NAME_MAX_LEN]; /**< Device name */
struct rte_cdx_id id; /**< CDX ID. */
struct rte_mem_resource mem_resource[RTE_CDX_MAX_RESOURCE];
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 5a823a0dcd..cb4f755b8e 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -333,7 +333,6 @@ cdx_probe_device(struct rte_driver *drv, struct rte_device *dev)
cdx_drv->driver.name, dev_name, ret);
goto error_probe;
}
- cdx_dev->driver = cdx_drv;
return ret;
@@ -395,7 +394,6 @@ cdx_detach_dev(struct rte_cdx_device *dev)
}
/* clear driver structure */
- dev->driver = NULL;
dev->device.driver = NULL;
rte_cdx_unmap_device(dev);
diff --git a/drivers/bus/dpaa/bus_dpaa_driver.h b/drivers/bus/dpaa/bus_dpaa_driver.h
index 7e5e9b2126..5ddb52c419 100644
--- a/drivers/bus/dpaa/bus_dpaa_driver.h
+++ b/drivers/bus/dpaa/bus_dpaa_driver.h
@@ -85,7 +85,6 @@ struct rte_dpaa_device {
struct rte_cryptodev *crypto_dev;
struct rte_dma_dev *dmadev;
};
- struct rte_dpaa_driver *driver;
struct dpaa_device_id id;
struct rte_intr_handle *intr_handle;
enum rte_dpaa_type device_type; /**< Ethernet or crypto type device */
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index 0fd6a8d7a6..ade093cd13 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -802,8 +802,6 @@ dpaa_bus_probe_device(struct rte_driver *drv, struct rte_device *dev)
ret = dpaa_drv->probe(dpaa_drv, dpaa_dev);
if (ret != 0)
DPAA_BUS_ERR("unable to probe: %s", dpaa_dev->name);
- else
- dpaa_dev->driver = dpaa_drv;
return ret;
}
@@ -828,7 +826,6 @@ dpaa_bus_cleanup(void)
rte_errno = errno;
return -1;
}
- dev->driver = NULL;
dev->device.driver = NULL;
}
dpaa_portal_finish((void *)DPAA_PER_LCORE_PORTAL);
diff --git a/drivers/bus/fslmc/bus_fslmc_driver.h b/drivers/bus/fslmc/bus_fslmc_driver.h
index ab8bc1c41d..44f81cf662 100644
--- a/drivers/bus/fslmc/bus_fslmc_driver.h
+++ b/drivers/bus/fslmc/bus_fslmc_driver.h
@@ -101,7 +101,6 @@ struct rte_dpaa2_device {
uint16_t ep_object_id; /**< Endpoint DPAA2 Object ID */
char ep_name[RTE_DEV_NAME_MAX_LEN];
struct rte_intr_handle *intr_handle; /**< Interrupt handle */
- struct rte_dpaa2_driver *driver; /**< Associated driver */
char name[FSLMC_OBJECT_MAX_LEN]; /**< DPAA2 Object name*/
};
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index e33b63259d..d855d6b36b 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -523,7 +523,6 @@ fslmc_bus_probe_device(struct rte_driver *driver, struct rte_device *rte_dev)
if (ret != 0) {
DPAA2_BUS_ERR("Unable to probe");
} else {
- dev->driver = drv;
DPAA2_BUS_INFO("%s Plugged", dev->device.name);
}
@@ -538,7 +537,6 @@ fslmc_bus_unplug(struct rte_device *rte_dev)
if (drv->remove != NULL) {
drv->remove(dev);
- dev->driver = NULL;
dev->device.driver = NULL;
DPAA2_BUS_INFO("%s Un-Plugged", dev->device.name);
return 0;
diff --git a/drivers/bus/ifpga/bus_ifpga_driver.h b/drivers/bus/ifpga/bus_ifpga_driver.h
index b56916167c..4df95485c9 100644
--- a/drivers/bus/ifpga/bus_ifpga_driver.h
+++ b/drivers/bus/ifpga/bus_ifpga_driver.h
@@ -75,7 +75,6 @@ struct rte_afu_device {
/**< AFU Memory Resource */
struct rte_afu_shared shared;
struct rte_intr_handle *intr_handle; /**< Interrupt handle */
- struct rte_afu_driver *driver; /**< Associated driver */
char path[IFPGA_BUS_BITSTREAM_PATH_MAX_LEN];
};
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index b13285230b..2c22329f65 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -272,17 +272,8 @@ ifpga_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
struct rte_afu_device *afu_dev = RTE_BUS_DEVICE(dev, *afu_dev);
struct rte_afu_driver *afu_drv = RTE_BUS_DRIVER(drv, *afu_drv);
- int ret;
-
- /* reference driver structure */
- afu_dev->driver = afu_drv;
-
- /* call the driver probe() function */
- ret = afu_drv->probe(afu_dev);
- if (ret)
- afu_dev->driver = NULL;
- return ret;
+ return afu_drv->probe(afu_dev);
}
/*
@@ -310,7 +301,6 @@ ifpga_cleanup(void)
rte_errno = errno;
error = -1;
}
- afu_dev->driver = NULL;
afu_dev->device.driver = NULL;
free:
diff --git a/drivers/bus/pci/bus_pci_driver.h b/drivers/bus/pci/bus_pci_driver.h
index b0e5428e64..cb7039f8d6 100644
--- a/drivers/bus/pci/bus_pci_driver.h
+++ b/drivers/bus/pci/bus_pci_driver.h
@@ -39,7 +39,6 @@ struct rte_pci_device {
struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE];
/**< PCI Memory Resource */
struct rte_intr_handle *intr_handle; /**< Interrupt handle */
- struct rte_pci_driver *driver; /**< PCI driver used in probing */
uint16_t max_vfs; /**< sriov enable if not zero */
enum rte_pci_kernel_driver kdrv; /**< Kernel driver passthrough */
char name[PCI_PRI_STR_SIZE+1]; /**< PCI location (ASCII) */
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index e927f4af7b..2bdb94a924 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -245,11 +245,9 @@ pci_probe_device(struct rte_driver *drv, struct rte_device *dev)
return -ENOMEM;
}
- pci_dev->driver = pci_drv;
if (pci_drv->drv_flags & RTE_PCI_DRV_NEED_MAPPING) {
ret = rte_pci_map_device(pci_dev);
if (ret != 0) {
- pci_dev->driver = NULL;
rte_intr_instance_free(pci_dev->vfio_req_intr_handle);
pci_dev->vfio_req_intr_handle = NULL;
rte_intr_instance_free(pci_dev->intr_handle);
@@ -268,7 +266,6 @@ pci_probe_device(struct rte_driver *drv, struct rte_device *dev)
if (already_probed)
return ret; /* no rollback if already succeeded earlier */
if (ret) {
- pci_dev->driver = NULL;
if ((pci_drv->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
/* Don't unmap if device is unsupported and
* driver needs mapped resources.
@@ -312,7 +309,6 @@ rte_pci_detach_dev(struct rte_pci_device *dev)
}
/* clear driver structure */
- dev->driver = NULL;
dev->device.driver = NULL;
if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
@@ -348,7 +344,6 @@ pci_cleanup(void)
rte_errno = errno;
error = -1;
}
- dev->driver = NULL;
dev->device.driver = NULL;
free:
diff --git a/drivers/bus/platform/bus_platform_driver.h b/drivers/bus/platform/bus_platform_driver.h
index 3912ed5b85..3ac405a201 100644
--- a/drivers/bus/platform/bus_platform_driver.h
+++ b/drivers/bus/platform/bus_platform_driver.h
@@ -95,7 +95,6 @@ struct rte_platform_resource {
*/
struct rte_platform_device {
struct rte_device device; /**< Core device */
- struct rte_platform_driver *driver; /**< Matching device driver */
char name[RTE_DEV_NAME_MAX_LEN]; /**< Device name */
unsigned int num_resource; /**< Number of device resources */
struct rte_platform_resource *resource; /**< Device resources */
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 433bab9b60..4492ed62ec 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -329,10 +329,8 @@ device_setup(struct rte_platform_device *pdev)
static int
driver_call_probe(struct rte_platform_driver *pdrv, struct rte_platform_device *pdev)
{
- if (pdrv->probe != NULL) {
- pdev->driver = pdrv;
+ if (pdrv->probe != NULL)
return pdrv->probe(pdev);
- }
return 0;
}
@@ -418,7 +416,6 @@ device_release_driver(struct rte_platform_device *pdev)
}
pdev->device.driver = NULL;
- pdev->driver = NULL;
}
static int
diff --git a/drivers/bus/uacce/bus_uacce_driver.h b/drivers/bus/uacce/bus_uacce_driver.h
index 04ced912c9..afed71bb67 100644
--- a/drivers/bus/uacce/bus_uacce_driver.h
+++ b/drivers/bus/uacce/bus_uacce_driver.h
@@ -44,7 +44,6 @@ struct rte_uacce_driver;
*/
struct rte_uacce_device {
struct rte_device device; /**< Inherit core device. */
- struct rte_uacce_driver *driver; /**< Driver used in probing. */
char name[RTE_DEV_NAME_MAX_LEN]; /**< Device name. */
char dev_root[RTE_UACCE_DEV_PATH_SIZE]; /**< Sysfs path with device name. */
char cdev_path[RTE_UACCE_DEV_PATH_SIZE]; /**< Device path in devfs. */
diff --git a/drivers/bus/uacce/uacce.c b/drivers/bus/uacce/uacce.c
index ead920d261..db49e887ad 100644
--- a/drivers/bus/uacce/uacce.c
+++ b/drivers/bus/uacce/uacce.c
@@ -366,7 +366,6 @@ uacce_probe_device(struct rte_driver *drv, struct rte_device *dev)
UACCE_BUS_ERR("probe device %s with driver %s failed %d",
dev_name, uacce_drv->driver.name, ret);
} else {
- uacce_dev->driver = uacce_drv;
UACCE_BUS_DEBUG("probe device %s with driver %s success",
dev_name, uacce_drv->driver.name);
}
@@ -395,7 +394,6 @@ uacce_cleanup(void)
rte_errno = errno;
error = -1;
}
- dev->driver = NULL;
dev->device.driver = NULL;
free:
@@ -420,7 +418,6 @@ uacce_detach_dev(struct rte_uacce_device *dev)
return ret;
}
- dev->driver = NULL;
dev->device.driver = NULL;
return 0;
diff --git a/drivers/bus/vmbus/bus_vmbus_driver.h b/drivers/bus/vmbus/bus_vmbus_driver.h
index d53bda2340..888d856141 100644
--- a/drivers/bus/vmbus/bus_vmbus_driver.h
+++ b/drivers/bus/vmbus/bus_vmbus_driver.h
@@ -37,7 +37,6 @@ enum hv_uio_map {
* A structure describing a VMBUS device.
*/
struct rte_vmbus_device {
- const struct rte_vmbus_driver *driver; /**< Associated driver */
struct rte_device device; /**< Inherit core device */
rte_uuid_t device_id; /**< VMBUS device id */
rte_uuid_t class_id; /**< VMBUS device type */
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index b96f4133dd..2b1730afc2 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -107,19 +107,14 @@ vmbus_probe_device(struct rte_driver *drv, struct rte_device *dev)
if (ret != 0)
return ret;
- /* reference driver structure */
- vmbus_dev->driver = vmbus_drv;
-
if (vmbus_dev->device.numa_node < 0 && rte_socket_count() > 1)
VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
/* call the driver probe() function */
VMBUS_LOG(INFO, " probe driver: %s", vmbus_drv->driver.name);
ret = vmbus_drv->probe(vmbus_drv, vmbus_dev);
- if (ret != 0) {
- vmbus_dev->driver = NULL;
+ if (ret != 0)
rte_vmbus_unmap_device(vmbus_dev);
- }
return ret;
}
@@ -158,7 +153,6 @@ rte_vmbus_cleanup(void)
rte_vmbus_unmap_device(dev);
- dev->driver = NULL;
dev->device.driver = NULL;
rte_bus_remove_device(&rte_vmbus_bus.bus, &dev->device);
free(dev);
--
2.53.0
^ permalink raw reply related
* [PATCH v3 19/25] drivers: rely on generic driver
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Rosen Xu, Chenbo Xia, Tomasz Duszynski, Chengwen Feng, Long Li,
Wei Hu, Kai Ji, Chengfei Han, Ming Ran, Christian Koue Muf,
Serhii Iliushyk, Jakub Palider, Akhil Goyal, Jingjing Wu,
Andrew Rybchenko
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
Now that the generic device object always references a generic driver, we
can rely on this reference and stop going through the bus-specific
driver reference.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/auxiliary/auxiliary_common.c | 12 +++++++-----
drivers/bus/cdx/cdx.c | 2 +-
drivers/bus/dpaa/dpaa_bus.c | 3 ++-
drivers/bus/fslmc/fslmc_bus.c | 2 +-
drivers/bus/fslmc/fslmc_vfio.c | 10 ++++++----
drivers/bus/ifpga/ifpga_bus.c | 6 ++++--
drivers/bus/pci/linux/pci_uio.c | 6 ++++--
drivers/bus/pci/pci_common.c | 20 +++++++++-----------
drivers/bus/platform/platform.c | 20 ++++++++++++--------
drivers/bus/uacce/uacce.c | 5 +++--
drivers/bus/vmbus/vmbus_common.c | 3 ++-
drivers/common/qat/qat_qp.c | 4 ++--
drivers/common/zsda/zsda_qp.c | 4 ++--
drivers/net/ntnic/ntnic_ethdev.c | 8 +++-----
drivers/raw/cnxk_bphy/cnxk_bphy.c | 2 +-
drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c | 2 +-
drivers/raw/cnxk_rvu_lf/cnxk_rvu_lf.c | 2 +-
drivers/raw/ifpga/afu_pmd_core.c | 2 +-
drivers/raw/ifpga/ifpga_rawdev.c | 2 +-
drivers/raw/ntb/ntb.c | 2 +-
lib/ethdev/ethdev_pci.h | 7 +++++--
21 files changed, 69 insertions(+), 55 deletions(-)
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index ba8c35ebbe..2000ffa369 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -131,7 +131,7 @@ auxiliary_probe_device(struct rte_driver *drv, struct rte_device *dev)
static int
rte_auxiliary_driver_remove_dev(struct rte_auxiliary_device *dev)
{
- struct rte_auxiliary_driver *drv = dev->driver;
+ const struct rte_auxiliary_driver *drv = RTE_BUS_DRIVER(dev->device.driver, *drv);
int ret = 0;
AUXILIARY_LOG(DEBUG, "Driver %s remove auxiliary device %s on NUMA node %i",
@@ -226,12 +226,13 @@ static int
auxiliary_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
struct rte_auxiliary_device *aux_dev = RTE_BUS_DEVICE(dev, *aux_dev);
+ const struct rte_auxiliary_driver *aux_drv = RTE_BUS_DRIVER(dev->driver, *aux_drv);
- if (aux_dev->driver->dma_map == NULL) {
+ if (aux_drv->dma_map == NULL) {
rte_errno = ENOTSUP;
return -1;
}
- return aux_dev->driver->dma_map(aux_dev, addr, iova, len);
+ return aux_drv->dma_map(aux_dev, addr, iova, len);
}
static int
@@ -239,12 +240,13 @@ auxiliary_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,
size_t len)
{
struct rte_auxiliary_device *aux_dev = RTE_BUS_DEVICE(dev, *aux_dev);
+ const struct rte_auxiliary_driver *aux_drv = RTE_BUS_DRIVER(dev->driver, *aux_drv);
- if (aux_dev->driver->dma_unmap == NULL) {
+ if (aux_drv->dma_unmap == NULL) {
rte_errno = ENOTSUP;
return -1;
}
- return aux_dev->driver->dma_unmap(aux_dev, addr, iova, len);
+ return aux_drv->dma_unmap(aux_dev, addr, iova, len);
}
static enum rte_iova_mode
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index c6a4da7692..5a823a0dcd 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -382,7 +382,7 @@ rte_cdx_unregister(struct rte_cdx_driver *driver)
static int
cdx_detach_dev(struct rte_cdx_device *dev)
{
- struct rte_cdx_driver *dr = dev->driver;
+ const struct rte_cdx_driver *dr = RTE_BUS_DRIVER(dev->device.driver, *dr);
int ret = 0;
CDX_BUS_DEBUG("detach device %s using driver: %s",
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index b7d0caaf61..0fd6a8d7a6 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -815,11 +815,12 @@ dpaa_bus_cleanup(void)
BUS_INIT_FUNC_TRACE();
RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
- struct rte_dpaa_driver *drv = dev->driver;
+ const struct rte_dpaa_driver *drv;
int ret = 0;
if (!rte_dev_is_probed(&dev->device))
continue;
+ drv = RTE_BUS_DRIVER(dev->device.driver, *drv);
if (drv->remove == NULL)
continue;
ret = drv->remove(dev);
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index ae69bd0da2..e33b63259d 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -534,7 +534,7 @@ static int
fslmc_bus_unplug(struct rte_device *rte_dev)
{
struct rte_dpaa2_device *dev = RTE_BUS_DEVICE(rte_dev, *dev);
- struct rte_dpaa2_driver *drv = dev->driver;
+ const struct rte_dpaa2_driver *drv = RTE_BUS_DRIVER(rte_dev->driver, *drv);
if (drv->remove != NULL) {
drv->remove(dev);
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index e38f3e9fe7..5784adaf90 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -1392,7 +1392,7 @@ fslmc_close_iodevices(struct rte_dpaa2_device *dev,
int vfio_fd)
{
struct rte_dpaa2_object *object = NULL;
- struct rte_dpaa2_driver *drv;
+ const struct rte_dpaa2_driver *drv;
int ret;
switch (dev->dev_type) {
@@ -1411,9 +1411,11 @@ fslmc_close_iodevices(struct rte_dpaa2_device *dev,
case DPAA2_ETH:
case DPAA2_CRYPTO:
case DPAA2_QDMA:
- drv = dev->driver;
- if (drv && drv->remove && drv->remove(dev))
- DPAA2_BUS_ERR("Unable to remove");
+ if (dev->device.driver != NULL) {
+ drv = RTE_BUS_DRIVER(dev->device.driver, *drv);
+ if (drv->remove && drv->remove(dev))
+ DPAA2_BUS_ERR("Unable to remove");
+ }
break;
default:
break;
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index a79a287fbe..b13285230b 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -296,11 +296,12 @@ ifpga_cleanup(void)
int error = 0;
RTE_BUS_FOREACH_DEV(afu_dev, &rte_ifpga_bus) {
- struct rte_afu_driver *drv = afu_dev->driver;
+ const struct rte_afu_driver *drv;
int ret = 0;
if (!rte_dev_is_probed(&afu_dev->device))
goto free;
+ drv = RTE_BUS_DRIVER(afu_dev->device.driver, *drv);
if (drv->remove == NULL)
goto free;
@@ -326,9 +327,10 @@ static int
ifpga_unplug(struct rte_device *dev)
{
struct rte_afu_device *afu_dev = RTE_BUS_DEVICE(dev, *afu_dev);
+ const struct rte_afu_driver *afu_drv = RTE_BUS_DRIVER(dev->driver, *afu_drv);
int ret;
- ret = afu_dev->driver->remove(afu_dev);
+ ret = afu_drv->remove(afu_dev);
if (ret)
return ret;
diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c
index 4c1d3327a9..40e96b1c67 100644
--- a/drivers/bus/pci/linux/pci_uio.c
+++ b/drivers/bus/pci/linux/pci_uio.c
@@ -301,8 +301,10 @@ pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx,
struct pci_map *maps;
int wc_activate = 0;
- if (dev->driver != NULL)
- wc_activate = dev->driver->drv_flags & RTE_PCI_DRV_WC_ACTIVATE;
+ if (dev->device.driver != NULL) {
+ const struct rte_pci_driver *pdrv = RTE_BUS_DRIVER(dev->device.driver, *pdrv);
+ wc_activate = pdrv->drv_flags & RTE_PCI_DRV_WC_ACTIVATE;
+ }
loc = &dev->addr;
maps = uio_res->maps;
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 5c37c5e543..e927f4af7b 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -245,11 +245,6 @@ pci_probe_device(struct rte_driver *drv, struct rte_device *dev)
return -ENOMEM;
}
- /*
- * Reference driver structure.
- * This needs to be before rte_pci_map_device(), as it enables
- * to use driver flags for adjusting configuration.
- */
pci_dev->driver = pci_drv;
if (pci_drv->drv_flags & RTE_PCI_DRV_NEED_MAPPING) {
ret = rte_pci_map_device(pci_dev);
@@ -298,7 +293,7 @@ static int
rte_pci_detach_dev(struct rte_pci_device *dev)
{
struct rte_pci_addr *loc;
- struct rte_pci_driver *dr = dev->driver;
+ const struct rte_pci_driver *dr = RTE_BUS_DRIVER(dev->device.driver, *dr);
int ret = 0;
loc = &dev->addr;
@@ -339,11 +334,12 @@ pci_cleanup(void)
int error = 0;
RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus.bus) {
- struct rte_pci_driver *drv = dev->driver;
+ const struct rte_pci_driver *drv;
int ret = 0;
if (!rte_dev_is_probed(&dev->device))
goto free;
+ drv = RTE_BUS_DRIVER(dev->device.driver, *drv);
if (drv->remove == NULL)
goto free;
@@ -545,9 +541,10 @@ static int
pci_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
struct rte_pci_device *pdev = RTE_BUS_DEVICE(dev, *pdev);
+ const struct rte_pci_driver *pdrv = RTE_BUS_DRIVER(dev->driver, *pdrv);
- if (pdev->driver->dma_map != NULL)
- return pdev->driver->dma_map(pdev, addr, iova, len);
+ if (pdrv->dma_map != NULL)
+ return pdrv->dma_map(pdev, addr, iova, len);
/**
* In case driver don't provides any specific mapping
* try fallback to VFIO.
@@ -564,9 +561,10 @@ static int
pci_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
struct rte_pci_device *pdev = RTE_BUS_DEVICE(dev, *pdev);
+ const struct rte_pci_driver *pdrv = RTE_BUS_DRIVER(dev->driver, *pdrv);
- if (pdev->driver->dma_unmap != NULL)
- return pdev->driver->dma_unmap(pdev, addr, iova, len);
+ if (pdrv->dma_unmap != NULL)
+ return pdrv->dma_unmap(pdev, addr, iova, len);
/**
* In case driver don't provides any specific mapping
* try fallback to VFIO.
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 671f36fac9..433bab9b60 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -408,7 +408,7 @@ platform_bus_probe_device(struct rte_driver *drv, struct rte_device *dev)
static void
device_release_driver(struct rte_platform_device *pdev)
{
- struct rte_platform_driver *pdrv = pdev->driver;
+ const struct rte_platform_driver *pdrv = RTE_BUS_DRIVER(pdev->device.driver, *pdrv);
int ret;
if (pdrv->remove != NULL) {
@@ -458,9 +458,10 @@ static int
platform_bus_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
struct rte_platform_device *pdev = RTE_BUS_DEVICE(dev, *pdev);
+ const struct rte_platform_driver *pdrv = RTE_BUS_DRIVER(dev->driver, *pdrv);
- if (pdev->driver->dma_map != NULL)
- return pdev->driver->dma_map(pdev, addr, iova, len);
+ if (pdrv->dma_map != NULL)
+ return pdrv->dma_map(pdev, addr, iova, len);
return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova, len);
}
@@ -469,9 +470,10 @@ static int
platform_bus_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
struct rte_platform_device *pdev = RTE_BUS_DEVICE(dev, *pdev);
+ const struct rte_platform_driver *pdrv = RTE_BUS_DRIVER(dev->driver, *pdrv);
- if (pdev->driver->dma_unmap != NULL)
- return pdev->driver->dma_unmap(pdev, addr, iova, len);
+ if (pdrv->dma_unmap != NULL)
+ return pdrv->dma_unmap(pdev, addr, iova, len);
return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova,
len);
@@ -480,12 +482,14 @@ platform_bus_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t
static enum rte_iova_mode
platform_bus_get_iommu_class(void)
{
- struct rte_platform_driver *pdrv;
+ const struct rte_platform_driver *pdrv;
struct rte_platform_device *pdev;
RTE_BUS_FOREACH_DEV(pdev, &platform_bus.bus) {
- pdrv = pdev->driver;
- if (pdrv != NULL && pdrv->drv_flags & RTE_PLATFORM_DRV_NEED_IOVA_AS_VA)
+ if (!rte_dev_is_probed(&pdev->device))
+ continue;
+ pdrv = RTE_BUS_DRIVER(pdev->device.driver, *pdrv);
+ if (pdrv->drv_flags & RTE_PLATFORM_DRV_NEED_IOVA_AS_VA)
return RTE_IOVA_VA;
}
diff --git a/drivers/bus/uacce/uacce.c b/drivers/bus/uacce/uacce.c
index aa33499f55..ead920d261 100644
--- a/drivers/bus/uacce/uacce.c
+++ b/drivers/bus/uacce/uacce.c
@@ -381,11 +381,12 @@ uacce_cleanup(void)
int error = 0;
RTE_BUS_FOREACH_DEV(dev, &uacce_bus.bus) {
- struct rte_uacce_driver *dr = dev->driver;
+ const struct rte_uacce_driver *dr;
int ret = 0;
if (!rte_dev_is_probed(&dev->device))
goto free;
+ dr = RTE_BUS_DRIVER(dev->device.driver, *dr);
if (dr->remove == NULL)
goto free;
@@ -408,7 +409,7 @@ uacce_cleanup(void)
static int
uacce_detach_dev(struct rte_uacce_device *dev)
{
- struct rte_uacce_driver *dr = dev->driver;
+ const struct rte_uacce_driver *dr = RTE_BUS_DRIVER(dev->device.driver, *dr);
int ret = 0;
UACCE_BUS_DEBUG("detach device %s using driver: %s", dev->device.name, dr->driver.name);
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 43652c0487..b96f4133dd 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -143,11 +143,12 @@ rte_vmbus_cleanup(void)
int error = 0;
RTE_BUS_FOREACH_DEV(dev, &rte_vmbus_bus.bus) {
- const struct rte_vmbus_driver *drv = dev->driver;
+ const struct rte_vmbus_driver *drv;
int ret;
if (!rte_dev_is_probed(&dev->device))
continue;
+ drv = RTE_BUS_DRIVER(dev->device.driver, *drv);
if (drv->remove == NULL)
continue;
diff --git a/drivers/common/qat/qat_qp.c b/drivers/common/qat/qat_qp.c
index 0d2bbdb8a5..fe28cb160d 100644
--- a/drivers/common/qat/qat_qp.c
+++ b/drivers/common/qat/qat_qp.c
@@ -174,7 +174,7 @@ qat_qp_setup(struct qat_pci_device *qat_dev,
snprintf(op_cookie_pool_name, RTE_RING_NAMESIZE,
"%s%d_cookies_%s_qp%hu",
- pci_dev->driver->driver.name, qat_dev->qat_dev_id,
+ pci_dev->device.driver->name, qat_dev->qat_dev_id,
qat_qp_conf->service_str, queue_pair_id);
QAT_LOG(DEBUG, "cookiepool: %s", op_cookie_pool_name);
@@ -252,7 +252,7 @@ qat_queue_create(struct qat_pci_device *qat_dev, struct qat_queue *queue,
*/
snprintf(queue->memz_name, sizeof(queue->memz_name),
"%s_%d_%s_%s_%d_%d",
- pci_dev->driver->driver.name, qat_dev->qat_dev_id,
+ pci_dev->device.driver->name, qat_dev->qat_dev_id,
qp_conf->service_str, "qp_mem",
queue->hw_bundle_number, queue->hw_queue_number);
qp_mz = queue_dma_zone_reserve(queue->memz_name, queue_size_bytes,
diff --git a/drivers/common/zsda/zsda_qp.c b/drivers/common/zsda/zsda_qp.c
index dab524e2d3..79172234af 100644
--- a/drivers/common/zsda/zsda_qp.c
+++ b/drivers/common/zsda/zsda_qp.c
@@ -592,12 +592,12 @@ zsda_queue_create(const uint8_t dev_id, struct zsda_queue *queue,
if (dir == RING_DIR_TX)
snprintf(queue->memz_name, sizeof(queue->memz_name),
- "%s_%d_%s_%s_%d", pci_dev->driver->driver.name, dev_id,
+ "%s_%d_%s_%s_%d", pci_dev->device.driver->name, dev_id,
qp_conf->service_str, "qptxmem",
queue->hw_queue_number);
else
snprintf(queue->memz_name, sizeof(queue->memz_name),
- "%s_%d_%s_%s_%d", pci_dev->driver->driver.name, dev_id,
+ "%s_%d_%s_%s_%d", pci_dev->device.driver->name, dev_id,
qp_conf->service_str, "qprxmem",
queue->hw_queue_number);
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 89e751e7e0..7cc90a7a5b 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2676,8 +2676,7 @@ nthw_pci_dev_deinit(struct rte_eth_dev *eth_dev __rte_unused)
}
static int
-nthw_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
- struct rte_pci_device *pci_dev)
+nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
{
int ret;
@@ -2711,9 +2710,8 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
pci_dev->id.subsystem_vendor_id, pci_dev->id.subsystem_device_id,
pci_dev->name[0] ? pci_dev->name : "NA",
pci_dev->device.numa_node,
- pci_dev->driver->driver.name ? pci_dev->driver->driver.name : "NA",
- pci_dev->driver->driver.alias ? pci_dev->driver->driver.alias : "NA");
-
+ (pci_drv->driver.name != NULL) ? pci_drv->driver.name : "NA",
+ (pci_drv->driver.alias != NULL) ? pci_drv->driver.alias : "NA");
ret = nthw_pci_dev_init(pci_dev);
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy.c b/drivers/raw/cnxk_bphy/cnxk_bphy.c
index 0c26cfbbe6..ea30a7100c 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy.c
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy.c
@@ -351,7 +351,7 @@ bphy_rawdev_probe(struct rte_pci_driver *pci_drv,
bphy_rawdev->dev_ops = &bphy_rawdev_ops;
bphy_rawdev->device = &pci_dev->device;
- bphy_rawdev->driver_name = pci_dev->driver->driver.name;
+ bphy_rawdev->driver_name = pci_dev->device.driver->name;
bphy_dev = (struct bphy_device *)bphy_rawdev->dev_private;
bphy_dev->mem.res0 = pci_dev->mem_resource[0];
diff --git a/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c b/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c
index c82589baa2..e0c7257027 100644
--- a/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c
+++ b/drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c
@@ -319,7 +319,7 @@ cnxk_bphy_cgx_rawdev_probe(struct rte_pci_driver *pci_drv,
rawdev->dev_ops = &cnxk_bphy_cgx_rawdev_ops;
rawdev->device = &pci_dev->device;
- rawdev->driver_name = pci_dev->driver->driver.name;
+ rawdev->driver_name = pci_dev->device.driver->name;
cgx = rawdev->dev_private;
cgx->rcgx = rte_zmalloc(NULL, sizeof(*rcgx), 0);
diff --git a/drivers/raw/cnxk_rvu_lf/cnxk_rvu_lf.c b/drivers/raw/cnxk_rvu_lf/cnxk_rvu_lf.c
index 60c2080740..57c7cc85f3 100644
--- a/drivers/raw/cnxk_rvu_lf/cnxk_rvu_lf.c
+++ b/drivers/raw/cnxk_rvu_lf/cnxk_rvu_lf.c
@@ -210,7 +210,7 @@ rvu_lf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
rvu_lf_rawdev->dev_ops = &rvu_lf_rawdev_ops;
rvu_lf_rawdev->device = &pci_dev->device;
- rvu_lf_rawdev->driver_name = pci_dev->driver->driver.name;
+ rvu_lf_rawdev->driver_name = pci_dev->device.driver->name;
roc_rvu_lf = (struct roc_rvu_lf *)rvu_lf_rawdev->dev_private;
roc_rvu_lf->pci_dev = pci_dev;
diff --git a/drivers/raw/ifpga/afu_pmd_core.c b/drivers/raw/ifpga/afu_pmd_core.c
index f650b76cfe..ca14ebf52f 100644
--- a/drivers/raw/ifpga/afu_pmd_core.c
+++ b/drivers/raw/ifpga/afu_pmd_core.c
@@ -310,7 +310,7 @@ static int afu_rawdev_create(struct rte_afu_device *afu_dev, int socket_id)
rawdev->dev_ops = &afu_rawdev_ops;
rawdev->device = &afu_dev->device;
- rawdev->driver_name = afu_dev->driver->driver.name;
+ rawdev->driver_name = afu_dev->device.driver->name;
dev = afu_rawdev_get_priv(rawdev);
if (!dev)
diff --git a/drivers/raw/ifpga/ifpga_rawdev.c b/drivers/raw/ifpga/ifpga_rawdev.c
index d1d54e9065..9f961c943f 100644
--- a/drivers/raw/ifpga/ifpga_rawdev.c
+++ b/drivers/raw/ifpga/ifpga_rawdev.c
@@ -1602,7 +1602,7 @@ ifpga_rawdev_create(struct rte_pci_device *pci_dev,
rawdev->dev_ops = &ifpga_rawdev_ops;
rawdev->device = &pci_dev->device;
- rawdev->driver_name = pci_dev->driver->driver.name;
+ rawdev->driver_name = pci_dev->device.driver->name;
/* must enumerate the adapter before use it */
ret = opae_adapter_enumerate(adapter);
diff --git a/drivers/raw/ntb/ntb.c b/drivers/raw/ntb/ntb.c
index 0ed4c14592..d54f2fb783 100644
--- a/drivers/raw/ntb/ntb.c
+++ b/drivers/raw/ntb/ntb.c
@@ -1478,7 +1478,7 @@ ntb_create(struct rte_pci_device *pci_dev, int socket_id)
rawdev->dev_ops = &ntb_ops;
rawdev->device = &pci_dev->device;
- rawdev->driver_name = pci_dev->driver->driver.name;
+ rawdev->driver_name = pci_dev->device.driver->name;
ret = ntb_init_hw(rawdev, pci_dev);
if (ret < 0) {
diff --git a/lib/ethdev/ethdev_pci.h b/lib/ethdev/ethdev_pci.h
index 2229ffa252..ce3e293818 100644
--- a/lib/ethdev/ethdev_pci.h
+++ b/lib/ethdev/ethdev_pci.h
@@ -39,10 +39,13 @@ rte_eth_copy_pci_info(struct rte_eth_dev *eth_dev,
eth_dev->intr_handle = pci_dev->intr_handle;
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ const struct rte_pci_driver *pci_drv;
+
+ pci_drv = RTE_BUS_DRIVER(pci_dev->device.driver, *pci_drv);
eth_dev->data->dev_flags = 0;
- if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_LSC)
+ if (pci_drv->drv_flags & RTE_PCI_DRV_INTR_LSC)
eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_LSC;
- if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_RMV)
+ if (pci_drv->drv_flags & RTE_PCI_DRV_INTR_RMV)
eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_RMV;
eth_dev->data->numa_node = pci_dev->device.numa_node;
--
2.53.0
^ permalink raw reply related
* [PATCH v3 18/25] bus: factorize driver reference
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Rosen Xu, Chenbo Xia, Tomasz Duszynski, Chengwen Feng, Long Li,
Wei Hu
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
Before this patch, setting the driver in the device object and error
rollback were the responsibility of each bus probe_device function.
This patch centralizes it in the EAL's local_dev_probe function.
The dev->driver field is now set before calling bus->probe_device,
ensuring that drivers can rely on it being set during probe.
On error, dev->driver is cleared only if the device was not previously
probed, to handle buses that support multiple probes
(RTE_PCI_DRV_PROBE_AGAIN).
As a consequence, each bus implementation no longer needs to set/clear
dev->driver and dev->device.driver.
In PCI bus, a side effect is that detecting a re-probe cannot use
rte_dev_is_probed (which checks dev->driver != NULL) anymore.
Switch to checking if intr_handle was allocated instead.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/auxiliary/auxiliary_common.c | 2 --
drivers/bus/cdx/cdx.c | 2 --
drivers/bus/dpaa/dpaa_bus.c | 6 ++----
drivers/bus/fslmc/fslmc_bus.c | 1 -
drivers/bus/ifpga/ifpga_bus.c | 2 --
drivers/bus/pci/pci_common.c | 6 ++----
drivers/bus/platform/platform.c | 8 +-------
drivers/bus/uacce/uacce.c | 1 -
drivers/bus/vmbus/vmbus_common.c | 2 --
lib/eal/common/eal_common_bus.c | 7 +++++++
lib/eal/common/eal_common_dev.c | 12 ++++++++++++
11 files changed, 24 insertions(+), 25 deletions(-)
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index 206b69bbd4..ba8c35ebbe 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -120,8 +120,6 @@ auxiliary_probe_device(struct rte_driver *drv, struct rte_device *dev)
aux_dev->driver = NULL;
rte_intr_instance_free(aux_dev->intr_handle);
aux_dev->intr_handle = NULL;
- } else {
- aux_dev->device.driver = &aux_drv->driver;
}
return ret;
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index e1405d1372..c6a4da7692 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -332,8 +332,6 @@ cdx_probe_device(struct rte_driver *drv, struct rte_device *dev)
CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
cdx_drv->driver.name, dev_name, ret);
goto error_probe;
- } else {
- cdx_dev->device.driver = &cdx_drv->driver;
}
cdx_dev->driver = cdx_drv;
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index 7f4a85a91d..b7d0caaf61 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -800,12 +800,10 @@ dpaa_bus_probe_device(struct rte_driver *drv, struct rte_device *dev)
int ret;
ret = dpaa_drv->probe(dpaa_drv, dpaa_dev);
- if (ret != 0) {
+ if (ret != 0)
DPAA_BUS_ERR("unable to probe: %s", dpaa_dev->name);
- } else {
+ else
dpaa_dev->driver = dpaa_drv;
- dpaa_dev->device.driver = &dpaa_drv->driver;
- }
return ret;
}
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 29ef3b58cf..ae69bd0da2 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -524,7 +524,6 @@ fslmc_bus_probe_device(struct rte_driver *driver, struct rte_device *rte_dev)
DPAA2_BUS_ERR("Unable to probe");
} else {
dev->driver = drv;
- dev->device.driver = &drv->driver;
DPAA2_BUS_INFO("%s Plugged", dev->device.name);
}
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index b2eb4f4413..a79a287fbe 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -281,8 +281,6 @@ ifpga_probe_device(struct rte_driver *drv, struct rte_device *dev)
ret = afu_drv->probe(afu_dev);
if (ret)
afu_dev->driver = NULL;
- else
- afu_dev->device.driver = &afu_drv->driver;
return ret;
}
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index a507360ef6..5c37c5e543 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -203,7 +203,7 @@ pci_probe_device(struct rte_driver *drv, struct rte_device *dev)
if (pci_dev->device.numa_node < 0 && rte_socket_count() > 1)
PCI_LOG(INFO, "Device %s is not NUMA-aware", pci_dev->name);
- already_probed = rte_dev_is_probed(&pci_dev->device);
+ already_probed = (pci_dev->intr_handle != NULL);
if (already_probed && !(pci_drv->drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
PCI_LOG(DEBUG, "Device %s is already probed", pci_dev->device.name);
return -EEXIST;
@@ -251,7 +251,7 @@ pci_probe_device(struct rte_driver *drv, struct rte_device *dev)
* to use driver flags for adjusting configuration.
*/
pci_dev->driver = pci_drv;
- if (pci_dev->driver->drv_flags & RTE_PCI_DRV_NEED_MAPPING) {
+ if (pci_drv->drv_flags & RTE_PCI_DRV_NEED_MAPPING) {
ret = rte_pci_map_device(pci_dev);
if (ret != 0) {
pci_dev->driver = NULL;
@@ -285,8 +285,6 @@ pci_probe_device(struct rte_driver *drv, struct rte_device *dev)
pci_dev->vfio_req_intr_handle = NULL;
rte_intr_instance_free(pci_dev->intr_handle);
pci_dev->intr_handle = NULL;
- } else {
- pci_dev->device.driver = &pci_drv->driver;
}
return ret;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 01c8239cbb..671f36fac9 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -329,17 +329,11 @@ device_setup(struct rte_platform_device *pdev)
static int
driver_call_probe(struct rte_platform_driver *pdrv, struct rte_platform_device *pdev)
{
- int ret;
-
if (pdrv->probe != NULL) {
pdev->driver = pdrv;
- ret = pdrv->probe(pdev);
- if (ret)
- return ret;
+ return pdrv->probe(pdev);
}
- pdev->device.driver = &pdrv->driver;
-
return 0;
}
diff --git a/drivers/bus/uacce/uacce.c b/drivers/bus/uacce/uacce.c
index 463d08bc12..aa33499f55 100644
--- a/drivers/bus/uacce/uacce.c
+++ b/drivers/bus/uacce/uacce.c
@@ -366,7 +366,6 @@ uacce_probe_device(struct rte_driver *drv, struct rte_device *dev)
UACCE_BUS_ERR("probe device %s with driver %s failed %d",
dev_name, uacce_drv->driver.name, ret);
} else {
- uacce_dev->device.driver = &uacce_drv->driver;
uacce_dev->driver = uacce_drv;
UACCE_BUS_DEBUG("probe device %s with driver %s success",
dev_name, uacce_drv->driver.name);
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index bd375ae6bb..43652c0487 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -119,8 +119,6 @@ vmbus_probe_device(struct rte_driver *drv, struct rte_device *dev)
if (ret != 0) {
vmbus_dev->driver = NULL;
rte_vmbus_unmap_device(vmbus_dev);
- } else {
- vmbus_dev->device.driver = &vmbus_drv->driver;
}
return ret;
diff --git a/lib/eal/common/eal_common_bus.c b/lib/eal/common/eal_common_bus.c
index 94b7be5f5f..ca13ccce5b 100644
--- a/lib/eal/common/eal_common_bus.c
+++ b/lib/eal/common/eal_common_bus.c
@@ -86,6 +86,7 @@ rte_bus_generic_probe(struct rte_bus *bus)
TAILQ_FOREACH(dev, &bus->device_list, next) {
struct rte_driver *drv = NULL;
+ bool was_probed;
int ret;
if (rte_bus_device_is_ignored(bus, dev->name))
@@ -102,7 +103,13 @@ rte_bus_generic_probe(struct rte_bus *bus)
if (drv == NULL)
continue;
+ was_probed = rte_dev_is_probed(dev);
+ if (!was_probed)
+ dev->driver = drv;
ret = bus->probe_device(drv, dev);
+ if (!was_probed && ret != 0)
+ dev->driver = NULL;
+
if (ret > 0)
goto next_driver;
diff --git a/lib/eal/common/eal_common_dev.c b/lib/eal/common/eal_common_dev.c
index 9047a01c4a..48b631532a 100644
--- a/lib/eal/common/eal_common_dev.c
+++ b/lib/eal/common/eal_common_dev.c
@@ -229,7 +229,19 @@ local_dev_probe(const char *devargs, struct rte_device **new_dev)
EAL_LOG(INFO, "Device %s is already probed", dev->name);
ret = -EEXIST;
} else {
+ bool was_probed = rte_dev_is_probed(dev);
+
+ /*
+ * Reference driver structure.
+ * This needs to be before .probe_device as some bus (like PCI) use
+ * driver flags for adjusting configuration.
+ */
+ if (!was_probed)
+ dev->driver = drv;
ret = dev->bus->probe_device(drv, dev);
+ if (!was_probed && ret != 0)
+ dev->driver = NULL;
+
if (ret > 0)
goto next_driver;
}
--
2.53.0
^ permalink raw reply related
* [PATCH v3 17/25] bus: implement probe in EAL
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Rosen Xu, Chenbo Xia, Tomasz Duszynski, Chengwen Feng, Long Li,
Wei Hu, Kevin Laatz
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
Add rte_bus_generic_probe() in EAL to eliminate duplicated device
probing logic across bus drivers. This function implements the common
pattern of iterating devices, finding matching drivers, and calling
the bus probe_device operation.
The generic probe includes rte_bus_device_is_ignored() check to skip
ignored devices during probing. Errors from probe_device() are logged
except for -EEXIST, which is silently skipped to support device reprobing.
Notes:
- rte_vmbus_probe() is kept as it's an exported public API, but
the .probe field is removed from rte_vmbus_bus structure.
- rte_ifpga_device_name() is removed as it has no user remaining.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/auxiliary/auxiliary_common.c | 39 +------------------
drivers/bus/cdx/cdx.c | 38 +-----------------
drivers/bus/dpaa/dpaa_bus.c | 30 +--------------
drivers/bus/fslmc/fslmc_bus.c | 29 +-------------
drivers/bus/ifpga/bus_ifpga_driver.h | 9 -----
drivers/bus/ifpga/ifpga_bus.c | 34 +---------------
drivers/bus/pci/pci_common.c | 42 +-------------------
drivers/bus/platform/platform.c | 33 +---------------
drivers/bus/uacce/uacce.c | 33 +---------------
drivers/bus/vdev/vdev.c | 37 +-----------------
drivers/bus/vmbus/vmbus_common.c | 35 +----------------
drivers/dma/idxd/idxd_bus.c | 21 +---------
lib/eal/common/eal_common_bus.c | 49 +++++++++++++++++++++++-
lib/eal/include/bus_driver.h | 23 ++++++++++-
14 files changed, 81 insertions(+), 371 deletions(-)
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index c210e303ce..206b69bbd4 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -152,43 +152,6 @@ rte_auxiliary_driver_remove_dev(struct rte_auxiliary_device *dev)
return 0;
}
-/*
- * Scan the content of the auxiliary bus, and call the probe function for
- * all registered drivers to try to probe discovered devices.
- */
-static int
-auxiliary_probe(void)
-{
- struct rte_auxiliary_device *dev = NULL;
- size_t probed = 0, failed = 0;
-
- RTE_BUS_FOREACH_DEV(dev, &auxiliary_bus.bus) {
- struct rte_driver *drv = NULL;
- int ret;
-
- probed++;
-
-next_driver:
- drv = rte_bus_find_driver(&auxiliary_bus.bus, drv, &dev->device);
- if (drv == NULL)
- continue;
-
- ret = auxiliary_bus.bus.probe_device(drv, &dev->device);
- if (ret < 0) {
- if (ret != -EEXIST) {
- AUXILIARY_LOG(ERR, "Requested device %s cannot be used",
- dev->name);
- rte_errno = errno;
- failed++;
- }
- } else if (ret > 0) {
- goto next_driver;
- }
- }
-
- return (probed && probed == failed) ? -1 : 0;
-}
-
static int
auxiliary_parse(const char *name, void *addr)
{
@@ -302,7 +265,7 @@ auxiliary_get_iommu_class(void)
struct rte_auxiliary_bus auxiliary_bus = {
.bus = {
.scan = auxiliary_scan,
- .probe = auxiliary_probe,
+ .probe = rte_bus_generic_probe,
.cleanup = auxiliary_cleanup,
.find_device = rte_bus_generic_find_device,
.match = auxiliary_bus_match,
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index 64fb0a8534..e1405d1372 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -347,42 +347,6 @@ cdx_probe_device(struct rte_driver *drv, struct rte_device *dev)
return ret;
}
-/*
- * Scan the content of the CDX bus, and call the probe() function for
- * all registered drivers that have a matching entry in its id_table
- * for discovered devices.
- */
-static int
-cdx_probe(void)
-{
- struct rte_cdx_device *dev = NULL;
- size_t probed = 0, failed = 0;
-
- RTE_BUS_FOREACH_DEV(dev, &rte_cdx_bus.bus) {
- struct rte_driver *drv = NULL;
- int ret;
-
- probed++;
-
-next_driver:
- drv = rte_bus_find_driver(&rte_cdx_bus.bus, drv, &dev->device);
- if (drv == NULL)
- continue;
-
- ret = rte_cdx_bus.bus.probe_device(drv, &dev->device);
- if (ret < 0) {
- CDX_BUS_ERR("Requested device %s cannot be used",
- dev->name);
- rte_errno = errno;
- failed++;
- } else if (ret > 0) {
- goto next_driver;
- }
- }
-
- return (probed && probed == failed) ? -1 : 0;
-}
-
static int
cdx_parse(const char *name, void *addr)
{
@@ -489,7 +453,7 @@ cdx_get_iommu_class(void)
struct rte_cdx_bus rte_cdx_bus = {
.bus = {
.scan = cdx_scan,
- .probe = cdx_probe,
+ .probe = rte_bus_generic_probe,
.find_device = rte_bus_generic_find_device,
.match = cdx_bus_match,
.probe_device = cdx_probe_device,
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index 43d71cefa4..7f4a85a91d 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -779,31 +779,6 @@ rte_dpaa_bus_scan(void)
return 0;
}
-static int
-rte_dpaa_bus_probe(void)
-{
- struct rte_dpaa_device *dev;
-
- /* For each registered driver, and device, call the driver->probe */
- RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
- struct rte_driver *driver = NULL;
- int ret;
-
-next_driver:
- driver = rte_bus_find_driver(&rte_dpaa_bus.bus, driver, &dev->device);
- if (driver == NULL)
- continue;
-
- ret = rte_dpaa_bus.bus.probe_device(driver, &dev->device);
- if (ret < 0)
- DPAA_BUS_ERR("Failed to probe device %s", dev->name);
- else if (ret > 0)
- goto next_driver;
- }
-
- return 0;
-}
-
/*
* Get iommu class of DPAA2 devices on the bus.
*/
@@ -824,9 +799,6 @@ dpaa_bus_probe_device(struct rte_driver *drv, struct rte_device *dev)
struct rte_dpaa_driver *dpaa_drv = RTE_BUS_DRIVER(drv, *dpaa_drv);
int ret;
- if (rte_bus_device_is_ignored(&rte_dpaa_bus.bus, dpaa_dev->name))
- return 0;
-
ret = dpaa_drv->probe(dpaa_drv, dpaa_dev);
if (ret != 0) {
DPAA_BUS_ERR("unable to probe: %s", dpaa_dev->name);
@@ -890,7 +862,7 @@ RTE_FINI_PRIO(dpaa_cleanup, 102)
static struct rte_dpaa_bus rte_dpaa_bus = {
.bus = {
.scan = rte_dpaa_bus_scan,
- .probe = rte_dpaa_bus_probe,
+ .probe = rte_bus_generic_probe,
.parse = rte_dpaa_bus_parse,
.dev_compare = dpaa_bus_dev_compare,
.find_device = rte_bus_generic_find_device,
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 156f4e295f..29ef3b58cf 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -446,33 +446,6 @@ rte_fslmc_close(void)
return 0;
}
-static int
-rte_fslmc_probe(void)
-{
- struct rte_dpaa2_device *dev;
- int ret;
-
- RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
- struct rte_driver *driver = NULL;
-
- if (rte_bus_device_is_ignored(&rte_fslmc_bus.bus, dev->device.name))
- continue;
-
-next_driver:
- driver = rte_bus_find_driver(&rte_fslmc_bus.bus, driver, &dev->device);
- if (driver == NULL)
- continue;
-
- ret = rte_fslmc_bus.bus.probe_device(driver, &dev->device);
- if (ret < 0)
- DPAA2_BUS_ERR("Failed to probe device %s", dev->device.name);
- else if (ret > 0)
- goto next_driver;
- }
-
- return 0;
-}
-
/*register a fslmc bus based dpaa2 driver */
RTE_EXPORT_INTERNAL_SYMBOL(rte_fslmc_driver_register)
void
@@ -578,7 +551,7 @@ fslmc_bus_unplug(struct rte_device *rte_dev)
struct rte_fslmc_bus rte_fslmc_bus = {
.bus = {
.scan = rte_fslmc_scan,
- .probe = rte_fslmc_probe,
+ .probe = rte_bus_generic_probe,
.cleanup = rte_fslmc_close,
.parse = rte_fslmc_parse,
.dev_compare = fslmc_dev_compare,
diff --git a/drivers/bus/ifpga/bus_ifpga_driver.h b/drivers/bus/ifpga/bus_ifpga_driver.h
index c1ff38bdb2..b56916167c 100644
--- a/drivers/bus/ifpga/bus_ifpga_driver.h
+++ b/drivers/bus/ifpga/bus_ifpga_driver.h
@@ -99,15 +99,6 @@ struct rte_afu_driver {
const struct rte_afu_uuid *id_table; /**< AFU uuid within FPGA. */
};
-__rte_internal
-static inline const char *
-rte_ifpga_device_name(const struct rte_afu_device *afu)
-{
- if (afu && afu->device.name)
- return afu->device.name;
- return NULL;
-}
-
/**
* Register a ifpga afu device driver.
*
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index d3f3370bc5..b2eb4f4413 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -287,38 +287,6 @@ ifpga_probe_device(struct rte_driver *drv, struct rte_device *dev)
return ret;
}
-/*
- * Scan the content of the Intel FPGA bus, and call the probe() function for
- * all registered drivers that have a matching entry in its id_table
- * for discovered devices.
- */
-static int
-ifpga_probe(void)
-{
- struct rte_afu_device *afu_dev = NULL;
- int ret = 0;
-
- RTE_BUS_FOREACH_DEV(afu_dev, &rte_ifpga_bus) {
- struct rte_driver *drv = NULL;
-
-next_driver:
- drv = rte_bus_find_driver(&rte_ifpga_bus, drv, &afu_dev->device);
- if (drv == NULL)
- continue;
-
- ret = rte_ifpga_bus.probe_device(drv, &afu_dev->device);
- if (ret == -EEXIST)
- continue;
- if (ret < 0)
- IFPGA_BUS_ERR("failed to initialize %s device",
- rte_ifpga_device_name(afu_dev));
- else if (ret > 0)
- goto next_driver;
- }
-
- return ret;
-}
-
/*
* Cleanup the content of the Intel FPGA bus, and call the remove() function
* for all registered devices.
@@ -421,7 +389,7 @@ ifpga_parse(const char *name, void *addr)
static struct rte_bus rte_ifpga_bus = {
.scan = ifpga_scan,
- .probe = ifpga_probe,
+ .probe = rte_bus_generic_probe,
.cleanup = ifpga_cleanup,
.find_device = rte_bus_generic_find_device,
.match = ifpga_bus_match,
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 02542a903a..a507360ef6 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -334,46 +334,6 @@ rte_pci_detach_dev(struct rte_pci_device *dev)
return 0;
}
-/*
- * Scan the content of the PCI bus, and call the probe() function for
- * all registered drivers that have a matching entry in its id_table
- * for discovered devices.
- */
-static int
-pci_probe(void)
-{
- struct rte_pci_device *dev = NULL;
- size_t probed = 0, failed = 0;
-
- RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus.bus) {
- struct rte_driver *drv = NULL;
- int ret;
-
- probed++;
-
-next_driver:
- drv = rte_bus_find_driver(&rte_pci_bus.bus, drv, &dev->device);
- if (drv == NULL)
- continue;
-
- ret = rte_pci_bus.bus.probe_device(drv, &dev->device);
- if (ret < 0) {
- if (ret != -EEXIST) {
- PCI_LOG(ERR, "Requested device " PCI_PRI_FMT " cannot be used",
- dev->addr.domain, dev->addr.bus,
- dev->addr.devid, dev->addr.function);
- rte_errno = errno;
- failed++;
- }
- ret = 0;
- } else if (ret > 0) {
- goto next_driver;
- }
- }
-
- return (probed && probed == failed) ? -1 : 0;
-}
-
static int
pci_cleanup(void)
{
@@ -825,7 +785,7 @@ struct rte_pci_bus rte_pci_bus = {
.bus = {
.allow_multi_probe = true,
.scan = rte_pci_scan,
- .probe = pci_probe,
+ .probe = rte_bus_generic_probe,
.cleanup = pci_cleanup,
.find_device = rte_bus_generic_find_device,
.match = pci_bus_match,
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 3d04ba4d25..01c8239cbb 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -401,40 +401,9 @@ platform_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
return match;
}
-static int
-platform_bus_probe(void)
-{
- struct rte_platform_device *pdev;
-
- RTE_BUS_FOREACH_DEV(pdev, &platform_bus.bus) {
- struct rte_driver *drv = NULL;
- int ret;
-
-next_driver:
- drv = rte_bus_find_driver(&platform_bus.bus, drv, &pdev->device);
- if (drv == NULL)
- continue;
-
- ret = platform_bus.bus.probe_device(drv, &pdev->device);
- if (ret == -EBUSY) {
- PLATFORM_LOG_LINE(DEBUG, "device %s already probed", pdev->name);
- continue;
- }
- if (ret < 0)
- PLATFORM_LOG_LINE(ERR, "failed to probe %s", pdev->name);
- else if (ret > 0)
- goto next_driver;
- }
-
- return 0;
-}
-
static int
platform_bus_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
- if (rte_bus_device_is_ignored(&platform_bus.bus, dev->name))
- return -EPERM;
-
if (!dev_is_bound_vfio_platform(dev->name))
return -EPERM;
@@ -547,7 +516,7 @@ platform_bus_cleanup(void)
struct rte_platform_bus platform_bus = {
.bus = {
.scan = platform_bus_scan,
- .probe = platform_bus_probe,
+ .probe = rte_bus_generic_probe,
.find_device = rte_bus_generic_find_device,
.match = platform_bus_match,
.probe_device = platform_bus_probe_device,
diff --git a/drivers/bus/uacce/uacce.c b/drivers/bus/uacce/uacce.c
index b34c263295..463d08bc12 100644
--- a/drivers/bus/uacce/uacce.c
+++ b/drivers/bus/uacce/uacce.c
@@ -375,37 +375,6 @@ uacce_probe_device(struct rte_driver *drv, struct rte_device *dev)
return ret;
}
-static int
-uacce_probe(void)
-{
- size_t probed = 0, failed = 0;
- struct rte_uacce_device *dev;
-
- RTE_BUS_FOREACH_DEV(dev, &uacce_bus.bus) {
- struct rte_driver *drv = NULL;
- int ret;
-
- probed++;
-
-next_driver:
- drv = rte_bus_find_driver(&uacce_bus.bus, drv, &dev->device);
- if (drv == NULL)
- continue;
-
- ret = uacce_bus.bus.probe_device(drv, &dev->device);
- if (ret < 0) {
- UACCE_BUS_LOG(ERR, "Requested device %s cannot be used",
- dev->name);
- rte_errno = errno;
- failed++;
- } else if (ret > 0) {
- goto next_driver;
- }
- }
-
- return (probed && probed == failed) ? -1 : 0;
-}
-
static int
uacce_cleanup(void)
{
@@ -597,7 +566,7 @@ rte_uacce_unregister(struct rte_uacce_driver *driver)
static struct rte_uacce_bus uacce_bus = {
.bus = {
.scan = uacce_scan,
- .probe = uacce_probe,
+ .probe = rte_bus_generic_probe,
.cleanup = uacce_cleanup,
.match = uacce_bus_match,
.probe_device = uacce_probe_device,
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 37820b9614..3121db1f79 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -546,41 +546,6 @@ vdev_scan(void)
return 0;
}
-static int
-vdev_probe(void)
-{
- struct rte_vdev_device *dev;
- int r, ret = 0;
-
- /* call the init function for each virtual device */
- RTE_BUS_FOREACH_DEV(dev, &rte_vdev_bus) {
- struct rte_driver *drv = NULL;
-
- /* we don't use the vdev lock here, as it's only used in DPDK
- * initialization; and we don't want to hold such a lock when
- * we call each driver probe.
- */
-
-next_driver:
- drv = rte_bus_find_driver(&rte_vdev_bus, drv, &dev->device);
- if (drv == NULL)
- continue;
-
- r = rte_vdev_bus.probe_device(drv, &dev->device);
- if (r < 0) {
- if (r == -EEXIST)
- continue;
- VDEV_LOG(ERR, "failed to initialize %s device",
- rte_vdev_device_name(dev));
- ret = -1;
- } else if (r > 0) {
- goto next_driver;
- }
- }
-
- return ret;
-}
-
static int
vdev_cleanup(void)
{
@@ -652,7 +617,7 @@ vdev_get_iommu_class(void)
static struct rte_bus rte_vdev_bus = {
.scan = vdev_scan,
- .probe = vdev_probe,
+ .probe = rte_bus_generic_probe,
.cleanup = vdev_cleanup,
.find_device = vdev_find_device,
.match = vdev_bus_match,
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 24b1c24f43..bd375ae6bb 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -135,38 +135,7 @@ RTE_EXPORT_SYMBOL(rte_vmbus_probe)
int
rte_vmbus_probe(void)
{
- struct rte_vmbus_device *dev;
- size_t probed = 0, failed = 0;
- char ubuf[RTE_UUID_STRLEN];
-
- RTE_BUS_FOREACH_DEV(dev, &rte_vmbus_bus.bus) {
- struct rte_driver *drv = NULL;
- int ret;
-
- probed++;
-
- rte_uuid_unparse(dev->device_id, ubuf, sizeof(ubuf));
-
- if (rte_bus_device_is_ignored(&rte_vmbus_bus.bus, ubuf))
- continue;
-
-next_driver:
- drv = rte_bus_find_driver(&rte_vmbus_bus.bus, drv, &dev->device);
- if (drv == NULL)
- continue;
-
- ret = rte_vmbus_bus.bus.probe_device(drv, &dev->device);
- if (ret < 0) {
- VMBUS_LOG(NOTICE,
- "Requested device %s cannot be used", ubuf);
- rte_errno = errno;
- failed++;
- } else if (ret > 0) {
- goto next_driver;
- }
- }
-
- return (probed && probed == failed) ? -1 : 0;
+ return rte_bus_generic_probe(&rte_vmbus_bus.bus);
}
static int
@@ -247,7 +216,7 @@ rte_vmbus_unregister(struct rte_vmbus_driver *driver)
struct rte_vmbus_bus rte_vmbus_bus = {
.bus = {
.scan = rte_vmbus_scan,
- .probe = rte_vmbus_probe,
+ .probe = rte_bus_generic_probe,
.cleanup = rte_vmbus_cleanup,
.find_device = rte_bus_generic_find_device,
.match = vmbus_bus_match,
diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c
index 493ddc0d1e..1a9c07580e 100644
--- a/drivers/dma/idxd/idxd_bus.c
+++ b/drivers/dma/idxd/idxd_bus.c
@@ -45,7 +45,6 @@ struct dsa_bus;
static int dsa_scan(void);
static bool dsa_match(const struct rte_driver *drv, const struct rte_device *dev);
static int dsa_probe_device(struct rte_driver *drv, struct rte_device *dev);
-static int dsa_probe(void);
static enum rte_iova_mode dsa_get_iommu_class(void);
static int dsa_addr_parse(const char *name, void *addr);
@@ -60,9 +59,9 @@ struct dsa_bus {
struct dsa_bus dsa_bus = {
.bus = {
.scan = dsa_scan,
+ .probe = rte_bus_generic_probe,
.match = dsa_match,
.probe_device = dsa_probe_device,
- .probe = dsa_probe,
.find_device = rte_bus_generic_find_device,
.get_iommu_class = dsa_get_iommu_class,
.parse = dsa_addr_parse,
@@ -264,24 +263,6 @@ is_for_this_process_use(const char *name)
return retval;
}
-static int
-dsa_probe(void)
-{
- struct rte_dsa_device *dev;
-
- RTE_BUS_FOREACH_DEV(dev, &dsa_bus.bus) {
- if (dsa_match(&dsa_bus.driver, &dev->device) &&
- !rte_bus_device_is_ignored(&dsa_bus.bus, dev->device.name)) {
- dev->device.driver = &dsa_bus.driver;
- dsa_probe_device(&dsa_bus.driver, &dev->device);
- continue;
- }
- IDXD_PMD_DEBUG("WQ '%s', not allocated to DPDK", dev->wq_name);
- }
-
- return 0;
-}
-
static bool dsa_match(const struct rte_driver *drv, const struct rte_device *dev)
{
const struct rte_dsa_device *dsa_dev = RTE_BUS_DEVICE(dev, *dsa_dev);
diff --git a/lib/eal/common/eal_common_bus.c b/lib/eal/common/eal_common_bus.c
index 88c417e8d3..94b7be5f5f 100644
--- a/lib/eal/common/eal_common_bus.c
+++ b/lib/eal/common/eal_common_bus.c
@@ -72,6 +72,51 @@ rte_bus_scan(void)
return 0;
}
+/*
+ * Generic probe function for buses.
+ * Iterates through all devices on the bus, finds matching drivers,
+ * and calls bus->probe_device() for each device.
+ */
+RTE_EXPORT_INTERNAL_SYMBOL(rte_bus_generic_probe)
+int
+rte_bus_generic_probe(struct rte_bus *bus)
+{
+ size_t probed = 0, failed = 0;
+ struct rte_device *dev;
+
+ TAILQ_FOREACH(dev, &bus->device_list, next) {
+ struct rte_driver *drv = NULL;
+ int ret;
+
+ if (rte_bus_device_is_ignored(bus, dev->name))
+ continue;
+
+ if (rte_dev_is_probed(dev) && !bus->allow_multi_probe) {
+ EAL_LOG(INFO, "Device %s is already probed", dev->name);
+ continue;
+ }
+
+ probed++;
+next_driver:
+ drv = rte_bus_find_driver(bus, drv, dev);
+ if (drv == NULL)
+ continue;
+
+ ret = bus->probe_device(drv, dev);
+ if (ret > 0)
+ goto next_driver;
+
+ if (ret < 0) {
+ if (ret == -EEXIST)
+ continue;
+ EAL_LOG(ERR, "Failed to probe device %s", dev->name);
+ failed++;
+ }
+ }
+
+ return (probed && probed == failed) ? -1 : 0;
+}
+
/* Probe all devices of all buses */
RTE_EXPORT_SYMBOL(rte_bus_probe)
int
@@ -86,14 +131,14 @@ rte_bus_probe(void)
continue;
}
- ret = bus->probe();
+ ret = bus->probe(bus);
if (ret)
EAL_LOG(ERR, "Bus (%s) probe failed.",
rte_bus_name(bus));
}
if (vbus) {
- ret = vbus->probe();
+ ret = vbus->probe(vbus);
if (ret)
EAL_LOG(ERR, "Bus (%s) probe failed.",
rte_bus_name(vbus));
diff --git a/lib/eal/include/bus_driver.h b/lib/eal/include/bus_driver.h
index a2fd3bd82d..3d04efbd3f 100644
--- a/lib/eal/include/bus_driver.h
+++ b/lib/eal/include/bus_driver.h
@@ -40,11 +40,14 @@ typedef int (*rte_bus_scan_t)(void);
*
* This is called while iterating over each registered bus.
*
+ * @param bus
+ * A pointer to the bus structure.
+ *
* @return
* 0 for successful probe
* !0 for any error while probing
*/
-typedef int (*rte_bus_probe_t)(void);
+typedef int (*rte_bus_probe_t)(struct rte_bus *bus);
/**
* Device iterator to find a device on a bus.
@@ -598,6 +601,24 @@ __rte_internal
struct rte_driver *rte_bus_find_driver(const struct rte_bus *bus, const struct rte_driver *start,
const struct rte_device *dev);
+/**
+ * Generic probe function for buses.
+ *
+ * Iterates through all devices on the bus, finds matching drivers using
+ * rte_bus_find_driver(), and calls bus->probe_device() for each device that has
+ * a matching driver.
+ *
+ * This function can be used by buses that don't require special probe
+ * logic and just need the standard device iteration and driver matching.
+ *
+ * @param bus
+ * Pointer to the bus to probe.
+ * @return
+ * 0 on success.
+ */
+__rte_internal
+int rte_bus_generic_probe(struct rte_bus *bus);
+
#ifdef __cplusplus
}
#endif
--
2.53.0
^ permalink raw reply related
* [PATCH v3 16/25] drivers/bus: initialize NXP bus specifics in scan
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Hemant Agrawal, Sachin Saxena,
Anatoly Burakov
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
Move bus-specific initialization code from probe() to scan() operations
for DPAA and FSLMC buses. This separates device discovery and bus setup
(scan) from driver matching and probing (probe), preparing for a future
generic probe() implementation in EAL.
Both buses now have a clean separation:
- scan(): discover devices + initialize bus
- probe(): match drivers + call probe_device()
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/dpaa/dpaa_bus.c | 76 +++++++++++++++---------------
drivers/bus/fslmc/fslmc_bus.c | 88 +++++++++++++++++------------------
2 files changed, 80 insertions(+), 84 deletions(-)
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index b0ed61ba39..43d71cefa4 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -569,40 +569,6 @@ dpaa_bus_dev_compare(const char *name1, const char *name2)
return strncmp(devname1, devname2, sizeof(devname1));
}
-#define DPAA_DEV_PATH1 "/sys/devices/platform/soc/soc:fsl,dpaa"
-#define DPAA_DEV_PATH2 "/sys/devices/platform/fsl,dpaa"
-
-static int
-rte_dpaa_bus_scan(void)
-{
- int ret;
-
- BUS_INIT_FUNC_TRACE();
-
- if ((access(DPAA_DEV_PATH1, F_OK) != 0) &&
- (access(DPAA_DEV_PATH2, F_OK) != 0)) {
- DPAA_BUS_LOG(DEBUG, "DPAA Bus not present. Skipping.");
- return 0;
- }
-
- if (rte_dpaa_bus.detected)
- return 0;
-
- rte_dpaa_bus.detected = 1;
-
- /* create the key, supplying a function that'll be invoked
- * when a portal affined thread will be deleted.
- */
- ret = pthread_key_create(&dpaa_portal_key, dpaa_portal_finish);
- if (ret) {
- DPAA_BUS_LOG(DEBUG, "Unable to create pthread key. (%d)", ret);
- dpaa_clean_device_list();
- return ret;
- }
-
- return 0;
-}
-
/* register a dpaa bus based dpaa driver */
RTE_EXPORT_INTERNAL_SYMBOL(rte_dpaa_driver_register)
void
@@ -701,20 +667,43 @@ static int rte_dpaa_setup_intr(struct rte_intr_handle *intr_handle)
return 0;
}
+#define DPAA_DEV_PATH1 "/sys/devices/platform/soc/soc:fsl,dpaa"
+#define DPAA_DEV_PATH2 "/sys/devices/platform/fsl,dpaa"
+
static int
-rte_dpaa_bus_probe(void)
+rte_dpaa_bus_scan(void)
{
- int ret = -1;
struct rte_dpaa_device *dev;
FILE *svr_file = NULL;
uint32_t svr_ver;
static int process_once;
char *penv;
+ int ret;
+
+ BUS_INIT_FUNC_TRACE();
+
+ if ((access(DPAA_DEV_PATH1, F_OK) != 0) &&
+ (access(DPAA_DEV_PATH2, F_OK) != 0)) {
+ DPAA_BUS_LOG(DEBUG, "DPAA Bus not present. Skipping.");
+ return 0;
+ }
- /* If DPAA bus is not present nothing needs to be done */
- if (!rte_dpaa_bus.detected)
+ if (rte_dpaa_bus.detected)
return 0;
+ rte_dpaa_bus.detected = 1;
+
+ /* create the key, supplying a function that'll be invoked
+ * when a portal affined thread will be deleted.
+ */
+ ret = pthread_key_create(&dpaa_portal_key, dpaa_portal_finish);
+ if (ret) {
+ DPAA_BUS_LOG(DEBUG, "Unable to create pthread key. (%d)", ret);
+ dpaa_clean_device_list();
+ return ret;
+ }
+
+ /* SoC version detection and configuration */
svr_file = fopen(DPAA_SOC_ID_FILE, "r");
if (svr_file) {
if (fscanf(svr_file, "svr:%x", &svr_ver) > 0)
@@ -786,9 +775,19 @@ rte_dpaa_bus_probe(void)
/* And initialize the PA->VA translation table */
dpaax_iova_table_populate();
+ dpaa_bus_global_init = 1;
+ return 0;
+}
+
+static int
+rte_dpaa_bus_probe(void)
+{
+ struct rte_dpaa_device *dev;
+
/* For each registered driver, and device, call the driver->probe */
RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
struct rte_driver *driver = NULL;
+ int ret;
next_driver:
driver = rte_bus_find_driver(&rte_dpaa_bus.bus, driver, &dev->device);
@@ -801,7 +800,6 @@ rte_dpaa_bus_probe(void)
else if (ret > 0)
goto next_driver;
}
- dpaa_bus_global_init = 1;
return 0;
}
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index e6db8f5100..156f4e295f 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -322,7 +322,6 @@ rte_fslmc_scan(void)
}
return 0;
}
- process_once = 1;
/* Now we only support single group per process.*/
group_name = getenv("DPRC");
@@ -368,6 +367,48 @@ rte_fslmc_scan(void)
/* If debugging is enabled, device list is dumped to log output */
dump_device_list();
+ /* Bus initialization - only if devices were found */
+ if (!TAILQ_EMPTY(&rte_fslmc_bus.bus.device_list)) {
+ static const struct rte_mbuf_dynfield dpaa2_seqn_dynfield_desc = {
+ .name = DPAA2_SEQN_DYNFIELD_NAME,
+ .size = sizeof(dpaa2_seqn_t),
+ .align = alignof(dpaa2_seqn_t),
+ };
+
+ dpaa2_seqn_dynfield_offset =
+ rte_mbuf_dynfield_register(&dpaa2_seqn_dynfield_desc);
+ if (dpaa2_seqn_dynfield_offset < 0) {
+ DPAA2_BUS_ERR("Failed to register mbuf field for dpaa sequence number");
+ return 0;
+ }
+
+ ret = fslmc_vfio_setup_group();
+ if (ret) {
+ DPAA2_BUS_ERR("Unable to setup VFIO %d", ret);
+ return 0;
+ }
+
+ /* Map existing segments as well as, in case of hotpluggable memory,
+ * install callback handler.
+ */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ ret = fslmc_vfio_dmamap();
+ if (ret) {
+ DPAA2_BUS_ERR("Unable to DMA map existing VAs: (%d)", ret);
+ DPAA2_BUS_ERR("FSLMC VFIO Mapping failed");
+ return 0;
+ }
+ }
+
+ ret = fslmc_vfio_process_group();
+ if (ret) {
+ DPAA2_BUS_ERR("Unable to setup devices %d", ret);
+ return 0;
+ }
+ }
+
+ process_once = 1;
+
return 0;
scan_fail_cleanup:
@@ -408,51 +449,8 @@ rte_fslmc_close(void)
static int
rte_fslmc_probe(void)
{
- int ret = 0;
-
struct rte_dpaa2_device *dev;
-
- static const struct rte_mbuf_dynfield dpaa2_seqn_dynfield_desc = {
- .name = DPAA2_SEQN_DYNFIELD_NAME,
- .size = sizeof(dpaa2_seqn_t),
- .align = alignof(dpaa2_seqn_t),
- };
-
- if (TAILQ_EMPTY(&rte_fslmc_bus.bus.device_list))
- return 0;
-
- dpaa2_seqn_dynfield_offset =
- rte_mbuf_dynfield_register(&dpaa2_seqn_dynfield_desc);
- if (dpaa2_seqn_dynfield_offset < 0) {
- DPAA2_BUS_ERR("Failed to register mbuf field for dpaa sequence number");
- return 0;
- }
-
- ret = fslmc_vfio_setup_group();
- if (ret) {
- DPAA2_BUS_ERR("Unable to setup VFIO %d", ret);
- return 0;
- }
-
- /* Map existing segments as well as, in case of hotpluggable memory,
- * install callback handler.
- */
- if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
- ret = fslmc_vfio_dmamap();
- if (ret) {
- DPAA2_BUS_ERR("Unable to DMA map existing VAs: (%d)",
- ret);
- /* Not continuing ahead */
- DPAA2_BUS_ERR("FSLMC VFIO Mapping failed");
- return 0;
- }
- }
-
- ret = fslmc_vfio_process_group();
- if (ret) {
- DPAA2_BUS_ERR("Unable to setup devices %d", ret);
- return 0;
- }
+ int ret;
RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
struct rte_driver *driver = NULL;
--
2.53.0
^ permalink raw reply related
* [PATCH v3 15/25] bus: support multiple probe
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Rosen Xu, Chenbo Xia, Tomasz Duszynski, Chengwen Feng, Long Li,
Wei Hu
In-Reply-To: <20260526085257.3148516-1-david.marchand@redhat.com>
Add infrastructure to declare support for multiple probe attempts
on the same device. This prepares for the introduction of
generic probe_device operation.
The PCI bus enables this feature to support drivers with
RTE_PCI_DRV_PROBE_AGAIN flag.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/auxiliary/auxiliary_common.c | 6 ------
drivers/bus/cdx/cdx.c | 5 -----
drivers/bus/dpaa/dpaa_bus.c | 3 ---
drivers/bus/fslmc/fslmc_bus.c | 3 ---
drivers/bus/ifpga/ifpga_bus.c | 7 -------
drivers/bus/pci/pci_common.c | 1 +
drivers/bus/platform/platform.c | 3 ---
drivers/bus/uacce/uacce.c | 5 -----
drivers/bus/vdev/vdev.c | 3 ---
drivers/bus/vmbus/vmbus_common.c | 6 ------
lib/eal/common/eal_common_dev.c | 3 +++
lib/eal/include/bus_driver.h | 1 +
12 files changed, 5 insertions(+), 41 deletions(-)
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index 7824c26f92..c210e303ce 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -94,12 +94,6 @@ auxiliary_probe_device(struct rte_driver *drv, struct rte_device *dev)
if (aux_dev->device.numa_node < 0 && rte_socket_count() > 1)
AUXILIARY_LOG(INFO, "Device %s is not NUMA-aware", aux_dev->name);
- if (rte_dev_is_probed(&aux_dev->device)) {
- AUXILIARY_LOG(DEBUG, "Device %s is already probed on auxiliary bus",
- aux_dev->device.name);
- return -EEXIST;
- }
-
iova_mode = rte_eal_iova_mode();
if ((aux_drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0 &&
iova_mode != RTE_IOVA_VA) {
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index c38eae325b..64fb0a8534 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -315,11 +315,6 @@ cdx_probe_device(struct rte_driver *drv, struct rte_device *dev)
const char *dev_name = cdx_dev->name;
int ret;
- if (rte_dev_is_probed(&cdx_dev->device)) {
- CDX_BUS_INFO("Device %s is already probed", dev_name);
- return -EEXIST;
- }
-
CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
cdx_drv->driver.name);
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index 14cd64cc32..b0ed61ba39 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -826,9 +826,6 @@ dpaa_bus_probe_device(struct rte_driver *drv, struct rte_device *dev)
struct rte_dpaa_driver *dpaa_drv = RTE_BUS_DRIVER(drv, *dpaa_drv);
int ret;
- if (rte_dev_is_probed(&dpaa_dev->device))
- return 0;
-
if (rte_bus_device_is_ignored(&rte_dpaa_bus.bus, dpaa_dev->name))
return 0;
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index a975e464c1..e6db8f5100 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -541,9 +541,6 @@ fslmc_bus_probe_device(struct rte_driver *driver, struct rte_device *rte_dev)
struct rte_dpaa2_driver *drv = RTE_BUS_DRIVER(driver, *drv);
int ret = 0;
- if (rte_dev_is_probed(&dev->device))
- return 0;
-
if (dev->device.devargs &&
dev->device.devargs->policy == RTE_DEV_BLOCKED) {
DPAA2_BUS_DEBUG("%s Blocked, skipping",
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index 92ad3513e0..d3f3370bc5 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -274,13 +274,6 @@ ifpga_probe_device(struct rte_driver *drv, struct rte_device *dev)
struct rte_afu_driver *afu_drv = RTE_BUS_DRIVER(drv, *afu_drv);
int ret;
- /* Check if a driver is already loaded */
- if (rte_dev_is_probed(&afu_dev->device)) {
- IFPGA_BUS_DEBUG("Device %s is already probed",
- rte_ifpga_device_name(afu_dev));
- return -EEXIST;
- }
-
/* reference driver structure */
afu_dev->driver = afu_drv;
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index b57320064e..02542a903a 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -823,6 +823,7 @@ rte_pci_pasid_set_state(const struct rte_pci_device *dev,
struct rte_pci_bus rte_pci_bus = {
.bus = {
+ .allow_multi_probe = true,
.scan = rte_pci_scan,
.probe = pci_probe,
.cleanup = pci_cleanup,
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 22979f31b3..3d04ba4d25 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -331,9 +331,6 @@ driver_call_probe(struct rte_platform_driver *pdrv, struct rte_platform_device *
{
int ret;
- if (rte_dev_is_probed(&pdev->device))
- return -EBUSY;
-
if (pdrv->probe != NULL) {
pdev->driver = pdrv;
ret = pdrv->probe(pdev);
diff --git a/drivers/bus/uacce/uacce.c b/drivers/bus/uacce/uacce.c
index 2ef1a27635..b34c263295 100644
--- a/drivers/bus/uacce/uacce.c
+++ b/drivers/bus/uacce/uacce.c
@@ -359,11 +359,6 @@ uacce_probe_device(struct rte_driver *drv, struct rte_device *dev)
const char *dev_name = uacce_dev->name;
int ret;
- if (rte_dev_is_probed(&uacce_dev->device)) {
- UACCE_BUS_INFO("device %s is already probed", dev_name);
- return -EEXIST;
- }
-
UACCE_BUS_DEBUG("probe device %s using driver %s", dev_name, uacce_drv->driver.name);
ret = uacce_drv->probe(uacce_drv, uacce_dev);
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index e55be7fb2c..37820b9614 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -190,9 +190,6 @@ vdev_probe_device(struct rte_driver *drv, struct rte_device *dev)
enum rte_iova_mode iova_mode;
int ret;
- if (rte_dev_is_probed(&vdev_dev->device))
- return -EEXIST;
-
name = rte_vdev_device_name(vdev_dev);
VDEV_LOG(DEBUG, "Search driver to probe device %s", name);
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index ba923a2669..24b1c24f43 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -91,12 +91,6 @@ vmbus_probe_device(struct rte_driver *drv, struct rte_device *dev)
char guid[RTE_UUID_STRLEN];
int ret;
- /* Check if a driver is already loaded */
- if (rte_dev_is_probed(&vmbus_dev->device)) {
- VMBUS_LOG(DEBUG, "VMBUS driver already loaded");
- return 0;
- }
-
rte_uuid_unparse(vmbus_dev->device_id, guid, sizeof(guid));
VMBUS_LOG(INFO, "VMBUS device %s on NUMA socket %i",
guid, vmbus_dev->device.numa_node);
diff --git a/lib/eal/common/eal_common_dev.c b/lib/eal/common/eal_common_dev.c
index a38c211e5d..9047a01c4a 100644
--- a/lib/eal/common/eal_common_dev.c
+++ b/lib/eal/common/eal_common_dev.c
@@ -225,6 +225,9 @@ local_dev_probe(const char *devargs, struct rte_device **new_dev)
drv = rte_bus_find_driver(dev->bus, drv, dev);
if (drv == NULL) {
ret = -ENOTSUP;
+ } else if (rte_dev_is_probed(dev) && !dev->bus->allow_multi_probe) {
+ EAL_LOG(INFO, "Device %s is already probed", dev->name);
+ ret = -EEXIST;
} else {
ret = dev->bus->probe_device(drv, dev);
if (ret > 0)
diff --git a/lib/eal/include/bus_driver.h b/lib/eal/include/bus_driver.h
index 0db4473bcc..a2fd3bd82d 100644
--- a/lib/eal/include/bus_driver.h
+++ b/lib/eal/include/bus_driver.h
@@ -327,6 +327,7 @@ typedef void *(*rte_bus_dev_iterate_t)(const struct rte_bus *bus,
struct rte_bus {
RTE_TAILQ_ENTRY(rte_bus) next; /**< Next bus object in linked list */
const char *name; /**< Name of the bus */
+ bool allow_multi_probe; /**< Allow probing devices multiple times */
rte_bus_scan_t scan; /**< Scan for devices attached to bus */
rte_bus_probe_t probe; /**< Probe devices on bus */
rte_bus_find_device_t find_device; /**< Find a device on the bus */
--
2.53.0
^ permalink raw reply related
* [PATCH v3 14/25] bus: refactor device probe
From: David Marchand @ 2026-05-26 8:52 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Rosen Xu, Chenbo Xia, Tomasz Duszynski, Chengwen Feng, Long Li,
Wei Hu, Kevin Laatz
In-Reply-To: <20260526084212.3145685-1-david.marchand@redhat.com>
Introduce a new rte_bus_probe_device_t operation with signature
(struct rte_driver *drv, struct rte_device *dev).
Replace the existing .plug field in the struct rte_bus with .probe_device.
Update all in-tree buses to use .probe_device instead of .plug.
Each bus probe() function now calls rte_bus_find_driver() (which uses the
match operation added in previous commit) and passes the found driver
to bus.probe_device(driver, device).
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/auxiliary/auxiliary_common.c | 98 ++++++---------
drivers/bus/cdx/cdx.c | 81 +++++--------
drivers/bus/dpaa/dpaa_bus.c | 63 +++++-----
drivers/bus/fslmc/fslmc_bus.c | 73 +++++-------
drivers/bus/ifpga/ifpga_bus.c | 68 ++++-------
drivers/bus/pci/pci_common.c | 145 +++++++++--------------
drivers/bus/platform/platform.c | 38 +++---
drivers/bus/uacce/uacce.c | 67 ++++-------
drivers/bus/vdev/vdev.c | 58 +++++----
drivers/bus/vmbus/vmbus_common.c | 84 ++++++-------
drivers/dma/idxd/idxd_bus.c | 23 ++--
lib/eal/common/eal_common_bus.c | 5 +-
lib/eal/common/eal_common_dev.c | 14 ++-
lib/eal/include/bus_driver.h | 17 ++-
14 files changed, 354 insertions(+), 480 deletions(-)
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index 21b5bcb416..7824c26f92 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -74,61 +74,60 @@ auxiliary_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
* Call the probe() function of the driver.
*/
static int
-rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
- struct rte_auxiliary_device *dev)
+auxiliary_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
+ struct rte_auxiliary_device *aux_dev = RTE_BUS_DEVICE(dev, *aux_dev);
+ struct rte_auxiliary_driver *aux_drv = RTE_BUS_DRIVER(drv, *aux_drv);
enum rte_iova_mode iova_mode;
int ret;
- /* Check if driver supports it. */
- if (!auxiliary_bus_match(&drv->driver, &dev->device))
- /* Match of device and driver failed */
- return 1;
+ if (!auxiliary_dev_exists(dev->name))
+ return -ENOENT;
/* No initialization when marked as blocked, return without error. */
- if (dev->device.devargs != NULL &&
- dev->device.devargs->policy == RTE_DEV_BLOCKED) {
+ if (aux_dev->device.devargs != NULL &&
+ aux_dev->device.devargs->policy == RTE_DEV_BLOCKED) {
AUXILIARY_LOG(INFO, "Device is blocked, not initializing");
return -1;
}
- if (dev->device.numa_node < 0 && rte_socket_count() > 1)
- AUXILIARY_LOG(INFO, "Device %s is not NUMA-aware", dev->name);
+ if (aux_dev->device.numa_node < 0 && rte_socket_count() > 1)
+ AUXILIARY_LOG(INFO, "Device %s is not NUMA-aware", aux_dev->name);
- if (rte_dev_is_probed(&dev->device)) {
+ if (rte_dev_is_probed(&aux_dev->device)) {
AUXILIARY_LOG(DEBUG, "Device %s is already probed on auxiliary bus",
- dev->device.name);
+ aux_dev->device.name);
return -EEXIST;
}
iova_mode = rte_eal_iova_mode();
- if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0 &&
+ if ((aux_drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0 &&
iova_mode != RTE_IOVA_VA) {
AUXILIARY_LOG(ERR, "Driver %s expecting VA IOVA mode but current mode is PA, not initializing",
- drv->driver.name);
+ aux_drv->driver.name);
return -EINVAL;
}
/* Allocate interrupt instance */
- dev->intr_handle =
+ aux_dev->intr_handle =
rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
- if (dev->intr_handle == NULL) {
+ if (aux_dev->intr_handle == NULL) {
AUXILIARY_LOG(ERR, "Could not allocate interrupt instance for device %s",
- dev->name);
+ aux_dev->name);
return -ENOMEM;
}
- dev->driver = drv;
+ aux_dev->driver = aux_drv;
AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
- drv->driver.name, dev->name, dev->device.numa_node);
- ret = drv->probe(drv, dev);
+ aux_drv->driver.name, aux_dev->name, aux_dev->device.numa_node);
+ ret = aux_drv->probe(aux_drv, aux_dev);
if (ret != 0) {
- dev->driver = NULL;
- rte_intr_instance_free(dev->intr_handle);
- dev->intr_handle = NULL;
+ aux_dev->driver = NULL;
+ rte_intr_instance_free(aux_dev->intr_handle);
+ aux_dev->intr_handle = NULL;
} else {
- dev->device.driver = &drv->driver;
+ aux_dev->device.driver = &aux_drv->driver;
}
return ret;
@@ -159,33 +158,6 @@ rte_auxiliary_driver_remove_dev(struct rte_auxiliary_device *dev)
return 0;
}
-/*
- * Call the probe() function of all registered drivers for the given device.
- * Return < 0 if initialization failed.
- * Return 1 if no driver is found for this device.
- */
-static int
-auxiliary_probe_all_drivers(struct rte_auxiliary_device *dev)
-{
- struct rte_auxiliary_driver *drv;
- int rc;
-
- RTE_BUS_FOREACH_DRV(drv, &auxiliary_bus.bus) {
- if (!drv->match(dev->name))
- continue;
-
- rc = rte_auxiliary_probe_one_driver(drv, dev);
- if (rc < 0)
- /* negative value is an error */
- return rc;
- if (rc > 0)
- /* positive value means driver doesn't support it */
- continue;
- return 0;
- }
- return 1;
-}
-
/*
* Scan the content of the auxiliary bus, and call the probe function for
* all registered drivers to try to probe discovered devices.
@@ -195,12 +167,19 @@ auxiliary_probe(void)
{
struct rte_auxiliary_device *dev = NULL;
size_t probed = 0, failed = 0;
- int ret = 0;
RTE_BUS_FOREACH_DEV(dev, &auxiliary_bus.bus) {
+ struct rte_driver *drv = NULL;
+ int ret;
+
probed++;
- ret = auxiliary_probe_all_drivers(dev);
+next_driver:
+ drv = rte_bus_find_driver(&auxiliary_bus.bus, drv, &dev->device);
+ if (drv == NULL)
+ continue;
+
+ ret = auxiliary_bus.bus.probe_device(drv, &dev->device);
if (ret < 0) {
if (ret != -EEXIST) {
AUXILIARY_LOG(ERR, "Requested device %s cannot be used",
@@ -208,7 +187,8 @@ auxiliary_probe(void)
rte_errno = errno;
failed++;
}
- ret = 0;
+ } else if (ret > 0) {
+ goto next_driver;
}
}
@@ -250,14 +230,6 @@ rte_auxiliary_unregister(struct rte_auxiliary_driver *driver)
rte_bus_remove_driver(&auxiliary_bus.bus, &driver->driver);
}
-static int
-auxiliary_plug(struct rte_device *dev)
-{
- if (!auxiliary_dev_exists(dev->name))
- return -ENOENT;
- return auxiliary_probe_all_drivers(RTE_BUS_DEVICE(dev, struct rte_auxiliary_device));
-}
-
static int
auxiliary_unplug(struct rte_device *dev)
{
@@ -340,7 +312,7 @@ struct rte_auxiliary_bus auxiliary_bus = {
.cleanup = auxiliary_cleanup,
.find_device = rte_bus_generic_find_device,
.match = auxiliary_bus_match,
- .plug = auxiliary_plug,
+ .probe_device = auxiliary_probe_device,
.unplug = auxiliary_unplug,
.parse = auxiliary_parse,
.dma_map = auxiliary_dma_map,
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index c898ce9271..c38eae325b 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -308,29 +308,23 @@ cdx_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
* driver.
*/
static int
-cdx_probe_one_driver(struct rte_cdx_driver *dr,
- struct rte_cdx_device *dev)
+cdx_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
- const char *dev_name = dev->name;
- bool already_probed;
+ struct rte_cdx_device *cdx_dev = RTE_BUS_DEVICE(dev, *cdx_dev);
+ struct rte_cdx_driver *cdx_drv = RTE_BUS_DRIVER(drv, *cdx_drv);
+ const char *dev_name = cdx_dev->name;
int ret;
- /* The device is not blocked; Check if driver supports it */
- if (!cdx_bus_match(&dr->driver, &dev->device))
- /* Match of device and driver failed */
- return 1;
-
- already_probed = rte_dev_is_probed(&dev->device);
- if (already_probed) {
+ if (rte_dev_is_probed(&cdx_dev->device)) {
CDX_BUS_INFO("Device %s is already probed", dev_name);
return -EEXIST;
}
CDX_BUS_DEBUG(" probe device %s using driver: %s", dev_name,
- dr->driver.name);
+ cdx_drv->driver.name);
- if (dr->drv_flags & RTE_CDX_DRV_NEED_MAPPING) {
- ret = cdx_vfio_map_resource(dev);
+ if (cdx_drv->drv_flags & RTE_CDX_DRV_NEED_MAPPING) {
+ ret = cdx_vfio_map_resource(cdx_dev);
if (ret != 0) {
CDX_BUS_ERR("CDX map device failed: %d", ret);
goto error_map_device;
@@ -338,50 +332,26 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
}
/* call the driver probe() function */
- ret = dr->probe(dr, dev);
+ ret = cdx_drv->probe(cdx_drv, cdx_dev);
if (ret) {
CDX_BUS_ERR("Probe CDX driver: %s device: %s failed: %d",
- dr->driver.name, dev_name, ret);
+ cdx_drv->driver.name, dev_name, ret);
goto error_probe;
} else {
- dev->device.driver = &dr->driver;
+ cdx_dev->device.driver = &cdx_drv->driver;
}
- dev->driver = dr;
+ cdx_dev->driver = cdx_drv;
return ret;
error_probe:
- cdx_vfio_unmap_resource(dev);
- rte_intr_instance_free(dev->intr_handle);
- dev->intr_handle = NULL;
+ cdx_vfio_unmap_resource(cdx_dev);
+ rte_intr_instance_free(cdx_dev->intr_handle);
+ cdx_dev->intr_handle = NULL;
error_map_device:
return ret;
}
-/*
- * If vendor/device ID match, call the probe() function of all
- * registered driver for the given device. Return < 0 if initialization
- * failed, return 1 if no driver is found for this device.
- */
-static int
-cdx_probe_all_drivers(struct rte_cdx_device *dev)
-{
- struct rte_cdx_driver *dr = NULL;
- int rc = 0;
-
- RTE_BUS_FOREACH_DRV(dr, &rte_cdx_bus.bus) {
- rc = cdx_probe_one_driver(dr, dev);
- if (rc < 0)
- /* negative value is an error */
- return rc;
- if (rc > 0)
- /* positive value means driver doesn't support it */
- continue;
- return 0;
- }
- return 1;
-}
-
/*
* Scan the content of the CDX bus, and call the probe() function for
* all registered drivers that have a matching entry in its id_table
@@ -392,17 +362,26 @@ cdx_probe(void)
{
struct rte_cdx_device *dev = NULL;
size_t probed = 0, failed = 0;
- int ret = 0;
RTE_BUS_FOREACH_DEV(dev, &rte_cdx_bus.bus) {
+ struct rte_driver *drv = NULL;
+ int ret;
+
probed++;
- ret = cdx_probe_all_drivers(dev);
+next_driver:
+ drv = rte_bus_find_driver(&rte_cdx_bus.bus, drv, &dev->device);
+ if (drv == NULL)
+ continue;
+
+ ret = rte_cdx_bus.bus.probe_device(drv, &dev->device);
if (ret < 0) {
CDX_BUS_ERR("Requested device %s cannot be used",
dev->name);
rte_errno = errno;
failed++;
+ } else if (ret > 0) {
+ goto next_driver;
}
}
@@ -470,12 +449,6 @@ cdx_detach_dev(struct rte_cdx_device *dev)
return 0;
}
-static int
-cdx_plug(struct rte_device *dev)
-{
- return cdx_probe_all_drivers(RTE_BUS_DEVICE(dev, struct rte_cdx_device));
-}
-
static int
cdx_unplug(struct rte_device *dev)
{
@@ -524,7 +497,7 @@ struct rte_cdx_bus rte_cdx_bus = {
.probe = cdx_probe,
.find_device = rte_bus_generic_find_device,
.match = cdx_bus_match,
- .plug = cdx_plug,
+ .probe_device = cdx_probe_device,
.unplug = cdx_unplug,
.parse = cdx_parse,
.dma_map = cdx_dma_map,
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index ca80fff6ec..14cd64cc32 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -706,7 +706,6 @@ rte_dpaa_bus_probe(void)
{
int ret = -1;
struct rte_dpaa_device *dev;
- struct rte_dpaa_driver *drv;
FILE *svr_file = NULL;
uint32_t svr_ver;
static int process_once;
@@ -789,25 +788,18 @@ rte_dpaa_bus_probe(void)
/* For each registered driver, and device, call the driver->probe */
RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
- RTE_BUS_FOREACH_DRV(drv, &rte_dpaa_bus.bus) {
- if (!dpaa_bus_match(&drv->driver, &dev->device))
- continue;
-
- if (rte_dev_is_probed(&dev->device))
- continue;
-
- if (rte_bus_device_is_ignored(&rte_dpaa_bus.bus, dev->name))
- continue;
-
- ret = drv->probe(drv, dev);
- if (ret) {
- DPAA_BUS_ERR("unable to probe: %s", dev->name);
- } else {
- dev->driver = drv;
- dev->device.driver = &drv->driver;
- }
- break;
- }
+ struct rte_driver *driver = NULL;
+
+next_driver:
+ driver = rte_bus_find_driver(&rte_dpaa_bus.bus, driver, &dev->device);
+ if (driver == NULL)
+ continue;
+
+ ret = rte_dpaa_bus.bus.probe_device(driver, &dev->device);
+ if (ret < 0)
+ DPAA_BUS_ERR("Failed to probe device %s", dev->name);
+ else if (ret > 0)
+ goto next_driver;
}
dpaa_bus_global_init = 1;
@@ -828,17 +820,27 @@ rte_dpaa_get_iommu_class(void)
}
static int
-dpaa_bus_plug(struct rte_device *dev __rte_unused)
+dpaa_bus_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
- /* No operation is performed while plugging the device */
- return 0;
-}
+ struct rte_dpaa_device *dpaa_dev = RTE_BUS_DEVICE(dev, *dpaa_dev);
+ struct rte_dpaa_driver *dpaa_drv = RTE_BUS_DRIVER(drv, *dpaa_drv);
+ int ret;
-static int
-dpaa_bus_unplug(struct rte_device *dev __rte_unused)
-{
- /* No operation is performed while unplugging the device */
- return 0;
+ if (rte_dev_is_probed(&dpaa_dev->device))
+ return 0;
+
+ if (rte_bus_device_is_ignored(&rte_dpaa_bus.bus, dpaa_dev->name))
+ return 0;
+
+ ret = dpaa_drv->probe(dpaa_drv, dpaa_dev);
+ if (ret != 0) {
+ DPAA_BUS_ERR("unable to probe: %s", dpaa_dev->name);
+ } else {
+ dpaa_dev->driver = dpaa_drv;
+ dpaa_dev->device.driver = &dpaa_drv->driver;
+ }
+
+ return ret;
}
static int
@@ -899,8 +901,7 @@ static struct rte_dpaa_bus rte_dpaa_bus = {
.find_device = rte_bus_generic_find_device,
.get_iommu_class = rte_dpaa_get_iommu_class,
.match = dpaa_bus_match,
- .plug = dpaa_bus_plug,
- .unplug = dpaa_bus_unplug,
+ .probe_device = dpaa_bus_probe_device,
.dev_iterate = rte_bus_generic_dev_iterate,
.cleanup = dpaa_bus_cleanup,
},
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 8cd9b1eb88..a975e464c1 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -411,7 +411,6 @@ rte_fslmc_probe(void)
int ret = 0;
struct rte_dpaa2_device *dev;
- struct rte_dpaa2_driver *drv;
static const struct rte_mbuf_dynfield dpaa2_seqn_dynfield_desc = {
.name = DPAA2_SEQN_DYNFIELD_NAME,
@@ -456,25 +455,21 @@ rte_fslmc_probe(void)
}
RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
- RTE_BUS_FOREACH_DRV(drv, &rte_fslmc_bus.bus) {
- if (!fslmc_bus_match(&drv->driver, &dev->device))
- continue;
+ struct rte_driver *driver = NULL;
- if (rte_dev_is_probed(&dev->device))
- continue;
+ if (rte_bus_device_is_ignored(&rte_fslmc_bus.bus, dev->device.name))
+ continue;
- if (rte_bus_device_is_ignored(&rte_fslmc_bus.bus, dev->device.name))
- continue;
+next_driver:
+ driver = rte_bus_find_driver(&rte_fslmc_bus.bus, driver, &dev->device);
+ if (driver == NULL)
+ continue;
- ret = drv->probe(drv, dev);
- if (ret) {
- DPAA2_BUS_ERR("Unable to probe");
- } else {
- dev->driver = drv;
- dev->device.driver = &drv->driver;
- }
- break;
- }
+ ret = rte_fslmc_bus.bus.probe_device(driver, &dev->device);
+ if (ret < 0)
+ DPAA2_BUS_ERR("Failed to probe device %s", dev->device.name);
+ else if (ret > 0)
+ goto next_driver;
}
return 0;
@@ -540,35 +535,29 @@ rte_dpaa2_get_iommu_class(void)
}
static int
-fslmc_bus_plug(struct rte_device *rte_dev)
+fslmc_bus_probe_device(struct rte_driver *driver, struct rte_device *rte_dev)
{
- int ret = 0;
struct rte_dpaa2_device *dev = RTE_BUS_DEVICE(rte_dev, *dev);
- struct rte_dpaa2_driver *drv;
-
- RTE_BUS_FOREACH_DRV(drv, &rte_fslmc_bus.bus) {
- if (!fslmc_bus_match(&drv->driver, &dev->device))
- continue;
+ struct rte_dpaa2_driver *drv = RTE_BUS_DRIVER(driver, *drv);
+ int ret = 0;
- if (rte_dev_is_probed(&dev->device))
- continue;
+ if (rte_dev_is_probed(&dev->device))
+ return 0;
- if (dev->device.devargs &&
- dev->device.devargs->policy == RTE_DEV_BLOCKED) {
- DPAA2_BUS_DEBUG("%s Blocked, skipping",
- dev->device.name);
- continue;
- }
+ if (dev->device.devargs &&
+ dev->device.devargs->policy == RTE_DEV_BLOCKED) {
+ DPAA2_BUS_DEBUG("%s Blocked, skipping",
+ dev->device.name);
+ return 0;
+ }
- ret = drv->probe(drv, dev);
- if (ret) {
- DPAA2_BUS_ERR("Unable to probe");
- } else {
- dev->driver = drv;
- dev->device.driver = &drv->driver;
- DPAA2_BUS_INFO("%s Plugged", dev->device.name);
- }
- break;
+ ret = drv->probe(drv, dev);
+ if (ret != 0) {
+ DPAA2_BUS_ERR("Unable to probe");
+ } else {
+ dev->driver = drv;
+ dev->device.driver = &drv->driver;
+ DPAA2_BUS_INFO("%s Plugged", dev->device.name);
}
return ret;
@@ -601,7 +590,7 @@ struct rte_fslmc_bus rte_fslmc_bus = {
.find_device = rte_bus_generic_find_device,
.get_iommu_class = rte_dpaa2_get_iommu_class,
.match = fslmc_bus_match,
- .plug = fslmc_bus_plug,
+ .probe_device = fslmc_bus_probe_device,
.unplug = fslmc_bus_unplug,
.dev_iterate = rte_bus_generic_dev_iterate,
},
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index 021171e955..92ad3513e0 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -268,57 +268,32 @@ ifpga_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
}
static int
-ifpga_probe_one_driver(struct rte_afu_driver *drv,
- struct rte_afu_device *afu_dev)
+ifpga_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
+ struct rte_afu_device *afu_dev = RTE_BUS_DEVICE(dev, *afu_dev);
+ struct rte_afu_driver *afu_drv = RTE_BUS_DRIVER(drv, *afu_drv);
int ret;
- if (!ifpga_bus_match(&drv->driver, &afu_dev->device))
- /* Match of device and driver failed */
- return 1;
+ /* Check if a driver is already loaded */
+ if (rte_dev_is_probed(&afu_dev->device)) {
+ IFPGA_BUS_DEBUG("Device %s is already probed",
+ rte_ifpga_device_name(afu_dev));
+ return -EEXIST;
+ }
/* reference driver structure */
- afu_dev->driver = drv;
+ afu_dev->driver = afu_drv;
/* call the driver probe() function */
- ret = drv->probe(afu_dev);
+ ret = afu_drv->probe(afu_dev);
if (ret)
afu_dev->driver = NULL;
else
- afu_dev->device.driver = &drv->driver;
+ afu_dev->device.driver = &afu_drv->driver;
return ret;
}
-static int
-ifpga_probe_all_drivers(struct rte_afu_device *afu_dev)
-{
- struct rte_afu_driver *drv = NULL;
- int ret = 0;
-
- /* Check if a driver is already loaded */
- if (rte_dev_is_probed(&afu_dev->device)) {
- IFPGA_BUS_DEBUG("Device %s is already probed",
- rte_ifpga_device_name(afu_dev));
- return -EEXIST;
- }
-
- RTE_BUS_FOREACH_DRV(drv, &rte_ifpga_bus) {
- ret = ifpga_probe_one_driver(drv, afu_dev);
- if (ret < 0)
- /* negative value is an error */
- return ret;
- if (ret > 0)
- /* positive value means driver doesn't support it */
- continue;
- return 0;
- }
- if ((ret > 0) && (afu_dev->driver == NULL))
- return 0;
- else
- return ret;
-}
-
/*
* Scan the content of the Intel FPGA bus, and call the probe() function for
* all registered drivers that have a matching entry in its id_table
@@ -331,12 +306,21 @@ ifpga_probe(void)
int ret = 0;
RTE_BUS_FOREACH_DEV(afu_dev, &rte_ifpga_bus) {
- ret = ifpga_probe_all_drivers(afu_dev);
+ struct rte_driver *drv = NULL;
+
+next_driver:
+ drv = rte_bus_find_driver(&rte_ifpga_bus, drv, &afu_dev->device);
+ if (drv == NULL)
+ continue;
+
+ ret = rte_ifpga_bus.probe_device(drv, &afu_dev->device);
if (ret == -EEXIST)
continue;
if (ret < 0)
IFPGA_BUS_ERR("failed to initialize %s device",
rte_ifpga_device_name(afu_dev));
+ else if (ret > 0)
+ goto next_driver;
}
return ret;
@@ -379,12 +363,6 @@ ifpga_cleanup(void)
return error;
}
-static int
-ifpga_plug(struct rte_device *dev)
-{
- return ifpga_probe_all_drivers(RTE_BUS_DEVICE(dev, struct rte_afu_device));
-}
-
static int
ifpga_unplug(struct rte_device *dev)
{
@@ -454,7 +432,7 @@ static struct rte_bus rte_ifpga_bus = {
.cleanup = ifpga_cleanup,
.find_device = rte_bus_generic_find_device,
.match = ifpga_bus_match,
- .plug = ifpga_plug,
+ .probe_device = ifpga_probe_device,
.unplug = ifpga_unplug,
.parse = ifpga_parse,
};
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index d7fda1752a..b57320064e 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -181,51 +181,42 @@ pci_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
* driver.
*/
static int
-rte_pci_probe_one_driver(struct rte_pci_driver *dr,
- struct rte_pci_device *dev)
+pci_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
- int ret;
+ struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev, *pci_dev);
+ struct rte_pci_driver *pci_drv = RTE_BUS_DRIVER(drv, *pci_drv);
+ struct rte_pci_addr *loc = &pci_dev->addr;
bool already_probed;
- struct rte_pci_addr *loc;
-
- if ((dr == NULL) || (dev == NULL))
- return -EINVAL;
-
- loc = &dev->addr;
-
- /* The device is not blocked; Check if driver supports it */
- if (!pci_bus_match(&dr->driver, &dev->device))
- /* Match of device and driver failed */
- return 1;
+ int ret;
PCI_LOG(DEBUG, "PCI device "PCI_PRI_FMT" on NUMA socket %i",
loc->domain, loc->bus, loc->devid, loc->function,
- dev->device.numa_node);
+ pci_dev->device.numa_node);
/* no initialization when marked as blocked, return without error */
- if (dev->device.devargs != NULL &&
- dev->device.devargs->policy == RTE_DEV_BLOCKED) {
+ if (pci_dev->device.devargs != NULL &&
+ pci_dev->device.devargs->policy == RTE_DEV_BLOCKED) {
PCI_LOG(INFO, " Device is blocked, not initializing");
return 1;
}
- if (dev->device.numa_node < 0 && rte_socket_count() > 1)
- PCI_LOG(INFO, "Device %s is not NUMA-aware", dev->name);
+ if (pci_dev->device.numa_node < 0 && rte_socket_count() > 1)
+ PCI_LOG(INFO, "Device %s is not NUMA-aware", pci_dev->name);
- already_probed = rte_dev_is_probed(&dev->device);
- if (already_probed && !(dr->drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
- PCI_LOG(DEBUG, "Device %s is already probed", dev->device.name);
+ already_probed = rte_dev_is_probed(&pci_dev->device);
+ if (already_probed && !(pci_drv->drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+ PCI_LOG(DEBUG, "Device %s is already probed", pci_dev->device.name);
return -EEXIST;
}
- PCI_LOG(DEBUG, " probe driver: %x:%x %s", dev->id.vendor_id,
- dev->id.device_id, dr->driver.name);
+ PCI_LOG(DEBUG, " probe driver: %x:%x %s", pci_dev->id.vendor_id,
+ pci_dev->id.device_id, pci_drv->driver.name);
if (!already_probed) {
enum rte_iova_mode dev_iova_mode;
enum rte_iova_mode iova_mode;
- dev_iova_mode = pci_device_iova_mode(dr, dev);
+ dev_iova_mode = pci_device_iova_mode(pci_drv, pci_dev);
iova_mode = rte_eal_iova_mode();
if (dev_iova_mode != RTE_IOVA_DC &&
dev_iova_mode != iova_mode) {
@@ -236,21 +227,21 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
}
/* Allocate interrupt instance for pci device */
- dev->intr_handle =
+ pci_dev->intr_handle =
rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
- if (dev->intr_handle == NULL) {
+ if (pci_dev->intr_handle == NULL) {
PCI_LOG(ERR, "Failed to create interrupt instance for %s",
- dev->device.name);
+ pci_dev->device.name);
return -ENOMEM;
}
- dev->vfio_req_intr_handle =
+ pci_dev->vfio_req_intr_handle =
rte_intr_instance_alloc(RTE_INTR_INSTANCE_F_PRIVATE);
- if (dev->vfio_req_intr_handle == NULL) {
- rte_intr_instance_free(dev->intr_handle);
- dev->intr_handle = NULL;
+ if (pci_dev->vfio_req_intr_handle == NULL) {
+ rte_intr_instance_free(pci_dev->intr_handle);
+ pci_dev->intr_handle = NULL;
PCI_LOG(ERR, "Failed to create vfio req interrupt instance for %s",
- dev->device.name);
+ pci_dev->device.name);
return -ENOMEM;
}
@@ -259,43 +250,43 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
* This needs to be before rte_pci_map_device(), as it enables
* to use driver flags for adjusting configuration.
*/
- dev->driver = dr;
- if (dev->driver->drv_flags & RTE_PCI_DRV_NEED_MAPPING) {
- ret = rte_pci_map_device(dev);
+ pci_dev->driver = pci_drv;
+ if (pci_dev->driver->drv_flags & RTE_PCI_DRV_NEED_MAPPING) {
+ ret = rte_pci_map_device(pci_dev);
if (ret != 0) {
- dev->driver = NULL;
- rte_intr_instance_free(dev->vfio_req_intr_handle);
- dev->vfio_req_intr_handle = NULL;
- rte_intr_instance_free(dev->intr_handle);
- dev->intr_handle = NULL;
+ pci_dev->driver = NULL;
+ rte_intr_instance_free(pci_dev->vfio_req_intr_handle);
+ pci_dev->vfio_req_intr_handle = NULL;
+ rte_intr_instance_free(pci_dev->intr_handle);
+ pci_dev->intr_handle = NULL;
return ret;
}
}
}
PCI_LOG(INFO, "Probe PCI driver: %s (%x:%04x) device: "PCI_PRI_FMT" (socket %i)",
- dr->driver.name, dev->id.vendor_id, dev->id.device_id,
+ pci_drv->driver.name, pci_dev->id.vendor_id, pci_dev->id.device_id,
loc->domain, loc->bus, loc->devid, loc->function,
- dev->device.numa_node);
+ pci_dev->device.numa_node);
/* call the driver probe() function */
- ret = dr->probe(dr, dev);
+ ret = pci_drv->probe(pci_drv, pci_dev);
if (already_probed)
return ret; /* no rollback if already succeeded earlier */
if (ret) {
- dev->driver = NULL;
- if ((dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
+ pci_dev->driver = NULL;
+ if ((pci_drv->drv_flags & RTE_PCI_DRV_NEED_MAPPING) &&
/* Don't unmap if device is unsupported and
* driver needs mapped resources.
*/
!(ret > 0 &&
- (dr->drv_flags & RTE_PCI_DRV_KEEP_MAPPED_RES)))
- rte_pci_unmap_device(dev);
- rte_intr_instance_free(dev->vfio_req_intr_handle);
- dev->vfio_req_intr_handle = NULL;
- rte_intr_instance_free(dev->intr_handle);
- dev->intr_handle = NULL;
+ (pci_drv->drv_flags & RTE_PCI_DRV_KEEP_MAPPED_RES)))
+ rte_pci_unmap_device(pci_dev);
+ rte_intr_instance_free(pci_dev->vfio_req_intr_handle);
+ pci_dev->vfio_req_intr_handle = NULL;
+ rte_intr_instance_free(pci_dev->intr_handle);
+ pci_dev->intr_handle = NULL;
} else {
- dev->device.driver = &dr->driver;
+ pci_dev->device.driver = &pci_drv->driver;
}
return ret;
@@ -343,33 +334,6 @@ rte_pci_detach_dev(struct rte_pci_device *dev)
return 0;
}
-/*
- * If vendor/device ID match, call the probe() function of all
- * registered driver for the given device. Return < 0 if initialization
- * failed, return 1 if no driver is found for this device.
- */
-static int
-pci_probe_all_drivers(struct rte_pci_device *dev)
-{
- struct rte_pci_driver *dr = NULL;
- int rc = 0;
-
- if (dev == NULL)
- return -EINVAL;
-
- RTE_BUS_FOREACH_DRV(dr, &rte_pci_bus.bus) {
- rc = rte_pci_probe_one_driver(dr, dev);
- if (rc < 0)
- /* negative value is an error */
- return rc;
- if (rc > 0)
- /* positive value means driver doesn't support it */
- continue;
- return 0;
- }
- return 1;
-}
-
/*
* Scan the content of the PCI bus, and call the probe() function for
* all registered drivers that have a matching entry in its id_table
@@ -380,12 +344,19 @@ pci_probe(void)
{
struct rte_pci_device *dev = NULL;
size_t probed = 0, failed = 0;
- int ret = 0;
RTE_BUS_FOREACH_DEV(dev, &rte_pci_bus.bus) {
+ struct rte_driver *drv = NULL;
+ int ret;
+
probed++;
- ret = pci_probe_all_drivers(dev);
+next_driver:
+ drv = rte_bus_find_driver(&rte_pci_bus.bus, drv, &dev->device);
+ if (drv == NULL)
+ continue;
+
+ ret = rte_pci_bus.bus.probe_device(drv, &dev->device);
if (ret < 0) {
if (ret != -EEXIST) {
PCI_LOG(ERR, "Requested device " PCI_PRI_FMT " cannot be used",
@@ -395,6 +366,8 @@ pci_probe(void)
failed++;
}
ret = 0;
+ } else if (ret > 0) {
+ goto next_driver;
}
}
@@ -595,12 +568,6 @@ pci_sigbus_handler(const void *failure_addr)
return ret;
}
-static int
-pci_plug(struct rte_device *dev)
-{
- return pci_probe_all_drivers(RTE_BUS_DEVICE(dev, struct rte_pci_device));
-}
-
static int
pci_unplug(struct rte_device *dev)
{
@@ -861,7 +828,7 @@ struct rte_pci_bus rte_pci_bus = {
.cleanup = pci_cleanup,
.find_device = rte_bus_generic_find_device,
.match = pci_bus_match,
- .plug = pci_plug,
+ .probe_device = pci_probe_device,
.unplug = pci_unplug,
.parse = pci_parse,
.dev_compare = pci_dev_compare,
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 3d6b6efe6e..22979f31b3 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -404,43 +404,36 @@ platform_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
return match;
}
-static int
-device_attach(struct rte_platform_device *pdev)
-{
- struct rte_platform_driver *pdrv;
-
- RTE_BUS_FOREACH_DRV(pdrv, &platform_bus.bus) {
- if (platform_bus_match(&pdrv->driver, &pdev->device))
- break;
- }
-
- if (pdrv == NULL)
- return -ENODEV;
-
- return driver_probe_device(pdrv, pdev);
-}
-
static int
platform_bus_probe(void)
{
struct rte_platform_device *pdev;
- int ret;
RTE_BUS_FOREACH_DEV(pdev, &platform_bus.bus) {
- ret = device_attach(pdev);
+ struct rte_driver *drv = NULL;
+ int ret;
+
+next_driver:
+ drv = rte_bus_find_driver(&platform_bus.bus, drv, &pdev->device);
+ if (drv == NULL)
+ continue;
+
+ ret = platform_bus.bus.probe_device(drv, &pdev->device);
if (ret == -EBUSY) {
PLATFORM_LOG_LINE(DEBUG, "device %s already probed", pdev->name);
continue;
}
- if (ret)
+ if (ret < 0)
PLATFORM_LOG_LINE(ERR, "failed to probe %s", pdev->name);
+ else if (ret > 0)
+ goto next_driver;
}
return 0;
}
static int
-platform_bus_plug(struct rte_device *dev)
+platform_bus_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
if (rte_bus_device_is_ignored(&platform_bus.bus, dev->name))
return -EPERM;
@@ -448,7 +441,8 @@ platform_bus_plug(struct rte_device *dev)
if (!dev_is_bound_vfio_platform(dev->name))
return -EPERM;
- return device_attach(RTE_BUS_DEVICE(dev, struct rte_platform_device));
+ return driver_probe_device(RTE_BUS_DRIVER(drv, struct rte_platform_driver),
+ RTE_BUS_DEVICE(dev, struct rte_platform_device));
}
static void
@@ -559,7 +553,7 @@ struct rte_platform_bus platform_bus = {
.probe = platform_bus_probe,
.find_device = rte_bus_generic_find_device,
.match = platform_bus_match,
- .plug = platform_bus_plug,
+ .probe_device = platform_bus_probe_device,
.unplug = platform_bus_unplug,
.parse = platform_bus_parse,
.dma_map = platform_bus_dma_map,
diff --git a/drivers/bus/uacce/uacce.c b/drivers/bus/uacce/uacce.c
index 16767a3b88..2ef1a27635 100644
--- a/drivers/bus/uacce/uacce.c
+++ b/drivers/bus/uacce/uacce.c
@@ -352,74 +352,59 @@ uacce_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
}
static int
-uacce_probe_one_driver(struct rte_uacce_driver *dr, struct rte_uacce_device *dev)
+uacce_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
- const char *dev_name = dev->name;
- bool already_probed;
+ struct rte_uacce_device *uacce_dev = RTE_BUS_DEVICE(dev, *uacce_dev);
+ struct rte_uacce_driver *uacce_drv = RTE_BUS_DRIVER(drv, *uacce_drv);
+ const char *dev_name = uacce_dev->name;
int ret;
- if (!uacce_bus_match(&dr->driver, &dev->device))
- /* Match of device and driver failed */
- return 1;
-
- already_probed = rte_dev_is_probed(&dev->device);
- if (already_probed) {
+ if (rte_dev_is_probed(&uacce_dev->device)) {
UACCE_BUS_INFO("device %s is already probed", dev_name);
return -EEXIST;
}
- UACCE_BUS_DEBUG("probe device %s using driver %s", dev_name, dr->driver.name);
+ UACCE_BUS_DEBUG("probe device %s using driver %s", dev_name, uacce_drv->driver.name);
- ret = dr->probe(dr, dev);
+ ret = uacce_drv->probe(uacce_drv, uacce_dev);
if (ret != 0) {
UACCE_BUS_ERR("probe device %s with driver %s failed %d",
- dev_name, dr->driver.name, ret);
+ dev_name, uacce_drv->driver.name, ret);
} else {
- dev->device.driver = &dr->driver;
- dev->driver = dr;
+ uacce_dev->device.driver = &uacce_drv->driver;
+ uacce_dev->driver = uacce_drv;
UACCE_BUS_DEBUG("probe device %s with driver %s success",
- dev_name, dr->driver.name);
+ dev_name, uacce_drv->driver.name);
}
return ret;
}
-static int
-uacce_probe_all_drivers(struct rte_uacce_device *dev)
-{
- struct rte_uacce_driver *dr;
- int rc;
-
- RTE_BUS_FOREACH_DRV(dr, &uacce_bus.bus) {
- rc = uacce_probe_one_driver(dr, dev);
- if (rc < 0)
- /* negative value is an error */
- return rc;
- if (rc > 0)
- /* positive value means driver doesn't support it */
- continue;
- return 0;
- }
-
- return 1;
-}
-
static int
uacce_probe(void)
{
size_t probed = 0, failed = 0;
struct rte_uacce_device *dev;
- int ret;
RTE_BUS_FOREACH_DEV(dev, &uacce_bus.bus) {
+ struct rte_driver *drv = NULL;
+ int ret;
+
probed++;
- ret = uacce_probe_all_drivers(dev);
+next_driver:
+ drv = rte_bus_find_driver(&uacce_bus.bus, drv, &dev->device);
+ if (drv == NULL)
+ continue;
+
+ ret = uacce_bus.bus.probe_device(drv, &dev->device);
if (ret < 0) {
UACCE_BUS_LOG(ERR, "Requested device %s cannot be used",
dev->name);
rte_errno = errno;
failed++;
+ } else if (ret > 0) {
+ goto next_driver;
}
}
@@ -457,12 +442,6 @@ uacce_cleanup(void)
return error;
}
-static int
-uacce_plug(struct rte_device *dev)
-{
- return uacce_probe_all_drivers(RTE_BUS_DEVICE(dev, struct rte_uacce_device));
-}
-
static int
uacce_detach_dev(struct rte_uacce_device *dev)
{
@@ -626,7 +605,7 @@ static struct rte_uacce_bus uacce_bus = {
.probe = uacce_probe,
.cleanup = uacce_cleanup,
.match = uacce_bus_match,
- .plug = uacce_plug,
+ .probe_device = uacce_probe_device,
.unplug = uacce_unplug,
.find_device = rte_bus_generic_find_device,
.parse = uacce_parse,
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 0308be5cbe..e55be7fb2c 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -182,32 +182,30 @@ vdev_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
}
static int
-vdev_probe_all_drivers(struct rte_vdev_device *dev)
+vdev_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
+ struct rte_vdev_device *vdev_dev = RTE_BUS_DEVICE(dev, *vdev_dev);
+ struct rte_vdev_driver *vdev_drv = RTE_BUS_DRIVER(drv, *vdev_drv);
const char *name;
- struct rte_vdev_driver *driver;
enum rte_iova_mode iova_mode;
int ret;
- if (rte_dev_is_probed(&dev->device))
+ if (rte_dev_is_probed(&vdev_dev->device))
return -EEXIST;
- name = rte_vdev_device_name(dev);
+ name = rte_vdev_device_name(vdev_dev);
VDEV_LOG(DEBUG, "Search driver to probe device %s", name);
- if (vdev_parse(name, &driver))
- return -1;
-
iova_mode = rte_eal_iova_mode();
- if ((driver->drv_flags & RTE_VDEV_DRV_NEED_IOVA_AS_VA) && (iova_mode == RTE_IOVA_PA)) {
+ if ((vdev_drv->drv_flags & RTE_VDEV_DRV_NEED_IOVA_AS_VA) && (iova_mode == RTE_IOVA_PA)) {
VDEV_LOG(ERR, "%s requires VA IOVA mode but current mode is PA, not initializing",
name);
return -1;
}
- ret = driver->probe(dev);
+ ret = vdev_drv->probe(vdev_dev);
if (ret == 0)
- dev->device.driver = &driver->driver;
+ vdev_dev->device.driver = &vdev_drv->driver;
return ret;
}
@@ -323,14 +321,23 @@ rte_vdev_init(const char *name, const char *args)
rte_spinlock_recursive_lock(&vdev_device_list_lock);
ret = insert_vdev(name, args, &dev, true);
if (ret == 0) {
- ret = vdev_probe_all_drivers(dev);
- if (ret) {
- if (ret > 0)
- VDEV_LOG(ERR, "no driver found for %s", name);
+ struct rte_driver *drv = NULL;
+
+next_driver:
+ drv = rte_bus_find_driver(&rte_vdev_bus, drv, &dev->device);
+ if (drv == NULL) {
+ VDEV_LOG(ERR, "no driver found for %s", name);
+ ret = -1;
+ } else {
+ ret = rte_vdev_bus.probe_device(drv, &dev->device);
+ }
+ if (ret < 0) {
/* If fails, remove it from vdev list */
rte_bus_remove_device(&rte_vdev_bus, &dev->device);
rte_devargs_remove(dev->device.devargs);
free(dev);
+ } else if (ret > 0) {
+ goto next_driver;
}
}
rte_spinlock_recursive_unlock(&vdev_device_list_lock);
@@ -393,8 +400,6 @@ struct vdev_param {
char name[RTE_DEV_NAME_MAX_LEN];
};
-static int vdev_plug(struct rte_device *dev);
-
/**
* This function works as the action for both primary and secondary process
* for static vdev discovery when a secondary process is booting.
@@ -552,18 +557,27 @@ vdev_probe(void)
/* call the init function for each virtual device */
RTE_BUS_FOREACH_DEV(dev, &rte_vdev_bus) {
+ struct rte_driver *drv = NULL;
+
/* we don't use the vdev lock here, as it's only used in DPDK
* initialization; and we don't want to hold such a lock when
* we call each driver probe.
*/
- r = vdev_probe_all_drivers(dev);
- if (r != 0) {
+next_driver:
+ drv = rte_bus_find_driver(&rte_vdev_bus, drv, &dev->device);
+ if (drv == NULL)
+ continue;
+
+ r = rte_vdev_bus.probe_device(drv, &dev->device);
+ if (r < 0) {
if (r == -EEXIST)
continue;
VDEV_LOG(ERR, "failed to initialize %s device",
rte_vdev_device_name(dev));
ret = -1;
+ } else if (r > 0) {
+ goto next_driver;
}
}
@@ -614,12 +628,6 @@ vdev_find_device(const struct rte_bus *bus, const struct rte_device *start,
return dev;
}
-static int
-vdev_plug(struct rte_device *dev)
-{
- return vdev_probe_all_drivers(RTE_BUS_DEVICE(dev, struct rte_vdev_device));
-}
-
static int
vdev_unplug(struct rte_device *dev)
{
@@ -651,7 +659,7 @@ static struct rte_bus rte_vdev_bus = {
.cleanup = vdev_cleanup,
.find_device = vdev_find_device,
.match = vdev_bus_match,
- .plug = vdev_plug,
+ .probe_device = vdev_probe_device,
.unplug = vdev_unplug,
.parse = vdev_parse,
.dma_map = vdev_dma_map,
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index d811f1a229..ba923a2669 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -79,85 +79,59 @@ vmbus_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
return false;
}
+
/*
* If device ID match, call the devinit() function of the driver.
*/
static int
-vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
- struct rte_vmbus_device *dev)
+vmbus_probe_device(struct rte_driver *drv, struct rte_device *dev)
{
+ struct rte_vmbus_device *vmbus_dev = RTE_BUS_DEVICE(dev, *vmbus_dev);
+ struct rte_vmbus_driver *vmbus_drv = RTE_BUS_DRIVER(drv, *vmbus_drv);
char guid[RTE_UUID_STRLEN];
int ret;
- if (!vmbus_bus_match(&dr->driver, &dev->device))
- return 1; /* not supported */
+ /* Check if a driver is already loaded */
+ if (rte_dev_is_probed(&vmbus_dev->device)) {
+ VMBUS_LOG(DEBUG, "VMBUS driver already loaded");
+ return 0;
+ }
- rte_uuid_unparse(dev->device_id, guid, sizeof(guid));
+ rte_uuid_unparse(vmbus_dev->device_id, guid, sizeof(guid));
VMBUS_LOG(INFO, "VMBUS device %s on NUMA socket %i",
- guid, dev->device.numa_node);
+ guid, vmbus_dev->device.numa_node);
/* no initialization when marked as blocked, return without error */
- if (dev->device.devargs != NULL &&
- dev->device.devargs->policy == RTE_DEV_BLOCKED) {
+ if (vmbus_dev->device.devargs != NULL &&
+ vmbus_dev->device.devargs->policy == RTE_DEV_BLOCKED) {
VMBUS_LOG(INFO, " Device is blocked, not initializing");
return 1;
}
/* map resources for device */
- ret = rte_vmbus_map_device(dev);
+ ret = rte_vmbus_map_device(vmbus_dev);
if (ret != 0)
return ret;
/* reference driver structure */
- dev->driver = dr;
+ vmbus_dev->driver = vmbus_drv;
- if (dev->device.numa_node < 0 && rte_socket_count() > 1)
+ if (vmbus_dev->device.numa_node < 0 && rte_socket_count() > 1)
VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
/* call the driver probe() function */
- VMBUS_LOG(INFO, " probe driver: %s", dr->driver.name);
- ret = dr->probe(dr, dev);
- if (ret) {
- dev->driver = NULL;
- rte_vmbus_unmap_device(dev);
+ VMBUS_LOG(INFO, " probe driver: %s", vmbus_drv->driver.name);
+ ret = vmbus_drv->probe(vmbus_drv, vmbus_dev);
+ if (ret != 0) {
+ vmbus_dev->driver = NULL;
+ rte_vmbus_unmap_device(vmbus_dev);
} else {
- dev->device.driver = &dr->driver;
+ vmbus_dev->device.driver = &vmbus_drv->driver;
}
return ret;
}
-/*
- * If device class GUID matches, call the probe function of
- * register drivers for the vmbus device.
- * Return -1 if initialization failed,
- * and 1 if no driver found for this device.
- */
-static int
-vmbus_probe_all_drivers(struct rte_vmbus_device *dev)
-{
- struct rte_vmbus_driver *dr;
- int rc;
-
- /* Check if a driver is already loaded */
- if (rte_dev_is_probed(&dev->device)) {
- VMBUS_LOG(DEBUG, "VMBUS driver already loaded");
- return 0;
- }
-
- RTE_BUS_FOREACH_DRV(dr, &rte_vmbus_bus.bus) {
- rc = vmbus_probe_one_driver(dr, dev);
- if (rc < 0) /* negative is an error */
- return -1;
-
- if (rc > 0) /* positive driver doesn't support it */
- continue;
-
- return 0;
- }
- return 1;
-}
-
/*
* Scan the vmbus, and call the devinit() function for
* all registered drivers that have a matching entry in its id_table
@@ -172,6 +146,9 @@ rte_vmbus_probe(void)
char ubuf[RTE_UUID_STRLEN];
RTE_BUS_FOREACH_DEV(dev, &rte_vmbus_bus.bus) {
+ struct rte_driver *drv = NULL;
+ int ret;
+
probed++;
rte_uuid_unparse(dev->device_id, ubuf, sizeof(ubuf));
@@ -179,11 +156,19 @@ rte_vmbus_probe(void)
if (rte_bus_device_is_ignored(&rte_vmbus_bus.bus, ubuf))
continue;
- if (vmbus_probe_all_drivers(dev) < 0) {
+next_driver:
+ drv = rte_bus_find_driver(&rte_vmbus_bus.bus, drv, &dev->device);
+ if (drv == NULL)
+ continue;
+
+ ret = rte_vmbus_bus.bus.probe_device(drv, &dev->device);
+ if (ret < 0) {
VMBUS_LOG(NOTICE,
"Requested device %s cannot be used", ubuf);
rte_errno = errno;
failed++;
+ } else if (ret > 0) {
+ goto next_driver;
}
}
@@ -272,6 +257,7 @@ struct rte_vmbus_bus rte_vmbus_bus = {
.cleanup = rte_vmbus_cleanup,
.find_device = rte_bus_generic_find_device,
.match = vmbus_bus_match,
+ .probe_device = vmbus_probe_device,
.parse = vmbus_parse,
.dev_compare = vmbus_dev_compare,
},
diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c
index cba26d0cdc..493ddc0d1e 100644
--- a/drivers/dma/idxd/idxd_bus.c
+++ b/drivers/dma/idxd/idxd_bus.c
@@ -44,6 +44,7 @@ struct rte_dsa_device {
struct dsa_bus;
static int dsa_scan(void);
static bool dsa_match(const struct rte_driver *drv, const struct rte_device *dev);
+static int dsa_probe_device(struct rte_driver *drv, struct rte_device *dev);
static int dsa_probe(void);
static enum rte_iova_mode dsa_get_iommu_class(void);
static int dsa_addr_parse(const char *name, void *addr);
@@ -60,6 +61,7 @@ struct dsa_bus dsa_bus = {
.bus = {
.scan = dsa_scan,
.match = dsa_match,
+ .probe_device = dsa_probe_device,
.probe = dsa_probe,
.find_device = rte_bus_generic_find_device,
.get_iommu_class = dsa_get_iommu_class,
@@ -210,32 +212,33 @@ read_device_int(struct rte_dsa_device *dev, const char *filename,
}
static int
-idxd_probe_dsa(struct rte_dsa_device *dev)
+dsa_probe_device(__rte_unused struct rte_driver *drv, struct rte_device *dev)
{
+ struct rte_dsa_device *dsa_dev = RTE_BUS_DEVICE(dev, *dsa_dev);
struct idxd_dmadev idxd = {0};
int ret = 0;
IDXD_PMD_INFO("Probing device %s on numa node %d",
- dev->wq_name, dev->device.numa_node);
- if (read_wq_int(dev, "size", &ret) < 0)
+ dsa_dev->wq_name, dsa_dev->device.numa_node);
+ if (read_wq_int(dsa_dev, "size", &ret) < 0)
return -1;
idxd.max_batches = ret;
- if (read_wq_int(dev, "max_batch_size", &ret) < 0)
+ if (read_wq_int(dsa_dev, "max_batch_size", &ret) < 0)
return -1;
idxd.max_batch_size = ret;
- idxd.qid = dev->addr.wq_id;
- idxd.u.bus.dsa_id = dev->addr.device_id;
+ idxd.qid = dsa_dev->addr.wq_id;
+ idxd.u.bus.dsa_id = dsa_dev->addr.device_id;
idxd.sva_support = 1;
- idxd.portal = idxd_bus_mmap_wq(dev);
+ idxd.portal = idxd_bus_mmap_wq(dsa_dev);
if (idxd.portal == NULL) {
IDXD_PMD_ERR("WQ mmap failed");
return -ENOENT;
}
- ret = idxd_dmadev_create(dev->wq_name, &dev->device, &idxd, &idxd_bus_ops);
+ ret = idxd_dmadev_create(dsa_dev->wq_name, dev, &idxd, &idxd_bus_ops);
if (ret) {
- IDXD_PMD_ERR("Failed to create dmadev %s", dev->wq_name);
+ IDXD_PMD_ERR("Failed to create dmadev %s", dsa_dev->wq_name);
return ret;
}
@@ -270,7 +273,7 @@ dsa_probe(void)
if (dsa_match(&dsa_bus.driver, &dev->device) &&
!rte_bus_device_is_ignored(&dsa_bus.bus, dev->device.name)) {
dev->device.driver = &dsa_bus.driver;
- idxd_probe_dsa(dev);
+ dsa_probe_device(&dsa_bus.driver, &dev->device);
continue;
}
IDXD_PMD_DEBUG("WQ '%s', not allocated to DPDK", dev->wq_name);
diff --git a/lib/eal/common/eal_common_bus.c b/lib/eal/common/eal_common_bus.c
index 4884cdfa50..88c417e8d3 100644
--- a/lib/eal/common/eal_common_bus.c
+++ b/lib/eal/common/eal_common_bus.c
@@ -36,8 +36,9 @@ rte_bus_register(struct rte_bus *bus)
RTE_VERIFY(bus->scan);
RTE_VERIFY(bus->probe);
RTE_VERIFY(bus->find_device);
- /* Buses supporting driver plug also require unplug. */
- RTE_VERIFY(!bus->plug || bus->unplug);
+
+ /* A bus providing probe_device requires match. */
+ RTE_VERIFY(!bus->probe_device || bus->match);
TAILQ_INIT(&bus->device_list);
TAILQ_INIT(&bus->driver_list);
diff --git a/lib/eal/common/eal_common_dev.c b/lib/eal/common/eal_common_dev.c
index 17e8901546..a38c211e5d 100644
--- a/lib/eal/common/eal_common_dev.c
+++ b/lib/eal/common/eal_common_dev.c
@@ -179,6 +179,7 @@ int
local_dev_probe(const char *devargs, struct rte_device **new_dev)
{
struct rte_device *dev;
+ struct rte_driver *drv;
struct rte_devargs *da;
int ret;
@@ -191,7 +192,7 @@ local_dev_probe(const char *devargs, struct rte_device **new_dev)
if (ret)
goto err_devarg;
- if (da->bus->plug == NULL) {
+ if (da->bus->probe_device == NULL) {
EAL_LOG(ERR, "Function plug not supported by bus (%s)",
da->bus->name);
ret = -ENOTSUP;
@@ -219,9 +220,16 @@ local_dev_probe(const char *devargs, struct rte_device **new_dev)
* those devargs shouldn't be removed manually anymore.
*/
- ret = dev->bus->plug(dev);
- if (ret > 0)
+ drv = NULL;
+next_driver:
+ drv = rte_bus_find_driver(dev->bus, drv, dev);
+ if (drv == NULL) {
ret = -ENOTSUP;
+ } else {
+ ret = dev->bus->probe_device(drv, dev);
+ if (ret > 0)
+ goto next_driver;
+ }
if (ret && !rte_dev_is_probed(dev)) { /* if hasn't ever succeeded */
EAL_LOG(ERR, "Driver cannot attach the device (%s)",
diff --git a/lib/eal/include/bus_driver.h b/lib/eal/include/bus_driver.h
index c4d9ac2719..0db4473bcc 100644
--- a/lib/eal/include/bus_driver.h
+++ b/lib/eal/include/bus_driver.h
@@ -85,6 +85,21 @@ typedef struct rte_device *
*/
typedef int (*rte_bus_plug_t)(struct rte_device *dev);
+/**
+ * Implementation specific probe function which is responsible for linking
+ * devices on that bus with applicable drivers.
+ *
+ * @param drv
+ * Driver that matches the device.
+ * @param dev
+ * Device pointer that was returned by a previous call to find_device.
+ *
+ * @return
+ * 0 on success.
+ * !0 on error.
+ */
+typedef int (*rte_bus_probe_device_t)(struct rte_driver *drv, struct rte_device *dev);
+
/**
* Implementation specific remove function which is responsible for unlinking
* devices on that bus from assigned driver.
@@ -316,7 +331,7 @@ struct rte_bus {
rte_bus_probe_t probe; /**< Probe devices on bus */
rte_bus_find_device_t find_device; /**< Find a device on the bus */
rte_bus_match_t match; /**< Check if driver matches device */
- rte_bus_plug_t plug; /**< Probe single device for drivers */
+ rte_bus_probe_device_t probe_device; /**< Probe single device with driver */
rte_bus_unplug_t unplug; /**< Remove single device from driver */
rte_bus_parse_t parse; /**< Parse a device name */
rte_bus_dev_compare_t dev_compare; /**< Compare two device names */
--
2.53.0
^ permalink raw reply related
* [PATCH v3 13/25] bus: factorize driver lookup
From: David Marchand @ 2026-05-26 8:41 UTC (permalink / raw)
To: dev
Cc: thomas, stephen, bruce.richardson, Parav Pandit, Xueming Li,
Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Rosen Xu, Chenbo Xia, Tomasz Duszynski, Chengwen Feng, Long Li,
Wei Hu, Kevin Laatz
In-Reply-To: <20260526084212.3145685-1-david.marchand@redhat.com>
Introduce a new bus operation 'match' that checks whether a driver can
handle a given device. This separates the matching logic from iteration,
with buses providing match logic and EAL providing generic iteration
through rte_bus_find_driver().
The match operation returns true if a driver matches a device.
Matching logic is bus-specific (e.g., ID table matching for PCI/CDX,
device type matching for DPAA/FSLMC, UUID matching for IFPGA/VMBUS,
name matching for vdev/platform, API/algorithm matching for uacce).
A generic helper rte_bus_find_driver() iterates through all drivers
on a bus and returns the next matching driver, eliminating the need
for each bus to duplicate iteration logic.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/bus/auxiliary/auxiliary_common.c | 14 ++++-----
drivers/bus/auxiliary/private.h | 6 ----
drivers/bus/cdx/cdx.c | 16 +++++-----
drivers/bus/dpaa/dpaa_bus.c | 21 ++++++-------
drivers/bus/fslmc/fslmc_bus.c | 23 +++++++-------
drivers/bus/ifpga/ifpga_bus.c | 14 +++++----
drivers/bus/pci/pci_common.c | 20 ++++++-------
drivers/bus/pci/private.h | 15 ----------
drivers/bus/platform/platform.c | 9 ++++--
drivers/bus/uacce/uacce.c | 13 ++++----
drivers/bus/vdev/vdev.c | 20 +++++++++++++
drivers/bus/vmbus/vmbus_common.c | 21 ++++---------
drivers/dma/idxd/idxd_bus.c | 35 ++++++++++++++--------
lib/eal/common/eal_common_bus.c | 19 ++++++++++++
lib/eal/include/bus_driver.h | 38 ++++++++++++++++++++++++
15 files changed, 172 insertions(+), 112 deletions(-)
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
index 05299db8fe..21b5bcb416 100644
--- a/drivers/bus/auxiliary/auxiliary_common.c
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -59,13 +59,12 @@ auxiliary_on_scan(struct rte_auxiliary_device *aux_dev)
aux_dev->device.devargs = rte_bus_find_devargs(&auxiliary_bus.bus, aux_dev->name);
}
-/*
- * Match the auxiliary driver and device using driver function.
- */
-bool
-auxiliary_match(const struct rte_auxiliary_driver *aux_drv,
- const struct rte_auxiliary_device *aux_dev)
+static bool
+auxiliary_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
{
+ const struct rte_auxiliary_driver *aux_drv = RTE_BUS_DRIVER(drv, *aux_drv);
+ const struct rte_auxiliary_device *aux_dev = RTE_BUS_DEVICE(dev, *aux_dev);
+
if (aux_drv->match == NULL)
return false;
return aux_drv->match(aux_dev->name);
@@ -82,7 +81,7 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
int ret;
/* Check if driver supports it. */
- if (!auxiliary_match(drv, dev))
+ if (!auxiliary_bus_match(&drv->driver, &dev->device))
/* Match of device and driver failed */
return 1;
@@ -340,6 +339,7 @@ struct rte_auxiliary_bus auxiliary_bus = {
.probe = auxiliary_probe,
.cleanup = auxiliary_cleanup,
.find_device = rte_bus_generic_find_device,
+ .match = auxiliary_bus_match,
.plug = auxiliary_plug,
.unplug = auxiliary_unplug,
.parse = auxiliary_parse,
diff --git a/drivers/bus/auxiliary/private.h b/drivers/bus/auxiliary/private.h
index 659d798cd6..116154eb56 100644
--- a/drivers/bus/auxiliary/private.h
+++ b/drivers/bus/auxiliary/private.h
@@ -44,10 +44,4 @@ int auxiliary_scan(void);
*/
void auxiliary_on_scan(struct rte_auxiliary_device *aux_dev);
-/*
- * Match the auxiliary driver and device by driver function.
- */
-bool auxiliary_match(const struct rte_auxiliary_driver *aux_drv,
- const struct rte_auxiliary_device *aux_dev);
-
#endif /* BUS_AUXILIARY_PRIVATE_H */
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index d6f83e2e80..c898ce9271 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -279,13 +279,12 @@ cdx_unmap_resource(void *requested_addr, size_t size)
requested_addr, size, rte_strerror(rte_errno));
}
}
-/*
- * Match the CDX Driver and Device using device id and vendor id.
- */
+
static bool
-cdx_match(const struct rte_cdx_driver *cdx_drv,
- const struct rte_cdx_device *cdx_dev)
+cdx_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
{
+ const struct rte_cdx_driver *cdx_drv = RTE_BUS_DRIVER(drv, *cdx_drv);
+ const struct rte_cdx_device *cdx_dev = RTE_BUS_DEVICE(dev, *cdx_dev);
const struct rte_cdx_id *id_table;
for (id_table = cdx_drv->id_table; id_table->vendor_id != 0;
@@ -298,10 +297,10 @@ cdx_match(const struct rte_cdx_driver *cdx_drv,
id_table->device_id != RTE_CDX_ANY_ID)
continue;
- return 1;
+ return true;
}
- return 0;
+ return false;
}
/*
@@ -317,7 +316,7 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr,
int ret;
/* The device is not blocked; Check if driver supports it */
- if (!cdx_match(dr, dev))
+ if (!cdx_bus_match(&dr->driver, &dev->device))
/* Match of device and driver failed */
return 1;
@@ -524,6 +523,7 @@ struct rte_cdx_bus rte_cdx_bus = {
.scan = cdx_scan,
.probe = cdx_probe,
.find_device = rte_bus_generic_find_device,
+ .match = cdx_bus_match,
.plug = cdx_plug,
.unplug = cdx_unplug,
.parse = cdx_parse,
diff --git a/drivers/bus/dpaa/dpaa_bus.c b/drivers/bus/dpaa/dpaa_bus.c
index b3a754cbf4..ca80fff6ec 100644
--- a/drivers/bus/dpaa/dpaa_bus.c
+++ b/drivers/bus/dpaa/dpaa_bus.c
@@ -626,19 +626,16 @@ rte_dpaa_driver_unregister(struct rte_dpaa_driver *driver)
rte_bus_remove_driver(&rte_dpaa_bus.bus, &driver->driver);
}
-static int
-rte_dpaa_device_match(struct rte_dpaa_driver *drv,
- struct rte_dpaa_device *dev)
+static bool
+dpaa_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
{
- if (!drv || !dev) {
- DPAA_BUS_DEBUG("Invalid drv or dev received.");
- return -1;
- }
+ const struct rte_dpaa_driver *dpaa_drv = RTE_BUS_DRIVER(drv, *dpaa_drv);
+ const struct rte_dpaa_device *dpaa_dev = RTE_BUS_DEVICE(dev, *dpaa_dev);
- if (drv->drv_type == dev->device_type)
- return 0;
+ if (dpaa_drv->drv_type == dpaa_dev->device_type)
+ return true;
- return -1;
+ return false;
}
static int
@@ -793,8 +790,7 @@ rte_dpaa_bus_probe(void)
/* For each registered driver, and device, call the driver->probe */
RTE_BUS_FOREACH_DEV(dev, &rte_dpaa_bus.bus) {
RTE_BUS_FOREACH_DRV(drv, &rte_dpaa_bus.bus) {
- ret = rte_dpaa_device_match(drv, dev);
- if (ret)
+ if (!dpaa_bus_match(&drv->driver, &dev->device))
continue;
if (rte_dev_is_probed(&dev->device))
@@ -902,6 +898,7 @@ static struct rte_dpaa_bus rte_dpaa_bus = {
.dev_compare = dpaa_bus_dev_compare,
.find_device = rte_bus_generic_find_device,
.get_iommu_class = rte_dpaa_get_iommu_class,
+ .match = dpaa_bus_match,
.plug = dpaa_bus_plug,
.unplug = dpaa_bus_unplug,
.dev_iterate = rte_bus_generic_dev_iterate,
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 716f0178b5..8cd9b1eb88 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -381,14 +381,16 @@ rte_fslmc_scan(void)
return 0;
}
-static int
-rte_fslmc_match(struct rte_dpaa2_driver *dpaa2_drv,
- struct rte_dpaa2_device *dpaa2_dev)
+static bool
+fslmc_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
{
+ const struct rte_dpaa2_driver *dpaa2_drv = RTE_BUS_DRIVER(drv, *dpaa2_drv);
+ const struct rte_dpaa2_device *dpaa2_dev = RTE_BUS_DEVICE(dev, *dpaa2_dev);
+
if (dpaa2_drv->drv_type == dpaa2_dev->dev_type)
- return 0;
+ return true;
- return 1;
+ return false;
}
static int
@@ -455,8 +457,7 @@ rte_fslmc_probe(void)
RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
RTE_BUS_FOREACH_DRV(drv, &rte_fslmc_bus.bus) {
- ret = rte_fslmc_match(drv, dev);
- if (ret)
+ if (!fslmc_bus_match(&drv->driver, &dev->device))
continue;
if (rte_dev_is_probed(&dev->device))
@@ -504,14 +505,12 @@ rte_fslmc_driver_unregister(struct rte_dpaa2_driver *driver)
static inline int
fslmc_all_device_support_iova(void)
{
- int ret = 0;
struct rte_dpaa2_device *dev;
struct rte_dpaa2_driver *drv;
RTE_BUS_FOREACH_DEV(dev, &rte_fslmc_bus.bus) {
RTE_BUS_FOREACH_DRV(drv, &rte_fslmc_bus.bus) {
- ret = rte_fslmc_match(drv, dev);
- if (ret)
+ if (!fslmc_bus_match(&drv->driver, &dev->device))
continue;
/* if the driver is not supporting IOVA */
if (!(drv->drv_flags & RTE_DPAA2_DRV_IOVA_AS_VA))
@@ -548,8 +547,7 @@ fslmc_bus_plug(struct rte_device *rte_dev)
struct rte_dpaa2_driver *drv;
RTE_BUS_FOREACH_DRV(drv, &rte_fslmc_bus.bus) {
- ret = rte_fslmc_match(drv, dev);
- if (ret)
+ if (!fslmc_bus_match(&drv->driver, &dev->device))
continue;
if (rte_dev_is_probed(&dev->device))
@@ -602,6 +600,7 @@ struct rte_fslmc_bus rte_fslmc_bus = {
.dev_compare = fslmc_dev_compare,
.find_device = rte_bus_generic_find_device,
.get_iommu_class = rte_dpaa2_get_iommu_class,
+ .match = fslmc_bus_match,
.plug = fslmc_bus_plug,
.unplug = fslmc_bus_unplug,
.dev_iterate = rte_bus_generic_dev_iterate,
diff --git a/drivers/bus/ifpga/ifpga_bus.c b/drivers/bus/ifpga/ifpga_bus.c
index 7d3331fe7e..021171e955 100644
--- a/drivers/bus/ifpga/ifpga_bus.c
+++ b/drivers/bus/ifpga/ifpga_bus.c
@@ -245,10 +245,11 @@ ifpga_scan(void)
/*
* Match the AFU Driver and AFU Device using the ID Table
*/
-static int
-rte_afu_match(const struct rte_afu_driver *afu_drv,
- const struct rte_afu_device *afu_dev)
+static bool
+ifpga_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
{
+ const struct rte_afu_driver *afu_drv = RTE_BUS_DRIVER(drv, *afu_drv);
+ const struct rte_afu_device *afu_dev = RTE_BUS_DEVICE(dev, *afu_dev);
const struct rte_afu_uuid *id_table;
for (id_table = afu_drv->id_table;
@@ -260,10 +261,10 @@ rte_afu_match(const struct rte_afu_driver *afu_drv,
afu_dev->id.uuid.uuid_high)
continue;
- return 1;
+ return true;
}
- return 0;
+ return false;
}
static int
@@ -272,7 +273,7 @@ ifpga_probe_one_driver(struct rte_afu_driver *drv,
{
int ret;
- if (!rte_afu_match(drv, afu_dev))
+ if (!ifpga_bus_match(&drv->driver, &afu_dev->device))
/* Match of device and driver failed */
return 1;
@@ -452,6 +453,7 @@ static struct rte_bus rte_ifpga_bus = {
.probe = ifpga_probe,
.cleanup = ifpga_cleanup,
.find_device = rte_bus_generic_find_device,
+ .match = ifpga_bus_match,
.plug = ifpga_plug,
.unplug = ifpga_unplug,
.parse = ifpga_parse,
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 70ce63eac7..d7fda1752a 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -141,13 +141,12 @@ pci_unmap_resource(void *requested_addr, size_t size)
} else
PCI_LOG(DEBUG, " PCI memory unmapped at %p", requested_addr);
}
-/*
- * Match the PCI Driver and Device using the ID Table
- */
-int
-rte_pci_match(const struct rte_pci_driver *pci_drv,
- const struct rte_pci_device *pci_dev)
+
+static bool
+pci_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
{
+ const struct rte_pci_driver *pci_drv = RTE_BUS_DRIVER(drv, *pci_drv);
+ const struct rte_pci_device *pci_dev = RTE_BUS_DEVICE(dev, *pci_dev);
const struct rte_pci_id *id_table;
for (id_table = pci_drv->id_table; id_table->vendor_id != 0;
@@ -171,10 +170,10 @@ rte_pci_match(const struct rte_pci_driver *pci_drv,
id_table->class_id != RTE_CLASS_ANY_ID)
continue;
- return 1;
+ return true;
}
- return 0;
+ return false;
}
/*
@@ -195,7 +194,7 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
loc = &dev->addr;
/* The device is not blocked; Check if driver supports it */
- if (!rte_pci_match(dr, dev))
+ if (!pci_bus_match(&dr->driver, &dev->device))
/* Match of device and driver failed */
return 1;
@@ -680,7 +679,7 @@ rte_pci_get_iommu_class(void)
RTE_BUS_FOREACH_DRV(drv, &rte_pci_bus.bus) {
enum rte_iova_mode dev_iova_mode;
- if (!rte_pci_match(drv, dev))
+ if (!pci_bus_match(&drv->driver, &dev->device))
continue;
dev_iova_mode = pci_device_iova_mode(drv, dev);
@@ -861,6 +860,7 @@ struct rte_pci_bus rte_pci_bus = {
.probe = pci_probe,
.cleanup = pci_cleanup,
.find_device = rte_bus_generic_find_device,
+ .match = pci_bus_match,
.plug = pci_plug,
.unplug = pci_unplug,
.parse = pci_parse,
diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
index 21637882f8..c54ea7b9d8 100644
--- a/drivers/bus/pci/private.h
+++ b/drivers/bus/pci/private.h
@@ -217,21 +217,6 @@ pci_uio_remap_resource(struct rte_pci_device *dev);
int pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx,
struct mapped_pci_resource *uio_res, int map_idx);
-/*
- * Match the PCI Driver and Device using the ID Table
- *
- * @param pci_drv
- * PCI driver from which ID table would be extracted
- * @param pci_dev
- * PCI device to match against the driver
- * @return
- * 1 for successful match
- * 0 for unsuccessful match
- */
-int
-rte_pci_match(const struct rte_pci_driver *pci_drv,
- const struct rte_pci_device *pci_dev);
-
/**
* OS specific callbacks for rte_pci_get_iommu_class
*
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 636f051049..3d6b6efe6e 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -371,8 +371,10 @@ driver_probe_device(struct rte_platform_driver *pdrv, struct rte_platform_device
}
static bool
-driver_match_device(struct rte_platform_driver *pdrv, struct rte_platform_device *pdev)
+platform_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
{
+ const struct rte_platform_driver *pdrv = RTE_BUS_DRIVER(drv, *pdrv);
+ const struct rte_platform_device *pdev = RTE_BUS_DEVICE(dev, *pdev);
bool match = false;
char *kdrv;
@@ -408,7 +410,7 @@ device_attach(struct rte_platform_device *pdev)
struct rte_platform_driver *pdrv;
RTE_BUS_FOREACH_DRV(pdrv, &platform_bus.bus) {
- if (driver_match_device(pdrv, pdev))
+ if (platform_bus_match(&pdrv->driver, &pdev->device))
break;
}
@@ -488,7 +490,7 @@ platform_bus_parse(const char *name, void *addr)
rte_strscpy(pdev.name, name, sizeof(pdev.name));
RTE_BUS_FOREACH_DRV(pdrv, &platform_bus.bus) {
- if (driver_match_device(pdrv, &pdev))
+ if (platform_bus_match(&pdrv->driver, &pdev.device))
break;
}
@@ -556,6 +558,7 @@ struct rte_platform_bus platform_bus = {
.scan = platform_bus_scan,
.probe = platform_bus_probe,
.find_device = rte_bus_generic_find_device,
+ .match = platform_bus_match,
.plug = platform_bus_plug,
.unplug = platform_bus_unplug,
.parse = platform_bus_parse,
diff --git a/drivers/bus/uacce/uacce.c b/drivers/bus/uacce/uacce.c
index af1ada0bd3..16767a3b88 100644
--- a/drivers/bus/uacce/uacce.c
+++ b/drivers/bus/uacce/uacce.c
@@ -315,15 +315,17 @@ uacce_match_api(const struct rte_uacce_device *dev, bool forward_compat,
}
static bool
-uacce_match(const struct rte_uacce_driver *dr, const struct rte_uacce_device *dev)
+uacce_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
{
+ const struct rte_uacce_driver *dr = RTE_BUS_DRIVER(drv, *dr);
+ const struct rte_uacce_device *uacce_dev = RTE_BUS_DEVICE(dev, *uacce_dev);
bool forward_compat = !!(dr->drv_flags & RTE_UACCE_DRV_FORWARD_COMPATIBILITY_DEV);
const struct rte_uacce_id *id_table;
const char *map;
uint32_t len;
for (id_table = dr->id_table; id_table->dev_api != NULL; id_table++) {
- if (!uacce_match_api(dev, forward_compat, id_table))
+ if (!uacce_match_api(uacce_dev, forward_compat, id_table))
continue;
if (id_table->dev_alg == NULL)
@@ -334,10 +336,10 @@ uacce_match(const struct rte_uacce_driver *dr, const struct rte_uacce_device *de
* algorithms: aaa, bbbb and cc.
* The id_table->dev_alg should be a single algrithm, e.g. bbbb.
*/
- map = strstr(dev->algs, id_table->dev_alg);
+ map = strstr(uacce_dev->algs, id_table->dev_alg);
if (map == NULL)
continue;
- if (map != dev->algs && map[-1] != '\n')
+ if (map != uacce_dev->algs && map[-1] != '\n')
continue;
len = strlen(id_table->dev_alg);
if (map[len] != '\0' && map[len] != '\n')
@@ -356,7 +358,7 @@ uacce_probe_one_driver(struct rte_uacce_driver *dr, struct rte_uacce_device *dev
bool already_probed;
int ret;
- if (!uacce_match(dr, dev))
+ if (!uacce_bus_match(&dr->driver, &dev->device))
/* Match of device and driver failed */
return 1;
@@ -623,6 +625,7 @@ static struct rte_uacce_bus uacce_bus = {
.scan = uacce_scan,
.probe = uacce_probe,
.cleanup = uacce_cleanup,
+ .match = uacce_bus_match,
.plug = uacce_plug,
.unplug = uacce_unplug,
.find_device = rte_bus_generic_find_device,
diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 4003805315..0308be5cbe 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -162,6 +162,25 @@ vdev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
return 0;
}
+/*
+ * Check if a vdev driver matches a vdev device by name.
+ */
+static bool
+vdev_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
+{
+ const char *name = dev->name;
+
+ /* Check driver name match */
+ if (strncmp(drv->name, name, strlen(drv->name)) == 0)
+ return true;
+
+ /* Check driver alias match */
+ if (drv->alias && strncmp(drv->alias, name, strlen(drv->alias)) == 0)
+ return true;
+
+ return false;
+}
+
static int
vdev_probe_all_drivers(struct rte_vdev_device *dev)
{
@@ -631,6 +650,7 @@ static struct rte_bus rte_vdev_bus = {
.probe = vdev_probe,
.cleanup = vdev_cleanup,
.find_device = vdev_find_device,
+ .match = vdev_bus_match,
.plug = vdev_plug,
.unplug = vdev_unplug,
.parse = vdev_parse,
diff --git a/drivers/bus/vmbus/vmbus_common.c b/drivers/bus/vmbus/vmbus_common.c
index 3260bd5395..d811f1a229 100644
--- a/drivers/bus/vmbus/vmbus_common.c
+++ b/drivers/bus/vmbus/vmbus_common.c
@@ -65,25 +65,15 @@ vmbus_unmap_resource(void *requested_addr, size_t size)
}
}
-/**
- * Match the VMBUS driver and device using UUID table
- *
- * @param drv
- * VMBUS driver from which ID table would be extracted
- * @param pci_dev
- * VMBUS device to match against the driver
- * @return
- * true for successful match
- * false for unsuccessful match
- */
static bool
-vmbus_match(const struct rte_vmbus_driver *dr,
- const struct rte_vmbus_device *dev)
+vmbus_bus_match(const struct rte_driver *drv, const struct rte_device *dev)
{
+ const struct rte_vmbus_driver *dr = RTE_BUS_DRIVER(drv, *dr);
+ const struct rte_vmbus_device *vmbus_dev = RTE_BUS_DEVICE(dev, *vmbus_dev);
const rte_uuid_t *id_table;
for (id_table = dr->id_table; !rte_uuid_is_null(*id_table); ++id_table) {
- if (rte_uuid_compare(*id_table, dev->class_id) == 0)
+ if (rte_uuid_compare(*id_table, vmbus_dev->class_id) == 0)
return true;
}
@@ -99,7 +89,7 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
char guid[RTE_UUID_STRLEN];
int ret;
- if (!vmbus_match(dr, dev))
+ if (!vmbus_bus_match(&dr->driver, &dev->device))
return 1; /* not supported */
rte_uuid_unparse(dev->device_id, guid, sizeof(guid));
@@ -281,6 +271,7 @@ struct rte_vmbus_bus rte_vmbus_bus = {
.probe = rte_vmbus_probe,
.cleanup = rte_vmbus_cleanup,
.find_device = rte_bus_generic_find_device,
+ .match = vmbus_bus_match,
.parse = vmbus_parse,
.dev_compare = vmbus_dev_compare,
},
diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c
index 269aac1946..cba26d0cdc 100644
--- a/drivers/dma/idxd/idxd_bus.c
+++ b/drivers/dma/idxd/idxd_bus.c
@@ -43,6 +43,7 @@ struct rte_dsa_device {
/* forward prototypes */
struct dsa_bus;
static int dsa_scan(void);
+static bool dsa_match(const struct rte_driver *drv, const struct rte_device *dev);
static int dsa_probe(void);
static enum rte_iova_mode dsa_get_iommu_class(void);
static int dsa_addr_parse(const char *name, void *addr);
@@ -58,6 +59,7 @@ struct dsa_bus {
struct dsa_bus dsa_bus = {
.bus = {
.scan = dsa_scan,
+ .match = dsa_match,
.probe = dsa_probe,
.find_device = rte_bus_generic_find_device,
.get_iommu_class = dsa_get_iommu_class,
@@ -126,7 +128,7 @@ idxd_bus_mmap_wq(struct rte_dsa_device *dev)
}
static int
-read_wq_string(struct rte_dsa_device *dev, const char *filename,
+read_wq_string(const struct rte_dsa_device *dev, const char *filename,
char *value, size_t valuelen)
{
char sysfs_node[PATH_MAX];
@@ -241,7 +243,7 @@ idxd_probe_dsa(struct rte_dsa_device *dev)
}
static int
-is_for_this_process_use(struct rte_dsa_device *dev, const char *name)
+is_for_this_process_use(const char *name)
{
char prefix[256];
int retval = 0;
@@ -256,9 +258,6 @@ is_for_this_process_use(struct rte_dsa_device *dev, const char *name)
if (strncmp(name, prefix, prefixlen) == 0 && name[prefixlen] == '_')
retval = 1;
- if (retval)
- retval = !rte_bus_device_is_ignored(&dsa_bus.bus, dev->device.name);
-
return retval;
}
@@ -268,14 +267,8 @@ dsa_probe(void)
struct rte_dsa_device *dev;
RTE_BUS_FOREACH_DEV(dev, &dsa_bus.bus) {
- char type[64], name[64];
-
- if (read_wq_string(dev, "type", type, sizeof(type)) < 0 ||
- read_wq_string(dev, "name", name, sizeof(name)) < 0)
- continue;
-
- if (strncmp(type, "user", 4) == 0 &&
- is_for_this_process_use(dev, name)) {
+ if (dsa_match(&dsa_bus.driver, &dev->device) &&
+ !rte_bus_device_is_ignored(&dsa_bus.bus, dev->device.name)) {
dev->device.driver = &dsa_bus.driver;
idxd_probe_dsa(dev);
continue;
@@ -286,6 +279,22 @@ dsa_probe(void)
return 0;
}
+static bool dsa_match(const struct rte_driver *drv, const struct rte_device *dev)
+{
+ const struct rte_dsa_device *dsa_dev = RTE_BUS_DEVICE(dev, *dsa_dev);
+
+ if (drv == &dsa_bus.driver) {
+ char type[64], name[64];
+
+ if (read_wq_string(dsa_dev, "type", type, sizeof(type)) >= 0 &&
+ read_wq_string(dsa_dev, "name", name, sizeof(name)) >= 0) {
+ return strncmp(type, "user", 4) == 0 && is_for_this_process_use(name);
+ }
+ }
+
+ return false;
+}
+
static int
dsa_scan(void)
{
diff --git a/lib/eal/common/eal_common_bus.c b/lib/eal/common/eal_common_bus.c
index 46a8e68532..4884cdfa50 100644
--- a/lib/eal/common/eal_common_bus.c
+++ b/lib/eal/common/eal_common_bus.c
@@ -473,3 +473,22 @@ rte_bus_generic_dev_iterate(const struct rte_bus *bus,
rte_kvargs_free(kvargs);
return dev;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_bus_find_driver)
+struct rte_driver *
+rte_bus_find_driver(const struct rte_bus *bus, const struct rte_driver *start,
+ const struct rte_device *dev)
+{
+ struct rte_driver *drv;
+
+ if (start != NULL)
+ drv = TAILQ_NEXT(start, next);
+ else
+ drv = TAILQ_FIRST(&bus->driver_list);
+ while (drv != NULL) {
+ if (bus->match(drv, dev))
+ break;
+ drv = TAILQ_NEXT(drv, next);
+ }
+ return drv;
+}
diff --git a/lib/eal/include/bus_driver.h b/lib/eal/include/bus_driver.h
index 9568d820e5..c4d9ac2719 100644
--- a/lib/eal/include/bus_driver.h
+++ b/lib/eal/include/bus_driver.h
@@ -233,6 +233,24 @@ typedef int (*rte_bus_sigbus_handler_t)(const void *failure_addr);
*/
typedef int (*rte_bus_cleanup_t)(void);
+/**
+ * Check if a driver matches a device.
+ *
+ * This function checks whether a driver can handle a given device.
+ * Matching logic is bus-specific (e.g., PCI uses ID tables, vdev uses
+ * name matching, fslmc uses device type).
+ *
+ * @param drv
+ * Driver to check.
+ * @param dev
+ * Device to check against.
+ *
+ * @return
+ * true if the driver matches the device, false otherwise.
+ */
+typedef bool (*rte_bus_match_t)(const struct rte_driver *drv,
+ const struct rte_device *dev);
+
/**
* Bus scan policies
*/
@@ -297,6 +315,7 @@ struct rte_bus {
rte_bus_scan_t scan; /**< Scan for devices attached to bus */
rte_bus_probe_t probe; /**< Probe devices on bus */
rte_bus_find_device_t find_device; /**< Find a device on the bus */
+ rte_bus_match_t match; /**< Check if driver matches device */
rte_bus_plug_t plug; /**< Probe single device for drivers */
rte_bus_unplug_t unplug; /**< Remove single device from driver */
rte_bus_parse_t parse; /**< Parse a device name */
@@ -544,6 +563,25 @@ void rte_bus_add_driver(struct rte_bus *bus, struct rte_driver *driver);
__rte_internal
void rte_bus_remove_driver(struct rte_bus *bus, struct rte_driver *driver);
+/**
+ * Find the first driver that matches a device on a bus.
+ *
+ * Iterates through all registered drivers on the bus and returns the next
+ * one that matches the given device according to the bus's match operation.
+ *
+ * @param bus
+ * A pointer to a rte_bus structure.
+ * @param start
+ * Starting iteration context.
+ * @param dev
+ * A pointer to a rte_device structure.
+ * @return
+ * Pointer to the matching driver, or NULL if no match found.
+ */
+__rte_internal
+struct rte_driver *rte_bus_find_driver(const struct rte_bus *bus, const struct rte_driver *start,
+ const struct rte_device *dev);
+
#ifdef __cplusplus
}
#endif
--
2.53.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox