[PATCH] doc: remove obsolete mempool creation advice

[PATCH] doc: remove obsolete mempool creation advice

[Morten Brørup]

The descriptions for the mempool creation functions contained advice for
choosing the optimum (in terms of memory usage) number of elements and
cache size.
The advice was based on implementation details, which was changed long
ago, making the advice completely irrelevant.

This obsolete advice was removed.

Also fixed a copy-paste bug in the rte_mempool_calc_obj_size() function
description.

Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
---
 lib/mbuf/rte_mbuf.h       | 12 +++---------
 lib/mempool/rte_mempool.h | 13 +++----------
 2 files changed, 6 insertions(+), 19 deletions(-)

diff --git a/lib/mbuf/rte_mbuf.h b/lib/mbuf/rte_mbuf.h
index 592af2388c..f571819211 100644
--- a/lib/mbuf/rte_mbuf.h
+++ b/lib/mbuf/rte_mbuf.h
@@ -780,9 +780,7 @@ void rte_pktmbuf_pool_init(struct rte_mempool *mp, void *opaque_arg);
  * @param name
  *   The name of the mbuf pool.
  * @param n
- *   The number of elements in the mbuf pool. The optimum size (in terms
- *   of memory usage) for a mempool is when n is a power of two minus one:
- *   n = (2^q - 1).
+ *   The number of elements in the mbuf pool.
  * @param cache_size
  *   Size of the per-core object cache. See rte_mempool_create() for
  *   details.
@@ -818,9 +816,7 @@ rte_pktmbuf_pool_create(const char *name, unsigned n,
  * @param name
  *   The name of the mbuf pool.
  * @param n
- *   The number of elements in the mbuf pool. The optimum size (in terms
- *   of memory usage) for a mempool is when n is a power of two minus one:
- *   n = (2^q - 1).
+ *   The number of elements in the mbuf pool.
  * @param cache_size
  *   Size of the per-core object cache. See rte_mempool_create() for
  *   details.
@@ -867,9 +863,7 @@ struct rte_pktmbuf_extmem {
  * @param name
  *   The name of the mbuf pool.
  * @param n
- *   The number of elements in the mbuf pool. The optimum size (in terms
- *   of memory usage) for a mempool is when n is a power of two minus one:
- *   n = (2^q - 1).
+ *   The number of elements in the mbuf pool.
  * @param cache_size
  *   Size of the per-core object cache. See rte_mempool_create() for
  *   details.
diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
index 2fa56464dc..2e54fc4466 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -1040,19 +1040,15 @@ rte_mempool_free(struct rte_mempool *mp);
  * @param name
  *   The name of the mempool.
  * @param n
- *   The number of elements in the mempool. The optimum size (in terms of
- *   memory usage) for a mempool is when n is a power of two minus one:
- *   n = (2^q - 1).
+ *   The number of elements in the mempool.
  * @param elt_size
  *   The size of each element.
  * @param cache_size
  *   If cache_size is non-zero, the rte_mempool library will try to
  *   limit the accesses to the common lockless pool, by maintaining a
  *   per-lcore object cache. This argument must be lower or equal to
- *   RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5. It is advised to choose
- *   cache_size to have "n modulo cache_size == 0": if this is
- *   not the case, some elements will always stay in the pool and will
- *   never be used. The access to the per-lcore table is of course
+ *   RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.
+ *   The access to the per-lcore table is of course
  *   faster than the multi-producer/consumer pool. The cache can be
  *   disabled if the cache_size argument is set to 0; it can be useful to
  *   avoid losing objects in cache.
@@ -1130,8 +1126,6 @@ rte_mempool_create(const char *name, unsigned n, unsigned elt_size,
  *   The name of the mempool.
  * @param n
  *   The maximum number of elements that can be added in the mempool.
- *   The optimum size (in terms of memory usage) for a mempool is when n
- *   is a power of two minus one: n = (2^q - 1).
  * @param elt_size
  *   The size of each element.
  * @param cache_size
@@ -1889,7 +1883,6 @@ struct rte_mempool *rte_mempool_lookup(const char *name);
  * @param flags
  *   The flags used for the mempool creation.
  *   Consult rte_mempool_create() for more information about possible values.
- *   The size of each element.
  * @param sz
  *   The calculated detailed size the mempool object. May be NULL.
  * @return
-- 
2.43.0

Re: [PATCH] doc: remove obsolete mempool creation advice

[Bruce Richardson]

On Thu, Mar 19, 2026 at 09:13:00AM +0000, Morten Brørup wrote:
> The descriptions for the mempool creation functions contained advice for
> choosing the optimum (in terms of memory usage) number of elements and
> cache size.
> The advice was based on implementation details, which was changed long
> ago, making the advice completely irrelevant.
> 

The comment is still correct in most cases, since the default backing
storage remains an rte_ring. If passing a power-of-2 size to mempool create
one will get a backing rte_ring store which is twice as large as requested,
leading to lots of ring slots being wasted. For example, for a pool with
16k elements, the actual ring size allocated will be 32k, leading to
wasting 128k of RAM, and also potentially cache too. The latter will occur
because of the nature of the ring to iterate through all mempool/ring
entries, meaning that even if only 16k of the 32k slots will ever be used,
all 32k slots will be passed through the cpu cache if it works on the
mempool directly and not just from the per-core cache.

On the other hand, I'd be in favour of removing this text if we switched
the default mempool in DPDK to being stack-based. While the stack may not
be lock-free like the ring, with per-lcore caches the number of accesses to
the stack should be small, and it gives much better cache utilization
overall - especially in cases where buffers are allocated on one core and
freed on a different one! Even in cases where we are not transferring
between cores, in a single-core case we still will get better reuse of
"hot" buffers than in an rte_ring-backed case.

/Bruce

RE: [PATCH] doc: remove obsolete mempool creation advice

[Morten Brørup]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Thursday, 19 March 2026 10.41
> 
> On Thu, Mar 19, 2026 at 09:13:00AM +0000, Morten Brørup wrote:
> > The descriptions for the mempool creation functions contained advice
> for
> > choosing the optimum (in terms of memory usage) number of elements
> and
> > cache size.
> > The advice was based on implementation details, which was changed
> long
> > ago, making the advice completely irrelevant.
> >
> 
> The comment is still correct in most cases, since the default backing
> storage remains an rte_ring. If passing a power-of-2 size to mempool
> create
> one will get a backing rte_ring store which is twice as large as
> requested,
> leading to lots of ring slots being wasted. For example, for a pool
> with
> 16k elements, the actual ring size allocated will be 32k, leading to
> wasting 128k of RAM, and also potentially cache too. The latter will
> occur
> because of the nature of the ring to iterate through all mempool/ring
> entries, meaning that even if only 16k of the 32k slots will ever be
> used,
> all 32k slots will be passed through the cpu cache if it works on the
> mempool directly and not just from the per-core cache.

You are right about the waste of memory in the ring driver. And good point about the CPU cache!

However, only pointer entries (8 byte each) are being wasted, not object entries (which are much larger). This is not 100 % clear from the advice.

Furthermore, with 16k mbufs of 2368 byte each, the mempool itself consumes 37 MB worth of memory, so do we really care about wasting 128 KB?

IMHO, removing the advice improves the quality of the documentation.
I don't think a detail about saving 0.3 % of the memory used by the mempool should be presented so prominently in the documentation.

Obviously, the memory waste percentage for a mempool holding smaller objects will be larger.
With 64 byte objects (+64 byte object header), the waste is 8 bytes in the ring driver per 128 bytes in the mempool itself, ca. 6 %.
Still relatively small.

Alternatively, we could keep the advice and change the wording. Something like:

If the mempool uses a ring driver, the optimum size (in terms of memory) is when n is a power of two minus one, n = (2^q - 1); otherwise the ring's array of pointers to the objects will be larger, as it's size is aligned to the next power of 2.

It's difficult to keep the advice short, if we add the fine print to it.

But then the advice is still missing for all the other mempool drivers using nonlinear memory allocation.
I'm not sure, but I think the "bucket" driver also uses nonlinear memory allocation.

Another alternative:
The mempool drivers or underlying libraries could log a debug message when allocating an oversize array for alignment reasons.

> 
> On the other hand, I'd be in favour of removing this text if we
> switched
> the default mempool in DPDK to being stack-based. While the stack may
> not
> be lock-free like the ring, with per-lcore caches the number of
> accesses to
> the stack should be small, and it gives much better cache utilization
> overall - especially in cases where buffers are allocated on one core
> and
> freed on a different one! Even in cases where we are not transferring
> between cores, in a single-core case we still will get better reuse of
> "hot" buffers than in an rte_ring-backed case.

<jokingly>
The lock free stack driver's reference to each object uses 2 pointers, so guaranteed waste of memory here.
</jokingly>

Let's discuss changing the default mempool driver some other day, and focus on the documentation for now.

I get your point, though.
It makes sense that the documentation for a library considers the underlying default implementation.

But if it does, the advice should come with a disclaimer mentioning when it applies.

Re: [PATCH] doc: remove obsolete mempool creation advice

[Bruce Richardson]

On Thu, Mar 19, 2026 at 11:55:11AM +0100, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Thursday, 19 March 2026 10.41
> > 
> > On Thu, Mar 19, 2026 at 09:13:00AM +0000, Morten Brørup wrote:
> > > The descriptions for the mempool creation functions contained advice
> > for
> > > choosing the optimum (in terms of memory usage) number of elements
> > and
> > > cache size.
> > > The advice was based on implementation details, which was changed
> > long
> > > ago, making the advice completely irrelevant.
> > >
> > 
> > The comment is still correct in most cases, since the default backing
> > storage remains an rte_ring. If passing a power-of-2 size to mempool
> > create
> > one will get a backing rte_ring store which is twice as large as
> > requested,
> > leading to lots of ring slots being wasted. For example, for a pool
> > with
> > 16k elements, the actual ring size allocated will be 32k, leading to
> > wasting 128k of RAM, and also potentially cache too. The latter will
> > occur
> > because of the nature of the ring to iterate through all mempool/ring
> > entries, meaning that even if only 16k of the 32k slots will ever be
> > used,
> > all 32k slots will be passed through the cpu cache if it works on the
> > mempool directly and not just from the per-core cache.
> 
> You are right about the waste of memory in the ring driver. And good point about the CPU cache!
> 
> However, only pointer entries (8 byte each) are being wasted, not object entries (which are much larger). This is not 100 % clear from the advice.
> 
> Furthermore, with 16k mbufs of 2368 byte each, the mempool itself consumes 37 MB worth of memory, so do we really care about wasting 128 KB?
> 
> IMHO, removing the advice improves the quality of the documentation.
> I don't think a detail about saving 0.3 % of the memory used by the mempool should be presented so prominently in the documentation.
> 

Ok, point taken. It would actually be the cache wastage that would worry me
more, but again the cache use from the extra ring space is probably small
compared to that from the buffers if we are cycling through the whole
mempool.

/Bruce

Re: [PATCH] doc: remove obsolete mempool creation advice

[Bruce Richardson]

On Thu, Mar 19, 2026 at 09:13:00AM +0000, Morten Brørup wrote:
> The descriptions for the mempool creation functions contained advice for
> choosing the optimum (in terms of memory usage) number of elements and
> cache size.
> The advice was based on implementation details, which was changed long
> ago, making the advice completely irrelevant.
> 
> This obsolete advice was removed.
> 
> Also fixed a copy-paste bug in the rte_mempool_calc_obj_size() function
> description.
> 
> Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
> ---
Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [PATCH] doc: remove obsolete mempool creation advice

[Andrew Rybchenko]

On 3/19/26 3:03 PM, Bruce Richardson wrote:
> On Thu, Mar 19, 2026 at 09:13:00AM +0000, Morten Brørup wrote:
>> The descriptions for the mempool creation functions contained advice for
>> choosing the optimum (in terms of memory usage) number of elements and
>> cache size.
>> The advice was based on implementation details, which was changed long
>> ago, making the advice completely irrelevant.
>>
>> This obsolete advice was removed.
>>
>> Also fixed a copy-paste bug in the rte_mempool_calc_obj_size() function
>> description.
>>
>> Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
>> ---
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

Re: [PATCH] doc: remove obsolete mempool creation advice

[Thomas Monjalon]

20/03/2026 06:44, Andrew Rybchenko:
> On 3/19/26 3:03 PM, Bruce Richardson wrote:
> > On Thu, Mar 19, 2026 at 09:13:00AM +0000, Morten Brørup wrote:
> >> The descriptions for the mempool creation functions contained advice for
> >> choosing the optimum (in terms of memory usage) number of elements and
> >> cache size.
> >> The advice was based on implementation details, which was changed long
> >> ago, making the advice completely irrelevant.
> >>
> >> This obsolete advice was removed.
> >>
> >> Also fixed a copy-paste bug in the rte_mempool_calc_obj_size() function
> >> description.
> >>
> >> Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
> >> ---
> > Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

Applied, thanks.