RE: mempool cache change

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
> Sent: Friday, 17 July 2026 19.10
> 
> Hi Bruce,
> 
> The below patch works fine for us. We tested all the different packet
> sizes.
> Thanks for the patch. Do you want to push this patch since it is not
> changing the ABI/API?

Too late in the release process.

Let's postpone the discussion for DPDK 26.11, where API/ABI breakage is allowed.

I'm not strongly opposed to Bruce's algorithm, targeting a fill level of 25 % from the edges and flushing/refilling up to 75 % of the cache when necessary. It does have its advantages for some mempool access patterns (which are not exotic).
I just prefer the current algorithm, targeting a fill level of 50 % and only flushing/refilling up to 50 % of the cache when necessary. It performs better at random get/put access patterns, and the backend transactions are smaller.

For DPDK 26.11, where we can break the API/ABI, we can simply double RTE_MEMPOOL_CACHE_MAX_SIZE to 1024, to compensate for reducing the effective cache size from 150 % to 100 %.
The mempool cache objs array will no longer be [RTE_MEMPOOL_CACHE_MAX_SIZE * 2], but only [RTE_MEMPOOL_CACHE_MAX_SIZE], so doubling RTE_MEMPOOL_CACHE_MAX_SIZE will not increase the memory footprint, but allow using a cache size up to 1024.

Please also note that the current implementation is carefully designed to keep the transfers to/from the mempool backend CPU cache aligned (assuming cache->size is 2^N and large enough).
Refer to the parameters passed to rte_mempool_ops_enqueue/dequeue_bulk().
E.g. with mempool cache size 256, backend transfers are 128 objects, 16 full cache lines.
Using CPU cache aligned transfers has a few advantages:
- There are no cache line ownership issues across different CPU cores repeatedly accessing the backend.
- The mempool backend drivers can be performance optimized for transferring full CPU cache lines. (Both source and destination addresses, and number of objects copied, are CPU cache aligned. Assuming all transfers go via the mempool cache.)

These details should be fine tuned in the implementation, if we do proceed with Bruce's algorithm.

-Morten

> 
> Rgds,
> Kishore
> 
> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Thursday, July 16, 2026 5:57 AM
> To: Kishore Padmanabha <kishore.padmanabha@broadcom.com>
> 
> On Wed, Jul 15, 2026 at 11:12:59AM -0400, Kishore Padmanabha wrote:
> >    On Wed, Jul 15, 2026 at 5:02 AM Morten Brørup
> >    <[1]mb@smartsharesystems.com> wrote:
> >
> >      > From: fengchengwen [mailto:[2]fengchengwen@huawei.com]
> >      > Sent: Wednesday, 15 July 2026 10.12
> >      >
> >      > On 7/15/2026 4:08 AM, Morten Brørup wrote:
> >      > > Hi Kishore,
> >      > >
> >      > > For your testing purposes, please follow the guidance
> provided
> >      to
> >      > Wisam Jaddo:
> >      > >
> >      >
> >
> [3]https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35F65964
> >      @smart
> >      > [4]server.smartshare.dk/
> >      >
> >      > We need to recompile in this case. We should try to avoid
> >      > recompilation.
> >      > I think it is necessary to adjust RTE_MEMPOOL_CACHE_MAX_SIZE
> to
> >      768 as
> >      > a default configuration.
> >      Changing RTE_MEMPOOL_CACHE_MAX_SIZE breaks both the API and the
> ABI;
> >      so it has to be done when building locally, where API/ABI
> breakage
> >      is acceptable.
> >      For DPDK 26.11, where API/ABI breakage is acceptable, we can
> discuss
> >      increasing the default from 512 to a higher value. I do have
> some
> >      input to that discussion, but let's postpone it until after DPDK
> >      26.07 has been released.
> >
> >    We should increase this value to avoid performance degradation, as
> >    users may not realize they need to change it.  We do not have do
> it
> >    right away for 26.07 release but we should it right after the
> >    release.
> >
> 
> Out of interest, does adjusting the 50% fill/flush threshold to be a
> 75%
> one, i.e. fill to 75% rather than 50%, flush to 25% rather than 50%,
> help at
> all? Draft patch below, can you test it quickly, perhaps?
> 
> /Bruce
> 
> diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
> index
> 50d958c7c6..2526a23903 100644
> --- a/lib/mempool/rte_mempool.h
> +++ b/lib/mempool/rte_mempool.h
> @@ -1417,6 +1417,7 @@ rte_mempool_do_generic_put(struct rte_mempool
> *mp,
> void * const *obj_table,
>                            unsigned int n, struct rte_mempool_cache
> *cache)
> {
>         void **cache_objs;
> +       uint32_t quarter, flush;
> 
>         /* No cache provided? */
>         if (unlikely(cache == NULL))
> @@ -1426,30 +1427,39 @@ rte_mempool_do_generic_put(struct rte_mempool
> *mp,
> void * const *obj_table,
>         RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
>         RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
> 
> +       /* A quarter (25%) of the cache size, computed with a shift to
> avoid
> +        * a divide. Draining the cache down to this level on overflow
> leaves
> +        * room for a burst of up to (size - quarter), i.e. three
> quarters
> +        * (75%), of the cache size.
> +        */
> +       quarter = cache->size >> 2;
> +
>         __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> -       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE /
> 2);
> +       __rte_assume(quarter <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
>         __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
>         __rte_assume(cache->len <= cache->size);
>         if (likely(cache->len + n <= cache->size)) {
>                 /* Sufficient room in the cache for the objects. */
>                 cache_objs = &cache->objs[cache->len];
>                 cache->len += n;
> -       } else if (n <= cache->size / 2) {
> +       } else if (n <= cache->size - quarter) {
>                 /*
>                  * The number of objects is within the cache bounce
> buffer
> limit,
>                  * but - as detected by the comparison above - the
> cache has
>                  * insufficient room for them.
>                  * Flush the cache to the backend to make room for the
> objects;
> -                * flush (size / 2) objects from the bottom of the
> cache,
> where
> -                * objects are less hot, and move down the remaining
> objects, which
> -                * are more hot, from the upper half of the cache.
> +                * flush objects from the bottom of the cache, where
> objects
> are
> +                * less hot, draining it down to a quarter (25%) of its
> size, and
> +                * move down the remaining quarter of objects, which
> are
> more hot,
> +                * from the upper part of the cache.
>                  */
> -               __rte_assume(cache->len > cache->size / 2);
> -               rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0],
> cache->size / 2);
> -               rte_memcpy(&cache->objs[0], &cache->objs[cache->size /
> 2],
> -                               sizeof(void *) * (cache->len - cache-
> >size /
> 2));
> -               cache_objs = &cache->objs[cache->len - cache->size /
> 2];
> -               cache->len = cache->len - cache->size / 2 + n;
> +               __rte_assume(cache->len > quarter);
> +               flush = cache->len - quarter;
> +               rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0],
> flush);
> +               rte_memcpy(&cache->objs[0], &cache->objs[flush],
> +                               sizeof(void *) * quarter);
> +               cache_objs = &cache->objs[quarter];
> +               cache->len = quarter + n;
>         } else {
>                 /* The request itself is too big for the cache. */
>                 goto driver_enqueue_stats_incremented; @@ -1557,6
> +1567,7 @@
> rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
>         int ret;
>         unsigned int remaining;
>         uint32_t index, len;
> +       uint32_t quarter, fill;
>         void **cache_objs;
> 
>         /* No cache provided? */
> @@ -1592,13 +1603,23 @@ rte_mempool_do_generic_get(struct rte_mempool
> *mp,
> void **obj_table,
>         for (index = 0; index < len; index++)
>                 *obj_table++ = *--cache_objs;
> 
> +       /* A quarter (25%) of the cache size, computed with a shift to
> avoid
> +        * a divide; 'fill' is the complementary three quarters (75%),
> which
> +        * is the amount fetched from the backend to fill the cache up
> to
> 75%,
> +        * and also the burst limit for this bounce buffer (since the
> cache
> +        * was just fully drained above, up to 'fill' objects can be
> filled
> +        * and handed back to the caller in one go).
> +        */
> +       quarter = cache->size >> 2;
> +       fill = cache->size - quarter;
> +
>         /* Dequeue below would exceed the cache bounce buffer limit? */
> -       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE /
> 2);
> -       if (unlikely(remaining > cache->size / 2))
> +       __rte_assume(fill <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> +       if (unlikely(remaining > fill))
>                 goto driver_dequeue;
> 
> -       /* Fill the cache from the backend; fetch (size / 2) objects.
> */
> -       ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache->size
> /
> 2);
> +       /* Fill the cache from the backend, up to 75% of its size. */
> +       ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, fill);
>         if (unlikely(ret < 0)) {
>                 /*
>                  * We are buffer constrained, and not able to fetch all
> that.
> @@ -1612,11 +1633,10 @@ rte_mempool_do_generic_get(struct rte_mempool
> *mp,
> void **obj_table,
>         RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
>         RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
> 
> -       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE /
> 2);
> -       __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> -       __rte_assume(remaining <= cache->size / 2);
> -       cache_objs = &cache->objs[cache->size / 2];
> -       cache->len = cache->size / 2 - remaining;
> +       __rte_assume(fill <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> +       __rte_assume(remaining <= fill);
> +       cache_objs = &cache->objs[fill];
> +       cache->len = fill - remaining;
>         for (index = 0; index < remaining; index++)
>                 *obj_table++ = *--cache_objs;