mempool cache change

mempool cache change

[Kishore Padmanabha]

[-- Attachment #1.1: Type: text/plain, Size: 738 bytes --]

Hi Thomas,



The recent change done as part of BUG ID: 1027, with commit id
https://git.dpdk.org/dpdk/commit/?id=f5e1310f16e0909e7e7f71807123644c63b23cba

Is causing performance drop for the bnxt NIC in 26.07 release.



The smaller, shallower cache increases the miss rate, causing refills to
hit the shared ring more frequently.
In the old code, the cache only touched the shared ring when it climbed
past 768 instead of 512. That extra 256 elements of headroom allowed
significantly more frees to remain local.



Could we make the change in config/rte_config.h for
RTE_MEMPOOL_CACHE_MAX_SIZE to be 768 instead of 512. Since we use
–mbcache=512 as argument for the performance tests.



Thanks,

Kishore

[-- Attachment #1.2: Type: text/html, Size: 2685 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5493 bytes --]

RE: mempool cache change

[Morten Brørup]

Hi Kishore,

For your testing purposes, please follow the guidance provided to Wisam Jaddo:
https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35F65964@smartserver.smartshare.dk/

Thank you for testing!

Venlig hilsen / Kind regards,
-Morten Brørup

From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com] 
Sent: Tuesday, 14 July 2026 18.49
To: Thomas Monjalon; Morten Brørup
Cc: dev@dpdk.org
Subject: mempool cache change

Hi Thomas,
 
The recent change done as part of BUG ID: 1027, with commit id  https://git.dpdk.org/dpdk/commit/?id=f5e1310f16e0909e7e7f71807123644c63b23cba
Is causing performance drop for the bnxt NIC in 26.07 release.
 
The smaller, shallower cache increases the miss rate, causing refills to hit the shared ring more frequently. 
In the old code, the cache only touched the shared ring when it climbed past 768 instead of 512. That extra 256 elements of headroom allowed significantly more frees to remain local.
 
Could we make the change in config/rte_config.h for RTE_MEMPOOL_CACHE_MAX_SIZE to be 768 instead of 512. Since we use –mbcache=512 as argument for the performance tests.
 
Thanks,
Kishore
 
 

Re: mempool cache change

[fengchengwen]

On 7/15/2026 4:08 AM, Morten Brørup wrote:
> Hi Kishore,
> 
> For your testing purposes, please follow the guidance provided to Wisam Jaddo:
> https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35F65964@smartserver.smartshare.dk/

We need to recompile in this case. We should try to avoid recompilation.
I think it is necessary to adjust RTE_MEMPOOL_CACHE_MAX_SIZE to 768 as a
default configuration.

Thanks

> 
> Thank you for testing!
> 
> Venlig hilsen / Kind regards,
> -Morten Brørup
> 
> From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com] 
> Sent: Tuesday, 14 July 2026 18.49
> To: Thomas Monjalon; Morten Brørup
> Cc: dev@dpdk.org
> Subject: mempool cache change
> 
> Hi Thomas,
>  
> The recent change done as part of BUG ID: 1027, with commit id  https://git.dpdk.org/dpdk/commit/?id=f5e1310f16e0909e7e7f71807123644c63b23cba
> Is causing performance drop for the bnxt NIC in 26.07 release.
>  
> The smaller, shallower cache increases the miss rate, causing refills to hit the shared ring more frequently. 
> In the old code, the cache only touched the shared ring when it climbed past 768 instead of 512. That extra 256 elements of headroom allowed significantly more frees to remain local.
>  
> Could we make the change in config/rte_config.h for RTE_MEMPOOL_CACHE_MAX_SIZE to be 768 instead of 512. Since we use –mbcache=512 as argument for the performance tests.
>  
> Thanks,
> Kishore
>  
>  

RE: mempool cache change

[Morten Brørup]

> From: fengchengwen [mailto:fengchengwen@huawei.com]
> Sent: Wednesday, 15 July 2026 10.12
> 
> On 7/15/2026 4:08 AM, Morten Brørup wrote:
> > Hi Kishore,
> >
> > For your testing purposes, please follow the guidance provided to
> Wisam Jaddo:
> >
> https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35F65964@smart
> server.smartshare.dk/
> 
> We need to recompile in this case. We should try to avoid
> recompilation.
> I think it is necessary to adjust RTE_MEMPOOL_CACHE_MAX_SIZE to 768 as
> a default configuration.

Changing RTE_MEMPOOL_CACHE_MAX_SIZE breaks both the API and the ABI; so it has to be done when building locally, where API/ABI breakage is acceptable.

For DPDK 26.11, where API/ABI breakage is acceptable, we can discuss increasing the default from 512 to a higher value. I do have some input to that discussion, but let's postpone it until after DPDK 26.07 has been released.

> 
> Thanks
> 
> >
> > Thank you for testing!
> >
> > Venlig hilsen / Kind regards,
> > -Morten Brørup
> >
> > From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
> > Sent: Tuesday, 14 July 2026 18.49
> > To: Thomas Monjalon; Morten Brørup
> > Cc: dev@dpdk.org
> > Subject: mempool cache change
> >
> > Hi Thomas,
> >
> > The recent change done as part of BUG ID: 1027, with commit id
> https://git.dpdk.org/dpdk/commit/?id=f5e1310f16e0909e7e7f71807123644c63
> b23cba
> > Is causing performance drop for the bnxt NIC in 26.07 release.
> >
> > The smaller, shallower cache increases the miss rate, causing refills
> to hit the shared ring more frequently.
> > In the old code, the cache only touched the shared ring when it
> climbed past 768 instead of 512. That extra 256 elements of headroom
> allowed significantly more frees to remain local.
> >
> > Could we make the change in config/rte_config.h for
> RTE_MEMPOOL_CACHE_MAX_SIZE to be 768 instead of 512. Since we use –
> mbcache=512 as argument for the performance tests.
> >
> > Thanks,
> > Kishore
> >
> >

Re: mempool cache change

[Kishore Padmanabha]

[-- Attachment #1.1: Type: text/plain, Size: 2473 bytes --]

On Wed, Jul 15, 2026 at 5:02 AM Morten Brørup <mb@smartsharesystems.com>
wrote:

> > From: fengchengwen [mailto:fengchengwen@huawei.com]
> > Sent: Wednesday, 15 July 2026 10.12
> >
> > On 7/15/2026 4:08 AM, Morten Brørup wrote:
> > > Hi Kishore,
> > >
> > > For your testing purposes, please follow the guidance provided to
> > Wisam Jaddo:
> > >
> > https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35F65964@smart
> > server.smartshare.dk/
> >
> > We need to recompile in this case. We should try to avoid
> > recompilation.
> > I think it is necessary to adjust RTE_MEMPOOL_CACHE_MAX_SIZE to 768 as
> > a default configuration.
>
> Changing RTE_MEMPOOL_CACHE_MAX_SIZE breaks both the API and the ABI; so it
> has to be done when building locally, where API/ABI breakage is acceptable.
>
> For DPDK 26.11, where API/ABI breakage is acceptable, we can discuss
> increasing the default from 512 to a higher value. I do have some input to
> that discussion, but let's postpone it until after DPDK 26.07 has been
> released.
>
We should increase this value to avoid performance degradation, as users
may not realize they need to change it.  We do not have do it right away
for 26.07 release but we should it right after the release.

>
> >
> > Thanks
> >
> > >
> > > Thank you for testing!
> > >
> > > Venlig hilsen / Kind regards,
> > > -Morten Brørup
> > >
> > > From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
> > > Sent: Tuesday, 14 July 2026 18.49
> > > To: Thomas Monjalon; Morten Brørup
> > > Cc: dev@dpdk.org
> > > Subject: mempool cache change
> > >
> > > Hi Thomas,
> > >
> > > The recent change done as part of BUG ID: 1027, with commit id
> > https://git.dpdk.org/dpdk/commit/?id=f5e1310f16e0909e7e7f71807123644c63
> > b23cba
> > > Is causing performance drop for the bnxt NIC in 26.07 release.
> > >
> > > The smaller, shallower cache increases the miss rate, causing refills
> > to hit the shared ring more frequently.
> > > In the old code, the cache only touched the shared ring when it
> > climbed past 768 instead of 512. That extra 256 elements of headroom
> > allowed significantly more frees to remain local.
> > >
> > > Could we make the change in config/rte_config.h for
> > RTE_MEMPOOL_CACHE_MAX_SIZE to be 768 instead of 512. Since we use –
> > mbcache=512 as argument for the performance tests.
> > >
> > > Thanks,
> > > Kishore
> > >
> > >
>
>

[-- Attachment #1.2: Type: text/html, Size: 3797 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5493 bytes --]

Re: mempool cache change

[Bruce Richardson]

On Wed, Jul 15, 2026 at 11:12:59AM -0400, Kishore Padmanabha wrote:
>    On Wed, Jul 15, 2026 at 5:02 AM Morten Brørup
>    <[1]mb@smartsharesystems.com> wrote:
> 
>      > From: fengchengwen [mailto:[2]fengchengwen@huawei.com]
>      > Sent: Wednesday, 15 July 2026 10.12
>      >
>      > On 7/15/2026 4:08 AM, Morten Brørup wrote:
>      > > Hi Kishore,
>      > >
>      > > For your testing purposes, please follow the guidance provided
>      to
>      > Wisam Jaddo:
>      > >
>      >
>      [3]https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35F65964
>      @smart
>      > [4]server.smartshare.dk/
>      >
>      > We need to recompile in this case. We should try to avoid
>      > recompilation.
>      > I think it is necessary to adjust RTE_MEMPOOL_CACHE_MAX_SIZE to
>      768 as
>      > a default configuration.
>      Changing RTE_MEMPOOL_CACHE_MAX_SIZE breaks both the API and the ABI;
>      so it has to be done when building locally, where API/ABI breakage
>      is acceptable.
>      For DPDK 26.11, where API/ABI breakage is acceptable, we can discuss
>      increasing the default from 512 to a higher value. I do have some
>      input to that discussion, but let's postpone it until after DPDK
>      26.07 has been released.
> 
>    We should increase this value to avoid performance degradation, as
>    users may not realize they need to change it.  We do not have do it
>    right away for 26.07 release but we should it right after the
>    release.
> 

Out of interest, does adjusting the 50% fill/flush threshold to be a 75%
one, i.e. fill to 75% rather than 50%, flush to 25% rather than 50%, help at
all? Draft patch below, can you test it quickly, perhaps?

/Bruce

diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
index 50d958c7c6..2526a23903 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -1417,6 +1417,7 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
                           unsigned int n, struct rte_mempool_cache *cache)
 {
        void **cache_objs;
+       uint32_t quarter, flush;
 
        /* No cache provided? */
        if (unlikely(cache == NULL))
@@ -1426,30 +1427,39 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
        RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
        RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
+       /* A quarter (25%) of the cache size, computed with a shift to avoid
+        * a divide. Draining the cache down to this level on overflow leaves
+        * room for a burst of up to (size - quarter), i.e. three quarters
+        * (75%), of the cache size.
+        */
+       quarter = cache->size >> 2;
+
        __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
-       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+       __rte_assume(quarter <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
        __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
        __rte_assume(cache->len <= cache->size);
        if (likely(cache->len + n <= cache->size)) {
                /* Sufficient room in the cache for the objects. */
                cache_objs = &cache->objs[cache->len];
                cache->len += n;
-       } else if (n <= cache->size / 2) {
+       } else if (n <= cache->size - quarter) {
                /*
                 * The number of objects is within the cache bounce buffer limit,
                 * but - as detected by the comparison above - the cache has
                 * insufficient room for them.
                 * Flush the cache to the backend to make room for the objects;
-                * flush (size / 2) objects from the bottom of the cache, where
-                * objects are less hot, and move down the remaining objects, which
-                * are more hot, from the upper half of the cache.
+                * flush objects from the bottom of the cache, where objects are
+                * less hot, draining it down to a quarter (25%) of its size, and
+                * move down the remaining quarter of objects, which are more hot,
+                * from the upper part of the cache.
                 */
-               __rte_assume(cache->len > cache->size / 2);
-               rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0], cache->size / 2);
-               rte_memcpy(&cache->objs[0], &cache->objs[cache->size / 2],
-                               sizeof(void *) * (cache->len - cache->size / 2));
-               cache_objs = &cache->objs[cache->len - cache->size / 2];
-               cache->len = cache->len - cache->size / 2 + n;
+               __rte_assume(cache->len > quarter);
+               flush = cache->len - quarter;
+               rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0], flush);
+               rte_memcpy(&cache->objs[0], &cache->objs[flush],
+                               sizeof(void *) * quarter);
+               cache_objs = &cache->objs[quarter];
+               cache->len = quarter + n;
        } else {
                /* The request itself is too big for the cache. */
                goto driver_enqueue_stats_incremented;
@@ -1557,6 +1567,7 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
        int ret;
        unsigned int remaining;
        uint32_t index, len;
+       uint32_t quarter, fill;
        void **cache_objs;
 
        /* No cache provided? */
@@ -1592,13 +1603,23 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
        for (index = 0; index < len; index++)
                *obj_table++ = *--cache_objs;
 
+       /* A quarter (25%) of the cache size, computed with a shift to avoid
+        * a divide; 'fill' is the complementary three quarters (75%), which
+        * is the amount fetched from the backend to fill the cache up to 75%,
+        * and also the burst limit for this bounce buffer (since the cache
+        * was just fully drained above, up to 'fill' objects can be filled
+        * and handed back to the caller in one go).
+        */
+       quarter = cache->size >> 2;
+       fill = cache->size - quarter;
+
        /* Dequeue below would exceed the cache bounce buffer limit? */
-       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
-       if (unlikely(remaining > cache->size / 2))
+       __rte_assume(fill <= RTE_MEMPOOL_CACHE_MAX_SIZE);
+       if (unlikely(remaining > fill))
                goto driver_dequeue;
 
-       /* Fill the cache from the backend; fetch (size / 2) objects. */
-       ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache->size / 2);
+       /* Fill the cache from the backend, up to 75% of its size. */
+       ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, fill);
        if (unlikely(ret < 0)) {
                /*
                 * We are buffer constrained, and not able to fetch all that.
@@ -1612,11 +1633,10 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
        RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
        RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
 
-       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
-       __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
-       __rte_assume(remaining <= cache->size / 2);
-       cache_objs = &cache->objs[cache->size / 2];
-       cache->len = cache->size / 2 - remaining;
+       __rte_assume(fill <= RTE_MEMPOOL_CACHE_MAX_SIZE);
+       __rte_assume(remaining <= fill);
+       cache_objs = &cache->objs[fill];
+       cache->len = fill - remaining;
        for (index = 0; index < remaining; index++)
                *obj_table++ = *--cache_objs;

RE: mempool cache change

[Kishore Padmanabha]

[-- Attachment #1: Type: text/plain, Size: 8443 bytes --]

Hi Bruce,

The below patch works fine for us. We tested all the different packet sizes.
Thanks for the patch. Do you want to push this patch since it is not
changing the ABI/API?

Rgds,
Kishore

-----Original Message-----
From: Bruce Richardson <bruce.richardson@intel.com>
Sent: Thursday, July 16, 2026 5:57 AM
To: Kishore Padmanabha <kishore.padmanabha@broadcom.com>
Cc: Morten Brørup <mb@smartsharesystems.com>; fengchengwen
<fengchengwen@huawei.com>; Thomas Monjalon <thomas@monjalon.net>;
dev@dpdk.org; Wisam Jaddo <wisamm@nvidia.com>
Subject: Re: mempool cache change

On Wed, Jul 15, 2026 at 11:12:59AM -0400, Kishore Padmanabha wrote:
>    On Wed, Jul 15, 2026 at 5:02 AM Morten Brørup
>    <[1]mb@smartsharesystems.com> wrote:
>
>      > From: fengchengwen [mailto:[2]fengchengwen@huawei.com]
>      > Sent: Wednesday, 15 July 2026 10.12
>      >
>      > On 7/15/2026 4:08 AM, Morten Brørup wrote:
>      > > Hi Kishore,
>      > >
>      > > For your testing purposes, please follow the guidance provided
>      to
>      > Wisam Jaddo:
>      > >
>      >
>      [3]https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35F65964
>      @smart
>      > [4]server.smartshare.dk/
>      >
>      > We need to recompile in this case. We should try to avoid
>      > recompilation.
>      > I think it is necessary to adjust RTE_MEMPOOL_CACHE_MAX_SIZE to
>      768 as
>      > a default configuration.
>      Changing RTE_MEMPOOL_CACHE_MAX_SIZE breaks both the API and the ABI;
>      so it has to be done when building locally, where API/ABI breakage
>      is acceptable.
>      For DPDK 26.11, where API/ABI breakage is acceptable, we can discuss
>      increasing the default from 512 to a higher value. I do have some
>      input to that discussion, but let's postpone it until after DPDK
>      26.07 has been released.
>
>    We should increase this value to avoid performance degradation, as
>    users may not realize they need to change it.  We do not have do it
>    right away for 26.07 release but we should it right after the
>    release.
>

Out of interest, does adjusting the 50% fill/flush threshold to be a 75%
one, i.e. fill to 75% rather than 50%, flush to 25% rather than 50%, help at
all? Draft patch below, can you test it quickly, perhaps?

/Bruce

diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h index
50d958c7c6..2526a23903 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -1417,6 +1417,7 @@ rte_mempool_do_generic_put(struct rte_mempool *mp,
void * const *obj_table,
                           unsigned int n, struct rte_mempool_cache *cache)
{
        void **cache_objs;
+       uint32_t quarter, flush;

        /* No cache provided? */
        if (unlikely(cache == NULL))
@@ -1426,30 +1427,39 @@ rte_mempool_do_generic_put(struct rte_mempool *mp,
void * const *obj_table,
        RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
        RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);

+       /* A quarter (25%) of the cache size, computed with a shift to avoid
+        * a divide. Draining the cache down to this level on overflow
leaves
+        * room for a burst of up to (size - quarter), i.e. three quarters
+        * (75%), of the cache size.
+        */
+       quarter = cache->size >> 2;
+
        __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
-       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+       __rte_assume(quarter <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
        __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
        __rte_assume(cache->len <= cache->size);
        if (likely(cache->len + n <= cache->size)) {
                /* Sufficient room in the cache for the objects. */
                cache_objs = &cache->objs[cache->len];
                cache->len += n;
-       } else if (n <= cache->size / 2) {
+       } else if (n <= cache->size - quarter) {
                /*
                 * The number of objects is within the cache bounce buffer
limit,
                 * but - as detected by the comparison above - the cache has
                 * insufficient room for them.
                 * Flush the cache to the backend to make room for the
objects;
-                * flush (size / 2) objects from the bottom of the cache,
where
-                * objects are less hot, and move down the remaining
objects, which
-                * are more hot, from the upper half of the cache.
+                * flush objects from the bottom of the cache, where objects
are
+                * less hot, draining it down to a quarter (25%) of its
size, and
+                * move down the remaining quarter of objects, which are
more hot,
+                * from the upper part of the cache.
                 */
-               __rte_assume(cache->len > cache->size / 2);
-               rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0],
cache->size / 2);
-               rte_memcpy(&cache->objs[0], &cache->objs[cache->size / 2],
-                               sizeof(void *) * (cache->len - cache->size /
2));
-               cache_objs = &cache->objs[cache->len - cache->size / 2];
-               cache->len = cache->len - cache->size / 2 + n;
+               __rte_assume(cache->len > quarter);
+               flush = cache->len - quarter;
+               rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0], flush);
+               rte_memcpy(&cache->objs[0], &cache->objs[flush],
+                               sizeof(void *) * quarter);
+               cache_objs = &cache->objs[quarter];
+               cache->len = quarter + n;
        } else {
                /* The request itself is too big for the cache. */
                goto driver_enqueue_stats_incremented; @@ -1557,6 +1567,7 @@
rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
        int ret;
        unsigned int remaining;
        uint32_t index, len;
+       uint32_t quarter, fill;
        void **cache_objs;

        /* No cache provided? */
@@ -1592,13 +1603,23 @@ rte_mempool_do_generic_get(struct rte_mempool *mp,
void **obj_table,
        for (index = 0; index < len; index++)
                *obj_table++ = *--cache_objs;

+       /* A quarter (25%) of the cache size, computed with a shift to avoid
+        * a divide; 'fill' is the complementary three quarters (75%), which
+        * is the amount fetched from the backend to fill the cache up to
75%,
+        * and also the burst limit for this bounce buffer (since the cache
+        * was just fully drained above, up to 'fill' objects can be filled
+        * and handed back to the caller in one go).
+        */
+       quarter = cache->size >> 2;
+       fill = cache->size - quarter;
+
        /* Dequeue below would exceed the cache bounce buffer limit? */
-       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
-       if (unlikely(remaining > cache->size / 2))
+       __rte_assume(fill <= RTE_MEMPOOL_CACHE_MAX_SIZE);
+       if (unlikely(remaining > fill))
                goto driver_dequeue;

-       /* Fill the cache from the backend; fetch (size / 2) objects. */
-       ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache->size /
2);
+       /* Fill the cache from the backend, up to 75% of its size. */
+       ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, fill);
        if (unlikely(ret < 0)) {
                /*
                 * We are buffer constrained, and not able to fetch all
that.
@@ -1612,11 +1633,10 @@ rte_mempool_do_generic_get(struct rte_mempool *mp,
void **obj_table,
        RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
        RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);

-       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
-       __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
-       __rte_assume(remaining <= cache->size / 2);
-       cache_objs = &cache->objs[cache->size / 2];
-       cache->len = cache->size / 2 - remaining;
+       __rte_assume(fill <= RTE_MEMPOOL_CACHE_MAX_SIZE);
+       __rte_assume(remaining <= fill);
+       cache_objs = &cache->objs[fill];
+       cache->len = fill - remaining;
        for (index = 0; index < remaining; index++)
                *obj_table++ = *--cache_objs;

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5493 bytes --]

RE: mempool cache change

[Morten Brørup]

> From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
> Sent: Friday, 17 July 2026 19.10
> 
> Hi Bruce,
> 
> The below patch works fine for us. We tested all the different packet
> sizes.
> Thanks for the patch. Do you want to push this patch since it is not
> changing the ABI/API?

Too late in the release process.

Let's postpone the discussion for DPDK 26.11, where API/ABI breakage is allowed.

I'm not strongly opposed to Bruce's algorithm, targeting a fill level of 25 % from the edges and flushing/refilling up to 75 % of the cache when necessary. It does have its advantages for some mempool access patterns (which are not exotic).
I just prefer the current algorithm, targeting a fill level of 50 % and only flushing/refilling up to 50 % of the cache when necessary. It performs better at random get/put access patterns, and the backend transactions are smaller.

For DPDK 26.11, where we can break the API/ABI, we can simply double RTE_MEMPOOL_CACHE_MAX_SIZE to 1024, to compensate for reducing the effective cache size from 150 % to 100 %.
The mempool cache objs array will no longer be [RTE_MEMPOOL_CACHE_MAX_SIZE * 2], but only [RTE_MEMPOOL_CACHE_MAX_SIZE], so doubling RTE_MEMPOOL_CACHE_MAX_SIZE will not increase the memory footprint, but allow using a cache size up to 1024.

Please also note that the current implementation is carefully designed to keep the transfers to/from the mempool backend CPU cache aligned (assuming cache->size is 2^N and large enough).
Refer to the parameters passed to rte_mempool_ops_enqueue/dequeue_bulk().
E.g. with mempool cache size 256, backend transfers are 128 objects, 16 full cache lines.
Using CPU cache aligned transfers has a few advantages:
- There are no cache line ownership issues across different CPU cores repeatedly accessing the backend.
- The mempool backend drivers can be performance optimized for transferring full CPU cache lines. (Both source and destination addresses, and number of objects copied, are CPU cache aligned. Assuming all transfers go via the mempool cache.)

These details should be fine tuned in the implementation, if we do proceed with Bruce's algorithm.

-Morten

> 
> Rgds,
> Kishore
> 
> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Thursday, July 16, 2026 5:57 AM
> To: Kishore Padmanabha <kishore.padmanabha@broadcom.com>
> 
> On Wed, Jul 15, 2026 at 11:12:59AM -0400, Kishore Padmanabha wrote:
> >    On Wed, Jul 15, 2026 at 5:02 AM Morten Brørup
> >    <[1]mb@smartsharesystems.com> wrote:
> >
> >      > From: fengchengwen [mailto:[2]fengchengwen@huawei.com]
> >      > Sent: Wednesday, 15 July 2026 10.12
> >      >
> >      > On 7/15/2026 4:08 AM, Morten Brørup wrote:
> >      > > Hi Kishore,
> >      > >
> >      > > For your testing purposes, please follow the guidance
> provided
> >      to
> >      > Wisam Jaddo:
> >      > >
> >      >
> >
> [3]https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35F65964
> >      @smart
> >      > [4]server.smartshare.dk/
> >      >
> >      > We need to recompile in this case. We should try to avoid
> >      > recompilation.
> >      > I think it is necessary to adjust RTE_MEMPOOL_CACHE_MAX_SIZE
> to
> >      768 as
> >      > a default configuration.
> >      Changing RTE_MEMPOOL_CACHE_MAX_SIZE breaks both the API and the
> ABI;
> >      so it has to be done when building locally, where API/ABI
> breakage
> >      is acceptable.
> >      For DPDK 26.11, where API/ABI breakage is acceptable, we can
> discuss
> >      increasing the default from 512 to a higher value. I do have
> some
> >      input to that discussion, but let's postpone it until after DPDK
> >      26.07 has been released.
> >
> >    We should increase this value to avoid performance degradation, as
> >    users may not realize they need to change it.  We do not have do
> it
> >    right away for 26.07 release but we should it right after the
> >    release.
> >
> 
> Out of interest, does adjusting the 50% fill/flush threshold to be a
> 75%
> one, i.e. fill to 75% rather than 50%, flush to 25% rather than 50%,
> help at
> all? Draft patch below, can you test it quickly, perhaps?
> 
> /Bruce
> 
> diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
> index
> 50d958c7c6..2526a23903 100644
> --- a/lib/mempool/rte_mempool.h
> +++ b/lib/mempool/rte_mempool.h
> @@ -1417,6 +1417,7 @@ rte_mempool_do_generic_put(struct rte_mempool
> *mp,
> void * const *obj_table,
>                            unsigned int n, struct rte_mempool_cache
> *cache)
> {
>         void **cache_objs;
> +       uint32_t quarter, flush;
> 
>         /* No cache provided? */
>         if (unlikely(cache == NULL))
> @@ -1426,30 +1427,39 @@ rte_mempool_do_generic_put(struct rte_mempool
> *mp,
> void * const *obj_table,
>         RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
>         RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
> 
> +       /* A quarter (25%) of the cache size, computed with a shift to
> avoid
> +        * a divide. Draining the cache down to this level on overflow
> leaves
> +        * room for a burst of up to (size - quarter), i.e. three
> quarters
> +        * (75%), of the cache size.
> +        */
> +       quarter = cache->size >> 2;
> +
>         __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> -       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE /
> 2);
> +       __rte_assume(quarter <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
>         __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
>         __rte_assume(cache->len <= cache->size);
>         if (likely(cache->len + n <= cache->size)) {
>                 /* Sufficient room in the cache for the objects. */
>                 cache_objs = &cache->objs[cache->len];
>                 cache->len += n;
> -       } else if (n <= cache->size / 2) {
> +       } else if (n <= cache->size - quarter) {
>                 /*
>                  * The number of objects is within the cache bounce
> buffer
> limit,
>                  * but - as detected by the comparison above - the
> cache has
>                  * insufficient room for them.
>                  * Flush the cache to the backend to make room for the
> objects;
> -                * flush (size / 2) objects from the bottom of the
> cache,
> where
> -                * objects are less hot, and move down the remaining
> objects, which
> -                * are more hot, from the upper half of the cache.
> +                * flush objects from the bottom of the cache, where
> objects
> are
> +                * less hot, draining it down to a quarter (25%) of its
> size, and
> +                * move down the remaining quarter of objects, which
> are
> more hot,
> +                * from the upper part of the cache.
>                  */
> -               __rte_assume(cache->len > cache->size / 2);
> -               rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0],
> cache->size / 2);
> -               rte_memcpy(&cache->objs[0], &cache->objs[cache->size /
> 2],
> -                               sizeof(void *) * (cache->len - cache-
> >size /
> 2));
> -               cache_objs = &cache->objs[cache->len - cache->size /
> 2];
> -               cache->len = cache->len - cache->size / 2 + n;
> +               __rte_assume(cache->len > quarter);
> +               flush = cache->len - quarter;
> +               rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0],
> flush);
> +               rte_memcpy(&cache->objs[0], &cache->objs[flush],
> +                               sizeof(void *) * quarter);
> +               cache_objs = &cache->objs[quarter];
> +               cache->len = quarter + n;
>         } else {
>                 /* The request itself is too big for the cache. */
>                 goto driver_enqueue_stats_incremented; @@ -1557,6
> +1567,7 @@
> rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
>         int ret;
>         unsigned int remaining;
>         uint32_t index, len;
> +       uint32_t quarter, fill;
>         void **cache_objs;
> 
>         /* No cache provided? */
> @@ -1592,13 +1603,23 @@ rte_mempool_do_generic_get(struct rte_mempool
> *mp,
> void **obj_table,
>         for (index = 0; index < len; index++)
>                 *obj_table++ = *--cache_objs;
> 
> +       /* A quarter (25%) of the cache size, computed with a shift to
> avoid
> +        * a divide; 'fill' is the complementary three quarters (75%),
> which
> +        * is the amount fetched from the backend to fill the cache up
> to
> 75%,
> +        * and also the burst limit for this bounce buffer (since the
> cache
> +        * was just fully drained above, up to 'fill' objects can be
> filled
> +        * and handed back to the caller in one go).
> +        */
> +       quarter = cache->size >> 2;
> +       fill = cache->size - quarter;
> +
>         /* Dequeue below would exceed the cache bounce buffer limit? */
> -       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE /
> 2);
> -       if (unlikely(remaining > cache->size / 2))
> +       __rte_assume(fill <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> +       if (unlikely(remaining > fill))
>                 goto driver_dequeue;
> 
> -       /* Fill the cache from the backend; fetch (size / 2) objects.
> */
> -       ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache->size
> /
> 2);
> +       /* Fill the cache from the backend, up to 75% of its size. */
> +       ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, fill);
>         if (unlikely(ret < 0)) {
>                 /*
>                  * We are buffer constrained, and not able to fetch all
> that.
> @@ -1612,11 +1633,10 @@ rte_mempool_do_generic_get(struct rte_mempool
> *mp,
> void **obj_table,
>         RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
>         RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
> 
> -       __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE /
> 2);
> -       __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
> -       __rte_assume(remaining <= cache->size / 2);
> -       cache_objs = &cache->objs[cache->size / 2];
> -       cache->len = cache->size / 2 - remaining;
> +       __rte_assume(fill <= RTE_MEMPOOL_CACHE_MAX_SIZE);
> +       __rte_assume(remaining <= fill);
> +       cache_objs = &cache->objs[fill];
> +       cache->len = fill - remaining;
>         for (index = 0; index < remaining; index++)
>                 *obj_table++ = *--cache_objs;

Re: mempool cache change

[Bruce Richardson]

On Sat, Jul 18, 2026 at 04:07:05PM +0200, Morten Brørup wrote:
> > From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
> > Sent: Friday, 17 July 2026 19.10
> > 
> > Hi Bruce,
> > 
> > The below patch works fine for us. We tested all the different packet
> > sizes.
> > Thanks for the patch. Do you want to push this patch since it is not
> > changing the ABI/API?
> 
> Too late in the release process.
> 
> Let's postpone the discussion for DPDK 26.11, where API/ABI breakage is allowed.

Agreed. Let's not unnecessarily rush this.

> 
> I'm not strongly opposed to Bruce's algorithm, targeting a fill level of 25 % from the edges and flushing/refilling up to 75 % of the cache when necessary. It does have its advantages for some mempool access patterns (which are not exotic).
> I just prefer the current algorithm, targeting a fill level of 50 % and only flushing/refilling up to 50 % of the cache when necessary. It performs better at random get/put access patterns, and the backend transactions are smaller.
> 
> For DPDK 26.11, where we can break the API/ABI, we can simply double RTE_MEMPOOL_CACHE_MAX_SIZE to 1024, to compensate for reducing the effective cache size from 150 % to 100 %.
> The mempool cache objs array will no longer be [RTE_MEMPOOL_CACHE_MAX_SIZE * 2], but only [RTE_MEMPOOL_CACHE_MAX_SIZE], so doubling RTE_MEMPOOL_CACHE_MAX_SIZE will not increase the memory footprint, but allow using a cache size up to 1024.

My concern with this approach is that it won't automatically fix the
problem if we have users who experience a performance regression due to the
mempool changes. While testpmd allows the mbcache size to be provided via
parameter, end applications are likely to have it hardcoded. That means
that if an app does experience a regression, the author/user has to be
either aware of the mempool changes, or has to debug it down to the mempool
and then know to increase the mempool cache size in the app.

It's not an insurmountable problem, but one that needs to be very clearly
called out in our documentation, what the change is, how it may affect
things and how to fix it.

On the other hand, in realworld, i.e. not just testpmd/l3fwd cases doing
little packet processesing, I would be fairly hopeful that regressions are
going to be few and very small.

> 
> Please also note that the current implementation is carefully designed to keep the transfers to/from the mempool backend CPU cache aligned (assuming cache->size is 2^N and large enough).
> Refer to the parameters passed to rte_mempool_ops_enqueue/dequeue_bulk().
> E.g. with mempool cache size 256, backend transfers are 128 objects, 16 full cache lines.
> Using CPU cache aligned transfers has a few advantages:
> - There are no cache line ownership issues across different CPU cores repeatedly accessing the backend.
> - The mempool backend drivers can be performance optimized for transferring full CPU cache lines. (Both source and destination addresses, and number of objects copied, are CPU cache aligned. Assuming all transfers go via the mempool cache.)
> 
> These details should be fine tuned in the implementation, if we do proceed with Bruce's algorithm.
> 
Yep, good points.

/Bruce

RE: mempool cache change

[Morten Brørup]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 20 July 2026 10.13
> 
> On Sat, Jul 18, 2026 at 04:07:05PM +0200, Morten Brørup wrote:
> > > From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
> > > Sent: Friday, 17 July 2026 19.10
> > >
> > > Hi Bruce,
> > >
> > > The below patch works fine for us. We tested all the different
> packet
> > > sizes.
> > > Thanks for the patch. Do you want to push this patch since it is
> not
> > > changing the ABI/API?
> >
> > Too late in the release process.
> >
> > Let's postpone the discussion for DPDK 26.11, where API/ABI breakage
> is allowed.
> 
> Agreed. Let's not unnecessarily rush this.
> 
> >
> > I'm not strongly opposed to Bruce's algorithm, targeting a fill level
> of 25 % from the edges and flushing/refilling up to 75 % of the cache
> when necessary. It does have its advantages for some mempool access
> patterns (which are not exotic).
> > I just prefer the current algorithm, targeting a fill level of 50 %
> and only flushing/refilling up to 50 % of the cache when necessary. It
> performs better at random get/put access patterns, and the backend
> transactions are smaller.
> >
> > For DPDK 26.11, where we can break the API/ABI, we can simply double
> RTE_MEMPOOL_CACHE_MAX_SIZE to 1024, to compensate for reducing the
> effective cache size from 150 % to 100 %.
> > The mempool cache objs array will no longer be
> [RTE_MEMPOOL_CACHE_MAX_SIZE * 2], but only
> [RTE_MEMPOOL_CACHE_MAX_SIZE], so doubling RTE_MEMPOOL_CACHE_MAX_SIZE
> will not increase the memory footprint, but allow using a cache size up
> to 1024.
> 
> My concern with this approach is that it won't automatically fix the
> problem if we have users who experience a performance regression due to
> the
> mempool changes. While testpmd allows the mbcache size to be provided
> via
> parameter, end applications are likely to have it hardcoded. That means
> that if an app does experience a regression, the author/user has to be
> either aware of the mempool changes, or has to debug it down to the
> mempool
> and then know to increase the mempool cache size in the app.
> 
> It's not an insurmountable problem, but one that needs to be very
> clearly
> called out in our documentation, what the change is, how it may affect
> things and how to fix it.

The release notes [1] do mention that the mempool algorithm has changed, and how some applications should adapt to it.
They also do mention that the effective mempool cache size has changed, but not that applications can restore the previously 50 % larger effective cache size by simply requesting a 50 % larger cache.

Maybe we should move the notes about the mempool changes from the New Features section to the API Changes section, and mention how to increase the cache size to regain the reduction in effective size?

[1]: https://github.com/DPDK/dpdk/blob/v26.07-rc4/doc/guides/rel_notes/release_26_07.rst?plain=1#L80

> 
> On the other hand, in realworld, i.e. not just testpmd/l3fwd cases
> doing
> little packet processesing, I would be fairly hopeful that regressions
> are
> going to be few and very small.

Yes.
And some real world applications (most run-to-completion applications?) will get fewer mempool cache misses with the new algorithm.
Best case a performance improvement exceeding the performance cost of the reduced effective cache size.

> 
> >
> > Please also note that the current implementation is carefully
> designed to keep the transfers to/from the mempool backend CPU cache
> aligned (assuming cache->size is 2^N and large enough).
> > Refer to the parameters passed to
> rte_mempool_ops_enqueue/dequeue_bulk().
> > E.g. with mempool cache size 256, backend transfers are 128 objects,
> 16 full cache lines.
> > Using CPU cache aligned transfers has a few advantages:
> > - There are no cache line ownership issues across different CPU cores
> repeatedly accessing the backend.
> > - The mempool backend drivers can be performance optimized for
> transferring full CPU cache lines. (Both source and destination
> addresses, and number of objects copied, are CPU cache aligned.
> Assuming all transfers go via the mempool cache.)
> >
> > These details should be fine tuned in the implementation, if we do
> proceed with Bruce's algorithm.
> >
> Yep, good points.
> 
> /Bruce

PS:
I have some API/ABI breaking mempool optimizations in the pipeline for DPDK 26.11.
But let's try to concentrate on DPDK 26.07 for now. ;-)

Re: mempool cache change

[Kishore Padmanabha]

[-- Attachment #1.1: Type: text/plain, Size: 4943 bytes --]

On Mon, Jul 20, 2026 at 6:54 AM Morten Brørup <mb@smartsharesystems.com>
wrote:

> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Monday, 20 July 2026 10.13
> >
> > On Sat, Jul 18, 2026 at 04:07:05PM +0200, Morten Brørup wrote:
> > > > From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
> > > > Sent: Friday, 17 July 2026 19.10
> > > >
> > > > Hi Bruce,
> > > >
> > > > The below patch works fine for us. We tested all the different
> > packet
> > > > sizes.
> > > > Thanks for the patch. Do you want to push this patch since it is
> > not
> > > > changing the ABI/API?
> > >
> > > Too late in the release process.
> > >
> > > Let's postpone the discussion for DPDK 26.11, where API/ABI breakage
> > is allowed.
> >
> > Agreed. Let's not unnecessarily rush this.
> >
> > >
> > > I'm not strongly opposed to Bruce's algorithm, targeting a fill level
> > of 25 % from the edges and flushing/refilling up to 75 % of the cache
> > when necessary. It does have its advantages for some mempool access
> > patterns (which are not exotic).
> > > I just prefer the current algorithm, targeting a fill level of 50 %
> > and only flushing/refilling up to 50 % of the cache when necessary. It
> > performs better at random get/put access patterns, and the backend
> > transactions are smaller.
> > >
> > > For DPDK 26.11, where we can break the API/ABI, we can simply double
> > RTE_MEMPOOL_CACHE_MAX_SIZE to 1024, to compensate for reducing the
> > effective cache size from 150 % to 100 %.
> > > The mempool cache objs array will no longer be
> > [RTE_MEMPOOL_CACHE_MAX_SIZE * 2], but only
> > [RTE_MEMPOOL_CACHE_MAX_SIZE], so doubling RTE_MEMPOOL_CACHE_MAX_SIZE
> > will not increase the memory footprint, but allow using a cache size up
> > to 1024.
> >
> > My concern with this approach is that it won't automatically fix the
> > problem if we have users who experience a performance regression due to
> > the
> > mempool changes. While testpmd allows the mbcache size to be provided
> > via
> > parameter, end applications are likely to have it hardcoded. That means
> > that if an app does experience a regression, the author/user has to be
> > either aware of the mempool changes, or has to debug it down to the
> > mempool
> > and then know to increase the mempool cache size in the app.
> >
> > It's not an insurmountable problem, but one that needs to be very
> > clearly
> > called out in our documentation, what the change is, how it may affect
> > things and how to fix it.
>
> The release notes [1] do mention that the mempool algorithm has changed,
> and how some applications should adapt to it.
> They also do mention that the effective mempool cache size has changed,
> but not that applications can restore the previously 50 % larger effective
> cache size by simply requesting a 50 % larger cache.
>
> Maybe we should move the notes about the mempool changes from the New
> Features section to the API Changes section, and mention how to increase
> the cache size to regain the reduction in effective size?
>
> [1]:
> https://github.com/DPDK/dpdk/blob/v26.07-rc4/doc/guides/rel_notes/release_26_07.rst?plain=1#L80
>
> >
> > On the other hand, in realworld, i.e. not just testpmd/l3fwd cases
> > doing
> > little packet processesing, I would be fairly hopeful that regressions
> > are
> > going to be few and very small.
>
> Yes.
> And some real world applications (most run-to-completion applications?)
> will get fewer mempool cache misses with the new algorithm.
> Best case a performance improvement exceeding the performance cost of the
> reduced effective cache size.
>
> >
> > >
> > > Please also note that the current implementation is carefully
> > designed to keep the transfers to/from the mempool backend CPU cache
> > aligned (assuming cache->size is 2^N and large enough).
> > > Refer to the parameters passed to
> > rte_mempool_ops_enqueue/dequeue_bulk().
> > > E.g. with mempool cache size 256, backend transfers are 128 objects,
> > 16 full cache lines.
> > > Using CPU cache aligned transfers has a few advantages:
> > > - There are no cache line ownership issues across different CPU cores
> > repeatedly accessing the backend.
> > > - The mempool backend drivers can be performance optimized for
> > transferring full CPU cache lines. (Both source and destination
> > addresses, and number of objects copied, are CPU cache aligned.
> > Assuming all transfers go via the mempool cache.)
> > >
> > > These details should be fine tuned in the implementation, if we do
> > proceed with Bruce's algorithm.
> > >
> > Yep, good points.
> >
> > /Bruce
>
> Sounds good. Documenting the limitation helps.


> PS:
> I have some API/ABI breaking mempool optimizations in the pipeline for
> DPDK 26.11.
> But let's try to concentrate on DPDK 26.07 for now. ;-)
>
>

[-- Attachment #1.2: Type: text/html, Size: 6300 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5493 bytes --]

Re: mempool cache change

[Stephen Hemminger]

[-- Attachment #1: Type: text/plain, Size: 3822 bytes --]

I can't help thinking that there needs to be a more dynamic heuristic.
Keep existing upper limit but only cache as much as gets used.

There are two main usage models.
One is fast direct recycling which run to completion generates. The other
is when pipeline is used and arrival and departure is on different cores.

On Mon, Jul 20, 2026, 10:13 Bruce Richardson <bruce.richardson@intel.com>
wrote:

> On Sat, Jul 18, 2026 at 04:07:05PM +0200, Morten Brørup wrote:
> > > From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
> > > Sent: Friday, 17 July 2026 19.10
> > >
> > > Hi Bruce,
> > >
> > > The below patch works fine for us. We tested all the different packet
> > > sizes.
> > > Thanks for the patch. Do you want to push this patch since it is not
> > > changing the ABI/API?
> >
> > Too late in the release process.
> >
> > Let's postpone the discussion for DPDK 26.11, where API/ABI breakage is
> allowed.
>
> Agreed. Let's not unnecessarily rush this.
>
> >
> > I'm not strongly opposed to Bruce's algorithm, targeting a fill level of
> 25 % from the edges and flushing/refilling up to 75 % of the cache when
> necessary. It does have its advantages for some mempool access patterns
> (which are not exotic).
> > I just prefer the current algorithm, targeting a fill level of 50 % and
> only flushing/refilling up to 50 % of the cache when necessary. It performs
> better at random get/put access patterns, and the backend transactions are
> smaller.
> >
> > For DPDK 26.11, where we can break the API/ABI, we can simply double
> RTE_MEMPOOL_CACHE_MAX_SIZE to 1024, to compensate for reducing the
> effective cache size from 150 % to 100 %.
> > The mempool cache objs array will no longer be
> [RTE_MEMPOOL_CACHE_MAX_SIZE * 2], but only [RTE_MEMPOOL_CACHE_MAX_SIZE], so
> doubling RTE_MEMPOOL_CACHE_MAX_SIZE will not increase the memory footprint,
> but allow using a cache size up to 1024.
>
> My concern with this approach is that it won't automatically fix the
> problem if we have users who experience a performance regression due to the
> mempool changes. While testpmd allows the mbcache size to be provided via
> parameter, end applications are likely to have it hardcoded. That means
> that if an app does experience a regression, the author/user has to be
> either aware of the mempool changes, or has to debug it down to the mempool
> and then know to increase the mempool cache size in the app.
>
> It's not an insurmountable problem, but one that needs to be very clearly
> called out in our documentation, what the change is, how it may affect
> things and how to fix it.
>
> On the other hand, in realworld, i.e. not just testpmd/l3fwd cases doing
> little packet processesing, I would be fairly hopeful that regressions are
> going to be few and very small.
>
> >
> > Please also note that the current implementation is carefully designed
> to keep the transfers to/from the mempool backend CPU cache aligned
> (assuming cache->size is 2^N and large enough).
> > Refer to the parameters passed to rte_mempool_ops_enqueue/dequeue_bulk().
> > E.g. with mempool cache size 256, backend transfers are 128 objects, 16
> full cache lines.
> > Using CPU cache aligned transfers has a few advantages:
> > - There are no cache line ownership issues across different CPU cores
> repeatedly accessing the backend.
> > - The mempool backend drivers can be performance optimized for
> transferring full CPU cache lines. (Both source and destination addresses,
> and number of objects copied, are CPU cache aligned. Assuming all transfers
> go via the mempool cache.)
> >
> > These details should be fine tuned in the implementation, if we do
> proceed with Bruce's algorithm.
> >
> Yep, good points.
>
> /Bruce
>

[-- Attachment #2: Type: text/html, Size: 4484 bytes --]

RE: mempool cache change

[Morten Brørup]

[-- Attachment #1: Type: text/plain, Size: 5177 bytes --]

The mempool cache size is configurable.

The application developer should configure it to strike a balance between the two performance parameters:

- Throughput: Large mempool cache reduces the number of backend transfers.

- Latency: Large mempool cache causes latency spikes (CPU cache misses) when transferring objects from/to cold backend memory.

 

However, none of the DPDK tests consider latency or jitter, only throughput, so larger cache sizes yield better performance.

 

Another detail (for completeness only): Compared to the size of the mempool itself, the mempool cache must also not be too big, or the entire mempool may be depleted by a few per-lcore mempool caches.

 

PS:

Loading 64 objects (a block of 512 byte) from L2 CPU Cache takes ~2.4 ns, while loading it from System RAM takes ~76 ns.

Loading 256 objects (a block of 2048 byte) from L2 CPU Cache takes ~9 ns, while loading it from System RAM takes ~90 ns.

Loading 512 objects (a block of 4096 byte) from L2 CPU Cache takes ~18 ns, while loading it from System RAM takes ~121 ns.

(Source: Google AI)

 

Venlig hilsen / Kind regards,

-Morten Brørup

 

From: Stephen Hemminger [mailto:stephen@networkplumber.org] 
Sent: Tuesday, 21 July 2026 08.35
To: Bruce Richardson
Cc: Morten Brørup; Kishore Padmanabha; fengchengwen; Thomas Monjalon; dev; Wisam Jaddo; Andrew Rybchenko
Subject: Re: mempool cache change

 

I can't help thinking that there needs to be a more dynamic heuristic.

Keep existing upper limit but only cache as much as gets used.

 

There are two main usage models.

One is fast direct recycling which run to completion generates. The other is when pipeline is used and arrival and departure is on different cores.

 

On Mon, Jul 20, 2026, 10:13 Bruce Richardson <bruce.richardson@intel.com> wrote:

	On Sat, Jul 18, 2026 at 04:07:05PM +0200, Morten Brørup wrote:
	> > From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
	> > Sent: Friday, 17 July 2026 19.10
	> > 
	> > Hi Bruce,
	> > 
	> > The below patch works fine for us. We tested all the different packet
	> > sizes.
	> > Thanks for the patch. Do you want to push this patch since it is not
	> > changing the ABI/API?
	> 
	> Too late in the release process.
	> 
	> Let's postpone the discussion for DPDK 26.11, where API/ABI breakage is allowed.
	
	Agreed. Let's not unnecessarily rush this.
	
	> 
	> I'm not strongly opposed to Bruce's algorithm, targeting a fill level of 25 % from the edges and flushing/refilling up to 75 % of the cache when necessary. It does have its advantages for some mempool access patterns (which are not exotic).
	> I just prefer the current algorithm, targeting a fill level of 50 % and only flushing/refilling up to 50 % of the cache when necessary. It performs better at random get/put access patterns, and the backend transactions are smaller.
	> 
	> For DPDK 26.11, where we can break the API/ABI, we can simply double RTE_MEMPOOL_CACHE_MAX_SIZE to 1024, to compensate for reducing the effective cache size from 150 % to 100 %.
	> The mempool cache objs array will no longer be [RTE_MEMPOOL_CACHE_MAX_SIZE * 2], but only [RTE_MEMPOOL_CACHE_MAX_SIZE], so doubling RTE_MEMPOOL_CACHE_MAX_SIZE will not increase the memory footprint, but allow using a cache size up to 1024.
	
	My concern with this approach is that it won't automatically fix the
	problem if we have users who experience a performance regression due to the
	mempool changes. While testpmd allows the mbcache size to be provided via
	parameter, end applications are likely to have it hardcoded. That means
	that if an app does experience a regression, the author/user has to be
	either aware of the mempool changes, or has to debug it down to the mempool
	and then know to increase the mempool cache size in the app.
	
	It's not an insurmountable problem, but one that needs to be very clearly
	called out in our documentation, what the change is, how it may affect
	things and how to fix it.
	
	On the other hand, in realworld, i.e. not just testpmd/l3fwd cases doing
	little packet processesing, I would be fairly hopeful that regressions are
	going to be few and very small.
	
	> 
	> Please also note that the current implementation is carefully designed to keep the transfers to/from the mempool backend CPU cache aligned (assuming cache->size is 2^N and large enough).
	> Refer to the parameters passed to rte_mempool_ops_enqueue/dequeue_bulk().
	> E.g. with mempool cache size 256, backend transfers are 128 objects, 16 full cache lines.
	> Using CPU cache aligned transfers has a few advantages:
	> - There are no cache line ownership issues across different CPU cores repeatedly accessing the backend.
	> - The mempool backend drivers can be performance optimized for transferring full CPU cache lines. (Both source and destination addresses, and number of objects copied, are CPU cache aligned. Assuming all transfers go via the mempool cache.)
	> 
	> These details should be fine tuned in the implementation, if we do proceed with Bruce's algorithm.
	> 
	Yep, good points.
	
	/Bruce


[-- Attachment #2: Type: text/html, Size: 10142 bytes --]

Re: mempool cache change

[Bruce Richardson]

On Tue, Jul 21, 2026 at 11:02:34AM +0200, Morten Brørup wrote:
>    The mempool cache size is configurable.
> 
>    The application developer should configure it to strike a balance
>    between the two performance parameters:
> 
>    - Throughput: Large mempool cache reduces the number of backend
>    transfers.
> 
>    - Latency: Large mempool cache causes latency spikes (CPU cache misses)
>    when transferring objects from/to cold backend memory.
> 
> 
Since 26.11 is an ABI break release, perhaps we can reconsider the option
to have the "keep" value configurable for the mempool caches, i.e. for
random alloc/free's set it to 50% like you have, or 25% as proposed by my
patch, while for pipeline apps it should be set to 0%. Rather than us
constantly arguing^H^H^H discussing what the value should be, we can make
it tunable for the developer?

As an alternative version of that proposal, we can simplify and just have a
couple of predefined simple profiles for 2 or 3 values, one for pipeline
apps, and one or two for run-to-completion, rather than exposing the whole
range of possibilities to the user.

/Bruce

RE: mempool cache change

[Morten Brørup]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Tuesday, 21 July 2026 11.12
> 
> On Tue, Jul 21, 2026 at 11:02:34AM +0200, Morten Brørup wrote:
> >    The mempool cache size is configurable.
> >
> >    The application developer should configure it to strike a balance
> >    between the two performance parameters:
> >
> >    - Throughput: Large mempool cache reduces the number of backend
> >    transfers.
> >
> >    - Latency: Large mempool cache causes latency spikes (CPU cache
> misses)
> >    when transferring objects from/to cold backend memory.
> >
> >
> Since 26.11 is an ABI break release, perhaps we can reconsider the
> option
> to have the "keep" value configurable for the mempool caches, i.e. for
> random alloc/free's set it to 50% like you have, or 25% as proposed by
> my
> patch, while for pipeline apps it should be set to 0%. Rather than us
> constantly arguing^H^H^H discussing what the value should be, we can
> make
> it tunable for the developer?
> 
> As an alternative version of that proposal, we can simplify and just
> have a
> couple of predefined simple profiles for 2 or 3 values, one for
> pipeline
> apps, and one or two for run-to-completion, rather than exposing the
> whole
> range of possibilities to the user.

At this time, I'm not planning major changes to the mempool cache algorithm.

I have sent a patch [DIV32] requiring the mempool cache size to be divisible by 32.
This opens a road towards operating on the mempool cache in chunks of 32 objects.

Bruce strongly hinted that the ring driver might not be optimal for mbuf pools,
so I played around with the lock-free stack mempool driver, and it is extremely slow.
To address this issue, I developed a faster stack-like "pile" [PILE-STACK] and a mempool driver [PILE-MEMPOOL] on top of it.

All this playing around and working with the details got me thinking. Which leads back to the subject of 32-object chunks...
The current mempool cache algorithm is targeting large flush/replenish operations.
If the mempool driver is slow (and its runtime scales with the number of objects), large operations will cause latency spikes.

A new algorithm could take a different approach to address latency spikes, targeting small driver operations instead of large driver operations:
When pulling objects from a cache with insufficient objects, only refill the necessary number of objects to satisfy the request, rounded up to 32-object chunks.
When pushing objects from a cache with insufficient space, only flush the necessary number of objects to satisfy the request, rounded up to 32-object chunks.

As a further optimization, instead of moving objects down in the stack when flushing from the bottom, we could use an integer offset to point at the bottom of the stack, and wrap around the index at size when accessing the array. E.g. for push:
for (i = 0; i < num; i++) cache->objs[(cache->bottom + cache->len + i) % size] = obj_table[i];

I don't have plans to implement such an algorithm (or the bottom offset) for DPDK 26.11.
Just sharing my thoughts.

[DIV32]: https://patchwork.dpdk.org/project/dpdk/patch/20260812120626.1772120-1-mb@smartsharesystems.com/
[PILE-STACK]: https://patchwork.dpdk.org/project/dpdk/patch/20260812134756.1829613-1-mb@smartsharesystems.com/
[PILE-MEMPOOL]: https://patchwork.dpdk.org/project/dpdk/patch/20260812141511.1854348-1-mb@smartsharesystems.com/

RE: mempool cache change

[Morten Brørup]

From: Stephen Hemminger [mailto:stephen@networkplumber.org] 
Sent: Tuesday, 21 July 2026 08.35

>I can't help thinking that there needs to be a more dynamic heuristic.
>Keep existing upper limit but only cache as much as gets used.
>
>There are two main usage models.
>One is fast direct recycling which run to completion generates. The other is when pipeline is used and arrival and departure is on different cores.

In the "different cores" usage model, only half the cache size is active on each thread.
In the "free/put only" thread, only the upper half is active (since we flush to size/2).
In the "alloc/get only" thread, only the lower half is active (since we replenish to size/2).

So, doubling the configured cache size will perform the same as an optimal algorithm.
But it will waste size/2 objects per "free/put only" thread as dead in the per-thread mempool cache's lower half.

Runtime heuristic would introduce a performance cost for checking usage model.