RE: mempool cache change

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 20 July 2026 10.13
> 
> On Sat, Jul 18, 2026 at 04:07:05PM +0200, Morten Brørup wrote:
> > > From: Kishore Padmanabha [mailto:kishore.padmanabha@broadcom.com]
> > > Sent: Friday, 17 July 2026 19.10
> > >
> > > Hi Bruce,
> > >
> > > The below patch works fine for us. We tested all the different
> packet
> > > sizes.
> > > Thanks for the patch. Do you want to push this patch since it is
> not
> > > changing the ABI/API?
> >
> > Too late in the release process.
> >
> > Let's postpone the discussion for DPDK 26.11, where API/ABI breakage
> is allowed.
> 
> Agreed. Let's not unnecessarily rush this.
> 
> >
> > I'm not strongly opposed to Bruce's algorithm, targeting a fill level
> of 25 % from the edges and flushing/refilling up to 75 % of the cache
> when necessary. It does have its advantages for some mempool access
> patterns (which are not exotic).
> > I just prefer the current algorithm, targeting a fill level of 50 %
> and only flushing/refilling up to 50 % of the cache when necessary. It
> performs better at random get/put access patterns, and the backend
> transactions are smaller.
> >
> > For DPDK 26.11, where we can break the API/ABI, we can simply double
> RTE_MEMPOOL_CACHE_MAX_SIZE to 1024, to compensate for reducing the
> effective cache size from 150 % to 100 %.
> > The mempool cache objs array will no longer be
> [RTE_MEMPOOL_CACHE_MAX_SIZE * 2], but only
> [RTE_MEMPOOL_CACHE_MAX_SIZE], so doubling RTE_MEMPOOL_CACHE_MAX_SIZE
> will not increase the memory footprint, but allow using a cache size up
> to 1024.
> 
> My concern with this approach is that it won't automatically fix the
> problem if we have users who experience a performance regression due to
> the
> mempool changes. While testpmd allows the mbcache size to be provided
> via
> parameter, end applications are likely to have it hardcoded. That means
> that if an app does experience a regression, the author/user has to be
> either aware of the mempool changes, or has to debug it down to the
> mempool
> and then know to increase the mempool cache size in the app.
> 
> It's not an insurmountable problem, but one that needs to be very
> clearly
> called out in our documentation, what the change is, how it may affect
> things and how to fix it.

The release notes [1] do mention that the mempool algorithm has changed, and how some applications should adapt to it.
They also do mention that the effective mempool cache size has changed, but not that applications can restore the previously 50 % larger effective cache size by simply requesting a 50 % larger cache.

Maybe we should move the notes about the mempool changes from the New Features section to the API Changes section, and mention how to increase the cache size to regain the reduction in effective size?

[1]: https://github.com/DPDK/dpdk/blob/v26.07-rc4/doc/guides/rel_notes/release_26_07.rst?plain=1#L80

> 
> On the other hand, in realworld, i.e. not just testpmd/l3fwd cases
> doing
> little packet processesing, I would be fairly hopeful that regressions
> are
> going to be few and very small.

Yes.
And some real world applications (most run-to-completion applications?) will get fewer mempool cache misses with the new algorithm.
Best case a performance improvement exceeding the performance cost of the reduced effective cache size.

> 
> >
> > Please also note that the current implementation is carefully
> designed to keep the transfers to/from the mempool backend CPU cache
> aligned (assuming cache->size is 2^N and large enough).
> > Refer to the parameters passed to
> rte_mempool_ops_enqueue/dequeue_bulk().
> > E.g. with mempool cache size 256, backend transfers are 128 objects,
> 16 full cache lines.
> > Using CPU cache aligned transfers has a few advantages:
> > - There are no cache line ownership issues across different CPU cores
> repeatedly accessing the backend.
> > - The mempool backend drivers can be performance optimized for
> transferring full CPU cache lines. (Both source and destination
> addresses, and number of objects copied, are CPU cache aligned.
> Assuming all transfers go via the mempool cache.)
> >
> > These details should be fine tuned in the implementation, if we do
> proceed with Bruce's algorithm.
> >
> Yep, good points.
> 
> /Bruce

PS:
I have some API/ABI breaking mempool optimizations in the pipeline for DPDK 26.11.
But let's try to concentrate on DPDK 26.07 for now. ;-)