Re: [RFC] cache guard

[Thomas Monjalon <thomas@monjalon.net>]

27/08/2023 10:34, Morten Brørup:
> +CC Honnappa and Konstantin, Ring lib maintainers
> +CC Mattias, PRNG lib maintainer
> 
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Friday, 25 August 2023 11.24
> > 
> > On Fri, Aug 25, 2023 at 11:06:01AM +0200, Morten Brørup wrote:
> > > +CC mempool maintainers
> > >
> > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > Sent: Friday, 25 August 2023 10.23
> > > >
> > > > On Fri, Aug 25, 2023 at 08:45:12AM +0200, Morten Brørup wrote:
> > > > > Bruce,
> > > > >
> > > > > With this patch [1], it is noted that the ring producer and
> > consumer data
> > > > should not be on adjacent cache lines, for performance reasons.
> > > > >
> > > > > [1]:
> > > >
> > https://git.dpdk.org/dpdk/commit/lib/librte_ring/rte_ring.h?id=d9f0d3a1f
> > fd4b66
> > > > e75485cc8b63b9aedfbdfe8b0
> > > > >
> > > > > (It's obvious that they cannot share the same cache line, because
> > they are
> > > > accessed by two different threads.)
> > > > >
> > > > > Intuitively, I would think that having them on different cache
> > lines would
> > > > suffice. Why does having an empty cache line between them make a
> > difference?
> > > > >
> > > > > And does it need to be an empty cache line? Or does it suffice
> > having the
> > > > second structure start at two cache lines after the start of the
> > first
> > > > structure (e.g. if the size of the first structure is two cache
> > lines)?
> > > > >
> > > > > I'm asking because the same principle might apply to other code
> > too.
> > > > >
> > > > Hi Morten,
> > > >
> > > > this was something we discovered when working on the distributor
> > library.
> > > > If we have cachelines per core where there is heavy access, having
> > some
> > > > cachelines as a gap between the content cachelines can help
> > performance. We
> > > > believe this helps due to avoiding issues with the HW prefetchers
> > (e.g.
> > > > adjacent cacheline prefetcher) bringing in the second cacheline
> > > > speculatively when an operation is done on the first line.
> > >
> > > I guessed that it had something to do with speculative prefetching,
> > but wasn't sure. Good to get confirmation, and that it has a measureable
> > effect somewhere. Very interesting!
> > >
> > > NB: More comments in the ring lib about stuff like this would be nice.
> > >
> > > So, for the mempool lib, what do you think about applying the same
> > technique to the rte_mempool_debug_stats structure (which is an array
> > indexed per lcore)... Two adjacent lcores heavily accessing their local
> > mempool caches seems likely to me. But how heavy does the access need to
> > be for this technique to be relevant?
> > >
> > 
> > No idea how heavy the accesses need to be for this to have a noticable
> > effect. For things like debug stats, I wonder how worthwhile making such
> > a
> > change would be, but then again, any change would have very low impact
> > too
> > in that case.
> 
> I just tried adding padding to some of the hot structures in our own application, and observed a significant performance improvement for those.
> 
> So I think this technique should have higher visibility in DPDK by adding a new cache macro to rte_common.h:

+1 to make more visibility in doc and adding a macro, good idea!