Re: Multi-core scalability problems

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Thu, 15 Oct 2020 14:04:51 +0200
Federico Parola <fede.parola@hotmail.it> wrote:

> On 14/10/20 16:26, Jesper Dangaard Brouer wrote:
> > On Wed, 14 Oct 2020 14:17:46 +0200
> > Federico Parola <fede.parola@hotmail.it> wrote:
> >   
> >> On 14/10/20 11:15, Jesper Dangaard Brouer wrote:  
> >>> On Wed, 14 Oct 2020 08:56:43 +0200
> >>> Federico Parola <fede.parola@hotmail.it> wrote:
> >>>
> >>> [...]  
> >>>>> Can you try to use this[2] tool:
> >>>>>     ethtool_stats.pl --dev enp101s0f0
> >>>>>
> >>>>> And notice if there are any strange counters.
> >>>>>
> >>>>>
> >>>>> [2]https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
[...]

> >> The only solution I've found so far is to reduce the size of the rx ring
> >> as I mentioned in my former post. However I still see a decrease in
> >> performance when exceeding 4 cores.  
> > 
> > What is happening when you are reducing the size of the rx ring is two
> > things. (1) i40e driver have reuse/recycle-pages trick that get less
> > efficient, but because you are dropping packets early you are not
> > affected. (2) the total size of L3 memory you need to touch is also
> > decreased.
> > 
> > I think you are hitting case (2).  The Intel CPU have a cool feature
> > called DDIO (Data-Direct IO) or DCA (Direct Cache Access), which can
> > deliver packet data into L3 cache memory (if NIC is directly PCIe
> > connected to CPU).  The CPU is in charge when this feature is enabled,
> > and it will try to avoid L3 trashing and disable it in certain cases.
> > When you reduce the size of the rx rings, then you are also needing
> > less L3 cache memory, to the CPU will allow this DDIO feature.
> > 
> > You can use the 'perf stat' tool to check if this is happening, by
> > monitoring L3 (and L2) cache usage.  
> 
> What events should I monitor? LLC-load-misses/LLC-loads?

Looking at my own results from xdp-paper[1], it looks like that it
results in real 'cache-misses' (perf stat -e cache-misses).

E.g I ran:
 sudo ~/perf stat -C3 -e cycles -e  instructions -e cache-references -e cache-misses -r 3 sleep 1

Notice how the 'insn per cycle' gets less efficient when we experience
these cache-misses.

Also how RX-size of queues affect XDP-redirect in [2].


[1] https://github.com/xdp-project/xdp-paper/blob/master/benchmarks/bench01_baseline.org
[2] https://github.com/xdp-project/xdp-paper/blob/master/benchmarks/bench05_xdp_redirect.org
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer