XDP redirect max rate on Intel XL710

XDP redirect max rate on Intel XL710

[Federico Parola]

Hello everybody,
when I redirect packets between two ports of an Intel XL710 40Gb card 
(e.g. with the xdp_redirect_map example fo the kernel) I can not achieve 
throughputs higher than ~31 Mpps.
This NIC is not able to reach the theoretical ~60 Mpps with small 
packets ([1] p. 23) but with DPDK I'm able to achieve ~40 Mpps with the 
testpmd application and 2 cores.
In XDP when dropping packets I achieve more or less the same throughput 
with 3 cores, but I'm not able to exceed 31 Mpps when forwarding, no 
matter how many cores I use.
I tried tuning the size of the RX/TX rings and the DDIO occupancy but 
with no success. I can scale with the number of cores more linearly but 
as soon as I reach the 31 threshold cores usage decreases and throughput 
remains the same.
I don't know if this is just related to my setup or my specific NIC (I 
exprimented on kernels 5.11 and 5.9), does anybody know what the reason 
could be?

Best regards,
Federico Parola

[1] 
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwicv__o6_jvAhWUwQIHHQtpBu8QFjABegQIBhAD&url=https%3A%2F%2Fwww.intel.com%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%2Fspecification-updates%2Fxl710-10-40-controller-spec-update.pdf&usg=AOvVaw27anCuRwRX25h8Dg3epgoy

Re: XDP redirect max rate on Intel XL710

[Jesper Dangaard Brouer]

On Mon, 12 Apr 2021 15:48:10 +0200
Federico Parola <federico.parola@polito.it> wrote:

> Hello everybody,
> when I redirect packets between two ports of an Intel XL710 40Gb card 
> (e.g. with the xdp_redirect_map example fo the kernel) I can not achieve 
> throughputs higher than ~31 Mpps.
> This NIC is not able to reach the theoretical ~60 Mpps with small

1/(84*8)*40*10^9 = 59,523,810 pps

> packets ([1] p. 23) but with DPDK I'm able to achieve ~40 Mpps with the 
> testpmd application and 2 cores.

Lets calculate the processing or arrival rate per packet (or between
packets arriving).

 31 Mpps = 32.25 nanosec  (1/31*(10^9/10^6))
 40 Mpps = 25    nanosec
 Difference = 7.26 nanosec.

Thus, you only have to optimize XDP redirect with 7 nanosec to catch up
with DPDK.  Assuming a 4GHz CPU (7*4) you get 28 cycles more with DPDK
that I'm sure your application really needs and use with carefully ;-)

> In XDP when dropping packets I achieve more or less the same throughput 
> with 3 cores, but I'm not able to exceed 31 Mpps when forwarding, no 
> matter how many cores I use.
> I tried tuning the size of the RX/TX rings and the DDIO occupancy but 
> with no success. I can scale with the number of cores more linearly but 
> as soon as I reach the 31 threshold cores usage decreases and throughput 
> remains the same.

If the CPU usage start do decrease (e.g. CPU gets idle cycles) then the
limitation is likely elsewhere, likely in the PCIe layer.  Or a limit
in the NIC hardware, but given DPDK is 7 nanosec faster then it might
not be the NIC hardware, but it could as DPDK could have disabled some
flow-dissector HW feature.  It can also be that XDP/kernel is doing more
PCIe operations than DPDK.

> I don't know if this is just related to my setup or my specific NIC (I 
> exprimented on kernels 5.11 and 5.9), does anybody know what the reason 
> could be?

Bjørn have recently optimized the kernel XDP redirect code path with
around 3-4 nanosec.  You can try with net-next kernel (Bjørn correct me
if this isn't true).

In your userspace AF_XDP application also make sure you don't use these
28 cycles / 7 nanosec for something that the DPDK testpmd doesn't.

 
> [1] 
> https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwicv__o6_jvAhWUwQIHHQtpBu8QFjABegQIBhAD&url=https%3A%2F%2Fwww.intel.com%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%2Fspecification-updates%2Fxl710-10-40-controller-spec-update.pdf&usg=AOvVaw27anCuRwRX25h8Dg3epgoy
> 



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer