Re: Kernel 4.19 network performance - forwarding/routing normal users traffic

[David Ahern <dsahern@gmail.com>]

On 10/31/18 3:57 PM, Paweł Staszewski wrote:
> Hi
> 
> So maybee someone will be interested how linux kernel handles normal
> traffic (not pktgen :) )
> 
> 
> Server HW configuration:
> 
> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
> 
> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
> 
> 
> Server software:
> 
> FRR - as routing daemon
> 
> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local numa
> node)
> 
> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa node)
> 
> 
> Maximum traffic that server can handle:
> 
> Bandwidth
> 
>  bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>   input: /proc/net/dev type: rate
>   \         iface                   Rx Tx                Total
> ==============================================================================
> 
>        enp175s0f1:          28.51 Gb/s           37.24 Gb/s          
> 65.74 Gb/s
>        enp175s0f0:          38.07 Gb/s           28.44 Gb/s          
> 66.51 Gb/s
> ------------------------------------------------------------------------------
> 
>             total:          66.58 Gb/s           65.67 Gb/s         
> 132.25 Gb/s
> 
> 
> Packets per second:
> 
>  bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>   input: /proc/net/dev type: rate
>   -         iface                   Rx Tx                Total
> ==============================================================================
> 
>        enp175s0f1:      5248589.00 P/s       3486617.75 P/s 8735207.00 P/s
>        enp175s0f0:      3557944.25 P/s       5232516.00 P/s 8790460.00 P/s
> ------------------------------------------------------------------------------
> 
>             total:      8806533.00 P/s       8719134.00 P/s 17525668.00 P/s
> 
> 
> After reaching that limits nics on the upstream side (more RX traffic)
> start to drop packets
> 
> 
> I just dont understand that server can't handle more bandwidth
> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX
> side are increasing.
> 
> Was thinking that maybee reached some pcie x16 limit - but x16 8GT is
> 126Gbit - and also when testing with pktgen i can reach more bw and pps
> (like 4x more comparing to normal internet traffic)
> 
> And wondering if there is something that can be improved here.

This is mainly a forwarding use case? Seems so based on the perf report.
I suspect forwarding with XDP would show pretty good improvement. You
need the vlan changes I have queued up though.