Re: Kernel 4.19 network performance - forwarding/routing normal users traffic

[David Ahern <dsahern@gmail.com>]

On 11/8/18 9:27 AM, Paweł Staszewski wrote:
>>> What hardware is this?
>>>
> mellanox connectx 4
> ethtool -i enp175s0f0
> driver: mlx5_core
> version: 5.0-0
> firmware-version: 12.21.1000 (SM_2001000001033)
> expansion-rom-version:
> bus-info: 0000:af:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: yes
> 
> ethtool -i enp175s0f1
> driver: mlx5_core
> version: 5.0-0
> firmware-version: 12.21.1000 (SM_2001000001033)
> expansion-rom-version:
> bus-info: 0000:af:00.1
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: yes
> 
>>> Start with:
>>>
>>> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
>>> cat /sys/kernel/debug/tracing/trace_pipe
>>  cat /sys/kernel/debug/tracing/trace_pipe
>>          <idle>-0     [045] ..s. 68469.467752: xdp_devmap_xmit:
>> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
>> from_ifindex=4 to_ifindex=5 err=-6

FIB lookup is good, the redirect is happening, but the mlx5 driver does
not like it.

I think the -6 is coming from the mlx5 driver and the packet is getting
dropped. Perhaps this check in mlx5e_xdp_xmit:

       if (unlikely(sq_num >= priv->channels.num))
                return -ENXIO;



>> swapper     0 [045] 68493.746274: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>
>> swapper     0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>
>> swapper     0 [045] 68495.794304: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>
>> swapper     0 [045] 68496.818308: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>
>> swapper     0 [045] 68497.842313: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms])