From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Ahern Subject: Re: Kernel 4.19 network performance - forwarding/routing normal users traffic Date: Thu, 8 Nov 2018 09:32:26 -0700 Message-ID: <68cc8279-5e3f-85c2-673c-aa3d4a47b353@gmail.com> References: <61697e49-e839-befc-8330-fc00187c48ee@itcare.pl> <3a88bb53-9d17-3e85-638e-a605f5bfe0fb@gmail.com> <20181101115522.10b0dd0a@redhat.com> <63198d68-6752-3695-f406-d86fb395c12b@itcare.pl> <7141e1e0-93e4-ab20-bce6-17f1e14682f1@gmail.com> <394a0bf2-fa97-1085-2eda-98ddf476895c@itcare.pl> <6ed1666d-47bc-24e7-d432-a0c0027452ed@gmail.com> <8dde3b32-59ce-38f3-5913-2ce08264e9dc@itcare.pl> <6165513d-1e27-31dc-8f94-9de029a73f93@gmail.com> <11199f9f-da21-527b-f5db-0bbf1e448a8b@itcare.pl> <87a2a15c-f9bf-743b-b4c5-7d37da0bd887@itcare.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Cc: netdev , Yoel Caspersen To: =?UTF-8?Q?Pawe=c5=82_Staszewski?= , Jesper Dangaard Brouer Return-path: Received: from mail-pl1-f196.google.com ([209.85.214.196]:33111 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726421AbeKICIp (ORCPT ); Thu, 8 Nov 2018 21:08:45 -0500 Received: by mail-pl1-f196.google.com with SMTP id w22-v6so6453863plk.0 for ; Thu, 08 Nov 2018 08:32:29 -0800 (PST) In-Reply-To: <87a2a15c-f9bf-743b-b4c5-7d37da0bd887@itcare.pl> Content-Language: en-US Sender: netdev-owner@vger.kernel.org List-ID: On 11/8/18 9:27 AM, Paweł Staszewski wrote: >>> What hardware is this? >>> > mellanox connectx 4 > ethtool -i enp175s0f0 > driver: mlx5_core > version: 5.0-0 > firmware-version: 12.21.1000 (SM_2001000001033) > expansion-rom-version: > bus-info: 0000:af:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: no > supports-priv-flags: yes > > ethtool -i enp175s0f1 > driver: mlx5_core > version: 5.0-0 > firmware-version: 12.21.1000 (SM_2001000001033) > expansion-rom-version: > bus-info: 0000:af:00.1 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: no > supports-priv-flags: yes > >>> Start with: >>> >>> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable >>> cat /sys/kernel/debug/tracing/trace_pipe >>  cat /sys/kernel/debug/tracing/trace_pipe >>          -0     [045] ..s. 68469.467752: xdp_devmap_xmit: >> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1 >> from_ifindex=4 to_ifindex=5 err=-6 FIB lookup is good, the redirect is happening, but the mlx5 driver does not like it. I think the -6 is coming from the mlx5 driver and the packet is getting dropped. Perhaps this check in mlx5e_xdp_xmit: if (unlikely(sq_num >= priv->channels.num)) return -ENXIO; >> swapper     0 [045] 68493.746274: fib:fib_table_lookup: table 254 oif >> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 >> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0 >>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms]) >> >> swapper     0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif >> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 >> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0 >>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms]) >> >> swapper     0 [045] 68495.794304: fib:fib_table_lookup: table 254 oif >> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 >> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0 >>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms]) >> >> swapper     0 [045] 68496.818308: fib:fib_table_lookup: table 254 oif >> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 >> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0 >>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms]) >> >> swapper     0 [045] 68497.842313: fib:fib_table_lookup: table 254 oif >> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 >> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0 >>             7fff818c13b5 fib_table_lookup ([kernel.kallsyms])