Re: NAT performance regression caused by vlan GRO support

["Rafał Miłecki" <zajec5@gmail.com>]

On 05.04.2019 10:12, Rafał Miłecki wrote:
> On 05.04.2019 09:58, Toshiaki Makita wrote:
>> On 2019/04/05 16:14, Felix Fietkau wrote:
>>> On 2019-04-05 09:11, Rafał Miłecki wrote:
>>>> I guess its GRO + csum_partial() to be blamed for this performance drop.
>>>>
>>>> Maybe csum_partial() is very fast on your powerful machine and few extra calls
>>>> don't make a difference? I can imagine it affecting much slower home router with
>>>> ARM cores.
>>> Most high performance Ethernet devices implement hardware checksum
>>> offload, which completely gets rid of this overhead.
>>> Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is
>>> why you're getting such crappy performance.
>>
>> Hmm... now I disabled rx checksum and tried the test again, and indeed I
>> see csum_partial from GRO path. But I also see csum_partial even without
>> GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete.
>> Probably Rafał disabled nf_conntrack_checksum sysctl knob?
>>
>> But anyway even with disabling rx csum offload my machine has better
>> performance with GRO. I'm sure in some cases GRO should be disabled, but
>> I guess it's difficult to determine whether we should disable GRO or not
>> automatically when csum offload is not available.
> 
> Few testing results:
> 
> 1) ethtool -K eth0 gro off; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum
> [  6]  0.0-60.0 sec  6.57 GBytes   940 Mbits/sec
> 
> 2) ethtool -K eth0 gro off; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum
> [  6]  0.0-60.0 sec  4.65 GBytes   666 Mbits/sec

For this case (GRO off and nf_conntrack_checksum enabled) I can confirm I see
csum_partial() in the perf output. It's taking 13,14% instead of 25,46% (as when
using GRO) though.

Samples: 38K of event 'cycles', Event count (approx.): 12209908413
   Overhead  Command          Shared Object           Symbol
+   13,14%  ksoftirqd/1      [kernel.kallsyms]       [k] csum_partial
+   10,16%  swapper          [kernel.kallsyms]       [k] v7_dma_inv_range
+    6,36%  swapper          [kernel.kallsyms]       [k] l2c210_inv_range
+    4,89%  swapper          [kernel.kallsyms]       [k] __irqentry_text_end
+    4,12%  ksoftirqd/1      [kernel.kallsyms]       [k] v7_dma_clean_range
+    3,78%  swapper          [kernel.kallsyms]       [k] bcma_host_soc_read32
+    2,76%  swapper          [kernel.kallsyms]       [k] arch_cpu_idle
+    2,45%  ksoftirqd/1      [kernel.kallsyms]       [k] __netif_receive_skb_core
+    2,37%  ksoftirqd/1      [kernel.kallsyms]       [k] l2c210_clean_range
+    1,76%  ksoftirqd/1      [kernel.kallsyms]       [k] bgmac_start_xmit
+    1,66%  swapper          [kernel.kallsyms]       [k] bgmac_poll
+    1,55%  ksoftirqd/1      [kernel.kallsyms]       [k] __dev_queue_xmit
+    1,11%  ksoftirqd/1      [kernel.kallsyms]       [k] skb_vlan_untag


> 3) ethtool -K eth0 gro on; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum
> [  6]  0.0-60.0 sec  4.02 GBytes   575 Mbits/sec
> 
> 4) ethtool -K eth0 gro on; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum
> [  6]  0.0-60.0 sec  4.04 GBytes   579 Mbits/sec