Re: Performance regression between 4.13 and 4.14

[Eric Dumazet <eric.dumazet@gmail.com>]

On 05/09/2018 11:43 AM, Ben Greear wrote:
> On 05/08/2018 10:10 AM, Eric Dumazet wrote:
>>
>>
>> On 05/08/2018 09:44 AM, Ben Greear wrote:
>>> Hello,
>>>
>>> I am trying to track down a performance regression that appears to be between 4.13
>>> and 4.14.
>>>
>>> I first saw the problem with a hacked version of pktgen on some ixgbe NICs.  4.13 can do
>>> right at 10G bi-directional on two ports, and 4.14 and later can do only about 6Gbps.
>>>
>>> I also tried with user-space UDP traffic on a stock kernel, and I can get about 3.2Gbps combined tx+rx
>>> on 4.14 and about 4.4Gbps on 4.13.
>>>
>>> Attempting to bisect seems to be triggering a weirdness in git, and also lots of commits
>>> crash or do not bring up networking, which makes the bisect difficult.
>>>
>>> Looking at perf top, it would appear that some lock is probably to blame.
>>
>>
>> perf record -a -g -e cycles:pp sleep 5
>> perf report
>>
>> Then you'll be able to tell us which lock (or call graph) is killing your perf.
>>
> 
> I seem to be chasing multiple issues.  For 4.13, at least part of my problem was that LOCKDEP was enabled,
> during my bisect, though it does NOT appear enabled in 4.16.  I think maybe CONFIG_LOCKDEP moved to CONFIG_PROVE_LOCKING
> in 4.16, or something like that?  My 4.16 .config does have CONFIG_LOCKDEP_SUPPORT enabled, and I see no option to disable it:
> 
> [greearb@ben-dt3 linux-4.16.x64]$ grep LOCKDEP .config
> CONFIG_LOCKDEP_SUPPORT=y
> 
> 
> For 4.16, I am disabling RETRAMPOLINE...are there any other such things I need
> to disable to keep from getting a performance hit from the spectre-related bug
> fixes?  At this point, I do not care about the security implications.
> 
> greearb@ben-dt3 linux-4.16.x64]$ grep RETPO .config
> # CONFIG_RETPOLINE is not set
> 
> 
> Thanks,
> Ben
> 

No idea really, you mention a 4.13 -> 4.14 regression and jump then to 4.16 :/

Before doing a (painful) dissection, the perf output would immediately tell you if
something is really wrong on your .config.