Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later)

[Steven Rostedt <rostedt@goodmis.org>]

We'd like this email archived in netdev list, but since netdev is
notorious for blocking outlook email as spam, it didn't go through. So
I'm replying here to help get it into the archives.

Thanks!

-- Steve


On Fri, 27 Apr 2018 23:05:46 +0000
Michael Wenig <mwenig@vmware.com> wrote:

> As part of VMware's performance testing with the Linux 4.15 kernel,
> we identified CPU cost and throughput regressions when comparing to
> the Linux 4.14 kernel. The impacted test cases are mostly TCP_STREAM
> send tests when using small message sizes. The regressions are
> significant (up 3x) and were tracked down to be a side effect of Eric
> Dumazat's RB tree changes that went into the Linux 4.15 kernel.
> Further investigation showed our use of the TCP_NODELAY flag in
> conjunction with Eric's change caused the regressions to show and
> simply disabling TCP_NODELAY brought performance back to normal.
> Eric's change also resulted into significant improvements in our
> TCP_RR test cases.
> 
> 
> 
> Based on these results, our theory is that Eric's change made the
> system overall faster (reduced latency) but as a side effect less
> aggregation is happening (with TCP_NODELAY) and that results in lower
> throughput. Previously even though TCP_NODELAY was set, system was
> slower and we still got some benefit of aggregation. Aggregation
> helps in better efficiency and higher throughput although it can
> increase the latency. If you are seeing a regression in your
> application throughput after this change, using TCP_NODELAY might
> help bring performance back however that might increase latency.
> 
> 
> 
> As such, we are not asking for a fix but simply want to document for
> others what we have found.
> 
> 
> 
> Michael Wenig
> 
> Performance Engineering
> 
> VMware, Inc.
>