From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven Rostedt Subject: Re: Performance regressions in TCP_STREAM tests in Linux 4.15 (and later) Date: Mon, 30 Apr 2018 12:31:36 -0400 Message-ID: <20180430123136.59caaf81@gandalf.local.home> References: <20180427231149.119db14c@vmware.local.home> <476bfc0f-eb2d-fe57-73d9-ec8a8392ad33@candelatech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Michael Wenig , "netdev@vger.kernel.org" , "eric.dumazet@gmail.com" , Shilpi Agarwal , Boon Ang , Darren Hart , Abdul Anshad Azeez To: Ben Greear Return-path: Received: from mail.kernel.org ([198.145.29.99]:43942 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754631AbeD3Qbj (ORCPT ); Mon, 30 Apr 2018 12:31:39 -0400 In-Reply-To: <476bfc0f-eb2d-fe57-73d9-ec8a8392ad33@candelatech.com> Sender: netdev-owner@vger.kernel.org List-ID: On Mon, 30 Apr 2018 09:14:04 -0700 Ben Greear wrote: > >> As part of VMware's performance testing with the Linux 4.15 kernel, > >> we identified CPU cost and throughput regressions when comparing to > >> the Linux 4.14 kernel. The impacted test cases are mostly TCP_STREAM > >> send tests when using small message sizes. The regressions are > >> significant (up 3x) and were tracked down to be a side effect of Eric > >> Dumazat's RB tree changes that went into the Linux 4.15 kernel. > >> Further investigation showed our use of the TCP_NODELAY flag in > >> conjunction with Eric's change caused the regressions to show and > >> simply disabling TCP_NODELAY brought performance back to normal. > >> Eric's change also resulted into significant improvements in our > >> TCP_RR test cases. > >> > >> > >> > >> Based on these results, our theory is that Eric's change made the > >> system overall faster (reduced latency) but as a side effect less > >> aggregation is happening (with TCP_NODELAY) and that results in lower > >> throughput. Previously even though TCP_NODELAY was set, system was > >> slower and we still got some benefit of aggregation. Aggregation > >> helps in better efficiency and higher throughput although it can > >> increase the latency. If you are seeing a regression in your > >> application throughput after this change, using TCP_NODELAY might > >> help bring performance back however that might increase latency. > > I guess you mean _disabling_ TCP_NODELAY instead of _using_ TCP_NODELAY? Yes, thank you for catching that. -- Steve