From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jesper Dangaard Brouer Subject: Re: Kernel 4.19 network performance - forwarding/routing normal users traffic Date: Sun, 11 Nov 2018 09:56:27 +0100 Message-ID: <20181111095627.6e2bf5cf@redhat.com> References: <61697e49-e839-befc-8330-fc00187c48ee@itcare.pl> <659fbf4b481c815f45a58b2351481cc9f761445b.camel@mellanox.com> <6486d01d-7a50-33c4-e27f-4ace8aa8e150@itcare.pl> <920c2665-781f-5f62-efbe-347e63063a24@itcare.pl> <162e25c6-dae2-7e1e-75f0-9c5b22453495@itcare.pl> <20181110203409.482f39ec@redhat.com> <5d44e0d9-d8d3-1b07-5392-838e4096ca74@itcare.pl> <20181110220131.655f7cc0@redhat.com> <78c37b54-dfd8-d528-1152-abe9ddbb08b2@itcare.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT Cc: Saeed Mahameed , "netdev@vger.kernel.org" , brouer@redhat.com To: =?UTF-8?B?UGF3ZcWC?= Staszewski Return-path: Received: from mx1.redhat.com ([209.132.183.28]:48460 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727334AbeKKSob (ORCPT ); Sun, 11 Nov 2018 13:44:31 -0500 In-Reply-To: <78c37b54-dfd8-d528-1152-abe9ddbb08b2@itcare.pl> Sender: netdev-owner@vger.kernel.org List-ID: On Sat, 10 Nov 2018 22:53:53 +0100 Paweł Staszewski wrote: > Now im messing with ring configuration for connectx5 nics. > And after reading that paper: > https://netdevconf.org/2.1/slides/apr6/network-performance/04-amir-RX_and_TX_bulking_v2.pdf > Do notice that some of the ideas in that slide deck, was never implemented. But they are still on my todo list ;-). Notice how that it show that TX bulking is very important, but based on your ethtool_stats.pl, I can see that not much TX bulking is happening in your case. This is indicated via the xmit_more counters. Ethtool(enp175s0) stat: 2630 ( 2,630) <= tx_xmit_more /sec Ethtool(enp175s0) stat: 4956995 ( 4,956,995) <= tx_packets /sec And the per queue levels are also avail: Ethtool(enp175s0) stat: 184845 ( 184,845) <= tx7_packets /sec Ethtool(enp175s0) stat:     78 (   78) <= tx7_xmit_more /sec This means that you are doing too many doorbell's to the NIC hardware at TX time, which I worry could be what cause the NIC and PCIe hardware not to operate at optimal speeds. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer