From mboxrd@z Thu Jan 1 00:00:00 1970 From: Duyck, Alexander H Date: Thu, 1 Jun 2017 19:18:05 +0000 Subject: [Intel-wired-lan] [i40e] regression on TCP stream and TCP maerts, kernel-4.12.0-0.rc2 In-Reply-To: <1496312080.3758.7.camel@redhat.com> References: <655ed9ab-853e-7497-9ea3-cc8d13f56043@redhat.com> <1496238507.22688.4.camel@redhat.com> <1496312080.3758.7.camel@redhat.com> Message-ID: <1496344683.27073.306.camel@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On Thu, 2017-06-01 at 12:14 +0200, Adrian Tomasov wrote: > On Wed, 2017-05-31 at 14:42 -0700, Alexander Duyck wrote: > > > > On Wed, May 31, 2017 at 6:48 AM, Adrian Tomasov > > wrote: > > > > > > > > > On Tue, 2017-05-30 at 18:27 -0700, Alexander Duyck wrote: > > > > > > > > > > > > On Tue, May 30, 2017 at 8:41 AM, Alexander Duyck > > > > wrote: > > > > > > > > > > > > > > > > > > > > On Tue, May 30, 2017 at 6:43 AM, Adam Okuliar > > > > com> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > we found regression on intel card(XL710) with i40e driver. > > > > > > Regression is > > > > > > about ~45% > > > > > > on TCP_STREAM and TCP_MAERTS test for IPv4 and IPv6. > > > > > > Regression > > > > > > was first > > > > > > visible in kernel-4.12.0-0.rc1. > > > > > > > > > > > > More details about results you can see in uploaded images in > > > > > > bugzilla. [0] > > > > > > > > > > > > > > > > > > [0] https://bugzilla.kernel.org/show_bug.cgi?id=195923 > > > > > > > > > > > > > > > > > > Best regards, / S pozdravom, > > > > > > > > > > > > Adri?n Toma?ov > > > > > > Kernel Performance QE > > > > > > atomasov at redhat.com > > > > > > > > > > I have added the i40e driver maintainer and the intel-wired-lan > > > > > mailing list so that we can make are developers aware of the > > > > > issue. > > > > > > > > > > Thanks. > > > > > > > > > > - Alex > > > > > > > > Adam, > > > > > > > > We are having some issues trying to reproduce what you reported. > > > > > > > > Can you provide some additional data. Specifically we would be > > > > looking > > > > for an "ethtool -i", and an "ethtool -S" for the port before and > > > > after > > > > the test. If you can attach it to the bugzilla that would be > > > > appreciated. > > > > > > > > Thanks. > > > > > > > > - Alex > > > > > > Hello Alex, > > > > > > requested files are updated in bugzilla. > > > > > > If you have any questions about testing feel free to ask. > > > > > > > > > Best regards, > > > > > > Adrian > > > > So looking at the data I wonder if we don't have an MTU mismatch in > > the network config. I notice the "after" has rx_length_errors being > > reported. Recent changes made it so that i40e doesn't support jumbo > > frames by default, whereas before we could. You might want to check > > for that as that could cause the kind of performance issues you are > > seeing. > > > > - Alex > > There isn't MTU mismatch. Traffic path is : server -> switch -> > server.? > > > Output from switch: > > > show interfaces et-0/0/18???? > Physical interface: et-0/0/18, Enabled, Physical link is Up > ? Interface index: 644, SNMP ifIndex: 538 > ? Link-level type: Ethernet, MTU: 1514, Speed: 40Gbps, BPDU Error: > None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: > Disabled, Flow control: Disabled, Media type: Fiber > ? Device flags???: Present Running > ? Interface flags: SNMP-Traps Internal: 0x4000 > ? Link flags?????: None > ? CoS queues?????: 12 supported, 12 maximum usable queues > ? Current address: d4:04:ff:90:5a:4b, Hardware address: > d4:04:ff:90:5a:4b > ? Last flapped???: 2017-06-01 10:09:32 CEST (01:21:29 ago) > ? Input rate?????: 432 bps (0 pps) > ? Output rate????: 8336 bps (11 pps) > ? Active alarms??: None > ? Active defects : None > ? Interface transmit statistics: Disabled > > ? Logical interface et-0/0/18.0 (Index 552) (SNMP ifIndex 539) > ????Flags: SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge > ????Input packets : 464041 > ????Output packets: 209210 > ????Protocol eth-switch, MTU: 1514 > ??????Flags: Is-Primary, Trunk-Mode > > > MTU is same for all et-0/0/x interfaces.? > > - Adrian One thing you might try try doing is toggling the legacy-rx flag using the "ethtool --show-priv-flags/--set-priv-flags" command to see if that has any impact. That will help to rule things out as the most significant change I can think of is the recent update of the Rx path to support XDP. Also one other thing you might try would be to use a fixed interrupt moderation rate by locking things down using "ethtool -C" to disable adaptive interrupt moderation and lock the Rx usecs and Tx usecs at some predefined values. I seem to recall there have been some interrupt moderation changes made recently that might be impacting the performance. Beyond that is there any chance you would be able to bisect the issue? Unfortunately we haven't be able to reproduce it internally so anything that would help us to narrow down the problem would be useful. Thanks. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Duyck, Alexander H" Subject: Re: [Intel-wired-lan] [i40e] regression on TCP stream and TCP maerts, kernel-4.12.0-0.rc2 Date: Thu, 1 Jun 2017 19:18:05 +0000 Message-ID: <1496344683.27073.306.camel@intel.com> References: <655ed9ab-853e-7497-9ea3-cc8d13f56043@redhat.com> <1496238507.22688.4.camel@redhat.com> <1496312080.3758.7.camel@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: "osabart@redhat.com" , "netdev@vger.kernel.org" , "aokuliar@redhat.com" , "intel-wired-lan@lists.osuosl.org" , "jhladky@redhat.com" To: "atomasov@redhat.com" , "alexander.duyck@gmail.com" Return-path: Received: from mga06.intel.com ([134.134.136.31]:20840 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751240AbdFATSR (ORCPT ); Thu, 1 Jun 2017 15:18:17 -0400 In-Reply-To: <1496312080.3758.7.camel@redhat.com> Content-Language: en-US Content-ID: <59EBFAFEE587AF4F97F2221C2F1FD4E9@intel.com> Sender: netdev-owner@vger.kernel.org List-ID: T24gVGh1LCAyMDE3LTA2LTAxIGF0IDEyOjE0ICswMjAwLCBBZHJpYW4gVG9tYXNvdiB3cm90ZToN Cj4gT24gV2VkLCAyMDE3LTA1LTMxIGF0IDE0OjQyIC0wNzAwLCBBbGV4YW5kZXIgRHV5Y2sgd3Jv dGU6DQo+ID4gDQo+ID4gT24gV2VkLCBNYXkgMzEsIDIwMTcgYXQgNjo0OCBBTSwgQWRyaWFuIFRv bWFzb3YgPGF0b21hc292QHJlZGhhdC5jb20+DQo+ID4gd3JvdGU6DQo+ID4gPiANCj4gPiA+IA0K PiA+ID4gT24gVHVlLCAyMDE3LTA1LTMwIGF0IDE4OjI3IC0wNzAwLCBBbGV4YW5kZXIgRHV5Y2sg d3JvdGU6DQo+ID4gPiA+IA0KPiA+ID4gPiANCj4gPiA+ID4gT24gVHVlLCBNYXkgMzAsIDIwMTcg YXQgODo0MSBBTSwgQWxleGFuZGVyIER1eWNrDQo+ID4gPiA+IDxhbGV4YW5kZXIuZHV5Y2tAZ21h aWwuY29tPiB3cm90ZToNCj4gPiA+ID4gPiANCj4gPiA+ID4gPiANCj4gPiA+ID4gPiANCj4gPiA+ ID4gPiBPbiBUdWUsIE1heSAzMCwgMjAxNyBhdCA2OjQzIEFNLCBBZGFtIE9rdWxpYXIgPGFva3Vs aWFyQHJlZGhhdC4NCj4gPiA+ID4gPiBjb20+DQo+ID4gPiA+ID4gd3JvdGU6DQo+ID4gPiA+ID4g PiANCj4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiBIZWxsbywNCj4gPiA+ ID4gPiA+IA0KPiA+ID4gPiA+ID4gd2UgZm91bmQgcmVncmVzc2lvbiBvbiBpbnRlbCBjYXJkKFhM NzEwKSB3aXRoIGk0MGUgZHJpdmVyLg0KPiA+ID4gPiA+ID4gUmVncmVzc2lvbiBpcw0KPiA+ID4g PiA+ID4gYWJvdXQgfjQ1JQ0KPiA+ID4gPiA+ID4gb24gVENQX1NUUkVBTSBhbmQgVENQX01BRVJU UyB0ZXN0IGZvciBJUHY0IGFuZCBJUHY2Lg0KPiA+ID4gPiA+ID4gUmVncmVzc2lvbg0KPiA+ID4g PiA+ID4gd2FzIGZpcnN0DQo+ID4gPiA+ID4gPiB2aXNpYmxlIGluIGtlcm5lbC00LjEyLjAtMC5y YzEuDQo+ID4gPiA+ID4gPiANCj4gPiA+ID4gPiA+IE1vcmUgZGV0YWlscyBhYm91dCByZXN1bHRz IHlvdSBjYW4gc2VlIGluIHVwbG9hZGVkIGltYWdlcyBpbg0KPiA+ID4gPiA+ID4gYnVnemlsbGEu IFswXQ0KPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiANCj4gPiA+ID4gPiA+IFswXSBodHRwczov L2J1Z3ppbGxhLmtlcm5lbC5vcmcvc2hvd19idWcuY2dpP2lkPTE5NTkyMw0KPiA+ID4gPiA+ID4g DQo+ID4gPiA+ID4gPiANCj4gPiA+ID4gPiA+IEJlc3QgcmVnYXJkcywgLyBTIHBvemRyYXZvbSwN Cj4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gQWRyacOhbiBUb21hxaFvdg0KPiA+ID4gPiA+ID4g S2VybmVsIFBlcmZvcm1hbmNlIFFFDQo+ID4gPiA+ID4gPiBhdG9tYXNvdkByZWRoYXQuY29tDQo+ ID4gPiA+ID4gDQo+ID4gPiA+ID4gSSBoYXZlIGFkZGVkIHRoZSBpNDBlIGRyaXZlciBtYWludGFp bmVyIGFuZCB0aGUgaW50ZWwtd2lyZWQtbGFuDQo+ID4gPiA+ID4gbWFpbGluZyBsaXN0IHNvIHRo YXQgd2UgY2FuIG1ha2UgYXJlIGRldmVsb3BlcnMgYXdhcmUgb2YgdGhlDQo+ID4gPiA+ID4gaXNz dWUuDQo+ID4gPiA+ID4gDQo+ID4gPiA+ID4gVGhhbmtzLg0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+ IC0gQWxleA0KPiA+ID4gPiANCj4gPiA+ID4gQWRhbSwNCj4gPiA+ID4gDQo+ID4gPiA+IFdlIGFy ZSBoYXZpbmcgc29tZSBpc3N1ZXMgdHJ5aW5nIHRvIHJlcHJvZHVjZSB3aGF0IHlvdSByZXBvcnRl ZC4NCj4gPiA+ID4gDQo+ID4gPiA+IENhbiB5b3UgcHJvdmlkZSBzb21lIGFkZGl0aW9uYWwgZGF0 YS4gU3BlY2lmaWNhbGx5IHdlIHdvdWxkIGJlDQo+ID4gPiA+IGxvb2tpbmcNCj4gPiA+ID4gZm9y IGFuICJldGh0b29sIC1pIiwgYW5kIGFuICJldGh0b29sIC1TIiBmb3IgdGhlIHBvcnQgYmVmb3Jl IGFuZA0KPiA+ID4gPiBhZnRlcg0KPiA+ID4gPiB0aGUgdGVzdC4gSWYgeW91IGNhbiBhdHRhY2gg aXQgdG8gdGhlIGJ1Z3ppbGxhIHRoYXQgd291bGQgYmUNCj4gPiA+ID4gYXBwcmVjaWF0ZWQuDQo+ ID4gPiA+IA0KPiA+ID4gPiBUaGFua3MuDQo+ID4gPiA+IA0KPiA+ID4gPiAtIEFsZXgNCj4gPiA+ IA0KPiA+ID4gSGVsbG8gQWxleCwNCj4gPiA+IA0KPiA+ID4gcmVxdWVzdGVkIGZpbGVzIGFyZSB1 cGRhdGVkIGluIGJ1Z3ppbGxhLg0KPiA+ID4gDQo+ID4gPiBJZiB5b3UgaGF2ZSBhbnkgcXVlc3Rp b25zIGFib3V0IHRlc3RpbmcgZmVlbCBmcmVlIHRvIGFzay4NCj4gPiA+IA0KPiA+ID4gDQo+ID4g PiBCZXN0IHJlZ2FyZHMsDQo+ID4gPiANCj4gPiA+IEFkcmlhbg0KPiA+IA0KPiA+IFNvIGxvb2tp bmcgYXQgdGhlIGRhdGEgSSB3b25kZXIgaWYgd2UgZG9uJ3QgaGF2ZSBhbiBNVFUgbWlzbWF0Y2gg aW4NCj4gPiB0aGUgbmV0d29yayBjb25maWcuIEkgbm90aWNlIHRoZSAiYWZ0ZXIiIGhhcyByeF9s ZW5ndGhfZXJyb3JzIGJlaW5nDQo+ID4gcmVwb3J0ZWQuIFJlY2VudCBjaGFuZ2VzIG1hZGUgaXQg c28gdGhhdCBpNDBlIGRvZXNuJ3Qgc3VwcG9ydCBqdW1ibw0KPiA+IGZyYW1lcyBieSBkZWZhdWx0 LCB3aGVyZWFzIGJlZm9yZSB3ZSBjb3VsZC4gWW91IG1pZ2h0IHdhbnQgdG8gY2hlY2sNCj4gPiBm b3IgdGhhdCBhcyB0aGF0IGNvdWxkIGNhdXNlIHRoZSBraW5kIG9mIHBlcmZvcm1hbmNlIGlzc3Vl cyB5b3UgYXJlDQo+ID4gc2VlaW5nLg0KPiA+IA0KPiA+IC0gQWxleA0KPiANCj4gVGhlcmUgaXNu J3QgTVRVIG1pc21hdGNoLiBUcmFmZmljIHBhdGggaXMgOiBzZXJ2ZXIgLT4gc3dpdGNoIC0+DQo+ IHNlcnZlci7CoA0KPiANCj4gDQo+IE91dHB1dCBmcm9tIHN3aXRjaDoNCj4gDQo+ICAgICA+IHNo b3cgaW50ZXJmYWNlcyBldC0wLzAvMTjCoMKgwqDCoA0KPiAgICAgUGh5c2ljYWwgaW50ZXJmYWNl OiBldC0wLzAvMTgsIEVuYWJsZWQsIFBoeXNpY2FsIGxpbmsgaXMgVXANCj4gICAgIMKgIEludGVy ZmFjZSBpbmRleDogNjQ0LCBTTk1QIGlmSW5kZXg6IDUzOA0KPiAgICAgwqAgTGluay1sZXZlbCB0 eXBlOiBFdGhlcm5ldCwgTVRVOiAxNTE0LCBTcGVlZDogNDBHYnBzLCBCUERVIEVycm9yOg0KPiAg ICAgTm9uZSwgTUFDLVJFV1JJVEUgRXJyb3I6IE5vbmUsIExvb3BiYWNrOiBEaXNhYmxlZCwgU291 cmNlIGZpbHRlcmluZzoNCj4gICAgIERpc2FibGVkLCBGbG93IGNvbnRyb2w6IERpc2FibGVkLCBN ZWRpYSB0eXBlOiBGaWJlcg0KPiAgICAgwqAgRGV2aWNlIGZsYWdzwqDCoMKgOiBQcmVzZW50IFJ1 bm5pbmcNCj4gICAgIMKgIEludGVyZmFjZSBmbGFnczogU05NUC1UcmFwcyBJbnRlcm5hbDogMHg0 MDAwDQo+ICAgICDCoCBMaW5rIGZsYWdzwqDCoMKgwqDCoDogTm9uZQ0KPiAgICAgwqAgQ29TIHF1 ZXVlc8KgwqDCoMKgwqA6IDEyIHN1cHBvcnRlZCwgMTIgbWF4aW11bSB1c2FibGUgcXVldWVzDQo+ ICAgICDCoCBDdXJyZW50IGFkZHJlc3M6IGQ0OjA0OmZmOjkwOjVhOjRiLCBIYXJkd2FyZSBhZGRy ZXNzOg0KPiAgICAgZDQ6MDQ6ZmY6OTA6NWE6NGINCj4gICAgIMKgIExhc3QgZmxhcHBlZMKgwqDC oDogMjAxNy0wNi0wMSAxMDowOTozMiBDRVNUICgwMToyMToyOSBhZ28pDQo+ICAgICDCoCBJbnB1 dCByYXRlwqDCoMKgwqDCoDogNDMyIGJwcyAoMCBwcHMpDQo+ICAgICDCoCBPdXRwdXQgcmF0ZcKg wqDCoMKgOiA4MzM2IGJwcyAoMTEgcHBzKQ0KPiAgICAgwqAgQWN0aXZlIGFsYXJtc8KgwqA6IE5v bmUNCj4gICAgIMKgIEFjdGl2ZSBkZWZlY3RzIDogTm9uZQ0KPiAgICAgwqAgSW50ZXJmYWNlIHRy YW5zbWl0IHN0YXRpc3RpY3M6IERpc2FibGVkDQo+IA0KPiAgICAgwqAgTG9naWNhbCBpbnRlcmZh Y2UgZXQtMC8wLzE4LjAgKEluZGV4IDU1MikgKFNOTVAgaWZJbmRleCA1MzkpDQo+ICAgICDCoMKg wqDCoEZsYWdzOiBTTk1QLVRyYXBzIDB4MjQwMjQwMDAgRW5jYXBzdWxhdGlvbjogRXRoZXJuZXQt QnJpZGdlDQo+ICAgICDCoMKgwqDCoElucHV0IHBhY2tldHMgOiA0NjQwNDENCj4gICAgIMKgwqDC oMKgT3V0cHV0IHBhY2tldHM6IDIwOTIxMA0KPiAgICAgwqDCoMKgwqBQcm90b2NvbCBldGgtc3dp dGNoLCBNVFU6IDE1MTQNCj4gICAgIMKgwqDCoMKgwqDCoEZsYWdzOiBJcy1QcmltYXJ5LCBUcnVu ay1Nb2RlDQo+IA0KPiANCj4gTVRVIGlzIHNhbWUgZm9yIGFsbCBldC0wLzAveCBpbnRlcmZhY2Vz LsKgDQo+IA0KPiAtIEFkcmlhbg0KDQpPbmUgdGhpbmcgeW91IG1pZ2h0IHRyeSB0cnkgZG9pbmcg aXMgdG9nZ2xpbmcgdGhlIGxlZ2FjeS1yeCBmbGFnIHVzaW5nDQp0aGUgImV0aHRvb2wgLS1zaG93 LXByaXYtZmxhZ3MvLS1zZXQtcHJpdi1mbGFncyIgY29tbWFuZCB0byBzZWUgaWYgdGhhdA0KaGFz IGFueSBpbXBhY3QuIFRoYXQgd2lsbCBoZWxwIHRvIHJ1bGUgdGhpbmdzIG91dCBhcyB0aGUgbW9z dA0Kc2lnbmlmaWNhbnQgY2hhbmdlIEkgY2FuIHRoaW5rIG9mIGlzIHRoZSByZWNlbnQgdXBkYXRl IG9mIHRoZSBSeCBwYXRoDQp0byBzdXBwb3J0IFhEUC4NCg0KQWxzbyBvbmUgb3RoZXIgdGhpbmcg eW91IG1pZ2h0IHRyeSB3b3VsZCBiZSB0byB1c2UgYSBmaXhlZCBpbnRlcnJ1cHQNCm1vZGVyYXRp b24gcmF0ZSBieSBsb2NraW5nIHRoaW5ncyBkb3duIHVzaW5nICJldGh0b29sIC1DIiB0byBkaXNh YmxlDQphZGFwdGl2ZSBpbnRlcnJ1cHQgbW9kZXJhdGlvbiBhbmQgbG9jayB0aGUgUnggdXNlY3Mg YW5kIFR4IHVzZWNzIGF0DQpzb21lIHByZWRlZmluZWQgdmFsdWVzLiBJIHNlZW0gdG8gcmVjYWxs IHRoZXJlIGhhdmUgYmVlbiBzb21lIGludGVycnVwdA0KbW9kZXJhdGlvbiBjaGFuZ2VzIG1hZGUg cmVjZW50bHkgdGhhdCBtaWdodCBiZSBpbXBhY3RpbmcgdGhlDQpwZXJmb3JtYW5jZS4NCg0KQmV5 b25kIHRoYXQgaXMgdGhlcmUgYW55IGNoYW5jZSB5b3Ugd291bGQgYmUgYWJsZSB0byBiaXNlY3Qg dGhlIGlzc3VlPw0KVW5mb3J0dW5hdGVseSB3ZSBoYXZlbid0IGJlIGFibGUgdG8gcmVwcm9kdWNl IGl0IGludGVybmFsbHkgc28gYW55dGhpbmcNCnRoYXQgd291bGQgaGVscCB1cyB0byBuYXJyb3cg ZG93biB0aGUgcHJvYmxlbSB3b3VsZCBiZSB1c2VmdWwuDQoNClRoYW5rcy4NCg0KLSBBbGV4