From mboxrd@z Thu Jan 1 00:00:00 1970 From: Anders K. Pedersen | Cohaesio Date: Sun, 22 Oct 2017 13:56:40 +0000 Subject: [Intel-wired-lan] Linux 4.12+ memory leak on router with i40e NICs In-Reply-To: References: <1507121766.30720.4.camel@cohaesio.com> <3d783736-a474-d9e3-2de2-e35c765f8249@itcare.pl> <39696136-2a4a-9c6c-3a63-4485ed2a1bf3@itcare.pl> <310ce203-0d65-bdf4-d9e4-897a349b3277@itcare.pl> <1704eb26-c4c5-b196-ca7a-5265e92ae4e6@itcare.pl> <7fd828d7-c586-2c32-3ba6-e0575bf9958c@itcare.pl> <99eba19f-327e-e01f-4f4a-87540b176e40@itcare.pl> <748b6d9d-4a3f-4eaa-ad24-27060c2e2642@itcare.pl> <16c9fa34-252f-e5a3-8b15-e5a8c4d8a46f@itcare.pl> <12670bc6-439c-7ef4-109a-fd20384b9ca2@itcare.pl> <888ec430-1d70-68e6-2ee8-bab53d21bb0f@itcare.pl> <1508415582.1186.49.camel@cohaesio.com> Message-ID: <1508680600.4970.26.camel@cohaesio.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On tor, 2017-10-19 at 08:40 -0700, Alexander Duyck wrote: > On Thu, Oct 19, 2017 at 5:19 AM, Anders K. Pedersen | Cohaesio > wrote: > > Hi Alex, > > > > On ons, 2017-10-18 at 16:37 -0700, Alexander Duyck wrote: > > > When we last talked I had asked if you could do a git bisect to > > > find > > > the memory leak and you said you would look into it. The most > > > useful > > > way to solve this would be to do a git bisect between your > > > current > > > kernel and the 4.11 kernel to find the point at which this > > > started. > > > If > > > we can do that then fixing this becomes much simpler as we just > > > have > > > to fix the patch that introduced the issue. > > > > We're also seeing a smaller memory leak (about 1 GB per day) than > > the > > original one even with the "Fix memory leak related filter > > programming > > status" fix applied. So far I've determined that the leak is > > present on > > 4.13.7 and was introduced between 4.11 and 4.12, so I'll do another > > round of bisection to identify the patch that introduced this. > > > > Since the router must run for a couple of hours before I can be > > sure > > whether a kernel is good or bad, and I can't reboot it during > > working > > hours, it'll probably be about a week before I have a result. > > > > -- > > Venlig hilsen / Best Regards > > > > Anders K. Pedersen > > Senior Technical Manager > > Anders, > > I'll do some digging on my side to see if I can find any other memory > leaks that might be floating around in the driver that could have > been > introduced during that time-frame. > > One thing you might try that would help with your testing would be to > just disable the ATR functionality in i40e. You can do that with the > ethtool command "ethtool --set-priv-flags flow-director-atr > off". That should allow you to bisect this without needing to deal > with the "programming status" patches since you won't be programming > ATR filters which is what caused that leak. > > Thanks for looking into this. > > - Alex Hi Alex, I began bisecting, where I applied the known fix patches to the steps, where they were applicable (i.e. without changing the flow-director-atr flag), but some of the steps had a high amount of packet drops, which caused problems for our network, so I couldn't leave them running for several hours, which is necessary to determine if the leak is present or not. The part of the bisection I got through had the same outcome as the last bisection, which led to "i40e: Fix support for flow director programming status". After that I experimented a bit with the flow-director-atr flag, and it turns out that if I disable this flag on all the NICs, then the memory leak is gone, so I suspected that the smaller memory leak was also caused by "i40e: Fix support for flow director programming status". I tried to revert this patch from 4.13 (with manual fixup for the trace point that had been added later), but that brought back the packet drops, so I couldn't let it run. This morning I saw your "i40e: Add programming descriptors to cleaned_count" patch, so I tried 4.13.9 with that patch and the previous "i40e: Fix memory leak related filter programming status" without turning off the flow-director-atr flag. So far this combination is running stable without any memory leaks. Thanks for fixing this. Regards, Anders From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Anders K. Pedersen | Cohaesio" Subject: Re: Linux 4.12+ memory leak on router with i40e NICs Date: Sun, 22 Oct 2017 13:56:40 +0000 Message-ID: <1508680600.4970.26.camel@cohaesio.com> References: <1507121766.30720.4.camel@cohaesio.com> <3d783736-a474-d9e3-2de2-e35c765f8249@itcare.pl> <39696136-2a4a-9c6c-3a63-4485ed2a1bf3@itcare.pl> <310ce203-0d65-bdf4-d9e4-897a349b3277@itcare.pl> <1704eb26-c4c5-b196-ca7a-5265e92ae4e6@itcare.pl> <7fd828d7-c586-2c32-3ba6-e0575bf9958c@itcare.pl> <99eba19f-327e-e01f-4f4a-87540b176e40@itcare.pl> <748b6d9d-4a3f-4eaa-ad24-27060c2e2642@itcare.pl> <16c9fa34-252f-e5a3-8b15-e5a8c4d8a46f@itcare.pl> <12670bc6-439c-7ef4-109a-fd20384b9ca2@itcare.pl> <888ec430-1d70-68e6-2ee8-bab53d21bb0f@itcare.pl> <1508415582.1186.49.camel@cohaesio.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: "pstaszewski@itcare.pl" , "netdev@vger.kernel.org" , "pavlos.parissis@gmail.com" , "intel-wired-lan@lists.osuosl.org" , "alexander.h.duyck@intel.com" To: "alexander.duyck@gmail.com" Return-path: Received: from bender.cohaesio.com ([212.97.128.130]:46026 "EHLO mail.cohaesio.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750984AbdJVN4o (ORCPT ); Sun, 22 Oct 2017 09:56:44 -0400 In-Reply-To: Content-Language: en-US Content-ID: Sender: netdev-owner@vger.kernel.org List-ID: T24gdG9yLCAyMDE3LTEwLTE5IGF0IDA4OjQwIC0wNzAwLCBBbGV4YW5kZXIgRHV5Y2sgd3JvdGU6 DQo+IE9uIFRodSwgT2N0IDE5LCAyMDE3IGF0IDU6MTkgQU0sIEFuZGVycyBLLiBQZWRlcnNlbiB8 IENvaGFlc2lvDQo+IDxha3BAY29oYWVzaW8uY29tPiB3cm90ZToNCj4gPiBIaSBBbGV4LA0KPiA+ IA0KPiA+IE9uIG9ucywgMjAxNy0xMC0xOCBhdCAxNjozNyAtMDcwMCwgQWxleGFuZGVyIER1eWNr IHdyb3RlOg0KPiA+ID4gV2hlbiB3ZSBsYXN0IHRhbGtlZCBJIGhhZCBhc2tlZCBpZiB5b3UgY291 bGQgZG8gYSBnaXQgYmlzZWN0IHRvDQo+ID4gPiBmaW5kDQo+ID4gPiB0aGUgbWVtb3J5IGxlYWsg YW5kIHlvdSBzYWlkIHlvdSB3b3VsZCBsb29rIGludG8gaXQuIFRoZSBtb3N0DQo+ID4gPiB1c2Vm dWwNCj4gPiA+IHdheSB0byBzb2x2ZSB0aGlzIHdvdWxkIGJlIHRvIGRvIGEgZ2l0IGJpc2VjdCBi ZXR3ZWVuIHlvdXINCj4gPiA+IGN1cnJlbnQNCj4gPiA+IGtlcm5lbCBhbmQgdGhlIDQuMTEga2Vy bmVsIHRvIGZpbmQgdGhlIHBvaW50IGF0IHdoaWNoIHRoaXMNCj4gPiA+IHN0YXJ0ZWQuDQo+ID4g PiBJZg0KPiA+ID4gd2UgY2FuIGRvIHRoYXQgdGhlbiBmaXhpbmcgdGhpcyBiZWNvbWVzIG11Y2gg c2ltcGxlciBhcyB3ZSBqdXN0DQo+ID4gPiBoYXZlDQo+ID4gPiB0byBmaXggdGhlIHBhdGNoIHRo YXQgaW50cm9kdWNlZCB0aGUgaXNzdWUuDQo+ID4gDQo+ID4gV2UncmUgYWxzbyBzZWVpbmcgYSBz bWFsbGVyIG1lbW9yeSBsZWFrIChhYm91dCAxIEdCIHBlciBkYXkpIHRoYW4NCj4gPiB0aGUNCj4g PiBvcmlnaW5hbCBvbmUgZXZlbiB3aXRoIHRoZSAiRml4IG1lbW9yeSBsZWFrIHJlbGF0ZWQgZmls dGVyDQo+ID4gcHJvZ3JhbW1pbmcNCj4gPiBzdGF0dXMiIGZpeCBhcHBsaWVkLiBTbyBmYXIgSSd2 ZSBkZXRlcm1pbmVkIHRoYXQgdGhlIGxlYWsgaXMNCj4gPiBwcmVzZW50IG9uDQo+ID4gNC4xMy43 IGFuZCB3YXMgaW50cm9kdWNlZCBiZXR3ZWVuIDQuMTEgYW5kIDQuMTIsIHNvIEknbGwgZG8gYW5v dGhlcg0KPiA+IHJvdW5kIG9mIGJpc2VjdGlvbiB0byBpZGVudGlmeSB0aGUgcGF0Y2ggdGhhdCBp bnRyb2R1Y2VkIHRoaXMuDQo+ID4gDQo+ID4gU2luY2UgdGhlIHJvdXRlciBtdXN0IHJ1biBmb3Ig YSBjb3VwbGUgb2YgaG91cnMgYmVmb3JlIEkgY2FuIGJlDQo+ID4gc3VyZQ0KPiA+IHdoZXRoZXIg YSBrZXJuZWwgaXMgZ29vZCBvciBiYWQsIGFuZCBJIGNhbid0IHJlYm9vdCBpdCBkdXJpbmcNCj4g PiB3b3JraW5nDQo+ID4gaG91cnMsIGl0J2xsIHByb2JhYmx5IGJlIGFib3V0IGEgd2VlayBiZWZv cmUgSSBoYXZlIGEgcmVzdWx0Lg0KPiA+IA0KPiA+IC0tDQo+ID4gVmVubGlnIGhpbHNlbiAvIEJl c3QgUmVnYXJkcw0KPiA+IA0KPiA+IEFuZGVycyBLLiBQZWRlcnNlbg0KPiA+IFNlbmlvciBUZWNo bmljYWwgTWFuYWdlcg0KPiANCj4gQW5kZXJzLA0KPiANCj4gSSdsbCBkbyBzb21lIGRpZ2dpbmcg b24gbXkgc2lkZSB0byBzZWUgaWYgSSBjYW4gZmluZCBhbnkgb3RoZXIgbWVtb3J5DQo+IGxlYWtz IHRoYXQgbWlnaHQgYmUgZmxvYXRpbmcgYXJvdW5kIGluIHRoZSBkcml2ZXIgdGhhdCBjb3VsZCBo YXZlDQo+IGJlZW4NCj4gaW50cm9kdWNlZCBkdXJpbmcgdGhhdCB0aW1lLWZyYW1lLg0KPiANCj4g T25lIHRoaW5nIHlvdSBtaWdodCB0cnkgdGhhdCB3b3VsZCBoZWxwIHdpdGggeW91ciB0ZXN0aW5n IHdvdWxkIGJlIHRvDQo+IGp1c3QgZGlzYWJsZSB0aGUgQVRSIGZ1bmN0aW9uYWxpdHkgaW4gaTQw ZS4gWW91IGNhbiBkbyB0aGF0IHdpdGggdGhlDQo+IGV0aHRvb2wgY29tbWFuZCAiZXRodG9vbCAt LXNldC1wcml2LWZsYWdzIDxpZmFjZT4gZmxvdy1kaXJlY3Rvci1hdHINCj4gb2ZmIi4gVGhhdCBz aG91bGQgYWxsb3cgeW91IHRvIGJpc2VjdCB0aGlzIHdpdGhvdXQgbmVlZGluZyB0byBkZWFsDQo+ IHdpdGggdGhlICJwcm9ncmFtbWluZyBzdGF0dXMiIHBhdGNoZXMgc2luY2UgeW91IHdvbid0IGJl IHByb2dyYW1taW5nDQo+IEFUUiBmaWx0ZXJzIHdoaWNoIGlzIHdoYXQgY2F1c2VkIHRoYXQgbGVh ay4NCj4gDQo+IFRoYW5rcyBmb3IgbG9va2luZyBpbnRvIHRoaXMuDQo+IA0KPiAtIEFsZXgNCg0K SGkgQWxleCwNCg0KSSBiZWdhbiBiaXNlY3RpbmcsIHdoZXJlIEkgYXBwbGllZCB0aGUga25vd24g Zml4IHBhdGNoZXMgdG8gdGhlIHN0ZXBzLA0Kd2hlcmUgdGhleSB3ZXJlIGFwcGxpY2FibGUgKGku ZS4gd2l0aG91dCBjaGFuZ2luZyB0aGUgZmxvdy1kaXJlY3Rvci1hdHIgDQpmbGFnKSwgYnV0IHNv bWUgb2YgdGhlIHN0ZXBzIGhhZCBhIGhpZ2ggYW1vdW50IG9mIHBhY2tldCBkcm9wcywgd2hpY2gN CmNhdXNlZCBwcm9ibGVtcyBmb3Igb3VyIG5ldHdvcmssIHNvIEkgY291bGRuJ3QgbGVhdmUgdGhl bSBydW5uaW5nIGZvcg0Kc2V2ZXJhbCBob3Vycywgd2hpY2ggaXMgbmVjZXNzYXJ5IHRvIGRldGVy bWluZSBpZiB0aGUgbGVhayBpcyBwcmVzZW50DQpvciBub3QuIFRoZSBwYXJ0IG9mIHRoZSBiaXNl Y3Rpb24gSSBnb3QgdGhyb3VnaCBoYWQgdGhlIHNhbWUgb3V0Y29tZSBhcw0KdGhlIGxhc3QgYmlz ZWN0aW9uLCB3aGljaCBsZWQgdG8gImk0MGU6IEZpeCBzdXBwb3J0IGZvciBmbG93DQpkaXJlY3Rv ciBwcm9ncmFtbWluZyBzdGF0dXMiLg0KDQpBZnRlciB0aGF0IEkgZXhwZXJpbWVudGVkIGEgYml0 IHdpdGggdGhlIGZsb3ctZGlyZWN0b3ItYXRyIGZsYWcsIGFuZCBpdA0KdHVybnMgb3V0IHRoYXQg aWYgSSBkaXNhYmxlIHRoaXMgZmxhZyBvbiBhbGwgdGhlIE5JQ3MsIHRoZW4gdGhlIG1lbW9yeQ0K bGVhayBpcyBnb25lLCBzbyBJIHN1c3BlY3RlZCB0aGF0IHRoZSBzbWFsbGVyIG1lbW9yeSBsZWFr IHdhcyBhbHNvDQpjYXVzZWQgYnkgImk0MGU6IEZpeCBzdXBwb3J0IGZvciBmbG93IGRpcmVjdG9y IHByb2dyYW1taW5nIHN0YXR1cyIuDQoNCkkgdHJpZWQgdG8gcmV2ZXJ0IHRoaXMgcGF0Y2ggZnJv bSA0LjEzICh3aXRoIG1hbnVhbCBmaXh1cCBmb3IgdGhlIHRyYWNlDQpwb2ludCB0aGF0IGhhZCBi ZWVuIGFkZGVkIGxhdGVyKSwgYnV0IHRoYXQgYnJvdWdodCBiYWNrIHRoZSBwYWNrZXQNCmRyb3Bz LCBzbyBJIGNvdWxkbid0IGxldCBpdCBydW4uDQoNClRoaXMgbW9ybmluZyBJIHNhdyB5b3VyICJp NDBlOiBBZGQgcHJvZ3JhbW1pbmcgZGVzY3JpcHRvcnMgdG8NCmNsZWFuZWRfY291bnQiIHBhdGNo LCBzbyBJIHRyaWVkIDQuMTMuOSB3aXRoIHRoYXQgcGF0Y2ggYW5kIHRoZQ0KcHJldmlvdXMgImk0 MGU6IEZpeCBtZW1vcnkgbGVhayByZWxhdGVkIGZpbHRlciBwcm9ncmFtbWluZyBzdGF0dXMiDQp3 aXRob3V0IHR1cm5pbmcgb2ZmIHRoZSBmbG93LWRpcmVjdG9yLWF0ciBmbGFnLiBTbyBmYXIgdGhp cyBjb21iaW5hdGlvbg0KaXMgcnVubmluZyBzdGFibGUgd2l0aG91dCBhbnkgbWVtb3J5IGxlYWtz Lg0KDQpUaGFua3MgZm9yIGZpeGluZyB0aGlzLg0KDQpSZWdhcmRzLA0KQW5kZXJz