From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id pBLE6Gf9019545 for ; Wed, 21 Dec 2011 08:06:17 -0600 Received: from mail.hq.newdream.net (mail.hq.newdream.net [66.33.206.127]) by cuda.sgi.com with ESMTP id u1qFT9VemtBI7X0M for ; Wed, 21 Dec 2011 06:06:14 -0800 (PST) Message-ID: <1324476371.3399.11.camel@doink> Subject: Re: Kswapd in 3.2.0-rc5 is a CPU hog From: Alex Elder Date: Wed, 21 Dec 2011 08:06:11 -0600 In-Reply-To: <20111221102449.GE27137@tiehlicka.suse.cz> References: <1324437036.4677.5.camel@hakkenden.homenet> <20111221095249.GA28474@tiehlicka.suse.cz> <1324462521.17322.12.camel@rybalov.eng.ttk.net> <20111221102449.GE27137@tiehlicka.suse.cz> Mime-Version: 1.0 Reply-To: elder@dreamhost.com List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Michal Hocko Cc: Alex Elder , linux-mm@kvack.org, linux-kernel@vger.kernel.org, xfs@oss.sgi.com, Christoph Hellwig , nowhere , Ben Myers T24gV2VkLCAyMDExLTEyLTIxIGF0IDExOjI0ICswMTAwLCBNaWNoYWwgSG9ja28gd3JvdGU6Cj4g W0xldCdzIGNjIGFsc28gc29tZSBmcyBhbmQgeGZzIHBlb3BsZV0KPiAKPiBPbiBXZWQgMjEtMTIt MTEgMTQ6MTU6MjEsIG5vd2hlcmUgd3JvdGU6Cj4gPiDQkiDQodGALiwgMjEvMTIvMjAxMSDQsiAx MDo1MiArMDEwMCwgTWljaGFsIEhvY2tvINC/0LjRiNC10YI6Cj4gPiA+IFtMZXQncyBDQyBsaW51 eC1tbV0KPiA+ID4gCj4gPiA+IE9uIFdlZCAyMS0xMi0xMSAwNzoxMDozNiwgTmlrb2xheSBTLiB3 cm90ZToKPiA+ID4gPiBIZWxsbywKPiA+ID4gPiAKPiA+ID4gPiBJJ20gdXNpbmcgMy4yLXJjNSBv biBhIG1hY2hpbmUsIHdoaWNoIGF0bSBkb2VzIGFsbW9zdCBub3RoaW5nIGV4Y2VwdAo+ID4gPiA+ IGZpbGUgc3lzdGVtIG9wZXJhdGlvbnMgYW5kIG5ldHdvcmsgaS9vIChpLmUuIGZpbGUgc2VydmVy KS4gQW5kIHRoZXJlIGlzCj4gPiA+ID4gYSBwcm9ibGVtIHdpdGgga3N3YXBkLgo+ID4gPiAKPiA+ ID4gV2hhdCBraW5kIG9mIGZpbGVzeXN0ZW0gZG8geW91IHVzZT8KPiA+IAo+ID4gV2VsbCwgdGhh dCBpcyBYRlMuCj4gPiBJIGhhdmUgYSBsYXJnZSB2b2x1bWUgd2l0aCB+MjAwMDAwIGZpbGVzLCBh bmQgYSBwZXJpb2RpYyBqb2IsIHdoaWNoCj4gPiBjaGVja3MgYWxsIGZpbGUncyB0aW1lc3RhbXBz IG9uY2UgcGVyIDMwIG1pbnV0ZXMgYW5kIG1ha2VzIGFjdGlvbnMgaWYKPiA+IHRpbWVzdGFtcCBo YXMgY2hhbmdlZC4KPiAKPiBJcyBpdCB0aGUgZmlyc3QgdGltZSB5b3UgYXJlIHNlZWluZyB0aGlz PyBJIGFtIG5vdCBmYW1pbGlhciB3aXRoIHhmcyBhdAo+IGFsbCBidXQgdGhlIG51bWJlciBvZiBm aWxlcyBzb3VuZHMgbGlrZSBkY2FjaGUgc2hyaW5rZXIgbWlnaHQgYmUgcmVhbGx5Cj4gYnVzeS4u Lgo+IAo+ID4gPiA+IEknbSBwbGF5aW5nIHdpdGggZGQ6Cj4gPiA+ID4gZGQgaWY9L3NvbWUvYmln L2ZpbGUgb2Y9L2Rldi9udWxsIGJzPThNCgoKSnVzdCB0byBiZSBjbGVhciwgaXMgaXQgYWx3YXlz IHRoZSBzYW1lIGZpbGUgeW91IGFyZQpvcGVyYXRpbmcgb24gaW4gYWxsIG9mIHRoZSBjYXNlcyB5 b3Ugc2hvdyBoZXJlPyAgSWYKc28sIHRoZXJlIGFueXRoaW5nIGVsc2UgaGFwcGVuaW5nIHRvIHRo YXQgZmlsZSBpbgp0aGUgbWVhbiB0aW1lPyAgSXMgaXQgYmVpbmcgd3JpdHRlbiB0byBhdCBhbGw/ CgpJcyB0aGUgZmlsZXN5c3RlbSBpbiBxdWVzdGlvbiAodGhlIG9uZSB3aXRoIHRoZQpiaWcgZmls ZSkgYWxzbyBiZWluZyBORlMgc2VydmVkPyAgQXJlIHRoZXJlIGFueQpvdGhlciBmaWxlc3lzdGVt cyBiZWluZyBzZXJ2ZWQsIGFuZCBpZiBzbywgYXJlCnRoZXkgYWxzbyBYRlM/CgpJIHByZXN1bWUg eW91IGFyZSBkb2luZyBub3RoaW5nIHNwZWNpYWwgd2l0aCByZXNwZWN0CnRvIHRoZSBmaWxlIHNl cnZpbmcsIGkuZS4gaW4gYWxsIHRoZXNlIGNhc2VzIGFuZAp0aW1pbmdzLCBldGMuIHRoZSBzeXN0 ZW0gaGFzIGJlZW4gYW5kIGNvbnRpbnVlcyB0bwphY3QgYXMgYW4gTkZTIHNlcnZlci4KCklzIHRo ZSBmaWxlc3lzdGVtIGFueXdoZXJlIG5lYXIgZnVsbD8KCkkgZG9uJ3Qga25vdyBpZiB0aGVzZSBh bnN3ZXJzIGNoYW5nZSBtdWNoIGJ1dCBpdAptaWdodCBoZWxwIHRvIGNvbXBsZXRlIHRoZSBwaWN0 dXJlLgoKVGhhbmtzLgoKCQkJCQktQWxleAo+ID4gPiA+IEkuZS4gSSdtIGZpbGxpbmcgcGFnZSBj YWNoZS4KPiA+ID4gPiAKPiA+ID4gPiBTbyB3aGVuIHRoZSBtYWNoaW5lIGlzIGp1c3QgcmVib290 ZWQsIGtzd2FwZCBkdXJpbmcgdGhpcyBvcGVyYXRpb24gaXMKPiA+ID4gPiBhbG1vc3QgaWRsZSwg anVzdCA1LTggcGVyY2VudCBhY2NvcmRpbmcgdG8gdG9wLgo+ID4gPiA+IAo+ID4gPiA+IEFmdGVy IH41IGRheXMgb2YgdXB0aW1lICg1IGRheXMsICAyOjEwKSwgdGhlIHNhbWUgb3BlcmF0aW9uIGRl bWFuZHMgfjcwJQo+ID4gPiA+IGZvciBrc3dhcGQ6Cj4gPiA+ID4gCj4gPiA+ID4gICBQSUQgVVNF UiAgICAgIFMgJUNQVSAlTUVNICAgIFRJTUUrICBTV0FQIENPTU1BTkQKPiA+ID4gPiAgIDQyMCBy b290ICAgICAgUiAgIDcwICAwLjAgIDIyOjA5LjYwICAgIDAga3N3YXBkMAo+ID4gPiA+IDE3NzE3 IG5vd2hlcmUgICBEICAgMjcgIDAuMiAgIDA6MDEuODEgIDEwbSBkZAo+ID4gPiA+IAo+ID4gPiA+ IEluIGZhY3QsIGtzd2FwZCBjcHUgdXNhZ2Ugb24gdGhpcyBvcGVyYXRpb24gc3RlYWRpbHkgaW5j cmVhc2VzIG92ZXIKPiA+ID4gPiB0aW1lLgo+ID4gPiA+IAo+ID4gPiA+IEFsc28gcmVhZCBwZXJm b3JtYW5jZSBkZWdyYWRlcyBvdmVyIHRpbWUuIEFmdGVyIHJlYm9vdDoKPiA+ID4gPiBkZCBpZj0v c29tZS9iaWcvZmlsZSBvZj0vZGV2L251bGwgYnM9OE0KPiA+ID4gPiAxMDE5KzEgcmVjb3JkcyBp bgo+ID4gPiA+IDEwMTkrMSByZWNvcmRzIG91dAo+ID4gPiA+IDg1NTM0OTQwMTggYnl0ZXMgKDgu NiBHQikgY29waWVkLCAxNi4yMTEgcywgNTI4IE1CL3MKPiA+ID4gPiAKPiA+ID4gPiBBZnRlciB+ NSBkYXlzIHVwdGltZToKPiA+ID4gPiBkZCBpZj0vc29tZS9iaWcvZmlsZSBvZj0vZGV2L251bGwg YnM9OE0KPiA+ID4gPiAxMDE5KzEgcmVjb3JkcyBpbgo+ID4gPiA+IDEwMTkrMSByZWNvcmRzIG91 dAo+ID4gPiA+IDg1NTM0OTQwMTggYnl0ZXMgKDguNiBHQikgY29waWVkLCAyOS4wNTA3IHMsIDI5 NCBNQi9zCj4gPiA+ID4gCj4gPiA+ID4gV2hlcmVhcyByYXcgZGlzayBzZXF1ZW50aWFsIHJlYWQg cGVyZm9ybWFuY2Ugc3RheXMgdGhlIHNhbWU6Cj4gPiA+ID4gZGQgaWY9L3NvbWUvYmlnL2ZpbGUg b2Y9L2Rldi9udWxsIGJzPThNIGlmbGFnPWRpcmVjdAo+ID4gPiA+IDEwMTkrMSByZWNvcmRzIGlu Cj4gPiA+ID4gMTAxOSsxIHJlY29yZHMgb3V0Cj4gPiA+ID4gODU1MzQ5NDAxOCBieXRlcyAoOC42 IEdCKSBjb3BpZWQsIDE0LjcyODYgcywgNTgxIE1CL3MKPiA+ID4gPiAKPiA+ID4gPiBBbHNvIGFm dGVyIGRyb3BwaW5nIGNhY2hlcywgc2l0dWF0aW9uIHNvbWVob3cgaW1wcm92ZXMsIGJ1dCBub3Qg dG8gdGhlCj4gPiA+ID4gc3RhdGUgb2YgZnJlc2hseSByZXN0YXJ0ZWQgc3lzdGVtOgo+ID4gPiA+ ICAgUElEIFVTRVIgICAgICBTICVDUFUgJU1FTSAgICBUSU1FKyAgU1dBUCBDT01NQU5ECj4gPiA+ ID4gICA0MjAgcm9vdCAgICAgIFMgICAzOSAgMC4wICAyMzozMS4xNyAgICAwIGtzd2FwZDAKPiA+ ID4gPiAxOTgyOSBub3doZXJlICAgRCAgIDI0ICAwLjIgICAwOjAyLjcyIDc3NjQgZGQKPiA+ID4g PiAKPiA+ID4gPiBwZXJmIHNob3dzOgo+ID4gPiA+IAo+ID4gPiA+ICAgICAzMS4yNCUgIGtzd2Fw ZDAgIFtrZXJuZWwua2FsbHN5bXNdICBba10gX3Jhd19zcGluX2xvY2sKPiA+ID4gPiAgICAgMjYu MTklICBrc3dhcGQwICBba2VybmVsLmthbGxzeW1zXSAgW2tdIHNocmlua19zbGFiCj4gPiA+ID4g ICAgIDE2LjI4JSAga3N3YXBkMCAgW2tlcm5lbC5rYWxsc3ltc10gIFtrXSBwcnVuZV9zdXBlcgo+ ID4gPiA+ICAgICAgNi41NSUgIGtzd2FwZDAgIFtrZXJuZWwua2FsbHN5bXNdICBba10gZ3JhYl9z dXBlcl9wYXNzaXZlCj4gPiA+ID4gICAgICA1LjM1JSAga3N3YXBkMCAgW2tlcm5lbC5rYWxsc3lt c10gIFtrXSBkb3duX3JlYWRfdHJ5bG9jawo+ID4gPiA+ICAgICAgNC4wMyUgIGtzd2FwZDAgIFtr ZXJuZWwua2FsbHN5bXNdICBba10gdXBfcmVhZAo+ID4gPiA+ICAgICAgMi4zMSUgIGtzd2FwZDAg IFtrZXJuZWwua2FsbHN5bXNdICBba10gcHV0X3N1cGVyCj4gPiA+ID4gICAgICAxLjgxJSAga3N3 YXBkMCAgW2tlcm5lbC5rYWxsc3ltc10gIFtrXSBkcm9wX3N1cGVyCj4gPiA+ID4gICAgICAwLjk5 JSAga3N3YXBkMCAgW2tlcm5lbC5rYWxsc3ltc10gIFtrXSBfX3B1dF9zdXBlcgo+ID4gPiA+ICAg ICAgMC4yNSUgIGtzd2FwZDAgIFtrZXJuZWwua2FsbHN5bXNdICBba10gX19pc29sYXRlX2xydV9w YWdlCj4gPiA+ID4gICAgICAwLjIzJSAga3N3YXBkMCAgW2tlcm5lbC5rYWxsc3ltc10gIFtrXSBm cmVlX3BjcHBhZ2VzX2J1bGsKPiA+ID4gPiAgICAgIDAuMTklICBrc3dhcGQwICBbcjgxNjldICAg ICAgICAgICAgW2tdIHJ0bDgxNjlfaW50ZXJydXB0Cj4gPiA+ID4gICAgICAwLjE1JSAga3N3YXBk MCAgW2tlcm5lbC5rYWxsc3ltc10gIFtrXSB0d2FfaW50ZXJydXB0Cj4gPiA+IAo+ID4gPiBRdWl0 ZSBhIGxvdCBvZiB0aW1lIHNwZW50IHNocmlua2luZyBzbGFiIChkY2FjaGUgSSBndWVzcykgYW5k IGEgbG90IG9mCj4gPiA+IHNwaW4gbG9jayBjb250ZW50aW9uLgo+ID4gCj4gPiBUaGlzIGlzIHNs YWJpbmZvLCBzb3J0ZWQgYnkgbnVtIG9iamVjdHM6Cj4gPiAKPiA+IHhmc19pbm9kZSAgICAgICAg IDE5Mjk0MSAxOTMyMDUgICAgOTYwICAgMTcgICAgNCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAw IDogc2xhYmRhdGEgIDExMzY1ICAxMTM2NSAgICAgIDAKPiA+IGRlbnRyeSAgICAgICAgICAgIDEx ODgxOCAxMTg4MTggICAgMTkyICAgMjEgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDog c2xhYmRhdGEgICA1NjU4ICAgNTY1OCAgICAgIDAKPiA+IGttYWxsb2MtMjU2ICAgICAgIDEwNzky MCAxMDc5MjAgICAgMjU2ICAgMTYgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xh YmRhdGEgICA2NzQ1ICAgNjc0NSAgICAgIDAKPiA+IGttYWxsb2MtNjQgICAgICAgICA1OTkxMiAx MDI2NTYgICAgIDY0ICAgNjQgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRh dGEgICAxNjA0ICAgMTYwNCAgICAgIDAKPiA+IHJhZGl4X3RyZWVfbm9kZSAgICAzMDYxOCAgMzM0 NzQgICAgNTY4ICAgMTQgICAgMiA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEg ICAyMzkxICAgMjM5MSAgICAgIDAKPiA+IGttYWxsb2MtOTYgICAgICAgICAyNzA5MiAgNDEyMDIg ICAgIDk2ICAgNDIgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAg OTgxICAgIDk4MSAgICAgIDAKPiA+IGJ1ZmZlcl9oZWFkICAgICAgICAyNDg5MiAgNjM4NDMgICAg MTA0ICAgMzkgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAxNjM3 ICAgMTYzNyAgICAgIDAKPiA+IGttYWxsb2MtMTkyICAgICAgICAyMzMzMiAgMzQ1MDMgICAgMTky ICAgMjEgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAxNjQzICAg MTY0MyAgICAgIDAKPiA+IHN5c2ZzX2Rpcl9jYWNoZSAgICAxNzQ0NCAgMTc0NDQgICAgMTQ0ICAg MjggICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgNjIzICAgIDYy MyAgICAgIDAKPiA+IGFycF9jYWNoZSAgICAgICAgICAxMjg2MyAgMTQ3OTYgICAgMzIwICAgMTIg ICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAxMjMzICAgMTIzMyAg ICAgIDAKPiA+IGttYWxsb2MtNTEyICAgICAgICAxMTA1MSAgMTE0NDAgICAgNTEyICAgMTYgICAg MiA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgNzE1ICAgIDcxNSAgICAg IDAKPiA+IGttYWxsb2MtMTI4ICAgICAgICAxMDYxMSAgMTMxNTIgICAgMTI4ICAgMzIgICAgMSA6 IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgNDExICAgIDQxMSAgICAgIDAK PiA+IGV4dDRfaW5vZGVfY2FjaGUgICAgOTY2MCAgMTgwMTggICAgODgwICAgMTggICAgNCA6IHR1 bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAxMDAxICAgMTAwMSAgICAgIDAKPiA+ IGttYWxsb2MtOCAgICAgICAgICAgODcwNCAgIDg3MDQgICAgICA4ICA1MTIgICAgMSA6IHR1bmFi bGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgIDE3ICAgICAxNyAgICAgIDAKPiA+IGV4 dDRfaW9fcGFnZSAgICAgICAgNjkxMiAgIDY5MTIgICAgIDE2ICAyNTYgICAgMSA6IHR1bmFibGVz ICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgIDI3ICAgICAyNyAgICAgIDAKPiA+IGFub25f dm1hX2NoYWluICAgICAgNjcwMSAgMTA4ODAgICAgIDQ4ICAgODUgICAgMSA6IHR1bmFibGVzICAg IDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgMTI4ICAgIDEyOCAgICAgIDAKPiA+IEFjcGktTmFt ZXNwYWNlICAgICAgNjYxMSAgIDgwNTggICAgIDQwICAxMDIgICAgMSA6IHR1bmFibGVzICAgIDAg ICAgMCAgICAwIDogc2xhYmRhdGEgICAgIDc5ICAgICA3OSAgICAgIDAKPiA+IGZzbm90aWZ5X2V2 ZW50X2hvbGRlciAgIDYyOTAgICA2OTcwICAgICAyNCAgMTcwICAgIDEgOiB0dW5hYmxlcyAgICAw ICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICA0MSAgICAgNDEgICAgICAwCj4gPiBrbWFsbG9jLTEw MjQgICAgICAgIDU4MTMgICA1ODg4ICAgMTAyNCAgIDE2ICAgIDQgOiB0dW5hYmxlcyAgICAwICAg IDAgICAgMCA6IHNsYWJkYXRhICAgIDM2OCAgICAzNjggICAgICAwCj4gPiB2bV9hcmVhX3N0cnVj dCAgICAgIDU2NjQgICA1NjY0ICAgIDE2OCAgIDI0ICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAg ICAgMCA6IHNsYWJkYXRhICAgIDIzNiAgICAyMzYgICAgICAwCj4gPiByZWlzZXJfaW5vZGVfY2Fj aGUgICAzOTkyICAgNTE5OCAgICA3MDQgICAyMyAgICA0IDogdHVuYWJsZXMgICAgMCAgICAwICAg IDAgOiBzbGFiZGF0YSAgICAyMjYgICAgMjI2ICAgICAgMAo+ID4gQWNwaS1QYXJzZUV4dCAgICAg ICAzODA4ICAgMzgwOCAgICAgNzIgICA1NiAgICAxIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAg OiBzbGFiZGF0YSAgICAgNjggICAgIDY4ICAgICAgMAo+ID4ga21hbGxvYy0yMDQ4ICAgICAgICAz NTg3ICAgMzg4OCAgIDIwNDggICAxNiAgICA4IDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBz bGFiZGF0YSAgICAyNDMgICAgMjQzICAgICAgMAo+ID4gcHJvY19pbm9kZV9jYWNoZSAgICAzNDk4 ICAgMzUxMCAgICA2MjQgICAxMyAgICAyIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFi ZGF0YSAgICAyNzAgICAgMjcwICAgICAgMAo+ID4gYW5vbl92bWEgICAgICAgICAgICAzMzgwICAg MzY0MCAgICAgNzIgICA1NiAgICAxIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0 YSAgICAgNjUgICAgIDY1ICAgICAgMAo+ID4ga21hbGxvYy0xNiAgICAgICAgICAzMDcyICAgMzA3 MiAgICAgMTYgIDI1NiAgICAxIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAg ICAgMTIgICAgIDEyICAgICAgMAo+ID4gaW5vZGVfY2FjaGUgICAgICAgICAzMDI0ICAgMzAyNCAg ICA1NjAgICAxNCAgICAyIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAgICAy MTYgICAgMjE2ICAgICAgMAo+ID4gZXh0NF9hbGxvY2F0aW9uX2NvbnRleHQgICAzMDAwICAgMzAw MCAgICAxMzYgICAzMCAgICAxIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAg ICAxMDAgICAgMTAwICAgICAgMAo+ID4gbmZfY29ubnRyYWNrX2ZmZmZmZmZmODE3NzZkNDAgICAy OTEwICAgMzU0OSAgICAzMTIgICAxMyAgICAxIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBz bGFiZGF0YSAgICAyNzMgICAgMjczICAgICAgMAo+ID4ga21hbGxvYy00MDk2ICAgICAgICAyNzky ICAgMzEzNiAgIDQwOTYgICAgOCAgICA4IDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFi ZGF0YSAgICAzOTIgICAgMzkyICAgICAgMAo+ID4gZXh0NF9mcmVlX2RhdGEgICAgICAyNzAxICAg MjcwMSAgICAgNTYgICA3MyAgICAxIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0 YSAgICAgMzcgICAgIDM3ICAgICAgMAo+ID4gcGlkX25hbWVzcGFjZSAgICAgICAyMTMwICAgMjEz MCAgIDIxMTIgICAxNSAgICA4IDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAg ICAxNDIgICAgMTQyICAgICAgMAo+ID4gbXF1ZXVlX2lub2RlX2NhY2hlICAgMjEyNCAgIDIxMjQg ICAgODk2ICAgMTggICAgNCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAg MTE4ICAgIDExOCAgICAgIDAKPiA+IGpiZDJfcmV2b2tlX3JlY29yZCAgIDE2NjQgICAxNjY0ICAg ICAzMiAgMTI4ICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAx MyAgICAgMTMgICAgICAwCj4gPiBrbWFsbG9jLTMyICAgICAgICAgIDE0MzQgICAzMDcyICAgICAz MiAgMTI4ICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAyNCAg ICAgMjQgICAgICAwCj4gPiBzaG1lbV9pbm9kZV9jYWNoZSAgIDE0MDAgICAxNTIxICAgIDYyNCAg IDEzICAgIDIgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgIDExNyAgICAx MTcgICAgICAwCj4gPiB4ZnNfaWxpICAgICAgICAgICAgIDExMzkgICAxMjYwICAgIDIxNiAgIDE4 ICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICA3MCAgICAgNzAg ICAgICAwCj4gPiBuZnNkNF9zdGF0ZWlkcyAgICAgIDExMzEgICAxNDk2ICAgIDEyMCAgIDM0ICAg IDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICA0NCAgICAgNDQgICAg ICAwCj4gPiBpZHJfbGF5ZXJfY2FjaGUgICAgICA5MDAgICAgOTAwICAgIDU0NCAgIDE1ICAgIDIg OiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICA2MCAgICAgNjAgICAgICAw Cj4gPiBqYmQyX2pvdXJuYWxfaGVhZCAgICA3NDIgICAxMDA4ICAgIDExMiAgIDM2ICAgIDEgOiB0 dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAyOCAgICAgMjggICAgICAwCj4g PiBmc25vdGlmeV9ldmVudCAgICAgICA2NDggICAgNjQ4ICAgIDExMiAgIDM2ICAgIDEgOiB0dW5h YmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAxOCAgICAgMTggICAgICAwCj4gPiBz b2NrX2lub2RlX2NhY2hlICAgICA1NzcgICAgNjI0ICAgIDY0MCAgIDEyICAgIDIgOiB0dW5hYmxl cyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICA1MiAgICAgNTIgICAgICAwCj4gPiB0d19z b2NrX1RDUCAgICAgICAgICA1MDQgICAgNTA0ICAgIDE5MiAgIDIxICAgIDEgOiB0dW5hYmxlcyAg ICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAyNCAgICAgMjQgICAgICAwCj4gPiBUQ1AgICAg ICAgICAgICAgICAgICAzNTYgICAgNDE0ICAgMTcyOCAgIDE4ICAgIDggOiB0dW5hYmxlcyAgICAw ICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAyMyAgICAgMjMgICAgICAwCj4gPiBSQVcgICAgICAg ICAgICAgICAgICAzNDIgICAgMzQyICAgIDgzMiAgIDE5ICAgIDQgOiB0dW5hYmxlcyAgICAwICAg IDAgICAgMCA6IHNsYWJkYXRhICAgICAxOCAgICAgMTggICAgICAwCj4gPiBqYmQyX2pvdXJuYWxf aGFuZGxlICAgIDM0MCAgICAzNDAgICAgIDI0ICAxNzAgICAgMSA6IHR1bmFibGVzICAgIDAgICAg MCAgICAwIDogc2xhYmRhdGEgICAgICAyICAgICAgMiAgICAgIDAKPiA+IGJsa2Rldl9yZXF1ZXN0 cyAgICAgIDMyMiAgICAzMjIgICAgMzQ0ICAgMjMgICAgMiA6IHR1bmFibGVzICAgIDAgICAgMCAg ICAwIDogc2xhYmRhdGEgICAgIDE0ICAgICAxNCAgICAgIDAKPiA+IHRhc2tfc3RydWN0ICAgICAg ICAgIDI5MyAgICAzNTcgICAxNTA0ICAgMjEgICAgOCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAw IDogc2xhYmRhdGEgICAgIDE3ICAgICAxNyAgICAgIDAKPiA+IFVEUCAgICAgICAgICAgICAgICAg IDI4NSAgICAyODUgICAgODMyICAgMTkgICAgNCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDog c2xhYmRhdGEgICAgIDE1ICAgICAxNSAgICAgIDAKPiA+IGZpbGVzX2NhY2hlICAgICAgICAgIDI3 NiAgICAyNzYgICAgNzA0ICAgMjMgICAgNCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xh YmRhdGEgICAgIDEyICAgICAxMiAgICAgIDAKPiA+IG5mc2Q0X29wZW5vd25lcnMgICAgIDIyMCAg ICAzMjAgICAgMzkyICAgMjAgICAgMiA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRh dGEgICAgIDE2ICAgICAxNiAgICAgIDAKPiA+IG1tX3N0cnVjdCAgICAgICAgICAgIDIxNiAgICAy MTYgICAgODk2ICAgMTggICAgNCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEg ICAgIDEyICAgICAxMiAgICAgIDAKPiA+IHNpZ2hhbmRfY2FjaGUgICAgICAgIDE5OSAgICAyMjUg ICAyMTEyICAgMTUgICAgOCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAg IDE1ICAgICAxNSAgICAgIDAKPiA+IG5mc2Q0X2RlbGVnYXRpb25zICAgIDE5OCAgICAxOTggICAg MzY4ICAgMjIgICAgMiA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICA5 ICAgICAgOSAgICAgIDAKPiA+IGttZW1fY2FjaGVfbm9kZSAgICAgIDE5MiAgICAxOTIgICAgIDY0 ICAgNjQgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICAzICAg ICAgMyAgICAgIDAKPiA+IHhmc19idWZfaXRlbSAgICAgICAgIDE2MiAgICAxNjIgICAgMjI0ICAg MTggICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICA5ICAgICAg OSAgICAgIDAKPiA+IGlwX2ZpYl90cmllICAgICAgICAgIDE0NiAgICAxNDYgICAgIDU2ICAgNzMg ICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICAyICAgICAgMiAg ICAgIDAKPiA+IGV4dDRfaW9fZW5kICAgICAgICAgIDE0MCAgICAxNTQgICAxMTI4ICAgMTQgICAg NCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgIDExICAgICAxMSAgICAg IDAKPiA+IGRub3RpZnlfbWFyayAgICAgICAgIDEyMCAgICAxMjAgICAgMTM2ICAgMzAgICAgMSA6 IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICA0ICAgICAgNCAgICAgIDAK PiA+IFRDUHY2ICAgICAgICAgICAgICAgIDEwNCAgICAxMTkgICAxODU2ICAgMTcgICAgOCA6IHR1 bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICA3ICAgICAgNyAgICAgIDAKPiA+ IGNmcV9xdWV1ZSAgICAgICAgICAgIDEwMiAgICAxMDIgICAgMjMyICAgMTcgICAgMSA6IHR1bmFi bGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICA2ICAgICAgNiAgICAgIDAKPiA+IEFj cGktU3RhdGUgICAgICAgICAgIDEwMiAgICAxMDIgICAgIDgwICAgNTEgICAgMSA6IHR1bmFibGVz ICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICAyICAgICAgMiAgICAgIDAKPiA+IHNpZ3F1 ZXVlICAgICAgICAgICAgIDEwMCAgICAxMDAgICAgMTYwICAgMjUgICAgMSA6IHR1bmFibGVzICAg IDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICA0ICAgICAgNCAgICAgIDAKPiA+IHhmc19lZmRf aXRlbSAgICAgICAgICA4MCAgICAgODAgICAgNDAwICAgMjAgICAgMiA6IHR1bmFibGVzICAgIDAg ICAgMCAgICAwIDogc2xhYmRhdGEgICAgICA0ICAgICAgNCAgICAgIDAKPiA+IHR3X3NvY2tfVENQ djYgICAgICAgICA2NCAgICAxNDQgICAgMjU2ICAgMTYgICAgMSA6IHR1bmFibGVzICAgIDAgICAg MCAgICAwIDogc2xhYmRhdGEgICAgICA5ICAgICAgOSAgICAgIDAKPiA+IGJkZXZfY2FjaGUgICAg ICAgICAgICA1NyAgICAgNTcgICAgODMyICAgMTkgICAgNCA6IHR1bmFibGVzICAgIDAgICAgMCAg ICAwIDogc2xhYmRhdGEgICAgICAzICAgICAgMyAgICAgIDAKPiA+IGJsa2Rldl9xdWV1ZSAgICAg ICAgICA1NCAgICAgNTQgICAxNzQ0ICAgMTggICAgOCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAw IDogc2xhYmRhdGEgICAgICAzICAgICAgMyAgICAgIDAKPiA+IG5ldF9uYW1lc3BhY2UgICAgICAg ICA1MiAgICAgNTIgICAyNDMyICAgMTMgICAgOCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDog c2xhYmRhdGEgICAgICA0ICAgICAgNCAgICAgIDAKPiA+IGttYWxsb2MtODE5MiAgICAgICAgICA1 MiAgICAgNTIgICA4MTkyICAgIDQgICAgOCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xh YmRhdGEgICAgIDEzICAgICAxMyAgICAgIDAKPiA+IGttZW1fY2FjaGUgICAgICAgICAgICA0MiAg ICAgNDIgICAgMTkyICAgMjEgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRh dGEgICAgICAyICAgICAgMiAgICAgIDAKPiA+IHhmc19sb2dfdGlja2V0ICAgICAgICA0MCAgICAg NDAgICAgMjAwICAgMjAgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEg ICAgICAyICAgICAgMiAgICAgIDAKPiA+IHhmc19idHJlZV9jdXIgICAgICAgICAzOCAgICAgMzgg ICAgMjA4ICAgMTkgICAgMSA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAg ICAyICAgICAgMiAgICAgIDAKPiA+IHJwY19pbm9kZV9jYWNoZSAgICAgICAzOCAgICAgMzggICAg ODMyICAgMTkgICAgNCA6IHR1bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICAy ICAgICAgMiAgICAgIDAKPiA+IG5mX2Nvbm50cmFja19leHBlY3QgICAgIDM0ICAgICAzNCAgICAy NDAgICAxNyAgICAxIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAgICAgIDIg ICAgICAyICAgICAgMAo+ID4geGZzX2RhX3N0YXRlICAgICAgICAgIDMyICAgICAzMiAgICA0ODgg ICAxNiAgICAyIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAgICAgIDIgICAg ICAyICAgICAgMAo+ID4gVURQdjYgICAgICAgICAgICAgICAgIDMyICAgICAzMiAgIDEwMjQgICAx NiAgICA0IDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAgICAgIDIgICAgICAy ICAgICAgMAo+ID4geGZzX3RyYW5zICAgICAgICAgICAgIDI4ICAgICAyOCAgICAyODAgICAxNCAg ICAxIDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAgICAgIDIgICAgICAyICAg ICAgMAo+ID4gdGFza3N0YXRzICAgICAgICAgICAgIDI0ICAgICAyNCAgICAzMjggICAxMiAgICAx IDogdHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAgICAgIDIgICAgICAyICAgICAg MAo+ID4gZGlvICAgICAgICAgICAgICAgICAgIDI0ICAgICAyNCAgICA2NDAgICAxMiAgICAyIDog dHVuYWJsZXMgICAgMCAgICAwICAgIDAgOiBzbGFiZGF0YSAgICAgIDIgICAgICAyICAgICAgMAo+ ID4gcG9zaXhfdGltZXJzX2NhY2hlICAgICAyMyAgICAgMjMgICAgMTc2ICAgMjMgICAgMSA6IHR1 bmFibGVzICAgIDAgICAgMCAgICAwIDogc2xhYmRhdGEgICAgICAxICAgICAgMSAgICAgIDAKPiA+ IGh1Z2V0bGJmc19pbm9kZV9jYWNoZSAgICAgMTQgICAgIDE0ICAgIDU2MCAgIDE0ICAgIDIgOiB0 dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMSAgICAgIDEgICAgICAwCj4g PiB4ZnJtX2RzdF9jYWNoZSAgICAgICAgIDAgICAgICAwICAgIDM4NCAgIDIxICAgIDIgOiB0dW5h YmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiB1 c2VyX25hbWVzcGFjZSAgICAgICAgIDAgICAgICAwICAgMTA3MiAgIDE1ICAgIDQgOiB0dW5hYmxl cyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBVRFBM SVRFdjYgICAgICAgICAgICAgIDAgICAgICAwICAgMTAyNCAgIDE2ICAgIDQgOiB0dW5hYmxlcyAg ICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBVRFAtTGl0 ZSAgICAgICAgICAgICAgIDAgICAgICAwICAgIDgzMiAgIDE5ICAgIDQgOiB0dW5hYmxlcyAgICAw ICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBrY29weWRfam9i ICAgICAgICAgICAgIDAgICAgICAwICAgMzI0MCAgIDEwICAgIDggOiB0dW5hYmxlcyAgICAwICAg IDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBmbG93X2NhY2hlICAg ICAgICAgICAgIDAgICAgICAwICAgIDEwNCAgIDM5ICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAg ICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBleHQyX3hhdHRyICAgICAg ICAgICAgIDAgICAgICAwICAgICA4OCAgIDQ2ICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAg MCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBleHQyX2lub2RlX2NhY2hlICAg ICAgIDAgICAgICAwICAgIDc1MiAgIDIxICAgIDQgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6 IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBkcXVvdCAgICAgICAgICAgICAgICAg IDAgICAgICAwICAgIDI1NiAgIDE2ICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNs YWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBkbV91ZXZlbnQgICAgICAgICAgICAgIDAg ICAgICAwICAgMjYwOCAgIDEyICAgIDggOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJk YXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBkbWEta21hbGxvYy05NiAgICAgICAgIDAgICAg ICAwICAgICA5NiAgIDQyICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRh ICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBkbWEta21hbGxvYy04MTkyICAgICAgIDAgICAgICAw ICAgODE5MiAgICA0ICAgIDggOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAg ICAgMCAgICAgIDAgICAgICAwCj4gPiBkbWEta21hbGxvYy04ICAgICAgICAgIDAgICAgICAwICAg ICAgOCAgNTEyICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAg MCAgICAgIDAgICAgICAwCj4gPiBkbWEta21hbGxvYy02NCAgICAgICAgIDAgICAgICAwICAgICA2 NCAgIDY0ICAgIDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAg ICAgIDAgICAgICAwCj4gPiBkbWEta21hbGxvYy01MTIgICAgICAgIDAgICAgICAwICAgIDUxMiAg IDE2ICAgIDIgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAg IDAgICAgICAwCj4gPiBkbWEta21hbGxvYy00MDk2ICAgICAgIDAgICAgICAwICAgNDA5NiAgICA4 ICAgIDggOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAg ICAgICAwCj4gPiBkbWEta21hbGxvYy0zMiAgICAgICAgIDAgICAgICAwICAgICAzMiAgMTI4ICAg IDEgOiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAg ICAwCj4gPiBkbWEta21hbGxvYy0yNTYgICAgICAgIDAgICAgICAwICAgIDI1NiAgIDE2ICAgIDEg OiB0dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAw Cj4gPiBkbWEta21hbGxvYy0yMDQ4ICAgICAgIDAgICAgICAwICAgMjA0OCAgIDE2ICAgIDggOiB0 dW5hYmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4g PiBkbWEta21hbGxvYy0xOTIgICAgICAgIDAgICAgICAwICAgIDE5MiAgIDIxICAgIDEgOiB0dW5h YmxlcyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBk bWEta21hbGxvYy0xNiAgICAgICAgIDAgICAgICAwICAgICAxNiAgMjU2ICAgIDEgOiB0dW5hYmxl cyAgICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBkbWEt a21hbGxvYy0xMjggICAgICAgIDAgICAgICAwICAgIDEyOCAgIDMyICAgIDEgOiB0dW5hYmxlcyAg ICAwICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBkbWEta21h bGxvYy0xMDI0ICAgICAgIDAgICAgICAwICAgMTAyNCAgIDE2ICAgIDQgOiB0dW5hYmxlcyAgICAw ICAgIDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiBic2dfY21kICAg ICAgICAgICAgICAgIDAgICAgICAwICAgIDMxMiAgIDEzICAgIDEgOiB0dW5hYmxlcyAgICAwICAg IDAgICAgMCA6IHNsYWJkYXRhICAgICAgMCAgICAgIDAgICAgICAwCj4gPiAKPiA+ID4gQ291bGQg eW91IGFsc28gdGFrZSBmZXcgc25hcHNob3RzIG9mIC9wcm9jLzQyMC9zdGFjayB0byBzZWUgd2hh dCBrc3dhcGQKPiA+ID4gaXMgZG9pbmcuCj4gPiAKPiA+IFVobSwgdGhlcmUgaXMgbm8gc3VjaCBl bnRyeSBpbiBwcm9jLiBHdWVzcyBJIG5lZWQgdG8gZW5hYmxlIHNvbWUga2VybmVsCj4gPiBvcHRp b24gYW5kIHJlY29tcGlsZT8KPiAKPiBZZXMsIHlvdSBuZWVkIENPTkZJR19TVEFDS1RSQUNFLiBC dXQgeW91IGNhbiBnZXQgYSBzaW1pbGFyIGluZm9ybWF0aW9uCj4gYnkgc3lzcnErdAo+IAo+ID4g VGhpcyB3aWxsIHJlc2V0IHVwdGltZSBhbmQgYSBwcm9ibGVtIGZvciBhbm90aGVyIDUtMTAgZGF5 cy4uCj4gCj4gVGhlbiBkb24ndCBkbyB0aGF0IDspCj4gCj4gPiA+ID4gUC5TLjogVGhlIG1lc3Nh Z2UgYWJvdmUgd2FzIHdyaXR0ZW4gY291cGxlIG9mIGRheXMgYWdvLiBOb3cgSSdtIGF0IDEwCj4g PiA+ID4gZGF5cyB1cHRpbWUsIGFuZCB0aGlzIGlzIHRoZSByZXN1bHQgYXMgb2YgdG9kYXkKPiA+ ID4gPiAgIFBJRCBVU0VSICAgICAgUyAlQ1BVICVNRU0gICAgVElNRSsgIFNXQVAgQ09NTUFORAo+ ID4gPiA+ICAgNDIwIHJvb3QgICAgICBSICAgOTMgIDAuMCAxMTA6NDguNDggICAgMCBrc3dhcGQw Cj4gPiA+ID4gMzAwODUgbm93aGVyZSAgIEQgICA0MiAgMC4yICAgMDowNC4zNiAgMTBtIGRkCj4g PiA+ID4gCj4gPiA+ID4gUFBTOiBQbGVhc2UgQ0MgbWUuCj4gCgoKCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCnhmcyBtYWlsaW5nIGxpc3QKeGZzQG9zcy5z Z2kuY29tCmh0dHA6Ly9vc3Muc2dpLmNvbS9tYWlsbWFuL2xpc3RpbmZvL3hmcwo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx156.postini.com [74.125.245.156]) by kanga.kvack.org (Postfix) with SMTP id 431A36B004D for ; Wed, 21 Dec 2011 09:06:14 -0500 (EST) Message-ID: <1324476371.3399.11.camel@doink> Subject: Re: Kswapd in 3.2.0-rc5 is a CPU hog From: Alex Elder Reply-To: elder@dreamhost.com Date: Wed, 21 Dec 2011 08:06:11 -0600 In-Reply-To: <20111221102449.GE27137@tiehlicka.suse.cz> References: <1324437036.4677.5.camel@hakkenden.homenet> <20111221095249.GA28474@tiehlicka.suse.cz> <1324462521.17322.12.camel@rybalov.eng.ttk.net> <20111221102449.GE27137@tiehlicka.suse.cz> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: nowhere , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , Dave Chinner , xfs@oss.sgi.com, Ben Myers , Alex Elder On Wed, 2011-12-21 at 11:24 +0100, Michal Hocko wrote: > [Let's cc also some fs and xfs people] >=20 > On Wed 21-12-11 14:15:21, nowhere wrote: > > =D0=92 =D0=A1=D1=80., 21/12/2011 =D0=B2 10:52 +0100, Michal Hocko =D0= =BF=D0=B8=D1=88=D0=B5=D1=82: > > > [Let's CC linux-mm] > > >=20 > > > On Wed 21-12-11 07:10:36, Nikolay S. wrote: > > > > Hello, > > > >=20 > > > > I'm using 3.2-rc5 on a machine, which atm does almost nothing exc= ept > > > > file system operations and network i/o (i.e. file server). And th= ere is > > > > a problem with kswapd. > > >=20 > > > What kind of filesystem do you use? > >=20 > > Well, that is XFS. > > I have a large volume with ~200000 files, and a periodic job, which > > checks all file's timestamps once per 30 minutes and makes actions if > > timestamp has changed. >=20 > Is it the first time you are seeing this? I am not familiar with xfs at > all but the number of files sounds like dcache shrinker might be really > busy... >=20 > > > > I'm playing with dd: > > > > dd if=3D/some/big/file of=3D/dev/null bs=3D8M Just to be clear, is it always the same file you are operating on in all of the cases you show here? If so, there anything else happening to that file in the mean time? Is it being written to at all? Is the filesystem in question (the one with the big file) also being NFS served? Are there any other filesystems being served, and if so, are they also XFS? I presume you are doing nothing special with respect to the file serving, i.e. in all these cases and timings, etc. the system has been and continues to act as an NFS server. Is the filesystem anywhere near full? I don't know if these answers change much but it might help to complete the picture. Thanks. -Alex > > > > I.e. I'm filling page cache. > > > >=20 > > > > So when the machine is just rebooted, kswapd during this operatio= n is > > > > almost idle, just 5-8 percent according to top. > > > >=20 > > > > After ~5 days of uptime (5 days, 2:10), the same operation deman= ds ~70% > > > > for kswapd: > > > >=20 > > > > PID USER S %CPU %MEM TIME+ SWAP COMMAND > > > > 420 root R 70 0.0 22:09.60 0 kswapd0 > > > > 17717 nowhere D 27 0.2 0:01.81 10m dd > > > >=20 > > > > In fact, kswapd cpu usage on this operation steadily increases ov= er > > > > time. > > > >=20 > > > > Also read performance degrades over time. After reboot: > > > > dd if=3D/some/big/file of=3D/dev/null bs=3D8M > > > > 1019+1 records in > > > > 1019+1 records out > > > > 8553494018 bytes (8.6 GB) copied, 16.211 s, 528 MB/s > > > >=20 > > > > After ~5 days uptime: > > > > dd if=3D/some/big/file of=3D/dev/null bs=3D8M > > > > 1019+1 records in > > > > 1019+1 records out > > > > 8553494018 bytes (8.6 GB) copied, 29.0507 s, 294 MB/s > > > >=20 > > > > Whereas raw disk sequential read performance stays the same: > > > > dd if=3D/some/big/file of=3D/dev/null bs=3D8M iflag=3Ddirect > > > > 1019+1 records in > > > > 1019+1 records out > > > > 8553494018 bytes (8.6 GB) copied, 14.7286 s, 581 MB/s > > > >=20 > > > > Also after dropping caches, situation somehow improves, but not t= o the > > > > state of freshly restarted system: > > > > PID USER S %CPU %MEM TIME+ SWAP COMMAND > > > > 420 root S 39 0.0 23:31.17 0 kswapd0 > > > > 19829 nowhere D 24 0.2 0:02.72 7764 dd > > > >=20 > > > > perf shows: > > > >=20 > > > > 31.24% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock > > > > 26.19% kswapd0 [kernel.kallsyms] [k] shrink_slab > > > > 16.28% kswapd0 [kernel.kallsyms] [k] prune_super > > > > 6.55% kswapd0 [kernel.kallsyms] [k] grab_super_passive > > > > 5.35% kswapd0 [kernel.kallsyms] [k] down_read_trylock > > > > 4.03% kswapd0 [kernel.kallsyms] [k] up_read > > > > 2.31% kswapd0 [kernel.kallsyms] [k] put_super > > > > 1.81% kswapd0 [kernel.kallsyms] [k] drop_super > > > > 0.99% kswapd0 [kernel.kallsyms] [k] __put_super > > > > 0.25% kswapd0 [kernel.kallsyms] [k] __isolate_lru_page > > > > 0.23% kswapd0 [kernel.kallsyms] [k] free_pcppages_bulk > > > > 0.19% kswapd0 [r8169] [k] rtl8169_interrupt > > > > 0.15% kswapd0 [kernel.kallsyms] [k] twa_interrupt > > >=20 > > > Quite a lot of time spent shrinking slab (dcache I guess) and a lot= of > > > spin lock contention. > >=20 > > This is slabinfo, sorted by num objects: > >=20 > > xfs_inode 192941 193205 960 17 4 : tunables 0 0= 0 : slabdata 11365 11365 0 > > dentry 118818 118818 192 21 1 : tunables 0 0= 0 : slabdata 5658 5658 0 > > kmalloc-256 107920 107920 256 16 1 : tunables 0 0= 0 : slabdata 6745 6745 0 > > kmalloc-64 59912 102656 64 64 1 : tunables 0 0= 0 : slabdata 1604 1604 0 > > radix_tree_node 30618 33474 568 14 2 : tunables 0 0= 0 : slabdata 2391 2391 0 > > kmalloc-96 27092 41202 96 42 1 : tunables 0 0= 0 : slabdata 981 981 0 > > buffer_head 24892 63843 104 39 1 : tunables 0 0= 0 : slabdata 1637 1637 0 > > kmalloc-192 23332 34503 192 21 1 : tunables 0 0= 0 : slabdata 1643 1643 0 > > sysfs_dir_cache 17444 17444 144 28 1 : tunables 0 0= 0 : slabdata 623 623 0 > > arp_cache 12863 14796 320 12 1 : tunables 0 0= 0 : slabdata 1233 1233 0 > > kmalloc-512 11051 11440 512 16 2 : tunables 0 0= 0 : slabdata 715 715 0 > > kmalloc-128 10611 13152 128 32 1 : tunables 0 0= 0 : slabdata 411 411 0 > > ext4_inode_cache 9660 18018 880 18 4 : tunables 0 0= 0 : slabdata 1001 1001 0 > > kmalloc-8 8704 8704 8 512 1 : tunables 0 0= 0 : slabdata 17 17 0 > > ext4_io_page 6912 6912 16 256 1 : tunables 0 0= 0 : slabdata 27 27 0 > > anon_vma_chain 6701 10880 48 85 1 : tunables 0 0= 0 : slabdata 128 128 0 > > Acpi-Namespace 6611 8058 40 102 1 : tunables 0 0= 0 : slabdata 79 79 0 > > fsnotify_event_holder 6290 6970 24 170 1 : tunables 0 = 0 0 : slabdata 41 41 0 > > kmalloc-1024 5813 5888 1024 16 4 : tunables 0 0= 0 : slabdata 368 368 0 > > vm_area_struct 5664 5664 168 24 1 : tunables 0 0= 0 : slabdata 236 236 0 > > reiser_inode_cache 3992 5198 704 23 4 : tunables 0 = 0 0 : slabdata 226 226 0 > > Acpi-ParseExt 3808 3808 72 56 1 : tunables 0 0= 0 : slabdata 68 68 0 > > kmalloc-2048 3587 3888 2048 16 8 : tunables 0 0= 0 : slabdata 243 243 0 > > proc_inode_cache 3498 3510 624 13 2 : tunables 0 0= 0 : slabdata 270 270 0 > > anon_vma 3380 3640 72 56 1 : tunables 0 0= 0 : slabdata 65 65 0 > > kmalloc-16 3072 3072 16 256 1 : tunables 0 0= 0 : slabdata 12 12 0 > > inode_cache 3024 3024 560 14 2 : tunables 0 0= 0 : slabdata 216 216 0 > > ext4_allocation_context 3000 3000 136 30 1 : tunables = 0 0 0 : slabdata 100 100 0 > > nf_conntrack_ffffffff81776d40 2910 3549 312 13 1 : tunabl= es 0 0 0 : slabdata 273 273 0 > > kmalloc-4096 2792 3136 4096 8 8 : tunables 0 0= 0 : slabdata 392 392 0 > > ext4_free_data 2701 2701 56 73 1 : tunables 0 0= 0 : slabdata 37 37 0 > > pid_namespace 2130 2130 2112 15 8 : tunables 0 0= 0 : slabdata 142 142 0 > > mqueue_inode_cache 2124 2124 896 18 4 : tunables 0 = 0 0 : slabdata 118 118 0 > > jbd2_revoke_record 1664 1664 32 128 1 : tunables 0 = 0 0 : slabdata 13 13 0 > > kmalloc-32 1434 3072 32 128 1 : tunables 0 0= 0 : slabdata 24 24 0 > > shmem_inode_cache 1400 1521 624 13 2 : tunables 0 0= 0 : slabdata 117 117 0 > > xfs_ili 1139 1260 216 18 1 : tunables 0 0= 0 : slabdata 70 70 0 > > nfsd4_stateids 1131 1496 120 34 1 : tunables 0 0= 0 : slabdata 44 44 0 > > idr_layer_cache 900 900 544 15 2 : tunables 0 0= 0 : slabdata 60 60 0 > > jbd2_journal_head 742 1008 112 36 1 : tunables 0 0= 0 : slabdata 28 28 0 > > fsnotify_event 648 648 112 36 1 : tunables 0 0= 0 : slabdata 18 18 0 > > sock_inode_cache 577 624 640 12 2 : tunables 0 0= 0 : slabdata 52 52 0 > > tw_sock_TCP 504 504 192 21 1 : tunables 0 0= 0 : slabdata 24 24 0 > > TCP 356 414 1728 18 8 : tunables 0 0= 0 : slabdata 23 23 0 > > RAW 342 342 832 19 4 : tunables 0 0= 0 : slabdata 18 18 0 > > jbd2_journal_handle 340 340 24 170 1 : tunables 0 = 0 0 : slabdata 2 2 0 > > blkdev_requests 322 322 344 23 2 : tunables 0 0= 0 : slabdata 14 14 0 > > task_struct 293 357 1504 21 8 : tunables 0 0= 0 : slabdata 17 17 0 > > UDP 285 285 832 19 4 : tunables 0 0= 0 : slabdata 15 15 0 > > files_cache 276 276 704 23 4 : tunables 0 0= 0 : slabdata 12 12 0 > > nfsd4_openowners 220 320 392 20 2 : tunables 0 0= 0 : slabdata 16 16 0 > > mm_struct 216 216 896 18 4 : tunables 0 0= 0 : slabdata 12 12 0 > > sighand_cache 199 225 2112 15 8 : tunables 0 0= 0 : slabdata 15 15 0 > > nfsd4_delegations 198 198 368 22 2 : tunables 0 0= 0 : slabdata 9 9 0 > > kmem_cache_node 192 192 64 64 1 : tunables 0 0= 0 : slabdata 3 3 0 > > xfs_buf_item 162 162 224 18 1 : tunables 0 0= 0 : slabdata 9 9 0 > > ip_fib_trie 146 146 56 73 1 : tunables 0 0= 0 : slabdata 2 2 0 > > ext4_io_end 140 154 1128 14 4 : tunables 0 0= 0 : slabdata 11 11 0 > > dnotify_mark 120 120 136 30 1 : tunables 0 0= 0 : slabdata 4 4 0 > > TCPv6 104 119 1856 17 8 : tunables 0 0= 0 : slabdata 7 7 0 > > cfq_queue 102 102 232 17 1 : tunables 0 0= 0 : slabdata 6 6 0 > > Acpi-State 102 102 80 51 1 : tunables 0 0= 0 : slabdata 2 2 0 > > sigqueue 100 100 160 25 1 : tunables 0 0= 0 : slabdata 4 4 0 > > xfs_efd_item 80 80 400 20 2 : tunables 0 0= 0 : slabdata 4 4 0 > > tw_sock_TCPv6 64 144 256 16 1 : tunables 0 0= 0 : slabdata 9 9 0 > > bdev_cache 57 57 832 19 4 : tunables 0 0= 0 : slabdata 3 3 0 > > blkdev_queue 54 54 1744 18 8 : tunables 0 0= 0 : slabdata 3 3 0 > > net_namespace 52 52 2432 13 8 : tunables 0 0= 0 : slabdata 4 4 0 > > kmalloc-8192 52 52 8192 4 8 : tunables 0 0= 0 : slabdata 13 13 0 > > kmem_cache 42 42 192 21 1 : tunables 0 0= 0 : slabdata 2 2 0 > > xfs_log_ticket 40 40 200 20 1 : tunables 0 0= 0 : slabdata 2 2 0 > > xfs_btree_cur 38 38 208 19 1 : tunables 0 0= 0 : slabdata 2 2 0 > > rpc_inode_cache 38 38 832 19 4 : tunables 0 0= 0 : slabdata 2 2 0 > > nf_conntrack_expect 34 34 240 17 1 : tunables 0 = 0 0 : slabdata 2 2 0 > > xfs_da_state 32 32 488 16 2 : tunables 0 0= 0 : slabdata 2 2 0 > > UDPv6 32 32 1024 16 4 : tunables 0 0= 0 : slabdata 2 2 0 > > xfs_trans 28 28 280 14 1 : tunables 0 0= 0 : slabdata 2 2 0 > > taskstats 24 24 328 12 1 : tunables 0 0= 0 : slabdata 2 2 0 > > dio 24 24 640 12 2 : tunables 0 0= 0 : slabdata 2 2 0 > > posix_timers_cache 23 23 176 23 1 : tunables 0 = 0 0 : slabdata 1 1 0 > > hugetlbfs_inode_cache 14 14 560 14 2 : tunables 0 = 0 0 : slabdata 1 1 0 > > xfrm_dst_cache 0 0 384 21 2 : tunables 0 0= 0 : slabdata 0 0 0 > > user_namespace 0 0 1072 15 4 : tunables 0 0= 0 : slabdata 0 0 0 > > UDPLITEv6 0 0 1024 16 4 : tunables 0 0= 0 : slabdata 0 0 0 > > UDP-Lite 0 0 832 19 4 : tunables 0 0= 0 : slabdata 0 0 0 > > kcopyd_job 0 0 3240 10 8 : tunables 0 0= 0 : slabdata 0 0 0 > > flow_cache 0 0 104 39 1 : tunables 0 0= 0 : slabdata 0 0 0 > > ext2_xattr 0 0 88 46 1 : tunables 0 0= 0 : slabdata 0 0 0 > > ext2_inode_cache 0 0 752 21 4 : tunables 0 0= 0 : slabdata 0 0 0 > > dquot 0 0 256 16 1 : tunables 0 0= 0 : slabdata 0 0 0 > > dm_uevent 0 0 2608 12 8 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-96 0 0 96 42 1 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-8 0 0 8 512 1 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-64 0 0 64 64 1 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-512 0 0 512 16 2 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-32 0 0 32 128 1 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-256 0 0 256 16 1 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-192 0 0 192 21 1 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-16 0 0 16 256 1 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-128 0 0 128 32 1 : tunables 0 0= 0 : slabdata 0 0 0 > > dma-kmalloc-1024 0 0 1024 16 4 : tunables 0 0= 0 : slabdata 0 0 0 > > bsg_cmd 0 0 312 13 1 : tunables 0 0= 0 : slabdata 0 0 0 > >=20 > > > Could you also take few snapshots of /proc/420/stack to see what ks= wapd > > > is doing. > >=20 > > Uhm, there is no such entry in proc. Guess I need to enable some kern= el > > option and recompile? >=20 > Yes, you need CONFIG_STACKTRACE. But you can get a similar information > by sysrq+t >=20 > > This will reset uptime and a problem for another 5-10 days.. >=20 > Then don't do that ;) >=20 > > > > P.S.: The message above was written couple of days ago. Now I'm a= t 10 > > > > days uptime, and this is the result as of today > > > > PID USER S %CPU %MEM TIME+ SWAP COMMAND > > > > 420 root R 93 0.0 110:48.48 0 kswapd0 > > > > 30085 nowhere D 42 0.2 0:04.36 10m dd > > > >=20 > > > > PPS: Please CC me. >=20 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752697Ab1LUOGQ (ORCPT ); Wed, 21 Dec 2011 09:06:16 -0500 Received: from mail.hq.newdream.net ([66.33.206.127]:42478 "EHLO mail.hq.newdream.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751689Ab1LUOGN convert rfc822-to-8bit (ORCPT ); Wed, 21 Dec 2011 09:06:13 -0500 DomainKey-Signature: a=rsa-sha1; c=nofws; d=dreamhost.com; h=message-id :subject:from:reply-to:to:cc:date:in-reply-to:references :content-type:mime-version:content-transfer-encoding; q=dns; s= dreamhost.com; b=eismv7T8nzofaMLf7ilNWv1ggxUzzfdJN1jLS/8fbAjYNTG cGDEJf1e759MN5/jJFRb+g/2hOOINVcrtdLSNh5iSOLf5IMxrqAgsrdexLuRpEg+ oZ0Jh0bClmp8IUCqGU3BaAXnETzqgwP6AypgFCY4PbRK2hYc3jeN2HEHARRA= Message-ID: <1324476371.3399.11.camel@doink> Subject: Re: Kswapd in 3.2.0-rc5 is a CPU hog From: Alex Elder Reply-To: elder@dreamhost.com To: Michal Hocko Cc: nowhere , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , Dave Chinner , xfs@oss.sgi.com, Ben Myers , Alex Elder Date: Wed, 21 Dec 2011 08:06:11 -0600 In-Reply-To: <20111221102449.GE27137@tiehlicka.suse.cz> References: <1324437036.4677.5.camel@hakkenden.homenet> <20111221095249.GA28474@tiehlicka.suse.cz> <1324462521.17322.12.camel@rybalov.eng.ttk.net> <20111221102449.GE27137@tiehlicka.suse.cz> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.1- Mime-Version: 1.0 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 2011-12-21 at 11:24 +0100, Michal Hocko wrote: > [Let's cc also some fs and xfs people] > > On Wed 21-12-11 14:15:21, nowhere wrote: > > В Ср., 21/12/2011 в 10:52 +0100, Michal Hocko пишет: > > > [Let's CC linux-mm] > > > > > > On Wed 21-12-11 07:10:36, Nikolay S. wrote: > > > > Hello, > > > > > > > > I'm using 3.2-rc5 on a machine, which atm does almost nothing except > > > > file system operations and network i/o (i.e. file server). And there is > > > > a problem with kswapd. > > > > > > What kind of filesystem do you use? > > > > Well, that is XFS. > > I have a large volume with ~200000 files, and a periodic job, which > > checks all file's timestamps once per 30 minutes and makes actions if > > timestamp has changed. > > Is it the first time you are seeing this? I am not familiar with xfs at > all but the number of files sounds like dcache shrinker might be really > busy... > > > > > I'm playing with dd: > > > > dd if=/some/big/file of=/dev/null bs=8M Just to be clear, is it always the same file you are operating on in all of the cases you show here? If so, there anything else happening to that file in the mean time? Is it being written to at all? Is the filesystem in question (the one with the big file) also being NFS served? Are there any other filesystems being served, and if so, are they also XFS? I presume you are doing nothing special with respect to the file serving, i.e. in all these cases and timings, etc. the system has been and continues to act as an NFS server. Is the filesystem anywhere near full? I don't know if these answers change much but it might help to complete the picture. Thanks. -Alex > > > > I.e. I'm filling page cache. > > > > > > > > So when the machine is just rebooted, kswapd during this operation is > > > > almost idle, just 5-8 percent according to top. > > > > > > > > After ~5 days of uptime (5 days, 2:10), the same operation demands ~70% > > > > for kswapd: > > > > > > > > PID USER S %CPU %MEM TIME+ SWAP COMMAND > > > > 420 root R 70 0.0 22:09.60 0 kswapd0 > > > > 17717 nowhere D 27 0.2 0:01.81 10m dd > > > > > > > > In fact, kswapd cpu usage on this operation steadily increases over > > > > time. > > > > > > > > Also read performance degrades over time. After reboot: > > > > dd if=/some/big/file of=/dev/null bs=8M > > > > 1019+1 records in > > > > 1019+1 records out > > > > 8553494018 bytes (8.6 GB) copied, 16.211 s, 528 MB/s > > > > > > > > After ~5 days uptime: > > > > dd if=/some/big/file of=/dev/null bs=8M > > > > 1019+1 records in > > > > 1019+1 records out > > > > 8553494018 bytes (8.6 GB) copied, 29.0507 s, 294 MB/s > > > > > > > > Whereas raw disk sequential read performance stays the same: > > > > dd if=/some/big/file of=/dev/null bs=8M iflag=direct > > > > 1019+1 records in > > > > 1019+1 records out > > > > 8553494018 bytes (8.6 GB) copied, 14.7286 s, 581 MB/s > > > > > > > > Also after dropping caches, situation somehow improves, but not to the > > > > state of freshly restarted system: > > > > PID USER S %CPU %MEM TIME+ SWAP COMMAND > > > > 420 root S 39 0.0 23:31.17 0 kswapd0 > > > > 19829 nowhere D 24 0.2 0:02.72 7764 dd > > > > > > > > perf shows: > > > > > > > > 31.24% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock > > > > 26.19% kswapd0 [kernel.kallsyms] [k] shrink_slab > > > > 16.28% kswapd0 [kernel.kallsyms] [k] prune_super > > > > 6.55% kswapd0 [kernel.kallsyms] [k] grab_super_passive > > > > 5.35% kswapd0 [kernel.kallsyms] [k] down_read_trylock > > > > 4.03% kswapd0 [kernel.kallsyms] [k] up_read > > > > 2.31% kswapd0 [kernel.kallsyms] [k] put_super > > > > 1.81% kswapd0 [kernel.kallsyms] [k] drop_super > > > > 0.99% kswapd0 [kernel.kallsyms] [k] __put_super > > > > 0.25% kswapd0 [kernel.kallsyms] [k] __isolate_lru_page > > > > 0.23% kswapd0 [kernel.kallsyms] [k] free_pcppages_bulk > > > > 0.19% kswapd0 [r8169] [k] rtl8169_interrupt > > > > 0.15% kswapd0 [kernel.kallsyms] [k] twa_interrupt > > > > > > Quite a lot of time spent shrinking slab (dcache I guess) and a lot of > > > spin lock contention. > > > > This is slabinfo, sorted by num objects: > > > > xfs_inode 192941 193205 960 17 4 : tunables 0 0 0 : slabdata 11365 11365 0 > > dentry 118818 118818 192 21 1 : tunables 0 0 0 : slabdata 5658 5658 0 > > kmalloc-256 107920 107920 256 16 1 : tunables 0 0 0 : slabdata 6745 6745 0 > > kmalloc-64 59912 102656 64 64 1 : tunables 0 0 0 : slabdata 1604 1604 0 > > radix_tree_node 30618 33474 568 14 2 : tunables 0 0 0 : slabdata 2391 2391 0 > > kmalloc-96 27092 41202 96 42 1 : tunables 0 0 0 : slabdata 981 981 0 > > buffer_head 24892 63843 104 39 1 : tunables 0 0 0 : slabdata 1637 1637 0 > > kmalloc-192 23332 34503 192 21 1 : tunables 0 0 0 : slabdata 1643 1643 0 > > sysfs_dir_cache 17444 17444 144 28 1 : tunables 0 0 0 : slabdata 623 623 0 > > arp_cache 12863 14796 320 12 1 : tunables 0 0 0 : slabdata 1233 1233 0 > > kmalloc-512 11051 11440 512 16 2 : tunables 0 0 0 : slabdata 715 715 0 > > kmalloc-128 10611 13152 128 32 1 : tunables 0 0 0 : slabdata 411 411 0 > > ext4_inode_cache 9660 18018 880 18 4 : tunables 0 0 0 : slabdata 1001 1001 0 > > kmalloc-8 8704 8704 8 512 1 : tunables 0 0 0 : slabdata 17 17 0 > > ext4_io_page 6912 6912 16 256 1 : tunables 0 0 0 : slabdata 27 27 0 > > anon_vma_chain 6701 10880 48 85 1 : tunables 0 0 0 : slabdata 128 128 0 > > Acpi-Namespace 6611 8058 40 102 1 : tunables 0 0 0 : slabdata 79 79 0 > > fsnotify_event_holder 6290 6970 24 170 1 : tunables 0 0 0 : slabdata 41 41 0 > > kmalloc-1024 5813 5888 1024 16 4 : tunables 0 0 0 : slabdata 368 368 0 > > vm_area_struct 5664 5664 168 24 1 : tunables 0 0 0 : slabdata 236 236 0 > > reiser_inode_cache 3992 5198 704 23 4 : tunables 0 0 0 : slabdata 226 226 0 > > Acpi-ParseExt 3808 3808 72 56 1 : tunables 0 0 0 : slabdata 68 68 0 > > kmalloc-2048 3587 3888 2048 16 8 : tunables 0 0 0 : slabdata 243 243 0 > > proc_inode_cache 3498 3510 624 13 2 : tunables 0 0 0 : slabdata 270 270 0 > > anon_vma 3380 3640 72 56 1 : tunables 0 0 0 : slabdata 65 65 0 > > kmalloc-16 3072 3072 16 256 1 : tunables 0 0 0 : slabdata 12 12 0 > > inode_cache 3024 3024 560 14 2 : tunables 0 0 0 : slabdata 216 216 0 > > ext4_allocation_context 3000 3000 136 30 1 : tunables 0 0 0 : slabdata 100 100 0 > > nf_conntrack_ffffffff81776d40 2910 3549 312 13 1 : tunables 0 0 0 : slabdata 273 273 0 > > kmalloc-4096 2792 3136 4096 8 8 : tunables 0 0 0 : slabdata 392 392 0 > > ext4_free_data 2701 2701 56 73 1 : tunables 0 0 0 : slabdata 37 37 0 > > pid_namespace 2130 2130 2112 15 8 : tunables 0 0 0 : slabdata 142 142 0 > > mqueue_inode_cache 2124 2124 896 18 4 : tunables 0 0 0 : slabdata 118 118 0 > > jbd2_revoke_record 1664 1664 32 128 1 : tunables 0 0 0 : slabdata 13 13 0 > > kmalloc-32 1434 3072 32 128 1 : tunables 0 0 0 : slabdata 24 24 0 > > shmem_inode_cache 1400 1521 624 13 2 : tunables 0 0 0 : slabdata 117 117 0 > > xfs_ili 1139 1260 216 18 1 : tunables 0 0 0 : slabdata 70 70 0 > > nfsd4_stateids 1131 1496 120 34 1 : tunables 0 0 0 : slabdata 44 44 0 > > idr_layer_cache 900 900 544 15 2 : tunables 0 0 0 : slabdata 60 60 0 > > jbd2_journal_head 742 1008 112 36 1 : tunables 0 0 0 : slabdata 28 28 0 > > fsnotify_event 648 648 112 36 1 : tunables 0 0 0 : slabdata 18 18 0 > > sock_inode_cache 577 624 640 12 2 : tunables 0 0 0 : slabdata 52 52 0 > > tw_sock_TCP 504 504 192 21 1 : tunables 0 0 0 : slabdata 24 24 0 > > TCP 356 414 1728 18 8 : tunables 0 0 0 : slabdata 23 23 0 > > RAW 342 342 832 19 4 : tunables 0 0 0 : slabdata 18 18 0 > > jbd2_journal_handle 340 340 24 170 1 : tunables 0 0 0 : slabdata 2 2 0 > > blkdev_requests 322 322 344 23 2 : tunables 0 0 0 : slabdata 14 14 0 > > task_struct 293 357 1504 21 8 : tunables 0 0 0 : slabdata 17 17 0 > > UDP 285 285 832 19 4 : tunables 0 0 0 : slabdata 15 15 0 > > files_cache 276 276 704 23 4 : tunables 0 0 0 : slabdata 12 12 0 > > nfsd4_openowners 220 320 392 20 2 : tunables 0 0 0 : slabdata 16 16 0 > > mm_struct 216 216 896 18 4 : tunables 0 0 0 : slabdata 12 12 0 > > sighand_cache 199 225 2112 15 8 : tunables 0 0 0 : slabdata 15 15 0 > > nfsd4_delegations 198 198 368 22 2 : tunables 0 0 0 : slabdata 9 9 0 > > kmem_cache_node 192 192 64 64 1 : tunables 0 0 0 : slabdata 3 3 0 > > xfs_buf_item 162 162 224 18 1 : tunables 0 0 0 : slabdata 9 9 0 > > ip_fib_trie 146 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 > > ext4_io_end 140 154 1128 14 4 : tunables 0 0 0 : slabdata 11 11 0 > > dnotify_mark 120 120 136 30 1 : tunables 0 0 0 : slabdata 4 4 0 > > TCPv6 104 119 1856 17 8 : tunables 0 0 0 : slabdata 7 7 0 > > cfq_queue 102 102 232 17 1 : tunables 0 0 0 : slabdata 6 6 0 > > Acpi-State 102 102 80 51 1 : tunables 0 0 0 : slabdata 2 2 0 > > sigqueue 100 100 160 25 1 : tunables 0 0 0 : slabdata 4 4 0 > > xfs_efd_item 80 80 400 20 2 : tunables 0 0 0 : slabdata 4 4 0 > > tw_sock_TCPv6 64 144 256 16 1 : tunables 0 0 0 : slabdata 9 9 0 > > bdev_cache 57 57 832 19 4 : tunables 0 0 0 : slabdata 3 3 0 > > blkdev_queue 54 54 1744 18 8 : tunables 0 0 0 : slabdata 3 3 0 > > net_namespace 52 52 2432 13 8 : tunables 0 0 0 : slabdata 4 4 0 > > kmalloc-8192 52 52 8192 4 8 : tunables 0 0 0 : slabdata 13 13 0 > > kmem_cache 42 42 192 21 1 : tunables 0 0 0 : slabdata 2 2 0 > > xfs_log_ticket 40 40 200 20 1 : tunables 0 0 0 : slabdata 2 2 0 > > xfs_btree_cur 38 38 208 19 1 : tunables 0 0 0 : slabdata 2 2 0 > > rpc_inode_cache 38 38 832 19 4 : tunables 0 0 0 : slabdata 2 2 0 > > nf_conntrack_expect 34 34 240 17 1 : tunables 0 0 0 : slabdata 2 2 0 > > xfs_da_state 32 32 488 16 2 : tunables 0 0 0 : slabdata 2 2 0 > > UDPv6 32 32 1024 16 4 : tunables 0 0 0 : slabdata 2 2 0 > > xfs_trans 28 28 280 14 1 : tunables 0 0 0 : slabdata 2 2 0 > > taskstats 24 24 328 12 1 : tunables 0 0 0 : slabdata 2 2 0 > > dio 24 24 640 12 2 : tunables 0 0 0 : slabdata 2 2 0 > > posix_timers_cache 23 23 176 23 1 : tunables 0 0 0 : slabdata 1 1 0 > > hugetlbfs_inode_cache 14 14 560 14 2 : tunables 0 0 0 : slabdata 1 1 0 > > xfrm_dst_cache 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 > > user_namespace 0 0 1072 15 4 : tunables 0 0 0 : slabdata 0 0 0 > > UDPLITEv6 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 > > UDP-Lite 0 0 832 19 4 : tunables 0 0 0 : slabdata 0 0 0 > > kcopyd_job 0 0 3240 10 8 : tunables 0 0 0 : slabdata 0 0 0 > > flow_cache 0 0 104 39 1 : tunables 0 0 0 : slabdata 0 0 0 > > ext2_xattr 0 0 88 46 1 : tunables 0 0 0 : slabdata 0 0 0 > > ext2_inode_cache 0 0 752 21 4 : tunables 0 0 0 : slabdata 0 0 0 > > dquot 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 > > dm_uevent 0 0 2608 12 8 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-512 0 0 512 16 2 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-256 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-192 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 > > dma-kmalloc-1024 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 > > bsg_cmd 0 0 312 13 1 : tunables 0 0 0 : slabdata 0 0 0 > > > > > Could you also take few snapshots of /proc/420/stack to see what kswapd > > > is doing. > > > > Uhm, there is no such entry in proc. Guess I need to enable some kernel > > option and recompile? > > Yes, you need CONFIG_STACKTRACE. But you can get a similar information > by sysrq+t > > > This will reset uptime and a problem for another 5-10 days.. > > Then don't do that ;) > > > > > P.S.: The message above was written couple of days ago. Now I'm at 10 > > > > days uptime, and this is the result as of today > > > > PID USER S %CPU %MEM TIME+ SWAP COMMAND > > > > 420 root R 93 0.0 110:48.48 0 kswapd0 > > > > 30085 nowhere D 42 0.2 0:04.36 10m dd > > > > > > > > PPS: Please CC me. >