From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id p5L59LgF229629 for ; Tue, 21 Jun 2011 00:09:21 -0500 Received: from ipmail06.adl6.internode.on.net (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 344A11DCC2 for ; Mon, 20 Jun 2011 22:09:18 -0700 (PDT) Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id MmdaiZ77NyWELJ4g for ; Mon, 20 Jun 2011 22:09:18 -0700 (PDT) Date: Tue, 21 Jun 2011 15:09:14 +1000 From: Dave Chinner Subject: Re: [PATCH 03/12] vmscan: reduce wind up shrinker->nr when shrinker can't do work Message-ID: <20110621050914.GO32466@dastard> References: <1306998067-27659-1-git-send-email-david@fromorbit.com> <1306998067-27659-4-git-send-email-david@fromorbit.com> <4DFE997C.2060805@jp.fujitsu.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <4DFE997C.2060805@jp.fujitsu.com> List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: KOSAKI Motohiro Cc: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, xfs@oss.sgi.com T24gTW9uLCBKdW4gMjAsIDIwMTEgYXQgMDk6NTE6MDhBTSArMDkwMCwgS09TQUtJIE1vdG9oaXJv IHdyb3RlOgo+ICgyMDExLzA2LzAyIDE2OjAwKSwgRGF2ZSBDaGlubmVyIHdyb3RlOgo+ID4gRnJv bTogRGF2ZSBDaGlubmVyIDxkY2hpbm5lckByZWRoYXQuY29tPgo+ID4gCj4gPiBXaGVuIGEgc2hy aW5rZXIgcmV0dXJucyAtMSB0byBzaHJpbmtfc2xhYigpIHRvIGluZGljYXRlIGl0IGNhbm5vdCBk bwo+ID4gYW55IHdvcmsgZ2l2ZW4gdGhlIGN1cnJlbnQgbWVtb3J5IHJlY2xhaW0gcmVxdWlyZW1l bnRzLCBpdCBhZGRzIHRoZQo+ID4gZW50aXJlIHRvdGFsX3NjYW4gY291bnQgdG8gc2hyaW5rZXIt Pm5yLiBUaGUgaWRlYSBlaGluZCB0aGlzIGlzIHRoYXQKPiA+IHdoZW50ZWggc2hyaW5rZXIgaXMg bmV4dCBjYWxsZWQgYW5kIGNhbiBkbyB3b3JrLCBpdCB3aWxsIGRvIHRoZSB3b3JrCj4gPiBvZiB0 aGUgcHJldmlvdXNseSBhYm9ydGVkIHNocmlua2VyIGNhbGwgYXMgd2VsbC4KPiA+IAo+ID4gSG93 ZXZlciwgaWYgYSBmaWxlc3lzdGVtIGlzIGRvaW5nIGxvdHMgb2YgYWxsb2NhdGlvbiB3aXRoIEdG UF9OT0ZTCj4gPiBzZXQsIHRoZW4gd2UgZ2V0IG1hbnksIG1hbnkgbW9yZSBhYm9ydHMgZnJvbSB0 aGUgc2hyaW5rZXJzIHRoYW4gd2UKPiA+IGRvIHN1Y2Nlc3NmdWwgY2FsbHMuIFRoZSByZXN1bHQg aXMgdGhhdCBzaHJpbmtlci0+bnIgd2luZHMgdXAgdG8KPiA+IGl0J3MgbWF4aW11bSBwZXJtaXNz aWJsZSB2YWx1ZSAodHdpY2UgdGhlIGN1cnJlbnQgY2FjaGUgc2l6ZSkgYW5kCj4gPiB0aGVuIHdo ZW4gdGhlIG5leHQgc2hyaW5rZXIgY2FsbCB0aGF0IGNhbiBkbyB3b3JrIGlzIGlzc3VlZCwgaXQK PiA+IGhhcyBlbm91Z2ggc2NhbiBjb3VudCBidWlsdCB1cCB0byBmcmVlIHRoZSBlbnRpcmUgY2Fj aGUgdHdpY2Ugb3Zlci4KPiA+IAo+ID4gVGhpcyBtYW5pZmVzdHMgaXRzZWxmIGluIHRoZSBjYWNo ZSBnb2luZyBmcm9tIGZ1bGwgdG8gZW1wdHkgaW4gYQo+ID4gbWF0dGVyIG9mIHNlY29uZHMsIGV2 ZW4gd2hlbiBvbmx5IGEgc21hbGwgcGFydCBvZiB0aGUgY2FjaGUgaXMKPiA+IG5lZWRlZCB0byBi ZSBlbXB0aWVkIHRvIGZyZWUgc3VmZmljaWVudCBtZW1vcnkuCj4gPiAKPiA+IFVuZGVyIG1ldGFk YXRhIGludGVuc2l2ZSB3b3JrbG9hZHMgb24gZXh0NCBhbmQgWEZTLCBJJ20gc2VlaW5nIHRoZQo+ ID4gVkZTIGNhY2hlcyBpbmNyZWFzZSBtZW1vcnkgY29uc3VtcHRpb24gdXAgdG8gNzUlIG9mIG1l bW9yeSAobm8gcGFnZQo+ID4gY2FjaGUgcHJlc3N1cmUpIG92ZXIgYSBwZXJpb2Qgb2YgMzAtNjBz LCBhbmQgdGhlbiB0aGUgc2hyaW5rZXIKPiA+IGVtcHRpZXMgdGhlbSBkb3duIHRvIHplcm8gaW4g dGhlIHNwYWNlIG9mIDItM3MuIFRoaXMgY3ljbGUgcmVwZWF0cwo+ID4gb3ZlciBhbmQgb3ZlciBh Z2Fpbiwgd2l0aCB0aGUgc2hyaW5rZXIgY29tcGxldGVseSB0cmFzaGluZyB0aGUg0ZZub2RlCj4g PiBhbmQgZGVudHJ5IGNhY2hlcyBldmVyeSBtaW51dGUgb3Igc28gdGhlIHdvcmtsb2FkIGNvbnRp bnVlcy4KPiA+IAo+ID4gVGhpcyBiZWhhdmlvdXIgd2FzIG1hZGUgb2J2aW91cyBieSB0aGUgc2hy aW5rX3NsYWIgdHJhY2Vwb2ludHMgYWRkZWQKPiA+IGVhcmxpZXIgaW4gdGhlIHNlcmllcywgYW5k IG1hZGUgd29yc2UgYnkgdGhlIHBhdGNoIHRoYXQgY29ycmVjdGVkCj4gPiB0aGUgY29uY3VycmVu dCBhY2NvdW50aW5nIG9mIHNocmlua2VyLT5uci4KPiA+IAo+ID4gVG8gYXZvaWQgdGhpcyBwcm9i bGVtLCBzdG9wIHJlcGVhdGVkIHNtYWxsIGluY3JlbWVudHMgb2YgdGhlIHRvdGFsCj4gPiBzY2Fu IHZhbHVlIGZyb20gd2luZGluZyBzaHJpbmtlci0+bnIgdXAgdG8gYSB2YWx1ZSB0aGF0IGNhbiBj YXVzZQo+ID4gdGhlIGVudGlyZSBjYWNoZSB0byBiZSBmcmVlZC4gV2Ugc3RpbGwgbmVlZCB0byBh bGxvdyBpdCB0byB3aW5kIHVwLAo+ID4gc28gdXNlIHRoZSBkZWx0YSBhcyB0aGUgImxhcmdlIHNj YW4iIHRocmVzaG9sZCBjaGVjayAtIGlmIHRoZSBkZWx0YQo+ID4gaXMgbW9yZSB0aGFuIGEgcXVh cnRlciBvZiB0aGUgZW50aXJlIGNhY2hlIHNpemUsIHRoZW4gaXQgaXMgYSBsYXJnZQo+ID4gc2Nh biBhbmQgYWxsb3dlZCB0byBjYXVzZSBsb3RzIG9mIHdpbmR1cCBiZWNhdXNlIHdlIGFyZSBjbGVh cmx5Cj4gPiBuZWVkaW5nIHRvIGZyZWUgbG90cyBvZiBtZW1vcnkuCj4gPiAKPiA+IElmIGl0IGlz bid0IGEgbGFyZ2Ugc2NhbiB0aGVuIGxpbWl0IHRoZSB0b3RhbCBzY2FuIHRvIGhhbGYgdGhlIHNp emUKPiA+IG9mIHRoZSBjYWNoZSBzbyB0aGF0IHdpbmR1cCBuZXZlciBpbmNyZWFzZXMgdG8gY29u c3VtZSB0aGUgd2hvbGUKPiA+IGNhY2hlLiBSZWR1Y2luZyB0aGUgdG90YWwgc2NhbiBsaW1pdCBm dXJ0aGVyIGRvZXMgbm90IGFsbG93IGVub3VnaAo+ID4gd2luZC11cCB0byBtYWludGFpbiB0aGUg Y3VycmVudCBsZXZlbHMgb2YgcGVyZm9ybWFuY2UsIHdoaWxzdCBhCj4gPiBoaWdoZXIgdGhyZXNo b2xkIGRvZXMgbm90IHByZXZlbnQgdGhlIHdpbmR1cCBmcm9tIGZyZWVpbmcgdGhlIGVudGlyZQo+ ID4gY2FjaGUgdW5kZXIgc3VzdGFpbmVkIHdvcmtsb2Fkcy4KPiA+IAo+ID4gU2lnbmVkLW9mZi1i eTogRGF2ZSBDaGlubmVyIDxkY2hpbm5lckByZWRoYXQuY29tPgo+ID4gLS0tCj4gPiAgbW0vdm1z Y2FuLmMgfCAgIDE0ICsrKysrKysrKysrKysrCj4gPiAgMSBmaWxlcyBjaGFuZ2VkLCAxNCBpbnNl cnRpb25zKCspLCAwIGRlbGV0aW9ucygtKQo+ID4gCj4gPiBkaWZmIC0tZ2l0IGEvbW0vdm1zY2Fu LmMgYi9tbS92bXNjYW4uYwo+ID4gaW5kZXggZGNlMjc2Ny4uMzY4OGY0NyAxMDA2NDQKPiA+IC0t LSBhL21tL3Ztc2Nhbi5jCj4gPiArKysgYi9tbS92bXNjYW4uYwo+ID4gQEAgLTI3Nyw2ICsyNzcs MjAgQEAgdW5zaWduZWQgbG9uZyBzaHJpbmtfc2xhYihzdHJ1Y3Qgc2hyaW5rX2NvbnRyb2wgKnNo cmluaywKPiA+ICAJCX0KPiA+ICAKPiA+ICAJCS8qCj4gPiArCQkgKiBBdm9pZCBleGNlc3NpdmUg d2luZHVwIG9uIGZpZWxzeXN0ZW0gc2hyaW5rZXJzIGR1ZSB0byBsYXJnZQo+ID4gKwkJICogbnVt YmVycyBvZiBHRlBfTk9GUyBhbGxvY2F0aW9ucyBjYXVzaW5nIHRoZSBzaHJpbmtlcnMgdG8KPiA+ ICsJCSAqIHJldHVybiAtMSBhbGwgdGhlIHRpbWUuIFRoaXMgcmVzdWx0cyBpbiBhIGxhcmdlIG5y IGJlaW5nCj4gPiArCQkgKiBidWlsdCB1cCBzbyB3aGVuIGEgc2hyaW5rIHRoYXQgY2FuIGRvIHNv bWUgd29yayBjb21lcyBhbG9uZwo+ID4gKwkJICogaXQgZW1wdGllcyB0aGUgZW50aXJlIGNhY2hl IGR1ZSB0byBuciA+Pj4gbWF4X3Bhc3MuICBUaGlzIGlzCj4gPiArCQkgKiBiYWQgZm9yIHN1c3Rh aW5pbmcgYSB3b3JraW5nIHNldCBpbiBtZW1vcnkuCj4gPiArCQkgKgo+ID4gKwkJICogSGVuY2Ug b25seSBhbGxvdyBuciB0byBnbyBsYXJnZSB3aGVuIGEgbGFyZ2UgZGVsdGEgaXMKPiA+ICsJCSAq IGNhbGN1bGF0ZWQuCj4gPiArCQkgKi8KPiA+ICsJCWlmIChkZWx0YSA8IG1heF9wYXNzIC8gNCkK PiA+ICsJCQl0b3RhbF9zY2FuID0gbWluKHRvdGFsX3NjYW4sIG1heF9wYXNzIC8gMik7Cj4gPiAr Cj4gPiArCQkvKgo+ID4gIAkJICogQXZvaWQgcmlza2luZyBsb29waW5nIGZvcmV2ZXIgZHVlIHRv IHRvbyBsYXJnZSBuciB2YWx1ZToKPiA+ICAJCSAqIG5ldmVyIHRyeSB0byBmcmVlIG1vcmUgdGhh biB0d2ljZSB0aGUgZXN0aW1hdGUgbnVtYmVyIG9mCj4gPiAgCQkgKiBmcmVlYWJsZSBlbnRyaWVz Lgo+IAo+IEkgZ3Vlc3MgIm1heF9wYXNzLzQiIGFuZCAibWluKHRvdGFsX3NjYW4sIG1heF9wYXNz IC8gMikiIGFyZSB5b3VyIGhldXJpc3RpYyB2YWx1ZS4gcmlnaHQ/CgpZZXMuCgo+IElmIHNvLCBw bGVhc2Ugd3JpdGUgeW91ciBiZW5jaG1hcmsgbmFtZSBhbmQgaXRzIHJlc3VsdCBpbnRvIHRoZSBk ZXNjcmlwdGlvbi4KClRha2UgeW91ciBwaWNrLgoKQW55dGhpbmcgdGhhdCBnZW5lcmF0ZXMgYSBs YXJnZSBhbW91bnQgb2YgdmZzIGNhY2hlIHByZXNzdXJlCmNvbWJpbmVkIHdpdGggR0ZQX05PRlMg bWVtb3J5IGFsbG9jYXRpb24gd2lsbCBzaG93IGNoYW5nZXMgaW4KLWJlaGF2aW91ci0gYXMgeW91 IG1vZGlmeSB0aGVzZSB2YXJpYWJsZXMuICBlLmcuIGNyZWF0aW5nIDUwIG1pbGxpb24KaW5vZGVz IGluIHBhcmFsbGVsIHdpdGggZnNfbWFyaywgcGFyYWxsZWwgY2htb2QgLVIgdHJhdmVyc2FscyBv ZgpzYWlkIGlub2RlcywgcGFyYWxsZWwgcm0gLXJmLCBldGMuIFRoZSBzYW1lIGNhY2hlIGJlaGF2 aW91ciBjYW4gYmUKb2JzZXJ2ZWQgd2l0aCBhbnkgb2YgdGhlc2Ugc29ydHMgb2YgY29sZCBjYWNo ZSB3b3JrbG9hZHMuCgpJIHNheSBjaGFuZ2VzIGluIGJlaGF2aW91ciByYXRoZXIgdGhhbiBwZXJm b3JtYW5jZSBiZWNhdXNlIGp1c3QKbWVhc3VyaW5nIHdhbGwgdGltZSBkb2VzIG5vdCBuZWNlc3Nh cmlseSBzaG93IGFueSBkaWZmZXJlbmNlIGluCnBlcmZvcm1hbmNlLiBUaGUgYnVnIHRoaXMgZml4 ZXMgaXMgdGhlIGNhY2hlIGJlaW5nIGNvbXBsZXRlIHRyYXNoZWQKcGVyaW9kaWNhbGx5LCBidXQg bm9uZSBvZiB0aGUgYWJvdmUgd29ya2xvYWRzIHNob3cgc2lnbmlmaWNhbnQgd2FsbAp0aW1lIGRp ZmZlcmVuY2VzIGluIGJlaGF2aW91ciBiZWNhdXNlIG9mIHRoaXMgbW9kaWZpY2F0aW9uIGFzIHRo ZXkKZG9uJ3QgcmVseSBvbiBjYWNoZSBoaXRzIGZvciBwZXJmb3JtYW5jZS4gSWYgeW91IGhhdmUg YSB3b3JrbG9hZAp0aGF0IGFjdHVhbGx5IHJlbGllcyBvbiByZXNpZGVudCBjYWNoZSBoaXRzIGZv ciBnb29kIHBlcmZvcm1hbmNlLAp0aGVuIHlvdSdsbCBzZWUgYSBkaWZmZXJlbmNlIGluIHBlcmZv cm1hbmNlIHRoYXQgeW91IGNhbiBtZWFzdXJlCndpdGggd2FsbCB0aW1lLi4uLgoKV2hhdCB0aGlz IGNoYW5nZSBkb2VzIGhhdmUgYW4gZWZmZWN0IG9uIGlzIHRoZSB2YXJpYW5jZSBvZiB0aGUKcmVz aWRlbnQgY2FjaGUgc2l6ZSwgd2hpY2ggSSBtb25pdG9yIHZpYSBsaXZlIGdyYXBoaW5nIG9mIHRo ZQp2YXJpb3VzIGNhY2hlIG1ldHJpY3MuIEl0IGlzIG5vdCBhcyBzaW1wbGUgdG8gbW9uaXRvciBh cyBoYXZpbmcgYQpzaW5nbGUgbnVtYmVyIGZhbGwgb3V0IHRoZSBib3R0b20gdGhhdCB5b3UgY2Fu IHBvaW50IHRvIGZvcgppbXByb3ZlbWVudC4gRGV0YWlscyBvZiB0aGlzIGlzIGFscmVhZHkgZG9j dW1lbnRlZCBpbiBwYXRjaCB6ZXJvIG9mCnRoZSBzZXJpZXM6CgpodHRwczovL2xrbWwub3JnL2xr bWwvMjAxMS82LzIvNDIKCkJzaWNhbGx5LCB0aGUgYWJvdmUgbnVtYmVycyBnYXZlIHRoZSBsb3dl c3QgdmFyaWFuY2UgaW4gdGhlIHJlc2lkZW50CmNhY2hlIHNpemUgd2l0aG91dCBwcmV2ZW50aW5n IHRoZSBzaHJpbmtlciBmcm9tIGJlaW5nIGFibGUgdG8gZnJlZQplbm91Z2ggbWVtb3J5IGZvciB0 aGUgc3lzdGVtIHRvIHdvcmsgZWZmZWN0aXZlbHkuCgo+IEkgbWVhbiwgY3VycmVudGx5IHNvbWUg bW0gZm9sa3MgcGxhbiB0byBlbmhhbmNlIHNocmlua2VyLiBTbywKPiBzaGFyaW5nIGJlbmNobWFy ayBtYXkgaGVscCB0byBhdm9pZCBhbiBhY2NpZGVudGFsIHJlZ3Jlc3Npb24uCgpJIHByZWRpY3Qg dGhhdCBJIHdpbGwgaGF2ZSBzb21lIGJ1ZyByZXBvcnRpbmcgdG8gZG8gaW4gZnV0dXJlLiA7KQoK Q2hlZXJzLAoKRGF2ZS4KLS0gCkRhdmUgQ2hpbm5lcgpkYXZpZEBmcm9tb3JiaXQuY29tCgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwp4ZnMgbWFpbGluZyBs aXN0Cnhmc0Bvc3Muc2dpLmNvbQpodHRwOi8vb3NzLnNnaS5jb20vbWFpbG1hbi9saXN0aW5mby94 ZnMK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932072Ab1FUFJW (ORCPT ); Tue, 21 Jun 2011 01:09:22 -0400 Received: from ipmail06.adl6.internode.on.net ([150.101.137.145]:18766 "EHLO ipmail06.adl6.internode.on.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750848Ab1FUFJT (ORCPT ); Tue, 21 Jun 2011 01:09:19 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av0EAKImAE55LChD/2dsb2JhbABUhEmiHniIc65fkQEOgR2DdYEKBKFx Date: Tue, 21 Jun 2011 15:09:14 +1000 From: Dave Chinner To: KOSAKI Motohiro Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, xfs@oss.sgi.com Subject: Re: [PATCH 03/12] vmscan: reduce wind up shrinker->nr when shrinker can't do work Message-ID: <20110621050914.GO32466@dastard> References: <1306998067-27659-1-git-send-email-david@fromorbit.com> <1306998067-27659-4-git-send-email-david@fromorbit.com> <4DFE997C.2060805@jp.fujitsu.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <4DFE997C.2060805@jp.fujitsu.com> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jun 20, 2011 at 09:51:08AM +0900, KOSAKI Motohiro wrote: > (2011/06/02 16:00), Dave Chinner wrote: > > From: Dave Chinner > > > > When a shrinker returns -1 to shrink_slab() to indicate it cannot do > > any work given the current memory reclaim requirements, it adds the > > entire total_scan count to shrinker->nr. The idea ehind this is that > > whenteh shrinker is next called and can do work, it will do the work > > of the previously aborted shrinker call as well. > > > > However, if a filesystem is doing lots of allocation with GFP_NOFS > > set, then we get many, many more aborts from the shrinkers than we > > do successful calls. The result is that shrinker->nr winds up to > > it's maximum permissible value (twice the current cache size) and > > then when the next shrinker call that can do work is issued, it > > has enough scan count built up to free the entire cache twice over. > > > > This manifests itself in the cache going from full to empty in a > > matter of seconds, even when only a small part of the cache is > > needed to be emptied to free sufficient memory. > > > > Under metadata intensive workloads on ext4 and XFS, I'm seeing the > > VFS caches increase memory consumption up to 75% of memory (no page > > cache pressure) over a period of 30-60s, and then the shrinker > > empties them down to zero in the space of 2-3s. This cycle repeats > > over and over again, with the shrinker completely trashing the Ń–node > > and dentry caches every minute or so the workload continues. > > > > This behaviour was made obvious by the shrink_slab tracepoints added > > earlier in the series, and made worse by the patch that corrected > > the concurrent accounting of shrinker->nr. > > > > To avoid this problem, stop repeated small increments of the total > > scan value from winding shrinker->nr up to a value that can cause > > the entire cache to be freed. We still need to allow it to wind up, > > so use the delta as the "large scan" threshold check - if the delta > > is more than a quarter of the entire cache size, then it is a large > > scan and allowed to cause lots of windup because we are clearly > > needing to free lots of memory. > > > > If it isn't a large scan then limit the total scan to half the size > > of the cache so that windup never increases to consume the whole > > cache. Reducing the total scan limit further does not allow enough > > wind-up to maintain the current levels of performance, whilst a > > higher threshold does not prevent the windup from freeing the entire > > cache under sustained workloads. > > > > Signed-off-by: Dave Chinner > > --- > > mm/vmscan.c | 14 ++++++++++++++ > > 1 files changed, 14 insertions(+), 0 deletions(-) > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index dce2767..3688f47 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -277,6 +277,20 @@ unsigned long shrink_slab(struct shrink_control *shrink, > > } > > > > /* > > + * Avoid excessive windup on fielsystem shrinkers due to large > > + * numbers of GFP_NOFS allocations causing the shrinkers to > > + * return -1 all the time. This results in a large nr being > > + * built up so when a shrink that can do some work comes along > > + * it empties the entire cache due to nr >>> max_pass. This is > > + * bad for sustaining a working set in memory. > > + * > > + * Hence only allow nr to go large when a large delta is > > + * calculated. > > + */ > > + if (delta < max_pass / 4) > > + total_scan = min(total_scan, max_pass / 2); > > + > > + /* > > * Avoid risking looping forever due to too large nr value: > > * never try to free more than twice the estimate number of > > * freeable entries. > > I guess "max_pass/4" and "min(total_scan, max_pass / 2)" are your heuristic value. right? Yes. > If so, please write your benchmark name and its result into the description. Take your pick. Anything that generates a large amount of vfs cache pressure combined with GFP_NOFS memory allocation will show changes in -behaviour- as you modify these variables. e.g. creating 50 million inodes in parallel with fs_mark, parallel chmod -R traversals of said inodes, parallel rm -rf, etc. The same cache behaviour can be observed with any of these sorts of cold cache workloads. I say changes in behaviour rather than performance because just measuring wall time does not necessarily show any difference in performance. The bug this fixes is the cache being complete trashed periodically, but none of the above workloads show significant wall time differences in behaviour because of this modification as they don't rely on cache hits for performance. If you have a workload that actually relies on resident cache hits for good performance, then you'll see a difference in performance that you can measure with wall time.... What this change does have an effect on is the variance of the resident cache size, which I monitor via live graphing of the various cache metrics. It is not as simple to monitor as having a single number fall out the bottom that you can point to for improvement. Details of this is already documented in patch zero of the series: https://lkml.org/lkml/2011/6/2/42 Bsically, the above numbers gave the lowest variance in the resident cache size without preventing the shrinker from being able to free enough memory for the system to work effectively. > I mean, currently some mm folks plan to enhance shrinker. So, > sharing benchmark may help to avoid an accidental regression. I predict that I will have some bug reporting to do in future. ;) Cheers, Dave. -- Dave Chinner david@fromorbit.com From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Chinner Subject: Re: [PATCH 03/12] vmscan: reduce wind up shrinker->nr when shrinker can't do work Date: Tue, 21 Jun 2011 15:09:14 +1000 Message-ID: <20110621050914.GO32466@dastard> References: <1306998067-27659-1-git-send-email-david@fromorbit.com> <1306998067-27659-4-git-send-email-david@fromorbit.com> <4DFE997C.2060805@jp.fujitsu.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, xfs@oss.sgi.com To: KOSAKI Motohiro Return-path: Content-Disposition: inline In-Reply-To: <4DFE997C.2060805@jp.fujitsu.com> Sender: owner-linux-mm@kvack.org List-Id: linux-fsdevel.vger.kernel.org On Mon, Jun 20, 2011 at 09:51:08AM +0900, KOSAKI Motohiro wrote: > (2011/06/02 16:00), Dave Chinner wrote: > > From: Dave Chinner > >=20 > > When a shrinker returns -1 to shrink_slab() to indicate it cannot do > > any work given the current memory reclaim requirements, it adds the > > entire total_scan count to shrinker->nr. The idea ehind this is that > > whenteh shrinker is next called and can do work, it will do the work > > of the previously aborted shrinker call as well. > >=20 > > However, if a filesystem is doing lots of allocation with GFP_NOFS > > set, then we get many, many more aborts from the shrinkers than we > > do successful calls. The result is that shrinker->nr winds up to > > it's maximum permissible value (twice the current cache size) and > > then when the next shrinker call that can do work is issued, it > > has enough scan count built up to free the entire cache twice over. > >=20 > > This manifests itself in the cache going from full to empty in a > > matter of seconds, even when only a small part of the cache is > > needed to be emptied to free sufficient memory. > >=20 > > Under metadata intensive workloads on ext4 and XFS, I'm seeing the > > VFS caches increase memory consumption up to 75% of memory (no page > > cache pressure) over a period of 30-60s, and then the shrinker > > empties them down to zero in the space of 2-3s. This cycle repeats > > over and over again, with the shrinker completely trashing the =D1=96= node > > and dentry caches every minute or so the workload continues. > >=20 > > This behaviour was made obvious by the shrink_slab tracepoints added > > earlier in the series, and made worse by the patch that corrected > > the concurrent accounting of shrinker->nr. > >=20 > > To avoid this problem, stop repeated small increments of the total > > scan value from winding shrinker->nr up to a value that can cause > > the entire cache to be freed. We still need to allow it to wind up, > > so use the delta as the "large scan" threshold check - if the delta > > is more than a quarter of the entire cache size, then it is a large > > scan and allowed to cause lots of windup because we are clearly > > needing to free lots of memory. > >=20 > > If it isn't a large scan then limit the total scan to half the size > > of the cache so that windup never increases to consume the whole > > cache. Reducing the total scan limit further does not allow enough > > wind-up to maintain the current levels of performance, whilst a > > higher threshold does not prevent the windup from freeing the entire > > cache under sustained workloads. > >=20 > > Signed-off-by: Dave Chinner > > --- > > mm/vmscan.c | 14 ++++++++++++++ > > 1 files changed, 14 insertions(+), 0 deletions(-) > >=20 > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index dce2767..3688f47 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -277,6 +277,20 @@ unsigned long shrink_slab(struct shrink_control = *shrink, > > } > > =20 > > /* > > + * Avoid excessive windup on fielsystem shrinkers due to large > > + * numbers of GFP_NOFS allocations causing the shrinkers to > > + * return -1 all the time. This results in a large nr being > > + * built up so when a shrink that can do some work comes along > > + * it empties the entire cache due to nr >>> max_pass. This is > > + * bad for sustaining a working set in memory. > > + * > > + * Hence only allow nr to go large when a large delta is > > + * calculated. > > + */ > > + if (delta < max_pass / 4) > > + total_scan =3D min(total_scan, max_pass / 2); > > + > > + /* > > * Avoid risking looping forever due to too large nr value: > > * never try to free more than twice the estimate number of > > * freeable entries. >=20 > I guess "max_pass/4" and "min(total_scan, max_pass / 2)" are your heuri= stic value. right? Yes. > If so, please write your benchmark name and its result into the descrip= tion. Take your pick. Anything that generates a large amount of vfs cache pressure combined with GFP_NOFS memory allocation will show changes in -behaviour- as you modify these variables. e.g. creating 50 million inodes in parallel with fs_mark, parallel chmod -R traversals of said inodes, parallel rm -rf, etc. The same cache behaviour can be observed with any of these sorts of cold cache workloads. I say changes in behaviour rather than performance because just measuring wall time does not necessarily show any difference in performance. The bug this fixes is the cache being complete trashed periodically, but none of the above workloads show significant wall time differences in behaviour because of this modification as they don't rely on cache hits for performance. If you have a workload that actually relies on resident cache hits for good performance, then you'll see a difference in performance that you can measure with wall time.... What this change does have an effect on is the variance of the resident cache size, which I monitor via live graphing of the various cache metrics. It is not as simple to monitor as having a single number fall out the bottom that you can point to for improvement. Details of this is already documented in patch zero of the series: https://lkml.org/lkml/2011/6/2/42 Bsically, the above numbers gave the lowest variance in the resident cache size without preventing the shrinker from being able to free enough memory for the system to work effectively. > I mean, currently some mm folks plan to enhance shrinker. So, > sharing benchmark may help to avoid an accidental regression. I predict that I will have some bug reporting to do in future. ;) Cheers, Dave. --=20 Dave Chinner david@fromorbit.com -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter= .ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail138.messagelabs.com (mail138.messagelabs.com [216.82.249.35]) by kanga.kvack.org (Postfix) with SMTP id 9AC996B0159 for ; Tue, 21 Jun 2011 01:09:28 -0400 (EDT) Date: Tue, 21 Jun 2011 15:09:14 +1000 From: Dave Chinner Subject: Re: [PATCH 03/12] vmscan: reduce wind up shrinker->nr when shrinker can't do work Message-ID: <20110621050914.GO32466@dastard> References: <1306998067-27659-1-git-send-email-david@fromorbit.com> <1306998067-27659-4-git-send-email-david@fromorbit.com> <4DFE997C.2060805@jp.fujitsu.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <4DFE997C.2060805@jp.fujitsu.com> Sender: owner-linux-mm@kvack.org List-ID: To: KOSAKI Motohiro Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, xfs@oss.sgi.com On Mon, Jun 20, 2011 at 09:51:08AM +0900, KOSAKI Motohiro wrote: > (2011/06/02 16:00), Dave Chinner wrote: > > From: Dave Chinner > > > > When a shrinker returns -1 to shrink_slab() to indicate it cannot do > > any work given the current memory reclaim requirements, it adds the > > entire total_scan count to shrinker->nr. The idea ehind this is that > > whenteh shrinker is next called and can do work, it will do the work > > of the previously aborted shrinker call as well. > > > > However, if a filesystem is doing lots of allocation with GFP_NOFS > > set, then we get many, many more aborts from the shrinkers than we > > do successful calls. The result is that shrinker->nr winds up to > > it's maximum permissible value (twice the current cache size) and > > then when the next shrinker call that can do work is issued, it > > has enough scan count built up to free the entire cache twice over. > > > > This manifests itself in the cache going from full to empty in a > > matter of seconds, even when only a small part of the cache is > > needed to be emptied to free sufficient memory. > > > > Under metadata intensive workloads on ext4 and XFS, I'm seeing the > > VFS caches increase memory consumption up to 75% of memory (no page > > cache pressure) over a period of 30-60s, and then the shrinker > > empties them down to zero in the space of 2-3s. This cycle repeats > > over and over again, with the shrinker completely trashing the N?node > > and dentry caches every minute or so the workload continues. > > > > This behaviour was made obvious by the shrink_slab tracepoints added > > earlier in the series, and made worse by the patch that corrected > > the concurrent accounting of shrinker->nr. > > > > To avoid this problem, stop repeated small increments of the total > > scan value from winding shrinker->nr up to a value that can cause > > the entire cache to be freed. We still need to allow it to wind up, > > so use the delta as the "large scan" threshold check - if the delta > > is more than a quarter of the entire cache size, then it is a large > > scan and allowed to cause lots of windup because we are clearly > > needing to free lots of memory. > > > > If it isn't a large scan then limit the total scan to half the size > > of the cache so that windup never increases to consume the whole > > cache. Reducing the total scan limit further does not allow enough > > wind-up to maintain the current levels of performance, whilst a > > higher threshold does not prevent the windup from freeing the entire > > cache under sustained workloads. > > > > Signed-off-by: Dave Chinner > > --- > > mm/vmscan.c | 14 ++++++++++++++ > > 1 files changed, 14 insertions(+), 0 deletions(-) > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index dce2767..3688f47 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -277,6 +277,20 @@ unsigned long shrink_slab(struct shrink_control *shrink, > > } > > > > /* > > + * Avoid excessive windup on fielsystem shrinkers due to large > > + * numbers of GFP_NOFS allocations causing the shrinkers to > > + * return -1 all the time. This results in a large nr being > > + * built up so when a shrink that can do some work comes along > > + * it empties the entire cache due to nr >>> max_pass. This is > > + * bad for sustaining a working set in memory. > > + * > > + * Hence only allow nr to go large when a large delta is > > + * calculated. > > + */ > > + if (delta < max_pass / 4) > > + total_scan = min(total_scan, max_pass / 2); > > + > > + /* > > * Avoid risking looping forever due to too large nr value: > > * never try to free more than twice the estimate number of > > * freeable entries. > > I guess "max_pass/4" and "min(total_scan, max_pass / 2)" are your heuristic value. right? Yes. > If so, please write your benchmark name and its result into the description. Take your pick. Anything that generates a large amount of vfs cache pressure combined with GFP_NOFS memory allocation will show changes in -behaviour- as you modify these variables. e.g. creating 50 million inodes in parallel with fs_mark, parallel chmod -R traversals of said inodes, parallel rm -rf, etc. The same cache behaviour can be observed with any of these sorts of cold cache workloads. I say changes in behaviour rather than performance because just measuring wall time does not necessarily show any difference in performance. The bug this fixes is the cache being complete trashed periodically, but none of the above workloads show significant wall time differences in behaviour because of this modification as they don't rely on cache hits for performance. If you have a workload that actually relies on resident cache hits for good performance, then you'll see a difference in performance that you can measure with wall time.... What this change does have an effect on is the variance of the resident cache size, which I monitor via live graphing of the various cache metrics. It is not as simple to monitor as having a single number fall out the bottom that you can point to for improvement. Details of this is already documented in patch zero of the series: https://lkml.org/lkml/2011/6/2/42 Bsically, the above numbers gave the lowest variance in the resident cache size without preventing the shrinker from being able to free enough memory for the system to work effectively. > I mean, currently some mm folks plan to enhance shrinker. So, > sharing benchmark may help to avoid an accidental regression. I predict that I will have some bug reporting to do in future. ;) Cheers, Dave. -- Dave Chinner david@fromorbit.com -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org