From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id p5K0pGGE095448 for ; Sun, 19 Jun 2011 19:51:17 -0500 Received: from fgwmail6.fujitsu.co.jp (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 165B3DDE42D for ; Sun, 19 Jun 2011 17:51:13 -0700 (PDT) Received: from fgwmail6.fujitsu.co.jp (fgwmail6.fujitsu.co.jp [192.51.44.36]) by cuda.sgi.com with ESMTP id isCphxldL3sG1H3d for ; Sun, 19 Jun 2011 17:51:13 -0700 (PDT) Received: from m2.gw.fujitsu.co.jp (unknown [10.0.50.72]) by fgwmail6.fujitsu.co.jp (Postfix) with ESMTP id 9B0F53EE0C7 for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Received: from smail (m2 [127.0.0.1]) by outgoing.m2.gw.fujitsu.co.jp (Postfix) with ESMTP id 73DB545DE6A for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Received: from s2.gw.fujitsu.co.jp (s2.gw.fujitsu.co.jp [10.0.50.92]) by m2.gw.fujitsu.co.jp (Postfix) with ESMTP id 5ACFF45DE69 for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Received: from s2.gw.fujitsu.co.jp (localhost.localdomain [127.0.0.1]) by s2.gw.fujitsu.co.jp (Postfix) with ESMTP id 476111DB803C for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Received: from ml14.s.css.fujitsu.com (ml14.s.css.fujitsu.com [10.240.81.134]) by s2.gw.fujitsu.co.jp (Postfix) with ESMTP id 020011DB8042 for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Message-ID: <4DFE997C.2060805@jp.fujitsu.com> Date: Mon, 20 Jun 2011 09:51:08 +0900 From: KOSAKI Motohiro MIME-Version: 1.0 Subject: Re: [PATCH 03/12] vmscan: reduce wind up shrinker->nr when shrinker can't do work References: <1306998067-27659-1-git-send-email-david@fromorbit.com> <1306998067-27659-4-git-send-email-david@fromorbit.com> In-Reply-To: <1306998067-27659-4-git-send-email-david@fromorbit.com> List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: david@fromorbit.com Cc: linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, xfs@oss.sgi.com KDIwMTEvMDYvMDIgMTY6MDApLCBEYXZlIENoaW5uZXIgd3JvdGU6Cj4gRnJvbTogRGF2ZSBDaGlu bmVyIDxkY2hpbm5lckByZWRoYXQuY29tPgo+IAo+IFdoZW4gYSBzaHJpbmtlciByZXR1cm5zIC0x IHRvIHNocmlua19zbGFiKCkgdG8gaW5kaWNhdGUgaXQgY2Fubm90IGRvCj4gYW55IHdvcmsgZ2l2 ZW4gdGhlIGN1cnJlbnQgbWVtb3J5IHJlY2xhaW0gcmVxdWlyZW1lbnRzLCBpdCBhZGRzIHRoZQo+ IGVudGlyZSB0b3RhbF9zY2FuIGNvdW50IHRvIHNocmlua2VyLT5uci4gVGhlIGlkZWEgZWhpbmQg dGhpcyBpcyB0aGF0Cj4gd2hlbnRlaCBzaHJpbmtlciBpcyBuZXh0IGNhbGxlZCBhbmQgY2FuIGRv IHdvcmssIGl0IHdpbGwgZG8gdGhlIHdvcmsKPiBvZiB0aGUgcHJldmlvdXNseSBhYm9ydGVkIHNo cmlua2VyIGNhbGwgYXMgd2VsbC4KPiAKPiBIb3dldmVyLCBpZiBhIGZpbGVzeXN0ZW0gaXMgZG9p bmcgbG90cyBvZiBhbGxvY2F0aW9uIHdpdGggR0ZQX05PRlMKPiBzZXQsIHRoZW4gd2UgZ2V0IG1h bnksIG1hbnkgbW9yZSBhYm9ydHMgZnJvbSB0aGUgc2hyaW5rZXJzIHRoYW4gd2UKPiBkbyBzdWNj ZXNzZnVsIGNhbGxzLiBUaGUgcmVzdWx0IGlzIHRoYXQgc2hyaW5rZXItPm5yIHdpbmRzIHVwIHRv Cj4gaXQncyBtYXhpbXVtIHBlcm1pc3NpYmxlIHZhbHVlICh0d2ljZSB0aGUgY3VycmVudCBjYWNo ZSBzaXplKSBhbmQKPiB0aGVuIHdoZW4gdGhlIG5leHQgc2hyaW5rZXIgY2FsbCB0aGF0IGNhbiBk byB3b3JrIGlzIGlzc3VlZCwgaXQKPiBoYXMgZW5vdWdoIHNjYW4gY291bnQgYnVpbHQgdXAgdG8g ZnJlZSB0aGUgZW50aXJlIGNhY2hlIHR3aWNlIG92ZXIuCj4gCj4gVGhpcyBtYW5pZmVzdHMgaXRz ZWxmIGluIHRoZSBjYWNoZSBnb2luZyBmcm9tIGZ1bGwgdG8gZW1wdHkgaW4gYQo+IG1hdHRlciBv ZiBzZWNvbmRzLCBldmVuIHdoZW4gb25seSBhIHNtYWxsIHBhcnQgb2YgdGhlIGNhY2hlIGlzCj4g bmVlZGVkIHRvIGJlIGVtcHRpZWQgdG8gZnJlZSBzdWZmaWNpZW50IG1lbW9yeS4KPiAKPiBVbmRl ciBtZXRhZGF0YSBpbnRlbnNpdmUgd29ya2xvYWRzIG9uIGV4dDQgYW5kIFhGUywgSSdtIHNlZWlu ZyB0aGUKPiBWRlMgY2FjaGVzIGluY3JlYXNlIG1lbW9yeSBjb25zdW1wdGlvbiB1cCB0byA3NSUg b2YgbWVtb3J5IChubyBwYWdlCj4gY2FjaGUgcHJlc3N1cmUpIG92ZXIgYSBwZXJpb2Qgb2YgMzAt NjBzLCBhbmQgdGhlbiB0aGUgc2hyaW5rZXIKPiBlbXB0aWVzIHRoZW0gZG93biB0byB6ZXJvIGlu IHRoZSBzcGFjZSBvZiAyLTNzLiBUaGlzIGN5Y2xlIHJlcGVhdHMKPiBvdmVyIGFuZCBvdmVyIGFn YWluLCB3aXRoIHRoZSBzaHJpbmtlciBjb21wbGV0ZWx5IHRyYXNoaW5nIHRoZSDRlm5vZGUKPiBh bmQgZGVudHJ5IGNhY2hlcyBldmVyeSBtaW51dGUgb3Igc28gdGhlIHdvcmtsb2FkIGNvbnRpbnVl cy4KPiAKPiBUaGlzIGJlaGF2aW91ciB3YXMgbWFkZSBvYnZpb3VzIGJ5IHRoZSBzaHJpbmtfc2xh YiB0cmFjZXBvaW50cyBhZGRlZAo+IGVhcmxpZXIgaW4gdGhlIHNlcmllcywgYW5kIG1hZGUgd29y c2UgYnkgdGhlIHBhdGNoIHRoYXQgY29ycmVjdGVkCj4gdGhlIGNvbmN1cnJlbnQgYWNjb3VudGlu ZyBvZiBzaHJpbmtlci0+bnIuCj4gCj4gVG8gYXZvaWQgdGhpcyBwcm9ibGVtLCBzdG9wIHJlcGVh dGVkIHNtYWxsIGluY3JlbWVudHMgb2YgdGhlIHRvdGFsCj4gc2NhbiB2YWx1ZSBmcm9tIHdpbmRp bmcgc2hyaW5rZXItPm5yIHVwIHRvIGEgdmFsdWUgdGhhdCBjYW4gY2F1c2UKPiB0aGUgZW50aXJl IGNhY2hlIHRvIGJlIGZyZWVkLiBXZSBzdGlsbCBuZWVkIHRvIGFsbG93IGl0IHRvIHdpbmQgdXAs Cj4gc28gdXNlIHRoZSBkZWx0YSBhcyB0aGUgImxhcmdlIHNjYW4iIHRocmVzaG9sZCBjaGVjayAt IGlmIHRoZSBkZWx0YQo+IGlzIG1vcmUgdGhhbiBhIHF1YXJ0ZXIgb2YgdGhlIGVudGlyZSBjYWNo ZSBzaXplLCB0aGVuIGl0IGlzIGEgbGFyZ2UKPiBzY2FuIGFuZCBhbGxvd2VkIHRvIGNhdXNlIGxv dHMgb2Ygd2luZHVwIGJlY2F1c2Ugd2UgYXJlIGNsZWFybHkKPiBuZWVkaW5nIHRvIGZyZWUgbG90 cyBvZiBtZW1vcnkuCj4gCj4gSWYgaXQgaXNuJ3QgYSBsYXJnZSBzY2FuIHRoZW4gbGltaXQgdGhl IHRvdGFsIHNjYW4gdG8gaGFsZiB0aGUgc2l6ZQo+IG9mIHRoZSBjYWNoZSBzbyB0aGF0IHdpbmR1 cCBuZXZlciBpbmNyZWFzZXMgdG8gY29uc3VtZSB0aGUgd2hvbGUKPiBjYWNoZS4gUmVkdWNpbmcg dGhlIHRvdGFsIHNjYW4gbGltaXQgZnVydGhlciBkb2VzIG5vdCBhbGxvdyBlbm91Z2gKPiB3aW5k LXVwIHRvIG1haW50YWluIHRoZSBjdXJyZW50IGxldmVscyBvZiBwZXJmb3JtYW5jZSwgd2hpbHN0 IGEKPiBoaWdoZXIgdGhyZXNob2xkIGRvZXMgbm90IHByZXZlbnQgdGhlIHdpbmR1cCBmcm9tIGZy ZWVpbmcgdGhlIGVudGlyZQo+IGNhY2hlIHVuZGVyIHN1c3RhaW5lZCB3b3JrbG9hZHMuCj4gCj4g U2lnbmVkLW9mZi1ieTogRGF2ZSBDaGlubmVyIDxkY2hpbm5lckByZWRoYXQuY29tPgo+IC0tLQo+ ICBtbS92bXNjYW4uYyB8ICAgMTQgKysrKysrKysrKysrKysKPiAgMSBmaWxlcyBjaGFuZ2VkLCAx NCBpbnNlcnRpb25zKCspLCAwIGRlbGV0aW9ucygtKQo+IAo+IGRpZmYgLS1naXQgYS9tbS92bXNj YW4uYyBiL21tL3Ztc2Nhbi5jCj4gaW5kZXggZGNlMjc2Ny4uMzY4OGY0NyAxMDA2NDQKPiAtLS0g YS9tbS92bXNjYW4uYwo+ICsrKyBiL21tL3Ztc2Nhbi5jCj4gQEAgLTI3Nyw2ICsyNzcsMjAgQEAg dW5zaWduZWQgbG9uZyBzaHJpbmtfc2xhYihzdHJ1Y3Qgc2hyaW5rX2NvbnRyb2wgKnNocmluaywK PiAgCQl9Cj4gIAo+ICAJCS8qCj4gKwkJICogQXZvaWQgZXhjZXNzaXZlIHdpbmR1cCBvbiBmaWVs c3lzdGVtIHNocmlua2VycyBkdWUgdG8gbGFyZ2UKPiArCQkgKiBudW1iZXJzIG9mIEdGUF9OT0ZT IGFsbG9jYXRpb25zIGNhdXNpbmcgdGhlIHNocmlua2VycyB0bwo+ICsJCSAqIHJldHVybiAtMSBh bGwgdGhlIHRpbWUuIFRoaXMgcmVzdWx0cyBpbiBhIGxhcmdlIG5yIGJlaW5nCj4gKwkJICogYnVp bHQgdXAgc28gd2hlbiBhIHNocmluayB0aGF0IGNhbiBkbyBzb21lIHdvcmsgY29tZXMgYWxvbmcK PiArCQkgKiBpdCBlbXB0aWVzIHRoZSBlbnRpcmUgY2FjaGUgZHVlIHRvIG5yID4+PiBtYXhfcGFz cy4gIFRoaXMgaXMKPiArCQkgKiBiYWQgZm9yIHN1c3RhaW5pbmcgYSB3b3JraW5nIHNldCBpbiBt ZW1vcnkuCj4gKwkJICoKPiArCQkgKiBIZW5jZSBvbmx5IGFsbG93IG5yIHRvIGdvIGxhcmdlIHdo ZW4gYSBsYXJnZSBkZWx0YSBpcwo+ICsJCSAqIGNhbGN1bGF0ZWQuCj4gKwkJICovCj4gKwkJaWYg KGRlbHRhIDwgbWF4X3Bhc3MgLyA0KQo+ICsJCQl0b3RhbF9zY2FuID0gbWluKHRvdGFsX3NjYW4s IG1heF9wYXNzIC8gMik7Cj4gKwo+ICsJCS8qCj4gIAkJICogQXZvaWQgcmlza2luZyBsb29waW5n IGZvcmV2ZXIgZHVlIHRvIHRvbyBsYXJnZSBuciB2YWx1ZToKPiAgCQkgKiBuZXZlciB0cnkgdG8g ZnJlZSBtb3JlIHRoYW4gdHdpY2UgdGhlIGVzdGltYXRlIG51bWJlciBvZgo+ICAJCSAqIGZyZWVh YmxlIGVudHJpZXMuCgpJIGd1ZXNzICJtYXhfcGFzcy80IiBhbmQgIm1pbih0b3RhbF9zY2FuLCBt YXhfcGFzcyAvIDIpIiBhcmUgeW91ciBoZXVyaXN0aWMgdmFsdWUuIHJpZ2h0PwpJZiBzbywgcGxl YXNlIHdyaXRlIHlvdXIgYmVuY2htYXJrIG5hbWUgYW5kIGl0cyByZXN1bHQgaW50byB0aGUgZGVz Y3JpcHRpb24uIEkgbWVhbiwKY3VycmVudGx5IHNvbWUgbW0gZm9sa3MgcGxhbiB0byBlbmhhbmNl IHNocmlua2VyLiBTbywgc2hhcmluZyBiZW5jaG1hcmsgbWF5IGhlbHAgdG8gYXZvaWQKYW4gYWNj aWRlbnRhbCByZWdyZXNzaW9uLgoKSSBtZWFuLCB5b3VyIGNvZGUgaXRzZWxmIGxvb2tzIHByZXR0 eSBnb29kIHRvIG1lLgoKdGhhbmtzLgoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fCnhmcyBtYWlsaW5nIGxpc3QKeGZzQG9zcy5zZ2kuY29tCmh0dHA6Ly9v c3Muc2dpLmNvbS9tYWlsbWFuL2xpc3RpbmZvL3hmcwo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751346Ab1FTAvN (ORCPT ); Sun, 19 Jun 2011 20:51:13 -0400 Received: from fgwmail5.fujitsu.co.jp ([192.51.44.35]:35078 "EHLO fgwmail5.fujitsu.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750765Ab1FTAvK (ORCPT ); Sun, 19 Jun 2011 20:51:10 -0400 X-SecurityPolicyCheck-FJ: OK by FujitsuOutboundMailChecker v1.3.1 Message-ID: <4DFE997C.2060805@jp.fujitsu.com> Date: Mon, 20 Jun 2011 09:51:08 +0900 From: KOSAKI Motohiro User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9.2.17) Gecko/20110414 Lightning/1.0b2 Thunderbird/3.1.10 MIME-Version: 1.0 To: david@fromorbit.com CC: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, xfs@oss.sgi.com Subject: Re: [PATCH 03/12] vmscan: reduce wind up shrinker->nr when shrinker can't do work References: <1306998067-27659-1-git-send-email-david@fromorbit.com> <1306998067-27659-4-git-send-email-david@fromorbit.com> In-Reply-To: <1306998067-27659-4-git-send-email-david@fromorbit.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org (2011/06/02 16:00), Dave Chinner wrote: > From: Dave Chinner > > When a shrinker returns -1 to shrink_slab() to indicate it cannot do > any work given the current memory reclaim requirements, it adds the > entire total_scan count to shrinker->nr. The idea ehind this is that > whenteh shrinker is next called and can do work, it will do the work > of the previously aborted shrinker call as well. > > However, if a filesystem is doing lots of allocation with GFP_NOFS > set, then we get many, many more aborts from the shrinkers than we > do successful calls. The result is that shrinker->nr winds up to > it's maximum permissible value (twice the current cache size) and > then when the next shrinker call that can do work is issued, it > has enough scan count built up to free the entire cache twice over. > > This manifests itself in the cache going from full to empty in a > matter of seconds, even when only a small part of the cache is > needed to be emptied to free sufficient memory. > > Under metadata intensive workloads on ext4 and XFS, I'm seeing the > VFS caches increase memory consumption up to 75% of memory (no page > cache pressure) over a period of 30-60s, and then the shrinker > empties them down to zero in the space of 2-3s. This cycle repeats > over and over again, with the shrinker completely trashing the Ń–node > and dentry caches every minute or so the workload continues. > > This behaviour was made obvious by the shrink_slab tracepoints added > earlier in the series, and made worse by the patch that corrected > the concurrent accounting of shrinker->nr. > > To avoid this problem, stop repeated small increments of the total > scan value from winding shrinker->nr up to a value that can cause > the entire cache to be freed. We still need to allow it to wind up, > so use the delta as the "large scan" threshold check - if the delta > is more than a quarter of the entire cache size, then it is a large > scan and allowed to cause lots of windup because we are clearly > needing to free lots of memory. > > If it isn't a large scan then limit the total scan to half the size > of the cache so that windup never increases to consume the whole > cache. Reducing the total scan limit further does not allow enough > wind-up to maintain the current levels of performance, whilst a > higher threshold does not prevent the windup from freeing the entire > cache under sustained workloads. > > Signed-off-by: Dave Chinner > --- > mm/vmscan.c | 14 ++++++++++++++ > 1 files changed, 14 insertions(+), 0 deletions(-) > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index dce2767..3688f47 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -277,6 +277,20 @@ unsigned long shrink_slab(struct shrink_control *shrink, > } > > /* > + * Avoid excessive windup on fielsystem shrinkers due to large > + * numbers of GFP_NOFS allocations causing the shrinkers to > + * return -1 all the time. This results in a large nr being > + * built up so when a shrink that can do some work comes along > + * it empties the entire cache due to nr >>> max_pass. This is > + * bad for sustaining a working set in memory. > + * > + * Hence only allow nr to go large when a large delta is > + * calculated. > + */ > + if (delta < max_pass / 4) > + total_scan = min(total_scan, max_pass / 2); > + > + /* > * Avoid risking looping forever due to too large nr value: > * never try to free more than twice the estimate number of > * freeable entries. I guess "max_pass/4" and "min(total_scan, max_pass / 2)" are your heuristic value. right? If so, please write your benchmark name and its result into the description. I mean, currently some mm folks plan to enhance shrinker. So, sharing benchmark may help to avoid an accidental regression. I mean, your code itself looks pretty good to me. thanks. From mboxrd@z Thu Jan 1 00:00:00 1970 From: KOSAKI Motohiro Subject: Re: [PATCH 03/12] vmscan: reduce wind up shrinker->nr when shrinker can't do work Date: Mon, 20 Jun 2011 09:51:08 +0900 Message-ID: <4DFE997C.2060805@jp.fujitsu.com> References: <1306998067-27659-1-git-send-email-david@fromorbit.com> <1306998067-27659-4-git-send-email-david@fromorbit.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, xfs@oss.sgi.com To: david@fromorbit.com Return-path: Received: from fgwmail5.fujitsu.co.jp ([192.51.44.35]:35078 "EHLO fgwmail5.fujitsu.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750765Ab1FTAvK (ORCPT ); Sun, 19 Jun 2011 20:51:10 -0400 In-Reply-To: <1306998067-27659-4-git-send-email-david@fromorbit.com> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: (2011/06/02 16:00), Dave Chinner wrote: > From: Dave Chinner >=20 > When a shrinker returns -1 to shrink_slab() to indicate it cannot do > any work given the current memory reclaim requirements, it adds the > entire total_scan count to shrinker->nr. The idea ehind this is that > whenteh shrinker is next called and can do work, it will do the work > of the previously aborted shrinker call as well. >=20 > However, if a filesystem is doing lots of allocation with GFP_NOFS > set, then we get many, many more aborts from the shrinkers than we > do successful calls. The result is that shrinker->nr winds up to > it's maximum permissible value (twice the current cache size) and > then when the next shrinker call that can do work is issued, it > has enough scan count built up to free the entire cache twice over. >=20 > This manifests itself in the cache going from full to empty in a > matter of seconds, even when only a small part of the cache is > needed to be emptied to free sufficient memory. >=20 > Under metadata intensive workloads on ext4 and XFS, I'm seeing the > VFS caches increase memory consumption up to 75% of memory (no page > cache pressure) over a period of 30-60s, and then the shrinker > empties them down to zero in the space of 2-3s. This cycle repeats > over and over again, with the shrinker completely trashing the =D1=96= node > and dentry caches every minute or so the workload continues. >=20 > This behaviour was made obvious by the shrink_slab tracepoints added > earlier in the series, and made worse by the patch that corrected > the concurrent accounting of shrinker->nr. >=20 > To avoid this problem, stop repeated small increments of the total > scan value from winding shrinker->nr up to a value that can cause > the entire cache to be freed. We still need to allow it to wind up, > so use the delta as the "large scan" threshold check - if the delta > is more than a quarter of the entire cache size, then it is a large > scan and allowed to cause lots of windup because we are clearly > needing to free lots of memory. >=20 > If it isn't a large scan then limit the total scan to half the size > of the cache so that windup never increases to consume the whole > cache. Reducing the total scan limit further does not allow enough > wind-up to maintain the current levels of performance, whilst a > higher threshold does not prevent the windup from freeing the entire > cache under sustained workloads. >=20 > Signed-off-by: Dave Chinner > --- > mm/vmscan.c | 14 ++++++++++++++ > 1 files changed, 14 insertions(+), 0 deletions(-) >=20 > diff --git a/mm/vmscan.c b/mm/vmscan.c > index dce2767..3688f47 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -277,6 +277,20 @@ unsigned long shrink_slab(struct shrink_control = *shrink, > } > =20 > /* > + * Avoid excessive windup on fielsystem shrinkers due to large > + * numbers of GFP_NOFS allocations causing the shrinkers to > + * return -1 all the time. This results in a large nr being > + * built up so when a shrink that can do some work comes along > + * it empties the entire cache due to nr >>> max_pass. This is > + * bad for sustaining a working set in memory. > + * > + * Hence only allow nr to go large when a large delta is > + * calculated. > + */ > + if (delta < max_pass / 4) > + total_scan =3D min(total_scan, max_pass / 2); > + > + /* > * Avoid risking looping forever due to too large nr value: > * never try to free more than twice the estimate number of > * freeable entries. I guess "max_pass/4" and "min(total_scan, max_pass / 2)" are your heuri= stic value. right? If so, please write your benchmark name and its result into the descrip= tion. I mean, currently some mm folks plan to enhance shrinker. So, sharing benchmark= may help to avoid an accidental regression. I mean, your code itself looks pretty good to me. thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel= " in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail6.bemta8.messagelabs.com (mail6.bemta8.messagelabs.com [216.82.243.55]) by kanga.kvack.org (Postfix) with ESMTP id E94196B0012 for ; Sun, 19 Jun 2011 20:51:20 -0400 (EDT) Received: from m3.gw.fujitsu.co.jp (unknown [10.0.50.73]) by fgwmail5.fujitsu.co.jp (Postfix) with ESMTP id 86FF73EE0CF for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Received: from smail (m3 [127.0.0.1]) by outgoing.m3.gw.fujitsu.co.jp (Postfix) with ESMTP id 6BA0345DE8F for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Received: from s3.gw.fujitsu.co.jp (s3.gw.fujitsu.co.jp [10.0.50.93]) by m3.gw.fujitsu.co.jp (Postfix) with ESMTP id 52EA645DE67 for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Received: from s3.gw.fujitsu.co.jp (localhost.localdomain [127.0.0.1]) by s3.gw.fujitsu.co.jp (Postfix) with ESMTP id 431E51DB803E for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Received: from ml14.s.css.fujitsu.com (ml14.s.css.fujitsu.com [10.240.81.134]) by s3.gw.fujitsu.co.jp (Postfix) with ESMTP id 0227C1DB8037 for ; Mon, 20 Jun 2011 09:51:09 +0900 (JST) Message-ID: <4DFE997C.2060805@jp.fujitsu.com> Date: Mon, 20 Jun 2011 09:51:08 +0900 From: KOSAKI Motohiro MIME-Version: 1.0 Subject: Re: [PATCH 03/12] vmscan: reduce wind up shrinker->nr when shrinker can't do work References: <1306998067-27659-1-git-send-email-david@fromorbit.com> <1306998067-27659-4-git-send-email-david@fromorbit.com> In-Reply-To: <1306998067-27659-4-git-send-email-david@fromorbit.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: owner-linux-mm@kvack.org List-ID: To: david@fromorbit.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, xfs@oss.sgi.com (2011/06/02 16:00), Dave Chinner wrote: > From: Dave Chinner > > When a shrinker returns -1 to shrink_slab() to indicate it cannot do > any work given the current memory reclaim requirements, it adds the > entire total_scan count to shrinker->nr. The idea ehind this is that > whenteh shrinker is next called and can do work, it will do the work > of the previously aborted shrinker call as well. > > However, if a filesystem is doing lots of allocation with GFP_NOFS > set, then we get many, many more aborts from the shrinkers than we > do successful calls. The result is that shrinker->nr winds up to > it's maximum permissible value (twice the current cache size) and > then when the next shrinker call that can do work is issued, it > has enough scan count built up to free the entire cache twice over. > > This manifests itself in the cache going from full to empty in a > matter of seconds, even when only a small part of the cache is > needed to be emptied to free sufficient memory. > > Under metadata intensive workloads on ext4 and XFS, I'm seeing the > VFS caches increase memory consumption up to 75% of memory (no page > cache pressure) over a period of 30-60s, and then the shrinker > empties them down to zero in the space of 2-3s. This cycle repeats > over and over again, with the shrinker completely trashing the N?node > and dentry caches every minute or so the workload continues. > > This behaviour was made obvious by the shrink_slab tracepoints added > earlier in the series, and made worse by the patch that corrected > the concurrent accounting of shrinker->nr. > > To avoid this problem, stop repeated small increments of the total > scan value from winding shrinker->nr up to a value that can cause > the entire cache to be freed. We still need to allow it to wind up, > so use the delta as the "large scan" threshold check - if the delta > is more than a quarter of the entire cache size, then it is a large > scan and allowed to cause lots of windup because we are clearly > needing to free lots of memory. > > If it isn't a large scan then limit the total scan to half the size > of the cache so that windup never increases to consume the whole > cache. Reducing the total scan limit further does not allow enough > wind-up to maintain the current levels of performance, whilst a > higher threshold does not prevent the windup from freeing the entire > cache under sustained workloads. > > Signed-off-by: Dave Chinner > --- > mm/vmscan.c | 14 ++++++++++++++ > 1 files changed, 14 insertions(+), 0 deletions(-) > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index dce2767..3688f47 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -277,6 +277,20 @@ unsigned long shrink_slab(struct shrink_control *shrink, > } > > /* > + * Avoid excessive windup on fielsystem shrinkers due to large > + * numbers of GFP_NOFS allocations causing the shrinkers to > + * return -1 all the time. This results in a large nr being > + * built up so when a shrink that can do some work comes along > + * it empties the entire cache due to nr >>> max_pass. This is > + * bad for sustaining a working set in memory. > + * > + * Hence only allow nr to go large when a large delta is > + * calculated. > + */ > + if (delta < max_pass / 4) > + total_scan = min(total_scan, max_pass / 2); > + > + /* > * Avoid risking looping forever due to too large nr value: > * never try to free more than twice the estimate number of > * freeable entries. I guess "max_pass/4" and "min(total_scan, max_pass / 2)" are your heuristic value. right? If so, please write your benchmark name and its result into the description. I mean, currently some mm folks plan to enhance shrinker. So, sharing benchmark may help to avoid an accidental regression. I mean, your code itself looks pretty good to me. thanks. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org