From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id p4U269kQ213250 for ; Sun, 29 May 2011 21:06:09 -0500 Received: from ipmail06.adl6.internode.on.net (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 5CEF21ECFA14 for ; Sun, 29 May 2011 19:06:07 -0700 (PDT) Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id cgGY7HE70x6hKMtI for ; Sun, 29 May 2011 19:06:07 -0700 (PDT) Date: Mon, 30 May 2011 12:06:04 +1000 From: Dave Chinner Subject: [regression, 3.0-rc1] dentry cache growth during unlinks, XFS performance way down Message-ID: <20110530020604.GC561@dastard> MIME-Version: 1.0 Content-Disposition: inline List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: linux-kernel@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com Rm9sa3MsCgpJIGp1c3QgYm9vdGVkIHVwIGEgMy4wLXJjMSBrZXJuZWwsIGFuZCBtb3VudGVkIGFu IFhGUyBmaWxlc3lzdGVtCndpdGggNTBNIGZpbGVzIGluIGl0LiBSdW5uaW5nOgoKJCBmb3IgaSBp biAvbW50L3NjcmF0Y2gvKjsgZG8gc3VkbyAvdXNyL2Jpbi90aW1lIHJtIC1yZiAkaSAyPiYxICYg ZG9uZQoKcnVucyBhbiA4LXdheSBwYXJhbGxlbCB1bmxpbmsgb24gdGhlIGZpbGVzLiBOb3JtYWxs eSB0aGlzIHJ1bnMgYXQKYXJvdW5kIDgwayB1bmxpbmtzL3MsIGFuZCBpdCBydW5zIHdpdGggYWJv dXQgNTAway0xbSBkZW50cmllcyBhbmQKaW5vZGVzIGNhY2hlZCBpbiB0aGUgc3RlYWR5IHN0YXRl LgoKVGhlIHN0ZWFkeSBzdGF0ZSBiZWhhdmlvdXIgd2l0aCAzLjAtcmMxIGlzIHRoYXQgdGhlcmUg YXJlIGFyb3VuZCAxMG0KY2FjaGVkIGRlbnRyaWVzIC0gYWxsIG5lZ2F0aXZlIGRlbnRyaWVzIC0g Y29uc3VtaW5nIGFib3V0IDEuNkdCIG9mClJBTSAob2YgNEdCIHRvdGFsKS4gUHJldmlvdXMgc3Rl YWR5IHN0YXRlIHdhcywgSUlSQywgYXJvdW5kIDIwME1CIG9mCmRlbnRyaWVzLiBNeSBpbml0aWFs IHN1c3BpY2lvbnMgYXJlIHRoYXQgdGhlIGRlbnRyeSB1bmhhc2hpbmcKY2hhbmdl0ZUgbWF5IGJl IHRoZSBjYXVzZSBvZiB0aGlzLi4uCgpQZXJmb3JtYW5jZSBpcyBub3cgYSB2ZXJ5IHJlZ3VsYXIg cGVhay90cm91Z2ggcGF0dGVuIHdpdGggYSBwZXJpb2QKb2YgYWJvdXQgMjBzLCB3aGVyZSB0aGUg cGVhayBpcyBhYm91dCA4MGsgdW5saW5rcy9zLCBhbmQgdGhlIHRyb3VnaAppcyBhcm91bmQgMjBr IHVubGlua3Mvcy4gVGhlIHJ1bnRpbWUgb2YgdGhlIDUwbSBpbm9kZSBkZWxldGUgaGFzCmdvbmUg ZnJvbSBhcm91bmQgMTBtIG9uIDIuNi4zOSwgdG86CgoxMS43MXVzZXIgNDcwLjA4c3lzdGVtIDE1 OjA3LjkxZWxhcHNlZCA1MyVDUFUgKDBhdmd0ZXh0KzBhdmdkYXRhIDEzMzE4NG1heHJlc2lkZW50 KWsKMGlucHV0cyswb3V0cHV0cyAoMzBtYWpvcis0OTcyMjhtaW5vcilwYWdlZmF1bHRzIDBzd2Fw cwoxMS41MHVzZXIgNDY4LjMwc3lzdGVtIDE1OjE0LjM1ZWxhcHNlZCA1MiVDUFUgKDBhdmd0ZXh0 KzBhdmdkYXRhIDEzMzE2OG1heHJlc2lkZW50KWsKMGlucHV0cyswb3V0cHV0cyAoNDJtYWpvcis0 OTcyNjhtaW5vcilwYWdlZmF1bHRzIDBzd2FwcwoxMS4zNHVzZXIgNDY2LjY2c3lzdGVtIDE1OjI2 LjA0ZWxhcHNlZCA1MSVDUFUgKDBhdmd0ZXh0KzBhdmdkYXRhIDEzMzIxNm1heHJlc2lkZW50KWsK MGlucHV0cyswb3V0cHV0cyAoMThtYWpvcis0OTcxMjFtaW5vcilwYWdlZmF1bHRzIDBzd2Fwcwox Mi4xNHVzZXIgNDcwLjQ2c3lzdGVtIDE1OjI2LjYwZWxhcHNlZCA1MiVDUFUgKDBhdmd0ZXh0KzBh dmdkYXRhIDEzMzIxNm1heHJlc2lkZW50KWsKMGlucHV0cyswb3V0cHV0cyAoNDRtYWpvcis0OTcz MDltaW5vcilwYWdlZmF1bHRzIDBzd2FwcwoxMi4wNnVzZXIgNDYzLjc0c3lzdGVtIDE1OjI4Ljg0 ZWxhcHNlZCA1MSVDUFUgKDBhdmd0ZXh0KzBhdmdkYXRhIDEzMzIzMm1heHJlc2lkZW50KWsKMGlu cHV0cyswb3V0cHV0cyAoMjVtYWpvcis0OTcwNDZtaW5vcilwYWdlZmF1bHRzIDBzd2FwcwoxMS4z N3VzZXIgNDY4LjE4c3lzdGVtIDE1OjI5LjA3ZWxhcHNlZCA1MSVDUFUgKDBhdmd0ZXh0KzBhdmdk YXRhIDEzMzE4NG1heHJlc2lkZW50KWsKMGlucHV0cyswb3V0cHV0cyAoNTVtYWpvcis0OTcwNTZt aW5vcilwYWdlZmF1bHRzIDBzd2FwcwoxMS42OXVzZXIgNDc0LjQ2c3lzdGVtIDE1OjQ3LjQ1ZWxh cHNlZCA1MSVDUFUgKDBhdmd0ZXh0KzBhdmdkYXRhIDEzMzIzMm1heHJlc2lkZW50KWsKMGlucHV0 cyswb3V0cHV0cyAoNjFtYWpvcis0OTcyODRtaW5vcilwYWdlZmF1bHRzIDBzd2FwcwoxMS4zMnVz ZXIgNDc2Ljkzc3lzdGVtIDE2OjA1LjE0ZWxhcHNlZCA1MCVDUFUgKDBhdmd0ZXh0KzBhdmdkYXRh IDEzMzE4NG1heHJlc2lkZW50KWsKMGlucHV0cyswb3V0cHV0cyAoMzBtYWpvcis0OTcyMjVtaW5v cilwYWdlZmF1bHRzIDBzd2FwcwoKQWJvdXQgMTYgbWludXRlcy4gSSdtIG5vdCBzdXJlIHlldCB3 aGV0aGVyIHRoaXMgY2hhbmdlIG9mIGNhY2hlCmJlaGF2aW91ciBpcyB0aGUgY2F1c2Ugb2YgdGhl IGVudGlyZSBwZXJmb3JtYW5jZSByZWdyZXNzaW9uLCBidXQKaXQncyBhIGdvb2QgY2hhbmNlIHRo YXQgaXQgaXMgYSBjb250cmlidXRpbmcgZmFjdG9yLgoKQ2hyaXN0b3BoLCBpdCBhcHBlYXJzIHRo YXQgdGhlcmUgaXMgYSBzaWduaWZpY2FudCBpbmNyZWFzZSBpbiBsb2cKZm9yY2VzIGR1cmluZyB0 aGlzIHVubGluayB3b3JrbG9hZCBjb21wYXJlZCB0byAyLjYuMzksIGFuZCB0aGF0J3MKcG9zc2li bHkgd2hlcmUgdGhlIHBlcmZvcm1hbmNlIGRlZ3JhZGF0aW9uIGlzIGNvbWluZyBmcm9tLiBJJ20g Z29pbmcKdG8gaGF2ZSB0byBiaXNlY3QsIEkgdGhpbmsuCgpUaGUgOC13YXkgY3JlYXRlIHJhdGUg Zm9yIHRoZSA1MG0gaW5vZGVzIGlzIGRvd24gYnkgMTAlIGFzIHdlbGwsIGJ1dCBJCmRvbid0IHRo aW5rIHRoYXQgaGFzIGFueXRoaW5nIHRvIGRvIHdpdGggZGVudHJ5IGNhY2hlIGJlaGF2aW91ciAt CmxvZyB3cml0ZSB0aHJvdWdocHV0IGlzIHVwIGJ5IGEgZmFjdG9yIG9mIDN4IG92ZXIgMi42LjM5 LiBDaHJpc3RvcGgsCkkgdGhpbmsgdGhhdCB0aGlzIGlzIG9uY2UgYWdhaW4gZHVlIHRvIGFuIGlu Y3JlYXNlIGluIGxvZyBmb3JjZXMsCmJ1dCBJIG5lZWQgdG8gZG8gbW9yZSBhbmFseXNpcyB0byBi ZSBzdXJlLi4uCgpDaGVlcnMsCgpEYXZlLgotLSAKRGF2ZSBDaGlubmVyCmRhdmlkQGZyb21vcmJp dC5jb20KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCnhm cyBtYWlsaW5nIGxpc3QKeGZzQG9zcy5zZ2kuY29tCmh0dHA6Ly9vc3Muc2dpLmNvbS9tYWlsbWFu L2xpc3RpbmZvL3hmcwo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753653Ab1E3CGK (ORCPT ); Sun, 29 May 2011 22:06:10 -0400 Received: from ipmail06.adl6.internode.on.net ([150.101.137.145]:63453 "EHLO ipmail06.adl6.internode.on.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752300Ab1E3CGJ (ORCPT ); Sun, 29 May 2011 22:06:09 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AkcFAAL64k15LCoegWdsb2JhbABUhEmTUY4OFQEBFiYltk+PUg6BHYNsgQcEn3s Date: Mon, 30 May 2011 12:06:04 +1000 From: Dave Chinner To: linux-kernel@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com Subject: [regression, 3.0-rc1] dentry cache growth during unlinks, XFS performance way down Message-ID: <20110530020604.GC561@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Folks, I just booted up a 3.0-rc1 kernel, and mounted an XFS filesystem with 50M files in it. Running: $ for i in /mnt/scratch/*; do sudo /usr/bin/time rm -rf $i 2>&1 & done runs an 8-way parallel unlink on the files. Normally this runs at around 80k unlinks/s, and it runs with about 500k-1m dentries and inodes cached in the steady state. The steady state behaviour with 3.0-rc1 is that there are around 10m cached dentries - all negative dentries - consuming about 1.6GB of RAM (of 4GB total). Previous steady state was, IIRC, around 200MB of dentries. My initial suspicions are that the dentry unhashing changeѕ may be the cause of this... Performance is now a very regular peak/trough patten with a period of about 20s, where the peak is about 80k unlinks/s, and the trough is around 20k unlinks/s. The runtime of the 50m inode delete has gone from around 10m on 2.6.39, to: 11.71user 470.08system 15:07.91elapsed 53%CPU (0avgtext+0avgdata 133184maxresident)k 0inputs+0outputs (30major+497228minor)pagefaults 0swaps 11.50user 468.30system 15:14.35elapsed 52%CPU (0avgtext+0avgdata 133168maxresident)k 0inputs+0outputs (42major+497268minor)pagefaults 0swaps 11.34user 466.66system 15:26.04elapsed 51%CPU (0avgtext+0avgdata 133216maxresident)k 0inputs+0outputs (18major+497121minor)pagefaults 0swaps 12.14user 470.46system 15:26.60elapsed 52%CPU (0avgtext+0avgdata 133216maxresident)k 0inputs+0outputs (44major+497309minor)pagefaults 0swaps 12.06user 463.74system 15:28.84elapsed 51%CPU (0avgtext+0avgdata 133232maxresident)k 0inputs+0outputs (25major+497046minor)pagefaults 0swaps 11.37user 468.18system 15:29.07elapsed 51%CPU (0avgtext+0avgdata 133184maxresident)k 0inputs+0outputs (55major+497056minor)pagefaults 0swaps 11.69user 474.46system 15:47.45elapsed 51%CPU (0avgtext+0avgdata 133232maxresident)k 0inputs+0outputs (61major+497284minor)pagefaults 0swaps 11.32user 476.93system 16:05.14elapsed 50%CPU (0avgtext+0avgdata 133184maxresident)k 0inputs+0outputs (30major+497225minor)pagefaults 0swaps About 16 minutes. I'm not sure yet whether this change of cache behaviour is the cause of the entire performance regression, but it's a good chance that it is a contributing factor. Christoph, it appears that there is a significant increase in log forces during this unlink workload compared to 2.6.39, and that's possibly where the performance degradation is coming from. I'm going to have to bisect, I think. The 8-way create rate for the 50m inodes is down by 10% as well, but I don't think that has anything to do with dentry cache behaviour - log write throughput is up by a factor of 3x over 2.6.39. Christoph, I think that this is once again due to an increase in log forces, but I need to do more analysis to be sure... Cheers, Dave. -- Dave Chinner david@fromorbit.com