From mboxrd@z Thu Jan 1 00:00:00 1970 From: Piergiorgio Sartor Subject: Re: md-RAID5/6 stripe_cache_size default value vs performance vs memory footprint Date: Sat, 21 Dec 2013 13:20:14 +0100 Message-ID: <20131221122014.GA3909@lazy.lzy> References: <52B102FF.8040404@pzystorm.de> <52B2FE9E.50307@hardwarefreak.com> <52B41B67.9030308@pzystorm.de> <201312202343.47895.arekm@maven.pl> <52B57912.5080000@hardwarefreak.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: Content-Disposition: inline In-Reply-To: <52B57912.5080000@hardwarefreak.com> Sender: linux-raid-owner@vger.kernel.org To: Stan Hoeppner Cc: Arkadiusz =?utf-8?Q?Mi=C5=9Bkiewicz?= , linux-raid@vger.kernel.org, "xfs@oss.sgi.com" List-Id: linux-raid.ids On Sat, Dec 21, 2013 at 05:18:42AM -0600, Stan Hoeppner wrote: > I renamed the subject as your question doesn't really apply to XFS, o= r > the OP, but to md-RAID. >=20 > On 12/20/2013 4:43 PM, Arkadiusz Mi=C5=9Bkiewicz wrote: >=20 > > I wonder why kernel is giving defaults that everyone repeatly recom= mends to=20 > > change/increase? Has anyone tried to bugreport that for stripe_cach= e_size=20 > > case? >=20 > The answer is balancing default md-RAID5/6 write performance against > kernel RAM consumption, with more weight given to the latter. The fo= rmula: >=20 > ((4096*stripe_cache_size)*num_drives)=3D RAM consumed for stripe cach= e >=20 > High stripe_cache_size values will cause the kernel to eat non trivia= l > amounts of RAM for the stripe cache buffer. This table demonstrates = the > effect today for typical RAID5/6 disk counts. >=20 > stripe_cache_size drives RAM consumed > 256 4 4 MB > 8 8 MB > 16 16 MB > 512 4 8 MB > 8 16 MB > 16 32 MB > 1024 4 16 MB > 8 32 MB > 16 64 MB > 2048 4 32 MB > 8 64 MB > 16 128 MB > 4096 4 64 MB > 8 128 MB > 16 256 MB >=20 > The powers that be, Linus in particular, are not fond of default > settings that create a lot of kernel memory structures. The default > md-RAID5/6 stripe_cache-size yields 1MB consumed per member device. >=20 > With SSDs becoming mainstream, and becoming ever faster, at some poin= t > the md-RAID5/6 architecture will have to be redesigned because of the > memory footprint required for performance. Currently the required si= ze > of the stripe cache appears directly proportional to the aggregate wr= ite > throughput of the RAID devices. Thus the optimal value will vary > greatly from one system to another depending on the throughput of the > drives. >=20 > For example, I assisted a user with 5x Intel SSDs back in January and > his system required 4096, or 80MB of RAM for stripe cache, to reach > maximum write throughput of the devices. This yielded 600MB/s or 60% > greater throughput than 2048, or 40MB RAM for cache. In his case 60M= B > more RAM than the default was well worth the increase as the machine = was > an iSCSI target server with 8GB RAM. >=20 > In the previous case with 5x rust RAID6 the 2048 value seemed optimal > (though not yet verified), requiring 40MB less RAM than the 5x Intel > SSDs. For a 3 modern rust RAID5 the default of 256, or 3MB, is close= to > optimal but maybe a little low. Consider that 256 has been the defau= lt > for a very long time, and was selected back when average drive > throughput was much much lower, as in 50MB/s or less, SSDs hadn't yet > been invented, and system memories were much smaller. >=20 > Due to the massive difference in throughput between rust and SSD, any > meaningful change in the default really requires new code to sniff ou= t > what type of devices constitute the array, if that's possible, and it > probably isn't, and set a lowish default accordingly. Again, SSDs > didn't exist when md-RAID was coded, nor when this default was set, a= nd > this throws a big monkey wrench into these spokes. Hi Stan, nice analytical report, as usual... My dumb suggestion would be to simply use udev to setup the drives. Everything, stripe_cache, read_ahead, stcerr, etc. can be configured, I suppose, by udev rules. bye, --=20 piergiorgio -- To unsubscribe from this list: send the line "unsubscribe linux-raid" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 5E70D7F3F for ; Sat, 21 Dec 2013 06:20:26 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id BF7ECAC001 for ; Sat, 21 Dec 2013 04:20:25 -0800 (PST) Received: from mail-in-03.arcor-online.net (mail-in-03.arcor-online.net [151.189.21.43]) by cuda.sgi.com with ESMTP id tSof5yoLLd4CCWEG (version=TLSv1 cipher=AES256-SHA bits=256 verify=NO) for ; Sat, 21 Dec 2013 04:20:23 -0800 (PST) Date: Sat, 21 Dec 2013 13:20:14 +0100 From: Piergiorgio Sartor Subject: Re: md-RAID5/6 stripe_cache_size default value vs performance vs memory footprint Message-ID: <20131221122014.GA3909@lazy.lzy> References: <52B102FF.8040404@pzystorm.de> <52B2FE9E.50307@hardwarefreak.com> <52B41B67.9030308@pzystorm.de> <201312202343.47895.arekm@maven.pl> <52B57912.5080000@hardwarefreak.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <52B57912.5080000@hardwarefreak.com> List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xfs-bounces@oss.sgi.com Sender: xfs-bounces@oss.sgi.com To: Stan Hoeppner Cc: linux-raid@vger.kernel.org, "xfs@oss.sgi.com" T24gU2F0LCBEZWMgMjEsIDIwMTMgYXQgMDU6MTg6NDJBTSAtMDYwMCwgU3RhbiBIb2VwcG5lciB3 cm90ZToKPiBJIHJlbmFtZWQgdGhlIHN1YmplY3QgYXMgeW91ciBxdWVzdGlvbiBkb2Vzbid0IHJl YWxseSBhcHBseSB0byBYRlMsIG9yCj4gdGhlIE9QLCBidXQgdG8gbWQtUkFJRC4KPiAKPiBPbiAx Mi8yMC8yMDEzIDQ6NDMgUE0sIEFya2FkaXVzeiBNacWba2lld2ljeiB3cm90ZToKPiAKPiA+IEkg d29uZGVyIHdoeSBrZXJuZWwgaXMgZ2l2aW5nIGRlZmF1bHRzIHRoYXQgZXZlcnlvbmUgcmVwZWF0 bHkgcmVjb21tZW5kcyB0byAKPiA+IGNoYW5nZS9pbmNyZWFzZT8gSGFzIGFueW9uZSB0cmllZCB0 byBidWdyZXBvcnQgdGhhdCBmb3Igc3RyaXBlX2NhY2hlX3NpemUgCj4gPiBjYXNlPwo+IAo+IFRo ZSBhbnN3ZXIgaXMgYmFsYW5jaW5nIGRlZmF1bHQgbWQtUkFJRDUvNiB3cml0ZSBwZXJmb3JtYW5j ZSBhZ2FpbnN0Cj4ga2VybmVsIFJBTSBjb25zdW1wdGlvbiwgd2l0aCBtb3JlIHdlaWdodCBnaXZl biB0byB0aGUgbGF0dGVyLiAgVGhlIGZvcm11bGE6Cj4gCj4gKCg0MDk2KnN0cmlwZV9jYWNoZV9z aXplKSpudW1fZHJpdmVzKT0gUkFNIGNvbnN1bWVkIGZvciBzdHJpcGUgY2FjaGUKPiAKPiBIaWdo IHN0cmlwZV9jYWNoZV9zaXplIHZhbHVlcyB3aWxsIGNhdXNlIHRoZSBrZXJuZWwgdG8gZWF0IG5v biB0cml2aWFsCj4gYW1vdW50cyBvZiBSQU0gZm9yIHRoZSBzdHJpcGUgY2FjaGUgYnVmZmVyLiAg VGhpcyB0YWJsZSBkZW1vbnN0cmF0ZXMgdGhlCj4gZWZmZWN0IHRvZGF5IGZvciB0eXBpY2FsIFJB SUQ1LzYgZGlzayBjb3VudHMuCj4gCj4gc3RyaXBlX2NhY2hlX3NpemUJZHJpdmVzCVJBTSBjb25z dW1lZAo+IDI1NgkJCSA0CSAgNCBNQgo+IAkJCSA4CSAgOCBNQgo+IAkJCTE2CSAxNiBNQgo+IDUx MgkJCSA0CSAgOCBNQgo+IAkJCSA4CSAxNiBNQgo+IAkJCTE2CSAzMiBNQgo+IDEwMjQJCQkgNAkg MTYgTUIKPiAJCQkgOAkgMzIgTUIKPiAJCQkxNgkgNjQgTUIKPiAyMDQ4CQkJIDQJIDMyIE1CCj4g CQkJIDgJIDY0IE1CCj4gCQkJMTYJMTI4IE1CCj4gNDA5NgkJCSA0CSA2NCBNQgo+IAkJCSA4CTEy OCBNQgo+IAkJCTE2CTI1NiBNQgo+IAo+IFRoZSBwb3dlcnMgdGhhdCBiZSwgTGludXMgaW4gcGFy dGljdWxhciwgYXJlIG5vdCBmb25kIG9mIGRlZmF1bHQKPiBzZXR0aW5ncyB0aGF0IGNyZWF0ZSBh IGxvdCBvZiBrZXJuZWwgbWVtb3J5IHN0cnVjdHVyZXMuICBUaGUgZGVmYXVsdAo+IG1kLVJBSUQ1 LzYgc3RyaXBlX2NhY2hlLXNpemUgeWllbGRzIDFNQiBjb25zdW1lZCBwZXIgbWVtYmVyIGRldmlj ZS4KPiAKPiBXaXRoIFNTRHMgYmVjb21pbmcgbWFpbnN0cmVhbSwgYW5kIGJlY29taW5nIGV2ZXIg ZmFzdGVyLCBhdCBzb21lIHBvaW50Cj4gdGhlIG1kLVJBSUQ1LzYgYXJjaGl0ZWN0dXJlIHdpbGwg aGF2ZSB0byBiZSByZWRlc2lnbmVkIGJlY2F1c2Ugb2YgdGhlCj4gbWVtb3J5IGZvb3RwcmludCBy ZXF1aXJlZCBmb3IgcGVyZm9ybWFuY2UuICBDdXJyZW50bHkgdGhlIHJlcXVpcmVkIHNpemUKPiBv ZiB0aGUgc3RyaXBlIGNhY2hlIGFwcGVhcnMgZGlyZWN0bHkgcHJvcG9ydGlvbmFsIHRvIHRoZSBh Z2dyZWdhdGUgd3JpdGUKPiB0aHJvdWdocHV0IG9mIHRoZSBSQUlEIGRldmljZXMuICBUaHVzIHRo ZSBvcHRpbWFsIHZhbHVlIHdpbGwgdmFyeQo+IGdyZWF0bHkgZnJvbSBvbmUgc3lzdGVtIHRvIGFu b3RoZXIgZGVwZW5kaW5nIG9uIHRoZSB0aHJvdWdocHV0IG9mIHRoZQo+IGRyaXZlcy4KPiAKPiBG b3IgZXhhbXBsZSwgSSBhc3Npc3RlZCBhIHVzZXIgd2l0aCA1eCBJbnRlbCBTU0RzIGJhY2sgaW4g SmFudWFyeSBhbmQKPiBoaXMgc3lzdGVtIHJlcXVpcmVkIDQwOTYsIG9yIDgwTUIgb2YgUkFNIGZv ciBzdHJpcGUgY2FjaGUsIHRvIHJlYWNoCj4gbWF4aW11bSB3cml0ZSB0aHJvdWdocHV0IG9mIHRo ZSBkZXZpY2VzLiAgVGhpcyB5aWVsZGVkIDYwME1CL3Mgb3IgNjAlCj4gZ3JlYXRlciB0aHJvdWdo cHV0IHRoYW4gMjA0OCwgb3IgNDBNQiBSQU0gZm9yIGNhY2hlLiAgSW4gaGlzIGNhc2UgNjBNQgo+ IG1vcmUgUkFNIHRoYW4gdGhlIGRlZmF1bHQgd2FzIHdlbGwgd29ydGggdGhlIGluY3JlYXNlIGFz IHRoZSBtYWNoaW5lIHdhcwo+IGFuIGlTQ1NJIHRhcmdldCBzZXJ2ZXIgd2l0aCA4R0IgUkFNLgo+ IAo+IEluIHRoZSBwcmV2aW91cyBjYXNlIHdpdGggNXggcnVzdCBSQUlENiB0aGUgMjA0OCB2YWx1 ZSBzZWVtZWQgb3B0aW1hbAo+ICh0aG91Z2ggbm90IHlldCB2ZXJpZmllZCksIHJlcXVpcmluZyA0 ME1CIGxlc3MgUkFNIHRoYW4gdGhlIDV4IEludGVsCj4gU1NEcy4gIEZvciBhIDMgbW9kZXJuIHJ1 c3QgUkFJRDUgdGhlIGRlZmF1bHQgb2YgMjU2LCBvciAzTUIsIGlzIGNsb3NlIHRvCj4gb3B0aW1h bCBidXQgbWF5YmUgYSBsaXR0bGUgbG93LiAgQ29uc2lkZXIgdGhhdCAyNTYgaGFzIGJlZW4gdGhl IGRlZmF1bHQKPiBmb3IgYSB2ZXJ5IGxvbmcgdGltZSwgYW5kIHdhcyBzZWxlY3RlZCBiYWNrIHdo ZW4gYXZlcmFnZSBkcml2ZQo+IHRocm91Z2hwdXQgd2FzIG11Y2ggbXVjaCBsb3dlciwgYXMgaW4g NTBNQi9zIG9yIGxlc3MsIFNTRHMgaGFkbid0IHlldAo+IGJlZW4gaW52ZW50ZWQsIGFuZCBzeXN0 ZW0gbWVtb3JpZXMgd2VyZSBtdWNoIHNtYWxsZXIuCj4gCj4gRHVlIHRvIHRoZSBtYXNzaXZlIGRp ZmZlcmVuY2UgaW4gdGhyb3VnaHB1dCBiZXR3ZWVuIHJ1c3QgYW5kIFNTRCwgYW55Cj4gbWVhbmlu Z2Z1bCBjaGFuZ2UgaW4gdGhlIGRlZmF1bHQgcmVhbGx5IHJlcXVpcmVzIG5ldyBjb2RlIHRvIHNu aWZmIG91dAo+IHdoYXQgdHlwZSBvZiBkZXZpY2VzIGNvbnN0aXR1dGUgdGhlIGFycmF5LCBpZiB0 aGF0J3MgcG9zc2libGUsIGFuZCBpdAo+IHByb2JhYmx5IGlzbid0LCBhbmQgc2V0IGEgbG93aXNo IGRlZmF1bHQgYWNjb3JkaW5nbHkuICBBZ2FpbiwgU1NEcwo+IGRpZG4ndCBleGlzdCB3aGVuIG1k LVJBSUQgd2FzIGNvZGVkLCBub3Igd2hlbiB0aGlzIGRlZmF1bHQgd2FzIHNldCwgYW5kCj4gdGhp cyB0aHJvd3MgYSBiaWcgbW9ua2V5IHdyZW5jaCBpbnRvIHRoZXNlIHNwb2tlcy4KCkhpIFN0YW4s CgpuaWNlIGFuYWx5dGljYWwgcmVwb3J0LCBhcyB1c3VhbC4uLgoKTXkgZHVtYiBzdWdnZXN0aW9u IHdvdWxkIGJlIHRvIHNpbXBseSB1c2UgdWRldiB0bwpzZXR1cCB0aGUgZHJpdmVzLgpFdmVyeXRo aW5nLCBzdHJpcGVfY2FjaGUsIHJlYWRfYWhlYWQsIHN0Y2VyciwgZXRjLgpjYW4gYmUgY29uZmln dXJlZCwgSSBzdXBwb3NlLCBieSB1ZGV2IHJ1bGVzLgoKYnllLAoKLS0gCgpwaWVyZ2lvcmdpbwoK X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KeGZzIG1haWxp bmcgbGlzdAp4ZnNAb3NzLnNnaS5jb20KaHR0cDovL29zcy5zZ2kuY29tL21haWxtYW4vbGlzdGlu Zm8veGZzCg==