From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Chinner Subject: Re: 12x performance drop on md/linux+sw raid1 due to barriers [xfs] Date: Wed, 17 Dec 2008 10:14:11 +1100 Message-ID: <20081216231411.GG32301@disturbed> References: <18757.33373.744917.457587@tree.ty.sabi.co.uk> <20081215223857.GF32301@disturbed> <200812161039.07700.Martin@lichtvoll.de> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: Content-Disposition: inline In-Reply-To: <200812161039.07700.Martin@lichtvoll.de> Sender: linux-raid-owner@vger.kernel.org To: Martin Steigerwald Cc: linux-xfs@oss.sgi.com, Peter Grandi , Linux RAID Linux XFS List-Id: linux-raid.ids On Tue, Dec 16, 2008 at 10:39:07AM +0100, Martin Steigerwald wrote: > Am Montag 15 Dezember 2008 schrieb Dave Chinner: > > On Sun, Dec 14, 2008 at 10:02:05PM +0000, Peter Grandi wrote: > > > The purpose of barriers is to guarantee that relevant data is > > > known to be on persistent storage (kind of hardware 'fsync'). > > > > > > In effect write barrier means "tell me when relevant data is on > > > persistent storage", or less precisely "flush/sync writes now > > > and tell me when it is done". Properties as to ordering are just > > > a side effect. > > > > No, that is incorrect. > > > > Barriers provide strong ordering semantics. I/Os issued before the > > barrier must be completed before the barrier I/O, and I/Os issued > > after the barrier write must not be started before the barrier writ= e > > completes. The elevators are not allowed to re-=D0=BErder I/Os arou= nd > > barriers. > > > > This is all documented in Documentation/block/barrier.txt. Please > > read it because most of what you are saying appears to be based on > > incorrect assumptions about what barriers do. >=20 > Hmmm, so I am not completely off track it seems ;-). >=20 > What I still do not understand then is: How can write barriers + writ= e=20 > cache be slower than no write barriers + no cache? Because frequent write barriers cause ordering constraints on I/O. =46or example, in XFS log I/Os are sequential. With barriers enabled they cannot be merged by the elevator, whereas without barriers they can be merged and issued as a single I/O. =46urther, if you have no barrier I/os queued in the elevator, sorting and merging occurs across the entire queue of I/Os, not just the I/Os that have been issued after the last barrier I/O. Effectively the ordering constraints of barriers introduce more seeks by reducing the efficiency of the elevator due to constraining sorting and merging ranges. In many cases, the ordering constraints impose a higher seek penalty than the write cache can mitigate - the whole purpose of the barrier IOs is to force the cache to be flushed - so write caching does not improve performance when frequent barriers are issued. In this case, barriers are the problem and hence turning of the cache and barriers will result in higher performance. > I still would expect=20 > write barriers + write cache be in between no barriers + write cache = and=20 > no barriers + no cache performance wise. Depends entirely on the disk and the workload. Some disks are faster with wcache and barriers (e.g. laptop drives), some are faster with no wcache and no barriers (e.g. server drives).... > And would see anything else as a=20 > regression basically. No, just your usual "pick the right hardware" problem. Cheers, Dave. --=20 Dave Chinner david@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-raid" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by oss.sgi.com (8.12.11.20060308/8.12.11/SuSE Linux 0.7) with ESMTP id mBGNEg2o023124 for ; Tue, 16 Dec 2008 17:14:42 -0600 Date: Wed, 17 Dec 2008 10:14:11 +1100 From: Dave Chinner Subject: Re: 12x performance drop on md/linux+sw raid1 due to barriers [xfs] Message-ID: <20081216231411.GG32301@disturbed> References: <18757.33373.744917.457587@tree.ty.sabi.co.uk> <20081215223857.GF32301@disturbed> <200812161039.07700.Martin@lichtvoll.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <200812161039.07700.Martin@lichtvoll.de> List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Martin Steigerwald Cc: linux-xfs@oss.sgi.com, Linux RAID Linux XFS T24gVHVlLCBEZWMgMTYsIDIwMDggYXQgMTA6Mzk6MDdBTSArMDEwMCwgTWFydGluIFN0ZWlnZXJ3 YWxkIHdyb3RlOgo+IEFtIE1vbnRhZyAxNSBEZXplbWJlciAyMDA4IHNjaHJpZWIgRGF2ZSBDaGlu bmVyOgo+ID4gT24gU3VuLCBEZWMgMTQsIDIwMDggYXQgMTA6MDI6MDVQTSArMDAwMCwgUGV0ZXIg R3JhbmRpIHdyb3RlOgo+ID4gPiBUaGUgcHVycG9zZSBvZiBiYXJyaWVycyBpcyB0byBndWFyYW50 ZWUgdGhhdCByZWxldmFudCBkYXRhIGlzCj4gPiA+IGtub3duIHRvIGJlIG9uIHBlcnNpc3RlbnQg c3RvcmFnZSAoa2luZCBvZiBoYXJkd2FyZSAnZnN5bmMnKS4KPiA+ID4KPiA+ID4gSW4gZWZmZWN0 IHdyaXRlIGJhcnJpZXIgbWVhbnMgInRlbGwgbWUgd2hlbiByZWxldmFudCBkYXRhIGlzIG9uCj4g PiA+IHBlcnNpc3RlbnQgc3RvcmFnZSIsIG9yIGxlc3MgcHJlY2lzZWx5ICJmbHVzaC9zeW5jIHdy aXRlcyBub3cKPiA+ID4gYW5kIHRlbGwgbWUgd2hlbiBpdCBpcyBkb25lIi4gUHJvcGVydGllcyBh cyB0byBvcmRlcmluZyBhcmUganVzdAo+ID4gPiBhIHNpZGUgZWZmZWN0Lgo+ID4KPiA+IE5vLCB0 aGF0IGlzIGluY29ycmVjdC4KPiA+Cj4gPiBCYXJyaWVycyBwcm92aWRlIHN0cm9uZyBvcmRlcmlu ZyBzZW1hbnRpY3MuICBJL09zIGlzc3VlZCBiZWZvcmUgdGhlCj4gPiBiYXJyaWVyIG11c3QgYmUg Y29tcGxldGVkIGJlZm9yZSB0aGUgYmFycmllciBJL08sIGFuZCBJL09zIGlzc3VlZAo+ID4gYWZ0 ZXIgdGhlIGJhcnJpZXIgd3JpdGUgbXVzdCBub3QgYmUgc3RhcnRlZCBiZWZvcmUgdGhlIGJhcnJp ZXIgd3JpdGUKPiA+IGNvbXBsZXRlcy4gVGhlIGVsZXZhdG9ycyBhcmUgbm90IGFsbG93ZWQgdG8g cmUt0L5yZGVyIEkvT3MgYXJvdW5kCj4gPiBiYXJyaWVycy4KPiA+Cj4gPiBUaGlzIGlzIGFsbCBk b2N1bWVudGVkIGluIERvY3VtZW50YXRpb24vYmxvY2svYmFycmllci50eHQuIFBsZWFzZQo+ID4g cmVhZCBpdCBiZWNhdXNlIG1vc3Qgb2Ygd2hhdCB5b3UgYXJlIHNheWluZyBhcHBlYXJzIHRvIGJl IGJhc2VkIG9uCj4gPiBpbmNvcnJlY3QgYXNzdW1wdGlvbnMgYWJvdXQgd2hhdCBiYXJyaWVycyBk by4KPiAKPiBIbW1tLCBzbyBJIGFtIG5vdCBjb21wbGV0ZWx5IG9mZiB0cmFjayBpdCBzZWVtcyA7 LSkuCj4gCj4gV2hhdCBJIHN0aWxsIGRvIG5vdCB1bmRlcnN0YW5kIHRoZW4gaXM6IEhvdyBjYW4g d3JpdGUgYmFycmllcnMgKyB3cml0ZSAKPiBjYWNoZSBiZSBzbG93ZXIgdGhhbiBubyB3cml0ZSBi YXJyaWVycyArIG5vIGNhY2hlPwoKQmVjYXVzZSBmcmVxdWVudCB3cml0ZSBiYXJyaWVycyBjYXVz ZSBvcmRlcmluZyBjb25zdHJhaW50cyBvbiBJL08uCkZvciBleGFtcGxlLCBpbiBYRlMgbG9nIEkv T3MgYXJlIHNlcXVlbnRpYWwuIFdpdGggYmFycmllcnMgZW5hYmxlZAp0aGV5IGNhbm5vdCBiZSBt ZXJnZWQgYnkgdGhlIGVsZXZhdG9yLCB3aGVyZWFzIHdpdGhvdXQgYmFycmllcnMKdGhleSBjYW4g YmUgbWVyZ2VkIGFuZCBpc3N1ZWQgYXMgYSBzaW5nbGUgSS9PLgoKRnVydGhlciwgaWYgeW91IGhh dmUgbm8gYmFycmllciBJL29zIHF1ZXVlZCBpbiB0aGUgZWxldmF0b3IsIHNvcnRpbmcKYW5kIG1l cmdpbmcgb2NjdXJzIGFjcm9zcyB0aGUgZW50aXJlIHF1ZXVlIG9mIEkvT3MsIG5vdCBqdXN0IHRo ZQpJL09zIHRoYXQgaGF2ZSBiZWVuIGlzc3VlZCBhZnRlciB0aGUgbGFzdCBiYXJyaWVyIEkvTy4K CkVmZmVjdGl2ZWx5IHRoZSBvcmRlcmluZyBjb25zdHJhaW50cyBvZiBiYXJyaWVycyBpbnRyb2R1 Y2UgbW9yZQpzZWVrcyBieSByZWR1Y2luZyB0aGUgZWZmaWNpZW5jeSBvZiB0aGUgZWxldmF0b3Ig ZHVlIHRvIGNvbnN0cmFpbmluZwpzb3J0aW5nIGFuZCBtZXJnaW5nIHJhbmdlcy4KCkluIG1hbnkg Y2FzZXMsIHRoZSBvcmRlcmluZyBjb25zdHJhaW50cyBpbXBvc2UgYSBoaWdoZXIgc2VlayBwZW5h bHR5CnRoYW4gdGhlIHdyaXRlIGNhY2hlIGNhbiBtaXRpZ2F0ZSAtIHRoZSB3aG9sZSBwdXJwb3Nl IG9mIHRoZSBiYXJyaWVyCklPcyBpcyB0byBmb3JjZSB0aGUgY2FjaGUgdG8gYmUgZmx1c2hlZCAt IHNvIHdyaXRlIGNhY2hpbmcgZG9lcyBub3QKaW1wcm92ZSBwZXJmb3JtYW5jZSB3aGVuIGZyZXF1 ZW50IGJhcnJpZXJzIGFyZSBpc3N1ZWQuIEluIHRoaXMgY2FzZSwKYmFycmllcnMgYXJlIHRoZSBw cm9ibGVtIGFuZCBoZW5jZSB0dXJuaW5nIG9mIHRoZSBjYWNoZSBhbmQgYmFycmllcnMKd2lsbCBy ZXN1bHQgaW4gaGlnaGVyIHBlcmZvcm1hbmNlLgoKCgo+IEkgc3RpbGwgd291bGQgZXhwZWN0IAo+ IHdyaXRlIGJhcnJpZXJzICsgd3JpdGUgY2FjaGUgYmUgaW4gYmV0d2VlbiBubyBiYXJyaWVycyAr IHdyaXRlIGNhY2hlIGFuZCAKPiBubyBiYXJyaWVycyArIG5vIGNhY2hlIHBlcmZvcm1hbmNlIHdp c2UuCgpEZXBlbmRzIGVudGlyZWx5IG9uIHRoZSBkaXNrIGFuZCB0aGUgd29ya2xvYWQuIFNvbWUg ZGlza3MgYXJlIGZhc3Rlcgp3aXRoIHdjYWNoZSBhbmQgYmFycmllcnMgKGUuZy4gbGFwdG9wIGRy aXZlcyksIHNvbWUgYXJlIGZhc3RlciB3aXRoCm5vIHdjYWNoZSBhbmQgbm8gYmFycmllcnMgKGUu Zy4gc2VydmVyIGRyaXZlcykuLi4uCgo+IEFuZCB3b3VsZCBzZWUgYW55dGhpbmcgZWxzZSBhcyBh IAo+IHJlZ3Jlc3Npb24gYmFzaWNhbGx5LgoKTm8sIGp1c3QgeW91ciB1c3VhbCAicGljayB0aGUg cmlnaHQgaGFyZHdhcmUiIHByb2JsZW0uCgpDaGVlcnMsCgpEYXZlLgotLSAKRGF2ZSBDaGlubmVy CmRhdmlkQGZyb21vcmJpdC5jb20KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fCnhmcyBtYWlsaW5nIGxpc3QKeGZzQG9zcy5zZ2kuY29tCmh0dHA6Ly9vc3Mu c2dpLmNvbS9tYWlsbWFuL2xpc3RpbmZvL3hmcwo=