From mboxrd@z Thu Jan 1 00:00:00 1970 From: Martin Steigerwald Subject: Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?) Date: Thu, 30 Apr 2015 11:00:05 +0200 Message-ID: <4154074.ZWLyZCMjhl@merkaba> References: <8f886f13-6550-4322-95be-93244ae61045@phunq.net> <1430334326.7360.25.camel@gmail.com> <20150430002008.GY15810@dastard> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: Theodore Ts'o , tux3@tux3.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Mike Galbraith , Daniel Phillips , OGAWA Hirofumi To: Dave Chinner Return-path: In-Reply-To: <20150430002008.GY15810@dastard> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: tux3-bounces@phunq.net Sender: "Tux3" List-Id: linux-fsdevel.vger.kernel.org QW0gRG9ubmVyc3RhZywgMzAuIEFwcmlsIDIwMTUsIDEwOjIwOjA4IHNjaHJpZWIgRGF2ZSBDaGlu bmVyOgo+IE9uIFdlZCwgQXByIDI5LCAyMDE1IGF0IDA5OjA1OjI2UE0gKzAyMDAsIE1pa2UgR2Fs YnJhaXRoIHdyb3RlOgo+ID4gSGVyZSdzIHNvbWV0aGluZyB0aGF0IF9taWdodF8gaW50ZXJlc3Qg eGZzIGZvbGtzLgo+ID4gCj4gPiBjZCBnaXQgKHNvdXJjZSByZXBvc2l0b3J5IG9mIGdpdCBpdHNl bGYpCj4gPiBtYWtlIGNsZWFuCj4gPiBlY2hvIDMgPiAvcHJvYy9zeXMvdm0vZHJvcF9jYWNoZXMK PiA+IHRpbWUgbWFrZSAtajggdGVzdAo+ID4gCj4gPiBleHQ0ICAgIDJtMjAuNzIxcwo+ID4geGZz ICAgICA2bTQxLjg4N3MgPC0tIGljawo+ID4gYnRyZnMgICAxbTMyLjAzOHMKPiA+IHR1eDMgICAg MW0zMC4yNjJzCj4gPiAKPiA+IFRlc3RpbmcgYnkgQXVudCBUaWxseTogbWtmcywgbm8gZmFuY3kg c3dpdGNoZXMsIG1vdW50IHRoZSB0aGluZywgdGVzdC4KPiAKPiBUTDtEUjogUmVzdWx0cyBhcmUg KnZlcnkgZGlmZmVyZW50KiBvbiBhIDI1NkdCIFNhbXN1bmcgODQwIEVWTyBTU0QKPiB3aXRoIHNs aWdodGx5IHNsb3dlciBDUFVzIChFNS00NjIwIEAgMi4yMEdIeilpLCBhbGwgZmlsZXN5c3RlbXMK PiB1c2luZyBkZWZhdWx0czoKPiAKPiAJcmVhbAkJdXNlcgkJc3lzCj4geGZzCTNtMTYuMTM4cwk3 bTguMzQxcwkxNG0zMi40NjJzCj4gZXh0NAkzbTE4LjA0NXMJN203Ljg0MHMJMTRtMzIuOTk0cwo+ IGJ0cmZzCTNtNDUuMTQ5cwk3bTEwLjE4NHMJMTZtMzAuNDk4cwo+IAo+IFdoYXQgeW91IGFyZSBz ZWVpbmcgaXMgcGh5c2ljYWwgc2VlayBkaXN0YW5jZXMgaW1wYWN0aW5nIHJlYWQKPiBwZXJmb3Jt YW5jZS4gIFhGUyBkb2VzIG5vdCBvcHRpbWlzZSBmb3IgbWluaW1hbCBwaHlzaWNhbCBzZWVrCj4g ZGlzdGFuY2UsIGFuZCBoZW5jZSBpcyBzbG93ZXIgdGhhbiBmaWxlc3l0c2VtcyB0aGF0IGRvIG9w dGltaXNlIGZvcgo+IG1pbmltYWwgc2VlayBkaXN0YW5jZS4gVGhpcyBzaG93cyB1cCBlc3BlY2lh bGx5IHdlbGwgb24gc2xvdyBzaW5nbGUKPiBzcGluZGxlcy4KPiAKPiBYRlMgaXMgKmFkZXF1YXRl KiBmb3IgdGhlIHVzZSBvbiBzbG93IHNpbmdsZSBkcml2ZXMsIGJ1dCBpdCBpcwo+IHJlYWxseSBk ZXNpZ25lZCBmb3IgYmVzdCBwZXJmb3JtYW5jZSBvbiBzdG9yYWdlIGhhcmR3YXJlIHRoYXQgaXMg bm90Cj4gc2VlayBkaXN0YW5jZSBzZW5zaXRpdmUuCj4gCj4gSU9XUywgWEZTIGp1c3QgaGF0ZXMg eW91ciBkaXNrLiBTcGVuZCAkNTAgYW5kIGJ1eSBhIGNoZWFwIFNTRCBhbmQKPiB0aGUgcHJvYmxl bSBnb2VzIGF3YXkuIDopCgoKSSBhbSBxdWl0ZSBzdXJwcmlzZWQgdGhhdCBhIHRyYWRpdGlvbmFs IGZpbGVzeXN0ZW0gdGhhdCB3YXMgY3JlYXRlZCBpbiB0aGUgCmFnZSBvZiByb3RhdGluZyBtZWRp YSBkb2VzIG5vdCBsaWtlIHRoaXMga2luZCBvZiBtZWRpYSBhbmQgZXZlbiBzZWVtcyB0byAKZXhj ZWwgb24gQlRSRlMgb24gdGhlIG5ldyBub24gcm90YXRpbmcgbWVkaWEgYXZhaWxhYmxlLgoKQnV0 4oCmCgo+IC0tLS0KPiAKPiBBbmQgbm93IGluIG1vcmUgZGV0YWlsLgo+IAo+IEl0J3MgZWFzeSB0 byBiZSBmYXN0IG9uIGVtcHR5IGZpbGVzeXN0ZW1zLiBYRlMgZG9lcyBub3QgYWltIHRvIGJlCj4g ZmFzdCBpbiBzdWNoIHNpdHVhdGlvbnMgLSBpdCBhaW1zIHRvIGhhdmUgY29uc2lzdGVudCBwZXJm b3JtYW5jZQo+IGFjcm9zcyB0aGUgbGlmZSBvZiB0aGUgZmlsZXN5c3RlbS4KCuKApiB0aGlzIGlz IGEgcXVpdGUgaW1wb3J0YW50IGFkZGl0aW9uLgoKPiBUaGluZyBpcywgb25jZSB5b3UndmUgYWJ1 c2VkIHRob3NlIGZpbGVzeXRzZW1zIGZvciBhIGNvdXBsZSBvZgo+IG1vbnRocywgdGhlIGZpbGVz IGluIGV4dDQsIGJ0cmZzIGFuZCB0dXgzIGFyZSBub3QgZ29pbmcgdG8gYmUgbGFpZAo+IG91dCBw ZXJmZWN0bHkgb24gdGhlIG91dGVyIGVkZ2Ugb2YgdGhlIGRpc2suIFRoZXknbGwgYmUgc3ByZWFk IGFsbAo+IG92ZXIgdGhlIHBsYWNlIGFuZCBzbyBhbGwgdGhlIGZpbGVzeXN0ZW1zIHdpbGwgYmUg c2VlaW5nIGxhcmdlIHNlZWtzCj4gb24gcmVhZC4gVGhlIHRoaW5nIGlzLCBYRlMgd2lsbCBoYXZl IHJvdWdobHkgdGhlIHNhbWUgcGVyZm9ybWFuY2UgYXMKPiB3aGVuIHRoZSBmaWxlc3lzdGVtIGlz IGVtcHR5IGJlY2F1c2UgdGhlIHNwcmVhZGluZyBvZiB0aGUgYWxsb2NhdGlvbgo+IGFsbG93cyBp dCB0byBtYWludGFpbiBiZXR0ZXIgbG9jYWxpdHkgYW5kIHNlcGFyYXRpb24gYW5kIGhlbmNlCj4g ZG9lc24ndCBmcmFnbWVudCBmcmVlIHNwYWNlIG5lYXJseSBhcyBiYWRseSBhcyB0aGUgb2hlciBm aWxlc3lzdGVtcy4KPiBGcmVlIHNwYWNlIGZyYWdtZW50YXRpb24gaXMgd2hhdCBsZWFkcyB0byBw ZXJmb3JtYW5jZSBkZWdyYWRhdGlvbiBpbgo+IGZpbGVzeXN0ZW1zLCBhbmQgYWxsIHRoZSBvdGhl ciBmaWxlc3lzdGVtIHdpbGwgaGF2ZSBkZWdyYWRlZCB0byBiZQo+ICptdWNoIHdvcnNlKiB0aGFu IFhGUy4KCkkgZXZlbiBzdGlsbCBzZWUgaHVuZ3Mgb24gd2hhdCBJIHRlbmQgdG8gc2VlIGFzIGZy ZWVzcGFjZSBmcmFnbWVudGF0aW9uIGluIApCVFJGUy4gTXkgL2hvbWUgb24gYSBEdWFsICghKSBC VFJGUyBTU0Qgc2V0dXAgY2FuIGJhc2ljYWxseSBzdGFsbCB0byBhIApoYWx0IHdoZW4gaXQgaGFz IHJlc2VydmVkIGFsbCBzcGFjZSBvZiB0aGUgZGV2aWNlIGZvciBjaHVua3MuIFNvIHRoaXMKCm1l cmthYmE6fj4gYnRyZnMgZmkgc2ggL2hvbWUKTGFiZWw6ICdob21lJyAgdXVpZDogW+KApl0KICAg ICAgICBUb3RhbCBkZXZpY2VzIDIgRlMgYnl0ZXMgdXNlZCAxMjkuNDhHaUIKICAgICAgICBkZXZp ZCAgICAxIHNpemUgMTcwLjAwR2lCIHVzZWQgMTQ2LjAzR2lCIHBhdGggL2Rldi9tYXBwZXIvbXNh dGEtCmhvbWUKICAgICAgICBkZXZpZCAgICAyIHNpemUgMTcwLjAwR2lCIHVzZWQgMTQ2LjAzR2lC IHBhdGggL2Rldi9tYXBwZXIvc2F0YS0KaG9tZQoKQnRyZnMgdjMuMTgKbWVya2FiYTp+PiBidHJm cyBmaSBkZiAvaG9tZQpEYXRhLCBSQUlEMTogdG90YWw9MTQyLjAwR2lCLCB1c2VkPTEyNi43Mkdp QgpTeXN0ZW0sIFJBSUQxOiB0b3RhbD0zMi4wME1pQiwgdXNlZD00OC4wMEtpQgpNZXRhZGF0YSwg UkFJRDE6IHRvdGFsPTQuMDBHaUIsIHVzZWQ9Mi43NkdpQgpHbG9iYWxSZXNlcnZlLCBzaW5nbGU6 IHRvdGFsPTUxMi4wME1pQiwgdXNlZD0wLjAwQgoKaXMgc2FmZSwgYnV0IG9uZSBJIGhhdmUgc2l6 ZSAxNzAgR2lCIHVzZXIgMTcwIEdpQiwgZXZlbiBpZiBpbnNpZGUgdGhlIApjaHVua3MgdGhlcmUg aXMgZW5vdWdoIGZyZWUgc3BhY2UgdG8gYWxsb2NhdGUgZnJvbSwgZW5vdWdoIGFzIGluIDMwLTQw IApHaUIsIGl0IGNhbiBoYXBwZW4gdGhhdCB3cml0ZXMgYXJlIHN0YWxsZWQgdXAgdG8gdGhlIHBv aW50IHRoYXQgCmFwcGxpY2F0aW9ucyBvbiB0aGUgZGVza3RvcCBmcmVlemUgYW5kIEkgc2VlIGh1 bmcgdGFzayBtZXNzYWdlcyBpbiBrZXJuZWwgCmxvZy4KClRoaXMgaXMgdGhlIGNhc2UgdXB0byBr ZXJuZWwgNC4wLiBJIGhhdmUgc2VlbiBDaHJpcyBNYXNvbiBmaXhpbmcgc29tZSB3cml0ZSAKc3Rh bGxzIGZvciBiaWcgZmFjZWJvb2sgc2V0dXBzLCBtYXliZSBpdCB3aWxsIGhlbHAgaGVyZSwgYnV0 IHVubGVzcyB0aGlzIAppc3N1ZSBpcyBmaXhlZCwgSSB0aGluayBCVFJGUyBpcyBub3QgeWV0IGZ1 bGx5IHByb2R1Y3Rpb24gcmVhZHksIHVubGVzcyB5b3UgCmxlYXZlICpodWdlKiBhbW91bnQgb2Yg ZnJlZSBzcGFjZSwgYXMgaW4gZm9yIDIwMCBHaUIgb2YgZGF0YSB5b3Ugd2FudCB0byAKd3JpdGUg bWFrZSBhIDQwMCBHaUIgdm9sdW1lLgoKPiBQdXQgc2ltcGx5OiBlbXB0eSBmaWxlc3lzdGVtIGJl bmNobWFya2luZyBkb2VzIG5vdCBzaG93IHRoZSByZWFsCj4gcGVyZm9ybWFuY2Ugb2YgdGhlIGZp bGVzeXN0ZW0gdW5kZXIgc3VzdGFpbmVkIHByb2R1Y3Rpb24gd29ya2xvYWRzLgo+IEhlbmNlIGJl bmNobWFya3MgbGlrZSB0aGlzIC0gd2hpbGUgaW50ZXJlc3RpbmcgZnJvbSBhIHRoZW9yZXRpY2Fs Cj4gcG9pbnQgb2YgdmlldyBhbmQgYXJlIHdpZGVseSB1c2VkIGZvciBicmFnZ2luZyBhYm91dCB3 aG9zZSBnb3QgdGhlCj4gZmFzdGVzdCAtIGFyZSBtb3N0bHkgaXJyZWxldmFudCB0byBkZXRlcm1p bmluZyBob3cgdGhlIGZpbGVzeXN0ZW0KPiB3aWxsIHBlcmZvcm0gaW4gcHJvZHVjdGlvbiBlbnZp cm9ubWVudHMuCj4gCj4gV2UgY2FuIGFsc28gbG9vayBhdCB0aGlzIGFsZ29yaXRobSBpbiBhIGRp ZmZlcmVudCB3YXk6IHRha2UgYSBsYXJnZQo+IGZpbGVzeXN0ZW0gKHNheSBhIGZldyBodW5kcmVk IFRCKSBhY3Jvc3MgYSBmZXcgdGVucyBvZiBkaXNrcyBpbiBhCj4gbGluZWFyIGNvbmNhdC4gIGV4 dDQsIGJ0cmZzIGFuZCB0dXgzIHdpbGwgb25seSBoaXQgdGhlIGZpcnN0IGRpc2sgaW4KPiB0aGUg Y29uY2F0LCBhbmQgc28gZ28gbm8gZmFzdGVyIGJlY2F1c2UgdGhleSBhcmUgc3RpbGwgYm91bmQg YnkKPiBwaHlzaWNhbCBzZWVrIHRpbWVzLiAgWEZTLCBob3dldmVyLCB3aWxsIHNwcmVhZCB0aGUg bG9hZCBhY3Jvc3MgbWFueQo+IChpZiBub3QgYWxsKSBvZiB0aGUgZGlza3MsIGFuZCBzbyBlZmZl Y3RpdmVseSByZWR1Y2UgdGhlIGF2ZXJhZ2UKPiBzZWVrIHRpbWUgYnkgdGhlIG51bWJlciBvZiBk aXNrcyBkb2luZyBjb25jdXJyZW50IElPLiBUaGVuIHlvdSdsbAo+IHNlZSB0aGF0IGFwcGxpY2F0 aW9uIGxldmVsIElPIGNvbmN1cnJlbmN5IGJlY29tZXMgdGhlIHBlcmZvcm1hbmNlCj4gbGltaXRh dGlvbiwgbm90IHRoZSBwaHlzaWNhbCBzZWVrIHRpbWUgb2YgdGhlIGhhcmR3YXJlLgoKVGhhdCBh cmUgdGhlIGFsbG9jYXRpb24gZ3JvdXBzLiBJIGFsd2F5cyB3b25kZXJlZCBob3cgaXQgY2FuIGJl IGJlbmVmaWNpYWwgCnRvIHNwcmVhZCB0aGUgYWxsb2NhdGlvbnMgb250byA0IGFyZWFzIG9mIG9u ZSBwYXJ0aXRpb24gb24gZXhwZW5zaXZlIHNlZWsgCm1lZGlhLiBOb3cgdGhhdCBtYWtlcyBiZXR0 ZXIgc2Vuc2UgZm9yIG1lLiBJIGFsd2F5cyBoYWQgdGhlIGd1dCBpbXByZXNzaW9uIAp0aGF0IFhG UyBtYXkgbm90IGJlIHRoZSBmYXN0ZXN0IGluIGFsbCBjYXNlcywgYnV0IGl0IGlzIG9uZSBvZiB0 aGUgCmZpbGVzeXN0ZW0gd2l0aCB0aGUgbW9zdCBjb25zaXN0ZW50IHBlcmZvcm1hbmNlIG92ZXIg dGltZSwgYnV0IG5ldmVyIHdhcyAKYWJsZSB0byBmdWxseSBleHBsYWluIHdoeSB0aGF0IGlzLgoK VGhhbmtzLAotLSAKTWFydGluICdIZWxpb3MnIFN0ZWlnZXJ3YWxkIC0gaHR0cDovL3d3dy5MaWNo dHZvbGwuZGUKR1BHOiAwM0IwIDBENkMgMDA0MCAwNzEwIDRBRkEgIEI4MkYgOTkxQiBFQUFDIEE1 OTkgODRDNwoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K VHV4MyBtYWlsaW5nIGxpc3QKVHV4M0BwaHVucS5uZXQKaHR0cDovL3BodW5xLm5ldC9tYWlsbWFu L2xpc3RpbmZvL3R1eDMK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751476AbbD3JAT (ORCPT ); Thu, 30 Apr 2015 05:00:19 -0400 Received: from mondschein.lichtvoll.de ([194.150.191.11]:48489 "EHLO mail.lichtvoll.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751056AbbD3JAO convert rfc822-to-8bit (ORCPT ); Thu, 30 Apr 2015 05:00:14 -0400 From: Martin Steigerwald To: Dave Chinner Cc: Mike Galbraith , Daniel Phillips , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, tux3@tux3.org, "Theodore Ts'o" , OGAWA Hirofumi Subject: Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?) Date: Thu, 30 Apr 2015 11:00:05 +0200 Message-ID: <4154074.ZWLyZCMjhl@merkaba> User-Agent: KMail/4.14.7 (Linux/4.0.0-tp520-btrfs-trim+; KDE/4.14.2; x86_64; git-38b5d90; 2015-04-16) In-Reply-To: <20150430002008.GY15810@dastard> References: <8f886f13-6550-4322-95be-93244ae61045@phunq.net> <1430334326.7360.25.camel@gmail.com> <20150430002008.GY15810@dastard> MIME-Version: 1.0 Content-Transfer-Encoding: 8BIT Content-Type: text/plain; charset="utf-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Am Donnerstag, 30. April 2015, 10:20:08 schrieb Dave Chinner: > On Wed, Apr 29, 2015 at 09:05:26PM +0200, Mike Galbraith wrote: > > Here's something that _might_ interest xfs folks. > > > > cd git (source repository of git itself) > > make clean > > echo 3 > /proc/sys/vm/drop_caches > > time make -j8 test > > > > ext4 2m20.721s > > xfs 6m41.887s <-- ick > > btrfs 1m32.038s > > tux3 1m30.262s > > > > Testing by Aunt Tilly: mkfs, no fancy switches, mount the thing, test. > > TL;DR: Results are *very different* on a 256GB Samsung 840 EVO SSD > with slightly slower CPUs (E5-4620 @ 2.20GHz)i, all filesystems > using defaults: > > real user sys > xfs 3m16.138s 7m8.341s 14m32.462s > ext4 3m18.045s 7m7.840s 14m32.994s > btrfs 3m45.149s 7m10.184s 16m30.498s > > What you are seeing is physical seek distances impacting read > performance. XFS does not optimise for minimal physical seek > distance, and hence is slower than filesytsems that do optimise for > minimal seek distance. This shows up especially well on slow single > spindles. > > XFS is *adequate* for the use on slow single drives, but it is > really designed for best performance on storage hardware that is not > seek distance sensitive. > > IOWS, XFS just hates your disk. Spend $50 and buy a cheap SSD and > the problem goes away. :) I am quite surprised that a traditional filesystem that was created in the age of rotating media does not like this kind of media and even seems to excel on BTRFS on the new non rotating media available. But… > ---- > > And now in more detail. > > It's easy to be fast on empty filesystems. XFS does not aim to be > fast in such situations - it aims to have consistent performance > across the life of the filesystem. … this is a quite important addition. > Thing is, once you've abused those filesytsems for a couple of > months, the files in ext4, btrfs and tux3 are not going to be laid > out perfectly on the outer edge of the disk. They'll be spread all > over the place and so all the filesystems will be seeing large seeks > on read. The thing is, XFS will have roughly the same performance as > when the filesystem is empty because the spreading of the allocation > allows it to maintain better locality and separation and hence > doesn't fragment free space nearly as badly as the oher filesystems. > Free space fragmentation is what leads to performance degradation in > filesystems, and all the other filesystem will have degraded to be > *much worse* than XFS. I even still see hungs on what I tend to see as freespace fragmentation in BTRFS. My /home on a Dual (!) BTRFS SSD setup can basically stall to a halt when it has reserved all space of the device for chunks. So this merkaba:~> btrfs fi sh /home Label: 'home' uuid: […] Total devices 2 FS bytes used 129.48GiB devid 1 size 170.00GiB used 146.03GiB path /dev/mapper/msata- home devid 2 size 170.00GiB used 146.03GiB path /dev/mapper/sata- home Btrfs v3.18 merkaba:~> btrfs fi df /home Data, RAID1: total=142.00GiB, used=126.72GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=4.00GiB, used=2.76GiB GlobalReserve, single: total=512.00MiB, used=0.00B is safe, but one I have size 170 GiB user 170 GiB, even if inside the chunks there is enough free space to allocate from, enough as in 30-40 GiB, it can happen that writes are stalled up to the point that applications on the desktop freeze and I see hung task messages in kernel log. This is the case upto kernel 4.0. I have seen Chris Mason fixing some write stalls for big facebook setups, maybe it will help here, but unless this issue is fixed, I think BTRFS is not yet fully production ready, unless you leave *huge* amount of free space, as in for 200 GiB of data you want to write make a 400 GiB volume. > Put simply: empty filesystem benchmarking does not show the real > performance of the filesystem under sustained production workloads. > Hence benchmarks like this - while interesting from a theoretical > point of view and are widely used for bragging about whose got the > fastest - are mostly irrelevant to determining how the filesystem > will perform in production environments. > > We can also look at this algorithm in a different way: take a large > filesystem (say a few hundred TB) across a few tens of disks in a > linear concat. ext4, btrfs and tux3 will only hit the first disk in > the concat, and so go no faster because they are still bound by > physical seek times. XFS, however, will spread the load across many > (if not all) of the disks, and so effectively reduce the average > seek time by the number of disks doing concurrent IO. Then you'll > see that application level IO concurrency becomes the performance > limitation, not the physical seek time of the hardware. That are the allocation groups. I always wondered how it can be beneficial to spread the allocations onto 4 areas of one partition on expensive seek media. Now that makes better sense for me. I always had the gut impression that XFS may not be the fastest in all cases, but it is one of the filesystem with the most consistent performance over time, but never was able to fully explain why that is. Thanks, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7