How do I pick the best fs?

How do I pick the best fs?

[Paul Furness]

Hi.

I'm afraid this is a bit of a broad question, but I can't think of a
better place to ask friendly people for some ideas. :)

I'm building a new production server. It has a big hardware RAID
attached, such that linux simply sees a single large SCSI disk (1.4TB to
start with). I'm using LVM to deal with partitioning (which is so far
working absolutely perfectly), but I need to choose a file system type
that will be the optimum for me. Not that it's likely to really matter
for the purposes of this question, but this thing is build on RedHat 7.3
+ patches, and then I have installed 2.4.20 kernel.

Oh, I should probably explain that the idea is to have a number of
partitions on the RAID, each shared separately. The shares will be both
NFS and Samba.

The three best looking candidates are XFS, Reiserfs and Ext3. Like
everyone, I want everything from this system - speed, reliability,
resilience to failures. Probably the most important single thing is
speed, but it's a photo finish with reliability. Resilience to things
like power failures is perhaps less important, since it's on our main
UPS and I take backups every day. :)

Ext3 is described as very reliable, resilient, but not the fastest
around. Both Reiserfs and XFS are described as quick, but I get the
feeling that they are more experimental.

So, has anyone actually compared them? Ideally, I'd love to find a nice
table somewhere on the internet labeled "File systems that Paul is
testing, with the best shown in red." :) 

I'm quite happy to do comparisons between them before it goes live, but
I have never really done this before, so I don't really know how to go
about it (I guess I'd have to think up a test that does a lot of file
operations that exceed the size of any and all caches on the machine).

Even better, I suppose, would be a linux based file system testing /
bench marking program. However, I know nothing at all about what to look
for.

Can anyone make any suggestions?

Tks.

Paul.


-- 
Paul Furness

Systems Manager
Visual Information Lab
Mitsubsihi Electric ITE BV
Guildford, UK
 __________________________________________________________
|  Fight Spam! Join EuroCAUCE: http://www.euro.cauce.org/  |
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Re: How do I pick the best fs?

[Jorge R . Csapo]

Hi Paul, this is not a direct answer to your question, but while reading it
something occurred to me that you may find useful. A few years ago I worked for
a bank and was asked to compare performance between different file servers.
My choices didn't include RAID (we're talking circa 1995 here) but I had to
decide to go with one of a) NFS server and PC-NFS equipped Windows clients,
b) Samba servers and Win 95 clients. The servers would be either Intel or Alpha
processors. After a lot of benchmarks I discovered that the number of
simultaneous clients on any given server mattered far more than the difference
between the servers. For instance, one NT server serving one Win station would
outperform everything else. However, when I hooked around 8 clients the
picture began to change. From 10 to 20 clients Samba was the best choice and
upwards of 60 clients the Alpha processor made enough of a difference that the
bank decided to pay about twice as much for those than it would for comparable
Intel machines.

So, when you run your benchmarks try to test with different numbers of clients
for each situation. It's worth the trouble.

Jorge

assim falou Paul Furness (em 18/03/2003):
> Hi.
> 
> I'm afraid this is a bit of a broad question, but I can't think of a
> better place to ask friendly people for some ideas. :)
> 
> I'm building a new production server. It has a big hardware RAID
> attached, such that linux simply sees a single large SCSI disk (1.4TB to
> start with). I'm using LVM to deal with partitioning (which is so far
> working absolutely perfectly), but I need to choose a file system type
> that will be the optimum for me. Not that it's likely to really matter
> for the purposes of this question, but this thing is build on RedHat 7.3
> + patches, and then I have installed 2.4.20 kernel.
> 
> Oh, I should probably explain that the idea is to have a number of
> partitions on the RAID, each shared separately. The shares will be both
> NFS and Samba.
> 
> The three best looking candidates are XFS, Reiserfs and Ext3. Like
> everyone, I want everything from this system - speed, reliability,
> resilience to failures. Probably the most important single thing is
> speed, but it's a photo finish with reliability. Resilience to things
> like power failures is perhaps less important, since it's on our main
> UPS and I take backups every day. :)
> 
> Ext3 is described as very reliable, resilient, but not the fastest
> around. Both Reiserfs and XFS are described as quick, but I get the
> feeling that they are more experimental.
> 
> So, has anyone actually compared them? Ideally, I'd love to find a nice
> table somewhere on the internet labeled "File systems that Paul is
> testing, with the best shown in red." :) 
> 
> I'm quite happy to do comparisons between them before it goes live, but
> I have never really done this before, so I don't really know how to go
> about it (I guess I'd have to think up a test that does a lot of file
> operations that exceed the size of any and all caches on the machine).
> 
> Even better, I suppose, would be a linux based file system testing /
> bench marking program. However, I know nothing at all about what to look
> for.
> 
> Can anyone make any suggestions?
> 
> Tks.
> 
> Paul.
> 
> 
> -- 
> Paul Furness
> 
> Systems Manager
> Visual Information Lab
> Mitsubsihi Electric ITE BV
> Guildford, UK
>  __________________________________________________________
> |  Fight Spam! Join EuroCAUCE: http://www.euro.cauce.org/  |
>  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-admin" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Jorge R. Csapo
--------------------------------------------------
 /"\
 \ / CAMPANHA DA FITA ASCII - CONTRA MAIL HTML
  X  ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
 / \
--------------------------------------------------
http://www.completo.com.br/~jorge
===========================================
With a PC, I always felt limited
by the software available.
On Unix, I am limited only by my knowledge.
--Peter J. Schoenster

Re: How do I pick the best fs?

[Jamie Harris]

I spent about a day looking at the various comparisions of ReiserFS Vs
other filesystems, it was mainly between ReiserFS/Ext3 in the end I went
for ReiserFS as it seems to have best worst-case performance, and my
server handles a lot of small (<1K quite often) files, which ReiserFS
seems pretty good at.  I can't say I've had any problems to date.

I'd say you need to have a damn good readup on all the file systems,
decide which points are most important (efficiency with you most common
file size, speed of different operations, recovery speed) and make a
decision based on that.

Any good resources you do find would be greatly appreciated as its a
subject that it seems takes quite a bit of time searching around!

cheers, good luck

Jamie...

-- 
**  This message was transmitted on 100% recycled electrons **

Re: How do I pick the best fs?

[urgrue]

ive been using reiserfs for over 3 years now, and i definitely recommend it.
reiser was at 2.x while ext3 was still 0.1x. granted that doesnt really 
prove anything, but id been using reiser in production use for over two 
years before ext3 became stable.
so basically my opinion is that your impression about reiser being more 
experimental is wrong. i definitely feel its the most mature of the three.
i think reiser's web page had some benchmarks and comparisons 
(www.namesys.com  iirc).
there's also an old filesystem benchmark program called bonnie, i think 
suse has updated it and released it as bonnie+ or something like that.



>Hi.
>
>I'm afraid this is a bit of a broad question, but I can't think of a
>better place to ask friendly people for some ideas. :)
>
>I'm building a new production server. It has a big hardware RAID
>attached, such that linux simply sees a single large SCSI disk (1.4TB to
>start with). I'm using LVM to deal with partitioning (which is so far
>working absolutely perfectly), but I need to choose a file system type
>that will be the optimum for me. Not that it's likely to really matter
>for the purposes of this question, but this thing is build on RedHat 7.3
>+ patches, and then I have installed 2.4.20 kernel.
>
>Oh, I should probably explain that the idea is to have a number of
>partitions on the RAID, each shared separately. The shares will be both
>NFS and Samba.
>
>The three best looking candidates are XFS, Reiserfs and Ext3. Like
>everyone, I want everything from this system - speed, reliability,
>resilience to failures. Probably the most important single thing is
>speed, but it's a photo finish with reliability. Resilience to things
>like power failures is perhaps less important, since it's on our main
>UPS and I take backups every day. :)
>
>Ext3 is described as very reliable, resilient, but not the fastest
>around. Both Reiserfs and XFS are described as quick, but I get the
>feeling that they are more experimental.
>
>So, has anyone actually compared them? Ideally, I'd love to find a nice
>table somewhere on the internet labeled "File systems that Paul is
>testing, with the best shown in red." :)
>
>I'm quite happy to do comparisons between them before it goes live, but
>I have never really done this before, so I don't really know how to go
>about it (I guess I'd have to think up a test that does a lot of file
>operations that exceed the size of any and all caches on the machine).
>
>Even better, I suppose, would be a linux based file system testing /
>bench marking program. However, I know nothing at all about what to look
>for.
>
>Can anyone make any suggestions?
>
>Tks.
>
>Paul.
>
>
>--
>Paul Furness
>
>Systems Manager
>Visual Information Lab
>Mitsubsihi Electric ITE BV
>Guildford, UK
>  __________________________________________________________
>|  Fight Spam! Join EuroCAUCE: http://www.euro.cauce.org/  |
>  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-admin" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How do I pick the best fs?

[Andreas Happe]

In article <5.1.0.14.0.20030318201306.00b91c28@mail.tumsan.fi>, urgrue wrote:
> ive been using reiserfs for over 3 years now, and i definitely recommend it.
> reiser was at 2.x while ext3 was still 0.1x. granted that doesnt really 
> prove anything,

yeah, that means absolutly nothing.

>but id been using reiser in production use for over two 
> years before ext3 became stable.

well, I've used reiser for 9 months and it destroyed my data twice, xfs
was a lot more stable and finally I've settled back for good old ext3
(Ok, i'm staying most of my time in "unstable" 2.5 kernels). Have you
considered all the good ext2 infrastructure that is included in ext3?

> so basically my opinion is that your impression about reiser being more 
> experimental is wrong.

NACK. I think that most alternatives (ext3|xfs) should suit him better.
And ext3 got the big advantage that it is backwards compatible to ext2
which means that you can use all that industry - strength tools like
ext2.fsck to recover your data _if_ you've lost data... last time i've
checked (i had lost data) fsck.reiserfs was, well, not so good.

>i definitely feel its the most mature of the three.

what the fsck have you been smoking?

I remember a time when I could play games guessing which files on my
reiserfs partition were overwrite by zeros.

> i think reiser's web page had some benchmarks and comparisons 
> (www.namesys.com  iirc).

lies, damn lies and statistics?

If you head for stability choose ext3 or xfs, if you also need realtime
read/write access to your files use xfs. If you want to face the "wow
it's new, it's shiny, you must use it" - crowd, use ext3.

Andreas
-- 
15. I will never employ any device with a digital countdown. If I find that 
    such a device is absolutely unavoidable, I will set it to activate when 
    the counter reaches 117 and the hero is just putting his plan into 
    --Peter Anspach's list of things to do as an Evil Overlord

Re: How do I pick the best fs?

[urgrue]

>well, I've used reiser for 9 months and it destroyed my data twice, xfs
>was a lot more stable and finally I've settled back for good old ext3

well, ive been using reiser for over three years on over 200 always-on 
production servers.
while ive only tried ext3 on maybe 5 or less computers, twice its caused 
problems (one may have been due to a faulty hard drive though)
but, these are all anecdotes, and the plural of anecdote is not data. i 
might be lucky, you might be unlucky, i might be using more stable versions 
while you tried buggy versions, whatever.

>(Ok, i'm staying most of my time in "unstable" 2.5 kernels). Have you

which, of course, might have something to do with bad experiences. 
admittedly though i dont know what version of reiserfs is in the 2.5 kernels.

>checked (i had lost data) fsck.reiserfs was, well, not so good.

ive never had the misfortune to need to use it, but yes admittedly the 
reiserfsck docs were filled with warnings about its unfinished/experimental 
nature - at least last i checked, which was easily a year ago.

 >lies, damn lies and statistics?

i know, i dont place almost any value on benchmarks, especially 
non-independent ones, but the poster was asking for them, so i obliged.

Re: How do I pick the best fs?

[Andreas Happe]

In article <5.1.0.14.0.20030319234642.00b76038@mail.tumsan.fi>, urgrue wrote:
> well, ive been using reiser for over three years on over 200 always-on 
> production servers.

~10 "can't call them production servers, but servers used 24/7" + 5 toy
pcs... and I've got lots of problems with reiser... pure luck?

> while ive only tried ext3 on maybe 5 or less computers, twice its caused 
> problems (one may have been due to a faulty hard drive though)
> but, these are all anecdotes, and the plural of anecdote is not data.

true and I think that with all the low latency work akpm and others have
done in the past ext3 should be one of the stablest fs around (it may
not scale as good as any of the new journaled fs, but I don't think that
the OP will build an 8+way server :)). It also reflects my experience
gained on lkml, plus fs related newsgroups. You should also not forget
the years of usage that ext2 had. In that light the "oldest" journaling
filesystem for linux is not quite that old.

> admittedly though i dont know what version of reiserfs is in the 2.5 kernels.

the newest ;). no, i don't mean reiser v4.. which looks very
interessting if it gets stable enough.
 
> ive never had the misfortune to need to use it, but yes admittedly the 
> reiserfsck docs were filled with warnings about its unfinished/experimental 
> nature - at least last i checked, which was easily a year ago.

well, I've needed them a little bit faster.

> >lies, damn lies and statistics?
> i know, i dont place almost any value on benchmarks, especially 
> non-independent ones, but the poster was asking for them, so i obliged.

I think the known online linux magazines had more than enough tests, I
think IBM developer works had a good series on filesystems (okay, they
may not be _that_ independent (JFS), but they're are good start)..
 
Andreas
-- 
"Red Hat 5.2 puts all the commercial stuff on a seperate CD so you
know clearly what is what, and you can stick that CD in your
microwave and watch the pretty lights if you wish." -- Alan Cox 

Re: How do I pick the best fs?

[Paul Furness]

Hmm, I can see I've opened a sacred cow... ;)

Thanks, everyone, for comments and experiences. Admittedly, I still
haven't decided, but the small amount of testing / experience I have had
suggests that reiserfs is definately faster than Ext3. Most of these
tests have been on my workstation, and have been far from scientific.
However, I can't say that I've lost data with either file system. I
haven't needed to do much with fsck, though.

Having said that, I don't expect I'll be needing to use fsck a lot; this
is not because I have any illusions about "it won't happen to me" but
more because I've designed the system from the start using new hardware
with lots of redundancy in it (dual PSU on both the disk array and the
server, dual CPU in the server, UPS powering the whole collection and
all the disks are in a hardware RAID array).  

XFS is very much an unknown to me, and I'm not sure if I can use it
fully anyway: I'm slightly constrained because I need to use Samba to
share the file systems as well as NFS. I'm not yet clear on whether the
acl feature of XFS will work with samba; if it won't, then the main
reason for picking XFS over the others evaporates and the field is again
level. In that case, reiserfs probably has a slight edge because I've
actually used it in anger (on my workstation) for a good couple of
months and it certainly seems quicker than Ext3.


The jury is still out on this; I'd still welcome any thoughts anyone
else wants to add.

Thanks again,

Paul.


-- 
Paul Furness

Systems Manager
Visual Information Lab
Mitsubsihi Electric ITE BV
Guildford, UK
 __________________________________________________________
|  Fight Spam! Join EuroCAUCE: http://www.euro.cauce.org/  |
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Re: How do I pick the best fs?

[Andreas Happe]

In article <1048160854.23760.21.camel@Zebra.vil.ite.mee.com>, Paul Furness wrote:
> Hmm, I can see I've opened a sacred cow... ;)

fs? vi/emacs? monolithic/microkernel? ;) it's all the same
 
> XFS is very much an unknown to me, and I'm not sure if I can use it
> fully anyway: I'm slightly constrained because I need to use Samba to
> share the file systems as well as NFS.

there were a couple of problems using reiserfs with nfs, but I think
they should be gone by now.

the "typical" filesystem which was in use with samba/acl is/was xfs
(because of its acl features and speed compared to ext3/reiserfs). I
dunno how this has changed since the acl bits have moved into a higher
level (at least in 2.5 if I remember it correctly).

>I'm not yet clear on whether the
> acl feature of XFS will work with samba;

as that was one of the main reasons why early linux based nas solutions
used xfs there should be no problems.

> In that case, reiserfs probably has a slight edge[...]

remember checking for reiserfs/nfs problems. The reiserfs semantics of
some file operations are slightly different than other fs (that was the
problem, that my Maildir was rather sluggish with reiserfs compared to
other fs.) but that has been known for long and should be changed by now
(just remember to use an up-to-date kernel)

> actually used it in anger (on my workstation) for a good couple of
> months and it certainly seems quicker than Ext3.

the best thing would be a benchmark dependend on your needs. Please post
your results (if you're willing to do such a benchmark) to this group
and the corresponding filesystem groups/lkml.

Andreas
-- 
26. No matter how attractive certain members of the rebellion are, there is 
    probably someone just as attractive who is not desperate to kill me. 
    Therefore, I will think twice before ordering a prisoner sent to my 
    bedchamber. --Peter Anspach's list of things to do as an Evil Overlord

Re: How do I pick the best fs?

[Milan P. Stanic]

On Thu, Mar 20, 2003 at 11:47:34AM +0000, Paul Furness wrote:
> Hmm, I can see I've opened a sacred cow... ;)

We learnt to live with it ;)
 
> Thanks, everyone, for comments and experiences. Admittedly, I still
> haven't decided, but the small amount of testing / experience I have had
> suggests that reiserfs is definately faster than Ext3. Most of these
> tests have been on my workstation, and have been far from scientific.
> However, I can't say that I've lost data with either file system. I
> haven't needed to do much with fsck, though.

ext3 driver have Hash Tree Directory indexing in kernel 2.5 which
greatly improves performances. I'm not sure but I think that it is
(or will be soon) backported to 2.4.
So we will have ext2 compatibility, speed, journaling and stability in
the *native* Linux FS, I hope. :)

Milan