Re: Best filesystems ?

Peter Grandi
2010-10-23T18:12:02Z

[ ... ]

>> But the ig deal here is that's not something that a filesystem
>> targeted at high bandwidth multistreaming loads should be
>> optimized for, at least by default.

> Has someone documented (recently, and correctly) what
> filesystems (XFS, ext3, ext4, reiser, JFS, VFAT etc.) are best
> for what tasks (mailboxes, database, general computing, video
> streaming etc.) ?

That's impossible. In part because it would take a lot of work,
but mostly because file system performance is so anisotropic with
storage and application profiles, and there are too many variables.
File systems also have somewhat different features. I have written
some impressions on what matters here:

http://www.sabi.co.uk/blog/0804apr.html#080415

The basic performance metric for a file systems used to be which
percent of the native performance of a hard disk it could deliver
to a single process, but currently most file systems are at 90%
or higher for most of that.

My current impressions are:

* Reiser3 is nice with the right parameters for the few cases
where there is a significant number of small files on a small
storage system accessed by a single process.

* 'ext2' is nice for small freshly loaded filesystems with not
too big files, and for MS-Windows compatibility.

* 'ext3' is good for widespread compatibility and for somewhat
"average", small or middling filesystems up to 2TB.

* 'ext4' is good for inplace upgradeability from 'ext3', and some
more scalability and features, but I can't see any real reason
why it was developed but offering an in-place upgrade path to RH
customers, given that JFS and XFS were there are fairly mature
already. I think it has a bit more scalability and performance
than 'ext3' (especially better internal parallelism).

* JFS is good for almost everything, including largish filesystems
on somewhat largish systems with lots of processes accessing
lots of files, and works equally well on 32b and 64b, is very
stable, and has a couple of nice features. Its major downside is
less care than XFS for barriers. I think that it can support
well filesystems up to 10-15TB, and perhaps beyond. It should
have been made the default for Linux for at least a decade
instead of 'ext3'.

* XFS is like JFS, and with somewhat higher scalability both as to
sizes and as to higher internal parallelism in the of multiple
processes accessing the same file, and has a couple of nice
features (mostly barrier support, but also small blocks and large
inodes). Its major limitation are internal complexity and should
only be used on 64b systems. It can support single filesystems
larger than 10-15TB, but that's stretching things.

* Lustre works well as a network parallel large file streaming
filesystem. It is however somewhat unstable and great care has to
be taken in integration testing because of that.

I currently think that JFS, XFS and Lustre cover more or less all
common workloads. I occasionally use 'ext2' or NTFS for data
exchange between Linux and MS-Windows.

There are some other interesting ones:

* UDF could be a very decent small-average sized file systems,
especially for interchange, but also general purpose use.
Implementations are a bit lacking.

* OCFS2 used in non cluster mode works well, and has pretty decent
performance, and can be used in shared-storage mode too, but it
seems still a bit too unstable.

* NILFS2 seems just the right thing for SSD based file systems,
and with a collector could be a general purpose file systems.

* GlusterFS seems quite interesting for the distributed case.

Reiser4 also looked fairly good, but it has been sort of succeeded
by BTRFS (whose author used to work on Reiser4), but BTRFS seems
like a non entirely justified successor to 'ext4', but it has also
a few nice extra features builtin that I am not sure really need to
be built in ( have the same feeling for ZFS).

Then there are traditional file systems which also have some
special interest, like OpenAFS or GFS2.

The major current issue with all of these is 'fsck' times and lack
of scalability to single-pool filesystems larger than several TB,
but doing that well is still a research issue (which hasn't stopped
happy-go-lucky people from creating much larger filesystems,
usually with XFS, "because you can", but good luck to them).