Re: Ext3 vs NTFS performance

[Valerie Henson <val_henson@linux.intel.com>]

On Fri, May 04, 2007 at 08:23:08AM -0400, Theodore Tso wrote:
> On Thu, May 03, 2007 at 02:14:52PM -0700, Valerie Henson wrote:
> 
> > I'd really like to see a generic VFS-level detection of
> > read()/write()/creat()/mkdir()/etc. patterns which could detect things
> > like "Oh, this file is likely to be deleted immediately, wait and see
> > if it goes away and don't bother sending it on to the FS immediately"
> > or "Looks like this file will grow pretty big, let's go pre-allocate
> > some space for it."  This is probably best done as a set of helper
> > functions in the usual way.
> 
> What patterns do you think means things like "this file is likely to
> be deleted immediate", or "this file will grow pretty big"?  I don't
> think there are any that would be generally valid.

I wouldn't have guessed that either, but it turns out there are:

http://www.eecs.harvard.edu/~ellard/pubs/able-usenix04.pdf

    We present evidence that attributes that are known to
    the file system when a file is created, such as its name,
    permission mode, and owner, are often strongly related
    to future properties of the file such as its ultimate size,
    lifespan, and access pattern. More importantly, we show
    that we can exploit these relationships to automatically
    generate predictive models for these properties, and that
    these predictions are sufficiently accurate to enable opti-
    mizations.

For example, lock files have predictable names and permissions, and
live for a fraction of second in most cases.  Files which are appended
a few hundred bytes at a time are probably log files and will continue
to grow in this manner.  Some of their predictions were 98% accurate!

In any case, any predictive algorithms we already do at the file
system level can be done at the VFS level, and shared between file
systems, instead of being reimplemented over and over again.  Just
food for thought.

-VAL