Re: Allocation strategy - dynamic zone for small files

[Andreas Dilger <adilger@clusterfs.com>]

On Nov 14, 2006  15:19 +0000, phillip@lougher.demon.co.uk wrote:
> adilger@clusterfs.com wrote:
> > At the filesystem summit we DID find a surprising number of small files
> > even when the whole system was examined.  We discussed storing small
> > files directly in the inode along with other EAs (this would require
> > larger inodes).  This improves data locality and performance (i.e. stat
> > of the file loads the small file data into cache), though the assumption
> > is that there will be an increasing number of EAs on files in the future.
> 
> So it won't be feasible to store EAs + small file in the inode, or in the
> future it won't be feasible to store just EAs in the inode for most files?

Sorry to be unclear.  What I meant was that part of the justification for
always increasing the inode size (which will cause internal fragmentation
itself) is that the assumption is the larger inode size can be efficiently
used for a variety of EAs (small file bodies, ACLs, selinux, etc).

Additionally, it was proposed is to use this EA space to store the file
name(s) belonging to the inode, and in fact turn the inode table into the
directory itself.  That means information needed for dirent->d_type, etc
is readily available, as is readdir+ (readdir + stat) information.  For
small files it means that readdir + stat + read operations like "grep -r"
or "find ... | xargs grep" or any number of others can be done without any
seeks, which is a HUGE performance improvement.

> Are there any stats showing the current amount/size of EAs per file, and
> what it is expected to be in the future?

No, just empirical evidence.  Using the EA space for filenames to create a
directory itself adds a significant amount of EAs if files are hard linked.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.