Re: huge filesystems

[Andreas Dilger <adilger@shaw.ca>]

On Mar 09, 2005  10:53 -0800, Dan Stromberg wrote:
> The group I work in has been experimenting with GFS and Lustre, and I did
> some NBD/ENBD experimentation on my own, described at
> http://dcs.nac.uci.edu/~strombrg/nbd.html
> 
> My question is, what is the current status of huge filesystems - IE,
> filesystems that exceed 2 terabytes, and hopefully also exceeding 16
> terabytes?

Lustre has run with filesystems up to 400TB (where it hits a Lustre limit
that should be removed shortly for a 900TB filesystem being deployed).
The caveat is that Lustre is made up of individual block devices and
filesystems of only 2TB or less in size.

> Am I correct in assuming that the usual linux buffer cache only goes to 16
> terabytes?

That is the block device limit, and also the file limit for 32-bit systems,
imposed by the size of a single VM mapping 2^32 * PAGE_SIZE.

> Does the FUSE API (or similar) happen to allow surpassing either the 2T or
> 16T limits?

Some 32-bit systems (PPC?) may allow larger PAGE_SIZE and will have a
larger limit for a single VM mapping.  For 64-bit platforms there is no 
2^32 limit for page->index and this also removes the 16TB limit.

> What about the "LBD" patches - what limits are involved there, and have
> they been rolled into a Linus kernel, or one or more vendor kernels?

These are part of stock 2.6 kernels.  The caveat here is that there have
been some problems reported (with ext3 at least) for filesystems > 2TB
so I don't think it has really been tested very much.

Cheers, Andreas
--
Andreas Dilger
http://members.shaw.ca/adilger/             http://members.shaw.ca/golinux/