Re: Data Deduplication with the help of an online filesystem check

[Bron Gondwana <brong@fastmail.fm>]

On Tue, Apr 28, 2009 at 04:58:15PM -0400, Chris Mason wrote:
> >         Assuming a 4 kbyte block size that would mean for a 1 Tbyte
> >         filesystem:
> > 
> >         1Tbyte / 4096 / 8 = 32 Mbyte of memory (this should of course
> >         be saved to disk from time to time and be restored on startup).
> > 
> 
> Sorry, a 1TB drive is teeny, I don't think a bitmap is practical across
> the whole FS.

32Mbyte of memory is teeny too.  You can pick up a server with 32Gb for
about $5k these days.  Using 10% of that gives you room for 100Tb of
storage, which is getting to be a reasonable size.  More than that and
you would be thinking of bigger machines anyway to have enough memory
for a decent cache.

(32Gb seems to be the nice price point, because 4Gb dimms are cheap as
chips, and 8Gb are way expensive.  We're buying 32Gb machines at the
moment)

Bron ( ok, so I don't work with giant enterprise systems.  I can see
       that there are cases where this might suck, but memory seems
       to be growing as fast as disk in the low end )