Re: Data De-duplication

[Oliver Mattos <oliver.mattos08@imperial.ac.uk>]

> It would be interesting to see how many duplicate *blocks* there are
> across the filesystem, agnostic to files...
> 
> Is this somthing your script does Oliver?

My script doesn't yet exist, although when created it would, yes.  I was
thinking of just making a BASH script and using dd to extract 512 byte
chunks of files, pipe through md5sum and save the result in a large
index file.  Next just iterate through the index file looking for
duplicate hashes.

In fact that sounds so easy I might do it right now...  (only to proof
of concept stage - a real utility would probably want to be written in a
compiled language and use proper trees for faster searching)