Re: Data De-duplication

Oliver Mattos
2008-12-11T00:18:11Z

> It would be interesting to see how many duplicate *blocks* there are
> across the filesystem, agnostic to files...
>
> Is this somthing your script does Oliver?

My script doesn't yet exist, although when created it would, yes. I was
thinking of just making a BASH script and using dd to extract 512 byte
chunks of files, pipe through md5sum and save the result in a large
index file. Next just iterate through the index file looking for
duplicate hashes.

In fact that sounds so easy I might do it right now... (only to proof
of concept stage - a real utility would probably want to be written in a
compiled language and use proper trees for faster searching)