Re: Data Deduplication with the help of an online filesystem check

Edward Shishkin
2009-04-28T17:29:19Z

Tomasz Chmielewski wrote:
> Thomas Glanzmann schrieb:
>
>> 300 Gbyte of used storage of several productive VMs with the following
>> Operatings systems running:
>> \begin{itemize}
>> \item Red Hat Linux 32 and 64 Bit (Release 3, 4 and 5)
>> \item SuSE Linux 32 and 64 Bit (SLES 9 and 10)
>> \item Windows 2003 Std. Edition 32 Bit
>> \item Windows 2003 Enterprise Edition 64 Bit
>> \end{itemize}
>> \begin{tabular}{r|r|r|l}
>> blocksize & Deduplicated Data \\
>> \hline
>> 128k & 29.9 G \\
>> 64k & 41.3 G \\
>> 32k & 59.2 G \\
>> 16k & 82 G \\
>> 8k & 112 G \\
>> \
>>
>> Bottom line with 8 K blocksize you can get more than 33% of deduped data
>> running a productive set of VMs.
>
> Did you just compare checksums,

I wouldn't rely on crc32: it is not a strong hash,
Such deduplication can lead to various problems,
including security ones.

> or did you also compare the data "bit after bit" if the checksums
> matched?