Re: Data Deduplication with the help of an online filesystem check

[Ric Wheeler <rwheeler@redhat.com>]

Thomas Glanzmann wrote:
> Hello Ric,
> 
>> (1) Block level or file level dedup?
> 
> what is the difference between the two?
> 
>> (2) Inband dedup (during a write) or background dedup?
> 
> I think inband dedup is way to intensive on ressources (memory) and also
> would kill every performance benchmark. So I think the offline dedup is
> the right way to go.

I would not categorize it as offline, but just not as inband (i.e., you can run 
a low priority background process to handle dedup).  Offline windows are 
extremely rare in production sites these days and it could take a very long time 
to do dedup at the block level over a large file system :-)

> 
>> (3) How reliably can you protect the pool of blocks? How reliably can
>> you protect the database that maps hashes to blocks?
> 
> You have to lock down the i/o requests for the blocks in question and
> compare them byte by byte anyway, just to make sure.

Yes, one advantage we had in centera was that the objects were read-only and 
whole files, so this was not an issue for us.

> 
>> (4) Can you give users who are somewhat jaded confidence in your
>> solution (this is where stats come in very handy!)
> 
> For virtual machines you can reduce the used data by 1/3. Of course it
> can blow in your face when you don't watch your physical resources
> closely.
> 
>         Thomas

1/3 is not sufficient for dedup in my opinion - you can get that with normal 
compression at the block level.

Put another way, if the baseline is bzip2 levels of compression for the block 
device, when is the complexity of dedup going to pay off?

I think that Chris already mentioned that you can (for virt OS images) also 
imagine using copy on write snapshots (of something that is mostly read-only 
like the OS system partitions).  Make 50 copy-on-write snapshots of "/" 
(excluding /home) and you have an effective compression of 98% until someone 
rudely starts writing at least :-)

ric