Re: Offline Deduplication for Btrfs

[Gordan Bobic <gordan@bobich.net>]

Tomasz Chmielewski wrote:
>> I have been thinking a lot about de-duplication for a backup application
>> I am writing. I wrote a little script to figure out how much it would
>> save me. For my laptop home directory, about 100 GiB of data, it was a
>> couple of percent, depending a bit on the size of the chunks. With 4 KiB
>> chunks, I would save about two gigabytes. (That's assuming no MD5 hash
>> collisions.) I don't have VM images, but I do have a fair bit of saved
>> e-mail. So, for backups, I concluded it was worth it to provide an
>> option to do this. I have no opinion on whether it is worthwhile to do
>> in btrfs.
> 
> Online deduplication is very useful for backups of big, multi-gigabyte 
> files which change constantly.
> Some mail servers store files this way; some MUA store the files like 
> this; databases are also common to pack everything in big files which 
> tend to change here and there almost all the time.
> 
> Multi-gigabyte files which only have few megabytes changed can't be 
> hardlinked; simple maths shows that even compressing multiple files 
> which have few differences will lead to greater space usage than a few 
> megabytes extra in each (because everything else is deduplicated).
> 
> And I don't even want to think about IO needed to offline dedup a 
> multi-terabyte storage (1 TB disks and bigger are becoming standard 
> nowadays) i.e. daily, especially when the storage is already heavily 
> used in IO terms.
> 
> 
> Now, one popular tool which can deal with small changes in files is 
> rsync. It can be used to copy files over the network - so that if you 
> want to copy/update a multi-gigabyte file which only has a few changes, 
> rsync would need to transfer just a few megabytes.
> 
> On disk however, rsync creates a "temporary copy" of the original file, 
> where it packs unchanged contents together with any changes made. For 
> example, while it copies/updates a file, we will have:
> 
> original_file.bin
> .temporary_random_name
> 
> Later, original_file.bin would be removed, and .temporary_random_name 
> would be renamed to original_file.bin. Here goes away any deduplication 
> we had so far, we have to start the IO over again.

You can tell rsync to either modify the file in place (--inplace) or to 
put the temp file somewhere else (--temp-dir=DIR).

Gordan