Re: Offline Deduplication for Btrfs

[Mike Hommey <mh@glandium.org>]

On Thu, Jan 06, 2011 at 10:37:46AM +0100, Tomasz Chmielewski wrote:
> >I have been thinking a lot about de-duplication for a backup application
> >I am writing. I wrote a little script to figure out how much it would
> >save me. For my laptop home directory, about 100 GiB of data, it was a
> >couple of percent, depending a bit on the size of the chunks. With 4 KiB
> >chunks, I would save about two gigabytes. (That's assuming no MD5 hash
> >collisions.) I don't have VM images, but I do have a fair bit of saved
> >e-mail. So, for backups, I concluded it was worth it to provide an
> >option to do this. I have no opinion on whether it is worthwhile to do
> >in btrfs.
> 
> Online deduplication is very useful for backups of big,
> multi-gigabyte files which change constantly.
> Some mail servers store files this way; some MUA store the files
> like this; databases are also common to pack everything in big files
> which tend to change here and there almost all the time.
> 
> Multi-gigabyte files which only have few megabytes changed can't be
> hardlinked; simple maths shows that even compressing multiple files
> which have few differences will lead to greater space usage than a
> few megabytes extra in each (because everything else is
> deduplicated).
> 
> And I don't even want to think about IO needed to offline dedup a
> multi-terabyte storage (1 TB disks and bigger are becoming standard
> nowadays) i.e. daily, especially when the storage is already heavily
> used in IO terms.
> 
> 
> Now, one popular tool which can deal with small changes in files is
> rsync. It can be used to copy files over the network - so that if
> you want to copy/update a multi-gigabyte file which only has a few
> changes, rsync would need to transfer just a few megabytes.
> 
> On disk however, rsync creates a "temporary copy" of the original
> file, where it packs unchanged contents together with any changes
> made. For example, while it copies/updates a file, we will have:
> 
> original_file.bin
> .temporary_random_name
> 
> Later, original_file.bin would be removed, and
> .temporary_random_name would be renamed to original_file.bin. Here
> goes away any deduplication we had so far, we have to start the IO
> over again.

Sounds like all you need is cp --reflink=always and rsync --inplace.

Haven't tested is that works well, though.

Mike