Re: [RFC PATCH v3 0/2] Online data deduplication

[Josef Bacik <jbacik@fusionio.com>]

On Wed, May 01, 2013 at 10:27:36AM -0600, Liu Bo wrote:
> NOTE: This leads to a FORMAT CHANGE, DO NOT use it on real data!
> 
> Data deduplication is a specialized data compression technique for eliminating
> duplicate copies of repeating data.[1]
> 
> This patch set is also related to "Content based storage" in project ideas[2].
> 
> PATCH 1 is a hang fix when deduplication is on, but it's also useful with no
> deduplication in practice use.
> 
> For more implementation details, please refer to PATCH 2.
> 
> TODO:
> * a bit-to-bit comparison callback.
> 
> All comments are welcome!
> 
> [1]: http://en.wikipedia.org/wiki/Data_deduplication
> [2]: https://btrfs.wiki.kernel.org/index.php/Project_ideas#Content_based_storage
> 
> 
> v3:
>   * add COMPRESS support
>   * add a real ioctl to enable dedup feature
>   * change the maximum allowed dedup blocksize to 128k because of compressed
>     range limit
> v2:
>   * To avoid enlarging the file extent item's size, add another index key used
>     for freeing dedup extent.
>   * Freeing dedup extent is now like how we delete checksum.
>   * Add support for alternative deduplicatin blocksize larger than PAGESIZE.
>   * Add a mount option to set deduplication blocksize.
>   * Add support for those writes that are smaller than deduplication blocksize.
> 
> =====================
> HOW To turn deduplication on:
> 
> There are 2 steps you need to do before using it,
> 1) mount /dev/disk /mnt_of_your_btrfs -o dedup
>    (or mount /dev/disk /mnt_of_your_btrfs -o dedup_bs=128K)
> 2) btrfs filesystem dedup-register /mnt_of_your_btrfs
> =====================
> 

You didn't use an INCOPMAT option for this so you need to deal with a user
mounting the file system with an older kernel or even forgetting to use mount -o
dedup.  Otherwise your dedup tree will become out of date and you could corrupt
peoples data.  So if you aren't going to use an INCOMPAT flag you need to at
least use a COMPAT flag so we know the option has been used at all and then you
need to have a mechanism to know if you need to invalidate the hash tree.

Users are also going to make the mistake of thinking dedup will make their
workload awesome, and when it doesn't they need a way to turn it off.  If you do
an INCOMPAT option then you need to have a way to delete the hash tree and unset
the INCOMPAT flag.  If you do the COMPAT route then you get this for free since
the user just needs to stop using -o dedup, but you'll probably also want to
provide a mechanism to delete the tree to free up space.  Thanks,

Josef