Re: status of inline deduplication in btrfs

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2017-08-28 06:32, Adam Borowski wrote:
> On Mon, Aug 28, 2017 at 12:49:10PM +0530, shally verma wrote:
>> Am bit confused over here, is your description based on offline-dedupe
>> here Or its with inline deduplication?
> 
> It doesn't matter _how_ you get to excessive reflinking, the resulting
> slowdown is the same.
> 
> By the way, you can try "bees", it does nearline-dedupe which is for
> practical purposes as good as fully online, and unlike the latter, has no
> way to damage your data in case of bugs (mistaken userland dedupe can at
> most make the kernel pointlessly read and compare data).
> 
> I haven't tried it myself, but what it does is dedupe using FILE_EXTENT_SAME
> asynchronously right after a write gets put into the page cache, which in
> most cases is quick enough to avoid writeout.
I would also recommend looking at 'bees'.  If you absolutely _must_ have 
online or near-online deduplication, then this is your best option 
currently from a data safety perspective.

That said, it's worth pointing out that in-line deduplication is not 
always the best answer.  In fact, it's quite often a sub-optimal answer 
compared to a combination of compression, sparse files, and batch 
deduplication.  Compression and usage of sparse files will get you about 
the same space savings most of the time as in-line deduplication (I've 
tested this on ZFS on FreeBSD using native in-line deduplication, and 
with BTRFS on Linux using bees) while using much less memory, and about 
the same amount of processor time.  In the event that you need better 
space savings than that, you're better off using batch deduplication 
because it gives you better control over when you're using more system 
resources and will often get better overall results than in-line 
deduplication.