Re: Why is dedup inline, not delayed (as opposed to offline)? Explain like I'm five pls.

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2016-01-16 13:07, Rich Freeman wrote:
> On Sat, Jan 16, 2016 at 9:10 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>> Al posted on Sat, 16 Jan 2016 12:27:16 +0000 as excerpted:
>>
>>> Is there any urgency for dedup? What's wrong with storing the hash on
>>> disk with the block and having a separate process dedup the written data
>>> over time;
>>
>> There's actually uses for both inline and out-of-line[1] aka delayed
>> dedup.  Btrfs already has a number of independent products doing various
>> forms of out-of-line dedup, so what's missing and being developed now is
>> the inline dedup option, which being directly in the write processing,
>> must be handled by btrfs itself -- it can't be primarily done by third
>> parties with just a few kernel calls, like out-of-line dedup can.
>
> Does the out-of-line dedup option actually utilize stored hashes, or
> is it forced to re-read all the data to compute hashes?  If it is
> collecting checksums/etc is this done efficiently?
AFAIK, duperemove has the option to store block hashes in a database to 
save them between runs (I'm pretty sure that it invalidates hashes if 
the file containing the block changed, but I'm not certain).
>
> I think he is actually suggesting a hybrid approach where a bit of
> effort is done during operations to greatly streamline out-of-line
> deduplication.  I'm not sure how close we are to that already, or if
> any room for improvement remains.
There isn't any implementation I know of that does this.  In theory, it 
would be pretty easy if we could somehow get block checksums  from BTRFS 
in userspace.