Re: Btrfs offline deduplication

[Austin S Hemmelgarn <ahferroin7@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2524 bytes --]

On 08/01/2014 02:55 PM, Mark Fasheh wrote:
> On Fri, Aug 01, 2014 at 10:16:08AM -0400, Austin S Hemmelgarn wrote:
>> On 2014-08-01 09:23, David Sterba wrote:
>>> On Fri, Aug 01, 2014 at 06:17:44AM -0400, Austin S Hemmelgarn wrote:
>>>> I do think however that having the option of a background thread doing
>>>> deduplication asynchronously is a good idea, but then you would have to
>>>> have some way to trigger it on individual files/trees, and triggering on
>>>> writes like the autodefrag thread does doesn't make much sense.  Having
>>>> some userspace program to tell it to run on a given set of files would
>>>> probably be the best approach for a trigger.  I don't remember if this
>>>> kind of thing was also included in the online deduplication patches that
>>>> got posted a while back or not.
>>>
>>> IIRC the proposed implementation only merged new writes with existing
>>> data.
>>>
>>> For the out-of-band ("off-line") dedup there's bedup
>>> (https://github.com/g2p/bedup) or Mark's duperemove tool
>>> (https://github.com/markfasheh/duperemove) that work on a set of files.
>>>
>> Something kernel-side to do the work asynchronously would be nice,
>> especially if it could leverage the check-sums that BTRFS already stores
>> for the blocks.  Having a userspace interface for offline deduplication
>> similar to that for scrub operations would even better.
> 
> Why does this have to be kernel side? There's userspace software already to
> dedupe that can be run on a regular basis. Exporting checksums is a
> differnet story (you can do that via ioctl) but running the dedupe software
> itself inside the kernel is exactly what we want to avoid by having the
> dedupe ioctl in the first place.
> 	--Mark
> 
> --
> Mark Fasheh
> 
Based on the same logic however, we don't need scrub to be done kernel
side, as it wouldn't take but one more ioctl to be able to tell it which
block out of a set to treat as valid.  I'm not saying that things need
to be done in the kernel, but duperemove doesn't use the ioctl interface
even if it exists, and bedup is buggy as hell (unless it's improved
greatly in the last two weeks), and neither of them is at all efficient.
 I do understand that this isn't something that is computationally
simple (especially on x86 with it's defficiency of registers), but rsync
does almost the same thing for data transmission over the network, and
it does so seemingly much more efficiently than either option available
at the moment.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]