Re: Btrfs offline deduplication

[Austin S Hemmelgarn <ahferroin7@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1913 bytes --]

On 07/31/2014 07:54 PM, Timofey Titovets wrote:
> Good time of day.
> I have several questions about data deduplication on btrfs.
> Sorry if i ask stupid questions or waste you time %)
> 
> What about implementation of offline data deduplication? I don't see
> any activity on this place, may be i need to ask a particular person?
> Where the problem? May be a can i try to help (testing as example)?
> 
> I could be wrong, but as i understand btrfs store crc32 checksum one
> per file, if this is true, may be make a sense to create small worker
> for dedup files? Like worker for autodefrag?
> With simple logic like:
> if sum1 == sum2 && file_size1 == file_size2; then
> if (bit_to_bit_identical(file1,2)); then merge(file1, file2);
> This can be first attempt to implement per file offline dedup
> What you think about it? could i be wrong? or this is a horrible crutch?
> (as i understand it not change format of fs)
> 
> (bedup and other tools, its cool, but have several problem with these
> tools and i think, what kernel implementation can work better).
> 
I think there may be some misunderstandings here about some of the
internals of BTRFS.  First of all, checksums are stored per block, not
per file, and secondly, deduplication can be done on a much finer scale
than individual files (you can deduplicate individual extents).

I do think however that having the option of a background thread doing
deduplication asynchronously is a good idea, but then you would have to
have some way to trigger it on individual files/trees, and triggering on
writes like the autodefrag thread does doesn't make much sense.  Having
some userspace program to tell it to run on a given set of files would
probably be the best approach for a trigger.  I don't remember if this
kind of thing was also included in the online deduplication patches that
got posted a while back or not.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]