Re: Data Deduplication with the help of an online filesystem check

[jim owens <jowens@hp.com>]

Andrey Kuzmin wrote:
> On Tue, Apr 28, 2009 at 2:02 PM, Chris Mason <chris.mason@oracle.com> wrote:
>> On Tue, 2009-04-28 at 07:22 +0200, Thomas Glanzmann wrote:
>>> Hello Chris,
>>>
>>>> There is a btrfs ioctl to clone individual files, and this could be used
>>>> to implement an online dedup.  But, since it is happening from userland,
>>>> you can't lock out all of the other users of a given file.
>>>> So, the dedup application would be responsible for making sure a given
>>>> file was not being changed while the dedup scan was running.
>>> I see, does that mean that I can not do ,,dedup'' for files that are
>>> currently opened by a userland program?
>> No, but it does mean the dedup done from userland is racey.  Picture
> 
> Race disappears if (background) dedupe is run against snapshot(s).

only if the snapshots are made read-only.

btrfs has writable snapshots.

jim

> 
> Regards,
> Andrey
> 
>> this:
>>
>> process A:
>>    create some_file # some_file matches the contents of another file
>>
>> dedup proc:
>>    check some_file
>>    decide to start block dedup
>>
>> process A:
>>     modify some_file
>>
>> dedup proc:
>>     progress through block dedup
>>
>> So, this will happily replace blocks in some_file with the dedup blocks.
>> But there's no way to atomically swap them.
>>
>> We could create new ioctls for this, basically a variant of the clone
>> file ioctl that makes sure a given set of pages has a given sum (or
>> strict memory contents) before doing the swap.
>>
>> But they don't exist yet.
>>
>> -chris
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>