Re: Data Deduplication with the help of an online filesystem check

[Chris Mason <chris.mason@oracle.com>]

On Tue, 2009-04-28 at 07:22 +0200, Thomas Glanzmann wrote:
> Hello Chris,
> 
> > There is a btrfs ioctl to clone individual files, and this could be used
> > to implement an online dedup.  But, since it is happening from userland,
> > you can't lock out all of the other users of a given file.
> 
> > So, the dedup application would be responsible for making sure a given
> > file was not being changed while the dedup scan was running.
> 
> I see, does that mean that I can not do ,,dedup'' for files that are
> currently opened by a userland program?

No, but it does mean the dedup done from userland is racey.  Picture
this:

process A:
    create some_file # some_file matches the contents of another file

dedup proc:
    check some_file
    decide to start block dedup

process A:
     modify some_file

dedup proc:
     progress through block dedup

So, this will happily replace blocks in some_file with the dedup blocks.
But there's no way to atomically swap them.

We could create new ioctls for this, basically a variant of the clone
file ioctl that makes sure a given set of pages has a given sum (or
strict memory contents) before doing the swap.

But they don't exist yet.

-chris