Re: "Appending" data to the middle of a file using btrfs-specific features

[Nirbheek Chauhan <nirbheek.chauhan@gmail.com>]

On Tue, Dec 7, 2010 at 1:05 AM, Freddie Cash <fjwcash@gmail.com> wrote:
> On Mon, Dec 6, 2010 at 11:14 AM, Nirbheek Chauhan
> <nirbheek.chauhan@gmail.com> wrote:
>> As an aside, my primary motivation for this was that doing an
>> incremental backup of things like git bare repositories and database=
s
>> using btrfs subvolume snapshots is expensive w.r.t. disk space. Even
>> though rsync calculates a binary delta before transferring data, it
>> has to write everything out (except if just appending). So in that
>> case, each "incremental" backup is hardly so.
>
> Since btrfs is Copy-on-Write, have you experimented with --inplace on
> the rsync command-line? =C2=A0That way, rsync writes the changes "ove=
r-top"
> of the existing file, thus allowing btrfs to only write out the block=
s
> that have changed, via CoW?
>
> We do this with our ZFS rsync backups, and found disk usage to go way
> down over the default "write out new data to new file, rename overtop=
"
> method that rsync uses.
>
> There's also the --no-whole-file option which causes rsync to only
> send delta changes for existing files, another useful feature with Co=
W
> filesystems.
>

I had tried the --inplace option, but it didn't seem to do anything
for me, so I didn't explore that further. However, after following
your suggestion and retrying with --no-whole-file, I see that the
behaviour is quite different! It seems that --whole-file is enabled by
default for local file transfers, and so --inplace had no effect.

But the behaviour of --inplace is not entirely to write out *only* the
blocks that have changed. From what I could make out, it does the
following:

(1) Calculate a delta b/w the src and trg files
(2) Seek to the first difference in the target file
(3) Start writing data

I'm glossing over the final step because I didn't look deeper, but I
think you can safely assume that after the first difference, all data
is rewritten. So this is halfway between "rewrite the whole file" and
"write only the changed bits into the file". It doesn't actually use
any CoW features from what I can see. There is lots of room for btrfs
reflinking magic. :)

Note that I tested this behaviour on a btrfs partition with a vanilla
rsync-3.0.7 tarball; the copy you use with ZFS might be doing some CoW
magic.

Thanks for the tip!

--=20
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html