Re: brtfs COW links and git

[Jeff King <peff@peff.net>]

On Sat, Mar 19, 2011 at 04:15:32PM -0400, Larry D'Anna wrote:

> I wish git could use COW links.  I wish I could put a large binary into git and
> have the only underlying filesystem operation be to cp --reflink and to save the
> metadata.  There are a few complications:

I have never used reflink, but my understanding is that the proposed
system call just lets us reflink one entire file. So basically the
useful points would be:

  1. on "git add", we could reflink the file into the object db

  2. on "git checkout", we could reflink the object into the working
     tree

The biggest stumbling block is that the object db does not currently
hold unadorned files. They have an object type and size header at the
beginning, and I believe even uncompressed files are stored with a zlib
header. So it would require a completely new section of the object db to
store these files (as opposed to the current loose objects and
packfiles).

Note also that during "git add" we will need to get the sha1 of the
data. So you'll still have to pull all the data from disk, though not
making a copy will save some space and time.

I'm not very knowledgeable on the current state of such things, but
is there any automatic de-duplication in btrfs? If so, does it depend on
data being at the same offsets within files?

> How does it know which files to reflink?  attributes?  a size limit?

Probably supporting both would make sense.

> What does git gc do with reflinks?

If we had a "giant literal blobs" section of the object database, we
would not want to pack those objects during a regular gc. It would kill
your reflink, but also there's just no point in copying some gigantic
file into a pack where it won't actually be delta-compressed.

> Should diff-delta be reflink-aware?  Perhaps it could query the fs for
> blocklists.

Wouldn't it just be sharing underlying data between the working tree and
the object database? How would that help us make deltas between objects?

> Before I dive into implementing this, I'd like to get your comments and advice,
> to maximize the chances of success.

I'm not exactly clear on what you want to implement.

-Peff