Re: git push to amazon s3 [was: [GSoC] What is status of Git's Google Summer of Code 2008 projects?]

["Shawn O. Pearce" <spearce@spearce.org>]

Tarmigan <tarmigan+git@gmail.com> wrote:
> (trimmed cc list to folks who've touched http-push.c)
> On Mon, Jul 7, 2008 at 9:19 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
> > Using Marek's pack generation code I added support for push over
> > the dumb sftp:// and amazon-s3:// protocols, with the latter also
> > supporting transparent client side encryption.
> 
> Can you describe the s3 support that you added?  Did you do any
> locking when you pushed?  The objects and packs seem likely to be
> naturally OK, but I was worried about refs/ and especially
> objects/info/packs and info/refs (fetch over http works currently out
> of the box with publicly accessable s3 repos).

It behaves like http push does in C git in that it is pretty
transparent to the end-user:

  # Create a bucket using other S3 tools.
  # I used jets3t's cockpit tool to creat "gitney".

  # Create a configuration file for jgit's S3 client:
  #
  $ touch ~/.jgit_s3_public
  $ chmod 600 ~/.jgit_s3_public
  $ cat >>~/.jgit_s3_public
  accesskey: AWSAccessKeyId
  secretkey: AWSSecretAccessKey
  acl: public
  EOF

  # Configure the remote and push
  #
  $ git remote add s3 amazon-s3://.jgit_s3_public@gitney/projects/egit.git/
  $ jgit push s3 refs/heads/master
  $ jgit push --tags s3

  # Future incremental updates are just as easy
  #
  $ jgit push s3 refs/heads/master

  (or)
  $ git config remote.s3.push refs/heads/master
  $ jgit push s3

This is now cloneable[*1*]:

  $ git clone http://gitney.s3.amazonaws.com/projects/egit.git

Pushes are incremental, rather than the approach you outlined, as
that causes a full re-upload of the repository.  Consequently there
is relatively little bandwidth usage during subsequent pushes.

A jgit amazon-s3 URL is organized as:

  amazon-s3://$config@$bucket/$prefix

where:

  $config = path to configuration in $GIT_DIR/$config or $HOME/$config
  $bucket = name of the Amazon S3 bucket holding the objects
  $prefix = prefix to apply to all objects, implicitly ends in "/"


Amazon S3 atomically replaces a file, but offers no locking support.
Our crude remote VFS abstraction offers two types of file write
operations:

  - Atomic write for small (in-memory) files <~1M
  - Stream write for large (e.g. pack) files >~1M

In the S3 implementation both operations are basically the same
code, since even large streams are atomic updates.  But in sftp://
our remote VFS writes to a "$path.lock" for the atomic case and
renames to "$path".  This is not the same as a real lock, but it
avoids readers from seeing an in-progress update.

We are very carefully to order the update operations to try and
avoid any sorts of race conditions:

  - Delete loose refs which are being deleted.
  - Upload new pack:
    - If same pack already exists:
      - (atomic) Remove it from objects/info/packs
      - Delete .idx
      - Delete .pack
    - Upload new .pack
    - Upload new .idx
    - (atomic) Add to front of objects/info/packs.
  - (atomic) Create/update loose refs.
  - (atomic) Update (if necessary) packed-refs.
  - (atomic) Update info/refs.

Since we are pushing over a dumb transport we assume readers
are pulling over the same dumb transport and thus rely upon the
objects/info/packs and info/refs files to obtain the listing of
what is available.  This isn't true though for jgit's sftp://
and amazon-s3:// protocols as both support navigation of the
objects/packs and refs/{heads,tags} tree directly.

Locking on S3 is difficult.  Multi-object writes may not sync
across the S3 cluster immediately.  This means you can write to A,
then to B, then read A and see the old content still there, then
seconds later read A again and see the new content suddenly arrive.
It all depends upon when the replicas update and which replica
the load-balancer sends you into during the request.  So despite
our attempts to order writes to S3 it is still possible for an S3
write to appear "late" and for a client to see a ref in info/refs
for which the corresponding pack is not listed in object/info/packs.

However, this is the same mirroring problem that kernel.org has for
its git trees.  I believe they are moved out to the public mirrors
by dumb rsync and not some sort of smart git-aware transport.
As rsync is free to order the writes out of order kernel.org has
the same issue.  ;-)

Actually I suspect the S3 replica update occurs more quickly than
the kernel.org mirrors update, so the window under which a client
can see out-of-order writes is likely smaller.


*1* You need a bug fix in jgit to correctly initialize HEAD during
    push to a new, non-existant repository stored on S3.  The patch
    is going to be posted later this evening, its still in my tree.

-- 
Shawn.