Re: Resumable clone/Gittorrent (again) - stable packs?

[Jeff King <peff@peff.net>]

On Thu, Jan 06, 2011 at 11:33:51PM -0500, Nicolas Pitre wrote:

> Here's what I suggest:
> 
> 	cd my_project
> 	BUNDLENAME=my_project_$(date "+%s").gitbundle
> 	git bundle create $BUNDLENAME --all
> 	maketorrent-console your_favorite_tracker $BUNDLENAME
> 
> Then start seeding that bundle, and upload $BUNDLENAME.torrent as 
> bundle.torrent inside my_project.git on your server.
> 
> Now... Git clients could be improved to first check for the availability 
> of the file "bundle.torrent" on the remote side, either directly in 
> my_project.git, or through some Git protocol extension.  Or even better, 
> the torrent hash could be stored in a Git ref, such as 
> refs/bittorrent/bundle and the client could use that to retrieve the 
> bundle.torrent file through some other means.

I really like the simplicity of this idea. It could even be generalized
to handle more traditional mirrors, too. Just slice up the refs/mirrors
namespace to provide different methods of fetching some initial set of
objects. For example, upload-pack might advertise (in addition to the
usual refs):

  refs/mirrors/bundle/torrent
  refs/mirrors/bundle/http
  refs/mirrors/fetch/git
  refs/mirrors/fetch/http

and the client can decide its preferred way of getting data: a bundle by
http or by torrent, or connecting directly to some other git repository
by git protocol or http. It would fetch the appropriate ref, which would
contain a blob in some method-specific format. For torrent, it would be
a torrent file. For the others, probably a newline-delimited set of
URLs. You could also provide a torrent-magnet ref if you didn't even
want to distribute the torrent file.

And no matter what the method used, at the end you have some set of refs
and objects, and you can re-try your (now much smaller fetch). And there
are a few obvious optimizations:

  1. When you get the initial set of refs from the master, remember
     them. If the mirror actually satisfies everything you were going to
     fetch, then you don't even have to reconnect for the final fetch.

  2. You can optionally cache the mirror list, and go straight to a
     mirror for future fetches instead of checking the master. This is
     only a reasonable thing to do if the mirrors are kept up to date,
     and provide good incremental access (i.e., only actual git-protocol
     mirrors, not torrent or http file).

-Peff