Re: Resumable clone/Gittorrent (again)

[Sam Vilain <sam@vilain.net>]

On 09/01/11 06:21, Luke Kenneth Casson Leighton wrote:
> On Sat, Jan 8, 2011 at 2:17 AM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>> there are a few characteristics of bittorrent pieces that i see:
>> verifiable, resumable, uniquely identifiable across peers and
>> reasonbly small in count.
>>
>> The fixed size helps peers uniquely identify any pieces by splitting
>> the whole transfer equally and indexing them in 1-dimension.
>  ok - you haven't answered the question: are the chains perfectly
> fixed identical sizes?
>
>  if so they can be slotted into the bittorrent protocol by simply
> pre-selecting the size to match.  with the downside that if there are
> a million such "chains" you now pretty much overwhelm the peers with
> the amount of processing, network traffic and memory requirements to
> maintain the "pieces" map.

I'll respond also to this sub-point.  This can be done; but instead of
doing it at the pack level, you take the list of objects between A and B
(for a fetch from A to B), order them by some deterministic order
(called the "commit reel" in the Gittorrent RFC) and then carve that
list up into chunks based on the uncompressed object sizes.

The ordering defined in the RFC is such that it is possible to create
"thin" packs for discrete ranges of commits using existing plumbing, so
that the total transfer size is relatively similar to a complete clone. 
In experiments the network overhead was found to be around 10-20% in
this way.

However I must discourage looking for "inspiration" from the Bittorrent
protocol; it reinvents many wheels unnecessarily, and contains much
shonky advice in it.  See the revision history for the gittorrent RFC
(github.com/samv/gittorrent) for the gory details.

Sam