resume downloads

resume downloads

[Thiago Farina]

Hi,

Is there links to discussion on this? I mean, is resume downloads a
feature that is still being considered?

Being able to download huge repos like WebKit, Linux, LibreOffice in
small parts seems like a good feature to me.

-- 
Thiago Farina

Re: resume downloads

[Junio C Hamano]

The current thinking is to model this after the "repo" tool.
Prepare a reasonably up-to-date bundle file on the server side,
add a protocol capability to advertise the URL to download that
bundle from upload-pack, and have "git clone" to pay attention to it.

Then, a "git clone" could become:

 - If the capability advertises such a prebuilt bundle, spawn "curl"
   or "wget" internally to fetch it. This can be resumed when the
   connection goes down and will grab majority of the data necessary.

 - Extract the bundle into temporary area inside .git/refs/ to help
   the next step.

 - Internally do a "git fetch" to the original server. Thanks to the
   bundle transfer that has already happened, this step will become
   a small incremental update.

 - Then prune away the temporary .git/refs/ refs that were in the
   bundle, as these are not the up-to-date refs that exist on the
   server side.

A few points that need to be considered by whoever is doing this
are:

 - Where to download the bundle, so that after killing "git clone"
   that is still in the bundle-download phase, the next invocation
   of "git clone" can notice and resume the bundle-download;

 - What kind of transfer protocols do we want to support? Is http
   and https from CDN sufficient? In other words, what exactly
   should the new capability say to point at the prebuilt bundle?

These (and probably there are several others) are not something
that "repo" does not have to worry about, but would become
issues when we try to fold this into "git clone".



On Sun, May 10, 2015 at 2:55 PM, Thiago Farina <tfransosi@gmail.com> wrote:
> Hi,
>
> Is there links to discussion on this? I mean, is resume downloads a
> feature that is still being considered?
>
> Being able to download huge repos like WebKit, Linux, LibreOffice in
> small parts seems like a good feature to me.
>
> --
> Thiago Farina
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: resume downloads

[Sitaram Chamarty]

On 05/11/2015 03:49 AM, Junio C Hamano wrote:
> The current thinking is to model this after the "repo" tool.
> Prepare a reasonably up-to-date bundle file on the server side,

<shameless plug (but not "commercial")>

For people using gitolite, the server side issues of generating a
reasonably up-to-date bundle *and* enabling it for resumable download
using rsync (with the same ssh key used to gain gitolite access), can
all be handled by gitolite.

</shameless plug>

Of course the client side issues still remain; gitolite can't help
there.

> add a protocol capability to advertise the URL to download that
> bundle from upload-pack, and have "git clone" to pay attention to it.
> 
> Then, a "git clone" could become:
> 
>  - If the capability advertises such a prebuilt bundle, spawn "curl"
>    or "wget" internally to fetch it. This can be resumed when the
>    connection goes down and will grab majority of the data necessary.
> 
>  - Extract the bundle into temporary area inside .git/refs/ to help
>    the next step.
> 
>  - Internally do a "git fetch" to the original server. Thanks to the
>    bundle transfer that has already happened, this step will become
>    a small incremental update.
> 
>  - Then prune away the temporary .git/refs/ refs that were in the
>    bundle, as these are not the up-to-date refs that exist on the
>    server side.
> 
> A few points that need to be considered by whoever is doing this
> are:
> 
>  - Where to download the bundle, so that after killing "git clone"
>    that is still in the bundle-download phase, the next invocation
>    of "git clone" can notice and resume the bundle-download;
> 
>  - What kind of transfer protocols do we want to support? Is http
>    and https from CDN sufficient? In other words, what exactly
>    should the new capability say to point at the prebuilt bundle?
> 
> These (and probably there are several others) are not something
> that "repo" does not have to worry about, but would become
> issues when we try to fold this into "git clone".
> 
> 
> 
> On Sun, May 10, 2015 at 2:55 PM, Thiago Farina <tfransosi@gmail.com> wrote:
>> Hi,
>>
>> Is there links to discussion on this? I mean, is resume downloads a
>> feature that is still being considered?
>>
>> Being able to download huge repos like WebKit, Linux, LibreOffice in
>> small parts seems like a good feature to me.
>>
>> --
>> Thiago Farina
>> --
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>