git not resuming push

git not resuming push

[Paul Brossier]

Hi!

I'm trying to upload large amount of data from a connection that often
cuts.

If the connection fails after uploading part of the data, it seems I
need to start over from zero again. Is there a way to resume the upload
instead?

Here is an example:

$ git push
Counting objects: 504, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (475/475), done.
Write failed: Broken pipe1/476), 533.78 MiB | 60 KiB/s
fatal: The remote end hung up unexpectedly
error: pack-objects died of signal 13
error: failed to push some refs to 'myhost.org:/git/myrepo/'

$ git push
Counting objects: 504, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (475/475), done.
Writing objects:  12% (61/476), 89.34 MiB | 23 KiB/s

Thanks, piem

Re: git not resuming push

[Jeff King]

On Tue, Nov 22, 2011 at 08:58:56PM -0700, Paul Brossier wrote:

> If the connection fails after uploading part of the data, it seems I
> need to start over from zero again. Is there a way to resume the upload
> instead?

No, there isn't a way to resume just using push.

If you have shell access on the server, one workaround you can do is to
create a bundle with the commits in question, upload it via some
resumable protocol (like sftp, http, rsync, etc), possibly taking many
attempts, and then fetch the result on the server side from the bundle
into the repository.

See "git help bundle" for some examples.

-Peff

Re: git not resuming push

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Tue, Nov 22, 2011 at 08:58:56PM -0700, Paul Brossier wrote:
>
>> If the connection fails after uploading part of the data, it seems I
>> need to start over from zero again. Is there a way to resume the upload
>> instead?
>
> No, there isn't a way to resume just using push.
>
> If you have shell access on the server, one workaround you can do is to
> create a bundle with the commits in question, upload it via some
> resumable protocol (like sftp, http, rsync, etc), possibly taking many
> attempts, and then fetch the result on the server side from the bundle
> into the repository.
>
> See "git help bundle" for some examples.

Another possibility, if it is the connection between you and the other
side that is the problem, is to chunk your push in smaller pieces. That
is, if you are trying to push out v3.0, you first push only to v1.0, then
to v2.0, and then finally to v3.0.

Peff, by the way, wouldn't this request reminds of us of a scenario we
discussed recently, which I said I would imagine would be common while you
dismissed as not likely to be common?

Re: git not resuming push

[Paul Brossier]

ok, that, with a bit of patience, should help!

thank you both for your answers,

piem

On 22/11/2011 23:24, Junio C Hamano wrote:
> Jeff King<peff@peff.net>  writes:
>
>> On Tue, Nov 22, 2011 at 08:58:56PM -0700, Paul Brossier wrote:
>>
>>> If the connection fails after uploading part of the data, it seems I
>>> need to start over from zero again. Is there a way to resume the upload
>>> instead?
>>
>> No, there isn't a way to resume just using push.
>>
>> If you have shell access on the server, one workaround you can do is to
>> create a bundle with the commits in question, upload it via some
>> resumable protocol (like sftp, http, rsync, etc), possibly taking many
>> attempts, and then fetch the result on the server side from the bundle
>> into the repository.
>>
>> See "git help bundle" for some examples.
>
> Another possibility, if it is the connection between you and the other
> side that is the problem, is to chunk your push in smaller pieces. That
> is, if you are trying to push out v3.0, you first push only to v1.0, then
> to v2.0, and then finally to v3.0.
>
> Peff, by the way, wouldn't this request reminds of us of a scenario we
> discussed recently, which I said I would imagine would be common while you
> dismissed as not likely to be common?
>
>

Re: git not resuming push

[Jeff King]

On Tue, Nov 22, 2011 at 10:24:12PM -0800, Junio C Hamano wrote:

> Another possibility, if it is the connection between you and the other
> side that is the problem, is to chunk your push in smaller pieces. That
> is, if you are trying to push out v3.0, you first push only to v1.0, then
> to v2.0, and then finally to v3.0.
> 
> Peff, by the way, wouldn't this request reminds of us of a scenario we
> discussed recently, which I said I would imagine would be common while you
> dismissed as not likely to be common?

I didn't say it wasn't common; I said I didn't think it was the majority
case, and that I didn't want a solution that only helped this case
without covering the other cases.

Also, it isn't clear to me that an immediate-reconnect scheme would
work. When the connection gets dropped, is it simply that TCP connection
that goes out, and another one could immediately be made to resume? Or
is the network actually out (even if only for a few seconds)?

-Peff