Incremental object transfer

Incremental object transfer

[Enrico Weigelt]

Hi folks,


as already said some time ago, I'm using git to back up maildirs
on a machine w/ relatively low ram. The biggest problem for now
is the initial push (maybe later larger subsequent pushes could
be also affected too): it takes quite a long time to get everything
packed, and if the connection breaks (the box is sitting behind
a dynamic-IP DSL link), everything has to be restarted :(

So my idea is to incrementally transfer objects in smaller packs,
disable gc on remote side and update refs when some commit is
complete.

Is there any way to do this ?


thx
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: Incremental object transfer

[Jonathan Nieder]

Enrico Weigelt wrote:

> as already said some time ago, I'm using git to back up maildirs
> on a machine w/ relatively low ram. The biggest problem for now
> is the initial push (maybe later larger subsequent pushes could
> be also affected too): it takes quite a long time to get everything
> packed, and if the connection breaks (the box is sitting behind
> a dynamic-IP DSL link), everything has to be restarted :(

Maybe use "git bundle" and then transfer by rsync?

Re: Incremental object transfer

[Jakub Narebski]

Enrico Weigelt <weigelt@metux.de> writes:

> As already said some time ago, I'm using git to back up maildirs
> on a machine w/ relatively low ram. The biggest problem for now
> is the initial push (maybe later larger subsequent pushes could
> be also affected too): it takes quite a long time to get everything
> packed, and if the connection breaks (the box is sitting behind
> a dynamic-IP DSL link), everything has to be restarted :(
> 
> So my idea is to incrementally transfer objects in smaller packs,
> disable gc on remote side and update refs when some commit is
> complete.
> 
> Is there any way to do this ?

This would work only for "dumb" transports: "dumb" HTTP transport and
deprecated rsync transport.

"Smart" transport (e.g. git://, SSH, new "smart" HTTP transport) all
create packs to send on the fly.  Those packs would rarely be
byte-for-byte the same, even if both server and client have the same
objects, unless perhaps on single code (unthreaded).

There were discussion about resumable download, by examining what got
downloaded in partial pack; but this is a hard problem.

So I would recommend either creating a bundle (see git-bundle
manpage), which can be resumably downloaded via HTTP, FTP, P2P - it is
an ordinary file.  Or you can try cloning via rsync (but the
repository should be quiescent).

-- 
Jakub Narebski
Poland
ShadeHawk on #git