git clone out of memory. alternatives?

git clone out of memory. alternatives?

[Jeffrey Chang]

Hello,

I am working with a large-ish git repository set up as a shared
repository.  I am the only user of this repository, and I access it
over ssh.

I am running into a problem where I can no longer clone the repository
on a new computer.  I get a message that seems to indicate that the
machine with the repository is running out of memory.

xigua:~/remotecvs jchang$ git clone [...]
Initialized empty Git repository in [...]
remote: Generating pack...
remote: Done counting 9122 objects.
remote: Deltifying 9122 objects...
error: git-upload-pack: git-pack-objects died with error.
fatal: git-upload-pack: aborting due to possible repository corruption
on the remote side.
remote: fatal: out of memoryremote:
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed
fetch-pack from [...] failed.
xigua:~/remotecvs jchang$

Moving the repository to another machine is not an option, nor is
adding more RAM to that machine.

Is there any way to get around this problem?  For example:
- Can I run git clone in a way that uses less memory, such as cloning
a piece of the repository at a time?
- Can I export the entire repository as a file that can be loaded on
my target machine, like "svnadmin dump" for subversion?
- Can I just rsync the repository from another computer that already has a copy?
- Are there any other work-arounds to set up a copy of the repository
on my local machine?

Thanks,
Jeff

Re: git clone out of memory. alternatives?

[Linus Torvalds]

On Thu, 27 Mar 2008, Jeffrey Chang wrote:
> 
> Is there any way to get around this problem?  For example:
> - Can I run git clone in a way that uses less memory, such as cloning
> a piece of the repository at a time?

Cloning really is pretty memory-intensive, because it involves going 
through every single object.

What you *can* do is to limit cloning to the rsync protocol, which is 
strictly quite horrible (none of the inherent sanity-checks of the native 
protocol), but it avoids the server-side costs.

However, you'll eventually hit other problems, like the fact that you also 
won't be able to do a full repack on the server side (because a full 
repack does the same thing).

> - Can I export the entire repository as a file that can be loaded on
> my target machine, like "svnadmin dump" for subversion?

That's essentially what a repack does. See above about the problem.

> - Can I just rsync the repository from another computer that already has a copy?

Yes. You can. And you can also repack on another host and then rsync the 
results back to the server. It's not pretty, but it should work.

		Linus

Re: git clone out of memory. alternatives?

[Johannes Schindelin]

Hi,

On Thu, 27 Mar 2008, Jeffrey Chang wrote:

> I am working with a large-ish git repository set up as a shared 
> repository.  I am the only user of this repository, and I access it over 
> ssh.
> 
> I am running into a problem where I can no longer clone the repository 
> on a new computer.  I get a message that seems to indicate that the 
> machine with the repository is running out of memory.

AFAICT the config variables pack.* (see config.txt) allow to tune the 
packing, even for cloning.

Hth,
Dscho