Re: git-p4 out of memory for very large repository

[Corey Thompson <cmtptr@gmail.com>]

On Mon, Sep 02, 2013 at 08:42:36PM +0100, Luke Diamand wrote:
> I guess you could try changing the OOM score for git-fast-import.
> 
> change /proc/<pid>/oomadj.
> 
> I think a value of -31 would make it very unlikely to be killed.
> 
> On 29/08/13 23:46, Pete Wyckoff wrote:
> >I usually just do "git p4 sync @505859".  The error message below
> >crops up when things get confused.  Usually after a previous
> >error.  I tend to destroy the repo and try again.  Sorry I don't
> >can't explain better what's happening here.  It's not a memory
> >issue; it reports only 24 MB used.
> >
> >Bizarre.  There is no good explanation why memory usage would go
> >up to 32 GB (?) within one top interval (3 sec ?).  My theory
> >about one gigantic object is debunked:  you have only the 118 MB
> >one.  Perhaps there's some container or process memory limit, as
> >Luke guessed, but it's not obvious here.
> >
> >The other big hammer is "strace".  If you're still interested in
> >playing with this, you could do:
> >
> >     strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....
> >
> >and hours later, see if something suggests itself toward the
> >end of that output file.
> >
> >		-- Pete
> 

Finally, I claim success!  Unfortunately I did not try either of the OOM
score or strace suggestions - sorry!  After spending so much time on
this, I've gotten to the point that I'm more interested in getting it to
work than in figuring out why the direct approach isn't working; it
sounds like you're both pretty confident that git is working as it
should, and I don't maintain the system I'm doing this on so I don't
doubt that there might be some artificial limit or other quirk here that
we just aren't seeing.

Anyway, what I found is that Pete's incremental method does work, I just
have to know how to do it properly!  This is what I WAS doing to
generate the error message I pasted several posts ago:

git clone //path/to/branch@<begin>,<stage1>
cd branch
git sync //path/to/branch@<stage2>
# ERROR!
# (I also tried //path/to/branch@<stage1+1>,<stage2>, same error)

Eventually what happened is that I downloaded the free 20-user p4d, set
up a very small repository with only 4 changes, and started some old
fashioned trial-and-error.  Here's what I should have been doing all
along:

git clone //path/to/branch@<begin>,<stage1>
cd branch
git sync //path/to/branch@<begin>,<stage2>
git sync //path/to/branch@<begin>,<stage3>
# and so on...

And syncing a few thousand changes every day over the course of the past
week, my git repo is finally up to the Perforce HEAD.  So I suppose
ultimately this was my own misunderstanding, partly because when you
begin your range at the original first change number the output looks
suspiciously like it's importing changes again that it's already
imported.  Maybe this is all documented somewhere, and if it is I just
failed to find it.

Thanks to both of you for all your help!
Corey