Re: [PATCH/RFC v2] git-p4: stream from perforce to speed up clones

[Pete Wyckoff <pw@padd.com>]

luke@diamand.org wrote on Sat, 11 Jul 2009 09:31 +0100:
> Change commit() to stream data from Perforce and into fast-import
> rather than reading into memory first, and then writing out. This
> hugely reduces the memory requirements when cloning non-incrementally.
> 
> Signed-off-by: Luke Diamand <luke@diamand.org>
> ---
> I've modified git-p4 so that it streams/pipes data into fast-import rather
> that reading everything into memory first. The old scheme meant that
> for a large repository (mine is around 2G) my PC just grinds to a
> halt and never actually finishes. With this change it takes around ten
> minutes.
> 
> This is a resend of a patch I sent earlier, which my MUA helpfully
> managed to word-wrap.

This is definitely good stuff.  While I'd like to get Daniel's
native transport code working in my environment, I had a need to
import some big trees quickly from p4 to git.  Usually I resort to a
rare big-memory machine to do the git-p4 import, but with your
change, I can use my laptop without going into swap or
out-of-memory.

For the content, I would recommend some changes.  Here are some
patches on top of yours:

    [PATCH 1/5] git-p4 stream: remove unused function
    [PATCH 2/5] git-p4 stream: do not pass branchPrefixes so much
    [PATCH 3/5] git-p4 stream: show relative path in debug messages
    [PATCH 4/5] git-p4 stream: check apple file type
    [PATCH 5/5] git-p4 stream: use existing p4CmdList with callback

They clean up the code and change your main iterator loop to reuse
the existing p4CmdList.

I'd suggest you merge these into your patch and send it back out.
It's a worthwhile change and we should get Simon to review it.

		-- Pete