Re: [PATCH] RFC: git lazy clone proof-of-concept

["Jon Smirl" <jonsmirl@gmail.com>]

On 2/10/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Sat, 9 Feb 2008, Nicolas Pitre wrote:
>
> > On Sat, 9 Feb 2008, Jan Holesovsky wrote:
> >
> > > On Friday 08 February 2008 20:00, Jakub Narebski wrote:
> > >
> > > > Both Mozilla import, and GCC import were packed below 0.5 GB. Warning:
> > > > you would need machine with large amount of memory to repack it
> > > > tightly in sensible time!
> > >
> > > As I answered elsewhere, unfortunately it goes out of memory even on 8G
> > > machine (x86-64), so...  But still trying.
> >
> > Try setting the following config variables as follows:
> >
> >       git config pack.deltaCacheLimit 1
> >       git config pack.deltaCacheSize 1
> >       git config pack.windowMemory 1g
> >
> > That should help keeping memory usage somewhat bounded.
>
> I tried that:
>
> $ git config pack.deltaCacheLimit 1
> $ git config pack.deltaCacheSize 1
> $ git config pack.windowMemory 2g
> $ #/usr/bin/time git repack -a -d -f --window=250 --depth=250
> $ du -s objects/
> 2548137 objects/
> $ /usr/bin/time git repack -a -d -f --window=250 --depth=250
> Counting objects: 2477715, done.
> fatal: Out of memory, malloc failed411764)
> Command exited with non-zero status 1
> 9356.95user 53.33system 2:38:58elapsed 98%CPU (0avgtext+0avgdata
> 0maxresident)k
> 0inputs+0outputs (31929major+18088744minor)pagefaults 0swaps
>
> Note that this is on a 2.4GHz Quadcode CPU with 3.5GB RAM.

Turning on multi-core support greatly increases the memory
consumption; at least double the single thread case.

Going over the original repository and deleting (get all copies out of
the history) those giant i18n files generated by programs than Sean
refers to would be my first step. If you have 5,000 revisions of a
10MB file I suspect it would take a huge amount of memory to pack.
Plus you have to copy all of that pointless history around.

>
> I'm retrying with smaller values, but at over 2.5 hours per try, this is
> getting tedious.
>
> Ciao,
> Dscho
>
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Jon Smirl
jonsmirl@gmail.com